Grayscale Upgrade for High-availability Clusters

This tutorial introduces how to grayscale upgrade (partial update) a high-availability (HA) cluster.

For a regular upgrade, all nodes must be shut down before you upgrade a cluster. A cluster is upgraded after all nodes are updated to the new version, and during the upgrade, all services and operations are interrupted.

In a grayscale upgrade, a cluster is upgraded on a node-by-node basis with uninterrupted service. After a node is upgraded and returns to service, another node is upgraded, and so on until all nodes are upgraded. During a grayscale upgrade, the system remains online. This requires that the cluster must be deployed for high availability.

Grayscale upgrades requires in-memory data formats, including transmission protocols, serialization protocols, etc., to be fully compatible between the new and the old versions.

Therefore, before a major upgrade, it is recommended to contact our technical support for a risk assessment.

Note: When the server is upgraded to a certain version, the plugin should also be upgraded to the corresponding version.

In the following example, the cluster of version 2.00.7.4 is upgraded to version 2.00.8.12.

controller.cfg

mode=controller
localSite=10.10.11.2:8920:HActl44
dfsHAMode=Raft
dfsReplicationFactor=2
dfsReplicaReliabilityLevel=1
dataSync=1
workerNum=4
localExecutors=3
maxConnections=512
maxMemSize=8
lanCluster=0
dfsRecoveryWaitTime=30000

cluster.nodes

localSite,mode
10.10.11.1:8920:HActl43,controller
10.10.11.2:8920:HActl44,controller
10.10.11.3:8920:HActl45,controller
10.10.11.1:8921:agent43,agent
10.10.11.2:8921:agent44,agent
10.10.11.3:8921:agent45,agent
10.10.11.1:8922:node43,datanode
10.10.11.2:8922:node44,datanode
10.10.11.3:8922:node45,datanode

pnodeRun(version)

	node	value
0	node43	2.00.7.4 2022.12.08
1	node44	2.00.7.4 2022.12.08
2	node45	2.00.7.4 2022.12.08

Suppose you have a database "dfs://stock" with a DFS table "stock" in DolphinDB (which can be created with the following script).

db = database("dfs://stock",VALUE,2023.01.01..2023.01.10,,'TSDB')
tmp = table(1:0,[`time,`name,`id],[TIMESTAMP,SYMBOL,INT])
pt = db.createPartitionedTable(tmp,`stock,`time,,`name,ALL)

We use Python API to write data to the DFS table in a highly available manner during the upgrade.

import dolphindb as ddb
import pandas as pd
import datetime
import time
import random
s = ddb.session()
sites = ["192.168.100.43:8922","192.168.100.44:8922","192.168.100.45:8922"]
s.connect(host="192.168.100.43",port=8922,userid="admin",password="123456", highAvailability=True, highAvailabilitySites=sites)
appender = ddb.tableAppender(dbPath="dfs://stock",tableName='stock',ddbSession=s,action="fitColumnType")
x = ['Apple','Microsoft','Meta','Google','Amazon']
i = 1
while i<=10000000:
    now = datetime.datetime.now()
    name = random.sample(x,1)
    data = pd.DataFrame({'time':now,'name':name,'id':i})
    appender.append(data)
    print(i)
    i = i+1
    time.sleep(1)