Cross-Cluster Asynchronous Replication

Cross-cluster asynchronous replication in DolphinDB enables automatic data replication from a master cluster to one or more slave clusters, ensuring data consistency and facilitating offsite disaster recovery. This feature enhances fault tolerance and simplifies maintenance procedures.

Supported Operations

Cross-cluster replication is exclusively supported for DFS databases. Operations related to access control, tiered storage, and storage engine configuration cannot be replicated.

The following table displays functions and DDL/DML operations supported in cluster replication.

Function Operation Type
append, tableInsert APPEND, APPEND_CHUNK_GRANULARITY
delete SQL_DELETE
update SQL_UPDATE
upsert! UPSERT
dropTable DROP_TABLE
dropPartition DROP_PARTITION
dropDatabase DROP_DB
addRangePartitions ADD_RANGE_PARTITION
addValuePartitions ADD_VALUE_PARTITION
database CREATE_DOMAIN
createcreatePartitionedTable, createDimensionTable CREATE_TABLE, CREATE_PARTITIONED_TABLE
addColumn ADD_COLUMN
dropColumns DROP_COLUMN
renameTable RENAME_TABLE
truncate TRUNCATE_TABLE
replaceColumn REPLACE_COLUMN
setColumnComment SET_COLUMN_COMMENT
rename! RENAME_COLUMN

Operational Workflow

  1. Configure asynchronous replication among clusters.
  2. Start the server. Call setDatabaseForClusterReplication on a data node of the master cluster to enable asynchronous replication for a specific database. For databases with high-priority tasks, you can use setDatabaseClusterReplicationExecutionSet and set executionSet to 1, which allows processing tasks separately from other databases.
  3. If a replication task fails repeatedly (which can be checked using function getClusterReplicationMetrics) and interrupts the overall replication process, you may try to:
    • Call getSlaveReplicationStatus on the controller of the slave cluster to check the failure reasons. If a failed task cannot be resolved, call skipClusterReplicationTask to skip the task.
    • Call startClusterReplication on the controller of the slave cluster to restart asynchronous replication.
  4. To stop the asynchronous replication for all databases, call stopClusterReplication on the controllers of the master and slave clusters separately. Ongoing tasks will continue, but the master cluster will stop putting replication tasks to the queue, and the slave cluster will stop pulling tasks.
  5. To stop asynchronous replication for a specific database, call setDatabaseForClusterReplication. Ongoing tasks will continue, but no new replication tasks will be generated for that database.

Implementation

Master Cluster

  1. Once a transaction is completed following the two-phase commit protocol, its metadata is persisted to disk on the controller.

  2. The controller sends the metadata of completed transactions into a send queue, assigning each task a task ID based on its entry order.

  3. Conflict detection is performed for tasks in the send queue. Two tasks are considered conflicting if they write to or modify the same chunk. Non-conflicting tasks are assigned the same group ID, which can be executed in parallel.

Slave Cluster

  1. The slave cluster maintains an offset indicating the ID of tasks last pulled from the master cluster. The controller pulls tasks and corresponding metadata based on the offset from the send queue.

  2. Tasks organized into execution set(s). Within each set, tasks are hash-distributed into execution queues by target database. In each execution queue, tasks are sorted by group ID.

  3. Execution queues are hash-mapped to data nodes of the slave cluster. Each data node retrieves task data from master cluster and initates the task. Multiple data nodes can concurrently execute tasks from the same task group.

  4. After a transaction is completed, the controller logs the ID of the committed task in the edit log and proceeds with steps 3 to 4.

  5. The controller sends the IDs of completed tasks to the master cluster, allowing these tasks to be garbage collected from the send queue. This step occurs simultaneously with step 1.

Note:

  • If a replication task on the slave cluster fails, the cluster replication stops. You can call startClusterReplication on the controller of the slave cluster to restart asynchronous replication.

  • Online scaling is only available for master clusters. If new nodes are added to a slave cluster, the replication tasks will not be assigned to these newly added nodes until the cluster is restarted.