High Availability

To ensure security and high availability of data, DolphinDB supports storing multiple replicas of data chunks on different servers. DolphinDB also adopts a two-phase commit (2PC) protocol to achieve strong consistency among data replicas.If data on one machine is corrupted, the database can still be accessed by visiting other machines. In cases where replicas become inconsistent due to data corruption or other issues, the system uses a recovery mechanism for replica consistency.

The number of replicas can be specified with the parameter dfsReplicationFactor in controller.cfg. Setting the number of replicas to 2 is recommended.

Metadata is stored in controllers. We can deploy multiple controllers in a cluster to ensure that metadata service is uninterrupted. All controllers in a cluster form a Raft group. One controller is the leader and the rest of the controllers are the followers. Data nodes can only interact with the leader. The leader writes logs locally and sends synchronization requests to all followers. Once a majority of nodes have synchronized the logs, the leader instructs the followers to change the status of related transactions to commited, ensuring strong consistency of metadata across controllers. If the leader is not available, a new leader is immediately elected to provide the metadata service. The Raft group can tolerate the failure of less than half of the controllers. For example, 1 cluster with 3 controllers can tolerate 1 unavailable controller, and 1 cluster with 5 controllers can tolerate 2 unavailable controllers. To enable high availability of metadata, the number of controllers is at least 3, and the number of replicas must be specified to be more than 1 with the configuration parameter dfsReplicationFactor in controller.cfg.

For stream processing scenarios, DolphinDB provides high availability of stream table s based on Raft Learner (requires pre-configuration of multi-cluster management).

A Raft Learner is a special member of the Raft consensus protocol that replicates logs from the Raft leader but does not participate in voting or leader election. Leveraging this mechanism, DolphinDB can achieve real-time replication of stream table data across clusters, enabling:

• Cross-cluster disaster recovery: Remote Learner nodes maintain a full copy of the data and can be used for recovery if the primary cluster becomes unavailable.
• Read-write isolation: Learner nodes can serve read-only stream computations or downstream consumers, reducing load on the primary cluster.
• High performance: Write operations remain confined to the primary cluster, avoiding cross-region network latency.

Traditionally, a Raft group is limited to a single cluster, and all Voter nodes (Leader and Follower) must communicate frequently. To support cross-cluster high availability, DolphinDB extends the Raft protocol as follows:

• Only Learner nodes are allowed to be deployed across clusters.
• Voter nodes (Leader and Follower) must remain within the same cluster and cannot span clusters.

This design extends the scope of high availability while preserving data consistency and performance, allowing DolphinDB to provide cross-cluster data protection and disaster recovery in stream processing scenarios.

DolphinDB APIs incorporate auto-reconnect and failover mechanisms to maintain high availability when interacting with DolphinDB server. When a data node or compute node which an API is connecting with becomes unavailable, the API will attempt to reconnect to the node. If the attempt fails, the API will automatically connect to another available data node or compute node.