Deployment on Kubernetes

Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications. DolphinDB-MGR (including DolphinDB-Operator and DolphinDB-Webserver) is an automated system deployed in Kubernetes for DolphinDB cluster operation and maintenance. It provides full lifecycle management including deployment, upgrade, scaling, and configuration changes. With DolphinDB-MGR, you can run DolphinDB seamlessly on a Kubernetes cluster deployed on-premise or in public cloud.

Introduction to DolphinDB-MGR

DolphinDB runs as containerized applications in Kubernetes.

Custom Resources: The custom resource ddb is an abstraction that describes DolphinDB in Kubernetes. It defines the properties that specify DolphinDB configurations in Kubernetes.
Resource Management: The service component 'DolphinDB-Operator' manages the resources of DolphinDB in Kubernetes. It parses ddb into Pods, Services, ConfigMaps, etc., and monitors these resources in real time to make sure DolphinDB runs properly in Kubernetes.
License Authentication: A license is required for authentication as a ConfigMap resource before running DolphinDB in Kubernetes.

Quick Deployment of DolphinDB

The environment and software versions used in this tutorial are as follows:

Servers: 3 servers on Alibaba Cloud with extranet IP addresses 192.168.100.10 (master), 192.168.100.11 (worker), and 192.168.100.12 (worker).
Operating System: Linux (kernel version 3.10 and above)
Memory: 8 GB
CPU: x86_64 (8 cores)
Docker version: v20.10.12. Deployed on all servers
Helm version: v3.7.2. Deployed on all servers
Kubernetes version: v1.23.3. Deployed on all servers
Kubectl and Kubeadm version: v1.21 and above. Deployed on all servers
DolphinDB-MGR version: v1.0.0, official release. Only deployed on the master.

DolphinDB-MGR is a Helm package, which includes a collection of files describing DolphinDB resources and management components. Below are the key components:

DolphinDB-Operator: custom resources manager
DolphinDB-Webserver: web interface of cluster manager

This tutorial introduces how to create a Kubernetes cluster, deploy DolphinDB-MGR, and create a high-availability cluster with 3 controllers:

controller1  => agent1 => 1 datanode
controller2  => agent2 => 1 datanode
controller3  => agent3 => 1 datanode

Create a Kubernetes Cluster

Before creating a Kubernetes cluster, please set up Docker and Helm. Please refer to the official documentation of Docker and Helm.

Once you install Docker, please execute the following Linux command. The installation is successful if the Docker version is displayed.

docker -v

Once you install Helm, execute the following Linux command. The installation is successful if the Helm version is displayed.

helm version

Below are 2 recommended ways to set up a Kubernetes cluster.

Single-node Kubernetes cluster: see minikube start
Multi-node Kubernetes cluster: see Installing kubeadm

Once you install Kubernetes, execute the following Linux command. The installation is successful if the Kubernetes version is displayed.

kubectl version

Deploy DolphinDB-MGR

Prerequisites:

For hardware prerequisites, please see Installing kubeadm.

Software prerequisites:

A Kubernetes cluster that can be accessed using kubectl (Kubeadm in this tutorial)
Successful installation of Helm
Core dumps enabled in Kubernetes. See Enable Core Dump in Kubernetes

To verify whether the installation is successful, see Create a Kubernetes Cluster. This section will introduce the deployment process.

Deploy Local Path Provisioner

Local path provisioner can be used as a Container Storage Interface (CSI) for local paths in a Kubernetes environment to dynamically allocate PV using the node's local path.

Note: Local path provisioner only provides the user with Storage Class to create Persistent Volume Claims (PVC). It is not required for users with other types of CSI.

(1) Download local-path-provisioner from GitHub.

wget https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

Expected output:

--2022-01-12 12:05:27--  https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml                              
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.109.133, ...                
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.          
HTTP request sent, awaiting response... 200 OK                          
Length: 3451 (3.4K) [text/plain]                                         
Saving to: 'local-path-storage.yaml'                                     
100%[=================================================================================================================================================================================================>] 3,451        402B/s   in 8.6s                                                               
2022-01-12 12:09:35 (402 B/s) - 'local-path-storage.yaml' saved [3451/3451]

(2) Modify configuration:

The selected paths are for data persistence in the container (see local-path-provisioner-configuration)

Note: To modify the local path for provisioning, find the config map "local-path-config" and specify a node path in the Kubernetes cluster for the selected "paths" under "data". If more than one path was specified, the path would be chosen randomly when provisioning.

(3) Deploy in a Kubernetes environment:

kubectl apply -f  local-path-storage.yaml

Expected output

namespace/local-path-storage created
serviceaccount/local-path-provisioner-service-account created
clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
deployment.apps/local-path-provisioner created
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created

Install DolphinDB-MGR

DolphinDB-MGR is integrated into a Helm chart package "dolphindb-mgr" and stored in the Helm repository. You can install it with Helm.

(1) Add a chart repository

helm repo add dolphindb https://dolphindbit.github.io/helm-chart/

Expected output:

"dolphindb" has been added to your repositories

(2) Install DolphinDB-MGR

helm install dolphindb-mgr dolphindb/dolphindb-mgr --set global.version="v1.0.0" --set global.serviceType=NodePort --set global.storageClass=local-path --set dolphindb-webserver.nodePortIP=192.168.100.10 --set-file license.content=$licensePath/license.lic  -ndolphindb --create-namespace

A license is required to start DolphinDB. Please add license.content=$licensePath/license.lic to the command and modify the path to where your license is located.

Expected output

NAME: dolphindb-mgr                                                     
LAST DEPLOYED: Wed Jan 12 14:39:11 2022                                 
NAMESPACE: dolphindb
STATUS: deployed
REVISION: 1
TEST SUITE: None

Main Parameters:

-ndolphindb --create-namespace: deploy DolphinDB-MGR in the namespace "dolphindb" (create it if the namespace does not exist)
license.content: the path of DolphinDB License
global.serviceType=NodePort, dolphindb-webserver.nodePortIP: the service type provided by DolphinDB-MGR in Kubernetes. It can be:
- clusterIP: Expose the Service on a cluster-internal IP. The service is only accessible within the cluster.
- NodePort (default): Expose the Service on a static port of each node. The service is accessible from outside the cluster by requesting <NodeIP>:<NodePort>.
- LoadBalancer: Expose the Service externally using a cloud provider's load balancer. The service is accessible from outside the cluster through load balancer in Kubernetes.
  This tutorial adopts service of the NodePort type. Modify the parameter if you need to use other Services.
global.version: DolphinDB-MGR version (v1.0.0 in the example).

Note: The DolphinDB License must be obtained officially. If you use an invalid license, error "persistentvolumeclaim log-ddb-t3-crt-0-0 not found" will occur.

Other Parameters:


Parameters	Descriptions
`global.registry`	The container registry. The images are pulled from Docker Hub if unspecified.
`global.repository`	The image repository. Do not change it unless necessary.
`global.storageClass`	The StorageClass to provision persistent volumes. The default StorageClass is used if unspecified.
`global.serviceAccount`	The RBAC resource ServiceAccount. Do not change it unless necessary.
`global.allNamespace`	Whether DolphinDB can run in all namespaces. true means DolphinDB can be deployed and managed in all namespaces; false means it can only be deployed and managed in the current namespace.
`dolphindb-operator.replicaCount`	The number of replicas of DolphinDB-Operator
`dolphindb-operator.imageTag`	DolphinDB-Operator version. It is the same as `global.version` if unspecified.
`dolphindb-webserver.replicaCount`	The number of replicas of DolphinDB-Webserver
`dolphindb-webserver.imageTag`	DolphinDB-Webserver version. It is the same as `global.version` if unspecified.
`dolphindb-webserver.nodePortIP`	The IP address exposed by DolphinDB-Webserver. The parameter is required if `global.serviceType` is NodePort. It can be IP address of any node in the cluster.
`dolphindb.version`	The default DolphinDB server version
`dolphindb.serviceType`	The service types provided by DolphinDB in Kubernetes. See `global.serviceType`.
`dolphindb.controllerDataSize`	The default size of the persistent data stored in each DolphinDB controller.
`dolphindb.datanodeDataSize`	The default size of the persistent data stored in each DolphinDB data node.
`license.name`	The name of the ConfigMap for DolphinDB License. Do not change it unless necessary.
`license.resources`	The default CPU and memory configurations for each DolphinDB container. It should be consistent with the DolphinDB License.

(3) Check DolphinDB-MGR deployment

helm list -ndolphindb

Expected output

NAME          NAMESPACE     REVISION    UPDATED                               STATUS          CHART                   APPVERSION 
dolphindb-mgr DolphinDB     1           2022-01-12 14:39:11.493079194+0800CST deployed        dolphindb-mgr-v1.0.0    1.0.0

(4) Check if all pods are running

kubectl get pods -ndolphindb

Expected output

NAME                                   READY   STATUS    RESTARTS   AGE                                                         
dolphindb-operator-0                   1/1     Running   0          20m                                                         
dolphindb-operator-1                   1/1     Running   0          12m                                                         
dolphindb-webserver-5487785cfd-msr5w   1/1     Running   0          20m                                                         
dolphindb-webserver-5487785cfd-ns5dq   1/1     Running   0          20m

Note: DolphinDB-MGR is successfully deployed when the helm install command returns 'deployed' status and pods are running.

Deploy and Connect to DolphinDB Clusters

Web User Interface

DolphinDB-MGR provides a web user interface, which is hosted by a nodeport service by default. After successfully deploying DolphinDB-MGR, look for "dolphindb-webserver" service in Kubernetes.

kubectl -ndolphindb get svc | grep dolphindb-webserver

Output:

dolphindb-webserver   NodePort    10.109.94.68    <none>        8080:30908/TCP   43m

You can visit the following link to access the interface:

http://$nodeIP:$NodePort/dolphindb-cloud

Parameters:

$nodeIP: IP address of any node in the Kubernetes cluster.
$NodePort: Port corresponding to the DolphinDB-Webserver displayed in the terminal ("30908" in this example).

Note: If you use minikube to deploy DolphinDB-MGR, please execute the following command:

kubectl port-forward svc/dolphindb-webserver 8080 -ndolphindb

In this example, the address is http://192.168.100.10:30908/dolphindb-cloud/

Deploy DolphinDB Clusters

(1) Create a cluster

(2) Configure the cluster

Note:

The CPU, memory, and other resources allocated to the controllers and data nodes cannot exceed the resources of the servers. Otherwise, the cluster cannot be created.
There are two ways to output the system log: stdout and output to files (recommended).
The controller replicas and data node replicas indicate the number of controllers and data nodes in the cluster to be created.
Comparatively, the standard PVC (PersistentVolumeClaims) is a more flexible way but local path is easier and recommended to use. This tutorial uses local path provisioner used as the Storage Class to dynamically provision Persistent Volume (PV).
The port refers to the container port to specify the port used in LoadBalancer.

(3) Wait till cluster is successfully deployed

The cluster is successfully deployed once the status turns "Available".

(4) Connect to a DolphinDB cluster

As shown above, the <NodeIP>:<NodePort> of the controller is 192.168.100.10:31436 and that of the data node is 192.168.100.10:31470. Click the link to connect to the corresponding node.

For a newly created cluster, if you want to connect to data nodes, you need to connect to the controller first, and start data nodes in the web interface of cluster management.

Note:

Currently, the ports of NodePort Services are assigned randomly and it cannot be specified.

Upgrade DolphinDB Clusters

With DolphinDB-MGR, the rolling upgrade of DolphinDB clusters can be simplified.

To upgrade the DolphinDB clusters to version 1.30.15, execute the following command and manually modify the version to be 1.30.15.

kubectl edit ddb $ddbname -ndolphindb

As shown below, the cluster is upgraded to v1.30.15 and the status is Available.

Note:

The functionality of upgrading DolphinDB in the web user interface is under development.

Update License Files

Follow the steps to update license file for a DolphinDB cluster.

(1) Get the cluster names on K8s

$ kubectl get cm -ndolphindb

(2) Select the yaml file of the target cluster where the license needs to be upgraded (ddb_test1 in this example).

$ kubectl edit cm ddb_test1 -oyaml -ndolphindb

Replace the part circled in red as shown above with a new license and save it.

(3) Restart the pods after updating the license.

$ kubectl get pod -ndolphindb | grep ddb_test1 | awk '{print $1}' | xargs kubectl delete pod -ndolphindb

(4) Execute the following command to obtain the status of all pods.

$ kubectl get pod -ndolphindb

If all pods are running as shown above, the license file is successfully updated.

Delete Clusters

Delete Kubernetes Clusters

To delete a Kubernetes cluster, please refer to this page if you create it using Kubeadm, and this page if you create it using Minikube.

Delete DolphinDB Clusters

There are 2 ways to delete DolphinDB clusters.

Method 1: Click on the Delete button in the web-based cluster manager

Method 2: Run the following command

kubectl delete ddb $ddbName  -ndolphindb

Parameter:

$ddbName: DolphinDB cluster name

Uninstall DolphinDB-MGR

Run the following command to uninstall DolphinDB-MGR:

helm uninstall dolphindb-mgr -ndolphindb

Enable Core Dump in Kubernetes

If core dump is enabled, when a fatal or unhandled error occurs in DolphinDB, the system terminates abnormally and a core dump is generated. Execute the following command in Kubernetes to enable core dump on the host of DolphinDB container.

echo "/data/ddb/core/core.%p"  | sudo tee /proc/sys/kernel/core_pattern

/data/ddb/core is the path to save the core dumps in the container. If it conflicts with the current host configuration, specify the parameter dolphindb.coreDumpDir when deploying DolphinDB-MGR.

FAQ

(1) If a Pod is not created after creating a cluster, you can troubleshoot in the following ways:

kubectl get pod -nlphindb
kubectl describe pod $podName -ndolphindb
kubectl get statefulsets -n dolphindb
kubectl describe statefulsets -n dolphindb

(2) Pod in the Pending state is usually caused by insufficient resources. The reasons could be:

i. For a Pod using PV for storage, the StorageClass of the PVC does not exist or the capacity of PV is insufficient.

ii. You have insufficient CPUs or memories on the available nodes.

You can check the exact cause using kubectl describe pod command.

kubectl describe po -n ${namespace} ${pod_name}