Best Practices for Log Monitoring in DolphinDB

A typical Loki monitoring architecture comprises three core components: Promtail, Loki, and Grafana.

• Promtail acts as the log collection agent. It reads logs from local sources (such as DolphinDB datanode), attaches metadata labels for classification, and streams logs to Loki via HTTP.
• Loki serves as the centralized log aggregator. It ingests and indexes logs based on labels rather than full-text content, enabling efficient storage and fast retrieval.
• Grafana offers visualization, query interfaces, and alerting dashboards.

The architecture diagram (Figure 1-1) illustrates the data flow: red arrows represent log ingestion, while blue arrows indicate query and alert flows.

Simulated Server Topology

The deployment environment consists of three nodes simulating a distributed DolphinDB cluster. Each node runs DolphinDB components and log monitoring services as shown below:

Software Versions

Note: Ensure Grafana is pre-installed on the monitoring node (vagrant1) before proceeding.

Before proceeding, ensure a high availability DolphinDB cluster is already deployed with multiple data nodes. Refer to High-availability Cluster Deployment or Multi-Container Deployment With Docker Compose for setup instructions.

mkdir -p /usr/local/logsCollect/loki

mkdir  /data/loki
mkdir  /data/loki/{chunks,index}

unzip loki-linux-amd64.zip -d /usr/local/logsCollect/loki

cd /usr/local/logsCollect/loki
vim config.yaml

auth_enabled: false  # Enable or disable authentication

server:
  http_listen_port: 3100  # HTTP service listening port

ingester:
  lifecycler:
    address: 10.0.0.80  # IP address of the monitoring server
    ring:
      kvstore:
        store: inmemory  # Storage backend for ring metadata. Options: inmemory, consul, etcd
      replication_factor: 1  # Sets the number of data replicas. A value of 1 disables replication.
    final_sleep: 0s  # Wait time before shutdown for graceful termination
  chunk_idle_period: 5m  # Marks a chunk as complete if it receives no logs for 5 minutes
  chunk_retain_period: 30s  # Waits 30 seconds after chunk completion before writing to storage

schema_config:
  configs:
    - from: 2024-04-01  # Effective start date of this schema configuration
      store: boltdb  # Storage backend for index data. Common values: boltdb, cassandra
      object_store: filesystem  # Object store type for chunk storage. Common values: filesystem, s3, gcs
      schema: v11  # Storage schema version used by Loki
      index:
        prefix: index_  # Prefix for index tables
        period: 168h  # Duration covered by each index table (7 days)

storage_config:
  boltdb:
    directory: /data/loki/index  # Directory path for index files
  filesystem:
    directory: /data/loki/chunks  # Directory path for chunk files

limits_config:
  enforce_metric_name: false  # Enforce requirement for metric name label in log streams
  reject_old_samples: true  # Reject log samples older than the allowed time window
  reject_old_samples_max_age: 168h  # Maximum age of accepted log samples (7 days)
  ingestion_rate_mb: 1024  # Global ingestion rate limit in MB/s
  ingestion_burst_size_mb: 2048  # Global burst ingestion limit in MB/s

chunk_store_config:
  max_look_back_period: 168h  # Maximum lookback duration for log queries (must align with index period)

table_manager:
  retention_deletes_enabled: true  # Enable automatic deletion of expired tables
  retention_period: 168h  # Log retention duration (7 days)

nohup ./loki-linux-amd64 -config.file=./config.yaml >./server.log 2>&1 &

kill -9 $(pgrep -f "loki-linux-amd64")

tail -200f server.log

mkdir -p /usr/local/logsCollect/promtail

unzip promtail-linux-amd64.zip -d /usr/local/logsCollect/promtail

cd /usr/local/logsCollect/promtail
vim promtail.yaml

server:
  http_listen_port: 9080        
  grpc_listen_port: 0           

positions:
  filename: ./positions.yaml    

clients:
  - url: http://10.0.0.80:3100/loki/api/v1/push  # Loki server URL for pushing logs

scrape_configs:
  # ucenter1
  - job_name: dolphinDB
    static_configs:
      - targets:
          - 10.0.0.80
        labels:
          job: dolphinDB
          host: 10.0.0.80
          __path__: /home/vagrant/v2.00.11.13/server/clusterDemo/log/*.log  # Match all *.log files in directory
    pipeline_stages:
      - regex:
          expression: '^(?P<ts>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d+)\s(?P<level><\w+>)\s:(?P<message>.*)$'
      - timestamp:
          source: ts
          format: 2006-01-02 15:04:05.000000
          timezone: "China/Beijing"
      - labels:
          level:
      - output:
          source: message
  - job_name: core_file_monitor
    static_configs:
      - targets:
          - 10.0.0.80
        labels:
          job: core_files
          host: 10.0.0.80
          __path__: /home/vagrant/v2.00.11.13/server/clusterDemo/log/core.*  # Match core.* files
    pipeline_stages:
      - labels:
          filename: __path__   # Extract filename from file path as label
      - output:
          source: filename     # Use filename as log content
      - limit:
          rate: 10             # Max 10 logs per second
          burst: 10            # Allow short bursts of up to 10 logs
          drop: true           # Drop logs exceeding rate limits

nohup ./promtail-linux-amd64 -config.file=./promtail.yaml >./server.log 2>&1 &

kill -9 $(pgrep -f "promtail-linux-amd64")

tail -200f server.log

[smtp]
enabled = true
host = smtp.163.com:465             # Specify the SMTP server and port (163 Mail in this example)
user = xxxxxxxxx@163.com            # Use your email address (e.g., a personal 163 Mail account)

password = xxxxxxxx                 # Use the app-specific authorization code (not your login password)
cert_file =
key_file =
skip_verify = false
from_address = xxxxxxxx@163.com     # Same as the user email above
from_name = Grafana
ehlo_identity =
startTLS_policy =

writeLogLevel(ERROR,"This is an ERROR message") 

Silence Alerts During After-Hours

To avoid unnecessary alerts outside trading hours, go to Notification policies and create a new mute timing entry. As shown in Figure 4-1, define silent periods using the following settings:

• Time range: Configure multiple entries to cover after-hours. For example, set 00:00–09:00 and 15:00–23:59.
• Days of the week: Set monday:friday to cover weekdays.
• Days of the month: Set 1:31 to include all days.
• Month: Set 1:12 to include all months.
• Years: Set 2025 to apply this configuration throughout the year.

Packet Loss Detection for High-Frequency Data Ingestion

In the SSE (Shanghai Stock Exchange) tick data stream, both stocks and funds share the same channel. Within each channel, OrderIndex and TradeIndex are expected to be continuous. A jump in sequence typically indicates a packet loss.

This can be monitored via changes in the SeqNo field. The Insight plugin already supports logging such events, and additional plugin support is under development.

To enable this alert, define a log rule in DolphinDB that outputs an error when a discontinuity is detected, as illustrated in Figure 4-2.

In Alert Rules, configure a LogQL expression to count error logs related to packet loss over a 5-minute window, with Evaluate every:10s, For:20s.

count_over_time({job="dolphinDB"} |= "wrong applseqnum" [5m])

Monitoring for Missing Stock ID or TradeDate in Real-Time Market Data

If a real-time quote record lacks either the stock ID or TradeDate, this should trigger an error log using writeLogLevel. Since these issues are reported with an ERROR level, you can set up a LogQL query to detect ERROR logs from the corresponding job. As shown in Figure 4-3:

The LogQL query is as follows, with additional parameters set to evaluate every 1 minute for a duration of 2 minutes, with an alert condition defined as WHEN last() OF A IS ABOVE 0.

count_over_time({job="dolphinDB"} |= "ERROR"[5m])

Client Connection Timeout Alert

To catch connection issues, define an alert that triggers when more than five logs containing timeout or connection failed appear within 5 minutes. The configuration is illustrated in Figure 4-4.

Metadata Recovery Log Monitoring

Trigger an alert if recovery failures exceed 10 times within 1 minute, configured as shown in Figure 4-5.

The LogQL query is as follows, with evaluation set to run every 15 seconds over a 30-second window, triggering when last() OF A IS ABOVE 10.

count_over_time({job="dolphinDB"} |~ "(?i)failed to incrementally recover chunk"[1m])

Alert on Missing Log Entries

Trigger an alert if no logs are generated by a service within 10 minutes, configured as in Figure 4-6.

The LogQL query is below, evaluated every 1 minute for 2 minutes, triggering when last() OF A IS BELOW 1.

Note:

• Ensure the query covers at least the time range from now-10m to now.
• To monitor specific services, add labels to the query, for example: count_over_time({filename="/home/vagrant/v2.00.11.13/server/clusterDemo/log/agent.log",job="dolphinDB"}[10m]). This counts log entries for the specified file (agent.log) in the past 10 minutes.

User Login Failure Monitoring

Trigger an alert if a user fails login more than 10 times within 1 hour, configured as in Figure 4-7.

The LogQL query is below:

sum by (remoteIP) (count_over_time({job="dolphinDB"} |~ "failed.*The user name or password is incorrect" | logfmt | remoteIP!="" [1h]))

This query counts login failures grouped by IP (remoteIP) and triggers alerts only when a specific IP exceeds the threshold. Adjust the query time range to cover at least now-1h to now.

This query is evaluated every 1 minute for 2 minutes, with the alert condition last() OF A IS ABOVE 10.

Out of Memory Log Monitoring

Trigger an alert if more than 2 "Out of memory" errors occur within 5 minutes, configured as in Figure 4-8.

The LogQL query is evaluated every 1 minute for 2 minutes and triggers when last() OF A IS ABOVE 2.

count_over_time({job="dolphinDB"} |= "Out of memory" [5m])

Core Dump Monitoring

Trigger an alert when a core dump file is generated.

The core_file_monitor job defined in the Promtail configuration (promtail.yaml) is specifically used to monitor the generation of core dump files, as shown in Figure 4-9.

Use the following LogQL expression, with alert evaluation every 1 minute over a 2-minute period, and the condition set to WHEN last() OF A IS ABOVE 0.

count_over_time({job="core_files"}[5m])

Shutdown Detection

Trigger an alert when a node goes offline.

Use the following LogQL query, with evaluations every 15 seconds for a 30-second window, and trigger condition WHEN last() OF A IS ABOVE 0, as shown in Figure 4-10.

Low Disk Space Monitoring

Trigger an alert when the system detects insufficient disk space.

Use the LogQL expression below, evaluated every 1 minute for 2 minutes, triggering when last() OF A IS ABOVE 0, as shown in Figure 4-11.

count_over_time({job="dolphinDB"} |~ "(?i)No space left on device" [5m])

Frequent Node Join/Leave or Network Instability Alert

Trigger an alert when a node frequently disconnects and reconnects—specifically, if more than 10 transitions occur within 3 minutes.

Use the following LogQL query, with evaluation every 15 seconds for a 30-second duration, and alert condition WHEN last() OF A IS ABOVE 10, as shown in Figure 4-12.

count_over_time({job="dolphinDB"} |~ "(?i)HeartBeatSender exception" [3m])

Promtail Timestamp Parsing Behavior

In the Loki and Promtail log monitoring architecture, if Promtail does not explicitly parse timestamps from log entries, Loki will use the ingestion time (i.e., the time the log is pushed to Loki) as the timestamp. To preserve original log timestamps, you must configure "promtail.yaml" accordingly:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: ./positions.yaml

clients:
  - url: http://10.0.0.80:3100/loki/api/v1/push  # Loki server endpoint

scrape_configs:
  - job_name: dolphinDB
    static_configs:
      - targets:
          - 10.0.0.80
        labels:
          job: dolphinDB
          host: 10.0.0.80
          __path__: /home/vagrant/v2.00.11.13/server/clusterDemo/log/*.log
    pipeline_stages:
      # Extract timestamp, log level, and message using regex
      - regex:
          expression: '^(?P<ts>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d+)\s(?P<level><\w+>)\s:(?P<message>.*)$'
      # Parse the extracted timestamp into standard format
      - timestamp:
          source: ts
          format: 2006-01-02 15:04:05.000000  # Go-style layout
          timezone: "China/Beijing"
      # Attach the log level as a label
      - labels:
          level:
      # Use the extracted message as the log output
      - output:
          source: message

After applying this configuration and restarting Promtail and Loki, you may encounter a "timestamp too old" error (see Figure 5-1).

This occurs because Promtail now uses the timestamp extracted from the log itself, and if the log is too old, Loki will reject it.

To resolve this, increase the reject_old_samples_max_age in Loki’s "limits_config" section:

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 1680h  # Increase max allowed age for logs
  ingestion_rate_mb: 1024
  ingestion_burst_size_mb: 2048

After restarting Grafana, you'll find that Loki can now filter logs using the level label (see Figure 5-2). Additionally, log entries display only a single timestamp (see Figure 5-3), indicating that the original timestamps extracted from the logs have been successfully parsed and applied. This confirms that the timestamp parsing configuration is working correctly, and Loki no longer attaches its own ingestion time.

Note: Once the level field is extracted as a label, it must be queried using label matchers instead of plain text searches. For instance, replace count_over_time({job="dolphinDB"} |= "ERROR"[5m]) with count_over_time({job="dolphinDB", level="ERROR"}[5m]). Be sure to update your alert rules accordingly.

Minimizing Alert Evaluation Delay

To improve alert responsiveness, reduce the evaluation interval and the alerting hold time (evaluation window). The configuration shown in Figure 5-4 sets the minimum recommended values, with Evaluate every: 10s (check every 10 seconds) and For: 20s (evaluate over 20 seconds before firing).

Note: The "For" duration must be at least twice the "Evaluate every" interval.For example, if Evaluate every = 1m, then "For" must be at least 2m.

In high-availability deployments of the distributed time-series database DolphinDB, log monitoring plays a vital role in ensuring system reliability and accelerating issue diagnosis. This document introduces a lightweight, efficient, and scalable logging solution built on Loki, Promtail, and Grafana, delivering a robust and cost-effective framework for log collection, analysis, and visualization tailored to DolphinDB’s operational needs.

• Promtail Linux amd64 installation package and configuration file:
  • promtail.yaml
  • promtail-linux-amd64.zip
• Loki Linux amd64 installation package and configuration file:
  • loki-linux-amd64.zip
  • config.yaml