Monitoring with Prometheus

Unzip package

The unzipped files are as follows:

demo@zhiyu:~/prometheus-2.26.0.linux-amd64$ ls
console_libraries  consoles  data  LICENSE  NOTICE  prometheus  prometheus.yml  promtool

Configure prometheus.yml

Modify the configuration file prometheus.yml:

global:
  scrape_interval:     15s 
  evaluation_interval: 15s

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 127.0.0.1:9093

rule_files:
  - "./avgLoadMonitor.yml"

scrape_configs:
  - job_name: 'DolphinDB'
    static_configs:
    - targets: ['115.239.209.122:8080','115.239.209.122:25667']

The targets in the alerting section specifies the port of Alertmanager.

The rule_files block specifies the alerting rules. The following section introduces the creation of file avgLoadMonitor.yml.

The last block scrape_configs controls what resources Prometheus monitors and the targets specifies the ip:port of DolphinDB nodes. This example monitors 2 nodes with IP address 115.239.209.122 and port numbers 8080 and 25667. You can also add a node in the format of "IP:PORT".

Create avgLoadMonitor.yml

The content of file avgLoadMonitor.yml is as follows:

groups:
- name: avgLoadMonitor
  rules:
  - alert: avgLoadMonitor
    expr: avgLoad > 0.1
    for: 15s
    labels:
      severity: 1
      team: node
    annotations:
      summary: "{{ $labels.instance }} avgLoad larger than 0.1!"

In this example, we use metrics avgLoad and set the alerting rule as avgLoad>0.1.

Unzip package

The unzipped files are as follows:

demo@zhiyu:~/alertmanager-0.21.0.linux-amd64$ ls
alertmanager  alertmanager.yml  amtool  LICENSE  NOTICE

Users can specify the alert receivers (such as email, PagerDuty, or OpsGenie) in the configuration file alertmanager.yml. Alerting rules in Prometheus servers send alerts to an Alertmanager specified in the file prometheus.yml. The Alertmanager then manages those alerts and sent out emails to the receivers.

Configure alertmanager.yml

You can refer to Configuration for the Alertmanager configuration and see the example file using email.

Start Prometheus with the following command:

demo@zhiyu:~/prometheus-2.26.0.linux-amd64$ nohup ./prometheus --config.file=prometheus.yml &

By default, you can browse to a status page about Prometheus at http://localhost:9090.

Start Alertmanager with the following command:

demo@zhiyu:~/alertmanager-0.21.0.linux-amd64$ nohup ./alertmanager --config.file=alertmanager.yml &

Alertmanager will now be reachable at http://localhost:9093. If the alerting rule is triggered in Prometheus, you can view the notifications in the above address. Then the Alertmanager will send alerts to the receivers. In this example, when the average load exceeds 0.1, alerting emails will be sent.