Skip to main content

Monitoring a validator node

Monitoring​

The monitoring wiki seems outdated; especially the described grafana templates don’t work. There is a project ongoing for updated templates. We made our own template which you can find here: https://grafana.com/grafana/dashboards/16421.

The monitoring setup exists of:

  • a monitoring instance which uses grafana, loki, node-exporter and alertmanager
  • validator nodes with promtail and node-exporter from which the data, including the build-in prometheus endpoint in polkadot/kusama is send to the monitoring node.

Grafana

Setting up grafana​

The wiki uses binaries but there is a apt repository available. To install the latest grafana server

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O /usr/share/keyrings/grafana.key https://packages.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana
sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server
sudo systemctl enable grafana-server.service

If you are running behind a proxy adjust grafana config:

vi /etc/grafana/grafana.ini
[server]
domain = <domain>
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
serve_from_sub_path = true

Prometheus​

Example prometheus scrape config

global:
  scrape_interval: 60s
  evaluation_interval: 60s

rule_files:
  - "rules.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["<host>:9090"]
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['<host>:<port>']
        labels:
           name: "<host>"
      - targets: ['<host2>:<port2>']
        labels:
           name: "<host2>"
  - job_name: "validators"
    scrape_interval: 5s
    static_configs:
      - targets: ["<host>:<port>"]
        labels:
           name: "<nodename>"
           host: "<host>"
      - targets: ["<host2>:<port2>"]
        labels:
           name: "<nodename2>"
           host: "<host2>"