Grafana

Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, InfluxDB & OpenTSDB. OpenStack runs Graphite which stores all the metrics related to Nodepool and Zuul (to name a few).

At a Glance

Hosts
Projects
Bugs

Overview

Apache is configured as a reverse proxy to Grafana running in a container, listening on port 3000.

Updating dashboards

Changes to config-files in project-config will trigger the infra-prod-run-grafana job to refresh the dashboards. If the container is restarted manually, you can run /usr/local/bin/update-grafana-dashboards on grafana.opendev.org to reload directly.

Local Development

To develop dashboards, you can run the Grafana container. Firstly, get setup:

$ cd <work dir>
$ mkdir secrets
$ echo "password" > secrets/admin_password
$ echo "admin" > secrets/admin_user
$ echo "key" > secrets/secret_key

$ git clone https://opendev.org/openstack/project-config

Then run the container with the following options:

$ cd <work dir>
$ sudo podman run \
  -p 3000:3000 \
  -v ./secrets:/etc/grafana/secrets \
  -e GF_AUTH_ANONYMOUS_ENABLED=true \
  -e GF_USERS_ALLOW_SIGN_UP=false \
  -e GF_SECURITY_ADMIN_PASSWORD__FILE=/etc/grafana/secrets/admin_password \
  -e GF_SECURITY_ADMIN_USER__FILE=/etc/grafana/secrets/admin_user \
  -e GF_SECURITY_SECRET_KEY__FILE=/etc/grafana/secrets/secret_key \
 docker.io/grafana/grafana-oss

At this point, Grafana will be running and listening on port 3000. You can log into as admin with password (or using your secrets above).

This is unconfigured and does not yet talk to the OpenDev Graphite instance. The dashboard definitions are kept in project-config/grafana. You need to run the grafyaml tool to load them.

$ echo 'GRAFANA_URL=http://admin:password@localhost:3000/' > grafyaml.env
$ docker run --rm -t --network=host \
   --env-file ./grafyaml.env \
   -v ./project-config/grafana:/grafana:ro \
   opendevorg/grafyaml

To work on dashboards, update the yaml files in project-config and re-run the grafyaml import as above, then reload them in the Grafana UI.

Alternatively, you can use the Grafana editor to make the dashboards, and then under “Dashboard Settings” (gear icon) select “JSON Model” and commit that (it seems you have to cut-and-paste, there isn’t currently a way to export the JSON as such). The raw JSON can be proposed as an addition to project-config.