ar51an / unbound-dashboard

Unbound Dashboard In Grafana With Prometheus & Loki

unbound dashboard

           

Light Theme

Summary

🔸 Unbound dashboard in Grafana
🔸 Prometheus time series database
🔸 Unbound metrics exporter in Go
🔸 Log aggregation with Loki
🔸 Unbound setup is available at unbound-redis
🔸 Refer to info.md for dashboard details and release.md for changes

Specs:

Grafana	Prometheus	Loki	Go	OS	HW
10.1.0	2.42.0	2.8.4	1.21.5	raspios-bookworm-arm64-lite	Raspberry Pi 4 Model B

Steps

  🔸 Grafana ➜ Prometheus ➜ Unbound Exporter ➜ Loki ➜ Import Dashboard

❯ Grafana

• Download:
  There are 2 versions OSS and Enterprise. OSS version is more than enough. Enterprise version installs too many extra packages (like unattended-upgrades and more). Below cmd downloads Grafana OSS for arm64.

  wget https://dl.grafana.com/oss/release/grafana_10.1.0_arm64.deb

• Install:

  sudo dpkg -i grafana_10.1.0_arm64.deb

  ℹ️ Note:
  A tweaked grafana.ini is available in the release. It reduces memory footprint, removes usage collection, stops calls to grafana server/repo and has few more optimizations. You can use grafana.ini either from the release or the default one. Default config is located at /etc/grafana/grafana.ini

• Start:

  sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

• UI:

  http://<RP-IP>:3000/
  Default user/pass ➟ admin/admin

❯ Prometheus

• Install:

  sudo apt install prometheus

• Config:
  Enable unbound-exporter scraping in prometheus. A trimmed down prometheus config, prometheus.yml is available in the release. Take a backup of existing prometheus.yml, if you are interested in the default config. Copy prometheus.yml from the release to /etc/prometheus/ dir.

  ℹ️ Note:
  Provided prometheus.yml has only unbound-exporter metric collection enabled. Default metric collection for node and prometheus exporters are removed. Scraping interval is set to 5m.

• Remove Node Exporter:
  Node exporter exports machine metrics. It is installed as part of prometheus pkg and runs as systemd service. It is not needed for unbound-dashboard. Unless you are already using it, remove node exporter. Below cmd will remove 8 node-exporter related pkgs.

  Remove:
  sudo apt --purge autoremove prometheus-node-exporter
  Disable scrape config:
  Provided prometheus.yml has node exporter scrapping config removed.

• UI:

  http://<RP-IP>:9090/

❯ Unbound Exporter

• I wrote my own exporter in Go. It is more efficient and tailored for this dashboard. A prebuilt binary (for arm64) is available in the release. Source code is available at unbound-exporter.

• Config:
  Modify Unbound config. Edit /etc/unbound/unbound.conf

  • Enable extended stats. Add option under server: tag

    extended-statistics: yes

  • Enable Unix domain socket for collecting stats. It is faster than default TCP. Add below options under remote-control: tag

    control-interface: "/var/run/unbound.sock"
control-use-cert: no

• Install:
  Copy unbound-exporter binary from the release to /usr/local/bin/ dir. Make sure it is under ownership of root and executable.

  Change ownership [If needed]:
  sudo chown root:root /usr/local/bin/unbound-exporter
  Make it executable [If needed]:
  sudo chmod +x /usr/local/bin/unbound-exporter

• Service:
  Create service to automatically run unbound-exporter at startup. Copy prometheus-unbound-exporter.service from the release to /etc/systemd/system/ dir. Make sure it is under the ownership of root. Enable and start the service.

  Change ownership [If needed]:
  sudo chown root:root /etc/systemd/system/prometheus-unbound-exporter.service
  Enable:
  sudo systemctl enable prometheus-unbound-exporter
  Start:
  sudo systemctl start prometheus-unbound-exporter
  Status:
  sudo systemctl status prometheus-unbound-exporter

  ℹ️ Note:
  Provided prometheus-unbound-exporter.service passes 2 paramaters. Blocklist file path and Unbound unix domain socket URI. Change them accordingly if you are using different path and name.

• Usage:

  unbound-exporter -h

  

❯ Loki

• Download:
  Download Loki and Promtail

  curl -O -L "https://github.com/grafana/loki/releases/download/v2.8.4/loki_2.8.4_arm64.deb"
curl -O -L "https://github.com/grafana/loki/releases/download/v2.8.4/promtail_2.8.4_arm64.deb"

• Install:

  sudo dpkg -i loki_2.8.4_arm64.deb
sudo dpkg -i promtail_2.8.4_arm64.deb

• Logging:
  Enable Unbound logging.

  • Edit /etc/unbound/unbound.conf. Add/Modify below options under server: tag

    log-replies: yes
log-tag-queryreply: yes
log-local-actions: yes
logfile: /var/log/unbound/unbound.log

    ℹ️ Note:
    Make sure verbosity: is set to 0.

  • Create log dir

    sudo mkdir /var/log/unbound
sudo chown unbound:unbound /var/log/unbound

  • Enable log rotation

    Copy file unbound from the release under logrotate dir to /etc/logrotate.d/ dir. Make sure it is under the ownership of root.

• Config:
  Replace /etc/loki/config.yml and /etc/promtail/config.yml with the config.yml files in the release under loki and promtail dirs respectively. Both should be under the ownership of root.

  ℹ️ Note:
  Provided loki config.yml is optimized to process large data set and metrics calculation for Unbound logs. It is tweaked after some thorough testing in Unbound logs specific dashboard with 6 panels. Default loki config can hardly handle 2 panels with small data set, it throws errors and timeouts with large data set and multiple requests.
  Provided promtail config.yml enables Unbound logs scrapping.

• Restart:

  sudo systemctl restart loki
sudo systemctl restart promtail

❯ Import Dashboard

• Open Grafana UI ➟ http://<RP-IP>:3000/

  • Click Data Sources under Configuration. Click Add data source select Prometheus. Add below options:

    Name ➟ Prometheus
    Default ➟ On
    Add URL ➟ http://localhost:9090
    Add Scrape interval ➟ 5m
    Hit ➟ Save & test

  • Click Data Sources under Configuration. Click Add data source select Loki. Add below options:

    Name ➟ Loki
    Add URL ➟ http://localhost:3100
    Add Maximum lines ➟ 100000
    Hit ➟ Save & test

  • Dashboard, unbound-dashboard.json is available in the release. Click Import under Dashboards. Click Upload JSON file. Select unbound-dashboard.json. Add below options:

    Folder ➟ General
    Select Prometheus ➟ Data Source
    Select Loki ➟ Data Source
    Hit ➟ Import

❯ ℹ️ Tips & Notes

• Grafana:
  How to ➟ Change grafana landing page to unbound dashboard & Switch between Dark (default) and Light theme.

  Open Grafana UI ➟ http://<RP-IP>:3000/
  Click Preferences under Configuration
  Select General/Unbound in Home Dashboard drop down
  Change UI Theme

  There is an additional panel in the dashboard at the top right, not visible in the preview. It shows unbound-exporter status and may be beneficial. If you are not interested in that simply remove it. Screenshot below:

  

• Prometheus:
  How to ➟ Remove time series (metrics) collected by prometheus instantly for fresh start & Reduce prometheus journal logging.

  Enable admin API:
  sudo nano /etc/default/prometheus
  Add at the top: ARGS="--web.enable-admin-api --log.level=warn"
  Save & Exit

  To delete all metrics of specific exporter add job_name as last argument in delete cmd:
  Delete:
  curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node"}'
  Purge:
  curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'
  Restart:
  sudo systemctl restart prometheus