Monitoring Docker Swarm with Prometheus and ELK stack.
This repository describes and publishes our setup of monitoring a Docker Swarm with the help of the ELK repository and Prometheus with it's scrapers.
As of version 2.x.x. of Prometheus they changed the user to nobody. This means that if you use a persistent data directory is has to be chmod'ed to 777. In this example this will be the directory /var/dockerdata/prometheus/data.
- Ubuntu (16.04 or higher) or RHEL host(s) or CentOS7 hosts
- Docker v1.13.1 (minimum) (Latest full run on 17.12.0-ce)
- Experimental Mode must be set to true (to be able to use "docker deploy" with compose v3 files)
- Must run in Swarm Mode
- 2 overlay networks ("monitoring" and "logging")
We have split up the monitoring into 2 basic parts:
| Service | Purpose |
|---|---|
| Prometheus | Central Metric Collecting |
| CAdvisor | Collecting Container information |
| Node-Exporter | Collecting Hardware and OS information |
| AlertManager | Sending out alerts raised from Prometheus |
| Prom2Teams | Alerter that sends alerts to MSTeams (currently DEV tag till proven) |
| Grafana | Dashboard on top of Prometheus |
| Service | Purpose |
|---|---|
| ElasticSearch | Central storage for Logdata |
| LogStash | Log formatter and processing pipeline |
| ElastAlert | Sending out alerts raised on Logs |
| Curator | Cleaning up Indices in Elasticsearch |
| Kibana | Dashboard on top of Elasticsearch |
Create the following directories:
- /var/dockerdata/prometheus
- /var/dockerdata/elasticsearch/data
- /var/dockerdata/elastalert/logs
- /var/dockerdata/alertmanager/data
- /var/dockerdata/grafana
- /var/dockerdata/curator
Host setting for ElasticSearch (Look here for more information)
$ sysctl -w vm.max_map_count=262144
Changing mod of persistent data directory (Only for Prometheus 2.x.x)
$ chmod 777 /var/dockerdata/prometheus/data
$ docker swarm init
$ docker network create -d overlay monitoring
$ docker network create -d overlay logging
Make sure to look at the compose files for the volume mappings. In this example everything is mapped to /var/dockerdata/{servicename}/{directories}. Adjust this to your own liking or create the same structure as used in this example.
| Config file | Needs to be in | Remarks |
|---|---|---|
| alertmanagerconfig.yml | /var/dockerdata/alertmanager/ | The alerts go through Slack/MS Teams. Use your Slack Key and Slack channel name and MSTeams Webhook URL for it to work |
| elastalert_supervisord.conf | /var/dockerdata/elastalert/config | - |
| elastalertconfig.yaml | /var/dockerdata/elastalert/config | - |
| prometheus.yml | /var/dockerdata/prometheus | - |
| Alert file | Needs to be in | Remarks |
|---|---|---|
| alertrules.nodes | /var/dockerdata/prometheus/rules | - |
| alertrules.task | /var/dockerdata/prometheus/rules | - |
| elastrules.error.yaml | /var/dockerdata/elastalert/rules | The alerts go through Slack. Use your Slack Key and channel name for it to work |
| File | Needs to be in | Remarks |
|---|---|---|
| action.yml | /var/dockerdata/curator | Action file for Curator |
$ docker deploy --compose-file docker-compose-logging.yml logging
$ docker deploy --compose-file docker-compose-monitoring.yml monitoring
In order to get the logs from the services/containers to Logstash you need to start them with a different logdriver.
Compose file:
logging:
driver: gelf
options:
gelf-address: "udp://127.0.0.1:12201"
tag: "<name of container for filtering in elasticsearch>"
Run command:
$ docker run \
--log-driver=gelf \
--log-opt gelf-address=udp://127.0.0.1:12201 \
--log-opt tag="<name of container for filtering in elasticsearch>" \
....
....
Basilio Vera's repo's (https://hub.docker.com/u/basi/) have been used for information. This got me a long way with building up a monitoring stack. Also using his version of Node-Exporter and some alert files so we have access to HOST_NAME and some startup alerts. He made a really nice Grafana Dashboard too which we used as a base. You can check it out here (https://grafana.net/dashboards/609).
The files are free to use and you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation.