Monitoring
Set of scripts for evaluating various monitoring stacks (log aggregation, metrics collection, metrics correlation, etc.)
Currently, the following stacks can be installed.
- Sensu (including Uchiwa)
- Flapjack
- Logstash (with Elasticsearch/Kibana and logstash-forwarder)
- Heka
- Sentry
- Graylog2 (including standard and streaming dashboards)
- Statsd
- Graphite (with Grafana)
- InfluxDB
- Flume
- Fluentd
- OpenTSDB and TCollector
This repository started off from the ideas on my Monitoring Everything blog series.
Goals
Scripts included in this project must:
- Follow best practices for Ubuntu and the applications being installed
- Document service boundaries (ports, sockets, etc.) and default credentials
- Be as minimal as possible
- Be as reusable as possible (middleware and persistence, for example, can be reused between applications)
- Try to stay close to each application's defaults (specifically port numbers and file locations)
This makes it easier for people to solve issues during evaluation, as well as prepare for deploying the applications into staging and production environments.
I won't accept scripts for the following applications. These applications are either too old or too bloated for cloud monitoring.
- Nagios
- Zabbix
Contributing
Want to add additional scripts? Improve existing scenarios? Just fork it, and submit a pull request. It's that simple.
Usage
Two Vagrant boxes are provided with this script.
monitoring
on 192.168.12.10app1
on 192.168.12.11
The monitoring
box is intended for the different monitoring stacks, while app1
is intended to try out clients.
Sensu/Logstash
The monitoring
box provides the following:
For client nodes, it provides:
- AMQP (TCP/5672)
- AMQP over SSL (TCP/5671)
- Redis (TCP/6379)
- Statsd (UDP/8125)
- Lumberjack receiver (TCP/5043)
- Graphite line receiver (TCP/2013)
- Graphite Pickle receiver (TCP/2014)
OpenTSDB
This package provides scripts to install OpenTSDB and TCollector. OpenTSDB depends on HBase, which is installed in pseudo-distributed mode alongside a standalone ZooKeeper.
vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-opentsdb.sh
This script also installs Grafana with the OpenTSDB backend configured. Elasticsearch is installed as a dashboard store for Grafana.
TCollector is installed on the monitoring host to provide some sample metrics. Note that this script can take more than 10 minutes to install, depending on your bandwidth.
To start collecting metrics from app1
, simply SSH to it and run the following.
sudo -i
cd /vagrant && ./install-tcollector.sh
Sentry
This package provides scripts to install Sentry with the following configuration.
- memcached
- supervisord
- Redis as work queue, update buffer and TSDB
- PostgreSQL as node store
You can use the following to perform the installation.
vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-sentry.sh
After installation, the Sentry web interface should be available. Login with the username admin
and the password secret
.
Graylog2
To install Graylog2 with the web interface and stream dashboard, simply use the following.
vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-graylog2.sh
After installation, you can access one of the following resources. Use the username admin
and the passwod password
.
Note that Graylog2 requires its own Elasticsearch cluster, so do not use this script on the same node as other installation scripts.
Heka
Heka was created by Mozilla as a lighter alternative to Logstash. This package provides scripts for installing a Heka router that outputs to Elasticsearch.
vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-heka.sh
This package provides the following:
- Heka dashboard
- Kibana
- ElasticHQ
- Heka Protobuf input on
192.168.12.10
at TCP port 5565
InfluxDB
InfluxDB is an open-source distributed time series database with no external dependencies. This package provides scripts for using InfluxDB as a general replacement for Graphite. It will install Sensu and InfluxDB, configuring Sensu to relay metrics to InfluxDB.
vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-influxdb.sh
This package provides the following:
- Grafana configured for InfluxDB
- InfluxDB admin interface with default credentials
- Uchiwa dashboard for Sensu
Use vagrant up app1
to start collecting metrics
Flume
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. This package provides scripts for using Flume with Elasticsearch and receivers for Avro and Syslog protocols.
vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-flume.sh
This package provides the following:
- Kibana
- ElasticHQ
- Avro input on
192.168.12.10
at TCP port 41414 - Syslog input on
192.168.12.10
at TCP port 1514
When using Kibana, you will need to change the index pattern to [flume-]YYYY-MM-DD
.
Fluentd
Fluentd is an open source data collector to unify log management. This package provides scripts for using Fluentd with Elasticsearch and receivers for Syslog and HTTP protocols.
vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-fluentd.sh
This package provides the following:
- Kibana
- ElasticHQ
- HTTP input on
192.168.12.10
at TCP port 9880 - Syslog input on
192.168.12.10
at TCP port 1514
Testing Syslog Receivers
To quickly test the functionality of a syslog-compatible receiver, you can use the logger
command on Ubuntu.
message="hello world"
# With TCP syslog receiver
logger -n localhost -P 1514 $message
# With UDP syslog receiver
logger -n localhost -P 1514 -d $message