docklogbeat

DEPRECATED: Please use a more robust solution, such as fluentd on Kubernetes. This repository is no longer maintained. It is archived as an example for future users.

docklogbeat is a tool which combines filebeat and docker-gen to transport data from Docker container logs to Logstash and Elasticsearch.

It is intended as a more reliable replacement for logspout.

The implementation was inspired by this blog post.

Instructions

Run the docklogbeat container on each physical host in your cluster. It will watch for new Docker containers to be spawned. If a container has a specific label, it will automatically begin shipping the logs to your Logstash or Elasticsearch cluster.

An example:

$ docker run -d \
    -v /var/lib/docker:/var/lib/docker:ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e LOGSTASH_HOSTS=logstash1.example.com:5044,logstash2.example.com:5044 \
    docklogbeat

Any containers with the label docklogbeat=true will have their Docker logs sent to the configured Logstash / Elasticsearch cluster.

An example:

$ docker run -l docklogbeat=true ...

Limitations

The docker-gen tool takes about 2 seconds to notice new containers and regenerate the filebeat configuration. If your containers are short lived, they may not have their log output sent to your Logstash / Elasticsearch cluster.

This container will read all log messages from Log Producing Containers when it starts, similar to docker logs --tail=all. If you need to restart docklogbeat without producing duplicate messages, you must make sure to store /var/lib/filebeat/registry on a persistent volume.

Rancher Integration

This container automatically detects if it is being run on a Rancher cluster and sets the DOCKLOGBEAT_REGISTRY_FILE_NAME to a unique value based on the hostname of the physical host running the container. This allows docklogbeat to be restarted safely without producing duplicate messages.

Environment Variables

These environment variables can be used to configure the docklogbeat container.

LOGSTASH_HOSTS - A comma-separated lists of the Elastic Beats servers which will receive logs.
DOCKLOGBEAT_REGISTRY_FILE_NAME - File to store the filebeat registry (default: /var/lib/filebeat/registry).
DOCKLOGBEAT_LABEL_KEY - Container Filter Label Key (default: docklogbeat).
DOCKLOGBEAT_LABEL_VALUE - Container Filter Label Value (default: true).
DOCKLOGBEAT_SCAN_FRECUENCY - Container Scan Logs Frecuency (default: 11s).

Labels for Log Producing Containers

docklogbeat - must be set to true for docklogbeat to collect this container's logs.
docklogbeat_document_type - set the document_type field for this container (optional).

irasnyd / docklogbeat