kylemcc / kube-filebeat

Automated log shipper for Kubernetes powered by annotations

kube-filebeat

kube-filebeat is a Docker container running filebeat and kube-gen. kube-gen watches for events on the Kubernetes API and generates filebeat configurations (based on Pod annotations) to harvest logs from applications running in Kubernetes and ship them to logstash.

Note: This project is mostly experimental. It relies on and exploits the mechanics of Docker's filesystem layer. The implementation here only works for Docker versions >= 1.10.0 and may break at any time.

Usage

Due to the mechanics of how kube-filebeat operates, it needs to be running on any node from which you would like to collect logs. The recommended way to acheive this is to run kube-filebeat as a Daemon Set. For example:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: "kube-filebeat"
  annotations:
    description: "automated log shipper powered by annotations"
spec:
  template:
    spec:
      containers:
        -
          name: "kube-filebeat"
          image: "kylemcc/kube-filebeat:latest"
          env:
            -
              name: LOGSTASH_HOSTS
              value:  logstash.default.svc.cluster.local:5044
            -
              name: KUBERNETES_API_URL
              value: http://10.1.2.3:8080
          volumeMounts:
            - name: docker
              mountPath: /var/lib/docker
          imagePullPolicy: "Always"
      restartPolicy: "Always"
      volumes:
        - name: docker
          hostPath:
            path: /var/lib/docker

Configuration

Annotations are used to inform kube-filebeat of files that should be harvested. For example:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kube_filebeat: >
      [
          {
              "log": "/var/log/example-app/output.log",
              "ignore_older": "24h",
              "close_older": "24h",
              "fields": {
                  "app": "example-app",
                  "version": "1.2.3"
              },
              "multiline": {
                  "pattern": "^(([[:alpha:]]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2})|([0-9]{4}-[0-9]{2}-[0-9]{2}))",
                  "negate": true,
                  "match": "after"
              }
          },
          {
              "log": "/var/log/nginx/access.log",
              "exclude_lines": [".*Go-http-client/1\\.1.*"],
              "ignore_older": "24h",
              "close_older": "24h",
              "fields": {
                  "app": "example-app",
                  "version": "1.2.3",
                  "type": "access_log"
              }
          }
      ]
  name: example-app
spec:
  containers:
    - image: example-app:1.2.3
      name: example-app

For multi-container pods, specify the container name in each filebeat config. E.g.:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kube_filebeat: >
      [
          {
              "container": "example-app",
              "log": "/var/log/app/logfile",
              ...
          },
          {
              "container": "nginx",
              "log": "/var/log/nginx/access.log",
              ...
          }
      ]
spec:
  containers:
    - image: example-app:1.2.3
      name: example-app
    - image: nginx:latest
      name: nginx