Fluent-bit daemonset for Kubernetes, using this helm chart and this Dockerfile.
Fluent Bit is a Data Forwarder for Linux, Embedded Linux, OSX and BSD family operating systems. It's part of the Fluentd Ecosystem. Fluent Bit allows collection of information from different sources, buffering and dispatching them to different outputs such as Fluentd, Elasticsearch, Nats or any HTTP end-point within others. It's fully supported on x86_64, x86 and ARM architectures.
Currently the daemonset reads Docker logs from /var/log/containers and journald logs from /var/log/journal and /run/log/journal. It adds Kubernetes metadata to the logs, and forwards everything to Fluentd.
helm repo add cnct https://charts.migrations.cnct.io
helm repo update
helm install cnct/fluent-bit
# or to pass in your own values file
helm install cnct/fluent-bit -f <your-values.yaml>
helm install --name my-release --namespace my-namespace ./fluent-bit
e.g.
helm install cnct/chart-fluent-bit. --name=fb-test --namespace=logging --set cluster_uuid=22222222-3333-0000-0000-000000000000
This input plugin reads from /var/log/journal, which contains kernel, dockerd, and rkt logs, among others. It is new as of v0.12.
This input plugin monitors text files as matched by a specified Path; in this case, /var/log/containers/*.log, excluding /var/log/containers/fluent*.log.
This filter adds the following data into the body of the log:
- namespace
- pod id
- pod name
- labels
- host
- container name
- container id
Forward is the protocol used by Fluentd to route messages between peers, and allows interoperability between Fluent Bit and Fluentd. Our default is set to forward. To enable secure forward mode, set the enableTlS value in values.yaml to true, then uncomment this section in the output plugin:
Shared_Key fluentd
Self_Hostname fluentd
tls on
tls.verify off
tls.debug 4
tls.ca_file /fluent-bit/ssl/ca.crt.pem
tls.crt_file /fluent-bit/ssl/client.crt.pem
tls.key_file /fluent-bit/ssl/client.key.pem
tls.key_passwd fbit
You will need to create client and server certs to use with both fluentd and fluent-bit to communicate securely. This information must be passed in as a Kubernetes Secret before this chart can install if you enable TLS. For more information, read this blog about Fluent Bit and Fluentd secure communication using TLS.
example secret creation:
kubectl create secret generic fluentd-tls \
--from-file=ca.crt.pem=./certs/ca.crt.pem \
--from-file=server.crt.pem=./certs/server.crt.pem \
--from-file=server.key.pem=./private/server.key.pem