SimoneLazzaris / http_loggo

Logs fluentbit HTTP output from kubernetes into a logfile

What is this?

It's a simple python script that listen for HTTP POST request generated by fluentbit http output component. Log are then written on a (rotated) logfile for easier consultation and longer storage.

Why is it useful?

Kubernetes logging is... erratic. Messages from deleted pods are lost, and only a (small) fraction is kept for running ones. It's also difficult to relate messages from different pods to events or to each other.

Fluentbit makes it easy to collect all messages, but usually requires complex solutions like elasticsearch or graylog. I wanted a simple log file to collect everything and keep it available for a reasonable amount of time.

For that, you could use the syslog output and then rely on the good old rsyslog or syslog-ng, but some details (pod, container, namespace) are lost.

With this simple script, those details are extracted from the json http payload and stored in the logfile.

Installation:

helm repo add fluent https://fluent.github.io/helm-charts
helm install -n fluent --create-namespace fluent fluent/fluent-bit -f fluent.yaml --set loggoPasswd=yourpasswordhere --set loggoHost=yourhostiphere

make install

Logfile is /var/log/fluentbit/fluentbit.log; by default, it will keep 20MB of logs, and then it will rotate and 5 rotations will be kept.

Requirements

You'll need:

• helm
• make (for installation, but you can also manually copy some files)
• python3 (tested on version 3.9)
• sdnotify for python3 (apt install python3-sdnotify on debian)