This project orignally was developped by Michael Marshall, unfortunately the original repo is not longer available, this is a copy
To add missing Log Derived Metrics functionality to Fluent Bit. When combined with Grafana Loki, Prometheus, Cortex, and Grafana UI, you get a fully functional Data Observability Platform.
The Output plugin leverages the prometheus pushgateway API by pushing the tracked metrics to pushgateway backed with persistent store. This allows for a fire-and-forget architecture to prevent fluent bit from blocking. Prometheus treats pushgateway as a metric scrape target.
While the plugin pairs nicely with Grafana Loki's Output plugin, it can be used standalone or coupled with other output plugins.
| Key | Description | Required | Default | Valid Options | Notes |
|---|---|---|---|---|---|
| id | Plugin instance id | Yes | Must be unique per [OUTPUT] section in a single fluent-bit.conf | ||
| job | Prometheus job label | Yes | |||
| url | HTTP Url for destination push gateway | Yes | Ex. http://127.0.0.1:9091 | ||
| push_gateway_retries | Number of retry attempts to connect to push gateway | No | 3 | ||
| metric_type | Prometheus metric type | Yes | none | Counter, Gauge, Summary, Histogram | |
| metric_name | Metric name sent to Prometheus | Yes | |||
| metric_help | Help string associated with metric | Yes | Enclose in double quotes | ||
| metric_constant_labels | Static JSON formatted key/value pairs to index metric | No | Although not required, {"instance":"1"} is recommended. Ex. {"instance":"1", "source":"fluent-bit"} |
||
| metric_variable_labels | Comma separated list of fluent bit fields to index metric. This is the key to log derived metrics. The value of these keys will vary depending on log line content. | No | Will be appended to any metric_constant_labels is configured. Ex. origin, status_code, method |
In addition to keys noted above.
See Prometheus Counter for details. No additional parameters are required Counter at this time.
See Prometheus Summary for details.
| Key | Description | Required for Specific Metric Type | Default | Valid Options | Notes |
|---|---|---|---|---|---|
| metric_summary_observe_key | Single fluent bit field to observe for Summary metric type. | Yes |
See Prometheus Histogram for details.
| Key | Description | Required for Specific Metric Type | Default | Valid Options | Notes |
|---|---|---|---|---|---|
| metric_histogram_bucket_type | Histogram bucket distribution | Yes | Linear, Exponential | ||
| metric_histogram_observe_key | Single fluent bit field to observe for Histogram metric type. | Yes |
Creates 'count' buckets, each 'width' wide, where the lowest bucket has an upper bound of 'start'. The final +Inf bucket is not counted and is not included. See Prometheus SDK reference - LinearBuckets() for details.
| Key | Description | Required for Specific Metric Type | Default | Valid Options | Notes |
|---|---|---|---|---|---|
| metric_histogram_linear_buckets_count | Count of buckets | Yes | > 0 | Ex. 5 | |
| metric_histogram_linear_buckets_width | Width of bucket | Yes | Ex. 5 | ||
| metric_histogram_linear_buckets_start | Lowest bucket has an upper bound of Start | Yes | Ex. 20 |
Creates 'count' buckets, where the lowest bucket has an upper bound of 'start' and each following bucket's upper bound is 'factor' times the previous bucket's upper bound. The final +Inf bucket is not counted and not included. See Prometheus SDK reference - ExponentialBuckets() for details.
| Key | Description | Required for Specific Metric Type | Default | Valid Options | Notes |
|---|---|---|---|---|---|
| metric_histogram_exponential_buckets_count | Count of buckets | Yes | > 0 | Ex. 5 | |
| metric_histogram_exponential_buckets_factor | Each additional bucket upper bound is Factor times the previous bucket's upper bound | Yes | Ex. 1.5 | ||
| metric_histogram_exponential_buckets_start | Lowest bucket has an upper bound of Start | Yes | Ex. 20 |
See Prometheus Gauge for details.
| Key | Description | Required for Specific Metric Type | Default | Valid Options | Notes |
|---|---|---|---|---|---|
| metric_gauge_method | Method selected to change the value of the Gauge | Yes | Set, Add, Sub, Inc, Dec | Set, Add, and Sub require a key as input. Inc and Dec only increment or decrement the gauge by 1. | |
| metric_gauge_set_key | Single fluent bit field as input to Set method. | Yes with Set | |||
| metric_gauge_add_key | Single fluent bit field as input to Add method. | Yes with Add | |||
| metric_gauge_sub_key | Single fluent bit field as input to Sub method. | Yes with Sub |
The example folder contains a set of configurations showing each type of metric currently supported by the plugin plus Grafana Loki logs. These were used to create the dashboard pictured above.
- Check out https://github.com/ycyr/fluent-bit-data-observability-platform for a full environment leveraging the example configuration.
- Either Docker Desktop, or docker and docker-compose.
- Make
- Curl
make all
make build
make start
Execute repeatedly to see the metrics generated using the plugin.
make show-metrics
Execute repeatedly to see the latest container logs
make logs
Shutdown demo environment.
make stop
The build-plugin configuration in the Makefile is referenced by the docker build environment.
https://fluentbit.io/
https://grafana.com/oss/grafana/
https://grafana.com/oss/loki/
https://grafana.com/oss/prometheus/
https://grafana.com/oss/cortex/
Reference Talk: FluentCon 2021: Fluent Bit - Swiss Army Tool of Observability Data Ingestion
- Michael Marshall - Project creation and development.
- Refer to our contribution guidelines to contribute to this project. See CONTRIBUTING.md.
- All contributions must follow our code of conduct. See CONDUCT.md.
- This project is licensed under the Apache 2.0 license. See LICENSE.
Thank you to the teams who create and maintain the individual projects listed above.