This Prometheus exporter consumes the __consumer_offsets topic of a Kafka cluser and exports the results as Prometheus gauge metrics. i.e. it shows the position of Kafka consumer groups, including their lag.
The high-water and low-water marks of the partitions of each topic are also exported.
The exporter requires Python 3 and Pip 3 to be installed.
To install the latest published version via Pip, run:
> pip3 install prometheus-kafka-consumer-group-exporter
Note that you may need to add the start script location (see pip output) to your PATH.
Once installed, you can run the exporter with the prometheus-kafka-consumer-group-exporter command.
By default, it will bind to port 9208 and connect to Kafka on localhost:9092. You can change these defaults as required by passing in arguments:
> prometheus-kafka-consumer-group-exporter -p <port> -b <kafka nodes>
Run with the -h flag to see details on all the available arguments.
Prometheus metrics can then be scraped from the /metrics path, e.g. http://localhost:9208/metrics. Metrics are currently actually exposed on all paths, but this may change in the future and /metrics is the standard path for Prometheus metric endpoints.
Nine main metrics are exported:
The latest committed offset of a consumer group in a given partition of a topic, as read from __consumer_offsets. Useful for calculating the consumption rate and lag of a consumer group.
The lag of a consumer group behind the head of a given partition of a topic - the difference between kafka_topic_highwater and kafka_consumer_group_offset. Useful for checking if a consumer group is keeping up with a topic.
The lead of a consumer group ahead of the tail of a given partition of a topic - the difference between kafka_consumer_group_offset and kafka_topic_lowwater. Useful for checking if a consumer group is at risk of missing messages due to the cleaner.
The number of commit messages read from __consumer_offsets by the exporter from a consumer group for a given partition of a topic. Useful for calculating the commit rate of a consumer group (i.e. are the consumers working).
The offset of the exporter's consumer in each partition of the __consumer_offset topic. Useful for calculating the lag of the exporter.
The lag of the exporter's consumer behind the head of each partition of the __consumer_offset topic. Useful for checking if the exporter is keeping up with __consumer_offset.
The lead of the exporter's consumer ahead of the tail of each partition of the __consumer_offset topic. Useful for checking if the exporter is at risk of missing messages due to the cleaner.
The offset of the head of a given partition of a topic, as reported by the lead broker for the partition. Useful for calculating the production rate of the producers for a topic, and the lag of a consumer group (or the exporter itself).
The offset of the tail of a given partition of a topic, as reported by the lead broker for the partition. Useful for calculating the lead of a consumer group (or the exporter itself) - i.e. how far ahead of the cleaner the consumer group is.
Lag metrics are exported for convenience, but they can also be calculated using other metrics if desired:
# Lag for a consumer group:
kafka_topic_highwater - on (topic, partition) kafka_consumer_group_offset{group="some-consumer-group"}
# Lag for the exporter:
kafka_topic_highwater{topic='__consumer_offsets'} - on (partition) kafka_consumer_group_exporter_offset
Note that as the offset and high-water metrics are updated separately the offset value can be more up-to-date than the high-water, resulting in a negative lag. This is often the case with the exporter lag, as the exporter offset is tracked internally rather than read from __consumer_offsets.
If you need to set Kafka consumer configuration that isn't supported by command line arguments, you can provided a standard Kafka consumer properties file:
> prometheus-kafka-consumer-group-exporter --consumer-config consumer.properties
See the Kafka docs for details on consumer properties. However, as the exporter doesn't use the official consumer implementation, all properties may not be supported. Check the kafka-python docs if you run into problems.
You can provide multiple files if that's helpful - they will be merged together, with later files taking precedence:
> prometheus-kafka-consumer-group-exporter --consumer-config consumer.properties --consumer-config another-consumer.properties
Note that where a command line argument relates to a consumer property (e.g. --bootstrap-brokers sets bootstrap.servers) a value provided via that argument will override any value for that property in a properties file. The argument default will only be used if the property isn't provided in either a file or an argument.
Docker images for released versions can be found on Docker Hub (note that no latest version is provided):
> sudo docker pull braedon/prometheus-kafka-consumer-group-exporter:<version>
To run a container successfully, you will need map container port 9208 to a port on the host. Any options placed after the image name (prometheus-kafka-consumer-group-exporter) will be passed to the process inside the container. For example, you will need to use this to configure the kafka node(s) using -b.
> sudo docker run --rm --name exporter \
-p <host port>:9208 \
braedon/prometheus-kafka-consumer-group-exporter:<version> -b <kafka nodes>
To install directly from the git repo, run the following in the root project directory:
> pip3 install .
The exporter can be installed in "editable" mode, using pip's -e flag. This allows you to test out changes without having to re-install.
> pip3 install -e .
To build a docker image directly from the git repo, run the following in the root project directory:
> sudo docker build -t <your repository name and tag> .
Send me a PR if you have a change you want to contribute!