How to use Prometheus to monitor Kong gateway

[xluffy]

Prometheus is an open source system monitor toolkit built at SoundCloud that is now widely adopted. StatsD was originally a simple daemon developed by Etsy to aggregate and summarize application metrics. Prometheus provides a StatsD exporter to collect metrics that are sent in StatsD format.

Kong Gateway supports both of the above for integrating with Prometheus. This enables Prometheus to pull metrics directly from the gateway or using a StatsD exporter in between to offload some work from the gateway.

Kong Gateway is built on top of OpenResty/Nginx, which is a multi-process single-threaded architecture. To collect and aggregate metrics from different processes, we implement the Prometheus plugin with shared memory.

Nginx handles requests in a non-blocking way and is normally very efficient. However, every read and write operation to the shared memory requires a mutex lock to lock the critical section and will block all worker processes from processing requests. When the plugin is used to monitor metrics with high cardinality, it can affect Kong Gateway performance significantly, especially increasing the long tail latencies. We recommend using the StatsD plugin as an alternative for such use cases.

https://konghq.com/blog/engineering/how-to-use-prometheus-to-monitor-kong-gateway