Giters

graylog-labs / graylog-plugin-metrics-reporter

Graylog Metrics Reporter Plugins

Home Page:https://www.graylog.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Prometheus - not all provided information are exposed (pipeline rules)

jalogisch opened this issue · comments

commented

Not all available metrics available in the UI are exposed via prometheus for the debugging of the processing pipeline rules.

Available

image

Exposed

0.5, 0.75, 0.95, 0.98, 0.99, 0.999 and count

org_graylog_plugins_pipelineprocessor_ast_Rule_58f74b8ecb08a070f6e8f144_58f74b20cb08a070f6e8f0ce_1_trace_evaluate_duration Generated from Dropwizard metric import 

(metric=org.graylog.plugins.pipelineprocessor.ast.Rule.58f74b8ecb08a070f6e8f144.58f74b20cb08a070f6e8f0ce.1.trace.evaluate.duration, type=com.codahale.metrics.Timer)

org_graylog_plugins_pipelineprocessor_ast_Rule_58f74b8ecb08a070f6e8f144_58f74b20cb08a070f6e8f0ce_1_trace_evaluate_duration summary org_graylog_plugins_pipelineprocessor_ast_Rule_58f74b8ecb08a070f6e8f144_58f74b20cb08a070f6e8f0ce_1_trace_evaluate_duration{quantile="0.5",} 6.294E-6 

org_graylog_plugins_pipelineprocessor_ast_Rule_58f74b8ecb08a070f6e8f144_58f74b20cb08a070f6e8f0ce_1_trace_evaluate_duration{quantile="0.75",} 9.741E-6 

org_graylog_plugins_pipelineprocessor_ast_Rule_58f74b8ecb08a070f6e8f144_58f74b20cb08a070f6e8f0ce_1_trace_evaluate_duration{quantile="0.95",} 1.1951000000000001E-5 

org_graylog_plugins_pipelineprocessor_ast_Rule_58f74b8ecb08a070f6e8f144_58f74b20cb08a070f6e8f0ce_1_trace_evaluate_duration{quantile="0.98",} 1.3784E-5 

org_graylog_plugins_pipelineprocessor_ast_Rule_58f74b8ecb08a070f6e8f144_58f74b20cb08a070f6e8f0ce_1_trace_evaluate_duration{quantile="0.99",} 1.4817E-5 

org_graylog_plugins_pipelineprocessor_ast_Rule_58f74b8ecb08a070f6e8f144_58f74b20cb08a070f6e8f0ce_1_trace_evaluate_duration{quantile="0.999",} 3.3044000000000004E-5 org_graylog_plugins_pipelineprocessor_ast_Rule_58f74b8ecb08a070f6e8f144_58f74b20cb08a070f6e8f0ce_1_trace_evaluate_duration_count 705.0
commented

The issue is that prometheus actually drops some of the exported stats because they aren't overly useful (the rates are actually decaying exponentially since start and aren't accurate if summarized again in prometheus): https://prometheus.io/docs/instrumenting/writing_exporters/#drop-less-useful-statistics

There's unlikely to be support from prometheus for this:

We won't be able to work around this.