patterns support for config file for kafka metrics

Question

patterns support for config file for kafka metrics

Janardhan78 opened this issue 7 months ago · comments

I am trying to creat a pattern for the below kafka metrics to be filtered

kafka_server_replica_fetcher_metrics_iotime_total

kafka_server_replica_fetcher_metrics_iotime_total Deprecated The total time the I/O thread spent doing I/O kafka.server:name=null,type=replica-fetcher-metrics,attribute=iotime-total

TYPE kafka_server_replica_fetcher_metrics_iotime_total untyped

kafka_server_replica_fetcher_metrics_iotime_total{broker_id="1",fetcher_id="0",} 4.0117128E7

tried the below pattern

pattern : kafka.server<type=replica-fetcher-metrics,attribute=iotime-total>

the filtering is not happening

is it mandatory to provide the name ,type, label in the rules?

Doug Hoard · Answer 1 · Thu Jan 11 2024 03:29:30 GMT+0800 (China Standard Time)

Can you share the rules as designed in your YAML file?

Janardhan78 · Answer 2 · Thu Jan 11 2024 17:14:03 GMT+0800 (China Standard Time)

pattern : kafka.server<type=replica-fetcher-metrics, broker-id=(\w+), fetcher-id=(\w+)>

Janardhan78 · Answer 3 · Mon Jan 22 2024 17:30:39 GMT+0800 (China Standard Time)

We also have issue with the excludeObjectNameAttributes

below is the config file

lowercaseOutputName: false
excludeObjectNameAttributes:
"kafka.network:type=RequestMetrics, name=ResponseSendTimeMs, request=AddPartitionsToTxn":
- "50thPercentile"
rules:

pattern : kafka.network<type=RequestMetrics, name=ResponseSendTimeMs, request=(\w+)><>50thPercentile

The metrics HELP is as below

HELP kafka_network_RequestMetrics_50thPercentile Attribute exposed for management kafka.network:name=ResponseS

endTimeMs,type=RequestMetrics,attribute=50thPercentile

but still the metrics is not filtered.

kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="AddPartitionsToTxn",} 0.0

Janardhan78 · Answer 4 · Tue Jan 23 2024 20:11:28 GMT+0800 (China Standard Time)

hi @dhoard
is there any help we can get on the filtering
Regards
Jana

Doug Hoard · Answer 5 · Tue Jan 23 2024 22:08:26 GMT+0800 (China Standard Time)

@Janardhan78 You should include name, labels, and type as part of your rule definition.

Regarding excludeObjectNameAttributes, which version of the exporter are you using?

Janardhan78 · Answer 6 · Tue Jan 23 2024 22:49:25 GMT+0800 (China Standard Time)

we are using 0.18.0

Doug Hoard · Answer 7 · Tue Jan 23 2024 23:01:54 GMT+0800 (China Standard Time)

This was added in a later version. Please upgrade to the latest version.

Janardhan78 · Answer 8 · Wed Jan 24 2024 18:31:46 GMT+0800 (China Standard Time)

hi @dhoard The latest available release version is 0.20.0 is the support present in 0.20.0

Janardhan78 · Answer 9 · Thu Jan 25 2024 20:08:58 GMT+0800 (China Standard Time)

hi @dhoard

We used the latest version of jmx_prometheus_javaagent-0.20.0.jar, and provided the configuration to excludeObjectNameAttributes

-javaagent:/usr/share/java/kafka/jmx_prometheus_javaagent-0.20.0.jar=9404:/etc/confluent/metrics/kafka_metrics_config.yaml

kafka_metrics_config.yaml contents as below

lowercaseOutputName: false
excludeObjectNameAttributes:
"kafka.network:type=RequestMetrics, name=ResponseSendTimeMs, request=AddPartitionsToTxn":
- "50thPercentile"
rules:

pattern : kafka.network<type=RequestMetrics, name=ResponseSendTimeMs, request=(\w+)><>50thPercentile

but the metrics "AddPartitionsToTxn" is not being excluded ? is this a bug

bash-4.4$ curl localhost:9404

HELP kafka_network_RequestMetrics_50thPercentile Attribute exposed for management kafka.network:name=ResponseSendTimeMs,type=RequestMetrics,attribute=50thPercentile

TYPE kafka_network_RequestMetrics_50thPercentile untyped

kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="AlterUserScramCredentials",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="ApiVersions",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="DescribeProducers",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="DeleteAcls",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="FetchConsumer",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="AlterPartitionReassignments",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="AddOffsetsToTxn",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="OffsetDelete",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="Fetch",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="OffsetForLeaderEpoch",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="CreateDelegationToken",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="CreatePartitions",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="DescribeAcls",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="DeleteGroups",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="DescribeLogDirs",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="DescribeGroups",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="UpdateFeatures",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="InitProducerId",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="DeleteTopics",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="ListPartitionReassignments",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="AddPartitionsToTxn",} 0.0

Janardhan78 · Answer 10 · Thu Jan 25 2024 20:43:34 GMT+0800 (China Standard Time)

hi @dhoard

with the below configuration with name and labels provided the metrics is not filtered i suppose excludeObjectNameAttributes is not supported in 0.20.0 or is it a bug ?

kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="AddPartitionsToTxn",} 0.0

bash-4.4$ curl localhost:9404 | grep ddPartitionsToTxn
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 17338 100 17338 0 0 110k 0 --:--:-- --:--:-- --:--:-- 110k

kafka_network_RequestMetrics_50thPercentile{request="AddPartitionsToTxn",} 0.0
bash-4.4$ cat /etc/confluent/metrics/kafka_metrics_config.yaml
lowercaseOutputName: false
excludeObjectNameAttributes:
"kafka.network:type=RequestMetrics, name=ResponseSendTimeMs, request=AddPartitionsToTxn":
- "50thPercentile"
rules:

pattern : kafka.network<type=RequestMetrics, name=ResponseSendTimeMs, request=(\w+)><>50thPercentile
name: kafka_network_RequestMetrics_50thPercentile
labels:
request: $1

Doug Hoard · Answer 11 · Fri Jan 26 2024 22:06:51 GMT+0800 (China Standard Time)

@Janardhan78 The functionality is in 0.20.0 and works as expected.

The configuration...

excludeObjectNameAttributes:
 "kafka.network:type=RequestMetrics, name=ResponseSendTimeMs, request=AddPartitionsToTxn":
 - "50thPercentile"

... is incorrect because the ObjectName is only kafka.network:type=RequestMetrics. name and request are MBean attributes.

Here is an example from the integration tests:

  "java.lang:type=Runtime":
    - "ObjectName"
    - "ClassPath"
    - "SystemProperties"

In this scenario, for the Runtime MBean, we will not include the ObjectName, ClassPath, or SystemProperties attributes.

Janardhan78 · Answer 12 · Mon Jan 29 2024 22:19:48 GMT+0800 (China Standard Time)

Hi @dhoard

We have a scenario where the attribute "request" has 10 values out of which we want to select only 7 values, so instead of providing all 7 values and select, is there a way to provide only 3 values which is not needed to be excluded in the output.

Doug Hoard · Answer 13 · Mon Jan 29 2024 23:52:55 GMT+0800 (China Standard Time)

@Janardhan78 to clarify, you have an MBean that has an attribute that returns an array of values?

Janardhan78 · Answer 14 · Tue Jan 30 2024 17:16:18 GMT+0800 (China Standard Time)

Hi @dhoard
for example the attribute request can have multiple values like "ApiVersions", "AddPartitionsToTxn" etc
so if i want to filter based upon the values in request attribute, i needed to provide the rules as below.
Is there any other way to filter the attribute value ? other than providing the below rule. so that we can exclude metrics based upon the attribute value

rules:

pattern : kafka.network<type=RequestMetrics, name=(\w+), request=([A][d][d][P][a][r][t][i][t][i][o][n][s][T][o][T][x][n]|[A][p][i][V][e][r][s][i][o][n][s])><>50thPercentile
name: kafka_network_RequestMetrics_50thPercentile
labels:
name: $1
request: $2

kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="ApiVersions",} 4.0
kafka_network_RequestMetrics_50thPercentile{name="RemoteTimeMs",request="ApiVersions",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="TotalTimeMs",request="AddPartitionsToTxn",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseSendTimeMs",request="AddPartitionsToTxn",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ThrottleTimeMs",request="AddPartitionsToTxn",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseQueueTimeMs",request="AddPartitionsToTxn",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="TotalTimeMs",request="ApiVersions",} 69.0
kafka_network_RequestMetrics_50thPercentile{name="RemoteTimeMs",request="AddPartitionsToTxn",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="RequestQueueTimeMs",request="AddPartitionsToTxn",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseQueueTimeMs",request="ApiVersions",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="LocalTimeMs",request="ApiVersions",} 33.0
kafka_network_RequestMetrics_50thPercentile{name="LocalTimeMs",request="AddPartitionsToTxn",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="RequestQueueTimeMs",request="ApiVersions",} 31.0
kafka_network_RequestMetrics_50thPercentile{name="RequestBytes",request="ApiVersions",} 25.0
kafka_network_RequestMetrics_50thPercentile{name="RequestBytes",request="AddPartitionsToTxn",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ThrottleTimeMs",request="ApiVersions",} 0.0

Janardhan78 · Answer 15 · Thu Feb 01 2024 18:09:06 GMT+0800 (China Standard Time)

HI @dhoard
Can you please help me with the query #903 (comment)

Regards
Jana

Doug Hoard · Answer 16 · Thu Feb 01 2024 23:45:51 GMT+0800 (China Standard Time)

@Janardhan78 apologizes for late response. (I'm on business travel/vacation for a few weeks.)

I would suggest using an online Java regular expression based tool using the raw metric line to get the correct matching inclusion/exclusion you require.

Janardhan78 · Answer 17 · Fri Feb 02 2024 17:47:05 GMT+0800 (China Standard Time)

hi @dhoard
can you share any comments for the below scenario

Scenario:
I want all the metrics for the kafka_network_RequestMetrics_50thPercentile

except the below 3,
kafka_network_RequestMetrics_50thPercentile{name="ResponseQueueTimeMs",request="FindCoordinator",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseQueueTimeMs",request="WriteTxnMarkers",} 0.0
kafka_network_RequestMetrics_50thPercentile{name="ResponseQueueTimeMs",request="AddPartitionsToTxn",} 0.0

for the above scenario to exclude row wise 3 rows of data Is it possible to exclude?

Doug Hoard · Answer 18 · Sat Feb 03 2024 00:32:20 GMT+0800 (China Standard Time)

@Janardhan78 for...

kafka_network_RequestMetrics_50thPercentile{name="ResponseQueueTimeMs",request="FindCoordinator",} 0.0

The actual MBean ObjectName is...

kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=FindCoordinator

If you don't want any metrics from the MBean, then exclude the MBean using...

excludeObjectNames: ["kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=FindCoordinator"]

The same configuration should be applied for other MBeans you want to exclude.