Duplicate metrics with NaN

[Joel Soares Rodrigues]

I have instantiated jmx-exporter-prometheus containers in many Kafka services. Some metrics are being exported in duplicate with one of the values being NaN.

For example in ksql service:

curl http://[TARGET_IP]:5556/metrics | grep kafka_consumer_consumer_fetch_manager_metrics_records_lag

kafka_consumer_consumer_fetch_manager_metrics_records_lag{clientId="client-12",partition="0",topic="topic-123",} NaN
kafka_consumer_consumer_fetch_manager_metrics_records_lag{clientId="client-12",partition="0",topic="topic-123",} 15.2

My jmx-exporter has the following configuration:

jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:5555/jmxrmi
lowercaseOutputName: true
rules:
  # kafka.streams:type=stream-thread-metrics,thread-id="{threadId}"
  - pattern: 'kafka.streams<type=stream-thread-metrics, thread-id=(.+)><>(.+-total|.+-rate|.+-avg)'
    name: kafka_streams_stream_thread_metrics_$2
    labels:
      threadId: "$1"
    help: "Kafka Streams JMX metric $2"
    type: GAUGE

  # kafka.streams:type=stream-task-metrics,thread-id="{threadId}",task-id="{taskId}"
  - pattern: 'kafka.streams<type=stream-task-metrics, thread-id=(.+), task-id=(.+)><>(.+-total|.+-rate|.+-ratio|.+-avg)'
    name: kafka_streams_stream_task_metrics_$3
    labels:
      threadId: "$1"
      taskId: "$2"
    help: "Kafka Streams JMX metric $3"
    type: GAUGE

  #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}",partition="{partition}"
  #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}",partition="{partition}"
  - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+), partition=(.+)><>(.+-total|.+-rate|.+-avg|.+-lag)
    name: kafka_$1_$2_metrics_$6
    labels:
      clientId: "$3"
      topic: "$4"
      partition: "$5"
    help: "Kafka $1 JMX metric type $2"
    type: GAUGE

  #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"
  #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}""
  - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+)><>(.+-total|.+-rate|.+-avg)
    name: kafka_$1_$2_metrics_$5
    labels:
      clientId: "$3"
      topic: "$4"
    help: "Kafka $1 JMX metric type $2"
    type: GAUGE

  #kafka.streams:type=streams-node-metrics,client-id="{clientid}",node-id="{nodeid}"
  #kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id="{nodeid}"
  - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), node-id=(.+)><>(.+-total|.+-avg)
    name: kafka_$1_$2_metrics_$5
    labels:
      clientId: "$3"
      nodeId: "$4"
    help: "Kafka $1 JMX metric type $2"
    type: UNTYPED

  #kafka.streams:type=kafka-metrics-theads,client-id="{clientid}"
  #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}"
  #kafka.consumer:type=consumer-coordinator-metrics,client-id="{clientid}"
  #kafka.consumer:type=consumer-metrics,client-id="{clientid}"
  #kafka.producer:type=producer-metrics,client-id="{clientid}"
  - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.*)><>(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-rate|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago)
    name: kafka_$1_$2_metrics_$4
    labels:
      clientId: "$3"
    help: "Kafka $1 JMX metric type $2"
    type: GAUGE

  #io.confluent.ksql.metrics:type=ksql-engine-query-stats:{serviceId}
  - pattern: 'io.confluent.ksql.metrics<type=ksql-engine-query-stats><>(_confluent-ksql-default_)(.+-total|.+-rate|.+-avg|.+-per-sec|num.+queries)'
    name: ksql_metrics_ksql_engine_query_stats_$2
    labels:
      serviceId: "$1"
    help: "ksql JMX metric $2"
    type: GAUGE
  - pattern: 'io.confluent.ksql.metrics<type=ksql-engine-query-stats><>(_confluent-ksql-default_ksql-engine-query-stats-)(.+-queries)'
    name: ksql_metrics_ksql_engine_query_stats_$2
    labels:
      serviceId: "$1"
    help: "ksql JMX metric $2"
    type: GAUGE

  #io.confluent.ksql.metrics:id={id},key={key},type=producer-metrics
  #io.confluent.ksql.metrics:id={id},key={key},type=consumer-metrics
  - pattern: 'io.confluent.ksql.metrics<id=(.*),key=(.*),type=(.+)-metrics><>(.+-per-sec|.+-messages|.+-bytes)'
    name: ksql_metrics_$3_metrics_$4
    labels:
      id: "$1"
      key: "$2"
    help: "ksql JMX m

Can help, pls?

[Doug Hoard]

@JoelRodrigues58 I saw your comment as well in #609. Did you follow the example in that issue?