JMX exporter high memory usage.

[harshal-choudhari-9393]

I'm employing an independent JMX exporter in conjunction with Kafka Connect to retrieve Kafka Connect metrics in Prometheus within a Kubernetes pod. Specifically, I'm utilizing version 0.20 of the JMX exporter. With approximately 9000 topics in Kafka, I've allocated 5G of memory to the pod. Upon initiating the JMX exporter, JMX memory utilization promptly surges to 100%, leading to the termination of the pod and consequently halting Kafka Connect. I retrieved high cardinality metrics and attempted to mitigate this issue by deactivating them using the provided configuration However, I am still receiving all the metrics that I have attempted to disable.

excludeObjectNames: ["kafka.log:type=Log, name=Size,*"......]

Could you please guide me on reducing memory usage for the JMX exporter and disabling unnecessary metrics? Also, Let me know if any further information is needed.

[Doug Hoard]

@harshal-choudhari-9393 can you provide your JMX Exporter YAML configuration file?

[harshal-choudhari-9393]

Sure. Here is the YAML for the JMX exporter.

lowercaseOutputName: true
excludeObjectNames: ["kafka.server:type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec,*","kafka.cluster:type=Partition, name=UnderReplicated,*", "kafka.log:name=Size,type=Log,*","kafka.cluster:type=Partition,name=ReplicasCount,*"]
rules:
# Special cases and very specific rules
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    topic: "$4"
    partition: "$5"
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    broker: "$4:$5"
- pattern: kafka.coordinator.(\w+)<type=(.+), name=(.+)><>Value
  name: kafka_coordinator_$1_$2_$3
  type: GAUGE

# Generic per-second counters with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER

# Generic gauges with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE

# Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
#
# Note that these are missing the '_sum' metric!
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
    quantile: "0.$8"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
  name: kafka_$1_$2_$3_value
  type: GAUGE
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>OneMinuteRate
  name: kafka_$1_$2_$3_rate
  type: GAUGE
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    quantile: "0.$6"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
  name: kafka_$1_$2_$3_value
  type: GAUGE
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>OneMinuteRate
  name: kafka_$1_$2_$3_rate
  type: GAUGE
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    quantile: "0.$4"```

[Doug Hoard]

@harshal-choudhari-9393 Can you provide your solution/resolution for future information?