out of memory when prometheus is enabled and url with path varaible
Ricard-Kollcaku opened this issue · comments
Having a spring boot service using spring boot 3.2.1 java 21 as native Image and webflux micrometer for tracing and prometheus.
service has a url that use path variable like get/{unique id}
if i enable prometheus looks like it saves results for every url like
reactor_netty_http_server_data_received_bytes_count{application="label-gateway",uri="/get/1",} 1.0
reactor_netty_http_server_data_received_bytes_sum{application="label-gateway",uri="/get/1",} 0.0
reactor_netty_http_server_data_received_bytes_count{application="label-gateway",uri="/get/2",} 1.0
reactor_netty_http_server_data_received_bytes_sum{application="label-gateway",uri="/get/2",} 0.0
and so one and the response from prometheus gets bigger and bigger till after some times service goes out of memory.
Not sure if this is a reactor issue or micrometer
@Ricard-Kollcaku I would say that you have misconfiguration. Most probably you enabled the metrics with this .metrics(true, Function.identity())
?
If yes you should have in the logs the following message (DEBUG level)
Metrics are enabled with [uriTagValue=Function#identity]. It is strongly recommended to provide template-like form for the URIs. Without a conversion to a template-like form, each distinct URI leads to the creation of a distinct tag, which takes a lot of memory for the metrics.
Also we've described that in the javadoc and the reference documentation
https://projectreactor.io/docs/netty/release/reference/index.html#_metrics_4
not really the only configuration i have is
Hooks.enableAutomaticContextPropagation(); ObservationThreadLocalAccessor.getInstance().setObservationRegistry(observationRegistry); Metrics.observationRegistry(observationRegistry);
and in props
management:
endpoint:
health:
probes:
enabled: true
show-details: always
endpoints:
web:
exposure:
include: [ "*" ]
base-path: "/management"
prometheus:
metrics:
export:
enabled: true` ``
@Ricard-Kollcaku You have label label-gateway
, is this Spring Cloud Gateway? Basically you need to find where the metrics are enabled with Function.identity()
. Reactor Netty does not enable the metrics by default so the user needs to configure this explicitly.
no is not spring cloud gateway. Ok i will check thank you
hi @violetagg i found the issue, thank u for the support. im closing the ticket