reactor / reactor-netty

TCP/HTTP/UDP/QUIC client/server with Reactor over Netty

Home Page:https://projectreactor.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

out of memory when prometheus is enabled and url with path varaible

Ricard-Kollcaku opened this issue · comments

Having a spring boot service using spring boot 3.2.1 java 21 as native Image and webflux micrometer for tracing and prometheus.
service has a url that use path variable like get/{unique id}
if i enable prometheus looks like it saves results for every url like
reactor_netty_http_server_data_received_bytes_count{application="label-gateway",uri="/get/1",} 1.0
reactor_netty_http_server_data_received_bytes_sum{application="label-gateway",uri="/get/1",} 0.0
reactor_netty_http_server_data_received_bytes_count{application="label-gateway",uri="/get/2",} 1.0
reactor_netty_http_server_data_received_bytes_sum{application="label-gateway",uri="/get/2",} 0.0

and so one and the response from prometheus gets bigger and bigger till after some times service goes out of memory.
Not sure if this is a reactor issue or micrometer

@Ricard-Kollcaku I would say that you have misconfiguration. Most probably you enabled the metrics with this .metrics(true, Function.identity())?

If yes you should have in the logs the following message (DEBUG level)

Metrics are enabled with [uriTagValue=Function#identity]. It is strongly recommended to provide template-like form for the URIs. Without a conversion to a template-like form, each distinct URI leads to the creation of a distinct tag, which takes a lot of memory for the metrics.

Also we've described that in the javadoc and the reference documentation

https://projectreactor.io/docs/netty/release/api/reactor/netty/http/server/HttpServer.html#metrics-boolean-java.util.function.Function-

https://projectreactor.io/docs/netty/release/reference/index.html#_metrics_4

not really the only configuration i have is
Hooks.enableAutomaticContextPropagation(); ObservationThreadLocalAccessor.getInstance().setObservationRegistry(observationRegistry); Metrics.observationRegistry(observationRegistry);

    and in props 
management:
  endpoint:
    health:
      probes:
        enabled: true
      show-details: always
  endpoints:
    web:
      exposure:
        include: [ "*" ]
      base-path: "/management"
  prometheus:
    metrics:
      export:
        enabled: true` ``
        

@Ricard-Kollcaku You have label label-gateway, is this Spring Cloud Gateway? Basically you need to find where the metrics are enabled with Function.identity(). Reactor Netty does not enable the metrics by default so the user needs to configure this explicitly.

no is not spring cloud gateway. Ok i will check thank you

hi @violetagg i found the issue, thank u for the support. im closing the ticket