Micrometer Prometheus metrics export fails due to conditional `proxy.address` tag
Stephan202 opened this issue · comments
With the introduction of the proxy.address metrics tag in #3081, we started to observe that Micrometer Prometheus metrics registration started to fail, due to the requirement that metrics with the same name have the same set of tags:
2024-05-01 14:03:54.728 ERROR 1 --- [or-http-epoll-4] PrometheusMetricsConfig () : Failed to register meter_id='MeterId{name='reactor.netty.http.client.data.received', tags=[tag(country=REDACTED),tag(proxy.address=REDACTED),tag(remote.address=REDACTED),tag(uri=http)]}': Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'reactor.netty.http.client.data.received' containing tag keys [country, remote_address, uri]. The meter you are attempting to register has keys [country, proxy_address, remote_address, uri].
Besides reduced observability, this causes a stream of error logs. This happens because in one of our Spring Boot applications we use Netty to connect to multiple other systems, only one of which involves a proxy.
Would be possible to e.g. either have the proxy.address tag be conditionally omitted, or to always include it, with a default value (the empty string?) where relevant? (Maybe there are other approaches; just proposing the two "obvious" ones 😄.)
NB: In the log line shown above, tag(uri=http) looks rather uninformative 👀. Didn't investigate that aspect.
All meters with the same name have the same set of tag keys. If mixed tag set used then Spring start fails.
If proxy not used the value like "na" should be set to it.
Sounds that some code path does not set this new tag --> conflict
Prometheus when scraping can mix tag sets.
@Stephan202 Thanks for the report. I confirm that this is an issue and it will be fixed for the next release.