reactor_netty_http_server_connections_active and reactor_netty_http_server_connections_total metrics inaccurate
jtorkkel opened this issue · comments
##Summary of the issue:
We have noticed that reported metrics contains multiple local_address, sometimes active count keep increasing and exceeding total
1. external host name with one or two port reported in local_address, port different that own listen port, normally 443 or 80 but can be also remote_address source port, tens of local_address host address can be found
reactor_netty_http_server_connections_total{instance="192.168.227.2:8080", local_address="192.168.227.2:8080", dnsname="example-service-v3"} 5
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="192.168.227.2:8080", dnsname="example-service-v3"} 5
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="example-service-v3.bbb.net.local:80", dnsname="example-service-v3"} 5
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="domainY.ccc.xxx.local:80", dnsname="example-service-v3"} 5
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="domainY.ccc.xxx.local:443:80", dnsname="example-service-v3"} 5
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="domainY.ccc.xxx.local:80", dnsname="example-service-v3"} 5
Services are running in Linux docker and in kubernetes, SpringBoot 2.7.x and 3.1.4
total count works correctly.
In few SpringBoot 2.7.8 docker service local host has contained remote port number
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="example-service-v3.bbb.net.local:12345:80", dnsname="example-service-v3"} 5
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="example-service-v3.bbb.net.local:12346:80", dnsname="example-service-v3"} 5
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="example-service-v3.bbb.net.local:12347:80", dnsname="example-service-v3"} 5
No Repro steps, cannot repro in local machine. But can be seen in all environments.
2. multiple local_address exposed, each reporting same active and total count instead of per local_address count
reactor_netty_http_server_connections_total{local_address="127.0.0.1:8082",springBoot="3.1.4",uri="http"} 21.0
reactor_netty_http_server_connections_total{local_address="0:0:0:0:0:0:0:1:8082",springBoot="3.1.4",uri="http"} 21.0
reactor_netty_http_server_connections_active{local_address="127.0.0.1:8082",springBoot="3.1.4",uri="http",ver="3.1.10-SNAPSHOT",} 21.0
reactor_netty_http_server_connections_active{local_address="0:0:0:0:0:0:0:1:8082",springBoot="3.1.4",uri="http"} 21.0
Reproduction steps:
Can be reproduced in localhost.
20 jMeter use IPv4 treads calling service API /xxx, persistent connections
Chrome browser calling prometheus /metrics endpoint, Chrome use IPv6
Separate local_address created for IPv4 and IPv6, IPv2 should contain up to 20, and IPv6 1, but combined results exposed to both local_address
3. sometimes active count explode, exceeding total count, tens of local_address host address can be found
reactor_netty_http_server_connections_total{instance="192.168.227.2:8080", local_address="192.168.227.2:8080", dnsname="example-service-v3"} 10
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="192.168.227.2:8080", dnsname="example-service-v3"} 4000
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="example-service-v3.bbb.net.local:80", dnsname="example-service-v3"} 4000
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="domainY.ccc.xxx.local:80", dnsname="example-service-v3"} 4000
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="domainY.ccc.xxx.local:443:80", dnsname="example-service-v3"} 4000
reactor_netty_http_server_connections_active{instance="192.168.227.2:8080", local_address="domainY.ccc.xxx.local:80", dnsname="example-service-v3", dnsname="example-service-v3"} 4000
total count works correctly.
Services are running in Linux docker, SpringBoot 2.7.x, only very few service.
No Repro steps.
Expected Behavior:
- only servers own IP-address or domain name reported (Seen in SpringBoot 2.7.x and 3.1.4), no random domain names reported
- two port number should not be reported.
- if multiple local_addresses then total and active should be per local_address, not so thate each combined (Seen in SpringBoot 2.7.x and 3.1.4)
- active should not keep increasing and exceed total (detected in Spring 2.7 only so far)
Actual Behavior:
We cannot explain where these hostnames with one or two 80/443/remote addr ports comes from. They comes from
- some X-Forwarded* headers
- metrics system try to resolve own IP-address to hostname
I tried to repro locally
- total use MicrometerChannelMetricsRecorder, recordServerConnectionOpened and recordServerConnectionClosed
- active use MicrometerHttpServerMetricsRecorder recordServerConnectionActive, recordServerConnectionInactive.
Both seems to to resolve socket address to local_Address using reactor.netty.Metrics.formatSocketAddress(localAddress)
We are using external DNS to resolve addresses and who knows what get returned if private docket/k8s server address used is reverse query because docket/k8s private address might be used outside cluster too. Services are access only through ingress/interlock etc.
In Windows localhost machine I try to repro this but formatSocketAddress InetSocketAddress.Java getHostString() end up calling getHostName() in InetAddress.Java which just return hostname/null instead of calling InetSocketAddress.Java which trigger DNS.
maybe in docker/k8s/linux it trigger DNS.
final class MicrometerHttpServerMetricsRecorder extends MicrometerHttpMetricsRecorder implements HttpServerMetricsRecorder {
@Override
public void recordServerConnectionActive(SocketAddress localAddress) {
LongAdder adder = getServerConnectionAdder(localAddress);
if (adder != null) {
adder.increment();
}
}
@Override
public void recordServerConnectionInactive(SocketAddress localAddress) {
LongAdder adder = getServerConnectionAdder(localAddress);
if (adder != null) {
adder.decrement();
}
}
@Nullable
private LongAdder getServerConnectionAdder(SocketAddress localAddress) {
String address = reactor.netty.Metrics.formatSocketAddress(localAddress);
return MapUtils.computeIfAbsent(activeConnectionsCache, address,
key -> {
Gauge gauge = filter(
Gauge.builder(CONNECTIONS_ACTIVE.getName(), activeConnectionsAdder, LongAdder::longValue)
.tags(HttpServerMeters.ConnectionsActiveTags.URI.asString(), PROTOCOL_VALUE_HTTP,
HttpServerMeters.ConnectionsActiveTags.LOCAL_ADDRESS.asString(), address)
.register(REGISTRY));
return gauge != null ? activeConnectionsAdder : null;
});
}
}
public class Metrics {
@Nullable
public static String formatSocketAddress(@Nullable SocketAddress socketAddress) {
if (socketAddress != null) {
if (socketAddress instanceof InetSocketAddress) {
InetSocketAddress address = (InetSocketAddress) socketAddress;
return address.getHostString() + ":" + address.getPort();
}
else {
return socketAddress.toString();
}
}
return null;
}
}
Possible Solution
- maybe simplest solution to address 1. is to remove local_address, or limit to numeric IP-address.
Your Environment
- Linux, Wndows
- Java 17
- SpringBoot 3.1.4
@jtorkkel Thanks for the report! Let me check the issue.
@jtorkkel I would recommend to update your Spring Boot 2.7.x version as we do have fixes related to these metrics (for example #2619). This might be the reason why you observe some issues on Spring Boot 2.7.x and not on Spring Boot 3.1.x
I'm working also on a fix for one of the issues reported here.
In Windows localhost machine I try to repro this but formatSocketAddress InetSocketAddress.Java getHostString() end up calling getHostName() in InetAddress.Java which just return hostname/null instead of calling InetSocketAddress.Java which trigger DNS.
maybe in docker/k8s/linux it trigger DNS.
There is a promise from JDK that getHostString() would never trigger DNS resolution, that's why we use this API. If you observe something else then this is a JDK issue.
We are in the process to migrate to SpringBoot 3.x but will take time until all services migrated. We skipped SpringBoot 3.0.x and started to use 3.1 directly, that is why propably why we did not notice active increase in SpringBoot 3.1.x.
One thing which confuse me is that
reactor_netty_http_server_connections_total never reports dns addresses, only IP-address. reactor_netty_http_server_connections_active reports almost always own dnsname plus other domain names in addition to ip-address.
"_active" is counted in
recordServerConnectionActive/recordServerConnectionInactive are in MicrometerHttpServerMetricsRecorder.class
whereas _total are counted in
recordServerConnectionOpened/recordServerConnectionClosed are from MicrometerChannelMetricsRecorder
This hints that issue is only happening in MicrometerHttpServerMetricsRecorder.class, but not clear why.
final class MicrometerHttpServerMetricsRecorder extends MicrometerHttpMetricsRecorder implements HttpServerMetricsRecorder {
public void recordServerConnectionActive(SocketAddress localAddress) {
LongAdder adder = this.getServerConnectionAdder(localAddress);
if (adder != null) {
adder.increment();
}
}
public void recordServerConnectionInactive(SocketAddress localAddress) {
LongAdder adder = this.getServerConnectionAdder(localAddress);
if (adder != null) {
adder.decrement();
}
}
@Nullable
private LongAdder getServerConnectionAdder(SocketAddress localAddress) {
String address = Metrics.formatSocketAddress(localAddress);
return (LongAdder)MapUtils.computeIfAbsent(this.activeConnectionsCache, address, (key) -> {
Gauge gauge = (Gauge)filter(Gauge.builder(HttpServerMeters.CONNECTIONS_ACTIVE.getName(), this.activeConnectionsAdder, LongAdder::longValue).tags(new String[]{ConnectionsActiveTags.URI.asString(), "http", ConnectionsActiveTags.LOCAL_ADDRESS.asString(), address}).register(Metrics.REGISTRY));
return gauge != null ? this.activeConnectionsAdder : null;
});
}
}
public class MicrometerChannelMetricsRecorder implements ChannelMetricsRecorder {
public void recordServerConnectionOpened(SocketAddress serverAddress) {
LongAdder totalConnectionAdder = this.getTotalConnectionsAdder(serverAddress);
if (totalConnectionAdder != null) {
totalConnectionAdder.increment();
}
}
public void recordServerConnectionClosed(SocketAddress serverAddress) {
LongAdder totalConnectionAdder = this.getTotalConnectionsAdder(serverAddress);
if (totalConnectionAdder != null) {
totalConnectionAdder.decrement();
}
}
@Nullable
protected static <M extends Meter> M filter(M meter) {
return meter instanceof NoopMeter ? null : meter;
}
public String name() {
return this.name;
}
public String protocol() {
return this.protocol;
}
@Nullable
private LongAdder getTotalConnectionsAdder(SocketAddress serverAddress) {
String address = Metrics.formatSocketAddress(serverAddress);
return (LongAdder)MapUtils.computeIfAbsent(this.totalConnectionsCache, address, (key) -> {
Gauge gauge = (Gauge)filter(Gauge.builder(this.name + ".connections.total", this.totalConnectionsAdder, LongAdder::longValue).tags(new String[]{ConnectionsTotalMeterTags.URI.asString(), this.protocol, ConnectionsTotalMeterTags.LOCAL_ADDRESS.asString(), address}).register(Metrics.REGISTRY));
return gauge != null ? this.totalConnectionsAdder : null;
});
}
}
Multiple ip-address like IPV4 and IPv6 is probably correct as tested in "local machine" but using LongAdder is not correct in my opinion as it result shared counters.
Multiple ip-address like IPV4 and IPv6 is probably correct as tested in "local machine" but using LongAdder is not correct in my opinion as it result shared counters.
That's correct. This can happen locally when server is bound on any local address. See #2953
One thing which confuse me is that
reactor_netty_http_server_connections_total never reports dns addresses, only IP-address. reactor_netty_http_server_connections_active reports almost always own dnsname plus other domain names in addition to ip-address.
This should be addressed with #2954