Spring_security_filterchains "after" longTaskTimer run forever and count increase constantly after external cancel() or random internal cancel()

Question

Spring_security_filterchains "after" longTaskTimer run forever and count increase constantly after external cancel() or random internal cancel()

jtorkkel opened this issue 9 months ago · comments

Current situation:

We are using SpringBoot 3.1 in preproduction and we have noticed that new longTaskTimermetrics "spring_security_filterchains_active_seconds_active_count{spring_security_reached_filter_name="none",spring_security_reached_filter_section="after"}" keep increasing constantly.

In 24h tests where we made 50 Million requests to service using ~20 parallel threads using connection pools, "spring_security_filterchains_active_seconds_active_count increases to value 8000, after 10d the value was still 8000 (server was running, no more requests).

Same issue happen when test client like Jmeter is forced to "stop" requests before getting response. When repeated > 10000 times the spring_security_filterchains_active_seconds_active_count increase to over 10000 indicating no max amount. Each longTaskTimeris stored to umeter registry, so thhis is considered as memory leak and eventually service will OOM or slow down.

When connection pool renew the socket, it close the socket gracefully after using it and open new socket.

Our analysis indicates that

When rest controller use Mono, normally stream ends to onComplete(), but randomly there is onComplete()+onCancel() or just cancel().
When rest controller use Flux, stream end always to onComplete(), there are no extra internal cancel()
when external client cancel the requests (user switch apps in their mobile the) or requests timeouts the client forced stop result external cancel() as expected , also for Flux

stackTrace does not clearly indicate when the cancel comes from, it originates from

at reactor.core.publisher.Operators.terminate(Operators.java:1277)

but also success case originates from same "terminate"

In all these cases the cancel() cause the following issues

http_server_requests metrics reports metrics twice when onComplete+OnCancel is detected. One metrics reported as status="200, outcome="SUCCESS", another as status="UNKNOWN", outcome="UNKNOWN"
http_server_requests metrics reports status="UNKNOWN", outcome="UNKNOWN" when there is only cancel()
spring_security_filterchains_active_seconds_active_count "after" filter is always started, even when cancel() happen before filter started, and never stopped when there is cancel()
http_server_requests filter is always stopped, also with external and internal cancel()

Issue status:

was reported in #31365, fixed #31417 (OnComplete()+Cancel()), and related status="UNKNOWN fixed in #31388
not fixed, root cause for internal random cancel() not known. This is root cause most issues. In SpringFramework there was similar case #22952 where EncoderHttpMessageWriter was using Mono.from(flux) instead of flux.singleOrEmpty(). In SpringFW 5.2 encodec was changed Encoder#encodeValue but apparently issue still happens somewhere. StackTrace in all cases indicate that flow starts from reactor netty terminate
seems that spring_security_filterchains_active_seconds_active_count is always started, also after cancel(), but onCancel does not stop it, including external triggered cancel()

I am not sure where I should raise this case, and it might be still multiple different case.

Desired situation:

there should never be random cancel() when Mono used
spring_security_filterchains_active_seconds_active_count should handle the cancel and probably report status success/cancel in spring_security_filterchains_seconds_count metrics

spring_security_filterchains_active_seconds_active_count if used in production might request longTaskTimercount to increase to very high values, probably to millions.

cancel() is not impacting to business logic and actual data streams, in testing no errors seen in HTTP responses. It seems only to impact "observations".

Environment:

SpringBoot 3.1.4 (also happen with SpringBoot 3.0, and cancel() also seen similar way in 2.5, 2.7.14 where no observations --> no longRunningTimer
Java 17
Linux, reproduced in Windows. In windows cancel() rate might be close to % whereas in Linux might be ~0.1%. So VM specific. "Not happening with single thread"

Repro steps:
Example application included. example.zip.
It also contains

netty server and client metrics (checks the real number of requests)
custom filter metrics (check the number filters called)
fixes to status #31388 and custom fix to exception and outcome,
CustomDefaultMeterObservationHandler to print filter onStart/onStop
I did not get fix to #31417 working as my fix created additional DefaultMeterObservationHandler , but counted metrics did not include extra entries so apparently fix works.
multiple endpoints to with Plain, Digest, Mono flux and with and without webClient

Make few thousands calls to endpoint "GET /monoCancel" with minimum 10 threads. If no issue use 50 threads. Issue is not reproducing if less threads are used.
Also seems that in Windows this is much more frequent than in linux, in Windows even 5-10% of the calls might result cancel().

If too many console entries without cancel(), use "GET /monoCancelNoLOg", then you see stackTrace when cancel.
set also properties

example.CustomDefaultMeterObservationHandler=false
example.CustomWebFilter=false

it disables logging from webfilter, and extra onStart/onStop logging

    // prints whole reactive stream, onSubscribe, request, onNext, onComplete and onCancel and stackTrace 
    @GetMapping("monoCancel")
    public Mono<String> monoCancel(){
        return Mono.just("test").log().doOnCancel(() -> new Exception().printStackTrace());
    }

    // print only stackTrace when cancel
    @GetMapping("monoCancelNoLog")
    public Mono<String> monoCancelnoLog(){
        return Mono.just("test").doOnCancel(() -> new Exception().printStackTrace());
    }

There are also few other API

/monoCancelBasicAuth, with basic auth, but cancel() and active leak happen also without credentials in use
/fluxSuccess, issue not happening with Flux, unless external cancel
/fluxSuccessBasicAuth, flux with basic auth, no random cance()
/status, calls external rest http://localhost:11112/mx/status usingwebClient
/statusLocal, calls internal rest http://localhost:8081/mx/status usingwebClient
/statusFlux, use flux instead of Mono, no random cancel()
Use mock server to external server, adding delay allow cancel while waiting for response.

TEST 1:

Expected request with onComplete ONLY when using external webclient

// GET /status
2023-10-16T17:16:16.127+03:00 ERROR 26052 --- [ctor-http-nio-3] e.e.CustomDefaultMeterObservationHandler : STARTname=http.server.requests
2023-10-16T17:16:16.278+03:00  INFO 26052 --- [     parallel-1] reactor.Mono.Defer.1                     : onSubscribe([Fuseable] MonoFlatMap.FlatMapMain)
2023-10-16T17:16:16.280+03:00  INFO 26052 --- [     parallel-1] reactor.Mono.Defer.1                     : request(unbounded)
2023-10-16T17:16:16.468+03:00  INFO 26052 --- [ctor-http-nio-3] reactor.Mono.Defer.1                     : onComplete()
2023-10-16T17:16:16.468+03:00 ERROR 26052 --- [ctor-http-nio-3] e.e.CustomDefaultMeterObservationHandler : STARTname= spring.security.filterchains, spring.security.reached.filter.section=after
2023-10-16T17:16:16.468+03:00 ERROR 26052 --- [ctor-http-nio-3] e.e.CustomDefaultMeterObservationHandler : STOPname= spring.security.filterchains, spring.security.reached.filter.section=after
2023-10-16T17:16:16.469+03:00 ERROR 26052 --- [ctor-http-nio-3] e.e.CustomDefaultMeterObservationHandler : STOPname=http.server.requests

TEST 2:
With additional Cancel(), can be also

//GET /status
2023-10-16T17:18:59.027+03:00  INFO 26052 --- [    parallel-11] reactor.Mono.Defer.1459                  : onSubscribe([Fuseable] MonoFlatMap.FlatMapMain)
2023-10-16T17:18:59.027+03:00  INFO 26052 --- [    parallel-11] reactor.Mono.Defer.1459                  : request(unbounded)
2023-10-16T17:18:59.033+03:00  INFO 26052 --- [ctor-http-nio-2] reactor.Mono.Defer.1459                  : cancel()
2023-10-16T17:18:59.032+03:00  INFO 26052 --- [    parallel-11] reactor.Mono.Defer.1459                  : onComplete()


// GET /monoCancel
2023-10-16T14:33:25.202+03:00  INFO 6956 --- [     parallel-5] reactor.Mono.Just.574                    : | onSubscribe([Synchronous Fuseable] Operators.ScalarSubscription)
2023-10-16T14:33:25.203+03:00  INFO 6956 --- [     parallel-5] reactor.Mono.Just.574                    : | request(unbounded)
2023-10-16T14:33:25.203+03:00  INFO 6956 --- [     parallel-5] reactor.Mono.Just.574                    : | onNext(test)
2023-10-16T14:33:25.203+03:00  INFO 6956 --- [     parallel-5] reactor.Mono.Just.574                    : | onComplete()
2023-10-16T14:33:25.205+03:00  INFO 6956 --- [ctor-http-nio-4] reactor.Mono.Just.574                    : | cancel()

TEST 3: I cannot repro with example application, but easily repro with real service.
with cancel() only

2023-10-12T09:58:00.474+03:00  INFO [service-v3,,] 25340 --- [ctor-http-nio-3] reactor.Mono.Defer.11446                 : onSubscribe([Fuseable] MonoFlatMap.FlatMapMain)
2023-10-12T09:58:00.474+03:00  INFO [service-v3,,] 25340 --- [ctor-http-nio-3] reactor.Mono.Defer.11446                 : request(unbounded)
2023-10-12T09:58:00.630+03:00  INFO [service-v3,,] 25340 --- [ctor-http-nio-3] reactor.Mono.Defer.11446                 : cancel()
java.lang.Exception
	at com.example.example.ExampleController.lambda$monoCancel$0(ExampleController.java:27)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.cancel(FluxPeekFuseable.java:152)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.cancel(FluxMapFuseable.java:176)
	at reactor.core.publisher.MonoSingle$SingleSubscriber.doOnCancel(MonoSingle.java:108)
	at reactor.core.publisher.Operators$MonoInnerProducerBase.cancel(Operators.java:2931)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.cancel(FluxPeekFuseable.java:798)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.cancel(FluxDoFinally.java:134)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.cancel(FluxPeekFuseable.java:452)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.cancel(FluxPeekFuseable.java:452)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.FluxMap$MapSubscriber.cancel(FluxMap.java:169)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.cancel(FluxPeekFuseable.java:452)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.cancel(FluxDoOnEach.java:113)
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.cancel(FluxPeekFuseable.java:798)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators.terminate(Operators.java:1277)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:481)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)

TEST 4: external Cancel(), call canceled during slow external webclient call

2023-10-16T17:16:29.989+03:00 ERROR 26052 --- [ctor-http-nio-6] e.e.CustomDefaultMeterObservationHandler : STARTname=http.server.requests
2023-10-16T17:16:30.080+03:00  INFO 26052 --- [     parallel-2] reactor.Mono.Defer.2                     : onSubscribe([Fuseable] MonoFlatMap.FlatMapMain)
2023-10-16T17:16:30.080+03:00  INFO 26052 --- [     parallel-2] reactor.Mono.Defer.2                     : request(unbounded)
2023-10-16T17:16:30.995+03:00  WARN 26052 --- [ctor-http-nio-6] reactor.netty.channel.FluxReceive        : [22f5e991-1, L:/127.0.0.1:8081 - R:/127.0.0.1:34772] An exception has been observed post termination, use DEBUG level to see the full stack: java.net.SocketException: Connection reset
2023-10-16T17:16:30.995+03:00 ERROR 26052 --- [ctor-http-nio-6] e.e.CustomDefaultMeterObservationHandler : STOPname=http.server.requests
2023-10-16T17:16:30.996+03:00 ERROR 26052 --- [ctor-http-nio-6] e.e.CustomDefaultMeterObservationHandler : STARTname= spring.security.filterchains, spring.security.reached.filter.section=after
2023-10-16T17:16:30.996+03:00  INFO 26052 --- [ctor-http-nio-6] reactor.Mono.Defer.2                     : cancel()
java.lang.Exception
at com.example.example.ExampleController.lambda$status$3(ExampleController.java:69)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.cancel(FluxPeekFuseable.java:152)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.cancel(FluxMapFuseable.java:176)
	at reactor.core.publisher.MonoSingle$SingleSubscriber.doOnCancel(MonoSingle.java:108)
	at reactor.core.publisher.Operators$MonoInnerProducerBase.cancel(Operators.java:2931)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.cancel(FluxPeekFuseable.java:798)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.cancel(FluxDoFinally.java:134)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.cancel(FluxPeekFuseable.java:452)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.cancel(FluxPeekFuseable.java:452)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.FluxMap$MapSubscriber.cancel(FluxMap.java:169)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:199)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.cancel(FluxPeekFuseable.java:452)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.cancel(FluxDoOnEach.java:113)
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.cancel(FluxPeekFuseable.java:798)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators.terminate(Operators.java:1277)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:481)
	at reactor.netty.http.server.HttpServerOperations.onInboundClose(HttpServerOperations.java:696)
	at reactor.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:73)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
	at reactor.netty.http.server.AbstractHttpServerMetricsHandler.channelInactive(AbstractHttpServerMetricsHandler.java:107)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376)
	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)

TEST 5: active count reach 135.0

calling GET /monoCancelNoLog
Send large number of parallel requests, for example after 5000 requests, 10 threads, 500 requests each, sent in 3s.

Logs shows 136 exception due cancel()
metrics shown

5126 http_server_server_requests metrics entries matching uri,
4836 with status="200, outcome="SUCCESS"
190 with status="UNKNOWN", outcome="UNKNOWN"
Extra 26 are due (#31365), fixed #31417 (OnComplete()+Cancel()), and related 26 status="UNKNOWN fixed in #31388, rest 116 are just cancel() without onComplete

these 136 cancel() will in addition to already fixed http_server_requests issue result

164 http_server_requests metrics reported outcome="UNKNOWN"
135 spring_security_filterchains_active_seconds_active_count{spring_security_reached_filter_name="none",spring_security_reached_filter_section="after"} longRunningTasks which are never completed

Detailed metrics.

custom_channel_invocations_total{source="None",uri="/monoCancelNoLog",} 5000.0
reactor_netty_http_server_data_sent_bytes_count{uri="/",} 5000.0
reactor_netty_http_server_data_received_time_seconds_count{method="GET",uri="/",} 5000.0

http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/monoCancelNoLog",} 4836.0
http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="UNKNOWN",status="UNKNOWN",uri="/monoCancelNoLog",} 190.0

spring_security_filterchains_seconds_count{error="none",spring_security_filterchain_position="13",spring_security_filterchain_size="13",spring_security_reached_filter_name="ServerWebExchangeReactorContextWebFilter",spring_security_reached_filter_section="after",} 4853.0

spring_security_filterchains_active_seconds_active_count{spring_security_filterchain_position="0",spring_security_filterchain_size="0",spring_security_reached_filter_name="none",spring_security_reached_filter_section="after",} 135.0
spring_security_filterchains_active_seconds_duration_sum{spring_security_filterchain_position="0",spring_security_filterchain_size="0",spring_security_reached_filter_name="none",spring_security_reached_filter_section="after",} 7701.453904792

TEST 6: active count 6.0

calling GET /monoCancel
1000 request, 10 threads, 100 each

2023-10-16T14:33:24.842+03:00 INFO 6956 --- [ parallel-5] reactor.Mono.Just.1 : | onSubscribe([Synchronous Fuseable] Operators.ScalarSubscription)
2023-10-16T14:33:24.846+03:00 INFO 6956 --- [ parallel-5] reactor.Mono.Just.1 : | request(unbounded)
2023-10-16T14:33:24.847+03:00 INFO 6956 --- [ parallel-1] reactor.Mono.Just.1 : | onNext(test)
2023-10-16T14:33:24.855+03:00 INFO 6956 --- [ parallel-1] reactor.Mono.Just.1 : | onComplete()

2023-10-16T14:33:25.058+03:00 INFO 6956 --- [ parallel-11] reactor.Mono.Just.262 : | onSubscribe([Synchronous Fuseable] Operators.ScalarSubscription)
2023-10-16T14:33:25.058+03:00 INFO 6956 --- [ parallel-11] reactor.Mono.Just.262 : | request(unbounded)
2023-10-16T14:33:25.058+03:00 INFO 6956 --- [ parallel-11] reactor.Mono.Just.262 : | onNext(test)
2023-10-16T14:33:25.058+03:00 INFO 6956 --- [ parallel-11] reactor.Mono.Just.262 : | onComplete()
2023-10-16T14:33:25.064+03:00 INFO 6956 --- [ctor-http-nio-6] reactor.Mono.Just.262 : | cancel()

2023-10-16T14:33:25.110+03:00 INFO 6956 --- [ parallel-8] reactor.Mono.Just.352 : | onSubscribe([Synchronous Fuseable] Operators.ScalarSubscription)
2023-10-16T14:33:25.110+03:00 INFO 6956 --- [ parallel-8] reactor.Mono.Just.352 : | request(unbounded)
2023-10-16T14:33:25.110+03:00 INFO 6956 --- [ parallel-8] reactor.Mono.Just.352 : | onNext(test)
2023-10-16T14:33:25.110+03:00 INFO 6956 --- [ parallel-8] reactor.Mono.Just.352 : | onComplete()
2023-10-16T14:33:25.115+03:00 INFO 6956 --- [tor-http-nio-10] reactor.Mono.Just.352 : | cancel()

2023-10-16T14:33:25.138+03:00 INFO 6956 --- [ parallel-9] reactor.Mono.Just.416 : | onSubscribe([Synchronous Fuseable] Operators.ScalarSubscription)
2023-10-16T14:33:25.138+03:00 INFO 6956 --- [ parallel-9] reactor.Mono.Just.416 : | request(unbounded)
2023-10-16T14:33:25.138+03:00 INFO 6956 --- [ parallel-9] reactor.Mono.Just.416 : | onNext(test)
2023-10-16T14:33:25.138+03:00 INFO 6956 --- [ parallel-9] reactor.Mono.Just.416 : | onComplete()
2023-10-16T14:33:25.142+03:00 INFO 6956 --- [ctor-http-nio-4] reactor.Mono.Just.416 : | cancel()

2023-10-16T14:33:25.202+03:00 INFO 6956 --- [ parallel-5] reactor.Mono.Just.574 : | onSubscribe([Synchronous Fuseable] Operators.ScalarSubscription)
2023-10-16T14:33:25.203+03:00 INFO 6956 --- [ parallel-5] reactor.Mono.Just.574 : | request(unbounded)
2023-10-16T14:33:25.203+03:00 INFO 6956 --- [ parallel-5] reactor.Mono.Just.574 : | onNext(test)
2023-10-16T14:33:25.203+03:00 INFO 6956 --- [ parallel-5] reactor.Mono.Just.574 : | onComplete()
2023-10-16T14:33:25.205+03:00 INFO 6956 --- [ctor-http-nio-4] reactor.Mono.Just.574 : | cancel()

2023-10-16T14:33:25.250+03:00 INFO 6956 --- [ parallel-1] reactor.Mono.Just.695 : | onSubscribe([Synchronous Fuseable] Operators.ScalarSubscription)
2023-10-16T14:33:25.251+03:00 INFO 6956 --- [ parallel-1] reactor.Mono.Just.695 : | request(unbounded)
2023-10-16T14:33:25.251+03:00 INFO 6956 --- [ parallel-1] reactor.Mono.Just.695 : | onNext(test)
2023-10-16T14:33:25.251+03:00 INFO 6956 --- [ parallel-1] reactor.Mono.Just.695 : | onComplete()
2023-10-16T14:33:25.260+03:00 INFO 6956 --- [tor-http-nio-12] reactor.Mono.Just.695 : | cancel()

http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/monoCancel",} 989.0
http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="UNKNOWN",status="UNKNOWN",uri="/monoCancel",} 18.0

reactor_netty_http_server_response_time_seconds_count{method="GET",status="200",uri="/",} 1000.0
reactor_netty_http_server_data_sent_time_seconds_count{method="GET",status="200",uri="/",} 1000.0
custom_channel_invocations_total{source="None",uri="/monoCancel",} 1000.0

spring_security_filterchains_seconds_count{error="none",spring_security_filterchain_position="13",spring_security_filterchain_size="13",spring_security_reached_filter_name="ServerWebExchangeReactorContextWebFilter",spring_security_reached_filter_section="after",} 993.0

spring_security_filterchains_active_seconds_active_count{spring_security_filterchain_position="0",spring_security_filterchain_size="0",spring_security_reached_filter_name="none",spring_security_reached_filter_section="after",} 6.0
spring_security_filterchains_active_seconds_duration_sum{spring_security_filterchain_position="0",spring_security_filterchain_size="0",spring_security_reached_filter_name="none",spring_security_reached_filter_section="after",} 2885.6595331

Violeta Georgieva commented 9 months ago

yes sure

Violeta Georgieva · Answer 1 · Mon Oct 16 2023 23:47:02 GMT+0800 (China Standard Time)

@jtorkkel As it is suggested here spring-projects/spring-framework#31365 (comment)
Please create an issue in the Spring Security project.

jtorkkel · Answer 2 · Tue Oct 17 2023 00:19:20 GMT+0800 (China Standard Time)

Thanks, I think there is two separate issue.

random Mono cancel --> spring FW, reactor, reactor netty ?
missing cancel handling for spring.security.filterchains, both random Mono cancel() and external cancel() should be handled

The random cancel is root cause in my opinion, but not clear where is the issue now. External cancel can and should not be avoided. Both need proper cancel() handling in observation but unnecessary random cancel() should be avoided as it will result inconsistent metrics too.

I can create spring security issue for 2. but not clear where is 1.

I already had case for spring framework regarding 1. and it was closed after fixing 2 http_server_requests related fixes.

Violeta Georgieva · Answer 3 · Tue Oct 17 2023 00:35:08 GMT+0800 (China Standard Time)

For the moment I don't see how Reactor Netty is involved in this issue

Violeta Georgieva · Answer 4 · Tue Oct 17 2023 14:14:22 GMT+0800 (China Standard Time)

@jtorkkel As this is related to metrics in Spring Security, I'm closing this issue. Please open an issue in Spring Security project. If you think Reactor Netty issues random cancel signals, please open a new issue and if possible provide a reproducible example that demonstrates this.

Violeta Georgieva · Answer 5 · Tue Oct 17 2023 14:23:41 GMT+0800 (China Standard Time)

About reactor.netty.channel.ChannelOperations.terminate invocation - we always guarantee that this is invoked in an even loop, the two stack traces above show that it is either executed immediately or scheduled to be executed in an event loop.

jtorkkel · Answer 6 · Tue Oct 17 2023 15:30:16 GMT+0800 (China Standard Time)

I have not been able to identify where the extra random cancel() comes from. Reactor debugging is out of my skip set.

I know it happen frequently impacting metrics/observations.

I can only reopen specific cancel() issue to spring framework or reactor netty as it is so easy to reproduce and maybe you could trace what cause it.

jtorkkel · Answer 7 · Tue Oct 17 2023 17:00:31 GMT+0800 (China Standard Time)

We have done some more analysis and we suspect that reactor netty ChannelOperations terminate() sometimes trigger extra cancel()

This exception indicates that cancel probably comes from ChannelOperations.

Normal

cancel() normally comes +5ms after onComplete(), which might explain why happen more frequently in Windows

2023-10-17T11:15:25.705+03:00  INFO 9724 --- [     parallel-8] reactor.Mono.Just.129                    : | onSubscribe([Synchronous Fuseable] Operators.ScalarSubscription)
2023-10-17T11:15:25.705+03:00  INFO 9724 --- [     parallel-8] reactor.Mono.Just.129                    : | request(unbounded)
2023-10-17T11:15:25.705+03:00  INFO 9724 --- [     parallel-8] reactor.Mono.Just.129                    : | onNext(monoCancel)
2023-10-17T11:15:25.705+03:00  INFO 9724 --- [     parallel-8] reactor.Mono.Just.129                    : | onComplete()
2023-10-17T11:15:25.711+03:00  INFO 9724 --- [ctor-http-nio-4] reactor.Mono.Just.129                    : | cancel()

Should I post new issue or ...

jtorkkel · Answer 8 · Tue Oct 17 2023 17:17:57 GMT+0800 (China Standard Time)

I added nettyChannelOperation=TRACE logs and now when cancel is happening I can see extra "eaception" logging

full logs having 5 cancel

channelOperationsExceptionCancel.txt

Violeta Georgieva · Answer 9 · Tue Oct 17 2023 19:03:18 GMT+0800 (China Standard Time)

Should I post new issue or ...

Yes please do so