Fatal exceptions not caught in onErrorDropped Hook

Question

Fatal exceptions not caught in onErrorDropped Hook

ResoluteError opened this issue 3 months ago · comments

In our current project, we have a reactive chain that consumes service bus message. When encountering a fatal exception (e.g. an OutOfMemory exception) within the chain, we need to gracefully shut down and restart the service. However, there seems to be no way of handling fatal exceptions. Other issues have referenced the onErrorDropped hook, but also does not seem to have the intended effect.

When encountering a fatal exception, the service currently just hangs and the chain no longer processes any SerivceBus messages.

Expected Behavior

There should be a way to catch fatal exceptions for graceful service termination.

Actual Behavior

Fatal exceptions seem to evade all forms of error handling.

Steps to Reproduce

@Test
void reproCase() {


    Hooks.onErrorDropped((err) -> {
      System.out.println("### An error was dropped"); // Never executed
    });

    Flux.range(0, 10)
        .delayElements(java.time.Duration.ofMillis(300))
        .flatMap(i -> {
          if (i == 5) {
            return Flux.error(new OutOfMemoryError("Test error"));
          } else {
            return Flux.just(i);
          }
        })
        .doOnError(err -> System.out.println("### An error occurred: " + err.getMessage())) // Never executed
        .doOnNext((i) -> System.out.println("Got " + i))
        .doOnComplete(() -> System.out.println("Completed"))
        .collectList()
        .block();
}

Returns logs:

Got 0
Got 1
Got 2
Got 3
Got 4
12:06:01.504 [parallel-6] WARN reactor.core.Exceptions -- throwIfFatal detected a jvm fatal exception, which is thrown and logged below:
java.lang.OutOfMemoryError: Test error
	at io.gfnw.nis.workflow.executor.debug.lambda$hooksTest$1(debug.java:20)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:388)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259)
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865)
	at reactor.core.publisher.MonoDelayUntil$DelayUntilCoordinator.complete(MonoDelayUntil.java:420)
	at reactor.core.publisher.MonoDelayUntil$DelayUntilTrigger.onComplete(MonoDelayUntil.java:533)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
12:06:01.512 [parallel-6] WARN reactor.core.Exceptions -- throwIfFatal detected a jvm fatal exception, which is thrown and logged below:
java.lang.OutOfMemoryError: Test error
	at io.gfnw.nis.workflow.executor.debug.lambda$hooksTest$1(debug.java:20)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:388)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259)
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865)
	at reactor.core.publisher.MonoDelayUntil$DelayUntilCoordinator.complete(MonoDelayUntil.java:420)
	at reactor.core.publisher.MonoDelayUntil$DelayUntilTrigger.onComplete(MonoDelayUntil.java:533)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
12:06:01.513 [parallel-6] WARN reactor.core.Exceptions -- throwIfFatal detected a jvm fatal exception, which is thrown and logged below:
java.lang.OutOfMemoryError: Test error
	at io.gfnw.nis.workflow.executor.debug.lambda$hooksTest$1(debug.java:20)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:388)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259)
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865)
	at reactor.core.publisher.MonoDelayUntil$DelayUntilCoordinator.complete(MonoDelayUntil.java:420)
	at reactor.core.publisher.MonoDelayUntil$DelayUntilTrigger.onComplete(MonoDelayUntil.java:533)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
12:06:01.513 [parallel-6] ERROR reactor.core.scheduler.Schedulers -- Scheduler worker in group main failed with an uncaught exception
java.lang.OutOfMemoryError: Test error
	at io.gfnw.nis.workflow.executor.debug.lambda$hooksTest$1(debug.java:20)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:388)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259)
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865)
	at reactor.core.publisher.MonoDelayUntil$DelayUntilCoordinator.complete(MonoDelayUntil.java:420)
	at reactor.core.publisher.MonoDelayUntil$DelayUntilTrigger.onComplete(MonoDelayUntil.java:533)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Possible Solution

Your Environment

reactor-core version 3.6.2
java version 21.0.1
OS: 23.2.0 Darwin Kernel Version 23.2.0 (ARGM64 - M1 Apple Silicone) | same behavior across OS
spring-cloud-azure-starter-servicebus 5.8.0
spring-boot-starter-webflux 3.2.2

As a side note, we are using the Spring @EventListener(ApplicationReadyEvent.class) lifecycle annotation to subscribe to the service bus. If there is a better way that would automatically propagate these errors and close the spring context & shut down the application without manual error handling, I'm happy for any suggestions.

I previously tried asking via StackOverflow but with no success.

Thank you for your support!

Dariusz Jędrzejczyk · Answer 1 · Tue Apr 30 2024 18:43:08 GMT+0800 (China Standard Time)

We can't do much in case of JVM fatal exceptions. There was at some point the logging handler added, that's why you see these logs. However, please refer to Simon's response and related discussions in that issue to understand more. Long story short, we can't handle Errors like Exceptions and propagate them via Reactor's mechanisms. You have to resort to traditional JVM control structures if you want to catch any Error. That is not recommended though and you should allow your JVM to exit as there might be no more memory left to even try to do any cleanup. External control is applicable - e.g. sizing your heap properly, responding to health checks (e.g. using Spring Boot Actuator's endpoint) and restarting the JVM using your orchestrator of choice (e.g. Kubernetes) if there's no timely response to the health checks.

Douglas Reiser · Answer 2 · Tue Apr 30 2024 19:19:43 GMT+0800 (China Standard Time)

Thanks for the explanation and immediate response - totally makes sense. Indeed using JVM controls via -XX:+ExitOnOutOfMemoryError works fine when encountering actual OOM errors. Much appreciated. :)