Application does not start with BlockHound and ReactorDebugAgent installed on java 8

Question

Application does not start with BlockHound and ReactorDebugAgent installed on java 8

robotmrv opened this issue 5 years ago · comments

BlockHound and ReactorDebugAgent cannot work together on java 8.
Application fails with error

java.util.ServiceConfigurationError: com.sun.tools.attach.spi.AttachProvider: Provider sun.tools.attach.WindowsAttachProvider could not be instantiated
java.lang.ExceptionInInitializerError
Caused by: java.lang.IllegalStateException: Error during attachment using: reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@53aac487
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:400)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:374)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:342)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:328)
	at reactor.tools.agent.ReactorDebugAgent.init(ReactorDebugAgent.java:41)
	at com.example.demo.ReactorDebugApplication.<clinit>(ReactorDebugApplication.java:27)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at reactor.tools.shaded.net.bytebuddy.agent.Attacher.install(Attacher.java:99)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:395)
	... 5 more
Caused by: com.sun.tools.attach.AttachNotSupportedException: no providers installed
	at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:203)
	... 11 more
Exception in thread "main"

If I try to start application using java 11 or just disable BlockHound or ReactorDebugAgent (in the code and spring-boot integration) everything works fine.

Expected Behavior

application starts

Actual Behavior

application start fails

Steps to Reproduce

try to start application on java 8
https://github.com/robotmrv/reactor-debug-error

Possible Solution

Your Environment

Reactor version(s) used:
io.projectreactor:reactor-core:3.3.0.RELEASE
io.projectreactor:reactor-tools:3.3.0.RELEASE
io.projectreactor.tools:blockhound:1.0.1.RELEASE
Other relevant libraries versions (eg. netty, ...):
JVM version (javar -version):
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)
OS and version (eg uname -a):
Win 10

Sergei Egorov · Answer 1 · Mon Nov 04 2019 00:36:47 GMT+0800 (China Standard Time)

HI @robotmrv,

Thanks for reporting!
Have you tried different combinations, e.g. "only BlockHound" or "only reactor-tools?

Just trying to understand whether it fails only if both are installed.

robotmrv · Answer 2 · Mon Nov 04 2019 00:48:19 GMT+0800 (China Standard Time)

Hi @bsideup,
If only one of BlockHound / ReactorDebugAgent is installed it works as expected

Sergei Egorov · Answer 3 · Tue Nov 05 2019 21:11:00 GMT+0800 (China Standard Time)

@robotmrv does it happen with BlockHound 1.0.0.RELEASE?

robotmrv · Answer 4 · Tue Nov 05 2019 21:43:49 GMT+0800 (China Standard Time)

@bsideup yes, the same with BlockHound 1.0.0.RELEASE

Sergei Egorov · Answer 5 · Tue Nov 05 2019 22:38:34 GMT+0800 (China Standard Time)

@robotmrv thanks for verifying! 👍 on it...

robotmrv · Answer 6 · Fri Nov 08 2019 17:56:33 GMT+0800 (China Standard Time)

Hi @bsideup
I've tested example application using linux based docker image openjdk:8

Linux c199ba7a5c0d 4.9.184-linuxkit #1 SMP Tue Jul 2 22:58:16 UTC 2019 x86_64 GNU/Linux
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

and got similar result.

So it is not Windows specific

 09:37:42.177 [main] DEBUG reactor.util.Loggers$LoggerFactory - Using Slf4j logging framework
 Exception in thread "main" java.lang.ExceptionInInitializerError
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
      at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
      at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
      at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
 Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Error during attachment using: reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@311d617d
      at reactor.blockhound.BlockHound$Builder.install(BlockHound.java:328)
      at reactor.blockhound.BlockHound.install(BlockHound.java:94)
      at com.example.demo.ReactorDebugApplication.<clinit>(ReactorDebugApplication.java:22)
      ... 8 more
 Caused by: java.lang.IllegalStateException: Error during attachment using: reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@311d617d
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:608)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:581)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:533)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:510)
      at reactor.blockhound.BlockHound$Builder.install(BlockHound.java:309)
      ... 10 more
 Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at reactor.shaded.net.bytebuddy.agent.Attacher.install(Attacher.java:106)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:603)
      ... 14 more
 Caused by: java.lang.UnsatisfiedLinkError: Native Library /usr/local/openjdk-8/jre/lib/amd64/libattach.so already loaded in another classloader
      at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1900)
      at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1838)
      at java.lang.Runtime.loadLibrary0(Runtime.java:870)
      at java.lang.System.loadLibrary(System.java:1122)
      at sun.tools.attach.LinuxVirtualMachine.<clinit>(LinuxVirtualMachine.java:342)
      at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
      at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
      ... 20 more

Sergei Egorov · Answer 7 · Fri Nov 08 2019 21:27:00 GMT+0800 (China Standard Time)

Caused by: java.lang.UnsatisfiedLinkError: Native Library /usr/local/openjdk-8/jre/lib/amd64/libattach.so already loaded in another classloader

Interesting...

/cc @raphw

Rafael Winterhalter · Answer 8 · Fri Nov 08 2019 22:09:15 GMT+0800 (China Standard Time)

Probably a bieffect of shading with the way attach works on Java 8 where tools.jar must be loaded into a new class loader. Byte Buddy checks if it already loaded tools.jar previously and reuses that attachment if possible but with the shading, I assume another, unshaded copy of Byte Buddy attempts the same what is impossible in Java 8 as the native code in tools.jar can only loaded once. What other tools you use load tools.jar? (Set a break point in com.sun.tools.attach.VirtualMachine.)

The solutions:

Use emulated attach (add JNA). It does not set any VM global state.
Avoid shading Byte Buddy if an unshaded copy is attempting the attach.
Upgrade to Java 9+ where this is fixed in the JVM.

Sergei Egorov · Answer 9 · Mon Nov 11 2019 18:52:54 GMT+0800 (China Standard Time)

@raphw thank you for the explanation!

I just tried adding JNA but it still fails, this time with:

Caused by: java.lang.IllegalStateException: Error during attachment using: reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@66498326
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:400)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:374)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:342)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:328)
	at reactor.tools.agent.ReactorDebugAgent.init(ReactorDebugAgent.java:41)
	at com.example.demo.ReactorDebugApplication.<clinit>(ReactorDebugApplication.java:22)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at reactor.tools.shaded.net.bytebuddy.agent.Attacher.install(Attacher.java:99)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:395)
	... 5 more
Caused by: java.lang.UnsatisfiedLinkError: Native Library /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre/lib/libattach.dylib already loaded in another classloader
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1907)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1845)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at sun.tools.attach.BsdVirtualMachine.<clinit>(BsdVirtualMachine.java:307)
	at sun.tools.attach.BsdAttachProvider.attachVirtualMachine(BsdAttachProvider.java:63)
	at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
	... 11 more

Rafael Winterhalter · Answer 10 · Tue Nov 12 2019 10:22:03 GMT+0800 (China Standard Time)

In this case you need to tell Byte Buddy explicitly to use attachment emulation rather then the built-in attachment. It will always prefer the latter if available and this exception occurs a bit too late in the process to fall back to something else, unfortunately.

Sergei Egorov · Answer 11 · Wed Nov 20 2019 22:18:22 GMT+0800 (China Standard Time)

It seems that there is no easy fix we can do in BlockHound or reactor-tools and we're affected by raphw/byte-buddy#670, the same way as seen in Kotlin/kotlinx.coroutines#1060

As a workaround, I would suggest using Java 9+ for running the tests

Sergei Egorov · Answer 12 · Wed Nov 20 2019 22:23:40 GMT+0800 (China Standard Time)

Also Mockk:
mockk/mockk#15

robotmrv · Answer 13 · Mon Sep 21 2020 04:57:02 GMT+0800 (China Standard Time)

@bsideup is it possible to add original artifact without shading (the same way as reactor-tools publishes it now). It seems that usage of the same non-shaded ByteBuddyAgent should solve the problem

Sergei Egorov · Answer 14 · Mon Oct 19 2020 23:40:54 GMT+0800 (China Standard Time)

@robotmrv yeah, I think we could do that. Alternatively, JNA can be added to classpath to make BB emulate the attachment.