Blockhound + Opentelemetry agent fail on install
friscoMad opened this issue · comments
We are using the opentelemtry java agent via JVM params and when Blockhound tries to install it fails as it tries to instrument opentelemetry loaded code.
[Byte Buddy] ERROR java.util.concurrent.FutureTask [null, module java.base, Thread[main,5,main], loaded=true] │
│ reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Resolution$NoSuchTypeException: Cannot resolve type description for io.opentelemetry.javaagent.bootstrap.VirtualFieldInstalledMarker ││ at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:190) │
│ at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType.toErasure(TypePool.java:6931) ││ at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType$TokenList.get(TypePool.java:7012) │
│ at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType$TokenList.get(TypePool.java:6958) ││ at java.base/java.util.AbstractList$Itr.next(AbstractList.java:371) │
│ at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default.doAnalyze(MethodGraph.java:745) ││ at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default.compile(MethodGraph.java:668) │
│ at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$AbstractBase.compile(MethodGraph.java:519) ││ at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.inline.DecoratingDynamicTypeBuilder.toTypeWriter(DecoratingDynamicTypeBuilder.java:496) │
│ at reactor.blockhound.shaded.net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$UsingTypeWriter.make(DynamicType.java:4062) ││ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12510) │
│ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12445) ││ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12154) │
│ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12936) ││ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12866) │
│ at java.base/java.security.AccessController.doPrivileged(Native Method) ││ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java) │
│ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12388) ││ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source) │
│ at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188) ││ at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563) │
│ at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method) ││ at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167) │
│ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ││ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) │
│ at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ││ at java.base/java.lang.reflect.Method.invoke(Method.java:566) │
│ at reactor.blockhound.shaded.net.bytebuddy.utility.Invoker$Dispatcher.invoke(Unknown Source) ││ at reactor.blockhound.shaded.net.bytebuddy.utility.dispatcher.JavaDispatcher$Dispatcher$ForNonStaticMethod.invoke(JavaDispatcher.java:1032) │
│ at reactor.blockhound.shaded.net.bytebuddy.utility.dispatcher.JavaDispatcher$ProxiedInvocationHandler.invoke(JavaDispatcher.java:1162) ││ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.$Proxy86.retransformClasses(Unknown Source) │
│ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:8395) ││ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:8210) │
│ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(AgentBuilder.java:5907) ││ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:11521) │
│ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:11421) ││ at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$Delegator.installOn(AgentBuilder.java:13218) │
│ at reactor.blockhound.BlockHound$Builder.instrument(BlockHound.java:597)
Expected Behavior
Open telemetry and blockhound can work together
Actual Behavior
Instrumentation fails
Steps to Reproduce
java -javaagent:/opt/otel/opentelemetry-javaagent-all.jar
@Test
void reproCase() {
BlockHound.install()
}Possible Solution
This reported issue in ByteBuddy seems to be related:
raphw/byte-buddy#1320
Your Environment
- Reactor version(s) used:
- Other relevant libraries versions (eg.
netty, ...): opentelemetry 1.24 (I will do some test with a recent version in case it was fixed there) - JVM version (
java -version): 11 - OS and version (eg
uname -a):
Hi @friscoMad ,
Can you try to run your test like this: instead of calling BlockHound.install() from your test in the java code, declare the blockhound agent using a -javaagent option, like you do with your open telemetry agent.
but make sure you declare the BlockHound agent before the open telemetry agent.
it works for me when I run my sample application like this:
java -javaagent:blockhound-1.0.9.RELEASE.jar -javaagent:opentelemetry-javaagent-2.4.0-SNAPSHOT.jar -jar target/echo-0.0.1-SNAPSHOT.jar
note that if I declare the blockhound agent after the opentelemetry agent, then I reproduce the same problem you have reported.
I have tested with jdk-8, jdk-11, jdk-17, jdk-21, and jdk-22
do not forget to use -XX:+AllowRedefinitionToAddDeleteMethods in case you use a JDK13+ version.
let me know.
thanks.
Thanks, we are trying to avoid adding more agents to the command line, so our solution is to move the OTel agent to attach it on runtime using this contrib lib:
https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/runtime-attach/README.md
So far it appears to be working and both seem to be doing their thing, even when Blockhound is attached after OTel, which is interesting.
ok, I'm closing this issue, reopen it if necessary.
thanks.