java.lang.ClassNotFoundException: com.sun.xml.bind.v2.ContextFactory Using Netty in Spring Webclient
apescione opened this issue · comments
Webclient of Spring Boot using HttpClient of netty throw a class not found exception only for 3 or more CPU core.
Expected Behavior
Everything works fine regardless of number of CPU cores as set.
Actual Behavior
My spring application caught this error as soon as I set up 3 or more CPUs in Kubernetes deployment.
Caused by: javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.
at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:232)
at javax.xml.bind.ContextFinder.find(ContextFinder.java:375)
at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:691)
at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:632)
at org.springframework.http.codec.xml.JaxbContextContainer.lambda$getJaxbContext$0(JaxbContextContainer.java:54)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source)
at org.springframework.http.codec.xml.JaxbContextContainer.getJaxbContext(JaxbContextContainer.java:52)
at org.springframework.http.codec.xml.JaxbContextContainer.createMarshaller(JaxbContextContainer.java:42)
at org.springframework.http.codec.xml.Jaxb2XmlEncoder.initMarshaller(Jaxb2XmlEncoder.java:145)
at org.springframework.http.codec.xml.Jaxb2XmlEncoder.encodeValue(Jaxb2XmlEncoder.java:126)
at org.springframework.http.codec.xml.Jaxb2XmlEncoder.lambda$encode$0(Jaxb2XmlEncoder.java:107)
at reactor.core.publisher.MonoCallable.call(MonoCallable.java:92)
at reactor.core.publisher.FluxCallable.call(FluxCallable.java:60)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:410)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:219)
at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onSubscribe(FluxTake.java:379)
at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onStateChange(DefaultPooledConnectionProvider.java:187)
at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnection.onStateChange(DefaultPooledConnectionProvider.java:444)
at reactor.netty.channel.ChannelOperationsHandler.channelActive(ChannelOperationsHandler.java:62)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:238)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelActive(CombinedChannelDuplexHandler.java:412)
at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:69)
at io.netty.channel.CombinedChannelDuplexHandler.channelActive(CombinedChannelDuplexHandler.java:211)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:260)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:238)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:231)
at reactor.netty.tcp.SslProvider$SslReadHandler.userEventTriggered(SslProvider.java:847)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:400)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368)
at io.netty.handler.logging.LoggingHandler.userEventTriggered(LoggingHandler.java:222)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368)
at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1848)
at io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:952)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1419)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: com.sun.xml.bind.v2.ContextFactory
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at javax.xml.bind.ServiceLoaderUtil.nullSafeLoadClass(ServiceLoaderUtil.java:92)
at javax.xml.bind.ServiceLoaderUtil.safeLoadClass(ServiceLoaderUtil.java:125)
at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:230)
... 65 common frames omitted
Steps to Reproduce
Code:
ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
.codecs(configurer -> {
configurer.defaultCodecs().enableLoggingRequestDetails(true);
configurer.defaultCodecs().jaxb2Encoder(new Jaxb2XmlEncoder());
configurer.defaultCodecs().jaxb2Decoder(new Jaxb2XmlDecoder());
})
.build();
HttpClient httpClient = HttpClient.create();
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.exchangeStrategies(exchangeStrategies)
.baseUrl(url)
Testing HttpClient Api, we notice that:
HttpClient.create() ---> it doesn't work
HttpClient.from(TcpClient.create()) --> it works but *from* Api is deprecated
HttpClient.create().runOn(configuration.loopResources(), configuration.isPreferNative()) ---> it works
Possible Solutions
we have difference alternative solutions:
- Change HttpClient API for creation
- Serialize XML using Jackson instead of JAXB
- Use Jetty instead of Netty as HttpClient for Webclient
But we want to find the root cause that causes the problem to avoid that can influence other things or scenarios.
Your Environment
Spring boot application and some Azure Java SDK.
Spring boot version: 2.6.15
- Reactor version(s) used: 1.0.24
- Netty: 4.1.92.Final
- Kubernates: 1.27.3
- com.microsoft.azure:msal4j: 1.13.8
- azure-core: 1.39.0
- Docker image from: alpine:3.18.4
- JVM version (
java -version): microsoft-jdk-11.0.20.1-alpine-x64
We have also found this issue 2582, but it is not linked to the CPU number and there is a root cause.
@apescione Did you try the solutions in the mentioned issue? Reactor Netty does not change class loaders. In your application try to find the component that changes the class loaders.
CPU number does affect the number of threads used by Reactor Netty as the number of the threads that Reactor Netty uses is equal to the CPU number.
Hi @violetagg, the warmup solution doesn't work for me.
Moreover, it doesn't work when the used pool is "reactor-http-epoll-", instead it works when it is "reactor-tcp-epoll-".
What is the difference between each other?
Thank you
@apescione Please provide a reproducible example.
@apescione "reactor-http-epoll-" is the default thread pool used by all HTTP clients, while "reactor-tcp-epoll-" is the default for all TCP clients.
If you configure you HTTP client to use "reactor-tcp-epoll-", it will share this pool with the rest of TCP clients.
How can I choose another custom pool for my HttpClient? Just to avoid that some other libraries can have some issues about Classloader?
public class Application {
public static void main(String[] args) {
LoopResources loop = LoopResources.create("event-loop", 1, 4, true);
HttpClient client =
HttpClient.create()
.runOn(loop);
client.get()
.uri("https://example.com/")
.responseContent()
.aggregate()
.asString()
.block();
}
}
See more here https://projectreactor.io/docs/netty/release/reference/index.html#client-tcp-level-configurations-event-loop-group
Also take a look here for more info about how to do this in Spring Framework (ReactorResourceFactory) https://docs.spring.io/spring-framework/reference/web/webflux-webclient/client-builder.html#webflux-client-builder-reactor
Hi @violetagg , you code fit perfectly for my situation. Using my own event loop I can isolate it from other libraries/frameworks, and moreover, I can tune better for my clients. I can close the issue, and I want to thank you very much for quick and useful answer. Just for note, I've also found what is the library that produces this behavior. The library is Azure App Configuration (Azure App configuration)[https://learn.microsoft.com/en-us/java/api/overview/azure/spring-cloud-starter-appconfiguration-config-readme?view=azure-java-stable], it initializes the event pool in spring bootstrap phase (much earlier) because need to fetch from Azure Configuration all the needed properties to load Spring Boot Application. In this phase, the Jaxb classes are not still loaded. Initializing the event loop after, it works fine. Most likely, the more CPUs speed up the properties fetching and cause the issue. I'll open an issue to Azure SDK, to invite them to use their own event pool as a Best Practice to avoid other strange behaviors.
@apescione Thanks for the update!