many exceptions after upgrade to spring boot 3.2 with apm agent 1.45.0
clockrun opened this issue · comments
We kindly ask to post all questions and issues on the Discuss forum first. In addition to awesome, knowledgeable community contributors, core APM developers are on the forums every single day to help you out as well, so your questions will reach a wider audience there. If we confirm that there is a bug, then you can open an issue in this repo.
Describe the bug
There are many exceptions printed out during startup after upgrade to spring boot 3.2.1. Our project is a simple spring cloud gateway project.
BTW, we are using this Java runtime image container-registry.oracle.com/graalvm/jdk:21-ol9
Steps to reproduce
Below is part of our pom.xml. When we build, we will use docker.build profile
<properties>
<java.version>21</java.version>
<spring-cloud.version>2023.0.0</spring-cloud.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-oauth2-client</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>co.elastic.logging</groupId>
<artifactId>logback-ecs-encoder</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.session</groupId>
<artifactId>spring-session-data-redis</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>build-info</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>code-coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.8</version>
<executions>
<execution>
<id>prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>coverage-report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>code-quality</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-pmd-plugin</artifactId>
<configuration>
<failOnViolation>true</failOnViolation>
<printFailingErrors>true</printFailingErrors>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.1.0</version>
</plugin>
<plugin>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs-maven-plugin</artifactId>
<version>3.1.12</version>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>docker.build</id>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-fabric8</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-fabric8-config</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-fabric8-loadbalancer</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>unpack</id>
<phase>package</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>${project.groupId}</groupId>
<artifactId>${project.artifactId}</artifactId>
<version>${project.version}</version>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<configuration>
<offline>true</offline>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>Expected behavior
startup application without error
Debug logs
Click to expand
2024-01-10T17:25:26.127392744+08:00 java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
2024-01-10T17:25:26.127964642+08:00 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-01-10T17:25:26.128114992+08:00 at java.base/java.lang.IndyBootstrapDispatcher.bootstrap(IndyBootstrapDispatcher.java:61)
2024-01-10T17:25:26.128122271+08:00 at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:147)
2024-01-10T17:25:26.128125037+08:00 at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:316)
2024-01-10T17:25:26.128127487+08:00 at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:274)
2024-01-10T17:25:26.128129929+08:00 at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:264)
2024-01-10T17:25:26.128144391+08:00 at co.elastic.apm.agent.bci.bytebuddy.Instrumented.isInstrumented(Instrumented.java:34)
2024-01-10T17:25:26.128147983+08:00 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2024-01-10T17:25:26.128151695+08:00 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-01-10T17:25:26.128154782+08:00 at co.elastic.apm.agent.bci.bytebuddy.InstallationListenerImpl.onAfterWarmUp(InstallationListenerImpl.java:65)
2024-01-10T17:25:26.128187460+08:00 at net.bytebuddy.agent.builder.AgentBuilder$InstallationListener$Compound.onAfterWarmUp(AgentBuilder.java:5419)
2024-01-10T17:25:26.128192560+08:00 at net.bytebuddy.agent.builder.AgentBuilder$Default$WarmupStrategy$Enabled.apply(AgentBuilder.java:11766)
2024-01-10T17:25:26.128196612+08:00 at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:11446)
2024-01-10T17:25:26.128199157+08:00 at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:11362)
2024-01-10T17:25:26.128201475+08:00 at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:300)
2
024-01-10T17:25:26.128204968+08:00 at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:180)
2024-01-10T17:25:26.128222577+08:00 at co.elastic.apm.agent.bci.ElasticApmAgent.initialize(ElasticApmAgent.java:166)
2024-01-10T17:25:26.128225926+08:00 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2024-01-10T17:25:26.128229174+08:00 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-01-10T17:25:26.128232307+08:00 at co.elastic.apm.agent.premain.AgentMain.loadAndInitializeAgent(AgentMain.java:157)
2024-01-10T17:25:26.128235437+08:00 at co.elastic.apm.agent.premain.AgentMain.init(AgentMain.java:98)
2024-01-10T17:25:26.128237765+08:00 at co.elastic.apm.agent.premain.AgentMain.premain(AgentMain.java:47)
2024-01-10T17:25:26.128239999+08:00 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2024-01-10T17:25:26.128256728+08:00 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-01-10T17:25:26.128260097+08:00 at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:560)
2024-01-10T17:25:26.128263124+08:00 at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:572)
2024-01-10T17:25:26.128433984+08:00 Caused by: java.util.ServiceConfigurationError: java.nio.file.spi.FileSystemProvider: Provider org.springframework.boot.loader.nio.file.NestedFileSystemProvider not found
2024-01-10T17:25:26.128441092+08:00 at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)
2024-01-10T17:25:26.128444762+08:00 at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1219)
2024-01-10T17:25:26.128536017+08:00 at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1228)
2024-01-10T17:25:26.128539824+08:00 at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.has
Next(ServiceLoader.java:1273)
2024-01-10T17:25:26.128542100+08:00 at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
2024-01-10T17:25:26.128544377+08:00 at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
2024-01-10T17:25:26.128546732+08:00 at java.base/java.nio.file.spi.FileSystemProvider.loadInstalledProviders(FileSystemProvider.java:156)
2024-01-10T17:25:26.128604532+08:00 at java.base/java.nio.file.spi.FileSystemProvider$1.run(FileSystemProvider.java:207)
2024-01-10T17:25:26.128617088+08:00 at java.base/java.nio.file.spi.FileSystemProvider$1.run(FileSystemProvider.java:204)
2024-01-10T17:25:26.128619621+08:00 at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
2024-01-10T17:25:26.128621930+08:00 at java.base/java.nio.file.spi.FileSystemProvider.installedProviders(FileSystemProvider.java:204)
2024-01-10T17:25:26.128763099+08:00 at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:336)
2024-01-10T17:25:26.128767677+08:00 at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:288)
at co.elastic.apm.agent.util.PackageScanner.doGetClassNames(PackageScanner.java:91)
at co.elastic.apm.agent.util.PackageScanner.getClassNames(PackageScanner.java:61)
at co.elastic.apm.agent.bci.IndyBootstrap.getClassNamesFromBundledPlugin(IndyBootstrap.java:543)
at co.elastic.apm.agent.bci.IndyBootstrap.internalBootstrap(IndyBootstrap.java:470)
at co.elastic.apm.agent.bci.IndyBootstrap.bootstrap(IndyBootstrap.java:415)
2024-01-10T17:25:26.128792069+08:00 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
... 26 more
2024-01-10T17:25:26.129412656+08:00 java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
2024-01-10T17:25:26.129567906+08:00 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.lang.IndyBootstrapDispatcher.bootstrap(IndyBootstrapDispatcher.java:61)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:147)
2024-01-10T17:25:26.129592353+08:00 at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:316)
2024-01-10T17:25:26.129619864+08:00 at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:274)
2024-01-10T17:25:26.129623069+08:00 at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:264)
2024-01-10T17:25:26.129625255+08:00 at co.elastic.apm.agent.bci.bytebuddy.Instrumented.isInstrumented(Instrumented.java:34)
2024-01-10T17:25:26.129634843+08:00 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2024-01-10T17:25:26.129637482+08:00 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-01-10T17:25:26.129779763+08:00 at co.elastic.apm.agent.bci.bytebuddy.InstallationListenerImpl.onAfterWarmUp(InstallationListenerImpl.java:65)
2024-01-10T17:25:26.129807594+08:00 at net.bytebuddy.agent.builder.AgentBuilder$InstallationListener$Compound.onAfterWarmUp(AgentBuilder.java:5419)
2024-01-10T17:25:26.129810569+08:00 at net.bytebuddy.agent.builder.AgentBuilder$Default$WarmupStrategy$Enabled.apply(AgentBuilder.java:11766)
2024-01-10T17:25:26.129848549+08:00 at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:11446)
2024-01-10T17:25:26.129854400+08:00 at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:11362)
2024-01-10T17:25:26.129973127+08:00 at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:300)
2024-01-10T17:25:26.129984131+08:00 at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:180)
2024-01-10T17:25:26.129986989+08:00 at co.elastic.apm.agent.bci.ElasticApmAgent.initialize(ElasticApmAgent.java:166)
2024-01-10T17:25:26.129990284+08:00 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleA
ccessor.java:103)
2024-01-10T17:25:26.130000721+08:00 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-01-10T17:25:26.130177383+08:00 at co.elastic.apm.agent.premain.AgentMain.loadAndInitializeAgent(AgentMain.java:157)
2024-01-10T17:25:26.130195074+08:00 at co.elastic.apm.agent.premain.AgentMain.init(AgentMain.java:98)
2024-01-10T17:25:26.130197811+08:00 at co.elastic.apm.agent.premain.AgentMain.premain(AgentMain.java:47)
2024-01-10T17:25:26.130200409+08:00 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2024-01-10T17:25:26.130202866+08:00 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-01-10T17:25:26.130205429+08:00 at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:560)
2024-01-10T17:25:26.130207834+08:00 at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:572)
2024-01-10T17:25:26.130213985+08:00 Caused by: java.lang.Error: Circular loading of installed providers detected
2024-01-10T17:25:26.130286647+08:00 at java.base/java.nio.file.spi.FileSystemProvider.installedProviders(FileSystemProvider.java:198)
2024-01-10T17:25:26.130290805+08:00 at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:336)
2024-01-10T17:25:26.130301136+08:00 at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:288)
2024-01-10T17:25:26.130330503+08:00 at co.elastic.apm.agent.util.PackageScanner.doGetClassNames(PackageScanner.java:91)
2024-01-10T17:25:26.130333892+08:00 at co.elastic.apm.agent.util.PackageScanner.getClassNames(PackageScanner.java:61)
2024-01-10T17:25:26.130344237+08:00 at co.elastic.apm.agent.bci.IndyBootstrap.getClassNamesFromBundledPlugin(IndyBootstrap.java:543)
2024-01-10T17:25:26.130360408+08:00 at co.elastic.apm.agent.bci.IndyBootstrap.internalBootstrap(IndyBootstrap.java:470)
2024-01-10T17:25:26.130392142+08:00 at co.elastic.apm.agent.bci.IndyBootstrap.bootstrap(IndyBootstrap.
java:415)
The corresponding error seems to not be related to the elastic-apm-agent, the agent is just the first to access FileSystems.newFileSystem which surfaces the error:
java.util.ServiceConfigurationError: java.nio.file.spi.FileSystemProvider: Provider org.springframework.boot.loader.nio.file.NestedFileSystemProvider not found
Apparently on your classpath an additional FileSystemProvider from spring is registered, which however does not exist. This causes failures when accessing FileSystems. So looks more like a classpath problem of your application to me.
Therefore closing this issue, feel free to reopen in case you have some additional information indicating that this error is actually caused by the agent.
Is there a way to solve this issue? Getting the same error using the new Spring boot 3.2.1 and elastic.apm 1.45.
When downgrading to spring 3.1.X it all works.
As stated in my previous comment, this doesn't look like an apm-agent issue, but like a spring issue. See also this issue.
If you still think this is apm-agent-related and the linked issue does not help, please provide a runnable sample application to reproduce the issue.