Issues running on Java 8 and aarch64
andrewserff opened this issue · comments
I'm having a very weird issue which I think I've narrowed down to something to do with the thymeleaf layout dialect. When running our webapp on Java 8 (Zulu 8.66.0.15-CA-macos-aarch64) on an M1 (aarch64) Mac, when we try to load a page, it is crashing the JVM with the following error:
=============== DEBUG MESSAGE: illegal bytecode sequence - method not verified ================
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x0000000102435354, pid=75488, tid=0x0000000000009803
#
# JRE version: OpenJDK Runtime Environment (Zulu 8.66.0.15-CA-macos-aarch64) (8.0_352-b08) (build 1.8.0_352-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.352-b08 mixed mode bsd-aarch64 compressed oops)
# Problematic frame:
# j org.codehaus.groovy.runtime.callsite.AbstractCallSite.<init>(Lorg/codehaus/groovy/runtime/callsite/CallSite;)V+0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/.../hs_err_pid75488.log
#
# If you would like to submit a bug report, please visit:
# http://www.azul.com/support/
#
The reason I think it's related to thymeleaf-layout-dialect is looking in the hs-err-pid...log I see the following:
Internal exceptions (10 events):
Event: 21.491 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': nz/net/ultraq/thymeleaf/layoutdialect/decorators/DecorateProcessorBeanInfo> (0x000000071c895a98) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/sy
Event: 21.492 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': org/thymeleaf/processor/element/AbstractAttributeModelProcessorBeanInfo> (0x000000071c8c44f8) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/syste
Event: 21.492 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': org/thymeleaf/processor/element/AbstractElementModelProcessorBeanInfo> (0x000000071c8f29c8) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/systemD
Event: 21.492 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': org/thymeleaf/processor/element/AbstractElementModelProcessorCustomizer> (0x000000071c9217c8) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/syste
Event: 21.493 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': org/thymeleaf/processor/element/AbstractAttributeModelProcessorCustomizer> (0x000000071c9556f8) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/sys
Event: 21.493 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': nz/net/ultraq/thymeleaf/layoutdialect/decorators/DecorateProcessorCustomizer> (0x000000071c988630) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/
Event: 21.494 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': nz/net/ultraq/thymeleaf/layoutdialect/includes/IncludeProcessorBeanInfo> (0x000000071c9f5848) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/syste
Event: 21.495 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': nz/net/ultraq/thymeleaf/layoutdialect/includes/IncludeProcessorCustomizer> (0x000000071ca24b30) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/sys
Event: 21.496 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': org/slf4j/LoggerFactoryBeanInfo> (0x000000071ca752d8) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 220]
Event: 21.496 Thread 0x0000000147971000 Exception <a 'java/lang/ClassNotFoundException': org/slf4j/LoggerFactoryCustomizer> (0x000000071ca920e8) thrown at [/Users/tester/jenkins/workspace/zulu-platform-build/zulu-src.git/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 220]
This all started happening with our upgrade to Spring Boot 2.7.x. I have tried upgrading all the thymeleaf dependencies to 3.1.0 (although does layoutdialect work with 3.1.0 yet?). I've also tried to make sure we are using Groovy 4.x as well (which I wasn't before), but that doesn't seem to help either. Those classes don't seem to exist any where so I can only assume they are generated by groovy?
This doesn't seem to happen on x86 arch, only on the M1. Other pieces of Spring (REST controllers) seem to work fine. Do you have any idea of what could be going on here?
Hmm, no idea unfortunately 🤔 I've never tried that combination of things (Java 8 Zulu JDK on an M1 Mac), and I've never seen those classes either (Customzer? BeanInfo?). The build process doesn't generate such class files, so they're definitely missing. A brief search for customizer/beaninfo brought up Spring a few times, so it might be some reflection/instrumentation thing they're doing, can't tell for sure.
As for working w/ Thymeleaf 3.1.0 - I've managed to convert the benchmark application in this repo to using it via Spring Boot 3.0, and the layout dialect worked on my machine without having to make any code changes to it (Java 17 as Spring Boot requires that as a minimum now, Temirin JDK, M1 Mac). Same for a personal project on Spring Boot 2.7.6.
Do you think you could create and share a minimum project/repo that can reproduce this error? Since I've also got an M1 Mac, if I download a Zulu JDK then I might be able to get the error too? Note however that I'm no expert when it comes to JDK differences, or the internals of Spring, so even with reproducing the problem I might just be scratching my head anyway 😕
Hi, sorry for the delay in getting back to you. I just pushed a sample project that exhibits the problem. You can find it here:
https://github.com/everwatchsolutions/thymeleaf-layout-java8-crash-test
In the MVCConfig, if you comment out adding the Layout dialect (line 79), it doesn't crash. Add the layout dialect back in and it crashes once you load the UI. The page I have in there is super simple, doesn't even use the layout dialect and it crashes with the output above.
I'm running this on this JVM (which is the latest and only Java 8 VM i know of on arm):
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (Zulu 8.66.0.15-CA-macos-aarch64) (build 1.8.0_352-b08)
OpenJDK 64-Bit Server VM (Zulu 8.66.0.15-CA-macos-aarch64) (build 25.352-b08, mixed mode)
I just confirmed with someone else it does not core dump on the same VM on an Intel mac.
If you run the same project on Java 11, it does not core dump and works as expected.
If you think I should submit a bug with Azul or Spring let me know.
I just downgraded from layout dialect 3.1.0 to 3.0.0 and the problem doesn't happen any more. Groovy 4.0 related?
I also tried to upgrade groovy to 4.0.6 but that didn't help. Luckily we are able to downgrade to layout dialect 3.0.0 and that fixes our problem, but this will likely be a problem with future spring boot problems as I assume they will pull in your latest. So hopefully you can determine the cause of this.
Thanks for the sample project, and glad to know that you at least have workarounds for now. I'll check it out over this holiday period.
I was able to replicate the problem with your sample project, and it seems the combination of the bytecode created by Groovy 4 and the Zulu 8 JDK for Apple Silicon is the only way to trigger this bug. I tried the other JDKs I had installed (Java 8, 11, and 17 from Eclipse Adoptium), tried Windows, and even tried the x86 version of the Zulu 8 JDK running through Rosetta, and none of these would get the problem 😕
With the more thorough core dumps, I was able to get lines pointing to Groovy calling out to the LoggerFactory.getLogger methods of SLF4J in a static context. Commenting out those lines would fix it, but it did mean removing logging, so I managed to get a workaround in place that kept the logging.
I've uploaded a 3.2.0-SNAPSHOT version of the layout dialect with this fix. If you're able to test this out to confirm that it works for you too, that'd be appreciated. I'll create a proper 3.2.0 release soon.
Excellent, thanks for looking into it and glad you were able to determine the problem! Does the 3.2.0-SNAPSHOT include the changes to work with Thymeleaf 3.1.0 as well? Or should I try this with 3.0.15 still?
The snapshot version doesn't have any specific changes for Thymeleaf 3.1.0. I've found that the layout dialect works with Thymeleaf 3.1.0 without any modifications, so you should be able to use either Thymeleaf 3.0.15 or 3.1.0.
So Christmas / New Year's happened, and I kinda forgot I had this release to do! 😅 I'm going to release version 3.2.0, which includes the fix for this issue - it should show up on Maven Central in a few hours at most.