ZstdConstants thows ExceptionInInitializerError on read-only volumes

[Alexander Silvennoinen]

Expected behavior

Like in Netty versions lower than 4.1.108.Final, using ZstdConstants should not throw an Exception on read-only volumes. Especially not if Zstd is not used at all.

Actual behavior

When the application that is using Netty 4.1.108.Final is running on a read-only volume, where no temporary java temp directory is available, it throws the following StackTrace, although Zstd is not used in http communication:

java.lang.ExceptionInInitializerError: Cannot unpack libzstd-jni-1.5.5-1: Read-only file system at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method) at java.base/java.io.File.createTempFile(File.java:2170) at com.github.luben.zstd.util.Native.load(Native.java:129) at com.github.luben.zstd.util.Native.load(Native.java:85) at com.github.luben.zstd.Zstd.<clinit>(Zstd.java:13) at io.netty.handler.codec.compression.ZstdConstants.<clinit>(ZstdConstants.java:25) at io.netty.handler.codec.compression.ZstdOptions.<clinit>(ZstdOptions.java:41) at io.netty.handler.codec.compression.StandardCompressionOptions.zstd(StandardCompressionOptions.java:54) at io.micronaut.http.server.netty.handler.Compressor.<init>(Compressor.java:62) at io.micronaut.http.server.netty.handler.PipeliningServerHandler.setCompressionStrategy(PipeliningServerHandler.java:142) at io.micronaut.http.server.netty.HttpPipelineBuilder$StreamPipeline.insertMicronautHandlers(HttpPipelineBuilder.java:622) at io.micronaut.http.server.netty.HttpPipelineBuilder$StreamPipeline.insertHttp1DownstreamHandlers(HttpPipelineBuilder.java:638) at io.micronaut.http.server.netty.HttpPipelineBuilder$ConnectionPipeline.configureForHttp1(HttpPipelineBuilder.java:380) at io.micronaut.http.server.netty.HttpPipelineBuilder$ConnectionPipeline.initChannel(HttpPipelineBuilder.java:299) at io.micronaut.http.server.netty.NettyHttpServer$Listener.initChannel(NettyHttpServer.java:892) at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1130) at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:840)

In versions lower than 4.1.108.Final, ZstdConstants were using plain numbers to initialize static constants.
In version 4.1.108.Final, ZstdConstants seems to use method calls to a zstd library to initialize these static constants, which fail on read-only volumes, because the zstd library tries to unpack libzstd-jni-1.5.5-1.

Applications based on e.g. Micronaut 4.3.7 are not even able to start anymore.

Netty version

4.1.108.Final

JVM version (e.g. java -version)

openjdk version "17.0.10" 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-17.0.10.7.1 (build 17.0.10+7-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.10.7.1 (build 17.0.10+7-LTS, mixed mode, sharing)

OS version (e.g. uname -a)

Linux c579b42069c4 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 Linux

[Chris Vest]

Does io.netty.handler.codec.compression.Zstd.isAvailable() return true on such read-only volumes?

[Daniel]

We run Micronaut 4.3.8 with Netty 4.1.108.Final in a Docker container with readOnlyRootFilesystem: true. I logged ZStd.isAvailable() before startup in the main method, the value is true.

[Violeta Georgieva]

Zstd#isAvailable() is a bit incomplete I think.
For example Brotli#isAvailable() checks also whether natives can be loaded, while there is no such check for Zstd

[Norman Maurer]

Sounds like something we need to fix in Zstd

[Daniel]

@AlexSilver9 We had exactly the same problem and simply excluded zstd-jni from our dependencies as a workaround for now, e.g.:

<dependency>
  <groupId>io.micronaut.kafka</groupId>
  <artifactId>micronaut-kafka</artifactId>
  <exclusions>
      <exclusion>
        <groupId>com.github.luben</groupId>
        <artifactId>zstd-jni</artifactId>
      </exclusion>
  </exclusions>
</dependency>

[Daniel]

Sounds like something we need to fix in Zstd

Maybe it's enough to check in Zstd#isAvailable() if com.github.luben.zstd.util.Native#load can create a temporary file?

[Norman Maurer]

@danielksb PRs welcome :)

[Daniel]

I will have some time later this day and try to provide a PR. But I am not sure, if this problem can be easily tested automatically.