Giters

xerial / snappy-java

Snappy compressor/decompressor for Java

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

org.xerial.snappy.SnappyError: [UNSUPPORTED_PLATFORM] pure-java snappy requires access to java.nio.Buffer raw address field

chiranth opened this issue · comments

commented

Hello Guys,

We recently upgraded to jdk 17 with Redhat as base image and ever since we started getting the below error
org.xerial.snappy.SnappyError: [UNSUPPORTED_PLATFORM] pure-java snappy requires access to java.nio.Buffer raw address field

snappy version : 1.1.8.4(works with jdk11 Redhat base image but not when we upgrade to jdk17 Redhat base image)

If i revert to jdk11 Redhat docker base image, the issue goes away. I was refering to #289 which is basically same but alpine based docker image is being used over there and the solution is wrt to a alpine package, What do i have to do in Redhat base docker mage for jdk 17? Appreciate any help on this

commented

We were able to workaround the problem by adding the runtime java env --add-opens=java.base/java.nio=ALL-UNNAMED which resolved the problem
Ref: https://stackoverflow.com/questions/69728301/unable-to-make-field-long-java-nio-buffer-address-accessible-module-java-base-d
https://blogs.oracle.com/javamagazine/post/its-time-to-move-your-applications-to-java-17-heres-why-and-heres-how

But the above seems like a last resort workaround as per the above link and probably might need the fix in snappy?

commented

We run a similar issue too, using Alpine as base Docker image. The solution was to install a native lib and tell to Spark to use it.

I wrote a small post : https://thomasdecaux.medium.com/use-snappy-compression-native-when-running-spark-on-alpine-5fee0ce28ed7

  1. apk add java-snappy-native
  2. spark.executor.extraJavaOptions: -Dorg.xerial.snappy.use.systemlib=true -Dorg.xerial.snappy.lib.path=/usr/lib/libsnappyjava.so
commented

Both gcompat and java-snappy-native can help to fix this error but with different configuration.

  1. With gcompat: Add gcompat package to your runtime environment (for e.g with Alpine linux apk add --no-cache gcompat)

  2. With java-snappy-native

  • Install package : apk updateapk --no-cache update && apk add --no-cache curl java-snappy-native
  • Add extra JVM arguments when starting your application: java -jar -Dorg.xerial.snappy.use.systemlib=true -Dorg.xerial.snappy.lib.path=/usr/lib/libsnappyjava.so application.jar

Note: Don't try to use extra JVM arguments with gcompat, if these arguments exist, then snappy will try to find java-snappy-native instead of gcompat

commented

We run a similar issue too, using Alpine as base Docker image. The solution was to install a native lib and tell to Spark to use it.

I wrote a small post : https://thomasdecaux.medium.com/use-snappy-compression-native-when-running-spark-on-alpine-5fee0ce28ed7

  1. apk add java-snappy-native
  2. spark.executor.extraJavaOptions: -Dorg.xerial.snappy.use.systemlib=true -Dorg.xerial.snappy.lib.path=/usr/lib/libsnappyjava.so

I found that after the java-snappy-native installation finished, my jdk became jdk8 instead of jdk17