Running code on Java > 8

[Simas Joneliunas]

With multiple Java JREs installed on the same machine, the default JRE is usually 11 or 17 (depending on the machine). Sometimes something in between.

However trying to run spark with one of the later JDKs (in my case, 17) the run fails with IllegalAccessError as shown below. is there any way to pass java path to the runner or otherwise have it coexist without setting java 8 as main java of the machine?

`PS C:\Projects\redacted> spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-3-2_2.12-2.1.0.jar dotnet SparkApp.dll input.txt
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/09/20 15:48:04 INFO DotnetRunner: Starting DotnetBackend with dotnet.
22/09/20 15:48:04 INFO DotnetBackend: The number of DotnetBackend threads is set to 10.
22/09/20 15:48:05 INFO DotnetRunner: Port number used by DotnetBackend is 5662
22/09/20 15:48:05 INFO DotnetRunner: Adding key=spark.jars and value=file:/.nuget/packages/microsoft.spark/2.1.0/jars/microsoft-spark-3-2_2.12-2.1.0.jar to environment
22/09/20 15:48:05 INFO DotnetRunner: Adding key=spark.app.name and value=org.apache.spark.deploy.dotnet.DotnetRunner to environment
22/09/20 15:48:05 INFO DotnetRunner: Adding key=spark.submit.pyFiles and value= to environment
22/09/20 15:48:05 INFO DotnetRunner: Adding key=spark.submit.deployMode and value=client to environment
22/09/20 15:48:05 INFO DotnetRunner: Adding key=spark.master and value=local to environment
[2022-09-20T07:48:05.7879582Z] [] [Info] [ConfigurationService] Using port 5662 for connection.
[2022-09-20T07:48:05.7918928Z] [] [Info] [JvmBridge] JvMBridge port is 5662
[2022-09-20T07:48:05.7927120Z] [] [Info] [JvmBridge] The number of JVM backend thread is set to 10. The max number of concurrent sockets in JvmBridge is set to 7.
22/09/20 15:48:06 INFO SparkContext: Running Spark version 3.2.1
22/09/20 15:48:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/09/20 15:48:06 INFO ResourceUtils: ==============================================================
22/09/20 15:48:06 INFO ResourceUtils: No custom resources configured for spark.driver.
22/09/20 15:48:06 INFO ResourceUtils: ==============================================================
22/09/20 15:48:06 INFO SparkContext: Submitted application: word_count_sample
22/09/20 15:48:06 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/09/20 15:48:06 INFO ResourceProfile: Limiting resource is cpu
22/09/20 15:48:06 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/09/20 15:48:06 INFO SecurityManager: Changing view acls to: --
22/09/20 15:48:06 INFO SecurityManager: Changing modify acls to: --
22/09/20 15:48:06 INFO SecurityManager: Changing view acls groups to:
22/09/20 15:48:06 INFO SecurityManager: Changing modify acls groups to:
22/09/20 15:48:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(--); groups with view permissions: Set(); users with modify permissions: Set(--); groups with modify permissions: Set()
22/09/20 15:48:06 INFO Utils: Successfully started service 'sparkDriver' on port 5664.
22/09/20 15:48:06 INFO SparkEnv: Registering MapOutputTracker
22/09/20 15:48:06 INFO SparkEnv: Registering BlockManagerMaster
22/09/20 15:48:06 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/09/20 15:48:06 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/09/20 15:48:06 ERROR DotnetBackendHandler: Failed to execute 'getOrCreate' on 'org.apache.spark.sql.SparkSession$Builder' with args=()
[2022-09-20T07:48:06.9638182Z] [] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'getOrCreate' failed for class '1' when called with no arguments
[2022-09-20T07:48:06.9638918Z] [] [Error] [JvmBridge] java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x55f616cf) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x55f616cf
at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:213)
at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala)
at org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:110)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
at org.apache.spark.SparkContext.(SparkContext.scala:460)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:165)
at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:105)
at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)

[2022-09-20T07:48:07.0357592Z] [] [Exception] [JvmBridge] JVM method execution failed: Nonstatic method 'getOrCreate' failed for class '1' when called with no arguments
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
Unhandled exception. System.Exception: JVM method execution failed: Nonstatic method 'getOrCreate' failed for class '1' when called with no arguments
---> Microsoft.Spark.JvmException: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x55f616cf) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x55f616cf
at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:213)
at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala)
at org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:110)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
at org.apache.spark.SparkContext.(SparkContext.scala:460)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:165)
at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:105)
at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)

--- End of inner exception stack trace ---
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference jvmObject, String methodName, Object[] args)
at Microsoft.Spark.Sql.Builder.GetOrCreate()
at TASparkApp.Actions.DemoAction2..ctor() in C:\Projects..\DemoAction2.cs:line 32
at TASparkApp.TASparkAppProgram.Main(String[] args) in C:\Projects..\SparkApp.cs:line 23
22/09/20 15:48:10 ERROR DotnetBackendHandler: Exception caught:
java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
22/09/20 15:48:10 INFO DotnetRunner: Closing DotnetBackend`