Unable to perform distributed training using tiny ImageNet example

[MoslemTCM]

Issue Description

I am trying to to reporduce your results for tinyimagenet example provided in distributed-training-examples
on my dockerized standlone spark cluster (numNodes = 1). However I am getting the following erreur:

21/07/12 18:40:45 INFO v2.ModelParameterServer: ModelParameterServer starting
21/07/12 18:40:46 INFO impl.AeronUdpTransport: Adding UDP connection: [aeron:udp?endpoint=172.23.0.3:49876]
21/07/12 18:40:46 ERROR executor.Executor: Exception in task 2.0 in stage 6.0 (TID 22)
io.aeron.exceptions.RegistrationException: Negative size
at io.aeron.ClientConductor.onError(ClientConductor.java:299)
at io.aeron.DriverEventsAdapter.onMessage(DriverEventsAdapter.java:75)
at org.agrona.concurrent.broadcast.CopyBroadcastReceiver.receive(CopyBroadcastReceiver.java:100)
at io.aeron.DriverEventsAdapter.receive(DriverEventsAdapter.java:55)
at io.aeron.ClientConductor.service(ClientConductor.java:436)
at io.aeron.ClientConductor.awaitResponse(ClientConductor.java:472)
at io.aeron.ClientConductor.addPublication(ClientConductor.java:179)
at io.aeron.Aeron.addPublication(Aeron.java:241)
at org.nd4j.parameterserver.distributed.v2.transport.impl.AeronUdpTransport.addConnection(AeronUdpTransport.java:315)
at org.nd4j.parameterserver.distributed.v2.transport.impl.AeronUdpTransport.launch(AeronUdpTransport.java:352)
at org.nd4j.parameterserver.distributed.v2.ModelParameterServer.launch(ModelParameterServer.java:377)
at org.deeplearning4j.spark.parameterserver.pw.SharedTrainingWrapper.run(SharedTrainingWrapper.java:363)
at org.deeplearning4j.spark.parameterserver.functions.SharedFlatMapPaths.call(SharedFlatMapPaths.java:86)
at org.deeplearning4j.spark.parameterserver.functions.SharedFlatMapPaths.call(SharedFlatMapPaths.java:41)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$4$1.apply(JavaRDDLike.scala:153)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$4$1.apply(JavaRDDLike.scala:153)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
21/07/12 18:40:46 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 24
21/07/12 18:40:46 INFO executor.Executor: Running task 4.0 in stage 6.0 (TID 24)
21/07/12 18:40:46 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks
21/07/12 18:40:46 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms

spark submit configuration:

EXECUTOR_MEMORY=15G
OFFHEAP_MEM_JAVACPP=15G
OFFHEAP_JAVACPP_MAX_PHYS=20G
AERON_BUFFER=1073741824

Version Information

Please indicate relevant versions, including, if relevant:

• Deeplearning4j version: 1.0.0-beta7
• platform information (OS, etc): ubuntu 20
• CUDA version, if used: Onlu CPU backend is used

[Adam Gibson]

Please use the community forums for further communication. Apologies, but this is right at the top of the readme.