java.lang.NegativeArraySizeException: -63465312 at py4j.Base64.decode(Base64.java:321)
siddhsql opened this issue · comments
I am trying to marshal a large numpy array from Python to Java using this function:
# If from Python, you pass a bytearray or bytes variable to the Java side, Py4J will convert it to a byte[].
# https://stackoverflow.com/a/36479534/147530
def marshal_float(numpy_ndarray):
if sys.byteorder != "little":
raise RuntimeError("not implemented")
body = array.array('f', numpy_ndarray.tolist())
return bytearray(body)
My array has 100M floats in it (100M * 4 is still within the range of an int
). Once I got following when I tried to do this:
py4j.protocol.Py4JError: An error occurred while calling o4.addMany. Trace:
java.lang.NegativeArraySizeException: -63465312
at py4j.Base64.decode(Base64.java:321)
at py4j.Protocol.getBytes(Protocol.java:178)
at py4j.Protocol.getObject(Protocol.java:299)
at py4j.commands.AbstractCommand.getArguments(AbstractCommand.java:82)
at py4j.commands.CallCommand.execute(CallCommand.java:77)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:1623)
at other time I got this in Java:
[WARNING]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange (Arrays.java:3822)
at java.lang.String.<init> (String.java:4531)
at java.lang.String.<init> (String.java:1487)
at java.lang.StringBuilder.toString (StringBuilder.java:453)
at py4j.commands.AbstractCommand.getStringArguments (AbstractCommand.java:105)
at py4j.commands.AbstractCommand.getArguments (AbstractCommand.java:79)
at py4j.commands.CallCommand.execute (CallCommand.java:77)
at py4j.GatewayConnection.run (GatewayConnection.java:238)
at java.lang.Thread.runWith (Thread.java:1636)
at java.lang.Thread.run (Thread.java:1623)
can anyone explain to me what is going on here?