py4j / py4j

Py4J enables Python programs to dynamically access arbitrary Java objects

Home Page:https://www.py4j.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

java.lang.NegativeArraySizeException: -63465312 at py4j.Base64.decode(Base64.java:321)

siddhsql opened this issue · comments

I am trying to marshal a large numpy array from Python to Java using this function:

# If from Python, you pass a bytearray or bytes variable to the Java side, Py4J will convert it to a byte[].
# https://stackoverflow.com/a/36479534/147530
def marshal_float(numpy_ndarray):
    if sys.byteorder != "little":
        raise RuntimeError("not implemented")
    body = array.array('f', numpy_ndarray.tolist())
    return bytearray(body)

My array has 100M floats in it (100M * 4 is still within the range of an int). Once I got following when I tried to do this:

py4j.protocol.Py4JError: An error occurred while calling o4.addMany. Trace:
java.lang.NegativeArraySizeException: -63465312
	at py4j.Base64.decode(Base64.java:321)
	at py4j.Protocol.getBytes(Protocol.java:178)
	at py4j.Protocol.getObject(Protocol.java:299)
	at py4j.commands.AbstractCommand.getArguments(AbstractCommand.java:82)
	at py4j.commands.CallCommand.execute(CallCommand.java:77)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:1623)

at other time I got this in Java:

[WARNING]
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange (Arrays.java:3822)
    at java.lang.String.<init> (String.java:4531)
    at java.lang.String.<init> (String.java:1487)
    at java.lang.StringBuilder.toString (StringBuilder.java:453)
    at py4j.commands.AbstractCommand.getStringArguments (AbstractCommand.java:105)
    at py4j.commands.AbstractCommand.getArguments (AbstractCommand.java:79)
    at py4j.commands.CallCommand.execute (CallCommand.java:77)
    at py4j.GatewayConnection.run (GatewayConnection.java:238)
    at java.lang.Thread.runWith (Thread.java:1636)
    at java.lang.Thread.run (Thread.java:1623)

can anyone explain to me what is going on here?