py4j.protocol.Py4JJavaError: An error occurred while calling o82.print - java.lang.reflect.InaccessibleObjectException: Unable to make field private final byte[] java.lang.String.value accessible: module java.base does not "opens java.lang" to unnamed module
jeanmonet opened this issue · comments
Trying to run the basic script mentioned here, I encounter the following error:
Traceback (most recent call last):
File "/Users/ebs/test_flink.py", line 28, in <module>
tutorial()
File "/Users/ebs/test_flink.py", line 20, in tutorial
ds.print()
File "/opt/homebrew/Caskroom/miniconda/base/envs/flink/lib/python3.9/site-packages/pyflink/datastream/data_stream.py", line 883, in print
j_data_stream_sink = self._align_output_type()._j_data_stream.print()
File "/opt/homebrew/Caskroom/miniconda/base/envs/flink/lib/python3.9/site-packages/py4j/java_gateway.py", line 1321, in __call__
return_value = get_return_value(
File "/opt/homebrew/Caskroom/miniconda/base/envs/flink/lib/python3.9/site-packages/pyflink/util/exceptions.py", line 146, in deco
return f(*a, **kw)
File "/opt/homebrew/Caskroom/miniconda/base/envs/flink/lib/python3.9/site-packages/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o82.print.
: java.lang.reflect.InaccessibleObjectException: Unable to make field private final byte[] java.lang.String.value accessible: module java.base does not "opens java.lang" to unnamed module @a803f94
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:106)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2308)
at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:202)
at org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1242)
at org.apache.flink.streaming.api.datastream.DataStream.print(DataStream.java:936)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:833)Environment:
# Installation with mamba (conda):
mamba create -n flink "python<3.10" openjdk cython
conda activate flink
pip install apache-flink --no-cache
# --- List of installed packages ---
# packages in environment at /opt/homebrew/Caskroom/miniconda/base/envs/flink:
#
# Name Version Build Channel
apache-beam 2.38.0 pypi_0 pypi
apache-flink 1.16.1 pypi_0 pypi
apache-flink-libraries 1.16.1 pypi_0 pypi
avro-python3 1.9.2.1 pypi_0 pypi
bzip2 1.0.8 h3422bc3_4 conda-forge
ca-certificates 2022.12.7 h4653dfc_0 conda-forge
certifi 2022.12.7 pypi_0 pypi
charset-normalizer 3.1.0 pypi_0 pypi
cloudpickle 2.1.0 pypi_0 pypi
crcmod 1.7 pypi_0 pypi
cython 0.29.33 py39h23fbdae_0 conda-forge
dill 0.3.1.1 pypi_0 pypi
docopt 0.6.2 pypi_0 pypi
fastavro 1.4.7 pypi_0 pypi
find-libpython 0.3.0 pypi_0 pypi
grpcio 1.51.3 pypi_0 pypi
hdfs 2.7.0 pypi_0 pypi
httplib2 0.19.1 pypi_0 pypi
idna 3.4 pypi_0 pypi
libcxx 15.0.7 h75e25f2_0 conda-forge
libffi 3.4.2 h3422bc3_5 conda-forge
libsqlite 3.40.0 h76d750c_0 conda-forge
libzlib 1.2.13 h03a7124_4 conda-forge
ncurses 6.3 h07bb92c_1 conda-forge
numpy 1.21.4 pypi_0 pypi
oauth2client 4.1.3 pypi_0 pypi
openjdk 17.0.3 hf913c23_6 conda-forge # ****
openssl 3.1.0 h03a7124_0 conda-forge
orjson 3.8.7 pypi_0 pypi
pandas 1.3.5 pypi_0 pypi
pemja 0.2.6 pypi_0 pypi
pip 23.0.1 pyhd8ed1ab_0 conda-forge
proto-plus 1.22.2 pypi_0 pypi
protobuf 3.20.3 pypi_0 pypi
py4j 0.10.9.3 pypi_0 pypi # ****
pyarrow 6.0.1 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pydot 1.4.2 pypi_0 pypi
pymongo 3.13.0 pypi_0 pypi
pyparsing 2.4.7 pypi_0 pypi
python 3.9.16 hea58f1e_0_cpython conda-forge
python-dateutil 2.8.0 pypi_0 pypi
python_abi 3.9 1_cp39 huggingface
pytz 2022.7.1 pypi_0 pypi
readline 8.1.2 h46ed386_0 conda-forge
requests 2.28.2 pypi_0 pypi
rsa 4.9 pypi_0 pypi
setuptools 67.6.0 pyhd8ed1ab_0 conda-forge
six 1.16.0 pypi_0 pypi
tk 8.6.12 he1e0b03_0 conda-forge
typing-extensions 4.5.0 pypi_0 pypi
tzdata 2022g h191b570_0 conda-forge
urllib3 1.26.15 pypi_0 pypi
wheel 0.40.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h57fd34a_0 conda-forgeIt appears to pop up in downstream of py4j - any idea why & how to fix it?
Many thanks
@jeanmonet which code did you run? Would be great to have a reproducer
Hi, more specifically, this is the test code (in test_flink.py):
from pyflink.common.serialization import Encoder
from pyflink.common.typeinfo import Types
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.connectors import StreamingFileSink
def tutorial():
env = StreamExecutionEnvironment.get_execution_environment()
env.set_parallelism(1)
ds = env.from_collection(
collection=[(1, 'aaa'), (2, 'bbb')],
type_info=Types.ROW([Types.INT(), Types.STRING()])
).map(lambda l: l, output_type=Types.ROW([Types.INT(), Types.STRING()]))
ds.print()
ds.add_sink(StreamingFileSink
.for_row_format('./output', Encoder.simple_string_encoder())
.build())
env.execute("tutorial_job")
if __name__ == '__main__':
tutorial()(Emphasis on environment with openjdk v17.)
Mind pointing out where you exactly invoke setAccessible? If you cannot do the reflection with JVM, there's nothing Py4J can do about it.
py4j.protocol.Py4JJavaError: An error occurred while calling o82.print.
: java.lang.reflect.InaccessibleObjectException: Unable to make field private final byte[] java.lang.String.value accessible: module java.base does not "opens java.lang" to unnamed module @a803f94
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:106)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2308)
at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:202)
at org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1242)
at org.apache.flink.streaming.api.datastream.DataStream.print(DataStream.java:936)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
I would not know to answer, since my interface is the PyFlink (apache-flink) package that uses py4j.
I also note similar error messages causing trouble due to there being (apparently) a change in Java's API, and the error arises starting with openjdk 17.
The temporary "fix" other suggested was to run the java app with --add-opens=java.base/java.lang=ALL-UNNAMED, but I didn't have success using that for the pyflink toy example above.
However, downgrading to openjdk 11 allows me to run the code, with the following warnings (warning that became an error in subsequent versions apparently):
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/opt/homebrew/Caskroom/miniconda/base/envs/flink/lib/python3.9/site-packages/pyflink/lib/flink-dist-1.16.1.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
In conclusion, not certain if this is a py4j issue or a apache flink issue?
If this is not doable with plain JDK, you cannot do Py4J issue. Assuming from what we have here, its very unlikely that the issue is specific to Py4J.