Tensorflow Version Compatibility
sgboakes opened this issue · comments
I am attempting to run tensorflow-agents with dm-reverb on a HPC with GPUs. I have installed tensorflow-gpu with conda, and tf-agents and dm-reverb with pip. When I attempt to import reverb, I get the following error:
>>> import reverb
2022-09-12 15:30:21.945651: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/pybind.py", line 4, in <module>
from .libpybind import *
ImportError: /lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/libpybind.so: undefined symbol: _ZN4absl12lts_2022062320raw_logging_internal21internal_log_functionB5cxx11E
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/__init__.py", line 27, in <module>
from reverb import item_selectors as selectors
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/item_selectors.py", line 19, in <module>
from reverb import pybind
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/pybind.py", line 6, in <module>
_load_op_library.reraise_wrapped_error(e)
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/platform/default/load_op_library.py", line 43, in reraise_wrapped_error
six.reraise(ImportError,
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/six.py", line 702, in reraise
raise value.with_traceback(tb)
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/pybind.py", line 4, in <module>
from .libpybind import *
ImportError:
Attempted to load a reverb dynamic library, but it could not find the required
symbols inside of TensorFlow. This commonly occurs when your version of
tensorflow and reverb are mismatched. For example, if you are using the python
package 'tf-nightly', make sure you use the python package 'dm-reverb-nightly'
built on the same or the next night. If you are using a release version package
'tensorflow', use a release package 'dm-reverb' built to be compatible with
that exact version. If all else fails, file a github issue on deepmind/reverb.
Current installed version of tensorflow: 2.4.1.
I have tried installing different versions of reverb, which provides the following error:
>>> import reverb
2022-09-12 15:34:30.457808: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/__init__.py", line 27, in <module>
from reverb import item_selectors as selectors
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/item_selectors.py", line 19, in <module>
from reverb import pybind
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/pybind.py", line 6, in <module>
_load_op_library.reraise_wrapped_error(e)
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/platform/default/load_op_library.py", line 47, in reraise_wrapped_error
raise error
File "/lustre/hpc/local/user/anaconda3/envs/agents/lib/python3.9/site-packages/reverb/pybind.py", line 4, in <module>
from .libpybind import *
ImportError: libpython3.9.so.1.0: cannot open shared object file: No such file or directory
Does anyone know which version of reverb is compatible with tensorflow 2.4.1? I am unable to change my tensorflow version.
Thank you
I think reverb 0.2.0 was built on top of tf 2.4.0, so it should be compatible. Try it: https://pypi.org/project/dm-reverb/0.2.0/
I did a quick check. For TF 2.4.x you want.
tf-agents 0.7.1. and Reverb 0.2.0. (EXACTLY not greater)
A bit later I started to PIN tf-agents to a specific Reverb / Tensorflow. The "easiest" way to figure this out is to find the tf-agents that matches the TF you are using and install it with tf-agents[reverb] == 0.7.1. Then look at what reveb it wants. WARNING: Until later on even installing it this way it would be OK with Reverb 0.9.0 or whatever because I had it set to >= until we realized that was not going to work.
I hope this helps you now and in the future.
Thank you for your replies. I am having trouble installing reverb 0.2.0.
I install tensorflow-gpu with conda, then tf-agents==0.7.1 with pip, but than cannot install dm-reverb==0.2.0
pip returns:
(rl) -bash-4.2$ pip install dm-reverb==0.2.0
ERROR: Could not find a version that satisfies the requirement dm-reverb==0.2.0 (from versions: 0.3.1rc0, 0.3.1, 0.4.0rc0, 0.4.0, 0.5.0rc0, 0.5.0rc1, 0.5.0, 0.6.0rc0, 0.6.0, 0.6.1rc0, 0.6.1, 0.7.0rc0, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0rc0, 0.8.0, 0.9.0rc0, 0.9.0rc1, 0.9.0)
ERROR: No matching distribution found for dm-reverb==0.2.0
(rl) -bash-4.2$
If I try to do pip install tf-agents[reverb]==0.7.1, pip installs dm-reverb==0.9.0.
Is there any way to install 0.2.0 with pip?
That is the correct syntax. Keep in mind reverb 0.2.0 is released for python 3.6, 3.7 and 3.8. Which aligns with what TF was releasing at the time. I also get the same error you get when installing on Python 3.10. Here is me installing it on 3.7.14.
What is weird is TF 2.4.1 did not officially support 3.9 so we did not and would not have been able to build reverb for it. My guess is who ever owns the Conda build created an unoffiical 3.9 build, which is fine.
tf-docker /tmp/reverb > python3 --version
Python 3.7.14
tf-docker /tmp/reverb > python3 --version -mpip install dm-reverb==0.2.0
Python 3.7.14
tf-docker /tmp/reverb > python3 -mpip install dm-reverb==0.2.0
Collecting dm-reverb==0.2.0
Downloading dm_reverb-0.2.0-cp37-cp37m-manylinux2010_x86_64.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 48.1 MB/s eta 0:00:00
Requirement already satisfied: portpicker in /usr/local/lib/python3.7/dist-packages (from dm-reverb==0.2.0) (1.4.0)
Requirement already satisfied: dm-tree in /usr/local/lib/python3.7/dist-packages (from dm-reverb==0.2.0) (0.1.7)
Installing collected packages: dm-reverb
Successfully installed dm-reverb-0.2.0
I want to stress that I do not know your constraints. I see a couple options. I realize these might be impossible; but I wanted to just share them in collaboration:
- You have python 3.9 so you can install any version of tensorflow you want. You might be constrained by your version of CUDA? FYI. TF 2.5.0 added Python 3.9 as did Reverb 0.3.0.
- Use Python 3.8.
These are the only options I can think of. I wish I had a more straight forward answer for you. Let me know how it turns out.
Running on Python 3.8 has allowed me to install reverb 0.2.0, thank you for the advice. I have some errors relating to opening libraries but that Is unrelated to this issue.
Thank you again for all the help