`OSError: libmkl_intel_lp64.so.1: cannot open shared object file` when using `to_hf_weights.py`
danyaljj opened this issue · comments
$ python3 to_hf_weights.py --input-ckpt gs://danielk-files/gpt-j-checkpoints_slim/step_318500 --config configs/6B_roto_256.json --output-path gs://danielk-files/danielk-files/gpt-j-checkpoints_slim_hf/step_318500 --dtype fp32 --cpu
Traceback (most recent call last):
File "to_hf_weights.py", line 21, in <module>
import torch
File "/usr/local/lib/python3.8/dist-packages/torch/__init__.py", line 198, in <module>
_load_global_deps()
File "/usr/local/lib/python3.8/dist-packages/torch/__init__.py", line 151, in _load_global_deps
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory
Here is the environment:
Package Version
---------------------------- --------------------
absl-py 1.0.0
aiohttp 3.8.1
aiohttp-cors 0.7.0
aioredis 2.0.1
aiosignal 1.2.0
anyio 3.5.0
asgiref 3.5.0
astunparse 1.6.3
async-timeout 4.0.2
attrs 19.3.0
Automat 0.8.0
bcrypt 3.2.0
best-download 0.0.9
black 22.1.0
blessings 1.7
blinker 1.4
cachetools 4.2.4
certifi 2021.10.8
cffi 1.15.0
chardet 3.0.4
charset-normalizer 2.0.11
chex 0.1.1
clang 5.0
click 8.0.4
cloud-init 21.4
cloud-tpu-client 0.10
cloudpickle 1.3.0
colorama 0.4.3
colorful 0.5.4
command-not-found 0.3
configobj 5.0.6
constantly 15.1.0
cryptography 2.8
Cython 0.29.14
DataProperty 0.55.0
datasets 1.15.1
dbus-python 1.2.16
Deprecated 1.2.13
dill 0.3.4
distlib 0.3.4
distro 1.4.0
distro-info 0.23ubuntu1
dm-haiku 0.0.5
dm-tree 0.1.6
docker-pycreds 0.4.0
dyNET38 2.1
einops 0.3.2
entrypoints 0.3
fabric 2.6.0
fastapi 0.75.0
filelock 3.4.2
Flask 1.1.4
flatbuffers 1.12
frozenlist 1.3.0
fsspec 2022.2.0
ftfy 6.1.1
func-timeout 4.3.5
future 0.18.2
gast 0.4.0
gitdb 4.0.9
GitPython 3.1.27
google-api-core 1.31.5
google-api-python-client 1.8.0
google-auth 1.35.0
google-auth-httplib2 0.1.0
google-auth-oauthlib 0.4.6
google-cloud-core 1.7.2
google-cloud-storage 1.36.2
google-crc32c 1.3.0
google-pasta 0.2.0
google-resumable-media 1.3.3
googleapis-common-protos 1.54.0
gpustat 0.6.0
grpcio 1.43.0
h11 0.13.0
h5py 3.1.0
httplib2 0.20.2
huggingface-hub 0.4.0
hyperlink 19.0.0
idna 2.10
importlib-metadata 4.10.1
incremental 16.10.1
iniconfig 1.1.1
intel-openmp 2022.0.2
invoke 1.7.0
itsdangerous 1.1.0
jax 0.2.12
jaxlib 0.1.68
jieba 0.42.1
Jinja2 2.10.1
jmp 0.0.2
joblib 1.1.0
jsonlines 2.0.0
jsonpatch 1.22
jsonpointer 2.0
jsonpointer 2.0
jsonschema 3.2.0
keras 2.6.0
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.2
keyring 18.0.1
language-selector 0.1
launchpadlib 1.10.13
lazr.restfulclient 0.14.2
lazr.uri 1.0.3
libclang 13.0.0
lm-dataformat 0.0.20
lm-eval 0.2.0
Markdown 3.3.6
MarkupSafe 1.1.0
mbstrdecoder 1.1.0
mkl 2022.0.2
mkl-include 2022.0.2
mock 4.0.3
more-itertools 4.2.0
msgfy 0.2.0
msgpack 1.0.3
multidict 6.0.2
multiprocess 0.70.12.2
mypy-extensions 0.4.3
nagisa 0.2.7
netifaces 0.10.4
nltk 3.7
numexpr 2.7.2
numpy 1.19.5
nvidia-ml-py3 7.352.0
oauth2client 4.1.3
oauthlib 3.1.0
openai 0.6.4
opencensus 0.8.0
opencensus-context 0.1.2
opt-einsum 3.3.0
optax 0.0.9
packaging 21.3
pandas 1.4.1
paramiko 2.10.3
pathlib2 2.3.7.post1
pathspec 0.9.0
pathtools 0.1.2
pathvalidate 2.5.0
pathy 0.6.1
pexpect 4.6.0
Pillow 9.0.0
pip 20.0.2
platformdirs 2.4.1
pluggy 0.13.1
portalocker 2.4.0
prometheus-client 0.13.1
promise 2.3
protobuf 3.19.4
psutil 5.9.0
py 1.11.0
py-spy 0.3.11
pyarrow 7.0.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pybind11 2.6.2
pycountry 20.7.3
pycparser 2.21
pydantic 1.9.0
PyGObject 3.36.0
PyHamcrest 1.9.0
PyJWT 1.7.1
pymacaroons 0.13.0
PyNaCl 1.3.0
pyOpenSSL 19.0.0
pyparsing 3.0.7
pyrsistent 0.15.5
pyserial 3.4
pytablewriter 0.58.0
pytest 6.2.3
python-apt 2.0.0+ubuntu0.20.4.6
python-dateutil 2.8.2
python-debian 0.1.36ubuntu1
pytz 2021.3
PyYAML 5.4.1
ray 1.4.1
redis 4.1.4
regex 2022.3.15
rehash 1.0.0
requests 2.25.1
requests-oauthlib 1.3.1
requests-unixsocket 0.2.0
rouge-score 0.0.4
rsa 4.8
sacrebleu 1.5.0
sacremoses 0.0.49
scikit-learn 1.0.2
scipy 1.8.0
SecretStorage 2.3.1
sentry-sdk 1.5.8
service-identity 18.1.0
setproctitle 1.2.2
setuptools 60.7.0
shortuuid 1.0.8
simplejson 3.16.0
six 1.16.0
smart-open 5.2.1
smmap 5.0.0
sniffio 1.2.0
sos 4.1
sqlitedict 1.6.0
ssh-import-id 5.10
starlette 0.17.1
systemd-python 234
tabledata 1.3.0
tabulate 0.8.9
tbb 2021.5.1
tcolorpy 0.1.2
tensorboard 2.6.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorflow 2.8.0
tensorflow-cpu 2.6.3
tensorflow-estimator 2.6.0
tensorflow-hub 0.12.0
tensorflow-io-gcs-filesystem 0.23.1
tensorflow-text 2.8.0rc0
termcolor 1.1.0
threadpoolctl 3.1.0
tokenizers 0.11.6
toml 0.10.2
tomli 2.0.1
toolz 0.11.2
torch 1.11.0a0+git06838ce
torch-xla 1.11+3413bd1
torchvision 0.12.0a0+563d9ca
tqdm 4.63.0
tqdm-multiprocess 0.0.11
transformers 4.17.0
Twisted 18.9.0
typepy 1.3.0
typer 0.4.0
typing-extensions 4.0.1
ubuntu-advantage-tools 27.5
ufw 0.36
ujson 5.1.0
unattended-upgrades 0.1
uritemplate 3.0.1
urllib3 1.26.8
uvicorn 0.17.6
virtualenv 20.13.0
wadllib 1.3.3
wandb 0.12.11
wcwidth 0.2.5
Werkzeug 1.0.1
wheel 0.37.1
wrapt 1.12.1
xxhash 3.0.0
yarl 1.7.2
yaspin 2.1.0
zipp 1.0.0
zope.interface 4.7.1
zstandard 0.15.2
and
$ python3 --version
Python 3.8.10
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"