'version `GFORTRAN_1.4'' not found when running tests
cjmcmurtrie opened this issue · comments
Hi Hugh, I'm now attempting a build on an Ubuntu 14.04 machine with CUDA 7.5. I have a fresh Torch install and Python 2.7.11 Anaconda distribution.
After building PyTorch, running the tests fails with the followings errors:
platform linux2 -- Python 2.7.11, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/dg/anaconda2/bin/python
cachedir: .cache
rootdir: /home/dg/pytorch, inifile:
collected 0 items / 8 errors
======================================================= ERRORS =======================================================
______________________________________ ERROR collecting test/testByteTensor.py _______________________________________
test/testByteTensor.py:3: in <module>
import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
???
E ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
_____________________________________ ERROR collecting test/testDoubleTensor.py ______________________________________
test/testDoubleTensor.py:3: in <module>
import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
???
E ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
______________________________________ ERROR collecting test/testFloatTensor.py ______________________________________
test/testFloatTensor.py:3: in <module>
import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
???
E ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
______________________________________ ERROR collecting test/testLongTensor.py _______________________________________
test/testLongTensor.py:3: in <module>
import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
???
E ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
_________________________________________ ERROR collecting test/test_nnx.py __________________________________________
test/test_nnx.py:2: in <module>
import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
???
E ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
_________________________________________ ERROR collecting test/test_pynn.py _________________________________________
test/test_pynn.py:2: in <module>
import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
???
E ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
_______________________________________ ERROR collecting test/test_pytorch.py ________________________________________
test/test_pytorch.py:5: in <module>
import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
???
E ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
________________________________________ ERROR collecting test/test_throw.py _________________________________________
test/test_throw.py:1: in <module>
import PyTorchHelpers
build/bdist.linux-x86_64/egg/PyTorchHelpers.py:1: in <module>
???
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
???
E ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
============================================== 8 error in 0.26 seconds ===============================================
Any idea what could be causing this? Since I was using the same distribution (Anaconda) with a successful build on OSX, I'm not sure what the problem could be here.
You need fortran installed. On other platforms, install-deps installs it, eg https://github.com/torch/distro/blob/master/install-deps#L181 , but for Mac it seems not to install it. Maybe it's normally present on Mac by default, and maybe you have it, but for some reason it's not being picked up, when running torch from python?
Can you first check that torch itself works ok?
luajit -l torch -e 'torch.test()'
luajit -l cutorch -e 'cutorch.test()'
luajit -l nn -e 'nn.test()'
luajit -l cunn -e 'nn.testcuda()'
If that works, let's try to find the fortran library that torch uses. Can you try something like:
otool -l ~/torch/install/lib/libTH.so
(or it might be:
otool -l ~/torch/install/lib/libTH.dylib
)
If otool -l
doenst work, can you try otool -L
instead?
Hi Hugh, all the above Torch tests run correctly. However not the Fortran tests, this is the outcome:
No command 'otool' found, did you mean:
Command 'atool' from package 'atool' (universe)
Command 'potool' from package 'potool' (universe)
Command 'btool' from package 'barry-util' (universe)
By the way, this is an Ubuntu 14.04 machine I'm on now, not a Mac
By the way, this is an Ubuntu 14.04 machine I'm on now, not a Mac
:-P
Ok... so, on linux, the command will be something like:
ldd ~/torch/install/lib/libTH.so
On my box this gives:
...
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f2d071ea000)
...
And also:
dpkg -l '*fortran*' | grep ii
This should give something like:
$ dpkg -l '*fortran*' | grep ii
ii gfortran 4:5.2.1-3ubuntu1 amd64 GNU Fortran 95 compiler
ii gfortran-5 5.2.1-22ubuntu2 amd64 GNU Fortran compiler
ii libgfortran-5-dev:amd64 5.2.1-22ubuntu2 amd64 Runtime library for GNU Fortran applications (development files)
ii libgfortran3:amd64 5.2.1-22ubuntu2 amd64 Runtime library for GNU Fortran applications
This is the result of ldd ~/torch/install/lib/libTH.so
. It looks like libgfortran
is in usr/lib but the the tests script is looking for it in Anaconda directories? :
$ ldd ~/torch/install/lib/libTH.so
linux-vdso.so.1 => (0x00007ffc6f34f000)
libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f9bfde9d000)
liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007f9bfd700000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9bfd3fa000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f9bfd1eb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9bfcfcd000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9bfcc07000)
/lib64/ld-linux-x86-64.so.2 (0x00005605544ad000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f9bfb3fe000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f9bfb0e4000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9bfaecd000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f9bfac91000)
This is the result of dpkg -l '*fortran*' | grep ii
:
$ dpkg -l '*fortran*' | grep ii
ii gfortran 4:4.8.2-1ubuntu6 amd64 GNU Fortran 95 compiler
ii gfortran-4.8 4.8.4-2ubuntu1~14.04.1 amd64 GNU Fortran compiler
ii libgfortran-4.8-dev:amd64 4.8.4-2ubuntu1~14.04.1 amd64 Runtime library for GNU Fortran applications (development files)
ii libgfortran3:amd64 4.8.4-2ubuntu1~14.04.1 amd64 Runtime library for GNU Fortran applications
Hmmmm. What if you preload the libgfortran, something like:
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 python myscript.py
managed to reproduce the problem, using conda, and LD_PRELOAD
seems to fix it:
$ which python
/mnt/anaconda/bin/python
ubuntu@ip-172-30-4-214:~/git/pytorch/simpleexample$ python pybit.py
Traceback (most recent call last):
File "pybit.py", line 3, in <module>
import PyTorch
ImportError: /mnt/anaconda/bin/../lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
ubuntu@ip-172-30-4-214:~/git/pytorch/simpleexample$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 python pybit.py
Luabit:__init( green )
green
type(luabit) <class 'PyTorchLua.Luabit'>
out from lua (1,.,.) =
0.8097 -0.6946 0.4558
-0.2452 0.4862 -0.1560
(2,.,.) =
-0.4325 -0.9738 -0.4096
0.5181 -0.0675 1.2093
[torch.FloatTensor of size 2x2x3]
outTensor [[[ 0.80968177 -0.69459283 0.45584014]
[-0.24516164 0.48619133 -0.15603113]]
[[-0.4325307 -0.97379291 -0.40962613]
[ 0.51810217 -0.0674812 1.20929408]]]
Luabit:printTable weather sunny
Luabit:printTable color red
Luabit:printTable anumber 10
Luabit:printTable afloat 1.2339999675751
somestring mistletoe
Luabit table2 row1 col1
Luabit table2 meta data
res {'bear': 'happy', 'foo': 'bar', 'result': 12.345}
I know it's a bit hacky...
Hmm this is a good spot! Do you think there's a more conclusive way we can fix it? It'll be painful running Python scripts like this from IDEs or in general with larger projects.
It's odd that this works with Anaconda build on OSX but not Linux...
Well... if you do:
CONDA_DIR=$(dirname $(dirname $(which conda)))
pushd $CONDA_DIR/lib
mkdir disabled
mv libgfortran* disabled
popd
... then pytorch will work ok. Not sure if that breaks anything inside conda though?
Which distribution of Python do you recommend with PyTorch?
Disabling Fortran sounds like a risky business, Numpy's backend is written in C/Fortran I think...
Fair enough. I'm hoping it will pick up the system one though: basically the fortran library supplied with anaconda seems to be old, I think.
Matrix multiplication might be a good test, but seems it uses mkl. In a first window, start python, run a matrix multiplication, which will load any required libraries, then leave the terminal open:
$ cat /tmp/test.py
import numpy as np
a = np.random.randn(3,20)
b = np.random.randn(20,2)
c = a.dot(b)
print('c', c)
$ python -i /tmp/test.py
c [[ -4.71386118 -2.23634845]
[ 2.39371308 -0.97695027]
[ 7.18042807 18.71131986]]
>>>
Leaving this open, in a second window, look at what libraries are loaded:
$ sudo bash
# cd /proc/$(ps -ef | grep test.py | grep -v grep | awk '{print $2}')
root:/proc/649# grep fort maps | awk '{print $6}' | uniq
root:/proc/649# grep blas maps | awk '{print $6}' | uniq
root:/proc/649# grep lapack maps | awk '{print $6}' | uniq
/media/ubuntu/data/norep/Downloads/anaconda/lib/python3.5/site-packages/numpy/linalg/lapack_lite.cpython-35m-x86_64-linux-gnu.so
root:/proc/649# grep mkl maps | awk '{print $6}' | uniq
/media/ubuntu/data/norep/Downloads/anaconda/lib/libmkl_avx2.so
/media/ubuntu/data/norep/Downloads/anaconda/lib/libmkl_core.so
/media/ubuntu/data/norep/Downloads/anaconda/lib/libmkl_intel_thread.so
/media/ubuntu/data/norep/Downloads/anaconda/lib/libmkl_intel_lp64.so
What can we test that will use fortran, and not use mkl?
(Alternatively ,we can just relink the fortran libraries in anaconda to point to the system ones perhaps, eg see http://stackoverflow.com/questions/9628273/libgfortran-version-gfortran-1-4-not-found/10817240#10817240
CONDA_DIR=$(dirname $(dirname $(which conda)))
pushd $CONDA_DIR/lib
mkdir disabled
mv libgfortran* disabled
ln -s /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 .
mv disabled/libgfortran.so.3 .
popd
I htink it would be good to have a test case of something that needs the fortran libraries though (other than torch). Maybe you can run a few of your projects/applications, and see if any choke with the fortran library removed from anaconda/lib directory?)
Which distribution of Python do you recommend with PyTorch?
Just noticed this question. Well, what I'm testing with is simply a standard python 3.4, then create a virtualenv, then pip install
stuff. Like:
sudo apt-get install python3 python3-virtualenv python3-dev
virtualenv -p python3 ~/env3
source ~/env3/bin/activate
pip install -U pip
pip install -U setuptools
pip install -U wheel
pip install -U numpy
pip install -U scipy
pip install -U pytest
pip install -U docopt
you can put all of these in a requirements.txt, and install by doing simply:
python install -r requirements.txt
$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 python pybit.py
This worked without a hitch for me too.
I htink it would be good to have a test case of something that needs the fortran libraries though (other than torch). Maybe you can run a few of your projects/applications, and see if any choke with the fortran library removed from anaconda/lib directory?)
Yes, I'm going to try this, but only if this doesn't work:
Alternatively ,we can just relink the fortran libraries in anaconda to point to the system ones perhaps
I think this is the best bet. I'm going to try it and will post my results
Ok, I re-linked the Anaconda fortran libraries as you described and ran the tests, they all passed
41 passed in 23.48 seconds
Also python runner.py
runs perfectly both on CPU and GPU. Great stuff!
I will still try to investigate the effects of re-linking fortran on other Python projects.
Cool. Sounds good :-)