hughperkins / pytorch

Python wrappers for torch and lua

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

'version `GFORTRAN_1.4'' not found when running tests

cjmcmurtrie opened this issue · comments

commented

Hi Hugh, I'm now attempting a build on an Ubuntu 14.04 machine with CUDA 7.5. I have a fresh Torch install and Python 2.7.11 Anaconda distribution.

After building PyTorch, running the tests fails with the followings errors:

platform linux2 -- Python 2.7.11, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/dg/anaconda2/bin/python
cachedir: .cache
rootdir: /home/dg/pytorch, inifile: 
collected 0 items / 8 errors 

======================================================= ERRORS =======================================================
______________________________________ ERROR collecting test/testByteTensor.py _______________________________________
test/testByteTensor.py:3: in <module>
    import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
    ???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
    ???
E   ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
_____________________________________ ERROR collecting test/testDoubleTensor.py ______________________________________
test/testDoubleTensor.py:3: in <module>
    import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
    ???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
    ???
E   ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
______________________________________ ERROR collecting test/testFloatTensor.py ______________________________________
test/testFloatTensor.py:3: in <module>
    import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
    ???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
    ???
E   ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
______________________________________ ERROR collecting test/testLongTensor.py _______________________________________
test/testLongTensor.py:3: in <module>
    import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
    ???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
    ???
E   ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
_________________________________________ ERROR collecting test/test_nnx.py __________________________________________
test/test_nnx.py:2: in <module>
    import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
    ???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
    ???
E   ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
_________________________________________ ERROR collecting test/test_pynn.py _________________________________________
test/test_pynn.py:2: in <module>
    import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
    ???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
    ???
E   ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
_______________________________________ ERROR collecting test/test_pytorch.py ________________________________________
test/test_pytorch.py:5: in <module>
    import PyTorch
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
    ???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
    ???
E   ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
________________________________________ ERROR collecting test/test_throw.py _________________________________________
test/test_throw.py:1: in <module>
    import PyTorchHelpers
build/bdist.linux-x86_64/egg/PyTorchHelpers.py:1: in <module>
    ???
build/bdist.linux-x86_64/egg/PyTorch.py:7: in <module>
    ???
build/bdist.linux-x86_64/egg/PyTorch.py:6: in __bootstrap__
    ???
E   ImportError: /home/dg/anaconda2/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
============================================== 8 error in 0.26 seconds ===============================================

Any idea what could be causing this? Since I was using the same distribution (Anaconda) with a successful build on OSX, I'm not sure what the problem could be here.

You need fortran installed. On other platforms, install-deps installs it, eg https://github.com/torch/distro/blob/master/install-deps#L181 , but for Mac it seems not to install it. Maybe it's normally present on Mac by default, and maybe you have it, but for some reason it's not being picked up, when running torch from python?

Can you first check that torch itself works ok?

luajit -l torch -e 'torch.test()'
luajit -l cutorch -e 'cutorch.test()'
luajit -l nn -e 'nn.test()'
luajit -l cunn -e 'nn.testcuda()'

If that works, let's try to find the fortran library that torch uses. Can you try something like:

otool -l ~/torch/install/lib/libTH.so

(or it might be:

otool -l ~/torch/install/lib/libTH.dylib

)
If otool -l doenst work, can you try otool -L instead?

commented

Hi Hugh, all the above Torch tests run correctly. However not the Fortran tests, this is the outcome:

No command 'otool' found, did you mean:
 Command 'atool' from package 'atool' (universe)
 Command 'potool' from package 'potool' (universe)
 Command 'btool' from package 'barry-util' (universe)

By the way, this is an Ubuntu 14.04 machine I'm on now, not a Mac

By the way, this is an Ubuntu 14.04 machine I'm on now, not a Mac

:-P

Ok... so, on linux, the command will be something like:

ldd ~/torch/install/lib/libTH.so

On my box this gives:

...
    libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f2d071ea000)
...

And also:

dpkg -l '*fortran*' | grep ii

This should give something like:

$ dpkg -l '*fortran*' | grep ii
ii  gfortran                4:5.2.1-3ubuntu1 amd64        GNU Fortran 95 compiler
ii  gfortran-5              5.2.1-22ubuntu2  amd64        GNU Fortran compiler
ii  libgfortran-5-dev:amd64 5.2.1-22ubuntu2  amd64        Runtime library for GNU Fortran applications (development files)
ii  libgfortran3:amd64      5.2.1-22ubuntu2  amd64        Runtime library for GNU Fortran applications
commented

This is the result of ldd ~/torch/install/lib/libTH.so. It looks like libgfortran is in usr/lib but the the tests script is looking for it in Anaconda directories? :

$ ldd ~/torch/install/lib/libTH.so
    linux-vdso.so.1 =>  (0x00007ffc6f34f000)
    libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f9bfde9d000)
    liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007f9bfd700000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9bfd3fa000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f9bfd1eb000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9bfcfcd000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9bfcc07000)
    /lib64/ld-linux-x86-64.so.2 (0x00005605544ad000)
    libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f9bfb3fe000)
    libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f9bfb0e4000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9bfaecd000)
    libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f9bfac91000)

This is the result of dpkg -l '*fortran*' | grep ii:

$ dpkg -l '*fortran*' | grep ii
ii  gfortran                                              4:4.8.2-1ubuntu6                                    amd64        GNU Fortran 95 compiler
ii  gfortran-4.8                                          4.8.4-2ubuntu1~14.04.1                              amd64        GNU Fortran compiler
ii  libgfortran-4.8-dev:amd64                             4.8.4-2ubuntu1~14.04.1                              amd64        Runtime library for GNU Fortran applications (development files)
ii  libgfortran3:amd64                                    4.8.4-2ubuntu1~14.04.1                              amd64        Runtime library for GNU Fortran applications

Hmmmm. What if you preload the libgfortran, something like:

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 python myscript.py

managed to reproduce the problem, using conda, and LD_PRELOAD seems to fix it:

$ which python
/mnt/anaconda/bin/python
ubuntu@ip-172-30-4-214:~/git/pytorch/simpleexample$ python pybit.py             
Traceback (most recent call last):
  File "pybit.py", line 3, in <module>
    import PyTorch
ImportError: /mnt/anaconda/bin/../lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3)
ubuntu@ip-172-30-4-214:~/git/pytorch/simpleexample$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 python pybit.py 
Luabit:__init(  green   )
green
type(luabit) <class 'PyTorchLua.Luabit'>
out from lua    (1,.,.) = 
  0.8097 -0.6946  0.4558
 -0.2452  0.4862 -0.1560

(2,.,.) = 
 -0.4325 -0.9738 -0.4096
  0.5181 -0.0675  1.2093
[torch.FloatTensor of size 2x2x3]

outTensor [[[ 0.80968177 -0.69459283  0.45584014]
  [-0.24516164  0.48619133 -0.15603113]]

 [[-0.4325307  -0.97379291 -0.40962613]
  [ 0.51810217 -0.0674812   1.20929408]]]
Luabit:printTable       weather sunny
Luabit:printTable       color   red
Luabit:printTable       anumber 10
Luabit:printTable       afloat  1.2339999675751
somestring      mistletoe
Luabit table2   row1    col1
Luabit table2   meta    data
res {'bear': 'happy', 'foo': 'bar', 'result': 12.345}

I know it's a bit hacky...

commented

Hmm this is a good spot! Do you think there's a more conclusive way we can fix it? It'll be painful running Python scripts like this from IDEs or in general with larger projects.

It's odd that this works with Anaconda build on OSX but not Linux...

Well... if you do:

CONDA_DIR=$(dirname $(dirname $(which conda)))
pushd $CONDA_DIR/lib
mkdir disabled
mv libgfortran* disabled
popd

... then pytorch will work ok. Not sure if that breaks anything inside conda though?

commented

Which distribution of Python do you recommend with PyTorch?

Disabling Fortran sounds like a risky business, Numpy's backend is written in C/Fortran I think...

Fair enough. I'm hoping it will pick up the system one though: basically the fortran library supplied with anaconda seems to be old, I think.

Matrix multiplication might be a good test, but seems it uses mkl. In a first window, start python, run a matrix multiplication, which will load any required libraries, then leave the terminal open:

$ cat /tmp/test.py
import numpy as np

a = np.random.randn(3,20)
b = np.random.randn(20,2)

c = a.dot(b)
print('c', c)

$ python -i /tmp/test.py
c [[ -4.71386118  -2.23634845]
 [  2.39371308  -0.97695027]
 [  7.18042807  18.71131986]]
>>> 

Leaving this open, in a second window, look at what libraries are loaded:

$ sudo bash
# cd /proc/$(ps -ef | grep test.py | grep -v grep | awk '{print $2}')
root:/proc/649# grep fort maps | awk '{print $6}' | uniq
root:/proc/649# grep blas maps | awk '{print $6}' | uniq
root:/proc/649# grep lapack maps | awk '{print $6}' | uniq
/media/ubuntu/data/norep/Downloads/anaconda/lib/python3.5/site-packages/numpy/linalg/lapack_lite.cpython-35m-x86_64-linux-gnu.so
root:/proc/649# grep mkl maps | awk '{print $6}' | uniq
/media/ubuntu/data/norep/Downloads/anaconda/lib/libmkl_avx2.so
/media/ubuntu/data/norep/Downloads/anaconda/lib/libmkl_core.so
/media/ubuntu/data/norep/Downloads/anaconda/lib/libmkl_intel_thread.so
/media/ubuntu/data/norep/Downloads/anaconda/lib/libmkl_intel_lp64.so

What can we test that will use fortran, and not use mkl?

(Alternatively ,we can just relink the fortran libraries in anaconda to point to the system ones perhaps, eg see http://stackoverflow.com/questions/9628273/libgfortran-version-gfortran-1-4-not-found/10817240#10817240

CONDA_DIR=$(dirname $(dirname $(which conda)))
pushd $CONDA_DIR/lib
mkdir disabled
mv libgfortran* disabled
ln -s /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 .
mv disabled/libgfortran.so.3 .
popd

I htink it would be good to have a test case of something that needs the fortran libraries though (other than torch). Maybe you can run a few of your projects/applications, and see if any choke with the fortran library removed from anaconda/lib directory?)

Which distribution of Python do you recommend with PyTorch?

Just noticed this question. Well, what I'm testing with is simply a standard python 3.4, then create a virtualenv, then pip install stuff. Like:

sudo apt-get install python3 python3-virtualenv python3-dev
virtualenv -p python3 ~/env3
source ~/env3/bin/activate
pip install -U pip
pip install -U setuptools
pip install -U wheel
pip install -U numpy
pip install -U scipy
pip install -U pytest
pip install -U docopt

you can put all of these in a requirements.txt, and install by doing simply:

python install -r requirements.txt
commented

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 python pybit.py

This worked without a hitch for me too.

I htink it would be good to have a test case of something that needs the fortran libraries though (other than torch). Maybe you can run a few of your projects/applications, and see if any choke with the fortran library removed from anaconda/lib directory?)

Yes, I'm going to try this, but only if this doesn't work:

Alternatively ,we can just relink the fortran libraries in anaconda to point to the system ones perhaps

I think this is the best bet. I'm going to try it and will post my results

commented

Ok, I re-linked the Anaconda fortran libraries as you described and ran the tests, they all passed

41 passed in 23.48 seconds

Also python runner.py runs perfectly both on CPU and GPU. Great stuff!

I will still try to investigate the effects of re-linking fortran on other Python projects.

Cool. Sounds good :-)