import shenfun: Illegal instruction (core dumped) error

Question

import shenfun: Illegal instruction (core dumped) error

atif4321 opened this issue 6 years ago · comments

Dear mikaem,

I am trying to install shenfun package through anaconda 3.5.0 with python-3.6.0
but it is not working and giving core dumped on exporting it,

please help me to resolve the issue.

$ python
Python 3.6.0 | packaged by conda-forge | (default, Feb 9 2017, 14:36:55)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import shenfun
Illegal instruction (core dumped)

Mikael Mortensen · Answer 1 · Fri Mar 08 2019 16:38:27 GMT+0800 (China Standard Time)

Hi

Thanks for reporting the issue, but this is not reproducible for my linux laptop:

$ conda create --name shenfuntest python=3.6.0 shenfun mpich nomkl
$ conda activate shenfuntest  
$ python
Python 3.6.0 | packaged by conda-forge | (default, Feb  9 2017, 14:36:55) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shenfun
>>>

For future reference, you need to provide at least something that shows where and how things go wrong. There is nothing to go by in just a segfault. Could be anything. This error is probably not an error in shenfun. For example, can you import mpi4py, numpy?

sag123kas · Answer 2 · Fri Mar 15 2019 17:05:44 GMT+0800 (China Standard Time)

Yes, I could able to import these two packages, snapshot of the same is pasted below for your information.
Let me know what other things could impact such failure.

$ conda activate shenfuntest
(shenfuntest) $ python
Python 3.6.0 | packaged by conda-forge | (default, Feb 9 2017, 14:36:55)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import numpy
import mpi4py
import shenfun
Illegal instruction
(shenfuntest) $

Francis J. Poulin · Answer 3 · Thu May 28 2020 23:26:50 GMT+0800 (China Standard Time)

I am trying to install shenfun on a fairly larger server and have installed Anaconda.

The good news is that when I installed mpi4py-fft it worked beautifully.

The bad news is that when I tried and set up shenfun, I get an illegal instruction error when I import shenfun. Using faulthandler, I found the following infomration that I can't really make heads or tails out of. Any idea what might be wrong?

`$ python -q -X faulthandler
>>> import shenfun
Fatal Python error: Illegal instruction

Current thread 0x00002ba78b6fcec0 (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1109 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load

File"/home/fpoulin/software/software/anaconda3/envs/shenfun/lib/python3.8/sitepackages/shenfun/optimization/cython/init.py", line 1 in
File "", line 219 in _call_with_frames_removed
File "", line 783 in exec_module
File "", line 671 in _load_unlocked
File "", line 975 in _find_and_load_unlocked
File "", line 991 in _find_and_load
File "/home/fpoulin/software/software/anaconda3/envs/shenfun/lib/python3.8/sitepackages/shenfun/chebyshev/la.py", line 6 in
File "", line 219 in _call_with_frames_removed
File "", line 783 in exec_module
File "", line 671 in _load_unlocked
File "", line 975 in _find_and_load_unlocked
File "", line 991 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1042 in _handle_fromlist
File "/home/fpoulin/software/software/anaconda3/envs/shenfun/lib/python3.8/sitepackages/shenfun/chebyshev/init.py", line 2 in
File "", line 219 in _call_with_frames_removed
File "", line 783 in exec_module
File "", line 671 in _load_unlocked
File "", line 975 in _find_and_load_unlocked
File "", line 991 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1042 in _handle_fromlist
File "/home/fpoulin/software/software/anaconda3/envs/shenfun/lib/python3.8/sitepackages/shenfun/init.py", line 37 in
File "", line 219 in _call_with_frames_removed
File "", line 783 in exec_module
File "", line 671 in _load_unlocked
File "", line 975 in _find_and_load_unlocked
File "", line 991 in _find_and_load
File "", line 1 in
Illegal instruction
`

Mikael Mortensen · Answer 4 · Thu May 28 2020 23:32:04 GMT+0800 (China Standard Time)

No idea, except that it's cython related. Is this installed from conda-forge? Perhaps you need to compile it by hand on the server?

Francis J. Poulin · Answer 5 · Thu May 28 2020 23:45:33 GMT+0800 (China Standard Time)

Thanks for pointing that out, now it seems pretty obvious that cython was the initial source of the problem.

Yes, I did use conda-forge and noticed that cython was not installed.

I pip installed cython but I fear that might have used a default version of cython that is on the server. I will try again with a new install and include cython on the list.

By default, the server is openmpi. Since shenfun needs mpich, I presume I should remove all openmpi libraries and load mpich, before I do the conda create? Or is conda smart enough to do all of that?

Mikael Mortensen · Answer 6 · Thu May 28 2020 23:50:03 GMT+0800 (China Standard Time)

Did you check that you used anaconda's pip? It's a common mistake to use system pip.

Openmpi should not be a problem. I had some issues with them in the past, but I don't remember why. You should probably use the mpi that has been installed on the server, assuming the people that installed them used systemspecific optimizations. Conda-forge versions do not know about you system:-)

Francis J. Poulin · Answer 7 · Fri May 29 2020 00:07:08 GMT+0800 (China Standard Time)

I believe that I did use the conda version of pip but I removed all that I had before and did a new conda create now with cython, just to be sure.

I also used python 3.7.6, which is what I have on my local machine, and that helped.

Good news: I can import cython and the errors I had previously go away.

Bad news: there is a new error, which I get when I install mpi4py_fft.

`$ python -q -X faulthandler
>>> import cython
>>> import mpi4py
>>> import mpi4py_fft

Traceback (most recent call last):
File "", line 1, in
File "/home/fpoulin/software/software/anaconda3/envs/shenfun/lib/python3.7/site-packages/mpi4py_fft/init.py", line 22, in
from .distarray import DistArray, newDistArray, Function
File "/home/fpoulin/software/software/anaconda3/envs/shenfun/lib/python3.7/site-packages/mpi4py_fft/distarray.py", line 4, in
from mpi4py import MPI
ImportError: libhcoll.so.1: cannot open shared object file: No such file or directory
>>> import shenfun
Traceback (most recent call last):
File "", line 1, in
File "/home/fpoulin/software/software/anaconda3/envs/shenfun/lib/python3.7/site-packages/shenfun/init.py", line 36, in
from mpi4py import MPI
ImportError: libhcoll.so.1: cannot open shared object file: No such file or directory
`

Francis J. Poulin · Answer 8 · Fri May 29 2020 00:40:32 GMT+0800 (China Standard Time)

Minor update:

I can confirm that when I install mpi4py-fft with python 3.8.2 it just works. However, when I use python 3.7.6, it doesn't seem to work on the server. I have another computer that has them both installed with 3.8.2 so I will try that again, and see if I have any more success.

Francis J. Poulin · Answer 9 · Fri May 29 2020 00:50:43 GMT+0800 (China Standard Time)

Minor update:

When I install shenfun using python 3.8.2 I get that mpi4py_fft works, but then I get the fatal python error: illegal instruction, when I try and run shenfun. Since this conda install worked on one of my computers I'm not sure why it doesn't work with 3.8.2 on the server. Hmmm....

Francis J. Poulin · Answer 10 · Mon Jun 08 2020 22:52:23 GMT+0800 (China Standard Time)

Sorry to bother you with this again but any idea why I might get a failure of shenfun when I use this particular version of python?

Mikael Mortensen · Answer 11 · Mon Jun 08 2020 23:12:29 GMT+0800 (China Standard Time)

Sorry, no idea. Shenfun is tested automatically on python 3.6, 3.7 and 3.8, so it's probably not a regular python thing.

Francis J. Poulin · Answer 12 · Tue Jun 09 2020 01:59:54 GMT+0800 (China Standard Time)

Thaks. It turns out that when I tried the exact line on my laptop I ended up getting the same error.

This leads me to believe that I should try and find out exactly what I did to create.

Unfortunately, when I try and create shenfun, as I did before, it doesn't seem to work.

conda create --name sf5 -c conda-forge shenfun mpich python=3.8.2

This yields similar errors as far as I can tell.

$ python -q -X faulthandler

import shenfun
Fatal Python error: Illegal instruction

Current thread 0x00007fa629660740 (most recent call first):
File "", line 219 in _call_with_frames_removed
File "", line 1109 in exec_module
File "", line 671 in _load_unlocked
File "", line 975 in _find_and_load_unlocked
File "", line 991 in _find_and_load
File "/home/fpoulin/software/anaconda3/envs/sf5/lib/python3.8/site-packages/shenfun/optimization/cython/init.py", line 1 in
File "", line 219 in _call_with_frames_removed
File "", line 783 in exec_module
File "", line 671 in _load_unlocked
File "", line 975 in _find_and_load_unlocked
File "", line 991 in _find_and_load
File "/home/fpoulin/software/anaconda3/envs/sf5/lib/python3.8/site-packages/shenfun/chebyshev/la.py", line 6 in
File "", line 219 in _call_with_frames_removed
File "", line 783 in exec_module
File "", line 671 in _load_unlocked
File "", line 975 in _find_and_load_unlocked
File "", line 991 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1042 in _handle_fromlist
File "/home/fpoulin/software/anaconda3/envs/sf5/lib/python3.8/site-packages/shenfun/chebyshev/init.py", line 2 in
File "", line 219 in _call_with_frames_removed
File "", line 783 in exec_module
File "", line 671 in _load_unlocked
File "", line 975 in _find_and_load_unlocked
File "", line 991 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1042 in _handle_fromlist
File "/home/fpoulin/software/anaconda3/envs/sf5/lib/python3.8/site-packages/shenfun/init.py", line 37 in
File "", line 219 in _call_with_frames_removed
File "", line 783 in exec_module
File "", line 671 in _load_unlocked
File "", line 975 in _find_and_load_unlocked
File "", line 991 in _find_and_load
File "", line 1 in
Illegal instruction (core dumped)

Mikael Mortensen · Answer 13 · Tue Jun 09 2020 02:29:38 GMT+0800 (China Standard Time)

conda deactivate
conda create --name sf5 -c conda-forge shenfun mpich python=3.8.2
conda activate sf5
export PYTHONPATH=''
python -c 'import shenfun'

Works for me on my mac. I can try in a docker linux image later.

Francis J. Poulin · Answer 14 · Tue Jun 09 2020 02:57:16 GMT+0800 (China Standard Time)

I have tried this on two linux machine and am getting the same problem so if you could check, sometime, that would be greatly appreciated.

If i wanted to check to see how my working version of shenfun was installed, is there an easy way to find that out? I should probably try that again as a sanity check.

Mikael Mortensen · Answer 15 · Tue Jun 09 2020 14:37:13 GMT+0800 (China Standard Time)

So I tried it in a docker image and I can reproduce the illegal instruction. Solution for me in the docker container, within sf5 environment:

conda install -c conda-forge comilers cython
git clone https://github.com/spectralDNS/shenfun.git
cd shenfun
python setup.py build_ext -i
export PYTHONPATH=~/shenfun
python -c 'import shenfun'

runs without mistake. So there must be an incompatibility between the shenfun built by conda-forge some of the packages you get when installing with python=3.8.2. I don't know what it is, but the solution is to build it locally.

Francis J. Poulin · Answer 16 · Tue Jun 09 2020 21:52:57 GMT+0800 (China Standard Time)

Thank you!

I needed to fix a typo above, comilers -> compilers and add numba to the list but I can say that shenfun now imports on the server!

I'm now going to test some demos but I can say that after adding matplotlib, RayleighBenardRK3.py successfully runs.

Thanks again Mikael, the help is greatly appreciated.

Mikael Mortensen · Answer 17 · Thu Sep 10 2020 20:55:06 GMT+0800 (China Standard Time)

@francispoulin I just found a bug that I think was causing this segfault! There's a line in setup.py that adds compiler options, like -O3 etc. For my native computer I had included -march=native. But I just found that as an unintended side-effect -march=native was also added to the conda-forge build! The bug has probably hit a lot more people than just you, so it's really annoying that it has been there for so long and I didn't figure it out before now.

Francis J. Poulin · Answer 18 · Thu Sep 10 2020 21:31:20 GMT+0800 (China Standard Time)

Bugs are rarely easy beats to tame and this one sounds especially challenging. Congratulations on finding it and I really appreciate you fixing it, and many others should too. I will update shenfun and make sure everything is good on my end.