Cython fails if shenfun is imported from conda

Question

Cython fails if shenfun is imported from conda

sebglane opened this issue 4 years ago · comments

Hi,
I am trying to get the current github version of shenfun running in a conda-environment. I installed shenfun using conda as described in the documentation and cloned the github repository.

I could fix an error which was related to missing include files for numpy. But the include files for ios seem to be missing too, see below. I am somehow struggling to configure Cython such that the import shenfun works fine.

Best wishes,
Sebastian

(shenfun) sg@Piola:~/Documents/Projects/shenfun$ python
Python 3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 08:20:52) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import pyximport
>>> pyximport.install(setup_args={"include_dirs":numpy.get_include()})
(None, <pyximport.pyximport.PyxImporter object at 0x7f854f0f95e0>)
>>> import shenfun
In file included from /home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1832:0,
                 from /home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /home/sg/.pyxbld/temp.linux-x86_64-3.8/pyrex/shenfun/optimization/cython/la.c:600:
/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
/home/sg/.pyxbld/temp.linux-x86_64-3.8/pyrex/shenfun/optimization/cython/la.c:602:10: fatal error: ios: No such file or directory
 #include "ios"
          ^~~~~
compilation terminated.
Traceback (most recent call last):
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/spawn.py", line 157, in _spawn_posix
    raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/home/sg/anaconda3/envs/shenfun/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 214, in load_module
    so_path = build_module(module_name, pyxfilename, pyxbuild_dir,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 186, in build_module
    so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyxbuild.py", line 102, in pyx_to_dll
    dist.run_commands()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command '/home/sg/anaconda3/envs/shenfun/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sg/Documents/Projects/shenfun/shenfun/__init__.py", line 33, in <module>
    from . import chebyshev
  File "/home/sg/Documents/Projects/shenfun/shenfun/chebyshev/__init__.py", line 2, in <module>
    from . import la
  File "/home/sg/Documents/Projects/shenfun/shenfun/chebyshev/la.py", line 6, in <module>
    from shenfun.optimization.cython import la
  File "/home/sg/Documents/Projects/shenfun/shenfun/optimization/cython/__init__.py", line 1, in <module>
    from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMA_SymSolve, \
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 459, in load_module
    module = load_module(fullname, self.path,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 231, in load_module
    raise exc.with_traceback(tb)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 214, in load_module
    so_path = build_module(module_name, pyxfilename, pyxbuild_dir,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 186, in build_module
    so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyxbuild.py", line 102, in pyx_to_dll
    dist.run_commands()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
ImportError: Building module shenfun.optimization.cython.la failed: ["distutils.errors.CompileError: command '/home/sg/anaconda3/envs/shenfun/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1\n"]
>>> import numpy
>>> import shenfun
In file included from /home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1832:0,
                 from /home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /home/sg/.pyxbld/temp.linux-x86_64-3.8/pyrex/shenfun/optimization/cython/la.c:600:
/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
/home/sg/.pyxbld/temp.linux-x86_64-3.8/pyrex/shenfun/optimization/cython/la.c:602:10: fatal error: ios: No such file or directory
 #include "ios"
          ^~~~~
compilation terminated.
Traceback (most recent call last):
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/spawn.py", line 157, in _spawn_posix
    raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/home/sg/anaconda3/envs/shenfun/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 214, in load_module
    so_path = build_module(module_name, pyxfilename, pyxbuild_dir,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 186, in build_module
    so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyxbuild.py", line 102, in pyx_to_dll
    dist.run_commands()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command '/home/sg/anaconda3/envs/shenfun/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sg/Documents/Projects/shenfun/shenfun/__init__.py", line 33, in <module>
    from . import chebyshev
  File "/home/sg/Documents/Projects/shenfun/shenfun/chebyshev/__init__.py", line 2, in <module>
    from . import la
  File "/home/sg/Documents/Projects/shenfun/shenfun/chebyshev/la.py", line 6, in <module>
    from shenfun.optimization.cython import la
  File "/home/sg/Documents/Projects/shenfun/shenfun/optimization/cython/__init__.py", line 1, in <module>
    from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMA_SymSolve, \
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 459, in load_module
    module = load_module(fullname, self.path,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 231, in load_module
    raise exc.with_traceback(tb)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 214, in load_module
    so_path = build_module(module_name, pyxfilename, pyxbuild_dir,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyximport.py", line 186, in build_module
    so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/pyximport/pyxbuild.py", line 102, in pyx_to_dll
    dist.run_commands()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/home/sg/anaconda3/envs/shenfun/lib/python3.8/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
ImportError: Building module shenfun.optimization.cython.la failed: ["distutils.errors.CompileError: command '/home/sg/anaconda3/envs/shenfun/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1\n"]

Mikael Mortensen · Answer 1 · Mon May 04 2020 21:22:56 GMT+0800 (China Standard Time)

Hi
Did you try to run

python setup.py build_ext -i

in the cloned shenfun repository? I have not tried this with pyximport before and I think the problem lies there somehow.

sglane · Answer 2 · Mon May 04 2020 21:41:50 GMT+0800 (China Standard Time)

Hi Mikael,
yes this works perfect! May it is worth to mention this somewhere in the documentations.
Closing this issue.
Thank you
Sebastian

Mikael Mortensen · Answer 3 · Mon May 04 2020 21:46:32 GMT+0800 (China Standard Time)

Yes, I see that the only suggestion for building yourself locally now (in the installation chapter on readthedocs) is

pip install .

I think that would work for you as well. But I will add a

python setup.py build_ext -i

since that is how I usually do it myself.