FilippoBovo / robustats

When I install robustats on my machine, it works. On my colleagues machine, however, it fails to install with a long output including this:

 creating build/temp.macosx-10.15-x86_64-3.7
    creating build/temp.macosx-10.15-x86_64-3.7/c
    compile options: ‘-I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/.../venv/include -I/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c’
    extra options: ‘-std=c99’
    clang: c/robustats.cclang: c/_robustats.c
    clang: c/base.c
    c/_robustats.c:3:10: fatal error: ‘numpy/arrayobject.h’ file not found
    #include <numpy/arrayobject.h>
             ^~~~~~~~~~~~~~~~~~~~~
    1 error generated.

When I run it on my machine, this part runs through with only a warning:

creating build/temp.macosx-10.15-x86_64-3.7
  creating build/temp.macosx-10.15-x86_64-3.7/c
  compile options: '-I/Users/.../.venv/lib/python3.7/site-packages/numpy/core/include -I/Users/.../.venv/include -I/Users/.../.pyenv/versions/3.7.6/include/python3.7m -c'
  extra options: '-std=c99'
  clang: c/_robustats.c
  clang: c/robustats.c
  clang: c/base.c
  In file included from c/_robustats.c:3:
  In file included from /Users/.../.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
  In file included from /Users/.../.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
  In file included from /Users/.../.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822:
  /Users/.../.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
  #warning "Using deprecated NumPy API, disable it with " \
   ^
  1 warning generated.

Our environment is very similar:

We both are on MacOS 10.15, use Python 3.7.6 and these packages:

numpy==1.19.1
wheel==0.35.1
setuptools==50.3.0

I installed python using pyenv, while he installed it using homebrew.

As you can see in these logs, on his machine the numpy include directory does not actually get passed to the compile options. I could not figure out what is different between our two environments, but I figured out how to fix his issue:

robustats/setup.py

Line 33 in 8d02900

include_dirs=numpy.distutils.misc_util.get_numpy_include_dirs(),

According to the distutils docs, this include_dirs parameter should be passed to the Extension constructor directly, but here it is passed directly to setup() instead. After cloning your project, changing this line and installing it, installation worked on both of our machines.

I sadly could not figure out why this still worked on my machine but not on his. It might have to do with the python installation (homebrew vs pyenv), but I don't know for sure.

I will also create a PR with a fix, I just wanted to write down our problem.

Hi, @pspeter. Thanks for your investigation. Could you please try to install this library in a virtual environment created with Virtualenv? I am asking this because I know that sometimes creating virtual environments with Pyenv gives issues.

You can create and activate the virtual environment with this command:

virtualenv --python=$(pyenv prefix 3.7.6)/bin/python venv
source venv/bin/activate

Then, try to install the robustats library and let me know if it works, please.

Also, could you try to replace numpy.distutils.misc_util.get_numpy_include_dirs with numpy.get_include() in the Extension only and do not use it in the setup function?

Oh, by the way, also try to upgrade the base Python libraries:

pip install --upgrade pip setuptools wheel

I am also linking this for future reference.

In addition, could you please show me the output of the following Python code?

import numpy as np

print(np.get_include())

import numpy.distutils.misc_util
print(np.distutils.misc_util.get_numpy_include_dirs())

Also, from the shell, could you show me the output of the following command?

which python

Thanks

Hey Filippo,

I'd love to help, but sadly I am currently switching jobs and don't have access to my and my colleague's work computers anymore. However, the StackOverflow answer you linked above seems like a good solution. Especially the comment by dashesy seems relevant:

include_dirs passed to setup() gets ignored in the latest distutils, it has to be passed to each Extension, at least on mac

Ok. Thanks, @pspeter. Good luck with your new job!

FYI. The patch presented by @pspeter fixed the issue on my local installation. Having this change in master would help out.

MacOS
Python 3.7.7

In addition, could you please show me the output of the following Python code?
import numpy as np

print(np.get_include())

import numpy.distutils.misc_util
print(np.distutils.misc_util.get_numpy_include_dirs())
Also, from the shell, could you show me the output of the following command?
which python
Thanks

@kidddw, thanks for your message. Could you please show me the output of the commands quoted above?

Also, are you using Pyenv?

@FilippoBovo

sure. I am installing dependencies from scratch within a fresh virtual environment using venv. That is, running pip install -e .

Upon successfully installing those dependencies, making use of the patch, the first code produces

/Users/daniel.kidd/PycharmProjects/robustats/venv/lib/python3.7/site-packages/numpy/core/include
['/Users/daniel.kidd/PycharmProjects/robustats/venv/lib/python3.7/site-packages/numpy/core/include']

The second code produces

/Users/daniel.kidd/PycharmProjects/robustats/venv/bin/python

And additionally, the bash code python3 -V returns

Python 3.7.7

I do not use Pyenv, though, interestingly, my coworker who does use Pyenv was able to install from master without the same problems. Pyenv seemed to be the only difference between our approaches.

@kidddw, thanks for running those tests.

It seems that distutils fails to get the correct numpy include directory in your case, which is odd, because I also have a Mac and use virtualenv.

One more question. Did you pass the include_dirs argument only to Extension or to setup as well?

Only to Extension.

That is, the setup.py which succeeds for me is @pspeter 's patch-1 branch merged into your master branch. Or to be completely clear,

from setuptools import setup, Extension

try:
    import numpy.distutils.misc_util
except ModuleNotFoundError:
    from setuptools import dist

    dist.Distribution().fetch_build_eggs(["numpy"])
    import numpy.distutils.misc_util


with open("README.md", "r") as f:
    long_description = f.read()


setup(
    name = 'robustats',
    version = '0.1.5',
    description = 'Robustats is a Python library for high-performance '
                  'computation of robust statistical estimators.',
    long_description=long_description,
    long_description_content_type='text/markdown',
    classifiers=[
        'Programming Language :: Python :: 3',
    ],
    url='https://github.com/FilippoBovo/robustats',
    download_url='https://github.com/FilippoBovo/robustats/archive/v0.1.5.tar.gz',
    author = 'Filippo Bovo',
    author_email = 'bovo.filippo@gmail.com',
    license='MIT',
    packages=['robustats'],
    install_requires=['numpy'],
    ext_modules=[
        Extension(
            name="_robustats",
            sources=["c/_robustats.c", "c/robustats.c", "c/base.c"],
            extra_compile_args=["-std=c99"],
            include_dirs=numpy.distutils.misc_util.get_numpy_include_dirs(),
        )
    ],
)

Without the code for explicitly importing numpy in lines 3--9, I also receive errors when attempting to reference numpy.distutils.misc_util.get_numpy_include_dirs() within Extension.

Ok, thanks. I’ll try to test the change over the weekend and, if it works, merge @pspeter’s pull request to master.

@kidddw, I merged the pull request to master. The new release is also available on Pypi.

thanks @FilippoBovo, working for me now with the latest release.

Great. Happy to hear that!

Numpy not found during compilation of C-extension - include_dirs is not configured correctly in setup.py