Installation issues in Ubuntu
passaglia opened this issue · comments
I'm running 64-bit Ubuntu 15.04. Installing esutil fails with both:
sudo pip3 install esutil
sudo python3 setup.py (in the appropriate folder)
I can install other packages and my python setup seems otherwise well behaved. I have python3-dev installed.
Here's a terminal dump from the pip install: http://pastebin.com/eXbAEqcG . I've spent a long time trying to parse it but I can't seem to find any 'fatal' errors. Any ideas what the cause is?
I don't think I've ported all the extensions over to python3 yet. A rerun of swig may be in order for python3, plus using the 2to3 portion of setuptools.
Ah I see. Thanks for the help. Based on the information that it isn't expected to work in Python3, I tried installing it in Python 2 with the command
sudo pip2 install esutil
This didn't work either, so perhaps the issue is on my end somewhere... I seem to be able to install other packages with pip2 though.
Here's the terminal dump for the pip2 install esutil failure: http://pastebin.com/u25JbSqK
The issue seems to me to be that a lot of these warnings are being treated as errors by gcc? Is this a problem due to some environment variable on my end or due to a call by the code itself?
Again, thanks for the help. I appreciate it.
ahh, interesting about the warnings. The compile flags match that of the python used for installation. Normally python is not compiled with warnings as errors.
what is your python install?
sam@veillantif:/clustering$ python -V/clustering$ python3 -V
Python 2.7.9
sam@veillantif:
Python 3.4.3
Perhaps I should try to reinstall python2 in the hopes that a reinstall will give me the correct compile flags?
Then again, when I do sudo pip2 install scipy --upgrade, a ton of warnings are displayed but in the end it installs successfully.
but where is the python from? Is this your own install, or through a package manager, or what?
Oh, I apologize for the misunderstanding.
My install is through the package manager on ubuntu, apt.
I have the same installation issue running python3.5.1 installed via astroconda on OS X. I think the primary error relates to how import_array() in records.cpp returns NULL when used with python3 but is expected to return nothing. An excerpt from the lengthy terminal dump is below. Any insights welcome!
In file included from /usr/include/_types.h:27:0,
from /usr/include/stdio.h:67,
from /Users/Meredith/anaconda/envs/astroconda/include/python3.5m/Python.h:25,
from esutil/recfile/records.hpp:1,
from esutil/recfile/records.cpp:1:
esutil/recfile/records.cpp: In constructor 'Records::Records(PyObject*, const char*, PyObject*, PyObject*, long long int, int)':
/Users/Meredith/anaconda/envs/astroconda/lib/python3.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:1713:35: error: returning a value from a constructor
#define NUMPY_IMPORT_ARRAY_RETVAL NULL
lots of work is needed to get esutil working on python3, due to swigging most of the C++ code.
and a potential show stopper is the getting of FILE* pointers from file objects at the C layer. This is no longer possible in python3, but a couple of codes do this
I shouldn't say "showstopper"; rather some functionality will no longer be supported
OK, understood; I was afraid my error might have been the tip of the iceberg. I will see if it's possible to use something other than esutil for now. Thanks.
what are you using from esutil? If it is not in the C++ code we can probably get something working faster
Maybe you and @jobovy should talk; it's his code that uses esutil and I'm not sure parts of it are critical or whether it would be simpler to use something else. See https://github.com/jobovy/apogee
The main thing that I'm using that other users of jobovy/apogee (like @mrawls) would want access to is esutil.numpy_util.add_fields and to a lesser extent esutil.numpy_util.remove_fields. I'm also using esutil.htm.HTM for catalog matching, but most users could probably do without that.
OK, thanks.
I've got esutil working on python3. Can you please try using the master branch?
Installation seems to work for me in python 3.4! I got a segmentation fault when I tried to run the htm matching, not sure why. Maybe @mrawls should also check, because I don't use python 3 very much.
Can you try with the latest git master?
Also what happens when you run this?
esutil.htm.unit_tests.test()I should note that these tests pass for me on ubuntu with anaconda and python 3.5.2 built with gcc 4.4.7 (that is standard anaconda), and with esutil extensions built with gcc 5.4.0 (5.4.0-6ubuntu1~16.04.1)
I still get the same error and
esutil.htm.unit_tests.test()
also segfaults. I'm using clang and I don't have python 3.5 installed, so am using anaconda's 3.4.
Anyway, this issue doesn't affect me directly, because I'm not using python 3 and it sounds like it should work if I installed the package properly. Thanks!
OK, this could be clang related. Also, I'm guessing you are running on a mac, which always entails more difficulties.
I don't have a mac for testing.
Can you please run the test within valgrind and send me the error? e.g. put that call into a scrtipt and run
valgrind script.py &> log
and send me the log?
@passaglia it should work now on ubuntu
I've never used valgrind before, so I don't know whether I'm using it correctly, but this seems to be the only thing it returns in the log
==53317== Memcheck, a memory error detector
==53317== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==53317== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==53317== Command: ./script.py
==53317==
Is that helpful? Yes, I am working on a Mac.
The script was
#!/usr/bin/env python
import esutil
esutil.htm.unit_tests.test()
Just to be sure I'm doing that correctly...
it should have a lot of output. maybe valgrind works differently on a mac
I see the problem. You need to run it like this:
valgrind python script.py &> log
I have consolidated some tests into a unit test sub-package.
import esutil as eu
eu.test()Here's the output from
valgrind python script.py &> log
Do you want me to run the eu.test() as well?
looks like a problem converting numpy arrays at the C level. Might be related to krugoffs error on a mac
I just installed python 3.5 and I don't get the htm error (all of the eu.test() tests run fine), so I guess this is a weird python 3.4 bug (or perhaps a bug in an older version of numpy). Sorry for leading us down this wild goose chase!
I'm sure there are many people using 3.4 on os x, so this is still a problem.
also still waiting to hear from @passaglia on ubuntu
@jobovy I think I may have fixed problems that occurred for python 3.4. Do you mind trying the full test suite on 3.4?
Now the installation fails with this output: http://pastebin.com/NRsWyt7H
Yep, bug for OS X, thanks for finding it. I've fixed that one, please try again.
Installation now works and all of the tests pass! 👍
I'm going to close the issue, but @passaglia if you see problems please reopen.
Actually, in Python 2.7 I now get this error when I run the tests: http://pastebin.com/HBHDJwfd
and in Python 3.5, esutil cannot be imported anymore: http://pastebin.com/yNU27NiD
So now Python 3.4 is the only version in which esutil works...
Do you use travis? If you have a test suite, it's pretty easy to setup to test different Python versions and even Mac/Linux I think (although I only ever use the Linux builds).
I think I fixed the problem for 2.7
For the other it is because your 3.4 had an old file laying around in the install directory, which is no longer in the repo, and some code was importing it incorrectly. I had the same problem when I cleaned my install dir.
I remove that incorrect import.
So I expect your tests to work on all three systems now.
(as for travis, I do use it in fitsio but someone else set it up for me)
Yes, everything works now (python 2.7, 3.4, and 3.5), thanks. Strange about the file, I thought I had installed directly from GitHub using
pip install -U --no-deps git+git://github.com/esheldon/esutil.git#egg=esutil
Anyway, works now!
Thanks so much to both of you for getting to the bottom of this!