lookup_id segfaults
SimonKrughoff opened this issue · comments
I just installed this on top of a miniconda install on a Mac running El Cap. When trying to use the lookup_id method, it segfaults unless another method is called first.
E.g.
Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec 6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import esutil
>>> htm = esutil.htm.HTM(8)
>>> htm.lookup_id(1., 2.)
Segmentation fault: 11
but
Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec 6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import esutil
>>> htm = esutil.htm.HTM(8)
>>> htm.intersect(1.,2.,0.0001)
array([1016022])
>>> htm.lookup_id(3., 4.)
array([1016776])
I haven't look into it, but it seems like there is some side effect that is making an assignment lookup_id expects.
Hi Simon. I cannot reproduce this seg fault on my systems (all anaconda on linux). Unfortunately, I don't have a mac for testing.
What version of esutil are you using?
What I mean is, are you using a tagged version, e.g. from pip or installed from master?
I installed from master. I can try the pip version. Is there any other info that would help? It could very easily be an El Cap problem. It's cause a lot of issues.
well, the master is newer actually. I see that gcc is fairly old, I wonder if that could be a problem.
So I just tried the pip installed version and I get:
Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec 6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import esutil
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/krughoff/miniconda/lib/python2.7/site-packages/esutil/__init__.py", line 113, in <module>
import integrate
ImportError: No module named integrate
Looking into it now.
Edit: it does look like it is actually missing.
$ ls miniconda//lib/python2.7/site-packages/esutil
__init__.py cosmology_purepy.pyc json_util.py oracle_util.pyc sfile.py wcsutil.pyc
__init__.pyc fits.py json_util.pyc ostools.py sfile.pyc xmltools.py
algorithm.py fits.pyc misc.py ostools.pyc sqlite_util.py xmltools.pyc
algorithm.pyc hdfs.py misc.pyc plotting.py sqlite_util.pyc
coords.py hdfs.pyc numpy_util.py plotting.pyc stomp_util.py
coords.pyc io.py numpy_util.pyc random.py stomp_util.pyc
cosmology_purepy.py io.pyc oracle_util.py random.pyc wcsutil.py
Regarding gcc version, that is the gcc used to build that version of python (I think). OS X has gone to clang and that version is:
$> gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/c++/4.2.1
Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin15.2.0
Thread model: posix
That's right, the anaconda folks are using an old gcc
One thing to try: in your git checkout, in the /esutil/esutil/htm/ directory, run
bash runswig.sh
If you don't have swig installed it will fail, but otherwise it should write out a new interface file.
Then go back to the root of esutil and reinstall (you might have to remove the build dir). Then repeat your test from a different directory.
O.K. I tried what you suggested and it didn't make a difference (it still seg faulted). I guess I'll have to run it through valgrind to see what's going on.
For reference, following is the output from git diff on the two files that changed when I reran SWIG.
$> git diff htmc.py
diff --git a/esutil/htm/htmc.py b/esutil/htm/htmc.py
index 08ea741..db7d81d 100644
--- a/esutil/htm/htmc.py
+++ b/esutil/htm/htmc.py
@@ -1,5 +1,5 @@
# This file was automatically generated by SWIG (http://www.swig.org).
-# Version 2.0.11
+# Version 3.0.2
#
# Do not make changes to this file unless you know what you are doing--modify
# the SWIG interface file instead.
$> git diff htmc_wrap.cc
diff --git a/esutil/htm/htmc_wrap.cc b/esutil/htm/htmc_wrap.cc
index 1ec86e6..f90c96b 100644
--- a/esutil/htm/htmc_wrap.cc
+++ b/esutil/htm/htmc_wrap.cc
@@ -1,6 +1,6 @@
/* ----------------------------------------------------------------------------
* This file was automatically generated by SWIG (http://www.swig.org).
- * Version 2.0.11
+ * Version 3.0.2
*
* This file is not intended to be easily readable and contains a number of
* coding conventions designed to improve portability and efficiency. Do not make
@@ -560,14 +560,14 @@ SWIG_MangledTypeQueryModule(swig_module_info *start,
swig_module_info *iter = start;
do {
if (iter->size) {
- register size_t l = 0;
- register size_t r = iter->size - 1;
+ size_t l = 0;
+ size_t r = iter->size - 1;
do {
/* since l+r >= 0, we can (>> 1) instead (/ 2) */
- register size_t i = (l + r) >> 1;
+ size_t i = (l + r) >> 1;
const char *iname = iter->types[i]->name;
if (iname) {
- register int compare = strcmp(name, iname);
+ int compare = strcmp(name, iname);
if (compare == 0) {
return iter->types[i];
} else if (compare < 0) {
@@ -611,7 +611,7 @@ SWIG_TypeQueryModule(swig_module_info *start,
of the str field (the human readable name) */
swig_module_info *iter = start;
do {
- register size_t i = 0;
+ size_t i = 0;
for (; i < iter->size; ++i) {
if (iter->types[i]->str && (SWIG_TypeEquiv(iter->types[i]->str, name)))
return iter->types[i];
@@ -630,10 +630,10 @@ SWIG_TypeQueryModule(swig_module_info *start,
SWIGRUNTIME char *
SWIG_PackData(char *c, void *ptr, size_t sz) {
static const char hex[17] = "0123456789abcdef";
- register const unsigned char *u = (unsigned char *) ptr;
- register const unsigned char *eu = u + sz;
+ const unsigned char *u = (unsigned char *) ptr;
+ const unsigned char *eu = u + sz;
for (; u != eu; ++u) {
- register unsigned char uu = *u;
+ unsigned char uu = *u;
*(c++) = hex[(uu & 0xf0) >> 4];
*(c++) = hex[uu & 0xf];
}
@@ -645,11 +645,11 @@ SWIG_PackData(char *c, void *ptr, size_t sz) {
*/
SWIGRUNTIME const char *
SWIG_UnpackData(const char *c, void *ptr, size_t sz) {
- register unsigned char *u = (unsigned char *) ptr;
- register const unsigned char *eu = u + sz;
+ unsigned char *u = (unsigned char *) ptr;
+ const unsigned char *eu = u + sz;
for (; u != eu; ++u) {
- register char d = *(c++);
- register unsigned char uu;
+ char d = *(c++);
+ unsigned char uu;
if ((d >= '0') && (d <= '9'))
uu = ((d - '0') << 4);
else if ((d >= 'a') && (d <= 'f'))
@@ -1326,7 +1326,7 @@ SWIG_Python_UnpackTuple(PyObject *args, const char *name, Py_ssize_t min, Py_ssi
}
if (!PyTuple_Check(args)) {
if (min <= 1 && max >= 1) {
- register int i;
+ int i;
objs[0] = args;
for (i = 1; i < max; ++i) {
objs[i] = 0;
@@ -1336,7 +1336,7 @@ SWIG_Python_UnpackTuple(PyObject *args, const char *name, Py_ssize_t min, Py_ssi
PyErr_SetString(PyExc_SystemError, "UnpackTuple() argument list is not a tuple");
return 0;
} else {
- register Py_ssize_t l = PyTuple_GET_SIZE(args);
+ Py_ssize_t l = PyTuple_GET_SIZE(args);
if (l < min) {
PyErr_Format(PyExc_TypeError, "%s expected %s%d arguments, got %d",
name, (min == max ? "" : "at least "), (int)min, (int)l);
@@ -1346,7 +1346,7 @@ SWIG_Python_UnpackTuple(PyObject *args, const char *name, Py_ssize_t min, Py_ssi
name, (min == max ? "" : "at most "), (int)max, (int)l);
return 0;
} else {
- register int i;
+ int i;
for (i = 0; i < l; ++i) {
objs[i] = PyTuple_GET_ITEM(args, i);
}
@@ -2461,7 +2461,7 @@ SWIG_Python_NewShadowInstance(SwigPyClientData *data, PyObject *swig_this)
}
} else {
#if PY_VERSION_HEX >= 0x03000000
- inst = PyBaseObject_Type.tp_new((PyTypeObject*) data->newargs, Py_None, Py_None);
+ inst = ((PyTypeObject*) data->newargs)->tp_new((PyTypeObject*) data->newargs, Py_None, Py_None);
if (inst) {
PyObject_SetAttr(inst, SWIG_This(), swig_this);
Py_TYPE(inst)->tp_flags &= ~Py_TPFLAGS_VALID_VERSION_TAG;
@@ -2962,7 +2962,7 @@ static swig_module_info swig_module = {swig_types, 3, 0, 0, 0, 0};
#endif
#define SWIG_name "_htmc"
-#define SWIGVERSION 0x020011
+#define SWIGVERSION 0x030002
#define SWIG_VERSION SWIGVERSION
@@ -4183,7 +4183,7 @@ static swig_const_info swig_const_table[] = {
* array with the correct data and linking the correct swig_cast_info
* structures together.
*
- * The generated swig_type_info structures are assigned staticly to an initial
+ * The generated swig_type_info structures are assigned statically to an initial
* array. We just loop through that array, and handle each type individually.
* First we lookup if this type has been already loaded, and if so, use the
* loaded structure instead of the generated one. Then we have to fill in the
@@ -4513,7 +4513,7 @@ extern "C" {
var = var->next;
}
if (res == NULL && !PyErr_Occurred()) {
- PyErr_SetString(PyExc_NameError,"Unknown C global variable");
+ PyErr_Format(PyExc_AttributeError, "Unknown C global variable '%s'", n);
}
return res;
}
@@ -4530,7 +4530,7 @@ extern "C" {
var = var->next;
}
if (res == 1 && !PyErr_Occurred()) {
- PyErr_SetString(PyExc_NameError,"Unknown C global variable");
+ PyErr_Format(PyExc_AttributeError, "Unknown C global variable '%s'", n);
}
return res;
}wow, the old swig tried to place data in registers by hand...
Hmm. I just tried running through valgrind and it said there were no debugging symbols. I added -g to the extra_compile_flags and now it no longer segfaults...
I also noticed that the -g flag is already in the compile command, so this shouldn't have done anything. I guess I'll close this until I can figure out how to debug more.
heisenbug
I'm going to try to borrow a mac to test, but might be a day or two
O.K. Thanks. Let me know what you find out if you do get your hands on a mac.
I've pushed a change to master that I think may fix this. Please give it a try.
Sweet! Sorry I bailed on you, but I'll give it a try ASAP.
I still see the same segfault, so I guess that wasn't the whole story.
The backtrace looks very similar to what we were seeing before.
Has there been any work on this, and is there anything I can do to help?
not yet, but I just got a mac donated to me that I might be able to use for testing this
I think I may have solved this while fixing another issue, and porting to python 3. Can you please give it a try with git master?
@esheldon I have not done extensive testing, but the failure mode I observed before does indeed seems to be gone. I can now call lookup_id without calling another method first. This is with a miniconda install using Python 2.7.
What was the fix?
One more question. I'd like to incorporate this new version. Are you going to make a new tag with this fix?
It was in the recent release. Also I'll be releasing a new version soon.