xxx.pyston.so vs xxx.so

[Sun]

Pyston set the SO to .pyston.so by:

build_time_vars = {
        ...
        "SO": ".pyston.so",
        ...
        }

https://github.com/dropbox/pyston/blob/master/cmake/_sysconfigdata.py.in#L10

---

In from_cpython/setup.py, Python try to find system libraries through self.compiler.find_library_file. For example:

do_readline = self.compiler.find_library_file(lib_dirs, 'readline')

in https://github.com/dropbox/pyston/blob/master/from_cpython/setup.py#L746

---

With the defintion in _sysconfigdata.py.in. It will try to find libreadline.pyston.so in lib_dirs. I can remove the .pyston prefix in setup.py by:

 from distutils.spawn import find_executable
+from _sysconfigdata import build_time_vars
+
+build_time_vars['SO'] = '.so'

But it cause the Python extension also build as xxx.so, not xxx.pyston.so:

$ ls build/Release-gcc/lib/python2.7/lib-dynload/
_ctypes.so  _ctypes_test.so  _curses.so

---

Without the .pyston notation. Pyston will report can not find xxx modules:

$ ./pyston_release_gcc
Pyston v0.6.0 (rev 60623a74691eaac685a1ec3fa5b0cb5063711703), targeting Python 2.7.7
>> import _curses
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named _curses

Why Pyston have to add the .pyston prefix to its extensions? Can we remove it? Or does there has a better solution?

@kmod @undingen .

[Marius Wachtler]

I think we moved to the .pyston.so extension to make sure that we don't load CAPI extensions which got compiled for cpython because they may load in pyston because export the same symbols but because our objects don't have the exact same layout it may crash at runtime and this errors may be hard to find.

But I'm not sure how to fix this problem. I guess there should normally be no CAPI extensions which dynamically links against another one (but I may be wrong) so we maybe able to just always change .pyston.so to .so inside ccompiler.py when searching for dynlibs.
@kmod I think you can give better advice on this.

[Kevin Modzelewski]

Yeah, we write out our shared libraries with slightly different extensions as a safety measure. The case I was worried about (loading a CPython extension by accident) isn't as bad as I thought it would be -- it just causes a link error, rather than getting to the point that it actually executes it. The link error is still quite annoying though because it's typically something obscure about Py_NoneStruct, and it's not always guaranteed to happen.

Anyway, this has caused issues in other areas as well, so I'm planning on changing this back to using the normal extension but having some other safety measure (probably just trying to guarantee the link error somehow).

Can you try this as a workaround? In from_cpython/Lib/distutils/unixccompiler.py, try changing shared_lib_extension = ".so" to ".pyston.so". At this point most of our stuff expects the "pyston.so" extension so it's probably easier to change this bit rather than everything else. At some point I'll get around to it and change everything back to just ".so".

[Sun]

@kmod Sorry, I don't understand. Why change shared_lib_extension = ".so" to ".pyston.so"? This just let compiler try to search shared libs which like xxx.pyston.so. And it still let self.compiler.find_library_file(lib_dirs, 'readline') try to find libreadline.pyston.so in system path.

For now, there have two requirements:

• let self.compiler.find_library_file(lib_dirs, 'readline') find libxxx.so in system path.
• let Pyston build extensions by from_cpython/setup.py which named as xxx.pyston.so.

[Kevin Modzelewski]

Oh sorry, I misunderstood the problem. Can we modify find_library_file to look for '.so'? Seems hacky for now but we'll do a larger fix later.

[Sun]

Yes, that was my workaround in my local code base. I thought it is a bit messy...

[Kevin Modzelewski]

Yeah it's not great, but I have this in my queue so it won't stay there for long. If you don't mind, it'd be great to have an assert like ext = ".pyston.so" right before setting ext = ".so" so that when we fix it we won't forget to change this spot.