unable to run bootrap file using py2app and python3.4
JPFrancoia opened this issue · comments
Hi again,
I can successfully run a simple hello world program using py2app, esky and python 2, python 3 however fails..
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
Running the second-level exe (not the boostrap one) works, the program runs smoothly. The issue seems to be related to #72.
I assumed python could not find some libraries, so I copied everything from the second level folder Contents into the first level folder Contents.
MyProgram.app/
Info.plist
Contents/
MacOS/
exe_esky
MyProgram-mac-os-x86_64/
MyProgram.app/
Contents/
Now when I run exe_esky, I get this:
Failed to import the site module
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1129, in _exec
File "<frozen importlib._bootstrap>", line 1467, in exec_module
File "<frozen importlib._bootstrap>", line 1678, in get_code
File "<frozen importlib._bootstrap>", line 656, in _compile_bytecode
ValueError: bad marshal data (unknown type code)
@rfk , any idea ? We are kind of stuck here :)
@JPFrancoia easiest might be to link or email me a tarball of the frozen app itself, that way I can poke around in its internals and see if I can see what's going on. I'll also try to repro locally.
(Wow, it's been a while since I actually tried to run this stuff myself)
With python3.4.3, py2app 0.9 I get the following error when trying to freeze an app:
AttributeError: 'ModuleGraph' object has no attribute 'scan_code'
Have you seen this before? Suggestion for a workaround?
Nope, but there is a fix here I guess:
http://stackoverflow.com/questions/25394320/py2app-modulegraph-missing-scan-code
I'll email you that right away.
No luck on tracking down the problem with encodings module, but I suspect this subsequent error:
Failed to import the site module
...
ValueError: bad marshal data (unknown type code)
May be because of the way we're writing bytecode in the py2app freezer here:
https://github.com/cloudmatrix/esky/blob/master/esky/bdist_esky/f_py2app.py#L138
Changing this to use the new esky.util.compile_to_bytecode function rather than calling marshal directly, may help things work after copying more resources into the top-level folder.
The encodings modules are in the zipfile at:
YourApp.app/YourApp-0.9.1.macosx-10_6-intel/YourApp.app/Contents/Resources/lib/python35.zip
Does copying this into the equivalent location in the top-level Contents directory help move things forward?
I had to copy both
YourApp.app/YourApp-0.9.1.macosx-10_6-intel/YourApp.app/Contents/Resources/lib/python35.zip
and
YourApp.app/YourApp-0.9.1.macosx-10_6-intel/YourApp.app/Contents/Resources/lib/python35/
Into the he equivalent location in the top-level Contents directory, otherwise I get this error:
Fatal Python error: Py_Initialize: unable to load the file system codec
zipimport.ZipImportError: can't decompress data; zlib not available
Current thread 0x00007fff7cd22000 (most recent call first):
When I do so, I go back to the error:
Failed to import the site module
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 658, in exec_module
File "<frozen importlib._bootstrap_external>", line 870, in get_code
File "<frozen importlib._bootstrap_external>", line 473, in _compile_bytecode
ValueError: bad marshal data (unknown type code)
I tried to change the line:
f.write(marshal.dumps(compile("", "site.py", "exec")))to
f.write(esky.util.compile_to_bytecode("", "site.py"))But I still get the bad marshal data error.
Did I made the proper modifications ? Do I have to make more ?
I tried to change the line:
f.write(marshal.dumps(compile("", "site.py", "exec")))
It should replace both the f.write(imp.get_magic()) and f.write(marshal.dumps) lines.
Ok. I made the changes, but got:
Traceback (most recent call last):
File "/Users/chembrows/Desktop/myprogram.app/dist/myrpogram.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 695, in <module>
sys.executable = environ["EXECUTABLEPATH"]
KeyError: 'EXECUTABLEPATH'
Hmm, I guess the behaviour of py2app must have changed in newer versions, either of it or of python3. To debug, I suggest replacing the _EXTRA_BOOTSTRAP_CODE section in f_py2app.py with the following:
_EXTRA_BOOTSTRAP_CODE = """
from posix import environ
print(list(environ.items()))
try:
sys.executable = environ["EXECUTABLEPATH"]
sys.argv[0] = environ["ARGVZERO"]
except KeyError:
pass
"""
Which will hopefully (1) allow the bootstrapping exe to continue to load the real frozen app, and (2) tell us what's in environ so we might re-create the old behaviour.
Here we go:
Traceback (most recent call last):
[(b'PYTHONHOME', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources'), (b'TERM', b'xterm-256color'), (b'SHELL', b'/bin/bash'), (b'XPC_SERVICE_NAME', b'0'), (b'HISTFILESIZE', b'10000'), (b'LANG', b'fr_FR.UTF-8'), (b'LD_LIBRARY_PATH', b'/usr/lib/jvm/java-7-openjdk/jre/lib/amd64/'), (b'HOME', b'/Users/chembrows'), (b'_', b'./gui'), (b'ARGVZERO', b'./gui'), (b'PATH', b'/Library/Frameworks/Python.framework/Versions/3.5/bin:/opt/local/bin:/opt/local/sbin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Users/chembrows/Gabedit64:/Users/chembrows/Gabedit64:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/home/djipey/scripts_perso'), (b'DISPLAY', b'/private/tmp/com.apple.launchd.2p2Rzg1yiw/org.macosforge.xquartz:0'), (b'TERM_PROGRAM_VERSION', b'361'), (b'DBUS_LAUNCHD_SESSION_BUS_SOCKET', b'/private/tmp/com.apple.launchd.XBAOC1iS5t/unix_domain_listener'), (b'LOGNAME', b'chembrows'), (b'TERM_PROGRAM', b'Apple_Terminal'), (b'Apple_PubSub_Socket_Render', b'/private/tmp/com.apple.launchd.fzy01S7PKa/Render'), (b'HISTSIZE', b'10000'), (b'SECURITYSESSIONID', b'186dc'), (b'XPC_FLAGS', b'0x0'), (b'USER', b'chembrows'), (b'MANPAGER', b'vimpager'), (b'PWD', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/MacOS'), (b'SHLVL', b'1'), (b'HISTTIMEFORMAT', b'[ %d/%m/%Y %H:%M:%S ]'), (b'SSH_AUTH_SOCK', b'/private/tmp/com.apple.launchd.Vp8BNWInk0/Listeners'), (b'EDITOR', b'/usr/bin/vim'), (b'RESOURCEPATH', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources'), (b'__CF_USER_TEXT_ENCODING', b'0x1F7:0x0:0x1'), (b'OLDPWD', b'/Users/chembrows/Desktop/myprogram'), (b'TERM_SESSION_ID', b'20B3A39C-D95D-437E-9694-6CB85BA27C49'), (b'TMPDIR', b'/var/folders/sv/3vhc3x596wq8vfyhlx21qz9m0000gq/T/'), (b'HISTCONTROL', b'ignoreboth'), (b'PYTHONPATH', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources'), (b'PYTHONDONTWRITEBYTECODE', b'1'), (b'EXECUTABLEPATH', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/MacOS/./gui'), (b'LC_CTYPE', b'en_US.UTF-8')]
File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 704, in <module>
bootstrap()
File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 335, in bootstrap
return chainload(pathjoin(vsdir,best_version))
File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 362, in chainload
_chainload(target_dir)
File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 412, in _chainload
execv(target_exe,[target_exe] + sys.argv[1:])
PermissionError: [Errno 13] Permission denied
2015-12-03 22:36:21.209 gui[7668:375362] myprogram Error
Interesting, is the matching gui executable in the frozen app marked as executable? Otherwise, I wonder if there's some new security policy in OSX that's blocking this from being executed...
b'EXECUTABLEPATH'
Aha, they values here are bytes rather than strings, that's probably what caused the KeyError.
Yep, the boostrap exe and the real exe are both marked as executable.
Yes they are bytes. Shall we convert the values before we use them ?
Also, is
/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/MacOS/./gui
A correct path ? Shouldn't it be
/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/MacOS/gui
The extra . in the path shouldn't matter IIUC.
Yes they are bytes. Shall we convert the values before we use them
No, I think it's OK to use them as bytes, but the EXTRA_BOOTSTRAP_CODE for py3 probably needs to be updated to use bytes for keys, like:
_EXTRA_BOOTSTRAP_CODE = """
from posix import environ
sys.executable = environ[b"EXECUTABLEPATH"]
...etc...
I changed the extra_bootstrap_code as you required:
Traceback (most recent call last):
Dec 3 23:03:02 gui[8405] <Notice>: File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 704, in <module>
Dec 3 23:03:02 gui[8405] <Notice>: bootstrap()
Dec 3 23:03:02 gui[8405] <Notice>: File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 310, in bootstrap
Dec 3 23:03:02 gui[8405] <Notice>: sys.executable = abspath(sys.executable)
Dec 3 23:03:02 gui[8405] <Notice>: File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 77, in abspath
Dec 3 23:03:02 gui[8405] <Notice>: path = pathjoin(getcwd(),path)
Dec 3 23:03:02 gui[8405] <Notice>: File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 254, in pathjoin
Dec 3 23:03:02 gui[8405] <Notice>: if isabs(arg):
Dec 3 23:03:02 gui[8405] <Notice>: File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 75, in isabs
Dec 3 23:03:02 gui[8405] <Notice>: return (path.startswith(SEP))
Dec 3 23:03:02 gui[8405] <Notice>: TypeError: startswith first arg must be bytes or a tuple of bytes, not str
Ugh, bytes. @JPFrancoia try this one:
_EXTRA_BOOTSTRAP_CODE = """
from posix import environ
def _decode(b):
return b.decode(sys.getfilesystemencoding())
sys.executable = _decode(environ[b"EXECUTABLEPATH"])
sys.argv[0] = _decode(environ[b"ARGVZERO"])
"""
Sorry, I haven't managed to find the time to get py2app up and running and make a fresh build myself yet, so I can't try it out myself.
Woohoo, with all of:
- Copying the files from
libinto the top-level app resources - Changing the byte-compiled representation of site.pyc
- The above modifications of the bootstrap code
I can successfully run the top-level script for the compiled app that you sent me! Hopefully it will also work for you. If so, we can work backwards to figure out what to change in esky itself to make this all work by default.
Paring lib-dynload down to the following still works:
$ ls ./ChemBrows.app/Contents/Resources/lib/python3.5/lib-dynload/
fcntl.so zlib.so
Paring the contents of python35.zip down to the following still works:
$ unzip ../python35.zip > /dev/null
$ ls
_weakrefset.pyc abc.pyc codecs.pyc encodings io.pyc
AFAICT this is the minimal set of modules required for python3 to start up successfully. So if this works for you @JPFrancoia, the fixes we need to make are:
- The change to use
compile_to_bytecode - The change to use bytes in the
EXTRA_BOOTSTRAP_CODE - Extend the code in
f_py2appthat currently copiesfcntl.sointo the bootstrap env, so that it also copieszlib.so, and a minimalpythonXY.zipcontaining the above modules required for startup
WOUHOUHOU :) It also works for me guys.
For now, I did:
- Copying the files from lib into the top-level app resources
- Changing the byte-compiled representation of site.pyc
- The above modifications of the bootstrap code
as @rfk suggested.
Now we should make everything look good, and everything is in the pocket :)
@timeyyy , I can' find your email address, can you PM it to me ?
@JPFrancoia thanks so much for sticking with this to work through the issue; I think the discoveries here will also help unblock #131 and related issues.
I'll try to patch Esky with what you guys suggested. @rfk , if I set the content of python35.zip down to the following:
_weakrefset.pyc abc.pyc codecs.pyc encodings io.pyc
I get the
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
error. Did you do something else to your zip file ? (I also set the the contents of lib-dynload to what you said).
Also, why copying to the bootstrap env ? Wouldn't a link work (something like ln -s) ?
Did you include all the contents of the "encodings" directory in the zipfile as well?
Yes, here is what I have in the top level Resources folder (I unzipped the python35 archive):
PythonApplet.icns __boot__.py __error__.sh include lib site.pyc
./include:
python3.5m
./include/python3.5m:
pyconfig.h
./lib:
python3.5 python35 python35.zip
./lib/python3.5:
lib-dynload site.pyc
./lib/python3.5/lib-dynload:
fcntl.so zlib.so
./lib/python35:
_weakrefset.pyc abc.pyc codecs.pyc encodings io.pyc
./lib/python35/encodings:
__init__.pyc cp1255.pyc cp860.pyc gb2312.pyc iso8859_15.pyc mac_croatian.pyc shift_jisx0213.pyc
aliases.pyc cp1256.pyc cp861.pyc gbk.pyc iso8859_16.pyc mac_cyrillic.pyc tis_620.pyc
ascii.pyc cp1257.pyc cp862.pyc hex_codec.pyc iso8859_2.pyc mac_farsi.pyc undefined.pyc
base64_codec.pyc cp1258.pyc cp863.pyc hp_roman8.pyc iso8859_3.pyc mac_greek.pyc unicode_escape.pyc
big5.pyc cp273.pyc cp864.pyc hz.pyc iso8859_4.pyc mac_iceland.pyc unicode_internal.pyc
big5hkscs.pyc cp424.pyc cp865.pyc idna.pyc iso8859_5.pyc mac_latin2.pyc utf_16.pyc
bz2_codec.pyc cp437.pyc cp866.pyc iso2022_jp.pyc iso8859_6.pyc mac_roman.pyc utf_16_be.pyc
charmap.pyc cp500.pyc cp869.pyc iso2022_jp_1.pyc iso8859_7.pyc mac_romanian.pyc utf_16_le.pyc
cp037.pyc cp65001.pyc cp874.pyc iso2022_jp_2.pyc iso8859_8.pyc mac_turkish.pyc utf_32.pyc
cp1006.pyc cp720.pyc cp875.pyc iso2022_jp_2004.pyc iso8859_9.pyc mbcs.pyc utf_32_be.pyc
cp1026.pyc cp737.pyc cp932.pyc iso2022_jp_3.pyc johab.pyc palmos.pyc utf_32_le.pyc
cp1125.pyc cp775.pyc cp949.pyc iso2022_jp_ext.pyc koi8_r.pyc ptcp154.pyc utf_7.pyc
cp1140.pyc cp850.pyc cp950.pyc iso2022_kr.pyc koi8_t.pyc punycode.pyc utf_8.pyc
cp1250.pyc cp852.pyc euc_jis_2004.pyc iso8859_1.pyc koi8_u.pyc quopri_codec.pyc utf_8_sig.pyc
cp1251.pyc cp855.pyc euc_jisx0213.pyc iso8859_10.pyc kz1048.pyc raw_unicode_escape.pyc uu_codec.pyc
cp1252.pyc cp856.pyc euc_jp.pyc iso8859_11.pyc latin_1.pyc rot_13.pyc zlib_codec.pyc
cp1253.pyc cp857.pyc euc_kr.pyc iso8859_13.pyc mac_arabic.pyc shift_jis.pyc
cp1254.pyc cp858.pyc gb18030.pyc iso8859_14.pyc mac_centeuro.pyc shift_jis_2004.pyc