Passing binary files as 'scripts' causes errors at develop and install.
opened this issue · comments
Originally reported by: davidoff (Bitbucket: davidoff, GitHub: davidoff)
If you specify a binary file to the scripts= parameter of setup, the setup.py develop command copies the script with bytes 0D converted to 0A, garbling the binary data.
I have traced this to the text mode file opening in setuptools/command/develop.py:165 (introduced in c9e11fdff773).
If I change from open(script_path, 'rU') to open(script_path, 'rb') the binary files are copied correctly.
Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):
David, I'm not sure distutils/setuptools should necessarily support arbitrary binaries as 'scripts'. In the Python documentation, it defines scripts as "Scripts are files containing Python source code". Under this assumption, it's reasonable for setuptools to open these files as text.
I acknowledge that you've created pull request 60 to address the regression, but I'm reluctant to accept that pull request as it adds quite a bit more undocumented special casing, creating more untested, implicit requirements on setuptools.
Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):
What I'm failing to understand is why easy_install.install_egg_scripts doesn't encounter this same error, since it apparently uses dist.get_metadata to load the script as a binary blob.
Can you provide a reference to a project (preferably one with minimal dependencies and system requirements) that experiences this failure?
Original comment by davidoff (Bitbucket: davidoff, GitHub: davidoff):
As I understood the feature, and the usage we have in a normal install, is that scripts are copied to the python Scripts installation directory. On windows, this brings a fine way to include things in a package that can be bundled and used with the package, binary or not.
For example, we bundle a custom version of the python interpreter built with another build system (but it could be any other auxiliary binary support file). Let's call it myPython.exe (with it's runtime dependencies (dll etc)). The package to be installed is ourPythonDriver.py. Now when people install ourPythonDriver.py, they get also get myPython.exe installed, and it is available in the environment.
It is very practical that this feature works also with the develop command, and having a different semantics from the install and the develop command seems like a bug. Even if binary support is not explicitly mentioned in the docs, using binary files specified as scripts break only on windows because of the line ending translation caused by text mode opening. This happens only with the develop command.
Original comment by davidoff (Bitbucket: davidoff, GitHub: davidoff):
You can easily clone a sample project I created on GitHub to see the problem
https://github.com/davidovich/develop_breakage_on_windows
With pull request #60, this does not crash.
Without it, it crashes.
Even not on Windows, the package presented fails to build on Python 3 on Unix:
$ hg clone gh://davidovich/develop_breakage_on_windows
destination directory: develop_breakage_on_windows
importing git objects into hg
updating to branch default
5 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd develop_breakage_on_windows
$ python setup.py develop --user
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'pacakges'
warnings.warn(msg)
running develop
running egg_info
creating setuptools_develop_breaker.egg-info
writing setuptools_develop_breaker.egg-info/PKG-INFO
writing dependency_links to setuptools_develop_breaker.egg-info/dependency_links.txt
writing top-level names to setuptools_develop_breaker.egg-info/top_level.txt
writing manifest file 'setuptools_develop_breaker.egg-info/SOURCES.txt'
reading manifest file 'setuptools_develop_breaker.egg-info/SOURCES.txt'
writing manifest file 'setuptools_develop_breaker.egg-info/SOURCES.txt'
running build_ext
Creating /Users/jaraco/Library/Python/3.5/lib/python/site-packages/setuptools-develop-breaker.egg-link (link to .)
Adding setuptools-develop-breaker 0.0.0 to easy-install.pth file
Traceback (most recent call last):
File "setup.py", line 6, in <module>
scripts=['binary_hello_world.exe']
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/develop.py", line 34, in run
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/develop.py", line 133, in install_for_development
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/easy_install.py", line 697, in process_distribution
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/develop.py", line 168, in install_egg_scripts
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte
$ python setup.py sdist
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'pacakges'
warnings.warn(msg)
running sdist
running egg_info
writing dependency_links to setuptools_develop_breaker.egg-info/dependency_links.txt
writing top-level names to setuptools_develop_breaker.egg-info/top_level.txt
writing setuptools_develop_breaker.egg-info/PKG-INFO
reading manifest file 'setuptools_develop_breaker.egg-info/SOURCES.txt'
writing manifest file 'setuptools_develop_breaker.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt
running check
warning: check: missing required meta-data: url
warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied
creating setuptools_develop_breaker-0.0.0
creating setuptools_develop_breaker-0.0.0/setuptools_develop_breaker.egg-info
making hard links in setuptools_develop_breaker-0.0.0...
hard linking binary_hello_world.exe -> setuptools_develop_breaker-0.0.0
hard linking setup.py -> setuptools_develop_breaker-0.0.0
hard linking setuptools_develop_breaker.egg-info/PKG-INFO -> setuptools_develop_breaker-0.0.0/setuptools_develop_breaker.egg-info
hard linking setuptools_develop_breaker.egg-info/SOURCES.txt -> setuptools_develop_breaker-0.0.0/setuptools_develop_breaker.egg-info
hard linking setuptools_develop_breaker.egg-info/dependency_links.txt -> setuptools_develop_breaker-0.0.0/setuptools_develop_breaker.egg-info
hard linking setuptools_develop_breaker.egg-info/top_level.txt -> setuptools_develop_breaker-0.0.0/setuptools_develop_breaker.egg-info
Writing setuptools_develop_breaker-0.0.0/setup.cfg
creating dist
Creating tar archive
removing 'setuptools_develop_breaker-0.0.0' (and everything under it)
$ ls dist
setuptools_develop_breaker-0.0.0.tar.gz
$ python -m easy_install --user dist/setuptools_develop_breaker-0.0.0.tar.gz
Processing setuptools_develop_breaker-0.0.0.tar.gz
Writing /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-bffnbre1/setuptools_develop_breaker-0.0.0/setup.cfg
Running setuptools_develop_breaker-0.0.0/setup.py -q bdist_egg --dist-dir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-bffnbre1/setuptools_develop_breaker-0.0.0/egg-dist-tmp-ouau29u5
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'pacakges'
warnings.warn(msg)
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tokenize.py", line 392, in find_cookie
line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 154, in save_modules
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 195, in setup_context
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 243, in run_setup
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 273, in run
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 242, in runner
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 46, in _execfile
File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-bffnbre1/setuptools_develop_breaker-0.0.0/setup.py", line 6, in <module>
scripts=['binary_hello_world.exe']
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/bdist_egg.py", line 191, in run
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/bdist_egg.py", line 147, in call_command
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/install_scripts.py", line 20, in run
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/command/install_scripts.py", line 43, in run
self.run_command('build_scripts')
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/command/build_scripts.py", line 50, in run
self.copy_scripts()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/command/build_scripts.py", line 82, in copy_scripts
encoding, lines = tokenize.detect_encoding(f.readline)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tokenize.py", line 433, in detect_encoding
encoding = find_cookie(first)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tokenize.py", line 397, in find_cookie
raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for 'binary_hello_world.exe'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/easy_install.py", line 5, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/easy_install.py", line 2241, in main
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/easy_install.py", line 391, in run
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/easy_install.py", line 621, in easy_install
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/easy_install.py", line 670, in install_item
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/easy_install.py", line 850, in install_eggs
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/easy_install.py", line 1078, in build_and_install
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/easy_install.py", line 1064, in run_setup
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 246, in run_setup
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 195, in setup_context
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 166, in save_modules
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 141, in resume
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/pkg_resources/_vendor/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 154, in save_modules
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 195, in setup_context
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 243, in run_setup
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 273, in run
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 242, in runner
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/sandbox.py", line 46, in _execfile
File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-bffnbre1/setuptools_develop_breaker-0.0.0/setup.py", line 6, in <module>
scripts=['binary_hello_world.exe']
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/bdist_egg.py", line 191, in run
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/bdist_egg.py", line 147, in call_command
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-20.10.1-py3.5.egg/setuptools/command/install_scripts.py", line 20, in run
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/command/install_scripts.py", line 43, in run
self.run_command('build_scripts')
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/command/build_scripts.py", line 50, in run
self.copy_scripts()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/command/build_scripts.py", line 82, in copy_scripts
encoding, lines = tokenize.detect_encoding(f.readline)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tokenize.py", line 433, in detect_encoding
encoding = find_cookie(first)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tokenize.py", line 397, in find_cookie
raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for 'binary_hello_world.exe'
Notice that it fails to develop and it fails to install, and the failure in install occurs in distutils, reinforcing the idea that scripts, as the namespace and documentation indicates, is for scripts and not for binary executables. Given the traceback, I suspect that even the patch in the referenced pull request would not correct the issue for installation.
It still feels wrong to me to be hacking scripts to support something which they weren't intended to support, so here are a few options to consider:
- A new feature, an option that allows installing arbitrary binary files to the
binorScriptsdirectories. - Use the
data_filesto install the relevant directory (although beware of #130). - Store the binary files as package data and have a separate command that links or copies those files into the scripts directory following installation (a custom post-install step).
Accordingly, as a bug, I'm declaring this invalid, though I welcome follow-on tickets to implement one of the suggestions above or perhaps another enhancement to better support this need.
I hit this problem too - and implemented the 3rd suggestion by @jaraco (custom post-install script to copy binary over to the scripts directory).
The code is here in case anyone is interested: https://github.com/benfred/py-spy/blob/290584dde76834599d66d74b64165dfe9a357ef5/setup.py#L42
I'm using this to distribute a rust binary with pypi (so you can go pip install py-spy and get a precompiled rust binary installed).
I just ran into this while porting to Python 3 a setup.py that used scripts to install a compiled binary, and a simple solution was to use data_files instead. I.e. I added
data_files = [ ('bin', [<my binary>]) ]One important difference vs using scripts is that for data_files the permissions don't get set automatically. So, be sure to chmod a+rx on the <my binary> first!
One important difference vs using
scriptsis that fordata_filesthe permissions don't get set automatically. So, be sure tochmod a+rxon the<my binary>first!
Actually I just tried and it looks that now the permissions are been correctly kept.
@ntc2 Hi, when I use data_files to add some binary file to bin, it seems that the file is copied to a temporary directory like build/bdist.linux-x86_64/egg/bin rather than installation_prefix/bin. And I use the setup function from setuptools module not distutils.core module. How can I solve this problem? Thanks!
@DongqingSun96 sorry, I don't remember anything about how this works, and it sounds like it might have changed since my comment above.
@DongqingSun96 Please open a new issue, describing your issue - what commands you ran, what versions of Python and Setuptools you were using, what you encountered, and what you expected instead. In particular, if you can describe the problem in a way that the team can replicate, that would go a long way to getting more insight into the issue.