Python Package with Dependencies
Hujun opened this issue · comments
As Python developers, we usually build Python package to share our work (devpi for internal & private scope and PyPI for global opensource work). Normally basic usage of setup.py is enough for these works. But there is still some specific scenarios which need more sophisticated packaging process. For example, in specific CI/CD workflow, you don't want take time to download package dependencies from PyPI (or devpi) every time when build the docker image. So how to do this?
Create wheel house
The first thing to do is to make all dependencies local packages (wheel). Pip has this function as:
pip wheel -r requirements.txt --wheel-dir <path>
The command will download all dependencies defined in requirements.txt from PyPI as usual, and make them as particular wheel package in the directory specified.
To simplify the operation, you can put it in setup.py command. The following snippet demonstrates the a package procedure with dependencies wheel making.
class PackageCommand(Command):
"""Support setup.py package"""
description = 'Build package.'
user_options = [
('with-deps', None, 'package with all wheel packages of dependencies'),
]
def initialize_options(self):
"""Set default values for options"""
self.with_deps=False
def finalize_options(self):
"""Post-process options"""
if self.with_deps:
self.with_deps=True
def run(self):
"""Run command"""
clear_files = [
os.path.join(pwd_path, 'build'),
os.path.join(pwd_path, 'dist'),
os.path.join(pwd_path, '*/*.egg-info'),
os.path.join(pwd_path, 'Youtiao.egg-info'),
]
for cf in clear_files:
print('rm {}'.format(cf))
subprocess.run(['rm', '-rf', cf])
# make sure that wheel is installed
subprocess.run(['python', 'setup.py', 'bdist', 'bdist_wheel', '--universal'])
if self.with_deps:
rqm_path = os.path.join(pwd_path, 'requirements.txt')
wheels_path = os.path.join(pwd_path, 'wheels')
subprocess.run(['rm', '-rf', wheels_path])
subprocess.run(['mkdir', '-p', wheels_path])
subprocess.run('pip wheel --wheel-dir={} -r {}'.format(wheels_path, rqm_path), shell=True)
sys.exit(0)
Put it in your setup cmdclass
of arguments.
Make docker image without dependency download
Normally we run simple pip install -r requirements.txt
in docker build script. It needs some modification if you want to avoid download from PyPI. A simple Dockerfile example as:
FROM python:3.6
COPY . /app
WORKDIR /app
# install wheel
# make sure that your project wheel under /app/dist
# and wheels of dependencies under /app/wheels
RUN pip install /app/dist/<project_wheel> --no-index --find-links /app/wheels
ENTRYPOINT ["python"]
CMD ["start_point_script.py"]
Voila