All scripts in /opt/venvs/<packagname>/bin modified when building with github actions
thuitaw opened this issue · comments
Hi,
I started using github actions to build debian packages for flask applications. Previously we used to do this on jenkins.
The problem is, every time the package is built, all the scripts that are located in /opt/venvs//bin now contain the line below as a shebang
#!bin/sh
'''exec' /home/runner/work/<package-name>/<package-name>/debian/<package-name>/opt/venvs/<package-name>/bin/python "$0" "$@"'
eg.
Alembic now contains
the alembic script now looks like
#!/bin/sh
'''exec' /home/runner/work/<package-name>/<package-name>/debian/<packag-name>/opt/venvs/<package-name>/bin/python "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from alembic.config import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())
Where it previously looked like
#!/opt/venvs/<package-name>/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from alembic.config import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())
Basically, the shebang appended to all scripts packaged in /opt/venvs/package-name/bin now contain the default path where the repository is cloned when running any github workflow.
link to documentation about $GITHUB_WORKSPACE
Is there something i'm missing?
Update:
I think it has something to do with the following settings:
dh-virtualenv/dh_virtualenv/deployment.py
Line 56 in 9f4e66f
dh-virtualenv/dh_virtualenv/deployment.py
Line 62 in 9f4e66f
dh-virtualenv/dh_virtualenv/deployment.py
Line 65 in 9f4e66f
Which are used in the function here:
dh-virtualenv/dh_virtualenv/deployment.py
Line 220 in 9f4e66f
From the documentation it seems that DH_VIRTUALENV_INSTALL_ROOT is by default set to /opt/venvs, but for some reason this setting is not automatically set in github workflow environment
I've added a sample of the workflow file i use for this
name: Python package
on:
push:
branches: [ master ]
jobs:
Test:
runs-on: ubuntu-18.04
services:
postgres:
image: postgres:10.8
env:
POSTGRES_USER: db_user
POSTGRES_PASSWORD: db_pass
POSTGRES_DB: db
ports:
- 5432:5432
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 1
- name: Set up Python 3.6.10
uses: actions/setup-python@v1
with:
python-version: 3.6.10
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
python setup.py install
- name: Test with pytest
run: |
pytest tests
build:
runs-on: ubuntu-18.04
strategy:
max-parallel: 1
steps:
- name: Update packages
run: sudo apt-get update
- name: Install dh virtual env
run: sudo apt-get install dh-virtualenv
- name: Install git build package
run: sudo apt-get install git-buildpackage
- name: Install python dev packages
run: sudo apt-get install build-essential libssl-dev
libffi-dev python3-dev python3-venv git-buildpackage
dh-virtualenv python3-pip libpq-dev
debhelper python3-setuptools python3-distutils
- name: Install package cloud and dependencies
run: sudo gem install rake && sudo gem install package_cloud
- name: Checkout code and generate tags
uses: actions/checkout@v2
- run: git fetch --prune --unshallow
- name: Generate latest tag
run: |
$GITHUB_WORKSPACE/version.sh --patch --release
- name: Print out latest tag
run: |
echo ::set-env name=PROPOSED_TAG::$(cat $GITHUB_WORKSPACE/version.properties | grep snapshot.version | awk -F '=' '{print $2}' | awk '{split($0,output,"/"); print output[2]}')
- name: Generate debian changelog
run: gbp dch --id-length=6 --auto --ignore-branch --release -N $PROPOSED_TAG --commit --spawn-editor=never
- name: Build Package
run: gbp buildpackage --git-tag --git-ignore-branch --git-ignore-new -us -uc
- name: Push tags
run: git push origin --tags
- name: Push changelog
run: git push origin
- name: Set full path for package to upload
run: |
echo ::set-env name=PACKAGE_PATH::"${GITHUB_WORKSPACE}_${PROPOSED_TAG}_amd64.deb"
- name: Print path to package name
run: echo $PACKAGE_PATH
- name: Push debian package to packagecloud
run: |
package_cloud push ${REPOSTIORY}/ubuntu/bionic $PACKAGE_PATH
env:
PACKAGECLOUD_TOKEN: ${{secrets.PACKAGECLOUD_TOKEN}}
Explicit is better than implicit - DO set up the env in the workflow file.
@jhermann I have set the DH_VIRTUALENV_INSTALL_ROOT in the workflow environment as you advised, but the shebangs in all the scripts in /opt/venvs/{package-name} are still of the form
#!bin/sh
'''exec' /home/runner/work/<package-name>/<package-name>/debian/<package-name>/opt/venvs/<package-name>/bin/python "$0" "$@"'
name: Python package
on:
push:
branches: [ master ]
jobs:
Test:
runs-on: ubuntu-18.04
services:
postgres:
image: postgres:10.8
env:
POSTGRES_USER: db_user
POSTGRES_PASSWORD: db_pass
POSTGRES_DB:db
ports:
- 5432:5432
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 1
- name: Set up Python 3.6.10
uses: actions/setup-python@v1
with:
python-version: 3.6.10
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
python setup.py install
- name: Test with pytest
run: |
pytest tests
build:
runs-on: ubuntu-18.04
strategy:
max-parallel: 1
steps:
- name: Update packages
run: sudo apt-get update
- name: Install dh virtual env
run: sudo apt-get install dh-virtualenv
- name: Install git build package
run: sudo apt-get install git-buildpackage
- name: Install python dev packages
run: sudo apt-get install build-essential libssl-dev
libffi-dev python3-dev python3-venv git-buildpackage
dh-virtualenv python3-pip libpq-dev
debhelper python3-setuptools python3-distutils
- name: Install package cloud and dependecies
run: sudo gem install rake && sudo gem install package_cloud
- name: Checkout code and generate tags
uses: actions/checkout@v2
- run: git fetch --prune --unshallow
- name: Generate latest tag
run: |
$GITHUB_WORKSPACE/version.sh --patch --release
- name: Print out latest tag
run: |
echo ::set-env name=PROPOSED_TAG::$(cat $GITHUB_WORKSPACE/version.properties | grep snapshot.version | awk -F '=' '{print $2}' | awk '{split($0,output,"/"); print output[2]}')
- name: Generate debian changelog
run: gbp dch --id-length=6 --auto --ignore-branch --release -N $PROPOSED_TAG --commit --spawn-editor=never
- name: Build Package
run: gbp buildpackage --git-tag --git-ignore-branch --git-ignore-new -us -uc
env:
DH_VIRTUALENV_INSTALL_ROOT: /opt/venvs/
- name: Push tags
run: git push origin --tags
- name: Push changelog
run: git push origin
- name: Set full path for package to upload
run: |
echo ::set-env name=PACKAGE_PATH::"${GITHUB_WORKSPACE}_${PROPOSED_TAG}_amd64.deb"
- name: Print path to package name
run: echo $PACKAGE_PATH
- name: Push debian package to packagecloud
run: |
package_cloud push eneza/debian/ubuntu/bionic $PACKAGE_PATH
env:
PACKAGECLOUD_TOKEN: ${{secrets.PACKAGECLOUD_TOKEN}}
@jhermann @nailor I finally got this working by using a horribly ancient version of pip(version 9.03)
My make file now looks like this
#!/usr/bin/make -f
# This is a makefile. Indentations are tab separated!
# Uncomment this to turn on verbose mode.
export DH_VERBOSE=1
export DH_VIRTUALENV_INSTALL_ROOT=/opt/venvs
%:
dh $@ --with python-virtualenv --python /usr/bin/python3
# Uncomment the --pypi-url section if you are using a local pypi caching server
# See http://doc.devpi.net/latest/
#
# Comment flake8 at your own peril
override_dh_virtualenv:
dh_virtualenv --python /usr/bin/python3 --setuptools --preinstall pip==9.0.3 --preinstall=no-manylinux1
# Make sure that we clean out build artefacts from a previous build
override_dh_clean:
dh_clean
find . \( -name '*.py[co]' -or -name '\._*' -or -name '\.DS_Store' \) -delete
find . \( -name '*.egg-info' -or -name 'build' \) -print0 | xargs -0 rm -rf
~
I got the idea to do this from this closed issue: #223
I would really like to close this issue, but If any of you could offer any insight as to why this only works with that version of pip, I'd be grateful
Well, it could be a managed transitive dependency, esp. setuptools since that writes the bangpaths in the end. I always manage setuptools and wheel versions, not just pip.
@thuitaw And the concrete results / changes were...?
@jhermann The only thing that worked was using that old version of PIP...pinning setuptools still resulted in the new style shebang being prepended to all scripts in /opt/venvs/bin
I'm hitting this too and I'm pinning to the ancient pip version isn't helping. This should probably be re-opened as the fix isn't very satisfactory or general.
We hit what looks like the same issue. For me I traced this down to a package name with more than 18 characters in.
@chrispbailey How did you fix this? Was it by shortening the package name?
I have run into an issue where a dependency in my requirements.txt requires a minimum version of pip, and I'm back to this same problem
Without having researched it, the likely culprit is a fix in pip or one of its libs, to move bang paths that are too long for bash to a normal exec call. In turn, the rewrite of bang paths in dh-venv fails to recognize these modified forms and (incorrectly) leaves them as-is.
@chrispbailey I fixed this issue by shortening the package name. Involved renaming the entire project and reconfiguring my imports. Thanks for the tip.
Not sure this is of use to anyone else, but I managed to avoid the shebang rewrite by copying all files needed for building the debian package to the same path I would want the package be deployed to eventually (/usr/share in my case).
I did not pin any pip or setoptools versions.
After that I no longer see these type of shebangs,
#!bin/sh
'''exec' /home/runner/work/<package-name>/<package-name>/debian/<package-name>/usr/share/venvs/<package-name>/bin/python "$0" "$@"'
The scripts (bin/pip, bin/gunicorn, etc.) show the correct shebang for me
#!/usr/share/<package-name>/bin/python
This does however get rid of the venv/ part in the path. Might need a bit more tweaking if you need the path structure mentioned above(/opt/venvs/<package-name>/bin/python)
Tested on GH Actions with Ubuntu 18.04, Python 3.7 and dh_virtualenv 1.0