This project is a simple example on how to use GH actions and poetry to
implement Continuos Integration and Continuos Delivery for a python project
using poetry and pytests
The master branch is protect and can only be changed via PRs. Each PR has a ci
workflow with linting and tests. After a merge on master, another workflow
is responsible to build the package. There is also Publish step that should
push the package to a private repository following
poetry docs
The architecture follows a simple structure with two workflows:
- The
ciworkflow with automation to runpre-commitfor linting and dependency management checks. Also test automation is included with test coverage report. This is executed on each PR tomasterand must succeed to the PR to receive green light to be merged - The
cdworkflow, which has steps to build the python package and publish it to a private package repository if the credentials are configured properly. This is executed after every merge onmaster. Since the tests on every change have been executed already from theciworkflow, we can skip to the deployment actions
Both workflows reuse a setup action to configure the machine with the required dependencies. This setup
also caches the python virtualenv when nothing has changed in the poetry.lock file, making subsequent
workflow executions much faster.
Each workflow has only one job with the same name. This is because every desired action must run sequentially and by using just one job, we can continue to reuse the same runner reusing the setup step, reducing some execution time.
Separation of jobs make sense when you want to run things in parallel (its default behavior). A common case is to run tests with several machine configurations, but since this is not the case of this project, one job is sufficient.
# Install poetry
curl -sSL https://install.python-poetry.org | python3 -
export PATH="${HOME}/.local/bin:$PATH"
# Install dependencies
poetry shell
poetry install
# pre-commit setup
pre-commit installđź’ˇ Tip: you can run
pre-commit run --all-filesto catch errors before committing
- Lint:
black *.py python_cicd/ - Test:
pytest - Build:
poetry build
Linting is already being done with black with pre-commit and is the recommended way to use it
locally or via GH workflows. The gh-action
from black docs could be used in the ci workflow, but pre-commit was used to streamline the process.
We can use act to test GH actions locally.
# Install act and GH cli
brew install act
brew install gh
# Authenticate with gh
gh auth login
# Test push-based workflows
act push --container-architecture linux/amd64 \
-s REPO_USERNAME="${REPO_USERNAME}" \
-s REPO_PASSWORD="${REPO_PASSWORD}" \
-s REPO_ENDPOINT="${REPO_ENDPOINT}"
# Test pull_request-test workflows (there is a known issue with the `Comment coverage` step)
act pull_request --container-architecture linux/amd64 \
-s GITHUB_TOKEN=$(gh auth status -t 2>&1 | grep Token | awk '{print $3}') \
-P ubuntu-latest=nektos/act-environments-ubuntu:18.04 # See https://github.com/dorny/paths-filter#notesNote: There is a known issues when using
act. With theciworkflow, the second step is to fetch the master branch from the remote. This works fine on GH, but locally it fails because the GH managed runners are not trusted
Poetry has documentation on how to configure and publish on private repositories
This repo has a step on the cd workflow that tries to configure access and publish to a private repository like
Sonatype Nexus
(assuming that a PyPI repository named pypi-internal has been create already). After configuring
the REPO_USERNAME and REPO_PASSWORD secrets in the repo, the Publish should execute on merges to master.
- workflow-builder: Great tips for GH workflows design
- Poetry docs
- Sonatype Nexus docs: As an example of PyPI repository