Pix4D / concourse-pypi-resource

A Concourse CI resource for Python PyPI packages.

PyPI Package Resource

A Concourse CI resource for Python PyPI packages.

It can be used to check/download existing packages and to manages your own builds as well. Internally it uses Pip 10.0.1 for check and in (downloads) and twine for output.

Docker image publicly available on Docker Hub: https://hub.docker.com/r/cfplatformeng/concourse-pypi-resource.

Resource Configuration

Parameter	default	status	description
PACKAGE SELECTION
name	-	required	name of the package
name_must_match	true	optional	require the project name and the packge name to match (see PEP-423)
pre_release	false	optional	check dev and pre-release versions (see PEP-440)
release	true	optional	check release versions
filename_match	-/-	optional	only include packages containing this string (e.g. py2.py3, .whl)
packaging	any	optional	only include source or binary (or any) packages
platform	-/-	optional	only include releases compatible with this platform (implicit default will be the os used for Concourse's workers)
python_abi	-/-	optional	TODO
python_implementation	-/-	optional	TODO
python_version	-/-	optional	only include packages compatible with this Python interpreter version number (see pip's --python-version)
REPOSITORY
repository.test	false	optional	set to true as shortcut to use the PyPI test server for index_url and repository_url
repository.index_url	PyPI	optional	url to a pip compatible index for check and download
repository.repository_url	PyPI	optional	url to a twine compatible repository for upload
repository.username	-/-	req. for uploads	username for PyPI server authentication
repository.password	-/-	req. for uploads	password for PyPI server authentication
repository.authenticate	out	optional	set to in to authenticate to a private repo for check and download only, always to authenticate to a private repository for upload, check and download.

Deprecated parameters (since version 0.2.0)

• repository: (special index-server name if it is specified in ~/.pypirc). This is no longer available to the current implementation of check and in. Also there's no way to inject a .pypirc file into this Concourse resource type.
• repository, test, username and password: get mapped to repository.<key>. This allows to configure private repositories through a single yaml-map parameter value, thus removing redundancy from the pipeline.

Examples

resource_types:
- name: pypi
  type: registry-image
  source:
    repository: cfplatformeng/concourse-pypi-resource

resources:
- name: my-public-package
  type: pypi
  source:
    name: my_package
    packaging: any
    repository:
      username: user
      password: pass

- name: my-private-package
  type: pypi
  source:
    name: my_package
    pre_release: true
    # In a live pipeline you would probably set the while repository configuration
    # through a single yaml parameter:
    # repository: ((my-private-repository))
    repository:
      authenticate: always
      index_url: http://nexus.local:8081/repository/pypi-private/simple
      repository_url: http://nexus.local:8081/repository/pypi-private/
      username: admin
      password: admin123

get: Download the latest version

• version.version: Optional, defaults to latest version
• count_retries: Optional Number of maximum retry before the task fails. By default 20 times.
• delay_between_retries: Optional Time to wait in sec between two iterations of retry. By default 3s.

Additional files populated

• version: Python version number of the downloaded package
• semver: Semver-formatted version number that can be processed with a Concourse SemVer Resource.

Example

plan:
- get: my-public-package
- get: my-private-package
  version: {version: '4.2'}

put: Upload a new version

• glob: Required A glob expression matching the package file to upload.

Note

You can modify count_retries and delay_between_retries in get_params to give enough time to PyPi to make available your package.

Example

plan:
- put: my-pypi-package
  params:
    glob: 'task-out-folder/my_package-*.tar.gz'
  get_params:
    count_retries: 10
    delay_between_retries: 30

Development

To run the unit tests, go to the root of the repository and run:

# install pipenv
pip3 install --user pipenv
# setup and run unittests
make test

To build the docker image for the resource:

# package
make dist
# optionally upload
make push

Run local test runs like this (nexus):

# check
docker run -i cfplatformeng/concourse-pypi-resource:latest-rc check < test/input/check-nexus.json | jq
# in
docker run --rm -i --volume $(pwd)/output:/destdir cfplatformeng/concourse-pypi-resource:latest-rc in /destdir < test/input/in-nexus.json | jq

or using public PyPi:

# check
docker run -i cfplatformeng/concourse-pypi-resource:latest-rc check < test/input/check-public.json | jq
# in
docker run --rm -i --volume $(pwd)/output:/destdir cfplatformeng/concourse-pypi-resource:latest-rc in /destdir < test/input/in-public.json | jq

Private repository integration tests (using Sonatype Nexus 3)

• Spin-up a docker instance of Nexus 3:

  docker run -d -p 8081:8081 --name nexus sonatype/nexus3

• Fetch the default admin password

  docker exec -it nexus cat /nexus-data/admin.password

• Use this password to log in for the first time as admin user
• During initial Setup wizard:
  • Change the password to admin123, as configured in test/nexus-integration.py.
  • Disable user anonymous access.
• Create pypi hosted repositories called pypi-private and pypi-release (enable redeploy).

# ensure test-packages are built
make test
pipenv run python -m pytest "test/nexus-integration.py"

Test using Concourse CI. Check that the provided params match your repository.

fly -t test sp -p demo -c test/ci/pipeline.yml -l test/ci/params.yml

Public Integration Tests

# provide login data to be able to upload concourse-pypi-resource
export TWINE_USERNAME=<...>
export TWINE_PASSWORD=<...>
python -m pytest -v test/pypi-integration.py 

# or skip the upload
python -m pytest -v -k 'not test_upload' test/pypi-integration.py