With OpenShift 4, container image signatures can be verified at deploy time by configuring CRI-O to use GPG keys. This repo contains a walkthrough/demo of this feature and a simple solution to signature storage. This demo will use:
- Openshift 4 as the container platform
- An external Sonatype Nexus instance to simulate a private docker registry
- GPG to sign/verify container images
- NGINX + LuaJIT as the application framework (OpenResty)
Signatures can also be stored inside Nexus, so if this is the intended workflow, the NGINX signature server is not required. Also, gpg signature verification is enabled for official RedHat repositories.
- Install Openshift 4.x
- Generate a GPG Key that will be used to sign images
# gpg --quick-gen-key demo@redhat.com
# gpg -k
[...]
pub rsa2048 2020-04-24 [SC] [expires: 2022-04-24]
01164344435F9572F7B8B06D48790DBE02151245
uid [ultimate] demo@redhat.com
sub rsa2048 2020-04-24 [E]
[...]
- Export the public key to file
# gpg --armor --export demo@redhat.com > nexus-key.gpg
- Generate another GPG Key that will be used to simulate a wrongly signed image
# gpg --quick-gen-key wrong@redhat.com
- Deploy an instance of Nexus
# oc create -f components/nexus-deployment.yaml
- On the nexus web interface, create a new Hosted Docker repository.
- If you are using a self-signed certificate on the ingress controllers, the local nexus needs to be added to the list of insecure registries:
# oc edit image.config.openshift.io/cluster
apiVersion: config.openshift.io/v1
kind: Image
metadata:
annotations:
release.openshift.io/create-only: "true"
creationTimestamp: 2020-05-05T05:21:57Z
generation: 2
name: cluster
resourceVersion: "80728"
selfLink: /apis/config.openshift.io/v1/images/cluster
uid: 5576bd84-9e0c-4d67-9498-9c8cf523cbd2
spec:
registrySources:
insecureRegistries:
- nexus-registry.apps.<ingress domain associated with the openshift cluster>
status:
internalRegistryHostname: image-registry.openshift-image-registry.svc:5000
The MachineConfigOperator monitors that resource for differences and applies the new config when appropriate.
- If the image registry created on nexus needs authentication, a pull secret needs to be created and linked to the correct ServiceAccount
# oc new-project signature-demo
# oc create secret docker-registry nexus-pull-secret --docker-server=nexus-registry.apps.<ingress domain associated with the openshift cluster> --docker-username=<username> --docker-password=<password> --docker-email=unused
For example, if the 'demo-sa' is used to deploy pods with a deploymentConfig, this pull secret needs to be linked to that SA:
# oc create sa demo-sa
# oc secrets link demo-sa nexus-pull-secret --for=pull
To assign the pull secret to the 'default' service account (the SA that is used when no other is specified):
# oc secrets link default nexus-pull-secret --for=pull
This demo uses a local instance of Nexus as an external image repository. We want images coming from that repo to be signed and verified. Worker (and masters optionally) nodes in an OCP cluster need to be made aware of a new repo that requires signature verification.
The policy.json file will contain all repositories that need signature verification. For example, the resulting policy.json file will look like this with the custom 'nexus-registry.apps.<ingress domain associated with the openshift cluster>' repository added in:
{
"default": [
{
"type": "insecureAcceptAnything"
}
],
"transports": {
"docker": {
"registry.access.redhat.com": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
}
],
"registry.redhat.io": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
}
],
"nexus-registry.apps.<ingress domain associated with the openshift cluster>": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/nexus-key.gpg"
}
]
},
"docker-daemon": {
"": [
{
"type": "insecureAcceptAnything"
}
]
}
}
}
Configuration files are automatically rendered with the provided 'gen-machineconfig.sh' script.
- Create a configuration file for every repo and fill in the address of the HTTP server that will host the signatures:
docker:
nexus-registry.apps.<ingress domain associated with the openshift cluster>:
sigstore: https://signature.apps.<ingress domain associated with the openshift cluster>/sigstore
Create a file like this for all custom/official repositories enumerated in the policy.json file and that need GPG signature verification.
- Generate the MachineConfig manifests with the provided script (under machineconfig/)
# ./gen-machineconfig.sh -k /path/to/nexus-key.gpg -r /path/to/nexus-registry.apps.<ingress domain associated with the openshift cluster>.yaml
This will create two MachineConfig manifest files under the ./rendered/ folder:
# oc create -f 02-master-rh-registry-trust.yaml
# oc create -f 02-worker-rh-registry-trust.yaml
After a while both configuration will be applied to the cluster.
# oc get machineconfigpool
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-36f5d702f485cde72df754013e17937f True False False 3 3 3 0 4d5h
worker rendered-worker-ec7bab1743d5d2a88bed9cf1280ff9f1 True False False 3 3 3 0 4d5h
Container images signatures are served by a simple HTTP server (nginx) with a couple service APIs baked in.
- Create a new project on OCP and set up the correct SCC for the sigserver service account
# oc new-project signature-server
# oc adm policy add-scc-to-user anyuid system:serviceaccount:signature-server:signature-sa
- Create the virtual host config map:
# oc create configmap nginx-sigstore-vhost --from-file=nginx/sigstore.conf
- Create the API configmap
# oc create configmap lua-api-sources --from-file=api/context_body.lua --from-file=api/signature_upload.lua --from-file=api/filesystem.lua
- Deploy the signature server
# oc create -f components/signature-server-deployment.yaml
This test makes use of three different small container images, to demonstrate these three use cases:
- A Correctly Signed image (signed with the approved and configured GPG key)
- An Image that has no signature
- An Images that has been signed with an unknown/wrong GPG key
Skopeo needs to be configured to store signatures in a known path, so that these can later be uploaded to a signature store:
[...]
# This is the default signature write location for docker registries.
default-docker:
# sigstore: file:///var/lib/containers/sigstore
sigstore-staging: file:///tmp/sigstore
[...]
the 'sigstore-staging' parameter is used by skopeo. After a successful sign operation, the signature is stored under that path:
# tree /tmp/sigstore
/tmp/sigstore/
└── docker
└── busybox@sha256=a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6
└── signature-1
- Upload an image without signature to nexus
# skopeo copy --dest-creds=<username>:<password> docker://docker.io/library/alpine:latest docker://nexus-registry.apps.<ingress domain associated with the openshift cluster>/docker/alpine:unsigned
- Upload an image signed with the wrong key to nexus
# skopeo copy --dest-creds=<username>:<password> --sign-by wrong@email.com docker://docker.io/library/busybox:latest docker://nexus-registry.apps.<ingress domain associated with the openshift cluster>/docker/busybox:wrongsig
- Upload an image signed with the correct gpg key to nexus
# skopeo copy --dest-creds=<username>:<password> --sign-by demo@redhat.com docker://docker.io/library/centos:latest docker://nexus-registry.apps.<ingress domain associated with the openshift cluster>/docker/centos:signed
After that, in this third case, the image signature needs to be uploaded to the signature server.
Uploading signature is achieved by calling the /upload API endpoint served by the signature server. All parameters need to be base64-encoded. There is only one POST method implemented and that accepts a JSON payload:
{
"repoName": "base64-encoded name of the repo on the remote docker registry",
"layerId": "base64-encoded sha digest of the signed container layer",
"signatureData": "base64-encoded signature of the image layer"
}
An helper script is provided under jenkins-agents/signer-agent/scripts:
# ./clients/signature-upload.py -r https://signature.apps.<ingress domain associated with the openshift cluster>/upload -a /tmp/sigstore/docker/busybox@sha256=a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6/signature-1
this script takes the absolute path to the local signature of the container, builds the json payload and sends that to the signature server via a POST HTTP call. Also, this script makes use of the python3 interpreter so a linux distro that supports at least:
- A fairly recent version of python3
- The python-requests library for python3
is absolutely mandatory.
RAW Repositories in Nexus3 can also host image signature files, so instead of deploying a separate signature server, the same Nexus used to store container images can be used to store signatures too.
- Create a RAW hosted repository called 'sigstore'
- Enable anonymous access
This is needed since in this demo CRI-O is configured without authentication support. Keep in mind that upload on the other hand needs authentication.
- Sign and upload the image as shown in previous paragraphs and then upload the signature to nexus
# ./clients/signature-upload.py -r https://nexus.apps.<ingress domain associated with the openshift cluster> -a /tmp/sigstore/docker/busybox@sha256=a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6/signature-1 --no-verify --nexus -s sigstore -u <username> -p <password>
- Update the repository configuration to use Nexus instead of the HTTP signature server and update the MachineConfig manifests:
docker:
nexus-registry.apps.<ingress domain associated with the openshift cluster>:
sigstore: https://nexus.apps.<ingress domain associated with the openshift cluster>/repository/sigstore
- Create the demo deploymentconfig
# oc project signature-demo
# oc create -f components/demo-deployment.yaml
- Check out the "unsigned" container:
# oc describe pod demo-unsigned-c5d8dddf6-5lkbs
[...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned signature-server/demo-unsigned-c5d8dddf6-5lkbs to ip-10-0-166-156.us-east-2.compute.internal
Normal Pulling 10s kubelet, ip-10-0-166-156.us-east-2.compute.internal Pulling image "nexus-registry.apps.ocp4.sandbox595.opentlc.com/docker/alpine:unsigned"
Warning Failed 10s kubelet, ip-10-0-166-156.us-east-2.compute.internal Failed to pull image "nexus-registry.apps.ocp4.sandbox595.opentlc.com/docker/alpine:unsigned": rpc error: code = Unknown desc = Source image rejected: A signature was required, but no signature exists
Warning Failed 10s kubelet, ip-10-0-166-156.us-east-2.compute.internal Error: ErrImagePull
Normal BackOff 8s (x2 over 9s) kubelet, ip-10-0-166-156.us-east-2.compute.internal Back-off pulling image "nexus-registry.apps.ocp4.sandbox595.opentlc.com/docker/alpine:unsigned"
Warning Failed 8s (x2 over 9s) kubelet, ip-10-0-166-156.us-east-2.compute.internal Error: ImagePullBackOff
The deployment is refused because images from nexus-registry need to be signed, but no signature has been uploaded to the sigstore for this image
- Check out the "wrongly signed" container:
# oc describe pod demo-wrong-signature-68fb74b784-7tqmb
[...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned signature-server/demo-wrong-signature-68fb74b784-7tqmb to ip-10-0-166-156.us-east-2.compute.internal
Normal BackOff 17s (x2 over 42s) kubelet, ip-10-0-166-156.us-east-2.compute.internal Back-off pulling image "nexus-registry.apps.ocp4.sandbox595.opentlc.com/docker/busybox:wrongsig"
Warning Failed 17s (x2 over 42s) kubelet, ip-10-0-166-156.us-east-2.compute.internal Error: ImagePullBackOff
Normal Pulling 5s (x3 over 43s) kubelet, ip-10-0-166-156.us-east-2.compute.internal Pulling image "nexus-registry.apps.ocp4.sandbox595.opentlc.com/docker/busybox:wrongsig"
Warning Failed 5s (x3 over 43s) kubelet, ip-10-0-166-156.us-east-2.compute.internal Failed to pull image "nexus-registry.apps.ocp4.sandbox595.opentlc.com/docker/busybox:wrongsig": rpc error: code = Unknown desc = Source image rejected: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"4F06789A5C76861E", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63723658926, loc:(*time.Location)(0x55f9f39502a0)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55f9f39502a0)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}
Warning Failed 5s (x3 over 43s) kubelet, ip-10-0-166-156.us-east-2.compute.internal Error: ErrImagePull
The deployment is refused because although the image is correctly signed, the signature cannot be verified because the signer private key used to sign the image does not match the public key used to verify the signature.
- Check out the "correctly signed" container:
# oc describe pod demo-signed-6c784b5957-4gpt7
[...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned signature-server/demo-signed-6c784b5957-4gpt7 to ip-10-0-166-156.us-east-2.compute.internal
Normal Pulling 14s kubelet, ip-10-0-166-156.us-east-2.compute.internal Pulling image "nexus-registry.apps.ocp4.sandbox595.opentlc.com/docker/centos:signed"
Normal Pulled 14s kubelet, ip-10-0-166-156.us-east-2.compute.internal Successfully pulled image "nexus-registry.apps.ocp4.sandbox595.opentlc.com/docker/centos:signed"
Normal Created 13s kubelet, ip-10-0-166-156.us-east-2.compute.internal Created container pause
Normal Started 13s kubelet, ip-10-0-166-156.us-east-2.compute.internal Started container pause
This deployment is approved because the signature is correctly found on the sigstore and the verification succeeded with the configured public key.
- Integrate into a Jenkins pipeline
- Make the scripts/manifests more generically usable, as for example domains are for now hardcoded in code.
- Remove shell scripts, migrate to Helm
- Improve documentation