error when merging multi-platform manifest(s) to custom tls config registry after successfully pushing for individual platforms
dweomer opened this issue · comments
Contributing guidelines
- I've read the contributing guidelines and wholeheartedly agree
I've found a bug and checked that ...
- ... the documentation does not mention anything about my problem
- ... there are no open or closed issues that are related to my problem
Description
It looks as if the imagetools.Opt
passed to the itpull := imagetools.New(imageopt)
line is lacking the necessary RegistryConfig
to connect to a private registry signed by a CA that isn't included in the system ca-certificates BUT that individual builders are able to push to without issue (meaning, they are configured properly ... the build pulls from the private build cache registry successfully, honoring the private registry cache importer).
Expected behaviour
Can successfully merge multi-platform manifests for blobs that have already been pushed to a private registry.
Actual behaviour
Cannot successfully merge multi-platform manifests for blobs that have already been pushed to a private registry.
Buildx version
github.com/docker/buildx v0.13.1+dweomer.1 5decc6f
Docker info
Client:
Version: 25.0.4
API version: 1.44
Go version: go1.21.8
Git commit: 1a576c5
Built: Wed Mar 6 16:32:02 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.1.3
API version: 1.45 (minimum version 1.24)
Go version: go1.21.10
Git commit: 8e96db1
Built: Thu May 16 08:33:58 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.15
GitCommit: 926c9586fe4a6236699318391cd44976a98e31f1
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
/usr/bin/docker info
Client:
Version: 25.0.4
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.13.1+dweomer.1
Path: /home/runner/.docker/cli-plugins/docker-buildx
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 26.1.3
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 926c9586fe4a6236699318391cd44976a98e31f1
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
Profile: builtin
Kernel Version: 5.10.130-118.517.amzn2.x86_64
Operating System: Alpine Linux v3.19 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 30.9GiB
Name: ip-10-10-11-198.us-gov-east-1.compute.internal
ID: 941f2083-c5f9-4f79-8d28-fb49661dfb6c
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://dhub.cache.svc/
Live Restore Enabled: false
Product License: Community Engine
Builders list
Name: builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e
Driver: kubernetes
Last Activity: 2024-05-25 13:46:09 +0000 UTC
Nodes:
Name: builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e0
Endpoint: kubernetes:///builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e?deployment=buildkit-495d6f30-f49f-491d-a811-0cf9049bccc6-8tfds&kubeconfig=
Driver Options: nodeselector="category=build" tolerations="key=category,value=build"
Status: running
BuildKit daemon flags: --allow-insecure-entitlement=network.host
BuildKit version: v0.13.2
Platforms: linux/amd64*, linux/amd64/v2*, linux/amd64/v3*, linux/amd64/v4*, linux/386*, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
Labels:
org.mobyproject.buildkit.worker.executor: oci
org.mobyproject.buildkit.worker.hostname: builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e0-6df88cfdc6-88rrb
org.mobyproject.buildkit.worker.network: host
org.mobyproject.buildkit.worker.oci.process-mode: sandbox
org.mobyproject.buildkit.worker.selinux.enabled: false
org.mobyproject.buildkit.worker.snapshotter: overlayfs
GC Policy rule#0:
All: false
Filters: type==source.local,type==exec.cachemount,type==source.git.checkout
Keep Duration: 48h0m0s
Keep Bytes: 488.3MiB
GC Policy rule#1:
All: false
Keep Duration: 1440h0m0s
Keep Bytes: 46.57GiB
GC Policy rule#2:
All: false
Keep Bytes: 46.57GiB
GC Policy rule#3:
All: true
Keep Bytes: 46.57GiB
Name: builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e1
Endpoint: kubernetes:///builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e?deployment=buildkit-9e169c6c-af0d-46c4-9ad3-1589f6a15580-5lp28&kubeconfig=
Driver Options: nodeselector="category=build-arm64" tolerations="key=category,value=build-arm64"
Status: running
BuildKit daemon flags: --allow-insecure-entitlement=network.host
BuildKit version: v0.13.2
Platforms: linux/arm/v6*, linux/arm/v7*, linux/arm64*
Labels:
org.mobyproject.buildkit.worker.executor: oci
org.mobyproject.buildkit.worker.hostname: builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e1-cb9c654df-kxqgr
org.mobyproject.buildkit.worker.network: host
org.mobyproject.buildkit.worker.oci.process-mode: sandbox
org.mobyproject.buildkit.worker.selinux.enabled: false
org.mobyproject.buildkit.worker.snapshotter: overlayfs
GC Policy rule#0:
All: false
Filters: type==source.local,type==exec.cachemount,type==source.git.checkout
Keep Duration: 48h0m0s
Keep Bytes: 488.3MiB
GC Policy rule#1:
All: false
Keep Duration: 1440h0m0s
Keep Bytes: 46.57GiB
GC Policy rule#2:
All: false
Keep Bytes: 46.57GiB
GC Policy rule#3:
All: true
Keep Bytes: 46.57GiB
Configuration
FROM library/alpine:edge
RUN echo 'unable to share this but the same dockerfile merges just fine to ghcr.io'
Build logs
#26 exporting to image
#26 ...
#27 exporting to image
#27 exporting layers
#27 ...
#26 exporting to image
#26 exporting layers 65.5s done
#26 exporting manifest sha256:d9be2cdb45c5b07b54691e153ad5b6b4c8d527356500323f9fea81df300876c5 done
#26 exporting config sha256:17675bb9b8dc515066bc0f326b2d548dfe1232579f588a1cbdbf5c45a7f726cd done
#26 exporting attestation manifest sha256:8c81834da520a243b0fd108537c970d94c4bee5e270a6bfcd74b9c8a38854e5f 0.0s done
#26 exporting manifest list sha256:dc6732014a6873697cadcc9531c31b7504193fbecac1411ea3491d8118152bcb done
#26 pushing layers
#26 pushing layers 3.8s done
#26 pushing manifest for build.cache.svc/my-project/my-image
#26 pushing manifest for build.cache.svc/my-project/my-image 0.0s done
#26 DONE 69.4s
#28 exporting cache to registry
#28 preparing build cache for export
#28 writing layer sha256:08b1720df82a0beee132289941ac9ee2eba74a7d2ad637c1a8352366d751fb25 done
#28 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
#28 writing layer sha256:561cb69653d56a9725be56e02128e4e96fb434a8b4b4decf2bdeb479a225feaf done
#28 writing layer sha256:8f665685b215c7daf9164545f1bbdd74d800af77d0d267db31fe0345c0c8fb8b done
#28 writing layer sha256:9361d72813976e1175ddb2fbce2e5f0ab01e71a419990d64e71bc36946edd884 done
#28 writing layer sha256:96ad531c39c935bc6319f19f3be8f9f4a6faa15ded833ad2bd50a95a0d95e8d2 done
#28 writing layer sha256:e5fca6c395a62ec277102af9e5283f6edb43b3e4f20f798e3ce7e425be226ba6 done
#28 writing layer sha256:f56be85fc22e46face30e2c3de3f7fe7c15f8fd7c4e5add29d7f64b87abdaa09 done
#28 writing layer sha256:fc07f0dda8ec1c1acc98ab6a4673371611db7184cff56ddef0eba11523eec347 done
#28 writing config sha256:5bf508bda394326c3229d6ad06bcb6bded9357713a60be9a5056503b68adbadf 0.0s done
#28 writing cache manifest sha256:e681f494875749ceb3083097acf67ee72e298cc01dce5ab63e5f856b65cbf12c
#28 preparing build cache for export 0.1s done
#28 writing cache manifest sha256:e681f494875749ceb3083097acf67ee72e298cc01dce5ab63e5f856b65cbf12c 0.0s done
#28 DONE 0.1s
#27 exporting to image
#27 exporting layers 68.0s done
#27 exporting manifest sha256:c3092a12a16f9d5411701e95592b1f0d0d64b24ff810727cf911128403848f11 done
#27 exporting config sha256:459fb84f04c080c7a977c605b777e903e4a135002b442ede68aed725320f5880 done
#27 exporting attestation manifest sha256:c13a9929f49c119f9dccbeeeb763a84548efa08dea86a7546ad9a128dbc5e9c5 0.0s done
#27 exporting manifest list sha256:fc95710499e7bb88684294ec76c284fa9c73444468793654a5c68ecb3b059397 done
#27 pushing layers
#27 pushing layers 3.7s done
#27 pushing manifest for build.cache.svc/my-project/my-image
#27 pushing manifest for build.cache.svc/my-project/my-image 0.0s done
#27 DONE 71.7s
#29 exporting cache to registry
#29 preparing build cache for export
#29 writing layer sha256:4cf6a83c0e2af3c780abcda02cc33f9e812fdcb40b610ed1838281cc9ab94ec8 done
#29 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
#29 writing layer sha256:5a63f40ac9bbdfab87854860e46116e14c81556e0d159437bcbd13ec83848687
#29 preparing build cache for export 0.1s done
#29 writing layer sha256:5a63f40ac9bbdfab87854860e46116e14c81556e0d159437bcbd13ec83848687 done
#29 writing layer sha256:683339ce8d6b9be2ca150a8de67b895e20ea5594b91d3911c95b0b8fea3e314c done
#29 writing layer sha256:686172e40c38722891b4004f55f6447548c8367968ac523a612591e0d92f9db3 done
#29 writing layer sha256:c41833b44d910632b415cd89a9cdaa4d62c9725dc56c99a7ddadafd6719960f9 done
#29 writing layer sha256:e83c0d77c542c0ae16eda4f948bdc6e84b0a82b8a00068b7eeb5a5a743b1b453 done
#29 writing layer sha256:ed43d91b02ce995d68736bc3af861c28500f6109fcb8d62179c71ffa023ce97a done
#29 writing layer sha256:fc1eefa94020698f74056fc3449798c2319f23cb42221d278064fa8f8ea616c0 done
#29 writing config sha256:95ee56b834bf8aa0dde7ef40d4fe16146f00da17d3c14ca69fabb7aafe8f9e87 0.0s done
#29 writing cache manifest sha256:777b29ca996df891e166c85a82232c6da4b94c19470a3d5ca32c0641144ede04 0.0s done
#29 DONE 0.1s
#30 merging manifest list build.cache.svc/my-project/my-image:my-tag,build.cache.svc/my-project/my-image:sha-cc220b522f58843a818603b89cf6195fd4b30643,build.cache.svc/my-project/my-image:latest
#30 ERROR: httpReadSeeker: failed open: failed to do request: Get "https://build.cache.svc/v2/my-project/my-image/manifests/sha256:fc95710499e7bb88684294ec76c284fa9c73444468793654a5c68ecb3b059397": tls: failed to verify certificate: x509: certificate signed by unknown authority
------
> merging manifest list build.cache.svc/my-project/my-image:my-tag,build.cache.svc/my-project/my-image:sha-cc220b522f58843a818603b89cf6195fd4b30643,build.cache.svc/my-project/my-image:latest:
------
Additional info
This is driven via github actions on a private runner, leveraging:
- https://github.com/marketplace/actions/docker-metadata-action
- https://github.com/marketplace/actions/docker-setup-buildx
- https://github.com/marketplace/actions/build-and-push-docker-images
- https://github.com/dweomer/buildx/releases/tag/v0.13.1%2Bdweomer.1
See also:
So, updating the system ca trust got me further, i.e.:
# runner startup tweaks
sudo ln -vs /etc/docker/certs.d/build.cache.svc/ca.crt /usr/local/share/ca-certificates/build-cache-svc.crt
sudo update-ca-certificates
But then I see this in the access logs for build.cache.svc:
10.42.165.22 - - [25/May/2024:16:14:08 +0000] "PUT /v2/my-project/my-image/manifests/my-tag HTTP/1.1" 201 0 "" "buildkit/v0.13"
2024-05-25T16:14:08.762990483Z time="2024-05-25T16:14:08.7627864Z" level=info msg="response completed" go.version=go1.20.8 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=build.cache.svc http.request.id=f0f62fae-b358-4b4f-af1c-64d8f2ddbdc8 http.request.method=PUT http.request.remoteaddr="10.42.165.22:58088" http.request.uri="/v2/my-project/my-image/manifests/my-tag" http.request.useragent="buildkit/v0.13" http.response.duration=7.41057ms http.response.status=201 http.response.written=0
2024/05/25 16:14:08 http: TLS handshake error from 10.42.53.0:63128: tls: client didn't provide a certificate
2024/05/25 16:14:08 http: TLS handshake error from 10.42.53.0:45160: EOF
Progress! Now the issue is the client (buildx imagetools, I believe) is neglecting to send a client certificate when pushing to the private build cache registry. So, I tried these additional tweaks at runner startup:
# match default expectations of https://github.com/docker/setup-buildx-action
# sans a DOCKER_CONFIG env override (which fails due to attempted writes on a read-only volumeMount)
ln -vs /etc/docker/certs.d ~/.docker/
# match the translated buildkitd.toml registry ca/keypair entries shipped to buildkitd backends
sudo mkdir -p /etc/buildkit
sudo ln -vs /etc/docker/certs.d /etc/buildkit/certs
... but still no joy.
AFAIK I have correctly followed the mTLS setup for my build-cache registry, i.e.:
runner@ip-10-10-11-198:~$ ls -alF /etc/docker/certs.d/build.cache.svc/
total 0
dr-xr-x--- 3 root docker 140 May 25 16:14 ./
drwxr-xr-x 7 root root 122 May 25 16:15 ../
drwxr-xr-x 2 root root 100 May 25 16:14 ..2024_05_25_16_14_57.1642649139/
lrwxrwxrwx 1 root root 32 May 25 16:14 ..data -> ..2024_05_25_16_14_57.1642649139/
lrwxrwxrwx 1 root root 13 May 25 16:14 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 18 May 25 16:14 client.cert -> ..data/client.cert
lrwxrwxrwx 1 root root 17 May 25 16:14 client.key -> ..data/client.key
hi @tonistiigi, i do not understand this edit:
our build.cache.svc
registry is secured via mTLS (with some dummy basic auth) which buildkitd backend(s) push to without issue but with the same config present on the client isn't recognized by buildx imagetools (some call into that go package, i forget) and so it doesnt expect the custom ca nor does it use the client certificate when negotiating an https connection. am i missing something? the "insecure registry" designation is a category error, from my perspective.
was digging into this further yesterday (because i am unreasonably angry that the awesome buildx imagetools fails like this) and it looks like a problem with the containerd resolver that imagetools is relying on (i even tried an /etc/containerd/certs.d/_default/hosts.toml
with no luck). something truncated in the setup, likely a gap between what ctr adds and imagetools expects.
also, docker push to this mTLS secured registry works fine via the tlscacert
, tlscert
, and tlskey
flags to docker but these are not picked up by buildx imagetools.
likely related issues: