Index annotations aren't pushed after multinode builds
treuherz opened this issue · comments
Contributing guidelines
- I've read the contributing guidelines and wholeheartedly agree
I've found a bug and checked that ...
- ... the documentation does not mention anything about my problem
- ... there are no open or closed issues that are related to my problem
Description
If building a multi-platform image using a multi-node buildx instance and specifying an annotation for the index, no annotation is on the pushed index. Single-platform builds, or multi-platform builds running on a single node all work fine.
Expected behaviour
- Set up buildkit builder with multiple nodes for different platforms
- Build an image for multiple platforms on that builder, with an index annotation:
docker buildx build \
--tag ${AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi \
--platform=linux/amd64,linux/arm64 \
--annotation index:com.example=test \
--file Dockerfile \
--push \
--builder kubernetes-multi \
.
- Get the manifest list and check annotations:
docker buildx imagetools inspect ${AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi --raw | jq '.annotations'
Expect to see the annotation you specified:
{
"com.example": "test"
}
Actual behaviour
No annotations on the returned JSON.
Buildx version
github.com/docker/buildx v0.14.1-desktop.1 5a0555e6c99a65811f4409bce8460a8fb89474f1
Docker info
Client:
Version: 26.1.4
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.14.1-desktop.1
Path: /Users/et/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.27.1-desktop.1
Path: /Users/et/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.32
Path: /Users/et/.docker/cli-plugins/docker-debug
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: /Users/et/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.24
Path: /Users/et/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: /Users/et/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.2.0
Path: /Users/et/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/et/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.9.3
Path: /Users/et/.docker/cli-plugins/docker-scout
Server:
Containers: 36
Running: 25
Paused: 0
Stopped: 11
Images: 9
Server Version: 26.1.4
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d2d58213f83a351ca8f528a95fbd145f5654e957
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.6.31-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 8
Total Memory: 9.708GiB
Name: docker-desktop
ID: 16cbac40-83fc-4e60-9d03-32f0303b90d0
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 23710
Goroutines: 23731
System Time: 2024-06-18T17:41:34.924223628Z
EventsListeners: 13
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/et/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: true
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Registry Mirrors:
https://$AWSACCOUNT.dkr.ecr.eu-west-2.amazonaws.com/
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile
Builders list
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
kubernetes-multi remote
\_ kubernetes-multi0 \_ tcp://localhost:1234 running v0.13.2 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
\_ kubernetes-multi1 \_ tcp://localhost:1235 running v0.13.2 linux/arm64, linux/arm/v7, linux/arm/v6
kubernetes-single* remote
\_ kubernetes-single0 \_ tcp://localhost:1234 running v0.13.2 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
default docker
\_ default \_ default running v0.13.2 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
desktop-linux docker
\_ desktop-linux \_ desktop-linux running v0.13.2 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
Configuration
# syntax=docker/dockerfile:1.7
FROM alpine
RUN echo "hello"
Build logs
❯ docker buildx build -t {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop --platform=linux/amd64,linux/arm64 --annotation index:com.example=test --file Dockerfile --push --builder desktop-linux --no-cache .
#0 building with "desktop-linux" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 98B done
#1 DONE 0.0s
#2 [auth] sharing credentials for 860982037103.dkr.ecr.eu-west-2.amazonaws.com
#2 DONE 0.0s
#3 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#3 DONE 1.7s
#4 docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#4 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e done
#4 CACHED
#5 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#5 ...
#6 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
#6 DONE 0.9s
#5 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#5 DONE 0.9s
#7 [internal] load .dockerignore
#7 transferring context: 2B done
#7 DONE 0.0s
#8 [linux/arm64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#8 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#8 CACHED
#9 [linux/amd64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#9 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#9 CACHED
#10 [linux/arm64 2/2] RUN echo "hello"
#10 0.145 hello
#10 DONE 0.2s
#11 [linux/amd64 2/2] RUN echo "hello"
#11 0.150 hello
#11 DONE 0.2s
#12 exporting to image
#12 exporting layers 0.0s done
#12 exporting manifest sha256:aa42ef74759614614cad53cf6be8499c69f52a9b4eecf72a6997b00657309e4c done
#12 exporting config sha256:d29bc21dc5feeb418c331f4d6e6f7df4c096d666d5ca6f4871698fd4d861c991 done
#12 exporting attestation manifest sha256:6a8172a2b3954728c9e4fd7732744dd3850bb94db05464c104a00655472de03f done
#12 exporting manifest sha256:bd22d6dfd1f3e41a10d896fbc7d2d7e2eba40e1bb77f8c6ebe02236cfec6e445 done
#12 exporting config sha256:ffc79a5fae7c0392f7697ad5c8318dea01f33c8618b5beac3590328010de9ed8 done
#12 exporting attestation manifest sha256:e797530febc2fe2c52d56f3d7022634266afa9765bdbbd12f8ad8bacd2cfef76 done
#12 exporting manifest list sha256:73707543d4357537d25945d49ca56fe438b9f7b7dcecba6fa2163ba49170edd3 done
#12 naming to {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop done
#12 unpacking to {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop done
#12 pushing layers
#12 ...
#13 [auth] sharing credentials for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com
#13 DONE 0.0s
#12 exporting to image
#12 pushing layers 0.9s done
#12 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop@sha256:73707543d4357537d25945d49ca56fe438b9f7b7dcecba6fa2163ba49170edd3
#12 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop@sha256:73707543d4357537d25945d49ca56fe438b9f7b7dcecba6fa2163ba49170edd3 1.2s done
#12 DONE 2.2s
#14 pushing {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop with docker
#14 pushing layer 50e0e13a788a
#14 pushing layer 225a1bb2fd79 0.1s
#14 pushing layer c36e8f02ed96 0.1s
#14 pushing layer d25f557d7f31 0.1s
#14 pushing layer 4f4fb700ef54 0.1s
#14 pushing layer 94747bd81234 0.2s
#14 pushing layer c36e8f02ed96 0.5s done
#14 pushing layer 50e0e13a788a 0.5s done
#14 pushing layer 225a1bb2fd79 0.5s done
#14 pushing layer d25f557d7f31 0.5s done
#14 pushing layer 4f4fb700ef54 0.5s done
#14 pushing layer 94747bd81234 0.5s done
#14 DONE 0.6s
View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/i0xgq94zfjpi52qahyu77m2wu
❯ docker buildx imagetools inspect {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop --raw | jq '.annotations'
{
"com.example": "test"
}
❯ docker buildx build --tag {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-single --platform=linux/amd64,linux/arm64 --annotation index:com.example=test --file Dockerfile --push --builder kubernetes-single --no-cache .
#0 building with "kubernetes-single" instance using remote driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 98B 0.1s done
#1 DONE 0.1s
#2 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#2 DONE 0.8s
#3 docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#3 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#3 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e 0.0s done
#3 CACHED
#4 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#4 ...
#5 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
#5 DONE 0.6s
#6 [internal] load .dockerignore
#6 transferring context: 2B 0.0s done
#6 DONE 0.1s
#4 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.7s
#7 [linux/arm64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#7 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#7 DONE 0.0s
#8 [linux/amd64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#8 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#8 CACHED
#7 [linux/arm64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#7 CACHED
#9 [linux/amd64 2/2] RUN echo "hello"
#9 0.280 hello
#9 DONE 0.3s
#10 [linux/arm64 2/2] RUN echo "hello"
#10 0.254 hello
#10 DONE 0.3s
#11 exporting to image
#11 exporting layers 0.2s done
#11 exporting manifest sha256:aef666dfc4ea829c6f1fe843c38a402a7165ff56f92f6081e2b26eb62f7340b2 0.0s done
#11 exporting config sha256:ed6608e24ef455dd269576b472943fed154e928fe5c80691099c129f71f835de 0.0s done
#11 exporting attestation manifest sha256:b8134296dd9e5402a24e5e94f1fec47ebc8b7525718981764fb98ee01ce03ab7 0.0s done
#11 exporting manifest sha256:2aeeff60f54eda915d0441270b09f1de5d342e661360dede1a3f7e269956dc14 0.0s done
#11 exporting config sha256:75290c5feabd960ee4689960132987bbf29bcae211acd71d651bbcb49285a9a0 0.0s done
#11 exporting attestation manifest sha256:0684c074a098d9f6d0a89b34f5b237843ccd4ab6749a445206658c5fff72e9d3 0.0s done
#11 exporting manifest list sha256:b08cd837ccc4b380cf7a614daf0b88012111c228ce3eefcd4c56d843e40f69d1
#11 ...
#12 [auth] sharing credentials for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com
#12 DONE 0.0s
#11 exporting to image
#11 exporting manifest list sha256:b08cd837ccc4b380cf7a614daf0b88012111c228ce3eefcd4c56d843e40f69d1 0.0s done
#11 pushing layers
#11 pushing layers 0.7s done
#11 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-single@sha256:b08cd837ccc4b380cf7a614daf0b88012111c228ce3eefcd4c56d843e40f69d1
#11 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-single@sha256:b08cd837ccc4b380cf7a614daf0b88012111c228ce3eefcd4c56d843e40f69d1 0.9s done
#11 DONE 1.9s
View build details: docker-desktop://dashboard/build/kubernetes-single/kubernetes-single0/q03fv726k4zl28frk5tx7e5td
❯ docker buildx imagetools inspect {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-single --raw | jq '.annotations'
{
"com.example": "test"
}
❯ docker buildx build --tag {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi --platform=linux/amd64,linux/arm64 --annotation index:com.example=test --file Dockerfile --push --builder kubernetes-multi --no-cache .
#0 building with "kubernetes-multi" instance using remote driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 98B 0.1s done
#1 DONE 0.1s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 98B 0.1s done
#2 DONE 0.1s
#3 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#3 DONE 0.3s
#4 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#4 ...
#5 docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#5 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e 0.0s done
#5 CACHED
#6 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#6 DONE 0.2s
#4 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#4 DONE 0.7s
#7 [internal] load .dockerignore
#7 transferring context: 2B 0.0s done
#7 DONE 0.0s
#8 docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#8 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e 0.0s done
#8 CACHED
#9 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
#9 ...
#10 [linux/amd64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#10 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#10 CACHED
#11 [linux/amd64 2/2] RUN echo "hello"
#11 0.125 hello
#11 DONE 0.1s
#12 exporting to image
#12 exporting layers 0.1s done
#12 exporting manifest sha256:9bcf08188b093c6279b03897c63e3da69f17df0fb0d51faccd4e6a76f927a671 0.0s done
#12 exporting config sha256:07126decdda8123dab09c4124cd2817df528f6e76a2922e57b9886d0b7264bd8 done
#12 exporting attestation manifest sha256:f15763740f79a676c5ef344147fcd8f8c6f4053b06150005e5df4c64a8a8a9dd 0.0s done
#12 exporting manifest list sha256:d7619152588a13b7541a89f828c650a6531321477fb37f3c4ad8e0af37fec6d4 0.0s done
#12 pushing layers
#12 ...
#9 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
#9 DONE 0.5s
#13 [auth] sharing credentials for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com
#13 DONE 0.0s
#14 [internal] load .dockerignore
#14 transferring context: 2B 0.0s done
#14 DONE 0.0s
#15 [linux/arm64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#15 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd done
#15 CACHED
#16 [linux/arm64 2/2] RUN echo "hello"
#16 0.055 hello
#16 DONE 0.1s
#17 exporting to image
#17 exporting layers 0.1s done
#17 exporting manifest sha256:8f3f7e0d6d9e2d38df203a7ab889156da10175dda202361eadd3677ce3642b4f done
#17 exporting config sha256:74aa40bd1cfc0591c63ffbcaf57d9c4fbcb25497af5040e532d4efc29124c17e done
#17 exporting attestation manifest sha256:cb1047a49ac3f540012a3f6860db892f93107cdcbccd7c8c9b157df13d490a18 0.0s done
#17 exporting manifest list sha256:9ceb415afa03a4edbd264e9a13ab0d7e653880501774b0812bb6b1c860d0122e done
#17 pushing layers 0.5s done
#17 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test
#17 ...
#12 exporting to image
#12 pushing layers 0.4s done
#12 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test 0.5s done
#12 DONE 1.1s
#17 exporting to image
#17 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test 0.5s done
#17 DONE 1.2s
#18 merging manifest list {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi
#18 DONE 0.5s
View build details: docker-desktop://dashboard/build/kubernetes-multi/kubernetes-multi1/9pj70lgrhj7rbmgyyta1vd4li
❯ docker buildx imagetools inspect {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi --raw | jq '.annotations'
null
Additional info
The main difference I noticed in the log for kubernetes-multi was that it says "merging manifest list", so possibly the annotations are getting lost in the merge. With a few pointers I'd be happy to contribute a PR for this but the code that produces that log was hard for me to follow from a cold start.
From a bit more reading, I've found part of the problem. The "merging manifest lists" task uses imagetools' Combine method to build the overall index but passes nil
to the ann
argument. I could fix this by plumbing the annotations through in Options
, and probably lifting the parsing and validation currently in Combine out to its other call-site since those limitations seem specific to buildx imagetools create
.
If that sounds alright as an approach then I can get a PR together quite quickly, although I'm not sure where to add tests