Custom registry, push error on self-signed cert
minusdelta opened this issue · comments
'buildx' errors, while 'docker build' succeeds:
cat <<'EOD' > Dockerfile
FROM alpine
RUN touch /test
EOD
docker buildx build \
-t img.service.consul/alpine:test \
--platform=linux/amd64,linux/arm64,linux/arm \
--push \
.
[+] Building 2.1s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 65B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [linux/arm/v7 internal] load metadata for docker.io/library/alpine:latest 1.5s
=> [linux/amd64 internal] load metadata for docker.io/library/alpine:latest 1.3s
=> [linux/arm64 internal] load metadata for docker.io/library/alpine:latest 1.3s
=> CACHED [linux/arm64 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> [linux/arm64 2/2] RUN touch /test 0.1s
=> CACHED [linux/amd64 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> [linux/amd64 2/2] RUN touch /test 0.1s
=> CACHED [linux/arm/v7 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> [linux/arm/v7 2/2] RUN touch /test 0.2s
=> ERROR exporting to image 0.4s
=> => exporting layers 0.2s
=> => exporting manifest sha256:879ac4adf9493121ff9bb12f8566ed993fa7079c59ae02b516a9287d6de7daea 0.0s
=> => exporting config sha256:42c2e158eb64d21ea6832e4842a3c11c9fd70c89afbf2fd0fbe2f16dd2698453 0.0s
=> => exporting manifest sha256:64b17d691e5c5ab257ad37b622c9ed219e50ea637bf5f7aa25e2f65f0bd0c26d 0.0s
=> => exporting config sha256:14af510e076d60389743c6fc7c99e2777ba56bdad11cbbffacb438e7c68f6321 0.0s
=> => exporting manifest sha256:2e87dbf064ba1c829a9d18525fa38e77baa26b654c04659b0fa3e75d6ea34ea5 0.0s
=> => exporting config sha256:f1b89d61d625bff13e65a679d2bdb1c513289999789ec2e13fe7acefca39adfd 0.0s
=> => exporting manifest list sha256:53f07aa12e20079138de3650629277928313e7bfdc59c3f22c93834fe11ba9f3 0.0s
=> => pushing layers 0.0s
------
> exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed to do request: Head https://img.service.consul/v2/alpine/blobs/sha256:8dc302c06141b7124ea05ccf2fdde10013ce594c28e5fe980047b0740891e398: x509: certificate signed by unknown authority
x509: certificate signed by unknown authority, but certificate chain is ok.
test:
: |openssl s_client -connect img.service.consul:443 [...] Verify return code: 0 (ok)
docker build + push works also:
docker build \
-t img.service.consul/x86_64/alpine:test \
.
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM alpine
---> 055936d39205
Step 2/2 : RUN touch /test
---> Using cache
---> 7bd5dcd02d4c
Successfully built 7bd5dcd02d4c
Successfully tagged img.service.consul/x86_64/alpine:test
docker push img.service.consul/x86_64/alpine:test
The push refers to repository [img.service.consul/x86_64/alpine]
b13e8440598c: Pushed
f1b5933fe4b5: Pushed
test: digest: sha256:e9c2e8f188d0bedc6d3c26b39a6a75c36be5b4cbeedb9defc4f3b48953b4ef45 size: 734
buildx imagetools inspect again works:
docker buildx imagetools inspect img.service.consul/x86_64/alpine:test
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1645,
"digest": "sha256:7bd5dcd02d4c340892fe431a40a39badf5695af58b669a33bd21b61159f4ffe5"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2757034,
"digest": "sha256:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 97,
"digest": "sha256:6e0d312d9ebb1961db61366af2a5a323ad84155db2018457d2c5168c4f86e410"
}
]
}
perhaps related to #57 (comment)
tested:
- docker beta-4 + 20190517171028-f4f33ba16d nightly,
- 'buildx' release + current master.
uname -mrov
4.19.44 #1-NixOS SMP Thu May 16 17:41:32 UTC 2019 x86_64 GNU/Linux
In my case, it wasn't a self-signed cert but a company-wide CA cert. It's very possible that CA cert wasn't present in the environment I was running buildx from though so I could look into that.
I'm seeing similar with a standalone registry server and locally created CA/certificates. I'm able to run a docker login after adding a /etc/docker/certs.d/<hostname>:5000/ca.crt file, but buildx doesn't appear to use this.
I managed to work-around this - by adding my own CA's cert to the generated container, and restarting it.
$ docker ps|grep 'moby/buildkit'
ee110c9e6dfc moby/buildkit:buildx-stable-1 "buildkitd" 7 minutes ago Up 7 minutes buildx_buildkit_mybuilder0
$ docker exec -it ee110c9e6dfc sh
$$ cat >> /etc/ssl/certs/ca-certificates.crt <<'EOF'
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOF
$$ exit
$ docker restart ee110c9e6dfc
It aint pretty, but it works!
A marginally more robust work-around, but still not pretty (no error checking etc):
BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDER
A possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.
as I just bumped into this issue, I'd like to add one more option instead of mangling an existing builder container. driver-opt has a image option (for docker-container driver).
I went with the option to have a 2 line Dockerfile that adds my internal CAs to moby/buildkit and use that image when creating the builder.
@fopina can you elaborate on how you did that? I have a 2 line dockerfile where I copy a cert into /usr/local/share/ca-certificates/, and I can run it and exec into it to verify that it's there. I create the image locally, tagging it like testbuilder:latest
When I run docker buildx create --name tester --builder testbuilder:latest, the resulting container that's spun up doesn't have the cert in that folder:
Create and verify the image:
% docker run --rm -ti testbuilder sh
INFO[0000] auto snapshotter: using native
WARN[0000] using host network as the default
INFO[0000] found worker "t3jdqv8o0ov117tdipqjmfnly", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:000f6536acfa org.mobyproject.buildkit.worker.snapshotter:native], platforms=[linux/amd64 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/arm/v7 linux/arm/v6]
WARN[0000] skipping containerd worker, as "/run/containerd/containerd.sock" does not exist
INFO[0000] found 1 workers, default="t3jdqv8o0ov117tdipqjmfnly"
WARN[0000] currently, only the default worker can be used.
INFO[0000] running server on /run/buildkit/buildkitd.sock
# In a different terminal window:
% docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
000f6536acfa testbuilder "buildkitd sh" 18 seconds ago Up 17 seconds jovial_blackwell
% docker exec -it jovial_blackwell sh
/ # cd /usr/local/share/ca-certificates/
/usr/local/share/ca-certificates # ls
certs.pemusing it as a builder?
% docker buildx create --name tester --builder testbuilder:latest
% docker buildx use tester
% docker buildx inspect --bootstrap
[+] Building 1.0s (1/1) FINISHED
=> [internal] booting buildkit 1.0s
=> => pulling image moby/buildkit:buildx-stable-1 0.4s
=> => creating container buildx_buildkit_tester0 0.6s
Name: tester
Driver: docker-container
Nodes:
Name: tester0
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
% docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d9c5e77418fc moby/buildkit:buildx-stable-1 "buildkitd" 9 seconds ago Up 8 seconds buildx_buildkit_tester0
docker exec -ti d9c5e77418fc sh
/ # ls /usr/local/share/ca-certificates/
/ # % < -- nothing here@bdharrington7 have you tried curl in that image (on the registry endpoint) to validate that they have been properly installed?
I believe you have to run update-ca command unless you already pushed the total file.
I didn't try curl, but I did try using the --push option when I built another image using buildx, and it still failed, because the registry I'm pushing to uses an internal cert. I assumed that the image I specified with the --builder flag wasn't being used, since it didn't have the same information in the one I actually built when I spun up testbuilder:latest manually (the cert I baked into the testbuilder image at /usr/local/share/ca-certificates/ isn't even there)
A marginally more robust work-around, but still not pretty (no error checking etc):
BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDERA possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.
Tried that, but it didn't work.
Considering this issue has been open for over two years, I was just curious if there is an update and/or plans to address?
Reboot fixed it for me, I'm not sure why..
I didn't try curl, but I did try using the
--pushoption when I built another image using buildx, and it still failed, because the registry I'm pushing to uses an internal cert. I assumed that the image I specified with the--builderflag wasn't being used, since it didn't have the same information in the one I actually built when I spun uptestbuilder:latestmanually (the cert I baked into the testbuilder image at/usr/local/share/ca-certificates/isn't even there)
To modify buildx builder image use buildx create --driver-opt image=yourimage
@tonistiigi is there any documentation on how to get thos to work now?
Hi, guys. We still hit this issue in buildx v0.8.1
➜ multi-build-template git:(master) ✗ docker buildx version
github.com/docker/buildx v0.8.1-docker 5fac64c2c49dae1320f2b51f1a899ca451935554
➜ multi-build-template git:(master) ✗ make build-demo-amd64
docker buildx build --builder demo --platform linux/amd64 \
-o type=docker \
-t demo/demo:v0.1-amd64 \
-f ./Dockerfile.amd64 \
--build-arg GOARCH=amd64 \
--build-arg GO_VERSION=1.13.15 \
--build-arg CGO_ENABLED=0 \
--build-arg GO111MODULE=on \
--build-arg APPLICATION_NAME=demo \
--build-arg FILE_NAME=./demo.go \
--build-arg COMMIT_ID=$(git rev-parse --short HEAD)+"dirty" .
[+] Building 0.2s (7/7) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile.amd64 0.0s
=> => transferring dockerfile: 915B 0.0s
=> ERROR [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/ubi8@sh 0.2s
=> ERROR [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/golang: 0.2s
=> [auth] library/golang:pull token for harbor.mysql.dbdns.repo-name.cn 0.0s
=> [auth] library/ubi8:pull token for harbor.mysql.dbdns.repo-name.cn 0.0s
=> [auth] library/ubi8:pull token forharbor.mysql.dbdns.repo-name.cn 0.0s
------
> [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/ubi8@sha256:fef1f0adb231b94c80f22be55bb7678b5c8898d3eb314ee7ceeae7c520d76958:
------
------
> [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/golang:1.13.15:
------
Dockerfile.amd64:2
--------------------
1 | ARG GO_VERSION
2 | >>> FROMharbor.mysql.dbdns.repo-name.cn/library/golang:$GO_VERSION as gobuilder
3 |
4 | ARG CGO_ENABLED
--------------------
error: failed to solve: failed to fetch oauth token: Post "https://harbor.mysql.dbdns.repo-name.cn/service/token": x509: certificate signed by unknown authority
make: *** [build-demo-amd64] Error 1
Here is the configuration of buildkit.toml
debug=true
[registry."otherrepo-workfine.io:5000"]
http = true
insecure = true
[registry."harbor.mysql.dbdns.repo-name.cn"]
ca=["/etc/docker/certs.d/harbor.mysql.dbdns.repo-name.cn/ca.crt"]
[[registry."harbor.mysql.dbdns.repo-name.cn".keypair]]
key="/etc/docker/certs.d/harbor.mysql.dbdns.repo-name.cn/harbor.mysql.dbdns.repo-name.cn.key"
cert="/etc/docker/certs.d/harbor.mysql.dbdns.repo-name.cn/harbor.mysql.dbdns.repo-name.cn.cert"
We checked all the certifications are copied into the buildx container however, it looks like did not work.
More information about the token, so, we believe the token is working well.
# Get token
curl -i -k -u admin:cddHarbor@1234 'https://harbor.mysql.dbdns.repo-name.cn/service/token?service=harbor-registry&scope=repository:library/golang:pull,push'
# Get information successfully from the harbor with `ca.crt` and `token` we got above.
curl -i --cacert /home/harbor/ca/ca.crt -H "Content-Type: application/json" -H "Authorization: Bearer `tokenStr`" \
'https://harbor.mysql.dbdns.repo-name.cn/v2/library/golang/manifests/1.15.15'
Finally, we solved this issue. And there are two steps we need to confirm before we use to build:
- Make sure we use
ca.crtwas created by Harbor(we found the correct pah of it is/home/harbor/ca/ca.crt), (it is not the one that is related to Harbor's domain). - If you are in the CentOS (yum) distribution GNU/Linux environment, make sure the host trusts Harbor's certification
# Execution of this command on the host
cp /home/harbor/ca/ca.crt /etc/pki/ca-trust/source/anchors/ update-ca-trust extractAll things are good to go.
I checked all the certifications are copied into the buildx container and "cp /home/harbor/ca/ca.crt /etc/pki/ca-trust/source/anchors/ update-ca-trust extract". however, it looks like did not work.
can we make buildx skip verify private registry cert like docker ( /etc/docker/daemon.json insecure-registries) ？ then we have no need to provide CA cert for buildx instance.
can we make buildx skip verify private registry cert like docker ( /etc/docker/daemon.json insecure-registries) ？ then we have no need to provide CA cert for buildx instance.