containerd can't pull image from Github Docker Package Registry
csantanapr opened this issue · comments
Using the new github docker registry containerd kubernetes can't pull image but using docker engine based k8s works fine.
Steps to reproduce the issue:
- Create a secret with github docker registry token
Follow instructions here:
https://help.github.com/en/articles/configuring-docker-for-use-with-github-package-registry#authenticating-to-github-package-registry
Using kubectl
kubectl create secret docker-registry regcred --docker-server=https://docker.pkg.github.com --docker-username=<user | org>--docker-password=15650cad4e8a6602284255f7caf76134eb977b45 --docker-email=<email>
- Create pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go
imagePullSecrets:
- name: regcred- Create a pod
kubectl apply -f pod.yamlDescribe the results you received:
Errors for the Pod pulling image
31s Normal Pulling Pod pulling image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go"
31s Warning Failed Pod Failed to pull image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go": rpc error: code = Unknown desc = failed to resolve image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest": no available registry endpoint: docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest not found
31s Warning Failed Pod Error: ErrImagePull
3s Normal BackOff Pod Back-off pulling image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go"
3s Warning Failed Pod Error: ImagePullBackOff
Describe the results you expected:
Pod is in Running State
Here is the output when running same scenario on minikube with docker engine
Normal Pulling 2m58s kubelet, minikube Pulling image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go"
Normal Pulled 2m48s kubelet, minikube Successfully pulled image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go"
Normal Created 2m48s kubelet, minikube Created container private-reg-container
Normal Started 2m47s kubelet, minikube Started container private-reg-container
Output of containerd --version:
I'm running on IKS here is the version 1.2.6 when running kubectl get nodes -o wide
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
10.187.176.105 Ready <none> 18d v1.13.5+IKS 10.187.176.105 169.*.*.* Ubuntu 18.04.2 LTS 4.15.0-47-generic containerd://1.2.6
@estesp ^^ this is the issue I ping you today at KubeCon
Does ctr --debug images pull shows interesting log?
I initially thought that this might be because that you have 4 fields in the image name, but it turns out that containerd handles that fine:
$ crictl pull gcr.io/google.com/noogler-kubernetes/busybox:test
Image is up to date for sha256:c2013f624d1666afd5e579d8628ed5375de1b32a54083274701c6df495b0e499
Can you enable debug and paste the containerd log here?
Figuring this out will likely require an HTTP capture. Often registries only implement the subset of the specification used by the Docker engine, not working with some other clients. Usually I need to see how the registry is responding to specific API endpoints to figure out what is not working correctly, this is common with many registries, especially newer implementations.
I'm not very familiar on how to configure and run ctr/crictl on my osx to setup the auth and pull the image.
if you have a simple way to do it or I can share my github docker registry token and you can try pulling my image, I can send my credentials via email to you @Random-Liu @dmcgowan
@csantanapr If possible, that would be helpful for us to debug. :)
I'm pinging someone from github to see if they can help as well.
@Random-Liu sent you my registry auth via DM on Slack.
Hi, I'm the Engineering Lead on GitHub Package Registry and am investigating this.
From what I can tell, there's a request coming in for the docker/knative-samples_helloworld-go:latest Docker Image, but it doesn't have any credentials associated with it.
If you notice the user value in this logline 👇
app=github env=production user=nil
user_agent=containerd/v1.2.0-580-gbcb6c8db47 accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *"
status=404
url="https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest"Maybe the regcred secret in the kube config is not being used when pulling the image?
I'm happy to jump on a video call to debug this with screensharing if someone from containerd is also up for joining us.
@Phanatic I would be happy to go over the registry API and how containerd client is using it.
The package registry is only returning 401 on the /v2 "ping" endpoint. This endpoint will be removed from future version of Docker. The expected use of the API is that 401 is returned by any endpoint which requires authorization. When an endpoint returns 404 when no auth is provided, then the client will not know to provide authorization. The expected flow from a client for any resource...if no auth then return 200, 401, or 404 (only when everything is public), if with auth then return 200 or 404. This is because the start of a registry interaction may begin on any resource, as the client may not need all resources.
This is all based on what is defined by https://github.com/opencontainers/distribution-spec/blob/master/spec.md rather than Docker's current very specific flow.
Interesting, thanks for the context @dmcgowan !
The package registry is only returning 401 on the /v2 "ping" endpoint.
This behavior is consistent with DockerHub's registry APIs,
curl -I https://registry-1.docker.io/v2/
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"Is there a change required here for GitHub Docker Registry to work with containerd?
Re: 👇
When an endpoint returns 404 when no auth is provided, then the client will not know to provide authorization.
I think I see the disconnect here, please correct me if I'm wrong, the containerd image pull flow doesn't provide credentials on first try, instead it tries first without credentials and if the endpoint returns a 401, it'll try again with auth. Is this accurate?
the containerd image pull flow doesn't provide credentials on first try
No credentials are provided until a 401 is received from a registry informing containerd what type of Authorization is expected. This could be basic or bearer.
Is there a change required here for GitHub Docker Registry to work with containerd?
containerd does not every contact https://registry-1.docker.io/v2/ as this endpoint provides no purpose to the overall registry flow. This endpoint was originally put in place to distinguish a v2 registry from a v1 registry index server (this has been long deprecated and never supported by containerd). Later this year Docker will no longer contact this endpoint either as v1 registry support has been completely removed in the upcoming version of Docker.
No credentials are provided until a 401 is received from a registry informing containerd what type of Authorization is expected. This could be basic or bearer.
alright, that was the 🔑 , I know how to fix this in GitHub Docker Registry 😄
I'll start working on a fix and comment on this thread when I have the fix deployed to production.
Thank you so much for trying out GPR and giving us valuable feedback ❤️
@csantanapr, @dmcgowan, I just deployed a fix to the GitHub Docker Registry to return the right status code if the user is unauthenticated. Please try the kube deploy again.
We now return a 401-UnAuthorized with the WWW-Authenticate header set to Basic.
curl -Ssv https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest
> GET /v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest HTTP/1.1
> Host: docker.pkg.github.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Basic realm="GitHub Docker Registry"
< Server: GitHub.com
* Connection #0 to host docker.pkg.github.com left intact
{"errors":[{"code":"UNAUTHORIZED","message":"GitHub Docker Registry needs login"}]}Thanks, the authentication seems to work now.
The manifest fetch by digest doesn't seem to work though.
Also I recommend returning the Docker-Content-Digest header on manifest requests to avoid making the client do an extra GET for digest computation.
More generally, fetching by tag is done to resolve to a digest. In this case the registry is used as a trusted source for what that named tag represents. Client may also use external ways to trust a name such as notary or always pinning their deployed images to a manifest digest.
awesome, thanks for the context! I'll read up on https://github.com/opencontainers/distribution-spec/blob/master/spec.md and try to setup a repro that I can use to debug this further. I'd love to get the gaps identified so we can tackle them all in one go.
@Phanatic any updates on this?
@Phanatic
The manifest fetch by digest doesn't seem to work for me too.
Is it because I'm missing something or this is a known issue/feature? Thanks.
Hello is this issue resolved?
I am getting 406 Not Acceptable when trying to pull from docker.pkg.github.com and I can't find any reference for this status code anywhere
I have tried both with user creds and without and I got the same output
ctr --debug images pull docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest
DEBU[2019-09-16T09:20:19.784036734Z] fetching image="docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest"
DEBU[2019-09-16T09:20:19.784162084Z] resolving
DEBU[2019-09-16T09:20:19.784183008Z] do request request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD url="https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest"
DEBU[2019-09-16T09:20:20.187236230Z] fetch response received response.headers="map[Content-Length:[0] Content-Security-Policy:[default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com] Content-Type:[text/html; charset=utf-8] Date:[Mon, 16 Sep 2019 09:20:20 GMT] Expect-Ct:[max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"] Referrer-Policy:[origin-when-cross-origin, strict-origin-when-cross-origin] Server:[GitHub.com] Status:[406 Not Acceptable] Strict-Transport-Security:[max-age=31536000; includeSubdomains; preload] X-Content-Type-Options:[nosniff] X-Frame-Options:[deny] X-Github-Backend:[Kubernetes] X-Github-Request-Id:[CF30:6C76:5C7D:6242A:5D7F53D3] X-Request-Id:[6b911397-a1f9-4ad8-8973-d1e1a703ad9d] X-Runtime:[0.004638] X-Runtime-Rack:[0.010905] X-Xss-Protection:[1; mode=block]]" status="406 Not Acceptable" url="https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest"
ctr: failed to resolve reference "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest": unexpected status code https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest: 406 Not Acceptable
ctr --debug images pull --user user:pass docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest
DEBU[2019-09-16T09:22:04.276359468Z] fetching image="docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest"
DEBU[2019-09-16T09:22:04.276496151Z] resolving
DEBU[2019-09-16T09:22:04.276524600Z] do request request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD url="https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest"
DEBU[2019-09-16T09:22:04.723342816Z] fetch response received response.headers="map[Content-Length:[0] Content-Security-Policy:[default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com] Content-Type:[text/html; charset=utf-8] Date:[Mon, 16 Sep 2019 09:22:04 GMT] Expect-Ct:[max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"] Referrer-Policy:[origin-when-cross-origin, strict-origin-when-cross-origin] Server:[GitHub.com] Status:[406 Not Acceptable] Strict-Transport-Security:[max-age=31536000; includeSubdomains; preload] X-Content-Type-Options:[nosniff] X-Frame-Options:[deny] X-Github-Backend:[Kubernetes] X-Github-Request-Id:[EC7A:54BA:BE46:89229:5D7F543C] X-Request-Id:[c1107b57-bfad-4ad8-96ae-dc0f093908a6] X-Runtime:[0.005667] X-Runtime-Rack:[0.015374] X-Xss-Protection:[1; mode=block]]" status="406 Not Acceptable" url="https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest"
ctr: failed to resolve reference "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest": unexpected status code https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest: 406 Not Acceptable
Hi, I am trying to use Kubernetes Kind and it seems I am hitting this issue, too . kubernetes-sigs/kind#870
I tried both 1.2 and 1.3 latest released versions and I am still getting the same error.
root@ubuntu-s-1vcpu-1gb-sfo2-01:~/bin# ./ctr version
Client:
Version: v1.2.10
Revision: b34a5c8af56e510852c35414db4c1f4fa6172339
Server:
Version: v1.2.10
Revision: b34a5c8af56e510852c35414db4c1f4fa6172339
root@ubuntu-s-1vcpu-1gb-sfo2-01:~/bin# ./ctr --debug images pull docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64
DEBU[0000] fetching image="docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
DEBU[0000] resolving
DEBU[0000] do request request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
DEBU[0000] fetch response received response.headers=map[Content-Length:[0] Content-Security-Policy:[default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com] Content-Type:[text/html; charset=utf-8] Date:[Thu, 26 Sep 2019 21:06:27 GMT] Expect-Ct:[max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"] Referrer-Policy:[origin-when-cross-origin, strict-origin-when-cross-origin] Server:[GitHub.com] Status:[406 Not Acceptable] Strict-Transport-Security:[max-age=31536000; includeSubdomains; preload] X-Content-Type-Options:[nosniff] X-Frame-Options:[deny] X-Github-Backend:[Kubernetes] X-Github-Request-Id:[A5AC:664D:2653:1EA75:5D8D2853] X-Request-Id:[2843882f-0329-45f1-8283-91703c6721f9] X-Runtime:[0.005880] X-Runtime-Rack:[0.014772] X-Xss-Protection:[1; mode=block]] status="406 Not Acceptable" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
ctr: failed to resolve reference "docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64": unexpected status code https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64: 406 Not Acceptable
root@ubuntu-s-1vcpu-1gb-sfo2-01:~/bin# ./ctr version
Client:
Version: v1.3.0-rc.3
Revision: da66333271f372204aed6b83c2ecf37fa7d9ae2c
Server:
Version: v1.3.0-rc.3
Revision: da66333271f372204aed6b83c2ecf37fa7d9ae2c
UUID: 53b136dc-1bf4-47d2-b079-70bf34ed10fd
root@ubuntu-s-1vcpu-1gb-sfo2-01:~/bin# ./ctr --debug images pull docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64
DEBU[0000] fetching image="docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
DEBU[0000] resolving host=docker.pkg.github.com
DEBU[0000] do request host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *" request.header.user-agent=containerd/v1.3.0-rc.3 request.method=HEAD url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
DEBU[0000] fetch response received host=docker.pkg.github.com response.header.content-length=0 response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="text/html; charset=utf-8" response.header.date="Thu, 26 Sep 2019 21:10:36 GMT" response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.status="406 Not Acceptable" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="84A8:17A6:67B2:4C076:5D8D294C" response.header.x-request-id=c01dc0e9-5dc5-4818-b0a0-c9a42915e9d3 response.header.x-runtime=0.007552 response.header.x-runtime-rack=0.018101 response.header.x-xss-protection="1; mode=block" response.status="406 Not Acceptable" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
ctr: failed to resolve reference "docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64": unexpected status code [manifests v0.9.0-rc.0-25-g1aa27c95_linux_amd64]: 406 Not Acceptable
We would like to see this issue resolved. This is blocking us from using Github registry.
It appears the 406 is a result of the Accept adding *, when I tried */* the registry responded fine. We'll investigate this on our side as well.
Thanks for the quick fix @dmcgowan ! Any idea when I can try it?
@tamalsaha you can try it on master, it will also be part of 1.3.0. I am not closing this since I don't know if all the remainder of the issues have been figured out around pull by digest. Once we verify everything is working we can close this.
Thanks to everyone working getting this addressed
Thanks @dmcgowan . Can this be backported to 1.2.x ?
@dmcgowan , the 406 issue is fixed but now there is a new issue. I built from master and this is what I got.
root@ubuntu-s-1vcpu-1gb-sfo2-01:~# ./ctr version
Client:
Version: v1.3.0-rc.3-14-ga668365e
Revision: a668365eca1ddcdc3046e04d8a6421f956ddefc7
Server:
Version: v1.3.0-rc.3-14-ga668365e
Revision: a668365eca1ddcdc3046e04d8a6421f956ddefc7
UUID: e6f60ed5-dabd-42b7-a809-cfaa70d68b38
root@ubuntu-s-1vcpu-1gb-sfo2-01:~# ./ctr --debug images pull --user stashed:****** docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64
DEBU[0000] fetching image="docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] resolving host=docker.pkg.github.com
DEBU[0000] do request host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=HEAD url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] fetch response received host=docker.pkg.github.com response.header.cache-control=no-cache response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:22 GMT" response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:22 -0000" response.header.status="401 Unauthorized" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.www-authenticate="Basic realm=\"GitHub Docker Registry\"" response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE78:3E15:4290:372C3:5D8D55C5" response.header.x-request-id=6eb2c9bf-6b62-4a78-93d9-c11edd61c11d response.header.x-runtime=0.004865 response.header.x-runtime-rack=0.012802 response.header.x-xss-protection="1; mode=block" response.status="401 Unauthorized" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] Unauthorized header="Basic realm=\"GitHub Docker Registry\"" host=docker.pkg.github.com
DEBU[0000] do request host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=HEAD url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] fetch response received host=docker.pkg.github.com response.header.cache-control="max-age=0, private, must-revalidate" response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/vnd.docker.distribution.manifest.v2+json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:22 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="W/\"3e803b76cf7154c538fe10333409779b\"" response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:22 -0000" response.header.status="200 OK" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE78:3E15:4291:372C4:5D8D55C6" response.header.x-github-user=1gtm response.header.x-request-id=98c91ee8-03f1-4bcd-aa73-3989f5d195cd response.header.x-runtime=0.334852 response.header.x-runtime-rack=0.374636 response.header.x-xss-protection="1; mode=block" response.status="200 OK" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] no Docker-Content-Digest header, fetching manifest instead host=docker.pkg.github.com
DEBU[0000] do request host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=GET url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0001] fetch response received host=docker.pkg.github.com response.header.cache-control="max-age=0, private, must-revalidate" response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/vnd.docker.distribution.manifest.v2+json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:22 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="W/\"3e803b76cf7154c538fe10333409779b\"" response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:22 -0000" response.header.status="200 OK" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE78:3E15:4292:372C6:5D8D55C6" response.header.x-github-user=1gtm response.header.x-request-id=11499acf-a7c6-45c2-aa25-2200eb0798af response.header.x-runtime=0.090834 response.header.x-runtime-rack=0.114095 response.header.x-xss-protection="1; mode=block" response.status="200 OK" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0001] resolved desc.digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" host=docker.pkg.github.com
DEBU[0001] fetch digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1373
DEBU[0001] do request digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=GET size=1373 url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7"
DEBU[0001] fetch response received digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:22 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:22 -0000" response.header.status="404 Not Found" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE78:3E15:4293:372C8:5D8D55C6" response.header.x-github-user=1gtm response.header.x-request-id=6a772f9f-fe3a-4781-a711-dd9fb6117766 response.header.x-runtime=0.041982 response.header.x-runtime-rack=0.049381 response.header.x-xss-protection="1; mode=block" response.status="404 Not Found" size=1373 url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7"
DEBU[0001] do request digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=GET size=1373 url="https://docker.pkg.github.com/v2/stashed/stash/stash/blobs/sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7"
DEBU[0001] fetch response received digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:23 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:23 -0000" response.header.status="404 Not Found" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE7A:2065:56B2:39BCD:5D8D55C7" response.header.x-github-user=1gtm response.header.x-request-id=c8bcc730-f8eb-457c-bc1a-0c2a59522173 response.header.x-runtime=0.042270 response.header.x-runtime-rack=0.053240 response.header.x-xss-protection="1; mode=block" response.status="404 Not Found" size=1373 url="https://docker.pkg.github.com/v2/stashed/stash/stash/blobs/sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7"
ctr: failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7 (application/vnd.docker.distribution.manifest.v2+json) from remote: not found
$ docker pull docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64
v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64: Pulling from stashed/stash/stash
092586df9206: Already exists
6b8bc148141c: Pull complete
6ceec8dc461a: Pull complete
53b0d3a7e6a2: Pull complete
bf8fa31a6e81: Pull complete
Digest: sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7
Status: Downloaded newer image for docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64
docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64
@tamalsaha This is the same issue that was reported earlier and a registry side fix. @Phanatic for an update.
I don't think I've seen this error message in the thread, has anyone else seen it?
ctr: failed to resolve reference "[...]": could not resolve digest for [...]
I get it on the latest 'container-optimized' GKE version, and the image pulls fine from docker from the same box. containerd://1.2.8
$ ctr --debug images pull --user danopia:redacted docker.pkg.github.com/danopia/reggie/reggie:097dbb9
DEBU[0000] fetching image="docker.pkg.github.com/danopia/reggie/reggie:097dbb9"
DEBU[0000] resolving
DEBU[0000] do request request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://docker.pkg.github.com/v2/danopia/reggie/reggie/manifests/097dbb9"
DEBU[0000] fetch response received response.headers=map[X-Github-Request-Id:[A0AA:3829:6C64:4CE79:5DA69C61] Content-Type:[application/json] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Xss-Protection:[1; mode=block] Date:[Wed, 16 Oct 2019 04:28:17 GMT] Content-Security-Policy:[default-src 'none';] Server:[GitHub Registry] Strict-Transport-Security:[max-age=31536000;] Content-Length:[52]] status="405 Method Not Allowed" url="https://docker.pkg.github.com/v2/danopia/reggie/reggie/manifests/097dbb9"
DEBU[0000] do request request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/1.2.8]] request.method=GET url="https://docker.pkg.github.com/v2/danopia/reggie/reggie/manifests/097dbb9"
DEBU[0000] fetch response received response.headers=map[Www-Authenticate:[Basic realm="GitHub Package Registry"] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Xss-Protection:[1; mode=block] Date:[Wed, 16 Oct 2019 04:28:17 GMT] Content-Security-Policy:[default-src 'none';] Strict-Transport-Security:[max-age=31536000;] Content-Type:[text/plain; charset=utf-8] X-Github-Request-Id:[A0AA:3829:6C65:4CE7A:5DA69C61] Server:[GitHub Registry] Content-Length:[84]] status="401 Unauthorized" url="https://docker.pkg.github.com/v2/danopia/reggie/reggie/manifests/097dbb9"
DEBU[0000] Unauthorized header="Basic realm="GitHub Package Registry""
DEBU[0000] do request request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/1.2.8]] request.method=GET url="https://docker.pkg.github.com/v2/danopia/reggie/reggie/manifests/097dbb9"
DEBU[0000] fetch response received response.headers=map[Content-Security-Policy:[default-src 'none';] Docker-Distribution-Api-Version:[registry/2.0] Server:[GitHub Registry] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Xss-Protection:[1; mode=block] Date:[Wed, 16 Oct 2019 04:28:17 GMT] Content-Length:[1160] Content-Type:[application/vnd.docker.distribution.manifest.v2+json] Strict-Transport-Security:[max-age=31536000;] X-Github-Request-Id:[A0AE:3D3A:E831:A15B9:5DA69C61]] status="200 OK" url="https://docker.pkg.github.com/v2/danopia/reggie/reggie/manifests/097dbb9"
ctr: failed to resolve reference "docker.pkg.github.com/danopia/reggie/reggie:097dbb9": could not resolve digest for docker.pkg.github.com/danopia/reggie/reggie:097dbb9
I'm currently trying to pull an image from a private github registry as well, I'm getting this error:
ctr: failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:16a875db2e95a197a847dac777362417c977a2a8b347eee8366fdfae71ee248d (application/vnd.docker.distribution.manifest.v2+json) from remote: not found
docker pull seems to work without any isses
I got an email today saying that Github package registry is public. Is it working for anyone with containerd?
Edit: I am still getting from remote: not found
ctr: failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7 (application/vnd.docker.distribution.manifest.v2+json) from remote: not found
I get the same issues with a custom hosted registry (version 2+).
Looks like I have the same/similar issue (using drone cd pipeline in a k3s cluster):
Warning Failed 43s (x4 over 2m20s) kubelet, k3s2 Failed to pull image "docker.pkg.github.com/vasiliys/drone/kubectl:v1.6.2": rpc error: code = NotFound desc = failed to pull and unpack image "docker.pkg.github.com/vasiliys/drone/kubectl:v1.6.2": failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:e43914fa8c69b3638f421842accf3e150471b1fe2e1f002ae41d072e25899d4e (application/vnd.docker.distribution.manifest.v2+json) from remote: not found
I am seeing the same issue with bintray registry:
ctr --debug images pull --user username docker-xxx.bintray.io/testimage:1.0.0
DEBU[2019-11-06T19:00:16.221444239Z] fetching image="docker-xxx.bintray.io/testimage:1.0.0"
DEBU[2019-11-06T19:00:16.221597472Z] resolving
DEBU[2019-11-06T19:00:16.221619377Z] do request request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD
url="https://docker-xxx.bintray.io/v2/testimage/manifests/1.0.0"
DEBU[2019-11-06T19:00:16.501325470Z] fetch response received response.headers="map[Connection:[keep-alive]
Content-Length:[73]
Content-Type:[application/json]
Date:[Wed, 06 Nov 2019 19:00:17 GMT]
Server:[nginx]]"
status="401 Unauthorized"
url="https://docker-xxx.bintray.io/v2/testimage/manifests/1.0.0"
DEBU[2019-11-06T19:00:16.501408964Z] Unauthorized
header=
ctr: failed to resolve reference "docker-xxx.bintray.io/testimage:1.0.0": unexpected status code https://docker-xxx.bintray.io/v2/testimage/manifests/1.0.0: 401 Unauthorized
@ArpithaDR , you are getting unauthorized error. You probably need to pass token/password.
ctr --debug images pull --user username:token docker-xxx.bintray.io/testimage:1.0.0
@ArpithaDR , you are getting unauthorized error. You probably need to pass token/password.
ctr --debug images pull --user username:token docker-xxx.bintray.io/testimage:1.0.0
@tamalsaha , I am passing the password too when prompted.
Had also tried passing the password directly through CLI, both didn't work.
ctr --debug images pull --user username:password docker-xxx.bintray.io/testimage:1.0.0
@ArpithaDR that is a separate issue and a different registry provider. JFrog's registry has a different set of issues that they also need to fix upstream, maybe related to #3556
Like I said, I am also hitting this problem but with https://hub.docker.com/_/registry + https://github.com/cesanta/docker_auth
edit:
This seems to be related to this issue. cesanta/docker_auth#265
@Phanatic / @dmcgowan
Is this supposed to work with public package repositories? I can't seem to pull a package that I built from a forked upstream repo (but recompiled for ARM).
My fork is public and the package is seemingly public as well.
root@k3s-master-01:~# k describe pod podinfo-645899b78f-s52vg
[...snip...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10s default-scheduler Successfully assigned default/podinfo-645899b78f-s52vg to k3s-node-01
Normal Pulling 9s kubelet, k3s-node-01 Pulling image "docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5"
Warning Failed 9s kubelet, k3s-node-01 Failed to pull image "docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5": rpc error: code = Unknown desc = failed to resolve image "docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5": no available registry endpoint: unexpected status code https://docker.pkg.github.com/v2/stgarf/podinfo/podinfo/manifests/3.1.5: 401 Unauthorized
Warning Failed 9s kubelet, k3s-node-01 Error: ErrImagePull
Normal BackOff 8s kubelet, k3s-node-01 Back-off pulling image "docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5"
Warning Failed 8s kubelet, k3s-node-01 Error: ImagePullBackOffError:
Failed to pull image "docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5": rpc error: code = Unknown desc = failed to resolve image "docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5": no available registry endpoint: unexpected status code https://docker.pkg.github.com/v2/stgarf/podinfo/podinfo/manifests/3.1.5: 401 Unauthorized
I'm running a k3s Kubernetes cluster v0.9.1 (755bd1c6) based on containerd runtime v1.2.8-k3s.1. ...depending on how you invoke crictl you get a different version. So maybe I'm running v1.15.0-k3s.2?
root@k3s-master-01:~# crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: v1.2.8-k3s.1
RuntimeApiVersion: v1alpha2
root@k3s-master-01:~# crictl --version
crictl github.com/rancher/containerd v1.15.0-k3s.2
root@k3s-master-01:~# ctr version
Client:
Version: v1.2.8-k3s.1
Revision:
Server:
Version: v1.2.8-k3s.1
Revision:
root@k3s-master-01:~# k3s -v
k3s version v0.9.1 (755bd1c6)ctr pull debug logs:
root@k3s-master-01:~# export GITHUB_TOKEN=redacted
root@k3s-master-01:~# ctr --debug images pull --user stgarf:$GITHUB_TOKEN docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5
DEBU[2019-11-13T23:22:34.694913500-08:00] fetching image="docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5"
DEBU[2019-11-13T23:22:34.695349743-08:00] resolving
DEBU[2019-11-13T23:22:34.695444637-08:00] do request request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD url="https://docker.pkg.github.com/v2/stgarf/podinfo/podinfo/manifests/3.1.5"
DEBU[2019-11-13T23:22:35.643191279-08:00] fetch response received response.headers="map[Content-Length:[52] Content-Security-Policy:[default-src 'none';] Content-Type:[application/json] Date:[Thu, 14 Nov 2019 07:22:35 GMT] Server:[GitHub Registry] Strict-Transport-Security:[max-age=31536000;] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Github-Request-Id:[E15A:68F3:1F91:121C5:5DCD00BA] X-Xss-Protection:[1; mode=block]]" status="405 Method Not Allowed" url="https://docker.pkg.github.com/v2/stgarf/podinfo/podinfo/manifests/3.1.5"
DEBU[2019-11-13T23:22:35.643568096-08:00] do request request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/v1.2.8-k3s.1]]" request.method=GET url="https://docker.pkg.github.com/v2/stgarf/podinfo/podinfo/manifests/3.1.5"
DEBU[2019-11-13T23:22:35.735864042-08:00] fetch response received response.headers="map[Content-Length:[84] Content-Security-Policy:[default-src 'none';] Content-Type:[text/plain; charset=utf-8] Date:[Thu, 14 Nov 2019 07:22:35 GMT] Server:[GitHub Registry] Strict-Transport-Security:[max-age=31536000;] Www-Authenticate:[Basic realm=\"GitHub Package Registry\"] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Github-Request-Id:[E15A:68F3:1F92:121C7:5DCD00BB] X-Xss-Protection:[1; mode=block]]" status="401 Unauthorized" url="https://docker.pkg.github.com/v2/stgarf/podinfo/podinfo/manifests/3.1.5"
DEBU[2019-11-13T23:22:35.736281795-08:00] Unauthorized header="Basic realm=\"GitHub Package Registry\""
DEBU[2019-11-13T23:22:35.736702205-08:00] do request request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/v1.2.8-k3s.1]]" request.method=GET url="https://docker.pkg.github.com/v2/stgarf/podinfo/podinfo/manifests/3.1.5"
DEBU[2019-11-13T23:22:36.080153452-08:00] fetch response received response.headers="map[Content-Length:[1580] Content-Security-Policy:[default-src 'none';] Content-Type:[application/vnd.docker.distribution.manifest.v2+json] Date:[Thu, 14 Nov 2019 07:22:36 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[GitHub Registry] Strict-Transport-Security:[max-age=31536000;] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Github-Request-Id:[E160:3BB9:2287:1335B:5DCD00BB] X-Xss-Protection:[1; mode=block]]" status="200 OK" url="https://docker.pkg.github.com/v2/stgarf/podinfo/podinfo/manifests/3.1.5"
ctr: failed to resolve reference "docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5": could not resolve digest for docker.pkg.github.com/stgarf/podinfo/podinfo:3.1.5
Any workarounds at containerd / k8s side?
I believe I see this error as well:
$ ctr --debug images pull docker.pkg.github.com/owner/repo/image:tag
DEBU[0000] fetching image="docker.pkg.github.com/owner/repo/image:tag"
DEBU[0000] resolving
DEBU[0000] do request request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[0000] fetch response received response.headers=map[Content-Length:[52] Content-Type:[application/json] Date:[Mon, 02 Dec 2019 14:06:03 GMT] X-Github-Request-Id:[DF8A:674C:1C4B:1D46C:5DE51A4B]] status="405 Method Not Allowed" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[0000] do request request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/1.2.10-0ubuntu1]] request.method=GET url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[0000] fetch response received response.headers=map[Content-Length:[84] Content-Security-Policy:[default-src 'none';] Content-Type:[text/plain; charset=utf-8] Date:[Mon, 02 Dec 2019 14:06:03 GMT] Server:[GitHub Registry] Strict-Transport-Security:[max-age=31536000;] Www-Authenticate:[Basic realm="GitHub Package Registry"] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Github-Request-Id:[DF8A:674C:1C4C:1D46D:5DE51A4B] X-Xss-Protection:[1; mode=block]] status="401 Unauthorized" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[0000] Unauthorized header="Basic realm="GitHub Package Registry""
ctr: failed to resolve reference "docker.pkg.github.com/owner/repo/image:tag": unexpected status code https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag: 401 UnauthorizedWhen I add the --user flag the error is different, yet still present:
$ k3s ctr --debug images pull --user riker09:$GITHUB_TOKEN docker.pkg.github.com/owner/repo/image:tag
DEBU[2019-12-02T15:29:00.411982181+01:00] fetching image="docker.pkg.github.com/owner/repo/image:tag"
DEBU[2019-12-02T15:29:00.414453549+01:00] resolving host=docker.pkg.github.com
DEBU[2019-12-02T15:29:00.414504221+01:00] do request host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=HEAD url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:00.824424265+01:00] fetch response received host=docker.pkg.github.com response.header.content-length=52 response.header.content-type=application/json response.header.date="Mon, 02 Dec 2019 14:29:00 GMT" response.header.x-github-request-id="B478:1FE2:16731:10F61A:5DE51FAC" response.status="405 Method Not Allowed" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:00.824488827+01:00] do request host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:01.128049683+01:00] fetch response received host=docker.pkg.github.com response.header.content-length=84 response.header.content-security-policy="default-src 'none';" response.header.content-type="text/plain; charset=utf-8" response.header.date="Mon, 02 Dec 2019 14:29:00 GMT" response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.www-authenticate="Basic realm=\"GitHub Package Registry\"" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B478:1FE2:16732:10F61B:5DE51FAC" response.header.x-xss-protection="1; mode=block" response.status="401 Unauthorized" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:01.128144926+01:00] Unauthorized header="Basic realm=\"GitHub Package Registry\"" host=docker.pkg.github.com
DEBU[2019-12-02T15:29:01.128256245+01:00] do request host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:01.570887786+01:00] fetch response received host=docker.pkg.github.com response.header.content-length=2000 response.header.content-security-policy="default-src 'none';" response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 02 Dec 2019 14:29:01 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B48A:3EF6:D341:A5D71:5DE51FAD" response.header.x-xss-protection="1; mode=block" response.status="200 OK" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:01.570999503+01:00] no Docker-Content-Digest header, fetching manifest instead host=docker.pkg.github.com
DEBU[2019-12-02T15:29:01.571038435+01:00] do request host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:02.152267193+01:00] fetch response received host=docker.pkg.github.com response.header.content-length=2000 response.header.content-security-policy="default-src 'none';" response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 02 Dec 2019 14:29:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B494:36BD:B6ED:94D55:5DE51FAD" response.header.x-xss-protection="1; mode=block" response.status="200 OK" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:02.152555755+01:00] resolved desc.digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" host=docker.pkg.github.com
DEBU[2019-12-02T15:29:02.152779654+01:00] fetch digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json size=2000
DEBU[2019-12-02T15:29:02.163557378+01:00] do request digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET size=2000 url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171"
DEBU[2019-12-02T15:29:02.423060126+01:00] fetch response received digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=206 response.header.content-security-policy="default-src 'none';" response.header.content-type="text/plain; charset=utf-8" response.header.date="Mon, 02 Dec 2019 14:29:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B494:36BD:B6EE:94D57:5DE51FAE" response.header.x-xss-protection="1; mode=block" response.status="404 Not Found" size=2000 url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171"
DEBU[2019-12-02T15:29:02.423468150+01:00] do request digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET size=2000 url="https://docker.pkg.github.com/v2/owner/repo/image/blobs/sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171"
DEBU[2019-12-02T15:29:02.971197798+01:00] fetch response received digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=209 response.header.content-security-policy="default-src 'none';" response.header.content-type="text/plain; charset=utf-8" response.header.date="Mon, 02 Dec 2019 14:29:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B4A4:16DD:E38E:A3798:5DE51FAE" response.header.x-xss-protection="1; mode=block" response.status="404 Not Found" size=2000 url="https://docker.pkg.github.com/v2/owner/repo/image/blobs/sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171"
ctr: failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171 (application/vnd.docker.distribution.manifest.v2+json) from remote: not foundPlease note: I have replaced the real image name with
owner/repo/image:tag.
Any known workarounds?
Im having the same issues and tried passing --auth and --creds:
$ crictl -D pull --auth 'somebase64hash' registry.domain.tld/image:latest DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:registry.domain.tld/image:latest,},Auth:&AuthConfig{Username:,Password:,Auth:somebase64hash,ServerAddress:,IdentityToken:,RegistryToken:,},SandboxConfig:nil,} DEBU[0000] PullImageResponse: nil FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.domain.tld/image:latest": failed to resolve reference "registry.domain.tld/image:latest": failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error
$ crictl -D pull --creds 'user:password' registry.domain.tld/image:latest DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:registry.domain.tld/image:latest,},Auth:&AuthConfig{Username:user,Password:password,Auth:,ServerAddress:,IdentityToken:,RegistryToken:,},SandboxConfig:nil,} DEBU[0000] PullImageResponse: nil FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.domain.tld/image:latest": failed to resolve reference "registry.domain.tld/image:latest": failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error
I try to setup a repro that I can use to debug this further
@Phanatic did you have any luck reproducing it inside a github's lab? I have the same results as #3291 (comment) when trying to use github registry from k3s:
$ sudo k3s --version
k3s version v1.17.0-rc.3-k3s.1 (5c374547)
$ sudo k3s ctr --version
ctr github.com/rancher/containerd v1.3.0-k3s.5
@narqo the way I solved was setting up a docker registry in front of the main registry acting as a proxy. Now I can pull/images using containerd.
For my testing use cases, I have "solved" it by moving the images from a private github registry to the one provided by gitlab. But I'm curious to know if GitHub folks are able to solve this for real
I spent far too long on this to then discover this issue - having the same problem with the GitHub registry;
Events:
Type Reason Age From Message
Normal Scheduled default-scheduler Successfully assigned openfaas-fn/profiles-follow-777c95599f-z94q7 to k3d-k3s-default-server
Normal Pulling 20s (x3 over 75s) kubelet, k3d-k3s-default-server Pulling image "docker.pkg.github.com/xxxx/xxx/profiles-follow:0.1-develop-70cbaa1"
Warning Failed 15s (x3 over 71s) kubelet, k3d-k3s-default-server Failed to pull image "docker.pkg.github.com/xxxx/xxx/profiles-follow:0.1-develop-70cbaa1": rpc error: code = NotFound desc = failed to pull and unpack image "docker.pkg.github.com/xxxx/xxx/profiles-follow:0.1-develop-70cbaa1": failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:xxxxxx (application/vnd.docker.distribution.manifest.v2+json) from remote: not found
Warning Failed 15s (x3 over 71s) kubelet, k3d-k3s-default-server Error: ErrImagePull
Warning Failed 2s (x3 over 70s) kubelet, k3d-k3s-default-server Error: ImagePullBackOff
Normal BackOff 2s (x3 over 70s) kubelet, k3d-k3s-default-server Back-off pulling image "docker.pkg.github.com/xxxx/xxx/profiles-follow:0.1-develop-70cbaa1"
Moving it into another repo (e.g. GCR) and it all works fine. Also running docker pull on the image is fine too.
@clarkbw wondering if there's any progress on GitHub's side on this issue?
I spoke with a PM from GH, seems the issue is on their backlog but it is the number one issue they are seeing with their registry implementation.
@cpuguy83 How long has this been on the backlog? I'm concerned that this is a breaking bug, opened almost a year ago and we're still discussing this. It's putting me off a little to "commit" to GH.
Is there an ETA on this?
@andrew-s You have all the info I have. :(
Just to be clear, is this issue being classed as a problem with GitHub's registry, rather than with containerd? Is there somewhere upstream that we can track it and link back to? Anybody have friends with GitHub engineering?
@pathcl We outlined the problem very clearly with what is not working with the Github registry, please read above before commenting. They are not implementing the distribution specification, so clients are having trouble.
I was pointed to support@github.com, if someone wants to open a ticket?
Containerd doesn't work with DTR? Can you log a separate issue with repro?
Seeing this error with IBM IKS, containerd, and the docker registry deployed within IKS.
The issue is that the docker registry was backed by an external auth provider. Containerd will not send authentication until it receives a 401 with the method that should be used.
So in the auth server, when authorization header is not present, you need to return a 401 with a header
if (!authorization) {
return res
.status(401)
.set("WWW-Authenticate", "Bearer")
.end();
}
I don't understand the details of the specification here. Just from layman's perspective, all these other docker registry's work with the docker command. Why can't containerd just do that docker is doing? I thought the recent Docker releases just uses containerd under the hood.
@tamalsaha it's possible containerd is trying to be more open by allowing the user to determine whether it should create a bearer or basic token. In practice it's just a lost day of engineering productivity waiting to happen.
@clarkbw wondering if there's any progress on GitHub's side on this issue?
👋 Ability to pull images by digest is in our backlog for this quarter. Apologies for the delay.
stay tuned
I have the same problem in Oracle container registry.
@clarkbw do you have any details on planning?
@clarkbw do you have any details on planning?
Yes, we're actively working on a solution. Give my team until the end of April and we'll have something for you all to test.
Thanks for checking in!
Dates slipped a little due to extenuating circumstances.
I’ll send a message to the maintainer group in mid May with the details for access and feedback.
Should be a public announcement by end of May.
Is there at least some workaround to make it work at the moment?
@clarkbw any news? I really think you're underestimating the seriousness of this issue. Simply nobody who's using Kubernetes with containerd (which is an uncontrollable combination in many cases) can use Github container registry.
It actually affects Docker Swarm as well (docker stack deploy doesn't like Github registry images at the moment)
Same problem with buildkit and cache-from, I haven't check it without the buildkit.
We'll be opening up to the maintainers group very soon and then we'll have a Beta to share more broadly. Sorry for the delays, I appreciate your patience here.
@clarkbw any news when this will be fixed?
I had this problem and spent 3 days on it, finally stumbled upon this thread that it was is an issue on GitHub's end.
Sorry you got stuck. We are working on a solution and will have news soon.
For anybody having issues with running outdated images with docker swarm, a workaround for me was the Following:
Instead of using a moving tag like :latest, tagging images with the commit sha instead, so for example :latest-5716e43 and then pulling that image. As there is only one image that has the tag I don't encounter any issues of outdated images on different nodes.
@clarkbw would still be nice to see this asap as the github package registry is pretty useless for Docker if you can't access images by sha
@jaschaio I tried it with GitHub sha, but it still doesn't work, once the stack deploy has run, it doesn't change and gives the same error. Am I missing something here:
I am using it like this
version: "3.3"
services:
name:
image: docker.pkg.github.com/user/repo/image_name:${GITHUB_SHA}
PS: I checked, the containers are getting deployed, but the error message is still there. Thanks for your help! Cheers!
A tip here, you can use a Github action which lets you delete docker images from the registry, with the combination of that and this approach I able to deploy and not have a bunch of images lying around.
stack deploy pins to the digest of the image (There may be an option to disable pinning, I can't remembesr).
The pinning is the problem here because the GH registry doesn't support fetch by digest.
@gurleensethi yes as I said, its a workaround not a solution. Error messages is still being displayed. But at least it works. What are you using to clean up outdated images from the registry with Github Actions?
@jaschaio Take a look at this Delete package version action. It is from GitHub itself.
@clarkbw is there any update from the folks at GH?
This is still a serious issue.....
We are beginning a private beta this week. Email me, clarkbw@github.com to gain access. We had planned to be in a public beta by this point but 2020, oh she had other plans.
Here's a nasty little workaround for thoses who:
- Don't mind loosing blue/green deploys until this is resolved
- Don't mind 10-15 secs app start-up time
- Use docker swarm / docker stack deploys
- Use CI scripts for deployment
In your CI scripts call:
$ docker stack rm {{ your_stack_name }}
$ until [ -z $(docker stack ps {{ your_stack_name }} -q) ]; do sleep 1; done
$ docker stack deploy --with-registry-auth -c docker-compose.yml {{ your_stack_name }}
Basically you ask Docker scheduler to stop all the services under {{ your_stack_name }} orchestrator. A little knack of docker swarm is that docker stack rm will immediately return even if some services are not properly closed chich may cause networking errors when you try to deploy again. That's why we use a small inline script until [ -z $(docker stack ps {{ your_stack_name }} -q) ]; do sleep 1; done to wait for the proper return.
Hopes it saves a few folks headaches. I guess a similar temporary fix will help you out.
This is quite a frustrating issue, for our apps that MUST use blue/green deploys we bought a private repo to fix the problem.
Hi @clarkbw Im still keeping an eye on this issue...just was wondering if is there any ETA for the fix?
We are beginning a private beta this week. Email me, clarkbw@github.com to gain access. We had planned to be in a public beta by this point but 2020, oh she had other plans.
The private beta is rolling out to a number of users. This is a phased rollout over time so you may only get your instructions over the next couple weeks. Thanks!
Same problem with buildkit and cache-from
Can also confirm this is the case for me, when running a docker command like
docker buildx build --progress plain --load --cache-from docker.pkg.github.com/mvgijssel/setup/base-image:refs_heads_feature_mediacenter-proxmox --cache-from docker.pkg.github.com/mvgijssel/setup/base-image:latest --cache-to=type=inline,mode=all --build-arg IMAGE_SHA_TAG=368dbd16ec4e450e1f8507bfb43fad1d9a91e26d --build-arg DOCKER_REGISTRY=docker.pkg.github.com/mvgijssel/setup --tag docker.pkg.github.com/mvgijssel/setup/base-image:refs_heads_feature_mediacenter-proxmox --tag docker.pkg.github.com/mvgijssel/setup/base-image:368dbd16ec4e450e1f8507bfb43fad1d9a91e26d --file ./base-image/Dockerfile ./base-imageit results into the following error in the logs:
#4 importing cache manifest from docker.pkg.github.com/mvgijssel/setup/base...
#4 ERROR: httpReaderSeeker: failed open: content at https://docker.pkg.github.com/v2/mvgijssel/setup/base-image/manifests/sha256:975bc337767ee38460ec75aa8a6db06114d9dc19a5d82d87a4fde681f7c27eec not found: not found
even thought the docker image with the associated tag actually exists on the GitHub registry and running the image works just fine:
docker run -it docker.pkg.github.com/mvgijssel/setup/base-image:refs_heads_feature_mediacenter-proxmox /bin/bash
I can also confirm this is an issue for us. We are using AWS Fargate 1.4 which uses containerd and we cannot pull images from GPR, and AWS Technical Support pointed us to this issue.
Sorry to be annoying, but do you have any update to share @clarkbw as it has been a couple of weeks since the last update. Getting tripped up on this again after coming back hoping this would've been resolved by now.
This is absurd.
I ❤️ GitHub but this really makes me question the integrity and professionalism of this service.
I'm currently paying for a product that doesn't even work. It's been a month since the private beta and literally nothing notably has happened. This is a serious issue......
Probably moving to Docker Hub this week... 💔
I have also moved to Dockerhub, but I have been wanting to try out digital ocean package registry, I think it's nice to have the registry on the same "app" than the production artifact.
I ❤️ GitHub but this really makes me question the integrity and professionalism of this service.
Right, thanks; this work is being done by humans with integrity and professionalism. I'm sorry we've let you down.
We are rolling out the feature flag to a number of people. I'm sorry if you've emailed me and haven't gotten into the beta yet. Still more coming this week.
I ❤️ GitHub but this really makes me question the integrity and professionalism of this service.
Right, thanks; this work is being done by humans with integrity and professionalism. I'm sorry we've let you down.
We are rolling out the feature flag to a number of people. I'm sorry if you've emailed me and haven't gotten into the beta yet. Still more coming this week.
Awesome; thank you @clarkbw for your personal involvement and supervision on this issue.
I'm sure everyone here appreciates that ❤️
@clarkbw sorry for hijacking the thread, but this is kind of related to this issue; are public GH package (docker) registries still on the radar?
I'm running into the same issue pulling images from a secured bintray repository. I'm using docker registry secrets in k8s, and the config works in docker for mac/windows however fails with microk8s using containerd:
Warning Failed 6s (x2 over 20s) kubelet, username-virtualbox Failed to pull image "host.bintray.io/my-image:1.0.0.0-alpine": rpc error: code = Unknown desc = failed to pull and unpack image "host.bintray.io/my-image:1.0.0.0-alpine": failed to copy: httpReaderSeeker: failed open: unexpected status code https://host.bintray.io/v2/my-image/manifests/sha256:XXXXX: 401 Unauthorized - Server message: unauthorized: Unauthorized
Following the thread in this issue I can't figure out is the problem with the container registry or is it with containerd?
The issue of direct sha access is with our current Docker offering. I've connected nearly everyone who reached out from this thread to the fix we have running. Please reach out to me clarkbw@github.com and I can get you setup as well. Hopefully soon we'll be able to have a public offering here.
I'm getting the following error when trying to pull with miicrok8s.
Failed to pull image "docker.pkg.github.com/resplendent-data/front-end/frontend": rpc error: code = NotFound desc = failed to pull and unpack image "docker.pkg.github.com/resplendent-data/front-end/frontend:latest": failed to copy: httpReaderSeeker: failed open: content at https://docker.pkg.github.com/v2/resplendent-data/front-end/frontend/manifests/sha256:XXXX not found: not found
Doing a docker pull works fine.
Hello, can anyone provide feedback on the private beta?
Does it involve some workaround / changing anything on your side? Or will it just fix your/your organization's Github and work as expected?
Thank you.
@clarkbw I'm still waiting fort the invite 🙏
Hello, can anyone provide feedback on the private beta?
Does it involve some workaround / changing anything on your side? Or will it just fix your/your organization's Github and work as expected?
Thank you.@clarkbw I'm still waiting fort the invite 🙏
It has been great for me! I had to change references to the new location, and get my kubernetes regcred working, but the beta itself is working smoothly.