Error reading manifest 1.0.0-SNAPSHOT-2020-03-12-18-56-25 in quay.io/open-cluster-management/multiclusterhub-operator-index: unauthorized: access to the requested resource is not authorized

Question

Error reading manifest 1.0.0-SNAPSHOT-2020-03-12-18-56-25 in quay.io/open-cluster-management/multiclusterhub-operator-index: unauthorized: access to the requested resource is not authorized

achuzhoy opened this issue 4 years ago · comments

Version: 4.4.0-0.nightly-2020-03-15-192859

open-cluster-management-registry pod is "ErrImagePull" status.

Trying to run the steps in https://github.com/open-cluster-management/deploy
Stuck on step #6.

[kni@provisionhost-0 multiclusterhub-operator]$ oc get pod -A|grep -v Run|grep -v Comple
NAMESPACE NAME READY STATUS RESTARTS AGE
open-cluster-management open-cluster-management-registry-7bf489f7fd-p2fzx 0/1 ErrImagePull 0 16s

oc describe -n open-cluster-management pod open-cluster-management-registry-7bf489f7fd-p2fzx
Name: open-cluster-management-registry-7bf489f7fd-p2fzx
Namespace: open-cluster-management
Priority: 0
Node: worker-0/192.168.123.126
Start Time: Mon, 16 Mar 2020 21:32:48 +0000
Labels: app=open-cluster-management-registry
pod-template-hash=7bf489f7fd
Annotations: k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.131.0.4"
],
"dns": {},
"default-route": [
"10.131.0.1"
]
}]
openshift.io/scc: restricted
Status: Pending
IP: 10.131.0.4
IPs:
IP: 10.131.0.4
Controlled By: ReplicaSet/open-cluster-management-registry-7bf489f7fd
Containers:
multiclusterhub-operator-index:
Container ID:
Image: quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25
Image ID:
Port: 50051/TCP
Host Port: 0/TCP
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-2zgxr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-2zgxr:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-2zgxr
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Normal Scheduled default-scheduler Successfully assigned open-cluster-management/open-cluster-management-registry-7bf489f7fd-p2fzx to worker-0
Normal BackOff 20s (x2 over 49s) kubelet, worker-0 Back-off pulling image "quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25"
Warning Failed 20s (x2 over 49s) kubelet, worker-0 Error: ImagePullBackOff
Normal Pulling 9s (x3 over 50s) kubelet, worker-0 Pulling image "quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25"
Warning Failed 8s (x3 over 50s) kubelet, worker-0 Failed to pull image "quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25": [rpc error: code = Unknown desc = Error reading manifest 1.0.0-SNAPSHOT-2020-03-12-18-56-25 in quay.io/open-cluster-management/multiclusterhub-operator-index: manifest unknown: manifest unknown, rpc error: code = Unknown desc = Error reading manifest 1.0.0-SNAPSHOT-2020-03-12-18-56-25 in quay.io/open-cluster-management/multiclusterhub-operator-index: unauthorized: access to the requested resource is not authorized]
Warning Failed 8s (x3 over 50s) kubelet, worker-0 Error: ErrImagePull

Tim Pouyer · Answer 1 · Tue Mar 17 2020 05:43:45 GMT+0800 (China Standard Time)

Does this pull secret exist?

https://github.com/open-cluster-management/deploy/blob/414d994397912c0993c96936b793115f2573b1e5/multiclusterhub-operator/deployment.yaml#L22

Alexander Chuzhoy · Answer 2 · Tue Mar 17 2020 05:49:16 GMT+0800 (China Standard Time)

[kni@provisionhost-0 multiclusterhub-operator]$ oc get secret -A|grep multiclust
open-cluster-management multiclusterhub-operator-pull-secret kubernetes.io/dockerconfigjson 1 17m
[kni@provisionhost-0 multiclusterhub-operator]$

Tim Pouyer · Answer 3 · Tue Mar 17 2020 06:15:59 GMT+0800 (China Standard Time)

is the pull-secret on the pod?

oc get pod open-cluster-management-registry-7bf489f7fd-p2fzx -o yaml

Can you pull the image from your terminal:

docker login quay.io
docker pull quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25

Alexander Chuzhoy · Answer 4 · Tue Mar 17 2020 08:08:22 GMT+0800 (China Standard Time)

Unable to pull the image with my user:
[kni@provisionhost-0 multiclusterhub-operator]$ podman login quay.io
Username: achuzhoy
Password:
Login Succeeded!

[kni@provisionhost-0 multiclusterhub-operator]$ podman pull quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25
Trying to pull quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25...ERRO[0000] Error pulling image ref //quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25: Error initializing source docker://quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25: Error reading manifest 1.0.0-SNAPSHOT-2020-03-12-18-56-25 in quay.io/open-cluster-management/multiclusterhub-operator-index: manifest unknown: manifest unknown

oc get pod open-cluster-management-registry-7bf489f7fd-8k8bf -o yaml -n open-cluster-management
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.v1.cni.cncf.io/networks-status: |-
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.131.0.6"
],
"dns": {},
"default-route": [
"10.131.0.1"
]
}]
openshift.io/scc: restricted
creationTimestamp: "2020-03-17T00:04:01Z"
generateName: open-cluster-management-registry-7bf489f7fd-
labels:
app: open-cluster-management-registry
pod-template-hash: 7bf489f7fd
name: open-cluster-management-registry-7bf489f7fd-8k8bf
namespace: open-cluster-management
ownerReferences:

apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: open-cluster-management-registry-7bf489f7fd
uid: 82f4629a-414b-4208-bc9d-c5f90ce86846
resourceVersion: "242219"
selfLink: /api/v1/namespaces/open-cluster-management/pods/open-cluster-management-registry-7bf489f7fd-8k8bf
uid: b211b793-2d2a-44e3-93e0-bfb43776301d
spec:
containers:
image: quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25
imagePullPolicy: IfNotPresent
name: multiclusterhub-operator-index
ports:
- containerPort: 50051
  protocol: TCP
  resources: {}
  securityContext:
  capabilities:
  drop:
  - KILL
  - MKNOD
  - SETGID
  - SETUID
    runAsUser: 1000660000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  name: default-token-m69fl
  readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
name: multiclusterhub-operator-pull-secret
nodeName: worker-0
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000660000
seLinuxOptions:
level: s0:c26,c5
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
name: default-token-m69fl
secret:
defaultMode: 420
secretName: default-token-m69fl
status:
conditions:
lastProbeTime: null
lastTransitionTime: "2020-03-17T00:04:01Z"
status: "True"
type: Initialized
lastProbeTime: null
lastTransitionTime: "2020-03-17T00:04:01Z"
message: 'containers with unready status: [multiclusterhub-operator-index]'
reason: ContainersNotReady
status: "False"
type: Ready
lastProbeTime: null
lastTransitionTime: "2020-03-17T00:04:01Z"
message: 'containers with unready status: [multiclusterhub-operator-index]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
lastProbeTime: null
lastTransitionTime: "2020-03-17T00:04:01Z"
status: "True"
type: PodScheduled
containerStatuses:
image: quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25
imageID: ""
lastState: {}
name: multiclusterhub-operator-index
ready: false
restartCount: 0
started: false
state:
waiting:
message: Back-off pulling image "quay.io/open-cluster-management/multiclusterhub-operator-index:1.0.0-SNAPSHOT-2020-03-12-18-56-25"
reason: ImagePullBackOff
hostIP: 192.168.123.126
phase: Pending
podIP: 10.131.0.6
podIPs:
ip: 10.131.0.6
qosClass: BestEffort
startTime: "2020-03-17T00:04:01Z"

Tim Pouyer · Answer 5 · Tue Mar 17 2020 20:00:54 GMT+0800 (China Standard Time)

That snapshot doesn't exist...

Pick a snapshot from the top of this page and try again: https://quay.io/repository/open-cluster-management/multiclusterhub-operator-index?tab=tags

Alexander Chuzhoy · Answer 6 · Wed Mar 18 2020 03:54:17 GMT+0800 (China Standard Time)

Thanks a lot.
Was able to pass this step with another snapshot referenced.