unauthorized: access to the requested resource is not authorized
achuzhoy opened this issue · comments
Attempted to follow the steps in https://github.com/open-cluster-management/deploy
After Step 6 checking the pods status:
open-cluster-management 15b2552bda98af9ca6f85d34f2bfb89e5e18d86f7c7267f112126cfd278g9hc 0/1 Init:ImagePullBackOff 0 21m
Checking what's wrong with that pod:
Failed to pull image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:ba919b34aa7c7c7135f4474791defb6240ff1af2491b876c25ba960af81c5267": rpc error: code = Unknown desc = Error reading manifest sha256:ba919b34aa7c7c7135f4474791defb6240ff1af2491b876c25ba960af81c5267 in quay.io/open-cluster-management/multicloudhub-operator-bundle: unauthorized: access to the requested resource is not authorized
Tried several users with access to quay. Same error.
You were in the qe
team on our quay org open-cluster-management
. That team did not have read permissions for the multicloudhub-operator-bundle
image nor the multicloudhub-operator-index
image... I've updated the permissions for those repos.
You can test this from your command line by seeing if you can successfully pull these two images using:
docker login quay.io
docker pull quay.io/open-cluster-management/multicloudhub-operator-index:1.0.0-SNAPSHOT-2020-03-10-15-49-00
docker pull quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:ba919b34aa7c7c7135f4474791defb6240ff1af2491b876c25ba960af81c5267
If you can pull those images then you should be good to go now
I'm able to pull the images upon login to quay.io.
Regenerated the key according to the guide to assure there's no error. Still fail on the same issue.
Normal Scheduled default-scheduler Successfully assigned open-cluster-management/3ed3478e5608f39e9a7f144fae67618bf4ed284b1561ed1bc28407ce858fqst to worker-1
Normal Pulled 23s kubelet, worker-1 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f1038e9a52b30deb7106c5ea6ed44f91493c0bdeffe32aebfd5eacc906550cb" already present on machine
Normal Created 22s kubelet, worker-1 Created container util
Normal Started 22s kubelet, worker-1 Started container util
Normal Pulling 22s kubelet, worker-1 Pulling image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830"
Warning Failed 21s kubelet, worker-1 Failed to pull image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830": rpc error: code = Unknown desc = Error reading manifest sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830 in quay.io/open-cluster-management/multicloudhub-operator-bundle: unauthorized: access to the requested resource is not authorized
Warning Failed 21s kubelet, worker-1 Error: ErrImagePull
Normal BackOff 21s kubelet, worker-1 Back-off pulling image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830"
Warning Failed 21s kubelet, worker-1 Error: ImagePullBackOff
I also got this on a 4.4 build. I thought I was just doing dumb PM stuff. Probably still am.
multicloudhub-operator/kustomization.yaml using tag:
newTag: 1.0.0-SNAPSHOT-2020-03-10-19-58-31
Failed to pull image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830": rpc error: code = Unknown desc = Error reading manifest sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830 in quay.io/open-cluster-management/multicloudhub-operator-bundle: unauthorized: access to the requested resource is not authorized
So you're credentials are working... I can see where your user achuzhoy
was able to pull the multicloudhub-operator-bundle
via it's sha
recently in the quay logs.
So there must be something wrong with the quay-secret.yaml
file... When you followed the instructions to generate a pull secret from quay did you save the file as quay-secret.yaml
in the multicloudhub-operator
directory?
Did you update the metadata.name
value in the quay-secret.yaml
file to use the name quay-secret
:
apiVersion: v1
kind: Secret
metadata:
name: quay-secret
...
Is the mulitcloudhub-operator-registry
deployment in a READY
state? Can you share the output of:
oc get deployment open-cluster-management-registry -n open-cluster-management -o yaml
and what about open-cluster-management
catalogsource? can you share the output of:
oc get catalogsource open-cluster-management -n open-cluster-management -o yaml
And can you share the content of your kustomization.yaml
file in the mulitcloudhub-operator
dir?
multicloudhub-operator/kustomization.yaml using tag:
newTag: 1.0.0-SNAPSHOT-2020-03-10-19-58-31
Failed to pull image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830": rpc error: code = Unknown desc = Error reading manifest sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830 in quay.io/open-cluster-management/multicloudhub-operator-bundle: unauthorized: access to the requested resource is not authorized
@berenss I added you to the dev
team in open-cluster-management
org on quay... please try again.
[kni@provisionhost-0 ~]$ oc get deployment open-cluster-management-registry -n open-cluster-management -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"open-cluster-management-registry"},"name":"open-cluster-management-registry","namespace":"open-cluster-management"},"spec":{"selector":{"matchLabels":{"app":"open-cluster-management-registry"}},"template":{"metadata":{"labels":{"app":"open-cluster-management-registry"}},"spec":{"containers":[{"image":"quay.io/open-cluster-management/multicloudhub-operator-index:1.0.0-SNAPSHOT-2020-03-10-19-58-31","name":"multicloudhub-operator-index","ports":[{"containerPort":50051}]}],"imagePullSecrets":[{"name":"quay-secret"}]}}}}
creationTimestamp: "2020-03-11T03:29:45Z"
generation: 1
labels:
app: open-cluster-management-registry
name: open-cluster-management-registry
namespace: open-cluster-management
resourceVersion: "2138493"
selfLink: /apis/apps/v1/namespaces/open-cluster-management/deployments/open-cluster-management-registry
uid: 900eb533-89f8-43ee-9bfa-3f17d77e93b7
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: open-cluster-management-registry
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: open-cluster-management-registry
spec:
containers:
- image: quay.io/open-cluster-management/multicloudhub-operator-index:1.0.0-SNAPSHOT-2020-03-10-19-58-31
imagePullPolicy: IfNotPresent
name: multicloudhub-operator-index
ports:
- containerPort: 50051
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: quay-secret
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2020-03-11T03:29:56Z"
lastUpdateTime: "2020-03-11T03:29:56Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available - lastTransitionTime: "2020-03-11T03:29:45Z"
lastUpdateTime: "2020-03-11T03:29:56Z"
message: ReplicaSet "open-cluster-management-registry-dbc98b957" has successfully
progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
[kni@provisionhost-0 ~]$
[kni@provisionhost-0 ~]$ oc get catalogsource open-cluster-management -n open-cluster-management -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"operators.coreos.com/v1alpha1","kind":"CatalogSource","metadata":{"annotations":{},"name":"open-cluster-management","namespace":"open-cluster-management"},"spec":{"address":"open-cluster-management-registry.open-cluster-management.svc:50051","sourceType":"grpc"}}
creationTimestamp: "2020-03-11T03:29:45Z"
generation: 1
name: open-cluster-management
namespace: open-cluster-management
resourceVersion: "2412749"
selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/open-cluster-management/catalogsources/open-cluster-management
uid: 3015fc78-5243-4a63-8f61-b9ce6c82a6da
spec:
address: open-cluster-management-registry.open-cluster-management.svc:50051
sourceType: grpc
status:
connectionState:
address: open-cluster-management-registry.open-cluster-management.svc:50051
lastConnect: "2020-03-11T14:15:24Z"
lastObservedState: READY
registryService:
createdAt: "2020-03-11T03:29:45Z"
protocol: grpc
[kni@provisionhost-0 ~]$
[kni@provisionhost-0 multicloudhub-operator]$ cat kustomization.yaml
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generatorOptions:
disableNameSuffixHash: true
# namespace to deploy all Resources to
namespace: open-cluster-management
images:
- name: multicloudhub-operator-index
newName: quay.io/open-cluster-management/multicloudhub-operator-index
newTag: 1.0.0-SNAPSHOT-2020-03-10-19-58-31
# list of Resource Config to be Applied
resources:
- quay-secret.yaml
- deployment.yaml
- service.yaml
- catalog-source.yaml
- operator-group.yaml
- subscription.yaml
@achuzhoy All this looks good... you were able to pull the multicloudhub-operator-index
image so you should have no issues pulling the multicloudhub-operator-bundle
image.
I made some changes to this repo last night to work with some changes that multicloudhub-operator
made this morning wrt new secrets... please pull the latest code and be sure to read the updated README.md
Reproduced the issue.
Note that I'm trying it on 4.4 (deviates from the README.md)
Warning Failed 22s kubelet, worker-1 Error: ImagePullBackOff
Normal Pulling 10s (x2 over 23s) kubelet, worker-1 Pulling image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:0edb06bb5c8e9c49a21bbb678709c524f549cbd23ec637987ac08feac8a9f5be"
Warning Failed 10s (x2 over 22s) kubelet, worker-1 Failed to pull image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:0edb06bb5c8e9c49a21bbb678709c524f549cbd23ec637987ac08feac8a9f5be": rpc error: code = Unknown desc = Error reading manifest sha256:0edb06bb5c8e9c49a21bbb678709c524f549cbd23ec637987ac08feac8a9f5be in quay.io/open-cluster-management/multicloudhub-operator-bundle: unauthorized: access to the requested resource is not authorized
@achuzhoy this is a hack but it will get you past the issue:
oc patch serviceaccount default -p '{"imagePullSecrets": [{"name": "multiclusterhub-operator-pull-secret"}]}'
I did find that some repos in quay didn't have read permissions set for some accounts - QE and Robots among them. I've set all existing repos now to have read permissions for everyone, so that might have an effect here.
oc patch serviceaccount default -p '{"imagePullSecrets": [{"name": "multiclusterhub-operator-pull-secret"}]}'
@tpouyer tried that on a new setup (and thus new clone) - same error persists.
I too am having the same issue. I can pull from open-cluster-management/multicloudhub-operator-index but dont seem to have permissions to pull from open-cluster-management/multicloudhub-operator-bundle
[root@sealusa11 multicloudhub-operator]# oc get pods
NAME READY STATUS RESTARTS AGE
0b302b2b0acd2947160616eec1ae06326a854a7e4a0e1d3925f0e40d1d2nff9 0/1 Init:ImagePullBackOff 0 3m54s
open-cluster-management-registry-c7dbc8f47-ctlkt 1/1 Running 0 4m1s
[root@sealusa11 multicloudhub-operator]# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.4.0-0.ci-2020-03-11-095511 True False 22h Cluster version is 4.4.0-0.ci-2020-03-11-095511
I have just pushed an update to the repo... I'm now patching the default service-account with the pull-secret that gets created as part of the prereqs
here: https://github.com/open-cluster-management/deploy/blob/master/prereqs/serviceaccount.yaml#L7
This should resolve the issue some people are having... I have not been able to isolate why some people are having auth issues pulling the bundle image and some are not... permissions are the same on the bundle repo as the index repo in quay... patching the default service account with the pull-secret seems to be the only way to get around the problem...
Ultimately these repos will all be opensourced at some point and the need for pull secrets to pull the index and bundle images will no longer be necessary.
Please pull the latest code down and try it out, be sure to rerun kubectl apply -k .
in the prereqs
dir to patch the service account even if you are reusing your OCP cluster and already have applied the prereqs
before.
Now failing with #6
The error is for another image, but the resolution may be the same in the end.
Still stuck.
Was able to pass this step.