hypershift service account is attempting to grant RBAC permissions not currently held

[Roman Bednář]

Hypershift installation fails on AWS. Approximately 2 days ago the same procedure worked fine for me so this bug must have landed recently with hypershift-operator:latest image.

$ oc logs deployment/operator -n hypershift

{"level":"error","ts":"2023-01-19T14:26:35Z","msg":"Reconciler error","controller":"hostedcluster","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedCluster","hostedCluster":{"name":"example-hypershift","namespace":"clusters"},"namespace":"clusters","name":"example-hypershift","reconcileID":"7a477039-3b2c-430b-8f94-4003611ff0d5","error":"failed to reconcile control plane operator: failed to reconcile controlplane operator role: roles.rbac.authorization.k8s.io \"control-plane-operator\" is forbidden: user \"system:serviceaccount:hypershift:operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:hypershift\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"image.openshift.io\"], Resources:[\"*\"], Verbs:[\"*\"]}","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}

This is my deployment:

$ oc -n hypershift get deployment/operator -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    hypershift.openshift.io/install-cli-version: openshift/hypershift 01a16bb380a1e49da2cfeef5109503b7538627e2
  creationTimestamp: "2023-01-19T14:05:03Z"
  generation: 1
  name: operator
  namespace: hypershift
  resourceVersion: "29630"
  uid: 1aac4550-0910-4274-b333-d2d6c6f6c8bf
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: operator
        hypershift.openshift.io/operator-component: operator
        name: operator
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: name
                operator: In
                values:
                - operator
            topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - run
        - --namespace=$(MY_NAMESPACE)
        - --pod-name=$(MY_NAME)
        - --metrics-addr=:9000
        - --enable-ocp-cluster-monitoring=false
        - --enable-ci-debug-output=false
        - --private-platform=None
        - --cert-dir=/var/run/secrets/serving-cert
        - --oidc-storage-provider-s3-bucket-name=rbednar-bucket-01
        - --oidc-storage-provider-s3-region=us-east-1
        - --oidc-storage-provider-s3-credentials=/etc/oidc-storage-provider-s3-creds/credentials
        command:
        - /usr/bin/hypershift-operator
        env:
        - name: MY_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: MY_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: METRICS_SET
          value: Telemetry
        image: quay.io/hypershift/hypershift-operator:latest
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /metrics
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 60
          successThreshold: 1
          timeoutSeconds: 5
        name: operator
        ports:
        - containerPort: 9000
          name: metrics
          protocol: TCP
        - containerPort: 9443
          name: manager
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /metrics
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 60
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          requests:
            cpu: 10m
            memory: 150Mi
        securityContext:
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run/secrets/serving-cert
          name: serving-cert
        - mountPath: /etc/oidc-storage-provider-s3-creds
          name: oidc-storage-provider-s3-creds
      dnsPolicy: ClusterFirst
      priorityClassName: hypershift-operator
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: operator
      serviceAccountName: operator
      terminationGracePeriodSeconds: 30
      volumes:
      - name: serving-cert
        secret:
          defaultMode: 420
          secretName: manager-serving-cert
      - name: oidc-storage-provider-s3-creds
        secret:
          defaultMode: 420
          secretName: hypershift-operator-oidc-provider-s3-credentials
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2023-01-19T14:06:03Z"
    lastUpdateTime: "2023-01-19T14:06:03Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2023-01-19T14:05:03Z"
    lastUpdateTime: "2023-01-19T14:06:03Z"
    message: ReplicaSet "operator-766d77995" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2

[Roman Bednář]

Resolved by: 2ad9143#diff-4edc0a874ce10f9d4f77594c9ff489c3275cb2adc788826a9ea2f48a64c336efR782

Closing.