[error retrieving webhook] Error occurred in cosigned-webhook-pods

[Shawnho]

[Environment]

• GKE v1.21.10
• Helm CLI: v3.6.2
• cosigned helm chart version v1.8.0

1. Deploy GKE standard cluster v1.21.10-gke.2000
2. Deploy Cosigned's helm chart with the following script:

kubectl create namespace cosign-system

kubectl create secret generic mysecret -n cosign-system \
--from-file=cosign.pub=./cosign.pub \
--from-file=cosign.key=./cosign.key \
--from-literal=cosign.password=${COSIGN_PASSWORD}

helm install cosigned -n cosign-system sigstore/cosigned --devel \
--set cosign.secretKeyRef.name=mysecret 

3. Running logs command to check cosigned-webhook and we'll see repeated errors below.

kubectl logs -n cosign-system cosigned-webhook-6c68bfb587-6c7hb

{"level":"error","ts":"2022-05-08T17:18:08.942Z","logger":"cosigned.DefaultingWebhook","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"9ef6b20","knative.dev/traceid":"00a8cb0b-180f-45a6-847c-6421a9eaa7b8","knative.dev/key":"cosign-system/webhook-certs","duration":0.000132104,"error":"error retrieving webhook: mutatingwebhookconfiguration.admissionregistration.k8s.io \"\\\"cosigned.sigstore.dev\\\"\" not found","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20220325200448-1f7514acd0c2/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20220325200448-1f7514acd0c2/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20220325200448-1f7514acd0c2/controller/controller.go:491"}
{"level":"error","ts":"2022-05-08T17:18:08.944Z","logger":"cosigned.ValidationWebhook","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"9ef6b20","knative.dev/traceid":"76a39714-c3f6-4491-a99f-5548e0a50d38","knative.dev/key":"cosign-system/webhook-certs","duration":0.000152002,"error":"error retrieving webhook: validatingwebhookconfiguration.admissionregistration.k8s.io \"\\\"cosigned.sigstore.dev\\\"\" not found","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20220325200448-1f7514acd0c2/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20220325200448-1f7514acd0c2/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20220325200448-1f7514acd0c2/controller/controller.go:491"}

I tested under such error, even I enable cosign policy in my namespace. It is not working as expected. Any comment is highly appreciated.

[Shawnho]

@hectorj2f thanks for taking the issue. My colleague and I worked together and found the possible root cause. Once we removed the quote mark in the arg:
--webhook-name="cosigned.sigstore.dev"
The pod could start correctly. Just a quick update for this issue.

    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  control-plane: cosigned-webhook
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - args:
        - --webhook-name=cosigned.sigstore.dev
        - -secret-name=mysecret
        env:

[Hector Fernandez]

@shawnho1018 Question: Does the policy webhook work too ? It also contains double quotes for the names.

[Shawnho]

Policy webhook wored. However, I checked its template but I don't see this yaml file contains any quote.
If you check deployment-webhook.yaml, you would find the quote in this line:

        - --webhook-name={{ required "A valid cosign.webhookName is required" .Values.cosign.webhookName | quote }}