policy controller taking longer then 30s to complete
mlbiam opened this issue · comments
Description
When running the policy controller on civo cloud, it's taking longer then 30s to complete validation/mutation. I'm not seeing this behavior on other clouds. I'd like to open an issue with Civo, but i don't really have any data to go back to them with. Here's the logs:
{"level":"info","ts":"2023-05-22T14:58:37.852Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000e19cb0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"3819\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000eb1140), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:3819, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.cosign-system.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.0.2:42738\", RequestURI:\"/mutations?timeout=30s\", TLS:(*tls.ConnectionState)(0xc0008f3c30), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc000eb1180)}","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-05-22T14:58:37.852Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc0005678c0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"3800\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000b22580), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:3800, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.cosign-system.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.2.4:48442\", RequestURI:\"/mutations?timeout=30s\", TLS:(*tls.ConnectionState)(0xc0002c8160), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc000b225c0)}","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-05-22T14:58:52.546Z","logger":"policy-controller","caller":"defaulting/defaulting.go:158","msg":"Kind: \"apps/v1, Kind=Deployment\" PatchBytes: [{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/image\",\"value\":\"public.ecr.aws/o6u7e2l7/pause@sha256:569cfa0ff435f3a076a1b06a1f45d772ee3f5d4fbf6b39242a573c0cff632d69\"}]","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"run-service","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller"}
{"level":"info","ts":"2023-05-22T14:58:52.546Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"run-service","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller","admissionreview/uid":"793b9e0d-a62e-49ad-b471-77a15170d239","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2023-05-22T14:58:52.567Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000e0f5f0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"3943\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000f68d80), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:3943, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.cosign-system.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.2.4:50294\", RequestURI:\"/validations?timeout=20s\", TLS:(*tls.ConnectionState)(0xc000a41ad0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc000f68dc0)}","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-05-22T14:58:54.593Z","logger":"policy-controller","caller":"defaulting/defaulting.go:158","msg":"Kind: \"apps/v1, Kind=Deployment\" PatchBytes: [{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/image\",\"value\":\"public.ecr.aws/o6u7e2l7/scratchpad@sha256:b5f91bab579a81b7367986ccdcfa6288b0511f5361046b81543f7e4c61e490b1\"}]","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"scrathpad","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller"}
{"level":"info","ts":"2023-05-22T14:58:54.593Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"scrathpad","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller","admissionreview/uid":"0f0068d4-b5d8-493e-9b64-3db58b9ad287","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2023-05-22T14:58:54.614Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc0010e0120), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"3962\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc00053ce80), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:3962, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.cosign-system.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.0.2:36546\", RequestURI:\"/validations?timeout=18s\", TLS:(*tls.ConnectionState)(0xc000b6ca50), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc00053cec0)}","commit":"89ef904-dirty"}
{"level":"error","ts":"2023-05-22T14:59:24.104Z","logger":"policy-controller","caller":"webhook/validator.go:763","msg":"failed validSignatures for authority keyless with fulcio for public.ecr.aws/o6u7e2l7/pause@sha256:569cfa0ff435f3a076a1b06a1f45d772ee3f5d4fbf6b39242a573c0cff632d69: Get \"https://public.ecr.aws/v2/\": context canceled","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"run-service","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller","stacktrace":"github.com/sigstore/policy-controller/pkg/webhook.ValidatePolicySignaturesForAuthority\n\tgithub.com/sigstore/policy-controller/pkg/webhook/validator.go:763\ngithub.com/sigstore/policy-controller/pkg/webhook.ValidatePolicy.func1\n\tgithub.com/sigstore/policy-controller/pkg/webhook/validator.go:529"}
{"level":"warn","ts":"2023-05-22T14:59:24.104Z","logger":"policy-controller","caller":"webhook/validator.go:1115","msg":"Failed to validate at least one policy for public.ecr.aws/o6u7e2l7/pause@sha256:569cfa0ff435f3a076a1b06a1f45d772ee3f5d4fbf6b39242a573c0cff632d69 wanted 1 policies, only validated 0","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"run-service","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller"}
{"level":"error","ts":"2023-05-22T14:59:24.105Z","logger":"policy-controller","caller":"validation/validation_admit.go:180","msg":"Failed the resource specific validation","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"run-service","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller","stacktrace":"knative.dev/pkg/webhook/resourcesemantics/validation.validate\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/resourcesemantics/validation/validation_admit.go:180\nknative.dev/pkg/webhook/resourcesemantics/validation.(*reconciler).Admit\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/resourcesemantics/validation/validation_admit.go:79\nknative.dev/pkg/webhook.admissionHandler.func1\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/admission.go:123\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2109\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2487\nknative.dev/pkg/webhook.(*Webhook).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/webhook.go:262\nknative.dev/pkg/network/handlers.(*Drainer).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/network/handlers/drain.go:113\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991"}
{"level":"info","ts":"2023-05-22T14:59:24.105Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"run-service","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller","admissionreview/uid":"e46ed4de-6d9c-400c-8541-bae22e306f9c","admissionreview/allowed":false,"admissionreview/result":"&Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:validation failed: context was canceled before validation completed: ,Reason:BadRequest,Details:nil,Code:400,}"}
{"level":"error","ts":"2023-05-22T14:59:25.854Z","logger":"policy-controller","caller":"webhook/validator.go:763","msg":"failed validSignatures for authority keyless with fulcio for public.ecr.aws/o6u7e2l7/scratchpad@sha256:b5f91bab579a81b7367986ccdcfa6288b0511f5361046b81543f7e4c61e490b1: Get \"https://public.ecr.aws/v2/\": context canceled","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"scrathpad","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller","stacktrace":"github.com/sigstore/policy-controller/pkg/webhook.ValidatePolicySignaturesForAuthority\n\tgithub.com/sigstore/policy-controller/pkg/webhook/validator.go:763\ngithub.com/sigstore/policy-controller/pkg/webhook.ValidatePolicy.func1\n\tgithub.com/sigstore/policy-controller/pkg/webhook/validator.go:529"}
{"level":"warn","ts":"2023-05-22T14:59:25.855Z","logger":"policy-controller","caller":"webhook/validator.go:1115","msg":"Failed to validate at least one policy for public.ecr.aws/o6u7e2l7/scratchpad@sha256:b5f91bab579a81b7367986ccdcfa6288b0511f5361046b81543f7e4c61e490b1 wanted 1 policies, only validated 0","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"scrathpad","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller"}
{"level":"error","ts":"2023-05-22T14:59:25.855Z","logger":"policy-controller","caller":"validation/validation_admit.go:180","msg":"Failed the resource specific validation","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"scrathpad","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller","stacktrace":"knative.dev/pkg/webhook/resourcesemantics/validation.validate\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/resourcesemantics/validation/validation_admit.go:180\nknative.dev/pkg/webhook/resourcesemantics/validation.(*reconciler).Admit\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/resourcesemantics/validation/validation_admit.go:79\nknative.dev/pkg/webhook.admissionHandler.func1\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/admission.go:123\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2109\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2487\nknative.dev/pkg/webhook.(*Webhook).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/webhook.go:262\nknative.dev/pkg/network/handlers.(*Drainer).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/network/handlers/drain.go:113\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991"}
{"level":"info","ts":"2023-05-22T14:59:25.855Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"apps/v1, Kind=Deployment","knative.dev/namespace":"user-ns-mlbiam","knative.dev/name":"scrathpad","knative.dev/operation":"CREATE","knative.dev/resource":"apps/v1, Resource=deployments","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:argocd:argocd-application-controller","admissionreview/uid":"9210de34-6cc1-4f14-87e9-b378a7797a5e","admissionreview/allowed":false,"admissionreview/result":"&Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:validation failed: context was canceled before validation completed: ,Reason:BadRequest,Details:nil,Code:400,}"}Version
policy controller - ghcr.io/sigstore/policy-controller/policy-controller@sha256:947693aa3a536992bc89f3c7ded8a7707b26cd4518972f293edd3e57e112438e
Closing this issue for now! It is related to the cloud provider where it got deployed.