Cert manager unable to update certs
MattCosturos opened this issue · comments
We are using the Kubernetes cert manager, version 2.4.2, along with Let's Encrypt.
Our cert expires in the next few days, and cert-manager is unable to update.
These logs are from the cert-manager-webhook pod.
I am seeing this vague error but I am not sure where to go next for further debugging.
2023-03-15T04:43:55-04:00 I0315 08:43:55.000294 1 dynamic_source.go:169] cert-manager/webhook "msg"="Serving certificate requires renewal, regenerating"
2023-03-15T04:43:55-04:00 I0315 08:43:55.035562 1 dynamic_source.go:267] cert-manager/webhook "msg"="Updated serving TLS certificate"
2023-03-15T20:43:54-04:00 I0316 00:43:54.473908 1 authority.go:312] cert-manager/webhook "msg"="Root CA certificate is nearing expiry. Regenerating..."
2023-03-15T20:43:54-04:00 I0316 00:43:54.519234 1 dynamic_source.go:160] cert-manager/webhook "msg"="Detected root CA rotation - regenerating serving certificates"
2023-03-15T20:43:54-04:00 I0316 00:43:54.551889 1 dynamic_source.go:267] cert-manager/webhook "msg"="Updated serving TLS certificate"
2023-03-16T06:43:57-04:00 W0316 10:43:57.028777 1 reflector.go:442] external/io_k8s_client_go/tools/cache/reflector.go:167: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
2023-03-20T12:43:54-04:00 I0320 16:43:54.001357 1 dynamic_source.go:169] cert-manager/webhook "msg"="Serving certificate requires renewal, regenerating"
2023-03-20T12:43:54-04:00 I0320 16:43:54.035635 1 dynamic_source.go:267] cert-manager/webhook "msg"="Updated serving TLS certificate"
Hello, this problem is not related to module i think.
But anyway, i am not sure that in this logs any issues exist.
Check also there:
kubectl get CertificateRequest -n NAMESPACE
kubectl get Certificate -n NAMESPACE
In the status you can check, is certificate new or can't request new one.
watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
That seems like it is "not good", but I agree with you it doesn't seem to be a problem with the module or the terraform block.
Running those 2 commands.
I see there is a Certificate Request that is 30d old, which is approved and ready.
The Cert is 89 days old.
We used v2.4.2 of the terraform block, which deploys v1.7.1 of the cert manager. we are in the process of updating to 2.5.0 and cert-manager 1.11.0 to see if this will get resolved.
Any other ideas where to look based on this cert / request info below?
kubectl get CertificateRequest -n my-namespace
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
le-cluster-domain-wr248 True True cert-manager-global system:serviceaccount:cert-manager:cert-manager 30d
kubectl get Certificate -n my-namespace
NAME READY SECRET AGE
le-cluster-domain True le-cluster-domain 89d
The cert is 89 days old since that was when I made the namespace. It was being updated correctly.
We are running into an issue with Azure AppGateway. The terraform module, and cert-manager pod are all working correctly.
Sorry to bother you!