Route is missing CertKey, no exposer route created

[Stephen Reaves]

What happened:

After installing openshift-acme and adding the annotations to an existing route, these logs keep repeating in one of the acme pods.

I1204 00:24:51.074684       1 route.go:496] Started syncing Route "homelab-main/portfolio"
I1204 00:24:51.074793       1 route.go:563] Route "homelab-main/portfolio" needs new certificate: Route is missing CertKey
I1204 00:24:51.075066       1 route.go:498] Finished syncing Route "homelab-main/portfolio"
E1204 00:24:51.075091       1 route.go:1301] homelab-main/portfolio failed with : can't get cert issuer: cert issuer acme/letsencrypt-live is missing required secret
I1204 00:25:07.546687       1 acme.go:271] Started syncing Account "acme/letsencrypt-live"
I1204 00:25:11.478149       1 acme.go:273] Finished syncing Account "acme/letsencrypt-live"
E1204 00:25:11.478201       1 acme.go:157] acme/letsencrypt-live failed with : Get https://acme-v02.api.letsencrypt.org/directory: x509: certificate is valid for *.apps.openshift.<domain.com>, not acme-v02.api.letsencrypt.org

What you expected to happen:

Route to portfolio app becomes encrypted

How to reproduce it (as minimally and precisely as possible):

oc apply -fhttps://raw.githubusercontent.com/tnozicka/openshift-acme/master/deploy/cluster-wide/{clusterrole,serviceaccount,issuer-letsencrypt-live,deployment}.yaml
oc create clusterrolebinding openshift-acme --clusterrole=openshift-acme --serviceaccount="$( oc project -q ):openshift-acme" --dry-run -o yaml | oc apply -f -

apiVersion: route.openshift.io/v1
metadata:
  annotations:
    kubernetes.io/tls-acme: 'true'
  selfLink: /apis/route.openshift.io/v1/namespaces/homelab-main/routes/portfolio
  resourceVersion: '1045914'
  name: portfolio
  uid: ae3b8054-b443-49da-91a5-d0e516468b7f
  creationTimestamp: '2020-12-03T20:36:44Z'
  managedFields:
    - manager: openshift-router
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2020-12-03T20:36:44Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:ingress': {}
    - manager: Mozilla
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2020-12-04T00:24:30Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubernetes.io/tls-acme': {}
          'f:labels':
            .: {}
            'f:app': {}
            'f:app.kubernetes.io/component': {}
            'f:app.kubernetes.io/instance': {}
            'f:app.kubernetes.io/name': {}
            'f:app.kubernetes.io/part-of': {}
            'f:app.openshift.io/runtime': {}
            'f:app.openshift.io/runtime-version': {}
        'f:spec':
          'f:host': {}
          'f:port':
            .: {}
            'f:targetPort': {}
          'f:tls':
            .: {}
            'f:insecureEdgeTerminationPolicy': {}
            'f:termination': {}
          'f:to':
            'f:kind': {}
            'f:name': {}
            'f:weight': {}
          'f:wildcardPolicy': {}
  namespace: homelab-main
  labels:
    app: portfolio
    app.kubernetes.io/component: portfolio
    app.kubernetes.io/instance: portfolio
    app.kubernetes.io/name: python
    app.kubernetes.io/part-of: portfolio-app
    app.openshift.io/runtime: python
    app.openshift.io/runtime-version: 3.8-ubi7
spec:
  host: portfolio.apps.openshift.<domain.com>
  to:
    kind: Service
    name: portfolio
    weight: 100
  port:
    targetPort: 8080-tcp
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None
status:
  ingress:
    - host: portfolio.apps.openshift.<domain.com>
      routerName: default
      conditions:
        - type: Admitted
          status: 'True'
          lastTransitionTime: '2020-12-03T20:36:44Z'
      wildcardPolicy: None
      routerCanonicalHostname: apps.openshift.<domain.com>

Anything else we need to know?:

This is a fresh cluster, but I am fairly inexperienced with openshift/k8s.

Environment:

• OpenShift/Kubernetes version (use oc/kubectl version):

Client Version: 4.6.6
Server Version: 4.6.8-0.okd-2020-12-01-053012
Kubernetes Version: v1.19.2-1003+1348ff864868a6-dirty

• Others:

@tnozicka

[Tomáš Nožička]

E1204 00:25:11.478201 1 acme.go:157] acme/letsencrypt-live failed with : Get https://acme-v02.api.letsencrypt.org/directory: x509: certificate is valid for *.apps.openshift.<domain.com>, not acme-v02.api.letsencrypt.org

This looks like while the controller is trying to reach out to acme-v02.api.letsencrypt.org it hits the OCP router, can you check your OCP installation? Have you tried oc run a pod and trying to reach https://acme-v02.api.letsencrypt.org/directory url from there?

[Stephen Reaves]

After banging my head against the wall all day yesterday, I came back first thing this morning and it just works now....

The only thing I can figure that happened was that I was running reverse-proxy with let's encrypt before upgrading to OKD, and I accidentally left that container running on a different host but for the same domain. I killed that container and manually revoked the certificate after lunch and maybe it took a while to process? have another route that's having a different issue but I think this issue can be marked closed. Thanks for taking the time to look at this.