Exposer route gets HostAlreadyClaimed

Question

Exposer route gets HostAlreadyClaimed

Alveel opened this issue 3 years ago · comments

What happened:

Upon adding the acme annotation to a route, the exposer gets created. However, the exposer route gets rejected by the default router with HostAlreadyClaimed.

Maybe duplicate of #140, but it still seems to occur even with the latest images.

What you expected to happen:

The acme controller creates appropriate routes and is able to get the certificate.

How to reproduce it (as minimally and precisely as possible):

Unsure, so far I have been able to reproduce it by adding the acme annotation to the oauth-openshift route in the openshift-authentication namespace.

Anything else we need to know?:

I have tried re-creating the router without the acme annotations with oc replace, as well as oc replace --force, and re-adding the annotations a little while later.

Environment:

OpenShift/Kubernetes version (use oc/kubectl version):

Client Version: 4.6.16
Server Version: 4.5.27
Kubernetes Version: v1.18.3+f561b20

Others: This is a Azure RedHat OpenShift (ARO) cluster.

@tnozicka

Alwyn Kik · Answer 1 · Tue Feb 16 2021 20:56:56 GMT+0800 (China Standard Time)

I couldn't find many relevant entries in the logs, but alas:

I0216 11:47:08.835968       1 route.go:496] Started syncing Route "openshift-authentication/oauth-openshift"
I0216 11:47:08.836330       1 route.go:563] Route "openshift-authentication/oauth-openshift" needs new certificate: Route is missing CertKey
I0216 11:47:08.836553       1 route.go:607] Using ACME client with DirectoryURL "https://acme-v02.api.letsencrypt.org/directory"
I0216 11:47:09.388951       1 route.go:650] Route "openshift-authentication/oauth-openshift": Order "https://acme-v02.api.letsencrypt.org/acme/order/112525390/7925055141" is in "pending" state
I0216 11:47:09.389005       1 route.go:655] Route "openshift-authentication/oauth-openshift": Order "https://acme-v02.api.letsencrypt.org/acme/order/112525390/7925055141" contains 1 authorization(s)
I0216 11:47:09.554541       1 route.go:663] Route "openshift-authentication/oauth-openshift": order "https://acme-v02.api.letsencrypt.org/acme/order/112525390/7925055141": authz "https://acme-v02.api.letsencrypt.org/acme/authz-v3/10892179595": is in "pending" state
I0216 11:47:09.554590       1 route.go:690] route "openshift-authentication/oauth-openshift": order "https://acme-v02.api.letsencrypt.org/acme/order/112525390/7925055141": authz "https://acme-v02.api.letsencrypt.org/acme/authz-v3/10892179595": challenge "pending" is in "pending" state
I0216 11:47:09.554719       1 route.go:1008] exposer Route openshift-authentication/exposer-lkckfmiq0dn9vtl2bui4m5c8bnt65e4q98gbd9jnm6mc1cpdseh0 isn't admitted yet
I0216 11:47:09.555069       1 route.go:498] Finished syncing Route "openshift-authentication/oauth-openshift"

Alwyn Kik · Answer 2 · Mon Apr 19 2021 20:32:33 GMT+0800 (China Standard Time)

I think the issue with HostAlreadyClaimed stopped occurring for some reason. However the issue in the previous comment is still there (with different routes/domains). Perhaps I should create a new issue?

Tomáš Nožička · Answer 3 · Tue May 04 2021 18:33:08 GMT+0800 (China Standard Time)

HostAlreadyClaimed means another namespace already has a Route with the same domain. The first namespace that claims it wins.

JT Smith · Answer 4 · Thu May 06 2021 04:23:29 GMT+0800 (China Standard Time)

@tnozicka this is happening to every route i add the annotation to. i'm not sure if it's spinning up the exposer too slowly or what, but every time we add the annotation, it tries to add the exposer but it gets rejected because the main route that we added the annotation already has that route.

⚡️•100% ➜ oc version
Client Version: v4.2.0-alpha.0-657-g51011e4
Server Version: 4.6.21
Kubernetes Version: v1.19.0+2f3101c

exposer route:

spec:
  host: example-nginx-default.apps.domain.com
  path: /.well-known/acme-challenge/blah
  to:
    kind: Service
    name: exposer-1n16ito7mf1f48eqoftks21tqtc7h4a5fiid238eroe5s5ogi57g
    weight: 100
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow
  wildcardPolicy: None
status:
  ingress:
    - host: example-nginx-default.apps.domain.com
      routerName: default
      conditions:
        - type: Admitted
          status: 'False'
          reason: HostAlreadyClaimed
          message: >-
            route example-nginx already exposes
            example-nginx-default.apps.domain.com and is older
          lastTransitionTime: '2021-05-05T20:14:47Z'
      wildcardPolicy: None
      routerCanonicalHostname: apps.domain.com

Tomáš Nožička · Answer 5 · Thu May 06 2021 13:57:38 GMT+0800 (China Standard Time)

Can you attach the full yaml (with redacted cert if any) for both of them?
oc get route/<original> route/<exposer> -o yaml

JT Smith · Answer 6 · Thu May 06 2021 21:55:26 GMT+0800 (China Standard Time)

I'm still fairly new to OpenShift (might explain my troubles) but I couldn't get the command to work so I just went to the "YAML" tab on the web interface and pulled these from there. Hopefully they're what you're looking for?

[EDIT: After posting this, I see that the command I copy/pasted out of the email that GH sent me about your update removed the <original> and <exposer> parts, explaining why the command wouldn't work. I can confirm that the YAML below is what you're requesting.]

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: example-nginx
  namespace: default
  selfLink: /apis/route.openshift.io/v1/namespaces/default/routes/example-nginx
  uid: 7619f74b-bbac-4a69-bceb-a467aa487311
  resourceVersion: '73498'
  creationTimestamp: '2021-05-05T20:14:18Z'
  annotations:
    acme.openshift.io/status: |
      provisioningStatus:
        earliestAttemptAt: "2021-05-05T20:14:46.808511663Z"
        orderStatus: pending
        orderURI: https://acme-v02.api.letsencrypt.org/acme/order/123456789/123456789
        startedAt: "2021-05-05T20:14:46.808511663Z"
    kubernetes.io/tls-acme: 'true'
    openshift.io/host.generated: 'true'
  managedFields:
    - manager: openshift-router
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2021-05-05T20:14:19Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:ingress': {}
    - manager: Mozilla
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2021-05-05T20:14:45Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:kubernetes.io/tls-acme': {}
        'f:spec':
          'f:port':
            .: {}
            'f:targetPort': {}
          'f:tls':
            .: {}
            'f:insecureEdgeTerminationPolicy': {}
            'f:termination': {}
          'f:to':
            'f:kind': {}
            'f:name': {}
            'f:weight': {}
          'f:wildcardPolicy': {}
    - manager: openshift-acme-controller
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2021-05-05T20:14:47Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:acme.openshift.io/status': {}
spec:
  host: example-nginx-default.apps.domain.com
  to:
    kind: Service
    name: example-nginx
    weight: 100
  port:
    targetPort: 8080
  tls:
    termination: passthrough
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None
status:
  ingress:
    - host: example-nginx-default.apps.domain.com
      routerName: default
      conditions:
        - type: Admitted
          status: 'True'
          lastTransitionTime: '2021-05-05T20:14:19Z'
      wildcardPolicy: None
      routerCanonicalHostname: apps.domain.com

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  annotations:
    acme.openshift.io/exposer-id: >-
      example-nginx-default.apps.domain.com:/.well-known/acme-challenge/abcdefghijklmnopqrstuvwyz
    acme.openshift.io/exposer-key: default/example-nginx
    acme.openshift.io/status: |
      provisioningStatus:
        earliestAttemptAt: "0001-01-01T00:00:00Z"
        orderStatus: pending
        orderURI: https://acme-v02.api.letsencrypt.org/acme/order/1234567890/1234567890
        startedAt: "2021-05-05T20:14:46.553448501Z"
    kubernetes.io/tls-acme: 'true'
    openshift.io/host.generated: 'true'
  selfLink: >-
    /apis/route.openshift.io/v1/namespaces/default/routes/exposer-zyxwvutsrqponmlkjihgfedcba
  resourceVersion: '73495'
  name: exposer-zyxwvutsrqponmlkjihgfedcba
  uid: c074a3c8-fdb6-4bc1-8161-7196c3116b58
  creationTimestamp: '2021-05-05T20:14:47Z'
  managedFields:
    - manager: Mozilla
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2021-05-05T20:14:45Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:port':
            .: {}
            'f:targetPort': {}
    - manager: openshift-acme-controller
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2021-05-05T20:14:47Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:acme.openshift.io/exposer-id': {}
            'f:acme.openshift.io/exposer-key': {}
            'f:acme.openshift.io/status': {}
            'f:kubernetes.io/tls-acme': {}
            'f:openshift.io/host.generated': {}
          'f:labels':
            .: {}
            'f:acme.openshift.io/exposer-uid': {}
            'f:acme.openshift.io/temporary': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"7619f74b-bbac-4a69-bceb-a467aa487311"}':
              .: {}
              'f:apiVersion': {}
              'f:controller': {}
              'f:kind': {}
              'f:name': {}
              'f:uid': {}
        'f:spec':
          'f:host': {}
          'f:path': {}
          'f:tls':
            .: {}
            'f:insecureEdgeTerminationPolicy': {}
            'f:termination': {}
          'f:to':
            'f:kind': {}
            'f:name': {}
            'f:weight': {}
          'f:wildcardPolicy': {}
    - manager: openshift-router
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2021-05-05T20:14:47Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:ingress': {}
  namespace: default
  ownerReferences:
    - apiVersion: route.openshift.io/v1
      kind: Route
      name: example-nginx
      uid: 7619f74b-bbac-4a69-bceb-a467aa487311
      controller: true
  labels:
    acme.openshift.io/exposer-uid: 7619f74b-bbac-4a69-bceb-a467aa487311
    acme.openshift.io/temporary: 'true'
spec:
  host: example-nginx-default.apps.domain.com
  path: /.well-known/acme-challenge/abcdefghijklmnopqrstuvwyz
  to:
    kind: Service
    name: exposer-aqswdefrgthyjuki
    weight: 100
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow
  wildcardPolicy: None
status:
  ingress:
    - host: example-nginx-default.apps.domain.com
      routerName: default
      conditions:
        - type: Admitted
          status: 'False'
          reason: HostAlreadyClaimed
          message: >-
            route example-nginx already exposes
            example-nginx-default.apps.domain.com and is older
          lastTransitionTime: '2021-05-05T20:14:47Z'
      wildcardPolicy: None
      routerCanonicalHostname: apps.domain.com

Tomáš Nožička · Answer 7 · Thu May 06 2021 22:17:42 GMT+0800 (China Standard Time)

your initial route is termination: passthrough, that's why the new one doesn't get admitted - related to #117

JT Smith · Answer 8 · Thu May 06 2021 23:03:08 GMT+0800 (China Standard Time)

Awesome, thank you! Removing it seems to have worked!

Alwyn Kik · Answer 9 · Wed May 19 2021 15:22:32 GMT+0800 (China Standard Time)

My assignment ended at the company where I experienced this issue, so I'm closing it. :(