exposer routes rejected for a passthrough route with a secret to mount
ntxt opened this issue · comments
What happened:
I have run the following script with a resource file below, the deployment and a passthrough route got created with the annotation pointing to a TLS secret. The secret gets created but the cert and key values are not populated, which makes the "docker-registry" fail looking for them (the secret is mounted to the container). The exposer route gets rejected with "HostAlreadyClaimed".
What you expected to happen:
The secret should get populated with the key and cert values, "docker-registry" should see and use them to secure the requests passed through by the route.
How to reproduce it (as minimally and precisely as possible):
ENV=staging #staging | live
oc new-project example-utils
oc apply -fhttps://raw.githubusercontent.com/tnozicka/openshift-acme/master/deploy/single-namespace/{role,serviceaccount,issuer-letsencrypt-${ENV},deployment}.yaml
oc create rolebinding openshift-acme --role=openshift-acme --serviceaccount="$( oc project -q ):openshift-acme" --dry-run -o yaml | oc apply -f -
sleep 5
oc apply -f docker-registry.yaml
# docker-registry.yaml
apiVersion: v1
kind: List
items:
- apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: s3-docker-registry
data:
REGISTRY_STORAGE_S3_ACCESSKEY: ***
REGISTRY_STORAGE_S3_SECRETKEY: ***
- apiVersion: v1
kind: Service
metadata:
labels:
app: docker-registry
app.kubernetes.io/component: docker-registry
app.kubernetes.io/instance: docker-registry
name: docker-registry
spec:
ports:
- name: 8443-tcp
port: 443
protocol: TCP
targetPort: 8443
selector:
deployment: docker-registry
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
- apiVersion: v1
kind: Service
metadata:
labels:
app: docker-registry
app.kubernetes.io/component: redis
app.kubernetes.io/instance: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: docker-registry
app.openshift.io/runtime-version: latest
name: redis
spec:
ports:
- name: 6379-tcp
port: 6379
protocol: TCP
targetPort: 6379
selector:
app: redis
deploymentconfig: redis
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
- apiVersion: route.openshift.io/v1
kind: Route
metadata:
annotations:
kubernetes.io/tls-acme: "true"
acme.openshift.io/secret-name: "docker-registry-tls"
labels:
app: docker-registry
app.kubernetes.io/component: docker-registry
app.kubernetes.io/instance: docker-registry
name: docker-registry
spec:
host: docker-registry.apps.example.io
port:
targetPort: 8443-tcp
tls:
insecureEdgeTerminationPolicy: Redirect
termination: passthrough
to:
kind: Service
name: docker-registry
weight: 100
wildcardPolicy: None
- apiVersion: apps/v1
kind: Deployment
metadata:
generation: 15
labels:
app: docker-registry
app.kubernetes.io/component: docker-registry
app.kubernetes.io/instance: docker-registry
app.kubernetes.io/part-of: docker-registry
name: docker-registry
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
deployment: docker-registry
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
openshift.io/generated-by: OpenShiftNewApp
creationTimestamp: null
labels:
deployment: docker-registry
spec:
containers:
- env:
- name: REGISTRY_HTTP_ADDR
value: 0.0.0.0:8443
- name: REGISTRY_HTTP_TLS_CERTIFICATE
value: /certs/tls.crt
- name: REGISTRY_HTTP_TLS_KEY
value: /certs/tls.key
- name: REGISTRY_STORAGE
value: s3
- name: REGISTRY_STORAGE_S3_ACCESSKEY
- name: REGISTRY_STORAGE_S3_BUCKET
value: example-docker-registry
- name: REGISTRY_STORAGE_S3_REGION
value: eu-west-1
- name: REGISTRY_STORAGE_S3_SECRETKEY
- name: REGISTRY_HTTP_SECRET
value: a random secret generated by hand
- name: REGISTRY_REDIS_ADDR
value: redis:6379
envFrom:
- secretRef:
name: s3-docker-registry
image: registry
imagePullPolicy: IfNotPresent
name: docker-registry
ports:
- containerPort: 8443
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/registry
name: docker-registry-volume-1
- mountPath: /certs
name: docker-registry-tls
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: docker-registry-volume-1
- name: docker-registry-tls
secret:
defaultMode: 420
secretName: docker-registry-tls
- apiVersion: apps/v1
kind: Deployment
name: redis
...
openshift-acme logs:
I1212 23:38:51.169507 1 openshift-acme-controller.go:192] No kubeconfig specified, using InClusterConfig.
I1212 23:38:51.171856 1 openshift-acme-controller.go:236] Managing namespaces: []string{"example-utils"}
I1212 23:38:51.172286 1 openshift-acme-controller.go:272] Leaderelection ID is "openshift-acme-5cf885c959-j2gt5_a18cdf74-f8ea-4fc3-a7dc-6d4291dda970"
I1212 23:38:51.172337 1 leaderelection.go:242] attempting to acquire leader lease example-utils/acme-controller-locks...
E1212 23:38:51.184525 1 leaderelection.go:331] error retrieving resource lock example-utils/acme-controller-locks: configmaps "acme-controller-locks" is forbidden: User "system:serviceaccount:example-utils:openshift-acme" cannot get resource "configmaps" in API group "" in the namespace "example-utils"
I1212 23:38:51.184547 1 leaderelection.go:247] failed to acquire lease example-utils/acme-controller-locks
I1212 23:39:07.385563 1 leaderelection.go:252] successfully acquired lease example-utils/acme-controller-locks
I1212 23:39:07.385628 1 openshift-acme-controller.go:329] Acquired leaderelection
I1212 23:39:07.385639 1 openshift-acme-controller.go:335] loglevel is set to "4"
I1212 23:39:07.385796 1 acme.go:89] Setting up kube informers for namespace "example-utils"
I1212 23:39:07.385903 1 route.go:136] Setting up route informers for namespace "example-utils"
I1212 23:39:07.385927 1 route.go:153] Setting up kube informers for namespace "example-utils"
I1212 23:39:07.385987 1 acme.go:114] Starting Account controller
I1212 23:39:07.385993 1 shared_informer.go:197] Waiting for caches to sync for account controller
I1212 23:39:07.386026 1 reflector.go:153] Starting reflector *v1.Secret (0s) from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386030 1 reflector.go:153] Starting reflector *v1.ConfigMap (0s) from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386040 1 reflector.go:188] Listing and watching *v1.Secret from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386045 1 reflector.go:188] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386035 1 reflector.go:153] Starting reflector *v1.LimitRange (0s) from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386077 1 reflector.go:188] Listing and watching *v1.LimitRange from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386080 1 reflector.go:153] Starting reflector *v1.Service (0s) from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386087 1 reflector.go:188] Listing and watching *v1.Service from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386033 1 reflector.go:153] Starting reflector *v1.ReplicaSet (0s) from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386211 1 reflector.go:188] Listing and watching *v1.ReplicaSet from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386220 1 route.go:1347] Starting Route controller
I1212 23:39:07.386223 1 reflector.go:153] Starting reflector *v1.Route (0s) from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.386228 1 shared_informer.go:197] Waiting for caches to sync for route controller
I1212 23:39:07.386232 1 reflector.go:188] Listing and watching *v1.Route from k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108
I1212 23:39:07.390220 1 acme.go:180] Adding ConfigMap example-utils/letsencrypt-staging UID=edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb RV=46127840
I1212 23:39:07.395158 1 route.go:214] Adding Route example-utils/docker-registry RV=46128006 UID=9bc42f0c-30a2-4632-b587-bc890b1db48a
I1212 23:39:07.486130 1 shared_informer.go:227] caches populated
I1212 23:39:07.486159 1 shared_informer.go:204] Caches are synced for account controller
I1212 23:39:07.486222 1 acme.go:271] Started syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:07.486316 1 shared_informer.go:227] caches populated
I1212 23:39:07.486334 1 shared_informer.go:204] Caches are synced for route controller
I1212 23:39:07.486383 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:07.486432 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.486471 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.486615 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.486627 1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.491395 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:07.491923 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.491977 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.492107 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.492122 1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.502472 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.502511 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.502601 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.502616 1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.523170 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.523218 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.523366 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.523387 1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.563808 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.563868 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.564019 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.564033 1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.644550 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.644612 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.644755 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.644771 1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.805042 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.805104 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.805228 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.805244 1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:08.125389 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:08.125466 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:08.125633 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:08.125652 1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:08.765793 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:08.765851 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:08.765953 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:08.765964 1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:09.137726 1 acme.go:50] By continuing running this program you agree to the CA's Terms of Service (https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf). If you do not agree exit the program immediately!
I1212 23:39:09.451178 1 acme.go:273] Finished syncing Account "example-utils/letsencrypt-staging"
E1212 23:39:09.451200 1 acme.go:157] example-utils/letsencrypt-staging failed with : secret "letsencrypt-staging" not found
I1212 23:39:09.451259 1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"example-utils", Name:"letsencrypt-staging", UID:"edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb", APIVersion:"v1", ResourceVersion:"46127840", FieldPath:""}): type: 'Normal' reason: 'AcmeAccountProvisioned' Provisioned new ACME account for issuer "example-utils/letsencrypt-staging" because its secret example-utils/letsencrypt-staging was missing.
E1212 23:39:09.453030 1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"letsencrypt-staging.16c02646b2bd087c", GenerateName:"", Namespace:"example-utils", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ConfigMap", Namespace:"example-utils", Name:"letsencrypt-staging", UID:"edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb", APIVersion:"v1", ResourceVersion:"46127840", FieldPath:""}, Reason:"AcmeAccountProvisioned", Message:"Provisioned new ACME account for issuer \"example-utils/letsencrypt-staging\" because its secret example-utils/letsencrypt-staging was missing.", Source:v1.EventSource{Component:"openshift-acme-acme-account-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc065bfe75ae4267c, ext:18287285589, loc:(*time.Location)(0x1ef3580)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc065bfe75ae4267c, ext:18287285589, loc:(*time.Location)(0x1ef3580)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:example-utils:openshift-acme" cannot create resource "events" in API group "" in the namespace "example-utils"' (will not retry!)
I1212 23:39:09.456304 1 acme.go:271] Started syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:09.891211 1 acme.go:410] Refreshed account object example-utils/letsencrypt-staging with data from ACME
I1212 23:39:09.897657 1 acme.go:193] Updating ConfigMap from example-utils/letsencrypt-staging UID=edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb RV=46127840 to example-utils/letsencrypt-staging UID=edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb,RV=46128167
I1212 23:39:09.897776 1 acme.go:273] Finished syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:09.897805 1 acme.go:271] Started syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:09.898269 1 acme.go:273] Finished syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:10.046150 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:10.046225 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:10.046443 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:10.646624 1 route.go:622] Created Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" for Route "example-utils/docker-registry"
I1212 23:39:10.647090 1 route.go:482] Updating status for Route example-utils/docker-registry to (*api.Status){ObservedGeneration:(int64)0 CertificateMeta:(*api.CertificateMeta)<nil> ProvisioningStatus:(api.CertProvisioningStatus){StartedAt:(time.Time)2021-12-12 23:39:10.646650622 +0000 UTC m=+19.482776539 EarliestAttemptAt:(time.Time)0001-01-01 00:00:00 +0000 UTC Failures:(int)0 OrderURI:(string)https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098 OrderStatus:(string)pending OrderError:(*api.OrderError)<nil> AccountHash:(string)} Signature:(string)}
I1212 23:39:10.660816 1 route.go:226] Updating Route example-utils/docker-registry RV=46128006->46128175 UID=9bc42f0c-30a2-4632-b587-bc890b1db48a->9bc42f0c-30a2-4632-b587-bc890b1db48a
I1212 23:39:10.660845 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.661080 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:10.661105 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:10.661228 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:10.661427 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:10.666873 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.666894 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.672001 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.677647 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.677663 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.688054 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.693478 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.693495 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.713614 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.719454 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.719471 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.759593 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.764840 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.764861 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.844979 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.850982 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.851002 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:11.011124 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:11.017311 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:11.017330 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:11.254352 1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:11.254380 1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:11.337463 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:11.343377 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:11.343395 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:11.406934 1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:11.406967 1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:11.407040 1 route.go:756] Exposer route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang not found, creating new one.
I1212 23:39:11.425042 1 route.go:762] Created exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang for Route example-utils/docker-registry
I1212 23:39:11.425091 1 route.go:812] Exposer secret example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang not found, creating new one.
I1212 23:39:11.429575 1 route.go:931] Exposer replica set example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang not found, creating new one.
I1212 23:39:11.441282 1 route.go:986] Exposer service example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang not found, creating new one.
I1212 23:39:11.451631 1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:11.451836 1 route.go:482] Updating status for Route example-utils/docker-registry to (*api.Status){ObservedGeneration:(int64)0 CertificateMeta:(*api.CertificateMeta)<nil> ProvisioningStatus:(api.CertProvisioningStatus){StartedAt:(time.Time)2021-12-12 23:39:10.646650622 +0000 UTC EarliestAttemptAt:(time.Time)2021-12-12 23:39:10.646650622 +0000 UTC Failures:(int)0 OrderURI:(string)https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098 OrderStatus:(string)pending OrderError:(*api.OrderError)<nil> AccountHash:(string)} Signature:(string)}
I1212 23:39:11.463416 1 route.go:226] Updating Route example-utils/docker-registry RV=46128175->46128197 UID=9bc42f0c-30a2-4632-b587-bc890b1db48a->9bc42f0c-30a2-4632-b587-bc890b1db48a
I1212 23:39:11.463453 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:11.465055 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:11.465095 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:11.465210 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:11.465405 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:11.472343 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:11.472368 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:11.983535 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:11.989825 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:11.989847 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:12.076813 1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:12.076851 1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:12.233865 1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:12.233897 1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:12.234006 1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:12.234231 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:12.234261 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:12.234339 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:12.234508 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:12.830891 1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:12.830921 1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:12.984471 1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:12.984501 1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:12.984597 1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:12.984854 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:14.550779 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:14.556672 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:14.556692 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:14.643382 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:14.643516 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:14.643819 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:15.249426 1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:15.249463 1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:15.407440 1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:15.407469 1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:15.407552 1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:15.407807 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:15.407833 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:15.407909 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:15.408026 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:16.014995 1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:16.015022 1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:16.167121 1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:16.167150 1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:16.167236 1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:16.167446 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:19.676934 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:19.682822 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:19.682838 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:26.451824 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:26.451974 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:26.452119 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:27.042360 1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:27.042399 1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:27.200379 1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:27.200414 1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:27.200519 1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:27.200794 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:29.922979 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:29.930752 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:29.930778 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:42.200736 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:42.200902 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:42.201098 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:42.790856 1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:42.790884 1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:42.944615 1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:42.944644 1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:42.944761 1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:42.945006 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:50.410919 1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:50.419740 1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:50.419757 1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:57.944974 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:57.945171 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:57.945398 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:58.543768 1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:58.543804 1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:58.701613 1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:58.701866 1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:58.701974 1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:58.702202 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:40:13.702144 1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:40:13.702306 1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:40:13.702448 1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:40:14.290053 1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:40:14.290086 1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:40:14.449247 1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:40:14.449283 1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:40:14.449392 1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:40:14.449680 1 route.go:498] Finished syncing Route "example-utils/docker-registry"
Anything else we need to know?:
I had this setup working for a few days (passthrough route + TLS secret mounted into a pod) but after recreating it in another namespace both stopped to work. No duplicate routes are present in any namespaces, double checked.
Environment:
- OpenShift/Kubernetes version :
Client Version: 4.8.13
Server Version: 4.8.20
Kubernetes Version: v1.21.4+6438632 - Others:
I'm also experiencing this in the context of RH ServiceMesh 2.0 on OpenShift 4.7. As @ntxt mentioned, this used to work. We noticed certificates are expired and not renewing. I see the same stanza with the exposer route not admitted due to HostAlreadyClaimed. In my case, the offending route is the one that gets created by RHSM (istio) Gateway.
(Note: RHSM copies annotations from the Gateway to the Openshift Route object. Details https://docs.openshift.com/container-platform/4.7/service_mesh/v2x/ossm-traffic-manage.html#ossm-auto-route-annotations_routing-traffic)
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: api-gateway
annotations:
kubernetes.io/tls-acme: "true"
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
httpsRedirect: true
hosts:
- api-foo-dev.example.com
- api-bar-dev.example.com
- api-baz-dev.example.com
- api-qixx-dev.example.com
- api-quxx-dev.example.com
That causes the main route to get created in the service mesh's namespace in addition to the exposer route. These two routes conflict. If I remove the annotation, only the main route exists.
$ oc get route -n develop-istio-system
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
api-gateway-1b49626edxx api-foo-dev.example.com istio-ingressgateway https passthrough/Redirect None
api-gateway-25ac42f2exx api-bar-dev.example.com istio-ingressgateway https passthrough/Redirect None
api-gateway-2e7f2c82bxx api-baz-dev.example.com istio-ingressgateway https passthrough/Redirect None
api-gateway-7142b43bcxx api-qixx-dev.example.com istio-ingressgateway https passthrough/Redirect None
api-gateway-ba7b8c932xx api-quxx-dev.example.com istio-ingressgateway https passthrough/Redirect None
exposer-4dc3fbdoncopqm03ssi4khkxx HostAlreadyClaimed /.well-known/acme-challenge/wnq3LxaXxxxx exposer-4dc3fbdoncopqm03ssi4khkxx <all> edge/Allow None
exposer-bo90lih5l4fnd336d163ur0xx HostAlreadyClaimed /.well-known/acme-challenge/2tvd6HrLxxxx exposer-bo90lih5l4fnd336d163ur0xx <all> edge/Allow None
exposer-osfb5gnj37o8rr68g24a8eqxx HostAlreadyClaimed /.well-known/acme-challenge/_pUV-Af0xxxx exposer-osfb5gnj37o8rr68g24a8eqxx <all> edge/Allow None
exposer-ota457irmli2gpdaetqbkbfxx HostAlreadyClaimed /.well-known/acme-challenge/51CLcTDyxxxx exposer-ota457irmli2gpdaetqbkbfxx <all> edge/Allow None
exposer-pjqm1ebiea7nn5p41tcptt6xx HostAlreadyClaimed /.well-known/acme-challenge/TFXPfY8Bxxxx exposer-pjqm1ebiea7nn5p41tcptt6xx <all> edge/Allow None
Any advice on a workaround is appreciated.