Giters

GoogleCloudPlatform / gke-managed-certs

Managed Certificates for Kubernetes clusters using GCLB

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Multiple Certs not working: Only picks up the first managedcert in the list

domparry opened this issue · comments

commented

The following config for my ingress create a LB with only the first cert. If I swap them around, I get the other one:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: om-static-ip
    networking.gke.io/managed-certificates: om-ssl-google-managed,om-no-www-ssl-google-managed
  name: om-prod-ssl
  namespace: default
spec:
  rules:
  - host: www.temp-om.simply.co.za
    http:
      paths:
      - backend:
          serviceName: om-tenandsix-prod
          servicePort: 8080
  - host: temp-om.simply.co.za
    http:
      paths:
      - backend:
          serviceName: om-tenandsix-prod
          servicePort: 8080

The resulting annotations copied from the ingress on cloud console:

ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-7d7ac878-3a4f-4fe7-b23d-483813bb6ac0
ingress.kubernetes.io/backends: {"k8s-be-30009--4d15a37c4c5becdc":"HEALTHY","k8s-be-31353--4d15a37c4c5becdc":"HEALTHY"}
ingress.kubernetes.io/forwarding-rule: k8s-fw-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/https-forwarding-rule: k8s-fws-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/https-target-proxy: k8s-tps-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/ssl-cert: mcrt-7d7ac878-3a4f-4fe7-b23d-483813bb6ac0
ingress.kubernetes.io/target-proxy: k8s-tp-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/url-map: k8s-um-default-om-prod-ssl--4d15a37c4c5becdc
kubernetes.io/ingress.global-static-ip-name: om-static-ip
networking.gke.io/managed-certificates: om-ssl-google-managed,om-no-www-ssl-google-managed```
commented

This has suddenly started working now...

commented

Hi,

we have currently the same problem. We have opened a case at the GCP support.

GKE Version: v1.14.10-gke.17

Here our configuration:

kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: images-recognition
  namespace: images-recognition
  annotations:
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: ingress-images-recognition
    networking.gke.io/managed-certificates: cert-imgreco,cert-imgreco-ancien-neuf,cert-imgreco-exterieur-elements,cert-imgreco-interieur-elements,cert-imgreco-interieur-matieres,cert-imgreco-interieur-pieces,cert-imgreco-visuels
spec:
  backend:
    serviceName: images-recognition
    servicePort: 5000
  rules:
    - host: imgreco.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition
            servicePort: 5000
    - host: imgreco-ancien-neuf.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-ancien-neuf
            servicePort: 5000
    - host: imgreco-exterieur-elements.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-exterieur-elements
            servicePort: 5000
    - host: imgreco-interieur-elements.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-interieur-elements
            servicePort: 5000
    - host: imgreco-interieur-matieres.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-interieur-matieres
            servicePort: 5000
    - host: imgreco-interieur-pieces.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-interieur-pieces
            servicePort: 5000
    - host: imgreco-visuels.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-visuels
            servicePort: 5000

Here the annotations:

ingress.kubernetes.io/https-forwarding-rule:       k8s-fws-images-recognition-images-recognition--bd24109445b008c0
ingress.kubernetes.io/backends:                    {"k8s1-bd241094-images-recog-images-recognition-exter-50-e31b7b75":"HEALTHY","k8s1-bd241094-images-recog-images-recognition-inter-50-5999c0d4":"HEALTHY","k8s1-bd241094-images-recog-images-recognition-inter-50-ebf20c03":"HEALTHY","k8s1-bd241094-images-recog-images-recognition-inter-50-fcba0df3":"HEALTHY","k8s1-bd241094-images-recogni-images-recognition-anc-50-41a97672":"HEALTHY","k8s1-bd241094-images-recognit-images-recognition-v-500-715a016e":"HEALTHY","k8s1-bd241094-images-recognition-images-recognitio-500-8ce5ddaa":"HEALTHY"}
ingress.kubernetes.io/https-target-proxy:          k8s-tps-images-recognition-images-recognition--bd24109445b008c0
ingress.kubernetes.io/ssl-cert:                    mcrt-1aa80f1a-174e-4f5f-9b94-a40d777d2a92
ingress.kubernetes.io/url-map:                     k8s-um-images-recognition-images-recognition--bd24109445b008cf
kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.allow-http":"false","kubernetes.io/ingress.global-static-ip-name":"ingress-images-recognition","networking.gke.io/managed-certificates":"cert-imgreco, cert-imgreco-ancien-neuf, cert-imgreco-exterieur-elements, cert-imgreco-interieur-elements, cert-imgreco-interieur-matieres, cert-imgreco-interieur-pieces, cert-imgreco-visuels"},"name":"images-recognition","namespace":"images-recognition"},"spec":{"backend":{"serviceName":"images-recognition","servicePort":5000},"rules":[{"host":"imgreco.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-ancien-neuf.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-ancien-neuf","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-exterieur-elements.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-exterieur-elements","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-interieur-elements.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-interieur-elements","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-interieur-matieres.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-interieur-matieres","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-interieur-pieces.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-interieur-pieces","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-visuels.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-visuels","servicePort":5000},"path":"/*"}]}}]}}

kubernetes.io/ingress.allow-http:             false
kubernetes.io/ingress.global-static-ip-name:  ingress-images-recognition
networking.gke.io/managed-certificates:       cert-imgreco,cert-imgreco-ancien-neuf,cert-imgreco-exterieur-elements,cert-imgreco-interieur-elements,cert-imgreco-interieur-matieres,cert-imgreco-interieur-pieces,cert-imgreco-visuels
ingress.gcp.kubernetes.io/pre-shared-cert:    mcrt-1aa80f1a-174e-4f5f-9b94-a40d777d2a92

For us, only the last one is took into account.

Any idea ?

commented

@domparry Could you reopen the case ?

commented

We're now on 1.15.8-gke.3, and it works well with the following:

metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: om-static-ip
    networking.gke.io/managed-certificates: om-collections-no-www-ssl-google-managed,om-group-www-ssl-google-managed,om-group-no-www-ssl-google-managed,om-admin-no-www-ssl-google-managed,om-admin-www-ssl-google-managed,om-home-ssl-google-managed,om-home-no-www-ssl-google-managed,om-callcentre-no-www-ssl-google-managed,om-postoffice-no-www-ssl-google-managed,om-app-no-www-ssl-google-managed
  name: om-prod-ssl
  namespace: default

Which results in the following annotations:

ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-118fd68b-4134-4694-968f-a19b26695427,mcrt-19374f70-94c9-4128-b540-d09a48311af1,mcrt-3389cdda-f3f8-45a0-89f3-7bd6c042d713,mcrt-3547e7ac-5b1e-4739-a743-ef6f247fa348,mcrt-3b1faa81-0deb-498a-a339-56249fbd83bf,mcrt-701e17a8-a3fe-4ec2-afc8-cf740878bc30,mcrt-75b73c73-0e27-4420-a691-5ccb13a9cbff,mcrt-91396b87-1137-414d-936c-02a297727fe0,mcrt-9668cbe6-b532-42bf-8741-c4b368741a29,mcrt-ba63e6ef-65e9-4266-8f46-29925759710d
ingress.kubernetes.io/backends: {"k8s-be-30009--4d15a37c4c5becdc":"HEALTHY","k8s-be-31029--4d15a37c4c5becdc":"HEALTHY","k8s-be-31353--4d15a37c4c5becdc":"HEALTHY","k8s-be-31438--4d15a37c4c5becdc":"HEALTHY","k8s-be-31522--4d15a37c4c5becdc":"HEALTHY","k8s-be-32031--4d15a37c4c5becdc":"HEALTHY","k8s-be-32676--4d15a37c4c5becdc":"HEALTHY"}
ingress.kubernetes.io/forwarding-rule: k8s-fw-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/https-forwarding-rule: k8s-fws-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/https-target-proxy: k8s-tps-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/ssl-cert: mcrt-118fd68b-4134-4694-968f-a19b26695427,mcrt-19374f70-94c9-4128-b540-d09a48311af1,mcrt-3389cdda-f3f8-45a0-89f3-7bd6c042d713,mcrt-3547e7ac-5b1e-4739-a743-ef6f247fa348,mcrt-3b1faa81-0deb-498a-a339-56249fbd83bf,mcrt-701e17a8-a3fe-4ec2-afc8-cf740878bc30,mcrt-75b73c73-0e27-4420-a691-5ccb13a9cbff,mcrt-91396b87-1137-414d-936c-02a297727fe0,mcrt-9668cbe6-b532-42bf-8741-c4b368741a29,mcrt-ba63e6ef-65e9-4266-8f46-29925759710d
ingress.kubernetes.io/target-proxy: k8s-tp-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/url-map: k8s-um-default-om-prod-ssl--4d15a37c4c5becdc
kubernetes.io/ingress.global-static-ip-name: om-static-ip
networking.gke.io/managed-certificates: om-collections-no-www-ssl-google-managed,om-group-www-ssl-google-managed,om-group-no-www-ssl-google-managed,om-admin-no-www-ssl-google-managed,om-admin-www-ssl-google-managed,om-home-ssl-google-managed,om-home-no-www-ssl-google-managed,om-callcentre-no-www-ssl-google-managed,om-postoffice-no-www-ssl-google-managed,om-app-no-www-ssl-google-managed```

On a different cluster however, I've opted to use a wildcard cert which works really well.
commented

You say that you use a Google Managed Wilcard Cert ?
I was thinking it isn't possible to do wildcard, are you sure ? Perhaps, you use a normal wilcard cert, not autogenerated ?

commented

Sorry @vrobert78 , I meant a multi domain cert, not a wildcard cert. It's defined like this:

gcloud beta compute ssl-certificates create cert-name --project=projectId --domains domain1.co.za,domain2.co.za,domain3.co.za

used like this:

metadata:
  annotations:
    ingress.kubernetes.io/ssl-cert: cert-name
commented

Ok. I understand.

We tested to create one, but unfortunately it's not supported yet in 1.14.10-gke.17.

commented

Hi, we solved the problem thanks the the Google Support.

We had to delete the certs in errors, but not by deleting the managedcertificates.networking.gke.io.
Instead, the resource to be deleted is mcrt.

You have to do a: kubectl delete mcrt xxx, wait 2 minutes, then recreate the cert by reapplying your yml.

commented

It was caused by a bug, which is already fixed in 0.4.2/GKE GKE 1.16.8-gke.3, sorry. See #45 for more info. I'm closing this issue and let's continue the discussion there.