Pass Vault agent annotations to Consul API Gateway controller

[Rosemary Wang]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

I deployed the Consul Helm chart with the Vault secrets backend and Consul API Gateway enabled. The Consul API Gateway was stuck initializing because the Vault agent needed to authenticate to HCP Vault, which requires a Vault namespace annotation.

.Values.global.secretsBackend.vault.agentAnnotations must to be added to api-gateway-controller-deployment.yaml to allow the Vault agent to properly authenticate to Vault.

Reproduction Steps

1. Run helm install with the following values.yml:

global:
  datacenter: "${CONSUL_DATACENTER}"
  name: consul
  secretsBackend:
    vault:
      enabled: true
      consulServerRole: ${CONSUL_SERVER_ROLE}
      consulClientRole: ${CONSUL_CLIENT_ROLE}
      consulCARole: ${CONSUL_CA_ROLE}
      manageSystemACLsRole: ${SERVER_ACL_INIT_ROLE}
      agentAnnotations: |
        "vault.hashicorp.com/namespace": "${VAULT_NAMESPACE}"
      connectCA:
        address: ${VAULT_ADDR}
        rootPKIPath: ${CONSUL_CONNECT_PKI_PATH_ROOT}
        intermediatePKIPath: ${CONSUL_CONNECT_PKI_PATH_INT}
        authMethodPath: ${KUBERNETES_AUTH_METHOD_PATH}
        additionalConfig: '"{"connect": [{ "ca_config": [{ "namespace": "${VAULT_NAMESPACE}"}]}]}"'
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: "${CONSUL_PKI_PATH}/cert/ca"
    caKey:
      secretName: "${CONSUL_PKI_PATH}/issue/${CONSUL_SERVER_ROLE}"
      secretKey: private_key
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: "${CONSUL_STATIC_PATH}/data/bootstrap"
      secretKey: token
  gossipEncryption:
    secretName: ${CONSUL_STATIC_PATH}/data/gossip
    secretKey: key

server:
  replicas: 1
  serverCert:
    secretName: "${CONSUL_PKI_PATH}/issue/${CONSUL_SERVER_ROLE}"

apiGateway:
  enabled: true
  image: "hashicorp/consul-api-gateway:0.2.1"
  managedGatewayClass:
    serviceType: LoadBalancer
    useHostPorts: true

3. The Consul API Gateway controller is stuck initializing.

$ kubectl get pods

NAME                                             READY   STATUS     RESTARTS   AGE
consul-api-gateway-controller-849649d5b8-vns64   0/2     Init:0/4   0          48s

Logs

Logs

$ kubectl logs consul-api-gateway-controller-849649d5b8-vns64 -c vault-agent-init
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.10.3
             Version Sha: af866591ee60485f05d6e32dd63dde93df686dfb

2022-05-26T20:19:00.146Z [INFO]  sink.file: creating file sink
2022-05-26T20:19:00.146Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2022-05-26T20:19:00.146Z [INFO]  auth.handler: starting auth handler
2022-05-26T20:19:00.146Z [INFO]  auth.handler: authenticating
2022-05-26T20:19:00.147Z [INFO]  template.server: starting template server
2022-05-26T20:19:00.147Z [INFO]  sink.server: starting sink server
2022-05-26T20:19:00.147Z [INFO] (runner) creating new runner (dry: false, once: false)
2022-05-26T20:19:00.148Z [INFO] (runner) creating watcher
2022-05-26T20:19:00.164Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  | 
  | URL: PUT https://REDACTED:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  | 
  | * permission denied
   backoff=1s

Expected behavior

Consul API Gateway should be running.

Environment details

• Consul Helm chart 0.44.0
• Consul 1.12.0
• HCP Vault (Vault version 1.10.3)
• Kubernetes v1.22.6-eks-14c7a48
• Cloud Provider EKS

Additional Context

Currently, I am patching the deployment with:

kubectl patch deployment consul-api-gateway-controller -p '{"spec": {"template":{"metadata":{"annotations":{"vault.hashicorp.com/namespace":"admin"}}}}}'

This allows the Vault agent container to authenticate correctly to Vault and inject the certificates for the controller.