Giters

hashicorp / vault-secrets-operator

The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.

Home Page:https://hashicorp.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Problems encountered during renewal after restarting the vault operator

towithyou opened this issue · comments

commented

Describe the bug
After restarting the vault operator, the lease will be automatically deleted at night. If the service is not restarted, long-term operation can renew the lease normally

Vault version
vault operator 0.4.3
vault server 1.15.4

VDS yaml， Normally, it will expire at 2024-05-11 10:10:34

Application deployment:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"secrets.hashicorp.com/v1beta1","kind":"VaultDynamicSecret","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"a-cloud-task-platform-online"},"name":"a-cloud-task-platform","namespace":"devops"},"spec":{"destination":{"create":true,"name":"a-cloud-task-platform-secret"},"mount":"a-cloud-db","path":"creds/task-platform-rw","renewalPercent":80,"rolloutRestartTargets":[{"kind":"Deployment","name":"a-cloud-task-platform"}],"vaultAuthRef":"devops/a-cloud-task-platform"}}
  creationTimestamp: "2024-05-08T02:10:33Z"
  finalizers:
  - vaultdynamicsecret.secrets.hashicorp.com/finalizer
  generation: 2
  labels:
    argocd.argoproj.io/instance: a-cloud-task-platform-online
  name: a-cloud-task-platform
  namespace: devops
  resourceVersion: "286348555"
  uid: eb9045a7-4f84-4562-a606-0019247ba6ed
spec:
  destination:
    create: true
    name: a-cloud-task-platform-secret
    overwrite: false
    transformation: {}
  mount: a-cloud-db
  path: creds/task-platform-rw
  renewalPercent: 80
  rolloutRestartTargets:
  - kind: Deployment
    name: a-cloud-task-platform
  vaultAuthRef: devops/a-cloud-task-platform
status:
  lastGeneration: 2
  lastRenewalTime: 1715134234
  lastRuntimePodUID: cfd3d88f-4a6f-4684-a8d2-e0c4d230bd01
  secretLease:
    duration: 259200
    id: a-cloud-db/creds/task-platform-rw/lONHRDFmJb3KHCoCa1kym3C1
    renewable: true
    requestID: 5ee6cda8-1efa-d2c7-f86c-22fd96f829f2
  staticCredsMetaData:
    lastVaultRotation: 0
    rotationPeriod: 0
    rotationSchedule: ""
    ttl: 0
---
kind: VaultAuth
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"secrets.hashicorp.com/v1beta1","kind":"VaultAuth","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"a-cloud-task-platform-online"},"name":"a-cloud-task-platform","namespace":"devops"},"spec":{"kubernetes":{"role":"a-cloud-task-platform","serviceAccount":"default"},"method":"kubernetes","mount":"group-k8s","vaultConnectionRef":"vault-connection"}}
  creationTimestamp: "2023-12-18T03:44:19Z"
  finalizers:
  - vaultauth.secrets.hashicorp.com/finalizer
  generation: 1
  labels:
    argocd.argoproj.io/instance: a-cloud-task-platform-online
  name: a-cloud-task-platform
  namespace: devops
  resourceVersion: "163972247"
  uid: 0827ee9a-20b2-49cb-871a-a4a249e7cd8d
spec:
  kubernetes:
    role: a-cloud-task-platform
    serviceAccount: default
    tokenExpirationSeconds: 600
  method: kubernetes
  mount: group-k8s
  vaultConnectionRef: vault-connection
status:
  error: ""
  valid: true

Vault server logs
2024-05-08T23:13:10.305Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-08T23:23:10.354Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-08T23:33:10.469Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-08T23:43:10.521Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-08T23:53:10.568Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T00:03:10.617Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T00:03:41.059Z [INFO] expiration: revoked lease: lease_id=auth/group-k8s/login/hde2c354672e648b920ec415f5c271d9e15da25655a3d2daf9373b3455efa4798
2024-05-09T00:03:41.078Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/task-platform-rw/lONHRDFmJb3KHCoCa1kym3C1
2024-05-09T00:04:22.233Z [INFO] expiration: revoked lease: lease_id=auth/group-k8s/login/h78af65f30b7af74ce2b7230cf94717a9f673973239ef9c63d3d1d0a963d0f4e1
2024-05-09T00:04:22.241Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/workflow-rw/gmGCdiSReSYLAAUYbwzRfKyy
2024-05-09T00:13:10.665Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T00:23:10.715Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T00:28:29.110Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/service-tree-rw/wyYsCXdvlieoU7DPIObv0PZC
2024-05-09T00:28:29.120Z [INFO] expiration: revoked lease: lease_id=auth/group-k8s/login/h6f1a708218e6e2ab23d6e2d0c755666f63da531ff7eeb2b3d94ab12e99a51b1d
2024-05-09T00:28:29.123Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/service-tree-rw/yyCMX1FRYrnNbovVUFzVsNrq
2024-05-09T00:33:10.822Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T00:40:49.342Z [INFO] expiration: revoked lease: lease_id=auth/group-k8s/login/hc319ff3f9273e3b9f61d146053c0a3e4707ad29a6518776b1b13fa860d3b9d40
2024-05-09T00:43:10.868Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T00:53:10.918Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T01:03:10.963Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T01:13:11.012Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T01:13:37.046Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/db-manager-r/iJdtQ4lt7iI61jXr0EDCASs9
2024-05-09T01:23:11.054Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T01:33:11.178Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T01:43:11.225Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T01:53:11.277Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T02:02:49.405Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/resource-utilization-rw/3fbCmUPGxt59MSbIYPaDf6KG
2024-05-09T02:03:11.326Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T02:13:11.376Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
2024-05-09T02:23:11.426Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms

Vault operator logs, From the logs, there are no obvious errors
2024-05-08T23:37:45Z DEBUG events Rollout restart triggered for {Deployment a-cloud-cmdb} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"286978772"}, "reason": "RolloutRestartTriggered"}
2024-05-09T00:23:12Z DEBUG events Lease renewal duration was truncated from 3600s to 682s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"286977175"}, "reason": "SecretLeaseRenewal"}
2024-05-09T00:23:12Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_service_tree-rw/OZWMNxMvQSwTSqCmpItunb5d", horizon=50m13.530685162s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287002316"}, "reason": "SecretRotated"}
2024-05-09T00:23:12Z DEBUG events Rollout restart triggered for {Deployment a-cloud-service-tree} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287002316"}, "reason": "RolloutRestartTriggered"}
2024-05-09T00:28:42Z DEBUG events Lease renewal duration was truncated from 3600s to 543s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"286978772"}, "reason": "SecretLeaseRenewal"}
2024-05-09T00:28:42Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_cmdb-rw/71By43ksdbAkBiyx7GzLPfxw", horizon=53m49.031526697s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287005035"}, "reason": "SecretRotated"}
2024-05-09T00:28:42Z DEBUG events Rollout restart triggered for {Deployment a-cloud-cmdb} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287005035"}, "reason": "RolloutRestartTriggered"}
2024-05-09T01:13:26Z DEBUG events Lease renewal duration was truncated from 3600s to 586s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287002316"}, "reason": "SecretLeaseRenewal"}
2024-05-09T01:13:26Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_service_tree-rw/s0pZr0mdRbGdyAM6Md5LWKGL", horizon=53m45.735179044s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028179"}, "reason": "SecretRotated"}
2024-05-09T01:13:26Z DEBUG events Rollout restart triggered for {Deployment a-cloud-service-tree} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028179"}, "reason": "RolloutRestartTriggered"}
2024-05-09T01:14:53Z DEBUG events Lease renewal duration was truncated from 7200s to 928s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-workflow","uid":"d34f9a84-1ac9-4eb8-9297-d963624763da","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"286975094"}, "reason": "SecretLeaseRenewal"}
2024-05-09T01:14:53Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_workflow-rw/6zJP1ph0j6akcxD5Dwxl4LzT", horizon=1h38m55.068554041s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-workflow","uid":"d34f9a84-1ac9-4eb8-9297-d963624763da","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028899"}, "reason": "SecretRotated"}
2024-05-09T01:14:53Z DEBUG events Rollout restart triggered for {Deployment a-cloud-workflow} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-workflow","uid":"d34f9a84-1ac9-4eb8-9297-d963624763da","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028899"}, "reason": "RolloutRestartTriggered"}
2024-05-09T01:22:31Z DEBUG events Lease renewal duration was truncated from 3600s to 371s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287005035"}, "reason": "SecretLeaseRenewal"}
2024-05-09T01:22:32Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_cmdb-rw/oVcjMlwDDNPaKV0nNWblE5Rd", horizon=51m11.582480918s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287032970"}, "reason": "SecretRotated"}
2024-05-09T01:22:32Z DEBUG events Rollout restart triggered for {Deployment a-cloud-cmdb} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287032970"}, "reason": "RolloutRestartTriggered"}
2024-05-09T02:07:11Z DEBUG events Lease renewal duration was truncated from 3600s to 375s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028179"}, "reason": "SecretLeaseRenewal"}
2024-05-09T02:07:12Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_service_tree-rw/4gZA9LVKzopkRJw4qjgyRQCV", horizon=50m52.643550801s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287055823"}, "reason": "SecretRotated"}
2024-05-09T02:07:12Z DEBUG events Rollout restart triggered for {Deployment a-cloud-service-tree} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287055823"}, "reason": "RolloutRestartTriggered"}
2024-05-09T02:13:43Z DEBUG events Lease renewal duration was truncated from 3600s to 529s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287032970"}, "reason": "SecretLeaseRenewal"}
2024-05-09T02:13:43Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_cmdb-rw/M6SR2Lw3xE5bXLQtHHAIdytq", horizon=51m20.519425641s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287059288"}, "reason": "SecretRotated"}
2024-05-09T02:13:43Z DEBUG events Rollout restart triggered for {Deployment a-cloud-cmdb} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287059288"}, "reason": "RolloutRestartTriggered"}

commented

Just to add
vault k8s auth token ttl 1 day
vault secret db ttl and max_ttl 3 day
Is the auth ttl time less than the secret ttl, which may have an impact when restarting the service？

commented

Hi @towithyou, it looks like you are running an older VSO release, would you be able to upgrade to v0.6.0 (current latest) and see if the problem persists?

Thanks,

Ben

commented

Thank you for your reply. Upgrading to the latest version(v0.6.0) still has issues after restart vault operator. I have learned from reviewing the source code that the connection of the vault client will be cached in memory. When the service restarts, a new vault client will be reinitialized. At this time, the previous vault auth ttl will automatically revoke the dynamic secret lease after it expires. My suggestion is to set the auth token ttl to be greater than or equal to the dynamic secret ttl. Is my understanding correct

commented

@towithyou - I see, if you are syncing VaultDynamicSecrets you will need to enable client cache storage. When the storage is enabled new VSO leaders will pick up the token renewals on when a new election occur, or whenever a new VSO Pod is replaced. Please see https://developer.hashicorp.com/vault/docs/platform/k8s/vso/sources/vault/client-cache for more information.

commented

I configured and solved this problem according to the document requirements. Thank you very much for your support