rollout in cycle with enabled rolloutRestartTargets
yotles opened this issue · comments
I've tried to use VaultStaticSecret
to sync secrets from vault. And for proper rotating keys in secret I want to trigger deployment after updating secret in Vault. For this purpose I've added RolloutRestartTriggered
to VaultStaticSecret
. But even if I pinned version for secret kv-v2 my deployment triggered each minutes with logs Rollout restart triggered for {Deployment ***}
. If I add for example to description RefreshAfter: 10s
, it will be triggered each 10s.
At this time in logs of vault-secret-operator:
DEBUG events Rollout restart triggered for {Deployment ***} {"type": "Normal", "object": {"kind":"VaultStaticSecret","namespace":"***","name":"***","uid":"***","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"***"}, "reason": "RolloutRestartTriggered"} DEBUG events Secret synced {"type": "Normal", "object": {"kind":"VaultStaticSecret","namespace":"***","name":"***","uid":"***","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"***"}, "reason": "SecretRotated"}
To stop this behavior I've found only one solution, configure RefreshAfter for long period, for example 24h, in this way it works as expected, rollout restart happened only once when secret really changed with rotating version.
My VaultStaticSecret:
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: ***
namespace: ***
spec:
type: kv-v2
mount: ***
path: ***
destination:
name: ***
create: true
version: 4
vaultAuthRef: ***
rolloutRestartTargets:
- kind: Deployment
name: ***
Expected behavior
RolloutRestartTriggered happened only when I've changed version of secret in VaultStaticSecret.
Environment
- Kubernetes version: v1.24.0
- Distribution: OpenShift
- vault-secrets-operator version: 0.4.1
@benashz Thank you for replay. As I can see https://github.com/hashicorp/vault-secrets-operator/blob/v0.4.1/chart/crds/secrets.hashicorp.com_vaultstaticsecrets.yaml hmacSecretData=true
is default value. But even if I set it explicitly in VaultStaticSecret config and configured for example refreshAfter: 1m
still get recreating pod each sync secret within 1 minute.
@yotles We had the same problem. After updating to version 0.5.0
it went away, but remember to update the CRD!
@bhaku Yep, looks like this problem disappeared after updating to 0.5.1
. Much appreciate for you advice!
Same issue here. We have deployments being rollout restarted every 30s (which is the refreshAfter
value)
Here's the static secret yaml we have:
spec:
destination:
create: true
name: secretkv
overwrite: false
hmacSecretData: true
mount: ***/
path: ***
refreshAfter: 30s
rolloutRestartTargets:
- kind: Deployment
name: ***
type: kv-v2
vaultAuthRef: static-auth
Updated to v0.5.1 didn't solve this issue.
@jameshwc did you remember about the CRD update? They will not be updated when you update the version using helm.
On version 0.5.2
, CRDs have been updated, but
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: argocd-secret
spec:
vaultAuthRef: argocd
mount: secret
type: kv-v2
path: argocd/argocd-secret
destination:
create: true
overwrite: true
name: argocd-secret
rolloutRestartTargets:
- kind: Deployment
name: argocd-server
results in argocd-server
coming up, working, and once confirmed working VSO triggers another rolling restart. This happens in an infinite ~1 min cycle so no downtime is detected, but we have some 10,000 revisions of the deployment over the last few days.