VSO is working fine but controller throws Invalid client error

[Parth Sharma]

I deployed VSO in its own namespace & a test-app is separate namespace. After all the configuration VSO is working fine, it is creating & rotating the secrets but vault-secrets-operator-controller-manager is throwing below error

ERROR   cachingClientFactory    Invalid client  {"controller": "vaultstaticsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultStaticSecret", "VaultStaticSecret": {"name":"vss-app-mysecret","namespace":"vso-testapp"}, "namespace": "vso-testapp", "name": "vss-app-mysecret", "reconcileID": "5129e8a3-90e5-49e0-aeb1-6e1ff5e0bc6f", "cacheKey": "approle-e414c5e7b83c6eaa17acfe", "error": "lifetime watcher not set"}
github.com/hashicorp/vault-secrets-operator/internal/vault.(*cachingClientFactory).Get
        /home/runner/work/vault-secrets-operator/vault-secrets-operator/internal/vault/client_factory.go:355
github.com/hashicorp/vault-secrets-operator/controllers.(*VaultStaticSecretReconciler).Reconcile
        /home/runner/work/vault-secrets-operator/vault-secrets-operator/controllers/vaultstaticsecret_controller.go:73
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227
{"level":"info","ts":"2024-05-02T18:21:36Z","logger":"clientCacheFactory","msg":"Handling client cache eviction","persist":false,"enforceEncryption":false,"cacheKey":"approle-e414c5e7b83c6eaa17acfe"}
2024-05-02T18:21:36Z    INFO    Close() called  {"id": ""}

We deployed VSO helm chart version: 0.6.0 on k8s version: v1.28.7

Below are the overrides

vault-secrets-operator:
  metricsService: {}
  telemetry: {}
  defaultVaultConnection:
    enabled: true
    address: "xxxxx"                        # enterprise version of vault. Outside our k8s cluster 
  defaultAuthMethod:
    enabled: true
    namespace: "abc"
    allowedNamespaces: []
    method: appRole
    mount: vso
    appRole:
      roleId: 'xxxx'                              # autheticating using approle 
      secretRef: vso-vault-secret       # this k8s secret deployed in app namespace

VSS

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vss-app-mysecret
  namespace: vso-testapp
spec:
  hmacSecretData: true
  namespace: "abc"
  mount: xxx                                          # secret-engine path
  path: testApp/mysecret
  type: kv-v2
  refreshAfter: 10s
  destination:
    create: true
    name: mysecret
  rolloutRestartTargets:
  - kind: Deployment
    name: test-app

vaultConnection' status

 status:
    valid: true

But vaultAuth' status

status:
  error: ""                 # does this means some error 
  valid: true

[Ben Ash]

@iamps5 thanks for reporting this issue! It looks like VSO is not properly handling a non-renewable Vault token. We should have a fix out for this soon. In the meantime, would you mind confirming your Vault authentication role setup? What method are you using, etc? Mostly wondering what is set for https://developer.hashicorp.com/vault/api-docs/auth/kubernetes#token_type (assuming K8s auth)

[Ben Ash]

@iamps5 - from the logs you provided it looks like you are using appRole, it would be great if you could provide us with the role's configuration for https://developer.hashicorp.com/vault/api-docs/auth/approle#token_type

[Parth Sharma]

@benashz thanks for quick response. Below is the roles' configuration

auth/vso/role/dev \
policies="default","vso-dev","vso-global" \
token_num_uses=100000000 \
token_max_ttl="48h" \
token_ttl="24h" \
secret_id_num_uses=0 \
secret_id_ttl="2160h"

[Ben Ash]

Released in https://github.com/hashicorp/vault-secrets-operator/releases/tag/v0.7.0