EKS: Kubernetes 1.21 404 error

[Kevin Lefevre]

Hi, I'm currently running EKS 1.21 with secrets store CSI as well as vault deployed with CSI and injector. I have updated the Kubernetes auth login to be compatible with the changes in 1.21 by settting the issuer which is somehting like this on EKS: https://oidc.eks.eu-west-1.amazonaws.com/id/REDACTED123456.

I have a vault cluster deployed in the vault namespace which is reachable at http://vault.vault:8200.

I have a KV store v2 with the following keys;

ssv/operator, inside I have to K/V, PK = <data> and SK=<data>

I have a policy giving access to ssv/* with read and list.

I have a vault role ssv-node binding service account node in namespace ssv giving access to the ssv/ K/V store.

I have tested with an injector on a pod and it works fine.

I have the following secret provider class:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: ssv-node
  namespace: ssv
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.vault:8200"
    roleName: "ssv-node"
    objects: |
      - objectName: "operator-key"
        secretPath: "ssv/operator"
        secretKey: "SK"
    secretObjects:
    - data:
     - key: "SK"
       objectName: "operator-key"
     secretName: ssv-node
     type: Opaque

and then the following inside my pods:

    - csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: ssv-node

...
      - mountPath: secrets                                                                                                                                                                                      
        name: secrets-store-inline  

In the pod description I get the following error:

 MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store object
s for pod ssv/node-0, err: rpc error: code = Unknown desc = error making mount request: couldn't read secret "operator-key": Error making API request.                                                           
                                                                                                        
URL: GET http://vault.vault:8200/v1/ssv/operator
Code: 404. Errors:                                 

[Kevin Lefevre]

I seems to work with :

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: ssv-node
  namespace: ssv
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.vault:8200"
    roleName: "ssv-node"
    objects: |
      - objectName: operator-key
        secretPath: ssv/data/operator
        secretKey: SK
  secretObjects:
  - data:
    - key: SK
      objectName: operator-key
    secretName: ssv-node
    type: Opaque

[Greg Taylor]

I wonder whether this may break in the future if we manually stick that data in there :/

[Kevin Lefevre]

@gtaylor what is the proper way of mounting secret with kv v2?

[Greg Taylor]

I think there's a bug, so this may be the only way. Once the bug is fixed, our workaround may no longer work.

I'm using the same workaround, not trying to suggest an alternative (I don't know of any, aside from fixing the bug).

[Tom Proctor]

The TL;DR here is that you've arrived at the correct solution, and we have no plans to change how that works. Read on for more context and detail.

Previously, the provider only supported KV secret engines, and did automatic detection of which KV version (1 or 2) was in use, and inserted a data/ element into the path accordingly. See the docs for reading KV v2 secret contents for context on why there is a data/ element at all. This feature also required an additional round trip to Vault and extra permissions to query the API endpoint that advertises which version of KV is in use.

As part of adding support for all secret engines to the provider, we removed that feature, and the path specified in the CRD is always what the provider will use for its API query. See #35 for some additional context too. Elsewhere in the Vault ecosystem, there are KV v2-aware examples where the data/ element is not required, such as vault kv get ssv/operator, which understandably causes a little confusion, but I'd recommend using the API docs as the reference point for what paths you set in the CRD. Note that in the CLI, you can also use vault read ssv/data/operator, and so the raw read/write etc sub-commands are the closer analogue to what the CSI provider is doing. Hope this helps clear it up!

[Sébastien OUDRY]

Thanks @tomhjp for this explanation.

Addind data/ was exactly the solution found yesterday around the same time after hours of investigation and troubleshooting 😄

[Aidan]

Would be nice if this was called out in the examples somewhere. I am comparing the vault injector to the vault csi provider, and one requires /data/ and the other does not. Glad I found this issue. I guess technically it is captured in the getting started docs but maybe it could be emphasized.