ArgoCD and GCP KMS not decrypting

[Daniel Mühlbachler-Pietrzykowski]

I have installed ArgoCD using the Helm Chart custom tooling resulting in my Helm values.yaml file like this:

  server:
    config:
      kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

  repoServer:
    env:
      - name: XDG_CONFIG_HOME
        value: /.config
      - name: GOOGLE_APPLICATION_CREDENTIALS
        value: /ksops-credentials/credentials.json
    volumes:
      - name: custom-tools
        emptyDir: {}
      - name: ksops-credentials
        secret:
          secretName: ksops-credentials
    initContainers:
      - name: install-ksops
        image: viaductoss/ksops:v4
        imagePullPolicy: Always
        command:
          - "/bin/sh"
          - "-c"
        args:
          - echo "Installing KSOPS...";
            mv ksops /custom-tools/;
            mv $GOPATH/bin/kustomize /custom-tools/;
            echo "Done.";
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools
    volumeMounts:
      - mountPath: /usr/local/bin/kustomize
        name: custom-tools
        subPath: kustomize
      - mountPath: /usr/local/bin/ksops
        name: custom-tools
        subPath: ksops
      - mountPath: /ksops-credentials
        readOnly: true
        name: ksops-credentials

Now I receive the error kustomize build .infrastructure/kustomizations/external-secrets-stores --enable-alpha-plugins --enable-exec failed exit status 1: failed to evaluate function: error decrypting file "./secrets/secret-doppler-auth-cert-manager.enc.yml" from manifest.Files: trouble decrypting file: Error getting data key: 0 successful groups required, got 0unable to generate manifests: error decrypting file "./secrets/secret-doppler-auth-cert-manager.enc.yml" from manifest.Files: trouble decrypting file: Error getting data key: 0 successful groups required, got 0Error: couldn't execute function: exit status 1

I also tried to exec into the pod and manually run kustomize build --enable-alpha-plugins --enable-exec . but it throws the same error. Also my GOOGLE_APPLICATION_CREDENTIALS are set correctly referring to a service account.

I verified the account permissions locally by doing the following:

• Set GOOGLE_APPLICATION_CREDENTIALS accordingly.
• Remove all permissions of the account and run kustomize.
• This is failing with the same error.
• Add the CryptoKey Encrypter/Decrypter permission (again) and run kustomize.
• The secrets get decrypted successfully.
  Hence, the service account permissions are correctly set to be able to encrypt/decrypt the secrets.

I also tried setting GOOGLE_CREDENTIALS but the same error occurs as well.

The version used are:

• ArgoCD: 2.6.7
• kustomize: v5.0.1+ksops.v4.1.1

For your reference, my kustomization files look like this.
kustomization.yaml:

generators:
  - ./secret-generator.yml

secret-generator.yml:

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: doppler-access-secrets-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops
files:
  - ./secret.yml

secret.yml:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
    name: my-secret
spec:
    data:
        token: <SOME_ENCRYPTED_STRING>
sops:
    kms: []
    gcp_kms:
        - resource_id: projects/<PROJECT>/locations/europe/keyRings/<KEY_RING>/cryptoKeys/<CRYPTO_KEY>
          created_at: "2023-05-01T15:01:07Z"
          enc: CiQA9PdEcJsJKv1HyfUN0fXTI5FjdhjJ/FAqGB0kw5VcSvfW4E4SSAA2N6cCKHK9B4ZdTrEDU2oExz/sRc/i1Tb8YOT889320eEw8HnCaoV53Qkq3qjLtr9hLs8AuOhd2JiqLGvJka33kq2gAV92LA==
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2023-05-01T15:01:08Z"
    mac: ENC[AES256_GCM,data:mEE5GJ0KZprbkvvl5nR1hSxex99aNkHNJ+YZarbtGf0WLdMU4eGB3pBX0y0De0a6K3mRhUJsd+vnfFxTp9vM2InXtPBPflz7PzXHb4/nx4MYVWOoI1l3aLIDHTTK96N0jAL9iXd+/1UPsM5grC/76knbCOw1etL8zCFf2yhq+Ck=,iv:SS2CItySoi5bo+OxeCUZYpn6Ddrcgu2CwJbn/TWv9Ww=,tag:rAp4PntjYlKi7Y7C2v8A9A==,type:str]
    pgp: []
    unencrypted_regex: ^(apiVersion|metadata|kind|type)$
    version: 3.7.3

Now I have no idea anymore on why (k)sops doesn't pick up the credentials in the ArgoCD pod correctly, and would welcome any suggestions.

[Daniel Mühlbachler-Pietrzykowski]

I finally got this resolved and my issue was two-fold:
1/ My server's date and time were out of sync which probably resulted in GCP not accepting authentication secrets.
2/ getsops/sops#1151 - I explicitly needed to set GOOGLE_CREDENTIALS for the repo server as GOOGLE_APPLICATION_CREDENTIALS did not work for me.