Impossible to decrypt with AWS credentials

[Lorenzo Maffioli]

Hello, I am trying to make ksops work with ArgoCD on EKS, but with no success. I know that the installation works and the repo-server setup is fine because I can run a dummy application, consisting of a single secret, encrypted with age.

Now I am trying to use KMS to encrypt the same secret, but I get trouble decrypting file Error getting data key: 0 successful groups required, got 0Error: failure in plugin configured via /tmp/kust-plugin-config-2510209872; exit status 1: exit status 1.

This is my setup:

values.yaml

global:
  image:
    repository: quay.io/argoproj/argocd
    tag: "v2.5.0"
    imagePullPolicy: IfNotPresent

configs:
  cm:
    create: true
    kustomize.buildOptions: "--enable-alpha-plugins"

repoServer:
  name: repo-server
  env:
    - name: XDG_CONFIG_HOME
      value: /.config
    - name: AWS_DEFAULT_REGION
      value: eu-central-1
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: argocd-aws-credentials
          key: aws-key-id
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: argocd-aws-credentials
          key: aws-secret-key
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
      name: custom-tools
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v3.0.2
      command: ["/bin/sh", "-c"]
      args:
        - echo "Installing KSOPS...";
          mv ksops /custom-tools/;
          mv $GOPATH/bin/kustomize /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

The secret with the credentials:

apiVersion: v1
data:
    aws-key-id: <SECRET_ID>
    aws-secret-key: <SECRET_KEY>
kind: Secret
metadata:
    name: argocd-aws-credentials
    namespace: argo-cd
type: Opaque

.sops.yaml

creation_rules:
  - encrypted_regex: "^(data|stringData)$"
    kms: arn:aws:kms:eu-central-1:xxxxxxxxxx:alias/xxxxxxx
    aws_profile: kms-user

kustomization.yaml

generators:
  - ./secret-generator.yaml

secret-generator.yaml

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  # Specify a name
  name: example-secret-generator
files:
  - ./secret.yaml

As for the rest I have:

• If I test it locally with kustomize build --enable-alpha-plugins . I get the secret unencrypted.
• The credentials I'm using work: they encrypt, decrypt and list the keys in the AWS account.

If I install ArgoCD in the cluster and I log in the repo server I have:

env | grep AWS

AWS_DEFAULT_REGION=eu-central-1
AWS_SECRET_ACCESS_KEY=xxxxxx
AWS_ACCESS_KEY_ID=xxxxxxx

env | grep XDG_CONFIG_HOME

XDG_CONFIG_HOME=/.config

I can run kustomize normally and ksops is the right directory.

For testing purposes I tried to run the viaductoss/ksops:v3.0.2 container locally: I moved ksops in /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops, then I did:

export XDG_CONFIG_HOME=/.config
export AWS_DEFAULT_REGION=eu-central-1
export AWS_SECRET_ACCESS_KEY=xxxxx
export AWS_ACCESS_KEY_ID=xxxxxxx

I installed awscli and I can use my key normally. Then I tried to copy my dummy application in the container and tried to kustomize build --enable-alpha-plugins ., but I get trouble decrypting file Error getting data key: 0 successful groups required, got 0Error: failure in plugin configured via /tmp/kust-plugin-config-2510209872; exit status 1: exit status 1.

I don't know what else to try.

EDIT: I tried downgrading to viaductoss/ksops:v2.6.0 but the issue remains.

EDIT 2: With an ubuntu container I downloaded all the components separately, encrypted a dummy secret I found on Kubernetes documentation, did kustomize build --enable-alpha-plugins . and it works fine.

root@2e64714d351f:/# wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv4.5.7/kustomize_v4.5.7_linux_arm64.tar.gz
root@2e64714d351f:/# tar -xvf kustomize_v4.5.7_linux_arm64.tar.gz
root@2e64714d351f:/# chmod +x kustomize
root@2e64714d351f:/# mv kustomize /usr/local/bin
root@2e64714d351f:/# wget https://github.com/viaduct-ai/kustomize-sops/releases/download/v3.0.2/ksops_3.0.2_Linux_arm64.tar.gz
root@2e64714d351f:/# tar -xvf ksops_3.0.2_Linux_arm64.tar.gz
root@2e64714d351f:/# chmod +x ksops
root@2e64714d351f:/# mkdir -p /.config/kustomize/plugin/viaduct.ai/v1/ksops
root@2e64714d351f:/# mv ksops /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
root@2e64714d351f:/# export XDG_CONFIG_HOME=/.config
root@2e64714d351f:/# cat <<EOF kustomization.yaml
> generators:
  - ./secret-generator.yaml
>
> EOF
root@2e64714d351f:/# cat <<EOF > secret-generator.yaml
> apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  # Specify a name
  name: example-secret-generator
files:
  - ./secret.yaml
>
> EOF
root@2e64714d351f:/# cat <<EOF > secret.yaml
> apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  USER_NAME: YWRtaW4=
  PASSWORD: MWYyZDFlMmU2N2Rm
>
> EOF
root@2e64714d351f:/# ./sops-v3.7.3.linux.arm64 -i -e secret.yaml
root@2e64714d351f:/# kustomize build --enable-alpha-plugins .
apiVersion: v1
data:
  PASSWORD: MWYyZDFlMmU2N2Rm
  USER_NAME: YWRtaW4=
kind: Secret
metadata:
  name: mysecret
type: Opaque

So basically if I encrypt a secret in the container, then the plugin works, but if I take a secret already encrypted, then it doesn't.

[Mateusz Kobus]

I have exactly the same error with similar configuration. Can someone help us?

[Lorenzo Maffioli]

@matemango I spent the entire day trying to figure this out and the solution is simpler than I expected (as always). I'll write it down here hoping that it will save someone a day or two of frustration.

Basically when you sops a secret, sops reads the rules from .sops.yaml that, in my case was this:

creation_rules:
  - encrypted_regex: "^(data|stringData)$"
    kms: arn:aws:kms:eu-central-1:xxxxxxxxxx:alias/xxxxxxx
    aws_profile: kms-user

and then appends a bunch of metadata to the encrypted secret. One of these metadata is aws_profile, so my secret was:

apiVersion: v1
data:
    password: ENC[AES256_GCM,data:co8VJr3M7HIYZaXgOWfsow==,iv:sF/wkc0XcCbFzSciH7gTls8BsWxUFg+sxuTfZusIBuQ=,tag:lryQXNZJQou4aYbENOTecQ==,type:str]
    username: ENC[AES256_GCM,data:xh67qjVMFMM=,iv:VLsJ4ieh9Gc3hAvwwgtyToz/BNPzSUjOZJGqzU0wTXI=,tag:cqmUAn6ilDvNR6muDV2K4A==,type:str]
kind: Secret
metadata:
    name: dummy-secret
type: Opaque
sops:
    kms:
        - arn: arn:aws:kms:eu-central-1:xxxxxxxxxxx:alias/xxxxxxxxx
          created_at: "2022-11-17T13:24:54Z"
          enc: AQICAHhgXf8PbXXK968kesG1a/Wqe0/pB6fnP7OgCN8BUUS8ygFQWWxHUo/Xjcydtdfd/950AAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM9sF87SU+ZGb2j7UhAgEQgDv11k149L4+SL5/qjvwxnVruCTIykC/P/8S3JxtIX2WJ5SXh4RSyp4zowzr4irVGQEIX7/g4y+9QVOasQ==
          aws_profile: kms-user <------------- THIS HAS TO MATCH!
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2022-11-17T13:24:55Z"
    mac: ENC[AES256_GCM,data:gxxkq0wVXqhgIZpxbJrKlAWUDbtsPeHhSndlvRG00ao4YLrQ2UZj2Ydxaa/pURVQY6unvyqMW1GKrliMP79sj+SIvHNsyIuFpV+be5vc9vGI6UZwUe38430CpQM0LWQ5GPRmRJduqqXbydxW00XBqSIIFzsWOKInKIgVzCmMlkY=,iv:q8KkVRRI+bG5A+IJlSpuveNHC3KKwylrjqOddhBzwT4=,tag:SJrO0zby3nAXAaDeybI4DA==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.7.3

So the aws_profile has to match the profile you use to decrypt. Now if I just delete the aws_profile entry in the secret, the repo server uses the default profile (with the credentials) and manages to decrypt the secret just fine.

As an experiment I tried to set AWS_PROFILE=kms-user in the repo server, but it doesn't work. I don't know why, but for now I'm happy with what I have.