Trouble decrypting file

[Dale-Kurt Murray]

Version

Argo CD v2.1.2
kustomizs-sops {Version:v4.3.0+ksops.v3.0.1 GitCommit:v4@v4.3.0 BuildDate:2021-09-01T20:13:35Z GoOs:linux GoArch:amd64

What I've done so far

I have a AWS EKS cluster with permissions to KMS decrypt. SOPS config is using the KMS key to encrypt locally on my machine, commit and push to git.

.sops.yaml

---
creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    kms: "redacted"

Using a custom Argo CD container image with the kustomize-sops

Custom Dockerfile

ARG ARGO_CD_VERSION="v2.1.2"
ARG KSOPS_VERSION="v3.0.1"

FROM viaductoss/ksops:$KSOPS_VERSION as ksops-builder

FROM argoproj/argocd:$ARGO_CD_VERSION


# Switch to root for the ability to perform install
USER root

# Set the kustomize home directory
ENV XDG_CONFIG_HOME=$HOME/.config
ENV KUSTOMIZE_PLUGIN_PATH=$XDG_CONFIG_HOME/kustomize/plugin/

ARG PKG_NAME=ksops

# Override the default kustomize executable with the Go built version
COPY --from=ksops-builder /go/bin/kustomize /usr/local/bin/kustomize

# Copy the plugin to kustomize plugin path
COPY --from=ksops-builder /go/src/github.com/viaduct-ai/kustomize-sops/*  $KUSTOMIZE_PLUGIN_PATH/viaduct.ai/v1/${PKG_NAME}/

# Switch back to non-root user
USER argocd

The updated argocd-cm ConfigMap with the following:

data:
  kustomize.buildOptions: "--enable-alpha-plugins"

Additionally, the following files are in the repo (and path)

kustomization.yaml

generators:
  - ./secret-generator.yaml

secret-generator.yaml

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: demo-secret-generator
  namespace: demo
files:
  - ./secrets.yaml

secrets.yaml

redacted

Issue

Executing kustomize build from within the argo-repo-server Pod

kustomize build /tmp/redacted/demo --enable-alpha-plugins
trouble decrypting file Error getting data key: 0 successful groups required, got 0Error: failure in plugin configured via /tmp/kust-plugin-config-288993863; exit status 1: exit status 1

[Devin Stein]

Hi @dalekurt, thanks for making an issue! This error

kustomize build /tmp/redacted/demo --enable-alpha-plugins
trouble decrypting file Error getting data key: 0 successful groups required, got 0Error: failure in plugin configured via /tmp/kust-plugin-config-288993863; exit status 1: exit status 1

indicates that the pod doesn't permission to access the KMS key. How are you injecting credentials to the pod? Do credentials have the correct IAM permissions to use the KMS key?

[Dale-Kurt Murray]

@devstein That was SUPER responsive, the AWS EKS cluster role has IAM permission to use the KMS key. However, I will do a sanity check the permissions.

[Dale-Kurt Murray]

@devstein An update -- The pod required having a service account in order to have access to AWS KMS which was never created initially. I've created it and now working on the next issue.

I will go ahead and close this issue.

[Divick]

@dalekurt I am having the same issue as you faced above. The only difference is that my cluster is built using kops instead of EKS and I am not using IRSA for linking the service accounts with roles. In such a case I have to specify the permissions / policies directly to instances. But even after applying all kms permissions, it gives the same error.

@devstein Wouldn't specifying the AWS_ACCESS_KEY_ID/SECRET defeat the purpose of encrypting the keys? Because if someone can see get hold of them, one can anyways decrypt the secrets?

[Devin Stein]

Hi @divick have you tried exec-ing into the pod to see what AWS role/user the pod has? If you are running into this issue, then the KMS permissions are either insufficient or not applied to the AWS role/user associated with the pod.