ksops + argocd + helm - plugin doesn't seem to be active/run

Question

ksops + argocd + helm - plugin doesn't seem to be active/run

conor-nsurely opened this issue 5 months ago · comments

Hi all,

I have installed Argocd using helm, with the helm values file used to configure the init container which installs ksops to Argocd .

I am using the helm values from here

And all the values seem to be present in the configmap / deployments in the cluster.

I am using Argocd v2.9 and viaductoss/ksops:v4.3.1

In the argo-repo-server logs I can see

│ install-ksops Installing KSOPS...                                                                                                                                                                                                              │
│ install-ksops Done.

And if I start a shell on the argo-repo-server, "kustomize" and "ksops" are present on the path and seem to be installed correctly.

When I add an encrypted secret to Git, Argo fails to deploy saying
one or more objects failed to apply, reason: Secret in version "v1" cannot be handled as a Secret: illegal base64 data at input byte 3 (retried 5 times).

It seems like the plugin is not running and Argocd is just trying to deploy the encrypted secret directly.

Where am I going wrong?

( I have tried $XDG_CONFIG_HOME = /.config also to no avail)

dosubot · Answer 1 · Tue Jan 16 2024 01:48:08 GMT+0800 (China Standard Time)

🤖

Hello @conor-nsurely! 👋

I'm Dosu, a friendly bot 🤖 here to assist while we wait for a human maintainer. I can help you solve bugs, answer questions, and guide you to become a contributor. Let's crack this nut together, shall we? 🌰

Based on the information you've provided and similar issues I found in the KSOPS repository, it seems like the issue might be related to the ArgoCD configuration not being set up correctly to use the KSOPS plugin.

Here are a few things you can check:

Ensure that the KSOPS plugin is correctly installed in the ArgoCD repo server. You can do this by using an init container to install the KSOPS plugin and Kustomize into an emptyDir volume, and then mounting this volume into the ArgoCD repo server container. This is shown in the "KSOPS Repo Sever Patch" section of the README.
Make sure that the ArgoCD server is configured to enable Kustomize alpha plugins and exec functionality. This can be done by setting the kustomize.buildOptions field in the ArgoCD ConfigMap to "--enable-alpha-plugins --enable-exec". This is shown in the "Argo CD Helm Chart with Custom Tooling" section of the README.
If you're using AWS or GCP KMS for encryption, ensure that the necessary credentials to decrypt the secrets are included in the ArgoCD repo server container's environment variables. This is mentioned in the "KSOPS Repo Sever Patch" section of the README.
If you're still having issues, you can try building a custom ArgoCD Docker image that includes the KSOPS plugin and Kustomize, as shown in the "Custom Argo CD w/ KSOPS Dockerfile" section of the README.

In addition to these, I found two similar issues in the KSOPS repository that might be helpful:

KSOPS not working with ArgoCD - no error which was last updated on January 04, 2024. The suggested solution is to ensure the -enable-exec flag is present in kustomize.buildOptions.
KSOPS Config With Argo Values.yaml - Unable to find plugin root which was last updated on October 24, 2023. The suggested solution is to set the XDG_CONFIG_HOME environment variable correctly and ensure the volume mounts are configured properly.

I hope this helps! If you're still having issues, could you provide more details about your setup? Specifically, it would be helpful to see the exact configuration you're using for the ArgoCD repo server and the KSOPS plugin.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

Devin Stein · Answer 2 · Tue Jan 16 2024 02:13:42 GMT+0800 (China Standard Time)

Hey @conor-nsurely thanks for making a detailed issue. At least we know that KSOPS is installed correctly.

Can you verify that kustomize.buildOptions: "--enable-alpha-plugins --enable-exec" is set on the ArgoCD config map?
If it is, are you able to share more information about your K8s manifests and directory structure?

Conor McDonald · Answer 3 · Tue Jan 16 2024 19:38:32 GMT+0800 (China Standard Time)

Hey @devstein

It is enabled but I did notice there's also a flag disabling exec. Not sure where that has come from, maybe a default in Helm

data:
  admin.enabled: "true"
  application.instanceLabelKey: argocd.argoproj.io/instance
  exec.enabled: "false"
  kustomize.buildOptions: --enable-alpha-plugins --enable-exec
  server.rbac.log.enforce.enable: "false"

I've chaged exec.enabled to true, but there doesn't seem to be a difference.

Here's the secret that Argo is failing to apply

apiVersion: v1
data:
    cache_secret: ENC[AES256_GCM,data:d0Yg4P60Vi5AcwxlSJ/oPw==,iv:VwcckTKniflsOFzBLCzDDgbJB8W63MAknlrx52afaX8=,tag:oF3Xoj5aVFdR7D74UJO9yQ==,type:str]
kind: Secret
metadata:
    name: 123-secret
    namespace: default
type: Opaque
sops:
    kms: []
    gcp_kms:
        - resource_id: projects/dev/locations/global/keyRings/ksops-test/cryptoKeys/gke
          created_at: "2024-01-15T16:07:55Z"
          enc: CiQAR1OX8+1vLMju1VB94xq2gJkVki4eI6847X9NEXv3cHPMGRUSSQAFusxYIaFO2vNbWnaXrIx+u1xaH9LfQfxYqcwXtzrrU1h+rliLX6MWJuWfClj3g04JaA8iElW3EGy6Us5w/WvfRDfZfjEGHB0=
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2024-01-15T16:07:55Z"
    mac: ENC[AES256_GCM,data:Zr+bXPfbkq+lQAr9y9Ybue6Qqf+sNPE05hpnGtzF0tdUlnAWOVMFzCGzQZoZnqQlUxO/SQiPR1ncJxWy8ab4LDY5k8IoI+HygDK6dC1igXB+dJawTGGfRKhCtM5tOOlxIYG6rYRg43SlNKzNE7C6f4m90qW5xsDT+NSefG9jUn4=,iv:atOxnkFhkRU/8uGpTNi2YdcX/N/M7oynpTR0WogH7MA=,tag:ARnoyZVsXyVVncVSCqYswQ==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.8.1

I have the secret in a a top level directory by itself. I have put other yaml files in there as a sanity check and they all deploy just fine.

I assume I'm generating the secret incorrectly.

Do I need to apply anything else to kubernetes?`Like the KSOPS kustomize Generator from step 5 of the general instructions.

kind: ksops
metadata:
  name: example-secret-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops

Is it possible to use Workflow Identity federation to retrieve the GCP key or does the key have to be passed via an environment variable? I assume there would be another error though if failing to decrypt was the sole issue.

Really appreciate the help!

Conor McDonald · Answer 4 · Tue Feb 06 2024 21:04:32 GMT+0800 (China Standard Time)

@devstein Is there anymore information I can provide to make the issue clearer?

hugopicq · Answer 5 · Wed Apr 24 2024 22:17:41 GMT+0800 (China Standard Time)

Hi ! I am running into the same problem here, any news?