KSOPS not working with ArgoCD - no error

Question

KSOPS not working with ArgoCD - no error

dunnjt opened this issue 7 months ago · comments

I'm unable to find the root cause for why KSOPS isn't working with ArgoCD. I can test my setup locally and everything works as expected. On Argo, the repo-server seems configured properly, but doesn't create the Secret. There are also no errors to help debug.

My Kustomization flow is as follows:

kustomization.yaml

...
generators:
  - secret-generator.yaml

secretGenerator:
- name: test-generator
  literals:
  - FRUIT=apple
  - VEGETABLE=carrot
....

secret-generator.yaml

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: test-secret-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops
secretFrom:
- metadata:
    name: my-secrets
    namespace: test
    labels:
      app: test
    annotations:
      kustomize.config.k8s.io/needs-hash: "false"
  type: Opaque
files:
  - secrets.yaml

secret.yaml is a standard SOPS file encrypted with an AWS KMS key.

Locally I can run kustomize build --enable-alpha-plugins --enable-exec . and I'll see the output of both the KSOPS generated secret and the standard secretFrom generators secret.

On ArgoCD we are using the Argo Helm chart version "5.45.0". Passing the chart values as described in the documentation with

repoServer:
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.2.5
      command: ["/bin/sh", "-c"]
      args:
        - echo "Installing KSOPS...";
          mv ksops /custom-tools/;
          mv kustomize /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops
  envFrom:
    - secretRef:
        name: argocd-aws-credentials

I can exec into the repo-server pod and run kustomize version which gives me v5.2.1+ksops.v4.2.5. I can echo $AWS_SECRET_ACCESS_KEY and it will provide me with my AWS credential. I have also set the kustomize.buildOptions in the argocd-cm.

When I sync the application in Argo, I only see my secretFrom secret created, NOT the KSOPS secret. The trouble is there are no errors. From Argo UI, the sync is healthy. I've put the repo-server into debug mode and found no logs related to ksops. I've even tried purposefully mucking up the implementation to force errors, which they do appear, however, when I put my config back to its original state, the errors go away, but no Secret is created.

Any idea where it could have gone wrong?

dosubot · Answer 1 · Thu Nov 09 2023 23:19:11 GMT+0800 (China Standard Time)

🤖

Hello @dunnjt,

Nice to meet you! I'm a bot here to help you with your KSOPS and ArgoCD issues while we wait for a human maintainer. I'm here to help you solve bugs, answer questions, and guide you through contributing to the project. I'm currently reviewing your issue and will provide a detailed response shortly.

dosubot · Answer 2 · Thu Nov 09 2023 23:29:55 GMT+0800 (China Standard Time)

🤖

Based on the information you've provided and similar issues I found in the KSOPS repository, there could be several reasons why KSOPS isn't working with ArgoCD in your case.

Kustomize Plugins Not Enabled in ArgoCD ConfigMap: Ensure that you have enabled Kustomize plugins in the ArgoCD ConfigMap. This is done by setting the kustomize.buildOptions to --enable-alpha-plugins --enable-exec. If these flags are not set, ArgoCD will not be able to use the KSOPS plugin.
KSOPS Not Installed in ArgoCD Repo Server: The KSOPS plugin needs to be installed in the ArgoCD repo server. This can be done using an init container to build KSOPS and volume mount to inject the KSOPS plugin. If this step is not done correctly, ArgoCD will not be able to use the KSOPS plugin.
Missing or Incorrect Credentials: If you're using AWS or GCP KMS for SOPS, you need to inject the necessary credentials into the ArgoCD + KSOPS build. If these credentials are missing or incorrect, decryption will fail.
Incorrect Secret Generator Configuration: Ensure that your secret generator configuration is correct. The KSOPS plugin uses a specific configuration to generate secrets from encrypted dotenv files. If this configuration is incorrect, the secret may not be created.
Incorrect Use of Kustomize Annotations: KSOPS supports certain kustomize annotations for generator options. If you're using these annotations incorrectly, it could lead to issues with secret creation. For example, if you're trying to replace or merge a base secret using the kustomize.config.k8s.io/behavior annotation, ensure that you're using the correct values (replace or merge) and that the encrypted data is correctly provided.

Please verify these configurations and make sure they are correctly set up. If the issue persists, it would be helpful to have more detailed logs or error messages.

Regarding your question about the compatibility of ArgoCD with KSOPS version v4.2.5, I wasn't able to find specific information about this in the KSOPS repository. However, it would be helpful to know the exact version of ArgoCD you're using to further investigate this issue.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

Devin Stein · Answer 3 · Sat Nov 11 2023 04:36:10 GMT+0800 (China Standard Time)

Hey @dunnjt thanks for making an issue.

At a glance it looks like it could be a yaml formatting issue. The files key is at the top-level rather than nested in secretsFrom. Try

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: test-secret-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops
secretFrom:
- metadata:
    name: my-secrets
    namespace: test
    labels:
      app: test
    annotations:
      kustomize.config.k8s.io/needs-hash: "false"
  type: Opaque
  files:
    - secrets.yaml

Also, is there a reason for the test generator? kustomize can be finicky about multiple generators, so maybe try remove it and see if it make a difference.

secretGenerator:
- name: test-generator
  literals:
  - FRUIT=apple
  - VEGETABLE=carrot

Let me know if either of these help

John · Answer 4 · Thu Jan 04 2024 00:52:24 GMT+0800 (China Standard Time)

@devstein Thanks for the follow-up.

The secretGenerator was just a test to verify that any secret could be created. Which it could, so I removed that.

To troubleshoot further, I created a secondary kube cluster with an installation of ArgoCD (same version), loaded KSOPS, used the same Kustomize repo as before and everything worked as expected. The only difference I can see is on the working installation it's deploying in-cluster, whereas on the non-working install, it's deploying to a remote cluster.

On the non-working install, I exec'ed into ArgoCD's repo-server found the location of my application code and ran,
/tmp/_argocd-repo/da5d910c-b169-4223-8662-967a4a454afd/argocd/testing/us$ kustomize build --enable-alpha-plugins --enable-exec and I do see the correct output with the KSOPS generated secret properly decrypted. However, argocd app resources <<APP_NAME>> does not show this secret in its list of managed resources. There is a disconnect somewhere.

At this point it seems more like an ArgoCD issue, but I haven't been able to unlock the root cause.

John · Answer 5 · Thu Jan 04 2024 23:01:29 GMT+0800 (China Standard Time)

Found the problem, the argocd-cm was being overwritten and -enable-exec flag was missing from kustomize.buildOptions.