KSOPS not working with ArgoCD - no error
dunnjt opened this issue · comments
I'm unable to find the root cause for why KSOPS isn't working with ArgoCD. I can test my setup locally and everything works as expected. On Argo, the repo-server seems configured properly, but doesn't create the Secret. There are also no errors to help debug.
My Kustomization flow is as follows:
kustomization.yaml
...
generators:
- secret-generator.yaml
secretGenerator:
- name: test-generator
literals:
- FRUIT=apple
- VEGETABLE=carrot
....
secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: test-secret-generator
annotations:
config.kubernetes.io/function: |
exec:
path: ksops
secretFrom:
- metadata:
name: my-secrets
namespace: test
labels:
app: test
annotations:
kustomize.config.k8s.io/needs-hash: "false"
type: Opaque
files:
- secrets.yaml
secret.yaml is a standard SOPS file encrypted with an AWS KMS key.
Locally I can run kustomize build --enable-alpha-plugins --enable-exec .
and I'll see the output of both the KSOPS generated secret and the standard secretFrom generators secret.
On ArgoCD we are using the Argo Helm chart version "5.45.0". Passing the chart values as described in the documentation with
repoServer:
volumes:
- name: custom-tools
emptyDir: {}
initContainers:
- name: install-ksops
image: viaductoss/ksops:v4.2.5
command: ["/bin/sh", "-c"]
args:
- echo "Installing KSOPS...";
mv ksops /custom-tools/;
mv kustomize /custom-tools/;
echo "Done.";
volumeMounts:
- mountPath: /custom-tools
name: custom-tools
volumeMounts:
- mountPath: /usr/local/bin/kustomize
name: custom-tools
subPath: kustomize
- mountPath: /usr/local/bin/ksops
name: custom-tools
subPath: ksops
envFrom:
- secretRef:
name: argocd-aws-credentials
I can exec into the repo-server pod and run kustomize version
which gives me v5.2.1+ksops.v4.2.5
. I can echo $AWS_SECRET_ACCESS_KEY
and it will provide me with my AWS credential. I have also set the kustomize.buildOptions
in the argocd-cm.
When I sync the application in Argo, I only see my secretFrom secret created, NOT the KSOPS secret. The trouble is there are no errors. From Argo UI, the sync is healthy. I've put the repo-server into debug mode and found no logs related to ksops. I've even tried purposefully mucking up the implementation to force errors, which they do appear, however, when I put my config back to its original state, the errors go away, but no Secret is created.
Any idea where it could have gone wrong?
🤖
Hello @dunnjt,
Nice to meet you! I'm a bot here to help you with your KSOPS and ArgoCD issues while we wait for a human maintainer. I'm here to help you solve bugs, answer questions, and guide you through contributing to the project. I'm currently reviewing your issue and will provide a detailed response shortly.
🤖
Based on the information you've provided and similar issues I found in the KSOPS repository, there could be several reasons why KSOPS isn't working with ArgoCD in your case.
-
Kustomize Plugins Not Enabled in ArgoCD ConfigMap: Ensure that you have enabled Kustomize plugins in the ArgoCD ConfigMap. This is done by setting the
kustomize.buildOptions
to--enable-alpha-plugins --enable-exec
. If these flags are not set, ArgoCD will not be able to use the KSOPS plugin. -
KSOPS Not Installed in ArgoCD Repo Server: The KSOPS plugin needs to be installed in the ArgoCD repo server. This can be done using an init container to build KSOPS and volume mount to inject the KSOPS plugin. If this step is not done correctly, ArgoCD will not be able to use the KSOPS plugin.
-
Missing or Incorrect Credentials: If you're using AWS or GCP KMS for SOPS, you need to inject the necessary credentials into the ArgoCD + KSOPS build. If these credentials are missing or incorrect, decryption will fail.
-
Incorrect Secret Generator Configuration: Ensure that your secret generator configuration is correct. The KSOPS plugin uses a specific configuration to generate secrets from encrypted dotenv files. If this configuration is incorrect, the secret may not be created.
-
Incorrect Use of Kustomize Annotations: KSOPS supports certain kustomize annotations for generator options. If you're using these annotations incorrectly, it could lead to issues with secret creation. For example, if you're trying to replace or merge a base secret using the
kustomize.config.k8s.io/behavior
annotation, ensure that you're using the correct values (replace
ormerge
) and that the encrypted data is correctly provided.
Please verify these configurations and make sure they are correctly set up. If the issue persists, it would be helpful to have more detailed logs or error messages.
Regarding your question about the compatibility of ArgoCD with KSOPS version v4.2.5, I wasn't able to find specific information about this in the KSOPS repository. However, it would be helpful to know the exact version of ArgoCD you're using to further investigate this issue.
Sources
- ArgoCD is trying to create a KSOPS Kubernete resource
- KSOPS Issue With Strategic Merge Patch In ArgoCD
- README.md
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
Hey @dunnjt thanks for making an issue.
At a glance it looks like it could be a yaml formatting issue. The files
key is at the top-level rather than nested in secretsFrom
. Try
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: test-secret-generator
annotations:
config.kubernetes.io/function: |
exec:
path: ksops
secretFrom:
- metadata:
name: my-secrets
namespace: test
labels:
app: test
annotations:
kustomize.config.k8s.io/needs-hash: "false"
type: Opaque
files:
- secrets.yaml
Also, is there a reason for the test generator? kustomize
can be finicky about multiple generators, so maybe try remove it and see if it make a difference.
secretGenerator:
- name: test-generator
literals:
- FRUIT=apple
- VEGETABLE=carrot
Let me know if either of these help
@devstein Thanks for the follow-up.
The secretGenerator
was just a test to verify that any secret could be created. Which it could, so I removed that.
To troubleshoot further, I created a secondary kube cluster with an installation of ArgoCD (same version), loaded KSOPS, used the same Kustomize repo as before and everything worked as expected. The only difference I can see is on the working installation it's deploying in-cluster, whereas on the non-working install, it's deploying to a remote cluster.
On the non-working install, I exec'ed into ArgoCD's repo-server
found the location of my application code and ran,
/tmp/_argocd-repo/da5d910c-b169-4223-8662-967a4a454afd/argocd/testing/us$ kustomize build --enable-alpha-plugins --enable-exec
and I do see the correct output with the KSOPS generated secret properly decrypted. However, argocd app resources <<APP_NAME>>
does not show this secret in its list of managed resources. There is a disconnect somewhere.
At this point it seems more like an ArgoCD issue, but I haven't been able to unlock the root cause.
Found the problem, the argocd-cm was being overwritten and -enable-exec
flag was missing from kustomize.buildOptions
.