Replace existing secret?

[Lars Kellogg-Stedman]

I originally asked this over in kubernetes-sigs/kustomize#3078, but they tell me this behavior needs to be implemented by the individual generator plugins, so I am asking here!

I am trying to replace a secretGenerator-generated resource in a base with a plugin generated (ksops-encrypted) secret, and it's failing because:

Error: merging from generator &{0xc0002a76a0 0xc00012d680 [./certificate.yml ./deploykeys.yml ./bitlbee_config.yml]}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", K
ind:"Secret"}, Name:"bitlbee-config", Namespace:""} exists; behavior must be merge or replace

In ../base/kustomization.yml, I have:

secretGenerator:
  - name: bitlbee-config
    files:
      - bitlbee.conf

In my local directory, I have:

bases:
  - ../base

generators:
  - secret-generator.yml

In secret-generator.yml I have:

---
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secret-generator
files:
  - ./bitlbee_config.yml

And in bitlbee_config.yml I have:

apiVersion: v1
kind: Secret
metadata:
    name: bitlbee-config
    annotations:
        kustomize.config.k8s.io/needs-hash: "true"
stringData:
    bitlbee.conf: |
      ...content goes here...

I want the secret created from the sops-encrypted file to replace the static secret in the base directory. I see that e.g. for configMapGenerator and secretGenerator I can set a behavior key to control create vs replace, but I don't see a similar option for generators, which is a list rather than a map and doesn't appear to support anything other than a list of files.

Is it possible to add a similar option to the ksops plugin?

[Devin Stein]

@larsks Thanks for making an issue! Yes it is possible, KSOPS use exec plugin annotations to control the generator behavior . You can see an example in the README. You'll just need to update your secret with the following annotation:

apiVersion: v1
kind: Secret
metadata:
    name: bitlbee-config
    annotations:
      kustomize.config.k8s.io/needs-hash: "true"
      kustomize.config.k8s.io/behavior: replace
stringData:
    bitlbee.conf: |
      ...content goes here...

Let me know if this solves your issue! If so, feel free to close the issue

[Lars Kellogg-Stedman]

@devstein thanks, that's exactly what I was looking for! Sorry for missing the bit in the README; I was looking all over for how to solve this from the kustomize side of things.

[Tyler Mapp]

Commenting here because I am trying to do this with the exact use case you call out in the README about the base ArgoCD secret. I've tried both with replace and merge as I'm just adding keys for SSO configuration, and it's not taking on my end. ArgoCD continues to complain about there already being a resource with that secret name, so it can't create another resource with the same name.

[Devin Stein]

Hey, @streetmapp 👋 would you mind sharing your kustomize directory structure and Argo CD secret overlay manifest and I can help debug?

Also, what version of ksops/kustomize are you using?

[Tyler Mapp]

I'm presently using the kustomize patch to install ksops. Right now I am using image version ksops:v2.1.1-go-1.14. When I was running the latest version out there, I was getting Go version conflicts which was confusing to me since kustomize and ksops appeared to be built using the same versions of Go. I had moved to this version and things started working.

├── argocd-cm.yaml
├── argocd-ingress.yaml
├── argocd-rbac-cm.yaml
├── argocd-repo-ksops.yaml
├── enc.argocd-secret.yaml
├── enc.repo-config.yaml
├── kustomization.yaml
└── secret-generator.yaml

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argo

resources:
  - ../../bases/argo-cd
  - argocd-ingress.yaml

patchesStrategicMerge:
  - argocd-cm.yaml
  - argocd-rbac-cm.yaml
  - argocd-repo-ksops.yaml

generators:
  - secret-generator.yaml

enc.argocd-secret.yaml

apiVersion: v1
kind: Secret
metadata:
    annotations:
        kustomize.config.k8s.io/behavior: replace
    labels:
        app.kubernetes.io/instance: internal-argocd
        app.kubernetes.io/name: argocd-secret
        app.kubernetes.io/part-of: argocd
    name: argocd-secret
type: Opaque
data:
  

secret-generator.yaml

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  # Specify a name
  name: argocd-secret-generator
files:
  - ./enc.repo-config.yaml
  - ./enc.argocd-secret.yaml

[Devin Stein]

@streetmapp At a glance, your setup looks correct. Can you share the exact error in Argo CD? I can try reproduce with the same docker image

[Tyler Mapp]

@devstein

Only FYI is that I had changed my setup to be a merge instead of the replace

rpc error: code = Unknown desc = `kustomize build /tmp/[e-mail]:gitops/internal/argo --enable_alpha_plugins` 
failed exit status 1: Error: merging from generator &{0xc00047ee80 0xc0000baae0 
[./enc.repo-config.yaml ./enc.argocd-secret.yaml]}: may not add resource with an already 
registered id: ~G_v1_Secret|~X|argocd-secret

[Devin Stein]

@streetmapp Unfortunately, I'm unable to reproduce on v2.1.1, but I know there is some tricky behavior in the ordering of evaluation for merging/replacing and the namespace transformer. Would you mind trying to add the argo namespace to your secret and retry?

apiVersion: v1
kind: Secret
metadata:
    annotations:
        kustomize.config.k8s.io/behavior: replace
    labels:
        app.kubernetes.io/instance: internal-argocd
        app.kubernetes.io/name: argocd-secret
        app.kubernetes.io/part-of: argocd
    name: argocd-secret
    namespace: argo
type: Opaque
data:

Also, if you build the manifests locally do you still see this error? Or only in Argo CD?

[Tyler Mapp]

@devstein so that changes the error to the following

rpc error: code = Unknown desc = `kustomize build /tmp/[e-mail]:gitops/internal/argo --enable_alpha_plugins` 
failed exit status 1: Error: namespace tranformation produces ID conflict: []*resource.Resource{(*resource.Resource)
(0xc00046ee60), (*resource.Resource)(0xc000640140)}

I'll have to get back to you on the local building. Thanks for your assistance on this.

[Devin Stein]

@streetmapp Were you able to reproduce locally?

[Tyler Mapp]

@devstein thanks for checking in. This is working now. The issue ended up being on my end, as I'm in the process of upgrading and was testing in parallel. And I ultimately had my versions crossed. So my initial tests were from the older version which not surprsingly had issues. Once I moved to test with the new version, the issues went away. Thanks again for the support.

[Devin Stein]

@streetmapp Glad to hear!

[Karl Schriek]

@devstein Any guidance on the mimimum kustomize/ksops versions where the above works? We are currently locked to kustomize 3.x.

[Devin Stein]

@karlschriek Yes, if you are stuck on kustomize 3.x, you'll want to use KSOPS v2.4.0