Problems running in openshift + gitops operator (ArgoCD)

Question

Problems running in openshift + gitops operator (ArgoCD)

aladrocMatiner opened this issue 8 months ago · comments

Hej, Im testing this repo on argo cd following the tutorial using openshift. For this I use a mix of this redhat article:
https://www.redhat.com/en/blog/a-guide-to-gitops-and-secret-management-with-argocd-operator-and-sops

and the repo i generated.
https://github.com/aladrocMatiner/poc-ksops-01

I modify as the readme says the repo argocd object on the gitops operator:

repo:
    initContainers:
      - args:
          - >-
            echo "Installing KSOPS..."; mv ksops /custom-tools/; mv kustomize
            /custom-tools/; echo "Done.";
        command:
          - /bin/sh
          - '-c'
        image: 'viaductoss/ksops:v4.3.1'
        name: install-ksops
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools
    resources:
      limits:
        cpu: '1'
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 256Mi
    volumeMounts:
      - mountPath: /usr/local/bin/kustomize
        name: custom-tools
        subPath: kustomize
      - mountPath: /usr/local/bin/ksops
        name: custom-tools
        subPath: ksops
    volumes:
      - emptyDir: {}
        name: custom-tools

And i can go to the container openshift-gitops-repo-server and I have there ksops and kustomize as it should be

sh-4.4$ cd /usr/local/bin/
sh-4.4$ ls
argocd                            argocd-cmp-server     argocd-repo-server  git-verify-wrapper.sh  ksops      uid_entrypoint.sh
argocd-application-controller     argocd-dex            argocd-server       gpg-wrapper.sh         kustomize
argocd-applicationset-controller  argocd-notifications  entrypoint.sh       helm                   tini

Then I configure argocd with my project...

and I get this message

Unable to create application: application spec for argo-test-01 is invalid: InvalidSpecError: Unable to generate manifests in ./: rpc error: code = Unknown desc = Manifest generation error (cached): `kustomize build <path to cached source> failed exit status 1: Error: loading generator plugins: failed to load generator: external plugins disabled; unable to load external plugin 'ksops'

if i run the command locally it works:

╰─± kustomize build --enable-alpha-plugins --enable-exec .
apiVersion: v1
data:
  password: MWYyZDFlMmU2N2Rm
  username: YWRtaW4=
kind: Secret
metadata:
  name: mysecret
type: Opaque

So i wonder what is wrong on the setup

dosubot · Answer 1 · Wed Jan 24 2024 07:27:14 GMT+0800 (China Standard Time)

🤖

Hej @aladrocMatiner! Glad to see you back here. How have you been?

The error message you're seeing suggests that Argo CD is unable to use the KSOPS plugin because external plugins are disabled. This is likely due to the fact that Argo CD, by default, disables the use of external plugins for security reasons.

To resolve this issue, you need to enable the use of external plugins in Argo CD. This can be done by setting the ARGOCD_KUSTOMIZE_BUILD_OPTIONS environment variable to --enable_alpha_plugins in the Argo CD deployment.

Here is an example of how you can set this environment variable in a Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
  namespace: argocd
spec:
  template:
    spec:
      containers:
      - name: argocd-server
        env:
        - name: ARGOCD_KUSTOMIZE_BUILD_OPTIONS
          value: --enable_alpha_plugins

After making this change, you should be able to use the KSOPS plugin with Argo CD.

The KSOPS plugin is integrated with Argo CD in the KSOPS codebase in three ways:

KSOPS Repo Server Patch: This method uses a strategic merge patch on the Argo CD repo server deployment. An init container is used to build KSOPS and a volume mount is used to inject the KSOPS plugin and optionally override the kustomize executable. The patch includes moving the KSOPS and kustomize executables to a custom-tools directory which is then mounted to the appropriate paths in the argocd-repo-server container.
Custom Argo CD with KSOPS Dockerfile: This method involves building a custom Docker image for more control and faster pod start times. The Dockerfile uses a multi-stage build where the first stage builds KSOPS and kustomize from the viaductoss/ksops image, and the second stage copies these executables into the argoproj/argocd image.
Argo CD Helm Chart with Custom Tooling: This method involves setting up KSOPS custom tooling in the Argo CD Chart with specific values. The Helm values enable Kustomize Alpha Plugins via the Argo CD ConfigMap, which is required for KSOPS. It also configures init containers to install KSOPS and kustomize into a custom-tools directory, which is then mounted to the appropriate paths in the argocd-repo-server container.

You can find more details about these methods in the KSOPS README.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

aladrocMatiner · Answer 2 · Wed Jan 24 2024 16:56:23 GMT+0800 (China Standard Time)

If i run the command from inside the container i get this:

sh-4.4$ cd lalala/
sh-4.4$ ls
README.md  kustomization.yaml  secret-generator.yaml  secret.enc.yaml  secret.yaml
sh-4.4$ ls -lha
total 20K
drwxr-sr-x. 3 1000690000 1000690000  150 Jan 24 08:53 .
drwxrwsrwx. 4 root       1000690000   72 Jan 24 08:51 ..
drwxr-sr-x. 7 1000690000 1000690000  150 Jan 24 08:53 .git
-rw-r--r--. 1 1000690000 1000690000  364 Jan 24 08:53 .sops.yaml
-rw-r--r--. 1 1000690000 1000690000    0 Jan 24 08:53 README.md
-rw-r--r--. 1 1000690000 1000690000   40 Jan 24 08:53 kustomization.yaml
-rw-r--r--. 1 1000690000 1000690000  375 Jan 24 08:53 secret-generator.yaml
-rw-r--r--. 1 1000690000 1000690000 1.7K Jan 24 08:53 secret.enc.yaml
-rw-r--r--. 1 1000690000 1000690000  124 Jan 24 08:53 secret.yaml
sh-4.4$ kustomize build --enable-alpha-plugins --enable-exec .
failed to evaluate function: error decrypting file "./secret.enc.yaml" from manifest.Files: trouble decrypting file: Error getting data key: 0 successful groups required, got 0unable to generate manifests: error decrypting file "./secret.enc.yaml" from manifest.Files: trouble decrypting file: Error getting data key: 0 successful groups required, got 0Error: couldn't execute function: exit status 1

aladrocMatiner · Answer 3 · Wed Jan 24 2024 18:24:45 GMT+0800 (China Standard Time)

keys added to argocd config map

aladrocMatiner · Answer 4 · Wed Jan 24 2024 18:37:14 GMT+0800 (China Standard Time)

noticed that the changes on argocd-cm are not persisting:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

aladrocMatiner · Answer 5 · Wed Jan 24 2024 19:05:27 GMT+0800 (China Standard Time)

managed to get buildOptions working by modifiying the argocd object: https://argocd-operator.readthedocs.io/en/latest/reference/argocd/#kustomize-build-options

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: example-argocd
  labels:
    example: kustomize-build-options
spec:
  kustomizeBuildOptions: --load_restrictor none

Now I get another error:

Unable to create application: application spec for argo-test-01 is invalid: InvalidSpecError: Unable to generate manifests in ./: rpc error: code = Unknown desc = `kustomize build <path to cached source> --enable-alpha-plugins --enable-exec` failed exit status 1: failed to evaluate function: error decrypting file "./secret.enc.yaml" from manifest.Files: trouble decrypting file: Error getting data key: 0 successful groups required, got 0unable to generate manifests: error decrypting file "./secret.enc.yaml" from manifest.Files: trouble decrypting file: Error getting data key: 0 successful groups required, got 0Error: couldn't execute function: exit status 1

aladrocMatiner · Answer 6 · Wed Jan 24 2024 20:35:25 GMT+0800 (China Standard Time)

Now trying to mount the private key to be available in the container but unsure how to find the correct path.

volumeMounts:
      - mountPath: /usr/local/bin/kustomize
        name: custom-tools
        subPath: kustomize
      - mountPath: /usr/local/bin/ksops
        name: custom-tools
        subPath: ksops
      - mountPath: /home/argocd
        name: sops-gpg
    volumes:
      - name: sops-gpg
        secret:
          items:
            - key: private-FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4.asc
              path: private-FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4.asc
          secretName: sops-gpg
      - emptyDir: {}
        name: custom-tools

aladrocMatiner · Answer 7 · Wed Jan 24 2024 20:55:37 GMT+0800 (China Standard Time)

seems gpg is not installed in the container so that might be the reason all this is not working. The funny part is that the tutorial does not mention nothinf like this and the container build neigther... perhaps Im doing something that i not suppose to or overcomplicating the situation?

aladrocMatiner · Answer 8 · Thu Jan 25 2024 18:16:38 GMT+0800 (China Standard Time)

Documentation is pretty broken.... took a while to figure this out...

gpg --export-secret-key -a "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4" > private-FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4.asc
gpg --export-secret-key -a "B611A2F9F11D0FF82568805119F9B5DAEA91FF86" > private-B611A2F9F11D0FF82568805119F9B5DAEA91FF86.asc
gpg --export-secret-key -a "D7229043384BCC60326C6FB9D8720D957C3D3074" > private-D7229043384BCC60326C6FB9D8720D957C3D3074.asc

Create a secret with the gpg keys and put it on the namespace openshift-gitops

oc create secret generic sops-gpg --namespace=openshift-gitops
--from-file=private-FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4.asc
--from-file=private-D7229043384BCC60326C6FB9D8720D957C3D3074.asc
--from-file=private-B611A2F9F11D0FF82568805119F9B5DAEA91FF86.asc
--dry-run=client -o yaml > secret-sops-gpg.yaml

Add all this things to your openshift gitops operator ( oc edit argocd). Not specially happy having to import a 3rd party container for the initContainer to use gpg... I hope viaduct include gpg to their container to be able to use it.

kustomizeBuildOptions: '--enable-alpha-plugins --enable-exec'
repo:
    initContainers:
      - args:
          - >-
            echo "[Installing Keys...]" && gpg --import /home/argocd/*.asc &&
            gpg --list-secret-keys && echo "[Done]"
        command:
          - /bin/sh
          - '-c'
        image: stevenctimm/gpgridvanilla
        name: install-gpg-keys
        resources: {}
        volumeMounts:
          - mountPath: /home/argocd/
            name: sops-gpg
          - mountPath: //.gnupg
            name: gnupg
      - args:
          - >-
            echo "Installing KSOPS..."; mv ksops /custom-tools/; mv kustomize
            /custom-tools/; echo "Done.";
        command:
          - /bin/sh
          - '-c'
        image: 'viaductoss/ksops:v4.3.1'
        name: install-ksops
        resources: {}
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools
    resources:
      limits:
        cpu: '1'
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 256Mi
    volumeMounts:
      - mountPath: /usr/local/bin/kustomize
        name: custom-tools
        subPath: kustomize
      - mountPath: /usr/local/bin/ksops
        name: custom-tools
        subPath: ksops
      - mountPath: /home/argocd/
        name: sops-gpg
      - mountPath: //.gnupg
        name: gnupg
    volumes:
      - name: sops-gpg
        secret:
          items:
            - key: private-FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4.asc
              path: private-FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4.asc
          secretName: sops-gpg
      - emptyDir: {}
        name: custom-tools
      - emptyDir: {}
        name: gnupg

I will prepare a guide to install on openshift and do a merge request later in the week or weekend for any questions contact me at jose.romero@aladroc.io

aladrocMatiner · Answer 9 · Thu Jan 25 2024 18:52:51 GMT+0800 (China Standard Time)

just to mention aswell the logs of the initPod "install-ksops" are not looking that good

Installing KSOPS...
mv: cannot remove 'ksops': Permission denied
mv: cannot remove 'kustomize': Permission denied
Done.

aladrocMatiner · Answer 10 · Wed Jan 31 2024 22:02:13 GMT+0800 (China Standard Time)

btw, a colegue told made a blaster comment instead of using the image "stevenctimm/gpgridvanilla" that includes gpg u can use the ubi-9 images since ubis use gpg already for the package management so they have it on their "DNA"