Deployment only works in the same namespace where the operator was installed

Question

Deployment only works in the same namespace where the operator was installed

rodrigobrito opened this issue 4 months ago · comments

Rodrigo da Silva Brito commented 4 months ago

Deployment only works in the same namespace where the operator was installed

Steps to reproduce the behavior:

Install the operator:

helm install mongodb-community-operator mongodb/community-operator --namespace mongodb-operator --create-namespace --set operator.watchNamespace="*"

Create a yaml file 'mongodb-replicaset.yaml' for deployment in a different namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: my-app-mongodb
---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: my-mongodb
  namespace: my-app-mongodb
spec:
  members: 3
  type: ReplicaSet
  version: "6.0.5"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: user
      db: admin
      passwordSecretRef: 
        name: admin-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram
  additionalMongodConfig:
    storage.wiredTiger.engineConfig.journalCompressor: zlib
---
apiVersion: v1
kind: Secret
metadata:
  name: admin-password
  namespace: my-app-mongodb
type: Opaque
stringData:
  password: 078a0e02133f4567a700

Perform the deployment:

kubectl apply -f mongodb-replicaset.yaml

The deployment will remain in pending status infinitely.
Operator image and version: quay.io/mongodb/mongodb-kubernetes-operator:0.9.0

What did you expect?
A successful deployment is expected to occur in the specified namespace.

What happened instead?
The deployment remains in a pending state infinitely. In the tests with the same namespace as the operator, the problem does not occur.

Operator Information

Operator Version: quay.io/mongodb/mongodb-kubernetes-operator:0.9.0
MongoDB Image used: docker.io/mongo:6.0.5

Kubernetes Cluster Information

Distribution: RKE2
Version: v1.26.12 +rke2r1

❯ kubectl get mdbc -n my-app-mongodb
NAME PHASE VERSION
my-mongodb Pending

❯ kubectl get sts -n my-app-mongodb
NAME READY AGE
my-mongodb 0/3 18m
my-mongodb-arb 0/0 18m

Operator deployment data:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '1'
    email: support@mongodb.com
    meta.helm.sh/release-name: mongodb-community-operator
    meta.helm.sh/release-namespace: mongodb-operator
  creationTimestamp: '2024-02-23T21:33:04Z'
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    owner: mongodb
  managedFields:
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:email: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/managed-by: {}
            f:owner: {}
        f:spec:
          f:progressDeadlineSeconds: {}
          f:replicas: {}
          f:revisionHistoryLimit: {}
          f:selector: {}
          f:strategy:
            f:rollingUpdate:
              .: {}
              f:maxSurge: {}
              f:maxUnavailable: {}
            f:type: {}
          f:template:
            f:metadata:
              f:labels:
                .: {}
                f:name: {}
            f:spec:
              f:affinity:
                .: {}
                f:podAntiAffinity:
                  .: {}
                  f:requiredDuringSchedulingIgnoredDuringExecution: {}
              f:containers:
                k:{"name":"mongodb-kubernetes-operator"}:
                  .: {}
                  f:command: {}
                  f:env:
                    .: {}
                    k:{"name":"AGENT_IMAGE"}:
                      .: {}
                      f:name: {}
                      f:value: {}
                    k:{"name":"MONGODB_IMAGE"}:
                      .: {}
                      f:name: {}
                      f:value: {}
                    k:{"name":"MONGODB_REPO_URL"}:
                      .: {}
                      f:name: {}
                      f:value: {}
                    k:{"name":"OPERATOR_NAME"}:
                      .: {}
                      f:name: {}
                      f:value: {}
                    k:{"name":"POD_NAME"}:
                      .: {}
                      f:name: {}
                      f:valueFrom:
                        .: {}
                        f:fieldRef: {}
                    k:{"name":"READINESS_PROBE_IMAGE"}:
                      .: {}
                      f:name: {}
                      f:value: {}
                    k:{"name":"VERSION_UPGRADE_HOOK_IMAGE"}:
                      .: {}
                      f:name: {}
                      f:value: {}
                    k:{"name":"WATCH_NAMESPACE"}:
                      .: {}
                      f:name: {}
                      f:value: {}
                  f:image: {}
                  f:imagePullPolicy: {}
                  f:name: {}
                  f:resources:
                    .: {}
                    f:limits:
                      .: {}
                      f:cpu: {}
                      f:memory: {}
                    f:requests:
                      .: {}
                      f:cpu: {}
                      f:memory: {}
                  f:terminationMessagePath: {}
                  f:terminationMessagePolicy: {}
              f:dnsPolicy: {}
              f:restartPolicy: {}
              f:schedulerName: {}
              f:securityContext:
                .: {}
                f:runAsNonRoot: {}
                f:runAsUser: {}
              f:serviceAccount: {}
              f:serviceAccountName: {}
              f:terminationGracePeriodSeconds: {}
      manager: helm
      operation: Update
      time: '2024-02-23T21:33:04Z'
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:deployment.kubernetes.io/revision: {}
        f:status:
          f:availableReplicas: {}
          f:conditions:
            .: {}
            k:{"type":"Available"}:
              .: {}
              f:lastTransitionTime: {}
              f:lastUpdateTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Progressing"}:
              .: {}
              f:lastTransitionTime: {}
              f:lastUpdateTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
          f:observedGeneration: {}
          f:readyReplicas: {}
          f:replicas: {}
          f:updatedReplicas: {}
      manager: kube-controller-manager
      operation: Update
      subresource: status
      time: '2024-02-23T21:33:09Z'
  name: mongodb-kubernetes-operator
  namespace: mongodb-operator
  resourceVersion: '44970321'
  uid: 703ae49e-302d-4042-8ffa-8db9a9e915c8
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: mongodb-kubernetes-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: mongodb-kubernetes-operator
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: name
                    operator: In
                    values:
                      - mongodb-kubernetes-operator
              topologyKey: kubernetes.io/hostname
      containers:
        - command:
            - /usr/local/bin/entrypoint
          env:
            - name: WATCH_NAMESPACE
              value: '*'
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: mongodb-kubernetes-operator
            - name: AGENT_IMAGE
              value: quay.io/mongodb/mongodb-agent:107.0.0.8465-1
            - name: VERSION_UPGRADE_HOOK_IMAGE
              value: >-
                quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
            - name: READINESS_PROBE_IMAGE
              value: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
            - name: MONGODB_IMAGE
              value: mongo
            - name: MONGODB_REPO_URL
              value: docker.io
          image: quay.io/mongodb/mongodb-kubernetes-operator:0.9.0
          imagePullPolicy: Always
          name: mongodb-kubernetes-operator
          resources:
            limits:
              cpu: 1100m
              memory: 1Gi
            requests:
              cpu: 500m
              memory: 200Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
      serviceAccount: mongodb-kubernetes-operator
      serviceAccountName: mongodb-kubernetes-operator
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
    - lastTransitionTime: '2024-02-23T21:33:04Z'
      lastUpdateTime: '2024-02-23T21:33:04Z'
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: 'True'
      type: Available
    - lastTransitionTime: '2024-02-23T21:33:04Z'
      lastUpdateTime: '2024-02-23T21:33:09Z'
      message: >-
        ReplicaSet "mongodb-kubernetes-operator-747bf7c54" has successfully
        progressed.
      reason: NewReplicaSetAvailable
      status: 'True'
      type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

jsmcnair · Answer 1 · Mon Feb 26 2024 16:17:31 GMT+0800 (China Standard Time)

I encountered issues as well. What you need to do is documented as part of the enterprise operator, which I found in another issue.

The reason you encounter issues is because the operator assumes certain resources to exist and does not deploy them for you. It's not very intuitive. Here's the documentation for setting the scope of the operator: https://www.mongodb.com/docs/kubernetes-operator/master/tutorial/set-scope-k8s-operator/#ns-scope-different-ns

What you need to do is deploy a subset of resources from the the operator chart in every namespace where you want to deploy MongoDBCommunity resources:

helm template mongodb/enterprise-operator \
  --set operator.namespace=<metadata.namespace> \
  --show-only templates/database-roles.yaml | kubectl apply -f -

This creates the missing service account, role and role binding needed by the pods.

Rodrigo da Silva Brito · Answer 2 · Mon Feb 26 2024 19:11:41 GMT+0800 (China Standard Time)

I used the following command, to try to configure the deployment in a different namespace:

kubectl apply -k config/rbac --namespace <my-namespace>

Using the files available here, and it still didn't work.

I followed the steps as described here:
https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/docs/install-upgrade.md#install-in-a-different-namespace-using-kubectl

github-actions · Answer 3 · Sat Apr 27 2024 09:48:21 GMT+0800 (China Standard Time)

This issue is being marked stale because it has been open for 60 days with no activity. Please comment if this issue is still affecting you. If there is no change, this issue will be closed in 30 days.

github-actions · Answer 4 · Mon May 27 2024 09:50:03 GMT+0800 (China Standard Time)

This issue was closed because it became stale and did not receive further updates. If the issue is still affecting you, please re-open it, or file a fresh Issue with updated information.

David Dumas · Answer 5 · Thu Jun 20 2024 22:04:44 GMT+0800 (China Standard Time)

For reference I used these commands to make it work (Operator Uses a Subset of Namespaces in the docs):

Install operator in to namespace mongodb-operator configured to watch namespace foo:

helm repo add mongodb https://mongodb.github.io/helm-charts
helm -n mongodb-operator install community-operator mongodb/community-operator --set operator.watchNamespace="foo"

Install RBAC in to other namespace foo where the MongoDBCommunity resource will be created:

helm -n foo template mongodb/community-operator --show-only templates/database_roles.yaml | kubectl apply -f -

I am not sure if I should use the key database.namespace from the operator Helm chart. The key operator.namespace does not exist.