Mongo v7 with 6.0 compatability fails to start; does not move from Pending phase
ctrmcubed opened this issue · comments
What did you do to encounter the bug?
Steps to reproduce the behavior:
# Clean up if necessary
# kubectl delete crd mongodbcommunity.mongodbcommunity.mongodb.com 2> /dev/null
#
cd $(mktemp -d)
git clone https://github.com/mongodb/mongodb-kubernetes-operator.git
cd mongodb-kubernetes-operator
kubectl delete deployment --namespace mongodb-test mongodb-kubernetes-operator 2> /dev/null
kubectl create namespace mongodb-test 2> /dev/null
kubectl apply -f config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml
kubectl get crd/mongodbcommunity.mongodbcommunity.mongodb.com
kubectl apply -k config/rbac/ --namespace mongodb-test
kubectl apply -f config/manager/manager.yaml --namespace mongodb-test
kubectl wait pod --namespace mongodb-test --for=condition=Ready --all --timeout=1h
kubectl get pods --namespace mongodb-test
kubectl apply --namespace mongodb-test -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: mongodb-test-password
namespace: mongodb-test
type: Opaque
stringData:
password: supersecret
---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: mongodb-test
namespace: mongodb-test
spec:
members: 1
type: ReplicaSet
version: "7.0.1"
featureCompatibilityVersion: "6.0"
security:
authentication:
modes: ["SCRAM-SHA-1"]
users:
- name: admin
db: admin
passwordSecretRef:
name: mongodb-test-password
roles:
- name: root
db: admin
- name: clusterAdmin
db: admin
- name: userAdminAnyDatabase
db: admin
scramCredentialsSecretName: scram-credentials
additionalMongodConfig:
storage.wiredTiger.engineConfig.journalCompressor: zlib
EOF
echo "Waiting for operator to process request"
kubectl wait MongoDBCommunity --namespace mongodb-test mongodb-test --for=jsonpath='{status.phase}'=Running --timeout=1h
kubectl get MongoDBCommunity --namespace mongodb-test
What did you expect?
NAME PHASE VERSION
mongodb-test Running 7.0.1
What happened instead?
NAME PHASE VERSION
mongodb-test Pending
Operator Information
- Operator Version: 0.8.2
- MongoDB Image used: docker.io/mongodb/mongodb-community-server:7.0.1-ubi8
Kubernetes Cluster Information
- Distribution: k3s
- Version: v1.27.3+k3s1
- Image Registry location (quay, or an internal registry): docker.io/mongodb
Additional context
Pod logs:
{"t":{"$date":"2023-09-21T15:33:17.186+00:00"},"s":"I", "c":"NETWORK", "id":6788700, "ctx":"conn245","msg":"Received first command on ingress connection since session start or auth handshake","attr":{"elapsedMillis":0}}
{"t":{"$date":"2023-09-21T15:33:17.186+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn244","msg":"Interrupted operation as its client disconnected","attr":{"opId":10105}}
{"t":{"$date":"2023-09-21T15:33:17.186+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn243","msg":"Connection ended","attr":{"remote":"10.42.0.40:36208","uuid":{"uuid":{"$uuid":"b1b4d44a-6b19-45e8-b11b-17393572e413"}},"connectionId":243,"connectionCount":11}}
{"t":{"$date":"2023-09-21T15:33:17.186+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn245","msg":"Connection ended","attr":{"remote":"10.42.0.40:36230","uuid":{"uuid":{"$uuid":"3e92d6d4-1953-4122-bad0-c911d3b841cc"}},"connectionId":245,"connectionCount":10}}
{"t":{"$date":"2023-09-21T15:33:17.186+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn244","msg":"Connection ended","attr":{"remote":"10.42.0.40:36224","uuid":{"uuid":{"$uuid":"0307e561-9394-436d-a187-2beb79acf391"}},"connectionId":244,"connectionCount":9}}
{"t":{"$date":"2023-09-21T15:33:17.201+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn241","msg":"Interrupted operation as its client disconnected","attr":{"opId":9929}}
{"t":{"$date":"2023-09-21T15:33:17.201+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn242","msg":"Connection ended","attr":{"remote":"10.42.0.40:41328","uuid":{"uuid":{"$uuid":"2b82ab73-2cf7-422f-9055-163ff4bce3be"}},"connectionId":242,"connectionCount":8}}
{"t":{"$date":"2023-09-21T15:33:17.201+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn240","msg":"Connection ended","attr":{"remote":"10.42.0.40:41318","uuid":{"uuid":{"$uuid":"c117fd21-298d-473b-8b03-f82a5aca8d9e"}},"connectionId":240,"connectionCount":7}}
{"t":{"$date":"2023-09-21T15:33:17.202+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn241","msg":"Connection ended","attr":{"remote":"10.42.0.40:41324","uuid":{"uuid":{"$uuid":"12f0e85c-ce4b-4222-b224-23b4f4f3bb13"}},"connectionId":241,"connectionCount":6}}
{"t":{"$date":"2023-09-21T15:33:17.408+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.42.0.40:36242","uuid":{"uuid":{"$uuid":"69ec3ecc-44a4-408b-8fd1-e02f4399371c"}},"connectionId":246,"connectionCount":7}}
{"t":{"$date":"2023-09-21T15:33:17.408+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.42.0.40:36244","uuid":{"uuid":{"$uuid":"6d2e0109-97dd-4ee7-983e-1a97acd2b899"}},"connectionId":247,"connectionCount":8}}
{"t":{"$date":"2023-09-21T15:33:17.408+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn246","msg":"Connection ended","attr":{"remote":"10.42.0.40:36242","uuid":{"uuid":{"$uuid":"69ec3ecc-44a4-408b-8fd1-e02f4399371c"}},"connectionId":246,"connectionCount":7}}
{"t":{"$date":"2023-09-21T15:33:17.408+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn247","msg":"Connection ended","attr":{"remote":"10.42.0.40:36244","uuid":{"uuid":{"$uuid":"6d2e0109-97dd-4ee7-983e-1a97acd2b899"}},"connectionId":247,"connectionCount":6}}
{"t":{"$date":"2023-09-21T15:33:17.408+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.42.0.40:36260","uuid":{"uuid":{"$uuid":"2cb7c52e-a7c8-407b-8f9a-add986698211"}},"connectionId":248,"connectionCount":7}}
{"t":{"$date":"2023-09-21T15:33:17.408+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.42.0.40:36270","uuid":{"uuid":{"$uuid":"821af6fc-3634-4707-9286-591cabbf4092"}},"connectionId":249,"connectionCount":8}}
{"t":{"$date":"2023-09-21T15:33:17.408+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn248","msg":"client metadata","attr":{"remote":"10.42.0.40:36260","client":"conn248","doc":{"driver":{"name":"mongo-go-driver","version":"v1.7.2+prerelease"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.20.6","application":{"name":"MongoDB Automation Agent v12.0.25.7724 (git: 856139c968154ba88c954022a49fa168b5fed71d)"}}}}
{"t":{"$date":"2023-09-21T15:33:17.408+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn249","msg":"client metadata","attr":{"remote":"10.42.0.40:36270","client":"conn249","doc":{"driver":{"name":"mongo-go-driver","version":"v1.7.2+prerelease"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.20.6","application":{"name":"MongoDB Automation Agent v12.0.25.7724 (git: 856139c968154ba88c954022a49fa168b5fed71d)"}}}}
{"t":{"$date":"2023-09-21T15:33:17.409+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.42.0.40:36286","uuid":{"uuid":{"$uuid":"79c60204-8232-43b3-b3dc-e8b951993b73"}},"connectionId":250,"connectionCount":9}}
{"t":{"$date":"2023-09-21T15:33:17.409+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn250","msg":"client metadata","attr":{"remote":"10.42.0.40:36286","client":"conn250","doc":{"driver":{"name":"mongo-go-driver","version":"v1.7.2+prerelease"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.20.6","application":{"name":"MongoDB Automation Agent v12.0.25.7724 (git: 856139c968154ba88c954022a49fa168b5fed71d)"}}}}
{"t":{"$date":"2023-09-21T15:33:17.409+00:00"},"s":"I", "c":"ACCESS", "id":5286307, "ctx":"conn250","msg":"Failed to authenticate","attr":{"client":"10.42.0.40:36286","isSpeculative":true,"isClusterMember":false,"mechanism":"SCRAM-SHA-256","user":"mms-automation","db":"admin","error":"BadValue: SCRAM-SHA-256 authentication is disabled","result":2,"metrics":{"conversation_duration":{"micros":34,"summary":{"0":{"step":1,"step_total":2,"duration_micros":6}}}},"extraInfo":{}}}
{"t":{"$date":"2023-09-21T15:33:17.409+00:00"},"s":"I", "c":"ACCESS", "id":6788604, "ctx":"conn250","msg":"Auth metrics report","attr":{"metric":"acquireUser","micros":0}}
{"t":{"$date":"2023-09-21T15:33:17.412+00:00"},"s":"I", "c":"ACCESS", "id":5286306, "ctx":"conn250","msg":"Successfully authenticated","attr":{"client":"10.42.0.40:36286","isSpeculative":false,"isClusterMember":false,"mechanism":"SCRAM-SHA-1","user":"mms-automation","db":"admin","result":0,"metrics":{"conversation_duration":{"micros":3026,"summary":{"0":{"step":1,"step_total":2,"duration_micros":70},"1":{"step":2,"step_total":2,"duration_micros":13}}}},"extraInfo":{}}}
{"t":{"$date":"2023-09-21T15:33:17.413+00:00"},"s":"I", "c":"NETWORK", "id":6788700, "ctx":"conn250","msg":"Received first command on ingress connection since session start or auth handshake","attr":{"elapsedMillis":0}}
{"t":{"$date":"2023-09-21T15:33:39.234+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn30","msg":"Interrupted operation as its client disconnected","attr":{"opId":10538}}
{"t":{"$date":"2023-09-21T15:33:39.235+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn31","msg":"Connection ended","attr":{"remote":"10.42.0.40:46950","uuid":{"uuid":{"$uuid":"c379b048-2154-4426-bf42-16be37a6560f"}},"connectionId":31,"connectionCount":8}}
{"t":{"$date":"2023-09-21T15:33:39.235+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn32","msg":"Connection ended","attr":{"remote":"10.42.0.40:46954","uuid":{"uuid":{"$uuid":"79ab3c13-9f2b-4356-97e7-a8600f2e652f"}},"connectionId":32,"connectionCount":7}}
{"t":{"$date":"2023-09-21T15:33:39.235+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn30","msg":"Connection ended","attr":{"remote":"10.42.0.40:46946","uuid":{"uuid":{"$uuid":"912cee3b-8dbc-42cd-8bb5-eb776de74daa"}},"connectionId":30,"connectionCount":6}}
Logs from Sep 21, 2023 to Sep 21, 2023 UTC
Could you please paste the content of /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json file from your container and the output of kubectl get pods -o yaml?
agent-health-status.json :
{
"mmsStatus": {
"mongodb-test-0": {
"errorCode": 0,
"errorString": "<mongodb-test-0> [08:21:13.191] Plan execution failed on step UpdateFeatureCompatibilityVersion as part of move UpdateFeatureCompatibilityVersion : <mongodb-test-0> [08:21:13.191] Failed to apply action. Result = <nil> : <mongodb-test-0> [08:21:13.191] Error running setFeatureCompatibilityVersion command : <mongodb-test-0> [08:21:13.191] Error executing WithClientFor() for cp=mongodb-test-0.mongodb-test-svc.mongodb-test.svc.cluster.local:27017 (local=false) connectMode=SingleConnect : <mongodb-test-0> [08:21:13.191] Error running command for runCommandWithTimeout(dbName=admin, cmd=[{setFeatureCompatibilityVersion 6.0}]) : result=null identityUsed=mms-automation@admin[[MONGODB-CR/SCRAM-SHA-1]][20] : (Location7369100) Once you have downgraded the FCV, if you choose to downgrade the binary version, it will require support assistance. Please re-run this command with 'confirm: true' to acknowledge this and continue with the FCV downgrade.",
"lastGoalVersionAchieved": -1,
"name": "mongodb-test-0",
"plans": [
{
"automationConfigVersion": 1,
"completed": null,
"moves": [
{
"move": "Start",
"moveDoc": "Start the process",
"steps": [
{
"completed": "2023-09-26T08:17:51.245483491Z",
"isWaitStep": false,
"result": "success",
"started": "2023-09-26T08:17:48.399397118Z",
"step": "StartFresh",
"stepDoc": "Start a mongo instance (start fresh)"
}
]
},
{
"move": "WaitAllRsMembersUp",
"moveDoc": "Wait until all members of this process' repl set are up",
"steps": [
{
"completed": "2023-09-26T08:17:51.245675773Z",
"isWaitStep": true,
"result": "success",
"started": "2023-09-26T08:17:51.245516237Z",
"step": "WaitAllRsMembersUp",
"stepDoc": "Wait until all members of this process' repl set are up"
}
]
},
{
"move": "RsInit",
"moveDoc": "Initialize a replica set including the current MongoDB process",
"steps": [
{
"completed": null,
"isWaitStep": false,
"result": "error",
"started": "2023-09-26T08:17:51.245709093Z",
"step": "RsInit",
"stepDoc": "Initialize a replica set"
}
]
},
{
"move": "WaitFeatureCompatibilityVersionCorrect",
"moveDoc": "Wait for featureCompatibilityVersion to be right",
"steps": [
{
"completed": null,
"isWaitStep": true,
"result": "",
"started": null,
"step": "WaitFeatureCompatibilityVersionCorrect",
"stepDoc": "Wait for featureCompatibilityVersion to be right"
}
]
}
],
"started": "2023-09-26T08:17:48.39937121Z"
},
{
"automationConfigVersion": 1,
"completed": null,
"moves": [
{
"move": "EnsureAutomationCredentials",
"moveDoc": "Ensure the automation user exists",
"steps": [
{
"completed": "2023-09-26T08:17:52.452428988Z",
"isWaitStep": false,
"result": "success",
"started": "2023-09-26T08:17:52.401781839Z",
"step": "EnsureAutomationCredentials",
"stepDoc": "Ensure the automation user exists"
}
]
},
{
"move": "AdjustUsers",
"moveDoc": "Adjust Users",
"steps": [
{
"completed": "2023-09-26T08:17:52.47687078Z",
"isWaitStep": false,
"result": "success",
"started": "2023-09-26T08:17:52.452472329Z",
"step": "AdjustUsers",
"stepDoc": "Adjust Users"
}
]
},
{
"move": "UpdateFeatureCompatibilityVersion",
"moveDoc": "Update the featureCompatibilityVersion",
"steps": [
{
"completed": null,
"isWaitStep": false,
"result": "error",
"started": "2023-09-26T08:17:52.476913385Z",
"step": "UpdateFeatureCompatibilityVersion",
"stepDoc": "Update the featureCompatibilityVersion"
}
]
}
],
"started": "2023-09-26T08:17:52.401767338Z"
}
]
}
},
"statuses": {
"mongodb-test-0": {
"ExpectedToBeUp": true,
"IsInGoalState": false,
"LastMongoUpTime": 1695716498,
"ReplicationStatus": 1
}
}
}
kubectl get pods -o yaml --namespace mongodb-test
apiVersion: v1
items:
- apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2023-09-26T08:17:36Z"
generateName: mongodb-kubernetes-operator-86756c8765-
labels:
name: mongodb-kubernetes-operator
pod-template-hash: 86756c8765
name: mongodb-kubernetes-operator-86756c8765-7k2fb
namespace: mongodb-test
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: mongodb-kubernetes-operator-86756c8765
uid: a5384147-4164-45fe-8392-ec923da3c487
resourceVersion: "42048412"
uid: 1aedfad2-1777-498e-8cb8-78d400602cca
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: name
operator: In
values:
- mongodb-kubernetes-operator
topologyKey: kubernetes.io/hostname
containers:
- command:
- /usr/local/bin/entrypoint
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: OPERATOR_NAME
value: mongodb-kubernetes-operator
- name: AGENT_IMAGE
value: quay.io/mongodb/mongodb-agent:12.0.25.7724-1
- name: VERSION_UPGRADE_HOOK_IMAGE
value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.7
- name: READINESS_PROBE_IMAGE
value: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.15
- name: MONGODB_IMAGE
value: mongodb-community-server
- name: MONGODB_REPO_URL
value: docker.io/mongodb
image: quay.io/mongodb/mongodb-kubernetes-operator:0.8.2
imagePullPolicy: Always
name: mongodb-kubernetes-operator
resources:
limits:
cpu: 1100m
memory: 1Gi
requests:
cpu: 500m
memory: 200Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 2000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-2xm7r
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: dalehead
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
seccompProfile:
type: RuntimeDefault
serviceAccount: mongodb-kubernetes-operator
serviceAccountName: mongodb-kubernetes-operator
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-2xm7r
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2023-09-26T08:17:36Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2023-09-26T08:17:39Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2023-09-26T08:17:39Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2023-09-26T08:17:36Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://a3b6e98100ffa7e6ea298d5be94ed980cb66802adacdbc26598acaf355f8cb5e
image: quay.io/mongodb/mongodb-kubernetes-operator:0.8.2
imageID: quay.io/mongodb/mongodb-kubernetes-operator@sha256:0a6c482d3b88bca283cd50e7d1b65d32b3358e3ed634ca50b81f9c056fc4db73
lastState: {}
name: mongodb-kubernetes-operator
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2023-09-26T08:17:39Z"
hostIP: 10.0.202.223
phase: Running
podIP: 10.42.0.88
podIPs:
- ip: 10.42.0.88
qosClass: Burstable
startTime: "2023-09-26T08:17:36Z"
- apiVersion: v1
kind: Pod
metadata:
annotations:
agent.mongodb.com/version: "-1"
creationTimestamp: "2023-09-26T08:17:40Z"
generateName: mongodb-test-
labels:
app: mongodb-test-svc
controller-revision-hash: mongodb-test-6cdb594554
statefulset.kubernetes.io/pod-name: mongodb-test-0
name: mongodb-test-0
namespace: mongodb-test
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: mongodb-test
uid: d8f48bfe-4938-4e0b-adb1-e73e0a2fdbef
resourceVersion: "42048521"
uid: 76bd5442-16b1-469f-8581-86ac4dff57af
spec:
containers:
- args:
- ""
command:
- /bin/sh
- -c
- |2+
#run post-start hook to handle version changes
/hooks/version-upgrade
# wait for config and keyfile to be created by the agent
while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;
# start mongod with this configuration
exec mongod -f /data/automation-mongod.conf;
env:
- name: AGENT_STATUS_FILEPATH
value: /healthstatus/agent-health-status.json
image: docker.io/mongodb/mongodb-community-server:7.0.1-ubi8
imagePullPolicy: IfNotPresent
name: mongod
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data-volume
- mountPath: /healthstatus
name: healthstatus
- mountPath: /hooks
name: hooks
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-test-keyfile
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-42w8t
readOnly: true
- command:
- /bin/bash
- -c
- |-
current_uid=$(id -u)
AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS} -logLevel ${AGENT_LOG_LEVEL}
env:
- name: AGENT_LOG_LEVEL
value: INFO
- name: AGENT_MAX_LOG_FILE_DURATION_HOURS
value: "24"
- name: AGENT_STATUS_FILEPATH
value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
- name: AUTOMATION_CONFIG_MAP
value: mongodb-test-config
- name: HEADLESS_AGENT
value: "true"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/mongodb/mongodb-agent:12.0.25.7724-1
imagePullPolicy: Always
name: mongodb-agent
readinessProbe:
exec:
command:
- /opt/scripts/readinessprobe
failureThreshold: 40
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/lib/automation/config
name: automation-config
readOnly: true
- mountPath: /data
name: data-volume
- mountPath: /var/log/mongodb-mms-automation/healthstatus
name: healthstatus
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-test-keyfile
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-42w8t
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostname: mongodb-test-0
initContainers:
- command:
- cp
- version-upgrade-hook
- /hooks/version-upgrade
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.7
imagePullPolicy: Always
name: mongod-posthook
resources: {}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /hooks
name: hooks
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-42w8t
readOnly: true
- command:
- cp
- /probes/readinessprobe
- /opt/scripts/readinessprobe
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.15
imagePullPolicy: Always
name: mongodb-agent-readinessprobe
resources: {}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-42w8t
readOnly: true
nodeName: dalehead
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccount: mongodb-database
serviceAccountName: mongodb-database
subdomain: mongodb-test-svc
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: data-volume-mongodb-test-0
- name: logs-volume
persistentVolumeClaim:
claimName: logs-volume-mongodb-test-0
- emptyDir: {}
name: agent-scripts
- name: automation-config
secret:
defaultMode: 416
secretName: mongodb-test-config
- emptyDir: {}
name: healthstatus
- emptyDir: {}
name: hooks
- emptyDir: {}
name: mongodb-test-keyfile
- emptyDir: {}
name: tmp
- name: kube-api-access-42w8t
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2023-09-26T08:17:47Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2023-09-26T08:17:44Z"
message: 'containers with unready status: [mongodb-agent]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2023-09-26T08:17:44Z"
message: 'containers with unready status: [mongodb-agent]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2023-09-26T08:17:44Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://f93cad0e98e8310d7c7777d535336cd5ca603d99b35379bb09e7e4cab3f9b8ed
image: docker.io/mongodb/mongodb-community-server:7.0.1-ubi8
imageID: docker.io/mongodb/mongodb-community-server@sha256:bbe995eaf0b1aa24016b73e1dacdefe732fc28b567013da9be8c06db9e7ab3be
lastState: {}
name: mongod
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2023-09-26T08:17:47Z"
- containerID: containerd://20555a5c1f913041164d3c0b70695aa848da4f7d8218e22c8fdd9416a4c79ead
image: quay.io/mongodb/mongodb-agent:12.0.25.7724-1
imageID: quay.io/mongodb/mongodb-agent@sha256:5ea2046331ea1bce66a01d22bc37b6b86854e6d1a54e830f5553b1858aff6b5e
lastState: {}
name: mongodb-agent
ready: false
restartCount: 0
started: true
state:
running:
startedAt: "2023-09-26T08:17:48Z"
hostIP: 10.0.202.223
initContainerStatuses:
- containerID: containerd://6765bb84b94e85d81489ee4a7dd06e2f44f1ca2d427fbd06cf0acb7727e64437
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.7
imageID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:a86a1ad8df61694dd3268bcb0c62a51b2a8edf1ad9e40093690400ec6f270576
lastState: {}
name: mongod-posthook
ready: true
restartCount: 0
state:
terminated:
containerID: containerd://6765bb84b94e85d81489ee4a7dd06e2f44f1ca2d427fbd06cf0acb7727e64437
exitCode: 0
finishedAt: "2023-09-26T08:17:46Z"
reason: Completed
startedAt: "2023-09-26T08:17:46Z"
- containerID: containerd://6dc8162ff4c6941d3ceb0403ffcf7886932bc28c06c24ac9f2499f566b862db8
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.15
imageID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:431d12d07bdb63e7d8f49cbb31f2b11a848ad53c913035e45103cec4e78c438c
lastState: {}
name: mongodb-agent-readinessprobe
ready: true
restartCount: 0
state:
terminated:
containerID: containerd://6dc8162ff4c6941d3ceb0403ffcf7886932bc28c06c24ac9f2499f566b862db8
exitCode: 0
finishedAt: "2023-09-26T08:17:47Z"
reason: Completed
startedAt: "2023-09-26T08:17:47Z"
phase: Running
podIP: 10.42.0.91
podIPs:
- ip: 10.42.0.91
qosClass: Burstable
startTime: "2023-09-26T08:17:44Z"
kind: List
metadata:
resourceVersion: ""
OK so with that info it looks like the set FCV is not passing {confirm: true} to setFeatureCompatibilityVersion
The 6.0 format was:
db.adminCommand({ setFeatureCompatibilityVersion: '6.0'})
The 7.0 format is now:
db.adminCommand({ setFeatureCompatibilityVersion: '7.0', confirm: true})
c.f. https://www.mongodb.com/docs/manual/reference/command/setFeatureCompatibilityVersion/#confirm
confirm
Required
New in version 7.0.
Set to true to confirm the feature compatibility change and allow the operation to proceed.
If you omit the confirm parameter or set confirm to a value other than true, the command fails and returns a warning about modifying the feature compatibility version. The warning states that after you upgrade or downgrade your cluster's fCV, you cannot downgrade the binary version without support assistance.
Hey there
MongoDB deployment with server version >= 7.0 is currently not supported by the community operator. We are working on adding this support to the community operator soon.
This issue is being marked stale because it has been open for 60 days with no activity. Please comment if this issue is still affecting you. If there is no change, this issue will be closed in 30 days.
up
This issue was closed because it became stale and did not receive further updates. If the issue is still affecting you, please re-open it, or file a fresh Issue with updated information.