Replica set does not start, stuck in phase: Pending
heruan opened this issue · comments
What did you do to encounter the bug?
Steps to reproduce the behavior:
- Follow the docs in https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/docs/deploy-configure.md
- Run
kubectl get mongodbcommunity --namespace mongodb
What did you expect?
NAME PHASE VERSION
example-mongodb Running 6.0.5
What happened instead?
NAME PHASE VERSION
example-mongodb Pending
Operator Information
- Operator Version: 0.8.2
- MongoDB Image used: the one from the deploy-configure docs
Kubernetes Cluster Information
- Distribution: docker-desktop
- Version: 4.23.0
Additional context
Operator logs:
2023-09-21T15:39:41.583Z INFO manager/main.go:74 Watching namespace: mongodb
2023-09-21T15:39:41.673Z INFO manager/main.go:91 Registering Components.
2023-09-21T15:39:41.676Z INFO manager/main.go:104 Starting the Cmd.
2023-09-21T15:40:06.420Z INFO controllers/replica_set_controller.go:139 Reconciling MongoDB {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.420Z DEBUG controllers/replica_set_controller.go:141 Validating MongoDB.Spec {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.421Z DEBUG controllers/replica_set_controller.go:151 Ensuring the service exists {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.630Z DEBUG agent/replica_set_port_manager.go:122 No port change required {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.658Z INFO controllers/replica_set_controller.go:470 Create/Update operation succeeded {"ReplicaSet": "mongodb/example-mongodb", "operation": "created"}
2023-09-21T15:40:06.661Z INFO controllers/replica_set_controller.go:362 Creating/Updating AutomationConfig {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.663Z DEBUG scram/scram.go:199 No existing credentials found, generating new credentials
2023-09-21T15:40:06.663Z DEBUG scram/scram.go:177 Generating new credentials and storing in secret/my-scram-scram-credentials
2023-09-21T15:40:06.757Z DEBUG scram/scram.go:188 Successfully generated SCRAM credentials
2023-09-21T15:40:06.762Z DEBUG agent/replica_set_port_manager.go:122 No port change required {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.763Z DEBUG agent/replica_set_port_manager.go:40 Calculated process port map: map[example-mongodb-0:27017 example-mongodb-1:27017 example-mongodb-2:27017] {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.763Z DEBUG controllers/replica_set_controller.go:537 AutomationConfigMembersThisReconciliation {"mdb.AutomationConfigMembersThisReconciliation()": 3}
2023-09-21T15:40:06.771Z DEBUG controllers/replica_set_controller.go:381 The existing StatefulSet did not have the readiness probe init container, skipping pod annotation check. {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.772Z INFO controllers/replica_set_controller.go:337 Creating/Updating StatefulSet {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.793Z INFO controllers/replica_set_controller.go:342 Creating/Updating StatefulSet for Arbiters {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.802Z DEBUG controllers/replica_set_controller.go:352 Ensuring StatefulSet is ready, with type: RollingUpdate {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.803Z INFO controllers/mongodb_status_options.go:110 ReplicaSet is not yet ready, retrying in 10 seconds
2023-09-21T15:40:06.825Z INFO controllers/replica_set_controller.go:139 Reconciling MongoDB {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.825Z DEBUG controllers/replica_set_controller.go:141 Validating MongoDB.Spec {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.825Z DEBUG controllers/replica_set_controller.go:151 Ensuring the service exists {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.831Z DEBUG agent/agent_readiness.go:106 The Pod 'example-mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.831Z DEBUG agent/replica_set_port_manager.go:122 No port change required {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.834Z INFO controllers/replica_set_controller.go:470 Create/Update operation succeeded {"ReplicaSet": "mongodb/example-mongodb", "operation": "updated"}
2023-09-21T15:40:06.835Z INFO controllers/replica_set_controller.go:362 Creating/Updating AutomationConfig {"ReplicaSet": "mongodb/example-mongodb"}
2023-09-21T15:40:06.910Z DEBUG scram/scram.go:172 Credentials have not changed, using credentials stored in: secret/my-scram-scram-credentials
and then it loops with the same lines.
Logs of the mongodb container:
2023-09-21T16:11:40.592Z INFO versionhook/main.go:32 Running version change post-start hook
2023-09-21T16:11:40.597Z INFO versionhook/main.go:39 Waiting for agent health status...
2023-09-21T16:11:41.603Z INFO versionhook/main.go:45 Agent health status file not found, mongod will start
and then nothing.
This is the YAML file I'm using, straight from the docs:
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: example-mongodb
spec:
members: 3
type: ReplicaSet
version: "6.0.5"
security:
authentication:
modes: ["SCRAM"]
users:
- name: my-user
db: admin
passwordSecretRef:
name: my-user-password
roles:
- name: clusterAdmin
db: admin
- name: userAdminAnyDatabase
db: admin
scramCredentialsSecretName: my-scram
additionalMongodConfig:
storage.wiredTiger.engineConfig.journalCompressor: zlibDescribe of the pending pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m22s default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
Normal Scheduled 2m20s default-scheduler Successfully assigned mongodb/example-mongodb-0 to docker-desktop
Normal Pulling 2m19s kubelet Pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.7"
Normal Pulled 2m18s kubelet Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.7" in 1.708012167s (1.70804721s including waiting)
Normal Created 2m18s kubelet Created container mongod-posthook
Normal Started 2m18s kubelet Started container mongod-posthook
Normal Pulling 2m17s kubelet Pulling image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.15"
Normal Pulled 2m16s kubelet Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.15" in 1.720200334s (1.720207668s including waiting)
Normal Created 2m16s kubelet Created container mongodb-agent-readinessprobe
Normal Started 2m15s kubelet Started container mongodb-agent-readinessprobe
Normal Pulled 2m15s kubelet Container image "docker.io/mongodb/mongodb-community-server:6.0.5-ubi8" already present on machine
Normal Created 2m15s kubelet Created container mongod
Normal Started 2m15s kubelet Started container mongod
Normal Pulling 2m15s kubelet Pulling image "quay.io/mongodb/mongodb-agent:12.0.25.7724-1"
Normal Pulled 2m13s kubelet Successfully pulled image "quay.io/mongodb/mongodb-agent:12.0.25.7724-1" in 1.967205626s (1.967212418s including waiting)
Normal Created 2m13s kubelet Created container mongodb-agent
Normal Started 2m13s kubelet Started container mongodb-agent
Warning Unhealthy 39s (x10 over 119s) kubelet Readiness probe failed:
Trying to access MongoDB from inside the container:
kubectl exec -it example-mongodb-0 -n mongodb -- /bin/bash
Defaulted container "mongod" out of: mongod, mongodb-agent, mongod-posthook (init), mongodb-agent-readinessprobe (init)
bash-4.4$ mongosh
Warning: Could not access file: EACCES: permission denied, mkdir '/data/db/.mongodb'
Current Mongosh Log ID: 650c6d3b735fc3da1aac25f7
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.10.6
MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017
bash-4.4$ tail /var/log/mongodb/mongod.log
tail: cannot open '/var/log/mongodb/mongod.log' for reading: Permission denied
Focusing in on 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
You will need PVs for your data; do you have a PV provisioner? If not you can create your own and configure the operator to use those PVs.
Thanks! I'm following the docs at deploy-configure.md, did I miss something about the PV?
If I run kubectl get pv I see the PV being created:
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-83e9bc02-7f8e-4135-8d14-9266475d4ef6 10G RWO Delete Bound mongodb/data-volume-example-mongodb-0 hostpath 147m
pvc-baf8ef90-720e-421f-b6e8-00c1f46758d9 2G RWO Delete Bound mongodb/logs-volume-example-mongodb-0 hostpath 147m
Thanks! I'm following the docs at deploy-configure.md, did I miss something about the PV?
If I run
kubectl get pvI see the PV being created:$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-83e9bc02-7f8e-4135-8d14-9266475d4ef6 10G RWO Delete Bound mongodb/data-volume-example-mongodb-0 hostpath 147m pvc-baf8ef90-720e-421f-b6e8-00c1f46758d9 2G RWO Delete Bound mongodb/logs-volume-example-mongodb-0 hostpath 147m
OK that's not the issue then.
Thank you for the feedback anyways! I noticed we both opened a similar issue at the same time, not sure if they're related. Hope to get this fixed soon!
It seems for some reason the Readiness Probe didn't return ready state:
Warning Unhealthy 39s (x10 over 119s) kubelet Readiness probe failed:
Could you please paste the content of the /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json and the /var/log/mongodb-mms-automation/readiness.log file from the container?
Thanks! Sure, here they are:
$ kubectl exec -it example-mongodb-0 -c mongodb-agent -n mongodb -- /bin/bash
groups: cannot find name for group ID 2000
I have no name!@example-mongodb-0:/$ cat /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
{"statuses":{"example-mongodb-0":{"IsInGoalState":false,"LastMongoUpTime":0,"ExpectedToBeUp":true,"ReplicationStatus":-1}},"mmsStatus":{"mongodb-rs-0":{"name":"example-mongodb-0","lastGoalVersionAchieved":-1,"plans":[{"automationConfigVersion":1,"started":"2023-09-26T07:58:02.694785842Z","completed":null,"moves":[{"move":"Start","moveDoc":"Start the process","steps":[{"step":"StartFresh","stepDoc":"Start a mongo instance (start fresh)","isWaitStep":false,"started":"2023-09-26T07:58:02.695242259Z","completed":null,"result":""}]},{"move":"WaitAllRsMembersUp","moveDoc":"Wait until all members of this process' repl set are up","steps":[{"step":"WaitAllRsMembersUp","stepDoc":"Wait until all members of this process' repl set are up","isWaitStep":true,"started":null,"completed":null,"result":""}]},{"move":"RsInit","moveDoc":"Initialize a replica set including the current MongoDB process","steps":[{"step":"RsInit","stepDoc":"Initialize a replica set","isWaitStep":false,"started":null,"completed":null,"result":""}]},{"move":"WaitFeatureCompatibilityVersionCorrect","moveDoc":"Wait for featureCompatibilityVersion to be right","steps":[{"step":"WaitFeatureCompatibilityVersionCorrect","stepDoc":"Wait for featureCompatibilityVersion to be right","isWaitStep":true,"started":null,"completed":null,"result":""}]}]}],"errorCode":0,"errorString":""}}}
and
$ kubectl exec -it example-mongodb-0 -c mongodb-agent -n mongodb -- /bin/bash
groups: cannot find name for group ID 2000
I have no name!@example-mongodb-0:/$ cat /var/log/mongodb-mms-automation/readiness.log
{"level":"info","ts":1695312716.3761523,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312716.3794947,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312716.6948318,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312716.6977067,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312726.3701475,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312726.3729086,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312736.3802369,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312736.3829937,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312746.3738763,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312746.3767989,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312756.3890557,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312756.3916657,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312766.3830001,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312766.386074,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312776.3749304,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312776.3773458,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312786.38313,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312786.3851404,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312796.3888578,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312796.3915386,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312797.448246,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312797.4511156,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312806.3795326,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312806.3825727,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312816.3652399,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312816.3678174,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312826.373138,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312826.3760192,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312836.3845918,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312836.387063,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312846.373052,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312846.3755276,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312856.3756347,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312856.377877,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312858.4724293,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312858.4753454,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312866.3744965,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312866.3768694,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312876.3787057,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312876.3813832,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312886.3916223,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312886.3961985,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312896.3766682,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312896.3790946,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312906.3693953,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312906.3724024,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312916.3794153,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312916.3821118,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312926.3776546,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312926.3819761,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312930.431642,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312930.4346752,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312936.3699937,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312936.3727434,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312946.3719578,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312946.37445,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312956.3791192,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312956.381972,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312966.3627295,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312966.36617,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312976.3743665,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312976.3769166,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312986.3686726,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312986.371401,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312996.38496,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312996.3875685,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695312999.412998,"msg":"Mongod is not ready"}
{"level":"info","ts":1695312999.433647,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313006.383269,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313006.3861408,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313016.3727274,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313016.3759387,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313026.3737447,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313026.3760746,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313036.3753963,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313036.3780744,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313046.3685672,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313046.3713136,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313056.370764,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313056.373624,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313066.371384,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313066.3740685,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313076.3665602,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313076.3693192,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313086.3664322,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313086.3689969,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313114.6524863,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313114.6559436,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313124.6464264,"msg":"The Agent has reached goal state. Returning ready."}
{"level":"info","ts":1695313124.9466996,"msg":"The Agent has reached goal state. Returning ready."}
{"level":"info","ts":1695313134.640301,"msg":"The Agent has reached goal state. Returning ready."}
{"level":"info","ts":1695313144.6818528,"msg":"The Agent has reached goal state. Returning ready."}
{"level":"info","ts":1695313154.6773405,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313154.6798635,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313164.6891134,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313164.6918461,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313174.6819024,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313174.6860795,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313184.6854167,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313184.6883616,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313194.6722212,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313194.6741483,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313204.685962,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313204.68928,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313214.675309,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313214.6784883,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313224.6714134,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313224.673464,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313234.6683872,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313234.671147,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313244.6750133,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313244.678348,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313254.6770575,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313254.6803813,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313264.6942031,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313264.7027066,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313274.6725745,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313274.675629,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313284.685739,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313284.6886654,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313294.6668863,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313294.6696293,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313304.6918561,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313304.6939044,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313314.6822872,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313314.6855478,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313324.6770878,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313324.680386,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313334.6697748,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313334.6724405,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313344.6746085,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313344.677905,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313354.6673555,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313354.6754143,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695313364.6622078,"msg":"Mongod is not ready"}
{"level":"info","ts":1695313364.6739702,"msg":"Reached the end of the check. Returning not ready."}
{"level":"error","ts":1695313374.6839797,"msg":"There was problem checking the health status: failed to fetch automation-config secret name: example-mongodb-config, err: secrets \"example-mongodb-config\" not found"}
{"level":"error","ts":1695313384.660656,"msg":"There was problem checking the health status: failed to fetch automation-config secret name: example-mongodb-config, err: secrets \"example-mongodb-config\" not found"}
{"level":"error","ts":1695313394.6531646,"msg":"There was problem checking the health status: failed to fetch automation-config secret name: example-mongodb-config, err: secrets \"example-mongodb-config\" not found"}
{"level":"info","ts":1695715093.3670304,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715093.3698885,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715093.6991317,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715093.7014601,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715103.3865538,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715103.3896227,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715113.3663273,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715113.3701966,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715123.3671596,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715123.3707056,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715133.354212,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715133.368764,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715143.3881218,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715143.3905241,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715153.3673022,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715153.3700907,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715163.3775995,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715163.380205,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715172.3848653,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715172.3877163,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715173.3711038,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715173.3739305,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715183.3777401,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715183.3799734,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715193.3484478,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715193.3684816,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715203.3683298,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715203.3706539,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715213.3786557,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715213.3813946,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715223.3809993,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715223.3836586,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715233.36764,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715233.3701656,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715236.3748634,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715236.377498,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715243.377472,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715243.3799725,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715253.3498514,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715253.370531,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715263.3701518,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715263.3727472,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715273.371194,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715273.3736343,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715283.3773427,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715283.3801985,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715293.3771265,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715293.3796954,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715303.3824048,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715303.3853998,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715313.3710415,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715313.3734798,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715320.3917735,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715320.395481,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715323.382934,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715323.3856268,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715333.3685658,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715333.3712955,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715343.3767338,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715343.3792107,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715353.3534365,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715353.3682754,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715363.3711793,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715363.3739078,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715373.3704822,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715373.3736026,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715383.3848758,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715383.3883517,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715393.3684726,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715393.371185,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715402.398974,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715402.4014869,"msg":"Reached the end of the check. Returning not ready."}
{"level":"info","ts":1695715403.3774443,"msg":"Mongod is not ready"}
{"level":"info","ts":1695715403.3802712,"msg":"Reached the end of the check. Returning not ready."}
Were the logs helpful to investigate on this? Can I provide more to help fixing this?
I pulled the latest commits and re-installed the operator, now the pods start. Not sure which change made the trick, but thanks!