Endless Rolling Restarts when using Tolerations
siegenthalerroger opened this issue · comments
What did you do to encounter the bug?
Had a MongoDB resource with v0.8.0 like below (incomplete) and then upgraded to v0.8.1.
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: mongodb
namespace: demo-mongodb
spec:
members: 2
type: ReplicaSet
version: "6.0.8"
statefulSet:
spec:
template:
metadata:
labels:
variant: global
spec:
containers:
- ... (resource configuration for agent + mongod, aswell as mongo-exporter sidecar)
tolerations:
- key: "optimised-for"
operator: "Equal"
value: "memory"
effect: "PreferNoSchedule"
What did you expect?
The cluster to continue to operate with the newer operator.
What happened instead?
There was an endless rolling restart loop between the two replicas.
Downgrading to 0.8.0 rectified the issue.
I triaged this to be due to the operator repeatedly adding the "toleration" defined above to the StatefulSet, causing a state difference which k8s reconciled by triggering a rolling update. I suspect this bug was introduced by #1314.
Following a cleaned-up variant of the operator-reconciled mongo StatefulSet:
```
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongodb
namespace: demo-mongodb
uid: 1922f590-e379-4bff-8362-8a7a5cbfc352
resourceVersion: '242788943'
generation: 2798
creationTimestamp: '2023-08-02T11:41:33Z'
ownerReferences:
- apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
name: mongodb
uid: 3bbe2f15-b9d8-4c26-8507-eb538927bcce
controller: true
blockOwnerDeletion: true
managedFields:
- manager: manager
operation: Update
apiVersion: apps/v1
time: '2023-08-02T11:41:33Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:ownerReferences:
.: {}
k:{"uid":"3bbe2f15-b9d8-4c26-8507-eb538927bcce"}: {}
f:spec:
f:podManagementPolicy: {}
f:replicas: {}
f:revisionHistoryLimit: {}
f:selector: {}
f:serviceName: {}
f:template:
f:metadata:
f:labels:
.: {}
f:app: {}
f:variant: {}
f:spec:
f:containers:
k:{"name":"mongo-exporter"}:
.: {}
f:args: {}
f:env:
.: {}
k:{"name":"MONGODB_URI"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources:
.: {}
f:limits:
.: {}
f:cpu: {}
f:memory: {}
f:requests:
.: {}
f:cpu: {}
f:memory: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
k:{"name":"mongod"}:
.: {}
f:args: {}
f:command: {}
f:env:
.: {}
k:{"name":"AGENT_STATUS_FILEPATH"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources:
.: {}
f:limits:
.: {}
f:cpu: {}
f:memory: {}
f:requests:
.: {}
f:cpu: {}
f:memory: {}
f:securityContext:
.: {}
f:allowPrivilegeEscalation: {}
f:readOnlyRootFilesystem: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/data"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/healthstatus"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/hooks"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/tmp"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/var/lib/mongodb-mms-automation/authentication"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/var/log/mongodb-mms-automation"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"name":"mongodb-agent"}:
.: {}
f:command: {}
f:env:
.: {}
k:{"name":"AGENT_LOG_LEVEL"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"AGENT_MAX_LOG_FILE_DURATION_HOURS"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"AGENT_STATUS_FILEPATH"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"AUTOMATION_CONFIG_MAP"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"HEADLESS_AGENT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"POD_NAMESPACE"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:readinessProbe:
.: {}
f:exec:
.: {}
f:command: {}
f:failureThreshold: {}
f:initialDelaySeconds: {}
f:periodSeconds: {}
f:successThreshold: {}
f:timeoutSeconds: {}
f:resources:
.: {}
f:limits:
.: {}
f:cpu: {}
f:memory: {}
f:requests:
.: {}
f:cpu: {}
f:memory: {}
f:securityContext:
.: {}
f:allowPrivilegeEscalation: {}
f:readOnlyRootFilesystem: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/data"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/opt/scripts"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/tmp"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/var/lib/automation/config"}:
.: {}
f:mountPath: {}
f:name: {}
f:readOnly: {}
k:{"mountPath":"/var/lib/mongodb-mms-automation/authentication"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/var/log/mongodb-mms-automation"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/var/log/mongodb-mms-automation/healthstatus"}:
.: {}
f:mountPath: {}
f:name: {}
f:dnsPolicy: {}
f:initContainers:
.: {}
k:{"name":"mongod-posthook"}:
.: {}
f:command: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:securityContext:
.: {}
f:allowPrivilegeEscalation: {}
f:readOnlyRootFilesystem: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/hooks"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"name":"mongodb-agent-readinessprobe"}:
.: {}
f:command: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:securityContext:
.: {}
f:allowPrivilegeEscalation: {}
f:readOnlyRootFilesystem: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/opt/scripts"}:
.: {}
f:mountPath: {}
f:name: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext:
.: {}
f:fsGroup: {}
f:runAsNonRoot: {}
f:runAsUser: {}
f:serviceAccount: {}
f:serviceAccountName: {}
f:terminationGracePeriodSeconds: {}
f:tolerations: {}
f:volumes:
.: {}
k:{"name":"agent-scripts"}:
.: {}
f:emptyDir: {}
f:name: {}
k:{"name":"automation-config"}:
.: {}
f:name: {}
f㊙️
.: {}
f:defaultMode: {}
f:secretName: {}
k:{"name":"healthstatus"}:
.: {}
f:emptyDir: {}
f:name: {}
k:{"name":"hooks"}:
.: {}
f:emptyDir: {}
f:name: {}
k:{"name":"mongodb-keyfile"}:
.: {}
f:emptyDir: {}
f:name: {}
k:{"name":"tmp"}:
.: {}
f:emptyDir: {}
f:name: {}
f:updateStrategy:
f:type: {}
f:volumeClaimTemplates: {}
- manager: kube-controller-manager
operation: Update
apiVersion: apps/v1
time: '2023-08-02T12:03:39Z'
fieldsType: FieldsV1
fieldsV1:
f:status:
f:availableReplicas: {}
f:collisionCount: {}
f:currentRevision: {}
f:observedGeneration: {}
f:readyReplicas: {}
f:replicas: {}
f:updateRevision: {}
f:updatedReplicas: {}
subresource: status
selfLink: /apis/apps/v1/namespaces/demo-mongodb/statefulsets/mongodb
status:
observedGeneration: 2798
replicas: 2
readyReplicas: 1
updatedReplicas: 1
currentRevision: mongodb-86c99f5dbd
updateRevision: mongodb-fc5fc6d9d
collisionCount: 0
availableReplicas: 1
spec:
replicas: 2
selector:
matchLabels:
app: mongodb-svc
template:
metadata:
creationTimestamp: null
labels:
app: mongodb-svc
variant: global
spec:
volumes:
- name: agent-scripts
emptyDir: {}
- name: automation-config
secret:
secretName: mongodb-config
defaultMode: 416
- name: healthstatus
emptyDir: {}
- name: hooks
emptyDir: {}
- name: mongodb-keyfile
emptyDir: {}
- name: tmp
emptyDir: {}
initContainers:
- name: mongod-posthook
image: >-
quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.7
command:
- cp
- version-upgrade-hook
- /hooks/version-upgrade
resources: {}
volumeMounts:
- name: hooks
mountPath: /hooks
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
- name: mongodb-agent-readinessprobe
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.15
command:
- cp
- /probes/readinessprobe
- /opt/scripts/readinessprobe
resources: {}
volumeMounts:
- name: agent-scripts
mountPath: /opt/scripts
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
containers:
- name: mongo-exporter
image: percona/mongodb_exporter:0.39.0
args:
- '--discovering-mode'
- '--collect-all'
- '--collector.dbstats'
- '--collector.topmetrics'
- '--collector.collstats-limit=-1'
- '--log.level=info'
env:
- name: MONGODB_URI
value: mongodb://LOREM-IPSUM
resources:
limits:
cpu: 250m
memory: 50Mi
requests:
cpu: 100m
memory: 50Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
- name: mongod
image: docker.io/mongo:6.0.8
command:
- /bin/sh
- '-c'
- |+
#run post-start hook to handle version changes
/hooks/version-upgrade
# wait for config and keyfile to be created by the agent
while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;
# start mongod with this configuration
exec mongod -f /data/automation-mongod.conf;
args:
- ''
env:
- name: AGENT_STATUS_FILEPATH
value: /healthstatus/agent-health-status.json
resources:
limits:
cpu: '1'
memory: 2500Mi
requests:
cpu: 250m
memory: 2500Mi
volumeMounts:
- name: data-volume
mountPath: /data
- name: healthstatus
mountPath: /healthstatus
- name: hooks
mountPath: /hooks
- name: logs-volume
mountPath: /var/log/mongodb-mms-automation
- name: mongodb-keyfile
mountPath: /var/lib/mongodb-mms-automation/authentication
- name: tmp
mountPath: /tmp
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
- name: mongodb-agent
image: quay.io/mongodb/mongodb-agent:12.0.24.7719-1
command:
- /bin/bash
- '-c'
- >-
current_uid=$(id -u)
AGENT_API_KEY="$(cat
/mongodb-automation/agent-api-key/agentApiKey)"
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent
-healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
-serveStatusPort=5000
-cluster=/var/lib/automation/config/cluster-config.json
-skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile
/var/log/mongodb-mms-automation/automation-agent.log
-maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS}
-logLevel ${AGENT_LOG_LEVEL}
env:
- name: AGENT_LOG_LEVEL
value: INFO
- name: AGENT_MAX_LOG_FILE_DURATION_HOURS
value: '24'
- name: AGENT_STATUS_FILEPATH
value: >-
/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
- name: AUTOMATION_CONFIG_MAP
value: mongodb-config
- name: HEADLESS_AGENT
value: 'true'
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
resources:
limits:
cpu: 200m
memory: 100Mi
requests:
cpu: 50m
memory: 100Mi
volumeMounts:
- name: agent-scripts
mountPath: /opt/scripts
- name: automation-config
readOnly: true
mountPath: /var/lib/automation/config
- name: data-volume
mountPath: /data
- name: healthstatus
mountPath: /var/log/mongodb-mms-automation/healthstatus
- name: logs-volume
mountPath: /var/log/mongodb-mms-automation
- name: mongodb-keyfile
mountPath: /var/lib/mongodb-mms-automation/authentication
- name: tmp
mountPath: /tmp
readinessProbe:
exec:
command:
- /opt/scripts/readinessprobe
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 40
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: mongodb-database
serviceAccount: mongodb-database
securityContext:
runAsUser: 2000
runAsNonRoot: true
fsGroup: 2000
schedulerName: default-scheduler
tolerations:
- key: optimised-for
operator: Equal
value: memory
effect: PreferNoSchedule
- key: optimised-for
operator: Equal
value: memory
effect: PreferNoSchedule
- key: optimised-for
operator: Equal
value: memory
effect: PreferNoSchedule
- key: optimised-for
operator: Equal
value: memory
effect: PreferNoSchedule
- ... quite literally thousands more copies
volumeClaimTemplates:
- kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: data-volume
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: ssd-retained
volumeMode: Filesystem
status:
phase: Pending
- kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: logs-volume
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2G
volumeMode: Filesystem
status:
phase: Pending
serviceName: mongodb-svc
podManagementPolicy: OrderedReady
updateStrategy:
type: RollingUpdate
revisionHistoryLimit: 10
</details>
**Operator Information**
- Operator Version: v0.8.1
- MongoDB Image used: v6.0.8
**Kubernetes Cluster Information**
- Distribution: AKS
- Version: 1.24
- Image Registry location: quay.io
@siegenthalerroger Thank you for raising this, it is indeed a bug addressed with #1346 . We're releasing a fix version shortly.
@siegenthalerroger Closing this as it was addressed with the 0.8.2 version. Please re-open if you're still encountering the issue. Thank you for bringing the issue up so quickly!
Also tested on my end, working as intended now :)