[QUESTION] Use reloader with Custom Resource Definitions that create deployments?
romancin opened this issue · comments
I am trying to use reloader with a Deployment that is created by a Custom Resource Definition managed by an operator. I annotated the deployment using the CRD. Reloader creates a new replica set correctly, but when the operator reconciles, it rolls back this change. Is there I way to use reloader for this cases? Thank you very much.
Sorry i need some more information around this to understand.
Can you share the exact change which is reverted?
Of course, I will try to explain it better. I am trying to manage custom configuration changes for Victoria Metrics Agent using reloader. We are deploying the Agent using Victoria Metrics Operator. To deploy a VMAgent using Victoria metrics operator, we create a CRD like this:
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
annotations:
argocd.argoproj.io/sync-wave: "95"
reloader.stakater.com/auto: "true"
labels:
app.kubernetes.io/instance: vmagent
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: victoria-metrics-k8s-stack
app.kubernetes.io/version: v1.94.0
helm.sh/chart: victoria-metrics-k8s-stack-0.18.5
name: vmagent
namespace: victoriametrics
spec:
additionalScrapeConfigs:
key: additional-scrape-configs.yaml
name: additional-scrape-configs
extraArgs:
promscrape.streamParse: "true"
remoteWrite.streamAggr.config: /etc/vm/stream-aggr-configs/stream-aggr-configs.config
remoteWrite.streamAggr.keepInput: "true"
image:
tag: v1.94.0
replicaCount: 2
resources:
limits:
memory: 1024Mi
requests:
cpu: "0.5"
memory: 512Mi
scrapeInterval: 60s
selectAllByDefault: true
volumeMounts:
- mountPath: /etc/vm/stream-aggr-configs
name: stream-aggr-configs
readOnly: true
volumes:
- configMap:
defaultMode: 420
name: stream-aggr-configs
name: stream-aggr-configs
As you can see on it, we are putting the annotation reloader.stakater.com/auto: "true". We want that when changes are made to stream-aggr-configs configMap, reloader takes care of reloading VMAgent. When this CRD is applied, it creates this deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
argocd.argoproj.io/sync-wave: "95"
deployment.kubernetes.io/revision: "7"
reloader.stakater.com/auto: "true"
labels:
app.kubernetes.io/component: monitoring
app.kubernetes.io/instance: vmagent
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: vmagent
app.kubernetes.io/version: v1.94.0
helm.sh/chart: victoria-metrics-k8s-stack-0.18.5
managed-by: vm-operator
name: vmagent-vmagent
namespace: victoriametrics
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: monitoring
app.kubernetes.io/instance: vmagent
app.kubernetes.io/name: vmagent
managed-by: vm-operator
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/component: monitoring
app.kubernetes.io/instance: vmagent
app.kubernetes.io/name: vmagent
managed-by: vm-operator
spec:
containers:
- args:
- --reload-url=http://localhost:8429/-/reload
- --config-envsubst-file=/etc/vmagent/config_out/vmagent.env.yaml
- --watched-dir=/etc/vm/relabeling
- --watched-dir=/etc/vm/stream-aggr
- --config-file=/etc/vmagent/config/vmagent.yaml.gz
command:
- /bin/prometheus-config-reloader
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: quay.io/prometheus-operator/prometheus-config-reloader:v0.68.0
imagePullPolicy: IfNotPresent
name: config-reloader
resources:
limits:
cpu: 100m
memory: 25Mi
requests:
cpu: 100m
memory: 25Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /etc/vmagent/config_out
name: config-out
- mountPath: /etc/vm/relabeling
name: relabeling-assets
readOnly: true
- mountPath: /etc/vm/stream-aggr
name: stream-aggr-conf
readOnly: true
- mountPath: /etc/vmagent/config
name: config
- args:
- -httpListenAddr=:8429
- -promscrape.config=/etc/vmagent/config_out/vmagent.env.yaml
- -promscrape.streamParse=true
- -remoteWrite.maxDiskUsagePerURL=1073741824
- -remoteWrite.streamAggr.config=/etc/vm/stream-aggr-configs/stream-aggr-configs.config
- -remoteWrite.streamAggr.keepInput=true
- -remoteWrite.tmpDataPath=/tmp/vmagent-remotewrite-data
image: victoriametrics/vmagent:v1.94.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 10
httpGet:
path: /health
port: 8429
scheme: HTTP
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
name: vmagent
ports:
- containerPort: 8429
name: http
protocol: TCP
readinessProbe:
failureThreshold: 10
httpGet:
path: /health
port: 8429
scheme: HTTP
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /tmp/vmagent-remotewrite-data
name: persistent-queue-data
- mountPath: /etc/vm/stream-aggr-configs
name: stream-aggr-configs
readOnly: true
- mountPath: /etc/vmagent/config_out
name: config-out
readOnly: true
- mountPath: /etc/vmagent-tls/certs
name: tls-assets
readOnly: true
- mountPath: /etc/vm/relabeling
name: relabeling-assets
readOnly: true
- mountPath: /etc/vm/stream-aggr
name: stream-aggr-conf
readOnly: true
- mountPath: /etc/vmagent/config
name: config
readOnly: true
dnsPolicy: ClusterFirst
initContainers:
- args:
- -c
- gunzip -c /etc/vmagent/config/vmagent.yaml.gz > /etc/vmagent/config_out/vmagent.env.yaml
command:
- /bin/sh
image: quay.io/prometheus-operator/prometheus-config-reloader:v0.68.0
imagePullPolicy: IfNotPresent
name: config-init
resources:
limits:
cpu: 100m
memory: 25Mi
requests:
cpu: 100m
memory: 25Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/vmagent/config
name: config
- mountPath: /etc/vmagent/config_out
name: config-out
restartPolicy: Always
schedulerName: default-scheduler
serviceAccount: vmagent-vmagent
serviceAccountName: vmagent-vmagent
terminationGracePeriodSeconds: 30
volumes:
- name: persistent-queue-data
- configMap:
defaultMode: 420
name: stream-aggr-configs
name: stream-aggr-configs
- name: tls-assets
secret:
defaultMode: 420
secretName: tls-assets-vmagent-vmagent
- name: config-out
- configMap:
defaultMode: 420
name: relabelings-assets-vmagent-vmagent
name: relabeling-assets
- configMap:
defaultMode: 420
name: stream-aggr-vmagent-vmagent
name: stream-aggr-conf
- name: config
secret:
defaultMode: 420
secretName: vmagent-vmagent
When a change to the configMap is made, I can see in reloader that it triggers a reload, but against the Deployment:
time="2023-10-20T13:21:17Z" level=info msg="Changes detected in 'stream-aggr-configs' of type 'CONFIGMAP' in namespace 'victoriametrics', Updated 'vmagent-vmagent' of type 'Deployment' in namespace 'victoriametrics'"
But just before this, here comes the problem. If I execute kubectl describe deploy ... I can see this:
...
events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Reloaded 5m12s reloader-configMaps Changes detected in 'stream-aggr-configs' of type 'CONFIGMAP' in namespace 'victoriametrics', Updated 'vmagent-vmagent' of type 'Deployment' in namespace 'victoriametrics'
Normal ScalingReplicaSet 5m12s deployment-controller Scaled up replica set vmagent-vmagent-5d777c9478 to 1
Normal ScalingReplicaSet 5m11s deployment-controller Scaled down replica set vmagent-vmagent-5d777c9478 to 0
After one second, the new replicaSet is scaled down. I suppose the operator is reverting the change, as something behind it (reloader) is making changes to the object, so the operator reconciles it again.
Can this be solved?
Thank you!
Reloader cannot control how the operator reconciles the custom resources. In your situation, if the old pod is deleted and either the operator or reloader creates the new pod, you should still be fine because the new pod will have the new changes from configmap.