Prometheus Sharding Name Too Long
alexstaroselsky opened this issue · comments
Describe the bug A clear and concise description of what the bug is.
When setting kube-prometheus-stack.prometheus.prometheusSpec.shards
to value 2
an error happens related to the name of the shard StatefulSet and/or Pod being too long, even with an extremely short helm release name like "sumo"
.
How can the name of the kube-prometheus-stack.prometheus
deployments be changed to help avoid exceeding maximum number of characters.
Logs Logs which can be helpful in investigating the issue.
Logs for stateful set for prometheus shard(s)"
Events: │
│ Type Reason Age From Message │
│ ---- ------ ---- ---- ------- │
│ Warning FailedCreate 3m12s (x18 over 14m) statefulset-controller create Pod prometheus-sumo-kube-prometheus-stack-prometheus-shard-1-0 in StatefulSet prometheus-sumo-kube-prometheus-stack-prometheus-shard-1 failed error: Pod "pr │
│ ometheus-sumo-kube-prometheus-stack-prometheus-shard-1-0" is invalid: metadata.labels: Invalid value: "prometheus-sumo-kube-prometheus-stack-prometheus-shard-1-74fdcd57b6": must be no more than 63 characters
Command used to install/upgrade Collection Direct command which you used to install/upgrade collection.
helm upgrade --install -n sumologic --create-namespace -f ./values.yaml sumo sumologic/sumologic
Configuration Configuration used for Collection e.g. user-values.yaml for helm.
sumologic:
accessId: someId
accessKey: someSecret
clusterName: my-cool-cluster
nodeSelector:
another-node-group: "true"
otellogs:
kubelet:
enabled: false
additionalDaemonSets:
active:
nodeSelector:
some-node-group: "true"
resources: {}
kube-prometheus-stack:
grafana:
enabled: true
defaultDashboardsEnabled: true
prometheus:
enabled: true
prometheusSpec:
retention: 8d
shards: 2
prometheusOperator:
enabled: true
kubelet:
serviceMonitor:
interval: 5m
To Reproduce Steps to reproduce the behavior.
Execute deployment using helm command above. Inspect events of StatefulSet(s)/Pod(s) related to -shard
in namespace sumologic
to see error.
Expected behavior A clear and concise description of what you expected to happen.
Name of StatefulSet(s) and/or Pod(s) for prometheus shard should be truncated to maximum of 63 characters to avoid errors. Otherwise provide mechanism to force to a different name to avoid issue.
Environment (please complete the following information):
- Collection version (e.g.
helm ls -n sumologic
):4.4.0
- Kubernetes version (e.g.
kubectl version
):v1.29.0-eks-c417bb3
- Cloud provider:
AWS
- Others:
Anything else do we need to know Add any other context about the problem here, e.g. issue happens only occasionally.
Attempted to override with specifying kube-prometheus-stack.prometheus.prometheusSpec.metadata.name
to value like "foobar"
to try to force to a shorter name, similar to this issue did not help.
Was able to resolve using kube-prometheus-stack.nameOverride
and kube-prometheus-stack.fullNameOverride
but still wanted to mention what happens without that override.