Giters

SumoLogic / sumologic-kubernetes-collection

Sumo Logic collection solution for Kubernetes

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Prometheus Sharding Name Too Long

alexstaroselsky opened this issue · comments

commented

Describe the bug A clear and concise description of what the bug is.

When setting kube-prometheus-stack.prometheus.prometheusSpec.shards to value 2 an error happens related to the name of the shard StatefulSet and/or Pod being too long, even with an extremely short helm release name like "sumo".

How can the name of the kube-prometheus-stack.prometheus deployments be changed to help avoid exceeding maximum number of characters.

Logs Logs which can be helpful in investigating the issue.

Logs for stateful set for prometheus shard(s)"

Events:                                                                                                                                                                                                                                    │
│   Type     Reason        Age                   From                    Message                                                                                                                                                             │
│   ----     ------        ----                  ----                    -------                                                                                                                                                             │
│   Warning  FailedCreate  3m12s (x18 over 14m)  statefulset-controller  create Pod prometheus-sumo-kube-prometheus-stack-prometheus-shard-1-0 in StatefulSet prometheus-sumo-kube-prometheus-stack-prometheus-shard-1 failed error: Pod "pr │
│ ometheus-sumo-kube-prometheus-stack-prometheus-shard-1-0" is invalid: metadata.labels: Invalid value: "prometheus-sumo-kube-prometheus-stack-prometheus-shard-1-74fdcd57b6": must be no more than 63 characters

Command used to install/upgrade Collection Direct command which you used to install/upgrade collection.

helm upgrade --install -n sumologic --create-namespace -f ./values.yaml sumo sumologic/sumologic

Configuration Configuration used for Collection e.g. user-values.yaml for helm.

sumologic:
  accessId: someId
  accessKey: someSecret
  clusterName: my-cool-cluster

  nodeSelector:
    another-node-group: "true"

otellogs:
  kubelet:
    enabled: false

  additionalDaemonSets:
    active:
      nodeSelector:
        some-node-group: "true"
      resources: {}

kube-prometheus-stack:
  grafana:
    enabled: true
    defaultDashboardsEnabled: true
  
  prometheus:
    enabled: true
    prometheusSpec:
      retention: 8d
      shards: 2

  prometheusOperator:
    enabled: true

  kubelet:
    serviceMonitor:
      interval: 5m

To Reproduce Steps to reproduce the behavior.

Execute deployment using helm command above. Inspect events of StatefulSet(s)/Pod(s) related to -shard in namespace sumologic to see error.

Expected behavior A clear and concise description of what you expected to happen.

Name of StatefulSet(s) and/or Pod(s) for prometheus shard should be truncated to maximum of 63 characters to avoid errors. Otherwise provide mechanism to force to a different name to avoid issue.

Environment (please complete the following information):

  • Collection version (e.g. helm ls -n sumologic): 4.4.0
  • Kubernetes version (e.g. kubectl version): v1.29.0-eks-c417bb3
  • Cloud provider: AWS
  • Others:

Anything else do we need to know Add any other context about the problem here, e.g. issue happens only occasionally.

Attempted to override with specifying kube-prometheus-stack.prometheus.prometheusSpec.metadata.name to value like "foobar" to try to force to a shorter name, similar to this issue did not help.

commented

Was able to resolve using kube-prometheus-stack.nameOverride and kube-prometheus-stack.fullNameOverride but still wanted to mention what happens without that override.