Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"aicloud\"" ts=2024-06-05T10:58:52.329Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"aicloud\"" ts=2024-06-05T10:58:52.329Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\"

[yang0009]

When i cretea servicemonitor but no found them in prometheus target lists with release 0.12

Did you expect to see some different?

How to reproduce it (as minimally and precisely as possible):

Environment

• Prometheus Operator version:

  Insert image tag or Git SHA here
  Name: prometheus-operator
  Namespace: monitoring
  CreationTimestamp: Wed, 05 Jun 2024 17:33:37 +0800
  Labels: app.kubernetes.io/component=controller
  app.kubernetes.io/name=prometheus-operator
  app.kubernetes.io/part-of=kube-prometheus
  app.kubernetes.io/version=0.62.0
  Annotations: deployment.kubernetes.io/revision: 1
  Selector: app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus
  Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
  StrategyType: RollingUpdate
  MinReadySeconds: 0
  RollingUpdateStrategy: 25% max unavailable, 25% max surge
  Pod Template:
  Labels: app.kubernetes.io/component=controller
  app.kubernetes.io/name=prometheus-operator
  app.kubernetes.io/part-of=kube-prometheus
  app.kubernetes.io/version=0.62.0
  Annotations: kubectl.kubernetes.io/default-container: prometheus-operator
  Service Account: prometheus-operator
  Containers:
  prometheus-operator:
  Image: quay.mirrors.ustc.edu.cn/prometheus-operator/prometheus-operator:v0.62.0
  Port: 8080/TCP
  Host Port: 0/TCP
  Args:
  --kubelet-service=kube-system/kubelet
  --prometheus-config-reloader=quay.mirrors.ustc.edu.cn/prometheus-operator/prometheus-config-reloader:v0.62.0
  Limits:
  cpu: 200m
  memory: 200Mi
  Requests:
  cpu: 100m
  memory: 100Mi
  Environment:
  Mounts:
  kube-rbac-proxy:
  Image: quay.mirrors.ustc.edu.cn/brancz/kube-rbac-proxy:v0.14.0
  Port: 8443/TCP
  Host Port: 0/TCP
  Args:
  --logtostderr
  --secure-listen-address=:8443
  --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
  --upstream=http://127.0.0.1:8080/
  Limits:
  cpu: 20m
  memory: 40Mi
  Requests:
  cpu: 10m
  memory: 20Mi
  Environment:
  Mounts:
  Volumes:
  Conditions:
  Type Status Reason

  ---

  Available True MinimumReplicasAvailable
  Progressing True NewReplicaSetAvailable
  OldReplicaSets:
  NewReplicaSet: prometheus-operator-6d49cb58c8 (1/1 replicas created)
  Events:

• Kubernetes version information:

  kubectl version
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:41:22Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.15-vke.13", GitCommit:"7781ced74658b436fe1a1b1ef90f246eb1d4bbc6", GitTreeState:"clean", BuildDate:"2023-08-16T03:58:51Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, tectonic-installer, etc.

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:

level=info ts=2024-06-05T09:37:53.146027961Z caller=operator.go:1162 component=prometheusoperator key=monitoring/prometheus-vci-gpu-l4 msg="sync prometheus"
level=info ts=2024-06-05T09:37:53.185092154Z caller=operator.go:1330 component=prometheusoperator key=monitoring/prometheus-vci-gpu-l4 msg="update prometheus status"
level=info ts=2024-06-05T09:38:02.608185405Z caller=operator.go:1330 component=prometheusoperator key=monitoring/prometheus-vci-gpu-a30 msg="update prometheus status"
level=info ts=2024-06-05T09:38:08.629134788Z caller=operator.go:1330 component=prometheusoperator key=monitoring/prometheus-vci-gpu-l4 msg="update prometheus status"

• Prometheus Logs:

ts=2024-06-05T10:57:28.858Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"aicloud\""
ts=2024-06-05T10:57:28.858Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"aicloud\""

Anything else we need to know?: