原生k8s环境下部署报错Unauthorized
zhaoge77 opened this issue · comments
原生k8s环境（非tke环境）的安装步骤有吗？需要改哪些配置？我在原生k8s环境下简单部署了一下，prometheus服务反复重建中：
$ k2 get sts,pod -n kstone
NAME READY AGE
statefulset.apps/prometheus-kstone-prometheus-prometheus 1/1 61s
NAME READY STATUS RESTARTS AGE
pod/kstone-backup-operator-5c77c949d8-m7s7w 1/1 Running 0 35m
pod/kstone-dashboard-5595764fd5-sk4r6 1/1 Running 0 35m
pod/kstone-dashboard-api-7cfc658f9f-vkwvm 1/1 Running 0 35m
pod/kstone-etcd-controller-6c6d996fcf-jsdtl 1/1 Running 0 35m
pod/kstone-etcd-operator-f9b4bd7-xbdwr 1/1 Running 0 35m
pod/kstone-grafana-5d5985ffdf-vkr7l 2/2 Running 0 35m
pod/kstone-inspection-controller-65d6578554-zxjx5 1/1 Running 0 35m
pod/kstone-prometheus-operator-85dd8b5d99-m4snl 1/1 Running 0 35m
pod/prometheus-kstone-prometheus-prometheus-0 2/2 Terminating 0 61s
这是prometheus-operator的报错日志：
level=info ts=2021-12-09T10:54:17.573352638Z caller=main.go:295 msg="Starting Prometheus Operator" version="(version=0.49.0, branch=master, revision=2388bfa55)"
level=info ts=2021-12-09T10:54:17.573426783Z caller=main.go:296 build_context="(go=go1.16.5, user=paulfantom, date=20210706-17:43:37)"
level=info ts=2021-12-09T10:54:17.580278215Z caller=server.go:54 msg="enabling server side TLS"
level=info ts=2021-12-09T10:54:17.592804524Z caller=main.go:125 msg="Starting secure server on [::]:10250"
level=info ts=2021-12-09T10:54:17.594563066Z caller=operator.go:309 component=thanosoperator msg="connection established" cluster-version=v1.18.9
level=info ts=2021-12-09T10:54:17.594518358Z caller=operator.go:421 component=prometheusoperator msg="connection established" cluster-version=v1.18.9
level=info ts=2021-12-09T10:54:17.595023766Z caller=operator.go:430 component=prometheusoperator msg="CRD API endpoints ready"
level=info ts=2021-12-09T10:54:17.595047532Z caller=operator.go:449 component=alertmanageroperator msg="connection established" cluster-version=v1.18.9
level=info ts=2021-12-09T10:54:17.595109887Z caller=operator.go:458 component=alertmanageroperator msg="CRD API endpoints ready"
level=info ts=2021-12-09T10:54:17.595084193Z caller=operator.go:318 component=thanosoperator msg="CRD API endpoints ready"
level=info ts=2021-12-09T10:54:17.796953373Z caller=operator.go:261 component=thanosoperator msg="successfully synced all caches"
level=info ts=2021-12-09T10:54:18.396087837Z caller=operator.go:351 component=prometheusoperator msg="successfully synced all caches"
level=info ts=2021-12-09T10:54:18.397634362Z caller=operator.go:1221 component=prometheusoperator key=kstone/kstone-prometheus-prometheus msg="sync prometheus"
level=info ts=2021-12-09T10:54:18.497522759Z caller=operator.go:281 component=alertmanageroperator msg="successfully synced all caches"
level=info ts=2021-12-09T10:54:18.497765803Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:18.576327944Z caller=operator.go:1306 component=prometheusoperator key=kstone/kstone-prometheus-prometheus statefulset=prometheus-kstone-prometheus-prometheus shard=0 msg="recre
ating StatefulSet because the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"
level=info ts=2021-12-09T10:54:18.601315559Z caller=operator.go:1221 component=prometheusoperator key=cattle-monitoring-system/rancher-monitoring-prometheus msg="sync prometheus"
level=info ts=2021-12-09T10:54:18.672826348Z caller=operator.go:804 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="recreating AlertManager StatefulSet because
the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"
level=info ts=2021-12-09T10:54:18.689319741Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:18.791717511Z caller=operator.go:804 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="recreating AlertManager StatefulSet because
the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"
level=info ts=2021-12-09T10:54:18.804561025Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:18.916061885Z caller=operator.go:804 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="recreating AlertManager StatefulSet because
the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"
level=info ts=2021-12-09T10:54:19.279129482Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:19.383887819Z caller=operator.go:1306 component=prometheusoperator key=cattle-monitoring-system/rancher-monitoring-prometheus statefulset=prometheus-rancher-monitoring-prometheu
s shard=0 msg="recreating StatefulSet because the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbid
den"
level=info ts=2021-12-09T10:54:19.395885355Z caller=operator.go:804 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="recreating AlertManager StatefulSet because
the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"
level=info ts=2021-12-09T10:54:19.417586393Z caller=operator.go:1221 component=prometheusoperator key=kstone/kstone-prometheus-prometheus msg="sync prometheus"
level=info ts=2021-12-09T10:54:19.430864859Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:19.522897026Z caller=operator.go:804 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="recreating AlertManager StatefulSet because
the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"
level=info ts=2021-12-09T10:54:19.58892879Z caller=operator.go:1306 component=prometheusoperator key=kstone/kstone-prometheus-prometheus statefulset=prometheus-kstone-prometheus-prometheus shard=0 msg="recrea
ting StatefulSet because the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"
level=info ts=2021-12-09T10:54:19.604894838Z caller=operator.go:1221 component=prometheusoperator key=cattle-monitoring-system/rancher-monitoring-prometheus msg="sync prometheus"
level=info ts=2021-12-09T10:54:20.127681649Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:20.234899709Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:20.380352807Z caller=operator.go:1306 component=prometheusoperator key=cattle-monitoring-system/rancher-monitoring-prometheus statefulset=prometheus-rancher-monitoring-prometheu
s shard=0 msg="recreating StatefulSet because the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbid
den"
level=info ts=2021-12-09T10:54:20.397125163Z caller=operator.go:1221 component=prometheusoperator key=kstone/kstone-prometheus-prometheus msg="sync prometheus"
level=info ts=2021-12-09T10:54:20.529716104Z caller=operator.go:1306 component=prometheusoperator key=kstone/kstone-prometheus-prometheus statefulset=prometheus-kstone-prometheus-prometheus shard=0 msg="recre
ating StatefulSet because the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"
level=info ts=2021-12-09T10:54:20.532849411Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:20.597040563Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:20.915831566Z caller=operator.go:1221 component=prometheusoperator key=kstone/kstone-prometheus-prometheus msg="sync prometheus"
level=info ts=2021-12-09T10:54:21.051243142Z caller=operator.go:1221 component=prometheusoperator key=kstone/kstone-prometheus-prometheus msg="sync prometheus"
level=info ts=2021-12-09T10:54:21.113887188Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:21.214160287Z caller=operator.go:742 component=alertmanageroperator key=cattle-monitoring-system/rancher-monitoring-alertmanager msg="sync alertmanager"
level=info ts=2021-12-09T10:54:21.233518201Z caller=operator.go:1221 component=prometheusoperator key=kstone/kstone-prometheus-prometheus msg="sync prometheus"
level=error ts=2021-12-09T10:54:21.341324336Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="Sync "cattle-monitoring-system/rancher-monitoring-alertmanager" failed: synchronizing gover
ning service failed: updating service object failed: Operation cannot be fulfilled on services "alertmanager-operated": the object has been modified; please apply your changes to the latest version and try
again"
可能原因1：该集群已经通过rancher部署了一套kube-prometheus集群，其底层也是prometheus-operator，其使用crd名称相同但版本不一致，产生了冲突？
打开dashboard页面有报错，报错提示：Unauthorized
1 kstone使用的prometheus-operator为v0.49.0，与其他版本的确会有CRD配置冲突的可能，可以考虑将kube-prometheus-stack文件夹中相关组件的replica调整为0，并将你们自身的prometheus query替换文件charts/charts/grafana/templates/configmap.yaml中的http://{{ .Release.Name }}-prometheus-prometheus.{{ .Release.Namespace }}.svc.cluster.local:9090，这样即可使用你们现有的prometheus替换kstone默认的kube-prometheus-stack。
2 Unauthorized的问题可以先检查下文件charts/charts/dashboard-api/values.yaml中的kube.token是否配置正确。
