vm-datasource not works, but prometheus data source works
ahfuzhang opened this issue · comments
At fisrt, I create a config map like this:
# datasource.yaml
apiVersion: 1
# List of data sources to insert/update depending on what's
# available in the database.
datasources:
- name: Prometheus-realtime-cluster
type: prometheus
access: direct
#orgId: 1
url: http://10.43.113.112:8481/select/0/prometheus/
#url: http://realtime-cluster-vm-select-service:8481/select/0/prometheus/
isDefault: false
version: 1
editable: true
- name: Prometheus-self-monitor-cluster
type: prometheus
access: direct
#orgId: 1
url: http://10.43.12.253:8481/self-monitor-cluster-select/select/0/prometheus/
#url: http://self-monitor-cluster-vm-select-service:8481/self-monitor-cluster-select/select/0/prometheus/
isDefault: false
version: 1
editable: true
# <string, required> Name of the VictoriaMetrics datasource
# displayed in Grafana panels and queries.
- name: VictoriaMetrics-realtime-cluster
# <string, required> Sets the data source type.
type: victoriametrics-datasource
# <string, required> Sets the access mode, either
# proxy or direct (Server or Browser in the UI).
# Some data sources are incompatible with any setting
# but proxy (Server).
access: direct
# <string> Sets default URL of the single node version of VictoriaMetrics
#url: http://realtime-cluster-vm-select-service:8481/select/0/prometheus/
url: http://10.43.113.112:8481/select/0/prometheus/
# <string> Sets the pre-selected datasource for new panels.
# You can set only one default data source per organization.
isDefault: false
editable: true
- name: VictoriaMetrics-self-monitor-cluster
# <string, required> Sets the data source type.
type: victoriametrics-datasource
# <string, required> Sets the access mode, either
# proxy or direct (Server or Browser in the UI).
# Some data sources are incompatible with any setting
# but proxy (Server).
access: direct
# <string> Sets default URL of the single node version of VictoriaMetrics
#url: http://self-monitor-cluster-vm-select-service:8481/self-monitor-cluster-select/select/0/prometheus/
url: http://10.43.12.253:8481/self-monitor-cluster-select/select/0/prometheus/
# <string> Sets the pre-selected datasource for new panels.
# You can set only one default data source per organization.
isDefault: false
editable: trueWhen I open grafana:
vm-datasource can not connect. But prometheus datasource works.
And I create a new vm-datasource with same address, it works again.
So I think there might be some problem with yaml configed vm-data-source.
Hi @ahfuzhang ! Can you try to test it with this PR
VictoriaMetrics/VictoriaMetrics#5363
I don't understand.
I use k8s env, not docker compose.
I don't understand. I use k8s env, not docker compose.
Hi! Can you share how you run your k8s? Is there vmauth present?
my deploy code is here: https://github.com/ahfuzhang/deploy_VictoriaMetrics_cluster/blob/main/terraform/realtime_cluster/grafana.tf
Can you share kubernetes manifests for the grafana which used for deploy (deployment, configmap)?
k8s deployment yaml:
Name: realtime-cluster-grafana
Namespace: test-devops
CreationTimestamp: Tue, 02 Jan 2024 09:23:24 +0800
Labels: kubernetes_deployment_name=realtime-cluster-grafana
Annotations: deployment.kubernetes.io/revision: 1
Selector: kubernetes_deployment_name=realtime-cluster-grafana
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: kubernetes_deployment_name=realtime-cluster-grafana
Init Containers:
realtime-cluster-grafana-init:
Image: alpine:3.18.4
Port: <none>
Host Port: <none>
Command:
/bin/sh
Args:
-x
/grafana/config/init.sh
Environment: <none>
Mounts:
/grafana/config/ from grafana-config-volume (rw)
/vm-data/ from realtime-cluster-pvc (rw)
Containers:
realtime-cluster-grafana:
Image: grafana/grafana:10.2.2
Port: 3000/TCP
Host Port: 0/TCP
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Environment:
CONTAINER_NAME: (v1:metadata.name)
CONTAINER_IP: (v1:status.podIP)
GOMAXPROCS: 1
GF_PATHS_DATA: /vm-data/grafana/data/
GF_PATHS_PLUGINS: /vm-data/grafana/plugins/
GF_PATHS_PROVISIONING: /vm-data/grafana/provisioning/
GF_PATHS_CONFIG: /vm-data/grafana/config/grafana.ini
Mounts:
/vm-data/ from realtime-cluster-pvc (rw)
Volumes:
realtime-cluster-pvc:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: realtime-cluster-pvc
ReadOnly: false
grafana-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: realtime-cluster-grafana-config
Optional: false
Conditions:
Type Status Reason
---- ------ ------
Progressing True NewReplicaSetAvailable
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: realtime-cluster-grafana-7976c8b455 (1/1 replicas created)
Events: <none>
config map yaml:
Name: realtime-cluster-grafana-config
Namespace: test-devops
Labels: <none>
Annotations: <none>
Data
====
init.sh:
----
if [ ! -d "/vm-data/grafana/plugins/victoriametrics-datasource/" ]; then
wget https://github.com/VictoriaMetrics/grafana-datasource/releases/download/v0.5.0/victoriametrics-datasource-v0.5.0.tar.gz
mkdir -p /vm-data/grafana/plugins/
tar -zxf victoriametrics-datasource-v0.5.0.tar.gz -C /vm-data/grafana/plugins/
fi
mkdir -p /vm-data/grafana/provisioning/datasources/ /vm-data/grafana/config/ /vm-data/grafana/data/ /vm-data/grafana/plugins/
cp /grafana/config/datasource.yaml /vm-data/grafana/provisioning/datasources/
cp /grafana/config/grafana.ini /vm-data/grafana/config/
#echo "1" > /vm-data/grafana/data/test.txt
chmod -R 777 /vm-data/grafana/
#sleep 10
datasource.yaml:
----
apiVersion: 1
# List of data sources to insert/update depending on what's
# available in the database.
datasources:
- name: Prometheus-realtime-cluster
type: prometheus
access: direct
#orgId: 1
url: http://10.43.113.112:8481/select/0/prometheus/
#url: http://realtime-cluster-vm-select-service:8481/select/0/prometheus/
isDefault: false
version: 1
editable: true
- name: Prometheus-self-monitor-cluster
type: prometheus
access: direct
#orgId: 1
url: http://10.43.12.253:8481/self-monitor-cluster-select/select/0/prometheus/
#url: http://self-monitor-cluster-vm-select-service:8481/self-monitor-cluster-select/select/0/prometheus/
isDefault: false
version: 1
editable: true
# <string, required> Name of the VictoriaMetrics datasource
# displayed in Grafana panels and queries.
- name: VictoriaMetrics-realtime-cluster
# <string, required> Sets the data source type.
type: victoriametrics-datasource
# <string, required> Sets the access mode, either
# proxy or direct (Server or Browser in the UI).
# Some data sources are incompatible with any setting
# but proxy (Server).
access: direct
# <string> Sets default URL of the single node version of VictoriaMetrics
#url: http://realtime-cluster-vm-select-service:8481/select/0/prometheus/
url: http://10.43.113.112:8481/select/0/prometheus/
# <string> Sets the pre-selected datasource for new panels.
# You can set only one default data source per organization.
isDefault: false
editable: true
- name: VictoriaMetrics-self-monitor-cluster
# <string, required> Sets the data source type.
type: victoriametrics-datasource
# <string, required> Sets the access mode, either
# proxy or direct (Server or Browser in the UI).
# Some data sources are incompatible with any setting
# but proxy (Server).
access: direct
# <string> Sets default URL of the single node version of VictoriaMetrics
#url: http://self-monitor-cluster-vm-select-service:8481/self-monitor-cluster-select/select/0/prometheus/
url: http://10.43.12.253:8481/self-monitor-cluster-select/select/0/prometheus/
# <string> Sets the pre-selected datasource for new panels.
# You can set only one default data source per organization.
isDefault: false
editable: true
grafana.ini:
----
[plugins[]
allow_loading_unsigned_plugins = victoriametrics-datasource
BinaryData
====
Events: <none>
k8s deployment yaml:
Name: realtime-cluster-grafana Namespace: test-devops CreationTimestamp: Tue, 02 Jan 2024 09:23:24 +0800 Labels: kubernetes_deployment_name=realtime-cluster-grafana Annotations: deployment.kubernetes.io/revision: 1 Selector: kubernetes_deployment_name=realtime-cluster-grafana Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: kubernetes_deployment_name=realtime-cluster-grafana Init Containers: realtime-cluster-grafana-init: Image: alpine:3.18.4 Port: <none> Host Port: <none> Command: /bin/sh Args: -x /grafana/config/init.sh Environment: <none> Mounts: /grafana/config/ from grafana-config-volume (rw) /vm-data/ from realtime-cluster-pvc (rw) Containers: realtime-cluster-grafana: Image: grafana/grafana:10.2.2 Port: 3000/TCP Host Port: 0/TCP Limits: cpu: 1 memory: 1Gi Requests: cpu: 100m memory: 128Mi Environment: CONTAINER_NAME: (v1:metadata.name) CONTAINER_IP: (v1:status.podIP) GOMAXPROCS: 1 GF_PATHS_DATA: /vm-data/grafana/data/ GF_PATHS_PLUGINS: /vm-data/grafana/plugins/ GF_PATHS_PROVISIONING: /vm-data/grafana/provisioning/ GF_PATHS_CONFIG: /vm-data/grafana/config/grafana.ini Mounts: /vm-data/ from realtime-cluster-pvc (rw) Volumes: realtime-cluster-pvc: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: realtime-cluster-pvc ReadOnly: false grafana-config-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: realtime-cluster-grafana-config Optional: false Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available True MinimumReplicasAvailable OldReplicaSets: <none> NewReplicaSet: realtime-cluster-grafana-7976c8b455 (1/1 replicas created) Events: <none>config map yaml:
Name: realtime-cluster-grafana-config Namespace: test-devops Labels: <none> Annotations: <none> Data ==== init.sh: ---- if [ ! -d "/vm-data/grafana/plugins/victoriametrics-datasource/" ]; then wget https://github.com/VictoriaMetrics/grafana-datasource/releases/download/v0.5.0/victoriametrics-datasource-v0.5.0.tar.gz mkdir -p /vm-data/grafana/plugins/ tar -zxf victoriametrics-datasource-v0.5.0.tar.gz -C /vm-data/grafana/plugins/ fi mkdir -p /vm-data/grafana/provisioning/datasources/ /vm-data/grafana/config/ /vm-data/grafana/data/ /vm-data/grafana/plugins/ cp /grafana/config/datasource.yaml /vm-data/grafana/provisioning/datasources/ cp /grafana/config/grafana.ini /vm-data/grafana/config/ #echo "1" > /vm-data/grafana/data/test.txt chmod -R 777 /vm-data/grafana/ #sleep 10 datasource.yaml: ---- apiVersion: 1 # List of data sources to insert/update depending on what's # available in the database. datasources: - name: Prometheus-realtime-cluster type: prometheus access: direct #orgId: 1 url: http://10.43.113.112:8481/select/0/prometheus/ #url: http://realtime-cluster-vm-select-service:8481/select/0/prometheus/ isDefault: false version: 1 editable: true - name: Prometheus-self-monitor-cluster type: prometheus access: direct #orgId: 1 url: http://10.43.12.253:8481/self-monitor-cluster-select/select/0/prometheus/ #url: http://self-monitor-cluster-vm-select-service:8481/self-monitor-cluster-select/select/0/prometheus/ isDefault: false version: 1 editable: true # <string, required> Name of the VictoriaMetrics datasource # displayed in Grafana panels and queries. - name: VictoriaMetrics-realtime-cluster # <string, required> Sets the data source type. type: victoriametrics-datasource # <string, required> Sets the access mode, either # proxy or direct (Server or Browser in the UI). # Some data sources are incompatible with any setting # but proxy (Server). access: direct # <string> Sets default URL of the single node version of VictoriaMetrics #url: http://realtime-cluster-vm-select-service:8481/select/0/prometheus/ url: http://10.43.113.112:8481/select/0/prometheus/ # <string> Sets the pre-selected datasource for new panels. # You can set only one default data source per organization. isDefault: false editable: true - name: VictoriaMetrics-self-monitor-cluster # <string, required> Sets the data source type. type: victoriametrics-datasource # <string, required> Sets the access mode, either # proxy or direct (Server or Browser in the UI). # Some data sources are incompatible with any setting # but proxy (Server). access: direct # <string> Sets default URL of the single node version of VictoriaMetrics #url: http://self-monitor-cluster-vm-select-service:8481/self-monitor-cluster-select/select/0/prometheus/ url: http://10.43.12.253:8481/self-monitor-cluster-select/select/0/prometheus/ # <string> Sets the pre-selected datasource for new panels. # You can set only one default data source per organization. isDefault: false editable: true grafana.ini: ---- [plugins[] allow_loading_unsigned_plugins = victoriametrics-datasource BinaryData ==== Events: <none>
Can you please share files of output with command
kubectl get deployment -o yaml > deployment.yaml
kubectl get configmap -o yaml > configmap.yaml
kubectl get pvc -o yaml > pvc.yaml
Thank you!
deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2024-01-02T01:23:24Z"
generation: 1
labels:
kubernetes_deployment_name: realtime-cluster-grafana
name: realtime-cluster-grafana
namespace: test-devops
resourceVersion: "64280263"
uid: 29fcf537-993e-442d-b4a1-62debd6dcef8
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
kubernetes_deployment_name: realtime-cluster-grafana
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
kubernetes_deployment_name: realtime-cluster-grafana
spec:
automountServiceAccountToken: true
containers:
- env:
- name: CONTAINER_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: CONTAINER_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: GOMAXPROCS
value: "1"
- name: GF_PATHS_DATA
value: /vm-data/grafana/data/
- name: GF_PATHS_PLUGINS
value: /vm-data/grafana/plugins/
- name: GF_PATHS_PROVISIONING
value: /vm-data/grafana/provisioning/
- name: GF_PATHS_CONFIG
value: /vm-data/grafana/config/grafana.ini
image: grafana/grafana:10.2.2
imagePullPolicy: IfNotPresent
name: realtime-cluster-grafana
ports:
- containerPort: 3000
protocol: TCP
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 100m
memory: 128Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /vm-data/
mountPropagation: None
name: realtime-cluster-pvc
dnsPolicy: ClusterFirst
enableServiceLinks: true
initContainers:
- args:
- -x
- /grafana/config/init.sh
command:
- /bin/sh
image: alpine:3.18.4
imagePullPolicy: IfNotPresent
name: realtime-cluster-grafana-init
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /vm-data/
mountPropagation: None
name: realtime-cluster-pvc
- mountPath: /grafana/config/
mountPropagation: None
name: grafana-config-volume
workingDir: /
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
shareProcessNamespace: false
terminationGracePeriodSeconds: 30
volumes:
- name: realtime-cluster-pvc
persistentVolumeClaim:
claimName: realtime-cluster-pvc
- configMap:
defaultMode: 420
name: realtime-cluster-grafana-config
optional: false
name: grafana-config-volume
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2024-01-02T01:23:24Z"
lastUpdateTime: "2024-01-02T01:23:30Z"
message: ReplicaSet "realtime-cluster-grafana-7976c8b455" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2024-01-16T09:59:59Z"
lastUpdateTime: "2024-01-16T09:59:59Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
configmap.yaml:
apiVersion: v1
data:
datasource.yaml: |+
apiVersion: 1
# List of data sources to insert/update depending on what's
# available in the database.
datasources:
- name: Prometheus-realtime-cluster
type: prometheus
access: direct
#orgId: 1
url: http://10.43.113.112:8481/select/0/prometheus/
#url: http://realtime-cluster-vm-select-service:8481/select/0/prometheus/
isDefault: false
version: 1
editable: true
- name: Prometheus-self-monitor-cluster
type: prometheus
access: direct
#orgId: 1
url: http://10.43.12.253:8481/self-monitor-cluster-select/select/0/prometheus/
#url: http://self-monitor-cluster-vm-select-service:8481/self-monitor-cluster-select/select/0/prometheus/
isDefault: false
version: 1
editable: true
# <string, required> Name of the VictoriaMetrics datasource
# displayed in Grafana panels and queries.
- name: VictoriaMetrics-realtime-cluster
# <string, required> Sets the data source type.
type: victoriametrics-datasource
# <string, required> Sets the access mode, either
# proxy or direct (Server or Browser in the UI).
# Some data sources are incompatible with any setting
# but proxy (Server).
access: direct
# <string> Sets default URL of the single node version of VictoriaMetrics
#url: http://realtime-cluster-vm-select-service:8481/select/0/prometheus/
url: http://10.43.113.112:8481/select/0/prometheus/
# <string> Sets the pre-selected datasource for new panels.
# You can set only one default data source per organization.
isDefault: false
editable: true
- name: VictoriaMetrics-self-monitor-cluster
# <string, required> Sets the data source type.
type: victoriametrics-datasource
# <string, required> Sets the access mode, either
# proxy or direct (Server or Browser in the UI).
# Some data sources are incompatible with any setting
# but proxy (Server).
access: direct
# <string> Sets default URL of the single node version of VictoriaMetrics
#url: http://self-monitor-cluster-vm-select-service:8481/self-monitor-cluster-select/select/0/prometheus/
url: http://10.43.12.253:8481/self-monitor-cluster-select/select/0/prometheus/
# <string> Sets the pre-selected datasource for new panels.
# You can set only one default data source per organization.
isDefault: false
editable: true
grafana.ini: |
[plugins]
allow_loading_unsigned_plugins = victoriametrics-datasource
init.sh: |+
if [ ! -d "/vm-data/grafana/plugins/victoriametrics-datasource/" ]; then
wget https://github.com/VictoriaMetrics/grafana-datasource/releases/download/v0.5.0/victoriametrics-datasource-v0.5.0.tar.gz
mkdir -p /vm-data/grafana/plugins/
tar -zxf victoriametrics-datasource-v0.5.0.tar.gz -C /vm-data/grafana/plugins/
fi
mkdir -p /vm-data/grafana/provisioning/datasources/ /vm-data/grafana/config/ /vm-data/grafana/data/ /vm-data/grafana/plugins/
cp /grafana/config/datasource.yaml /vm-data/grafana/provisioning/datasources/
cp /grafana/config/grafana.ini /vm-data/grafana/config/
#echo "1" > /vm-data/grafana/data/test.txt
chmod -R 777 /vm-data/grafana/
#sleep 10
immutable: false
kind: ConfigMap
metadata:
creationTimestamp: "2023-12-28T08:13:51Z"
name: realtime-cluster-grafana-config
namespace: test-devops
resourceVersion: "59465742"
uid: a819d1ee-6234-4abe-8d7b-cf49fb3f90cf
pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: org.democratic-csi.nfs
volume.kubernetes.io/storage-provisioner: org.democratic-csi.nfs
creationTimestamp: "2023-12-28T08:12:17Z"
finalizers:
- kubernetes.io/pvc-protection
name: realtime-cluster-pvc
namespace: test-devops
resourceVersion: "59108745"
uid: ebec8d1a-669a-4cd5-ba45-7e324f493e13
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Ti
storageClassName: freenas-nfs-csi
volumeMode: Filesystem
volumeName: pvc-ebec8d1a-669a-4cd5-ba45-7e324f493e13
status:
accessModes:
- ReadWriteMany
capacity:
storage: 2Ti
phase: Bound
Hi @ahfuzhang ! Thank you for your configuration files. The main problem is that you are using access: direct in the datasource config. But in the documentation we are using access: proxy.
If you are using access: direct you should choose
Access Server (default) because all requests should go via the plugin backend. But if you use the Browser access it will sends the requests directly to the URL http://10.43.12.253:8481/self-monitor-cluster-select/select/0/prometheus/ from the browser.
Here is the working example
and the response from the server
Thank you so much.
I will try it tomorrow.
@ahfuzhang did the proposed configuration example work for you?
fixed. Thank you so much.