VMInsert: unsupported path requested
hansedong opened this issue · comments
After I deployed the VM Cluster using VictoriaMetrics Operator, the VMAgent instance cannot write data to the VMInsert instance. The following error logs can be seen from both VMAgent and VMInsert (this issue has been bothering me for 2 days):
VMInsert:
2024-03-15T03:08:11.510Z warn VictoriaMetrics/lib/httpserver/httpserver.go:365 remoteAddr: "10.131.81.101:39916"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
2024-03-15T03:08:14.124Z warn VictoriaMetrics/lib/httpserver/httpserver.go:365 remoteAddr: "10.131.81.100:43155"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
2024-03-15T03:08:18.124Z warn VictoriaMetrics/lib/httpserver/httpserver.go:365 remoteAddr: "10.131.81.100:43155"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
2024-03-15T03:08:27.510Z warn VictoriaMetrics/lib/httpserver/httpserver.go:365 remoteAddr: "10.131.81.101:39916"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
2024-03-15T03:08:28.124Z warn VictoriaMetrics/lib/httpserver/httpserver.go:365 remoteAddr: "10.131.81.100:43155"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
2024-03-15T03:08:30.510Z warn VictoriaMetrics/lib/httpserver/httpserver.go:365 remoteAddr: "10.131.81.101:39916"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
2024-03-15T03:08:33.125Z warn VictoriaMetrics/lib/httpserver/httpserver.go:365 remoteAddr: "10.131.81.100:43155"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
VMAgent:
[vmagent-vmagent-ha-2-0 vmagent] 2024-03-15T02:46:11.216Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:433 sending a block with size 220417 bytes to "1:secret-url:0:0" was rejected (skipping the block): status code 400; response body: remoteAddr: "10.131.81.101:15502"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
[vmagent-vmagent-ha-2-0 vmagent] 2024-03-15T02:46:11.511Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:433 sending a block with size 31437 bytes to "1:secret-url:0:0" was rejected (skipping the block): status code 400; response body: remoteAddr: "10.131.81.101:15502"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
[vmagent-vmagent-ha-4-0 vmagent] 2024-03-15T02:46:15.620Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:433 sending a block with size 47168 bytes to "1:secret-url:0:0" was rejected (skipping the block): status code 400; response body: remoteAddr: "10.131.81.100:59750"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
[vmagent-vmagent-ha-1-0 vmagent] 2024-03-15T02:46:18.125Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:433 sending a block with size 19413 bytes to "1:secret-url:0:0" was rejected (skipping the block): status code 400; response body: remoteAddr: "10.131.81.100:7019"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
[vmagent-vmagent-ha-0-0 vmagent] 2024-03-15T02:46:18.241Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:433 sending a block with size 29306 bytes to "1:secret-url:0:0" was rejected (skipping the block): status code 400; response body: remoteAddr: "10.131.81.100:60405"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
[vmagent-vmagent-ha-0-0 vmagent] 2024-03-15T02:46:19.241Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:433 sending a block with size 30165 bytes to "1:secret-url:0:0" was rejected (skipping the block): status code 400; response body: remoteAddr: "10.131.81.100:60405"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
[vmagent-vmagent-ha-3-0 vmagent] 2024-03-15T02:46:22.714Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:433 sending a block with size 19608 bytes to "1:secret-url:0:0" was rejected (skipping the block): status code 400; response body: remoteAddr: "10.131.81.101:7382"; requestURI: /insert/0:0/prometheus/api/v1/write; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
I may need to clarify a few points:
- I deployed VictoriaMetrics using the Operator.
- I have deployed VictoriaMetrics in cluster mode.
- I directly send data to the VMInsert instance, and the error is the same.
- I carefully read the URL format mentioned in https://docs.victoriametrics.com/cluster-victoriametrics/#url-format. From the URL format, it seems normal. I feel the issue lies with VMInsert, but I can't pinpoint where the problem is with VMInsert.
curl -v -d 'metric_name{foo="bar"} 123' -X POST http://10.131.120.215:8480/insert/0:0/prometheus/api/v1/write
remoteAddr: "10.131.81.103:44650"; requestURI: /insert/0:0/prometheus/api/v1/write?metric_name%7Bfoo=%22bar%22%7D+123; unsupported path requested: "/insert/0:0/prometheus/api/v1/write"
Here is my specific deployment configuration.：
VMCluster
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: vmcluster
namespace: monitor
spec:
replicationFactor: 2
retentionPeriod: 60d
vmstorage:
replicaCount: 5
vmBackup:
acceptEULA: false
concurrency: 5
destination: "s3://product/victoria-metrics-backup"
customS3Endpoint: https://microservices-1258295365.cos.ap-beijing.myqcloud.com
credentialsSecret:
name: tencent-s3-remote-storage-keys
key: credentials.json
extraArgs:
dedup.minScrapeInterval: 3s
image:
repository: hub.company.com/kubernetes/victoriametrics/vmstorage
tag: v1.96.0-cluster
pullPolicy: Always
storageDataPath: "/vm-data"
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: In
values:
- "vmstorage"
topologyKey: "kubernetes.io/hostname"
storage:
volumeClaimTemplate:
spec:
storageClassName: openebs-hostpath
resources:
requests:
storage: 250Gi
resources:
limits:
cpu: "20"
memory: 10240Mi
vmselect:
replicaCount: 3
image:
repository: hub.company.com/kubernetes/victoriametrics/vmselect
tag: v1.96.0-cluster
pullPolicy: Always
cacheMountPath: "/select-cache"
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: In
values:
- "vmselect"
topologyKey: "kubernetes.io/hostname"
storage:
volumeClaimTemplate:
spec:
storageClassName: openebs-hostpath
resources:
requests:
storage: 100Gi
resources:
limits:
cpu: "8"
memory: "4096Mi"
vminsert:
replicaCount: 5
image:
repository: hub.company.com/kubernetes/victoriametrics/vmselect
tag: v1.96.0-cluster
pullPolicy: Always
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: In
values:
- "vminsert"
topologyKey: "kubernetes.io/hostname"
resources:
limits:
cpu: "8"
memory: "4096Mi"
VMAgent
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
namespace: monitor
name: vmagent-ha
spec:
# ...
selectAllByDefault: true
vmAgentExternalLabelName: vmagent_ha
# Replication:
scrapeInterval: 30s
replicaCount: 1
# StatefulMode:
statefulMode: true
statefulStorage:
volumeClaimTemplate:
spec:
storageClassName: openebs-hostpath
resources:
requests:
storage: 50Gi
# Sharding
shardCount: 5
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
shard-num: '%SHARD_NUM%'
topologyKey: kubernetes.io/hostname
weight: 1
# Remote Write Configuration
remoteWrite:
# - url: "http://vminsert-vmcluster.monitor.svc.cluster.local:8480/insert/multitenant/prometheus/api/v1/write"
# - url: "http://vminsert-vmcluster.monitor.svc.cluster.local:8480/api/v1/write"
- url: "http://vminsert-vmcluster.monitor.svc.cluster.local:8480"
remoteWriteSettings:
useMultiTenantMode: true
label:
vm_account_id: "0"
vm_project_id: "0"
VMUser
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
name: vmuser
spec:
name: vmuser
username: admin
password: admin
targetRefs:
# vmui + vmselect
- crd:
kind: VMCluster/vmselect
name: vmcluster
namespace: monitor
target_path_suffix: "/select/0"
paths:
- "/vmui"
- "/vmui/.*"
- "/prometheus/api/v1/query"
- "/prometheus/api/v1/query_range"
- "/prometheus/api/v1/series"
- "/prometheus/api/v1/status/.*"
- "/prometheus/api/v1/label/"
- "/prometheus/api/v1/label/[^/]+/values"
- crd:
kind: VMCluster/vminsert
name: vmcluster
namespace: monitor
target_path_suffix: "/insert/0"
- crd:
kind: VMAgent
name: vmagent-ha
namespace: monitor
paths: ["/targets/api/v1","/targets","/metrics"]
VMAuth
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: victoria-metrics
namespace: monitor
spec:
selectAllByDefault: true
userNamespaceSelector: {}
userSelector: {}
ingress:
class_name: traefik-lb # <-- change this to your ingress-controller
host: vm-seashell.mfwdev.com
Please help to identify the possible reasons for this issue, thanks.
Hello!
Seems like you're using the wrong image for vminsert here.
vminsert:
replicaCount: 5
image:
repository: hub.company.com/kubernetes/victoriametrics/vmselect ##should be /vminsert
tag: v1.96.0-cluster
pullPolicy: Always
Hello! Seems like you're using the wrong image for vminsert here.
vminsert: replicaCount: 5 image: repository: hub.company.com/kubernetes/victoriametrics/vmselect ##should be /vminsert tag: v1.96.0-cluster pullPolicy: Always
@Haleygo Yes, what a waking sentence to wake up the dreamer, thank you so much