keda-operator-metrics-apiserver /metrics endpoint connection refused
trampfox opened this issue · comments
- Our Prometheus instance is getting a connection refused error while trying to scrape the metrics from the
/metrics
endpoint after a clean installation (using the default values) via Helm Chart version 2.11.1. - A Prometheus operator ServiceMonitor is enabled for the Keda Metrics Server
- The
keda-operator-metrics-apiserver
is deployed and is running without any problems, except for the/metrics
endpoint where we get a connection refused error
Expected behavior
Keda Metrics Server service monitor targets are all up and Prometheus is able to scrape the metrics from keda-operator-metrics-apiserver
instances.
Actual Behavior
Prometheus is not able to scrape the metrics from the /metrics
endpoint of due to the following error
Get "http://10.24.1.112:8080/metrics": dial tcp 10.24.1.112:8080: connect: connection refused
We have also tried to get the metrics by enabling a port-forwarding to the 8080 port of the keda-operator-metrics-apiserver
and we got the same connection refused error
$ kubectl port-forward keda-operator-metrics-apiserver-57586d6c44-vtm5s 8080:8080
and when we call the /metrics
endpoint we see the following error
E0720 11:05:39.957032 60120 portforward.go:400] an error occurred forwarding 8080 -> 8080: error forwarding port 8080 to pod c7354feb5414f5e29681e7d45354b5b0424053ab52eddead40b18908b9f0773e, uid : failed to execute portforward in network namespace "/var/run/netns/cni-c62a7094-5ffa-c294-3107-6fac86c49f95": failed to connect to localhost:8080 inside namespace "c7354feb5414f5e29681e7d45354b5b0424053ab52eddead40b18908b9f0773e", IPv4: dial tcp4 127.0.0.1:8080: connect: connection refused IPv6 dial tcp6 [::1]:8080: connect: connection refused
Steps to Reproduce the Problem
- Install Keda via helm chart version 2.11.1
Specifications
- KEDA Version: 2.11.1
- Kubernetes Version: 1.24
- Scaler(s): N/A
I'm able to connect when I port-forward, but I get an empty reply from the endpoint.
$ kubectl -n keda port-forward keda-operator-metrics-apiserver-5bb88966d5-6v2mk metrics
Forwarding from 127.0.0.1:9022 -> 9022
Forwarding from [::1]:9022 -> 9022
Handling connection for 9022
Handling connection for 9022
Handling connection for 9022
when I try to use curl, I get an empty reply.
$ curl -v http://localhost:9022/metrics
* Trying 127.0.0.1:9022...
* Connected to localhost (127.0.0.1) port 9022 (#0)
> GET /metrics HTTP/1.1
> Host: localhost:9022
> User-Agent: curl/7.88.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server
I also have curl: (52) Empty reply from server
Check this release: https://github.com/kedacore/keda/releases/tag/v2.11.2
It works now. Many thanks!
You can probably close this issue since it was fixed in v2.11.2 @trampfox