HTTP Error Bad Gateway
erichegit opened this issue · comments
I followed 1:1 the Readme - all Containers are up and running.
But I receive "HTTP Error Bad Gateway" in Grafana when I look at my default Datasource.
Browsing to IP:9090 shows me
502 Bad Gateway
How can I fix this?
Grafana http://:3000 = Working
Prometheus http://:9090 = BAD Gateway
Prometheus-Pushgateway http://:9091 = Working
AlertManager http://:9093 = BAD Gateway
Had this issue as well, I believe it has something to do with the default Docker IP subnet.
Fix: when I did a docker inspect prometheus | grep IPAddress on the Prometheus container to get its IP, and set that in Grafana's Data source config for Prometheus, it worked.
e.g. docker logs grafana showed that it was trying to query "127.0.0.1:9090". The IP that worked was "172.24.0.1:9090".
Note that this can change when you redeploy.
I'm not sure what circumstances are required on a server for this to work out of the box.
The data source that Grafana provisions for Prometheus shouldn't reference 127.0.0.1.
It should be referencing Prometheus by its DNS name.
Then this is a DNS configuration issue, at least for me.
nslookup is supposed to work in the container, right? It's not working here, with nameserver 127.0.0.11. It could be that whatever method Grafana uses to resolve prometheus also fails. However ping, wget, arp do resolve prometheus from the Grafana container.
My host is a company server so I can't modify /etc/resolv.conf to troubleshoot. It does include ("search") some company domains by default, which might cause local lookups to fail.
According to https://docs.docker.com/config/containers/container-networking/:
containers that use a custom network use Docker’s embedded DNS server, which forwards external DNS lookups to the DNS servers configured on the host
Is there an easy way to avoid putting the host's search domains in the containers?
nslookup fails for me too, however I do not experience the same issues. Grafana seems to still be able to resolve prometheus to it's respective Docker network IP:
/usr/share/grafana $ ping prometheus
PING prometheus (172.19.0.8): 56 data bytes
ping: permission denied (are you root?)
/usr/share/grafana $ nslookup prometheus
Server: 127.0.0.11
Address: 127.0.0.11:53
** server can't find prometheus.tokyo-3.vpn: NXDOMAIN
** server can't find prometheus.tokyo-3: NXDOMAIN
** server can't find prometheus.guest: NXDOMAIN
** server can't find prometheus.tokyo-3.wlan: NXDOMAIN
** server can't find prometheus.tokyo-3.vpn: NXDOMAIN
** server can't find prometheus.cygnus: NXDOMAIN
** server can't find prometheus.tokyo-3: NXDOMAIN
** server can't find prometheus.tokyo-3.wlan: NXDOMAIN
** server can't find prometheus.guest: NXDOMAIN
** server can't find prometheus.cygnus: NXDOMAIN
** server can't find prometheus.goomba: NXDOMAIN
** server can't find prometheus.goomba: NXDOMAIN
** server can't find prometheus.eva01: NXDOMAIN
** server can't find prometheus.eva01: NXDOMAIN
So I'm not sure if this is a red herring but I don't think that's the actual issue that you may be experiencing.
I've also tried running this up in a brand new VM with no modifications without any issues.
Have you tried a fresh clone?
Closing as I was not able to reproduce this. If you continue to have issues please request for this to be reopened and provide steps to reproduce the problem.