Can't connect to nodeexporter locally
copolycube opened this issue · comments
Hello,
There are 2 nodeexporter definitions
- (A) 1 in docker-compose.yml
- (B) 1 in docker-compose.exporter.yml
If I use (A), and defined the scraping at prometheus, it connects, but doesn't show the local filesystem data.
If I use (B), I am able to curl it from the command line, but it fails to be scraped (tried with the nodeexporter:9100 container name, with localhost:9100 and with the server name http://<url>:9100 ) from prometheus.yml but none work and it is still seen as beeing down.
I'm surely missing a obvious piece there, but I can't figure it, any insights would be great !
Thanks in advance for your help
You've assumed correctly that if using B you need to update the definition in the prometheus.yml to point to the host as opposed to the container name.
If the hostname doesn't work try utilising the IP address of the host, there could be a DNS resolution problem occuring.
So for example if your IP of the host was 10.10.10.1 you would have the following:
scrape_configs:
- job_name: 'nodeexporter'
scrape_interval: 5s
static_configs:
- targets: ['10.10.10.1:9100']
(note) : I also opened an issue on docker community forum about the issue (A), cf. https://forums.docker.com/t/mounted-volume-wrong-df-value-reported/112807/3
Have you checked the specific query required to report for your filesystem?
The README covers the section:
For storage and particularly Free Storage graph, you have to specify the fstype in grafana graph request. You can find it in grafana/dashboards/docker_host.json, at line 480 :
"expr": "sum(node_filesystem_free_bytes{fstype=\"btrfs\"})",
I work on BTRFS, so i need to change aufs to btrfs.
You can find right value for your system in Prometheus http://<host-ip>:9090 launching this request :
node_filesystem_free_bytes
All my servers are using Ubuntu 18.04, including the one where dockprom is running, including the faulty nodeexporter.
That's quite odd, I'm honestly not sure why you cannot see any stats specifically for those mount points.
I would suggest you lodge an issue at https://github.com/prometheus/node_exporter/issues, I had a quick look at some of the previous issues with similar symptoms and couldn't see anything obvious that would resolve the issue.
I resolved this issue by reinstalling docker. Using the "snap" install was failing, while using the official package didn't cause this issue.