Failure on Ansible Task Ensure internet-monitoring environment is running - Grafana Failure
KunalGehlot opened this issue · comments
I'm facing a failure on the specified task, I'm running the latest Raspberry Pi OS and I noticed on Grafana Docker Hub that they don't have an image for linux/arm/v8.
I believe forcing docker to install v7 should work but I'm not sure if that's correct and where I can do that within the ansible Task.
Here's the detailed traceback:
TASK [Ensure internet-monitoring environment is running.] ********************************************************************************************************************************************************
task path: /home/zack/internet-pi/tasks/internet-monitoring.yml:47
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: zack
<127.0.0.1> EXEC /bin/sh -c 'echo ~zack && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/zack/.ansible/tmp `"&& mkdir "` echo /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369 `" && echo ansible-tmp-1696255007.7593706-5892-48847898244369="` echo /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369 `" ) && sleep 0'
Using module file /home/zack/.local/lib/python3.9/site-packages/ansible_collections/community/docker/plugins/modules/docker_compose.py
<127.0.0.1> PUT /home/zack/.ansible/tmp/ansible-local-5349i998sglj/tmp3ugo1lxe TO /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/AnsiballZ_docker_compose.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/ /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/AnsiballZ_docker_compose.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/AnsiballZ_docker_compose.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
File "/tmp/ansible_community.docker.docker_compose_payload_ol_b5jul/ansible_community.docker.docker_compose_payload.zip/ansible_collections/community/docker/plugins/modules/docker_compose.py", line 861, in cmd_up
File "/usr/local/lib/python3.9/dist-packages/compose/project.py", line 664, in up
svc.ensure_image_exists(do_build=do_build, silent=silent, cli=cli)
File "/usr/local/lib/python3.9/dist-packages/compose/service.py", line 358, in ensure_image_exists
self.pull(silent=silent)
File "/usr/local/lib/python3.9/dist-packages/compose/service.py", line 1251, in pull
return progress_stream.get_digest_from_pull(event_stream)
File "/usr/local/lib/python3.9/dist-packages/compose/progress_stream.py", line 99, in get_digest_from_pull
for event in events:
File "/usr/local/lib/python3.9/dist-packages/compose/service.py", line 1223, in _do_pull
yield from stream_output(output, sys.stdout)
File "/usr/local/lib/python3.9/dist-packages/compose/progress_stream.py", line 27, in stream_output
print_output_event(event, stream, is_terminal)
File "/usr/local/lib/python3.9/dist-packages/compose/progress_stream.py", line 59, in print_output_event
raise StreamOutputError(event['errorDetail']['message'])
fatal: [127.0.0.1]: FAILED! => {
"changed": false,
"errors": [],
"invocation": {
"module_args": {
"api_version": "auto",
"build": false,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"debug": false,
"definition": null,
"dependencies": true,
"docker_host": "unix://var/run/docker.sock",
"env_file": null,
"files": null,
"hostname_check": false,
"nocache": false,
"profiles": null,
"project_name": null,
"project_src": "/home/zack/internet-monitoring/",
"pull": false,
"recreate": "smart",
"remove_images": null,
"remove_orphans": false,
"remove_volumes": false,
"restarted": false,
"scale": null,
"services": null,
"ssl_version": null,
"state": "present",
"stopped": false,
"timeout": null,
"tls": false,
"tls_hostname": null,
"use_ssh_client": false,
"validate_certs": false
}
},
"module_stderr": "",
"module_stdout": "latest: Pulling from grafana/grafana\n",
"msg": "Error starting project no matching manifest for linux/arm/v8 in the manifest list entries"
}
Looks like I was on the wrong track, I did force docker to use v7 by adding a line in /internet-pi/templates/docker-compose.yml.j2 - platform: linux/arm/v7.
But now it says Error starting project no matching manifest for linux/arm/v7 in the manifest list entries.
P.S: Turns out grafana docker hub does not have the ARM build for the latest tag.
Kunal, I'm getting the same thing.
TASK [Ensure internet-monitoring environment is running.] **************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "errors": [], "module_stderr": "", "module_stdout": "latest: Pulling from miguelndecarvalho/speedtest-exporter\nDigest: sha256:f1064d49124c7fc45faabb87c6c876a2fd04e92b3dc14d4b871301217ba30fed\nStatus: Downloaded newer image for miguelndecarvalho/speedtest-exporter:latest\nlatest: Pulling from prom/prometheus\nDigest: sha256:c5dd3503828713c4949ae1bccd1d8d69f382c33d441954674a6b78ebe69c3331\nStatus: Downloaded newer image for prom/prometheus:latest\nlatest: Pulling from grafana/grafana\n", "msg": "Error starting project no matching manifest for linux/arm/v8 in the manifest list entries"}
It's driving me nuts.
Here is a little on the issue
`pi@internetpi1:~ $ cd internet-monitoring/
pi@internetpi1:~/internet-monitoring $ docker-compose up -d
Pulling grafana (grafana/grafana:latest)...
ERROR: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)`
Also this
pi@internetpi1:~/internet-monitoring $ docker-compose up -d
Pulling grafana (grafana/grafana:latest)...
latest: Pulling from grafana/grafana
ERROR: no matching manifest for linux/arm/v8 in the manifest list entries
Alright I've learned something. This seems to be an issue with the docker-compose.yml file pulling the latest version of Grafana and Grafana not having the linux/arm/v8 in their manifest. I tried changing latest to a few version back of Grafana and that didnt work either, so I went like a year back and tried 9.4.2. That fixed it! I don't actually know where the cut-off version is, but I'll test more later. For now here's how you can get it running.
cd ~/internet-pi/templates
nano docker-compose.yml.j2
Change grafana/grafana:latest to grafana/grafana:9.2.4
feel free to play around and find the most recent release that still works. You can find the releases here: https://github.com/grafana/grafana/releases
Well, that worked to get just blank grafana running using the ~/internet-monitoring/ folder and docker-compose up -d, but it is not working inside of the ansible-playbook... so idk and I've got to go to bed unfortunately
Wait, it actually works i think... but you have to do some weird stuff. change the docker-compose.yml.j2 file like I said above. Then run the playbook. This will copy the yml.j2 file to ~/internet-monitoring/docker-compose.yml. The play book will get stuck at the Internet Monitoring Check again. Ctrl+C out of it. Then cd ~/internet-monitoring/. Then docker-compose up -d. Then back to cd ~/internet-pi/. Run ansible-playbook main.yml. Everything should succeed this time.
Finally, go to your pi's IP Address :3030. Then click the magnifying glass for search dashboards. Click the General folder then the Internet Connection dashboard. Star it for easier access. Then I think you're good. What a PITA! Glad I got it, I'll check back tomorrow for connection logs.
Thanks a lot @benlbaum !
Tried 10.1.2 and it worked! listed here as the latest release with armv7 support: https://hub.docker.com/r/grafana/grafana/tags
Wait, it actually works i think... but you have to do some weird stuff. change the docker-compose.yml.j2 file like I said above. Then run the playbook. This will copy the yml.j2 file to ~/internet-monitoring/docker-compose.yml. The play book will get stuck at the Internet Monitoring Check again. Ctrl+C out of it. Then cd ~/internet-monitoring/. Then docker-compose up -d. Then back to cd ~/internet-pi/. Run ansible-playbook main.yml. Everything should succeed this time.
I'm close, but I'm getting the following error on docker-compose up -d
network internet-monitoring-back-tier was found but has incorrect label com.docker.compose.network set to "internet-monitoring-back-tier"
Update:
Looks like Docker is not reading the custom network names, change the compose file to
networks:
internet-monitoring-front-tier:
internet-monitoring-back-tier:
and updated the network reference accordingly.
But now it says
Error response from daemon: driver failed programming external connectivity on endpoint internet-monitoring-nginx-proxy-1 (b69305b73c74f0cd0cc0f1e7c7cb41f4d0ba4a5b1d461e61d7257ba48c6ca805): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use
Sidenote: When I run docker-compose up -d in ~/internet-monitoring/ I get the error
parsing /home/zack/internet-monitoring/docker-compose.yml: yaml: unmarshal errors:
line 70: mapping key "depends_on" already defined at line 61
and in-fact, there is two mentions of depends_on under grafana, just combined it into one.
Update: This happens when you set domain_name_enable to true in config.yml
Going for a hard-reset of the whole system...
Update: It worked! After all the patches discussed above, and additionally changing become: true at pi-hole.yml at Line 24 did the trick.
Now the only problem remains is that I cannot log into Grafana :/
Realised, the devs might want to patch this, re-opening.
this worked! i was ready to give up. I'm in!!!
nano docker-compose.yml.j2
Confirming this also worked for me :)