Prometheus memory usage for containers is always 0

[Tobias Zander]

I just set up the default system using docker-compose.
Unfortunately, in Grafana is see all metric except of the Container Memory Usage and Cached Container Memory Usage.

Unfortunately already when I search in Prometheus for memory, all values are 0.

I'm using a PI4, probably the issue is related to that. Did anyone have similar issues and has an idea how to fix it or what would be the next step to look into, when Prometheus has the wrong/no values?

[Amir Zarrinkafsh]

Is your cadvisor container running?
As far as I know the official cadvisor container doesn't support ARM so you would have had to modify the docker-compose.yml to utilise an image that does.

[Tobias Zander]

Is your cadvisor container running?
As far as I know the official cadvisor container doesn't support ARM so you would have had to modify the docker-compose.yml to utilise an image that does.

Indeed that's what I did, easily replaced it with: image: budry/cadvisor-arm

I don't know all these tools enough, is cadvisor actually the tool that gets the raw container memory data? If yes I would look deeper into that specific arm-image.

[Amir Zarrinkafsh]

Yes it is I would try run this query in Prometheus directly to see what shows up:
container_memory_usage_bytes{image!="",container_label_org_label_schema_group=""}

[Tobias Zander]

This is the result:

<!--StartFragment-->
ontainer_memory_usage_bytes{container_label_com_docker_compose_config_hash="2554b9322524180575582f3546abe6114846a48c2ba54071ec3731dde4cda855",&nbsp;container_label_com_docker_compose_container_number="1",&nbsp;container_label_com_docker_compose_oneoff="False",&nbsp;container_label_com_docker_compose_project="openhab",&nbsp;container_label_com_docker_compose_service="openhab",&nbsp;container_label_com_docker_compose_version="1.21.0",&nbsp;container_label_maintainer="openHAB &lt;[e-mail]&gt;",&nbsp;container_label_org_label_schema_build_date="2021-01-27T21:40:00Z",&nbsp;container_label_org_label_schema_description="An open source, technology agnostic home automation platform",&nbsp;container_label_org_label_schema_docker_dockerfile="/Dockerfile",&nbsp;container_label_org_label_schema_license="EPL-2.0",&nbsp;container_label_org_label_schema_name="openHAB",&nbsp;container_label_org_label_schema_url="https://www.openhab.com/",&nbsp;container_label_org_label_schema_vcs_ref="75d0fffe8526866c2830d83612fbd7ed64a866bb",&nbsp;container_label_org_label_schema_vcs_type="Git",&nbsp;container_label_org_label_schema_vcs_url="https://github.com/openhab/openhab-docker.git",&nbsp;container_label_org_label_schema_vendor="openHAB Foundation e.V.",&nbsp;container_label_org_label_schema_version="2.5.12",&nbsp;id="/docker/f1b366c9b8cef49fe90bcdeb3476045ae0acbddde557fd622bb1875a79ebcd94",&nbsp;image="openhab_openhab",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="openhab"} | 0
-- | --
container_memory_usage_bytes{container_label_com_docker_compose_config_hash="e2cdee5c548ccaece03d188ba0ac441d6ae9f38178f329cd361079a54d15dae5",&nbsp;container_label_com_docker_compose_container_number="1",&nbsp;container_label_com_docker_compose_oneoff="False",&nbsp;container_label_com_docker_compose_project="openhab3",&nbsp;container_label_com_docker_compose_service="openhab",&nbsp;container_label_com_docker_compose_version="1.21.0",&nbsp;container_label_maintainer="openHAB &lt;[e-mail]&gt;",&nbsp;container_label_org_label_schema_build_date="2021-04-21T12:56:20Z",&nbsp;container_label_org_label_schema_description="An open source, technology agnostic home automation platform",&nbsp;container_label_org_label_schema_docker_dockerfile="/Dockerfile",&nbsp;container_label_org_label_schema_license="EPL-2.0",&nbsp;container_label_org_label_schema_name="openHAB",&nbsp;container_label_org_label_schema_url="https://www.openhab.com/",&nbsp;container_label_org_label_schema_vcs_ref="d4bc2cfb570ee70d74369fccd168ccbd078a4c79",&nbsp;container_label_org_label_schema_vcs_type="Git",&nbsp;container_label_org_label_schema_vcs_url="https://github.com/openhab/openhab-docker.git",&nbsp;container_label_org_label_schema_vendor="openHAB Foundation e.V.",&nbsp;container_label_org_label_schema_version="3.0.2",&nbsp;id="/docker/63adaa6b9219ba98ae1e58774adaba79d25a0beba57e49135cc887c40c0e6f62",&nbsp;image="openhab3_openhab",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="openhab3"} | 0
container_memory_usage_bytes{container_label_restartcount="1",&nbsp;id="/docker/a0a96fa86ff26b06b022e281cde454882b93c43c5db822092648a2387882de7e",&nbsp;image="marthoc/deconz",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="deconz"} | 0
container_memory_usage_bytes{container_label_restartcount="9",&nbsp;id="/docker/cd860d38fad0d7bd999c36ae043e6a6bac2756fd1f689e5090412d7085a03f5a",&nbsp;image="pi/heos-api",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="heos-api"} | 0
container_memory_usage_bytes{id="/docker/550fa5b733dd182cbf17c273ff5f687c592bd158cd8393020377d38c41e8ed7c",&nbsp;image="pi/shelly-proxy",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="shelly-proxy"}

<!--EndFragment-->

[Amir Zarrinkafsh]

That's the same query that's being run so you should definitely be seeing results in Grafana.

[Chintan Pathak]

@airbone42 Not sure if you have the same problem, but the new default on Raspbian is to not report out the memory. The docker stats command does not work as expected too, as reported here: docker/for-linux#1112 and the fix there was to append the following to the \boot\cmdline.txt: cgroup_memory=1 cgroup_enable=memory. Once I did that and restarted my Pi 4, the memory section in Grafana displays a non-zero value. Still need to verify if it is correct.

[Amir Zarrinkafsh]

Other thing to make sure is that you're not running the SNAP version of Docker, this seems to also cause issues.

[Amir Zarrinkafsh]

Did you manage to resolve this @airbone42?

[Amir Zarrinkafsh]

Closing due to inactivity if you're still having this issue please let me know and I'll reopen the issue.