bug: podman_images_* series include metrics for non-existing images
wally007 opened this issue · comments
Hello,
I've created a dashboard to monitor amount of images on the podman host and noticed that the metrics dont line up with the host.
There is a lot of <none> images that do not show up in cli when running podman images
Where are these coming from ? (I've tried to run podman system prune but that made no difference as there were no dangling images on the system)
root@podman1:~>podman system prune
WARNING! This command removes:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
root@podman1:~>
podman images -> 14 items
root@podman1:~>podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
refinst-docker-dev-local.artifactory.io/grafana-agent1 latest eabd977236ca 2 days ago 457 MB
refinst-docker-dev-local.artifactory.io/grafana_agent_latest latest ea71004b1aaf 2 days ago 457 MB
refinst-docker-dev-local.artifactory.io/vmware-exporter latest 5e883f1bedfc 5 days ago 937 MB
refinst-docker-dev-local.artifactory.io/veeam-em-exporter latest e09053d9feab 2 weeks ago 883 MB
refinst-docker-dev-local.artifactory.io/hpilo-exporter latest c81f066fce6d 2 weeks ago 882 MB
refinst-docker-dev-local.artifactory.io/python39 latest c801b1ebfa4b 2 weeks ago 880 MB
refinst-docker-dev-local.artifactory.io/web_discard latest 516038f49add 2 weeks ago 404 MB
refinst-docker-dev-local.artifactory.io/netapp-harvest-exporter latest 50b147ce0c4b 2 weeks ago 316 MB
refinst-docker-dev-local.artifactory.io/web_dashboard latest 24dd88e557b0 5 weeks ago 319 MB
refinst-docker-dev-local.artifactory.io/web_tq latest 9d2260fd7289 6 weeks ago 413 MB
refinst-docker-dev-local.artifactory.io/web_gapcheck latest 7a64a7f08f58 6 weeks ago 413 MB
refinst-docker-dev-local.artifactory.io/web_rds latest 3a07e63ee435 7 weeks ago 414 MB
refinst-docker-dev-local.artifactory.io/web_rdf latest 8a5e4434c4b0 8 weeks ago 416 MB
localhost/podman-pause 4.6.1-1692961697 4ce25834cda0 4 weeks ago 810 kB
root@podman1:~>
podman_image_info -> 38 items
root@podman1:~>curl -s http://127.0.0.1:9882/metrics | grep -i "podman_image_info"
# HELP podman_image_info Image information.
# TYPE podman_image_info gauge
podman_image_info{digest="sha256:0a6ed5c7ac19ac85ea3408247032e36590c97fcf9b92202131dde940994cbc9e",id="2905c0f74e2b",parent_id="ec7bf4cae86f",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:0aca56789602469679f03cd9a74abcc5cfb23e505f6a9dade9dac872fc7bf9d3",id="4ce25834cda0",parent_id="",repository="localhost/podman-pause",tag="4.6.1-1692961697"} 1
podman_image_info{digest="sha256:0c033e7a4a25b4bb3857aae3a9985eb6cc51674fa6841c5d6ab6e520b744037c",id="6f7fa9a134b1",parent_id="897420c14ff9",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:113446045e5beb631735d4b78606865b417d43aec3f5d0e10cce7138da10fe20",id="7a64a7f08f58",parent_id="",repository="refinst-docker-dev-local.artifactory.io/web_gapcheck",tag="latest"} 1
podman_image_info{digest="sha256:1673345dc3f19269ae8c87c337e6d0bbfc42ab3596abbfc2635fc5a33d8a1987",id="9a2807da1e7b",parent_id="ef5d4631a596",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:33442ed707926c7a5f95cee14ef921c79e01f6f0b6faa8c52821b4f89b9c1040",id="6d03d1c6deee",parent_id="ed57561df995",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:33c34a64e8b27039fbc546580395ef509f8943b55a2d4382c9efc75654bc2e1c",id="ed57561df995",parent_id="a207e1233108",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:3b76c4fa1298ce14181b89e09f336db94241698ef03571bda82618fd5ca33794",id="24dd88e557b0",parent_id="",repository="refinst-docker-dev-local.artifactory.io/web_dashboard",tag="latest"} 1
podman_image_info{digest="sha256:3fd696005267537e7901cea0e49d9498cea6889fc0ee17e5ea90be418d6f09b8",id="ec7bf4cae86f",parent_id="9a2807da1e7b",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:488e9d0d8eed2abacbc8e9aea9355e87e65f6a0c51acb387cf8994249202d741",id="a207e1233108",parent_id="",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:48bd1c5e531374e5763509cc21d0973ace66b43b0278a4f32645eb56264ee22c",id="8a5e4434c4b0",parent_id="",repository="refinst-docker-dev-local.artifactory.io/web_rdf",tag="latest"} 1
podman_image_info{digest="sha256:4fbe71d302dfcdd052541087701895dd0897cc05fa5f94d112efdad50a138021",id="50b147ce0c4b",parent_id="6f29191cc431",repository="refinst-docker-dev-local.artifactory.io/netapp-harvest-exporter",tag="latest"} 1
podman_image_info{digest="sha256:656bc3649d5854472637fdf47a46c56234b8bb11ad6ff2296349a61b360ce0a3",id="ea71004b1aaf",parent_id="",repository="refinst-docker-dev-local.artifactory.io/grafana_agent_latest",tag="latest"} 1
podman_image_info{digest="sha256:680500d57432c0d0efb312f7b4419b74e38db15ce54a85677beec73107d17432",id="516038f49add",parent_id="06e50700f88a",repository="refinst-docker-dev-local.artifactory.io/web_discard",tag="latest"} 1
podman_image_info{digest="sha256:7016d3a807af13f15cf6d06e06e7ca0c6457eabe795ad48ed8a074a7e438142f",id="87131e84e92f",parent_id="32c82d478acf",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:72fbd7c8d40fba25881926da0320c928bc6d96fa2e4ecf82c55c2e98948049e5",id="75547269d5dd",parent_id="c801b1ebfa4b",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:7435f71b84f40e8e11a79a00ef6ddf5b3a87fac4831c53b827d404e77eccf9f7",id="3a07e63ee435",parent_id="",repository="refinst-docker-dev-local.artifactory.io/web_rds",tag="latest"} 1
podman_image_info{digest="sha256:84fb87a5fdf985050f71d9bdea1478ee38216efc5ce575a543ec33fa54a82e48",id="a1534bf3bdb4",parent_id="2905c0f74e2b",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:8ea031fa8ae8966d5c98da5befd753adde29e6d456db4159302a8829da9a172f",id="32c82d478acf",parent_id="b4dcaebd7ad6",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:8ffa2a6e92204235f7c352c84101c1cba46cbbce6441c99d2abbf8cbd289c031",id="ef5d4631a596",parent_id="c801b1ebfa4b",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:92902126a72240481acc03525b8ead478a5a73750c29bf866d46f2e79a6b83af",id="c79f51b7c547",parent_id="5a7233b9a946",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:95bfa65e3fc9b2537d59f701edd5dc02db0fe4238e2ff2b68ff7bfe4889a516c",id="513fe78ae290",parent_id="ea71004b1aaf",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:986d1e26e6aed7df4ede6b236cc7fb6f3ac6f83bae752cfd13a0315f89fb08c1",id="c801b1ebfa4b",parent_id="",repository="refinst-docker-dev-local.artifactory.io/python39",tag="latest"} 1
podman_image_info{digest="sha256:996de721756a9656eda92bf31adadeedfec4246ecb4747accf301cba97e43806",id="d87698835e56",parent_id="c79f51b7c547",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:a08b74bfd4dee20fede7459800d26de0e65bf9ea6ad328083e600e3b1731ff9b",id="6f29191cc431",parent_id="87131e84e92f",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:a5f22f326e8c3376014ccbd0a3e81edda656f9e603e0367b25a40470e3e1e4d2",id="5a7233b9a946",parent_id="5d2e027dce4f",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:aad54800aa00ff2837730b51be85d00da21edb40dcd0f5e41c5a6d4334fffa65",id="55cb705da813",parent_id="c801b1ebfa4b",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:aed453013c0f102e953bacebb05dfa3d1355947f572665813834d439b5d67dc3",id="9d2260fd7289",parent_id="",repository="refinst-docker-dev-local.artifactory.io/web_tq",tag="latest"} 1
podman_image_info{digest="sha256:bce8cc01a18804d07163078d78febfe89144c3a6501900ad04e2a79473ac0b1a",id="897420c14ff9",parent_id="75547269d5dd",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:c364e2faffae61d860b23651baad3d73586c58a779387305cfe6e2af1c609f1d",id="06e50700f88a",parent_id="6d03d1c6deee",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:d0192458dbbbf72f0fe9f59cb0c98fc46cc3573918e866a0331d3177429a448f",id="c81f066fce6d",parent_id="bf4fc3d5cec8",repository="refinst-docker-dev-local.artifactory.io/hpilo-exporter",tag="latest"} 1
podman_image_info{digest="sha256:d601b23ecef1ceafefcfe2f122858aebb7d2873586d47fc9afd0fe5af67c80fd",id="bf4fc3d5cec8",parent_id="a1534bf3bdb4",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:e03b62dae1dc0163b6b9c90e1defe4d3f4a1808306ca133d8bdb8ee5aad58c9f",id="b4dcaebd7ad6",parent_id="4399df180324",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:e05ca2be96483cfce3ebc425ce478d24a4da427b065d744d1e9ff5a038861d46",id="5d2e027dce4f",parent_id="55cb705da813",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:e7f4b3705e8575ddd73bb326cb756686b1e874ba2257bb01178ba9d1d29a69ce",id="e09053d9feab",parent_id="6f7fa9a134b1",repository="refinst-docker-dev-local.artifactory.io/veeam-em-exporter",tag="latest"} 1
podman_image_info{digest="sha256:f80f755d353c7367cabf625b1b7723bfe8f850dbab3f72c8ee3b07d037ad8c53",id="4399df180324",parent_id="",repository="<none>",tag="<none>"} 1
podman_image_info{digest="sha256:f81453457f17aff712282c95d5c6ddb6407f487ae94f5c5c4b281291b6f2e055",id="5e883f1bedfc",parent_id="d87698835e56",repository="refinst-docker-dev-local.artifactory.io/vmware-exporter",tag="latest"} 1
podman_image_info{digest="sha256:fff48b6cab26df80408134707a6d8fcf10227c5dc6cf3c59755201f383985b4e",id="eabd977236ca",parent_id="513fe78ae290",repository="refinst-docker-dev-local.artifactory.io/grafana-agent1",tag="latest"} 1
root@podman1:~>
Hi @wally007
can you also send me podman image ls --all command output.
Regards
Hi @wally007 can you also send me
podman image ls --allcommand output.Regards
Yes, these show the "missing" images
root@podman1:~>podman images --all
REPOSITORY TAG IMAGE ID CREATED SIZE
refinst-docker-dev-local.artifactory.io/grafana-agent1 latest eabd977236ca 3 days ago 457 MB
<none> <none> 513fe78ae290 3 days ago 457 MB
refinst-docker-dev-local.artifactory.io/grafana_agent_latest latest ea71004b1aaf 3 days ago 457 MB
refinst-docker-dev-local.artifactory.io/vmware-exporter latest 5e883f1bedfc 5 days ago 937 MB
<none> <none> d87698835e56 5 days ago 937 MB
<none> <none> c79f51b7c547 5 days ago 937 MB
<none> <none> 5a7233b9a946 5 days ago 880 MB
<none> <none> 5d2e027dce4f 2 weeks ago 880 MB
<none> <none> 55cb705da813 2 weeks ago 880 MB
refinst-docker-dev-local.artifactory.io/veeam-em-exporter latest e09053d9feab 2 weeks ago 883 MB
<none> <none> 6f7fa9a134b1 2 weeks ago 883 MB
<none> <none> 897420c14ff9 2 weeks ago 883 MB
<none> <none> 75547269d5dd 2 weeks ago 880 MB
refinst-docker-dev-local.artifactory.io/hpilo-exporter latest c81f066fce6d 2 weeks ago 882 MB
<none> <none> bf4fc3d5cec8 2 weeks ago 882 MB
<none> <none> a1534bf3bdb4 2 weeks ago 882 MB
<none> <none> 2905c0f74e2b 2 weeks ago 880 MB
<none> <none> ec7bf4cae86f 2 weeks ago 880 MB
<none> <none> 9a2807da1e7b 2 weeks ago 880 MB
<none> <none> ef5d4631a596 2 weeks ago 880 MB
refinst-docker-dev-local.artifactory.io/python39 latest c801b1ebfa4b 2 weeks ago 880 MB
refinst-docker-dev-local.artifactory.io/web_discard latest 516038f49add 2 weeks ago 404 MB
<none> <none> 06e50700f88a 2 weeks ago 404 MB
<none> <none> 6d03d1c6deee 2 weeks ago 404 MB
<none> <none> ed57561df995 2 weeks ago 404 MB
<none> <none> a207e1233108 2 weeks ago 404 MB
refinst-docker-dev-local.artifactory.io/netapp-harvest-exporter latest 50b147ce0c4b 2 weeks ago 316 MB
<none> <none> 6f29191cc431 3 weeks ago 316 MB
<none> <none> 87131e84e92f 3 weeks ago 316 MB
<none> <none> 32c82d478acf 3 weeks ago 316 MB
<none> <none> b4dcaebd7ad6 3 weeks ago 316 MB
<none> <none> 4399df180324 3 weeks ago 316 MB
localhost/podman-pause 4.6.1-1692961697 4ce25834cda0 4 weeks ago 810 kB
refinst-docker-dev-local.artifactory.io/web_dashboard latest 24dd88e557b0 6 weeks ago 319 MB
refinst-docker-dev-local.artifactory.io/web_tq latest 9d2260fd7289 6 weeks ago 413 MB
refinst-docker-dev-local.artifactory.io/web_gapcheck latest 7a64a7f08f58 6 weeks ago 413 MB
refinst-docker-dev-local.artifactory.io/web_rds latest 3a07e63ee435 7 weeks ago 414 MB
refinst-docker-dev-local.artifactory.io/web_rdf latest 8a5e4434c4b0 8 weeks ago 416 MB
root@podman1:~>
What are these images ?
How come these are not pruned with podman system prune ?
We prune images on schedule in our dev env. and do not want any dangling images - yet it seems there are still some left. Thanks to this exporter, I know now better :-)
Those are intermediate layers, you can try podman image prune -a or podman system prune -a in order to remove them as well.
Hi @wally007
Since this is not a bug, I am going to close it.
Thanks for trying and using the exporter :)