Error when deleting images from Google Artifact Registry: MANIFEST UNKNOWN
maxgoepp opened this issue · comments
TL;DR
Images in Google Artifact Registry can't be deleted due to a MANIFEST UNKNOWN error.
Expected behavior
When using gcr-cleaner on a Container Registry Repository it deletes either all untagged images or all images specified with the -tag-filter-any or -tag-filter-all flags. So it works as expected.
Expected behaviour for Artifact Registry Repositories containing the same images is that these images would be deleted, too.
Observed behavior
gcr-cleaner fails to delete images due to a MANIFEST UNKNOWN error:
docker run -v "${HOME}/.config/gcloud:/.config/gcloud" -it europe-docker.pkg.dev/gcr-cleaner/gcr-cleaner/gcr-cleaner-cli -repo <location>-docker.pkg.dev/<project>/<repository>/staging/ -grace 48h -tag-filter-any "staging-*"
Deleting refs older than 2022-10-26T07:56:06Z on 1 repo(s)...
<location>-docker.pkg.dev/<project>/<repository>/staging/
✗ no refs were deleted
8 errors occurred:
* failed to delete <location>-docker.pkg.dev/<project>/<repository>/staging/@sha256:14963418838edc03fa6ae67935b318605f1df1dcda25cc0602fdbd44197c1d7c: GET https://<location>-docker.pkg.dev/v2/<project>/<repository>/staging/manifests/sha256:14963418838edc03fa6ae67935b318605f1df1dcda25cc0602fdbd44197c1d7c: MANIFEST_UNKNOWN: Manifest has media type "application/vnd.docker.distribution.manifest.v2+json" but client accepts []
The images I try to delete in Artifact Registry have the same mediaType as the images in Container Registry:
application/vnd.docker.distribution.manifest.v2+json
Debug log output
docker run -v "${HOME}/.config/gcloud:/.config/gcloud" -e GCRCLEANER_LOG=debug -it europe-docker.pkg.dev/gcr-cleaner/gcr-cleaner/gcr-cleaner-cli -repo <location>-docker.pkg.dev/<project>/<repository>/staging/ -grace 48h -tag-filter-any "staging-*"
Deleting refs older than 2022-10-26T08:10:19Z on 1 repo(s)...
<location>-docker.pkg.dev/<project>/<repository>/staging/
{"message":"computed repo","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:19Z"}
{"keep":0,"manifests":[null,null,null,null,null,null,null,null,null,{"created":"2022-10-27T05:18:25Z","digest":"sha256:795f73080e891bf7d25a4a2f727011f94b25701250b0b476a8fc2a0bef1af128","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","tags":["base"],"uploaded":"2022-10-27T05:19:51Z"},{"created":"2022-10-25T05:35:44Z","digest":"sha256:14963418838edc03fa6ae67935b318605f1df1dcda25cc0602fdbd44197c1d7c","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","tags":[],"uploaded":"2022-10-25T05:36:41Z"},{"created":"2022-10-24T11:27:34Z","digest":"sha256:ba7a1671a51bf5a3244c2566e6748e59f0024bab60b68a856ce908630d25aebe","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","tags":["staging-106"],"uploaded":"2022-10-24T11:27:56Z"},{"created":"2022-10-24T09:52:00Z","digest":"sha256:55875ad052c05999de938ac2e28595caff4cfce8941b9a73ee58b97d99a94f2b","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","tags":["staging-105"],"uploaded":"2022-10-24T09:52:23Z"},{"created":"2022-10-17T05:35:31Z","digest":"sha256:3fa13fc27b40e21848f025abd5141b2efb67f1978b9b0bee1591517b08bc7008","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","tags":[],"uploaded":"2022-10-17T05:36:28Z"},{"created":"2022-10-14T13:18:49Z","digest":"sha256:19ceb7e67b633cd0011317eb963e3ffe4a42c27bd98ca688aa9e6076b3fc6185","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","tags":["staging-104"],"uploaded":"2022-10-14T13:19:46Z"},{"created":"2022-10-14T09:35:44Z","digest":"sha256:a6f4d20c0b86fc128a15674f0952de5d82f5f7e4b7f4d7582355090d576b9ba1","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","tags":["staging-103"],"uploaded":"2022-10-14T09:36:04Z"},{"created":"2022-10-10T13:31:40Z","digest":"sha256:7d1efe974d937d6227b42efef9f9d8efdc49b19710df69fca24357831210795b","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","tags":["staging-3"],"uploaded":"2022-10-10T13:32:01Z"},{"created":"2022-10-10T12:59:23Z","digest":"sha256:bb6c1c530180e042b3d467983ebd69f698c8c3d10f4b7e97db7f738ddc8dc978","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","tags":[],"uploaded":"2022-10-10T13:00:30Z"}],"message":"computed all manifests","severity":"DEBUG","time":"2022-10-28T08:10:19Z"}
{"created":"2022-10-27T05:18:25Z","digest":"sha256:795f73080e891bf7d25a4a2f727011f94b25701250b0b476a8fc2a0bef1af128","message":"processing manifest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tags":["base"],"time":"2022-10-28T08:10:19Z","uploaded":"2022-10-27T05:19:51Z"}
{"created":"2022-10-27T05:18:25Z","delta":"21h9m32.497653291s","digest":"sha256:795f73080e891bf7d25a4a2f727011f94b25701250b0b476a8fc2a0bef1af128","message":"should not delete","reason":"too new","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","since":"2022-10-26T08:10:19Z","time":"2022-10-28T08:10:19Z","uploaded":"2022-10-27T05:19:51Z"}
{"created":"2022-10-25T05:35:44Z","digest":"sha256:14963418838edc03fa6ae67935b318605f1df1dcda25cc0602fdbd44197c1d7c","message":"processing manifest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tags":[],"time":"2022-10-28T08:10:19Z","uploaded":"2022-10-25T05:36:41Z"}
{"digest":"sha256:14963418838edc03fa6ae67935b318605f1df1dcda25cc0602fdbd44197c1d7c","message":"should delete","reason":"no tags","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:19Z"}
{"created":"2022-10-24T11:27:34Z","digest":"sha256:ba7a1671a51bf5a3244c2566e6748e59f0024bab60b68a856ce908630d25aebe","message":"processing manifest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tags":["staging-106"],"time":"2022-10-28T08:10:19Z","uploaded":"2022-10-24T11:27:56Z"}
{"digest":"sha256:ba7a1671a51bf5a3244c2566e6748e59f0024bab60b68a856ce908630d25aebe","message":"should delete","reason":"matches tag filter","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag_filter":"any(staging-*)","tags":["staging-106"],"time":"2022-10-28T08:10:19Z"}
{"digest":"sha256:ba7a1671a51bf5a3244c2566e6748e59f0024bab60b68a856ce908630d25aebe","message":"deleting tag","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag":"staging-106","time":"2022-10-28T08:10:19Z"}
{"digest":"sha256:14963418838edc03fa6ae67935b318605f1df1dcda25cc0602fdbd44197c1d7c","message":"deleting digest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:19Z"}
{"created":"2022-10-24T09:52:00Z","digest":"sha256:55875ad052c05999de938ac2e28595caff4cfce8941b9a73ee58b97d99a94f2b","message":"processing manifest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tags":["staging-105"],"time":"2022-10-28T08:10:20Z","uploaded":"2022-10-24T09:52:23Z"}
{"digest":"sha256:55875ad052c05999de938ac2e28595caff4cfce8941b9a73ee58b97d99a94f2b","message":"should delete","reason":"matches tag filter","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag_filter":"any(staging-*)","tags":["staging-105"],"time":"2022-10-28T08:10:20Z"}
{"digest":"sha256:55875ad052c05999de938ac2e28595caff4cfce8941b9a73ee58b97d99a94f2b","message":"deleting tag","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag":"staging-105","time":"2022-10-28T08:10:20Z"}
{"digest":"sha256:ba7a1671a51bf5a3244c2566e6748e59f0024bab60b68a856ce908630d25aebe","message":"deleting digest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:20Z"}
{"created":"2022-10-17T05:35:31Z","digest":"sha256:3fa13fc27b40e21848f025abd5141b2efb67f1978b9b0bee1591517b08bc7008","message":"processing manifest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tags":[],"time":"2022-10-28T08:10:20Z","uploaded":"2022-10-17T05:36:28Z"}
{"digest":"sha256:3fa13fc27b40e21848f025abd5141b2efb67f1978b9b0bee1591517b08bc7008","message":"should delete","reason":"no tags","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:20Z"}
{"created":"2022-10-14T13:18:49Z","digest":"sha256:19ceb7e67b633cd0011317eb963e3ffe4a42c27bd98ca688aa9e6076b3fc6185","message":"processing manifest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tags":["staging-104"],"time":"2022-10-28T08:10:20Z","uploaded":"2022-10-14T13:19:46Z"}
{"digest":"sha256:19ceb7e67b633cd0011317eb963e3ffe4a42c27bd98ca688aa9e6076b3fc6185","message":"should delete","reason":"matches tag filter","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag_filter":"any(staging-*)","tags":["staging-104"],"time":"2022-10-28T08:10:20Z"}
{"digest":"sha256:19ceb7e67b633cd0011317eb963e3ffe4a42c27bd98ca688aa9e6076b3fc6185","message":"deleting tag","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag":"staging-104","time":"2022-10-28T08:10:20Z"}
{"digest":"sha256:55875ad052c05999de938ac2e28595caff4cfce8941b9a73ee58b97d99a94f2b","message":"deleting digest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:20Z"}
{"digest":"sha256:3fa13fc27b40e21848f025abd5141b2efb67f1978b9b0bee1591517b08bc7008","message":"deleting digest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:20Z"}
{"created":"2022-10-14T09:35:44Z","digest":"sha256:a6f4d20c0b86fc128a15674f0952de5d82f5f7e4b7f4d7582355090d576b9ba1","message":"processing manifest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tags":["staging-103"],"time":"2022-10-28T08:10:20Z","uploaded":"2022-10-14T09:36:04Z"}
{"digest":"sha256:19ceb7e67b633cd0011317eb963e3ffe4a42c27bd98ca688aa9e6076b3fc6185","message":"deleting digest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:20Z"}
{"digest":"sha256:a6f4d20c0b86fc128a15674f0952de5d82f5f7e4b7f4d7582355090d576b9ba1","message":"should delete","reason":"matches tag filter","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag_filter":"any(staging-*)","tags":["staging-103"],"time":"2022-10-28T08:10:20Z"}
{"digest":"sha256:a6f4d20c0b86fc128a15674f0952de5d82f5f7e4b7f4d7582355090d576b9ba1","message":"deleting tag","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag":"staging-103","time":"2022-10-28T08:10:20Z"}
{"created":"2022-10-10T13:31:40Z","digest":"sha256:7d1efe974d937d6227b42efef9f9d8efdc49b19710df69fca24357831210795b","message":"processing manifest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tags":["staging-3"],"time":"2022-10-28T08:10:21Z","uploaded":"2022-10-10T13:32:01Z"}
{"digest":"sha256:7d1efe974d937d6227b42efef9f9d8efdc49b19710df69fca24357831210795b","message":"should delete","reason":"matches tag filter","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag_filter":"any(staging-*)","tags":["staging-3"],"time":"2022-10-28T08:10:21Z"}
{"digest":"sha256:7d1efe974d937d6227b42efef9f9d8efdc49b19710df69fca24357831210795b","message":"deleting tag","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tag":"staging-3","time":"2022-10-28T08:10:21Z"}
{"digest":"sha256:a6f4d20c0b86fc128a15674f0952de5d82f5f7e4b7f4d7582355090d576b9ba1","message":"deleting digest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:21Z"}
{"created":"2022-10-10T12:59:23Z","digest":"sha256:bb6c1c530180e042b3d467983ebd69f698c8c3d10f4b7e97db7f738ddc8dc978","message":"processing manifest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","tags":[],"time":"2022-10-28T08:10:21Z","uploaded":"2022-10-10T13:00:30Z"}
{"digest":"sha256:bb6c1c530180e042b3d467983ebd69f698c8c3d10f4b7e97db7f738ddc8dc978","message":"should delete","reason":"no tags","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:21Z"}
{"digest":"sha256:bb6c1c530180e042b3d467983ebd69f698c8c3d10f4b7e97db7f738ddc8dc978","message":"deleting digest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:21Z"}
{"digest":"sha256:7d1efe974d937d6227b42efef9f9d8efdc49b19710df69fca24357831210795b","message":"deleting digest","repo":"<location>-docker.pkg.dev/<project>/<repository>/staging/","severity":"DEBUG","time":"2022-10-28T08:10:21Z"}
✗ no refs were deleted
8 errors occurred:
* failed to delete <location>-docker.pkg.dev/<project>/<repository>/staging/@sha256:14963418838edc03fa6ae67935b318605f1df1dcda25cc0602fdbd44197c1d7c: GET https://<location>-docker.pkg.dev/v2/<project>/<repository>/staging/manifests/sha256:14963418838edc03fa6ae67935b318605f1df1dcda25cc0602fdbd44197c1d7c: MANIFEST_UNKNOWN: Manifest has media type "application/vnd.docker.distribution.manifest.v2+json" but client accepts []
* failed to delete <location>-docker.pkg.dev/<project>/<repository>/staging/@sha256:19ceb7e67b633cd0011317eb963e3ffe4a42c27bd98ca688aa9e6076b3fc6185: GET https://<location>-docker.pkg.dev/v2/<project>/<repository>/staging/manifests/sha256:19ceb7e67b633cd0011317eb963e3ffe4a42c27bd98ca688aa9e6076b3fc6185: MANIFEST_UNKNOWN: Manifest has media type "application/vnd.docker.distribution.manifest.v2+json" but client accepts []
* failed to delete <location>-docker.pkg.dev/<project>/<repository>/staging/@sha256:3fa13fc27b40e21848f025abd5141b2efb67f1978b9b0bee1591517b08bc7008: GET https://<location>-docker.pkg.dev/v2/<project>/<repository>/staging/manifests/sha256:3fa13fc27b40e21848f025abd5141b2efb67f1978b9b0bee1591517b08bc7008: MANIFEST_UNKNOWN: Manifest has media type "application/vnd.docker.distribution.manifest.v2+json" but client accepts []
* failed to delete <location>-docker.pkg.dev/<project>/<repository>/staging/@sha256:55875ad052c05999de938ac2e28595caff4cfce8941b9a73ee58b97d99a94f2b: GET https://<location>-docker.pkg.dev/v2/<project>/<repository>/staging/manifests/sha256:55875ad052c05999de938ac2e28595caff4cfce8941b9a73ee58b97d99a94f2b: MANIFEST_UNKNOWN: Manifest has media type "application/vnd.docker.distribution.manifest.v2+json" but client accepts []
* failed to delete <location>-docker.pkg.dev/<project>/<repository>/staging/@sha256:7d1efe974d937d6227b42efef9f9d8efdc49b19710df69fca24357831210795b: GET https://<location>-docker.pkg.dev/v2/<project>/<repository>/staging/manifests/sha256:7d1efe974d937d6227b42efef9f9d8efdc49b19710df69fca24357831210795b: MANIFEST_UNKNOWN: Manifest has media type "application/vnd.docker.distribution.manifest.v2+json" but client accepts []
* failed to delete <location>-docker.pkg.dev/<project>/<repository>/staging/@sha256:a6f4d20c0b86fc128a15674f0952de5d82f5f7e4b7f4d7582355090d576b9ba1: GET https://<location>-docker.pkg.dev/v2/<project>/<repository>/staging/manifests/sha256:a6f4d20c0b86fc128a15674f0952de5d82f5f7e4b7f4d7582355090d576b9ba1: MANIFEST_UNKNOWN: Manifest has media type "application/vnd.docker.distribution.manifest.v2+json" but client accepts []
* failed to delete <location>-docker.pkg.dev/<project>/<repository>/staging/@sha256:ba7a1671a51bf5a3244c2566e6748e59f0024bab60b68a856ce908630d25aebe: GET https://<location>-docker.pkg.dev/v2/<project>/<repository>/staging/manifests/sha256:ba7a1671a51bf5a3244c2566e6748e59f0024bab60b68a856ce908630d25aebe: MANIFEST_UNKNOWN: Manifest has media type "application/vnd.docker.distribution.manifest.v2+json" but client accepts []
* failed to delete <location>-docker.pkg.dev/<project>/<repository>/staging/@sha256:bb6c1c530180e042b3d467983ebd69f698c8c3d10f4b7e97db7f738ddc8dc978: GET https://<location>-docker.pkg.dev/v2/<project>/<repository>/staging/manifests/sha256:bb6c1c530180e042b3d467983ebd69f698c8c3d10f4b7e97db7f738ddc8dc978: MANIFEST_UNKNOWN: Manifest has media type "application/vnd.docker.distribution.manifest.v2+json" but client accepts []
Additional information
No response
How are these images built and published?
The images are built and pushed with the "docker/build-push-action@v3" Github Action. Both for the images in Artifact Registry and Container Registry.
What version of gcr-cleaner? Do you have a minimal reproduction case you can share?
I just figured it out. The last / in <location>-docker.pkg.dev/<project>/<repository>/staging/ was too much. The path went one level too deep.
Still thank you for your reply! And thanks for creating this tool! And sorry for being confused.