Replaced quay.io/skopeo/stable images keep being deleted

[Ethan Kent]

When attempting to use a SHA-256 digest for the Skopeo image, it appears that non-current digests are not available. This limitation is causing disruptions in our DevOps workflow.

[Miloslav Trmač]

Thanks for reaching out.

Please provide a reproducer / expected results / actual results.

[Ethan Kent]

Steps to Reproduce:

# Set up a docker dev environment compatible with `docker compose`
mkdir skopeo-test
cat <<EOT >> ./skopeo-test/docker-compose.yml
version: "3.9"

services:
  skopeo:
    image: quay.io/skopeo/stable@sha256:69d85072fe33f3f827930f6ad766154d5b411f6b0c7975a6840fda1c6cb2a80a
    pull_policy: always
EOT

# Test
docker compose -f ./skopeo-test/docker-compose.yml up

Expected results: the skopeo image is pulled and the build completes

Actual results: (error output)

[+] Running 1/1
 ✘ skopeo Error                                                            1.9s 
Error response from daemon: manifest for quay.io/skopeo/stable@sha256:69d85072fe33f3f827930f6ad766154d5b411f6b0c7975a6840fda1c6cb2a80a not found: manifest unknown: manifest unknown
➜  

Impact:

This results in build environments that fail the moment they become slightly out-of-date. Which in turn causes staff to lose trust in the the build environment.

Internal Workarounds:

We try to keep the images perfectly up to date, but this still is subject to inherent lag-time where build environments will simply break. We could specify the image via its version tag, but this tends to show up as a configuration error. See checkov docs for an example of such a configuration error policy.

Possible Solution:

I'm not familiar with your build/publish workflow, so I'm not able to diagnose any of the root causes of this behavior. But if I were attempting to fix this issue, I imagine I might experiment a couple of different ways.

1. Mirror the image on another service - this might eliminate potential limitations from the image registry.
2. Add an extra tag to each image that's released as an official version - this might prevent an automated process from deleting old images simply because they are not tagged.
3. Release an image based on scratch - this may remove spurious releases that are not directly related to core software updates (which might make the process of keeping these images updated less labor-intensive).

Thanks for looking in to this. I think skopeo is a great tool & I'm looking forward to using it more.

[Miloslav Trmač]

Ah, thanks, I didn’t realize this talks about the images we publish.

Yes, the most recent images keep being rebuilt to include security updates and the like. And quay.io is fairly aggressively removing untagged images.

@cevich is working on changing the image publishing workflow/design; I’m not sure if there is a specific issue to follow progress.

[Ethan Kent]

Ah yes. I think I see it now: containers/automation_images#310 and #2133.

I'm not seeing any indication of progress however. Maybe the original issues don't communicate any consequential impact. For me, this is essentially breaking our ability to pin these images to a sha256 digest - which is a must for our more compliance oriented use cases.

Thanks again @mtrmac for taking time to look into this.

[Miloslav Trmač]

Compliance-oriented use cases might also need to have a contractual relationship with some promise of security updates; what happens in this Open Source project is not really that.

Consider something like registry.access.redhat.com/ubi9/skopeo. Now I’m not saying that they are actually preserving every past version — I wouldn’t know! — but there are build teams and build processes dedicated to making those images enterprise-consumable.

[Chris Evich]

@cevich is working on changing the image publishing workflow/design

Oh, completely forgot about that open containers/automation_images#310

Miloslav is correct, I have a chain of work scheduled to improve this situation. It will most likely end with us publishing (one time only) a tag having a -stable suffix.

Those tags will be left alone forever, so you can freely use the tag+sha scheme w/o breakage. I'll update that issue to this effect.

[Ethan Kent]

@cevich that sounds like a great solution. Thanks for looking into this.

[Chris Evich]

Great. I'll close this and track it over in containers/automation_images#310