Production workers are running out of container space for sandboxes
calebbrown opened this issue · comments
Kubernetes is configured to use a StatefulSet for the worker, with a persistent 30GB volume to cache Sandbox images.
Currently we only call podman image prune -f during Sandbox init to clean up dangling images.
Unfortunately this doesn't remove previous versions of images from the storage.
- All images removed - except those tagged with "latest" and
OSSF_SANDBOX_IMAGE_TAG. - This code must not run outside of
worker. - This code should only run if
OSSF_SANDBOX_IMAGE_TAGis set.
Finally, podman image prune has an argument --filter that can be used as well. However we may want to add labels to our docker images to facilitate this work using docker build --label ...
Alternative we may want to consider switching away form StatefulSet