Docker-ce 25.0.0 breaks image squashing
Chekov2k opened this issue · comments
There is an issue with using docker-squash and the latest stable docker release. Using the example Dockefile
ARG BASE_IMAGE="docker"
FROM ${BASE_IMAGE}
ARG VERSION="0.48.3"
ARG SUFFIX="64bit"
RUN wget --no-verbose https://github.com/aquasecurity/trivy/releases/download/v${VERSION}/trivy_${VERSION}_Linux-${SUFFIX}.tar.gz -O - | tar -zxvf - && \
mv trivy /usr/bin
and building it via
docker build -t test .
I can successfully squash an image with docker-ce version 24.0.7, see
v24.0.7.log
However, if I use docker-ce 25.0.0 I get
docker-squash -v --tmp-dir scratch ${DOCKER_SQUASH_FROM} -t test test
2024-01-22 06:56:48,858 root DEBUG Running version 1.1.0
2024-01-22 06:56:48,858 root DEBUG Preparing Docker client...
2024-01-22 06:56:48,858 docker.utils.config DEBUG Trying paths: ['/home/infrastructure/.docker/config.json', '/home/infrastructure/.dockercfg']
2024-01-22 06:56:48,858 docker.utils.config DEBUG Found file at path: /home/infrastructure/.docker/config.json
2024-01-22 06:56:48,873 urllib3.connectionpool DEBUG http://localhost:None "GET /version HTTP/1.1" 200 843
2024-01-22 06:56:48,875 urllib3.connectionpool DEBUG http://localhost:None "GET /v1.44/_ping HTTP/1.1" 200 2
2024-01-22 06:56:48,876 root DEBUG Docker client ready
2024-01-22 06:56:48,890 urllib3.connectionpool DEBUG http://localhost:None "GET /v1.44/version HTTP/1.1" 200 843
2024-01-22 06:56:48,890 root INFO docker-squash version 1.1.0, Docker 25.0.0, API 1.44...
2024-01-22 06:56:48,891 root INFO Using v2 image format
2024-01-22 06:56:48,891 root DEBUG Using scratch as the temporary directory
2024-01-22 06:56:48,894 urllib3.connectionpool DEBUG http://localhost:None "GET /v1.44/images/test/json HTTP/1.1" 200 None
2024-01-22 06:56:48,900 urllib3.connectionpool DEBUG http://localhost:None "GET /v1.44/images/sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5/history HTTP/1.1" 200 None
2024-01-22 06:56:48,901 root INFO Old image has 31 layers
2024-01-22 06:56:48,901 root DEBUG Old layers: ['<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', 'sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5']
2024-01-22 06:56:48,901 root DEBUG We detected number of layers as the argument to squash
2024-01-22 06:56:48,901 root INFO Checking if squashing is necessary...
2024-01-22 06:56:48,901 root INFO Attempting to squash last 31 layers...
2024-01-22 06:56:48,901 root DEBUG Layers to squash: ['<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', 'sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5']
2024-01-22 06:56:48,901 root DEBUG Layers to move: []
2024-01-22 06:56:48,901 root INFO Saving image sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5 to scratch/old directory...
2024-01-22 06:56:48,901 root DEBUG Try #1...
2024-01-22 06:56:53,129 urllib3.connectionpool DEBUG http://localhost:None "GET /v1.44/images/sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5/get HTTP/1.1" 200 None
2024-01-22 06:56:53,130 root DEBUG Extracting image using iterator over raw data
2024-01-22 06:56:53,939 root INFO Image saved!
2024-01-22 06:56:53,939 root INFO Squashing image 'test'...
2024-01-22 06:56:53,940 root DEBUG Reading 'scratch/old/manifest.json' JSON file...
2024-01-22 06:56:53,940 root DEBUG Reading 'scratch/old/blobs/sha256/e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5' JSON file...
2024-01-22 06:56:53,940 root INFO Starting squashing...
2024-01-22 06:56:53,940 root INFO Squashing file 'scratch/old/blobs/layer.tar'...
2024-01-22 06:56:53,941 root DEBUG Cleaning up scratch temporary directory
2024-01-22 06:56:53,993 root ERROR [Errno 2] No such file or directory: 'scratch/old/blobs/layer.tar'
Traceback (most recent call last):
File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/cli.py", line 126, in run
squash.Squash(
File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/squash.py", line 90, in run
return self.squash(image)
File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/squash.py", line 125, in squash
new_image_id = image.squash()
File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/image.py", line 77, in squash
ret = self._squash()
File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/v2_image.py", line 41, in _squash
self._squash_layers(self.layer_paths_to_squash, self.layer_paths_to_move)
File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/image.py", line 779, in _squash_layers
layer_tar: tarfile.TarFile = tarfile.open(
File "/usr/lib/python3.8/tarfile.py", line 1603, in open
return func(name, "r", fileobj, **kwargs)
File "/usr/lib/python3.8/tarfile.py", line 1667, in gzopen
fileobj = GzipFile(name, mode + "b", compresslevel, fileobj)
File "/usr/lib/python3.8/gzip.py", line 173, in __init__
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'scratch/old/blobs/layer.tar'
2024-01-22 06:56:53,996 root ERROR Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!
I'm going to look further into this but thought raising an issue already might help
Might be related to moby/moby#44598 The docker image save tarball output is now OCI compliant.
Looks like the exported image old/blobs dir has stripped the /layer.tar and looks like this now
ls -l old/blobs/sha256
total 250168
-r--r--r-- 1 infrastructure staff 1.5K 1 Jan 1970 0d979313e9f9262364f038e05249664ce08e9ae049b5f4145a10b45fb64151f3
-r--r--r-- 1 infrastructure staff 1.1K 1 Jan 1970 136d873267a8e1955991a56e76b6bf86f6af0ef234d12de7d73c5f7208e1a50e
-r--r--r-- 1 infrastructure staff 544B 1 Jan 1970 1471633a84a6349cfeb08bf4b9e8879be2fdbdb8d55b489543be73d31bf91c57
-r--r--r-- 1 infrastructure staff 1.9M 1 Jan 1970 1d4959de38143abac793d31e4e8592696721305ff06281f8b9284aee380afcfe
-r--r--r-- 1 infrastructure staff 3.2K 1 Jan 1970 37c9a49ec4c63595dcad7f349573111e79ea86a4345028cfd8f9ea1c4374f044
-r--r--r-- 1 infrastructure staff 3.4K 1 Jan 1970 4993a3d53d15a092f04c547ffeff56f3bd98c746e4981166268602977605aa1d
-r--r--r-- 1 infrastructure staff 91K 1 Jan 1970 4b221b641f720e676dc1fe4410dbb9fbd8a397497ac9a35fa99b2a43cc96f6f9
-r--r--r-- 1 infrastructure staff 566B 1 Jan 1970 4b31d8bab091aef7ea07d8dbc3dcaa8b79dd43beeec9a9073a3aa424e8087dae
-r--r--r-- 1 infrastructure staff 9.1M 1 Jan 1970 4e62c66ee4db0c28bdd0a8d371e3034736195c22a9cfc2221c71fd75403feffe
-r--r--r-- 1 infrastructure staff 32B 1 Jan 1970 4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
-r--r--r-- 1 infrastructure staff 15M 1 Jan 1970 5598890badaf41ad29ca6e7a62aecf9b18e64ff1308118644d6ecda191bee6ae
-r--r--r-- 1 infrastructure staff 15M 1 Jan 1970 594895876a66f55998b0c3d9c175ec0cfa723898879d7fe8f837de3b0e9e01e6
-r--r--r-- 1 infrastructure staff 13K 1 Jan 1970 5f808b0b65434bc22dee0c2293dada9530071a11efc6208bcc79157591716686
-r--r--r-- 1 infrastructure staff 116B 1 Jan 1970 934f51c79a118bc2c9ef86a286aee9f3c33f0c413bd6c9ffa6c949a71275b217
-r--r--r-- 1 infrastructure staff 856B 1 Jan 1970 9ad316ac6f2cfb87a38894f741f377b29e9313b9def6ad15fd307eca8d145d1c
-r--r--r-- 1 infrastructure staff 167B 1 Jan 1970 9ceb83de43d622e69841979ca0985ea131b55405352bca6e9496dc6753435594
-r--r--r-- 1 infrastructure staff 1.3K 1 Jan 1970 b0686185000f25f48b2caaea5bba0d9a7c01edd8b1e365dd62827707ec488560
-r--r--r-- 1 infrastructure staff 16M 1 Jan 1970 b72e0c5c013142ab0f105384e856690145c739b4c62798a667ba44e8b100815b
-r--r--r-- 1 infrastructure staff 548B 1 Jan 1970 bf613c8d3eefad31faacee38beeedf68069e48ea01d8f083fa0a57ce343ed827
-r--r--r-- 1 infrastructure staff 3.2M 1 Jan 1970 c303524923177661067f7eb378c3dd5277088c2676ebd1cd78e68397bb80fdbf
-r--r--r-- 1 infrastructure staff 13M 1 Jan 1970 d46db1bcf02a720cf4e32fb0c58cc23f2310a263bde464ca9ce928d473c159e9
-r--r--r-- 1 infrastructure staff 1.0K 1 Jan 1970 d64d5b7dddfb9901291c707c8b672a987b74197fe26a2a8c58747cf784a5c08d
-r--r--r-- 1 infrastructure staff 49M 1 Jan 1970 e09e5717c3fbfa466bfb8a152c3718c081fb34db45e79fa281023aace028913d
We had the same issue after docker-ce and docker-ce-cli was auto-upgraded from version 24.0.7 to 25.0.0 on our build machine. As a workaround we downgraded to the 24.0.7 docker packages which resolved the issue.
2024-01-22T06:43:20.0552376Z Command: �[33mdocker-squash --output-path CloudCe_docker.tar -t cloudce:MAIN-42257 cloudce_full:MAIN-42257�[39m
2024-01-22T06:43:21.2208160Z 2024-01-22 06:43:21,217 root INFO docker-squash version 1.1.0, Docker 25.0.0, API 1.44...
2024-01-22T06:43:21.2210772Z 2024-01-22 06:43:21,218 root INFO Using v2 image format
2024-01-22T06:43:21.2298204Z 2024-01-22 06:43:21,227 root INFO Old image has 27 layers
2024-01-22T06:43:21.2354938Z 2024-01-22 06:43:21,227 root INFO Checking if squashing is necessary...
2024-01-22T06:43:21.2355181Z 2024-01-22 06:43:21,228 root INFO Attempting to squash last 27 layers...
2024-01-22T06:43:21.2355901Z 2024-01-22 06:43:21,228 root INFO Saving image sha256:0ac958c7ed1f7f38c3dc35b5022891aea77a6bd5e790b4db11bbe427e9198dd9 to /tmp/docker-squash-88vhom9s/old directory...
2024-01-22T06:43:41.3599383Z 2024-01-22 06:43:41,355 root INFO Image saved!
2024-01-22T06:43:41.3600393Z 2024-01-22 06:43:41,356 root INFO Squashing image 'cloudce_full:MAIN-42257'...
2024-01-22T06:43:41.4089020Z 2024-01-22 06:43:41,394 root INFO Starting squashing...
2024-01-22T06:43:41.4237307Z 2024-01-22 06:43:41,394 root INFO Squashing file '/tmp/docker-squash-88vhom9s/old/blobs/layer.tar'...
2024-01-22T06:43:41.5180141Z 2024-01-22 06:43:41,510 root ERROR [Errno 2] No such file or directory: '/tmp/docker-squash-88vhom9s/old/blobs/layer.tar'
2024-01-22T06:43:41.5181543Z 2024-01-22 06:43:41,511 root ERROR Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!
Do we need to manually generate a tarball now? What's the suggestion for compatibility with 25.0.0?
We had the same issue after docker-ce and docker-ce-cli was auto-upgraded from version 24.0.7 to 25.0.0 on our build machine. As a workaround we downgraded to the 24.0.7 docker packages which resolved the issue.
2024-01-22T06:43:20.0552376Z Command: �[33mdocker-squash --output-path CloudCe_docker.tar -t cloudce:MAIN-42257 cloudce_full:MAIN-42257�[39m 2024-01-22T06:43:21.2208160Z 2024-01-22 06:43:21,217 root INFO docker-squash version 1.1.0, Docker 25.0.0, API 1.44... 2024-01-22T06:43:21.2210772Z 2024-01-22 06:43:21,218 root INFO Using v2 image format 2024-01-22T06:43:21.2298204Z 2024-01-22 06:43:21,227 root INFO Old image has 27 layers 2024-01-22T06:43:21.2354938Z 2024-01-22 06:43:21,227 root INFO Checking if squashing is necessary... 2024-01-22T06:43:21.2355181Z 2024-01-22 06:43:21,228 root INFO Attempting to squash last 27 layers... 2024-01-22T06:43:21.2355901Z 2024-01-22 06:43:21,228 root INFO Saving image sha256:0ac958c7ed1f7f38c3dc35b5022891aea77a6bd5e790b4db11bbe427e9198dd9 to /tmp/docker-squash-88vhom9s/old directory... 2024-01-22T06:43:41.3599383Z 2024-01-22 06:43:41,355 root INFO Image saved! 2024-01-22T06:43:41.3600393Z 2024-01-22 06:43:41,356 root INFO Squashing image 'cloudce_full:MAIN-42257'... 2024-01-22T06:43:41.4089020Z 2024-01-22 06:43:41,394 root INFO Starting squashing... 2024-01-22T06:43:41.4237307Z 2024-01-22 06:43:41,394 root INFO Squashing file '/tmp/docker-squash-88vhom9s/old/blobs/layer.tar'... 2024-01-22T06:43:41.5180141Z 2024-01-22 06:43:41,510 root ERROR [Errno 2] No such file or directory: '/tmp/docker-squash-88vhom9s/old/blobs/layer.tar' 2024-01-22T06:43:41.5181543Z 2024-01-22 06:43:41,511 root ERROR Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!
Same issues on our end as well with Docker 25.0.0 and 25.0.1. We've reverted back to 24.0.7 and can confirm that resolves these issues.
Thanks for the report. I can confirm that due to Docker 25.x switching to the OCI archive format it is incompatible with docker-squash which currently only supports docker-archive format. Note that podman currently saves by default in docker-archive format but is also planning on switching in its next major release. It would effectively need a completely new v3_image.py implementation to handle the different format.
I am currently looking into this and examining the differences between the different formats.
Primarily instead of e.g. 4007b9f4932dd241d3f16a73029e79520a73b1fda673d3b7eeacbab313d343ff/layer.tar one might have blobs/sha256/22694763edf2e6f5beec6a058c8ebc695c8976c2677aba5626f7a684df54bc20. There are also differences between the manifest.json , index.json and potentially the archive might also be compressed (I think - although I haven't seen Docker doing that in any experiments I have done).
Is there any chance of getting a new tagged release so we don't have to install from git to get this fix?
Thank you for the quick fix :)
We should be now good to go!
https://pypi.org/project/docker-squash/1.2.0/
https://github.com/goldmann/docker-squash/releases/tag/1.2.0
Big ups for @rnc for stepping in to fix this!