WCOW buildkit/buildx caching
fruzitent opened this issue · comments
Contributing guidelines
- I've read the contributing guidelines and wholeheartedly agree
I've found a bug and checked that ...
- ... the documentation does not mention anything about my problem
- ... there are no open or closed issues that are related to my problem
Description
For some reason, specifically buildx, does not cache unpacked Windows Servercore image, but does Windows Nanoserver.
Buildkit, on the other hand, does cache stages from both images properly.
Steps to reproduce
1.1. Build nanoserver image using buildx
1.2. Build nanoserver image using buildx again (caching works)
1.3. Build servercore image using buildx
1.4. Build servercore image using buildx again (caching does not work)
2.1 Build nanoserver image using buildctl
2.2 Build nanoserver image using buildctl again (caching works)
2.3 Build servercore image using buildctl
2.4 Build servercore image using buildctl again (caching works)
3.1 Build nanoserver image using buildx (it loaded cache from buildkit node)
3.2 Build servercore image using buildx (it loaded cache from buildkit node)
Expected behaviour
Following builds to skip unchanged stages.
Actual behaviour
Buildx does not save stage cache, however, is able to import it (if there is one) from Buildkit.
Buildx version
github.com/docker/buildx v0.13.1-desktop.1 2368b2ee41873b948cb607251cec573ed00f00f9
Docker info
Client:
Version: 26.0.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.13.1-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe
compose: Docker Compose (Docker Inc.)
Version: v2.26.1-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe
debug: Get a shell into any image or container. (Docker Inc.)
Version: 0.0.27
Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: C:\Program Files\Docker\cli-plugins\docker-dev.exe
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.23
Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.4
Path: C:\Program Files\Docker\cli-plugins\docker-feedback.exe
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.1.0
Path: C:\Program Files\Docker\cli-plugins\docker-init.exe
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe
scout: Docker Scout (Docker Inc.)
Version: v1.6.3
Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 5
Server Version: 26.0.0
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics internal l2bridge l2tunnel nat null overlay private transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 22631 (22621.1.amd64fre.ni_release.220506-1250)
Operating System: Microsoft Windows Version 23H2 (OS Build 22631.3447)
OSType: windows
Architecture: x86_64
CPUs: 16
Total Memory: 15.38GiB
Name: DESKTOP-A7O2H12
ID: e9984d83-abc3-4b05-a038-96db7debaeb5
Docker Root Dir: C:\ProgramData\Docker
Debug Mode: false
Labels:
com.docker.desktop.address=npipe://\\.\pipe\docker_cli
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Builders list
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
buildkit-exp* remote
\_ buildkit-exp0 \_ npipe:////./pipe/buildkitd inactive
default docker
\_ default \_ default running v0.13.1 windows/amd64
desktop-linux docker
\_ desktop-linux \_ desktop-linux running v0.13.1 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
desktop-windows docker
\_ desktop-windows \_ desktop-windows running v0.13.1 windows/amd64
Configuration
FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
RUN <<EOF
echo "Hello, nanoserver!"
EOF
FROM mcr.microsoft.com/windows/servercore:ltsc2022
RUN <<EOF
echo "Hello, servercore!"
EOF
Build logs
No response
Additional info
There is a seemingly related issue on the buildkit repo moby/buildkit#2274.
I tried compiling buildkitd from the master after that fix was merged and repeating the same steps, but it did not change anything.
step1_buildkitd.log
step1_buildx.log
step1_containerd.log
cc @profnandaa
I wonder if this is the GC default storage limit kicking in and servercore
is the bigger image. Check buildx inspect
and buildx du
.
➜ ~ docker buildx inspect
Name: buildkit-exp
Driver: remote
Last Activity: 2024-04-15 13:39:20 +0000 UTC
Nodes:
Name: buildkit-exp0
Endpoint: npipe:////./pipe/buildkitd
Status: running
BuildKit version: v0.13.1
Platforms: windows/amd64
Labels:
org.mobyproject.buildkit.worker.containerd.namespace: buildkit
org.mobyproject.buildkit.worker.containerd.uuid: 80aae2c4-eafe-43af-809c-2bbee660b401
org.mobyproject.buildkit.worker.executor: containerd
org.mobyproject.buildkit.worker.hostname: DESKTOP-A7O2H12
org.mobyproject.buildkit.worker.network: cni
org.mobyproject.buildkit.worker.selinux.enabled: false
org.mobyproject.buildkit.worker.snapshotter: windows
GC Policy rule#0:
All: false
Filters: type==source.local,type==exec.cachemount,type==source.git.checkout
Keep Duration: 48h0m0s
Keep Bytes: 488.3MiB
GC Policy rule#1:
All: false
Keep Duration: 1440h0m0s
Keep Bytes: 1.863GiB
GC Policy rule#2:
All: false
Keep Bytes: 1.863GiB
GC Policy rule#3:
All: true
Keep Bytes: 1.863GiB
➜ ~ docker buildx du
ID RECLAIMABLE SIZE LAST ACCESSED
fozyq86h7n81d4gr68b00jkjb true 18.01GB* 9 hours ago
2ayz1tef4e4bq7lyoqadlox0d true 17.98GB* 7 hours ago
ifkou0cwokqo6zgea4jlqmuzt true 7.336GB* 11 hours ago
7vvwuu919c9ffzy4ot852qxmo true 5.941GB* 11 hours ago
mtxdriimlm219j2s6tynkm78w true 939.6MB* 7 hours ago
cq9f7xsqzf50nemwfzje7ugs2 true 746.8MB 11 hours ago
xxv5vxg8bur524vwnglgwudgf true 673.5MB 11 hours ago
o2a1dzv5jasbiqpfnw8dy6zfy true 174.9MB 11 hours ago
ns32won5iib26msan08j9xs4w* true 4.933kB 7 hours ago
n4qs13u4o8k7650v9fxm57gnr* true 101B 11 hours ago
n6vytn04tfgpb41ki2rd1mrcc* true 0B 7 hours ago
hg2roawofuky4ch58ty2mbhng* true 0B 11 hours ago
Shared: 50.21GB
Private: 1.595GB
Reclaimable: 51.81GB
Total: 51.81GB
GC configured at 1.8 GB is probably not what you want with such big images. Set --containerd-worker-gc false
to turn off the GC (or keepstorage to tweak it).
@profnandaa We need to look into how such a small limit happened. And maybe set a minimum for wcow as it tends to use huge images.
Docker's Desktop built-in runner (does not support buildkit) says it has unlimited storage
That's not what you are using to build. Based on your other output the builder you are using is called buildkit-exp
tested with buildkitd --containerd-worker-gc=false
it works
➜ wcow-test ✗ docker buildx inspect
Name: buildkit-exp
Driver: remote
Last Activity: 2024-04-16 03:59:55 +0000 UTC
Nodes:
Name: buildkit-exp0
Endpoint: npipe:////./pipe/buildkitd
Status: running
BuildKit version: v0.13.1
Platforms: windows/amd64
Labels:
org.mobyproject.buildkit.worker.containerd.namespace: buildkit
org.mobyproject.buildkit.worker.containerd.uuid: b84d108d-a0e9-4a2e-a687-006295e0a86f
org.mobyproject.buildkit.worker.executor: containerd
org.mobyproject.buildkit.worker.hostname: DESKTOP-A7O2H12
org.mobyproject.buildkit.worker.network: cni
org.mobyproject.buildkit.worker.selinux.enabled: false
org.mobyproject.buildkit.worker.snapshotter: windows
➜ wcow-test docker buildx du
ID RECLAIMABLE SIZE LAST ACCESSED
af3fw2gwc53dxsd89n8s2sfrl true 7.336GB 14 minutes ago
sax8i4cwg8z1lpjqgrbvpajmg true 5.941GB 2 minutes ago
n2n3th25eybkblxaj6dn1vtay true 746.7MB 32 seconds ago
m1rgmim2liy644g4g4122gi6c* true 97B 32 seconds ago
i6dkls59gv6qviaapskbsdckz* true 0B 32 seconds ago
Reclaimable: 14.02GB
Total: 14.02GB
@profnandaa I think I found the issue.
If no GC is configured by the user then in Linux the default is 10% of disk: https://github.com/moby/buildkit/blob/master/cmd/buildkitd/config/gcpolicy_unix.go#L11
But on wcow it is just a constant 2GB 😬 https://github.com/moby/buildkit/blob/master/cmd/buildkitd/config/gcpolicy_windows.go#L7
@tonistiigi -- thanks! You think we should just do the same for WCOW for consistency? I can work on the PR.
Let's transfer this to moby/buildkit