accessing secret very slow
pongnguy opened this issue · comments
Description
Any step that uses a build secret is very slow. Should be fast since file is local and of a very small size.
Reproduce
docker build --secret id=secret-key,src=secret-source -t image-name .
Expected behavior
Step involving key should take the same amount of time as other steps of similar complexity
docker version
Client: Docker Engine - Community
Version: 26.0.0
API version: 1.45
Go version: go1.21.8
Git commit: 2ae903e
Built: Wed Mar 20 15:17:48 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.0.0
API version: 1.45 (minimum version 1.24)
Go version: go1.21.8
Git commit: 8b79278
Built: Wed Mar 20 15:17:48 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.28
GitCommit: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client: Docker Engine - Community
Version: 26.0.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.13.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.25.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 5
Running: 3
Paused: 0
Stopped: 2
Images: 39
Server Version: 26.0.0
Storage Driver: overlay2
Backing Filesystem: btrfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: active
NodeID: nrfo0xui5r0hhpj5zzkafjjqd
Is Manager: true
ClusterID: uh2qrtf90k9xa3yimp07wsu60
Managers: 2
Nodes: 2
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.0.151
Manager Addresses:
10.0.0.151:2377
10.0.0.248:2377
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-107-generic
Operating System: Ubuntu 22.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 62.78GiB
Name: r73043420
ID: d9387ddf-63d1-4828-a1dc-b60c6bc81f4a
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: Running Swarm in a two-manager configuration. This configuration provides
no fault tolerance, and poses a high risk to lose control over the cluster.
Refer to https://docs.docker.com/engine/swarm/admin_guide/ to configure the
Swarm for fault-tolerance.
Additional Info
No response
Let me transfer this to the build repository, which is where the CLI side of build is maintained
Can you put together a reproducer?
I'm also seeing this issue. Here is a reproducer:
FROM python:latest
RUN echo "without mount"
RUN --mount=type=secret,id=SECRET,uid=1000 \
echo "with mount"
The second command takes ~5s to start running.
I've seen run commands take more than a minute to start running when mounting secrets during the build of an image. In this case I'm building a much larger image, so I'm wondering if this time scales with the size of the image
We have also seen that versions starting with 26.0.0 have increased build times when mounting secrets. Builds are very fast up until the mount step, and then it just freezes for a couple minutes then eventually continues. We downgraded to Docker version <26 (25.0.5 atm) and builds are fast again. So far we've confirmed even the most recent Version 27 upgrade suffers from the same issue.
We've observed the same slow secret behavior in our builds starting with 26.0.0 and have similarly downgraded to 25.0.5 to maintain build performance. Observed slowness can be in the minutes for steps involving mounting of secrets, like something is hung pending a timeout (pure conjecture).
Ticket is tagged needs more info. What more is needed?
I will take a look, is someone able to repro with a container builder as well?