System.IO.IOException: No space left on device
zspitzer opened this issue · comments
Description
Run df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 11M 1002K 9.7M 10% /
devtmpfs 8[6](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:7)[7](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:8)K 450 [8](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:9)67K 1% /dev
tmpfs 868K 5 868K 1% /dev/shm
tmpfs 868K 75[9](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:10) 868K 1% /run
tmpfs 868K 3 868K 1% /run/lock
tmpfs 868K 19 868K 1% /sys/fs/cgroup
/dev/loop0 486 486 0 [10](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:11)0% /snap/snapd/16292
/dev/loop2 [12](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:13)K 12K 0 100% /snap/core20/1518
/dev/loop1 802 802 0 100% /snap/lxd/22753
/dev/sdb15 0 0 0 - /boot/efi
/dev/sda1 896K [13](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:14) 896K 1% /mnt
tmpfs 868K 22 868K 1% /run/user/1001
https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true
Platforms affected
- Azure DevOps
- GitHub Actions
Virtual environments affected
- Ubuntu 18.04
- Ubuntu 20.04
- Ubuntu 22.04
- macOS 10.15
- macOS 11
- macOS 12
- Windows Server 2019
- Windows Server 2022
Image version and build link
Virtual Environment
Environment: ubuntu-20.04
Version: 20220717.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20220717.1/images/linux/Ubuntu2004-Readme.md
Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20220717.1
Is it regression?
yes
Expected behavior
runner should have more than 9.7MB free on /
Actual behavior
build
Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.294.0/_diag/Worker_20220722-085135-utc.log'
at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan1 buffer, Int64 fileOffset) at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite() at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at System.Diagnostics.TraceSource.Flush() at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing) at GitHub.Runner.Common.TraceManager.Dispose() at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing) at GitHub.Runner.Common.HostContext.Dispose() at GitHub.Runner.Worker.Program.Main(String[] args) System.IO.IOException: No space left on device : '/home/runner/runners/2.294.0/_diag/Worker_20220722-085135-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan
1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.294.0/_diag/Worker_20220722-085135-utc.log'
at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
at GitHub.Runner.Common.Tracing.Error(Exception exception)
at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
Repro steps
run the work flow https://github.com/lucee/extension-compress/actions/workflows/main.yml
@zspitzer hi!
We will take a look
@zspitzer hello!
I see that:
- we do not have fs quota enabled on runners
- you still have enough free space before tests run (I mean plain
df -h
) - even if I lift ext4 fs reservation to 0% via
tune2fs -m 0
the problem is still existing
So my only guess is that space is exhausted somewhere in the middle of the failed step.
Could you try to attach strace to the run itself or the process so it is possible to save logs and see what happens under the hood?
I am going to close this issue as we do not have any updates for the time being, feel free to contact us again if you have more details or any questions left.