System.IO.IOException: No space left on device

Question

System.IO.IOException: No space left on device

zspitzer opened this issue 2 years ago · comments

Description

Run df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/root         11M 1002K  9.7M   10% /
devtmpfs         8[6](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:7)[7](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:8)K   450  [8](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:9)67K    1% /dev
tmpfs            868K     5  868K    1% /dev/shm
tmpfs            868K   75[9](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:10)  868K    1% /run
tmpfs            868K     3  868K    1% /run/lock
tmpfs            868K    19  868K    1% /sys/fs/cgroup
/dev/loop0        486   486     0  [10](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:11)0% /snap/snapd/16292
/dev/loop2        [12](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:13)K   12K     0  100% /snap/core20/1518
/dev/loop1        802   802     0  100% /snap/lxd/22753
/dev/sdb15          0     0     0     - /boot/efi
/dev/sda1        896K    [13](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:14)  896K    1% /mnt
tmpfs            868K    22  868K    1% /run/user/1001

https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true

Platforms affected

Azure DevOps
GitHub Actions

Virtual environments affected

Image version and build link

Virtual Environment
Environment: ubuntu-20.04
Version: 20220717.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20220717.1/images/linux/Ubuntu2004-Readme.md
Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20220717.1

Is it regression?

yes

Expected behavior

runner should have more than 9.7MB free on /

Actual behavior

build
Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.294.0/_diag/Worker_20220722-085135-utc.log'
at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan1 buffer, Int64 fileOffset) at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite() at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at System.Diagnostics.TraceSource.Flush() at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing) at GitHub.Runner.Common.TraceManager.Dispose() at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing) at GitHub.Runner.Common.HostContext.Dispose() at GitHub.Runner.Worker.Program.Main(String[] args) System.IO.IOException: No space left on device : '/home/runner/runners/2.294.0/_diag/Worker_20220722-085135-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.294.0/_diag/Worker_20220722-085135-utc.log'
at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
at GitHub.Runner.Common.Tracing.Error(Exception exception)
at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)

Repro steps

run the work flow https://github.com/lucee/extension-compress/actions/workflows/main.yml

Mikhail Koliada · Answer 1 · Fri Jul 22 2022 19:23:13 GMT+0800 (China Standard Time)

@zspitzer hi!

We will take a look

Mikhail Koliada · Answer 2 · Wed Jul 27 2022 02:57:45 GMT+0800 (China Standard Time)

@zspitzer hello!

I see that:

we do not have fs quota enabled on runners
you still have enough free space before tests run (I mean plain df -h)
even if I lift ext4 fs reservation to 0% via tune2fs -m 0 the problem is still existing
So my only guess is that space is exhausted somewhere in the middle of the failed step.
Could you try to attach strace to the run itself or the process so it is possible to save logs and see what happens under the hood?

Mikhail Koliada · Answer 3 · Thu Aug 04 2022 00:09:31 GMT+0800 (China Standard Time)

I am going to close this issue as we do not have any updates for the time being, feel free to contact us again if you have more details or any questions left.