Microsoft.Testing.Extensions.HangDump does not work on macOS or Linux in native AOT mode

Question

Microsoft.Testing.Extensions.HangDump does not work on macOS or Linux in native AOT mode

martincostello opened this issue 4 months ago · comments

Describe the bug

Following this tip #3095 (comment) I added the Microsoft.Testing.Extensions.HangDump package to a native AoT test project of mine to try and diagnose a hanging test (which in my case I can only repro on Linux).

Upon making the changes martincostello/alexa-london-travel#1298 and running the CI, the tests fail with a unique exception message each on macOS and Linux as shown below. I can also repo this outside of GitHub Actions CI with WSL.

macOS

Unhandled Exception: System.ArgumentOutOfRangeException: The path '/var/folders/dm/88b38gj92jj53dgxdsm12qf00000gn/T/hangdumpgeneratorpipename.240706229_b5437ec2c2144a679172c9d6fe2a036d/.p' is of an invalid length for use with domain sockets on this platform.  The length must be between 1 and 104 characters, inclusive. (Parameter 'path')
Actual value was /var/folders/dm/88b38gj92jj53dgxdsm12qf00000gn/T/hangdumpgeneratorpipename.240706229_b5437ec2c2144a679172c9d6fe2a036d/.p.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.<InternalRunAsync>d__9.MoveNext() + 0x710
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.<InternalRunAsync>d__9.MoveNext() + 0xfd0
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x2d4
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x670
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Hosts.TestHostControlledHost.<RunAsync>d__4.MoveNext() + 0xec
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Builder.TestApplication.<RunAsync>d__17.MoveNext() + 0xc4
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Helpers.TaskExtensions.<>c.<<TimeoutAfterAsync>b__2_0>d.MoveNext() + 0xa8
   at TestingPlatformEntryPoint.<Main>d__0.MoveNext() + 0x284
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at TestingPlatformEntryPoint.<Main>(String[] args) + 0x3c
   at LondonTravel.Skill!<BaseAddress>+0xae7ca8
Unhandled Exception: System.InvalidOperationException: Unexpected state in file '/_/src/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs' at line '185'
   at Microsoft.Testing.Platform.Helpers.ApplicationStateGuard.Ensure(Boolean, String, Int32) + 0xb0
   at Microsoft.Testing.Extensions.Diagnostics.HangDumpProcessLifetimeHandler.<OnTestHostProcessStartedAsync>d__42.MoveNext() + 0x6c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Hosts.TestHostControllersTestHost.<InternalRunAsync>d__21.MoveNext() + 0x238c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.TestHostControllersTestHost.<InternalRunAsync>d__21.MoveNext() + 0x2d48
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x2d4
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x670
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Builder.TestApplication.<RunAsync>d__17.MoveNext() + 0xc4
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at TestingPlatformEntryPoint.<Main>d__0.MoveNext() + 0x284
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at TestingPlatformEntryPoint.<Main>(String[] args) + 0x3c
   at LondonTravel.Skill!<BaseAddress>+0xae7ca8

Linux

Unhandled Exception: System.IO.IOException: Broken pipe
 ---> System.Net.Sockets.SocketException (32): Broken pipe
   at System.Exception.SetCurrentStackTrace() + 0x63
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.SetCurrentStackTrace(Exception) + 0x18
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError, Boolean) + 0x72
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsync(Socket, CancellationToken) + 0xe1
   at System.Net.Sockets.Socket.SendAsync(ReadOnlyMemory`1, SocketFlags, CancellationToken) + 0xea
   at System.IO.Pipes.PipeStream.<WriteAsyncCore>d__83.MoveNext() + 0x93
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine&) + 0x42
   at System.IO.Pipes.PipeStream.WriteAsyncCore(ReadOnlyMemory`1, CancellationToken) + 0x4a
   at System.IO.Pipes.PipeStream.WriteAsync(ReadOnlyMemory`1, CancellationToken) + 0xb4
   at Microsoft.Testing.Platform.IPC.NamedPipeClient.<RequestReplyAsync>d__13`2.MoveNext() + 0x8fb
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine&) + 0x5e
   at Microsoft.Testing.Platform.IPC.NamedPipeClient.RequestReplyAsync[TRequest,TResponse](TRequest, CancellationToken) + 0x63
   at Microsoft.Testing.Extensions.Diagnostics.HangDumpActivityIndicator.<OnTestSessionFinishingAsync>d__33.MoveNext() + 0x13f
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine&) + 0x42
   at Microsoft.Testing.Extensions.Diagnostics.HangDumpActivityIndicator.OnTestSessionFinishingAsync(SessionUid, CancellationToken) + 0x35
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<NotifyTestSessionEndAsync>d__12.MoveNext() + 0x176
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) + 0x8c
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) + 0x66
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) + 0xa7
   at System.Threading.Tasks.Task.RunContinuations(Object) + 0xa8
   at System.Threading.Tasks.Task`1.TrySetResult(TResult) + 0x88
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1, TResult) + 0x39
   at Microsoft.Testing.Platform.Messages.MessageBusProxy.<DrainDataAsync>d__6.MoveNext() + 0xef
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) + 0x8c
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) + 0x66
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) + 0xa7
   at System.Threading.Tasks.Task.RunContinuations(Object) + 0xa8
   at System.Threading.Tasks.Task`1.TrySetResult(TResult) + 0x88
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1, TResult) + 0x39
   at Microsoft.Testing.Platform.Messages.AsynchronousMessageBus.<DrainDataAsync>d__17.MoveNext() + 0x8a3
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) + 0x8c
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) + 0x66
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) + 0xa7
   at System.Threading.Tasks.Task.RunContinuations(Object) + 0xa8
   at System.Threading.Tasks.Task`1.TrySetResult(TResult) + 0x86
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1, TResult) + 0x38
   at Microsoft.Testing.Platform.Messages.AsyncConsumerDataProcessor.<DrainDataAsync>d__14.MoveNext() + 0x329
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) + 0x8c
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) + 0x66
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) + 0xa7
   at System.Threading.Tasks.Task.RunContinuations(Object) + 0xa8
   at System.Threading.Tasks.Task.TrySetResult() + 0x6f
   at System.Threading.Tasks.Task.DelayPromise.CompleteTimedOut() + 0x13
   at System.Threading.TimerQueueTimer.Fire(Boolean) + 0x60
   at System.Threading.TimerQueue.FireNextTimers() + 0x23b
   at System.Threading.ThreadPoolWorkQueue.Dispatch() + 0x2d1
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() + 0x15a
   at System.Threading.Thread.StartThread(IntPtr) + 0xee
   at System.Threading.Thread.ThreadEntryPoint(IntPtr) + 0x19
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at System.IO.Pipes.PipeStream.<WriteAsyncCore>d__83.MoveNext() + 0x16b
   --- End of inner exception stack trace ---
   at System.IO.Pipes.PipeStream.<WriteAsyncCore>d__83.MoveNext() + 0x23e
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.IPC.NamedPipeClient.<RequestReplyAsync>d__13`2.MoveNext() + 0x94c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at Microsoft.Testing.Platform.Helpers.TaskExtensions.<>c.<<TimeoutAfterAsync>b__2_0>d.MoveNext() + 0x96
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<ExecuteRequestAsync>d__8.MoveNext() + 0x318
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.<InternalRunAsync>d__9.MoveNext() + 0x7b6
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.<InternalRunAsync>d__9.MoveNext() + 0x1094
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x2ae
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x5f3
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Hosts.TestHostControlledHost.<RunAsync>d__4.MoveNext() + 0xd9
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Builder.TestApplication.<RunAsync>d__17.MoveNext() + 0xaf
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at TestingPlatformEntryPoint.<Main>d__0.MoveNext() + 0x244
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at TestingPlatformEntryPoint.<Main>(String[] args) + 0x2e
   at LondonTravel.Skill!<BaseAddress>+0xa10b7c

Steps To Reproduce

Clone martincostello/alexa-london-travel@b93a00a onto a Linux or macOS machine.
Run ./build.ps1 in the root of the repository.

Expected behavior

Either:

The tests pass, or:
The tests produce a dump file relating to a hanging test.

Actual behavior

The process exits with an error of either ArgumentOutOfRangeException on macOS or IOException on Linux.

Additional context

In case it was an issue with the file name I was providing, I tried not specifying a file name at all in the hope of it having a default like dotnet test does. In that case, a different failure is observed:

Unhandled Exception: System.InvalidOperationException: Cannot find mutex 'TESTINGPLATFORM_HANGDUMP_MUTEXNAME_d28475404b504b3aacf951ff5509d4c6'
   at Microsoft.Testing.Extensions.Diagnostics.HangDumpProcessLifetimeHandler.<ActivityTimerAsync>d__44.MoveNext() + 0x39c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Helpers.SystemTask.<>c__DisplayClass4_0.<RunLongRunning>b__0() + 0x92
   at System.Threading.Thread.StartThread(IntPtr) + 0xee
   at System.Threading.Thread.ThreadEntryPoint(IntPtr) + 0x19

Amaury Levé · Answer 1 · Fri Jun 14 2024 23:40:40 GMT+0800 (China Standard Time)

@MarcoRossignoli I think it's another bug of pipe name under non-windows, could you check it please?

Marco Rossignoli · Answer 2 · Sat Jun 15 2024 03:41:00 GMT+0800 (China Standard Time)

Thanks @martincostello for reporting it, I'll check the length name issue, anyway I don't know if hang/crash works for native aot version.

@brianrob @tommcdon we're using the environment variable for the crash dump https://learn.microsoft.com/en-us/dotnet/core/diagnostics/collect-dumps-crash and the Microsoft.Diagnostics.NETCore.Client to take a dump in case of hang, is it working in native aot mode?