Unreasonably large ELFHeader when generating symbols from coredump

Question

Unreasonably large ELFHeader when generating symbols from coredump

taion809 opened this issue 5 years ago · comments

Hello,
We've created a coredump using the createdump utility. Upon trying to use dotnet-dump analyze we see the following:

> clrstack -all
...
00007F85D2FFA4D0 00007F85FE7F08B0 /app/NLog.dll!Unknown
00007F85D2FFA500 00007F85FCF5B2AF /app/NLog.dll!Unknown
00007F85D2FFA570 00007F85FCF55E96 /app/NLog.dll!Unknown
00007F85D2FFA5B0 00007F85FCF55DC2 /app/NLog.dll!Unknown
00007F85D2FFA5F0 00007F85FCF559C5 /app/NLog.dll!Unknown
...

Following this documentation, when attempting to generate symbols from the dump we see the following

> dotnet-symbol ./3f13aaf3-3485-01c7-1ee6-82a583e97eb2.coredump
Downloading from http://msdl.microsoft.com/download/symbols/
ERROR: Unreasonably large ELFHeader ProgramHeaderCount

How should we proceed with debugging this?

dotnet info:

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100
 Commit:    04339c3a26

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/3.0.100/

Host (useful for support):
  Version: 3.0.0
  Commit:  95a0a61858

.NET Core SDKs installed:
  2.2.402 [/usr/share/dotnet/sdk]
  3.0.100 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Mike McLaughlin · Answer 1 · Fri Oct 11 2019 00:21:19 GMT+0800 (China Standard Time)

Is there anyway you can zip up the coredump and attach?

Your clrstack -all UNKNOWN problem may be that you need your app assemblies on the machine you are loading the coredump. I'm assuming that /app/NLog.dll is part of your app. If so, dotnet-symbol isn't going to find it on the symbol servers. If you are loading the coredump on a different machine that it was generated, you need to copy your app binaries to a directory and include it in the symbol search with the SOS commmand setsymbolserver -directory <path-to-app>.

Nicholas Johns · Answer 2 · Fri Oct 11 2019 01:33:58 GMT+0800 (China Standard Time)

Hello,
Unfortunately I wouldn't be able to provide the coredump. I was able to pull the dll's from our docker container and everything is now resolving properly so thanks a bunch for that.

After pulling the dlls:

00007F85D2FFA3A0 00007f86746a09f3 [HelperMethodFrame_1OBJ: 00007f85d2ffa3a0] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F85D2FFA4D0 00007F85FE7F08B0 NLog.Targets.Wrappers.ConcurrentRequestQueue.WaitForBelowRequestLimit()
00007F85D2FFA500 00007F85FCF5B2AF NLog.Targets.Wrappers.ConcurrentRequestQueue.Enqueue(NLog.Common.AsyncLogEventInfo)
00007F85D2FFA570 00007F85FCF55E96 NLog.Targets.Wrappers.AsyncTargetWrapper.Write(NLog.Common.AsyncLogEventInfo)
00007F85D2FFA5B0 00007F85FCF55DC2 NLog.Targets.Wrappers.AsyncTargetWrapper.WriteAsyncThreadSafe(NLog.Common.AsyncLogEventInfo)
00007F85D2FFA5F0 00007F85FCF559C5 NLog.Targets.Target.WriteAsyncLogEvent(NLog.Common.AsyncLogEventInfo)
00007F85D2FFA660 00007F85FCF5537E NLog.LoggerImpl.WriteToTargetWithFilterChain(NLog.Targets.Target, NLog.Filters.FilterResult, NLog.LogEventInfo, NLog.Common.AsyncContinuation)
00007F85D2FFA690 00007F85FCF54AFC NLog.LoggerImpl.Write(System.Type, NLog.Internal.TargetWithFilterChain, NLog.LogEventInfo, NLog.LogFactory)

I think the dotnet-symbols issue may still be a valid bug but at this time we're able to setup dotnet-dump correctly.

Mike McLaughlin · Answer 3 · Fri Oct 11 2019 02:03:35 GMT+0800 (China Standard Time)

FYI, if you want source/line number info, you also need to copy the PDBs for your assemblies.

Mike McLaughlin · Answer 4 · Fri Oct 11 2019 02:04:59 GMT+0800 (China Standard Time)

Can you send/attach the text output of readelf -a <coredump>? It shouldn't contain any sensitive info and may help me diagnose the dotnet-symbol problem.

Nicholas Johns · Answer 5 · Fri Oct 11 2019 03:27:10 GMT+0800 (China Standard Time)

readelf: Warning: [ 0]: Unexpected value (20860) in info field.

readelf.log

Mike McLaughlin · Answer 6 · Fri Oct 11 2019 07:54:41 GMT+0800 (China Standard Time)

FYI, it looks like (even though readelf gave that warning) that your coredump is valid but has a lot of program headers (20860). dotnet-symbol has a check that limits them to 10,000. I'll increase that limit sometime. It was pretty arbitrary.

I also have some ideas why createdump is adding so many "LOAD" program headers too: it basically adding a header for invalid/non-memory-backed regions and it probably shouldn't. I already had fixed this but haven't merged it yet because it could be risky so it will go into 5.0 (the coreclr master branch).

Nicholas Johns · Answer 7 · Fri Oct 11 2019 21:08:31 GMT+0800 (China Standard Time)

Sounds great, thanks so much for looking into it 👍

Mikhail Khalizev · Answer 8 · Sun Apr 30 2023 22:09:47 GMT+0800 (China Standard Time)

Hi.
I have the same problem:

# dotnet-symbol core-gdb
Downloading from https://msdl.microsoft.com/download/symbols/
ERROR: Unreasonably large ELFHeader ProgramHeaderCount

readelf output shows 45285 headers:
readelf.zip