minidump-stackwalk support windows sym file?

Question

minidump-stackwalk support windows sym file?

anranziruo opened this issue a year ago · comments

my sym file example just like this

MODULE windows x86_64 422EF214A6564D9BB0DE413F3592229033 AstrolabeDemo.pdb
INFO CODE_ID 6350F9CD25000 AstrolabeDemo.exe
FILE 1 C:\Qt\Qt5.12.12\5.12.12\msvc2017_64\include\QtCore\qiodevice.h

when i use minidump-stackwalk --symbols-path=apm_symbols/ 111.dmp

apm_symbols dir :

AstrolabeDemo.pdb/422EF214A6564D9BB0DE413F3592229033/AstrolabeDemo.sym

but the result is not symbol success,the result just like this,

0  AstrolabeDemo.exe + 0x33d2
     rax = 0x0000000000000000    rdx = 0x00007ff6227bdedf
     rcx = 0x000000611e0ff680    rbx = 0x0000000000000013
     rsi = 0x000000611e0fa390    rdi = 0x0000000000000000
     rbp = 0x000000611e0ff680    rsp = 0x000000611e0fa218
      r8 = 0x00007ff6227b0000     r9 = 0x000000611e0fa390
     r10 = 0x0000000000000000    r11 = 0x000000611e0fa320
     r12 = 0x000000611e0ff680    r13 = 0x000000611e0fa390
     r14 = 0x0000000000000009    r15 = 0x00000186d39f03d0
     rip = 0x00007ff6227b33d2
    Found by: given as instruction pointer in context

Gabriele Svelto · Answer 1 · Fri Jan 20 2023 19:47:33 GMT+0800 (China Standard Time)

How did you generate the 111.dmp file? Keep in mind that whenever you rebuild your executable you also need to dump its symbols again, otherwise the debug ID will be different between the executable and the symbol file.

Gabriele Svelto · Answer 2 · Fri Jan 20 2023 19:49:51 GMT+0800 (China Standard Time)

To double-check if your .sym file matches what's in the minidump you can run minidump-stackwalk --dump 111.dmp and check the output. It should have a MINIDUMP_MODULE entry where the (debug_file) is the name of the PDB of your executable and (debug_identifier) is 422EF214A6564D9BB0DE413F3592229033. If the (debug_identifier) has a different value then it means that you've modified the executable after dumping the symbols and the two don't match anymore.

Alcestis Miller · Answer 3 · Fri Jan 20 2023 20:39:06 GMT+0800 (China Standard Time)

To double-check if your .sym file matches what's in the minidump you can run minidump-stackwalk --dump 111.dmp and check the output. It should have a MINIDUMP_MODULE entry where the (debug_file) is the name of the PDB of your executable and (debug_identifier) is 422EF214A6564D9BB0DE413F3592229033. If the (debug_identifier) has a different value then it means that you've modified the executable after dumping the symbols and the two don't match anymore.

(debug_file) = "C:\astrolabe2\astrolabe-demo\bin\release\AstrolabeDemo.pdb"
(debug_identifier) = "422ef214-a656-4d9b-b0de-413f35922290-33"
i find the error

TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=1 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=1 fname=""}: couldn't find symbols for address, cannot use cfi

the sym file content eg is

MODULE windows x86_64 422EF214A6564D9BB0DE413F3592229033 AstrolabeDemo.pdb
FILE 1 C:\Qt\Qt5.12.12\5.12.12\msvc2017_64\include\QtCore\qiodevice.h

Gabriele Svelto · Answer 4 · Fri Jan 20 2023 21:16:49 GMT+0800 (China Standard Time)

I see what's going on, the stackwalker finds the symbol file but there's no unwinding tables nor symbols inside the symbol file. How did you generate the PDB and what tool did you use to create the .sym file? If you're using our implementation of dump_syms you need to call it on the executable and it should find the PDB on his own. Given both files you should have a symbol file with both the symbols (PUBLIC and FUNC entries) and the unwinding directives (CFI INIT). Try re-generating your symbol file and walking the stack again.

nerix · Answer 5 · Fri Jan 27 2023 03:36:37 GMT+0800 (China Standard Time)

I think I have a similar issue. The (debug_identifier) match in the symbol file and minidump.

I generated the symbol file like this:

dump_syms chatterino.exe -o chatterino.sym

Then ran minidump-stackwalk like this:

minidump-stackwalk db252ac3-eb1c-4dfd-9baa-5a65cd2a0bcb.dmp --symbols-path chatterino.sym --verbose trace

Start of output

 WARN failed to pretty print instruction: failed to read memory at instruction pointer
 WARN failed to get instruction memory access: failed to read memory at instruction pointer
TRACE unwind_thread{idx=0 tid=8968 tname=""}: starting stack unwind of thread 8968
TRACE unwind_thread{idx=0 tid=8968 tname=""}: locating symbols for module G:\CLionProjects\c2-setup\build\bin\chatterino.exe
TRACE unwind_thread{idx=0 tid=8968 tname=""}:symbols{module="chatterino.exe"}:locate_file{file_kind=BreakpadSym module="chatterino.exe"}: SimpleSymbolSupplier search
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007ff6134521ba
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=0 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=0 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=0 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=0 fname=""}: scan seems valid -- caller_ip: 0x7ff613c97250, caller_sp: 0x3b93961240
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007ff613c9724f
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=1 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=1 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=1 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=1 fname=""}: scan seems valid -- caller_ip: 0x7ff613c972a0, caller_sp: 0x3b939612a8
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007ff613c9729f
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=2 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=2 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=2 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=2 fname=""}: scan seems valid -- caller_ip: 0x7ff613c97250, caller_sp: 0x3b939612b0
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007ff613c9724f
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: locating symbols for module G:\CLionProjects\c2-setup\build\bin\Qt5Cored.dll
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}:symbols{module="Qt5Cored.dll"}:locate_file{file_kind=BreakpadSym module="Qt5Cored.dll"}: SimpleSymbolSupplier search
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: scan seems valid -- caller_ip: 0x7fffadd97128, caller_sp: 0x3b939612b8
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007fffadd97127
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=4 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=4 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=4 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=4 fname=""}: scan seems valid -- caller_ip: 0x7ff6138580bb, caller_sp: 0x3b939613b0
...

This zip contains the generated symbol file (about 50mb) and minidump (if that helps): chatterino.zip. The minidump was generated by crashpad.

Alcestis Miller · Answer 6 · Sun Jan 29 2023 11:47:40 GMT+0800 (China Standard Time)

I think I have a similar issue. The (debug_identifier) match in the symbol file and minidump.

I generated the symbol file like this:

dump_syms chatterino.exe -o chatterino.sym

Then ran minidump-stackwalk like this:

minidump-stackwalk db252ac3-eb1c-4dfd-9baa-5a65cd2a0bcb.dmp --symbols-path chatterino.sym --verbose trace

Start of output

 WARN failed to pretty print instruction: failed to read memory at instruction pointer
 WARN failed to get instruction memory access: failed to read memory at instruction pointer
TRACE unwind_thread{idx=0 tid=8968 tname=""}: starting stack unwind of thread 8968
TRACE unwind_thread{idx=0 tid=8968 tname=""}: locating symbols for module G:\CLionProjects\c2-setup\build\bin\chatterino.exe
TRACE unwind_thread{idx=0 tid=8968 tname=""}:symbols{module="chatterino.exe"}:locate_file{file_kind=BreakpadSym module="chatterino.exe"}: SimpleSymbolSupplier search
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007ff6134521ba
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=0 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=0 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=0 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=0 fname=""}: scan seems valid -- caller_ip: 0x7ff613c97250, caller_sp: 0x3b93961240
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007ff613c9724f
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=1 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=1 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=1 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=1 fname=""}: scan seems valid -- caller_ip: 0x7ff613c972a0, caller_sp: 0x3b939612a8
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007ff613c9729f
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=2 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=2 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=2 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=2 fname=""}: scan seems valid -- caller_ip: 0x7ff613c97250, caller_sp: 0x3b939612b0
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007ff613c9724f
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: locating symbols for module G:\CLionProjects\c2-setup\build\bin\Qt5Cored.dll
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}:symbols{module="Qt5Cored.dll"}:locate_file{file_kind=BreakpadSym module="Qt5Cored.dll"}: SimpleSymbolSupplier search
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=3 fname=""}: scan seems valid -- caller_ip: 0x7fffadd97128, caller_sp: 0x3b939612b8
TRACE unwind_thread{idx=0 tid=8968 tname=""}: unwinding 0x00007fffadd97127
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=4 fname=""}: trying cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=4 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=4 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=8968 tname=""}:unwind_frame{idx=4 fname=""}: scan seems valid -- caller_ip: 0x7ff6138580bb, caller_sp: 0x3b939613b0
...

This zip contains the generated symbol file (about 50mb) and minidump (if that helps): chatterino.zip. The minidump was generated by crashpad.

your symbol path is not correct,
eg1 is the right demo

AstrolabeDemo.pdb/422EF214A6564D9BB0DE413F3592229033/AstrolabeDemo.sym
···

Alcestis Miller · Answer 7 · Sun Jan 29 2023 11:58:37 GMT+0800 (China Standard Time)

I see what's going on, the stackwalker finds the symbol file but there's no unwinding tables nor symbols inside the symbol file. How did you generate the PDB and what tool did you use to create the .sym file? If you're using our implementation of dump_syms you need to call it on the executable and it should find the PDB on his own. Given both files you should have a symbol file with both the symbols (PUBLIC and FUNC entries) and the unwinding directives (CFI INIT). Try re-generating your symbol file and walking the stack again.

The sym file have PUBLIC and FUNC，

when i use dump_syms get new sym file,when i use our

minidump-stackwalk --verbose=trace --symbols-path=symbols/ /Users/chao.zhang02/app/docker/minidump_stackwalker/eb4b7e49-a50b-4a73-a7f7-ebf5139168fa.dmp

But still hava error

TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=0 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=0 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=0 fname=""}: scan seems valid -- caller_ip: 0x7ff6227bddd1, caller_sp: 0x611e0fa220

When i use mozilla minidump-stackwalk ,it can get right stack,eg1 is

nerix · Answer 8 · Sun Jan 29 2023 20:56:33 GMT+0800 (China Standard Time)

your symbol path is not correct,
eg1 is the right demo

Thanks! I had to do the following:

dump_syms chatterino.exe -s <folder>
minidump-stackwalk db252ac3-eb1c-4dfd-9baa-5a65cd2a0bcb.dmp --symbols-path <folder>

I think the help from minidump-stackwalk might be misleading:

--symbols-path <SYMBOLS_PATH>
    Path to a symbol file

This suggests I have to supply a symbol file, but instead a path to a store is expected.

Alcestis Miller · Answer 9 · Mon Jan 30 2023 11:49:43 GMT+0800 (China Standard Time)

you can see this article,https://wiki.mozilla.org/Breakpad:Symbols#Flash_Player_symbols

Gabriele Svelto · Answer 10 · Mon Jan 30 2023 19:04:02 GMT+0800 (China Standard Time)

Ah yes, the symbols must be in a folder at the appropriate paths. One way to have dump_syms do this automatically is to use the --store option. You can even dump several files that way and they'll all be in the right place under the symbols directory. E.g.:

dump_syms --store symbols program.exe
minidump-stackwalk minidump.dmp symbols

Gabriele Svelto · Answer 11 · Mon Jan 30 2023 19:06:30 GMT+0800 (China Standard Time)

But still hava error

TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=0 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=0 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=0 fname=""}: scan seems valid -- caller_ip: 0x7ff6227bddd1, caller_sp: 0x611e0fa220

It should work but maybe you've hit some corner case we haven't considered. Without access to the minidump and symbol file I can't help you though.

Alcestis Miller · Answer 12 · Tue Feb 07 2023 22:20:13 GMT+0800 (China Standard Time)

But still hava error
TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=0 fname=""}: couldn't find symbols for address, cannot use cfi
TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=0 fname=""}: trying scan
TRACE unwind_thread{idx=0 tid=16896 tname=""}:unwind_frame{idx=0 fname=""}: scan seems valid -- caller_ip: 0x7ff6227bddd1, caller_sp: 0x611e0fa220
It should work but maybe you've hit some corner case we haven't considered. Without access to the minidump and symbol file I can't help you though.

sorry,this is our company project,but thank you for our advice,you help me to finish windows dmp symboling in linux.you can close this issue