Cannot get backtrace (stack trace) on Android devices; does this lib support android?

Question

Cannot get backtrace (stack trace) on Android devices; does this lib support android?

fzyzcjy opened this issue 3 years ago · comments

Hi thanks for the lib! However, I cannot get backtrace (stack trace) on Android devices.

Thus I wonder does this lib support android? If so, could you please provide a working example? Thanks!

For example (ignore the long prefix - unrelated to the question)

hello I am an error
2021-10-13 19:17:00.873 8044-8167/? I/flutter: Stack backtrace:
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    0: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    1: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    2: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    3: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    4: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    5: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    6: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    7: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    8: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:    9: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:   10: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:   11: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:   12: <unknown>
2021-10-13 19:17:00.873 8044-8167/? I/flutter:   13: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   14: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   15: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   16: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   17: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   18: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   19: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   20: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   21: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   22: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   23: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   24: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   25: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   26: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   27: <unknown>
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   28: _ZL15__pthread_startPv
2021-10-13 19:17:00.874 8044-8167/? I/flutter:   29: __start_thread, null)

If I manually print out the frames:

2021-10-13 19:16:58.434 8044-8099/? I/vision_utils_rs::api: debug_throw start mode=RETURN_ERR
2021-10-13 19:16:58.435 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8aba6bda, symbol_address: 0x8aba6bda }
2021-10-13 19:16:58.436 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8aba6c1e, symbol_address: 0x8aba6c1e }
2021-10-13 19:16:58.437 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8ab9124a, symbol_address: 0x8ab9124a }
2021-10-13 19:16:58.439 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8abad984, symbol_address: 0x8abad984 }
2021-10-13 19:16:58.440 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8ab95b62, symbol_address: 0x8ab95b62 }
2021-10-13 19:16:58.441 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8aba5a4a, symbol_address: 0x8aba5a4a }
2021-10-13 19:16:58.442 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8ab5d138, symbol_address: 0x8ab5d138 }
2021-10-13 19:16:58.444 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0d6704, symbol_address: 0x8b0d6704 }
2021-10-13 19:16:58.445 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8abb15d4, symbol_address: 0x8abb15d4 }
2021-10-13 19:16:58.446 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8aba422a, symbol_address: 0x8aba422a }
2021-10-13 19:16:58.447 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8aba4586, symbol_address: 0x8aba4586 }
2021-10-13 19:16:58.448 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8aba3d16, symbol_address: 0x8aba3d16 }
2021-10-13 19:16:58.450 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8ab907da, symbol_address: 0x8ab907da }
2021-10-13 19:16:58.451 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8abb1972, symbol_address: 0x8abb1972 }
2021-10-13 19:16:58.452 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8abb147e, symbol_address: 0x8abb147e }
2021-10-13 19:16:58.453 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0e8bd2, symbol_address: 0x8b0e8bd2 }
2021-10-13 19:16:58.454 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0e10aa, symbol_address: 0x8b0e10aa }
2021-10-13 19:16:58.455 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0e6054, symbol_address: 0x8b0e6054 }
2021-10-13 19:16:58.456 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0e3022, symbol_address: 0x8b0e3022 }
2021-10-13 19:16:58.457 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0ea36c, symbol_address: 0x8b0ea36c }
2021-10-13 19:16:58.458 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0eaa7c, symbol_address: 0x8b0eaa7c }
2021-10-13 19:16:58.459 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0ea30c, symbol_address: 0x8b0ea30c }
2021-10-13 19:16:58.460 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0e304a, symbol_address: 0x8b0e304a }
2021-10-13 19:16:58.461 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0e5f6c, symbol_address: 0x8b0e5f6c }
2021-10-13 19:16:58.462 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b0e130a, symbol_address: 0x8b0e130a }
2021-10-13 19:16:58.463 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0x8b263d08, symbol_address: 0x8b263d08 }
2021-10-13 19:16:58.464 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0xb2a3f394, symbol_address: 0xb2a3f394 }
2021-10-13 19:16:58.464 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace resolve_frame symbol=Symbol { name: "_ZL15__pthread_startPv" }
2021-10-13 19:16:58.464 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace frame=Frame { ip: 0xb2a11abe, symbol_address: 0xb2a11abe }
2021-10-13 19:16:58.465 8044-8099/? I/vision_utils_rs::utils: capture_and_print_backtrace resolve_frame symbol=Symbol { name: "__start_thread" }

How I use it: cargo ndk to build into a .so file. Then call it by other language's ffi.

bjorn3 · Answer 1 · Wed Oct 13 2021 20:02:22 GMT+0800 (China Standard Time)

Looks like cargo ndk strips symbols from the compiled library: https://github.com/bbqsrc/cargo-ndk/blob/c9fe5acf7fac063f0d77ad08de0ff877b8abb299/src/cargo.rs#L115

Try removing the call to this function.

fzyzcjy · Answer 2 · Wed Oct 13 2021 20:08:07 GMT+0800 (China Standard Time)

No it is not striped. I use file myfile.so and it says not stripped. Indeed I have tweaked a bit for cargo ndk so it seems not to strip it. I can upload file for you to see:

fzyzcjy · Answer 3 · Wed Oct 13 2021 20:13:25 GMT+0800 (China Standard Time)

well github seems not to allow me to upload. but I can share command line output:

file libvision_utils_rs.so 
libvision_utils_rs.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_info, not strippe

Thanks for the comments all the same!

fzyzcjy · Answer 4 · Wed Oct 13 2021 20:15:32 GMT+0800 (China Standard Time)

By the way, I have tried to symbolize it by myself as well, but failed. I do not know where to see the starting address of the .so file - since 0x8aba6bda is too big to be a real address in the .so file (please correct me if I am wrong)

fzyzcjy · Answer 5 · Wed Oct 13 2021 20:25:17 GMT+0800 (China Standard Time)

So, I do not necessarily need a way to see it symbolized when printed. If there is a way to symbolize it on my computer afterwards, it is also completely ok! (And I think that is the only way when build in mobile production environment where everything is stripped).

Could you please provide some suggestions? Thanks!

fzyzcjy · Answer 6 · Wed Oct 13 2021 20:30:16 GMT+0800 (China Standard Time)

By the way, as for why it is not stripped: https://github.com/bbqsrc/cargo-ndk/blob/c9fe5acf7fac063f0d77ad08de0ff877b8abb299/src/cli.rs#L223 I just do not provide that command line argument, so the whole logic including strip do not execute.

bjorn3 · Answer 7 · Wed Oct 13 2021 20:44:00 GMT+0800 (China Standard Time)

#415 which was merged 6 months ago should have added android support for android sdk version 21+.

By the way, as for why it is not stripped: https://github.com/bbqsrc/cargo-ndk/blob/c9fe5acf7fac063f0d77ad08de0ff877b8abb299/src/cli.rs#L223 I just do not provide that command line argument, so the whole logic including strip do not execute.

I see.

Can someone more familiar with android have a look at why this doesn't work? @kjvalencik maybe?

fzyzcjy · Answer 8 · Wed Oct 13 2021 20:48:18 GMT+0800 (China Standard Time)

@bjorn3 @kjvalencik Thank you so much!

K.J. Valencik · Answer 9 · Wed Oct 13 2021 20:59:08 GMT+0800 (China Standard Time)

@fzyzcjy There are a couple of potential issues. NDK 21+ is required, but that's pretty old and likely not the cause.

The more likely cause is that newer versions of Android support loading shared libraries directly from the packaged app and this is now the default behavior since it saves disk space and improves startup time. However, gimli is not able to read symbols from this location.

Setting the app to extract the lib allows backtraces to function properly. android:extractNativeLibs="true"

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
          xmlns:tools="http://schemas.android.com/tools"
          package="com.myapp">

    <application
            tools:replace="android:extractNativeLibs"
            android:extractNativeLibs="true" />

</manifest>

fzyzcjy · Answer 10 · Wed Oct 13 2021 21:08:16 GMT+0800 (China Standard Time)

@kjvalencik

The more likely cause is that newer versions of Android support loading shared libraries directly from the packaged app and this is now the default behavior since it saves disk space and improves startup time. However, gimli is not able to read symbols from this location.

Seems yes! Let me have a look.

fzyzcjy · Answer 11 · Wed Oct 13 2021 21:13:54 GMT+0800 (China Standard Time)

@kjvalencik You are a genius! It works! Thank you soooooo much!
@bjorn3 Also thank you very much for your help!

fzyzcjy · Answer 12 · Wed Oct 13 2021 21:15:05 GMT+0800 (China Standard Time)

So, the last question is: How can I symbolize it when the .so file is stripped? (Probably get only address values in mobile phone, and then run symbolize tools on computers given the address values and unstripped files)

I guess I should use something like addr2line. But how can I provide the input? The huge address like 0x8aba6bda seems not to be a correct input. For example, here mentions lib starting addr but backtrace does not seem to provide that.

Alex Crichton · Answer 13 · Wed Oct 13 2021 23:03:20 GMT+0800 (China Standard Time)

The general answer to this question is yes, Android should be supported now (as discovered). This crate doesn't currently help with stripped libraries and you'll typically need to use other or external means to extract addresses that are appropriate to pass to addr2line later. Typically this means acquiring the offset of the dynamic library in memory, so the address can be adjusted to match what's located in debuginfo.

fzyzcjy · Answer 14 · Thu Oct 14 2021 06:24:48 GMT+0800 (China Standard Time)

Thank you very much!

Peter de Kraker · Answer 15 · Tue Feb 28 2023 04:52:59 GMT+0800 (China Standard Time)

I am unable to get a backtrace printed on Android. (Tested with Android 13 on Pixel 6). I tried the extractNativeLibs, but that doesn't solve the issue.
Using latest NDK (25.2.9519653)
Debugsymbols are there.

ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

 error!("{:?}", Backtrace::new());

E  panicked at 'called `Option::unwrap()` on a `None` value', src/file.rs:324:13
2023-02-27 21:41:48.471 15919-16013        E     0: <unknown>
1: <unknown>
2: <unknown>
3: <unknown>
[..]      
17: _ZL15__pthread_startPv
18: __start_thread

When I force a unwrap outside my threaded code in Rust, I get a partial stacktrace, but only the C/C++ part, and not the Rust part:

0: <unknown>
1: <unknown>
2: <unknown>
3: <unknown>
4: <unknown>
5: <unknown>
6: <unknown>
7: <unknown> 
8: Java_com_some_app_JNI_method (JNI entry into Rust)
9: art_quick_generic_jni_trampoline
10: art_quick_invoke_static_stub
11:_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE
12:_ZN3art11interpreter20ExecuteSwitchImplCppILb0ELb0EEEvPNS0_17SwitchImplContextE
13: ExecuteSwitchImplAsm
14:_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEbb.__uniq.112435418011751916792819755956732575238.llvm.15953120228497039992
15: artQuickToInterpreterBridge
16: art_quick_to_interpreter_bridge
17: nterp_helper
18: nterp_helper
19: nterp_helper
20: nterp_helper
21: nterp_helper
22: nterp_helper
23: nterp_helper
24: nterp_helper
25: art_quick_invoke_stub
26: _ZN3art9ArtMethod6InvokeEPNS_6ThreadEPjjPNS_6JValueEPKc
27:_ZN3art35InvokeVirtualOrInterfaceWithJValuesIPNS_9ArtMethodEEENS_6JValueERKNS_33ScopedObjectAccessAlreadyRunnableEP8_jobjectT_PK6jvalue
28: _ZN3art6Thread14CreateCallbackEPv
29: _ZL15__pthread_startPv
30: __start_thread

What can I do to get a trace properly?

Peter de Kraker · Answer 16 · Wed Mar 01 2023 02:21:50 GMT+0800 (China Standard Time)

It turns out the actual .so on my Android device was stripped. So it works indeed.

Workaround to test it was:

 packagingOptions{
        doNotStrip "**/*.so"
    }

However, unfortunately Bugsnag NDK crash handling does not symbolicate remotely when the doNotStrip is removed. So I am unable to get stacktraces in production code.