crash on Android 8.1.0
August1996 opened this issue · comments
Describe the bug
some devices are works, but some not.
crash info
bugreport files
crash detail info refer to file tombstone_00
device info
https://www.tme.net/device/vivo-x20a/
Android 8.1.0
there are some logs, crash happens when i rendering complex glb model.
2022-04-13 17:10:27.546 28266-28350/com.tme.k3d W/Adreno-GSL: <ioctl_kgsl_syncobj_create:3977>: (37, 19, 4233) fail 22 Invalid argument
2022-04-13 17:10:27.561 28266-28350/com.tme.k3d W/Adreno-GSL: <gsl_ldd_control:541>: ioctl fd 54 code 0xc0200933 (IOCTL_KGSL_TIMESTAMP_EVENT) failed: errno 22 Invalid argument
2022-04-13 17:10:27.561 28266-28350/com.tme.k3d W/Adreno-GSL: <ioctl_kgsl_syncobj_create:3977>: (37, 19, 4233) fail 22 Invalid argument
2022-04-13 17:10:27.564 28266-28350/com.tme.k3d W/Adreno-GSL: <gsl_ldd_control:541>: ioctl fd 54 code 0xc0200933 (IOCTL_KGSL_TIMESTAMP_EVENT) failed: errno 22 Invalid argument
2022-04-13 17:10:27.564 28266-28350/com.tme.k3d W/Adreno-GSL: <ioctl_kgsl_syncobj_create:3977>: (37, 19, 4233) fail 22 Invalid argument
i found that most of devices under api level 29 will carsh.
i have try many versions, and found that the problem first appears on v1.18.0.
As you can see in the tombstone, this is not a crash in filament. It's a crash in libhwui:
#00 pc 000000000001e728 /system/lib64/libc.so (abort+120)
#01 pc 0000000000008348 /system/lib64/liblog.so (__android_log_assert+296)
#02 pc 00000000000469f8 /system/lib64/libhwui.so (android::uirenderer::debug::GlesErrorCheckWrapper::assertNoErrors(char const*)+384)
#03 pc 0000000000088af0 /system/lib64/libhwui.so (android::uirenderer::Caches::flush(android::uirenderer::Caches::FlushMode)+164)
#04 pc 000000000007482c /system/lib64/libhwui.so (android::uirenderer::renderthread::Bridge_destroyHardwareResources(android::uirenderer::renderthread::destroyHardwareResourcesArgs*)+12)
#05 pc 0000000000075bac /system/lib64/libhwui.so (android::uirenderer::renderthread::MethodInvokeRenderTask::run()+24)
#06 pc 0000000000075f48 /system/lib64/libhwui.so (android::uirenderer::renderthread::SignalingRenderTask::run()+28)
It is in fact an assertion:
Abort message: 'glFinish error! GL_INVALID_VALUE (0x501)'
I don't know what is causing this, but it's not related to filament.
As you can see in the tombstone, this is not a crash in filament. It's a crash in libhwui:
#00 pc 000000000001e728 /system/lib64/libc.so (abort+120) #01 pc 0000000000008348 /system/lib64/liblog.so (__android_log_assert+296) #02 pc 00000000000469f8 /system/lib64/libhwui.so (android::uirenderer::debug::GlesErrorCheckWrapper::assertNoErrors(char const*)+384) #03 pc 0000000000088af0 /system/lib64/libhwui.so (android::uirenderer::Caches::flush(android::uirenderer::Caches::FlushMode)+164) #04 pc 000000000007482c /system/lib64/libhwui.so (android::uirenderer::renderthread::Bridge_destroyHardwareResources(android::uirenderer::renderthread::destroyHardwareResourcesArgs*)+12) #05 pc 0000000000075bac /system/lib64/libhwui.so (android::uirenderer::renderthread::MethodInvokeRenderTask::run()+24) #06 pc 0000000000075f48 /system/lib64/libhwui.so (android::uirenderer::renderthread::SignalingRenderTask::run()+28)It is in fact an assertion:
Abort message: 'glFinish error! GL_INVALID_VALUE (0x501)'I don't know what is causing this, but it's not related to filament.
very strangely, this is an inevitable phenomenon on some devices.
Not actionable on our end unfortunately 😕
There are many devices crash on Android 8.x. And i found that their GPU are Adreno 5.x. May be it is a gpu specific problem.
I upload some bugreports which may help you.
@romainguy @pixelflinger
bugreport-PD1709-OPM1.171019.011-2022-05-16-20-38-55.zip
bugreport-LDN-AL00-HUAWEILDN-AL00-2022-05-16-20-43-00.zip
bugreport-ARE-AL00-HONORARE-AL00-2022-05-16-20-53-40.zip
There are many devices crash on Android 8.x. And i found that their GPU are Adreno 5.x. May be it is a gpu specific problem. I upload some bugreports which may help you. @romainguy @pixelflinger
bugreport-PD1709-OPM1.171019.011-2022-05-16-20-38-55.zip bugreport-LDN-AL00-HUAWEILDN-AL00-2022-05-16-20-43-00.zip bugreport-ARE-AL00-HONORARE-AL00-2022-05-16-20-53-40.zip
Both of OpenGL and Vulkan are crashs.
@August1996 did you see the crash happening with our GLTF viewer sample or is this with your application?
@August1996 did you see the crash happening with our GLTF viewer sample or is this with your application?
Both of them will crash, this is the crash info for sample-gltf-viewer which i replace the model.
the official model in the apk is fine, so i guess that it may be related to some render features.
@August1996 did you see the crash happening with our GLTF viewer sample or is this with your application?
bugreport-ARE-AL00-HONORARE-AL00-2022-05-17-14-41-58.zip
@pixelflinger tombstone_04 and tombstone_05 are related to sample-gltf-viewer.
@pixelflinger @romainguy What is the android version supported by Filament？
I have the same problem as you, have you solved it now?
the official model in the apk is fine，the crash info for sample-gltf-viewer which i replace my model.
i have try many versions, and found that the problem first appears on v1.18.0.
thanks, that's very useful.
@August1996 how frequent is this crash on devices where it happens? Is it immediate, do you have to start/stop the app, or just wait?
@August1996 how frequent is this crash on devices where it happens? Is it immediate, do you have to start/stop the app, or just wait?
It crashs immediate with some models, i have test some devices.
| Version | Result | GPU |
|---|---|---|
| 8.1.0 | fine | Mali-G51 |
| 8.1.0 | fine | Mali-G51 |
| 8.1.0 | crash | Adreno509 |
| 8.0.0 | fine | Mali-T830 MP2 |
| 8.0.0 | crash | Adreno 430 |
| 8.0.0 | crash | Adreno 505 |
| 8.1.0 | crash | Adreno 512 |
so, i guess that some features of shaders may result in this crash on some old Adreno GPUs.
I know it's a lot to ask, but would you be able to conduct a git bisect between v1.17.0 and v1.18.0?
I cannot reproduce the problem here because I don't have access to a device that shows the problem.
It's possible that one of our change triggers a memory corruption-type bug in older Adreno drivers. In any case, if we can narrow it down, we might be able to at least have a workaround.
I know it's a lot to ask, but would you be able to conduct a
git bisectbetween v1.17.0 and v1.18.0?I cannot reproduce the problem here because I don't have access to a device that shows the problem.
It's possible that one of our change triggers a memory corruption-type bug in older Adreno drivers. In any case, if we can narrow it down, we might be able to at least have a workaround.
@pixelflinger I executed git bisect start cdadb43e501dd9f6af7f6862afe70f58f713519d b19a73cc50fc69e109b178a8bb31cf873c6aafec and mark bad or good commit. so this crash my be from this commit.
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[5417d0953b157f0e60ab27ea9de9a2102eefc63e] Allow screen space refraction and reflection together
Fantastic, thank you.
hello
the crash info for the official sample-gltf-viewer demo
vivo X9s Plus
OS: Android 7.1.1
Pixel: 1920x1080
Display: 5.87
2022/6/13 14:32:4914577F*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2022/6/13 14:32:4914577FBuild fingerprint: 'vivo/PD1635/PD1635:7.1.1/NMF26F/compil10251828:user/release-keys'
2022/6/13 14:32:4914577FRevision: '0'
2022/6/13 14:32:4914577FABI: 'arm64
'2022/6/13 14:32:4914577Fpid: 14527, tid: 14548, name: FEngine::loop >>> com.google.android.filament.gltf <<<
2022/6/13 14:32:49 14577 F signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x44
2022/6/13 14:32:49 14577 F x0 0000000000000bd0 x1 0000007f68ddeb40 x2 0000000000000001 x3 0000007f6411f000
2022/6/13 14:32:49 14577 F x4 0000000000000000 x5 0000007f77433000 x6 0000000000000000 x7 0000000000000004
2022/6/13 14:32:49 14577 F x8 0000000000000000 x9 0000000000000004 x10 0000000000000001 x11 0000007f64bb0a50
2022/6/13 14:32:49 14577 F x12 0000000000000000 x13 0000007f557b96a0 x14 0000000000000000 x15 0000000000000020
2022/6/13 14:32:49 14577 F x16 0000000000000030 x17 0000000000000038 x18 000000000000ffff x19 0000000000000010
2022/6/13 14:32:49 14577 F x20 0000007f77901800 x21 0000000000000001 x22 0000000000000000 x23 0000000000000000
2022/6/13 14:32:49 14577 F x24 0000007f68ddeb40 x25 0000000000000000 x26 0000000000000000 x27 0000000000000847
2022/6/13 14:32:49 14577 F x28 0000000000000001 x29 0000007f67307870 x30 0000007f77292894
2022/6/13 14:32:49 14577 F sp 0000007f67307870 pc 0000007f772a3a58 pstate 0000000060000000
2022/6/13 14:32:49 14577 F backtrace:
2022/6/13 14:32:49 14577 F x0 0000000000000bd0 x1 0000007f68ddeb40 x2 0000000000000001 x3 0000007f6411f000
2022/6/13 14:32:49 14577 F x4 0000000000000000 x5 0000007f77433000 x6 0000000000000000 x7 0000000000000004
2022/6/13 14:32:49 14577 F x8 0000000000000000 x9 0000000000000004 x10 0000000000000001 x11 0000007f64bb0a50
2022/6/13 14:32:49 14577 F x12 0000000000000000 x13 0000007f557b96a0 x14 0000000000000000 x15 0000000000000020
2022/6/13 14:32:4914577 F x16 0000000000000030 x17 0000000000000038 x18 000000000000ffff x19 0000000000000010
2022/6/13 14:32:4914577 F x20 0000007f77901800 x21 0000000000000001 x22 0000000000000000 x23 0000000000000000
2022/6/13 14:32:4914577 F x24 0000007f68ddeb40 x25 0000000000000000 x26 0000000000000000 x27 0000000000000847
2022/6/13 14:32:4914577 F x28 0000000000000001 x29 0000007f67307870 x30 0000007f77292894
2022/6/13 14:32:4914577 F sp 0000007f67307870 pc 0000007f772a3a58 pstate 0000000060000000
2022/6/13 14:32:4914577Fbacktrace:
2022/6/13 14:32:4914577F#00 pc 00000000001eaa58 /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN9EsxGfxMem15UpdateTimestampEPK10EsxContext13EsxAccessTypeP20EsxBucketIdReference+472)
2022/6/13 14:32:4914577F#01 pc 00000000001d9890 /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN9EsxCmdMgr16GfxMemReferencedEP9EsxGfxMem13EsxAccessType+48)
2022/6/13 14:32:4914577F#02 pc 00000000002cfd3c /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZNK10A5xProgram25GenerateShaderCBSlotStateEP10A5xContext13EsxCmdBufType23EsxCompiledHwShaderTypejP20A5xShaderCBSlotState+1084)
2022/6/13 14:32:49 14577 F #03 pc 00000000002cda08 /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN11A5xPipeline13WriteUboStateEP10A5xContext13EsxCmdBufType23EsxCompiledHwShaderTypejP14A5xStateBuffer+168)
2022/6/13 14:32:49 14577 F #04 pc 00000000002f2664 /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10A5xContext13ValidateStateEPK17EsxDrawDescriptor+2148)
2022/6/13 14:32:49 14577 F #05 pc 00000000002f2d30 /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10A5xContext18HwValidateGfxStateEPK17EsxDrawDescriptor+16)
2022/6/13 14:32:49 14577 F #06 pc 00000000001364a0 /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10EsxContext16ValidateGfxStateEPK17EsxDrawDescriptor+768)
2022/6/13 14:32:49 14577 F #07 pc 0000000000128bdc /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10EsxContext21DrawElementsInstancedE11EsxPrimTypej10EsxPixTypePKvji+444)
2022/6/13 14:32:49 14577 F #08 pc 0000000000100118 /system/vendor/lib64/egl/libGLESv2_adreno.so (glDrawRangeElements+88)
2022/6/13 14:32:49 14577 F #09 pc 0000000000115d38 /data/app/com.google.android.filament.gltf-1/lib/arm64/libfilament-jni.so
2022/6/13 14:32:49 4030 E vivo-add txpackets:39535 , tx_failed:31 , , tx_retry:6929 , rx_packets:2724063
2022/6/13 14:32:50 511 E Error writing /proc/14527/oom_score_adj; errno=
222022/6/13 14:32:50 2562 E Open /sys/touchscreen/app_name failed
2022/6/13 14:32:50 4441 E monitor view is null, no need to update fullscreen
2022/6/13 14:32:51 5514 E Unable to decode stream: java.io.FileNotFoundException: /oem/etc/theme/icons/dynamic_icon/com.vivo.weather/res/drawable-sw360dp-xxhdpi/dynamic_weather_cloudy.png (No such file or directory)
2022/6/13 14:32:51 5514 E Unable to decode stream: java.io.FileNotFoundException: /oem/etc/theme/icons/dynamic_icon/com.vivo.weather/res/drawable-sw360dp-xxhdpi/dynamic_weather_degree.png (No such file or directory)
2022/6/13 14:32:51 5514 E Unable to decode stream: java.io.FileNotFoundException: /oem/etc/theme/icons/dynamic_icon/com.vivo.weather/res/drawable-sw360dp-xxhdpi/dynamic_weather_minus.png (No such file or directory)
2022/6/13 14:32:52 4030 E vivo-add txpackets:39535 , tx_failed:31 , , tx_retry:6929 , rx_packets:27240632022/6/13 14:32:554577EgetConfigForPhoneId subId is invalid
@August1996 so you can confirm that you don't see the crash at the change immediately before 5417d0953b157f0e60ab27ea9de9a2102eefc63e? (but you do see it this commit).
That's so very strange, this commit should affect only screen-space refraction/reflections.
@pixelflinger
Has this crash found the cause of the problem? thank you！
I'm seeing a similar stack trace for my own app, but I'm on a two year old fork of Filament (version 1.8 I think):
06-22 14:15:01.907 11562 11562 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
06-22 14:15:01.907 11562 11562 F DEBUG : Build fingerprint: 'Android/msm8952_64/msm8952_64:7.1.1/NMF26F/sim3sc04211953:user/release-keys'
06-22 14:15:01.907 11562 11562 F DEBUG : Revision: '0'
06-22 14:15:01.907 11562 11562 F DEBUG : ABI: 'arm'
06-22 14:15:01.908 11562 11562 F DEBUG : pid: 11398, tid: 11453, name: FEngine::loop >>> avn.app.hubs <<<
06-22 14:15:01.908 11562 11562 F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x60
06-22 14:15:01.908 11562 11562 F DEBUG : r0 00000020 r1 00000004 r2 80000000 r3 00000000
06-22 14:15:01.908 11562 11562 F DEBUG : r4 c3882fc8 r5 00000010 r6 00000002 r7 00000001
06-22 14:15:01.908 11562 11562 F DEBUG : r8 00000001 r9 c188f220 sl 000014ee fp 00000000
06-22 14:15:01.908 11562 11562 F DEBUG : ip 00000058 sp c3882fb8 lr 00000000 pc db19c710 cpsr 000f0030
06-22 14:15:01.910 11562 11562 F DEBUG :
06-22 14:15:01.910 11562 11562 F DEBUG : backtrace:
06-22 14:15:01.911 11562 11562 F DEBUG : #00 pc 0016d710 /system/vendor/lib/egl/libGLESv2_adreno.so (_ZN9EsxGfxMem15UpdateTimestampEPK10EsxContext13EsxAccessTypeP20EsxBucketIdReference+335)
06-22 14:15:01.911 11562 11562 F DEBUG : #01 pc 0015e3f9 /system/vendor/lib/egl/libGLESv2_adreno.so (_ZN9EsxCmdMgr16GfxMemReferencedEP9EsxGfxMem13EsxAccessType+24)
06-22 14:15:01.911 11562 11562 F DEBUG : #02 pc 0021a115 /system/vendor/lib/egl/libGLESv2_adreno.so (_ZNK10A5xProgram25GenerateShaderCBSlotStateEP10A5xContext13EsxCmdBufType23EsxCompiledHwShaderTypejP20A5xShaderCBSlotState+788)
06-22 14:15:01.911 11562 11562 F DEBUG : #03 pc 0021876b /system/vendor/lib/egl/libGLESv2_adreno.so (_ZN11A5xPipeline13WriteUboStateEP10A5xContext13EsxCmdBufType23EsxCompiledHwShaderTypejP14A5xStateBuffer+106)
06-22 14:15:01.911 11562 11562 F DEBUG : #04 pc 00225fbb /system/vendor/lib/egl/libGLESv2_adreno.so (_ZN10A5xContext29WriteConstantBufferStateGroupE13EsxCmdBufTypej+26)
06-22 14:15:01.911 11562 11562 F DEBUG : #05 pc 00235e51 /system/vendor/lib/egl/libGLESv2_adreno.so (_ZN10A5xContext13ValidateStateEPK17EsxDrawDescriptor+1968)
06-22 14:15:01.911 11562 11562 F DEBUG : #06 pc 00236325 /system/vendor/lib/egl/libGLESv2_adreno.so (_ZN10A5xContext18HwValidateGfxStateEPK17EsxDrawDescriptor+4)
06-22 14:15:01.911 11562 11562 F DEBUG : #07 pc 000e4037 /system/vendor/lib/egl/libGLESv2_adreno.so (_ZN10EsxContext16ValidateGfxStateEPK17EsxDrawDescriptor+758)
06-22 14:15:01.911 11562 11562 F DEBUG : #08 pc 000d9413 /system/vendor/lib/egl/libGLESv2_adreno.so (_ZN10EsxContext21DrawElementsInstancedE11EsxPrimTypej10EsxPixTypePKvji+338)
06-22 14:15:01.911 11562 11562 F DEBUG : #09 pc 000bb3ad /system/vendor/lib/egl/libGLESv2_adreno.so (glDrawRangeElements+44)
06-22 14:15:01.911 11562 11562 F DEBUG : #10 pc 000f7629 /data/app/avn.app.hubs-1/lib/arm/libfilament-jni.so
06-22 14:15:02.966 4051 4051 D wpa_supplicant: wlan0: Control interface command 'SIGNAL_POLL'
06-22 14:15:02.984 4051 4051 D wpa_supplicant: CTRL-DEBUG: global_ctrl_sock-sendto: sock=8 sndbuf=212992 outq=0 send_len=49
06-22 14:15:03.046 11398 11426 I HUB : Assigning geometry for Masts2 0 from node 76
06-22 14:15:03.046 11398 11398 I Choreographer: Skipped 72 frames! The application may be doing too much work on its main thread.
It only happens when I load certain GLTF models, but I'm not using the Filament GLTF library.
I've narrowed down the cause of my crash to models with bones (like the "complex model" referenced above) and found that it goes away if I drop CONFIG_MAX_BONE_COUNT from 256 to 64. I'm still trying to work out why though, but the most obvious culprit is still some sort of driver bug.
hello，I changed to constexpr size_t CONFIG_MAX_BONE_COUNT = 64;
Using the official filament demo in vivo x9s plus android 7.1.1 still crashes.
the official filament-1.22.1 version
hello，I changed to constexpr size_t CONFIG_MAX_BONE_COUNT = 64; Using the official filament demo in vivo x9s plus android 7.1.1 still crashes.
the official filament-1.22.1 version
While you are there it might be worth trying disabling skinning altogether just to see if it's a factor.
hello，I changed to constexpr size_t CONFIG_MAX_BONE_COUNT = 64; Using the official filament demo in vivo x9s plus android 7.1.1 still crashes.
the official filament-1.22.1 versionWhile you are there it might be worth trying disabling skinning altogether just to see if it's a factor.
How can I try to modify it?
Either modify the glTF viewer to ignore bones during import or hack the skin definition out of the glTF file itself. It might also be worth trying an even lower number for CONFIG_MAX_BONE_COUNT.
Ultimately if you keep hacking things out of the code you will find the cause. It's a pretty brutal way to debug, but graphics debugging is very opaque.
@hzzengxiaoqi
I used the filament demo in 1.22 to verify no problem on the crashed phone, thanks a lot!
@pixelflinger My phone is Xiaomi Mix2 Android 8.0.0, with adreno 540, This crash maybe relates to Skinning, I remove all Skinning from my models, crash disappears.
And i try to render a very simple Skinning sample from KhronosGroup official, it crashs too.
So, i think that you can find a phone on Android 8.x with adreno gpu, and renders a model with Skinning may reproduces the crash.
hello，I changed to constexpr size_t CONFIG_MAX_BONE_COUNT = 64; Using the official filament demo in vivo x9s plus android 7.1.1 still crashes.
the official filament-1.22.1 versionWhile you are there it might be worth trying disabling skinning altogether just to see if it's a factor.
I try to render a very simple Skinning sample from KhronosGroup official, it crashs too. And crash disappears when i remove all Skinning from my models. So it may relates to Skinning.
hello，I changed to constexpr size_t CONFIG_MAX_BONE_COUNT = 64; Using the official filament demo in vivo x9s plus android 7.1.1 still crashes.
the official filament-1.22.1 versionWhile you are there it might be worth trying disabling skinning altogether just to see if it's a factor.
I try to render a very simple Skinning sample from KhronosGroup official, it crashs too. And crash disappears when i remove all
Skinningfrom my models. So it may relates toSkinning.
You can modify it to fix the problem code submit：#5754
hello，I changed to constexpr size_t CONFIG_MAX_BONE_COUNT = 64; Using the official filament demo in vivo x9s plus android 7.1.1 still crashes.
the official filament-1.22.1 versionWhile you are there it might be worth trying disabling skinning altogether just to see if it's a factor.
I try to render a very simple Skinning sample from KhronosGroup official, it crashs too. And crash disappears when i remove all
Skinningfrom my models. So it may relates toSkinning.You can modify it to fix the problem code submit：#5754
amazing!!!.it works!!!!thanks!!!
Should be fixed by #5754