SSE Error?
TaeHagen opened this issue · comments
Describe the bug
When adding a new function and building the bridge, the function in the file panics in rust when called. It works on android, panics on linux. I don't even know why DartAttachment was being deserialized in the backtrace, it should have nothing to do with this call at all. If I add it at certain points it will make the next function fail too. I think there might be a memory corruption vuln somewhere.
The function I added is:
pub async fn get_user_name(state: &Arc<PushState>) -> anyhow::Result<String> { let inner = state.0.read().await; let (first, last) = inner.account.as_ref().unwrap().get_name(); Ok(format!("{first} {last}")) }
Steps to reproduce
Honestly no clue how what the conditions are, this is almost an ask for (what the hell is going on?)
Logs
thread '<unnamed>' panicked at src/frb_generated.rs:1611:56:
called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }
stack backtrace:
0: 0x7f53763d1be6 - std::backtrace_rs::backtrace::libunwind::trace::hbee8a7973eeb6c93
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: 0x7f53763d1be6 - std::backtrace_rs::backtrace::trace_unsynchronized::hc8ac75eea3aa6899
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x7f53763d1be6 - std::sys_common::backtrace::_print_fmt::hc7f3e3b5298b1083
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
3: 0x7f53763d1be6 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbb235daedd7c6190
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
4: 0x7f53763fbb50 - core::fmt::rt::Argument::fmt::h76c38a80d925a410
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
5: 0x7f53763fbb50 - core::fmt::write::h3ed6aeaa977c8e45
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
6: 0x7f53763ced3f - std::io::Write::write_fmt::h78b18af5775fedb5
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
7: 0x7f53763d19c4 - std::sys_common::backtrace::_print::h5d645a07e0fcfdbb
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
8: 0x7f53763d19c4 - std::sys_common::backtrace::print::h85035a511aafe7a8
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
9: 0x7f53763d3097 - std::panicking::default_hook::{{closure}}::hcce8cea212785a25
10: 0x7f53763d2df9 - std::panicking::default_hook::hf5fcb0f213fe709a
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
11: 0x7f53763d3528 - std::panicking::rust_panic_with_hook::h095fccf1dc9379ee
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
12: 0x7f53763d3402 - std::panicking::begin_panic_handler::{{closure}}::h032ba12139b353db
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
13: 0x7f53763d20e6 - std::sys_common::backtrace::__rust_end_short_backtrace::h9259bc2ff8fd0f76
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
14: 0x7f53763d3160 - rust_begin_unwind
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
15: 0x7f5375986705 - core::panicking::panic_fmt::h784f20a50eaab275
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
16: 0x7f5375986c53 - core::result::unwrap_failed::h03d8a5018196e1cd
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
17: 0x7f5375ae7e8f - <alloc::vec::Vec<u8> as rust_lib_bluebubbles::frb_generated::SseDecode>::sse_decode::hcb8704b144073230
18: 0x7f5375a8cf6e - <rust_lib_bluebubbles::api::api::DartMMCSFile as rust_lib_bluebubbles::frb_generated::SseDecode>::sse_decode::h685f1135e30769d1
19: 0x7f5375a8b94b - <rust_lib_bluebubbles::api::api::DartAttachment as rust_lib_bluebubbles::frb_generated::SseDecode>::sse_decode::h9c8f77de8941c5f1
20: 0x7f5375a1d0f5 - std::panicking::try::h1957b085e343ff26
21: 0x7f5375ad3a48 - rust_lib_bluebubbles::frb_generated::pde_ffi_dispatcher_primary_impl::h394a799a15fd2536
22: 0x7f5386906abb - <unknown>Expected behavior
The function should call and work like normal. An unrelated function should not stop working when I add a completely different function
Generated binding code
No response
OS
Linux
Version of flutter_rust_bridge_codegen
2.0.0-dev.28
Flutter info
[✓] Flutter (Channel stable, 3.19.4, on Ubuntu 22.04.3 LTS 6.6.13-amd64, locale en_US.UTF-8)
• Flutter version 3.19.4 on channel stable at /home/tae/Documents/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 68bfaea224 (3 weeks ago), 2024-03-20 15:36:31 -0700
• Engine revision a5c24f538d
• Dart version 3.3.2
• DevTools version 2.31.1
[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
• Android SDK at /home/tae/Android/Sdk
✗ cmdline-tools component is missing
Run `path/to/sdkmanager --install "cmdline-tools;latest"`
See https://developer.android.com/studio/command-line for more details.
✗ Android license status unknown.
Run `flutter doctor --android-licenses` to accept the SDK licenses.
See https://flutter.dev/docs/get-started/install/linux#android-setup for more details.
[✗] Chrome - develop for the web (Cannot find Chrome executable at google-chrome)
! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[✓] Linux toolchain - develop for Linux desktop
• Ubuntu clang version 14.0.0-1ubuntu1.1
• cmake version 3.22.1
• ninja version 1.10.1
• pkg-config version 0.29.2
[✓] Android Studio (version 2023.2)
• Android Studio at /home/tae/Downloads/android-studio
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 17.0.9+0-17.0.9b1087.7-11185874)
[✓] VS Code (version 1.85.1)
• VS Code at /usr/share/code
• Flutter extension version 3.86.0
[✓] Connected device (1 available)
• Linux (desktop) • linux • linux-x64 • Ubuntu 22.04.3 LTS 6.6.13-amd64
[✓] Network resources
• All expected network resources are available.Version of clang++
14.0.0-1ubuntu1.1
Additional context
Running for linux
Hi, firstly, is it open source such that I can reproduce it? If not, here are some suggestions to debug it on your own:
- Try to put some prints here and there and see what is going on. For example,
DartAttachment's sse_decode function (usually in frb_generated.rs). - At pde_ffi_dispatcher_primary_impl, you can just print out the whole byte array to see the exact bytes.
In addition, if the app is not open sourceable, maybe you can just copy-paste the relavent parts (and remember to remove any sensitive information) and paste it here.
Btw, have you run codegen and ensure to use both the latest generated code on Dart and Rust side? My first guess is that the auto-generated routing logic somehow fails, e.g. on Dart side you put thing A with label 100, but on Rust side the label 100 somehow is understood as thing B.
Indeed the debugging strategy is very simple: Just follow the flow of a method call and see what is going on, and usually there will be a place when the expected behavior is different from actual behavior. There is no magic hidden somewhere, since everything is explicitly generated as plain Rust/Dart code, and you can add whatever printing/debugging/... code to it.
Thank you for the timely advice! The entire app is not ready to be open sourced so you won't be able to build it, but you can see the problematic file here: https://github.com/TaeHagen/bluebubbles-app/blob/rustpush/rust/src/api/api.rs
A big part of my problem has been lack of debugging tools. I am 100% sure the codegen was updated, as I tried many times. If you have any advice for debugging tools (note this problem only happens when I build for linux) it could be very helpful.
I'd like to debug pde_ffi_dispatcher_primary_impl and see why it is calling DartAttachment because that should be unrelated for the method I am calling (get_user_name). However, when I try to add a log (see below) nothing ends up printed. Am I adding it in the wrong spot?
Thank you!
A big part of my problem has been lack of debugging tools.
I personally just print here and there, and it is usually sufficient (at least in this specific case).
nothing ends up printed.
Maybe try log::error!(...), and it will print if the default hook is setup (which is true for auto-generated templates)
Btw, try to run in debug (not release) mode to see the line numbers and maybe some more sanity checks
I am running in debug mode. I tried log::error, nothing still is printed. I think I might be adding it in wrong spot?
Also I know macros are helpful, but in this case I think pde_ffi_dispatcher_primary_impl is called by a macro, so I'm not sure where to go from here.
I also considered building with ASAN, but not sure how to get flutter_rust_bridge to build with rust nightly
I also considered building with ASAN, but not sure how to get flutter_rust_bridge to build with rust nightly
I guess it may be a more normal problem than memory issues at first glance since you are using SSE codec. But in case you are interested, have a look at ci.yaml, where flutter_rust_bridge has full tests running using ASAN and Valgrind.
I am running in debug mode. I tried log::error, nothing still is printed. I think I might be adding it in wrong spot?
Maybe try to firstly log it in a good function to see whether the logs are shown.
EDIT: Seems that you are using https://github.com/TaeHagen/bluebubbles-app/blob/03a40bcfd0e35c37a496e3adaff9ab967b2c5ce5/rust/src/api/api.rs#L49 for logging setup. Maybe copy-paste
to let it auto called at startup to setup logging to terminal.In addition, it would be great to provide a compilable project (maybe just remove all your real logic by things like unimplemented!() and throw Exception()), then I can directly look at it!
Thank you! Also do you have any ideas why it would work fine on Android, but panic on Linux?
You are welcome!
Also do you have any ideas why it would work fine on Android, but panic on Linux?
Hmm good question! Indeed it can be anything, yes including memory issues you mentioned, or other things like platform specific code, or different ABI, etc. Anyway, providing a compilable project will help a lot.
Also I know macros are helpful, but in this case I think pde_ffi_dispatcher_primary_impl is called by a macro, so I'm not sure where to go from here.
Forgot to reply this... One way is to expand that macro to see real code (IDE has this feature; or use cargo expand or whatever), and then just remove the original macro call and paste the real code there. Then can add whatever logs.
I've been working around this for the past few days, but seems like the plague has spread to android too. It comes in multiple variants, one is:
(the size differs)
This appears to be triggered after i add a method. I will add a new method, rebuild, and then this crap will start happening
Wait, that one might have been caused by me hot reloading and not deploying... oops.
Wait, that one might have been caused by me hot reloading and not deploying... oops.
Yes, Rust requires recompiling currently...
As for this specific point, maybe we can make a very simple check: e.g. generate a dart const and a rust const (random number, content hash, etc), and when RustLib.init, ensure the two consts agree with each other.
I've been working around this for the past few days
Sorry to hear that :/ Personally speaking this may not be super hard to spot at least the shallow cause... Just follow the flow of data and see which exact part goes wrong (e.g. suppose the byte array is good at Dart side but bad at Rust side, then we know it is the FFI; suppose it is already bad at Dart side, then we know it is serialization problem, etc.) In addition, since you have already open sourced the bridge-related code, maybe is it possible to remove all real business logic and provide a compilable demo, such that I can have a look at it.
For me, it's mostly a debugging issue, and not a huge blocker for now (can just use android). Also, I recently re-pulled my repo, so hopefully it fixes itself? / was a build artifact?
I see. Feel free to ping me when need help / the problem re-occurs / it is open sourced (thus have compilable reproducible sample)!
was a build artifact?
That looks also quite possible: Many function ids change if you add one new function. And if you have new Dart code + old Rust code (e.g. forget to recompile the whole project, but only use a hot restart), that will make it call the wrong Rust function wrapper. Then, the wrong Rust function wrapper will decode the binary message (if using SSE codec) into the wrong struct type, then the error occurs. (since there is no reproducible sample, all my words are only guesses)
I re-built everything from scratch (pulled git repo in diff folder, actually funny story accidentally rm -rf /'d my working tree, wasn't that bad tho).
It's working now. I think it was some weird build cache thing. I'll leave this up to you if you want to investigate further.
Thank you for all your work!
You are welcome and happy to see it working! After #1878 is released, you can try latest version, and see whether the sanity check directly fails if one day the bug re-appears.
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new issue.