Service crashes when keyboard is unplugged
jhakonen opened this issue · comments
I have Logitech G710+ connected to Lenovo Thinkpad TB3 dock and that is connected to Dell Precision 5530. I have Ubuntu 20.04 installed.
While sidewinderd's service is running if I unplug my laptop from the dock, or alternatively unplug keyboard's usb cable (the one with keyboard's logo on it) from the dock, the service crashes. After plugging the keyboard back in, I need to restart the service to get the my macro keys working again.
Service's status after unplugging (happened at 11:27:59):
$ sudo systemctl status sidewinderd.service
● sidewinderd.service - Support for Microsoft SideWinder X4 / X6 and Logitech G105 / G710+
Loaded: loaded (/usr/local/lib/systemd/system/sidewinderd.service; enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Tue 2020-07-14 11:27:59 EEST; 35s ago
Process: 1219617 ExecStart=/usr/local/bin/sidewinderd (code=dumped, signal=SEGV)
Main PID: 1219617 (code=dumped, signal=SEGV)
heinä 14 11:27:23 codebox sidewinderd[1219617]: Unable to find parent device.
heinä 14 11:27:23 codebox sidewinderd[1219617]: Unable to find parent device.
heinä 14 11:27:23 codebox sidewinderd[1219617]: Unable to find parent device.
heinä 14 11:27:23 codebox sidewinderd[1219617]: Unable to find parent device.
heinä 14 11:27:23 codebox sidewinderd[1219617]: Unable to find parent device.
heinä 14 11:27:23 codebox sidewinderd[1219617]: Unable to find parent device.
heinä 14 11:27:23 codebox sidewinderd[1219617]: Found device: 046d:c24d
heinä 14 11:27:59 codebox sidewinderd[1219617]: Keyboard Destructor
heinä 14 11:27:59 codebox systemd[1]: sidewinderd.service: Main process exited, code=dumped, status=11/SEGV
heinä 14 11:27:59 codebox systemd[1]: sidewinderd.service: Failed with result 'core-dump'.
Here's dump output from another crash:
$ coredumpctl
TIME PID UID GID SIG COREFILE EXE
Tue 2020-07-14 11:45:43 EEST 1224923 0 0 11 error /usr/local/bin/sidewinderd
$ sudo coredumpctl gdb
PID: 1224923 (sidewinderd)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Tue 2020-07-14 11:45:43 EEST (1min 3s ago)
Command Line: /usr/local/bin/sidewinderd
Executable: /usr/local/bin/sidewinderd
Control Group: /system.slice/sidewinderd.service
Unit: sidewinderd.service
Slice: system.slice
Boot ID: 344c2ebd5fed47ffba65a76592acdab5
Machine ID: 39aa1007b3e64b41954a3f3652df2f95
Hostname: codebox
Storage: /var/lib/systemd/coredump/core.sidewinderd.0.344c2ebd5fed47ffba65a76592acdab5.1224923.1594716343000000000000.lz4
Message: Process 1224923 (sidewinderd) of user 0 dumped core.
Stack trace of thread 1224923:
#0 0x000055cf5fc19762 _ZN8Keyboard11isConnectedEv (sidewinderd + 0x1e762)
#1 0x000055cf5fc151b0 _ZN13DeviceManager6unbindEv (sidewinderd + 0x1a1b0)
#2 0x000055cf5fc143de _ZN13DeviceManager7monitorEv (sidewinderd + 0x193de)
#3 0x000055cf5fc12ade main (sidewinderd + 0x17ade)
#4 0x00007fc2270c90b3 __libc_start_main (libc.so.6 + 0x270b3)
#5 0x000055cf5fc1204e _start (sidewinderd + 0x1704e)
GNU gdb (Ubuntu 9.1-0ubuntu1) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/sidewinderd...
(No debugging symbols found in /usr/local/bin/sidewinderd)
[New LWP 1224923]
warning: Unexpected size of section `.reg-xstate/1224923' in core file.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/sidewinderd'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Unexpected size of section `.reg-xstate/1224923' in core file.
#0 0x000055cf5fc19762 in Keyboard::isConnected() ()
(gdb) thread apply all bt
Thread 1 (Thread 0x7fc226f51dc0 (LWP 1224923)):
#0 0x000055cf5fc19762 in Keyboard::isConnected() ()
#1 0x000055cf5fc151b0 in DeviceManager::unbind() ()
#2 0x000055cf5fc143de in DeviceManager::monitor() ()
#3 0x000055cf5fc12ade in main ()
(gdb)
Is there anything else I should provide?
Debug build of Sidewinderd (cmake -DCMAKE_BUILD_TYPE=Debug ..) does not crash and seems to work fine. Survives over unplug / replug and macro keys work after replugging.
Release build with debug symbols (cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..) did reproduce though.
Dump output:
$ sudo coredumpctl gdb
PID: 1365279 (sidewinderd)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Tue 2020-07-14 12:04:44 EEST (45s ago)
Command Line: /usr/local/bin/sidewinderd
Executable: /usr/local/bin/sidewinderd
Control Group: /system.slice/sidewinderd.service
Unit: sidewinderd.service
Slice: system.slice
Boot ID: 344c2ebd5fed47ffba65a76592acdab5
Machine ID: 39aa1007b3e64b41954a3f3652df2f95
Hostname: codebox
Storage: /var/lib/systemd/coredump/core.sidewinderd.0.344c2ebd5fed47ffba65a76592acdab5.1365279.1594717484000000000000.lz4
Message: Process 1365279 (sidewinderd) of user 0 dumped core.
Stack trace of thread 1365279:
#0 0x000055cc711efb74 _ZN8Keyboard11isConnectedEv (sidewinderd + 0xdb74)
#1 0x000055cc711ed2c5 _ZN13DeviceManager6unbindEv (sidewinderd + 0xb2c5)
#2 0x000055cc711ee69b _ZN13DeviceManager7monitorEv (sidewinderd + 0xc69b)
#3 0x000055cc711eacf7 main (sidewinderd + 0x8cf7)
#4 0x00007f0cf1dac0b3 __libc_start_main (libc.so.6 + 0x270b3)
#5 0x000055cc711eb13e _start (sidewinderd + 0x913e)
GNU gdb (Ubuntu 9.1-0ubuntu1) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/sidewinderd...
[New LWP 1365279]
warning: Unexpected size of section `.reg-xstate/1365279' in core file.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/sidewinderd'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Unexpected size of section `.reg-xstate/1365279' in core file.
#0 Keyboard::isConnected (this=0x7000700070007) at /home/hakonen/git/sidewinderd/src/core/keyboard.cpp:29
29 return isConnected_;
(gdb) thread apply all bt
Thread 1 (Thread 0x7f0cf1c34dc0 (LWP 1365279)):
#0 Keyboard::isConnected (this=0x7000700070007) at /home/hakonen/git/sidewinderd/src/core/keyboard.cpp:29
#1 0x000055cc711ed2c5 in DeviceManager::unbind (this=this@entry=0x7ffe396574b0) at /usr/include/c++/9/bits/unique_ptr.h:360
#2 0x000055cc711ee69b in DeviceManager::monitor (this=0x7ffe396574b0) at /home/hakonen/git/sidewinderd/src/core/device_manager.cpp:110
#3 0x000055cc711eacf7 in main (argc=<optimized out>, argv=<optimized out>) at /home/hakonen/git/sidewinderd/src/main.cpp:148
(gdb)
I looked more into this, and I think the issue is in inside unbind() method. The code inside erases items from connected_ while iterating over it, and I think that causes the iterator to become invalidated. Following 1st answer at stackoverflow (https://stackoverflow.com/questions/8234779/how-to-remove-from-a-map-while-iterating-it) I made (possibly ugly) fix that seems to fix the crash:
diff --git a/src/core/device_manager.cpp b/src/core/device_manager.cpp
index 86412e3..7f92371 100644
--- a/src/core/device_manager.cpp
+++ b/src/core/device_manager.cpp
@@ -191,9 +191,11 @@ int DeviceManager::probe(struct Device *device, struct sidewinderd::DevNode *dev
}
void DeviceManager::unbind() {
- for (auto &it : connected_) {
- if (!it.second->isConnected()) {
- connected_.erase(it.first);
+ for (auto it = connected_.cbegin(); it != connected_.cend();) {
+ if (it->second->isConnected()) {
+ ++it;
+ } else {
+ it = connected_.erase(it);
}
}
}
Thank you for the bug report and extensive debugging!
I was able to reproduce the issue using Ubuntu 20.04. Tried re-plugging the device multiple times and eventhough sidewinderd didn't crash everytime, it crashed like 1 out of 4 times.
Haven't deeply debugged this, but from what I've understood so far, it's problematic how we unbind devices and call ~Keyboard() destructor. Also the code seems to be unnecessarily complicated, as we keep track of connection status both in DeviceManager and in Keyboard.
Not 100% sure yet, but I think we can call DeviceManager::unbind() with parameters and unbind the specific device which we retrieved from udev (action == remove), instead of iterating over all connected keyboards and looking for disconnected ones. I will try out a few things now.
Cheers,
Tolga
I modified DeviceManager::unbind() as stated above, which fixed the issue but unveiled another, more critical one: inside ~Keyboard() destructor we're indirectly referencing multiple virtual functions via listen().
I remember facing this error a few times, but it was very rare. Modifying unbind() somehow triggers the bug every single time for me. It's fixable, but quite a rabbit hole and involves redesigning a few classes.
Therefore, I plan to merge your quick & dirty fix for now, until we have the proper fix. Please send a Pull Request.
Cheers,
Tolga
Thanks for quick answer!
I noticed as well that the crash did not always happen. Some times it did every time, and sometimes the chances were lower. Additionally, when I added more and more std::cerr calls to the code (in order to pinpoint where the crash happens) the less and less likely it was that crash would occur. Weird.
Anyways, I'll make a PR right away.
Thanks for the Pull Request!
I have tested it, but I still received the same error I was facing above, caused by incorrect usage of the destructor. Essentially, we have 2 issues here: first caused by incorrect usage of the iterator for std::map, which your PR fixes. And second, destructing our keyboard objects.
I've worked on this again and I think it's properly fixed now: as stated above, I have modified unbind() so we don't manually iterate over std::map anymore. Also, I have now declared ~Keyboard() destructor as virtual, so derived classes are now guaranteed to have their instances properly destructed.
Please try out https://github.com/tolga9009/sidewinderd/tree/fix-unplug2 and let me know, if any error messages show up during plug / replug / unplug of your keyboard or starting / stoppind sidewinderd service. If everything works, I will merge it into master and bump to 0.4.3.
Thank you very much!
Cheers,
Tolga
Hi Tolga,
I tried the fix-unplug2 branch and it seems to be working fine. 👍
I tested by plugging/unplugging several times both the keyboard from dock, and dock from laptop, and I can't reproduce the crash anymore. I didn't see any errors, except for excessive amount of "Unable to find parent device" and "Found device: 046d:c24d" messages, although those came before the fix too.
Thanks,
Janne
Tested also service start/stop, and I don't see any error messages there either.
Thanks for the feedback!
Commits 854d637 and ea4bd69 fix the issue. I have now merged to master and released 0.4.3. AUR is also updated.
Regarding the console messages, I think debugging messages should be tagged as debug and only shown, if sidewinderd is launched with verbose flag. So, sidewinderd would only output in case it needs user attention. But thats a different topic, closing here.