[ROS2] Revamp Callback Groups (and/or Executors)
amalnanavati opened this issue · comments
Currently, most of our nodes use the default callback group, which only lets one topic/service/action/timer process a callback at a time. This is a huge problem, because it slows down execution and can result in some callbacks not getting called for a while because other callbacks are called first.
We should be very intentional with what executor and callback group we are using for each node and each callback within the node. See the below resources for more details:
In #56 we are doing it in a jank-y way by resetting a protected access class attribute in create_action_servers.py. There has to be a better way to do it.
#90 and ada_ros2#19 should address this, by putting all callbacks into a ReentrantCallbackGroup (allowing them all to be executed in parallel).
While this should work in theory, I'll leave this issue open so someone can go back and more intentionally consider every pair of callbacks and whether they can truly be executed in parallel or not.
Note: the ReentrantCallbackGroups appear to be too much for t0b1's and lovelace's processing powers. I am seeing jerky motions and dropped watchdog messages on t0b1, which I never saw before, and dropped watchdog messages on lovelace. Perhaps the watchdog should have its own callback group, the MoveIt object should have its own callback group, and the rest should be mutually exclusive? Because I don't think we need any of the others in parallel.
In forque_sensor_hardware as well, I'm noticing Error reading from telnet socket. and Error writing to telnet socket. on t0b1 if the sensor has been left on for > 1.5 mins. Perhaps the timer and the service should be in separate MutuallyExclusiveCallbacks, but not in a single ReentrantCallback?
#92 mostly addresses this, lingering issues will be documented there.