Command - reaction latency
iluetkeb opened this issue · comments
I've noticed a control dead time of approximately 200 milliseconds (time between the control command is sent to the hardware to the first change in odometry readings) in the Turtlebot2 firmware. This is on firmware version 1.2.0. See the following image for a plot (but note that I did the actual measurement based on a recorded rosbag, so it's not just determined from the plot).
I'm not sure this is a bug, but it is certainly very noticeable and impairs performance of the robot in dynamic situations. It would be great to learn whether this is intentional and whether the dead time can be reduced.
Hi @iluetkeb , thanks for posting this issue, I was wondering if you have solved the issue and found the root cause.
-- Luke
@lukelu0520 This is caused by the firmware of the Kobuki base. The source code to the firmware is not publicly available. Therefore, only Yujin can change this behavior.
@lukelu0520 This is caused by the firmware of the Kobuki base. The source code to the firmware is not publicly available. Therefore, only Yujin can change this behavior.
@iluetkeb thank you, can't believe we could get your reply in 2020 !
200 milliseconds sounds a bit large for real-time application, e.g. telepresence. Do you find any firmware update or any custom solution in 2020 to improve the latency further ?
So, the title of this ticket (originally "control dead time") was not 100% correct. The latency is not the "control dead time". It's the time between the command timestamp and the first sensor message timestamp showing any changes. Since both of these timestamps are generated in the Linux host, they include transmission delay and other effects. E.g., the Kobuki messages don't have an absolute timestamp, so there's also uncertainty there.
Finally, back in the day I managed to get it to between 120-150ms using some optimization on the Linux side.
That's still a lot, but keep in mind the Kobuki base is an old, repurposed robotic vacuum which was never designed for low-latency applications such as telepresence.
So, the title of this ticket (originally "control dead time") was not 100% correct. The latency is not the "control dead time". It's the time between the command timestamp and the first sensor message timestamp showing any changes. Since both of these timestamps are generated in the Linux host, they include transmission delay and other effects. E.g., the Kobuki messages don't have an absolute timestamp, so there's also uncertainty there.
Finally, back in the day I managed to get it to between 120-150ms using some optimization on the Linux side.
That's still a lot, but keep in mind the Kobuki base is an old, repurposed robotic vacuum which was never designed for low-latency applications such as telepresence.
I see, glad to hear you managed to get below 150ms, it's impressive when considering the age of Kobuki. Are you @iluetkeb working on related projects such as robot moving base now ? In 2020, we were wondering if any robot platform could give low-latency control, e.g. <100ms. So far we found Kobuki is still one of the best, hope to work on more options.
-- Luke
Worth noting that #382 discovered a regression in ftdi settings that hammered Kobuki's packet stream (circa 2017) on the usb-serial connection. Empirical measurements would seem to indicate that it frequently adds a very irregular 20+ms worth of latency.
The 16ms FTDI issue was known to me (it's been around for a while, on and off) and was addressed prior to making these measurements.
Can confirm 100-120ms latency with a test program on foxy (no ROS involved):
| Timestamp | Command | Pose2D |
|---|---|---|
| [966718.231485232] | [0.3,0] | [0,0] |
| [966718.255552732] | [0.3,0] | [0,0] |
| [966718.272513144] | [0.3,0] | [0,0] |
| [966718.291353923] | [0.3,0] | [0,0] |
| [966718.311167736] | [0.3,0] | [0,0] |
| [966718.331446064] | [0.3,0] | [0,0] |
| [966718.351330369] | [0.3,0] | [0.000255876,0] |
| [966718.371246106] | [0.3,0] | [0.00524546,-0.000370835] |
| [966718.392084524] | [0.3,0] | [0.0124953,-0.00259585] |
I do know that when they were building the firmware & software, engineering hours weren't available to dedicate towards product level optimisations, but certainly, 100-120ms is more than I expected. I cannot recall if this is working as intended, or something has crept in since a first release.