Subscribing with Theora compression reduces publish rate
s-m-asjad opened this issue · comments
This has been tested on both, a local and remote system setting, with ROS2 Humble.
Occasionally, the publisher's frequency of publishing images will drop from 30FPS to ~15FPS and lower depending on the size of image. Theora seems to stand strong when streaming HD images at 30FPS but going FHD will occasionally drop the publish rate to 15 FPS (it will stay at the lower FPS until the subscriber is killed).
There have been instances where streaming at 2K was successful without a drop in the publisher's rate of publishing although we lost frames at the subscriber. This however, does not seem persistent as other times (as mentioned previously) subscribing to the theora topic would reduce the publish rate of the publisher too.
If the setup is tested in a remote setting, subscribing to the default compressed topic, we see a large amount of data (usually in MB/s) going out towards the local system which is expected behavior. In the case of theora, when the aforementioned issue is not encountered, data usually in MB/s is seen being streamed (also expected behavior). However, when subscribing to the theora topic reduces the FPS of the publisher, we observe significantly reduced data being streamed, usually under in 200s of kB/s. This implies that the network has higher bandwidth for streaming, however, theora seems to not be using it and hence introducing latency.
We observed the data streamed out using ifstat -i and the topic publish rate via ros2 topic hz
I was able to work around this issue by running the following command in another terminal.
ros2 run image_transport republish raw theora --ros-args --remap in:=<raw image topic> --remap out/theora:=<new image topic>
For example if I have a topic called camera/image then I can republish to another topic high_fps_camera/image. This will create a new topic for the theora compression only with the name high_fps_camera/image/theora. Subscribing to this new theora topic will not drop the publish rate of the original publisher.
It seems that the reason the publish rate is dropping when someone subscribes to the original theora topic from the image_transport is bad QoS settings. Although I had not modified my QoS settings for either case but something seems to be overriding them for the original scenario.
By "original scenario" I mean that I have a publisher that is an image_transport::Publisher object and publishes a sensor_msgs::msg::Image::SharedPtr message through a image_transport::ImageTransport streamer node
Hi @s-m-asjad,
image_transport compression happened in publisher's node since ROS1.
Looking at Humble code
- https://github.com/ros-perception/image_common/blob/e947b47a45971e3edb59d8e34bc8e7cd2a41f2e6/image_transport/src/publisher.cpp#L164
- it is still the same
- publisher goes through all plugins and asks for compression
Unless ROS camera driver (sensor, your source) is written taking above into account (for example by decoupling internal driver logic into thread) the total time ROS driver has will be limited by driver internal handling and compression time.
A lot of ROS drivers are written not taking above into account.
I was able to work around this issue by running the following command in another terminal.
ros2 run image_transport republish raw theora --ros-args --remap in:=<raw image topic> --remap out/theora:=<new image topic>
As you have noticed you may decouple compression time from driver internals using republish.
In such scenario both driver/source and compression should have full frame time available for own processing.
I don't know about QoS settings but above may be one of the causes of your problems.
Also, I am not sure but it is possible that theora may emit multiple packets (ROS messages) per single image. You would need to double check.
In case message delivery is not guaranteed loosing one of those messages might prevent decoding image. In particular loosing one of keyframe sub-messages might prevent receiving images until next keyframe.
Hi @bmegli
Thankyou for your that great insight.
It might as well be the first issue you mentioned of not having a decent driver. My driver followed the same basic skeleton as this for the initial tests but I do plan to eventually isolate driver logic into its own separate thread. I shall look into this and hopefully it indeed pops up as the root of all evil.
I am certain that the issue is not caused by
In case message delivery is not guaranteed loosing one of those messages might prevent decoding image. In particular loosing one of keyframe sub-messages might prevent receiving images until next keyframe.
(I could be mistaken though, please feel free to correct me)
because I tried doing ros2 topic echo <theora topic name> on the local system itself and that too throttled the publisher's publish rate. If I understand things correctly then the decoding happens by the subscriber and echoing should only display the contents of the message.
because I tried doing ros2 topic echo
theora topic nameon the local system itself and that too throttled the publisher's publish rate. If I understand things correctly then the decoding happens by the subscriber and echoing should only display the contents of the message.
Yes, your understanding is correct:
- decoding happens in subscriber
- if echoing
theora topic namedirectly no decompression will happen, you echo compressed data
Even if driver doesn't take into account that compression may consume part of its processing time, it's easily fixable with republish.
Also, you may just measure (even with simple timer logic and debug output from node):
- how much time you need to process data internally
- how much time compression/publishing takes
- if sum of above exceeds time between frames you will typically have drops/queued messages if you allow queues
Core of the issue might be this ros2/ros2#1434
Does the theora compression thread on the publisher use close to 100% of one of your CPU cores?