LOT - Low Overhead Transport

General description

This repository contains the C++ implementation of the shared memory mechanism described in the paper Smart Pointers and Shared Memory Synchronisation for Efficient Inter-process Communication in ROS on an Autonomous Vehicle, along with detailed build and deployment instructions.

The implementation is based on a fork of ros_comm kinetic 1.12.14. along with a suite of benchmarks that were used to generate the LOT benchmark results.

The code can be built either directly on the host machine or within a Docker container via the provided Dockerfile. The former assumes the installation of ROS 1.12.14, Boost 1.58.0 and python2 on the host machine. The latter only requires Docker installation on the host machine. See the Docker website for details of setting up Docker on the host machine.

The benchmarks - either the entire suite or individual tests - can be executed following the procedure described below. Currently, four protocols are supported: TCP, UDP, LOT and TZC.

When executed from within the Docker containers, the benchmarks results are generated under ${HOME}/lot folder. Otherwise, environment variable lot_HOME dictates the results folder.

Cloning the repository and switching to the target branch

Note that we clone the repository into $HOME/ros_comm/src/ros_comm in this step. We assume this directory in subsequent steps. If you prefer to use a different working directory, the subsequent steps will need to be appropriately modified.

git clone git@github.com:fiveai/ros_comm.git $HOME/ros_comm/src/ros_comm
cd $HOME/ros_comm/src/ros_comm
git checkout lot

Building and running benchmarks natively

The following steps build the code directly on the host machine.

Compiling the entire workspace

cd $HOME/ros_comm
source /opt/ros/kinetic/setup.sh  # this assumes ROS kinetic 1.12.14 is already installed
catkin_make_isolated
  --source ./src/  \
  --build $HOME/ros_comm/build-release \
  --devel $HOME/ros_comm/devel-release \
  --install-space $HOME/ros_comm/install-release  \
  --install \
  --cmake-args -DCMAKE_BUILD_TYPE=Release

Executing the benchmarks specified in launch.xml, overriding some of the parameters

cd $HOME/ros_comm
source ./devel-release/setup.sh
roslaunch --screen -v  benchmark launch.xml   \
  use_case:=5p1s_same_host sub_stats_file_path:=$HOME \
  transport:=shm pub_queue_size:=200 pub_image_count:=200

Building and running benchmarks within Docker containers

Building the Docker image

cd $HOME/ros_comm/clients/benchmark/docker
export DOCKER_BUILDKIT=1
docker build --ssh default --tag lot .

Launch the docker containers, and run the benchmarks from within 1 of them

Launch 6 containers based on the above image, and attach to one of them:

cd $HOME/ros_comm/clients/benchmark/docker
docker-compose up -d
docker attach node1

From within the container, benchmarks can be executed using commands of the form:

python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py --tcp=no --shm=yes --use_case=5p1s_separate_docker

Once you're finished, exit the docker container and shutdown the others

docker-compose stop

You should find the results in $HOME/lot

The following section provides examples of benchmark commands.

Benchmark execution command examples

Execute the benchmark suite for 1p5s using TZC protocol, in separate Docker containers, enforcing the subscribers start up order, allocating 16GB of shared memory to be used by the publishers, and telling the subscriber to wait 15 seconds before starting publishing messages, TCP_NODELAY enabled.

python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py      \
        --tcp=no --shm=no --tzc=yes --udp=no                     \
        --use_case=1p5s_separate_docker --no_pool=yes --pool=no  \
        --extra_params sub_enable_synch_startup:=true            \
                       pub_extra_delay_ms:=15000                 \
                       shm_size_mega_bytes:=3000

Execute the benchmarks suite for 5p1s using TZC protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the subscriber start up delayed by 15secs, TCP_NODELAY enabled.

python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py      \
        --tcp=no --shm=no --tzc=yes --udp=no                     \
        --use_case=5p1s_separate_docker --no_pool=yes --pool=no  \
        --extra_params  pub_wait_for_subscribers:=true           \
                        sub_extra_delay_ms:=15000

Execute the benchmarks suite for 5p1s using TZC protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the publishers start up order enforced, TCP_NODELAY enabled, overriding the default results path.

python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py      \
        --tcp=no --shm=no --tzc=yes --udp=no                     \
        --use_case=5p1s_separate_docker --no_pool=yes --pool=no  \
        --extra_params  pub_wait_for_subscribers:=true           \
                        pub_enable_synch_startup:=true           \
                        sub_stats_file_path:=/path/to/results

Execute the benchmarks suite for 5p1s using LOT protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the publishers start up order enforced, and image pools disabled.

python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py      \
        --tcp=no --shm=yes --tzc=no --udp=no                     \
        --use_case=5p1s_separate_docker --no_pool=yes --pool=no  \
        --extra_params  pub_wait_for_subscribers:=true           \
                        pub_enable_synch_startup:=true

Execute one single test using TCP protocol, in same docker container, with image pool disabled.

roslaunch --screen -v  benchmark launch.xml     \
        pub_image_count:=200                    \
        use_case:=1p5s_same_docker              \
        transport:=tcp                          \
        image_width_pixels:=512                 \
        image_height_pixels:=512                \
        pub_pool_size:=0

Licence

LOT is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

This work is based on a fork of ros_comm kinetic 1.12.14. We apply the following fixes to ros_comm, in addition to our own code:

#1115 roslaunch - pass through command-line args to the xmlloader when using the API.
replace boost::condition_variable variable with std::condition_variable in callback_queue.cpp.

Boost.Process 1_65_0 has been imported into the source tree at ./clients/roscpp/include/boost_1.65.0 to support unit tests. We note that it is released under the Boost Software License.

For convenience of running the benchmarks, we have also included code from TZC at ./clients/roscpp/include/tzc. TZC was developed by a group researchers affiliated to Tsinghua University, China and University of Maryland, USA and is described more fully in their paper TZC: Efficient Inter-Process Communication for Robotics Middleware with Partial Serialization. We note it is released it under a BSD license.

We thank the authors for making their code available.

fiveai / ros_comm