Arena-bench (RA-L and IROS 22)

This repository provides the code used in our paper "Arena-Bench: A Benchmarking Suite for Obstacle Avoidance Approaches in Highly Dynamic Environments" (accepted for publication in the IEEE Robotics and Automation Letters). The scope of this project is to provide a benchmark suite to compare and evaluate classic and learning-based dynamic obstacle avoidance approaches on different robotic systems in highly dynamic simulation environments. Link to demo video.

Benchmark scenarios
Benchmark architecture
Running the Benchmark
Thanks

If you find this repository useful, please cite our paper

@ARTICLE{9827557,

  author={Kästner, Linh and Bhuiyan, Teham and Le, Tuan Anh and Treis, Elias and Cox, Johannes and Meinardus, Boris and Kmiecik, Jacek and Carstens, Reyk and Pichel, Duc and Fatloun, Bassel and Khorsandi, Niloufar and Lambrecht, Jens},
  journal={IEEE Robotics and Automation Letters}, 
  title={Arena-Bench: A Benchmarking Suite for Obstacle Avoidance Approaches in Highly Dynamic Environments}, 
  year={2022},
  pages={1-8},
  doi={10.1109/LRA.2022.3190086}}

Benchmark scenarios

Benchmarking scenarios were developed uniquely combining the following components

Robot	World	Planner	Obstacles
- `jackal` - `burger` - `robotino` (rto)	- `small_warehouse` - `map2` - `map5`	- `teb` - `dwa` - `mpc` - `rosnav` - `cadrl`	- `5` - `10`

The following clips show examples of the implemented scenarios:

Benchmark architecture

The repository is structured into a modular architecture with four models at its core.

Arena-Tools: this module is being responsible to create custom worlds, scenarios, and obstacles and adding those to the respective location. It is possible to use arena-bench without arena-tools as several worlds, scenarios and obstacles have already been included, including all scenarios and worlds to reproduce this paper.
Arena-Rosnav: this module contains a custom task-generator and the 2D simulation engine Flatland, enabling robot simulation in 2D Flatland environments (note that this repository has been added recently to the arena-bench repository, and is therefore not part of the original paper).
Arena-Rosnav-3D: this module contains a custom task-generator and the 3D simulation engine Gazebo, enabling robot simulation in 3D Gazebo environments.
Arena-Evaluation: this module can be used to record and evaluate simulation runs and visualize Robot performance by creating qualitative and quantitative plots.

NOTE: All modular components have their individual repository and are currently being extended with additional features

Running the Benchmark

Prerequisites

Below is the software we used. We cannot guarantee older versions of the software to work. Yet, newer software is most likely working just fine.

Software	Version
OS	Ubuntu 20.04.4
Python	3.8.10

Installation

To install the repo run:

wget https://raw.githubusercontent.com/ignc-research/arena-bench/main/setup.sh -O - | bash

Manually running the benchmark

You can run specific scenarios, be using the following syntax:

roslaunch arena_bringup start_arena_gazebo.launch local_planner:=PLANNER world:=WORLD model:=ROBOT scenario_file:=SCENARIO_FILE

with the choice of:

PLANNER: dwa, teb, mpc, rosnav, cadrl
WORLD: small_warehouse, map2, map5
ROBOT: turtlebot3_burger, jackal, rto
SCENARIO_FILE: map2_obs05.json, map2_obs10.json, map5_obs05.json, map5_obs10.json, small_warehouse_obs05.json, small_warehouse_obs10.json

Example command:

workon rosnav
roslaunch arena_bringup start_arena_gazebo.launch local_planner:=dwa world:=map2 model:=turtlebot3_burger scenario_file:=map2_obs05.json

ignc-research / arena-bench