Collision Detection Benchmarks [WIP]

This repo contains the benchmarks for the paper "Collision Detection Accelerated: An Optimization Perspective" published at RSS 2022. You can find the paper here and the project page here. There are two main benchmarks: the ellipsoid benchmark (strictly-convex shapes) and the convex mesh benchmark (non-strictly convex shapes), which are intended to compare the GJK algorithm and our method: Nesterov accelerated GJK.

These benchmarks call the HPPFCL C++ library in which both GJK and Nesterov-accelerated GJK are implemented.

For prototyping, we have also reimplemented GJK and Nesterov-accelerated GJK in Python.

Installation

To make the install easy, we recommend using conda to isolate the required packages needed to run the benchmarks from your system.

Clone this repo: git clone --recursive https://github.com/lmontaut/collision-detection-benchmark.git && cd collision-detection-benchmark
Install conda and a new conda environment: conda create -n collision_benchmark python=3.8 && conda activate collision_benchmark
Install dependencies: conda install cmake pinocchio pandas tqdm qhull. For pinocchio, add the conda-forge channel conda config --add channels conda-forge.
Re-activate the conda env conda activate collision_benchmark for the cmake path to take effect.
Install hppfcl:
- mkdir hpp-fcl/build && cd hpp-fcl/build
- git submodule update --init
- cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DCMAKE_BUILD_TYPE=Release -DHPP_FCL_HAS_QHULL=ON ..
- make install
Go back to the root of this repo (cd ../.. if you are in hpp-fcl/build) and install this python library on the conda env: pip install -e .

This was succesfully installed and tested on Manjaro 5.15.50 and Ubuntu 20.04. The tested compilers were g++ version 9.4.0 and 12.1.0 and clang++ version 13.0.1. The required version for eigen is 3.4.0.

ShapeNet download

Please visit https://shapenet.org. Download ShapeNetCore.v2 and place it in exp/shapenet/data.

To generate a subset of ShapeNet to run the benchmarks, run python exp/shapenet/generate_subshapenet.py

Quick benchmarks:

To launch a quick benchmark:

Ellipsoids: python exp/continuous_ellipsoids/ellipsoids_quick_benchmark.py [--opts]
Meshes: python exp/shapenet/shapenet_quick_benchmark.py [--opts].

The param --opts can be:

--python: also runs the quick benchmark with the solvers written in Python, off by default
--measure_time: measures execution times, off by default
--distance_category: overlapping, close-proximity, distant
--num_pairs: number of collision pairs
--num_poses: number of relative poses btw each collision pair

To compare the performances between Nesterov-accelerated GJK and vanilla GJK, we measure both the performance on boolean collision detection and distance computation.

For distance computation, both algorithms run until they have computed the distance which separates the shapes. This is measured by the Frank-Wolfe duality-gap reaching a certain tolerance; please read the paper for more info.
For boolean collision check, both algos stop as soon as they find a separating hyperplane between the shapes or when a point inside their intersection has been found.

We thus measure the following metrics for distance computation:

dist_to_vanilla: distance of the solution found by the solver to the solution found by vanilla GJK.
numit: number of iterations to converge.
execution_time
the suffix rel relates to the relative performance to vanilla GJK. Given a solver, a metric and a collision problem, we do metric of GJK on problem P / metric of solver on problem P. We add the suffix early to numit and execution_time to track the performance of the boolean collision check (early because boolean collision check is an early stop of distance computation).

Large benchmarks:

The plots from the paper where obtained from the following benchmarks. You will need to have pandas to save results to .csv files and jupyter to plot the results: conda install pandas jupyterlab

1 - Ellipsoids

Launch the benchmark: ./ellipsoids_benchmark.sh
View the results: jupyter lab then go to plot_exp/continuous_ellipsoids/continuous_ellipsoids_plots.ipynb and run the notebook.

Citing this repo

To cite Nesterov accelerated GJK and/or the associated benchmarks, please use the following bibtex lines:

@inproceedings{montaut2022GJKNesterov,
  title = {Collision Detection Accelerated: An Optimization Perspective},
  author = {Montaut, Louis and Le Lidec, Quentin and Petrik, Vladimir and Sivic, Josef and Carpentier, Justin},
  booktitle = {Robotics: Science and Systems},
  year = {2022}
}

lmontaut / collision-detection-benchmark