GRASPA (GRASPA is a Robot Arm graSping Performance benchmArk) is a benchmarking protocol and a set of metrics to evaluate the performance of grasp planners. It aims to fairly compare methodologies tested on different robots by measuring and accounting for platform limitations that might affect the performance of the algorithm itself.
Version 1.0 features:
- 3 printable layouts of predefined grasping scenarios (composed of a subset of YCB objects and localization markers) to enhance test reproducibility.
- A protocol to assess the robot reachability and the calibration of the vision system within the defined grasping setup area.
- Usage of a popular grasp quality metric to evaluate user-provided grasping poses.
- A score to assess the grasp stability during the practical execution on the robot.
- The possibility to benchmark the algorithm either in isolation or in the clutter, with the definition of a further metric to evaluate the obstacle avoidance in the clutter modality.
- A composite score to quantify the overall performance of the algorithm accounting for reachability and calibration limits of the test platform.
We have packaged a GitPod environment with every dependency compiled and already installed. Click on the button to open the cloud environment in your browser.
To ease the installation process, please use the Docker image we prepared.
docker pull ghcr.io/hsp-iit/graspa-benchmark:latest
GitPod is a cloud-based development environment that allows users to test code from a GitHub repository without having to set up the environment and dependencies on their local machine. We have set up such an environment for this repo to allow users to test the code in the most hassle-free way.
To further smoothen user experience, our GitPod instance comes with a Fluxbox desktop environment packed in. Once the environment is ready (it takes a while), GitPod will signal the user that a service is available on port 8080. By clicking on the Open Browser option, you will be automatically connected to the in-browser desktop. Otherwise, click Remote Explore tab in Gitpod, the Gitpod Workspace dropdown shows several ports. Select Open Browser for the 8080 port and click on vnc_auto.html to visualize the remote desktop.
To test the code on the data that we gathered for the paper and see the outcome, simply open a terminal in the remote desktop and type
cd /path/to/GRASPA-benchmark
$ ./src/script_launcher.sh
and the bash script will perform every step required to compute the benchmark score of our platform.
You can use the Docker image you just pulled. Fire it up with
docker run -it --rm -p 8080:8080 ghcr.io/hsp-iit/graspa-benchmark
and then navigate from your favourite web browser to http://localhost:8080. At this point, open a terminal and proceed as outlined in the GitPod instructions section.
Once the data required by the benchmark are available (see here instructions on how to collect them), all the scores can be computed by:
- Properly pointing to the folders including the data in the
src/script_launcher.sh. More information on how to fill correctly the script is available here. - Launching the script:
An example of the benchmark output is available here.
cd $GRASPA_DIR $ ./src/script_launcher.sh
GRASPA 1.0 comes with several resources and tools:
data/collects the:YCB objects modelsused in the grasping layouts[1];- the
3D renders of the grasping layouts; - the
printable versions; - the poses to be reached for the
reachability testand thecamera-calibration test. template files, thexmlsto be filled with the data collected on your robot in order to compute the benchmark scores.
src/includes several scripts and executables:- a
launcher scriptto automatically compute all the scores; - a
C++ executableto compute the grasp quality score using Simox; - a
python scriptto combine all the information to get the benchmark scores; - a
viewerto visualize the data provided within GRASPA and those collected by the user; - a
python scriptto automatically generate the printable layouts of the grasping scenarios defined by GRASPA 1.0.
- a
[1]This repo includes a subset of the YCB object dataset meshes for user convenience. We re-distribute these files under the same Creative Commons Attribution 4.0 International (CC BY 4.0) license.