This is the main repository for Hazards&Robots: A Dataset for Visual Anomaly Detection in Robotics and relative papers.
The dataset can be find on Zenodo:
- (v1) TAROS version : https://zenodo.org/record/7035788
- (v2) RAL extension : https://zenodo.org/record/7074958
-
(v3) Data in Brief final version: https://zenodo.org/record/7859211
Dario Mantegazza, Carlos Redondo, Fran Espada, Luca M. Gambardella, Alessandro Giusti and Jerome Guzzi
We consider the problem of detecting, in the visual sensing data stream of an
autonomous mobile robot, semantic patterns that are unusual (i.e., anomalous) with
respect to the robot's previous experience in similar environments. These
anomalies might indicate unforeseen hazards and, in scenarios where failure is
costly, can be used to trigger an avoidance behavior. We contribute three novel
image-based datasets acquired in robot exploration scenarios, comprising a total
of more than 200k labeled frames, spanning various types of anomalies. On these
datasets, we study the performance of an anomaly detection approach based on
autoencoders operating at different scales.
In the Proceedings of 23rd TAROS 2022 Conference
DOI: https://doi.org/10.1007/978-3-031-15908-4_17
ArXiv: https://arxiv.org/abs/2110.14706
Dario Mantegazza, Alessandro Giusti, Luca M. Gambardella and Jerome Guzzi
We consider the problem of building visual anomaly detection systems for mobile
robots. Standard anomaly detection models are trained using large datasets composed
only of non-anomalous data. However, in robotics applications, it is often the case
that (potentially very few) examples of anomalies are available. We tackle the
problem of exploiting these data to improve the performance of a Real-NVP anomaly
detection model, by minimizing, jointly with the Real-NVP loss, an auxiliary outlier
exposure margin loss. We perform quantitative experiments on a novel dataset (which
we publish as supplementary material) designed for anomaly detection in an indoor
patrolling scenario. On a disjoint test set, our approach outperforms alternatives
and shows that exposing even a small number of anomalous frames yields significant
performance improvements.
Published in Robotics and Automation Letters October 2022 Volume 7 Issue 4
DOI: https://doi.org/10.1109/LRA.2022.3192794
ArXiv: https://arxiv.org/abs/2209.09786
Dario Mantegazza, Alind Xhyra, Luca M. Gambardella, Alessandro Giusti, Jérôme Guzzi
We propose Hazards&Robots, a dataset for Visual Anomaly Detection in Robotics.
The dataset is composed of 324,408 RGB frames, and corresponding feature vectors;
it contains 145,470 normal frames and 178,938 anomalous ones categorized in 20
different anomaly classes. The dataset can be used to train and test current and
novel visual anomaly detection methods such as those based on deep learning vision models.
The data is recorded with a DJI Robomaster S1 front facing camera. The ground robot,
controlled by a human operator, traverses university corridors. Considered anomalies
include presence of humans, unexpected objects on the floor, defects to the robot.
DOI: https://doi.org/10.1016/j.dib.2023.109264
This is an Open-Access paper published in Data in Brief Volume 48, June 2023, Journal
Under ./code
you can find the code used for the TAROS paper under ./code/OLD_CODE
and the code for RAL paper under ./code/Latest
; the code for the Data in Brief is available on the Zenodo repository.
We use python 3.8 and the requirements in ./code/Latest/requirements.txt
; follow the README.md under the ./code/Latest
to install and run the models.
The dataset is composed of three different scenarios:
- Tunnel
- Factory
- Corridors
The TAROS version paper the Corridors scenario has 52'607 samples and 8 anomalies.
In the RAL paper we extended this scenario up to 132'838 frames and 16 anomalies.
The latest Data in Brief release has 324'408 frames and 20 anomalies; for the first time we provide 512-dimension features vectors extracted with CLIP.
Examples of samples of the Corridors scenario from the Data in Brief paper
This work was supported as a part of NCCR Robotics, a National Centre of Competence in Research, funded by the Swiss National Science Foundation (grant number 51NF40_185543) and by the European Commission through the Horizon 2020 project 1-SWARM, grant ID 871743.
- If you have questions please contact us via email dario (dot) mantegazza (at) idsia (dot) ch
- Questions or problems with the code? Just open an ISSUE, we will do our best to answer you as soon as possible :)
- For more information about us visit our site https://idsia-robotics.github.io/
If you use this dataset please cite it using the following bib
@ARTICLE{mantegazza2022outlier,
author={Mantegazza, Dario and Giusti, Alessandro and Gambardella, Luca Maria and Guzzi, Jérôme},
journal={IEEE Robotics and Automation Letters},
title={An Outlier Exposure Approach to Improve Visual Anomaly Detection Performance for Mobile Robots.},
year={2022},
volume={7},
number={4},
pages={11354-11361},
doi={10.1109/LRA.2022.3192794}
}
Across the three scenarios described before, we recorded various normal situations and numerous anomalies. The anomalies are the following:
Click for high resolution examples
![tun_normal](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/tunnel/normal1.jpg)
Normal - Empty underground man made tunnel
![wet](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/tunnel/wet1.jpg)
Wet - Water condensation on the tunnel walls and ceiling
![root](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/tunnel/root1.jpg)
Root - Roots coming down from the ceiling and walls
![dust](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/tunnel/dust1.jpg)
Dust - Dust moved by the drone
Click for high resolution examples
![fact_normal](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/factory/normal1.jpg)
Normal - Empty factory facility
![mist](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/factory/mist1.jpg)
Mist - Mist coming from a smoke machine
![tape](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/factory/tape1.jpg)
Tape - Signaling tape stretched across the facility
Click for high resolution examples
![corridor_normal](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/normal1.jpg)
![corridor_normal2](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/normal2.jpg)
![corridor_normal3](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/normal3.jpg)
Normal - Empty university corridors (on different floors)
![box](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/box.jpg)
Box - Cardboard boxes placed in front/near of the robot
![cable](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/cable.jpg)
Cable - Various cables layed on the floor around and in front of the robot
![debris](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/debris.jpg)
Debris - Various debris
![defects](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/defects.jpg)
Defects - Defects of the robot
![door](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/door.jpg)
Door - Open doors where doors should be closed
![human](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/human.jpg)
Human - Human presence
![clutter](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/clutter.jpg)
Clutter - Chairs, tables and furniture moved around the corridor
![foam](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/foam.jpg)
Foam - Foam placed on the floor
![sawdust](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/sawdust.jpg)
Sawdust - Sawdust placed on the floor
![cellophane](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/cellophane.jpg)
Cellophane - Cellophane foil stretched between walls
![floor](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/floor.jpg)
Floor - Fake flooring different than original floor
![screws](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/screws.jpg)
Screws - Small screws and bolts placed in front of the robot
![water](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/water.jpg)
Water - Water puddle in front of robot
![cones](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/cones.jpg)
Cones - Multiple orange cones placed in the corridor
![hanghingcables](https://raw.githubusercontent.com/jackyccl/hazard-detection/main/images/corridor/hanging_cable.jpg)
Hanging cables - Cables hanging from the ceiling