tomas-wood / therobotrix

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

The RobotriX

video_preview

Enter the RobotriX, an extremely photorealistic indoor dataset designed to enable the application of deep learning techniques to a wide variety of robotic vision problems. The RobotriX consists of hyperrealistic indoor scenes which are explored by robot agents which also interact with objects in a visually realistic manner in that simulated world. Photorealistic scenes and robots are rendered by Unreal Engine into a virtual reality headset which captures gaze so that a human operator can move the robot and use controllers for the robotic hands; scene information is dumped on a per-frame basis so that it can be reproduced offline using UnrealCV to generate raw data and ground truth labels. By taking this approach we were able to generate a dataset of 38 semantic classes across 512 sequences totaling 8M stills recorded at +60 frames per second with full HD resolution. For each frame, RGB-D and 3D information is provided with full annotations in both spaces. Thanks to the high quality and quantity of both raw information and annotations, the RobotriX will serve as a new milestone for investigating 2D and 3D robotic vision tasks with large-scale data-driven techniques.

seq0 seq0_depth seq0_mask seq1 seq1_depth seq1_mask seq2 seq2_depth seq2_mask

Contents

  1. Data
  2. UnrealROX
  3. Assets
  4. Troubleshooting
  5. License
  6. Contact

Data

We generated a dataset of 512 sequences recorded on 16 different indoor layouts at +60 FPS with a duration that spans between one and five minutes each. That adds up to a total of approximately eight million individual frames. This initial release of the dataset contains 32 detection and 39 semantic classes. The categories were selected from the most common and useful household goods in indoor environments for social robots.

Type 00
c0
01
c1
02
c2
03
c3
04
c4
05
c5
06
c6
07
c7
08
c8
09
c9
10
c10
11
c11
12
c12
Semantic void wall floor ceiling window door table chair lamp sofa cupboard screen hand
Detection - - - - - - table chair lamp sofa cupboard screen hand
Type 13
c13
14
c14
15
c15
16
c16
17
c17
18
c18
19
c19
20
c20
21
c21
22
c22
23
c23
24
c24
25
c25
Semantic frame bed fridge whiteboard book bottle plant furniture toilet phone bathtub cup mat
Detection frame bed fridge whiteboard book bottle plant - lamp toilet phone bathtub cup
Type 26
c26
27
c27
28
c28
29
c29
30
c30
31
c31
32
c32
33
c33
34
c34
35
c35
36
c36
37
c37
38
c38
Semantic mirror sink box mouse keyboard bin cushion shelf bag curtain kitchen_stuff bath_stuff prop
Detection mirror sink box mouse keyboard bin cushion shelf bag - kitchen_stuff bath_stuff prop

Due to the huge size of the data (~7 TiB), we are currently distributing part of it via private links to our FTP server to avoid excessive traffic (drop a mail to agarcia@dtic.ua.es for them). However, half of the dataset is already available (and increasing every day) through OSF at https://osf.io/b3g2y/ (OSF is a free, open source web application that connects and supports the research workflow, enabling scientists to increase the efficiency and effectiveness of their research. Researchers use OSF to collaborate, document, archive, share, and register research projects, materials, and data. OSF is the flagship product of the non-profit Center for Open Science).

The following data is available at the OSF project page:

ID Scene Robot Interactable Objects Cameras Duration Frames Total
TOTAL TOTAL TOTAL TOTAL TOTAL 3.039.252
000 HamburgHaus 11:48 63.714 318.570
000 HamburgHaus Mannequin 0 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:27 7838 39190
001 HamburgHaus Mannequin 1 (Moka_Chaleira_Moka_10) 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:18 7022 35110
002 HamburgHaus Mannequin 1 (Vase_Rounded_46) 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:02 5554 27770
003 HamburgHaus Mannequin 3 (Vase_Rounded_46, DEC_living_book_6, and DEC_living_book_4) 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:49 9786 48930
004 HamburgHaus Mannequin 5 (Cadeira_Eames_Cadeira_Eames_137, DEC_living_book_6, DEC_living_book_4, Vase_Rounded_46, Moka_Chaleira_Moka_10) 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:59 10704 53520
005 HamburgHaus Mannequin 2 (Moka_Chaleira_Moka_10, Vase_Rounded_46) 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:54 10269 51345
006 HamburgHaus Mannequin 0 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:12 6482 32410
007 HamburgHaus Mannequin 0 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:07 6059 30295
001 Viennese 127.521 679.736
000 Viennese Mannequin 3 (Fruit_Pear, Fruit_Apple, Fruit_Orange2) 5 (Head, LeftHand, RightHand, TopViewCamera, CornerCamera0) 04:18 23202 116.010
001 Viennese Mannequin 7 (Fruit_Pear, Fruit_Apple, Fruit_Orange2, Pot, Pot2, Fruit_Apple2, Fruit_Orange) 5 (Head, LeftHand, RightHand, TopViewCamera, CornerCamera0) 07:09 38627 193.135
002 Viennese Mannequin 8 (Moka_Coffe, Pot, Fruit_Pear, Fruit_Apple, Fruit_Orange2, Fruit_Apple2, Fruit_Orange) 5 (Head, LeftHand, RightHand, TopViewCamera, CornerCamera0) 04:29 23661 118.305
003 Viennese Mannequin 14 (Flow_chair, Flow_chair2, chair_table_corners2, Fruit_Pear, Fruit_Apple, Fruit_Orange2, Fruit_Apple2, Fruit_Orange, Moka_Coffe, table_decoration1_, table_decoration_2, Pot, Pot2, DEC_dining_table_vase) 6 (Head, LeftHand, RightHand, TopViewCamera, CornerCamera0, CornerCamera1) 07:57 42031 252.186
002 InteractiveHouse 20:08 103.352 516.760
000 InteractiveHouse Mannequin 0 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:39 8587 42935
001 InteractiveHouse Mannequin 4 (Vase_13, Flower_Pot2, SM_MERGED_Plant_Bromelia_10, Chair_93) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:40 8510 42550
002 InteractiveHouse Mannequin 7 (Vase_13, Flower_Pot2, SM_MERGED_Plant_Bromelia_10, Chair_93, Chair2, Chair3, Chair4) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 02:00 10800 54000
003 InteractiveHouse Mannequin 6 (Vase_13, Flower_Pot2, SM_MERGED_Plant_Bromelia_10, DEC_dining_vase_332, DEC_dining_vase_320) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 02:31 13564 67820
004 InteractiveHouse Mannequin 7 (Vase_13, DEC_dining_vase_332, DEC_dining_vase_320, DEC_living_book_6, Moka_Coffe_27, table_decoration_91, DEC_living_vase_410) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 02:47 14649 73245
005 InteractiveHouse Mannequin 9 (Flower_Pot2, SM_MERGED_Plant_Bromelia_10, Vase_13, DEC_dining_vase_332, DEC_dining_vase_320, DEC_living_book_6, Moka_Coffe_27, table_decoration_91, DEC_living_vase_410) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 02:56 14091 70455
006 InteractiveHouse Mannequin 4 (Chair_93, Chair2, Chair3, Chair4) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:37 8727 43635
007 InteractiveHouse Mannequin 3 (DEC_dining_table_vase_302, DEC_dining_table_sphere_004_302, DEC_dining_table_sphere_300) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 02:20 10888 54440
008 InteractiveHouse Mannequin 6 (DEC_dining_table_vase_302, DEC_dining_table_sphere_004_302, DEC_dining_table_sphere_300, Vase_13, Moka_Coffe_27,table_decoration_91) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:35 8485 42425
009 InteractiveHouse Mannequin 0 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:06 5051 25255
003 StudioApartment 05:23 28.768 143.480
000 StudioApartment Mannequin 0 5 (FirstPerson, LeftHand, RightHand, Entrance, Bedroom) 00:38 3366 16380
001 StudioApartment Mannequin 5 (FirstPerson, LeftHand, RightHand, Entrance, Bedroom) 01:51 9793 48965
002 StudioApartment Mannequin 5 (Pot_91, Fruit_Orange_130, Fruit_Apple_133, Moka_Coffee_144, SM_Kitchen_Deco_04_0) 5 (FirstPerson, LeftHand, RightHand, Entrance, Bedroom) 01:28 7829 39145
003 StudioApartment Mannequin 5 (SM_Kitchen_Deco_05, SM_Kitchen_Deco_04, SM_Kitchen_Deco_02, SM_Kitchen_Deco_04_0, Moka_Coffee_144) 5 (FirstPerson, LeftHand, RightHand, Entrance, Bedroom) 01:26 7798 38990
004 BerlinFlat 16:32 88.356 441.780
000 BerlinFlat Mannequin 0 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:33 7847 39235
001 BerlinFlat Mannequin 5 (glass1, glass3, glass4, glass5, glass6) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:45 9414 47070
002 BerlinFlat Mannequin 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:53 9961 49805
003 BerlinFlat Mannequin 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:46 9429 47145
004 BerlinFlat Mannequin 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:42 9204 46020
005 BerlinFlat Mannequin 0 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:13 6531 32655
006 BerlinFlat Mannequin 6 (glass1, glass2, glass3, glass4, glass5, glass6) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:50 9852 49260
007 BerlinFlat Mannequin 5 (chair5, chair6, chair7, chair8, chair9) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:20 7170 35850
008 BerlinFlat Mannequin 6 (glass1, glass2, glass3, glass4, glass5, glass6, ) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:48 9795 48975
009 BerlinFlat Mannequin 9 (glass1, glass2, glass3, glass4, glass5, glass6, table_decoration_44, book1) 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) 01:42 9153 45765
005 Singapore 87.951 439.755
000 Singapore Mannequin 0 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) 01:58 9923 49615
001 Singapore Mannequin 0 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) 01:40 8992 44960
002 Singapore Mannequin 5 (SM_Dining_Glass_01_31, SM_Dining_Glass_02_34, SM_Dining_Glass_3, SM_Dining_Glass_4, SM_Dining_Glass_8, SM_Dining_Glass_9, SM_Dining_Glass_10) 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) 02:26 13119 65595
003 Singapore Mannequin 5 (SM_Kitchen_Tableware_14, SM_Kitchen_Tableware_13, SM_Kitchen_Tableware_12, SM_Cactus_4, SM_Cactus_5) 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) 01:34 7890 39450
004 Singapore Mannequin 5 (SM_Dining_Glass_01_31, SM_Dining_Glass_02_34, SM_Dining_Glass_3, SM_Dining_Glass_4, SM_Dining_Glass_8, SM_Dining_Glass_9, SM_Dining_Glass_10) 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) 02:17 12112 60560
005 Singapore Mannequin 6 (SM_Dinning_Chair_9, SM_Dinning_Chair_7, SM_Dinning_Chair_6, SM_Dinning_Chair_3, SM_Dinning_Chair_2, SM_Dinning_Chair_01_8) 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) 02:32 13601 68005
006 Singapore Mannequin 4 (SM_Dinning_Chair_3, SM_MBook_8, SM_Dinning_Glass_02_34, SM_Kitchen_Tableware_13) 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) 01:41 9147 45735
007 Singapore Mannequin 17 (SM_Dinning_Chair_9, SM_Dinning_Chair_3, SM_Dinning_Chair_2, SM_Dinning_Chair_01_8,
SM_MBook_8, SM_Kitchen_Tableware_14, SM_Kitchen_Tableware_13, SM_Kitchen_Tableware_12,
SM_Kitchen_D7,
SM_Kitchen_D5_90,
SM_Cactus_5,
SM_Cactus_4,
SM_Dinning_Glass_01_31,
SM_Dinning_Glass_02_34,
SM_Dinning_Glass_3,
SM_Dinning_Glass_4,
SM_Dinning_Glass_8)
5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) 02:27 13167 65835
006 WarmHarbor 17:35 499.171
000 WarmHarbor Mannequin Exploration secuence of the entire scene 7 (FirstPerson, LeftHand, RightHand, Kitchen, Lounge, Lounge2, Lounge3) 03:37 19175 134225
001 WarmHarbor Mannequin 5 (SM_SoupPot1, SM_MokaCoffe01, SM_Ginger01, SM_Wok1, SM_Decoration29) 4 (FirstPerson, LeftHand, RightHand, Kitchen) 04:23 23395 93580
002 WarmHarbor Mannequin 6 (SM_Wok1, SM_Melon01, SM_Decoration29, SM_SoupPot1, SM_MokaCoffe01, SM_BarChair2) 4 (FirstPerson, LeftHand, RightHand, Kitchen) 03:16 17389 69556
003 WarmHarbor Mannequin 6 (SM_Decoration28, SM_Decoration23, SM_Side1, SM_Decoration26, SM_Drink5, SM_DiningChair6) 6(FirstPerson, LeftHand, RightHand, Lounge, Lounge2, Lounge3) 03:37 19163 114978
004 WarmHarbor Mannequin 7 (SM_Decoration3, SM_Decoration4, SM_Side1, SM_Decoration27, SM_Drink5, SM_Decoration23, SM_Decoration28) 6(FirstPerson, LeftHand, RightHand, Lounge, Lounge2, Lounge3) 02:42 14472 86832
007 Wooden 473.110
000 Wooden Mannequin 0 (Exploration sequence) 5 (FirstPerson, LeftHand, RightHand, SecondaryRoomCamera, SecondaryRoomCamera2) 01:25 7339 36695
001 Wooden Mannequin 0 (Exploration sequence) 5 (FirstPerson, LeftHand, RightHand, SecondaryRoomCamera, SecondaryRoomCamera2) 01:16 6860 34300
002 Wooden Mannequin 0 (Exploration sequence) 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) 01:31 7915 39575
003 Wooden Mannequin 0 (Exploration sequence) 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) 01:34 8407 42035
004 Wooden Mannequin 3 (SM_Moka_Coffe_01, SM_Dinning_Glass_207, SM_Dinning_Glass_20) 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) 01:35 8452 42260
005 Wooden Mannequin 5 (SM_Pepper_1, SM_KW_00, SM_KSauce_44, SM_KW_02, SM_KW_01) 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) 01:50 9759 48795
006 Wooden Mannequin 9 (SM_Chair_08, SM_Chair_05, SM_Chair_06, SM_Chair_07, SM_Dinning_Glass_07, SM_Dinning_Glass_11, SM_Dinning_Glass_12, SM_Dinning_Glass_04, SM_Dinning_Glass_02_138) 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) 01:28 7858 39290
007 Wooden Mannequin 18 (SM_Dinning_Glass_01_141, SM_Dinning_Glass_02_138,
SM_Dinning_Glass_03,
SM_Dinning_Glass_05,
SM_Dinning_Glass_06,
SM_Dinning_Glass_08,
SM_Dinning_Glass_09,
SM_Dinning_Glass_10,
SM_Dinning_Glass_174,
SM_Dinning_Glas_13,
SM_Dinning_Glass_14,
SM_Dinning_Glass_15,
SM_Dinning_Glass_16,
SM_Dinning_Glass_189,
SM_Dinning_Glass_19,
SM_Dinning_Glass_07, SM_Dinning_Glass_11, SM_Dinning_Glass_12, SM_Dinning_Glass_04)
5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) 04:15 22788 113940
008 Wooden Mannequin 7 (SM_Moka_Coffe_01, SM_Dinning_Glass_207,
SM_Dinning_Glass_20, SM_KW_00, SM_KSauce_44, SM_KW_02, SM_KW_01)
5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) 01:55 9500 47500
009 Wooden Mannequin 1 (SM_Deco_00) 5 (FirstPerson, LeftHand, RightHand, CorridorCamera, DiningCamera) 01:05 5744 28720
008 ModernCozy 339.097
000 ModernCozy Mannequin 01:40 8866 44330
001 ModernCozy Mannequin 02:10 11617 58085
002 ModernCozy Mannequin 01:38 8791 43955
003 ModernCozy Mannequin 02:30 12729 63645
004 ModernCozy Mannequin 01:34 8306 41530
005 ModernCozy Mannequin 00:39 3567 14268
006 ModernCozy Mannequin 01:24 7582 22746
007 ModernCozy Mannequin 03:09 16846 50538

Sequences

Each sequence is recorded as a TXT file that describes it and allows its offline playback to generate the aforementioned data. As a matter of fact, those TXT files are processed and converted into JSON files for improved readability and to make them easier to parse. Those sequence descriptor files contain all the information needed to generate the images and extract ground truth.

Raw Data

For each frame, we provide the following data:

  • 3D poses for the cameras, objects, and robot joints.
  • RGB image at 1920x1080 resolution in 24-bit JPG(95%) format (instead of PNG for reduced size).
  • Depth map at 1920x1080 resolution in 16-bit grayscale PNG format.
  • 2D instance mask at 1920x1080 resolution in RGB 24-bit PNG format.

Ground Truth

For each frame, we provide the tools to generate the following annotations:

  • 2D class mask at 1920x1080 resolution in RGB 24-bit PNG format.
  • 2D/3D object instance bounding boxes in XML format.
  • 3D point cloud in PLY format with RGB color.
  • 3D instance/class mask in PLY format with RGB color.

Due to the excessive size of that large-scale ground truth, we provide the needed tools and instructions so that anyone can generate the annotations they need locally using the aforementioned raw data.

UnrealROX

IMPORTANT

The data for this paper was generated using a deprecated tool which extended UnrealCV to generate all the data we needed. This tool has been superseded by UnrealROX, a compatible and home-brewed C++ solution for UnrealEngine that allows efficient and flexible data recording in virtual reality and offline generation with annotations. Since this tool is better suited for our purposes (more efficient, flexible, and complete), we have removed all previous references in this repository to old RobotriX tools that we mention in the paper (although they can be obtained through commit history).

UnrealROX is described in detail at the following arXiv paper and it is released at the following GitHub/3dperceptionlab/unrealrox repository.

IMPORTANT

Assets

Assets for this project are originated from two sources: UE4Arch and Unreal Engine Marketplace so they can be acquired from there. At this moment, we are still in conversations with both parties to release the modified assets as we used them in our scenes.

Troubleshooting

For any kind of problems, configurations, or instructions, please read carefully UnrealROX's documentation.

We encourage any user to submit any issue related to the data itself using GitHub's built-in issue system within this repository. For any other issue related to the data generation process or tool, please submit the adequate issue to UnrealROX's repository. Improvements and critics are welcome at all fronts!

License

Both the data for The RobotriX and the code for UnrealROX are released under the MIT license.

Citation

If you use this dataset or the UnrealROX tool, please cite:

  • Garcia-Garcia, A., Martinez-Gonzalez, P., Oprea, S., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Jover-Alvarez, A. (2018, October). The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 6790-6797). IEEE.
@inproceedings{garcia2018robotrix,
  title={The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions},
  author={Garcia-Garcia, Alberto and Martinez-Gonzalez, Pablo and Oprea, Sergiu and Castro-Vargas, John Alejandro and Orts-Escolano, Sergio and Garcia-Rodriguez, Jose and Jover-Alvarez, Alvaro},
  booktitle={2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={6790--6797},
  year={2018},
  organization={IEEE}
}
  • Martinez-Gonzalez, P., Oprea, S., Garcia-Garcia, A., Jover-Alvarez, A., Orts-Escolano, S., & Garcia-Rodriguez, J. (2018). UnrealROX: An eXtremely Photorealistic Virtual Reality Environment for Robotics Simulations and Synthetic Data Generation. arXiv preprint arXiv:1810.06936.
@article{martinez2018unrealrox,
  title={UnrealROX: An eXtremely Photorealistic Virtual Reality Environment for Robotics Simulations and Synthetic Data Generation},
  author={Martinez-Gonzalez, Pablo and Oprea, Sergiu and Garcia-Garcia, Alberto and Jover-Alvarez, Alvaro and Orts-Escolano, Sergio and Garcia-Rodriguez, Jose},
  journal={arXiv preprint arXiv:1810.06936},
  year={2018}
}

Contact

Please contact Albert Garcia at agarcia@dtic.ua.es if you have any questions or requests.

About

License:MIT License


Languages

Language:Python 100.0%