Below is a demo video of navigating to the target table (highlighted with red mask) in apartment_0 using habitat-sim simulator (Link).
In order to navigate to certain area, the agent needs to have knowledge about the environment first.
Walking through the first floor (left figure) and second floor (right figure) in apartment_0 meanwhile collecting rgbd and semantic images at each steps.
Unproject the depth image into 3D point clouds, then align point clouds to the first frame using ICP algorithm, we obtain the 3D rgb maps of first floor (left figure) and second floor (right figure) in apartment_0.
After creating maps for the environment, the agent can compute a collision-free trajectory and navigate to the target.
With 3D semantic map (left figure), the agent knows about the location of each objects. In this example, we choose to navigate to the target table (middle figure).
Using RRT algorithm, the agent can compute a collision-free trajectory to the target starting from current position (right figure).
The agent will perfrom left, right, forward these three actions according to the computed trajectory. The result can be seen on the top demo video.
