Your open-source robotic mobile manipulation stack!
HomeRobot lets you get started running a range of robotics tasks on a low-cost mobile manipulator, starting with Open Vocabulary Mobile Manipulation, or OVMM. OVMM is a challenging task which means that, in an unknown environment, a robot must:
- Explore its environment
- Find an object
- Find a receptacle -- a location on which it must place this object
- Put the object down on the receptacle.
Check out the Neurips 2023 HomeRobot Open-Vocabulary Mobile Manipulation Challenge!
When you're ready, follow these instructions to participate.
This package assumes you have a low-cost mobile robot with limited compute -- initially a Hello Robot Stretch -- and a "workstation" with more GPU compute. Both are assumed to be running on the same network.
This is the recommended workflow for hardware robots:
- Turn on your robot; for the Stretch, run
stretch_robot_home.pyto get it ready to use. - From your workstation, SSH into the robot and start a ROS launch file which brings up necessary low-level control and hardware drivers.
- If desired, run rviz on the workstation to see what the robot is seeing.
- Start running your AI code on the workstation - For example, you can run
python projects/stretch_grasping/eval_episode.pyto run the OVMM task.
We provide a couple connections for useful perception libraries like Detic and Contact Graspnet, which you can then use as a part of your methods.
HomeRobot requires Python 3.9. Installation on a workstation requires conda and mamba. Installation on a robot assumes Ubuntu 20.04 and ROS Noetic.
To set up the hardware stack on a Hello Robot Stretch, see the ROS installation instructions in home_robot_hw.
You may need a calibrated URDF for our inverse kinematics code to work well; see calibration notes.
Follow the network setup guide to set up your robot to use the network, and make sure that it can communicate between workstation and robot via ROS. On the robot side, start up the controllers with:
roslaunch home_robot_hw startup_stretch_hector_slam.launch
To set up your workstation, follow these instructions. We will assume that your system supports CUDA 11.8 or better for pytorch; earlier versions should be fine, but may require some changes to the conda environment.
If necessary, install mamba in your base conda environment. Optionally: install ROS noetic on your workstation.
# Create a conda env - use the version in home_robot_hw if you want to run on the robot
mamba env create -n home-robot -f src/home_robot_hw/environment.yml
# Otherwise, use the version in src/home_robot
mamba env create -n home-robot -f src/home_robot/environment.yml
conda activate home-robot
This should install pytorch; if you run into trouble, you may need to edit the installation to make sure you have the right CUDA version. See the pytorch install notes for more.
Optionally, setup a catkin workspace to use improved ROS visualizations.
Make sure you have the correct environment variables set: CUDA_HOME should point to your cuda install, matching the one used by your python environment. We recommend 11.7, and it's what will be automatically installed above. You can download it from nvidia's downloads page. Download the runfile, and make sure to check the box NOT to install your drivers.
Then make sure the environment variables are set to something reasonable:
HOME_ROBOT_ROOT=$USER/src/home-robot
CUDA_HOME=/usr/local/cuda-11.7
Finally, you can run the install script to download submodules, model checkpoints, and build Detic for open-vocabulary object detection:
conda activate home-robot
cd $HOME_ROBOT_ROOT
./install_deps.sh
If you run into issues, check out the step-by-step instructions.
To set up the simulation stack with Habitat, train DDPPO skills and run evaluations: see the installation instructions in home_robot_sim.
For more details on the OVMM challenge, see the Habitat OVMM readme. You can start by running the install script to download all the necessary data:
$HOME_ROBOT_ROOT/projects/habitat_ovmm/install.sh
You should then be able to run the Stretch OVMM example.
Run a grasping server; either Contact Graspnet or our simple grasp server.
# For contact graspnet
cd $HOME_ROBOT_ROOT/src/third_party/contact_graspnet
conda activate contact_graspnet_env
python contact_graspnet/graspnet_ros_server.py --local_regions --filter_grasps
# For simple grasping server
cd $HOME_ROBOT_ROOT
conda activate home-robot
python src/home_robot_hw/home_robot_hw/nodes/simple_grasp_server.py
Then you can run the OVMM example script:
cd $HOME_ROBOT_ROOT
python projects/real_world_ovmm/eval_episode.py
We welcome contributions to HomeRobot.
There are two main classes in HomeRobot that you need to be concerned with:
- Environments extend the abstract Environment class and provide observations of the world, and a way to apply actions.
- Agents extend the abstract Agent class, which takes in an observation and produces an action.
Generally, new methods will be implemented as Agents.
See the robot hardware development guide for some advice that may make developing code on the Stretch easier.
HomeRobot is broken up into three different packages:
| Resource | Description |
|---|---|
| home_robot | Core package containing agents and interfaces |
| home_robot_sim | OVMM simulation environment based on AI Habitat |
| home_robot_hw | ROS package containing hardware interfaces for the Hello Robot Stretch |
The home_robot package contains embodiment-agnostic agent code, such as our ObjectNav agent (finds objects in scenes) and our hierarchical OVMM agent. These agents can be extended or modified to implement your own solution.
Importantly, agents use a fixed set of interfaces which are overridden to provide access to
The home_robot_sim package contains code for interface
We use linters for enforcing good code style. The lint test will not pass if your code does not conform.
Install the git pre-commit hooks by running
python -m pip install pre-commit
cd $HOME_ROBOT_ROOT
pre-commit installTo format manually, run: pre-commit run --show-diff-on-failure --all-files
Home Robot is MIT licensed. See the LICENSE for details.
- hello-robot/stretch_body
- Base API for interacting with the Stretch robot
- Some scripts for interacting with the Stretch
- hello-robot/stretch_ros
- Builds on top of stretch_body
- ROS-related code for Stretch
- RoboStack/ros-noetic
- Conda stream with ROS binaries