hejia-zhang / manicast

ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting

Website | Paper

This is a repository containing datasets, visualizations and model checkpoints for the CoRL 2023 paper:
ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting
Kushal Kedia, Prithwish Dan, Atiksh Bhardwaj, Sanjiban Choudhury

Real-World Collaborative Manipulation

Our framework ManiCast learns cost-aware human motion forecasts and plans with such forecasts for collaborative manipulation tasks.

At train time, we fine-tune pre-trained human motion forecasting models on task specific datasets by upsampling transition points and upweighting joint dimensions that dominate the cost of the robot's planned trajectory. At inference time, we feed these forecasts into a model predictive control (MPC) planner to compute robot plans that are reactive and keep a safe distance from the human.

Collaborative Manipulation Dataset (CoMaD)

We release a high-quality dataset collected using a motion capture system, consisting of two humans collaborating to perform daily household activities. Episodes of each task along with train, validation and test splits can be found here.

Setup

Setup environments following the SETUP.md

We release checkpoints of all the models used in our paper. The next two sections provide instructions on how to use these models.

Visualization

Play any data episode via any model.

python eval/comad_visualization.py --data_dir {handover, reactive_stirring, table_setting} --visualize_from {train, val, test} --ep_num <EPISODE_NUMBER> --load_path <MODEL_NAME>

Python notebook demo through eval/comad_visualization.ipynb.

Evaluation

Generate evaluation metrics on Object Handovers.

python eval/handover.py --ep_num 2

Generate evaluation metrics on Reactive Stirring.

python eval/reactive_stirring.py --ep_num 4

Generate evaluation metrics on CoMaD Dataset.

python eval/test_comad.py

Training

Pretrain model on large-scale data (requires following SETUP.md to install AMASS data).

python src/pretrain.py --weight 1

Finetune the above models using the ManiCast framework. Add --load_path default/<epoch num> to load a model trained with pretrain.py. Upweighting wrist/hand joints can be done easily through the --weight command line argument.

python src/finetune.py --weight 1 `--load_path default/<epoch num>`

Acknowledgement

This repository borrows code from STS-GCN.

Citation

@inproceedings{kedia2023manicast,
    title={ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting},
    author={Kushal Kedia and Prithwish Dan and Atiksh Bhardwaj and Sanjiban Choudhury},
    booktitle={7th Annual Conference on Robot Learning},
    year={2023},
    url={https://openreview.net/forum?id=rxlokRzNWRq}
}