sizhe-li / DexDeform

[ICLR 2023] DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

DexDeform

Code and data for paper DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics at ICLR 2023.

Alt Text

Installation

conda env create -f environment.yml
conda activate dexdeform
pip install -e .
Install Sinkhorn Distance Metric
pip install pykeops
pip install geomloss

Download Demonstrations

Download here. For loading demonstrations, checkout tutorials/demonstration_loading.ipynb.

Tutorials

  • [Environment Loading] tutorials/1_environment_loading.ipynb
  • [Trajectory Optimization] tutorials/2_trajectory_optimization.ipynb
  • [Leap motion tracking module] leap_motion/
  • [Demonstration Loading] tutorials/3_demonstration_loading.ipynb
  • [Computing Score] tutorials/4_computing_score.ipynb

Implementation Details

  • Our simulation backend supports full differentiability and communications with PyTorch modules.
  • For optimal performance, the simulation backend is written in CUDA and implements PlasticineLab.
  • We provide python wrapper for the dexterous hand environment, located inside hand.py.

Acknowledgements

  • Our physics simulation is written based on PlasticineLab.
  • Our leap motion tracking module is written based on this repo.

TODO

  • Support for Human Teleoperation (Leap motion tracking module released, synchronization with simulation coming soon)
  • Release demonstrations
  • Support for DexDeform Algorithm (template uploaded, cleanup needed to support dataloding.)

Citation

@inproceedings{
li2023dexdeform,
title={DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics},
author={Sizhe Li and Zhiao Huang and Tao Chen and Tao Du and Hao Su and Joshua B. Tenenbaum and Chuang Gan},
booktitle={International Conference on Learning Representations},
year={2023},
url={https://openreview.net/forum?id=LIV7-_7pYPl}
}

About

[ICLR 2023] DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics

License:MIT License


Languages

Language:Jupyter Notebook 63.7%Language:Python 23.7%Language:C++ 7.5%Language:Cuda 4.1%Language:Cython 1.0%Language:C 0.0%