QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfer

Project | Gradio Demo | Video

The implementation of the paper QuasiSim, presenting a parameterized family of quasi-physical simulators for transferring kinematics-only human manipulation demonstrations to dexterous robot hand simulations. We tackle the optimization challenge posed by the complex dynamics inherent in dexterous manipulation by solving the control trajectory optimization problem through a physics curriculum.

demo-video-4-compressed.mp4

We enable accurately controlling a simulated dexterous robot hand to track complex human manipulations with changing contact, non-trivial object motions, and intricate tool-using in a physically realistic simulated environment.

The repository contains

Analytical part of the parameterized quasi-physical simulator;
Detailed instructions on the optimization process for a manipulation sequence example (first two stages).

We will add the remaining code and instructions on the last optimization stage, as well as the data and more manipulation examples.

Getting Started

This code was tested on Ubuntu 20.04.5 LTS and requires:

Python 3.8.8
conda3 or miniconda3
CUDA capable GPU (one is enough)

Environment setup

Creat a virtual environment and install necessary dependencies

Create a virtual environment

conda create -n quasisim python==3.8.8
conda activate quasisim

Install torch2.2.0+cu121

conda install pytorch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 pytorch-cuda=12.1 -c pytorch -c nvidia

Install torch_cluster:

mkdir whls
cd whls
wget https://data.pyg.org/whl/torch-2.2.0%2Bcu121/torch_cluster-1.6.3%2Bpt22cu121-cp38-cp38-linux_x86_64.whl
pip install torch_cluster-1.6.3+pt22cu121-cp38-cp38-linux_x86_64.whl

Install other dependences

pip install -r requirements.txt

Setup DiffHand:

cd DiffHand
cd core
python setup.py install
### For testing the installation ###
cd ..
cd examples
python test_redmax.py

It's better to install it from this project since we have made modifications to support our purpose.

Get data

Examples data

Download the example data and organize them in the same way as we do in OnDrive. The expected file structure is as follows:

data
 |-- grab
     |-- 102
         |-- 102_obj.npy
         |-- 102_obj.obj
         |-- 102_sv_dict.npy
         |-- 102_sv_dict_st_0_ed_108.npy
     ...

Checkpoints

Download pre-optimized checkpoints and organize them in the same way as we do in OnDrive. The expected file structure is as follows:

ckpts
 |-- grab
     |-- 102
         |-- xxx1.pt
         |-- xxx2.pt
     ...

URDFs and others

Download rsc.zip and raw_data.zip. Extract them in the root folder of the project.

Important: Download mano_pts.zip. Extract them in the folder rsc/mano.

Example Usage

We provide the data, detailed instructions, and the results of a specific example with the aim of elucidating the inverse dynamics problem within contact-rich manipulation scenarios, which we address leveraging the physics curriculum programmed from QuasiSim. Currently, we release the analytical part of QuasiSim as well as the first two optimization stages. The thrid stage, alongside with more data and their optimization code will be added.

In the case of the example sequence data/grab/102, which depicts a human hand rotating a mouse, the human manipulation demonstration, transferred manipulation to the simulated Shadow hand in the QuasiSim's analytical environment, and the manipulation optimized in the Bullet simulator are shown as follows.

Human Manipulation	Transferred to Shadow	Transferred to Shadow in Bullet

The following instructions aim to optimize the control trajectory to enable the Shadow hand to complete the tracking task, as demonstrated in the middle demo, within the stiffest analytical environment of QuasiSim.

Stage 1

Transferring human demonstrations via point set dynamics. In this stage, we represent the MANO model and the Shadow model within the simulation as parameterized point sets. The contact model is adjusted to its softest level. The objective is to optimize a trajectory of point sets for the Shadow hand to successfully accomplish manipulation tracking.

This stage is divided into four steps as described below.

Step 1: Optimizing the trajectory of the simulated dynamic MANO hand

In this step, we optimize a control trajectory for the dynamic MANO hand model to track the reference manipulation. Please execute the following commands sequentially:

bash scripts_new/train_grab_mano.sh
bash scripts_new/train_grab_mano_wreact.sh
bash scripts_new/train_grab_mano_wreact_optacts.sh

Step 2: Optimizing the control trajectory for the point set constructed from the MANO hand

Execute the following four commands sequentially for this step:

bash scripts_new/train_grab_pointset_points_dyn_s1.sh
bash scripts_new/train_grab_pointset_points_dyn_s2.sh
bash scripts_new/train_grab_pointset_points_dyn_s3.sh
bash scripts_new/train_grab_pointset_points_dyn_s4.sh

Step 3: Optimizing the trajectory of the kinematic Shadow hand

In this step, we optimize the trajectory of the kinematic Shadow hand based on keypoint-based correspondences and mesh surface point-based correspondences. Execute the following command for this step:

bash scripts_new/train_grab_sparse_retar.sh

Step 4: Optimizing the control trajectory for the point set constructed from the simulated Shadow hand

Please execute the following commands for this step:

bash scripts_new/train_grab_pointset_points_dyn_retar.sh
bash scripts_new/train_grab_pointset_points_dyn_retar_pts.sh

Stage 2

Tracking via a contact curriculum. In this stage, we optimize the control trajectory of the simulated Shadow hand to complete the tracking task through a curriculum of contact models. Initially, the contact model is tuned to the softest level. Subsequently, we gradually adjust parameters to tune it to the stiffest level.

Run the following command:

bash scripts_new/train_grab_stage_2_dm_curriculum.sh

TODOs

Contact

Please contact xymeow7@gmail.com or raise a github issue if you have any questions.

Acknowledgments

This code is standing on the shoulders of giants. We want to thank the following contributors that our code is based on: DiffHand and NeuS.

License

This code is distributed under an MIT LICENSE.

KailinLi / QuasiSim