PhysMoP: Incorporating Physics Principles for Precise Human Motion Prediction

Incorporating Physics Principles for Precise Human Motion Prediction
Yufei Zhang, Jeffrey O. Kephart, Qiang Ji
WACV2024, Paper | SupMat

This repository includes the training and evaluation code for the above paper.

Installation

Environment

conda create -n physmop python=3.8
conda activate physmop
pip install -r requirements.txt

3D Human Model

Please download SMPL and SMPL-H, and store them under the current directory as

|--dataset
|  |--smpl_official
|     |--female
|       |--model.npz
|     |--male
|       |--model.npz
|     |--neutral
|       |--model.npz
|     |-- processed_basicModel_neutral_lbs_10_207_0_v1.0.0.pkl

PhysMoP Model Training & Evaluation

Please follow the procedure below to conduct the experiments on AMASS.

Data Preparation

Download the raw 3D motion sequence data from AMASS (should be similar for Human3.6M and 3DPW), and store them under the current directory as

|--dataset
|  |--data_raw
|     |--AMASS
|       |--ACCAD
|       |--BioMotionLab_NTroje
|       |--CMU
|       |-- .
|       |-- .
|       |-- .
|  |--data_processed

Preprocess the raw data

python AMASS.py

Training

PhysMoP is trained in three steps:

1. Train the data-driven model

python train_script.py --name data --data True --num_epochs 5 --keypoint_loss_weight_data 1 --pose_loss_weight_data 2

2. Train the physics-based model with the checkpoint (DATA_CHECKPOINT) obtained from step 1.

python train_script.py --name data_physics --data True --physics True --pretrained_checkpoint DATA_CHECKPOINT --keypoint_loss_weight_physics_gt 1 --pose_loss_weight_physics_gt 2 --num_epochs 2 --keypoint_loss_weight_data 1 --pose_loss_weight_data 2 --resume

3. Fix the data-driven and physics-based model and train the fusion model with the checkpoint (PHYSICS_CHECKPOINT) obtained from step 2.

python train_script.py --name data_physics_fusion --data True --physics True --fix_weight True --fusion True --pretrained_checkpoint PHYSICS_CHECKPOINT --keypoint_loss_weight_physics_gt 1 --pose_loss_weight_physics_gt 2 --num_epochs 2 --keypoint_loss_weight_data 1 --pose_loss_weight_data 2 --keypoint_loss_weight_fusion 1 --pose_loss_weight_fusion 2

The saved model checkpoint can be found at './logs'.

Evaluation

To evaluate a model saved at CHECKPOINT_PATH, please run

python eval.py --checkpoint CHECKPOINT_PATH

After code re-organization, we re-produced the results as

AMASS-Test	80ms	160ms	320ms	400ms	560ms	720ms	880ms	1000ms
MPJPE (mm)	0.4	2.0	9.4	14.2	24.4	36.1	48.0	61.9

The evaluated checkpoint, which includes model weights and training log, can be downloaded via FINAL_MODEL.

Citation

If you find our work useful, please consider citing the paper:

@InProceedings{Zhang_2024_WACV,
    author    = {Zhang, Yufei and Kephart, Jeffrey O. and Ji, Qiang},
    title     = {Incorporating Physics Principles for Precise Human Motion Prediction},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {January},
    year      = {2024},
    pages     = {6164-6174}
}

If you have questions or encouter any issues when running the code, feel free to open an issue or directly contact me via: zhangy76@rpi.edu.

References

Part of our code is developed based on siMLPe. The SMPL model data is downloaded from SMPL-X model. We thank them for generously sharing their outstanding work.

zhangy76 / PhysMoP