kakaobrain / flame

Official implementation of the paper "FLAME: Free-form Language-based Motion Synthesis & Editing"

FLAME: Free-form Language-based Motion Synthesis & Editing

[Project Page] | [Paper] | [Video]

Official Implementation of the paper FLAME: Free-form Language-based Motion Synthesis & Editing (AAAI'23)

Generated Samples

Environment

This project is tested on the following environment. Please install them on your running environment.

• Python 3.8
• PyTorch: 1.11
• PyTorch3D: >=0.7.0

Prerequisites

Packages

You may need following packages to run this repo.

apt install libboost-dev libglfw3-dev libgles2-mesa-dev freeglut3-dev libosmesa6-dev libgl1-mesa-glx 

Dataset

❗ We cannot directly provide original data files to abide by the license.

AMASS Dataset

Visit https://amass.is.tue.mpg.de/ to download AMASS dataset. We used SMPL+H G of following datasets in AMASS:

• ACCAD
• BMLhandball
• BMLmovi
• BMLrub
• CMU
• DanceDB
• DFaust
• EKUT
• EyesJapanDataset
• HDM05
• Human4D
• HumanEva
• KIT
• Mosh
• PosePrior
• SFU
• SSM
• TCDHands
• TotalCapture
• Transitions

Downloaded data are compressed in bz2 format. All downloaded files need to be located at data/amass_download_smplhg .

BABEL

Visit https://babel.is.tue.mpg.de/ to download BABEL dataset. At the time of experiment, we used babel_v1.0_release . BABEL dataset should be loacated at data/babel_v1.0_release . File structures under data/babel_v1.0_release looks like:

.
├── extra_train.json
├── extra_val.json
├── test.json
├── train.json
└── val.json

HumanML3D

You can access full HumanML3D dataset at HumanML3D. However, we used original AMASS SMPL data instead of a customized rig. What you will need to prepare to run this repo is:

./data/HumanML3D/
├── humanact12
├── HumanML3D.csv
├── test.txt
├── texts.zip
├── train.txt
└── val.txt

Note that the files above are located at: data/HumanML3D/. Please download humanact12 and HumanML3D.csv. You can download other files from the original repo.

SMPL & DMPL Models

You may need SMPL and DMPL to preprocess motion data. Please refer to AMASS for this. smpl_model and dmpl_model should be located in the project root directory.

SMPL

smpl_model/
├── female
│   └── model.npz
├── info.txt
├── LICENSE.txt
├── male
│   └── model.npz
└── neutral
    └── model.npz

DMPL

dmpl_model/
├── female
│   └── model.npz
├── LICENSE.txt
├── male
│   └── model.npz
└── neutral
    └── model.npz

External Sources

You may need the following packages for visualization.

• VPoser
• PyOpenGL and PyOpenGL_Accelerate

Installation

1. Create a virtual environment and activate it.

   conda create -n flame python=3.8
   conda activate flame

2. Install required packages. Recommend to install corresponding version of PyTorch and PyTorch3D first.

   pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"  # PyTorch3D

3. Install VPoser and PyOpenGL and PyOpenGL_Accelerate from their installation guide.

4. Install other required packages.

   pip install -r requirements.txt

Preprocessing

1. Preprocess AMASS dataset.

   ./scripts/unzip_dataset.sh

   This will unzip downloaded AMASS data into data/amass_smplhg. You can also unzip data manually.

2. Prepare HumanML3D dataset.

   python scripts/prepare_humanml3d.py

3. Prepare BABEL dataset.

   python scripts/prepare_babel_dataset.py

Training

You can train your own model by running the following command. Training configs can be set by config files in configs/ or by command line arguments (hydra format).

python train.py

Testing

Testing takes a long time, since it needs to generate all samples in testset. You need to run test.py with proper config settings at configs/test.yaml. Then, you can run eval_util.py to evaluate the results.

Sampling

Text-to-Motion Generation

Set your sampling config at configs/t2m_sample.yaml. Sampled results will be saved at outputs/. You can export json output to visualize in Unity Engine. Exported json includes the root joint's position and rotation of all other joints in quaternion format.

python t2m_sample.py

Text-to-Motion Editing

Set your text-to-motion editing config at configs/edit_motion.yaml. You can choose a motion to be edited, editing joints, and text prompt. Sampled results will be saved at outputs/.

python edit_motion.py

Joint Index

• 00: Pelvis
• 01: L_Hip
• 02: R_Hip
• 03: Spine1
• 04: L_Knee
• 05: R_Knee
• 06: Spine2
• 07: L_Ankle
• 08: R_Ankle
• 09: Spine3
• 10: L_Foot
• 11: R_Foot
• 12: Neck
• 13: L_Collar
• 14: R_Collar
• 15: Head
• 16: L_Shoulder
• 17: R_Shoulder
• 18: L_Elbow
• 19: R_Elbow
• 20: L_Wrist
• 21: R_Wrist
• 22: L_Hand
• 23: R_Hand

Pretrained Weights

HumanML3D

• Model / mCLIP

BABEL

• Model / mCLIP

Citation

@article{kim2022flame,
  title={Flame: Free-form language-based motion synthesis \& editing},
  author={Kim, Jihoon and Kim, Jiseob and Choi, Sungjoon},
  journal={arXiv preprint arXiv:2209.00349},
  year={2022}
}

License

Copyright (c) 2022 Korea University and Kakao Brain Corp. All Rights Reserved. Licensed under the Apache License, Version 2.0. (see LICENSE for details)