Action-GPT: Leveraging Large-scale Language Models for Improved and Generalized Action Generation

Sai Shashank Kalakonda · Shubh Maheshwari · Ravi Kiran Sarvadevabhatla

ICME 2023

Action-GPT provides a plug and play framework for incorporating Large Language Models (LLMs) into text-based action generation models.

For more details please refer to the Paper or the project website.

Description
Getting Started
Running the Demo
Training
Sampling and Evaluation
Citation
Acknowledgments

Description

This implementation:

Can generate motion sequences using Action-GPT framework on TEACH, TEMOS and MotionCLIP for arbitrary text descriptions provided by users.
Can retrain Action-GPT on TEACH, TEMOS and MotionCLIP, allowing users to change details in the training configuration.

Getting started

To install the dependencies please follow the next steps:

Clone this repository:

git clone https://github.com/actiongpt/actiongpt.git
cd actiongpt

Install the dependencies of respective models by following the steps below:
- Action-GPT-TEACH
  - ```
   cd Action-GPT_TEACH_k-4/
```
- Install DistillBERT and requirements from here
- Download and setup the data as mentioned here
- Action-GPT-TEMOS
  - Since TEACH is an extension of TEMOS, the same installation and data setups used for Action-GPT-TEACH can be used here.
- Action-GPT-MotionCLIP
  - ```
   cd Action-GPT_MotionCLIP_k-4/
```
- Install requirements from here
- Download and setup the data as mentioned here
After installing the dependencies of the respective models, install openai as mentioned below to use GPT,
```
pip install openai
```

GPT descriptions

We provided gpt3_annotations.json for all the models which consists of the gpt3 descriptions for the train and test action phrases of the respective data loaders.
OpenAI API key
- OpenAI API key need to be provided to generate GPT-3 descriptions for the action phrases which doesn't exist in gpt3_annotations.json.
- Update your API-Key in gpt_annotator.py

Running the Demo

Action-GPT-TEACH (or) Action-GPT-TEMOS
- Check out the steps to run the demo on any arbitary text descriptions from here
- The path/to/experiment directory is pretrained_model in the respective Action-GPT_TEACH_k-4 or Action-GPT_TEMOS_k-4 directory, which can be downloaded from here.
- NOTE : As Action-GPT-TEMOS is trained for single text descriptions, the demo can be executed for only single text prompts.
Action-GPT-MotionCLIP:
- Check out the steps to generate motion from text here
- The ./exps/paper-model directory is pretrained_model in Action-GPT_MotionCLIP_k-4 directory, which can be downloaded from here

Training

Action-GPT-TEACH (or) Action-GPT-TEMOS
- Check out the steps to train the model from here
Action-GPT-MotionCLIP:
- Check out the steps to train the model from here

Sampling and Evaluation

Action-GPT-TEACH (or) Action-GPT-TEMOS
- Check out the steps to sample and evaluate the model from here
- NOTE : For Action-GPT-TEMOS there are no align and slerp parameters to be passed as the model is trained for single text descriptions.
Action-GPT-MotionCLIP:
- Follow the below command to sample from the test set. The below command creates two directories named ground_truth and action_gpt in the path provided for the parameter generations. Both the directories ground_truth and action_gpt contain the npy files of the motion sequences corresponding to the test set text descriptions. Using the npy files created and the evaluation code provided in TEACH one can generate the metrics as provided in the paper.
- python sampling.py ./exps/paper-model/checkpoint_0100.pth.tar --generations path/to/store/sampled/generations

Citation

@inproceedings{Action-GPT,
  title = {Action-GPT: Leveraging Large-scale Language Models for Improved and Generalized Action Generation},
  author = {Kalakonda, Sai Shashank and Maheshwari, Shubh and Sarvadevabhatla, Ravi Kiran},
  booktitle = {IEEE International Conference on Multimedia and Expo ({ICME})},
  year = {2023},
  url = {https://actiongpt.github.io/}
}

Acknowledgments

This work is an implementation of Action-GPT framework on T2M-models such as TEACH, TEMOS and MotionCLIP. Many part of this code were based on the official implementation of the respective T2M-Models. We thank all the corresponding authors for making thieir code available.

This template was adapted from the GitHub repository of GOAL.

sniler / actiongpt