How to evaluate on HumanAct12?

[fyyakaxyy]

Hi, first of all, thank you for providing this excellent project! I am currently running code to evaluate the effectiveness of MoDi.
Currently, I have completed some of the testing, but have encountered some issues.

1. How to evaluate the performance of the model on the HumanAct12 dataset? I haven't found relevant data or prompts. The paper states that for evaluation on the Mixamo dataset, 2000 results will be sampled; Evaluate on the HumanAct12 dataset and sample 1000 results. The program I am currently running is:

   python evaluate.py --ckpt data/pretrained_model/ckpt.pt --path data/edge_rot_data.npy --act_rec_gt_path data/motion.npy
   

   python evaluate.py --ckpt data/pretrained_model/ckpt.pt --path data/train_edge_rot_data.npy --act_rec_gt_path data/motion.npy
   

   They display 2000 sampled results, so they are all from the Mixamo dataset. Results:
   

2. I completed some tests on the Mixamo dataset and used 2 different data:edge_rot_data.npy and train_edge_rot_data.npy, all of which were able to complete the tests. What are the differences between them?

[fyyakaxyy]

Hello, I processed the dataset using the data from the HumanAct12 project and the code you provided, and obtained humanact12motion.npy. I used these two commands to test the performance of the pre trained model on HumanAct12,

python evaluate.py --ckpt data/pretrained_model/ckpt.pt --path data/edge_rot_data.npy --act_rec_gt_path evaluation/data/humanact12motion.npy

python evaluate.py --ckpt data/pretrained_model/ckpt.pt --path data/train_edge_rot_data.npy --act_rec_gt_path evaluation/data/humanact12motion.npy

but the results obtained were significantly different from the data in your paper. The FID actually exceeded 100! while the FID in the paper was below 20.

So I want to ask, is this an error in my code? Which of the two instructions should be used?

[Inbal-leibovitch]

Hello,
The provided pretrained model was trained on the Mixamo dataset. To evaluate on the HumanAct12 dataset you cannot use the provided model, you need to use a model that was trained on HumanAct12.
The two datasets have different distributions, explaining the high FID in the examples you provided.

As for the difference between edge_rot_data.npy and train_edge_rot_data.npy -
edge_rot_data.npy is the data the model was trained on (Mixamo) and is the file you should use. train_edge_rot_data.npy is a train split relevant for the encoder only.

[fyyakaxyy]

Hello, The provided pretrained model was trained on the Mixamo dataset. To evaluate on the HumanAct12 dataset you cannot use the provided model, you need to use a model that was trained on HumanAct12. The two datasets have different distributions, explaining the high FID in the examples you provided.

As for the difference between edge_rot_data.npy and train_edge_rot_data.npy - edge_rot_data.npy is the data the model was trained on (Mixamo) and is the file you should use. train_edge_rot_data.npy is a train split relevant for the encoder only.

Thank you very much for your answer!
Could you please provide me with a pretrained model on the HumanAct12 dataset to evaluate MoDi on HumanAct12?

[Inbal-leibovitch]

Sure,
Use this pretrained model and humanact12motion.npy data file.

To run the evaluation use:
python evaluate.py --dataset humanact12 --ckpt MoDi_humanact12_079999.pt --path humanact12motion.npy

[fyyakaxyy]

Sure, Use this pretrained model and humanact12motion.npy data file.

To run the evaluation use: python evaluate.py --dataset humanact12 --ckpt MoDi_humanact12_079999.pt --path humanact12motion.npy

Thank you for your answer and the provided documents. My problem has been resolved!