Accuracy of models trained form scratch
XueDx opened this issue · comments
Hi @tteepe
I am very interested in this work and have read your paper.
I tried to train the resgcn model from scratch on casia-b dataset (by download_data.sh) following experiments/1_train_casia-b.sh .
As shown blew, the accuracy of best models are lower than gaitgraph_resgcn-n39-r8_coco_seq_60.pth.
The mean acc of CL is 50.8 (51.1 after tune) while that of gaitgraph_resgcn-n39-r8_coco_seq_60.pth is 66.3.
Anything wrong in my experiment settings? Does it need a pretrained weight file from https://github.com/yfsong0709/ResGCNv1?
- experiments/1_train_casia-b.sh
| 0 | 18 | 36 | 54 | 72 | 90 | 108 | 126 | 144 | 162 | 180 | mean | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NM#5-6 | 75.1 | 77.4 | 81.4 | 82.9 | 78.6 | 80.5 | 84 | 78.5 | 82.6 | 77.7 | 70.8 | 79 |
| BG#1-2 | 65.2 | 67.4 | 73 | 67.7 | 66.3 | 67.7 | 65.8 | 65.2 | 65 | 70.3 | 60.7 | 66.8 |
| CL#1-2 | 51 | 51.8 | 41.6 | 49 | 51.2 | 49.7 | 54.9 | 54.7 | 52.4 | 52.6 | 50 | 50.8 |
- experiments/1_train_casia-b_fine.sh
| 0 | 18 | 36 | 54 | 72 | 90 | 108 | 126 | 144 | 162 | 180 | mean | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NM#5-6 | 77.9 | 80.1 | 80.9 | 82.9 | 78.4 | 81.5 | 84.7 | 77.5 | 82 | 78.7 | 71.6 | 79.7 |
| BG#1-2 | 67.5 | 69.5 | 73.6 | 70.1 | 67 | 67.8 | 66.4 | 65.2 | 64.4 | 71.5 | 61.4 | 67.7 |
| CL#1-2 | 52.4 | 52.4 | 43.5 | 48 | 52.2 | 51.4 | 54.8 | 53 | 51.8 | 53.2 | 49.6 | 51.1 |
Hi,
thanks for your interest!
Look like to me that you are using the weights at the end of each training. Due to the overfitting setup of the training, these won't be that best weights.
Could you re-run the evaluation on the weights that are called ckpt_epoch_best.pth?
Also the fine-tuning should be done on the best weights of the initial training.
If that's not the case, could you send me the accuracy graph from tensorboard?
Cheers,
Torben
Both evaluate and fine-tune are done on the best.pth
I have tried to train resgcn-n39-r8 instead of resgcn-n39-r4, the results are similar.
- 1_train_casia-b.sh orange line
- 1_train_casia-b_fine.sh red line
- gaitgraph_resgcn-n39-r8_coco_seq_60.pth gray line
okay, I just realized the experiment files were outdated. Sorry about that.
I pushed the updated parameters that also now match the description in the paper.
Could you pull the new files and re-run the experiments and let me if you can reproduce the results now?
Hello, I also met the same problem, may I ask how you solve it, thank you for your reply.
@tteepe can you point me to the checkpoint file "ckpt_epoch_best.pth" you mentioned in the comments above.
Hi @tteepe Thanks for your code! I recently trained the model following your instructions (using configurations in "experiments/1_train_casia-b.sh & 1_train_casia-b_fine.sh") and got the following performance
| 0 | 18 | 36 | 54 | 72 | 90 | 108 | 126 | 144 | 162 | 180 | mean | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NM#5-6 | 83.7 | 84.6 | 87.2 | 88.8 | 85.3 | 86.2 | 85.6 | 85 | 83.2 | 84.1 | 78.9 | 84.8 |
| BG#1-2 | 73.9 | 74.4 | 74.4 | 73.3 | 67.6 | 70.2 | 71.5 | 71.7 | 71 | 74.4 | 60.5 | 71.2 |
| CL#1-2 | 64 | 63.4 | 65.1 | 62.4 | 57.9 | 59.8 | 67.3 | 57.2 | 62.3 | 62.8 | 59.2 | 61.9 |
There's still a performance gap compared with the reported results. Would you please kindly check the training configurations and see if there's any problems? Thanks very much.
Hi, I want to know why trained twice 1_train_casia-b.sh and 1_train_casia-b_fine.sh?
Hi, I want to know why trained twice 1_train_casia-b.sh and 1_train_casia-b_fine.sh?