Metrics Score reported in Official Git Repo is inconsistent with the Paper
aviralchharia opened this issue · comments
I have retrained the model and tested it, confirming that the correct results can be obtained. I have placed the checkpoint file and train_log in the Google Drive folder.
It appears to be a printing error from an intermediate version of the evaluation script, normally F@05 would not be higher than F@15.
Below are two metrics. These metrics are obtained from two independent training processes
###########################################
Evaluation 3D KP results:
auc=0.000, mean_kp3d_avg=70.70 cm
Evaluation 3D KP ALIGNED results:
auc=0.887, mean_kp3d_avg=0.57 cm
Evaluation 3D MESH results:
auc=0.000, mean_kp3d_avg=70.69 cm
Evaluation 3D MESH ALIGNED results:
auc=0.881, mean_kp3d_avg=0.60 cm
F-scores
F@5.0mm = 0.0000 F_aligned@5.0mm = 0.7717
F@15.0mm = 0.0000 F_aligned@15.0mm = 0.9858
#############################################
Evaluation 3D KP results:
auc=0.000, mean_kp3d_avg=70.69 cm
Evaluation 3D KP ALIGNED results:
auc=0.886, mean_kp3d_avg=0.57 cm
Evaluation 3D MESH results:
auc=0.000, mean_kp3d_avg=70.69 cm
Evaluation 3D MESH ALIGNED results:
auc=0.880, mean_kp3d_avg=0.60 cm
F-scores
F@5.0mm = 0.0000 F_aligned@5.0mm = 0.7695
F@15.0mm = 0.0000 F_aligned@15.0mm = 0.9862