patienceFromZhou / simpleHand

This is the project page for paper "A Simple Baseline for Efficient Hand Mesh Reconstruction, CVPR2024"

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Metrics Score reported in Official Git Repo is inconsistent with the Paper

aviralchharia opened this issue · comments

Hi, the metrics scores on Freihand Benchmark seems inconsistent between the official arXiv paper and the official repository. Can the authors comment on which is correct? The score reported in the Github readme is:

while the paper reports:

I have retrained the model and tested it, confirming that the correct results can be obtained. I have placed the checkpoint file and train_log in the Google Drive folder.
It appears to be a printing error from an intermediate version of the evaluation script, normally F@05 would not be higher than F@15.
Below are two metrics. These metrics are obtained from two independent training processes
###########################################
Evaluation 3D KP results:
auc=0.000, mean_kp3d_avg=70.70 cm
Evaluation 3D KP ALIGNED results:
auc=0.887, mean_kp3d_avg=0.57 cm

Evaluation 3D MESH results:
auc=0.000, mean_kp3d_avg=70.69 cm
Evaluation 3D MESH ALIGNED results:
auc=0.881, mean_kp3d_avg=0.60 cm

F-scores
F@5.0mm = 0.0000 F_aligned@5.0mm = 0.7717
F@15.0mm = 0.0000 F_aligned@15.0mm = 0.9858
#############################################
Evaluation 3D KP results:
auc=0.000, mean_kp3d_avg=70.69 cm
Evaluation 3D KP ALIGNED results:
auc=0.886, mean_kp3d_avg=0.57 cm

Evaluation 3D MESH results:
auc=0.000, mean_kp3d_avg=70.69 cm
Evaluation 3D MESH ALIGNED results:
auc=0.880, mean_kp3d_avg=0.60 cm

F-scores
F@5.0mm = 0.0000 F_aligned@5.0mm = 0.7695
F@15.0mm = 0.0000 F_aligned@15.0mm = 0.9862