Evaluation of pretrained model on HICO-Det is lower than reported.
boringwar opened this issue · comments
Hi, I downloaded your CDN-S/B/L pretrained model, and evaluate them locally with your script.
The results are all lower than the reported results. Specifically,
Model S mAP: 0.3150153102455715 mAP rare: 0.27220402283111156 mAP non-rare: 0.32780309739534524 mean max recall:
0.6390865334740454
Model B mAP: 0.31671305044814135 mAP rare: 0.27124143008976015 mAP non-rare: 0.3302954825032422 mean max recall:
0.6436087461863998
Model L mAP: 0.3196075598370539 mAP rare: 0.2724063291553318 mAP non-rare: 0.3337066287419839 mean max recall:
0.6492344626931886
I'm wondering what caused the inconsistency. what's the version of opencv-python in your environment? I'm using opencv 4.5.1.
The reported results are evaluated with batch size 16 (2x8 or 4x4) by the official Matlab script. The inconsistency may be caused by such two aspects.
OK, I'll check it.