About confusing comparison in experiments
echoofluoc opened this issue · comments
In 4.1, you guys say you use the improved training annotationsfrom [22] and test annotations from [25]. so the results you obtained is evaluated on the improved test annotations , but in Table 3, methods above MSDS-RCNN like IATDNN+IAMSS with 26.37MR are reported on the original test annotations and it has a 14.95MR performance on the improved test annotations as you used, but methods like MSDS-RCNN and your results are reported on the improved test annotations,
this also happens in other comparison experiments like single-modality detectors in table 2, which is quite unfair and confusing
[22]Li, C., Song, D., Tong, R., Tang, M.: Multispectral pedestrian detection via simul-taneous detection and segmentation. In: Proc. of British Machine Vision Confer-ence (BMVC) (2018)
[25]Liu, J., Zhang, S., Wang, S., Metaxas, D.N.: Multispectral deep neural networksfor pedestrian detection. arXiv preprint arXiv:1611.02644 (2016)
Hi, could you tell me where to find the result of IATDNN on the improved set? their paper only showed on the original set? For results with *, I have their detector's results and evaluated on the improved test set by the provided evaluation tool. The remaining papers (we did not have detector's results), we cited from their official papers. If you get any detector's results and tested on the provided evaluation tool, it does not guarantee a better result, even it's worse in some cases. Thanks & best regards.
please refer to https://github.com/CalayZhou/MBNet
In 6. Comparison with other Methods , the author have collected some SOTA results