my results are lower than what is indicated
sofsoo1995 opened this issue · comments
Congratulation for your COCO Challenge result, and thank you for sharing your code.
I'm testing your code and the problem is I have these results on the validation set of 2014:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.430
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.682
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.458
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.385
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.511
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.534
Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.780
Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.566
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.463
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.631
-I'm using ResNet-101 with an input size of 384x288
-I haven't almost changed anything in the code(except the config file, in mptest I put dump_method=1 as argument for the function MultiProc).
-I'm using the pretrained model you have trained.
-Also, for the dataset, I have downloaded the 2014 version(train and val).
-I'm using the annotation file and also the bounding box you've given.
-I don't understand why is there a huge difference between your results and mine ! Have I done something wrong ?
I've downloaded the pretrained model from web and ran it again using 8 gpus, got 72.9.
Have you tried other pretrained model? Did they get the right result?
Thx you for your answers,
The results I'm reporting is with the pretrained model and not my training. I only have tried res101.384x288 and res50.256x192 and the results are approximately the same(I've just reported res101.384x288).
qualitatively, the model works on simple cases but it does a lot of errors for more complicated cases.
I will try the other models and will also try to train(with 2 Titan X and 1 GTX 1080 )
Do you check the validation dataset?
minival dataset: https://drive.google.com/drive/folders/15loPFQCMQnJqLK1viSMeIwTFT-KbNzdG
minival det: https://drive.google.com/drive/folders/1BllF9--dN9uV3FRROcmuIbwNCcn7cCP0?usp=sharing
Ok, I've re downloaded pre trained model and I've re tested everything.
and actually res50.256x192 has good results (as it is indicated) !
However the other methods failed (res50.384x288, and res101.384x288). So the problem might come from the input size.
Is it possible that it is a problem of hardware(because I only use 3 GPUs) ?
I put the details here(with pretrained model):
res50.256x192
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.697
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.883
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.770
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.662
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.761
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.764
Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.927
Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.823
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.715
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.830
res50.384x288
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.206
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.459
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.162
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.170
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.268
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.310
Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.591
Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.280
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.250
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.392
res101.384x288
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.430
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.682
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.458
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.385
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.511
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.534
Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.780
Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.566
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.463
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.631
I think it can't be the problem of hardware. Since it's worse in 384x288 size, did you run the pre-trained model in the corresponding model folder?
I found the mistake. I modified config and with silly copy-pastes I put the wrong i input size in the two networks. My Bad !
However, I still don't understand why is there this huge difference of results if I put a wrong input size for testing.
But thank you very much for helping me !