I can't reach the car moderate precision which I trained for 80 epochs
Eaton2022 opened this issue · comments
I trained the CasA-V for 80 epochs, 3 classes., while I found I can't reach car moderate 3D precision which recall is 11 and by the way ,the other two classes reached the precision as you write in paper.
I think my batchsize is too small, it's two. I wonder can I use two batchsize to obtain high precision? How to reset learning rate?
Here is the result.
*************** EPOCH 80 EVALUATION *****************
bbox AP:90.7547, 89.7766, 89.5010
bev AP:90.2147, 88.2992, 88.0006
3d AP:89.6871, 79.6853, 79.2249
aos AP:90.74, 89.67, 89.31
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:96.8148, 95.5741, 93.2506
bev AP:93.6572, 89.8203, 87.5100
3d AP:92.9158, 84.1482, 83.7333
This may be caused by the high score threshold, you can use lower score threshold (0.7->0.6 or 0.7->0.65) in the CasA-V.yaml to test your model.
POST_PROCESSING:
RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
SCORE_THRESH: 0.6 # tuning the score threshold here to obtain good results
OUTPUT_RAW_SCORE: False
Why is the network so sensitive to the threshold? By adjusting this threshold, I also achieved a similar AP to the paper.
bbox AP:98.1848, 89.9826, 89.6064
bev AP:90.3334, 88.6443, 88.1494
3d AP:89.6972, 86.2327, 79.2771
aos AP:98.13, 89.89, 89.46
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:98.9700, 95.7557, 93.3304
bev AP:93.7706, 92.0452, 89.7328
3d AP:92.8850, 86.1569, 83.7628
aos AP:98.93, 95.63, 93.15
Car AP@0.70, 0.50, 0.50:
bbox AP:98.1848, 89.9826, 89.6064
bev AP:98.2297, 89.9412, 89.6159
3d AP:98.2073, 89.9306, 89.5879
aos AP:98.13, 89.89, 89.46
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:98.9700, 95.7557, 93.3304
bev AP:98.9805, 95.7699, 93.3410
3d AP:98.8878, 95.7337, 93.3036
aos AP:98.93, 95.63, 93.15
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:80.2600, 73.4548, 71.9500
bev AP:71.2903, 68.6034, 61.9450
3d AP:70.3801, 67.5366, 60.8072
aos AP:77.47, 70.51, 68.57
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:81.3972, 75.1650, 70.4979
bev AP:74.0048, 68.8206, 62.4292
3d AP:72.8491, 66.2445, 59.8847
aos AP:78.36, 71.88, 67.02
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:80.2600, 73.4548, 71.9500
bev AP:82.9656, 80.9756, 73.6741
3d AP:83.0630, 80.9756, 73.6741
aos AP:77.47, 70.51, 68.57
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:81.3972, 75.1650, 70.4979
bev AP:86.9962, 82.2175, 75.5535
3d AP:87.0576, 82.2166, 75.5526
aos AP:78.36, 71.88, 67.02
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:89.0014, 78.5029, 78.0384
bev AP:87.7107, 74.6787, 67.6883
3d AP:86.5453, 72.9849, 66.3992
aos AP:88.70, 78.05, 77.54
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:94.9959, 79.1564, 76.5699
bev AP:93.3758, 75.2013, 70.7382
3d AP:91.8668, 71.9886, 67.4627
aos AP:94.58, 78.67, 76.05
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:89.0014, 78.5029, 78.0384
bev AP:88.0737, 75.3276, 74.3367
3d AP:88.0737, 75.3276, 74.3367
aos AP:88.70, 78.05, 77.54
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:94.9959, 79.1564, 76.5699
bev AP:93.8242, 75.7949, 73.0799
3d AP:93.8242, 75.7949, 73.0799
aos AP:94.58, 78.67, 76.05
@yiri2022
Hi, what is the threshold set to achieve similar results in the paper? 0.65 ？ 0.6？
:72.8491, 66.2445, 59.8847
0.6. But I can get the result which threshold is 0.7 when I set batchsize is 2 per GPU.
@jlqzzz , @Eaton2022 @yiri2022 ,thanks for discuss this problem, I have updated this code. Now, one can easily attain a good performance by training the default config. We show a CasA-V results reproduced by 2080 GPU with spconv2 :
2022-11-29 02:52:34,960 INFO Average predicted number of objects(3769 samples): 4.774
2022-11-29 02:52:45,250 INFO Car AP@0.70, 0.70, 0.70:
bbox AP:98.5548, 89.9622, 89.6169
bev AP:90.3222, 88.6719, 88.2400
3d AP:89.6201, 86.6608, 79.3748
aos AP:98.51, 89.88, 89.47
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:99.2572, 95.8205, 93.3752
bev AP:96.0735, 92.1997, 89.8395
3d AP:93.0469, 86.4311, 83.9911
aos AP:99.23, 95.70, 93.20