After 24 epoches trained, its loss still can't reach convergence

Question

After 24 epoches trained, its loss still can't reach convergence

GulpFire opened this issue 2 years ago · comments

`2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000
2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070
2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874
2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832
2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs
2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox...
2022-11-20 01:45:01,473 - mmdet - INFO -
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016

2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py
2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001
`

ZeroCodePro · Answer 1 · Thu Nov 24 2022 23:00:06 GMT+0800 (China Standard Time)

`2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000 2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070 2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874 2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832 2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs 2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox... 2022-11-20 01:45:01,473 - mmdet - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016

2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py 2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001 `

Me too, How did you solve it?

GulpFire · Answer 2 · Fri Nov 25 2022 11:37:23 GMT+0800 (China Standard Time)

2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000 2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070 2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874 2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832 2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs 2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox... 2022-11-20 01:45:01,473 - mmdet - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016 2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py 2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001

Me too, How did you solve it?

我没能复现，但你或许可以试试这个方案https://github.com/felix-schmitt/FormulaNet

ZeroCodePro · Answer 3 · Fri Nov 25 2022 16:13:08 GMT+0800 (China Standard Time)

2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000 2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070 2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874 2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832 2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs 2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox... 2022-11-20 01:45:01,473 - mmdet - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016 2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py 2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001

Me too, How did you solve it?

我没能复现，但你或许可以试试这个方案https://github.com/felix-schmitt/FormulaNet
mfd单卡训练降低学习率试试

ZeroCodePro · Answer 4 · Sat Nov 26 2022 22:30:21 GMT+0800 (China Standard Time)

8个gpu完全一样的配置文件，loss训练几轮之后就nan了，[FormulaNet]的数据集太难搞了

GulpFire · Answer 5 · Mon Nov 28 2022 09:41:48 GMT+0800 (China Standard Time)

2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000 2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070 2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874 2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832 2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs 2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox... 2022-11-20 01:45:01,473 - mmdet - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016 2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py 2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001

Me too, How did you solve it?

我没能复现，但你或许可以试试这个方案https://github.com/felix-schmitt/FormulaNet
mfd单卡训练降低学习率试试

我试过调整学习率，放大或缩小一个数量级，但并没有什么效果

yuxzho · Answer 6 · Sun Dec 25 2022 11:26:06 GMT+0800 (China Standard Time)

@GulpFire @ZeroCodePro Hi, the bug was fixed, please pull the latest config file and take it a try.