模型在训练的时候很奇怪

Question

模型在训练的时候很奇怪

cxq1 opened this issue a year ago · comments

不知道为什么训练的时长会突然变长，很奇怪

Vanilla · Answer 1 · Mon Dec 04 2023 22:20:59 GMT+0800 (China Standard Time)

不知道为什么训练的时长会突然变长，很奇怪

Hello, this occurs because the angle branch is introduced after the "burn-in step 1" phase (as mentioned in the paper). You can adjust "burn_in_steps1" and "burn_in_steps2" in the config file.

Chen Xiangqi · Answer 2 · Mon Dec 04 2023 22:26:21 GMT+0800 (China Standard Time)

thanks for your reply. it's a great work!!
How long will it take you to train the model using two Gpus

Chen Xiangqi · Answer 3 · Mon Dec 04 2023 22:28:56 GMT+0800 (China Standard Time)

Whether the model training will become inaccurate, when the training is interrupted, retraining re-enters the ""burn-in step 1" phase

Vanilla · Answer 4 · Mon Dec 04 2023 22:37:54 GMT+0800 (China Standard Time)

thanks for your reply. it's a great work!! How long will it take you to train the model using two Gpus

If you use the default configs, it takes about 15 hours on the DOTA-v1.0 and about 16 hours on DIOR-R.

Vanilla · Answer 5 · Mon Dec 04 2023 22:41:35 GMT+0800 (China Standard Time)

Whether the model training will become inaccurate, when the training is interrupted, retraining re-enters the ""burn-in step 1" phase

If interrupted, it is recommended to manually set the "iter_count" in the configs to the state at which the training was interrupted. Then, follow the resume procedure provided by MMDetection.

Chen Xiangqi · Answer 6 · Tue Dec 05 2023 08:58:48 GMT+0800 (China Standard Time)

Whether the model training will become inaccurate, when the training is interrupted, retraining re-enters the ""burn-in step 1" phase

If interrupted, it is recommended to manually set the "iter_count" in the configs to the state at which the training was interrupted. Then, follow the resume procedure provided by MMDetection.

What version of mmcv are you using? Is pytorch the default 1.9? I was curious, why did I train on an A100 GPU for more than 5 days

Vanilla · Answer 7 · Tue Dec 05 2023 10:33:16 GMT+0800 (China Standard Time)

Whether the model training will become inaccurate, when the training is interrupted, retraining re-enters the ""burn-in step 1" phase

If interrupted, it is recommended to manually set the "iter_count" in the configs to the state at which the training was interrupted. Then, follow the resume procedure provided by MMDetection.

What version of mmcv are you using? Is pytorch the default 1.9? I was curious, why did I train on an A100 GPU for more than 5 days

My environment:

PyTorch: 1.9.0
CUDA Runtime: 11.1
CuDNN: 8.0.5
TorchVision: 0.10.0
OpenCV: 4.8.1
MMCV: 1.4.5
MMDetection: 2.13.0+c820f32