模型在训练的时候很奇怪
cxq1 opened this issue · comments
thanks for your reply. it's a great work!!
How long will it take you to train the model using two Gpus
Whether the model training will become inaccurate, when the training is interrupted, retraining re-enters the ""burn-in step 1" phase
thanks for your reply. it's a great work!! How long will it take you to train the model using two Gpus
If you use the default configs, it takes about 15 hours on the DOTA-v1.0 and about 16 hours on DIOR-R.
Whether the model training will become inaccurate, when the training is interrupted, retraining re-enters the ""burn-in step 1" phase
If interrupted, it is recommended to manually set the "iter_count" in the configs to the state at which the training was interrupted. Then, follow the resume procedure provided by MMDetection.
Whether the model training will become inaccurate, when the training is interrupted, retraining re-enters the ""burn-in step 1" phase
If interrupted, it is recommended to manually set the "iter_count" in the configs to the state at which the training was interrupted. Then, follow the resume procedure provided by MMDetection.
What version of mmcv are you using? Is pytorch the default 1.9? I was curious, why did I train on an A100 GPU for more than 5 days
Whether the model training will become inaccurate, when the training is interrupted, retraining re-enters the ""burn-in step 1" phase
If interrupted, it is recommended to manually set the "iter_count" in the configs to the state at which the training was interrupted. Then, follow the resume procedure provided by MMDetection.
What version of mmcv are you using? Is pytorch the default 1.9? I was curious, why did I train on an A100 GPU for more than 5 days
My environment:
- PyTorch: 1.9.0
- CUDA Runtime: 11.1
- CuDNN: 8.0.5
- TorchVision: 0.10.0
- OpenCV: 4.8.1
- MMCV: 1.4.5
- MMDetection: 2.13.0+c820f32