retinanet训练问题
Joejwu opened this issue · comments
您好，想问一下，为啥我在训练retinanet的时候总是执行一段出现几个warning之后就自动停下来了，而且也不报错；我一开始以为是用来apex的问题，设置为false之后还是自动停下来了；后面我给换成多卡的也是同样的，请问大佬知道是为啥嘛？
`root@container-ab78119f3c-c31dcd5b:~/SimpleAICV-pytorch-ImageNet-COCO-training-master/detection_training/coco/res50_retinanet_retinaresize800# sh train.sh
======================1======================
No pretrained model file!
loading annotations into memory...
Done (t=16.43s)
creating index...
index created!
Dataset Size:117266
Dataset Class Num:80
loading annotations into memory...
Done (t=0.51s)
creating index...
index created!
Dataset Size:5000
Dataset Class Num:80
======================2======================
Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.
Defaults for this optimization level are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'")
Warning: apex was installed without --cpp_ext. Falling back to Python flatten and unflatten.`
找到原因了，我给关啦！
您好，想问一下，为啥我在训练retinanet的时候总是执行一段出现几个warning之后就自动停下来了，而且也不报错；我一开始以为是用来apex的问题，设置为false之后还是自动停下来了；后面我给换成多卡的也是同样的，请问大佬知道是为啥嘛？ `root@container-ab78119f3c-c31dcd5b:~/SimpleAICV-pytorch-ImageNet-COCO-training-master/detection_training/coco/res50_retinanet_retinaresize800# sh train.sh ======================1====================== No pretrained model file! loading annotations into memory... Done (t=16.43s) creating index... index created! Dataset Size:117266 Dataset Class Num:80 loading annotations into memory... Done (t=0.51s) creating index... index created! Dataset Size:5000 Dataset Class Num:80 ======================2====================== Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.
Defaults for this optimization level are: enabled : True opt_level : O1 cast_model_type : None patch_torch_functions : True keep_batchnorm_fp32 : None master_weights : None loss_scale : dynamic Processing user overrides (additional kwargs that are not None)... After processing overrides, optimization options are: enabled : True opt_level : O1 cast_model_type : None patch_torch_functions : True keep_batchnorm_fp32 : None master_weights : None loss_scale : dynamic Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'") Warning: apex was installed without --cpp_ext. Falling back to Python flatten and unflatten.`
hi，很高兴你已经解决了问题，我补充一下，当使用apex时，出现类似以下信息时：
======================2======================
Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.
Defaults for this optimization level are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'")
Warning: apex was installed without --cpp_ext. Falling back to Python flatten and unflatten.
模型此时已经在训练了，只是使用apex时并不会在terminal窗口中刷新log，所有log记录在log/train.info.log中。由于train_config.py中print_interval = 100，每过100个batch才会在train.info.log中更新一次信息。如果你想快速看到log信息，可以将这个值设为1。
另外补充一下，无论是否使用apex，log/train.info.log中都会记录所有训练log，一般来说，只要terminal命令没有中断，就是在正常训练。
呜呜！收到！感谢大佬！
您好，我还想问一下关于batchsize与学习率对应的问题！我看mmdetection有个学习率自动缩放的操作，
_比如：在 4 块 GPU 并且每张 GPU 上有 2 张图片的情况下 lr=0.01，那么在 16 块 GPU 并且每张 GPU 上有 4 张图片的情况下, LR 会自动缩放至lr=0.08。_
所以，基于您的开源代码训练retinanet，是否也存在这个学习率与bs之间的对应关系呢？因为我现在用的gpu只能设置bs为2或4，不知道用您之前默认的学习率0.0001是不是还合适；另一方面自己跑完完整的训练时间实在太长了，所以来这边打扰一下您！
目前需要自己调整学习率。
好嘞！收到！谢谢大佬！