Giters

bowenc0221 / panoptic-deeplab

This is Pytorch re-implementation of our CVPR 2020 paper "Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation" (https://arxiv.org/abs/1911.10194)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

RuntimeError: CUDA error: out of memory

yjlin0223 opened this issue · comments

commented

Traceback (most recent call last):
File "/home/ubuntu/yunjian/panoptic_deeplab/tools_d2/train_panoptic_deeplab.py", line 182, in
args=(args,),
File "/home/ubuntu/anaconda3/envs/panoptic/lib/python3.6/site-packages/detectron2/engine/launch.py", line 62, in launch
main_func(*args)
File "/home/ubuntu/yunjian/panoptic_deeplab/tools_d2/train_panoptic_deeplab.py", line 161, in main
model = Trainer.build_model(cfg)
File "/home/ubuntu/anaconda3/envs/panoptic/lib/python3.6/site-packages/detectron2/engine/defaults.py", line 452, in build_model
model = build_model(cfg)
File "/home/ubuntu/anaconda3/envs/panoptic/lib/python3.6/site-packages/detectron2/modeling/meta_arch/build.py", line 22, in build_model
model.to(torch.device(cfg.MODEL.DEVICE))
File "/home/ubuntu/anaconda3/envs/panoptic/lib/python3.6/site-packages/torch/nn/modules/module.py", line 612, in to
return self._apply(convert)
File "/home/ubuntu/anaconda3/envs/panoptic/lib/python3.6/site-packages/torch/nn/modules/module.py", line 359, in _apply
module._apply(fn)
File "/home/ubuntu/anaconda3/envs/panoptic/lib/python3.6/site-packages/torch/nn/modules/module.py", line 359, in _apply
module._apply(fn)
File "/home/ubuntu/anaconda3/envs/panoptic/lib/python3.6/site-packages/torch/nn/modules/module.py", line 381, in _apply
param_applied = fn(param)
File "/home/ubuntu/anaconda3/envs/panoptic/lib/python3.6/site-packages/torch/nn/modules/module.py", line 610, in convert
return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: CUDA error: out of memory

the following is the full error report. Here is the command I use python train_panoptic_deeplab.py --config-file configs/Cityscapes-PanopticSegmentation/panoptic_deeplab_X_65_os16_mg124_poly_90k_bs32_crop_512_1024.yaml --eval-only MODEL.WEIGHTS /home/ubuntu/model/model_yunjian/panoptic/evaluation/panoptic_deeplab_X_65_os16_mg124_poly_90k_bs32_crop_512_1024.pkl

I just change corresponding contents in yaml: 'IMS_PER_BATCH:4' , 'GPUS:(1,2)' and 'DATASET ROOT'. I don't know why this error happens.

commented

It means your GPU memory is not enough to hold two images per GPU