out of memory
Lywzz opened this issue · comments
i try to test on davis dataset with two GPU ,11GB,but it got error,please help me to solve it, thanks.
batchSize: 1
temperature: 1.0
gridSize: 9
classNum: 49
videoLen: 8
cropSize: 320
cropSize2: 80
0,1,2,3
False
self.T: 0.04419417382415922
Total params: 26.01M
==> Resuming from checkpoint..
Evaluation only
gridx: 4 gridy: 4
total_frame_num: 77
(77, 320, 320, 3)
[array([0, 0, 0], dtype=uint8), array([ 0, 128, 0], dtype=uint8), array([128, 0, 0], dtype=uint8)]
[85088, 10181, 7129]
20.661283493041992 relabel 0.456728458404541 label
0
Traceback (most recent call last):
File "test_davis.py", line 458, in
test_loss = test(val_loader, model, 1, use_cuda)
File "test_davis.py", line 238, in test
corrfeat2_now = model(imgs_tensor, target_tensor)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 119, in forward
inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 130, in scatter
return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 35, in scatter_kwargs
inputs = scatter(inputs, target_gpus, dim) if inputs else []
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 28, in scatter
return scatter_map(inputs)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 15, in scatter_map
return list(zip(*map(scatter_map, obj)))
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 13, in scatter_map
return Scatter.apply(target_gpus, None, dim, obj)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 87, in forward
outputs = comm.scatter(input, ctx.target_gpus, ctx.chunk_sizes, ctx.dim, streams)
File "/opt/conda/lib/python3.6/site-packages/torch/cuda/comm.py", line 142, in scatter
return tuple(torch._C._scatter(tensor, devices, chunk_sizes, dim, streams))
RuntimeError: CUDA error: out of memory (allocate at /opt/conda/conda-bld/pytorch_1532579805626/work/aten/src/THC/THCCachingAllocator.cpp:510)
The bottleneck is here:
Line 418 in 16d33ac
You can try to cut the input images into 2 imgs_tensor, and forward 2 times and then concat the outputs together.
thank you,i solved the problem