CPU memory usage is high

[Lihe Yang]

The cpu memory usage is high when training on Cityscapes. I didn't crop the image, and input size is 1024x2048. Larger size may lead to higher GPU memory usage but my GPU memory usage is okay(around 11G), the cpu memory usage is high(around 80%). Sometimes it may be killed because it cannot allocate memory.

[Kentaro Wada]

Maybe this visualized image is too large in your case?

pytorch-fcn/torchfcn/trainer.py

Lines 126 to 129 in e86a0f0

 if len(visualizations) < 9: 
     viz = fcn.utils.visualize_segmentation( 
         lbl_pred=lp, lbl_true=lt, img=img, n_class=n_class) 
     visualizations.append(viz) 

Those visualized images are tiled to create another large image.

pytorch-fcn/torchfcn/trainer.py

Lines 133 to 137 in e86a0f0

 out = osp.join(self.out, 'visualization_viz') 
 if not osp.exists(out): 
     os.makedirs(out) 
 out_file = osp.join(out, 'iter%012d.jpg' % self.iteration) 
 skimage.io.imsave(out_file, fcn.utils.get_tile_image(visualizations))