Memory requirements

[Luis Carlos Rivera]

Thanks for the awesome repository!
I am trying to run the cell graph generation example but I get CUDA out of memory errors. I am using a GPU with 8.5Gb, not running anything else or shared in any way.
Is there a minimum requirement for graph representation inference?

[Antonio Foncubierta Rodríguez]

With GPU it is normally OK to run the examples, can you try modifying the batch_size ?
Anyway, can you post your error log to have a look at it ?

[Luis Carlos Rivera]

But as far as I understand the batch_size in the example script is 1. isn't it?

I tried the code in a bigger GPU and I always get the same out of memory error:

Using backend: pytorch

Patch-level nuclei detection:   0%|          | 0/2 [00:00<?, ?it/s]

Traceback (most recent call last):
  File "cell_graph_generation.py", line 91, in <module>
    generate_cell_graph(image_path=os.path.join('output', 'images'))
  File "cell_graph_generation.py", line 64, in generate_cell_graph
    nuclei_map, _ = nuclei_detector.process(image)
  File "/home/rivera/.local/lib/python3.8/site-packages/histocartography/pipeline.py", line 138, in process
    return self._process(*args, **kwargs)
  File "/home/rivera/.local/lib/python3.8/site-packages/histocartography/preprocessing/nuclei_extraction.py", line 97, in _process
    return self._extract_nuclei(input_image, tissue_mask)
  File "/home/rivera/.local/lib/python3.8/site-packages/histocartography/preprocessing/nuclei_extraction.py", line 139, in _extract_nuclei
    out = self.model(image_batch).cpu()
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rivera/.local/lib/python3.8/site-packages/histocartography/ml/models/hovernet.py", line 44, in forward
    d = self.encode(images)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rivera/.local/lib/python3.8/site-packages/histocartography/ml/models/hovernet.py", line 105, in forward
    x2 = self.group0(x1)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rivera/.local/lib/python3.8/site-packages/histocartography/ml/models/hovernet.py", line 201, in forward
    shortcut = getattr(self, 'block0_convshortcut')(in_feats)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rivera/.local/lib/python3.8/site-packages/histocartography/ml/models/hovernet.py", line 385, in forward
    x = self.conv(x)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0; 15.75 GiB total capacity; 767.27 MiB already allocated; 1.60 GiB free; 786.00 MiB reserved in total by PyTorch)

[Guillaume Jaume]

It seems to be an issue in the nuclei detector.

1. Could you tell me if you run on your own images or on the set of dummy images?
2. If on your own, can you make sure that the input you provide to the nuclei extraction module is a numpy array
3. Antonio was referring to the batch size of the patches when processing the image. As the images are larger than what a ANN can handle, we have to break them down into patches. By default, it is set to 32 with cuda available (and 2 with cpu). Can you try reducing it, e.g., to 8, when declaring the NucleiExtractor, i.e., NucleiExtractor(batch_size=8)

Hope this will solve the issue :)

[Luis Carlos Rivera]

1. I am trying both. But for now to make the example script work.
2. Ok I will keep that in mind
3. Ok, but I think the way it is implemented does not allow to use of the batch_size as a parameter of the NucleiExtractor. When I try I get that 'NucleiExtractor' object has no attribute 'batch_size'.
   But if I go to the implementation of the extractor (nuclei_extraction.py) and change the default value from 32 to 8 it works.

So I think there is a small bug there to allow the parameter to be changed. Maybe the default value also should be smaller to allow implementation in most GPUs

[Guillaume Jaume]

Yes, you're right, thanks for reporting the issue - will make a change.

[Guillaume Jaume]

Addressed in #9