cannot location any cell

Question

cannot location any cell

RNA429 opened this issue 2 months ago · comments

Describe the bug
A clear and concise description of what the bug is.
I have a good result when i am training, but it cannot location any cell when i am inerfence.
To Reproduce
Steps to reproduce the behavior:

python '/data/CellViT/cell_segmentation/inference/cell_detection.py' --model /data/CellViT/log/2024-05-10T092116_cellvit_256/checkpoints/model_best.pth --batch_size 4 --gpu 1 --geojson process_wsi --wsi_path ./example/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE.svs --patched_slide_path ./example/output/preprocessing/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE
Error Traceback
INFO: Pandarallel will run on 12 workers.
INFO: Pandarallel will use Memory file system to transfer data between the main process and workers.
2024-05-10 11:21:47,272 [INFO] - Loading model: /data/CellViT/log/2024-05-10T092116_cellvit_256/checkpoints/model_best.pth
1:CellViT256Shared
2024-05-10 11:21:48,148 [INFO] -
2024-05-10 11:21:50,200 [INFO] - Loading inference transformations
2024-05-10 11:21:50,201 [INFO] - Processing single WSI file
2024-05-10 11:21:50,204 [INFO] - Processing WSI: TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE
0%| | 0/59 [00:00<?, ?it/s]Token-Shape: torch.Size([4, 384, 64, 64])
7%|██ | 4/59 [00:04<04:06, 4.48s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
14%|████▏ | 8/59 [00:06<00:56, 1.11s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
20%|██████ | 12/59 [00:08<00:37, 1.26it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
27%|████████▏ | 16/59 [00:11<00:30, 1.43it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
34%|██████████▏ | 20/59 [00:12<00:22, 1.73it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
41%|████████████▏ | 24/59 [00:14<00:20, 1.72it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
47%|██████████████▏ | 28/59 [00:16<00:15, 2.00it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
54%|████████████████▎ | 32/59 [00:17<00:12, 2.09it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
61%|██████████████████▎ | 36/59 [00:19<00:10, 2.15it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
68%|████████████████████▎ | 40/59 [00:21<00:08, 2.14it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
75%|██████████████████████▎ | 44/59 [00:23<00:07, 2.06it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
81%|████████████████████████▍ | 48/59 [00:25<00:05, 2.11it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
88%|██████████████████████████▍ | 52/59 [00:27<00:03, 1.97it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([4, 384, 64, 64])
95%|████████████████████████████▍ | 56/59 [00:29<00:01, 2.03it/s, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([3, 384, 64, 64])
100%|██████████████████████████████| 59/59 [00:30<00:00, 2.40it/s, Cells=0, Memory=0.00 MB]2024-05-10 11:22:20,951 [INFO] - Detected cells before cleaning: 0
2024-05-10 11:22:20,951 [INFO] - Initializing Cell-Postprocessor
Traceback (most recent call last):
File "/data/CellViT/cell_segmentation/inference/cell_detection.py", line 1069, in cell_segmentation.process_wsi(
File "/data/CellViT/cell_segmentation/inference/cell_detection.py", line 430, in process_wsi
keep_idx = self.post_process_edge_cells(cell_list=cell_dict_wsi)
File "/data/CellViT/cell_segmentation/inference/cell_detection.py", line 538, in post_process_edge_cells
cell_processor = CellPostProcessor(cell_list, self.logger)
File "/data/CellViT/cell_segmentation/inference/cell_detection.py", line 624, in init
self.cell_df = self.cell_df.parallel_apply(convert_coordinates, axis=1)
File "/home/ubuntu/miniconda3/envs/cellvit/lib/python3.9/site-packages/pandarallel/core.py", line 294, in closure
pool = CONTEXT.Pool(nb_workers)
File "/home/ubuntu/miniconda3/envs/cellvit/lib/python3.9/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/home/ubuntu/miniconda3/envs/cellvit/lib/python3.9/multiprocessing/pool.py", line 205, in init
raise ValueError("Number of processes must be at least 1")
ValueError: Number of processes must be at least 1
100%|██████████████████████████████| 59/59 [00:31<00:00, 1.90it/s, Cells=0, Memory=0.00 MB]

RNA429 · Answer 1 · Fri May 10 2024 12:49:41 GMT+0800 (China Standard Time)

I also tried the model and the .svs both you provided, but i get the same result.
(cellvit) ubuntu@ubuntu:/data/CellViT-main$ python3 ./cell_segmentation/inference/cell_detection.py --model ./models/pretrained/CellViT/CellViT-SAM-H-x40.pth --gpu 1 --batch_size 2
--geojson process_wsi --wsi_path ./example/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE.svs --patched_slide_path ./example/output/preprocessing/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE
INFO: Pandarallel will run on 12 workers.
INFO: Pandarallel will use Memory file system to transfer data between the main process and workers.
2024-05-10 12:44:41,627 [INFO] - Loading model: models/pretrained/CellViT/CellViT-SAM-H-x40.pth
2024-05-10 12:45:00,633 [INFO] -
2024-05-10 12:45:09,167 [INFO] - Loading inference transformations
2024-05-10 12:45:09,167 [INFO] - Processing single WSI file
2024-05-10 12:45:09,171 [INFO] - Processing WSI: TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE
0%| | 0/59 [00:00<?, ?it/s]Token-Shape: torch.Size([2, 1280, 64, 64])
3%|█ | 2/59 [00:03<03:43, 3.91s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
7%|██ | 4/59 [00:06<01:40, 1.82s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
10%|███▏ | 6/59 [00:08<01:14, 1.41s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
14%|████▏ | 8/59 [00:10<01:05, 1.28s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
17%|█████ | 10/59 [00:13<01:07, 1.39s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
20%|██████ | 12/59 [00:15<01:01, 1.32s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
24%|███████ | 14/59 [00:18<00:57, 1.28s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
27%|████████▏ | 16/59 [00:20<00:52, 1.23s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
31%|█████████▏ | 18/59 [00:22<00:46, 1.14s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
34%|██████████▏ | 20/59 [00:24<00:43, 1.11s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
37%|███████████▏ | 22/59 [00:26<00:41, 1.11s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
41%|████████████▏ | 24/59 [00:28<00:38, 1.10s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
44%|█████████████▏ | 26/59 [00:30<00:34, 1.05s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
47%|██████████████▏ | 28/59 [00:32<00:31, 1.01s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
51%|███████████████▎ | 30/59 [00:34<00:29, 1.03s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
54%|████████████████▎ | 32/59 [00:36<00:27, 1.03s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
58%|█████████████████▎ | 34/59 [00:38<00:26, 1.04s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
61%|██████████████████▎ | 36/59 [00:40<00:23, 1.01s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
64%|███████████████████▎ | 38/59 [00:42<00:21, 1.00s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
68%|████████████████████▎ | 40/59 [00:45<00:19, 1.05s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
71%|█████████████████████▎ | 42/59 [00:47<00:18, 1.06s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
75%|██████████████████████▎ | 44/59 [00:49<00:16, 1.07s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
78%|███████████████████████▍ | 46/59 [00:51<00:13, 1.02s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
81%|████████████████████████▍ | 48/59 [00:53<00:11, 1.06s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
85%|█████████████████████████▍ | 50/59 [00:55<00:09, 1.09s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
88%|██████████████████████████▍ | 52/59 [00:58<00:07, 1.08s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
92%|███████████████████████████▍ | 54/59 [00:59<00:05, 1.05s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
95%|████████████████████████████▍ | 56/59 [01:02<00:03, 1.06s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([2, 1280, 64, 64])
98%|█████████████████████████████▍| 58/59 [01:04<00:01, 1.03s/it, Cells=0, Memory=0.00 MB]Token-Shape: torch.Size([1, 1280, 64, 64])
100%|██████████████████████████████| 59/59 [01:04<00:00, 1.16it/s, Cells=0, Memory=0.00 MB]2024-05-10 12:46:14,260 [INFO] - Detected cells before cleaning: 0
2024-05-10 12:46:14,260 [INFO] - Initializing Cell-Postprocessor
Traceback (most recent call last):
File "/data/CellViT-main/./cell_segmentation/inference/cell_detection.py", line 1066, in
cell_segmentation.process_wsi(
File "/data/CellViT-main/./cell_segmentation/inference/cell_detection.py", line 427, in process_wsi
keep_idx = self.post_process_edge_cells(cell_list=cell_dict_wsi)
File "/data/CellViT-main/./cell_segmentation/inference/cell_detection.py", line 535, in post_process_edge_cells
cell_processor = CellPostProcessor(cell_list, self.logger)
File "/data/CellViT-main/./cell_segmentation/inference/cell_detection.py", line 621, in init
self.cell_df = self.cell_df.parallel_apply(convert_coordinates, axis=1)
File "/home/ubuntu/miniconda3/envs/cellvit/lib/python3.9/site-packages/pandarallel/core.py", line 294, in closure
pool = CONTEXT.Pool(nb_workers)
File "/home/ubuntu/miniconda3/envs/cellvit/lib/python3.9/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/home/ubuntu/miniconda3/envs/cellvit/lib/python3.9/multiprocessing/pool.py", line 205, in init
raise ValueError("Number of processes must be at least 1")
ValueError: Number of processes must be at least 1
100%|██████████████████████████████| 59/59 [01:05<00:00, 1.11s/it, Cells=0, Memory=0.00 MB]

Fabian Hörst · Answer 2 · Fri May 10 2024 14:41:46 GMT+0800 (China Standard Time)

Hi,

I am going to investigate this. We are currently working on an updated version with in-runtime inference, therefore this might need some days to complete.

RNA429 · Answer 3 · Fri May 10 2024 16:09:17 GMT+0800 (China Standard Time)

Hi,

I am going to investigate this. We are currently working on an updated version with in-runtime inference, therefore this might need some days to complete.

Very much looking forward to your updates

Fabian Hörst · Answer 4 · Fri May 10 2024 16:17:17 GMT+0800 (China Standard Time)

Could you pull the changes again and test again?

I receive the following output with the code:

2024-05-10 08:05:45,806 [INFO] - Loading model: /home/jovyan/cellvit-data/cellvit/checkpoints/CellViT-SAM-H-x40-AMP.pth
2024-05-10 08:05:58,001 [INFO] - <All keys matched successfully>
2024-05-10 08:05:58,734 [INFO] - Loading inference transformations
2024-05-10 08:05:58,734 [INFO] - Processing single WSI file
2024-05-10 08:05:58,739 [INFO] - Processing WSI: TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 59/59 [00:40<00:00,  1.86it/s, Cells=14127, Memory=105.43 MB]2024-05-10 08:06:39,425 [INFO] - Detected cells before cleaning: 14127
2024-05-10 08:06:39,425 [INFO] - Initializing Cell-Postprocessor
2024-05-10 08:06:41,542 [INFO] - Finding edge-cells for merging
2024-05-10 08:06:41,577 [INFO] - Removal of cells detected multiple times
2024-05-10 08:06:42,027 [INFO] - Iteration 0: Found overlap of # cells: 1001
2024-05-10 08:06:42,218 [INFO] - Iteration 1: Found overlap of # cells: 15
2024-05-10 08:06:42,405 [INFO] - Iteration 2: Found overlap of # cells: 0
2024-05-10 08:06:42,405 [INFO] - Found all overlapping cells
2024-05-10 08:06:42,472 [INFO] - Detected cells after cleaning: 12242
2024-05-10 08:06:42,473 [INFO] - Processed all patches. Storing final results: /home/jovyan/cellvit-data/cellvit-old/CellViT/example/output/preprocessing/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE/cell_detection/cells.json and cell_detection.json
2024-05-10 08:06:42,804 [INFO] - Converting segmentation to geojson
2024-05-10 08:06:43,283 [INFO] - Converting detection to geojson
2024-05-10 08:06:43,346 [INFO] - Create cell graph with embeddings and save it under: /home/jovyan/cellvit-data/cellvit-old/CellViT/example/output/preprocessing/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE/cell_detection/cells.pt
2024-05-10 08:06:44,766 [INFO] - Finished with cell detection for WSI TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE
2024-05-10 08:06:44,766 [INFO] - Stats:
2024-05-10 08:06:44,766 [INFO] - {'Connective': 9876, 'Epithelial': 1392, 'Inflammatory': 627, 'Neoplastic': 347}
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 59/59 [00:46<00:00,  1.28it/s, Cells=14127, Memory=105.43 MB]

RNA429 · Answer 5 · Fri May 10 2024 16:50:25 GMT+0800 (China Standard Time)

Could you pull the changes again and test again?

I receive the following output with the code:

2024-05-10 08:05:45,806 [INFO] - Loading model: /home/jovyan/cellvit-data/cellvit/checkpoints/CellViT-SAM-H-x40-AMP.pth
2024-05-10 08:05:58,001 [INFO] - <All keys matched successfully>
2024-05-10 08:05:58,734 [INFO] - Loading inference transformations
2024-05-10 08:05:58,734 [INFO] - Processing single WSI file
2024-05-10 08:05:58,739 [INFO] - Processing WSI: TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 59/59 [00:40<00:00,  1.86it/s, Cells=14127, Memory=105.43 MB]2024-05-10 08:06:39,425 [INFO] - Detected cells before cleaning: 14127
2024-05-10 08:06:39,425 [INFO] - Initializing Cell-Postprocessor
2024-05-10 08:06:41,542 [INFO] - Finding edge-cells for merging
2024-05-10 08:06:41,577 [INFO] - Removal of cells detected multiple times
2024-05-10 08:06:42,027 [INFO] - Iteration 0: Found overlap of # cells: 1001
2024-05-10 08:06:42,218 [INFO] - Iteration 1: Found overlap of # cells: 15
2024-05-10 08:06:42,405 [INFO] - Iteration 2: Found overlap of # cells: 0
2024-05-10 08:06:42,405 [INFO] - Found all overlapping cells
2024-05-10 08:06:42,472 [INFO] - Detected cells after cleaning: 12242
2024-05-10 08:06:42,473 [INFO] - Processed all patches. Storing final results: /home/jovyan/cellvit-data/cellvit-old/CellViT/example/output/preprocessing/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE/cell_detection/cells.json and cell_detection.json
2024-05-10 08:06:42,804 [INFO] - Converting segmentation to geojson
2024-05-10 08:06:43,283 [INFO] - Converting detection to geojson
2024-05-10 08:06:43,346 [INFO] - Create cell graph with embeddings and save it under: /home/jovyan/cellvit-data/cellvit-old/CellViT/example/output/preprocessing/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE/cell_detection/cells.pt
2024-05-10 08:06:44,766 [INFO] - Finished with cell detection for WSI TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE
2024-05-10 08:06:44,766 [INFO] - Stats:
2024-05-10 08:06:44,766 [INFO] - {'Connective': 9876, 'Epithelial': 1392, 'Inflammatory': 627, 'Neoplastic': 347}
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 59/59 [00:46<00:00,  1.28it/s, Cells=14127, Memory=105.43 MB]

maybe it caused by openslide_python and opencv_python_headless, i reinstall these to packages, then i get the right result!
It's my bad. Thanks for your reply!

Fabian Hörst · Answer 6 · Fri May 10 2024 16:55:38 GMT+0800 (China Standard Time)

No problem. Unfortunately, dealing with packages is also highly system-dependent. Nonetheless, thanks to your hint, I was able to fix another small bug.