Issue With PLIP

[Abdullah-Tauqeer01]

Hey I am facing an issue in forwarding the image batch through the PLIP model, I followed everything exactly as instructed in the readme file, and I have installed the clam_latest environment from official clam repository, the error is given below:

quantimb02 44 % cd /local/data0/home/atauqeer/Code/CLAM ; /usr/bin/env /local/data0/software/miniconda3-23.11/envs/clam_latest/bin/python /local/data0/home/atauqeer/pc_vscode/q01/.vscode-server/extensions/ms-python.debugpy-2024.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher 49945 -- /local/data0/home/atauqeer/Code/CLAM/extract_features_fp.py --data_h5_dir /local/data0/data/CAMELYON_16_AND_17/CAMELYON16/Processed_CAMELYON_512X512/ --data_slide_dir /local/data0/data/CAMELYON_16_AND_17/CAMELYON16/images/ --csv_path /local/data0/data/CAMELYON_16_AND_17/CAMELYON16/Processed_CAMELYON_512X512/process_list_autogen.csv --feat_dir /local/data0/data/CAMELYON_16_AND_17/CAMELYON16/Processed_CAMELYON_512X512/Features_dir/ --batch_size 256 --slide_ext .tif
initializing dataset
0%| | 0/379 [00:00<?, ?it/s]
progress: 0/379
normal_001
downsample [1. 1.]
downsampled_level_dim [ 97792 221184]
level_dim [ 97792 221184]
name normal_001
patch_level 0
patch_size 512
save_path /local/data0/data/CAMELYON_16_AND_17/CAMELYON16/Processed_CAMELYON_512X512/patches

feature extraction settings
transformations: Compose(
ToTensor()
Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711))
)
processing a total of 8 batches
0%| | 0/8 [05:58<?, ?it/s]
0%| | 0/379 [06:00<?, ?it/s]
Traceback (most recent call last):
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/local/data0/home/atauqeer/pc_vscode/q01/.vscode-server/extensions/ms-python.debugpy-2024.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in
cli.main()
File "/local/data0/home/atauqeer/pc_vscode/q01/.vscode-server/extensions/ms-python.debugpy-2024.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/local/data0/home/atauqeer/pc_vscode/q01/.vscode-server/extensions/ms-python.debugpy-2024.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="main")
File "/local/data0/home/atauqeer/pc_vscode/q01/.vscode-server/extensions/ms-python.debugpy-2024.8.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "/local/data0/home/atauqeer/pc_vscode/q01/.vscode-server/extensions/ms-python.debugpy-2024.8.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/local/data0/home/atauqeer/pc_vscode/q01/.vscode-server/extensions/ms-python.debugpy-2024.8.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "/local/data0/home/atauqeer/Code/CLAM/extract_features_fp.py", line 111, in
output_file_path = compute_w_loader(output_path, loader = loader, model = model, verbose = 1)
File "/local/data0/home/atauqeer/Code/CLAM/extract_features_fp.py", line 42, in compute_w_loader
features = model(batch)
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/local/data0/home/atauqeer/Code/CLAM/models/plip.py", line 10, in forward
return self.model(input).image_embeds
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 1298, in forward
vision_outputs = self.vision_model(
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 849, in forward
hidden_states = self.embeddings(pixel_values)
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/local/data0/software/miniconda3-23.11/envs/clam_latest/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 189, in forward
embeddings = embeddings + self.position_embedding(self.position_ids)
RuntimeError: The size of tensor a (257) must match the size of tensor b (50) at non-singleton dimension 1

[Wenhao Tang]

I'm sorry, but I can't offer any valuable replies about your question. But I think you can print the size of the embedding and pos for the model runtime and check if you have any error about the size when cropping the patches.