4_test.py
enochsol opened this issue · comments
When I run 4_test.py, it throws an error: ValueError: max() arg is an empty sequence.
What would be your suggestion?
############### CONFIGURATION ###############
all_wsi: all_wsi
val_wsi_per_class: 20
test_wsi_per_class: 30
keep_orig_copy: True
num_workers: 8
patch_size: 224
wsi_train: wsi_train
wsi_val: wsi_val
wsi_test: wsi_test
labels_train: labels_train.csv
labels_val: labels_val.csv
labels_test: labels_test.csv
train_folder: train_folder
patches_eval_train: patches_eval_train
patches_eval_val: patches_eval_val
patches_eval_test: patches_eval_test
num_train_per_class: 8000
type_histopath: True
purple_threshold: 100
purple_scale_size: 15
slide_overlap: 3
gen_val_patches_overlap_factor: 1.5
image_ext: jpg
by_folder: True
color_jitter_brightness: 0.5
color_jitter_contrast: 0.5
color_jitter_saturation: 0.5
color_jitter_hue: 0.2
num_epochs: 20
num_layers: 18
learning_rate: 0.001
batch_size: 16
weight_decay: 0.0001
learning_rate_decay: 0.85
resume_checkpoint: False
save_interval: 1
checkpoints_folder: checkpoints
checkpoint_file: xyz.pt
pretrain: False
log_folder: logs
auto_select: True
preds_train: preds_train
preds_val: preds_val
preds_test: preds_test
inference_train: inference_train
inference_val: inference_val
inference_test: inference_test
vis_train: vis_train
vis_val: vis_val
vis_test: vis_test
device: cpu
classes: ['benign', 'tumor']
num_classes: 2
train_patches: train_folder/train
val_patches: train_folder/val
path_mean: [167953056.0, 4.742346702121871e+30, 4.739335267905211e+30]
path_std: [nan, nan, nan]
resume_checkpoint_path: checkpoints/xyz.pt
log_csv: logs/log_272022_101945.csv
eval_model: checkpoints/xyz.pt
threshold_search: (0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)
colors: ('red', 'white', 'blue', 'green', 'purple', 'orange', 'black', 'pink', 'yellow')
#####################################################
+++++ Running 4_test.py +++++
----- Finding validation patch predictions -----
Traceback (most recent call last):
File "code/4_test.py", line 21, in
pretrain=config.args.pretrain)
File "/home/SebliLo/deepslide/code/utils_model.py", line 564, in get_predictions
checkpoints_folder=checkpoints_folder) if auto_select else eval_model
File "/home/SebliLo/deepslide/code/utils_model.py", line 534, in get_best_model
key=operator.itemgetter(1))[0]
ValueError: max() arg is an empty sequence
I believe this is a side effect of the problem raised in issue #46. Once that issue is fixed, please try running this code again.
@JosephDiPalma thank you, yes it could be. Will check it once we fix issue #46
After fixing issue #46, running the test gives the same issue.
############### CONFIGURATION ###############
all_wsi: all_wsi
val_wsi_per_class: 20
test_wsi_per_class: 30
keep_orig_copy: True
num_workers: 8
patch_size: 224
wsi_train: wsi_train
wsi_val: wsi_val
wsi_test: wsi_test
labels_train: labels_train.csv
labels_val: labels_val.csv
labels_test: labels_test.csv
train_folder: train_folder
patches_eval_train: patches_eval_train
patches_eval_val: patches_eval_val
patches_eval_test: patches_eval_test
num_train_per_class: 8000
type_histopath: True
purple_threshold: 100
purple_scale_size: 15
slide_overlap: 3
gen_val_patches_overlap_factor: 1.5
image_ext: jpg
by_folder: True
color_jitter_brightness: 0.5
color_jitter_contrast: 0.5
color_jitter_saturation: 0.5
color_jitter_hue: 0.2
num_epochs: 20
num_layers: 18
learning_rate: 0.001
batch_size: 16
weight_decay: 0.0001
learning_rate_decay: 0.85
resume_checkpoint: False
save_interval: 1
checkpoints_folder: checkpoints
checkpoint_file: xyz.pt
pretrain: False
log_folder: logs
auto_select: True
preds_train: preds_train
preds_val: preds_val
preds_test: preds_test
inference_train: inference_train
inference_val: inference_val
inference_test: inference_test
vis_train: vis_train
vis_val: vis_val
vis_test: vis_test
device: cpu
classes: ['benign', 'tumor']
num_classes: 2
train_patches: train_folder/train
val_patches: train_folder/val
path_mean: [0.16684997, 0.16684997, 0.16684997]
path_std: [0.40847272, 0.40847272, 0.40847272]
resume_checkpoint_path: checkpoints/xyz.pt
log_csv: logs/log_2162022_22214.csv
eval_model: checkpoints/xyz.pt
threshold_search: (0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)
colors: ('red', 'white', 'blue', 'green', 'purple', 'orange', 'black', 'pink', 'yellow')
#####################################################
+++++ Running 4_test.py +++++
----- Finding validation patch predictions -----
Traceback (most recent call last):
File "code/4_test.py", line 21, in
pretrain=config.args.pretrain)
File "/home/SebliLo/deepslide/code/utils_model.py", line 564, in get_predictions
checkpoints_folder=checkpoints_folder) if auto_select else eval_model
File "/home/SebliLo/deepslide/code/utils_model.py", line 534, in get_best_model
key=operator.itemgetter(1))[0]
ValueError: max() arg is an empty sequence
The traceback point corresponds to a snapshot selection. Could you double check that your checkpoints folder has N+1 snapshot files where N is the number of epochs you trained, and each file contains validation performance string?
checkpoints folder has only 20 file which is the default number of epochs
I copy/paste one of the existing file from/to the checkpoints folder to make the number of file N+1. But still the same issue.
+++++ Running 4_test.py +++++
----- Finding validation patch predictions -----
Traceback (most recent call last):
File "code/4_test.py", line 21, in
pretrain=config.args.pretrain)
File "/home/SebliLo/deepslide/code/utils_model.py", line 564, in get_predictions
checkpoints_folder=checkpoints_folder) if auto_select else eval_model
File "/home/SebliLo/deepslide/code/utils_model.py", line 534, in get_best_model
key=operator.itemgetter(1))[0]
ValueError: max() arg is an empty sequence
Can you provide the name of one of the checkpoint files in that folder?
Please see all. I added manually the one in bold.
resnet18_e0_va0.75000.pt resnet18_e12_va0.85000.pt resnet18_e15_va1.00000.pt resnet18_e18_va0.95000.pt resnet18_e20_va0.90000.pt resnet18_e4_va0.52500.pt resnet18_e7_va0.50000.pt
resnet18_e10_va0.50000.pt resnet18_e13_va0.95000.pt resnet18_e16_va0.77500.pt resnet18_e19_va0.90000.pt resnet18_e2_va0.95000.pt resnet18_e5_va0.60000.pt resnet18_e8_va0.50000.pt
resnet18_e11_va1.00000.pt resnet18_e14_va1.00000.pt resnet18_e17_va0.92500.pt resnet18_e1_va0.50000.pt resnet18_e3_va0.52500.pt resnet18_e6_va0.77500.pt resnet18_e9_va0.95000.pt
In utils_model.py can you add the following code above line 530 in the get_best_model() function:
print(get_image_paths(folder=checkpoints_folder))
It is empty
[]
This means that your checkpoint folder is empty. Can you provide the directory structure of the project?
(ds) [SebliLo@tesla deepslide]$ ll
total 148
drwxrwxr-x. 4 SebliLo SebliLo 4096 Feb 6 19:42 all_wsi
drwxrwxr-x. 2 SebliLo SebliLo 4096 Feb 17 20:16 checkpoints
drwxrwxr-x. 7 SebliLo SebliLo 4096 Feb 18 17:17 code
drwxrwxr-x. 2 SebliLo SebliLo 4096 Feb 3 21:22 figures
drwxrwxr-x. 2 SebliLo SebliLo 4096 Feb 3 21:22 LICENSE
drwxrwxr-x. 2 SebliLo SebliLo 4096 Feb 3 21:22 README.md
-rw-rw-r--. 1 SebliLo SebliLo 2612 Feb 16 21:14 labels_test.csv
-rw-rw-r--. 1 SebliLo SebliLo 84501 Feb 16 21:14 labels_train.csv
-rw-rw-r--. 1 SebliLo SebliLo 1756 Feb 16 21:14 labels_val.csv
drwxrwxr-x. 2 SebliLo SebliLo 4096 Feb 16 21:19 logs
-rw-------. 1 SebliLo SebliLo 8195 Feb 16 22:00 nohup.out
drwxrwxr-x. 62 SebliLo SebliLo 4096 Feb 16 21:16 patches_eval_test
drwxrwxr-x. 42 SebliLo SebliLo 4096 Feb 16 21:16 patches_eval_val
drwxrwxr-x. 4 SebliLo SebliLo 4096 Feb 16 21:16 train_folder
drwxrwxr-x. 4 SebliLo SebliLo 4096 Feb 16 21:14 wsi_test
drwxrwxr-x. 4 SebliLo SebliLo 4096 Feb 16 21:14 wsi_train
drwxrwxr-x. 4 SebliLo SebliLo 4096 Feb 16 21:14 wsi_val
(ds) [SebliLo@tesla deepslide]$
And dataset and deepslide folders are in the same directory.
Just to confirm, the checkpoints you provided in a prior reply are located in the checkpoints folder?
Yes it is
How are you running the file 4_test.py? Specifically, the python command and the directory in which you're running it.
From the "deepslide" directory (the same directory I ran the previous steps), I ran "python code/4_test.py"
Can you try running it from within the code directory?
It is not finding the all_wsi
(py3) [SebliLo@tesla code]$ python 4_test.py
Traceback (most recent call last):
File "4_test.py", line 1, in
import config
File "/home/SebliLo/deepslide/code/config.py", line 360, in
classes = get_classes(folder=args.all_wsi)
File "/home/SebliLo/deepslide/code/utils.py", line 26, in get_classes
return sorted([f.name for f in folder.iterdir() if
File "/home/SebliLo/deepslide/code/utils.py", line 26, in
return sorted([f.name for f in folder.iterdir() if
File "/home/SebliLo/anaconda3/envs/py3/lib/python3.6/pathlib.py", line 1081, in iterdir
for name in self._accessor.listdir(self):
File "/home/SebliLo/anaconda3/envs/py3/lib/python3.6/pathlib.py", line 387, in wrapped
return strfunc(str(pathobj), *args)
FileNotFoundError: [Errno 2] No such file or directory: 'all_wsi'
Could you provide us with your python environment setup?
(py3) [SebliLo@tesla deepslide]$ conda list
packages in environment at /home/SebliLo/anaconda3/envs/py3:
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
blas 1.0 mkl anaconda
ca-certificates 2021.10.26 h06a4308_2
cairo 1.16.0 hf32fb01_1
certifi 2020.6.20 pyhd3eb1b0_3
cloudpickle 1.6.0 py_0 anaconda
cpuonly 1.0 0 pytorch
cudatoolkit 11.3.1 h2bc3f7f_2
cycler 0.10.0 py36_0 anaconda
cytoolz 0.11.0 py36h7b6447c_0 anaconda
dask-core 2.30.0 py_0 anaconda
dataclasses 0.8 pyh4f3eec9_6
dbus 1.13.18 hb2f20db_0 anaconda
decorator 4.4.2 py_0 anaconda
expat 2.2.10 he6710b0_2 anaconda
fontconfig 2.13.1 h6c09931_0
freetype 2.10.4 h0708190_1 conda-forge
gdk-pixbuf 2.42.6 h8cc273a_5
giflib 5.2.1 h7b6447c_0 anaconda
glib 2.69.1 h4ff587b_1
gobject-introspection 1.68.0 py36hc6cf493_2
gst-plugins-base 1.14.0 h8213a91_2
gstreamer 1.14.0 h28cd5cc_2
icu 58.2 he6710b0_3 anaconda
imageio 2.9.0 py_0 anaconda
intel-openmp 2021.4.0 h06a4308_3561
jinja2 3.0.2 pyhd3eb1b0_0
joblib 0.17.0 py_0 anaconda
jpeg 9d h7f8727e_0
kiwisolver 1.2.0 py36hfd86e86_0 anaconda
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.35.1 h7274673_9
libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgfortran-ng 7.5.0 ha8ba4b0_17
libgfortran4 7.5.0 ha8ba4b0_17
libgomp 9.3.0 h5101ec6_17
libiconv 1.16 h516909a_0 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libstdcxx-ng 9.3.0 hd4cf53a_17
libtiff 4.2.0 h85742a9_0
libuuid 1.0.3 h7f8727e_2
libuv 1.40.0 h7b6447c_0
libwebp 1.2.0 h89dd481_0
libwebp-base 1.2.0 h27cfd23_0
libxcb 1.14 h7b6447c_0
libxml2 2.9.10 hb55368b_3 anaconda
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
markupsafe 2.0.1 py36h27cfd23_0
matplotlib 3.1.1 py36h5429711_0 anaconda
mkl 2019.4 243 anaconda
mkl-service 2.3.0 py36he904b0f_0 anaconda
mkl_fft 1.2.0 py36h23d657b_0 anaconda
mkl_random 1.0.4 py36hd81dba3_0
ncurses 6.3 h7f8727e_2
networkx 2.5 py_0 anaconda
ninja 1.10.2 h5e70eb0_2
numpy 1.19.1 py36hbc911f0_0 anaconda
numpy-base 1.19.1 py36hfa32c7d_0 anaconda
olefile 0.46 py_0 anaconda
openjpeg 2.4.0 hb52868f_1 conda-forge
openslide 3.4.1 h8137273_1 conda-forge/label/cf202003
openssl 1.1.1m h7f8727e_0
packaging 20.4 py_0 anaconda
pandas 0.25.3 py36he6710b0_0
pcre 8.45 h9c3ff4c_0 conda-forge
pillow 8.3.1 py36h5aabda8_0
pip 21.2.2 py36h06a4308_0
pixman 0.40.0 h36c2ea0_0 conda-forge
pyparsing 2.4.7 py_0 anaconda
pyqt 5.9.2 py36h22d08a2_1 anaconda
python 3.6.13 h12debd9_1
python-dateutil 2.8.1 py_0 anaconda
pytorch 1.7.0 py3.6_cpu_0 [cpuonly] pytorch
pytorch-mutex 1.0 cuda pytorch
pytz 2020.1 py_0 anaconda
pywavelets 1.0.3 py36hdd07704_1 anaconda
pyyaml 5.3.1 py36h7b6447c_1 anaconda
qt 5.9.7 h5867ecd_1
readline 8.1.2 h7f8727e_1
scikit-image 0.15.0 py36he6710b0_0 anaconda
scikit-learn 0.23.2 py36h0573a6f_0 anaconda
scipy 1.5.2 py36h0b6359f_0 anaconda
setuptools 58.0.4 py36h06a4308_0
sip 4.19.24 py36he6710b0_0 anaconda
six 1.15.0 py_0 anaconda
sqlite 3.37.0 hc218d9a_0
threadpoolctl 2.1.0 pyh5ca1d4c_0 anaconda
tk 8.6.11 h1ccaba5_0
toolz 0.11.1 py_0 anaconda
torchvision 0.8.1 py36_cpu [cpuonly] pytorch
tornado 6.0.4 py36h7b6447c_1 anaconda
typing_extensions 3.10.0.2 pyh06a4308_0
tzdata 2021e hda174b7_0
wheel 0.37.1 pyhd3eb1b0_0
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0 anaconda
zlib 1.2.11 h7f8727e_4
zstd 1.4.9 ha95c52a_0 conda-forge
(py3) [SebliLo@tesla deepslide]$ pip list
Package Version
certifi 2020.6.20
cloudpickle 1.6.0
cycler 0.10.0
cytoolz 0.11.0
dask 2.30.0
dataclasses 0.8
decorator 4.4.2
imageio 2.9.0
Jinja2 3.0.2
joblib 0.17.0
kiwisolver 1.2.0
MarkupSafe 2.0.1
matplotlib 3.1.1
mkl-fft 1.2.0
mkl-random 1.0.4
mkl-service 2.3.0
networkx 2.5
numpy 1.19.1
olefile 0.46
packaging 20.4
pandas 0.25.3
Pillow 8.3.1
pip 21.2.2
pyparsing 2.4.7
python-dateutil 2.8.1
pytz 2020.1
PyWavelets 1.0.3
PyYAML 5.3.1
scikit-image 0.15.0
scikit-learn 0.23.2
scipy 1.5.2
setuptools 58.0.4
sip 4.19.24
six 1.15.0
threadpoolctl 2.1.0
toolz 0.11.1
torch 1.7.0
torchvision 0.8.1
tornado 6.0.4
typing-extensions 3.10.0.2
wheel 0.37.1
We've identified the issue and a fix will be merged into the master branch soon. Thanks for bringing this to our attention.
Thank you so much for assistance!!
Thank you! The issue is resolved