OSError: Unable to open file (unable to open file: name = 'sample_output/heatmaps/heatmap_benign/0_L_CC.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
BirgerMoell opened this issue · comments
Hi!
Thank you for a great project.
I'm trying to run the project on a K80 on Google Cloud using their provided Pytorch image.
The model fails when trying to create the heatmats with the following error.
I've tried creating the missing directories and reading through the source code.
Any help would be greatly appreciated.
Stage 1: Crop Mammograms
Error: the directory to save cropped images already exists.
Stage 2: Extract Centers
Stage 3: Generate Heatmaps
Traceback (most recent call last):
File "src/heatmaps/run_producer.py", line 29, in
import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
Stage 4a: Run Classifier (Image)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:19<00:00, 5.10s/it]
Stage 4b: Run Classifier (Image+Heatmaps)
0%| | 0/4 [00:00<?, ?it/s]
Traceback (most recent call last):
File "src/modeling/run_model.py", line 195, in
main()
File "src/modeling/run_model.py", line 190, in main
parameters=parameters,
File "src/modeling/run_model.py", line 148, in load_run_save
predictions = run_model(model, exam_list, parameters)
File "src/modeling/run_model.py", line 80, in run_model
horizontal_flip=datum["horizontal_flip"],
File "/home/birgermoell/chexnet/breast_cancer_classifier/src/data_loading/loading.py", line 72, in load_heatmaps
benign_heatmap = load_image(benign_heatmap_path, view, horizontal_flip)
File "/home/birgermoell/chexnet/breast_cancer_classifier/src/data_loading/loading.py", line 59, in load_image
image = read_image_mat(image_path)
File "/home/birgermoell/chexnet/breast_cancer_classifier/src/utilities/reading_images.py", line 37, in read_image_mat
data = h5py.File(file_name, 'r')
File "/opt/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py", line 312, in init
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/opt/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py", line 142, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 78, in h5py.h5f.open
OSError: Unable to open file (unable to open file: name = 'sample_output/heatmaps/heatmap_benign/0_L_CC.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Hi,
It seems that src/heatmaps/run_producer.py was importing tensorflow even though it was not necessary. I removed this unnecessary import statement so it should work now. Please pull the changes.
Hi,
It seems that I'm experiencing the same issue as above.
I can't get the run.sh to work with the sample data.
Below is the output with error: unable to open the image
Also attached is the environment created in conda.
Thank you,
Ioannis
NanWu_env.txt
(NanWu) ioannis@ioannis-ML:~/Documents/models/errors/breast_cancer_classifier-master_NanWu$ ./run.sh
Stage 1: Crop Mammograms
/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Stage 2: Extract Centers
/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Stage 3: Generate Heatmaps
/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Traceback (most recent call last):
File "src/heatmaps/run_producer.py", line 376, in
main()
File "src/heatmaps/run_producer.py", line 371, in main
model, device = load_model(parameters)
File "src/heatmaps/run_producer.py", line 315, in load_model
model = models.ModifiedDenseNet121(num_classes=parameters['number_of_classes'])
File "/home/ioannis/Documents/models/errors/breast_cancer_classifier-master_NanWu/src/heatmaps/models.py", line 39, in init
self.densenet = densenet.densenet121(*args, **kwargs)
File "/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/torchvision/models/densenet.py", line 26, in densenet121
**kwargs)
File "/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/torchvision/models/densenet.py", line 142, in init
bn_size=bn_size, growth_rate=growth_rate, drop_rate=drop_rate)
File "/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/torchvision/models/densenet.py", line 98, in init
layer = _DenseLayer(num_input_features + i * growth_rate, growth_rate, bn_size, drop_rate)
File "/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/torchvision/models/densenet.py", line 77, in init
self.add_module('norm.1', nn.BatchNorm2d(num_input_features)),
File "/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/torch/nn/modules/module.py", line 178, in add_module
raise KeyError("module name can't contain "."")
KeyError: 'module name can't contain "."'
Stage 4a: Run Classifier (Image)
/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
100%|██████████████████████████████████████████████████████████████████████████████████| 4/4 [00:19<00:00, 4.88s/it]
Stage 4b: Run Classifier (Image+Heatmaps)
/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
0%| | 0/4 [00:00<?, ?it/s]
Traceback (most recent call last):
File "src/modeling/run_model.py", line 238, in
main()
File "src/modeling/run_model.py", line 233, in main
parameters=parameters,
File "src/modeling/run_model.py", line 189, in load_run_save
predictions = run_model(model, device, exam_list, parameters)
File "src/modeling/run_model.py", line 91, in run_model
horizontal_flip=datum["horizontal_flip"],
File "/home/ioannis/Documents/models/errors/breast_cancer_classifier-master_NanWu/src/data_loading/loading.py", line 72, in load_heatmaps
benign_heatmap = load_image(benign_heatmap_path, view, horizontal_flip)
File "/home/ioannis/Documents/models/errors/breast_cancer_classifier-master_NanWu/src/data_loading/loading.py", line 59, in load_image
image = read_image_mat(image_path)
File "/home/ioannis/Documents/models/errors/breast_cancer_classifier-master_NanWu/src/utilities/reading_images.py", line 37, in read_image_mat
data = h5py.File(file_name, 'r')
File "/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/h5py/_hl/files.py", line 269, in init
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/home/ioannis/anaconda3/envs/NanWu/lib/python3.6/site-packages/h5py/_hl/files.py", line 99, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 78, in h5py.h5f.open
OSError: Unable to open file (unable to open file: name = 'sample_output/heatmaps/heatmap_benign/0_L_CC.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Thank you for your quick reply
@mammoguy, your stage 4B is failing because earlier stage 3 failed (the same error as #25 occurred). Heatmap files were not generated, therefore heatmap classifiers are not working.
According to @zphang in #25, our recommendation is to use Pytorch 0.4.1 and torchvision 0.2.0.
I am already using Pytorch 0.4.1 and torchvision 0.2.0. as you can see from the attached txt file in my original post.
What do you suggest I do next?
Thanks