Error on custom dataset training

[lidiahhhh]

I followed the instruction giude and tried to train a custom dataset.
Dataset is as follows:
fcdd\data\datasets\custom\train\dashboard\normal
fcdd\data\datasets\custom\test\dashboard\normal
fcdd\data\datasets\custom\test\dashboard\anomalous

The error i get is
`python runners/run_custom.py

Plotting ROC for completed classes up to 0...
Traceback (most recent call last):
File ".\fcdd\python\fcdd\runners\run_custom.py", line 49, in
runner.run()
File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\runners\bases.py", line 203, in run
self.run_classes(**vars(self.args))
File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\runners\bases.py", line 221, in run_classes
res = self.run_seeds(
File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\runners\bases.py", line 181, in run_seeds
res = self.run_one(
File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\runners\bases.py", line 100, in run_one
setup = trainer_setup(
File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\training\setup.py", line 104, in trainer_setup
ds = load_dataset(
File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\datasets_init_.py", line 67, in load_dataset
dataset = ADImageFolderDataset(
File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\datasets\image_folder.py", line 95, in init
self.mean, self.std = self.extract_mean_std(self.trainpath, normal_class)
File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\datasets\image_folder.py", line 194, in extract_mean_std
for x, _ in loader:
File "...\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 359, in iter
return self._get_iterator()
File "...\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 305, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "...\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 918, in init
w.start()
File "...\Anaconda3\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "...\Anaconda3\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "...\Anaconda3\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "...\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "...\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'ADImageFolderDataset.extract_mean_std..'

$ Traceback (most recent call last):
File "", line 1, in
File "...\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "...\Anaconda3\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
`

[liznerski]

This seems to be the same issue as #18. You can either switch to Linux, or you use the windows branch of this repository and move the ADImageFolderDataset.extract_mean_std function our of the class.

[lidiahhhh]

Thanks for the reply.. I actually managed to successfully run the code on Google colab.

I have another question on the results, if you can help me:
I'd expected the heatmaps of the normal class to be completly blue, but i got that the heatmaps have a lot of dark red spots.. Am i doing something wrong?

Thanks

[liznerski]

Well...

The heatmaps need to be normalized so that they can be interpreted as images; the values need to be in [0, 1]). We provide "locally normalized" heatmaps, where each each heatmap is normalized w.r.t. itself so that it has at least one fully red pixel (value 1) and one fully blue pixel (value 0). In this case, heatmaps for normal samples will always have red spots. We also provide "globally normalized" heatmaps, where each heatmap is normalized w.r.t to the complete set of samples (e.g., test set). If this set contains a sufficient amount of anomalies, there shouldn't be too many red spots in the "globally normalized" heatmaps for normal samples.

Another reason for your red spots could be that the quantile hyperparamter is set suboptimally. Incerasing it--so that less scores are mapped to 1--might help. Have a look at Appendix C of our paper for more details.

Hope that helps :)