General fail in preprocessing
cissoidx opened this issue · comments
I did a full preprocessing using your script on the winter21_whole.tar.gz dataset. Is it normal to have tons of general fail messages? Will these fails of images impact final training results?
no. try again.
check for bugs (no memory, bad paths)
@mrT23 thanks for reply.
Have you tried this script on the recently released winter21_whole.tar.gz dataset on you own? It seems the resizing process is running normally, but the there are bad images in it. what is the md5sum of the winter21_whole.tar.gz file?
no. i processed fall11 variant.
winter21 variant was processed by image-net.org. notice that they offer the processed version for direct download, so
maybe you dont need to process it by yourself.
you can also contact them regarding the errors.
cannot find the download link. I only have the winter21_whole.tar.gz for the moment. Could you please paste here the md5sum of the winter21_whole.tar.gz file if you have it?
thanks for info
FYI, the md5 of my copy of the dataset matches the one shown above. So I guess there are indeed corrupted images in this tar file.
ab313ce03179fd803a401b02c651c0a2 winter21_whole.tar.gz
It seems I have one more image in training set than what you stated in the processing_instructions.md file although I had several tens of general failures. Imagenet.org does not validate my email, cannot contact them.
where do you get the following files:
files_train_set = './resources/datasets_imagenet21K_P_train_folder_index.pkl'
files_val_set = './resources/datasets_imagenet21K_P_val_folder_index.pkl'Could you please provide download links?
they are for fall11. you are processing winter21, so they won't help you
ok. thanks for reply