OverFlow Error during pickle -- multiprocessing
klamkiew opened this issue · comments
When huge amounts of data (tested: PB2 of IAV, >60.000 sequences) are given as input,
pickle complains about the size.
Chunking the dataset seems to be the best idea here.