MemoryError: during preprocessing
mominali12 opened this issue · comments
python main.py --mode prepro --data_file hotpot_train_v1.1.json --para_limit 2250 --data_split train
Getting memory error after 11048 tasks are done.
I have 8GB of RAM and I believe that it runs out of memory while doing pre processing. Is there any way so that I could perhaps perform pre processing in chunks or some other way around ?
Try reducing n_jobs in Parallel calls. More parallel jobs speed up processing, but also consume more memory.