Bus error (core dump) during training
ChenYutongTHU opened this issue · comments
Hi, thanks for sharing this project.
I prepared all the required h5py files and caption annotations files for COCO Caption finetuning as instructed in README. The training went normally at the beginning, but got killed (bus error (core dumped)) after around 70k~100k iteration.
I wonder if it was an out-of-memory issue caused by data loading. It seemed that huge memory was progressively consumed by the program, perhaps due to reading more and more image features from h5py files. Using del or gc.collect() didn't help free unreferenced objects' memory.
Is there any good solution to save memory for the multimodal training? Or idea on what was going on in my case. Thanks a lot!
@ChenYutongTHU With a quick google search, this one seems to be relevant: https://stackoverflow.com/questions/14653259/how-to-set-cache-settings-while-using-h5py-high-level-interface You will at some point stop caching the upcoming .h5 files I believe (I am nowhere near being an expert on this so please let me know once you find a good solution).