More RAM is consumed as training epoch increases
hongvin opened this issue · comments
Koay Hong Vin commented
I have repeated the experiment for many times and eventually at the 14th epoch, the process is killed, mainly due to my CPU RAM is exhausted. My current configuration is 8 core CPUs, 30GB CPU RAM, P5000 16GB GPU.
I have tried to reduce memory bank size, but it's still the same.
SeonWoo-Lee commented
reduce num_worker thread and batch size.
CreatedTRYNA commented
I have the same problem when I try to train this model, and anything can I do?