memory keep increasing in training
MillyMaMa opened this issue · comments
I ran the training code directly ,during my training, the memory usage(RAM, not GPU memory) kept increasing (at begining 1.8%, after training 3 epochs, it increased to 40%). After training several epochs ,the memory occupied too much, and the system killed the training process.
This project is borrowed from https://github.com/paul007pl/ECG.git. I only add a model file to the project. RAM issues ususally happened due to repeated data loading. I checked the codes but did not find redundant data loading operations. So I am also not sure why this happened.