Memory Leakage?

[Zihan Ding]

Hi,

Thanks for providing such a geat code as a baseline for research purpose!

However, when applying the code, I notice there may be a memory (RAM) leakage during training. However, the code looks fine to me, and I have no idea where it possibly comes from. Perhaps you have some idea about it. Specifically, what I observed are as following:

• When training with NFSP on 'LaserTag-small4-v0', the initial memory usage is 17.1% on my local machine with ~16 GB RAM, and it increases to 18.5% after 10^5 frames (when the buffer becomes full by defaut parameters), I can understand this part since the ReservoirBuffer is being fed during the process. However, the memory usage of this training process keeps growing to more than 19% after $3.4\times10^5$ frames.
• It becomes more obvious when I apply NFSP on a larger-scale environment like Pong in Atari, with image size 3*84*84 as observation, rather than 3*21*20 in LaserFrame. Specifically, on a server with ~370 GB RAM, it initially used ~1.2% memory but increases to 12.4% memory after $4\times10^5$ frames.

Not sure if you have any idea where the memory leakage could come from and whether there is any solution for this. Look forward to your reply!

Best,
Zihan