MemoryError
KassemKallas opened this issue · comments
Hi all,
I am trying to train the GAN network on my own dataset which consists of 8600 images of size 192x192. Using the default settings or not, I am always obtaining an error at epoch 6, the program fails to load the images and gives me the following error:
data=np.zeros((batch_size,3,npx,npx)) # NOTE: assumes 3 channels!
MemoryError
PS: I am using pycharm IDE on windows 10.
Thanks in advance,
Kassem Kallas
Hi
it seems that the problem is with the system RAM, not the deep learning GPU ram. How much RAM do you have, our program uses a lot since to speed-up iteration it stores all the texture images in memory. Maybe you can try to load only hald of the 8600 images and observe what happens with memory.
We have tested our system in Ubuntu with 10000 or more textures of larger size than yours. We have not tested in in Windows, which may have different memory footprint.
Hi Thank you for the response. Actually, I have found the memory leak, but still I could not solve it.
The leak is in data_io.py --> get_texture_iter --> while True: , the object data that will be returned to the main of psgan.py doesn't release its place in memory after being used. Each time the method is called, it is creating another instance in memory to store the big structure/object without cleaning the previous one from the previous epoch. Do you have an idea how to solve this?
You can try to move " data=np.zeros((batch_size,3,npx,npx)) " 2 lines upwards, so that it is before the "while(true) yield ..." loop.
A bit hacky, but may solve the problem.
Unfortunately, it did not solve the problem.
Hi,
The error was solved by replacing the "While True" with a "for loop + yield" for the iterative function.
Note that you need to pass to the function get_texture_iter the variable iters as a parameter since you need to for-loop up to iters-1.
In this way, if watch the memory usage, you can see that it is released after each epoch.
Best,
Kassem Kallas