Memory leak in batch processing
andreped opened this issue · comments
The leakage is likely due to TensorFlow and the fact that we are running inference in separate threads, which ideally should be run in different processes. However, when freezing python applications you cannot launch new processes by default.
Potential fix, but might require a newer version of Python:
https://superfastpython.com/multiprocessing-freeze-support-in-python/
Tried some more tests, but this is definitely not trivial. It has to do with processes should be created inside the if __name__ = "__main__" block, which is especially important on windows, to avoid some infinite loops occuring. We are running the process stuff in a module (or submodule) and in this case the name attribute is not available.
I believe we may create something that works on UNIX-systems, but I highly doubt that this will work on windows, sadly.
I guess we will have to get back to this at a later point.
More info regarding this problem can be found here:
https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Multiprocessing
But it does not address the problem of creating processes in (sub-)modules. Note the self regarding this topic can be found here:
https://stackoverflow.com/questions/42548602/multiprocessing-launch-from-within-module-or-class-not-from-main
https://stackoverflow.com/questions/42602584/how-to-use-multiprocessing-pool-in-an-imported-module
https://stackoverflow.com/questions/37737085/is-it-possible-to-use-multiprocessing-in-a-module-with-windows
The latter seems to work, but looks like the type of hack that should put you in jail.
But just a thought, as we are not doing any GPU-stuff, why is there a memory leak? I would expect CPU leakage to be quite low. Perhaps some objects are not deleted when running batch mode?
https://stackoverflow.com/questions/51005147/keras-release-memory-after-finish-training-process
Might not necessarily be TF-related.
Fixed in 411e3b1.