A minor issue with the data loader
russellizadi opened this issue · comments
In the data loader, a list of SoundFile objects is kept in self.files which may cause problems in loading because of having too many files open when the memory is limited. I suggest to keep the file names instead and create the object in the __getitem__ method.
Thanks for the suggestion.
As I remember, the SoundFile object only keep a reference to the file, so the memory consumption should be very small.
How many files have you tried to open, in your case? I think it might be some other factors that cause the problem.
If it's not the case I will consider create them in __getitem__.
I tried the LJSpeech and it was fine when I closed the f_obj At the end of the __init__ method. When keeping all the files open, it raises a SystemError Which is related to memory. I’m not confident but that was my understanding after searching for this issue. One way to test this is to use psutil and monitor the memory during the __init__ method.
Ok I also encountered the same error. Will fix it asap.
5a10ff9 should fix the problem.