Memory Leak
kurtommy opened this issue · comments
Hi, i'm using your wonderful library for read a lot of ID3 metadata form a large mp3 library, i'm using chrome, all works good but i receive always memory leaks, how i can flush memory and free RAM ?
For the massive use of cpu i used a delayed calls and all is ok, but for the memory i haven't found a solution.
Thank you.
Tommaso.
I would need to take a look into it, but to be honest I've been lacking the time lately because of other projects.
From the top of my head I'd say it's the ID3._files object that is used to memoize all calls to .get(all)Tags, have you tried clearing that one up?
Be careful because the minimized version doesn't have a property with that name (google closure renames it).
I guess the ideal would be to change the memoization to only keep in memory the last X files.
I call in a loop the function loadFromFile
var fileList, fileListLength, i, counter;
var founded = {};
var fApiReader;
$(document).ready(function(){
$('#file_input').change(function(e){
i = 0;
counter = 0;
fileList = e.target.files;
fileListLength = fileList.length;
founded = {};
readID3();
});
});
readID3 = function(){
if(i < fileListLength){
loadFromFile(fileList[i++]);
$('#console').html("Counter: "+(++counter));
setTimeout(readID3, 10);
//$('#debug').append(file.webkitRelativePath+'<br/>');
}
};
loadFromFile call loadUrl.In loadUrl there are this two lines that call ID3 class.
How i can destroy all references to the ID3 class for freeing memory? Thank you so much. Tommaso
function loadUrl(url, callback, reader) {
var startDate = new Date().getTime();
ID3.loadTags(url, function() {
var endDate = new Date().getTime();
if (typeof console !== "undefined") console.log("Time: " + ((endDate-startDate)/1000)+"s");
var tags = ID3.getAllTags(url);
Not sure if I understand what you mean. As far as I can tell in that piece of code there are no references to the ID3 object other than the ID3 variable itself that is provided by the library.
Btw, I made a mistake in my previous message, _files is not a property of the ID3 object, it's just in the closure where ID3 is created: https://github.com/aadsm/JavaScript-ID3-Reader/blob/master/src/id3.js#L13
Hi, tonight i made some debug with the chrome console, i found that the row that cause memory leaks is this:
var dataReader = options["dataReader"] || BufferedBinaryAjax;
in Buffered Binary Ajax 0.2.1 library
I checked the console of chrome to the tab "timeline, memory" and does not seem to be used a lot of memory, comes up to 20MB but checking in process monitor I see that the process of chrome running the script consumes all available RAM in my PC (I have 8GB). It seems that it is not memory occupied by variables in the script but that allocates resources to that class.
Do you have any idea how to libreare memory using that process for each new call?
Thank you very much
Tommaso
Uhm... without a closer look I can't really say why to be honest, 8GB is really insane, but like you said Chrome only reports you 20MB...
Do you have the same problem if you use the FileAPI instead of the BufferedBinaryAjax? Giving an "instance" of https://github.com/aadsm/JavaScript-ID3-Reader/blob/master/src/filereader.js as the dataReader.
The same problem, memory leak.
Using FileAPIReader the assignament row:
var dataReader = options["dataReader"] || FileAPIReader;
don't cause memory leak but when the code below will be executed yes.
this.loadTags = function(url, cb, options) {
options = options || {};
var dataReader = options["dataReader"] || FileAPIReader;//BufferedBinaryAjax;
_files = {};
dataReader(url, function(data) {
// preload the format identifier
data.loadRange(_formatIDRange, function() {
var reader = getTagReader(data);
reader.loadData(data, function() {
readTags(reader, data, url, options["tags"]);
if( cb ) cb();
});
});
});
};
you can try yourself for watichin what do the memory in your pc simply use this code with your library.
I change ID3 into class for generate a new instance each loop cicle, but don't solve the memory leak.
$(document).ready(function(){
$('#file_input').change(function(e){
i = 0;
counter = 0;
fileList = e.target.files;
fileListLength = fileList.length;
founded = {};
readID3();
});
});
readID3 = function(){
if(i < fileListLength){
ID3 = new ID3Obj();
loadFromFile(fileList[i++]);
$('#console').html("<br/>Counter: "+(++counter));
//$('#console').append("<br/>Filename: "+(fileList[i++].name));
setTimeout(readID3, 100);
//$('#debug').append(file.webkitRelativePath+'<br/>');
}
};
Thank you so much.
Tommaso.
I'll try to take a look this weekend.. you should try to clear out the _files object though.
Add a function to ID3 like: ID3.clearCache = function() { _files = {} } but don't forget it needs to be inside the ID3 closure.
I've tried yesterday but still the same problem, I hope you have a little 'time to find the bug this weekend.
thanks
Tommaso
hello, I'm still working on the code you should use your library, I hope you have time to check where the bug, you need that i send to you the code where I'm using?
Thank you so much!
Tommaso.
I'll try to replicate the issue on my own first but thanks!
I only remember that you load the files from my hard drive using a USB and are not locally on the server. Waiting for your answer ;) thanks again
Hi,
It seems that I'm able to see the memory leak even without the ID3 library, just by loading files.
I created a small example in jsFiddle where I can see that happening, I load the same file 100 times and Chrome's Task Manager shows me >500MB memory usage on the tab.
Does this match the problems you were experiencing?
Yes, also in your example the problem is in this line:
reader.readAsBinaryString(file);
The same way the library do to read files.
I use your FileAPIReader in this way:
var far = FileAPIReader(file);
loadUrl(url, function(){}, far);
And FileAPIReader use:
var reader = new FileReader();
reader.readAsBinaryString(file);
A lot of reader.readAsBinaryString cause memory leak.
Do you know some workaround or something to solve that problem?
Thank you,
Tommaso.
Nice find on stackoverflow.
I guess we need to create an object that implements the BinaryFile API ( https://github.com/aadsm/JavaScript-ID3-Reader/blob/master/src/binaryajax.js#L9-L143 ) that is able to read from a Blob, it shouldn't be that hard, mostly implementing the getByteAt.
I'll try to see what I can do, but again, can't promise anything, I'm really buried in work.
the library works really well and I regret not use it for this stupid problem, but I'm not very good with javascript and I do not think that I can be able to make this change alone. I'll have to wait for your help. thank you again.
can i ask to the main author of that library? cupboy@gmail.com do you know him?
I was taking a closer look at the object url but that's not really an option here because we need to read the data, in the stackoverflow example only a reference to the data is needed.
I think I need to do something similar to the BufferedBinaryAjax, the good news is that the logic to manage chunked data is there already, it's just a matter of slicing the File instead of doing range requests to a server. I'll see what I can do next weekend.
I don't want to put words in his mouth but I don't think Jacob (the original author of BinaryFile) has the time either...
However, in your current use case you're not dealing with the FileAPI right? If that's the case then this fix won't solve your problem anyway.
it seems that object url is only a reference to the file, and then not good. I have just read from my hard drive my music library because I want to expose the web by creating a virtual file system in order to have a reference to my favorite music at your fingertips. My goal is therefore to read many files with a selecting them from a folder and only chrome allows other browsers to make multiple selections of files but not folders. The problem is the way in which the library reads each file exactly as we have already seen these two lines of code:
var reader = new FileReader ();
reader.readAsBinaryString (file);
in this lib:
https://github.com/aadsm/JavaScript-ID3-Reader/blob/master/src/filereader.js
also using bufferedbinaryajax.js it has the usual problem of memory.
you have suggestions to make differently?
Tank you very much
Tommaso.
My suggestion is to create an Object similar to BufferedBinaryAjax that only loads chunks of the File instead of the whole thing which is what happens when you use readAsBinaryString.
However, you still have the problem with the current BufferedBinaryAjax object, but I haven't debugged that one yet. I'm curious to see what's going on because that one only loads chunks of the remote file.
Again, it's easier for me to take a look at it over the weekend.
Ok!!! see you on weekend! :)
I did some tests by loading 500 mp3 files. This will take ~70MB of memory in the heap because the cache doesn't have any limit. I've implemented a clearCache() function that once called it will bring the heap down to 1.5MB.
I checked these numbers using the "Heap Snapshot" feature of the Profile tab on Chrome's Developer Tools.
In Chrome's Task Manager I saw the memory going from ~75MB to ~160MB. When I call clearCache() it doesn't do much for the process memory if I stay in the current tab, however, if I try to load the 500 files again it will go down to around 100MB and it will go up again to 160MB while loading all the files. No matter how many times I cleared the cache and loaded the files again it was never using more than ~160MB.
If after clearCache() I switch focus to another tab or another app after a few seconds the process memory goes down to 75MB again.
I wasn't able to replicate the problem you reported of using up to 8GB :\ .
I will add a limit to the cache size, but other than that there's really not much that I can do.
Edit: These tests are related to BufferedBinaryAjax
Do you have committed this change? I don't see the clearCache() in
https://github.com/aadsm/JavaScript-ID3-Reader/blob/master/src/bufferedbinaryajax.js
No, that was just for a quick test. I think the best way is to probably add an option of how many tags the cache should handle.
I'll try to do it this week, it shouldn't take long.