Async unzip maxing out CPU and memory for some files
tmatson98 opened this issue · comments
How to reproduce
The issue can be reproduced with the following code:
const sync = false;
const contents = {};
for (let i = 0; i < 80; i += 1) {
contents[`file-${i}`] = new Uint8Array(Array.from({ length: 1024 * 1024 }, () => Math.floor(Math.random() * 256)));
}
const zipped = zipSync(contents);
console.log('Unzipping');
const start = performance.now();
if (sync) {
unzipSync(zipped);
console.log(`${performance.now() - start} ms`);
} else {
unzip(zipped, null, () => {
console.log(`${performance.now() - start} ms`);
});
}
The problem
Trying to decompress a moderately large ZIP file (~80 MB) containing files of 200 kB to 1 MB using the async unzip
function maxes out my system memory and CPU. Using unzipSync
instead works perfectly and takes less an 100ms in the above code, while unzip
takes a full minute. Increasing the ZIP size to 150 MB can cause unzip
to fail with this error:
Uncaught DOMException: The object could not be cloned.
cbify browser.js:910
inflate browser.js:1174
_loop_3 browser.js:2547
unzip browser.js:2556
Decompressing a ZIP with 8 10MB files works okay so it seems to be mainly caused by the number of files, but the unzipSync
function is still much faster decompressing it. Using a performance profiler the issue seems to be with the postMessage WebWorker function.
Test machine has an i7 6700, 16GB RAM
I'll try to address this in the next update. unzip
should be strictly faster than unzipSync
for sufficiently large ZIPs.
Reproduced locally. I have a potential fix for the slow performance but the crashing on ZIPs with hundreds of medium-size files might be more related to #58 than anything.
Fixed in v0.8.2.