High RAM usage when zipping (and streaming from expressjs) large number of files

[Zlatoslav Petukhov]

Hi 2 all,

First of all, I want to say a big 'thank you' to the author for the lib :)
There are some competitors out there but I find none of them are much better.

So, back to the issue...
When I stream a large number of files as .zip via expressJS, the library consumes too much RAM.
If the volume is big enough, all available RAM is consumed and the process is hung (and RAM is not freed).

I did some research and was able to resolve the issue.
After that I can stream gigabytes of files and the app is not consuming much RAM.
👇👇👇
The FIX is here:
zpetukhov/s3-files@14eca1c
👆👆👆
The change is actually in the underlying 's3-files', but I guess many users comes to this repo first.

Below is my [consuming] code in the express app (hope that may help to somebody):

const s3Zip = require('s3-zip');
const { S3 } = require('aws-sdk');
const s3creds = {
  accessKeyId: process.env.AWS_ACCESS_KEY,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
};
const s3 = new S3(s3creds);

...

mymethod(req, res) {
   const ar = [.......]; // an array of s3 keys is prepared here - I simplified the code here
   res.setHeader('Content-Type', 'application/zip');
   res.setHeader('Content-Dispositon', 'attachment; filename=" + "archive.zip"');
   s3Zip.setArchiverOptions({ level: 0 }).archive({ s3, bucket: 'mybucket' }, path, ar)
     .pipe(res);
}

Possibly related issues:

• #5
• #60

[Ronald Luitwieler]

hi @zpetukhov, thank you for the nice words. Would you mind creating a PR so I can easily merge and create new versions?

Thank you for your contribution :)

[Zlatoslav Petukhov]

I have created a PR
orangewise/s3-files#20

[Ronald Luitwieler]

v.3.2.1 is released, thanks again!

[Zlatoslav Petukhov]

According to my experience, to reduce RAM consuming, the following is very essential:

.setArchiverOptions({ level: 0 })

(the full code example is above)

Without setting compression to 0, I still facing with quite high RAM usage.
In my particular case I stream jpg files, so compression is not really needed.