Giters

orangewise / s3-zip

Download selected files from an Amazon S3 bucket as a zip file

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Is there any way to stream source files to s3-zip?

phillip-odam opened this issue · comments

commented

Looking at the examples, they all take an array of files to zip. Is it already possible or easy to extend so that .archive can be called multiple times as progressively more files are 'discovered' needing to be zipped up?

For example listObjects returns at most 1,000 objects, so to include more than 1,000 files in the zip without multiple calls to .archive we need to wait to finish traversing s3 before beginning the generation of the zip.

Am I overlooking the obvious? Is it already possible to incrementally add more files to the zip?

Thanks
Phillip

commented

I think I have a possible solution:

https://gist.github.com/arturojain/2fff0680e2f4d6b62d0313ef0f9ac559

My guess is that this can work on both express and other output options.

commented

Hi Arturo

Thanks for that. Correct me if I'm misreading it though, it looks as if your solution first collects all the s3 object references in to the keys array before passing it on to the resolve callback to ultimately be passed over to s3Zip's archive call.

This is what I was hoping I could avoid, I am looking for a way to not block on the zip creation until I have all the files I will eventually want included in the zip.

commented

@phillip-odam I think https://github.com/alukach/ZipStream will do what you want and a few more things. Note that it's AGPLv3, so you'll have to provide a download link for the source code you deploy.