.sync() can only delete remote files within the first 1000 files
peruukki opened this issue · comments
I noticed that some of our remote files were not deleted when using .sync(), and eventually found out that it's because the sync method enumerates the remote files with a simple call to listObjects that only returns a maximum of 1000 files at a time.
I'm not sure how this should be solved, as xml-json is used for the XML to JSON conversion, and it doesn't seem to support converting the whole XML response tree. It looks like you'd need at least the IsTruncated and existing Key value from the response, but they don't fall under a common subtree.
I guesss we didnt pay attention when we replaced the old code to work with the awssdk
I thought that this call will stream all the data
client.listObjects({ Prefix: prefix }).createReadStream()
An easy solution would be to create a stream that wrap this api and keep emitting until we reach the end
Isn't it about time to fix this?
javascript is beginning to look like lisp with all the partially working projects.
There is now a PR that handles this:
If you can't wait it's here: https://github.com/rmoskal/gulp-awspublish
