scan_bucket.py struggles when too many objects in the bucket

Question

scan_bucket.py struggles when too many objects in the bucket

cbp123 opened this issue 4 years ago · comments

We have a bucket with millions of objects in it. Because scan_bucket.py loads all objects in the bucket into memory before scanning them, it can freeze for a long time at startup. I imagine in the worst case it could start to run out of memory.

I modified the code to load and scan the objects in pages instead and it worked much better. If you think this is a better method, let me know and I can submit a PR.

Dennis Webb · Answer 1 · Fri Jan 08 2021 22:34:59 GMT+0800 (China Standard Time)

Yes a PR would be great. Thanks.

Jeremy Deppen · Answer 2 · Mon May 03 2021 22:52:36 GMT+0800 (China Standard Time)

Hey @cbp123, any update on this?