scan_bucket.py struggles when too many objects in the bucket
cbp123 opened this issue · comments
We have a bucket with millions of objects in it. Because scan_bucket.py loads all objects in the bucket into memory before scanning them, it can freeze for a long time at startup. I imagine in the worst case it could start to run out of memory.
I modified the code to load and scan the objects in pages instead and it worked much better. If you think this is a better method, let me know and I can submit a PR.
Yes a PR would be great. Thanks.
Hey @cbp123, any update on this?