support s3 directories with more than 1000 keys
andrewresnikoff opened this issue · comments
Each list-objects-v2 API call only returns 1000 keys at a time as per the AWS Docs.
Returns some or all (up to 1,000) of the objects in a bucket.
When there are more than 1000 keys, we get back NextContinuationToken
NextContinuationToken is sent when isTruncated is true, which means there are more keys in the bucket that can be listed. The next list requests to Amazon S3 can be continued with this NextContinuationToken . NextContinuationToken is obfuscated and is not a real key
to paginate the remaining results.