expose RateLimiter for s3 input
ryanpodonnell1 opened this issue · comments
ryanpodonnell1 commented
We are currently unable to toggle the client side rate throttling on the s3 input plugin. We'd like to expose it as an input configuration, this appears to be the code where it is set:
https://github.com/benthosdev/benthos/blob/main/internal/impl/aws/session.go#L24
sdk docs:
https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/retries-timeouts/#client-side-rate-limiting
example input:
input:
label: "s3"
aws_s3:
bucket: "bucket"
prefix: ""
codec: gzip/all-bytes
rate_limiter_max_tokens: 1000 <- set as 0 to disable or a number to modify the default of 500
sqs:
url: "https://xxxxx"
key_path: detail.object.key
bucket_path: detail.bucket.name
max_messages: 10
wait_time_seconds: 0