benthosdev / benthos

Fancy stream processing made operationally mundane

Home Page:https://www.benthos.dev

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

expose RateLimiter for s3 input

ryanpodonnell1 opened this issue · comments

We are currently unable to toggle the client side rate throttling on the s3 input plugin. We'd like to expose it as an input configuration, this appears to be the code where it is set:

https://github.com/benthosdev/benthos/blob/main/internal/impl/aws/session.go#L24

sdk docs:
https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/retries-timeouts/#client-side-rate-limiting

example input:

input:
  label: "s3"
  aws_s3:
    bucket: "bucket"
    prefix: ""
    codec: gzip/all-bytes
    rate_limiter_max_tokens: 1000 <- set as 0 to disable or a number to modify the default of 500
    sqs:
      url: "https://xxxxx"
      key_path: detail.object.key
      bucket_path: detail.bucket.name
      max_messages: 10
      wait_time_seconds: 0