Giters

logstash-plugins / logstash-input-s3

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

java.util.concurrent.ScheduledThreadPoolExecutor related to S3 input resulting in all pipeline logstash failure.

agrant1 opened this issue · comments

commented

Logstash information:

Please include the following information:

  1. Logstash version (e.g. bin/logstash --version)
    Using bundled JDK: /usr/share/logstash/jdk
    logstash 8.5.1
  2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
    RPM via yum
  3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
    systemd
  4. How was the Logstash Plugin installed
    Via RPM

JVM (e.g. java -version):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

  1. JVM version (java -version)
  2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
  3. Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:
S3 input causes a complete failure of logstash. Each S3 bucket is configured in a different pipeline, there are thousands of files in some of the s3 buckets.

The issue occurs randomly as far as I can tell. When it occurs the below error shows, logstash will never recover, and the only solution is to restart the service. The error will repeat over and over for each s3 input pipeline however, it is not limited to only an s3 input failure. All other pipelines silently fail with no error and the built in api in logstash do not return data.
pipeline.workers: use in the pipelines.yml seems to limit how often it occurs. However, this is an app breaking issue.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

I do not have a consistent way to reproduce this. The logs do not show anything out of the ordinary even in debug before it occurs.

Example input config:

input {
        s3{
        access_key_id => "<REDACTED>"
        secret_access_key => "${secret}"
        bucket => "<REDACTED>"
        region => "ca-central-1"
        prefix => "<REDACTED>
        "exclude_pattern" => "(\/202[0-1]\/|\/2022\/0\d\/|\/processing\-failed\/)"
        additional_settings => {
          force_path_style => true
          follow_redirects => true
          }
        }
}

Provide logs (if relevant):

[2023-03-02T16:16:33,530][ERROR][logstash.javapipeline    ][pipeline-1][52b512a94aa93bce1b8fc1cbaa5515accf8367fd3435706fa260182bb4dea35b] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:pipeline-1
  Plugin: <LogStash::Inputs::S3 access_key_id=>"<REDACTED>", bucket=>"<REDACTED>", additional_settings=>{"force_path_style"=>"true", "follow_redirects"=>"true"}, prefix=>"<REDACTED>", secret_access_key=><password>, exclude_pattern=>"(\\/202[0-1]\\/|\\/2022\\/0\\d\\/|\\/processing\\-failed\\/)", id=>"52b512a94aa93bce1b8fc1cbaa5515accf8367fd3435706fa260182bb4dea35b", region=>"ca-central-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_6debca34-0cd6-48ec-b7a7-1242fbc472f0", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", delete=>false, interval=>60, watch_for_new_files=>true, temporary_directory=>"/tmp/logstash", include_object_properties=>false, gzip_pattern=>".gz(ip)?$">
  Error: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1c0e5f31[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@37ec1c71[Wrapped task = org.jruby.ext.timeout.Timeout$TimeoutTask@79b15f5b]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@63c57f7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 15]
  Exception: Java::JavaUtilConcurrent::RejectedExecutionException
  Stack: java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
org.jruby.ext.timeout.Timeout.yieldWithTimeout(Timeout.java:145)
org.jruby.ext.timeout.Timeout.timeout(Timeout.java:118)
org.jruby.ext.timeout.Timeout.timeout(Timeout.java:92)
org.jruby.ext.timeout.Timeout$INVOKER$s$timeout.call(Timeout$INVOKER$s$timeout.gen)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$connect$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:949)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$do_start$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:934)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$start$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:929)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$start$0$__VARARGS__(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:919)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyBasicObject.send(RubyBasicObject.java:1733)
org.jruby.RubyBasicObject$INVOKER$i$send.call(RubyBasicObject$INVOKER$i$send.gen)
org.jruby.ir.targets.indy.InvokeSite.fail(InvokeSite.java:237)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.delegate.RUBY$method$method_missing$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/delegate.rb:83)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:192)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.connection_pool.RUBY$method$start_session$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/connection_pool.rb:307)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.connection_pool.RUBY$method$session_for$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/connection_pool.rb:100)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.handler.RUBY$method$session$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/handler.rb:128)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.handler.RUBY$method$transmit$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/handler.rb:76)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.handler.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/handler.rb:50)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.content_length.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/content_length.rb:24)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.request_callback.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/request_callback.rb:85)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.s3_signer.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_signer.rb:132)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.s3_signer.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_signer.rb:63)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.s3_host_id.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_host_id.rb:17)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.xml.error_handler.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/xml/error_handler.rb:10)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.transfer_encoding.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/transfer_encoding.rb:26)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.helpful_socket_errors.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.s3_signer.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_signer.rb:110)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.redirects.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/redirects.rb:20)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.retry_errors.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/retry_errors.rb:360)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.http_checksum.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/http_checksum.rb:19)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.endpoint_pattern.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.accelerate.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/accelerate.rb:67)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.checksum_algorithm.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:136)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.dualstack.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/dualstack.rb:41)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.expect_100_continue.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/expect_100_continue.rb:22)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.bucket_name_restrictions.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:26)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.arn.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/arn.rb:62)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.rest.handler.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/rest/handler.rb:10)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.recursion_detection.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/recursion_detection.rb:18)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.user_agent.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/user_agent.rb:13)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.endpoint_discovery.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:80)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.endpoint.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/endpoint.rb:47)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.param_validator.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/param_validator.rb:26)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.arn.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/arn.rb:88)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.raise_response_errors.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/raise_response_errors.rb:16)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.sse_cpk.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.dualstack.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/dualstack.rb:27)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.accelerate.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/accelerate.rb:56)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.checksum_algorithm.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.jsonvalue_converter.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.idempotency_token.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.param_converter.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/param_converter.rb:26)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.request_callback.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/request_callback.rb:71)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.response_paging.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/response_paging.rb:12)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.response_target.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/response_target.rb:24)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.request.RUBY$method$send_request$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/request.rb:72)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.client.RUBY$method$list_objects_v2$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/client.rb:8556)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.bucket.RUBY$block$objects$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/bucket.rb:912)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
org.jruby.runtime.Block.call(Block.java:143)
org.jruby.RubyProc.call(RubyProc.java:309)
org.jruby.RubyGenerator.each(RubyGenerator.java:99)
org.jruby.RubyGenerator$INVOKER$i$0$0$each.call(RubyGenerator$INVOKER$i$0$0$each.gen)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyClass.finvoke(RubyClass.java:502)
org.jruby.runtime.Helpers.invoke(Helpers.java:645)
org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:387)
org.jruby.RubyEnumerator.__each__(RubyEnumerator.java:396)
org.jruby.RubyEnumerator.each(RubyEnumerator.java:392)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.resources.collection.RUBY$block$non_empty_batches$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/resources/collection.rb:101)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
org.jruby.runtime.Block.call(Block.java:143)
org.jruby.RubyProc.call(RubyProc.java:309)
org.jruby.RubyGenerator.each(RubyGenerator.java:99)
org.jruby.RubyGenerator$INVOKER$i$0$0$each.call(RubyGenerator$INVOKER$i$0$0$each.gen)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyClass.finvoke(RubyClass.java:502)
org.jruby.runtime.Helpers.invoke(Helpers.java:645)
org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:387)
org.jruby.RubyEnumerator.__each__(RubyEnumerator.java:396)
org.jruby.RubyEnumerator.each(RubyEnumerator.java:392)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.resources.collection.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/resources/collection.rb:52)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
org.jruby.runtime.Block.call(Block.java:143)
org.jruby.RubyProc.call(RubyProc.java:309)
org.jruby.RubyGenerator.each(RubyGenerator.java:99)
org.jruby.RubyGenerator$INVOKER$i$0$0$each.call(RubyGenerator$INVOKER$i$0$0$each.gen)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyClass.finvoke(RubyClass.java:502)
org.jruby.runtime.Helpers.invoke(Helpers.java:645)
org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:387)
org.jruby.RubyEnumerator.__each__(RubyEnumerator.java:396)
org.jruby.RubyEnumerator.each(RubyEnumerator.java:392)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.resources.collection.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/resources/collection.rb:58)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_0.lib.logstash.inputs.s3.RUBY$method$list_new_files$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.0/lib/logstash/inputs/s3.rb:141)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_0.lib.logstash.inputs.s3.RUBY$method$process_files$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.0/lib/logstash/inputs/s3.rb:183)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_0.lib.logstash.inputs.s3.RUBY$block$run$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.0/lib/logstash/inputs/s3.rb:130)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.Block.call(Block.java:147)
org.jruby.RubyProc.call(RubyProc.java:272)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.stud_minus_0_dot_0_dot_23.lib.stud.interval.RUBY$method$interval$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/stud-0.0.23/lib/stud/interval.rb:20)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_0.lib.logstash.inputs.s3.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.0/lib/logstash/inputs/s3.rb:129)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:411)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:406)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:208)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:402)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
org.jruby.runtime.Block.call(Block.java:143)
org.jruby.RubyProc.call(RubyProc.java:309)
org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)
java.base/java.lang.Thread.run(Thread.java:833)
commented

Following up