Misleading error returned with the latest AWS sdk

[Pier-Hugues Pellerin]

When a permission error is raised, the error is hidden with an internal error. The Aws-sdk rely on lazy loading of the libraries file using autoload, there is some issues concerning autoloading with ruby or jruby in general that makes theses classes to not loaded correctly when we actually need them.

One way to fix the problem is to use the eager loading function of the library and to use this patch https://github.com/logstash-plugins/logstash-output-s3/blob/master/lib/logstash/outputs/s3/patch.rb

This make sure the Error class is available when we need it.

[2016-11-07T03:17:22,325][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::S3 bucket=>"some-bucket", prefix=>"elb", type=>"elb", id=>"b5f5a4538615404515f06ab3ad934351b048b509-2", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_b4023de1-84e1-4e61-b74d-01eb189f7415", enable_metric=>true, charset=>"UTF-8">, region=>"us-east-1", delete=>false, interval=>60, temporary_directory=>"/tmp/logstash">
  Error: uninitialized constant Aws::Client::Errors

Created from elastic/logstash#6201

[Rodney Beede]

I installed Logstash 5.2 today and noticed that I have the file

/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-4.0.5/lib/logstash/outputs/s3/patch.rb

With the exact content as mentioned.

I still get the Error: uninitialized constant Aws::Client::Errors message though. Is there something I need to do to apply the fix?

[Alexander]

+1 @rbeede
I'm not a ruby guy, but ran into this issue and don't know how to fix it. It would be great, if someone could make a patch that needs to be copied/pasted only.

[Alexander]

@rbeede
I managed to apply this patch. This is what I did for the input plugin:

1. Created a directory <<your_logstash_dir>>/vendor/bundle/jruby/1.9/gems/logstash-input-s3-3.1.1/lib/logstash/inputs/s3
2. Put the patch file into this directory
3. Add require "logstash/inputs/s3/patch" to the register method (the method has several register commands in the beginning) of the Logstash::Inputs::S3 class in the <<your_logstash_dir>>/vendor/bundle/jruby/1.9/gems/logstash-input-s3-3.1.1/lib/logstash/inputs/s3.rb file.

 public
   def register
     require "fileutils"
     require "digest/md5"
     require "aws-sdk-resources"
     require "logstash/inputs/s3/patch"
 
     @logger.info("Registering s3 input", :bucket => @bucket, :region => @region)
 
     s3 = get_s3object
 
     @s3bucket = s3.bucket(@bucket)
 
     unless @backup_to_bucket.nil?
       @backup_bucket = s3.bucket(@backup_to_bucket)
       begin
         s3.client.head_bucket({ :bucket => @backup_to_bucket})
       rescue Aws::S3::Errors::NoSuchBucket
         s3.create_bucket({ :bucket => @backup_to_bucket})
       end
     end
 
     unless @backup_to_dir.nil?
       Dir.mkdir(@backup_to_dir, 0700) unless File.exists?(@backup_to_dir)
     end
 
     FileUtils.mkdir_p(@temporary_directory) unless Dir.exist?(@temporary_directory)
   end

Hope this is helpful for non ruby guys like me.