S3 v4 signatures for logstash input plugin

[abeyt]

Hi ,

I was trying to collect all my AWS config logs from S3 via logstash and could see it list the filenames fine but once listing is complete it starts to download the S3 file and that fails with an error "Error: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.". Once it gets this error it will restart the plugin and start listing the files again and then error.

Here is the complete log (replaced file paths)

S3 input: Download remote file {:remote_key=>"logs/111/file_brqvFXbF63.json.gz", :local_filename=>"/tmp/logstash/file_brqvFXbF63.json.gz", :level=>:debug, :file=>"logstash/inputs/s3.rb", :line=>"344", :method=>"download_remote_file"}
A plugin had an unrecoverable error. Will restart this plugin.
Plugin: "s3bucket", codec=>"UTF-8">, interval=>600, prefix=>"logs/111/", region=>"eu-west-1", use_ssl=>true, delete=>false, temporary_directory=>"/tmp/logstash">
Error: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.
Exception: AWS::S3::Errors::InvalidArgument
Stack: /opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in return_or_raise'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:inclient_request'
(eval):3:in get_object'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:1371:inget_object'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:1090:in read'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:346:indownload_remote_file'
org/jruby/RubyIO.java:1183:in open'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:345:indownload_remote_file'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:321:in process_log'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:151:inprocess_files'
org/jruby/RubyArray.java:1613:in each'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:146:inprocess_files'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:102:in run'
org/jruby/RubyProc.java:281:incall'
/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/interval.rb:20:in interval'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:101:inrun'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/pipeline.rb:331:in inputworker'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/pipeline.rb:325:instart_input' {:level=>:error, :file=>"logstash/pipeline.rb", :line=>"342", :method=>"inputworker"}
S3 input: Found key {:key=>"logs/111/fil2_akbtQDyyS6nHW.json.gz", :level=>:debug, :file=>"logstash/inputs/s3.rb", :line=>"111", :method=>"list_new_files"}

I can see in logstash-output-s3 plugin "Support S3 v4 signatures" was added some time this year. Does the input plugin has support for AWS S3 v4 signatures?

Regards,
Abey

[tunafish0805]

take a look inside logstash/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/s3/config.rb

you'll notice this:

 add_option : s3_signature_version do |config, value|
  v3_regions = %w(
  us-east-1
  us-west-1
  us-west-2
  ap-northeast-1
  ap-southeast-1
  ap-southeast-2
  sa-east-1
  eu-west-1
  us-gov-west-1
)
if value
  value
elsif config.s3 && config.s3[:signature_version]
  config.s3[:signature_version]
elsif v3_regions.include?(config.s3_region)
  :v3
else
  :v4
end

I implemented a workaround by removing the regions I'm using from v3_regions. This is certainty not a solution but will work if your in a pinch.

[abeyt]

Thanks @tunafish0805. That fix worked. Yes i understand it is not a solution.

[Alexander]

I've run into the same issue. It's quite strange why it's not possible to configure the signature version, because the s3 output plugin had same issue and it was fixed in this issue.
I'm using older version of logstash (2.3.4) and found another solution for the logstash versions which use LogStash::PluginMixins::AwsConfig::V1

 public
  def aws_options_hash
    opts = {}
    #    ...
    if @access_key_id && @secret_access_key
      opts = {
        :access_key_id => @access_key_id,
        :secret_access_key => @secret_access_key
      }
      opts[:session_token] = @session_token if @session_token
    elsif @aws_credentials_file
      opts = YAML.load_file(@aws_credentials_file)
    end
    # ...
    return opts
end

The config reads all properties from a credentials file. So it is possible to define the signature version in this file. If you're using IAM based authentication it still works, if you keep credentials in the config file, they should be moved to a separate file that has a signature version as well.
Example of a credentials file with signature version:

:s3_signature_version: "v4"

I thinks it's worth to allow configuring the signature process version because amazon can introduce a new version in the future and it can lead to same issues.