[Feature Request] Add the ability to work with a S3 Compatible storage

[Nimrod Becker]

In its current state, the plugin can only work with AWS. I've tried to add the capability to work with any S3 compatible storage (which supports the AWS authentication) by adding the endpoint into the S3 object creation.

However, it seems that the current SDK version does not support endpoint in the initialization of the S3 object (V2 of the SDK supports it via the AWS::Client object).

Edit:
For some reason I'm able to use the Aws::S3::Resource object (although according to the docs, it should only be available at the v2 of the SDK). Using it I can

client = Aws::S3::Client.new(
        ...
        :endpoint => @s3_endpoint
        )
s3 = Aws::S3::Resource.new(client: client)

Working with the s3 object, I can access the S3 compatible bucket, however, the entire plug-in breaks (obviously) as it expects an s3 object with .buckets (and not a single resource). Sadly I'm not proficient in Ruby enough to go over and fix all the flows.

[Salim B]

FWIW, the following pull request is adding the ability to change the default endpoint in the logstash-output-s3 plugin :
logstash-plugins/logstash-output-s3#100 (review)

@andrewgaul any chance to see you work on the same feature for the input plugin ?

[Andrew Gaul]

@sboulkour I do not plan to further contribute to logstash. Upstream merging my original PR looks doubtful after four months.

@nimrod-becker The Ruby v1 API supports custom endpoints via another syntax. You can see it in an older version of my logstash-output-s3 PR:

https://github.com/andrewgaul/logstash-output-s3/tree/custom-endpoint-old

Hope this helps.

[Salim B]

@andrewgaul thx for your inputs

I read your PR. I didn't understand what you do with uri = URI(@endpoint) when creatin the endpoint hash : any hint ?

[João Duarte]

the mixin used in all s3 plugins now has endpoint setting available

As soon as the mixin is released and documentation is added, this feature will be supported in the s3, kinesis and sqs plugins.