Add Needed Metatdata
Ralnoc opened this issue · comments
Currently the S3 plugin only adds a Message entry to the general message that gets sent to Elasticsearch. While this at least gets some of the information there, it would be extremely helpful to have the following added:
bucket -> Name of the Bucket being processed
path -> Path to the file line is extracted from
prefix -> S3 prefix being processed
This is metadata that would be extremely useful to have access to. For example when consuming EMR Logging. The Path would have the Cluster ID on it which would allow us to grok that into it's own field for searching.
The key will be in plugins version 3.1.3