Cannot access metadata any more since 3.1.9
eye8 opened this issue · comments
I use the plugin and extract the S3 file key from the metadata, which worked correctly in 3.1.8. But the same code does not work any more in 3.1.9 and 3.2.0. I confirmed it by uninstalling logstash-input-s3 then install again with --version 3.1.8. 3.1.8 ingests the files but the other two versions ingest no file at all.
The issue is encountered on Mac with VirtualBox. Haven't tested Linux. My config is as follows:
input {
s3 {
access_key_id => "{{ .Env.S3_KEY }}"
secret_access_key => "{{ .Env.S3_SECRETE }}"
bucket => "{{ .Env.S3_BUCKET }}"
prefix => "{{ .Env.S3_PREFIX }}"
interval => 60
region => "us-east-1"
}
}
filter {
mutate {
add_field => {
"file_key" => "%{[@metadata][s3][key]}"
}
}
}
output {
elasticsearch {
hosts => ["elasticsearch"]
index => "s3-file"
codec => rubydebug {
metadata => true
}
}
}
Some additional info. I turned on logstash debug. In 3.1.9 and 3.2.0, it seems the logstash-input-s3 plugin ignores/skips the input event somehow. In s3.rb you have the following code:
public
def list_new_files
objects = {}
@s3bucket.objects(:prefix => @prefix).each do |log|
@logger.debug("S3 input: Found key", :key => log.key)
unless ignore_filename?(log.key)
if sincedb.newer?(log.last_modified) && log.content_length > 0
objects[log.key] = log.last_modified
@logger.debug("S3 input: Adding to objects[]", :key => log.key)
@logger.debug("objects[] length is: ", :length => objects.length)
end
end
end
return objects.keys.sort {|a,b| objects[a] <=> objects[b]}
end # def fetch_new_files
Supposedly I should see all three debug messages (i.e. S3 input: Found key, S3 input: Adding to objects[], and objects[] length is: ). However I only see the Found Key message. Perhaps log.content_length isn't found (since 3.1.8 doesn't check log.content_length)?
@eye8 Is this still an issue? I tried to reproduce the bug, but the files were processed, and the file_key property was written correctly to the event from the metadata.
Also, I'm a little unclear as to whether the issue is that the metadata is not being written, or that no files are being processed at all.