s3 unrecoverable error: unexpected token
cdenneen opened this issue · comments
logstash-plugins/logstash-codec-cloudtrail#1 (comment)
{:timestamp=>"2015-06-11T16:12:27.808000-0400", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::S3 bucket=>\"our.cloudtrail.NNNNNNNN\", credentials=>[\"XXXXXXXXXXXXX\", \"YYYYYYYYYYYY\"], region_endpoint=>\"us-east-1\", sincedb_path=>\"/var/log/logstash/cloudtrail.NNNNNNN.sincedb\", type=>\"cloudtrail\">\n Error: unexpected token at '0c22b8e6593b3eabfb00cf5f1ed73cba1a1200fdf19aeb0646b9da1d01522010 produsaphotoevent [07/Jun/2015:23:40:37 +0000] XXX.XXX.XXX.10 arn:aws:iam::NNNNNNNN:user/Prod_User 5B8075E1A6D42855 REST.GET.BUCKET - \"GET /?prefix=HeartBeat%2FHeartBeat.txt HTTP/1.1\" 200 - 605 - 59 58 \"-\" \"aws-sdk-dotnet/1.5.18.0 .NET Runtime/4.0 .NET Framework/4.0 OS/6.1.7601.65536 S3Sync\" -\n'", :level=>:error}
I've purged the bucket and then new message seems to crash s3 plugin.
Is this a bug in the cloudtrail codec or the s3 input or both?
Could be both, do you have a sample log to reproduce it?
ok so found the root cause. the same bucket that is used for cloudtrail logs also had a folder with other logs that were blowing up the plugin.
So the question is... to avoid that I think it might be good idea for the cloudtrail codec to always use the AWSlogs folder as it's root?
Cloudtrail always dumps to /AWSLogs/{accountid}/CloudTrail/{region}