logstash-plugins / logstash-input-s3

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How can i get file name as index name from S3 to logstash

dilipaim11 opened this issue · comments

Hi
i have s3 bucket from where i taking csv file and uploading in to ES .
i stuck on how to get file name which i have in s3 , beacuse i am using s3 file name as Index name .
My config file -


input{
	s3{
	bucket => "***"
	access_key_id => "**"
	secret_access_key => "***"
	region => "**"

}
}

filter {
	csv {
		separator => ","
		columns  => [ "name","uid","startdate","enddate"]

	}
	mutate {convert =>[uid , "integer"]}
	
	  date {
      match => [ "startdate", "ISO8601", "YYYY/MM/dd HH:mm:ss","YYYY/MM/dd HH:mm" ]
      target => "startdate"
      locale => "en"
	  timezone => "Asia/Dubai"
    }
	
	  date {
      match => [ "enddate", "ISO8601", "YYYY/MM/dd HH:mm","YYYY/MM/dd HH:mm:s" ]
      target => "enddate"
      locale => "en"
	  timezone => "Asia/Dubai"
    }
	  mutate {
    add_field => {
      "index_name" => "%{[@metadata][s3][key]}"
    }
   
  }

  }

output {
	elasticsearch {
		hosts => "localhost"
		index => "%{index_name}"
	}
	stdout{}

}

Any suggestion

Looks like a duplicate of #104.

This is a duplicate of a closed issue and can probably be closed...