Configuration for shard iterator AT_TIMESTAMP
santoshkulk opened this issue · comments
KCL provides an option to specify the initial position in the stream from where the application starts reading from. It can be one of LATEST, TRIM_HORIZON or AT_TIMESTAMP. I do not see a configuration option currently available in the logstash configuration to specify the same. Is there anything I am missing here?
Nope, it's just never come up so far. Right now it's hard-coded to TRIM_HORIZON: https://github.com/logstash-plugins/logstash-input-kinesis/blob/master/lib/logstash/inputs/kinesis.rb#L73
Making this configurable is certainly reasonable though.
I second this request.
We have a number of streams with 7 day retention, so as we're setting up new logstash stream processors, each one has to chew through a week worth of data that we've already processed with our existing system. It would be great to be able to tell it to just start processing from now and ignore the existing data.
We have the same use case @tschran so would love to see this feature.
hey @codekitchen I took a stab at adding support for LATEST here.
I have never written ruby code in my life, so I'm sure it will need a little revision.
+1
LATEST is now supported thanks to a PR from @aloisbarreras
AT_TIMESTAMP support involves further work if anybody is interested in tackling it -- most likely another config parameter for the timestamp. I'm not positive if logstash natively supports timestamp config vars or if we'd need to parse it out from a string in this plugin.
Hello,
Is this configuration available for use in kinesis input plugin now?