Mechanism to avoid processing old logs when there's an input rate issue
mkaranasou opened this issue · comments
Whenever filebeat-logstash stall, we get much older logs to process at once. We eventually catch up but processing hours of old logs means that we're also sending out old challenge request to Banjax.
Two ways to handle this:
- We do calculate
F.min('stop')
at some point. Use that to see if a time-window is irrelevant to current timestamp. - Use a watermark if possible (see here