deflect-ca / baskerville

Security Analytics Engine - Anomaly Detection in Web Traffic

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Mechanism to avoid processing old logs when there's an input rate issue

mkaranasou opened this issue · comments

Whenever filebeat-logstash stall, we get much older logs to process at once. We eventually catch up but processing hours of old logs means that we're also sending out old challenge request to Banjax.
Two ways to handle this:

  • We do calculate F.min('stop') at some point. Use that to see if a time-window is irrelevant to current timestamp.
  • Use a watermark if possible (see here