how to store processed lines events?
crunchie84 opened this issue · comments
The RX pipeline does this:
- find new or modified files
- foreach file determine the not-yet-emitted byte length (string)
- chunk the given string to lines (log lines)
- do something with those lines
Somehow i need to have a meta stream so i can persist which files i have processed how far so if i restart the process i can continue where we left off without data loss or duplication. But i have no clue how to do this yet.