Logster does not process lines on first call with new file
alastic opened this issue · comments
I create a file, call it "logfile" with just 3 lines in it. "ParserLogster.py" contains nothing but a print(line) in parse_line().
Running
logster --output=stdout ParserLogster logfile
Does not produce any output the first time. Running it again prints out all the lines.
On the first run of logster against a log file it hasn't seen before, it'll write out a state file for logtail2 then exit. It won't attempt to process any lines from the existing file.
This is by design. It avoids the situation where a very large file has logster run against it for the first time which attempts to ingest all lines from the file, potentially causing performance and resource issues on the host machine.
You can see the relevant code here
Thanks for the response.
In my tests logster does actually process all the lines in the file on the second run (the file is unchanged between runs, but I can try adding a couple of lines). Is that what you would expect?
That's certainly not the behaviour I'd expect. Are you able to provide a test case (with example log data and parser code) to reproduce the issue?
Sorry, it looks like I made a mistake with the last tests I ran. Started over and it only processes the added lines on the new example. Thanks for your time
Hello,
I am trying to make a parser and I was wondering if there's an argument I could pass to go into "testing" mode? The parse_line has stopped from being called and I think it might be because of a state file somewhere. I tried deleing /var/run/logtail-myparser.CustomParsertest.log.state but it still doesn't call parse_line. I think I may have missed something.