Question: Do I have to invoke logster every minute?

[Mark Crossfield]

Apologies if this is not the right forum.

Inspired by the Etsy talk at Velocity, I'm analysing a few Apache files with logster and the total run (admittedly on a VM on my desktop) takes around 3 minutes sometimes, depending on how smoothly our servers flush their log volume. As a result I was getting warnings when running logster every minute. I've switched to every 5, but now I end up with gaps in my data in Graphite. I've included my invocation of logster below (I've renamed SampleLogster to ApacheLogster). Am I missing something?

/usr/sbin/logster --output=graphite --graphite-host=graphite:2003 -p apache.dc1.liveapp125 ApacheLogster /home/vmplanet/dws/dc1/liveapp125/apache/access.log

Thanks in advance, Mark

[Mike Brittain]

I would surprise me that if parsing 1 minute worth of logs takes 3 minutes for you, but 5 minutes worth does not?

You can run logster at whatever interval you want. If you're sending data to Graphite and you have gaps, it's likely that you are not sending data often enough for the buckets provided in the data retention schema setup in Graphite. You can customize your retention so you only expect a datapoint every 5 minutes. You should be able to find that in the Graphite docs.

[Mark Crossfield]

Thanks - I hadn't understood the relationship between the carbon retention bucket sizes and the cron schedule when I submitted that question. I've had it explained to me now :)