Giters

etsy / logster

Parse log files, generate metrics for Graphite and Ganglia

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Multiple output types causes problems with prefix

MrSecure opened this issue · comments

commented

I noticed that passing multiple --output arguments is supported, however, there seems to be a bug when used with a prefix. In my test, I attempted to send to stdout, statsd, and graphite. Below is the shell output & logster.log data for one test without a prefix, and one with.

I'm not strong on python, but I would guess that the bug is somehow related to the multiple calls to submit_XXX(metrics, ...) actually modifying the contents of the metrics data when adding the prefix.

Thank you for looking into this.

[cosmo:/tmp]$ logster --output=stdout --output=statsd --output=graphite --graphite-host=127.0.0.1:2003 --statsd-host=127.0.0.1:8125 --state-dir=/tmp/logster --debug SampleLogster /tmp/access.log
1377652162 http_1xx 0.0
1377652162 http_2xx 0.0
1377652162 http_3xx 0.0
1377652162 http_4xx 0.0
1377652162 http_5xx 0.0
[cosmo:/tmp]$ logster --output=stdout --output=statsd --output=graphite --graphite-host=127.0.0.1:2003 --statsd-host=127.0.0.1:8125 --state-dir=/tmp/logster -p BUG --debug SampleLogster /tmp/access.log
1377652194 BUG_BUG.http_1xx 0.0
1377652194 BUG_BUG.http_2xx 0.0
1377652194 BUG_BUG.http_3xx 0.0
1377652194 BUG_BUG.http_4xx 0.0
1377652194 BUG_BUG.http_5xx 0.0
[cosmo:/tmp]$

and the corresponding /var/log/logster/logster.log

2013-08-27 20:09:22,492 INFO     Executing parser logster.parsers.SampleLogster.SampleLogster on logfile /tmp/access.log
2013-08-27 20:09:22,492 DEBUG    Using state file /tmp/logster/logtail-logster.parsers.SampleLogster.SampleLogster-tmp-access.log.state
2013-08-27 20:09:22,493 DEBUG    Locking successful
2013-08-27 20:09:22,493 DEBUG    Setting duration to 437.0 seconds.
2013-08-27 20:09:22,512 DEBUG    Submitting Graphite metric: http_1xx 0.0 1377652162
2013-08-27 20:09:22,512 DEBUG    Submitting Graphite metric: http_2xx 0.0 1377652162
2013-08-27 20:09:22,513 DEBUG    Submitting Graphite metric: http_3xx 0.0 1377652162
2013-08-27 20:09:22,513 DEBUG    Submitting Graphite metric: http_4xx 0.0 1377652162
2013-08-27 20:09:22,513 DEBUG    Submitting Graphite metric: http_5xx 0.0 1377652162
2013-08-27 20:09:22,514 DEBUG    Submitting statsd metric: http_1xx:0.0|g
2013-08-27 20:09:22,514 DEBUG    Submitting statsd metric: http_2xx:0.0|g
2013-08-27 20:09:22,514 DEBUG    Submitting statsd metric: http_3xx:0.0|g
2013-08-27 20:09:22,515 DEBUG    Submitting statsd metric: http_4xx:0.0|g
2013-08-27 20:09:22,515 DEBUG    Submitting statsd metric: http_5xx:0.0|g
2013-08-27 20:09:22,515 INFO     Total execution time: 0.0 seconds.
2013-08-27 20:09:22,515 DEBUG    Unlocking successful

2013-08-27 20:09:54,159 INFO     Executing parser logster.parsers.SampleLogster.SampleLogster on logfile /tmp/access.log
2013-08-27 20:09:54,159 DEBUG    Using state file /tmp/logster/logtail-logster.parsers.SampleLogster.SampleLogster-tmp-access.log.state
2013-08-27 20:09:54,159 DEBUG    Locking successful
2013-08-27 20:09:54,160 DEBUG    Setting duration to 32.0 seconds.
2013-08-27 20:09:54,180 DEBUG    Submitting Graphite metric: BUG.http_1xx 0.0 1377652194
2013-08-27 20:09:54,180 DEBUG    Submitting Graphite metric: BUG.http_2xx 0.0 1377652194
2013-08-27 20:09:54,180 DEBUG    Submitting Graphite metric: BUG.http_3xx 0.0 1377652194
2013-08-27 20:09:54,180 DEBUG    Submitting Graphite metric: BUG.http_4xx 0.0 1377652194
2013-08-27 20:09:54,180 DEBUG    Submitting Graphite metric: BUG.http_5xx 0.0 1377652194
2013-08-27 20:09:54,181 DEBUG    Submitting statsd metric: BUG.BUG_BUG.http_1xx:0.0|g
2013-08-27 20:09:54,181 DEBUG    Submitting statsd metric: BUG.BUG_BUG.http_2xx:0.0|g
2013-08-27 20:09:54,181 DEBUG    Submitting statsd metric: BUG.BUG_BUG.http_3xx:0.0|g
2013-08-27 20:09:54,181 DEBUG    Submitting statsd metric: BUG.BUG_BUG.http_4xx:0.0|g
2013-08-27 20:09:54,181 DEBUG    Submitting statsd metric: BUG.BUG_BUG.http_5xx:0.0|g
2013-08-27 20:09:54,181 INFO     Total execution time: 0.0 seconds.
2013-08-27 20:09:54,181 DEBUG    Unlocking successful

 commented 
def submit_stdout(metrics, options):
    for metric in metrics:
        if (options.metric_prefix != ""):
            metric.name = options.metric_prefix + options.stdout_separator + metric.name


Will that code alter the "metrics" variable that was passed into submit_stdout() ?
 commented 
Yup I'm seeing this as well... each submit_* should have a local metric_name (or so) to alter. I'm going to test out such a change ...
 commented 
Sent a pull request fwiw
 commented 
Fixed by #73