earthgecko / skyline

Anomaly detection

Home Page:http://earthgecko-skyline.readthedocs.io/en/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Analyzer also alerting on Mirage metrics now

earthgecko opened this issue · comments

And so is Mirage.

The previous change that reintroduced 432e3da#diff-54c72b335b6c00a0c5e1b1f4720341a2R683 has resulted in the order matters being compromised so to speak.

The issue presents itself when a metric namespace or wildcard has the absolute or wildcard path mapped to a tuple with the SECOND_ORDER_RESOLUTION_HOURS declared, such as

Such as:

            ('mysql.counters', 'smtp', 3600, 168),
...
...
            ('stats.mysql-master-1.mysql.counters.handlerRead_rnd', 'smtp', 3600, 168),
...
...
            ('stats.*', 'smtp', 3600),

stats.mysql-master-1.mysql.counters.handlerRead_rnd matches and gets sent to Mirage, OK.

2016-08-14 17:06:49 :: 21921 :: mirage check :: stats.mysql-master-1.mysql.counters.handlerRead_rnd at 168 hours - matched by substring
2016-08-14 17:06:49 :: 21921 :: added mirage check :: stats.mysql-master-1.mysql.counters.handlerRead_rnd,742316.0,168
2016-08-14 17:06:49 :: 21921 :: mirage check :: stats.mysql-master-1.mysql.counters.handlerRead_rnd at 168 hours - matched by regex
2016-08-14 17:06:49 :: 21921 :: added mirage check :: stats.mysql-master-1.mysql.counters.handlerRead_rnd,742316.0,168

But if there is an ALERTS tuple further down the ALERTS tuples such as:

            ('stats.*', 'smtp', 3600),

This run will send the metric to the Analyzer alert, as it should. However that
does make it more difficult to separate out Mirage metrics by wildcards

2016-08-14 17:06:49 :: 21921 :: triggering alert :: stats.mysql-master-1.mysql.counters.handlerRead_rnd 742316.0 via smtp - matched by regex

So the simple solution here will be to add a Mirage Redis metrics keys for any
Mirage metrics and the alert expiry that are encountered higher up the ALERTS
tuples and have the analyzer_metrics block first check to see if a Mirage
metrics key exists for the metric and only process via Analyzer alerters if not.

That should do it.

Introduced a mirage.metrics Redis key namespace for any Mirage metric. This
acts as a dynamic SKIP_LIST for Analyzer so to speak and allows for wildcard and
substring matching and prevents Analyzer alerting on Mirage metrics.