weaveworks / alertdiff

Expose metrics detailing differences between two alert sources

Introduction

alertdiff is a service that continuously compares the alerts generated by a Prometheus-compatible implementation (e.g. Weave Cortex) with a reference Prometheus instance, and exports metrics which detail spurious or missing alerts.

Usage

• Deploy alertdiff
• Append e.g. http://alertdiff:8080/expected/api/v1/alerts to the --alertmanager.url of your reference Prometheus server
• Add http://alertdiff:8080/actual/api/v1/alerts as an alertmanager to Cortex
• Ensure both Prometheus & Cortex are configured with the same set of alerting rules and are receiving identical metrics

Then add some alert definitions based on the exported metrics:

ALERT AlertDiffMissingAlert
  IF alertdiff_missing_alert > 0
  FOR 1m

ALERT AlertDiffSpuriousAlert
  IF alertdiff_spurious_alert > 0
  FOR 1m

Getting Help

If you have any questions about, feedback for or problems with alertdiff:

• Invite yourself to the Weave Users Slack.
• Ask a question on the #general slack channel.
• File an issue.

Weaveworks follows the CNCF Code of Conduct. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting a Weaveworks project maintainer, or Alexis Richardson ([e-mail]).

Your feedback is always welcome!