Giters

stefanprodan / swarmprom

Docker Swarm instrumentation with Prometheus, Grafana, cAdvisor, Node Exporter and Alert Manager

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Stack have message from Alert Manager, but Alert Manager dashboard and Unsee display nothing

hieu29791 opened this issue · comments

commented

Hi, I deploy stack as you said. Slack has message from AlertManager, but AlertManager dashboard show nothing, so Unsee can't get message from alert manager. When I clicked message link from slack, it shows containerID:9093, I think issue because of it. But I don't know how to fix. Could someone help me figure out this issue.
screen shot 2018-10-25 at 5 43 15 pm
screen shot 2018-10-25 at 5 45 38 pm 2
screen shot 2018-10-25 at 5 43 57 pm 2
screen shot 2018-10-25 at 5 44 03 pm 2

commented

Hi, if the alert condition later went away, it would no longer appear in Alertmanager or via Unsee.

To address the container names being sent in Alertmanager emails and in the links within Slack, you'll want to add the web.external-url command to your docker-compose.yml in both the alertmanager: and prometheus: services command: sections:

prometheus:
image: stefanprodan/swarmprom-prometheus:v2.2.0-rc.0
networks:
- net
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--web.external-url=https://externalhostname.example.com:9093/'

I think this also fixes the source links back into Prometheus.

I've been working on a project called swarmstack recently that extends Stefan's swarmprom, it's available at https://github.com/swarmstack

commented

@mh720 Thank you so much for your reply. If I want to see all alert from Slack post to Alertmanager and Unsee, what can I do? As you said, if alert went away, it no longer appears again.

commented

Hi @hieu29791 , It's the other way around. You should see all message posted FROM Alertmanager within your slack channel (where they would remain). Alertmanager only cares about the CURRENT state of alerts raised by Prometheus and manual silences in effect. Unsee only reads the state of one or more Alertmanager instances (single pane of glass to watch multiple Alertmanager instances). If you want to see historical alerts that Prometheus sent, if you have Prometheus scraping your Alertmanager(s), in PromQL you can graph the query "alertmanager_alerts" and see when alerts were greater than 0. Beyond that, you might need to go looking through Prometheus or Alertmanager log files to find traces of historical alerts.