Alerts triggered status is not persistent
macmoritz opened this issue · comments
Describe the bug
The triggered status of a monitor is only momentary. After a restart of gatus the information is gone.
Such restarts are daily business in cloud environments.
What do you see?
No response
What do you expect to see?
Gatus should only send one alert for a failing monitor.
List the steps that must be taken to reproduce this issue
- gatus starts
- after some time a monitored service fails
- gatus sends an alert after checking if alert was sended before
- gatus will not send more notifications for this monitor, which is correct (log:
[watchdog][handleAlertsToTrigger] Sending matrix alert because alert for endpoint=my-endpoint='' has been TRIGGERED) - gatus restarts
- gatus will send an alert for the monitor again
Version
No response
Additional information
Fix could be to use the events from the database for checking if an alert was already sended.
Yes, you're right. It's also something I've been wanting to take care of for a while.
I would also add something like UUID of the sent alert and keep the same UUID when the alert gets resolved or sending is repeated by any reason. That UUID should be available in the alert templating.