garbage collection does not work correctly after removing alerts from prometheus

[Marcin Kowalski]

I had some misconfigured alerts defined in prometheus. I eventually removed them, but signalilo keeps restoring them to icinga2. and they hang around with unknown status.

How can i clean them up? They are no longer present in my prometheus configuration.

[Simon Gerber]

[...] signalilo keeps restoring them to icinga2

Signalilo itself doesn't keep any state about alerts, so this should only happen if alerts are repeated by Alertmanager.

How can i clean them up? They are no longer present in my prometheus configuration.

Since Signalilo creates "passive" services in Icinga2, you should be able to mark stale alerts as resolved (status OK) by clicking "Check Now" in Icingaweb, at which point Signalilo will consider them for garbage collection. Alternatively you can manually delete services with curl using

signalilo_user=<icinga api user for signalilo>
signalilo_password=<icinga api pw for signalilo>
curl -u "${signalilo_user}:${signalilo_password}" 'https://icinga-api.example.com:5665/v1/objects/services/signalilo_host!servicename' -XDELETE

If the services move back to Unknown after you've cleaned them up using either approach, you'll probably want to check your Prometheus/Alertmanager stack for stale replicas which still have the misconfigured alerts.

[Simon Gerber]

Since I haven't heard back, I'm closing this issue, as it's been working as intended for us.