How to setup a second notification channel to be used only when the first is down ?
tradexsrl opened this issue · comments
⚠️ Please verify that this question has NOT been raised before.
- I checked and didn't find similar issue
🛡️ Security Policy
- I agree to have read this project Security Policy
📝 Describe your problem
Hi , i have installed uptime kuma and setup ntfy notification
ntfy is self hosted in our datacenter , how can i setup for example pushover or another external notification service to be used only if ntfy for some reason is down?
📝 Error Message(s) or Log
No response
🐻 Uptime-Kuma Version
1.23.11
💻 Operating System and Arch
docker
🌐 Browser
chrome - firefox
🖥️ Deployment Environment
docker
How do you determine that ntfy is down?
You can set up a monitor to notify you of this via a different notification provider.
if the notification from uptime kuma doesn't go through ntfy i think it reply with an error code like 404 or similar , so in this way uptime kuma can select the second notification method
yes i can monitor ntfy docker and send me a notification via pushover or other external system but if i have other down these doesn't go through because the default is ntfy....
i mean other down at the same time of ntfy down and this can happend because the docker host is only one with many services with also ntfy
yes i can monitor ntfy docker and send me a notification via pushover or other external system but if I have other down these doesn't go through because the default is ntfy....
I mean other down at the same time of ntfy down and this can happen because the docker host is only one with many services with also ntfy.
I don't quite get what you want to convey here.
If I get a notification that my notification system is down, I would just assume everything is on fire => look into the dashboard.
I am not convinced that adding a failover subsystem is necessary.
i'll try to explain: as much as i can , i self host everything but i need to have a least external way to get notification if something very bad happend , that's why i ask for a failover on notification
another example: we self host chat system - mail system and so on , in the worst case scenario where in our datacenter these system went down (already happenend) i used a whatsapp group specially design to tell our collegue that there are these problems , avoiding tons of call requesting info.
i want to add that if my notification system is down doesn't mean that everything is on fire...maybe just this docker container is exited and other system are in good shape...that's why we have two system for both critical services we host ; if one goes down no problem to hurry to solve....even in weekend but i cannot have 2 system for hosting 2 ntfy because ip address change and i have no failover system on uptime kuma
Then why don't you route the alerts that the chat system (or whatever critical piece of infra) is down via all notification channels (including said group)?
but i cannot have 2 system for hosting 2 ntfy because ip address change
What do you mean by this part. You likely have some sort of reverse proxy in front of said contaiers => why not handle this redundancy in traefik/nginx?
do you mean to setup a default notification channel to external pushover system instead of ntfy?
because we do not want to use external services that are not controllable by us.
we allow to use external service only in the worst case scenario that in 99% never happend
but i cannot have 2 system for hosting 2 ntfy because ip address change
What do you mean by this part. You likely have some sort of reverse proxy in front of said contaiers => why not handle this redundancy in traefik/nginx?
Because in that case i should install another instance of ntfy container on another host that must have another ip + add traefik/nginx on a third host and setup al the stuff...i think is complicated and i don't have a third host
do you mean to setup a notification provider to external pushover system instead of ntfy?
If your self-hosted chat system fails, then yes:
Add an external notification provider for that one. Using the failed service is unlikely to work.
Notify via all providers for critical infrastructure (getting more notifications for critical failures is fine!).
Depending on how critical your infra is:
Don't trust one monitor, add another one in a different Geografic region and a different provider.
my infra is very critical since all services are provided from the main HQ.
i can add another notification channell on the critical service so maybe i will receive twice the notification ....but gain i hope uptime kuma will consider failover notification.
i'm start to thinking of using webhook and pass the notification to another engine like n8n - activepieces (ifttt like) to implement the feature i need.
Sorry, but I think solving your need via the existing tools is sufficient to solve your usecase.
You would need to go further into why receiving multiple notifications (if you have set up multiple notification providers and all of them are up) for your primary notification provider being down is unacceptable/bad.
If your reliability requirements are that strict you likely want to use multiple monitoring service anyway instead of trusting on one single service like uptime kuma.
Just add a monitor to monitor your notification provider which notifies via an external provider?1
=> this way we don't add unnecessary complexity.
This Feature request is closely related to #1315
I think that failovers and business continuity should be handled by the notification provider.
When you say running a second server is out of the question
Footnotes
I have the same setup. Kuma with ntfy.
What i do is monitor the ntfy healthckeck endpoint with kuma https://docs.ntfy.sh/config/#health-checks
If ntfy is down, then kuma alerts to pushover. I also have uptimerobot monitor the kuma status page, and that alerts to pushover if kuma is down.
You can go on and on like this.
"Who watches the watchmen?" 🥸
I have the same setup. Kuma with ntfy. What i do is monitor the ntfy healthckeck endpoint with kuma https://docs.ntfy.sh/config/#health-checks If ntfy is down, then kuma alerts to pushover. I also have uptimerobot monitor the kuma status page, and that alerts to pushover if kuma is down.
You can go on and on like this.
"Who watches the watchmen?" 🥸
Hi , i have created an account on uptimerobot.com and setup the integration to notify via pushover but i never gest a push message...i saw i only need to insert user key but in uptime kuma i need to insert user key and token application
how do you setup pushover notification ?