Allow alerting on external endpoints that do not receive a push within a configurable time frame

[Devin Buhl]

For step 3. , will it also include the negative "Said endpoint must automatically check if an alert has not been triggered" i.e. raise an alert if no result received within a specified period ?
(aka push-based monitoring)

Originally posted by @r3mi in #722 (comment)

[TwiN]

Thank you for creating the issue.

Now that external endpoints have been implemented (#722, #724), this should probably be the next feature related to external endpoints that gets implemented as they kind of go hand in hand, especially for those using external endpoints to test connectivity. If there's no connectivity, Gatus' API won't be reachable, which means that Gatus wouldn't be able to trigger an alert without this feature.

The feature in question should allow the user to configure a duration under which an update is expected to be received.

Should the duration elapse with no new status update, a status should be created to indicate a failure to receive an update within the expected time frame.

This should in turn cause

gatus/watchdog/alerting.go

Lines 13 to 22 in 2833968

 func HandleAlerting(endpoint *core.Endpoint, result *core.Result, alertingConfig *alerting.Config, debug bool) { 
 	if alertingConfig == nil { 
 		return 
 	} 
 	if result.Success { 
 		handleAlertsToResolve(endpoint, result, alertingConfig, debug) 
 	} else { 
 		handleAlertsToTrigger(endpoint, result, alertingConfig, debug) 
 	} 
 } 

to be called, which would then lead to handleAlertsToTrigger being called (due to the new result indicating failure to receive an update having its Success field set to false), incrementing NumberOfFailuresInARow

gatus/watchdog/alerting.go

Lines 24 to 27 in 2833968

 func handleAlertsToTrigger(endpoint *core.Endpoint, result *core.Result, alertingConfig *alerting.Config, debug bool) { 
 	endpoint.NumberOfSuccessesInARow = 0 
 	endpoint.NumberOfFailuresInARow++ 
 	for _, endpointAlert := range endpoint.Alerts { 

& triggering whichever alerts should be triggered.

The only proper name I can think of for this feature is "dead man's switch", but as silly as it may sound, I don't like how that'd look on the configuration:

external-endpoints:
  - name: ...
    dead-man-switch:
      blackout-duration-until-automatic-failure: 1h
    alerts: 
      - type: slack
        send-on-resolved: true

Another consideration to make is the interaction between this feature and maintenance. While the maintenance period should prevent alerts from being triggered, should failure status be pushed anyways? Perhaps this should be an additional parameter on the maintenance configuration (e.g. maintenance.silence-dead-man-switch)?

Some food for thoughts.

[Devin Buhl]

The only proper name I can think of for this feature is "dead man's switch", but as silly as it may sound, I don't like how that'd look on the configuration:

I've seen other services call this heartbeat instead of dead man's switch, and they also have a grace period that is configurable.

external-endpoints:
  - name: ...
    heartbeat:
      interval: 5m
      grace-period: 5m
    alerts: 
      - type: slack
        send-on-resolved: true