Health Checks are 404ing
rocketnova opened this issue · comments
Description
After the R2 maintenance release, @lomky noticed that the calls to /claimstatus/api/health were all returning 404s. After some research, we found that the health checks in all environments are returning 404s.
We confirmed that when we (open the network firewall and) curl prod /api/health, we get a 200, but when we curl /claimstatus/api/health, we get 404.
We checked to see if this behavior could be caused by the WAF, but confirmed that the WAF configuration is the same between Prod and Staging. We also didn't change the WAF settings for Prod as part of the release last night.
We isolated the change in behavior in the Staging env to having started with release-171 that was a proof of concept of #508. As a result, we believe the correct Health Check configuration for Front Door is to hit the App Service url (not the Front Door url) and hit /api/health (not /claimstatus/api/health anymore).
We were also surprised to not receive any alerts around this (we think the failure anomolies alert should have caught this), so we'll configure some additional alerts specifically for the health check.
Acceptance Criteria
- Ask CDT to update the Front Door health check url to remove
/claimstatuson the Staging env - If that works, ask CDT to update in all environments
- Set an alarm for health check failures in all environments
Today, CDT helped us update the Front Door health probe for Staging from /claimstatus/api/health to /api/health, which resolved the 404s we were seeing on Staging. We'll continue working with CDT to resolve the other environments.
CDT has completed the fix on all other environments, I have confirmed the fix is working, and I have set up the alerts.