The fictional Smartlink company UrlSpark is running a highly redundant geographically distributed service serving Millions of users daily. The product Teams are fairly independent teams responsible for their own product and applications including deployments but not necessarily the infrastructure itself. The Infrastructure team oversees the Cloud infrastructure, the managed services and the self-hosted systems supplying the product teams with necessary systems and tools for them to succeed.
This is a task for the Cloud Operations Manager role, it tackles a common process with a dilemma scenario meant to facilitate a discussion on the topic. Please document your approach and thought process throughout. You should make your own assumption for all unknowns based on your past experience.
An underlying system is failing. Alerts are triggered across multiple applications spanning various teams during normal work hours. Engineers are engaged in finding a solution, customers are complaining and SLA's are being affected.
Please describe a common disaster recovery process which can be applied to this scenario. Assuming the process is followed, how could it play out throughout the steps of your plan.
The incident is affecting our internal SLO's and our SLA's towards certain customer segments. Engineers have devised a plan for a fix which will restore the critical services, but due to time constraints and dependencies the fix does not adhere to QA gates and production readiness checklists. The fix will negatively impact the teams quarterly objectives for ensuring high quality software and optimized systems which eventually leads to exceeding financial operational budgets and slower development due to tech debt.
Using your past experience and own assumptions of the variables how do you deal with the situation.