The Disaster Recovery Project Everyone Blamed

I didn't start this project. By the time it landed on my desk, a multi-region active-active disaster recovery initiative had already been a company-level priority for six months, and it had delivered nothing. Worse, it had become the explanation for everything else that went wrong. Any unrelated production incident, any missed deadline, any awkward question from a client: "it's because of the DR project" had become a convenient answer whether it was true or not.

If you've ever inherited an initiative in that state, you know the first feeling isn't technical. It's "where do I even start, and who do I trust to tell me the truth about where this actually stands?"

What is disaster recovery

Before we get into it, let's give a quick overview of what disaster recovery is and the problems it aims to solve. At its core, disaster recovery is about keeping a system running, or getting it back up quickly, when something takes out the infrastructure it normally runs on: a region outage, a bad dependency, anything that makes the primary environment unavailable. How that's solved generally falls on a spectrum. Active-passive keeps a secondary region warm but idle, ready to take traffic if the primary fails. Pilot light / standby goes a step leaner: core infrastructure exists in the second region but stays mostly powered down until it's needed, trading cost for a slower recovery. Active-active, what this project was building, runs both regions live and serving production traffic at the same time, so failover means shifting traffic rather than waking anything up. It's the fastest recovery, but the hardest to build and keep correct, because both regions have to behave identically at all times.

That spectrum is a cost curve as much as a technical one. Active-passive and pilot-light keep the bill down by leaving the secondary region mostly or fully idle until it's actually needed, in exchange for a slower recovery. Active-active means paying to run full production capacity in two regions at once, every day, whether or not either region ever fails: not a one-time setup cost, but an ongoing one that shows up on every monthly AWS bill. So the real question isn't "which option is most resilient," it's "what recovery time can the business actually tolerate, and is that worth paying for continuously." In this case, the enterprise client requirements that made active-active a sales checkbox were also what justified the spend; for a lot of companies, pilot-light or active-passive is the right call, not a lesser one.

What I found in the first few weeks

The decision to build active-active disaster recovery across two AWS regions had been made before I got there, for reasons that made sense: the company operates in fintech, and enterprise clients ask hard questions about availability. Active-active DR wasn't just an engineering nicety, it was a checkbox on the security questionnaires that let large financial institutions sign off on using the product.

What I didn't inherit was a clear picture of where the project actually stood. There weren't many design documents to go back to, and the backlog was a scatter of tickets with no obvious throughline. So I did what I could: talked to the individual contributors who'd been doing the implementation, talked to stakeholders outside the team, and used AI to chew through the repositories and old tickets and spit out mermaid diagrams of what had actually been built. That gave me a usable mental model in days instead of weeks: a brain dump I could react to and correct, rather than a blank page.

Here's where I have to separate what I was told from what I concluded myself. I was told the infrastructure team had been struggling to make progress. What I concluded, after looking at the codebase and sitting in on how decisions got made, was that the project had stalled for a mix of reasons: nobody seemed to agree on what specific problem it was solving beyond the company-level mandate, there wasn't strong technical leadership steering the implementation, and the Terraform codebase underneath it all was its own obstacle to making any confident change.

The lessons

Disaster recovery is a product problem before it's an infrastructure problem. The infrastructure team had been absorbing issues quietly: hitting a wall, trying to patch it themselves even without the domain knowledge to do it safely, and not asking for help. The fix wasn't a better architecture diagram. It was going to other teams and stakeholders and sharing what we'd actually found, then prioritizing the work that depended on them. A problem shared is a problem halved, and a lot of what looked like a stuck infrastructure project was really a stuck cross-team coordination problem wearing an infrastructure costume.

Your health check has to test the thing you're actually trying to detect. One of the first things I reviewed was how failover decisions got made. The system was using a GraphQL endpoint that, internally, checked the availability of around thirty microservices. Two problems followed from that: a bad deploy that went out to both regions at once could make the check fail everywhere simultaneously, and a single unresponsive service didn't actually mean the region hosting it was down. We were measuring service health and calling it region health. That gap had to close before anything else about the failover plan meant anything.

Untested failover is a belief, not a fact. The traffic split between regions, managed through Route 53 weighted DNS records, defaulted almost all traffic to the primary region. That's a comfortable setup right up until the day the secondary region has a problem nobody noticed because nothing was running there. We moved to an 80/20 split specifically to surface secondary-region issues early, rather than discovering them during a real incident. Then we ran the test that actually matters: shifting 100% of traffic to the secondary region. It passed cleanly, and we left it running there for a few days afterward, watching alerts, because a clean first hour doesn't rule out a problem that only shows up on day three.

Hardcoded assumptions don't survive contact with a second region. Partway through, we found microservices that assumed which AWS region they were running in (hardcoded values, in some cases baked into database records). None of that is fixable from inside the infrastructure team. It required going to other product teams directly, getting the region made configurable per environment instead of assumed, and cleaning up the data that had baked the assumption in. Multi-region isn't an infrastructure feature you bolt on; it's a constraint the whole system either respects or doesn't.

The foundation matters more than the failover plan sitting on top of it. One of the bigger surprises: the VPC wasn't built the way AWS's own Well-Architected Framework recommends. Instead of three availability zones, each with its own NAT gateway, it was running on two AZs behind a single NAT gateway: a single point of failure sitting directly underneath a disaster recovery initiative. No amount of clever failover logic fixes a foundation that wasn't built for the failure modes you're trying to survive. And this isn't a hypothetical risk reserved for unlucky companies: every major cloud provider has had a regional-scale outage in the last few years, which is exactly why the regional redundancy this project was chasing mattered in the first place.

Complexity doesn't stay contained, it compounds. The health-check logic had grown complicated enough that people genuinely struggled to reason about what was happening during an actual failover. Separately, and just as consequential, the Terraform codebase had over a thousand resources living in a single remote state, with symbolic links threading a shared base configuration across three AWS accounts and two regions (six environments, all wired together). Changing anything safely was hard, people kept stepping on each other, and that discovery turned into its own remediation project.

What I'd do differently

Honestly, not much about the approach. Walking in and spending real time understanding the history, the people, and the state of things before proposing fixes is exactly what I'd do again. The one thing I'd double down on even earlier is what already worked well: using AI to rapidly reconstruct context, summarizing repos and generating diagrams from scattered tickets, instead of trying to rebuild that picture by memory and meetings alone. It's the fastest way I've found to go from "I have no idea what state this is in" to "I can ask a useful question" in days rather than weeks.

The initiative itself got delivered. What had been stalled for six months, and had become everyone's go-to explanation for unrelated problems, is now a working active-active setup across both regions, tested and running, not a slide in a status update.

If you're inheriting a stalled initiative

A few things are usually true when an initiative has stalled: nobody can give you a one-line answer for what problem it's actually solving, the team closest to it has been absorbing problems alone instead of escalating them, the checks meant to validate it don't test the failure mode they claim to, and somewhere underneath it all is a foundational gap nobody has circled back to fix. Before you propose anything, go find the people who've been closest to the implementation, reconstruct whatever documentation exists (or rebuild it yourself), and verify that your monitoring actually detects the thing you think it detects. Everything else is easier once that's solid.

Unblocking and delivering a stalled, cross-team initiative like this is exactly the kind of Architecture Review and Strategic Advisory work I can help with at Cloud with Gus.