How I Cut Terraform Plan Time From 20 Minutes to 2 (a 90% Reduction)

The 20-minute plan

One Terraform workspace I worked in had grown to a little over 1,000 resources, spanning three environments (dev, staging, prod) and two AWS regions. It had grown that way gradually, over years: every new service, every new environment, every migrated legacy resource landed in the same state file, because that's where everything else already lived. By the time I got involved, terraform plan took about 20 minutes. Every time. Whether the change touched three resources or three hundred.

Twenty minutes doesn't sound catastrophic in isolation. But run it several times a day, multiply it across a team, and put it in front of every merge in CI/CD, and it becomes a real drag on how the team works. Engineers started batching changes to avoid running plan more than they had to. That's the opposite of the tight feedback loop Terraform is supposed to give you. Some stopped running plan locally at all and just pushed to CI, so at least the wait happened in the background. Reviews slowed down because a PR's plan output wasn't ready by the time anyone had time to look at it. None of this was really a Terraform problem. It was an architecture problem that happened to live inside Terraform.

Why size compounds

The root cause is straightforward once you see it: terraform plan has to build a full picture of the dependency graph and reconcile the state of every resource in it before it can tell you what's changing, even when your change is scoped to one small piece of that graph. There's no way to tell Terraform "just check this one subnet." It walks the whole graph, refreshes state for everything in it, and computes a diff against all of it, regardless of how small the actual delta is.

That's what was happening here: the workspace's size wasn't just a number in a state file. It was multiplying the cost of every single Terraform operation, no matter how targeted the actual change was.

There's a team-productivity cost hiding in that same mechanism, separate from raw plan time. A single state file means a single lock: only one apply can hold it at a time. When a thousand resources spanning three environments sit behind one lock, an engineer deploying a completely unrelated service can end up waiting on someone else's unrelated apply, or gets skittish about running one at all in case it collides. A large, shared state also discourages small, incremental changes: when every plan's blast radius is "this could touch anything in the account," the safe-feeling move is to batch several changes into one bigger, less frequent apply. That's the opposite of what makes infrastructure changes easy to reason about and easy to revert.

Measuring it before you fix it

You don't have to guess whether this is happening to you: two commands will tell you almost everything you need to know. terraform state list | wc -l gives you a resource count for the state you're standing in; if that number is creeping into the many hundreds, pay attention. And time terraform plan turns "plans feel slow" into an actual number you can track over time, instead of relying on gut feel or complaints in Slack. Run both before you touch anything, and again after, so the improvement is measured, not just felt.

The fix: scoped folders, scoped state

The fix, conceptually, was simple: stop keeping every resource in one basket. In practice, restructuring a workspace with this much history and this many interdependencies was significant enough that we didn't just start moving folders around. We wrote a design document first, laying out the target structure, the state boundaries, and the migration steps, and worked through it with a staff engineer before touching anything. Terraform state surgery on production infrastructure isn't something to improvise, and a second set of experienced eyes caught a couple of dependency edges we'd otherwise have discovered mid-migration.

Environment and region were the first cut, but they weren't the last one. A widely cited rule of thumb (Google's Terraform guidance, echoed across most large-scale write-ups on this) is to keep a single state well under a thousand resources, and ideally down in the low hundreds. Past that, plan and refresh time start climbing noticeably. Splitting only by environment and region still left individual states in the many-hundreds range, so we cut a second axis through each one: logical component. Things like core networking, datastores, serverless workloads, and application services each got their own folder and state within an environment/region, instead of sharing one.

# before
infra/
  main.tf
  variables.tf
  ...              # ~1,000 resources, one shared state

# after
infra/
  modules/
    networking/
    datastores/
    serverless/
    services/
    ...
  dev/
    us-east-1/
      core/
      datastores/
      serverless/
      services/
    us-west-2/
      ...
  staging/
    ...
  prod/
    ...

This split wasn't arbitrary. It followed how the infrastructure actually changed. Core networking and datastores are touched rarely and carry a lot of downstream risk if something goes wrong; application services change constantly as part of normal delivery. Grouping a frequently-changed service alongside a rarely-touched database was exactly the kind of pairing that made a small, routine change carry state and dependencies it had no business touching. Once core, datastores, serverless, and services each had their own state, a change to one service's configuration no longer had any reason to walk through the datastore or networking graph at all.

The part that took real care wasn't the splitting itself. It was avoiding the duplication a naive split invites. Copy-pasting the same resource definitions into every environment, region, and component folder just trades one big problem (slow plans) for several smaller ones (drifted, hand-maintained copies of the same configuration). We kept the actual logic in shared modules and used conditional logic plus per-environment/per-region/per-component variables in each folder to call those modules. The folders differ in which variables they set and which resources are toggled on, not in the underlying module code.

Deciding on the target structure was the easy part. Actually getting resources from one state into another needed just as much care, for two reasons. First, correctness: this has to happen with terraform state mv, not by deleting resources from one config and re-declaring them in another. Get that wrong and Terraform's next plan reads a "missing" resource as one to destroy and a "new" one to create, not a risk you take with a live database or load balancer. We moved resources state by state, verifying with terraform plan after each move that the destination showed zero diff before moving on, and sequenced the moves so nothing was ever absent from every state at once, keeping the whole migration zero-downtime.

Second, coordination. Once resources start moving between states, anyone else running plan or apply against the workspace mid-migration is effectively racing it: planning against a state that's half-moved, or applying on top of a state mv in progress. We handled this the low-tech way: a short Terraform code freeze, communicated to the team, covering the specific states being migrated. Nobody ran plan or apply against them until the migration was verified complete. A day or two of inconvenience is far cheaper than reconciling a state two people modified at the same time.

The result

Plan time on the largest of the new, scoped workspaces dropped from roughly 20 minutes to about 2 (a 90% reduction). That number alone would have justified the effort, but the more meaningful change was behavioral. CI/CD pipelines that used to have a 20-minute plan step blocking every merge sped up dramatically, so PRs could be reviewed and merged same-day instead of queuing behind a slow pipeline. Engineers started running plan locally again before pushing, since it no longer meant abandoning what they were doing for twenty minutes. And because a change to one component's folder no longer produced a diff touching resources everywhere, plan output became reviewable again: a reviewer could look at a short diff and trust it showed the real blast radius, not noise from an oversized dependency graph.

State lock contention between unrelated teams largely disappeared too. A networking change and an application service deploy no longer waited on the same lock, because they no longer lived in the same state. Incremental, small changes stopped feeling risky by default, which in turn made people more willing to ship them incrementally instead of batching everything into infrequent, larger applies.

There's a sharper version of this same benefit during an incident. A 20-minute plan is tolerable for routine work, but it's a very different problem when the fix for an active incident is an infrastructure change. Waiting 20 minutes for CI, or for a platform engineer, to safely plan and apply that change turns your MTTR (mean time to recover) into "the incident plus 20 minutes," every single time. With plans landing in 2 minutes instead of 20, the Terraform step stopped being the bottleneck in incident response.

Signs your Terraform setup has outgrown a single workspace

If your team's Terraform has grown organically for a few years, there's a decent chance you're heading toward the same wall, even if you're not there yet. A few things worth watching for:

• Plan times have been creeping up gradually, and nobody remembers the last time a plan took under a minute.
• A single state has grown past a few hundred resources. Well past that, and you're in the range where most large infrastructure teams report plan and refresh time becoming a real bottleneck.
• Frequently-changed resources (application services) and rarely-changed, high-blast-radius ones (core networking, databases) live in the same state, so routine changes carry risk they don't need to.
• Engineers occasionally hit a state lock error caused by someone else applying a completely unrelated part of the infrastructure, a sign unrelated work is contending for the same lock.
• Engineers avoid running terraform plan until they have to, or batch changes specifically to run it less often.
• A one-line change (a tag, a variable default) produces a plan diff touching dozens of unrelated resources.
• The Terraform plan/apply step has quietly become one of the slowest parts of your CI pipeline.

None of these are fatal on their own, but together they're a sign a single workspace has outgrown its boundaries the way the one above had. Restructuring an existing workspace safely, without losing state, breaking dependencies, or introducing drift, is exactly the kind of Infrastructure as Code and GitOps work I take on with clients at Cloud with Gus. If any of this sounds familiar, it's very fixable.