Infrastructure as Code: Where Teams Get It Wrong

Most teams adopt infrastructure as code for good reasons. They want environments that match, changes that can be reviewed, and a way to rebuild things that doesn't depend on one engineer's memory. And most teams get some of that. A year later, though, many of them find the code no longer matches what's running, nobody wants to touch the shared modules, and changes still go through the one person who understands it.

This guide covers the eight mistakes we see most often, and what to do instead. It applies whichever tool you use. These are infrastructure as code best practices learned from the problems, not from the documentation.

Why infrastructure as code goes wrong

Infrastructure as code means defining servers, networks, databases and permissions in files that are versioned, reviewed and applied by a tool, instead of creating them by hand in a cloud console. The tool then makes the real infrastructure match the files.

The idea is simple. What makes it hard is that infrastructure as code changes how a team works, not just what tool it uses. Every mistake below is a case where the tooling was adopted but the working habits weren't.

Mistake 1: Treating it as a one-time project

A team spends a quarter writing code for their infrastructure, celebrates, and moves on. Three months later there's an incident at 2 a.m. Someone opens the cloud console, increases a database's size and adds a firewall rule, and the incident ends. Nobody updates the code. The next time the code is applied, it quietly reverts both changes, and the incident comes back.

The cause isn't carelessness. It's that the team never agreed what happens in an emergency. When the only official path is a pull request with review, and the system is down, people will take the faster path every time.

What to do instead: define a "break-glass" process before you need it.

• Emergency console access exists, but it's time-limited, logged and granted to on-call engineers only.
• Every emergency change gets a follow-up pull request within one working day, either codifying the change or reverting it.
• Incident reviews check for it. "Did we change anything outside the code?" becomes a standard question in every post-incident review.

The goal isn't to forbid console changes. It's to make sure every one of them ends up back in the code.

Mistake 2: Codifying the mess

When existing infrastructure is brought under code, the easiest route is to describe exactly what's there. That's often the wrong goal. You end up with code that perfectly reproduces three naming conventions, untagged resources, a security group open to the internet "temporarily" since 2022, and a test database nobody can identify.

Adopting infrastructure as code is the cheapest moment you'll ever have to fix these, because you're already reading every resource. Use it to set a baseline:

Tagging deserves special attention. Required tags enforced in code are what make cloud costs traceable to a team or product, which is the starting point for any serious cost work. We cover that in why your cloud bill keeps growing.

Do this in stages, though. Don't combine "bring it under code" and "change it" in the same step for production systems. Codify first, confirm the tool proposes no changes, then improve in small, separate releases.

Mistake 3: Changes too big to review

Infrastructure as code promises that every change is reviewed. In practice, a pull request that touches forty resources produces a plan hundreds of lines long, and the reviewer scrolls to the bottom and approves it. That's not review. It's a ritual.

The fix is to make changes small enough that a reviewer can actually understand them, and to teach reviewers what to look for first:

• Deletions and replacements. Any resource the tool plans to destroy or recreate deserves the most attention, especially databases, storage and networks.
• Permission changes. New roles, broader policies or public access.
• Unexpected changes. Anything in the plan the pull request didn't mean to change usually points to drift or a version change.

A good habit is to post a short plan summary on every pull request, such as "3 to add, 1 to change, 0 to destroy," so a destroy can't hide in a long output. If a change needs more than a screen of plan output to explain, split it.

Mistake 4: Building a module platform too early

Reusable modules are one of the best features of infrastructure as code. They're also where teams over-engineer most. A team with one application builds a module for a network, a module for a service that wraps the network module, and a configurable "platform" module with forty inputs that wraps both. Six months later, nobody can change anything without understanding three layers of abstraction.

A practical rule is the rule of three: write something directly the first time, copy it the second time, and turn it into a module the third time, when you actually know what varies between uses.

• Keep modules small and focused. A module should do one recognizable thing, such as "a database with backups and monitoring."
• Prefer fewer inputs. Every option is something to test and document. Make the secure, sensible choice the default instead of a setting.
• Version shared modules. Once more than one team uses a module, changes to it should be released and adopted deliberately, not picked up automatically.

Readable, slightly repetitive code is easier to maintain than clever code that nobody wants to touch.

Mistake 5: Testing with the plan and nothing else

For many teams, "testing" infrastructure code means running a plan and checking that it doesn't error. A plan tells you what will change. It doesn't tell you whether the change is secure, affordable or actually works.

Infrastructure code can be tested in layers, from cheap and fast to slow and thorough:

The last layer used to need extra tooling. Since Terraform 1.6, released in October 2023, Terraform has included a native test framework that provisions the configuration defined in a test file, checks your assertions and automatically destroys the infrastructure at the end. Other tools offer similar options. For shared modules, this is how you find out a change breaks something before every team that uses the module does.

Mistake 6: Forgetting that data isn't code

Infrastructure as code makes servers disposable. That's the point. But databases, storage buckets and their contents aren't disposable, and the same code that can rebuild a server in minutes can delete a database in seconds. A renamed resource, a refactor or a removed block can all lead a tool to plan a destroy and recreate.

Protect data in more than one place:

• Turn on the cloud provider's deletion protection. On AWS, for example, an RDS database with deletion protection enabled cannot be deleted by any user until the setting is turned off. Azure resource locks and Google Cloud deletion protection settings serve the same purpose.
• Add protection in the code as well. In Terraform, prevent_destroy makes Terraform reject any plan that would destroy the resource. But HashiCorp's documentation notes the protection is removed along with the resource block, so it can't be the only safeguard.
• Make "destroy" on stateful resources fail review automatically. A policy check that blocks any plan destroying a database or bucket, unless explicitly approved, catches the mistake before a person has to.
• Test restores, not just backups. A backup you've never restored is an assumption. Rebuilding an environment from code is only useful if the data can follow.

Mistake 7: One person owns all of it

Many teams replace "only Sam knows how the servers are set up" with "only Sam understands the Terraform." The knowledge moved from the console to a repository, but it's still in one head. When that person is out, changes wait or get made in the console, and Mistake 1 begins again.

• Write down conventions. A short document covering naming, tagging, module structure and how to make an emergency change is enough to start.
• Assign code owners for each part of the infrastructure, so reviews go to the right people automatically and no single reviewer is a bottleneck.
• Rotate who makes changes. Pair a second engineer on the next few infrastructure changes. Understanding spreads faster by doing than by reading.
• Keep a runbook next to the code for the tasks that come up under pressure, such as scaling a database, rotating a credential or restoring from backup.

Mistake 8: Applying from laptops

If infrastructure changes are applied from engineers' own machines, you can't be sure which version of the code was applied, with which tool version, or by whom. Two people can apply conflicting changes the same afternoon.

The fix is the same discipline teams apply to application code: changes are proposed in a pull request, the plan runs automatically and is reviewed, and the apply runs from a pipeline with its own short-lived credentials. Engineers keep read access for investigation, and write access goes to the pipeline.

This is the setup we built in our cloud infrastructure migration case study, where the entire target environment was defined in Terraform. Provisioning a new environment now takes minutes instead of the week-long process the client had before.

Where does your team stand?

Most teams sit somewhere on this path. Find your stage, and focus on the next step rather than the last one.

Skipping stages rarely works. A team at stage 1 that tries to build stage 4 tooling usually ends up with impressive modules and a production environment still changed by hand.

Frequently asked questions

What are the most important infrastructure as code best practices for a small team?

Start with four: keep state remote and locked, make every change through a reviewed pull request, apply only from a pipeline, and agree how emergency changes get back into the code. Add required tags and secure defaults while you codify. Automated security and policy checks come next, once the basics are habits rather than rules.

How do we handle emergency changes made in the cloud console?

Allow them, but make them traceable. Give on-call engineers time-limited, logged console access for incidents, and require a follow-up pull request within one working day that either codifies the change or reverts it. Ask in every incident review whether anything changed outside the code. That keeps emergency fixes from silently disappearing on the next apply.

Should we import existing infrastructure as-is or clean it up first?

Import it as-is first, then clean it up in separate steps. Bringing a resource under code and changing it at the same time makes problems hard to diagnose. Once the code matches what's running and the tool proposes no changes, fix naming, add required tags, tighten security defaults and remove unexplained resources in small releases.

When should we start writing reusable modules?

When you've built the same thing about three times and know what actually varies between uses. Modules written earlier tend to guess wrong and grow dozens of options. Keep modules small, give them secure defaults instead of extra settings, and version them once more than one team depends on them.

How do you test infrastructure as code?

In layers. Format and validation checks run on every commit. Static security scanning, policy checks and cost estimates run on every pull request. A person reviews the plan, looking first at deletions and permission changes. For shared modules, test frameworks such as Terraform's native one provision real infrastructure, check assertions and destroy it afterward.

How do we stop infrastructure as code from deleting a production database?

Use several safeguards, because each one can fail. Turn on the cloud provider's deletion protection for databases and storage. Add code-level protection such as Terraform's prevent_destroy, knowing it disappears if the resource block is removed. Add a policy check that blocks plans destroying stateful resources without explicit approval, and test restores from backup regularly.