Test Data Is Lying To You – Why Environments Drift

I have lost more time to test data than to any tool, framework, or flaky assertion combined.

The bug that I cannot reproduce because my account has different permissions. The test that passes locally and fails in CI because the seed data is different. The half-completed environment refresh from three months ago that left some tables current and others stale. The PII that turned up in staging because someone copied prod down “just for a day.”

Test data is a quiet killer. It doesn’t crash anything per se, and it doesn’t show up on a dashboard. It just slowly degrades trust the team has in the test cases and environments they’re testing against, until all that trust erodes leaving only production.

Test data management aint glamorous, it gets zero budget, and it almost never gets any sprint allocation. Which is why most teams have a test data problem they have learned to put up with rather than solve.

Why Environments Drift

Every test environment starts clean. Then real life happens.

Someone runs a test that updates a record and it never get reset. Someone manually fixes a fixture to reproduce a bug and then nobody returns that fixture to its original state. Someone copies a snapshot of prod data down because they are debugging a customer issue, but it’s never anonymised. Someone changes some configuration in the settings, but doesn’t revert it back. Someone schedules a refresh job that runs once successfully and then starts to fail silently for the next six months.

We’ve all been there, that’s how environments drift. Not with a bang, but in a hundred small whimpering modifications, none documented, and all of which add up to “staging doesn’t work like it used to anymore.”

The effort involved to update environments is likely managed by other team members, the Devops team for example. They have other deadlines and other issues to manage, so test data concerns usually get pushed out, for week or months, sometimes forever.

I have seen teams have weeks-long arguments about why a test is flaky, where the answer was always going to be “the data is different in CI than it is on your machine.” Nobody owned or allowed time for test data maintenance, so nobody fixed it. Even the best laid plans to finally deal with it, by the business, are trumped due to more “urgent” work.

The Two-Axis Problem

Test data is hard because it has to satisfy two competing requirements.

It has to be realistic enough to test the system properly. Edge cases need to exist, the volume needs to look like prod, and the relationships between tables needs to make sense.

It also has to be clean. No PII, no real customer data, and no real payment details. Nothing that, if leaked or accidentally committed, would put the business in front of a regulator.

Most teams pick one and live with the consequences of the other, mostly due to giving up. Either staging looks nothing like prod, and bugs that would obviously surface in real conditions never get found. Or staging looks too much like prod, and there is real customer data sitting in an environment with weaker controls than production.

Synthetic data is the answer to both. But, synthetic data is also more work upfront than just copying production down.

What Good Looks Like

A good test data setup is not complicated. It’s just maintained, by doing a few good things.

Test data lives in code, not in someone’s head. Whether that is SQL seed scripts, Faker-driven generators, or fixtures committed to the repo, the data definition is versioned alongside the application. When the schema changes, the test data changes. When a test needs a new shape, that shape gets added in code, not by manually editing the database.

Refreshes are scheduled and observed. The job runs on a cadence the team trusts, and somebody is responsible for noticing when it fails. If the refresh broke six weeks ago, that is a problem that should be visible to the team, not buried in the logs noone looks at.

Anonymisation is the default for any prod-derived data. If a snapshot has to come down from prod to debug something, it gets sanitised before it lands in a non-prod environment. There are tools that can help with this, nothing new to see here.

Test data is owned and maintained. Not by everyone, as that usually means nobody. By a specific person or small group who can answer the question “is this data current and trustworthy” on any given day.

None of this is rocket science. It is just discipline that most teams trade away because there is always something more visibly urgent or impactful driving delivery needs.

Synthetic Data, Properly

Synthetic data is the right answer for almost every case. It just needs to be done well.

Things fail when someone writes a Faker script that generates 10,000 rows of plausible-looking nonsense, none of which actually exercise the system. Names that don’t match locales, addresses that don’t validate, and order histories with no relationship to the customer who placed them. All statistically valid, but all semantically broken.

What works best is built the same way the application builds it. If the system creates a customer through a sign-up flow and an order through a checkout flow, the synthetic data generator should produce data that could plausibly have come through those flows. That means it has to know about the relationships, the constraints, and the business rules. That is more work than Faker, but it is also data you can actually test against and trust.

For systems where the relationships are simple, Faker plus a bit of glue works fine. For systems with complex domain rules, you probably need a small in-house generator, or one of the synthetic data products that has emerged in the last few years. Either way, the principle is the same. Generate data that looks like the data your application would produce, not data that misses the intended use.

Where AI Fits

AI has shifted the test data conversation in a few ways.

It’s now genuinely fast to prototype a synthetic data generator. Describe your schema, describe the constraints, describe the kinds of edge cases you care about, and you can have a working generator in an afternoon. It might not be perfect, but it’ll be a working version you can start with. That has changed what is reasonable to expect from a small team.

It’s also pretty quick to anonymise production data. Custom anonymisation scripts that used to take a sprint to write properly can be sketched out in an hour with Claude, or preferably your own local AI. Names get replaced with realistic alternatives, email addresses get domain-swapped, and phone numbers get regenerated to valid formats. The end result is PII gets stripped or shuffled to be mostly unrecognisable. Done well, you can have a sanitised snapshot of prod that is safe to use in staging.

The risk is the same as anywhere else with AI. The tooling is plausible-looking and may not actually be correct. A synthetic data generator that produces output that “looks fine” but quietly misses is worse than no generator, because the team trusts the data and the data is just crap. AI accelerates the building of these systems, but it cannot replace the verification. Structure your input planning to cover things properly, do rush it and don’t skip parts, it’ll just create more work and distrust later.

A quick word on security and privacy of this kind of generated test data. Use local AI for this task 100%, or you risk production data being captured in the perpetual AI crap in/crap out cycle.

The Real Cost of Bad Test Data

Bad test data is invisible until it bites. Then it bites in a few ways.

Flaky tests get blamed on the framework when they are actually caused by data drift. CI builds become unreliable because the seed differs subtly from local. Engineers stop trusting test failures, so when the tests are fine and the data is not, trust erodes anyway.

Bugs that should have been caught in QA make it to production because staging does not look like prod. The volume is wrong, the mix of customer types is wrong, configurations do not match, and the edge cases that exist in prod do not exist in staging. The QA team passes everything they were given to test, yet a bug shows up the day after release, leaving everyone a little perplexed.

Over time, everyone becomes numb to it and these things multiply. The team just gets used to it, fatigued is likely the better term. “Yeah, we just retry that test.” “Yeah, we know staging is a bit weird.” “Yeah, we test that one in prod direct instead.” That just aint test data management, it’s “silent quitting”, or “test data resignation”, and it aint QA.

Treating Test Data like Code

The reframe that fixes most of this is somewhat simple. Treat test data like code, give it time and resources every sprint.

If your test data is in a database somewhere with no version control, that is the problem. Move the definition into the repo. Commit the seed scripts, commit the generators, and commit the anonymisation logic. Whatever exists outside of the repo today, get it into code, treat it like “test data as code”.

Once test data is in code, all the discipline you already apply to code applies to it by default. Pull request reviews, versioning, branches, and rollbacks. Test data drift becomes something the team can see in diffs, which is usually the only kind of drift teams actually catch. It’s also likely the easiest place to catch it as it’s right there in front of you, clear as day.

This is one of those things that looks like overhead until you do it a few times and then realise most of the “weird, why is this happening” issues just stop.

It’s not easy and may require a culture change of some kind, but I know from experience if nothing is done at all, everyone deals with the growing impact of it, not just QA.

The Take

Test data management is the silent killer of QA (even Dev, Product, or Support) productivity. It never gets budget, its never added to a sprint, and it never feels urgent until it is already a problem you have learned to live with and/or workaround. Worse it becomes a blocker.

The fix is not a new magic tool. Synthetic data that respects your domain. Refreshes that run and are observed with anonymisation by default. Test data just needs attention, it lives in code, owned by someone, and versioned alongside the application. Applied correctly, AI helps with this, but like with everything, it still needs verification.

The team that gets test data right stops having weeks-long arguments about flaky tests. Stops wasting hours fixing the wrong thing, potentially creating other issues. The team that does not gets used to staging being unreliable, then stops trusting any environment that is not prod.

Build it in to your work, don’t put yourself in a position where you “deal with it later.” That’s a mugs game and will bite you quicker than you’d expect. I’ll put money it.

A Note on Context

Every business and every project is different. What works in one place won’t work in another, and that’s the point.

Nothing here is meant to be a step-by-step prescription. It’s guidance, drawn from my own experiences, and deliberately kept general to avoid pointing fingers anywhere.

Take what’s useful, ignore what isn’t, and adapt it to your own context. Or as Joe Colantonio always says: “Test everything and keep the good.”