Table Of Contents
In the last article, I talked about reporting – who needs what, and why one report rarely serves everyone. But there’s a question that sits underneath all of that: what should you actually be measuring in the first place?
This is where a lot of teams get stuck. They know they need metrics. They know leadership wants dashboards. So they start measuring everything they can, hoping something useful emerges.
It rarely does. What usually emerges is noise.
The Trap: Measuring Everything Because You Can
Modern test management tools make it easy to track almost anything. Test cases executed. Defects logged. Pass rates. Fail rates. Time to fix. Time to retest. Coverage percentages. The list goes on.
The temptation is to capture all of it. More data is better, right?
Not really. More data just means more data. It doesn’t automatically mean more insight.
When you measure everything, you end up with dashboards full of numbers that no one quite knows how to interpret. People start asking “what does this mean?” and the answer is often “I’m not sure, but it looked important.”
That’s not measurement. That’s decoration. Is there any value in that?
Vanity Metrics vs Actionable Metrics
There’s a difference between metrics that look good and metrics that drive action.
Vanity metrics are the ones that sound impressive but don’t tell you much. “We executed 500 test cases this sprint.” Okay, cool, but were they the right ones? Did they find anything? Did they give you confidence?
Actionable metrics are the ones that answer real questions and point to something you can do. “Our defect escape rate increased this release, we’re finding more issues in production than in testing.” That tells you something. That prompts a conversation about coverage, about environments, about where your process might be leaking.
The question to ask about any metric: if this number changed, would we do something differently? If the answer is no, it’s probably not worth tracking.
What Makes a Good QA Metric
Here are some metrics I’ve found genuinely useful over the years. Not all of them will suit every team, but they’re a good starting point.
Defect escape rate. How many defects are found in production versus found in testing? This is one of the clearest indicators of whether your testing is actually catching things. A rising escape rate is a red flag worth investigating.
Defect age. How long do defects sit open before they’re resolved? Long-lived defects often signal prioritisation problems, unclear ownership, or issues that are harder than they first appeared. Tracking this helps surface blockers.
Test coverage trends. Not just “how much is covered” but “how is coverage changing over time?” Are you keeping up with new features? Are there areas that are consistently under-tested?
Cycle time. How long does it take from code commit to tested and ready? This isn’t purely a QA metric, but QA contributes to it. If cycle time is growing, it’s worth asking whether testing is a bottleneck, and if so, why.
Flakiness rate. What percentage of your automated tests fail intermittently for reasons unrelated to real defects? Flaky tests erode trust in your suite. Tracking this keeps the problem visible and creates pressure to fix it.
Defect reopen rate. How often do “fixed” defects come back? A high reopen rate might indicate unclear acceptance criteria, inadequate fix verification, or communication gaps between QA and development, or even project owner/management and the client.
Metrics that Can Mislead
Some metrics sound useful but can actually point you in the wrong direction if you’re not careful.
Raw test case counts. “We have 2,000 test cases” tells you nothing about quality. Are they maintained? Are they relevant? Do they cover the right risks? Volume isn’t value.
Pass rate without context. A 98% pass rate sounds great — until you realise the 2% that failed were critical paths, or that the suite hasn’t been updated in six months and isn’t testing anything new.
Defects logged (quantity). Counting how many defects a tester logs can incentivise the wrong behaviour. You get testers logging trivial issues to hit a number, rather than focusing on what matters. Quality of findings beats quantity every time.
Test execution velocity. How fast you run tests isn’t inherently meaningful. Fast execution of the wrong tests isn’t better than slower execution of the right ones.
The common thread: these metrics measure activity, not outcomes. They’re easy to game and easy to misinterpret.
The Foundation: Clear Work Items
Here’s something that often gets overlooked. Your metrics are only as good as the data feeding them. And that data depends on how well your work items are defined.
If your user stories are vague, your test cases will be vague. If your defects aren’t categorised consistently, your defect metrics will be unreliable. If “done” means different things to different people, your cycle time numbers won’t mean much.
Before you worry too much about which metrics to track, make sure the basics are in place:
- Stories and tasks are clearly defined with acceptance criteria
- Defects are logged consistently with severity, priority, and component
- Test cases are linked to requirements or stories, where possible
- Your teams have a shared understanding of what “done” means
This isn’t glamorous work, but it’s the foundation. Without it, you’re building castles on sand, next to an incoming ocean tide.
Metrics for the Team Vs Metrics for Leadership
Just like reporting, metrics serve different audiences.
Team metrics should be granular and actionable. Flakiness rate, defect age, reopen rate — these help the team improve day to day. They belong in sprint retros and team standups.
Leadership metrics should be summary-level and outcome-focused. Defect escape rate, overall cycle time, release confidence, as these tell the bigger story. They belong in steering committees and stakeholder updates.
There’s overlap, of course. Defect escape rate matters to everyone. But the level of detail and the frequency of review will differ.
The mistake is showing leadership the team-level metrics and expecting them to know what to do with them. Or giving the team only high-level numbers that don’t help them act.
Match the metric to the audience.
Start Small
If you’re just getting started with metrics, or if your current approach isn’t working, don’t try to fix everything all at once.
Pick three to five metrics that answer real questions your team or stakeholders are asking, or issues you can clearly see occurring. Make sure you can actually gather the data reliably. Then track them consistently for a few sprints and review them in retros.
Then iterate. Drop the ones that aren’t useful. Add new ones as new questions emerge.
Metrics aren’t set-and-forget; they’re a tool for learning. Treat them that way.
A Note on Context
Every business and every project is different. What works in one place won’t work in another, and that’s the point.
Nothing here is meant to be a step-by-step prescription. It’s guidance, drawn from my own experiences, and deliberately kept general to avoid pointing fingers anywhere.
Take what’s useful, ignore what isn’t, and adapt it to your own context. Or as Joe Colantonio always says: “Test everything and keep the good.”
