Regression Testing: What It Is, When to Run It, and How to Keep It Useful

Regression testing is the least glamorous part of quality work and the part that most reliably decides whether a release is safe. Nobody is promoted for it. It gets cut first when a deadline tightens. And the moment it stops working, teams discover that most of their confidence was coming from it.

It is also the area where the gap between theory and practice is widest. The theory is straightforward: after a change, check that things which used to work still work. The practice, in most organisations, is a suite that takes forty minutes, fails intermittently for reasons nobody has investigated, and gets re-run until it goes green.

This guide covers what regression testing actually is, how it differs from adjacent practices that share its vocabulary, how to decide what belongs in the suite, and how to keep it from decaying into ceremony.

What regression testing is

Regression testing re-verifies previously working behaviour after a change. The change can be anything: a new feature, a bug fix, a refactor, a dependency upgrade, an infrastructure or configuration edit. What makes it a regression concern is not the size of the change but the fact that software has non-local effects — a shared component, a database migration, a modified default, a library upgrade that alters date parsing.

The name comes from the direction of the failure. The product does not fail to advance; it regresses to a state where something it could previously do, it now cannot.

Two properties follow, and both are the source of most regression pain:

• The relevant tests are mostly unrelated to the change. You are checking the blast radius, not the diff. That is why regression suites are broad and why "just test what changed" is harder advice than it sounds.
• The tests must already exist. Regression testing depends entirely on having encoded previous behaviour somewhere. Behaviour nobody wrote a test for cannot regress detectably; it simply breaks.

What it is not

The vocabulary in this area overlaps enough that teams routinely talk past each other. Three distinctions worth keeping straight:

Practice	Question it answers	Scope	Typical trigger
Retesting	Is this specific defect fixed?	One defect, using its reproduction steps	A fix lands
Regression testing	Did this change break anything that worked?	Previously working behaviour, broadly	Any change
Smoke testing	Is this build worth testing at all?	A thin slice of critical paths	A build is produced
Sanity testing	Does this narrow area behave after a targeted change?	One feature area	A focused change

The most consequential confusion is between retesting and regression. A team that only retests has verified that the reported bug is gone and learned nothing about what the fix cost elsewhere — which is precisely how fixes ship that break two other things.

Types of regression testing

The literature lists several strategies; in practice teams use a blend, and the useful distinction is how much of the suite runs.

Full regression. Everything runs. Correct but expensive, and only viable if the suite is fast. Appropriate before a major release, after a dependency upgrade with wide reach, or after infrastructure changes where the blast radius is genuinely unknown.

Selective regression. A subset chosen by relationship to the change — same module, same dependency graph, same data model. Faster, and correct only insofar as your model of the dependencies is correct. This is where most misses come from: the coupling nobody knew about.

Risk-based regression. A subset chosen by business consequence rather than by code proximity. Checkout runs on every change regardless of what changed, because the cost of it silently breaking dominates the cost of running it. This is usually the most defensible default; Risk-Based Testing sets out how to rank.

Progressive regression. New tests written for changed behaviour, replacing the old tests that encoded the behaviour deliberately superseded. This is the maintenance half of regression that teams forget, and skipping it is how suites end up asserting rules the product abandoned two years ago.

In practice: run a fast, risk-selected subset on every pull request, and the full suite on every merge to the mainline. Reserve genuinely full runs for changes whose reach you cannot bound.

Choosing what goes in the suite

A regression suite is not "all our tests". It is a deliberately maintained set, and the selection criteria matter more than the size.

Include behaviour whose failure has a consequence someone would escalate. Money paths, authentication, permissions, data integrity, anything with legal or contractual weight.

Include everything that has broken before. Defect history is the single best predictor of future defects in the same area. A regression test written after an incident carries evidence, which makes it easy to defend when someone proposes trimming the suite.

Include the paths users actually take, which are frequently not the ones the team assumes. Critical User Journeys: How to Find and Protect Them covers how to establish these from real data.

Exclude tests that cannot fail independently. If a test can only fail in a way another test already catches, it is cost without coverage. This is the most under-used exclusion criterion in the industry.

Exclude assertions on presentation detail. Copy, spacing and element ordering change constantly and almost never carry business risk. Tests that assert them generate failures that are always dismissed, which trains the team to dismiss failures.

When to run it

The honest answer is: on every change, automatically, with results in the pull request. Anything slower means defects are found further from the change that caused them, and the cost of a defect rises steeply with that distance — a point examined in The Real Cost of a Bug in Production.

Practical placement:

• On every pull request: the fast risk-selected subset. Target a few minutes. If it takes longer, developers start merging without waiting for it, and the gate becomes decorative.
• On every merge to mainline: the full suite. Failures here still precede release.
• Before release: nothing new, ideally. If your release process requires a distinct regression phase, that is usually a sign the earlier gates are not trusted.
• After deploy: a smoke subset against production, plus monitoring. Some classes of failure only appear against real infrastructure and real data — see Continuous Testing vs Continuous Monitoring.

Why regression suites decay

Nearly every long-lived suite develops the same three problems, and they compound.

Monotonic growth. Tests are added with every feature and removed almost never, because removal requires someone to argue that a test is not needed and nobody wants to be wrong about that. The suite gets slower every quarter until it no longer fits in the feedback loop it was built for.

Flakiness. Timing assumptions, shared mutable data, hardcoded dates, third-party dependencies. Individually each is a small problem; collectively they mean the suite fails for reasons unrelated to the code. Once a team learns that red sometimes means nothing, red always means nothing. Flaky Tests: Why E2E Suites Break covers the mechanics and the fixes.

Maintenance debt. A redesign breaks two hundred selectors. The repair is tedious and produces no visible value, so it is deferred, and the suite is quarantined "temporarily". Temporarily is doing heavy lifting in that sentence.

The common root is that traditional regression tests are coupled to implementation detail while the behaviour they encode is stable. The intent — "a returning customer can check out with a saved card" — has not changed in five years. The two hundred lines expressing it have changed constantly.

Keeping the suite honest

Four habits that keep regression working:

1. Budget the runtime, not the test count. Decide the suite must finish in, say, ten minutes, and treat exceeding that as a defect. A budget forces the pruning conversation that otherwise never happens.
2. Quarantine flaky tests immediately, and fix or delete them within a fixed window. A flaky test left in the suite costs more than the coverage it provides.
3. Delete superseded tests as part of the change that supersedes them. Progressive regression is a step in the definition of done, not a cleanup project.
4. Review failures for signal, not just for green. Track how often a failure was a real defect. That ratio, not test count or code coverage, tells you whether the suite is working — a theme we develop in Software Quality Metrics That Actually Predict Failure.

How AI changes regression testing

Two of the three decay mechanisms above are maintenance problems rather than testing problems, and maintenance is what generative systems are best at removing.

When a test is derived from a plain-language description of intent rather than hand-written against selectors, a redesign does not break two hundred tests — it changes the interface the description resolves against, and the steps are regenerated. The durable artifact becomes the sentence "a returning customer checks out with a saved card", which was always the part that was stable. This does not eliminate flakiness caused by shared data or genuine timing issues, and it does not decide what belongs in the suite. But it removes the repair backlog that quarantines suites, and it makes deletion cheap, which is what allows a suite to stay small enough to run on every change.

The selection judgement stays human. Knowing that seat-limit handling is where your billing disputes originate, or that a particular integration has failed twice this year, is not something a model can infer from your codebase. The End of Manual Regression Testing looks at how the role shifts as a result.

Where BuniOD fits

BuniOD approaches regression from the intent side. You describe the flows that must keep working in plain language — one sentence each — and it generates the end-to-end scenarios, with the steps, test data and assertions. Those scenarios run on every release rather than on a schedule, adapt when the interface changes rather than breaking, and reproduce suspected failures before anyone is alerted, so a red result means something.

What you maintain is the list of sentences: the honest, readable record of what your product must never stop doing. Adding to it costs a sentence, and pruning it costs a deletion — which is what makes a regression suite sustainable over years.

Conclusion

Regression testing is re-verifying that what worked still works, and it is distinct from retesting a fix or smoke-testing a build. A good suite is selected by business risk and defect history rather than by code proximity, runs automatically on every change rather than on a calendar, and is pruned as deliberately as it is grown.

Its characteristic failure is not missing coverage but lost trust: a suite that is slow, flaky and ignored provides no protection no matter how many tests it holds. Watch the runtime budget, the flake rate and the proportion of failures that turn out to be real defects — those three numbers will tell you whether your regression testing is working long before an incident does.

Want a regression suite that maintains itself? Request access and describe the flows that must never break.