Flaky Tests: Why End-to-End Suites Break and How to Fix Them

A flaky test is one that produces a different result — pass or fail — without anything in the code changing. Run it ten times and it might fail twice for reasons that have nothing to do with the behavior it is supposed to check. Every engineering team that runs an end-to-end suite eventually meets flakiness, and most underestimate how expensive it is.

The direct cost is obvious: reruns, wasted CI minutes, blocked merges. The indirect cost is worse. Once a suite is flaky, people stop believing red builds. They re-run the job "to see if it passes this time," and the moment that becomes a habit, the suite has stopped protecting you. A test that cries wolf is not a safety net — it is noise that happens to be colored red.

This article is about the real, mechanical causes of flaky end-to-end tests, how to diagnose them, and how to stop treating "just re-run it" as a strategy.

What flakiness actually is

It helps to define the failure precisely. A test has three possible outcomes, not two:

• True pass — the behavior works and the test confirms it.
• True fail — the behavior is broken and the test catches it.
• False result — the test outcome disagrees with reality: a false fail (behavior works, test fails) or, more dangerously, a false pass (behavior is broken, test passes anyway).

Flakiness is the presence of false results that vary run-to-run. The variance is the tell. A test that always fails is broken but honest — you can fix it. A test that fails 15% of the time is lying intermittently, and intermittent lies are much harder to trust, triage, or delete.

The real causes, in order of how often they bite

1. Waiting on time instead of state

By far the most common cause. A test does something, then waits a fixed number of milliseconds, then asserts. On a fast local machine the wait is generous; on a loaded CI runner the app hasn't finished rendering yet, and the assertion fires into an empty DOM.

// Flaky: hard-coded sleep. Fast enough locally, too short under CI load.
await page.click("#submit");
await page.waitForTimeout(500);
await expect(page.locator(".confirmation")).toBeVisible();

// Stable: wait for the actual condition, with a bounded timeout.
await page.click("#submit");
await expect(page.locator(".confirmation")).toBeVisible({ timeout: 10_000 });

The fix is a principle, not a snippet: never wait for time, wait for state. Wait for the element, the network response, the URL change, the disabled button to re-enable. Modern frameworks like Playwright auto-wait on assertions for exactly this reason — most waitForTimeout calls in a suite are flakiness waiting to happen.

2. Tests that share state

When tests are not isolated, order and concurrency start to matter. Test A creates a user named test@example.com; test B assumes that user doesn't exist yet; run them in parallel and one loses. The suite passes when they happen to run in a lucky order and fails when they don't.

The fix is isolation: each test sets up its own data with unique identifiers, and tears it down (or runs against a per-test transaction or a fresh seeded state). If two tests can't run in either order and still pass, one of them is holding a hidden dependency.

3. Animations, transitions, and focus

An element can be present in the DOM but not yet clickable — mid-fade, sliding in, or covered by a modal backdrop that is animating out. The click lands on nothing, or on the wrong element. Disabling animations in the test environment (prefers-reduced-motion, or a test-only CSS flag) removes a whole category of timing races.

4. Selector churn

This one is not strictly flakiness — it's fragility — but it produces the same symptom: tests that break for reasons unrelated to behavior. A test keyed to .btn-primary.mt-4 breaks when a designer changes a margin utility class. The behavior is fine; the selector is stale.

The fix is to select by meaning, not by markup: prefer roles and accessible names (getByRole("button", { name: "Checkout" })) or explicit data-testid attributes that the team commits to keeping stable. Never bind a test to styling classes or deep DOM structure that changes on a whim.

5. The environment underneath

Real third-party calls, real time, real randomness, real clocks. A test that hits a live payment sandbox will flake when that sandbox is slow. A test asserting "created just now" flakes across a midnight or timezone boundary. A test relying on Math.random() output flakes by definition.

Control what you can: mock external dependencies at the network boundary, freeze the clock, seed randomness. Reserve real integrations for a small, separate suite you expect to be slower and treat differently.

How to diagnose flakiness without guessing

You cannot fix what you cannot see. The teams that beat flakiness make it measurable:

• Quarantine, don't ignore. When a test is known-flaky, move it to a quarantine group that still runs and reports but doesn't block merges. This keeps the signal without holding the pipeline hostage — and, crucially, keeps the flaky test visible instead of silently .skip-ped forever.
• Track a flake rate per test. Record pass/fail history across runs. A test that fails 8% of the time is a data point, not a mystery. Ranking by flake rate tells you exactly where to spend repair effort.
• Re-run failures once, and count it. A single automatic retry is a reasonable safety valve, but a retry that silently turns red into green is how flakiness hides. If a test only passes on retry, that is a recorded flake, not a pass.
• Reproduce under load. Most flakiness only appears under CI-like conditions: parallelism, constrained CPU, cold caches. Run the suspect test 50 times in parallel on a throttled runner before you declare it fixed.

Where "just re-run it" comes from — and why it fails

Retries are seductive because they work often enough to feel like a fix. The problem is that a blanket retry policy treats every failure as noise. Sooner or later a real regression fails once, gets retried, passes on a fluke of timing, and ships. The retry that was protecting your pipeline from flaky tests just laundered a genuine bug into production.

The honest position is that retries are a triage tool, not a cure. They buy you time to fix the underlying cause. A suite whose green status depends on retries is not green — it is amber with the warning light unplugged.

The deeper fix: validate failures instead of trusting them

Everything above reduces flakiness. But at AI-generation speed, when UIs churn constantly and suites grow faster than any team can hand-maintain, a second layer becomes necessary: automatic validation of every failure before it reaches a human.

The idea is straightforward. When a test fails, don't immediately alert. Reproduce the failure in a clean run. If it reproduces consistently, it is a real regression — surface it with the root cause and the change that caused it. If it doesn't reproduce, it is a flake — record it, rank it, and don't wake anyone up. This turns the three-outcome problem into a decision the pipeline makes itself, instead of a judgment call an on-call engineer makes at 2 a.m.

This is exactly the shift from reporting failures to confirming them — and it is the difference between a monitoring system that generates alert fatigue and one that generates trust.

Where BuniOD fits

Reducing flakiness by hand — auditing waits, enforcing isolation, stabilizing selectors — is real, ongoing work, and it doesn't scale on its own when the application changes faster than the tests do.

BuniOD approaches the problem from both ends. Because coverage is generated from the application's actual flows and expressed as business intent rather than hard-coded selectors, tests survive UI churn instead of breaking on a renamed class — removing an entire category of fragility at the source. And every result is validated before it is reported: a suspected failure is reproduced and confirmed as a real regression, or dismissed as noise, so what reaches your team is signal rather than a red build to re-run. You can read about the security model or how it fits QA teams.

Conclusion

Flaky tests are not bad luck. They are the predictable result of waiting on time instead of state, sharing data between tests, ignoring animations, binding to fragile selectors, and trusting an uncontrolled environment. Each cause has a concrete fix, and the discipline of measuring flake rates turns a vague frustration into a ranked, solvable backlog.

The teams that keep their pipelines fast and their releases safe are the ones that refuse to normalize the amber light — that treat every false result as a defect in the test system itself. In an era where code, and the tests around it, are increasingly machine-generated, the final safeguard is a layer that validates failures rather than assuming them. That is what turns a suite back into what it was always supposed to be: a reason to ship with confidence.