Flaky Tests Are Not Your Fault: How to Detect and Fix Them Automatically

There’s a specific kind of dread that comes from seeing a familiar test fail in CI. Not the good kind of failure, the one that caught a real bug. The other kind. The one where you already know, before you even open the logs, that if you just re-run the pipeline, it’ll probably pass.

I’ve watched entire teams normalize this. Someone pushes a PR, the build goes red, and the first response isn’t “let’s investigate”, its “re-run it, that test is flaky.” Everyone nods. Nobody’s wrong. And that’s exactly the problem: re-running a flaky test until it goes green isn’t a workaround, it’s a habit that quietly erodes the entire point of having tests.

If this sounds familiar, I want to say something clearly up front: this isn’t because your team writes bad tests. Flaky tests aren’t primarily a skill problem. They’re an architecture problem, the predictable result of testing dynamic, asynchronous, environment-dependent software with tools that assume everything happens in a fixed, predictable order. Good engineers on well-run teams produce flaky suites constantly, because the thing they’re testing was never actually deterministic to begin with.

This guide covers what test flakiness actually is, what causes it, how the current generation of flaky test detection tools (BuildPulse, Trunk, LambdaTest) approaches the problem, and, the part most guides stop short of, how to move from detecting and hiding flaky tests to actually fixing them.

What is a Flaky Test Actually Is (and Why It’s Not About Skill)

A flaky test is any test that produces different results, pass one run, fail the next, without any change to the code being tested. Simple definition. Expensive problem: flaky tests are estimated to cost engineering teams 6 to 8 hours of lost time every week, between re-running builds, investigating false failures, and the compounding effect of teams losing trust in their own test suite.

Google’s own testing infrastructure research is worth citing here because it’s one of the few large-scale, credible studies on the topic: their data shows that larger tests, measured by binary size and memory usage, are significantly more likely to be flaky. That’s not a statement about test-writing skill. It’s a statement about surface area. Bigger, more complex tests touch more moving parts, and more moving parts means more opportunities for timing, state, and environment to diverge between runs.

The root causes cluster into a few consistent categories:

• Timing and synchronization issues: A test asserts on something before the UI has finished rendering, before an async call resolved, or before an animation completed. This is the single most common source of flakiness in UI and mobile testing.
• Test isolation failures: One test leaves data, state, or a mock in a condition that affects the next test. Tests that pass in isolation fail when run in a particular order or in parallel.
• Environment and infrastructure noise: CI runner load, network latency, shared database contention, or resource constraints introduce variability that has nothing to do with the code under test.
• Device and OS fragmentation: Especially in mobile testing, a test that’s stable on one device/OS combination fails intermittently on another due to hardware or platform-specific timing differences.
• Locator and selector fragility: A UI element’s identifier shifts slightly between renders, or multiple similar elements create ambiguity for the test to resolve.

None of these are the result of an engineer not knowing how to write a test. They’re the predictable consequence of testing systems that are inherently asynchronous, distributed, and non-deterministic, using an assertion model that assumes determinism. That mismatch is the actual root cause, and it’s architectural, not a training gap.

How Flaky Test Detection Tools Work Today

Before getting to the gap, it’s worth being clear about what the current generation of flaky test detection tools does well, because it’s genuinely useful, and any team without a detection process should start here before anything else.

BuildPulse

BuildPulse connects to your CI system and monitors test results over time, flagging tests that flip outcomes without a corresponding code change. Its core value is quantification: it shows you exactly how much engineering time your team loses to flaky tests each week, which becomes the evidence you need to justify investing hours into fixing test reliability. It’s framework-agnostic and focused specifically on detection and impact estimation, by design, it doesn’t try to fix or contain the underlying issue itself.

Trunk (Flaky Tests)

Trunk takes detection a step further into workflow: it tracks pass/fail history per commit, calculates a flakiness rating, and, its most distinctive feature, can automatically quarantine tests it judges flaky. Quarantined tests stop blocking pull request merges but stay tracked on a list for follow-up, with ownership routing so someone is accountable for eventually fixing them. For teams whose main pain point is flaky tests blocking otherwise-good PRs, this is a meaningful improvement over a bare red X everyone learns to ignore.

LambdaTest / TestMu AI

LambdaTest (now operating as TestMu AI) applies ML algorithms to test execution data to detect flakiness patterns, particularly useful for teams running cross-browser and cross-device test matrices where flakiness often correlates with specific browser/OS/device combinations rather than the test logic itself.

All three tools solve the same core problem well: they tell you a test is flaky, and in Trunk’s case, they stop it from blocking your build. That’s real, valuable work. But notice what none of them do.

The Gap: Quarantine Detects Flakiness. It Doesn’t Fix It.

Here’s the part that took me longer than it should have to internalize: a quarantined flaky test is still a broken test. It’s just a broken test that’s no longer yelling at you.

Quarantine is a triage tool, not a resolution. It buys your team breathing room, genuinely valuable when a flaky test is blocking ten PRs a day, but the test still doesn’t provide reliable signal. It’s sitting on a list, routed to an owner, waiting for someone to find the time to actually diagnose and rewrite it. In practice, on most teams I’ve seen, that list grows faster than it shrinks. “Fix flaky tests” competes for sprint time against feature work and rarely wins.

Detection tells you which tests are lying to you. Quarantine stops them from lying loudly. Neither one makes them tell the truth.

The actual fix requires addressing the root cause, the timing issue, the isolation failure, the locator fragility, not just suppressing the symptom. And doing that manually, test by test, at the rate most teams accumulate flaky tests, is a losing race against the pace of feature development.

What Actually Fixes Flaky Tests (Not Just Hides Them)

Fixing flaky tests at the root, rather than quarantining them, comes down to addressing the specific failure category:

For timing and synchronization issues

The fix is proper wait conditions, waiting for the actual state you care about (element is interactable, network request resolved, animation completed) instead of a fixed sleep duration. Frameworks like Playwright ship this as auto-waiting by default. Self-healing platforms take it further: instead of a human writing the correct wait condition, an AI-driven test observes actual application state and adapts its timing dynamically, without anyone specifying a wait strategy at all.

For test isolation failures

The fix is ensuring every test starts from a known, clean state, proper setup/teardown, isolated test data, and avoiding shared mutable state between tests. This is often more of a test architecture discipline than a tooling problem, though better test data management platforms help enforce it.

For locator and selector fragility

This is where self-healing technology has matured the most. Rather than relying on a single brittle identifier (an XPath, a CSS selector, an element ID that can shift between renders), self-healing systems identify UI elements using multiple signals, visual context, text content, layout position, accessibility attributes, so a single changed attribute doesn’t break the test. When the intended element still exists but its identifier changed, the test adapts instead of failing.

For device and OS fragmentation

The fix is testing on real devices and real OS/browser combinations rather than emulators alone, combined with monitoring that correlates flakiness to specific device profiles so you can distinguish “this test is fundamentally flaky” from “this test only fails on one specific Android version.”

The common thread: real fixes require the test to adapt to legitimate variability in the application, rather than asserting on a single expected state and failing whenever reality doesn’t match exactly. That’s a fundamentally different capability than detection, it requires the testing system itself to understand what “correct” looks like across a range of valid states, not just compare against one recorded expectation.

How Sofy Detects and Fixes Flaky Tests Automatically

This is where Sofy’s approach differs from the detection-and-quarantine model. Sofy’s AI agents don’t rely on static locators in the first place, which eliminates the single largest source of UI test flakiness before it happens. When an agent executes a test, it identifies elements using multiple contextual signals, not a single brittle selector, so minor UI changes that would break a traditional script don’t register as failures at all.

When the application genuinely changes in a way that affects the test, a workflow restructures, a step is added, the agent adapts its approach proactively, before the test fails, rather than failing first and waiting for a human or a quarantine system to intervene after the fact.

• Root-cause resilience, not pattern detection: Sofy’s agents don’t need to observe a test flip outcomes several times before flagging it as flaky, because the architecture that causes most flakiness (locator fragility, rigid timing assumptions) isn’t present in how the agent operates.
• Adapts instead of quarantines: When something does change, the agent updates its approach and continues, no manual triage list, no test sitting in quarantine waiting for engineering time that never arrives.
• Failure analysis when something is genuinely wrong: When a test does fail, Sofy’s failure analysis agents identify whether it’s a real regression or an environmental issue, with root cause context, not just a pass/fail flip flagged for someone to investigate later.
• Complements existing detection tools: If your team already uses Trunk or BuildPulse for CI-level flake tracking across your broader test suite, Sofy reduces the inflow of new flaky UI/mobile tests those tools have to detect in the first place.

When Quarantine Tools Are Still the Right Call

In fairness: self-healing test execution reduces flakiness in the tests it runs. It doesn’t give you visibility into flakiness across a large, pre-existing suite of scripted tests you’re not ready to migrate, and it doesn’t give you the cross-CI analytics and cost quantification that BuildPulse and Trunk are built for.

If your immediate problem is a legacy suite of thousands of Selenium or Appium tests accumulating flaky results faster than your team can triage them, a detection and quarantine tool is the right first move, it stops the bleeding and gives you the data to prioritize. Self-healing, agent-native testing is the better fit for new test coverage going forward, and for teams ready to stop writing brittle tests in the first place.

The two approaches aren’t mutually exclusive. Many teams will run both: a detection layer across the full CI pipeline, and self-healing execution for the tests being written or migrated now.

Frequently Asked Questions

What causes flaky tests?

Flaky tests are most commonly caused by timing and synchronization issues (asserting before the app finished responding), test isolation failures (shared state between tests), environment and infrastructure noise (CI runner load, network variability), device/OS fragmentation, and fragile locators or selectors that shift between UI renders. None of these are primarily a test-writing skill issue, they’re architectural mismatches between deterministic assertions and non-deterministic systems.

What is flaky test detection?

Flaky test detection is the process of identifying tests that pass and fail inconsistently without a corresponding code change, typically by monitoring pass/fail history across multiple CI runs. Tools like BuildPulse and Trunk automate this by tracking results over time and flagging tests whose outcomes flip unpredictably.

Does quarantining a flaky test fix it?

No. Quarantining stops a flaky test from blocking builds and merges, but the underlying issue causing the flakiness, a timing problem, a fragile locator, a test isolation failure, remains unresolved. The test still doesn’t provide reliable signal; it’s simply no longer blocking other work while it waits to be fixed.

How is self-healing testing different from flaky test quarantine?

Quarantine detects a test that’s already flaky and isolates it from your build so it stops causing problems, while the root cause remains until a human fixes it. Self-healing testing addresses the root cause proactively, by identifying UI elements contextually instead of through brittle locators, and adapting to legitimate application changes, so tests are less likely to become flaky in the first place, and less likely to require quarantine at all.

Can AI reduce test flakiness?

Yes, specifically by removing the two most common root causes: locator fragility and rigid timing assumptions. AI-driven test execution that identifies elements using multiple contextual signals, rather than a single selector, is inherently more resilient to the minor UI changes that cause most flaky failures. AI can also help diagnose root causes faster during failure analysis, distinguishing genuine regressions from environmental noise.

Should I use a flaky test detection tool or a self-healing test platform?

They solve different problems and often work well together. Detection tools like BuildPulse and Trunk are the right choice for gaining visibility and control over an existing, large test suite’s flakiness. Self-healing platforms are the better fit for reducing how much new flakiness gets introduced in the first place, particularly for UI and mobile tests where locator fragility is the dominant cause.

Stop Managing Flakiness. Start Eliminating It.

Detection and quarantine tools do genuinely useful work, they gave your team visibility and stopped flaky tests from blocking releases. But visibility into a problem isn’t the same as solving it. If your quarantine list keeps growing faster than your team can clear it, the fix isn’t a better dashboard. It’s testing that doesn’t break the same way in the first place. Sofy’s AI agents test your SAP or Dynamics 365 environment, autonomously, with zero scripts, and with audit-ready results.