Visual Regression Testing: Catching the Bugs Your Functional Tests Pass

Visual Regression Testing: Functional tests pass while layouts break. How visual testing for web apps works: baselines, masking, tolerance thresholds, and noise vs real bugs.

Your checkout test passed. It found the Place order button, confirmed the label, clicked it and checked that the order went through. Meanwhile, on a small phone, a cookie banner sits on top of half the button, the price has wrapped onto two lines, and the product image is a grey box. A real customer would have struggled to buy anything. The test had no idea.

That isn’t a flaw in the test. Functional tests check what the page is: elements, text, values, and states. Users experience what the page looks like. Most of the time those agree. When they don’t, you get a passing build and a broken screen.

Visual regression testing closes that gap by comparing what the page actually looks like with what it looked like when you last approved it. This guide covers the bugs it catches, how it works, and the four decisions that decide whether it helps your team or buries it in false alarms: baselines, masking, tolerance and knowing when a difference is just noise.

Functional tests read the page. Users look at it. Visual regression testing is how you test what they actually see.

A functional assertion asks a narrow question: is this element present, does it hold this text, is it enabled. A layout bug usually leaves every one of those answers unchanged.

Visual bugWhy the functional test still passesWhat a visual test sees
Overlapping elementsBoth elements exist and hold the right textOne covers the other
Truncated or wrapped textThe full string is in the page; the CSS hides or wraps it“Free shipping on ord…” or a price split over two lines
Wrong theme or colourTheme and colour aren’t usually assertedWhite text on a white background in dark mode
Missing imageThe image element exists; the file failed to loadA blank box or broken-image icon
Off-screen or squeezed contentThe element is in the page, just not where users can reach itA button pushed below the fold or out of the viewport
Font fallbackText content is correctA different font, wider spacing and lines that no longer fit
Spacing and alignment driftNo assertion covers marginsA form that looks broken, though every field works

Most of these come from changes nobody thinks of as risky: a CSS refactor, a design-system update, a new third-party banner, a browser update, a longer translation. You could write a functional assertion for each one. Nobody does, because there are thousands of them. One screenshot covers all of them at once.

Every visual testing tool, open source or commercial, follows the same loop.

  • Capture a baseline. Render the page or component in a known state and save the image as the approved version.
  • Capture again on change. On the next build, render the same page in the same state.
  • Compare. The tool compares the two images and highlights what changed, from a simple pixel difference to more selective AI-based comparison.
  • Review. A person decides whether each difference is a bug to fix or an intended change. If it’s intended, the new image becomes the baseline.

The comparison is the easy part. The review is where visual testing succeeds or fails. If every run produces fifty differences and forty-eight are meaningless, people stop looking, and then the two real bugs get approved with the rest. Everything below is about keeping that review small and worth doing.

A baseline is only as useful as the conditions it was captured in. The same page renders differently on different machines, and a visual test can’t tell a rendering difference from a bug.

Playwright’s visual comparison documentation puts it plainly: browser rendering can vary based on the host OS, version, settings, hardware, power source (battery vs. power adapter), headless mode and other factors, so for consistent screenshots, run tests in the same environment where the baselines were generated. That advice applies whatever tool you use.

In practice that means four rules:

  • Generate baselines where the tests run. If comparisons run in CI, generate baselines in CI, not on a developer’s laptop.
  • One baseline per rendering context. Each browser, viewport and theme you care about needs its own baseline. Chrome on desktop and Safari on an iPhone are different expectations, not variations of one.
  • Control the page state. Use fixed test data, a fixed date and a logged-in user whose account doesn’t change. A baseline of a page that changes on its own will fail on its own.
  • Treat baseline updates like code changes. Keep baselines in version control or your tool’s approval history, and make someone review an update before it’s accepted. Accepting a new baseline without looking turns a regression into the new expectation.

Some content is supposed to change: timestamps, live prices, ads, rotating banners, recommendations, user avatars, a clock. Left alone, each produces a difference on every run. Most tools let you mask a region, covering it before comparison so that changes inside it are ignored.

Masking is necessary and easy to overdo. A mask over the whole header hides the logo, the navigation and the search box along with the timestamp. Keep masks as tight as the dynamic element, and prefer fixing the content over hiding it:

  • Freeze data where you can: seeded test accounts, fixed prices, a frozen clock. A known value is better than a hidden one.
  • Stop motion. Disable or finish CSS animations and transitions, hide the text cursor, and pause carousels before capture.
  • Wait for the page to settle. Capture after web fonts and images have loaded, not on the first paint. Playwright, for example, waits until two consecutive screenshots match before it compares.
  • Mask what’s left. Third-party ads and embeds you can’t control are the right candidates for a mask.

Review your masks occasionally. A mask added for a banner that was removed six months ago is now hiding part of the page for no reason.

No two renders are guaranteed to be identical at the pixel level, even in the same environment. Anti-aliasing and sub-pixel text rendering can change the color of a few edge pixels between runs. So visual tools let you set a tolerance, usually in two forms:

Tolerance typeWhat it controlsUse it for
Per-pixel colour thresholdHow different a pixel’s colour must be before it counts as changedAbsorbing anti-aliasing and tiny shading differences
Changed-pixel count or ratioHow many changed pixels the image can have before the test failsAllowing small, scattered differences on a large page
Region or layout comparisonWhether elements moved or changed, rather than individual pixelsPages where content legitimately varies but the layout shouldn’t

In Playwright, these are the threshold, maxDiffPixels and maxDiffPixelRatio options. Commercial tools use their own names, and some replace pixel counts with AI that tries to judge whether a change is perceptible to a person.

Set tolerance by what’s on the screen. A logo, an icon or a design-system button can be held to almost zero difference. A full marketing page with photography needs more room. And never widen a threshold just to make a failing test pass. If a test keeps failing on noise, fix the source of the noise: the environment, the data or a mask. A threshold wide enough to hide noise is often wide enough to hide a bug.

The review step comes down to one question for each difference: would a user notice, and would they care? This table is a starting point for answering it.

Usually noiseUsually a real bug
Scattered single pixels along text edges (anti-aliasing)Text that’s cut off, wrapped differently or overflows its box
A whole-page difference after a browser or OS updateElements that overlap or cover each other
A change inside a masked or known-dynamic regionA missing image, icon or button
A one-pixel shift of the entire pageOne element moved while everything around it stayed
Different frames of an animationA colour change on a brand, status or call-to-action element
 A component that renders in light theme inside a dark-theme page

Two patterns help. If every page changed, suspect the environment before the code. If one thing changed and everything else stayed still, take it seriously, because that’s what a real regression usually looks like.

Everything above applies to mobile, with more rendering contexts to cover. A layout that’s fine on a large phone can break on a small one, and the same app looks different across operating system versions, themes and accessibility settings. These are the mobile cases that most often pass functional tests and fail visually:

  • Small screens. Prices, buttons and labels that wrap or truncate on the narrowest supported device.
  • Notches and rounded corners. Content placed under a camera cutout or the home indicator, where it can’t be read or tapped.
  • Dark mode. A hard-coded color that was invisible in nobody’s test because every test ran in light mode.
  • Larger text sizes. Users who turn up the system font size see labels that overflow and buttons that no longer fit.
  • Longer translations. A German or French string that’s half as long again as the English one it replaced.

You can’t baseline every device, so pick the extremes: the smallest and largest screens you support, both themes, and the largest text size on your key screens. Our mobile app testing checklist covers the full device and OS matrix; visual testing is how you check what that matrix looks like.

Visual testing doesn’t replace functional tests. It covers a different kind of failure, and it works best when it’s added deliberately rather than everywhere.

  • Start with the pages that earn money or trust: home, search results, product page, cart, checkout, sign-in, and the key screens of your app.
  • Capture at meaningful moments, such as the end of a step in a functional flow, rather than screenshotting every step.
  • Run it on every pull request for the pages a change touches, so the person who made the change reviews the difference while it’s fresh.
  • Cover the browsers your users actually use. Safari renders differently from Chrome, and a visual test in one doesn’t cover the other. Our guide to cross-browser testing tools covers how teams choose that matrix.

There’s also a relationship worth understanding with self-healing tests. A self-healing functional test keeps running when a button moves or a selector changes, which is what you want for keeping a suite stable. But it means the functional test will adapt quietly to a layout change. Visual regression testing is the check that asks whether that change was intended. Our explainer on self-healing test automation covers the first half; this post covers the second.

Visual checks also earn their place before big trading days, when a promotional banner or a last-minute style change can break a layout without failing a single functional test. Our Black Friday app testing guide covers the rest of that freeze-week list.

Sofy’s web testing agents run functional flows across Chrome, Safari, Firefox and Edge, at desktop, tablet and mobile viewports, from tests described in plain English. According to the product page, the agents detect selector changes, layout shifts and updated content and rewrite the affected steps automatically, and runs can be triggered from CI/CD pipelines or on a schedule.

On mobile, Sofy runs tests on real iOS and Android devices, and in January 2024 it introduced a Visual Match feature aimed at testing visual elements in mobile apps. Sofy’s Solutions menu also lists Visual QA for finding visual inaccuracies across devices.

The combination is the point of this post: functional agents that keep running through layout changes, alongside visual checks that ask whether those changes were meant to happen.

Free Sofy AI Agents Demo

Get Your Free Sofy AI Agents Demo

See how Sofy AI Agents transform application testing with autonomous, intelligent execution.

Sofy values your privacy by using your information for account management, product delivery, and occasional updates with your consent.

A visual tool can tell you that something changed. It can’t tell you whether the change was intended, whether it matches the design, or whether a user would care. That review, approving a new baseline or sending a difference back as a bug, is the human part of visual testing, and it’s the part that makes it worth running.

Someone also has to decide what to cover and how strict to be: which pages matter, which devices are the extremes, how much tolerance a page deserves. Get those decisions right, and the review stays small enough that people actually do it.

What is visual regression testing?

Visual regression testing compares screenshots of a page or screen with an approved baseline to catch unintended visual changes: overlaps, truncated text, missing images, wrong colours or broken layouts that functional tests don’t check.

How is visual testing different from functional testing?

Functional tests check what the page contains and does: elements, text, values and behaviour. Visual tests check what it looks like. A page can pass every functional test and still be unusable because of a layout bug.

Why do visual tests produce false positives?

Usually because of differences in the rendering environment, dynamic content or animation. Generating baselines where tests run, freezing test data, disabling animations and masking truly dynamic regions remove most of them.

What tolerance threshold should I use?

It depends on the content. Logos, icons and design-system components can be held to near zero; full pages with photography need more room. Fix sources of noise before widening a threshold, because a wide threshold can hide real bugs.

Does visual testing work for mobile apps?

Yes. The same principles apply, with more contexts to cover: screen sizes, notches, dark mode, larger text sizes and translations. Baseline the extremes on your key screens rather than every device.

Functional tests and users look at different things. Tests read the page; users see it. Visual regression testing is how you test the second without writing an assertion for every margin, color and overlap.

Make it useful by getting four decisions right: generate baselines where tests run, freeze and mask what changes on its own, set tolerance by what’s on the screen, and review differences with one question in mind: would a user notice? Do that, and the bugs your functional tests pass stop reaching your customers.

See Sofy in action. Book your demo.

We’ll show you exactly how it works for your team in 30 minutes.

Scriptless test automation—no coding or framework setup

Run tests on hundreds of real iOS and Android devices

Integrate with your CI/CD in minutes

Self-healing test that adapt as your app changes

Real-time debugging with logs, crash reports, and performance data