Skip to content

skip_area selectors that match nothing block for 5s each — measured at 44% of a real suite #272

Closed

Description

@pftg

The bug

lib/snap_diff/browser_helpers.rb:102:

def self.all_visible_regions_for(selector)
  BrowserHelpers.session.all(selector, visible: true).map { |el| region_for(el) }
end

Capybara's all defaults to minimum: 1 (capybara/node/finders.rb:259: options = { minimum: 1 }.merge(options) unless minimum_specified) and then blocks in synchronize(query.wait) (:264). So every skip_area selector with zero matches waits the full Capybara.default_max_wait_time — 5 seconds by default — before returning empty.

skip_area is a mask: "exclude whatever is currently there". Waiting for an element to appear is precisely the wrong semantic. A selector that matches nothing has nothing to mask, and that answer is available immediately.

Measured cost, from a real project

From jetthoughts.github.io's application_system_test_case.rb, a comment written after debugging it:

skip_area CSS selectors resolve via Capybara all(sel, visible: true) INSIDE the gem, which waits default_max_wait_time (5s) for EVERY selector with zero matches — %w[picture img] on an image-less page = 10s per screenshot (measured; it was 44% of the whole suite).

Their workaround is to wrap every call:

final_options[:wait] ||= Capybara.default_max_wait_time
Capybara.using_wait_time(0) do
  assert_matches_screenshot(name, **final_options)
end

Note what that costs them: they must also re-pin wait explicitly, because with an implicit wait of 0 the stability loop rejects stability_time_limit > wait. A one-word gem bug forces a four-line workaround plus a comment explaining it.

This is very likely a larger share of the "visual tests are extremely expensive" complaint than stability_time_limit, which was previously measured at ~660 ms per screenshot (#271). 10 s beats 660 ms.

The fix

BrowserHelpers.session.all(selector, visible: true, minimum: 0)

minimum: 0 makes all return immediately with whatever currently matches. Selectors that do match are unaffected — they resolve instantly either way, as the report above notes.

Checklist

  • minimum: 0 at browser_helpers.rb:102
  • Audit every other Capybara finder in lib/ for the same implicit wait — all, find, has_css?. Fix the class, not the instance
  • Guard: a skip_area selector matching nothing must return in ~0 ms, not default_max_wait_time. Assert on elapsed time with a real Capybara session, not a stub — a stubbed session cannot exhibit this bug
  • Confirm a matching selector still resolves correctly and is still masked
  • Mutation-check: revert to the implicit default and watch the timing guard red
  • Once fixed, the Capybara.using_wait_time(0) wrapper becomes unnecessary — worth a note in docs/configuration.md for anyone who copied it

Why this was not caught

Every test in this repo stubs the browser. A stubbed session returns instantly whether or not the selector matches, so no unit test can observe a five-second wait. Same root cause as the unguarded git-baseline lookup found in #267 — the seam that only exists against a real browser.

Activity

  1. added this to the 2.0.0 milestone on Aug 24, 2026
  2. pftg commented on Aug 24, 2026

    @pftg
    CollaboratorAuthor

    Reframing from the maintainer: skip_area is the primary flakiness remedy

    This is not a niche option. skip_area is the mechanism users reach for when a region is unstable — a clock, a carousel, an animation, a font-swap reflow. It is the thing you add to make a flaky visual test reliable.

    Which makes the bug a trap with a feedback loop:

    1. A page is flaky → add skip_area to mask the unstable region.
    2. The same helper is applied across many pages, so some selectors match nothing on some pages.
    3. Every non-matching selector costs 5 seconds.
    4. The suite becomes slow because you fixed the flakiness.

    The remedy taxes you for using it, and the tax scales with how broadly you apply it — which is exactly what a shared assert_screenshot helper does. The real project's %w[picture img] on an image-less page is that pattern precisely: sensible, general, and 10 s per screenshot.

    Worse, the cost is invisible in the direction of causation. Nobody suspects the flakiness fix of being the slowness cause, so the natural response to a slow suite is to reach for tolerances or sleeps — which blind the tests further. The project in question ended up with all three: skip_areas, 0.03 tolerances, and stability retries, plus a Capybara.using_wait_time(0) wrapper once they finally traced it.

    Consequences for the fix

    Beyond minimum: 0, this argues skip_area deserves first-class treatment rather than being an option that happens to work:

    • A non-matching selector is not an error, but it may be a mistake. After the wait is gone, consider reporting selectors that matched nothing — a typo'd selector currently masks nothing and says nothing, which is the silent-no-op class this release has been eliminating everywhere else. It must not become noise for the legitimate "this page has no images" case, so think about where it belongs (debug output, the summary line, or once per process per selector).
    • Documentation should present skip_area as the flakiness answer, since that is what it is for, and pair it with the instability report naming the region to mask (Instability is undiagnosable, so people add sleeps instead — and the wait is the dominant cost #271). Those two features are one workflow: the tool tells you what moved, you mask it. Today it tells you nothing and masking is slow.

    Raising priority accordingly. The one-word fix stands; the framing changes what else belongs with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions