Repository navigation
skip_area selectors that match nothing block for 5s each — measured at 44% of a real suite #272
Description
The bug
lib/snap_diff/browser_helpers.rb:102:
def self.all_visible_regions_for(selector)
BrowserHelpers.session.all(selector, visible: true).map { |el| region_for(el) }
endCapybara's all defaults to minimum: 1 (capybara/node/finders.rb:259: options = { minimum: 1 }.merge(options) unless minimum_specified) and then blocks in synchronize(query.wait) (:264). So every skip_area selector with zero matches waits the full Capybara.default_max_wait_time — 5 seconds by default — before returning empty.
skip_area is a mask: "exclude whatever is currently there". Waiting for an element to appear is precisely the wrong semantic. A selector that matches nothing has nothing to mask, and that answer is available immediately.
Measured cost, from a real project
From jetthoughts.github.io's application_system_test_case.rb, a comment written after debugging it:
skip_areaCSS selectors resolve via Capybaraall(sel, visible: true)INSIDE the gem, which waitsdefault_max_wait_time(5s) for EVERY selector with zero matches —%w[picture img]on an image-less page = 10s per screenshot (measured; it was 44% of the whole suite).
Their workaround is to wrap every call:
final_options[:wait] ||= Capybara.default_max_wait_time
Capybara.using_wait_time(0) do
assert_matches_screenshot(name, **final_options)
endNote what that costs them: they must also re-pin wait explicitly, because with an implicit wait of 0 the stability loop rejects stability_time_limit > wait. A one-word gem bug forces a four-line workaround plus a comment explaining it.
This is very likely a larger share of the "visual tests are extremely expensive" complaint than stability_time_limit, which was previously measured at ~660 ms per screenshot (#271). 10 s beats 660 ms.
The fix
BrowserHelpers.session.all(selector, visible: true, minimum: 0)minimum: 0 makes all return immediately with whatever currently matches. Selectors that do match are unaffected — they resolve instantly either way, as the report above notes.
Checklist
-
minimum: 0atbrowser_helpers.rb:102 - Audit every other Capybara finder in
lib/for the same implicit wait —all,find,has_css?. Fix the class, not the instance - Guard: a
skip_areaselector matching nothing must return in ~0 ms, notdefault_max_wait_time. Assert on elapsed time with a real Capybara session, not a stub — a stubbed session cannot exhibit this bug - Confirm a matching selector still resolves correctly and is still masked
- Mutation-check: revert to the implicit default and watch the timing guard red
- Once fixed, the
Capybara.using_wait_time(0)wrapper becomes unnecessary — worth a note indocs/configuration.mdfor anyone who copied it
Why this was not caught
Every test in this repo stubs the browser. A stubbed session returns instantly whether or not the selector matches, so no unit test can observe a five-second wait. Same root cause as the unguarded git-baseline lookup found in #267 — the seam that only exists against a real browser.
Activity
pftg commentedon Aug 24, 2026
Reframing from the maintainer:
skip_areais the primary flakiness remedyThis is not a niche option.
skip_areais the mechanism users reach for when a region is unstable — a clock, a carousel, an animation, a font-swap reflow. It is the thing you add to make a flaky visual test reliable.Which makes the bug a trap with a feedback loop:
skip_areato mask the unstable region.The remedy taxes you for using it, and the tax scales with how broadly you apply it — which is exactly what a shared
assert_screenshothelper does. The real project's%w[picture img]on an image-less page is that pattern precisely: sensible, general, and 10 s per screenshot.Worse, the cost is invisible in the direction of causation. Nobody suspects the flakiness fix of being the slowness cause, so the natural response to a slow suite is to reach for tolerances or sleeps — which blind the tests further. The project in question ended up with all three:
skip_areas, 0.03 tolerances, and stability retries, plus aCapybara.using_wait_time(0)wrapper once they finally traced it.Consequences for the fix
Beyond
minimum: 0, this arguesskip_areadeserves first-class treatment rather than being an option that happens to work:skip_areaas the flakiness answer, since that is what it is for, and pair it with the instability report naming the region to mask (Instability is undiagnosable, so people add sleeps instead — and the wait is the dominant cost #271). Those two features are one workflow: the tool tells you what moved, you mask it. Today it tells you nothing and masking is slow.Raising priority accordingly. The one-word fix stands; the framing changes what else belongs with it.
fix: skip_area selectors that match nothing no longer block for 5s (#272
fix: skip_area wait, an invisible summary line, and a phantom asserti…
feat: an optional readiness block, and a run-level tally of selectors…
feat: name the region that would not settle, and what the waiting cost (
chore: prepare v2.0.0.beta4 (#286)
feat: name the region that would not settle, and what the waiting cost (
feat: name the region that would not settle, and what the waiting cost (
feat: name the region that would not settle, and document the readine…