fix: stop passing green when a screenshot has no committed baseline

[pftg]

Two boot/behaviour bugs a customer persona hit in their first afternoon, plus the message lies around them.

1. The false green (release blocker)

Baselines are read from git (git show HEAD:<path>), never from disk, and fail_if_new defaults to !ENV["CI"].nil? — false locally. A screenshot with no committed baseline was therefore not compared at all: need_to_compare? false, no assertion registered, test green, screenshot written over whatever was on disk.

Reproduced end to end in a scratch app (static page + real headless Chrome, git repo, no committed baseline):

Before

run 1 (heading blue)   -> 1 runs, 1 assertions, 0 failures   md5 6aff9207...
edit page: blue -> red
run 2 (heading red)    -> 1 runs, 1 assertions, 0 failures   md5 e2c6a13f...

After

run 1 (heading blue)
[snap_diff] No committed baseline for .../doc/screenshots/heading.png -- nothing was compared. Commit it to enable comparison.
1 runs, 1 assertions, 0 failures
[snap_diff] 1 screenshot had no committed baseline and was NOT compared: heading. Commit the captured file(s) to enable comparison.

edit page: blue -> red

run 2 (heading red)
[snap_diff] No committed baseline for .../doc/screenshots/heading.png (the file already there is not a baseline until it is committed) -- nothing was compared. Commit it to enable comparison.
1 runs, 1 assertions, 0 failures
[snap_diff] 1 screenshot had no committed baseline and was NOT compared: heading. Commit the captured file(s) to enable comparison.

Positive control, same app: git add doc/screenshots/heading.png && git commit, then change the page — the run fails, as it should. The feature is untouched; only the silence is gone.

fail_if_new = false is a deliberate, documented choice, so this warns rather than raises. Once per screenshot, naming the real path and the action. The end-of-run summary gets a matching line: the reporters count what was compared, so N screenshots compared, no failures was silent about exactly the screenshots that passed unlooked-at.

2. RSpec-only bundles could not boot the gem at all

Bundler.require — the default — requires the gem's own name, so lib/capybara-screenshot-diff.rb and lib/snap_diff-capybara.rb load for every consumer. Both hard-required capybara_screenshot_diff/minitest → require "minitest", which is not a declared runtime dependency (the gemspec declares capybara only):

There was an error while trying to load the gem 'capybara-screenshot-diff'. (Bundler::GemRequireError)
Gem Load Error is: cannot load such file -- minitest

…from a gem that ships a first-class RSpec integration. Verified with a real bundle install (rspec, no minitest) before and after.

minitest is not added as a runtime dependency — that would force it on RSpec users to fix a problem caused by assuming it. The gem-name entry now loads the gem, then feature-detects:

• minitest present → the documented zero-require Rails path is unchanged; assert_matches_screenshot is still there with no explicit require.
• minitest absent → boots fine, and says so, naming the integration to require and the require: false that silences the line. A gem that loads and then does nothing, silently, is its own bug report.

The v1 gem-name file forwards to the surviving one, so there is one copy of the detection rather than two that drift. Explicit requires are deliberately untouched: require "snap_diff/integrations/minitest" still hard-requires minitest and still fails loudly — that is a line the user wrote at a path they chose. snap_diff/static does the same and is left alone for the same reason (docs note, not a code change).

3. Messages that were not true

• RECORD_SCREENSHOTS=1 bundle exec rake test — nothing in lib/ has ever read that variable; it is this repo's own test-suite convention. Deleted rather than implemented. Measured first: one run captures every changed screenshot to its real path, including the second screenshot of a test whose first screenshot already differed (verification happens at teardown, not at the assert). So git add <screenshot> && git commit already is the bulk-record path; an env var would only add a way to accept regressions without looking at them, and would need a CI guard to avoid being exactly that.
• <name>.base.png in the new-screenshot error — a generated temp file nobody creates or commits. Now names the real screenshot.
• Capybara::Screenshot::Diff.fail_if_new = false — the v1 namespace, in the release whose headline is SnapDiff. Now the canonical name.

4. Unsilenceable deprecation warning

integrations/minitest.rb warned when the caller chain included capybara-screenshot-diff.rb — the gem-name Bundler entry point, which Bundler.require loads for everyone with the gem in their Gemfile whatever they required explicitly. Verified through a real Bundler.require before and after. Only capybara/screenshot/diff is a genuinely deprecated choice; the remedy line now names snap_diff/integrations/minitest instead of another deprecated require. It was also a bare Kernel#warn, so SnapDiff.silence_deprecations / SNAP_DIFF_SILENCE_DEPRECATIONS did not reach it — it now honours the documented switch.

Guards

Every fix has a test that failed before it, and each was mutation-checked (break the fix, watch the test red, restore with a targeted edit, watch it green):

Mutation	Test that reds
drop the missing-baseline warning	3 matcher tests + the summary test
dedupe always returns true	"warns once per screenshot"
warning names .base.png	"must name the real screenshot"
drop the on-disk clause	"flags an uncommitted screenshot already on disk"
warn after capture instead of before	"already there" must not appear for a fresh screenshot
drop the summary line	"finalize! names the screenshots that had no committed baseline"
summary always returns a string	"says nothing about baselines when every screenshot had one"
restore the old error message	"the fail_if_new error tells the truth"
re-add the gem-name file to the activation check	"the gem-name entry point does not warn"
never warn	"the v1 namespace entry point still warns"
ignore the silence switch	"silenced by the documented deprecation switch"
hard-require the minitest integration	"loads in a bundle without minitest" (canonical + v1)
drop the "activated nothing" line	"says where to go when it activated nothing"
never load the minitest integration	"still auto-activates … when minitest is present"
v1 gem-name file stops forwarding	"the v1 gem-name entry point loads without minitest"

The minitest-absence probes chdir out of the project (inside it, RubyGems re-adds -rbundler/setup and the gemspec unshifts the real lib/) and shadow minitest with a raising stub first on the load path, so require "minitest" fails exactly as in a bundle without it. Every probe opens with a gate line asserting the environment really is what it claims, and both gate lines were shown to abort when it is not (gate-true with the shim installed, gate-false without). A real bundle install reproduced the crash and confirmed the fix by hand; it has no place in the unit suite.

rake test:unit (602 runs), rake test:canonical (494 runs) and standardrb lib test are green. No existing test expectation was edited. Two deletion-simulation fixtures were updated to track the new shape of snap_diff-capybara.rb — legacy_deletion_test.rb's EDITS and the canonical_suite_has_no_legacy_refs_test.rb allowlist — both of which document that going red on such a change is their job.

Docs follow-ups (not touched here — docs/ is another lane's)

• docs/ci-integration.md:190,229, docs/docker-testing.md:19, docs/migration-guide.md:251 still tell users to run RECORD_SCREENSHOTS=1. It never did anything. Replace with: run the suite, then git add doc/screenshots/*.png && git commit.
• README.md:104 — "If the change is intentional, delete the baseline and re-run to update it." Verified wrong: baselines come from git, so deleting the file changes nothing and the run fails again. The accept step is git add <screenshot> && git commit (the failing run already left the new screenshot at that path).
• README.md:107-110 lists three artifacts; a failing run leaves five: homepage.png, homepage.base.png, homepage.diff.png, homepage.base.diff.png, homepage.heatmap.diff.png. A passing run cleans up all but homepage.png — verified, so .base.png never survives a green run and needs no code change, only a row in that table. The README gitignore block already covers all five (*.base.png plus *.diff.png, which globs .base.diff.png and .heatmap.diff.png).
• RSpec/Cucumber setup should now recommend gem "capybara-screenshot-diff", require: false — it is also what silences the new "minitest is not in this bundle" boot line.
• snap_diff/static (and capybara_screenshot_diff/static) still hard-require the Minitest integration, so an RSpec user who requires it gets a LoadError. Deliberate (loud failure at a line the user wrote), but worth documenting as minitest-only.

🤖 Generated with Claude Code

https://claude.ai/code/session_014BQJX6eWzBj2UTm5zQsjEs

[sourcery-ai]

Sorry @pftg, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

Next included review available in 12 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 05dd0a94-2d83-4c85-b6d0-c79188c636a4

📥 Commits

Reviewing files that changed from the base of the PR and between 4bd7416 and dd5e3b0.

📒 Files selected for processing (13)

• lib/capybara-screenshot-diff.rb
• lib/snap_diff-capybara.rb
• lib/snap_diff/integrations/minitest.rb
• lib/snap_diff/reporting.rb
• lib/snap_diff/screenshot_matcher.rb
• test/legacy/legacy_entry_point_probe_test.rb
• test/legacy/minitest_activation_warning_test.rb
• test/test_helper.rb
• test/unit/canonical_suite_has_no_legacy_refs_test.rb
• test/unit/gem_name_entry_point_test.rb
• test/unit/legacy_deletion_test.rb
• test/unit/reporter_interplay_test.rb
• test/unit/screenshot_matcher_test.rb

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Clarifies and scopes the deprecation warning source, introduces tracking and run‑summary reporting for screenshots without committed baselines, adjusts matcher behavior and messaging around missing baselines, and adds tests to lock in the new behavior and warning semantics.

Sequence diagram for missing screenshot baseline handling

sequenceDiagram
    participant Matcher as ScreenshotMatcher
    participant VCS as SnapshotVCS
    participant Reporting as SnapDiff::Reporting
    participant FS as Filesystem
    participant Test as TestRun

    Matcher->>VCS: checkout_base_screenshot()
    VCS-->>Matcher: committed baseline missing
    alt fail_if_new enabled
        Matcher->>Matcher: raise ExpectationNotMet
    else fail_if_new disabled
        Matcher->>Reporting: record_missing_baseline(screenshot_full_name)
        Reporting-->>Matcher: first occurrence
        Matcher->>FS: path.exist?
        FS-->>Matcher: on-disk file status
        Matcher-->>Test: warn no committed baseline
        Matcher->>Matcher: capture_screenshot()
        Test->>Reporting: finalize!()
        Reporting-->>Test: missing_baselines_summary()
    end

Loading

File-Level Changes

Change	Details	Files
Deprecation warning in the Minitest integration now only fires for the deprecated namespace entrypoint and points to the canonical SnapDiff require path.	Simplified deprecated entrypoint detection to only match capybara/screenshot/diff.rb in the call stack. Updated deprecation warning text to describe the behavior of require "capybara/screenshot/diff" and direct users to require "snap_diff/integrations/minitest".	lib/snap_diff/integrations/minitest.rb test/legacy/minitest_activation_warning_test.rb
Reporting collects screenshots that lacked committed baselines and prints a summarized warning line at the end of the run.	Added @missing_baselines state and thread‑safe record_missing_baseline and reset_missing_baselines! helpers to Reporting. Extended finalize! to emit a summary line when any missing baselines were recorded, with proper pluralization and names. Ensured test isolation by resetting missing baseline state in test teardown. Added unit tests verifying the presence/absence of the summary output depending on whether missing baselines exist.	lib/snap_diff/reporting.rb test/test_helper.rb test/unit/reporter_interplay_test.rb
ScreenshotMatcher warns when a screenshot has no committed baseline, once per screenshot, and improves the fail_if_new error message to reference the real file path and correct remediation steps.	Required the Reporting module and used record_missing_baseline from the matcher to gate per‑screenshot warnings. Changed baseline presence check to early‑return when base_path exists, and to raise ExpectationNotMet using the actual screenshot path and updated messaging when fail_if_new is true. Introduced warn_no_committed_baseline to emit a user‑facing warning when a screenshot had no committed baseline and fail_if_new is false, including special wording when a file already exists on disk. Added unit tests that cover warning content, once‑per‑screenshot behavior, handling of uncommitted on‑disk screenshots, quiet behavior when a committed baseline exists, and truthfulness of the fail_if_new error message.	lib/snap_diff/screenshot_matcher.rb test/unit/screenshot_matcher_test.rb

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.