fix: never print a path or command that is not derived from live state (#260)

[pftg]

Two 2.0 items, one commit each.

1. The error names a file that will never exist — fixes #260

No existing screenshot found for <path>! To record it: run the test, then \git add `was unfollowable on the exact run that printed it.build_screenshot_assertioncalledcheck_base_screenshot*before*capture_screenshot, and fail_if_new` defaults to on under CI — so in CI the raise came first and nothing was ever written to the path the message named.

Measured end to end against a real browser (cuprite), before:

$ CI=true  → SnapDiff::ExpectationNotMet: No existing screenshot found for …/issue_260_no_baseline.png!
             ls …/issue_260_no_baseline.png  → no such file
$ CI=       → no exception raised
             ls …/issue_260_no_baseline.png  → file IS on disk

After: the file is on disk in both cases, so git add <path> is a command the user can run right now — and a failing CI job leaves the new screenshot behind for an artifact upload instead of nothing.

Chosen fix: capture first. The alternative — branching the message on whether the file was written — needs two messages kept in sync, and the CI branch still could not offer git add; it would have to send the user somewhere else. Capturing first makes the state match the message in every case, with one code path.

Only the raise moved. The git checkout that drives need_to_compare? stays in check_base_screenshot, and so does the no-baseline warning, which has to read @snapshot.path before the capture overwrites it. Because the two are now separate, the warning would otherwise fire alongside the raise, so check_base_screenshot bows out when fail_if_new is on — the raise says the same thing with the fix attached.

Unguarded seam found by mutation: deleting @snapshot.checkout_base_screenshot outright left the entire unit suite green. Every test stubs Vcs.checkout_vcs, so nothing asserted the matcher ever asks git for the baseline — while need_to_compare?, the warning and the raise all hang off that one call. Added the missing guard.

2. Explicit config outranks the env sniff

fail_if_new stored !ENV["CI"].nil? at require time, which made "the user asked for false" and "CI was absent when the gem loaded" the same false — so the environment, not the user, had the last word. It now has no stored default: nil means nobody said, and only then does the CI sniff answer, read live.

SnapDiff.config.fail_if_new = false   # false, even under CI=true

Same ordering insta narrowed to in insta#924 after Ruff hit it ("normally, CLI flags take precedence over environment variables"), and the inverse of jest#12288, where reading argv.ci instead of detected CI state made CI=1 jest and jest --ci disagree for a whole major version.

The default itself is unchanged. Failing only under CI stays, deliberately — a locally recorded baseline is often worthless across OS. This is a precedence rule only.

test/unit/config_default_timing_test.rb is updated, not deleted: it still pins the two contracts that did not change (root frozen at require time, default_options[:wait] live per call), and its CI probe now pins the new rule instead of the old freeze. The probes are re-run from the v1 entry points by test/legacy/legacy_config_default_timing_test.rb, so the legacy mattr_accessor view gets the same precedence for free.

Evidence

Guard test first for each item; every guard mutation-checked (break it → red → restore with a targeted edit → green):

Mutation	Result
raise moved back before the capture	red — the message says \git add …` -- that file has to exist by then`
drop the fail_if_new bail-out in check_base_screenshot	red — warning fires alongside the raise
delete @snapshot.checkout_base_screenshot	green before the new guard, red after
fail_if_new reader ignores the ivar (env always wins)	red — explicit false outranks CI=1
fail_if_new frozen back into storage at require	red — CI unset after require is seen

rake test:unit (643 runs), rake test:canonical (527 runs), standardrb lib test — all green.

Summary by Sourcery

Make missing-screenshot errors actionable by capturing the screenshot first and give explicit fail_if_new settings precedence over live CI detection.

Bug Fixes:

• Ensure missing-baseline failures capture and persist the new screenshot before reporting the path and git add instruction.
• Prevent duplicate no-baseline warnings when fail_if_new raises.

Enhancements:

• Make explicit fail_if_new configuration take precedence over the live CI environment while preserving CI-based defaults.
• Add coverage for baseline checkout, actionable error paths, warning suppression, and configuration precedence.

Documentation:

• Update configuration and architecture documentation to describe live CI fallback and explicit configuration precedence.

Tests:

• Expand unit and legacy configuration timing tests and screenshot matcher tests for the new behavior.

[sourcery-ai]

Sorry @pftg, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

Next included review available in 18 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a6693fe9-4d23-4f13-a2fa-bc4ae793dd33

📥 Commits

Reviewing files that changed from the base of the PR and between 9988234 and aaddb19.

📒 Files selected for processing (10)

• docs/architecture.md
• docs/configuration.md
• lib/capybara/screenshot/diff/config_legacy.rb
• lib/snap_diff/config.rb
• lib/snap_diff/screenshot_matcher.rb
• test/legacy/legacy_config_default_timing_test.rb
• test/unit/config_default_timing_test.rb
• test/unit/dsl_test.rb
• test/unit/screenshot_matcher_test.rb
• test/unit/snap_diff_config_test.rb

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

The PR fixes misleading missing-screenshot guidance by capturing before enforcing fail_if_new, adds a guard that preserves baseline checkout behavior, and changes fail_if_new so explicit configuration outranks a live CI environment fallback while retaining CI-only failure as the default.

Sequence diagram for screenshot capture before missing-baseline enforcement

sequenceDiagram
    participant Matcher as ScreenshotMatcher
    participant Snapshot
    participant Config
    participant Filesystem

    Matcher->>Snapshot: checkout_base_screenshot()
    Snapshot->>Filesystem: Check base_path.exists?
    Matcher->>Matcher: check_base_screenshot()
    Matcher->>Matcher: capture_screenshot(capture_options, comparison_options)
    Matcher->>Filesystem: Write screenshot to path
    Matcher->>Matcher: fail_if_new_screenshot()
    Matcher->>Config: fail_if_new()
    Config-->>Matcher: CI fallback or explicit setting
    alt New screenshot and fail_if_new enabled
        Matcher-->>Matcher: Raise ExpectationNotMet with git add path
    else Existing baseline or failure disabled
        Matcher->>Matcher: need_to_compare?()
    end

Loading

Flow diagram for fail_if_new configuration precedence

flowchart TD
    A[Read fail_if_new] --> B{@fail_if_new is nil?}
    B -- No --> C[Use explicit setting]
    B -- Yes --> D[Read ENV CI live]
    D --> E{CI is non-empty?}
    E -- Yes --> F[Return true]
    E -- No --> G[Return false]
    C --> H[Explicit value outranks environment]
    F --> H
    G --> H

Loading

File-Level Changes

Change	Details	Files
Reorders screenshot capture and missing-baseline validation so error instructions reference a file that exists.	Captures the screenshot before raising for a missing baseline when fail_if_new is enabled. Keeps VCS baseline checkout and pre-capture warning logic in check_base_screenshot. Suppresses the duplicate warning when the post-capture error will be raised. Adds tests for VCS checkout invocation, on-disk error paths, and warning suppression.	lib/snap_diff/screenshot_matcher.rb test/unit/dsl_test.rb test/unit/screenshot_matcher_test.rb
Makes fail_if_new configuration explicitly override a live CI environment fallback.	Stores nil rather than a require-time CI-derived default. Implements a reader that consults ENV['CI'] only when no explicit value is set. Supports assigning nil to restore environment-based behavior. Documents the revised default timing and precedence contract. Adds canonical and legacy entry-point timing and precedence coverage.	lib/snap_diff/config.rb lib/capybara/screenshot/diff/config_legacy.rb docs/architecture.md docs/configuration.md test/unit/config_default_timing_test.rb test/legacy/legacy_config_default_timing_test.rb test/unit/snap_diff_config_test.rb

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#260	Capture a new screenshot before raising the fail_if_new error, so the path named in the message actually exists and can be added to git, including in CI.	✅
#260	Preserve the baseline checkout performed by check_base_screenshot while moving only the error raise, and avoid emitting the separate no-baseline warning when fail_if_new will raise.	✅
#260	Add regression coverage that exercises the user-facing screenshot flow and verifies both that the named file is written before the error and that the baseline checkout remains intact.	✅

Possibly linked issues

• The new-screenshot error prints a path that will never exist in CI #260: PR directly fixes issue The new-screenshot error prints a path that will never exist in CI #260 by capturing before raising, ensuring the reported screenshot path exists in CI.
• The new-screenshot error prints a path that will never exist in CI #260: PR directly implements the issue’s first requirement: capture missing baselines, then fail with an actionable live path.
• #ADR-004: The PR implements two specific 2.0 items explicitly documented in this issue: The new-screenshot error prints a path that will never exist in CI #260 and config precedence.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.