feat: make the failure message legible

[pftg]

Two people ran the product and could not read its most important output.

Before (real run, test/fixtures/summary_line_case.rb, real git baselines)

  1) Failure:
SummaryLineCase#test_changed [test/fixtures/summary_line_case.rb:53]:
Screenshot does not match for 'changed': ({"area_size":100.0,"region":[20.0,15.0,30.0,25.0]})
/Users/me/app/screenshots/changed.png
/Users/me/app/screenshots/changed.base.diff.png
/Users/me/app/screenshots/changed.diff.png
/Users/me/app/screenshots/changed.heatmap.diff.png
/Users/me/app/test/fixtures/summary_line_case.rb:53:in 'block (2 levels) in <class:SummaryLineCase>'
...18 more minitest frames...

region is four bare numbers. area_size has no denominator. The four paths are
unlabelled and sit against the backtrace, so they read as stack noise -- and the
baseline (.base.png) is not among them: the file listed first is the capture that
just overwrote it.

After (same run)

  1) Failure:
SummaryLineCase#test_changed [test/fixtures/summary_line_case.rb:53]:
Screenshot does not match for 'changed': the change spans 100 of 6400 px (1.56% of the 80x80 image)
  changed region: [20.0,15.0,30.0,25.0] (left,top,right,bottom edges)
  judged against: no tolerance thresholds configured (any difference fails)
  baseline:           screenshots/changed.base.png
  actual:             screenshots/changed.png
  baseline annotated: screenshots/changed.base.diff.png
  actual annotated:   screenshots/changed.diff.png
  heatmap:            screenshots/changed.heatmap.diff.png
/Users/me/app/test/fixtures/summary_line_case.rb:53:in 'block (2 levels) in <class:SummaryLineCase>'
...

With a tolerance set, the metric block also carries its unit:

  difference level: 0.00765625 (0.77% of the image area)
  judged against: tolerance 0.001, color_distance_limit 20

What changed

• Labels and order. Baseline first, then the actual capture, then what we drew
  on them. A label is printed only when that file is on disk -- chunky_png
  produces no diff mask, so it gets no heatmap line (the old message printed one
  regardless; same family as The new-screenshot error prints a path that will never exist in CI #260).
• Numbers mean something. area_size gets a denominator and a percentage of
  total pixels plus the image dimensions. region is stated as edge coordinates --
  verified in Region#to_edge_coordinates, not guessed. difference_level is the
  changed share of the image area, so it is printed with that unit and only when a
  tolerance made us compute it.
• The threshold is restated (VCR-style), or the message says plainly that none
  was configured and any difference fails.
• Relative paths (to SnapDiff.config.root), absolute when the file is outside
  it. Chosen because most terminals resolve relative paths against the working
  directory, so they stay click-through-able while being short enough to read --
  and short paths visually separate the artifact block from the absolute backtrace
  frames below it, which was half the "reads as noise" complaint.

Not changed

Pass/fail behaviour, exit codes, and the end-of-run summary line
([snap_diff] 1 verified, 1 changed, 0 new) are untouched -- the after-run above
still prints it verbatim.

Evidence

• Seven guard tests in test/unit/reporters/default_test.rb, each mutation-checked:
  the denominator, the region interpretation, the difference-level unit, the
  threshold line, the labels, the on-disk filter, and both directions of the
  relative/absolute path rule. Every mutation went red; each was restored by a
  targeted edit and confirmed green.
• Surfacing checked under Minitest and RSpec with real runs in fresh
  processes (RSpec indents the block under Failure/Error: and it holds up).
  Cucumber re-raises the same ExpectationNotMet string with no message handling
  of its own.
• rake test:unit (617 runs), rake test:canonical (512 runs), standardrb lib test
  all green.

Expected test churn

Two expectations in test/unit/dsl_test.rb pinned the old wording verbatim and are
updated -- a commissioned wording change, not a behaviour break. One of them was
asserting a heatmap.diff.png path that chunky_png never writes.

Summary by Sourcery

Improve screenshot mismatch failure messages so their metrics, thresholds, and diagnostic artifacts are immediately understandable.

New Features:

• Make screenshot mismatch failures easier to interpret with contextual metrics, threshold information, and clearly labeled artifact paths.

Bug Fixes:

• Avoid listing screenshot artifacts that were not created and prevent unreadable or misleading failure details.

Enhancements:

• Report changed area as pixels and image percentage, clarify region coordinates and difference-level units, and use concise relative paths where possible.

Tests:

• Expand reporter coverage for metrics, thresholds, artifact labeling and filtering, and relative versus absolute paths.

[sourcery-ai]

Sorry @pftg, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

Next included review available in 52 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8ff09382-1007-4b77-8512-63cb431bea44

📥 Commits

Reviewing files that changed from the base of the PR and between f9ef044 and 95b6ae1.

📒 Files selected for processing (3)

• lib/snap_diff/reporters/default.rb
• test/unit/dsl_test.rb
• test/unit/reporters/default_test.rb

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Refactors the default screenshot comparison failure message to be structured, explanatory, and artifact-aware, including denominated metrics, threshold restatement, and labeled, relative artifact paths, with tests updated and expanded to lock in the new behavior.

Sequence diagram for structured screenshot failure reporting

sequenceDiagram
    participant Comparison
    participant DefaultReporter
    participant Driver
    participant SnapDiffConfig
    participant Filesystem

    Comparison->>DefaultReporter: build_error_message()
    DefaultReporter->>Driver: dimension(comparison.base_image)
    Driver-->>DefaultReporter: width, height
    DefaultReporter->>Comparison: difference.region_area_size
    DefaultReporter->>Comparison: difference.coordinates
    DefaultReporter->>Comparison: difference.ratio
    DefaultReporter->>Comparison: difference.options
    DefaultReporter->>Comparison: difference.meta
    DefaultReporter->>SnapDiffConfig: config.root
    DefaultReporter->>Filesystem: path.exist? for each artifact
    Filesystem-->>DefaultReporter: on-disk artifact paths
    DefaultReporter-->>Comparison: headline, metric_lines, artifact_lines

Loading

File-Level Changes

Change	Details	Files
Rework failure message generation to produce a structured, human-readable summary with denominated metrics and threshold information.	Replace the old JSON dump plus raw paths with a composed message built from a headline, metric lines, and artifact lines. Compute total pixels and changed area to express the difference both as absolute pixels and percentage of the image, including dimensions. Include explicit interpretation of the region coordinates and conditionally print difference level and max distances with units.	lib/snap_diff/reporters/default.rb
Add artifact labeling and on-disk filtering with relative/absolute path logic for comparison outputs.	Define ordered artifact labels mapping to underlying path methods for baseline, actual, annotated images, and heatmap. Filter artifacts to only those whose files actually exist before printing, and align labels for readability. Implement display_path to show paths relative to SnapDiff.config.root when inside it, and absolute otherwise.	lib/snap_diff/reporters/default.rb
Introduce threshold listing logic and supporting constants for failure messages.	Add THRESHOLDS constant to define the order and names of configurable limits relevant to comparisons. Implement thresholds helper to render applied threshold values or a clear message when none are configured. Ensure the metric block always includes a "judged against" line summarizing configuration context.	lib/snap_diff/reporters/default.rb
Update DSL tests to expect the new structured failure message format and relative artifact paths.	Adjust assert_image_not_changed expectations to match the new headline, metric, and artifact lines. Change expected paths to be relative to SnapDiff.config.root and omit non-existent heatmap artifacts. Preserve test intent that behavior is unchanged aside from message wording and artifact listing correctness.	test/unit/dsl_test.rb
Expand reporter tests to cover message structure, numeric interpretations, thresholds, artifact labeling, and path rules.	Replace previous metric presence checks with assertions against the new message fields and absence of leaked Ruby object signatures. Add tests verifying denominators for area and region interpretation, and that difference_level is expressed with its unit. Add tests ensuring thresholds are restated, artifacts are labeled and filtered by existence, and paths obey relative/absolute rules. Introduce helper methods to set SnapDiff.config.root and to build comparisons with configurable options.	test/unit/reporters/default_test.rb

Possibly linked issues

• #issue: PR implements issue items 3 and 4 by improving failure diagnostics, artifact labels, thresholds, and relative paths.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.