Skip to content

Two tests share a screenshot name, so a failure overwrites its own evidence #284

Description

@pftg

Two integration tests write the same screenshot name:

test "compare crops only when other part is not working" do
  assert_matches_screenshot("index-without-img-cropped", crop: "form", color_distance_limit: 40)

test "crop accepts css selector" do
  assert_matches_screenshot("index-without-img-cropped", crop: "form")

A differing screenshot is written to its baseline path by design. So when the first test
fails, the second re-captures the same name, matches, and leaves the file clean — and the
failure's own evidence is gone before artifacts upload.

Verified on a real failing run

Run 32740082766, artifact screenshots-selenium_headless-chunky_png-diffs. The job
reported a 26.06% difference; its uploaded .base.png, .png and the committed
baseline are all the same file:

5fccb56f4a47d41b3a8c8cdf38531da6  index-without-img-cropped.base.png   (artifact)
5fccb56f4a47d41b3a8c8cdf38531da6  index-without-img-cropped.png        (artifact)
5fccb56f4a47d41b3a8c8cdf38531da6  index-without-img-cropped.png        (committed baseline)

The job reported a difference and shipped an artifact showing none.

Why it matters beyond tidiness

Anyone downloading that artifact to debug the flake finds two identical PNGs and reasonably
concludes the failure was spurious. The proof was destroyed by a name collision, not by
absence of a bug — which is why #285 has survived undiagnosed.

Fix

Give the two tests distinct screenshot names. Then #285 can reproduce with real artifacts.

  • Distinct names for the two tests
  • Confirm a deliberately-failed run uploads a genuinely differing .png vs .base.png

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions