Two integration tests write the same screenshot name:
test "compare crops only when other part is not working" do
assert_matches_screenshot("index-without-img-cropped", crop: "form", color_distance_limit: 40)
test "crop accepts css selector" do
assert_matches_screenshot("index-without-img-cropped", crop: "form")
A differing screenshot is written to its baseline path by design. So when the first test
fails, the second re-captures the same name, matches, and leaves the file clean — and the
failure's own evidence is gone before artifacts upload.
Verified on a real failing run
Run 32740082766, artifact screenshots-selenium_headless-chunky_png-diffs. The job
reported a 26.06% difference; its uploaded .base.png, .png and the committed
baseline are all the same file:
5fccb56f4a47d41b3a8c8cdf38531da6 index-without-img-cropped.base.png (artifact)
5fccb56f4a47d41b3a8c8cdf38531da6 index-without-img-cropped.png (artifact)
5fccb56f4a47d41b3a8c8cdf38531da6 index-without-img-cropped.png (committed baseline)
The job reported a difference and shipped an artifact showing none.
Why it matters beyond tidiness
Anyone downloading that artifact to debug the flake finds two identical PNGs and reasonably
concludes the failure was spurious. The proof was destroyed by a name collision, not by
absence of a bug — which is why #285 has survived undiagnosed.
Fix
Give the two tests distinct screenshot names. Then #285 can reproduce with real artifacts.
Two integration tests write the same screenshot name:
A differing screenshot is written to its baseline path by design. So when the first test
fails, the second re-captures the same name, matches, and leaves the file clean — and the
failure's own evidence is gone before artifacts upload.
Verified on a real failing run
Run
32740082766, artifactscreenshots-selenium_headless-chunky_png-diffs. The jobreported a 26.06% difference; its uploaded
.base.png,.pngand the committedbaseline are all the same file:
The job reported a difference and shipped an artifact showing none.
Why it matters beyond tidiness
Anyone downloading that artifact to debug the flake finds two identical PNGs and reasonably
concludes the failure was spurious. The proof was destroyed by a name collision, not by
absence of a bug — which is why #285 has survived undiagnosed.
Fix
Give the two tests distinct screenshot names. Then #285 can reproduce with real artifacts.
.pngvs.base.png