Infrastructure: Add native Storybook visual comparison reports for Windows, Win32, and macOS #4328

Goal

Give pull-request reviewers native before / after / diff evidence for selected components in Windows Fabric, Win32 Paper, and macOS Storybooks, identifying the component, story/state, theme, and endpoint without requiring a native build.

Parent: #4275.

Status - 2026-10-01

Open; comparison/report delivery is not implemented. The repository has native window/element capture, story discovery, correlated readiness, named WDIO cases, and artifact-uploading PR jobs. It does not yet have the opt-in visual cohort, reproducible base/candidate comparison, accepted noise settings, baseline lifecycle, static image-diff report, or reviewer-facing visual PR summary required here.

The original capture/pipeline delivery issues are now closed milestones. Their qualification follow-ups are #4348 (Windows/Win32 capture) and #4349 (CI authority, reliability, and policy); #4345 owns driver release qualification.

Policy remains advisory visual comparison, not a pixel-difference merge gate. Changed pixels are review evidence; unavailable/invalid capture is an explicit infrastructure/coverage result, never an unchanged-image pass.

Existing foundation and implementation decision

• Reuse Windows Graphics Capture on Windows/Win32 and ScreenCaptureKit on macOS, including exact target identity, element cropping, scale, and explicit capture errors. Do not reintroduce WinAppDriver, Appium, browser rendering, or whole-desktop capture scripts.
• Reuse the owned Storybook/WDIO lifecycle and authenticated story/native-root readiness. Keep comparison/reporting outside production component bundles and Storybook-specific selection outside the generic driver.
• The September 23 research recommends a small Node image comparator such as pixelmatch, PNG decoding, and a lightweight static report. This is a researched starting point, not a landed implementation or finalized decision.
• @wdio/visual-service compatibility with the Appium-free native desktop endpoint remains unproven. Adopt only after a bounded spike verifies the existing screenshot/session contract without DOM operations or a second automation stack.
• Playwright/Chromatic are workflow references, not proof of these three native desktop appearances.

Proposed scope

Opt-in cases and deterministic capture

• Start with Button, Checkbox, and Input, then expand through explicit story/test selectors. Support a selected component locally; changed-file filtering alone can miss shared token/theme changes.
• Capture a stable fixture/preview region via native testID/bounds, excluding chrome and unrelated desktop content without hiding focus rings, shadows, borders, overflow, or explicit secondary-window cases.
• Fix args/content, theme, locale, viewport, and scale; wait for authenticated readiness, fonts/assets/layout, and bounded consecutive stable captures rather than a fixed sleep alone.
• Document light/dark and representative disabled, checked, and keyboard-focused states, high-contrast coverage/exclusions, and required input/capture capabilities. Stabilize animation/caret/pointer only where justified without masking the feature under review.

Comparisons and baseline lifecycle

• Compare each endpoint only against its own reference, never cross-platform pixel equality.
• Initially pair captures of the exact PR base SHA and tested candidate SHA in the same controlled environment. Identify head versus synthetic merge commits; reuse immutable base artifacts only when case/environment/commit fingerprints match.
• Key images by endpoint/renderer, story/case/state, theme, viewport, and scale/DPI. Record OS/runner/font/runtime/helper/manifest/backend identities and comparator settings.
• Explicitly report added/removed/renamed cases, missing/stale/incompatible references, dimension changes, instability, capability skips, capture errors, and blank images. Preserve originals; do not resize to manufacture equality.
• Measure repeat-capture noise before approving tolerances. Keep color/antialiasing settings separate from changed-pixel count/percentage. Masks must be narrow, justified, and reviewable.
• Define baseline/environment ownership, retention, regeneration, and reviewed updates. A candidate cannot silently approve itself, and a missing baseline cannot be a match.

Reviewer-facing results and safe CI

• Produce baseline, candidate, and highlighted diff PNGs plus a self-contained static HTML report with side-by-side and overlay/toggle views, grouped by component/story/state/theme/endpoint.
• Publish a bounded job summary and one updateable PR comment/check linking to the report/artifact, clearly separating changes, coverage gaps, and infrastructure failures. Do not assume GitHub renders artifact HTML inline.
• Upload images/metadata on failure under ignored artifact directories; document retention, reproduction, runner ownership, and redaction.
• Preserve fork safety: no write credentials in candidate execution, trusted reporting code, metadata verified against the source run/PR/SHA, no execution of downloaded report scripts, and an explicit safe policy before untrusted code uses preauthorized persistent desktops.

Platform rollout

Prove same-commit noise and recognizable intentional changes on all three endpoints before publishing advisory reports. Semantic smoke alone does not complete this issue.

Acceptance criteria

• Record the comparator/report decision and any WDIO visual-service compatibility findings.
• Button, Checkbox, and Input produce native fixture comparisons on all three endpoints, including light/dark and a documented state matrix; every exclusion has a reason.
• A sample PR with intentional color, spacing, border, or typography changes yields recognizable before/after/diff evidence per endpoint, accessible from the PR without building the app.
• At least 10 repeated same-commit captures per initial case measure qualified-environment noise and justify settings without hiding intentional changes.
• Missing/new/removed cases, size changes, stale/incompatible baselines, unstable/blank captures, denied authority, and capture failures are explicit results, never unchanged passes.
• Baselines are reproducible from recorded SHAs/environment metadata; candidate output cannot silently bless itself, and accepted appearance changes have a reviewed update workflow.
• Failure artifacts, machine-readable metadata, bounded reviewer summaries, retention/runner/fork policy, and capability gaps are documented.
• Document adding a case, selecting a component locally, reproducing CI, interpreting diffs, and handling baseline/environment changes without regressing semantic tests or rendering.
• Pixel differences remain advisory under Infrastructure workstream #4275; no required visual gate has been introduced.

Dependencies and non-goals

Coordinate #4348/#4349/#4345 and the native catalog/state bugs #4344/#4346. A pilot can start while qualification proceeds, but all-platform completion requires qualified captures.

Not in scope: mobile/web testing, cross-platform pixel equality, replacing semantic tests, rebuilding capture backends, blanket baseline approval, or an initially exhaustive catalog/DPI matrix.

Sources

• Windows provider
• macOS capture
• Executable tests
• Native CI authority
• Pixelmatch, WDIO visual testing, Playwright comparisons, Chromatic native scope (September 23 research).