Skip to content

ci(ios): a regular depth-1 snapshot carries XCTest tree quality metadata when the AX bridge probe circuit is open #3328

Description

@thymikee

Purpose

File the currently untracked iOS Smoke failure signature "regular depth-1 snapshot must not carry XCTest tree quality metadata" so #2491's lane report covers every signature that fails on main, and decide which side owns it: the acquisition fallback or the assertion.

Evidence (both on main)

  • Run 37145696803 (2026-10-03, d396f3b50), job Smoke Tests
  • Run 37459193672 (2026-10-06, 1163b7ac0), job Smoke Tests

Failing assertion, test/integration/ios-simulator-e2e/live-snapshot-depth-frontier.ts:126 (assertSimulatorBridgeSnapshot, reached from assertRegularVisibleDepthFrontier on the step capture regular visible-depth frontier):

AssertionError [ERR_ASSERTION]: regular depth-1 snapshot must not carry XCTest tree quality metadata
+ actual   'tree'
- expected undefined

The response it asserts on keeps the fallback disclosure on a regular (non-raw) snapshot:

"snapshotQuality": { "state": "healthy", "backend": "tree", "timing": { "acquisitionMs": 64.7, "presentationMs": 0.09 } },
"warnings": ["Simulator AX snapshot unavailable (circuit-disabled); used XCTest for this app generation."],
"snapshotDiagnostics": { "stats": { "backends": { "xctest": 53 }, "maxMs": 27409 } }

The same two runs' runner log shows the acquisition path that produced it:

AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_DEPTH_RETRY depth=64 error=Could not match active AX application for XCTest application
... depth=56, depth=40, depth=24, depth=12 ...
AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_FAILED=Could not match active AX application for XCTest application
AGENT_DEVICE_RUNNER_SNAPSHOT_BACKEND_FAILED backend=tree error=the XCTest tree capture exceeded its 8s time slice

So the bridge was circuit-disabled for that app generation, the capture fell back to the XCTest tree, and a regular snapshot carried the tree backend's quality metadata that the lane asserts must be absent.

Required behavior

  • State the contract explicitly: may a regular (non-raw) snapshot ever carry XCTest snapshotQuality/backend disclosure on the simulator? Today the runner and the assertion disagree.
  • If a bridge-disabled fallback is legitimate for a regular snapshot, the assertion must assert the disclosed fallback through a typed acquisition reason instead of demanding absence. If it is not legitimate, the fallback must not be reachable for a regular snapshot, and the owning seam is snapshot acquisition classification (refactor(apple-runner): classify AX snapshot failure once at acquisition #2972), not a call-site guard.
  • Key the decision on typed acquisition reasons and snapshotQuality state, never on the warning string.
  • Decide whether the combination "probe circuit open for the app generation" plus "XCTest tree capture exceeded its time slice" is a product defect or a lane condition ci(ios): the iOS simulator smoke lane is failing on main across several rotating signatures #2491 should classify explicitly.

Completion

Related

#2972 (classify AX snapshot failure once at acquisition), #2492 (bridge probe circuit follow-ups), #2491 (the lane report this belongs to).

Activity

  1. thymikee commented on Oct 8, 2026

    @thymikee
    MemberAuthor

    Ledger update from the #3336 human review (note 5): the new assertion in #3336 (98321a2) accepts a runner-fallback-served regular capture because the route's circuit-disabled/fallback reason is only a diagnostic event on the wire today. This issue stays open for exactly two items: (1) the live-run completion leg is now satisfied — the re-run 37843633972 @ 98321a2 passed smoke:regular-visible-depth-frontier live with the healthy/tree + circuit-disabled disclosure — and (2) OWEED: upgrading the assertion to key on a typed fallback reason once #2972 puts the route decision on the wire as typed data. Closing this needs #2972's wire typing, not just the assertion swap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions