iOS: first-class widget and Live Activity testing through a system UI target

[thymikee]

iOS: first-class widget and Live Activity testing through a system UI target

Status update (2026-07-24): simulator prototype gate passed; spec corrected

A live probe on iPhone 16 simulator (iOS 26.2, Xcode 26.2) through the shipping CLI with zero code changes proved most of the raw capability already exists:

• agent-device open com.apple.springboard --platform ios opens a working session bound to SpringBoard. (Earlier static analysis predicted simctl launch would reject SpringBoard; it does not.)
• The entire widget-install workflow is drivable semantically: home snapshot (42 labeled nodes: icons, dock, page indicator) → longpress <x y> on empty space → edit mode (Edit/Done/per-icon DeleteButtons) → Edit menu cells (Add Widget/Customize/Edit Wallpaper/Edit Pages) → gallery search field (fillable) → size picker fully semantic (page 1 of 6, widget description text, Add Widget button) → widget installed and visible in the home tree → long-press installed widget → context menu (Edit Widget/Remove Widget, size cells) → removal confirmation alert, all via click 'label="…"'.
• A real system permission alert appeared mid-flow and was captured and dismissed semantically — system-modal handling composes with this workflow.
• The private AX bridge already works against SpringBoard unchanged — it engaged automatically as the fallback tier. The "adapt the bridge to SpringBoard" work item in the original architecture section is unnecessary.

One real capture gap found: the widget-gallery sheet trips the complex/slow-tree heuristic, capture falls back to the private-AX backend, and the search-result cells come back as unlabeled [other] nodes while the screenshot plainly shows the result row. That single step needed coordinates. First SpringBoard capture was ~6.8s; subsequent captures ~1.5s p95 (channel penalization then keeps captures on private-AX). This is the same family as the #758 depth-ladder/attribute-mapping work.

Spec corrections:

1. Voltra cannot build iOS home-screen widgets. Voltra's iOS support is Live Activities + Dynamic Island only; its widget support is Android (Glance). PR sections below are re-scoped accordingly.
2. The CLI has a dedicated home command; earlier drafts said press home, which is not valid syntax.
3. Cross-target ref rejection is smaller than specced: the single-frame ref model (ADR 0014) already expires all prior refs on every new snapshot, so misuse is mostly prevented by construction; what's needed is a target-provenance field and one new typed rejection reason, not a new mechanism.

Still unverified (remaining gate items): physical iPhone; Live Activity lock-screen/Dynamic Island semantics (needs the fixture first); app↔system alternation state retention (the probe used SpringBoard as the session app — exactly the ergonomic gap --ui-target closes).

The delivery plan below is reordered so value ships in the order it is unblocked.

Motivation

Agent Device can launch an app, go home, and take a full-screen screenshot, and — as verified above — can drive SpringBoard by opening it as the session app. What is missing is request-scoped system targeting that keeps the app session intact while individual commands target SpringBoard, plus capture quality in the widget gallery, plus a deterministic Live Activity fixture.

The goal is first-class support for testing widget and Live Activity experiences while preserving the distinction between:

• raw capability: target iOS system UI for snapshots, selectors, and interactions;
• workflow guidance: explain how an agent can add/edit a widget using the UI exposed by the current iOS version;
• visual verification: use screenshots for surfaces such as Dynamic Island where semantic AX coverage can be sparse.

Architecture decision

Use the existing XCTest runner and XCUIApplication(bundleIdentifier: "com.apple.springboard") as the production system-UI backend.

Do not add idb as a runtime dependency and do not make idb/CoreSimulator private APIs the contract:

• idb's ui describe-all is explicitly simulator-only;
• its current implementation calls CoreSimulator's private sendAccessibilityRequestAsync API and AccessibilityPlatformTranslation/AXPTranslator;
• that is a useful prototype oracle for SpringBoard hierarchy quality, but it cannot cover physical devices and adds another Xcode-private compatibility surface.

Agent Device already has a narrower private XCTest AX snapshot bridge in RunnerAXSnapshotBridge.m. It is fully parameterized on the target XCUIApplication and verified working against SpringBoard; keep that path simulator-only and fallback-only. Do not introduce idb or a second host-side hierarchy implementation.

Production priority:

1. Public XCTest APIs against SpringBoard on simulator and physical device.
2. Existing private AX bridge as a bounded simulator snapshot fallback (already functional for SpringBoard; needs the PR B label-quality fix below).
3. Screenshot plus coordinate interaction when the system surface does not expose useful accessibility nodes.

Proposed command contract

Add a request-scoped automation target rather than changing session ownership:

• CLI: --ui-target app|system
• Node/MCP input: uiTarget: "app" | "system"
• default: app

Do not reuse --target; it already describes the device family (mobile, tv, desktop). Do not store system as the session app bundle. The app session must remain intact while individual commands target SpringBoard.

Apply the option to semantic and interactive commands that need a UI root:

• snapshot, find, get, is, wait
• click, fill, press, longpress, scroll, gesture

Full-screen iOS screenshots already observe the display; they do not need a different capture backend. They should still compose correctly in a system-target workflow.

For uiTarget=system:

• resolve the runner application/root to com.apple.springboard;
• do not activate or relaunch the session app during command preflight;
• preserve the existing session app bundle and lifecycle;
• expose target provenance once per response, not once per node;
• bind snapshot refs to their UI target and reject an app ref used by a system command (and vice versa) with a typed, actionable error;
• include the target in recording/replay identity where needed so replay cannot silently cross roots.

Do not add a widget add command. Widget installation/editing remains ordinary system UI automation.

Delivery plan (reordered by what is unblocked)

PR A: bless the works-today SpringBoard workflow (help/skill only, no product code)

The workflow already works via open com.apple.springboard. Ship an iOS system-surfaces help topic now:

1. open the app and prepare widget/Live Activity state;
2. open com.apple.springboard --platform ios (or home first from an app session, then open SpringBoard);
3. snapshot -i to inspect the current localized SpringBoard controls;
4. long-press an empty coordinate to enter edit mode — the one deliberate coordinate step;
5. use selectors from the fresh snapshot to open the Edit menu → widget gallery → search → size picker → Add Widget → done;
6. long-press an installed widget and use the fresh hierarchy (Edit Widget/Remove Widget) to edit or remove it;
7. use screenshots for visual assertions and as the fallback when AX is sparse — today that means the gallery search-result rows (tap by coordinates from the screenshot until PR B lands);
8. reopen the app bundle to return to app automation.

The help must say that system labels vary by iOS version and locale. Agents should discover them from the current snapshot instead of relying on literal Edit/Done strings.

PR B: widget-gallery snapshot quality

Fix the one live-found capture gap: SpringBoard's gallery sheet must not lose cell labels when the complex/slow-tree fallback engages.

• Reproduce: SpringBoard session → edit mode → Edit menu → Add Widget → fill search. The delivered tree has the search field, Sheet Grabber, keyboard keys — but result cells are unlabeled [other] nodes.
• Suspect area: private-AX attribute mapping/depth budget for sheet-hosted collection views (add-sheet-collection-view), same family as the fix: add iOS private AX snapshot fallback #758 depth-ladder fix; also review whether the xctest-channel penalization keyed on the bundle id is too sticky for SpringBoard (first capture ~6.8s, then permanently private-AX).
• Acceptance: the searched widget result row is selector-addressable (click 'label="Calendar"'-style) on the supported simulator; snapshotQuality explains any remaining fallback.

PR C: system UI target and provenance (the --ui-target contract)

The seams are mapped; implementers should not rediscover them.

Swift runner (small — the seam exists):

• RunnerTests+Models.swift: add uiTarget to Command.
• RunnerTests+CommandExecution.swift: extend prepareActiveCommandContext(command:routeToSpringboard:) — a uiTarget == system branch mirrors the existing routeToSpringboard branch (activeApp = springboard, skip activation preflight, and do not mutate currentApp/currentBundleId session caches, which key retries/penalties). An explicit uiTarget: system should short-circuit the blocking-system-modal probe (probing SpringBoard for a modal is pointless when the caller already targets SpringBoard). Foreground guards / existence waits that assume "resolved app must be activatable" must be no-ops for the system root.
• RunnerTests.swift, RunnerAXSnapshotBridge.m/.h, snapshot/interaction files: no changes — already app-parameterized. There is a unit-test template: testPrepareActiveCommandContextRoutesBlockingSystemModalToSpringboard.
• Do not grow RunnerTests+CommandExecution.swift into another mixed-responsibility file; extract a focused application/root resolver if the change would otherwise branch throughout command execution.

TypeScript (~15–20 files, ~250–350 non-test LOC; the flag registry part is cheap, the cost is in these enumerated hand-maintained seams — each needs its one-line-or-so addition, and missing any of them is a silent-drop bug of the #1303 class):

• src/contracts/ new UiTarget type (mirror session-surface.ts) + cli-flags.ts + src/commands/cli-grammar/flag-definitions-target.ts + flag-groups.ts (one definition, one group).
• Descriptor allowedFlags splices: src/commands/capture/snapshot.ts, src/commands/capture/wait.ts, and the 8 command schemas in src/commands/interaction/index.ts.
• src/commands/command-flags.ts (buildFlags) and src/daemon/context.ts (contextFromFlags) — the flat compatibility mappers.
• src/client/client-types.ts: a UiTargetCommandOptions mixin spliced into the ~10 affected *Options types, plus the hand-rebuilding toXxxOptions() projectors in src/commands/interaction/index.ts (this is exactly where fix(replay): default-exclude observation-only reads from repair heals, add --record opt-in (#1271 stage 2) #1303-style drops happen).
• Root override + provenance: thread uiTarget through DispatchContext/RunnerContext/RunnerCommand (src/core/dispatch-context.ts, src/core/interactor-types.ts, src/platforms/apple/core/runner/runner-contract.ts, plus the appBundleId:-setting call sites in src/platforms/apple/interactor.ts/interactions.ts). Carry uiTarget alongside appBundleId, not as a bundle-id swap — the runner must distinguish "explicit target override" from "launched app" to skip activation preflight.
• Ref provenance: src/daemon/ref-frame.ts + src/daemon/types.ts (refFrameTarget), set at the two frame-publication sites (snapshot-runtime.ts, session-snapshot.ts), one new rejection reason in admitRefMutation/interaction-ref-policy.ts. Note the single-frame model already expires cross-snapshot refs; this adds the typed reason, not a new mechanism.
• Recording/replay identity: SANITIZED_FLAG_KEYS in src/daemon/session-action-recorder.ts, appendScriptSeriesFlags in src/replay/script-utils.ts, and the per-command parse blocks in src/replay/script.ts. (src/replay/target-identity.ts is element-binding identity — unrelated, do not touch.)
• ADR-0011: one new guarantee row in src/contracts/interaction-guarantees.ts (tsc forces all 6 path cells; only the click/fill/press/longpress intersection is in the matrix — the other command families are covered by dedicated contract scenarios instead).
• Platform gating: reject --ui-target system on non-iOS-family targets with a typed error. Preserve macOS surface (session-scoped, platform-disjoint) and Android behavior unchanged.

PR D: deterministic Live Activity fixture (+ optional widget shim)

Re-scoped after the Voltra correction. The repo test app (examples/test-app, Expo prebuild from app.config.js; no extension targets exist today) gets:

• Live Activity fixture via Voltra (@use-voltra/ios-client): one Live Activity with deterministic start, update, and end controls in the fixture UI; compact/minimal/expanded Dynamic Island presentations and a lock-screen presentation; the App Group/config the extension needs (Voltra's config plugin generates the extension target and App Group wiring at prebuild). Local ActivityKit start/update/end first; APNs remote updates are a later physical-device extension.
• Home-screen widget: not Voltra on iOS. Two options, decided in the PR: (a) a small authored SwiftUI WidgetKit extension (order ~50–100 LOC Swift, e.g. via @bacons/apple-targets or a hand-authored config plugin) with deterministic text and, where the deployment target permits, one configurable state and one interactive action; or (b) defer the custom widget — the system add/edit/remove workflow is fully testable with built-in Apple widgets (verified live with Calendar), and a custom widget is only required to assert app-specific widget content.
• Keep fixture UI compact and purpose-built for automation. Track generated/build dependency size separately from authored source size; measure app/extension binary and build-time deltas before/after and report in the PR body. Note the CI build cache (.github/workflows/test-app-build-cache.yml) keys on the Expo fingerprint — a new native target changes native inputs; verify the cache key still behaves.

PR E: final help + live evidence

• Update the PR A help topic for --ui-target (replacing the reopen-SpringBoard dance) and add Live Activity guidance: start/update/end in the app, then inspect SpringBoard/lock-screen semantics where available and capture screenshots for Dynamic Island.
• Focused SkillGym coverage for the widget and Live Activity workflows.
• Physical-device evidence (see gate below).

Prototype gate — remaining

The simulator portion of the original gate is done (see status update; evidence: iPhone 16 sim, iOS 26.2, Xcode 26.2, full command transcript with semantic snapshots at every step, screenshots of the gallery and widget context menu). The idb-oracle comparison is moot — the tree quality question is answered, and the one deficiency is precisely located (PR B). Remaining before the cross-device contract is claimed:

1. On a physical iPhone, prove at minimum a SpringBoard snapshot and one selector-driven interaction through public XCTest. Private AX/idb are not acceptable evidence there.
2. Live Activity semantics: with the PR D fixture, record what the OS exposes semantically on lock screen and Dynamic Island per state (compact/minimal/expanded); screenshots are first-class where AX is absent.
3. App↔system alternation: verify an app session's state survives interleaved system commands (this is what --ui-target adds over the works-today reopen workflow).

If normal XCTest cannot produce a useful SpringBoard hierarchy on physical iOS, stop and report the capability boundary before claiming a cross-device contract.

Acceptance criteria

• An app session remains open while app and system commands alternate; switching roots does not relaunch or lose app state.
• home followed by snapshot --ui-target system -i returns a useful SpringBoard tree on a supported simulator and physical iPhone.
• An agent can add a widget using a coordinate long-press plus selector-driven system interactions discovered from snapshots — including selector-addressable gallery search results (PR B).
• An agent can long-press the installed widget and enter its edit/configuration path when the widget exposes one (Edit Widget is already semantic today).
• App refs and system refs are not interchangeable; misuse returns a typed error and recovery hint.
• Snapshot, selector, recording/replay, timeout, warnings, and diagnostics behavior remains consistent with existing contracts.
• Live Activity start/update/end is deterministic from the fixture app.
• Lock-screen and Dynamic Island screenshots demonstrate each supported fixture state. Semantic assertions are required only where the OS exposes stable AX nodes.
• Live evidence includes simulator and physical-device results, exact OS/Xcode/device identifiers, commands, screenshots/artifacts, and any AX gaps.
• Required gates pass: affected checks, provider integration/coverage for response changes, interaction contracts, pnpm build:xcuitest, formatting, and live session cleanup.

Non-goals

• No direct WidgetKit database manipulation or private API to install a widget.
• No widget-specific orchestration command.
• No hard-coded English SpringBoard workflow.
• No idb dependency or host-side clone of idb's accessibility serializer.
• No promise that Dynamic Island content is fully semantic; visual evidence is first-class there.
• No lock/unlock/passcode automation contract in the first slice.
• No APNs server or remote push setup in the first slice.
• No Android widget fixture in this arc (Voltra Glance widgets could be a separate follow-up issue).

Risks to report, not hide

• SpringBoard AX shape and labels vary across iOS/Xcode versions and locale (verified evidence above is iOS 26.2/Xcode 26.2 only).
• The gallery-sheet complex-tree fallback currently loses result-cell labels (PR B); until fixed, that step is screenshot+coordinates.
• First SpringBoard capture is slow (~7s observed) and channel penalization then pins captures to private-AX; PR B should review whether that stickiness is right for SpringBoard.
• Private AX fallback is simulator-only and Xcode-private; selector availability must be feature-detected and failure must fall back cleanly.
• System UI may expose controls only after animations settle; use existing settle/timeout semantics rather than fixed sleeps (--settle worked throughout the live probe).
• Empty-space long-press is inherently coordinate-based; document it as the one deliberate coordinate step.
• Widget extension packaging can inflate checkout/build/app size; measure actual deltas before merge.

Source references for the architecture decision

• idb accessibility docs: website/docs/accessibility.mdx states these primitives are for iOS Simulators.
• idb implementation: FBSimulatorAccessibilityCommands.swift imports AccessibilityPlatformTranslation and calls CoreSimulator's private accessibility request selector.
• Agent Device existing fallback: RunnerAXSnapshotBridge.m uses XCTest's private accessibility interface, is compiled only for the iOS simulator by RunnerTests+AXSnapshotFallback.swift, and is verified working against SpringBoard as-is.
• Agent Device already owns a SpringBoard XCUIApplication in RunnerTests.swift and a root-resolution seam in RunnerTests+CommandExecution.swift (prepareActiveCommandContext(command:routeToSpringboard:)); extend that seam instead of adding another backend.
• Voltra platform support (README, verified 2026-07-24): iOS = Live Activities + Dynamic Island; Android = home-screen widgets (Glance).

[thymikee]

Issue body updated after a live simulator prototype run (2026-07-24, iPhone 16 sim / iOS 26.2 / Xcode 26.2). Summary of what changed, for anyone who read the earlier revision:

• Simulator prototype gate: passed, with zero code changes. open com.apple.springboard --platform ios works through the shipping CLI, and the full widget add/edit/remove workflow is selector-drivable end-to-end — except widget-gallery search results, which lose their labels when the complex-tree fallback engages. That gap is now its own work item (PR B).
• Voltra correction: Voltra's iOS support is Live Activities + Dynamic Island only; home-screen widgets are Android/Glance. The fixture PR (now PR D) is re-scoped: Voltra for the Live Activity, and either a small authored WidgetKit shim or built-in Apple widgets for the home-screen workflow.
• Delivery reordered into PRs A–E (help topic for the works-today workflow → gallery capture fix → --ui-target contract → fixture → final help/evidence), with an explicit seam map for PR C so implementers don't rediscover it: the runner already has the root-resolution seam (prepareActiveCommandContext's routeToSpringboard branch) and the private AX bridge works against SpringBoard as-is; the TS cost is ~6–8 enumerated hand-maintained seams (contextFromFlags, SANITIZED_FLAG_KEYS, replay script read/write, client option types, ref-frame provenance, ADR-0011 row).
• Remaining gate items before the cross-device contract is claimed: physical-iPhone SpringBoard snapshot + one selector interaction, Live Activity AX semantics (needs the fixture), and app↔system alternation state retention.
• Minor: the workflow command is home, not press home.