From 67e1d04f907851a6eb8fb7fe21321cfaa4eb733f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 10 Jul 2026 09:02:58 +0200 Subject: [PATCH 1/7] docs: add ADR 0012 for interactive replay, resolution disclosure, and retiring --update healing Records the decision to retire --update healing as a silent actor (repurposing its candidate machinery as ranked suggestions), disclose selector disambiguation in every interaction response, verify replay steps against record-time identity evidence, and add an interactive replay --from loop with a structured divergence report for all callers. --- docs/adr/0012-interactive-replay.md | 235 ++++++++++++++++++++++++++++ docs/adr/README.md | 1 + 2 files changed, 236 insertions(+) create mode 100644 docs/adr/0012-interactive-replay.md diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md new file mode 100644 index 0000000000..cb408f682a --- /dev/null +++ b/docs/adr/0012-interactive-replay.md @@ -0,0 +1,235 @@ +# ADR 0012: Interactive Replay (agent-in-the-loop repair, resolution disclosure, retiring `--update` healing) + +## Status + +Proposed (2026-07-10). Nothing in this ADR is implemented yet. + +## Context + +Replay today is deterministic. `.ad` scripts are plain text — one action per line, `#` comments, a +`context platform=... device=... theme=...` header (`src/replay/script.ts`) — recorded via +`open --save-script` (`src/daemon/session-action-recorder.ts`, `src/daemon/session-script-writer.ts`) +or hand-written, and executed step-by-step by `runReplayScriptFile` +(`src/daemon/handlers/session-replay-runtime.ts`) under the daemon's `replay`/`test` commands +(`src/daemon/handlers/session-replay.ts`). Recorded touch/fill/get targets are selector chains with +`||` alternates (`buildSelectorChainForNode(...).join(' || ')`, +`src/commands/interaction/runtime/resolution.ts:242`, mirrored in +`src/daemon/handlers/session-replay-heal.ts:131-135`); Maestro YAML flows import through `--maestro` +(`src/compat/maestro/`); progress is step-indexed (`stepIndex`/`stepTotal` in +`emitReplayTestActionProgress`, `session-replay-runtime.ts:243-260`). + +Recovery is opt-in `--update`/`-u` healing (`replayUpdate` flag, +`src/cli/parser/cli-flags.ts:1041-1047`). It only fires after a step has already returned a hard +failure (`session-replay-runtime.ts:118-149`: `if (!shouldUpdate) return failure; ... +healReplayAction(...)`), and it only retries the SAME recorded selector material — +`collectReplaySelectorCandidates` (`session-replay-heal.ts:39-81`) gathers the step's originally +recorded `selectorChain`/positionals, then `resolveSelectorChain` re-resolves those exact candidate +strings against a freshly captured snapshot (`session-replay-heal.ts:122-135`). If the identifying term +itself changed — an id or label rename — the same string will not match the new tree either, so heal +cannot rescue renames; it can only recover drift the ORIGINAL selector still matches (a moved or +re-rendered node with the same id). PR #297 (closing #279) already trimmed heal once, removing +`refLabel`-synthesis and numeric `get text` drift healing to keep it "centered on recorded selectors and +explicit selector expressions" — heal has a maintained history of narrowing, not growing. + +**Benchmark evidence** (2026-07-09/10, `~/.agent-device-bench/rnnav-matrix.py`, external harness): the +`--settle` quiet-window loop is now at its 1-snapshot floor, so wall time for a QA flow is dominated by +model turn latency, not device I/O. A happy-path agent-driven QA flow costs O(steps) model turns +end-to-end; a deterministic replay of the same flow costs O(divergences). The entire economic case for +replay is collapsing the per-step model-turn cost toward zero on the happy path and paying only where +reality diverged from the recording. + +**Audit evidence** (2026-07-10) on where that divergence cost actually goes: + +- **(a) Heal is narrow and mostly unable to act.** Per the mechanism above, heal only recovers + same-selector drift. Most real replay failures are renames or removals heal's candidate-recycling + cannot reach. +- **(b) The real mis-binding surface is not heal — it is silent disambiguation in ORDINARY resolution**, + live and replay alike. `resolveSelectorInteractionTarget` calls `resolveSelectorChain(..., { + disambiguateAmbiguous: true })` on every press/click/fill (`resolution.ts:170-183`); when a selector + matches N>1 nodes, `accumulateDisambiguationCandidate`/`compareDisambiguationCandidates` + (`src/daemon/selectors-resolve.ts:153-204`) silently pick a winner — visible candidates over + off-screen ones, then deepest node, then smallest on-screen area, only an exact tie failing. + `describeResolvedInteractionNode` (`resolution.ts:227-249`), the response's entire identity payload, + carries `node`/`selectorChain`/`refLabel`/`targetHittable`/`hint` — no match count, no signal a + tiebreak happened at all. This was live-reproduced during the audit on an RN playground screen with + two identical-rect "Prevent Remove" buttons, where scroll position alone decided which one a selector + hit. The general policy is documented (`agent-device help workflow`, + `src/cli/parser/cli-help.ts:243,384`: "does not fail by default ... auto-resolves deepest node first + ... then smallest on-screen area") but never disclosed per response — an agent that hasn't read the + help topic, or whose target moved between recording and replay, gets no signal a heuristic rather than + an exact match chose its target. +- **(c) No outcome verification exists anywhere in this path.** `--verify` + (`captureEvidenceBaseline`, `resolution.ts:45-58,104-134`; the `verifyEvidence` guarantee cell in ADR + 0011's registry) attaches a pre/post-action node diff so the caller can see SOMETHING changed — it + says nothing about whether the CORRECT node was the one tapped. A wrong-but-plausible pick (the + sibling "Prevent Remove" button) produces a real, visible diff and is still the wrong action. +- **(d) Heal auditability is a bare count.** A successful `--update` run returns + `{ replayed, healed, ... }` (`session-replay-runtime.ts:186-195`) — `healed` is a number, nothing + else — and rewrites the `.ad` file in place via `writeReplayScript` + (`session-replay-runtime.ts:182-184`, `src/replay/script.ts:459-484`) with no diff shown anywhere in + the response. +- **(e) This silent-pick default is in real tension with this repo's general posture toward ambiguity.** + Elsewhere, ambiguous input is refused and hinted about rather than silently guessed — `start`/`restart` + are deliberately left out of the CLI alias-suggestion table because `start` is "genuinely ambiguous, so + a hint beats silently guessing" (`src/cli/parser/command-suggestions.ts:16-17`). Selector resolution + took the opposite default, and ADR 0011's own registry records that choice precisely: the + `disambiguation` cell for `runtime-selector` is classified `{ kind: 'runtime', via: + '...selectors-resolve.ts#resolveSelectorChain' }` (`src/contracts/interaction-guarantees.ts:176-179`) + — proving the heuristic runs consistently across paths, not that the caller is told it ran. That + default is not being revisited here; see the rejected hard-reject alternative below for why. +- **(f) Issue #1037 / PR #1040 is the direct, partial precedent.** A UNIQUE-but-wrong match (Apple + Maps' `text="Anthropic - Headquarters"` exact-matching a 30x30 map-pin annotation instead of the + recents row) now surfaces as `targetHittable:false` plus a hint + (`describeNonHittableTarget`, `resolution.ts:259-268`) — disclosed, but not prevented; the tap still + lands on the wrong element, just no longer silently. Disambiguation (N>1 matches, as opposed to one + unique-but-non-hittable match) has no equivalent disclosure today. +- **(g) Issues #279/#297 are precedent for trimming heal rather than growing it** when the evidence + says a heuristic isn't earning its complexity — see above. + +A related, currently under-used precedent: recorded `@ref` steps already carry an optional identity +hint in the `.ad` file. `appendRefLabel` (`src/daemon/session-script-writer.ts:235-240`) writes the +node's label as a trailing token, parsed back into `action.result.refLabel` +(`src/replay/script.ts:269,295,315`). Today that label is used only as a fallback LOOKUP key +(`tryResolveRefNode`'s `fallbackLabel`, `resolution.ts:393,413-430`) when the ref itself fails to +resolve, and to scope the pre-action snapshot capture (`buildScopedSnapshotAction`, +`session-script-writer.ts:136-155`) — never as a check against what disambiguation actually picked. It +establishes the pattern this ADR's decision 3 extends into a verification role: per-step identity +already travels in the `.ad` file. + +## Decision + +### 1. Retire `--update` healing as an actor; repurpose its candidate machinery as ranked suggestions + +`--update`/`-u` stops silently rewriting `.ad` files. The two pieces of machinery it already has — +`collectReplaySelectorCandidates` (recorded-chain/positional extraction) and the `resolveSelectorChain` +re-resolution it drives — are repurposed to populate a ranked list of selector suggestions inside the +divergence report (decision 4), not to act unattended. With an agent in the loop, adjudicating a heal +proposal costs one cheap model turn — cheaper than discovering a silent wrong repair later — and the +audit ((a) above) already found heal rarely able to act. A proposal an agent can accept, reject, or edit +is strictly more valuable than the same proposal applied blind. + +### 2. Disclose disambiguation in every interaction response, live and replay + +When a selector's resolution matched N>1 candidates and a heuristic (not a unique match) chose the +winner, the response — press/click/fill/longpress, live or replayed — carries the match count, the +chosen node's ref, the tiebreak reason (visible / deepest / smallest-area), and a capped list of the +other candidates' refs. This follows the #1040 precedent exactly (disclose, don't change resolution) and +slots into the single existing response-construction site (`buildInteractionResponseData`, ADR 0011 +Layer 2, `src/daemon/handlers/interaction-touch-response.ts`) so it cannot be dropped by a hand-rolled +branch the way `evidence` once was (#1064). It is **not** a behavior change to +`resolveSelectorChain`/`accumulateDisambiguationCandidate` — same heuristic, same winner, same policy +already documented in `help workflow`. Only the caller's visibility into the decision changes. + +### 3. Record-time identity verification for replay + +`open --save-script` recording captures the winning node's identity evidence (id/role/label/rect) for +each step, not just the label hint `refLabel` already carries. At replay time, when disambiguation +(decision 2) selects a node, its identity is compared against the recorded evidence for that step. A +mismatch is a **divergence** — reported exactly like a hard failure (decision 4) — not a silent success, +even though the command itself "succeeded" (tapped something, got a result back). This is outcome +verification via recorded ground truth, and it is what actually catches the "Prevent Remove" class of +bug: the command doesn't error, so nothing else would flag it. + +### 4. Interactive replay loop + +**(a)** One new flag, `replay --from ` — a range selector on an existing script, not a new +mode or command. `--from` starts execution at step N and never re-runs steps `1..N-1`. + +**(b)** On step failure — a hard failure OR a decision-3 identity divergence — the response, for ALL +callers (no agent-only mode), becomes a structured divergence report: + +- step index and the `.ad` source line — both already tracked (`actionLines`/`index` in + `runReplayScriptFile`, `session-replay-runtime.ts:98-131`, and rendered into + `withReplayFailureContext`'s `details`, `session-replay-runtime.ts:340-369`); +- the failing command and its error; +- current screen evidence with actionable refs. Minted refs must be blessed into the session the same + way settle refs are: `replay`/`test` are today in neither `REF_ISSUING_TOOLS` nor + `SETTLE_REF_ISSUING_TOOLS` (`src/mcp/command-tools.ts:114,125-129`), so any ref a failure response + handed back today would pass through unpinned at the MCP layer and fall to the coarse + `STALE_SNAPSHOT_REFS_WARNING` floor at best (`src/daemon/session-snapshot.ts:11-12,104-116`). The + divergence report must instead be an issuing response — carrying a `refsGeneration` the way + `settle.refsGeneration` does (`interaction-touch-response.ts:64-73,131-140`) and consumed by the same + merge-only pin bookkeeping settle uses (`mergeIssuedRefPins`/`mergeSettleIssuedRefPins`, + `src/mcp/command-tools.ts:167-217`) — so the agent's very next command can use a ref this report just + handed it at full precision, not the coarse warning floor; +- ranked selector suggestions from decision 1's retired heal machinery; +- when decision 3 applies, the recorded-vs-observed identity mismatch. + +**(c)** The loop protocol — run, read the report, then either fix reality and `--from N`, or perform the +step manually and `--from N+1`, or edit the plain-text `.ad` file and `--from N` — is documented as a +help topic (`agent-device help ...`, alongside the existing disambiguation-policy paragraph in +`src/cli/parser/cli-help.ts`). Loop until exit 0. There is no agent-mode flag: deterministic/CI callers +get the same richer failure output as an interactive agent — they simply don't act on the suggestions, +the same way they already ignore `hint` strings today. + +### 5. Validation + +Extend the settle benchmark (`~/.agent-device-bench/rnnav-matrix.py` pattern, external harness) with a +replay arm: author a script from one session, then measure (i) clean-replay cost — should be 0 model +turns, matching the O(divergences) claim — and (ii) induced-divergence repair cost — break one selector +deliberately, measure agent turns to green through the `--from` loop. + +## Consequences + +- `--from` resumability makes app-state **preconditions the caller's responsibility**. The daemon has + no way to know the app is actually in the state step N expects; the divergence report hands over + current screen evidence specifically so the caller can check that before resuming, but nothing + enforces it. +- **Non-idempotent scripts are exactly why `--from` must never re-run steps `1..N-1`**: a script that + creates a record, navigates, then asserts on it would double-create on any re-run of its early steps. + This is a hard constraint on the flag's semantics, not an implementation nicety. +- **Retiring heal-as-actor removes CI self-repair for agentless callers.** A nightly `test --update` run + that used to silently patch a renamed selector and go green now stays red with a suggestion in the + divergence report nobody reads. This is a real regression for that use case, accepted because the + audit found heal rarely able to act anyway (rename is exactly the case it cannot rescue), and a + silently patched selector was already a correctness risk of the kind decision 3 closes — "recorded and + current selector agree" is not the same claim as "they agree on the right element." +- **Disclosure adds bytes to every response where resolution was ambiguous.** A capped candidate list + keeps this bounded, but it is a token-cost increase on exactly the interactions that were already + hardest to get right, not a free win. +- **Recorded identity evidence adds bytes to every `.ad` file**, proportional to steps that target + selectors (ref-only steps already carry `refLabel` at similar cost today). `.ad` stays plain text; + this is per-line growth, not a format change. +- **The disambiguation heuristic itself (visible → deepest → smallest-area) is unchanged.** The + rejected alternative was hard-reject: fail any non-unique match instead of picking one. That would + break benign, common cases the heuristic exists for — react-navigation's Maestro suite alone has 185 + `tapOn`s on short/duplicated labels (`'Albums'` x9, `'Go back'` x16, per #1040) that resolve correctly + only because deepest/smallest-area picks the leaf button over its ancestor row/tab. Disclosure was + chosen over rejection because the cost is asymmetric: most ambiguous matches are benign + (tab+header+row sharing a label) and disclosure is nearly free for those, while rejection would fail + all of them to catch the rare "Prevent Remove"-style case decision 3 is built to catch structurally + instead. + +## Alternatives considered + +- **Guarded sequences as a new batch engine**: rejected — replay already is a step-sequenced engine + with progress and failure reporting; the gap is disclosure/verification/resumability on the existing + engine, not a second one. +- **An `--agent`/agent-mode flag on `replay`**: rejected — no semantic fork is needed once the + divergence report is simply the richer default failure shape. A deterministic CI caller does not need + protection from a richer error payload it can ignore. +- **Keep `--update` auto-heal, add outcome verification to it**: rejected for now — decision 3's + recorded-identity check subsumes what a verified auto-heal would buy (a heal that knows it healed + correctly) more simply, without heal's own retry-and-rewrite complexity. Revisit if the agentless-CI + regression noted in Consequences proves costlier than expected. +- **Auto-heal tiers** (safe-tier heals applied automatically, risky-tier surfaced): deferred, not + rejected outright — there is no current evidence base for which heals are "safe," and tiering now + would be speculative. Revisit if agentless CI demand for some self-repair materializes. + +## Migration plan + +Each step lands independently useful, in order: + +1. **Resolution disclosure** (decision 2) — additive fields on the existing single response-construction + site, no flag, no format change, immediately useful for live commands. +2. **`.ad` recorded identity evidence** (decision 3, recording side only) — the format change and the + replay-side comparison land separately so each is independently reviewable; recorded evidence stays + inert (captured but unchecked) until step 3. +3. **`replay --from` + the structured divergence report** (decision 4) — the report is what `--from` + resumes from, so they land together; wires in decision 3's comparison and decision 1's + retired-heal suggestions at the same time. +4. **`--update` retirement** (decision 1, the removal of its rewrite path) — lands once step 3's + suggestions are a proven substitute, not before, so there is no gap where healing regresses with + nothing in its place. +5. **Benchmark extension** (decision 5) validates 1–4 against the O(divergences) claim before this + ADR's economics are treated as proven rather than designed-for. diff --git a/docs/adr/README.md b/docs/adr/README.md index e8bbedc584..47ba47f2f1 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -13,6 +13,7 @@ | [0009 Apple Platform Consolidation](0009-apple-platform-consolidation.md) | Apple platform family, apple/appleOs axes, the apple-leak guard | | [0010 Error system conventions](0010-error-system.md) | error codes, hints, normalizeError, typed error signals | | [0011 Interaction Guarantee Contract](0011-interaction-guarantee-contract.md) | interaction dispatch paths, fast paths, guards, the guarantee matrix, parity tables | +| [0012 Interactive Replay](0012-interactive-replay.md) | replay healing/`--update`, selector disambiguation disclosure, `.ad` recorded identity, `replay --from` and the divergence report | ADRs record *why*; the registries and gates they describe are the living source of truth — when prose and a registry disagree, the registry wins and the ADR needs a follow-up. From ccf752b1e77d5eed9ef19d46c1d0cdf878d40960 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 10 Jul 2026 09:10:43 +0200 Subject: [PATCH 2/7] docs(adr-0012): ground in live replay evidence; require step provenance for --from Adds hands-on evidence from driving replay on the RN playground (silent text-mode success, app-state divergence heal cannot fix, Maestro step-index shift from runFlow flattening, per-format hint/code inconsistency, recordings carrying zero observation steps), makes step provenance (source file + line, including through Maestro runFlow inlining) a requirement of the divergence report plus an optional replay --list-steps dry-run, and adds a one-line text-mode success summary as decision 4d. --- docs/adr/0012-interactive-replay.md | 81 +++++++++++++++++++++++++++-- 1 file changed, 76 insertions(+), 5 deletions(-) diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md index cb408f682a..40b555ad0d 100644 --- a/docs/adr/0012-interactive-replay.md +++ b/docs/adr/0012-interactive-replay.md @@ -86,6 +86,53 @@ reality diverged from the recording. - **(g) Issues #279/#297 are precedent for trimming heal rather than growing it** when the evidence says a heuristic isn't earning its complexity — see above. +**Live hands-on evidence** (2026-07-10, driving replay by hand on the RN playground, iOS simulator, +both `.ad` and Maestro paths) grounds the same conclusions from the caller's seat: + +- **Successful replay is silent in text mode.** Exit 0, zero output; `replayed: 5` appears only under + `--json`. Structurally: replay's success payload (`{ replayed, healed, session, artifactPaths }`, + `session-replay-runtime.ts:186-195`) has no `message` field, so the generic CLI success path prints + nothing (`writeGenericCliOutput` → `readCommandMessage` → `writeCommandOutput`, + `src/cli/commands/generic.ts:68-71`, `src/utils/success-text.ts:12-14`, + `src/cli/commands/shared.ts:4-15`). An agent pays a verification turn just to learn what happened. +- **Failure output today is step + action + selector + a generic hint — no screen evidence.** The live + divergence hit was pure app state: the RN example app persists navigation state, so relaunch+deeplink + restored the Article screen and a perfectly correct selector legitimately missed. Heal can never fix + that class (the selector isn't wrong; reality is), while one line of screen evidence ("current + screen: Article") would have made the repair instant. The only recovery available was a full re-run — + no `--from` — and re-running earlier steps is precisely what makes state-restoring apps + nondeterministic across attempts. +- **Maestro step indices are untraceable to source today.** Breaking `tapOn: Push Input` — the 4th + top-level YAML step — failed as "Replay failed at step 5 (`__maestroTapOn` ...)": the flow's + `runFlow file: ../launch.yml` include had expanded into the linear plan and shifted every subsequent + index, and no file or line appears anywhere in the failure. Code-verified: `--maestro` input flattens + at parse time (`parseReplayInput`, `src/compat/replay-input.ts:47-68`) — `runFlow file:` inlines the + included file's actions (`convertRunFlow`/`readRunFlowActions`, + `src/compat/maestro/flow-control.ts:40-41,123-124`, via `parseRunFlowFile`, + `src/compat/maestro/replay-flow.ts:267-280`), platform/`true` `when` conditions are evaluated at + parse time (`flow-control.ts:47-48`), and `repeat.times` expands deterministically + (`flow-control.ts:84-87`). Provenance is lost in two stages: every action converted from one root + command inherits that PARENT command's YAML line (`convertRootCommands`, `replay-flow.ts:76-83`), + and `parseRunFlowFile`'s callers keep only `.actions`, discarding the included file's own line table + and path entirely. Even for `.ad`, the tracked line never reaches the caller: `actionLines` flows + into the per-action ndjson trace (`appendReplayTraceEvent`, + `src/daemon/handlers/session-replay-action-runtime.ts:47-56`) but `withReplayFailureContext` + (`session-replay-runtime.ts:349-369`) puts only `replayPath` + `step` in the error details. +- **The same failure class reports differently per format.** An `.ad` selector miss is + `COMMAND_FAILED` with the targeted hint "Run snapshot -i ... or use find ..." + (`selectorFailureHint`, `src/daemon/selectors-resolve.ts:84-97`, thrown at `resolution.ts:199-203`); + the equivalent Maestro miss is `ELEMENT_NOT_FOUND` constructed with no hint + (`src/compat/maestro/runtime-interactions.ts:644-652`), falling through to the generic default + "Retry with --debug and inspect diagnostics log for details." (`defaultHintForCode`, + `src/kernel/errors.ts:253-254`). +- **Recordings contain zero verification steps.** The script writer strips every recorded `snapshot` + action (`buildOptimizedActions`, `src/daemon/session-script-writer.ts:69`: `if (action.command === + 'snapshot') continue;` — only synthetic ref-scoped snapshots are re-inserted, as resolution aids, not + observations), and the record-time flag allowlist (`SANITIZED_FLAG_KEYS`, + `src/daemon/session-action-recorder.ts:46-77`) carries neither `settle`/`settleQuietMs` nor `verify`, + so `--settle`/`--verify` are dropped from recorded steps. A recording therefore replays actions with + no outcome observation at all — exactly the gap decision 3's record-time identity evidence fills. + A related, currently under-used precedent: recorded `@ref` steps already carry an optional identity hint in the `.ad` file. `appendRefLabel` (`src/daemon/session-script-writer.ts:235-240`) writes the node's label as a trailing token, parsed back into `action.result.refLabel` @@ -138,9 +185,23 @@ mode or command. `--from` starts execution at step N and never re-runs steps `1. **(b)** On step failure — a hard failure OR a decision-3 identity divergence — the response, for ALL callers (no agent-only mode), becomes a structured divergence report: -- step index and the `.ad` source line — both already tracked (`actionLines`/`index` in - `runReplayScriptFile`, `session-replay-runtime.ts:98-131`, and rendered into - `withReplayFailureContext`'s `details`, `session-replay-runtime.ts:340-369`); +- **step provenance: step index AND the source file + line of the failing step.** For `.ad` this is + mostly plumbing: `actionLines` is already tracked per step (`runReplayScriptFile`, + `session-replay-runtime.ts:98-131`) and already reaches the ndjson trace, but + `withReplayFailureContext` (`session-replay-runtime.ts:349-369`) currently renders only + `replayPath` + `step` — the line must be added to the report. For Maestro this is a requirement on + the parse: as the live evidence shows, includes and conditionals flatten into the linear `actions[]` + at parse time, so a failing step's index is meaningless against the YAML the caller is editing. + The Maestro parse must carry per-step source positions (file + line) through `runFlow` inlining — + today `convertRootCommands` (`replay-flow.ts:76-83`) assigns the parent entry's line to every + expanded action and `parseRunFlowFile`'s callers (`flow-control.ts:41,124`) discard the included + file's path and line table. Without this, `--from` is unusable on Maestro flows. Index determinism + makes this sufficient: platform/`true` `when` blocks and includes flatten at parse time + (`flow-control.ts:47-48`), `repeat.times` expands deterministically (`flow-control.ts:84-87`), and + visible/notVisible `when` becomes a single runtime control step (`wrapRunFlowCondition`, + `flow-control.ts:411-430`), so step indices are stable per platform and `--from N` re-targets the + same step the report named. Optionally, a `replay --list-steps` dry-run prints the flattened plan + with per-step provenance so a caller can map indices to source before running anything; - the failing command and its error; - current screen evidence with actionable refs. Minted refs must be blessed into the session the same way settle refs are: `replay`/`test` are today in neither `REF_ISSUING_TOOLS` nor @@ -162,6 +223,12 @@ help topic (`agent-device help ...`, alongside the existing disambiguation-polic get the same richer failure output as an interactive agent — they simply don't act on the suggestions, the same way they already ignore `hint` strings today. +**(d)** The success path stops being silent for text callers: a successful replay prints a one-line +summary (replayed N, wall time). Today text mode emits nothing on success (see the live evidence in +Context — the success payload has no `message`, so the generic renderer prints zero bytes), which +costs an agent a verification turn just to confirm the run happened. One line closes that for free; +`--json` output is unchanged. + ### 5. Validation Extend the settle benchmark (`~/.agent-device-bench/rnnav-matrix.py` pattern, external harness) with a @@ -174,7 +241,8 @@ deliberately, measure agent turns to green through the `--from` loop. - `--from` resumability makes app-state **preconditions the caller's responsibility**. The daemon has no way to know the app is actually in the state step N expects; the divergence report hands over current screen evidence specifically so the caller can check that before resuming, but nothing - enforces it. + enforces it. The live nav-state-persistence divergence in Context is the canonical case: the app, + not the script, decides what screen a relaunch lands on. - **Non-idempotent scripts are exactly why `--from` must never re-run steps `1..N-1`**: a script that creates a record, navigates, then asserts on it would double-create on any re-run of its early steps. This is a hard constraint on the flag's semantics, not an implementation nicety. @@ -227,7 +295,10 @@ Each step lands independently useful, in order: inert (captured but unchecked) until step 3. 3. **`replay --from` + the structured divergence report** (decision 4) — the report is what `--from` resumes from, so they land together; wires in decision 3's comparison and decision 1's - retired-heal suggestions at the same time. + retired-heal suggestions at the same time. Step provenance (the `.ad` line in the report and + Maestro per-step source positions) is part of this step, not an optional follow-up — without it + `--from` is unusable on Maestro flows. The one-line success summary (decision 4d) can land any + time, independently. 4. **`--update` retirement** (decision 1, the removal of its rewrite path) — lands once step 3's suggestions are a proven substitute, not before, so there is no gap where healing regresses with nothing in its place. From fe8d79801c8922d0b79603eb2d89aeeeea092c4d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 10 Jul 2026 09:25:36 +0200 Subject: [PATCH 3/7] docs: make interactive replay ADR implementable --- docs/adr/0012-interactive-replay.md | 267 ++++++++++++++++------------ docs/adr/README.md | 2 +- 2 files changed, 152 insertions(+), 117 deletions(-) diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md index 40b555ad0d..162174ae4d 100644 --- a/docs/adr/0012-interactive-replay.md +++ b/docs/adr/0012-interactive-replay.md @@ -58,7 +58,7 @@ reality diverged from the recording. ... then smallest on-screen area") but never disclosed per response — an agent that hasn't read the help topic, or whose target moved between recording and replay, gets no signal a heuristic rather than an exact match chose its target. -- **(c) No outcome verification exists anywhere in this path.** `--verify` +- **(c) No target-binding verification exists anywhere in this path.** `--verify` (`captureEvidenceBaseline`, `resolution.ts:45-58,104-134`; the `verifyEvidence` guarantee cell in ADR 0011's registry) attaches a pre/post-action node diff so the caller can see SOMETHING changed — it says nothing about whether the CORRECT node was the one tapped. A wrong-but-plausible pick (the @@ -155,109 +155,151 @@ proposal costs one cheap model turn — cheaper than discovering a silent wrong audit ((a) above) already found heal rarely able to act. A proposal an agent can accept, reject, or edit is strictly more valuable than the same proposal applied blind. -### 2. Disclose disambiguation in every interaction response, live and replay - -When a selector's resolution matched N>1 candidates and a heuristic (not a unique match) chose the -winner, the response — press/click/fill/longpress, live or replayed — carries the match count, the -chosen node's ref, the tiebreak reason (visible / deepest / smallest-area), and a capped list of the -other candidates' refs. This follows the #1040 precedent exactly (disclose, don't change resolution) and -slots into the single existing response-construction site (`buildInteractionResponseData`, ADR 0011 -Layer 2, `src/daemon/handlers/interaction-touch-response.ts`) so it cannot be dropped by a hand-rolled -branch the way `evidence` once was (#1064). It is **not** a behavior change to -`resolveSelectorChain`/`accumulateDisambiguationCandidate` — same heuristic, same winner, same policy -already documented in `help workflow`. Only the caller's visibility into the decision changes. - -### 3. Record-time identity verification for replay - -`open --save-script` recording captures the winning node's identity evidence (id/role/label/rect) for -each step, not just the label hint `refLabel` already carries. At replay time, when disambiguation -(decision 2) selects a node, its identity is compared against the recorded evidence for that step. A -mismatch is a **divergence** — reported exactly like a hard failure (decision 4) — not a silent success, -even though the command itself "succeeded" (tapped something, got a result back). This is outcome -verification via recorded ground truth, and it is what actually catches the "Prevent Remove" class of -bug: the command doesn't error, so nothing else would flag it. - -### 4. Interactive replay loop - -**(a)** One new flag, `replay --from ` — a range selector on an existing script, not a new -mode or command. `--from` starts execution at step N and never re-runs steps `1..N-1`. - -**(b)** On step failure — a hard failure OR a decision-3 identity divergence — the response, for ALL -callers (no agent-only mode), becomes a structured divergence report: - -- **step provenance: step index AND the source file + line of the failing step.** For `.ad` this is - mostly plumbing: `actionLines` is already tracked per step (`runReplayScriptFile`, - `session-replay-runtime.ts:98-131`) and already reaches the ndjson trace, but - `withReplayFailureContext` (`session-replay-runtime.ts:349-369`) currently renders only - `replayPath` + `step` — the line must be added to the report. For Maestro this is a requirement on - the parse: as the live evidence shows, includes and conditionals flatten into the linear `actions[]` - at parse time, so a failing step's index is meaningless against the YAML the caller is editing. - The Maestro parse must carry per-step source positions (file + line) through `runFlow` inlining — - today `convertRootCommands` (`replay-flow.ts:76-83`) assigns the parent entry's line to every - expanded action and `parseRunFlowFile`'s callers (`flow-control.ts:41,124`) discard the included - file's path and line table. Without this, `--from` is unusable on Maestro flows. Index determinism - makes this sufficient: platform/`true` `when` blocks and includes flatten at parse time - (`flow-control.ts:47-48`), `repeat.times` expands deterministically (`flow-control.ts:84-87`), and - visible/notVisible `when` becomes a single runtime control step (`wrapRunFlowCondition`, - `flow-control.ts:411-430`), so step indices are stable per platform and `--from N` re-targets the - same step the report named. Optionally, a `replay --list-steps` dry-run prints the flattened plan - with per-step provenance so a caller can map indices to source before running anything; -- the failing command and its error; -- current screen evidence with actionable refs. Minted refs must be blessed into the session the same - way settle refs are: `replay`/`test` are today in neither `REF_ISSUING_TOOLS` nor - `SETTLE_REF_ISSUING_TOOLS` (`src/mcp/command-tools.ts:114,125-129`), so any ref a failure response - handed back today would pass through unpinned at the MCP layer and fall to the coarse - `STALE_SNAPSHOT_REFS_WARNING` floor at best (`src/daemon/session-snapshot.ts:11-12,104-116`). The - divergence report must instead be an issuing response — carrying a `refsGeneration` the way - `settle.refsGeneration` does (`interaction-touch-response.ts:64-73,131-140`) and consumed by the same - merge-only pin bookkeeping settle uses (`mergeIssuedRefPins`/`mergeSettleIssuedRefPins`, - `src/mcp/command-tools.ts:167-217`) — so the agent's very next command can use a ref this report just - handed it at full precision, not the coarse warning floor; -- ranked selector suggestions from decision 1's retired heal machinery; -- when decision 3 applies, the recorded-vs-observed identity mismatch. - -**(c)** The loop protocol — run, read the report, then either fix reality and `--from N`, or perform the -step manually and `--from N+1`, or edit the plain-text `.ad` file and `--from N` — is documented as a -help topic (`agent-device help ...`, alongside the existing disambiguation-policy paragraph in -`src/cli/parser/cli-help.ts`). Loop until exit 0. There is no agent-mode flag: deterministic/CI callers -get the same richer failure output as an interactive agent — they simply don't act on the suggestions, -the same way they already ignore `hint` strings today. - -**(d)** The success path stops being silent for text callers: a successful replay prints a one-line -summary (replayed N, wall time). Today text mode emits nothing on success (see the live evidence in -Context — the success payload has no `message`, so the generic renderer prints zero bytes), which -costs an agent a verification turn just to confirm the run happened. One line closes that for free; -`--json` output is unchanged. - -### 5. Validation +### 2. Disclose daemon-tree disambiguation and identify fast-path responses + +The daemon-tree selector path (`runtime-selector`) adds an additive `resolution` response field. A unique +tree resolution is `{ source: "runtime", kind: "unique" }`; a heuristic resolution is +`{ source: "runtime", kind: "disambiguated", matchCount, winnerRef, tiebreak, alternatives }`. +`tiebreak` is one of `visible`, `deepest`, or `smallest-area`; `alternatives` contains actionable refs for +at most **5** losing candidates. The selected ref is not included in `alternatives`. This discloses the +existing heuristic without changing `resolveSelectorChain` or its winner. + +The accepted direct-iOS selector fast path has no daemon tree and the XCTest response cannot truthfully +provide a match count, candidate refs, or a runtime tiebreak. It remains enabled for ordinary simple +`press`/`fill`, but its canonical response instead carries +`resolution: { source: "direct-ios", kind: "not-observed" }`. It must never fabricate a unique-match or +identity claim. `--verify` and `--settle` continue to disable this fast path and therefore produce a +runtime resolution. Recording likewise disables it for any action for which target-binding evidence is +required by decision 3. + +ADR 0011's matrix must add a `resolutionDisclosure` guarantee. `runtime-selector` enforces the complete +runtime shape through the shared response builder; `direct-ios-selector` enforces only the explicit +`not-observed` shape through that builder. Its existing `disambiguation` and `responseIdentity` success +path waivers remain, and the exact waived-cell test must continue to list them. Layer-3 coverage must add +a runtime ambiguity/tiebreak/cap case and a direct-iOS no-snapshot case asserting `not-observed`. No +selection-parity table is claimed or added for the direct path: such a table would falsely imply that +XCTest selection has runtime parity. A future runner-side diagnostic design must replace the two waivers, +add a Swift/TypeScript parity fixture, and add the corresponding provider contract cases in the same +change. + +### 3. Versioned `.ad` target-binding evidence + +Recording writes evidence for every action that resolves an element target. The plain-text format is a +versioned comment immediately before the action it annotates: + +```text +# agent-device:target-v1 {"id":"save","role":"button","label":"Save","rect":{"x":12,"y":48,"width":80,"height":44}} +click @e12 "Save" +``` + +The prefix is ASCII and the payload is one JSON object encoded on one line. JSON supplies all quoting and +escaping; writers must use canonical `JSON.stringify` field order `id`, `role`, `label`, `rect` and rect +order `x`, `y`, `width`, `height`. `id`, `role`, and `label` are optional non-empty strings; `rect` is an +optional object of four finite numbers. The writer normalizes strings to Unicode NFC, omits missing or +empty fields, and emits no annotation when no field is available. A v1 parser accepts those fields in any +JSON object order, ignores unknown fields, normalizes known strings to NFC, and rejects malformed v1 +annotations or invalid known field types with `INVALID_ARGS`. An unknown future `target-vN` comment is an +ordinary comment to a v1 reader. + +The annotation binds only to the next physical action line. A blank line or any intervening line leaves +it unbound and is rejected as `INVALID_ARGS`; this prevents an edit from silently moving evidence to a +different target. Parser/writer tests must prove parse-write-parse semantic equality, embedded quotes, +backslashes, Unicode, and the unbound/malformed cases. + +Old readers ignore the comment and execute the action unchanged. New readers accept old scripts with no +annotation and perform no target-binding check for those actions. A writer that reads then rewrites a +script preserves v1 annotations in canonical form; it must not silently discard them. This is an additive +`.ad` format change, not merely per-line growth. + +At replay, every annotated resolved target is checked before its action is sent. A field present in the +recording but absent in the observed node is a mismatch. `id` and `role` compare exactly after NFC; +`label` compares after NFC plus trim and internal whitespace collapse; rects match when every coordinate +and dimension differs by at most **8** recorded coordinate units. Every recorded field must match. An old +unannotated action remains executable without this check. Any mismatch is a +**target-binding divergence**, reported before the device action, even when resolution was unique; this +catches a unique-but-wrong rebind as well as a changed ambiguity winner. This is not general outcome +verification: `--verify` remains post-action change evidence with a different contract. + +### 4. Divergence wire contract and replay-only resume + +**Divergence is a structured error, not success data.** The daemon returns `ok:false` with code +`REPLAY_DIVERGENCE` and a `details.divergence` object for both an action failure and a target-binding +mismatch. The object has version `1` and contains `kind`, `step` (`index`, `source.path`, `source.line`), +`action`, `cause`, `screen`, `suggestions`, and, for binding failures, `targetBinding` +(`recorded`, `observed`, `mismatches`). `step.index` is the 1-based executable-plan ordinal, not a source +line. Its source location is diagnostic only. A Maestro parser must preserve the original file and line +through includes so that source location is actionable. + +`screen` is a fresh, actionable snapshot digest with `refsGeneration`. It contains no raw tree. Existing +response levels define its bounds: compact (`--level digest`) carries at most **8** refs and no selector +suggestions; default carries at most **20** refs and at most **5** ranked suggestions; full carries the +same hard caps with the full fields for those entries. The 20-ref and 5-candidate limits are absolute, +including error payloads. The report declares truncation when either limit is reached. This keeps the +failure path bounded while preserving enough current-screen evidence to act. + +The same daemon error is preserved end to end. The Node client rejects with `AppError` retaining +`details.divergence`. CLI exits nonzero; text renders a compact report and JSON includes the complete +structured error. The MCP tool returns `isError: true`, exposes the object as `structuredContent`, and +renders the same compact text summary. MCP treats this error as a ref-issuing result: it merges and pins +every `screen` ref with `refsGeneration` before returning it, including on the error path. CLI and direct +client callers receive the unpinned refs and generation already present in the daemon error. No caller +gets a text-only divergence that loses its repair data. + +`--from N` is a `replay`-only flag. `test` must reject it as `INVALID_ARGS`; test shares replay execution +but must remain a full, deterministic suite run. `N` is a 1-based index into the fully expanded +executable plan and must be in range. It is never a YAML line number, fractional source-step number, or a +repeat iteration label. Static includes, platform conditions, and fixed-count repeats expand before +indexing, so repeated source lines are distinguished by their plan index. + +Resume does not reconstruct execution state. For `N > 1`, preflight must reject with `INVALID_ARGS` when +any skipped action can produce `outputEnv` values, or when the skipped range or resume target is inside +runtime control flow (conditional, retry, or dynamic repeat). The only variables available after a resume +are explicit script/header, CLI, and shell inputs; if the planner cannot prove that, it rejects rather +than invoking with an incomplete scope. The daemon also never infers app state: the caller must put the +app into the required state before resuming. This conservative rule is intentionally the first release +scope; deterministic state reconstruction is deferred until it can be specified and tested separately. + +The loop is therefore: run, read the divergence, repair app or script state, then replay from the reported +plan index (or the next index after completing the failed action manually). Help documents that protocol +and its resume rejections. Successful text replay prints one line with replayed count and wall time; +`--json` remains structured. + +### 5. Mandatory validation + +Implementation is not accepted on benchmark evidence alone. Required automated coverage is: + +- matrix and provider contracts for runtime ambiguity disclosure, the five-alternative cap, direct-iOS + `not-observed` disclosure, and the retained direct-path waiver list; +- parser/writer unit cases for v1 identity round trips, old/new reader compatibility, escaping, + normalization, rect tolerance, malformed annotations, and mismatch-before-action behavior; +- replay runtime tests for every annotated target, unique-but-wrong matches, compact/default/full caps, + `--from` indexing, variable-output and control-flow preflight rejection, and `test --from` rejection; +- daemon/client/CLI/MCP contracts proving the typed divergence survives failure, JSON and MCP structured + output retain it, and MCP pins its error-path refs; and +- `--update` retirement tests proving it never rewrites the source file and only returns bounded + suggestions. Extend the settle benchmark (`~/.agent-device-bench/rnnav-matrix.py` pattern, external harness) with a -replay arm: author a script from one session, then measure (i) clean-replay cost — should be 0 model -turns, matching the O(divergences) claim — and (ii) induced-divergence repair cost — break one selector -deliberately, measure agent turns to green through the `--from` loop. +replay arm only after these contracts pass: measure clean replay and one induced divergence repaired +through the allowed `--from` loop. ## Consequences -- `--from` resumability makes app-state **preconditions the caller's responsibility**. The daemon has - no way to know the app is actually in the state step N expects; the divergence report hands over - current screen evidence specifically so the caller can check that before resuming, but nothing - enforces it. The live nav-state-persistence divergence in Context is the canonical case: the app, - not the script, decides what screen a relaunch lands on. +- `--from` makes app state the caller's responsibility, and only accepts a resume when the planner can + prove its variable and control-flow state is independent of skipped execution. The daemon has no way to + know that the app is actually in the state step N expects. The live nav-state-persistence divergence in + Context is the canonical case: the app, not the script, decides what screen a relaunch lands on. - **Non-idempotent scripts are exactly why `--from` must never re-run steps `1..N-1`**: a script that creates a record, navigates, then asserts on it would double-create on any re-run of its early steps. This is a hard constraint on the flag's semantics, not an implementation nicety. -- **Retiring heal-as-actor removes CI self-repair for agentless callers.** A nightly `test --update` run - that used to silently patch a renamed selector and go green now stays red with a suggestion in the - divergence report nobody reads. This is a real regression for that use case, accepted because the - audit found heal rarely able to act anyway (rename is exactly the case it cannot rescue), and a - silently patched selector was already a correctness risk of the kind decision 3 closes — "recorded and - current selector agree" is not the same claim as "they agree on the right element." -- **Disclosure adds bytes to every response where resolution was ambiguous.** A capped candidate list - keeps this bounded, but it is a token-cost increase on exactly the interactions that were already - hardest to get right, not a free win. -- **Recorded identity evidence adds bytes to every `.ad` file**, proportional to steps that target - selectors (ref-only steps already carry `refLabel` at similar cost today). `.ad` stays plain text; - this is per-line growth, not a format change. +- **Retiring heal-as-actor removes CI self-repair for agentless callers.** `--update` may return bounded + suggestions but never rewrites the script. A nightly run that once patched a selector now stays red. + This is accepted because the audit found the mechanism rarely useful and a silent patch is a + target-binding risk: selector agreement is not proof of the same target. +- **Disclosure adds bounded bytes.** Runtime ambiguity responses carry at most five alternatives; direct + iOS responses pay only the explicit `not-observed` provenance marker. +- **Recorded identity evidence is an additive `.ad` format change.** It adds one reserved JSON comment + before each supported recorded target action; scripts without the comment remain valid. - **The disambiguation heuristic itself (visible → deepest → smallest-area) is unchanged.** The rejected alternative was hard-reject: fail any non-unique match instead of picking one. That would break benign, common cases the heuristic exists for — react-navigation's Maestro suite alone has 185 @@ -276,10 +318,9 @@ deliberately, measure agent turns to green through the `--from` loop. - **An `--agent`/agent-mode flag on `replay`**: rejected — no semantic fork is needed once the divergence report is simply the richer default failure shape. A deterministic CI caller does not need protection from a richer error payload it can ignore. -- **Keep `--update` auto-heal, add outcome verification to it**: rejected for now — decision 3's - recorded-identity check subsumes what a verified auto-heal would buy (a heal that knows it healed - correctly) more simply, without heal's own retry-and-rewrite complexity. Revisit if the agentless-CI - regression noted in Consequences proves costlier than expected. +- **Keep `--update` auto-heal, add target-binding verification to it**: rejected — decision 3 verifies + the resolved target without retry-and-rewrite behavior. Revisit only if agentless CI needs a separately + specified and testable repair policy. - **Auto-heal tiers** (safe-tier heals applied automatically, risky-tier surfaced): deferred, not rejected outright — there is no current evidence base for which heals are "safe," and tiering now would be speculative. Revisit if agentless CI demand for some self-repair materializes. @@ -288,19 +329,13 @@ deliberately, measure agent turns to green through the `--from` loop. Each step lands independently useful, in order: -1. **Resolution disclosure** (decision 2) — additive fields on the existing single response-construction - site, no flag, no format change, immediately useful for live commands. -2. **`.ad` recorded identity evidence** (decision 3, recording side only) — the format change and the - replay-side comparison land separately so each is independently reviewable; recorded evidence stays - inert (captured but unchecked) until step 3. -3. **`replay --from` + the structured divergence report** (decision 4) — the report is what `--from` - resumes from, so they land together; wires in decision 3's comparison and decision 1's - retired-heal suggestions at the same time. Step provenance (the `.ad` line in the report and - Maestro per-step source positions) is part of this step, not an optional follow-up — without it - `--from` is unusable on Maestro flows. The one-line success summary (decision 4d) can land any - time, independently. -4. **`--update` retirement** (decision 1, the removal of its rewrite path) — lands once step 3's - suggestions are a proven substitute, not before, so there is no gap where healing regresses with - nothing in its place. -5. **Benchmark extension** (decision 5) validates 1–4 against the O(divergences) claim before this - ADR's economics are treated as proven rather than designed-for. +1. **Resolution disclosure** (decision 2) — update the matrix, exact waiver list, and provider contracts + together. It is additive to response data and does not claim direct-iOS selection parity. +2. **`.ad` target annotations** (decision 3) — land parser/writer round trips and compatibility before + recording. Recording and pre-action target-binding verification then land together. +3. **Structured divergence + `replay --from`** (decision 4) — land daemon, client, CLI, and MCP error + propagation together with plan provenance and conservative resume preflight. `test` does not expose + `--from`. +4. **`--update` retirement** (decision 1) — remove its write path only after divergence suggestions are + available, with a no-write regression test. +5. **Benchmark extension** (decision 5) follows the mandatory contracts and measures the economic claim. diff --git a/docs/adr/README.md b/docs/adr/README.md index 47ba47f2f1..0dcefce740 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -13,7 +13,7 @@ | [0009 Apple Platform Consolidation](0009-apple-platform-consolidation.md) | Apple platform family, apple/appleOs axes, the apple-leak guard | | [0010 Error system conventions](0010-error-system.md) | error codes, hints, normalizeError, typed error signals | | [0011 Interaction Guarantee Contract](0011-interaction-guarantee-contract.md) | interaction dispatch paths, fast paths, guards, the guarantee matrix, parity tables | -| [0012 Interactive Replay](0012-interactive-replay.md) | replay healing/`--update`, selector disambiguation disclosure, `.ad` recorded identity, `replay --from` and the divergence report | +| [0012 Interactive Replay](0012-interactive-replay.md) | replay healing/`--update`, resolution disclosure, `.ad` target-binding evidence, divergence wire/error handling, and replay-only `--from` semantics | ADRs record *why*; the registries and gates they describe are the living source of truth — when prose and a registry disagree, the registry wins and the ADR needs a follow-up. From 8828470b867e2fa650feeb4a49e6ea10780720a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 10 Jul 2026 09:53:39 +0200 Subject: [PATCH 4/7] docs: tighten interactive replay contracts --- docs/adr/0012-interactive-replay.md | 177 +++++++++++++++++++--------- docs/adr/README.md | 2 +- 2 files changed, 122 insertions(+), 57 deletions(-) diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md index 162174ae4d..7aa96ac6ab 100644 --- a/docs/adr/0012-interactive-replay.md +++ b/docs/adr/0012-interactive-replay.md @@ -158,11 +158,21 @@ is strictly more valuable than the same proposal applied blind. ### 2. Disclose daemon-tree disambiguation and identify fast-path responses The daemon-tree selector path (`runtime-selector`) adds an additive `resolution` response field. A unique -tree resolution is `{ source: "runtime", kind: "unique" }`; a heuristic resolution is -`{ source: "runtime", kind: "disambiguated", matchCount, winnerRef, tiebreak, alternatives }`. -`tiebreak` is one of `visible`, `deepest`, or `smallest-area`; `alternatives` contains actionable refs for -at most **5** losing candidates. The selected ref is not included in `alternatives`. This discloses the -existing heuristic without changing `resolveSelectorChain` or its winner. +tree resolution is `{ source: "runtime", phase: "pre-action", kind: "unique" }`; a heuristic resolution +is `{ source: "runtime", phase: "pre-action", kind: "disambiguated", matchCount, winnerDiagnostic, +tiebreak, alternatives }`. `tiebreak` is one of `visible`, `deepest`, or `smallest-area`; `alternatives` +contains at most **5** losing `diagnosticRef` entries. The selected diagnostic is not included in +`alternatives`. `winnerDiagnostic` and each alternative are `{ diagnosticRef, role?, label? }`, where +`diagnosticRef` is an opaque non-`@` diagnostic token; every optional string is capped at **256 UTF-8 +bytes** with a truncation marker. This discloses the existing heuristic without changing +`resolveSelectorChain` or its winner. + +These are **pre-action diagnostics**, not issued refs. The selector-resolution snapshot can be invalid +after a mutating press/fill, so neither `winnerDiagnostic` nor `alternatives` carries `refsGeneration`, is +MCP-pinned, or may be reused as an `@ref` target. A caller that wants to act on an alternative must take a +fresh `snapshot`/`find`. A post-action `--settle` diff remains a separate, actionable issuer and may carry +fresh pinned refs. In contrast, a target-binding divergence sends no action; its fresh report snapshot is +an actionable issuer as defined in decision 4. The accepted direct-iOS selector fast path has no daemon tree and the XCTest response cannot truthfully provide a match count, candidate refs, or a runtime tiebreak. It remains enabled for ordinary simple @@ -172,15 +182,19 @@ identity claim. `--verify` and `--settle` continue to disable this fast path and runtime resolution. Recording likewise disables it for any action for which target-binding evidence is required by decision 3. -ADR 0011's matrix must add a `resolutionDisclosure` guarantee. `runtime-selector` enforces the complete -runtime shape through the shared response builder; `direct-ios-selector` enforces only the explicit -`not-observed` shape through that builder. Its existing `disambiguation` and `responseIdentity` success -path waivers remain, and the exact waived-cell test must continue to list them. Layer-3 coverage must add -a runtime ambiguity/tiebreak/cap case and a direct-iOS no-snapshot case asserting `not-observed`. No -selection-parity table is claimed or added for the direct path: such a table would falsely imply that -XCTest selection has runtime parity. A future runner-side diagnostic design must replace the two waivers, -add a Swift/TypeScript parity fixture, and add the corresponding provider contract cases in the same -change. +ADR 0011's matrix must add a `resolutionDisclosure` guarantee with all six honest cells: `runtime-selector` +enforces the complete pre-action diagnostic shape; `runtime-ref` and `native-ref` enforce +`{ source: "ref", phase: "pre-action", kind: "exact" }`; `direct-ios-selector` enforces only the explicit +`{ source: "direct-ios", kind: "not-observed" }` shape; `coordinate` is inapplicable because no element +was resolved; and `maestro-non-hittable-fallback` is inapplicable because Maestro owns matching and the +fallback is coordinate execution. The four enforced cells use the shared response builder. Its existing +direct-path `disambiguation` and `responseIdentity` waivers remain, and the exact waived-cell test must +continue to list them. Layer-3 coverage must claim every enforced/delegated cell: runtime +ambiguity/tiebreak/cap plus non-actionable diagnostics after mutation, exact-ref provenance for runtime +and native refs, and a direct-iOS no-snapshot `not-observed` case. No selection-parity table is claimed or +added for the direct path: such a table would falsely imply XCTest selection has runtime parity. A future +runner-side diagnostic design must replace the two waivers, add a Swift/TypeScript parity fixture, and add +the corresponding provider contract cases in the same change. ### 3. Versioned `.ad` target-binding evidence @@ -188,18 +202,33 @@ Recording writes evidence for every action that resolves an element target. The versioned comment immediately before the action it annotates: ```text -# agent-device:target-v1 {"id":"save","role":"button","label":"Save","rect":{"x":12,"y":48,"width":80,"height":44}} +# agent-device:target-v1 {"id":"save","role":"button","label":"Save","rect":{"x":12,"y":48,"width":80,"height":44},"ancestry":[{"role":"window"},{"role":"toolbar","id":"editor"}],"sibling":0,"verification":"verified"} click @e12 "Save" ``` The prefix is ASCII and the payload is one JSON object encoded on one line. JSON supplies all quoting and -escaping; writers must use canonical `JSON.stringify` field order `id`, `role`, `label`, `rect` and rect -order `x`, `y`, `width`, `height`. `id`, `role`, and `label` are optional non-empty strings; `rect` is an -optional object of four finite numbers. The writer normalizes strings to Unicode NFC, omits missing or -empty fields, and emits no annotation when no field is available. A v1 parser accepts those fields in any -JSON object order, ignores unknown fields, normalizes known strings to NFC, and rejects malformed v1 -annotations or invalid known field types with `INVALID_ARGS`. An unknown future `target-vN` comment is an -ordinary comment to a v1 reader. +escaping; writers must use canonical `JSON.stringify` field order `id`, `role`, `label`, `rect`, +`ancestry`, `sibling`, `verification`, `matchCount` and rect order `x`, `y`, `width`, `height`. `id`, +`role`, and `label` are optional non-empty strings; `rect` is an optional object of four finite numbers. +`verification` is `"verified"` or `"unverifiable"`; `matchCount` is required only for the latter. `role` is +`normalizeType(node.type ?? "")`, exactly the normalized type used by `buildSelectorChainForNode`; it is +never the raw optional `node.role`. `ancestry` is up to eight root-to-parent entries of the same normalized +`role` plus optional id/label, derived through `parentIndex`; `sibling` is the zero-based ordinal among +siblings with the same local identity. The writer normalizes strings to Unicode NFC and omits missing or +empty fields. A v1 payload is at most **4 KiB** UTF-8; each string field is at most **256 bytes** after +normalization; `ancestry` has at most eight entries; and `matchCount`, when present, is a positive safe +integer. The parser rejects a v1 annotation exceeding these bounds with `INVALID_ARGS`. + +The writer must test the tuple against the record-time tree. It writes `verification: "verified"` only +when exactly one node matches id/role/label/rect/ancestry/sibling. If zero or multiple nodes match, it +writes `verification: "unverifiable"` and `matchCount`; replay reports an +`identity-unverifiable` target-binding divergence before acting. At replay, the observed tree must also +produce exactly one tuple match. This closes the duplicate-label/identical-rect case even if the stronger +structural context is itself duplicated: ambiguity is a visible divergence, never a silent binding. + +A v1 parser accepts known fields in any JSON object order, ignores unknown fields, normalizes known +strings to NFC, and rejects malformed annotations or invalid known field types with `INVALID_ARGS`. An +unknown future `target-vN` comment is an ordinary comment to a v1 reader. The annotation binds only to the next physical action line. A blank line or any intervening line leaves it unbound and is rejected as `INVALID_ARGS`; this prevents an edit from silently moving evidence to a @@ -212,10 +241,11 @@ script preserves v1 annotations in canonical form; it must not silently discard `.ad` format change, not merely per-line growth. At replay, every annotated resolved target is checked before its action is sent. A field present in the -recording but absent in the observed node is a mismatch. `id` and `role` compare exactly after NFC; -`label` compares after NFC plus trim and internal whitespace collapse; rects match when every coordinate -and dimension differs by at most **8** recorded coordinate units. Every recorded field must match. An old -unannotated action remains executable without this check. Any mismatch is a +recording but absent in the observed node is a mismatch. `id` and normalized `role` compare exactly after +NFC; `label` compares after NFC plus trim and internal whitespace collapse; rects match when every +coordinate and dimension differs by at most **8** recorded coordinate units; ancestry and sibling compare +exactly after their component normalization. Every recorded field must match. An old unannotated action +remains executable without this check. Any mismatch is a **target-binding divergence**, reported before the device action, even when resolution was unique; this catches a unique-but-wrong rebind as well as a changed ambiguity winner. This is not general outcome verification: `--verify` remains post-action change evidence with a different contract. @@ -225,17 +255,30 @@ verification: `--verify` remains post-action change evidence with a different co **Divergence is a structured error, not success data.** The daemon returns `ok:false` with code `REPLAY_DIVERGENCE` and a `details.divergence` object for both an action failure and a target-binding mismatch. The object has version `1` and contains `kind`, `step` (`index`, `source.path`, `source.line`), -`action`, `cause`, `screen`, `suggestions`, and, for binding failures, `targetBinding` +`action`, `cause`, `screen`, `suggestions`, `resume`, and, for binding failures, `targetBinding` (`recorded`, `observed`, `mismatches`). `step.index` is the 1-based executable-plan ordinal, not a source line. Its source location is diagnostic only. A Maestro parser must preserve the original file and line through includes so that source location is actionable. -`screen` is a fresh, actionable snapshot digest with `refsGeneration`. It contains no raw tree. Existing -response levels define its bounds: compact (`--level digest`) carries at most **8** refs and no selector -suggestions; default carries at most **20** refs and at most **5** ranked suggestions; full carries the -same hard caps with the full fields for those entries. The 20-ref and 5-candidate limits are absolute, -including error payloads. The report declares truncation when either limit is reached. This keeps the -failure path bounded while preserving enough current-screen evidence to act. +`screen` is discriminated. `{ state: "available", refsGeneration, refs, truncated }` is a fresh, +healthy snapshot digest and the only form that issues actionable refs. `{ state: "unavailable", reason, +hint }` is returned when capture fails or is sparse; it has no refs or generation and must not fall back to +the old session tree. Screen-capture failure never replaces or masks the original replay cause. + +Response levels bound the entire serialized UTF-8 `details.divergence` object, not merely its arrays: +compact (`--level digest`) is at most **8 KiB**, default at most **24 KiB**, and full at most **64 KiB**. +Compact carries at most **8** screen refs and no suggestions; default and full carry at most **20** screen +refs and **5** ranked suggestions. These counts are absolute, including error payloads. Individual +labels, ids, selectors, source paths, mismatch values, cause messages, and hints are UTF-8 truncated to +**256 bytes**; an action summary has no positional array, and fill text, expanded variables, and arbitrary +nested cause details are never serialized. All rendered strings and any overflow artifact pass through the +central diagnostics redactor before truncation. The report sets truncation/redaction markers for every +omission. + +When the bounded form would omit material, the daemon writes the same redacted, bounded-per-field detail +to a session-scoped divergence artifact and returns its path plus `overflow: { omittedBytes, artifactPath +}`. If that artifact cannot be written, it returns `artifactUnavailable: true` and preserves the original +error. No raw snapshot tree or unredacted input is written to the artifact. The same daemon error is preserved end to end. The Node client rejects with `AppError` retaining `details.divergence`. CLI exits nonzero; text renders a compact report and JSON includes the complete @@ -251,6 +294,14 @@ executable plan and must be in range. It is never a YAML line number, fractional repeat iteration label. Static includes, platform conditions, and fixed-count repeats expand before indexing, so repeated source lines are distinguished by their plan index. +Every divergence includes `resume: { allowed, from, reason?, planDigest }`. `planDigest` is SHA-256 over +the canonical fully expanded plan, including each action's command, normalized inputs, control shape, +platform-conditioned expansion, and source provenance. A resume requires both `--from N` and +`--plan-digest ` from the report. The daemon rebuilds the current plan and rejects +`INVALID_ARGS` before any action when its digest differs, so edits, include changes, or environment-driven +expansion cannot silently retarget ordinal N. `allowed: false` explains why no resume is safe; its digest +is still diagnostic, not an authorization to bypass preflight. + Resume does not reconstruct execution state. For `N > 1`, preflight must reject with `INVALID_ARGS` when any skipped action can produce `outputEnv` values, or when the skipped range or resume target is inside runtime control flow (conditional, retry, or dynamic repeat). The only variables available after a resume @@ -259,23 +310,32 @@ than invoking with an incomplete scope. The daemon also never infers app state: app into the required state before resuming. This conservative rule is intentionally the first release scope; deterministic state reconstruction is deferred until it can be specified and tested separately. -The loop is therefore: run, read the divergence, repair app or script state, then replay from the reported -plan index (or the next index after completing the failed action manually). Help documents that protocol -and its resume rejections. Successful text replay prints one line with replayed count and wall time; -`--json` remains structured. +The loop is therefore: run, read the divergence, repair app state, then replay with the reported plan +digest and index (or the next index after completing the failed action manually). Editing a script requires +a fresh full replay that produces a new digest. Help documents that protocol and its resume rejections. +Successful text replay prints one line with replayed count and wall time; `--json` remains structured. ### 5. Mandatory validation Implementation is not accepted on benchmark evidence alone. Required automated coverage is: -- matrix and provider contracts for runtime ambiguity disclosure, the five-alternative cap, direct-iOS - `not-observed` disclosure, and the retained direct-path waiver list; +- matrix and provider contracts for all six `resolutionDisclosure` cells: runtime ambiguity/tiebreak and + the five-alternative limit, runtime/native exact-ref provenance, direct-iOS `not-observed`, coordinate + and Maestro inapplicability, and the retained direct-path waiver list; +- an interaction mutation contract proving pre-action resolution diagnostics are not ref-issued or + MCP-pinned, a fresh snapshot is required before using an alternative, and a no-action target-binding + divergence can issue and pin its fresh report refs; - parser/writer unit cases for v1 identity round trips, old/new reader compatibility, escaping, - normalization, rect tolerance, malformed annotations, and mismatch-before-action behavior; -- replay runtime tests for every annotated target, unique-but-wrong matches, compact/default/full caps, - `--from` indexing, variable-output and control-flow preflight rejection, and `test --from` rejection; + normalized-role source, ancestry/sibling structural context, duplicate/unverifiable record and replay + evidence, rect tolerance, malformed annotations, and mismatch-before-action behavior; +- replay runtime tests for every annotated target, unique-but-wrong and duplicate-evidence divergences, + compact/default/full field and byte ceilings, redaction, overflow artifacts and artifact-write failure, + available versus sparse/capture-failed screen forms, and preservation of the original cause; +- replay resume tests for plan-digest emission and mismatch rejection after script/include/expansion + changes, `resume.allowed` reasons, `--from` indexing, variable-output and control-flow rejection, and + `test --from` rejection; - daemon/client/CLI/MCP contracts proving the typed divergence survives failure, JSON and MCP structured - output retain it, and MCP pins its error-path refs; and + output retain it, MCP pins only actionable error-path refs, and no text-only path drops the report; and - `--update` retirement tests proving it never rewrites the source file and only returns bounded suggestions. @@ -286,9 +346,10 @@ through the allowed `--from` loop. ## Consequences - `--from` makes app state the caller's responsibility, and only accepts a resume when the planner can - prove its variable and control-flow state is independent of skipped execution. The daemon has no way to - know that the app is actually in the state step N expects. The live nav-state-persistence divergence in - Context is the canonical case: the app, not the script, decides what screen a relaunch lands on. + prove its variable and control-flow state is independent of skipped execution **and** its plan digest + matches the reported plan. The daemon has no way to know that the app is actually in the state step N + expects. The live nav-state-persistence divergence in Context is the canonical case: the app, not the + script, decides what screen a relaunch lands on. - **Non-idempotent scripts are exactly why `--from` must never re-run steps `1..N-1`**: a script that creates a record, navigates, then asserts on it would double-create on any re-run of its early steps. This is a hard constraint on the flag's semantics, not an implementation nicety. @@ -296,10 +357,12 @@ through the allowed `--from` loop. suggestions but never rewrites the script. A nightly run that once patched a selector now stays red. This is accepted because the audit found the mechanism rarely useful and a silent patch is a target-binding risk: selector agreement is not proof of the same target. -- **Disclosure adds bounded bytes.** Runtime ambiguity responses carry at most five alternatives; direct - iOS responses pay only the explicit `not-observed` provenance marker. +- **Disclosure adds bounded diagnostic bytes, not reusable targets.** Runtime ambiguity responses carry at + most five pre-action alternatives; direct iOS responses pay only the explicit `not-observed` provenance + marker. A fresh capture is the cost of acting on a diagnostic alternative. - **Recorded identity evidence is an additive `.ad` format change.** It adds one reserved JSON comment - before each supported recorded target action; scripts without the comment remain valid. + before each supported recorded target action; scripts without the comment remain valid. A duplicate that + survives structural evidence is intentionally a pre-action unverifiable divergence, not a best guess. - **The disambiguation heuristic itself (visible → deepest → smallest-area) is unchanged.** The rejected alternative was hard-reject: fail any non-unique match instead of picking one. That would break benign, common cases the heuristic exists for — react-navigation's Maestro suite alone has 185 @@ -329,13 +392,15 @@ through the allowed `--from` loop. Each step lands independently useful, in order: -1. **Resolution disclosure** (decision 2) — update the matrix, exact waiver list, and provider contracts - together. It is additive to response data and does not claim direct-iOS selection parity. -2. **`.ad` target annotations** (decision 3) — land parser/writer round trips and compatibility before - recording. Recording and pre-action target-binding verification then land together. -3. **Structured divergence + `replay --from`** (decision 4) — land daemon, client, CLI, and MCP error - propagation together with plan provenance and conservative resume preflight. `test` does not expose - `--from`. +1. **Resolution disclosure** (decision 2) — update all six matrix cells, the exact waiver list, and + provider mutation contracts together. It is additive to response data and does not claim direct-iOS + selection parity or issue pre-action refs. +2. **`.ad` target annotations** (decision 3) — land bounded parser/writer round trips, compatibility, + structural uniqueness, and duplicate detection before recording. Recording and pre-action + target-binding verification then land together. +3. **Structured divergence + `replay --from`** (decision 4) — land bounded/redacted error propagation, + actionable-or-unavailable screen semantics, error-path MCP pinning, plan digest validation, and + conservative resume preflight together. `test` does not expose `--from`. 4. **`--update` retirement** (decision 1) — remove its write path only after divergence suggestions are available, with a no-write regression test. 5. **Benchmark extension** (decision 5) follows the mandatory contracts and measures the economic claim. diff --git a/docs/adr/README.md b/docs/adr/README.md index 0dcefce740..a8245fd562 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -13,7 +13,7 @@ | [0009 Apple Platform Consolidation](0009-apple-platform-consolidation.md) | Apple platform family, apple/appleOs axes, the apple-leak guard | | [0010 Error system conventions](0010-error-system.md) | error codes, hints, normalizeError, typed error signals | | [0011 Interaction Guarantee Contract](0011-interaction-guarantee-contract.md) | interaction dispatch paths, fast paths, guards, the guarantee matrix, parity tables | -| [0012 Interactive Replay](0012-interactive-replay.md) | replay healing/`--update`, resolution disclosure, `.ad` target-binding evidence, divergence wire/error handling, and replay-only `--from` semantics | +| [0012 Interactive Replay](0012-interactive-replay.md) | replay healing/`--update`, diagnostic resolution disclosure, bounded `.ad` target-binding evidence, bounded divergence wire/error handling, and plan-bound replay-only `--from` semantics | ADRs record *why*; the registries and gates they describe are the living source of truth — when prose and a registry disagree, the registry wins and the ADR needs a follow-up. From 1a31a646be0d591d308a509f1baac613870d8f9b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 10 Jul 2026 10:39:38 +0200 Subject: [PATCH 5/7] docs(adr-0012): demote geometry to disambiguation signal, fix matchCount, define matching algorithm MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reworks the target-v1 contract per review: identity is recorded id, else role + normalized label, plus a leaf-anchored ancestry prefix (K=8, nearest ancestors kept, root-side truncation only); absolute rects are demoted to never-compared diagnostics with the ±8 tolerance removed rather than tuned; duplicates disambiguate by recorded sibling order among the matching set, then viewport-relative order within the recorded scroll region — never absolute pixels; ties are identity-unverifiable divergences with candidates listed. matchCount is redefined as the replay-time recorded-selector match count (0..N, always present), with selector-miss (0) and identity-mismatch (>=1, no identity candidate) as distinct classes in an explicit six-path verification classification. Also inlines the quantitative benchmark numbers (3.67->1.00 snapshots, 14.3 vs 23.3/26.7 commands, 38/38 in 539s) so the evidence is durable without the external harness directory. --- docs/adr/0012-interactive-replay.md | 152 +++++++++++++++++++++------- 1 file changed, 114 insertions(+), 38 deletions(-) diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md index 7aa96ac6ab..36b7eedf7e 100644 --- a/docs/adr/0012-interactive-replay.md +++ b/docs/adr/0012-interactive-replay.md @@ -31,12 +31,21 @@ re-rendered node with the same id). PR #297 (closing #279) already trimmed heal `refLabel`-synthesis and numeric `get text` drift healing to keep it "centered on recorded selectors and explicit selector expressions" — heal has a maintained history of narrowing, not growing. -**Benchmark evidence** (2026-07-09/10, `~/.agent-device-bench/rnnav-matrix.py`, external harness): the -`--settle` quiet-window loop is now at its 1-snapshot floor, so wall time for a QA flow is dominated by -model turn latency, not device I/O. A happy-path agent-driven QA flow costs O(steps) model turns -end-to-end; a deterministic replay of the same flow costs O(divergences). The entire economic case for -replay is collapsing the per-step model-turn cost toward zero on the happy path and paying only where -reality diverged from the recording. +**Benchmark evidence** (2026-07-09/10, iOS simulator, react-navigation/RN playground matrix; harness +follows the `~/.agent-device-bench/rnnav-matrix.py` pattern, external — the key numbers are recorded +here so the evidence stays durable without the harness directory): + +| Measurement | Result | +| --- | --- | +| Snapshot captures per interaction, `--settle` off → on | 3.67 → 1.00 (the 1-snapshot floor) | +| Commands per task, settled arm vs unsettled arms | 14.3 vs 23.3 / 26.7 | +| react-navigation Maestro suite via deterministic replay | 38/38 flows green in 539 s, zero model turns | + +With the settle loop at its snapshot floor, wall time for an agent-driven QA flow is dominated by model +turn latency, not device I/O. A happy-path agent-driven QA flow costs O(steps) model turns end-to-end; a +deterministic replay of the same flow costs O(divergences) — the 38/38 sweep is that limit realized at +zero divergences. The entire economic case for replay is collapsing the per-step model-turn cost toward +zero on the happy path and paying only where reality diverged from the recording. **Audit evidence** (2026-07-10) on where that divergence cost actually goes: @@ -202,29 +211,69 @@ Recording writes evidence for every action that resolves an element target. The versioned comment immediately before the action it annotates: ```text -# agent-device:target-v1 {"id":"save","role":"button","label":"Save","rect":{"x":12,"y":48,"width":80,"height":44},"ancestry":[{"role":"window"},{"role":"toolbar","id":"editor"}],"sibling":0,"verification":"verified"} +# agent-device:target-v1 {"id":"save","role":"button","label":"Save","ancestry":[{"role":"toolbar","label":"Editor"},{"role":"window"}],"sibling":0,"viewportOrder":0,"scrollRegion":{"role":"scrollview","id":"editor-scroll"},"verification":"verified"} click @e12 "Save" ``` The prefix is ASCII and the payload is one JSON object encoded on one line. JSON supplies all quoting and -escaping; writers must use canonical `JSON.stringify` field order `id`, `role`, `label`, `rect`, -`ancestry`, `sibling`, `verification`, `matchCount` and rect order `x`, `y`, `width`, `height`. `id`, -`role`, and `label` are optional non-empty strings; `rect` is an optional object of four finite numbers. -`verification` is `"verified"` or `"unverifiable"`; `matchCount` is required only for the latter. `role` is -`normalizeType(node.type ?? "")`, exactly the normalized type used by `buildSelectorChainForNode`; it is -never the raw optional `node.role`. `ancestry` is up to eight root-to-parent entries of the same normalized -`role` plus optional id/label, derived through `parentIndex`; `sibling` is the zero-based ordinal among -siblings with the same local identity. The writer normalizes strings to Unicode NFC and omits missing or -empty fields. A v1 payload is at most **4 KiB** UTF-8; each string field is at most **256 bytes** after -normalization; `ancestry` has at most eight entries; and `matchCount`, when present, is a positive safe -integer. The parser rejects a v1 annotation exceeding these bounds with `INVALID_ARGS`. - -The writer must test the tuple against the record-time tree. It writes `verification: "verified"` only -when exactly one node matches id/role/label/rect/ancestry/sibling. If zero or multiple nodes match, it -writes `verification: "unverifiable"` and `matchCount`; replay reports an -`identity-unverifiable` target-binding divergence before acting. At replay, the observed tree must also -produce exactly one tuple match. This closes the duplicate-label/identical-rect case even if the stronger -structural context is itself duplicated: ambiguity is a visible divergence, never a silent binding. +escaping; writers must use canonical `JSON.stringify` field order `id`, `role`, `label`, `ancestry`, +`sibling`, `viewportOrder`, `scrollRegion`, `rect`, `verification`, and rect order `x`, `y`, `width`, +`height`. `verification` is `"verified"` or `"unverifiable"`. The payload has **three tiers** with +different comparison roles: + +- **Identity** (compared exactly): `id` when recorded, else `role` plus normalized `label`, plus the + leaf-anchored `ancestry` prefix. `role` is `normalizeType(node.type ?? "")`, exactly the normalized + type used by `buildSelectorChainForNode`; it is never the raw optional `node.role`. +- **Disambiguation signals** (consulted only when several current nodes share the identity): `sibling`, + then `viewportOrder` + `scrollRegion` — normalized, relative signals, never absolute pixels. +- **Diagnostics** (never compared): optional `rect`, carried only so divergence reports can show where + the recorded target was. + +Absolute geometry is deliberately demoted out of identity: an absolute rect is the least stable component +of a target's identity — scroll offset, device rotation, dynamic type, iPad/macOS window resizing, and +ordinary RN layout shifts all move rects between healthy runs — and the audit's identical-rect +"Prevent Remove" sibling pair proves absolute geometry cannot even separate identical siblings in the +worst case. No absolute-coordinate tolerance exists in v1: the earlier draft's ±8-unit rect comparison is +removed rather than tuned, because no measured drift distribution exists to justify any particular +constant. If a future revision reintroduces an absolute tolerance, it must carry measured evidence. + +**Normalization.** All strings are Unicode NFC. `label` additionally trims leading/trailing whitespace +and collapses internal whitespace runs to a single space. Comparison is case-sensitive after +normalization (a label case change is a real UI change). A string that is empty after normalization is +omitted by the writer and treated as absent by the comparator. Each string field is at most **256 UTF-8 +bytes** after normalization; the whole payload is at most **4 KiB**; `ancestry` has at most **eight** +entries; `sibling` and `viewportOrder` are non-negative safe integers. The parser rejects a v1 annotation +exceeding these bounds with `INVALID_ARGS`. + +**Local identity.** Two nodes share local identity when both carry `id` and the normalized ids are equal; +or, when the recording carries no `id`, when their normalized roles are equal and their normalized labels +are equal (label absent on both sides counts as equal; label present on exactly one side is a mismatch). +A recorded `id` never matches a node without that id. + +**Ancestry.** The chain is the nearest **K = 8** ancestors of the target, ordered **leaf→root** (nearest +ancestor first), each entry `{ role, label? }` under the same normalization (`role` may be the empty +string when the node has no type; `label` is omitted when empty). Truncation drops entries from the +**root side only** — the nearest ancestors are always kept. Comparison is a **leaf-anchored prefix +match**: recorded chain R matches observed chain O iff for every index `i < |R|`, `O[i]` exists, the +roles are equal, and — when `R[i]` carries a label — the labels are equal (a label absent in `R[i]` is +unconstrained). `|O| < |R|` is a mismatch. An inserted or removed wrapper ancestor therefore changes +identity by design: structure is part of identity. + +**Record-time write.** + +1. Resolve the action's winner and compute its identity tuple from the record-time tree. +2. Compute the record-time identity set: all nodes sharing the winner's local identity with a matching + leaf-anchored ancestry prefix. +3. `sibling` is the winner's zero-based ordinal within that set in tree (document) order. + `viewportOrder` is the winner's zero-based ordinal within that set ordered by rect center, + top-to-bottom then left-to-right; members without rects sort last, in tree order. `scrollRegion` is + the local identity (`role` + `id`/`label`) of the winner's nearest scrollable ancestor, omitted when + none exists. +4. Run the replay-time verification algorithm below against the record-time tree itself. If it isolates + exactly the winner, write `verification: "verified"`; otherwise write `verification: "unverifiable"`. + An unverifiable annotation makes the step an `identity-unverifiable` divergence at replay, before + acting — the evidence declares its own limits at record time instead of permitting a silent best + guess later. A v1 parser accepts known fields in any JSON object order, ignores unknown fields, normalizes known strings to NFC, and rejects malformed annotations or invalid known field types with `INVALID_ARGS`. An @@ -240,15 +289,35 @@ annotation and perform no target-binding check for those actions. A writer that script preserves v1 annotations in canonical form; it must not silently discard them. This is an additive `.ad` format change, not merely per-line growth. -At replay, every annotated resolved target is checked before its action is sent. A field present in the -recording but absent in the observed node is a mismatch. `id` and normalized `role` compare exactly after -NFC; `label` compares after NFC plus trim and internal whitespace collapse; rects match when every -coordinate and dimension differs by at most **8** recorded coordinate units; ancestry and sibling compare -exactly after their component normalization. Every recorded field must match. An old unannotated action -remains executable without this check. Any mismatch is a -**target-binding divergence**, reported before the device action, even when resolution was unique; this -catches a unique-but-wrong rebind as well as a changed ambiguity winner. This is not general outcome -verification: `--verify` remains post-action change evidence with a different contract. +**Replay-time verification.** Every annotated resolved target is checked before its action is sent, by +this exact classification. `matchCount` is the number of current nodes matching the **recorded selector** +at replay time — the same match set resolution itself used — with range **0..N** and **always present** +in the report's `targetBinding`. Identity verification applies only when `matchCount >= 1`. + +1. Recorded `verification` is `"unverifiable"` → **identity-unverifiable** divergence, before any + resolution. +2. `matchCount == 0` → **selector-miss** divergence: the recorded selector no longer matches anything. + This class is distinct from an identity mismatch — the repair is a selector repair. +3. `matchCount >= 1`; the identity set I (matched nodes sharing the recorded local identity with a + matching ancestry prefix) is empty → **identity-mismatch** divergence: the selector still matches, + but nothing carries the recorded identity. +4. `|I| == 1` and the resolution winner W is that member → **verified**; the action proceeds. This is + the only path that sends the action. +5. `|I| == 1` and W is a different node → **identity-mismatch** divergence: a unique-but-wrong rebind or + a changed ambiguity winner, caught even when resolution was unique. +6. `|I| > 1` → apply the disambiguation signals in order: (i) order I in tree order; if the recorded + `sibling` ordinal is in range, the evidence denotes that member — compare with W as in paths 4/5. + (ii) Otherwise filter I to members whose nearest scrollable ancestor matches the recorded + `scrollRegion` local identity (no filter when none was recorded), order by rect center top-to-bottom + then left-to-right (rect-less members last, in tree order); if the recorded `viewportOrder` ordinal is + in range of the filtered set, the evidence denotes that member — compare with W as in paths 4/5. + If neither signal isolates a member, the step is an **identity-unverifiable** divergence with up to + **5** candidates listed — never a silent pick. That refusal is the point of this ADR. + +A field present in the recording but absent on the compared node is a mismatch; `rect` is never compared. +An old unannotated action remains executable without this check. All three divergence classes are +target-binding divergences reported before the device action. This is not general outcome verification: +`--verify` remains post-action change evidence with a different contract. ### 4. Divergence wire contract and replay-only resume @@ -256,7 +325,10 @@ verification: `--verify` remains post-action change evidence with a different co `REPLAY_DIVERGENCE` and a `details.divergence` object for both an action failure and a target-binding mismatch. The object has version `1` and contains `kind`, `step` (`index`, `source.path`, `source.line`), `action`, `cause`, `screen`, `suggestions`, `resume`, and, for binding failures, `targetBinding` -(`recorded`, `observed`, `mismatches`). `step.index` is the 1-based executable-plan ordinal, not a source +(`classification`, `matchCount`, `recorded`, `observed`, `mismatches`, `candidates`). `kind` is one of +`action-failure`, `selector-miss`, `identity-mismatch`, or `identity-unverifiable` — the latter three are +decision 3's target-binding classes, and `targetBinding.matchCount` is always present for them (0..N). +`step.index` is the 1-based executable-plan ordinal, not a source line. Its source location is diagnostic only. A Maestro parser must preserve the original file and line through includes so that source location is actionable. @@ -326,9 +398,13 @@ Implementation is not accepted on benchmark evidence alone. Required automated c MCP-pinned, a fresh snapshot is required before using an alternative, and a no-action target-binding divergence can issue and pin its fresh report refs; - parser/writer unit cases for v1 identity round trips, old/new reader compatibility, escaping, - normalized-role source, ancestry/sibling structural context, duplicate/unverifiable record and replay - evidence, rect tolerance, malformed annotations, and mismatch-before-action behavior; -- replay runtime tests for every annotated target, unique-but-wrong and duplicate-evidence divergences, + normalized-role source, leaf-anchored ancestry prefix matching (including root-side truncation and + inserted-wrapper mismatch), duplicate/unverifiable record and replay evidence, rect-never-compared, + malformed annotations, and mismatch-before-action behavior; +- replay runtime tests covering all six verification paths of decision 3 — recorded-unverifiable, + selector-miss (`matchCount == 0`), empty identity set, verified, unique-but-wrong rebind, and + post-signal tie (including out-of-range `sibling`/`viewportOrder` ordinals and `scrollRegion` + filtering) — plus divergence-report tests for compact/default/full field and byte ceilings, redaction, overflow artifacts and artifact-write failure, available versus sparse/capture-failed screen forms, and preservation of the original cause; - replay resume tests for plan-digest emission and mismatch rejection after script/include/expansion From 6b5a1b2c7e8acf34bc7073a6f9fc166ff91b38fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 10 Jul 2026 10:54:54 +0200 Subject: [PATCH 6/7] docs(adr-0012): unify positional-signal candidate domains between record and replay MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes the P1 domain mismatch: sibling becomes a genuine same-parent child index (parent already captured as ancestry[0], no new field; identical by definition on both sides, non-isolating when the same index recurs under different parents); viewportOrder gets one region-scoped domain — the identity set partitioned by scroll region, ordinal within the recorded partition on both sides, unavailable (never compared cross-region) when the recorded region no longer exists; document order (pre-order index) is the canonical total order making every ordering deterministic, including equal rect centers. Residual ties stay identity-unverifiable with candidates listed. Record-time write, replay verification, and mandatory validation updated in lockstep; the six-path classification is unchanged. --- docs/adr/0012-interactive-replay.md | 66 ++++++++++++++++++++--------- 1 file changed, 47 insertions(+), 19 deletions(-) diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md index 36b7eedf7e..14806ed5a4 100644 --- a/docs/adr/0012-interactive-replay.md +++ b/docs/adr/0012-interactive-replay.md @@ -224,8 +224,10 @@ different comparison roles: - **Identity** (compared exactly): `id` when recorded, else `role` plus normalized `label`, plus the leaf-anchored `ancestry` prefix. `role` is `normalizeType(node.type ?? "")`, exactly the normalized type used by `buildSelectorChainForNode`; it is never the raw optional `node.role`. -- **Disambiguation signals** (consulted only when several current nodes share the identity): `sibling`, - then `viewportOrder` + `scrollRegion` — normalized, relative signals, never absolute pixels. +- **Disambiguation signals** (consulted only when several current nodes share the identity): `sibling` + (a genuine same-parent child index), then `viewportOrder` scoped to the recorded `scrollRegion` + partition — normalized, relative signals, never absolute pixels, with document order as the final + deterministic tie-break for every ordering. - **Diagnostics** (never compared): optional `rect`, carried only so divergence reports can show where the recorded target was. @@ -259,19 +261,33 @@ roles are equal, and — when `R[i]` carries a label — the labels are equal (a unconstrained). `|O| < |R|` is a mismatch. An inserted or removed wrapper ancestor therefore changes identity by design: structure is part of identity. -**Record-time write.** +**Record-time write.** Both positional signals are defined over candidate domains that record and +replay compute identically — never one domain at record time and another at replay. **Document order** +— a node's pre-order tree-traversal index — is the canonical total order of this contract: every +enumeration, ordering tie, and candidate listing below resolves by document order, so every comparison +is total and deterministic. 1. Resolve the action's winner and compute its identity tuple from the record-time tree. 2. Compute the record-time identity set: all nodes sharing the winner's local identity with a matching leaf-anchored ancestry prefix. -3. `sibling` is the winner's zero-based ordinal within that set in tree (document) order. - `viewportOrder` is the winner's zero-based ordinal within that set ordered by rect center, - top-to-bottom then left-to-right; members without rects sort last, in tree order. `scrollRegion` is - the local identity (`role` + `id`/`label`) of the winner's nearest scrollable ancestor, omitted when - none exists. -4. Run the replay-time verification algorithm below against the record-time tree itself. If it isolates +3. `sibling` is the winner's zero-based index among its **parent's children** in the tree — a genuine + same-parent structural ordinal, independent of scroll regions and cheap to read off the + accessibility tree. The parent is already captured as `ancestry[0]` in the leaf-anchored chain, so + no additional field is recorded; record and replay compute this ordinal identically by definition. +4. Partition the identity set by **scroll region**: the partition key is the local identity (`role` + + `id`/`label`) of a member's nearest scrollable ancestor, or *none* when it has no scrollable + ancestor. `scrollRegion` is the winner's partition key (omitted when *none*). `viewportOrder` is the + winner's zero-based ordinal **within its own partition** — not the whole identity set — ordered by + rect center top-to-bottom then left-to-right, with equal centers resolved by document order and + rect-less members last, in document order. The partition is the ordinal's domain on both sides, so + recorded and replayed `viewportOrder` always refer to the same candidate domain. +5. Run the replay-time verification algorithm below against the record-time tree itself. If it isolates exactly the winner, write `verification: "verified"`; otherwise write `verification: "unverifiable"`. - An unverifiable annotation makes the step an `identity-unverifiable` divergence at replay, before + Because both ordinals are computed from the winner over deterministic total orders, this self-check + succeeds by construction whenever the capture supplies the needed structural data; `unverifiable` at + record time therefore marks a capture anomaly — a signal that could not be computed (e.g. missing + parent linkage) — and the branch is kept as a fail-closed safety valve, not an expected path. An + unverifiable annotation makes the step an `identity-unverifiable` divergence at replay, before acting — the evidence declares its own limits at record time instead of permitting a silent best guess later. @@ -305,14 +321,23 @@ in the report's `targetBinding`. Identity verification applies only when `matchC the only path that sends the action. 5. `|I| == 1` and W is a different node → **identity-mismatch** divergence: a unique-but-wrong rebind or a changed ambiguity winner, caught even when resolution was unique. -6. `|I| > 1` → apply the disambiguation signals in order: (i) order I in tree order; if the recorded - `sibling` ordinal is in range, the evidence denotes that member — compare with W as in paths 4/5. - (ii) Otherwise filter I to members whose nearest scrollable ancestor matches the recorded - `scrollRegion` local identity (no filter when none was recorded), order by rect center top-to-bottom - then left-to-right (rect-less members last, in tree order); if the recorded `viewportOrder` ordinal is - in range of the filtered set, the evidence denotes that member — compare with W as in paths 4/5. +6. `|I| > 1` → apply the disambiguation signals in order, each over the SAME candidate domain record + time used: (i) **sibling** — the members of I whose zero-based index among their own parent's + children equals the recorded `sibling`. Exactly one qualifying member: the evidence denotes it — + compare with W as in paths 4/5. Zero or several qualifying members (the same child index can recur + under different parents): the signal does not isolate; fall through. (ii) **region-scoped + viewportOrder** — restrict I to the partition whose scroll-region key equals the recorded + `scrollRegion` (the *none* partition when none was recorded). An empty partition means the recorded + scroll region no longer exists: `viewportOrder` is **unavailable** and is never compared across + regions; fall through. Otherwise order the partition by rect center top-to-bottom then + left-to-right (equal centers by document order; rect-less members last, in document order); if the + recorded `viewportOrder` ordinal is in range, the evidence denotes that member — compare with W as + in paths 4/5; out of range falls through. If neither signal isolates a member, the step is an **identity-unverifiable** divergence with up to - **5** candidates listed — never a silent pick. That refusal is the point of this ADR. + **5** candidates listed in document order — never a silent pick. Document order makes every ordering + above total, so a residual tie would require two nodes at identical positions in an identical tree — + impossible under pre-order indexing — and even that residual case is identity-unverifiable, not a + pick. That refusal is the point of this ADR. A field present in the recording but absent on the compared node is a mismatch; `rect` is never compared. An old unannotated action remains executable without this check. All three divergence classes are @@ -403,8 +428,11 @@ Implementation is not accepted on benchmark evidence alone. Required automated c malformed annotations, and mismatch-before-action behavior; - replay runtime tests covering all six verification paths of decision 3 — recorded-unverifiable, selector-miss (`matchCount == 0`), empty identity set, verified, unique-but-wrong rebind, and - post-signal tie (including out-of-range `sibling`/`viewportOrder` ordinals and `scrollRegion` - filtering) — plus divergence-report tests for + post-signal fall-through — including same-parent `sibling` semantics with the same child index + recurring under different parents, region-partitioned `viewportOrder` domains proven identical at + record and replay, a recorded scroll region that no longer exists (unavailable, never compared + cross-region), out-of-range ordinals, and document-order determinism for equal rect centers and + rect-less members — plus divergence-report tests for compact/default/full field and byte ceilings, redaction, overflow artifacts and artifact-write failure, available versus sparse/capture-failed screen forms, and preservation of the original cause; - replay resume tests for plan-digest emission and mismatch rejection after script/include/expansion From 054682a57d684a8ee69f1cd67dce77f11c066b36 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 10 Jul 2026 11:34:58 +0200 Subject: [PATCH 7/7] docs(adr-0012): conditional matchCount, dependency-ordered migration, writer invariant, suggestion ranking contract --- docs/adr/0012-interactive-replay.md | 93 ++++++++++++++++++++++------- 1 file changed, 70 insertions(+), 23 deletions(-) diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md index 14806ed5a4..550602944b 100644 --- a/docs/adr/0012-interactive-replay.md +++ b/docs/adr/0012-interactive-replay.md @@ -158,8 +158,20 @@ already travels in the `.ad` file. `--update`/`-u` stops silently rewriting `.ad` files. The two pieces of machinery it already has — `collectReplaySelectorCandidates` (recorded-chain/positional extraction) and the `resolveSelectorChain` -re-resolution it drives — are repurposed to populate a ranked list of selector suggestions inside the -divergence report (decision 4), not to act unattended. With an agent in the loop, adjudicating a heal +re-resolution it drives — are repurposed to populate the ranked `suggestions` list inside the +divergence report (decision 4), not to act unattended. + +**Ranking is a total order**: (1) candidates satisfying more identity components rank first — a +recorded-id match outranks a role+label match, which outranks a label-only match; (2) among equals, +candidates in the same `scrollRegion` as recorded rank before candidates in other regions; (3) document +order is the final tie-break. Suggestions are deduplicated by node: a node reachable through several +recorded selector terms appears once, tagged with its strongest match basis. The list is bounded by +decision 4's suggestion cap. Response levels affect only report content, never file behavior: before +retirement lands (migration step 6), `--update` keeps its legacy rewrite semantics regardless of level; +after retirement, `--update` at any level performs no rewrite and returns the same bounded suggestions +object, with `--level digest` omitting suggestion entries but carrying `suggestionCount` per decision 4. + +With an agent in the loop, adjudicating a heal proposal costs one cheap model turn — cheaper than discovering a silent wrong repair later — and the audit ((a) above) already found heal rarely able to act. A proposal an agent can accept, reject, or edit is strictly more valuable than the same proposal applied blind. @@ -247,6 +259,16 @@ bytes** after normalization; the whole payload is at most **4 KiB**; `ancestry` entries; `sibling` and `viewportOrder` are non-negative safe integers. The parser rejects a v1 annotation exceeding these bounds with `INVALID_ARGS`. +**Writer-parser invariant.** The recorder must never emit a payload its own parser rejects. When a +payload would exceed the 4 KiB ceiling after per-field truncation, the writer reduces it +deterministically: drop `ancestry` entries one at a time from the **root side** — the same side ancestry +truncation already drops from — until the payload fits. If it still overflows with only `ancestry[0]` +(the parent) retained, the writer downgrades the annotation to `verification: "unverifiable"` +(fail-closed) rather than writing an invalid or silently-lossy script; with the per-field 256-byte caps +in force, a parent-only payload fits arithmetically, so the downgrade branch is a terminal guarantee, +not an expected path. The record-time self-check (step 5 below) runs against the reduced tuple, so a +`verified` claim is always honest for exactly what was written. + **Local identity.** Two nodes share local identity when both carry `id` and the normalized ids are equal; or, when the recording carries no `id`, when their normalized roles are equal and their normalized labels are equal (label absent on both sides counts as equal; label present on exactly one side is a mismatch). @@ -307,8 +329,13 @@ script preserves v1 annotations in canonical form; it must not silently discard **Replay-time verification.** Every annotated resolved target is checked before its action is sent, by this exact classification. `matchCount` is the number of current nodes matching the **recorded selector** -at replay time — the same match set resolution itself used — with range **0..N** and **always present** -in the report's `targetBinding`. Identity verification applies only when `matchCount >= 1`. +at replay time — the same match set resolution itself used — with range **0..N**. It is **required on +every path that performs resolution (paths 2–6 below) and absent on path 1** — the key is omitted per +the drop-empty-keys convention, never `null` — because path 1 fires before any resolution. No +diagnostic-only count is computed there: a recorded-unverifiable annotation means there is no +trustworthy recorded identity to resolve against, so a count would invite misreading and add capture +cost on a path that by definition cannot verify. Identity verification applies only when +`matchCount >= 1`. 1. Recorded `verification` is `"unverifiable"` → **identity-unverifiable** divergence, before any resolution. @@ -352,7 +379,11 @@ mismatch. The object has version `1` and contains `kind`, `step` (`index`, `sour `action`, `cause`, `screen`, `suggestions`, `resume`, and, for binding failures, `targetBinding` (`classification`, `matchCount`, `recorded`, `observed`, `mismatches`, `candidates`). `kind` is one of `action-failure`, `selector-miss`, `identity-mismatch`, or `identity-unverifiable` — the latter three are -decision 3's target-binding classes, and `targetBinding.matchCount` is always present for them (0..N). +decision 3's target-binding classes, and `targetBinding.classification` always equals the top-level +`kind`. `targetBinding.matchCount` follows decision 3's presence rule exactly: present (0..N) for +`selector-miss`, `identity-mismatch`, and an `identity-unverifiable` reached through resolution (path 6); +absent — key omitted, never `null` — when `identity-unverifiable` arose from a recorded-unverifiable +annotation (path 1), which fires before any resolution. `step.index` is the 1-based executable-plan ordinal, not a source line. Its source location is diagnostic only. A Maestro parser must preserve the original file and line through includes so that source location is actionable. @@ -364,8 +395,10 @@ the old session tree. Screen-capture failure never replaces or masks the origina Response levels bound the entire serialized UTF-8 `details.divergence` object, not merely its arrays: compact (`--level digest`) is at most **8 KiB**, default at most **24 KiB**, and full at most **64 KiB**. -Compact carries at most **8** screen refs and no suggestions; default and full carry at most **20** screen -refs and **5** ranked suggestions. These counts are absolute, including error payloads. Individual +Compact carries at most **8** screen refs and no suggestion entries — it carries `suggestionCount` (the +number of suggestions available at default/full) so a caller knows whether a re-fetch at a higher level +has material; default and full carry at most **20** screen refs and **5** suggestions ranked per +decision 1's total order. These counts are absolute, including error payloads. Individual labels, ids, selectors, source paths, mismatch values, cause messages, and hints are UTF-8 truncated to **256 bytes**; an action summary has no positional array, and fill text, expanded variables, and arbitrary nested cause details are never serialized. All rendered strings and any overflow artifact pass through the @@ -433,7 +466,8 @@ Implementation is not accepted on benchmark evidence alone. Required automated c record and replay, a recorded scroll region that no longer exists (unavailable, never compared cross-region), out-of-range ordinals, and document-order determinism for equal rect centers and rect-less members — plus divergence-report tests for - compact/default/full field and byte ceilings, redaction, overflow artifacts and artifact-write failure, + compact/default/full field and byte ceilings (including digest-level `suggestionCount` with entries + omitted), redaction, overflow artifacts and artifact-write failure, available versus sparse/capture-failed screen forms, and preservation of the original cause; - replay resume tests for plan-digest emission and mismatch rejection after script/include/expansion changes, `resume.allowed` reasons, `--from` indexing, variable-output and control-flow rejection, and @@ -441,7 +475,7 @@ Implementation is not accepted on benchmark evidence alone. Required automated c - daemon/client/CLI/MCP contracts proving the typed divergence survives failure, JSON and MCP structured output retain it, MCP pins only actionable error-path refs, and no text-only path drops the report; and - `--update` retirement tests proving it never rewrites the source file and only returns bounded - suggestions. + suggestions ranked and deduplicated per decision 1's total order. Extend the settle benchmark (`~/.agent-device-bench/rnnav-matrix.py` pattern, external harness) with a replay arm only after these contracts pass: measure clean replay and one induced divergence repaired @@ -494,17 +528,30 @@ through the allowed `--from` loop. ## Migration plan -Each step lands independently useful, in order: - -1. **Resolution disclosure** (decision 2) — update all six matrix cells, the exact waiver list, and - provider mutation contracts together. It is additive to response data and does not claim direct-iOS - selection parity or issue pre-action refs. -2. **`.ad` target annotations** (decision 3) — land bounded parser/writer round trips, compatibility, - structural uniqueness, and duplicate detection before recording. Recording and pre-action - target-binding verification then land together. -3. **Structured divergence + `replay --from`** (decision 4) — land bounded/redacted error propagation, - actionable-or-unavailable screen semantics, error-path MCP pinning, plan digest validation, and - conservative resume preflight together. `test` does not expose `--from`. -4. **`--update` retirement** (decision 1) — remove its write path only after divergence suggestions are - available, with a no-write regression test. -5. **Benchmark extension** (decision 5) follows the mandatory contracts and measures the economic claim. +Steps are ordered so every dependency lands before its consumer; each step is independently useful, and +each states its dependencies explicitly. + +1. **Resolution disclosure** (decision 2) — no dependencies. Update all six matrix cells, the exact + waiver list, and provider mutation contracts together. Additive to response data; does not claim + direct-iOS selection parity or issue pre-action refs. +2. **Structured divergence transport** (decision 4, report only) — no dependencies. `REPLAY_DIVERGENCE` + with `kind: "action-failure"` attaches to the EXISTING replay failure paths: step provenance (source + path + line preserved through Maestro includes), bounded/redacted payloads, actionable-or-unavailable + screen semantics, error-path MCP pinning, ranked suggestions (decision 1's candidate machinery, + read-only), and the one-line text success summary. Immediately useful on its own — this closes the + provenance/evidence gaps the live hands-on evidence documents — and introduces no verification + semantics. +3. **`.ad` target annotations, inert** (decision 3, parser/writer only) — no dependencies. Bounded + parser/writer round trips, the writer-parser invariant with root-side reduction, old/new reader + compatibility, structural uniqueness, and duplicate detection. Recordings gain annotations; replay + parses and preserves them but does not yet enforce. +4. **Target-binding verification** (decision 3, enforcement) — depends on 2 (reports through the + divergence transport, adding the `selector-miss`/`identity-mismatch`/`identity-unverifiable` kinds) + and on 3 (consumes the annotations). +5. **`replay --from` + `--plan-digest` resume** (decision 4, resume) — depends on 2 only (the report + supplies `resume` and `planDigest`); may land before, with, or after 3/4. `test` does not expose + `--from`. +6. **`--update` retirement** (decision 1) — depends on 2 (ranked suggestions must be available in the + report before the write path is removed), with a no-write regression test. +7. **Benchmark extension** (decision 5) — follows the mandatory contracts; measures the economic claim + (clean replay plus one induced divergence repaired through the allowed `--from` loop).