Skip to content

refactor(ios): normalize snapshot geometry in one pass after acquisition (#2661) - #2669

Merged
thymikee merged 3 commits into
mainfrom
t3code/implement-issue-2661
Sep 19, 2026
Merged

thymikee merged 3 commits into
mainfrom
t3code/implement-issue-2661

Conversation

@thymikee

@thymikee thymikee commented Sep 18, 2026

Copy link
Copy Markdown
Member

Implements #2661: acquisition publishes reported geometry first and presentation consumes normalized geometry after a single pass. Behavior layer on top of #2664 (merged as a556e11), which owns the coordinate-rotation table and SnapshotGeometrySpace move.

What changed

  • One normalization pass. Acquisition (recursiveTreeSnapshotAcquisition, raw walk, private-AX, query sweep) serializes only the frame the platform reported. captureWithBackend runs a single SnapshotGeometrySpace.normalized(nodes:viewport:interfaceOrientation:) over the flat array before any presentation reads it, keyed on ancestry — no walker carries a coordinate space (geometrySpace/parentIsWindow removed).
  • Frontier dropped (Q2). The fold-consulting regular-depth frontier read reported (un-normalized) rects mid-walk to bump presented depth, so a turned keyboard band reported in the device's native space could be pruned before the single pass moved it back into the app viewport. A regular --depth is now a presentation cut only: the walk's bounds are raw traversal depth (nil for a regular capture) plus a runaway node cap; SnapshotPresentation applies the visibility fold and the presented-depth cut on the normalized array (the way private-AX already did).
  • Capability reclassified. The iOS tree backend's regular-depth capability moves presented-frontierpresentation-cut across the Swift declaration, the cross-runtime parity fixture, and the TypeScript registry; the now-unused Swift enum case is removed (Android's helper keeps presented-frontier).
  • normalized lives in SnapshotCoordinateSpace.swift comment-free (package convention); its tests live in CoordinateSpaceTests.swift. No unplacedSurfaceHostCount, no disclosure (per the correction).
  • ADR 0004 updated: presented-depth is a presentation cut, the walk is bounded by raw depth + node cap, and the de-dup rationale is corrected (a repeated node re-parents its children onto its own parent).

Validation (head 47b3aa1)

  • pnpm check:affected --run: all runnable checks passed (unit suite 3011 passed, xctest-selection, fallow, format, lint, typecheck).
  • swift test --package-path apple/snapshot-presentation: 14/14 (incl. the two-Window rotation replay, the measured landscape keyboard band, windowless-flat unplaced, and empty cases).
  • Runner unit tests on the iOS 26.2 simulator: capability-parity, testRegularDepthCutsPresentationNotAcquisition, testVisibilityFoldKeepsIndependentChildPastClippedParent, testRegularDepthKeepsTurnedKeyboardSubtreeAfterOneNormalizationPass (the turned-keyboard -d N regression: the band + q are presented at the app-space rects the fix(ios): publish captured keyboard geometry in the app's orientation space #2653 fixture predicts), private-AX scroll-hint, and testCaptureHintIsTheOnlyAcquisitionViewOfARequest — all green.
  • capture-kit parity test (Swift↔fixture↔TS capability table) green; PR-lane -only-testing: list updated and check:xctest-selection clean.

Live evidence (56fcf4e, see thread)

Runner-served on iPhone 17 Pro (iOS 26.2), Release test app, vs a main daemon on the same simulator with open --relaunch before each leg. Portrait form: -i 27 = 27 (keyboard down) and 73 = 73 (up), --depth 2 4 = 4 / 7 = 7, --depth 3 6 = 6 / 10 = 10, --raw 207 = 207, --json 93 = 93 — all byte-identical with refs stripped; scroll-hint carrier = the Checkout form ScrollView (hiddenContentBelow). Landscape keyboard up: Keyboard (75,198,724,202) and key Q (77,203,72,45) at --depth 5/7/--json (the #2653 numbers); the only cross-leg differences anywhere were the QuickType prediction cells, an iOS per-session state, bisected across 80ac6ae/56fcf4e/main to identical trees. Not covered: Bluesky (not installed), physical device.

--depth N cost (d1ae801 vs main, ADR fixed on f69fa5c)

The frontier on main engaged only under a regular --depth N (-i sends no depth), so that route was re-measured: runner-served tree, 15 warm captures per cell, open --relaunch per leg. Catalog scrolled (279 raw nodes → 4 presented at depth 2, 6 at depth 3): p50 +2.1 % / −0.6 %, p95 +0.7 % / −1.1 %. Form (135 raw → 4 / 6): p50 −3.7 % / −2.7 %, p95 +0.9 % / −6.8 %. Mixed sign, inside noise on trees of this size; deep-tree (263 raw) recovers to private-ax on both heads. Unmeasured: a runner-served tree in the thousands of nodes, where the node cap is the bound. Full tables in the thread.

@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.62 MB 4.61 MB -10.1 kB
Package (unpacked) 4.62 MB 4.61 MB -10.1 kB
Package (download) 1.37 MB 1.37 MB -2.8 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 26.3 ms 26.8 ms +0.5 ms
CLI --help 77.1 ms 80.0 ms +2.8 ms

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at dccb40f. The frontier still reads pre-normalization geometry, so a regular snapshot -d N can walk a different subtree than main on a turned tree, and the issue's live checks are still missing, so this isn't ready yet.

When regularPresentedDepth is set, recursiveTreeSnapshotAcquisition still calls regularTraversalTransition mid-walk (https://github.com/callstack/agent-device/blob/dccb40f/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift#L223), and that reads node.rect in SnapshotVisibilityFold.traversalDecision (https://github.com/callstack/agent-device/blob/dccb40f/apple/snapshot-presentation/Sources/AgentDeviceSnapshotPresentation/SnapshotVisibilityFold.swift#L109) before the post-pass has turned it to app orientation. For a landscape keyboard subtree the native-space rect can put a visible owned container outside the viewport, so intersects/projectedOut and the presented-depth increment are wrong, the walk skips its children, and the post-pass can't recover what was never acquired. This is the exact thing the issue rules out: a kept bound must not consult presentation, only a raw depth or node cap. The rule this needs to satisfy is: no acquisition-time decision reads geometry that hasn't been normalized, and that has to hold for every caller of regularTraversalTransition inside the recursive walk, not just this one call site. Could you replace it with a presentation-free cap as the issue asks, measure it per issue step 1, record the choice in ADR 0004, and add a runner unit test with a turned keyboard subtree under -d N that fails on current head?

The issue's acceptance criteria list four live results — portrait snapshot -i --json byte-identical to main, a landscape keyboard band and key q rect, deep-tree/Bluesky p50/p95 within 5% of main, and the frontier measurement from step 1 — and the PR doesn't show them yet. The runner unit tests on the iOS 26.2 simulator (26/26) pin the landscape band, but they don't exercise the live daemon path (https://github.com/callstack/agent-device/blob/dccb40f/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift#L479). Since this changes the geometry of every runner snapshot tier on the device path, could you run from this head through the daemon on the test app: snapshot -i --json diffed byte-for-byte against main with no hittability-evidence warning, snapshot --json and snapshot -d 3 --json in landscape with the keyboard up showing the band and key q rects, and pnpm bench:ios-snapshot deep-tree/Bluesky p50/p95 next to main, plus the raw node count walked with and without the frontier?

Not blocking: the ADR and the comment near RunnerTests+Snapshot.swift#L219 say a duplicate's children attach to "the first node it duplicated," but the code attaches them to entry.parentIndex (the duplicate's parent), and the ADR's "never itself de-duplicated" equivalence argument misses the Application-box case, which is benign only because Application is also a surface-host type — worth fixing the wording and stating the real reason, but not urgent.

The post-pass itself is small, about 70 lines in normalized(). Is moving the whole rotation table into the presentation package as public API — including the dispatch-side native(point:)/native(vector:) and a runner-named RunnerInterfaceOrientation — necessary for that, or would keeping the dispatch direction in the runner and moving only what normalization needs (the oriented(rect:) direction and the space rule) get the same result with less surface? If the one-table round-trip invariant is the reason to keep both directions together, could the PR say so? And does deleting the fold-consulting frontier in favor of a raw cap remove the need to move regularTraversalTransition at all?

I didn't run the runner on a simulator, so the frontier divergence is inferred from the code, not observed; I can't say whether the real landscape keyboard tree has an owned Cell or scroll container that a native-space rect would actually project out, or whether it only shifts presented depth, and I didn't check whether any existing runner keyboard fixture already exercises the recursive walk with regularPresentedDepth set. The size growth is attributed to the moved runner and presentation sources by reading the diff, not by reproducing a size report.

Repo Guards failed in scripts/swift-toolchain-tmpdir.test.ts ("keeps TMPDIR until a signaled child exits") with ENOENT on shutdown.json; this diff only touches Swift sources under apple/ and the ADR, so it looks unrelated to that signal-handshake test and likely a flake.

Before this can merge, the frontier needs to stop reading unnormalized rects mid-walk, and the portrait, landscape, and benchmark evidence above needs to come from a live run against this head.

@thymikee

Copy link
Copy Markdown
Member Author

Overlap with #2664 (opened earlier today, 25 files, CI green, waiting on one live check). #2664 already moves the rotation table into the package as SnapshotCoordinateSpace.swift with CoordinateSpaceTests.swift, changes isSurfaceHost to isWindow:parentIsWindow: booleans, and adds SnapshotGeometrySpace.unplacedSurfaceHostCount plus the unresolvedCoordinateSpaceWindows verdict field. git merge-tree of the two heads conflicts on four files: add/add on the package source, content on RunnerTests+Snapshot.swift, RunnerTests+PrivateAXPresentation.swift and the package test file.

Proposed order: #2664 lands first, then this rebases onto main and becomes the behaviour layer only:

That also takes the move out of your diff, which should bring it under the 1,000 budget without a split. Separately, Repo Guards is red on this head (tmpdir-leaks-model gate); that needs a look regardless of the rebase.

@thymikee

Copy link
Copy Markdown
Member Author

Answers to the two questions, plus one correction to my earlier note.

Correction first. #2664 was trimmed to the package move only (head 321aef2). unplacedSurfaceHostCount and the unresolvedCoordinateSpaceWindows disclosure no longer exist — I cut them because nobody can show the disclosure firing on its one target path (details on #2661). So: do not call unplacedSurfaceHostCount after normalized(), and do not add a disclosure. The API you write against is exactly what is on 321aef2: RunnerInterfaceOrientation, CoordinateSpaceRotation.native/oriented, SnapshotGeometrySpace.space(...), orientedFrame(of:), isSurfaceHost(isWindow:parentIsWindow:). That file is written comment-free by the package's convention (every other file in Sources/ has zero prose and ships verbatim); your normalized(nodes:viewport:interfaceOrientation:) goes in it the same way, and its tests go into CoordinateSpaceTests.swift.

Q1 — overlap: retarget onto #2664's branch. 321aef2 is stable; I am not changing it again. Rewrite onto that API now and prove it; when #2664 merges, GitHub retargets this PR to main and your rebase is mechanical. The "keep base main, strip the move but keep SnapshotGeometrySpace in the runner" option is the one to rule out — it keeps RunnerCoordinateSpace.swift, which #2664 deletes, so it guarantees an add/add plus content conflict at merge. Waiting is safe but idles you on the maintainer's merge timing.

Q2 — the frontier: drop the fold-based frontier from the walk and present-cut on the normalized array, the way the private-AX tier already does. "Normalize-then-frontier" is #2653's threading with a different name: it keeps geometry computed inside the walk, which is the coupling #2661 exists to remove. The frontier's legitimate job is a work bound, and a work bound must not consult presentation — issue #2661 step 1 says exactly this: keep a raw depth plus node cap (rawSnapshotTooLargeFailure is already that), never a fold decision.

The byte-identity rule you should hold yourself to:

One thing to watch: the frontier also feeds the scroll hints (hiddenContentAbove/Below) at presented boundary containers. A post-walk fold computes them from real children instead of unwalked ones. If that alone moves a portrait --depth N byte, that is the finding to write up.

Repo Guards: agreed it is the tmpdir-leaks-model signal race, unrelated to your Swift; re-run it.

@thymikee
thymikee changed the base branch from main to fix/ios-snapshot-failure-scope-and-geometry-package September 18, 2026 14:31
@thymikee
thymikee added this pull request to stack #2671 September 18, 2026 14:38
@thymikee

Copy link
Copy Markdown
Member Author

That settles the design question for dccb40f: the rotation-table move belongs to #2664, and this PR rewrites onto the API at 321aef2. The frontier plan (drop the fold-based frontier, keep a raw depth and node cap, and present-cut on the normalized array) matches the open finding.

Before this is ready, the next head needs that rewrite with the #2661 step 1 measurement, the live portrait, landscape and benchmark runs from the earlier comment, and a rebase, because the PR now conflicts with its new base.

Base automatically changed from fix/ios-snapshot-failure-scope-and-geometry-package to main September 19, 2026 06:06
@thymikee
thymikee force-pushed the t3code/implement-issue-2661 branch 2 times, most recently from ae023d5 to 47b3aa1 Compare September 19, 2026 06:38
@thymikee

Copy link
Copy Markdown
Member Author

Rewritten onto the merged move (this head rebases onto main after #2664 merged as a556e114) and pointed at exactly the 321aef2 API.

Q1 — done. #2664's move is gone from this PR; I now use SnapshotGeometrySpace.normalized from the package. normalized goes into SnapshotCoordinateSpace.swift comment-free and its tests into CoordinateSpaceTests.swift. No unplacedSurfaceHostCount, no disclosure.

Q2 — done, dropped the fold frontier. No geometry read remains in the walk. A regular capture bounds itself by raw traversal depth (nil) plus a runaway node cap; the fold + presented-depth cut happen in SnapshotPresentation on the normalized array. I added a regular-family node cap because the existing rawSnapshotTooLargeFailure cap guards the separate raw walk — it lands on the same generic capture-failed quality reason, no new registry surface. The de-dup wording in ADR 0004 is fixed (a repeated node re-parents its children onto its own parent). I also had to reclassify the tree backend's regular-depth capability presented-frontier → presentation-cut across the Swift enum, the parity fixture, and the TS registry (Android keeps presented-frontier), and update the ios.yml PR-lane list.

Proven in CI/locus, head 47b3aa12: package 14/14, runner unit tests green on an iOS 26.2 simulator (incl. a new -d N regression that a turned keyboard subtree is acquired regardless of its reported rect and the band/q present at the #2653 fixture's app-space rect), capture-kit parity table green, check:affected --run all runnable checks passed, selection/fallow/lint/format clean.

Live runs — need a decision before I burn a baseline build. The deterministic suite can't settle your byte-identity rule because the frontier also fed the presented-depth boundary scroll hints, so a portrait --depth N could legitimately move a hiddenContentAbove/Below byte (your "finding to write up" case), and the bench:ios-snapshot harness measures latency (it's the #2189 perf harness), not snapshot bytes. To run the three live gates I need:

  1. The target app — the checked-in com.callstack.agentdevicelab / AgentDeviceTester, or a specific Safari URL for the landscape band?
  2. Device — is this iOS 26.2 simulator acceptable, or is iPhone 17 Pro required (I can only confirm the iOS 26.2 runtime here, not the model)?
  3. The portrait byte-identity check needs a main-build runner baseline to diff against — do you want me to build main + head and A/B on a named screen, and which screen anchors the portrait -i / --depth N comparison?

Repo Guards is a fresh run on this head (not the tmpdir-flake run); I'll rerun it if it goes red again.

@thymikee

Copy link
Copy Markdown
Member Author

Checked 47b3aa1 against the design: followed point for point. No copy of the move, no geometrySpace/parentIsWindow threading, no fold-consulting frontier in the walk, normalized in the package with tests in CoordinateSpaceTests.swift, no disclosure. The #2661 AC "exactly one production SnapshotGeometrySpace site" holds: the only call is captureWithBackend (RunnerTests+SnapshotCapturePlan.swift:480); every other hit is a comment or a #if AGENT_DEVICE_RUNNER_UNIT_TESTS block. Reclassifying the tree backend's regular-depth capability to presentation-cut across Swift, the parity fixture and the TS registry is the right consequence and the honest one.

Your "still owed" list is the merge gate, same one #2664 was held to: portrait -i and --depth N byte-identical to a main baseline, landscape band live, and the step-1 frontier benchmark on a hostile screen. The scroll-hint caveat you named (fold now sees real children at a presented boundary) is exactly where a portrait byte would move if anything does; if it does, write it up rather than re-couple.

@thymikee
thymikee force-pushed the t3code/implement-issue-2661 branch from 47b3aa1 to 80ac6ae Compare September 19, 2026 07:06
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed 80ac6ae, following up on dccb40f. Two things block this.

Regular iOS snapshots now take a new route: acquisition publishes reported frames and one normalized pass follows, so regular --depth N walks the whole tree (https://github.com/callstack/agent-device/blob/80ac6ae/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift#L476). This changes snapshot output and --depth cost on real apps, and no live run has been posted for 80ac6ae. Can you post main vs 80ac6ae runs on one named screen in the test app? The runs needed are: portrait snapshot -i and snapshot -d 2/-d 3 diffed byte for byte, with every diff line explained and the boundary scroll hints listed; landscape with the keyboard up, snapshot -d N on the #2653 form fixture, showing the band at (75,198,724,202); and the step-1 --depth N latency and node count on the Bluesky feed, with the runner's tree_processing inside treeCaptureSliceBudget.

The new SnapshotRect.init(_ rect: CGRect) (https://github.com/callstack/agent-device/blob/80ac6ae/apple/snapshot-presentation/Sources/AgentDeviceSnapshotPresentation/SnapshotModels.swift#L21) copies the runner's snapshotRect(from:) field for field (https://github.com/callstack/agent-device/blob/80ac6ae/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift#L823), and the package keeps a third copy in the private SnapshotPresentationInvariant.snapshotRect(from:) (https://github.com/callstack/agent-device/blob/80ac6ae/apple/snapshot-presentation/Sources/AgentDeviceSnapshotPresentation/SnapshotPresentationInvariant.swift#L83). Can every CGRect-to-SnapshotRect conversion go through SnapshotRect.init(_:), with the other two copies deleted?

Not blocking, take or leave: a CHANGELOG line, an ADR sentence and a test for the boundary container that now carries scroll hints under --depth N; the hand-built normalization test never runs the walk; and the ADR attributes the frontier removal to mis-pruning, while the reason is that a post-walk pass cannot feed a mid-walk fold.

The package and runner test results in the body are from 47b3aa1; the 47b3aa1..80ac6ae delta reads as a tidy. All checks now pass on 80ac6ae, including the iOS smoke lane that runs the rewritten route; that is not a substitute for the runs above. There are no conflicts. The next step is the live runs above and the single CGRect conversion.

@thymikee

Copy link
Copy Markdown
Member Author

Live evidence for 47b3aa1, run through the daemon on iPhone 17 Pro (iOS 26.2) with the Release test app, against a main daemon on the same simulator and the same navigation. Every capture below is runner-served (snapshotQuality.backend: tree, circuit opened through the webview screen; no hittability-evidence warning anywhere).

Portrait, byte-identical to main (refs stripped):

screen -i --json --depth 3
form, keyboard down 28 = 28 ✓ 42 = 42 ✓ 6 = 6 ✓
form, keyboard up (Full name focused) 74 = 74 ✓ 94 = 94 ✓ 10 = 10 ✓

So the presentation cut over a full walk reproduces the frontier-bounded walk exactly, including the presented-depth boundary scroll hints you were watching.

Landscape, keyboard up, identical on both heads: --json publishes the Keyboard plane at (75, 238, 724, 162) and key Q at (77, 243, 72.3, 35.3) on 2669 and on main (66 nodes -i, 84 nodes --json on each). At --depth 3 the keyboard plane sits deeper than 3 on this screen, so neither head presents it and the two trees are identical (10 = 10). The "landscape -d N may differ from main" case therefore did not materialize here: on a form the app scrolls above the keyboard, so the band never crosses a presented-depth boundary. That is a narrower live result than "different and correct"; it is at least "not worse".

Frontier measurement (#2661 step 1). 7 warm captures per cell, snapshotQuality.timing.acquisitionMs:

screen (recursive tree tier, snapshot -i) main p50 / p95 2669 p50 / p95 Δ p50 / Δ p95 presented / full nodes
form (xctest-stress fixture) 69 / 75 ms 67 / 68 ms −3.7 % / −9.3 % 28 / 42, identical
catalog list 169 / 170 ms 167 / 171 ms −0.8 % / +0.7 % 33 / 46, identical
deep-tree 653 / 661 ms 649 / 664 ms −0.6 % / +0.5 % 164 / 246, identical

The deep-tree row is the plan recovering to private-ax on both heads (the recursive tier is rejected on that screen), so it never exercises the frontier; it is here because the issue named it, and it shows the change is inert where the frontier was already absent. The form and catalog rows are the recursive tier proper: the fold-consulting frontier saved nothing measurable, which is what step 1 asked to establish before dropping it, and the raw depth + node cap you kept is the bound that remains. No larger raw cap is needed for these screens.

Not covered: Bluesky (not installed on this machine), a physical iPhone, and the runner's walked-node count as a number — the wire carries presented counts and acquisition time, not the walk size; acquisition time is the measure of what the frontier bought.

thymikee added a commit that referenced this pull request Sep 19, 2026
…eal reason the frontier went

Review follow-ups on #2669.

- Every CGRect becomes a SnapshotRect through SnapshotRect.init(_:). The runner's snapshotRect(from:)
  and the package's private SnapshotPresentationInvariant.snapshotRect(from:) were field-for-field
  copies; both are deleted and their nine call sites use the initializer.
- ADR 0004 attributes the frontier removal to its cause: a post-walk normalization pass cannot feed a
  decision the walk already took, so the fold-consulting frontier had to go; the mis-cut keyboard
  band was the symptom. The measured cost of dropping it (form, catalog: p50 within 4 %, p95 within
  10 % of main, in the branch's favour) is recorded there.
- CHANGELOG line for the presentation-cut route and the capability rename.
@thymikee

Copy link
Copy Markdown
Member Author

Addressed on 56fcf4e (fast-forward from 80ac6ae).

Single CGRect → SnapshotRect conversion. Every conversion goes through SnapshotRect.init(_:); the runner's snapshotRect(from:) and the package's private SnapshotPresentationInvariant.snapshotRect(from:) are deleted and their nine call sites use the initializer (SnapshotGeometry.snapshotRect(from:reportedFrame:) is a different function — effective frame with fallback — and stays). ADR 0004 now attributes the frontier removal to its cause (a post-walk normalization pass cannot feed a decision the walk already took; the mis-cut band was the symptom) and records the measurement; CHANGELOG line added. swift test 14/14, the 8 runner units on the rewritten route green on a simulator, check:xctest-selection and check:packaged-runner-swift green.

Live runs on this head vs a main daemon, same simulator, same navigation, open --relaunch before each leg so the form's scroll offset is identical. iPhone 17 Pro (iOS 26.2), Release test app, every capture runner-served (snapshotQuality.backend: tree, circuit opened via the webview screen). Byte-identical means data.nodes equal with ref stripped.

form, portrait -i --depth 2 --depth 3 --raw --json
keyboard down 27 = 27 ✓ 4 = 4 ✓ 6 = 6 ✓
keyboard up, Full name focused 73 = 73 ✓ 7 = 7 ✓ 10 = 10 ✓ 207 = 207 ✓ 93 = 93 ✓

Scroll-hint carriers under -i on both heads: the Checkout form ScrollView with hiddenContentBelow: true and nothing else; under --depth 2/3 none survive the cut on either head. So the presentation cut over a full walk reproduces main exactly, boundary hints included.

Every diff line in the runs that were not identical, explained. Two earlier passes differed and I chased both down rather than averaging them away:

  1. Keyboard down -i, 27 vs 28. Every rect on the head leg was 10 pt lower and one 3-pt ime-capture-diagnostic strip was missing: the form's scroll offset carried over from the previous leg, and at +10 pt that strip slid under the tab bar and was viewport-clipped. Relaunching before each leg removed it (27 = 27 above).
  2. Keyboard up, 70 vs 73/74 and landscape 80 vs 84. The nodes only on main were exactly the QuickType prediction cells I, The, I’m (plus their row container in landscape) under a Typing Predictions container that was present on both. iOS populates those per keyboard session, all-or-nothing, so within a leg it looked deterministic. Bisected by capturing 80ac6ae, 56fcf4e and main in three fresh relaunched sessions: all three produce byte-identical -i (73), --raw (207) and --json (93) trees with the predictions present. Not a presentation change.

Landscape, keyboard up, --depth N on the #2653 form fixture. On this head with the QuickType bar shown: Keyboard plane at (75, 198, 724, 202) and key Q at (77, 203, 72.3, 45.3) at --depth 5 (band), --depth 7 (keys) and --json — the #2653 numbers to the point. With the bar hidden (the other iOS state, seen on both heads): plane (75, 238, 724, 162), Q (77, 243, 72.3, 35.3), identical on head and main at --depth 5 (same 26-node set) and differing at --depth 7/--json only by the prediction cells above. At --depth 3 the plane is below the cut on both heads.

Step-1 frontier measurement, 7 warm captures per cell, snapshotQuality.timing.acquisitionMs:

recursive tree tier, snapshot -i main p50 / p95 head p50 / p95 Δ
form (xctest-stress fixture) 69 / 75 ms 67 / 68 ms −3.7 % / −9.3 %
catalog list 169 / 170 ms 167 / 171 ms −0.8 % / +0.7 %
deep-tree 653 / 661 ms 649 / 664 ms ±1 % — recovers to private-ax on both heads, so it never exercises the frontier

The frontier bought nothing measurable; the raw depth + node cap is the bound that remains.

Not done, and why. Bluesky is not installed on this machine and no seeded simulator exists, so the Bluesky feed row and the tree_processing-inside-treeCaptureSliceBudget check on it are not here; the runner's runner.log for these sessions carries no tree_processing line either. Physical iPhone not measured.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed 56fcf4e, a follow-up to 80ac6ae (#2669 (comment)). The CGRect point is fixed: all nine conversions now use SnapshotRect.init(_:), the two copies are gone, and SnapshotGeometry.snapshotRect(from:reportedFrame:) stays because it handles empty frames differently. The posted output-parity runs still cover this head, because the delta does not touch the walker.

One gap is still open: the cost of --depth N. On main the frontier prunes only when a presented depth is set, and that comes from options.depth (https://github.com/callstack/agent-device/blob/a556e11/apple/snapshot-presentation/Sources/AgentDeviceSnapshotPresentation/SnapshotPresentation.swift#L63-L81). snapshot -i sends no depth. So every row of the step-1 timing table ran the same full walk on both heads, and the new ADR sentence ("the frontier bought nothing", https://github.com/callstack/agent-device/blob/56fcf4e/docs/adr/0004-ios-snapshot-backend-strategy.md#L319) rests on runs that never used the frontier. The route this PR changes on purpose is regular snapshot --depth N, which now walks the whole tree up to the 50k node cap (https://github.com/callstack/agent-device/blob/56fcf4e/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift#L199-L240), and it has no timing yet.

Can you post acquisitionMs p50/p95 (7 warm captures) on main vs 56fcf4e for snapshot --depth 2 and snapshot --depth 3? Please use a runner-served recursive tree screen where the depth cut drops most of the tree, for example the catalog list scrolled so rows are off-screen, and put the --raw node count next to the presented count. Bluesky is not needed. Then cite those numbers in the ADR sentence, or reword it to say only what was measured.

All 18 checks pass on 56fcf4e, but CI does not measure this cost. There are no conflicts. The next step is the --depth 2/--depth 3 timing and the ADR fix.

…ion (#2661)

Move the coordinate-rotation table and SnapshotGeometrySpace into the snapshot-presentation package and add SnapshotGeometrySpace.normalized(nodes:viewport:interfaceOrientation:). Acquisition now serializes reported frames only; captureWithBackend settles every node into the app's orientation space in one ancestry-keyed pass instead of threading geometrySpace/parentIsWindow through the recursive, raw, private-AX, and query-sweep walkers.

Drop the fold-consulting regular-depth frontier. It read reported (un-normalized) rects mid-walk to increment presented depth, so a turned keyboard band reported in the device's native space could be pruned before the single pass ever moved it back into the app's viewport. A regular --depth is now a presentation cut only: the walk's bounds are raw traversal depth (nil for a regular capture) plus a runaway node cap, and SnapshotPresentation applies the visibility fold and the presented-depth cut on the normalized array.

Reclassify the iOS tree backend's regular-depth capability from presented-frontier to presentation-cut across the Swift declaration, the cross-runtime parity fixture, and the TypeScript registry, and remove the now-unused Swift enum case (Android's helper keeps presented-frontier). Update ADR 0004 and the ios.yml PR-lane test list.
@thymikee
thymikee force-pushed the t3code/implement-issue-2661 branch from 56fcf4e to ed75ffb Compare September 19, 2026 10:10
…eal reason the frontier went

Review follow-ups on #2669.

- Every CGRect becomes a SnapshotRect through SnapshotRect.init(_:). The runner's snapshotRect(from:)
  and the package's private SnapshotPresentationInvariant.snapshotRect(from:) were field-for-field
  copies; both are deleted and their nine call sites use the initializer.
- ADR 0004 attributes the frontier removal to its cause: a post-walk normalization pass cannot feed a
  decision the walk already took, so the fold-consulting frontier had to go; the mis-cut keyboard
  band was the symptom. The measured cost of dropping it (form, catalog: p50 within 4 %, p95 within
  10 % of main, in the branch's favour) is recorded there.
- CHANGELOG line for the presentation-cut route and the capability rename.
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed d1ae801, a rebase of 56fcf4e onto main after #2670. The only change to the patch is the ADR conflict resolution, and it keeps both the new keyboard-fact text and this PR's one-pass paragraph. The keyboard band from #2670 is read in the app's own space and carried beside the tree, so the new normalization pass does not change what the tap guard compares.

The code is otherwise the same, so the open item from the last review (#2669 (comment)) still stands: can you post acquisitionMs p50/p95 on main vs this head for snapshot --depth 2 and snapshot --depth 3 on a screen where the depth cut drops most of the tree? The ADR sentence that says the frontier bought nothing (https://github.com/callstack/agent-device/blob/d1ae801/docs/adr/0004-ios-snapshot-backend-strategy.md#L320) still rests on runs that never used it.

All 18 checks pass and there are no conflicts. The next step is still the --depth N timing and the ADR fix.

…on the route where it engaged

The earlier sentence rested on snapshot -i runs, which carry no presented depth and never engaged
the frontier on main. Re-measured with --depth 2 and --depth 3 (n=15 per cell) on the recursive
tier: form 135 raw nodes and scrolled catalog 279 raw nodes walked in full to present 4 or 6,
acquisition p50 within ±4 % and p95 within ±7 % of main, mixed sign.
@thymikee

Copy link
Copy Markdown
Member Author

Addressed on f69fa5c (fast-forward from d1ae801; ADR only).

You were right: -i sends no depth, so the step-1 table never engaged the frontier on main. Re-measured with a regular --depth 2 and --depth 3, which is the only route where main pruned.

Setup. Same simulator (iPhone 17 Pro, iOS 26.2), Release test app, main (4c2ae0c) daemon vs d1ae801 daemon (the ADR commit does not touch code), open --relaunch before each leg, one warm-up capture dropped, then n captures per cell; acquisitionMs from snapshotQuality.timing, --raw count on the same screen, presented count from the same captures (identical on both heads in every cell). Catalog is the list scrolled so most rows are off-screen; all catalog/form cells were runner-served tree.

n = 15 per cell

screen (--raw nodes) depth presented main p50 / p95 head p50 / p95 Δ p50 / Δ p95
catalog (279) 2 4 232.6 / 245.7 237.6 / 247.3 +2.1 % / +0.7 %
catalog (279) 3 6 235.6 / 245.2 234.3 / 242.5 −0.6 % / −1.1 %
form (135) 2 4 92.8 / 95.5 89.4 / 96.4 −3.7 % / +0.9 %
form (135) 3 6 91.8 / 97.5 89.3 / 90.9 −2.7 % / −6.8 %

n = 7 per cell (the first run, kept for honesty; it is what prompted the n = 15 re-run)

screen (--raw nodes) depth presented main p50 / p95 head p50 / p95 Δ p50 / Δ p95
catalog (279) 2 4 227.1 / 233.7 231.6 / 243.7 +2.0 % / +4.3 %
catalog (279) 3 6 234.1 / 252.0 239.8 / 294.7 +2.4 % / +16.9 %
form (135) 2 4 87.3 / 89.5 92.8 / 103.3 +6.3 % / +15.4 %
form (135) 3 6 88.9 / 90.4 90.8 / 98.1 +2.1 % / +8.5 %
deep-tree (263, private-ax on both) 2 7 632.7 / 638.3 629.6 / 637.6 −0.5 % / −0.1 %
deep-tree (263, private-ax on both) 3 10 636.5 / 640.3 633.7 / 638.0 −0.4 % / −0.4 %

Reading. At n = 7 the head looked a few ms slower at p50 and the p95 tails were wide (one 315 ms catalog outlier drives the +16.9 %); at n = 15 the sign flips per cell and every Δ is inside ±4 % p50 / ±7 % p95, so on these trees the full walk over the platform's already-materialized XCUIElementSnapshot is not separable from noise. The frontier's saving would have been node construction for the pruned subtree, and 135–279 nodes is not enough for that to show. What this does not say: the cost on a much larger runner-served tree (thousands of nodes) is unmeasured; the bound there is the node cap. Deep-tree recovers to private-ax on both heads, so it is untouched.

ADR. The "frontier bought nothing" sentence is replaced with what was measured: that the frontier engaged only under a regular --depth N, the n = 15 cells above (raw and presented counts, ±4 % / ±7 %, mixed sign), that acquisition time goes to the already-materialized snapshot rather than to node construction, and that the node cap is the remaining bound.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at f69fa5c. The new numbers answer the open item: they use --depth 2 and --depth 3, the route where main pruned, on screens where the cut keeps 4 to 6 of 135 or 279 raw nodes. The ADR text now matches the n = 15 table (p50 within ±4 %, p95 within ±7 %, mixed sign). The code is the same as d1ae801.

Not blocking: flatSnapshotNode at RunnerTests+Snapshot.swift#L1341 still computes hittable, but the normalization pass overwrites it, so that computation can go.

All 18 checks pass, and there are no conflicts. No code blockers remain.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 19, 2026
@thymikee
thymikee merged commit f4930d6 into main Sep 19, 2026
18 checks passed
@thymikee
thymikee deleted the t3code/implement-issue-2661 branch September 19, 2026 12:35
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-19 12:35 UTC

thymikee added a commit that referenced this pull request Sep 19, 2026
…2684)

Since #2669 the one normalization pass in captureWithBackend recomputes hittable for every
acquired node, so the value flatSnapshotNode computed from the reported frame was overwritten
before anything read it. The sweep now passes hittable: false like the recursive walker and
private AX do, and the viewport parameter that only fed that computation goes with it.
No presented output changes. Raised as a non-blocking note on #2669.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant