fix(ios): publish captured keyboard geometry in the app's orientation space - #2653
Conversation
|
Reviewed at 9b0935c. The rotation logic looks right: the rect transform is the exact inverse of the touch rotation, only nodes under a turned surface host are rotated, and the bridge refusal sends only that one capture to the runner, keyed on the typed One size question before this is ready. The production diff is about 771 lines, which is over the 700-line review threshold, although much of The unit tests do not cover the walk wiring. Every In Two smaller notes. In Two questions on the live run. The reported key Only the CodeQL checks have run on this head. The main CI jobs, including Integration Tests and Coverage, have not reported yet. The PR conflicts with main in |
… space iOS hosts the system keyboard in the device's native portrait space, so in landscape its whole subtree arrives quarter-turned: a key measured 45x72 at x 154 in an 874x402 app, drawing a strip down the left edge where the screenshot shows a 724x204 band docked at y 198. Rules that read those numbers refused app content the keyboard was nowhere near and let a tap land on a key (#2612). The Apple runner now reads the app's interface orientation at capture time and publishes every rect under a turned surface host in the app's space, through the exact inverse of the rotation its synthesized touches rotate forward by, so a reported rect and a performed tap cannot disagree about which pixel is which. One table serves both directions, and the new coordinate-space module owns it. An orientation that names no quarter turn declares nothing, and the collapsed-tab expansion, which reads live element frames, stays out of a declared native space rather than turning them twice. The Simulator AX bridge reader carries no interface orientation, so it counts surface hosts reporting the app box quarter-turned and refuses that capture with window-coordinate-space-unresolved; the route serves the runner for that capture alone rather than retiring the app generation, so the next capture of a healthy app still uses the bridge. Both languages replay one table, contracts/fixtures/window-coordinate-space.json, which pins the turn and the box that declares it.
9b0935c to
bc80c28
Compare
|
Rebased and addressed at Size question — took it. Walk wiring —
Portrait/unknown and upside-down — an orientation that names no quarter turn now declares nothing, so no app-frame origin is ever added back and the doc comment says what the code does. Upside-down is called out as what the box rule cannot decide: an upside-down app's own box is already the native box, which is also why the consumer rules stay. Symmetry — The two live-run questions. The Still open: the main CI jobs (Integration Tests, Coverage) on this head, and iPad or physical-device keyboards, which stay unmeasured. |
Size Report
Startup median (7 runs, lower is better):
|
|
Reviewed at bc80c28. The points from the 9b0935c review are addressed. Not blocking: after the guard at line 1172, I did not run the gated Swift XCTests; I checked the new test's expected rect by hand. CI is green: 18 checks, 0 failing at bc80c28. |
thymikee
added
the
ready-for-human
|
1 participant
Summary
iOS hosts
UIRemoteKeyboardWindowin the device's native portrait space, so in landscape the keyboard's whole subtree arrives quarter-turned: a key measured 45x72 atx 154inside an 874x402 app, drawing a strip down the left edge where the screenshot shows a 724x204 band aty 198. Rules that read those numbers refused app content the keyboard was nowhere near and let a tap land on a key.The Apple runner now reads the app's interface orientation at capture time and publishes every rect under a turned surface host in the app's space, using the exact inverse of the rotation its synthesized touches rotate forward by — one table, both directions, so a reported rect and a performed tap cannot disagree about which pixel is which. The Simulator AX bridge reader carries no orientation, so it refuses such a capture (
window-coordinate-space-unresolved) and the route serves the runner for that capture only, without retiring the app generation. Contracts pins both rules in one table replayed in TypeScript and Swift.21 files, 630 production lines; the 1.64k gross adds 372 unit-test-only Swift lines behind
#if AGENT_DEVICE_RUNNER_UNIT_TESTS, the golden table and its replays, and an ADR amendment. The rotation table moved into the newRunnerCoordinateSpace.swiftout ofRunnerTests+Interaction.swift; the rule's TypeScript home is private tosnapshot-source, so no new package export is declared.Closes #2612.
Validation
Head
9b0935cc61.pnpm check:affected --runexit 0; unit suite 1333 files / 10,383 tests; lint and typecheck clean. 6 runner XCTest units on iPhone 17 Pro (iOS 26.2). Device: afterorientation landscape-leftwith the keyboard up, the capture warns it left the bridge and publishes the band aty 238and keyqat(77,243,72,35); pressing the tab item inside the band is refused, the next capture is served by the bridge again, and portrait is unchanged. Unresolved: coverage and Integration Tests jobs on this head; iPad and physical-device keyboards unmeasured.