Treat interaction targets behind the visible keyboard as occluded/off-screen (parity with scroll)

[thymikee]

Purpose

When the software keyboard is up, elements it visually covers still report hittable: true with an on-screen rect, so press/click/fill/longpress tap through the keyboard and silently activate whatever the keyboard's own hit-testing routes the point to (or nothing). Keyboard occlusion is a normal, expected iOS/Android pattern — apps are not going to move their tab bar above the IME — so the fix belongs in agent-device: an interaction target whose tap point falls under the visible keyboard should be treated as occluded/off-screen and refused with a typed reason and a recovery hint, exactly the way scroll already does.

This came out of dogfooding MaintainX on a physical iPhone (thymikee-iphone, v5.39-311017).

Context / evidence

On the Messages tab of MaintainX, focusing the search field raised the keyboard over the bottom tab bar. The tab-bar nodes stayed in the app snapshot with their layout geometry and were still hittable:

Button "More"     center=(340,822)  rect={"x":303,"y":795,"width":74,"height":54}
Button "Messages" center=(274,822)  ...

The keyboard occupies roughly y≈540..874, so the More tab is fully behind it. press 340 822 (and press @more) did not navigate; the touch landed on the keyboard. The AX overlay below shows the tab-bar refs drawn under the keys:

The annotated overlay (tab-bar refs @e40–@e46 drawn under the keys) and a reproduction video are in the dogfood evidence bundle (bug3-keyboard-covers-tabbar.png, bug3-keyboard-tabbar.mp4); they need a manual web upload to attach here.

(Reproduction video: local bug3-keyboard-tabbar.mp4 — More tab works with no keyboard, then is dead once the search field is focused. Same class of silent misfire on Android, whose IME is also a separate window.)

Why the current guards miss it

• occlusion (ADR 0011) refuses targets "covered by another visible element" — but occlusion is computed from same-window drawing order in the snapshot tree. The keyboard is a separate window (a system surface), so it never appears as a covering sibling of app elements.
• offscreen refuses targets whose tap point is outside the root Application/Window viewport (isTapPointInsideViewport / isVisibleOnScreen in packages/contracts/src/snapshot-visibility.ts). The covered tab bar is still inside the app window rect, so it passes.

scroll already has this exact notion and does the right thing: SCROLL_KEYBOARD_OCCLUDES_SURFACE_REASON (packages/contracts/src/scroll-command.ts, packages/contracts/src/scroll-gesture.ts, golden table contracts/fixtures/scroll-keyboard-policy.json) and the documented behavior in src/commands/interaction/metadata.ts:

A visible keyboard shortens the swiped band instead of being dismissed; when too little is left, the command refuses with scroll_keyboard_occludes_surface.

Interaction tap paths have no equivalent, which is the asymmetry this issue closes.

Required behavior

• On press/click/fill/longpress (and their --settle/native-ref fast paths), when a software keyboard is visible, compute the keyboard's occluding frame and treat any target whose tap point (rect center) falls inside it as occluded/off-screen.
• Fail with a typed reason on that interaction guarantee (mirroring scroll_keyboard_occludes_surface), plus a hint to dismiss the keyboard first (reference keyboard dismiss, and note iOS only dismisses via the keyboard's own control per fix(ios): give keyboard dismiss a safe-area-tap fallback (#1598) #1606 / mechanism: 'dismissKey' in packages/contracts/src/keyboard.ts).
• Do NOT auto-dismiss the keyboard as a side effect — dismissal has semantic consequences (e.g. commits/cancels edit state); refuse and let the caller decide, consistent with the scroll policy.
• Apply to both platforms: iOS software keyboard and Android IME (focusedPackage/inputMethodPackage path already surfaced in KeyboardCommandResult). Coordinate-only taps must be guarded too, since that is how agents hit-tested past the tree here.
• Keyboard-frame source: derive from the already-captured snapshot (inputView / keyboard nodes are present in the iOS tree) and/or the runner's keyboard-frame API; on Android from the visible IME / visible-to-user inset. If the keyboard frame is unavailable on a backend, fail open (tap proceeds) to avoid regressing current behavior, but record the miss on a typed reason/detail rather than guessing.

Completion conditions

• A coordinate or @ref tap on an app element fully behind the visible keyboard is refused with the new typed reason + dismiss hint, and a follow-up after keyboard dismiss succeeds.
• An element only partially under the keyboard whose tap point (center) is still above it continues to tap normally (center rule, consistent with isTapPointInsideViewport).
• Golden/parity coverage: a keyboard-occlusion fixture under contracts/fixtures/ (like scroll-keyboard-policy.json) and an entry for the new path/guarantee stance in packages/contracts/src/interaction-guarantees.ts for occlusion and/or offscreen.
• Versioned CLI help / metadata updated to name the new refusal reason.

API / CLI shape

# before: silently no-ops (tap hits the keyboard)
agent-device press @more --settle

# after:
Error (COMMAND_FAILED): Target @more is behind the visible keyboard and cannot be tapped
Hint: Run `agent-device keyboard dismiss` (iOS taps the keyboard's own dismiss control) and retry.
details: { reason: 'tap_keyboard_occludes_target', keyboardFrame: {...}, targetRect: {...} }

Dependencies / open questions

• Needs a reliable keyboard-frame provider on the iOS runner (incl. physical devices) and Android; confirm both are exposed at snapshot time before choosing snapshot-derived vs. separate API.
• Decide whether this lands as a new guarantee cell or extends the existing offscreen enforcement set; if it can be computed purely from the captured snapshot node, it likely belongs in the shared runtime preflight (preflightNativeRefInteraction, Interaction guarantee gaps (ADR 0011 umbrella) #1081) so the fast path can't silently succeed.

[thymikee]

One wording mismatch to settle here, plus one orientation this work does not cover.

Coordinate-only taps warn instead of refusing, and that is the intended stance. #2602 says Closes #2589, so the issue should own the decision rather than the PR body quietly narrowing it.

What refuses: every acting path that names an element — @ref, selector, the native-ref fast path, and both drag endpoints. They fail with COMMAND_FAILED and details.reason: tap_keyboard_occludes_target, carrying the target's rect and the measured keyboard frame, and the backend is never called.

What discloses: a bare coordinate. The tap goes through and the response names the reason in a warning, with the same typed reason and the band it measured. Three reasons, in order of how much they matter:

• a raw coordinate names a point, not an element, so there is no named element whose activation could be silently stolen — the caller asked for a place on the screen and got it;
• it is the only way to tap the keyboard's own controls, so the disclosure stays quiet when the point lands on a control the capture reports as the keyboard's, and refusing all coordinates would refuse keyboard taps too;
• the coordinate path never captures the tree it is measured against. Its evidence is the last-known snapshot, which can be arbitrarily stale; refusing on that would block real taps on a keyboard that may no longer be there.

If refusing coordinates is what you want, that is a decision to spend a fresh capture on every coordinate tap — and it still cannot tell a deliberate keyboard tap from an accidental one whenever the tree does not report a control under the point. Say the word and we will take that trade.

Landscape iPhone is not covered by this fix, and cannot be by a rule. Apple reports the landscape iPhone keyboard's rects in the keyboard's own rotated space: measured on iPhone 17 Pro, the screenshot shows the keyboard full width across the bottom 327 pt of an 874 x 402 viewport, while the tree says the key plane is 162 pt wide and the globe dock button is at y 8. There is no band to measure from that, so the classifier refuses to invent one and the tap fails open. Tracked in #2612 against the Apple capture.

Portrait iPhone was verified on device: the covered tab-bar item refuses with the typed reason, the keyboard's own dock key still taps, a coordinate behind the keyboard warns, and the same item presses normally once the keyboard is gone. Android was verified against a real Gboard capture on an emulator: the band derives as x 0, y 1517, 1080 x 883 from nodes carrying com.google.android.inputmethod.latin, and points inside it read as occluded.