Skip to content

feat: type into web text fields with key events so page scripts see the input - #82

Merged
iamnbutler merged 1 commit into
mainfrom
feat/web-type
Oct 17, 2026
Merged

iamnbutler merged 1 commit into
mainfrom
feat/web-type

Conversation

@iamnbutler

Copy link
Copy Markdown
Contributor

Item 3 from #75. desktop_type sets AXValue, which browsers accept for a web field without firing the page's input/change events, so React-style forms ignore it.

Peekaboo patch peekaboo-web-text-value.patch (appended last)

  • ActionInputDriver.setValue now checks a string written to an AXTextField/AXTextArea/AXComboBox/search field first. It walks the element's ancestry with the same bounded walk as upstream's TextInputRoute, via new resolve(container:), which skips the focus check.
    • Web ancestry: an AXWebArea ancestor throws the public WebTextFieldValueRefusal, a pre-dispatch refusal. Its hint points to desktop_type, so desktop_set_value gets the same pointer.
    • Unresolved ancestry: treated the same as web, so a write the page can't see is never reported as done.
  • Other roles in web content keep the value route.
  • New test in ActionInputDriverTests, covering:
    • web and unresolved ancestry refused with nothing written;
    • a native field still set;
    • a slider in web content still set.
  • All 90 tests pass (isolated root, narrow filter).

Client (actions.swift)

  • On that exact refusal: desktop_type finishes the lease unconsumed. It then checks that the target element is the snapshot's focused element (same role, frame within 1 pt) and re-leases.
  • Typing: it sends typeActionsWithOutcome([.clear, .text(text)]) to that focused control. That is background, pid-targeted key events: ⌘A plus Delete, then Unicode text. Upstream's exact-literal confirmation reads the final value back.
  • Outcome: anything short of a confirmed change or no-change is unknown (TYPING_UNVERIFIED).
  • Refused before input:
    • Field not focused: hint to click it, inspect, and repeat. It never clicks implicitly.
    • Long or control-character text: over 1000 UTF-16 code units, or containing control characters, gets a hint to use desktop_insert route paste.
    • Secure keyboard entry on, or runtime without targeted typing.
  • Native fields: unchanged.

Docs: desktop_type description, tools.md, and the patches README.

Live check: pending, screen locked. Next step is a dev host with Safari or Chrome on a React controlled input: type into it focused and unfocused, check that the page state updates, and set a native TextEdit field to confirm it's unchanged. The signed dev host builds and starts.

If #80 (system-defined-key) lands first, I'll rebase and append after it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant