You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6316e10
Browse filesBrowse the repository at this point in the historyBrowse files
feat(input): iOS simulator input through the HID injector (#1037)
Closes#1019
### Status
Implement: done Review: passed (round 2) Hardware: unavailable (disk
14.8 GiB free, the lane needs 15) Gate: waiting for the stack
Next: needs-hardware. Run Done when line 3 on a machine with disk space
and an iOS simulator
## Assumptions
- Assumption: where the iOS device input lives —
`src/drivers/ios/input/` (an `index.ts` and `device-input.ts`, as
Android's `input/`) instead of the spec's single `input.ts`, so its test
goes through an `index.ts` (testing rule 6).
## Done when
- [x] The two Tests lines and the `byte-pipe` test pass in `pnpm test`,
and `src/source-hygiene.test.ts` passes.
- [x] `pnpm build:capture` builds `simlock-screen-capture` with the
input mode where Xcode is selected.
- [x] On a simulator, `e2e/slow-agent-device.test.ts` opens a lease and
the Appduct fixture through agent-device, opens `WS
/v1/leases/{id}/input`, gets 0x83 and 0x82, taps the centre of
`tab-status` as agent-device's snapshot places it, and the next snapshot
shows the Status tab's screen (slow lane, iOS simulator). — slow lane on
2ffb9ae, 2026-10-27: passed (18 passed, 17 Android cases skipped)
- [x] `watching-the-screen.mdx` lists what input does on iOS: touch,
scroll, Home, keys on iOS 26 runtimes; not the app switcher, Lock,
rotation or keys on iOS 27 (#1024).
- [x] An agent token streams its own lease through a gateway and through
a standalone worker, and gets 404 for anyone else's. — #1015 (PR #1025),
`src/daemon/stream-gateway.e2e.test.ts`
- [x] A test client that follows T3 Code's message order (waits for
0x83, reads 0x82, sends touches and keys) drives a faked device through
a gateway, in the fast lane. — #1017 (PR #1034),
`src/daemon/input-lease.e2e.test.ts`
- [x] A real iOS simulator and an Android emulator take a tap through
the input socket: a tap on a known control opens the next screen, as
agent-device's snapshot shows (slow lane). A person confirms a tap on an
Android phone (slow lane, person at the phone). — simulator and emulator
taps passed on the slow lane 2026-10-27; the phone tap
(`e2e/slow-android-physical-manual.test.ts`) was confirmed by the
maintainer at the phone
- [x] Ending the lease ends both the stream (with a 0x07 frame) and the
input socket (close 4001). — #1015 (`src/stream/create.test.ts`), #1017
(`src/daemon/input-lease.e2e.test.ts`)
## Review
Spec review: 0 blocking, 0 fixed, 0 notes. Code review: 1 blocking, 1
fixed, 0 notes. Claims review: 0 blocking, 0 fixed, 0 notes.
Rounds: 2. Tokens: 28.5M.
Runs: 11. Cost USD: 6.0555. Agent seconds: 2652.
Assessment: complete
_Written by an agent._
Copy file name to clipboardExpand all lines: docs/internal/VENDORED-CODE.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@ license text(s) they require under `LICENSES/`, and a `NOTICE.md`.
14
14
15
15
| What | Upstream | Version / commit | License | Where it lives | Consumed by | Re-verify with |
16
16
|---|---|---|---|---|---|---|
17
-
| serve-sim's captureand encode pipeline — CoreSimulator framebuffer capture and the H.264 encode path, patched by us, with four files rewritten by us that stay serve-sim's modified files |`EvanBacon/serve-sim`| commit `6d009686ed08f7d8bd19bca9f9f67a9237931f44`| Apache-2.0 for serve-sim's files and our rewrites of four of them; MIT for the files that are ours entirely |`vendor/screen-capture/Sources/`, licenses at `vendor/screen-capture/LICENSES/serve-sim.APACHE-2.0.txt` and `…/idevice.MIT.txt`| the `simlock-screen-capture` and `simlock-device-capture` programs `pnpm build` puts in dist/bin, run by `src/drivers/ios/screen-capture.ts`| the `curl … \| diff -u -` command in [`vendor/screen-capture/NOTICE.md`](../../vendor/screen-capture/NOTICE.md), per file |
17
+
| serve-sim's capture, encode and HID-input pipeline — CoreSimulator framebuffer capture, the H.264 encode path and the `HIDInjector` that injects touches and keys, patched by us, with four files rewritten by us that stay serve-sim's modified files |`EvanBacon/serve-sim`| commit `6d009686ed08f7d8bd19bca9f9f67a9237931f44`| Apache-2.0 for serve-sim's files and our rewrites of four of them; MIT for the files that are ours entirely |`vendor/screen-capture/Sources/`, licenses at `vendor/screen-capture/LICENSES/serve-sim.APACHE-2.0.txt` and `…/idevice.MIT.txt`| the `simlock-screen-capture` and `simlock-device-capture` programs `pnpm build` puts in dist/bin, run by `src/drivers/ios/screen-capture.ts` and, in its `--input` mode, `src/drivers/ios/input/`| the `curl … \| diff -u -` command in [`vendor/screen-capture/NOTICE.md`](../../vendor/screen-capture/NOTICE.md), per file |
18
18
| the HEVC capability string the iPhone's media-stream offer advertises for payload 100 — one string inside one of our own files |`jkcoxson/idevice`|`master` at the time of taking; the string is what its receiver documents | MIT |`vendor/screen-capture/LICENSES/idevice.MIT.txt`, the string in `vendor/screen-capture/Sources/SimlockMediaBridge/SimlockMediaBridge.m`|`simlock-device-capture`, whose offer a physical iPhone accepts without painting torn pictures | the string in `Sources/SimlockMediaBridge/SimlockMediaBridge.m` against idevice's receiver, and its `LICENSE.txt` at `https://raw.githubusercontent.com/jkcoxson/idevice/master/LICENSE.txt`|
19
19
|`scrcpy-server` — the Android screen-streaming server artifact, byte-identical to upstream's release asset |`Genymobile/scrcpy`| release `v5.0.1`; revision built `dc8a55a24c27ae78ae202b5a3288da681d0f141a`| Apache-2.0; the Kotlin standard library inside it adds BSD-3-Clause (ThreeTen-BP) and BSL-1.0 (Boost) code |`vendor/scrcpy/scrcpy-server-v5.0.1`, licenses at `vendor/scrcpy/LICENSES/scrcpy.APACHE-2.0.txt` and `vendor/scrcpy/LICENSES/threetenbp.BSD-3-Clause.txt`, digest in `vendor/scrcpy/SHA256SUMS.txt`| the Android driver's screen capture, which pushes it unmodified to a leased emulator; the client that drives it is ours, in `src/drivers/android/`, and is not vendored |`cd vendor/scrcpy && shasum -a 256 -c SHA256SUMS.txt`, and the same against upstream's own release `SHA256SUMS.txt`, whose GPG signature `vendor/scrcpy/NOTICE.md` says how to check |
Copy file name to clipboardExpand all lines: docs/internal/adr/README.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -68,10 +68,10 @@ the status is stale.
68
68
|[0027](0027-dependency-cruiser-checks-every-component-boundary.md)| dependency-cruiser checks every component's boundary and the module graph | Accepted |
69
69
|[0028](0028-the-uplink-carries-byte-channels-and-small-messages-go-first.md)| The uplink carries byte channels beside its messages, and small messages go first | Accepted; §4 narrowed by 0049 |
70
70
|[0029](0029-host-load-is-an-optional-module-whose-readings-ride-on-status.md)| Host load is an optional module whose readings ride on status | Accepted |
71
-
|[0030](0030-optional-modules-share-one-contract-and-load-only-when-turned-on.md)| Optional modules share one contract and load only when turned on | Accepted — not yet implemented; §2 narrowed by 0046; §3 narrowed by 0049 |
71
+
|[0030](0030-optional-modules-share-one-contract-and-load-only-when-turned-on.md)| Optional modules share one contract and load only when turned on | Accepted; §2 narrowed by 0046; §3 narrowed by 0049 |
72
72
|[0031](0031-a-physical-iphone-streams-through-coredevice-behind-a-screen-source-seam.md)| A physical iPhone streams through CoreDevice's media stream, behind a screen-source seam | Accepted |
73
73
|[0032](0032-the-event-history-is-a-sqlite-database-with-projections-beside-it.md)| The event history is a SQLite database, with projections beside it | Accepted — not yet implemented; §1 narrowed by 0036 |
74
-
|[0033](0033-a-sign-in-proxy-is-a-second-way-to-sign-in-and-auth-is-one-module.md)| A sign-in proxy is a second way to sign in, and auth is one module | Accepted — not yet implemented |
74
+
|[0033](0033-a-sign-in-proxy-is-a-second-way-to-sign-in-and-auth-is-one-module.md)| A sign-in proxy is a second way to sign in, and auth is one module | Accepted |
75
75
|[0034](0034-one-worker-per-user-and-the-folder-says-what-a-daemon-is.md)| One worker per user, and the folder says what a daemon is | Accepted — not yet implemented |
76
76
|[0035](0035-a-gateway-sends-chosen-fleet-events-to-webhook-destinations.md)| A gateway sends chosen fleet events to webhook destinations | Accepted — not yet implemented |
77
77
|[0036](0036-daemon-state-is-a-sqlite-database-whose-tables-each-owner-keeps.md)| Daemon state is a SQLite database, and each owner keeps its own tables | Accepted — not yet implemented |
@@ -87,4 +87,4 @@ the status is stale.
87
87
|[0046](0046-a-standalone-worker-serves-agent-device-by-default-with-a-private-profile.md)| A standalone worker serves agent-device by default and hands local agents a private profile | Accepted — not yet implemented; narrows 0023 §1, §3–4, 0030 §2, 0039 and 0043 §4 |
88
88
|[0047](0047-device-exec-fills-in-the-leases-device.md)|`device.exec` fills in the lease's device, and refuses another | Accepted — not yet implemented; extends 0005 §19a; narrows 0003 §11 for one MCP tool |
89
89
|[0048](0048-an-ios-worker-may-keep-its-simulators-in-xcodes-device-set.md)| An iOS worker may keep its simulators in Xcode's device set | Accepted; narrows 0001 §1, §2, §3, §5, §6 and §7 for iOS under `deviceSet: "default"`|
90
-
|[0049](0049-a-lease-holder-watches-and-drives-its-devices-screen.md)| A lease holder watches and drives its device's screen | Accepted — not yet implemented |
90
+
|[0049](0049-a-lease-holder-watches-and-drives-its-devices-screen.md)| A lease holder watches and drives its device's screen | Accepted |
Copy file name to clipboardExpand all lines: e2e/slow-agent-device.test.ts
+1-2Lines changed: 1 addition & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -236,8 +236,7 @@ function defineSuite(target: Target): void {
236
236
},
237
237
);
238
238
239
-
// why: the input socket's iOS device input arrives with #1019, so the case waits for it there.
240
-
it.skipIf(platform==="ios")(
239
+
it(
241
240
"an agent taps the centre of tab-status through the lease's input socket, as agent-device's snapshot places it, and the next snapshot shows the Status tab's screen",
|`prerequisites.ts`|`iosPrerequisites`: Xcode selected, license accepted, first launch done |
19
+
|`input/`|`createIosDeviceInput`: the lease's input socket on a simulator, through `simlock-screen-capture --input`; a physical iPhone answers unavailable |
20
+
|`runtime-version.ts`|`iosRuntimeVersionFromId`: the iOS major and minor a runtime id names |
19
21
|`screen-capture.ts`|`createIosScreenCapture`: runs `vendor/screen-capture/` on a simulator or a physical iPhone |
20
22
21
23
## How it works
@@ -93,6 +95,10 @@ Programs: dist/bin from `pnpm build`, or `drivers.ios.screenCaptureCommand` and
93
95
program enforces its pin: a CoreDevice major other than 642 exits 69 with the reason, a 503 for
94
96
the viewer. Asked to stop while still starting, a program stops and goes inside its 2 s.
95
97
98
+
Input — `input` (offered with `screenCapture`, same launcher): `simlock-screen-capture <udid> --input` with the same `SIMLOCK_IOS_DEVICE_SET` rule. stdout: one `0x82``{width,height}` (the device type's native size) within 10 s of spawn, else the open rejects (also for a missing program or an early exit; the helper is stopped).
99
+
stdin: envelopes `[len][tag]{json}` for touch `0x03`, key `0x06` and scroll `0x0b`; a failed write or a later exit calls `onFailed`. Home runs `xcrun simctl [--set <root>] launch <udid> com.apple.springboard` (10 s; a failure is logged); the app switcher, Lock and Back are dropped.
100
+
The config is always `portrait`, with `keyboardUnavailable: true` when the runtime is iOS 27 or later or unreadable, and keys are then dropped. Close: stdin, 2 s, SIGKILL. A physical iPhone answers `{unavailable}` and starts nothing.
101
+
96
102
Advisories — `runtime-cache-unreclaimable` (asset bundles no installed runtime mounts);
97
103
`slim-runtime-unsupported` (installed runtimes below 18.5, only when `ios.defaultMode` is `slim`).
98
104
@@ -122,6 +128,7 @@ record of a device's mode; `driverData` holds the slim signature.
122
128
123
129
## Known gaps
124
130
131
+
-**Input:** no app switcher, Lock, rotation or keys on iOS 27 runtimes (#1024). The input follows the device's native frame, so after a rotation taps and picture stay in the same frame. After the health monitor reboots a crashed simulator the helper's HID client is bound to the old boot, so taps may vanish until the lease's input reopens.
125
132
-**A restart between `simctl create` and the durable mark under `default` (#968)** leaves one unlisted `simlock-<id>` simulator in Xcode's set. Delete it with `xcrun simctl delete <udid>`.
126
133
-**A runtime download outlives the runtime (#79).**`simctl runtime delete` leaves its ~8.5 GiB asset bundle in `/System/Library/AssetsV2`. Simlock never deletes it; `doctor` and `component remove` report it. Reclaim in Xcode → Settings → Platforms.
127
134
-**Slim costs (#87, #172).** Every reclaim erases, so the next boot re-slims: two boots each
0 commit comments