Skip to content

Commit 6316e10

Browse files
authored
feat(input): iOS simulator input through the HID injector (#1037)
Closes #1019 ### Status Implement: done Review: passed (round 2) Hardware: unavailable (disk 14.8 GiB free, the lane needs 15) Gate: waiting for the stack Next: needs-hardware. Run Done when line 3 on a machine with disk space and an iOS simulator ## Assumptions - Assumption: where the iOS device input lives — `src/drivers/ios/input/` (an `index.ts` and `device-input.ts`, as Android's `input/`) instead of the spec's single `input.ts`, so its test goes through an `index.ts` (testing rule 6). ## Done when - [x] The two Tests lines and the `byte-pipe` test pass in `pnpm test`, and `src/source-hygiene.test.ts` passes. - [x] `pnpm build:capture` builds `simlock-screen-capture` with the input mode where Xcode is selected. - [x] On a simulator, `e2e/slow-agent-device.test.ts` opens a lease and the Appduct fixture through agent-device, opens `WS /v1/leases/{id}/input`, gets 0x83 and 0x82, taps the centre of `tab-status` as agent-device's snapshot places it, and the next snapshot shows the Status tab's screen (slow lane, iOS simulator). — slow lane on 2ffb9ae, 2026-10-27: passed (18 passed, 17 Android cases skipped) - [x] `watching-the-screen.mdx` lists what input does on iOS: touch, scroll, Home, keys on iOS 26 runtimes; not the app switcher, Lock, rotation or keys on iOS 27 (#1024). - [x] An agent token streams its own lease through a gateway and through a standalone worker, and gets 404 for anyone else's. — #1015 (PR #1025), `src/daemon/stream-gateway.e2e.test.ts` - [x] A test client that follows T3 Code's message order (waits for 0x83, reads 0x82, sends touches and keys) drives a faked device through a gateway, in the fast lane. — #1017 (PR #1034), `src/daemon/input-lease.e2e.test.ts` - [x] A real iOS simulator and an Android emulator take a tap through the input socket: a tap on a known control opens the next screen, as agent-device's snapshot shows (slow lane). A person confirms a tap on an Android phone (slow lane, person at the phone). — simulator and emulator taps passed on the slow lane 2026-10-27; the phone tap (`e2e/slow-android-physical-manual.test.ts`) was confirmed by the maintainer at the phone - [x] Ending the lease ends both the stream (with a 0x07 frame) and the input socket (close 4001). — #1015 (`src/stream/create.test.ts`), #1017 (`src/daemon/input-lease.e2e.test.ts`) ## Review Spec review: 0 blocking, 0 fixed, 0 notes. Code review: 1 blocking, 1 fixed, 0 notes. Claims review: 0 blocking, 0 fixed, 0 notes. Rounds: 2. Tokens: 28.5M. Runs: 11. Cost USD: 6.0555. Agent seconds: 2652. Assessment: complete _Written by an agent._
1 parent 0bbe8f6 commit 6316e10

29 files changed

Lines changed: 1358 additions & 46 deletions

‎docs/internal/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ worker daemon ── uplink (WebSocket to /v1/uplink, worker dials) ──┘
4343
| core/warm-pool | What stays booted, and warm targets | [warm-pool](../../src/core/warm-pool/AGENTS.md) |
4444
| core/capacity | Limits, RAM budget, reservations | [capacity](../../src/core/capacity/AGENTS.md) |
4545
| core/usage | Usage figures from the history rows, read on a thread | [usage](../../src/core/usage/AGENTS.md) |
46-
| drivers | The `Driver` port's two implementations | [drivers](../../src/drivers/AGENTS.md) · [android](../../src/drivers/android/AGENTS.md) · [android screen](../../src/drivers/android/screen/AGENTS.md) · [android input](../../src/drivers/android/input/AGENTS.md) · [ios](../../src/drivers/ios/AGENTS.md) |
46+
| drivers | The `Driver` port's two implementations | [drivers](../../src/drivers/AGENTS.md) · [android](../../src/drivers/android/AGENTS.md) · [android screen](../../src/drivers/android/screen/AGENTS.md) · [android input](../../src/drivers/android/input/AGENTS.md) · [ios](../../src/drivers/ios/AGENTS.md) · [ios input](../../src/drivers/ios/input/AGENTS.md) |
4747
| gateway | Fleet queue, routing, forwarding to workers, relay | [gateway](../../src/gateway/AGENTS.md) · [routing](../../src/gateway/routing/AGENTS.md) |
4848
| alerts | Gateway only: chosen fleet events POSTed to webhook destinations | [alerts](../../src/alerts/AGENTS.md) |
4949
| link | Byte channels over the uplink | [link](../../src/link/AGENTS.md) |

‎docs/internal/VENDORED-CODE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ license text(s) they require under `LICENSES/`, and a `NOTICE.md`.
1414

1515
| What | Upstream | Version / commit | License | Where it lives | Consumed by | Re-verify with |
1616
|---|---|---|---|---|---|---|
17-
| serve-sim's capture and encode pipeline — CoreSimulator framebuffer capture and the H.264 encode path, patched by us, with four files rewritten by us that stay serve-sim's modified files | `EvanBacon/serve-sim` | commit `6d009686ed08f7d8bd19bca9f9f67a9237931f44` | Apache-2.0 for serve-sim's files and our rewrites of four of them; MIT for the files that are ours entirely | `vendor/screen-capture/Sources/`, licenses at `vendor/screen-capture/LICENSES/serve-sim.APACHE-2.0.txt` and `…/idevice.MIT.txt` | the `simlock-screen-capture` and `simlock-device-capture` programs `pnpm build` puts in dist/bin, run by `src/drivers/ios/screen-capture.ts` | the `curl … \| diff -u -` command in [`vendor/screen-capture/NOTICE.md`](../../vendor/screen-capture/NOTICE.md), per file |
17+
| serve-sim's capture, encode and HID-input pipeline — CoreSimulator framebuffer capture, the H.264 encode path and the `HIDInjector` that injects touches and keys, patched by us, with four files rewritten by us that stay serve-sim's modified files | `EvanBacon/serve-sim` | commit `6d009686ed08f7d8bd19bca9f9f67a9237931f44` | Apache-2.0 for serve-sim's files and our rewrites of four of them; MIT for the files that are ours entirely | `vendor/screen-capture/Sources/`, licenses at `vendor/screen-capture/LICENSES/serve-sim.APACHE-2.0.txt` and `…/idevice.MIT.txt` | the `simlock-screen-capture` and `simlock-device-capture` programs `pnpm build` puts in dist/bin, run by `src/drivers/ios/screen-capture.ts` and, in its `--input` mode, `src/drivers/ios/input/` | the `curl … \| diff -u -` command in [`vendor/screen-capture/NOTICE.md`](../../vendor/screen-capture/NOTICE.md), per file |
1818
| the HEVC capability string the iPhone's media-stream offer advertises for payload 100 — one string inside one of our own files | `jkcoxson/idevice` | `master` at the time of taking; the string is what its receiver documents | MIT | `vendor/screen-capture/LICENSES/idevice.MIT.txt`, the string in `vendor/screen-capture/Sources/SimlockMediaBridge/SimlockMediaBridge.m` | `simlock-device-capture`, whose offer a physical iPhone accepts without painting torn pictures | the string in `Sources/SimlockMediaBridge/SimlockMediaBridge.m` against idevice's receiver, and its `LICENSE.txt` at `https://raw.githubusercontent.com/jkcoxson/idevice/master/LICENSE.txt` |
1919
| `scrcpy-server` — the Android screen-streaming server artifact, byte-identical to upstream's release asset | `Genymobile/scrcpy` | release `v5.0.1`; revision built `dc8a55a24c27ae78ae202b5a3288da681d0f141a` | Apache-2.0; the Kotlin standard library inside it adds BSD-3-Clause (ThreeTen-BP) and BSL-1.0 (Boost) code | `vendor/scrcpy/scrcpy-server-v5.0.1`, licenses at `vendor/scrcpy/LICENSES/scrcpy.APACHE-2.0.txt` and `vendor/scrcpy/LICENSES/threetenbp.BSD-3-Clause.txt`, digest in `vendor/scrcpy/SHA256SUMS.txt` | the Android driver's screen capture, which pushes it unmodified to a leased emulator; the client that drives it is ours, in `src/drivers/android/`, and is not vendored | `cd vendor/scrcpy && shasum -a 256 -c SHA256SUMS.txt`, and the same against upstream's own release `SHA256SUMS.txt`, whose GPG signature `vendor/scrcpy/NOTICE.md` says how to check |
2020

‎docs/internal/adr/0030-optional-modules-share-one-contract-and-load-only-when-turned-on.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# 0030. Optional modules share one contract and load only when turned on
22

3-
- **Status:** Accepted — not yet implemented; §2 narrowed by [ADR 0046](0046-a-standalone-worker-serves-agent-device-by-default-with-a-private-profile.md); §3 narrowed by [ADR 0049](0049-a-lease-holder-watches-and-drives-its-devices-screen.md)
3+
- **Status:** Accepted; §2 narrowed by [ADR 0046](0046-a-standalone-worker-serves-agent-device-by-default-with-a-private-profile.md); §3 narrowed by [ADR 0049](0049-a-lease-holder-watches-and-drives-its-devices-screen.md)
44
- **Date:** 2026-10-15
55
- **Issue:** [#591](https://github.com/callstackincubator/simlock/issues/591)
66
- **Supersedes:** nothing. Narrows [ADR

‎docs/internal/adr/0033-a-sign-in-proxy-is-a-second-way-to-sign-in-and-auth-is-one-module.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# 0033. A sign-in proxy is a second way to sign in, and auth is one module
22

3-
- **Status:** Accepted — not yet implemented
3+
- **Status:** Accepted
44
- **Date:** 2026-10-19
55
- **Issue:** [#701](https://github.com/callstackincubator/simlock/issues/701)
66
- **Supersedes:** nothing. Amends [ADR

‎docs/internal/adr/0049-a-lease-holder-watches-and-drives-its-devices-screen.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# 0049. A lease holder watches and drives its device's screen
22

3-
- **Status:** Accepted — not yet implemented
3+
- **Status:** Accepted
44
- **Date:** 2026-10-26
55
- **Issue:** [#1013](https://github.com/callstackincubator/simlock/issues/1013)
66
- **Narrows:** [ADR 0045](0045-a-devices-screen-streams-to-operators-not-a-leases.md)

‎docs/internal/adr/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,10 +68,10 @@ the status is stale.
6868
| [0027](0027-dependency-cruiser-checks-every-component-boundary.md) | dependency-cruiser checks every component's boundary and the module graph | Accepted |
6969
| [0028](0028-the-uplink-carries-byte-channels-and-small-messages-go-first.md) | The uplink carries byte channels beside its messages, and small messages go first | Accepted; §4 narrowed by 0049 |
7070
| [0029](0029-host-load-is-an-optional-module-whose-readings-ride-on-status.md) | Host load is an optional module whose readings ride on status | Accepted |
71-
| [0030](0030-optional-modules-share-one-contract-and-load-only-when-turned-on.md) | Optional modules share one contract and load only when turned on | Accepted — not yet implemented; §2 narrowed by 0046; §3 narrowed by 0049 |
71+
| [0030](0030-optional-modules-share-one-contract-and-load-only-when-turned-on.md) | Optional modules share one contract and load only when turned on | Accepted; §2 narrowed by 0046; §3 narrowed by 0049 |
7272
| [0031](0031-a-physical-iphone-streams-through-coredevice-behind-a-screen-source-seam.md) | A physical iPhone streams through CoreDevice's media stream, behind a screen-source seam | Accepted |
7373
| [0032](0032-the-event-history-is-a-sqlite-database-with-projections-beside-it.md) | The event history is a SQLite database, with projections beside it | Accepted — not yet implemented; §1 narrowed by 0036 |
74-
| [0033](0033-a-sign-in-proxy-is-a-second-way-to-sign-in-and-auth-is-one-module.md) | A sign-in proxy is a second way to sign in, and auth is one module | Accepted — not yet implemented |
74+
| [0033](0033-a-sign-in-proxy-is-a-second-way-to-sign-in-and-auth-is-one-module.md) | A sign-in proxy is a second way to sign in, and auth is one module | Accepted |
7575
| [0034](0034-one-worker-per-user-and-the-folder-says-what-a-daemon-is.md) | One worker per user, and the folder says what a daemon is | Accepted — not yet implemented |
7676
| [0035](0035-a-gateway-sends-chosen-fleet-events-to-webhook-destinations.md) | A gateway sends chosen fleet events to webhook destinations | Accepted — not yet implemented |
7777
| [0036](0036-daemon-state-is-a-sqlite-database-whose-tables-each-owner-keeps.md) | Daemon state is a SQLite database, and each owner keeps its own tables | Accepted — not yet implemented |
@@ -87,4 +87,4 @@ the status is stale.
8787
| [0046](0046-a-standalone-worker-serves-agent-device-by-default-with-a-private-profile.md) | A standalone worker serves agent-device by default and hands local agents a private profile | Accepted — not yet implemented; narrows 0023 §1, §3–4, 0030 §2, 0039 and 0043 §4 |
8888
| [0047](0047-device-exec-fills-in-the-leases-device.md) | `device.exec` fills in the lease's device, and refuses another | Accepted — not yet implemented; extends 0005 §19a; narrows 0003 §11 for one MCP tool |
8989
| [0048](0048-an-ios-worker-may-keep-its-simulators-in-xcodes-device-set.md) | An iOS worker may keep its simulators in Xcode's device set | Accepted; narrows 0001 §1, §2, §3, §5, §6 and §7 for iOS under `deviceSet: "default"` |
90-
| [0049](0049-a-lease-holder-watches-and-drives-its-devices-screen.md) | A lease holder watches and drives its device's screen | Accepted — not yet implemented |
90+
| [0049](0049-a-lease-holder-watches-and-drives-its-devices-screen.md) | A lease holder watches and drives its device's screen | Accepted |

‎e2e/slow-agent-device.test.ts‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -236,8 +236,7 @@ function defineSuite(target: Target): void {
236236
},
237237
);
238238

239-
// why: the input socket's iOS device input arrives with #1019, so the case waits for it there.
240-
it.skipIf(platform === "ios")(
239+
it(
241240
"an agent taps the centre of tab-status through the lease's input socket, as agent-device's snapshot places it, and the next snapshot shows the Status tab's screen",
242241
{ timeout: 600_000 },
243242
async () => {

‎src/daemon/main.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1325,6 +1325,7 @@ async function discoverIosDriver(
13251325
filesystem: options.filesystem,
13261326
idGenerator: options.idGenerator,
13271327
instanceId: options.instanceId,
1328+
logger: logger.child("ios"),
13281329
onSlimmed: emitSlimDiagnostic(options.eventBus),
13291330
screenCapture: {
13301331
baseEnvironment: process.env,

‎src/drivers/ios/AGENTS.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ Parent: [drivers](../AGENTS.md). Flows: [device selection](../../../docs/interna
1616
| `devicectl-exec.ts` | `devicectlExecCommand`: the reviewed `devicectl` list |
1717
| `slim-labels.ts` | `SLIM_CATEGORIES`, `resolveSlimCategories`, `labelsFor`, `slimSignature` |
1818
| `prerequisites.ts` | `iosPrerequisites`: Xcode selected, license accepted, first launch done |
19+
| `input/` | `createIosDeviceInput`: the lease's input socket on a simulator, through `simlock-screen-capture --input`; a physical iPhone answers unavailable |
20+
| `runtime-version.ts` | `iosRuntimeVersionFromId`: the iOS major and minor a runtime id names |
1921
| `screen-capture.ts` | `createIosScreenCapture`: runs `vendor/screen-capture/` on a simulator or a physical iPhone |
2022

2123
## How it works
@@ -93,6 +95,10 @@ Programs: dist/bin from `pnpm build`, or `drivers.ios.screenCaptureCommand` and
9395
program enforces its pin: a CoreDevice major other than 642 exits 69 with the reason, a 503 for
9496
the viewer. Asked to stop while still starting, a program stops and goes inside its 2 s.
9597

98+
Input — `input` (offered with `screenCapture`, same launcher): `simlock-screen-capture <udid> --input` with the same `SIMLOCK_IOS_DEVICE_SET` rule. stdout: one `0x82` `{width,height}` (the device type's native size) within 10 s of spawn, else the open rejects (also for a missing program or an early exit; the helper is stopped).
99+
stdin: envelopes `[len][tag]{json}` for touch `0x03`, key `0x06` and scroll `0x0b`; a failed write or a later exit calls `onFailed`. Home runs `xcrun simctl [--set <root>] launch <udid> com.apple.springboard` (10 s; a failure is logged); the app switcher, Lock and Back are dropped.
100+
The config is always `portrait`, with `keyboardUnavailable: true` when the runtime is iOS 27 or later or unreadable, and keys are then dropped. Close: stdin, 2 s, SIGKILL. A physical iPhone answers `{unavailable}` and starts nothing.
101+
96102
Advisories — `runtime-cache-unreclaimable` (asset bundles no installed runtime mounts);
97103
`slim-runtime-unsupported` (installed runtimes below 18.5, only when `ios.defaultMode` is `slim`).
98104

@@ -122,6 +128,7 @@ record of a device's mode; `driverData` holds the slim signature.
122128

123129
## Known gaps
124130

131+
- **Input:** no app switcher, Lock, rotation or keys on iOS 27 runtimes (#1024). The input follows the device's native frame, so after a rotation taps and picture stay in the same frame. After the health monitor reboots a crashed simulator the helper's HID client is bound to the old boot, so taps may vanish until the lease's input reopens.
125132
- **A restart between `simctl create` and the durable mark under `default` (#968)** leaves one unlisted `simlock-<id>` simulator in Xcode's set. Delete it with `xcrun simctl delete <udid>`.
126133
- **A runtime download outlives the runtime (#79).** `simctl runtime delete` leaves its ~8.5 GiB asset bundle in `/System/Library/AssetsV2`. Simlock never deletes it; `doctor` and `component remove` report it. Reclaim in Xcode → Settings → Platforms.
127134
- **Slim costs (#87, #172).** Every reclaim erases, so the next boot re-slims: two boots each

‎src/drivers/ios/index.ts‎

Lines changed: 27 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import {
66
type ComponentInstallResult,
77
type ComponentReceipt,
88
type ComponentRemoval,
9+
type DeviceInputFactory,
910
COMPONENT_REMOVAL_TIMEOUT_MS,
1011
type ExactDeviceRequest,
1112
type Driver,
@@ -50,6 +51,7 @@ import type {
5051
Clock,
5152
Filesystem,
5253
IdGenerator,
54+
Logger,
5355
ProcessResult,
5456
ProcessRunner,
5557
} from "../../ports/index.js";
@@ -58,6 +60,8 @@ import { refuseOwnSetLeftovers, refuseXcodeSetLeftovers } from "./device-set-lef
5860
import { devicectlExecCommand } from "./devicectl-exec.js";
5961
import { fillSimctlDevice } from "./simctl-device.js";
6062
import { IosPhysicalHandler } from "./physical.js";
63+
import { createIosDeviceInput } from "./input/index.js";
64+
import { iosRuntimeVersionFromId } from "./runtime-version.js";
6165
import { createIosScreenCapture } from "./screen-capture.js";
6266
import { labelsFor, resolveSlimCategories, slimSignature } from "./slim-labels.js";
6367

@@ -216,6 +220,8 @@ export interface IosSimctlDriverOptions {
216220
readonly bootTimeoutMs: number;
217221
};
218222
readonly slimByDefault?: boolean;
223+
// why: the device input logs a Home press that did nothing; nothing else in the driver logs through it.
224+
readonly logger?: Logger;
219225
// why: absent when this host has no way to run the capture program; the driver then offers no
220226
// screen at all rather than one that fails on every open.
221227
readonly screenCapture?: {
@@ -349,9 +355,11 @@ export class IosSimctlDriver implements Driver {
349355
? {}
350356
: { onLog: options.screenCapture.onLog }),
351357
});
358+
this.input = createInputFor(options, deviceRoot, rootOptions !== undefined);
352359
}
353360

354361
readonly screenCapture: ScreenCapture | undefined;
362+
readonly input: DeviceInputFactory | undefined;
355363

356364
/**
357365
* why: Establishes the device set this driver owns before it can be asked to do anything,
@@ -1778,6 +1786,24 @@ class IosRuntimeUnavailableForDownloadError extends RuntimeMissingError {
17781786
}
17791787
}
17801788

1789+
// why: the input is offered exactly when the screen capture is, because both are the one program.
1790+
function createInputFor(
1791+
options: IosSimctlDriverOptions,
1792+
deviceRoot: string,
1793+
ownedRoot: boolean,
1794+
): DeviceInputFactory | undefined {
1795+
if (options.screenCapture === undefined) return undefined;
1796+
return createIosDeviceInput({
1797+
baseEnvironment: options.screenCapture.baseEnvironment,
1798+
clock: options.clock,
1799+
command: configuredScreenCaptureCommand(options) ?? options.screenCapture.command,
1800+
...(ownedRoot ? { deviceSet: deviceRoot } : {}),
1801+
...(options.logger === undefined ? {} : { logger: options.logger }),
1802+
launcher: options.screenCapture.launcher,
1803+
processRunner: options.processRunner,
1804+
});
1805+
}
1806+
17811807
function configuredScreenCaptureCommand(options: IosSimctlDriverOptions): string | undefined {
17821808
const configured = options.driverConfig["screenCaptureCommand"];
17831809
return typeof configured === "string" && configured !== "" ? configured : undefined;
@@ -2257,18 +2283,7 @@ function iosDriverData(device: DriverDevice): IosDriverData {
22572283
};
22582284
}
22592285

2260-
export function iosRuntimeVersionFromId(runtimeId: string): readonly [number, number] | undefined {
2261-
const match = /iOS-(\d+)(?:-(\d+))?(?:-(\d+))?$/.exec(runtimeId);
2262-
if (match === null) {
2263-
return undefined;
2264-
}
2265-
const major = Number.parseInt(match[1] ?? "", 10);
2266-
const minor = match[2] === undefined ? 0 : Number.parseInt(match[2], 10);
2267-
if (Number.isNaN(major) || Number.isNaN(minor)) {
2268-
return undefined;
2269-
}
2270-
return [major, minor];
2271-
}
2286+
export { iosRuntimeVersionFromId };
22722287

22732288
export function supportsPersistentSlim(version: readonly [number, number] | undefined): boolean {
22742289
if (version === undefined) {

0 commit comments

Comments
 (0)