Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions dist/mcp.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { createInterface } from "node:readline";
import { parseArgs } from "node:util";
import { createDesktop } from "./client.js";
import { isDesktopAction } from "./protocol.js";
import { desktopRequest, desktopTools } from "./tools.js";
import { desktopRequest, desktopToolRules, desktopTools } from "./tools.js";
const { values } = parseArgs({
options: { client: { type: "string" }, socket: { type: "string" }, name: { type: "string" } },
strict: true,
Expand All @@ -19,7 +19,7 @@ const desktop = createDesktop(config);
const { version } = JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8"));
// Newest first; an unsupported client version gets the newest, as the MCP lifecycle requires.
const VERSIONS = ["2025-11-25", "2025-06-18", "2025-03-26", "2024-11-05"];
// Guidance mirrors Ace's desktop extension so agents behave alike; tools come from the shared definitions.
// Guidance mirrors Ace's desktop extension so agents behave alike; tools and their shared rules come from the shared definitions.
const instructions = [
`Use desktop_* tools to observe and operate native apps on this Mac through ${host}, which holds the macOS permissions.`,
"Shell commands remain appropriate for builds, files, and preparing clipboard fixtures directly.",
Expand All @@ -33,6 +33,8 @@ const instructions = [
"Action results start with their outcome: completed, refused, or unknown.",
"For a refusal before dispatch, follow its concrete recovery hint once when applicable, then refresh the observation; a refusal alone does not prove the capability is missing.",
"After completed, interrupted, or uncertain input, inspect the current target before deciding whether another action is needed. Never replay input just because its follow-up observation failed.",
"",
...desktopToolRules(host),
].join("\n");
const tools = desktopTools({ place: "this Mac", app: host });
const byName = new Map(tools.map((tool) => [tool.name, tool]));
Expand Down
5 changes: 5 additions & 0 deletions dist/tools.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ export type DesktopTool = {
/** JSON Schema for the tool's arguments; its properties are the request's fields besides `op`. */
parameters: JsonSchema;
};
/**
* Rules every desktop tool shares, which the tool descriptions don't repeat; a host's agent guidance should include them.
* `app` names the signed app that holds the macOS permissions, whose menu pauses input.
*/
export declare function desktopToolRules(app: string): string[];
export declare function desktopTools({ place, app }: {
place: string;
app: string;
Expand Down
281 changes: 102 additions & 179 deletions dist/tools.js

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ command = "npx"
args = ["-y", "--package=github:githubnext/desktop-tools#v0.10.0", "desktop-tools-mcp"]
```

The tool names, descriptions, and argument schemas come from `@githubnext/desktop-tools/tools` (`desktopTools`, `desktopRequest`), which Ace's `desktop_*` tools also use. [Tools](tools.md) lists them.
The tool names, descriptions, and argument schemas come from `@githubnext/desktop-tools/tools` (`desktopTools`, `desktopRequest`), which Ace's `desktop_*` tools also use. [Tools](tools.md) lists them. The descriptions don't repeat the rules every tool shares: exact targets, single-use snapshots, background and foreground input, what each outcome means, pausing, `desktop_wait` and `expect`, and the recipes. `desktopToolRules(app)` returns those rules, and the server adds them to its instructions; a host with its own agent guidance should include them too.

The server names the agent in the host's window highlight from the MCP client's `clientInfo`: Claude, Codex, Cursor, Gemini or Copilot when the client name contains one, otherwise the client's `title`, otherwise "Agent".

Expand Down
53 changes: 52 additions & 1 deletion docs/tools.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Tools

The [MCP server](mcp.md) and `desktopTools()` expose 45 tools, one per protocol operation. Each tool's description and argument schema in `src/tools.ts` is the full contract; this page is the map.
The [MCP server](mcp.md) and `desktopTools()` expose 47 tools, one per protocol operation. Each tool's description in `src/tools.ts` says, in a few sentences, what it does, whether it is foreground or background input, when its result is `completed` rather than `unknown`, and its main refusal and recovery. Rules shared by every tool, such as single-use snapshots and never replaying input, are in the MCP server's instructions. This page holds the details. The client validates every argument, so the schemas name fields and keep only the limits that help choose values.

[Recipes](recipes.md) give tool sequences, derived from the code, for tasks that take several calls, such as printing to PDF, installing from a DMG or filling a web form.

Expand Down Expand Up @@ -193,6 +193,57 @@ Listing runs in the client process and needs no permission. Selecting goes throu

An action refused because its window was minimized after inspection reports `WINDOW_MINIMIZED`, with the low-level reason as its `cause`. Restore the window with `desktop_restore`, then inspect it again.

## Details by tool

These are the rest of each tool's contract, beyond its description.

**`desktop_clipboard_read`.**
- **Text.** It returns the complete text, up to a 24 KB JSON result. An empty string is distinct from absent text (`present: false`).
- **Image.** The preview is oriented, with separate source and preview metadata. A large preview may flatten transparency onto a white JPEG.
- **Files.** It returns up to 32 advertised local file URLs and their decoded paths. It neither opens the files nor confirms they exist.
- **Refused.** Text and image reads need exactly one clipboard item. File promises, legacy-only filename lists, items mixing files with other content, and unreadable or oversized content are refused.
- **Side effects.** It needs macOS to allow clipboard reading. It never changes the clipboard or releases a pending paste reservation.

**`desktop_clipboard_write`.**
- **Image.** Its original bytes, orientation and transparency are kept. It may be at most 10 MiB and 64 million pixels.
- **Files.** Each path must be an existing file, folder or symbolic link. The check reads only metadata, not contents, and doesn't resolve links. References keep their order and duplicates within the 24 KB read-result limit. `file_count` counts the references requested, not what a receiver copies or pastes.
- **Persistence.** The write lasts until the next copy or write.

**`desktop_launch` and `desktop_open`.**
- **Choosing a copy.** A bundle ID lets macOS choose the installation; a path selects one copy. One launch can include several counted activation attempts.
- **`desktop_open` limits.** Items are at most 4096 UTF-16 units, and the whole request at most 16 KiB. Paths keep Unicode and spaces, with no shell expansion. URLs must already be encoded and are never repaired. Any scheme, including `file` and custom schemes, has its receiving app's normal effects.
- **Neither tool** batches items, creates extra instances or handles dialogs.

**`desktop_menus` and `desktop_menu`.**
- **Reading.** A read is bound to the same process generation before and after. Native results can come from a 2-second cache, so their time and completeness, including lazy or budget-limited submenus, are unknown. Result truncation is counted separately (`truncated`, `omitted`). Enabled and checked state are left out, because the native service substitutes defaults for attributes it can't read. Paths are for discovery, not reusable action targets. Accessibility reads are synchronous and can delay cancellation.
- **Running a command.** `desktop_menu` resolves the path afresh. It refuses missing, ambiguous, disabled or unavailable lazy paths, and presses only the final item once. Its reads and the press run off the GUI actor, with a finite messaging timeout. Cancellation waits for the native call to return and can't undo delivered input.

**`desktop_menubar_click`.** Before pressing, the host lists the menu bar again. It refuses if the item at that index no longer has the same owner generation, identifier, title, bounds or neighbours. Where an item has its own window, it is clicked through that window. The result lists the windows that appeared within about a second, front to back, with pid, window ID, owner, layer and bounds. It captures their region as an image without a snapshot.

**`desktop_dock`.** Titles match exactly, including case, punctuation and Unicode. The right-click menu's submenus can't be reached.

**`desktop_quit` and `desktop_close`.** A normal quit or close can open an unsaved-work dialog. The result is then `unknown`, with fresh inventory when available. The host never retries, force-closes or answers the dialog. `force: true` terminates that exact process immediately, with no save prompt.

**`desktop_inspect`.**
- **Accessibility mode** (the default) can return an action snapshot with element IDs.
- **Pixels mode** returns a snapshot with no element IDs and no focused control. It authorizes only point clicks, point scrolls and drags in that exact window.
- **After an action,** the automatic observation uses accessibility mode. If it fails, inspect again with pixels mode.

**`desktop_click`.** A single left click on a supported editable text field, by element or point, also requests keyboard focus. The result says whether focus was verified; use `desktop_select` to choose a range or caret position. Points stay bound to the captured window, even when the returned screenshot was scaled.

**`desktop_scroll`.** `amount` is in native scroll units. Their distance depends on the route the target supports, not on pixels.

**`desktop_set_value`.**
- **Dates.** A date is written as the control's date value and must read back to the second. The result shows it in UTC. A control that drops the time or seconds reports `unknown`. If a date control refuses, the result's `keyboard_route` explains entering it field by field.
- **Toggles.** A toggle whose value is read-only but which advertises `AXPress` gets at most one press, and none when it already has the requested state.
- **Pop-up buttons.** To pick an item, use `desktop_element_action` with `AXPick`, or `desktop_click`, on the item from the fresh observation.

**`desktop_insert`, paste route.**
- **Requirements.** The field's text and selection must be readable, and macOS must allow clipboard reading so the earlier contents can be restored. The original and resulting values must each fit 65,536 UTF-16 units.
- **Delivery.** The paste is bound to the exact process, window and focused control. An exact receiver in the active frontmost app gets the paste chord directly. Any other target first gets a guarded click on a blank part of its title bar, which keeps the editor and selection; unsupported window chrome is refused. The route never changes after input begins, and the app isn't brought forward.
- **Clipboard.** The clipboard is restored only after a meaningful observed edit. After uncertain delivery, the replacement may stay on the clipboard, and anything copied since is kept. The result reports delivery separately from `consumption`, `clipboard_cleanup` and `clipboard_ownership`.
- **Reservation.** A reserved ownership refuses later automated clipboard writes until the edit is verified or the receiving process exits. Restarting the host doesn't clear it.

## Not supported

- **Confirmed insertion outside text fields.** Where `desktop_insert` can't observe a text control with a readable selection, such as terminals, canvases and custom views, it types the text instead, and the result is `unknown`, because nothing can read back what the receiver did with it. Typed insertion can't contain newlines or tabs.
Expand Down
6 changes: 4 additions & 2 deletions src/mcp.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { parseArgs } from "node:util";

import { type DesktopConfig, createDesktop } from "./client.js";
import { type DesktopRequest, isDesktopAction } from "./protocol.js";
import { type DesktopTool, desktopRequest, desktopTools } from "./tools.js";
import { type DesktopTool, desktopRequest, desktopToolRules, desktopTools } from "./tools.js";

const { values } = parseArgs({
options: { client: { type: "string" }, socket: { type: "string" }, name: { type: "string" } },
Expand All @@ -29,7 +29,7 @@ const { version } = JSON.parse(
// Newest first; an unsupported client version gets the newest, as the MCP lifecycle requires.
const VERSIONS = ["2025-11-25", "2025-06-18", "2025-03-26", "2024-11-05"];

// Guidance mirrors Ace's desktop extension so agents behave alike; tools come from the shared definitions.
// Guidance mirrors Ace's desktop extension so agents behave alike; tools and their shared rules come from the shared definitions.
const instructions = [
`Use desktop_* tools to observe and operate native apps on this Mac through ${host}, which holds the macOS permissions.`,
"Shell commands remain appropriate for builds, files, and preparing clipboard fixtures directly.",
Expand All @@ -43,6 +43,8 @@ const instructions = [
"Action results start with their outcome: completed, refused, or unknown.",
"For a refusal before dispatch, follow its concrete recovery hint once when applicable, then refresh the observation; a refusal alone does not prove the capability is missing.",
"After completed, interrupted, or uncertain input, inspect the current target before deciding whether another action is needed. Never replay input just because its follow-up observation failed.",
"",
...desktopToolRules(host),
].join("\n");

const tools = desktopTools({ place: "this Mac", app: host });
Expand Down
Loading
Loading