Skip to content

feat: capture a whole display with desktop_screen - #36

Merged
iamnbutler merged 1 commit into
mainfrom
feat/display-capture
Oct 17, 2026
Merged

iamnbutler merged 1 commit into
mainfrom
feat/display-capture

Conversation

@iamnbutler

@iamnbutler iamnbutler commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Removes the "Whole-screen capture" limitation from docs/tools.md.

What

New tool desktop_screen (op screen), observation only:

  • Optional display index (0–63), default 0, the main display.
  • Returns a screenshot of the whole display, resized the same way as desktop_inspect (JPEG, at most 1600 px and 900 KB).
  • Returns every active display with its index, display ID, name, bounds in global top-left points, scale and main flag.
  • Returns the on-screen windows that intersect the captured display, front to back: pid, window_id, app name, title, bounds and layer. When native inventory matches a window, it also includes the process generation, observation capability and the desktop_windows target object.
  • Returns no snapshot_id. Input stays bound to exact windows through desktop_inspect, and the tool description says so.

How

  • New client subcommand screen <absolute-output-path> [display] (native/sources/client/screen.swift). It calls desktopObservationWithOutcome with .screen(index:), detection .none, saveSnapshot: false and inline pixels. The runtime already allows this through .desktopObservation, so there are no runtime changes and no Peekaboo patch.
  • Display indices follow CGGetActiveDisplayList, which lists the main display first. Peekaboo indexes its capture engine's own display list, so the client checks the captured display's bounds against the requested display. It accepts the legacy engine's Cocoa-flipped origin too. If the bounds don't match, it tries the other indices before refusing.
  • Windows come from CGWindowListCopyWindowInfo for order and bounds, limited to levels below the Dock. Each visible app's Bridge window inventory then fills in titles and exact targets, so titles don't depend on the calling process's Screen Recording grant. Inventory is read for at most 32 apps, and a failure adds a warning.
  • The TS host reuses screenshot() and bounded() from inspect and truncates windows if the result is too large.

Verified live

Two displays: Studio Display (main, 2560×1440 pt) and the built-in display (1512×982 pt at 553,1440). I ran the signed client from this branch against the installed app's socket:

  • Default, 0 and 1 each captured the correct whole display, menu bar included, and I looked at both images. Each took about 0.5–2.3 s.
  • The window lists matched what was on screen, front to back, with targets.
  • display 5 was refused with the valid range.
  • Through createDesktop in dist/client.js, I got a JPEG plus bounded JSON and confirmed a non-integer display is refused. Passing a listed window's pid and window_id to desktop_inspect succeeded.

Adds an observation-only desktop_screen tool. It captures one display
through the Bridge's existing desktopObservation operation with a
.screen(index:) target and no detection, and returns the active displays
and the on-screen windows on that display front to back, so an agent can
find what it sees and then desktop_inspect the exact window. It returns
no input snapshot. No Peekaboo patch is needed.
@iamnbutler
iamnbutler merged commit b6c2987 into main Oct 17, 2026
2 checks passed
@iamnbutler
iamnbutler deleted the feat/display-capture branch October 17, 2026 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant