Repository navigation
feat: capture a whole display with desktop_screen - #36
Merged
Merged
Conversation
Adds an observation-only desktop_screen tool. It captures one display through the Bridge's existing desktopObservation operation with a .screen(index:) target and no detection, and returns the active displays and the on-screen windows on that display front to back, so an agent can find what it sees and then desktop_inspect the exact window. It returns no input snapshot. No Peekaboo patch is needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Removes the "Whole-screen capture" limitation from docs/tools.md.
What
New tool
desktop_screen(opscreen), observation only:displayindex (0–63), default 0, the main display.desktop_inspect(JPEG, at most 1600 px and 900 KB).desktop_windowstarget object.snapshot_id. Input stays bound to exact windows throughdesktop_inspect, and the tool description says so.How
screen <absolute-output-path> [display](native/sources/client/screen.swift). It callsdesktopObservationWithOutcomewith.screen(index:), detection.none,saveSnapshot: falseand inline pixels. The runtime already allows this through.desktopObservation, so there are no runtime changes and no Peekaboo patch.CGGetActiveDisplayList, which lists the main display first. Peekaboo indexes its capture engine's own display list, so the client checks the captured display's bounds against the requested display. It accepts the legacy engine's Cocoa-flipped origin too. If the bounds don't match, it tries the other indices before refusing.CGWindowListCopyWindowInfofor order and bounds, limited to levels below the Dock. Each visible app's Bridge window inventory then fills in titles and exact targets, so titles don't depend on the calling process's Screen Recording grant. Inventory is read for at most 32 apps, and a failure adds a warning.screenshot()andbounded()from inspect and truncateswindowsif the result is too large.Verified live
Two displays: Studio Display (main, 2560×1440 pt) and the built-in display (1512×982 pt at 553,1440). I ran the signed client from this branch against the installed app's socket:
0and1each captured the correct whole display, menu bar included, and I looked at both images. Each took about 0.5–2.3 s.display 5was refused with the valid range.createDesktopindist/client.js, I got a JPEG plus bounded JSON and confirmed a non-integer display is refused. Passing a listed window's pid and window_id todesktop_inspectsucceeded.