feat: report focus, appearance, session and input state with the app inventory

dist/tools.js

@@ -175,7 +175,7 @@ export function desktopTools({ place, app }) {
         {
             name: "desktop_apps",
             op: "apps",
-            description: `List native applications on ${place}. Optional query searches names and bundle IDs case-insensitively before result truncation; use it to find apps omitted from a large inventory. A query must contain non-whitespace text and at most 256 characters. Filter counts cover only the native inventory returned by this call; native completeness and truncation still apply. Use an application's PID with desktop_windows to select a window to inspect. Activity and visibility are unknown unless is_active_known and is_hidden_known respectively are true; read metadata_warnings for missing evidence.`,
+            description: `List native applications on ${place}. Optional query searches names and bundle IDs case-insensitively before result truncation; use it to find apps omitted from a large inventory. A query must contain non-whitespace text and at most 256 characters. Filter counts cover only the native inventory returned by this call; native completeness and truncation still apply. Use an application's PID with desktop_windows to select a window to inspect. Activity and visibility are unknown unless is_active_known and is_hidden_known respectively are true; read metadata_warnings for missing evidence. system describes the desktop as a whole: focus is the frontmost app's pid, bundle_id and name, its frontmost window_id and its focused element (role, title, identifier, bounds in global desktop points, window_id; never its value), which is where keyboard input and desktop_system_key would land; appearance gives mode dark or light, whether it switches automatically, the accent color and the accessibility display settings (increase_contrast, reduce_motion, reduce_transparency, invert_colors, differentiate_without_color); session gives locked, on_console, screensaver_running, displays_asleep and secure_keyboard_entry, which blocks typed text; input gives the keyboard layout desktop_key maps characters through and the pointer position with its display index; menu_bar_autohides and dock_autohides explain visible_bounds. A focused element that can't be read within 300 ms is null, with a warning in system.warnings. This reads only and sends no input.`,
             parameters: {
                 type: "object",
                 properties: {

docs/tools.md

@@ -76,6 +76,8 @@ Window operations other than focus use background Accessibility and don't activa
 
 Both `desktop_spaces` and `desktop_screen` describe each display: its `desktop_screen` index, name, whether it is the main display (index 0, which holds the menu bar and the coordinate origin), `bounds` and `visible_bounds` in global desktop points with a top-left origin, as window bounds use, `scale` (pixels per point), `pixel_width` and `pixel_height`, and `rotation` in degrees. `visible_bounds` is what is left after the menu bar and Dock, the area `desktop_maximize` fills; a display left of or above the main one has negative coordinates. `desktop_spaces` reads this without a capture, so prefer it when only the arrangement is needed.
 
+`desktop_apps` also returns `system`, the desktop-wide state worth checking before acting. `focus` names the frontmost app, its frontmost window and its focused element by role, title, identifier and bounds, which is where keyboard input and `desktop_system_key` would land; the element's value is never included, so field contents stay behind `desktop_read`. `appearance` gives dark or light mode, automatic switching, the accent color and the accessibility display settings that change how apps draw. `session` says whether the screen is locked, the screensaver runs, every display sleeps, or secure keyboard entry is on, which refuses typed text. `input` gives the keyboard layout, which `desktop_key` uses to find character keys, and the pointer's position and display. `menu_bar_autohides` and `dock_autohides` explain `visible_bounds`. Everything but the focused element comes from the client process without Accessibility; that one read goes through the host with its own 300 ms limit, after the inventory, so a slow app leaves `focus.element` null with a warning instead of delaying the call. Focus modes such as Do Not Disturb have no public read and are not reported.
+
 `desktop_switch_space` is foreground input. It presses Move left a space or Move right a space (Control-Left and Control-Right by default) once per step, reading the Spaces back after each, and reports `completed` only when WindowServer shows the target Space as current. Otherwise the result is `unknown` and names the Space that is current now. macOS applies those shortcuts to the display that holds the pointer or the key window, so with separate Spaces per display the switch is refused before any key unless both are on the target Space's display. `focus` in `desktop_spaces` shows where they are; use `desktop_hover` to move the pointer and `desktop_focus` to bring a window there forward. It is also refused when either shortcut is turned off or rebound to something other than an arrow key with modifiers in Keyboard Shortcuts > Mission Control.
 
 Mission Control and App Exposé are `desktop_system_key` with `control` and `up` or `down`.

native/sources/client/client.swift

@@ -148,7 +148,9 @@ private enum Client {
 				apps: inventory.items.map { App($0, metadata: matched[$0.processIdentifier], activityKnown: activityKnown) },
 				inventory_completeness: inventory.completeness.rawValue,
 				inventory_warnings: inventory.warnings,
-				metadata_warnings: warnings
+				metadata_warnings: warnings,
+				// Read after the inventory; its focus read has its own short limit.
+				system: await systemState(client, handshake: handshake)
 			))
 		case "windows":
 			let pid = try processID(args[2])
@@ -337,6 +339,7 @@ private struct Apps: Encodable {
 	let inventory_completeness: String
 	let inventory_warnings: [String]
 	let metadata_warnings: [String]
+	let system: SystemState
 }
 
 private struct App: Encodable {

native/sources/client/spaces.swift

@@ -120,12 +120,12 @@ private func display(for uuid: String, in displays: [Display]) -> Display? {
 	}
 }
 
-private func rect(_ bounds: Bounds) -> CGRect {
+func rect(_ bounds: Bounds) -> CGRect {
 	CGRect(x: bounds.x, y: bounds.y, width: bounds.width, height: bounds.height)
 }
 
 /// The display holding most of a rectangle in global top-left points.
-private func displayIndex(of area: CGRect, in displays: [Display]) -> Int? {
+func displayIndex(of area: CGRect, in displays: [Display]) -> Int? {
 	var best: (index: Int, area: CGFloat)?
 	for display in displays {
 		let overlap = rect(display.bounds).intersection(area)
@@ -143,7 +143,7 @@ private struct Focus {
 	let windowDisplay: Int?
 }
 
-private func normalWindows(_ options: CGWindowListOption) -> [(pid: Int32, id: Int, app: String?, bounds: CGRect)] {
+func normalWindows(_ options: CGWindowListOption) -> [(pid: Int32, id: Int, app: String?, bounds: CGRect)] {
 	let own = getpid()
 	let list = CGWindowListCopyWindowInfo(options, kCGNullWindowID) as? [[String: Any]] ?? []
 	return list.compactMap { info in

native/sources/client/system.swift

@@ -0,0 +1,190 @@
+import AppKit
+import Carbon
+import CoreGraphics
+import Foundation
+import PeekabooAutomationKit
+import PeekabooBridge
+
+/// The focused element read waits this long at most, so a slow app never delays the inventory.
+private let focusReadLimit: Duration = .milliseconds(300)
+
+/// Desktop-wide state an agent reads before acting: where input would land, appearance, session and input.
+struct SystemState: Encodable {
+	let focus: SystemFocus
+	let appearance: Appearance
+	let session: Session
+	let input: Input
+	let menu_bar_autohides: Bool
+	let dock_autohides: Bool
+	let warnings: [String]
+	let note = "Read when this inventory was taken. focus is where keyboard input to the frontmost app would land; element omits the control's value, so read text with desktop_read. Positions are global desktop points."
+}
+
+struct SystemFocus: Encodable {
+	let pid: Int32?
+	let bundle_id: String?
+	let app_name: String?
+	let window_id: Int?
+	let element: FocusedElement?
+}
+
+struct FocusedElement: Encodable {
+	let role: String
+	let title: String?
+	let identifier: String?
+	let bounds: Bounds
+	let window_id: Int?
+}
+
+struct Appearance: Encodable {
+	let mode: String
+	let switches_automatically: Bool
+	let accent_color: String?
+	let increase_contrast: Bool
+	let reduce_motion: Bool
+	let reduce_transparency: Bool
+	let invert_colors: Bool
+	let differentiate_without_color: Bool
+}
+
+struct Session: Encodable {
+	let locked: Bool
+	let on_console: Bool
+	let screensaver_running: Bool
+	let displays_asleep: Bool
+	let secure_keyboard_entry: Bool
+}
+
+struct Input: Encodable {
+	let keyboard_layout: KeyboardLayout?
+	let pointer: Pointer?
+}
+
+struct KeyboardLayout: Encodable {
+	let id: String
+	let name: String?
+}
+
+struct Pointer: Encodable {
+	let x: Double
+	let y: Double
+	let display_index: Int?
+}
+
+func systemState(_ client: PeekabooBridgeClient, handshake: PeekabooBridgeHandshakeResponse) async -> SystemState {
+	var warnings: [String] = []
+	let workspace = NSWorkspace.shared
+	let front = workspace.frontmostApplication
+	let pid = front?.processIdentifier
+	let window = pid.flatMap { pid in
+		normalWindows([.optionOnScreenOnly, .excludeDesktopElements]).first { $0.pid == pid }?.id
+	}
+	var element: FocusedElement?
+	if let pid {
+		if (handshake.enabledOperations ?? handshake.supportedOperations).contains(.getFocusedElement) {
+			switch await focusedElement(client, pid: pid) {
+			case let .read(info):
+				element = info.map {
+					FocusedElement(
+						role: $0.role, title: $0.title, identifier: $0.identifier, bounds: Bounds($0.frame),
+						window_id: $0.windowID
+					)
+				}
+			case .timedOut:
+				warnings.append("The focused element read took longer than 300 ms, so focus.element is unknown.")
+			case .failed:
+				warnings.append("The focused element could not be read through Accessibility, so focus.element is unknown.")
+			}
+		} else {
+			warnings.append("This desktop runtime does not read the focused element; update it to see focus.element.")
+		}
+	}
+	let session = CGSessionCopyCurrentDictionary() as NSDictionary?
+	let displays = (try? activeDisplays()) ?? []
+	let pointer = CGEvent(source: nil).map(\.location).map { point in
+		Pointer(
+			x: point.x, y: point.y,
+			display_index: displays.first { rect($0.bounds).contains(point) }?.index
+		)
+	}
+	let defaults = UserDefaults.standard
+	let dark = await MainActor.run {
+		NSApplication.shared.effectiveAppearance.bestMatch(from: [.aqua, .darkAqua]) == .darkAqua
+	}
+	var asleep = false
+	var count: UInt32 = 0
+	var ids = [CGDirectDisplayID](repeating: 0, count: 16)
+	if CGGetOnlineDisplayList(16, &ids, &count) == .success, count > 0 {
+		asleep = ids.prefix(Int(count)).allSatisfy { CGDisplayIsAsleep($0) != 0 }
+	}
+	return SystemState(
+		focus: SystemFocus(
+			pid: pid, bundle_id: front?.bundleIdentifier, app_name: front?.localizedName,
+			// A login or full-screen panel isn't a normal-level window; its focused element still names it.
+			window_id: window ?? element?.window_id,
+			element: element
+		),
+		appearance: Appearance(
+			mode: dark ? "dark" : "light",
+			switches_automatically: defaults.bool(forKey: "AppleInterfaceStyleSwitchesAutomatically"),
+			accent_color: accentColor(defaults.object(forKey: "AppleAccentColor") as? Int),
+			increase_contrast: workspace.accessibilityDisplayShouldIncreaseContrast,
+			reduce_motion: workspace.accessibilityDisplayShouldReduceMotion,
+			reduce_transparency: workspace.accessibilityDisplayShouldReduceTransparency,
+			invert_colors: workspace.accessibilityDisplayShouldInvertColors,
+			differentiate_without_color: workspace.accessibilityDisplayShouldDifferentiateWithoutColor
+		),
+		session: Session(
+			locked: session?["CGSSessionScreenIsLocked"] as? Bool == true,
+			on_console: session?["kCGSSessionOnConsoleKey"] as? Bool ?? false,
+			screensaver_running: workspace.runningApplications.contains {
+				$0.bundleIdentifier == "com.apple.ScreenSaver.Engine"
+			},
+			displays_asleep: asleep,
+			secure_keyboard_entry: IsSecureEventInputEnabled()
+		),
+		input: Input(keyboard_layout: keyboardLayout(), pointer: pointer),
+		menu_bar_autohides: defaults.bool(forKey: "_HIHideMenuBar"),
+		dock_autohides: UserDefaults(suiteName: "com.apple.dock")?.bool(forKey: "autohide") ?? false,
+		warnings: warnings
+	)
+}
+
+private enum FocusRead {
+	case read(UIFocusInfo?)
+	case timedOut
+	case failed
+}
+
+/// Races the host's focused element read against its own short limit; a late answer is discarded.
+private func focusedElement(_ client: PeekabooBridgeClient, pid: Int32) async -> FocusRead {
+	await withTaskGroup(of: FocusRead.self) { group in
+		group.addTask {
+			do { return .read(try await client.getFocusedElement(targetProcessIdentifier: pid)) } catch { return .failed }
+		}
+		group.addTask {
+			try? await Task.sleep(for: focusReadLimit)
+			return .timedOut
+		}
+		let first = await group.next() ?? .failed
+		group.cancelAll()
+		return first
+	}
+}
+
+/// System Settings' accent colors by their stored index; no value means the multicolor default.
+private func accentColor(_ index: Int?) -> String? {
+	guard let index else { return "multicolor" }
+	return [-1: "graphite", 0: "red", 1: "orange", 2: "yellow", 3: "green", 4: "blue", 5: "purple", 6: "pink"][index]
+}
+
+private func keyboardLayout() -> KeyboardLayout? {
+	guard let source = TISCopyCurrentKeyboardLayoutInputSource()?.takeRetainedValue(),
+		let id = TISGetInputSourceProperty(source, kTISPropertyInputSourceID)
+	else { return nil }
+	let name = TISGetInputSourceProperty(source, kTISPropertyLocalizedName)
+	return KeyboardLayout(
+		id: Unmanaged<CFString>.fromOpaque(id).takeUnretainedValue() as String,
+		name: name.map { Unmanaged<CFString>.fromOpaque($0).takeUnretainedValue() as String }
+	)
+}

native/sources/runtime/runtime.swift

@@ -81,7 +81,7 @@ private final class Desktop {
 					allowlistedTeams: [identity.team],
 					allowlistedBundles: [client],
 					allowedOperations: [
-						.listApplications, .listWindows, .listMenus, .menuCommand, .desktopObservation, .inspectAccessibilityTree,
+						.listApplications, .getFocusedElement, .listWindows, .listMenus, .menuCommand, .desktopObservation, .inspectAccessibilityTree,
 						.createSnapshot, .cleanSnapshot, .ownsSnapshot, .getDetectionResult, .beginSnapshotMutation,
 						.finishSnapshotMutation,
 						.targetedClick, .exactWindowTargetedClick, .setValue, .performAction, .exactWindowTargetedHotkey, .hotkey, .exactWindowTargetedTypeActions,

src/tools.ts

@@ -193,7 +193,7 @@ export function desktopTools({ place, app }: { place: string; app: string }): De
 			name: "desktop_apps",
 			op: "apps",
 			description:
-				`List native applications on ${place}. Optional query searches names and bundle IDs case-insensitively before result truncation; use it to find apps omitted from a large inventory. A query must contain non-whitespace text and at most 256 characters. Filter counts cover only the native inventory returned by this call; native completeness and truncation still apply. Use an application's PID with desktop_windows to select a window to inspect. Activity and visibility are unknown unless is_active_known and is_hidden_known respectively are true; read metadata_warnings for missing evidence.`,
+				`List native applications on ${place}. Optional query searches names and bundle IDs case-insensitively before result truncation; use it to find apps omitted from a large inventory. A query must contain non-whitespace text and at most 256 characters. Filter counts cover only the native inventory returned by this call; native completeness and truncation still apply. Use an application's PID with desktop_windows to select a window to inspect. Activity and visibility are unknown unless is_active_known and is_hidden_known respectively are true; read metadata_warnings for missing evidence. system describes the desktop as a whole: focus is the frontmost app's pid, bundle_id and name, its frontmost window_id and its focused element (role, title, identifier, bounds in global desktop points, window_id; never its value), which is where keyboard input and desktop_system_key would land; appearance gives mode dark or light, whether it switches automatically, the accent color and the accessibility display settings (increase_contrast, reduce_motion, reduce_transparency, invert_colors, differentiate_without_color); session gives locked, on_console, screensaver_running, displays_asleep and secure_keyboard_entry, which blocks typed text; input gives the keyboard layout desktop_key maps characters through and the pointer position with its display index; menu_bar_autohides and dock_autohides explain visible_bounds. A focused element that can't be read within 300 ms is null, with a warning in system.warnings. This reads only and sends no input.`,
 			parameters: {
 				type: "object",
 				properties: {