fix(browser): recover stalled screenshots and preserve image coordinates

[waleedlatif1]

Summary

• Recover screenshots after a native capture stalls using a bounded, cancellable frame subscription. Keep the native-capture guard and existing page identity, focus, and viewport checks.
• Put actual crop bounds, encoded dimensions, and independent X/Y coordinate mapping in model-visible image captions. Share the production formatter with the BrowserGym evaluator.
• Align coordinate guidance and regenerate the paired browser tool descriptions.

Type of Change

• Bug fix

Testing

• 17 real Electron tests, 1,657 desktop unit tests, and 81 browser result/execution tests passed. The injected permanent-capture-stall regression fails on baseline and passes with this change.
• All 26 workspace type-checks, lint/lint-check, all 46 audits including API validation, generators, and manifest checks passed after syncing staging.
• A same-base comparison found exactly one new, dependency-free formatter module in each affected home/chat/workflow graph. Regenerated those four graph baselines after syncing staging; audit tolerances are unchanged.
• Paired BrowserGym capture comparison: normal development 39/40→37/40 and held-out seeds 36/40→37/40, so no general improvement claim. Injected permanent-capture-stall tasks improved 5/16→16/16 in development and 5/16→14/16 on held-out seeds; holdout screenshot failures fell from 63/63 to 0/22. One development baseline closed its task tab and is retained as an adjudicated failure with missing usage. Different runner versions are reported separately; this does not establish production fault frequency or general site success rates.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Companion PRs

• simstudioai/mothership#506

Companion: simstudioai/mothership#506

Merge this caption/geometry change first, then the paired Mothership guidance/contract update. Both target staging; the backend parser correction is independently compatible.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 16, 2026 8:23pm UTC

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[github-actions]

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

• ❌ simstudioai/mothership#506 — OPEN, not merged (targets staging) — fix(browser): preserve Gemini tool identities and add BrowserGym evaluation

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no new actionable defect or outstanding repository-rule violation was identified.

Summary

This PR makes desktop browser screenshots recover from permanently stalled native captures without replacing the active page, and reports precise encoded-image and crop geometry for model coordinate conversion.

• Falls back from a timed-out native surface copy to a bounded, cancellable frame subscription.
• Preserves page, viewport, and element identity checks while serializing screenshot capture per tab.
• Reports rounded and clamped crop bounds plus independent X/Y image scales.
• Shares screenshot result formatting between production execution and evaluator paths.
• Adds unit and real-Electron coverage for stalled captures, cancellation, cleanup, repeated recovery, and fractional crops.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Request[browser_screenshot] --> Guard{Capture already active?}
  Guard -->|Yes| Reject[Reject overlapping capture]
  Guard -->|No| Pending{Native capture already pending?}
  Pending -->|No| Native[Attempt bounded capturePage]
  Native -->|Success| Validate[Validate page, viewport, and image]
  Native -->|Timeout| Frame[Subscribe to one bounded frame]
  Pending -->|Yes| Frame
  Frame --> Validate
  Validate --> Crop[Round and clamp optional crop]
  Crop --> Encode[Resize and JPEG encode]
  Encode --> Geometry[Return imageSize, viewport, and actual clip]
  Geometry --> Caption[Format independent X/Y coordinate mapping]
  Caption --> Model[Image attachment and model-visible caption]

Loading

_{Reviews (2) · Last reviewed commit: "chore(browser): sync staging and regener..."}

[cubic-dev-ai]

No issues found across 13 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 13 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}