Repository navigation
feat(desktop-gpui): track masks through motion, with points around any shape - #3674
thesammykins wants to merge 14 commits into
Conversation
Expose existing mask keyframes through editable markers and local motion tracking. Four-point tracking combines corner patches with region matching to follow resizing content, with adjustable handles and an explicit start before writing a simplified path. Keep the project format unchanged and share interpolation between preview, export and canvas edits. Hold lost targets, flag results for review, preserve partial results on Stop, and reject stale jobs after concurrent edits. Share frame pyramids across trackers without changing their saved paths.
Keep precise tracking tips visible with offset numbered droplets. Allow supplemental anchors after the four bounding corners, while retaining rectangular mask output and the existing four-corner path when it succeeds. Show tracking changes only the editor preview; saved masks and export configuration remain intact. Kernel tracking tests pass, including corner loss during resizing. Scoped editor tests pass (258 passed, two ignored), kernel Clippy passes, and native placement, dragging and visibility were exercised. Nine adversarial clips retain their four-point paths with supplemental anchors.
A tracked path is simplified before it becomes mask keyframes, and the simplification lets each edge drift up to the tolerance away from the box the tracker found. On a sensitive mask tracked tightly to its target that drift uncovers a sliver of what it hides: in the editor, 101 of 916 frames across six moving-content clips leaked a few pixels at an edge. Sensitive masks now grow by twice the tolerance when a tracked path is applied, so the simplification error falls outside the content; those clips then leak on 3 frames, all slivers of a card a window is sliding over. Highlight masks keep the tracked size. Re-tracking a range starts from the grown box, so repeated runs widen it slightly, which errs towards hiding more. A regression test checks that every tracked box stays inside the simplified sensitive mask, and fails with the margin removed.
Track Points followed a rectangle's four corners with patches centred on each corner, three quarters of which is the background behind the target. When a panel moved or resized over a still page the patches stuck to the page, and the whole-box fallback could only scale uniformly, so the mask slid across a resizing panel instead of stretching (about 56% coverage). Points around a round or irregular shape had no model at all. Four or more clicked points now define the mask as the box around them. Each point is followed by a patch on it and one inside the shape, and the box is fitted per axis to the patches that agree, so it can widen without growing taller; the whole-box tracker covers frames where too few agree, and a fit is limited to 25% size change per frame. A side the user clicked along (as with a rectangle's corners) is also kept on its straight edge when its outline is one colour the page beside it does not share: a resized window's border moves while its content and the page stay still, which patch matching cannot see. Masks from points are widened by about four pixels a side, since clicks land on or just inside the outline. The picker and Tracking panel now use the canvas's selection language: hollow handles and label chips in the selection blue, the outline around the points with the mask's box drawn faintly, a dark instruction pill, the mask panel's card-style buttons and a progress bar. A selected tracked mask shows its motion path. In the editor, Track Points now leaves no leaking frame on a window resized from its corner, a stretched panel, a round badge with eight points and a rounded card; a hexagon's tips still show slivers on 24 of 179 frames when scored against its bounding box. Scenario tests in kernels/tests/mask_tracking_scenarios.rs generate these motions and score the masks as the editor saves them. Tracking sees the editor's render, not the source video; debug builds can dump those frames with CAP_GPUI_MASK_TRACK_DUMP for the mask_track_replay example.
|
@thesammykins is attempting to deploy a commit to the Cap Software Inc Team on Vercel. A member of the Team first needs to authorize it. |
| deviation(start_center, end_center, center, t) | ||
| .max(deviation(start_size, end_size, size, t) / 2.0), |
There was a problem hiding this comment.
simplify_path bounds center error and half-size error separately, but those errors add at an edge. With tolerance 0.003, let the endpoints have x-center 0.5 and width 0.2, and the middle sample have x-center 0.4976 and width 0.2048. Both measured errors are 0.0024, so the middle sample is dropped. The grown mask's left edge is then 0.397, while the tracked box reaches 0.3952, leaving sensitive content exposed.
Measure the error of all four edges directly, or grow the margin enough to cover both errors.
How this was verified: The simplifier permits both errors together, while the renderer places each edge at center plus or minus half the size.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/project/src/mask_tracking.rs
Line: 144-145
Comment:
**Saved masks expose an edge**
`simplify_path` bounds center error and half-size error separately, but those errors add at an edge. With tolerance `0.003`, let the endpoints have x-center `0.5` and width `0.2`, and the middle sample have x-center `0.4976` and width `0.2048`. Both measured errors are `0.0024`, so the middle sample is dropped. The grown mask's left edge is then `0.397`, while the tracked box reaches `0.3952`, leaving sensitive content exposed.
Measure the error of all four edges directly, or grow the margin enough to cover both errors.
**How this was verified:** The simplifier permits both errors together, while the renderer places each edge at center plus or minus half the size.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Valid, thanks. Fixed in 82a66a0: the simplifier now measures how far each of the four mask edges strays from the line between kept samples, which is what the margin is sized against. A test with your case (centre 0.4976, width 0.2048 between samples at 0.5 and 0.2) keeps the sample and covers the tracked edge; it fails on the old measure.
| let (region, lost) = self | ||
| .region | ||
| .map_or_else(|| (Vec::new(), Vec::new()), TrackSession::finish); |
There was a problem hiding this comment.
PointTrackSession can start without a whole-box tracker, but finish then returns no lost spans regardless of unresolved. Points spanning a 4–7-pixel-wide target pass the point tracker's minimum size, while MotionTracker requires 8 pixels. The surrounding 16-pixel patches can still have enough detail to start tracking.
If those patches later lose the target, the saved mask holds without the promised review warning. Build lost spans from unresolved frames that remain unrecovered, including when region is None.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/kernels/src/point_track.rs
Line: 218-220
Comment:
**Lost targets go unreported**
`PointTrackSession` can start without a whole-box tracker, but `finish` then returns no lost spans regardless of `unresolved`. Points spanning a 4–7-pixel-wide target pass the point tracker's minimum size, while `MotionTracker` requires 8 pixels. The surrounding 16-pixel patches can still have enough detail to start tracking.
If those patches later lose the target, the saved mask holds without the promised review warning. Build lost spans from unresolved frames that remain unrecovered, including when `region` is `None`.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Valid. Fixed in e17e7d4: lost spans now come from the frames point tracking left unplaced, less any the whole-box tracker recovered, so they are reported whether or not that tracker exists. The corner test now checks that its hidden frames come back as one span for review.
| /// A tracked mask's path on the canvas while it is selected: a faint line | ||
| /// through its markers' centres, each marker a small handle-style dot. |
There was a problem hiding this comment.
The new render_motion_path comment only describes what the function draws. AGENTS.md requires comments to capture non-obvious context and explicitly bans comments that narrate the code. Remove this comment or replace it with a reason the implementation needs explaining. The same pattern appears above tracking_label_centers and points_outline.
This repository requirement must be satisfied before merging.
Context Used: AGENTS.md (source)
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/editor_mask_tracking.rs
Line: 111-112
Comment:
**Comments repeat the code**
The new `render_motion_path` comment only describes what the function draws. `AGENTS.md` requires comments to capture non-obvious context and explicitly bans comments that narrate the code. Remove this comment or replace it with a reason the implementation needs explaining. The same pattern appears above `tracking_label_centers` and `points_outline`.
This repository requirement must be satisfied before merging.
**Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md))
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Agreed. Fixed in a7ca064: the comments on render_motion_path, tracking_label_centers and points_outline now give the reasons (labels sit off their points so they never hide the pixel being aimed at; the hull is used because clicks around a shape come in any order) or are removed.
… a pixel On a real screen recording of a terminal window being dragged, Track Motion lost the window as soon as it moved and leaked on 125 of 249 frames. At tracking resolution the terminal's text is a pixel or two tall, and a shift by a fraction of a pixel resamples it into a different pattern: the correct match scored about 0.47 against the 0.5 lost threshold, while the window at rest scored 0.999. When the sharp match fails, the tracker now retries on lightly smoothed frames and templates ([1, 2, 1] in each direction), where the same match scores about 0.75, and accepts it from 0.65. Smoothing is only a fallback: applied to every match it also let a feed of look-alike cards match each other and broke the corner tests. Frames buffered while the target is lost are kept as pyramids so the smoothed levels are built once. Track Points now accepts any box the whole-box tracker reports for a frame, rather than only matches above its patch threshold, which discarded these. On the recording, Track Motion now leaves no leaking frame (0 of 249) and Track Points 8. A test drags a window of one-pixel text strokes by fractional pixels, and loses it with the fallback disabled. The mask_track_replay example now takes the frame size, since the editor tracks at 960 wide or less to keep a recording's aspect ratio.
When a remembered straight edge is missed for a frame, its side used to stay where the edge was last seen. That holds a resized window's side in place, which is the point, but pinned the side of a window dragged fast enough to outrun the edge search, uncovering up to a quarter of it for a few frames. A held side now keeps its correction to the estimate instead, moving by as much as the estimate has since the edge was last found; on a resize the estimate barely moves, so the side still holds. Also replaces a closure passed to bool::then that clippy rejects.
Simplifying a tracked path bounded the centre's error and half the size's error separately, but a mask edge sits at the centre plus or minus half the size, so the two errors add there. A sample whose centre and width each moved just under the tolerance could be dropped while its left edge moved by nearly twice it, more than the sensitive mask's margin covers. The simplifier now measures how far each of the four edges strays from the line between the kept samples, which is what the margin is sized against. A test with the case from review (centre 0.4976, width 0.2048 between two samples at 0.5 and 0.2) keeps the sample and covers the tracked edge; it fails on the old measure.
Point tracking reported lost spans only from its whole-box tracker, which is not created for targets under eight pixels across. Frames neither the points nor anything else could place then went unreported, and the editor saved the held mask without asking the user to review it. Lost spans now come from the frames point tracking left unplaced, less any the whole-box tracker recovered afterwards, so they are reported either way. The corner test checks that frames where its target is hidden come back as one span for review.
Replace comments that restated what three picker helpers draw with the reasons behind them: labels sit off their points so they never hide the pixel being aimed at, and the outline is the convex hull because clicks around a shape come in any order.
…g window On a screen recording of a terminal window dragged around a desktop, Track Points still uncovered part of the window on 9 of 249 frames, and more when the corners were clicked a pixel or two inside the window. What went wrong when the window moved fast: - A straight edge seen near where it was last found was trusted even when the whole window had moved: dragged up by its title bar's height, the line under the title bar lands where the top edge was. An edge now stays put only while the target as a whole (both opposite sides of the estimate) moved less than an edge may drift; otherwise the qualifying line nearest where the estimate puts the edge wins, not the one most positions agree on. - A point fit backed by three patches could come from two clicked points, each point having up to two patches, and overrule a whole-box track that was right. A fit now needs patches from three distinct points. - Where estimates disagreed, one was picked and could be the wrong one. Sensitive masks now cover all of them when they broadly agree (overlap of at least half): the point fit, the whole-box track and the box snapped to straight edges. A slightly larger mask hides nothing it should show; highlight masks keep following the target closely. A missed edge takes whichever of its held position and the estimate lies further out. - Clicks inside the outline left the mask short of it. The outline search now starts a few pixels outside the clicked side and the box starts on it; where no straight outline can be found (a side over a dark dock), tracked masks get a margin of 0.6% of the frame's width a side, in equal pixels on both axes. Sensitive Track Motion masks get the same margin, for the few pixels a box lags a fast zoom. On that recording, scored against the window's position found at full resolution, Track Motion and Track Points leave every frame fully covered, with corner clicks from a pixel outside the outline to four inside it, and no frame of the exported video keeps readable detail in the window or along its edges. The generated scenarios are fully covered except one 25 px² frame where a window slides over the card and three at a hexagon's tips.
|
@greptile review |
| .filter(|found| found.score >= SMOOTHED_SCORE)?; | ||
| return Some(self.accept(pyramid, found, 1.0)); |
There was a problem hiding this comment.
Zooming text outgrows its mask
When fine text needs the new smoothed fallback during a zoom, follow accepts the match with scale 1.0 and returns before trying other sizes. The box stays the same size on those frames even as the target grows. Once that growth exceeds the fixed margin, a sensitive mask can expose content without a review warning, because these matches count as found.
Run the size search on the smoothed frames too before accepting the match.
How this was verified: The smoothed branch returns before the size search, and TrackSession saves successful matches without marking those frames as lost.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/kernels/src/motion_track.rs
Line: 677-678
Comment:
**Zooming text outgrows its mask**
When fine text needs the new smoothed fallback during a zoom, `follow` accepts the match with scale `1.0` and returns before trying other sizes. The box stays the same size on those frames even as the target grows. Once that growth exceeds the fixed margin, a sensitive mask can expose content without a review warning, because these matches count as found.
Run the size search on the smoothed frames too before accepting the match.
**How this was verified:** The smoothed branch returns before the size search, and `TrackSession` saves successful matches without marking those frames as lost.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Valid, thanks. Fixed in efe81db: the size search now runs on whichever frames found the target, sharp or smoothed, with that path's own score floor. A new test zooms one-pixel text by 1.2% a frame while it drifts; the box now stays within 3.5 px of it, where the old code held its first size (160 px wide against 218) and lost the target.
When fine text was only found on the smoothed frames, the tracker accepted that match at the box's current size and skipped the search over nearby sizes. Text zooming while it moved kept its first size on those frames, so a sensitive mask fell behind the growth until the margin no longer covered it, and the frames still counted as found, so nothing asked for review. The size search now runs on whichever frames found the target, sharp or smoothed, with that path's own score floor. A test zooms one-pixel text by 1.2% a frame while it drifts: the box now stays within 3.5 pixels of it (the 3.5% size steps), where before it held 160 pixels wide against 218 and lost the target. On the dragged-window recording both tracking modes still leave every frame covered.
|
@greptile review |
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
|
@greptile review |
|
Thanks! Can you rebase on main to clear the conflicts? Then we'll take a look. |
Sensitive masks in the GPUI editor stay where they are drawn, so any blur or pixelate over moving content leaks as soon as the content moves. This adds motion tracking to mask segments: Track Motion follows the box as drawn, and Track Points follows four or more points clicked on the target (a rectangle's corners, or points around a rounded or irregular shape). Tracking writes the mask's existing position and size keyframes, so the project format is unchanged and preview, export and canvas edits share one interpolation. Lost targets are held and flagged for review, Stop keeps partial results, and a job is discarded if the project changes while it runs.
On a real recording
A screen recording of a terminal window being dragged around a busy desktop (3600×2338, imported through File › Import Media), with a sensitive Pixelate mask drawn over the window on the first frame. The window's true box in every frame was found by matching its first-frame image at full resolution, and a frame counts as failed if any visible window pixel falls outside the mask:
In the exported video, no frame keeps readable detail inside the window or along any of its edges (the static mask leaves 18 frames readable).
Track Points
Each point is followed by a small patch on it and one inside the shape, and the box is fitted per axis to the patches that agree (from at least three different points), so it can widen without growing taller. A whole-box tracker runs alongside for frames where too few points agree. A side the user clicked along, as with a rectangle's corners, is also kept on its straight edge when the outline there is one colour the page beside it does not share: a window resized from its corner has a border that moves while its content and the page stay still, which patch matching alone cannot see. The box around a round shape only touches its sides at a point, so round shapes rely on the patches and the whole-box tracker.
For sensitive masks, where the point fit, the whole-box track and the edge-snapped box disagree but broadly overlap, the mask covers all of them: a mask a little too large hides nothing it should show. Highlight masks keep the most trusted estimate and follow the target closely.
The window-resize recording uses a Highlight mask so the tracked box is visible; pixelating a page of text-like dashes looks much like the dashes.
Fine text
At tracking resolution a terminal's text is a pixel or two tall, and a window dragged by a fraction of a pixel resamples it into a different pattern; the correct match scored below the "lost" threshold, so the window was lost as soon as it moved. When a sharp match fails, the tracker retries on lightly smoothed frames, where the same match scores well, and measures how the target grew or shrank on those frames too, so zooming text keeps its box growing. Smoothing is only a fallback because it also lets look-alikes match each other.
Margins
Tracked paths are simplified into keyframes, and simplification may move each mask edge up to the tolerance (0.3% of the frame); sensitive masks grow by that much so the error falls outside the content. Tracked masks also grow by 0.6% of the frame's width a side, the same number of pixels on both axes: for Track Points this covers clicks landing a few pixels inside the outline on a side with no straight outline to find, and for sensitive Track Motion masks the few pixels a box lags a fast zoom.
Picker and panel
The picker and the Tracking section use the canvas's selection language: hollow handles and label chips in the selection blue, the outline around the points with the mask's box drawn faintly, the dark instruction pill used elsewhere on the canvas, and the mask panel's card-style buttons. Tracking shows a progress bar, and a selected tracked mask shows its motion path.
Generated scenarios
Synthetic 1280×720 clips with the target's true box known every frame, tracked in the built editor and scored on the saved mask (any visible target pixel outside the mask fails the frame):
The hexagon is scored against its bounding box, which counts the empty corners around each tip as content.
kernels/tests/mask_tracking_scenarios.rsgenerates these motions in code and scores each mask the way the editor saves it.Tracking sees the editor's render rather than the source video, and the two differ in colour and sharpness enough to change results. Debug builds write the frames the tracker sees with
CAP_GPUI_MASK_TRACK_DUMP=<file>, and themask_track_replaykernels example replays them outside the editor.Worth a close look:
kernels/src/box_edges.rs(which sides qualify as edges, when an edge may stay put, and where a missed edge goes) and the estimate selection inkernels/src/point_track.rs, since they decide when the tracker overrides what the patches report.

Confidence Score: 5/5
The PR appears safe to merge; the previous findings are addressed and the latest change only removes a comment.Summary
This PR adds motion and point tracking for editor masks, saves tracked paths as existing keyframes, and adds tracking controls and scenario tests.
Reviews (4) · Last reviewed commit: "Update apps/desktop-gpui/kernels/src/mot..." · Reviewed by Greptile