Skip to content

feat(desktop-gpui): track masks through motion, with points around any shape - #3674

Open
thesammykins wants to merge 14 commits into
CapSoftware:mainfrom
thesammykins:feat/gpui-mask-tracking
Open

thesammykins wants to merge 14 commits into
CapSoftware:mainfrom
thesammykins:feat/gpui-mask-tracking

Conversation

@thesammykins

@thesammykins thesammykins commented Nov 3, 2026 •

Copy link
Copy Markdown

Sensitive masks in the GPUI editor stay where they are drawn, so any blur or pixelate over moving content leaks as soon as the content moves. This adds motion tracking to mask segments: Track Motion follows the box as drawn, and Track Points follows four or more points clicked on the target (a rectangle's corners, or points around a rounded or irregular shape). Tracking writes the mask's existing position and size keyframes, so the project format is unchanged and preview, export and canvas edits share one interpolation. Lost targets are held and flagged for review, Stop keeps partial results, and a job is discarded if the project changes while it runs.

On a real recording

A screen recording of a terminal window being dragged around a busy desktop (3600×2338, imported through File › Import Media), with a sensitive Pixelate mask drawn over the window on the first frame. The window's true box in every frame was found by matching its first-frame image at full resolution, and a frame counts as failed if any visible window pixel falls outside the mask:

Mask Frames failed (of 249)
Static 220
Track Motion 0
Track Points, corners clicked from 1 px outside to 4 px inside the window 0

In the exported video, no frame keeps readable detail inside the window or along any of its edges (the static mask leaves 18 frames readable).

Static mask, Track Motion and Track Points on a dragged terminal window

Track Points

Each point is followed by a small patch on it and one inside the shape, and the box is fitted per axis to the patches that agree (from at least three different points), so it can widen without growing taller. A whole-box tracker runs alongside for frames where too few points agree. A side the user clicked along, as with a rectangle's corners, is also kept on its straight edge when the outline there is one colour the page beside it does not share: a window resized from its corner has a border that moves while its content and the page stay still, which patch matching alone cannot see. The box around a round shape only touches its sides at a point, so round shapes rely on the patches and the whole-box tracker.

For sensitive masks, where the point fit, the whole-box track and the edge-snapped box disagree but broadly overlap, the mask covers all of them: a mask a little too large hides nothing it should show. Highlight masks keep the most trusted estimate and follow the target closely.

Static mask vs Track Points on a window resized from its corner

Track Points on a round badge, a rounded card and a hexagon

The window-resize recording uses a Highlight mask so the tracked box is visible; pixelating a page of text-like dashes looks much like the dashes.

Fine text

At tracking resolution a terminal's text is a pixel or two tall, and a window dragged by a fraction of a pixel resamples it into a different pattern; the correct match scored below the "lost" threshold, so the window was lost as soon as it moved. When a sharp match fails, the tracker retries on lightly smoothed frames, where the same match scores well, and measures how the target grew or shrank on those frames too, so zooming text keeps its box growing. Smoothing is only a fallback because it also lets look-alikes match each other.

Margins

Tracked paths are simplified into keyframes, and simplification may move each mask edge up to the tolerance (0.3% of the frame); sensitive masks grow by that much so the error falls outside the content. Tracked masks also grow by 0.6% of the frame's width a side, the same number of pixels on both axes: for Track Points this covers clicks landing a few pixels inside the outline on a side with no straight outline to find, and for sensitive Track Motion masks the few pixels a box lags a fast zoom.

Picker and panel

The picker and the Tracking section use the canvas's selection language: hollow handles and label chips in the selection blue, the outline around the points with the mask's box drawn faintly, the dark instruction pill used elsewhere on the canvas, and the mask panel's card-style buttons. Tracking shows a progress bar, and a selected tracked mask shows its motion path.

Before After
Old picker Picker with four corners
Old panel Motion path and Tracking panel
Picker on a hexagon Picker on a rounded card

Generated scenarios

Synthetic 1280×720 clips with the target's true box known every frame, tracked in the built editor and scored on the saved mask (any visible target pixel outside the mask fails the frame):

Clip Mode Frames failed
Pan, feed scroll among look-alike cards, Cap zoom, brightness and grain, leaving and returning Track Motion 0
Window sliding over the card Track Motion 1 of 179, a 25 px² sliver
Window resized from its corner, panel stretched out of proportion Track Points, 4 0
Round badge growing as it moves Track Points, 4 and 8 0
Rounded card Track Points, 12 0
Hexagon through a zoom Track Points, 6 3 of 179, at most 301 px² at its tips

The hexagon is scored against its bounding box, which counts the empty corners around each tip as content. kernels/tests/mask_tracking_scenarios.rs generates these motions in code and scores each mask the way the editor saves it.

Tracking sees the editor's render rather than the source video, and the two differ in colour and sharpness enough to change results. Debug builds write the frames the tracker sees with CAP_GPUI_MASK_TRACK_DUMP=<file>, and the mask_track_replay kernels example replays them outside the editor.

Worth a close look: kernels/src/box_edges.rs (which sides qualify as edges, when an edge may stay put, and where a missed edge goes) and the estimate selection in kernels/src/point_track.rs, since they decide when the tracker overrides what the patches report.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; the previous findings are addressed and the latest change only removes a comment.

Summary

This PR adds motion and point tracking for editor masks, saves tracked paths as existing keyframes, and adds tracking controls and scenario tests.

  • Mask tracking follows boxes or clicked points through moving footage.
  • The editor saves tracked mask paths as editable keyframes.

Reviews (4) · Last reviewed commit: "Update apps/desktop-gpui/kernels/src/mot..." · Reviewed by Greptile

Expose existing mask keyframes through editable markers and local motion tracking. Four-point tracking combines corner patches with region matching to follow resizing content, with adjustable handles and an explicit start before writing a simplified path.

Keep the project format unchanged and share interpolation between preview, export and canvas edits. Hold lost targets, flag results for review, preserve partial results on Stop, and reject stale jobs after concurrent edits. Share frame pyramids across trackers without changing their saved paths.
Keep precise tracking tips visible with offset numbered droplets. Allow supplemental anchors after the four bounding corners, while retaining rectangular mask output and the existing four-corner path when it succeeds.

Show tracking changes only the editor preview; saved masks and export configuration remain intact.

Kernel tracking tests pass, including corner loss during resizing. Scoped editor tests pass (258 passed, two ignored), kernel Clippy passes, and native placement, dragging and visibility were exercised. Nine adversarial clips retain their four-point paths with supplemental anchors.
A tracked path is simplified before it becomes mask keyframes, and the
simplification lets each edge drift up to the tolerance away from the box
the tracker found. On a sensitive mask tracked tightly to its target that
drift uncovers a sliver of what it hides: in the editor, 101 of 916
frames across six moving-content clips leaked a few pixels at an edge.

Sensitive masks now grow by twice the tolerance when a tracked path is
applied, so the simplification error falls outside the content; those clips
then leak on 3 frames, all slivers of a card a window is sliding over.
Highlight masks keep the tracked size. Re-tracking a range starts from the
grown box, so repeated runs widen it slightly, which errs towards hiding
more.

A regression test checks that every tracked box stays inside the simplified
sensitive mask, and fails with the margin removed.
Track Points followed a rectangle's four corners with patches centred on
each corner, three quarters of which is the background behind the target.
When a panel moved or resized over a still page the patches stuck to the
page, and the whole-box fallback could only scale uniformly, so the mask
slid across a resizing panel instead of stretching (about 56% coverage).
Points around a round or irregular shape had no model at all.

Four or more clicked points now define the mask as the box around them.
Each point is followed by a patch on it and one inside the shape, and the
box is fitted per axis to the patches that agree, so it can widen without
growing taller; the whole-box tracker covers frames where too few agree,
and a fit is limited to 25% size change per frame. A side the user clicked
along (as with a rectangle's corners) is also kept on its straight edge
when its outline is one colour the page beside it does not share: a
resized window's border moves while its content and the page stay still,
which patch matching cannot see. Masks from points are widened by about
four pixels a side, since clicks land on or just inside the outline.

The picker and Tracking panel now use the canvas's selection language:
hollow handles and label chips in the selection blue, the outline around
the points with the mask's box drawn faintly, a dark instruction pill, the
mask panel's card-style buttons and a progress bar. A selected tracked
mask shows its motion path.

In the editor, Track Points now leaves no leaking frame on a window resized
from its corner, a stretched panel, a round badge with eight points and a
rounded card; a hexagon's tips still show slivers on 24 of 179 frames when
scored against its bounding box. Scenario tests in
kernels/tests/mask_tracking_scenarios.rs generate these motions and score
the masks as the editor saves them. Tracking sees the editor's render, not
the source video; debug builds can dump those frames with
CAP_GPUI_MASK_TRACK_DUMP for the mask_track_replay example.
@vercel

vercel Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

@thesammykins is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

Comment thread crates/project/src/mask_tracking.rs Outdated
Comment on lines +144 to +145
deviation(start_center, end_center, center, t)
.max(deviation(start_size, end_size, size, t) / 2.0),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security Saved masks expose an edge

simplify_path bounds center error and half-size error separately, but those errors add at an edge. With tolerance 0.003, let the endpoints have x-center 0.5 and width 0.2, and the middle sample have x-center 0.4976 and width 0.2048. Both measured errors are 0.0024, so the middle sample is dropped. The grown mask's left edge is then 0.397, while the tracked box reaches 0.3952, leaving sensitive content exposed.

Measure the error of all four edges directly, or grow the margin enough to cover both errors.

How this was verified: The simplifier permits both errors together, while the renderer places each edge at center plus or minus half the size.

Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/project/src/mask_tracking.rs
Line: 144-145

Comment:
**Saved masks expose an edge**

`simplify_path` bounds center error and half-size error separately, but those errors add at an edge. With tolerance `0.003`, let the endpoints have x-center `0.5` and width `0.2`, and the middle sample have x-center `0.4976` and width `0.2048`. Both measured errors are `0.0024`, so the middle sample is dropped. The grown mask's left edge is then `0.397`, while the tracked box reaches `0.3952`, leaving sensitive content exposed.

Measure the error of all four edges directly, or grow the margin enough to cover both errors.

**How this was verified:** The simplifier permits both errors together, while the renderer places each edge at center plus or minus half the size.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, thanks. Fixed in 82a66a0: the simplifier now measures how far each of the four mask edges strays from the line between kept samples, which is what the margin is sized against. A test with your case (centre 0.4976, width 0.2048 between samples at 0.5 and 0.2) keeps the sample and covers the tracked edge; it fails on the old measure.

Comment on lines +218 to +220
let (region, lost) = self
.region
.map_or_else(|| (Vec::new(), Vec::new()), TrackSession::finish);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Lost targets go unreported

PointTrackSession can start without a whole-box tracker, but finish then returns no lost spans regardless of unresolved. Points spanning a 4–7-pixel-wide target pass the point tracker's minimum size, while MotionTracker requires 8 pixels. The surrounding 16-pixel patches can still have enough detail to start tracking.

If those patches later lose the target, the saved mask holds without the promised review warning. Build lost spans from unresolved frames that remain unrecovered, including when region is None.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/kernels/src/point_track.rs
Line: 218-220

Comment:
**Lost targets go unreported**

`PointTrackSession` can start without a whole-box tracker, but `finish` then returns no lost spans regardless of `unresolved`. Points spanning a 4–7-pixel-wide target pass the point tracker's minimum size, while `MotionTracker` requires 8 pixels. The surrounding 16-pixel patches can still have enough detail to start tracking.

If those patches later lose the target, the saved mask holds without the promised review warning. Build lost spans from unresolved frames that remain unrecovered, including when `region` is `None`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. Fixed in e17e7d4: lost spans now come from the frames point tracking left unplaced, less any the whole-box tracker recovered, so they are reported whether or not that tracker exists. The corner test now checks that its hidden frames come back as one span for review.

Comment on lines +111 to +112
/// A tracked mask's path on the canvas while it is selected: a faint line
/// through its markers' centres, each marker a small handle-style dot.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Comments repeat the code

The new render_motion_path comment only describes what the function draws. AGENTS.md requires comments to capture non-obvious context and explicitly bans comments that narrate the code. Remove this comment or replace it with a reason the implementation needs explaining. The same pattern appears above tracking_label_centers and points_outline.

This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/editor_mask_tracking.rs
Line: 111-112

Comment:
**Comments repeat the code**

The new `render_motion_path` comment only describes what the function draws. `AGENTS.md` requires comments to capture non-obvious context and explicitly bans comments that narrate the code. Remove this comment or replace it with a reason the implementation needs explaining. The same pattern appears above `tracking_label_centers` and `points_outline`.

This repository requirement must be satisfied before merging.

**Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Fixed in a7ca064: the comments on render_motion_path, tracking_label_centers and points_outline now give the reasons (labels sit off their points so they never hide the pixel being aimed at; the hull is used because clicks around a shape come in any order) or are removed.

@thesammykins
thesammykins marked this pull request as draft November 3, 2026 04:47
… a pixel

On a real screen recording of a terminal window being dragged, Track Motion
lost the window as soon as it moved and leaked on 125 of 249 frames. At
tracking resolution the terminal's text is a pixel or two tall, and a shift
by a fraction of a pixel resamples it into a different pattern: the correct
match scored about 0.47 against the 0.5 lost threshold, while the window at
rest scored 0.999.

When the sharp match fails, the tracker now retries on lightly smoothed
frames and templates ([1, 2, 1] in each direction), where the same match
scores about 0.75, and accepts it from 0.65. Smoothing is only a fallback:
applied to every match it also let a feed of look-alike cards match each
other and broke the corner tests. Frames buffered while the target is lost
are kept as pyramids so the smoothed levels are built once. Track Points
now accepts any box the whole-box tracker reports for a frame, rather than
only matches above its patch threshold, which discarded these.

On the recording, Track Motion now leaves no leaking frame (0 of 249) and
Track Points 8. A test drags a window of one-pixel text strokes by
fractional pixels, and loses it with the fallback disabled. The
mask_track_replay example now takes the frame size, since the editor tracks
at 960 wide or less to keep a recording's aspect ratio.
When a remembered straight edge is missed for a frame, its side used to
stay where the edge was last seen. That holds a resized window's side in
place, which is the point, but pinned the side of a window dragged fast
enough to outrun the edge search, uncovering up to a quarter of it for a
few frames. A held side now keeps its correction to the estimate instead,
moving by as much as the estimate has since the edge was last found; on a
resize the estimate barely moves, so the side still holds.

Also replaces a closure passed to bool::then that clippy rejects.
Simplifying a tracked path bounded the centre's error and half the size's
error separately, but a mask edge sits at the centre plus or minus half the
size, so the two errors add there. A sample whose centre and width each
moved just under the tolerance could be dropped while its left edge moved
by nearly twice it, more than the sensitive mask's margin covers.

The simplifier now measures how far each of the four edges strays from the
line between the kept samples, which is what the margin is sized against.
A test with the case from review (centre 0.4976, width 0.2048 between two
samples at 0.5 and 0.2) keeps the sample and covers the tracked edge; it
fails on the old measure.
Point tracking reported lost spans only from its whole-box tracker, which
is not created for targets under eight pixels across. Frames neither the
points nor anything else could place then went unreported, and the editor
saved the held mask without asking the user to review it.

Lost spans now come from the frames point tracking left unplaced, less any
the whole-box tracker recovered afterwards, so they are reported either
way. The corner test checks that frames where its target is hidden come
back as one span for review.
Replace comments that restated what three picker helpers draw with the
reasons behind them: labels sit off their points so they never hide the
pixel being aimed at, and the outline is the convex hull because clicks
around a shape come in any order.
…g window

On a screen recording of a terminal window dragged around a desktop, Track
Points still uncovered part of the window on 9 of 249 frames, and more when
the corners were clicked a pixel or two inside the window. What went wrong
when the window moved fast:

- A straight edge seen near where it was last found was trusted even when
  the whole window had moved: dragged up by its title bar's height, the
  line under the title bar lands where the top edge was. An edge now stays
  put only while the target as a whole (both opposite sides of the
  estimate) moved less than an edge may drift; otherwise the qualifying
  line nearest where the estimate puts the edge wins, not the one most
  positions agree on.
- A point fit backed by three patches could come from two clicked points,
  each point having up to two patches, and overrule a whole-box track that
  was right. A fit now needs patches from three distinct points.
- Where estimates disagreed, one was picked and could be the wrong one.
  Sensitive masks now cover all of them when they broadly agree (overlap of
  at least half): the point fit, the whole-box track and the box snapped to
  straight edges. A slightly larger mask hides nothing it should show;
  highlight masks keep following the target closely. A missed edge takes
  whichever of its held position and the estimate lies further out.
- Clicks inside the outline left the mask short of it. The outline search
  now starts a few pixels outside the clicked side and the box starts on
  it; where no straight outline can be found (a side over a dark dock),
  tracked masks get a margin of 0.6% of the frame's width a side, in equal
  pixels on both axes. Sensitive Track Motion masks get the same margin,
  for the few pixels a box lags a fast zoom.

On that recording, scored against the window's position found at full
resolution, Track Motion and Track Points leave every frame fully covered,
with corner clicks from a pixel outside the outline to four inside it, and
no frame of the exported video keeps readable detail in the window or along
its edges. The generated scenarios are fully covered except one 25 px²
frame where a window slides over the card and three at a hexagon's tips.
@thesammykins

Copy link
Copy Markdown
Author

@greptile review

@thesammykins
thesammykins marked this pull request as ready for review November 3, 2026 07:37
Comment on lines +677 to +678
.filter(|found| found.score >= SMOOTHED_SCORE)?;
return Some(self.accept(pyramid, found, 1.0));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security Zooming text outgrows its mask

When fine text needs the new smoothed fallback during a zoom, follow accepts the match with scale 1.0 and returns before trying other sizes. The box stays the same size on those frames even as the target grows. Once that growth exceeds the fixed margin, a sensitive mask can expose content without a review warning, because these matches count as found.

Run the size search on the smoothed frames too before accepting the match.

How this was verified: The smoothed branch returns before the size search, and TrackSession saves successful matches without marking those frames as lost.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/kernels/src/motion_track.rs
Line: 677-678

Comment:
**Zooming text outgrows its mask**

When fine text needs the new smoothed fallback during a zoom, `follow` accepts the match with scale `1.0` and returns before trying other sizes. The box stays the same size on those frames even as the target grows. Once that growth exceeds the fixed margin, a sensitive mask can expose content without a review warning, because these matches count as found.

Run the size search on the smoothed frames too before accepting the match.

**How this was verified:** The smoothed branch returns before the size search, and `TrackSession` saves successful matches without marking those frames as lost.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, thanks. Fixed in efe81db: the size search now runs on whichever frames found the target, sharp or smoothed, with that path's own score floor. A new test zooms one-pixel text by 1.2% a frame while it drifts; the box now stays within 3.5 px of it, where the old code held its first size (160 px wide against 218) and lost the target.

When fine text was only found on the smoothed frames, the tracker accepted
that match at the box's current size and skipped the search over nearby
sizes. Text zooming while it moved kept its first size on those frames, so
a sensitive mask fell behind the growth until the margin no longer covered
it, and the frames still counted as found, so nothing asked for review.

The size search now runs on whichever frames found the target, sharp or
smoothed, with that path's own score floor. A test zooms one-pixel text by
1.2% a frame while it drifts: the box now stays within 3.5 pixels of it
(the 3.5% size steps), where before it held 160 pixels wide against 218
and lost the target. On the dragged-window recording both tracking modes
still leave every frame covered.
@thesammykins

Copy link
Copy Markdown
Author

@greptile review

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
@thesammykins

Copy link
Copy Markdown
Author

@greptile review

@richiemcilroy

Copy link
Copy Markdown
Member

Thanks! Can you rebase on main to clear the conflicts? Then we'll take a look.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants