Skip to content

feat(desktop-gpui): hide sensitive data with an OCR scan of the recording - #4418

Open
AatmanAJ wants to merge 12 commits into
CapSoftware:mainfrom
AatmanAJ:capy/auto-hide-sensitive-data
Open

AatmanAJ wants to merge 12 commits into
CapSoftware:mainfrom
AatmanAJ:capy/auto-hide-sensitive-data

Conversation

@AatmanAJ

@AatmanAJ AatmanAJ commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds Hide sensitive data to the editor, in a new Privacy tab of the sidebar and in the Mask panel. It scans a finished recording for text that shouldn't be published and turns each find into a keyframed Sensitive mask that follows the text as it scrolls or zooms. It finds:

  • email addresses
  • phone numbers
  • card numbers that pass the Luhn check
  • API keys and tokens (known prefixes such as sk_live_, ghp_, AKIA and xoxb-, labelled values such as password:, and high-entropy strings)
  • any custom phrases the user adds, which are saved with the project (sensitivePhrases in the project config)

The masks are ordinary mask segments, so the existing renderer and exporter draw them. Users can move, resize, edit or delete them like any mask they drew by hand. Generated masks are clamped to the recording's duration and packed into the fewest mask lanes that keep them from overlapping, reusing free space in existing lanes.

Demo

Hide sensitive data: Cap finds it before you publish. Click to play the 8 s cut.

The film plays the real end-to-end exports. On a 26 s scrolling recording, the scan added 23 Sensitive masks, and a separate OCR read no secret in any of the 781 masked frames, or in the 421 frames of a 14 s fast-flick recording. Masks widen while the page scrolls and settle back on the value when it stops. The scan is shown sped up: it took 174 s in a debug build.

Full film (35 s) · 8 s cut · Square cut

How it works

The scan runs in a new crate, apps/desktop-gpui/sensitive-scan (cap-sensitive-scan), in four steps:

  1. Frames. The editor renders composited frames at 1920x1080 through render_frame_with_editor, so the scan sees what the export will show (crop, zoom, background, camera). Layers drawn above masks (captions, keyboard, image and shape overlays, annotations) are left out of the scan input, because a mask can't cover them. Text overlays stay in the scan input with their drawn text hidden, so fullscreen and split-layout text keeps its timing and layout, and the masks land at the right times.
  2. Change-driven OCR. Frames are sampled at 15 fps while the picture is still and at 30 fps while it changes, so a fast scroll can't alias onto repeating content between samples. A cheap tracker on a 960 px-wide greyscale copy follows already-found text between frames, using per-box matching plus a global motion estimate from a patch grid. Tracks snap to the global motion when they agree with it, and a match that disagrees with both the global motion and standing still is ignored. Full OCR runs when the content changes and then settles, after a cut (at most every 0.25 s when the page changes in place, as with a playing video), after half a screen of travel, or after at most 1 s of change. A small change on an otherwise still screen, below the tile threshold, is still read within 1 s. In the benchmark below that is about one OCR pass per second of video.
  3. Classification. OCR boxes on the same row are joined before classifying, so a label and its value (password: followed by hunter2!…) are seen together even when OCR splits them. The classifiers are plain Rust with unit tests, including negatives: versions, dates, build numbers, hashes in prose, order numbers and hyphenated slugs.
  4. Masks. Each track becomes one MaskSegment with keyframes.position and keyframes.size, using times relative to the segment start. The keyframes are simplified with a Ramer–Douglas–Peucker pass (0.15% of the frame) so long scrolls don't produce thousands of keys. Each keyframe also covers the neighbouring samples' positions, so a mask can't trail the text by a frame. A rescan leaves alone any found text that is already covered by an existing sensitive mask.

OCR engine

The scan uses PaddleOCR PP-OCRv4 (detection and recognition) through ONNX Runtime, via the ort crate with load-dynamic. It loads the same ONNX Runtime library that cap-camera-effects already ships.

  • The models are Apache-2.0. They aren't bundled. The first scan downloads them (about 15 MB) from a pinned Hugging Face commit into the app's data directory, checks their SHA-256 and size, and shows download progress in the panel.
  • I tried ocrs 0.14 with rten first, because a pure-Rust engine would be simpler. It reads short crops well, but on full 1080p pages it often drops the first characters of a line ("AKIA…" became "KIA…" and ghp_ was lost). Those prefixes are exactly what the classifier needs. With ocrs, the same benchmark gave 70.1% recall, a 90.8% leaked-frame rate and 318 s per minute of 1080p.

Results

Synthetic benchmark. cargo run --release -p cap-sensitive-scan --features synthetic --example scan-bench renders four 20 s, 30 fps 1080p scenes of a page full of fake secrets: light with smooth scrolling, dark with wheel scrolling, light with fast flicks, and light with zoom. The fast-flick scene throws the page 1100 px every 2.5 s with an ease-out that peaks near 275 px per frame, and wraps back to the top. It runs the scanner and compares the masks against the ground truth for every frame. Results on a 4-core AMD EPYC Linux VM:

Scenario Scan time Masks Frames with a visible secret not covered
light, smooth scroll 65.2 s 107 0 of 600
dark, wheel scroll 72.5 s 95 0 of 600
light, fast flick 78.4 s 79 0 of 600
light, zoom 36.8 s 27 0 of 600
Measure Result
Detection recall (fully visible secrets) 100.0% (321 of 321)
Detection precision 100.0% (322 of 322)
Frames with any secret leaked 0.00% (0 of 2400)
Masks on no secret 3 of 308
OCR passes 62 per minute of video
Scan speed 190 s per minute of 1080p (4 threads)

End to end on Linux. I recorded two scrolling admin pages with 14 fake secrets under Xvfb using the CLI: a 26 s recording with ordinary scrolling, and a 14 s recording with fast flicks, wheel bursts and a jump back to the top (both 30 fps). The covered values were 3 emails, 3 phone numbers, 2 cards, 4 keys, a labelled password and a custom phrase. I scanned each recording through the editor path, added the masks and exported with cap export. Then I ran an independent OCR pass (RapidOCR) over every frame of each masked export:

Recording Masked export: frames with a readable secret Unmasked control
Ordinary scrolling (26 s) 0 of 781 all 14 secrets readable, 157 of 157 sampled frames
Fast flicks (14 s) 0 of 421 all 14 secrets readable, 85 of 85 sampled frames

The scans produced 23 and 48 masks, packed into 10 and 12 lanes (the number of secrets on screen at once). They took 174 s and 231 s in a debug build; I didn't time a release build of the app.

On macOS, after the review fixes. The runs were on Apple silicon with 18 threads. The synthetic benchmark (release build) leaked 0 of 2400 frames, with recall 321 of 321, precision 322 of 322, 3 of 308 masks on no secret, 62 OCR passes per minute of video and 44 s per minute of 1080p. The end-to-end projects use the Linux recordings above as display tracks. Ordinary scrolling leaked 0 of 781 frames (23 masks), and fast flicks 0 of 421 (51 masks). A third project adds a 2 s fullscreen title at 3 s to the ordinary-scrolling recording. It leaked secrets in 526 of 841 frames before the text-hold fix and in 0 of 841 after (28 masks).

Over-coverage. On the 26 s recording, sampling every third frame, 137 of 261 frames have a non-secret text line at least 20% covered by a mask. Masks are tight on a still page, where the padding touches the row below, and grow taller during a scroll in proportion to its speed, because each keyframe covers the positions one sample either side.

Relationship to #3674

This builds on the mask format on main and doesn't depend on #3674. The keyframes it writes use the same model that #3674's tracking edits: relative-time position/size vectors with linear interpolation. So generated masks should show up as tracked masks there, with markers. The render_frame_with_editor change in crates/export/src/preview.rs is the same function #3674 adds. Whichever PR merges second just drops that hunk.

Risks and limits

  • Untested on Windows. The scan, export and benchmark ran on Linux x86_64 and on macOS (Apple silicon). Loading ONNX Runtime on Windows should follow the same path cap-camera-effects uses, but I haven't run the scan there.
  • Review before publishing. No frame leaked in the benchmark or in either recording, but OCR can still miss text, so users should check the result.
  • Masks widen during scrolls. A mask covers neighbouring text while the page moves fast, and returns to the value's own box when it stops.
  • Overlays aren't scanned. A secret typed into a text overlay, caption or annotation isn't found, because those layers render above masks.
  • Many lanes. A screen with ten secrets needs ten mask lanes. The timeline keeps its height after a scan and the lanes scroll inside it; I checked that in the running app on Linux under Xvfb.
  • Video regions. Text that appears inside a playing video or animation can be visible for up to 0.25 s before it is read.
  • Speed. Scan time is about 3x the video length on 4 cores, and OCR is CPU-only. GPU execution providers are a possible follow-up.
  • Not included: blur during recording, and faces.

Testing

Tested on macOS 27.0.1 (Apple M5 Max): cargo test passes for cap-sensitive-scan (29), cap-project (359), cap-export (87) and cap-editor (273), and the cap-gpui bin tests pass (2158 passed, run on stable 1.99.1 because the 1.95.0 proc-macro dylibs don't load on macOS 27). cap-gpui builds in release. cargo clippy -p cap-sensitive-scan --all-targets --features synthetic -- -D warnings is clean. For cap-project, cap-export, cap-editor and the app crate, clippy adds no errors; the ones it reports there are identical on main. In a release build, scanning the 26 s end-to-end recording took 15.6 s on the M5 Max (mean of 2 runs with the models already downloaded) and found the same 23 tracks.

  • cargo test -p cap-sensitive-scan --features synthetic (29 tests: classifiers with negatives, tracking, motion sampling, mask generation, lane packing and clamping, and small text changes read within the maximum OCR interval and at the end of a scan)
  • cargo clippy -p cap-sensitive-scan --all-targets --features synthetic -- -D warnings
  • cargo check -p cap-project -p cap-editor -p cap-export on the root toolchain (1.88), and a round-trip test for sensitivePhrases
  • GPUI tests: typing in the custom-phrase field redraws the sidebar, phrases are trimmed and de-duplicated, and the scan input drops the layers drawn above masks
  • GPUI i18n surface tests for the new strings in all 7 locales
  • GPUI tests for the review fixes:
    • the scan keeps the timing of fullscreen text holds
    • the scan reads the screen without the layers drawn above masks (updated)
    • a phrase added during a scan makes its result stale
    • first scans that download a model together both install it, with two overlapping downloads from a local HTTP server
    • a download dropped part way leaves no staging file

RetriggerView in GreptileConfidence Score: 5/5

The changes appear safe to merge; no blocking issue remains.

Summary

Adds OCR-based scanning that turns sensitive text into tracked masks, with custom phrases saved in the project.

  • The scan finds sensitive text and tracks it as the recording moves.
  • Findings become editable masks that follow sensitive text.
  • A Privacy tab lets people scan recordings and keep custom phrases with each project.

Reviews (6) · Last reviewed commit: "fix: remove a model's staging file when ..." · Reviewed by Greptile

@vercel

vercel Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@AatmanAJ is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant