Skip to content

improve: one-second keyframes for Studio screen recordings on macOS - #4430

Draft
richiemcilroy wants to merge 4 commits into
mainfrom
perf/studio-vt-keyframe-interval
Draft

richiemcilroy wants to merge 4 commits into
mainfrom
perf/studio-vt-keyframe-interval

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Draft, on hold: the size win costs editor seek latency, and that trade needs a product decision. Do not merge.

Studio screen recordings on macOS encode through h264_videotoolbox in realtime mode, and that branch of get_codec_and_options never sets g. VideoToolbox then gets FFmpeg's default GOP of 12 frames: five keyframes a second at 60 fps, which is where most of a file's bytes go. Every other encoder, and the VideoToolbox export path (#2502), already sets a keyframe interval.

This adds an opt-in keyframe interval that only realtime h264_videotoolbox reads, and turns it on (1 s) for the in-process macOS Studio screen recording at Balanced and Compatibility quality.

  • H264EncoderBuilder::with_realtime_videotoolbox_keyframe_interval, SegmentedVideoEncoderConfig::realtime_videotoolbox_keyframe_interval and MacOSFragmentedM4SMuxerConfig::realtime_videotoolbox_keyframe_interval, all None by default.
  • STUDIO_DISPLAY_VIDEOTOOLBOX_KEYFRAME_INTERVAL (1 s) is set by capture_pipeline.rs for Studio, except Ultra. With longer GOPs VideoToolbox reaches Ultra's much higher bitrate target on motion, so Ultra files would grow.
  • Unchanged: Ultra, Instant mode, the camera track, the out-of-process muxer, recovery and upload re-encodes, exports, and every encoder on Windows and Linux.
  • The crash suite's in-process arm now records with the Studio interval, so verify-crash covers it.

Size and quality

Same scripted scene (static UI, text scroll, typing, a video window) recorded with the cap CLI on an M4 Mac mini, scored against lossless captures of the scene.

Recording Mbps Keyframes Text PSNR static / scroll / typing
main, Balanced 10.6 to 10.8 every 0.2 s 41.7 / 51.4 / 62.7 to 63.8 dB
this PR, Balanced 8.7 every 1 s 41.9 / 51.4 / 64.0 dB

On the encoder alone (same lossless input through SegmentedVideoEncoder, paced in real time): Balanced 12.3 to 10.0 Mbps, Compatibility 7.2 to 5.3 Mbps. Scrolling is 2.3x smaller, typing 2.7x smaller, and video gains 1 dB at the same size. VMAF is equal, per-frame encode time is unchanged, and CPU during recording is unchanged.

Seek latency (why it is on hold)

decode-benchmark random access, same content:

Machine GOP 12 p50 / p95 1 s p50 / p95
M4 Mac mini 11 / 15 ms 25 / 33 to 52 ms
CI Windows (Media Foundation) 55 / 85 ms 147 / 240 ms
CI Linux (FFmpeg) 43 / 68 ms 113 / 181 ms

In scrub-benchmark on the M4, idle random seek p50 goes from 26 to 32 ms to 31 to 38 ms. A slow backward drag presents 50 to 62 frames/s instead of 91. Forward drags are unchanged.

Checks

  • Windows and Linux: recording output byte-identical to main, two runs each.
  • realtime_keyframe_interval_only_changes_realtime_videotoolbox: every other encoder's options are identical with or without the setting, for both recording and export.
  • realtime_videotoolbox_keyframe_interval_keeps_two_second_segments: same segment count, 1 s keyframes.
  • 1 s files read on every decoder: AVAssetReader and FFmpeg exports from main, 0.6.3 and 0.6.2 all complete with identical results; Media Foundation, FFmpeg and AVAssetReader decode all frames.
  • Engine verify (full), verify-recording and verify-crash pass.

@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Ignored Ignored Preview Nov 20, 2026 5:15pm UTC

This branch was successfully deployed

1 active deployment
Preview – cap-web — d56b93de Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant