Skip to content

perf(enc-ffmpeg): read capture-rate timestamps from the MP4 index - #4416

Open
AatmanAJ wants to merge 4 commits into
CapSoftware:mainfrom
AatmanAJ:capy/editor-open-index-capture-fps
Open

AatmanAJ wants to merge 4 commits into
CapSoftware:mainfrom
AatmanAJ:capy/editor-open-index-capture-fps

Conversation

@AatmanAJ

@AatmanAJ AatmanAJ commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Demo

Long recordings open sooner, without reading the whole file first. Click to play the 8 s cut.

Opening a long constant-rate recording in the editor no longer reads its whole display track. On a 10-minute 1080p30 recording the editor reads 8.8 MB instead of 1,191 MB while it opens, and with a cold disk cache the first frame shows 2.8x sooner. With a warm cache it's 1.5x sooner, and a 5-second recording opens in the same time as before.

Full film (24 s) · 8 s cut · Square cut

Why it was slow

Every editor open runs editor_window::preflight, which runs track_heal::heal_stretched_tracks. Its first step, heal_average_display_fps, checks whether a recording's stored display fps is the track's average rate (an older finalization bug). When it is, it calls get_video_capture_fps, which demuxed up to 20,000 video packets to measure the capture rate from their timestamps.

A healthy constant-rate recording (video, games, steady scrolling) has a stored fps equal to its average and a capture rate equal to both. Nothing gets rewritten, so the check repeats on every open. At 1080p30 that means reading the first 11 minutes of the file; at 60 fps, the first 5.5 minutes.

What changed

  • get_video_capture_fps reads the timestamps from the container's sample index (avformat_index_get_entry), which the MP4 demuxer builds from the moov box without touching sample data. The timestamps, the 20,000-sample cap and capture_fps_from_timestamps are unchanged, so the result is the same.
  • It falls back to the old packet scan when the index can't stand in for it: the track has any non-zero composition offset (the index holds decode times, the scan used presentation times), the file is fragmented or isn't MP4, the entry count doesn't match the track's frame count, the stream reorders frames (video_delay > 0), or an entry is marked discard by an edit list.
  • The composition-offset check is the existing track_has_no_composition_offsets from crates/video-decode/src/avassetreader.rs. It moves, with its box-parsing helpers and unchanged, to cap_enc_ffmpeg::mp4_boxes so both crates share one copy. avassetreader.rs is macOS-only and now imports it.
  • New crates/recording/examples/track-heal-benchmark.rs times heal_stretched_tracks on a project and reports bytes read from /proc/self/io.

Results

Linux x86_64 VM, 4 vCPU, release builds, mean of 3 alternating runs. Cold runs drop the page cache first (echo 3 > /proc/sys/vm/drop_caches). The 10-minute project is 1080p30 testsrc2 with libx264 (1.18 GB display track, 18,000 frames) plus an Opus mic track. The 5-second project is the same recipe.

Editor open, measured in the GPUI app under Xvfb from the editor window opened log line, with whole-process rchar at first frame:

Metric Before After Delta
10-min, cold: window to first frame 1,426 ms 518 ms 2.8x faster
10-min, cold: window to project validated 1,041 ms 134 ms 7.8x faster
10-min, warm: window to first frame 724 ms 472 ms 1.5x faster
10-min, warm: window to project validated 331 ms 126 ms 2.6x faster
10-min: bytes read by the app during open 1,191 MB 8.8 MB 135x less
5-s, cold: window to first frame 432 ms 464 ms within run-to-run spread (419 to 478 ms)
5-s, warm: window to first frame 500 ms 407 ms within run-to-run spread (388 to 551 ms)
5-s: bytes read by the app during open 17.2 MB 7.5 MB 2.3x less

The heal check alone (track-heal-benchmark):

Project Cache Before After Bytes read before → after
10-min cold 1,934 ms (745 to 4,240) 77 ms 1,182,067,604 → 848,519
10-min warm 172 ms 21 ms same
5-s cold 81 ms 66 ms 10,077,673 → 472,784
5-s warm 21.6 ms 18.8 ms same
cargo build --release -p cap-recording --example track-heal-benchmark
target/release/examples/track-heal-benchmark <project.cap> 3

Raw output:

rep=1 before-long cold validated_ms=923 instance_ms=1110 first_frame_ms=1260 process_read_mb=1190.6
rep=1 after-long cold validated_ms=146 instance_ms=351 first_frame_ms=545 process_read_mb=9.1
rep=1 before-short cold validated_ms=112 instance_ms=296 first_frame_ms=452 process_read_mb=17.5
rep=1 after-short cold validated_ms=130 instance_ms=328 first_frame_ms=478 process_read_mb=7.5
rep=1 before-long warm validated_ms=349 instance_ms=571 first_frame_ms=774 process_read_mb=1191.2
rep=1 after-long warm validated_ms=127 instance_ms=354 first_frame_ms=466 process_read_mb=9.4
rep=1 before-short warm validated_ms=119 instance_ms=251 first_frame_ms=403 process_read_mb=17.1
rep=1 after-short warm validated_ms=112 instance_ms=267 first_frame_ms=388 process_read_mb=7.2
rep=2 before-long cold validated_ms=1086 instance_ms=1343 first_frame_ms=1508 process_read_mb=1190.7
rep=2 after-long cold validated_ms=106 instance_ms=323 first_frame_ms=497 process_read_mb=8.8
rep=2 before-short cold validated_ms=111 instance_ms=298 first_frame_ms=424 process_read_mb=17.4
rep=2 after-short cold validated_ms=112 instance_ms=311 first_frame_ms=444 process_read_mb=7.5
rep=2 before-long warm validated_ms=335 instance_ms=533 first_frame_ms=695 process_read_mb=1190.1
rep=2 after-long warm validated_ms=117 instance_ms=318 first_frame_ms=473 process_read_mb=9.4
rep=2 before-short warm validated_ms=157 instance_ms=409 first_frame_ms=546 process_read_mb=16.8
rep=2 after-short warm validated_ms=112 instance_ms=271 first_frame_ms=394 process_read_mb=7.5
rep=3 before-long cold validated_ms=1114 instance_ms=1311 first_frame_ms=1511 process_read_mb=1189.9
rep=3 after-long cold validated_ms=151 instance_ms=404 first_frame_ms=512 process_read_mb=9.2
rep=3 before-short cold validated_ms=119 instance_ms=296 first_frame_ms=419 process_read_mb=17.2
rep=3 after-short cold validated_ms=126 instance_ms=316 first_frame_ms=471 process_read_mb=7.8
rep=3 before-long warm validated_ms=310 instance_ms=530 first_frame_ms=704 process_read_mb=1190.4
rep=3 after-long warm validated_ms=133 instance_ms=373 first_frame_ms=476 process_read_mb=7.0
rep=3 before-short warm validated_ms=106 instance_ms=323 first_frame_ms=551 process_read_mb=17.5
rep=3 after-short warm validated_ms=100 instance_ms=300 first_frame_ms=440 process_read_mb=7.5
before lib1/Long.cap cold rep=1 {"bytesRead":1182067604,"elapsedMs":4239.732213,"healed":false,"run":0}
before lib1/Long.cap warm rep=1 {"bytesRead":1182067608,"elapsedMs":168.506102,"healed":false,"run":2}
after lib1/Long.cap cold rep=1 {"bytesRead":848519,"elapsedMs":75.49356999999999,"healed":false,"run":0}
after lib1/Long.cap warm rep=1 {"bytesRead":848518,"elapsedMs":19.314369000000003,"healed":false,"run":2}
before lib600/Rec-001.cap cold rep=1 {"bytesRead":10077673,"elapsedMs":109.39856,"healed":false,"run":0}
before lib600/Rec-001.cap warm rep=1 {"bytesRead":10077673,"elapsedMs":21.423826,"healed":false,"run":2}
after lib600/Rec-001.cap cold rep=1 {"bytesRead":472784,"elapsedMs":66.168983,"healed":false,"run":0}
after lib600/Rec-001.cap warm rep=1 {"bytesRead":472781,"elapsedMs":18.422956,"healed":false,"run":2}
before lib1/Long.cap cold rep=2 {"bytesRead":1182067604,"elapsedMs":816.013196,"healed":false,"run":0}
before lib1/Long.cap warm rep=2 {"bytesRead":1182067608,"elapsedMs":169.779369,"healed":false,"run":2}
after lib1/Long.cap cold rep=2 {"bytesRead":848519,"elapsedMs":78.262294,"healed":false,"run":0}
after lib1/Long.cap warm rep=2 {"bytesRead":848518,"elapsedMs":19.404647,"healed":false,"run":2}
before lib600/Rec-001.cap cold rep=2 {"bytesRead":10077673,"elapsedMs":75.899591,"healed":false,"run":0}
before lib600/Rec-001.cap warm rep=2 {"bytesRead":10077673,"elapsedMs":20.829879,"healed":false,"run":2}
after lib600/Rec-001.cap cold rep=2 {"bytesRead":472784,"elapsedMs":70.29159,"healed":false,"run":0}
after lib600/Rec-001.cap warm rep=2 {"bytesRead":472781,"elapsedMs":18.605526,"healed":false,"run":2}
before lib1/Long.cap cold rep=3 {"bytesRead":1182067604,"elapsedMs":744.9825450000001,"healed":false,"run":0}
before lib1/Long.cap warm rep=3 {"bytesRead":1182067608,"elapsedMs":176.142994,"healed":false,"run":2}
after lib1/Long.cap cold rep=3 {"bytesRead":848519,"elapsedMs":76.02135799999999,"healed":false,"run":0}
after lib1/Long.cap warm rep=3 {"bytesRead":848518,"elapsedMs":23.166151,"healed":false,"run":2}
before lib600/Rec-001.cap cold rep=3 {"bytesRead":10077673,"elapsedMs":57.836594,"healed":false,"run":0}
before lib600/Rec-001.cap warm rep=3 {"bytesRead":10077673,"elapsedMs":22.428354000000002,"healed":false,"run":2}
after lib600/Rec-001.cap cold rep=3 {"bytesRead":472784,"elapsedMs":62.612863000000004,"healed":false,"run":0}
after lib600/Rec-001.cap warm rep=3 {"bytesRead":472781,"elapsedMs":19.230311,"healed":false,"run":2}

Tests

Tested on macOS 27.0.1 (Apple M5 Max, rustc 1.88.0): cargo check --all-targets and cargo clippy --all-targets -- -D warnings pass for cap-enc-ffmpeg and cap-video-decode, so avassetreader.rs compiles with the moved helpers. cargo test passes for cap-enc-ffmpeg (268), cap-video-decode (24, including the avassetreader composition-offset and reordering tests), the cap-recording lib (852) and oop_muxer (13). With --all-targets, cap-recording's clippy hits 9 errors that are identical on main and sit in files this PR doesn't touch (cursor.rs, recovery/needs_remux_tests.rs), and the hardware_* suites need Screen Recording permission, as they do on main. cap-gpui builds in release. On a 10-minute 1080p30 project with a warm cache (mean of 5 alternating launches), the editor's first frame drops from 291 to 199 ms after the window opens, and project validation drops from 185 to 82 ms. The heal check alone drops from 94.5 to 7.1 ms. A 5-second project stays within run-to-run spread (206 vs 194 ms to first frame). Cold-cache runs and bytes read weren't measured on macOS, because there's no /proc and no root to purge the cache.

  • indexed_timestamps_match_a_full_packet_scan: on constant-rate, bursty variable-rate, jittered and paused tracks written by Cap's own MP4File + H264Encoder, the index gives exactly the timestamps a full packet scan does.
  • varying_composition_offsets_fall_back_to_reading_packets: on the repo's h264-varying-composition-offset.mp4 fixture the index is refused and the capture rate equals the one computed from a packet scan; the h264-no-reordering.mp4 fixture still uses the index and matches the scan.
  • a_fragmented_track_falls_back_to_reading_packets: an init segment plus fragments has no usable index; the scan still runs and returns 30 fps.
  • a_jittered_variable_rate_display_at_its_average_is_healed and a_display_fps_that_differs_from_its_track_is_left_alone join the existing heal tests. All of them pass before and after, so heal decisions are unchanged.
  • cargo test -p cap-enc-ffmpeg: 265 passed. cargo test -p cap-video-decode: 17 passed on Linux. cargo test -p cap-recording --lib --tests: 989 lib tests plus every integration suite pass (oop_muxer needs cargo build -p cap-muxer first).
  • cargo clippy --all-targets -- -D warnings is clean for cap-enc-ffmpeg and cap-video-decode. cargo clippy -p cap-recording --no-deps reports only errors that already exist on main on Linux, in files this PR doesn't touch (screen_capture/linux.rs, cursor.rs, recovery/needs_remux_tests.rs), plus two in cap-utils. cargo fmt --all is clean.

Risks and what wasn't tested

  • The index timestamps are decode timestamps. The index is used only when every composition offset in the track is zero, where they equal presentation timestamps; every other track takes the old path.
  • avassetreader.rs only compiles on macOS and wasn't compiled here. Its change is one import replacing the moved functions, and its two existing tests for track_has_no_composition_offsets stay with it.
  • macOS and Windows weren't run. The change is FFmpeg demuxer behavior and platform-independent, but real recordings from VideoToolbox and Media Foundation weren't checked; all fixtures here are synthetic.
  • Recordings that don't trigger this check (most screen recordings, whose average rate sits below the stored fps) open exactly as before.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no blocking issues were found.

Summary

This PR speeds up capture-rate checks by reading safe MP4 sample indexes instead of scanning video packets.

  • Eligible MP4 capture-rate checks now read timestamps from the sample index.

No actionable issues were found. Tests were not run during this review.

Reviews (2) · Last reviewed commit: "Merge branch 'main' into capy/editor-ope..." · Reviewed by Greptile

get_video_capture_fps demuxed up to 20,000 video packets, so opening a constant-rate recording in the editor read the whole display track on every open. It now reads the timestamps from the container sample index, and falls back to the packet scan when the index does not cover the track (fragmented files), the stream reorders frames, or an entry is marked discard.
…d tracks

Adds a variable-rate fixture that is healed to its capture rate and a constant-rate fixture whose stored fps differs from its track, plus a track-heal-benchmark example that reports the time and bytes read per heal check.
…osition offsets

The sample index holds decode times while the packet scan used presentation times, so a track with varying composition offsets could measure a different capture rate. The index is now used only when track_has_no_composition_offsets holds; that check and its box helpers move unchanged from cap-video-decode to cap_enc_ffmpeg::mp4_boxes so both crates share them.
@vercel

vercel Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@AatmanAJ is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

@richiemcilroy

Copy link
Copy Markdown
Member

@greptileai review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants