fix(stt): preserve media-clock audio offsets

[satwiksharma01]

Summary

When video starts at 0s but its audio stream starts at 2s, native transcription extraction currently puts the first sound near 0s in the PCM sent to Whisper. Every word timestamp can then be two seconds early, affecting transcript-based cuts.

Decode through aresample=first_pts=0 to preserve the media-clock offset. Two real-FFmpeg regression tests generate tiny lossless media: delayed audio in a video retains two seconds of leading silence, while standalone audio with a nonzero container origin remains relative to its own start. The delayed-audio test fails on the original code.

FFmpeg also preserves interior timestamp gaps with this setting. A separate real-binary check confirmed that a 0.5s audio gap adds 8,000 silence samples at 16 kHz. This is media-clock preservation, not an aligner accuracy or performance change. The extraction API remains unchanged.

Related issue

Refs #948

This fixes one extraction clock defect; it does not claim to complete the broader word-timing benchmark or alignment work.

Type of change

• Bug fix

Release impact

• Patch

Desktop impact

• Not platform-specific

Screenshots / video

No UI changes. Behavioral tests assert actual decoded PCM timing.

Testing

• All 13 extraction tests pass, including the two new real-FFmpeg cases. Integration tests warn and skip explicitly if FFmpeg is unavailable on PATH.
• Both app and test TypeScript checks, full lint, docs and renderer/Electron production build pass. Lint reports 26 existing warnings.
• Full suite passes: 309 files, 4,307 tests passed and 5 existing skips. The two real-FFmpeg integration tests ran in this validation.
• Independent review found no actionable defects. An additional actual extraction check with AAC MP4 moved the first sound from 0.0640625s to 2.0000625s, matching its video clock; standalone-audio behavior remained unchanged.

Validated with pinned Node 22.22.1, npm 10.9.4 and real FFmpeg 9.0.1. AI assistance was used. Synthetic fixtures validate the source-clock behavior; no native hardware benchmark or full transcription quality result is claimed. FFmpeg's documented first_pts behavior.

Summary by CodeRabbit

• Bug Fixes
  • Audio extracted from video now retains its timing relative to the start of the media, including when the audio track begins after the video.
  • Standalone audio continues to start at the beginning of the extracted audio, with no change to its mono, 16 kHz output format. This provides more consistent synchronization when working with media that contains delayed audio tracks.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 5e8c1c8f-5f1c-4f10-91df-c9b009c1fe50

📥 Commits

Reviewing files that changed from the base of the PR and between 870196c and 324277b.

📒 Files selected for processing (2)

• electron/stt/extractAudio.integration.test.ts
• electron/stt/extractAudio.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

---

📝 Walkthrough

Walkthrough

extractMono16kPcm now uses FFmpeg to pad delayed audio and align sample zero with the media clock. Integration tests use generated fixtures to check delayed audio in video and standalone audio.

Changes

Audio timestamp alignment

Layer / File(s)	Summary
Align extracted audio and verify timestamps electron/stt/extractAudio.ts, electron/stt/extractAudio.integration.test.ts	The FFmpeg arguments apply aresample=first_pts=0. Integration tests check that delayed video audio has two seconds of silence before its first audible sample, while standalone audio starts near sample zero. Tests skip when FFmpeg is unavailable and remove temporary fixtures.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 32427

Transcription extraction now preserves the audio offset when a video's audio starts later than its video, so word timings stay aligned with the media clock. Standalone audio behavior is unchanged. No merge-blocking risk was found.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description check		The description follows the repository template. It explains the bug, implementation, related issue, change type, release impact, platform impact, and validation results.
Title check		The title clearly and concisely describes the main change: preserving media-clock audio offsets during STT extraction.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[EtienneLescot]

Reviewed: aligns the transcript with the export's raw pts; PCM identical for normal recordings. Thanks @satwiksharma01!