Skip to content

feat(tools): added speech to text with openai whisper, elevenlabs, and deepgram - #2068

Merged
waleedlatif1 merged 4 commits into
stagingfrom
feat/speech-to-text
Nov 20, 2025
Merged

feat(tools): added speech to text with openai whisper, elevenlabs, and deepgram#2068
waleedlatif1 merged 4 commits into
stagingfrom
feat/speech-to-text

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • added speech to text with openai whisper, elevenlabs, and deepgram

Type of Change

  • Bug fix

Testing

Tested manually

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Nov 20, 2025

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Preview Comments Updated (UTC)
docs Skipped Skipped Nov 20, 2025 4:41am

@waleedlatif1
waleedlatif1 marked this pull request as ready for review November 20, 2025 04:58
@greptile-apps

greptile-apps Bot commented Nov 20, 2025

Copy link
Copy Markdown
Contributor

Greptile Summary

  • Implemented speech-to-text functionality with three AI providers (OpenAI Whisper, Deepgram, ElevenLabs) supporting audio/video transcription with timestamps and speaker diarization
  • Added ffmpeg integration for video-to-audio extraction with proper Docker and Next.js configuration
  • Extended file upload validation and MIME type handling to support audio formats (mp3, wav, m4a, etc.) and video formats (mp4, mov, avi, mkv)

Confidence Score: 3/5

  • This PR has a critical concurrency bug that could cause file conflicts in production under load.
  • The temporary file naming in extractor.ts uses only Date.now() which has millisecond precision, creating a race condition where concurrent requests could collide and overwrite each other's files. This will cause transcription failures and potential data corruption under moderate to high traffic.
  • Pay close attention to apps/sim/lib/audio/extractor.ts - the temp file collision bug must be fixed before merging.

Important Files Changed

Filename Overview
apps/sim/lib/audio/extractor.ts Audio/video extraction using ffmpeg with temp file name collision bug in concurrent requests
apps/sim/app/api/proxy/stt/route.ts API route handling STT transcription with proper auth, file handling, and provider integration

Sequence Diagram

sequenceDiagram
    participant User
    participant "STT Block" as Block
    participant "API Route" as API
    participant "Audio Extractor" as Extractor
    participant "STT Provider" as Provider
    
    User->>Block: "Upload audio/video file"
    Block->>API: "POST /api/proxy/stt with file"
    API->>API: "Authenticate request"
    API->>API: "Download file from storage"
    alt Video File
        API->>Extractor: "Extract audio from video"
        Extractor->>Extractor: "Create temp files with Date.now()"
        Extractor->>Extractor: "Convert using ffmpeg"
        Extractor->>API: "Return audio buffer"
    end
    API->>Provider: "Send audio for transcription"
    Provider->>API: "Return transcript with segments"
    API->>Block: "Return transcript data"
    Block->>User: "Display transcript"
Loading

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

26 files reviewed, 3 comments

Edit Code Review Agent Settings | Greptile
React with 👍 or 👎 to share your feedback on this new summary format

Comment thread apps/sim/lib/audio/extractor.ts
Comment thread apps/sim/lib/audio/extractor.ts
Comment thread apps/sim/lib/audio/extractor.ts
@waleedlatif1
waleedlatif1 merged commit e64b1c9 into staging Nov 20, 2025
9 checks passed
@waleedlatif1
waleedlatif1 deleted the feat/speech-to-text branch November 20, 2025 05:03
@waleedlatif1 waleedlatif1 mentioned this pull request Nov 20, 2025
10 tasks
waleedlatif1 added a commit that referenced this pull request Aug 7, 2026
Three zero-reference exports, found by walking every export in the modules the
merge touched and counting non-test references.

- `resolveMediaMimeType` + `MEDIA_FALLBACK_MIME` (`lib/uploads/utils/file-utils`).
  Staging added them in #6341 for `MediaPreview`'s blob path; this branch rewrote
  that preview to stream from the serve route, so the merge orphaned them. Their
  tests go too, and the `DUAL_CONTAINER_MIME` doc stops pointing at a function
  that no longer exists.

  Worth recording, since deleting the helper deletes the fix: it retagged a
  dual-container `.webm` to the element the viewer had already chosen. The serve
  route now declares the type instead, and derives it from the filename, where
  `webm` maps to `video/webm` — so an audio-only `.webm` reaches an `<audio>`
  element labelled `video/webm`. Browsers sniff `src=` responses rather than
  trusting the header, so this is inert in practice, but it is a real narrowing
  and the public-share route (which uses the stored `file.contentType`) does not
  share it.

- `tableWorkspaceId` (`resources/table-source.ts`, the whole file). Added by this
  branch's own first commit and never called — `cell-render` takes the workspace
  id as a plain argument. Its TSDoc claimed to be "the value that decides whether
  a cell may render a sim-resource chip", which nothing enforced. The table
  migration will want this helper; it can arrive with a caller and an accurate
  docstring.

- `isMediaFileType` (`lib/uploads/utils/file-utils`). Not ours — dead since #2068,
  zero references including tests. Removed while the file was open.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant