Skip to content

fix(desktop): preserve huddle speech boundaries - #6397

Merged
tlongwell-block merged 4 commits into
mainfrom
wren/vad-endpointing
Aug 20, 2026
Merged

fix(desktop): preserve huddle speech boundaries#6397
tlongwell-block merged 4 commits into
mainfrom
wren/vad-endpointing

Conversation

@tlongwell-block

Copy link
Copy Markdown
Collaborator

Problem and intent

Buzz huddles have been clipping speech into fragments: one live turn arrived as the single letter “M,” and another ended mid-sentence. The detector was making each frame decision independently, but the surrounding endpointing policy had no onset confirmation or pre-roll, used one threshold for both entering and leaving speech, and silently discarded short segments.

This PR keeps Earshot 1.1.0 and fixes that policy around it. It is intentionally model-independent so the separate Earshot/Silero bake-off can evaluate detectors under the same segmentation behavior.

What changed

One production file changed: desktop/src-tauri/src/huddle/stt.rs (+367/−79 versus main).

  • Add a pure VadEndpoint state machine around Earshot probabilities.
  • Preserve 256 ms (16 frames) of pre-roll before confirmed onset.
  • Require three consecutive frames above 0.50 to enter speech.
  • Use 0.35 to leave speech, preserving hysteresis-band audio.
  • Retain 96 ms (6 frames) of hangover while keeping the existing 304 ms silence-flush window.
  • Make short-segment drops visible in logs instead of silent.
  • Add boundary, onset, hysteresis, hangover, drop-path, and PTT policy tests.
  • Bind the thresholds to Earshot 1.1.0 in source. Earshot 1.2.x is deliberately parked pending a matched-policy bake-off; chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick #6392 separately prevents Renovate from silently crossing that boundary.

Boundary behavior and known tradeoffs

A hard message boundary clears pre-roll. That prevents segment N audio from reaching segment N+1, but a fast follow-up turn may receive less than the full 256 ms onset window.

On the 121-clip corpus, 12 of 25 non-first segments had truncated pre-roll, with 32 ms worst observed at the shipping constants. This result holds at both 208 ms and 256 ms on this corpus; it is not independent of pre-roll. The observed follow-ups were bimodal—fast cases clustered at 2–10 frames and the next case was 19—so no larger pre-roll budget could reach those fast cases in this corpus.

If a future refactor makes pre-roll survive a boundary, segment N reaches segment N+1 under the strict predicate:

gap < VAD_PRE_ROLL_FRAMES - VAD_ONSET_FRAMES

At the shipped values that means gaps 0–12 leak and gap 13 is the first clean case. Hangover and the silence-flush window do not enter this bound because leakage rides the pre-roll deque. Marker-origin tests exercise the real flush/reset and next-onset drain; a rolling-buffer mutant leaks 3,328 samples and fails them.

Other known behavior:

  • voiced_frames now counts frames above the 0.35 exit threshold once speech begins, not only frames above 0.50. Corpus drop counts measured under > 0.50 counting must be re-baselined rather than compared directly; the old “19 silent drops” figure is not a valid before/after baseline.
  • If PTT releases while the manually-open microphone flips or remains on, there is no combined-transmit falling edge. The uninterrupted utterance correctly closes on normal VAD timing instead of an edge flush.
  • The GUI-to-push_audio_pcm wiring is unchanged. The final acoustic gate exercises the checked-in release-profile latency harness against the real production SttPipeline and TtsPipeline; Tyler’s hand-test covers the live GUI leg by design.

Verification ledger

All gates attest exact head 2dad6bb0e2e2a0af115a9a06c26cf6183fedb633.

  • Implementation: complete pre-push gate green; remote SHA matched local HEAD.
  • Exact-SHA verification: cargo fmt --check; cargo clippy --all-targets -- -D warnings; full desktop/src-tauri tests: 2,702 passed, 0 failed, 18 ignored; HEAD re-confirmed unchanged afterward.
  • Mutation review: engine region SHA-256 3c161d8f6e4d22c02dff3c76498430964beda51a034d34651dbdb03e8ff69aa0; 5/5 policy mutants killed, including deleted-clear and rolling-buffer regressions. Direct gap sweep leaked at 0–12 and was clean from 13.
  • Release acoustic gate: negative control deliberately overstated the expected segment count and failed with exit 101. Soft/short onset produced 1/1 segment (“I'm happy.”). Natural-pause fixture used two exact 700 ms pauses (>304 ms) and produced the pre-registered 3/3 intact segments at the scripted boundaries.
  • Release CPU beside Pocket TTS: soft fixture median 32.1%, p95/max 56.8% (5 sparse samples); natural fixture median 14.55%, p95/max 17.2% (8 sparse samples). These are scoped process samples from synthesis start through append acceptance, not a general desktop CPU benchmark.

Local review receipts (not committed to the repository):

  • .scratch/vad-live-release-2dad6bb0e/
  • RESEARCH/VAD_ARM_B_MUTATION_RIG_2026_08_20/
  • RESEARCH/EARSHOT_1_1_0_TO_1_2_2_MEASUREMENT_2026_08_20.md

Hand-test focus

Before merge, exercise the two live shapes that originally failed:

  1. Soft short openers such as “M” and “yes.”
  2. Natural mid-sentence pauses.

Also try a reply immediately after the previous message commits; that is intentionally the least-protected onset case because hard boundaries clear pre-roll.


Authorship disclosure: this change was implemented and the PR opened by Wren, Tyler’s Buzz agent, using Tyler’s GitHub identity after prior disclosure and authorization in the originating Buzz thread. Tyler remains the accountable human reviewer/operator.

Wren added 2 commits August 20, 2026 10:37
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner August 20, 2026 15:47
Wren added 2 commits August 20, 2026 13:03
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
@tlongwell-block
tlongwell-block merged commit 7ebe3ea into main Aug 20, 2026
24 checks passed
@tlongwell-block
tlongwell-block deleted the wren/vad-endpointing branch August 20, 2026 18:06
wpfleger96 added a commit that referenced this pull request Aug 20, 2026
…-scoping

* origin/main:
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  fix(acp): guard against unrequested public relay skills (#6394)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
atishpatel added a commit that referenced this pull request Aug 20, 2026
…ounting-and-solo

* origin/main: (48 commits)
  fix(hooks): scope pre-push lanes to branch merge-base diff (#6423)
  Enforce a three-day dependency cooldown (#6426)
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  fix(acp): guard against unrequested public relay skills (#6394)
  feat(desktop): refine context-aware Projects collaboration (#6396)
  fix(desktop): distinguish duplicate agent devices (#6337)
  feat(desktop): close Buzz window with Cmd+W (#6314)
  refactor(prompt): simplify Buzz agent guidance (#6340)
  feat(desktop): make Projects workspaces selectable (#6368)
  Add Buzz-native collaboration benchmarks (#6264)
  Polish mobile timeline and emoji interactions (#6297)
  feat(desktop): make the Projects overview follow the selected section (#6335)
  refactor(desktop): coordinate TTS playback (#6341)
  ...

Signed-off-by: Atish Patel <atishpatel2012@gmail.com>
brow added a commit that referenced this pull request Aug 20, 2026
…ifications-pr

* origin/main: (33 commits)
  fix(hooks): scope pre-push lanes to branch merge-base diff (#6423)
  Enforce a three-day dependency cooldown (#6426)
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  fix(acp): guard against unrequested public relay skills (#6394)
  feat(desktop): refine context-aware Projects collaboration (#6396)
  fix(desktop): distinguish duplicate agent devices (#6337)
  feat(desktop): close Buzz window with Cmd+W (#6314)
  refactor(prompt): simplify Buzz agent guidance (#6340)
  feat(desktop): make Projects workspaces selectable (#6368)
  Add Buzz-native collaboration benchmarks (#6264)
  Polish mobile timeline and emoji interactions (#6297)
  feat(desktop): make the Projects overview follow the selected section (#6335)
  refactor(desktop): coordinate TTS playback (#6341)
  ...

Signed-off-by: Tom Brow <tomb@block.xyz>
TheSentinel454 added a commit that referenced this pull request Aug 21, 2026
* origin/main: (64 commits)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)
  feat(desktop-messages): show compact Buzz link metadata (#6252)
  feat(workflows): reply in-thread from send_message action (#6178)
  perf(desktop): split discover_acp_providers into cheap and forced paths (#6330)
  fix(desktop): restore recent channel sorting (#6402)
  fix(desktop): isolate main timeline stacking context from focus drawer (#6398)
  fix(desktop): make reconnect repair lossless (#6415)
  fix(hooks): scope pre-push lanes to branch merge-base diff (#6423)
  Enforce a three-day dependency cooldown (#6426)
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  ...

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
TheSentinel454 added a commit that referenced this pull request Aug 21, 2026
…kflow-management

* origin/main: (27 commits)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)
  feat(desktop-messages): show compact Buzz link metadata (#6252)
  feat(workflows): reply in-thread from send_message action (#6178)
  perf(desktop): split discover_acp_providers into cheap and forced paths (#6330)
  fix(desktop): restore recent channel sorting (#6402)
  fix(desktop): isolate main timeline stacking context from focus drawer (#6398)
  fix(desktop): make reconnect repair lossless (#6415)
  fix(hooks): scope pre-push lanes to branch merge-base diff (#6423)
  Enforce a three-day dependency cooldown (#6426)
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  ...

Signed-off-by: Luke Tornquist <tornquist@squareup.com>

# Conflicts:
#	desktop/src/features/workflows/ui/WorkflowDeleteDialog.tsx
#	desktop/src/features/workflows/ui/WorkflowsView.tsx
cursor Bot pushed a commit to Nuncio-hq/crew that referenced this pull request Aug 23, 2026
Upstream sync of block#5825 and block#6397 plus the always-on part of block#5671.

block#5825 — 20 Hz speaker levels and the ~30 Hz mic analyser lived in the same
context value as huddle lifecycle state, so every tick re-rendered every
`useHuddle()` consumer (and their subtrees) across the whole app. Levels now
live in a separate `HuddleLevelsContext` (`useHuddleLevels()`), the mic
analyser moves to `useMicLevelAnalyser`, and the lifecycle value is memoized.
Meters still see every tick; ordinary consumers see none (20 -> 0 re-renders
per second of huddle audio).

block#6397 — endpointing was a single threshold plus a silence counter, so a
one-frame blip opened a segment, a dipping vowel started the countdown, and
word onsets/offsets were clipped. Speech boundaries now live in a pure
`VadEndpoint`: onset needs consecutive above-threshold frames, confirmation
prepends bounded pre-roll, offset uses a lower threshold (hysteresis), flush
keeps only a hangover tail, and reset drops buffered audio so nothing leaks
between segments. Short segments now log why they were dropped.

block#5671 — only the push-to-talk grouping fix is portable here: a held shortcut
no longer silence-flushes mid-hold when the mic is also manually open
(`vad_flush_allowed`). The env-gated STT/TTS levers live in TTS internals this
branch does not touch; documented as such in docs/crew/HERMES.md, and
BUZZ_STT_FLUSH_MS stays removed.

Fixes #275

Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
cursor Bot pushed a commit to Nuncio-hq/crew that referenced this pull request Aug 23, 2026
Upstream sync of block#5825 and block#6397 plus the always-on part of block#5671.

block#5825 — 20 Hz speaker levels and the ~30 Hz mic analyser lived in the same
context value as huddle lifecycle state, so every tick re-rendered every
`useHuddle()` consumer (and their subtrees) across the whole app. Levels now
live in a separate `HuddleLevelsContext` (`useHuddleLevels()`), the mic
analyser moves to `useMicLevelAnalyser`, and the lifecycle value is memoized.
Meters still see every tick; ordinary consumers see none (20 -> 0 re-renders
per second of huddle audio).

block#6397 — endpointing was a single threshold plus a silence counter, so a
one-frame blip opened a segment, a dipping vowel started the countdown, and
word onsets/offsets were clipped. Speech boundaries now live in a pure
`VadEndpoint`: onset needs consecutive above-threshold frames, confirmation
prepends bounded pre-roll, offset uses a lower threshold (hysteresis), flush
keeps only a hangover tail, and reset drops buffered audio so nothing leaks
between segments. Short segments now log why they were dropped.

block#5671 — only the push-to-talk grouping fix is portable here: a held shortcut
no longer silence-flushes mid-hold when the mic is also manually open
(`vad_flush_allowed`). The env-gated STT/TTS levers live in TTS internals this
branch does not touch; documented as such in docs/crew/HERMES.md, and
BUZZ_STT_FLUSH_MS stays removed.

Fixes #275

Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
BradGroux pushed a commit to BradGroux/buzz that referenced this pull request Aug 23, 2026
## Problem and intent

Buzz huddles have been clipping speech into fragments: one live turn
arrived as the single letter “M,” and another ended mid-sentence. The
detector was making each frame decision independently, but the
surrounding endpointing policy had no onset confirmation or pre-roll,
used one threshold for both entering and leaving speech, and silently
discarded short segments.

This PR keeps Earshot 1.1.0 and fixes that policy around it. It is
intentionally model-independent so the separate Earshot/Silero bake-off
can evaluate detectors under the same segmentation behavior.

## What changed

One production file changed: `desktop/src-tauri/src/huddle/stt.rs`
(**+367/−79** versus `main`).

- Add a pure `VadEndpoint` state machine around Earshot probabilities.
- Preserve 256 ms (16 frames) of pre-roll before confirmed onset.
- Require three consecutive frames above 0.50 to enter speech.
- Use 0.35 to leave speech, preserving hysteresis-band audio.
- Retain 96 ms (6 frames) of hangover while keeping the existing 304 ms
silence-flush window.
- Make short-segment drops visible in logs instead of silent.
- Add boundary, onset, hysteresis, hangover, drop-path, and PTT policy
tests.
- Bind the thresholds to Earshot 1.1.0 in source. Earshot 1.2.x is
deliberately parked pending a matched-policy bake-off; block#6392 separately
prevents Renovate from silently crossing that boundary.

## Boundary behavior and known tradeoffs

A hard message boundary clears pre-roll. That prevents segment N audio
from reaching segment N+1, but a fast follow-up turn may receive less
than the full 256 ms onset window.

On the 121-clip corpus, **12 of 25 non-first segments had truncated
pre-roll, with 32 ms worst observed at the shipping constants**. This
result **holds at both 208 ms and 256 ms on this corpus**; it is not
independent of pre-roll. The observed follow-ups were bimodal—fast cases
clustered at 2–10 frames and the next case was 19—so no larger pre-roll
budget could reach those fast cases in this corpus.

If a future refactor makes pre-roll survive a boundary, segment N
reaches segment N+1 under the strict predicate:

```text
gap < VAD_PRE_ROLL_FRAMES - VAD_ONSET_FRAMES
```

At the shipped values that means gaps 0–12 leak and gap 13 is the first
clean case. Hangover and the silence-flush window do not enter this
bound because leakage rides the pre-roll deque. Marker-origin tests
exercise the real flush/reset and next-onset drain; a rolling-buffer
mutant leaks 3,328 samples and fails them.

Other known behavior:

- `voiced_frames` now counts frames above the 0.35 exit threshold once
speech begins, not only frames above 0.50. Corpus drop counts measured
under `> 0.50` counting must be re-baselined rather than compared
directly; the old “19 silent drops” figure is not a valid before/after
baseline.
- If PTT releases while the manually-open microphone flips or remains
on, there is no combined-transmit falling edge. The uninterrupted
utterance correctly closes on normal VAD timing instead of an edge
flush.
- The GUI-to-`push_audio_pcm` wiring is unchanged. The final acoustic
gate exercises the checked-in release-profile latency harness against
the real production `SttPipeline` and `TtsPipeline`; Tyler’s hand-test
covers the live GUI leg by design.

## Verification ledger

All gates attest exact head `2dad6bb0e2e2a0af115a9a06c26cf6183fedb633`.

- **Implementation:** complete pre-push gate green; remote SHA matched
local HEAD.
- **Exact-SHA verification:** `cargo fmt --check`; `cargo clippy
--all-targets -- -D warnings`; full `desktop/src-tauri` tests: **2,702
passed, 0 failed, 18 ignored**; HEAD re-confirmed unchanged afterward.
- **Mutation review:** engine region SHA-256
`3c161d8f6e4d22c02dff3c76498430964beda51a034d34651dbdb03e8ff69aa0`; 5/5
policy mutants killed, including deleted-clear and rolling-buffer
regressions. Direct gap sweep leaked at 0–12 and was clean from 13.
- **Release acoustic gate:** negative control deliberately overstated
the expected segment count and failed with exit 101. Soft/short onset
produced 1/1 segment (`“I'm happy.”`). Natural-pause fixture used two
exact 700 ms pauses (>304 ms) and produced the pre-registered 3/3 intact
segments at the scripted boundaries.
- **Release CPU beside Pocket TTS:** soft fixture median 32.1%, p95/max
56.8% (5 sparse samples); natural fixture median 14.55%, p95/max 17.2%
(8 sparse samples). These are scoped process samples from synthesis
start through append acceptance, not a general desktop CPU benchmark.

Local review receipts (not committed to the repository):

- `.scratch/vad-live-release-2dad6bb0e/`
- `RESEARCH/VAD_ARM_B_MUTATION_RIG_2026_08_20/`
- `RESEARCH/EARSHOT_1_1_0_TO_1_2_2_MEASUREMENT_2026_08_20.md`

## Hand-test focus

Before merge, exercise the two live shapes that originally failed:

1. Soft short openers such as “M” and “yes.”
2. Natural mid-sentence pauses.

Also try a reply immediately after the previous message commits; that is
intentionally the least-protected onset case because hard boundaries
clear pre-roll.

---

Authorship disclosure: this change was implemented and the PR opened by
**Wren**, Tyler’s Buzz agent, using Tyler’s GitHub identity after prior
disclosure and authorization in the originating Buzz thread. Tyler
remains the accountable human reviewer/operator.

---------

Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant