Skip to content

fix: agent reliability — no restart on channel-add, visible dead-letter notice#1468

Merged
wesbillman merged 2 commits into
mainfrom
brain/agent-reliability-fixes
Jul 2, 2026
Merged

fix: agent reliability — no restart on channel-add, visible dead-letter notice#1468
wesbillman merged 2 commits into
mainfrom
brain/agent-reliability-fixes

Conversation

@wesbillman

Copy link
Copy Markdown
Collaborator

Two focused fixes for the dominant agent-reliability failure modes discussed in #agent-reliability: turns silently dying when a running agent is added to a channel, and dead turns leaving channels waiting forever with no visible signal.

1. Don't restart running local agents on channel-add (desktop)

Adding a running local agent to a channel did stopManagedAgentstartManagedAgent to pick up the new subscription — killing any in-flight turn. That branch dates to #86, before buzz-acp learned dynamic subscriptions; the harness now subscribes to new channels live via member-added notifications, and the desktop doesn't set BUZZ_ACP_CHANNELS, so nothing blocks the dynamic path.

The fix is mostly deletion: remove the restart branch in attachManagedAgentToChannel plus the restarted result plumbing and stale UI copy.

2. Visible notice when a turn is dead-lettered (buzz-acp, buzz-agent)

When retries were exhausted, the queue dead-lettered the batch with only an ERROR log — the channel never learned the turn died. Now:

  • EventQueue::requeue returns the exhausted batch instead of discarding it.
  • handle_prompt_result posts a short kind:9 into the triggering thread: "⚠️ I couldn't process the last request after multiple retries (<reason>). Please re-send if it's still needed."
  • Transient failures still retry quietly with exponential backoff — only final dead-letter is surfaced, so no noise.
  • The notice is best-effort (5s timeout, errors logged and swallowed) and posted from a spawned task so it can never stall the main loop.

Also raises the buzz-agent LLM call timeout default 120s → 240s so slow-but-successful LLM calls stop being classified as failures and burning retries.

Testing

  • cargo test: buzz-acp 405 passed, buzz-agent 122 passed (includes a new dead-letter queue test)
  • cargo clippy clean; root + desktop-tauri rustfmt clean
  • desktop: pnpm test 1494 passed; typecheck clean
  • Playwright smoke: agents / profile / active-turn-resilience / profile-active-turn / smoke specs — 22 passed

Co-authored-by: Brain 21994759fc7a6fa6b965551d35cfd7897d262f2495467f2d78694ddcfa6a5c7e@sprout-oss.stage.blox.sqprod.co

wesbillman and others added 2 commits July 2, 2026 11:12
Adding a running local agent to a channel used stop+start to pick up
the new subscription, killing any in-flight turn. The harness has
handled membership additions dynamically since buzz-acp learned live
member-added subscriptions, so the restart branch is dead weight —
remove it and the associated "restarted" result plumbing/copy.

Co-authored-by: Brain <21994759fc7a6fa6b965551d35cfd7897d262f2495467f2d78694ddcfa6a5c7e@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Silent turn deaths left channels waiting on replies that would never
come. EventQueue::requeue now returns the exhausted batch instead of
discarding it, and handle_prompt_result posts a short kind:9 notice
("couldn't process the last request after multiple retries") into the
triggering thread. Transient failures still retry with backoff and stay
quiet — only final dead-letter is surfaced.

Also raises the buzz-agent LLM call timeout default from 120s to 240s
so slow-but-successful LLM calls stop being classified as failures.

Co-authored-by: Brain <21994759fc7a6fa6b965551d35cfd7897d262f2495467f2d78694ddcfa6a5c7e@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
@wesbillman wesbillman merged commit d9c4e4a into main Jul 2, 2026
29 checks passed
@wesbillman wesbillman deleted the brain/agent-reliability-fixes branch July 2, 2026 18:16
wpfleger96 added a commit that referenced this pull request Jul 2, 2026
…n-metrics

* origin/main:
  feat: per-community workspace icon set by admins, served via NIP-11 (#1463)
  perf(relay): batch outbound websocket data frames (#1464)
  Make reaction ingest atomic (#1458)
  Serialize fan-out EVENT frames once (#1459)
  fix: agent reliability — no restart on channel-add, visible dead-letter notice (#1468)

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>

# Conflicts:
#	crates/buzz-relay/src/handlers/event.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants