fix(server): tell the provider when a restart killed its background work

[juliusmarminge]

When the server restarts after a turn has settled while its background work was still running (Claude/Codex background subagents, background shells, monitors), startup recovery marks that work cancelled. The provider session then resumes from its own transcript, which has no record of the death, so the model keeps waiting for work that will never report back. Only T3 knows it happened.

What changed

• Recorded as an event, no migration. Reconciliation collects the background work it cancels on already-settled turns: open subagent / command_execution / dynamic_tool items, plus the provider-thread roster (the Claude local_bash path), deduped by native id. It records a short { kind, label } list on the thread's latest started run with a small new event, run.background-work-cancelled ({ runId, restartCancelledBackgroundWork }). The projectors merge only that optional run field (SQL: json_set on the row's payload). A run snapshot read before the commit therefore can't regress a lifecycle change made since, such as a checkpoint completing the run. Work on a running run is not recorded, because it dies with its turn and the existing "Continue" path covers that. Shutdown reconciliation records too, since a graceful restart cancels the work there first.
• Delivered once, through the handoff prefix. ProviderTurnStartService prepends Note: the T3 server restarted, and this background work was cancelled before it finished. It will not report back: plus the list to the provider input. It uses the same startWithHandoffs prefix (…\n\nUser message:\n<text>) as context handoffs. "Delivered" is derived, not cleared: the note is pending until a later run's attempt has a provider turn. Compactions and restart continuations don't count as delivery (Codex resumes those without a prompt). Rolled-back runs don't count either. The note belongs to the provider thread that lost the work: after a provider switch, turns on the other provider neither receive it nor consume it. It is bounded (≤10 entries plus "and K more", each label ≤160 chars) and counted in the handoff budget, since it travels with the user text.
• Not a fake user message. The note reaches the provider only. The timeline keeps the user's text, and the existing cancelled markers already show the user what happened.
• Provider-agnostic. The note is built from projection state, not adapter state, so Grok background tasks benefit once v2/grok-bg-restart makes them trackable. The ACP adapter is untouched.
• Test harness: runOrchestratorV2ProviderReplayScenario gets opt-in recoverOnStartup (runs ProviderRuntimeRecoveryService.recover before the effect worker starts, as serverRuntimeStartup does) and continueThreadsAfterServerUpdate.

Verification

• New OrchestratorReplayRestartBackgroundNote.integration.test.ts. It replays the recorded claude_background_subagent_after_root session across two runtimes and one DB, cut by a restart right after the root turn's result, with the subagent still running. The fresh runtime runs startup recovery, then two user turns. The replay runner pins each outbound prompt.offer: the first carries the note exactly once, the second doesn't. The subagent ends cancelled, and the timeline's user messages are unchanged.
  • Fails without the ProviderTurnStartService change (ClaudeReplayFrameMismatchError at prompt.offer:1).
  • Fails with delivery tracking disabled (prompt.offer:2 carries the note again).
• ProviderRuntimeRecoveryService.test.ts: the waiting-run case now asserts that no run.updated is emitted and the work is recorded through run.background-work-cancelled (roster and item deduped).
• ProjectionStore.test.ts: a run that completed after recovery's read stays completed once the event applies (fails with a full-snapshot run.updated).
• RestartBackgroundNote.test.ts: on a switch Claude→Codex→Claude, the note is not sent to or consumed by Codex and is delivered on return to Claude (fails without the provider-thread scoping). The note is capped at 10 entries plus "and 15 more", and labels at ≤160 chars.
• vp test run over ProviderRuntimeRecoveryService{,.regression}, RestartContinuation, ProviderTurnStartService, OrchestratorReplayRecovery, OrchestratorReplayRestartBackgroundNote, and OrchestratorReplayFixtures (full replay suite): all pass.
• vp test run on client-runtime orchestrationV2Projection.test.ts.
• tsc --noEmit for apps/server, packages/contracts, packages/client-runtime, apps/web and apps/mobile: clean. knip (files/deps, plus --exports for server, contracts and client-runtime): clean. vp lint / vp fmt on touched files: clean.
• Not run: repo-wide checks and a live provider run. Clients get the new event through client-runtime only (the projection reducer; the awareness relay ignores it), with no UI change.
• No Codex case: no Codex fixture records background work that outlives a settled turn.

Model: Claude Opus 5.5 (Claude Code)

🤖 Generated with Claude Code

---

[macroscopeapp]

🟠 High orchestration-v2/ProviderTurnStartService.ts:905

A run on a different provider thread can consume the restart note, so cancelled background work on the current provider thread is never reported when the user switches back. pendingRestartCancelledBackgroundWork receives all projection.runs and projection.providerTurns; scope both inputs to providerThread.id before calculating the note.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @apps/server/src/orchestration-v2/ProviderTurnStartService.ts around line 905:

A run on a different provider thread can consume the restart note, so cancelled background work on the current provider thread is never reported when the user switches back. `pendingRestartCancelledBackgroundWork` receives all `projection.runs` and `projection.providerTurns`; scope both inputs to `providerThread.id` before calculating the note.

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	—	4.9 KiB	—	6.8 KiB	✅
Codex	Thread snapshot wire	—	3.7 KiB	—	4.9 KiB	✅
Codex	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Codex	Live turn WebSocket decoded	—	20.4 KiB	—	29.3 KiB	✅
Codex	Live turn messages	—	2	—	8	✅
Claude	Total thread wire	—	4.9 KiB	—	6.8 KiB	✅
Claude	Thread snapshot wire	—	3.7 KiB	—	4.9 KiB	✅
Claude	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Claude	Live turn WebSocket decoded	—	20.8 KiB	—	29.3 KiB	✅
Claude	Live turn messages	—	2	—	8	✅

Baseline: unavailable · PR result: 85eb9a9 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 106.1 KiB
• Claude decoded thread snapshot: 106.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This is a substantial cross-cutting runtime change that records restart-cancelled work, persists new state, and injects provider-facing recovery context on later turns; it is more than a narrowly scoped bug fix. An unresolved high-severity finding also concerns provider-thread delivery behavior, so the recovery and delivery paths need human review.

Not approved because:

• 1 blocking correctness issue found at or above your repo's Minimum Blocking Severity

^{Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.}