Skip to content

fix(nightly-v2): resume interrupted runs and release held queues after an opted-in restart - #99

Closed
saphid wants to merge 1 commit into
mainfrom
fix/nightly-v2-restart-run-recovery
Closed

saphid wants to merge 1 commit into
mainfrom
fix/nightly-v2-restart-run-recovery

Conversation

@saphid

@saphid saphid commented Sep 30, 2026

Copy link
Copy Markdown
Owner

What Changed

Pin the restart recovery fix (branch fix/restart-run-recovery). V2 only. Fork main carries release configuration, not V2 source, so this PR only appends the manifest entry. The source commit is on the branch named below. Incident: research/t3-v2-server-update-outage-2026-09-30.md. Pins append at the end of .github/downstream-nightly-v2.json, so the PRs in this series conflict textually; whichever merges later rebases and keeps every entry. CI on this fork has been stuck, so verification is local.

Why

The 2026-09-30 restarts cancelled about 19 live runs. With continueThreadsAfterServerUpdate on (it was), recovery only continued a run whose saved provider state passed a strict check, and skipped every thread with queued work behind the interrupted run. About 13 threads needed a manual "continue", and four held queues sat for hours.

Now, with the opt-in, startup also continues the interrupted run when that check fails (the same "Continue where you left off" a manual resume sends), and when queued work follows the cancelled run, delivery releases the held queue through queue.resume instead of prompting. Runs waiting on the user, including running runs with an unanswered question, are left alone. Opted-out environments are unchanged. Each queue release has its own command id so a replay after another restart does not hit the old receipt.

Not built: a one-click "resume all" for opted-out environments. Clients already show a held queue with a resume action; a cross-thread action would need a new command and UI on web and mobile.

Verification

  • RestartContinuation.test.ts and ProviderRuntimeRecoveryService*.test.ts: 25 passed, with new cases for the relaxed selection, held-queue release, question-blocked runs, and the recovery commit carrying the continue effect.
  • tsc: no errors; lint and fmt clean. Not exercised against a live provider.

Implementation: Claude Sonnet 5.5 through T3 Code. Cross-vendor review: GPT-6 Astra (high) via codex exec read-only, because T3 delegate_task reported the Codex provider unavailable.

🤖 Generated with Claude Code

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XS labels Sep 30, 2026
@saphid

saphid commented Oct 1, 2026

Copy link
Copy Markdown
Owner Author

Not going upstream as a PR. The main gap (Codex/Claude sessions left 'ready' were never continued) was fixed upstream in pingdotgg#14182. Releasing held queues conflicts with the deliberate 'hold queued runs after restart' design, so it needs an Ideas discussion with maintainer approval first.

@saphid saphid closed this Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant