[Bug]: Delegated Codex child completes, but its subagent node stays Running and the parent is never woken #17512

Activity

1. juliusmarminge commented on Oct 9, 2026

   

   juliusmarminge

   on Oct 9, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Thanks for the detailed report and the database counts. I read the code on main (ec80933). I didn't reproduce this, so treat the following as a likely lead rather than a diagnosis.

   How the row should move to completed

   • The orchestration_v2_projection_subagents row is an upsert keyed on subagent_id (the parent's task node), driven by subagent.updated on the parent thread: ProjectionStore.ts#L2030-L2096.
   • For app-owned children, the only live writer of the terminal row is finalizeAppOwnedSubagent, which runs from the terminal run.updated listener on the child thread: Orchestrator.ts#L10630-L10697. It writes the terminal subagent.updated, the result transfer, and the wake together: Orchestrator.ts#L9766-L9871.

   Where it can return without writing anything
   finalizeAppOwnedSubagent has several early returns before that write: Orchestrator.ts#L9586-L9665. Your row (updated_at equal to creation time, no completed_at) fits one of them firing. The most plausible one seems to be L9600-L9601: it only proceeds when delegatedTaskProgress reports result_available. That function reports waiting_for_children if the child thread has any subagent row still active, a child task whose completion delivery is pending/claimed, or a provider thread with pendingBackgroundTasks: subagentProjection.ts#L237-L256. On a completed run, leftover run-owned subagents are not cascaded to a terminal status (the cascade covers only interrupted/failed/cancelled): RunExecutionService.ts#L162-L165.

   If that check is what fired, nothing re-runs finalize later while the server stays up. The listener reacts only to terminal run.updated events, so a nested item that settles a few seconds after the run (your trace shows a write at 13:54:14, after the run completed at 13:54:01) does not trigger it again. The catch-up pass recoverDelegatedTasks runs only at startup: Orchestrator.ts#L10704-L10731, ProjectionStore.ts#L3569-L3593. Any error inside finalize is also only logged as a warning ("Failed to react to terminal V2 run"), with no retry.

   The overlapping parent runs probably don't explain the stuck row. Finalize looks up the parent run by task.runId and writes the terminal status whatever that run's state is. The parent run's state only affects whether a wake is offered: Orchestrator.ts#L9403-L9436.

   Why Stop doesn't clear it: stopping the parent run marks the task's completionDelivery as disposed but copies its status unchanged: Orchestrator.ts#L2115-L2139. The child has no live run left to interrupt, so no new terminal run.updated arrives to re-run finalize.

   Related: #15888 (there the Grok child run itself stays non-terminal), #16355 and #17154 (nested subagents left running, task stuck in waiting_for_children), which may share this root cause if the Codex child had a nested item open. Open PR #15201 changes task_status reads, startup recovery, and pending-background-work derivation. From its diff it doesn't seem to add a live re-check of finalizeAppOwnedSubagent, so it may not cover this case on its own.

   Useful next data, if you still have the stuck case:

   1. What task_status returns for that task, in particular workState.
   2. In the child thread: any orchestration_v2_projection_subagents rows still running, and pendingBackgroundTasks on its provider thread. What wrote the 13:54:14 event?
   3. Any Failed to react to terminal V2 run warning around 13:54:01.
   4. Whether restarting the server settles the node and wakes the parent. The startup recovery pass would run finalize again.
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Oct 9, 2026
3. Neonsy commented on Oct 9, 2026

   

   Neonsy

   on Oct 9, 2026

   Last edited by Neonsy

   More actions

   Another case on the same nightly (0.0.46-nightly.20261009.2861), this time with a Claude child. It includes the data asked for above

   Setup

   • A Claude parent runs delegate_task (async) to a Claude child
   • The child starts native background work and ends its turn
   • A background-task notification queues a wake run 25 ms before run 1 is recorded complete
   • The wake run finishes in about 0.15 s with no reply
   • The parent then calls task_cancel

   Expected

   The Lineage row shows the child as finished, and Stop is not offered

   Actual

   • The header reads "Lineage · 4 running" and counts this child
   • Its timer counts from the task's original start (1.3h and rising)
   • Stop subagent does nothing

   Timeline (UTC)

   Time	Event
   16:05:05.954	Run 2 (wake) requested
   16:05:05.979	Run 1 completed
   16:05:09.090	Run 2 completed
   16:05:20	First task_cancel returns cancel_requested with no run active. It records run_interrupt_request on the already completed run 2
   16:22	task_status returns the values below
   16:22	Second task_cancel returns completed. t3_thread_interrupt returns no_active_run

   The child thread itself reports completed with no active run

   task_status at 16:22

   status: completed
   workState: result_available
   hasPendingChildRuns: false
   resultContextTransferId: null
   

   How it fits the lead above

   1. At run 1's end, run 2 was already queued, so finalize skipped
   2. At run 2's end, finalize skipped again. The task was still waiting on background work, inferred from step 3
   3. The first task_cancel returned cancel_requested with no active run, which only happens in waiting_for_children
   4. The cancel cleared that work but produced no new terminal run.updated, so finalize never re-ran
   5. The app-owned row stayed running. The Lineage edge falls back to that row when the child has no live run, so the count, the timer and the Stop button all follow it

   Also observed

   • Settling the child thread does not clear the row. It still showed Running 1h 31m about 12 minutes after settling, while the server still reported the thread as completed and settled

   Not tried

   • Whether a server restart's recovery pass settles it