Skip to content

fix(server): keep the shared Codex app-server scope alive after a tool OOM kill - #18329

Open
only21mil wants to merge 2 commits into
pingdotgg:mainfrom
only21mil:fix/17662-shared-codex-scope-oom-continue
Open

only21mil wants to merge 2 commits into
pingdotgg:mainfrom
only21mil:fix/17662-shared-codex-scope-oom-continue

Conversation

@only21mil

@only21mil only21mil commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem

#17662 introduced a regression for the shared Codex app-server on systemd 252 or newer by putting it and its tools in a scope with OOMPolicy=stop where supported. A kernel OOM kill of one tool now stops the whole scope, taking down the app-server used by every Codex thread on that provider instance. #16502 and #17045, both filed before #17662 merged, report threads staying broken after the shared app-server exits; #17046 and #18139 address that recovery.

An October 11 comment on #17045 reports exit code 0 after an earlier kernel OOM kill in a T3 agent scope on an October 10 nightly, consistent with a scope stop but not proof of one. The commenter notes no OOM near the later failures and says OOM does not explain the repeated fresh-thread failures, which are the recovery bug.

Change

  • Add optional shared to AgentScope.wrap for a process used by several threads.
  • Pass shared: true in the Codex app-server spawn helper, the only caller changed.
  • On systemd 253+, use OOMPolicy=continue, remove the launching thread's scope tracking and previous OOM answer, and add --collect because no thread reads the scope result. This avoids the roughly two-second poll after a tool OOM, since a continuing scope never records Result=oom-kill.
  • Keep older-systemd launch policy, tracking and OOM answers unchanged. Tests cover the shared launch policy, immediate answer and fallback.

apps/server/src/cloud/bootService.ts already keeps the server unit on OOMPolicy=continue because a helper's kernel OOM kill would otherwise stop the whole unit.

Behavior and limits

  • The kernel can still pick the app-server itself as the OOM victim. On 253+, systemd no longer stops the scope; tools the app-server left behind can keep running in the scope until they exit. This PR adds no cleanup for them.
  • A systemd-oomd kill still ends the whole scope.
  • systemd older than 253 cannot set OOMPolicy on scopes. On 252 a tool's OOM kill still stops the shared scope, so this PR does not help there. On 251 and older a plain scope never stops for an OOM kill, so the problem does not arise.
  • OpenCode's server in OpenCodeRuntime.ts is shared the same way and keeps stop here; it can take shared: true separately. All other scope policies stay unchanged.
  • On 253+, if the app-server itself is OOM-killed, systemd records no OOM result, so the opening thread can no longer get "Killed: out of memory." Codex turns rarely reach that label today: fix(server): agents run in their own systemd scopes so an OOM kill spares the server #17662 notes Codex has no process-exit handling.
  • Session recovery remains the work in #17046 and #18139.

Scope and approval

CONTRIBUTING permits: "A very small, focused fix for an obvious bug can be submitted without a prior issue or discussion."

There is no maintainer-triaged issue for this exact change; it fixes one defect, a tool OOM stopping unrelated Codex threads through the shared scope, and the exception does not imply maintainer approval.

Refs #17662, #17045, #16502, #17046, #18139.

Verification

Environment: Linux x64, systemd 259, Node 24.21.
Regression results at 3c38e194ec, with the same nine-test file against main's source and the branch. The branch image's c3d03a7eb3 has the same tree as 3c38e194ec; only the author and committer differ:

(cd apps/server && vp test run --reporter=verbose src/process/agentScope.test.ts)

Main source: 2 failed | 7 passed (9), missing the shared policy and immediate answer.
Main regression test output

Branch: 9 passed (9), including shared policy, immediate answer and older-systemd fallback.
Branch regression test output

With the policy swap alone, shared scopes still tracked as on main, 2 failed | 7 passed (9): the shared scope loses --collect and the OOM answer waits again. This distinguishes the tracking fix from the policy change.
These tests use fake systemd and do not create a real Codex OOM.

A scratch run used T3's Codex spawn helper and AgentScope service with a stand-in app-server on real systemd, a verified 64 MiB scope cap and no swap; the slice write was a logged no-op and the unit was renamed so local drop-ins did not apply.

Observation Main 2625327090 This branch
Launch option --property=OOMPolicy=stop --property=OOMPolicy=continue --collect
Tool exit Not printed; stand-in killed before reporting it 137, kernel OOM SIGKILL
Server alive No server_alive=yes, scope active/running, oom_kill=1
Stand-in exit SIGTERM, systemd stopped the scope 0
Scope Result oom-kill, scope failed success; unit collected after the stand-in exited
oomKilled("e2e-thread") true false

The image's ef46d65baf is this branch before its rebase onto bed923986f, with the same patch ID as 9796521680.
Real systemd main and branch output using a stand-in app-server

Focused checks at 9796521680: 161 Codex adapter tests and 1 shared test passed; lint, format, Knip and both typechecks exited 0, with two existing lint warnings and typecheck suggestions. Lint, format, Knip and the server typecheck exited 0 again on 3c38e194ec.

Not checked

  • A real Codex app-server under memory pressure or a UI-driven run.
  • Global or ancestor-slice OOM, systemd-oomd kills, or systemd older than 253 on a live machine.
  • Tool descendants after an app-server OOM kill.
  • macOS or Windows, which do not use these scopes.
  • The full test suite and a build.

Model and harness: Claude Opus 5.5 in Claude Code and GPT-6.1 Sol in Codex, run from T3 Code.

🤖 Generated with Claude Code

…l OOM kill

Since pingdotgg#17662 every agent scope starts with OOMPolicy=stop. The Codex
app-server is shared by every thread on a provider instance, and the
commands its tools run live in the same scope. When the kernel
OOM-killed one tool process, systemd stopped the whole scope, so the
app-server died and every Codex thread on that instance failed.

AgentScope.wrap now takes `shared`, and the Codex app-server spawn
passes it. On systemd 253 or newer, shared scopes start with
OOMPolicy=continue, so systemd logs the kill and the app-server keeps
running. Claude and terminal scopes keep OOMPolicy=stop. The probe and
the plain-scope fallback for older systemd are unchanged.

With OOMPolicy=continue systemd never records Result=oom-kill, so such
a scope can never explain one thread's failure. It is no longer
tracked for the thread that opened it. Otherwise every later failed
turn on that thread would poll the scope for about 2 seconds, waiting
for a stop that never comes. Launching it still ends the thread's
previous answer, and nothing reads its result, so it gets --collect
like thread-less scopes.

Older user managers cannot set OOMPolicy on a scope. There a shared
scope starts as before and stays tracked for its thread without
--collect, because systemd can still stop it on an OOM kill and record
Result=oom-kill (systemd 252 does).

Not changed:
- The kernel still picks the victim. If it kills the app-server
  itself, every thread on the instance still loses it.
- A systemd-oomd kill still ends the whole scope.
- On systemd 253 or newer, a Codex turn that fails because its
  app-server was itself killed for memory is no longer labeled
  "Killed: out of memory.", since systemd records no OOM result under
  OOMPolicy=continue. Before, only the thread that opened the session
  got that label.
- Recovering a session whose app-server died is pingdotgg#17046 and pingdotgg#18139.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Oct 11, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 11, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 3c38e19

Macroscope's review found this PR approvable — This is a narrowly scoped fix for shared Codex app-server scopes being terminated when a tool is OOM-killed. The new policy is isolated to shared Codex launches, preserves existing behavior for other scopes and older systemd versions, and is covered by focused tests.

You can add or adjust custom eligibility rules. Learn more.

@juliusmarminge juliusmarminge added the macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews label Oct 11, 2026 — with ChatGPT Codex Connector
@macroscopeapp
macroscopeapp Bot dismissed their stale review October 11, 2026 19:59

Dismissing prior approval to re-evaluate d25189d

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Path: .coderabbit.config.ts
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6ea80dd9-33df-46a5-a69e-1c3d96adfd0d

📥 Commits

Reviewing files that changed from the base of the PR and between 9796521 and 3c38e19.


📒 Files selected for processing (2)
  • apps/server/src/process/agentScope.test.ts
  • apps/server/src/process/agentScope.ts

🚧 Files skipped from review as they are similar to previous changes (1)
  • apps/server/src/process/agentScope.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.



📝 Walkthrough
📝 Walkthrough

Walkthrough

Codex app-server launches now mark their process as shared. AgentScope applies shared-scope OOM handling when supported and updates thread OOM tracking. Tests cover systemd support and fallback behavior.

Changes

Shared Codex app-server OOM handling

Layer / File(s) Summary
Shared scope contract and OOM handling
packages/shared/src/AgentScope.ts, apps/server/src/process/agentScope.ts
AgentScope.wrap accepts a shared flag. Shared scopes use OOMPolicy=continue when supported and are excluded from an individual thread’s tracked OOM scope.
Codex shared launch and validation
apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts, apps/server/src/process/agentScope.test.ts
Codex app-server commands set shared: true. Tests cover OOM policy selection, shared-scope tracking, and behavior when OOM policy is unavailable.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant CodexAdapterV2
  participant AgentScope
  participant systemd
  participant CodexAppServer
  CodexAdapterV2->>AgentScope: Wrap launch with shared true
  AgentScope->>systemd: Launch scope with OOMPolicy=continue when supported
  systemd->>CodexAppServer: Start app-server command
Loading

Suggested reviewers: t3dotgg


Merge Risk: ⚪ Minimal · up to 3c38e

The change addresses tool OOM containment for the shared Codex app-server, and no concrete additional user-impacting regression was established. It appears ready to merge subject to normal checks.

Architecture Summary

Architecture risk: 🔵 Low · up to 3c38e

The change affects 2 systems.

Changed systems: apps/server, packages/shared

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — apps/server (service) was modified; 3 changed files map to changed impact.
  • observed — packages/shared (library) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts: makeCodexAppServerSpawnCommand now passes shared: true to AgentScope.wrap; previously it did not mark the app-server as shared.
  • observed — Modified behavior in packages/shared/src/AgentScope.ts: The wrap input gains the optional shared flag, documented as indicating a process shared by several threads and describing its systemd 253+ OOM behavior and older-systemd behavior.
  • observed — Modified behavior in apps/server/src/process/agentScope.test.ts: The test imports add assertion, AgentScope shape, Fiber, FileSystem, ChildProcess, and the Codex app-server command factory; the prior imports of assert-free test utilities and only ChildProcessSpawner are replaced or expanded.
  • observed — Modified behavior in apps/server/src/process/agentScope.test.ts: The fake scope unit model adds an optional oomKills count, and withFakeSystemd accepts an oomPolicy option defaulting to true. Its callback receives the updated unit map, and the output helper now accepts a process exit code, defaulting to zero.

Pre-merge checks | Passed 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description covers the problem, change, scope and approval rationale, verification results, limitations, and agent details. It provides focused test evidence and clearly identifies unchecked scena…
Title check Passed The title is concise, conventional, and accurately identifies the main change: keeping the shared Codex app-server scope alive after a tool OOM kill.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

The test now says it checks the shared app-server launch options and also asserts that the shared scope is launched with --collect. The probe comment now says which older systemd user managers keep the OOM label: 252 still stops a plain scope on an OOM kill and records it, while 251 and older do neither.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@only21mil
only21mil force-pushed the fix/17662-shared-codex-scope-oom-continue branch from d25189d to 9796521 Compare October 12, 2026 11:30

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews size:M 30-99 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants