fix: detect token invalidation in 401 responses and stop cascade rotation#496
Conversation
…tion
Upstream responses containing explicit token-invalidation messages
("invalidated oauth token", "authentication token has been invalidated",
etc.) are now detected separately from generic 401 errors. When detected:
- A long cooldown (default 5 min, CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS)
is applied to the affected account.
- The 401 is returned directly to the client instead of rotating to the
next account, preventing the cascade where each successive account token
is invalidated in turn by OpenAI's anti-abuse detection.
- Session affinity for that session key is cleared.
Generic 401 responses (expired tokens, wrong credentials) continue to
rotate as before.
Fixes #495
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughadds token-invalidation-cooldown config and detection to the runtime proxy. on matching upstream 401 bodies the proxy refunds the runtime token, marks the account with an auth-failure cooldown, clears session affinity, persists account state, records a token_invalidated observability event, and returns the upstream 401 to the client without rotating. see lib/config.ts:1407-1448 and lib/runtime-rotation-proxy.ts:1729-1758. Changestoken invalidation cooldown + rotation
Sequence Diagram(s)sequenceDiagram
participant client as client
participant proxy as lib/runtime-rotation-proxy.ts
participant upstream as upstream/auth or model
participant store as account store / persistence
client->>proxy: request using runtime token
proxy->>upstream: forward request
upstream-->>proxy: 401 with body
proxy->>proxy: isTokenInvalidationError(body)
alt invalidation detected
proxy->>store: refund token, mark coolingUntil using tokenInvalidationCooldownMs
proxy->>store: clear session affinity, persist account state
proxy-->>client: forward upstream 401 (status/headers/body)
proxy->>store: record usage errorCode: "token_invalidated"
else generic 401
proxy->>proxy: apply auth-failure cooldown (short)
proxy->>proxy: rotate to next account and retry
proxy-->>client: recovered stream (200) or final response
end
estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes possibly related PRs
suggested labels
notes:
🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (2 warnings, 1 inconclusive)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const TOKEN_INVALIDATION_PHRASES = [ | ||
| "invalidated oauth token", | ||
| "authentication token has been invalidated", | ||
| "oauth token has been invalidated", | ||
| "token has been invalidated", | ||
| ] as const; |
There was a problem hiding this comment.
phrases 2 and 3 ("authentication token has been invalidated" and "oauth token has been invalidated") are both substrings of phrase 4 ("token has been invalidated"), so the .some() check will already match them via the broader phrase. phrases 2 and 3 are dead code. either remove phrase 4 to keep the more specific set, or drop phrases 2 and 3 since phrase 4 already covers them.
| const TOKEN_INVALIDATION_PHRASES = [ | |
| "invalidated oauth token", | |
| "authentication token has been invalidated", | |
| "oauth token has been invalidated", | |
| "token has been invalidated", | |
| ] as const; | |
| const TOKEN_INVALIDATION_PHRASES = [ | |
| "invalidated oauth token", | |
| "token has been invalidated", | |
| ] as const; |
Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/runtime-rotation-proxy.ts
Line: 124-129
Comment:
**redundant phrases in TOKEN_INVALIDATION_PHRASES**
phrases 2 and 3 (`"authentication token has been invalidated"` and `"oauth token has been invalidated"`) are both substrings of phrase 4 (`"token has been invalidated"`), so the `.some()` check will already match them via the broader phrase. phrases 2 and 3 are dead code. either remove phrase 4 to keep the more specific set, or drop phrases 2 and 3 since phrase 4 already covers them.
```suggestion
const TOKEN_INVALIDATION_PHRASES = [
"invalidated oauth token",
"token has been invalidated",
] as const;
```
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| accountManager.markAccountCoolingDown( | ||
| refreshed.account, | ||
| tokenInvalidationCooldownMs, | ||
| "auth-failure", | ||
| ); |
There was a problem hiding this comment.
both the invalidation path and the generic 401 path record "auth-failure" as the CooldownReason. since CooldownReason is a union in lib/storage/migrations.ts, adding "token-invalidated" would let operators distinguish a 5-minute invalidation cooldown from a 30-second auth cooldown in stored account state. the current state makes codex-multi-auth rotation status output ambiguous for affected accounts.
Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/runtime-rotation-proxy.ts
Line: 1672-1676
Comment:
**cooldownReason does not distinguish token invalidation from generic auth failure**
both the invalidation path and the generic 401 path record `"auth-failure"` as the `CooldownReason`. since `CooldownReason` is a union in `lib/storage/migrations.ts`, adding `"token-invalidated"` would let operators distinguish a 5-minute invalidation cooldown from a 30-second auth cooldown in stored account state. the current state makes `codex-multi-auth rotation status` output ambiguous for affected accounts.
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
test/schemas.test.ts (1)
33-77: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick winadd
tokenInvalidationCooldownMsto the full config test for consistency.the test includes
networkErrorCooldownMs(line 68) andserverErrorCooldownMs(line 69) but omits the newtokenInvalidationCooldownMsfield. for completeness and to verify the field integrates correctly with the full config object, add it here.suggested addition
networkErrorCooldownMs: 6000, serverErrorCooldownMs: 4000, + tokenInvalidationCooldownMs: 300000, preemptiveQuotaEnabled: true,🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/schemas.test.ts` around lines 33 - 77, The full-config unit test is missing the new tokenInvalidationCooldownMs field; update the test that constructs the config object in the "accepts valid full config" case so it includes tokenInvalidationCooldownMs with an appropriate numeric value (near other cooldowns like networkErrorCooldownMs and serverErrorCooldownMs) before calling PluginConfigSchema.safeParse; this ensures PluginConfigSchema (used in the test) validates the new field along with networkErrorCooldownMs and serverErrorCooldownMs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/runtime-rotation-proxy.ts`:
- Around line 121-129: Add a comment block above TOKEN_INVALIDATION_PHRASES
explaining these strings were observed in OpenAI/Microsoft error response bodies
when an OAuth token was explicitly revoked (not just expired), include the
provenance (source provider and date/CI run where observed) and note that this
list is used to detect non-retryable revoked-token errors in
runtime-rotation-proxy; instruct maintainers to add new phrases to
TOKEN_INVALIDATION_PHRASES when encountering different wording in production,
record the source and date in the comment, update any associated tests that
assert revoked-token detection (and run the test suite), and keep the list in
sync with provider docs/changelog.
In `@test/runtime-rotation-proxy.test.ts`:
- Around line 1843-1889: Add edge-case tests to runtime-rotation-proxy.test.ts
targeting phrase-detection robustness: (1) a 401 with an empty body should NOT
be treated as an explicit invalidation and should trigger rotation — use
AccountManager, createRecordingFetch((_call, attempt) => attempt === 1 ? new
Response("", { status: HTTP_STATUS.UNAUTHORIZED }) : textEventStream(...)) and
assert two upstream calls and proxy.getStatus().rotations increased; (2) a 401
with a non-JSON/html body containing the invalidation phrase (e.g.,
"<html>...oauth token has been invalidated...</html>") should be detected as
invalidation — return that Response via createRecordingFetch and assert no
rotation and accountManager.getAccountByIndex(0)?.cooldownReason ===
"auth-failure"; (3) simulate concurrent invalidation races by firing multiple
postResponses(proxy, ...) in parallel against a fetchImpl that returns the
invalidation response for the first N concurrent calls and ensure the account is
cooled once and rotations/call counts match expected behavior; reuse startProxy,
postResponses, createRecordingFetch and assert calls, headers
(OPENAI_HEADERS.ACCOUNT_ID), cooldownReason and proxy.getStatus().rotations
accordingly.
- Around line 1866-1889: Add a deterministic assertion that the first account's
coolingDownUntil uses the short auth-failure cooldown: after the simulated
generic 401, check AccountManager (the local accountManager variable used to
startProxy) for the first account's coolingDownUntil and assert it is
approximately now + 30_000 (use a small delta window, e.g. between now + 29_900
and now + 30_100) to ensure the generic 401 path sets the short cooldown rather
than the long invalidation cooldown.
- Around line 1843-1864: Update the test so it verifies the invalidation
cooldown duration and that session affinity was cleared: after calling
postResponses(proxy, ...) assert
accountManager.getAccountByIndex(0)?.coolingDownUntil is approximately now +
300_000 (5 minutes) rather than 30_000, and also send a request with
metadata.session_id (e.g. "session-invalidated") and assert that the session was
forgotten (i.e., subsequent requests are not routed to the same account;
reference the forgetSession behavior invoked in lib/runtime-rotation-proxy.ts),
keeping existing checks for cooldownReason ("auth-failure") and rotations; use
startProxy, postResponses, AccountManager.getAccountByIndex and
proxy.getStatus() to locate the relevant state.
---
Outside diff comments:
In `@test/schemas.test.ts`:
- Around line 33-77: The full-config unit test is missing the new
tokenInvalidationCooldownMs field; update the test that constructs the config
object in the "accepts valid full config" case so it includes
tokenInvalidationCooldownMs with an appropriate numeric value (near other
cooldowns like networkErrorCooldownMs and serverErrorCooldownMs) before calling
PluginConfigSchema.safeParse; this ensures PluginConfigSchema (used in the test)
validates the new field along with networkErrorCooldownMs and
serverErrorCooldownMs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: e003d052-2904-4e54-86f7-bd9bd4efdb84
📒 Files selected for processing (7)
lib/config.tslib/runtime-rotation-proxy.tslib/schemas.tstest/codex-manager-cli.test.tstest/plugin-config.test.tstest/runtime-rotation-proxy.test.tstest/schemas.test.ts
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Greptile Review
🧰 Additional context used
📓 Path-based instructions (11)
**/*.{ts,tsx,js,mjs}
📄 CodeRabbit inference engine (AGENTS.md)
Use ESM only (
"type": "module"), Node >= 18
Files:
test/schemas.test.tstest/codex-manager-cli.test.tstest/runtime-rotation-proxy.test.tslib/schemas.tslib/config.tstest/plugin-config.test.tslib/runtime-rotation-proxy.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use
as any,@ts-ignore, or@ts-expect-errorTypeScript assertions
Files:
test/schemas.test.tstest/codex-manager-cli.test.tstest/runtime-rotation-proxy.test.tslib/schemas.tslib/config.tstest/plugin-config.test.tslib/runtime-rotation-proxy.ts
{**/scripts/**/*.js,**/test/**/*.ts}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use bare recursive delete logic in Windows-sensitive scripts/tests without retry handling for transient
EBUSY/EPERM/ENOTEMPTYerrors
Files:
test/schemas.test.tstest/codex-manager-cli.test.tstest/runtime-rotation-proxy.test.tstest/plugin-config.test.ts
test/**/*.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
test/**/*.test.ts: Vitest globals (describe,it,expect) are enabled and should be used without explicit imports
Maintain 80% coverage threshold across statements, branches, functions, and lines
UseremoveWithRetryfor Windows filesystem cleanup instead of barefs.rmto handle EBUSY/EPERM/ENOTEMPTY backoff
Use source files in tests, not compileddist/files; test the source directly
Do not skip tests without justification; include rationale if a test must be skipped
Relax ESLint rules for test files as specified ineslint.config.js
Files:
test/schemas.test.tstest/codex-manager-cli.test.tstest/runtime-rotation-proxy.test.tstest/plugin-config.test.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (README.md)
**/*.{ts,tsx,js,jsx}: Implement default-on runtime Responses rotation for request-bearing forwarded Codex CLI/app sessions, with opt-out support viaCODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY=0
Store project-scoped accounts under~/.codex/multi-auth/projects/<project-key>/openai-codex-accounts.jsonfor repo-specific workflows
Support environment variable overrides for configuration includingCODEX_MULTI_AUTH_DIR,CODEX_MODE,CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY,CODEX_TUI_COLOR_PROFILE, and others as documented
Implement account health checks withcodex-multi-auth checkcommand that validates saved account credentials and state
Implement quota forecasting and budget guards to prevent exhausting account quota within a session
Record local usage in a ledger at~/.codex/multi-auth/usage/usage-ledger.jsonlfor per-project tracking and reporting
Implement bounded outbound request budget so one prompt cannot walk the full account pool indefinitely
Trigger short cooldown instead of continuing aggressive rotation when repeated cross-account 5xx bursts are detected
Stagger proactive token refresh to reduce background refresh bursts across the account pool
Expose recent runtime request metrics incodex-multi-auth statustext output and machine-readable metrics incodex-multi-auth report --json
Make OAuth callback listen on port 1455 for login flows
Support device authorization flow viacodex-multi-auth login --device-authas an alternate to browser-based OAuth for headless environments
Implement dashboard hotkeys including Up/Down for navigation, Enter for selection, 1-9 for quick switch, / for search, ? for help, Q to back, S to set account, R to refresh, E to enable/disable, and D to delete
Support named local pool backup export with filename prompt in the Settings > Experimental menu
Implementcodex-multi-auth doctor --fixcommand to diagnose and apply the safest fixes for storage or account state issues
Implementcodex-multi-auth fix --dry-run...
Files:
test/schemas.test.tstest/codex-manager-cli.test.tstest/runtime-rotation-proxy.test.tslib/schemas.tslib/config.tstest/plugin-config.test.tslib/runtime-rotation-proxy.ts
test/**
⚙️ CodeRabbit configuration file
tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.
Files:
test/schemas.test.tstest/codex-manager-cli.test.tstest/runtime-rotation-proxy.test.tstest/plugin-config.test.ts
test/**/codex-manager-cli.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
Test CLI settings with question cancellation across all 5 panels and EBUSY/concurrent race conditions
Files:
test/codex-manager-cli.test.ts
lib/**/*.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/**/*.ts: All public exports should flow throughlib/index.tsor documented package subpaths
Never import fromdist/in source tests or library code
Never suppress type errors
Files:
lib/schemas.tslib/config.tslib/runtime-rotation-proxy.ts
lib/**
⚙️ CodeRabbit configuration file
focus on auth rotation, windows filesystem IO, and concurrency. verify every change cites affected tests (vitest) and that new queues handle EBUSY/429 scenarios. check for logging that leaks tokens or emails.
Files:
lib/schemas.tslib/config.tslib/runtime-rotation-proxy.ts
**/lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (AGENTS.md)
**/lib/runtime-rotation-proxy.ts: Keep runtime rotation default-on behavior aligned with explicit release and migration documentation
Do not expose account emails or tokens in runtime proxy client response headers or logs
Files:
lib/runtime-rotation-proxy.ts
lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/runtime-rotation-proxy.ts: Runtime rotation code must preserve pass-through semantics except for auth/provider headers that intentionally change
Runtime proxy client-facing headers must not expose account emails or tokens
Runtime rotation should fail open to normal official Codex forwarding when startup helpers are unavailable
Never add account emails/tokens to runtime proxy client responses
Files:
lib/runtime-rotation-proxy.ts
🔇 Additional comments (9)
lib/schemas.ts (1)
73-73: LGTM!lib/config.ts (3)
201-201: LGTM!
1406-1424: LGTM!
1846-1850: LGTM!test/schemas.test.ts (2)
101-101: LGTM!
136-136: LGTM!test/codex-manager-cli.test.ts (1)
1178-1178: LGTM!test/plugin-config.test.ts (1)
163-163: LGTM!Also applies to: 235-235, 551-551, 624-624, 691-691
lib/runtime-rotation-proxy.ts (1)
1666-1688: LGTM!
…hrase provenance - Add comment above TOKEN_INVALIDATION_PHRASES explaining observed source providers (OpenAI/Microsoft), how to update the list, and reference to issue #495 for context. - Invalidation test: assert coolingDownUntil is ~5min (not 30s generic), confirming the long cooldown path is reached. - Generic 401 test: assert coolingDownUntil is ~30s (not 5min), confirming the short fallback path is unchanged. - Add edge case: empty 401 body does not trigger invalidation detection, rotation proceeds normally. - Add edge case: HTML 401 body containing invalidation phrase is detected and stops cascade rotation.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
test/runtime-rotation-proxy.test.ts (1)
1843-1867: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick winassert the upstream 401 body is forwarded unchanged.
these tests prove status and rotation, but not the passthrough contract in
lib/runtime-rotation-proxy.ts:1682-1683. addexpect(await response.text()).toBe(invalidationBody)intest/runtime-rotation-proxy.test.ts:1843-1867and the equivalent html-body assertion intest/runtime-rotation-proxy.test.ts:1918-1936, otherwise a future change could still rewrite the body while keeping these tests green.as per coding guidelines, "
lib/runtime-rotation-proxy.ts: runtime rotation code must preserve pass-through semantics except for auth/provider headers that intentionally change".Also applies to: 1918-1936
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/runtime-rotation-proxy.test.ts` around lines 1843 - 1867, Add assertions that the upstream 401 response body is forwarded unchanged: in the test case using invalidationBody (the block calling postResponses and asserting HTTP_STATUS.UNAUTHORIZED, calls length, headers, cooldowns and proxy.getStatus().rotations) add expect(await response.text()).toBe(invalidationBody) to verify passthrough semantics for JSON; likewise add an equivalent expect(await response.text()).toBe(htmlInvalidationBody) in the other test block (the HTML-response case) so both tests assert the exact body is returned unchanged by the runtime rotation proxy code paths that modify only auth/provider headers.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@test/runtime-rotation-proxy.test.ts`:
- Around line 1843-1867: Add assertions that the upstream 401 response body is
forwarded unchanged: in the test case using invalidationBody (the block calling
postResponses and asserting HTTP_STATUS.UNAUTHORIZED, calls length, headers,
cooldowns and proxy.getStatus().rotations) add expect(await
response.text()).toBe(invalidationBody) to verify passthrough semantics for
JSON; likewise add an equivalent expect(await
response.text()).toBe(htmlInvalidationBody) in the other test block (the
HTML-response case) so both tests assert the exact body is returned unchanged by
the runtime rotation proxy code paths that modify only auth/provider headers.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8cf9ff80-2230-449e-bed1-533f43a0d6b9
📒 Files selected for processing (2)
lib/runtime-rotation-proxy.tstest/runtime-rotation-proxy.test.ts
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Greptile Review
🧰 Additional context used
📓 Path-based instructions (10)
**/*.{ts,tsx,js,mjs}
📄 CodeRabbit inference engine (AGENTS.md)
Use ESM only (
"type": "module"), Node >= 18
Files:
lib/runtime-rotation-proxy.tstest/runtime-rotation-proxy.test.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use
as any,@ts-ignore, or@ts-expect-errorTypeScript assertions
Files:
lib/runtime-rotation-proxy.tstest/runtime-rotation-proxy.test.ts
**/lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (AGENTS.md)
**/lib/runtime-rotation-proxy.ts: Keep runtime rotation default-on behavior aligned with explicit release and migration documentation
Do not expose account emails or tokens in runtime proxy client response headers or logs
Files:
lib/runtime-rotation-proxy.ts
lib/**/*.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/**/*.ts: All public exports should flow throughlib/index.tsor documented package subpaths
Never import fromdist/in source tests or library code
Never suppress type errors
Files:
lib/runtime-rotation-proxy.ts
lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/runtime-rotation-proxy.ts: Runtime rotation code must preserve pass-through semantics except for auth/provider headers that intentionally change
Runtime proxy client-facing headers must not expose account emails or tokens
Runtime rotation should fail open to normal official Codex forwarding when startup helpers are unavailable
Never add account emails/tokens to runtime proxy client responses
Files:
lib/runtime-rotation-proxy.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (README.md)
**/*.{ts,tsx,js,jsx}: Implement default-on runtime Responses rotation for request-bearing forwarded Codex CLI/app sessions, with opt-out support viaCODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY=0
Store project-scoped accounts under~/.codex/multi-auth/projects/<project-key>/openai-codex-accounts.jsonfor repo-specific workflows
Support environment variable overrides for configuration includingCODEX_MULTI_AUTH_DIR,CODEX_MODE,CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY,CODEX_TUI_COLOR_PROFILE, and others as documented
Implement account health checks withcodex-multi-auth checkcommand that validates saved account credentials and state
Implement quota forecasting and budget guards to prevent exhausting account quota within a session
Record local usage in a ledger at~/.codex/multi-auth/usage/usage-ledger.jsonlfor per-project tracking and reporting
Implement bounded outbound request budget so one prompt cannot walk the full account pool indefinitely
Trigger short cooldown instead of continuing aggressive rotation when repeated cross-account 5xx bursts are detected
Stagger proactive token refresh to reduce background refresh bursts across the account pool
Expose recent runtime request metrics incodex-multi-auth statustext output and machine-readable metrics incodex-multi-auth report --json
Make OAuth callback listen on port 1455 for login flows
Support device authorization flow viacodex-multi-auth login --device-authas an alternate to browser-based OAuth for headless environments
Implement dashboard hotkeys including Up/Down for navigation, Enter for selection, 1-9 for quick switch, / for search, ? for help, Q to back, S to set account, R to refresh, E to enable/disable, and D to delete
Support named local pool backup export with filename prompt in the Settings > Experimental menu
Implementcodex-multi-auth doctor --fixcommand to diagnose and apply the safest fixes for storage or account state issues
Implementcodex-multi-auth fix --dry-run...
Files:
lib/runtime-rotation-proxy.tstest/runtime-rotation-proxy.test.ts
lib/**
⚙️ CodeRabbit configuration file
focus on auth rotation, windows filesystem IO, and concurrency. verify every change cites affected tests (vitest) and that new queues handle EBUSY/429 scenarios. check for logging that leaks tokens or emails.
Files:
lib/runtime-rotation-proxy.ts
{**/scripts/**/*.js,**/test/**/*.ts}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use bare recursive delete logic in Windows-sensitive scripts/tests without retry handling for transient
EBUSY/EPERM/ENOTEMPTYerrors
Files:
test/runtime-rotation-proxy.test.ts
test/**/*.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
test/**/*.test.ts: Vitest globals (describe,it,expect) are enabled and should be used without explicit imports
Maintain 80% coverage threshold across statements, branches, functions, and lines
UseremoveWithRetryfor Windows filesystem cleanup instead of barefs.rmto handle EBUSY/EPERM/ENOTEMPTY backoff
Use source files in tests, not compileddist/files; test the source directly
Do not skip tests without justification; include rationale if a test must be skipped
Relax ESLint rules for test files as specified ineslint.config.js
Files:
test/runtime-rotation-proxy.test.ts
test/**
⚙️ CodeRabbit configuration file
tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.
Files:
test/runtime-rotation-proxy.test.ts
🔇 Additional comments (1)
test/runtime-rotation-proxy.test.ts (1)
1843-1867: still missing the invalidation affinity-clearing regression.
lib/runtime-rotation-proxy.ts:1680forgets the session key on token invalidation, buttest/runtime-rotation-proxy.test.ts:1843-1867never seeds a session id or proves the next request is no longer pinned to the cooled account. this auth-rotation path still needs the regression coverage called out earlier.as per coding guidelines, "
test/**: tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior."
…tching Adds a global per-proxy rotation throttle that biases account selection toward the last successfully-served account within a configurable time window (default 60 seconds, env: CODEX_AUTH_MIN_ROTATION_INTERVAL_MS). When the last served account is within the window and still available, it receives a large score boost (1000) in the hybrid selection algorithm, overriding the freshness weight that previously caused the proxy to eagerly rotate to idle accounts on every request. This reduces how often different OAuth tokens are presented from the same IP in quick succession -- the primary fingerprint that triggers OpenAI's anti-abuse detection and causes cascade token invalidation (issue #495). The boost is applied only to available accounts, so rate-limited or cooling-down accounts are still skipped and rotation proceeds naturally. Setting minRotationIntervalMs to 0 disables the throttle entirely.
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
lib/runtime-rotation-proxy.ts (1)
1685-1712:⚠️ Potential issue | 🟠 Majorprevent cooldown shortening on concurrent 401s
- in
lib/runtime-rotation-proxy.ts:1691-1712, the token-invalidation 401 path marks a long cooldown and returns, but another 401 path can mark the same account with the shorter"auth-failure"cooldown (DEFAULT_AUTH_FAILURE_COOLDOWN_MS) for concurrent requests.- in
lib/accounts.ts:1078-1086,markAccountCoolingDown(...): voidoverwritesaccount.coolingDownUntil = nowMs() + msunconditionally (no max-deadline/min-clamp), so a later shorter call can shrink the invalidation window under concurrency.- add a vitest that fires concurrent 401s for the same account where one is
isTokenInvalidationError(bodyText)and the other is the generic auth-failure branch, then assertcoolingDownUntilnever decreases (current tests liketest/accounts.test.ts:1254-1258andtest/rotation-integration.test.ts:174-178cover single cooldown state, not concurrent overwrites).- align the implementations so the sync
markAccountCoolingDown(...): voidinlib/accounts.ts:1078-1086cannot regress the longer window—either funnel through the mutexed cooldown updater atlib/accounts.ts:1016-1021or implement max-window semantics in the sync path.- windows edge case: both branches persist via
saveToDiskDebounced()inlib/runtime-rotation-proxy.ts:1691-1712; add coverage that debounced persistence ordering can’t write back a shorter cooldown value after races (esp. with windows filesystem/ebusy scenarios).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/runtime-rotation-proxy.ts` around lines 1685 - 1712, The token-invalidation 401 path can set a long cooldown but a concurrent generic 401 can later overwrite it with a shorter window because markAccountCoolingDown(...) writes coolingDownUntil unconditionally; change markAccountCoolingDown in lib/accounts.ts to never shorten an existing deadline (set account.coolingDownUntil = max(account.coolingDownUntil || 0, nowMs() + ms)) or delegate the sync path to the existing mutexed updater (the updater used around the cooldown logic at the mutexed block), and keep saveToDiskDebounced() calls as-is; add a Vitest that sends concurrent 401 responses (one where isTokenInvalidationError(bodyText) is true and one generic auth-failure) against the same account and asserts coolingDownUntil never decreases and that persisted disk state also never ends up with the shorter value after debounced writes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@lib/runtime-rotation-proxy.ts`:
- Around line 1685-1712: The token-invalidation 401 path can set a long cooldown
but a concurrent generic 401 can later overwrite it with a shorter window
because markAccountCoolingDown(...) writes coolingDownUntil unconditionally;
change markAccountCoolingDown in lib/accounts.ts to never shorten an existing
deadline (set account.coolingDownUntil = max(account.coolingDownUntil || 0,
nowMs() + ms)) or delegate the sync path to the existing mutexed updater (the
updater used around the cooldown logic at the mutexed block), and keep
saveToDiskDebounced() calls as-is; add a Vitest that sends concurrent 401
responses (one where isTokenInvalidationError(bodyText) is true and one generic
auth-failure) against the same account and asserts coolingDownUntil never
decreases and that persisted disk state also never ends up with the shorter
value after debounced writes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: a2601d97-9fab-443e-94c8-280a7a8ea31a
📒 Files selected for processing (7)
lib/config.tslib/runtime-rotation-proxy.tslib/schemas.tstest/codex-manager-cli.test.tstest/plugin-config.test.tstest/runtime-rotation-proxy.test.tstest/schemas.test.ts
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Greptile Review
🧰 Additional context used
📓 Path-based instructions (11)
**/*.{ts,tsx,js,mjs}
📄 CodeRabbit inference engine (AGENTS.md)
Use ESM only (
"type": "module"), Node >= 18
Files:
lib/schemas.tstest/codex-manager-cli.test.tstest/schemas.test.tstest/plugin-config.test.tslib/config.tstest/runtime-rotation-proxy.test.tslib/runtime-rotation-proxy.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use
as any,@ts-ignore, or@ts-expect-errorTypeScript assertions
Files:
lib/schemas.tstest/codex-manager-cli.test.tstest/schemas.test.tstest/plugin-config.test.tslib/config.tstest/runtime-rotation-proxy.test.tslib/runtime-rotation-proxy.ts
lib/**/*.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/**/*.ts: All public exports should flow throughlib/index.tsor documented package subpaths
Never import fromdist/in source tests or library code
Never suppress type errors
Files:
lib/schemas.tslib/config.tslib/runtime-rotation-proxy.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (README.md)
**/*.{ts,tsx,js,jsx}: Implement default-on runtime Responses rotation for request-bearing forwarded Codex CLI/app sessions, with opt-out support viaCODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY=0
Store project-scoped accounts under~/.codex/multi-auth/projects/<project-key>/openai-codex-accounts.jsonfor repo-specific workflows
Support environment variable overrides for configuration includingCODEX_MULTI_AUTH_DIR,CODEX_MODE,CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY,CODEX_TUI_COLOR_PROFILE, and others as documented
Implement account health checks withcodex-multi-auth checkcommand that validates saved account credentials and state
Implement quota forecasting and budget guards to prevent exhausting account quota within a session
Record local usage in a ledger at~/.codex/multi-auth/usage/usage-ledger.jsonlfor per-project tracking and reporting
Implement bounded outbound request budget so one prompt cannot walk the full account pool indefinitely
Trigger short cooldown instead of continuing aggressive rotation when repeated cross-account 5xx bursts are detected
Stagger proactive token refresh to reduce background refresh bursts across the account pool
Expose recent runtime request metrics incodex-multi-auth statustext output and machine-readable metrics incodex-multi-auth report --json
Make OAuth callback listen on port 1455 for login flows
Support device authorization flow viacodex-multi-auth login --device-authas an alternate to browser-based OAuth for headless environments
Implement dashboard hotkeys including Up/Down for navigation, Enter for selection, 1-9 for quick switch, / for search, ? for help, Q to back, S to set account, R to refresh, E to enable/disable, and D to delete
Support named local pool backup export with filename prompt in the Settings > Experimental menu
Implementcodex-multi-auth doctor --fixcommand to diagnose and apply the safest fixes for storage or account state issues
Implementcodex-multi-auth fix --dry-run...
Files:
lib/schemas.tstest/codex-manager-cli.test.tstest/schemas.test.tstest/plugin-config.test.tslib/config.tstest/runtime-rotation-proxy.test.tslib/runtime-rotation-proxy.ts
lib/**
⚙️ CodeRabbit configuration file
focus on auth rotation, windows filesystem IO, and concurrency. verify every change cites affected tests (vitest) and that new queues handle EBUSY/429 scenarios. check for logging that leaks tokens or emails.
Files:
lib/schemas.tslib/config.tslib/runtime-rotation-proxy.ts
{**/scripts/**/*.js,**/test/**/*.ts}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use bare recursive delete logic in Windows-sensitive scripts/tests without retry handling for transient
EBUSY/EPERM/ENOTEMPTYerrors
Files:
test/codex-manager-cli.test.tstest/schemas.test.tstest/plugin-config.test.tstest/runtime-rotation-proxy.test.ts
test/**/*.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
test/**/*.test.ts: Vitest globals (describe,it,expect) are enabled and should be used without explicit imports
Maintain 80% coverage threshold across statements, branches, functions, and lines
UseremoveWithRetryfor Windows filesystem cleanup instead of barefs.rmto handle EBUSY/EPERM/ENOTEMPTY backoff
Use source files in tests, not compileddist/files; test the source directly
Do not skip tests without justification; include rationale if a test must be skipped
Relax ESLint rules for test files as specified ineslint.config.js
Files:
test/codex-manager-cli.test.tstest/schemas.test.tstest/plugin-config.test.tstest/runtime-rotation-proxy.test.ts
test/**/codex-manager-cli.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
Test CLI settings with question cancellation across all 5 panels and EBUSY/concurrent race conditions
Files:
test/codex-manager-cli.test.ts
test/**
⚙️ CodeRabbit configuration file
tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.
Files:
test/codex-manager-cli.test.tstest/schemas.test.tstest/plugin-config.test.tstest/runtime-rotation-proxy.test.ts
**/lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (AGENTS.md)
**/lib/runtime-rotation-proxy.ts: Keep runtime rotation default-on behavior aligned with explicit release and migration documentation
Do not expose account emails or tokens in runtime proxy client response headers or logs
Files:
lib/runtime-rotation-proxy.ts
lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/runtime-rotation-proxy.ts: Runtime rotation code must preserve pass-through semantics except for auth/provider headers that intentionally change
Runtime proxy client-facing headers must not expose account emails or tokens
Runtime rotation should fail open to normal official Codex forwarding when startup helpers are unavailable
Never add account emails/tokens to runtime proxy client responses
Files:
lib/runtime-rotation-proxy.ts
🔇 Additional comments (3)
test/codex-manager-cli.test.ts (1)
1179-1179: LGTM!test/plugin-config.test.ts (1)
164-164: LGTM!Also applies to: 237-237, 554-554, 628-628, 696-696
test/schemas.test.ts (1)
102-103: LGTM!Also applies to: 138-139
When the OAuth token refresh endpoint itself returns an explicit invalidation message (e.g. Microsoft/Outlook SSO tokens being revoked server-side on first use through the proxy), apply the long cooldown and return 401 directly to the client without rotating to the next account. This covers the second cascade vector reported in issue #495: Account 4 (Outlook) was getting invalidated immediately on the first request because ensureFreshAccessToken was calling queuedRefresh on a token with a missing or short expiry, and the refresh itself triggered Microsoft's session revocation. Rotating to the next account after that would then present another fresh token, cascading the invalidation. The fix adds tokenInvalidationCooldownMs to ensureFreshAccessToken, checks isTokenInvalidationError against the refresh failure message, and when matched: applies the long cooldown, clears session affinity, and returns the auth error to the client instead of continuing the loop.
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
lib/runtime-rotation-proxy.ts (1)
902-903:⚠️ Potential issue | 🟠 Major | ⚡ Quick winmerge sticky and policy boosts additively.
lib/runtime-rotation-proxy.ts:975-978spreadsstickyBoostByAccountoverpolicy?.scoreBoostByAccount, so the sticky boost replaces any existing policy boost for the same account instead of biasing on top of it. that can erase routing-profile scoring in the exact path this change is trying to stabilize. please sum overlapping boosts before callinggetCurrentOrNextForFamilyHybrid, and add a vitest that covers overlapping policy + sticky boosts intest/runtime-rotation-proxy.test.ts.suggested fix
+ const mergedScoreBoostByAccount: Record<number, number> = { + ...(policy?.scoreBoostByAccount ?? {}), + }; + for (const [accountIndex, boost] of Object.entries( + stickyBoostByAccount ?? {}, + )) { + const index = Number(accountIndex); + mergedScoreBoostByAccount[index] = + (mergedScoreBoostByAccount[index] ?? 0) + boost; + } const selected = accountManager.getCurrentOrNextForFamilyHybrid(family, model, { - scoreBoostByAccount: { - ...(policy?.scoreBoostByAccount ?? {}), - ...(stickyBoostByAccount ?? {}), - }, + scoreBoostByAccount: mergedScoreBoostByAccount, });as per coding guidelines
lib/**:focus on auth rotation, windows filesystem io, and concurrency. verify every change cites affected tests (vitest) and that new queues handle EBUSY/429 scenarios.Also applies to: 975-978
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/runtime-rotation-proxy.ts` around lines 902 - 903, The sticky boost map (stickyBoostByAccount) is currently spread over policy?.scoreBoostByAccount which overwrites policy boosts instead of adding; update the logic that prepares the combined boosts (before calling getCurrentOrNextForFamilyHybrid) to sum values for matching account IDs (e.g., iterate keys and add stickyBoostByAccount[id] to policy.scoreBoostByAccount[id] or vice versa) so overlapping boosts are additive, and add a vitest in test/runtime-rotation-proxy.test.ts that constructs a policy with scoreBoostByAccount and a non-empty stickyBoostByAccount for the same account to assert the resulting boost passed into getCurrentOrNextForFamilyHybrid is the sum.test/runtime-rotation-proxy.test.ts (3)
1843-1868:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winverify session affinity clearing when invalidation is detected.
past review requested verifying that
forgetSessionis called when token invalidation is detected (impl atlib/runtime-rotation-proxy.ts:1677). current test doesn't send a request withmetadata.session_id, so session affinity clearing isn't verified.add a second request with
metadata: { session_id: "session-invalidated" }before the invalidation response, then verify that a subsequent request with the same session_id does not stick to the invalidated account.🧪 add session affinity clearing verification
const proxy = await startProxy({ accountManager, fetchImpl }); + + // establish session affinity to acc_1 + const firstBody = { model: "gpt-5-codex", metadata: { session_id: "session-inv" } }; + await (await postResponses(proxy, firstBody)).text(); + expect(calls).toHaveLength(1); - const response = await postResponses(proxy, { model: "gpt-5-codex" }); + // trigger invalidation for acc_1 + const response = await postResponses(proxy, firstBody); expect(response.status).toBe(HTTP_STATUS.UNAUTHORIZED); - expect(calls).toHaveLength(1); + expect(calls).toHaveLength(2); expect(calls[0]?.headers.get(OPENAI_HEADERS.ACCOUNT_ID)).toBe("acc_1"); + expect(calls[1]?.headers.get(OPENAI_HEADERS.ACCOUNT_ID)).toBe("acc_1"); expect(accountManager.getAccountByIndex(0)?.cooldownReason).toBe("auth-failure"); const coolingDownUntil = accountManager.getAccountByIndex(0)?.coolingDownUntil ?? 0; expect(coolingDownUntil).toBeGreaterThan(now + 250_000); expect(coolingDownUntil).toBeLessThan(now + 350_000); - expect(proxy.getStatus().rotations).toBe(0); + + // verify session affinity was cleared: next request should pick acc_2 + const thirdFetch = createRecordingFetch(() => textEventStream("data: recovered\n\n")); + // swap fetchImpl to allow success on acc_2 + const proxyWithRecovery = await startProxy({ accountManager, fetchImpl: thirdFetch.fetchImpl }); + const afterClear = await postResponses(proxyWithRecovery, firstBody); + expect(afterClear.status).toBe(HTTP_STATUS.OK); + expect(thirdFetch.calls[0]?.headers.get(OPENAI_HEADERS.ACCOUNT_ID)).toBe("acc_2");as per coding guidelines test/**: "demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/runtime-rotation-proxy.test.ts` around lines 1843 - 1868, The test must exercise session-affinity clearing: send an initial request that includes metadata.session_id = "session-invalidated" (using postResponses) so the proxy binds that session to acc_1, then trigger the invalidation response as currently done and assert that lib/runtime-rotation-proxy.ts's forgetSession behavior runs (i.e., a subsequent postResponses call with metadata.session_id = "session-invalidated" does NOT reuse the invalidated account id "acc_1" and instead gets a different account or causes a new assignment); update assertions around calls/cookies to confirm the second request’s OPENAI_HEADERS.ACCOUNT_ID is not "acc_1" (or that accountManager no longer returns the session mapping), and keep the existing checks for 401, cooldownReason, coolingDownUntil, and rotations.
1945-1964: 🧹 Nitpick | 🔵 Trivial | ⚖️ Poor tradeoffconsider testing rotation after sticky window expires.
current test verifies that requests within the
minRotationIntervalMswindow stick to the last served account. it doesn't verify that rotation can occur after the window expires.a complete test would use
vi.useFakeTimers()to advance time past the 60s window and verify that a third request rotates toacc_2.this is optional since the sticky-boost logic is straightforward, but it would provide more complete coverage of the window boundary behavior.
🧪 add sticky window expiry verification
it("rotates after minRotationIntervalMs window expires", async () => { vi.useFakeTimers(); vi.stubEnv("CODEX_AUTH_MIN_ROTATION_INTERVAL_MS", "60000"); try { const now = Date.now(); const accountManager = new AccountManager(undefined, createStorage(now, 2)); const { calls, fetchImpl } = createRecordingFetch(() => textEventStream("data: ok\n\n"), ); const proxy = await startProxy({ accountManager, fetchImpl }); await (await postResponses(proxy, { model: "gpt-5-codex" })).text(); vi.advanceTimersByTime(30_000); // within window await (await postResponses(proxy, { model: "gpt-5-codex" })).text(); vi.advanceTimersByTime(40_000); // past window (30+40=70s) // force rotation by making acc_1 unavailable accountManager.getAccountByIndex(0)?.markAccountCoolingDown(10_000, "rate-limit"); await (await postResponses(proxy, { model: "gpt-5-codex" })).text(); expect(calls[0]?.headers.get(OPENAI_HEADERS.ACCOUNT_ID)).toBe("acc_1"); expect(calls[1]?.headers.get(OPENAI_HEADERS.ACCOUNT_ID)).toBe("acc_1"); expect(calls[2]?.headers.get(OPENAI_HEADERS.ACCOUNT_ID)).toBe("acc_2"); } finally { vi.useRealTimers(); vi.unstubAllEnvs(); } });as per coding guidelines test/**: "demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/runtime-rotation-proxy.test.ts` around lines 1945 - 1964, Add a complementary test that verifies rotation after the minRotationIntervalMs sticky window expires: use vi.useFakeTimers(), vi.stubEnv("CODEX_AUTH_MIN_ROTATION_INTERVAL_MS","60000"), create AccountManager(...) and proxy via startProxy with createRecordingFetch, then send a first request, vi.advanceTimersByTime(30_000), send a second request (still within window), vi.advanceTimersByTime(40_000) to move past the 60s window, mark the first account cooling down via accountManager.getAccountByIndex(0)?.markAccountCoolingDown(...), send a third request via postResponses(...), and assert calls[0/1/2] headers use OPENAI_HEADERS.ACCOUNT_ID equal to "acc_1","acc_1","acc_2" respectively; finally restore timers and env with vi.useRealTimers() and vi.unstubAllEnvs().
1843-1984:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winmissing concurrent invalidation race test flagged in past review.
past review comment (lines 1843-1897) requested: "simulate concurrent invalidation races by firing multiple postResponses(proxy, ...) in parallel against a fetchImpl that returns the invalidation response for the first N concurrent calls and ensure the account is cooled once and rotations/call counts match expected behavior."
this was marked "✅ addressed in commit b354205" but the test is not present. concurrent invalidation could expose race conditions in:
- cooldown duration (could a generic-401 handler overwrite the 5-min invalidation cooldown with 30s?)
forgetSessionbeing called multiple times- multiple
token_invalidatedevents recordedexisting test at
test/runtime-rotation-proxy.test.ts:1213-1265covers concurrent deactivation (402) but not concurrent 401 invalidation.🧪 add concurrent invalidation race test
it("handles concurrent invalidation errors on same account without race", async () => { const now = Date.now(); const accountManager = new AccountManager(undefined, createStorage(now, 1)); const recordFailureSpy = vi.spyOn(accountManager, "recordFailure"); let invalidationCalls = 0; let releaseInvalidationCalls: (() => void) | null = null; const allInvalidationCallsArrived = new Promise<void>((resolve) => { releaseInvalidationCalls = resolve; }); const invalidationBody = JSON.stringify({ error: { message: "invalidated oauth token for user" }, }); const { calls, fetchImpl } = createRecordingFetch(async (call) => { if (call.headers.get(OPENAI_HEADERS.ACCOUNT_ID) === "acc_1") { invalidationCalls += 1; if (invalidationCalls === 2) releaseInvalidationCalls?.(); await allInvalidationCallsArrived; return new Response(invalidationBody, { status: HTTP_STATUS.UNAUTHORIZED, headers: { "content-type": "application/json" }, }); } return textEventStream("data: unreachable\n\n"); }); const proxy = await startProxy({ accountManager, fetchImpl }); const responses = await Promise.all([ postResponses(proxy, { model: "gpt-5-codex" }), postResponses(proxy, { model: "gpt-5-codex" }), ]); expect(responses.map((r) => r.status)).toEqual([ HTTP_STATUS.UNAUTHORIZED, HTTP_STATUS.UNAUTHORIZED, ]); expect(calls).toHaveLength(2); expect(calls.map((c) => c.headers.get(OPENAI_HEADERS.ACCOUNT_ID))).toEqual([ "acc_1", "acc_1", ]); // verify account is cooled once with long invalidation cooldown const account = accountManager.getAccountByIndex(0); expect(account?.cooldownReason).toBe("auth-failure"); const coolingDownUntil = account?.coolingDownUntil ?? 0; expect(coolingDownUntil).toBeGreaterThan(now + 250_000); // ~5min expect(coolingDownUntil).toBeLessThan(now + 350_000); // verify recordFailure called once or twice (depending on dedup), not more expect(recordFailureSpy.mock.calls.length).toBeLessThanOrEqual(2); expect(proxy.getStatus().rotations).toBe(0); });as per coding guidelines test/**: "demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/runtime-rotation-proxy.test.ts` around lines 1843 - 1984, Add a new test that simulates concurrent 401 token-invalidation responses: instantiate AccountManager(createStorage(now, 1)), spy on accountManager.recordFailure, use createRecordingFetch to return the invalidation JSON response for multiple simultaneous calls to postResponses(proxy, ...), block the fetchImpl until both requests arrive (using a promise) so they race, then assert both responses are 401, the fetch calls used the same accountId header (OPENAI_HEADERS.ACCOUNT_ID "acc_1"), the account's cooldownReason is "auth-failure" and coolingDownUntil is ~5min from now, recordFailure was not invoked an unbounded number of times (<=2), and proxy.getStatus().rotations remains 0; use startProxy and postResponses helpers and mirror the structure/expectations shown in the provided suggested test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/runtime-rotation-proxy.ts`:
- Around line 737-748: Two concurrent auth-failure paths can race and shorten an
existing long "invalidated" cooldown; modify the logic around
accountManager.markAccountCoolingDown (calls from the refresh error path that
use isTokenInvalidationError, tokenInvalidationCooldownMs, and
DEFAULT_AUTH_FAILURE_COOLDOWN_MS at both locations) so cooldown updates are
monotonic: before calling markAccountCoolingDown either (A) update
markAccountCoolingDown to compare the requested cooldown expiry against the
current stored expiry and only extend it if the new expiry is later (or to
preserve an "invalidated" stronger state), or (B) read the account's current
cooldown/invalidated state and only call markAccountCoolingDown when the new
duration is strictly longer or when setting invalidation to true; apply this
change at both call sites (the refresh error path and the other 1740-1744 site)
and add a vitest regression in test/runtime-rotation-proxy.test.ts that
simulates two in-flight failures racing to ensure the longer
tokenInvalidationCooldownMs wins and shorter cooldowns do not overwrite it.
In `@test/runtime-rotation-proxy.test.ts`:
- Around line 1918-1943: Add an assertion that a previously-bound session_id
loses affinity after a refresh invalidation: before triggering the refresh, bind
a session (use AccountManager and whatever creates/records affinity in the test
harness, e.g., call postResponses with a session_id or use the same helper that
binds sessions to accounts) to confirm it maps to account 0, then run the
refresh-invalidation scenario (the existing refreshAccessToken mock and
postResponses call) and assert that the session affinity table no longer routes
that session to the invalidated account (i.e., subsequent requests with the same
session_id should be routed to a healthy account or cause a rebalance);
reference the refreshAccessToken mock, AccountManager.getAccountByIndex /
AccountManager methods that inspect coolingDownUntil, postResponses, and
startProxy to locate where to add the binding and the post-invalidation
verification.
---
Outside diff comments:
In `@lib/runtime-rotation-proxy.ts`:
- Around line 902-903: The sticky boost map (stickyBoostByAccount) is currently
spread over policy?.scoreBoostByAccount which overwrites policy boosts instead
of adding; update the logic that prepares the combined boosts (before calling
getCurrentOrNextForFamilyHybrid) to sum values for matching account IDs (e.g.,
iterate keys and add stickyBoostByAccount[id] to policy.scoreBoostByAccount[id]
or vice versa) so overlapping boosts are additive, and add a vitest in
test/runtime-rotation-proxy.test.ts that constructs a policy with
scoreBoostByAccount and a non-empty stickyBoostByAccount for the same account to
assert the resulting boost passed into getCurrentOrNextForFamilyHybrid is the
sum.
In `@test/runtime-rotation-proxy.test.ts`:
- Around line 1843-1868: The test must exercise session-affinity clearing: send
an initial request that includes metadata.session_id = "session-invalidated"
(using postResponses) so the proxy binds that session to acc_1, then trigger the
invalidation response as currently done and assert that
lib/runtime-rotation-proxy.ts's forgetSession behavior runs (i.e., a subsequent
postResponses call with metadata.session_id = "session-invalidated" does NOT
reuse the invalidated account id "acc_1" and instead gets a different account or
causes a new assignment); update assertions around calls/cookies to confirm the
second request’s OPENAI_HEADERS.ACCOUNT_ID is not "acc_1" (or that
accountManager no longer returns the session mapping), and keep the existing
checks for 401, cooldownReason, coolingDownUntil, and rotations.
- Around line 1945-1964: Add a complementary test that verifies rotation after
the minRotationIntervalMs sticky window expires: use vi.useFakeTimers(),
vi.stubEnv("CODEX_AUTH_MIN_ROTATION_INTERVAL_MS","60000"), create
AccountManager(...) and proxy via startProxy with createRecordingFetch, then
send a first request, vi.advanceTimersByTime(30_000), send a second request
(still within window), vi.advanceTimersByTime(40_000) to move past the 60s
window, mark the first account cooling down via
accountManager.getAccountByIndex(0)?.markAccountCoolingDown(...), send a third
request via postResponses(...), and assert calls[0/1/2] headers use
OPENAI_HEADERS.ACCOUNT_ID equal to "acc_1","acc_1","acc_2" respectively; finally
restore timers and env with vi.useRealTimers() and vi.unstubAllEnvs().
- Around line 1843-1984: Add a new test that simulates concurrent 401
token-invalidation responses: instantiate AccountManager(createStorage(now, 1)),
spy on accountManager.recordFailure, use createRecordingFetch to return the
invalidation JSON response for multiple simultaneous calls to
postResponses(proxy, ...), block the fetchImpl until both requests arrive (using
a promise) so they race, then assert both responses are 401, the fetch calls
used the same accountId header (OPENAI_HEADERS.ACCOUNT_ID "acc_1"), the
account's cooldownReason is "auth-failure" and coolingDownUntil is ~5min from
now, recordFailure was not invoked an unbounded number of times (<=2), and
proxy.getStatus().rotations remains 0; use startProxy and postResponses helpers
and mirror the structure/expectations shown in the provided suggested test.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 43e52085-87f3-48eb-ba5e-c38150d9c964
📒 Files selected for processing (2)
lib/runtime-rotation-proxy.tstest/runtime-rotation-proxy.test.ts
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Greptile Review
🧰 Additional context used
📓 Path-based instructions (10)
**/*.{ts,tsx,js,mjs}
📄 CodeRabbit inference engine (AGENTS.md)
Use ESM only (
"type": "module"), Node >= 18
Files:
test/runtime-rotation-proxy.test.tslib/runtime-rotation-proxy.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use
as any,@ts-ignore, or@ts-expect-errorTypeScript assertions
Files:
test/runtime-rotation-proxy.test.tslib/runtime-rotation-proxy.ts
{**/scripts/**/*.js,**/test/**/*.ts}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use bare recursive delete logic in Windows-sensitive scripts/tests without retry handling for transient
EBUSY/EPERM/ENOTEMPTYerrors
Files:
test/runtime-rotation-proxy.test.ts
test/**/*.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
test/**/*.test.ts: Vitest globals (describe,it,expect) are enabled and should be used without explicit imports
Maintain 80% coverage threshold across statements, branches, functions, and lines
UseremoveWithRetryfor Windows filesystem cleanup instead of barefs.rmto handle EBUSY/EPERM/ENOTEMPTY backoff
Use source files in tests, not compileddist/files; test the source directly
Do not skip tests without justification; include rationale if a test must be skipped
Relax ESLint rules for test files as specified ineslint.config.js
Files:
test/runtime-rotation-proxy.test.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (README.md)
**/*.{ts,tsx,js,jsx}: Implement default-on runtime Responses rotation for request-bearing forwarded Codex CLI/app sessions, with opt-out support viaCODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY=0
Store project-scoped accounts under~/.codex/multi-auth/projects/<project-key>/openai-codex-accounts.jsonfor repo-specific workflows
Support environment variable overrides for configuration includingCODEX_MULTI_AUTH_DIR,CODEX_MODE,CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY,CODEX_TUI_COLOR_PROFILE, and others as documented
Implement account health checks withcodex-multi-auth checkcommand that validates saved account credentials and state
Implement quota forecasting and budget guards to prevent exhausting account quota within a session
Record local usage in a ledger at~/.codex/multi-auth/usage/usage-ledger.jsonlfor per-project tracking and reporting
Implement bounded outbound request budget so one prompt cannot walk the full account pool indefinitely
Trigger short cooldown instead of continuing aggressive rotation when repeated cross-account 5xx bursts are detected
Stagger proactive token refresh to reduce background refresh bursts across the account pool
Expose recent runtime request metrics incodex-multi-auth statustext output and machine-readable metrics incodex-multi-auth report --json
Make OAuth callback listen on port 1455 for login flows
Support device authorization flow viacodex-multi-auth login --device-authas an alternate to browser-based OAuth for headless environments
Implement dashboard hotkeys including Up/Down for navigation, Enter for selection, 1-9 for quick switch, / for search, ? for help, Q to back, S to set account, R to refresh, E to enable/disable, and D to delete
Support named local pool backup export with filename prompt in the Settings > Experimental menu
Implementcodex-multi-auth doctor --fixcommand to diagnose and apply the safest fixes for storage or account state issues
Implementcodex-multi-auth fix --dry-run...
Files:
test/runtime-rotation-proxy.test.tslib/runtime-rotation-proxy.ts
test/**
⚙️ CodeRabbit configuration file
tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.
Files:
test/runtime-rotation-proxy.test.ts
**/lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (AGENTS.md)
**/lib/runtime-rotation-proxy.ts: Keep runtime rotation default-on behavior aligned with explicit release and migration documentation
Do not expose account emails or tokens in runtime proxy client response headers or logs
Files:
lib/runtime-rotation-proxy.ts
lib/**/*.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/**/*.ts: All public exports should flow throughlib/index.tsor documented package subpaths
Never import fromdist/in source tests or library code
Never suppress type errors
Files:
lib/runtime-rotation-proxy.ts
lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/runtime-rotation-proxy.ts: Runtime rotation code must preserve pass-through semantics except for auth/provider headers that intentionally change
Runtime proxy client-facing headers must not expose account emails or tokens
Runtime rotation should fail open to normal official Codex forwarding when startup helpers are unavailable
Never add account emails/tokens to runtime proxy client responses
Files:
lib/runtime-rotation-proxy.ts
lib/**
⚙️ CodeRabbit configuration file
focus on auth rotation, windows filesystem IO, and concurrency. verify every change cites affected tests (vitest) and that new queues handle EBUSY/429 scenarios. check for logging that leaks tokens or emails.
Files:
lib/runtime-rotation-proxy.ts
🔇 Additional comments (1)
lib/runtime-rotation-proxy.ts (1)
1821-1824: still unresolved: make the sticky window sliding.
lib/runtime-rotation-proxy.ts:1821-1824still only refresheslastGlobalSwitchAtwhen the account index changes. if the same account serves again inside the window, the next request can rotate as soon as the original timestamp expires even though that account just served successfully. the mocked-clock regression intest/runtime-rotation-proxy.test.tsis still needed here.as per coding guidelines
lib/**:focus on auth rotation, windows filesystem io, and concurrency. verify every change cites affected tests (vitest) and that new queues handle EBUSY/429 scenarios.
ndycode
left a comment
ndycode
left a comment
There was a problem hiding this comment.
All 4 CR actionable comments addressed across commits b354205 and b4209bb:
- Phrase provenance comment added above TOKEN_INVALIDATION_PHRASES
- cooldown duration assertions in both invalidation and generic-401 tests
- Edge case tests: empty body, HTML body, refresh-phase invalidation
- All tests pass (53/53 proxy, 4049+ full suite), TypeScript clean
@coderabbitai review
…tests - Sliding sticky window: update lastGlobalSwitchAt on every successful serve, not only when the account index changes. Previously a request at t=61s would rotate even if the same account served at t=55s (only 6s ago) because the anchor stayed at t=0. Now the window slides so the 60s interval is measured from the last actual serve. - Monotonic auth-failure cooldown: replace direct markAccountCoolingDown calls in auth-failure paths with applyMonotonicAuthCooldown helper that only extends the cooldown if the proposed deadline exceeds the existing one. Two concurrent requests can no longer race so that a generic 401 truncates a longer invalidation cooldown already written by a parallel request. - Regression test: three requests across the window boundary using vi.useFakeTimers to verify the sliding behavior. - Session affinity test: refresh-invalidation test sends session_id and verifies subsequent request with same session routes to healthy account.
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/runtime-rotation-proxy.ts`:
- Around line 659-672: Two direct calls to markAccountCoolingDown in
lib/runtime-rotation-proxy.ts (the refresh-commit failure and missing-account-id
branches around the previous calls at lines near 785-789 and 1547-1551) must be
routed through applyMonotonicAuthCooldown(accountManager, account, cooldownMs)
so the monotonic "only extend" semantics are preserved; replace those
markAccountCoolingDown(...) invocations with calls to applyMonotonicAuthCooldown
using the same accountManager, account, and cooldownMs values. Add a vitest
regression that races each of those branches against the invalidation paths
already covered by test/runtime-rotation-proxy.test.ts (the blocks at ~1918-1991
and ~1444-1565), reproducing the concurrent sequence to assert the cooldown
remains the longer value; follow existing test patterns and ensure the new tests
account for filesystem/concurrency error handling (EBUSY/429) as per lib/**
guidelines.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 84923871-1a4d-40d9-a4a3-964b1a73ed6b
📒 Files selected for processing (2)
lib/runtime-rotation-proxy.tstest/runtime-rotation-proxy.test.ts
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Greptile Review
🧰 Additional context used
📓 Path-based instructions (10)
**/*.{ts,tsx,js,mjs}
📄 CodeRabbit inference engine (AGENTS.md)
Use ESM only (
"type": "module"), Node >= 18
Files:
test/runtime-rotation-proxy.test.tslib/runtime-rotation-proxy.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use
as any,@ts-ignore, or@ts-expect-errorTypeScript assertions
Files:
test/runtime-rotation-proxy.test.tslib/runtime-rotation-proxy.ts
{**/scripts/**/*.js,**/test/**/*.ts}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use bare recursive delete logic in Windows-sensitive scripts/tests without retry handling for transient
EBUSY/EPERM/ENOTEMPTYerrors
Files:
test/runtime-rotation-proxy.test.ts
test/**/*.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
test/**/*.test.ts: Vitest globals (describe,it,expect) are enabled and should be used without explicit imports
Maintain 80% coverage threshold across statements, branches, functions, and lines
UseremoveWithRetryfor Windows filesystem cleanup instead of barefs.rmto handle EBUSY/EPERM/ENOTEMPTY backoff
Use source files in tests, not compileddist/files; test the source directly
Do not skip tests without justification; include rationale if a test must be skipped
Relax ESLint rules for test files as specified ineslint.config.js
Files:
test/runtime-rotation-proxy.test.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (README.md)
**/*.{ts,tsx,js,jsx}: Implement default-on runtime Responses rotation for request-bearing forwarded Codex CLI/app sessions, with opt-out support viaCODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY=0
Store project-scoped accounts under~/.codex/multi-auth/projects/<project-key>/openai-codex-accounts.jsonfor repo-specific workflows
Support environment variable overrides for configuration includingCODEX_MULTI_AUTH_DIR,CODEX_MODE,CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY,CODEX_TUI_COLOR_PROFILE, and others as documented
Implement account health checks withcodex-multi-auth checkcommand that validates saved account credentials and state
Implement quota forecasting and budget guards to prevent exhausting account quota within a session
Record local usage in a ledger at~/.codex/multi-auth/usage/usage-ledger.jsonlfor per-project tracking and reporting
Implement bounded outbound request budget so one prompt cannot walk the full account pool indefinitely
Trigger short cooldown instead of continuing aggressive rotation when repeated cross-account 5xx bursts are detected
Stagger proactive token refresh to reduce background refresh bursts across the account pool
Expose recent runtime request metrics incodex-multi-auth statustext output and machine-readable metrics incodex-multi-auth report --json
Make OAuth callback listen on port 1455 for login flows
Support device authorization flow viacodex-multi-auth login --device-authas an alternate to browser-based OAuth for headless environments
Implement dashboard hotkeys including Up/Down for navigation, Enter for selection, 1-9 for quick switch, / for search, ? for help, Q to back, S to set account, R to refresh, E to enable/disable, and D to delete
Support named local pool backup export with filename prompt in the Settings > Experimental menu
Implementcodex-multi-auth doctor --fixcommand to diagnose and apply the safest fixes for storage or account state issues
Implementcodex-multi-auth fix --dry-run...
Files:
test/runtime-rotation-proxy.test.tslib/runtime-rotation-proxy.ts
test/**
⚙️ CodeRabbit configuration file
tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.
Files:
test/runtime-rotation-proxy.test.ts
**/lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (AGENTS.md)
**/lib/runtime-rotation-proxy.ts: Keep runtime rotation default-on behavior aligned with explicit release and migration documentation
Do not expose account emails or tokens in runtime proxy client response headers or logs
Files:
lib/runtime-rotation-proxy.ts
lib/**/*.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/**/*.ts: All public exports should flow throughlib/index.tsor documented package subpaths
Never import fromdist/in source tests or library code
Never suppress type errors
Files:
lib/runtime-rotation-proxy.ts
lib/runtime-rotation-proxy.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/runtime-rotation-proxy.ts: Runtime rotation code must preserve pass-through semantics except for auth/provider headers that intentionally change
Runtime proxy client-facing headers must not expose account emails or tokens
Runtime rotation should fail open to normal official Codex forwarding when startup helpers are unavailable
Never add account emails/tokens to runtime proxy client responses
Files:
lib/runtime-rotation-proxy.ts
lib/**
⚙️ CodeRabbit configuration file
focus on auth rotation, windows filesystem IO, and concurrency. verify every change cites affected tests (vitest) and that new queues handle EBUSY/429 scenarios. check for logging that leaks tokens or emails.
Files:
lib/runtime-rotation-proxy.ts
🔇 Additional comments (1)
test/runtime-rotation-proxy.test.ts (1)
1934-1952: LGTM!Also applies to: 1955-1991
…t SSO warning - troubleshooting.md: add rows for progressive OAuth token invalidation and Microsoft/Outlook SSO immediate invalidation, with recovery steps and CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS guidance. - configuration.md: add CODEX_AUTH_MIN_ROTATION_INTERVAL_MS and CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS to the env var table. Expand the Runtime Rotation Proxy section with an "Anti-abuse protection" note explaining the two mitigations added in issue #495 (token-invalidation detection + rotation-rate throttle), and a Microsoft/Outlook SSO note for accounts that get invalidated on first proxy request.
…path (#497) * fix(runtime): emit consistent token_invalidated body on upstream-401 path The upstream-401 invalidation path forwarded the raw OpenAI body, so clients keying off error.code saw a stable "token_invalidated" code on the refresh-failure path but not here (Greptile P3 on #496). - Add buildTokenInvalidationBody(): wraps both paths in the same { error: { message, code: "token_invalidated" } } shape, preserving the upstream message when present and falling back to a stable message for non-JSON bodies (no markup echoed to clients). - Force content-type: application/json and drop content-length/-encoding via responseHeadersForClient since the body is rewritten. - Document that applyMonotonicAuthCooldown intentionally uses Date.now() (not the injected now()) to match the markAccountCoolingDown/nowMs() write domain, guarding against a regression that would defeat the monotonic cooldown race protection. - Strengthen existing upstream-401 and html-body tests to assert the client contract (code + message). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(runtime): route refresh-failure 401 through shared invalidation builder + unit-test branches Addresses two Greptile P2 comments on this PR: - The refresh-failure invalidation exit still hardcoded the body literals the new constants were meant to centralise, reopening the exact drift risk this PR closes. Route it through buildTokenInvalidationBody(""), which yields the same { error: { message: <fallback>, code: "token_invalidated" } } so both paths can no longer diverge. - Export buildTokenInvalidationBody and add 8 unit tests covering all message-extraction branches: empty input, top-level message, nested error.message, top-level-wins priority, blank-top-level -> nested fallback, non-JSON body, and no-usable-message fallback. Full suite: 4062 tests pass (4054 + 8). typecheck/lint/build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Prerelease that ships the cascade OAuth token-invalidation fix from issue #495 to npm under the `beta` dist-tag: the 401 handler now detects explicit token-invalidation responses and returns them to the client instead of rotating through every account (the rotation itself was tripping OpenAI's anti-abuse detection and invalidating accounts in sequence). Invalidated accounts get a monotonic 5-minute cooldown, session affinity is cleared, and both invalidation exit paths emit a consistent token_invalidated error body. Also adds a configurable minRotationIntervalMs sticky window. Carries forward multi-workspace support (beta.1) and the pinned-account 503 diagnostic (beta.0). Stable v2.1.13 will land once the issue #486 root cause is identified and patched. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
Summary
Fixes #495 — rotation gateway triggers mass OAuth token invalidation across accounts.
"invalidated oauth token","authentication token has been invalidated", etc.), distinguishing them from generic expired-token 401s.CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MSortokenInvalidationCooldownMsin config) vs the generic 30-second auth-failure cooldown.Generic 401 responses (expired tokens, wrong credentials) continue to rotate as before.
Test plan
tsc --noEmit)tokenInvalidationCooldownMsadded to Zod schema with min(0), allows-zero and rejects-string coverage addednote: greptile review for oc-chatgpt-multi-auth. cite files like
lib/foo.ts:123. confirm regression tests + windows concurrency/token redaction coverage.Greptile Summary
adds token-invalidation detection to the 401 handler (both the upstream response path and the refresh-endpoint failure path), stops cascade rotation by returning the 401 directly to the client, and applies a configurable 5-minute cooldown. a second feature (
minRotationIntervalMs) adds a sticky score boost toward the last-served account to reduce how often a different OAuth token is presented from the same IP within the configured window.TOKEN_INVALIDATION_PHRASES+isTokenInvalidationError) andapplyMonotonicAuthCooldownaddress the cascade failure from issue Rotation gateway triggers mass OAuth token invalidation across accounts #495; both refresh-failure and upstream-401 paths clear session affinity on detection.minRotationIntervalMssliding anchor (lastGlobalSwitchAtupdated every successful serve, not only on account change) prevents the sticky window from expiring prematurely between back-to-back requests on the same account.Confidence Score: 5/5
safe to merge; the invalidation-detection and cascade-stop logic is correct and well-tested, with no regressions to the generic-401 rotation path.
the two new 401 exit paths are both exercised by dedicated tests, the monotonic-cooldown helper correctly gates writes on the live account state, and the sliding-anchor test guards the key regression. the findings here are design-level observations that don't affect correctness in any current code path.
lib/runtime-rotation-proxy.ts — the two token-invalidation exit paths produce structurally different response bodies, and applyMonotonicAuthCooldown uses Date.now() rather than the proxy's injected now function.
Important Files Changed
Sequence Diagram
sequenceDiagram participant C as Client participant P as Proxy participant AM as AccountManager participant U as Upstream (OpenAI) C->>P: POST /responses rect rgb(220, 240, 220) Note over P: minRotationIntervalMs sticky-boost applied to chooseAccount P->>AM: ensureFreshAccessToken() alt refresh endpoint returns invalidation message AM-->>P: "{ok:false, invalidated:true}" P->>AM: applyMonotonicAuthCooldown(5 min) P->>P: forgetSession(sessionKey) P-->>C: "401 {error:{code:"token_invalidated"}}" Note over P: return — no rotation else refresh succeeds AM-->>P: "{ok:true, accessToken}" P->>U: proxied request alt upstream returns 401 with invalidation body U-->>P: 401 "invalidated oauth token" P->>AM: applyMonotonicAuthCooldown(5 min) P->>P: forgetSession(sessionKey) P-->>C: 401 raw upstream body Note over P: return — no rotation else upstream returns generic 401 U-->>P: 401 "Unauthorized" P->>AM: applyMonotonicAuthCooldown(30 s) P->>P: continue loop → rotate account else upstream returns 200 U-->>P: 200 stream P->>AM: recordSuccess() P->>P: "lastGlobalSwitchAt = now()" P-->>C: 200 streamed response end end endComments Outside Diff (1)
lib/runtime-rotation-proxy.ts, line 1666-1700 (link)markAccountCoolingDownis an unconditional overwrite (account.coolingDownUntil = nowMs() + ms). if two concurrent requests hit the same account and one gets an invalidation 401 while the other gets a generic 401, the generic path (executing after the invalidation path sets 5 min) overwrites it with 30 seconds. the window is narrow but non-zero — a request inflight before invalidation is detected racing against the first detected invalidation response. a "take the maximum" check inmarkAccountCoolingDownwould close this.Prompt To Fix With AI
Prompt To Fix All With AI
Reviews (6): Last reviewed commit: "docs: document token-invalidation anti-a..." | Re-trigger Greptile