Skip to content

[Bug]: Usage-limit auto-resume gives up permanently when the continuation is rejected right after the reset #33708

Open

Description

@brazyptian96

Before submitting

Area

apps/server

Summary

Auto-resume fired on time, but the provider rejected the continuation run with usageLimitExceeded a few seconds after the reported reset. That failure was recorded without a usable future resetAt, so UsageLimitRecoveryWorker never re-armed and the thread stayed failed until I continued it by hand about 2 hours later. One rejected continuation ends recovery for good.

This is separate from #27773. Even when the reset time is recovered (for example via the account/rateLimits/read probe in #27787), the probe may return the window that just expired, or null, at this moment. limitRecoveryCommand rejects any resetAt that is not later than the run's completion time, so it would still return null.

Steps to reproduce

  1. Run a Codex (app-server driver) thread on a ChatGPT-plan account until the 5h window is exhausted. The failure carries resetAt from the limitId: "codex" snapshot.
  2. Arm Resume at reset (auto-resume).
  3. Wait for the reset. The worker dispatches Continue where you left off. about 1–2 s after resetAt.
  4. If the backend still rejects that first request (usageLimitExceeded), recovery stops permanently.

Observed once (T3 Code 0.1.7). This is timing-dependent, and I have not reproduced it on demand.

Timeline (UTC, from statev2.sqlite and the provider event log)

Time Event
22:00:04 account/rateLimits/updated limitId:"codex", primary.usedPercent:100, resetsAt:1794268919 (00:01:59Z). Run fails usage_limit, resetAt:"…T00:01:59.000Z".
22:03:07 limit-arm:…:1794268919000, autoResume:true.
00:02:00.579 limit-resume:… dispatched; continuation run created (+1.6 s after reset).
00:02:06.775 Only rate-limit notification: limitId:"premium", primary:null, secondary:null (ignored by mergeCodexRateLimits).
00:02:06.781 error: usageLimitExceeded, "…try again at 7:01 PM." (server-local time = 00:01Z, already past), willRetry:false.
00:02:06.815 Failure item: class:"usage_limit", resetAt:null. No further limit-arm / limit-resume.
01:59:19 Manual continue runs normally. The next limitId:"codex" snapshot shows primary.usedPercent:0 in a new window.

Relevant code (0.1.7 bundle)

  • UsageLimitRecoveryWorker.limitRecoveryCommand returns null when !thread.usageLimitResetAt, or when resetMs <= latestRunCompletedAt. There is no fallback or retry for a failed continuation.
  • CodexProvider: codexUsageLimitResetAt(rateLimitSnapshot) is the only source of failure.resetAt. The error text's "try again at …" is not used, and here it was already in the past.

Expected behavior

If a run started by limit-resume (usageLimitContinuationOfRunId set) fails again with usage_limit, and there is no reset time later than now, auto-resume should stay armed. It should retry with a bounded backoff (for example +2 min, +5 min, +15 min, then give up and show the existing "retry manually" state), still respecting cancel, settle, snooze, and archive.

A small grace delay after resetAt before dispatching (for example 30–60 s) would also make the first attempt less likely to land before the provider's counters roll over. On its own, though, it doesn't fix the dead end above.

Actual behavior

The thread remains failed with no armed recovery. The UI gives no indication that auto-resume was attempted and abandoned.

Notes

The cause on the provider side (why a request 7 s after the reported reset was still rejected) is inferred, not proven: possibly propagation lag in the backend limiter. The client-side dead end is confirmed from the event store and the bundled source.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions