Skip to content

fix(selfhost): cap total wall-clock/subprocess spend when a CLI provider times out repeatedly in one review call #3927

Description

@JSONbored

Context

runWorkersOpinion (src/services/ai-review.ts) retries every model in the self-host CLI fallback chain (Claude Code / Codex, via createSelfHostAi) up to 3 times regardless of failure type, including subscription_cli_timeout -- a full SIGKILL after the effort-based timeout (e.g. 180s at medium effort, up to 600s at max effort, per the timeouts c7073949 corrected). There is no check that short-circuits a retry when the SAME failure (a timeout) just occurred -- every attempt unconditionally re-spawns the full CLI subprocess. The identical unbounded attempt < 3 pattern also drives runDualAiTieBreakJudgeCall, invoked up to 2x on reviewer disagreement.

Worst case for a single review call with a 2-provider fallback chain, both consistently timing out at max effort: 2 models x 3 attempts x 600s = up to 3600s of subprocess time and subscription-quota burn for zero output, per review dispatch.

This is a distinct bug from the one c7073949 (#3747) fixed: that commit capped the OUTER cross-sweep-tick re-dispatch of the SAME PR/SHA (via the audit_events ledger), bounding how many TIMES a stuck PR gets re-swept. This inner loop -- how many times ONE dispatch retries before giving up on a single model -- was never touched, before or after that commit, and traces back unchanged to the original dual-AI feature (dab0ce47).

Only reachable through the self-host Node entrypoint (src/server.ts's createSelfHostAi) when a self-hoster configures a CLI-based AI_PROVIDER (Claude Code / Codex) -- the hosted Cloudflare Worker has no AI binding and never executes this path, so this is a self-host-only cost/reliability gap, not a hosted-production one.

Requirements

  • Short-circuit remaining retries within a single runWorkersOpinion/runDualAiTieBreakJudgeCall call once a subscription_cli_timeout (or another clearly-non-transient failure class) has occurred once, rather than blindly retrying the same failure up to 3 times.
  • Keep retrying transient/ambiguous failure classes (e.g. a single dropped connection) as today.

Deliverables

  • Updated retry logic in src/services/ai-review.ts's runWorkersOpinion and runDualAiTieBreakJudgeCall.
  • Tests proving a repeated timeout on the same model stops retrying that model after the first timeout, while a transient error still gets its full retry budget.

Acceptance criteria

  • A self-host deployment whose configured CLI provider is genuinely timing out on a large/complex diff no longer burns 3x the effort-timeout per model per dispatch.
  • No change in behavior for non-timeout, transient failure classes.

Parent: #1667

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions