Skip to content

Distinguish a rejected credential from an exhausted quota in provider-failure signals #9549

Description

@JSONbored

Problem

A provider auth failure and a provider quota exhaustion are the same shape in every surface we expose:

  • Metrics: loopover_ai_provider_failures_total{provider} has no reason dimension.
  • Logs: selfhost_ai_provider_failed carries the raw error string only.

So claude_code_error_401 and claude_code_error_429 are indistinguishable without grepping raw logs and decoding the status by hand. The two need opposite first responses:

401 / 403 429
Meaning credential rejected credential fine, plan limit spent
Fix rotate the credential wait for the window, or raise the limit
Self-heals? No — degrades every review until someone acts Yes

This is not hypothetical. A burst of claude_code_error_429 on the hosted ORB box read as "the token is dead" and prompted a credential rotation that could not possibly have helped — the credential was valid the whole time.

Scope

  • Classify a provider failure into an actionable reason (credential_invalid, quota_exhausted, timeout, not_configured, other), matching on the structured error shapes the module itself throws rather than free-text provider prose.
  • Expose it as a reason label on a new counter (not a new label on the existing failures counter, which shipped alert rules and dashboards already query) and as a reason field on the failure log.
  • Two alert rules with the severity split that encodes the difference: a rejected credential pages, an exhausted quota warns, and each runbook says explicitly whether rotating helps.

Notes

Anything unrecognised must stay other. A confidently wrong label sends an operator to the wrong runbook, which is worse than an honest unknown.

Follow-up to #9543.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions