Problem
A provider auth failure and a provider quota exhaustion are the same shape in every surface we expose:
- Metrics:
loopover_ai_provider_failures_total{provider} has no reason dimension.
- Logs:
selfhost_ai_provider_failed carries the raw error string only.
So claude_code_error_401 and claude_code_error_429 are indistinguishable without grepping raw logs and decoding the status by hand. The two need opposite first responses:
|
401 / 403 |
429 |
| Meaning |
credential rejected |
credential fine, plan limit spent |
| Fix |
rotate the credential |
wait for the window, or raise the limit |
| Self-heals? |
No — degrades every review until someone acts |
Yes |
This is not hypothetical. A burst of claude_code_error_429 on the hosted ORB box read as "the token is dead" and prompted a credential rotation that could not possibly have helped — the credential was valid the whole time.
Scope
- Classify a provider failure into an actionable reason (
credential_invalid, quota_exhausted, timeout, not_configured, other), matching on the structured error shapes the module itself throws rather than free-text provider prose.
- Expose it as a
reason label on a new counter (not a new label on the existing failures counter, which shipped alert rules and dashboards already query) and as a reason field on the failure log.
- Two alert rules with the severity split that encodes the difference: a rejected credential pages, an exhausted quota warns, and each runbook says explicitly whether rotating helps.
Notes
Anything unrecognised must stay other. A confidently wrong label sends an operator to the wrong runbook, which is worse than an honest unknown.
Follow-up to #9543.
Problem
A provider auth failure and a provider quota exhaustion are the same shape in every surface we expose:
loopover_ai_provider_failures_total{provider}has no reason dimension.selfhost_ai_provider_failedcarries the rawerrorstring only.So
claude_code_error_401andclaude_code_error_429are indistinguishable without grepping raw logs and decoding the status by hand. The two need opposite first responses:This is not hypothetical. A burst of
claude_code_error_429on the hosted ORB box read as "the token is dead" and prompted a credential rotation that could not possibly have helped — the credential was valid the whole time.Scope
credential_invalid,quota_exhausted,timeout,not_configured,other), matching on the structured error shapes the module itself throws rather than free-text provider prose.reasonlabel on a new counter (not a new label on the existing failures counter, which shipped alert rules and dashboards already query) and as areasonfield on the failure log.Notes
Anything unrecognised must stay
other. A confidently wrong label sends an operator to the wrong runbook, which is worse than an honest unknown.Follow-up to #9543.