test(ai-proxy): stop failing the LLM suite on a provider outage

[Scra3]

LLM Integration Tests (ai-proxy) fails the whole suite when OpenAI has an outage, blocking PRs that have nothing to do with ai-proxy.

What happened

It failed twice in a row on #1815, while main and two other feature branches were green — and a run on another branch that succeeded landed between my two failures, so it is correlated with time, not with the branch.

Same cause both times:

❌ 1 OpenAI model(s) failed tool support: gpt-3.5-turbo-16k
   error: 'AIProviderUnavailableError: OpenAI server error (HTTP 500):
           500 The server had an error while processing your request.'

Why the test is wrong here

The suite exists to answer one question: does this model support tool calls? A provider 5xx carries no signal about that. It is also not actionable through the mechanism this test feeds — per packages/ai-proxy/CLAUDE.md, a model that fails here is meant to be added to the supported-models.ts denylist, and denylisting a model because the provider had a bad minute would encode an outage as a permanent capability claim.

The existing isInfrastructureError branch already separates rate limits, auth and network errors from real capability failures. Provider 5xx simply was not in that list, so it fell through to "failed tool support".

The change

Models whose call raises AIProviderUnavailableError are collected separately and reported as skipped with a warning, instead of counted as failures. A genuine capability failure still fails the test.

To keep the skip from hiding a real problem, the test now also asserts that at least one model was actually verified — so a total provider outage fails loudly rather than passing green having checked nothing.

Verification

• yarn workspace @forestadmin/ai-proxy test — 469 tests, 39 suites, green.
• test/llm.integration.test.ts run against the real APIs — 40 tests green, so OpenAI is healthy again and the check still does its job end to end.

One honest limitation: because every model was available during that run, the new skip branch was not itself exercised. It is matched on the String(error) prefix observed verbatim in the CI logs above.

Note

Skip provider unavailability errors in the LLM integration test suite

Updates the all models should support tool calls test in llm.integration.test.ts to distinguish AIProviderUnavailableError failures from real test failures. Models that return this error are collected separately, logged as a warning, and excluded from the failure count. A final assertion checks that at least one model was actually tested, preventing a total outage from producing a false pass.

^{Macroscope summarized efafeb7.}

[qltysh]

Coverage Impact

This PR will not change total coverage.

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.