test(model-onboarding): cover chat, tool calls, file processing and IS activity tools

[denispetre]

What

The model-onboarding testcase now runs four capability payloads for every get_chat_model code path, all rolled into the single success boolean:

payload	what it checks
simple	plain ainvoke → non-empty completion
tools	full tool-calling round trip (get_weather): the tool is executed and the final answer must use its result
files/<name>	the coded file-processing agent (Studio Web "Clone as Coded Agent" of the low-code FileProcessingAgent): production ReAct stack (create_agent + Analyze Files internal tool), test file uploaded as a real platform attachment. When the org forbids attachment creation (403 for the CI principal), degrades to the direct multimodal invoke with an explicit marker
is_tools	call-only Activity-tool check: bind the coded copy's real IS tools (Slack Send Message to Channel, Outlook 365 Send Email) and assert the model returns a well-formed tool call — known bound name + required args. Tools are never executed, nothing is sent, no connections needed

API flavor as input

paths entries (workflow input or input.json) accept two forms, freely mixed:

• registry keys: azure_responses, azure_chat_completions, vertex, bedrock_converse, bedrock_invoke, anthropic_sdk
• vendor_type:api_flavor pairs passed straight to get_chat_model (exact enum values), e.g. awsbedrock:converse, openai:responses, vertexai:generate-content, awsbedrock:AnthropicMessages. vendor: alone autodetects the flavor.

Every payload — including the coded agents — runs with the model built for that path/flavor.

Design

• Agents live under src/agents/, one subfolder per coded agent (src/agents/README.md).
• Each agent.py is the Studio Web generated main.py (downloaded via uip agent file get) with exactly one structural change: the hardcoded llm = get_chat_model('gpt-5.4', ...) becomes build_graph(llm, ...) / build_tools(...) so the testcase injects the model from model_spec — model + API flavor stay configurable; run.sh / input.json / dispatch contract unchanged.
• Provenance (alpha org llm_gateway_automated_testing): FileProcessingAgent (solution 1823d2d6-…, coded copy e7b8598c-…), IsToolsAgent (solution 925fc5db-…, coded copy 0959f23b-…).
• The vertex registry path pins vendor_type + api_flavor=GENERATE_CONTENT explicitly (was autodetect-only).
• Workflow fix: the dispatch input builder now uses python3 — the uv container does not ship jq (first live dispatch failed with exit 127).

Live proof (dispatched via model_onboarding.yml on this branch)

Run 30250006856 — registry path, and run 30277990899 — flavor-pair path (openai:chat-completions), both success on alpha with gpt-5.2-2025-12-11:

simple:      ✓
tools:       ✓
files/*:     ✓ (direct multimodal; attachments unavailable in CI org)
is_tools:    ✓ (Send_Message_to_Channel, Send_Email)

Supersedes #1007. No src/ changes — test + workflow only.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[denispetre]

Live proof via the dispatch workflow — model_onboarding.yml invoked on this branch (gpt-5.2-2025-12-11, azure_responses, alpha): run 30250006856 → success.

simple:      ✓
tools:       ✓
files/image: ✓ (direct multimodal; attachments unavailable: EnrichedException)
files/pdf:   ✓ (direct multimodal; attachments unavailable: EnrichedException)
is_tools:    ✓ (Send_Message_to_Channel, Send_Email)

Dispatching also surfaced and fixed three real issues now in this PR: the workflow's input builder used jq, which the uv container doesn't ship (exit 127 → rewritten in python3); a wrong get_httpx_client_kwargs import path; and the CI principal gets 403 on Orchestrator attachment creation, so the files cells degrade to the direct multimodal call with an explicit marker (orgs with attachment permissions get the full generated-agent + Analyze Files path).

[denispetre]

Added: forgiving string coercion in get_chat_model (commit aa36733, folded in from #1011 per review preference).

vendor_type / api_flavor / routing_mode strings are now matched case-insensitively against enum names and values with -/_ interchangeable — so workflow inputs like paths="awsbedrock:anthropic_messages" or "AWSBedrock:Converse" resolve instead of failing on exact spelling; unknown strings raise an immediate ValueError listing valid values. 23 new tests in tests/chat/test_chat_model_factory_coercion.py; enum/None inputs pass through unchanged.

[denispetre]

Removed the get_chat_model string-coercion commit per review (aa36733 dropped, force-pushed — the earlier comment about it is obsolete). The PR no longer touches src/; the vendor_type:api_flavor paths input works as strict pass-through with exact enum values, as documented in the workflow's paths description.

[tudormatei1]

i think f-string nested inside the other f-string with the same quotes is a problem. looks like this pattern became legal in pyrthon 3.12 but pyproject says requires-python = ">=3.11" so i suppose if you try running it on 3.11 it might break. I suppose CI uses a newer version so thats why it goes green? If u plan on changing it it also happens on line 372, 380, 392 and 404. link to this: https://peps.python.org/pep-0701/

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[tudormatei1]

these seem quite flaky no? refusal can be stated in many ways, i dont think its reliable to test like this.

my idea to test would be to have a simple token in the pdf like a code lets say: 12345, now theres no way the model can just guess it or say the file is a placeholder, it either read it or it didnt.

and then you can just do an exact match and drop the marker list entirely (cause you just check for code).

for images its a bit different i think cause its quite unfair to make it read a code from the image, so maybe use something that is less probable to be guessed (in this case dog is the first thing that comes to mind so it might just say dog without looking at the image)

one example: have a purple duck image and ask what color it is

[tudormatei1]

im not really sure about this but does the file name in this case the "dog.jpg" create a bias? maybe the model cannot open the file and just says dog.jpg because of the filename? i believe the model has direct access to the file name inside:

return [
        SystemMessage(content=enhanced_system_prompt),
        HumanMessage(content=interpolate_legacy_message("""{{prompt}}

{{fileIn}}""", state.model_dump())),
    ]

this can be solved with the comment from below i guess

[tudormatei1]

im not sure about this but i though we talked yesterday to only include one simple tool call like you had with get_weather_data (or whatever it was called)?

we had that discussion with either keeping one simple tool call test like that or have a really difficult and complicated one. right now we only test the analyse file tool maybe thats enough?