fix(create): default the LiteLLM harness model to a keyless bedrock model - #2168
Conversation
|
Claude Security Review: no high-confidence findings. (run) |
There was a problem hiding this comment.
AgentCore Harness Review
Verdict: Looks good
Two well-scoped changes, both with clear reasoning and tests:
-
transcript.tsx— unindexedcontentBlockDeltaevents for text/reasoning now fold into block 0 (instead of being dropped), while unindexedtoolUse/toolResultdeltas are still ignored.contentBlockStopfalls back to 0 rather than -1. The reordering of the guard (delta === undefinedfirst, then the kind-specific handling with an innercontentBlockIndex === undefinedshort-circuit beforetoolUse/toolResult) is correct: text/reasoning get the?? 0treatment while tool deltas keep requiring a real index. -
create/index.ts+screen.tsx— the wizard now sources per-provider defaults from the exportedHARNESS_DEFAULT_MODEL_IDS, eliminating the wizard/flag-path drift. The litellm default flips fromanthropic/claude-sonnet-4-5(needs an API key the wizard never prompts for) tobedrock/${DEFAULT_HARNESS_MODEL.modelId}(keyless), which matches the wizard's UX of leavingapiKeyArn/apiBaseoptional. The newcreate.screen.test.tsxcase explicitly asserts thebedrock/prefix so a future revert can't slip through, andproject.test.tswas updated to match.
One minor behavioral note (not a blocker): with the new fallback, an unindexed contentBlockStop now settles whatever occupies block 0. In a mixed-indexing stream that could target the wrong block, but real providers emit either all-indexed or all-unindexed events, so this is the pragmatic choice.
No telemetry needed — this is a rendering bugfix plus a default-value correction.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## refactor #2168 +/- ##
============================================
- Coverage 97.14% 97.14% -0.01%
============================================
Files 519 519
Lines 35492 35486 -6
============================================
- Hits 34478 34472 -6
Misses 1014 1014 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Claude Security Review: no high-confidence findings. (run) |
4005451 to
d515573
Compare
|
Claude Security Review: no high-confidence findings. (run) |
…odel The lite_llm default was anthropic/claude-sonnet-4-5, which routes LiteLLM at Anthropic directly and needs an Anthropic API key. Because apiKeyArn is optional for LiteLLM, a default project deployed READY and then failed its first invoke with a LiteLLM AuthenticationError. A bedrock/ model ID routes LiteLLM at Bedrock, which it signs with SigV4 using the harness execution role, so no API key is required. Derive the default from DEFAULT_HARNESS_MODEL so the bedrock and lite_llm defaults cannot drift.
The wizard declared its own copy of every provider default, so changing the flag path's lite_llm default left the TUI still pre-filling anthropic/claude-sonnet-4-5. That value is form state rather than placeholder text, so pressing through the wizard deployed a harness that reached READY and then failed its first invoke with a LiteLLM AuthenticationError. Export HARNESS_DEFAULT_MODEL_IDS and read it for both the pre-filled value and the placeholder, leaving MODEL_PROVIDERS to describe only labels. The two paths can no longer drift.
d515573 to
0e920b4
Compare
|
Claude Security Review: no high-confidence findings. (run) |
MODEL_PROVIDERS no longer declares a per-provider default, so the wizard and the flag path read one table and cannot disagree. project.test.ts already pins the value, which is the only remaining way to regress it.
|
Claude Security Review: no high-confidence findings. (run) |
Follow-up to #2150, which fixed the harness-first create defects the #2146 bug bash turned up. This one fixes the LiteLLM default that same bug bash surfaced.
The default LiteLLM model required a key the CLI never asks for
Before:
HARNESS_DEFAULT_MODEL_IDS.lite_llmwasanthropic/claude-sonnet-4-5. That prefix routes LiteLLM at Anthropic directly, which needs an Anthropic API key — butapiKeyArnis optional for LiteLLM, so nothing stopped you. A defaultproject create --model-provider lite_llmdeployed toREADYand then failed its first invoke withlitellm.AuthenticationError: Missing Anthropic API Key.After: the default derives from
DEFAULT_HARNESS_MODEL, givingbedrock/global.anthropic.claude-sonnet-4-6. Abedrock/prefix routes LiteLLM at Bedrock, which it signs with SigV4 from the harness execution role — no API key, noapiKeyArn. Deriving rather than hardcoding also stops thebedrockandlite_llmdefaults drifting apart.Confirmed live in us-east-1:
apiKeyArnanthropic/claude-sonnet-4-5AuthenticationErrorbedrock/global.anthropic.claude-sonnet-4-6end_turn, nonzerooutputTokensbedrock/us.amazon.nova-lite-v1:0end_turn, nonzerooutputTokensThis matches the documented contract — the Harness models guide states LiteLLM models using the
bedrock/prefix use the execution role and need no API key, while direct providers requireapiKeyArn.anthropic/...keeps requiring a key; onlybedrock/...is keyless.The wizard shipped its own copy of that default
Found while fixing the above, and it would have silently undone it for TUI users.
create/screen.tsxdeclareddefaultModelIdper provider, duplicating all four values includinglite_llm: "anthropic/claude-sonnet-4-5". Fixing the flag path alone left the wizard pre-filling the broken id — and that value is form state, not placeholder text, so pressing enter through the wizard deployed it:HARNESS_DEFAULT_MODEL_IDSis now exported and supplies both the pre-filled value and the placeholder;MODEL_PROVIDERSdescribes only labels. The flag path and the wizard read one table, so they cannot disagree again.Testing
bun test: 2735 pass, 0 fail.typecheck,lint:check,format:checkclean.project.test.tsalready pins the flag path's litellm default, and after this change the wizard reads the same table, so that one assertion covers both paths. No new wizard test:MODEL_PROVIDERSno longer declares a per-provider default, so there is no second literal left to drift.project create --model-provider lite_llmwrites"modelId": "bedrock/global.anthropic.claude-sonnet-4-6", and that harness authenticates and completes. All AWS test resources were deleted.Note on the empty transcript, for anyone who tested this earlier
Until today,
agentcore harness invokerendered no assistant text for alite_llmharness — the callsucceeded with
end_turnand nonzerooutputTokens, but nothing appeared. That was a harness-sidecontract bug, not a CLI one: the LiteLLM provider path omitted
contentBlockIndex, whichinvokeHarness.smithymarks@requiredon all three content-block events, and the CLI correctlyrejects events that lack it.
The harness team had already fixed it (
LoopyRuntimeArtifacts, "Emit contentBlockIndex as it is partof the invoke harness contract") on 2026-08-27. It reached us-east-1 on 2026-09-02 at 16:37Z. Our
capture was taken on 2026-09-01, before the wave arrived, which is why we saw it.
No CLI change is needed for that, and this branch carries none. Mentioning it only because a reviewer
who tried a LiteLLM harness in the last few days may remember an empty transcript.