fix(eval): keep CLI-only commands out of the TUI menu - #2191
Conversation
|
Claude Security Review: no high-confidence findings. (run) |
There was a problem hiding this comment.
AgentCore Harness Review
Verdict: Looks good
Small, focused fix that adds supportedTuiCommands to the eval and eval batch-evaluation routers so CLI-only commands (ondemand, recommendation, evaluate, simulate) stop showing up in the interactive menu. The categorization matches each subhandler's actual state (either no .default(renderTui...) or an explicit empty supportedTuiCommands()), and the pattern mirrors what's already in place for gateway, runtime, identity, etc.
The new tests validate menu membership through the real compiled router with TestCoreClient, consistent with gateway.test.tsx. No excessive mocking, no telemetry surface changed. LGTM.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## refactor #2191 +/- ##
=========================================
Coverage 97.15% 97.15%
=========================================
Files 542 542
Lines 37728 37740 +12
=========================================
+ Hits 36653 36665 +12
Misses 1075 1075 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The eval router and its batch-evaluation subgroup open an interactive TUI on a bare invocation but never restricted which children the menu offers via supportedTuiCommands(), so the menu surfaced commands that have no interactive screen: - eval menu listed ondemand (help-only) and recommendation (no screen) - batch-evaluation menu listed evaluate and simulate (imperative jobs) Selecting one navigated to a route with no screen. Add the allowlists, matching the existing batch-insights / gateway pattern, so only groups and leaves with a real screen appear. Behaviour of the commands themselves is unchanged; this only gates TUI-menu membership.
d55c9b5 to
a3cc129
Compare
|
Claude Security Review: no high-confidence findings. (run) |
Problem
The
evalrouter and itsbatch-evaluationsubgroup open an interactive TUI on a bare invocation (.default(renderTui(...))) but never called.supportedTuiCommands(...), so the menu (RouterScreen→isTuiCommandSupported) offered commands that have no interactive screen:evalmenu listedondemand(help-only default) andrecommendation(no screen).eval batch-evaluationmenu listedevaluateandsimulate(imperative job-starting commands with required flags).Selecting any of them navigated to a route with no screen. Every sibling with the same shape (
batch-insights,ab-test,online-eval,dataset,config-bundle, …) already restricts the menu to("get", "list"); these two were the outliers.Fix
Add the missing allowlists, matching the existing
batch-insights/gatewaypattern:eval→ the 8 groups that actually have an interactive screen (dropsondemand+recommendation).eval batch-evaluation→("get", "list")(dropsevaluate+simulate).Behaviour of the commands themselves is unchanged —
agentcore eval ondemandstill prints help,batch-evaluation simulatestill runs from the CLI. This only gates TUI-menu membership.Tests
Mirror the gateway command-hierarchy suite (
supportsTuihelper +test.eachfor supported vs CLI-only):src/handlers/eval/eval.test.tsx— the 8 screen-backed groups are offered;ondemand+recommendationare hidden.batch-evaluation.test.tsx—get/list(and bare group) offered;evaluate/simulatehidden.bun test src/handlers/eval/→ 377 pass, 0 fail.oxlint+prettierclean on changed files.