feat(project): add project add evaluator code-based - #2144
Draft
jariy17 wants to merge 18 commits into
Draft
Conversation
added 18 commits
August 27, 2026 21:20
…ms, failures/sessions)
Adds `agentcore eval ab-test config-bundle run` — the first create in the ab-test family. Runs an A/B test between two config-bundle versions on one gateway. --control / --treatment / --gateway-filter each accept inline JSON, file://<path>, or - (stdin) via SourceResolver (same shape as online-eval create's --filters). Core provisions an IAM execution role when --role-arn is omitted (mirrors online-eval create + retryWhileRolePropagates), and rolls the role back if CreateABTest fails. Validation is server-side (gateway READY, bundles, online-eval enabled iff enableOnCreate) — the CLI resolves ids to ARNs and surfaces the service's 4xx cleanly. Note: --runtime dropped (not a CreateABTest field) and deviates from doc.
- Reject control/treatment only when the (config-bundle, bundle-version) pair is identical, not on version-string collision across different bundles. - Retry CreateABTest on data-plane AccessDenied (403), not just the control-plane role-not-propagated phrasing, so a freshly provisioned role that is mid-propagation is retried. - Extract accountId via a throwing helper instead of a silent '*' fallback. - --treatment-weight must be an integer. - Add unit tests for the execution-role module (name cap, trust + inline policy, create vs reuse).
Match online-eval create's flag ergonomics: replace the boolean opt-out --disable-on-create with a value flag --enable-on-create <true|false> (default true). Input carries enableOnCreate?: boolean; core sends enableOnCreate ?? true.
Drop the duplicate retryWhileRoleUnassumable I added; broaden the existing retryWhileRolePropagates to also retry on data-plane AccessDenied/403 (how a freshly-provisioned role surfaces on CreateABTest) and reuse it. Removes the cross-file name collision with harness's helper.
Record a self-contained config-bundle run golden in account 685197708687 (matches the config-bundle fixtures): create a bundle (v1 -> v2), a paused online-eval on a real runtime, then run the paused A/B test; afterAll tears down the ab-test, online-eval, provisioned role, and bundle. Broaden retryWhileRolePropagates to also retry ValidationException 'unable to assume the provided IAM role' -- how CreateABTest surfaces a freshly provisioned role mid-propagation. Drop the TestCoreClient happy-path mapping tests the golden now covers; keep the local validation/error cases.
… file Merge ab-test.write.test.tsx + ab-test.create.test.tsx into a single ab-test.test.tsx (mirrors batch-evaluation.test.tsx): hierarchy, get/list/ pause/resume/stop/delete happy paths, and every unhappy path in one place -- missing --id (now covers get, which regressed), Core-error surfacing per op (not-found / invalid-transition / not-stopped), and config-bundle run validation (required flags, malformed + mis-shaped JSON, identical variants, weight bounds). Golden fixture files unchanged.
Adds `agentcore eval ab-test target-based run` — an A/B test between two
gateway targets and their per-variant online evaluations. --control/--treatment
take {gateway-target, online-eval} JSON (inline/file/stdin via SourceResolver);
variants use variantConfiguration.target and a perVariantOnlineEvaluationConfig.
Extract a shared EvalClient.createABTest helper (GetGateway -> account, role
provision + AccessDenied/assume retry + rollback) and drive both config-bundle
and target-based create through it, removing the duplicated role/retry block.
Consolidated ab-test.test.tsx covers the target-based hierarchy, validation
(required flags, mis-shaped JSON, identical targets), and flag->request
mapping. Golden fixture deferred (needs a gateway with two wired targets +
two online-evals).
…run' into refactor
Declarative code-based evaluators via projects. Mode is inferred from flags (mirrors CodeBasedConfigSchema managed XOR external): --lambda-arn -> external (BYO Lambda) --metric <library.Metric> -> managed 3P (deepeval/autoevals), scaffolded neither -> managed empty stub you fill in Scaffolds app/<name>/ from ported evaluator templates (python/deepeval/autoevals lambda), hardcodes codeLocation, and auto-wires additionalPolicies= [execution-role-policy.json]. Also enables `project remove evaluator`.
…aluators Runtimes, harnesses, and evaluators all scaffold into app/<name>, but the duplicate-name guard is per-resource-type and the tree write happens outside the rollback try/catch. An evaluator whose name matches an existing runtime/ harness dir (or a leftover from a removed evaluator) threw a raw 'File already exists' mid-write and orphaned partial files. Fail up front with a clear InputValidationError when app/<name> already exists.
…or code-based evaluators - Reject a namespaced/multi-dot metric class (e.g. deepeval.metrics.Faithfulness) that would render invalid Python; require a single class identifier. - --model is Bedrock-only: accept a bare model id / inference-profile-or- foundation-model ARN, optionally prefixed with bedrock/, validated via isValidBedrockModelId (same forms the llm-as-a-judge handler accepts). Non-Bedrock or slashless values now error instead of being silently dropped (deepeval) or passed to the wrong client (autoevals). - autoevals template prefixes bedrock/ for litellm routing now that Model is the bare id.
Print notes after add: the empty stub returns Pass for every session until implemented, and managed evaluators are scaffolded but not yet provisioned by 'project deploy' (no CDK/L3 support). External (--lambda-arn) prints neither.
agentcore-devx-automation
Bot
added
agentcore-harness-reviewing
![agentcore-devx-automation[bot]](https://avatars.githubusercontent.com/u/2232217?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
AgentCore Harness Review
Verdict: Looks good
Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:
- Empty stub silently returns
Passfor every session (verified againstsrc/assets/evaluators/python-lambda/lambda_function.py, which returnslabel="Pass"). - Managed code-based evaluators aren't yet provisioned by
project deploy.
Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.
Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.
Nothing blocking.
agentcore-devx-automation
Bot
removed
the
agentcore-harness-reviewing
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## refactor #2144 +/- ##
============================================
- Coverage 97.25% 97.13% -0.13%
============================================
Files 472 480 +8
Lines 29018 29819 +801
============================================
+ Hits 28221 28964 +743
- Misses 797 855 +58 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
agentcore project add evaluator code-based— declarative code-based evaluators via projects, plus enablesagentcore project remove evaluator. Follows theproject addpattern from #2034 / #2004 / #2035 / #1998 / #2037.No selector flag — mode is inferred from what you pass (mirrors
CodeBasedConfigSchema'smanaged XOR external):--metric <library.Metric>@custom_code_based_evaluator()stub you fill in--lambda-arn <arn>Managed scaffolds
app/<name>/(lambda_function.py+pyproject.toml+execution-role-policy.json, ported from the old CLI), hardcodescodeLocation, and auto-wiresadditionalPolicies: ["execution-role-policy.json"].Commits
268305a7feat — the command + 3 templates + remove enabled8eb5227fix — guardapp/<name>collisions (cross-resource name clash no longer orphans files)821b337dfix — validate--metricclass + require a Bedrock--model(no more invalid Python / silently-dropped model)303c8db6fix — echo inferred mode; warn on always-Pass empty stub and on not-yet-deployed managed evaluatorsTesting
bun run buildOK ·bun test src/handlers/project src/core/project→ 536 pass / 0 fail. All four CLI paths smoke-tested in a scratch project; generated Pythonast.parse-verified for deepeval + autoevals.Known gap (why this is a draft)
project deploydoes not yet provision project evaluators — nothing insrc/assets/cdk/**(or@aws/agentcore-cdk) readsspec.evaluators.cdk synthemits zero evaluator resources. This PR is the CLI/authoring half; the CDK-L3 provisioning + evaluator code-packaging is follow-up work in the L3 repo. Commit303c8db6surfaces this ataddtime instead of failing silently.Full bug-bash report (cases + fixes + design decisions): shared separately.
🤖 Draft — CLI/authoring layer only; do not merge until L3 provisioning lands.