feat(eval): ondemand simulate — replay a dataset, evaluate synchronously

[jariy17]

What

eval ondemand simulate — replay a dataset against a runtime, then evaluate the sessions synchronously, client-side (scores print inline). The on-demand twin of batch-evaluation simulate.

Pipeline: invokeDataset (replay) → getTracesForAgent (CloudWatch) → evaluate (Evaluate API).

Follows the batch-evaluation simulate pattern

• Same invoke flags + --ingestion-wait-ms (default 180000; 0 to skip)
• Same SIGINT abort + invokeDataset composition
• Refuses when 0 invoked, naming the first failure (failures[0])
• Output adds sessions[] (exampleId ↔ sessionId) + failures[]

Differs by design: no --name / --kms-key-arn (no async job); output is inline scores, not a job id.

Output

{
  "sessionsEvaluated": 1,
  "results": [ { "evaluatorId": "Builtin.Helpfulness", "value": 0.83, "label": "Very Helpful", "explanation": "…", "tokenUsage": {…} } ],
  "examplesInvoked": 1,
  "examplesFailed": 0,
  "sessions": [ { "exampleId": "greet", "sessionId": "…" } ],
  "failures": []
}

Tests (batch pattern)

• Edge tests — ondemand.test.tsx (TestCoreClient): required-flag validation, refuse-when-nothing-invoked, --ingestion-wait-ms passthrough.
• Fixture golden — ondemand.fixture.test.tsx: real invoke → CloudWatch traces → Evaluate, recorded against a live agent, replayed offline via matchGolden. Deterministic via the injected newSessionId seam plus a now() clock seam — on-demand's trace-query window is otherwise Date.now()-based, so its StartQuery fixture key would drift between record and replay (batch has no client query; evaluate pins an explicit --start-time/--end-time).
• Live-validated across 10 dataset scenarios against a real agent in the EXPLORE account (all real Builtin.Helpfulness scores; edge/negative paths handled).

Notes

• Builtin.Helpfulness ignores ground-truth refs (ignoredReferenceInputFields); the handler still forwards them (adapter exercised).
• Rebased onto current refactor (was stacked on the pre-merge batch-simulate work).

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.16%. Comparing base (f2b73d6) to head (0cfacfe).

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2071      +/-   ##
============================================
+ Coverage     97.14%   97.16%   +0.02%     
============================================
  Files           494      495       +1     
  Lines         32550    32676     +126     
============================================
+ Hits          31622    31751     +129     
+ Misses          928      925       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[nborges-aws]

this and --header should be marked sensitive