Agent-task eval: rich vs POSIX tool surfaces

[phernandez]

Part of the POSIX tool surface + memory evals tracker.

Scope

An agent-in-the-loop eval where the tool surface is the provider axis: the same agent, model, budget, and BM project completes memory tasks with rich-tools (today's MCP surface) vs posix-tools (the flagged surface). Measures what retrieval-only evals cannot: task success, total tokens (in+out per turn), tool-call count, latency.

Task sources

• The shipped skills, treated as eval specs: memory-continue ("pick up where we left off on X"), memory-curate (find orphan notes), memory-metadata-search (frontmatter queries), memory-tasks.
• SPEC-47's manual demo chain: "how do I write a note" → grep → man page → targeted read.
• A fixed seeded corpus (a snapshot project) so tasks are reproducible.

Grading

• Deterministic where possible: expected permalink found, expected note created at path, expected metadata result set.
• LLM-judge for open-ended outcomes, reusing the benchmarks package's qa/review/rejudge stages and artifact/provenance format.
• Headline metric: tokens per successfully completed task (xAFS framing), with accuracy and tool-call count alongside — never accuracy alone.

Fairness

Same tasks, same model, same max-turn/token budget, same corpus snapshot. Only the tool surface varies. Provider failures explicit in artifacts per the existing contract.

Acceptance

• bm-bench run agent-tasks --provider rich-tools --provider posix-tools produces a side-by-side artifact.
• ≥10 tasks spanning at least 3 skills plus the man-page chain.
• Result: a defensible token/accuracy comparison that decides whether the POSIX surface graduates from the flag.

🤖 Generated with Claude Code

https://claude.ai/code/session_014pmKq6bqCi6Zp6BTHuZjrp

[claude]

Claude finished @phernandez's task in 33s —— View job

---

Triage

Type: Enhancement — new agent-in-the-loop evaluation harness, not a defect.

Component: MCP tools / benchmarks package (bm-bench), plus the SPEC-47 POSIX tool surface flag. No repository cloud label applies.

Priority: Medium — this is evaluation tooling that informs a graduation decision for the POSIX tool surface flag, not a user-facing break.

Complexity: Complex — requires a seeded snapshot corpus, a new agent-tasks provider axis in bm-bench comparing rich-tools vs posix-tools, deterministic + LLM-judge grading reusing the benchmarks package's qa/review/rejudge stages, and ≥10 tasks spanning 3+ skills plus the SPEC-47 man-page chain. Multi-part, spans the eval framework and both tool surfaces.

Duplicates: None. This is a sub-task of the tracker in #1398 ("POSIX tool surface and memory evals (SPEC-47 A/B, BEAM, xAFS)"), alongside sibling issues #1399 (POSIX read-side MCP tools flag), #1400 (BEAM benchmark integration), #1402 (xAFS dataset adapter), #1403 (section indexing), and #1404 (bm CLI POSIX verbs). No overlapping open issue duplicates this scope.

Scope, task sources, grading approach, fairness constraints, and acceptance criteria are already well specified in the issue body — no clarifying questions needed before implementation.