counterfactual: the replay harness CLI — score a prompt/model variant against historical judgments

[JSONbored]

Problem

Counterfactual sub-epic phase 3. The actual runner: given a fixture set and a judge variant (prompt version + provider/model/effort), replay offline, map outputs through the scoring contract, and print the same comparison report every other backtest uses.

Requirements

⚠️ Required pattern. Pure-core/thin-IO split exactly like backtest-corpus-export: all mapping/aggregation logic in a unit-tested core; the wrapper owns provider IO. Budget + cursor discipline from the design contract (partial runs persist scores + resume). Local ollama is the smoke path; BYOK providers behind explicit flags.

• tsx scripts/counterfactual-replay.ts --fixtures <file> --variant <prompt-version|model spec> [--budget N] [--baseline <scores file>] → runs the variant, scores with scoreBacktest via the contract's classifier mapping, and with --baseline prints compareBacktestScores' Pareto-floored comparison + the shared renderer's report.
• Abstentions counted per the contract; per-fixture raw outputs cached to a local artifacts dir (never committed) so re-scoring is free.
• Never posts anywhere; exit code reflects operational success only, never verdict (the Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138 advisory guarantee).

Deliverables

• Core (100% covered) + wrapper + docs section with a worked ollama example
• A recorded first comparison on the production fixture set posted to this issue

Links & Resources

The sub-epic (parent), the design + assembler issues (blocked-by), scripts/backtest-corpus-export.ts (the split to mirror)

Boundaries

Offline only; spend-capped; no GitHub writes; artifacts stay local.

maintainer-only — model spend + judgment-layer evaluation.

[linear-code]

JSO-5705

[JSONbored]

Recorded first comparison on the production fixture set

Setup (fully reproducible): corpus = all 460 raw-context production cases for ai_consensus_defect (export checksum 86c3b4b55cae…); seeded deterministic sample counterfactual-replay-v1:production-first-comparison, 40 fixtures; judge model qwen3:8b on the ORB box's ollama (temperature 0); raw outputs cached under the artifacts dir.

Variant A — the canonical production judge prompt (review-prompt-v1, 4,963 chars, extracted via print-review-prompt.ts):

Metric	Value
Scored / abstained	40 / 0
True positives	0
False positives	0
Precision	N/A (never flagged)
Recall	0

Variant B — the minimal judge prompt (the harness default, 3 lines):

Metric	Value
Scored / abstained	40 / 0
True positives	5
False positives	7
Precision	0.417
Recall	0.278

Pareto verdict (A → B): improved — no axis regressed (recall 0 → 0.278; precision axis excluded on the N/A side, per the unknown-stays-unknown discipline).

Reading, with the honest caveats: on an 8B local smoke model, the full production system prompt suppresses flagging entirely — 40/40 clean parses, zero blockers ever emitted — while a three-line prompt at least engages the task. That is precisely the class of prompt×model interaction the harness exists to make visible before it ships anywhere: the long prompt isn't "safer" on this model, it's inert. Caveats: this is the smoke-path model, not a production judge; 40 fixtures is the seeded bounded sample, not the full set; and the labels are realized human reversal outcomes on decisions the OLD pipeline surfaced (selection effects apply). The number that matters here is the mechanism: end-to-end replay, zero abstentions, zero accounting drift, budget 119k/250k (A) + 73k/250k (B) neurons.