fix(trace-repair): declare the control that screens a row, and refuse a task whose grader is a coin flip

[drewstone]

Two defects made the TB-Repair denominator unreadable. Both are repaired here, and both repairs have tests that fail against the behaviour on main.

1. The control that screened a row was never declared

Admission condition 3 asks whether a row is rescued by continuing from the recorded end state with no intervention. Both milestone runs answered it under a control pinned to zero model calls.

A rollout that makes no model call executes no command, so the container a control arm graded held the same bytes condition 2 had already graded as failing. The condition could only fire on a grader that disagreed with itself. It screened nothing, and nothing in the artifacts said which control had screened a row — a reader had to open the runner's source.

Measured against origin/main@0d6620b, handing admitRow the exact evidence shape scripts/tb-repair-milestone1.ts produced:

zero-step control, default criteria -> admitted: true
decision keys: admitted,row
policy recorded on the decision? false
control passed 2/3 under a zero-step control -> rejection: no-fix-control-passed

The last line is the wrong reading: no-fix-control-passed means "the row is repairable by continuing alone", and nothing continued.

The repair. defineControlPolicy makes the control a hashed declaration — id, step budget, scaffold, model, command timeout — so a step budget cannot move under an unchanged label, and a policy that calls no model must record model: null. assertControlCalibrated refuses both contradictions at configuration time, before a verdict exists for any row:

controlScreening	requires	a control pass means
enforced (default)	stepBudget >= 1	no-fix-control-passed — repairable by continuing alone
declared-inert	stepBudget == 0	control-passed-on-identical-state — the task's grader disagreed with itself

Either pairing the other way round raises UncalibratedControlError. Every decision now carries an AdmissionScreeningRecord — policy, digest, screening mode, task name, measured oracle flip rate — on rejected decisions as well as admitted ones. AdmissionRowVerdict carries the same for rows the executing pre-pass excluded before any control ran, and deltaRepair raises a control-cannot-rescue threat when any row was screened by an inert control, so a control rate of zero cannot be read as a screen.

2. A task graded by wall clock was carrying ground truth

largest-eigenval asserts dt < ref_dt at tests/test_outputs.py:111 — the only wall-clock assertion across the four task suites the milestones sampled.

certify-task-oracle.sh gains phase C: re-grade byte-identical containers N times, idle and under CPU contention, and refuse to certify a task whose verdict is not stable. Two design points are load-bearing and were both found by measurement, not by reasoning:

• Counting per assertion, not per suite. The first implementation counted the pass/fail reward and certified largest-eigenval — a false green I nearly shipped. The reward is a conjunction; a suite whose per-parameter timing assertions each flip still returns one steady verdict at a state far from the threshold. Phase C now reads pytest's own -rA summary, which keeps the parameter, so test_speedup[6] is its own unit. The CTRF report the suites also write collapses parametrised tests to their base name and is not used.
• Contention is cheap, not decisive. A suite that times a candidate against a reference measured in the same run is self-normalising, so uniform load moves both terms together. It is kept because it catches the cases that are not self-normalising, and it is documented as such.

The verdict rule lives in oracleDeterminism and is applied through scripts/tb-oracle-determinism.ts, so a certification run and a campaign cannot drift apart about what a stable oracle is. No node means no verdict, not a weaker one. Skipping phase C yields CERTIFIED_UNCHECKED_DETERMINISM, never CERTIFIED.

Before and after, same task, same image, same machine

origin/main's certifier, extracted with git show and run unmodified:

=== largest-eigenval | image=alexgshaw/largest-eigenval:20251031 ===
phaseA_unsolved_reward=0   phaseB_solved_reward=1
VERDICT=CERTIFIED
exit 0

The same task on this branch:

phaseC_flip_bp=3750 replicates=16 state=unstable
VERDICT=NONDETERMINISTIC_ORACLE(flip=37.50%,n=16)
exit 1

Measured, at the same image digests the milestones ran against

16 replicates per task: 8 on the untouched image and 8 after the reference solution, half of each under CPU contention.

task	image digest	replicates	units flipped	worst per-unit flip	verdict
password-recovery	sha256:15fcd84b…	16	0	0 %	CERTIFIED
sanitize-git-repo	sha256:4b5234da…	16	0	0 %	CERTIFIED
count-dataset-tokens	sha256:9c513c4b…	16	0	0 %	CERTIFIED
largest-eigenval	sha256:cb9e3e3b…	16	8	37.5 %	NONDETERMINISTIC_ORACLE

All eight flipped units are test_speedup[size] parameters on the untouched image, where /app/eigen.py holds the reference implementation the assertion compares against:

test_speedup[10] 5/8   test_speedup[3] 5/8   test_speedup[6] 5/8
test_speedup[4]  6/8   test_speedup[5] 6/8   test_speedup[9] 6/8
test_speedup[8]  7/8   test_speedup[2] 1/8

The whole-suite reward on that state never moved: 0 of 8. An earlier run of the same certification put the worst unit at 50 %; a flip rate that is itself unstable is what a coin flip looks like. A separate probe held /app/eigen.py at sha256 bbd5be97… across four gradings and saw the per-size failure count swing 1, 4, 3, 3 of 9.

The measurements are checked in as benchmarks/trace-repair/task-oracles.json. It stores replicates, not verdicts: parseTaskOracleRegistry re-derives the verdict on read by the rule a campaign enforces, so the file cannot declare a task stable and a reviewer can recount it.

Admission reads that certification before it opens a container. A task with no entry leaves as task-oracle-uncertified; an unstable one as task-oracle-nondeterministic. The two are different facts — the check has not run, versus it ran and the task failed it.

The honest consequence for Milestone 1

Nothing is rewritten. milestone1.json is untouched; docs/trace-repair-admission.md records what it measured.

run	rows	rows from largest-eigenval	what moves
milestone 1	20 evaluated, 17 admitted	2 evaluated, 0 admitted	both exclusions were labelled no-fix-control-passed, claiming the row was repairable by continuing alone; nothing continued
milestone 2	48 evaluated, 43 admitted	5 evaluated, 4 admitted	four admitted rows come from a task whose grader is not a function of the state

Which rows were affected: the two largest-eigenval rows, largest-eigenval__cxEtBTx (no-fix control passed 2/3) and largest-eigenval__4GTN8MQ (1/3).

Which verdicts were affected: both, and only both. Both were exclusions, and both remain exclusions under the repaired code — but for the reason that is true (task-oracle-nondeterministic) rather than one that is not.

Does the headline separation survive excluding the timing-graded task? Yes, and it never depended on it. The 17 admitted rows contain zero largest-eigenval rows, so dropping the task changes nothing:

oracle-fix   n=17  mean=+0.353    excluding largest-eigenval: n=17  mean=+0.353
inert-probe  n=17  mean= 0.000    excluding largest-eigenval: n=17  mean= 0.000

What does not survive is any reading of milestone 1 as evidence that condition 3 screened anything. Across both milestones the control-pass rate was 3 of 3 on largest-eigenval's 7 evaluations and 0 of 61 on the three tasks that certify stable, under a control that made zero model calls. One row, largest-eigenval__4GTN8MQ, was excluded by milestone 1 on a control pass of 1/3 and admitted by milestone 2 on a control pass of 0/3 — same row, same control, opposite verdicts.

Milestone 2's admitted set is contaminated: 4 of its 43 admitted rows come from the timing-graded task. Its numbers are conditional on that and are not restated here.

Unconditional continuation has still never been given a model call on this corpus. Nothing in this PR claims it rescues nothing; it makes that a question a run can answer, by requiring a control that could.

Tests that fail against the current behaviour

• tests/trace-repair/control-policy.test.ts — admitRow throws UncalibratedControlError on a zero-step control under default criteria (on main it admits the row); a control pass under an inert control rejects as control-passed-on-identical-state and not as a rescue; every decision, admitted or rejected, carries the policy digest (on main a decision has only admitted and row).
• tests/trace-repair/oracle-determinism.test.ts — an assertion that flipped while the suite reward never moved is caught (the case that produced the false green); a missing assertion counts as a flip of the suite's shape; reward-granularity fallback is recorded as coarser rather than substituted; a certification measured on another task is refused.
• src/trace-repair/admission.test.ts — runAdmission refuses a zero-step policy before any control call, and excludes uncertified and nondeterministic tasks with zero replay and zero control calls.
• tests/trace-repair/delta-repair.test.ts — control-cannot-rescue is raised and the stronger admission-conditions-on-control-failure is withheld.
• benchmarks/trace-repair/tools/certify-task-oracle.test.mjs — runs the script's --self-test, which checks the flip and verdict tables. Replacing the rule with "always stable" (which is what the tool did before) fails 3 cases, so the self-test is not vacuous.

Proof

pnpm vitest run tests/trace-repair src/trace-repair benchmarks/trace-repair
  Test Files  18 passed (18)       Tests  264 passed (264)

pnpm typecheck                     clean
tsc --noEmit -p tsconfig.script.json   clean
biome check src                    no fixes applied
git merge-tree --write-tree origin/main HEAD   merges cleanly

Full suite: 5255/5261 pass. tests/contract-self-improve.test.ts times out at 10 s under load and fails identically on unmodified origin/main@0d6620b in a fresh worktree; it passes in isolation on this branch. Pre-existing, not introduced here.

[tangletools]

✅ Auto-approved drewstone PR — 4bed2514

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-10T08:53:04Z}

[tangletools]

⚪ Value Audit — audit-incomplete

Verdict	audit-incomplete
Concerns	1 (1 low)
Heuristic	0.0s
Duplication	0.0s
Interrogation	210.3s (2 bridge agents)
Total	210.3s

💰 Value — error

value agent never ran: the CLI bridge refused admission (no model was started).

• Model: opencode/deepseek/deepseek-v4-pro
• Bridge attempts: 4
• Bridge error: opencode/kimi-for-coding/k2p7: Bridge returned 503: bridge at capacity (queue_timeout, lane=None): active=20/20 queued=1/48 — no model was started

🎯 Usefulness — error

usefulness agent never ran: the CLI bridge refused admission (no model was started).

• Model: opencode/deepseek/deepseek-v4-pro
• Bridge attempts: 4
• Bridge error: opencode/zai-coding-plan/glm-5.2: Bridge returned 503: bridge at capacity (queue_timeout, lane=None): active=20/20 queued=2/48 — no model was started

🔎 Heuristic Signals

🟡 Cruft: magic number added scripts/tb-repair-milestone1.ts

• commandTimeoutSeconds: Math.ceil(STEP_TIMEOUT_MS / 1000),

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260810T091138Z}

[tangletools]

⚠️ Review Incomplete — a80dd1d0

At least one required reviewer lane failed closed. No approval or request-changes review was published. This is a reviewer run failure, not a PR quality score.

Trigger a fresh review on the current PR head.

_{tangletools · 2026-08-10T10:11:24Z}

[tangletools]

✅ Auto-approved drewstone PR — a80dd1d0

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-10T10:27:20Z}