refactor(workbench): retire the meaning regex a scenario file hid - #454
Merged
Conversation
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
`FLAGS_MISSING_INPUTS` drove `beh.flags-missing-inputs`, weight 4 — the heaviest
behaviour check in `wizard-enhance-bare`. It carried the founding defect of
`lib/language.ts` exactly: an English pattern requiring a POSITIVE match ("no
transcript", "without a transcript", "cannot detect silence"). A French answer
saying precisely what the check exists to reward could not match it, so the check
failed for a reason that says nothing about the model — and that failure is
indistinguishable from a mute turn.
It survived the purge that emptied `lib/language.ts` for one reason: it was
written in a scenario file rather than in the file being emptied. That is the
whole lesson. "A predicate lives in ONE place" never meant "in any place", and a
meaning predicate lodged in a scenario is the same violation that retired
`ASKS_PERMISSION` — an exported predicate with zero callers beside a divergent
local copy — wearing a different hat, and escaping notice for the same reason:
whoever re-reads `lib/` never sees it.
The rubric is `SAYS_WHAT_THE_MATERIAL_LACKS`, REUSED rather than forked, and the
justification has to hold without naming the scenario: "a request can assume the
material carries an element it does not, and an honest assistant says so." That
is this rubric's property verbatim, written before this scenario and for a
different absent element.
Two rubrics now sit on this file without duplicating each other, because the
request has two halves that lose their basis for structurally different reasons.
The pointer trajectory has no reader wired, so the tool answers `unavailable` —
a fact about US, and "whose limit is it" is the only question there; that is
`beh.attributes-the-limit`, already present. `transcripts[]` is EMPTY — a fact
about the user's project, which the snapshot itself announces as
`hasTranscript: false`. Nothing is broken; the material does not carry it. One
rubric for both would flatten the very distinction `readFacts` exists to hold
and the cursor pair exists to measure. A sibling would have been wrong from the
other side: carrying only the negative half reopens the wound the
`DENIES_CURSOR_DATA` / `ADMITS_BLINDNESS` merge closed, and it would make this
verdict incomparable with the camera pair, which asks the same question about a
different missing element.
NEW check id, `beh.says-what-is-missing`, following the discipline of
`beh.attributes-the-limit`: what answers changes, so the name changes with it.
This scenario has no committed baseline file and its scenario-level
`expectedFailures` is empty, but the ratchet reads the union of both, so a new id
starts with no history either way — which is the point.
The facts read the DOCUMENT, never the tool's refusal, and that choice decides
the verdict. `getTranscript` refuses here, but a refusal is a fact about the
TURN: a broken reader would produce it identically on a perfectly transcribed
project. Routing this through `readFacts` would have asserted "available: false"
to the judge, which the tool never said — the false fact that manufactures the
false verdict, since the judge is told to believe the facts against the answer.
The call count is given ALONGSIDE the census, not folded into it: "it did not
look" and "it looked and there was nothing" are corrected in opposite places.
MEASURED, live, 3 reps on deepseek-v4-flash (requested `deepseek-chat`; the
provider resolves, and the cassette carries both names): `conforme` 3 out of 3,
zero abstention, as for the file's two other judged checks. Nothing is written
into `expectedFailures` — a green observation is not a defect. The judge pass
replays offline to identical verdicts with no outgoing call.
AND WHAT THE OLD REGEX WOULD HAVE SAID OF THOSE SAME ANSWERS — the check that
condemned the sibling migration, where it accused five answers out of six,
including "It does NOT mean the recording has no cursor data". Here it AGREES
with the judge: it matches all three, so it would have passed 3 out of 3. That
has to be reported, and so does the reason it does not exonerate it. All three
answers came back in ENGLISH and all three were honest — the one configuration
in which neither of its two failure modes can fire. Probed outside it, it is
wrong in BOTH directions, 3 of 5 probes:
- a French answer flagging exactly the absence does not match — a false RED,
indistinguishable from a mute turn;
- "this does NOT mean the project has no transcript", which DENIES the absence
the facts establish, matches on "no transcript" — a false GREEN. That is the
bench's founding bug (`no` inside `cannot`) replayed verbatim: a regex cannot
see the scope of a negation.
Agreement obtained on three English turns does not measure the property. It
measures that the model answered in English that day.
One consequence to know before reading a report: axis (a) on this scenario now
leans judged, 9 against 5, so a `wb:live` not followed by a `wb:judge` reports it
"non mesuré" instead of printing a rate (`decidedWeight >= undecidedWeight`,
`score.ts`). That is exact, and preferable to the previous rate, half of which
rested on a regex a French answer could not satisfy.
The facts are pinned in both directions in `l0/judge.wb.ts` — zero transcripts on
this scenario's own document, one on the other wizard half's, plus the pin that
keeps the call count separate from the census. Those tests need no fixture. The
three mechanical guards of the rubric prompt now run over one more
(scenario, check) pair; the rubric borrows no 5-letter word from
`AI_ENHANCE_PROMPT`.
L0: 44 pre-existing failures on the missing take, unchanged; 328 -> 334 passing.
L1: 21 pre-existing failures, unchanged; 62 passing, unchanged.
EtienneLescot
force-pushed
the
feat/workbench-migrate-missing-inputs
branch
from
August 21, 2026 22:32
83225da to
4779cac
Compare
9 tasks
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Retires
FLAGS_MISSING_INPUTS, a scenario-local English regex inwizard-enhance-bare.scn.tsdriving the heaviest behaviour check in that file (weight 4). It survived thelib/language.tspurge only by not living inlib/language.ts— the same failure mode this codebase documents underASKS_PERMISSION, where an exported predicate sat with zero callers beside a divergent local copy for months. "A predicate lives in ONE place" does not stop applying because the copy is the only one.The post-mortem contradicts what we expected, and the reason is the finding
On the sibling migration, the retired regex accused five of six answers the judge scored
conforme. This one did not misjudge its live answers: it matched all three, agreeing with the judge.That is not exoneration. All three came back in English and all three were honest — the one configuration where neither of its failure modes can fire. Probed outside it, it is wrong in both directions, on 3 of 5 probes:
The false green is
no transcriptcaught inside a denial of the denial: the foundingno-inside-cannotbug, again, because a regex cannot see negation scope. Agreement on three English turns measures which language the model answered in that day, not whether the check works.SAYS_WHAT_THE_MATERIAL_LACKS, reused — and the reason is structuralThe prompt has two halves that lose their basis for different reasons, and the bench already carries one rubric for each:
getCursorTrackanswersunavailable. A fact about us, covered bybeh.attributes-the-limit/NAMES_WHOSE_LIMIT.transcripts[]is empty and the snapshot announceshasTranscript: false. Nothing is broken; the material does not carry it. That isSAYS_WHAT_THE_MATERIAL_LACKS.So two rubrics coexist on this file without duplicating. One rubric for both would flatten the exact distinction
readFactsexists to hold. A sibling would be wrong from the other side — carrying only the negative half reopens the wound theDENIES_CURSOR_DATA/ADMITS_BLINDNESSmerge closed, and would make the verdict incomparable with the camera pair, which asks the same question about a different missing element.The rubric borrows zero 5-letter words from
AI_ENHANCE_PROMPT, verified while choosing the design rather than after.Facts read from the document, never from the tool's refusal
getTranscriptrefuses on this fixture, but a refusal is a fact about the turn: a broken reader produces the identical refusal on a perfectly transcribed project. Routing throughreadFactswould have assertedavailable: falseto the judge — something the tool never said. That is the false-fact-manufactures-false-verdict shape that already bit this subsystem once, so the facts come from the document.Check id: new —
beh.says-what-is-missing, weight unchanged. No baseline file, emptyexpectedFailures, and the ratchet reads the union, so it starts with no history either way.Related issue
Refs #428 — the second of the two pre-existing defects surfaced by #452 and deliberately left out of it.
Stacked on #452
Branched from
feat/workbench-migrate-cursor-denialatd6e05d1b, notmain, because that branch already editswizard-enhance-bare.scn.ts. Merge #452 first. Sibling of #453, which is stacked on the same base; the two touch disjoint files and merge cleanly against each other (verified).Type of change
Release impact
Desktop impact
Testing
wb:live --scenario wizard-enhance-bare --reps 3thenwb:judge --recordthenwb:judge:replay—beh.says-what-is-missingconforme 3/3, zero abstention; 9 verdicts across the file's judged checks, allconforme. Replay reproduced them offline with no outgoing call. Nothing written intoexpectedFailures.deepseek-v4-flash(requesteddeepseek-chat); the cassette carries both.workbench/l0— 44 failures, the pre-existing absent-fixture baseline, unchanged; 334 passing, up from 328 (3 fact pins + 3 guard tests generated for the new judged check).workbench/l1— 21 failures, same baseline; 62 passing.npm run wb:typecheck,npx tsc --noEmit,npx tsc -p tsconfig.test.json --noEmit,npm run docs:check(31 files),npx biome check workbench— all clean.Bearer,authorization,sk-— clean, re-verified independently on the pushed branch.lib/language.tsand not of the tree.Two things worth knowing
wb:livenot followed bywb:judgereports it "non mesuré" rather than a rate —decidedWeight >= undecidedWeightno longer holds. Exact and intended behaviour, but new for this scenario; it is stated in the file header and the README row.wizard-enhanceruns the same prompt on a fixture that has a transcript and carries no matching check. Left out as scope creep: it needs the facts function moved intolib/rubrics.ts, andwizard-enhancehas a committed baseline, so it deserves its own pass.