Skip to content

calibration: detect one-shot-culture reversals — bot-closed PR superseded by a merged successor (reversal_superseded) #8166

Description

@JSONbored

Problem

The reversal detector (recordReversalSignals, src/review/outcomes-wire.ts) keys the "bot was wrong" ground-truth signal on a human REOPENING a bot-closed PR (or reverting a bot merge). Verified against production D1 (2026-07-23): zero reversal_reopened/reversal_reverted events ever, zero owner-reopen-pending markers, and zero bot-closed PRs later merged — not because the bot has never been wrong, but because this gate's own one-shot design tells contributors "recovery = open a fresh PR." The reopen-shaped signal is structurally near-impossible in this culture, which starves the ENTIRE calibration/backtest epic (#8082) of its positive class: precision reads 100% forever, reversed labels never accumulate, and #8105's flip-to-block evidence can never form.

Requirements

⚠️ Required pattern. Detect the culture's actual reversal shape: bot CLOSES PR #N → a SUCCESSOR PR #M (same author or same linked issue, overlapping files/title) is opened and later MERGES. Reuse the existing overlap machinery — duplicate_pr_risk's same-linked-issue/similarity signals and the open-PR file-collision detector — do NOT invent a new similarity engine.

  • On a PR merge, look back over a bounded window for bot-closed PRs in the same repo satisfying the successor heuristics; record a reversal_superseded event (a NEW event type — do not overload reversal_reopened) against the ORIGINAL closed PR's target, carrying the successor's number and which heuristics matched.
  • Conservative by design: require at least (same linked issue) OR (same author AND file-overlap above a threshold). A false "the bot was wrong" poisons calibration worse than a miss — precision over recall here.
  • Wire it into the same consumers as the other reversal events: computeRulePrecision-feeding override recording (a HumanOverrideEvent with verdict reversed for the rules that blocked #N), the accuracy trend's reversal source, and the backtest corpus labels.
  • Every event carries metadata.heuristics so a later audit can re-examine borderline matches.

Deliverables

  • reversal_superseded detection (pure heuristics core + wire), recorded on successor-merge
  • HumanOverrideEvent(reversed) recording for the original PR's blocking rules
  • Tests: both heuristic paths, the conservative-miss paths, no-double-record idempotency

Test Coverage Requirements

99%+ Codecov patch coverage (branch-counted) — src/** wiring; both sides of every heuristic guard.

Links & Resources

Boundaries

Never removes the existing reopen/revert signals. New event type only. No auto-action taken on detection — this is ground-truth recording, nothing else.

maintainer-only — calibration ground-truth integrity.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions