Skip to content

Resolve dead aiReviewCloseConfidence floor (decision + fix) #4603

Description

@JSONbored

Context

Part of the review-stack architecture audit (parent epic).

Root cause (git-blame confirmed): commit 311b7613d ("fix(review): enforce blocker-first PR
verdicts (#1781)", 2026-06-29) intentionally removed the old isLowConfidenceAiReviewHold mechanism, so
an ai_consensus_defect / ai_review_split finding now hard-blocks unconditionally whenever
aiReviewGateMode: "block", regardless of the model's own confidence. The commit correctly rewrote
one docstring (src/types.ts:832-836, the RepositorySettings.aiReviewCloseConfidence field) to say
the floor is now "retained as configurable context, not a manual-review downgrade" — but left five
other copies of the old claim
unedited, scattered across the exact code path that produces and carries
the value:

Location Stale claim
src/types.ts:457-461 (AdvisoryFinding.confidence) "an AI defect blocks ONLY when this clears the floor"
src/services/ai-review.ts:395-398 (ModelReview.confidence) "Drives the gate's aiReviewCloseConfidence floor"
src/services/ai-review.ts:1423-1426 (consensusDefectOf) "the calibrated confidence (#8) ADDS a numeric floor on top"
src/services/ai-review.ts:2207-2209 (runGittensoryAiReview return) "gate ai_review_split on the same floor as a consensus defect"
src/queue/processors.ts:7589,7604-7607 "clears aiReviewCloseConfidence ⇒ block; below it ⇒ human-review hold"

The actual gate code, isConfiguredGateBlocker (src/rules/advisory.ts:882-886):

if (code === "ai_consensus_defect" || code === "ai_review_split") {
  void (policy.aiReviewCloseConfidence ?? DEFAULT_AI_REVIEW_CLOSE_CONFIDENCE);
  return gateMode(policy.aiReviewGateMode ?? "advisory") === "block";
}

literally voids the confidence value and returns based on aiReviewGateMode alone.
finding.confidence is never read anywhere else in the gate path to decide block-vs-hold — it is now
pure telemetry.

Concrete failure scenario

Repo has gate.aiReview.mode: block (documented closeConfidence default 0.93). Two independent AI
reviewers both flag the same ambiguous line as a blocker but each self-report low confidence (0.3-0.4) —
consensusDefectOf sets confidence: Math.min(a.confidence, b.confidence) exactly as designed. Per
every one of the five stale comments, a sub-0.93 confidence should degrade this to a neutral
"held for human review." Instead the PR is auto-CLOSED one-shot (contributor PRs are one-shot). This is
also user-facing config-as-code — .gittensory.yml.example:346-351's comment (correctly, unlike the
internal docs) already says the floor no longer downgrades a blocker, so a maintainer tuning
closeConfidence today is tuning a no-op.

This issue needs a decision before code changes

Two valid resolutions — pick one before implementing:

  1. The post-fix(review): enforce blocker-first PR verdicts #1781 behavior (confidence is pure telemetry, no downgrade) is intentional → fix the 5 stale
    comments to match src/types.ts:832-836's already-correct language. No behavior change.
  2. The confidence floor should still downgrade low-confidence consensus defects to a hold →
    restore the comparison in isConfiguredGateBlocker, matching what all 5 comments (and the
    closeConfidence config field's continued existence) still promise callers.

Acceptance criteria

  • Decision made and recorded in the PR description (which of the two resolutions, and why).
  • All 6 locations (the 5 stale comments + isConfiguredGateBlocker if behavior changes) are
    consistent with each other.
  • If behavior changes: regression test proving a sub-floor consensus defect degrades to hold, not
    close.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions