Skip to content

fix(review): reputation-based AI-review skip must fail closed - #9041

Merged
JSONbored merged 2 commits into
mainfrom
fix/reputation-skip-fail-closed
Jul 26, 2026
Merged

fix(review): reputation-based AI-review skip must fail closed#9041
JSONbored merged 2 commits into
mainfrom
fix/reputation-skip-fail-closed

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Closes #9015.

LOOPOVER_REVIEW_REPUTATION=true is live on edge-nl-01, so this was an active production exposure, not a theoretical one.

The defect: when shouldSkipAiForReputation fired (low signal, or the submissions>=8 && merged<1 burst), aiReviewWillRun went false with no compensating hold. The contributor-controlled skip beside it fails CLOSED via maybeAddRequiredAutoReviewSkipHold; the reputation skip set no autoReviewSkipReason, so no hold branch fired. The PR then proceeded on deterministic checks only — CI, coverage, guardrails, secrets, linked-issue — none of which read code semantics — and could auto-merge with zero AI defect detection. A suspected abuser therefore received LESS scrutiny than a trusted contributor, exactly inverting the feature's intent. The issue verified three submitters sitting in the burst state at report time.

Fix 1 — fail closed (maybeAddReputationSkipHold): the exact sibling of the contributor-controlled hold, sharing its shouldRequirePublicAiReviewForAdvisory predicate, adding an ai_review_inconclusive finding so the gate holds for a human instead of passing silently. No-ops when no skip fired (the overwhelmingly common path) and when AI review is off for the repo (nothing was suppressed).

Fix 2 — restore the half-blind leg: getSubmitterCadence queried review_targets, which stopped receiving writes at the 2026-06-22 self-host cutover (newest row frozen there), so the machine-paced cadence signal never fired for anyone. It now reads the live pull_requests ledger (repo_full_name + case-insensitive author_login, created_at as the submission timestamp), making the second leg of the feature actually functional.

Tests: the hold fires under a reputation skip and stays silent on both no-skip and review-off arms; a query-shape assertion pins the cadence source to pull_requests and away from review_targets so this cannot silently regress at the next cutover. Changed-line coverage: all statements and branches.

Acceptance (from the issue): a burst-flagged author's PR now either gets a review or a named human hold — never a silent deterministic-only auto-merge.

LOOPOVER_REVIEW_REPUTATION is live in production. When the reputation
signal downgraded a submitter (low signal, or the submissions>=8 /
merged<1 burst), AI review was suppressed with NO compensating hold, so
the PR proceeded on deterministic checks alone — none of which read code
semantics — and could auto-merge with zero defect detection. Suspicion
bought LESS scrutiny, inverting the feature's intent; three submitters
were in the burst state at the time of the report.

maybeAddReputationSkipHold mirrors the contributor-controlled skip's
fail-closed hold exactly, adding ai_review_inconclusive wherever public
AI review was expected to run. Also repoints the cadence query from
review_targets — frozen since the 2026-06-22 self-host cutover, leaving
the machine-paced leg silently inert — to the live pull_requests
ledger.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 26, 2026
@JSONbored
JSONbored force-pushed the fix/reputation-skip-fail-closed branch from e368608 to 609ef30 Compare July 26, 2026 16:20
@JSONbored
JSONbored merged commit c25fa1e into main Jul 26, 2026
3 of 4 checks passed
@JSONbored
JSONbored deleted the fix/reputation-skip-fail-closed branch July 26, 2026 16:29
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
21561 1 21560 21
View the top 1 failed test(s) by shortest run time
test/unit/reputation-wiring.test.ts > shouldSkipAiForReputation (helper) > submission-cadence signal (#4514) > FLAG-ON: true for a machine-paced submitter even though every submission itself looks fine (quality-neutral)
Stack Traces | 0.295s run time
AssertionError: expected false to be true // Object.is equality

- Expected
+ Received

- true
+ false

 ❯ test/unit/reputation-wiring.test.ts:299:105

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

JSONbored added a commit that referenced this pull request Jul 26, 2026
… to pull_requests (#9097)

#9041 repointed getSubmitterCadence's query from review_targets (frozen since the
2026-06-22 self-host cutover) to the live pull_requests ledger, but the local
seedReviewTarget helper inside "submission-cadence signal (#4514)" still inserted
into review_targets -- so its own test seeded a table the code no longer reads,
making "FLAG-ON: true for a machine-paced submitter" fail on main for everyone
(cadence always resolves to 0 samples against the real query).

Renamed to seedCadencePullRequest and repointed at pull_requests
(repo_full_name/author_login, matching the real query shape) so the test again
exercises the code path it names. The module-level seedReviewTarget (used
elsewhere in this file for the separate quality/burst signal, which #9041 did not
change) is untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

orb(security): reputation burst-downgrade fails OPEN — suspected abusers get LESS review (LIVE in prod)

1 participant