Skip to content

fix(review): rate-limit the ci_stuck_review_repeat_suppressed log (#4998) - #5037

Merged
JSONbored merged 1 commit into
mainfrom
fix/4998-ci-stuck-repeat-log-ratelimit
Jul 11, 2026
Merged

fix(review): rate-limit the ci_stuck_review_repeat_suppressed log (#4998)#5037
JSONbored merged 1 commit into
mainfrom
fix/4998-ci-stuck-repeat-log-ratelimit

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Fixes fix(review): ci_stuck_review_repeat_suppressed logs itself as an error on every suppression (650 Sentry events) #4998: ci_stuck_review_repeat_suppressed — a log line that exists specifically to announce "I am suppressing a repeat notification" — fired as its own console.error (Sentry-forwarded) on every single suppression: 649 events over 4 days, escalating.
  • Root cause matched Sentry Seer's summary directly this time: the #orb-ci-stuck-repeat guard (prReadyForReview, src/queue/processors.ts) correctly caps the expensive part (re-running a full paid AI review) to once per head SHA, but the log announcing that cap re-fires on every later evaluation of the same still-stuck PR — a webhook re-trigger or a sweep pass, at roughly a 5-10 minute cadence based on the raw event timestamps.
  • Fix: rate-limit the log only, once per (repo, pr, headSha) per day, via the same SELFHOST_TRANSIENT_CACHE-backed helper pattern already used by ciPendingDeferStuck/ciReReviewCoalesced in this file (getTransientKey/putTransientKey). The underlying defer/suppression behavior is untouched — it still runs on every evaluation; only the Sentry-visible log is coalesced. A new head SHA (a fresh commit) is a fresh key, so a genuinely new incident is never silently swallowed by an old PR's coalesce window.

Investigation: what's actually wrong with the stuck PR(s) (issue requirement #2)

Pulled the raw event list behind this Sentry issue rather than trusting its title/culprit summary, since forwardStructuredLogToSentry groups every ci_stuck_review_repeat_suppressed occurrence into one Sentry issue regardless of repo/PR/headSha (grouped by the fixed message template) — the issue's displayed "(JSONbored/awesome-claude#4816) headSha=..." is just the latest contributing event, not a single PR stuck for 4 days as the issue title implied. Confirmed: PR awesome-claude#4816 itself was only open ~77 minutes end-to-end (created 04:44 UTC, merged 06:01 UTC the same day) — not stuck at all.

The real 649-event history spans several distinct PRs across gittensory and awesome-claude (at least #4343, #4344, #4346, #4349, #4371 in gittensory and #4739, #4745, #4816 in awesome-claude, from a 25-event sample), each re-evaluated repeatedly at a ~5-10 minute cadence while stuck — the exact "steady few-minute cadence" burn pattern the original #orb-ci-stuck-repeat guard's own code comment describes, just now visible as noisy logging instead of wasted AI spend.

Checked the most-repeated one, gittensory#4349, directly: it carries a non-required Contributor trust check (a third-party Action, superagent.sh) that resolved to conclusion ACTION_REQUIRED in 3 seconds — but main's actual branch-protection required-status-checks list is only ["Superagent Security Scan", "validate"], so that check is not itself the blocker. All sampled PRs eventually merged cleanly once a later commit landed or a required check finally completed, so this is not a permanent hang — it's multiple PRs sitting through a slow-to-settle CI run (or a required check needing a manual fork-Action approval) for anywhere from tens of minutes to a few hours each, during which the sweep/webhook cadence re-evaluates and re-logs. No code change beyond this PR's rate-limit is proposed for that underlying CI-settling latency; it's a candidate follow-up (documented in the parent observability epic) rather than a fix bundled here.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (Closes #4998).

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • test:coverage (full unsharded): not run end-to-end — ran scoped vitest --coverage for test/unit/queue.test.ts (159 tests) and confirmed via lcov that every changed line and both branches of the new coalescing guard are covered.
  • actionlint / test:workers / build:mcp / test:mcp-pack / ui:openapi:check / ui:lint / ui:typecheck / ui:build / npm audit: not run — this change touches only src/queue/processors.ts (an existing transient-cache-backed guard, no new API/schema/binding/dependency surface) and its tests; no workflow, MCP, UI, or dependency-manifest surface changed.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth surface touched.)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A.)
  • UI changes use live API data or real empty/error/loading states. (N/A.)
  • Visible UI changes include a UI Evidence section. (N/A.)
  • Public docs/changelogs are updated where needed. (N/A — internal engine behavior; changelog is not edited in a normal PR.)

Notes

Part of a batch of 13 bug fixes filed from a Sentry-issue triage this session (#4994#5006). This is #5 by priority.

)

The log line that announces a repeat suppression fired as its own
console.error on EVERY suppressed evaluation, not once per incident --
649 events over 4 days. The underlying suppression (capping the
finalize/review spend to once per head SHA) was correct; only the
observability side never got the same treatment. Rate-limit the log
to once per (repo, pr, headSha) per day via the existing self-host
transient-cache helpers; the defer itself is untouched and still runs
on every evaluation.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.36%. Comparing base (5b2334e) to head (079d367).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5037   +/-   ##
=======================================
  Coverage   94.36%   94.36%           
=======================================
  Files         459      459           
  Lines       39285    39291    +6     
  Branches    14329    14331    +2     
=======================================
+ Hits        37072    37078    +6     
  Misses       1558     1558           
  Partials      655      655           
Flag Coverage Δ
shard-1 46.45% <100.00%> (-0.28%) ⬇️
shard-2 33.63% <14.28%> (+0.25%) ⬆️
shard-3 31.36% <14.28%> (-0.12%) ⬇️
shard-4 32.57% <14.28%> (-0.26%) ⬇️
shard-5 34.34% <14.28%> (+0.88%) ⬆️
shard-6 45.41% <14.28%> (+0.30%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/queue/processors.ts 95.39% <100.00%> (+<0.01%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit 80ee061 into main Jul 11, 2026
17 checks passed
@JSONbored
JSONbored deleted the fix/4998-ci-stuck-repeat-log-ratelimit branch July 11, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

fix(review): ci_stuck_review_repeat_suppressed logs itself as an error on every suppression (650 Sentry events)

1 participant