Skip to content

fix(queue): reconcile-open-prs has no GitHub rate-limit admission or job dedup #4505

Description

@JSONbored

Part of #4496. P8 — medium severity, high confidence, but zero current blast radius (feature flag off by default today) — fix before enabling.

Context

The reconcile-open-prs job type does real, per-repo paginated GitHub REST work every 10 minutes (isReconciliationWindow = minute % 10 === 0, src/index.ts:114, on the */2 * * * * cron) but is excluded from BOTH rate-limit admission and job-key dedup:

  • It's missing from GITHUB_BUDGET_BACKGROUND_TYPES (src/selfhost/queue-common.ts:86-98), so it never yields to shouldWaitForGitHubRateLimit at dequeue time — unlike every other real GitHub-REST-calling background job in this file.
  • src/index.ts:183 pushes it unconditionally with NO !sweepThrottledUntil check — contrast the very next block (index.ts:165-169) which DOES gate backfill-registered-repos this way.
  • jobCoalesceKey (queue-common.ts:882-1023) has no case "reconcile-open-prs", so it falls through to return null — a null key means pg-queue.ts's enqueue-time coalesce/supersede path is skipped entirely and a fresh duplicate row is inserted every tick (pg-queue.ts:911-942), even while a prior scan is still processing.
  • src/selfhost/maintenance-admission.ts's own header comment (lines 5-7) asserts every MAINTENANCE_JOB_TYPES member "already yield[s] to an EXHAUSTED GitHub REST budget (shouldWaitForGitHubRateLimit)" — reconcile-open-prs IS in MAINTENANCE_JOB_TYPES (line 62) but only gets that module's local-queue-depth/host-load admission, never the GitHub-budget one this comment claims — a real, confirmable contradiction between the module's own doc comment and reality.

Meanwhile runOpenPrReconciliation (src/review/pr-reconciliation.ts:94-124) sequentially loops every watched repo, and reconcileOpenPullRequests (src/github/backfill.ts:3254-3274) makes up to RECONCILE_OPEN_PRS_MAX_PAGES = 10 sequential GET /pulls?state=open&per_page=100 calls per repo, then sequentially awaits catchUpMissingPullRequest for every missing PR found — real, potentially large GitHub REST volume with none of the protections its sibling scheduled jobs have.

This cannot fire in production todayisPrReconciliationEnabled defaults OFF (GITTENSORY_PR_RECONCILIATION unset ⇒ false) — but the moment an operator enables this flag, the dual gap (no GitHub-budget admission + no job-key dedup, stacked) becomes real.

Requirements

  • Add "reconcile-open-prs" to GITHUB_BUDGET_BACKGROUND_TYPES in queue-common.ts (or give it its own admission check at enqueue time mirroring backfill-registered-repos's !sweepThrottledUntil guard).
  • Add a case "reconcile-open-prs": return type; to jobCoalesceKey so a second 10-minute tick coalesces into an already-pending/processing scan instead of creating a duplicate row.
  • Correct maintenance-admission.ts's header comment to accurately describe which job types get GitHub-budget admission vs. only local-load admission.
  • Invariant + regression tests (non-negotiable): an invariant test asserting reconcile-open-prs correctly yields when shouldWaitForGitHubRateLimit reports an exhausted budget (mirroring the equivalent test for backfill-registered-repos); a regression test asserting a second reconcile-open-prs enqueue while a prior one is pending/processing coalesces rather than duplicates; a test enabling the flag in a test harness and confirming both protections actually engage end-to-end (since this path has zero test coverage of the gap today, per the audit).

Deliverables

  • reconcile-open-prs added to GITHUB_BUDGET_BACKGROUND_TYPES (or equivalent enqueue-time guard)
  • jobCoalesceKey case added for dedup
  • maintenance-admission.ts header comment corrected
  • Invariant test: rate-limit yield behavior
  • Regression test: duplicate-tick coalescing
  • End-to-end test with the flag enabled

Expected outcome

reconcile-open-prs gets the same GitHub-budget admission and job-key dedup every comparable scheduled GitHub-calling job already has, so enabling GITTENSORY_PR_RECONCILIATION in the future doesn't reintroduce the exact rate-limit/duplication incident class fixed elsewhere tonight.

References

  • src/selfhost/queue-common.ts:86-98, 882-1023 (the missing-from-both-sets job type + the coalesce switch)
  • src/index.ts:114, 165-169, 183 (the cron cadence + the sibling's guard this job lacks)
  • src/selfhost/maintenance-admission.ts:5-7, 40-66 (the incorrect header comment + the actual membership)
  • src/review/pr-reconciliation.ts:94-124 (the sequential per-repo reconciliation loop)
  • src/github/backfill.ts:3230-3274 (the paginated per-repo GitHub calls)
  • src/selfhost/pg-queue.ts:911-942 (the pending-only coalesce mechanism this job never reaches)

Effort

S

Metadata

Metadata

Assignees

Labels

gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions