fix(github): attribute rate limits by token scope

[JSONbored]

Summary

Closes #1940.

This fixes GitHub rate-limit attribution so self-host queue admission can distinguish installation-token work, public-token work, and unknown/no-token work instead of collapsing too much traffic into an ambiguous bucket.

What changed

• Threaded installation ids through scheduled/API backfill, RAG indexing, and re-gate fan-out jobs.
• Passed installation/public admission keys into GitHub REST and GraphQL reads used by backfill, live PR freshness, live CI/merge facts, linked-issue fallback reads, and review file refreshes.
• Split rate-limit metric/admission scope labels into installation, public, global, unknown, and other.
• Added regression coverage for installed vs non-installed repo fan-out, public-token attribution, installation-token attribution, no-token fallback behavior, and queue/API payload propagation.

Why

The self-host stack needs to keep unrelated GitHub work moving while respecting the actual depleted token bucket. Without stable attribution, observed rate-limit pressure can be too broad or too vague, which makes queue admission less useful during webhook and scheduled-work bursts.

Validation

• git diff --check
• npm run typecheck
• npx vitest run test/unit/github-client.test.ts test/unit/selfhost-queue-common.test.ts test/unit/selfhost-sqlite-queue.test.ts test/unit/selfhost-pg-queue.test.ts
• npx vitest run test/unit/backfill.test.ts test/unit/rag-index.test.ts test/unit/queue.test.ts test/unit/fetch-live-pr-state.test.ts test/unit/fetch-live-pull-request.test.ts test/integration/api.test.ts test/integration/routes-errors.test.ts
• npx vitest run test/integration/api.test.ts test/unit/backfill.test.ts test/unit/rag-index.test.ts test/unit/queue.test.ts
• npm run test:coverage
• Local changed-line LCOV check: 369 changed source lines, 74 changed-line branches, 0 uncovered changed-line branches.

[loopover-orb]

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

_{Review updated: 2026-07-01 06:43:30 UTC}

17 files · 1 AI reviewer · no blockers · readiness 93/100 · CI pending · blocked

⏸️ Suggested Action - Manual Review

• Touches a guarded path — held for manual review

Review summary
The change consistently threads rate-limit admission keys through the main GitHub read paths and updates the metric scope taxonomy to distinguish public-token, installation-token, explicit global, unknown, and other keys. The visible diff correctly derives the key from the same token used for each request in the core queue/backfill paths, including the linked-issue hard-rule path where the new `installationId` argument is passed by `maybeRunAgentMaintenance`. I do not see a reachable break in the provided diff; the main remaining review points are maintainability around duplicated fan-out payload construction and making the token-scope contract harder to misuse.

Nits — 7 non-blocking

• nit: src/queue/processors.ts:1000 and src/queue/processors.ts:1105 duplicate the same repository-to-`{ fullName, installationId }` fan-out shaping, so this should be extracted before the next queue-message field gets threaded through both places.
• nit: src/review/linked-issue-hard-rules.ts:154 keeps `installationId` optional even though a non-public `ciToken` can only be attributed when the caller also supplies the matching installation id; consider a discriminated argument shape or a small helper at the call site to make that contract explicit.
• nit: src/github/backfill.ts:321 introduces `repoInstallationPayload`, but the same conditional payload construction is repeated in `src/queue/processors.ts` and `src/api/routes.ts`; a shared helper would reduce drift for future `JobMessage` metadata.
• In `src/queue/processors.ts`, add a small helper such as `jobRepoPayload(repo)` and use it in both `fanOutAgentRegateSweepJobs` and `fanOutRagIndexJobs`.
• In `src/review/linked-issue-hard-rules.ts`, tighten the args type so `ciToken` and its `installationId` cannot be separated accidentally, or derive both from a single token context object.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
• Touches a guarded path — held for manual review — A maintainer must review and merge this change.

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	✅ Linked	#1940
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (size label size:L; 1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 68 registered-repo PR(s), 58 merged, 209 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 68 PR(s), 209 issue(s).
Gate result	⚠️ Not blocking	Advisory; not blocking this PR.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository registration is not available in the local Gittensory cache.
• Public profile languages: not available
• Official Gittensor activity: 68 PR(s), 209 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Triage stale or unlinked PRs.
• No action.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.66%. Comparing base (1ac3ff3) to head (e758093).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1951      +/-   ##
==========================================
+ Coverage   95.63%   95.66%   +0.02%     
==========================================
  Files         220      220              
  Lines       24400    24452      +52     
  Branches     8847     8867      +20     
==========================================
+ Hits        23336    23393      +57     
  Misses        436      436              
+ Partials      628      623       -5     

Files with missing lines	Coverage Δ
src/api/routes.ts	94.65% <100.00%> (+0.02%)	⬆️
src/github/backfill.ts	96.12% <100.00%> (+0.37%)	⬆️
src/github/client.ts	100.00% <100.00%> (ø)
src/github/pr-freshness.ts	100.00% <100.00%> (ø)
src/queue/processors.ts	89.86% <100.00%> (+0.24%)	⬆️
src/review/linked-issue-hard-rules.ts	100.00% <100.00%> (ø)
src/selfhost/queue-common.ts	91.66% <100.00%> (+0.09%)	⬆️

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[JSONbored]

Reworked and rebased onto main to resolve the flagged blocker and land cleanly on top of the recently-merged #1980 (GraphQL status rollup), #1985 (merged-PR file-hydration skip), and #1976 (conditional backfill).

Blocker fixed — linked-issue-hard-rules.ts installation-token attribution. The reviewer was right: the inline resolver only produced the public-token key, so an installation-token linked-issue fact read recorded as unknown. Rather than duplicate the token→key resolution in yet another spot (which is how it drifted), resolveLinkedIssueHardRule now takes the admission key as an injected arg — maybeRunAgentMaintenance already resolves the correct key (installation vs public) for exactly this ciToken, so it passes that in. Added a regression test asserting an installation key reaches fetchLinkedIssueFacts (a cross-module vi.spyOn).

Scope-helper divergence reconciled. client.ts's githubAdmissionKeyScope now classifies identically to queue-common's githubRateLimitAdmissionKeyScope (installation / public / global / unknown / other) and is exported + directly unit-tested (mirroring the queue-common test), so the two metric surfaces can't drift again.

Threaded the admission key through the paths that didn't exist at this PR's base (so attribution is complete, not partial):

• #1985 hydrateMergedPullRequestFiles — now takes + forwards the key to the per-PR file reads.
• #1980 fetchLiveCiAggregateViaGraphQl / fetchLiveCiAggregatePreferGraphQl — now accept + forward the key (the GraphQL rollup read is attributed too), and the check-suites backstop fetch inside fetchLiveCiAggregate gets it.

Conflicts resolved by keeping main's structure (GraphQL-preferring CI path, hydrateMergedPullRequestFiles) and threading this PR's admission key through it. Local gate green (npm run test:ci + npm audit).

[JSONbored]

Addressed the API-drift note (duplicated token→key derivation + a helper that took a token without requiring the matching key):

• One resolver. Collapsed the three copies (backfill.repoAdmissionKeyForToken, processors.githubAdmissionKeyForToken, pr-freshness's inline ternary) into a single exported githubRateLimitAdmissionKeyForToken(env, token, installationId) in client.ts. The two call-shaped wrappers now just delegate to it, so there's one place the public/installation/unattributed decision lives. Added a direct unit test covering every bucket (no-token, public, installation, non-public-without-id, non-finite-id).
• Token can't travel without its key. resolveLinkedIssueHardRule no longer takes a separable admissionKey; it takes installationId and derives the key from the same token it will read with, via that one resolver. So the key is always in sync with the token — there's no signature that lets a caller pass a token and omit/mismatch the key. Regression test updated to assert the derived installation key reaches the fact reads.

Gate green again locally (npm run test:ci + npm audit).

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	gittensory-ui	e758093	Commit Preview URL Branch Preview URL	Jul 01 2026, 06:33 AM