perf(queue): fan build-contributor-evidence out into per-batch jobs (config-driven)

[JSONbored]

Summary

The scheduled build-contributor-evidence job resolved evidence for up to 500 contributors in ONE job — each doing a /users/{login} read plus its (uncached, paginated) /users/{login}/repos reads — so the whole set bursts against GitHub in a single execution during the 6-hourly full-sync window.

This makes the scheduled trigger fan out into per-batch jobs so those per-login reads spread across the queue's paced execution + rate-limit admission instead of bursting:

• The cron trigger (no login) derives the contributor set from stored PRs + issues, then enqueues one build-contributor-evidence job per batch of CONTRIBUTOR_EVIDENCE_BATCH_SIZE logins (staggered), and returns.
• Each batch job loads the shared aggregate data once and processes its batch; the single-login re-index path is unchanged.
• Config-driven: CONTRIBUTOR_EVIDENCE_BATCH_SIZE (default 150, 0 = disable → the pre-existing single job). A set that fits one batch stays a single job, so small instances see no change.

Trade-off (called out honestly): because each per-login computation needs the full aggregate (buildContributorFit / buildContributorOutcomeHistory cross-reference all PRs/issues), each batch re-loads that shared data. So fan-out trades a bounded ceil(N/batchSize)× DB re-read for spreading the GitHub burst — which is the right trade on a rate-limited instance, and the batch size lets operators tune it (or turn it off).

Footgun avoided: a fanned-out batch coalesces by its first login (batches are disjoint slices, so heads are unique) — distinct batches never collapse into one :all key (which would silently drop all-but-one batch). A duplicate re-enqueue of the same batch still coalesces.

Advances #1936 (rec #2 of the #1941 read-reduction audit).

Validation

• git diff --check, npm run typecheck
• npm run test:coverage — new tests: the fan-out enqueues one batch job per login above the size; the disabled (0) path stays a single inline job; the batch-size reader defaults on unset/invalid/negative; and the batch coalesce key is unique per batch (never :all). Existing single-login + inline evidence tests still pass.
• npm run test:ci, npm audit --audit-level=moderate
• No cf-typegen: CONTRIBUTOR_EVIDENCE_BATCH_SIZE is read via process.env (works on cloud + self-host), documented in .env.example.

Safety

• No secrets/etc.; output is byte-identical (same evidence for the same contributors), only the job topology changes.
• Default 150 keeps small instances on the current single-job path; 0 fully disables the fan-out.

[loopover-orb]

Caution

🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥

🛑 Gittensory review result - fixes required

_{Review updated: 2026-07-01 08:25:18 UTC}

6 files · 1 AI reviewer · no blockers · readiness 91/100 · CI failing · unstable

🛑 Suggested Action - Manual Review

• Touches a guarded path — held for manual review

Review summary
The change cleanly separates the scheduled trigger from per-batch execution and preserves the single-login path, with coalescing updated so batch jobs do not all collapse into the scheduled trigger key. The core fan-out behavior is coherent in the visible diff, and the tests cover the enabled, disabled, empty, and coalescing cases. The main risk is the new configuration source: it is documented as cloud + self-host, but the implementation reads only `process.env`, which needs an explicit runtime guarantee outside Node.

Nits — 6 non-blocking

• nit: `src/queue/processors.ts:2547` should verify that `process.env` is available in the Cloudflare Worker runtime, or read this through the existing environment/config surface so the new setting is actually cloud-configurable.
• nit: `src/queue/processors.ts:2579` enqueues all batches with `Promise.all`, so the queue receives up to four hundred-some sends at once for the current 500-login cap; that is probably acceptable, but a small sequential loop would better match the stated goal of avoiding bursts.
• nit: `src/selfhost/queue-common.ts:665` applies the new `logins` fallback to both `build-contributor-evidence` and `build-contributor-decision-packs`, even though only evidence jobs have `logins`; split the cases or add a comment if you intentionally want malformed decision-pack payloads to key as batches.
• In `src/queue/processors.ts:2547`, either route `CONTRIBUTOR_EVIDENCE_BATCH_SIZE` through an existing env/config helper or add a focused test/fixture proving the Worker runtime has `process.env` populated for this code path.
• In `src/queue/processors.ts:2579`, consider replacing `Promise.all(batches.map(...))` with a `for` loop that awaits each `env.JOBS.send`, keeping only the job execution staggered via `delaySeconds`.
• Touches a guarded path — held for manual review — A maintainer must review and merge this change.

CI checks failing

• codecov/patch — 93.75% of diff hit (target 99.00%)

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	⚠️ Missing	No linked issue or no-issue rationale found.
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (size label size:M; no linked issue context).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 68 registered-repo PR(s), 58 merged, 202 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 68 PR(s), 202 issue(s).
Gate result	⚠️ Not blocking	Advisory; not blocking this PR.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository registration is not available in the local Gittensory cache.
• Public profile languages: not available
• Official Gittensor activity: 68 PR(s), 202 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Explain no-issue PR.
• No action.
• Link the issue being solved, or explicitly explain why this is a no-issue PR.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[JSONbored]

Rebased onto main to resolve the `queue-common.ts` conflict from #2001 (the event-job coalescing that just merged): jobCoalesceKey's destructure now carries both the event-job id fields and this PR's logins, and the two switch cases sit side by side.

Also tightened the coalescing contract the review flagged: a fanned-out batch can no longer fall through to the :all key (the scheduled-trigger slot) — a non-empty logins batch keys by its first login, and a batch with no usable head is left uncoalesced (null) rather than collapsed into :all (which would have dropped its work). Added a test for that edge. Gate + npm audit green.

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
5558	1	5557	13

View the full list of 1 ❄️ flaky test(s)

test/unit/codecov-policy.test.ts > Codecov policy > fails closed when the backend coverage report is missing or cannot upload

Flake rate in main: 100.00% (Passed 0 times, Failed 1 times)

Stack Traces | 0.0505s run time

AssertionError: expected '${{ success() && (github.event_name =…' to be '${{ success() && (github.event_name =…' // Object.is equality

Expected: "${{ success() && (github.event_name == 'push' || needs.changes.outputs.backend == 'true') && github.event.pull_request.head.repo.fork != true }}"
Received: "${{ success() && (github.event_name == 'push' || needs.changes.outputs.backend == 'true') }}"

 ❯ test/unit/codecov-policy.test.ts:55:27

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}