perf(db): index ai_usage_events(status, created_at) for the daily-budget query - #657
Conversation
…get query sumAiEstimatedNeuronsSince runs SELECT sum(estimated_neurons) WHERE status='ok' AND created_at >= ? — verified via EXPLAIN QUERY PLAN on prod to be a full `SCAN ai_usage_events` (no covering index; existing indexes are on feature/actor). It runs on every AI review/summary (the daily-budget check) and the table grows with each AI call, so the scan cost grows unbounded once AI is enabled. Add the covering index (status equality + created_at range). EXPLAIN confirms the query now SEARCHes via it. Investigation found this is the ONLY un-indexed read in the codebase — the per-webhook/gate hot path (issues/pull_requests/ files/bounties by repo, official_miner_detections by login, settings/repos by PK) is already fully indexed, so the recent prod D1 overload is capacity/DB-size related, not a missing-index problem on the hot path. Migration numbered 0028 to avoid colliding with 0026/0027 reserved by #652.
|
gittensory · advisory review Reviewed 3 changed file(s) — two independent AI reviewers. Suggested action: ✅ Safe to merge — both reviewers found no blocking issues. Reviewer A · Suggestions
Worth double-checking
Reviewer B · Worth double-checking
|
|
Note Gittensory Gate skippedPR closed before full evaluation. No late first comment was created.
💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers. |
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Investigation: are hot-path queries doing full table scans?
Audited the prod D1 index inventory against every query the code runs. Conclusion: the per-webhook / gate hot path is fully indexed —
issues/pull_requests/pull_request_files/bountiesbyrepo_full_name,official_miner_detectionsbylogin(PK),repositories/repository_settingsby PK. The recent prod D1 overload is therefore capacity / DB-size related (~2.87 GB, blob-heavy tables), not a missing-index problem on the hot path.A few things I checked and ruled out:
webhook_events(39,780 rows) — the app only reads it bydelivery_id(PK). The 40k-row scans I saw earlier were my own ad-hoc diagnostic queries, not the app. No index needed.audit_events(15,876 rows) — both app reads filter byevent_typefirst, covered byaudit_events_type_created_idx. No index needed.The one real gap
sumAiEstimatedNeuronsSince(the daily AI-neuron budget check) runs:EXPLAIN QUERY PLANon prod confirms a fullSCAN ai_usage_events— the existing indexes are(feature, created_at)and(actor, created_at), neither of which covers astatus-only filter. This query runs on every AI review/summary (it's the budget gate), andai_usage_eventsgrows with each AI call, so the scan cost grows unbounded once AI is enabled — which #652 (AI review) is about to do. It also serves the existingai-summariesbudget query onmain.Fix
Add a covering index
ai_usage_events_status_created_idx (status, created_at)(schema + migration0028). A test asserts viaEXPLAIN QUERY PLANthat the budget query now SEARCHes via the index instead of scanning. Full suite green.Migration numbered
0028to avoid colliding with0026/0027reserved by #652; the D1 runner applies un-applied migrations in order and tolerates the gap.Out of scope (flagged separately)
The actual prod D1 overload looks like capacity/size, not query plans — worth watching D1 metrics / considering retention pruning on the large append-only + blob tables, but that's an ops decision, not a code fix.