feat(db): scheduled data-retention pruning for log/snapshot tables

[JSONbored]

What

A conservative, auto-pruning retention job so high-volume append-only / superseded-snapshot tables don't grow unbounded. Precautionary D1 hygiene (the DB isn't at its size limit yet; the recent overload was load/concurrency, not size).

Policy (src/db/retention.ts → RETENTION_POLICY)

Table	Column	Window
webhook_events	received_at	30d
audit_events	created_at	90d
ai_usage_events	created_at	90d
product_usage_events	occurred_at	180d
github_rate_limit_observations	observed_at	30d
signal_snapshots	generated_at	90d
score_previews	generated_at	90d
repo_snapshots	fetched_at	90d

Only append-only logs / usage metrics / superseded snapshots (latest-matters) are pruned. Current-state and reference tables are intentionally excluded — repositories, repository_settings, pull_requests, issues, contributors, repository_ai_keys, focus manifests, registry/scoring snapshots.

How

• pruneExpiredRecords deletes rows older than each window in bounded rowid batches (1000/batch, capped at 50k/table/run so a first prune of a backlog can't blow the D1 statement budget; the daily cron drains the rest). Table/column names come only from the hardcoded policy and are identifier-validated (^[a-z_]+$); the cutoff is bound as a parameter.
• Wired as a daily 03:00 UTC prune-retention queue job via the existing cron; processJob runs it and audits the outcome (retention.prune).
• GET /v1/internal/retention/preview — read-only dry-run that reports the rows the next prune would delete, per table. Deletes nothing. Use it to see what the cron will remove.

Safety / reversibility

• The cron deletes on the schedule once merged+deployed (as requested). Preview the impact first via the route above.
• Historical rows with the old literal "CURRENT_TIMESTAMP" in a timestamp column sort after real ISO dates, so they're naturally skipped by col < cutoff (a bounded, non-growing legacy set).

Tests

Dry-run vs real delete, the multi-batch loop + per-table cap, the identifier guard, the audit event, processJob, the preview route, and an assertion that protected tables are excluded from the policy. Full suite green; coverage holds above the 97% gate.

5th and final PR from the gate-hang investigation thread (after #652, #655, #656, #657).

[ghost]

Note

Gittensory Gate skipped

PR closed before full evaluation. No late first comment was created.

Signal	Result	Evidence	Action
Gate result	⚠️ Skipped	#660 is no longer open.	No action.

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[ghost]

gittensory · advisory review

Reviewed 6 changed file(s) — two independent AI reviewers.

Suggested action: ✅ Safe to merge — both reviewers found no blocking issues.

Reviewer A · gpt-oss-120b — recommends ✅ merge
This PR introduces a data‑retention pruning system with a new internal preview endpoint, scheduled job integration, processor support, type definitions, and comprehensive tests. The implementation validates identifiers, batches deletions, and audits outcomes, and appears well‑tested and safe.

Suggestions

• Verify that the new /v1/internal/retention/preview route is protected by the same internal auth middleware used by other internal endpoints.
• Consider adding a brief comment in pruneExpiredRecords about the D1 statement budget implications for future maintainers.
• Document the expected schedule (03:00 UTC) in the project README or cron documentation.

Worth double-checking

• If the internal auth check is missing, the preview endpoint could be exposed publicly.
• Large backlogs could cause multiple hourly runs to approach the D1 statement limit before the cap is reached.

Reviewer B · nemotron-3-120b-a12b — recommends ✅ merge
This PR adds scheduled data-retention pruning for log/snapshot tables, including a new retention module, internal preview API endpoint, scheduled job, and comprehensive tests. The changes are well-structured, include safety checks (identifier validation, batched deletes), respect the public/private boundary (internal-only endpoints, no forbidden terms), and maintain determinism. The test coverage is strong and validates edge cases.

Suggestions

• Consider adding a comment in src/db/retention.ts explaining why MAX_DELETED_PER_TABLE is set to 50,000 (e.g., to avoid exceeding D1 statement limits).
• In src/queue/processors.ts, the runRetentionPrune function could validate that requestedBy is one of the allowed literals (though the type system already enforces this).
• Ensure the internal route /v1/internal/retention/preview is protected by the same internal job token verification as other internal routes (verified in test).

Worth double-checking

• Verify that the internal job token (INTERNAL_JOB_TOKEN) is kept confidential and not exposed to public clients (existing infrastructure concern).
• Confirm that the RETENTION_POLICY does not inadvertently include any table containing private/private context data (test already validates this).
• Check that the batched delete loop handles edge cases like zero rows correctly (covered in tests).