Skip to content

fix(selfhost): make retention pruning work on postgres - #2485

Merged
JSONbored merged 2 commits into
mainfrom
claude/trusting-payne-4b45ce
Jul 2, 2026
Merged

fix(selfhost): make retention pruning work on postgres#2485
JSONbored merged 2 commits into
mainfrom
claude/trusting-payne-4b45ce

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Data-retention pruning (src/db/retention.ts) batches its deletes via SQLite's rowid pseudo-column and binds its cutoff with SQLite's numbered-placeholder syntax (?1). Both are SQLite/D1-only. On the self-host Postgres backend, rowid reaches Postgres unmodified and raises column "rowid" does not exist, dead-lettering the prune-retention job every run — this is the live self-host incident (_selfhost_jobs.id=61132, 5 attempts, last_error = column "rowid" does not exist").
  • Fix lives in the shared SQLite → Postgres SQL translator (src/selfhost/pg-dialect.ts) that every query already passes through on the Postgres backend, not in retention.ts itself:
    • translateRowid(): rewrites the bare rowid token to Postgres's ctid system column. Both are stable per-row identifiers for the lifetime of a single statement — exactly how this codebase uses rowid (bounded batched delete, ORDER BY tie-breaking), never for durable row identity. SQLite/D1 is untouched, since this translator only runs on the Postgres adapter.
    • toNumberedPlaceholders(): fixed to recognize a numbered placeholder (?1, ?2, …) and reuse its index directly as $1/$2, instead of folding the trailing digit into the anonymous-placeholder counter (which corrupted ?1 into $11 — a bind index nothing supplies). This was required for retention's own cutoff bind to resolve correctly on Postgres, and incidentally fixes the same latent bug in repositories.ts's claimRegateFanoutSlot().
  • retention.ts itself is unchanged — it keeps emitting the same SQLite-dialect SQL that already runs correctly on D1; batching/maxPerTable, dryRun, and the audit_events durable-event exclusion are all untouched and re-verified by the new tests.

Sentry triage (no code changes beyond the above — kept this PR narrow):

  • review_context_fetch_failed (REES 403) and orb_broker_unavailable are real gaps (a startup config-readiness check, and retry backoff/dedup respectively) — deferred as follow-up work.
  • The AI-review-inconclusive/unparseable cluster and the PR-publish-failed cluster look already addressed by two recently landed commits (9d05cb6a, 8da2d62f) — worth confirming no fresh occurrences post-deploy before closing those Sentry issues.
  • The bundlephobia analyzer HTTP-error noise is unrelated to retention/Postgres and is deferred.

The live dead-lettered job (_selfhost_jobs.id=61132) should be safe to retry once this deploys.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • No issue linked — this is a direct fix for a live, precisely root-caused self-host incident (a dead-lettered production job with the exact error captured); the change is narrowly scoped to the Postgres SQL-translation layer.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally (unsharded) — src/selfhost/pg-dialect.ts is at 100% lines/branches/functions on the diff (confirmed via the raw lcov output); full suite: 6063 passed, 0 failed.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate — 0 vulnerabilities
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries — see below. Also ran the real-Postgres integration suite locally against an actual Postgres 17 instance (PG_TEST_URL=... npx vitest run test/integration/selfhost-pg.test.ts), including the new retention regression case — all green.

Tests added:

  • test/unit/selfhost-pg-dialect.test.ts: translateRowid unit tests, a regression test against the exact batched-delete SQL shape retention.ts emits, numbered-placeholder regression tests (?1/?2$1/$2, not $11/$21).
  • test/unit/selfhost-pg-retention.test.ts (new): pruneExpiredRecords/processJob exercised against a mocked pg.Pool that throws the real Postgres error if untranslated rowid SQL ever reaches it again — covers dry-run, bounded batching + the per-table cap, the audit_events durable-event exclusion surviving translation, and a named regression test for the live dead-letter (processJob({ type: "prune-retention" }) no longer throws/dead-letters on Postgres).
  • test/integration/selfhost-pg.test.ts: a real-Postgres end-to-end case (skipped unless PG_TEST_URL is set, same convention as the rest of that suite).

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth/session/CORS surface touched.)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A — no API/OpenAPI/MCP surface touched.)
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (N/A — no UI changes.)
  • Visible UI changes include a UI Evidence section. (N/A — backend-only change, no visible UI change.)
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. (N/A — no changelog edit.)

Notes

  • Sentry MCP access was unavailable in the session that produced this PR, so the triage above is based on reading the relevant code paths (REES enrichment client, Orb broker client, AI review parser, heavy-dependency analyzer, review-publish path) rather than live Sentry data.

Data-retention pruning (src/db/retention.ts) batches its deletes via
SQLite's rowid pseudo-column, and its cutoff WHERE clause uses SQLite's
numbered-placeholder syntax (`?1`). Both are SQLite/D1-only: on the
self-host Postgres backend, `rowid` reaches Postgres unmodified and
raises `column "rowid" does not exist`, dead-lettering the
prune-retention job every run (live self-host: _selfhost_jobs.id=61132,
5 attempts).

Rather than special-case retention.ts, the fix lives in the shared
SQLite -> Postgres SQL translator (src/selfhost/pg-dialect.ts) that
every query already passes through on the Postgres backend:

- translateRowid(): rewrites the bare `rowid` token to Postgres's
  `ctid` system column. Both are stable per-row identifiers for the
  lifetime of a single statement, which is exactly how this codebase
  uses rowid (bounded batched delete, ORDER BY tie-breaking) - never
  for durable row identity. SQLite/D1 is untouched: this translator
  only runs on the Postgres adapter.
- toNumberedPlaceholders(): fixed to recognize a numbered placeholder
  (`?1`, `?2`, ...) and reuse its index directly as `$1`/`$2`, instead
  of folding the trailing digit into the anonymous-placeholder counter
  (which corrupted `?1` into `$11` - a bind index nothing supplies).
  This is required for retention's own cutoff bind to resolve
  correctly on Postgres, and also fixes the same latent bug in
  repositories.ts's claimRegateFanoutSlot().

retention.ts itself is unchanged - it keeps emitting the same
SQLite-dialect SQL that already runs correctly on D1.

Tests added:
- selfhost-pg-dialect.test.ts: translateRowid unit tests, a regression
  test against the exact batched-delete SQL shape retention.ts emits,
  and numbered-placeholder regression tests.
- selfhost-pg-retention.test.ts (new): pruneExpiredRecords/processJob
  against a mocked pg.Pool that throws the real Postgres error if
  untranslated rowid SQL ever reaches it again, covering dry-run,
  bounded batching, the audit_events durable-event exclusion, and a
  named regression test for the live dead-letter.
- test/integration/selfhost-pg.test.ts: a real-Postgres end-to-end
  case (skipped unless PG_TEST_URL is set, same as the rest of that
  suite) - validated locally against a real Postgres 17 instance.

Validation: npm run test:ci; npm run test:coverage (unsharded,
src/selfhost/pg-dialect.ts at 100% lines/branches/functions on the
diff); npm audit --audit-level=moderate; the new integration test
against a real local Postgres 17.

The live dead-lettered job (_selfhost_jobs.id=61132) should be safe to
retry once this deploys.

Sentry triage (no code changes in this PR beyond the above):
- review_context_fetch_failed (REES 403) and orb_broker_unavailable
  are real gaps (a startup config-readiness check and retry
  backoff/dedup, respectively) - deferred as follow-up work, not
  bundled here to keep this fix narrow.
- The AI-review-inconclusive and PR-publish-failed clusters look
  already addressed by two recently landed commits (9d05cb6,
  8da2d62); worth confirming no fresh occurrences post-deploy before
  closing those.
- The bundlephobia analyzer HTTP-error noise is unrelated to
  retention/Postgres and is deferred.

no issue because: this is a direct fix for a live, precisely
root-caused self-host incident (a dead-lettered production job with
the exact error captured); the change is narrowly scoped to the
Postgres SQL-translation layer.
@loopover-orb

loopover-orb Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 Gittensory is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing

@loopover-orb loopover-orb Bot added gittensor gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jul 2, 2026
…nymous one

toNumberedPlaceholders() reused a numbered placeholder's own index
(?1 -> $1) without raising the separate anonymous-placeholder counter,
so a later bare `?` in the same query could reuse an already-assigned
index (e.g. "a=?1 AND b=?" translated to "a=$1 AND b=$1" instead of
"a=$1 AND b=$2"). SQLite's own rule for a bare `?` is "one greater
than the largest parameter number already assigned", so the counter
must track numbered indices too.

Neither of the two real call sites (retention.ts's retentionWhere,
repositories.ts's claimRegateFanoutSlot) mixes numbered and anonymous
forms in one query, so this didn't affect production behavior, but the
translator is general-purpose and the added test had baked the wrong
contract in as if it were correct.

Caught by gate review on #2485.
@JSONbored
JSONbored merged commit 8d4e82a into main Jul 2, 2026
10 checks passed
@JSONbored
JSONbored deleted the claude/trusting-payne-4b45ce branch July 2, 2026 06:17
@github-project-automation github-project-automation Bot moved this from Todo to Done in gittensory - v1 roadmap Jul 2, 2026
@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.95%. Comparing base (118d537) to head (80948c3).
⚠️ Report is 29 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2485      +/-   ##
==========================================
+ Coverage   95.93%   95.95%   +0.01%     
==========================================
  Files         225      226       +1     
  Lines       25338    25395      +57     
  Branches     9218     9235      +17     
==========================================
+ Hits        24308    24367      +59     
  Misses        417      417              
+ Partials      613      611       -2     
Files with missing lines Coverage Δ
src/selfhost/pg-dialect.ts 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant