fix(selfhost): make the Postgres connection pool size operator-tunable - #2583
Conversation
The self-host Postgres connection pool is created with no `max` option, so it silently uses the pg driver's hardcoded default of 10 connections. That single pool is shared by every HTTP handler AND every queue worker's own database operations, including jobs that intentionally fan out several concurrent writes (e.g. hydrateMergedPullRequestFiles). A realistic burst -- a handful of concurrent job workers each running a fan-out step, plus normal webhook traffic -- can plausibly want more concurrent connections than the pool provides, well before Postgres's own connection ceiling or the existing GittensoryPostgresConnectionPressure alert would trip. There was no environment variable to raise this without a code change. - Add PGPOOL_MAX, wired into the pool construction in src/server.ts via a small extracted, directly-testable resolvePostgresPoolMax() in queue-common.ts (server.ts itself has no test infrastructure -- it boots a real server at module load and is Codecov-ignored for exactly that reason -- so the resolution logic needed to live somewhere side-effect-free to be unit-tested at all). - Default (10) matches the pg driver's own prior implicit default exactly, so this is a pure opt-in tuning knob with no behavior change when unset. - Documented in .env.example and the self-hosting backup/scaling doc, including the distinction from PgBouncer (which pools connections between instances and Postgres; this pools connections within one instance) and what to watch (GittensoryPostgresConnectionPressure) before raising it further. Validation: full local gate green; resolvePostgresPoolMax() is fully unit tested (default, override, invalid-value fallback). src/server.ts itself carries no Codecov patch-coverage obligation (already ignored in codecov.yml, alongside pg-adapter.ts/pg-queue.ts, as self-host process-entry code validated by the Docker build+boot smoke test instead).
|
Tip 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 ✅ Gittensory review result - approve/merge recommendedReview updated: 2026-07-02 10:40:00 UTC
✅ Suggested Action - Approve/Merge
Review summary Nits — 5 non-blocking
Review context
Contributor next steps
Signal definitions
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.
|
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
gittensory-ui | d6b5902 | Commit Preview URL Branch Preview URL |
Jul 02 2026, 10:20 AM |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2583 +/- ##
==========================================
+ Coverage 95.97% 95.98% +0.01%
==========================================
Files 226 229 +3
Lines 25533 25811 +278
Branches 9293 9389 +96
==========================================
+ Hits 24505 24775 +270
- Misses 417 425 +8
Partials 611 611
🚀 New features to boost your workflow:
|
Summary
Closes #2536.
The self-host Postgres backend (
src/server.ts'sbuildPostgresBackend) constructed itspg.Poolwith nomaxset, so it silently ran onpg's own hardcoded default of 10 connections shared across every HTTP handler and every queue worker's DB traffic in that instance — including jobs that fan out several concurrent writes (e.g.hydrateMergedPullRequestFiles). On a small/idle instance that's fine; at real volume (many registered repos, higherQUEUE_CONCURRENCY), a fixed pool of 10 becomes an app-side bottleneck well before Postgres's ownmax_connectionsor the existingGittensoryPostgresConnectionPressurealert would ever fire, and there was no way for an operator to raise it without a code change.What changed
resolvePostgresPoolMax()tosrc/selfhost/queue-common.ts, reading a newPGPOOL_MAXenv var via the module's existingparsePositiveIntEnvhelper (min 1, fallback 10 — the same defaultpg.Poolused implicitly before).src/server.ts'sbuildPostgresBackend:new pg.Pool({ connectionString: url, max: resolvePostgresPoolMax() }).PGPOOL_MAXin.env.exampleand in the self-host backup/scaling docs (docs.self-hosting-backup-scaling.tsx), explaining the distinction between this per-instance app pool and PgBouncer's pooling, and pointing operators at theGittensoryPostgresConnectionPressurealert as the signal to watch when raising it.The pool-sizing logic was extracted into
queue-common.tsrather than left inline inserver.tsbecauseserver.tsrunsmain()at module top level (unsafe to import in a test) and is explicitly excluded from Codecov's patch-coverage gate for exactly that reason —queue-common.tsis the already-proven-testable home for this kind of self-host config accessor.Correctness notes
PGPOOL_MAXunset still resolves to 10, matchingpg's own prior implicit default.parsePositiveIntEnv-based knob in this module.Validation
npm run typechecknpx vitest run test/unit/selfhost-queue-common.test.tsnpm run test:coverage(unsharded, full suite)npm run db:migrations:checknpm audit --audit-level=moderategit diff --checkScope
site/,CNAME,**/lovable/**, orCHANGELOG.mdSafety
PGPOOL_MAXis an operator-controlled env var validated the same way as existing pool/queue knobs