Skip to content

fix(server): stop fsyncing SQLite on every commit so slow disks no longer drop clients - #19205

Merged
t3dotgg merged 1 commit into
mainfrom
t3/sqlite-synchronous-normal
Oct 14, 2026
Merged

t3dotgg merged 1 commit into
mainfrom
t3/sqlite-synchronous-normal

Conversation

@t3dotgg

@t3dotgg t3dotgg commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Problem

Users keep seeing "reconnecting" while agents run. SQLite runs at the default synchronous = FULL, so every commit fsyncs the WAL on the main thread, and every provider event is its own commit. On a slow or busy disk (HDD, LUKS/btrfs, ext4 under build load, WSL with discard), one fsync takes 100 ms to several seconds. The event loop stops, heartbeats miss the timeout, and clients drop. The data is in #15707.

Change

Set PRAGMA synchronous = NORMAL with the other connection pragmas. In WAL mode this fsyncs at checkpoints, not on every commit.

Tradeoff: an app crash still loses nothing, and the database cannot corrupt. An OS crash or power loss can roll back commits made since the last checkpoint. That is the standard WAL setting, and macOS FULL already does not use F_FULLFSYNC (#13544).

Scope and approval

Maintainer change. It is the first option in the #15707 triage, and users have measured it on live servers. It does not move SQLite off the main thread (#18736, #14703), so slow reads (#14701) still stall.

Verification

  • User A/B on a live server in [Bug]: Orchestrator V2 event writes stall the server on a high-latency disk (156 ms fsync), environment flips Offline #15707: fsync time went from 17.6 s to 0.4 s per minute, the slowest single fsync from 6.7 s to 0.25 s, and the reconnects stopped. On the slow disk from the original report, median commit time went from 41.8 ms to 0.05 ms.
  • Local real-disk benchmark (ext4), 500 commits shaped like event writes: fsyncs went from 514 to 11, and median commit time from 1.85 ms to 0.60 ms.
  • New test in Sqlite.test.ts: it checks that a file-backed database opens at NORMAL. It fails without the change (expected 2 to equal 1). Sqlite.test.ts passes 5/5. Lint and format pass.
  • GPT-6.1 Sol reviewed the change and approved it. It confirmed that every production connection to statev2.sqlite goes through this setup.

Created with Claude Opus 5.5 in Claude Code, running in T3 Code. Reviewed with sol-loop: 1 round with GPT-6.1 Sol.

🤖 Generated with Claude Code

Closes #15707

…nger drop clients

Set PRAGMA synchronous = NORMAL. In WAL mode this only fsyncs at
checkpoints instead of on every commit, which ran on the main thread and
blocked heartbeats on slow or busy disks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XS 0-9 changed lines (additions + deletions). labels Oct 14, 2026
@juliusmarminge juliusmarminge added the macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews label Oct 14, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This changes SQLite's automatic durability setting for all server persistence connections, trading per-commit fsync latency for possible loss of commits since the last checkpoint after an OS or power failure. The production default change and its persistence tradeoff warrant human review.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 20.9 KiB 20.9 KiB 0 B (0.0%) 29.3 KiB ✅
Codex Live turn messages 2 2 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB −24 B (−0.5%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB −24 B (−2.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.2 KiB 21.2 KiB −41 B (−0.2%) 29.3 KiB ✅
Claude Live turn messages 2 1 −1 (−50.0%) 8 ✅

Baseline: aef0e9c · PR result: d72c261 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Path: .coderabbit.config.ts
  • Review profile: CHILL
  • Plan: Team
  • Run ID: 8b1682fb-1446-4f82-9388-3423f44e5d77
📥 Commits

Reviewing files that changed from the base of the PR and between aef0e9c and d72c261.

📒 Files selected for processing: 2
  • apps/server/src/persistence/Sqlite.test.ts
  • apps/server/src/persistence/Sqlite.ts

Included review availability: This review used your included allowance. 8 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 10 reviews per hour.

📝 Walkthrough

SQLite connections now use synchronous = NORMAL. A persistence test opens a temporary file-backed database and verifies that SQLite reports the setting as 1.

Changes: SQLite persistence

Layer / File(s) Summary
Set and verify synchronous mode
apps/server/src/persistence/Sqlite.ts, apps/server/src/persistence/Sqlite.test.ts
The shared SQLite setup sets synchronous = NORMAL. A test verifies the setting on a temporary file-backed database. Comments describe the WAL durability tradeoff.

Priority: ⬆️ High

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: otavio

Merge Risk: ⚪ Minimal · up to d72c2

This change trades some power-loss durability for lower commit latency, as intended. No merge-blocking issue is established by the supplied evidence.

Pre-merge checks: Passed 4
Passed Passed checks: 4 passed
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly and concisely describes the main change: changing SQLite commit durability behavior to prevent slow-disk client disconnects.
Description check Passed The description includes the required Problem, Change, Scope and approval, and Verification sections. It explains the durability tradeoff, approval context, test coverage, benchmark results, and limit…
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@t3dotgg
t3dotgg merged commit a74ce48 into main Oct 14, 2026
34 checks passed
@t3dotgg
t3dotgg deleted the t3/sqlite-synchronous-normal branch October 14, 2026 00:54
aorwall added a commit to aorwall/t3code that referenced this pull request Oct 14, 2026
Merges `pingdotgg/t3code` up to `6002c5c68c` (42 commits, base
`5070a83a32`), using a merge commit. No open merge PR to stack on; this
branches from `main` at `882bb0a638`.

## Resolution

- **`package.json`** (conflict): dropped upstream's `tsc-rs` 0.6.0 bump
(pingdotgg#19187). The fork removed `tsc-rs`, and the `tsc-rs` tripwire holds
that. New path-policy entry: `root-package-json` (`decide`).
- **`pnpm-lock.yaml`** (conflict, `theirs`): took upstream's copy. Its
seven `tsc-rs@0.6.0` entries fail `minimumReleaseAge` before resolution
can prune them, so I stripped them and let `install.mjs` re-derive the
rest. The fork edges (`moatless-api`, `mermaid`) are back, and `verify
--only lockfile` passes.
- 14 other files that both sides touched merged without conflict.
`duplicate-adds` and `resolution-check` are both clean.
`FilePreviewPanel.tsx` (`decide`) took upstream's `threadRef` prop and
kept both fork deltas. The two unlisted files now have `converged`
entries: `terminal-drawer-open-in-editor` and `thread-shell-subtasks`.
In `threadShell.ts`, upstream's `isChildRunLive` and the fork's listing
filter merged together correctly.
- File counts: 136 files landed, against 136 in the upstream range. The
only differences are `package.json` (missing, because its one upstream
change was the dropped bump) and `docs/fork/inventory.json` (extra,
fork-only). The fork delta is 809 files.
- **Script fix:** Moatless replaced its dispatch match arms with a
`MethodTable` (moatless#1138). Because of that,
`unsupported-methods.mjs` found no arms and exited 2. It now reads
`.detached`/`.session`/`.stream` registrations and follows handlers
passed by name. Result: ADD 0, DROP 0, KEEP 2 (`vcs.switchRef`,
`git.preparePullRequestThread`), which matches `gaps.md`.

## Verification

`verify.mjs`: these checks pass: duplicate-adds, tripwires,
resolution-check, unsupported-methods (after the fix), lockfile, fmt,
lint, typecheck, build. `test` fails. Each of the failing files also
failed when run alone, and they are the same five files as in the
2026-10-13 merge, all caused by the sandbox: shared `AgentScope`, server
`ServerBrowserPage` (no Playwright browser), `AcpAdapterV2`,
`OrchestratorReplayFixtures`, and `OpenCodeServerLedger`.

## Usable as-is

- Full-screen visual replies with zoom (pingdotgg#17561). Links inside HTML
renders open the way chat links do (pingdotgg#18897). App shortcuts keep working
after you click into an HTML render (pingdotgg#18898).
- Touch-scrolling in terminal output (pingdotgg#11305). Fixed: answering a
question reset the editor caret (pingdotgg#11337). Links in agent questions are
now clickable (pingdotgg#19197).
- Fixed: "Subagent needs input" stayed after the subagent finished
(pingdotgg#18996). This is client-side and uses `activityRunStatus` with a
fallback to `status`.
- Slash skill search now works mid-message (pingdotgg#19107). Fixed: removing a
project opened a new thread for it (pingdotgg#15864). Settings now warn when an
environment silently drops a setting (pingdotgg#18334).
- Smaller fixes: context handoff details use the available width
(pingdotgg#19098), pace icon alignment (pingdotgg#18580), tool-call collection perf
(pingdotgg#18853), git file paths keep whitespace in the contract schema
(pingdotgg#12465), Android beta links (pingdotgg#18955).
- Pending context-handoff preview when you pick a model (pingdotgg#18369). It is
computed on the client from `server.providers`
(`requiresNewThreadForModelChange`, `continuation.groupKey`). If
Moatless does not report those fields, the preview shows a handoff on
every provider switch.

## Unsupported in Moatless / needs implementation

- Delegated tasks in separate worktrees (pingdotgg#19102): `isolation:
"worktree"` on the orchestrator MCP `delegate_task`, and
`workspaceStrategy` on the child-launch command. Moatless does not
create worktrees (`FEATURES.worktreeSelection`).
- These land behind flags that are already gated: scheduled-task dialog
fix (pingdotgg#18940, `scheduledTasks`), PR merge/auto-merge button sizing
(pingdotgg#18478, pingdotgg#19111, `pullRequestSurface`), remembering editor selections
(pingdotgg#19078, `openInEditor`), device tool startup failures (pingdotgg#18502,
`deviceHub`), npm "update everything" (pingdotgg#18709, `serverUpdateBanner`),
clone folder names in the welcome wizard (pingdotgg#12474), Cmd+Enter
create-project in the palette (pingdotgg#18901).
- No new RPC methods (ADD 0).

## Backend behavior to consider reproducing in Moatless

I added these to the "Runtime fixes upstream made to its own server"
entry in `docs/fork/gaps.md`:
- Worktree isolation for delegated tasks (pingdotgg#19102).
- Snoozed threads still take queued automatic wakes (pingdotgg#19099).
- Indexed failed-thread error lookup (pingdotgg#18628).
- SQLite: no fsync on every commit, and planner stats collected on open
(pingdotgg#19205, pingdotgg#18471).
- Distinct CI checks with overlapping names (pingdotgg#19162).
- Bitbucket short-hash heads in review guides (pingdotgg#18900).
- Webhook template backtracking (pingdotgg#19160).
- OpenCode timeout formatting (pingdotgg#12469).
- Pi tool schemas and turn-start queueing (pingdotgg#18643, pingdotgg#18639).
- Usage hubs without a host, and Antigravity accounts deduplicated
(pingdotgg#11368, pingdotgg#19207).
- Haiku 5.5 as the default for background text generation (pingdotgg#18781).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---
Moatless task:
https://moatless.soaplabstest.com/tasks/db18f6dc-70c1-4d89-ada7-cd4b4f19a247
hezhaoqian1 added a commit to hezhaoqian1/t3code that referenced this pull request Oct 14, 2026
Every provider event is its own commit, and synchronous=FULL fsyncs the WAL
on the main thread each time. On a slow or antivirus-scanned disk that stalls
the event loop and the desktop shows "reconnecting". In WAL mode NORMAL only
fsyncs at checkpoints: an app crash loses nothing and the database cannot be
corrupted; a power loss can roll back the newest commits.

Port upstream a74ce48 (pingdotgg#19205).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AdEx-Partners-DE pushed a commit to AdEx-Partners-DE/t3code that referenced this pull request Oct 14, 2026
…nger drop clients (pingdotgg#19205)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a74ce48)
github-actions Bot added a commit to omarcresp/t3code-flake that referenced this pull request Oct 14, 2026
## What's Changed
* fix(claude): default background text generation to haiku 5.5 by @maria-rcks in pingdotgg/t3code#18781
* chore(typecheck): bump tsc-rs to 0.6.0 by @t3dotgg in pingdotgg/t3code#19187
* fix(server): keep OpenCode timeouts from crashing the error formatter by @tachytelicdetonation in pingdotgg/t3code#12469
* feat(server): run delegated tasks in separate worktrees by @Bil0000 in pingdotgg/t3code#19102
* fix(git): preserve whitespace in file paths (#12464) by @Defmon3 in pingdotgg/t3code#12465
* fix(server): stop fsyncing SQLite on every commit so slow disks no longer drop clients by @t3dotgg in pingdotgg/t3code#19205
* fix(chat): make links in agent questions clickable by @t3dotgg in pingdotgg/t3code#19197
* fix(usage): show one Antigravity account once, not once per machine by @t3dotgg in pingdotgg/t3code#19207
* fix(web): Android beta links point to the APK release, not the banned Google group by @shivamhwp in pingdotgg/t3code#18955
* fix(client): "Subagent needs input" no longer sticks after the subagent finishes by @shivamhwp in pingdotgg/t3code#18996
* fix(web): stop question answers from resetting the editor caret by @shivamhwp in pingdotgg/t3code#11337
* fix(web): app shortcuts keep working after clicking into an HTML render by @maria-rcks in pingdotgg/t3code#18898
* fix(source-control): keep distinct CI checks with overlapping names by @Lucenx9 in pingdotgg/t3code#19162
* fix(server): avoid excessive backtracking in webhook templates by @Lucenx9 in pingdotgg/t3code#19160
* fix(web): auto-merge button fits its label and animates width changes by @flamboh in pingdotgg/t3code#19111
* fix(web): scroll terminal output with touch gestures by @shivamhwp in pingdotgg/t3code#11305
* feat(web): view visual replies full screen with zoom by @Bil0000 in pingdotgg/t3code#17561
* fix(web): removing a project no longer opens a new thread for it by @shivamhwp in pingdotgg/t3code#15864
* fix(server): usage hubs with a host-less URL no longer vanish from Limits by @shivamhwp in pingdotgg/t3code#11368
* fix(mobile): search bar animates in on iOS 26+ instead of popping by @juliusmarminge in pingdotgg/t3code#19209

## New Contributors
* @Defmon3 made their first contribution in pingdotgg/t3code#12465

**Full Changelog**: pingdotgg/t3code@v0.0.46-nightly.20261013.3080...v0.0.46-nightly.20261014.3093

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.46-nightly.20261014.3093
github-actions Bot added a commit to davidvanderklay/t3code-flake that referenced this pull request Oct 14, 2026
## What's Changed
* fix(claude): default background text generation to haiku 5.5 by @maria-rcks in pingdotgg/t3code#18781
* chore(typecheck): bump tsc-rs to 0.6.0 by @t3dotgg in pingdotgg/t3code#19187
* fix(server): keep OpenCode timeouts from crashing the error formatter by @tachytelicdetonation in pingdotgg/t3code#12469
* feat(server): run delegated tasks in separate worktrees by @Bil0000 in pingdotgg/t3code#19102
* fix(git): preserve whitespace in file paths (#12464) by @Defmon3 in pingdotgg/t3code#12465
* fix(server): stop fsyncing SQLite on every commit so slow disks no longer drop clients by @t3dotgg in pingdotgg/t3code#19205
* fix(chat): make links in agent questions clickable by @t3dotgg in pingdotgg/t3code#19197
* fix(usage): show one Antigravity account once, not once per machine by @t3dotgg in pingdotgg/t3code#19207
* fix(web): Android beta links point to the APK release, not the banned Google group by @shivamhwp in pingdotgg/t3code#18955
* fix(client): "Subagent needs input" no longer sticks after the subagent finishes by @shivamhwp in pingdotgg/t3code#18996
* fix(web): stop question answers from resetting the editor caret by @shivamhwp in pingdotgg/t3code#11337
* fix(web): app shortcuts keep working after clicking into an HTML render by @maria-rcks in pingdotgg/t3code#18898
* fix(source-control): keep distinct CI checks with overlapping names by @Lucenx9 in pingdotgg/t3code#19162
* fix(server): avoid excessive backtracking in webhook templates by @Lucenx9 in pingdotgg/t3code#19160
* fix(web): auto-merge button fits its label and animates width changes by @flamboh in pingdotgg/t3code#19111
* fix(web): scroll terminal output with touch gestures by @shivamhwp in pingdotgg/t3code#11305
* feat(web): view visual replies full screen with zoom by @Bil0000 in pingdotgg/t3code#17561
* fix(web): removing a project no longer opens a new thread for it by @shivamhwp in pingdotgg/t3code#15864
* fix(server): usage hubs with a host-less URL no longer vanish from Limits by @shivamhwp in pingdotgg/t3code#11368
* fix(mobile): search bar animates in on iOS 26+ instead of popping by @juliusmarminge in pingdotgg/t3code#19209

## New Contributors
* @Defmon3 made their first contribution in pingdotgg/t3code#12465

**Full Changelog**: pingdotgg/t3code@v0.0.46-nightly.20261013.3080...v0.0.46-nightly.20261014.3093

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.46-nightly.20261014.3093
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews size:XS 0-9 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Orchestrator V2 event writes stall the server on a high-latency disk (156 ms fsync), environment flips Offline

2 participants