Reduce archived conversation disk usage with cold storage

[Quicksaver]

Summary

With moderate use, .t3 can grow past 30 GB because archiving a conversation leaves its full event history, projections, diagnostics, and attachments in hot storage. Deleting conversations also leaves storage that needs later cleanup. Archived conversations still need to be recoverable, so simply discarding that data is not an option.

This moves archived SQL rows and attachments into bounded, gzip-compressed chunks in a separate archive.sqlite database. state.sqlite keeps only the archived thread shell needed by the Archive page and command receipts needed for idempotent retries. Unarchive restores the conversation before committing the domain command, while permanent deletion removes both hot and cold data.

The result is much less hot-database growth without weakening diagnostics for active conversations. A durable lifecycle queue recovers interrupted work, reclaims freed pages, and keeps the server, web app, and mobile app in sync as conversations move between active, archived, restored, and deleted states.

Reopened

This reopens #4016. That PR was closed because its relationship to #2829 was only tangential. The behavior here is independent. This branch continues to track main, and #2829 can be integrated here later if it lands first.

Together with #7387 and #4384, this is useful now and does not need to wait for orchestration v2.

What changed

• Added ThreadColdStorage, which writes thread-scoped SQL rows and attachments as validated, gzip-compressed chunks in archive.sqlite. It preserves binary values, pages reads, accepts compatible older schemas, and rejects unknown tables, chunk kinds, or unsafe attachment names.
• Kept only the archived shell and command receipts hot. Archive removes the event stream, messages, activities, turns, checkpoints, plans, sessions, runtime rows, and content attachments from state.sqlite; provider diagnostics and terminal logs are deleted rather than restored later.
• Moved archive, restore, and delete work into a durable per-project lifecycle queue. Tree locks cover eligibility checks and provider, terminal, log-writer, and preview quiescence so stale jobs cannot disrupt a conversation that was unarchived while waiting.
• Made archive creation and restore retry-safe across failures and restarts. cleanup_pending and restored manifests preserve the operation boundary, abandoned reservations recover on startup, archive epochs prevent stale finalization from blocking a later archive, transient restore failures roll back partially restored tree members while remaining retryable with the same command ID, and interrupts cannot trigger rollback after an accepted receipt commits.
• Restored or reserved storage before dispatching unarchive. Failed commands move the rows and files cold again, successful commands finalize only the bundle restored by that request, and receipt replay retries bundle cleanup without publishing a duplicate event.
• Made permanent deletion remove the hot shell, projections, receipts, cold bundle, attachments, terminal history, and exact provider-log rotations. Cleanup metadata remains durable until filesystem cleanup and page reclamation finish, and project deletion applies the same path to archived-only members without bypassing live-thread checks.
• Added lifecycle migrations for existing archived and soft-deleted conversations, plus compatibility reruns for databases that already recorded conflicting upstream migration IDs. The branch keeps upstream title, pinning, turn-index, project-environment, and favicon migrations under non-conflicting IDs.
• Added one-time hot-database compaction after legacy queues drain. It remains pending while archive or delete work is still discoverable, then later lifecycle operations use bounded incremental free-page reclamation for both databases.
• Added the Archive settings view and made web and mobile lifecycle actions consistent. Partial environment failures remain visible alongside successfully loaded archive groups; archived-row menus use typed action metadata for Unarchive and a separated, destructive Delete action across web and desktop; optimistically archived threads disappear from sidebar, project ordering, keyboard navigation, drafts, and command-palette actions; duplicate archive and unarchive operations keep the correct pending state; invalid mobile archive dates sort last.
• Made the socket-owned shell snapshot authoritative after live buffering begins and when the app returns to the foreground. Web and mobile detail caches use eviction tombstones, generation checks, and retryable cleanup so archived threads cannot reappear from stale local data.
• Closed server and renderer preview state at archive and delete boundaries. Unarchive clears stale previews again, while the renderer preserves environment baselines long enough to reconcile removals and release capture and desktop-tab leases.
• Added t3-sqlite-state --database archive as the supported inspection path for cold manifests and chunks.

Validation

• pnpm exec vp check passed with 9 pre-existing lint warnings outside the changed files.
• pnpm exec vp run typecheck passed.
• pnpm exec vp run lint:mobile passed; optional SwiftLint, ktlint, and detekt checks were skipped because those tools were not installed locally.
• Focused tests passed for cold storage, deletion, migrations, orchestration, SQLite inspection, shell and detail-cache synchronization, sidebar and project deletion, mobile archive sorting, optimistic archive concurrency, command-palette behavior, and preview cleanup. Recorded review runs included 6 files with 79 tests, 3 files with 52 tests, 14 ThreadDeletionReactor tests, and 1 command-palette file with 2 tests.
• Changed-file formatting, lint, and type-aware diagnostics passed. Server, mobile, and client-runtime typechecks passed, as did targeted web typechecking; the package-wide web check still reports unrelated existing diagnostics in five untouched components.
• Isolated Playwright runs verified archive removal from rendered and keyboard-addressable sidebar state, authoritative shell synchronization, Archive route loading, project ordering, and command-palette exclusion without console or page errors.
• An isolated controlled-browser pass verified parsed cross-offset archive ordering and a rapid duplicate unarchive on web. A disposable Android emulator pass verified the same ordering and rapid duplicate unarchive, with exactly one projection restore; two pre-existing disconnected mobile environments produced an unrelated archive-refresh error after the verified operation.
• The iOS development client built, installed, and launched on an iPhone 17 Pro simulator. The archive-screen UI pass was blocked before application code by this worktree's Metro resolution of strict pnpm symlinks; focused mobile archive-list tests cover both sort directions.
• pnpm exec vp test run apps/web/src/contextMenuFallback.test.ts apps/desktop/src/electron/ElectronMenu.test.ts passed (2 files, 17 tests).
• pnpm exec vp test run apps/server/src/orchestration/Layers/OrchestrationEngine.test.ts apps/server/src/orchestration/ThreadColdStorage.test.ts apps/web/src/components/settings/ArchivedThreadsPanel.environment.test.tsx passed (3 files, 41 tests).
• Changed-file vp check passed for all 6 review-fix files; type-aware and type-check lint reported only one pre-existing string-spread warning in ThreadColdStorage.ts.
• An isolated Playwright pass loaded the authenticated Archive route with its navigation, breadcrumb, and empty state and reported zero console errors. The partial-environment error branch is covered by the focused component regression.

🤖 Generated by GPT-5 in Codex via T3 Code

---

Note

High Risk
Touches orchestration commands, SQLite migrations, filesystem cleanup, and data recovery paths for archived threads—errors could cause data loss or stuck lifecycle state.

Overview
Archived conversations move off hot state.sqlite into gzip-compressed chunks in archive.sqlite, with a durable background lifecycle queue and coordinated unarchive/restore semantics.

ThreadColdStorage archives thread projections, events, and attachments while keeping a lightweight archived shell and hot command receipts for idempotent retries; provider/terminal logs are dropped on archive rather than bundled. ThreadDeletionReactor now schedules archive, delete, and legacy-compaction jobs with provider/preview/terminal quiescence, coalesced retries, and startup rediscovery of pending work. OrchestrationEngine restores (or rolls back) cold bundles around thread.unarchive inside an uninterruptible commit boundary and retries bundle finalization on accepted receipt replay.

Tooling and clients: deriveServerPaths exposes archiveDbPath; t3-sqlite-state gains --database archive (docs updated). Mobile archive list treats invalid archive timestamps as absent, sorts them last, and blocks duplicate unarchive via per-thread keys; mobile SQLite schema v2 clears persisted thread-detail cache on upgrade. Tests and harness layers wire ThreadColdStorage (or no-op) across orchestration suites.

^{Reviewed by Cursor Bugbot for commit d7b897c. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add cold storage for archived conversations to reduce disk usage

• Introduces ThreadColdStorage service on the server that archives thread event trees to a gzip-compressed archive.sqlite, restores them on unarchive with rollback on failure, and deletes them permanently on thread.deleted
• Reworks ThreadDeletionReactor to process archive/delete as idempotent lifecycle jobs with quiescence checks and retry loops; OrchestrationEngine now restores cold-storage data pre-commit before thread.unarchive
• Adds client-side threadCache.ts with generation tokens, eviction on archive, and revival on unarchive; the shell and detail state coordinate cache persistence and evict stale entries when threads disappear from snapshots
• Adds migrations 035–044 creating thread_archive_manifests and thread_cleanup_queue tables and backfilling existing archived/deleted threads; bumps client IndexedDB to v5 and mobile SQLite to v2, both clearing cached thread data on upgrade
• Adds UI support: ArchivedThreadsPanel for browsing/unarchiving, optimistic archive hiding via useOptimisticThreadArchiveStore, sidebar/command-palette filtering, deleteArchivedThreads flag in project.delete, and --database archive on t3-sqlite-state CLI
• Risk: migration 035 backfills thread_archive_manifests for all existing archived threads — servers with many archived threads will see a large pending queue on first boot. The t3-sqlite-state CLI now defaults --database to state; scripts inspecting archives must pass --database archive. Client DB upgrades from <v5 (web) or <v2 (mobile) clear all cached thread data, causing a one-time refetch.

^{Macroscope summarized d7b897c.}

[Quicksaver]

@macroscopeapp Addressed the newly reported command-palette fallback-search blocker in 0d740e0. The rendered Remote fallback is now included in project search metadata, with focused regression coverage; the inline thread has a direct response and is resolved.

🤖 Generated by GPT-5 in Codex via T3 Code

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 65c62f4. Configure here.}

[t3dotgg]

Note

🤖 GPT-5.6 Sol responding on behalf of Theo

Closing this PR after an automated pass over open pull requests. Adds a second SQLite database, archive compression, migration, and restore systems across 94 files.

[Quicksaver]

@t3dotgg could use an explanation for the close, that's just a list of spec changes the PR does 🙃 unwanted? superseded? redundant?