fix(code-index): preserve and bound live migration replay

[HashemKhalifa]

Summary

Retained pre-paging evidence manifests can now be authenticated and restored without allocating owned identity maps for every symbol and chunk in the corpus. Encountered legacy markers resolve directly against canonical descriptors and decoded files. The full manifest parser and streaming retention reader share bounded layout validation while retaining separate authentication responsibilities; missing pages identifies historical format, and explicit null is rejected.

Semantic generation restart and publication use fresh, finite 15-minute background authority per phase. Tests cover the exact budget and lifecycle cancellation during snapshot refresh. The existing lexical cursor preservation and adaptive subdivision of refused page batches remain included.

Validation of the follow-up fixes

• Codec: 10 unit tests and 2 integration tests passed.
• Semantic vector generations: 31 tests passed, including cancellation and fresh per-phase authority.
• Formatting and diff whitespace checks passed.
• Two scoped commits: 74da6279f (codec) and 316e8e73f (semantic authority).

Earlier live evidence

The earlier revision resumed a retained generation, completed 3,863 source files, and remounted the verified generation after restart without replay. This is historical evidence; matching-runtime and BOT CLI/MCP verification of the follow-up revision is still in progress.

Acceptance gaps

• A genuine historical legacy-fixture peak-RSS measurement remains outstanding. Removing pages from a current-writer fixture is not valid historical evidence.
• Approved, green fork workflows remain outstanding; a skipped integrity check is not CI acceptance.
• Strict semantic profile activation and BOT CLI/MCP usability on the follow-up runtime are not yet claimed complete.

Follow-up validation for #707 and #792; shared validation addresses #886. Related: #753 and #837.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 3c83c9e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-05T05:07:34.357629Z	a239f44	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a239f44418

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ScriptedAlchemy]

Architecture review of a239f44418f61293eaba19714827775123685c91 in the context of #707.

Filed #886 for a concrete simplification exposed by this patch: parse_partitioned_manifest and partitioned_segment_identities_from_reader independently validate the same evidence descriptor invariants, so this migration now adds its legacy/paged decision in both places. Share a small private descriptor validator, not the whole reader. Retention's caller-authenticated metadata projection must remain distinct from serving authentication and must not start decoding the full generation.

The existing inline review already covers the two more serious concerns, so I am not filing duplicate issues for them: the corpus-wide owned identity maps during legacy restore, and replacing the interactive deadline with a very long background budget without a cost/progress proof. Keep the memory acceptance attached to #811/#852 and the semantic lifecycle acceptance attached to #753/#837. Successful completion on the reported 3,863-file profile is useful evidence, but it does not by itself bound peak memory or prove prompt cancellation of a stalled restart.

For the compatibility acceptance in #886, retain historical-writer fixture bytes with real marker-bearing evidence. The added test formed by removing pages from a current manifest is useful for descriptor compatibility, but is not by itself evidence for every historical identity-restoration case. Preserve the supported live format; do not delete recovery support merely because it is called legacy.

Source/diff review only; I did not run the reported tests or reproduce the operator journey.

[ScriptedAlchemy]

Two P1s remain unresolved and should block this PR from landing into #707:

1. Legacy restore must not allocate corpus-wide identity maps. The current compatibility decoder retains the decoded files + evidence DOM and then builds owned BTreeMap<String, String> entries for every symbol/chunk. Replace that with marker parsing/direct indexed resolution (or another demonstrably bounded lookup) so migration cost is proportional to evidence actually encountered, not total corpus identities. This is especially important because #852 already establishes that simultaneous generation representations are the dominant memory risk.

2. Do not replace a 30s timeout with the 168-hour background budget. Corpus-scale restart can have a different budget from an interactive request, but it still needs a measured finite bound plus progress/cancellation checkpoints. The acceptance should prove a stalled phase releases the semantic worker promptly; successful completion of the 3,863-file operator profile proves the happy path, not bounded failure behavior.

Also complete #886 while touching this codec: share the common paged/legacy descriptor validation between the full parser and streaming retention reader without merging their authentication responsibilities.

The missing changeset is not independently concerning if this PR is intentionally absorbed into unreleased #707 before release; do not add release metadata merely to silence the bot in that case.

After the fixes: rerun the focused codec/usecase tests already listed in the PR, add a peak-memory measurement for the legacy migration fixture, add a cancellation/stall test for restart, and get the forked-workflow runs approved/green. Current workflow results are action_required, so there is no CI evidence on this head yet.

[ScriptedAlchemy]

Current-head deep review (316e8e73f). The earlier corpus-wide identity-map P1 and 168-hour authority P1 have been materially addressed: legacy markers now resolve directly from persisted indices, descriptor layout validation is shared, explicit null is rejected, and background graph authority is finite at 15 minutes. Those prior objections should not be carried forward unchanged.

One new merge blocker remains in the current branch-add wait implementation: BRANCH_GENERATION_SEATING_POLL_INTERVAL = 10ms drives the whole exact-generation wait loop, and the loop calls dashboard_freshness(...).await before every select. A long but healthy generation can therefore execute ~100 freshness probes/sec for up to 30 minutes (~180k probes per waiter), plus serving-scope reads, with unrelated publication wakeups causing more iterations. Multiple branch-add callers multiply that background demand. The comment says polling exists only to bridge the narrow publication-before-seating race; scope it to that race instead of polling the entire generation lifetime. Prefer a seating/serving-generation notification tied to the existing owner; otherwise only start a bounded, slower confirmation poll after a matching publication event. Keep the hard deadline and idle-progress semantics.

I would also keep the PR's own stated acceptance gaps explicit blockers for claiming #707 live-migration acceptance: green fork workflows on this exact head, matching-runtime BOT CLI/MCP verification, and at least one genuine historical pre-paging fixture peak-RSS measurement. The synthetic 'remove pages from current writer' fixture proves descriptor compatibility, not historical memory behavior.

The 15-minute corpus authority is now qualitatively correct (finite + lifecycle cancellation), but the acceptance should record the measurement that justifies 15 minutes rather than only asserting the constant in a unit test. A constant-equality test protects configuration, not the cost model.

[ScriptedAlchemy]

Addressed the branch_add seating-poll finding in c6130cb.

Root cause. The scheduler broadcasts a generation publication when reconcile seals (registry.rs, publish_generation in the worker loop), which happens before the sealed generation takes the serving slot (ServingSwapOutcomeV1 swap, several hundred lines later). await_exact_branch_generation needs the seated slot, and no second publication follows the seat — so the only way it could observe the transition was to poll for it. That is what BRANCH_GENERATION_SEATING_POLL_INTERVAL (10ms) was doing, for up to the 30 minute hard deadline: ~180k iterations per waiter, each doing two async registry lookups.

Fix. CodeIndexSchedulerRegistryV1 now carries a tokio::sync::watch seating counter (serving_seats / subscribe_serving_seats) advanced after the serving slot is written, in both seat paths:

• the reconcile serving swap, when ServingSwapOutcomeV1::installs()
• the ignored-dependency serving swap

branch_add subscribes before it requests the refresh and awaits that transition alongside the publication stream. The remaining timer branch is sleep_until(idle_deadline.min(hard_deadline)), so both existing bounds still fire and neither is polled for. A closed seat channel disables that select branch rather than spinning; the deadline branch still bounds the wait.

I reused the existing publication broadcast rather than adding a second one — it fires too early for this waiter by design, so a distinct seating signal was needed. No existing notify/watch covered seating (grep -rn "serving_generation_epoch" shows the two seat sites bump only an AtomicU64, which is not awaitable).

Evidence.

New test serving_seat_wake_arrives_only_after_the_slot_is_seated (crates/tracedecay-code-index-runtime/src/code_index_scheduler/tests.rs) subscribes to the seat counter before mounting, then asserts the wake arrives and that the serving slot is already populated at that moment — the invariant that makes the poll unnecessary. It fails if anyone moves the signal back before the seat.

test code_index_scheduler::tests::serving_seat_wake_arrives_only_after_the_slot_is_seated ... ok
test result: ok. 1 passed; 0 failed; ... finished in 0.37s

0.37s against the 10ms×N poll it replaces, with no poll interval left in the loop.

New test the_hard_deadline_bounds_every_seating_wake covers the deadline selection, including an idle deadline extended past the hard bound:

test daemon::branch_add::wait_policy_tests::the_hard_deadline_bounds_every_seating_wake ... ok
test daemon::branch_add::wait_policy_tests::pending_graph_activation_keeps_exact_branch_wait_live ... ok
test result: ok. 2 passed; 0 failed

cargo clippy -p tracedecay -p tracedecay-code-index-runtime --all-targets -- -D warnings — 0 errors, 0 warnings. That required fixing two pre-existing clippy failures in branch_add.rs (items_after_test_module, duration_suboptimal_units); both reproduce on 316e8e7 without my change and would block CI on their own.

cargo test -p tracedecay --features test-helpers,test-transport --test daemon_suite git_watch_test fails 3 of 5 (concurrent_reconciliations_converge_on_one_sealed_generation, linked_worktree_requires_mount_then_serves_only_its_exact_generation, one_reconciliation_covers_a_fifty_commit_frontier). Identical failing set on 316e8e7 with the change stashed — pre-existing on this branch (indexing does not converge in time on a loaded box), not a regression from this commit. Flagging rather than fixing, as it is outside this finding.

[ScriptedAlchemy]

Validation on head c6130cb (2026-09-05)

Item	Verdict	Evidence
Seating wake tests	PASS	serving_seat_wake_arrives_only_after_the_slot_is_seated 1 passed (code-index-runtime); daemon::branch_add 2 passed (the_hard_deadline_bounds_every_seating_wake, pending_graph_activation_keeps_exact_branch_wait_live)
cargo clippy --workspace --all-targets --locked -- -D warnings	PASS	0 warnings, 4m39s
Exact-runtime CLI/MCP generation identity	PASS	isolated daemon (scripts/with-isolated-tracedecay-daemon.sh), tracedecay_runtime, CLI tracedecay_search and MCP tools/call tracedecay_search all report the same generation.v1.8ece8036.00000001…, freshness current/fresh, exact/graph/lexical coverage complete
Semantic stage on exact runtime	UNVERIFIED	semantic stage reports calibration_unavailable until tracedecay semantic activate runs; activation evaluation exceeded the 10 min probe window. Re-running with a longer budget.
Legacy (pre-paging) restore peak RSS	FAIL	no historical writer fixture exists on this head; scaled synthetic harness: 500 files → generation 21.2 MB, legacy VmHWM delta 52,428 KiB (2.53×), paged 256 KiB; 2000 files → 85.0 MB, legacy 221,488 KiB (2.67×), paged 272 KiB. Legacy restore is linear in corpus: decode_legacy_generation_evidence reads the whole evidence segment (whole_reads=2001), parses a full serde_json::Value DOM, mutates identities in place, then from_values. The identity-map P1 is fixed (index-based lookups); the remaining cost is the DOM. Extrapolated to the 3,863-file operator profile: ~430 MiB transient.
Trial merge onto codex/tracedecay-total-redesign-plan-reopened (e5dffc9)	one cosmetic conflict	code_index_scheduler/registry.rs: base renamed publish_generation → broadcast_generation_publication; #882 inserted subscribe_serving_seats/record_serving_seat above it. Keep both, use the new name.

Not merging yet. Next: stream legacy evidence without the DOM so restore RSS stays bounded, keep the RSS harness as a real bounded test, then re-run the semantic activation probe.

[ScriptedAlchemy]

Semantic activation probe (run 3, after waiting for index readiness): tracedecay semantic activate is rejected with hybrid-conservative:train fallback bytes changed, and tracedecay_runtime reports semantic_runtime.state=degraded reason=invalid_runtime_status before any activation. CLI and MCP search agree on the exact generation but semantic stays calibration_unavailable. Filed as #905; the exact-runtime semantic validation item stays open until that is fixed.

[ScriptedAlchemy]

Legacy restore blocker addressed in 4f9091548, with the base merged in 3c83c9e7a.

Root cause. decode_legacy_generation_evidence read the whole pre-paging evidence segment into a Vec<u8>, parsed a full serde_json::Value DOM from it, rewrote identity markers in that tree, then from_valued into the typed payload. Peak RSS was segment + DOM + payload and grew with the corpus — reproduced at 2.348x the on-disk generation at 500 files and 2.370x at 2000, i.e. ~5.5x the evidence segment itself in both cases.

Fix. Two changes, both reusing what the paged form already does:

1. The pre-paging segment is now read through PartitionedEvidencePageReaderV1. It carries no page table, so the reader ranges it in 256 KiB chunks and authenticates it with the aggregate segment digest it always had — there are no per-page digests to skip. verify_partitioned_sealed takes the same path, so verification no longer materializes a legacy segment either (that branch is now deleted rather than special-cased).
2. Identity markers are restored by a serde transcoder wrapped around that reader, so the typed payload is built directly from the stream. The classification rules are the replaced DOM walk's, unchanged: a string is substituted by the object key that encloses it, reset at every object member and inherited through arrays. An invalid symbol or chunk key still fails with its original contract message, carried out of serde's error type rather than stringified into it.

Numbers. Measured as peak RSS (VmHWM) growth over a baseline warmed by a paged restore of the same generation. Warming isolates the pre-paging path's own cost: the restored generation's own footprint (~1.2x the on-disk size, unavoidable — decode_partitioned_sealed returns it) is already in the allocator, so what is left is the extra the legacy path asks for.

files	generation on disk	evidence segment	before	after
300	12,750,156 B	5,402,171 B	1.279x (15.9 MB)	0.232x (2.9 MB)
1500	63,747,800 B	27,005,771 B	1.272x (79.2 MB)	0.168x (10.5 MB)

The ratio now shrinks as the corpus grows instead of holding constant, which is the linearity the review flagged. Absolute peak for a full restore at 500 files went from 47.6 MB to the paged form's own ~24 MB.

Test. The measurement harness is a real bounded test now, not an #[ignore]d probe: legacy_generation_restore_does_not_materialize_its_evidence_segment builds a 300-file generation, rewrites its descriptor into the pre-paging shape, and fails if the legacy restore's peak growth reaches half the on-disk generation size. It takes its reading in a child process because VmHWM is process-wide, and returns early where /proc/self/status and clear_refs are unavailable. Confirmed it fails on the pre-fix code (1.279x and 1.272x against a 0.5x bound) and passes after (0.232x / 0.168x, stable across runs).

partitioned_codec_reads_pre_paging_evidence_descriptor now asserts the read shape this produces — no whole-segment read of the evidence, and no single read larger than one page — instead of the previous "exactly one whole read". Its identity assertion (a legacy manifest restores byte-identically to the paged one) is unchanged.

Verification. cargo test -p tracedecay-code-index green (all suites). cargo clippy --workspace --all-targets --locked -- -D warnings clean. cargo test -p tracedecay-code-index-runtime --lib has two failures — registry_feeds_publications_and_bounded_freshness_reads and semantic_schedule_order_tests::panicking_semantic_hook_does_not_retire_later_reconciliation, both Elapsed(()) waiting on a generation publication — that reproduce identically on origin/codex/tracedecay-total-redesign-plan-reopened at e5dffc9cc with none of this branch's work, so they are pre-existing on the base and not from this change or the merge.

Merge. 3c83c9e7a merges the base. The only conflict was registry.rs, where the base renamed publish_generation to broadcast_generation_publication and this branch inserted subscribe_serving_seats / record_serving_seat immediately above it; both sides are kept, with the base's new name and its doc comment.