perf(memory): scan graph source once per settled reconciliation

[ScriptedAlchemy]

Problem

Project memory graph projection (project_memory_graph) and reconciliation (reconcile_project_memory_graph_pass → finish_reconciliation_watermark) each scanned the full owner source twice per operation, including every payload_json and event_json. The second scan existed only to recompute the ID-derived source_watermark and compare it against the first — but the watermark's MENTIONS relations derive from payload entities, so "just rehash IDs" still forced a full payload reload and parse. On a live daemon with a large project memory this doubles source work, bytes materialized, and transient RSS on every settled pass and every graph read.

Change (runtime-core only)

load_source now also captures the highest memory_v2_lineage_events.event_sequence inside the same read snapshot (LoadedMemoryGraphSource.lineage_stamp). Both recheck sites first ask source_unchanged_since, a single indexed-scalar read on a fresh snapshot:

• stamp unchanged → the source is provably identical, the recheck settles without reloading anything (this is the old watermark-match path; memory_graph_reconciliation_pending() remains consulted only on the mismatch/fallback path, where it has always suppressed the conflict);
• stamp moved or absent → full load_source + watermark compare, byte-identical to the previous behavior.

Why the stamp is a sound change token: memory_v2_lineage_events is append-only (schema triggers reject UPDATE/DELETE) with an AUTOINCREMENT sequence that is never reused, and every committed mutation that can change the projected source — new facts, assertions, evidence, payload rows, curated relations, active-assertion or payload-access transitions — records at least one lineage event in the same transaction on both fact write paths (the facade commit_fact_tx, which derives every memory_v2_current_facts change from batch.events(), and the storage-runtime FactExecutor::execute_write, which rejects event-less batches). Source-neutral writes (retrieval/feedback counters, superseded-payload purges) never touch rows the source reads; unrelated writes only advance the stamp and trigger the conservative fallback.

The durable graph_transaction_generation metadata counter was considered and rejected: it is only bumped by the Database facade commit and is bypassed by the storage-runtime writer path, so it cannot prove source stability.

Evidence (isolated crate tests, production journeys, tempdir fixtures only)

store::memory::graph_reconciliation_tests::reconciliation_pass_loads_large_payload_source_once seeds 8 facts with 16 KiB payload_json each, baselines one canonical source load via an unmounted publication pass (which records exactly one load before failing at the graph mount), then runs one settled publish_project_memory_graph_after_write pass and asserts it performs exactly one load through the same telemetry:

metric per settled pass	before	after
source rows loaded	48 (2× per-load 24)	24
source bytes loaded	278,656 (2× per-load 139,328)	139,328

Before-numbers captured by temporarily forcing the fallback on this branch; the test fails on the legacy double-load behavior (left: 48, right: 24).

store::memory::graph_reconciliation_tests::source_mutation_during_publication_conflicts_and_republishes parks a publication inside the verified-graph reconcile, commits a fact mid-pass (which settles through the ordinary write journey with exactly one load), then releases the parked pass: with a stale stamp, a stale watermark, and nothing pending, the finish reloads the canonical source, surfaces GraphConflict, and the publisher schedules exactly one republication — the post-release tail is exactly two loads of the mutated source (the conflicted finish's fallback reload plus the scheduled republication's own load).

Verification

• cargo test -p tracedecay-runtime-core --lib store::memory — 129 passed, 0 failed.
• cargo clippy -p tracedecay-runtime-core --all-features --tests — clean; cargo fmt -p tracedecay-runtime-core --check — clean.
• Pre-existing, unrelated on this VM: store_runtime::resolver::tests::project_graph_locator_...fails_closed fails on the clean base too (overlayfs locality check). The head CI Format failure is inherited from the base branch (crates/tracedecay-sessions/.../jsonl_observation_admission/tests.rs, src/commands/daemon.rs — both byte-identical to the base tip and last touched by base commits).

Files

• crates/tracedecay-runtime-core/src/store/memory/graph.rs
• crates/tracedecay-runtime-core/src/store/memory/graph_reconciliation_tests.rs

  

[changeset-bot]

⚠️ No Changeset found

Latest commit: 32ce033

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a6a2677b1d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ScriptedAlchemy]

code-index consumer view (source / load)

We are not a caller of load_source / project_memory_graph. Same class of leftover as our graph-manifest memo: a cheaper recheck must not look like a seat.

What looks good:

• Stamp is captured in the same read snapshot as the source (load_source → lineage_stamp_tx).
• source_unchanged_since: absent stamp is false (reload). Equal stamp is the old watermark-match Ok. Moved stamp falls through to full load_source + watermark compare.
• finish_reconciliation_watermark stamp-equal returns after this pass already published. It does not skip close/reopen or recovered-digest on our port.
• Serve path still GraphConflicts if the verified generation does not match the first-load expectation, or if the stamp moved and the reloaded watermark differs.
• Tests drive publish_project_memory_graph_after_write, not a test-only load port. Codex P1 (load_source_for_test / finish_reconciliation_watermark_for_test) is gone on d011d96ff.

Not a silent reinterpret of seating. Stamp skip ≠ Plan 39 seating.

What’s blocking ready

Nothing must-fix from this crate. Still draft. Isolated store::memory tests are the evidence; don’t retrigger CI. Leave the base. Don’t flatten.

[ScriptedAlchemy]

Cross-review of draft d011d96ff. Comments only. Not editing runtime-core. Not changing the base. Not retriggering CI.

Crate — not blocking ready: load_source still returns a lineage_stamp; source_unchanged_since is the cheap recheck. Stamp-equal Ok in finish_reconciliation_watermark does not swallow pending — worker clears pending when a pass starts and reruns if it is set again during the pass (memory_graph_reconciliation_pending only suppresses GraphConflict on watermark mismatch). Earlier inline on that was wrong; correction is on the thread and I resolved it. load_source_for_test / finish_reconciliation_watermark_for_test are gone; remaining *_for_test helpers are #[cfg(test)] + pub(in crate::store::memory). Codex P1 on those ports is outdated (resolved).

Blocking ready: draft; mergeable unknown; no checks on the rollup. Ready watch is empty until Zack names one.

Leftover: none in this crate from this pass.