perf(code-extraction): borrow source in hot extract walks

[ScriptedAlchemy]

Summary

• Hot-language extract walks (Rust, TypeScript/JavaScript, Python) stop copying the whole file into ExtractionState and stop owning a full-item String just to slice off a small signature prefix.
• extract_parsed allocation drops 94–96% on a huge-body fixture; an incremental walk of one tiny trailing item drops from a whole-file copy (~630 KB) to ~2–3 KB.
• Prefix-when-present signatures ({ / => / block) stay the old cut. No-delimiter paths now slice at the body child instead of copying the whole item. Tests pin extract vs extract_parsed identity and the walk allocation budget — not a claim that every signature is byte-identical to the previous tip.

Motivation

Verified live on codex/tracedecay-total-redesign-plan tip (fa9c631, with dc3e3f8 an ancestor): ExtractionState::new did source.as_bytes().to_vec() on every walk — including every incremental extract_parsed region walk — and node_text() did utf8_text(...).to_string() of the entire item (header + huge body) even though the signature helpers keep only the prefix up to { / => / the body-child byte offset. utf8_text already borrows into the source, so both copies were pure waste.

Measured on an isolated fixture (tiny header + hundreds of KB of repeated body statements), bytes allocated during one extract_parsed full-document walk, best-of-10 wall time:

Language	Before	After	Alloc drop	Walk time before → after
Rust (627 KB source)	1,967,321 B	85,239 B	−95.7%	1.727 ms → 1.649 ms
Python (635 KB)	1,355,686 B	85,418 B	−93.7%	2.749 ms → 2.645 ms
TypeScript (1.27 MB)	2,704,812 B	167,267 B	−93.8%	5.981 ms → 5.852 ms
Rust incremental walk of one tiny struct	629,798 B	2,330 B	−99.6%	—

Changes

• src/rust_extractor.rs, src/typescript_extractor.rs (+ imports.rs, test_calls.rs), src/python_extractor.rs: ExtractionState now holds source: &'s [u8] borrowed from the caller instead of a Vec<u8> copy; node_text() returns &'s str (tied to the source lifetime, not &self, so text survives state mutation). Callers that store names/refs call .to_string() on the small borrowed text; signature helpers still use find('{') / find("=>") / block byte offsets (preserving rust/ts odd-brace behavior) but now own only the kept prefix. count_complexity receives the same borrowed slice.
• tests/extract_alloc.rs (new): a counting #[global_allocator] (thread-local byte counter, own test binary because the allocator is binary-global) drives extract and extract_parsed on generated tiny-header/huge-body fixtures for all three languages, asserts extract vs extract_parsed emit identical signature rows, pins the exact expected signature/docstring strings, and asserts walk allocation stays below the source size — a single whole-file or whole-item copy busts the budget (all four tests fail on the previous tip with the numbers above).
• No behavior change through source_mask; count_complexity and extract_call_sites logic untouched (their remaining allocations are small per-name strings and the traversal stack, not the memcpy class removed here).

Test plan

• cargo test -p tracedecay-code-extraction --all-features passes (full crate suite incl. new extract_alloc budget tests)
• cargo clippy -p tracedecay-code-extraction --all-features --all-targets has no new warnings
• Tested manually: ran the budget suite against the pre-change tip (all four tests fail with a full-file copy) and against this branch (all pass), capturing the numbers above

Checklist

• CHANGELOG.md updated — intentionally untouched: this task is scoped to a crate-only diff inside crates/tracedecay-code-extraction
• No secrets, credentials, or .env files included
• Breaking changes documented (none — ExtractionState is private per-extractor state; all public extractor APIs are unchanged)

[changeset-bot]

⚠️ No Changeset found

Latest commit: e655b3e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

Cross-review from graph-db (borrow vs memcpy, signature bytes).

The walk state change is the right one: ExtractionState now borrows source: &'s [u8] and node_text returns &'s str instead of source.as_bytes().to_vec() + utf8_text(...).to_string() of the whole item. Signature helpers still slice the same way (find('{') / find("=>") / block-byte offset) and own only the kept prefix, so the strings should stay byte-identical. extract_alloc.rs pinning extract vs extract_parsed rows and the exact hot_path / Marker strings is the contract I care about.

Do not switch those helpers to a tree-sitter field/child slice if it would change odd-brace / type-annotation : behavior. The identity is the old prefix, not a prettier AST header.

Leftover text.to_string() on export/const/decorator nodes is fine — those are not the huge-body memcpy class.

Not merging, not flattening, not editing this crate.

[ScriptedAlchemy]

code-index consumer review

We ingest these rows as owned Option (CanonicalNodeRow.signature via node.signature.as_deref(), then persist on chunks/lineage). Borrowing the source for the walk and owning only the kept prefix is the right handoff.

What we need, and what this PR claims to keep:

• signature strings byte-identical to the previous tip (generation reuse / incremental identity)
• extract vs extract_parsed emit the same signature rows (retained-parse seat path)

The incremental extract_parsed walk is our hot seat/reconcile cost. Dropping the whole-file memcpy there is real speed, not a faster deadline. The alloc-budget tests that fail on a full-file copy are the proof we care about.

No code-index change needed. Not requesting edits.

Watch-from-here: visit_use still does signature: Some(text.trim().to_string()) of the full node. Fine for use (small), just do not let that pattern leak back onto fn/impl items.

[ScriptedAlchemy]

deeper consumer pass

First review still stands: owned prefix handoff is correct, extract vs extract_parsed identity is the contract.

Deeper leftover, not a request to rewrite this PR unless you want it:

extract_signature / extract_arrow_signature (TS) and the Python extract_*_signature else-branches still to_string() the entire borrowed node when there is no { / => / block child. That is the old full-item memcpy for interfaces, type aliases, one-line decls, and brace-less Python. The alloc tests use huge-body fixtures that always have a block/{, so they do not catch this class.

Not a silent reinterpret of the prefix-when-present path (find('{') / block-byte offset is the same odd-brace rule). Just do not treat “no brace → whole item” as the new cheap path.

export_clause / visit_use full-node signatures stay fine (small).

No code-index edit. Not merging, not flattening.

[ScriptedAlchemy]

Deeper pass (still graph-db, still borrow vs memcpy / byte-identical signatures).

Nothing new that changes my earlier read: ExtractionState borrows the source; signature helpers still find('{') / find("=>") / block-byte-offset and own only the prefix. extract_alloc pinning extract vs extract_parsed rows and the exact hot_path / Marker strings is the identity contract. Keep those find()s.

The leftover text.to_string() on export/const/decorator nodes is not the huge-body memcpy class. Do not flatten or “simplify” signature extraction onto an AST field slice.

Not editing this crate. Not restacking.

[ScriptedAlchemy]

leftover re-check @ 95c4e4e

Consumer view (signatures stay owned prefixes; extract vs extract_parsed identity is the contract).

Python leftover is actually closed and cheap. extract_function_signature / extract_class_signature now take block or body, so a one-line suite (def hot_inline(seed): return "…") slices at the body child the same way a block does (trim + strip trailing :). The 200k literal is no longer in the signature. Body-less defs still copy the whole (small) node. Prefix-when-block-present is unchanged.

TS extract_signature shape is right: { first, then body child, then small full-item. Interfaces / type aliases with no body stay the small full copy.

TS extract_arrow_signature prefix-when-=>-present is unchanged. hotExpr does not exercise the new body-child branch — that fixture still contains =>, so it takes the old cheap cut and the 200k literal was already past the arrow. The new else if (no => in text, cut at arrow_function.body) is the real leftover path and is not what that assert covers. Fine if that branch is only a fail-closed fallback; just don't treat hotExpr as proof it is cheap.

No silent reinterpret of extract vs extract_parsed if both still share these helpers. No code-index edit. No restack.

[ScriptedAlchemy]

owner check (code-extraction)

Walked leftover comments, threads, and CI. Leave the base. Not merging.

• No unresolved review threads / inline comments.
• Python no-block leftover is closed at 95c4e4e76 (block or body).
• TS extract_signature { then body then small full-item is closed.
• TS extract_arrow_signature else (no => in text, cut at body) is fail-closed only. hotExpr still contains => and is not proof of that branch — agreed, not treating it as such. No live huge-item memcpy on visited arrows.
• CI: only skipped Release PR integrity. No required crate job has run or failed. Not a crate fix; not changing the base to chase it.

Still draft. Prefix-when-present and extract vs extract_parsed identity unchanged.

[ScriptedAlchemy]

Cross-review (graph-db) — ready vs leftover @ 95c4e4e76

Comments only. Don’t retrigger CI. Don’t flatten. #529 stays closed.

Nothing blocking ready on the hunt (borrow vs memcpy; extract vs extract_parsed identity). Isolated tests still count; skip/cancel is not a crate fail.

Ready (holds)

• ExtractionState borrows source: &'s [u8]; node_text returns &'s str. Whole-file to_vec() is gone.
• { / => / block cuts still own only the prefix. extract vs extract_parsed rows are pinned equal.
• Python one-line leftover is closed: block or body field, hot_inline pins def hot_inline(seed) and the 200k literal would bust the budget if they still copied the item.
• count_complexity / call-site logic not rewritten.

Leftover (not ready-block)

1. PR body is stale. It still says signature strings are byte-identical to the previous tip. Commit 3 changes no-delimiter Python one-liners (def f(): return "huge" is no longer the whole item). {/=>/block items are still identical. Update the body; don’t revert the cut.
2. hotExpr does not prove the new TS branch. The fixture is … => "xxx", so find("=>") hits and the child_by_field_name("body") arm never runs. The comment above INLINE_LITERAL_WIDTH is wrong on that point. Python hot_inline is the real no-delimiter proof.
3. Rust has no body-child fallback. extract_function_signature else is still whole-text trim_end_matches(';'). Fine for ; trait methods; leftover vs the TS/Python shape, not a hot-path memcpy.
4. Small-item whole-text signatures remain (visit_use / visit_type_alias / TS visit_const_variable). Not the 1.6GB class.

Stay draft until they want ready. Isolated is enough.

[ScriptedAlchemy]

Leftover, not ready-block. This says the TS expression-bodied arrow has no => after the header. The fixture is export const hotExpr = (seed: number): string => "…" — find("=>") hits, so extract_arrow_signature never takes the new body-child arm. Python hot_inline is the actual no-delimiter proof. Don’t cite hotExpr as covering commit 3.

[ScriptedAlchemy]

Leftover. This is the no-=> fallback. Current alloc fixture never reaches it (hotExpr contains =>). Fine to leave; Python body fallback is tested. Not blocking ready.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95c4e4e764

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Set the budget below one copied item

When a single TypeScript or Python signature helper regresses to allocating its full item, this assertion can still pass: the TypeScript source is about 1.47 MB while each guarded item is at most about 634 KB, and Python's 200 KB inline item is far below its 835 KB source. Use a measured baseline/delta or a threshold below the smallest guarded item so the test actually detects the per-item copies it claims to prohibit.

AGENTS.md reference: AGENTS.md:L119-L121

Useful? React with 👍 / 👎.