diff --git a/.changeset/ast-hash-duplication.md b/.changeset/ast-hash-duplication.md new file mode 100644 index 00000000..9a05af2d --- /dev/null +++ b/.changeset/ast-hash-duplication.md @@ -0,0 +1,5 @@ +--- +"@stainless-code/codemap": minor +--- + +Add structural duplicate detection: `symbols.body_hash` at index time (canonical function body AST) and bundled `duplicates` recipe. Function-shaped symbols only; trivial one-line bodies skipped. Triage collisions with `snippet` — shared control-flow skeletons can false-positive. diff --git a/docs/architecture.md b/docs/architecture.md index fe18b6b1..611131b3 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -301,6 +301,7 @@ All base tables use `STRICT` mode; **`source_fts`** is an FTS5 virtual table (no | return_type | TEXT | Stringified return type for function-shaped symbols; NULL when unannotated or N/A | | is_async | INTEGER | 1 for async function-shaped symbols (`function`, `method`, arrow-assigned `function` kind) | | is_generator | INTEGER | 1 for generator function-shaped symbols | +| body_hash | TEXT | SHA-256 hex of canonicalized function **body** AST (identifiers → `$id`, literals → kind only, absent returns → `Literal:nullish`). Populated for function-shaped symbols when `body_line_count >= 2`; NULL otherwise. Powers `duplicates` recipe. Partial index `idx_symbols_body_hash` | ### `calls` — Function-scoped call edges, deduped per file (`STRICT`) diff --git a/docs/glossary.md b/docs/glossary.md index 0b2f8e51..c44bee8b 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -147,6 +147,10 @@ Per-function decision-point count (REAL column on `symbols`). Computed by the pa SonarSource-inspired cognitive complexity (INTEGER on `symbols`) for the same function-shaped symbols as cyclomatic `complexity`. Penalizes nested control flow; computed in the same parser walk as McCabe. Recipes: `high-cognitive-complexity` (`min_score` default 15, Sonar rule threshold); `high-complexity-untested` includes the column while filtering on cyclomatic `complexity`. +### `symbols.body_hash` / structural duplicate bodies + +SHA-256 hex of a canonicalized function **body** AST (not raw source). Normalization (v1): every identifier → `$id`; literals → kind only (`Literal:string`, …); absent returns (`null`, `undefined`, `void 0`, bare `return`) → `Literal:nullish`; template literals walked structurally. Populated for function-shaped symbols (`function`, `method`, `getter`, `setter`) when `body_line_count >= 2`; NULL for trivial one-liners and non-functions. Recipe **`duplicates`** groups rows sharing a hash. Distinct from token-level suffix-array / copy-paste clone detectors — catches rename-insensitive structural twins; may false-positive on shared control-flow skeletons (triage with `snippet`). + ### `source_fts` (FTS5 virtual table) / `--with-fts` / opt-in full-text Opt-in FTS5 virtual table over file content (`tokenize='porter unicode61'`). Always created (near-zero space when empty); populated only when the resolved config has FTS5 enabled (`.codemap/config.ts` `fts5: true` OR `--with-fts` CLI flag at index time; CLI wins, logs stderr override). Demonstrates the FTS5 ⨯ `symbols` ⨯ `coverage` JOIN composability that ripgrep can't match — bundled recipe `text-in-deprecated-functions` exemplifies the JOIN. Toggle change auto-detects via `meta.fts5_enabled` and forces a full rebuild so `source_fts` is consistently populated. Stderr telemetry `[fts5] source_fts populated: files / KB` on first populate. Distinct from `coverage` — `source_fts` is an FTS5 **virtual** table; `coverage` is a regular `STRICT, WITHOUT ROWID` table. Default OFF preserves `.codemap/index.db` size for non-users (~30–50% growth on text-heavy projects). diff --git a/docs/golden-queries.md b/docs/golden-queries.md index f8411f07..5ced88d6 100644 --- a/docs/golden-queries.md +++ b/docs/golden-queries.md @@ -78,6 +78,10 @@ Some bundled recipes add optional **`reason`** (TEXT) and **`evidence_json`** (T `coverage-confirmed-dead` adds **`confidence`** (`high` \| `medium`) on each row — **`high`** when static dead and ingested `coverage_pct = 0`; **`medium`** when static dead but the symbol has no ingested coverage row. Also **`reason`**, **`caller_count`**. Goldens: `coverage-confirmed-dead` (post-ingest mix) and `coverage-confirmed-dead-no-ingest` (`preSetup: clear-coverage`, `everyRowFieldEquals` on `confidence: medium`). +### Duplication columns (`duplicates` recipe) + +`duplicates` returns one row per function-shaped symbol in a **`body_hash`** collision group: **`name`**, **`kind`**, **`file_path`**, **`line_start`**, **`line_end`**, **`body_hash`**, **`body_line_count`**, **`duplicate_count`** (in-scope group size after `path_prefix` / `min_body_lines`). Substrate column **`symbols.body_hash`** is populated at index for function-shaped symbols (`function`, `method`, `getter`, `setter`) when `body_line_count >= 2`. Goldens: `duplicates` (includes `src/bench/duplicate-body-{a,b}.ts` pair). False positives possible when unrelated functions share control-flow skeleton or sync vs async/generator bodies match — triage with `snippet`. Recipe caps at **50 rows** (no truncation marker). + --- ## Status diff --git a/docs/plans/ast-hash-duplication.md b/docs/plans/ast-hash-duplication.md deleted file mode 100644 index cd941b87..00000000 --- a/docs/plans/ast-hash-duplication.md +++ /dev/null @@ -1,151 +0,0 @@ -# AST-hash duplication — plan - -> **Status:** open · **Priority:** P2 · **Effort:** M (~2 weeks) -> -> **Motivator:** Agents and maintainers need to find **structurally identical** function bodies across files — same control-flow shape, not merely copy-pasted text with renamed identifiers. Token-level suffix-array engines solve a different problem (literal clones). Codemap exposes duplication as **substrate + recipe**: `symbols.body_hash` at parse time + bundled `duplicates` recipe (`GROUP BY body_hash HAVING COUNT(*) > 1`). No severity primitive, no suppression-by-default. -> -> **Roadmap:** [§ Core substrate & platform](../roadmap.md#core-substrate--platform) - ---- - -## Agent start here - -Ship **`body_hash` column + migration + one parse fixture** before the `duplicates` recipe. Add a **new extractor** (or extend `symbolsExtractor` pop path) in the **same oxc visitor pass** ([substrate-extraction R.1](./substrate-extraction.md#pre-locked-decisions)). Hash only **function-shaped** symbols in slice 1. - -### Key touchpoints - -| File | What to read | -| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -| [`src/extractors/symbols.ts`](../../src/extractors/symbols.ts) | Function/method enter + `functionShapeColumns`; where `line_start`/`line_end` are set | -| [`src/extractors/complexity.ts`](../../src/extractors/complexity.ts) | Pattern for per-symbol body-scoped visitor state | -| [`src/parser.ts`](../../src/parser.ts) | `EXTRACTORS` registration order; single-pass walk | -| [`src/db.ts`](../../src/db.ts) | `SymbolRow` (~L886), `insertSymbols`, `SCHEMA_VERSION` migration pattern | -| [`src/parser.ts`](../../src/parser.ts) | `EXTRACTORS` array — register new extractor after `complexityExtractor` | -| [`src/extractors/types.ts`](../../src/extractors/types.ts) | `TierExtractor` contract for `bodyHashExtractor` | -| [`src/hash.ts`](../../src/hash.ts) | `hashContent` (SHA-256) for canonical body serialization | -| [`templates/recipes/`](../../templates/recipes/) | Recipe `.sql` + `.md` pair (e.g. `fan-in`) | -| [`docs/golden-queries.md`](../golden-queries.md) | Register golden scenario for `duplicates` recipe | - -### Architecture - -```text -oxc visitor (existing symbol walk) - → on function-shaped symbol exit: serialize normalized body AST → hashContent → body_hash - → symbol row persisted in symbols.body_hash (nullable for non-function kinds) -recipe duplicates - → SQL GROUP BY body_hash HAVING COUNT(*) > 1 - → rows: hash group + member symbols (file_path, name, line_start) - → query / MCP / HTTP (Moat A — no new verb) -``` - -**Not** suffix-array / LCP semantic clones — different problem class (literal copy-paste); stay deferred unless `body_hash` proves insufficient. - -### Tracer bullet (slice 1) - -1. `body_hash` on `FunctionDeclaration` bodies only; two fixtures with isomorphic bodies → same hash, different names. 2. `SCHEMA_VERSION` bump. 3. `duplicates.sql` returns the pair. Expand to arrows/methods in slice 2. - -### Out of scope (v1) - -Suffix-array semantic duplication engine; verdict / severity on duplicate groups; default suppressions; hashing type/interface bodies; comment-aware hashing. - ---- - -## Pre-locked decisions - -| # | Decision | Source | -| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | -| B.1 | **New column** `symbols.body_hash TEXT` — nullable; populated for function-shaped symbols only in v1. | [Moat B](../roadmap.md#moats-load-bearing) | -| B.2 | **Single-pass extraction** — compute hash in the existing oxc visitor; no second AST walk. | [substrate-extraction R.1](./substrate-extraction.md#pre-locked-decisions) | -| B.3 | **Structural, not textual** — hash canonical serialization of the function **body** subtree (not raw `source.slice`), so whitespace-normalized identical logic matches. | Roadmap differentiation vs suffix-array dupes | -| B.4 | **Moat-A exposure** — bundled recipe id `duplicates` (SQL join on `body_hash`); consumer applies `LIMIT` / directory filters. | [Moat A](../roadmap.md#moats-load-bearing) | -| B.5 | **SHA-256 hex** — reuse `hashContent` on the canonical body string (same convention as `files.content_hash`). | [`src/hash.ts`](../../src/hash.ts) | -| B.6 | **No verdict primitive** — recipe returns rows; no `pass`/`fail` on duplicate count. | Moat A | - ---- - -## Normalization sketch (v1 default — confirm in impl PR) - -Canonical string built from a depth-first walk of the body AST: - -- Node `type` + ordered child slots -- **Identifier tokens → placeholder** `$id` (rename-insensitive structural match) -- **Literal values → kind** (`string`, `number`, …) not value (so `"a"` vs `"b"` still match structure-only mode — document false-positive class) -- Skip `loc` / comment attachment -- Exclude `doc_comment` on the symbol row (comments not in body_hash) - -Document the exact rules in `architecture.md` when landed so agents can predict matches. - ---- - -## Recipe SQL sketch - -```sql --- illustrative; final SQL in templates/recipes/duplicates.sql -SELECT body_hash, - COUNT(*) AS duplicate_count, - GROUP_CONCAT(file_path || ':' || name, ', ') AS members -FROM symbols -WHERE body_hash IS NOT NULL -GROUP BY body_hash -HAVING COUNT(*) > 1 -ORDER BY duplicate_count DESC; -``` - -v1 may emit one row per group or one row per symbol with `duplicate_group_size` — pick in impl PR (golden-query ergonomics). - ---- - -## Implementation steps - -1. **`body-hash.ts` extractor** (or module) — `canonicalizeBody(node): string` + `hashContent`. -2. Wire on function exit in `symbols.ts` (or dedicated `bodyHashExtractor` registered after symbols). -3. Extend `SymbolRow` type + `insertSymbols` + migration in `db.ts`. -4. **`templates/recipes/duplicates.sql` + `.md`** — params: optional `min_count`, `path_prefix`. -5. Golden fixture: two files, same structure different param names → one duplicate group. -6. Negative fixture: same name different bodies → different hashes. -7. Docs — `architecture.md` `symbols.body_hash`; `glossary.md` disambiguate vs suffix-array dupes. - ---- - -### Verification - -```bash -bun test src/extractors/*.test.ts # add body-hash fixtures -bun test src/parser.test.ts # if parse integration tests exist for fixtures -bun src/index.ts --files # reindex duplicate fixture -bun src/index.ts query --recipe duplicates --json -bun run typecheck # SymbolRow + insertSymbols column touch db.ts types -``` - -Register golden scenario per [`docs/golden-queries.md`](../golden-queries.md); guard via `scripts/query-golden-coverage-matrix.test.mjs`. - ---- - -## Acceptance - -- [ ] Two isomorphic function bodies (renamed locals) share `body_hash` -- [ ] Different control flow → different `body_hash` -- [ ] `codemap query --recipe duplicates --json` returns groups with `COUNT > 1` -- [ ] Non-function symbols have `body_hash IS NULL` -- [ ] Incremental reindex updates hash for changed files -- [ ] No new pass/fail CLI verb - ---- - -## Open decisions (impl PR) - -| # | Question | -| --- | ------------------------------------------------------------------------------------------------ | -| Q1 | v1 kinds: `FunctionDeclaration` only, or include arrows / methods / class methods in slice 1? | -| Q2 | Identifier normalization: all → `$id`, or preserve exported param names for stricter matching? | -| Q3 | Recipe row shape: one row per duplicate **group** vs one row per **symbol** with group metadata? | -| Q4 | Minimum body size gate (skip `() => x` one-liners) — default off or `min_body_lines` param? | -| Q5 | Index on `symbols(body_hash)` for recipe perf — add in v1 or measure first? | - ---- - -## Dependencies - -- Shipped: `symbols` extraction, `hashContent`, recipe loader -- Independent of [churn-complexity-hotspots](./churn-complexity-hotspots.md), [`symbols.cognitive_complexity`](../glossary.md#symbolscognitive_complexity--cognitive-complexity) -- Supersedes motivation for suffix-array semantic dupes (stay deferred) diff --git a/docs/roadmap.md b/docs/roadmap.md index 08029a4e..37d3733b 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -112,7 +112,7 @@ Predicate-as-API only — enrich row shape and audit deltas; no standalone pass/ - [ ] **`codemap audit` verdict + thresholds** (v1.x) — `verdict: "pass" | "warn" | "fail"` driven by an `audit.deltas[].{added_max, action}` field on the config object (`.codemap/config.{ts,js,json}`). Triggers: two consumers ship `jq`-based threshold scripts with similar shapes, OR one consumer asks with a concrete config sketch. Until then, raw deltas + consumer-side `jq` is the CI exit-code idiom. **Likely accelerant:** the Marketplace Action (next item) shipping is the most plausible path to firing the trigger — once `- uses: stainless-code/codemap@v1` is the dominant CI path, real `jq` threshold scripts will surface. - [ ] **GitHub Marketplace Action — publish + listing finish** — core Action implementation is in-tree: root `action.yml`, `query --ci`, `audit --format sarif` / `--ci`, package-manager detection, dogfood smoke, and opt-in `pr-comment` summary renderer have shipped. Remaining work is the release/listing slice: `MARKETPLACE.md`, `v1.0.0` / floating `v1` tags, Marketplace setup, sacrificial-repo smoke, and making `action-smoke` blocking once the Action tag exists. Action version stream is independent of CLI version (`package.json` currently drives CLI/npm version; Action publishes at its own `v1.0.0`). Plan: [`plans/github-marketplace-action.md`](./plans/github-marketplace-action.md). Effort: S. - [ ] **Churn × complexity hotspots** — `file_churn` table (git `log --numstat` over indexed paths, recency-weighted commits, optional trend) + bundled recipe **`churn-complexity-hotspots`** JOINing `symbols.complexity` for ranked refactor targets. Distinct from outcome alias `hotspots` → `fan-in`. Score is a recipe column, not a verdict ([Moat A](./roadmap.md#moats-load-bearing)). Plan: [`plans/churn-complexity-hotspots.md`](./plans/churn-complexity-hotspots.md). Effort: L–M. -- [ ] **AST-hash duplication** — `symbols.body_hash` column (normalized AST hash via oxc, computed at parse time — Rust-native, fast) + bundled `duplicates` recipe joining on `body_hash` (`GROUP BY body_hash HAVING COUNT(*) > 1`). **Different shape from token-level suffix-array dupes** (catches structurally-identical functions, not copy-paste with renamed variables). Substrate addition — consumer writes the JOIN that decides "this is a problem"; no severity, no suppression-by-default. Plan: [`plans/ast-hash-duplication.md`](./plans/ast-hash-duplication.md). Effort: M. +- [x] **AST-hash duplication** — `symbols.body_hash` (canonical body AST, identifiers → `$id`, literals → kind, absent returns → `Literal:nullish`; function-shaped symbols; skip `body_line_count < 2`) + partial index + bundled `duplicates` recipe (per-symbol rows, CTE `GROUP BY`). **Different shape from token-level suffix-array dupes.** Contract: [architecture § `symbols` table](./architecture.md#symbols--functions-constants-classes-interfaces-types-enums-strict), [glossary § body_hash](./glossary.md#symbolsbody_hash--structural-duplicate-bodies). Effort: M. - [ ] **Falsifiable benchmark CI on named external fixtures** — structural-cost A/B (indexed queries vs `find` + `grep` + `Read`-loop discovery) on zod, fastify, vue-core, next.js. Numbers land in [`docs/benchmark.md`](./benchmark.md); headline figures surface in `MARKETPLACE.md` only after external runs land. Harness: [benchmark § Agent eval harness](./benchmark.md#agent-eval-harness) + external fixture extension; pair with **Agent eval: quality × tokens × wall** for scored completion metrics. **Partial:** manual [`.github/workflows/agent-eval-external.yml`](../.github/workflows/agent-eval-external.yml) for in-repo fixture paths (not zod/fastify/nightly). Effort: M. **Self-index regression guardrail shipped** (#96): `bun run check:perf-baseline` + weekly scheduled workflow (demoted from PR hard gate — GHA runner variance). - [ ] **In-repo test bench scale (optional)** — if `fixtures/minimal` outgrows one corpus: add committed `fixtures/bench/` or rename `minimal`→`bench`. Harness map: [`testing-coverage.md`](./testing-coverage.md), [`fixtures/README.md`](../fixtures/README.md). diff --git a/fixtures/CAPABILITIES.json b/fixtures/CAPABILITIES.json index 154f5c62..cbebddd7 100644 --- a/fixtures/CAPABILITIES.json +++ b/fixtures/CAPABILITIES.json @@ -175,6 +175,15 @@ ], "setup": ["ingest-coverage"] }, + { + "id": "duplication.body-hash", + "description": "symbols.body_hash structural fingerprint and duplicates recipe", + "fixtureFiles": [ + "src/bench/duplicate-body-a.ts", + "src/bench/duplicate-body-b.ts" + ], + "goldenScenarios": ["duplicates"] + }, { "id": "boundaries.suppressions", "description": "boundary_rules, suppressions, config-driven violations", diff --git a/fixtures/golden/minimal/barrel-files.json b/fixtures/golden/minimal/barrel-files.json index cb872d27..c29bd6b0 100644 --- a/fixtures/golden/minimal/barrel-files.json +++ b/fixtures/golden/minimal/barrel-files.json @@ -64,19 +64,19 @@ "exports": 1 }, { - "file_path": "src/bench/homonym-consumer-a.ts", + "file_path": "src/bench/duplicate-body-a.ts", "exports": 1 }, { - "file_path": "src/bench/homonym-consumer-b.ts", + "file_path": "src/bench/duplicate-body-b.ts", "exports": 1 }, { - "file_path": "src/bench/homonym-helper-a.ts", + "file_path": "src/bench/homonym-consumer-a.ts", "exports": 1 }, { - "file_path": "src/bench/homonym-helper-b.ts", + "file_path": "src/bench/homonym-consumer-b.ts", "exports": 1 } ] diff --git a/fixtures/golden/minimal/coverage-confirmed-dead-no-ingest.json b/fixtures/golden/minimal/coverage-confirmed-dead-no-ingest.json index 54a3c68d..cc98ee08 100644 --- a/fixtures/golden/minimal/coverage-confirmed-dead-no-ingest.json +++ b/fixtures/golden/minimal/coverage-confirmed-dead-no-ingest.json @@ -9,6 +9,26 @@ "confidence": "medium", "reason": "no_callers_and_coverage_unmeasured" }, + { + "name": "duplicateAlpha", + "kind": "function", + "file_path": "src/bench/duplicate-body-a.ts", + "line_start": 1, + "coverage_pct": 0, + "caller_count": 0, + "confidence": "medium", + "reason": "no_callers_and_coverage_unmeasured" + }, + { + "name": "duplicateBeta", + "kind": "function", + "file_path": "src/bench/duplicate-body-b.ts", + "line_start": 1, + "coverage_pct": 0, + "caller_count": 0, + "confidence": "medium", + "reason": "no_callers_and_coverage_unmeasured" + }, { "name": "useHelperA", "kind": "function", diff --git a/fixtures/golden/minimal/coverage-confirmed-dead.json b/fixtures/golden/minimal/coverage-confirmed-dead.json index b0bbe23c..c472b361 100644 --- a/fixtures/golden/minimal/coverage-confirmed-dead.json +++ b/fixtures/golden/minimal/coverage-confirmed-dead.json @@ -9,6 +9,26 @@ "confidence": "medium", "reason": "no_callers_and_coverage_unmeasured" }, + { + "name": "duplicateAlpha", + "kind": "function", + "file_path": "src/bench/duplicate-body-a.ts", + "line_start": 1, + "coverage_pct": 0, + "caller_count": 0, + "confidence": "medium", + "reason": "no_callers_and_coverage_unmeasured" + }, + { + "name": "duplicateBeta", + "kind": "function", + "file_path": "src/bench/duplicate-body-b.ts", + "line_start": 1, + "coverage_pct": 0, + "caller_count": 0, + "confidence": "medium", + "reason": "no_callers_and_coverage_unmeasured" + }, { "name": "useHelperA", "kind": "function", diff --git a/fixtures/golden/minimal/duplicates.json b/fixtures/golden/minimal/duplicates.json new file mode 100644 index 00000000..37d934c8 --- /dev/null +++ b/fixtures/golden/minimal/duplicates.json @@ -0,0 +1,172 @@ +[ + { + "name": "helper", + "kind": "function", + "file_path": "src/bench/homonym-helper-a.ts", + "line_start": 1, + "line_end": 3, + "body_hash": "e8d16be3d228ad8e8c6031f8519bdcebd650b1f8b98d74cc237d0fb06693e44c", + "body_line_count": 3, + "duplicate_count": 4 + }, + { + "name": "helper", + "kind": "function", + "file_path": "src/bench/homonym-helper-b.ts", + "line_start": 1, + "line_end": 3, + "body_hash": "e8d16be3d228ad8e8c6031f8519bdcebd650b1f8b98d74cc237d0fb06693e44c", + "body_line_count": 3, + "duplicate_count": 4 + }, + { + "name": "ignoredExport", + "kind": "function", + "file_path": "src/orphan.ts", + "line_start": 2, + "line_end": 4, + "body_hash": "e8d16be3d228ad8e8c6031f8519bdcebd650b1f8b98d74cc237d0fb06693e44c", + "body_line_count": 3, + "duplicate_count": 4 + }, + { + "name": "orphanHelper", + "kind": "function", + "file_path": "src/orphan.ts", + "line_start": 7, + "line_end": 9, + "body_hash": "e8d16be3d228ad8e8c6031f8519bdcebd650b1f8b98d74cc237d0fb06693e44c", + "body_line_count": 3, + "duplicate_count": 4 + }, + { + "name": "legacyClient", + "kind": "function", + "file_path": "src/api/client.ts", + "line_start": 46, + "line_end": 48, + "body_hash": "4a40f1a28d1725db9acbb5e8dc3ff8a19e8d4a89d55d5468076797f1732e17d2", + "body_line_count": 3, + "duplicate_count": 3 + }, + { + "name": "useHelperA", + "kind": "function", + "file_path": "src/bench/homonym-consumer-a.ts", + "line_start": 3, + "line_end": 5, + "body_hash": "4a40f1a28d1725db9acbb5e8dc3ff8a19e8d4a89d55d5468076797f1732e17d2", + "body_line_count": 3, + "duplicate_count": 3 + }, + { + "name": "useHelperB", + "kind": "function", + "file_path": "src/bench/homonym-consumer-b.ts", + "line_start": 3, + "line_end": 5, + "body_hash": "4a40f1a28d1725db9acbb5e8dc3ff8a19e8d4a89d55d5468076797f1732e17d2", + "body_line_count": 3, + "duplicate_count": 3 + }, + { + "name": "ping", + "kind": "method", + "file_path": "src/bench/method-call-sites.ts", + "line_start": 3, + "line_end": 5, + "body_hash": "9f55d2e87401c5a13848194ee9cb869a9c56340a0ea5452f75bc1052d15e0941", + "body_line_count": 3, + "duplicate_count": 3 + }, + { + "name": "dropMe", + "kind": "function", + "file_path": "src/bench/stale-multi-helpers.ts", + "line_start": 5, + "line_end": 7, + "body_hash": "9f55d2e87401c5a13848194ee9cb869a9c56340a0ea5452f75bc1052d15e0941", + "body_line_count": 3, + "duplicate_count": 3 + }, + { + "name": "keepMe", + "kind": "function", + "file_path": "src/bench/stale-multi-helpers.ts", + "line_start": 1, + "line_end": 3, + "body_hash": "9f55d2e87401c5a13848194ee9cb869a9c56340a0ea5452f75bc1052d15e0941", + "body_line_count": 3, + "duplicate_count": 3 + }, + { + "name": "_hiResEpoch", + "kind": "function", + "file_path": "src/utils/date.ts", + "line_start": 26, + "line_end": 28, + "body_hash": "ead8bca8e91caab31f4deb71f88930b2a9075b5d61624b7ee1a7907fe70482f6", + "body_line_count": 3, + "duplicate_count": 3 + }, + { + "name": "now", + "kind": "function", + "file_path": "src/utils/date.ts", + "line_start": 5, + "line_end": 7, + "body_hash": "ead8bca8e91caab31f4deb71f88930b2a9075b5d61624b7ee1a7907fe70482f6", + "body_line_count": 3, + "duplicate_count": 3 + }, + { + "name": "epochMs", + "kind": "function", + "file_path": "src/utils/format.ts", + "line_start": 5, + "line_end": 8, + "body_hash": "ead8bca8e91caab31f4deb71f88930b2a9075b5d61624b7ee1a7907fe70482f6", + "body_line_count": 4, + "duplicate_count": 3 + }, + { + "name": "handshake", + "kind": "function", + "file_path": "src/api/client.ts", + "line_start": 37, + "line_end": 40, + "body_hash": "fa2629b5493dec080e79ebc18a614bd18cf3de043aaf1ab266d9b8932e7c51af", + "body_line_count": 4, + "duplicate_count": 2 + }, + { + "name": "duplicateAlpha", + "kind": "function", + "file_path": "src/bench/duplicate-body-a.ts", + "line_start": 1, + "line_end": 6, + "body_hash": "82eac87bd797830b98f82147a6d1deec2d2cc45241694705827b475c449941b2", + "body_line_count": 6, + "duplicate_count": 2 + }, + { + "name": "duplicateBeta", + "kind": "function", + "file_path": "src/bench/duplicate-body-b.ts", + "line_start": 1, + "line_end": 6, + "body_hash": "82eac87bd797830b98f82147a6d1deec2d2cc45241694705827b475c449941b2", + "body_line_count": 6, + "duplicate_count": 2 + }, + { + "name": "UiPanel", + "kind": "function", + "file_path": "src/bench/jsx-ui-namespace.ts", + "line_start": 1, + "line_end": 3, + "body_hash": "fa2629b5493dec080e79ebc18a614bd18cf3de043aaf1ab266d9b8932e7c51af", + "body_line_count": 3, + "duplicate_count": 2 + } +] diff --git a/fixtures/golden/minimal/files-count.json b/fixtures/golden/minimal/files-count.json index beb71b49..c68f12eb 100644 --- a/fixtures/golden/minimal/files-count.json +++ b/fixtures/golden/minimal/files-count.json @@ -1,5 +1,5 @@ [ { - "n": 43 + "n": 45 } ] diff --git a/fixtures/golden/minimal/files-hashes.json b/fixtures/golden/minimal/files-hashes.json index 8436d04f..f181d16f 100644 --- a/fixtures/golden/minimal/files-hashes.json +++ b/fixtures/golden/minimal/files-hashes.json @@ -41,6 +41,18 @@ "language": "ts", "line_count": 5 }, + { + "path": "src/bench/duplicate-body-a.ts", + "content_hash": "4cb1486e1d487fecbc40344655d815604d9b7c5da792b9871c967ebff5a14c76", + "language": "ts", + "line_count": 7 + }, + { + "path": "src/bench/duplicate-body-b.ts", + "content_hash": "b0844c34fae42b69fc37a90754e27b700ec9701e0e32fb64170101bab81385cc", + "language": "ts", + "line_count": 7 + }, { "path": "src/bench/homonym-consumer-a.ts", "content_hash": "8b1e408d2dee9f4e02afa3a9c3a7b8cc67a348eb8a828c1c3f88b0b2cc504f35", diff --git a/fixtures/golden/minimal/index-summary.json b/fixtures/golden/minimal/index-summary.json index 6f0cb243..fee69116 100644 --- a/fixtures/golden/minimal/index-summary.json +++ b/fixtures/golden/minimal/index-summary.json @@ -1,7 +1,7 @@ [ { - "files": 43, - "symbols": 106, + "files": 45, + "symbols": 110, "imports": 26, "components": 5, "dependencies": 23 diff --git a/fixtures/golden/minimal/index-table-stats.json b/fixtures/golden/minimal/index-table-stats.json index 47f6445c..b618b394 100644 --- a/fixtures/golden/minimal/index-table-stats.json +++ b/fixtures/golden/minimal/index-table-stats.json @@ -1,9 +1,9 @@ [ { - "files": 43, - "symbols": 106, + "files": 45, + "symbols": 110, "imports": 26, - "exports": 62, + "exports": 64, "components": 5, "dependencies": 23, "markers": 7, @@ -13,17 +13,17 @@ "css_vars": 2, "css_classes": 2, "css_keyframes": 1, - "scopes": 108, - "ref_count": 319, - "bindings": 276, + "scopes": 112, + "ref_count": 325, + "bindings": 282, "import_specifiers": 31, - "function_params": 18, + "function_params": 20, "runtime_markers": 6, "test_suites": 7, "re_export_chains": 4, "module_cycles": 2, "dynamic_imports": 1, - "file_metrics": 35, + "file_metrics": 37, "unresolved_calls": 1, "jsx_elements": 10, "async_calls": 1, diff --git a/fixtures/golden/minimal/refactor-risk-ranking.json b/fixtures/golden/minimal/refactor-risk-ranking.json index 55a9a623..1f742e14 100644 --- a/fixtures/golden/minimal/refactor-risk-ranking.json +++ b/fixtures/golden/minimal/refactor-risk-ranking.json @@ -95,6 +95,22 @@ "measured_symbols": 0, "risk_score": 200 }, + { + "file_path": "src/bench/duplicate-body-a.ts", + "exported_count": 1, + "fan_in": 0, + "avg_coverage_pct": 0, + "measured_symbols": 0, + "risk_score": 100 + }, + { + "file_path": "src/bench/duplicate-body-b.ts", + "exported_count": 1, + "fan_in": 0, + "avg_coverage_pct": 0, + "measured_symbols": 0, + "risk_score": 100 + }, { "file_path": "src/bench/homonym-consumer-a.ts", "exported_count": 1, @@ -222,21 +238,5 @@ "avg_coverage_pct": 66.7, "measured_symbols": 3, "risk_score": 66.7 - }, - { - "file_path": "src/components/shop/ProductCard.tsx", - "exported_count": 1, - "fan_in": 0, - "avg_coverage_pct": 100, - "measured_symbols": 2, - "risk_score": 0 - }, - { - "file_path": "src/usePermissions.ts", - "exported_count": 1, - "fan_in": 2, - "avg_coverage_pct": 100, - "measured_symbols": 1, - "risk_score": 0 } ] diff --git a/fixtures/golden/minimal/source-fts-row-count.json b/fixtures/golden/minimal/source-fts-row-count.json index beb71b49..c68f12eb 100644 --- a/fixtures/golden/minimal/source-fts-row-count.json +++ b/fixtures/golden/minimal/source-fts-row-count.json @@ -1,5 +1,5 @@ [ { - "n": 43 + "n": 45 } ] diff --git a/fixtures/golden/minimal/unimported-exports.json b/fixtures/golden/minimal/unimported-exports.json index 537358f5..8fc71787 100644 --- a/fixtures/golden/minimal/unimported-exports.json +++ b/fixtures/golden/minimal/unimported-exports.json @@ -44,6 +44,24 @@ "reason": "no_direct_import", "evidence_json": "[]" }, + { + "name": "duplicateAlpha", + "kind": "value", + "file_path": "src/bench/duplicate-body-a.ts", + "is_default": 0, + "re_export_source": null, + "reason": "no_direct_import", + "evidence_json": "[]" + }, + { + "name": "duplicateBeta", + "kind": "value", + "file_path": "src/bench/duplicate-body-b.ts", + "is_default": 0, + "re_export_source": null, + "reason": "no_direct_import", + "evidence_json": "[]" + }, { "name": "useHelperA", "kind": "value", diff --git a/fixtures/golden/minimal/untested-and-dead.json b/fixtures/golden/minimal/untested-and-dead.json index 7e9d42f6..df29d48a 100644 --- a/fixtures/golden/minimal/untested-and-dead.json +++ b/fixtures/golden/minimal/untested-and-dead.json @@ -5,6 +5,18 @@ "line_start": 46, "coverage_pct": 0 }, + { + "name": "duplicateAlpha", + "file_path": "src/bench/duplicate-body-a.ts", + "line_start": 1, + "coverage_pct": 0 + }, + { + "name": "duplicateBeta", + "file_path": "src/bench/duplicate-body-b.ts", + "line_start": 1, + "coverage_pct": 0 + }, { "name": "useHelperA", "file_path": "src/bench/homonym-consumer-a.ts", diff --git a/fixtures/golden/minimal/worst-covered-exports.json b/fixtures/golden/minimal/worst-covered-exports.json index 41da549b..cccd2972 100644 --- a/fixtures/golden/minimal/worst-covered-exports.json +++ b/fixtures/golden/minimal/worst-covered-exports.json @@ -29,6 +29,18 @@ "line_start": 46, "coverage_pct": 0 }, + { + "name": "duplicateAlpha", + "file_path": "src/bench/duplicate-body-a.ts", + "line_start": 1, + "coverage_pct": 0 + }, + { + "name": "duplicateBeta", + "file_path": "src/bench/duplicate-body-b.ts", + "line_start": 1, + "coverage_pct": 0 + }, { "name": "useHelperA", "file_path": "src/bench/homonym-consumer-a.ts", @@ -106,17 +118,5 @@ "file_path": "src/components/shop/ShopButton.tsx", "line_start": 8, "coverage_pct": 0 - }, - { - "name": "prefetch", - "file_path": "src/consumer.ts", - "line_start": 15, - "coverage_pct": 0 - }, - { - "name": "run", - "file_path": "src/consumer.ts", - "line_start": 22, - "coverage_pct": 0 } ] diff --git a/fixtures/golden/scenarios.json b/fixtures/golden/scenarios.json index a92df0b4..bbd6d341 100644 --- a/fixtures/golden/scenarios.json +++ b/fixtures/golden/scenarios.json @@ -457,6 +457,11 @@ "prompt": "Functions with cognitive complexity >= default threshold (15).", "recipe": "high-cognitive-complexity" }, + { + "id": "duplicates", + "prompt": "Function-shaped symbols with identical structural body_hash across files.", + "recipe": "duplicates" + }, { "id": "circular-imports", "prompt": "Files in import cycles (SCCs of size >= 2) via Tarjan.", diff --git a/fixtures/minimal/src/bench/duplicate-body-a.ts b/fixtures/minimal/src/bench/duplicate-body-a.ts new file mode 100644 index 00000000..2c2e0f38 --- /dev/null +++ b/fixtures/minimal/src/bench/duplicate-body-a.ts @@ -0,0 +1,6 @@ +export function duplicateAlpha(x: number): number { + if (x > 0) { + return x; + } + return 0; +} diff --git a/fixtures/minimal/src/bench/duplicate-body-b.ts b/fixtures/minimal/src/bench/duplicate-body-b.ts new file mode 100644 index 00000000..b7b4089f --- /dev/null +++ b/fixtures/minimal/src/bench/duplicate-body-b.ts @@ -0,0 +1,6 @@ +export function duplicateBeta(y: number): number { + if (y > 0) { + return y; + } + return 0; +} diff --git a/scripts/agent-eval/scenarios.json b/scripts/agent-eval/scenarios.json index 429d876c..c11eff39 100644 --- a/scripts/agent-eval/scenarios.json +++ b/scripts/agent-eval/scenarios.json @@ -162,6 +162,15 @@ "regex": "export\\s+function\\s+\\w+", "mode": "matches" } + }, + { + "id": "duplicates-recipe", + "goldenId": "duplicates", + "traditional": { + "globs": ["**/*.{ts,tsx}"], + "regex": "export\\s+function\\s+\\w+", + "mode": "files" + } } ] } diff --git a/scripts/duplicates-recipe-scope.test.mjs b/scripts/duplicates-recipe-scope.test.mjs new file mode 100644 index 00000000..b03306fb --- /dev/null +++ b/scripts/duplicates-recipe-scope.test.mjs @@ -0,0 +1,69 @@ +import { describe, expect, it } from "bun:test"; +import { join } from "node:path"; + +/** + * Locks scoped duplicate_count semantics (filtered CTE before GROUP BY). + * Run via `bun run test:scripts`. + */ +import { $ } from "bun"; + +const REPO_ROOT = join(import.meta.dir, ".."); + +describe("duplicates recipe scoped grouping", () => { + it("path_prefix excludes collision groups with only one in-scope symbol", async () => { + await $`bun src/index.ts --full --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); + const result = + await $`bun src/index.ts query --recipe duplicates --json --params path_prefix=src/bench/duplicate-body-a.ts --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); + expect(result.exitCode).toBe(0); + const rows = JSON.parse(result.stdout.toString()); + expect( + rows.some((r) => r.file_path === "src/bench/duplicate-body-a.ts"), + ).toBe(false); + }); + + it("path_prefix duplicate_count reflects in-scope peers only", async () => { + await $`bun src/index.ts --full --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); + const result = + await $`bun src/index.ts query --recipe duplicates --json --params path_prefix=src/bench/duplicate-body- --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); + expect(result.exitCode).toBe(0); + const rows = JSON.parse(result.stdout.toString()); + const alpha = rows.find((r) => r.name === "duplicateAlpha"); + const beta = rows.find((r) => r.name === "duplicateBeta"); + expect(alpha?.duplicate_count).toBe(2); + expect(beta?.duplicate_count).toBe(2); + }); + + it("scoped duplicate_count is below global when prefix trims the group", async () => { + await $`bun src/index.ts --full --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); + const globalResult = + await $`bun src/index.ts query --recipe duplicates --json --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); + const scopedResult = + await $`bun src/index.ts query --recipe duplicates --json --params path_prefix=src/bench/homonym-helper- --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); + expect(globalResult.exitCode).toBe(0); + expect(scopedResult.exitCode).toBe(0); + const globalRows = JSON.parse(globalResult.stdout.toString()); + const scopedRows = JSON.parse(scopedResult.stdout.toString()); + const globalHelper = globalRows.find( + (r) => r.file_path === "src/bench/homonym-helper-a.ts", + ); + const scopedHelper = scopedRows.find( + (r) => r.file_path === "src/bench/homonym-helper-a.ts", + ); + expect(globalHelper?.duplicate_count).toBeGreaterThan(2); + expect(scopedHelper?.duplicate_count).toBe(2); + }); +}); diff --git a/scripts/spike-crap-reachability.test.mjs b/scripts/spike-crap-reachability.test.mjs index 33926cb9..3d5997fe 100644 --- a/scripts/spike-crap-reachability.test.mjs +++ b/scripts/spike-crap-reachability.test.mjs @@ -15,7 +15,7 @@ const SPIKE_SQL = readFileSync( ); describe("spike-crap-reachability (fixtures/minimal)", () => { - it("assigns 85/40/0% tiers to 1/4/39 function-shaped symbols", async () => { + it("assigns 85/40/0% tiers to 1/4/41 function-shaped symbols", async () => { const result = await $`bun src/index.ts query --json ${SPIKE_SQL} --root fixtures/minimal` .cwd(REPO_ROOT) @@ -27,6 +27,6 @@ describe("spike-crap-reachability (fixtures/minimal)", () => { ); expect(byTier[85]).toBe(1); expect(byTier[40]).toBe(4); - expect(byTier[0]).toBe(39); + expect(byTier[0]).toBe(41); }); }); diff --git a/src/db.ts b/src/db.ts index 37aaa9bb..0e7ebf3e 100644 --- a/src/db.ts +++ b/src/db.ts @@ -3,7 +3,7 @@ import type { CodemapDatabase, BindValues } from "./sqlite-db"; /** Bump only on rebuild-forcing DDL changes (NOT on additive tables/columns). * See `docs/architecture.md` § Schema Versioning. */ -export const SCHEMA_VERSION = 38; +export const SCHEMA_VERSION = 39; /** Moat-A: default call-graph surfaces exclude callback-synthesis edges. */ export const CALLS_AST_ONLY_SQL = "(provenance IS NULL OR provenance = 'ast')"; @@ -71,7 +71,8 @@ export function createTables(db: CodemapDatabase) { nesting_depth INTEGER, return_type TEXT, is_async INTEGER NOT NULL DEFAULT 0, - is_generator INTEGER NOT NULL DEFAULT 0 + is_generator INTEGER NOT NULL DEFAULT 0, + body_hash TEXT ) STRICT; -- One row per indexed file. Pure counters from the AST walk. @@ -601,6 +602,8 @@ export function createIndexes(db: CodemapDatabase) { WHERE visibility IS NOT NULL; CREATE INDEX IF NOT EXISTS idx_symbols_async ON symbols(file_path, name, return_type) WHERE is_async = 1; + CREATE INDEX IF NOT EXISTS idx_symbols_body_hash ON symbols(body_hash) + WHERE body_hash IS NOT NULL; CREATE INDEX IF NOT EXISTS idx_imports_source ON imports(source, file_path); CREATE INDEX IF NOT EXISTS idx_imports_resolved ON imports(resolved_path, file_path); @@ -932,6 +935,12 @@ export interface SymbolRow { return_type?: string | null; is_async?: number; is_generator?: number; + /** + * SHA-256 of canonicalized function body AST for function-shaped symbols + * (`function`, `method`, `getter`, `setter`). NULL for non-functions and + * trivial bodies (`body_line_count < 2`). + */ + body_hash?: string | null; } // SQLite 3.32+ (2020+) default; bun:sqlite + better-sqlite3 12.x both ship @@ -1006,8 +1015,8 @@ export function insertSymbols(db: CodemapDatabase, symbols: SymbolRow[]) { batchInsert( db, symbols, - "INSERT INTO symbols (file_path, name, kind, line_start, line_end, signature, is_exported, is_default_export, members, doc_comment, value, parent_name, visibility, complexity, cognitive_complexity, name_column_start, name_column_end, scope_local_id, body_line_count, param_count, nesting_depth, return_type, is_async, is_generator)", - "(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", + "INSERT INTO symbols (file_path, name, kind, line_start, line_end, signature, is_exported, is_default_export, members, doc_comment, value, parent_name, visibility, complexity, cognitive_complexity, name_column_start, name_column_end, scope_local_id, body_line_count, param_count, nesting_depth, return_type, is_async, is_generator, body_hash)", + "(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", (s, v) => v.push( s.file_path, @@ -1034,6 +1043,7 @@ export function insertSymbols(db: CodemapDatabase, symbols: SymbolRow[]) { s.return_type ?? null, s.is_async ?? 0, s.is_generator ?? 0, + s.body_hash ?? null, ), ); } diff --git a/src/extractors/body-hash.test.ts b/src/extractors/body-hash.test.ts new file mode 100644 index 00000000..506bc9d2 --- /dev/null +++ b/src/extractors/body-hash.test.ts @@ -0,0 +1,406 @@ +import { describe, expect, it } from "bun:test"; + +import { extractFileData } from "../parser"; +import { canonicalizeBody, hashFunctionBody } from "./body-hash"; + +describe("canonicalizeBody", () => { + it("normalizes identifiers to $id", () => { + const a = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "ReturnStatement", + argument: { type: "Identifier", name: "foo" }, + }, + ], + }); + const b = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "ReturnStatement", + argument: { type: "Identifier", name: "bar" }, + }, + ], + }); + expect(a).toBe(b); + expect(a).toContain("$id"); + }); + + it("normalizes absent returns to Literal:nullish", () => { + const nullRet = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "ReturnStatement", + argument: { type: "Literal", value: null }, + }, + ], + }); + const undefRet = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "ReturnStatement", + argument: { type: "Identifier", name: "undefined" }, + }, + ], + }); + const voidRet = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "ReturnStatement", + argument: { + type: "UnaryExpression", + operator: "void", + prefix: true, + argument: { type: "Literal", value: 0 }, + }, + }, + ], + }); + const bareRet = canonicalizeBody({ + type: "BlockStatement", + body: [{ type: "ReturnStatement" }], + }); + expect(nullRet).toBe(undefRet); + expect(nullRet).toBe(voidRet); + expect(nullRet).toBe(bareRet); + expect(nullRet).toContain("Literal:nullish"); + }); + + it("void 0 is nullish but void call is not", () => { + const voidZero = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "ReturnStatement", + argument: { + type: "UnaryExpression", + operator: "void", + prefix: true, + argument: { type: "Literal", value: 0 }, + }, + }, + ], + }); + const voidCall = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "ReturnStatement", + argument: { + type: "UnaryExpression", + operator: "void", + prefix: true, + argument: { + type: "CallExpression", + callee: { type: "Identifier", name: "sideEffect" }, + arguments: [], + }, + }, + }, + ], + }); + expect(voidZero).toContain("Literal:nullish"); + expect(voidCall).not.toContain("Literal:nullish"); + expect(voidZero).not.toBe(voidCall); + }); + + it("does not nullish-normalize outside return position", () => { + const nullCheck = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "IfStatement", + test: { + type: "BinaryExpression", + operator: "===", + left: { type: "Identifier", name: "x" }, + right: { type: "Literal", value: null }, + }, + consequent: { type: "BlockStatement", body: [] }, + }, + ], + }); + const undefCheck = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "IfStatement", + test: { + type: "BinaryExpression", + operator: "===", + left: { type: "Identifier", name: "x" }, + right: { type: "Identifier", name: "undefined" }, + }, + consequent: { type: "BlockStatement", body: [] }, + }, + ], + }); + expect(nullCheck).not.toBe(undefCheck); + expect(nullCheck).toContain("Literal:null"); + expect(undefCheck).toContain("$id"); + }); + + it("normalizes literal values to kind only", () => { + const a = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "ReturnStatement", + argument: { type: "Literal", value: "foo" }, + }, + ], + }); + const b = canonicalizeBody({ + type: "BlockStatement", + body: [ + { + type: "ReturnStatement", + argument: { type: "Literal", value: "bar" }, + }, + ], + }); + expect(a).toBe(b); + expect(a).toContain("Literal:string"); + }); +}); + +describe("body_hash extraction", () => { + it("null/undefined/void0/bare return variants share body_hash", () => { + const mk = (ret: string) => `export function fn(): unknown { + const n = 1; + ${ret} +} +`; + const nullFn = extractFileData( + "/proj/a.ts", + mk("return null;"), + "a.ts", + ).symbols.find((s) => s.name === "fn"); + const undefFn = extractFileData( + "/proj/b.ts", + mk("return undefined;"), + "b.ts", + ).symbols.find((s) => s.name === "fn"); + const voidFn = extractFileData( + "/proj/c.ts", + mk("return void 0;"), + "c.ts", + ).symbols.find((s) => s.name === "fn"); + const bareFn = extractFileData( + "/proj/d.ts", + mk("return;"), + "d.ts", + ).symbols.find((s) => s.name === "fn"); + expect(nullFn?.body_hash).toBeTruthy(); + expect(nullFn?.body_hash).toBe(undefFn?.body_hash); + expect(nullFn?.body_hash).toBe(voidFn?.body_hash); + expect(nullFn?.body_hash).toBe(bareFn?.body_hash); + }); + + it("FunctionDeclaration body_hash lands on function row not param rows", () => { + const src = `export function fn(a: number, b: string): void { + const x = a; + return; +} +`; + const data = extractFileData("/proj/x.ts", src, "x.ts"); + const fn = data.symbols.find( + (s) => s.name === "fn" && s.kind === "function", + ); + const params = data.symbols.filter((s) => s.kind === "param"); + expect(fn?.body_hash).toBeTruthy(); + expect(params.length).toBeGreaterThan(0); + for (const p of params) { + expect(p.body_hash ?? null).toBeNull(); + } + }); + + it("isomorphic FunctionDeclaration bodies share body_hash", () => { + const aSrc = `export function alpha(x: number): number { + if (x > 0) { + return x; + } + return 0; +} +`; + const bSrc = `export function beta(y: number): number { + if (y > 0) { + return y; + } + return 0; +} +`; + const a = extractFileData("/proj/a.ts", aSrc, "bench/duplicate-a.ts"); + const b = extractFileData("/proj/b.ts", bSrc, "bench/duplicate-b.ts"); + const aSym = a.symbols.find((s) => s.name === "alpha"); + const bSym = b.symbols.find((s) => s.name === "beta"); + expect(aSym?.body_hash).toBeTruthy(); + expect(aSym?.body_hash).toBe(bSym?.body_hash); + }); + + it("different control flow yields different body_hash", () => { + const aSrc = `export function plain(n: number): number { + return n; +} +`; + const bSrc = `export function branch(n: number): number { + if (n > 0) return n; + return 0; +} +`; + const a = extractFileData("/proj/a.ts", aSrc, "a.ts").symbols.find( + (s) => s.name === "plain", + ); + const b = extractFileData("/proj/b.ts", bSrc, "b.ts").symbols.find( + (s) => s.name === "branch", + ); + expect(a?.body_hash).toBeTruthy(); + expect(b?.body_hash).toBeTruthy(); + expect(a?.body_hash).not.toBe(b?.body_hash); + }); + + it("skips hash when body_line_count < 2", () => { + const src = `export function tiny() { return 1; }`; + const sym = extractFileData("/proj/x.ts", src, "x.ts").symbols.find( + (s) => s.name === "tiny", + ); + expect(sym?.body_line_count).toBe(1); + expect(sym?.body_hash ?? null).toBeNull(); + }); + + it("leaves body_hash null on non-function symbols", () => { + const src = `export const x = 1;`; + const sym = extractFileData("/proj/x.ts", src, "x.ts").symbols.find( + (s) => s.name === "x", + ); + expect(sym?.body_hash ?? null).toBeNull(); + }); + + it("hashFunctionBody returns null for trivial span", () => { + expect( + hashFunctionBody({ type: "BlockStatement", body: [] }, 1), + ).toBeNull(); + }); + + it("isomorphic named arrow bodies share body_hash", () => { + const aSrc = `export const arrowA = (x: number): number => { + if (x > 0) { + return x; + } + return 0; +}; +`; + const bSrc = `export const arrowB = (y: number): number => { + if (y > 0) { + return y; + } + return 0; +}; +`; + const a = extractFileData("/proj/a.ts", aSrc, "a.ts").symbols.find( + (s) => s.name === "arrowA", + ); + const b = extractFileData("/proj/b.ts", bSrc, "b.ts").symbols.find( + (s) => s.name === "arrowB", + ); + expect(a?.body_hash).toBeTruthy(); + expect(a?.body_hash).toBe(b?.body_hash); + }); + + it("same template shape with different quasi text shares body_hash", () => { + const aSrc = `export function a(): string { + return \`hello \${x}\`; +} +`; + const bSrc = `export function b(): string { + return \`world \${y}\`; +} +`; + const a = extractFileData("/proj/a.ts", aSrc, "a.ts").symbols.find( + (s) => s.name === "a", + ); + const b = extractFileData("/proj/b.ts", bSrc, "b.ts").symbols.find( + (s) => s.name === "b", + ); + expect(a?.body_hash).toBe(b?.body_hash); + }); + + it("isomorphic class getters share body_hash", () => { + const aSrc = `class A { + get val(): number { + if (this.x > 0) return this.x; + return 0; + } +} +`; + const bSrc = `class B { + get val(): number { + if (this.y > 0) return this.y; + return 0; + } +} +`; + const a = extractFileData("/proj/a.ts", aSrc, "a.ts").symbols.find( + (s) => s.name === "val" && s.kind === "getter", + ); + const b = extractFileData("/proj/b.ts", bSrc, "b.ts").symbols.find( + (s) => s.name === "val" && s.kind === "getter", + ); + expect(a?.body_hash).toBeTruthy(); + expect(a?.body_hash).toBe(b?.body_hash); + }); + + it("isomorphic class setters share body_hash", () => { + const aSrc = `class A { + set val(x: number) { + if (x > 0) this.x = x; + else this.x = 0; + } +} +`; + const bSrc = `class B { + set val(y: number) { + if (y > 0) this.y = y; + else this.y = 0; + } +} +`; + const a = extractFileData("/proj/a.ts", aSrc, "a.ts").symbols.find( + (s) => s.name === "val" && s.kind === "setter", + ); + const b = extractFileData("/proj/b.ts", bSrc, "b.ts").symbols.find( + (s) => s.name === "val" && s.kind === "setter", + ); + expect(a?.body_hash).toBeTruthy(); + expect(a?.body_hash).toBe(b?.body_hash); + }); + + it("isomorphic class methods share body_hash", () => { + const aSrc = `class A { + run(x: number): number { + if (x > 0) return x; + return 0; + } +} +`; + const bSrc = `class B { + go(y: number): number { + if (y > 0) return y; + return 0; + } +} +`; + const a = extractFileData("/proj/a.ts", aSrc, "a.ts").symbols.find( + (s) => s.name === "run", + ); + const b = extractFileData("/proj/b.ts", bSrc, "b.ts").symbols.find( + (s) => s.name === "go", + ); + expect(a?.body_hash).toBeTruthy(); + expect(a?.body_hash).toBe(b?.body_hash); + }); +}); diff --git a/src/extractors/body-hash.ts b/src/extractors/body-hash.ts new file mode 100644 index 00000000..8aa296d9 --- /dev/null +++ b/src/extractors/body-hash.ts @@ -0,0 +1,186 @@ +/** + * Structural body fingerprint for function-shaped symbols. Canonical AST walk + * on function bodies; symbol index via `complexity.markArrowSymbol` / `getArrowSymbol`. + */ + +import type { SymbolRow } from "../db"; +import { hashContent } from "../hash"; +import type { TierExtractor } from "./types"; + +/** Return-position absent values (`null` / `undefined` / `void 0` / bare `return`) → this token. */ +const NULLISH_LITERAL = "Literal:nullish"; + +const SKIP_KEYS = new Set([ + "loc", + "start", + "end", + "range", + "raw", + "cooked", + "name", + "value", + "bigint", + "regex", + "flags", + "optional", + "decorators", + "typeAnnotation", + "returnType", + "typeParameters", +]); + +/** Depth-first canonical serialization of a function `body` subtree. */ +export function canonicalizeBody(body: unknown): string { + const parts: string[] = []; + walk(body, parts); + return parts.join(""); +} + +/** SHA-256 hex of canonical body; NULL when body missing or `body_line_count < 2`. */ +export function hashFunctionBody( + body: unknown, + bodyLineCount: number | null | undefined, +): string | null { + if (!body || (bodyLineCount ?? 0) < 2) return null; + return hashContent(canonicalizeBody(body)); +} + +function walk(node: unknown, parts: string[], normalizeNullish = false): void { + if (node == null) return; + + if (Array.isArray(node)) { + parts.push("["); + for (const item of node) walk(item, parts, normalizeNullish); + parts.push("]"); + return; + } + + if (typeof node !== "object") return; + + const n = node as { type?: string }; + + if (n.type === "Identifier" || n.type === "BindingIdentifier") { + if (normalizeNullish && (node as { name?: string }).name === "undefined") { + parts.push(NULLISH_LITERAL); + return; + } + parts.push("$id"); + return; + } + + if ( + normalizeNullish && + n.type === "UnaryExpression" && + (node as { operator?: string; prefix?: boolean }).operator === "void" && + (node as { prefix?: boolean }).prefix + ) { + const voidArg = (node as { argument?: { type?: string; value?: unknown } }) + .argument; + if (voidArg?.type === "Literal" && voidArg.value === 0) { + parts.push(NULLISH_LITERAL); + return; + } + } + + if (n.type === "ReturnStatement") { + parts.push(n.type); + const arg = (node as { argument?: unknown }).argument; + parts.push("argument"); + if (arg == null) { + parts.push(NULLISH_LITERAL); + } else { + walk(arg, parts, true); + } + return; + } + + if (n.type === "PrivateIdentifier") { + parts.push("$id"); + return; + } + + if (n.type === "Literal") { + const kind = literalKind( + node as { value?: unknown; bigint?: string; regex?: unknown }, + ); + if (normalizeNullish && kind === "nullish") { + parts.push(NULLISH_LITERAL); + } else { + parts.push(`Literal:${kind}`); + } + return; + } + + if (n.type === "TemplateElement") { + parts.push("TemplateElement"); + return; + } + + if (n.type === "TemplateLiteral") { + const tl = node as { quasis?: unknown[]; expressions?: unknown[] }; + parts.push("TemplateLiteral"); + walk(tl.quasis, parts, normalizeNullish); + walk(tl.expressions, parts, normalizeNullish); + return; + } + + if (!n.type) return; + + parts.push(n.type); + const record = node as Record; + const keys = Object.keys(record) + .filter((k) => k !== "type" && !SKIP_KEYS.has(k)) + .sort(); + for (const key of keys) { + parts.push(key); + walk(record[key], parts, normalizeNullish); + } +} + +function literalKind(node: { + value?: unknown; + bigint?: string; + regex?: unknown; +}): string { + if (node.regex != null) return "regexp"; + if (node.bigint != null) return "bigint"; + if (node.value === null) return "nullish"; + return typeof node.value; +} + +const FUNCTION_SHAPED_KINDS = new Set([ + "function", + "method", + "getter", + "setter", +]); + +function assignBodyHashForSymbolIndex( + symbols: SymbolRow[], + symbolIndex: number | undefined, + body: unknown, +): void { + if (symbolIndex === undefined || symbolIndex < 0) return; + const sym = symbols[symbolIndex]; + if (!sym || !FUNCTION_SHAPED_KINDS.has(sym.kind)) return; + sym.body_hash = hashFunctionBody(body, sym.body_line_count); +} + +export const bodyHashExtractor: TierExtractor = { + tierId: "body-hash", + register(visitor, ctx) { + const onFnExit = (node: { body?: unknown }) => { + assignBodyHashForSymbolIndex( + ctx.symbols, + ctx.complexity.getArrowSymbol(node), + node.body, + ); + }; + + Object.assign(visitor, { + "FunctionDeclaration:exit": onFnExit, + "ArrowFunctionExpression:exit": onFnExit, + "FunctionExpression:exit": onFnExit, + }); + }, +}; diff --git a/src/extractors/symbols.ts b/src/extractors/symbols.ts index c5feeb2d..26024f10 100644 --- a/src/extractors/symbols.ts +++ b/src/extractors/symbols.ts @@ -106,6 +106,7 @@ function registerSymbolHandlers( ...functionShapeColumns(node), }); complexity.pushFor(symbolIndex); + complexity.markArrowSymbol(node, symbolIndex); scopes.push(name, "function", lineStart, lineEnd); ctx.claimedScopeNodes.add(node); diff --git a/src/extractors/types.ts b/src/extractors/types.ts index a44eecd7..af699578 100644 --- a/src/extractors/types.ts +++ b/src/extractors/types.ts @@ -55,10 +55,10 @@ export interface ComponentDetector { * `symbolIndex = -1` = anonymous (callbacks, IIFEs) — counted but * never persisted so branches don't bleed into the outer scope. * - * `markArrowSymbol(node, idx)` / `getArrowSymbol(node)` bridge the - * two-visit gap for arrow inits: symbol row pushed at - * `VariableDeclaration`, complexity counter pushed at - * `ArrowFunctionExpression`. WeakMap keyed by the init AST node. + * `markArrowSymbol(node, idx)` / `getArrowSymbol(node)` O(1) symbol index + * by function-shaped AST node (WeakMap). Arrows: row at `VariableDeclaration`, + * counter at `ArrowFunctionExpression`. `FunctionDeclaration`: row at enter, + * hash at exit — same map after `pushParams` would make `length - 1` wrong. */ export interface ComplexityTracker { pushFor(symbolIndex: number): void; diff --git a/src/parser.ts b/src/parser.ts index abf7d140..5ccece46 100644 --- a/src/parser.ts +++ b/src/parser.ts @@ -28,6 +28,7 @@ import type { DynamicImportRow, } from "./db"; import { behavioralExtractor } from "./extractors/behavioral"; +import { bodyHashExtractor } from "./extractors/body-hash"; import { callsExtractor } from "./extractors/calls"; import { complexityExtractor, @@ -117,6 +118,7 @@ const EXTRACTORS: readonly TierExtractor[] = [ symbolsExtractor, scopesExtractor, complexityExtractor, + bodyHashExtractor, callsExtractor, componentsExtractor, referencesExtractor, diff --git a/templates/agent-content/rule/00-full.md b/templates/agent-content/rule/00-full.md index f9658444..83de3b50 100644 --- a/templates/agent-content/rule/00-full.md +++ b/templates/agent-content/rule/00-full.md @@ -70,6 +70,7 @@ If the question matches any of these, use the index instead of grepping: | "What's high-complexity AND undertested?" | `--recipe high-complexity-untested` (needs `ingest-coverage`; without ingest prefer `high-crap-score`) | | "Complex + undertested without coverage ingest?" | `--recipe high-crap-score` (graph-estimated tiers; `coverage_source: estimated`) | | "What's cognitively complex (nesting-heavy)?" | `--recipe high-cognitive-complexity` (default `min_score=15`; `--params min_score=20` to tighten) | +| "Structurally duplicate function bodies?" | `--recipe duplicates` (rename-insensitive `body_hash`; triage with `snippet` before refactor) | ## Quick reference queries diff --git a/templates/recipes/duplicates.md b/templates/recipes/duplicates.md new file mode 100644 index 00000000..ca1eb432 --- /dev/null +++ b/templates/recipes/duplicates.md @@ -0,0 +1,34 @@ +--- +params: + - name: min_count + type: number + required: false + default: 2 + description: Minimum symbols sharing the same body_hash to surface (default 2) + - name: path_prefix + type: string + required: false + default: "" + description: Optional file_path prefix filter (empty = all indexed paths) + - name: min_body_lines + type: number + required: false + default: 2 + description: Minimum function span in lines (default 2; excludes one-line arrows and tiny bodies) +actions: + - type: review-duplicate-bodies + auto_fixable: false + description: "Function bodies with identical structural body_hash — rename-insensitive (identifiers and literal values erased). Triage with snippet; extract shared helper when confirmed." +--- + +# duplicates + +Symbols whose **`body_hash`** collides — structurally identical function bodies (top-level `function`, named arrow/const inits, class methods/getters/setters). Distinct from token-level copy-paste / suffix-array duplication engines. + +```bash +codemap query --recipe duplicates +codemap query --recipe duplicates --params path_prefix=src/lib/ +codemap query --recipe duplicates --params min_body_lines=3 +``` + +False positives are possible when unrelated functions share the same control-flow skeleton, or when sync vs async / generator flags differ but the block body matches — use `codemap snippet` before refactoring. Results cap at 50 rows per query (no truncation marker). diff --git a/templates/recipes/duplicates.sql b/templates/recipes/duplicates.sql new file mode 100644 index 00000000..e7b05ec8 --- /dev/null +++ b/templates/recipes/duplicates.sql @@ -0,0 +1,30 @@ +WITH params(min_count, path_prefix, min_body_lines) AS ( + SELECT ?, ?, ? +), +filtered AS ( + SELECT s.* + FROM symbols s + CROSS JOIN params p + WHERE s.body_hash IS NOT NULL + AND (p.path_prefix = '' OR s.file_path LIKE p.path_prefix || '%') + AND COALESCE(s.body_line_count, 0) >= p.min_body_lines +), +grouped AS ( + SELECT body_hash, COUNT(*) AS duplicate_count + FROM filtered + GROUP BY body_hash + HAVING COUNT(*) >= (SELECT min_count FROM params) +) +SELECT + s.name, + s.kind, + s.file_path, + s.line_start, + s.line_end, + s.body_hash, + s.body_line_count, + g.duplicate_count +FROM filtered s +INNER JOIN grouped g ON g.body_hash = s.body_hash +ORDER BY g.duplicate_count DESC, s.file_path, s.name +LIMIT 50;