Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
250 changes: 235 additions & 15 deletions Cargo.lock

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -135,12 +135,17 @@ self-replace = "1"
memmap2 = "0.9"
crossterm = "0.28"
fs2 = "0.4"
reflink-copy = "0.1"
sysinfo = { version = "0.32", default-features = false, features = ["system"] }
tree-sitter-hlsl = { version = "0.2.0", optional = true }
amari-holographic = "0.23.0"
regex = "1.12.3"
open = "5.3"
tiktoken-rs = { version = "0.12", optional = true }
# Raw filesystem watcher for the daemon git-metadata watcher (src/daemon/git_watch.rs).
# We deliberately use the raw watcher (not notify-debouncer-full) and debounce
# ourselves so a monorepo's git-ref churn coalesces into a single sync.
notify = "6"

[target.'cfg(not(windows))'.dependencies]
flate2 = "1"
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ Hermes wrappers run from Hermes' current working directory, use a 600-second tim
For Codex install details, see [Codex plugin installs](#codex-plugin-installs). For Cursor, both global and `--local` install put the plugin in `~/.cursor/plugins/local/tracedecay` and require a Cursor reload. The plugin MCP config runs `tracedecay serve --path ${workspaceFolder}`, so it resolves the active workspace instead of the plugin directory and uses that workspace's active project store rather than the legacy global Cursor MCP registration. Cursor install no longer writes `.cursor/mcp.json`, `.cursor/hooks.json`, `.cursor/rules/tracedecay.mdc` (legacy artifact name), or `.cursor/permissions.json`; approvals are left to Cursor approval/run-mode behavior. The plugin hooks are:

- `sessionStart` — fire-and-forget; injects context steering the Agent toward tracedecay MCP tools and reports index freshness (suggests `tracedecay init` when no initialized project store is found).
- `postToolUse` (unmatched — Cursor's docs enumerate no matcher value for semantic search) — fail-open; injects a soft `additional_context` hint after broad search/read tools (Grep, Glob, Read, semantic search, shell `rg`) so Cursor switches to `tracedecay_context`, `tracedecay_search`, `tracedecay_outline`, or `tracedecay_files`. Each hint category is emitted at most once per session (persisted in `.tracedecay/tool_hints_seen.json`).
- `postToolUse` (unmatched — Cursor's docs enumerate no matcher value for semantic search) — fail-open; injects a soft `additional_context` hint after broad search/read tools (Grep, Glob, Read, semantic search, shell `rg`) so Cursor switches to `tracedecay_grep`, `tracedecay_context`, `tracedecay_search`, `tracedecay_outline`, or `tracedecay_files`. Each hint category is emitted at most once per session (persisted in `.tracedecay/tool_hints_seen.json`).
- `beforeSubmitPrompt` — resets the local token counter for the new turn and ingests the current Cursor transcript into the active project session store when `transcript_path` is present.
- `afterFileEdit` (unmatched, so every Agent edit tool counts) — runs a **targeted single-file** sync of just the edited path(s) via `sync_if_stale_silent`, never a full-tree scan (which would scale with repo size, not edit size).
- `afterShellExecution` — on Agent-run `git checkout`/`switch`/`worktree add`, bootstraps/maintains tracedecay branch tracking (`branch add`); on other state-changing git commands (pull/merge/rebase/reset/cherry-pick/stash apply|pop), runs a coalesced incremental sync.
Expand Down Expand Up @@ -446,7 +446,7 @@ The default query set targets patterns present in most application codebases (CL

Each repo is shallow-cloned (`git init` + `git fetch --progress --depth 1 origin <ref>` + `checkout FETCH_HEAD`) on first use and cached locally; subsequent runs reuse the checkout. Git output is streamed to the terminal so the multi-GB fetch shows real-time progress.

**Tools covered (5 queries each).** Read tools — `search`, `context`, `callers`, `callees`, `node`, `by_qualified_name`, `signature`, `impact`, `body`, `files`, `complexity`, `doc_coverage`, `largest`, `hotspots`, `god_class`, `module_api`, `derives`, `dead_code`, `rank`, `coupling`, `circular`, and `outline` when `ast-grep` >= 0.44 is on `PATH`. Write tools — `str_replace`, `multi_str_replace`, `insert_at`, and (when the user-installed `ast-grep` CLI is available) `ast_grep_rewrite`.
**Tools covered (5 queries each).** Read tools — `grep`, `search`, `context`, `callers`, `callees`, `node`, `by_qualified_name`, `signature`, `impact`, `body`, `files`, `complexity`, `doc_coverage`, `largest`, `hotspots`, `god_class`, `module_api`, `derives`, `dead_code`, `rank`, `coupling`, `circular`, and `outline` when `ast-grep` >= 0.44 is on `PATH`. Write tools — `str_replace`, `multi_str_replace`, `insert_at`, and (when the user-installed `ast-grep` CLI is available) `ast_grep_rewrite`.

**Force-sync on every run.** Before any benchmark fires, the harness runs the equivalent of `tracedecay sync --force` on each repo (`index_all()` regardless of `.tracedecay/` freshness) so timings always reflect the pinned source.

Expand Down
12 changes: 12 additions & 0 deletions TEST-QUERIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,18 @@ Expected: Returns matching symbols with IDs, file paths, line numbers, and signa

---

## tracedecay_grep

> Search indexed code for the literal string "mcpServers".

Test:
```
tracedecay_grep(pattern="mcpServers", fixed_strings=true, path_glob="src/**/*.rs", context_lines=1)
```
Expected: Returns matching source lines with file paths, line numbers, and enclosing symbol metadata. Use this for literal strings, regexes, and config keys inside indexed code; use `tracedecay_search` for symbol names.

---

## tracedecay_context

> Build context for the task: "understand how the MCP server handles incoming tool calls"
Expand Down
9 changes: 6 additions & 3 deletions docs/PLUGIN-VALIDATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ JSON artifacts in the bundles are validated against vendored JSON Schemas in
| `plugin/.codex-plugin/plugin.json` | `cursor-schemas/plugin.schema.json` + `interface` extension | `tests/agent_suite/plugin_manifest_schema_test.rs` |
| `plugin/.claude-plugin/plugin.json` | `claude-schemas/plugin.schema.json` | `tests/agent_suite/claude_plugin_schema_test.rs` |
| `plugin/.claude-plugin/marketplace.json` | `claude-schemas/marketplace.schema.json` | `tests/agent_suite/claude_plugin_schema_test.rs` |
| `plugin/.mcp.json` | `cursor-schemas/mcp.schema.json` | `tests/agent_suite/plugin_config_schema_test.rs` |
| `plugin/mcp-cursor.json` (deploys as `mcp.json`) | `cursor-schemas/mcp.schema.json` | `tests/agent_suite/plugin_config_schema_test.rs` |
| `plugin/hooks/hooks-cursor.json` and `plugin/hooks/hooks-codex.json` | `cursor-schemas/hooks.schema.json` | `tests/agent_suite/plugin_config_schema_test.rs` |
| `plugin/hooks/hooks-claude.json` | `claude-schemas/hooks.schema.json` | `tests/agent_suite/claude_plugin_schema_test.rs` |
Expand All @@ -67,10 +68,12 @@ that host manifests share the same plugin `name`. The config-schema tests
include negative cases proving the mcp/hooks schemas actually reject
malformed configs (missing `command`, unknown fields, typo'd event names).

The Cursor plugin schema declares `additionalProperties: false`, and Codex
The Cursor plugin/hooks schemas declare `additionalProperties: false`. Codex
marketplaces read an `interface` display-metadata block that Cursor's schema
doesn't define. The Codex manifest is therefore validated against the Cursor
schema plus exactly that one extra key, derived in the test.
doesn't define, and the repo-local Codex hook seed carries a top-level
`description` explaining why its `hooks` object is empty. Those two Codex
surfaces are validated against the Cursor schemas plus exactly those
host-specific keys, derived in the tests.

### 2. Skill contract tests (cargo test)

Expand Down
5 changes: 3 additions & 2 deletions docs/USER-GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ This is the default. It registers the MCP server in `~/.claude/settings.json`, g
- `UserPromptSubmit` resets the per-turn token-savings counter.
- `Stop` ingests new session transcript data and prints a cost receipt.
- `SessionStart` reports index freshness (or a `tracedecay init` nudge) and, when the session restarts from compaction, injects the LCM context-recovery hint.
- `PostToolUse` (matcher `Edit|MultiEdit|Write|NotebookEdit|Bash`) notifies the daemon so edits and shell commands trigger targeted incremental sync.
- `PostToolUse` (matcher `Edit|MultiEdit|Write|NotebookEdit|Grep|Glob|Read|Bash`) notifies the daemon so edits and shell commands trigger targeted incremental sync, and broad search/read tools get routed toward TraceDecay equivalents.

The install also ships three read-only custom subagents into `~/.claude/agents/` — `code-explorer`, `code-health-auditor`, and `session-historian` — the same tracedecay subagents the Cursor plugin bundles. They are only replaced or removed when the file is tracedecay-managed; a same-named agent you authored yourself is left untouched. `tracedecay update-plugin` refreshes installed copies.

Expand Down Expand Up @@ -313,7 +313,7 @@ Cursor install is plugin-based:
- Cursor install no longer writes `.cursor/mcp.json`, `.cursor/hooks.json`, `.cursor/rules/tracedecay.mdc`, or `.cursor/permissions.json`; approvals are left to Cursor approval/run-mode behavior.
- The plugin bundles Cursor-specific, fail-open hooks. File and shell hooks notify the TraceDecay daemon; if no daemon is available they return success without indexing:
- `sessionStart` injects context steering the Agent toward tracedecay MCP tools and reports index freshness (suggests `tracedecay init` when uninitialized).
- `postToolUse` (unmatched) injects a nonblocking `additional_context` hint after broad search/read tools (Grep, Glob, Read, semantic search, shell `rg`) so Cursor can switch to `tracedecay_context`, `tracedecay_search`, `tracedecay_outline`, or `tracedecay_files`; each hint category fires at most once per session.
- `postToolUse` (unmatched) injects a nonblocking `additional_context` hint after broad search/read tools (Grep, Glob, Read, semantic search, shell `rg`) so Cursor can switch to `tracedecay_grep`, `tracedecay_context`, `tracedecay_search`, `tracedecay_outline`, or `tracedecay_files`; each hint category fires at most once per session.
- `beforeSubmitPrompt` resets the local token counter and ingests the current Cursor transcript into the active project session store when `transcript_path` is present.
- `afterFileEdit` (unmatched, so every Agent edit tool counts) sends the edited path(s) to the daemon, whose MCP server runs a **targeted single-file** sync — not a full-tree scan — so it stays cheap on large codebases even when the Agent edits many files per turn.
- `afterShellExecution` sends shell command effects to the daemon, whose MCP server makes branch handling automatic: Agent-run `git checkout`/`switch`/`worktree add` bootstraps/maintains tracedecay branch tracking (`branch add`), while other state-changing git commands (pull/merge/rebase/reset/cherry-pick/stash apply|pop) trigger a coalesced incremental sync.
Expand Down Expand Up @@ -565,6 +565,7 @@ When running as an MCP server, tracedecay exposes more than 70 tools that AI age
| Tool | What it does |
|------|-------------|
| `tracedecay_context` | Given a task description, returns relevant symbols, relationships, and code snippets. This is the go-to starting point for any coding task. |
| `tracedecay_grep` | Search indexed code content by literal string or regex, with each hit annotated by its enclosing symbol. |
| `tracedecay_search` | Find symbols by name. Supports filtering by kind (function, class, method, etc.). |
| `tracedecay_node` | Get full details for a specific symbol: source code, location, complexity metrics, and relationships. |
| `tracedecay_files` | List indexed files, optionally filtered by directory or glob pattern. |
Expand Down
140 changes: 140 additions & 0 deletions eval/hermetic/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
# Hermetic eval harness

Triggering evals must exercise the tracedecay build **under development in this
worktree** — its binary *and* its plugin bundle — never the system-installed
`tracedecay` and never the user's real Claude Code config. Live concurrent
sessions depend on the real `~/.claude`, `~/.tracedecay`, and
`~/.cargo/bin/tracedecay`, so the harness touches none of them.

```bash
# One-shot: build + isolate + install + index + one trivial scenario.
eval/hermetic/run.sh smoke --debug --keep

# Full corpus against a reusable env:
ENV=$(eval/hermetic/run.sh setup --debug)
eval/hermetic/run.sh index --env-dir "$ENV" --project /path/to/repo
eval/hermetic/run.sh run --env-dir "$ENV" --corpus my-corpus.jsonl --model sonnet
eval/hermetic/run.sh teardown --env-dir "$ENV"
```

## Why a naive PATH override is not enough

An eval session launched via `claude -p` resolves tracedecay **three** ways:

1. **MCP server command** — the plugin registers an MCP server whose command is
the bare string `tracedecay`, resolved via `PATH` at session start.
2. **Hook commands** — baked as **absolute paths** at install time. The
installer substitutes `__TRACEDECAY_BIN__` with a concrete path
(`src/agents/claude.rs`), so a `PATH` override does *not* affect already
installed hooks.
3. **Plugin bundle** — skills / hooks / agents JSON that Claude Code loads from
its plugin marketplace under the config dir.

A `PATH` override alone only fixes (1). The harness must control all three.

## What the harness isolates (verified mechanisms)

| Concern | Lever | Evidence |
| --- | --- | --- |
| Claude config, transcripts, plugin bundle | `CLAUDE_CONFIG_DIR` | Smoke test: a throwaway `CLAUDE_CONFIG_DIR` gets its own `projects/`, `sessions/`, `.claude.json`; the session transcript lands at `<config>/projects/<slug>/<session_id>.jsonl`. |
| Where the installer writes the plugin | `HOME` (installer uses `home_dir()` → `$HOME`, then `$HOME/.claude`) | `src/agents/mod.rs::home_dir()` reads `$HOME`; `src/agents/claude.rs` writes `ctx.home/.claude/plugins/marketplaces/tracedecay`. |
| tracedecay graph/data home | `TRACEDECAY_DATA_DIR` | `src/config.rs::user_data_dir()` returns `$TRACEDECAY_DATA_DIR` when set, else `~/.tracedecay`. |
| tracedecay daemon socket | derives from the data dir; also pinned via `TRACEDECAY_DAEMON_SOCKET` | `src/daemon/service.rs::default_socket_path()` = `tracedecay_data_dir()/daemon.sock`, overridable by `TRACEDECAY_DAEMON_SOCKET`. Isolating the data dir already isolates the socket, so the harness never fights the real daemon. |
| Which binary the installer bakes | **staged copy of the dev binary at a non-cargo-target path** | `src/agents/mod.rs::which_tracedecay_from()` deliberately **refuses** a path under a cargo target dir (`target/{debug,release}`) and falls back to `PATH` — which would bake the *system* binary. See below. |
| Auth for `claude -p` | copy `~/.claude/.credentials.json` into the isolated config (or `ANTHROPIC_API_KEY`) | Smoke test: without it the isolated session prints `Not logged in`; with the copied credential it returns a real result and `session_id`. |

`HOME` and `CLAUDE_CONFIG_DIR` are pointed at the **same** physical directory
(`<env>/home/.claude`) so the installer's `$HOME/.claude` writes and Claude
Code's `CLAUDE_CONFIG_DIR` reads refer to one bundle.

### The cargo-target-binary trap (the crux)

`which_tracedecay()` treats any binary living under a cargo target dir as
ephemeral and **will not** bake it into hooks; it prefers a `PATH` match
instead. So running `target/release/tracedecay install` directly would silently
bake the **system** `tracedecay`, defeating the whole point.

The harness sidesteps this by copying the freshly built binary to
`<env>/bin/tracedecay` (a stable, non-cargo location) and running the installer
from **that** copy with `<env>/bin` first on `PATH`. Then:

* `current_exe` is the staged copy → baked into hook commands (fixes #2), and
* the MCP `tracedecay` command resolves to the staged copy via `PATH` (fixes #1),
* the plugin bundle is deployed from this worktree's `plugin/` dir (fixes #3).

`setup` asserts the staged path actually appears in the baked hook JSON and
warns loudly if it does not.

## Env dir layout

```
<TMPDIR>/eval-env-<timestamp>-<pid>/
bin/tracedecay staged dev binary (baked into hooks + first on PATH)
home/ fake $HOME
home/.claude/ == CLAUDE_CONFIG_DIR (plugin bundle, transcripts)
tracedecay-data/ == TRACEDECAY_DATA_DIR (graph db, daemon.sock, logs)
results/ results.jsonl, summary.md, per-scenario json/stderr
env.sh sourceable export block (for reuse and manual debugging)
```

`--keep` preserves the env for inspection; otherwise a freshly created env is
removed on exit. `teardown --env-dir` refuses to delete anything that is not an
`eval-env-*` dir under `$TMPDIR`.

## Corpus schema

One JSON object per line (the schema used by the session scratchpad corpora):

```json
{"id":"ev-001","category":"context","project_dir":"/abs/path/to/repo",
"prompt":"...","expected_tools":["tracedecay_context"],
"anti_tools":["Grep","Glob"],"providers":["sonnet"],"success":"..."}
```

`run` executes each `prompt` via `claude -p --output-format json` inside
`project_dir` (falling back to the indexed default project if the dir is
missing), recovers the `session_id`, reads that session's transcript from the
isolated config, and counts `tool_use` blocks. `score.py` classifies each tool
as tracedecay (name contains `tracedecay`) or native.

A scenario **passes** when at least one tracedecay tool was used and no
`anti_tools` appear. This is a deliberately simple end-state judge — the harness
exists to guarantee *isolation*, not to be a sophisticated grader; layer a
stricter judge on top of `results.jsonl` if needed.

Outputs land in `<env>/results/`: `results.jsonl` (one scored object per
scenario) and `summary.md` (pass count + per-scenario table).

## What it cannot isolate / limitations

- **Model non-determinism.** Tool-use counts vary run to run; treat pass/fail as
a signal over a corpus, not a single scenario.
- **The `tracedecay init` index cost.** Indexing the tracedecay repo is not
free; reuse an env dir with `--keep` across corpus runs.
- **Network / model backend.** Evals hit the real Anthropic API using the copied
credential. There is no offline mode.
- **Codex.** `codex exec` uses `CODEX_HOME` for its config, and
`tracedecay install --agent codex` exists, so a `--codex` mode is feasible.
It is intentionally **left as follow-up**: codex-exec auth handling and the
codex plugin-bundle deploy path need their own validation pass, out of scope
for the claude-focused smoke here. The scaffolding (`CODEX_HOME` env, staged
binary, isolated data dir) transfers directly when added.
- **Global cargo caches** (`~/.cargo/registry`) are shared — only the *output*
(`CARGO_TARGET_DIR=<worktree>/target`) is worktree-scoped. The build cannot
corrupt the user's install because it never writes to `~/.cargo/bin`.

## Post-merge re-eval protocol

After the plugin-suite changes merge, re-run to compare against the baseline
facts recorded in tracedecay memory:

1. **Rebuild** from the merged checkout: `run.sh setup` (drop `--debug` for a
representative release build).
2. **Re-index** the same target project into the fresh env.
3. **Rerun the same corpus** at the same `--model`.
4. **Compare** the new `summary.md` pass rate and per-scenario tracedecay-vs-
native counts against the baseline. Store the new baseline as a durable fact
(`tracedecay_fact_store`) so future regressions are visible.

Because every run is hermetic, differences between two runs are attributable to
the code change (plus model noise), not to drift in the user's real environment.
Loading
Loading