Skip to content

feat: EP / graphical baselines + analytic parity cell (search-extensibility B4b) - #5

Merged
Jammy2211 merged 2 commits into
mainfrom
feature/search-ext-b4b-ep-baseline
Oct 15, 2026
Merged

Jammy2211 merged 2 commits into
mainfrom
feature/search-ext-b4b-ep-baseline

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Summary

B4b of the search-extensibility epic: ports the EP / graphical-model baselines from autofit_workspace_developer/{ep,graphical} and adds a profiling cell for the analytic Gaussian parity benchmark, reproduced on the B4a laptop host. The exporter reads two sibling result schemas and exports a runtime wall plus a memory host_peak_rss record per row on the existing profiling-summary@2 axes (no Pulse registry row, no contract bump). Findings: wiki/campaigns/ep_graphical_baselines.md.

Closes #4

Witness not met as written, for the human to judge: the witness asks for the reproduced N3 and N10 walls to be within runtime-drift-2x-1ms of the committed baseline.json. They are not. All six cells (EP and graphical, N3/N10/N30) are more than 2x faster (ratios 0.14–0.48, status improved). Correctness reproduces exactly: every truth logL sum matches bit-for-bit, and both N30 sanity FAILs reproduce. Nothing regressed. The committed baselines (2026-05-20) predate the changes behind the developer postfix re-runs, and the new rows are the baselines from here on.

Decision taken (flagged)

  • Host pin: the epic's release-sweep pin (euclid-ral-gpu-2, A100) does not fit CPU Dynesty runs; B4b pins to the same laptop as B4a (DESKTOP-H143S82, WSL2, platform.node() in every row) so Pulse's same-host rule pairs these records with B4a's. Stated in the wiki page and the catalogue LIMITATIONS. Cross-machine reproduction on RAL is a later campaign.
  • Drift notes are not Pulse comparisons: the committed developer baselines recorded no host, so no record pair exists; each row carries its runtime-drift-2x-1ms note in reference, and comparisons stays empty.
  • Banked sigma_q95: the prompt says 12.18; the benchmark docstrings bank 12.15 and the closed form gives 12.1536. The cell checks against 12.15.

Reproduction (PyAutoFit df6d218f1, JAX CPU float64, one fresh-process fit per cell, serial)

method N wall s committed s ratio drift peak RSS MiB truth logL sum exact sanity (committed)
graphical 3 6.0 22.8 0.263 improved 420 yes PASS (PASS)
graphical 10 7.1 50.7 0.141 improved 446 yes PASS (PASS)
graphical 30 19.0 132.8 0.143 improved 515 yes FAIL (FAIL)
EP 3 37.1 251.9 0.147 improved 353 yes PASS (PASS)
EP 10 152.9 566.9 0.270 improved 403 yes PASS (PASS)
EP 30 842.8 1758.7 0.479 improved 642 yes FAIL (FAIL)
  • Correctness reproduces: all truth logL sums equal the committed floats bit-for-bit; both N30 sanity FAILs reproduce (graphical: max-logL rule only, −1658.694 vs threshold −1650.561; EP: shared centre only, 49.26 ± 0.110, |z| 6.68).
  • Witness not met as written: N3/N10 walls are not within 2x of baseline.json; all six are faster (improved). Against the developer postfix re-runs, graphical N3 is flat (1.06x), graphical N10 and EP N3 still faster (0.48x, 0.49x). No regression.
  • Another session's CPU jobs shared the host (load 2.5 → 4.7), so the seconds are conservative; EP N30's N10→N30 scaling should wait for a quiet-host repeat.

Analytic parity (seed 0, N = 5): PARITY: PASS (41/41), both hard caps pass, wall 130.0 s, peak RSS 332 MiB, closed-form sigma_q95 12.1536. analytic_gaussian_priors.py is recorded as parked NEEDS_FIX, not run.

Scripts Changed

  • scripts/gaussian_1d_ep/shared.py: shared simulator, sanity rules, harness and row writer, ported from the developer simulator/util/sanity.py.
  • scripts/gaussian_1d_ep/{ep,graphical}/baseline.py: the EP and graphical N3/N10/N30 cells.
  • scripts/gaussian_1d_ep/config/: verbatim developer config.
  • scripts/analytic_gaussian/parity/benchmark.py: runs autofit_workspace_test's analytic_gaussian.py unchanged and records its verdicts.
  • scripts/misc/tooling/build_catalogue.py: reads the autofit-profiling-baseline and -parity schemas; adds runtime + memory records, HTML sections and the host-pin and drift LIMITATIONS.
  • scripts/misc/test/test_build_catalogue.py: 5 new tests (16 total).
  • results/**, dashboard/**, wiki/campaigns/ep_graphical_baselines.md, README.md, AGENTS.md.

No PyAutoFit change.

Test Plan

  • ruff check ., ruff format --check .
  • pytest -q scripts/misc/test: 16 passed
  • build_catalogue.py --check --validate-with ../PyAutoPulse: current, ok
  • lychee: not installed locally (CI runs it)

Validation checklist (--auto run, plan was not pre-approved)

  • Effective level: supervised (header: supervised, cap: feature/medium → safe)
  • Plan: on the issue (feat: B4b EP / graphical baseline port + analytic benchmark cell (search-extensibility) #4)
  • Gate: tests 16/16 · smoke n/a (profiling repo, no downstream script surface) · review: independent reviewer (fresh-context agent, not the author) returned FINDINGS ×2, both enacted in the second commit (wiki parity sha corrected to 26877bc69; rows carry provenance.host_contention and the catalogue names it in validation.reason; cells now record load averages). Its claim dispositions are all basis-cited: port faithful (side-by-side + diff -r config), truth logL bit-exact, N30 FAILs reproduced, ratios recomputed, deviations (a)–(d) disclosed honestly; Pulse re-ingest is pending sequencing · Heart STALE 85, acknowledged reason list: "release validation incomplete: no rehearsal for current source". STALE passes this leg. An earlier RED ("PyAutoFit: CI failure") was a timing flake in test_emcee_call_path_runs_at_jitted_speed on the 3.12 leg; the failed jobs were rerun green, then pyauto-heart tick
  • Human: plan sound in hindsight?
  • Human: diff matches plan (no scope creep)?
  • Human: merge, amend, or reject, then log the outcome

🤖 Generated with Claude Code

Jammy2211 and others added 2 commits October 14, 2026 15:47
…sibility B4b)

Port autofit_workspace_developer/{ep,graphical} as scripts/gaussian_1d_ep/
{ep,graphical}/baseline.py (shared harness, verbatim config) and add
scripts/analytic_gaussian/parity/benchmark.py, which runs
autofit_workspace_test's analytic_gaussian.py parity script unchanged and
records its verdicts. Reproduced N3/N10/N30 on the B4a laptop host
(DESKTOP-H143S82, PyAutoFit df6d218f1): every truth logL sum matches the
committed value exactly, both N30 sanity FAILs reproduce, and every wall is
faster than the committed developer baseline beyond the 2x policy (improved).

build_catalogue.py reads two sibling result schemas
(autofit-profiling-baseline, autofit-profiling-parity) and exports a runtime
wall plus a memory host_peak_rss record per row on the existing axes; the
reproduction table is wiki/campaigns/ep_graphical_baselines.md.

Closes #4

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- wiki: the parity run used autofit_workspace_test 26877bc69 (after the
  825362d seeding change), not 8b86eca55 (the earlier probe).
- rows: provenance.host_contention records the observed load (another
  session shared the host); the exporter names it in validation.reason;
  cells now record load_average_before/after.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the decision-taken Autonomous run took a flagged decision; human judges at merge label Oct 14, 2026
@Jammy2211
Jammy2211 merged commit e63a891 into main Oct 15, 2026
1 check passed
@Jammy2211
Jammy2211 deleted the feature/search-ext-b4b-ep-baseline branch October 15, 2026 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

decision-taken Autonomous run took a flagged decision; human judges at merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: B4b EP / graphical baseline port + analytic benchmark cell (search-extensibility)

1 participant