diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 5897c09..d7c296a 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -1,9 +1,9 @@ # lint.yml — PR / push-to-main gate. # -# Cheap, fast, CPU-only. Runs on every PR and on push to main. It lints only what exists -# in the skeleton: ruff (check + format) and a lychee link check over README.md and -# AGENTS.md. No PyAuto library is installed and no script runs; the pytest and smoke legs -# arrive with the harness in a later phase of the search-extensibility epic. +# Cheap, fast, CPU-only. Runs on every PR and on push to main: ruff (check + format), the +# stdlib-only tests under scripts/misc/test, the "committed catalogue is current and +# validates with the PyAutoPulse reader" check, and a lychee link check over README.md and +# AGENTS.md. No PyAuto library is installed and no measurement cell runs. name: lint @@ -30,7 +30,7 @@ jobs: - name: Install lint tools run: | - pip install ruff + pip install ruff pytest pyyaml # lychee is a Rust binary; fetch the prebuilt release. The tarball # nests everything under lychee-x86_64-unknown-linux-gnu/, so strip # that leading directory and extract just the binary. @@ -43,6 +43,19 @@ jobs: - name: ruff format --check run: ruff format --check . + # Checked out after ruff so the Pulse tree is never linted as this repo's code. + - name: Checkout PyAutoPulse (the profiling-summary@2 reader) + uses: actions/checkout@v4 + with: + repository: PyAutoLabs/PyAutoPulse + path: _pulse + + - name: pytest (stdlib-only tooling tests) + run: python -m pytest -q scripts/misc/test + + - name: catalogue up to date and valid profiling-summary@2 + run: python scripts/misc/tooling/build_catalogue.py --check --validate-with _pulse + - name: lychee — markdown link-rot run: | # Exclude GitHub UI links that lychee tends to false-positive on, and the diff --git a/.github/workflows/pages_dashboard.yml b/.github/workflows/pages_dashboard.yml new file mode 100644 index 0000000..04f0119 --- /dev/null +++ b/.github/workflows/pages_dashboard.yml @@ -0,0 +1,79 @@ +name: Pages Dashboard + +# Publishes the producer page (`dashboard/index.html`), the PyAutoPulse read contract +# (`dashboard/catalogue.json`, profiling-summary@2) and the project badge feed +# (`dashboard/state.json`) at https://pyautolabs.github.io/autofit_profiling/. The files +# are rendered and committed by `scripts/misc/tooling/build_catalogue.py`; this workflow +# only validates and publishes them, then tells PyAutoPulse to re-read the summary. +# Mirrors autolens_profiling's pages_dashboard.yml (project-feed identity: +# PyAutoMind complete/2026/10/pulse-refresh-project-identity.md). + +on: + push: + branches: [main] + paths: + - "dashboard/**" + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: true + +jobs: + deploy: + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - uses: actions/checkout@v4 + - uses: actions/checkout@v4 + with: + repository: PyAutoLabs/PyAutoBrain + path: _brain + - uses: actions/checkout@v4 + with: + repository: PyAutoLabs/PyAutoPulse + path: _pulse + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Validate the catalogue with the Pulse reader and check it is current + run: | + pip install pyyaml + python scripts/misc/tooling/build_catalogue.py --check --validate-with _pulse + # enablement: true creates the Pages site on first run. + - uses: actions/configure-pages@v5 + with: + enablement: true + - name: Stage the dashboard as the site + run: | + mkdir -p _site + cp dashboard/index.html dashboard/catalogue.json dashboard/state.json _site/ + - name: Validate the project badge feed against the Brain contract + run: python3 _brain/board/_state.py _site/state.json + - uses: actions/upload-pages-artifact@v3 + with: + path: _site + - id: deployment + uses: actions/deploy-pages@v4 + - name: Tell the PyAutoPulse dashboard to refresh (repository_dispatch pulse-refresh) + env: + GH_TOKEN: ${{ secrets.PAT_PYAUTOLABS }} + run: | + if [ -z "$GH_TOKEN" ]; then + echo "::warning::PAT_PYAUTOLABS is not available to this repo — skipping the pulse-refresh dispatch to PyAutoLabs/PyAutoPulse." + exit 0 + fi + sha=$(git rev-parse HEAD) + gh api repos/PyAutoLabs/PyAutoPulse/dispatches \ + -f event_type=pulse-refresh \ + -f "client_payload[repo]=${{ github.repository }}" \ + -f "client_payload[sha]=${sha}" \ + -f "client_payload[summary]=dashboard/catalogue.json" + echo "Dispatched pulse-refresh to PyAutoLabs/PyAutoPulse (${{ github.repository }}@${sha})." diff --git a/AGENTS.md b/AGENTS.md index e70bd1f..4b47d6e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,8 +12,10 @@ It is a collection of standalone scripts, **not** an installable package — the `pyproject.toml`. These are the canonical, agent-agnostic instructions; `README.md` is the human-facing overview. -**Status:** skeleton born 2026-10-07 (PyAutoMind#492), phase B1 of the -`search-extensibility` epic. There is no harness, no exporter and no result yet. +**Status:** born 2026-10-07 (PyAutoMind#492, phase B1 of the `search-extensibility` +epic). B4a (2026-10-14) added the first cell — the Nautilus `search.fit` breakdown on +gaussian_x1 — the `profiling-summary@2` exporter and the Pages board; the repo reports to +PyAutoPulse as instance `fit`. The EP / graphical baseline port is B4b. ## Where this repo is going @@ -26,7 +28,7 @@ The work arrives in later phases of the `search-extensibility` epic, whose ledge `fit` (`profiling-summary@2`). This adopts Pulse's task `autofit_profiling_bootstrap`. - **B4b** — the EP / graphical-model baseline port. -Until then, do not add scripts, results or exporters outside the phase that owns them. +Do not add scripts, results or exporters outside the phase that owns them. PyAutoPulse validates the published summary contract; the Brain's profiling conductor judges it. Nothing here issues a verdict. @@ -35,10 +37,33 @@ judges it. Nothing here issues a verdict. ``` ruff.toml lint config AND the root sentinel (leaf scripts walk up to it) activate.sh RAL shared venv + PYTHONPATH (PyAutoNerves, PyAutoFit only) +config/ PyAutoConf overrides (library output defaults kept on purpose) +scripts///.py measurement cells, e.g. gaussian_x1/nautilus/search_fit_breakdown.py +scripts/misc/profiling/ stdlib helpers shared by cells (exclusive component timer) +scripts/misc/tooling/build_catalogue.py results/ -> dashboard/{catalogue.json,index.html,state.json} +scripts/misc/test/ stdlib-only tests (no PyAuto install) +results/ committed result rows (schema autofit-profiling-breakdown), one file per cell variant +dashboard/ generated: catalogue.json (profiling-summary@2), index.html, state.json +wiki/campaigns/ findings pages; search_fit_bottlenecks.md is epic 1's ranked table hpc/ hpc/sync (laptop-side RAL driver) + the batch_cpu submit template ``` -`scripts/`, `results/` and the summary exporter arrive in B4a. +## Producing a result and publishing it + +1. `source activate.sh`, run a cell (e.g. `python scripts/gaussian_x1/nautilus/search_fit_breakdown.py`); + it writes its row under `results/`. Laptop CPU cells need no RAL. +2. `python scripts/misc/tooling/build_catalogue.py --validate-with ../PyAutoPulse` renders + `dashboard/` and validates the catalogue with the Pulse reader; commit rows and + `dashboard/` together. The renderer is deterministic (its stamp is the newest + evidence time), so `--check` is the lint gate. +3. On merge, `.github/workflows/pages_dashboard.yml` publishes `dashboard/` to Pages + (`https://pyautolabs.github.io/autofit_profiling/`) and dispatches `pulse-refresh` to + PyAutoPulse (payload repo / sha / `dashboard/catalogue.json`; needs the org secret + `PAT_PYAUTOLABS`, warns and skips without it). `dashboard/state.json` is this + project's own badge feed; the Brain cockpit reads the PyAutoPulse organ feed. + +Every catalogue record is declared `unreviewed`: the producer measures, the Brain's +profiling conductor judges. ## RAL @@ -53,8 +78,9 @@ detail in [`hpc/README.md`](hpc/README.md). ## Testing The PR gate is `.github/workflows/lint.yml` on Python 3.12: `ruff check .`, -`ruff format --check .` and a `lychee` link check over `README.md` and `AGENTS.md`. It -installs no PyAuto library and runs no script. +`ruff format --check .`, `pytest -q scripts/misc/test`, `build_catalogue.py --check +--validate-with _pulse` (PyAutoPulse checked out beside) and a `lychee` link check over +`README.md` and `AGENTS.md`. It installs no PyAuto library and runs no measurement cell. ## Related Repos diff --git a/README.md b/README.md index 1eb2db4..e352ebd 100644 --- a/README.md +++ b/README.md @@ -9,10 +9,12 @@ It is the sibling of [autofit_inference](https://github.com/PyAutoLabs/autofit_i (PyAutoFit search benchmarking) and the PyAutoFit counterpart of [autolens_profiling](https://github.com/PyAutoLabs/autolens_profiling). -**Status: skeleton — no results yet.** The repo was born 2026-10-07 as phase B1 of the -[search-extensibility epic](https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/research/autofit/search_extensibility_epic.md). The breakdown exporter and first results arrive in -B4a, when the repo starts reporting to -[PyAutoPulse](https://github.com/PyAutoLabs/PyAutoPulse); the EP baseline port follows in -B4b. +**Status:** the first cell — where the time in one Nautilus `search.fit` goes on a fast +1-D Gaussian likelihood — landed in B4a of the +[search-extensibility epic](https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/research/autofit/search_extensibility_epic.md), +with its ranked table in [`wiki/campaigns/search_fit_bottlenecks.md`](wiki/campaigns/search_fit_bottlenecks.md). +The repo reports to [PyAutoPulse](https://github.com/PyAutoLabs/PyAutoPulse) as instance +`fit` (`dashboard/catalogue.json`, `profiling-summary@2`); its own board is published to +GitHub Pages from `dashboard/`. The EP baseline port follows in B4b. Agents and contributors: start with [`AGENTS.md`](AGENTS.md). diff --git a/config/general.yaml b/config/general.yaml new file mode 100644 index 0000000..44e5096 --- /dev/null +++ b/config/general.yaml @@ -0,0 +1,4 @@ +# PyAutoConf overrides for runs launched from this repo. Output keeps the library +# defaults on purpose: a breakdown cell measures what a user's search.fit pays. +version: + workspace_version_check: false # this repo tracks PyAutoFit main, not a release tag; the mismatch warning is expected noise. diff --git a/dashboard/catalogue.json b/dashboard/catalogue.json new file mode 100644 index 0000000..1a090b0 --- /dev/null +++ b/dashboard/catalogue.json @@ -0,0 +1,1447 @@ +{ + "schema": "profiling-summary", + "version": 2, + "project": "autofit_profiling", + "scope": "search-fit-breakdown", + "generated_at": "2026-10-14T09:19:45Z", + "evidence_updated_at": "2026-10-14T09:19:45Z", + "valid_until": null, + "producer_revision": null, + "comparison_policy": { + "id": "no-temporal-comparisons" + }, + "coverage": { + "expected": { + "cells": 18 + }, + "observed": { + "setups": 2, + "records": 17, + "selected": 17 + }, + "excluded": [] + }, + "setups": [ + { + "id": "gaussian_x1/nautilus/search_fit", + "label": "af.Nautilus search.fit (PyAutoFit, checkpointing and output on)", + "dataset": "gaussian_x1", + "model": "gaussian", + "instrument": null, + "unknowns": { + "instrument": "toy 1-D dataset; no instrument" + }, + "configuration_id": "gaussian_x1/nautilus/search_fit@1", + "configuration": { + "data_pixels": { + "value": 100, + "unit": "count" + }, + "free_parameters": { + "value": 3, + "unit": "count" + }, + "likelihood": { + "value": "numpy, no JAX", + "unit": "name" + }, + "n_live": { + "value": 200, + "unit": "count" + }, + "n_networks": { + "value": 4, + "unit": "count" + }, + "number_of_cores": { + "value": 1, + "unit": "count" + }, + "output_config": { + "value": "config/general.yaml (library output defaults; version check off)", + "unit": "name" + }, + "search": { + "value": "Nautilus", + "unit": "name" + }, + "seed": { + "value": 1, + "unit": "dimensionless" + }, + "variant": { + "value": "search_fit", + "unit": "name" + } + } + }, + { + "id": "gaussian_x1/nautilus/bare_sampler", + "label": "bare nautilus.Sampler (the findings page's measurement; no PyAutoFit, no checkpoint)", + "dataset": "gaussian_x1", + "model": "gaussian", + "instrument": null, + "unknowns": { + "instrument": "toy 1-D dataset; no instrument" + }, + "configuration_id": "gaussian_x1/nautilus/bare_sampler@1", + "configuration": { + "data_pixels": { + "value": 100, + "unit": "count" + }, + "free_parameters": { + "value": 3, + "unit": "count" + }, + "likelihood": { + "value": "numpy, no JAX", + "unit": "name" + }, + "n_live": { + "value": 200, + "unit": "count" + }, + "n_networks": { + "value": 4, + "unit": "count" + }, + "number_of_cores": { + "value": 1, + "unit": "count" + }, + "output_config": { + "value": "config/general.yaml (library output defaults; version check off)", + "unit": "name" + }, + "search": { + "value": "Nautilus", + "unit": "name" + }, + "seed": { + "value": 1, + "unit": "dimensionless" + }, + "variant": { + "value": "bare_sampler", + "unit": "name" + } + } + } + ], + "records": [ + { + "id": "gaussian_x1/nautilus/search_fit/runtime/wall", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "runtime", + "metric": "wall", + "unit": "s", + "measurement": { + "wall": 28.41256609999982 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/wall", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/wall_s" + } + }, + { + "id": "gaussian_x1/nautilus/search_fit/breakdown/nn_training", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "nn_training", + "unit": "s", + "measurement": { + "nn_training": 11.872448799996164 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/components_s/nn_training" + } + }, + { + "id": "gaussian_x1/nautilus/search_fit/breakdown/bound_geometry", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "bound_geometry", + "unit": "s", + "measurement": { + "bound_geometry": 5.604948700006389 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/components_s/bound_geometry" + } + }, + { + "id": "gaussian_x1/nautilus/search_fit/breakdown/proposal_sampling", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "proposal_sampling", + "unit": "s", + "measurement": { + "proposal_sampling": 0.21894529999826773 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/components_s/proposal_sampling" + } + }, + { + "id": "gaussian_x1/nautilus/search_fit/breakdown/likelihood", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "likelihood", + "unit": "s", + "measurement": { + "likelihood": 0.3219049000008454 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/components_s/likelihood" + } + }, + { + "id": "gaussian_x1/nautilus/search_fit/breakdown/checkpoint_io", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "checkpoint_io", + "unit": "s", + "measurement": { + "checkpoint_io": 7.0924251000005825 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/components_s/checkpoint_io" + } + }, + { + "id": "gaussian_x1/nautilus/search_fit/breakdown/sampler_bookkeeping", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "sampler_bookkeeping", + "unit": "s", + "measurement": { + "sampler_bookkeeping": 0.667890200003967 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/components_s/sampler_bookkeeping" + } + }, + { + "id": "gaussian_x1/nautilus/search_fit/breakdown/samples_output", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "samples_output", + "unit": "s", + "measurement": { + "samples_output": 0.48924609999903623 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/components_s/samples_output" + } + }, + { + "id": "gaussian_x1/nautilus/search_fit/breakdown/visualisation", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "visualisation", + "unit": "s", + "measurement": { + "visualisation": 1.1609661000002234 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/components_s/visualisation" + } + }, + { + "id": "gaussian_x1/nautilus/search_fit/breakdown/other", + "setup_id": "gaussian_x1/nautilus/search_fit", + "run_id": "gaussian_x1/nautilus/search_fit@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "other", + "unit": "s", + "measurement": { + "other": 0.29422839999915595 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "search-fit-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown.json", + "fragment": "/components_s/other" + } + }, + { + "id": "gaussian_x1/nautilus/bare_sampler/runtime/wall", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "run_id": "gaussian_x1/nautilus/bare_sampler@2026-10-14T09:19:45Z", + "axis": "runtime", + "metric": "wall", + "unit": "s", + "measurement": { + "wall": 22.533835900000668 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "bare-sampler-exclusive-component-wall/wall", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown_bare_sampler.json", + "fragment": "/wall_s" + } + }, + { + "id": "gaussian_x1/nautilus/bare_sampler/breakdown/nn_training", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "run_id": "gaussian_x1/nautilus/bare_sampler@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "nn_training", + "unit": "s", + "measurement": { + "nn_training": 12.11267449999832 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "bare-sampler-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown_bare_sampler.json", + "fragment": "/components_s/nn_training" + } + }, + { + "id": "gaussian_x1/nautilus/bare_sampler/breakdown/bound_geometry", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "run_id": "gaussian_x1/nautilus/bare_sampler@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "bound_geometry", + "unit": "s", + "measurement": { + "bound_geometry": 5.800184900003842 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "bare-sampler-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown_bare_sampler.json", + "fragment": "/components_s/bound_geometry" + } + }, + { + "id": "gaussian_x1/nautilus/bare_sampler/breakdown/proposal_sampling", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "run_id": "gaussian_x1/nautilus/bare_sampler@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "proposal_sampling", + "unit": "s", + "measurement": { + "proposal_sampling": 1.3069437999929505 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "bare-sampler-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown_bare_sampler.json", + "fragment": "/components_s/proposal_sampling" + } + }, + { + "id": "gaussian_x1/nautilus/bare_sampler/breakdown/likelihood", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "run_id": "gaussian_x1/nautilus/bare_sampler@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "likelihood", + "unit": "s", + "measurement": { + "likelihood": 2.658846300001642 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "bare-sampler-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown_bare_sampler.json", + "fragment": "/components_s/likelihood" + } + }, + { + "id": "gaussian_x1/nautilus/bare_sampler/breakdown/sampler_bookkeeping", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "run_id": "gaussian_x1/nautilus/bare_sampler@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "sampler_bookkeeping", + "unit": "s", + "measurement": { + "sampler_bookkeeping": 0.6964637999963088 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "bare-sampler-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown_bare_sampler.json", + "fragment": "/components_s/sampler_bookkeeping" + } + }, + { + "id": "gaussian_x1/nautilus/bare_sampler/breakdown/other", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "run_id": "gaussian_x1/nautilus/bare_sampler@2026-10-14T09:19:45Z", + "axis": "breakdown", + "metric": "other", + "unit": "s", + "measurement": { + "other": 3.700000888784416e-06 + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "method": { + "id": "bare-sampler-exclusive-component-wall/components", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat" + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "has_provenance": true, + "qualified": true + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage" + }, + "evidence": { + "path": "results/gaussian_x1/nautilus/breakdown_bare_sampler.json", + "fragment": "/components_s/other" + } + } + ], + "selections": [ + { + "id": "select/gaussian_x1/nautilus/search_fit/runtime/wall", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "runtime", + "metric": "wall", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/wall", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/runtime/wall", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/search_fit/breakdown/nn_training", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "breakdown", + "metric": "nn_training", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/breakdown/nn_training", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/search_fit/breakdown/bound_geometry", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "breakdown", + "metric": "bound_geometry", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/breakdown/bound_geometry", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/search_fit/breakdown/proposal_sampling", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "breakdown", + "metric": "proposal_sampling", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/breakdown/proposal_sampling", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/search_fit/breakdown/likelihood", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "breakdown", + "metric": "likelihood", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/breakdown/likelihood", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/search_fit/breakdown/checkpoint_io", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "breakdown", + "metric": "checkpoint_io", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/breakdown/checkpoint_io", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/search_fit/breakdown/sampler_bookkeeping", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "breakdown", + "metric": "sampler_bookkeeping", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/breakdown/sampler_bookkeeping", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/search_fit/breakdown/samples_output", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "breakdown", + "metric": "samples_output", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/breakdown/samples_output", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/search_fit/breakdown/visualisation", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "breakdown", + "metric": "visualisation", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/breakdown/visualisation", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/search_fit/breakdown/other", + "setup_id": "gaussian_x1/nautilus/search_fit", + "axis": "breakdown", + "metric": "other", + "unit": "s", + "method_id": "search-fit-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/search_fit/breakdown/other", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/bare_sampler/runtime/wall", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "axis": "runtime", + "metric": "wall", + "unit": "s", + "method_id": "bare-sampler-exclusive-component-wall/wall", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/bare_sampler/runtime/wall", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/bare_sampler/breakdown/nn_training", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "axis": "breakdown", + "metric": "nn_training", + "unit": "s", + "method_id": "bare-sampler-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/bare_sampler/breakdown/nn_training", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/bare_sampler/breakdown/bound_geometry", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "axis": "breakdown", + "metric": "bound_geometry", + "unit": "s", + "method_id": "bare-sampler-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/bare_sampler/breakdown/bound_geometry", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/bare_sampler/breakdown/proposal_sampling", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "axis": "breakdown", + "metric": "proposal_sampling", + "unit": "s", + "method_id": "bare-sampler-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/bare_sampler/breakdown/proposal_sampling", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/bare_sampler/breakdown/likelihood", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "axis": "breakdown", + "metric": "likelihood", + "unit": "s", + "method_id": "bare-sampler-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/bare_sampler/breakdown/likelihood", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/bare_sampler/breakdown/checkpoint_io", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "axis": "breakdown", + "metric": "checkpoint_io", + "unit": "s", + "method_id": "bare-sampler-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "inapplicable", + "record_id": null, + "reason": "the bare sampler is run without a checkpoint file, so it writes none" + }, + { + "id": "select/gaussian_x1/nautilus/bare_sampler/breakdown/sampler_bookkeeping", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "axis": "breakdown", + "metric": "sampler_bookkeeping", + "unit": "s", + "method_id": "bare-sampler-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/bare_sampler/breakdown/sampler_bookkeeping", + "reason": "the cell's committed median row" + }, + { + "id": "select/gaussian_x1/nautilus/bare_sampler/breakdown/other", + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "axis": "breakdown", + "metric": "other", + "unit": "s", + "method_id": "bare-sampler-exclusive-component-wall/components", + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d", + "nautilus-sampler": "1.0.5", + "numpy": "2.2.6", + "python": "3.12.10", + "scikit-learn": "1.5.1", + "scipy": "1.17.1" + } + }, + "host": "DESKTOP-H143S82", + "status": "unreviewed", + "record_id": "gaussian_x1/nautilus/bare_sampler/breakdown/other", + "reason": "the cell's committed median row" + } + ], + "hazards": [], + "recommendations": [], + "comparisons": [], + "limitations": [ + "Laptop CPU only (single host, WSL2); wall times vary up to \u00b140 % between sessions, so shares are the comparable quantity, not seconds.", + "Component times are exclusive monkey-patched wall-clock intervals; 'other' is wall minus the instrumented total, not a measured phase.", + "Every record is unreviewed: the producer declares measurements, the Brain's profiling conductor judges them; nothing here is a verdict.", + "No temporal comparisons yet: one baseline per setup, no drift policy." + ] +} diff --git a/dashboard/index.html b/dashboard/index.html new file mode 100644 index 0000000..a3b1377 --- /dev/null +++ b/dashboard/index.html @@ -0,0 +1,48 @@ + + + +autofit_profiling + +

autofit_profiling

+

Where the time in one PyAutoFit search.fit goes. Producer page for the PyAutoPulse instance fit (catalogue.json, profiling-summary@2). Measurements are declared, not judged.

+

gaussian_x1/nautilus/search_fit

+

af.Nautilus search.fit (PyAutoFit, checkpointing and output on) · wall 28.41 s (median of 3) · 7501 likelihood calls · cpu/numpy · PyAutoFit 2026.10.12.1 · measured 2026-10-14T09:19:45Z · result row

+ + + + + + + + + + +
componentsecondsshare of wall
nn_training11.8743.4 %
checkpoint_io7.0924.5 %
bound_geometry5.6020.4 %
visualisation1.164.1 %
sampler_bookkeeping0.672.3 %
samples_output0.491.7 %
likelihood0.321.1 %
other0.291.0 %
proposal_sampling0.220.8 %
+

Findings-page shares (sampler_internal_shares, information only): nn_training 51.7 % (Δ -4.7 pp), bound_geometry 25.2 % (Δ -3.4 pp), likelihood 14.3 % (Δ -13.1 pp) — outside ±10 pp.

+

gaussian_x1/nautilus/bare_sampler

+

bare nautilus.Sampler (the findings page's measurement; no PyAutoFit, no checkpoint) · wall 22.53 s (median of 3) · 16400 likelihood calls · cpu/numpy · PyAutoFit 2026.10.12.1 · measured 2026-10-14T09:19:45Z · result row

+ + + + + + + + +
componentsecondsshare of wall
nn_training12.1153.8 %
bound_geometry5.8025.3 %
likelihood2.6611.6 %
proposal_sampling1.315.7 %
sampler_bookkeeping0.703.1 %
other0.000.0 %
checkpoint_io0.000.0 %
+

Findings-page shares (shares_of_wall, reproduction gate): nn_training 51.7 % (Δ +2.1 pp), bound_geometry 25.2 % (Δ +0.1 pp), likelihood 14.3 % (Δ -2.7 pp) — within ±10 pp.

+

Limitations

    +
  • Laptop CPU only (single host, WSL2); wall times vary up to ±40 % between sessions, so shares are the comparable quantity, not seconds.
    • +
  • Component times are exclusive monkey-patched wall-clock intervals; 'other' is wall minus the instrumented total, not a measured phase.
    • +
  • Every record is unreviewed: the producer declares measurements, the Brain's profiling conductor judges them; nothing here is a verdict.
    • +
  • No temporal comparisons yet: one baseline per setup, no drift policy.
    • +
+

Rendered from 2 result rows by build_catalogue.py; evidence updated 2026-10-14T09:19:45Z.

+ diff --git a/dashboard/state.json b/dashboard/state.json new file mode 100644 index 0000000..fb739ee --- /dev/null +++ b/dashboard/state.json @@ -0,0 +1,10 @@ +{ + "schema_version": 1, + "organ": "autofit_profiling", + "repo": "autofit_profiling", + "status": "green", + "headline": "2 setups \u00b7 17 records (search.fit breakdown); no drift policy yet \u2014 baseline only, not a verdict", + "updated": "2026-10-14T09:19:45Z", + "pages_url": "https://pyautolabs.github.io/autofit_profiling/", + "items": [] +} diff --git a/results/gaussian_x1/nautilus/breakdown.json b/results/gaussian_x1/nautilus/breakdown.json new file mode 100644 index 0000000..d7a4d3b --- /dev/null +++ b/results/gaussian_x1/nautilus/breakdown.json @@ -0,0 +1,192 @@ +{ + "schema": "autofit-profiling-breakdown", + "version": 1, + "cell": "gaussian_x1/nautilus/search_fit_breakdown", + "script": "scripts/gaussian_x1/nautilus/search_fit_breakdown.py", + "axis": "breakdown", + "unit": "s", + "setup": { + "id": "gaussian_x1/nautilus/search_fit", + "label": "af.Nautilus search.fit (PyAutoFit, checkpointing and output on)", + "dataset": "gaussian_x1", + "dataset_description": "1-D Gaussian, 100 pixels, noise 0.01, numpy seed 1 (findings page scenario 1)", + "model": "Gaussian (centre, normalization, sigma), 3 free parameters, uniform priors", + "search": "Nautilus", + "search_settings": { + "n_live": 200, + "n_networks": 4, + "seed": 1, + "number_of_cores": 1 + }, + "likelihood": "numpy, no JAX", + "config": "config/general.yaml (library output defaults; version check off)" + }, + "method": { + "id": "search-fit-exclusive-component-wall", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat", + "timer": "time.perf_counter, exclusive per component (scripts/misc/profiling/exclusive_timer.py)" + }, + "variant": "search_fit", + "wall_s": 28.41256609999982, + "components_s": { + "nn_training": 11.872448799996164, + "bound_geometry": 5.604948700006389, + "proposal_sampling": 0.21894529999826773, + "likelihood": 0.3219049000008454, + "checkpoint_io": 7.0924251000005825, + "sampler_bookkeeping": 0.667890200003967, + "samples_output": 0.48924609999903623, + "visualisation": 1.1609661000002234, + "other": 0.29422839999915595 + }, + "shares_of_wall": { + "nn_training": 0.43368608465957864, + "bound_geometry": 0.20430233455886074, + "proposal_sampling": 0.0075734870583560356, + "likelihood": 0.011329666559080962, + "checkpoint_io": 0.2453324629833977, + "sampler_bookkeeping": 0.023102837951638554, + "samples_output": 0.016923400533024366, + "visualisation": 0.0408610083268836, + "other": 0.010355572916701734 + }, + "sampler_internal_shares": { + "nn_training": 0.47008187873136503, + "bound_geometry": 0.21801666072550563, + "proposal_sampling": 0.008081877096881192, + "likelihood": 0.01219978059220303, + "checkpoint_io": 0.26180104335417803, + "sampler_bookkeeping": 0.02465367610396828 + }, + "likelihood_function_calls": 7501, + "deterministic_calls": true, + "reference": { + "source": "autofit_workspace_developer/searches_minimal/nautilus_bottleneck_findings.md \u00a71 (nautilus 1.0.5, n_live=200, seed=1, bare Sampler, laptop CPU)", + "shares": { + "nn_training": 0.517, + "bound_geometry": 0.252, + "likelihood": 0.143 + }, + "compared": "sampler_internal_shares", + "tolerance_abs": 0.1, + "deltas": { + "nn_training": -0.04691812126863498, + "bound_geometry": -0.03398333927449437, + "likelihood": -0.13080021940779696 + }, + "within_tolerance": false, + "gates_reproduction": false + }, + "repeats": [ + { + "wall_s": 25.84897439999986, + "components_s": { + "nn_training": 11.210340500001621, + "bound_geometry": 5.358583000000181, + "proposal_sampling": 0.16643590000148833, + "likelihood": 0.29093590000320546, + "checkpoint_io": 6.236064299997452, + "sampler_bookkeeping": 0.5852741999960926, + "samples_output": 0.4220662000006996, + "visualisation": 1.3870600000000195, + "other": 0.19221439999910217 + }, + "sampler_run_s": 23.84763380000004, + "likelihood_function_calls": 7501, + "component_calls": { + "bound_geometry": 26, + "checkpoint_io": 103, + "likelihood": 75, + "nn_training": 26, + "proposal_sampling": 75, + "sampler_bookkeeping": 1, + "samples_output": 5, + "visualisation": 1 + }, + "max_log_likelihood": -38.1725465805512, + "log_evidence": -57.51149618752003 + }, + { + "wall_s": 28.41256609999982, + "components_s": { + "nn_training": 11.872448799996164, + "bound_geometry": 5.604948700006389, + "proposal_sampling": 0.2239162000023498, + "likelihood": 0.3219049000008454, + "checkpoint_io": 7.7090770000031625, + "sampler_bookkeeping": 0.6715516999911415, + "samples_output": 0.5535243000003902, + "visualisation": 1.1609661000002234, + "other": 0.29422839999915595 + }, + "sampler_run_s": 26.403847300000052, + "likelihood_function_calls": 7501, + "component_calls": { + "bound_geometry": 26, + "checkpoint_io": 103, + "likelihood": 75, + "nn_training": 26, + "proposal_sampling": 75, + "sampler_bookkeeping": 1, + "samples_output": 5, + "visualisation": 1 + }, + "max_log_likelihood": -38.1725465805512, + "log_evidence": -57.51149618752003 + }, + { + "wall_s": 28.909443999999894, + "components_s": { + "nn_training": 12.869246400003249, + "bound_geometry": 5.906266899998627, + "proposal_sampling": 0.21894529999826773, + "likelihood": 0.33612309999534773, + "checkpoint_io": 7.0924251000005825, + "sampler_bookkeeping": 0.667890200003967, + "samples_output": 0.48924609999903623, + "visualisation": 0.9943709999997736, + "other": 0.3349299000010433 + }, + "sampler_run_s": 27.09089700000004, + "likelihood_function_calls": 7501, + "component_calls": { + "bound_geometry": 26, + "checkpoint_io": 103, + "likelihood": 75, + "nn_training": 26, + "proposal_sampling": 75, + "sampler_bookkeeping": 1, + "samples_output": 5, + "visualisation": 1 + }, + "max_log_likelihood": -38.1725465805512, + "log_evidence": -57.51149618752003 + } + ], + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "nautilus-sampler": "1.0.5", + "scikit-learn": "1.5.1", + "numpy": "2.2.6", + "scipy": "1.17.1", + "python": "3.12.10", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d" + } + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "platform": "Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.35", + "cpu_count": 8, + "pyautofit_checkout_dirty": false + } +} diff --git a/results/gaussian_x1/nautilus/breakdown_bare_sampler.json b/results/gaussian_x1/nautilus/breakdown_bare_sampler.json new file mode 100644 index 0000000..04eb5ce --- /dev/null +++ b/results/gaussian_x1/nautilus/breakdown_bare_sampler.json @@ -0,0 +1,173 @@ +{ + "schema": "autofit-profiling-breakdown", + "version": 1, + "cell": "gaussian_x1/nautilus/search_fit_breakdown", + "script": "scripts/gaussian_x1/nautilus/search_fit_breakdown.py", + "axis": "breakdown", + "unit": "s", + "setup": { + "id": "gaussian_x1/nautilus/bare_sampler", + "label": "bare nautilus.Sampler (the findings page's measurement; no PyAutoFit, no checkpoint)", + "dataset": "gaussian_x1", + "dataset_description": "1-D Gaussian, 100 pixels, noise 0.01, numpy seed 1 (findings page scenario 1)", + "model": "Gaussian (centre, normalization, sigma), 3 free parameters, uniform priors", + "search": "Nautilus", + "search_settings": { + "n_live": 200, + "n_networks": 4, + "seed": 1, + "number_of_cores": 1 + }, + "likelihood": "numpy, no JAX", + "config": "config/general.yaml (library output defaults; version check off)" + }, + "method": { + "id": "bare-sampler-exclusive-component-wall", + "statistic": "median", + "repetitions": 3, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat", + "timer": "time.perf_counter, exclusive per component (scripts/misc/profiling/exclusive_timer.py)" + }, + "variant": "bare_sampler", + "wall_s": 22.533835900000668, + "components_s": { + "nn_training": 12.11267449999832, + "bound_geometry": 5.800184900003842, + "proposal_sampling": 1.3069437999929505, + "likelihood": 2.658846300001642, + "checkpoint_io": 0.0, + "sampler_bookkeeping": 0.6964637999963088, + "other": 3.700000888784416e-06 + }, + "shares_of_wall": { + "nn_training": 0.537532737601855, + "bound_geometry": 0.2529185867842278, + "proposal_sampling": 0.05665423573593358, + "likelihood": 0.11632621736069092, + "checkpoint_io": 0.0, + "sampler_bookkeeping": 0.03144706169292557, + "other": 1.6419756073506802e-07 + }, + "sampler_internal_shares": { + "nn_training": 0.537532737601855, + "bound_geometry": 0.2529185867842278, + "proposal_sampling": 0.05665423573593358, + "likelihood": 0.11632621736069092, + "checkpoint_io": 0.0, + "sampler_bookkeeping": 0.03144706169292557 + }, + "likelihood_function_calls": 16400, + "deterministic_calls": true, + "reference": { + "source": "autofit_workspace_developer/searches_minimal/nautilus_bottleneck_findings.md \u00a71 (nautilus 1.0.5, n_live=200, seed=1, bare Sampler, laptop CPU)", + "shares": { + "nn_training": 0.517, + "bound_geometry": 0.252, + "likelihood": 0.143 + }, + "compared": "shares_of_wall", + "tolerance_abs": 0.1, + "deltas": { + "nn_training": 0.020532737601855033, + "bound_geometry": 0.0009185867842277928, + "likelihood": -0.02667378263930907 + }, + "within_tolerance": true, + "gates_reproduction": true + }, + "repeats": [ + { + "wall_s": 23.068774700000176, + "components_s": { + "nn_training": 12.542782399998941, + "bound_geometry": 5.800184900003842, + "proposal_sampling": 1.3069437999929505, + "likelihood": 2.683503299997028, + "checkpoint_io": 0.0, + "sampler_bookkeeping": 0.735355900007562, + "other": 4.399999852466863e-06 + }, + "sampler_run_s": 23.068774700000176, + "likelihood_function_calls": 16400, + "component_calls": { + "bound_geometry": 26, + "likelihood": 164, + "nn_training": 26, + "proposal_sampling": 164, + "sampler_bookkeeping": 1 + }, + "max_log_likelihood": -38.1725465805512, + "log_evidence": -57.45518380555804 + }, + { + "wall_s": 22.533835900000668, + "components_s": { + "nn_training": 12.11267449999832, + "bound_geometry": 5.8514490000025035, + "proposal_sampling": 1.2527613999991445, + "likelihood": 2.620483500003502, + "checkpoint_io": 0.0, + "sampler_bookkeeping": 0.6964637999963088, + "other": 3.700000888784416e-06 + }, + "sampler_run_s": 22.533835900000668, + "likelihood_function_calls": 16400, + "component_calls": { + "bound_geometry": 26, + "likelihood": 164, + "nn_training": 26, + "proposal_sampling": 164, + "sampler_bookkeeping": 1 + }, + "max_log_likelihood": -38.1725465805512, + "log_evidence": -57.45518380555804 + }, + { + "wall_s": 21.812791500000458, + "components_s": { + "nn_training": 11.624765800002024, + "bound_geometry": 5.516860399999132, + "proposal_sampling": 1.3263674000018, + "likelihood": 2.658846300001642, + "checkpoint_io": 0.0, + "sampler_bookkeeping": 0.6859481999954369, + "other": 3.4000004234258085e-06 + }, + "sampler_run_s": 21.812791500000458, + "likelihood_function_calls": 16400, + "component_calls": { + "bound_geometry": 26, + "likelihood": 164, + "nn_training": 26, + "proposal_sampling": 164, + "sampler_bookkeeping": 1 + }, + "max_log_likelihood": -38.1725465805512, + "log_evidence": -57.45518380555804 + } + ], + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": { + "PyAutoFit": "911fe05e1c6b570df51789f0ab5b2cbf54c1f43a", + "nautilus-sampler": "1.0.5", + "scikit-learn": "1.5.1", + "numpy": "2.2.6", + "scipy": "1.17.1", + "python": "3.12.10", + "PyAutoNerves": "7f3055b7f3d1967f372bdb41ee732135bca0e68d" + } + }, + "provenance": { + "host": "DESKTOP-H143S82", + "measured_at": "2026-10-14T09:19:45Z", + "platform": "Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.35", + "cpu_count": 8, + "pyautofit_checkout_dirty": false + } +} diff --git a/scripts/gaussian_x1/nautilus/search_fit_breakdown.py b/scripts/gaussian_x1/nautilus/search_fit_breakdown.py new file mode 100644 index 0000000..4d4ca7f --- /dev/null +++ b/scripts/gaussian_x1/nautilus/search_fit_breakdown.py @@ -0,0 +1,490 @@ +""" +gaussian_x1 · Nautilus · ``search.fit`` breakdown +================================================= + +Where the wall time of one PyAutoFit ``search.fit`` goes, on a likelihood fast enough +that the overhead has nowhere to hide. Axis: ``breakdown``. + +Ported from ``autofit_workspace_developer/searches_minimal/nautilus_profile.py`` +(scenario 1, the 3-parameter Gaussian fit; pre-delete commit recorded in the +autofit_workspace_developer retirement PR). The original timed a bare +``nautilus.Sampler``; this cell times the full ``af.Nautilus(...).fit(model, analysis)`` +so the PyAutoFit side of the fit (samples output, visualisation, checkpointing, setup +and result construction) is measured too. The findings it reproduces live in +``wiki/campaigns/search_fit_breakdown.md``. + +Components (exclusive wall time — a nested call is charged to the innermost one, see +``scripts/misc/profiling/exclusive_timer.py``): + + nn_training nautilus NeuralNetworkEmulator.train (sklearn MLP ensembles) + bound_geometry NautilusBound.compute minus NN training (ellipsoid unions, volume MC) + proposal_sampling Sampler.sample_shell + likelihood Sampler.evaluate_likelihood (prior transform + PyAutoFit Fitness) + checkpoint_io Sampler.write / write_shell_update (nautilus HDF5 checkpoint) + sampler_bookkeeping the rest of Sampler.run (shell updates, status, bookkeeping) + samples_output PyAutoFit pre/post-fit output, samples, latents, summary + visualisation PyAutoFit SearchUpdater.visualize + other wall minus all of the above (setup, result construction) + +Two variants, both run by default: + + search_fit the epic-1 measurement: one ``af.Nautilus(...).fit`` → ``breakdown.json`` + bare_sampler the findings page's own measurement re-run (bare ``nautilus.Sampler``, + no PyAutoFit, no checkpoint file) → ``breakdown_bare_sampler.json``. + Its shares of wall must reproduce the findings page's NN training, + bound geometry and likelihood shares within ``TOLERANCE``; the exit + status is non-zero when they do not. + +Deterministic in its sampling (``seed=1``: every repeat makes the same likelihood calls); +wall times vary between sessions (the findings page measured ±40 % on this laptop), +so each variant reports the median over ``--repeats`` fresh fits. + +Run (laptop CPU; no RAL): + + source activate.sh + python scripts/gaussian_x1/nautilus/search_fit_breakdown.py [--repeats 3] + +Writes ``results/gaussian_x1/nautilus/breakdown*.json``; then rebuild the catalogue with +``python scripts/misc/tooling/build_catalogue.py``. +""" + +from __future__ import annotations + +import argparse +import json +import os +import platform +import shutil +import statistics +import subprocess +import sys +import time +from datetime import UTC, datetime +from pathlib import Path + + +def _repo_root() -> Path: + here = Path(__file__).resolve() + for parent in here.parents: + if (parent / "ruff.toml").is_file(): + return parent + raise RuntimeError("ruff.toml root sentinel not found above this script") + + +ROOT = _repo_root() +sys.path[:0] = [str(ROOT), str(ROOT / "scripts" / "misc")] + +import autofit as af +import numpy as np +from profiling.exclusive_timer import ExclusiveTimer, shares, within_tolerance + +CELL = "gaussian_x1/nautilus/search_fit_breakdown" +RESULT = ROOT / "results" / "gaussian_x1" / "nautilus" / "breakdown.json" +OUTPUT = ROOT / "output" / "gaussian_x1" / "nautilus" / "search_fit_breakdown" + +SEARCH_SETTINGS = {"n_live": 200, "n_networks": 4, "seed": 1, "number_of_cores": 1} + +SAMPLER_COMPONENTS = ( + "nn_training", + "bound_geometry", + "proposal_sampling", + "likelihood", + "checkpoint_io", + "sampler_bookkeeping", +) +FIT_COMPONENTS = ("samples_output", "visualisation") +COMPONENTS = SAMPLER_COMPONENTS + FIT_COMPONENTS + +# The anatomy this cell re-homes: nautilus_bottleneck_findings.md §1, 3-parameter +# Gaussian fit column, shares of the bare sampler's wall. Compared with this cell's +# sampler-internal shares (component / Sampler.run time). +FINDINGS_SHARES = {"nn_training": 0.517, "bound_geometry": 0.252, "likelihood": 0.143} +TOLERANCE = 0.10 # absolute, per share (10 percentage points) +FINDINGS_SOURCE = ( + "autofit_workspace_developer/searches_minimal/nautilus_bottleneck_findings.md §1 " + "(nautilus 1.0.5, n_live=200, seed=1, bare Sampler, laptop CPU)" +) + + +# ----------------------------------------------------------------------------- data --- + + +class Gaussian: + def __init__(self, centre=30.0, normalization=1.0, sigma=5.0): + self.centre = centre + self.normalization = normalization + self.sigma = sigma + + def model_data_from(self, xvalues): + return (self.normalization / (self.sigma * np.sqrt(2.0 * np.pi))) * np.exp( + -0.5 * np.square((xvalues - self.centre) / self.sigma) + ) + + +def make_dataset(): + """The findings page's 1-D Gaussian (identical data and seed).""" + np.random.seed(1) + xvalues = np.arange(100) + data = Gaussian(centre=50.0, normalization=25.0, sigma=10.0).model_data_from(xvalues) + noise_map = np.full(data.shape, 0.01) + data = data + np.random.normal(0.0, 0.01, data.shape) + return xvalues, data, noise_map + + +class AnalysisGaussianX1(af.Analysis): + def __init__(self, xvalues, data, noise_map): + super().__init__() + self.xvalues = xvalues + self.data = data + self.noise_map = noise_map + + def log_likelihood_function(self, instance): + model_data = instance.model_data_from(xvalues=self.xvalues) + chi_squared = np.sum(((self.data - model_data) / self.noise_map) ** 2.0) + return float(-0.5 * chi_squared) + + +def make_model(): + model = af.Model(Gaussian) + model.centre = af.UniformPrior(lower_limit=0.0, upper_limit=100.0) + model.normalization = af.UniformPrior(lower_limit=0.0, upper_limit=50.0) + model.sigma = af.UniformPrior(lower_limit=0.0, upper_limit=50.0) + return model + + +# ------------------------------------------------------------------ instrumentation --- + + +def instrument(timer: ExclusiveTimer): + """Monkey-patch the timed methods; returns a callable that restores them.""" + from autofit.non_linear.search.abstract_search import NonLinearSearch + from autofit.non_linear.search.updater import SearchUpdater + from nautilus import Sampler + from nautilus.bounds import NautilusBound + from nautilus.neural import NeuralNetworkEmulator + + patches = [] + + def patch(owner, name, component, classmethod_=False): + original = owner.__dict__[name] + func = original.__func__ if classmethod_ else original + wrapped = timer.wrap(component, func) + setattr(owner, name, classmethod(wrapped) if classmethod_ else wrapped) + patches.append((owner, name, original)) + + patch(NeuralNetworkEmulator, "train", "nn_training", classmethod_=True) + patch(NautilusBound, "compute", "bound_geometry", classmethod_=True) + patch(Sampler, "sample_shell", "proposal_sampling") + patch(Sampler, "evaluate_likelihood", "likelihood") + patch(Sampler, "write", "checkpoint_io") + patch(Sampler, "write_shell_update", "checkpoint_io") + patch(Sampler, "run", "sampler_bookkeeping") + for name in ("_save_samples", "_compute_latent_samples", "_profile_and_summarize"): + patch(SearchUpdater, name, "samples_output") + patch(SearchUpdater, "visualize", "visualisation") + for name in ("pre_fit_output", "post_fit_output"): + patch(NonLinearSearch, name, "samples_output") + + def restore(): + for owner, name, original in reversed(patches): + setattr(owner, name, original) + + return restore + + +# ----------------------------------------------------------------------- provenance --- + + +def _git(path: Path, *args) -> str | None: + try: + out = subprocess.run( + ["git", "-C", str(path), *args], capture_output=True, text=True, check=True + ) + return out.stdout.strip() or None + except (OSError, subprocess.CalledProcessError): + return None + + +def software(): + import autofit + import nautilus + import scipy + import sklearn + + fit_dir = Path(autofit.__file__).resolve().parents[1] + sw = { + "PyAutoFit": _git(fit_dir, "rev-parse", "HEAD"), + "nautilus-sampler": nautilus.__version__, + "scikit-learn": sklearn.__version__, + "numpy": np.__version__, + "scipy": scipy.__version__, + "python": platform.python_version(), + } + try: + import autonerves + + sw["PyAutoNerves"] = _git( + Path(autonerves.__file__).resolve().parents[1], "rev-parse", "HEAD" + ) + except ImportError: + pass + # A source checkout's autofit.__version__ is stale (it is stamped at release), so the + # library version is the nearest release tag of the checkout that was measured. + tag = _git(fit_dir, "describe", "--tags", "--abbrev=0") + return ( + {k: v for k, v in sw.items() if v}, + tag or autofit.__version__, + _git(fit_dir, "status", "--porcelain"), + ) + + +# --------------------------------------------------------------------------- runner --- + + +def run_once(timer: ExclusiveTimer, xvalues, data, noise_map): + shutil.rmtree(OUTPUT, ignore_errors=True) # a fresh fit, never a resume + model = make_model() + analysis = AnalysisGaussianX1(xvalues=xvalues, data=data, noise_map=noise_map) + search = af.Nautilus( + path_prefix=str(Path("gaussian_x1") / "nautilus"), + name="search_fit_breakdown", + verbose=False, + **SEARCH_SETTINGS, + ) + calls = {"n": 0} + original = analysis.log_likelihood_function + + def counted(instance): + calls["n"] += 1 + return original(instance) + + analysis.log_likelihood_function = counted + + timer.reset() + t0 = time.perf_counter() + result = search.fit(model=model, analysis=analysis) + wall = time.perf_counter() - t0 + + seconds = shares(timer.totals, wall, COMPONENTS) + sampler_run = sum(seconds[c] for c in SAMPLER_COMPONENTS) + samples = result.samples + return { + "wall_s": wall, + "components_s": seconds, + "sampler_run_s": sampler_run, + "likelihood_function_calls": calls["n"], + "component_calls": dict(sorted(timer.counts.items())), + "max_log_likelihood": float(max(samples.log_likelihood_list)), + "log_evidence": float(samples.log_evidence), + } + + +def run_bare_once(timer: ExclusiveTimer, xvalues, data, noise_map): + """The findings page's own measurement: a bare ``nautilus.Sampler`` (no PyAutoFit + search, no checkpoint file), same data, priors, ``n_live`` and seed.""" + from nautilus import Sampler + + model = make_model() + calls = {"n": 0} + + def prior_transform(cube): + return np.array(model.vector_from_unit_vector(cube)) + + def log_likelihood(params): + calls["n"] += 1 + instance = model.instance_from_vector(vector=params) + residual = data - instance.model_data_from(xvalues=xvalues) + return float(-0.5 * np.sum((residual / noise_map) ** 2.0)) + + sampler = Sampler( + prior=prior_transform, + likelihood=log_likelihood, + n_dim=model.prior_count, + n_live=SEARCH_SETTINGS["n_live"], + n_networks=SEARCH_SETTINGS["n_networks"], + seed=SEARCH_SETTINGS["seed"], + ) + timer.reset() + t0 = time.perf_counter() + sampler.run(verbose=False) + wall = time.perf_counter() - t0 + seconds = shares(timer.totals, wall, SAMPLER_COMPONENTS) + _, _, log_l = sampler.posterior() + return { + "wall_s": wall, + "components_s": seconds, + "sampler_run_s": wall, + "likelihood_function_calls": calls["n"], + "component_calls": dict(sorted(timer.counts.items())), + "max_log_likelihood": float(np.max(log_l)), + "log_evidence": float(sampler.log_z), + } + + +def summarise(repeats, components): + keys = (*components, "other") + wall = statistics.median(r["wall_s"] for r in repeats) + comp = {c: statistics.median(r["components_s"][c] for r in repeats) for c in keys} + share = { + c: statistics.median(r["components_s"][c] / r["wall_s"] for r in repeats) for c in keys + } + internal = { + c: statistics.median(r["components_s"][c] / r["sampler_run_s"] for r in repeats) + for c in SAMPLER_COMPONENTS + } + return wall, comp, share, internal + + +VARIANTS = { + "search_fit": { + "result": RESULT, + "setup_id": "gaussian_x1/nautilus/search_fit", + "label": "af.Nautilus search.fit (PyAutoFit, checkpointing and output on)", + "runner": run_once, + "components": COMPONENTS, + "method_id": "search-fit-exclusive-component-wall", + }, + "bare_sampler": { + "result": RESULT.with_name("breakdown_bare_sampler.json"), + "setup_id": "gaussian_x1/nautilus/bare_sampler", + "label": "bare nautilus.Sampler (the findings page's measurement; no PyAutoFit, no checkpoint)", + "runner": run_bare_once, + "components": SAMPLER_COMPONENTS, + "method_id": "bare-sampler-exclusive-component-wall", + }, +} + + +def measure(variant, n_repeats, timer, dataset): + spec = VARIANTS[variant] + repeats = [] + for i in range(n_repeats): + row = spec["runner"](timer, *dataset) + print( + f"{variant} repeat {i + 1}/{n_repeats}: wall {row['wall_s']:.2f} s, " + f"{row['likelihood_function_calls']} likelihood calls" + ) + repeats.append(row) + wall, comp, share, internal = summarise(repeats, spec["components"]) + calls = sorted({r["likelihood_function_calls"] for r in repeats}) + # The findings page reports shares of the bare sampler's wall; the search.fit + # variant is compared on the same sampler-internal basis, for information only. + basis = share if variant == "bare_sampler" else internal + ok, deltas = within_tolerance(basis, FINDINGS_SHARES, TOLERANCE) + return { + "variant": variant, + "label": spec["label"], + "setup_id": spec["setup_id"], + "method_id": spec["method_id"], + "wall_s": wall, + "components_s": comp, + "shares_of_wall": share, + "sampler_internal_shares": internal, + "likelihood_function_calls": calls[0] if len(calls) == 1 else calls, + "deterministic_calls": len(calls) == 1, + "reference": { + "source": FINDINGS_SOURCE, + "shares": FINDINGS_SHARES, + "compared": "shares_of_wall" + if variant == "bare_sampler" + else "sampler_internal_shares", + "tolerance_abs": TOLERANCE, + "deltas": deltas, + "within_tolerance": ok, + "gates_reproduction": variant == "bare_sampler", + }, + "repeats": repeats, + } + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__.strip().splitlines()[0]) + parser.add_argument("--repeats", type=int, default=3) + parser.add_argument("--variant", choices=[*VARIANTS, "all"], default="all") + parser.add_argument("--no-write", action="store_true", help="print only") + args = parser.parse_args(argv) + + af.conf.instance.push(new_path=ROOT / "config", output_path=ROOT / "output") + dataset = make_dataset() + timer = ExclusiveTimer() + variants = list(VARIANTS) if args.variant == "all" else [args.variant] + restore = instrument(timer) + try: + measured = [measure(v, args.repeats, timer, dataset) for v in variants] + finally: + restore() + + sw, library_version, dirty = software() + measured_at = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ") + status = 0 + for m in measured: + row = { + "schema": "autofit-profiling-breakdown", + "version": 1, + "cell": CELL, + "script": f"scripts/{CELL}.py", + "axis": "breakdown", + "unit": "s", + "setup": { + "id": m.pop("setup_id"), + "label": m.pop("label"), + "dataset": "gaussian_x1", + "dataset_description": "1-D Gaussian, 100 pixels, noise 0.01, numpy seed 1 (findings page scenario 1)", + "model": "Gaussian (centre, normalization, sigma), 3 free parameters, uniform priors", + "search": "Nautilus", + "search_settings": SEARCH_SETTINGS, + "likelihood": "numpy, no JAX", + "config": "config/general.yaml (library output defaults; version check off)", + }, + "method": { + "id": m.pop("method_id"), + "statistic": "median", + "repetitions": args.repeats, + "warmup": "none; each repeat is a fresh fit (output removed, never a resume)", + "synchronization": "none (numpy, single process)", + "cache_state": "warm process: imports and first-touch costs fall in the first repeat", + "timer": "time.perf_counter, exclusive per component (scripts/misc/profiling/exclusive_timer.py)", + }, + **m, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": library_version, + "software": sw, + }, + "provenance": { + "host": platform.node(), + "measured_at": measured_at, + "platform": platform.platform(), + "cpu_count": os.cpu_count(), + "pyautofit_checkout_dirty": bool(dirty), + }, + } + print( + json.dumps( + { + k: row[k] + for k in ( + "variant", + "wall_s", + "shares_of_wall", + "sampler_internal_shares", + "reference", + ) + }, + indent=2, + ) + ) + if row["reference"]["gates_reproduction"] and not row["reference"]["within_tolerance"]: + status = 1 + if not args.no_write: + path = VARIANTS[row["variant"]]["result"] + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(row, indent=2) + "\n") + print(f"wrote {path.relative_to(ROOT)}") + return status + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/misc/profiling/__init__.py b/scripts/misc/profiling/__init__.py new file mode 100644 index 0000000..afba725 --- /dev/null +++ b/scripts/misc/profiling/__init__.py @@ -0,0 +1 @@ +"""Shared, stdlib-only helpers for the autofit_profiling cells.""" diff --git a/scripts/misc/profiling/exclusive_timer.py b/scripts/misc/profiling/exclusive_timer.py new file mode 100644 index 0000000..84b39b7 --- /dev/null +++ b/scripts/misc/profiling/exclusive_timer.py @@ -0,0 +1,67 @@ +"""Exclusive (self) wall-time accounting for monkey-patched methods (stdlib only). + +A breakdown cell wraps a handful of library methods. Some of them nest — a Nautilus +bound update (``NautilusBound.compute``) trains the neural-network emulators +(``NeuralNetworkEmulator.train``) inside it, and a PyAutoFit output update may evaluate +likelihoods. Timing each wrapper inclusively would count the inner time twice, so the +timer keeps a call stack and charges every interval to the innermost open component +only. The components' exclusive totals therefore never sum past the wall time, and +``other`` (wall minus the instrumented total) is never negative by construction. +""" + +from __future__ import annotations + +import functools +import time + + +class ExclusiveTimer: + """Charge wall time to the innermost active component.""" + + def __init__(self, clock=time.perf_counter): + self.clock = clock + self.totals: dict[str, float] = {} + self.counts: dict[str, int] = {} + self._stack: list[list] = [] # [component, start, child_time] + + def reset(self): + self.totals.clear() + self.counts.clear() + self._stack.clear() + + def enter(self, component: str): + self._stack.append([component, self.clock(), 0.0]) + + def exit(self): + component, start, child = self._stack.pop() + inclusive = self.clock() - start + self.totals[component] = self.totals.get(component, 0.0) + inclusive - child + self.counts[component] = self.counts.get(component, 0) + 1 + if self._stack: + self._stack[-1][2] += inclusive + + def wrap(self, component: str, func): + """Return ``func`` wrapped so its exclusive time is charged to ``component``.""" + + @functools.wraps(func) + def timed(*args, **kwargs): + self.enter(component) + try: + return func(*args, **kwargs) + finally: + self.exit() + + return timed + + +def shares(totals: dict[str, float], wall: float, order) -> dict[str, float]: + """Exclusive seconds per component in ``order`` plus ``other`` = wall − instrumented.""" + seconds = {name: float(totals.get(name, 0.0)) for name in order} + seconds["other"] = max(wall - sum(seconds.values()), 0.0) + return seconds + + +def within_tolerance(measured: dict[str, float], reference: dict[str, float], tol: float): + """``(ok, deltas)``: each reference share reproduced within ``tol`` (absolute, 0–1).""" + deltas = {k: measured.get(k, 0.0) - v for k, v in reference.items()} + return all(abs(d) <= tol for d in deltas.values()), deltas diff --git a/scripts/misc/test/conftest.py b/scripts/misc/test/conftest.py new file mode 100644 index 0000000..327630c --- /dev/null +++ b/scripts/misc/test/conftest.py @@ -0,0 +1,7 @@ +"""Put scripts/misc and its tooling on sys.path; stdlib + pytest only (no PyAuto install).""" + +import sys +from pathlib import Path + +MISC = Path(__file__).resolve().parents[1] +sys.path[:0] = [str(MISC), str(MISC / "tooling")] diff --git a/scripts/misc/test/test_build_catalogue.py b/scripts/misc/test/test_build_catalogue.py new file mode 100644 index 0000000..7247936 --- /dev/null +++ b/scripts/misc/test/test_build_catalogue.py @@ -0,0 +1,139 @@ +import copy +import json +import shutil +from pathlib import Path + +import build_catalogue as bc +import pytest + +REPO = Path(__file__).resolve().parents[3] +# CI checks Pulse out at ./_pulse; a workspace has it beside this repo. +PULSE = next( + (p for p in (REPO / "_pulse", REPO.parent / "PyAutoPulse") if (p / "pulse").is_dir()), + REPO / "_pulse", +) + + +def _row(variant="search_fit", **components): + components = components or {"nn_training": 6.0, "likelihood": 1.0, "other": 3.0} + return { + "schema": bc.RESULT_SCHEMA, + "version": 1, + "variant": variant, + "axis": "breakdown", + "unit": "s", + "setup": { + "id": f"gaussian_x1/nautilus/{variant}", + "label": variant, + "dataset": "gaussian_x1", + "search": "Nautilus", + "search_settings": {"n_live": 200, "n_networks": 4, "seed": 1, "number_of_cores": 1}, + "likelihood": "numpy, no JAX", + "config": "config/general.yaml", + }, + "method": { + "id": f"{variant}-method", + "statistic": "median", + "repetitions": 3, + "warmup": "none", + "synchronization": "none", + "cache_state": "warm process", + }, + "wall_s": 10.0, + "components_s": components, + "shares_of_wall": {k: v / 10.0 for k, v in components.items()}, + "likelihood_function_calls": 100, + "reference": { + "shares": {"nn_training": 0.5}, + "deltas": {"nn_training": 0.1}, + "compared": "shares_of_wall", + "tolerance_abs": 0.1, + "within_tolerance": True, + "gates_reproduction": variant == "bare_sampler", + }, + "identity": { + "device": "cpu", + "backend": "numpy", + "precision": "float64", + "library_version": "2026.10.12.1", + "software": {"PyAutoFit": "0" * 40, "nautilus-sampler": "1.0.5"}, + }, + "provenance": {"host": "test-host", "measured_at": "2026-10-14T10:00:00Z"}, + } + + +@pytest.fixture +def root(tmp_path): + (tmp_path / "results" / "x").mkdir(parents=True) + return tmp_path + + +def _write(root, name, row): + (root / "results" / "x" / name).write_text(json.dumps(row)) + + +def test_one_record_per_component_plus_wall_and_a_selection_each(root): + _write(root, "a.json", _row()) + doc, rows = bc.build(root) + assert len(rows) == 1 and len(doc["setups"]) == 1 + assert sorted(r["metric"] for r in doc["records"]) == [ + "likelihood", + "nn_training", + "other", + "wall", + ] + assert {r["axis"] for r in doc["records"] if r["metric"] == "wall"} == {"runtime"} + assert {r["axis"] for r in doc["records"] if r["metric"] != "wall"} == {"breakdown"} + assert doc["coverage"]["expected"]["cells"] == len(doc["selections"]) == 4 + assert all(r["validation"]["status"] == "unreviewed" for r in doc["records"]) + assert doc["comparisons"] == [] + + +def test_zero_inapplicable_component_is_a_slot_not_a_zero_record(root): + _write(root, "a.json", _row("bare_sampler", nn_training=5.0, checkpoint_io=0.0, other=5.0)) + doc, _ = bc.build(root) + assert "checkpoint_io" not in {r["metric"] for r in doc["records"]} + slot = next(s for s in doc["selections"] if s["metric"] == "checkpoint_io") + assert slot["status"] == "inapplicable" and slot["record_id"] is None + assert doc["coverage"]["observed"]["selected"] == len(doc["selections"]) - 1 + + +def test_non_result_json_is_ignored_and_render_is_deterministic(root): + _write(root, "a.json", _row()) + _write(root, "other.json", {"schema": "something-else"}) + first, second = bc.render(root), bc.render(root) + assert first == second + assert json.loads(first["catalogue.json"])["generated_at"] == "2026-10-14T10:00:00Z" + + +def test_check_mode_reports_stale_then_current(root, capsys): + _write(root, "a.json", _row()) + assert bc.main(["--root", str(root), "--check"]) == 1 + assert bc.main(["--root", str(root)]) == 0 + assert bc.main(["--root", str(root), "--check"]) == 0 + row = _row() + row["wall_s"] = 11.0 + _write(root, "a.json", row) + assert bc.main(["--root", str(root), "--check"]) == 1 + + +def test_committed_catalogue_is_current(): + assert bc.main(["--root", str(REPO), "--check"]) == 0 + + +@pytest.mark.skipif(not (PULSE / "pulse" / "summary.py").is_file(), reason="no PyAutoPulse beside") +def test_committed_results_validate_with_the_pulse_reader(root): + shutil.copytree(REPO / "results", root / "results", dirs_exist_ok=True) + assert bc.validate(bc.render(root), PULSE) == "ok" + + +@pytest.mark.skipif(not (PULSE / "pulse" / "summary.py").is_file(), reason="no PyAutoPulse beside") +def test_pulse_reader_refuses_a_broken_catalogue(root): + _write(root, "a.json", _row()) + outputs = bc.render(root) + doc = json.loads(outputs["catalogue.json"]) + broken = copy.deepcopy(doc) + broken["records"][0]["unit"] = "MiB" + outputs["catalogue.json"] = json.dumps(broken) + with pytest.raises(ValueError): + bc.validate(outputs, PULSE) diff --git a/scripts/misc/test/test_exclusive_timer.py b/scripts/misc/test/test_exclusive_timer.py new file mode 100644 index 0000000..0e617c8 --- /dev/null +++ b/scripts/misc/test/test_exclusive_timer.py @@ -0,0 +1,56 @@ +from profiling.exclusive_timer import ExclusiveTimer, shares, within_tolerance + + +class FakeClock: + def __init__(self): + self.t = 0.0 + + def __call__(self): + return self.t + + +def test_nested_time_is_charged_to_the_innermost_component(): + clock = FakeClock() + timer = ExclusiveTimer(clock=clock) + + def inner(): + clock.t += 3.0 + + def outer(): + clock.t += 1.0 + timed_inner() + clock.t += 2.0 + + timed_inner = timer.wrap("nn_training", inner) + timer.wrap("bound_geometry", outer)() + assert timer.totals == {"nn_training": 3.0, "bound_geometry": 3.0} + assert timer.counts == {"nn_training": 1, "bound_geometry": 1} + + +def test_exception_still_closes_the_interval(): + clock = FakeClock() + timer = ExclusiveTimer(clock=clock) + + def boom(): + clock.t += 1.0 + raise RuntimeError + + try: + timer.wrap("likelihood", boom)() + except RuntimeError: + pass + assert timer.totals == {"likelihood": 1.0} + assert not timer._stack + + +def test_shares_other_is_wall_minus_instrumented_and_never_negative(): + seconds = shares({"a": 2.0, "b": 1.0}, 5.0, ("a", "b", "c")) + assert seconds == {"a": 2.0, "b": 1.0, "c": 0.0, "other": 2.0} + assert shares({"a": 6.0}, 5.0, ("a",))["other"] == 0.0 + + +def test_within_tolerance(): + ok, deltas = within_tolerance({"x": 0.55, "y": 0.1}, {"x": 0.5, "y": 0.25}, 0.1) + assert not ok + assert round(deltas["x"], 6) == 0.05 and round(deltas["y"], 6) == -0.15 + assert within_tolerance({"x": 0.55}, {"x": 0.5}, 0.1)[0] diff --git a/scripts/misc/tooling/build_catalogue.py b/scripts/misc/tooling/build_catalogue.py new file mode 100644 index 0000000..403574f --- /dev/null +++ b/scripts/misc/tooling/build_catalogue.py @@ -0,0 +1,350 @@ +"""Build the profiling-summary@2 catalogue and the Pages board (stdlib only). + +Run ``python scripts/misc/tooling/build_catalogue.py [--check] [--validate-with PULSE]``. + +Reads every committed result row under ``results/**/*.json`` (schema +``autofit-profiling-breakdown``) and writes, deterministically: + +- ``dashboard/catalogue.json`` — the PyAutoPulse read contract ``profiling-summary`` v2 + (setups, one record per measured component plus the wall, explicit selections). The + shape follows autolens_profiling's ``build_catalogue.py``; nothing here judges a + timing, picks a "latest" row or infers acceptance — every row is ``unreviewed``. +- ``dashboard/index.html`` — a minimal page listing each cell's breakdown. +- ``dashboard/state.json`` — the project badge feed (PyAutoBrain ``board/_state.py`` v1); + the Brain cockpit reads the PyAutoPulse organ feed, not this one. + +``--check`` exits 1 when any committed output differs from a fresh render (the lint gate). +``--validate-with PATH`` validates the catalogue with the Pulse checkout's own reader +(``pulse.summary.classify``) before anything is written. +""" + +from __future__ import annotations + +import argparse +import html +import json +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[3] +RESULT_SCHEMA = "autofit-profiling-breakdown" +PROJECT = "autofit_profiling" +REPO_URL = "https://github.com/PyAutoLabs/autofit_profiling" +PAGES_URL = "https://pyautolabs.github.io/autofit_profiling/" +OUTPUTS = ("catalogue.json", "index.html", "state.json") + +# Components whose exact-zero value means "this variant has no such phase", declared +# as an inapplicable slot instead of a zero-cost record (REFERENCE.md: missing cells +# belong in selections, not zero-valued records). +INAPPLICABLE_REASON = { + "checkpoint_io": "the bare sampler is run without a checkpoint file, so it writes none", +} + +LIMITATIONS = [ + "Laptop CPU only (single host, WSL2); wall times vary up to ±40 % between sessions, so shares are the comparable quantity, not seconds.", + "Component times are exclusive monkey-patched wall-clock intervals; 'other' is wall minus the instrumented total, not a measured phase.", + "Every record is unreviewed: the producer declares measurements, the Brain's profiling conductor judges them; nothing here is a verdict.", + "No temporal comparisons yet: one baseline per setup, no drift policy.", +] + + +def load_rows(root: Path): + rows = [] + for path in sorted((root / "results").rglob("*.json")): + doc = json.loads(path.read_text()) + if isinstance(doc, dict) and doc.get("schema") == RESULT_SCHEMA: + rows.append((path.relative_to(root).as_posix(), doc)) + return rows + + +def _setup(doc): + s = doc["setup"] + config = { + "search": {"value": s["search"], "unit": "name"}, + "variant": {"value": doc["variant"], "unit": "name"}, + "likelihood": {"value": s["likelihood"], "unit": "name"}, + "free_parameters": {"value": 3, "unit": "count"}, + "data_pixels": {"value": 100, "unit": "count"}, + "output_config": {"value": s["config"], "unit": "name"}, + } + for key, value in s["search_settings"].items(): + unit = "count" if key in ("n_live", "n_networks", "number_of_cores") else "dimensionless" + config[key] = {"value": value, "unit": unit} + return { + "id": s["id"], + "label": s["label"], + "dataset": s["dataset"], + "model": "gaussian", + "instrument": None, + "unknowns": {"instrument": "toy 1-D dataset; no instrument"}, + "configuration_id": f"{s['id']}@{doc['version']}", + "configuration": dict(sorted(config.items())), + } + + +def _identity(doc): + ident = doc["identity"] + return { + "device": ident["device"], + "backend": ident["backend"], + "precision": ident["precision"], + "library_version": ident["library_version"], + "software": dict(sorted(ident["software"].items())), + } + + +def _method(doc, suffix): + m = doc["method"] + return { + "id": f"{m['id']}/{suffix}", + "statistic": m["statistic"], + "repetitions": m["repetitions"], + "warmup": m["warmup"], + "synchronization": m["synchronization"], + "cache_state": m["cache_state"], + } + + +def _record(doc, path, setup_id, axis, metric, value, method, fragment): + prov = doc["provenance"] + return { + "id": f"{setup_id}/{axis}/{metric}", + "setup_id": setup_id, + "run_id": f"{setup_id}@{prov['measured_at']}", + "axis": axis, + "metric": metric, + "unit": "s", + "measurement": {metric: value}, + "identity": _identity(doc), + "method": method, + "provenance": { + "host": prov["host"], + "measured_at": prov["measured_at"], + "has_provenance": True, + "qualified": True, + }, + "validation": { + "status": "unreviewed", + "reason": "producer measurement on one laptop host; awaits the profiling conductor's triage", + }, + "evidence": {"path": path, "fragment": fragment}, + } + + +def _selection(record, status="unreviewed", reason="the cell's committed median row"): + return { + "id": f"select/{record['id']}", + "setup_id": record["setup_id"], + "axis": record["axis"], + "metric": record["metric"], + "unit": record["unit"], + "method_id": record["method"]["id"], + "identity": record["identity"], + "host": record["provenance"]["host"], + "status": status, + "record_id": record["id"], + "reason": reason, + } + + +def build(root: Path): + rows = load_rows(root) + setups, records, selections = [], [], [] + for path, doc in rows: + setup = _setup(doc) + sid = setup["id"] + setups.append(setup) + wall = _record( + doc, path, sid, "runtime", "wall", doc["wall_s"], _method(doc, "wall"), "/wall_s" + ) + records.append(wall) + selections.append(_selection(wall)) + method = _method(doc, "components") + for metric, value in doc["components_s"].items(): + if value == 0 and metric in INAPPLICABLE_REASON: + slot = _selection( + _record(doc, path, sid, "breakdown", metric, 0, method, None), + "inapplicable", + INAPPLICABLE_REASON[metric], + ) + slot["record_id"] = None + selections.append(slot) + continue + rec = _record( + doc, path, sid, "breakdown", metric, value, method, f"/components_s/{metric}" + ) + records.append(rec) + selections.append(_selection(rec)) + measured = sorted(r["provenance"]["measured_at"] for r in records) + latest = measured[-1] if measured else None + catalogue = { + "schema": "profiling-summary", + "version": 2, + "project": PROJECT, + "scope": "search-fit-breakdown", + # Deterministic: the render stamp is the newest evidence time, never the clock, + # so --check is stable and an unchanged result set renders byte-identically. + "generated_at": latest or "1970-01-01T00:00:00Z", + "evidence_updated_at": latest, + **({} if latest else {"evidence_updated_at_reason": "no result rows yet"}), + "valid_until": None, + "producer_revision": None, + "comparison_policy": {"id": "no-temporal-comparisons"}, + "coverage": { + "expected": {"cells": len(selections)}, + "observed": { + "setups": len(setups), + "records": len(records), + "selected": sum(1 for s in selections if s["record_id"]), + }, + "excluded": [], + }, + "setups": setups, + "records": records, + "selections": selections, + "hazards": [], + "recommendations": [], + "comparisons": [], + "limitations": LIMITATIONS, + } + return catalogue, rows + + +def render_state(catalogue): + n = catalogue["coverage"]["observed"] + return { + "schema_version": 1, + "organ": PROJECT, + "repo": PROJECT, + "status": "grey" if not n["records"] else "green", + "headline": ( + f"{n['setups']} setups · {n['records']} records (search.fit breakdown); " + "no drift policy yet — baseline only, not a verdict" + ), + "updated": catalogue["generated_at"], + "pages_url": PAGES_URL, + "items": [], + } + + +def _pct(x): + return f"{100 * x:.1f} %" + + +def render_html(rows, catalogue): + out = [ + "", + '', + '', + "autofit_profiling", + "", + "

autofit_profiling

", + "

Where the time in one PyAutoFit search.fit goes. Producer page for the " + 'PyAutoPulse instance fit ' + '(catalogue.json, profiling-summary@2). ' + "Measurements are declared, not judged.

", + ] + for path, doc in rows: + s = doc["setup"] + ref = doc["reference"] + out.append(f"

{html.escape(s['id'])}

") + out.append( + f"

{html.escape(s['label'])} · wall {doc['wall_s']:.2f} s (median of " + f"{doc['method']['repetitions']}) · {doc['likelihood_function_calls']} likelihood calls · " + f"{html.escape(doc['identity']['device'])}/{html.escape(doc['identity']['backend'])} · " + f"PyAutoFit {html.escape(doc['identity']['library_version'])} · measured " + f"{html.escape(doc['provenance']['measured_at'])} · " + f'result row

' + ) + out.append( + "" + ) + ranked = sorted(doc["components_s"].items(), key=lambda kv: -kv[1]) + for name, sec in ranked: + share = doc["shares_of_wall"][name] + out.append( + f"" + f'' + ) + out.append("
componentsecondsshare of wall
{html.escape(name)}{sec:.2f}{_pct(share)}
") + verdict = "within" if ref["within_tolerance"] else "outside" + role = "reproduction gate" if ref["gates_reproduction"] else "information only" + out.append( + f"

Findings-page shares ({ref['compared']}, {role}): " + + ", ".join( + f"{html.escape(k)} {_pct(v)} (Δ {100 * ref['deltas'][k]:+.1f} pp)" + for k, v in ref["shares"].items() + ) + + f" — {verdict} ±{100 * ref['tolerance_abs']:.0f} pp.

" + ) + out.append("

Limitations

    ") + out.extend(f"
  • {html.escape(x)}
    • " for x in catalogue["limitations"]) + out.append("
") + out.append( + f"

Rendered from {len(rows)} result rows by " + f'build_catalogue.py; ' + f"evidence updated {html.escape(str(catalogue['evidence_updated_at']))}.

" + ) + out.append("") + return "\n".join(out) + "\n" + + +def render(root: Path): + catalogue, rows = build(root) + return { + "catalogue.json": json.dumps(catalogue, indent=2) + "\n", + "index.html": render_html(rows, catalogue), + "state.json": json.dumps(render_state(catalogue), indent=2) + "\n", + } + + +def validate(outputs, pulse: Path): + sys.path.insert(0, str(pulse.resolve())) + from pulse.summary import classify + + outcome, errors = classify(json.loads(outputs["catalogue.json"]), ("profiling-summary", 2)) + if outcome != "ok": + raise ValueError(f"Pulse v2 validation {outcome}: " + "; ".join(errors[:20])) + return outcome + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--root", type=Path, default=ROOT) + parser.add_argument("--check", action="store_true") + parser.add_argument( + "--validate-with", type=Path, help="PyAutoPulse checkout whose reader validates the feed" + ) + args = parser.parse_args(argv) + root = args.root.resolve() + outputs = render(root) + if args.validate_with: + print(f"pulse validator: {validate(outputs, args.validate_with)} (profiling-summary@2)") + out = root / "dashboard" + stale = [n for n in OUTPUTS if not (out / n).is_file() or (out / n).read_text() != outputs[n]] + if args.check: + if stale: + print(f"catalogue: STALE ({', '.join(stale)}); run build_catalogue.py") + return 1 + print("catalogue: current") + return 0 + out.mkdir(parents=True, exist_ok=True) + for name in OUTPUTS: + (out / name).write_text(outputs[name]) + doc = json.loads(outputs["catalogue.json"]) + print( + f"catalogue: wrote {len(doc['setups'])} setups, {len(doc['records'])} records, " + f"{len(doc['selections'])} selections" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/wiki/campaigns/search_fit_bottlenecks.md b/wiki/campaigns/search_fit_bottlenecks.md new file mode 100644 index 0000000..d045c49 --- /dev/null +++ b/wiki/campaigns/search_fit_bottlenecks.md @@ -0,0 +1,77 @@ +# Epic 1 — where one `search.fit` spends its time (fast likelihood) + +The ranked bottleneck table for epic 1 of the PyAutoFit profiling campaign (Pulse task +`autofit_profiling_bootstrap`, campaign `autofit`; search-extensibility B4a, +autofit_profiling#2): one `af.Nautilus(...).fit` on gaussian_x1, a likelihood fast enough +that the overhead has nowhere to hide. Epic 2 (the EP loop) is **not** opened: it waits +for epic 1's top findings to ship. + +This table is a producer measurement and a ranking of *where time goes*, not a verdict +on what to change. The Brain's profiling conductor judges; fixes take the normal Mind +route as one-PR prompts. + +## Measurement + +Rows: [`results/gaussian_x1/nautilus/breakdown.json`](../../results/gaussian_x1/nautilus/breakdown.json) +(cell [`search_fit_breakdown.py`](../../scripts/gaussian_x1/nautilus/search_fit_breakdown.py), +findings [`search_fit_breakdown.md`](search_fit_breakdown.md)). Laptop CPU (WSL2, 8 +cores), numpy fp64, PyAutoFit `911fe05e1` (2026.10.12.1+1), nautilus 1.0.5, +`n_live=200`, `n_networks=4`, `seed=1`, one core, library output defaults; median of 3 +fresh fits; wall 28.4 s (repeats 25.8 / 28.4 / 28.9 s); 7 501 likelihood calls; +exclusive component times. + +## Ranked table + +| rank | component | owner | seconds | share of wall | +|---:|---|---|---:|---:| +| 1 | NN ensemble training (`NeuralNetworkEmulator.train`, 26 bound updates) | nautilus | 11.87 | 43.4 % | +| 2 | checkpoint I/O (`Sampler.write` / `write_shell_update`, 103 HDF5 writes) | nautilus, switched on by PyAutoFit's `filepath=checkpoint.hdf5` | 7.09 | 24.5 % | +| 3 | bound geometry (`NautilusBound.compute` minus NN) | nautilus | 5.61 | 20.4 % | +| 4 | visualisation (`SearchUpdater.visualize`) | PyAutoFit | 1.16 | 4.1 % | +| 5 | sampler bookkeeping (rest of `Sampler.run`) | nautilus | 0.67 | 2.3 % | +| 6 | samples output (pre/post-fit output, samples, latents, summary) | PyAutoFit | 0.49 | 1.7 % | +| 7 | likelihood (`Sampler.evaluate_likelihood`: prior transform + `Fitness`) | PyAutoFit + user | 0.32 | 1.1 % | +| 8 | other (setup, result construction — unattributed remainder) | PyAutoFit | 0.29 | 1.0 % | +| 9 | proposal sampling (`Sampler.sample_shell`) | nautilus | 0.22 | 0.8 % | + +PyAutoFit-attributable time (ranks 2's switch, 4, 6, 7's wrapper, 8) is dominated by the +checkpoint write, which is the only PyAutoFit lever above 5 %: the sampler's own +machinery (ranks 1, 3, 5, 9) is 67 % of wall and was ruled on 2026-07-09 (findings §5). + +## `likelihood_share` cross-check (B2/B3 rows, autofit_inference) + +`likelihood_share` (= `per_call_s × likelihood_evals / wall`, basis `estimated`) from the +B3 wave-1 pilot rows, gaussian_x3 (10 free parameters), laptop CPU, PyAutoFit `0dbf258c` +(pre-A2). Read from autofit_inference `dashboard/summary.json` (inference-summary@1) at +commit [`637713b2`](https://github.com/PyAutoLabs/autofit_inference/blob/637713b253ab149f87a08ea0b02fcb009b4aa24e/dashboard/summary.json) +(branch `feature/search-ext-b3-postmerge-fixes`, PR #7; values identical on main +`900648b5`). `catalogue/search_catalogue.json` at the same commit carries no +`likelihood_share` field (it aggregates legs by task), so the per-run summary records are +the source. + +| search | backend | rows with a share | median share | range | +|---|---|---:|---:|---:| +| Nautilus | numpy | 11 | 1.9 % | 1.1–3.4 % | +| Nautilus | JAX CPU | 12 | 0.01 % | 0.00–0.02 % | +| DynestyStatic | numpy | 4 | 32.7 % | 19.0–52.3 % | +| DynestyStatic | JAX CPU | 4 | 4.5 % | 2.6–7.3 % | +| DynestyDynamic | numpy | 4 | 26.1 % | 21.1–53.1 % | +| DynestyDynamic | JAX CPU | 4 | 2.5 % | 1.4–4.4 % | +| SMC | JAX CPU | 3 | 0.01 % | 0.01–0.03 % | + +The Nautilus numpy pilot share (1.9 %, 10 parameters) agrees with this cell's measured +1.1 % (3 parameters): on a fast likelihood the likelihood is ~1–2 % of a Nautilus +`search.fit` and the remaining ~98 % is search and wrapper overhead. Under JAX the +likelihood share is ~10⁻⁴, so every row above it in the ranked table is the whole cost. +Searches without a share (optimisers, emcee, zeus, NUTS) recorded none in wave 1. + +## Candidate one-PR prompts (for the human / profiling conductor; not filed here) + +1. Nautilus checkpoint cadence under PyAutoFit (rank 2): measure the write cost against + `iterations_per_full_update` / an opt-out, preserving resume. +2. Visualisation on a fast likelihood (rank 4): the end-of-fit visualise is 1.2 s here. +3. Bound geometry (rank 3) and NN training (rank 1) stay nautilus-side and ruled + (findings §5); revisit only with new evidence. + +Filed as a Pulse campaign entry: PyAutoPulse `tasks/autofit_profiling_bootstrap.md` +(check-in 2026-10-14). diff --git a/wiki/campaigns/search_fit_breakdown.md b/wiki/campaigns/search_fit_breakdown.md new file mode 100644 index 0000000..35a415d --- /dev/null +++ b/wiki/campaigns/search_fit_breakdown.md @@ -0,0 +1,130 @@ +# Nautilus `search.fit` breakdown — gaussian_x1 (findings) + +Cell: [`scripts/gaussian_x1/nautilus/search_fit_breakdown.py`](../../scripts/gaussian_x1/nautilus/search_fit_breakdown.py) +· rows: [`results/gaussian_x1/nautilus/breakdown.json`](../../results/gaussian_x1/nautilus/breakdown.json) +(the `search.fit`) and [`breakdown_bare_sampler.json`](../../results/gaussian_x1/nautilus/breakdown_bare_sampler.json) +(the reproduction) · catalogue: `dashboard/catalogue.json` (`profiling-summary@2`, Pulse instance `fit`). + +Re-homed on 2026-10-14 (search-extensibility B4a, autofit_profiling#2) from +`autofit_workspace_developer/searches_minimal/{nautilus_profile.py,nautilus_bottleneck_findings.md}` +(research task Jammy2211/autofit_workspace_developer#18, verdict 2026-07-09). §1–§5 below +are the original findings, kept verbatim in substance; §0 is the B4a re-measurement. + +## 0. B4a re-measurement (2026-10-14, laptop CPU) + +Laptop `DESKTOP-H143S82` (WSL2, 8 cores), numpy fp64, PyAutoFit `911fe05e1` +(2026.10.12.1+1), nautilus 1.0.5, scikit-learn 1.5.1, `n_live=200`, `n_networks=4`, +`seed=1`, one core; median of 3 fresh fits. Component times are exclusive (a nested call +is charged to the innermost component). + +**Reproduction (bare `nautilus.Sampler`, the original measurement):** wall 22.5 s, +16 400 likelihood calls in every repeat, log Z −57.455 (the original reference value). + +| component | findings §1 | B4a | Δ | +|---|---:|---:|---:| +| NN ensemble training | 51.7 % | 53.8 % | +2.1 pp | +| bound geometry | 25.2 % | 25.3 % | +0.1 pp | +| likelihood (incl. transform) | 14.3 % | 11.6 % | −2.7 pp | +| proposal sampling | 5.2 % | 5.7 % | +0.5 pp | + +All three gated shares (NN training, bound geometry, likelihood) are within the cell's +stated tolerance of ±10 percentage points; the cell exits non-zero when they are not. + +**The full `af.Nautilus(...).fit`** (what epic 1 profiles): wall 28.4 s, 7 501 likelihood +calls (PyAutoFit's default `n_eff=500` stops earlier than bare nautilus's 10 000), log Z +−57.511. Ranked: NN training 43.4 %, **nautilus checkpoint I/O 24.5 %**, bound geometry +20.4 %, PyAutoFit visualisation 4.1 %, sampler bookkeeping 2.3 %, PyAutoFit samples output +1.7 %, likelihood 1.1 %, other 1.0 %, proposal sampling 0.8 %. The checkpoint I/O is new: +the bare sampler writes no checkpoint, whereas PyAutoFit always hands nautilus +`filepath=checkpoint.hdf5`, and nautilus's 103 HDF5 writes cost ~69 ms each here. The +ranked epic-1 table, joined with autofit_inference's `likelihood_share`, is +[`search_fit_bottlenecks.md`](search_fit_bottlenecks.md). + +The likelihood itself is cheaper under PyAutoFit (~43 µs/call through `Fitness` and the +vectorised prior, against ~160 µs/call for the bare script's `instance_from_vector` +callback), so the sampler-internal likelihood share (1.2 %) is *not* a reproduction of +the 14.3 % row — that comparison is recorded in the row for information only. + +## 1. Where the time goes (Phase 1, 2026-07) + +With a fast likelihood, Nautilus wall time is almost entirely sampler overhead, +and it is **not only the neural networks**: + +| component | 3-param Gaussian fit | fast 10-d likelihood (5 µs/call) | +|---|---:|---:| +| NN ensemble training | 51.7 % | 40.2 % | +| bound geometry (ellipsoid unions + 1000-pt volume MC) | 25.2 % | 38.2 % | +| proposal sampling | 5.2 % | 18.0 % | +| likelihood (incl. transform) | 14.3 % | **0.6 %** | + +nautilus 1.0.5, `n_live=200`, `seed=1`, laptop CPU (WSL2, 8 cores). Wall times vary +±40 % between sessions (thermal/contention); every comparison is within-run. The fast +10-d scenario was not re-homed (it is not a PyAutoFit `search.fit`). + +Mechanics (all in nautilus source): each bound update trains one +`NeuralNetworkEmulator` **per ellipsoid** — `n_networks=4` sklearn MLPs +(hidden layers 100/50/20, `tol=0`, early stop after 10 flat epochs), each +pinned to a single thread (`neural.py`, `threadpool_limits(1)`). Training +parallelises only across the ensemble via the sampler pool (`pool_s`). + +**PyAutoFit wrapper finding** (`autofit/non_linear/search/nest/nautilus/search.py`): +`fit_x1_cpu` — the JAX/GPU path — passes `pool=None`, so in exactly the +fast-GPU-likelihood regime every MLP trains serially. + +## 2. Zero-code knobs (Phase 2) + +All configs preserved quality (log Z within 0.02 nats of baseline both +scenarios, ESS ~10 000, reference log Z −57.5 for the Gaussian fit): + +| config | fast 10-d | 3-param fit | note | +|---|---:|---:|---| +| `n_networks=0` (pure ellipsoid bounds) | **3.2×** | **2.0×** | +1–10 % evals; log Z −57.451 vs −57.455 | +| `n_update=2*n_live` (retrain half as often) | 3.0× | — | quality intact | +| `pool=(None, 4)` (parallel MLP training) | 2.3× | 1.1× | `os.fork()` + JAX threads ⇒ deadlock risk | +| `n_networks=2` | 2.2× | — | | +| small MLP `(32,)` | 2.0× | 1.0× | | +| `vectorized=True` (batched callback) | 2.0× | — | the hook a vmapped GPU likelihood needs | +| `n_networks=0` + `n_update=2x` combined | 2.7× | 1.9× | ≈ `n_networks=0` alone; they don't stack | + +`n_networks=0` is an explicitly supported nautilus mode (`bounds/neural.py:74`): bounds +fall back to the ellipsoid union, i.e. MultiNest-style region sampling. Its cost is +sampling efficiency — here only +1 % (10-d) to +10–27 % (3-param) more likelihood calls. + +**Caveat that decided against it (see §5):** evals-to-ML on the 3-param fit degraded +4278 → 6481 (+50 %) — the sampler takes materially longer to *find the peak* without the +NN bounds, even on an easy unimodal problem. On real curved/degenerate lens posteriors +at 20–30 parameters the ellipsoid-only efficiency loss is expected to be much larger. + +## 3. JAX/GPU MLP swap prototype (Phase 3) — negative on CPU + +A single jitted JAX program replacing the sklearn ensemble (vmap over the 4 networks, +full-batch Adam, padded shapes, jitted predict) matched sklearn's quality but was +**4–6× slower** on CPU (fast 10-d: 5.3 s → 24–28 s NN time; 3-param fit: 37 s → 118 s+). +NN training is ≤ 52 % of wall, so even free training beats it by at most ~2× — less than +`n_networks=0` delivers for zero code. **Parked**; revisit only if a GPU-node measurement +of a real lens fit contradicts this. + +## 4. Can nautilus be JAX-ified wholesale? No. + +The sampler's control flow is adaptive and host-side by design (data-dependent while +loops, a growing point store, per-bound sklearn objects, HDF5 checkpointing). Porting +that to XLA is writing a different sampler, which is what NSS already is. The long-term +answer remains a JAX-native sampler swap. + +## 5. Verdict (human decision 2026-07-09) + +The `n_networks=0` route was **rejected by the maintainer**: for real lens-model sampling +the NN bounds are what keep Nautilus's sampling efficiency high. No PyAutoFit config +change and no GPU validation follow-up were filed. What stands: + +1. **The overhead anatomy (§1, reproduced in §0) is the reference** for Nautilus speed-up + work: NN training 40–54 %, bound geometry 20–38 %; they must be attacked together for + a win beyond ~2×. +2. **Do not wire a process pool into the JAX path** (`pool=(None, N)`): modest win, and + `os.fork()` under a JAX-threaded process risks deadlock. +3. **The JAX/GPU MLP swap is parked with a measured negative** (§3). +4. **The long-term answer remains a JAX-native sampler swap** (§4). +5. If Nautilus overhead work is revisited, the next targets in order: bound geometry + (~0.2–0.9 s/bound; profile `Union.compute`/split vs the 1000-pt volume MC first), then + early-stopping-preserving training parallelism without process forks. B4a adds a + third, PyAutoFit-side target: the per-update nautilus checkpoint write (§0).