Certified-solver phase C1: captured Nautilus lane-rate replay on A100 (#304) by Jammy2211 · Pull Request #309 · PyAutoLabs/autolens_profiling

Jammy2211 · 2026-09-25T19:06:52Z

Summary

Certified-positive-solver phase C1 (#304): real af.Nautilus proposal batches (HST fixed-light, Delaunay N=1500 + rectangular 39x39, pix1 + pix2 stages, regularization free) captured on RAL (arrays 350659 / 350766) and replayed on an A100 80GB in fp64 (array 350768). The result is the uncertified-lane rate and matched jit(vmap) vs scalar timings at B=16/20/50/100.

Pre-registered gate FAILED (own-composition 1e-9 pin). The result is recorded as it came out; the gate was not loosened. The failures are on catastrophic lanes about 6e4–2.2e5 nats below the peak, where PDIP disagrees with itself (≤ 0.86 nats), and on the uncertified none iterates that the design expects.
Uncertified rate (certified+none, B=20): 1.9–4.9% overall, 0 in the late half of every run.
B=50 jit(vmap) returns wrong likelihoods on 44–50/50 lanes, and the gate cannot see it. This is filed as a separate bug: PyAutoMind draft/bug/autoarray/batched_jit_vmap_b50_wrong_log_likelihood_a100.md.
B=100 runs out of memory (73–74 GiB single allocation).
Human decisions (2026-09-25) are recorded in the note's ## Decisions (human, 2026-09-25) section. C2 proceeds under a pre-registered near-peak nats gate. The C2 build rule passes in 3 of 4 cells; rectangular pix1 is at 11%, below the 15% threshold.

Closes nothing on merge — /prm closes #304.

Scripts Changed

scripts/imaging/likelihood_breakdown/fixed_light_trace.py: new --lanes captured-lane replay mode (time pass with per-lane gate + rate pass) and forwarding of the PyAutoArray #572 preconditioning setting.
scripts/imaging/likelihood_breakdown/nautilus_batch_capture.py: new. Captures real Nautilus proposal batches (pix1 / pix2, regularization free, production priors) to npz with a fingerprint.
scripts/misc/likelihood_breakdown/nautilus_batches.py: new. Loads captures, checks fingerprints and builds the lanes.
scripts/misc/likelihood_breakdown/library_solver_injection.py, fixed_light_numpy_solvers.py, fixed_light_numba_memo_policy_steps.py: forward preconditioning in the harness wrappers.
hpc/batch_gpu/submit_breakdown_imaging_fixed_light_certified_lane_rate_a100_hst_fp64: new RAL submit (capture + replay array, pre-registered gate in its header).
scripts/misc/test/test_nautilus_batches.py, test_fixed_light_certified_lane_rate_submit.py, test_fixed_light_cell.py: tests for the above.
Results: 28 JSON + 24 PNG under results/breakdown/imaging/*captured*; note results/notes/certified_solver_phase_c1_lane_rate_2026_09.md; provenance sidecar results/notes/certified_solver_phase_c1_job350768.json.

Test Plan

scripts/misc/tooling/build_readme.py --check passes
scripts/misc/wall/check_submits.py --check passes (11 contracted submits, 0 failing)
profiling CI (lint / README check) green on the PR

Heart RED override

Development-only override per PyAutoBrain/AUTONOMY.md "Human override for Heart RED (development only)". It permits commit, push and opening this PR only.

Authorization (live human, 2026-09-25, this session). The agent asked:

"Heart RED override for shipping C1. The RED is release validation FAILED (stage integrate), and the YELLOW reasons are the autolens cluster/weak notebook failures and manifest drift. None of it is from this branch. The PR body is drafted and ready, and on your go I push and open the PR. Merging still waits for green CI and your /prm."

The human replied, verbatim: "do these:" followed by that item.

Heart RED reasons at ship time (pyauto-heart readiness --json, re-read 2026-09-25 just before the push, verbatim):

release validation FAILED (stage integrate)

YELLOW at the same read: workspace validation not passing (4 failed, cloud#35579888156: autolens notebooks/cluster/modeling.ipynb, autolens notebooks/weak/a2744.ipynb, autolens scripts/cluster/modeling.py, +1 more); manifest drift: hub organism blurb (organs present) — 7 mismatch(es) vs PyAutoMind/repos.yaml; manifest drift: workspace checkouts (manifest ↔ disk) — 1 mismatch(es) vs PyAutoMind/repos.yaml.

Branch gates passed (re-run at ship time on 78cd546):

scripts/misc/tooling/build_readme.py --check: exit 0
scripts/misc/wall/check_submits.py --check: 11 contracted submits, 0 failing
pytest scripts/misc/test/: 834 passed, 5 skipped

Merge requires a separate human /prm with all required checks green. Heart remains RED for release purposes. This PR does not fix Heart.

Issue: #304
Bug filed from this phase: https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autoarray/batched_jit_vmap_b50_wrong_log_likelihood_a100.md

Also in this PR

XLA-attribution anchor re-pin (46251c9): scripts/misc/likelihood_breakdown/xla_attribution.py pinned PyAutoArray line ranges that PyAutoArray#571 (07145ea9) moved — e.g. _log_det_symmetric_from 924-968 -> 951-995, both log-det callers, the edge subset, regularization_term / mapped_reconstruction / reconstruction_scatter, and the inversion_util.py Jacobi / PDIP ranges. Re-pinned against PyAutoArray main 5f8a8dee (LIBRARY_ANCHOR_REVISION updated; two test fixture lines follow). This was stale on main since #571 and is why main's lint is red; together with the #572 preconditioning forwarding in this PR it makes main's lint green.

🤖 Generated with Claude Code

…play (#304) New scripts/imaging/likelihood_breakdown/nautilus_batch_capture.py fits the phase-B S3 fixed-light system with af.Nautilus at the production source_pix[1] settings (n_live=150, n_batch=20, use_jax_vmap=True, library PDIP) and records every batch reaching Fitness.call_wrap (script-scoped patch of call_wrap and nautilus.Sampler.evaluate_likelihood for the phase tag) to nautilus_batches_<mesh>.npz + provenance JSON. Caps: --n-like-max and --capture-seconds; the post-fit perform_update is skipped. fixed_light_trace.py gains --lanes captured --batches <npz> [--captured-pass time|rate] [--batch-sample K] (library mode only; S3 fingerprint check refuses a mismatched capture). time = phase-B harness and unchanged 1e-9 gate on a window of real batches; rate = untimed per-lane certified/passes over the capture in jit(vmap) chunks, overall / by Nautilus phase / by call half, with recorded comparisons against the capture's PDIP values and a scalar-PDIP recheck of uncertified lanes. New 28-task A100 submit (2 captures, 2 rate passes, mesh x B{16,20,50,100} x 3 phase-B rows) with a documented two-step dependency, static tests, and nautilus_batches unit tests. No library edits, no results committed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… stages (#304) nautilus_batch_capture.py gains --stage pix1|pix2 (production source_pix_1: mass+shear+regularization free, n_live=150; source_pix_2: mass+shear fixed at the S3 fiducial, regularization free, n_live=75). The regularization is free at production priors (AdaptSplit inner/outer LogUniform 1e-6..1e6, signal_scale Uniform 0..1 = packaged adapt_split.yaml = pipeline config). Rectangular cannot build AdaptSplit (no split cross-points) and frees its Constant coefficient (LogUniform 1e-6..1e6), recorded as a deviation. One builder, nautilus_batches.capture_model_from, serves capture and replay. Replay evaluates captured lanes as parameter vectors through the Fitness path (instance_from_vector under the trace), so every lane's regularization reaches the likelihood in the rate and timed passes. The S3 fingerprint now checks the regularization class and S0 fiducial, not per-lane coefficients, and the rebuilt model's parameter order must equal the capture's. Submit grows to 38 tasks: 4 captures + 4 rate passes (mesh x stage), pix1 time grid B=16/20/50/100, pix2 at B=20. Schema v2; tests updated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…_pix_2 RAL task 350659_3 (rectangular pix2) died with "Cannot run Nautilus with less than 2 parameters": the rectangular cell froze mass+shear and freed only the Constant coefficient. Every SLaM source_pix_2 on a rectangular mesh uses RectangularBilinearAdaptImage + al.reg.Adapt, so the "production has no rectangular stage" prose was wrong. - nautilus_batches.regularization_class_for(mesh, stage): AdaptSplit on Delaunay (both stages), Adapt on rectangular pix2, Constant on rectangular pix1 (recorded departure). Adapt priors = packaged adapt.yaml (identical to the pipeline's config copy). - capture: stage-threaded class; docstring, priors_source and departures_from_production corrected (rectangular mesh weight_power / weight_floor fixed where source_pix_2 frees them). - replay: fingerprint regularization_class compared against the stage-dependent class the capture freed; S0 identity still checked through regularization_s0_fiducial and the S3 figure of merit. - submit header prose corrected; wall/cell rows untouched. Rectangular pix2 now has 3 free parameters; delaunay pix1 10, pix2 3, rectangular pix1 8 unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…x1 frees Adapt 1. PyAutoArray #572 (07145ea9) added `preconditioning=` to inversion_util.reconstruction_positive_only_from, which the inversion always passes by name. Every harness wrapper that mirrors that signature now accepts it and forwards it only to a library that declares it: library_solver_injection (certified_solver_injected + its PDIP fallback, library_solver_observed), fixed_light_numpy_solvers.numpy_solver_injected, and fixed_light_numba_memo_policy_steps.policy_outer (which also gains the #566 solver/stats keywords it was missing). The certified active-set solve stays on the Jacobi-scaled system, as the library's own does. 2. Rectangular pix1 now frees al.reg.Adapt, as production SLaM source_pix_1 (regularization_init=al.reg.Adapt): regularization_class_for returns Adapt on rectangular in both stages, the unused Constant priors are dropped, and rect pix1 has 10 free parameters (3 Adapt + 7 mass/shear). Existing rectangular captures recorded Constant and are refused by the replay fingerprint; the submit header's two-step prose says to re-capture tasks 1 and 3 first. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…e FAILED Pull the 28 captured-lane JSONs (4 rate, 24 time) and 24 PNGs of RAL arrays 350766/350768 into results/breakdown/imaging/, add the provenance sidecar results/notes/certified_solver_phase_c1_job350768.json (sacct rows, footers, B=100 OOM and B=50 allocator excerpts, artifact sha256s) and the phase-C1 note. The pre-registered per-lane 1e-9 own-composition gate failed on all 24 timed tasks; B=100 (6 tasks) is the memory-limit row (73-74 GiB allocation in the vmap PDIP reference). Per-lane diagnosis: PDIP/cert+PDIP failures at B<=20 are all on catastrophic lanes (captured logL <= 0) where library PDIP is not reproducible at 1e-9 across separately compiled programs (<= 0.86 nats); certified+none adds the uncertified lanes (passes=16, up to 57 nats). At B=50 jit(vmap) returns wrong values on nearly every lane in all six tasks (the only tasks logging the bfc_allocator 18.3 GiB "not a failure" warning); the gate cannot see it. Rate: 1.9-4.9% uncertified, all in the early half. Policy questions left open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

B=50 jit(vmap) fault filed as its own bug; C2 proceeds under a pre-registered near-peak nats gate; C1 gate stays FAILED as recorded; C2 build rule evaluated (3/4 cells pass, rectangular pix1 11%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ver-phase-c1-lane-rate Resolved conflicts (both sides kept): scripts/imaging/likelihood_breakdown/fixed_light_trace.py (nautilus_batches + logdet_reuse_injection imports), scripts/misc/test/test_fixed_light_cell.py (certified_lane_rate + logdet_reuse submit exclusions). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Jammy2211 · 2026-09-25T19:21:09Z

Merged origin/main into the branch (merge commit dc4d2f7, no rebase/force-push) to clear the conflict with #303/#306 phase 4, #305 and #307.

Resolved files, keeping both sides:

scripts/imaging/likelihood_breakdown/fixed_light_trace.py: the import list now has both nautilus_batches (C1) and logdet_reuse_injection (phase 4). The rest auto-merged. --lanes captured needs library mode and --vmap-batch, and --logdet-candidate rejects --vmap-batch, so the two modes stay mutually exclusive. No flags or helpers are duplicated.
scripts/misc/test/test_fixed_light_cell.py: the fiducial-submit filter now excludes both _fixed_light_certified_lane_rate_ and _fixed_light_logdet_reuse_.

Local gates on the merge: ruff check and ruff format --check pass (220 files); build_readme.py --check is clean; check_submits.py --check shows 14 contracts, 0 failing; the cell passes py_compile and the AUTOLENS_PROFILING_SMOKE run. pytest scripts/misc/test/ gives 911 passed, 5 skipped, 1 failed. The failure is test_xla_attribution.py::test_the_log_det_anchors_enclose_the_installed_library_functions, and it was already there before this merge. The test and xla_attribution.py are identical to main. They pin _log_det_symmetric_from at line 924, but PyAutoArray main 07145ea9 (#571) moved it to line 951. So main fails this test too against current PyAutoArray main, and it needs its own anchor re-pin outside this PR.

Heart red_reasons have not changed: ["release validation FAILED (stage integrate)"]. That is covered by the development-only override already recorded here.

Jammy2211 · 2026-09-25T19:21:45Z

Follow-up: I checked the one failing test in a detached scratch worktree of clean origin/main (00211d3), with the same env and PYTHONPATH. It fails there with the same assertion, assert (951, 995) == (924, 968) (1 failed). The merge did not cause it; it comes from drift on main. PR CI (lint.yml) checks out PyAutoArray's default branch and runs pytest scripts/misc/test/, so this PR's lint check will go red on this test for a reason outside the branch. The lint run on main at 00211d3 is already failing: 5 failed / 881 passed. That includes this anchor test and the preconditioning wrapper failures that this branch's #572 forwarding fixes.

PyAutoArray#571 (07145ea9) inserted positive_only_preconditioning_used in inversion/inversion/abstract.py (+25..+27 from line 603) and the raw-forward PDIP branch in inversion_util.py, leaving xla_attribution's line anchors stale (red on main lint). Re-pinned against PyAutoArray main 5f8a8dee: abstract.py - _log_det_symmetric_from 924-968 -> 951-995 - log_det_curvature_reg_matrix_term 970-1051 -> 997-1078 - log_det_regularization_matrix_term 1071-1132 -> 1098-1159 - edge subset (reconstruction) 671-678 -> 696-703 - regularization_term 858-922 -> 885-949 - mapped_reconstruction 795-857 -> 822-884 - reconstruction_scatter 634-670 -> 659-695 - reconstruction_scatter 679-793 -> 704-820 inversion_util.py - Jacobi build + certified *D 433-450 -> 489-506 - Jacobi PDIP *D 455-464 -> 508-516 - reconstruction_positive_only_from 291-593 -> 291-644 LIBRARY_ANCHOR_REVISION -> "PyAutoArray main 5f8a8dee (2026-09-25)". Test fixtures at the moved lines follow (edge subset 675->700, Jacobi 435->491). Unchanged: curvature add 389, reduced gathers 392-415, inversion_util 12-288, convolver/delaunay/imaging anchors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Jammy2211 · 2026-09-25T19:33:09Z

Anchor re-pin pushed as 46251c9 (xla_attribution line ranges re-pinned to PyAutoArray main 5f8a8dee after PyAutoArray#571). Local gates on the new head, against the PyAutoArray main CI checks out: ruff check . all passed; ruff format --check . 220 files already formatted; build_readme.py --check clean; check_submits.py --check 14 submits with a wall-basis contract, 0 failing, 115 scanned; pytest scripts/misc/test/ 912 passed, 5 skipped, 0 failed (before: test_the_log_det_anchors_enclose_the_installed_library_functions failed with (951, 995) == (924, 968)). Heart readiness is RED only for 'release validation FAILED (stage integrate)', which the recorded development override covers.

🤖 Generated with Claude Code

Jammy2211 and others added 6 commits September 24, 2026 16:56

Jammy2211 added the pending-release PR queued for the next release build label Sep 25, 2026

Jammy2211 mentioned this pull request Sep 25, 2026

perf: certified solver phase C1 — uncertified-lane rate on real Nautilus batches #304

Closed

Jammy2211 merged commit 21875f3 into main Sep 25, 2026
1 check passed

Jammy2211 deleted the feature/certified-solver-phase-c1-lane-rate branch September 25, 2026 19:47

Jammy2211 removed the pending-release PR queued for the next release build label Oct 7, 2026

Navigation Menu

Repository navigation

Certified-solver phase C1: captured Nautilus lane-rate replay on A100 (#304) - #309

Certified-solver phase C1: captured Nautilus lane-rate replay on A100 (#304)#309
Jammy2211 merged 8 commits into
mainfrom
feature/certified-solver-phase-c1-lane-rate

Jammy2211 commented Sep 25, 2026 •

edited

Loading

Uh oh!

Jammy2211 commented Sep 25, 2026

Uh oh!

Jammy2211 commented Sep 25, 2026

Uh oh!

Jammy2211 commented Sep 25, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Jammy2211 commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scripts Changed

Test Plan

Heart RED override

Also in this PR

Uh oh!

Jammy2211 commented Sep 25, 2026

Uh oh!

Jammy2211 commented Sep 25, 2026

Uh oh!

Jammy2211 commented Sep 25, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Jammy2211 commented Sep 25, 2026 •

edited

Loading