[Jammy2211]

Summary

Streaming visibilities phase 4 (epic streaming-visibilities, Discussion https://github.com/orgs/PyAutoLabs/discussions/13; issue PyAutoLabs/PyAutoArray#598). ag.FitInterferometer with ordinary (non-linear) light profiles now runs on an array-free dataset (Interferometer.from_stream / from_sparse_terms), with or without an inversion, in numpy and under jax.jit, matching the in-memory dense fit at rel ≤ 1e-8 (measured 1.8e-15). The light profiles enter through their real-space profile_image and the identity Σ|d − F i_p|²/σ² = data_term − 2 i_pᵀd̃ + i_pᵀW̃i_p; profile_visibilities is never formed (a transformer spy test proves it). Requires the PyAutoArray companion PR (DatasetInterface.data_term, sparse_profile_terms_from, sparse_chi_squared hook).

Red-check recorded in the test file history: with the old raise removed but the override not wired, LP+pixelization log_evidence was −37.0125 vs dense −37.9117 (−2.4 %) and LP+linear −24.3192 vs −25.2183 (−3.6 %), silently. The override closes that.

In-memory sparse fits with ordinary light are unchanged bit-for-bit (they keep passing the profile-subtracted visibilities); only dataset.data is None takes the identity path.

API Changes

Mostly additive. The array-free raise in profile_visibilities is gone (it returns None); model_data, galaxy_model_visibilities_dict and model_visibilities_of_galaxies_list now raise a typed DatasetException on an array-free dataset pointing at model_image_natural / dirty_model_image_natural. New module helper sparse_profile_terms_from; uses_precomputed_data_term_from gains the array-free rule.
See full details below.

Review (Codex gpt-6-astra, 3 independent runs, 2026-10-01)

Four distinct findings, all reproduced before editing (numbers in the second commit's tests):

• F1 (high, introduced) — fixed: a fit subclass overriding noise_map/data on an array-free dataset made _uses_precomputed_data_term false while profile_subtracted_visibilities stayed None, so the inversion paired the subtracted dirty image with the raw operator data_term (LP+pix fom −37.0125 vs dense −37.9117). Now _require_no_array_free_overrides raises a typed DatasetException (array-free datasets support no data/noise-map overrides).
• F2 (high, introduced) — fixed: without an inversion sparse_chi_squared ignored an overridden noise map (χ² 5.2983 unscaled vs 1.3246 dense while noise_normalization switched). Same guard.
• F3 (medium) — fixed: array-free log_likelihood with an inversion returned fast_chi_squared, which carries eps·Σs² from no_regularization_add_to_curvature_diag_value (packaged 1e-3; −0.25 nats at ×100 data). sparse_chi_squared now applies the identity to the total real-space model image (profile_image + inversion.mapped_reconstructed_data), matching the dense log_likelihood at rel 1e-15 for every source type, numpy and jax.jit. fast_chi_squared / log_evidence unchanged.
• F4 (medium, pre-existing on main) — filed, not fixed: galaxy_image_dict dict-merge overwrites a mixed ordinary+linear galaxy's ordinary image (also fit_imaging.py); Mind draft draft/bug/autogalaxy/galaxy_image_dict_mixed_galaxy_overwrites_ordinary_light.md.

Test Plan

• New parity tests vs the in-memory dense fit (rel 1e-8): LP-only (figure_of_merit == log_likelihood), LP+pixelization, LP+linear LP, LP+Basis; jax.jit LP-only and LP+pixelization (traced intensity re-evaluated to prove it is not frozen); TransformerNUFFT/TransformerDFT.visibilities_from spy never called; model_data typed raise; aggregator round-trip and visualizer with an ordinary LP on array-free data.
• Deleted test__fit_figure_of_merit__array_free_dataset__light_profile__raises (behaviour it asserted no longer exists).
• test_autogalaxy: 1305 passed (main: 1293).
• CI green on all legs; merge after PyAutoArray.

Full API Changes (for automation & release notes)

Added

• autogalaxy.interferometer.fit_interferometer.sparse_profile_terms_from(dataset, galaxies, image, xp=np) → (sparse_dirty_image, data_term); (None, None) when no sparse operator or no ordinary light.
• ag.FitInterferometer.sparse_chi_squared — the identity on the total real-space model image (profile_image + inversion.mapped_reconstructed_data); None without a sparse operator. Raises DatasetException if the fit overrides data/noise_map on an array-free dataset.

Changed Behaviour

• ag.FitInterferometer.profile_visibilities — on an array-free dataset returns None instead of raising DatasetException when ordinary light profiles are present.
• ag.FitInterferometer.profile_subtracted_visibilities — returns None on array-free without evaluating profile_visibilities.
• ag.FitInterferometer.model_data, galaxy_model_visibilities_dict, model_visibilities_of_galaxies_list — raise aa.exc.DatasetException on an array-free dataset (previously only reachable for pixelization/linear fits, where they failed with None + array).
• uses_precomputed_data_term_from — returns True for any array-free dataset (dataset.data is None), with the subtracted data_term supplied to the inversion.
• ag.FitInterferometer.log_likelihood / chi_squared on an array-free fit equal the dense residual-map values (identity on the total model image); figure_of_merit / log_evidence unchanged.

Internal

• sparse_dirty_image_from now wraps sparse_profile_terms_from (same result).
• _require_transformer(fit, quantity), _require_no_array_free_overrides(fit), sparse_chi_squared_from(fit, galaxies) helpers (reused by PyAutoLens).

Heart RED development override (human-authorized, 2026-10-01)

Shipped under the AUTONOMY.md "Human override for Heart RED (development only)". Live human authorization in the CLI session (Fable 5.1), quoted: "Yes, ship (push + PR-open)" for task streaming-p4-light-profile-identity (PyAutoLabs/PyAutoArray#598). Exact RED reasons at ship time (pyauto-heart readiness):

• release validation FAILED (stage integrate)
• workspace validation not passing (0 failed, 1 timeout, cloud#36404726969: autolens_test scripts/multi_dataset/rectangular.py)
• manifest drift: public front-door organ tables (generated) — 1 mismatch(es) vs PyAutoMind/repos.yaml

None relate to this branch. Branch gates passed: test_autoarray 1903, test_autogalaxy 1305, test_autolens 793 + 1 xfailed; Codex review 4 findings (3 fixed in-branch, 1 pre-existing filed). Scope: push + PR-open only. Merge requires a separate human /prm with every required check green; this is not a release and does not claim Heart is healthy.

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code