feat: v1 semantic convention - #717
Draft
FBumann wants to merge 97 commits into
Draft
Conversation
The design goals and transitioning goals for linopy's v1 arithmetic convention, under arithmetics-design/goals.md. The convention itself and the bug catalogue (meta issue #714) follow separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Placeholder for the v1 convention document, to be written. Goals are in arithmetics-design/goals.md; the bug catalogue is the meta issue #714. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
FBumann
force-pushed
the
feat/arithmetic-convention
branch
from
May 21, 2026 12:48
40af9d6 to
1e336a4
Compare
Flesh out convention.md from the placeholder into the full spec — thirteen numbered sections in three groups: absence (§1–§7), coordinate alignment (§8–§11), and constraints and reductions (§12–§13). Covers the strict exact-match alignment model and the propagate-don't-fill NaN/absence convention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The convention governs coordinate alignment, absence/NaN handling, constraints, and reductions — not just arithmetic operators — so retitle convention.md and goals.md to "The v1 convention". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce linopy.options["semantics"] — legacy (default) or v1 — with LinopySemanticsWarning, a FutureWarning shown to users by default and exported at top level. Add the autouse `semantics` conftest fixture that runs every test under both conventions, plus legacy/v1 markers to pin a test to one. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 22, 2026
Merged
`_align_constant` branches on `options["semantics"]`: v1 uses exact alignment via `xr.align(join="exact")`; legacy keeps the size-aware positional/left-join behaviour and emits `LinopySemanticsWarning` when v1 would diverge. `_add_constant`/`_apply_constant_op` raise on a NaN in a user-supplied constant under v1, warn under legacy. `Variable.__mul__(DataArray)` now routes through `to_linexpr() * other` so the LinearExpression checks fire; the scalar fast-path is preserved (a NaN scalar diverts to the expression path so v1 raises). Marks the bug-class test groups `TestCoordinateAlignment` (#708/#586/ #550), `TestConstraintCoordinateAlignment`, `TestNaNMasking`, `test_auto_mask_constraint_model`, and four piecewise NaN-padding tests as `@pytest.mark.legacy` — they assert the very behaviour v1 forbids. v1 coverage of those bug classes accretes via later slices. `test/test_legacy_violations.py` (new) adds 22 paired tests covering §5/§8/§9 plus the PyPSA #1683 `0*inf=NaN` case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`merge` now pre-validates that all operands agree on the labels of every shared *user* dimension before concatenating. Helper dims (`_term`, `_factor`) and the concat dim itself are excluded — those legitimately vary between operands. v1 raises on mismatch; legacy keeps current size-based override/outer behaviour and emits `LinopySemanticsWarning` when v1 would diverge. The check uses a new `_merge_shared_user_coords_differ` helper. The existing override/outer decision is unchanged for the actual `xr.concat` call — the new check only gates whether legacy/v1 accept the merge, never how the concat itself runs. Adds 8 paired tests for var+var, var-var, expr+expr, broadcast guard, and warning emission on the merge path. Reclassifies as `@pytest.mark.legacy`: `test_non_aligned_variables` (deliberately disjoint coords), `test_linear_expression_sum` / `test_linear_expression_sum_with_const` (assert `v.loc[:9]+v.loc[10:]` merges), `TestJoinParameter` cases that build `a*b` from mismatched- coord vars, and two SOS2 reformulation tests. File-level legacy mark on `test_piecewise_constraints.py` + `test_piecewise_feasibility.py` until `linopy/piecewise.py` itself is made v1-aware (tracked as Slice P). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Variable.to_linexpr() now produces a LinearExpression whose absent slots (labels == -1) carry NaN coeffs and NaN const under v1, so downstream arithmetic has something to propagate. The expression constant operators (_add_constant, _apply_constant_op) no longer fillna(0) self.const / self.coeffs under v1 — NaN flows through. `merge` sums const along _term with skipna=False under v1, so a slot that's absent in any operand stays absent in the result. Legacy paths keep the silent-fill behaviour verbatim. LinearExpression.isnull() now returns `const.isnull()` under v1: a slot is absent iff its const is NaN. ``vars == -1`` is a dead-term signal (the slot can still be a present constant after fillna), not a slot-level absence marker. Legacy keeps the historical ``(vars == -1).all() & const.isnull()`` formula for byte-for-byte compatibility. Variable.fillna(numeric) now returns a LinearExpression (a constant isn't a variable). Variable.fillna(Variable) stays Variable, as before. Adds 11 tests for §6 propagation (mul/add/sub/div preserve absence, absent-vs-zero distinguishable, present + absent propagates) and §7 resolution (fillna numeric on expr / Variable, present-zero revival). Reclassifies test_masked_variable_model as @pytest.mark.legacy — its assertion "x bound to 10 at masked-y slots" only holds because legacy collapses absent y to 0. The v1 way is x + y.fillna(0) >= 10; a counterpart test in test_legacy_violations.py pins this. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The convention spec names ``reindex`` and ``reindex_like`` among the absence-creating mechanisms (alongside ``mask=``, ``.where()``, ``.shift()``, and ``.unstack()``), but master only had them on ``LinearExpression``. Add them on ``Variable``, with the sentinel fill values (``labels=-1``, ``lower=upper=NaN``) so new positions slot cleanly into §6 propagation. The methods work the same way under both semantics — under legacy the sentinels exist but downstream arithmetic still collapses them back to 0 (the #712 bug), so the user-visible effect of reindex-as- absence only really lands under v1. Adds 5 tests: extend with absent, subset drops, reindex_like with another Variable, and the §4 + §6 hand-off (a reindex-introduced absent flows through ``* 3`` and is visible via ``isnull()``). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Slice C propagated NaN const cleanly but left the storage half-absent after a merge: `(1*x) + xs` at the absent slot kept the `1*x` term's valid coefficient and label even though `const` was NaN there. The §1/§2 promise "absence is one concept, whatever the dtype" only holds if `const.isnull()` at a slot ⇒ every term at that slot has `coeffs = NaN`, `vars = -1`. Add `_absorb_absence(ds)` and call it at the end of `merge` under v1. The constant-operand paths (`_add_constant`, `_apply_constant_op`) don't need explicit absorption — their NaN-propagation naturally preserves the invariant when the input is already v1-compliant (NaN * anything = NaN; dead terms stay dead). Only `merge` opens the gap by concatenating one operand's live term with another operand's absent slot along `_term`. `convention.md` §2 now states the invariant explicitly and introduces the *dead term* terminology, so `fillna(value)` reviving a slot while leaving the sentinel term in place reads as a feature, not a glitch. Adds `test_outer_fillna_then_add_collapses_to_just_added` pinning `(x + y.shift()).fillna(0) + x` — at the previously-absent slot the result has exactly one live term (`1·x[0]`) with `const = 0`, algebraically equal to `x[0]`. At present slots all three terms stay live (`2·x[i] + y[i-1]`), so fillna placement is load-bearing — moving it inside (`x + y.shift().fillna(0) + x`) would double-count `x` at the absent slot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`.add/.sub/.mul/.div/.le/.ge/.eq` already accepted a `join=` argument; this slice's job is just §12's RHS handling under v1. `to_constraint` branches on `options["semantics"]`. Under v1 it skips the legacy `reindex_like(self.const, fill_value=NaN)` step that silently padded a subset RHS, so a coord mismatch with the LHS now flows through `self.sub(rhs)` and gets caught by §8's exact alignment. A NaN in a user-supplied constant RHS raises at construction (§5) — including the PyPSA #1683 case of `min_pu * nominal_fix` with `p_nom=inf` and `p_min_pu=0`. An absent slot in the LHS (propagated from §6) still produces a NaN RHS at that row; downstream auto-mask drops the constraint there, which is exactly §12's "absent slot yields no row." Legacy keeps the old auto-mask path verbatim and adds a `LinopySemanticsWarning` whenever a NaN RHS is observed, so users get the rollout signal without behaviour change. Adds 11 paired tests: TestNamedMethodJoin (inner/outer/left across .add/.mul/.le, plus a "bare op still raises" guard) and TestConstraintRHS (subset RHS raises, NaN RHS raises, PyPSA #1683 on the constraint side, §6→§12 hand-off where the absent LHS slot yields NaN RHS, plus the paired legacy auto-mask documentation and warning-emission tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Coming next, in order:
Then a final pass on docs (user-facing migration / rollout — deferred so far). |
Three internal patterns were violating §8 / §11:
1. ``_add_incremental`` in ``linopy/piecewise.py`` builds
``delta_hi <= delta_lo`` from two ``.isel(piece_dim=slice)`` slices
of the same variable. ``drop=True`` is a no-op for slice indexers
so ``piece_dim`` stays on both with *different* labels (first n-1
vs last n-1 of piece_index) — v1 §8 rejects. Relabel the high
slice onto the low slice's labels so the comparison aligns by
label (the explicit-positional path of §10). Same fix for
``binary_hi <= delta_lo``.
2. ``_incremental_weighted`` computes ``bp0 = bp.isel({dim: 0})``
without ``drop=True``, leaving the breakpoint dim as a scalar
coord on the resulting expression. When that expression appears
as the RHS of ``links.eq_expr == ...`` it conflicts with the LHS,
which has no such coord — §11 aux-coord conflict. Add ``drop=True``.
3. ``reformulate_sos2`` builds its first/last constraints from
scalar isels at different positions on ``sos_dim`` (``x``/``M`` at
``n-1`` paired with ``z`` at ``n-2``, etc.). All without
``drop=True``, so the scalar ``sos_dim`` coord differs across
operands — §11 aux-coord conflict. Add ``drop=True`` to all three
sites.
Removes the module-level ``pytestmark = pytest.mark.legacy`` from
``test_piecewise_constraints.py`` and ``test_piecewise_feasibility.py``
and the method-level marks from the two SOS2 multidim tests. Suite is
+598 tests under v1 vs Slice E (legacy → v1 broadened coverage),
0 failures under either semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§13 falls out of xarray's ``skipna=True`` default; no code changes needed. Adds 4 tests so future drift is caught: sum over a dim, sum without a dim, sum of all-absent (the zero expression), and groupby.sum across heterogeneously-present groups. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `_conflicting_aux_coord(datasets)` and wires it into both `merge` and `_align_constant`. When two operands carry an aux coord of the same name with disagreeing values, v1 raises with a pointer to the explicit resolutions (``.drop_vars(...)`` or ``.assign_coords(...)``). xarray silently drops the conflict — the #295 bug — and legacy keeps that behaviour but now emits a `LinopySemanticsWarning`. The helper guards against string-dtype coord values (no `equal_nan=True` there) so the multiindex case keeps working. `_merge_shared_user_coords_differ` refactored to compare bare ``d.indexes[k]`` instead of ``d.coords[k]``: aux coords no longer leak into the §8 check, so §11 owns aux-coord conflicts cleanly and §8 owns dim-coord mismatches with a separate message. Convention §11 expanded from one paragraph: aux coords are validated and propagated but never computed with — they describe the data, they don't enter the math. Goal #4 in `goals.md` picks this up: user-attached auxiliary coordinates are the user's, linopy never silently rewrites them. `test_linear_expression.py::test_merge` adds ``drop=True`` to its ``.sel`` setup — the test was leaving a leftover scalar coord that v1 now correctly catches as a §11 conflict; the fix preserves the test's intent of exercising merge with differing term counts. Conflict-raising tests (TestAuxCoordConflict) cover expr+const, var+var, scalar-isel-without-drop, the ``drop=True`` escape hatch, plus the paired legacy left-wins documentation and warning-emission tests. Propagation guarantees land in a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Regression coverage on the half of §11 that wasn't tested before: non-conflicting aux coords carry through every binary operator and into constraints. xarray already preserves them; the tests guard against future drift (e.g. a reduction or helper accidentally dropping a non-dim coord). TestAuxCoordPropagation covers ``3*v``, ``v+5`` (single-operand, fast paths), ``v+v`` with matching aux (the merge path), ``v<=10`` (the constraint path), ``x*a`` / ``x+a`` / ``x/a`` / ``x<=a`` where only the constant DataArray carries the coord (the ``_align_constant`` path), and the var+var case where only one side has the coord. Together: every operator times every "one side / both sides" arrangement, since only conflicts on both sides raise. Runs under both semantics — the legacy behaviour matches the v1 behaviour for the non-conflict cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… solve Fills the convention-coverage gaps surfaced by review of the branch: - §1/§2 dead-term storage invariant: pin that after a merge with an absent slot, coeffs=NaN AND vars=-1, not just const=NaN. The existing propagation tests read through isnull() which only checks const, so a regression in _absorb_absence would have passed them. Multi-operand variant catches binary-only-absorption regressions. - §12 equality: mirror the existing <=/>= TestConstraintRHS coverage for ==. Subset RHS raises, NaN RHS raises, absence in LHS drops the row. - §11 extra operators: add mul-constant and == constraint cases to the existing TestAuxCoordConflict. The class already covered +-constant and var+var; these extend coverage to the other call-site shapes. - §13 scope note: mean/resample/coarsen aren't yet on LinearExpression (tracked in #703); the spec text is the rule those will follow when implemented. Docstring note in TestReductionsSkipAbsent makes this explicit so the gap doesn't read as missing coverage. - End-to-end v1 solve: test_masked_variable_model_v1_drops_constraint pins the v1 outcome at the solver layer — con0 masked at absent slots (solver-independent) and x bound to 0 where the constraint still binds. _v1_fillna_binds confirms the §7 escape hatch recovers the legacy outcome. Catches the regression where v1 silently produces wrong solutions instead of raising. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls the seven v1-specific helpers and the user-NaN message out of ``expressions.py`` and into a dedicated ``linopy/semantics.py`` module — a single home for "what v1 means" that imports cleanly from ``config`` and ``constants`` only. Adds a tiny ``is_v1()`` predicate so the 16 scattered ``options["semantics"] == V1_SEMANTICS`` checks collapse to a one-line call. Helpers (renamed to drop the leading underscore now that they're a real module API): ``check_user_nan_scalar``, ``check_user_nan_array``, ``dim_coords_differ`` (was ``_shared_coords_differ`` — clearer name, matches ``merge_shared_user_coords_differ``), ``merge_shared_user_coords_differ``, ``conflicting_aux_coord``, ``absorb_absence``, plus ``is_v1``. No behaviour change — same checks, same warnings, same raises. The diff is mechanical: imports flipped, two local ``is_v1 = options[...]`` bindings replaced by the imported predicate, one missed ``_USER_NAN_MESSAGE`` reference in ``to_constraint`` routed through ``check_user_nan_array`` for consistency. ``expressions.py`` shrinks by ~105 lines. Future v1-only API surface (e.g. exposing ``is_v1()`` as ``linopy.is_v1()`` for downstream code) and the eventual legacy removal at 1.0 both reduce to deletions of ``semantics.py`` and its import sites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three test clusters in ``test_legacy_violations.py`` had near-identical
``test_add_X``, ``test_mul_X``, ``test_div_X`` triples that varied only
by which binary operator they exercised. Collapse each into a single
``@pytest.mark.parametrize("op", ...)`` test:
- TestExactAlignmentConstant: same-size-different-labels and
subset-constant raises, parameterized over add/sub/mul/div.
- TestUserNaNRaises: NaN-DataArray raises over add/sub/mul/div, NaN
scalar over add/sub/mul (div scalar shares the same ``_apply_constant_op``
code path as mul, but ``x / nan`` trips ``__div__``'s unary-negate
TypeError before our check fires; the dispatch needs a separate
fix that's not worth pulling into this refactor).
- TestAbsencePropagation: ``shifted OP scalar`` preserves absence,
parameterized over add/sub/mul/div. Adds a per-op present-slot
value check so the parameterization broadens rather than narrows
the assertion.
Adds a module-level ``_OPS`` dict mapping name → ``operator``
callable so the parameter is the readable name (``"add"``,
``"div"``) while the test still calls the actual operator.
Cuts ~50 lines off ``test_legacy_violations.py`` and makes adding a
new operator a one-line change. Test IDs become e.g.
``test_same_size_different_labels_raises[v1-add]`` — slightly less
self-describing than the explicit-method names but cheap to read.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both methods had v1 and legacy logic interleaved via a ``fillna0`` closure that was identity under v1 and ``da.fillna(0)`` under legacy. Pull them apart into: - ``_add_constant`` / ``_apply_constant_op`` — two-line dispatchers. - ``*_v1`` — v1's implementation, reads as a single coherent story. - ``*_legacy`` — legacy's implementation, ``# LEGACY: remove at 1.0`` marker on each. At 1.0 the removal is mechanical: delete the ``_legacy`` methods and inline the ``_v1`` body into the dispatcher (or rename it back to the public name). Future readers don't have to mentally subtract the legacy branches to understand what v1 does. Add ``LEGACY: remove at 1.0`` marker comments at the other mixed sites in ``expressions.py`` so ``grep`` finds every place that needs touching: ``_align_constant``'s size-aware default fallback, ``to_constraint``'s auto-mask fallthrough, ``LinearExpression.isnull``'s historical AND, and the two warn-on-divergence sites in ``merge``. New ``arithmetics-design/legacy-removal.md`` is the master checklist for the 1.0 cut: every file, function, test, doc edit, and the safe order to do them in. The intent is that the eventual legacy removal takes an afternoon, not a week of grep-archaeology. No behaviour change — same checks, same warns, same raises. Suite is 7282 passed, 0 failures under both semantics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two distinct CI failures both rooted in the v1 harness commit: 1. **Test collection crash on every linopy/*.py module.** ``test/conftest.py`` imported ``linopy.config`` at module top, which loaded linopy from site-packages before pytest's ``--doctest-modules`` collection walked the source tree. The resulting __file__ mismatch broke all 22 module collections. ``pyproject.toml`` already documents this exact failure mode in the ``filterwarnings`` block. Fix: keep the constant *values* (``"legacy"`` / ``"v1"``) inline in conftest as ``_LEGACY_SEMANTICS`` etc. so the parametrize decorator doesn't force an import, and defer the ``LinopySemanticsWarning`` / ``options`` import into the fixture body. The original import comment in pyproject is now mirrored at the top of conftest. 2. **mypy: 72 "no-untyped-def" errors in test_legacy_violations.py.** The new tests were missing parameter type annotations on the fixture-injected params (``x``, ``xs``, ``op``, ``unsilenced``, ``subset``, ``A``, ``da_aux_B``, ...). ``disallow_untyped_defs`` is set globally, so test files need them too. Filled in the types (``Variable``, ``str``, ``None``, ``xr.DataArray``, ``pd.Index``), added an ``isinstance(result, LinearExpression)`` narrowing in ``test_variable_fillna_zero_revives_slot_as_present_zero`` so mypy can pick the right branch of ``fillna``'s return union. Local: 7282 passed, 0 failures under both semantics; ``mypy .`` Success. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three v1 raises were under-informative — naming the rule violated but not the operand, dim, or values involved. Make each message carry the information the helper already has: - **§5 user-NaN**: the old message conflated the two intents the user might have had — *data error* (fix with ``.fillna(value)``) vs *intended absence* (mark on the variable with ``mask=`` / ``.where`` / ``.reindex`` / ``.shift``). The new message separates them and points each to its own remedy. - **§8 merge mismatch**: rename ``merge_shared_user_coords_differ`` (bool) to ``merge_shared_user_coord_mismatch`` (tuple ``(dim, left, right) | None``). Raise text now includes the offending dim name and both sides' labels (truncated), plus the full set of resolution paths from §10: ``.sel`` / ``.reindex`` / ``.assign_coords`` / ``linopy.align`` / ``join=`` on ``.add`` / ``.sub`` / ``.mul`` / ``.div`` / ``.le`` / ``.ge`` / ``.eq``. - **§11 aux-coord conflict**: ``conflicting_aux_coord`` returns ``(name, left_vals, right_vals) | None``. Raise text includes the coord name, both value snippets, and all three resolution paths (``.drop_vars`` / ``.assign_coords`` / ``isel(drop=True)`` — ``.assign_coords`` was previously omitted). The text is now centralized in ``semantics.py`` so the two raise sites in ``expressions.py`` (``_align_constant`` and ``merge``) share one voice instead of paraphrasing each other. New ``TestErrorMessageContent`` pins the rich content in three tests — that the §5 message names both intents, that the §8 message names the dim and both label lists, and that the §11 message names the coord, both value lists, and lists all three §11 fixes (the ``.assign_coords`` omission would have slipped through ``match= "Auxiliary coordinate"`` substrings). Section references (``§5``, ``§8``, ``§11``) deliberately omitted from user-visible text — spec jargon, not a navigation aid for downstream callers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the small-but-real holes in the §1–§13 coverage map. New tests
mostly, plus one code fix that the test surfaced.
§4 — absence creation
- test_where_creates_absence: §4 names ``.where(cond)`` but only
``mask=`` / ``.reindex`` were tested.
- test_unstack_creates_absence_at_missing_combinations: the
non-rectangular MultiIndex case (``stack`` preserves, ``unstack``
fills) is the asymmetry that earns its own test. Hit a real bug
on the way — ``Variable.unstack`` was producing float NaN in the
integer ``labels`` field instead of the ``FILL_VALUE`` sentinel
(-1), violating §2. Fixed by passing ``fill_value=_fill_value``
to the underlying ``Dataset.unstack`` (same pattern as ``shift``).
Audited the rest of the varwrap calls — only ``shift`` and
``unstack`` introduce new positions; the others either preserve
shape (``assign_*``, ``rename``, ``swap_dims``, ``set_index``,
``roll``, ``stack``), select existing positions (``sel`` /
``isel`` / ``drop_*``), or broadcast existing data without fill
(``broadcast_like``, ``expand_dims``).
- test_data_preserving_methods_do_not_create_absence: parameterized
over ``.roll`` / ``.sel`` / ``.isel``, regression-guards §4's
explicit contrast against the creators.
§10 — named-method join= argument
- test_add_join_override_aligns_positionally: positional-mode is the
surprising one in the join= set; pin it explicitly.
- test_reindex_like_resolves_mismatch_before_bare_op and
test_assign_coords_resolves_mismatch_before_bare_op: §10 names
these as the canonical user fixes; pin that the post-fix bare
operator actually accepts the once-mismatched operand.
§11 — auxiliary-coordinate conflicts
- test_assign_coords_resolves_conflict: §11 lists three escape
hatches; only ``.drop_vars`` / ``isel(drop=True)`` were tested.
- test_multi_operand_merge_aux_conflict_raises: the merge-path
check inspects all operands; a 3-way ``v + w + u`` with the
third disagreeing exercises that.
§12 — constraints follow the same rules
- Parameterize the existing subset / NaN / absence-propagation
tests in ``TestConstraintRHS`` over the three signs (``le`` /
``ge`` / ``eq``) via a new module-level ``_SIGNS`` dispatch.
Folds the previous ``<=`` and ``==`` duplicates together and
fills in ``>=`` for each rule (which was the explicit gap).
The PyPSA #1683 test stays separate — it's tied to ``>=`` by
the real-world case it documents.
Suite: 7303 passed, 515 skipped, 0 failures under both semantics.
``mypy .`` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…to MI row (#855) The migration guide's fix table covered the core absence + alignment + aux-coord + MultiIndex-dim + groupby-result cases, but three issue-backed v1 changes were under-covered. Add rows and mirror two into the release-notes v1 summary: - Unlabeled operand (numpy / list / polars) pairs to dims by size; ambiguous or no-size-match raises (#736) — the object-scope premise. - A reordered/mismatched groupby grouper raises rather than reindexing positionally (#827). - MultiIndex-level projection: v1 rejects the MultiIndex *dimension* at construction, so the per-level-input case is folded into the existing MultiIndex row — decompose with reset_index, then project by the level aux coord. (The convention.md §11 example still shows the legacy-shaped `.get_level_values` snippet, which cannot run under v1 — flagged for the spec pass.) Release notes already carry the level-projection deprecation separately, so only #736 and the grouper point are mirrored there. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raduation (#856) The §11 example used `expr.indexes["snapshot"].get_level_values("period")`, which cannot run under v1: v1 rejects a first-class `pd.MultiIndex` dimension at construction, so `snapshot` is a flat dim with `period`/`timestep` as aux coords and `.indexes["snapshot"]` is a flat index, not a MultiIndex. Replace it with the working aux-coord mapping (verified on the branch) and add a one-line note on the flat representation. Also record in open-items.md that convention.md graduates into the rendered Sphinx docs at the 1.0 reframe (add myst-parser, toctree, internal cross-refs); until then it stays a design-record linked by URL — the rules are permanent but the transitional framing would only be re-done, so rendering now is wasted. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 20, 2026
…e) (#858) Positional indexing leaves a scalar coordinate behind: `x.isel(time=0)` drops `time` as a dimension but keeps it as a scalar coord (the first label). So a cyclic/boundary constraint like `x.isel(time=0) == x.isel(time=-1)` — extremely common in storage/energy models — hits §11's aux-coord conflict under v1 (`time` = first vs last) and raises, where legacy silently dropped it. The aux-coord row already covered this in principle, but listed only `.drop_vars` / `.assign_coords` as fixes and didn't name the scalar-leftover cause. Broaden the migration-guide row and add the natural fix — drop the coord at the indexing site with `.isel(..., drop=True)` / `.sel(..., drop=True)` — and add the same note to convention.md §11. Verified on the branch: `x.isel(time=0) == x.isel(time=-1)` raises "Auxiliary coordinate 'time' has conflicting values ... left=0, right=3" under v1; `drop=True` on both sides builds cleanly. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(v1): polish nan-scalar + duplicate-label merge tests (dedup, trim comments) * test(v1): polish the #717 convention test suite Applies the maintainer's polish guidelines across the v1/legacy semantics tests (test_legacy_violations.py, test_convention.py): - Assert FULL error/warning message text (hardcoded literals) instead of partial `match="..."` substrings, so every message change surfaces as a test diff. Covers all v1 ValueError raises and legacy LinopySemanticsWarning sites across the 17 convention classes. - DRY: dedupe repeated operand/coord setup into fixtures; merge near-identical tests via @pytest.mark.parametrize (per-operator NaN/mismatch cases, quadratic build paths, join modes, object-scope operand kinds, ...). - Trim essay-length docstrings/inline comments to one crisp line where the rationale isn't load-bearing. Coverage is preserved or expanded (500 vs 480 collected invocations); no tests removed. Suite green under both semantics; ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
@FabianHofmann I fixed all issues you found. We need to decide where to place the documentation of the convention. |
This was referenced Jul 21, 2026
Collaborator
you are a hero. this sounds good, let's do it. I added two tracker prs
I will gather feedback from those here. atm it looks like there is not big issues |
FBumann
added a commit
to flixOpt/flixopt
that referenced
this pull request
Jul 21, 2026
linopy's upcoming "v1" arithmetic convention (PyPSA/linopy#717) replaces legacy's silent coordinate realignment and NaN-fill with strict, explicit rules that raise on ambiguity. flixopt's constraint construction relied on the legacy behaviour in three places; this makes every site explicit so the model builds and solves identically under both `linopy.options["semantics"] = "legacy"` and `= "v1"`. Categories fixed: - Coordinate mismatch on a shared dim (adjacent-step constraints). Sites that combine `x.isel(dim=slice(1, None))` with `x.isel(dim=slice(None, -1))` carried different labels on the shared dim; legacy aligned by position, v1 raises. New `modeling._lead(var, dim)` relabels the leading slice onto the trailing slice's labels (linopy's recommended `.assign_coords` fix), which is identical under both semantics. Applied to the storage energy balance, on/off duration forward/backward/lb, state- and continuous-transition bounds, level tracking, and linked-period sizing. - Conflicting scalar auxiliary coordinate. Endpoint constraints like `x.isel(time=0) == x.isel(time=-1)` leave a scalar `time`/`scenario`/ `cluster_boundary` coord whose values disagree; legacy drops it silently, v1 raises. Added `drop=True` at each scalar-index site (cyclic/initial storage & status constraints, scenario-equality, intercluster linking, previous-state selection). - Deprecated `Constraint.lhs` setter. Replaced `con.lhs +=/-=` with `con.update(lhs=...)` (linopy 0.8 deprecation) at the transmission-loss, bus-balance, and share-allocation sites. The white-box `assert_conequal` tests in tests/deprecated reconstruct these constraints; updated their expected expressions to the same explicit form. Verified: full suite (tests/, excl. slow tutorials) is identical under both semantics — 1615 passed, same 4 pre-existing clustering failures (solution degeneracy in cyclic-cluster SOC level, unrelated to the semantics flag). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
|
@FBumann quick update, making good progress on the pypsa compatibility side. will continue tomorrow, but if it continues like that, I would say we should be able to merge this by the end of the week |
Collaborator
Author
|
I also updated flixOpt here for reference including a reference of transferring patterns |
…#861) Relocates the load-bearing v1 specs out of the untracked-by-Sphinx arithmetics-design/ folder into the rendered docs site: - Convert convention.md / goals.md / legacy-removal.md → reStructuredText under doc/design/ and wire them into a "Design & Internals" toctree in doc/index.rst. rst (not markdown) so no myst-parser dependency is needed. - Update every path reference: linopy/semantics.py and the five `LEGACY: remove at 1.0` comments in linopy/expressions.py, the test_legacy_violations.py header, and the arithmetics-design/ scaffolding files (open-items.md, docs-plan.md) that link to them. - release_notes.rst and migrating-to-v1.rst now cross-reference the rendered convention page via :doc: instead of a GitHub blob URL. The three process/scaffolding files (docs-plan.md, open-items.md, multiindex-feasibility.md) stay in arithmetics-design/ as internal notes. Docs build clean (sphinx-build exit 0, all three pages render, no new warnings); ruff + the convention test suite pass. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 24, 2026
FBumann
added a commit
to FBumann/lpspec
that referenced
this pull request
Jul 28, 2026
Adopts linopy's v1 reading of absence as the language's own, and makes v1 the
oracle rather than a mode we happen to survive.
**What changes.** A term whose variable is masked out no longer contributes
zero — it makes the row absent, so `x + y >= 10` is *no constraint* where `y`
is masked rather than `x >= 10`. The old reading is how
x - rel_max * size <= 0
silently became `x <= 0` on an unsized component: feasible model, plausible
answer, no error. That is goal 1 of the v1 convention ("no silent wrong
answers") and the whole of PyPSA/linopy#712, and it was reachable here.
**Two things deliberately do not propagate.** A *reduction* skips absent slots
(§13), so `sum(x, over=d)` stays defined when only some of `d` exists — without
that, one masked component would delete a system-wide accounting row. A
*parameter* covering only some coordinates is sparse encoding, not absence: its
missing rows mean a zero coefficient (SPEC §8), which is what lets a coefficient
table hold live entries only. Absence is a property of variables.
**How the engine tells them apart.** `TermFragment.presence` carries the
variable's own coordinates beside the term stream, because once `coeff x var`
are multiplied the frame cannot say which side removed a row. It is set only
for a variable whose declaration has a `where` — decided off the plan, before
data — so an unmasked variable never imposes the cost, and `_label_frame` keeps
both of its arithmetic paths (#152, #178) for every equation that does not turn
on the difference.
**The oracle is v1, and it raises rather than skips.** A skip would be the worst
outcome available: the suite would go green having stopped comparing the lanes
on exactly the cases the convention changed. No release carries the option yet,
so `[tool.uv.sources]` pins PyPSA/linopy#717 by branch — by branch and not by
rev on purpose, since a stale rev would measure us against a spec that has moved.
Not included, and it is the follow-up this needs: `defined(v)` (#219). Dropping
the row is now the only reading available, and the way to ask for the other one
is complementary `where` clauses over a variable's existence — which is not yet
sayable. Until it lands, a model wanting "keep the row, treat the term as zero"
has to carry a parameter mirroring the variable's mask.
`test_a_constraint_row_left_with_no_variables` stays xfailed and is *not* this:
raw linopy builds a term-less row under both conventions (`labels=[0,1]`,
`vars=[-1]`), so that divergence lives in our own eager lane and wants its own
diagnosis.
Refs #8, #219
FBumann
added a commit
to FBumann/lpspec
that referenced
this pull request
Jul 28, 2026
…the answer (#236) The oracle lane runs clean under `linopy.options["semantics"] = "v1"`, which issue #8 has been parked on since the tracked PR (PyPSA/linopy#591) was closed unmerged. The live work is PyPSA/linopy#717; its spec is `doc/design/ convention.rst` on `feat/arithmetic-convention`. Two things the convention changes for us, and one it turns out it cannot: **§8 never fires.** Its "shared dimensions must carry identical labels" exists because linopy operands are arbitrary xarray objects with independent indexes, so a mismatch is ambiguous between "different data" and "subset". We resolve a master coordinate per dimension before any data binds (SPEC §8) and reindex every operand to it, so a superset already raises, a subset is filled and a reorder is fixed by label. Measured: four orderings of `a + factor + b` with a sparse factor and a masked variable produce byte-identical rows, so the associativity break v1 §8 exists to stop (PyPSA/linopy#711 — chained left-joins dropping coordinates) has no mechanism here. **§5 does fire, on our central idiom.** `load_parameters` reindexes to the master coordinates, so a parameter covering a subset of its dims arrives at linopy as NaN — and v1 refuses a NaN in a user-supplied constant, because from inside linopy a deliberate absence and a data error are indistinguishable. That is every sparse parameter, which is what SPEC §8 means by "sparse data gives sparse variables" and what a generated binder emits by the tableful. The 414 tests passing beforehand were not evidence otherwise; the suite had no sparse float parameter. It has one now, and it fails without this change. **So the answers are given per position, not once in the loader.** The same missing row means three different things: zero in a coefficient, an error in `bounds:` (unbounded is not bounded-at-zero), false in a `where` (SPEC §6's bare name is "non-null and finite"). A single fill in the loader would have to pick one and be wrong for the other two. `linopy/semantics.py` is where those answers live, mirroring linopy's own module of that name and for its reason: the evaluator stays about evaluating, and a later change to the convention is a single-file diff. All three functions are correct under the legacy convention too — they resolve absence to the value legacy reached implicitly — so none is conditional on the option. What v1 changed is that staying silent stopped being available. Verified on three configurations: released linopy 0.9.0 (415 passed), the v1 branch under `semantics="v1"` (415 passed), and the same branch under `semantics="legacy"` (413 passed, 2 failing only on `LinopySemanticsWarning` promoted to an error by our own filterwarnings). Refs #8
FBumann
added a commit
to FBumann/lpspec
that referenced
this pull request
Jul 28, 2026
…ow (#234) * feat(compat): answer linopy's v1 convention where the position knows the answer The oracle lane runs clean under `linopy.options["semantics"] = "v1"`, which issue #8 has been parked on since the tracked PR (PyPSA/linopy#591) was closed unmerged. The live work is PyPSA/linopy#717; its spec is `doc/design/ convention.rst` on `feat/arithmetic-convention`. Two things the convention changes for us, and one it turns out it cannot: **§8 never fires.** Its "shared dimensions must carry identical labels" exists because linopy operands are arbitrary xarray objects with independent indexes, so a mismatch is ambiguous between "different data" and "subset". We resolve a master coordinate per dimension before any data binds (SPEC §8) and reindex every operand to it, so a superset already raises, a subset is filled and a reorder is fixed by label. Measured: four orderings of `a + factor + b` with a sparse factor and a masked variable produce byte-identical rows, so the associativity break v1 §8 exists to stop (PyPSA/linopy#711 — chained left-joins dropping coordinates) has no mechanism here. **§5 does fire, on our central idiom.** `load_parameters` reindexes to the master coordinates, so a parameter covering a subset of its dims arrives at linopy as NaN — and v1 refuses a NaN in a user-supplied constant, because from inside linopy a deliberate absence and a data error are indistinguishable. That is every sparse parameter, which is what SPEC §8 means by "sparse data gives sparse variables" and what a generated binder emits by the tableful. The 414 tests passing beforehand were not evidence otherwise; the suite had no sparse float parameter. It has one now, and it fails without this change. **So the answers are given per position, not once in the loader.** The same missing row means three different things: zero in a coefficient, an error in `bounds:` (unbounded is not bounded-at-zero), false in a `where` (SPEC §6's bare name is "non-null and finite"). A single fill in the loader would have to pick one and be wrong for the other two. `linopy/semantics.py` is where those answers live, mirroring linopy's own module of that name and for its reason: the evaluator stays about evaluating, and a later change to the convention is a single-file diff. All three functions are correct under the legacy convention too — they resolve absence to the value legacy reached implicitly — so none is conditional on the option. What v1 changed is that staying silent stopped being available. Verified on three configurations: released linopy 0.9.0 (415 passed), the v1 branch under `semantics="v1"` (415 passed), and the same branch under `semantics="legacy"` (413 passed, 2 failing only on `LinopySemanticsWarning` promoted to an error by our own filterwarnings). Refs #8 * refactor(engine): a fragment loses rows for two reasons; carry them apart Prerequisite for adopting v1's absence semantics, and inert on its own — no behaviour changes, 415 tests unmoved. A term fragment can lose rows at a coordinate for two unrelated reasons, and a constraint row has to react to exactly one of them. A **masked variable** is genuinely absent there. A **sparse parameter** is a compressed dense array whose missing rows mean a zero coefficient — SPEC §8's "sparse data gives sparse variables", and what a generated binder emits by the tableful. Once the two are multiplied into one frame the distinction is gone, which is why "drop the row where a variable is absent" could not be written as an anti-join against the term stream. So the variable's own coordinates ride alongside as `TermFragment.presence`, rewritten by the same shape operators that rewrite the term. The propagation rule follows from the convention rather than from convenience: variable leaf the variable's frame parameter, constant none — sparsity is an encoding, not absence a * b, a / b, -a the variable side's, unchanged: a sparse coefficient zeroes a term, it does not unmake the variable under it sum, group_sum cleared — §13 has reductions *skip* absent slots rather than propagate them, which is what keeps a cross-module accounting equation summing over a partly-masked dim roll, shift still to do: the coordinate map has to be applied to presence too, and shift's vacated edge unioned back in, since SPEC §7 declares it contributes zero Refs #8 * feat!: absence propagates and drops the row, on both lanes Adopts linopy's v1 reading of absence as the language's own, and makes v1 the oracle rather than a mode we happen to survive. **What changes.** A term whose variable is masked out no longer contributes zero — it makes the row absent, so `x + y >= 10` is *no constraint* where `y` is masked rather than `x >= 10`. The old reading is how x - rel_max * size <= 0 silently became `x <= 0` on an unsized component: feasible model, plausible answer, no error. That is goal 1 of the v1 convention ("no silent wrong answers") and the whole of PyPSA/linopy#712, and it was reachable here. **Two things deliberately do not propagate.** A *reduction* skips absent slots (§13), so `sum(x, over=d)` stays defined when only some of `d` exists — without that, one masked component would delete a system-wide accounting row. A *parameter* covering only some coordinates is sparse encoding, not absence: its missing rows mean a zero coefficient (SPEC §8), which is what lets a coefficient table hold live entries only. Absence is a property of variables. **How the engine tells them apart.** `TermFragment.presence` carries the variable's own coordinates beside the term stream, because once `coeff x var` are multiplied the frame cannot say which side removed a row. It is set only for a variable whose declaration has a `where` — decided off the plan, before data — so an unmasked variable never imposes the cost, and `_label_frame` keeps both of its arithmetic paths (#152, #178) for every equation that does not turn on the difference. **The oracle is v1, and it raises rather than skips.** A skip would be the worst outcome available: the suite would go green having stopped comparing the lanes on exactly the cases the convention changed. No release carries the option yet, so `[tool.uv.sources]` pins PyPSA/linopy#717 by branch — by branch and not by rev on purpose, since a stale rev would measure us against a spec that has moved. Not included, and it is the follow-up this needs: `defined(v)` (#219). Dropping the row is now the only reading available, and the way to ask for the other one is complementary `where` clauses over a variable's existence — which is not yet sayable. Until it lands, a model wanting "keep the row, treat the term as zero" has to carry a parameter mirroring the variable's mask. `test_a_constraint_row_left_with_no_variables` stays xfailed and is *not* this: raw linopy builds a term-less row under both conventions (`labels=[0,1]`, `vars=[-1]`), so that divergence lives in our own eager lane and wants its own diagnosis. Refs #8, #219 * feat: a bare variable name in a where asks whether it exists The escape hatch the previous commit made necessary. Absence now takes the row with it, so a model wanting the other reading — keep the row, treat the term as zero — needs a way to name those coordinates. Without it the only spelling is a parameter mirroring the variable's own mask: two sources for one fact, and when they drift the failure is not an error but a row quietly pinned to zero. Surface is the one the language already implies rather than a new one. A bare *parameter* name in a where asks "does this have a value here"; a bare *variable* name asks "does this exist here". Same grammar, same shape, and resolution already had a branch for the case — it just raised. - expression: x - rel_max * size <= 0 where: "size" - expression: x <= 0 where: "NOT size" Pointwise on both lanes: a semi-join against the variable's frame relationally, `labels != -1` eagerly. `_predicate_dims` reads it through the variable's `foreach` exactly as a parameter is read through its dims, so `_free_prefix` keeps its arithmetic path for the leading dims a mask cannot see. **A self-reference is now a load error**, found by the parity sweep, which masks `variables.p.where` and so asked `p` whether `p` exists. It died in linopy with `KeyError: 'p'` — nothing in this package's voice. A variable's own where cannot ask whether it exists, because that mask is what decides it. Two limits, both deliberate and both visible in the tests. The dim rule refuses a variable whose foreach exceeds the frame — masking `balance` over `[snapshot]` by `p` over `[snapshot, generator]` would silently widen the mask, and saying which reduction is meant needs ROADMAP Track 1 item 6 (`all(x, over=d)`). And the predicate has no home in the dispatch-model parity sweep for that same reason, so `COVERED_ELSEWHERE` maps it to the test that does exercise it rather than letting the coverage guard be quietly weakened. Closes #219. Refs #8
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The strict v1 semantic convention for linopy — predictable coordinate alignment and NaN handling.
This is the master PR for the new semantic convention in linopy. It starts with our
Design & transitioning goals, which is carried out in ourNew Semantics spec. Both files are tracked in this branch. WHat you read is the current state.Both might change until this PR is merged
The concrete rollout checklist — the three stages (opt-in → default → 1.0) and their per-stage status — is tracked in
arithmetics-design/open-items.md.Scope
The convention ships behind
linopy.options["semantics"]—v1opt-in,legacythe default. This PR carries the design, spec, tests and implementation; documentation notebooks follow separately.Testing
All tests in linopy will be executed for both semantics.
Differing behaviour will be tested using
pytest.markers.This will increase ci time temporarily until v1 is released.
Defered to a follow up PR
Docs: Documentation, Migration guide, Release notes
Note
Issue links below added by AI. These are the milestone issues the v1
convention fixes; they auto-close when this PR merges. #703 (extend xarray
method coverage) stays open — it is a deferred follow-up (§13), not fixed here.
Closes #714
Closes #713
Closes #712
Closes #711
Closes #708
Closes #670
Closes #586
Closes #570
Closes #295
Closes #257
Closes #736
Closes #256