Repository navigation
feat(zarr-metadata)!: read what zarr-python writes in a ZARR_PYTHON_CONTEXT scope - #4583
Merged
d-v-b merged 13 commits intoNov 1, 2026
Merged
Conversation
…ONTEXT scope Add definitions for the extensions zarr-python writes. zarr-extensions registers `fixed_length_utf32` and the `vlen-utf8` and `vlen-bytes` codecs, so they join CORE_AND_EXTENSIONS_CONTEXT. zarr-python's own `null_terminated_bytes` and `variable_length_bytes`, which no specification registers, are only in the new ZARR_PYTHON_CONTEXT, so the default scope stays spec-pure. Rename CORE to CORE_CONTEXT and CORE_AND_EXTENSIONS to CORE_AND_EXTENSIONS_CONTEXT, so that a scope's name says it is a context. Fixes #423 Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
github-actions
Bot
added
needs release notes
Documentation build overview
13 files changed ·
|
Resolve the conflict in test_pipelines.py: keep the unclaimed `acme.vlen` codec, at the codec-level location zarr-developers#4574 gives a codec's chunk problems. Rename CORE to CORE_CONTEXT in main's new test_rule_locations.py, and move the expectations in test_zarr_python_scope.py to zarr-developers#4574's locations: a codec's chunk problem at the codec, a missing key at the object that lacks it. Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
…ules Keep `CORE` and `CORE_AND_EXTENSIONS` importable from `zarr_metadata.v3` and `zarr_metadata.v3.definition` as the same scopes with a DeprecationWarning, so a consumer such as pydantic-zarr keeps working across the rename. Check that `vlen-utf8` encodes `string` and `vlen-bytes` encodes `bytes` (or zarr-python's `variable_length_bytes`), as zarr-extensions says, instead of accepting any data type whose values vary in size. Do not count trailing null bytes against the size of a `null_terminated_bytes` fill value, since they are padding. Spell a `fixed_length_utf32` fill value without its trailing U+0000 padding, so `"a"` and `"a\u0000"` have one canonical spelling. Assisted-by: ClaudeCode:claude-opus-5-5
Add zarr-python's `raw_bytes` data type (NumPy's `V`) to ZARR_PYTHON_CONTEXT, so the scope reads every data type zarr-python writes. Its names start with `ZarrPython`, since `RawBytes` already names the spec's raw bits. A fill value longer than `length_bytes` is a problem, because zarr-python cuts it short, and the canonical spelling pads it with null bytes, as zarr-python writes it. Rename CORE_V2 to CORE_V2_CONTEXT, so every scope's name ends in `_CONTEXT`. The old name keeps working with a DeprecationWarning, through the same `renamed_scope` helper as the v3 names, which moves to `zarr_metadata._renamed`. Share the base64 decoding and spelling of fill values in `_byte.py`, and say that the vlen codecs' `configuration` key is optional, since zarr-python writes an empty object. Assisted-by: ClaudeCode:claude-opus-5-5
Move CORE_V2_CONTEXT and its table of old names into `v2/_core.py`, as `v3/_core.py` holds the v3 scopes, so `zarr_metadata.v2` and `zarr_metadata.v2.definition` share one table. Say why a `fixed_length_utf32` fill value counts its trailing U+0000 code points, as the spec requires, while `null_terminated_bytes` follows zarr-python and does not, and test a value that only its padding makes too long. Assisted-by: ClaudeCode:claude-opus-5-5
d-v-b
marked this pull request as ready for review
October 31, 2026 22:08
Contributor
Author
|
internal to zarr-metadata and so self-merging |
… definitions The Hypothesis strategies merged from main (zarr-developers#4575) imported `CORE`, `CORE_AND_EXTENSIONS` and `CORE_V2` from modules that now hold only the `_CONTEXT` names, and their coverage test requires a strategy for every definition in `CORE_AND_EXTENSIONS_CONTEXT`. Rename the imports, and add strategies for `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes`; each vlen codec draws nothing for a data type it does not read. Assisted-by: ClaudeCode:claude-opus-5-5
…strategies `DEFAULT_STRATEGIES` holds a strategy for each definition the package defines, but `null_terminated_bytes`, `variable_length_bytes` and `raw_bytes` had none, so `fields()` raised `LookupError` for them. Draw each, and check the strategies against `ZARR_PYTHON_CONTEXT`, which holds every definition. Say that the documents in test_zarr_python_scope.py leave out the compressors zarr-python adds. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…IONS_CONTEXT The strategies are now checked against `ZARR_PYTHON_CONTEXT`, which no longer proves that `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes` are in `CORE_AND_EXTENSIONS_CONTEXT`. Assert it directly. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each vlen codec strategy named its compatible data types a second time; draw nothing for exactly the chunks the codec's own chunk rules refuse, so the two cannot drift. Draw `fixed_length_utf32` fill values with trailing U+0000 padding too, which the canonical spelling removes. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 2, 2026
Resolve the rename against the four zarr-metadata changes main gained: the ZARR_PYTHON_CONTEXT scope and the *_CONTEXT constant names (zarr-developers#4583), the v3 submodule re-exports (zarr-developers#4588), the Hypothesis strategies (zarr-developers#4575) and the docstrings for every public object (zarr-developers#4590). Every conflict was the rename on one side and new upstream text on the other, so each is resolved as the upstream text with the rename applied. The rename is carried into upstream's additions: the new docstrings say "field" for a JSON key and "declaration" for an extension-point value, and the strategies module's `fields` strategy is `declarations`, with `FieldForm` now `DeclarationForm` in its tests. The zarr-developers#4575 changelog fragment uses the new name, since neither name has been released. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 2, 2026
This branch was squash-merged upstream as zarr-developers#4575 (4a72dd4), whose tree matches this branch's head. Each conflict is main's later change to the same files (zarr-developers#4583, zarr-developers#4588, zarr-developers#4590, zarr-developers#4592), so every file takes main's side, and the branch now matches main. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 2, 2026
Main already holds this branch as zarr-developers#4583 (bc177b3, whose tree is this branch's tip), followed by later work such as the declaration rename in zarr-developers#4592. Every conflict therefore resolves to main's side, and the merged tree is main's. Assisted-by: ClaudeCode:claude-opus-5-5
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 2, 2026
…adata-nested-strategies The base now matches main, which carries zarr-developers#4575 and main's later zarr-metadata changes. zarr-developers#4592 renames an extension-point value to a declaration: `fields` is `declarations`, `data_type_fields` is `data_type_declarations`, `AcceptedField` and its siblings are `*Declaration`, and the contexts are `CORE_CONTEXT`, `CORE_AND_EXTENSIONS_CONTEXT` and `ZARR_PYTHON_CONTEXT`; this branch's code and prose follow, where struct fields keep their name. zarr-developers#4583's strategies for zarr-python's data types and the vlen codecs take `Strategies` last, as every record function here does, so strings and variable-length bytes now get pipelines too. The struct test derives which data types vary in size from their storage, and the tests read in `ZARR_PYTHON_CONTEXT`, which holds every definition the strategies draw. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
zarr-metadatatoday only ships two contexts (sets of data types, codecs, etc that are in-scope for metadata resolution). They areCOREandCORE_AND_EXTENSIONS, neither of which hold all the data types / codecs that zarr-python uses. This PR adds such a context, and also appends the word_CONTEXTto these contexts to make it clear that they are contexts.Author attestation
TODO
docs/user-guide/*.mdchanges/