From f8aaa3eeae18437372874127f3c250c11f957064 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 11:53:55 +0000 Subject: [PATCH 01/56] Start the PR 4 plan: header, constraints, scope and findings Zarr loading. Sections through "Findings that refine the spec", which records seventeen points checked against zarr-python 3.4.1 while the plan was written -- including the answer to the spec's open question Q2, and three places the spec needs amending. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 358 ++++++++++++++++++ 1 file changed, 358 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-13-zarr-io-pr4.md diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md new file mode 100644 index 0000000000..804478a2b6 --- /dev/null +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -0,0 +1,358 @@ +# PR 4 — Zarr loading + +> **For agentic workers:** REQUIRED SUB-SKILL: Use +> `superpowers:subagent-driven-development` (recommended) or +> `superpowers:executing-plans` to implement this plan task-by-task. Steps use +> checkbox (`- [ ]`) syntax for tracking. + +> **This plan ships with its implementation.** Plan and code are delivered on +> the same branch and in the same pull request. If implementation proves a step +> wrong, correct the step here on the branch — the document is frozen when the +> work merges, not while it is under way. Programme status lives in §12 of the +> [design spec](../specs/2026-09-21-zarr-io-design.md); nothing here duplicates +> it. + +| | | +|---|---| +| **Spec** | [`../specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md) §2.4, §3, §4.1, §4.2, §4.4, §4.6, §4.7, §5 (PR 4), §6 | +| **Implements** | The fourth of the eight pull requests in spec §5. The subject of [#6979](https://github.com/SciTools/iris/issues/6979) | +| **Branch** | `zarr-io-pr-4`, targeting `SciTools/iris:brownfield` | +| **Builds on** | **PR 3** ([#7316](https://github.com/SciTools/iris/pull/7316)), which must merge first: this plan is written against `iris.fileformats.cf.loader`, `CFDatasetVariable.read_data` and the `cache_key=` keyword on `as_lazy_data`, none of which exist on `brownfield` yet | +| **Baseline** | The spec branch at `be3cc7965`, Iris `3.17.0.dev`, zarr-python `3.4.1`, dask `2025.x`, NumPy `2.5.2` | +| **Delivers** | `iris.fileformats.zarr` — `_dataset.py`, `_decode.py`, `loader.py` — the `"Zarr"` format specification, `iris.io.load_stores`, and `iris.load("store.zarr")` working for version 2 and version 3 stores, local and remote | +| **Behaviour change** | Three, all previously errors: a local `.zarr` directory and an `s3`/`gs`/`az`/`abfs` URL now load instead of raising; `CFVariable.spans` matches on `(name, size)` pairs rather than bare names, which is a widening no netCDF file can observe | +| **Public API change** | Additive: the new public module `iris.fileformats.zarr` (`load_cubes`), `iris.io.load_stores`, `CFDataset.format`, `CFDataset.encoding` and `CFDatasetVariable.encoding` | + +**Created:** 2026-10-13 + +Claims marked **[verified]** were checked by running code in the `iris-dev` +environment against zarr-python 3.4.1 while this plan was written; everything +else is read from source or from the spec. + +**Goal:** Make `iris.load("store.zarr")` and +`iris.load("s3://bucket/store.zarr")` return cubes from a native Zarr version 2 +or version 3 store, by implementing the `CFDataset` interface over zarr-python +and teaching Iris's load chain that a target can be a directory or a URL rather +than a seekable file. + +**Architecture:** `iris/fileformats/zarr/_dataset.py` implements `CFDataset` and +`CFDatasetVariable` over a `zarr.Group` and its `zarr.Array`s, which is all the +CF layer from PR 2 and PR 3 needs in order to classify variables and build +cubes. There is no data proxy: `zarr.Array` is picklable, so `read_data` hands +it straight to `as_lazy_data` and adds CF decoding — masking, `valid_range`, +`scale_factor`/`add_offset` — as explicit `map_blocks` layers over the raw +array, implemented as pure functions in `_decode.py`. `iris/fileformats/zarr/ +loader.py` is a thin `load_cubes` that opens a `ZarrDataset`, hands it to +`CFReader`, and runs the same `_load_cube` loop the netCDF loader runs. +`iris.io.load_stores` is the new entry point for a target that is not a file, +and a `ZarrStore` file element lets `FORMAT_AGENT` recognise one. + +**Tech Stack:** Python 3.12–3.14, zarr-python ≥3.0.8, NumPy, Dask, pytest + +pytest-xdist, pre-commit (ruff, numpydoc, mypy, sort-all), `abc.ABC`, +`collections.abc.MutableMapping`. + +**Spec:** [`../specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md) +— §2.4 lists the Zarr facts that shape the design, §3 the constraints agreed +before design, §4.1 the module layout, §4.2 the `CFDataset` interface, §4.4 the +whole of reading, §4.6 groups, §4.7 format detection and the load chain, §5's +fourth entry is this pull request, §6 the testing rules. The plan argues from +the spec; read both. + +--- + +## Global Constraints + +Project-wide requirements every task below implicitly carries. Values are +copied from the spec and the repository's agent rules, not restated from +memory. + +| Constraint | Value | Source | +|---|---|---| +| Target branch | `brownfield`. **Not** `main` | spec §5 | +| Labels | `Agentic` and `Type: Feature Branch` | spec §5 | +| Issue linkage | `Part of #6979` — never a closing keyword; those live on the merge-back | spec §5 | +| Changelog | Fragments land with the merge-back; the pull request body must say the omission is deliberate | spec §5 | +| Dependency | `zarr` is **optional** and lazily imported. "Absent, Zarr tests skip and Zarr use raises a clear `ImportError`" | spec §3 | +| Version pin | `zarr >=3.0.8`, "which is above the known data-loss bug" | spec §8 | +| Read support | **Version 2 and version 3** | spec §3 | +| Write support | Not here. The `ZarrDataset` write surface is PR 6 | spec §5 | +| Remote stores | "URLs are passed through to zarr-python. Iris adds no dependency and handles no credentials" — no fsspec, s3fs or gcsfs in Iris's requirements | spec §3, §9 | +| Groups | "Root group by default; a `group=` keyword selects another"; Iris loads **one** group per call and does not walk nested groups | spec §3, §4.6 | +| No implementation-type tests | "No code above `cf/dataset.py` tests the implementation type. No `isinstance`, no checking a class name, no `hasattr` probe standing in for one" | spec §4.2 | +| `format` is not a branch key | "Where the generic layer needs a capability, it asks for the capability, not the format. `format` is for messages, diagnostics and the user" | spec §4.2 | +| Dimension contract | A dimension name may be absent, null, repeated within an array, or shared across arrays at different lengths. "All four are conforming Zarr. None may raise" | spec §4.4 | +| Masking precedence | Version 3: CF attributes only. Version 2: `Array.fill_value` when no CF attribute is present | spec §4.4 | +| Decode layers | "Decoding is applied as **explicit dask graph layers** over the raw array … not hidden inside an opaque `__getitem__`" | spec §4.4 | +| No `ZarrDataProxy` | "Do not mirror `NetCDFDataProxy` on the Zarr side" | spec §4.4 | +| Cache key | `(store location, array path, json.dumps(Array.metadata.to_dict(), sort_keys=True))`, treated as opaque | spec §4.4 | +| Read unit | `chunking` returns `Array.chunks`, never `Array.shards`; alignment aggregates **upward** only | spec §4.4 | +| Format detection | Local: a directory holding `zarr.json` or `.zgroup`. Remote: scheme in `s3`, `gs`, `az`, `abfs` **and** the path ends `.zarr`. "**Do not add a third rule matching any path that ends in `.zarr`**" | spec §4.7 | +| `load_files` | "is not taught about directories" — a separate `iris.io.load_stores` answers "this target is not a file", once, for local and remote alike | spec §4.7 | +| NCZarr | Unaffected. `_uri_is_nczarr` is left exactly as it is | spec §4.7, §9 | +| Attribute policy | A nested JSON object or array is carried onto the cube unchanged and not interpreted | spec §4.4 | +| Tests | "pytest style, no network, no unittest classes in new files" | spec §6 | +| Skipping | "A single `pytest.importorskip(\"zarr\")` fixture in the Zarr test package. Iris without zarr installed must show no new failures" | spec §6 | +| Remote tests | "covered by an in-memory store and a local directory store only. No test touches the network" | spec §6 | +| Every commit | `pre-commit run --files ` then `pytest`, both **before** the commit. Never `--no-verify` | `AGENTS.md` | +| Test tiers | `pytest lib/iris/tests/unit//` per edit (no `-n auto` on small runs); `-n auto` only for parent-directory and full-suite runs | `lib/iris/tests/AGENTS.md` | +| Test data | `OVERRIDE_TEST_DATA_REPOSITORY` must be set **and verified** before trusting any run; read the skip count, not just the failures | `lib/iris/tests/AGENTS.md` | +| Line length | 88 characters, Ruff default | `AGENTS.md` | +| Docstrings | NumPy style, numpydoc-validated. Module docstring mandatory outside tests (D100); a public module needs a `z_reference` item in its first 25 lines | `AGENTS.md`, `docs/AGENTS.md` | +| New `.py` files | Four-line copyright header, checked by `check-license-headers` | `AGENTS.md` | +| netCDF access | No direct `import netCDF4` — always `iris.fileformats.netcdf._thread_safe_nc` | `AGENTS.md` | +| New modules | Aim under ~1000 lines; cohesion wins over line count | `lib/iris/AGENTS.md` | +| Warning classes | Every `UserWarning` subclass descends from `IrisUserWarning`; load-time CF complaints use `IrisCfLoadWarning`, chunking complaints `IrisLoadWarning` | `AGENTS.md`, `iris.warnings` | +| Prose | No citation of a plan number, task number or phase in code, tests or docs. Cite an issue, a pull request, the design document, or the convention's section | `lib/iris/tests/AGENTS.md`, `docs/AGENTS.md` | +| Attribution | Commits end `Co-Authored-By: Claude Opus 5 `; every GitHub post opens `(posted by Claude Opus)` | `AGENTS.md` | + +--- + +## Review Focus + +Five failure modes the spec implies that **no task's own tests would otherwise +exercise**, most likely first. Each names the input or condition and the +behaviour a reasonable person expects; each has a test added to the task that +owns the code, written out in that task's own step style. + +1. **A version 2 store's `_ARRAY_DIMENSIONS` attribute landing on the loaded + cube.** It is how a version 2 store records dimension names, so under the + §4.2 split it is a *property*, not a CF attribute — and every attribute a + `CFVariable` does not consume reaches `cube.attributes` + (`cf/loader.py::_add_unused_attributes`). Nothing in `_CF_ATTRS` excludes + it, so unless `ZarrDatasetVariable.attributes` withholds it, every cube + loaded from a version 2 store gains a stray `_ARRAY_DIMENSIONS` attribute + and fails to compare equal to the same cube loaded from version 3. The user + expects the two to be indistinguishable. → **Task 3, Steps 9–10**. + +2. **A store whose root group holds no arrays.** Spec §4.6 requires "a message + that names the groups that do", and §7 records a real store in that shape — + the ESA EOPF Sentinel-2 product, whose root holds only subgroups. The + failure to defend against is not the message but its absence: an empty + `CubeList` returned silently from `iris.load` of a store that plainly + contains data is the worst available answer, and the one that falls out of + writing the loop without thinking about it. → **Task 9, Steps 7–8**. + +3. **`iris.load("data/*")` matching a mix of `.nc` files and `.zarr` + directories.** Spec §4.7 states the requirement and names the consequence — + the file-or-store decision has to happen after glob expansion and before + format dispatch. The quiet failure is partial: `_generate_cubes` partitions + what `expand_filespecs` returns, so a bug that mis-partitions raises + `IsADirectoryError` from `load_files` for the stores while the netCDF files + load fine, and a user who did not expect the glob to match a store reads it + as an Iris bug. → **Task 11, Steps 11–12**. + +4. **A `scale_factor` or `add_offset` that changes the result dtype.** + `cf/loader.py::_get_actual_dtype` computes the post-unpacking dtype and the + netCDF path gets the arithmetic itself from netCDF4. On the Zarr path Iris + does the arithmetic, in `_decode.py`, so the dask array's declared dtype and + what the decode layer actually produces are two separately-written + statements that must agree. When they disagree dask does not complain: the + graph is built, and the mismatch surfaces later as a wrong dtype on the + cube, or as a `ValueError` from a downstream operation. A `uint16` band with + `scale_factor = 0.0001` — the EOPF reflectance case in spec §7 — must give a + `float64` cube whose lazy and realised dtypes match. → **Task 6, Steps 11–12**. + +5. **A `zarr` that is installed but too old, or not installed at all.** Spec §3 + requires "a clear `ImportError`" and §6 requires that "Iris without zarr + installed must show no new failures". Two distinct paths reach a user here — + `iris.load` of a `.zarr` target, and `import iris.fileformats.zarr` — and the + second is the one that breaks the first rule by accident, because + `iris.fileformats.__init__` imports every format module it registers. If the + Zarr import is not deferred, installing Iris without zarr stops `import iris` + working at all. → **Task 1, Steps 5–8**. + +--- + +## 1. Scope boundary + +**In.** + +- `lib/iris/fileformats/zarr/` — `__init__.py`, `_dataset.py`, `_decode.py`, + `loader.py`. +- `CFDataset.format` / `CFDatasetVariable.format`, and `CFDataset.encoding` / + `CFDatasetVariable.encoding`, on the interface and on both implementations. +- `CFVariable.spans` and its three overrides matching on `(name, size)` pairs. +- `CFReader` accepting an already-open `CFDataset`. +- The `"Zarr"` `FormatSpecification` and its `ZarrStore` file element. +- The four load-chain changes of spec §4.7, including `iris.io.load_stores`. +- `zarr` added to the optional-dependency section of + `requirements/py3{12,13,14}.yml`, and the lock files regenerated once. +- User-guide and API-reference documentation for Zarr loading, including the + remote URL forms and `group=`; and the S3 page rewritten. +- Unit and integration tests for all of the above. + +**Out.** + +- **Saving.** No `zarr/saver.py`, no `ZarrDataset` write path, no + `consolidate_metadata`, no registration in `_savers`. `ZarrDataset`'s + write-surface members exist because the ABC declares them and raise + `NotImplementedError`. That is PR 6. +- **The write half of the shared `CFDataset` contract.** It cannot run against + a dataset that refuses to create variables; §2 F10 explains the split. +- **Real-world fixtures and benchmarks.** The NOAA GFS and ESA EOPF samples, + and the ASV coverage, are PR 7. Every fixture here is synthetic, built in + `tmp_path` or in memory. +- **xarray as a test dependency.** Spec §6 lands it in PR 6, with the saver it + validates. Nothing here needs a second reader. +- **The netCDF read path's own sub-chunk re-read trap** (spec §12.3 Q4). The + alignment guard is added on the Zarr path only. +- **UGRID meshes from a Zarr store.** `iris.mesh.load_meshes` gains the ability + to *reach* a Zarr store (spec §4.7), because its scheme dispatch is a copy of + `loading.py`'s and the two move together. Nothing here builds or tests a mesh + from Zarr. +- **Changing the NCZarr path**, beyond leaving it alone. + +--- + +## 2. Findings that refine the spec + +Checked while writing this plan. Each is either a correction the implementer +would otherwise hit as a failing test, or an answer the spec asked for. + +**F1 — `zarr.Array` has no `dimension_names` property.** Version 3 names live +at `Array.metadata.dimension_names`; `ArrayV2Metadata` has no such attribute +and raises `AttributeError`, not the `ValueError` spec §2.4 quotes +**[verified]**. So `ZarrDatasetVariable.dimensions` must branch on +`metadata.zarr_format` and read `attrs["_ARRAY_DIMENSIONS"]` on version 2 — +never `try: … except ValueError`. + +**F2 — zarr-python rejects a wrong-length `dimension_names` at creation.** +`create_array(..., shape=(2, 2), dimension_names=("y",))` raises +``ValueError: `dimension_names` and `shape` need to have the same number of +dimensions`` **[verified]**. Spec §4.4 requires Iris to warn and load +anonymously for such a store, so the fixture for that test cannot be built +through the zarr-python API — it has to write `zarr.json` by hand. + +**F3 — spec §12.3 Q2 is answered: ` From 2149b9598d5957282a344220b1774dcf9fe3eee7 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 11:55:08 +0000 Subject: [PATCH 02/56] Add the PR 4 plan's file structure and interface design Every file created or modified, the order the tasks run in and why, and the signatures of ZarrDataset, ZarrDatasetVariable, the decode functions and the load-chain additions -- so that each task below can be read on its own without guessing a neighbour's names. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 295 ++++++++++++++++++ 1 file changed, 295 insertions(+) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index 804478a2b6..f75dbe1175 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -354,5 +354,300 @@ unnoticed. but the comment naming netCDF4 does, and `valid_min`/`valid_max`/`valid_range` are deliberately **not** in that set, so they reach the cube on both paths. +--- +## 3. File Structure + +### Created + +| Path | Responsibility | Rough size | +|---|---|---| +| `lib/iris/fileformats/zarr/__init__.py` | Public surface: `load_cubes`, re-exported, plus the package docstring that says `import zarr` inside this package still resolves to the third-party distribution (spec §4.1) | ~60 | +| `lib/iris/fileformats/zarr/_dataset.py` | `ZarrDataset`, `ZarrDatasetVariable`, `_ZarrAttributes`, `_align_chunks`. The only module that knows both the CF interface and the zarr-python API | ~550 | +| `lib/iris/fileformats/zarr/_decode.py` | The CF rules netCDF4 applies for free: missing-value masking, `valid_range` masking, `scale_factor`/`add_offset` unpacking, and base64 `_FillValue` decoding. Pure functions of `(block, parameters)` | ~250 | +| `lib/iris/fileformats/zarr/loader.py` | `load_cubes(file_sources, callback=None, constraints=None, group=None)` | ~180 | +| `lib/iris/tests/unit/fileformats/zarr/__init__.py` | The package-wide `pytest.importorskip("zarr")` and the store-building fixtures | ~120 | +| `lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py` | Empty package marker | 5 | +| `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py` | The shared read contract run against Zarr, plus `ZarrDataset`'s own cases | ~280 | +| `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py` | The dimension contract, `fill_value`, `chunking`, `encoding`, attributes | ~320 | +| `lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py` | The alignment guard, on `chunking` values alone | ~140 | +| `lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py` | Laziness, the cache key's staleness and sensitivity cases, the sharded read unit | ~300 | +| `lib/iris/tests/unit/fileformats/zarr/test__decode.py` | Each decode rule on plain ndarrays, and `_FillValue` in both encodings | ~280 | +| `lib/iris/tests/unit/fileformats/zarr/loader/__init__.py` | Empty package marker | 5 | +| `lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py` | One group per call, the empty-root message, `group=` | ~200 | +| `lib/iris/tests/unit/fileformats/cf/test_spans.py` | The `(name, size)` suite, run against both backends | ~260 | +| `lib/iris/tests/unit/io/test_load_stores.py` | Partitioning, scheme routing, and what `load_stores` refuses | ~180 | +| `lib/iris/tests/unit/io/format_picker/test_ZarrStore.py` | The local marker-file test and the remote scheme test, and that neither swallows the other | ~150 | +| `lib/iris/tests/integration/zarr/__init__.py` | Empty package marker | 5 | +| `lib/iris/tests/integration/zarr/test_load.py` | `iris.load` end to end, version 2 and version 3, including an NCZarr-written version 2 store | ~260 | +| `docs/src/user_manual/howto/zarr_io.rst` | How to load a Zarr store, local and remote, with `group=` | ~160 | + +### Modified + +| Path | Change | +|---|---| +| `lib/iris/fileformats/cf/dataset.py` | Add `format` and `encoding` to both abstract classes (spec §4.2, F11) | +| `lib/iris/fileformats/netcdf/_dataset.py` | Implement `format` from `file_format`, and `encoding` from `filters()`/`chunking()`/`createVariable`'s keywords | +| `lib/iris/fileformats/cf/_variables.py` | `CFVariable.dimensions` admits `None` entries; `spans` and the three overrides match `(name, size)` pairs; `_is_scalar` unchanged | +| `lib/iris/fileformats/cf/_reader.py` | A third `__init__` branch for an already-open `CFDataset`; module docstring updated — it is no longer "the one module still coupled to `iris.fileformats.netcdf`" in the way it says | +| `lib/iris/fileformats/__init__.py` | Register the `"Zarr"` `FormatSpecification`; import the Zarr loader lazily | +| `lib/iris/io/format_picker.py` | Add the `ZarrStore` file element | +| `lib/iris/io/__init__.py` | `expand_filespecs` leaves a recognised remote scheme alone; new `load_stores`; `find_saver` tolerates a trailing separator; `_CUBELIST_SAVER_MODULES` replaces the substring test | +| `lib/iris/loading.py` | `_generate_cubes` partitions files from stores and routes the remote schemes | +| `lib/iris/fileformats/netcdf/ugrid_load.py` | The same scheme dispatch, updated in step | +| `lib/iris/tests/unit/fileformats/cf/dataset/contract.py` | Split into `CFDatasetReadContract` and `CFDatasetWriteContract` (F10); add the `format` and `encoding` cases | +| `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py` | Inherit both contract halves | +| `requirements/py312.yml`, `py313.yml`, `py314.yml` | `zarr >=3.0.8` in the optional-dependency section | +| `requirements/locks/*.lock` | Regenerated, once, by `make lockfiles` | +| `docs/src/user_manual/tutorial/s3_io.rst` | Replace "S3 is unsupported, use a FUSE mount" with the Zarr route (spec §4.7) | +| `docs/src/user_manual/howto/index.rst` (or the sibling toctree) | List the new page | +| `docs/superpowers/specs/2026-09-21-zarr-io-design.md` | §12.1 state, §12.3 Q2 closed, §12.4 entries for F10 and F11, §5's PR 4/PR 5 split amended | + +### Sequencing, and why it is this way + +The order is: make a Zarr store answerable as a `CFDataset`, then make the CF +layer able to hold one, then make Iris able to find one. + +1. **Tasks 1–6 build `ZarrDataset` bottom-up** and are testable without any of + Iris's loading machinery. Each one ends with a store built in `tmp_path` and + questions asked of it directly. +2. **Tasks 7–8 are the two changes to generic code.** They are separated from + the Zarr work because they are the only parts a netCDF-only reviewer has to + check, and because `spans` is the one place in this pull request where a + mistake silently changes what loads from an existing netCDF file. +3. **Task 9 is the loader**, which can only be written once 1–8 exist. +4. **Tasks 10–11 are the load chain**, which is independent of everything above + and could be reviewed by someone who knows nothing about Zarr. They come + last among the code tasks because until Task 9 exists there is nothing for + the chain to dispatch *to*. +5. **Tasks 12–14** are integration, documentation and the pull request. + +The dependency that matters in the other direction is **PR 3**. This plan calls +`CFDatasetVariable.read_data`, `iris.fileformats.cf.loader._get_actual_dtype`, +`_LAZYVAR_MIN_BYTES` and `as_lazy_data(..., cache_key=...)`, none of which exist +until [#7316](https://github.com/SciTools/iris/pull/7316) merges. If this branch +is cut before then, rebase onto `brownfield` once it has merged and re-run +Task 6 and Task 9's tests before trusting anything. + +--- + +## 4. The Zarr implementation, as designed + +Signatures the tasks below fill in. Read this before Task 1; it is the single +place the names and types agree. + +```python +# lib/iris/fileformats/cf/dataset.py (additions) + +class CFDatasetVariable(ABC): + @property + @abstractmethod + def format(self) -> str: + """The concrete format of the dataset holding this variable.""" + + @property + @abstractmethod + def encoding(self) -> Mapping[str, Any]: + """The storage settings this variable actually holds.""" + + +class CFDataset(ABC): + @property + @abstractmethod + def format(self) -> str: + """The concrete format: "NETCDF4", "NETCDF3_CLASSIC", "ZARR_V2", "ZARR_V3".""" + + @property + @abstractmethod + def encoding(self) -> Mapping[str, Any]: + """The store-wide settings this store actually holds.""" +``` + +```python +# lib/iris/fileformats/zarr/_dataset.py + +#: What CFDataset.format answers, by the store's Zarr version. +_FORMAT_BY_ZARR_VERSION = {2: "ZARR_V2", 3: "ZARR_V3"} + +#: The version 2 attribute that carries dimension names. It is storage, not a +#: CF attribute, so it is withheld from `attributes` and read by `dimensions`. +_ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS" + + +def _align_chunks( + chunks: tuple[int, ...], + store_chunks: tuple[int, ...] | None, + shape: tuple[int, ...], + name: str, +) -> tuple[int, ...]: + """Round `chunks` up to whole multiples of the store's own chunk shape.""" + + +class _ZarrAttributes(MutableMapping): + """A zarr node's attributes as a mapping, withholding the storage ones.""" + + def __init__(self, node, *, withhold: frozenset[str] = frozenset()): ... + + +class ZarrDatasetVariable(CFDatasetVariable): + def __init__(self, array: "zarr.Array", location: str, format: str): ... + + name: str # array.basename + location: str # the dataset's location, verbatim + format: str # "ZARR_V2" | "ZARR_V3" + dimensions: tuple[str | None, ...] # see the table in spec 4.4 + shape: tuple[int, ...] + dtype: np.dtype + size: int + fill_value: Any | None # array.fill_value: STORAGE, not a mask + chunking: tuple[int, ...] | None # array.chunks, never array.shards + encoding: Mapping[str, Any] # chunks, shards, codecs, zarr_format + attributes: MutableMapping # array.attrs, less _ARRAY_DIMENSIONS + + def __getitem__(self, keys) -> np.ndarray: ... + def __setitem__(self, keys, values) -> None: ... + def read_data(self, chunking_policy: Callable[[], tuple]) -> Any: ... + def write_handle(self) -> "ZarrDatasetVariable": ... # returns self + + @property + def cache_key(self) -> str: + """`(location, array path, the metadata document)`, as one string.""" + + @property + def array(self) -> "zarr.Array": + """The backing zarr array. Zarr-only; nothing generic may read it.""" + + +class ZarrDataset(CFDataset): + def __init__( + self, + location: str, + mode: str = "r", + *, + group: str | None = None, + ): ... + + location: str # the target as given, with the group path appended + format: str # "ZARR_V2" | "ZARR_V3" + mode: str + closed: bool + variables: Mapping[str, ZarrDatasetVariable] + dimensions: Mapping[str, int] # names used at one consistent size + encoding: Mapping[str, Any] # zarr_format, consolidated + attributes: MutableMapping # group.attrs + + def create_dimension(self, name, size) -> None: ... # NotImplementedError + def create_variable(self, name, dtype, dimensions=(), *, + fill_value=None, **encoding): ... # NotImplementedError + def sync(self) -> None: ... # no-op; a Zarr write is not buffered + def finalise(self) -> None: ... # NotImplementedError until PR 6 + def close(self) -> None: ... # records closure; a store is not a handle + + @property + def subgroup_names(self) -> tuple[str, ...]: + """The names of this group's subgroups. Zarr-only; for the no-data message.""" +``` + +```python +# lib/iris/fileformats/zarr/_decode.py + +def decode_fill_value(value: Any, dtype: np.dtype, variable_name: str) -> Any | None: + """Return a CF `_FillValue`/`missing_value` as a value of `dtype`, or None. + + Accepts a JSON number, and the base64 of a little-endian float64 that + released xarray writes for every floating dtype. Anything else warns and + returns None. + """ + + +def mask_and_scale( + block: np.ndarray, + *, + sentinels: tuple, + valid_min: Any | None, + valid_max: Any | None, + scale_factor: Any | None, + add_offset: Any | None, + dtype: np.dtype, +) -> np.ndarray: + """Apply the CF decode rules netCDF4 applies for free, in CF order.""" + + +def mask_by_sentinels(block: np.ndarray, sentinels: tuple) -> np.ndarray: ... +def mask_by_valid_range(block, valid_min, valid_max) -> np.ndarray: ... +def unpack(block, scale_factor, add_offset, dtype) -> np.ndarray: ... + + +def decode_parameters(variable: "ZarrDatasetVariable") -> dict: + """Read the CF decode parameters off a variable's attributes and storage. + + This is where spec section 4.4's version-aware masking precedence lives: + on version 3 only the CF attributes are a mask source; on version 2 + `Array.fill_value` is one as well, when no CF attribute is present. + """ +``` + +```python +# lib/iris/fileformats/zarr/loader.py + +def load_cubes(file_sources, callback=None, constraints=None, group=None): + """Generate cubes from Zarr stores.""" +``` + +```python +# lib/iris/io/__init__.py (additions) + +#: Modules whose `save` takes a CubeList directly, rather than one cube at a +#: time with `append=True`. One line per format. +_CUBELIST_SAVER_MODULES = frozenset({"iris.fileformats.netcdf.saver"}) + +#: URI schemes Iris passes through to a store library rather than resolving as +#: a local path. +REMOTE_STORE_SCHEMES = frozenset({"s3", "gs", "az", "abfs"}) + + +def load_stores(targets, callback, constraints=None): + """Create a generator of Cubes from targets that are stores, not files.""" +``` + +```python +# lib/iris/io/format_picker.py (addition) + +class ZarrStore(FileElement): + """A :class:`FileElement` that returns a target that may be a Zarr store. + + Carries the whole target, not just its basename, because recognising a + local store means reading a marker file inside it. + """ + + def __init__(self): + super().__init__(requires_fh=False) + + def get_element(self, basename, file_handle): ... # returns basename +``` + +Three points about this shape are worth stating once, here, rather than +repeating in every task. + +**`fill_value` means different things on the two backends, deliberately.** +`NetCDFDatasetVariable.fill_value` returns the `_FillValue` *attribute*; +`ZarrDatasetVariable.fill_value` returns `Array.fill_value`, the *storage* +field. On netCDF the two coincide. On Zarr they are different things (spec +§4.4), and conflating them masks every valid zero in an array that never +declared a sentinel. Nothing generic reads `fill_value` to decide masking — +`zarr/_decode.py::decode_parameters` does, and it reads the CF attributes +first. + +**`encoding` is readable and never interpreted.** Generic code may print it, +record it, or hand it back to a user; it must not branch on a key. Its keys are +the backend's own spelling: `chunks`, `shards`, `codecs`, `zarr_format` for +Zarr; `chunksizes`, `zlib`, `complevel` for netCDF. `chunking` is the typed, +guaranteed member that the chunk calculation reads (spec §4.2). + +**`cache_key` is a member of `ZarrDatasetVariable`, not of the interface.** +`read_data` is the only caller and both implementations build their own key — +netCDF from `repr(proxy)`, Zarr from the metadata document. Putting it on the +ABC would add a member no generic code reads, which is the indirection +`lib/iris/AGENTS.md` bans for a single caller. + --- From 81ef978965251bfa382fa9ffda3646831bdf79f1 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 12:00:05 +0000 Subject: [PATCH 03/56] Add PR 4 plan tasks 1-3: the dependency, the interface, the variable Task 1 makes zarr optional and proves Iris imports without it. Task 2 adds format and encoding to the CF dataset interface and splits the shared contract into read and write halves. Task 3 presents a Zarr array through the interface, which is where the dimension contract lands. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 1218 ++++++++++++++++- 1 file changed, 1212 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index f75dbe1175..1e36d1bf20 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -209,12 +209,15 @@ owns the code, written out in that task's own step style. Checked while writing this plan. Each is either a correction the implementer would otherwise hit as a failing test, or an answer the spec asked for. -**F1 — `zarr.Array` has no `dimension_names` property.** Version 3 names live -at `Array.metadata.dimension_names`; `ArrayV2Metadata` has no such attribute -and raises `AttributeError`, not the `ValueError` spec §2.4 quotes -**[verified]**. So `ZarrDatasetVariable.dimensions` must branch on -`metadata.zarr_format` and read `attrs["_ARRAY_DIMENSIONS"]` on version 2 — -never `try: … except ValueError`. +**F1 — `zarr.Array` has no `dimension_names` property, and the error spec §2.4 +quotes is a *write* error.** Version 3 names live at +`Array.metadata.dimension_names`. `create_array(..., dimension_names=…)` on a +version 2 group raises `ValueError: Zarr format 2 arrays do not support +dimension names.`, which is the message the spec cites — but **reading** +`ArrayV2Metadata.dimension_names` raises `AttributeError`, because the +attribute simply is not there **[verified]**. So `ZarrDatasetVariable.dimensions` +must branch on `metadata.zarr_format` and read `attrs["_ARRAY_DIMENSIONS"]` on +version 2. A `try: … except ValueError` would not catch anything. **F2 — zarr-python rejects a wrong-length `dimension_names` at creation.** `create_array(..., shape=(2, 2), dimension_names=("y",))` raises @@ -649,5 +652,1208 @@ netCDF from `repr(proxy)`, Zarr from the metadata document. Putting it on the ABC would add a member no generic code reads, which is the indirection `lib/iris/AGENTS.md` bans for a single caller. +--- +## 5. Tasks + +### 5.0 The green check every task ends with + +Every task's last step is a commit, and every commit is preceded by this, +unchanged: + +```bash +# 1. Confirm the test-data path took effect. A missing repository SKIPS tests +# silently, so an unset variable removes coverage without failing anything. +echo "$OVERRIDE_TEST_DATA_REPOSITORY" +ls "$OVERRIDE_TEST_DATA_REPOSITORY" > /dev/null + +# 2. Hooks rewrite files in place: re-stage and re-run until clean. +pre-commit run --files +git add +pre-commit run --files + +# 3. The tier the task's own steps name. Never --no-verify on the commit. +``` + +Read the **skip count** as well as the failure count. A Zarr test that skips +because `zarr` is missing from the environment is a test that did not run, and +`pytest -q` reports it in the same line as the passes. + +--- + +### Task 1: `zarr` as an optional dependency, and a package that does not need it + +**Files:** +- Modify: `requirements/py312.yml`, `requirements/py313.yml`, + `requirements/py314.yml` +- Modify: `requirements/locks/py312-linux-64.lock`, `py313-linux-64.lock`, + `py314-linux-64.lock` +- Create: `lib/iris/fileformats/zarr/__init__.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/__init__.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/stores.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py` + +**Interfaces:** +- Consumes: nothing. +- Produces: the importable package `iris.fileformats.zarr`; the test package + whose `__init__.py` skips everything below it when `zarr` is absent; and + `stores.py`'s builders — `make_store(path, *, zarr_format=3, chunks="auto", + shards=None)`, `make_array(group, name, shape, dtype, zarr_format, *, + dimensions=None, chunks="auto", shards=None, fill_value=None, + attributes=None)`, and the constants `DIMENSIONS`, `DATA`, `VARIABLE_ATTRS`, + `GLOBALS` — which every later Zarr test builds its fixtures from. + +The package's `__init__.py` re-exports nothing yet; `loader.py` does not exist +until Task 10, and a re-export of a name that is not there is an import error. +Task 10 adds the `from .loader import load_cubes` line. The registration that +gives a user a clear `ImportError` is Task 11, beside the format +specification, which is the only place that needs it. + +- [ ] **Step 1: Add the dependency to all three requirement files** + +In each of `requirements/py312.yml`, `py313.yml` and `py314.yml`, under the +`# Optional dependencies.` heading, after ` - affine`: + +```yaml + - zarr >=3.0.8 +``` + +The floor is spec §8: "Pin `>=3.0.8`, which is above the known data-loss bug". + +- [ ] **Step 2: Regenerate the lock files** + +```bash +make lockfiles +git status --short requirements/locks/ +``` + +Expected: the three `py3{12,13,14}-linux-64.lock` files change, each gaining +`zarr`, `numcodecs`, `donfig` and their own dependencies. + +This needs network access and takes minutes. If it cannot run here, say so in +the pull request body and ask a maintainer to regenerate — do **not** hand-edit +a lock file. + +- [ ] **Step 3: Create the package** + +Create `lib/iris/fileformats/zarr/__init__.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Native Zarr support, built on zarr-python. + +.. z_reference:: iris.fileformats.zarr + :tags: topic_load_save + + API reference + +Reads Zarr version 2 and version 3 stores, local or remote. Not NCZarr: that +is netCDF-c's own Zarr driver, reached through a URL fragment such as +``file:///path/store.zarr#mode=nczarr,file``, and it stays where it is in +:mod:`iris.fileformats.netcdf`. + +``zarr`` is an optional dependency. Nothing in :mod:`iris.fileformats` imports +this package until a Zarr target is actually loaded, so Iris installs and +imports without it. + +This package shadows the third-party ``zarr`` distribution by name. Python 3 +resolves ``import zarr`` absolutely, so inside these modules it still finds the +third-party package; xarray's ``xarray.backends.zarr`` has the same +arrangement. + +See sections 4.1 and 4.4 of +``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. + +""" +``` + +- [ ] **Step 4: Create the test package, with the skip that guards it** + +Create `lib/iris/tests/unit/fileformats/zarr/__init__.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :mod:`iris.fileformats.zarr`. + +``zarr`` is optional, so every module below this one is skipped when it is +absent. The skip lives here rather than in a ``conftest.py``: the same call +there raises during collection and is reported as an **error**, not a skip. +""" + +import pytest + +pytest.importorskip("zarr", minversion="3.0.8") +``` + +- [ ] **Step 5: Write the failing test for the import guard** + +Create `lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Iris must import, and load netCDF, without zarr-python installed.""" + +import subprocess +import sys + + +def _imports_zarr(statement): + """Run `statement` in a fresh interpreter; report whether zarr was imported.""" + probe = f"{statement}\nimport sys\nprint('zarr' in sys.modules)\n" + completed = subprocess.run( + [sys.executable, "-c", probe], + capture_output=True, + text=True, + check=True, + ) + return completed.stdout.strip() == "True" + + +def test_importing_iris_does_not_import_zarr(): + assert not _imports_zarr("import iris") + + +def test_importing_the_format_registry_does_not_import_zarr(): + # iris.fileformats imports every format module it registers a handler + # from. If the Zarr loader joined that list, Iris would stop installing + # without zarr-python. + assert not _imports_zarr("import iris.fileformats") + + +def test_importing_the_zarr_package_does_not_import_zarr(): + # The package docstring is reachable without the dependency; only the + # modules under it need zarr itself. + assert not _imports_zarr("import iris.fileformats.zarr") +``` + +- [ ] **Step 6: Run the tests to see them pass, and know why** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py -v +``` + +Expected: 3 passed. These pass from the start — that is the point. They are a +*regression* guard, not a red-green cycle: the way they fail is by someone +later adding `from . import zarr` to `iris/fileformats/__init__.py`, or an +`import zarr` to `iris/fileformats/zarr/__init__.py`. Prove they can fail +before trusting them, by temporarily adding `import zarr` to the package +`__init__.py` and re-running: + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py -v +``` + +Expected with the temporary line: `test_importing_the_zarr_package_does_not_import_zarr` +FAILS. Remove the line. + +- [ ] **Step 7: Write the shared store builders** + +Create `lib/iris/tests/unit/fileformats/zarr/stores.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Builders for the small Zarr stores these tests ask questions of. + +One canonical store, deliberately the same contents as the shared CFDataset +contract writes, so that a Zarr answer and a netCDF answer can be compared +without translating between two sets of fixtures. +""" + +import numpy as np +import zarr + +#: The canonical store's dimensions, in order. +DIMENSIONS = {"y": 3, "x": 2} +DATA = np.arange(6, dtype="f8").reshape(3, 2) +VARIABLE_ATTRS = {"units": "K", "long_name": "surface temperature"} +GLOBALS = {"Conventions": "CF-1.7", "title": "contract"} + +#: The version 2 attribute carrying dimension names. +ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS" + + +def make_array( + group, + name, + shape, + dtype, + zarr_format, + *, + dimensions=None, + chunks="auto", + shards=None, + fill_value=None, + attributes=None, +): + """Create one array, recording dimension names the way its version does. + + zarr-python refuses `dimension_names` on a version 2 array, so version 2 + names are written to the `_ARRAY_DIMENSIONS` attribute by hand - which is + what xarray and NCZarr both do. + """ + extra = {} + if zarr_format == 3 and dimensions is not None: + extra["dimension_names"] = dimensions + array = group.create_array( + name, + shape=shape, + dtype=dtype, + chunks=chunks, + shards=shards, + fill_value=fill_value, + **extra, + ) + if zarr_format == 2 and dimensions is not None: + array.attrs[ARRAY_DIMENSIONS] = list(dimensions) + for key, value in (attributes or {}).items(): + array.attrs[key] = value + return array + + +def make_store(path, *, zarr_format=3, chunks="auto", shards=None): + """Write the canonical store at `path`, and return `path` as a string.""" + root = zarr.open_group(store=path, mode="w", zarr_format=zarr_format) + for name, value in GLOBALS.items(): + root.attrs[name] = value + + array = make_array( + root, + "air_temperature", + DATA.shape, + "f8", + zarr_format, + dimensions=tuple(DIMENSIONS), + chunks=chunks, + shards=shards, + fill_value=-1.0, + attributes=VARIABLE_ATTRS, + ) + array[:] = DATA + + # A dimensionless variable: what a grid mapping is, and the one shape a + # CFDatasetVariable must handle without a leading dimension. + scalar = make_array( + root, + "grid", + (), + "i4", + zarr_format, + dimensions=(), + attributes={"grid_mapping_name": "latitude_longitude"}, + ) + scalar[()] = 0 + + return str(path) +``` + +- [ ] **Step 8: Prove the builders build both versions** + +Append to `test_optional_dependency.py`: + +```python +import zarr + +from . import stores + + +def test_make_store_version_3(tmp_path): + location = stores.make_store(tmp_path / "v3.zarr") + group = zarr.open_group(store=location, mode="r") + assert group["air_temperature"].metadata.zarr_format == 3 + assert group["air_temperature"].metadata.dimension_names == ("y", "x") + + +def test_make_store_version_2(tmp_path): + location = stores.make_store(tmp_path / "v2.zarr", zarr_format=2) + group = zarr.open_group(store=location, mode="r") + array = group["air_temperature"] + assert array.metadata.zarr_format == 2 + assert array.attrs[stores.ARRAY_DIMENSIONS] == ["y", "x"] +``` + +- [ ] **Step 9: Run them** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/ -v +``` + +Expected: 5 passed. + +- [ ] **Step 10: Commit** + +Run §5.0, then: + +```bash +git add requirements lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr +git commit -m "Add zarr as an optional dependency, and the package that uses it + +The package holds only its docstring for now: iris.fileformats does not +import it, so Iris still installs and imports without zarr-python. The +test package skips itself when the dependency is absent." +``` + +--- + +### Task 2: `format` and `encoding` on the interface, and on netCDF + +**Files:** +- Modify: `lib/iris/fileformats/cf/dataset.py` +- Modify: `lib/iris/fileformats/netcdf/_dataset.py` +- Modify: `lib/iris/tests/unit/fileformats/cf/dataset/contract.py` +- Modify: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py` +- Modify: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py` + +**Interfaces:** +- Consumes: the existing `CFDataset` / `CFDatasetVariable` from PR 2. +- Produces: `CFDataset.format`, `CFDataset.encoding`, + `CFDatasetVariable.format`, `CFDatasetVariable.encoding`; and the contract + split into `CFDatasetReadContract` and `CFDatasetWriteContract` (F10), which + Task 5 subclasses. + +Spec §5 gives `format` to PR 5. It has to be here instead, because PR 4 merges +first and a `ZarrDataset` that cannot say which Zarr version it is cannot +implement §4.4's masking rule or name itself in a warning (F11). +`supports_dtype` is a saver question and stays in PR 5. + +- [ ] **Step 1: Write the failing contract tests** + +In `lib/iris/tests/unit/fileformats/cf/dataset/contract.py`, add to +`CFDatasetContract`, under the `# -- The dataset itself ---` heading: + +```python + def test_format_is_a_non_empty_string(self, readable): + assert isinstance(readable.format, str) + assert readable.format + + def test_encoding_is_a_readable_mapping(self, readable): + # Its keys are the backend's own spelling and generic code must never + # branch on one, but it must be possible to read them: otherwise a + # caller that wants to know how a store is laid out has to reach + # through to the netCDF4 or zarr object, which is the abstraction + # leaking at the point it exists to hold. + assert isinstance(readable.encoding, Mapping) + dict(readable.encoding) +``` + +and under `# -- A variable ---`: + +```python + def test_variable_format_matches_its_dataset(self, variable, readable): + assert variable.format == readable.format + + def test_variable_encoding_reports_the_chunking(self, variable): + encoding = variable.encoding + assert isinstance(encoding, Mapping) + # Whatever a backend spells it, the chunk shape it reports through + # `chunking` has to be somewhere in `encoding` too, or `encoding` is + # not reporting what the store holds. + assert variable.chunking in (None, *encoding.values()) +``` + +Add `from collections.abc import Mapping` to the module's imports. + +- [ ] **Step 2: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/fileformats/netcdf/dataset/ -v -k "format or encoding" +``` + +Expected: FAIL, `AttributeError: 'NetCDFDataset' object has no attribute 'format'`. + +- [ ] **Step 3: Add the two members to the interface** + +In `lib/iris/fileformats/cf/dataset.py`, in `CFDatasetVariable`, after +`location`: + +```python + @property + @abstractmethod + def format(self) -> str: + """The concrete format of the dataset holding this variable. + + Repeated here from the dataset so that a variable can answer for + itself, as :attr:`location` is. For messages and diagnostics only: + generic code asks for a capability, never for a format. + + """ +``` + +and after `chunking`: + +```python + @property + @abstractmethod + def encoding(self) -> Mapping: + """The storage settings this variable holds, in the backend's spelling. + + ``zlib``, ``complevel``, ``chunksizes`` for netCDF; ``chunks``, + ``shards``, ``codecs``, ``zarr_format`` for Zarr. Readable, never + interpreted: no generic code may branch on a key. The typed, + guaranteed-present member is :attr:`chunking`. + + """ +``` + +In `CFDataset`, after `mode`: + +```python + @property + @abstractmethod + def format(self) -> str: + """The concrete format: ``NETCDF4``, ``NETCDF3_CLASSIC``, ``ZARR_V2``, ``ZARR_V3``. + + What the loader, the saver and ``LOAD_PROBLEMS`` *say* when they have + to name the format. Reconstructing it from the implementation class + would be the type test this interface exists to forbid. + + """ + + @property + @abstractmethod + def encoding(self) -> Mapping: + """The store-wide settings this store holds, in the backend's spelling.""" +``` + +`Mapping` is already imported in that module. + +- [ ] **Step 4: Implement them for netCDF** + +In `lib/iris/fileformats/netcdf/_dataset.py`, in `NetCDFDatasetVariable`, after +`location`: + +```python + @property + def format(self) -> str: + """The netCDF format of the file holding this variable.""" + return self._format +``` + +taking a `format` argument in `__init__` — add it to the signature after +`location`, store it as `self._format`, and pass +`self._dataset.file_format` from both construction sites in `NetCDFDataset` +(`_materialised_variables` and `create_variable`). + +After `chunking`: + +```python + @property + def encoding(self) -> Mapping: + """The netCDF storage settings this variable holds. + + Read back off the file, so a variable created with `zlib=True` reports + the compression netCDF4 actually applied rather than the keyword it + was given. + + """ + filters = self._variable.filters() or {} + return MappingProxyType({**filters, "chunksizes": self.chunking}) +``` + +Add `from types import MappingProxyType` to the module's imports. + +In `NetCDFDataset`, after `mode`: + +```python + @property + def format(self) -> str: + """The netCDF format of this file, as netCDF4 reports it.""" + assert self._dataset is not None + return self._dataset.file_format + + @property + def encoding(self) -> Mapping: + """The store-wide netCDF settings. Only the format is one.""" + return MappingProxyType({"format": self.format}) +``` + +- [ ] **Step 5: Run the contract** + +```bash +pytest lib/iris/tests/unit/fileformats/netcdf/dataset/ -v +``` + +Expected: all pass, including the four new cases. + +- [ ] **Step 6: Split the contract into its read and write halves** + +Rename `CFDatasetContract` to `CFDatasetReadContract`, keeping everything +above the `# -- Writing ---` heading and the `readable`, `variable` and +`scalar` fixtures. Move everything below that heading, plus the `writable` and +`reopen` fixtures and `populate()`, into a new class +`CFDatasetWriteContract` in the same module. Update the module docstring: + +```python +"""One test body, run against every :class:`CFDataset` implementation. + +Subclass :class:`CFDatasetReadContract` in a module named for the +implementation, supply the fixtures it declares, and every test here runs +against it. Both classes deliberately lack a ``Test`` prefix, so pytest does +not collect them where they are written, only where they are subclassed. + +The split is read from write because an implementation may arrive in two +pieces: the Zarr loader can answer every question in the read half before a +Zarr saver exists. Skipping half of one class instead would hide an +incomplete implementation behind a skip, and the point of a shared body is +that divergence fails loudly. + +``docs/superpowers/specs/2026-09-21-zarr-io-design.md`` section 6: the +interface is only worth having if both sides of it agree, and agreement is +cheapest to check by asking the same questions twice. + +Deliberately not exercised here: :meth:`CFDatasetVariable.deprecated_netcdf_member`. +It is an explicitly netCDF4-specific compatibility escape hatch, so asserting +either its presence or its absence in a shared body would itself be +backend-specific - the omission is intentional, not a gap. +""" +``` + +In `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`, +change the test class to inherit both: + +```python +class TestNetCDFDataset(contract.CFDatasetReadContract, contract.CFDatasetWriteContract): +``` + +- [ ] **Step 7: Run the netCDF contract again and compare the count** + +```bash +pytest lib/iris/tests/unit/fileformats/netcdf/dataset/ -q +``` + +Expected: the same number of tests as before the split, plus the four added in +Step 1. The split must move tests, not lose them — count before and after. + +- [ ] **Step 8: Commit** + +Run §5.0, then: + +```bash +git add lib/iris/fileformats/cf/dataset.py lib/iris/fileformats/netcdf/_dataset.py lib/iris/tests/unit/fileformats +git commit -m "Add format and encoding to the CF dataset interface + +format names the concrete format for a message or a diagnostic; encoding +reports the storage settings a store actually holds, readable but never +interpreted. Splits the shared contract into its read and write halves, so +that an implementation arriving in two pieces can be held to the first." +``` + +--- + +### Task 3: `ZarrDatasetVariable` — the properties, and the dimension contract + +**Files:** +- Create: `lib/iris/fileformats/zarr/_dataset.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py` + +**Interfaces:** +- Consumes: `CFDatasetVariable` with `format` and `encoding` (Task 2); + `stores.make_array`, `stores.make_store` (Task 1). +- Produces: `ZarrDatasetVariable(array, location, format)` with every read + member of the interface except `read_data`, which is Task 7; the constants + `_FORMAT_BY_ZARR_VERSION` and `_ARRAY_DIMENSIONS`; and the module docstring + that later tasks add to. + +Spec §4.4's dimension table is the whole of this task: + +| store | `dimensions` | +|---|---| +| version 3, `dimension_names` present | its entries, null becoming `None` | +| version 3, `dimension_names` absent | `(None,) * ndim` | +| version 2, `_ARRAY_DIMENSIONS` present | its entries | +| version 2, `_ARRAY_DIMENSIONS` absent | `(None,) * ndim` | +| any version, scalar array | `()` | +| any version, wrong length | `(None,) * ndim`, with an `IrisCfLoadWarning` | + +- [ ] **Step 1: Write the failing tests for the dimension table** + +Create `lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py` with only the +copyright header, then +`lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDatasetVariable`.""" + +import json + +import numpy as np +import pytest +import zarr + +from iris.fileformats.zarr._dataset import ZarrDatasetVariable +from iris.warnings import IrisCfLoadWarning + +from .. import stores + + +def wrap(array, location="store.zarr"): + """Return the ZarrDatasetVariable over `array`, naming its own format.""" + zarr_format = array.metadata.zarr_format + return ZarrDatasetVariable(array, location, f"ZARR_V{zarr_format}") + + +@pytest.fixture(params=[2, 3], ids=["v2", "v3"]) +def zarr_format(request): + return request.param + + +@pytest.fixture +def group(tmp_path, zarr_format): + return zarr.open_group( + store=tmp_path / "probe.zarr", mode="w", zarr_format=zarr_format + ) + + +def test_named_dimensions(group, zarr_format): + array = stores.make_array( + group, "v", (3, 2), "f8", zarr_format, dimensions=("y", "x") + ) + assert wrap(array).dimensions == ("y", "x") + + +def test_unnamed_dimensions_are_anonymous(group, zarr_format): + array = stores.make_array(group, "v", (3, 2), "f8", zarr_format) + assert wrap(array).dimensions == (None, None) + + +def test_scalar_has_no_dimensions(group, zarr_format): + array = stores.make_array(group, "v", (), "i4", zarr_format) + assert wrap(array).dimensions == () + + +def test_partially_named_dimensions_version_3(tmp_path): + # The version 3 core specification: "an unnamed dimension is indicated by + # the null object". A store may name some dimensions and not others. + group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3) + array = group.create_array( + "v", shape=(3, 2), dtype="f8", dimension_names=("y", None) + ) + assert wrap(array).dimensions == ("y", None) + + +def test_repeated_name_within_one_array(tmp_path): + # "It is recommended but not required that all non-null dimension names + # are distinct." Recommended is not required, so this must not raise. + group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3) + array = group.create_array( + "v", shape=(2, 2), dtype="f8", dimension_names=("d", "d") + ) + assert wrap(array).dimensions == ("d", "d") +``` + +- [ ] **Step 2: Add the malformed-store test** + +zarr-python rejects a wrong-length `dimension_names` at creation (F2), so this +one writes the metadata by hand. Append: + +```python +def test_wrong_length_dimension_names_warns_and_loads_anonymous(tmp_path): + """A `dimension_names` whose length is not the rank is a malformation. + + The version 3 core specification says the field "must be an array of + strings or null objects with the same length as `shape`". zarr-python + refuses to write one, so the metadata is edited directly. + """ + group = zarr.open_group(store=tmp_path / "bad.zarr", mode="w", zarr_format=3) + group.create_array("v", shape=(3, 2), dtype="f8", dimension_names=("y", "x")) + + metadata_path = tmp_path / "bad.zarr" / "v" / "zarr.json" + document = json.loads(metadata_path.read_text()) + document["dimension_names"] = ["y"] + metadata_path.write_text(json.dumps(document)) + + array = zarr.open_group(store=tmp_path / "bad.zarr", mode="r")["v"] + with pytest.warns(IrisCfLoadWarning, match="dimension_names"): + dimensions = wrap(array).dimensions + assert dimensions == (None, None) +``` + +- [ ] **Step 3: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/dataset/ -v +``` + +Expected: collection error, +`ModuleNotFoundError: No module named 'iris.fileformats.zarr._dataset'`. + +- [ ] **Step 4: Write the module, and the variable's dimension logic** + +Create `lib/iris/fileformats/zarr/_dataset.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The Zarr implementation of the :mod:`iris.fileformats.cf.dataset` interface. + +This is the only module that knows both the CF interface and the zarr-python +API. Everything Zarr-specific the CF layer could want - where dimension names +are recorded, what a chunk is, what the store's own fill value is - either has +a named member on the interface or lives here as a Zarr-only member. + +Three things about Zarr shape this module, and each is a trap if forgotten. + +**A dimension name guarantees less than a netCDF one.** The version 3 core +specification makes ``dimension_names`` optional, allows a null entry, only +*recommends* that names are distinct within an array, and places no +restriction at all on reusing a name across arrays. All of that is conforming +Zarr, so none of it may raise. A dimension with no name is **anonymous**, which +Iris has always had: a cube dimension carrying no coordinate. + +**``Array.fill_value`` is not ``_FillValue``.** It says what unwritten storage +reads back as; it does not say the value is missing. zarr-python defaults it to +zero, so reading it as a missing-value sentinel masks every valid zero in an +array that never declared one. :attr:`ZarrDatasetVariable.fill_value` reports +it as the storage fact it is, and the masking decision is made in +:mod:`iris.fileformats.zarr._decode` from the CF attributes. + +**The read unit is the inner chunk, even when the array is sharded.** Zarr's +sharding codec is indexed, so a reader fetches only the inner chunks it needs. +:attr:`ZarrDatasetVariable.chunking` therefore returns ``Array.chunks``, never +``Array.shards``, and :func:`_align_chunks` aggregates upward into whole +multiples of it. + +See sections 4.2, 4.4 and 4.6 of +``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. + +""" + +from collections.abc import Iterator, Mapping, MutableMapping +from types import MappingProxyType +from typing import Any +import warnings + +import numpy as np +import zarr + +from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable +import iris.warnings + +#: What :attr:`CFDataset.format` answers, by the store's Zarr version. +_FORMAT_BY_ZARR_VERSION = {2: "ZARR_V2", 3: "ZARR_V3"} + +#: The version 2 attribute carrying dimension names, as xarray and NCZarr +#: write it. Storage, not a CF attribute, so it is withheld from `attributes`. +_ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS" + + +def _dimension_names(array) -> tuple: + """Return one name or None per dimension of `array`. + + Parameters + ---------- + array : :class:`zarr.Array` + The array to read names from. Version 3 carries them in its metadata; + version 2 carries them in the ``_ARRAY_DIMENSIONS`` attribute, which + is a de-facto convention rather than part of the Zarr specification. + + Returns + ------- + tuple + Of length ``array.ndim``. ``None`` where the store names no dimension. + + Warns + ----- + IrisCfLoadWarning + If the store names a number of dimensions other than ``array.ndim``. + The version 3 specification says the lengths "must" match, so such a + store is malformed; its names are ignored and the array loads with + anonymous dimensions. + + """ + anonymous = (None,) * array.ndim + if not array.ndim: + return () + + if array.metadata.zarr_format == 3: + names = array.metadata.dimension_names + else: + names = array.attrs.get(_ARRAY_DIMENSIONS) + + if names is None: + return anonymous + + if len(names) != array.ndim: + message = ( + "Ignoring malformed dimension_names, which names " + f"{len(names)} dimensions of a {array.ndim}-dimensional array: " + f"{array.basename}" + ) + warnings.warn(message, category=iris.warnings.IrisCfLoadWarning) + return anonymous + + return tuple(names) +``` + +- [ ] **Step 5: Write the class** + +Append to `_dataset.py`: + +```python +class ZarrDatasetVariable(CFDatasetVariable): + """One array of a Zarr store, presented through the CF interface.""" + + def __init__(self, array, location: str, format: str): + """Wrap `array`. + + Parameters + ---------- + array : :class:`zarr.Array` + The array this variable presents. + location : str + The path or URL of the dataset this array belongs to. + format : str + ``ZARR_V2`` or ``ZARR_V3``, as its dataset reports. + + """ + self._array = array + self._location = location + self._format = format + self._dimensions = _dimension_names(array) + self._attributes = _ZarrAttributes(array, withhold={_ARRAY_DIMENSIONS}) + + @property + def name(self) -> str: + """The array's name within its group.""" + return self._array.basename + + @property + def location(self) -> str: + """The path or URL of the dataset holding this variable.""" + return self._location + + @property + def format(self) -> str: + """``ZARR_V2`` or ``ZARR_V3``.""" + return self._format + + @property + def dimensions(self) -> tuple: + """One name per dimension, ``None`` where the store names none.""" + return self._dimensions + + @property + def shape(self) -> tuple: + """The array's shape.""" + return tuple(self._array.shape) + + @property + def dtype(self) -> np.dtype: + """The array's stored data type.""" + return self._array.dtype + + @property + def size(self) -> int: + """The total number of elements in the array.""" + return int(self._array.size) + + @property + def fill_value(self) -> Any: + """What unwritten storage reads back as. **Not** a missing-value flag. + + Version 3 requires this field, and zarr-python defaults it to zero, so + it says nothing about whether a value is missing. The CF + ``_FillValue`` and ``missing_value`` attributes say that, and + :mod:`iris.fileformats.zarr._decode` reads them. + + """ + return self._array.fill_value + + @property + def chunking(self) -> tuple | None: + """The array's inner chunk shape, never its shard shape.""" + chunks = self._array.chunks + return tuple(chunks) if chunks else None + + @property + def encoding(self) -> Mapping: + """The Zarr storage settings this array holds.""" + return MappingProxyType( + { + "chunks": self._array.chunks, + "shards": self._array.shards, + "codecs": self._array.metadata.to_dict().get("codecs"), + "zarr_format": self._array.metadata.zarr_format, + } + ) + + @property + def attributes(self) -> MutableMapping: + """The array's CF attributes, less the ones that are really storage.""" + return self._attributes + + def __getitem__(self, keys) -> np.ndarray: + """Return the indexed portion of the array's data.""" + return self._array[keys] + + def __setitem__(self, keys, values) -> None: + """Write `values` into the indexed portion of the array.""" + self._array[keys] = values + + def write_handle(self) -> "ZarrDatasetVariable": + """Return self: a Zarr write target needs no proxy. + + ``zarr.Array`` is picklable and a chunk is an independent object in a + key-value store, so there is no open file handle to avoid holding and + no lock to acquire. + + """ + return self + + def __repr__(self) -> str: + """Return a string representation.""" + return f"{self.__class__.__name__}({self.name!r}, {self.location!r})" + + # Zarr-only members below. + + @property + def array(self): + """The backing :class:`zarr.Array`. Nothing generic may read this.""" + return self._array +``` + +Note `chunking` returns `None` for a scalar array, because `Array.chunks` is +`()` there and an empty tuple is not a chunk shape. + +- [ ] **Step 6: Write the attributes mapping** + +Append to `_dataset.py`, above `ZarrDatasetVariable`: + +```python +class _ZarrAttributes(MutableMapping): + """A Zarr node's attributes as a mapping, less the ones that are storage. + + ``_ARRAY_DIMENSIONS`` is how a version 2 store records dimension names. + Under the interface's split that is a property, reported by + :attr:`ZarrDatasetVariable.dimensions`, not a CF attribute - so it is + withheld here. Leaving it in would put it on every cube loaded from a + version 2 store, since every attribute the loading rules do not consume + reaches ``cube.attributes``. + """ + + def __init__(self, node, *, withhold: frozenset = frozenset()): + """Materialise the attributes of `node`, a zarr array or group.""" + self._node = node + self._withhold = frozenset(withhold) + self._values = { + key: value + for key, value in node.attrs.items() + if key not in self._withhold + } + + def __getitem__(self, key: str) -> Any: + """Return `key`'s value.""" + return self._values[key] + + def __setitem__(self, key: str, value: Any) -> None: + """Set `key`'s value, here and in the store.""" + if key in self._withhold: + msg = f"Cannot set a storage attribute through `attributes`: {key}" + raise KeyError(msg) + self._node.attrs[key] = value + self._values[key] = value + + def __delitem__(self, key: str) -> None: + """Remove `key`, here and from the store.""" + del self._node.attrs[key] + del self._values[key] + + def __iter__(self) -> Iterator[str]: + """Return an iterator over the attribute names, in store order.""" + return iter(self._values) + + def __len__(self) -> int: + """Return the number of attributes.""" + return len(self._values) + + def __repr__(self) -> str: + """Return a string representation.""" + return f"{self.__class__.__name__}({self._values!r})" +``` + +- [ ] **Step 7: Run the dimension tests** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/dataset/ -v +``` + +Expected: all pass. + +- [ ] **Step 8: Write the tests for the remaining properties** + +Append to `test_ZarrDatasetVariable.py`: + +```python +def test_plain_properties(tmp_path, zarr_format): + location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format) + array = zarr.open_group(store=location, mode="r")["air_temperature"] + variable = wrap(array, location) + + assert variable.name == "air_temperature" + assert variable.location == location + assert variable.format == f"ZARR_V{zarr_format}" + assert variable.shape == stores.DATA.shape + assert variable.dtype == np.dtype("f8") + assert variable.size == stores.DATA.size + assert variable.ndim == stores.DATA.ndim + assert len(variable) == stores.DATA.shape[0] + np.testing.assert_array_equal(variable[:], stores.DATA) + np.testing.assert_array_equal(variable[1], stores.DATA[1]) + + +def test_fill_value_is_the_storage_field(tmp_path, zarr_format): + """`fill_value` is what unwritten storage reads as, not `_FillValue`. + + zarr-python defaults the field to zero, so an array that declares no CF + sentinel still reports one here. Reading it as a mask source is what + masks every valid zero. + """ + group = zarr.open_group( + store=tmp_path / "f.zarr", mode="w", zarr_format=zarr_format + ) + array = stores.make_array(group, "v", (3,), "i2", zarr_format) + assert wrap(array).fill_value == 0 + + declared = stores.make_array( + group, "w", (3,), "i2", zarr_format, fill_value=-999 + ) + assert wrap(declared).fill_value == -999 + + +def test_chunking_is_the_inner_chunk_not_the_shard(tmp_path): + """A sharded array reports `Array.chunks`. + + Zarr's sharding codec is indexed: a reader fetches the shard index, then + byte ranges for only the inner chunks it needs. The inner chunk is the + independently decodable unit. + """ + group = zarr.open_group(store=tmp_path / "sh.zarr", mode="w", zarr_format=3) + array = group.create_array( + "v", shape=(1024,), dtype="i4", chunks=(8,), shards=(1024,) + ) + variable = wrap(array) + assert variable.chunking == (8,) + assert variable.encoding["shards"] == (1024,) + + +def test_encoding_reports_the_store(tmp_path, zarr_format): + location = stores.make_store(tmp_path / "e.zarr", zarr_format=zarr_format) + array = zarr.open_group(store=location, mode="r")["air_temperature"] + encoding = wrap(array, location).encoding + assert encoding["chunks"] == array.chunks + assert encoding["zarr_format"] == zarr_format +``` + +- [ ] **Step 9: Write the test for Review Focus 1 — `_ARRAY_DIMENSIONS`** + +Append: + +```python +def test_array_dimensions_is_not_a_cf_attribute(tmp_path): + """`_ARRAY_DIMENSIONS` is storage, so it must not reach `attributes`. + + It is how a version 2 store records dimension names. Every attribute the + loading rules do not consume reaches `cube.attributes`, so leaving it in + would put it on every cube loaded from a version 2 store, and make that + cube unequal to the same data loaded from version 3. + """ + group = zarr.open_group(store=tmp_path / "v2.zarr", mode="w", zarr_format=2) + array = stores.make_array( + group, + "v", + (3, 2), + "f8", + 2, + dimensions=("y", "x"), + attributes={"units": "K"}, + ) + variable = wrap(array) + + assert variable.dimensions == ("y", "x") + assert stores.ARRAY_DIMENSIONS in array.attrs + assert stores.ARRAY_DIMENSIONS not in variable.attributes + assert dict(variable.attributes) == {"units": "K"} + + +def test_version_2_and_version_3_present_the_same_attributes(tmp_path): + v2 = stores.make_store(tmp_path / "a2.zarr", zarr_format=2) + v3 = stores.make_store(tmp_path / "a3.zarr", zarr_format=3) + from_2 = zarr.open_group(store=v2, mode="r")["air_temperature"] + from_3 = zarr.open_group(store=v3, mode="r")["air_temperature"] + assert dict(wrap(from_2, v2).attributes) == dict(wrap(from_3, v3).attributes) +``` + +- [ ] **Step 10: Write the test for nested and vector attributes (F14)** + +Append: + +```python +def test_structured_attribute_values_are_carried_unchanged(tmp_path): + """A JSON object or array attribute is carried, not interpreted. + + The version 3 specification allows an attribute to be any JSON literal, + and CF cannot express a nested one at any rank. Two independent producers + write them - NOAA GFS gives `valid_time` a `statistics_approximate` + object, and the ESA EOPF product root carries `stac_discovery`. + A flat numeric array is an ordinary CF vector attribute and is not special. + """ + group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "v", + (2,), + "f8", + 3, + attributes={ + "statistics_approximate": {"min": 0.0, "max": 1.0}, + "valid_range": [0.0, 1.0], + "nested_list": [[1, 2], [3, 4]], + }, + ) + attributes = wrap(array).attributes + assert attributes["statistics_approximate"] == {"min": 0.0, "max": 1.0} + assert attributes["valid_range"] == [0.0, 1.0] + assert attributes["nested_list"] == [[1, 2], [3, 4]] +``` + +- [ ] **Step 11: Run the module** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/ -v +``` + +Expected: all pass. + +- [ ] **Step 12: Commit** + +Run §5.0, then: + +```bash +git add lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr +git commit -m "Present a Zarr array through the CF dataset interface + +The dimension contract is the substance: a Zarr dimension may be unnamed, +and an unnamed one is anonymous rather than an error. _ARRAY_DIMENSIONS is +storage, so it is withheld from the CF attributes, and fill_value is +reported as the storage fact it is rather than as a missing-value flag." +``` + --- From 714930aa486a7906666f6d966d634d22e8aba8aa Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 12:02:02 +0000 Subject: [PATCH 04/56] Add PR 4 plan tasks 4-5: the Zarr group, and the shared contract Task 4 assembles CFDataset.dimensions from the arrays and decides what a name used at two sizes means. Task 5 asks the shared read contract of both Zarr versions, and adds the check that the cf package names no implementation class. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 615 ++++++++++++++++++ 1 file changed, 615 insertions(+) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index 1e36d1bf20..90ce5afb0a 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -1855,5 +1855,620 @@ storage, so it is withheld from the CF attributes, and fill_value is reported as the storage fact it is rather than as a missing-value flag." ``` +--- +### Task 4: `ZarrDataset` — the group, its variables and its dimensions + +**Files:** +- Modify: `lib/iris/fileformats/zarr/_dataset.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py` + +**Interfaces:** +- Consumes: `ZarrDatasetVariable`, `_ZarrAttributes`, + `_FORMAT_BY_ZARR_VERSION` (Task 3). +- Produces: `ZarrDataset(location, mode="r", *, group=None)` with + `location`, `format`, `mode`, `closed`, `variables`, `dimensions`, + `encoding`, `attributes`, `close`, `sync`, and the Zarr-only + `subgroup_names`. The write members exist and raise. + +Spec §4.4's second rule is the substance here: **`CFDataset.dimensions` is the +names that are used consistently.** A name that appears at two different sizes +is not an error and is not silently merged — it is omitted from the mapping, +with an `IrisCfLoadWarning` naming the dimension and the sizes. The variables +that use it keep it in their own `dimensions`, where the `(name, size)` rule of +Task 8 stops it linking them. + +- [ ] **Step 1: Write the failing tests** + +Create `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDataset`.""" + +import numpy as np +import pytest +import zarr + +from iris.fileformats.zarr._dataset import ZarrDataset +from iris.warnings import IrisCfLoadWarning + +from .. import stores + + +@pytest.fixture(params=[2, 3], ids=["v2", "v3"]) +def zarr_format(request): + return request.param + + +def test_opens_and_reports_itself(tmp_path, zarr_format): + location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format) + dataset = ZarrDataset(location) + + assert dataset.location == location + assert dataset.format == f"ZARR_V{zarr_format}" + assert dataset.mode == "r" + assert dataset.closed is False + assert sorted(dataset.variables) == ["air_temperature", "grid"] + assert dict(dataset.attributes) == stores.GLOBALS + assert dataset.encoding["zarr_format"] == zarr_format + + +def test_dimensions_are_assembled_from_the_arrays(tmp_path, zarr_format): + location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format) + assert dict(ZarrDataset(location).dimensions) == stores.DIMENSIONS + + +def test_a_name_used_at_two_sizes_is_omitted_and_warns(tmp_path): + """Two arrays may share a dimension name at different lengths. + + The version 3 specification "does not place any restrictions on the use of + the same dimension name across multiple arrays", so this is conforming + Zarr and must not raise. Merging the two would produce a wrong cube. + """ + group = zarr.open_group(store=tmp_path / "c.zarr", mode="w", zarr_format=3) + stores.make_array(group, "short", (2,), "f8", 3, dimensions=("t",)) + stores.make_array(group, "long", (5,), "f8", 3, dimensions=("t",)) + + with pytest.warns(IrisCfLoadWarning, match="t"): + dimensions = dict(ZarrDataset(str(tmp_path / "c.zarr")).dimensions) + + assert "t" not in dimensions + + +def test_anonymous_dimensions_are_not_in_the_mapping(tmp_path): + group = zarr.open_group(store=tmp_path / "a.zarr", mode="w", zarr_format=3) + stores.make_array(group, "named", (2,), "f8", 3, dimensions=("t",)) + stores.make_array(group, "unnamed", (4,), "f8", 3) + + dataset = ZarrDataset(str(tmp_path / "a.zarr")) + assert dict(dataset.dimensions) == {"t": 2} + + +def test_close_is_idempotent(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + dataset = ZarrDataset(location) + dataset.close() + dataset.close() + assert dataset.closed is True + + +def test_context_manager_closes(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + dataset = ZarrDataset(location) + with dataset as entered: + assert entered is dataset + assert dataset.closed is True + + +def test_group_keyword_selects_a_subgroup(tmp_path): + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + sub = root.create_group("measurements/r10m") + stores.make_array(sub, "b02", (2,), "u2", 3, dimensions=("p",)) + + dataset = ZarrDataset(str(tmp_path / "n.zarr"), group="measurements/r10m") + assert list(dataset.variables) == ["b02"] + assert dataset.location.endswith("measurements/r10m") + + +def test_subgroup_names_lists_only_the_groups(tmp_path): + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + root.create_group("conditions") + root.create_group("measurements") + stores.make_array(root, "v", (2,), "f8", 3) + + dataset = ZarrDataset(str(tmp_path / "n.zarr")) + assert sorted(dataset.subgroup_names) == ["conditions", "measurements"] + + +def test_nested_groups_are_not_walked(tmp_path): + """Iris loads one group per call, like xarray. + + A Zarr store can legitimately hold unrelated datasets, and merging them + into one CubeList would be wrong. + """ + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + stores.make_array(root, "top", (2,), "f8", 3) + stores.make_array(root.create_group("sub"), "inner", (2,), "f8", 3) + + assert list(ZarrDataset(str(tmp_path / "n.zarr")).variables) == ["top"] + + +def test_the_write_surface_refuses(tmp_path): + # Saving to Zarr is not implemented. The members exist because the + # interface declares them; refusing is the honest answer. + dataset = ZarrDataset(stores.make_store(tmp_path / "s.zarr")) + with pytest.raises(NotImplementedError, match="Saving"): + dataset.create_dimension("t", 3) + with pytest.raises(NotImplementedError, match="Saving"): + dataset.create_variable("v", np.dtype("f4"), ("t",)) + with pytest.raises(NotImplementedError, match="Saving"): + dataset.finalise() + + +def test_a_write_mode_refuses(tmp_path): + stores.make_store(tmp_path / "s.zarr") + with pytest.raises(NotImplementedError, match="Saving"): + ZarrDataset(str(tmp_path / "s.zarr"), mode="w") +``` + +- [ ] **Step 2: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py -v +``` + +Expected: collection error, `ImportError: cannot import name 'ZarrDataset'`. + +- [ ] **Step 3: Write the dimension assembly** + +Append to `lib/iris/fileformats/zarr/_dataset.py`, after `_dimension_names`: + +```python +def _assemble_dimensions(variables) -> dict: + """Return the dimension sizes the store's arrays agree on. + + Parameters + ---------- + variables : mapping of str to :class:`ZarrDatasetVariable` + Every array in one group. + + Returns + ------- + dict + Name to size, for every name used at exactly one size. Anonymous + dimensions have no name and so are absent. + + Warns + ----- + IrisCfLoadWarning + For each name used at more than one size. Zarr places no restriction + on reusing a name across arrays, so this is conforming and must not + raise; merging the two would produce a wrong cube, so the name is + omitted instead and the arrays that use it keep it in their own + `dimensions`. + + """ + sizes: dict[str, int] = {} + conflicts: dict[str, set[int]] = {} + for variable in variables.values(): + for name, size in zip(variable.dimensions, variable.shape): + if name is None: + continue + if name not in sizes: + sizes[name] = size + elif sizes[name] != size: + conflicts.setdefault(name, {sizes[name]}).add(size) + + for name, found in conflicts.items(): + message = ( + "Ignoring a dimension name used at more than one size, " + f"{sorted(found)}: {name}" + ) + warnings.warn(message, category=iris.warnings.IrisCfLoadWarning) + del sizes[name] + + return sizes +``` + +- [ ] **Step 4: Write the class** + +Append to `_dataset.py`: + +```python +#: What `mode` a dataset may be opened in. Writing is not implemented. +_READ_MODES = ("r",) + +#: The one message every unimplemented write member raises. +_NO_SAVING = "Saving to Zarr is not supported." + + +class ZarrDataset(CFDataset): + """A Zarr store, or one group of one, presented through the CF interface.""" + + def __init__(self, location, mode: str = "r", *, group: str | None = None): + """Open a Zarr store. + + Parameters + ---------- + location : str or :class:`pathlib.Path` + The store's path or URL. A URL is handed to zarr-python + unaltered; Iris adds no store library and handles no credentials, + so an unreachable scheme fails with zarr-python's own message. + mode : str, default="r" + Only ``"r"`` is supported. + group : str, optional + A ``/``-separated path to a group within the store. The root group + by default. Iris reads one group per call and does not walk + nested groups. + + Raises + ------ + NotImplementedError + For any `mode` other than ``"r"``. + FileNotFoundError + If `location` does not exist. + zarr.errors.GroupNotFoundError + If `location` is not a Zarr store, or holds no such `group`. + + """ + # Set first, so that close() is safe if opening fails. + self._group = None + self._closed = False + self._variables: dict[str, ZarrDatasetVariable] | None = None + self._attributes: _ZarrAttributes | None = None + self._dimensions: dict[str, int] | None = None + + if mode not in _READ_MODES: + raise NotImplementedError(f"{_NO_SAVING} Cannot open with mode={mode!r}.") + + self._mode = mode + self._group_path = group + target = str(location) + self._location = target if group is None else f"{target.rstrip('/')}/{group}" + + self._group = zarr.open_group(store=target, mode=mode, path=group) + self._format = _FORMAT_BY_ZARR_VERSION[self._group.metadata.zarr_format] + + @property + def location(self) -> str: + """The store's path or URL, with the group path when there is one.""" + return self._location + + @property + def format(self) -> str: + """``ZARR_V2`` or ``ZARR_V3``.""" + return self._format + + @property + def mode(self) -> str: + """The mode the store was opened in.""" + return self._mode + + @property + def closed(self) -> bool: + """Whether :meth:`close` has been called. + + A Zarr store is a namespace rather than an open file handle, so there + is nothing to release and nothing else to ask. + + """ + return self._closed + + @property + def variables(self) -> Mapping: + """The group's arrays, by name. Arrays in subgroups are not included.""" + if self._variables is None: + self._variables = { + name: ZarrDatasetVariable(array, self._location, self._format) + for name, array in self._group.arrays() + } + return self._variables + + @property + def dimensions(self) -> Mapping: + """The dimension sizes this group's arrays agree on.""" + if self._dimensions is None: + self._dimensions = _assemble_dimensions(self.variables) + return self._dimensions + + @property + def encoding(self) -> Mapping: + """The store-wide Zarr settings.""" + consolidated = self._group.metadata.consolidated_metadata + return MappingProxyType( + { + "zarr_format": self._group.metadata.zarr_format, + "consolidated": consolidated is not None, + } + ) + + @property + def attributes(self) -> MutableMapping: + """The group's attributes.""" + if self._attributes is None: + self._attributes = _ZarrAttributes(self._group) + return self._attributes + + def create_dimension(self, name: str, size: int | None) -> None: + """Raise: saving to Zarr is not supported.""" + raise NotImplementedError(_NO_SAVING) + + def create_variable( + self, name: str, dtype, dimensions=(), *, fill_value=None, **encoding + ): + """Raise: saving to Zarr is not supported.""" + raise NotImplementedError(_NO_SAVING) + + def sync(self) -> None: + """Do nothing: a Zarr write is not buffered.""" + + def finalise(self) -> None: + """Raise: saving to Zarr is not supported. + + A Zarr store's completion step is consolidating its metadata, which + only a writer performs. + + """ + raise NotImplementedError(_NO_SAVING) + + def close(self) -> None: + """Record that this dataset is finished with. + + A store is a namespace, not an open handle, so nothing is released. + The flag exists because callers - :class:`iris.fileformats.cf.CFReader` + among them - must be able to ask. + + """ + self._closed = True + + def __repr__(self) -> str: + """Return a string representation.""" + return f"{self.__class__.__name__}({self.location!r}, mode={self.mode!r})" + + # Zarr-only members below. + + @property + def subgroup_names(self) -> tuple: + """The names of this group's subgroups. + + What the loader names when a group holds no data variables but the + store plainly holds data elsewhere. + + """ + return tuple(self._group.group_keys()) +``` + +- [ ] **Step 5: Run the tests** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py -v +``` + +Expected: all pass. + +- [ ] **Step 6: Add the remote and error-path tests** + +Spec §6: "Remote stores are covered by an in-memory store and a local +directory store only. No test touches the network." Append to +`test_ZarrDataset.py`: + +```python +def test_an_in_memory_store_is_just_another_location(tmp_path): + """A store need not be a directory. zarr-python decides what it is.""" + memory = zarr.storage.MemoryStore() + group = zarr.open_group(store=memory, mode="w", zarr_format=3) + stores.make_array(group, "v", (2,), "f8", 3, dimensions=("t",)) + + dataset = ZarrDataset(memory) + assert list(dataset.variables) == ["v"] + + +def test_a_missing_store_raises_zarrs_own_error(tmp_path): + with pytest.raises(FileNotFoundError): + ZarrDataset(str(tmp_path / "nothing.zarr")) + + +def test_a_directory_that_is_not_a_store_raises(tmp_path): + (tmp_path / "plain").mkdir() + with pytest.raises(zarr.errors.GroupNotFoundError): + ZarrDataset(str(tmp_path / "plain")) + + +def test_a_missing_group_raises(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + with pytest.raises(zarr.errors.GroupNotFoundError): + ZarrDataset(location, group="nope") +``` + +`ZarrDataset(memory)` passes a store object where the signature says "str or +Path". That is deliberate and documented in the docstring's `location` +paragraph as "handed to zarr-python": `str(store)` of a `MemoryStore` is not a +path, so `location` becomes zarr-python's own rendering of it, which is the +best name available for a store with no address. Add that sentence to the +`location` docstring when this test is written. + +- [ ] **Step 7: Run and commit** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/ -v +``` + +Expected: all pass. Then run §5.0 and: + +```bash +git add lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr +git commit -m "Present a Zarr group through the CF dataset interface + +Dimensions are assembled from the arrays, and a name used at two sizes is +omitted with a warning rather than merged: Zarr places no restriction on +reusing a name across arrays, so a store meaning two things by one name is +conforming, and merging them would produce a wrong cube." +``` + +--- + +### Task 5: the shared read contract, run against Zarr + +**Files:** +- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py` + +**Interfaces:** +- Consumes: `CFDatasetReadContract` (Task 2); `ZarrDataset` (Task 4); + `stores.make_store` (Task 1). +- Produces: nothing other than coverage. This task exists so that a + netCDF/Zarr divergence fails loudly, which is spec §6's whole argument for a + shared body. + +The contract's canonical store is the one `stores.make_store` writes, which is +why Task 1 wrote it to match `contract.populate()` exactly — same dimensions, +same data, same attributes, same scalar `grid` variable. + +- [ ] **Step 1: Write the test module** + +Create +`lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The shared CFDataset read contract, asked of the Zarr implementation. + +Only the read half: the Zarr write path does not exist, and `ZarrDataset` +refuses every member of the write surface. The write half of the contract +arrives with the saver. +""" + +import pytest + +from iris.fileformats.zarr._dataset import ZarrDataset +from iris.tests.unit.fileformats.cf.dataset import contract + +from .. import stores + + +class TestZarrDatasetVersion3(contract.CFDatasetReadContract): + @pytest.fixture + def readable(self, tmp_path): + return ZarrDataset(stores.make_store(tmp_path / "contract.zarr")) + + +class TestZarrDatasetVersion2(contract.CFDatasetReadContract): + @pytest.fixture + def readable(self, tmp_path): + location = stores.make_store(tmp_path / "contract.zarr", zarr_format=2) + return ZarrDataset(location) +``` + +- [ ] **Step 2: Run it, and expect disagreements** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py -v +``` + +This is the step that earns the task. Expect failures, and resolve each one +by deciding **which side is wrong**, not by loosening the contract: + +- `test_variable_fill_value` asserts `-1.0`, which `stores.make_store` sets as + the array's `fill_value`. Zarr answers from storage and netCDF from the + `_FillValue` attribute (§4, "`fill_value` means different things"), so both + give `-1.0` here and the test should pass. If it does not, the Zarr store + builder is not setting what it claims to. +- `test_variable_chunking_is_none_or_a_shape` already admits `np.integer`, + which was written for Zarr. +- `test_mode_is_a_read_mode` asserts `mode in ("r", "r+", "a")`; `ZarrDataset` + answers `"r"`. +- `test_scalar_variable` expects `len(scalar)` to raise `TypeError` matching + "unsized". `CFDatasetVariable.__len__` supplies that for any implementation + with an empty `shape`, and `ZarrDatasetVariable` does not override it. + +Record each resolution in the commit message. A contract test that needs a +Zarr-shaped exception is a design signal, not a test problem. + +- [ ] **Step 3: Confirm the netCDF side is untouched** + +```bash +pytest lib/iris/tests/unit/fileformats/netcdf/dataset/ -q +``` + +Expected: the same count as at the end of Task 2. + +- [ ] **Step 4: Add the no-isinstance check** + +Spec §6: "A separate check greps the `cf` package for `isinstance` against an +implementation class and asserts there are none — the rule in §4.2 is only +worth writing down if something notices when it is broken." + +Create `lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The CF layer must not know which storage implementation it holds. + +Testing the interface - `isinstance(thing, CFDataset)` - is how generic code +is *supposed* to accept any backend. Testing an implementation is what makes +a third backend a rewrite instead of a new module. +""" + +from pathlib import Path + +import iris.fileformats.cf + +#: Implementation class names no module in the `cf` package may name. +IMPLEMENTATION_NAMES = ( + "NetCDFDataset", + "NetCDFDatasetVariable", + "ZarrDataset", + "ZarrDatasetVariable", +) + + +def cf_package_sources(): + """Yield `(path, text)` for every module in `iris.fileformats.cf`.""" + package = Path(iris.fileformats.cf.__file__).parent + for path in sorted(package.glob("*.py")): + yield path, path.read_text(encoding="utf-8") + + +def test_no_module_names_an_implementation_class(): + offenders = [] + for path, text in cf_package_sources(): + for line_number, line in enumerate(text.splitlines(), start=1): + if line.lstrip().startswith("#"): + continue + for name in IMPLEMENTATION_NAMES: + if name in line: + offenders.append(f"{path.name}:{line_number}: {line.strip()}") + assert offenders == [] +``` + +Note this test will **fail** until Task 9 removes `cf/_reader.py`'s import of +`NetCDFDataset`. Write it here, see it fail, and let Task 9 turn it green; +Task 9's steps say so. Mark it `@pytest.mark.xfail(strict=True, reason=...)` +in between **only if** you need an intervening commit to be green — and delete +the marker in Task 9, because an xfail that starts passing is a failure +(`AGENTS.md` gotcha 1). + +- [ ] **Step 5: Commit** + +Run §5.0, then: + +```bash +git add lib/iris/tests/unit/fileformats +git commit -m "Ask the shared CFDataset read contract of the Zarr store + +Both Zarr versions answer the same questions the netCDF implementation +does, so a divergence between the two fails here rather than somewhere in +the loader. Adds the check that no module in the cf package names an +implementation class." +``` + --- From c379be7c938878ecf0f2f23b4e567c1aa533e4c0 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 12:07:14 +0000 Subject: [PATCH 05/56] Add PR 4 plan tasks 6-7: CF decoding, and the lazy read Task 6 implements masking, valid-range masking and unpacking as pure functions, with the version-aware fill-value precedence and both _FillValue encodings. Task 7 is the lazy read: no proxy, chunking rounded up to the store's own, and a cache key that carries the array's metadata. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 1338 +++++++++++++++++ 1 file changed, 1338 insertions(+) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index 90ce5afb0a..a98e946cf5 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -2470,5 +2470,1343 @@ the loader. Adds the check that no module in the cf package names an implementation class." ``` +--- +### Task 6: `_decode.py` — the CF rules netCDF4 applies for free + +**Files:** +- Create: `lib/iris/fileformats/zarr/_decode.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/test__decode.py` + +**Interfaces:** +- Consumes: nothing. Every function here is pure and takes plain values, which + is the point: spec §6 wants decoding "unit-testable without a store at all". +- Produces: `decode_fill_value(value, dtype, variable_name)`, + `decode_parameters(attributes, *, dtype, storage_fill_value, zarr_format, + variable_name)`, `decoding_required(parameters)`, + `mask_and_scale(block, *, sentinels, valid_min, valid_max, scale_factor, + add_offset, dtype)`, and the three rules it composes. + +Two rules from spec §4.4 are the substance: + +- **The CF attribute governs masking and the storage field does not** — except + on version 2, where `Array.fill_value` is the mask source when no CF + attribute is present, "because version 2 stores predate the convention and + this is what their writers meant". This matches released xarray's + `use_zarr_fill_value_as_mask`. +- **A floating `_FillValue` is base64 on version 3, and that is a convention, + not a malformation.** `base64(struct.pack(" Any: + """Return a CF missing-value sentinel as a value of `dtype`. + + Parameters + ---------- + value : Any + The attribute's value, as the store holds it: a JSON number, or - for + a floating variable - the base64 of a little-endian ``float64``. + dtype : numpy.dtype + The variable's own dtype. The result is narrowed to it, because a + sentinel of a different width never equals anything in the array. + variable_name : str + Named in the warning, so the user can find the variable. + + Returns + ------- + numpy.generic or None + ``None`` when `value` is neither form, which is a producer error. + + Warns + ----- + IrisCfLoadWarning + When `value` cannot be read as a sentinel of `dtype`. + + """ + if isinstance(value, (bool, np.bool_)): + # A JSON true/false is a number to Python and nothing to CF. + decoded = None + elif isinstance(value, (int, float, np.number)): + decoded = value + elif isinstance(value, str) and dtype.kind == "f": + decoded = _decode_packed_double(value) + else: + decoded = None + + if decoded is None: + message = ( + "Ignoring a missing-value attribute that is neither a number nor " + f"the base64 of a float64, on variable {variable_name}: {value!r}" + ) + warnings.warn(message, category=iris.warnings.IrisCfLoadWarning) + return None + + return dtype.type(decoded) + + +def _decode_packed_double(value: str) -> float | None: + """Return the float in a base64 little-endian float64, or None.""" + try: + raw = base64.b64decode(value, validate=True) + except (binascii.Error, ValueError): + return None + if len(raw) != _PACKED_DOUBLE_BYTES: + return None + return struct.unpack(_PACKED_DOUBLE, raw)[0] +``` + +- [ ] **Step 4: Run the first group** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/test__decode.py -v +``` + +Expected: the seven `decode_fill_value` tests pass. + +- [ ] **Step 5: Write the failing tests for the masking precedence** + +Append to `test__decode.py`: + +```python +def parameters(attributes, *, dtype="i2", storage_fill_value=0, zarr_format=3): + return _decode.decode_parameters( + attributes, + dtype=np.dtype(dtype), + storage_fill_value=storage_fill_value, + zarr_format=zarr_format, + variable_name="v", + ) + + +def test_version_3_ignores_the_storage_fill_value(): + """An int16 [0, 1, 2] with no CF attributes loads with nothing masked. + + zarr-python defaults `fill_value` to zero, so reading it as a sentinel + masks valid data in the ordinary case, not the exotic one. + """ + assert parameters({}, storage_fill_value=0)["sentinels"] == () + + +def test_version_2_uses_the_storage_fill_value_when_there_is_no_attribute(): + result = parameters({}, storage_fill_value=0, zarr_format=2) + assert result["sentinels"] == (0,) + + +def test_a_cf_attribute_beats_the_storage_fill_value_on_version_2(): + result = parameters( + {"_FillValue": -999}, storage_fill_value=0, zarr_format=2 + ) + assert result["sentinels"] == (np.int16(-999),) + + +def test_both_cf_missing_value_attributes_are_sentinels(): + result = parameters({"_FillValue": -999, "missing_value": -888}) + assert set(result["sentinels"]) == {np.int16(-999), np.int16(-888)} + + +def test_valid_range_sets_both_bounds(): + result = parameters({"valid_range": [0, 10]}) + assert result["valid_min"] == 0 + assert result["valid_max"] == 10 + + +def test_valid_min_and_valid_max_are_read_separately(): + result = parameters({"valid_min": 1, "valid_max": 9}) + assert (result["valid_min"], result["valid_max"]) == (1, 9) + + +def test_packing_parameters_and_the_resulting_dtype(): + result = parameters({"scale_factor": 0.0001, "add_offset": 1.0}, dtype="u2") + assert result["scale_factor"] == 0.0001 + assert result["add_offset"] == 1.0 + assert result["dtype"] == np.dtype("f8") + + +def test_decoding_required_is_false_for_a_plain_array(): + assert _decode.decoding_required(parameters({})) is False + + +def test_decoding_required_is_true_for_each_rule(): + assert _decode.decoding_required(parameters({"_FillValue": -999})) is True + assert _decode.decoding_required(parameters({"valid_min": 1})) is True + assert _decode.decoding_required(parameters({"scale_factor": 2})) is True +``` + +- [ ] **Step 6: Write `decode_parameters` and `decoding_required`** + +Append to `_decode.py`: + +```python +def decode_parameters( + attributes, + *, + dtype: np.dtype, + storage_fill_value: Any, + zarr_format: int, + variable_name: str, +) -> dict: + """Return the CF decode parameters for one variable. + + Parameters + ---------- + attributes : mapping + The variable's CF attributes. + dtype : numpy.dtype + The variable's stored dtype, before any unpacking. + storage_fill_value : Any + ``zarr.Array.fill_value`` - what unwritten storage reads back as. A + mask source only on version 2, and only when the variable declares no + CF missing-value attribute. + zarr_format : int + 2 or 3. The one place the version changes what is decoded. + variable_name : str + Named in any warning. + + Returns + ------- + dict + The keyword arguments :func:`mask_and_scale` takes, always with the + same keys. :func:`decoding_required` says whether they do anything. + + """ + sentinels = [] + for name in ("_FillValue", "missing_value"): + if name in attributes: + decoded = decode_fill_value(attributes[name], dtype, variable_name) + if decoded is not None: + sentinels.append(decoded) + + if ( + not sentinels + and zarr_format < _FILL_VALUE_IS_A_MASK_BEFORE + and storage_fill_value is not None + ): + sentinels.append(storage_fill_value) + + valid_min = attributes.get("valid_min") + valid_max = attributes.get("valid_max") + valid_range = attributes.get("valid_range") + if valid_range is not None and len(valid_range) == 2: + valid_min, valid_max = valid_range + + scale_factor = attributes.get("scale_factor") + add_offset = attributes.get("add_offset") + + return { + "sentinels": tuple(sentinels), + "valid_min": valid_min, + "valid_max": valid_max, + "scale_factor": scale_factor, + "add_offset": add_offset, + "dtype": _unpacked_dtype(dtype, scale_factor, add_offset), + } + + +def _unpacked_dtype(dtype, scale_factor, add_offset) -> np.dtype: + """Return the dtype the unpacked values will have. + + The same calculation as + :func:`iris.fileformats.cf.loader._get_actual_dtype`, done here on plain + values so that the dtype the Dask graph declares and the dtype the decode + layer produces are one statement rather than two. + """ + probe = np.zeros(1, dtype=dtype) + if scale_factor is not None: + probe = scale_factor * probe + if add_offset is not None: + probe = add_offset + probe + return probe.dtype + + +def decoding_required(parameters: dict) -> bool: + """Whether these parameters would change anything. + + A variable with nothing to decode gets no ``map_blocks`` layer and an + unmasked Dask meta. Adding a masked meta without a layer beneath it would + be a meta that lies about its own graph. + """ + if parameters["sentinels"]: + return True + return any( + parameters[name] is not None + for name in ("valid_min", "valid_max", "scale_factor", "add_offset") + ) +``` + +- [ ] **Step 7: Run, then write the failing tests for `mask_and_scale`** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/test__decode.py -v +``` + +Expected: all pass so far. Then append: + +```python +def test_mask_by_sentinels(): + block = np.array([0, 1, -999], dtype="i2") + result = _decode.mask_by_sentinels(block, (np.int16(-999),)) + assert result.tolist() == [False, False, True] + + +def test_mask_by_a_nan_sentinel(): + """NaN never equals itself, so a NaN sentinel needs `isnan`, not `==`. + + The base64 `_FillValue` nearly every xarray-written store carries is NaN, + so this is the common case rather than the exotic one. + """ + block = np.array([1.0, np.nan], dtype="f4") + result = _decode.mask_by_sentinels(block, (np.float32("nan"),)) + assert result.tolist() == [False, True] + + +def test_mask_by_valid_range(): + block = np.array([-1, 5, 11], dtype="i2") + result = _decode.mask_by_valid_range(block, 0, 10) + assert result.tolist() == [True, False, True] + + +def test_unpack(): + block = np.array([1, 2], dtype="u2") + result = _decode.unpack(block, 0.5, 1.0, np.dtype("f8")) + np.testing.assert_allclose(result, [1.5, 2.0]) + assert result.dtype == np.dtype("f8") + + +def test_mask_and_scale_masks_the_packed_values(): + """Masking is applied before unpacking, as netCDF4 does it. + + CF section 2.5.1: `_FillValue` is "of the same type as its variable", so + it is a packed value and comparing it with an unpacked one matches + nothing. + """ + block = np.array([-999, 1, 2], dtype="i2") + result = _decode.mask_and_scale( + block, + sentinels=(np.int16(-999),), + valid_min=None, + valid_max=None, + scale_factor=0.5, + add_offset=0.0, + dtype=np.dtype("f8"), + ) + assert result.mask.tolist() == [True, False, False] + np.testing.assert_allclose(result.compressed(), [0.5, 1.0]) + + +def test_mask_and_scale_always_returns_a_masked_array(): + # The Dask meta says masked, so every block must be, mask or no mask. + block = np.array([1, 2], dtype="i2") + result = _decode.mask_and_scale( + block, + sentinels=(np.int16(-999),), + valid_min=None, + valid_max=None, + scale_factor=None, + add_offset=None, + dtype=np.dtype("i2"), + ) + assert isinstance(result, np.ma.MaskedArray) + assert not result.mask.any() +``` + +- [ ] **Step 8: Write the masking and unpacking functions** + +Append to `_decode.py`: + +```python +def mask_by_sentinels(block: np.ndarray, sentinels: tuple) -> np.ndarray: + """Return a boolean mask of the elements equal to any sentinel. + + A NaN sentinel is matched with :func:`numpy.isnan`, because NaN does not + equal itself and NaN is what nearly every version 3 store's base64 + ``_FillValue`` decodes to. + """ + mask = np.zeros(block.shape, dtype=bool) + for sentinel in sentinels: + if isinstance(sentinel, (float, np.floating)) and np.isnan(sentinel): + mask |= np.isnan(block) + else: + mask |= block == sentinel + return mask + + +def mask_by_valid_range(block: np.ndarray, valid_min, valid_max) -> np.ndarray: + """Return a boolean mask of the elements outside the valid range.""" + mask = np.zeros(block.shape, dtype=bool) + if valid_min is not None: + mask |= block < valid_min + if valid_max is not None: + mask |= block > valid_max + return mask + + +def unpack(block: np.ndarray, scale_factor, add_offset, dtype: np.dtype) -> np.ndarray: + """Return `block` unpacked by CF's scale and offset, as `dtype`.""" + result = block + if scale_factor is not None: + result = scale_factor * result + if add_offset is not None: + result = add_offset + result + return result.astype(dtype, copy=False) + + +def mask_and_scale( + block: np.ndarray, + *, + sentinels: tuple, + valid_min, + valid_max, + scale_factor, + add_offset, + dtype: np.dtype, +) -> np.ma.MaskedArray: + """Apply the CF decode rules to one block, in CF's order. + + Masking first, on the **packed** values, because CF declares a sentinel to + be "of the same type as its variable"; unpacking second. The result is + always a masked array, so that every block matches the Dask meta the + loader declares for the layer. + + Parameters + ---------- + block : numpy.ndarray + One Dask block of the raw stored values. + sentinels : tuple + Values meaning "missing", already narrowed to the stored dtype. + valid_min, valid_max : optional + CF's valid range. Values outside it are masked. + scale_factor, add_offset : optional + CF's packing. ``value * scale_factor + add_offset``. + dtype : numpy.dtype + The dtype of the result, which the Dask graph has already declared. + + Returns + ------- + numpy.ma.MaskedArray + + """ + mask = mask_by_sentinels(block, sentinels) + mask |= mask_by_valid_range(block, valid_min, valid_max) + return np.ma.masked_array( + unpack(block, scale_factor, add_offset, dtype), mask=mask + ) +``` + +- [ ] **Step 9: Run the module** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/test__decode.py -v +``` + +Expected: all pass. + +- [ ] **Step 10: Commit** + +Run §5.0, then: + +```bash +git add lib/iris/fileformats/zarr/_decode.py lib/iris/tests/unit/fileformats/zarr/test__decode.py +git commit -m "Implement the CF decoding Zarr does not do for itself + +Masking, valid-range masking and scale/offset unpacking, as pure functions +of values. The two rules worth reading twice: a floating _FillValue is +base64 on version 3, which is xarray's convention rather than a defect; +and Array.fill_value is a mask source on version 2 only, and only when the +variable declares no CF attribute of its own." +``` + +--- + +### Task 7: `read_data` — laziness, chunk alignment and the cache key + +**Files:** +- Modify: `lib/iris/fileformats/zarr/_dataset.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py` + +**Interfaces:** +- Consumes: `_decode.decode_parameters`, `decoding_required`, `mask_and_scale` + (Task 6); `ZarrDatasetVariable` (Task 3); + `iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES` and `_get_actual_dtype`, and + `iris._lazy_data.as_lazy_data(..., cache_key=...)` (**PR 3**). +- Produces: `ZarrDatasetVariable.read_data(chunking_policy)`, + `ZarrDatasetVariable.cache_key`, `ZarrDatasetVariable.zarr_format`, and the + module function `_align_chunks(chunks, store_chunks, shape, dtype, + dims_fixed, name)`. + +Three spec rules meet here. + +**No proxy.** `zarr.Array` is picklable, a chunk is an independent object in a +key-value store, and a store is a namespace rather than an open handle, so +every reason `NetCDFDataProxy` exists is absent. The array goes to +`as_lazy_data` directly. + +**Alignment aggregates upward only.** `_optimum_chunksize` expands a small +store chunk to a whole multiple, which is right; but when the store chunk is +*larger* than dask's target it shrinks below it, and subdividing a Zarr chunk +buys nothing — a read of any part of one fetches and decompresses the whole +object (F7 has the two reproductions). So the computed chunking is rounded +**up** to a whole multiple of the store chunk, never below, and the variable +warns when the store's own chunking exceeds the dask target. + +**The key carries the metadata, not just the address.** `CACHE` is a +process-wide `LRUCache(100)`, so an address goes stale the moment a store is +rewritten in place. + +- [ ] **Step 1: Write the failing tests for `_align_chunks`** + +Create `lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for the Zarr read-side chunk alignment. + +No store is built here: the guard is a function of chunk shapes, which is +most of the point of having it as one. +""" + +import numpy as np +import pytest + +from iris.fileformats.zarr._dataset import _align_chunks +from iris.warnings import IrisLoadWarning + +F8 = np.dtype("f8") + + +def test_a_whole_multiple_is_left_alone(): + assert _align_chunks((4096, 4096), (512, 512), (4096, 4096), F8, None, "v") == ( + 4096, + 4096, + ) + + +def test_a_sub_multiple_is_rounded_up(): + """A dask chunk that subdivides a store chunk costs N re-reads. + + A read of any part of a Zarr chunk fetches and decompresses the whole + object, so N dask blocks over one store chunk is N times the bytes and N + times the CPU - and, over a remote store, N HTTP GETs for one object. + """ + with pytest.warns(IrisLoadWarning, match="v"): + result = _align_chunks((2000, 8000), (8000, 8000), (16000, 16000), F8, None, "v") + assert result == (8000, 8000) + + +def test_rounding_up_is_capped_at_the_shape(): + assert _align_chunks((2000, 6000), (6000, 6000), (6000, 6000), F8, None, "v") == ( + 6000, + 6000, + ) + + +def test_an_unchunked_store_is_left_alone(): + assert _align_chunks((10,), None, (100,), F8, None, "v") == (10,) + + +def test_an_explicit_chunk_control_setting_wins_but_warns(): + """A deliberate instruction is honoured, on the same terms. + + CHUNK_CONTROL marks the dimensions it set as fixed. Overriding them would + make the keyword a suggestion. + """ + with pytest.warns(IrisLoadWarning, match="v"): + result = _align_chunks( + (100, 8000), (8000, 8000), (16000, 16000), F8, (True, False), "v" + ) + assert result[0] == 100 + + +def test_a_small_store_chunk_does_not_warn(): + # Dask's target is the binding constraint here, which is the ordinary + # case and nothing for the user to act on. + with warnings.catch_warnings(): + warnings.simplefilter("error") + result = _align_chunks((1024, 1024), (512, 512), (4096, 4096), F8, None, "v") + assert result == (1024, 1024) + + +@pytest.mark.parametrize( + "chunks, store_chunks, shape", + [ + ((2000, 8000), (8000, 8000), (16000, 16000)), + ((1024, 1024), (512, 512), (4096, 4096)), + ((3000, 10000), (1000, 1000), (10000, 10000)), + ], +) +def test_the_result_tiles_the_store_grid_exactly(chunks, store_chunks, shape): + with warnings.catch_warnings(): + warnings.simplefilter("ignore", IrisLoadWarning) + result = _align_chunks(chunks, store_chunks, shape, F8, None, "v") + for size, store_size, extent in zip(result, store_chunks, shape): + assert size % store_size == 0 or size == extent +``` + +`import warnings` goes at the top of the module with the others. + +- [ ] **Step 2: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py -v +``` + +Expected: `ImportError: cannot import name '_align_chunks'`. + +- [ ] **Step 3: Write `_align_chunks`** + +Append to `lib/iris/fileformats/zarr/_dataset.py`, after `_assemble_dimensions`: + +```python +def _fixed_flags(dims_fixed, ndim: int) -> tuple: + """Return one "do not change this" flag per dimension. + + `iris.fileformats.cf.loader._chunks_from_chunk_control` answers ``(None,)`` + when nothing is fixed, whatever the rank, so a length that does not match + means nothing is fixed. + """ + if dims_fixed is None or len(dims_fixed) != ndim: + return (False,) * ndim + return tuple(bool(flag) for flag in dims_fixed) + + +def _align_chunks( + chunks: tuple, + store_chunks: tuple | None, + shape: tuple, + dtype: np.dtype, + dims_fixed, + name: str, +) -> tuple: + """Round `chunks` up to whole multiples of the store's own chunk shape. + + A Zarr chunk is the atomic unit of storage: reading any part of one + fetches and decompresses the whole object. So a Dask chunking that + subdivides the store's costs a re-read per subdivision - and over a remote + store, one HTTP GET per subdivision for the same object. Aggregating + upward costs nothing, because peak memory per task is a whole decompressed + store chunk either way. + + Parameters + ---------- + chunks : tuple of int + The chunking computed for this variable, already through + :func:`iris._lazy_data._optimum_chunksize`. + store_chunks : tuple of int, optional + The store's own chunk shape. ``None`` for an unchunked store, which + constrains nothing. + shape : tuple of int + The variable's full shape. A result is never larger than this. + dtype : numpy.dtype + The stored dtype, for deciding whether one store chunk already exceeds + Dask's target chunk size. + dims_fixed : sequence of bool, optional + Dimensions an explicit ``CHUNK_CONTROL`` setting pinned. These are + left exactly as given: the keyword is an instruction, not a + suggestion. + name : str + The variable's name, for the warning. + + Returns + ------- + tuple of int + + Warns + ----- + IrisLoadWarning + When the result cannot tile the store grid without subdividing it - + either because one store chunk already exceeds Dask's target size, so + the store's layout is the binding constraint, or because an explicit + setting subdivides it. + + """ + if store_chunks is None: + return tuple(chunks) + + fixed = _fixed_flags(dims_fixed, len(shape)) + aligned = [] + subdivided = False + for size, store_size, extent, is_fixed in zip(chunks, store_chunks, shape, fixed): + if is_fixed: + aligned.append(size) + subdivided |= size < store_size + continue + whole = -(-size // store_size) * store_size + aligned.append(min(whole, extent)) + subdivided |= whole > extent and extent % store_size != 0 + + if _store_chunk_exceeds_dask_target(store_chunks, dtype): + subdivided = True + + if subdivided: + message = ( + "A Zarr chunk is read whole, so this store's chunking is the " + "binding constraint on memory per task rather than Dask's target " + f"chunk size: {name}" + ) + warnings.warn(message, category=iris.warnings.IrisLoadWarning) + + return tuple(aligned) + + +def _store_chunk_exceeds_dask_target(store_chunks: tuple, dtype: np.dtype) -> bool: + """Whether one whole store chunk is larger than Dask's target chunk size.""" + target = dask.utils.parse_bytes(dask.config.get("array.chunk-size")) + return int(np.prod(store_chunks)) * dtype.itemsize > target +``` + +Add `import dask`, `import dask.config` and `import dask.utils` to the module +imports. + +- [ ] **Step 4: Run the alignment tests** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py -v +``` + +Expected: all pass. The two regression cases are F7's: `(8000, 8000)` over +`(16000, 16000)` and `(6000, 6000)` over `(6000, 6000)`, both of which +`_optimum_chunksize` subdivides on its own. + +- [ ] **Step 5: Write the failing tests for `read_data`** + +Create `lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :meth:`ZarrDatasetVariable.read_data`.""" + +import dask.array as da +import numpy as np +import pytest +import zarr + +from iris.fileformats.cf import loader as cf_loader +from iris.fileformats.zarr._dataset import ZarrDataset + +from .. import stores + + +def variable(location, name="air_temperature"): + return ZarrDataset(location).variables[name] + + +def policy(): + """The chunking policy a loader outside any CHUNK_CONTROL supplies.""" + return None, None + + +def test_a_small_variable_is_read_whole(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + result = variable(location).read_data(policy) + assert not isinstance(result, da.Array) + np.testing.assert_array_equal(np.asarray(result), stores.DATA) + + +def test_a_small_variable_never_consults_the_policy(tmp_path): + """Under CHUNK_CONTROL.from_file() the policy raises for an unchunked store. + + A variable small enough to read whole must reach its data without asking, + or `from_file` starts rejecting small cubes that load today. + """ + + def raising_policy(): + raise AssertionError("chunking policy consulted for a small variable") + + location = stores.make_store(tmp_path / "s.zarr") + variable(location).read_data(raising_policy) + + +def test_a_large_variable_is_lazy(tmp_path, monkeypatch): + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + location = stores.make_store(tmp_path / "s.zarr") + result = variable(location).read_data(policy) + assert isinstance(result, da.Array) + np.testing.assert_array_equal(result.compute(), stores.DATA) + + +def test_a_plain_array_gets_an_unmasked_meta(tmp_path, monkeypatch): + # No decode layer, so no masked meta: a masked meta with nothing beneath + # it is a meta that lies about its own graph. + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + location = stores.make_store(tmp_path / "s.zarr") + result = variable(location).read_data(policy) + assert not isinstance(result._meta, np.ma.MaskedArray) + + +def test_a_decoded_array_gets_a_masked_meta(tmp_path, monkeypatch): + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + group = zarr.open_group(store=tmp_path / "m.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, "v", (4,), "i2", 3, dimensions=("t",), + attributes={"_FillValue": -999}, + ) + array[:] = [-999, 1, 2, 3] + + result = ZarrDataset(str(tmp_path / "m.zarr")).variables["v"].read_data(policy) + assert isinstance(result._meta, np.ma.MaskedArray) + realised = result.compute() + assert realised.mask.tolist() == [True, False, False, False] +``` + +- [ ] **Step 6: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py -v +``` + +Expected: FAIL, +`TypeError: Can't instantiate abstract class ZarrDatasetVariable with abstract method read_data` +— `read_data` is abstract on the interface, so until it is written nothing in +the Zarr package instantiates at all. If Tasks 3–5 passed, the class already +has a stub; delete it and watch this failure appear, so that the test is known +to be able to fail. + +- [ ] **Step 7: Write `read_data`, `cache_key` and `zarr_format`** + +In `ZarrDatasetVariable`, add to the Zarr-only members: + +```python + @property + def zarr_format(self) -> int: + """2 or 3. The one Zarr fact that changes how a value is read.""" + return self._array.metadata.zarr_format + + @property + def cache_key(self) -> str: + """An identity for this array that goes stale when the array does. + + `iris._lazy_data.CACHE` is process-wide, so an address alone is not + enough: a store rewritten in place keeps its address and changes its + contents. The metadata document covers shape, chunks, shards, dtype, + fill value, codecs, dimension names and attributes in one value, and + is treated as opaque - listing the parts Iris cares about would go + stale, silently, the first time zarr-python added a field that + mattered. + """ + return json.dumps( + [self._location, self._array.path, self._array.metadata.to_dict()], + sort_keys=True, + ) +``` + +and, among the interface members, after `__setitem__`: + +```python + def read_data(self, chunking_policy): + """Return this array's data, lazily where a lazy array is worth it. + + Parameters + ---------- + chunking_policy : callable + Called with no arguments, returning the ``(chunks, dims_fixed)`` + pair to build a lazy array with. Called only once the result is + known to be lazy. + + Returns + ------- + numpy.ndarray, numpy.ma.MaskedArray or dask.array.Array + Real data for a small array, lazy data otherwise. Masked whenever + the variable declares anything to decode. + + Notes + ----- + There is no data proxy. ``zarr.Array`` is picklable, a chunk is an + independent object in a key-value store so concurrent reads need no + lock, and a store is a namespace rather than an open file handle. CF + decoding is a ``map_blocks`` layer over the raw array rather than + something hidden inside ``__getitem__``, so it is visible in the + graph and testable without a store. + + """ + # Deferred, and the module rather than its members: the CF package + # reaches this one through iris.fileformats.zarr.loader, so a + # module-level import would be a cycle; and the tests patch + # _LAZYVAR_MIN_BYTES on the module object. + from iris.fileformats.cf import loader + + parameters = _decode.decode_parameters( + self._attributes, + dtype=self.dtype, + storage_fill_value=self.fill_value, + zarr_format=self.zarr_format, + variable_name=self.name, + ) + decoding = _decode.decoding_required(parameters) + + if self.size * self.dtype.itemsize < loader._LAZYVAR_MIN_BYTES: + # A lazy array would cost more memory AND more time to access. + block = np.asanyarray(self._array[...]) + if decoding: + return _decode.mask_and_scale(block, **parameters) + return block + + chunks, dims_fixed = chunking_policy() + raw_meta = np.empty((0,) * self.ndim, dtype=self.dtype) + + if chunks == "auto": + # CHUNK_CONTROL.as_dask(): the user asked for Dask's own policy, + # so the store's layout is deliberately not consulted. + lazy = as_lazy_data( + self._array, chunks="auto", meta=raw_meta, cache_key=self.cache_key + ) + else: + chunks = _optimum_chunksize( + chunks, shape=self.shape, dtype=self.dtype, dims_fixed=dims_fixed + ) + chunks = _align_chunks( + chunks, self.chunking, self.shape, self.dtype, dims_fixed, self.name + ) + # Fixed in every dimension, because the alignment above is the + # final word: as_lazy_data would otherwise re-run the + # optimisation and undo it. + lazy = as_lazy_data( + self._array, + chunks=chunks, + meta=raw_meta, + dims_fixed=(True,) * self.ndim, + cache_key=self.cache_key, + ) + + if not decoding: + return lazy + + return da.map_blocks( + _decode.mask_and_scale, + lazy, + dtype=parameters["dtype"], + meta=np.ma.masked_array(np.empty((0,) * self.ndim, parameters["dtype"])), + **parameters, + ) +``` + +Add to the module imports: `import json`, `import dask.array as da`, +`from iris._lazy_data import _optimum_chunksize, as_lazy_data`, and +`from . import _decode`. + +Note `chunks` arrives as a list from `_chunks_from_chunk_control`; +`_optimum_chunksize` returns a tuple, and `_align_chunks` returns one, so what +reaches `as_lazy_data` is hashable — which matters, because it goes into the +cache key. + +- [ ] **Step 8: Run, then write the cache-key tests** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py -v +``` + +Expected: all pass. Then append the staleness and sensitivity cases from +spec §6: + +```python +def key_of(location, name="v"): + return ZarrDataset(location).variables[name].cache_key + + +def test_the_key_is_stable_across_a_reopen(tmp_path): + group = zarr.open_group(store=tmp_path / "k.zarr", mode="w", zarr_format=3) + stores.make_array(group, "v", (4,), "i4", 3, dimensions=("t",)) + location = str(tmp_path / "k.zarr") + assert key_of(location) == key_of(location) + + +def test_the_key_changes_when_the_shape_does(tmp_path): + """A store rewritten in place keeps its address and changes its contents. + + Keyed on the address alone, an array reopened after its shape changed + from (4,) to (6,) returns the cached four values, silently truncating - + and that is weaker than the netCDF key it replaces, since + repr(NetCDFDataProxy) already carries shape and dtype. + """ + location = str(tmp_path / "k.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + stores.make_array(group, "v", (4,), "i4", 3, dimensions=("t",), chunks=(2,)) + before = key_of(location) + + group = zarr.open_group(store=location, mode="a", zarr_format=3) + group.create_array( + "v", shape=(6,), dtype="i4", chunks=(2,), dimension_names=("t",), + overwrite=True, + ) + assert key_of(location) != before + + +def test_the_key_changes_when_the_fill_value_does(tmp_path): + location = str(tmp_path / "k.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + group.create_array("v", shape=(4,), dtype="i4", fill_value=-1) + before = key_of(location) + + group = zarr.open_group(store=location, mode="a", zarr_format=3) + group.create_array("v", shape=(4,), dtype="i4", fill_value=-2, overwrite=True) + assert key_of(location) != before + + +@pytest.mark.parametrize( + "change", + [ + {"shape": (8,)}, + {"chunks": (4,)}, + {"shards": (8,)}, + {"dtype": "f8"}, + {"fill_value": -7}, + {"dimension_names": ("other",)}, + ], +) +def test_the_key_is_sensitive_to_every_metadata_field(tmp_path, change): + """`Array.metadata.to_dict()` is zarr-python's rendering, not a guarantee. + + Nothing in the Zarr specification defines what that method returns, so + these are what fail if a release stops carrying a field the key needs. + """ + base = dict( + shape=(4,), dtype="i4", chunks=(2,), fill_value=-1, + dimension_names=("t",), shards=None, + ) + location = str(tmp_path / "k.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + group.create_array("v", **base) + before = key_of(location) + + group = zarr.open_group(store=location, mode="a", zarr_format=3) + group.create_array("v", **{**base, **change}, overwrite=True) + assert key_of(location) != before + + +def test_the_key_is_sensitive_to_attributes(tmp_path): + location = str(tmp_path / "k.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + array = group.create_array("v", shape=(4,), dtype="i4") + before = key_of(location) + array.attrs["units"] = "K" + assert key_of(location) != before + + +def test_two_cubes_sharing_an_array_share_one_dask_array(tmp_path, monkeypatch): + # What the cache is for: many cubes in one store sharing a coordinate. + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + location = stores.make_store(tmp_path / "s.zarr") + first = variable(location).read_data(policy) + second = variable(location).read_data(policy) + assert first.name == second.name +``` + +The sensitivity test's `shards` case needs `chunks` to divide `shards`; keep +`chunks=(2,)` and `shards=(8,)` together when writing it, and drop the case +with a comment if zarr-python refuses the combination rather than loosening the +assertion. + +- [ ] **Step 9: Write the sharded read-unit test** + +Append to `test_read_data.py`: + +```python +class CountingStore(zarr.storage.WrapperStore): + """Wrap a store, adding every byte it returns to a caller-owned tally. + + zarr-python copies a store when it opens it read-only, through + `with_read_only` and `_with_store`, so a tally kept on the instance is + never the one the live store writes to, and the test passes vacuously. + """ + + def __init__(self, store, tally): + super().__init__(store=store) + self.tally = tally + + def _with_store(self, store): + return type(self)(store, self.tally) + + async def get(self, key, prototype, byte_range=None): + buffer = await self._store.get(key, prototype, byte_range) + if buffer is not None: + self.tally.append(len(buffer)) + return buffer + + async def get_partial_values(self, prototype, key_ranges): + buffers = await self._store.get_partial_values(prototype, key_ranges) + self.tally.extend(len(b) for b in buffers if b is not None) + return buffers + + +def bytes_to_read_first_chunk(location, dask_chunks): + """Return the bytes fetched reading the first eight values of "v".""" + tally = [] + store = CountingStore(zarr.storage.LocalStore(location), tally) + array = zarr.open_group(store=store, mode="r")["v"] + tally.clear() + lazy = da.from_array(array, chunks=dask_chunks, meta=np.empty((0,), "i4")) + lazy = da.map_blocks( + lambda block: np.ma.masked_array(block), lazy, dtype="i4" + ) + lazy[:8].compute() + return sum(tally) + + +def test_a_sharded_array_is_read_by_the_inner_chunk(tmp_path): + """Shard-sized Dask blocks fetch the whole shard, because of the layer. + + Bare Dask slicing can push a slice down into the array's own __getitem__, + but a map_blocks layer in between cannot, so the block is materialised in + full. The absolute byte counts are zarr-version-dependent; the comparison + is not. + """ + location = str(tmp_path / "sh.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + array = group.create_array( + "v", shape=(1024,), dtype="i4", chunks=(8,), shards=(1024,) + ) + array[:] = np.arange(1024, dtype="i4") + + inner = bytes_to_read_first_chunk(location, 8) + whole_shard = bytes_to_read_first_chunk(location, 1024) + assert inner < whole_shard + + # And the variable chooses the inner chunk for exactly this reason. + assert ZarrDataset(location).variables["v"].chunking == (8,) +``` + +- [ ] **Step 10: Run the module** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/ -v +``` + +Expected: all pass. + +- [ ] **Step 11: Write the test for Review Focus 4 — the unpacked dtype** + +Append to `test_read_data.py`: + +```python +def test_the_declared_dtype_matches_what_the_decode_layer_produces(tmp_path, monkeypatch): + """A packed variable's lazy dtype must equal its realised dtype. + + On the netCDF path netCDF4 does the arithmetic and reports the dtype. On + the Zarr path Iris does both, in two separately written statements, and + Dask does not check them against each other: a mismatch builds a graph + and surfaces later as a wrong dtype on the cube. + """ + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "b02", + (4,), + "u2", + 3, + dimensions=("p",), + attributes={"scale_factor": 0.0001, "add_offset": -0.1}, + ) + array[:] = [1000, 2000, 3000, 4000] + + lazy = ZarrDataset(str(tmp_path / "p.zarr")).variables["b02"].read_data(policy) + realised = lazy.compute() + + assert lazy.dtype == realised.dtype + assert lazy.dtype == np.dtype("f8") + np.testing.assert_allclose(realised, [0.0, 0.1, 0.2, 0.3]) + + +def test_the_declared_dtype_matches_the_cf_loader_calculation(tmp_path): + """And it is the same answer `_get_actual_dtype` gives. + + The cube's data variable and its coordinates must agree about what a + packed value becomes, and they reach that answer through different code. + """ + group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3) + stores.make_array( + group, "v", (4,), "u2", 3, + attributes={"scale_factor": np.float32(0.01)}, + ) + variable = ZarrDataset(str(tmp_path / "p.zarr")).variables["v"] + from iris.fileformats.zarr import _decode + + parameters = _decode.decode_parameters( + variable.attributes, + dtype=variable.dtype, + storage_fill_value=variable.fill_value, + zarr_format=variable.zarr_format, + variable_name=variable.name, + ) + assert parameters["dtype"] == cf_loader._get_actual_dtype(variable) +``` + +- [ ] **Step 12: Run everything built so far, and commit** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/ lib/iris/tests/unit/fileformats/cf/ lib/iris/tests/unit/lazy_data/ -q +``` + +Expected: all pass. Then run §5.0 and: + +```bash +git add lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr +git commit -m "Read Zarr data lazily, aligned to the store's own chunks + +No proxy: a zarr.Array is picklable and a store is a namespace, so the +array goes to as_lazy_data directly and CF decoding is a visible map_blocks +layer. Chunking is rounded up to whole multiples of the store's own, never +below, because reading any part of a Zarr chunk fetches the whole object. +The Dask cache key carries the array's metadata document rather than its +address, which alone goes stale when a store is rewritten in place." +``` + --- From cb7f407386d0a13528ba9e66509c60f5b6e989a1 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 12:10:29 +0000 Subject: [PATCH 06/56] Add PR 4 plan tasks 8-10: spans, CFReader, and the Zarr loader Task 8 is the one change a netCDF file could notice, so it runs its suite on both backends. Task 9 lets CFReader take an already-open CF dataset of any format. Task 10 builds cubes, and decides what a group holding no data variables says to the user. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 970 +++++++++++++++++- 1 file changed, 954 insertions(+), 16 deletions(-) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index a98e946cf5..2e2676f7b0 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -2410,18 +2410,21 @@ Create `lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py`: # # This file is part of Iris and is released under the BSD license. # See LICENSE in the root of the repository for full licensing details. -"""The CF layer must not know which storage implementation it holds. - -Testing the interface - `isinstance(thing, CFDataset)` - is how generic code -is *supposed* to accept any backend. Testing an implementation is what makes -a third backend a rewrite instead of a new module. +"""The CF layer must not branch on which storage implementation it holds. + +`isinstance(thing, CFDataset)` tests the *interface*, which is how generic +code is supposed to accept any backend, and `CFReader` does exactly that. +What is banned is testing an *implementation*: that is what makes a third +backend a rewrite rather than a new module. Constructing one is a third +thing again, and legitimate - `CFReader` opens a `NetCDFDataset` for a bare +path, which is a default, not a branch on something it was handed. """ from pathlib import Path import iris.fileformats.cf -#: Implementation class names no module in the `cf` package may name. +#: Implementation class names that may not appear in a type test. IMPLEMENTATION_NAMES = ( "NetCDFDataset", "NetCDFDatasetVariable", @@ -2429,6 +2432,9 @@ IMPLEMENTATION_NAMES = ( "ZarrDatasetVariable", ) +#: The spellings of a type test. `__class__` catches the hand-rolled form. +TYPE_TESTS = ("isinstance", "issubclass", "__class__", "type(") + def cf_package_sources(): """Yield `(path, text)` for every module in `iris.fileformats.cf`.""" @@ -2437,24 +2443,23 @@ def cf_package_sources(): yield path, path.read_text(encoding="utf-8") -def test_no_module_names_an_implementation_class(): +def test_no_module_type_tests_an_implementation_class(): offenders = [] for path, text in cf_package_sources(): for line_number, line in enumerate(text.splitlines(), start=1): if line.lstrip().startswith("#"): continue - for name in IMPLEMENTATION_NAMES: - if name in line: - offenders.append(f"{path.name}:{line_number}: {line.strip()}") + names_one = any(name in line for name in IMPLEMENTATION_NAMES) + tests_type = any(spelling in line for spelling in TYPE_TESTS) + if names_one and tests_type: + offenders.append(f"{path.name}:{line_number}: {line.strip()}") assert offenders == [] ``` -Note this test will **fail** until Task 9 removes `cf/_reader.py`'s import of -`NetCDFDataset`. Write it here, see it fail, and let Task 9 turn it green; -Task 9's steps say so. Mark it `@pytest.mark.xfail(strict=True, reason=...)` -in between **only if** you need an intervening commit to be green — and delete -the marker in Task 9, because an xfail that starts passing is a failure -(`AGENTS.md` gotcha 1). +This passes as written, and the way it fails is by someone adding the branch +it forbids. Prove it can fail before trusting it: temporarily add +`if isinstance(self._dataset, NetCDFDataset): pass` to `cf/_reader.py`, re-run, +see one offender, and remove it. - [ ] **Step 5: Commit** @@ -3808,5 +3813,938 @@ The Dask cache key carries the array's metadata document rather than its address, which alone goes stale when a store is rewritten in place." ``` +--- +### Task 8: `spans` matches on `(name, size)` pairs + +**Files:** +- Modify: `lib/iris/fileformats/cf/_variables.py` +- Create: `lib/iris/tests/unit/fileformats/cf/test_spans.py` + +**Interfaces:** +- Consumes: `ZarrDataset` (Task 4), `NetCDFDataset`, `stores.make_array`. +- Produces: `CFVariable._dimension_identities()` and the module-private + `_AnonymousDimension`; `CFVariable.spans` and the three overrides in + `CFBoundaryVariable`, `CFClimatologyVariable` and `CFLabelVariable` matching + on pairs. + +This is the one change in this pull request that can alter what loads from an +existing netCDF file, so it is on its own and its tests run on both backends. +Spec §4.4: + +> **The identity that links two variables is the pair `(name, size)`, and an +> anonymous dimension has no identity.** +> +> - Two variables share a dimension when they name it identically *and* it is +> the same length in both. A name reused at two lengths links nothing. +> - An anonymous dimension never matches anything — not a named dimension, and +> not another anonymous dimension, including within the same variable. +> - A variable may name a dimension the dataset does not list, and may name +> some of its dimensions and not others. Neither is an error. + +For netCDF this changes nothing by construction: dimension names are unique +within a group, so `(name, size)` and `name` are the same key, and no netCDF +variable has an anonymous dimension. + +- [ ] **Step 1: Write the failing shared suite** + +Create `lib/iris/tests/unit/fileformats/cf/test_spans.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""`CFVariable.spans`, asked of both storage backends. + +Dimension matching is a pair, `(name, size)`, not a bare name - which is a +widening to admit what Zarr allows, not a change to what netCDF does. These +tests run against both so that the second half of that sentence is checked +rather than asserted. +""" + +import numpy as np +import pytest + +from iris.fileformats.cf import ( + CFBoundaryVariable, + CFDataVariable, + CFLabelVariable, +) + +zarr = pytest.importorskip("zarr") + +from iris.fileformats.netcdf._dataset import NetCDFDataset # noqa: E402 +from iris.fileformats.zarr._dataset import ZarrDataset # noqa: E402 + + +def build_netcdf(path, described): + """Write `described` as a netCDF file and return its open dataset.""" + dataset = NetCDFDataset(path, mode="w") + sizes = {} + for dimensions, shape in described.values(): + for name, size in zip(dimensions, shape): + sizes.setdefault(name, size) + for name, size in sizes.items(): + dataset.create_dimension(name, size) + for name, (dimensions, shape) in described.items(): + dataset.create_variable(name, np.dtype("f4"), dimensions) + return dataset + + +def build_zarr(path, described): + """Write `described` as a Zarr store and return its open dataset.""" + group = zarr.open_group(store=path, mode="w", zarr_format=3) + for name, (dimensions, shape) in described.items(): + group.create_array( + name, shape=shape, dtype="f4", dimension_names=dimensions + ) + return ZarrDataset(str(path)) + + +@pytest.fixture(params=[build_netcdf, build_zarr], ids=["netcdf", "zarr"]) +def build(request, tmp_path): + """Return a callable giving `{name: CFDataVariable}` for a description.""" + suffix = ".nc" if request.param is build_netcdf else ".zarr" + + def make(described): + dataset = request.param(tmp_path / f"spans{suffix}", described) + return { + name: CFDataVariable(name, dataset.variables[name]) + for name in described + } + + return make + + +def test_identical_dimensions_span(build): + variables = build({"data": (("y", "x"), (3, 2)), "aux": (("y", "x"), (3, 2))}) + assert variables["aux"].spans(variables["data"]) is True + + +def test_a_subset_spans(build): + variables = build({"data": (("y", "x"), (3, 2)), "aux": (("y",), (3,))}) + assert variables["aux"].spans(variables["data"]) is True + + +def test_a_different_name_does_not_span(build): + variables = build({"data": (("y", "x"), (3, 2)), "aux": (("t",), (3,))}) + assert variables["aux"].spans(variables["data"]) is False + + +def test_a_scalar_always_spans(build): + variables = build({"data": (("y", "x"), (3, 2)), "grid": ((), ())}) + assert variables["grid"].spans(variables["data"]) is True + + +def test_a_bounds_variable_ignores_its_extent_dimension(build): + variables = build( + {"data": (("y",), (3,)), "bounds": (("y", "bnds"), (3, 2))} + ) + bounds = CFBoundaryVariable("bounds", variables["bounds"].cf_data) + assert bounds.spans(variables["data"]) is True + + +def test_a_label_variable_ignores_its_string_length_dimension(build): + variables = build( + {"data": (("y",), (3,)), "label": (("y", "strlen"), (3, 8))} + ) + label = CFLabelVariable("label", variables["label"].cf_data) + assert label.spans(variables["data"]) is True +``` + +- [ ] **Step 2: Run it against today's code** + +```bash +pytest lib/iris/tests/unit/fileformats/cf/test_spans.py -v +``` + +Expected: all pass on netCDF, all pass on Zarr. These are the cases that must +be **unchanged** by this task — run them before the change so that the +"netCDF behaviour is unchanged" claim has a before as well as an after. + +- [ ] **Step 3: Write the failing tests for the new rules** + +Append to `test_spans.py`: + +```python +def test_a_name_at_two_sizes_does_not_link(tmp_path): + """Zarr-only: a name guarantees nothing across arrays. + + The version 3 core specification "does not place any restrictions on the + use of the same dimension name across multiple arrays within the same Zarr + hierarchy", so two arrays may mean different things by one name. Matching + on the name alone would produce a cube whose coordinate is the wrong + length. netCDF cannot express this: a dimension name has one size per + group. + """ + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array("data", shape=(3,), dtype="f4", dimension_names=("t",)) + group.create_array("aux", shape=(5,), dtype="f4", dimension_names=("t",)) + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + data = CFDataVariable("data", dataset.variables["data"]) + aux = CFDataVariable("aux", dataset.variables["aux"]) + assert aux.spans(data) is False + + +def test_an_anonymous_dimension_matches_nothing(tmp_path): + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array("data", shape=(3,), dtype="f4", dimension_names=("t",)) + group.create_array("aux", shape=(3,), dtype="f4") + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + data = CFDataVariable("data", dataset.variables["data"]) + aux = CFDataVariable("aux", dataset.variables["aux"]) + assert aux.spans(data) is False + + +def test_two_anonymous_dimensions_of_one_size_do_not_match(tmp_path): + """Not even each other, and not within one variable. + + There is no information to match on, so claiming a match would be + invention. + """ + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array("data", shape=(3, 3), dtype="f4") + group.create_array("aux", shape=(3,), dtype="f4") + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + data = CFDataVariable("data", dataset.variables["data"]) + aux = CFDataVariable("aux", dataset.variables["aux"]) + assert aux.spans(data) is False + + +def test_a_partly_named_variable_matches_on_the_named_part(tmp_path): + # "A variable may name some of its dimensions and not others." The named + # ones still have to be present in the target, and the anonymous one + # still matches nothing - so this does not span. + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array( + "data", shape=(3, 2), dtype="f4", dimension_names=("y", "x") + ) + group.create_array( + "aux", shape=(3, 2), dtype="f4", dimension_names=("y", None) + ) + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + data = CFDataVariable("data", dataset.variables["data"]) + aux = CFDataVariable("aux", dataset.variables["aux"]) + assert aux.spans(data) is False +``` + +- [ ] **Step 4: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/fileformats/cf/test_spans.py -v -k "two_sizes or anonymous or partly" +``` + +Expected: four FAILs — today's `set(self.dimensions).issubset(...)` matches a +bare name, so a name at two sizes spans, and `None` matches `None`. + +- [ ] **Step 5: Add the identity helper** + +In `lib/iris/fileformats/cf/_variables.py`, after `_NCZARR_SCALAR_DIMENSION`: + +```python +class _AnonymousDimension: + """One dimension the store gives no name. + + A distinct instance per dimension, relying on object identity for + equality, so that an anonymous dimension matches nothing: not a named + dimension, not another anonymous one, and not itself as seen from a + second variable. There is nothing to match on, so claiming a match would + be invention. + + Zarr's version 3 core specification makes ``dimension_names`` optional and + allows a null entry, so a conforming store can present this and it must + not be an error. No netCDF variable ever has one. + """ + + def __repr__(self) -> str: + return "" +``` + +and on `CFVariable`, beside `spans`: + +```python + def _dimension_identities(self) -> tuple: + """Return one hashable identity per dimension this variable spans. + + A named dimension's identity is the pair ``(name, size)``: two + variables share a dimension when they name it identically *and* it is + the same length in both. A name reused at two lengths links nothing, + which is the only reading that cannot produce a wrong cube. + + An unnamed dimension gets a fresh :class:`_AnonymousDimension`, which + equals nothing but itself. + + Relies on the :class:`~iris.fileformats.cf.dataset.CFDatasetVariable` + invariant that ``dimensions`` and ``shape`` have the same length. + """ + return tuple( + (name, size) if name is not None else _AnonymousDimension() + for name, size in zip(self.dimensions, self.shape) + ) +``` + +- [ ] **Step 6: Rewrite the four `spans` methods** + +`CFVariable.spans` (keep the docstring, replace the comparison): + +```python + # Scalar variables always span the target variable. + if self._is_scalar(): + return True + + source = self._dimension_identities() + target = cf_variable._dimension_identities() + return set(source).issubset(target) +``` + +`CFBoundaryVariable.spans`, `CFClimatologyVariable.spans` and +`CFLabelVariable.spans` keep their own comments about which extent dimension +they ignore, and become: + +```python + # Scalar variables always span the target variable. + result = True + if not self._is_scalar(): + source = self._dimension_identities() + target = cf_variable._dimension_identities() + # Ignore the bounds extent dimension. + result = set(source[:-1]).issubset(target) or set(source[1:]).issubset( + target + ) + return result +``` + +with "bounds" replaced by "climatology" and "label string length" in the other +two, exactly as they read today. + +- [ ] **Step 7: Run the whole suite** + +```bash +pytest lib/iris/tests/unit/fileformats/cf/test_spans.py -v +pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/netcdf/ -q +``` + +Expected: the new tests pass, and **nothing else changes**. This is the claim +the task exists to support: a netCDF file cannot tell the difference. + +- [ ] **Step 8: Pin the two claims that keep this change contained (F16)** + +Append to `test_spans.py`: + +```python +def test_a_non_spanning_variable_is_dropped_and_promoted(tmp_path): + """An anonymous-dimension variable becomes a cube of its own. + + This is what keeps `None` out of `iris.fileformats._nc_load_rules.helpers`, + which matches dimensions by name and would read `None in (None, "x")` as a + match. `CFReader._build_cf_groups` drops any variable whose `spans` is + False from the referrer's group and promotes it. + """ + from iris.fileformats.cf import CFReader + + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + data = group.create_array( + "data", shape=(3,), dtype="f4", dimension_names=("t",) + ) + data.attrs["coordinates"] = "aux" + group.create_array("aux", shape=(3,), dtype="f4") + + with CFReader(ZarrDataset(str(tmp_path / "s.zarr"))) as reader: + assert "aux" not in reader.cf_group["data"].cf_group + assert "aux" in reader.cf_group.promoted + + +def test_an_anonymous_dimension_is_not_a_coordinate_variable(tmp_path): + """A one-dimensional array is a coordinate variable by name coincidence. + + `CFCoordinateVariable.identify` requires the variable's own name to be + among its dimension names, which no anonymous dimension satisfies - so + the second route into the name-matching helpers is closed too. + """ + from iris.fileformats.cf import CFCoordinateVariable + + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array("t", shape=(3,), dtype="f4") + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + assert CFCoordinateVariable.identify(dataset.variables) == {} +``` + +These two depend on Task 9, which teaches `CFReader` to take an open +`CFDataset`. Write them here and run them at the end of Task 9. + +- [ ] **Step 9: Commit** + +Run §5.0, then: + +```bash +git add lib/iris/fileformats/cf/_variables.py lib/iris/tests/unit/fileformats/cf/test_spans.py +git commit -m "Match CF dimensions on (name, size), not on the name alone + +Zarr places no restriction on reusing a dimension name across arrays, so a +name alone can link two variables that mean different things by it. An +unnamed dimension is anonymous and matches nothing, including another +anonymous dimension. netCDF names are unique within a group, so this is a +widening that no netCDF file can observe." +``` + +--- + +### Task 9: `CFReader` accepts an already-open `CFDataset` + +**Files:** +- Modify: `lib/iris/fileformats/cf/_reader.py` +- Modify: `lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py` + +**Interfaces:** +- Consumes: `CFDataset` (the ABC), `ZarrDataset` (Task 4). +- Produces: `CFReader(file_source)` where `file_source` may be a `CFDataset`, + which is how Task 10's loader hands a Zarr store to the CF layer. + +`CFReader.__init__` has two branches today: a `str` opens a `NetCDFDataset`, +and anything else is passed to `NetCDFDataset.from_existing`. A `ZarrDataset` +falls into the second and is wrapped as a netCDF dataset (F12). The new branch +goes first and borrows rather than owns, because whoever opened the dataset is +responsible for closing it. + +- [ ] **Step 1: Write the failing test** + +Append to `lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py`: + +```python +def test_an_already_open_cf_dataset_is_used_as_is(tmp_path): + """A loader that has already opened a store hands it straight over. + + The alternative - a path and a format keyword - would put the choice of + backend inside CFReader, which is the branch the dataset interface exists + to remove. + """ + from iris.fileformats.netcdf._dataset import NetCDFDataset + + path = str(tmp_path / "source.nc") + with NetCDFDataset(path, mode="w") as writer: + writer.create_dimension("x", 2) + writer.create_variable("v", np.dtype("f4"), ("x",)) + + dataset = NetCDFDataset(path, mode="r") + with CFReader(dataset) as reader: + assert reader.filename == dataset.location + assert "v" in reader.cf_group + # Borrowed, not owned: leaving the context must not close it. + assert dataset.closed is False + dataset.close() +``` + +and, in the Zarr unit tests, a matching one — +`lib/iris/tests/unit/fileformats/zarr/test_CFReader.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The CF layer reads a Zarr store through the same reader netCDF uses.""" + +from iris.fileformats.cf import CFReader +from iris.fileformats.zarr._dataset import ZarrDataset + +from . import stores + + +def test_the_cf_reader_classifies_a_zarr_store(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + with CFReader(ZarrDataset(location)) as reader: + assert "air_temperature" in reader.cf_group.data_variables + assert "grid" in reader.cf_group.grid_mappings + assert reader.cf_group.global_attributes["title"] == "contract" + assert reader.filename == location +``` + +- [ ] **Step 2: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/test_CFReader.py -v +``` + +Expected: FAIL. `NetCDFDataset.from_existing` is handed a `ZarrDataset`, which +has no `THREAD_SAFE_FLAG`, so `EncodedDataset.from_existing` raises on it. + +- [ ] **Step 3: Add the branch** + +In `lib/iris/fileformats/cf/_reader.py`, add `from .dataset import CFDataset` +to the imports, and make `__init__`'s dispatch read: + +```python + # Ensure safe operation for destructor, should init fail. + self._own_file = False + if isinstance(file_source, CFDataset): + # Already open, in whatever format: use it, don't own it. The + # caller that opened it is responsible for closing it. + self._dataset = file_source + self._filename = file_source.location + elif isinstance(file_source, str): + # Create from filepath : open it + own it (=close when we die). + if not urlparse(file_source).scheme: + self._filename = Path(file_source).expanduser() + else: + self._filename = file_source + + self._dataset = NetCDFDataset( + self._filename, mode="r", warn_legacy_format=warn + ) + self._own_file = True + else: + # We have been passed an open dataset. + # We use it but don't own it (don't close it). + self._dataset = NetCDFDataset.from_existing( + file_source, warn_legacy_format=warn + ) + self._filename = self._dataset.location +``` + +- [ ] **Step 4: Update the module docstring** + +The paragraph beginning "This is the one module in :mod:`iris.fileformats.cf` +still coupled to :mod:`iris.fileformats.netcdf`" is now only half true, and a +stale docstring is worse than none. Replace it with: + +```python +``CFReader`` takes one of three things. A path opens a netCDF file, because +netCDF is the default a bare filename means. An open netCDF4 dataset, or an +emulator of one such as ncdata passes, is wrapped. And a +:class:`~iris.fileformats.cf.dataset.CFDataset` of any format is used as it +stands, which is how a Zarr store reaches this machinery: its loader opens +the store and hands it over, so nothing here has to know that Zarr exists. +Only the first case mentions netCDF, and only because something has to be +the default. +``` + +- [ ] **Step 5: Run the tests** + +```bash +pytest lib/iris/tests/unit/fileformats/cf/ lib/iris/tests/unit/fileformats/zarr/ -v +``` + +Expected: all pass, including Task 8's two deferred tests and Task 5's +no-type-test check. + +- [ ] **Step 6: Commit** + +Run §5.0, then: + +```bash +git add lib/iris/fileformats/cf/_reader.py lib/iris/tests/unit/fileformats +git commit -m "Let CFReader read any already-open CF dataset + +A loader that has opened a store hands it straight over, so the choice of +backend stays with the loader rather than becoming a branch inside the +reader. A borrowed dataset is not closed by the reader that borrowed it." +``` + +--- + +### Task 10: `zarr/loader.py` — `load_cubes` + +**Files:** +- Create: `lib/iris/fileformats/zarr/loader.py` +- Modify: `lib/iris/fileformats/zarr/__init__.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/loader/__init__.py` +- Create: `lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py` + +**Interfaces:** +- Consumes: `ZarrDataset` (Task 4); `CFReader` taking one (Task 9); + `iris.fileformats.cf.loader._actions_engine`, `_load_cube`, + `_load_aux_factory`, `_translate_constraints_to_var_callback` (**PR 3**); + `iris.io.run_callback`. +- Produces: `iris.fileformats.zarr.load_cubes(file_sources, callback=None, + constraints=None, group=None)`, re-exported from the package. + +The shape is `netcdf/loader.py::load_cubes` with the mesh handling removed: +UGRID from Zarr is out of scope here (§1), and `_meshes_from_cf` lives in +`netcdf/ugrid_load.py`, which this package must not import. + +Spec §4.6's rule is the one piece that is not a copy: "a store whose root +group holds no data variables raises a message that names the groups that do". + +- [ ] **Step 1: Write the failing tests** + +Create `lib/iris/tests/unit/fileformats/zarr/loader/__init__.py` with only the +copyright header, then `.../loader/test_load_cubes.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :func:`iris.fileformats.zarr.load_cubes`.""" + +import numpy as np +import pytest +import zarr + +from iris.cube import Cube +from iris.exceptions import TranslationError +from iris.fileformats.zarr import load_cubes + +from .. import stores + + +def test_one_cube_per_data_variable(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + cubes = list(load_cubes(location)) + assert len(cubes) == 1 + cube = cubes[0] + assert isinstance(cube, Cube) + assert cube.var_name == "air_temperature" + assert cube.units == "K" + np.testing.assert_array_equal(cube.data, stores.DATA) + + +def test_a_version_2_store_loads_the_same_cube(tmp_path): + v3 = list(load_cubes(stores.make_store(tmp_path / "v3.zarr")))[0] + v2 = list(load_cubes(stores.make_store(tmp_path / "v2.zarr", zarr_format=2)))[0] + assert v2 == v3 + + +def test_a_callback_can_reject_a_cube(tmp_path): + from iris.exceptions import IgnoreCubeException + + def reject(cube, field, filename): + raise IgnoreCubeException + + location = stores.make_store(tmp_path / "s.zarr") + assert list(load_cubes(location, callback=reject)) == [] + + +def test_the_group_keyword_selects_a_group(tmp_path): + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + sub = root.create_group("measurements") + band = stores.make_array( + sub, "b02", (3,), "u2", 3, dimensions=("p",), attributes={"units": "1"} + ) + band[:] = [1, 2, 3] + + cubes = list(load_cubes(str(tmp_path / "n.zarr"), group="measurements")) + assert [cube.var_name for cube in cubes] == ["b02"] + + +def test_several_stores_in_one_call(tmp_path): + first = stores.make_store(tmp_path / "a.zarr") + second = stores.make_store(tmp_path / "b.zarr") + assert len(list(load_cubes([first, second]))) == 2 +``` + +- [ ] **Step 2: Write the failing tests for Review Focus 2 — an empty root** + +Append: + +```python +def test_a_group_with_no_arrays_names_the_groups_that_have_them(tmp_path): + """A real store in this shape exists, and silence is the worst answer. + + The ESA EOPF Sentinel-2 products hold no arrays in the root group, only + subgroups - `conditions/geometry`, `measurements/reflectance/r10m` and + others. Iris loads one group per call, so loading the default root yields + nothing; returning an empty CubeList from a store that plainly holds data + leaves the user with no way to find out why. + """ + root = zarr.open_group(store=tmp_path / "e.zarr", mode="w", zarr_format=3) + root.create_group("conditions") + measurements = root.create_group("measurements") + stores.make_array(measurements, "b02", (2,), "u2", 3, dimensions=("p",)) + + with pytest.raises(TranslationError, match="measurements"): + list(load_cubes(str(tmp_path / "e.zarr"))) + + +def test_the_message_says_so_when_there_are_no_groups_either(tmp_path): + zarr.open_group(store=tmp_path / "empty.zarr", mode="w", zarr_format=3) + with pytest.raises(TranslationError, match="no data variables"): + list(load_cubes(str(tmp_path / "empty.zarr"))) + + +def test_nested_groups_are_not_walked(tmp_path): + """Iris loads one group per call, like xarray. + + A Zarr store can legitimately hold unrelated datasets, and merging them + into one CubeList would be wrong. + """ + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + top = stores.make_array( + root, "top", (2,), "f8", 3, dimensions=("p",), attributes={"units": "1"} + ) + top[:] = [1.0, 2.0] + inner = stores.make_array( + root.create_group("sub"), "inner", (2,), "f8", 3, dimensions=("p",) + ) + inner[:] = [3.0, 4.0] + + cubes = list(load_cubes(str(tmp_path / "n.zarr"))) + assert [cube.var_name for cube in cubes] == ["top"] +``` + +- [ ] **Step 3: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/loader/ -v +``` + +Expected: `ImportError: cannot import name 'load_cubes'`. + +- [ ] **Step 4: Write the loader** + +Create `lib/iris/fileformats/zarr/loader.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Load Iris cubes from Zarr stores. + +.. z_reference:: iris.fileformats.zarr.loader + :tags: topic_load_save + + API reference + +A thin loop: open the store as a +:class:`~iris.fileformats.zarr._dataset.ZarrDataset`, hand it to +:class:`~iris.fileformats.cf.CFReader`, and build one cube per CF data +variable with :mod:`iris.fileformats.cf.loader` - the same machinery the +netCDF loader runs. Everything Zarr-specific happens below this module, in +:mod:`iris.fileformats.zarr._dataset`. + +Iris loads **one** group per call. A Zarr store can legitimately hold +unrelated datasets, and merging them into one ``CubeList`` would be wrong, so +nested groups are not walked and ``group=`` selects which one to read. + +UGRID meshes are not built here. :func:`iris.mesh.load_meshes` reaches a Zarr +store through the same format dispatch, but the mesh machinery lives in +:mod:`iris.fileformats.netcdf.ugrid_load` and this package does not import it. + +See sections 4.6 and 4.7 of +``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. + +""" + +from collections.abc import Iterable +import warnings + +from iris.exceptions import TranslationError +from iris.fileformats.cf import CFReader +from iris.fileformats.cf.loader import ( + _actions_engine, + _load_aux_factory, + _load_cube, + _translate_constraints_to_var_callback, +) +import iris.warnings + +from ._dataset import ZarrDataset + + +def _check_has_data_variables(reader, dataset) -> None: + """Raise if this group holds nothing to make a cube from. + + Returning an empty ``CubeList`` from a store that plainly holds data + elsewhere leaves the user nothing to act on, and a store whose root group + holds only subgroups is a real shape - the ESA EOPF Sentinel products are + built that way. + """ + if reader.cf_group.data_variables or reader.cf_group.promoted: + return + + subgroups = dataset.subgroup_names + if subgroups: + message = ( + f"No data variables in Zarr group {dataset.location!r}. Load one " + "of its groups instead, with the 'group' keyword: " + f"{', '.join(sorted(subgroups))}" + ) + else: + message = f"No data variables in Zarr store {dataset.location!r}." + raise TranslationError(message) + + +def load_cubes(file_sources, callback=None, constraints=None, group=None): + """Load cubes from one or more Zarr stores. + + Parameters + ---------- + file_sources : str or iterable of str + The stores to load. A local directory path, or a URL that + zarr-python can open - ``s3://``, ``gs://``, ``az://`` and ``abfs://`` + need the matching store library installed, which Iris does not + depend on. + callback : callable, optional + Passed to :func:`iris.io.run_callback`. + constraints : optional + Used to skip data variables that cannot match, before they are built. + group : str, optional + A ``/``-separated path to a group within each store. The root group by + default. + + Yields + ------ + iris.cube.Cube + + Raises + ------ + iris.exceptions.TranslationError + If the selected group holds no data variables. The message names the + store's other groups, when it has any. + + Notes + ----- + This function maintains laziness; it does not realise data. + + """ + from iris.io import run_callback + + var_callback = _translate_constraints_to_var_callback(constraints) + engine = _actions_engine() + + if isinstance(file_sources, str) or not isinstance(file_sources, Iterable): + file_sources = [file_sources] + + for file_source in file_sources: + with ZarrDataset(file_source, group=group) as dataset: + with CFReader(dataset) as cf: + _check_has_data_variables(cf, dataset) + + data_variables = list(cf.cf_group.data_variables.values()) + list( + cf.cf_group.promoted.values() + ) + for cf_var in data_variables: + if var_callback and not var_callback(cf_var): + continue + + cube = _load_cube(engine, cf, cf_var, cf.filename) + + try: + _load_aux_factory(engine, cube) + except ValueError as exception: + warnings.warn( + str(exception), + category=iris.warnings.IrisLoadWarning, + ) + + cube = run_callback(callback, cube, cf_var, file_source) + if cube is None: + # The callback rejected it. + continue + + yield cube +``` + +- [ ] **Step 5: Re-export it** + +In `lib/iris/fileformats/zarr/__init__.py`, after the docstring: + +```python +from .loader import load_cubes + +__all__ = ["load_cubes"] +``` + +This is the line Task 1 deferred. Re-run Task 1's import-guard tests +immediately: `iris.fileformats.zarr` now imports `zarr` through `loader`, so +`test_importing_the_zarr_package_does_not_import_zarr` **must now fail**, and +that is correct — the test was about the package before it had a loader. +Replace it with the one that still matters: + +```python +def test_importing_the_zarr_package_imports_zarr(): + # Once the package exports a loader it needs the dependency. What must + # stay true is that nothing in `iris` or `iris.fileformats` imports this + # package to find out that it exists. + assert _imports_zarr("import iris.fileformats.zarr") +``` + +- [ ] **Step 6: Run the loader tests** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/ -v +``` + +Expected: all pass. + +- [ ] **Step 7: Check what a cube from a Zarr store actually carries** + +Append to `test_load_cubes.py`: + +```python +def test_the_cube_is_lazy(tmp_path, monkeypatch): + from iris.fileformats.cf import loader as cf_loader + + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + cube = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))[0] + assert cube.has_lazy_data() + + +def test_the_grid_mapping_variable_does_not_become_a_cube(tmp_path): + cubes = list(load_cubes(stores.make_store(tmp_path / "s.zarr"))) + assert "grid" not in [cube.var_name for cube in cubes] + + +def test_cf_reserved_attributes_do_not_reach_the_cube(tmp_path): + """`units` becomes a property, so it must not also be an attribute. + + Which attributes survive is decided by which ones the loading rules did + not read, so an attribute Iris consumes but fails to record as read + arrives on the cube as well. + """ + cube = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))[0] + assert "units" not in cube.attributes + assert "long_name" not in cube.attributes + assert cube.attributes["Conventions"] == "CF-1.7" + + +def test_a_structured_attribute_reaches_the_cube_unchanged(tmp_path): + """CF cannot express a nested attribute; Zarr allows one. Carry it. + + Two independent producers write them - NOAA GFS gives `valid_time` a + `statistics_approximate` object, and the ESA EOPF product root carries + `stac_discovery`. Interpreting them needs the attribute-model discussion + in SciTools/iris#7288; dropping them loses the publisher's metadata. + """ + group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "v", + (2,), + "f8", + 3, + dimensions=("p",), + attributes={"stac_discovery": {"id": "abc"}}, + ) + array[:] = [1.0, 2.0] + + cube = list(load_cubes(str(tmp_path / "n.zarr")))[0] + assert cube.attributes["stac_discovery"] == {"id": "abc"} +``` + +- [ ] **Step 8: Run and commit** + +```bash +pytest lib/iris/tests/unit/fileformats/zarr/ -v +``` + +Expected: all pass. Then run §5.0 and: + +```bash +git add lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr +git commit -m "Build cubes from a Zarr store + +Open the store, hand it to CFReader, and run the same cube-building loop +the netCDF loader runs. One group per call: a store can hold unrelated +datasets, so a group with no data variables names the groups that have +some rather than returning nothing." +``` + --- From 00bbb11a69af4c1447a89e4db92846a31831a4e0 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 12:13:17 +0000 Subject: [PATCH 07/56] Add PR 4 plan tasks 11-12: format detection and the load chain Task 11 recognises a store -- by a marker file locally, by scheme and suffix remotely, and never by the suffix alone. It also has to say in code that an NCZarr URL is still NCZarr's, because the store it names is real. Task 12 is the chain: load_stores, the partition after glob expansion, and the two saver-side changes section 4.7 asks for. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 726 ++++++++++++++++++ 1 file changed, 726 insertions(+) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index 2e2676f7b0..51a2babd8d 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -4746,5 +4746,731 @@ datasets, so a group with no data variables names the groups that have some rather than returning nothing." ``` +--- +### Task 11: format detection — the `ZarrStore` element and the `"Zarr"` specification + +**Files:** +- Modify: `lib/iris/io/format_picker.py` +- Modify: `lib/iris/fileformats/__init__.py` +- Create: `lib/iris/tests/unit/io/format_picker/test_ZarrStore.py` + +**Interfaces:** +- Consumes: `iris.io.decode_uri`, `iris.io._is_nczarr_fragment`; + `iris.fileformats.zarr.load_cubes` (Task 10). +- Produces: `ZarrStore(FileElement)`; `iris.io.REMOTE_STORE_SCHEMES`; + `iris.fileformats._is_zarr_store(target)` and `_load_zarr(*args, **kwargs)`; + and the registered `"Zarr"` `FormatSpecification`, which Task 12 dispatches + to. + +Spec §4.7 sets the matching rule and forbids the obvious simplification: + +> - **locally**, the target is a directory containing `zarr.json` (version 3) +> or `.zgroup` (version 2); +> - **remotely**, the URI scheme is one of `s3`, `gs`, `az`, `abfs` and the +> path ends in `.zarr`. +> +> **Do not add a third rule matching any path that ends in `.zarr`.** Joined by +> "or" it swallows the remote rule whole, leaving the scheme test unreachable. + +One case the spec's prose does not spell out and the code must handle: the +format agent is asked about NCZarr URLs too, because `load_http` calls +`get_spec(url, None)` with the fragment still attached. A local NCZarr target +— `file:///path/store.zarr#mode=nczarr,file` — is a real Zarr store on disk, so +the marker-file test says yes unless the fragment is checked first. Spec §4.7's +claim that "the native reader and the NCZarr one are never candidates for the +same target" is true of what the *user* asked for, and this is what makes the +code agree with it. + +- [ ] **Step 1: Write the failing tests** + +Create `lib/iris/tests/unit/io/format_picker/test_ZarrStore.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Recognising a Zarr store, locally by its marker file and remotely by its URL.""" + +import pytest + +from iris.fileformats import _is_zarr_store +from iris.io.format_picker import ZarrStore + + +def test_the_element_carries_the_whole_target(): + # Not the basename: recognising a local store means reading a file inside + # it, so the element needs the path, not the last component of it. + element = ZarrStore() + assert element.requires_fh is False + assert element.get_element("/data/store.zarr", None) == "/data/store.zarr" + + +def test_a_version_3_store_is_recognised(tmp_path): + store = tmp_path / "v3.zarr" + store.mkdir() + (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}') + assert _is_zarr_store(str(store)) is True + + +def test_a_version_2_store_is_recognised(tmp_path): + store = tmp_path / "v2.zarr" + store.mkdir() + (store / ".zgroup").write_text('{"zarr_format": 2}') + assert _is_zarr_store(str(store)) is True + + +def test_a_directory_named_zarr_with_no_marker_is_not_a_store(tmp_path): + """The suffix alone is not the local test, and must not become one. + + Reading the marker file is what distinguishes a real store from a + directory someone happened to name `.zarr`, and locally that costs one + stat. + """ + store = tmp_path / "pretend.zarr" + store.mkdir() + assert _is_zarr_store(str(store)) is False + + +def test_a_plain_file_is_not_a_store(tmp_path): + path = tmp_path / "data.nc" + path.write_bytes(b"CDF\x01") + assert _is_zarr_store(str(path)) is False + + +@pytest.mark.parametrize("scheme", ["s3", "gs", "az", "abfs"]) +def test_a_remote_store_is_recognised_by_scheme_and_suffix(scheme): + assert _is_zarr_store(f"{scheme}://bucket/path/data.zarr") is True + assert _is_zarr_store(f"{scheme}://bucket/path/data.zarr/") is True + + +def test_a_remote_url_without_the_suffix_is_not_a_store(): + # Remotely the suffix is all there is to go on: reading a marker file + # would cost a network round trip before Iris knows it wants the target. + assert _is_zarr_store("s3://bucket/path/data") is False + + +def test_an_http_url_is_not_a_store(): + # OPeNDAP, which the netCDF loader already claims. + assert _is_zarr_store("https://example.invalid/data.zarr") is False + + +def test_an_nczarr_url_is_not_a_native_store(tmp_path): + """A user who writes `#mode=nczarr` has asked for NCZarr by name. + + The store on disk is a real one, so the marker-file test says yes unless + the fragment is checked first - and `load_http` passes the fragment + through to the format agent. + """ + store = tmp_path / "s.zarr" + store.mkdir() + (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}') + assert _is_zarr_store(f"file://{store}#mode=nczarr,file") is False + + +def test_a_non_string_target_is_not_a_store(): + # `load_data_objects` passes open datasets, not paths. + assert _is_zarr_store(object()) is False +``` + +- [ ] **Step 2: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/io/format_picker/test_ZarrStore.py -v +``` + +Expected: `ImportError: cannot import name 'ZarrStore'`. + +- [ ] **Step 3: Add the file element** + +In `lib/iris/io/format_picker.py`, after `UriProtocol`: + +```python +class ZarrStore(FileElement): + """A :class:`FileElement` that returns a target that may be a Zarr store. + + Carries the whole target rather than its basename, because recognising a + local store means reading a marker file inside it, and recognising a + remote one means reading its scheme. + + """ + + def __init__(self): + FileElement.__init__(self, requires_fh=False) + + def get_element(self, basename, file_handle): + """Return `basename`, which here is the whole path or URL.""" + return basename +``` + +- [ ] **Step 4: Add the schemes and the matching rule** + +In `lib/iris/io/__init__.py`, beside `_is_nczarr_fragment`: + +```python +#: URI schemes Iris hands to a store library rather than resolving as a local +#: path. Iris depends on none of them: zarr-python raises its own ImportError +#: naming what to install. +REMOTE_STORE_SCHEMES = frozenset({"s3", "gs", "az", "abfs"}) +``` + +In `lib/iris/fileformats/__init__.py`, after `_uri_is_nczarr`: + +```python +#: What a version 3 and a version 2 Zarr store put at the root of a store. +_ZARR_MARKERS = ("zarr.json", ".zgroup") + + +def _is_zarr_store(target): + """Whether `target` names a native Zarr store. + + Locally the test is a marker file, which costs one stat and is what tells + a real store from a directory someone named ``.zarr``. Remotely it is the + scheme and the suffix, because reading a marker file would cost a network + round trip before Iris knows it wants the target at all; a remote + ``.zarr`` that turns out not to be a store fails when zarr-python opens + it, with zarr-python's message. + + These are two tests for two kinds of target, not two chances for one. A + third rule matching any path ending in ``.zarr`` would subsume the remote + one and leave the scheme test unreachable. + """ + if not isinstance(target, str): + # iris.io.load_data_objects passes open datasets, not paths. + return False + + scheme, part, fragment = decode_uri(target) + if _is_nczarr_fragment(fragment): + # The user asked for netCDF-c's Zarr driver by name. The store on + # disk is a real one, so the local test below would otherwise claim it. + return False + + if scheme in REMOTE_STORE_SCHEMES: + return part.rstrip("/").endswith(".zarr") + + directory = Path(part) + return any((directory / marker).is_file() for marker in _ZARR_MARKERS) + + +def _load_zarr(*args, **kwargs): + """Load from Zarr stores, reporting clearly when zarr-python is absent.""" + try: + from iris.fileformats.zarr import load_cubes + except ImportError as exception: + message = ( + "Unable to load a Zarr store - the 'zarr' package is not " + "installed. Install zarr >=3.0.8, for example with " + "'conda install -c conda-forge zarr'." + ) + raise ImportError(message) from exception + + return load_cubes(*args, **kwargs) +``` + +Add `from pathlib import Path` and extend the existing +`from iris.io import _is_nczarr_fragment` to +`from iris.io import REMOTE_STORE_SCHEMES, _is_nczarr_fragment, decode_uri`, +and add `ZarrStore` to the `iris.io.format_picker` import list. + +`_load_zarr` follows `_load_grib`, three functions above it: a module-level +shim is how Iris already defers an optional format dependency, and it keeps +`iris.fileformats` importable without zarr. + +- [ ] **Step 5: Register the specification** + +In `lib/iris/fileformats/__init__.py`, under the `# Zarr files.` heading, +**before** the `NcZarr` specification so that the two read in version order: + +```python +FORMAT_AGENT.add_spec( + FormatSpecification( + "Zarr", + ZarrStore(), + _is_zarr_store, + _load_zarr, + priority=5, + constraint_aware_handler=True, + ) +) +``` + +- [ ] **Step 6: Run the tests** + +```bash +pytest lib/iris/tests/unit/io/ -v +``` + +Expected: all pass. + +- [ ] **Step 7: Prove the registration does not disturb the other formats** + +Append to `test_ZarrStore.py`: + +```python +def test_the_agent_still_picks_netcdf_for_a_netcdf_file(tmp_path): + """A `requires_fh=False` spec is consulted for every target with no buffer. + + Adding one risks claiming a target another format already handles, and + the format agent returns the first match rather than the best. + """ + import iris.tests as tests + from iris.fileformats import FORMAT_AGENT + + path = tests.get_data_path(("NetCDF", "global", "xyt", "SMALL_hires.nc")) + with open(path, "rb") as handle: + spec = FORMAT_AGENT.get_spec("SMALL_hires.nc", handle) + assert spec.name.startswith("NetCDF") + + +def test_the_agent_picks_zarr_for_a_store(tmp_path): + from iris.fileformats import FORMAT_AGENT + + store = tmp_path / "s.zarr" + store.mkdir() + (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}') + assert FORMAT_AGENT.get_spec(str(store), None).name == "Zarr" + + +def test_the_agent_still_picks_nczarr_for_an_nczarr_url(tmp_path): + from iris.fileformats import FORMAT_AGENT + + store = tmp_path / "s.zarr" + store.mkdir() + (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}') + uri = f"file://{store}#mode=nczarr,file" + assert FORMAT_AGENT.get_spec(uri, None).name == "NcZarr" +``` + +The first needs `iris-test-data`; it is a skip if the repository is missing, +which is why §5.0 checks the path. + +- [ ] **Step 8: Run, then commit** + +```bash +pytest lib/iris/tests/unit/io/ lib/iris/tests/unit/fileformats/ -q +``` + +Expected: all pass. Then run §5.0 and: + +```bash +git add lib/iris/io/format_picker.py lib/iris/fileformats/__init__.py lib/iris/tests/unit/io +git commit -m "Teach the format agent to recognise a Zarr store + +Locally by the marker file a store puts at its root, remotely by the URI +scheme and the .zarr suffix. Not by the suffix alone: that would claim any +directory someone named .zarr, and joined with the remote rule it would +leave the scheme test unreachable. An NCZarr URL is still NCZarr's, which +needs saying in code because the store it names is a real one." +``` + +--- + +### Task 12: the load chain — a target that is not a file + +**Files:** +- Modify: `lib/iris/io/__init__.py` +- Modify: `lib/iris/loading.py` +- Modify: `lib/iris/fileformats/netcdf/ugrid_load.py` +- Create: `lib/iris/tests/unit/io/test_load_stores.py` +- Modify: `lib/iris/tests/unit/io/test_expand_filespecs.py` (or the module + holding those tests today) + +**Interfaces:** +- Consumes: `REMOTE_STORE_SCHEMES`, the `"Zarr"` specification (Task 11). +- Produces: `iris.io.load_stores(targets, callback, constraints=None)`; + `iris.io._is_remote_store_uri(spec)`; `iris.io._CUBELIST_SAVER_MODULES`; and + `iris.load("store.zarr")` and `iris.load("s3://bucket/store.zarr")` working. + +Spec §4.7's four changes, and the one consequence it names: + +> The one thing this costs is that the file-or-store decision has to happen +> *after* glob expansion and *before* format dispatch, because `iris.load` +> must still accept `"data/*"` matching a mix of `.nc` files and `.zarr` +> stores. + +- [ ] **Step 1: Write the failing tests for `load_stores`** + +Create `lib/iris/tests/unit/io/test_load_stores.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :func:`iris.io.load_stores`.""" + +import numpy as np +import pytest + +import iris +from iris.io import _is_remote_store_uri, load_stores + +zarr = pytest.importorskip("zarr", minversion="3.0.8") + +from iris.tests.unit.fileformats.zarr import stores # noqa: E402 + + +def test_a_local_store_loads(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + cubes = list(load_stores([location], None)) + assert [cube.var_name for cube in cubes] == ["air_temperature"] + + +def test_several_stores_in_one_call(tmp_path): + first = stores.make_store(tmp_path / "a.zarr") + second = stores.make_store(tmp_path / "b.zarr") + assert len(list(load_stores([first, second], None))) == 2 + + +def test_constraints_reach_the_handler(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + constraint = iris.Constraint(cube_func=lambda cube: False) + assert list(load_stores([location], None, constraint)) == [] + + +@pytest.mark.parametrize( + "uri, expected", + [ + ("s3://bucket/data.zarr", True), + ("gs://bucket/data.zarr", True), + ("az://container/data.zarr", True), + ("abfs://container/data.zarr", True), + ("https://example.invalid/data.nc", False), + ("/local/path/data.zarr", False), + ("relative/data.zarr", False), + ], +) +def test_which_uris_are_remote_stores(uri, expected): + assert _is_remote_store_uri(uri) is expected +``` + +- [ ] **Step 2: Write the failing tests for the chain** + +Append: + +```python +def test_iris_load_accepts_a_local_store(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + cubes = iris.load(location) + assert len(cubes) == 1 + np.testing.assert_array_equal(cubes[0].data, stores.DATA) + + +def test_iris_load_accepts_a_remote_url_and_does_not_absolutise_it(): + """A remote URL must reach zarr-python intact. + + `expand_filespecs` would otherwise turn `s3://bucket/data.zarr` into + `//s3:/bucket/data.zarr`. No network is touched: opening the store + fails, and what matters is that the message still names the URL. + """ + with pytest.raises(Exception) as caught: + iris.load("s3://bucket/data.zarr") + assert "s3://bucket/data.zarr" in str(caught.value) or isinstance( + caught.value, ImportError + ) + + +def test_a_glob_matching_both_files_and_stores(tmp_path): + """`iris.load("data/*")` must accept a mix. + + The file-or-store decision happens after glob expansion and before format + dispatch. Mis-partitioning raises IsADirectoryError from `load_files` for + the stores while the netCDF files load fine, which reads as an Iris bug + to a user who did not expect the glob to match a store. + """ + import iris.tests as tests + + source = iris.load_cube( + tests.get_data_path(("NetCDF", "global", "xyt", "SMALL_hires.nc")) + ) + iris.save(source, str(tmp_path / "a.nc")) + stores.make_store(tmp_path / "b.zarr") + + cubes = iris.load(str(tmp_path / "*")) + names = sorted(cube.var_name or cube.name() for cube in cubes) + assert "air_temperature" in names + assert len(cubes) == 2 +``` + +- [ ] **Step 3: Run them to verify they fail** + +```bash +pytest lib/iris/tests/unit/io/test_load_stores.py -v +``` + +Expected: `ImportError: cannot import name 'load_stores'`, and once that is +written, `IsADirectoryError` from `iris.load`. + +- [ ] **Step 4: Teach `expand_filespecs` about remote schemes** + +In `lib/iris/io/__init__.py`, beside `REMOTE_STORE_SCHEMES`: + +```python +def _is_remote_store_uri(spec): + """Whether `spec` names a store Iris hands to a store library untouched. + + Such a URI must not be resolved as a local path: `.absolute()` turns + ``s3://bucket/data.zarr`` into ``//s3:/bucket/data.zarr``. + """ + if not isinstance(spec, str): + return False + scheme, _part, _fragment = decode_uri(spec) + return scheme in REMOTE_STORE_SCHEMES +``` + +and in `expand_filespecs`, replace the two statements that build `filenames` +and `glob_expanded`: + +```python + # Remove any hostname component - currently unused. A remote store URI is + # passed through: it is not a path, and resolving it mangles it. + filenames = [ + fn + if _is_remote_store_uri(fn) + else str(pathlib.Path(fn.removeprefix("//")).expanduser().absolute()) + for fn in file_specs + ] + + if files_expected: + # Try to expand all filenames as globs. A remote URI stands for + # itself; Iris does not list a remote store's neighbours. + glob_expanded = OrderedDict( + [ + [fn, [fn] if _is_remote_store_uri(fn) else sorted(glob.glob(fn))] + for fn in filenames + ] + ) +``` + +- [ ] **Step 5: Write `load_stores`** + +In `lib/iris/io/__init__.py`, after `load_files`: + +```python +def load_stores(targets, callback, constraints=None): + """Create a generator of Cubes from targets that are stores, not files. + + A store is a directory or a URL: something with no single seekable file + to open, so :func:`load_files` cannot reach it and the format agent is + asked without a buffer. + + Parameters + ---------- + targets : iterable of str + Local directory paths or remote URLs, already expanded. + callback : callable + Passed to the format handler. + constraints : optional + Passed to a constraint-aware format handler. + + Yields + ------ + iris.cube.Cube + + Notes + ----- + Typically, this function should not be called directly; instead, the + intended interface for loading is :func:`iris.load`. + + """ + from iris.fileformats import FORMAT_AGENT + + handler_map = collections.defaultdict(list) + for target in targets: + handling_format_spec = FORMAT_AGENT.get_spec(target, None) + handler_map[handling_format_spec].append(target) + + for handling_format_spec in sorted(handler_map): + names = handler_map[handling_format_spec] + if handling_format_spec.constraint_aware_handler: + for cube in handling_format_spec.handler(names, callback, constraints): + yield cube + else: + for cube in handling_format_spec.handler(names, callback): + yield cube +``` + +- [ ] **Step 6: Partition in `_generate_cubes`** + +In `lib/iris/loading.py`, replace the `file` branch and the scheme list: + +```python + if category == "file": + part_names = [part for _cat, _scheme, part in groups] + # The file-or-store decision has to happen after glob expansion + # and before format dispatch, because a glob may match both. + # Neither routine below learns about the other's kind of target. + expanded = iris.io.expand_filespecs(part_names) + files = [name for name in expanded if not Path(name).is_dir()] + stores = [name for name in expanded if Path(name).is_dir()] + if files: + for cube in iris.io.load_files(files, callback, constraints): + yield cube + if stores: + for cube in iris.io.load_stores(stores, callback, constraints): + yield cube + elif category in ["http", "https", "nczarr"]: + urls = [f"{scheme}:{part}" for _cat, scheme, part in groups] + for cube in iris.io.load_http(urls, callback): + yield cube + elif category in iris.io.REMOTE_STORE_SCHEMES: + urls = [f"{scheme}:{part}" for _cat, scheme, part in groups] + for cube in iris.io.load_stores(urls, callback, constraints): + yield cube + elif category == "data": +``` + +Add `from pathlib import Path` to `loading.py`'s imports. + +- [ ] **Step 7: Run the chain tests** + +```bash +pytest lib/iris/tests/unit/io/ -v +``` + +Expected: all pass. + +- [ ] **Step 8: Update the saver-side pair** + +Still in `lib/iris/io/__init__.py`. Beside `_savers`: + +```python +#: Modules whose `save` takes a CubeList directly, rather than one cube at a +#: time with `append=True`. One line per format. +_CUBELIST_SAVER_MODULES = frozenset({"iris.fileformats.netcdf.saver"}) +``` + +In `find_saver`: + +```python + _check_init_savers() + # A store is a directory, so "out.zarr/" is a natural thing to type. + candidate = filespec.rstrip("/\\").lower() + matches = [ + ext for ext in _savers if candidate.endswith("." + ext) or candidate == ext + ] +``` + +In `save`, replace the substring test: + +```python + # Only allow cubelist saving for those fileformats that are capable. + if saver.__module__ not in _CUBELIST_SAVER_MODULES: +``` + +- [ ] **Step 9: Test the saver-side pair** + +Append to `test_load_stores.py`: + +```python +def test_find_saver_tolerates_a_trailing_separator(): + from iris.io import find_saver + + assert find_saver("out.nc") is find_saver("out.nc/") + + +def test_the_cubelist_saver_set_names_the_netcdf_saver(): + """The substring test it replaces matched by accident of module naming. + + `"iris.fileformats.netcdf" not in saver.__module__` is true of any module + whose path happens to contain that string, and says nothing a reader can + check. The set is one line per format. + """ + from iris.fileformats.netcdf import save as netcdf_save + from iris.io import _CUBELIST_SAVER_MODULES + + assert netcdf_save.__module__ in _CUBELIST_SAVER_MODULES + + +def test_saving_a_cubelist_to_netcdf_still_takes_one_call(tmp_path): + import iris.tests as tests + + cubes = iris.load(tests.get_data_path(("NetCDF", "global", "xyt", "SMALL_hires.nc"))) + iris.save(cubes, str(tmp_path / "out.nc")) + assert len(iris.load(str(tmp_path / "out.nc"))) == len(cubes) +``` + +- [ ] **Step 10: Update the mesh loader's copy of the dispatch** + +In `lib/iris/fileformats/netcdf/ugrid_load.py`, the `load_meshes` scheme +dispatch is a second copy of `_generate_cubes`'s and moves with it: + +```python + for category, groups in groupby(uri_tuples, key=lambda x: x[0]): + # Call each scheme handler with the appropriate URIs + if category == "file": + filenames = [part for _cat, _scheme, part in groups] + sources = expand_filespecs(filenames) + elif category in ["http", "https", "nczarr"]: + sources = [f"{scheme}:{part}" for _cat, scheme, part in groups] + elif category in REMOTE_STORE_SCHEMES: + sources = [f"{scheme}:{part}" for _cat, scheme, part in groups] + else: + message = f"Iris cannot handle the URI scheme: {category}" + raise ValueError(message) + + for source in sources: + if category == "file" and not Path(source).is_dir(): + with open(source, "rb") as fh: + handling_format_spec = FORMAT_AGENT.get_spec(Path(source).name, fh) + else: + handling_format_spec = FORMAT_AGENT.get_spec(source, None) + + if handling_format_spec.handler == nc_loader.load_cubes: + valid_sources.append(source) + else: + message = f"Ignoring non-NetCDF file: {source}" + logger.info(msg=message, extra=dict(cls=None)) +``` + +Add `REMOTE_STORE_SCHEMES` to its `iris.io` import. + +A Zarr store reaches `CFReader(source)` as a path, which opens a +`NetCDFDataset` and fails — so `load_meshes` does not yet build a mesh from +Zarr, and must not claim to. The handler check above rejects it: the `"Zarr"` +specification's handler is `_load_zarr`, not `nc_loader.load_cubes`, so a Zarr +store is logged and skipped exactly as a PP file is. That is the honest +behaviour for this pull request, and it is what the test below pins. + +- [ ] **Step 11: Test the mesh dispatch** + +Append to `test_load_stores.py`: + +```python +def test_load_meshes_skips_a_zarr_store_rather_than_raising(tmp_path): + """Reaching a store is this pull request's change; reading a mesh from + one is not. What must not happen is the ValueError the scheme dispatch + raised for anything that was not a file. + """ + from iris.mesh import load_meshes + + stores.make_store(tmp_path / "s.zarr") + assert load_meshes(str(tmp_path / "s.zarr")) == {} +``` + +- [ ] **Step 12: Run the neighbourhood** + +```bash +pytest -n auto lib/iris/tests/unit/io/ lib/iris/tests/unit/fileformats/ lib/iris/tests/unit/test_load*.py -q +pytest -n auto lib/iris/tests/integration/netcdf/ -q +``` + +Expected: all pass, with no change to the netCDF counts. + +- [ ] **Step 13: Commit** + +Run §5.0, then: + +```bash +git add lib/iris/io lib/iris/loading.py lib/iris/fileformats/netcdf/ugrid_load.py lib/iris/tests/unit +git commit -m "Let the load chain reach a target that is not a file + +A new iris.io.load_stores answers 'this target is not a file', once, for a +local directory and a remote URL alike, rather than teaching load_files +about directories in one place and the scheme dispatch in another. The +partition happens after glob expansion, so iris.load('data/*') still +accepts a mix of netCDF files and Zarr stores." +``` + --- From dfb58e9ae852be5a73c4670705baaf02746b1c2f Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 12:16:35 +0000 Subject: [PATCH 08/56] Finish the PR 4 plan: integration, docs, the pull request, verification Task 13 loads realistic stores both ways round -- version 2 through Iris's own NCZarr writer, version 3 through a fixture shaped like the NOAA GFS archive. Task 14 documents loading. Task 15 updates the spec, including closing its open question about fixed-length unicode. Sections 6 and 7 say what done is measured against. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 819 +++++++++++++++++- 1 file changed, 818 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index 51a2babd8d..b89989659d 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -5473,4 +5473,821 @@ accepts a mix of netCDF files and Zarr stores." ``` --- - +### Task 13: integration — a realistic cube, both Zarr versions + +**Files:** +- Create: `lib/iris/tests/integration/zarr/__init__.py` +- Create: `lib/iris/tests/integration/zarr/fixtures.py` +- Create: `lib/iris/tests/integration/zarr/test_load.py` + +**Interfaces:** +- Consumes: everything above; `iris.tests.stock.realistic_4d_w_everything`; + the existing NCZarr save path. +- Produces: coverage only. + +PR 4 has no saver, so there is no Iris round trip to make. Two substitutes do +the work, and between them they are better evidence than a round trip would +be, because neither is Iris marking its own homework: + +- **Version 2** comes from Iris's own NCZarr `mode=xarray` save path, which + already writes a version 2 store with `_ARRAY_DIMENSIONS` (spec §6). Save + `realistic_4d_w_everything()` through it, load it back natively, and compare + cubes. That is a direct check that the native reader and the netCDF-c writer + agree. +- **Version 3** comes from a fixture written with zarr-python, carrying the CF + features that matter: dimension coordinates, a two-dimensional auxiliary + coordinate, bounds, a scalar grid mapping, packing, a base64 `_FillValue` + and sharding. Those are the properties spec §7 says make the real NOAA GFS + fixture worth having; PR 7 replaces this with the real thing. + +- [ ] **Step 1: Create the package and the version 3 fixture builder** + +Create `lib/iris/tests/integration/zarr/__init__.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Integration tests for native Zarr loading. + +Skipped when zarr-python is absent, which is also how the unit tests do it. +""" + +import pytest + +pytest.importorskip("zarr", minversion="3.0.8") +``` + +Create `lib/iris/tests/integration/zarr/fixtures.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""A version 3 store carrying the CF features a real one does. + +Modelled on the NOAA GFS archive at dynamical.org, whose properties the Zarr +I/O design names as the ones worth exercising: version 3 `dimension_names`, a +sharded codec, a scalar grid-mapping variable, a two-dimensional auxiliary +coordinate, and the base64 `_FillValue` that released xarray writes for every +floating dtype. A cut-down copy of the real store replaces this. +""" + +import base64 +import struct + +import numpy as np +import zarr + +SHAPE = (2, 4, 6, 5) +DIMENSIONS = ("time", "level", "latitude", "longitude") + + +def float_fill_value(value): + """Return `value` in the encoding released xarray writes for a float.""" + return base64.b64encode(struct.pack("`_ stores an array as many independently compressed +chunks under a prefix, rather than as one file. Iris reads Zarr version 2 and +version 3 stores natively, through +`zarr-python `_. + +``zarr`` is an **optional dependency**. Install it to load Zarr stores:: + + conda install -c conda-forge zarr + +Iris without it behaves exactly as before, and raises an ``ImportError`` +naming the package if you ask it to load a store. + +Loading +------- + +A store is a directory, or a URL, so pass it where you would pass a file:: + + cubes = iris.load("/data/forecast.zarr") + +Iris recognises a local store by the metadata file at its root - ``zarr.json`` +for version 3, ``.zgroup`` for version 2 - not by the ``.zarr`` suffix, so a +directory that merely has the suffix is not mistaken for one. + +Remote stores +------------- + +A URL with one of the ``s3``, ``gs``, ``az`` or ``abfs`` schemes, ending in +``.zarr``, is passed to zarr-python unaltered:: + + cubes = iris.load("s3://noaa-gfs-bdp-pds/forecast.zarr") + +Iris adds no store library of its own and handles no credentials. Install the +one your scheme needs -- ``s3fs``, ``gcsfs``, ``adlfs`` -- and configure it as +that package documents. zarr-python raises an ``ImportError`` naming the +package if it is missing. + +Groups +------ + +A Zarr store may hold several groups, and they need not describe the same +thing. Iris loads **one** group per call, like xarray, rather than merging +them: ``iris.load`` reads the root group, and a store whose root holds no data +variables raises a message naming the groups that do. + +To read another group, call the Zarr loader directly:: + + from iris.cube import CubeList + import iris.fileformats.zarr + + cubes = CubeList( + iris.fileformats.zarr.load_cubes( + "/data/sentinel.zarr", group="measurements/reflectance/r10m" + ) + ) + +Dimension names +--------------- + +A netCDF dimension has a name, one length, and a meaning shared across the +whole file. A Zarr dimension name is weaker in three ways, and all three are +permitted by the version 3 specification: + +* an array need not name its dimensions at all; +* it may name some and not others; +* the same name in two arrays guarantees nothing -- not even the same length. + +So Iris matches dimensions on the pair *(name, length)*. Two arrays share a +dimension when they name it identically **and** it is the same length in both. +A dimension with no name is **anonymous**, and never matches anything; the +cube simply has a dimension with no coordinate on it, which is an ordinary +Iris cube. A dimension name used at two different lengths is reported with a +warning and links nothing. + +Version 2 stores have no dimension-name field, and carry names in an +``_ARRAY_DIMENSIONS`` attribute instead -- the convention xarray established. +Iris reads it, and does not pass it on to the cube. + +Missing data +------------ + +Zarr applies no masking of its own, so Iris does it, from the CF attributes: +``_FillValue`` and ``missing_value`` for sentinels, +``valid_min``/``valid_max``/``valid_range`` for a range, and +``scale_factor``/``add_offset`` for packed data. + +A Zarr array also has a ``fill_value`` in its metadata. This is **not** a +missing-value declaration: it says what a chunk that was never written reads +back as, and zarr-python defaults it to zero. Iris therefore ignores it as a +mask source on version 3, where it is a required field, and uses it on version +2 only when the array declares no CF attribute -- which is what version 2 +writers meant by it, and what xarray does. + +Chunking +-------- + +Iris chooses Dask chunks from the store's own chunking, as it does for netCDF, +and :data:`~iris.fileformats.cf.loader.CHUNK_CONTROL` works the same way for +both formats. + +One rule is specific to Zarr. A Zarr chunk is the atomic unit of storage: +reading any part of one fetches and decompresses the whole object. So Iris +rounds the computed chunking **up** to a whole multiple of the store's +chunking, never below it. When a single store chunk is already larger than +Dask's target chunk size, the store's layout is the binding constraint on +memory per task, and Iris warns to say so. + +A **sharded** array is chunked on its inner chunks, not its shards. Zarr's +sharding codec is indexed, so a reader fetches only the inner chunks it needs. + +Attributes +---------- + +A Zarr attribute value may be any JSON literal, including a nested object or +an array of arrays, which CF cannot express at any rank. Iris carries such a +value onto the cube unchanged and does not interpret it. See +:issue:`7288` for the wider question of how the CF attribute model should map +onto JSON. + +NCZarr +------ + +Iris has long been able to read and write Zarr through netCDF-c's NCZarr +driver, by way of a URL fragment:: + + iris.load("file:///data/forecast.zarr#mode=nczarr,file") + +That is unchanged, and is still how you reach NCZarr. NCZarr implements Zarr +version 2 and writes scalar variables as length-one arrays; the native reader +described above handles both versions and is what a plain path or URL now +uses. +``` + +Check `:issue:` is the role `common_links.inc` provides; if it is not, write +the full URL. + +- [ ] **Step 2: List it** + +In `docs/src/user_manual/section_indexes/general.rst`, add to the toctree, +keeping the alphabetical order the list is in: + +```rst + ../explanation/zarr_io +``` + +- [ ] **Step 3: Rewrite the S3 page's premise** + +`docs/src/user_manual/tutorial/s3_io.rst` says S3 is unsupported and +recommends a FUSE mount. That is still true for netCDF and no longer true for +Zarr. Change the sphinx-needs summary and add a section near the top, before +"TL;DR": + +```rst +Zarr stores are accessed directly +--------------------------------- + +Everything below concerns **files** -- netCDF, PP, GRIB -- which Iris reaches +through a file-system path. A :ref:`Zarr store ` is different: Iris +passes an ``s3://`` URL straight to zarr-python, which reads the objects it +needs over the network:: + + cubes = iris.load("s3://my-bucket/data.zarr") + +That needs ``s3fs`` installed and configured, and no mount. If your data is in +Zarr, prefer this to the FUSE approach described below. +``` + +and amend the paragraph beginning "While it would be possible for Iris to +support S3 access directly" to say that this is what the Zarr path now does, +and that the limitation it describes applies to the file formats. + +- [ ] **Step 4: Check the metadata and build** + +```bash +pre-commit run check-docs-page-metadata +cd docs && make html-noplot +``` + +Expected: the hook passes, and the build is warning-free. Warnings are errors +in the standard build, so a bad cross-reference fails CI. + +- [ ] **Step 5: Run the doctests** + +The page has no `>>>` examples — the blocks above are `::` literals, not +doctests, deliberately, because none of them can run without a store or a +network. If you add a `>>>` example, run: + +```bash +cd docs/src && make doctest +``` + +- [ ] **Step 6: Commit** + +Run §5.0, then: + +```bash +git add docs/src +git commit -m "Document Zarr loading + +The page a user comparing formats will look for, beside the netCDF one: +what a store is, how a remote one is reached, what a Zarr dimension name +does and does not guarantee, why the array fill value is not a +missing-value flag, and why chunking rounds up. The S3 page no longer +begins by saying this is impossible." +``` + +--- + +### Task 15: the spec, the full-suite diff, and the pull request + +**Files:** +- Modify: `docs/superpowers/specs/2026-09-21-zarr-io-design.md` +- Modify: `docs/superpowers/plans/2026-10-13-zarr-io-pr4.md` (this file) + +**Interfaces:** +- Consumes: everything. +- Produces: an open pull request, and a spec that tells the next reader the + truth. + +- [ ] **Step 1: Run the full suite, twice** + +```bash +echo "$OVERRIDE_TEST_DATA_REPOSITORY" +git stash list # expect empty + +git switch --detach brownfield +pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/baseline.txt + +git switch - +pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/branch.txt +diff /tmp/baseline.txt /tmp/branch.txt +``` + +Expected: the branch adds passes and adds no failures. Compare **like with +like** — a tier run says nothing about a full-suite one — and read the **skip** +count as well as the failures. A skip that is new on the branch is a test that +never ran. + +- [ ] **Step 2: Update the spec's progress record** + +In `docs/superpowers/specs/2026-09-21-zarr-io-design.md`: + +- §12.1: set PR 4's state to `In review` and record its number, as soon as the + pull request is opened. +- The header table's **Progress** and **Next action** rows. +- §12.3: move **Q2** to §12.4 with today's date and the answer from F3 — + fixed-length unicode arrays read as themselves on both versions, verified + against zarr-python 3.4.1, so no special handling is needed. +- §12.4: append two entries, dated, for the decisions this pull request took + that change what the spec says: + + ```markdown + **2026-10-13 — during PR 4** + + - `CFDataset.format` lands in PR 4, not PR 5. `ZarrDataset` cannot answer + `ZARR_V2` versus `ZARR_V3` without it, and PR 4 merges first. + `supports_dtype` is a saver question and stays in PR 5. §5's PR 4 and PR 5 + entries are amended to match. + - The shared `CFDataset` contract is split into a read half and a write + half. PR 4 implements the Zarr read path only, so the write half is + unrunnable against it; one class with half of it skipped would hide an + incomplete implementation behind a skip, which is the opposite of what §6 + asks a shared body to do. PR 6 adds the write half for Zarr. + ``` + +- §5, PR 4 and PR 5 entries: move `CFDataset.format` from PR 5's sentence to + PR 4's list of generic-code changes. + +- [ ] **Step 3: Record what the implementation taught you** + +Add a section to this plan, below §2, titled **"What changed during +implementation"**, listing every step whose code differed from what is written +here and why. The plan ships with the pull request, so a reader comparing the +two must not have to guess which one is true. + +- [ ] **Step 4: Open the pull request** + +```bash +git push -u origin zarr-io-pr-4 +gh pr create --base brownfield --title "Zarr I/O PR 4: Zarr loading" --body-file /tmp/pr4-body.md +gh pr edit --add-label "Agentic" --add-label "Type: Feature Branch" +``` + +The body opens `(posted by Claude Opus)` before anything else, per +`AGENTS.md`, which **replaces** any trailing attribution — except the +`🤖 Generated with Claude Code` line the harness asks for, which stays at the +end. Never hard-wrap it. It must cover: + +- **What this does.** `iris.load("store.zarr")` and + `iris.load("s3://bucket/store.zarr")` return cubes, for version 2 and + version 3 stores. +- **Part of #6979.** Not `Closes` — a closing keyword only fires on a merge + into the default branch, and this targets `brownfield`. +- **The changelog fragment is deferred to the merge-back**, deliberately, per + spec §5. Say so, so the omission does not read as an oversight. +- **Behaviour changes**, all three, each with what used to happen: a local + `.zarr` directory raised `IsADirectoryError`; an `s3://` URL raised + `ValueError("Iris cannot handle the URI scheme")`; and `CFVariable.spans` + now matches `(name, size)` pairs, which no netCDF file can observe and which + the test suite asserts on both backends. +- **Public API added**: `iris.fileformats.zarr`, `iris.io.load_stores`, + `CFDataset.format`, `CFDataset.encoding`, `CFDatasetVariable.encoding`. +- **The spec amendments** from Step 2, each in one line, so a reviewer who + knows the spec is not surprised. +- **What is not here**: saving, real-world fixtures, benchmarks, xarray as a + test dependency, UGRID meshes from Zarr. +- **How to review it**: the task order in §3 of this plan is the reading + order, and Tasks 8 and 12 are the two a netCDF-only reviewer should check. +- **The dependency on #7316**, and whether this branch was rebased after it + merged. + +- [ ] **Step 5: Commit the spec and plan updates** + +Run §5.0, then: + +```bash +git add docs/superpowers +git commit -m "Record PR 4 in the spec, and what implementation changed + +Closes open question Q2: fixed-length unicode arrays read as themselves on +both Zarr versions, so #6961's note describes the specification rather than +zarr-python. Logs the two decisions that move work between pull requests: +format lands here rather than with the saver, and the shared dataset +contract splits into read and write halves." +``` + +--- + +## 6. Verification + +What "done" is measured against, and how to measure it. + +### 6.1 The green check, per task + +Every task ends with §5.0: the test-data path confirmed, `pre-commit` clean on +the paths touched, and the task's own test tier passing. No task's commit is +made with `--no-verify`. + +### 6.2 The full suite, against the base + +Run once at the start of implementation and once at the end, on the same +machine, with `OVERRIDE_TEST_DATA_REPOSITORY` set and verified: + +```bash +pytest -n auto lib/iris/tests -q +``` + +Record `passed`, `skipped`, `failed` and `errors` for both. The branch must +add passes, add no failures, and **add no skips**. A new skip is a test that +never ran, and the usual causes — a missing optional dependency, a missing +`iris-test-data` — are exactly the ones this pull request introduces a new +instance of. + +`zarr` must be installed in the environment the final run uses, or every test +this pull request adds is a skip and the comparison is meaningless. + +### 6.3 Without zarr + +The other half of spec §6's "Iris without zarr installed must show no new +failures". In an environment with zarr removed: + +```bash +python -c "import iris; print(iris.__version__)" +pytest -n auto lib/iris/tests -q +``` + +Expected: Iris imports; the Zarr unit and integration packages skip as whole +directories; nothing fails. If a clean environment is not available, state +that in the pull request rather than claiming the check was made. + +### 6.4 The documentation build + +```bash +pre-commit run check-docs-page-metadata --all-files +cd docs && make html +``` + +Expected: no warnings. The standard build uses `-W --keep-going`, so any new +warning fails CI. + +### 6.5 The claims this pull request makes, and what checks each + +| Claim | Checked by | +|---|---| +| A netCDF file loads exactly as it did | §6.2's diff, plus `test_spans.py` running the same suite on both backends (Task 8) | +| `_ARRAY_DIMENSIONS` does not reach a cube | Task 3 Step 9, and Task 13 Step 4 end to end | +| A version 3 array's zeros are not masked | Task 6 Step 5 | +| A version 2 array's fill value *is* a mask source | Task 6 Step 5 | +| Both `_FillValue` encodings read | Task 6 Steps 1–2, Task 13 Step 2 | +| A sharded array is read by the inner chunk | Task 7 Step 9, on bytes fetched through an instrumented store | +| The chunking tiles the store grid | Task 7 Steps 1–4, Task 13 Step 3 | +| The dask cache key goes stale when the array does | Task 7 Step 8 | +| An anonymous dimension matches nothing | Task 8 Steps 3–4 | +| A group with no data variables says so | Task 10 Step 2 | +| A `.zarr`-suffixed directory that is not a store is not claimed | Task 11 Step 1 | +| NCZarr still reaches NCZarr | Task 11 Steps 1 and 7 | +| A glob matching files and stores works | Task 12 Step 2 | +| Iris imports without zarr | Task 1 Steps 5–6, §6.3 | +| The native reader agrees with a second writer | Task 13 Step 4, against NCZarr | + +### 6.6 What is deliberately not checked here + +- **Against xarray.** Spec §6 adds xarray as a test dependency in PR 6, with + the saver it validates. Nothing in PR 4 produces a store for another reader + to open. +- **Against a real published store.** PR 7. +- **Over a network.** Spec §6: "No test touches the network. The URL path is + exercised by checking that the right store is constructed, not by fetching." +- **Performance.** The ASV benchmarks are PR 7. + +--- + +## 7. Definition of done + +- [ ] Every task's checkboxes are ticked, and every commit was made with + `pre-commit` clean and its tier of tests passing. +- [ ] `pytest -n auto lib/iris/tests` adds passes, adds no failures and adds + no skips against `brownfield`, with `zarr` installed and + `OVERRIDE_TEST_DATA_REPOSITORY` set (§6.2). +- [ ] Iris imports and its suite runs with `zarr` absent (§6.3), or the pull + request says the check could not be made. +- [ ] `cd docs && make html` is warning-free (§6.4). +- [ ] `iris.load("store.zarr")` works for a version 2 and a version 3 store, + and `iris.fileformats.zarr.load_cubes(store, group=...)` reads a + subgroup. +- [ ] `requirements/py3{12,13,14}.yml` carry `zarr >=3.0.8` and the lock files + are regenerated — or the pull request asks a maintainer to regenerate + them. +- [ ] The spec's §12.1, §12.3 and §12.4 are updated, and §5's PR 4 and PR 5 + entries reflect where `format` landed (Task 15 Step 2). +- [ ] This plan carries a "What changed during implementation" section, or + states that nothing did. +- [ ] The pull request is open against `brownfield`, labelled `Agentic` and + `Type: Feature Branch`, says `Part of #6979`, opens + `(posted by Claude Opus)`, and states that its changelog fragment is + deferred to the merge-back. + From d365da67dccdb017d2f9febd356826b516e2799a Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 12:19:47 +0000 Subject: [PATCH 09/56] Self-review the PR 4 plan: fix the pointers, close the gaps Review Focus now points at the tasks it means. The interface section's _align_chunks and decode_parameters signatures match what the tasks write. Adds F18, which is the gap the review found: this pull request creates the first way to get a nested attribute onto a cube, and the netCDF saver cannot write one -- so Task 10 characterises it and Task 14 documents it. Also adds the version-aware masking tests at the loaded-cube level, the async-concurrency note the spec asks for, and a step to read the generated API pages. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 209 ++++++++++++++++-- 1 file changed, 196 insertions(+), 13 deletions(-) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index b89989659d..348ac9ed27 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -130,7 +130,7 @@ owns the code, written out in that task's own step style. failure to defend against is not the message but its absence: an empty `CubeList` returned silently from `iris.load` of a store that plainly contains data is the worst available answer, and the one that falls out of - writing the loop without thinking about it. → **Task 9, Steps 7–8**. + writing the loop without thinking about it. → **Task 10, Step 2**. 3. **`iris.load("data/*")` matching a mix of `.nc` files and `.zarr` directories.** Spec §4.7 states the requirement and names the consequence — @@ -139,7 +139,7 @@ owns the code, written out in that task's own step style. what `expand_filespecs` returns, so a bug that mis-partitions raises `IsADirectoryError` from `load_files` for the stores while the netCDF files load fine, and a user who did not expect the glob to match a store reads it - as an Iris bug. → **Task 11, Steps 11–12**. + as an Iris bug. → **Task 12, Step 2**. 4. **A `scale_factor` or `add_offset` that changes the result dtype.** `cf/loader.py::_get_actual_dtype` computes the post-unpacking dtype and the @@ -150,7 +150,7 @@ owns the code, written out in that task's own step style. graph is built, and the mismatch surfaces later as a wrong dtype on the cube, or as a `ValueError` from a downstream operation. A `uint16` band with `scale_factor = 0.0001` — the EOPF reflectance case in spec §7 — must give a - `float64` cube whose lazy and realised dtypes match. → **Task 6, Steps 11–12**. + `float64` cube whose lazy and realised dtypes match. → **Task 7, Step 11**. 5. **A `zarr` that is installed but too old, or not installed at all.** Spec §3 requires "a clear `ImportError`" and §6 requires that "Iris without zarr @@ -159,7 +159,7 @@ owns the code, written out in that task's own step style. second is the one that breaks the first rule by accident, because `iris.fileformats.__init__` imports every format module it registers. If the Zarr import is not deferred, installing Iris without zarr stops `import iris` - working at all. → **Task 1, Steps 5–8**. + working at all. → **Task 1, Steps 5–6**, and §6.3. --- @@ -357,6 +357,18 @@ unnoticed. but the comment naming netCDF4 does, and `valid_min`/`valid_max`/`valid_range` are deliberately **not** in that set, so they reach the cube on both paths. +**F18 — this pull request creates the first way to get a nested attribute onto +a cube, and the netCDF saver cannot write one.** Spec §4.4 says Iris carries a +JSON object or array attribute onto the cube and "does not write them back +through the netCDF saver"; spec §4.5 says checking attribute content before +saving is [#7329](https://github.com/SciTools/iris/issues/7329), out of scope. +Until then, `iris.load("store.zarr")` followed by `iris.save(cube, "out.nc")` +fails inside netCDF4, with a message naming neither the attribute nor the +cube. Nothing in this pull request can fix that without changing the netCDF +saver, which PR 5 relocates unchanged — so **Task 10 characterises it with a +test and Task 14 documents it**, which is the difference between a known +limitation and a surprise. + --- ## 3. File Structure @@ -382,7 +394,8 @@ are deliberately **not** in that set, so they reach the cube on both paths. | `lib/iris/tests/unit/io/format_picker/test_ZarrStore.py` | The local marker-file test and the remote scheme test, and that neither swallows the other | ~150 | | `lib/iris/tests/integration/zarr/__init__.py` | Empty package marker | 5 | | `lib/iris/tests/integration/zarr/test_load.py` | `iris.load` end to end, version 2 and version 3, including an NCZarr-written version 2 store | ~260 | -| `docs/src/user_manual/howto/zarr_io.rst` | How to load a Zarr store, local and remote, with `group=` | ~160 | +| `lib/iris/tests/integration/zarr/fixtures.py` | A version 3 store shaped like the NOAA GFS archive | ~110 | +| `docs/src/user_manual/explanation/zarr_io.rst` | Loading a Zarr store: local and remote, groups, dimension names, missing data, chunking, attributes | ~170 | ### Modified @@ -402,7 +415,7 @@ are deliberately **not** in that set, so they reach the cube on both paths. | `requirements/py312.yml`, `py313.yml`, `py314.yml` | `zarr >=3.0.8` in the optional-dependency section | | `requirements/locks/*.lock` | Regenerated, once, by `make lockfiles` | | `docs/src/user_manual/tutorial/s3_io.rst` | Replace "S3 is unsupported, use a FUSE mount" with the Zarr route (spec §4.7) | -| `docs/src/user_manual/howto/index.rst` (or the sibling toctree) | List the new page | +| `docs/src/user_manual/section_indexes/general.rst` | List the new page in the toctree, beside `netcdf_io` | | `docs/superpowers/specs/2026-09-21-zarr-io-design.md` | §12.1 state, §12.3 Q2 closed, §12.4 entries for F10 and F11, §5's PR 4/PR 5 split amended | ### Sequencing, and why it is this way @@ -476,10 +489,18 @@ _FORMAT_BY_ZARR_VERSION = {2: "ZARR_V2", 3: "ZARR_V3"} _ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS" +def _dimension_names(array: "zarr.Array") -> tuple[str | None, ...]: ... +def _assemble_dimensions(variables: Mapping) -> dict[str, int]: ... +def _fixed_flags(dims_fixed, ndim: int) -> tuple[bool, ...]: ... +def _store_chunk_exceeds_dask_target(store_chunks, dtype) -> bool: ... + + def _align_chunks( chunks: tuple[int, ...], store_chunks: tuple[int, ...] | None, shape: tuple[int, ...], + dtype: np.dtype, + dims_fixed, name: str, ) -> tuple[int, ...]: """Round `chunks` up to whole multiples of the store's own chunk shape.""" @@ -515,6 +536,10 @@ class ZarrDatasetVariable(CFDatasetVariable): def cache_key(self) -> str: """`(location, array path, the metadata document)`, as one string.""" + @property + def zarr_format(self) -> int: + """2 or 3. Zarr-only; the generic layer never asks which version.""" + @property def array(self) -> "zarr.Array": """The backing zarr array. Zarr-only; nothing generic may read it.""" @@ -580,13 +605,28 @@ def mask_by_valid_range(block, valid_min, valid_max) -> np.ndarray: ... def unpack(block, scale_factor, add_offset, dtype) -> np.ndarray: ... -def decode_parameters(variable: "ZarrDatasetVariable") -> dict: - """Read the CF decode parameters off a variable's attributes and storage. +def decode_parameters( + attributes: Mapping, + *, + dtype: np.dtype, + storage_fill_value: Any, + zarr_format: int, + variable_name: str, +) -> dict: + """Read the CF decode parameters for one variable. - This is where spec section 4.4's version-aware masking precedence lives: + Plain arguments, not a variable, so that spec section 4.4's version-aware + masking precedence is testable without building a store for each case: on version 3 only the CF attributes are a mask source; on version 2 `Array.fill_value` is one as well, when no CF attribute is present. + + Always returns the same keys - the keyword arguments `mask_and_scale` + takes. `decoding_required` says whether they do anything. """ + + +def decoding_required(parameters: dict) -> bool: ... +def _unpacked_dtype(dtype, scale_factor, add_offset) -> np.dtype: ... ``` ```python @@ -608,10 +648,28 @@ _CUBELIST_SAVER_MODULES = frozenset({"iris.fileformats.netcdf.saver"}) REMOTE_STORE_SCHEMES = frozenset({"s3", "gs", "az", "abfs"}) +def _is_remote_store_uri(spec) -> bool: ... + + def load_stores(targets, callback, constraints=None): """Create a generator of Cubes from targets that are stores, not files.""" ``` +```python +# lib/iris/fileformats/__init__.py (additions) + +#: What a version 3 and a version 2 Zarr store put at the root of a store. +_ZARR_MARKERS = ("zarr.json", ".zgroup") + + +def _is_zarr_store(target) -> bool: + """Whether `target` names a native Zarr store.""" + + +def _load_zarr(*args, **kwargs): + """Load from Zarr stores, reporting clearly when zarr-python is absent.""" +``` + ```python # lib/iris/io/format_picker.py (addition) @@ -4726,6 +4784,90 @@ def test_a_structured_attribute_reaches_the_cube_unchanged(tmp_path): cube = list(load_cubes(str(tmp_path / "n.zarr")))[0] assert cube.attributes["stac_discovery"] == {"id": "abc"} + + +def test_a_structured_attribute_cannot_then_be_saved_to_netcdf(tmp_path): + """Known limitation, characterised so that it is not a surprise. + + Zarr allows any JSON literal as an attribute value and netCDF does not, + so a store carrying one produces a cube Iris cannot save as netCDF. + Checking attribute content before saving is SciTools/iris#7329, and + fixing it means changing the netCDF saver. Strip the attribute, or save + to Zarr once that is supported. + """ + import iris + + group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "v", + (2,), + "f8", + 3, + dimensions=("p",), + attributes={"stac_discovery": {"id": "abc"}}, + ) + array[:] = [1.0, 2.0] + cube = list(load_cubes(str(tmp_path / "n.zarr")))[0] + + with pytest.raises(Exception): + iris.save(cube, str(tmp_path / "out.nc")) + + # And the documented workaround works. + del cube.attributes["stac_discovery"] + iris.save(cube, str(tmp_path / "out.nc")) + + +def test_version_3_zeros_are_not_masked(tmp_path): + """An int16 [0, 1, 2] with no CF attributes loads with nothing masked. + + zarr-python defaults `fill_value` to zero, so reading the storage field + as a sentinel masks valid data in the ordinary case. + """ + group = zarr.open_group(store=tmp_path / "z3.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, "v", (3,), "i2", 3, dimensions=("p",), attributes={"units": "1"} + ) + array[:] = [0, 1, 2] + + cube = list(load_cubes(str(tmp_path / "z3.zarr")))[0] + assert not np.ma.is_masked(cube.data) + np.testing.assert_array_equal(cube.data, [0, 1, 2]) + + +def test_version_2_zeros_are_masked(tmp_path): + """The same array as version 2 does use `fill_value` as the mask source. + + Version 2 stores predate the CF `_FillValue` convention, and this is what + their writers meant - which is what released xarray's + `use_zarr_fill_value_as_mask` does. + """ + group = zarr.open_group(store=tmp_path / "z2.zarr", mode="w", zarr_format=2) + array = stores.make_array( + group, "v", (3,), "i2", 2, dimensions=("p",), attributes={"units": "1"} + ) + array[:] = [0, 1, 2] + + cube = list(load_cubes(str(tmp_path / "z2.zarr")))[0] + assert cube.data.mask.tolist() == [True, False, False] + + +def test_a_cf_fill_value_beats_the_storage_one(tmp_path): + group = zarr.open_group(store=tmp_path / "z.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "v", + (3,), + "i2", + 3, + dimensions=("p",), + fill_value=0, + attributes={"units": "1", "_FillValue": -999}, + ) + array[:] = [0, -999, 2] + + cube = list(load_cubes(str(tmp_path / "z.zarr")))[0] + assert cube.data.mask.tolist() == [False, True, False] ``` - [ ] **Step 8: Run and commit** @@ -5964,6 +6106,12 @@ memory per task, and Iris warns to say so. A **sharded** array is chunked on its inner chunks, not its shards. Zarr's sharding codec is indexed, so a reader fetches only the inner chunks it needs. +zarr-python fetches chunks concurrently, under its own +``zarr.config`` setting ``async.concurrency``. Iris does not change it. Dask's +worker count and that concurrency multiply, so a large Dask pool over a remote +store can open far more connections than either number suggests; if you are +rate-limited, lower one of them. + Attributes ---------- @@ -5973,6 +6121,15 @@ value onto the cube unchanged and does not interpret it. See :issue:`7288` for the wider question of how the CF attribute model should map onto JSON. +.. note:: + + A cube carrying such an attribute **cannot be saved as netCDF**, because + netCDF attribute values are scalars, strings and one-dimensional arrays. + The failure comes from the netCDF library and does not name the attribute; + see :issue:`7329`. Delete the attribute before saving:: + + del cube.attributes["stac_discovery"] + NCZarr ------ @@ -6035,7 +6192,24 @@ cd docs && make html-noplot Expected: the hook passes, and the build is warning-free. Warnings are errors in the standard build, so a bad cross-reference fails CI. -- [ ] **Step 5: Run the doctests** +- [ ] **Step 5: Read the generated API pages** + +`sphinxcontrib-apidoc` writes one page per non-private module from its +docstring, so `iris.fileformats.zarr` and `iris.fileformats.zarr.loader` get +pages from what Tasks 1 and 10 wrote. Open them and read them: + +```bash +cd docs && make html-noplot +python -m webbrowser src/_build/html/generated/api/iris/fileformats/zarr.html +``` + +`docs/AGENTS.md` gotcha 5 is the reason this is a step rather than an +assumption: a re-exporting `__init__.py` has three failure modes, two of them +silent. Check that `load_cubes` appears on the package page — it is in +`__all__`, which is what autodoc needs — and that it is described once, not +twice. + +- [ ] **Step 6: Run the doctests** The page has no `>>>` examples — the blocks above are `::` literals, not doctests, deliberately, because none of them can run without a store or a @@ -6045,7 +6219,7 @@ network. If you add a `>>>` example, run: cd docs/src && make doctest ``` -- [ ] **Step 6: Commit** +- [ ] **Step 7: Commit** Run §5.0, then: @@ -6240,8 +6414,10 @@ warning fails CI. |---|---| | A netCDF file loads exactly as it did | §6.2's diff, plus `test_spans.py` running the same suite on both backends (Task 8) | | `_ARRAY_DIMENSIONS` does not reach a cube | Task 3 Step 9, and Task 13 Step 4 end to end | -| A version 3 array's zeros are not masked | Task 6 Step 5 | -| A version 2 array's fill value *is* a mask source | Task 6 Step 5 | +| A version 3 array's zeros are not masked | Task 6 Step 5 (the rule), Task 10 Step 7 (a loaded cube) | +| A version 2 array's fill value *is* a mask source | Task 6 Step 5, Task 10 Step 7 | +| A CF `_FillValue` beats the storage one | Task 6 Step 5, Task 10 Step 7 | +| A nested attribute reaches the cube, and saving it as netCDF fails loudly | Task 10 Step 7, documented in Task 14 | | Both `_FillValue` encodings read | Task 6 Steps 1–2, Task 13 Step 2 | | A sharded array is read by the inner chunk | Task 7 Step 9, on bytes fetched through an instrumented store | | The chunking tiles the store grid | Task 7 Steps 1–4, Task 13 Step 3 | @@ -6263,6 +6439,13 @@ warning fails CI. - **Over a network.** Spec §6: "No test touches the network. The URL path is exercised by checking that the right store is constructed, not by fetching." - **Performance.** The ASV benchmarks are PR 7. +- **A masked cube round-tripping its mask.** Spec §6 lists it among the fill + value cases; it is a save test, and it catches a saver that sets the storage + `fill_value` without also writing the `_FillValue` attribute. PR 6. +- **Saving a nested attribute to netCDF.** Task 10 pins that it fails, not + what the message says: fixing the message means changing the netCDF saver + ([#7329](https://github.com/SciTools/iris/issues/7329)), and PR 5 relocates + that code unchanged. --- From adbb712995e27877fabb6633af207f03b16c21fa Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 12:20:45 +0000 Subject: [PATCH 10/56] Correct the PR 4 plan against the files that are actually there The netCDF contract subclass lives in test_NetCDFDataset__contract.py, not test_NetCDFDataset.py. The sequencing paragraph now uses the task numbers the tasks ended up with, and the file table lists every test module the tasks create. Co-Authored-By: Claude Opus 5 Session-Id: 91122c91-913f-44c8-841f-f69ed66ac830 --- .../plans/2026-10-13-zarr-io-pr4.md | 42 ++++++++++++------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index 348ac9ed27..f09aa8ec56 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -122,7 +122,8 @@ owns the code, written out in that task's own step style. it, so unless `ZarrDatasetVariable.attributes` withholds it, every cube loaded from a version 2 store gains a stray `_ARRAY_DIMENSIONS` attribute and fails to compare equal to the same cube loaded from version 3. The user - expects the two to be indistinguishable. → **Task 3, Steps 9–10**. + expects the two to be indistinguishable. → **Task 3, Step 9**, and + **Task 13, Step 4** end to end. 2. **A store whose root group holds no arrays.** Spec §4.6 requires "a message that names the groups that do", and §7 records a real store in that shape — @@ -382,7 +383,10 @@ limitation and a surprise. | `lib/iris/fileformats/zarr/loader.py` | `load_cubes(file_sources, callback=None, constraints=None, group=None)` | ~180 | | `lib/iris/tests/unit/fileformats/zarr/__init__.py` | The package-wide `pytest.importorskip("zarr")` and the store-building fixtures | ~120 | | `lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py` | Empty package marker | 5 | -| `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py` | The shared read contract run against Zarr, plus `ZarrDataset`'s own cases | ~280 | +| `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py` | `ZarrDataset`'s own cases: dimensions, groups, errors, the refused write surface | ~230 | +| `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py` | The shared read contract, run against both Zarr versions | ~40 | +| `lib/iris/tests/unit/fileformats/zarr/test_CFReader.py` | `CFReader` classifying a Zarr store | ~30 | +| `lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py` | No module in the `cf` package type-tests an implementation class | ~50 | | `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py` | The dimension contract, `fill_value`, `chunking`, `encoding`, attributes | ~320 | | `lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py` | The alignment guard, on `chunking` values alone | ~140 | | `lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py` | Laziness, the cache key's staleness and sensitivity cases, the sharded read unit | ~300 | @@ -423,19 +427,20 @@ limitation and a surprise. The order is: make a Zarr store answerable as a `CFDataset`, then make the CF layer able to hold one, then make Iris able to find one. -1. **Tasks 1–6 build `ZarrDataset` bottom-up** and are testable without any of - Iris's loading machinery. Each one ends with a store built in `tmp_path` and - questions asked of it directly. -2. **Tasks 7–8 are the two changes to generic code.** They are separated from +1. **Tasks 1–7 build the Zarr dataset bottom-up** — dependency, interface, + variable, group, contract, decoding, reading — and are testable without any + of Iris's loading machinery. Each one ends with a store built in `tmp_path` + and questions asked of it directly. +2. **Tasks 8–9 are the two changes to generic code.** They are separated from the Zarr work because they are the only parts a netCDF-only reviewer has to check, and because `spans` is the one place in this pull request where a mistake silently changes what loads from an existing netCDF file. -3. **Task 9 is the loader**, which can only be written once 1–8 exist. -4. **Tasks 10–11 are the load chain**, which is independent of everything above +3. **Task 10 is the loader**, which can only be written once 1–9 exist. +4. **Tasks 11–12 are the load chain**, which is independent of everything above and could be reviewed by someone who knows nothing about Zarr. They come - last among the code tasks because until Task 9 exists there is nothing for + last among the code tasks because until Task 10 exists there is nothing for the chain to dispatch *to*. -5. **Tasks 12–14** are integration, documentation and the pull request. +5. **Tasks 13–15** are integration, documentation and the pull request. The dependency that matters in the other direction is **PR 3**. This plan calls `CFDatasetVariable.read_data`, `iris.fileformats.cf.loader._get_actual_dtype`, @@ -1069,8 +1074,9 @@ test package skips itself when the dependency is absent." - Modify: `lib/iris/fileformats/cf/dataset.py` - Modify: `lib/iris/fileformats/netcdf/_dataset.py` - Modify: `lib/iris/tests/unit/fileformats/cf/dataset/contract.py` -- Modify: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py` -- Modify: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py` +- Modify: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py` + — this is where `TestNetCDFDatasetContract` lives, not in + `test_NetCDFDataset.py` **Interfaces:** - Consumes: the existing `CFDataset` / `CFDatasetVariable` from PR 2. @@ -1276,11 +1282,17 @@ backend-specific - the omission is intentional, not a gap. """ ``` -In `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`, -change the test class to inherit both: +In `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py`, +change the test class to inherit both, and update its import: ```python -class TestNetCDFDataset(contract.CFDatasetReadContract, contract.CFDatasetWriteContract): +from iris.tests.unit.fileformats.cf.dataset.contract import ( + CFDatasetReadContract, + CFDatasetWriteContract, +) + + +class TestNetCDFDatasetContract(CFDatasetReadContract, CFDatasetWriteContract): ``` - [ ] **Step 7: Run the netCDF contract again and compare the count** From adfd266c2d27d550516afe3701ec6ae0c1079bb6 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 13:17:19 +0000 Subject: [PATCH 11/56] Point the PR 4 plan's baseline at the merged spec The spec review merged as #7330. Its tree is identical to the spec branch the plan was written against, so every [verified] claim stands unchanged; only the baseline pointer moves. Co-Authored-By: Claude Opus 5 Session-Id: 436a543d-0543-48a7-92b3-f8bef5d37d8a --- docs/superpowers/plans/2026-10-13-zarr-io-pr4.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index f09aa8ec56..d32c75111e 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -18,7 +18,7 @@ | **Implements** | The fourth of the eight pull requests in spec §5. The subject of [#6979](https://github.com/SciTools/iris/issues/6979) | | **Branch** | `zarr-io-pr-4`, targeting `SciTools/iris:brownfield` | | **Builds on** | **PR 3** ([#7316](https://github.com/SciTools/iris/pull/7316)), which must merge first: this plan is written against `iris.fileformats.cf.loader`, `CFDatasetVariable.read_data` and the `cache_key=` keyword on `as_lazy_data`, none of which exist on `brownfield` yet | -| **Baseline** | The spec branch at `be3cc7965`, Iris `3.17.0.dev`, zarr-python `3.4.1`, dask `2025.x`, NumPy `2.5.2` | +| **Baseline** | `brownfield` at `40d377d26`, where the spec review of [#7330](https://github.com/SciTools/iris/pull/7330) merged; Iris `3.17.0.dev`, zarr-python `3.4.1`, dask `2025.x`, NumPy `2.5.2` | | **Delivers** | `iris.fileformats.zarr` — `_dataset.py`, `_decode.py`, `loader.py` — the `"Zarr"` format specification, `iris.io.load_stores`, and `iris.load("store.zarr")` working for version 2 and version 3 stores, local and remote | | **Behaviour change** | Three, all previously errors: a local `.zarr` directory and an `s3`/`gs`/`az`/`abfs` URL now load instead of raising; `CFVariable.spans` matches on `(name, size)` pairs rather than bare names, which is a widening no netCDF file can observe | | **Public API change** | Additive: the new public module `iris.fileformats.zarr` (`load_cubes`), `iris.io.load_stores`, `CFDataset.format`, `CFDataset.encoding` and `CFDatasetVariable.encoding` | From 31ebd5563fceb5c6a71e6b0703e64fef5f40fb2b Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 14:47:50 +0000 Subject: [PATCH 12/56] Base the PR 4 plan on zarr-io-pr-3, with a fork preview first PR 3 is still in review, so PR 4 is cut from its branch rather than waiting for brownfield. The base branch of a pull request has to live in the repository it is raised against, and zarr-io-pr-3 is on the fork, so the draft is raised fork-internally: its diff is PR 4's work alone instead of PR 4 plus PR 3's ten commits. Task 15 gains a sixth step for when #7316 merges - rebase onto brownfield with the three-argument form, re-run the two tasks that read PR 3's code, re-compare the full suite against brownfield, raise the real pull request and close the preview. The full-suite baseline moves to zarr-io-pr-3 for the same reason: it is the only comparison that isolates this pull request's effect. Co-Authored-By: Claude Opus 5 Session-Id: 4aeb6748-437c-4241-b20c-c5e1be85150e --- .../plans/2026-10-13-zarr-io-pr4.md | 182 +++++++++++++++--- 1 file changed, 157 insertions(+), 25 deletions(-) diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md index d32c75111e..71b4ce3ef1 100644 --- a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md +++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md @@ -16,9 +16,9 @@ |---|---| | **Spec** | [`../specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md) §2.4, §3, §4.1, §4.2, §4.4, §4.6, §4.7, §5 (PR 4), §6 | | **Implements** | The fourth of the eight pull requests in spec §5. The subject of [#6979](https://github.com/SciTools/iris/issues/6979) | -| **Branch** | `zarr-io-pr-4`, targeting `SciTools/iris:brownfield` | -| **Builds on** | **PR 3** ([#7316](https://github.com/SciTools/iris/pull/7316)), which must merge first: this plan is written against `iris.fileformats.cf.loader`, `CFDatasetVariable.read_data` and the `cache_key=` keyword on `as_lazy_data`, none of which exist on `brownfield` yet | -| **Baseline** | `brownfield` at `40d377d26`, where the spec review of [#7330](https://github.com/SciTools/iris/pull/7330) merged; Iris `3.17.0.dev`, zarr-python `3.4.1`, dask `2025.x`, NumPy `2.5.2` | +| **Branch** | `zarr-io-pr-4`, cut from `zarr-io-pr-3`. A **preview** pull request opens on the fork, `trexfeathers/iris`, with `zarr-io-pr-3` as its base, so that its diff is PR 4's changes and nothing else. The real pull request opens against `SciTools/iris:brownfield` once PR 3 merges (Task 15, Step 6) | +| **Builds on** | **PR 3** ([#7316](https://github.com/SciTools/iris/pull/7316)), which is this branch's **base**, not a prerequisite waited on: this plan is written against `iris.fileformats.cf.loader`, `CFDatasetVariable.read_data` and the `cache_key=` keyword on `as_lazy_data`, none of which are on `brownfield` yet | +| **Baseline** | `zarr-io-pr-3` at `231cd019e`, which carries `brownfield` at `40d377d26` — where the spec review of [#7330](https://github.com/SciTools/iris/pull/7330) merged — plus PR 3's own ten commits; Iris `3.17.0.dev`, zarr-python `3.4.1`, dask `2025.x`, NumPy `2.5.2` | | **Delivers** | `iris.fileformats.zarr` — `_dataset.py`, `_decode.py`, `loader.py` — the `"Zarr"` format specification, `iris.io.load_stores`, and `iris.load("store.zarr")` working for version 2 and version 3 stores, local and remote | | **Behaviour change** | Three, all previously errors: a local `.zarr` directory and an `s3`/`gs`/`az`/`abfs` URL now load instead of raising; `CFVariable.spans` matches on `(name, size)` pairs rather than bare names, which is a widening no netCDF file can observe | | **Public API change** | Additive: the new public module `iris.fileformats.zarr` (`load_cubes`), `iris.io.load_stores`, `CFDataset.format`, `CFDataset.encoding` and `CFDatasetVariable.encoding` | @@ -68,7 +68,7 @@ memory. | Constraint | Value | Source | |---|---|---| -| Target branch | `brownfield`. **Not** `main` | spec §5 | +| Target branch | `brownfield`. **Not** `main`. Reached by rebase once PR 3 merges; until then the branch sits on `zarr-io-pr-3` and the preview pull request is based there | spec §5 | | Labels | `Agentic` and `Type: Feature Branch` | spec §5 | | Issue linkage | `Part of #6979` — never a closing keyword; those live on the merge-back | spec §5 | | Changelog | Fragments land with the merge-back; the pull request body must say the omission is deliberate | spec §5 | @@ -444,10 +444,28 @@ layer able to hold one, then make Iris able to find one. The dependency that matters in the other direction is **PR 3**. This plan calls `CFDatasetVariable.read_data`, `iris.fileformats.cf.loader._get_actual_dtype`, -`_LAZYVAR_MIN_BYTES` and `as_lazy_data(..., cache_key=...)`, none of which exist -until [#7316](https://github.com/SciTools/iris/pull/7316) merges. If this branch -is cut before then, rebase onto `brownfield` once it has merged and re-run -Task 6 and Task 9's tests before trusting anything. +`_LAZYVAR_MIN_BYTES` and `as_lazy_data(..., cache_key=...)`, none of which are +on `brownfield` until [#7316](https://github.com/SciTools/iris/pull/7316) +merges. So this branch is cut from `zarr-io-pr-3` rather than waiting, and +lands on `brownfield` later: + +```bash +git switch -c zarr-io-pr-4 zarr-io-pr-3 +``` + +Work proceeds on that base, and the preview pull request described in Task 15 +is raised against it on the fork. When #7316 merges, Task 15's Step 6 moves +this branch's own commits across: + +```bash +git fetch upstream +git rebase --onto upstream/brownfield zarr-io-pr-3 zarr-io-pr-4 +``` + +and re-runs Task 6 and Task 9's tests — the two that read PR 3's code most +closely — before anything is trusted again. Nothing between here and there +assumes `brownfield`; every instruction that names a base names +`zarr-io-pr-3`. --- @@ -6248,7 +6266,7 @@ begins by saying this is impossible." --- -### Task 15: the spec, the full-suite diff, and the pull request +### Task 15: the spec, the full-suite diff, and the pull requests **Files:** - Modify: `docs/superpowers/specs/2026-09-21-zarr-io-design.md` @@ -6256,16 +6274,20 @@ begins by saying this is impossible." **Interfaces:** - Consumes: everything. -- Produces: an open pull request, and a spec that tells the next reader the +- Produces: a preview pull request on the fork now, the real one against + `brownfield` when PR 3 merges, and a spec that tells the next reader the truth. +Steps 1–5 run now, on the `zarr-io-pr-3` base. **Step 6 runs later**, when +#7316 merges, and is the only step that touches `brownfield`. + - [ ] **Step 1: Run the full suite, twice** ```bash echo "$OVERRIDE_TEST_DATA_REPOSITORY" git stash list # expect empty -git switch --detach brownfield +git switch --detach zarr-io-pr-3 pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/baseline.txt git switch - @@ -6273,6 +6295,12 @@ pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/branch.txt diff /tmp/baseline.txt /tmp/branch.txt ``` +The baseline is `zarr-io-pr-3`, **not** `brownfield`: this branch's changes are +the ones made on top of PR 3, and measuring against `brownfield` would fold +PR 3's ten commits into the comparison and attribute their effects here. Step 6 +repeats the comparison against `brownfield` after the rebase, which is where +that question gets its own answer. + Expected: the branch adds passes and adds no failures. Compare **like with like** — a tier run says nothing about a full-suite one — and read the **skip** count as well as the failures. A skip that is new on the branch is a test that @@ -6282,8 +6310,10 @@ never ran. In `docs/superpowers/specs/2026-09-21-zarr-io-design.md`: -- §12.1: set PR 4's state to `In review` and record its number, as soon as the - pull request is opened. +- §12.1: set PR 4's state to `Built, awaiting PR 3` and say that the work is + complete on a branch based on `zarr-io-pr-3`. The state becomes `In review` + and gains a number in Step 6, when the real pull request opens — a fork + preview has no number worth recording in a document that outlives it. - The header table's **Progress** and **Next action** rows. - §12.3: move **Q2** to §12.4 with today's date and the answer from F3 — fixed-length unicode arrays read as themselves on both versions, verified @@ -6315,14 +6345,35 @@ implementation"**, listing every step whose code differed from what is written here and why. The plan ships with the pull request, so a reader comparing the two must not have to guess which one is true. -- [ ] **Step 4: Open the pull request** +- [ ] **Step 4: Open the preview pull request, on the fork** + +The base branch of a pull request has to live in the repository the pull +request is raised against, and `zarr-io-pr-3` lives on the fork. Raising this +against `SciTools/iris:brownfield` now would therefore show PR 3's ten commits +as part of PR 4. Instead the preview is raised fork-internally, where +`zarr-io-pr-3` *is* available as a base and the diff is exactly this pull +request's work: ```bash git push -u origin zarr-io-pr-4 -gh pr create --base brownfield --title "Zarr I/O PR 4: Zarr loading" --body-file /tmp/pr4-body.md -gh pr edit --add-label "Agentic" --add-label "Type: Feature Branch" +gh pr create --repo trexfeathers/iris --base zarr-io-pr-3 --head zarr-io-pr-4 \ + --draft --title "Zarr I/O PR 4: Zarr loading" --body-file /tmp/pr4-body.md ``` +No labels: `Agentic` and `Type: Feature Branch` are SciTools labels and may not +exist on the fork. They go on in Step 6. + +Know what this preview is and is not. It **is** a rehearsal — the real diff, +the real body, somewhere a reviewer can be sent by link, and a place to +self-review before anyone else reads it. It is **not** the pull request: it +does not appear in SciTools' list, a `Part of #6979` in it does not reach the +issue, its CI runs under the fork without the upstream's secrets and may not +run at all, and any review comment left on it stays behind when Step 6 opens +the real one. Do not solicit review on it that you would mind re-soliciting. + +Write the body now and keep the file: Step 6 reuses it unchanged, which is the +point of writing it this early. + The body opens `(posted by Claude Opus)` before anything else, per `AGENTS.md`, which **replaces** any trailing attribution — except the `🤖 Generated with Claude Code` line the harness asks for, which stays at the @@ -6332,7 +6383,9 @@ end. Never hard-wrap it. It must cover: `iris.load("s3://bucket/store.zarr")` return cubes, for version 2 and version 3 stores. - **Part of #6979.** Not `Closes` — a closing keyword only fires on a merge - into the default branch, and this targets `brownfield`. + into the default branch, and the real pull request targets `brownfield`. The + line is written the same way in the preview, where it is inert: a reference + from a fork does not reach the parent repository's issue. - **The changelog fragment is deferred to the merge-back**, deliberately, per spec §5. Say so, so the omission does not read as an oversight. - **Behaviour changes**, all three, each with what used to happen: a local @@ -6348,8 +6401,10 @@ end. Never hard-wrap it. It must cover: test dependency, UGRID meshes from Zarr. - **How to review it**: the task order in §3 of this plan is the reading order, and Tasks 8 and 12 are the two a netCDF-only reviewer should check. -- **The dependency on #7316**, and whether this branch was rebased after it - merged. +- **The dependency on #7316.** This branch is cut from `zarr-io-pr-3` and + calls four names that #7316 introduces. Say, in the preview, that it will be + rebased onto `brownfield` before the real pull request opens; in the real + one, say that it was. - [ ] **Step 5: Commit the spec and plan updates** @@ -6366,6 +6421,65 @@ format lands here rather than with the saver, and the shared dataset contract splits into read and write halves." ``` +- [ ] **Step 6: When #7316 merges — rebase, re-verify, raise the real pull + request** + +This step is deferred by definition. Everything above is done and pushed; this +is what turns a rehearsal into a pull request, and none of it can happen until +PR 3 is on `brownfield`. + +```bash +git fetch upstream +git log --oneline -1 upstream/brownfield # confirm #7316 is actually in +git rebase --onto upstream/brownfield zarr-io-pr-3 zarr-io-pr-4 +``` + +The three-argument form moves **only** this branch's own commits: everything +after `zarr-io-pr-3` is replayed onto `brownfield`, and PR 3's commits are left +behind because they are already there. A plain `git rebase upstream/brownfield` +would try to replay them too. + +Expect conflicts in `docs/superpowers/specs/2026-09-21-zarr-io-design.md` +alone, and only if PR 3 changed §12 on its way through review. The code should +not conflict: PR 3 moved the CF loader, and this branch calls it rather than +touching it. + +Then re-verify, because a rebase is a rewrite and nothing on the old commits +was tested in this arrangement: + +```bash +# The two tasks that read PR 3's code most closely. +pytest lib/iris/tests/unit/fileformats/zarr/test__decode.py \ + lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py -q + +# Then the whole suite, now against brownfield - the comparison Step 1 +# deliberately could not make. +git switch --detach upstream/brownfield +pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/baseline-brownfield.txt +git switch - +pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/branch-final.txt +diff /tmp/baseline-brownfield.txt /tmp/branch-final.txt +``` + +Update the spec's §12.1 state from `Built, awaiting PR 3` to `In review` with +the number, once there is one. Then: + +```bash +git push --force-with-lease origin zarr-io-pr-4 +gh pr create --repo SciTools/iris --base brownfield \ + --title "Zarr I/O PR 4: Zarr loading" --body-file /tmp/pr4-body.md +gh pr edit --add-label "Agentic" --add-label "Type: Feature Branch" +gh pr close --repo trexfeathers/iris \ + --comment "(posted by Claude Opus) + +Superseded by SciTools/iris#, now that #7316 has merged and this branch sits on brownfield." +``` + +The body is Step 4's file, with its #7316 paragraph changed from *will be +rebased* to *was rebased*. Close the preview rather than leaving it: two pull +requests showing the same work, one of them stale, is how a reviewer ends up +commenting on the wrong one. + --- ## 6. Verification @@ -6387,6 +6501,11 @@ machine, with `OVERRIDE_TEST_DATA_REPOSITORY` set and verified: pytest -n auto lib/iris/tests -q ``` +The base is `zarr-io-pr-3` while the branch sits there, and `brownfield` after +Task 15's Step 6 rebases it. Both comparisons get made, in that order: the +first is the only one that isolates this pull request's effect, and the second +is the one the eventual reviewer sees. + Record `passed`, `skipped`, `failed` and `errors` for both. The branch must add passes, add no failures, and **add no skips**. A new skip is a test that never ran, and the usual causes — a missing optional dependency, a missing @@ -6466,8 +6585,9 @@ warning fails CI. - [ ] Every task's checkboxes are ticked, and every commit was made with `pre-commit` clean and its tier of tests passing. - [ ] `pytest -n auto lib/iris/tests` adds passes, adds no failures and adds - no skips against `brownfield`, with `zarr` installed and - `OVERRIDE_TEST_DATA_REPOSITORY` set (§6.2). + no skips against `zarr-io-pr-3`, with `zarr` installed and + `OVERRIDE_TEST_DATA_REPOSITORY` set (§6.2) — and again against + `brownfield` after the rebase. - [ ] Iris imports and its suite runs with `zarr` absent (§6.3), or the pull request says the check could not be made. - [ ] `cd docs && make html` is warning-free (§6.4). @@ -6477,12 +6597,24 @@ warning fails CI. - [ ] `requirements/py3{12,13,14}.yml` carry `zarr >=3.0.8` and the lock files are regenerated — or the pull request asks a maintainer to regenerate them. -- [ ] The spec's §12.1, §12.3 and §12.4 are updated, and §5's PR 4 and PR 5 - entries reflect where `format` landed (Task 15 Step 2). +- [ ] The spec's §12.1 records PR 4 as `Built, awaiting PR 3`; §12.3 and §12.4 + are updated, and §5's PR 4 and PR 5 entries reflect where `format` + landed (Task 15 Step 2). - [ ] This plan carries a "What changed during implementation" section, or states that nothing did. -- [ ] The pull request is open against `brownfield`, labelled `Agentic` and - `Type: Feature Branch`, says `Part of #6979`, opens +- [ ] The preview pull request is open on `trexfeathers/iris`, based on + `zarr-io-pr-3`, in draft, with the body Step 6 will reuse. + +Then, once #7316 has merged (Task 15, Step 6 — the rest of this list can be +complete without it): + +- [ ] The branch is rebased onto `brownfield` with + `git rebase --onto upstream/brownfield zarr-io-pr-3`, and Task 6's and + Task 9's tests pass on the rebased commits. +- [ ] The pull request is open against `SciTools/iris:brownfield`, labelled + `Agentic` and `Type: Feature Branch`, says `Part of #6979`, opens `(posted by Claude Opus)`, and states that its changelog fragment is deferred to the merge-back. +- [ ] The preview pull request is closed, pointing at the real one. +- [ ] The spec's §12.1 records PR 4 as `In review`, with its number. From 54d5ac602e4db19b4dfb275db86c45cc178366b6 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 15:08:09 +0000 Subject: [PATCH 13/56] Add zarr as an optional dependency, and the package that uses it The package holds only its docstring for now: iris.fileformats does not import it, so Iris still installs and imports without zarr-python. The test package skips itself when the dependency is absent. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/zarr/__init__.py | 29 ++++++ .../tests/unit/fileformats/zarr/__init__.py | 14 +++ .../tests/unit/fileformats/zarr/stores.py | 96 +++++++++++++++++++ .../zarr/test_optional_dependency.py | 57 +++++++++++ requirements/py312.yml | 1 + requirements/py313.yml | 1 + requirements/py314.yml | 1 + 7 files changed, 199 insertions(+) create mode 100644 lib/iris/fileformats/zarr/__init__.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/__init__.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/stores.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py diff --git a/lib/iris/fileformats/zarr/__init__.py b/lib/iris/fileformats/zarr/__init__.py new file mode 100644 index 0000000000..e2cf16967d --- /dev/null +++ b/lib/iris/fileformats/zarr/__init__.py @@ -0,0 +1,29 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Native Zarr support, built on zarr-python. + +.. z_reference:: iris.fileformats.zarr + :tags: topic_load_save + + API reference + +Reads Zarr version 2 and version 3 stores, local or remote. Not NCZarr: that +is netCDF-c's own Zarr driver, reached through a URL fragment such as +``file:///path/store.zarr#mode=nczarr,file``, and it stays where it is in +:mod:`iris.fileformats.netcdf`. + +``zarr`` is an optional dependency. Nothing in :mod:`iris.fileformats` imports +this package until a Zarr target is actually loaded, so Iris installs and +imports without it. + +This package shadows the third-party ``zarr`` distribution by name. Python 3 +resolves ``import zarr`` absolutely, so inside these modules it still finds the +third-party package; xarray's ``xarray.backends.zarr`` has the same +arrangement. + +See sections 4.1 and 4.4 of +``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. + +""" diff --git a/lib/iris/tests/unit/fileformats/zarr/__init__.py b/lib/iris/tests/unit/fileformats/zarr/__init__.py new file mode 100644 index 0000000000..2016f9f175 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/__init__.py @@ -0,0 +1,14 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :mod:`iris.fileformats.zarr`. + +``zarr`` is optional, so every module below this one is skipped when it is +absent. The skip lives here rather than in a ``conftest.py``: the same call +there raises during collection and is reported as an **error**, not a skip. +""" + +import pytest + +pytest.importorskip("zarr", minversion="3.0.8") diff --git a/lib/iris/tests/unit/fileformats/zarr/stores.py b/lib/iris/tests/unit/fileformats/zarr/stores.py new file mode 100644 index 0000000000..5b3603dab0 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/stores.py @@ -0,0 +1,96 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Builders for the small Zarr stores these tests ask questions of. + +One canonical store, deliberately the same contents as the shared CFDataset +contract writes, so that a Zarr answer and a netCDF answer can be compared +without translating between two sets of fixtures. +""" + +import numpy as np +import zarr + +#: The canonical store's dimensions, in order. +DIMENSIONS = {"y": 3, "x": 2} +DATA = np.arange(6, dtype="f8").reshape(3, 2) +VARIABLE_ATTRS = {"units": "K", "long_name": "surface temperature"} +GLOBALS = {"Conventions": "CF-1.7", "title": "contract"} + +#: The version 2 attribute carrying dimension names. +ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS" + + +def make_array( + group, + name, + shape, + dtype, + zarr_format, + *, + dimensions=None, + chunks="auto", + shards=None, + fill_value=None, + attributes=None, +): + """Create one array, recording dimension names the way its version does. + + zarr-python refuses `dimension_names` on a version 2 array, so version 2 + names are written to the `_ARRAY_DIMENSIONS` attribute by hand - which is + what xarray and NCZarr both do. + """ + extra = {} + if zarr_format == 3 and dimensions is not None: + extra["dimension_names"] = dimensions + array = group.create_array( + name, + shape=shape, + dtype=dtype, + chunks=chunks, + shards=shards, + fill_value=fill_value, + **extra, + ) + if zarr_format == 2 and dimensions is not None: + array.attrs[ARRAY_DIMENSIONS] = list(dimensions) + for key, value in (attributes or {}).items(): + array.attrs[key] = value + return array + + +def make_store(path, *, zarr_format=3, chunks="auto", shards=None): + """Write the canonical store at `path`, and return `path` as a string.""" + root = zarr.open_group(store=path, mode="w", zarr_format=zarr_format) + for name, value in GLOBALS.items(): + root.attrs[name] = value + + array = make_array( + root, + "air_temperature", + DATA.shape, + "f8", + zarr_format, + dimensions=tuple(DIMENSIONS), + chunks=chunks, + shards=shards, + fill_value=-1.0, + attributes=VARIABLE_ATTRS, + ) + array[:] = DATA + + # A dimensionless variable: what a grid mapping is, and the one shape a + # CFDatasetVariable must handle without a leading dimension. + scalar = make_array( + root, + "grid", + (), + "i4", + zarr_format, + dimensions=(), + attributes={"grid_mapping_name": "latitude_longitude"}, + ) + scalar[()] = 0 + + return str(path) diff --git a/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py b/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py new file mode 100644 index 0000000000..6c05313f19 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py @@ -0,0 +1,57 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Iris must import, and load netCDF, without zarr-python installed.""" + +import subprocess +import sys + + +def _imports_zarr(statement): + """Run `statement` in a fresh interpreter; report whether zarr was imported.""" + probe = f"{statement}\nimport sys\nprint('zarr' in sys.modules)\n" + completed = subprocess.run( + [sys.executable, "-c", probe], + capture_output=True, + text=True, + check=True, + ) + return completed.stdout.strip() == "True" + + +def test_importing_iris_does_not_import_zarr(): + assert not _imports_zarr("import iris") + + +def test_importing_the_format_registry_does_not_import_zarr(): + # iris.fileformats imports every format module it registers a handler + # from. If the Zarr loader joined that list, Iris would stop installing + # without zarr-python. + assert not _imports_zarr("import iris.fileformats") + + +def test_importing_the_zarr_package_does_not_import_zarr(): + # The package docstring is reachable without the dependency; only the + # modules under it need zarr itself. + assert not _imports_zarr("import iris.fileformats.zarr") + + +import zarr + +from . import stores + + +def test_make_store_version_3(tmp_path): + location = stores.make_store(tmp_path / "v3.zarr") + group = zarr.open_group(store=location, mode="r") + assert group["air_temperature"].metadata.zarr_format == 3 + assert group["air_temperature"].metadata.dimension_names == ("y", "x") + + +def test_make_store_version_2(tmp_path): + location = stores.make_store(tmp_path / "v2.zarr", zarr_format=2) + group = zarr.open_group(store=location, mode="r") + array = group["air_temperature"] + assert array.metadata.zarr_format == 2 + assert array.attrs[stores.ARRAY_DIMENSIONS] == ["y", "x"] diff --git a/requirements/py312.yml b/requirements/py312.yml index 311a91f39d..d37a734226 100644 --- a/requirements/py312.yml +++ b/requirements/py312.yml @@ -37,6 +37,7 @@ dependencies: - python-stratify - rasterio - affine + - zarr >=3.0.8 # Test dependencies. - asv_runner diff --git a/requirements/py313.yml b/requirements/py313.yml index 4f27ae73e6..dfb47e25b0 100644 --- a/requirements/py313.yml +++ b/requirements/py313.yml @@ -37,6 +37,7 @@ dependencies: - python-stratify - rasterio - affine + - zarr >=3.0.8 # Test dependencies. - asv_runner diff --git a/requirements/py314.yml b/requirements/py314.yml index 79965dcdde..11187c6165 100644 --- a/requirements/py314.yml +++ b/requirements/py314.yml @@ -37,6 +37,7 @@ dependencies: - python-stratify - rasterio - affine + - zarr >=3.0.8 # Test dependencies. - asv_runner From ec0e5a04267e179d07fefa59c5721d617b12f0b3 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 15:16:08 +0000 Subject: [PATCH 14/56] Split the Zarr store-builder tests into their own module test_optional_dependency.py now checks only what its name says: Iris importing safely without zarr-python. The two tests proving make_store builds both Zarr format versions move to test_stores.py, named for the module they exercise, with their imports at the top where they belong. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- .../zarr/test_optional_dependency.py | 20 -------------- .../unit/fileformats/zarr/test_stores.py | 26 +++++++++++++++++++ 2 files changed, 26 insertions(+), 20 deletions(-) create mode 100644 lib/iris/tests/unit/fileformats/zarr/test_stores.py diff --git a/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py b/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py index 6c05313f19..e993e2021c 100644 --- a/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py +++ b/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py @@ -35,23 +35,3 @@ def test_importing_the_zarr_package_does_not_import_zarr(): # The package docstring is reachable without the dependency; only the # modules under it need zarr itself. assert not _imports_zarr("import iris.fileformats.zarr") - - -import zarr - -from . import stores - - -def test_make_store_version_3(tmp_path): - location = stores.make_store(tmp_path / "v3.zarr") - group = zarr.open_group(store=location, mode="r") - assert group["air_temperature"].metadata.zarr_format == 3 - assert group["air_temperature"].metadata.dimension_names == ("y", "x") - - -def test_make_store_version_2(tmp_path): - location = stores.make_store(tmp_path / "v2.zarr", zarr_format=2) - group = zarr.open_group(store=location, mode="r") - array = group["air_temperature"] - assert array.metadata.zarr_format == 2 - assert array.attrs[stores.ARRAY_DIMENSIONS] == ["y", "x"] diff --git a/lib/iris/tests/unit/fileformats/zarr/test_stores.py b/lib/iris/tests/unit/fileformats/zarr/test_stores.py new file mode 100644 index 0000000000..2cc6200f01 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/test_stores.py @@ -0,0 +1,26 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The builders in :mod:`stores` produce the stores the rest of these tests +ask questions of; this checks they build both Zarr versions correctly. +""" + +import zarr + +from . import stores + + +def test_make_store_version_3(tmp_path): + location = stores.make_store(tmp_path / "v3.zarr") + group = zarr.open_group(store=location, mode="r") + assert group["air_temperature"].metadata.zarr_format == 3 + assert group["air_temperature"].metadata.dimension_names == ("y", "x") + + +def test_make_store_version_2(tmp_path): + location = stores.make_store(tmp_path / "v2.zarr", zarr_format=2) + group = zarr.open_group(store=location, mode="r") + array = group["air_temperature"] + assert array.metadata.zarr_format == 2 + assert array.attrs[stores.ARRAY_DIMENSIONS] == ["y", "x"] From 66a544d5ced7206774115747a05d6420b6be34a5 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 15:27:37 +0000 Subject: [PATCH 15/56] Add format and encoding to the CF dataset interface format names the concrete format for a message or a diagnostic; encoding reports the storage settings a store actually holds, readable but never interpreted. Splits the shared contract into its read and write halves, so that an implementation arriving in two pieces can be held to the first. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/cf/dataset.py | 39 +++++ lib/iris/fileformats/netcdf/_dataset.py | 42 ++++- .../unit/fileformats/cf/dataset/contract.py | 164 +++++++++++------- .../fileformats/cf/dataset/test_CFDataset.py | 8 + .../dataset/test_NetCDFDatasetVariable.py | 17 +- .../dataset/test_NetCDFDataset__contract.py | 8 +- .../dataset/test_NetCDFDataset__write.py | 5 +- 7 files changed, 205 insertions(+), 78 deletions(-) diff --git a/lib/iris/fileformats/cf/dataset.py b/lib/iris/fileformats/cf/dataset.py index a153bb41eb..f1258ab3ce 100644 --- a/lib/iris/fileformats/cf/dataset.py +++ b/lib/iris/fileformats/cf/dataset.py @@ -148,6 +148,17 @@ def name(self) -> str: def location(self) -> str: """The path or URL of the dataset holding this variable.""" + @property + @abstractmethod + def format(self) -> str: + """The concrete format of the dataset holding this variable. + + Repeated here from the dataset so that a variable can answer for + itself, as :attr:`location` is. For messages and diagnostics only: + generic code asks for a capability, never for a format. + + """ + @property @abstractmethod def dimensions(self) -> tuple: @@ -178,6 +189,18 @@ def fill_value(self) -> Any: def chunking(self) -> tuple | None: """The variable's storage chunk shape, or ``None`` when unchunked.""" + @property + @abstractmethod + def encoding(self) -> Mapping: + """The storage settings this variable holds, in the backend's spelling. + + ``zlib``, ``complevel``, ``chunksizes`` for netCDF; ``chunks``, + ``shards``, ``codecs``, ``zarr_format`` for Zarr. Readable, never + interpreted: no generic code may branch on a key. The typed, + guaranteed-present member is :attr:`chunking`. + + """ + @property @abstractmethod def attributes(self) -> MutableMapping: @@ -263,6 +286,22 @@ def location(self) -> str: def mode(self) -> str: """The mode the store was opened in: ``r``, ``r+``, ``a``, ``w`` or ``w-``.""" + @property + @abstractmethod + def format(self) -> str: + """The concrete format: ``NETCDF4``, ``NETCDF3_CLASSIC``, ``ZARR_V2``, ``ZARR_V3``. + + What the loader, the saver and ``LOAD_PROBLEMS`` *say* when they have + to name the format. Reconstructing it from the implementation class + would be the type test this interface exists to forbid. + + """ + + @property + @abstractmethod + def encoding(self) -> Mapping: + """The store-wide settings this store holds, in the backend's spelling.""" + @property @abstractmethod def closed(self) -> bool: diff --git a/lib/iris/fileformats/netcdf/_dataset.py b/lib/iris/fileformats/netcdf/_dataset.py index fb82b84a6d..9dd1efbe54 100644 --- a/lib/iris/fileformats/netcdf/_dataset.py +++ b/lib/iris/fileformats/netcdf/_dataset.py @@ -23,6 +23,7 @@ """ from collections.abc import Iterator, Mapping, MutableMapping +from types import MappingProxyType from typing import Any import warnings @@ -113,7 +114,9 @@ def __repr__(self) -> str: class NetCDFDatasetVariable(CFDatasetVariable): """One variable of a netCDF file, presented through the CF interface.""" - def __init__(self, variable, location: str, *, write_lock_factory=None): + def __init__( + self, variable, location: str, format: str, *, write_lock_factory=None + ): """Wrap ``variable``, a thread-safe netCDF variable wrapper. Parameters @@ -125,6 +128,8 @@ def __init__(self, variable, location: str, *, write_lock_factory=None): file's own. This class reports whatever the wrapper reports. location : str The path or URL of the dataset this variable belongs to. + format : str + The netCDF format of the dataset this variable belongs to. write_lock_factory : callable, optional Returns the lock shared by every variable of one dataset. Called by :meth:`write_handle`. A callable rather than a lock, so that a @@ -133,6 +138,7 @@ def __init__(self, variable, location: str, *, write_lock_factory=None): """ self._variable = variable self._location = location + self._format = format self._write_lock_factory = write_lock_factory self._attributes = _NetCDFAttributes(variable) @@ -146,6 +152,11 @@ def location(self) -> str: """The path or URL of the dataset holding this variable.""" return self._location + @property + def format(self) -> str: + """The netCDF format of the file holding this variable.""" + return self._format + @property def dimensions(self) -> tuple: """The names of the dimensions this variable spans, in order.""" @@ -179,6 +190,18 @@ def chunking(self) -> tuple | None: return None return tuple(chunks) + @property + def encoding(self) -> Mapping: + """The netCDF storage settings this variable holds. + + Read back off the file, so a variable created with `zlib=True` reports + the compression netCDF4 actually applied rather than the keyword it + was given. + + """ + filters = self._variable.filters() or {} + return MappingProxyType({**filters, "chunksizes": self.chunking}) + @property def attributes(self) -> MutableMapping: """The variable's CF attributes.""" @@ -532,6 +555,17 @@ def mode(self) -> str: """The mode the file was opened in.""" return self._mode + @property + def format(self) -> str: + """The netCDF format of this file, as netCDF4 reports it.""" + assert self._dataset is not None + return self._dataset.file_format + + @property + def encoding(self) -> Mapping: + """The store-wide netCDF settings. Only the format is one.""" + return MappingProxyType({"format": self.format}) + @property def closed(self) -> bool: """Whether the file has been released, by this object or by its owner. @@ -556,6 +590,7 @@ def _materialised_variables(self) -> dict[str, "NetCDFDatasetVariable"]: name: NetCDFDatasetVariable( variable, self._location, + self._dataset.file_format, write_lock_factory=self._write_lock_factory, ) for name, variable in self._dataset.variables.items() @@ -606,7 +641,10 @@ def create_variable( name, dtype, tuple(dimensions), fill_value=fill_value, **encoding ) wrapped = NetCDFDatasetVariable( - variable, self._location, write_lock_factory=self._write_lock_factory + variable, + self._location, + self._dataset.file_format, + write_lock_factory=self._write_lock_factory, ) # Register this wrapper, so that a later lookup by name returns it # rather than a second wrapper with its own attribute cache. diff --git a/lib/iris/tests/unit/fileformats/cf/dataset/contract.py b/lib/iris/tests/unit/fileformats/cf/dataset/contract.py index 7a00f43f99..ef91395955 100644 --- a/lib/iris/tests/unit/fileformats/cf/dataset/contract.py +++ b/lib/iris/tests/unit/fileformats/cf/dataset/contract.py @@ -4,14 +4,20 @@ # See LICENSE in the root of the repository for full licensing details. """One test body, run against every :class:`CFDataset` implementation. -Subclass :class:`CFDatasetContract` in a module named for the implementation, -supply the three fixtures it declares, and every test here runs against it. -The class deliberately has no ``Test`` prefix, so pytest does not collect it -where it is written, only where it is subclassed. +Subclass :class:`CFDatasetReadContract` in a module named for the +implementation, supply the fixtures it declares, and every test here runs +against it. Both classes deliberately lack a ``Test`` prefix, so pytest does +not collect them where they are written, only where they are subclassed. -``docs/superpowers/specs/2026-09-21-zarr-io-design.md`` section 6: the interface -is only worth having if both sides of it agree, -and agreement is cheapest to check by asking the same questions twice. +The split is read from write because an implementation may arrive in two +pieces: the Zarr loader can answer every question in the read half before a +Zarr saver exists. Skipping half of one class instead would hide an +incomplete implementation behind a skip, and the point of a shared body is +that divergence fails loudly. + +``docs/superpowers/specs/2026-09-21-zarr-io-design.md`` section 6: the +interface is only worth having if both sides of it agree, and agreement is +cheapest to check by asking the same questions twice. Deliberately not exercised here: :meth:`CFDatasetVariable.deprecated_netcdf_member`. It is an explicitly netCDF4-specific compatibility escape hatch, so asserting @@ -19,6 +25,8 @@ backend-specific - the omission is intentional, not a gap. """ +from collections.abc import Mapping + import numpy as np import pytest @@ -31,50 +39,14 @@ CONTRACT_GLOBALS = {"Conventions": "CF-1.7", "title": "contract"} -def populate(dataset): - """Fill an empty writable dataset with the canonical contents. - - Uses only the CFDataset API, so this doubles as the write half of the - contract: an implementation that cannot build this cannot be used to save. - """ - for name, size in CONTRACT_DIMENSIONS.items(): - dataset.create_dimension(name, size) - for name, value in CONTRACT_GLOBALS.items(): - dataset.attributes[name] = value - - variable = dataset.create_variable( - "air_temperature", np.dtype("f8"), tuple(CONTRACT_DIMENSIONS), fill_value=-1.0 - ) - for name, value in CONTRACT_VARIABLE_ATTRS.items(): - variable.attributes[name] = value - variable[:] = CONTRACT_DATA - - # A dimensionless variable: what a grid mapping is, and the one shape a - # CFDatasetVariable must handle without a leading dimension. - scalar = dataset.create_variable("grid", np.dtype("i4")) - scalar.attributes["grid_mapping_name"] = "latitude_longitude" - scalar[()] = 0 - return dataset - - -class CFDatasetContract: - """What every CFDataset implementation must do, whatever it stores into.""" +class CFDatasetReadContract: + """What every CFDataset implementation must answer, whatever it reads from.""" @pytest.fixture def readable(self): """Return an open read-mode dataset holding what populate() writes.""" raise NotImplementedError("supply a 'readable' fixture") - @pytest.fixture - def writable(self): - """Return an open, empty, write-mode dataset.""" - raise NotImplementedError("supply a 'writable' fixture") - - @pytest.fixture - def reopen(self): - """Return a callable giving a fresh read-mode dataset over 'writable'.""" - raise NotImplementedError("supply a 'reopen' fixture") - # -- The dataset itself ------------------------------------------------ def test_is_a_cf_dataset(self, readable): @@ -90,20 +62,6 @@ def test_mode_is_a_read_mode(self, readable): def test_starts_open(self, readable): assert readable.closed is False - def test_close_then_closed(self, writable): - writable.close() - assert writable.closed is True - - def test_close_is_idempotent(self, writable): - writable.close() - writable.close() - assert writable.closed is True - - def test_context_manager_returns_self_and_closes(self, writable): - with writable as entered: - assert entered is writable - assert writable.closed is True - def test_dimensions(self, readable): assert dict(readable.dimensions) == CONTRACT_DIMENSIONS @@ -113,6 +71,19 @@ def test_global_attributes(self, readable): def test_variable_names(self, readable): assert sorted(readable.variables) == ["air_temperature", "grid"] + def test_format_is_a_non_empty_string(self, readable): + assert isinstance(readable.format, str) + assert readable.format + + def test_encoding_is_a_readable_mapping(self, readable): + # Its keys are the backend's own spelling and generic code must never + # branch on one, but it must be possible to read them: otherwise a + # caller that wants to know how a store is laid out has to reach + # through to the netCDF4 or zarr object, which is the abstraction + # leaking at the point it exists to hold. + assert isinstance(readable.encoding, Mapping) + dict(readable.encoding) + # -- A variable -------------------------------------------------------- @pytest.fixture @@ -132,6 +103,17 @@ def test_variable_name(self, variable): def test_variable_location_matches_its_dataset(self, variable, readable): assert variable.location == readable.location + def test_variable_format_matches_its_dataset(self, variable, readable): + assert variable.format == readable.format + + def test_variable_encoding_reports_the_chunking(self, variable): + encoding = variable.encoding + assert isinstance(encoding, Mapping) + # Whatever a backend spells it, the chunk shape it reports through + # `chunking` has to be somewhere in `encoding` too, or `encoding` is + # not reporting what the store holds. + assert variable.chunking in (None, *encoding.values()) + def test_variable_dimensions(self, variable): assert variable.dimensions == tuple(CONTRACT_DIMENSIONS) @@ -202,10 +184,64 @@ def test_scalar_variable(self, scalar): with pytest.raises(TypeError, match="unsized"): len(scalar) - # -- Writing ----------------------------------------------------------- + +class CFDatasetWriteContract: + """What every CFDataset implementation must do, whatever it stores into.""" + + @pytest.fixture + def writable(self): + """Return an open, empty, write-mode dataset.""" + raise NotImplementedError("supply a 'writable' fixture") + + @pytest.fixture + def reopen(self): + """Return a callable giving a fresh read-mode dataset over 'writable'.""" + raise NotImplementedError("supply a 'reopen' fixture") + + def populate(self, dataset): + """Fill an empty writable dataset with the canonical contents. + + Uses only the CFDataset API, so this doubles as the write half of the + contract: an implementation that cannot build this cannot be used to save. + """ + for name, size in CONTRACT_DIMENSIONS.items(): + dataset.create_dimension(name, size) + for name, value in CONTRACT_GLOBALS.items(): + dataset.attributes[name] = value + + variable = dataset.create_variable( + "air_temperature", + np.dtype("f8"), + tuple(CONTRACT_DIMENSIONS), + fill_value=-1.0, + ) + for name, value in CONTRACT_VARIABLE_ATTRS.items(): + variable.attributes[name] = value + variable[:] = CONTRACT_DATA + + # A dimensionless variable: what a grid mapping is, and the one shape a + # CFDatasetVariable must handle without a leading dimension. + scalar = dataset.create_variable("grid", np.dtype("i4")) + scalar.attributes["grid_mapping_name"] = "latitude_longitude" + scalar[()] = 0 + return dataset + + def test_close_then_closed(self, writable): + writable.close() + assert writable.closed is True + + def test_close_is_idempotent(self, writable): + writable.close() + writable.close() + assert writable.closed is True + + def test_context_manager_returns_self_and_closes(self, writable): + with writable as entered: + assert entered is writable + assert writable.closed is True def test_populate_then_read_back(self, writable, reopen): - populate(writable) + self.populate(writable) writable.close() with reopen() as reader: np.testing.assert_array_equal( @@ -242,7 +278,7 @@ def test_setitem_then_getitem(self, writable): np.testing.assert_array_equal(variable[:], [1.0, 2.0, 3.0]) def test_attribute_write_through(self, writable, reopen): - populate(writable) + self.populate(writable) writable.variables["air_temperature"].attributes["comment"] = "added" writable.close() with reopen() as reader: @@ -254,7 +290,7 @@ def test_write_handle_works_after_close(self, writable, reopen): # threading.Lock, and only the distributed scheduler - which supplies # a picklable one - ever ships a handle to another process. Using the # handle after close is the portable approximation. - populate(writable) + self.populate(writable) handle = writable.variables["air_temperature"].write_handle() writable.close() @@ -266,7 +302,7 @@ def test_write_handle_works_after_close(self, writable, reopen): ) def test_sync_and_finalise_are_callable(self, writable): - populate(writable) + self.populate(writable) assert writable.sync() is None assert writable.finalise() is None assert writable.closed is False diff --git a/lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py b/lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py index 859771a20f..2525657bd4 100644 --- a/lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py +++ b/lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py @@ -15,12 +15,14 @@ class MinimalVariable(CFDatasetVariable): name = "air_temperature" location = "" + format = "NETCDF4" dimensions = ("time", "lat") shape = (3, 4) dtype = np.dtype("f4") size = 12 fill_value = None chunking = None + encoding: dict = {} attributes: dict = {} def __getitem__(self, keys): @@ -41,10 +43,12 @@ class MinimalDataset(CFDataset): location = "" mode = "r" + format = "NETCDF4" closed = False variables: dict = {} dimensions: dict = {} attributes: dict = {} + encoding: dict = {} def __init__(self): self.closes = 0 @@ -79,12 +83,14 @@ def test_dataset_cannot_be_instantiated(self): [ "name", "location", + "format", "dimensions", "shape", "dtype", "size", "fill_value", "chunking", + "encoding", "attributes", "ndim", "__getitem__", @@ -103,10 +109,12 @@ def test_variable_declares_member(self, name): [ "location", "mode", + "format", "closed", "variables", "dimensions", "attributes", + "encoding", "create_dimension", "create_variable", "sync", diff --git a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py index 1d6018e2f2..131714e278 100644 --- a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py +++ b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py @@ -35,7 +35,10 @@ def encoded(sample_path): def wrap(dataset, name, location, write_lock_factory=None): return _dataset.NetCDFDatasetVariable( - dataset.variables[name], location, write_lock_factory=write_lock_factory + dataset.variables[name], + location, + dataset.file_format, + write_lock_factory=write_lock_factory, ) @@ -131,7 +134,7 @@ def test_values_are_read_once(self, mocker, sample_location): variable = mocker.Mock() variable.ncattrs.return_value = ["units"] variable.getncattr.return_value = "K" - wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location) + wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4") _ = wrapped.attributes["units"] _ = wrapped.attributes["units"] assert variable.ncattrs.call_count == 1 @@ -156,7 +159,7 @@ def test_unreadable_attribute_becomes_empty_string(self, mocker, sample_location variable.getncattr.side_effect = lambda name: ( "m" if name == "units" else _raise(AttributeError(name)) ) - wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location) + wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4") assert wrapped.attributes["broken"] == "" @@ -183,7 +186,7 @@ def test_variable_exposes_the_backing_wrapper(self, raw, sample_location): # so the check has to compare against the one instance actually # passed in, not a second, independently fetched wrapper. variable = raw.variables["air_temperature"] - wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location) + wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4") assert wrapped.variable is variable def test_not_variable_length(self, air): @@ -209,7 +212,7 @@ def test_emulated_round_trip(self, sample_location): # The emulation hook ncdata uses (issue #4994): an emulating variable # carries its own array instead of file storage. variable = _EmulatedVariable() - wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location) + wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4") variable._data_array = np.zeros(3) assert wrapped.is_emulated is True np.testing.assert_array_equal(wrapped.emulated_data_array, np.zeros(3)) @@ -248,7 +251,7 @@ def test_emulated_array_is_returned_as_it_stands(self, sample_location, mocker): mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0) variable = _EmulatedVariable() variable._data_array = np.arange(6, dtype="f4").reshape(2, 3) - wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location) + wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4") result = wrapped.read_data(_unreachable_policy) assert result is variable._data_array @@ -259,7 +262,7 @@ def test_emulated_array_builds_no_proxy(self, sample_location, mocker): proxy_class = mocker.patch.object(_thread_safe_nc, "NetCDFDataProxy") variable = _EmulatedVariable() variable._data_array = np.arange(6, dtype="f4").reshape(2, 3) - wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location) + wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4") wrapped.read_data(_unreachable_policy) assert proxy_class.call_count == 0 diff --git a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py index 0836f8c404..912c5462d1 100644 --- a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py +++ b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py @@ -8,19 +8,19 @@ from iris.fileformats.netcdf import _dataset from iris.tests.unit.fileformats.cf.dataset.contract import ( - CFDatasetContract, - populate, + CFDatasetReadContract, + CFDatasetWriteContract, ) -class TestNetCDFDatasetContract(CFDatasetContract): +class TestNetCDFDatasetContract(CFDatasetReadContract, CFDatasetWriteContract): """The netCDF side of the contract.""" @pytest.fixture def readable(self, tmp_path): path = tmp_path / "contract.nc" with _dataset.NetCDFDataset(path, mode="w", netcdf_format="NETCDF4") as dataset: - populate(dataset) + self.populate(dataset) with _dataset.NetCDFDataset(path) as dataset: yield dataset diff --git a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py index e8eca7d9dc..458efdfde7 100644 --- a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py +++ b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py @@ -145,7 +145,10 @@ def test_write_handle_of_an_unencoded_variable_still_encodes(self, path): raw.createDimension("x", 2) raw.createVariable("air", "f4", ("x",)) variable = _dataset.NetCDFDatasetVariable( - raw.variables["air"], str(path), write_lock_factory=None + raw.variables["air"], + str(path), + raw.file_format, + write_lock_factory=None, ) assert not isinstance( variable.variable, _bytecoding_datasets.EncodedVariable From 3bc38ca573f54db1a2cad56d024bd7e23feefb91 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 15:30:52 +0000 Subject: [PATCH 16/56] Keep close and context-manager coverage in the read half of the contract close, closed and the context manager are read-side behaviour per spec section 4.2, not write-side: a read-only implementation must still answer them correctly, and the whole point of splitting the contract is that a read-only Zarr store gets checked against every read-side question. Moving these three tests to the write half, as the first cut did, would let that divergence through unchecked. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- .../unit/fileformats/cf/dataset/contract.py | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/lib/iris/tests/unit/fileformats/cf/dataset/contract.py b/lib/iris/tests/unit/fileformats/cf/dataset/contract.py index ef91395955..67495f4ae6 100644 --- a/lib/iris/tests/unit/fileformats/cf/dataset/contract.py +++ b/lib/iris/tests/unit/fileformats/cf/dataset/contract.py @@ -62,6 +62,20 @@ def test_mode_is_a_read_mode(self, readable): def test_starts_open(self, readable): assert readable.closed is False + def test_close_then_closed(self, readable): + readable.close() + assert readable.closed is True + + def test_close_is_idempotent(self, readable): + readable.close() + readable.close() + assert readable.closed is True + + def test_context_manager_returns_self_and_closes(self, readable): + with readable as entered: + assert entered is readable + assert readable.closed is True + def test_dimensions(self, readable): assert dict(readable.dimensions) == CONTRACT_DIMENSIONS @@ -226,20 +240,6 @@ def populate(self, dataset): scalar[()] = 0 return dataset - def test_close_then_closed(self, writable): - writable.close() - assert writable.closed is True - - def test_close_is_idempotent(self, writable): - writable.close() - writable.close() - assert writable.closed is True - - def test_context_manager_returns_self_and_closes(self, writable): - with writable as entered: - assert entered is writable - assert writable.closed is True - def test_populate_then_read_back(self, writable, reopen): self.populate(writable) writable.close() From eac443e92e7b20a70e76868d5e1e021e5884da32 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 15:41:04 +0000 Subject: [PATCH 17/56] Reflow CFDataset.format's docstring under the 88-character line limit The one-line summary copied from the plan's interface sketch ran to 91 characters; nothing in the pre-commit toolchain catches an over-length docstring line (E501 is disabled repo-wide, and W505 needs a max-doc-length the repository does not set). Splitting it into a short summary plus an extended description also fixes it ahead of the Zarr docstrings a later task writes from the same sketch. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/cf/dataset.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/iris/fileformats/cf/dataset.py b/lib/iris/fileformats/cf/dataset.py index f1258ab3ce..4e6962f0ad 100644 --- a/lib/iris/fileformats/cf/dataset.py +++ b/lib/iris/fileformats/cf/dataset.py @@ -289,7 +289,9 @@ def mode(self) -> str: @property @abstractmethod def format(self) -> str: - """The concrete format: ``NETCDF4``, ``NETCDF3_CLASSIC``, ``ZARR_V2``, ``ZARR_V3``. + """The concrete format of this dataset. + + One of ``NETCDF4``, ``NETCDF3_CLASSIC``, ``ZARR_V2`` or ``ZARR_V3``. What the loader, the saver and ``LOAD_PROBLEMS`` *say* when they have to name the format. Reconstructing it from the implementation class From 06b8c5b07d0b4112b741e837f7827ffd23a7fdad Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 15:52:29 +0000 Subject: [PATCH 18/56] Present a Zarr array through the CF dataset interface The dimension contract is the substance: a Zarr dimension may be unnamed, and an unnamed one is anonymous rather than an error. _ARRAY_DIMENSIONS is storage, so it is withheld from the CF attributes, and fill_value is reported as the storage fact it is rather than as a missing-value flag. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/zarr/_dataset.py | 285 ++++++++++++++++++ .../unit/fileformats/zarr/dataset/__init__.py | 5 + .../zarr/dataset/test_ZarrDatasetVariable.py | 208 +++++++++++++ .../tests/unit/fileformats/zarr/stores.py | 14 +- 4 files changed, 510 insertions(+), 2 deletions(-) create mode 100644 lib/iris/fileformats/zarr/_dataset.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py diff --git a/lib/iris/fileformats/zarr/_dataset.py b/lib/iris/fileformats/zarr/_dataset.py new file mode 100644 index 0000000000..3ca496f41d --- /dev/null +++ b/lib/iris/fileformats/zarr/_dataset.py @@ -0,0 +1,285 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The Zarr implementation of the :mod:`iris.fileformats.cf.dataset` interface. + +This is the only module that knows both the CF interface and the zarr-python +API. Everything Zarr-specific the CF layer could want - where dimension names +are recorded, what a chunk is, what the store's own fill value is - either has +a named member on the interface or lives here as a Zarr-only member. + +Three things about Zarr shape this module, and each is a trap if forgotten. + +**A dimension name guarantees less than a netCDF one.** The version 3 core +specification makes ``dimension_names`` optional, allows a null entry, only +*recommends* that names are distinct within an array, and places no +restriction at all on reusing a name across arrays. All of that is conforming +Zarr, so none of it may raise. A dimension with no name is **anonymous**, which +Iris has always had: a cube dimension carrying no coordinate. + +**``Array.fill_value`` is not ``_FillValue``.** It says what unwritten storage +reads back as; it does not say the value is missing. zarr-python defaults it to +zero, so reading it as a missing-value sentinel masks every valid zero in an +array that never declared one. :attr:`ZarrDatasetVariable.fill_value` reports +it as the storage fact it is, and the masking decision is made in +:mod:`iris.fileformats.zarr._decode` from the CF attributes. + +**The read unit is the inner chunk, even when the array is sharded.** Zarr's +sharding codec is indexed, so a reader fetches only the inner chunks it needs. +:attr:`ZarrDatasetVariable.chunking` therefore returns ``Array.chunks``, never +``Array.shards``, and :func:`_align_chunks` aggregates upward into whole +multiples of it. + +See sections 4.2, 4.4 and 4.6 of +``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. + +""" + +from collections.abc import Callable, Iterator, Mapping, MutableMapping +from types import MappingProxyType +from typing import Any +import warnings + +import numpy as np +import zarr + +from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable +import iris.warnings + +#: What :attr:`CFDataset.format` answers, by the store's Zarr version. +_FORMAT_BY_ZARR_VERSION = {2: "ZARR_V2", 3: "ZARR_V3"} + +#: The version 2 attribute carrying dimension names, as xarray and NCZarr +#: write it. Storage, not a CF attribute, so it is withheld from `attributes`. +_ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS" + + +def _dimension_names(array) -> tuple: + """Return one name or None per dimension of `array`. + + Parameters + ---------- + array : :class:`zarr.Array` + The array to read names from. Version 3 carries them in its metadata; + version 2 carries them in the ``_ARRAY_DIMENSIONS`` attribute, which + is a de-facto convention rather than part of the Zarr specification. + + Returns + ------- + tuple + Of length ``array.ndim``. ``None`` where the store names no dimension. + + Warns + ----- + IrisCfLoadWarning + If the store names a number of dimensions other than ``array.ndim``. + The version 3 specification says the lengths "must" match, so such a + store is malformed; its names are ignored and the array loads with + anonymous dimensions. + + """ + anonymous = (None,) * array.ndim + if not array.ndim: + return () + + if array.metadata.zarr_format == 3: + names = array.metadata.dimension_names + else: + names = array.attrs.get(_ARRAY_DIMENSIONS) + + if names is None: + return anonymous + + if len(names) != array.ndim: + message = ( + "Ignoring malformed dimension_names, which names " + f"{len(names)} dimensions of a {array.ndim}-dimensional array: " + f"{array.basename}" + ) + warnings.warn(message, category=iris.warnings.IrisCfLoadWarning) + return anonymous + + return tuple(names) + + +class _ZarrAttributes(MutableMapping): + """A Zarr node's attributes as a mapping, less the ones that are storage. + + ``_ARRAY_DIMENSIONS`` is how a version 2 store records dimension names. + Under the interface's split that is a property, reported by + :attr:`ZarrDatasetVariable.dimensions`, not a CF attribute - so it is + withheld here. Leaving it in would put it on every cube loaded from a + version 2 store, since every attribute the loading rules do not consume + reaches ``cube.attributes``. + """ + + def __init__(self, node, *, withhold: frozenset = frozenset()): + """Materialise the attributes of `node`, a zarr array or group.""" + self._node = node + self._withhold = frozenset(withhold) + self._values = { + key: value for key, value in node.attrs.items() if key not in self._withhold + } + + def __getitem__(self, key: str) -> Any: + """Return `key`'s value.""" + return self._values[key] + + def __setitem__(self, key: str, value: Any) -> None: + """Set `key`'s value, here and in the store.""" + if key in self._withhold: + msg = f"Cannot set a storage attribute through `attributes`: {key}" + raise KeyError(msg) + self._node.attrs[key] = value + self._values[key] = value + + def __delitem__(self, key: str) -> None: + """Remove `key`, here and from the store.""" + del self._node.attrs[key] + del self._values[key] + + def __iter__(self) -> Iterator[str]: + """Return an iterator over the attribute names, in store order.""" + return iter(self._values) + + def __len__(self) -> int: + """Return the number of attributes.""" + return len(self._values) + + def __repr__(self) -> str: + """Return a string representation.""" + return f"{self.__class__.__name__}({self._values!r})" + + +class ZarrDatasetVariable(CFDatasetVariable): + """One array of a Zarr store, presented through the CF interface.""" + + def __init__(self, array, location: str, format: str): + """Wrap `array`. + + Parameters + ---------- + array : :class:`zarr.Array` + The array this variable presents. + location : str + The path or URL of the dataset this array belongs to. + format : str + ``ZARR_V2`` or ``ZARR_V3``, as its dataset reports. + + """ + self._array = array + self._location = location + self._format = format + self._dimensions = _dimension_names(array) + self._attributes = _ZarrAttributes( + array, withhold=frozenset({_ARRAY_DIMENSIONS}) + ) + + @property + def name(self) -> str: + """The array's name within its group.""" + return self._array.basename + + @property + def location(self) -> str: + """The path or URL of the dataset holding this variable.""" + return self._location + + @property + def format(self) -> str: + """``ZARR_V2`` or ``ZARR_V3``.""" + return self._format + + @property + def dimensions(self) -> tuple: + """One name per dimension, ``None`` where the store names none.""" + return self._dimensions + + @property + def shape(self) -> tuple: + """The array's shape.""" + return tuple(self._array.shape) + + @property + def dtype(self) -> np.dtype: + """The array's stored data type.""" + return self._array.dtype + + @property + def size(self) -> int: + """The total number of elements in the array.""" + return int(self._array.size) + + @property + def fill_value(self) -> Any: + """What unwritten storage reads back as. **Not** a missing-value flag. + + Version 3 requires this field, and zarr-python defaults it to zero, so + it says nothing about whether a value is missing. The CF + ``_FillValue`` and ``missing_value`` attributes say that, and + :mod:`iris.fileformats.zarr._decode` reads them. + + """ + return self._array.fill_value + + @property + def chunking(self) -> tuple | None: + """The array's inner chunk shape, never its shard shape.""" + chunks = self._array.chunks + return tuple(chunks) if chunks else None + + @property + def encoding(self) -> Mapping: + """The Zarr storage settings this array holds.""" + return MappingProxyType( + { + "chunks": self._array.chunks, + "shards": self._array.shards, + "codecs": self._array.metadata.to_dict().get("codecs"), + "zarr_format": self._array.metadata.zarr_format, + } + ) + + @property + def attributes(self) -> MutableMapping: + """The array's CF attributes, less the ones that are really storage.""" + return self._attributes + + def __getitem__(self, keys) -> np.ndarray: + """Return the indexed portion of the array's data.""" + return self._array[keys] + + def __setitem__(self, keys, values) -> None: + """Write `values` into the indexed portion of the array.""" + self._array[keys] = values + + def read_data(self, chunking_policy: Callable[[], tuple]) -> Any: + """Not yet implemented. + + A later change adds this, following §4.4 of + ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. + + """ + raise NotImplementedError + + def write_handle(self) -> "ZarrDatasetVariable": + """Return self: a Zarr write target needs no proxy. + + ``zarr.Array`` is picklable and a chunk is an independent object in a + key-value store, so there is no open file handle to avoid holding and + no lock to acquire. + + """ + return self + + def __repr__(self) -> str: + """Return a string representation.""" + return f"{self.__class__.__name__}({self.name!r}, {self.location!r})" + + # Zarr-only members below. + + @property + def array(self): + """The backing :class:`zarr.Array`. Nothing generic may read this.""" + return self._array diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py b/lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py new file mode 100644 index 0000000000..66f11b11ae --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py @@ -0,0 +1,5 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDatasetVariable`.""" diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py new file mode 100644 index 0000000000..6be8a5e5f5 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py @@ -0,0 +1,208 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDatasetVariable`.""" + +import numpy as np +import pytest +import zarr + +from iris.fileformats.zarr._dataset import ZarrDatasetVariable +from iris.warnings import IrisCfLoadWarning + +from .. import stores + + +def wrap(array, location="store.zarr"): + """Return the ZarrDatasetVariable over `array`, naming its own format.""" + zarr_format = array.metadata.zarr_format + return ZarrDatasetVariable(array, location, f"ZARR_V{zarr_format}") + + +@pytest.fixture(params=[2, 3], ids=["v2", "v3"]) +def zarr_format(request): + return request.param + + +@pytest.fixture +def group(tmp_path, zarr_format): + return zarr.open_group( + store=tmp_path / "probe.zarr", mode="w", zarr_format=zarr_format + ) + + +def test_named_dimensions(group, zarr_format): + array = stores.make_array( + group, "v", (3, 2), "f8", zarr_format, dimensions=("y", "x") + ) + assert wrap(array).dimensions == ("y", "x") + + +def test_unnamed_dimensions_are_anonymous(group, zarr_format): + array = stores.make_array(group, "v", (3, 2), "f8", zarr_format) + assert wrap(array).dimensions == (None, None) + + +def test_scalar_has_no_dimensions(group, zarr_format): + array = stores.make_array(group, "v", (), "i4", zarr_format) + assert wrap(array).dimensions == () + + +def test_partially_named_dimensions_version_3(tmp_path): + # The version 3 core specification: "an unnamed dimension is indicated by + # the null object". A store may name some dimensions and not others. + group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3) + array = group.create_array( + "v", shape=(3, 2), dtype="f8", dimension_names=("y", None) + ) + assert wrap(array).dimensions == ("y", None) + + +def test_repeated_name_within_one_array(tmp_path): + # "It is recommended but not required that all non-null dimension names + # are distinct." Recommended is not required, so this must not raise. + group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3) + array = group.create_array( + "v", shape=(2, 2), dtype="f8", dimension_names=("d", "d") + ) + assert wrap(array).dimensions == ("d", "d") + + +def test_wrong_length_dimension_names_warns_and_loads_anonymous(tmp_path): + """A dimension-name count that disagrees with `ndim` is a malformation. + + Version 2 only: the version 3 core specification says `dimension_names` + "must" be the same length as `shape`, and zarr-python enforces that on + every read, not just on write - a hand-edited `zarr.json` with the wrong + length raises `ValueError` from `Array.open` itself, before Iris sees an + array at all. `_ARRAY_DIMENSIONS` carries no such validation, so this is + reachable only through a version 2 store. + """ + group = zarr.open_group(store=tmp_path / "bad.zarr", mode="w", zarr_format=2) + array = stores.make_array(group, "v", (3, 2), "f8", 2, dimensions=("y", "x")) + array.attrs[stores.ARRAY_DIMENSIONS] = ["y"] + + with pytest.warns(IrisCfLoadWarning, match="dimension_names"): + dimensions = wrap(array).dimensions + assert dimensions == (None, None) + + +def test_plain_properties(tmp_path, zarr_format): + location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format) + array = zarr.open_group(store=location, mode="r")["air_temperature"] + variable = wrap(array, location) + + assert variable.name == "air_temperature" + assert variable.location == location + assert variable.format == f"ZARR_V{zarr_format}" + assert variable.shape == stores.DATA.shape + assert variable.dtype == np.dtype("f8") + assert variable.size == stores.DATA.size + assert variable.ndim == stores.DATA.ndim + assert len(variable) == stores.DATA.shape[0] + np.testing.assert_array_equal(variable[:], stores.DATA) + np.testing.assert_array_equal(variable[1], stores.DATA[1]) + + +def test_fill_value_is_the_storage_field(tmp_path, zarr_format): + """`fill_value` is what unwritten storage reads as, not `_FillValue`. + + zarr-python defaults the field to zero, so an array that declares no CF + sentinel still reports one here. Reading it as a mask source is what + masks every valid zero. + """ + group = zarr.open_group( + store=tmp_path / "f.zarr", mode="w", zarr_format=zarr_format + ) + array = stores.make_array(group, "v", (3,), "i2", zarr_format) + assert wrap(array).fill_value == 0 + + declared = stores.make_array(group, "w", (3,), "i2", zarr_format, fill_value=-999) + assert wrap(declared).fill_value == -999 + + +def test_chunking_is_the_inner_chunk_not_the_shard(tmp_path): + """A sharded array reports `Array.chunks`. + + Zarr's sharding codec is indexed: a reader fetches the shard index, then + byte ranges for only the inner chunks it needs. The inner chunk is the + independently decodable unit. + """ + group = zarr.open_group(store=tmp_path / "sh.zarr", mode="w", zarr_format=3) + array = group.create_array( + "v", shape=(1024,), dtype="i4", chunks=(8,), shards=(1024,) + ) + variable = wrap(array) + assert variable.chunking == (8,) + assert variable.encoding["shards"] == (1024,) + + +def test_encoding_reports_the_store(tmp_path, zarr_format): + location = stores.make_store(tmp_path / "e.zarr", zarr_format=zarr_format) + array = zarr.open_group(store=location, mode="r")["air_temperature"] + encoding = wrap(array, location).encoding + assert encoding["chunks"] == array.chunks + assert encoding["zarr_format"] == zarr_format + + +def test_array_dimensions_is_not_a_cf_attribute(tmp_path): + """`_ARRAY_DIMENSIONS` is storage, so it must not reach `attributes`. + + It is how a version 2 store records dimension names. Every attribute the + loading rules do not consume reaches `cube.attributes`, so leaving it in + would put it on every cube loaded from a version 2 store, and make that + cube unequal to the same data loaded from version 3. + """ + group = zarr.open_group(store=tmp_path / "v2.zarr", mode="w", zarr_format=2) + array = stores.make_array( + group, + "v", + (3, 2), + "f8", + 2, + dimensions=("y", "x"), + attributes={"units": "K"}, + ) + variable = wrap(array) + + assert variable.dimensions == ("y", "x") + assert stores.ARRAY_DIMENSIONS in array.attrs + assert stores.ARRAY_DIMENSIONS not in variable.attributes + assert dict(variable.attributes) == {"units": "K"} + + +def test_version_2_and_version_3_present_the_same_attributes(tmp_path): + v2 = stores.make_store(tmp_path / "a2.zarr", zarr_format=2) + v3 = stores.make_store(tmp_path / "a3.zarr", zarr_format=3) + from_2 = zarr.open_group(store=v2, mode="r")["air_temperature"] + from_3 = zarr.open_group(store=v3, mode="r")["air_temperature"] + assert dict(wrap(from_2, v2).attributes) == dict(wrap(from_3, v3).attributes) + + +def test_structured_attribute_values_are_carried_unchanged(tmp_path): + """A JSON object or array attribute is carried, not interpreted. + + The version 3 specification allows an attribute to be any JSON literal, + and CF cannot express a nested one at any rank. Two independent producers + write them - NOAA GFS gives `valid_time` a `statistics_approximate` + object, and the ESA EOPF product root carries `stac_discovery`. + A flat numeric array is an ordinary CF vector attribute and is not special. + """ + group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "v", + (2,), + "f8", + 3, + attributes={ + "statistics_approximate": {"min": 0.0, "max": 1.0}, + "valid_range": [0.0, 1.0], + "nested_list": [[1, 2], [3, 4]], + }, + ) + attributes = wrap(array).attributes + assert attributes["statistics_approximate"] == {"min": 0.0, "max": 1.0} + assert attributes["valid_range"] == [0.0, 1.0] + assert attributes["nested_list"] == [[1, 2], [3, 4]] diff --git a/lib/iris/tests/unit/fileformats/zarr/stores.py b/lib/iris/tests/unit/fileformats/zarr/stores.py index 5b3603dab0..617fb8510e 100644 --- a/lib/iris/tests/unit/fileformats/zarr/stores.py +++ b/lib/iris/tests/unit/fileformats/zarr/stores.py @@ -21,6 +21,15 @@ #: The version 2 attribute carrying dimension names. ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS" +#: `make_array`'s own default for `fill_value`, distinct from `None`. +#: +#: `Group.create_array` treats an explicit `fill_value=None` as "this array +#: has no fill value" on a version 2 store, which is not the same thing as +#: "the caller did not ask for one". Passing this sentinel through instead +#: means `make_array`'s own default lets zarr-python choose its own default +#: fill value, on both versions, the way omitting the argument would. +_DEFAULT_FILL_VALUE = object() + def make_array( group, @@ -32,7 +41,7 @@ def make_array( dimensions=None, chunks="auto", shards=None, - fill_value=None, + fill_value=_DEFAULT_FILL_VALUE, attributes=None, ): """Create one array, recording dimension names the way its version does. @@ -44,13 +53,14 @@ def make_array( extra = {} if zarr_format == 3 and dimensions is not None: extra["dimension_names"] = dimensions + if fill_value is not _DEFAULT_FILL_VALUE: + extra["fill_value"] = fill_value array = group.create_array( name, shape=shape, dtype=dtype, chunks=chunks, shards=shards, - fill_value=fill_value, **extra, ) if zarr_format == 2 and dimensions is not None: From 77b1d3fa4603d52e7b398bc9e0282f1b5e712e6c Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 16:05:29 +0000 Subject: [PATCH 19/56] Present a Zarr group through the CF dataset interface Dimensions are assembled from the arrays, and a name used at two sizes is omitted with a warning rather than merged: Zarr places no restriction on reusing a name across arrays, so a store meaning two things by one name is conforming, and merging them would produce a wrong cube. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/zarr/_dataset.py | 219 ++++++++++++++++++ .../zarr/dataset/test_ZarrDataset.py | 157 +++++++++++++ 2 files changed, 376 insertions(+) create mode 100644 lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py diff --git a/lib/iris/fileformats/zarr/_dataset.py b/lib/iris/fileformats/zarr/_dataset.py index 3ca496f41d..f230bca89f 100644 --- a/lib/iris/fileformats/zarr/_dataset.py +++ b/lib/iris/fileformats/zarr/_dataset.py @@ -103,6 +103,52 @@ def _dimension_names(array) -> tuple: return tuple(names) +def _assemble_dimensions(variables) -> dict: + """Return the dimension sizes the store's arrays agree on. + + Parameters + ---------- + variables : mapping of str to :class:`ZarrDatasetVariable` + Every array in one group. + + Returns + ------- + dict + Name to size, for every name used at exactly one size. Anonymous + dimensions have no name and so are absent. + + Warns + ----- + IrisCfLoadWarning + For each name used at more than one size. Zarr places no restriction + on reusing a name across arrays, so this is conforming and must not + raise; merging the two would produce a wrong cube, so the name is + omitted instead and the arrays that use it keep it in their own + `dimensions`. + + """ + sizes: dict[str, int] = {} + conflicts: dict[str, set[int]] = {} + for variable in variables.values(): + for name, size in zip(variable.dimensions, variable.shape): + if name is None: + continue + if name not in sizes: + sizes[name] = size + elif sizes[name] != size: + conflicts.setdefault(name, {sizes[name]}).add(size) + + for name, found in conflicts.items(): + message = ( + "Ignoring a dimension name used at more than one size, " + f"{sorted(found)}: {name}" + ) + warnings.warn(message, category=iris.warnings.IrisCfLoadWarning) + del sizes[name] + + return sizes + + class _ZarrAttributes(MutableMapping): """A Zarr node's attributes as a mapping, less the ones that are storage. @@ -283,3 +329,176 @@ def __repr__(self) -> str: def array(self): """The backing :class:`zarr.Array`. Nothing generic may read this.""" return self._array + + +#: What `mode` a dataset may be opened in. Writing is not implemented. +_READ_MODES = ("r",) + +#: The one message every unimplemented write member raises. +_NO_SAVING = "Saving to Zarr is not supported." + + +class ZarrDataset(CFDataset): + """A Zarr store, or one group of one, presented through the CF interface.""" + + def __init__(self, location, mode: str = "r", *, group: str | None = None): + """Open a Zarr store. + + Parameters + ---------- + location : str or :class:`pathlib.Path` + The store's path or URL. A URL is handed to zarr-python + unaltered; Iris adds no store library and handles no credentials, + so an unreachable scheme fails with zarr-python's own message. A + store object - such as a :class:`zarr.storage.MemoryStore`, which + has no path - is also accepted and handed through unaltered; + `location` then becomes zarr-python's own rendering of it, the + best name available for a store with no address. + mode : str, default="r" + Only ``"r"`` is supported. + group : str, optional + A ``/``-separated path to a group within the store. The root group + by default. Iris reads one group per call and does not walk + nested groups. + + Raises + ------ + NotImplementedError + For any `mode` other than ``"r"``. + FileNotFoundError + If `location` does not exist. + zarr.errors.GroupNotFoundError + If `location` is not a Zarr store, or holds no such `group`. + + """ + # Set first, so that close() is safe if opening fails. + self._group: zarr.Group | None = None + self._closed = False + self._variables: dict[str, ZarrDatasetVariable] | None = None + self._attributes: _ZarrAttributes | None = None + self._dimensions: dict[str, int] | None = None + + if mode not in _READ_MODES: + raise NotImplementedError(f"{_NO_SAVING} Cannot open with mode={mode!r}.") + + self._mode = mode + self._group_path = group + target = str(location) + self._location = target if group is None else f"{target.rstrip('/')}/{group}" + + # `location` itself, not its stringified `target`, is handed to + # zarr-python: stringifying a store object such as a `MemoryStore` + # loses the store, leaving only a name zarr-python cannot reopen. + self._group = zarr.open_group(store=location, mode=mode, path=group) + self._format = _FORMAT_BY_ZARR_VERSION[self._group.metadata.zarr_format] + + @property + def location(self) -> str: + """The store's path or URL, with the group path when there is one.""" + return self._location + + @property + def format(self) -> str: + """``ZARR_V2`` or ``ZARR_V3``.""" + return self._format + + @property + def mode(self) -> str: + """The mode the store was opened in.""" + return self._mode + + @property + def closed(self) -> bool: + """Whether :meth:`close` has been called. + + A Zarr store is a namespace rather than an open file handle, so there + is nothing to release and nothing else to ask. + + """ + return self._closed + + @property + def variables(self) -> Mapping: + """The group's arrays, by name. Arrays in subgroups are not included.""" + if self._variables is None: + assert self._group is not None + self._variables = { + name: ZarrDatasetVariable(array, self._location, self._format) + for name, array in self._group.arrays() + } + return self._variables + + @property + def dimensions(self) -> Mapping: + """The dimension sizes this group's arrays agree on.""" + if self._dimensions is None: + self._dimensions = _assemble_dimensions(self.variables) + return self._dimensions + + @property + def encoding(self) -> Mapping: + """The store-wide Zarr settings.""" + assert self._group is not None + consolidated = self._group.metadata.consolidated_metadata + return MappingProxyType( + { + "zarr_format": self._group.metadata.zarr_format, + "consolidated": consolidated is not None, + } + ) + + @property + def attributes(self) -> MutableMapping: + """The group's attributes.""" + if self._attributes is None: + self._attributes = _ZarrAttributes(self._group) + return self._attributes + + def create_dimension(self, name: str, size: int | None) -> None: + """Raise: saving to Zarr is not supported.""" + raise NotImplementedError(_NO_SAVING) + + def create_variable( + self, name: str, dtype, dimensions=(), *, fill_value=None, **encoding + ): + """Raise: saving to Zarr is not supported.""" + raise NotImplementedError(_NO_SAVING) + + def sync(self) -> None: + """Do nothing: a Zarr write is not buffered.""" + + def finalise(self) -> None: + """Raise: saving to Zarr is not supported. + + A Zarr store's completion step is consolidating its metadata, which + only a writer performs. + + """ + raise NotImplementedError(_NO_SAVING) + + def close(self) -> None: + """Record that this dataset is finished with. + + A store is a namespace, not an open handle, so nothing is released. + The flag exists because callers - :class:`iris.fileformats.cf.CFReader` + among them - must be able to ask. + + """ + self._closed = True + + def __repr__(self) -> str: + """Return a string representation.""" + return f"{self.__class__.__name__}({self.location!r}, mode={self.mode!r})" + + # Zarr-only members below. + + @property + def subgroup_names(self) -> tuple: + """The names of this group's subgroups. + + What the loader names when a group holds no data variables but the + store plainly holds data elsewhere. + + """ + assert self._group is not None + return tuple(self._group.group_keys()) diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py new file mode 100644 index 0000000000..601b0ea6f1 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py @@ -0,0 +1,157 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDataset`.""" + +import numpy as np +import pytest +import zarr + +from iris.fileformats.zarr._dataset import ZarrDataset +from iris.warnings import IrisCfLoadWarning + +from .. import stores + + +@pytest.fixture(params=[2, 3], ids=["v2", "v3"]) +def zarr_format(request): + return request.param + + +def test_opens_and_reports_itself(tmp_path, zarr_format): + location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format) + dataset = ZarrDataset(location) + + assert dataset.location == location + assert dataset.format == f"ZARR_V{zarr_format}" + assert dataset.mode == "r" + assert dataset.closed is False + assert sorted(dataset.variables) == ["air_temperature", "grid"] + assert dict(dataset.attributes) == stores.GLOBALS + assert dataset.encoding["zarr_format"] == zarr_format + + +def test_dimensions_are_assembled_from_the_arrays(tmp_path, zarr_format): + location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format) + assert dict(ZarrDataset(location).dimensions) == stores.DIMENSIONS + + +def test_a_name_used_at_two_sizes_is_omitted_and_warns(tmp_path): + """Two arrays may share a dimension name at different lengths. + + The version 3 specification "does not place any restrictions on the use of + the same dimension name across multiple arrays", so this is conforming + Zarr and must not raise. Merging the two would produce a wrong cube. + """ + group = zarr.open_group(store=tmp_path / "c.zarr", mode="w", zarr_format=3) + stores.make_array(group, "short", (2,), "f8", 3, dimensions=("t",)) + stores.make_array(group, "long", (5,), "f8", 3, dimensions=("t",)) + + with pytest.warns(IrisCfLoadWarning, match="t"): + dimensions = dict(ZarrDataset(str(tmp_path / "c.zarr")).dimensions) + + assert "t" not in dimensions + + +def test_anonymous_dimensions_are_not_in_the_mapping(tmp_path): + group = zarr.open_group(store=tmp_path / "a.zarr", mode="w", zarr_format=3) + stores.make_array(group, "named", (2,), "f8", 3, dimensions=("t",)) + stores.make_array(group, "unnamed", (4,), "f8", 3) + + dataset = ZarrDataset(str(tmp_path / "a.zarr")) + assert dict(dataset.dimensions) == {"t": 2} + + +def test_close_is_idempotent(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + dataset = ZarrDataset(location) + dataset.close() + dataset.close() + assert dataset.closed is True + + +def test_context_manager_closes(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + dataset = ZarrDataset(location) + with dataset as entered: + assert entered is dataset + assert dataset.closed is True + + +def test_group_keyword_selects_a_subgroup(tmp_path): + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + sub = root.create_group("measurements/r10m") + stores.make_array(sub, "b02", (2,), "u2", 3, dimensions=("p",)) + + dataset = ZarrDataset(str(tmp_path / "n.zarr"), group="measurements/r10m") + assert list(dataset.variables) == ["b02"] + assert dataset.location.endswith("measurements/r10m") + + +def test_subgroup_names_lists_only_the_groups(tmp_path): + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + root.create_group("conditions") + root.create_group("measurements") + stores.make_array(root, "v", (2,), "f8", 3) + + dataset = ZarrDataset(str(tmp_path / "n.zarr")) + assert sorted(dataset.subgroup_names) == ["conditions", "measurements"] + + +def test_nested_groups_are_not_walked(tmp_path): + """Iris loads one group per call, like xarray. + + A Zarr store can legitimately hold unrelated datasets, and merging them + into one CubeList would be wrong. + """ + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + stores.make_array(root, "top", (2,), "f8", 3) + stores.make_array(root.create_group("sub"), "inner", (2,), "f8", 3) + + assert list(ZarrDataset(str(tmp_path / "n.zarr")).variables) == ["top"] + + +def test_the_write_surface_refuses(tmp_path): + # Saving to Zarr is not implemented. The members exist because the + # interface declares them; refusing is the honest answer. + dataset = ZarrDataset(stores.make_store(tmp_path / "s.zarr")) + with pytest.raises(NotImplementedError, match="Saving"): + dataset.create_dimension("t", 3) + with pytest.raises(NotImplementedError, match="Saving"): + dataset.create_variable("v", np.dtype("f4"), ("t",)) + with pytest.raises(NotImplementedError, match="Saving"): + dataset.finalise() + + +def test_a_write_mode_refuses(tmp_path): + stores.make_store(tmp_path / "s.zarr") + with pytest.raises(NotImplementedError, match="Saving"): + ZarrDataset(str(tmp_path / "s.zarr"), mode="w") + + +def test_an_in_memory_store_is_just_another_location(tmp_path): + """A store need not be a directory. zarr-python decides what it is.""" + memory = zarr.storage.MemoryStore() + group = zarr.open_group(store=memory, mode="w", zarr_format=3) + stores.make_array(group, "v", (2,), "f8", 3, dimensions=("t",)) + + dataset = ZarrDataset(memory) + assert list(dataset.variables) == ["v"] + + +def test_a_missing_store_raises_zarrs_own_error(tmp_path): + with pytest.raises(FileNotFoundError): + ZarrDataset(str(tmp_path / "nothing.zarr")) + + +def test_a_directory_that_is_not_a_store_raises(tmp_path): + (tmp_path / "plain").mkdir() + with pytest.raises(zarr.errors.GroupNotFoundError): + ZarrDataset(str(tmp_path / "plain")) + + +def test_a_missing_group_raises(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + with pytest.raises(zarr.errors.GroupNotFoundError): + ZarrDataset(location, group="nope") From 2b496b3efaee4c1dd8813d341e653ff8b7743b47 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 16:06:26 +0000 Subject: [PATCH 20/56] Assert variables keep an omitted dimension name in their own dimensions The inconsistent-dimension test checked the warning and the mapping's omission but not that the two arrays still report the name themselves, which is the fact that lets a later (name, size) match tell them apart. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- .../unit/fileformats/zarr/dataset/test_ZarrDataset.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py index 601b0ea6f1..a1294379ad 100644 --- a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py +++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py @@ -48,10 +48,15 @@ def test_a_name_used_at_two_sizes_is_omitted_and_warns(tmp_path): stores.make_array(group, "short", (2,), "f8", 3, dimensions=("t",)) stores.make_array(group, "long", (5,), "f8", 3, dimensions=("t",)) + dataset = ZarrDataset(str(tmp_path / "c.zarr")) with pytest.warns(IrisCfLoadWarning, match="t"): - dimensions = dict(ZarrDataset(str(tmp_path / "c.zarr")).dimensions) + dimensions = dict(dataset.dimensions) assert "t" not in dimensions + # The arrays keep the name in their own dimensions: nothing is renamed or + # dropped, only the dataset-wide mapping declines to link them. + assert dataset.variables["short"].dimensions == ("t",) + assert dataset.variables["long"].dimensions == ("t",) def test_anonymous_dimensions_are_not_in_the_mapping(tmp_path): From d07f0249deb258e7d4c38085eaf8f980458e2acb Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 16:18:10 +0000 Subject: [PATCH 21/56] Implement the CF decoding Zarr does not do for itself Masking, valid-range masking and scale/offset unpacking, as pure functions of values. The two rules worth reading twice: a floating _FillValue is base64 on version 3, which is xarray's convention rather than a defect; and Array.fill_value is a mask source on version 2 only, and only when the variable declares no CF attribute of its own. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/zarr/_decode.py | 287 ++++++++++++++++++ .../unit/fileformats/zarr/test__decode.py | 197 ++++++++++++ 2 files changed, 484 insertions(+) create mode 100644 lib/iris/fileformats/zarr/_decode.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/test__decode.py diff --git a/lib/iris/fileformats/zarr/_decode.py b/lib/iris/fileformats/zarr/_decode.py new file mode 100644 index 0000000000..1ab18fde1b --- /dev/null +++ b/lib/iris/fileformats/zarr/_decode.py @@ -0,0 +1,287 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The CF decoding netCDF4 does for free and Zarr does not. + +netCDF4 applies ``_FillValue``/``missing_value`` masking, +``valid_min``/``valid_max``/``valid_range`` masking and +``scale_factor``/``add_offset`` unpacking on every read. Zarr returns a plain +``numpy.ndarray`` and applies none of it, so these rules are implemented here. +They live on the Zarr side rather than in the shared CF layer because the +netCDF path gets the behaviour from the C library and has nothing to reuse. + +Everything here is a pure function of values, with no store and no Dask in +sight. :func:`mask_and_scale` is what the loader lays over a raw array as a +``map_blocks`` layer - visible in the graph, fusable with its neighbours, and +testable on a plain ndarray. + +Two traps, both from the Zarr specification rather than from CF. + +**A floating ``_FillValue`` is base64 on version 3.** Released xarray encodes +it as base64 of a little-endian ``float64`` for *every* floating dtype, +including ``float32``, and its reader requires that form. It is the de-facto +encoding, not a producer's mistake, so Iris reads both it and a plain JSON +number. CF section 2.5.1 asks for "the scalar attribute with the name +``_FillValue`` and of the same type as its variable"; this is a deliberate, +narrow deviation, taken because interoperability with the other Zarr readers +is the reason for native Zarr support. + +**``Array.fill_value`` is not a missing-value declaration.** It says what +unwritten storage reads back as, and zarr-python defaults it to zero. On +version 3 it is never a mask source: an ``int16`` array of ``[0, 1, 2]`` with +no CF attributes would otherwise load with its valid zero masked. On version 2 +it is one, but only when no CF attribute is present, because version 2 stores +predate the convention - which is what released xarray's +``use_zarr_fill_value_as_mask`` does. + +See section 4.4 of ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. + +""" + +import base64 +import binascii +import struct +from typing import Any +import warnings + +import numpy as np + +import iris.warnings + +#: The width of the base64 `_FillValue` encoding: one little-endian float64. +_PACKED_DOUBLE = " Any: + """Return a CF missing-value sentinel as a value of `dtype`. + + Parameters + ---------- + value : Any + The attribute's value, as the store holds it: a JSON number, or - for + a floating variable - the base64 of a little-endian ``float64``. + dtype : numpy.dtype + The variable's own dtype. The result is narrowed to it, because a + sentinel of a different width never equals anything in the array. + variable_name : str + Named in the warning, so the user can find the variable. + + Returns + ------- + numpy.generic or None + ``None`` when `value` is neither form, which is a producer error. + + Warns + ----- + IrisCfLoadWarning + When `value` cannot be read as a sentinel of `dtype`. + + """ + if isinstance(value, (bool, np.bool_)): + # A JSON true/false is a number to Python and nothing to CF. + decoded = None + elif isinstance(value, (int, float, np.number)): + decoded = value + elif isinstance(value, str) and dtype.kind == "f": + decoded = _decode_packed_double(value) + else: + decoded = None + + if decoded is None: + message = ( + "Ignoring a missing-value attribute that is neither a number nor " + f"the base64 of a float64, on variable {variable_name}: {value!r}" + ) + warnings.warn(message, category=iris.warnings.IrisCfLoadWarning) + return None + + return dtype.type(decoded) + + +def _decode_packed_double(value: str) -> float | None: + """Return the float in a base64 little-endian float64, or None.""" + try: + raw = base64.b64decode(value, validate=True) + except (binascii.Error, ValueError): + return None + if len(raw) != _PACKED_DOUBLE_BYTES: + return None + return struct.unpack(_PACKED_DOUBLE, raw)[0] + + +def decode_parameters( + attributes, + *, + dtype: np.dtype, + storage_fill_value: Any, + zarr_format: int, + variable_name: str, +) -> dict: + """Return the CF decode parameters for one variable. + + Parameters + ---------- + attributes : mapping + The variable's CF attributes. + dtype : numpy.dtype + The variable's stored dtype, before any unpacking. + storage_fill_value : Any + ``zarr.Array.fill_value`` - what unwritten storage reads back as. A + mask source only on version 2, and only when the variable declares no + CF missing-value attribute. + zarr_format : int + 2 or 3. The one place the version changes what is decoded. + variable_name : str + Named in any warning. + + Returns + ------- + dict + The keyword arguments :func:`mask_and_scale` takes, always with the + same keys. :func:`decoding_required` says whether they do anything. + + """ + sentinels = [] + for name in ("_FillValue", "missing_value"): + if name in attributes: + decoded = decode_fill_value(attributes[name], dtype, variable_name) + if decoded is not None: + sentinels.append(decoded) + + if ( + not sentinels + and zarr_format < _FILL_VALUE_IS_A_MASK_BEFORE + and storage_fill_value is not None + ): + sentinels.append(storage_fill_value) + + valid_min = attributes.get("valid_min") + valid_max = attributes.get("valid_max") + valid_range = attributes.get("valid_range") + if valid_range is not None and len(valid_range) == 2: + valid_min, valid_max = valid_range + + scale_factor = attributes.get("scale_factor") + add_offset = attributes.get("add_offset") + + return { + "sentinels": tuple(sentinels), + "valid_min": valid_min, + "valid_max": valid_max, + "scale_factor": scale_factor, + "add_offset": add_offset, + "dtype": _unpacked_dtype(dtype, scale_factor, add_offset), + } + + +def _unpacked_dtype(dtype, scale_factor, add_offset) -> np.dtype: + """Return the dtype the unpacked values will have. + + The same calculation as + :func:`iris.fileformats.cf.loader._get_actual_dtype`, done here on plain + values so that the dtype the Dask graph declares and the dtype the decode + layer produces are one statement rather than two. + """ + probe = np.zeros(1, dtype=dtype) + if scale_factor is not None: + probe = scale_factor * probe + if add_offset is not None: + probe = add_offset + probe + return probe.dtype + + +def decoding_required(parameters: dict) -> bool: + """Whether these parameters would change anything. + + A variable with nothing to decode gets no ``map_blocks`` layer and an + unmasked Dask meta. Adding a masked meta without a layer beneath it would + be a meta that lies about its own graph. + """ + if parameters["sentinels"]: + return True + return any( + parameters[name] is not None + for name in ("valid_min", "valid_max", "scale_factor", "add_offset") + ) + + +def mask_by_sentinels(block: np.ndarray, sentinels: tuple) -> np.ndarray: + """Return a boolean mask of the elements equal to any sentinel. + + A NaN sentinel is matched with :func:`numpy.isnan`, because NaN does not + equal itself and NaN is what nearly every version 3 store's base64 + ``_FillValue`` decodes to. + """ + mask = np.zeros(block.shape, dtype=bool) + for sentinel in sentinels: + if isinstance(sentinel, (float, np.floating)) and np.isnan(sentinel): + mask |= np.isnan(block) + else: + mask |= block == sentinel + return mask + + +def mask_by_valid_range(block: np.ndarray, valid_min, valid_max) -> np.ndarray: + """Return a boolean mask of the elements outside the valid range.""" + mask = np.zeros(block.shape, dtype=bool) + if valid_min is not None: + mask |= block < valid_min + if valid_max is not None: + mask |= block > valid_max + return mask + + +def unpack(block: np.ndarray, scale_factor, add_offset, dtype: np.dtype) -> np.ndarray: + """Return `block` unpacked by CF's scale and offset, as `dtype`.""" + result = block + if scale_factor is not None: + result = scale_factor * result + if add_offset is not None: + result = add_offset + result + return result.astype(dtype, copy=False) + + +def mask_and_scale( + block: np.ndarray, + *, + sentinels: tuple, + valid_min, + valid_max, + scale_factor, + add_offset, + dtype: np.dtype, +) -> np.ma.MaskedArray: + """Apply the CF decode rules to one block, in CF's order. + + Masking first, on the **packed** values, because CF declares a sentinel to + be "of the same type as its variable"; unpacking second. The result is + always a masked array, so that every block matches the Dask meta the + loader declares for the layer. + + Parameters + ---------- + block : numpy.ndarray + One Dask block of the raw stored values. + sentinels : tuple + Values meaning "missing", already narrowed to the stored dtype. + valid_min, valid_max : optional + CF's valid range. Values outside it are masked. + scale_factor, add_offset : optional + CF's packing. ``value * scale_factor + add_offset``. + dtype : numpy.dtype + The dtype of the result, which the Dask graph has already declared. + + Returns + ------- + numpy.ma.MaskedArray + + """ + mask = mask_by_sentinels(block, sentinels) + mask |= mask_by_valid_range(block, valid_min, valid_max) + return np.ma.masked_array(unpack(block, scale_factor, add_offset, dtype), mask=mask) diff --git a/lib/iris/tests/unit/fileformats/zarr/test__decode.py b/lib/iris/tests/unit/fileformats/zarr/test__decode.py new file mode 100644 index 0000000000..8d559cd022 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/test__decode.py @@ -0,0 +1,197 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :mod:`iris.fileformats.zarr._decode`.""" + +import base64 +import struct + +import numpy as np +import pytest + +from iris.fileformats.zarr import _decode +from iris.warnings import IrisCfLoadWarning + + +def base64_double(value): + """Return `value` in the encoding released xarray writes for a float.""" + return base64.b64encode(struct.pack(" Date: Tue, 13 Oct 2026 16:27:35 +0000 Subject: [PATCH 22/56] Stop an out-of-range missing-value attribute crashing the load decode_fill_value narrowed a decoded sentinel to the variable's dtype unconditionally, so a value that overflowed it - signed, negative or unsigned - raised OverflowError out of the whole load instead of warning and degrading, same as any other undecodable sentinel. Also adds the end-to-end test for the rule the module exists for: a version 3 array of zeros with no CF attributes comes back through decode_parameters and mask_and_scale with its zeros intact, not just with an empty sentinels tuple. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/zarr/_decode.py | 17 ++++++++-- .../unit/fileformats/zarr/test__decode.py | 33 +++++++++++++++++++ 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/lib/iris/fileformats/zarr/_decode.py b/lib/iris/fileformats/zarr/_decode.py index 1ab18fde1b..d368aa5033 100644 --- a/lib/iris/fileformats/zarr/_decode.py +++ b/lib/iris/fileformats/zarr/_decode.py @@ -75,7 +75,8 @@ def decode_fill_value(value: Any, dtype: np.dtype, variable_name: str) -> Any: Returns ------- numpy.generic or None - ``None`` when `value` is neither form, which is a producer error. + ``None`` when `value` is neither form, or does not fit `dtype` - + both are producer errors. Warns ----- @@ -101,7 +102,19 @@ def decode_fill_value(value: Any, dtype: np.dtype, variable_name: str) -> Any: warnings.warn(message, category=iris.warnings.IrisCfLoadWarning) return None - return dtype.type(decoded) + try: + return dtype.type(decoded) + except (OverflowError, ValueError): + # Out of range for dtype: a producer error, not a reason to fail + # the whole load. A float dtype never raises here - an overflowing + # float narrows to +/-inf instead, which is ordinary numpy + # behaviour and not something to guard against. + message = ( + f"Ignoring a missing-value attribute that does not fit {dtype}, " + f"on variable {variable_name}: {value!r}" + ) + warnings.warn(message, category=iris.warnings.IrisCfLoadWarning) + return None def _decode_packed_double(value: str) -> float | None: diff --git a/lib/iris/tests/unit/fileformats/zarr/test__decode.py b/lib/iris/tests/unit/fileformats/zarr/test__decode.py index 8d559cd022..99e11d809d 100644 --- a/lib/iris/tests/unit/fileformats/zarr/test__decode.py +++ b/lib/iris/tests/unit/fileformats/zarr/test__decode.py @@ -68,6 +68,25 @@ def test_a_base64_string_on_an_integer_variable_warns(): assert result is None +def test_an_out_of_range_integer_warns_and_is_ignored(): + # 70000 does not fit an int16, however a producer came to write it. + with pytest.warns(IrisCfLoadWarning, match="v"): + result = _decode.decode_fill_value(70000, np.dtype("i2"), "v") + assert result is None + + +def test_a_negative_out_of_range_integer_warns_and_is_ignored(): + with pytest.warns(IrisCfLoadWarning, match="v"): + result = _decode.decode_fill_value(-70000, np.dtype("i2"), "v") + assert result is None + + +def test_an_out_of_range_unsigned_integer_warns_and_is_ignored(): + with pytest.warns(IrisCfLoadWarning, match="v"): + result = _decode.decode_fill_value(300, np.dtype("u1"), "v") + assert result is None + + def parameters(attributes, *, dtype="i2", storage_fill_value=0, zarr_format=3): return _decode.decode_parameters( attributes, @@ -87,6 +106,20 @@ def test_version_3_ignores_the_storage_fill_value(): assert parameters({}, storage_fill_value=0)["sentinels"] == () +def test_version_3_zeros_survive_masking_and_scaling_unchanged(): + """The property the rule above exists for, demonstrated end to end. + + An int16 array of zeros, no CF attributes, storage fill value zero: + pushed through the actual masking function with the actual parameters + `decode_parameters` produces, not hand-written keyword arguments. + """ + params = parameters({}, storage_fill_value=0, zarr_format=3) + block = np.array([0, 0, 1], dtype="i2") + result = _decode.mask_and_scale(block, **params) + assert not result.mask.any() + assert result.tolist() == [0, 0, 1] + + def test_version_2_uses_the_storage_fill_value_when_there_is_no_attribute(): result = parameters({}, storage_fill_value=0, zarr_format=2) assert result["sentinels"] == (0,) From 790f7a94996cc3eff7d1da48229faa7a65c417f4 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 16:43:16 +0000 Subject: [PATCH 23/56] Read Zarr data lazily, aligned to the store's own chunks No proxy: a zarr.Array is picklable and a store is a namespace, so the array goes to as_lazy_data directly and CF decoding is a visible map_blocks layer. Chunking is rounded up to whole multiples of the store's own, never below, because reading any part of a Zarr chunk fetches the whole object. The Dask cache key carries the array's metadata document rather than its address, which alone goes stale when a store is rewritten in place. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/zarr/_dataset.py | 223 +++++++++++- .../zarr/dataset/test__align_chunks.py | 89 +++++ .../zarr/dataset/test_read_data.py | 339 ++++++++++++++++++ 3 files changed, 646 insertions(+), 5 deletions(-) create mode 100644 lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py diff --git a/lib/iris/fileformats/zarr/_dataset.py b/lib/iris/fileformats/zarr/_dataset.py index f230bca89f..2466482d15 100644 --- a/lib/iris/fileformats/zarr/_dataset.py +++ b/lib/iris/fileformats/zarr/_dataset.py @@ -37,16 +37,25 @@ """ from collections.abc import Callable, Iterator, Mapping, MutableMapping +from functools import partial +import json from types import MappingProxyType from typing import Any import warnings +import dask +import dask.array as da +import dask.config +import dask.utils import numpy as np import zarr +from iris._lazy_data import _optimum_chunksize, as_lazy_data from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable import iris.warnings +from . import _decode + #: What :attr:`CFDataset.format` answers, by the store's Zarr version. _FORMAT_BY_ZARR_VERSION = {2: "ZARR_V2", 3: "ZARR_V3"} @@ -149,6 +158,103 @@ def _assemble_dimensions(variables) -> dict: return sizes +def _fixed_flags(dims_fixed, ndim: int) -> tuple: + """Return one "do not change this" flag per dimension. + + `iris.fileformats.cf.loader._chunks_from_chunk_control` answers ``(None,)`` + when nothing is fixed, whatever the rank, so a length that does not match + means nothing is fixed. + """ + if dims_fixed is None or len(dims_fixed) != ndim: + return (False,) * ndim + return tuple(bool(flag) for flag in dims_fixed) + + +def _align_chunks( + chunks: tuple, + store_chunks: tuple | None, + shape: tuple, + dtype: np.dtype, + dims_fixed, + name: str, +) -> tuple: + """Round `chunks` up to whole multiples of the store's own chunk shape. + + A Zarr chunk is the atomic unit of storage: reading any part of one + fetches and decompresses the whole object. So a Dask chunking that + subdivides the store's costs a re-read per subdivision - and over a remote + store, one HTTP GET per subdivision for the same object. Aggregating + upward costs nothing, because peak memory per task is a whole decompressed + store chunk either way. + + Parameters + ---------- + chunks : tuple of int + The chunking computed for this variable, already through + :func:`iris._lazy_data._optimum_chunksize`. + store_chunks : tuple of int, optional + The store's own chunk shape. ``None`` for an unchunked store, which + constrains nothing. + shape : tuple of int + The variable's full shape. A result is never larger than this. + dtype : numpy.dtype + The stored dtype, for deciding whether one store chunk already exceeds + Dask's target chunk size. + dims_fixed : sequence of bool, optional + Dimensions an explicit ``CHUNK_CONTROL`` setting pinned. These are + left exactly as given: the keyword is an instruction, not a + suggestion. + name : str + The variable's name, for the warning. + + Returns + ------- + tuple of int + + Warns + ----- + IrisLoadWarning + When the result cannot tile the store grid without subdividing it - + either because one store chunk already exceeds Dask's target size, so + the store's layout is the binding constraint, or because an explicit + setting subdivides it. + + """ + if store_chunks is None: + return tuple(chunks) + + fixed = _fixed_flags(dims_fixed, len(shape)) + aligned = [] + subdivided = False + for size, store_size, extent, is_fixed in zip(chunks, store_chunks, shape, fixed): + if is_fixed: + aligned.append(size) + subdivided |= size < store_size + continue + whole = -(-size // store_size) * store_size + aligned.append(min(whole, extent)) + subdivided |= whole > extent and extent % store_size != 0 + + if _store_chunk_exceeds_dask_target(store_chunks, dtype): + subdivided = True + + if subdivided: + message = ( + "A Zarr chunk is read whole, so this store's chunking is the " + "binding constraint on memory per task rather than Dask's target " + f"chunk size: {name}" + ) + warnings.warn(message, category=iris.warnings.IrisLoadWarning) + + return tuple(aligned) + + +def _store_chunk_exceeds_dask_target(store_chunks: tuple, dtype: np.dtype) -> bool: + """Whether one whole store chunk is larger than Dask's target chunk size.""" + target = dask.utils.parse_bytes(dask.config.get("array.chunk-size")) + return int(np.prod(store_chunks)) * dtype.itemsize > target + + class _ZarrAttributes(MutableMapping): """A Zarr node's attributes as a mapping, less the ones that are storage. @@ -301,13 +407,99 @@ def __setitem__(self, keys, values) -> None: self._array[keys] = values def read_data(self, chunking_policy: Callable[[], tuple]) -> Any: - """Not yet implemented. + """Return this array's data, lazily where a lazy array is worth it. - A later change adds this, following §4.4 of - ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. + Parameters + ---------- + chunking_policy : callable + Called with no arguments, returning the ``(chunks, dims_fixed)`` + pair to build a lazy array with. Called only once the result is + known to be lazy. + + Returns + ------- + numpy.ndarray, numpy.ma.MaskedArray or dask.array.Array + Real data for a small array, lazy data otherwise. Masked whenever + the variable declares anything to decode. + + Notes + ----- + There is no data proxy. ``zarr.Array`` is picklable, a chunk is an + independent object in a key-value store so concurrent reads need no + lock, and a store is a namespace rather than an open file handle. CF + decoding is a ``map_blocks`` layer over the raw array rather than + something hidden inside ``__getitem__``, so it is visible in the + graph and testable without a store. """ - raise NotImplementedError + # Deferred, and the module rather than its members: the CF package + # reaches this one through iris.fileformats.zarr.loader, so a + # module-level import would be a cycle; and the tests patch + # _LAZYVAR_MIN_BYTES on the module object. + from iris.fileformats.cf import loader + + parameters = _decode.decode_parameters( + self._attributes, + dtype=self.dtype, + storage_fill_value=self.fill_value, + zarr_format=self.zarr_format, + variable_name=self.name, + ) + decoding = _decode.decoding_required(parameters) + + if self.size * self.dtype.itemsize < loader._LAZYVAR_MIN_BYTES: + # A lazy array would cost more memory AND more time to access. + block = np.asanyarray(self._array[...]) + if decoding: + return _decode.mask_and_scale(block, **parameters) + return block + + chunks, dims_fixed = chunking_policy() + raw_meta = np.empty((0,) * self.ndim, dtype=self.dtype) + + if chunks == "auto": + # CHUNK_CONTROL.as_dask(): the user asked for Dask's own policy, + # so the store's layout is deliberately not consulted. + lazy = as_lazy_data( + self._array, chunks="auto", meta=raw_meta, cache_key=self.cache_key + ) + else: + if chunks is None: + # What `as_lazy_data` makes of None, done here because this + # path reaches `_optimum_chunksize` first and that does not + # accept it: one chunk spanning the array, divided down below. + chunks = self.shape + chunks = _optimum_chunksize( + chunks, shape=self.shape, dtype=self.dtype, dims_fixed=dims_fixed + ) + chunks = _align_chunks( + chunks, self.chunking, self.shape, self.dtype, dims_fixed, self.name + ) + # Fixed in every dimension, because the alignment above is the + # final word: as_lazy_data would otherwise re-run the + # optimisation and undo it. + lazy = as_lazy_data( + self._array, + chunks=chunks, + meta=raw_meta, + dims_fixed=(True,) * self.ndim, + cache_key=self.cache_key, + ) + + if not decoding: + return lazy + + # `partial` rather than `**parameters` on the `map_blocks` call: + # `dtype` is a keyword `map_blocks` consumes for itself rather than + # forwarding, and `mask_and_scale` needs it too. Binding it once here + # also makes the dtype Dask declares and the dtype the layer produces + # one value rather than two statements that could disagree. + return da.map_blocks( + partial(_decode.mask_and_scale, **parameters), + lazy, + dtype=parameters["dtype"], + meta=np.ma.masked_array(np.empty((0,) * self.ndim, parameters["dtype"])), + ) def write_handle(self) -> "ZarrDatasetVariable": """Return self: a Zarr write target needs no proxy. @@ -330,6 +522,28 @@ def array(self): """The backing :class:`zarr.Array`. Nothing generic may read this.""" return self._array + @property + def zarr_format(self) -> int: + """2 or 3. The one Zarr fact that changes how a value is read.""" + return self._array.metadata.zarr_format + + @property + def cache_key(self) -> str: + """An identity for this array that goes stale when the array does. + + `iris._lazy_data.CACHE` is process-wide, so an address alone is not + enough: a store rewritten in place keeps its address and changes its + contents. The metadata document covers shape, chunks, shards, dtype, + fill value, codecs, dimension names and attributes in one value, and + is treated as opaque - listing the parts Iris cares about would go + stale, silently, the first time zarr-python added a field that + mattered. + """ + return json.dumps( + [self._location, self._array.path, self._array.metadata.to_dict()], + sort_keys=True, + ) + #: What `mode` a dataset may be opened in. Writing is not implemented. _READ_MODES = ("r",) @@ -382,7 +596,6 @@ def __init__(self, location, mode: str = "r", *, group: str | None = None): raise NotImplementedError(f"{_NO_SAVING} Cannot open with mode={mode!r}.") self._mode = mode - self._group_path = group target = str(location) self._location = target if group is None else f"{target.rstrip('/')}/{group}" diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py new file mode 100644 index 0000000000..a51d9d602b --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py @@ -0,0 +1,89 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for the Zarr read-side chunk alignment. + +No store is built here: the guard is a function of chunk shapes, which is +most of the point of having it as one. +""" + +import warnings + +import numpy as np +import pytest + +from iris.fileformats.zarr._dataset import _align_chunks +from iris.warnings import IrisLoadWarning + +F8 = np.dtype("f8") + + +def test_a_whole_multiple_is_left_alone(): + assert _align_chunks((4096, 4096), (512, 512), (4096, 4096), F8, None, "v") == ( + 4096, + 4096, + ) + + +def test_a_sub_multiple_is_rounded_up(): + """A dask chunk that subdivides a store chunk costs N re-reads. + + A read of any part of a Zarr chunk fetches and decompresses the whole + object, so N dask blocks over one store chunk is N times the bytes and N + times the CPU - and, over a remote store, N HTTP GETs for one object. + """ + with pytest.warns(IrisLoadWarning, match="v"): + result = _align_chunks( + (2000, 8000), (8000, 8000), (16000, 16000), F8, None, "v" + ) + assert result == (8000, 8000) + + +def test_rounding_up_is_capped_at_the_shape(): + assert _align_chunks((2000, 6000), (6000, 6000), (6000, 6000), F8, None, "v") == ( + 6000, + 6000, + ) + + +def test_an_unchunked_store_is_left_alone(): + assert _align_chunks((10,), None, (100,), F8, None, "v") == (10,) + + +def test_an_explicit_chunk_control_setting_wins_but_warns(): + """A deliberate instruction is honoured, on the same terms. + + CHUNK_CONTROL marks the dimensions it set as fixed. Overriding them would + make the keyword a suggestion. + """ + with pytest.warns(IrisLoadWarning, match="v"): + result = _align_chunks( + (100, 8000), (8000, 8000), (16000, 16000), F8, (True, False), "v" + ) + assert result[0] == 100 + + +def test_a_small_store_chunk_does_not_warn(): + # Dask's target is the binding constraint here, which is the ordinary + # case and nothing for the user to act on. + with warnings.catch_warnings(): + warnings.simplefilter("error") + result = _align_chunks((1024, 1024), (512, 512), (4096, 4096), F8, None, "v") + assert result == (1024, 1024) + + +@pytest.mark.parametrize( + ("chunks", "store_chunks", "shape"), + [ + ((2000, 8000), (8000, 8000), (16000, 16000)), + ((1024, 1024), (512, 512), (4096, 4096)), + ((3000, 10000), (1000, 1000), (10000, 10000)), + ], +) +def test_the_result_tiles_the_store_grid_exactly(chunks, store_chunks, shape): + with warnings.catch_warnings(): + warnings.simplefilter("ignore", IrisLoadWarning) + result = _align_chunks(chunks, store_chunks, shape, F8, None, "v") + for size, store_size, extent in zip(result, store_chunks, shape): + assert size % store_size == 0 or size == extent diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py new file mode 100644 index 0000000000..d2482a50f4 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py @@ -0,0 +1,339 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :meth:`ZarrDatasetVariable.read_data`.""" + +import dask +import dask.array as da +import dask.config +import numpy as np +import pytest +import zarr + +from iris.fileformats.cf import loader as cf_loader +from iris.fileformats.zarr._dataset import ZarrDataset +from iris.warnings import IrisLoadWarning + +from .. import stores + + +def variable(location, name="air_temperature"): + return ZarrDataset(location).variables[name] + + +def policy(): + """The chunking policy a loader outside any CHUNK_CONTROL supplies.""" + return None, None + + +def test_a_small_variable_is_read_whole(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + result = variable(location).read_data(policy) + assert not isinstance(result, da.Array) + np.testing.assert_array_equal(np.asarray(result), stores.DATA) + + +def test_a_small_variable_never_consults_the_policy(tmp_path): + """Under CHUNK_CONTROL.from_file() the policy raises for an unchunked store. + + A variable small enough to read whole must reach its data without asking, + or `from_file` starts rejecting small cubes that load today. + """ + + def raising_policy(): + raise AssertionError("chunking policy consulted for a small variable") + + location = stores.make_store(tmp_path / "s.zarr") + variable(location).read_data(raising_policy) + + +def test_a_large_variable_is_lazy(tmp_path, monkeypatch): + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + location = stores.make_store(tmp_path / "s.zarr") + result = variable(location).read_data(policy) + assert isinstance(result, da.Array) + np.testing.assert_array_equal(result.compute(), stores.DATA) + + +def test_a_plain_array_gets_an_unmasked_meta(tmp_path, monkeypatch): + # No decode layer, so no masked meta: a masked meta with nothing beneath + # it is a meta that lies about its own graph. + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + location = stores.make_store(tmp_path / "s.zarr") + result = variable(location).read_data(policy) + assert not isinstance(result._meta, np.ma.MaskedArray) + + +def test_a_decoded_array_gets_a_masked_meta(tmp_path, monkeypatch): + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + group = zarr.open_group(store=tmp_path / "m.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "v", + (4,), + "i2", + 3, + dimensions=("t",), + attributes={"_FillValue": -999}, + ) + array[:] = [-999, 1, 2, 3] + + result = ZarrDataset(str(tmp_path / "m.zarr")).variables["v"].read_data(policy) + assert isinstance(result._meta, np.ma.MaskedArray) + realised = result.compute() + assert realised.mask.tolist() == [True, False, False, False] + + +def key_of(location, name="v"): + return ZarrDataset(location).variables[name].cache_key + + +def test_the_key_is_stable_across_a_reopen(tmp_path): + group = zarr.open_group(store=tmp_path / "k.zarr", mode="w", zarr_format=3) + stores.make_array(group, "v", (4,), "i4", 3, dimensions=("t",)) + location = str(tmp_path / "k.zarr") + assert key_of(location) == key_of(location) + + +def test_the_key_changes_when_the_shape_does(tmp_path): + """A store rewritten in place keeps its address and changes its contents. + + Keyed on the address alone, an array reopened after its shape changed + from (4,) to (6,) returns the cached four values, silently truncating - + and that is weaker than the netCDF key it replaces, since + repr(NetCDFDataProxy) already carries shape and dtype. + """ + location = str(tmp_path / "k.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + stores.make_array(group, "v", (4,), "i4", 3, dimensions=("t",), chunks=(2,)) + before = key_of(location) + + group = zarr.open_group(store=location, mode="a", zarr_format=3) + group.create_array( + "v", + shape=(6,), + dtype="i4", + chunks=(2,), + dimension_names=("t",), + overwrite=True, + ) + assert key_of(location) != before + + +def test_the_key_changes_when_the_fill_value_does(tmp_path): + location = str(tmp_path / "k.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + group.create_array("v", shape=(4,), dtype="i4", fill_value=-1) + before = key_of(location) + + group = zarr.open_group(store=location, mode="a", zarr_format=3) + group.create_array("v", shape=(4,), dtype="i4", fill_value=-2, overwrite=True) + assert key_of(location) != before + + +@pytest.mark.parametrize( + "change", + [ + {"shape": (8,)}, + {"chunks": (4,)}, + {"shards": (8,)}, + {"dtype": "f8"}, + {"fill_value": -7}, + {"dimension_names": ("other",)}, + ], +) +def test_the_key_is_sensitive_to_every_metadata_field(tmp_path, change): + """`Array.metadata.to_dict()` is zarr-python's rendering, not a guarantee. + + Nothing in the Zarr specification defines what that method returns, so + these are what fail if a release stops carrying a field the key needs. + """ + base = dict( + shape=(4,), + dtype="i4", + chunks=(2,), + fill_value=-1, + dimension_names=("t",), + shards=None, + ) + location = str(tmp_path / "k.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + group.create_array("v", **base) + before = key_of(location) + + group = zarr.open_group(store=location, mode="a", zarr_format=3) + group.create_array("v", **{**base, **change}, overwrite=True) + assert key_of(location) != before + + +def test_the_key_is_sensitive_to_attributes(tmp_path): + location = str(tmp_path / "k.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + array = group.create_array("v", shape=(4,), dtype="i4") + before = key_of(location) + array.attrs["units"] = "K" + assert key_of(location) != before + + +def test_two_cubes_sharing_an_array_share_one_dask_array(tmp_path, monkeypatch): + # What the cache is for: many cubes in one store sharing a coordinate. + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + location = stores.make_store(tmp_path / "s.zarr") + first = variable(location).read_data(policy) + second = variable(location).read_data(policy) + assert first.name == second.name + + +class CountingStore(zarr.storage.WrapperStore): + """Wrap a store, adding every byte it returns to a caller-owned tally. + + zarr-python copies a store when it opens it read-only, through + `with_read_only` and `_with_store`, so a tally kept on the instance is + never the one the live store writes to, and the test passes vacuously. + """ + + def __init__(self, store, tally): + super().__init__(store=store) + self.tally = tally + + def _with_store(self, store): + return type(self)(store, self.tally) + + async def get(self, key, prototype, byte_range=None): + buffer = await self._store.get(key, prototype, byte_range) + if buffer is not None: + self.tally.append(len(buffer)) + return buffer + + async def get_partial_values(self, prototype, key_ranges): + buffers = await self._store.get_partial_values(prototype, key_ranges) + self.tally.extend(len(b) for b in buffers if b is not None) + return buffers + + +def bytes_to_read_first_chunk(location, dask_chunks): + """Return the bytes fetched reading the first eight values of "v".""" + tally = [] + store = CountingStore(zarr.storage.LocalStore(location), tally) + array = zarr.open_group(store=store, mode="r")["v"] + tally.clear() + lazy = da.from_array(array, chunks=dask_chunks, meta=np.empty((0,), "i4")) + lazy = da.map_blocks(lambda block: np.ma.masked_array(block), lazy, dtype="i4") + lazy[:8].compute() + return sum(tally) + + +def test_a_sharded_array_is_read_by_the_inner_chunk(tmp_path): + """Shard-sized Dask blocks fetch the whole shard, because of the layer. + + Bare Dask slicing can push a slice down into the array's own __getitem__, + but a map_blocks layer in between cannot, so the block is materialised in + full. The absolute byte counts are zarr-version-dependent; the comparison + is not. + """ + location = str(tmp_path / "sh.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + array = group.create_array( + "v", shape=(1024,), dtype="i4", chunks=(8,), shards=(1024,) + ) + array[:] = np.arange(1024, dtype="i4") + + inner = bytes_to_read_first_chunk(location, 8) + whole_shard = bytes_to_read_first_chunk(location, 1024) + assert inner < whole_shard + + # And the variable chooses the inner chunk for exactly this reason. + assert ZarrDataset(location).variables["v"].chunking == (8,) + + +def test_the_declared_dtype_matches_what_the_decode_layer_produces( + tmp_path, monkeypatch +): + """A packed variable's lazy dtype must equal its realised dtype. + + On the netCDF path netCDF4 does the arithmetic and reports the dtype. On + the Zarr path Iris does both, in two separately written statements, and + Dask does not check them against each other: a mismatch builds a graph + and surfaces later as a wrong dtype on the cube. + """ + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "b02", + (4,), + "u2", + 3, + dimensions=("p",), + attributes={"scale_factor": 0.0001, "add_offset": -0.1}, + ) + array[:] = [1000, 2000, 3000, 4000] + + lazy = ZarrDataset(str(tmp_path / "p.zarr")).variables["b02"].read_data(policy) + realised = lazy.compute() + + assert lazy.dtype == realised.dtype + assert lazy.dtype == np.dtype("f8") + np.testing.assert_allclose(realised, [0.0, 0.1, 0.2, 0.3]) + + +@pytest.mark.parametrize( + ("scale_factor", "expected"), + # A Zarr attribute is JSON, so a scale factor reaches Iris as a Python + # float or a Python int and never as a NumPy scalar - zarr-python refuses + # to write one. Both forms are here because they promote differently: + # a weak Python int leaves a uint16 variable uint16. + [(0.01, "f8"), (1, "u2")], +) +def test_the_declared_dtype_matches_the_cf_loader_calculation( + tmp_path, scale_factor, expected +): + """And it is the same answer `_get_actual_dtype` gives. + + The cube's data variable and its coordinates must agree about what a + packed value becomes, and they reach that answer through different code. + """ + group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3) + stores.make_array( + group, + "v", + (4,), + "u2", + 3, + attributes={"scale_factor": scale_factor}, + ) + variable = ZarrDataset(str(tmp_path / "p.zarr")).variables["v"] + from iris.fileformats.zarr import _decode + + parameters = _decode.decode_parameters( + variable.attributes, + dtype=variable.dtype, + storage_fill_value=variable.fill_value, + zarr_format=variable.zarr_format, + variable_name=variable.name, + ) + assert parameters["dtype"] == cf_loader._get_actual_dtype(variable) + assert parameters["dtype"] == np.dtype(expected) + + +def test_a_lazy_read_aligns_to_the_store_chunks(tmp_path, monkeypatch): + """The alignment reaches the array, not just the function that computes it. + + Dask's target is shrunk below one store chunk, so an unaligned result + would be the 8 elements the target asks for, subdividing the store's 16. + """ + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + location = str(tmp_path / "b.zarr") + group = zarr.open_group(store=location, mode="w", zarr_format=3) + array = group.create_array( + "v", shape=(64,), dtype="i4", chunks=(16,), dimension_names=("t",) + ) + array[:] = np.arange(64, dtype="i4") + + with dask.config.set({"array.chunk-size": "32B"}): + with pytest.warns(IrisLoadWarning, match="v"): + lazy = ZarrDataset(location).variables["v"].read_data(policy) + + assert lazy.chunks == ((16, 16, 16, 16),) + np.testing.assert_array_equal(lazy.compute(), np.arange(64)) From 41d77fe58acf2133669eea078f98bfbdf2256b19 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 17:00:03 +0000 Subject: [PATCH 24/56] Prove the Zarr read uses the cache, and stop a warning that cried wolf The cache test compared two Dask names, which Dask makes equal by deterministic tokenisation whether or not a cache key is passed, so it passed with the key removed from read_data entirely. It now replaces the process-wide cache and asserts the first read fills it and the second reuses it. Chunk alignment warned whenever rounding up overshot the array extent and the extent did not tile the store grid. Capping at the extent leaves one block spanning the dimension, which reads each store chunk exactly once, so nothing was subdivided and the warning was false - on an ordinary daily (365, 500, 1000) store, on every load. The two conditions that do mean something, an oversize store chunk and a subdivided fixed dimension, are kept and now have tests saying so. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/zarr/_dataset.py | 7 ++-- .../zarr/dataset/test__align_chunks.py | 38 +++++++++++++++++-- .../zarr/dataset/test_read_data.py | 20 +++++++++- 3 files changed, 57 insertions(+), 8 deletions(-) diff --git a/lib/iris/fileformats/zarr/_dataset.py b/lib/iris/fileformats/zarr/_dataset.py index 2466482d15..87a9d4d839 100644 --- a/lib/iris/fileformats/zarr/_dataset.py +++ b/lib/iris/fileformats/zarr/_dataset.py @@ -231,9 +231,10 @@ def _align_chunks( aligned.append(size) subdivided |= size < store_size continue - whole = -(-size // store_size) * store_size - aligned.append(min(whole, extent)) - subdivided |= whole > extent and extent % store_size != 0 + # Rounding up never subdivides, and capping at the extent leaves one + # block spanning the dimension, which cannot subdivide either - so a + # free dimension has nothing to report. + aligned.append(min(-(-size // store_size) * store_size, extent)) if _store_chunk_exceeds_dask_target(store_chunks, dtype): subdivided = True diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py index a51d9d602b..f178c60d3a 100644 --- a/lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py +++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py @@ -41,10 +41,13 @@ def test_a_sub_multiple_is_rounded_up(): def test_rounding_up_is_capped_at_the_shape(): - assert _align_chunks((2000, 6000), (6000, 6000), (6000, 6000), F8, None, "v") == ( - 6000, - 6000, - ) + # Store chunks small enough to stay under Dask's target, so that this + # case tests only the capping it is named for. Rounding 2000 up to a + # multiple of 3000 overshoots the 2500 extent, so the result is 2500. + with warnings.catch_warnings(): + warnings.simplefilter("error") + result = _align_chunks((2000, 2500), (3000, 2500), (2500, 2500), F8, None, "v") + assert result == (2500, 2500) def test_an_unchunked_store_is_left_alone(): @@ -64,6 +67,33 @@ def test_an_explicit_chunk_control_setting_wins_but_warns(): assert result[0] == 100 +def test_an_oversize_store_chunk_warns(): + """One of the two things worth telling the user about. + + A store chunk larger than Dask's target makes the store's layout the + binding constraint on memory per task, whatever chunking is asked for. + """ + with pytest.warns(IrisLoadWarning, match="v"): + _align_chunks((8000, 8000), (8000, 8000), (16000, 16000), F8, None, "v") + + +def test_an_ordinary_layout_does_not_warn(): + """And rounding up on its own is not worth telling the user about. + + A year of daily (500, 1000) fields chunked (1, 300, 1000) is an ordinary + store. Its middle dimension rounds 500 up to 600 and is capped back to + 500, which leaves one block spanning the dimension - a block that reads + each store chunk exactly once and so subdivides nothing. Warning here + would warn on every load of a store with nothing wrong with it. + """ + with warnings.catch_warnings(): + warnings.simplefilter("error") + result = _align_chunks( + (31, 500, 1000), (1, 300, 1000), (365, 500, 1000), F8, None, "v" + ) + assert result == (31, 500, 1000) + + def test_a_small_store_chunk_does_not_warn(): # Dask's target is the binding constraint here, which is the ordinary # case and nothing for the user to act on. diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py index d2482a50f4..9abb5e0086 100644 --- a/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py +++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py @@ -11,6 +11,7 @@ import pytest import zarr +from iris import _lazy_data from iris.fileformats.cf import loader as cf_loader from iris.fileformats.zarr._dataset import ZarrDataset from iris.warnings import IrisLoadWarning @@ -177,11 +178,28 @@ def test_the_key_is_sensitive_to_attributes(tmp_path): def test_two_cubes_sharing_an_array_share_one_dask_array(tmp_path, monkeypatch): - # What the cache is for: many cubes in one store sharing a coordinate. + """What the cache is for: many cubes in one store sharing a coordinate. + + Comparing the two Dask names proves nothing on its own: Dask tokenises a + zarr.Array deterministically, so two independently opened arrays get the + same name whether or not `read_data` passes a cache key at all. What has + to be shown is that the first read puts an entry in the process-wide + cache and the second read comes back out of it. + + The cache is replaced for the test rather than measured in place, because + it is process-wide and holds only its last hundred entries - a test that + counted the real one would depend on what ran before it. + """ monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + cache = _lazy_data.LRUCache(100) + monkeypatch.setattr(_lazy_data, "CACHE", cache) location = stores.make_store(tmp_path / "s.zarr") + first = variable(location).read_data(policy) + assert len(cache._cache) == 1, "the first read did not populate the cache" + second = variable(location).read_data(policy) + assert len(cache._cache) == 1, "the second read built a second array" assert first.name == second.name From 0b7656b6d2a235ea3faca6230d407910867cd283 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 17:06:37 +0000 Subject: [PATCH 25/56] Ask the shared CFDataset read contract of the Zarr store Both Zarr versions answer the same questions the netCDF implementation does, so a divergence between the two fails here rather than somewhere in the loader. Adds the check that no module in the cf package names an implementation class. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- .../cf/test_no_implementation_coupling.py | 48 +++++++++++++++++++ .../zarr/dataset/test_ZarrDataset_contract.py | 30 ++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py diff --git a/lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py b/lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py new file mode 100644 index 0000000000..65eaa1c7e9 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py @@ -0,0 +1,48 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The CF layer must not branch on which storage implementation it holds. + +`isinstance(thing, CFDataset)` tests the *interface*, which is how generic +code is supposed to accept any backend, and `CFReader` does exactly that. +What is banned is testing an *implementation*: that is what makes a third +backend a rewrite rather than a new module. Constructing one is a third +thing again, and legitimate - `CFReader` opens a `NetCDFDataset` for a bare +path, which is a default, not a branch on something it was handed. +""" + +from pathlib import Path + +import iris.fileformats.cf + +#: Implementation class names that may not appear in a type test. +IMPLEMENTATION_NAMES = ( + "NetCDFDataset", + "NetCDFDatasetVariable", + "ZarrDataset", + "ZarrDatasetVariable", +) + +#: The spellings of a type test. `__class__` catches the hand-rolled form. +TYPE_TESTS = ("isinstance", "issubclass", "__class__", "type(") + + +def cf_package_sources(): + """Yield `(path, text)` for every module in `iris.fileformats.cf`.""" + package = Path(iris.fileformats.cf.__file__).parent + for path in sorted(package.glob("*.py")): + yield path, path.read_text(encoding="utf-8") + + +def test_no_module_type_tests_an_implementation_class(): + offenders = [] + for path, text in cf_package_sources(): + for line_number, line in enumerate(text.splitlines(), start=1): + if line.lstrip().startswith("#"): + continue + names_one = any(name in line for name in IMPLEMENTATION_NAMES) + tests_type = any(spelling in line for spelling in TYPE_TESTS) + if names_one and tests_type: + offenders.append(f"{path.name}:{line_number}: {line.strip()}") + assert offenders == [] diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py new file mode 100644 index 0000000000..af8f96fcf9 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py @@ -0,0 +1,30 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The shared CFDataset read contract, asked of the Zarr implementation. + +Only the read half: the Zarr write path does not exist, and `ZarrDataset` +refuses every member of the write surface. The write half of the contract +arrives with the saver. +""" + +import pytest + +from iris.fileformats.zarr._dataset import ZarrDataset +from iris.tests.unit.fileformats.cf.dataset import contract + +from .. import stores + + +class TestZarrDatasetVersion3(contract.CFDatasetReadContract): + @pytest.fixture + def readable(self, tmp_path): + return ZarrDataset(stores.make_store(tmp_path / "contract.zarr")) + + +class TestZarrDatasetVersion2(contract.CFDatasetReadContract): + @pytest.fixture + def readable(self, tmp_path): + location = stores.make_store(tmp_path / "contract.zarr", zarr_format=2) + return ZarrDataset(location) From eddf08413d74795f77a77674052f1bf70c7e0122 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 17:37:11 +0000 Subject: [PATCH 26/56] Match CF dimensions on (name, size), not on the name alone Zarr places no restriction on reusing a dimension name across arrays, so a name alone can link two variables that mean different things by it. An unnamed dimension is anonymous and matches nothing, including another anonymous dimension. netCDF names are unique within a group, so this is a widening that no netCDF file can observe. Two pre-existing unit-test doubles modelled `dimensions` but not `shape`, which the old bare-name `spans()` never read and the new pair-based one does: `_NetCDFVarWithDimensions` (shared by the CFBoundaryVariable, CFClimatologyVariable and CFLabelVariable span tests) gains a `shape` property derived from a deterministic per-name size map, and the two `test_CFVariable.py` cases that compared against a bare `MagicMock` now compare against the same concrete stub on both sides - a MagicMock collaborator was auto-answering the new `_dimension_identities()` call with something that iterates as empty, silently flipping an assertion instead of failing loudly. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/cf/_variables.py | 61 +++++- .../unit/fileformats/cf/identify_mixins.py | 30 +++ .../unit/fileformats/cf/test_CFVariable.py | 38 ++-- .../tests/unit/fileformats/cf/test_spans.py | 183 ++++++++++++++++++ 4 files changed, 287 insertions(+), 25 deletions(-) create mode 100644 lib/iris/tests/unit/fileformats/cf/test_spans.py diff --git a/lib/iris/fileformats/cf/_variables.py b/lib/iris/fileformats/cf/_variables.py index 61dafa1786..e2c2898e5a 100644 --- a/lib/iris/fileformats/cf/_variables.py +++ b/lib/iris/fileformats/cf/_variables.py @@ -75,6 +75,25 @@ # NCZarr stores scalar variables as size-1 arrays on this pseudo-dimension. _NCZARR_SCALAR_DIMENSION = "_scalar_" + +class _AnonymousDimension: + """One dimension the store gives no name. + + A distinct instance per dimension, relying on object identity for + equality, so that an anonymous dimension matches nothing: not a named + dimension, not another anonymous one, and not itself as seen from a + second variable. There is nothing to match on, so claiming a match would + be invention. + + Zarr's version 3 core specification makes ``dimension_names`` optional and + allows a null entry, so a conforming store can present this and it must + not be an error. No netCDF variable ever has one. + """ + + def __repr__(self) -> str: + return "" + + # NetCDF variable attributes handled by the netCDF4 module and # therefore automatically classed as "used" attributes. _CF_ATTRS_IGNORE = set(["_FillValue", "add_offset", "missing_value", "scale_factor"]) @@ -195,6 +214,25 @@ def _is_scalar(self) -> bool: """ return not self.dimensions or self.dimensions == (_NCZARR_SCALAR_DIMENSION,) + def _dimension_identities(self) -> tuple: + """Return one hashable identity per dimension this variable spans. + + A named dimension's identity is the pair ``(name, size)``: two + variables share a dimension when they name it identically *and* it is + the same length in both. A name reused at two lengths links nothing, + which is the only reading that cannot produce a wrong cube. + + An unnamed dimension gets a fresh :class:`_AnonymousDimension`, which + equals nothing but itself. + + Relies on the :class:`~iris.fileformats.cf.dataset.CFDatasetVariable` + invariant that ``dimensions`` and ``shape`` have the same length. + """ + return tuple( + (name, size) if name is not None else _AnonymousDimension() + for name, size in zip(self.dimensions, self.shape) + ) + def spans(self, cf_variable): """Determine dimensionality coverage. @@ -218,8 +256,9 @@ def spans(self, cf_variable): if self._is_scalar(): return True - result = set(self.dimensions).issubset(cf_variable.dimensions) - return result + source = self._dimension_identities() + target = cf_variable._dimension_identities() + return set(source).issubset(target) def __eq__(self, other): # CF variable names are unique. @@ -235,7 +274,11 @@ def __hash__(self): @property def dimensions(self) -> tuple: - """The names of the dimensions this variable spans, in order.""" + """The names of the dimensions this variable spans, in order. + + An entry is ``None`` where the store names no dimension - only a + Zarr store does this; no netCDF variable has an anonymous dimension. + """ return tuple(self.cf_data.dimensions) @property @@ -527,8 +570,8 @@ def spans(self, cf_variable): # Scalar variables always span the target variable. result = True if not self._is_scalar(): - source = self.dimensions - target = cf_variable.dimensions + source = self._dimension_identities() + target = cf_variable._dimension_identities() # Ignore the bounds extent dimension. result = set(source[:-1]).issubset(target) or set(source[1:]).issubset( target @@ -603,8 +646,8 @@ def spans(self, cf_variable): # Scalar variables always span the target variable. result = True if not self._is_scalar(): - source = self.dimensions - target = cf_variable.dimensions + source = self._dimension_identities() + target = cf_variable._dimension_identities() # Ignore the climatology extent dimension. result = set(source[:-1]).issubset(target) or set(source[1:]).issubset( target @@ -926,8 +969,8 @@ def spans(self, cf_variable): # Scalar variables always span the target variable. result = True if not self._is_scalar(): - source = self.dimensions - target = cf_variable.dimensions + source = self._dimension_identities() + target = cf_variable._dimension_identities() # Ignore label string length dimension. result = set(source[:-1]).issubset(target) or set(source[1:]).issubset( target diff --git a/lib/iris/tests/unit/fileformats/cf/identify_mixins.py b/lib/iris/tests/unit/fileformats/cf/identify_mixins.py index d2c54f56bf..e23476e78a 100644 --- a/lib/iris/tests/unit/fileformats/cf/identify_mixins.py +++ b/lib/iris/tests/unit/fileformats/cf/identify_mixins.py @@ -41,6 +41,25 @@ def attributes(self): return {name: getattr(self, name) for name in self.ncattrs()} +#: The size each dimension name has stood for, the first time a stub used it. +#: `spans()` now checks `(name, size)` pairs, so two stubs that name a +#: dimension identically must also agree on its length, or a test exercising +#: that agreement would be testing nothing. Keyed by name and never cleared, +#: so the same name means the same size for every stub built in the process - +#: regardless of which test runs first. Pinned for +#: ``_NCZARR_SCALAR_DIMENSION``, matching the size NCZarr itself gives it; +#: every other name claims the next unused integer, once, the first time +#: :func:`_stub_dimension_size` is asked for it. +_STUB_DIMENSION_SIZES: dict[str, int] = {_NCZARR_SCALAR_DIMENSION: 1} + + +def _stub_dimension_size(name: str) -> int: + """Return `name`'s fixed size, claiming the next unused one if new.""" + if name not in _STUB_DIMENSION_SIZES: + _STUB_DIMENSION_SIZES[name] = len(_STUB_DIMENSION_SIZES) + 1 + return _STUB_DIMENSION_SIZES[name] + + class _NetCDFVarWithDimensions(_NetCDFVar): """Stub NetCDF variable with dimensions for spans() tests.""" @@ -50,6 +69,17 @@ def __init__(self, name, dimensions, dtype=int): super().__init__(name, dtype=dtype) self.dimensions = dimensions + @property + def shape(self) -> tuple: + """One size per dimension, agreeing with every other stub naming it. + + A property, not an instance attribute set in ``__init__``: `ncattrs` + reads `self.__dict__`, so a stored value would leak onto + `attributes` exactly as an unguarded `dimensions` would - a property + is never in `self.__dict__` and cannot leak that way. + """ + return tuple(_stub_dimension_size(name) for name in self.dimensions) + def assert_warning_gated(operation, warning_category, warning_regex): """Shared convenience for warning asserts.""" diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFVariable.py index 2309a7d4b3..d7e1bcab93 100644 --- a/lib/iris/tests/unit/fileformats/cf/test_CFVariable.py +++ b/lib/iris/tests/unit/fileformats/cf/test_CFVariable.py @@ -11,6 +11,8 @@ from iris.fileformats.cf import _variables from iris.fileformats.cf.dataset import TrackedAttributes +from .identify_mixins import _NetCDFVarWithDimensions + class CFVariableSub(cf.CFVariable): """A subclass of CFVariable for testing purposes.""" @@ -102,28 +104,32 @@ def test_scalar_dimension_always_true(self, mocker, nc_var): assert cf_var.spans(other) - def test_is_subset_check(self, mocker, nc_vars): - lhs_nc_var, other_nc_var, _ = nc_vars - lhs_nc_var.dimensions = ("time",) - lhs = CFVariableSub("lhs", lhs_nc_var) - - rhs = mocker.MagicMock() - rhs.dimensions = ("time", "lat") - + def test_is_subset_check(self): + # `spans()` now reads `shape` as well as `dimensions` on both sides of + # the comparison, so a bare `mocker.MagicMock()` collaborator will not + # do here: an unconfigured mock auto-answers `_dimension_identities()` + # with something that iterates as empty, which would make `spans()` + # trivially "true" instead of exercising the real comparison - see + # the Zarr I/O design doc, section 4.4, and this task's own history. + lhs = CFVariableSub("lhs", _NetCDFVarWithDimensions("lhs_var", ("time",))) + rhs = CFVariableSub("rhs", _NetCDFVarWithDimensions("rhs_var", ("time", "lat"))) assert lhs.spans(rhs) - other_nc_var.dimensions = ("height",) - other = CFVariableSub("other", other_nc_var) + other = CFVariableSub( + "other", _NetCDFVarWithDimensions("other_var", ("height",)) + ) assert not other.spans(rhs) - def test_scalar_dimension_with_others_is_not_scalar(self, mocker, nc_var): + def test_scalar_dimension_with_others_is_not_scalar(self): # _scalar_ means "this variable is scalar", which a variable with a # second dimension is not. It is not a wildcard. - nc_var.dimensions = (_variables._NCZARR_SCALAR_DIMENSION, "bnds") - cf_var = CFVariableSub("not_scalar", nc_var) - - other = mocker.MagicMock() - other.dimensions = ("time",) + cf_var = CFVariableSub( + "not_scalar", + _NetCDFVarWithDimensions( + "not_scalar_var", (_variables._NCZARR_SCALAR_DIMENSION, "bnds") + ), + ) + other = CFVariableSub("other", _NetCDFVarWithDimensions("other_var", ("time",))) assert not cf_var.spans(other) diff --git a/lib/iris/tests/unit/fileformats/cf/test_spans.py b/lib/iris/tests/unit/fileformats/cf/test_spans.py new file mode 100644 index 0000000000..2776b4d60d --- /dev/null +++ b/lib/iris/tests/unit/fileformats/cf/test_spans.py @@ -0,0 +1,183 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""`CFVariable.spans`, asked of both storage backends. + +Dimension matching is a pair, `(name, size)`, not a bare name - which is a +widening to admit what Zarr allows, not a change to what netCDF does. These +tests run against both so that the second half of that sentence is checked +rather than asserted. +""" + +import numpy as np +import pytest + +from iris.fileformats.cf import ( + CFBoundaryVariable, + CFDataVariable, + CFLabelVariable, +) + +zarr = pytest.importorskip("zarr") + +from iris.fileformats.netcdf._dataset import NetCDFDataset # noqa: E402 +from iris.fileformats.zarr._dataset import ZarrDataset # noqa: E402 + + +def build_netcdf(path, described): + """Write `described` as a netCDF file and return its open dataset.""" + dataset = NetCDFDataset(path, mode="w") + sizes = {} + for dimensions, shape in described.values(): + for name, size in zip(dimensions, shape): + sizes.setdefault(name, size) + for name, size in sizes.items(): + dataset.create_dimension(name, size) + for name, (dimensions, shape) in described.items(): + dataset.create_variable(name, np.dtype("f4"), dimensions) + return dataset + + +def build_zarr(path, described): + """Write `described` as a Zarr store and return its open dataset.""" + group = zarr.open_group(store=path, mode="w", zarr_format=3) + for name, (dimensions, shape) in described.items(): + group.create_array(name, shape=shape, dtype="f4", dimension_names=dimensions) + return ZarrDataset(str(path)) + + +@pytest.fixture(params=[build_netcdf, build_zarr], ids=["netcdf", "zarr"]) +def build(request, tmp_path): + """Return a callable giving `{name: CFDataVariable}` for a description.""" + suffix = ".nc" if request.param is build_netcdf else ".zarr" + + def make(described): + dataset = request.param(tmp_path / f"spans{suffix}", described) + return { + name: CFDataVariable(name, dataset.variables[name]) for name in described + } + + return make + + +def test_identical_dimensions_span(build): + variables = build({"data": (("y", "x"), (3, 2)), "aux": (("y", "x"), (3, 2))}) + assert variables["aux"].spans(variables["data"]) is True + + +def test_a_subset_spans(build): + variables = build({"data": (("y", "x"), (3, 2)), "aux": (("y",), (3,))}) + assert variables["aux"].spans(variables["data"]) is True + + +def test_a_different_name_does_not_span(build): + variables = build({"data": (("y", "x"), (3, 2)), "aux": (("t",), (3,))}) + assert variables["aux"].spans(variables["data"]) is False + + +def test_a_scalar_always_spans(build): + variables = build({"data": (("y", "x"), (3, 2)), "grid": ((), ())}) + assert variables["grid"].spans(variables["data"]) is True + + +def test_a_bounds_variable_ignores_its_extent_dimension(build): + variables = build({"data": (("y",), (3,)), "bounds": (("y", "bnds"), (3, 2))}) + bounds = CFBoundaryVariable("bounds", variables["bounds"].cf_data) + assert bounds.spans(variables["data"]) is True + + +def test_a_label_variable_ignores_its_string_length_dimension(build): + variables = build({"data": (("y",), (3,)), "label": (("y", "strlen"), (3, 8))}) + label = CFLabelVariable("label", variables["label"].cf_data) + assert label.spans(variables["data"]) is True + + +def test_a_name_at_two_sizes_does_not_link(tmp_path): + """Zarr-only: a name guarantees nothing across arrays. + + The version 3 core specification "does not place any restrictions on the + use of the same dimension name across multiple arrays within the same Zarr + hierarchy", so two arrays may mean different things by one name. Matching + on the name alone would produce a cube whose coordinate is the wrong + length. netCDF cannot express this: a dimension name has one size per + group. + """ + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array("data", shape=(3,), dtype="f4", dimension_names=("t",)) + group.create_array("aux", shape=(5,), dtype="f4", dimension_names=("t",)) + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + data = CFDataVariable("data", dataset.variables["data"]) + aux = CFDataVariable("aux", dataset.variables["aux"]) + assert aux.spans(data) is False + + +def test_an_anonymous_dimension_matches_nothing(tmp_path): + """Not even another anonymous dimension, of the same size, elsewhere. + + Bare-name matching treats `None == None` as a match: `aux` would span + `data` under the old rule, for a dimension neither variable actually + named. There is nothing shared between two differently-sourced "no + name"s, so claiming a match would be invention. + """ + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array("data", shape=(3,), dtype="f4") + group.create_array("aux", shape=(3,), dtype="f4") + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + data = CFDataVariable("data", dataset.variables["data"]) + aux = CFDataVariable("aux", dataset.variables["aux"]) + assert aux.spans(data) is False + + +def test_two_anonymous_dimensions_of_one_size_do_not_match(tmp_path): + """Not even each other, and not within one variable. + + There is no information to match on, so claiming a match would be + invention. + """ + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array("data", shape=(3, 3), dtype="f4") + group.create_array("aux", shape=(3,), dtype="f4") + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + data = CFDataVariable("data", dataset.variables["data"]) + aux = CFDataVariable("aux", dataset.variables["aux"]) + assert aux.spans(data) is False + + +def test_an_anonymous_dimension_is_not_a_coordinate_variable(tmp_path): + """A one-dimensional array is a coordinate variable by name coincidence. + + `CFCoordinateVariable.identify` requires the variable's own name to be + among its dimension names, which no anonymous dimension satisfies - so an + anonymous one-dimensional array is never mistaken for a coordinate + variable, whatever it is named. + """ + from iris.fileformats.cf import CFCoordinateVariable + + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array("t", shape=(3,), dtype="f4") + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + assert CFCoordinateVariable.identify(dataset.variables) == {} + + +def test_a_partly_named_variable_matches_on_the_named_part(tmp_path): + # "A variable may name some of its dimensions and not others." The named + # part ("y") is present in the target and would be enough to span under + # the old rule once the anonymous part is set aside - so `data` carries + # an anonymous dimension of its own, of the same size as `aux`'s, to show + # that one does not rescue the match either: bare-name matching would + # read both `None`s as the same dimension and wrongly call this a span. + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + group.create_array( + "data", shape=(3, 2, 2), dtype="f4", dimension_names=("y", "x", None) + ) + group.create_array("aux", shape=(3, 2), dtype="f4", dimension_names=("y", None)) + + dataset = ZarrDataset(str(tmp_path / "s.zarr")) + data = CFDataVariable("data", dataset.variables["data"]) + aux = CFDataVariable("aux", dataset.variables["aux"]) + assert aux.spans(data) is False From e234da07e28e5a0ccca273a566d3fd57923689c3 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 17:57:36 +0000 Subject: [PATCH 27/56] Let CFReader read any already-open CF dataset A loader that has opened a store hands it straight over, so the choice of backend stays with the loader rather than becoming a branch inside the reader. A borrowed dataset is not closed by the reader that borrowed it. Also wires the canonical Zarr test store's data variable to its grid mapping variable by name, which CF classification needs to recognise the grid mapping variable as one at all - it was present but unreferenced. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/cf/_reader.py | 26 +++++++++++-------- .../fileformats/cf/test_CFReader__dataset.py | 23 ++++++++++++++++ .../tests/unit/fileformats/cf/test_spans.py | 20 ++++++++++++++ .../tests/unit/fileformats/zarr/stores.py | 6 ++++- .../unit/fileformats/zarr/test_CFReader.py | 19 ++++++++++++++ 5 files changed, 82 insertions(+), 12 deletions(-) create mode 100644 lib/iris/tests/unit/fileformats/zarr/test_CFReader.py diff --git a/lib/iris/fileformats/cf/_reader.py b/lib/iris/fileformats/cf/_reader.py index bd056f2e7a..2b9caceb43 100644 --- a/lib/iris/fileformats/cf/_reader.py +++ b/lib/iris/fileformats/cf/_reader.py @@ -39,16 +39,14 @@ before anything that can raise, so the destructor is safe even if ``__init__`` fails part way. -This is the one module in :mod:`iris.fileformats.cf` still coupled to -:mod:`iris.fileformats.netcdf`: opening a file means building a -:class:`~iris.fileformats.netcdf._dataset.NetCDFDataset`. Everything else in -the package works against the format-agnostic -:class:`~iris.fileformats.cf.dataset.CFDataset` / -:class:`~iris.fileformats.cf.dataset.CFDatasetVariable` interface, so a Zarr -store can be read by the same machinery once a -:class:`~iris.fileformats.cf.dataset.CFDataset` implementation exists for it --- see §4.2 of the native Zarr I/O design, -``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. +``CFReader`` takes one of three things. A path opens a netCDF file, because +netCDF is the default a bare filename means. An open netCDF4 dataset, or an +emulator of one such as ncdata passes, is wrapped. And a +:class:`~iris.fileformats.cf.dataset.CFDataset` of any format is used as it +stands, which is how a Zarr store reaches this machinery: its loader opens +the store and hands it over, so nothing here has to know that Zarr exists. +Only the first case mentions netCDF, and only because something has to be +the default. """ @@ -79,6 +77,7 @@ CFUGridMeshVariable, _CFFormulaTermsVariable, ) +from .dataset import CFDataset #: Supported dimensionless vertical coordinate reference surface/phemomenon #: formula terms. Ref: [CF] Appendix D. @@ -134,7 +133,12 @@ class CFReader: def __init__(self, file_source, warn=False, monotonic=False): # Ensure safe operation for destructor, should init fail. self._own_file = False - if isinstance(file_source, str): + if isinstance(file_source, CFDataset): + # Already open, in whatever format: use it, don't own it. The + # caller that opened it is responsible for closing it. + self._dataset = file_source + self._filename = file_source.location + elif isinstance(file_source, str): # Create from filepath : open it + own it (=close when we die). if not urlparse(file_source).scheme: self._filename = Path(file_source).expanduser() diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py b/lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py index 97a185290b..2d91747616 100644 --- a/lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py +++ b/lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py @@ -205,3 +205,26 @@ def test_an_unread_attribute_reaches_the_cube(self, sample_path): # This does not pin the reverse: that no attribute Iris *read* reaches # the cube. Every attribute read here is in saver.py's _CF_ATTRS, and # _add_unused_attributes drops those before tracking is consulted. + + +def test_an_already_open_cf_dataset_is_used_as_is(tmp_path): + """A loader that has already opened a store hands it straight over. + + The alternative - a path and a format keyword - would put the choice of + backend inside CFReader, which is the branch the dataset interface exists + to remove. + """ + from iris.fileformats.netcdf._dataset import NetCDFDataset + + path = str(tmp_path / "source.nc") + with NetCDFDataset(path, mode="w") as writer: + writer.create_dimension("x", 2) + writer.create_variable("v", np.dtype("f4"), ("x",)) + + dataset = NetCDFDataset(path, mode="r") + with CFReader(dataset) as reader: + assert reader.filename == dataset.location + assert "v" in reader.cf_group + # Borrowed, not owned: leaving the context must not close it. + assert dataset.closed is False + dataset.close() diff --git a/lib/iris/tests/unit/fileformats/cf/test_spans.py b/lib/iris/tests/unit/fileformats/cf/test_spans.py index 2776b4d60d..2bcfcc93d0 100644 --- a/lib/iris/tests/unit/fileformats/cf/test_spans.py +++ b/lib/iris/tests/unit/fileformats/cf/test_spans.py @@ -181,3 +181,23 @@ def test_a_partly_named_variable_matches_on_the_named_part(tmp_path): data = CFDataVariable("data", dataset.variables["data"]) aux = CFDataVariable("aux", dataset.variables["aux"]) assert aux.spans(data) is False + + +def test_a_non_spanning_variable_is_dropped_and_promoted(tmp_path): + """An anonymous-dimension variable becomes a cube of its own. + + This is what keeps `None` out of `iris.fileformats._nc_load_rules.helpers`, + which matches dimensions by name and would read `None in (None, "x")` as a + match. `CFReader._build_cf_groups` drops any variable whose `spans` is + False from the referrer's group and promotes it. + """ + from iris.fileformats.cf import CFReader + + group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3) + data = group.create_array("data", shape=(3,), dtype="f4", dimension_names=("t",)) + data.attrs["coordinates"] = "aux" + group.create_array("aux", shape=(3,), dtype="f4") + + with CFReader(ZarrDataset(str(tmp_path / "s.zarr"))) as reader: + assert "aux" not in reader.cf_group["data"].cf_group + assert "aux" in reader.cf_group.promoted diff --git a/lib/iris/tests/unit/fileformats/zarr/stores.py b/lib/iris/tests/unit/fileformats/zarr/stores.py index 617fb8510e..9c1316d5af 100644 --- a/lib/iris/tests/unit/fileformats/zarr/stores.py +++ b/lib/iris/tests/unit/fileformats/zarr/stores.py @@ -15,7 +15,11 @@ #: The canonical store's dimensions, in order. DIMENSIONS = {"y": 3, "x": 2} DATA = np.arange(6, dtype="f8").reshape(3, 2) -VARIABLE_ATTRS = {"units": "K", "long_name": "surface temperature"} +VARIABLE_ATTRS = { + "units": "K", + "long_name": "surface temperature", + "grid_mapping": "grid", +} GLOBALS = {"Conventions": "CF-1.7", "title": "contract"} #: The version 2 attribute carrying dimension names. diff --git a/lib/iris/tests/unit/fileformats/zarr/test_CFReader.py b/lib/iris/tests/unit/fileformats/zarr/test_CFReader.py new file mode 100644 index 0000000000..e318d9ec05 --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/test_CFReader.py @@ -0,0 +1,19 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""The CF layer reads a Zarr store through the same reader netCDF uses.""" + +from iris.fileformats.cf import CFReader +from iris.fileformats.zarr._dataset import ZarrDataset + +from . import stores + + +def test_the_cf_reader_classifies_a_zarr_store(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + with CFReader(ZarrDataset(location)) as reader: + assert "air_temperature" in reader.cf_group.data_variables + assert "grid" in reader.cf_group.grid_mappings + assert reader.cf_group.global_attributes["title"] == "contract" + assert reader.filename == location From 8fc1dd410fbaa91eebdf5676920040bb5a803fee Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 18:17:15 +0000 Subject: [PATCH 28/56] Build cubes from a Zarr store Open the store, hand it to CFReader, and run the same cube-building loop the netCDF loader runs. One group per call: a store can hold unrelated datasets, so a group with no data variables names the groups that have some rather than returning nothing. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/zarr/__init__.py | 37 +++ lib/iris/fileformats/zarr/loader.py | 139 ++++++++++ .../unit/fileformats/zarr/loader/__init__.py | 5 + .../zarr/loader/test_load_cubes.py | 238 ++++++++++++++++++ 4 files changed, 419 insertions(+) create mode 100644 lib/iris/fileformats/zarr/loader.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/loader/__init__.py create mode 100644 lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py diff --git a/lib/iris/fileformats/zarr/__init__.py b/lib/iris/fileformats/zarr/__init__.py index e2cf16967d..78e7e08303 100644 --- a/lib/iris/fileformats/zarr/__init__.py +++ b/lib/iris/fileformats/zarr/__init__.py @@ -27,3 +27,40 @@ ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. """ + +__all__ = ["load_cubes"] + + +def __getattr__(name): + """Forward ``load_cubes`` to :mod:`iris.fileformats.zarr.loader`. + + PEP 562, not a plain ``from .loader import load_cubes``: ``loader`` + imports ``._dataset``, which imports ``zarr`` at module level, so an + eager import here would make the dependency load with the package rather + than with a Zarr load - breaking the promise this module's own docstring + makes, that nothing imports ``zarr`` until a Zarr target is actually + loaded. Resist "simplifying" this back to a plain import. + + Parameters + ---------- + name : str + The attribute being looked up. Python calls this only for names that + are not already module attributes. + + Returns + ------- + Any + ``load_cubes``, the only name this forwards. + + Raises + ------ + AttributeError + For any other name. + + """ + if name != "load_cubes": + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + + from .loader import load_cubes + + return load_cubes diff --git a/lib/iris/fileformats/zarr/loader.py b/lib/iris/fileformats/zarr/loader.py new file mode 100644 index 0000000000..f1d1ef5a2c --- /dev/null +++ b/lib/iris/fileformats/zarr/loader.py @@ -0,0 +1,139 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Load Iris cubes from Zarr stores. + +.. z_reference:: iris.fileformats.zarr.loader + :tags: topic_load_save + + API reference + +A thin loop: open the store as a +:class:`~iris.fileformats.zarr._dataset.ZarrDataset`, hand it to +:class:`~iris.fileformats.cf.CFReader`, and build one cube per CF data +variable with :mod:`iris.fileformats.cf.loader` - the same machinery the +netCDF loader runs. Everything Zarr-specific happens below this module, in +:mod:`iris.fileformats.zarr._dataset`. + +Iris loads **one** group per call. A Zarr store can legitimately hold +unrelated datasets, and merging them into one ``CubeList`` would be wrong, so +nested groups are not walked and ``group=`` selects which one to read. + +UGRID meshes are not built here. :func:`iris.mesh.load_meshes` reaches a Zarr +store through the same format dispatch, but the mesh machinery lives in +:mod:`iris.fileformats.netcdf.ugrid_load` and this package does not import it. + +See sections 4.6 and 4.7 of +``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. + +""" + +from collections.abc import Iterable +import warnings + +from iris.exceptions import TranslationError +from iris.fileformats.cf import CFReader +from iris.fileformats.cf.loader import ( + _actions_engine, + _load_aux_factory, + _load_cube, + _translate_constraints_to_var_callback, +) +import iris.warnings + +from ._dataset import ZarrDataset + + +def _check_has_data_variables(reader, dataset) -> None: + """Raise if this group holds nothing to make a cube from. + + Returning an empty ``CubeList`` from a store that plainly holds data + elsewhere leaves the user nothing to act on, and a store whose root group + holds only subgroups is a real shape - the ESA EOPF Sentinel products are + built that way. + """ + if reader.cf_group.data_variables or reader.cf_group.promoted: + return + + subgroups = dataset.subgroup_names + if subgroups: + message = ( + f"Zarr group {dataset.location!r} has no data variables. Load " + "one of its groups instead, with the 'group' keyword: " + f"{', '.join(sorted(subgroups))}" + ) + else: + message = f"Zarr store {dataset.location!r} has no data variables." + raise TranslationError(message) + + +def load_cubes(file_sources, callback=None, constraints=None, group=None): + """Load cubes from one or more Zarr stores. + + Parameters + ---------- + file_sources : str or iterable of str + The stores to load. A local directory path, or a URL that + zarr-python can open - ``s3://``, ``gs://``, ``az://`` and ``abfs://`` + need the matching store library installed, which Iris does not + depend on. + callback : callable, optional + Passed to :func:`iris.io.run_callback`. + constraints : optional + Used to skip data variables that cannot match, before they are built. + group : str, optional + A ``/``-separated path to a group within each store. The root group by + default. + + Yields + ------ + iris.cube.Cube + + Raises + ------ + iris.exceptions.TranslationError + If the selected group holds no data variables. The message names the + store's other groups, when it has any. + + Notes + ----- + This function maintains laziness; it does not realise data. + + """ + from iris.io import run_callback + + var_callback = _translate_constraints_to_var_callback(constraints) + engine = _actions_engine() + + if isinstance(file_sources, str) or not isinstance(file_sources, Iterable): + file_sources = [file_sources] + + for file_source in file_sources: + with ZarrDataset(file_source, group=group) as dataset: + with CFReader(dataset) as cf: + _check_has_data_variables(cf, dataset) + + data_variables = list(cf.cf_group.data_variables.values()) + list( + cf.cf_group.promoted.values() + ) + for cf_var in data_variables: + if var_callback and not var_callback(cf_var): + continue + + cube = _load_cube(engine, cf, cf_var, cf.filename) + + try: + _load_aux_factory(engine, cube) + except ValueError as exception: + warnings.warn( + str(exception), + category=iris.warnings.IrisLoadWarning, + ) + + cube = run_callback(callback, cube, cf_var, file_source) + if cube is None: + # The callback rejected it. + continue + + yield cube diff --git a/lib/iris/tests/unit/fileformats/zarr/loader/__init__.py b/lib/iris/tests/unit/fileformats/zarr/loader/__init__.py new file mode 100644 index 0000000000..dc37340f8b --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/loader/__init__.py @@ -0,0 +1,5 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :func:`iris.fileformats.zarr.load_cubes`.""" diff --git a/lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py b/lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py new file mode 100644 index 0000000000..f4fdaaddba --- /dev/null +++ b/lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py @@ -0,0 +1,238 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :func:`iris.fileformats.zarr.load_cubes`.""" + +import numpy as np +import pytest +import zarr + +from iris.cube import Cube +from iris.exceptions import TranslationError +from iris.fileformats.zarr import load_cubes + +from .. import stores + + +def test_one_cube_per_data_variable(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + cubes = list(load_cubes(location)) + assert len(cubes) == 1 + cube = cubes[0] + assert isinstance(cube, Cube) + assert cube.var_name == "air_temperature" + assert cube.units == "K" + np.testing.assert_array_equal(cube.data, stores.DATA) + + +def test_a_version_2_store_loads_the_same_cube(tmp_path): + v3 = list(load_cubes(stores.make_store(tmp_path / "v3.zarr")))[0] + v2 = list(load_cubes(stores.make_store(tmp_path / "v2.zarr", zarr_format=2)))[0] + assert v2 == v3 + + +def test_a_callback_can_reject_a_cube(tmp_path): + from iris.exceptions import IgnoreCubeException + + def reject(cube, field, filename): + raise IgnoreCubeException + + location = stores.make_store(tmp_path / "s.zarr") + assert list(load_cubes(location, callback=reject)) == [] + + +def test_the_group_keyword_selects_a_group(tmp_path): + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + sub = root.create_group("measurements") + band = stores.make_array( + sub, "b02", (3,), "u2", 3, dimensions=("p",), attributes={"units": "1"} + ) + band[:] = [1, 2, 3] + + cubes = list(load_cubes(str(tmp_path / "n.zarr"), group="measurements")) + assert [cube.var_name for cube in cubes] == ["b02"] + + +def test_several_stores_in_one_call(tmp_path): + first = stores.make_store(tmp_path / "a.zarr") + second = stores.make_store(tmp_path / "b.zarr") + assert len(list(load_cubes([first, second]))) == 2 + + +def test_a_group_with_no_arrays_names_the_groups_that_have_them(tmp_path): + """A real store in this shape exists, and silence is the worst answer. + + The ESA EOPF Sentinel-2 products hold no arrays in the root group, only + subgroups - `conditions/geometry`, `measurements/reflectance/r10m` and + others. Iris loads one group per call, so loading the default root yields + nothing; returning an empty CubeList from a store that plainly holds data + leaves the user with no way to find out why. + """ + root = zarr.open_group(store=tmp_path / "e.zarr", mode="w", zarr_format=3) + root.create_group("conditions") + measurements = root.create_group("measurements") + stores.make_array(measurements, "b02", (2,), "u2", 3, dimensions=("p",)) + + with pytest.raises(TranslationError, match="measurements"): + list(load_cubes(str(tmp_path / "e.zarr"))) + + +def test_the_message_says_so_when_there_are_no_groups_either(tmp_path): + zarr.open_group(store=tmp_path / "empty.zarr", mode="w", zarr_format=3) + with pytest.raises(TranslationError, match="no data variables"): + list(load_cubes(str(tmp_path / "empty.zarr"))) + + +def test_nested_groups_are_not_walked(tmp_path): + """Iris loads one group per call, like xarray. + + A Zarr store can legitimately hold unrelated datasets, and merging them + into one CubeList would be wrong. + """ + root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + top = stores.make_array( + root, "top", (2,), "f8", 3, dimensions=("p",), attributes={"units": "1"} + ) + top[:] = [1.0, 2.0] + inner = stores.make_array( + root.create_group("sub"), "inner", (2,), "f8", 3, dimensions=("p",) + ) + inner[:] = [3.0, 4.0] + + cubes = list(load_cubes(str(tmp_path / "n.zarr"))) + assert [cube.var_name for cube in cubes] == ["top"] + + +def test_the_cube_is_lazy(tmp_path, monkeypatch): + from iris.fileformats.cf import loader as cf_loader + + monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1) + cube = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))[0] + assert cube.has_lazy_data() + + +def test_the_grid_mapping_variable_does_not_become_a_cube(tmp_path): + cubes = list(load_cubes(stores.make_store(tmp_path / "s.zarr"))) + assert "grid" not in [cube.var_name for cube in cubes] + + +def test_cf_reserved_attributes_do_not_reach_the_cube(tmp_path): + """`units` becomes a property, so it must not also be an attribute. + + Which attributes survive is decided by which ones the loading rules did + not read, so an attribute Iris consumes but fails to record as read + arrives on the cube as well. + """ + cube = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))[0] + assert "units" not in cube.attributes + assert "long_name" not in cube.attributes + assert cube.attributes["Conventions"] == "CF-1.7" + + +def test_a_structured_attribute_reaches_the_cube_unchanged(tmp_path): + """CF cannot express a nested attribute; Zarr allows one. Carry it. + + Two independent producers write them - NOAA GFS gives `valid_time` a + `statistics_approximate` object, and the ESA EOPF product root carries + `stac_discovery`. Interpreting them needs the attribute-model discussion + in SciTools/iris#7288; dropping them loses the publisher's metadata. + """ + group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "v", + (2,), + "f8", + 3, + dimensions=("p",), + attributes={"stac_discovery": {"id": "abc"}}, + ) + array[:] = [1.0, 2.0] + + cube = list(load_cubes(str(tmp_path / "n.zarr")))[0] + assert cube.attributes["stac_discovery"] == {"id": "abc"} + + +def test_a_structured_attribute_cannot_then_be_saved_to_netcdf(tmp_path): + """Known limitation, characterised so that it is not a surprise. + + Zarr allows any JSON literal as an attribute value and netCDF does not, + so a store carrying one produces a cube Iris cannot save as netCDF. + Checking attribute content before saving is SciTools/iris#7329, and + fixing it means changing the netCDF saver. Strip the attribute, or save + to Zarr once that is supported. + """ + import iris + + group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "v", + (2,), + "f8", + 3, + dimensions=("p",), + attributes={"stac_discovery": {"id": "abc"}}, + ) + array[:] = [1.0, 2.0] + cube = list(load_cubes(str(tmp_path / "n.zarr")))[0] + + with pytest.raises(TypeError, match="stac_discovery"): + iris.save(cube, str(tmp_path / "out.nc")) + + # And the documented workaround works. + del cube.attributes["stac_discovery"] + iris.save(cube, str(tmp_path / "out.nc")) + + +def test_version_3_zeros_are_not_masked(tmp_path): + """An int16 [0, 1, 2] with no CF attributes loads with nothing masked. + + zarr-python defaults `fill_value` to zero, so reading the storage field + as a sentinel masks valid data in the ordinary case. + """ + group = zarr.open_group(store=tmp_path / "z3.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, "v", (3,), "i2", 3, dimensions=("p",), attributes={"units": "1"} + ) + array[:] = [0, 1, 2] + + cube = list(load_cubes(str(tmp_path / "z3.zarr")))[0] + assert not np.ma.is_masked(cube.data) + np.testing.assert_array_equal(cube.data, [0, 1, 2]) + + +def test_version_2_zeros_are_masked(tmp_path): + """The same array as version 2 does use `fill_value` as the mask source. + + Version 2 stores predate the CF `_FillValue` convention, and this is what + their writers meant - which is what released xarray's + `use_zarr_fill_value_as_mask` does. + """ + group = zarr.open_group(store=tmp_path / "z2.zarr", mode="w", zarr_format=2) + array = stores.make_array( + group, "v", (3,), "i2", 2, dimensions=("p",), attributes={"units": "1"} + ) + array[:] = [0, 1, 2] + + cube = list(load_cubes(str(tmp_path / "z2.zarr")))[0] + assert cube.data.mask.tolist() == [True, False, False] + + +def test_a_cf_fill_value_beats_the_storage_one(tmp_path): + group = zarr.open_group(store=tmp_path / "z.zarr", mode="w", zarr_format=3) + array = stores.make_array( + group, + "v", + (3,), + "i2", + 3, + dimensions=("p",), + fill_value=0, + attributes={"units": "1", "_FillValue": -999}, + ) + array[:] = [0, -999, 2] + + cube = list(load_cubes(str(tmp_path / "z.zarr")))[0] + assert cube.data.mask.tolist() == [False, True, False] From c2612d20306817c8f7c1194cb7c70c4f771e0bb0 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 18:32:23 +0000 Subject: [PATCH 29/56] Teach the format agent to recognise a Zarr store Locally by the marker file a store puts at its root, remotely by the URI scheme and the .zarr suffix. Not by the suffix alone: that would claim any directory someone named .zarr, and joined with the remote rule it would leave the scheme test unreachable. An NCZarr URL is still NCZarr's, which needs saying in code because the store it names is a real one. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/__init__.py | 67 +++++++++- lib/iris/io/__init__.py | 6 + lib/iris/io/format_picker.py | 17 +++ .../unit/io/format_picker/test_ZarrStore.py | 121 ++++++++++++++++++ 4 files changed, 210 insertions(+), 1 deletion(-) create mode 100644 lib/iris/tests/unit/io/format_picker/test_ZarrStore.py diff --git a/lib/iris/fileformats/__init__.py b/lib/iris/fileformats/__init__.py index 873ca644ef..93c6be23e6 100644 --- a/lib/iris/fileformats/__init__.py +++ b/lib/iris/fileformats/__init__.py @@ -10,7 +10,9 @@ API reference """ -from iris.io import _is_nczarr_fragment +from pathlib import Path + +from iris.io import REMOTE_STORE_SCHEMES, _is_nczarr_fragment, decode_uri from iris.io.format_picker import ( DataSourceObjectProtocol, FileExtension, @@ -19,6 +21,7 @@ LeadingLine, MagicNumber, UriProtocol, + ZarrStore, ) from . import name, netcdf, nimrod, pp, um @@ -44,6 +47,56 @@ def _uri_is_nczarr(uri_parts): return _is_nczarr_fragment(fragment) +#: What a version 3 and a version 2 Zarr store put at the root of a store. +_ZARR_MARKERS = ("zarr.json", ".zgroup") + + +def _is_zarr_store(target): + """Whether `target` names a native Zarr store. + + Locally the test is a marker file, which costs one stat and is what tells + a real store from a directory someone named ``.zarr``. Remotely it is the + scheme and the suffix, because reading a marker file would cost a network + round trip before Iris knows it wants the target at all; a remote + ``.zarr`` that turns out not to be a store fails when zarr-python opens + it, with zarr-python's message. + + These are two tests for two kinds of target, not two chances for one. A + third rule matching any path ending in ``.zarr`` would subsume the remote + one and leave the scheme test unreachable. + """ + if not isinstance(target, str): + # iris.io.load_data_objects passes open datasets, not paths. + return False + + scheme, part, fragment = decode_uri(target) + if _is_nczarr_fragment(fragment): + # The user asked for netCDF-c's Zarr driver by name. The store on + # disk is a real one, so the local test below would otherwise claim it. + return False + + if scheme in REMOTE_STORE_SCHEMES: + return part.rstrip("/").endswith(".zarr") + + directory = Path(part) + return any((directory / marker).is_file() for marker in _ZARR_MARKERS) + + +def _load_zarr(*args, **kwargs): + """Load from Zarr stores, reporting clearly when zarr-python is absent.""" + try: + from iris.fileformats.zarr import load_cubes + except ImportError as exception: + message = ( + "Unable to load a Zarr store - the 'zarr' package is not " + "installed. Install zarr >=3.0.8, for example with " + "'conda install -c conda-forge zarr'." + ) + raise ImportError(message) from exception + + return load_cubes(*args, **kwargs) + + # # PP files. # @@ -169,6 +222,18 @@ def _load_grib(*args, **kwargs): # # Zarr files. # +FORMAT_AGENT.add_spec( + FormatSpecification( + "Zarr", + ZarrStore(), + _is_zarr_store, + _load_zarr, + priority=5, + constraint_aware_handler=True, + ) +) + + FORMAT_AGENT.add_spec( FormatSpecification( "NcZarr", diff --git a/lib/iris/io/__init__.py b/lib/iris/io/__init__.py index ddeb6a7157..66edc284e9 100644 --- a/lib/iris/io/__init__.py +++ b/lib/iris/io/__init__.py @@ -38,6 +38,12 @@ def _is_nczarr_fragment(fragment): return False +#: URI schemes Iris hands to a store library rather than resolving as a local +#: path. Iris depends on none of them: zarr-python raises its own ImportError +#: naming what to install. +REMOTE_STORE_SCHEMES = frozenset({"s3", "gs", "az", "abfs"}) + + # Saving routines, indexed by file extension. class _SaversDict(dict): """A dictionary that can only have string keys with no overlap.""" diff --git a/lib/iris/io/format_picker.py b/lib/iris/io/format_picker.py index ac80e03611..b1e334e631 100644 --- a/lib/iris/io/format_picker.py +++ b/lib/iris/io/format_picker.py @@ -377,6 +377,23 @@ def get_element(self, basename, file_handle): return decode_uri(basename) +class ZarrStore(FileElement): + """A :class:`FileElement` that returns a target that may be a Zarr store. + + Carries the whole target rather than its basename, because recognising a + local store means reading a marker file inside it, and recognising a + remote one means reading its scheme. + + """ + + def __init__(self): + FileElement.__init__(self, requires_fh=False) + + def get_element(self, basename, file_handle): + """Return `basename`, which here is the whole path or URL.""" + return basename + + class DataSourceObjectProtocol(FileElement): """A :class:`FileElement` that simply returns the URI entry itself. diff --git a/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py b/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py new file mode 100644 index 0000000000..5dace9df69 --- /dev/null +++ b/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py @@ -0,0 +1,121 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Recognising a Zarr store, locally by its marker file and remotely by its URL.""" + +import pytest + +from iris.fileformats import _is_zarr_store +from iris.io.format_picker import ZarrStore + + +def test_the_element_carries_the_whole_target(): + # Not the basename: recognising a local store means reading a file inside + # it, so the element needs the path, not the last component of it. + element = ZarrStore() + assert element.requires_fh is False + assert element.get_element("/data/store.zarr", None) == "/data/store.zarr" + + +def test_a_version_3_store_is_recognised(tmp_path): + store = tmp_path / "v3.zarr" + store.mkdir() + (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}') + assert _is_zarr_store(str(store)) is True + + +def test_a_version_2_store_is_recognised(tmp_path): + store = tmp_path / "v2.zarr" + store.mkdir() + (store / ".zgroup").write_text('{"zarr_format": 2}') + assert _is_zarr_store(str(store)) is True + + +def test_a_directory_named_zarr_with_no_marker_is_not_a_store(tmp_path): + """The suffix alone is not the local test, and must not become one. + + Reading the marker file is what distinguishes a real store from a + directory someone happened to name `.zarr`, and locally that costs one + stat. + """ + store = tmp_path / "pretend.zarr" + store.mkdir() + assert _is_zarr_store(str(store)) is False + + +def test_a_plain_file_is_not_a_store(tmp_path): + path = tmp_path / "data.nc" + path.write_bytes(b"CDF\x01") + assert _is_zarr_store(str(path)) is False + + +@pytest.mark.parametrize("scheme", ["s3", "gs", "az", "abfs"]) +def test_a_remote_store_is_recognised_by_scheme_and_suffix(scheme): + assert _is_zarr_store(f"{scheme}://bucket/path/data.zarr") is True + assert _is_zarr_store(f"{scheme}://bucket/path/data.zarr/") is True + + +def test_a_remote_url_without_the_suffix_is_not_a_store(): + # Remotely the suffix is all there is to go on: reading a marker file + # would cost a network round trip before Iris knows it wants the target. + assert _is_zarr_store("s3://bucket/path/data") is False + + +def test_an_http_url_is_not_a_store(): + # OPeNDAP, which the netCDF loader already claims. + assert _is_zarr_store("https://example.invalid/data.zarr") is False + + +def test_an_nczarr_url_is_not_a_native_store(tmp_path): + """A user who writes `#mode=nczarr` has asked for NCZarr by name. + + The store on disk is a real one, so the marker-file test says yes unless + the fragment is checked first - and `load_http` passes the fragment + through to the format agent. + """ + store = tmp_path / "s.zarr" + store.mkdir() + (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}') + assert _is_zarr_store(f"file://{store}#mode=nczarr,file") is False + + +def test_a_non_string_target_is_not_a_store(): + # `load_data_objects` passes open datasets, not paths. + assert _is_zarr_store(object()) is False + + +def test_the_agent_still_picks_netcdf_for_a_netcdf_file(tmp_path): + """A `requires_fh=False` spec is consulted for every target with no buffer. + + Adding one risks claiming a target another format already handles, and + the format agent returns the first match rather than the best. + """ + from iris.fileformats import FORMAT_AGENT + import iris.tests as tests + + path = tests.get_data_path( + ("NetCDF", "global", "xyt", "SMALL_hires_wind_u_for_ipcc4.nc") + ) + with open(path, "rb") as handle: + spec = FORMAT_AGENT.get_spec("SMALL_hires_wind_u_for_ipcc4.nc", handle) + assert spec.name.startswith("NetCDF") + + +def test_the_agent_picks_zarr_for_a_store(tmp_path): + from iris.fileformats import FORMAT_AGENT + + store = tmp_path / "s.zarr" + store.mkdir() + (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}') + assert FORMAT_AGENT.get_spec(str(store), None).name == "Zarr" + + +def test_the_agent_still_picks_nczarr_for_an_nczarr_url(tmp_path): + from iris.fileformats import FORMAT_AGENT + + store = tmp_path / "s.zarr" + store.mkdir() + (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}') + uri = f"file://{store}#mode=nczarr,file" + assert FORMAT_AGENT.get_spec(uri, None).name == "NcZarr" From f0562975bae8f7acc8f792bff5d42b96ae960efe Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 18:35:48 +0000 Subject: [PATCH 30/56] Skip the NetCDF-detection test when iris-test-data is absent test_the_agent_still_picks_netcdf_for_a_netcdf_file opened a real file with no skip_data guard, so missing test data raised FileNotFoundError instead of skipping - the opposite of lib/iris/tests/AGENTS.md's "missing data skips affected tests by design, silently, and without failing". Follows the same @_shared_utils.skip_data idiom already used in test_load_meshes.py. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/tests/unit/io/format_picker/test_ZarrStore.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py b/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py index 5dace9df69..6b4bd2b8a9 100644 --- a/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py +++ b/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py @@ -8,6 +8,7 @@ from iris.fileformats import _is_zarr_store from iris.io.format_picker import ZarrStore +from iris.tests import _shared_utils def test_the_element_carries_the_whole_target(): @@ -85,16 +86,16 @@ def test_a_non_string_target_is_not_a_store(): assert _is_zarr_store(object()) is False -def test_the_agent_still_picks_netcdf_for_a_netcdf_file(tmp_path): +@_shared_utils.skip_data +def test_the_agent_still_picks_netcdf_for_a_netcdf_file(): """A `requires_fh=False` spec is consulted for every target with no buffer. Adding one risks claiming a target another format already handles, and the format agent returns the first match rather than the best. """ from iris.fileformats import FORMAT_AGENT - import iris.tests as tests - path = tests.get_data_path( + path = _shared_utils.get_data_path( ("NetCDF", "global", "xyt", "SMALL_hires_wind_u_for_ipcc4.nc") ) with open(path, "rb") as handle: From 461801431ffe3336cc109617c8a2b91965ebd0b7 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 19:16:05 +0000 Subject: [PATCH 31/56] Let the load chain reach a target that is not a file A new iris.io.load_stores answers 'this target is not a file', once, for a local directory and a remote URL alike, rather than teaching load_files about directories in one place and the scheme dispatch in another. The partition happens after glob expansion, so iris.load('data/*') still accepts a mix of netCDF files and Zarr stores. load_files no longer re-expands its input: _generate_cubes already runs expand_filespecs before partitioning files from stores, and expanding an already-expanded list silently collapses two identical paths produced by two different glob patterns into one, which broke the existing duplicate- match loading behaviour. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/netcdf/ugrid_load.py | 11 +- lib/iris/io/__init__.py | 98 ++++++++++-- lib/iris/loading.py | 19 ++- .../tests/unit/io/test__generate_cubes.py | 6 + .../tests/unit/io/test_expand_filespecs.py | 11 ++ lib/iris/tests/unit/io/test_load_stores.py | 146 ++++++++++++++++++ 6 files changed, 274 insertions(+), 17 deletions(-) create mode 100644 lib/iris/tests/unit/io/test_load_stores.py diff --git a/lib/iris/fileformats/netcdf/ugrid_load.py b/lib/iris/fileformats/netcdf/ugrid_load.py index dc4335144b..34d9419081 100644 --- a/lib/iris/fileformats/netcdf/ugrid_load.py +++ b/lib/iris/fileformats/netcdf/ugrid_load.py @@ -29,7 +29,12 @@ from iris.common.mixin import CFVariableMixin from iris.config import get_logger from iris.coords import AuxCoord -from iris.io import _is_nczarr_fragment, decode_uri, expand_filespecs +from iris.io import ( + REMOTE_STORE_SCHEMES, + _is_nczarr_fragment, + decode_uri, + expand_filespecs, +) from iris.loading import LoadProblems from iris.mesh.components import Connectivity, MeshXY from iris.util import guess_coord_axis @@ -169,12 +174,14 @@ def _categorise(decoded: tuple[str, str, str | None]) -> tuple[str, str, str]: sources = expand_filespecs(filenames) elif category in ["http", "https", "nczarr"]: sources = [f"{scheme}:{part}" for _cat, scheme, part in groups] + elif category in REMOTE_STORE_SCHEMES: + sources = [f"{scheme}:{part}" for _cat, scheme, part in groups] else: message = f"Iris cannot handle the URI scheme: {category}" raise ValueError(message) for source in sources: - if category == "file": + if category == "file" and not Path(source).is_dir(): with open(source, "rb") as fh: handling_format_spec = FORMAT_AGENT.get_spec(Path(source).name, fh) else: diff --git a/lib/iris/io/__init__.py b/lib/iris/io/__init__.py index 66edc284e9..b99d8bf50c 100644 --- a/lib/iris/io/__init__.py +++ b/lib/iris/io/__init__.py @@ -44,6 +44,18 @@ def _is_nczarr_fragment(fragment): REMOTE_STORE_SCHEMES = frozenset({"s3", "gs", "az", "abfs"}) +def _is_remote_store_uri(spec): + """Whether `spec` names a store Iris hands to a store library untouched. + + Such a URI must not be resolved as a local path: `.absolute()` turns + ``s3://bucket/data.zarr`` into ``//s3:/bucket/data.zarr``. + """ + if not isinstance(spec, str): + return False + scheme, _part, _fragment = decode_uri(spec) + return scheme in REMOTE_STORE_SCHEMES + + # Saving routines, indexed by file extension. class _SaversDict(dict): """A dictionary that can only have string keys with no overlap.""" @@ -61,6 +73,10 @@ def __setitem__(self, key, value): _savers = _SaversDict() +#: Modules whose `save` takes a CubeList directly, rather than one cube at a +#: time with `append=True`. One line per format. +_CUBELIST_SAVER_MODULES = frozenset({"iris.fileformats.netcdf.saver"}) + def run_callback(callback, cube, field, filename): """Run the callback mechanism given the appropriate arguments. @@ -196,15 +212,24 @@ def expand_filespecs(file_specs, files_expected=True): If files_expected is ``False``: A list of expanded file paths. """ - # Remove any hostname component - currently unused + # Remove any hostname component - currently unused. A remote store URI is + # passed through: it is not a path, and resolving it mangles it. filenames = [ - str(pathlib.Path(fn.removeprefix("//")).expanduser().absolute()) + fn + if _is_remote_store_uri(fn) + else str(pathlib.Path(fn.removeprefix("//")).expanduser().absolute()) for fn in file_specs ] if files_expected: - # Try to expand all filenames as globs - glob_expanded = OrderedDict([[fn, sorted(glob.glob(fn))] for fn in filenames]) + # Try to expand all filenames as globs. A remote URI stands for + # itself; Iris does not list a remote store's neighbours. + glob_expanded = OrderedDict( + [ + [fn, [fn] if _is_remote_store_uri(fn) else sorted(glob.glob(fn))] + for fn in filenames + ] + ) # If any of the specs expanded to an empty list then raise an error all_expanded = glob_expanded.values() @@ -227,8 +252,8 @@ def expand_filespecs(file_specs, files_expected=True): def load_files(filenames, callback, constraints=None): """Create a generator of Cubes from given files. - Take a list of filenames which may also be globs, and optionally a - constraint set and a callback function, and returns a + Take a list of filenames, already expanded by :func:`expand_filespecs`, + and optionally a constraint set and a callback function, and returns a generator of Cubes from the given files. Notes @@ -236,14 +261,18 @@ def load_files(filenames, callback, constraints=None): Typically, this function should not be called directly; instead, the intended interface for loading is :func:`iris.load`. + `filenames` is not re-expanded here: a glob matching the same file twice, + by two different patterns, must reach the format handler as two entries, + and :func:`expand_filespecs` keys its own glob cache by the spec string, + so expanding an already-expanded list collapses two identical paths into + one. + """ from iris.fileformats import FORMAT_AGENT - all_file_paths = expand_filespecs(filenames) - # Create default dict mapping iris format handler to its associated filenames handler_map = collections.defaultdict(list) - for fn in all_file_paths: + for fn in filenames: with open(fn, "rb") as fh: handling_format_spec = FORMAT_AGENT.get_spec(pathlib.Path(fn).name, fh) handler_map[handling_format_spec].append(fn) @@ -259,6 +288,49 @@ def load_files(filenames, callback, constraints=None): yield cube +def load_stores(targets, callback, constraints=None): + """Create a generator of Cubes from targets that are stores, not files. + + A store is a directory or a URL: something with no single seekable file + to open, so :func:`load_files` cannot reach it and the format agent is + asked without a buffer. + + Parameters + ---------- + targets : iterable of str + Local directory paths or remote URLs, already expanded. + callback : callable + Passed to the format handler. + constraints : optional + Passed to a constraint-aware format handler. + + Yields + ------ + iris.cube.Cube + + Notes + ----- + Typically, this function should not be called directly; instead, the + intended interface for loading is :func:`iris.load`. + + """ + from iris.fileformats import FORMAT_AGENT + + handler_map = collections.defaultdict(list) + for target in targets: + handling_format_spec = FORMAT_AGENT.get_spec(target, None) + handler_map[handling_format_spec].append(target) + + for handling_format_spec in sorted(handler_map): + names = handler_map[handling_format_spec] + if handling_format_spec.constraint_aware_handler: + for cube in handling_format_spec.handler(names, callback, constraints): + yield cube + else: + for cube in handling_format_spec.handler(names, callback): + yield cube + + def load_http(urls, callback): """Create generator of Cubes from the given OPeNDAP or NcZarr URLs. @@ -398,10 +470,10 @@ def find_saver(filespec): """ _check_init_savers() + # A store is a directory, so "out.zarr/" is a natural thing to type. + candidate = filespec.rstrip("/\\").lower() matches = [ - ext - for ext in _savers - if filespec.lower().endswith("." + ext) or filespec.lower() == ext + ext for ext in _savers if candidate.endswith("." + ext) or candidate == ext ] # Multiple matches could occur if one of the savers included a '.': # e.g. _savers = {'.dot.png': dot_png_saver, '.png': png_saver} @@ -533,7 +605,7 @@ def save(source, target, saver=None, **kwargs): isinstance(source, (list, tuple)) and all([isinstance(i, Cube) for i in source]) ): # Only allow cubelist saving for those fileformats that are capable. - if "iris.fileformats.netcdf" not in saver.__module__: + if saver.__module__ not in _CUBELIST_SAVER_MODULES: # Make sure the saver accepts an append keyword if "append" not in saver.__code__.co_varnames: raise ValueError( diff --git a/lib/iris/loading.py b/lib/iris/loading.py index 89e2e43059..fc8d2e6572 100644 --- a/lib/iris/loading.py +++ b/lib/iris/loading.py @@ -13,6 +13,7 @@ from contextlib import contextmanager from dataclasses import dataclass import itertools +from pathlib import Path import threading from traceback import TracebackException from typing import Any, Iterable, Type @@ -97,12 +98,26 @@ def _categorise(decoded: tuple[str, str, str | None]) -> tuple[str, str, str]: # Call each scheme handler with the appropriate URIs if category == "file": part_names = [part for _cat, _scheme, part in groups] - for cube in iris.io.load_files(part_names, callback, constraints): - yield cube + # The file-or-store decision has to happen after glob expansion + # and before format dispatch, because a glob may match both. + # Neither routine below learns about the other's kind of target. + expanded = iris.io.expand_filespecs(part_names) + files = [name for name in expanded if not Path(name).is_dir()] + stores = [name for name in expanded if Path(name).is_dir()] + if files: + for cube in iris.io.load_files(files, callback, constraints): + yield cube + if stores: + for cube in iris.io.load_stores(stores, callback, constraints): + yield cube elif category in ["http", "https", "nczarr"]: urls = [f"{scheme}:{part}" for _cat, scheme, part in groups] for cube in iris.io.load_http(urls, callback): yield cube + elif category in iris.io.REMOTE_STORE_SCHEMES: + urls = [f"{scheme}:{part}" for _cat, scheme, part in groups] + for cube in iris.io.load_stores(urls, callback, constraints): + yield cube elif category == "data": data_objects = [part for _cat, _scheme, part in groups] for cube in iris.io.load_data_objects(data_objects, callback): diff --git a/lib/iris/tests/unit/io/test__generate_cubes.py b/lib/iris/tests/unit/io/test__generate_cubes.py index 406ea2fa32..d8ff2ec08d 100644 --- a/lib/iris/tests/unit/io/test__generate_cubes.py +++ b/lib/iris/tests/unit/io/test__generate_cubes.py @@ -22,6 +22,9 @@ def test_pathlib_paths(self, mocker): decode_uri_mock = mocker.patch( "iris.iris.io.decode_uri", return_value=("file", None, None) ) + # `decode_uri` always reporting a `None` part is not a real path, so + # the file-or-store partition has nothing to resolve or classify. + mocker.patch("iris.iris.io.expand_filespecs", return_value=[]) mocker.patch("iris.iris.io.load_files") for gc_arg, du_arg in test_variants: @@ -48,6 +51,9 @@ def test_nczarr_uri(self, uri, mocker): def test_non_nczarr_uri(self, mocker): uri = "file:///foo#bar=baz" + # The partition needs a real-looking part name to classify, not a + # file-system lookup against a path that was never written. + mocker.patch("iris.iris.io.expand_filespecs", side_effect=lambda names: names) load_files_mock = mocker.patch("iris.iris.io.load_files", return_value=[]) load_http_mock = mocker.patch("iris.iris.io.load_http", return_value=[]) diff --git a/lib/iris/tests/unit/io/test_expand_filespecs.py b/lib/iris/tests/unit/io/test_expand_filespecs.py index 0a33f87c15..9ff879867e 100644 --- a/lib/iris/tests/unit/io/test_expand_filespecs.py +++ b/lib/iris/tests/unit/io/test_expand_filespecs.py @@ -105,3 +105,14 @@ def test_false_bool_relative(self): assert item_out == item_in finally: os.chdir(cwd) + + def test_remote_store_uri_is_passed_through_unresolved(self): + # `.absolute()` would otherwise turn this into + # "//s3:/bucket/data.zarr", which zarr-python cannot open. + uri = "s3://bucket/data.zarr" + assert iio.expand_filespecs([uri]) == [uri] + + def test_remote_store_uri_is_not_globbed(self): + # A remote URI stands for itself: Iris does not list its neighbours. + uri = "s3://bucket/data.zarr" + assert iio.expand_filespecs([uri], False) == [uri] diff --git a/lib/iris/tests/unit/io/test_load_stores.py b/lib/iris/tests/unit/io/test_load_stores.py new file mode 100644 index 0000000000..20f31cf3c8 --- /dev/null +++ b/lib/iris/tests/unit/io/test_load_stores.py @@ -0,0 +1,146 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for :func:`iris.io.load_stores`.""" + +import numpy as np +import pytest + +import iris +from iris.io import _is_remote_store_uri, load_stores +from iris.tests import _shared_utils + +zarr = pytest.importorskip("zarr", minversion="3.0.8") + +from iris.tests.unit.fileformats.zarr import stores # noqa: E402 + + +def test_a_local_store_loads(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + cubes = list(load_stores([location], None)) + assert [cube.var_name for cube in cubes] == ["air_temperature"] + + +def test_several_stores_in_one_call(tmp_path): + first = stores.make_store(tmp_path / "a.zarr") + second = stores.make_store(tmp_path / "b.zarr") + assert len(list(load_stores([first, second], None))) == 2 + + +def test_constraints_reach_the_handler(tmp_path): + """A constraint the handler can pre-filter on rules out a data variable. + + `load_stores` forwards `constraints` to a constraint-aware handler rather + than applying it itself - the general `cube_func` form is only enforced + later, by `_CubeFilter.add` - so the constraint used here has to be one + `_translate_constraints_to_var_callback` can act on: a `NameConstraint` + with no `STASH` component. + """ + location = stores.make_store(tmp_path / "s.zarr") + constraint = iris.NameConstraint(var_name="not_air_temperature") + assert list(load_stores([location], None, constraint)) == [] + + +@pytest.mark.parametrize( + ("uri", "expected"), + [ + ("s3://bucket/data.zarr", True), + ("gs://bucket/data.zarr", True), + ("az://container/data.zarr", True), + ("abfs://container/data.zarr", True), + ("https://example.invalid/data.nc", False), + ("/local/path/data.zarr", False), + ("relative/data.zarr", False), + ], +) +def test_which_uris_are_remote_stores(uri, expected): + assert _is_remote_store_uri(uri) is expected + + +def test_iris_load_accepts_a_local_store(tmp_path): + location = stores.make_store(tmp_path / "s.zarr") + cubes = iris.load(location) + assert len(cubes) == 1 + np.testing.assert_array_equal(cubes[0].data, stores.DATA) + + +def test_iris_load_accepts_a_remote_url_and_does_not_absolutise_it(): + """A remote URL must reach zarr-python intact. + + `expand_filespecs` would otherwise turn `s3://bucket/data.zarr` into + `//s3:/bucket/data.zarr`. No network is touched: opening the store + fails, and what matters is that the message still names the URL. + """ + # The failure mode depends on what is installed to talk to S3, so no + # single exception type or message is guaranteed - checked below instead. + with pytest.raises(Exception) as caught: # noqa: PT011 + iris.load("s3://bucket/data.zarr") + assert "s3://bucket/data.zarr" in str(caught.value) or isinstance( + caught.value, ImportError + ) + + +@_shared_utils.skip_data +def test_a_glob_matching_both_files_and_stores(tmp_path): + """`iris.load("data/*")` must accept a mix. + + The file-or-store decision happens after glob expansion and before format + dispatch. Mis-partitioning raises IsADirectoryError from `load_files` for + the stores while the netCDF files load fine, which reads as an Iris bug + to a user who did not expect the glob to match a store. + """ + source = iris.load_cube( + _shared_utils.get_data_path( + ("NetCDF", "global", "xyt", "SMALL_hires_wind_u_for_ipcc4.nc") + ) + ) + iris.save(source, str(tmp_path / "a.nc")) + stores.make_store(tmp_path / "b.zarr") + + cubes = iris.load(str(tmp_path / "*")) + names = sorted(cube.var_name or cube.name() for cube in cubes) + assert "air_temperature" in names + assert len(cubes) == 2 + + +def test_find_saver_tolerates_a_trailing_separator(): + from iris.io import find_saver + + assert find_saver("out.nc") is find_saver("out.nc/") + + +def test_the_cubelist_saver_set_names_the_netcdf_saver(): + """The substring test it replaces matched by accident of module naming. + + `"iris.fileformats.netcdf" not in saver.__module__` is true of any module + whose path happens to contain that string, and says nothing a reader can + check. The set is one line per format. + """ + from iris.fileformats.netcdf import save as netcdf_save + from iris.io import _CUBELIST_SAVER_MODULES + + assert netcdf_save.__module__ in _CUBELIST_SAVER_MODULES + + +@_shared_utils.skip_data +def test_saving_a_cubelist_to_netcdf_still_takes_one_call(tmp_path): + cubes = iris.load( + _shared_utils.get_data_path( + ("NetCDF", "global", "xyt", "SMALL_hires_wind_u_for_ipcc4.nc") + ) + ) + iris.save(cubes, str(tmp_path / "out.nc")) + assert len(iris.load(str(tmp_path / "out.nc"))) == len(cubes) + + +def test_load_meshes_skips_a_zarr_store_rather_than_raising(tmp_path): + """Reaching a Zarr store is new; building a mesh from one is not. + + What must not happen is the `ValueError` the scheme dispatch raised for + anything that was not a file. + """ + from iris.mesh import load_meshes + + stores.make_store(tmp_path / "s.zarr") + assert load_meshes(str(tmp_path / "s.zarr")) == {} From 4b63e0e5f29eec897692a7d9a34045d60108a625 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 19:24:23 +0000 Subject: [PATCH 32/56] Update the known-formats snapshot for the registered Zarr spec test_known_formats compares iris.fileformats.FORMAT_AGENT's registered specs against this snapshot. Registering the "Zarr" FormatSpecification added an entry the snapshot never gained, because the sweep that landed it covered lib/iris/tests/unit/io/ and lib/iris/tests/unit/fileformats/, and this file lives outside both. One line, in the alphabetical position FormatSpecification._sort_key already puts it: after the other priority-5 entries and before the first priority-4 one. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/tests/results/file_load/known_loaders.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/iris/tests/results/file_load/known_loaders.txt b/lib/iris/tests/results/file_load/known_loaders.txt index 51824c5273..7c3ee89ec6 100644 --- a/lib/iris/tests/results/file_load/known_loaders.txt +++ b/lib/iris/tests/results/file_load/known_loaders.txt @@ -4,6 +4,7 @@ * NetCDF 64 bit offset format (priority 5) * NetCDF_v4 (priority 5) * UM Post Processing file (PP) (priority 5) + * Zarr (priority 5) * NetCDF dataset (priority 4) * UM Fieldsfile (FF) post v5.2 (priority 4) * ABF (priority 3) From 60088c3643eb54e5e67c0bb75b23bd8c46c6d4f8 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 19:46:52 +0000 Subject: [PATCH 33/56] Restore load_files's public contract, and guard the dispatch split load_files is public API with no leading underscore and a documented promise that filenames may be globs; narrowing that silently to already-expanded paths broke the promise for any future caller. Its body moves to a private _load_files that expects pre-expanded paths - which is what the duplicate-collapsing fix actually needed - and public load_files goes back to expanding once, then delegating. _generate_cubes, the only caller, now calls _load_files directly with its own already-expanded, partitioned list, so the double-expansion bug stays fixed. Cross-reference comments at both independent file-or-store checks (loading.py's _generate_cubes and ugrid_load.py's load_meshes) name each other, so an edit to one is more likely to carry to the other - the duplication itself is intentional and out of scope here. The remote-URL dispatch test no longer calls iris.load against a real S3 URL: with s3fs installed (a common transitive Dask dependency) that would issue a genuine outbound request. It now mocks load_stores and asserts the URL reaches it unmangled, which is the actual claim under test, and was verified to fail under a reintroduced mangling bug. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/fileformats/netcdf/ugrid_load.py | 3 ++ lib/iris/io/__init__.py | 24 +++++++---- lib/iris/loading.py | 7 +++- .../tests/unit/io/test__generate_cubes.py | 8 ++-- lib/iris/tests/unit/io/test_load_stores.py | 40 ++++++++++++++----- 5 files changed, 60 insertions(+), 22 deletions(-) diff --git a/lib/iris/fileformats/netcdf/ugrid_load.py b/lib/iris/fileformats/netcdf/ugrid_load.py index 34d9419081..cb32e99e6e 100644 --- a/lib/iris/fileformats/netcdf/ugrid_load.py +++ b/lib/iris/fileformats/netcdf/ugrid_load.py @@ -181,6 +181,9 @@ def _categorise(decoded: tuple[str, str, str | None]) -> tuple[str, str, str]: raise ValueError(message) for source in sources: + # iris.loading._generate_cubes makes this same file-or-store + # distinction independently, for general cube loading - keep + # both in step. if category == "file" and not Path(source).is_dir(): with open(source, "rb") as fh: handling_format_spec = FORMAT_AGENT.get_spec(Path(source).name, fh) diff --git a/lib/iris/io/__init__.py b/lib/iris/io/__init__.py index b99d8bf50c..567a06aee0 100644 --- a/lib/iris/io/__init__.py +++ b/lib/iris/io/__init__.py @@ -252,8 +252,8 @@ def expand_filespecs(file_specs, files_expected=True): def load_files(filenames, callback, constraints=None): """Create a generator of Cubes from given files. - Take a list of filenames, already expanded by :func:`expand_filespecs`, - and optionally a constraint set and a callback function, and returns a + Take a list of filenames which may also be globs, and optionally a + constraint set and a callback function, and returns a generator of Cubes from the given files. Notes @@ -261,11 +261,21 @@ def load_files(filenames, callback, constraints=None): Typically, this function should not be called directly; instead, the intended interface for loading is :func:`iris.load`. - `filenames` is not re-expanded here: a glob matching the same file twice, - by two different patterns, must reach the format handler as two entries, - and :func:`expand_filespecs` keys its own glob cache by the spec string, - so expanding an already-expanded list collapses two identical paths into - one. + """ + yield from _load_files(expand_filespecs(filenames), callback, constraints) + + +def _load_files(filenames, callback, constraints=None): + """Create a generator of Cubes from files already expanded by `expand_filespecs`. + + `filenames` is not expanded here: a glob matching the same file twice, by + two different patterns, must reach the format handler as two entries, and + :func:`expand_filespecs` keys its own glob cache by the spec string, so + expanding an already-expanded list collapses two identical paths into + one. :func:`load_files` is the public entry point that expands once and + delegates here; :func:`iris.loading._generate_cubes` is the other caller, + expanding once for itself so it can partition files from stores before + either sees the list. """ from iris.fileformats import FORMAT_AGENT diff --git a/lib/iris/loading.py b/lib/iris/loading.py index fc8d2e6572..d80abbaa62 100644 --- a/lib/iris/loading.py +++ b/lib/iris/loading.py @@ -101,11 +101,16 @@ def _categorise(decoded: tuple[str, str, str | None]) -> tuple[str, str, str]: # The file-or-store decision has to happen after glob expansion # and before format dispatch, because a glob may match both. # Neither routine below learns about the other's kind of target. + # iris.fileformats.netcdf.ugrid_load.load_meshes makes this same + # distinction independently, for mesh loading - keep both in step. expanded = iris.io.expand_filespecs(part_names) files = [name for name in expanded if not Path(name).is_dir()] stores = [name for name in expanded if Path(name).is_dir()] if files: - for cube in iris.io.load_files(files, callback, constraints): + # `expanded` is already the result of `expand_filespecs`, so + # the private, non-expanding `_load_files` is used here - see + # its docstring for why expanding twice would be wrong. + for cube in iris.io._load_files(files, callback, constraints): yield cube if stores: for cube in iris.io.load_stores(stores, callback, constraints): diff --git a/lib/iris/tests/unit/io/test__generate_cubes.py b/lib/iris/tests/unit/io/test__generate_cubes.py index d8ff2ec08d..b207955a83 100644 --- a/lib/iris/tests/unit/io/test__generate_cubes.py +++ b/lib/iris/tests/unit/io/test__generate_cubes.py @@ -25,7 +25,7 @@ def test_pathlib_paths(self, mocker): # `decode_uri` always reporting a `None` part is not a real path, so # the file-or-store partition has nothing to resolve or classify. mocker.patch("iris.iris.io.expand_filespecs", return_value=[]) - mocker.patch("iris.iris.io.load_files") + mocker.patch("iris.iris.io._load_files") for gc_arg, du_arg in test_variants: decode_uri_mock.reset_mock() @@ -41,7 +41,7 @@ def test_pathlib_paths(self, mocker): ], ) def test_nczarr_uri(self, uri, mocker): - load_files_mock = mocker.patch("iris.iris.io.load_files", return_value=[]) + load_files_mock = mocker.patch("iris.iris.io._load_files", return_value=[]) load_http_mock = mocker.patch("iris.iris.io.load_http", return_value=[]) list(_generate_cubes(uri, None, None)) @@ -54,7 +54,9 @@ def test_non_nczarr_uri(self, mocker): # The partition needs a real-looking part name to classify, not a # file-system lookup against a path that was never written. mocker.patch("iris.iris.io.expand_filespecs", side_effect=lambda names: names) - load_files_mock = mocker.patch("iris.iris.io.load_files", return_value=[]) + # The "file" branch calls the private, non-expanding `_load_files` + # directly, since it has already expanded - see its docstring. + load_files_mock = mocker.patch("iris.iris.io._load_files", return_value=[]) load_http_mock = mocker.patch("iris.iris.io.load_http", return_value=[]) list(_generate_cubes(uri, None, None)) diff --git a/lib/iris/tests/unit/io/test_load_stores.py b/lib/iris/tests/unit/io/test_load_stores.py index 20f31cf3c8..1128f670c1 100644 --- a/lib/iris/tests/unit/io/test_load_stores.py +++ b/lib/iris/tests/unit/io/test_load_stores.py @@ -65,20 +65,17 @@ def test_iris_load_accepts_a_local_store(tmp_path): np.testing.assert_array_equal(cubes[0].data, stores.DATA) -def test_iris_load_accepts_a_remote_url_and_does_not_absolutise_it(): - """A remote URL must reach zarr-python intact. +def test_iris_load_accepts_a_remote_url_and_does_not_absolutise_it(mocker): + """A remote URL must reach `load_stores` unmangled, not resolved as a local path. `expand_filespecs` would otherwise turn `s3://bucket/data.zarr` into - `//s3:/bucket/data.zarr`. No network is touched: opening the store - fails, and what matters is that the message still names the URL. + `//s3:/bucket/data.zarr`. The dispatch is what is under test, so + `load_stores` itself is mocked out: no store library is invoked, and no + network is touched, whatever is installed alongside zarr-python. """ - # The failure mode depends on what is installed to talk to S3, so no - # single exception type or message is guaranteed - checked below instead. - with pytest.raises(Exception) as caught: # noqa: PT011 - iris.load("s3://bucket/data.zarr") - assert "s3://bucket/data.zarr" in str(caught.value) or isinstance( - caught.value, ImportError - ) + mock_load_stores = mocker.patch("iris.io.load_stores", return_value=iter(())) + iris.load("s3://bucket/data.zarr") + mock_load_stores.assert_called_once_with(["s3://bucket/data.zarr"], None, None) @_shared_utils.skip_data @@ -104,6 +101,27 @@ def test_a_glob_matching_both_files_and_stores(tmp_path): assert len(cubes) == 2 +def test_load_files_still_expands_a_glob(tmp_path, mocker): + """Public `load_files` keeps its documented promise: filenames may be globs. + + The duplicate-collapsing bug fixed alongside `load_stores` lives in + calling `expand_filespecs` twice, once in a caller and again inside + `load_files` - not in `load_files` expanding at all. Pinning this + guards against quietly narrowing the public contract while fixing that. + """ + from iris.io import load_files + + (tmp_path / "a.nc").write_bytes(b"") + (tmp_path / "b.nc").write_bytes(b"") + mock_load = mocker.patch("iris.io._load_files", return_value=iter(())) + + list(load_files([str(tmp_path / "*.nc")], None)) + + mock_load.assert_called_once_with( + [str(tmp_path / "a.nc"), str(tmp_path / "b.nc")], None, None + ) + + def test_find_saver_tolerates_a_trailing_separator(): from iris.io import find_saver From e209e64257cd05e97eaee4ceb8f7e3a03aca8263 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 13 Oct 2026 20:07:30 +0000 Subject: [PATCH 34/56] Load realistic Zarr stores end to end, both versions Version 2 comes from Iris's own NCZarr writer, so the native reader and netCDF-c are checked against each other. Version 3 comes from a fixture shaped like the NOAA GFS archive: sharded, packed, with a scalar grid mapping, a two-dimensional auxiliary coordinate and a base64 fill value. Co-Authored-By: Claude Opus 5 Session-Id: 7a880931-8be4-4c5c-a2cd-afa8e486f9e4 --- lib/iris/tests/integration/zarr/__init__.py | 12 ++ lib/iris/tests/integration/zarr/fixtures.py | 116 ++++++++++++ lib/iris/tests/integration/zarr/test_load.py | 181 +++++++++++++++++++ 3 files changed, 309 insertions(+) create mode 100644 lib/iris/tests/integration/zarr/__init__.py create mode 100644 lib/iris/tests/integration/zarr/fixtures.py create mode 100644 lib/iris/tests/integration/zarr/test_load.py diff --git a/lib/iris/tests/integration/zarr/__init__.py b/lib/iris/tests/integration/zarr/__init__.py new file mode 100644 index 0000000000..b7761f385a --- /dev/null +++ b/lib/iris/tests/integration/zarr/__init__.py @@ -0,0 +1,12 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Integration tests for native Zarr loading. + +Skipped when zarr-python is absent, which is also how the unit tests do it. +""" + +import pytest + +pytest.importorskip("zarr", minversion="3.0.8") diff --git a/lib/iris/tests/integration/zarr/fixtures.py b/lib/iris/tests/integration/zarr/fixtures.py new file mode 100644 index 0000000000..551f9f088a --- /dev/null +++ b/lib/iris/tests/integration/zarr/fixtures.py @@ -0,0 +1,116 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""A version 3 store carrying the CF features a real one does. + +Modelled on the NOAA GFS archive at dynamical.org, whose properties the Zarr +I/O design names as the ones worth exercising: version 3 `dimension_names`, a +sharded codec, a scalar grid-mapping variable, a two-dimensional auxiliary +coordinate, and the base64 `_FillValue` that released xarray writes for every +floating dtype. A cut-down copy of the real store replaces this. +""" + +import base64 +import struct + +import numpy as np +import zarr + +SHAPE = (2, 4, 6, 5) +DIMENSIONS = ("time", "level", "latitude", "longitude") + + +def float_fill_value(value): + """Return `value` in the encoding released xarray writes for a float.""" + return base64.b64encode(struct.pack("