diff --git a/.github/WRITING.md b/.github/WRITING.md
index 17b65a30d2..55e01d461d 100644
--- a/.github/WRITING.md
+++ b/.github/WRITING.md
@@ -80,7 +80,8 @@ never fixed at all.
- **Never cite what decays silently**: a line number in another file, a count
of call sites, a time relative to writing ("currently", "this PR", "the new
behaviour"). Nothing checks any of them, and your "now" is never the
- reader's.
+ reader's. However, user-facing messages such as warnings/errors remain
+ appropriate places for phrases such as "currently" or "in future".
- **Never cite an artefact of how the work was done**: a review finding, a
task or plan number, a phase, a checklist item. These resolve to nothing for
a later reader. Cite an issue, a pull request, a design document, or a
diff --git a/.hooks/check_docs_page_metadata.py b/.hooks/check_docs_page_metadata.py
index ad733275b8..3b54912a72 100644
--- a/.hooks/check_docs_page_metadata.py
+++ b/.hooks/check_docs_page_metadata.py
@@ -221,8 +221,10 @@ def check_docstring(text: str, expected_type: str) -> list[str]:
if docstring is None:
return [
- "Missing a module docstring, so the page generated from it has no "
- "sphinx-needs item."
+ (
+ "Missing a module docstring, so the page generated from it has "
+ "no sphinx-needs item."
+ )
]
return check_item(docstring, expected_type)
diff --git a/docs/src/user_manual/explanation/zarr_io.rst b/docs/src/user_manual/explanation/zarr_io.rst
index 76efe4374f..a1c956bb19 100644
--- a/docs/src/user_manual/explanation/zarr_io.rst
+++ b/docs/src/user_manual/explanation/zarr_io.rst
@@ -1,7 +1,7 @@
.. explanation:: Zarr I/O Handling in Iris
:tags: topic_load_save
- Read about how Iris loads Zarr stores.
+ Read about how Iris loads and saves Zarr stores.
.. include:: ../../common_links.inc
@@ -19,9 +19,9 @@ version 3 stores natively, through
`zarr-python `_.
``zarr`` is an **optional dependency**, and Iris requires at least version
-3.0.8. Install it to load Zarr stores::
+3.1.0. Install it to load Zarr stores::
- conda install -c conda-forge "zarr>=3.0.8"
+ conda install -c conda-forge "zarr>=3.1.0"
Iris without it installs and runs normally, raising an ``ImportError`` that
names the package and its minimum required version if you ask it to load a
@@ -44,16 +44,122 @@ A store holding UGRID mesh data loads as a mesh cube, the same as the
equivalent netCDF file would. :func:`iris.mesh.load_meshes` reads a store
too.
+.. note::
+
+ A Zarr attribute may be any JSON value, so a cube loaded from a store
+ may carry one that netCDF cannot hold. Saving that cube as netCDF fails.
+ See :ref:`zarr_io_attributes`.
+
Saving
------
-Iris does not save to a native Zarr store. ``iris.save(cube, "out.zarr")``
-raises ``ValueError: Cannot save; no saver``, naming neither Zarr nor an
-alternative. Save to netCDF instead, or write Zarr through netCDF-c's NCZarr
-driver, covered below in :ref:`zarr_io_nczarr`.
+A store is a directory, or a URL, so pass it where you would pass a file::
+
+ iris.save(cubes, "/data/forecast.zarr")
+
+Iris writes Zarr **version 3** stores. A whole :class:`~iris.cube.CubeList`
+goes into one store, which is opened once. Saving through netCDF-c's NCZarr
+driver writes **version 2** instead, covered below in :ref:`zarr_io_nczarr`.
+
+.. warning::
+
+ A save that is interrupted leaves a partial store behind, and Iris cannot
+ roll it back. The damage is quiet -- see
+ :ref:`zarr_io_interrupted_save`.
+
+Unlike the netCDF saver, this one does **not** overwrite. ``iris.save`` raises
+if the target already exists, because replacing a store means deleting many
+objects one at a time, and an interrupted replacement destroys what was there
+without completing what replaces it. Say so explicitly to opt in::
+
+ iris.save(cubes, "/data/forecast.zarr", mode="w") # replace
+ iris.save(more_cubes, "/data/forecast.zarr", mode="a") # add what is not there
+
+``mode="a"`` adds a *new* array or group to a store that already holds others.
+It never extends, rewrites, resizes or re-chunks an array that is already
+there. Saving a cube whose own data variable already exists in the target
+group is an error rather than an overwrite, so saving the same cube twice with
+``mode="a"`` raises rather than duplicating it. Appending to data is not
+supported, for Zarr or for netCDF: deciding whether incoming data is
+compatible with what a store already holds is a data-model problem, and the
+storage format does not solve it.
+
+Everything else a cube carries -- its coordinates, bounds and cell measures --
+is not matched against what the store already holds. Adding a cube to a store
+that already carries a ``latitude`` array writes a second one, named
+``latitude_0``, rather than reusing the first. Deciding whether two
+coordinates are "the same" across unequal values, differing units and
+different calendars is its own piece of work, so Iris does not attempt it.
+
+Storage keywords
+^^^^^^^^^^^^^^^^
+
+The Zarr saver takes **zarr-python's** vocabulary, not netCDF's::
+
+ iris.save(
+ cube,
+ "out.zarr",
+ chunks=(1, 50, 50),
+ shards=(4, 100, 100),
+ compressors=[zarr.codecs.BloscCodec(cname="zstd")],
+ )
+
+``chunks``, ``shards``, ``compressors``, ``filters`` and ``serializer`` are
+passed through as zarr-python defines them. A netCDF keyword such as
+``zlib=True`` raises :class:`TypeError` naming the Zarr replacement where there
+is one, and saying there is none where there is not. ``order`` raises too: it
+is zarr-python's own keyword, but it has no effect on a version 3 array.
+
+For a cube holding **lazy** data, ``chunks`` defaults to the cube's own lazy
+chunking. Real data has no chunking of its own to inherit, and gets
+zarr-python's automatic chunking instead. Passing ``chunks`` explicitly
+overrides either case. The default is also dropped when you pass ``shards``: a
+shard holds a whole number of inner chunks, and zarr-python picks the inner
+chunk in that case.
+
+Deferred saving
+^^^^^^^^^^^^^^^
+
+``compute=False`` returns a :class:`dask.delayed.Delayed` that writes the lazy
+data when computed, and consolidates the store's metadata after the last chunk::
+
+ result = iris.save(cubes, "deferred.zarr", compute=False)
+ result.compute()
-A cube loaded from Zarr may carry an attribute netCDF cannot hold, which
-fails the save. See :ref:`zarr_io_attributes`.
+Consolidated metadata
+^^^^^^^^^^^^^^^^^^^^^
+
+Iris writes a consolidated metadata document at the store's root when the save
+completes, which makes a remote store usable: a reader fetches one object
+instead of one per array. Pass ``consolidate_metadata=False`` to decline.
+
+When Iris adds to a store that already carries one, it rewrites it -- or, if
+you declined consolidation for that save, removes it. A consolidated document
+that does not mention what was just written is what a default reader believes,
+so a stale one hides data; having none at all is slower to read but tells the
+truth.
+
+.. _zarr_io_interrupted_save:
+
+What an interrupted save leaves behind
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An interrupted save leaves a partial store, and Iris offers no rollback. Zarr
+has no multi-object transaction: a store's metadata and its chunks are separate
+objects, so a save interrupted between them leaves the two disagreeing, and a
+missing chunk is not an error -- it reads back as the fill value, with no
+warning. The usual mitigation, writing to a temporary name and renaming on
+success, is not adopted: a directory rename is atomic on a local filesystem but
+object stores have no equivalent, so the guarantee would evaporate exactly
+where Zarr is most used. Callers who need transactional writes want
+`Icechunk `_.
+
+Meshes
+^^^^^^
+
+A cube with an unstructured :class:`~iris.mesh.MeshXY` saves correctly: the
+store carries the full CF-UGRID structure, the same mesh, topology and
+connectivity variables a netCDF save would write.
Remote stores
-------------
@@ -87,6 +193,16 @@ To read another group, call the Zarr loader directly::
)
)
+Saving takes a group more directly: ``iris.save`` itself takes a ``group=``
+keyword, naming a ``/``-separated path within the store::
+
+ iris.save(cubes, "/data/sentinel.zarr", group="measurements/reflectance/r10m")
+
+Passing ``group=`` implies ``mode="a"``, because writing a group into a store
+means the store may already hold others. ``iris.load`` and ``iris.load_cube``
+have no equivalent keyword: reading a group other than the root means calling
+the Zarr loader directly, as above.
+
Dimension names
---------------
@@ -129,6 +245,20 @@ That one case can mask a genuine zero, so Iris warns when it happens, naming
the variable. Setting a ``_FillValue`` attribute on the array prevents both
the masking and the warning.
+On save, Iris writes the missing-value declaration as well as the storage fill:
+a masked cube gets a ``_FillValue`` attribute even when it carried none of its
+own, because on version 3 the attribute is what declares missing data and the
+storage ``fill_value`` is not. Without it, every masked point would read back
+unmasked. Pass ``fill_value=`` to choose the sentinel; the default is the same
+one netCDF uses for that dtype.
+
+A floating ``_FillValue`` is written as the base64 of a little-endian
+``float64``, which is what xarray and the wider Zarr ecosystem read. This
+deviates from CF section 2.5.1, which asks for a value of the variable's own
+type. The deviation is deliberate and load-bearing: a plain JSON number there
+makes the whole store unreadable by default xarray, which is a worse outcome
+than a convention deviation that every reader understands.
+
Chunking
--------
diff --git a/docs/src/user_manual/tutorial/s3_io.rst b/docs/src/user_manual/tutorial/s3_io.rst
index c03d78de7e..35b22078c8 100644
--- a/docs/src/user_manual/tutorial/s3_io.rst
+++ b/docs/src/user_manual/tutorial/s3_io.rst
@@ -2,7 +2,8 @@
:tags: topic_load_save
A lesson on using s3-fuse with Iris to load/save netCDF, PP and GRIB
- data from/to S3 buckets, and on reading a Zarr store from S3 directly.
+ data from/to S3 buckets, and on loading from and saving to a Zarr store
+ on S3 directly.
.. _s3_io:
@@ -33,6 +34,15 @@ the objects it needs over the network::
That needs ``s3fs`` installed and configured, and no mount. If your data is in
Zarr, prefer this to the FUSE approach described below.
+Saving works the same way::
+
+ iris.save(cubes, "s3://bucket/forecast.zarr")
+
+The same store library and credentials apply, and the same defaults: an
+existing store is not overwritten unless you pass ``mode="w"``, and the
+metadata is consolidated when the save completes, which is what makes the store
+cheap to open again over the network.
+
Other filetypes via a virtual file system
-----------------------------------------
diff --git a/docs/superpowers/plans/2026-10-15-zarr-io-pr6.md b/docs/superpowers/plans/2026-10-15-zarr-io-pr6.md
new file mode 100644
index 0000000000..946512256b
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-15-zarr-io-pr6.md
@@ -0,0 +1,4510 @@
+# Zarr I/O PR 6 — Zarr saving: Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+| | |
+|---|---|
+| **Spec** | [`docs/superpowers/specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md) §4.1, §4.2, §4.5, §4.6, §4.7, §5 (PR 6), §6 |
+| **Implements** | Pull request 6 of 8, the subject of [#6980](https://github.com/SciTools/iris/issues/6980). `Part of #6980` — never a closing keyword. |
+| **Branch** | `zarr-io-pr-6`, branched from `zarr-io-pr-4-5-base`, targeting `brownfield` |
+| **Baseline** | `52c478bf9`, PR 4 ([#7336](https://github.com/SciTools/iris/pull/7336)) and PR 5 ([#7338](https://github.com/SciTools/iris/pull/7338)) merged together on a local branch, neither yet in `brownfield`; Iris `3.17.0.dev16`; zarr `3.4.1`; xarray `2026.9.0`; dask `2026.8.0`; numpy `2.5.2` |
+| **Delivers** | `iris.save(cubes, "out.zarr")` and `iris.save(cubes, "s3://bucket/out.zarr")`, writing Zarr version 3 stores that xarray and every other Zarr reader can open |
+| **Behaviour change** | New capability only. `iris.save(cube, "out.zarr")` raises `ValueError("Cannot save; no saver")` at the baseline, so nothing that works today changes meaning. The one edit to existing netCDF behaviour is a hook that leaves what lands in a netCDF file byte for byte unchanged (F2). |
+| **Public API change** | `iris.fileformats.zarr.save` and `iris.fileformats.zarr.Saver` are new. `iris.fileformats.netcdf._attribute_handlers` moves to `iris.fileformats.cf._attribute_handlers`; it is private and has no deprecation obligation (F3). |
+
+**Goal:** Make `iris.save(cubes, "out.zarr")` write a CF-conforming Zarr version 3 store, by adding a write path to the `ZarrDataset` PR 4 built and a `Saver` subclassing the `CFSaver` PR 5 extracted.
+
+**Architecture:** PR 5 left `cf/saver.py` holding every format-agnostic part of saving a cube — dimension naming, coordinate, bounds, mesh, cell-measure and grid-mapping variables, attribute compliance, lazy write collection — driven through the `CFDataset` interface and four hooks. PR 4 left `zarr/_dataset.py` holding a read-only `ZarrDataset` whose write members raise `NotImplementedError`. This plan fills in those members, adds a fifth hook for grid mappings so that nothing in generic code reaches for a netCDF4 object, and adds `zarr/saver.py`: a `Saver` that opens a store, a `save()` that takes Zarr's own storage vocabulary, and a `delayed_completion()` that returns a real `Delayed` owning the final metadata consolidation.
+
+**Tech Stack:** Python 3.12–3.14, zarr-python ≥3.0.8, NumPy, Dask, xarray (test-only, for cross-reader conformance), pytest + pytest-mock, pre-commit (ruff, numpydoc, mypy, sort-all, and the repository's local hooks), Sphinx.
+
+---
+
+## Global Constraints
+
+Every task's requirements implicitly include this section.
+
+| Constraint | Value, verbatim from its source |
+|---|---|
+| Target branch | `brownfield`, not `main` (spec §5) |
+| PR labels | `Agentic` and `Type: Feature Branch` (spec §5) |
+| Issue reference | `Part of #6980`. **Never** a closing keyword — the issue closes at merge-back (spec §5) |
+| Changelog | No `changelog/` fragment in this PR. Fragments are written once at merge-back; the PR body must say so (spec §5) |
+| GitHub posts | Every PR description, comment and review reply opens with `(posted by Claude Opus)`, before anything else. This **replaces** any trailing harness attribution (root `AGENTS.md`) |
+| Commit messages | Out of scope for the above; they keep `Co-Authored-By:` trailers |
+| Zarr version written | **Version 3 only.** "`ZarrDataset.__init__` takes `zarr_format: int = 3`. Today it raises `NotImplementedError` for anything but 3, in one place" (spec §4.5) |
+| No version in generic code | "`cf/saver.py` never mentions a Zarr version. Everything version-specific lives in `ZarrDataset`" (spec §4.5) |
+| Zarr's vocabulary, not netCDF's | "`compressors=`, `chunks=`, `shards=`, `filters=`, `serializer=` — the names zarr-python uses — plus `consolidate_metadata=True`. A netCDF keyword passed to the Zarr saver raises `TypeError` naming the Zarr equivalent where there is one, and saying there is none where there is not" (spec §4.5) |
+| No keyword translation | "**Do not translate netCDF keywords onto Zarr codecs** — `zlib`/`complevel` onto Blosc or Gzip, `shuffle` onto Blosc shuffle, `chunksizes` onto `chunks`, `fletcher32` onto Crc32c" (spec §4.5) |
+| Write-alignment invariant | "Every Zarr array Iris writes has a *write grid* that the lazy source's dask chunking tiles exactly, where the write grid is `Array.shards` when the array is sharded and `Array.chunks` otherwise. Each concurrent task owns at least one whole write-grid cell; no cell is touched by two tasks. A truncated final cell is permitted" (spec §4.5) |
+| Create-once invariant | "Array metadata is written when the array is created and never edited afterwards. Iris does not resize, rechunk, or change the dtype, `fill_value`, codecs or dimension names of an array that already exists" (spec §4.5) |
+| Default mode | "`"w-"`: create, and raise if the target already exists." "`mode="w"` opts back in to replacement; `mode="a"` adds to an existing store and is what `group=` needs. The error message names `mode`" (spec §4.5) |
+| No append | "It never extends, rewrites, resizes or re-chunks an array that is already there... There is no `mode="a+"` and no `append=` keyword" (spec §4.5) |
+| Attributes are JSON-native | "NumPy scalar to its Python equivalent, NumPy array to a list, `bytes` to `str` via ASCII, everything else unchanged. Anything still unserialisable raises with the attribute name in the message" (spec §4.5) |
+| Converted before handing over | "Values are converted to JSON-native types **before** being handed to zarr, never after, because a rejected value poisons the attributes mapping" (spec §4.5, reproduced as F6) |
+| No stringify-and-warn | "An attribute the format cannot hold is an error, and is not quietly turned into a string" (spec §4.5) |
+| `_FillValue` encoding | floating: "base64 of `struct.pack("=3.0.8`, in `requirements/py3{12,13,14}.yml` and in `_MINIMUM_VERSION` (PR 4) |
+| netCDF4 imports | No direct `import netCDF4`; always `iris.fileformats.netcdf._thread_safe_nc` (root `AGENTS.md`, enforced by the `check-netcdf4-imports` hook) |
+| Copyright header | Every new Python file opens with the four-line BSD header (root `AGENTS.md`) |
+| Line length | 88 characters (Ruff default) |
+| Docstrings | NumPy style, strictly validated by the `numpydoc-validation` hook |
+| Public module docstrings | Must open with a `z_reference` item within the first 25 lines (`lib/iris/AGENTS.md`, enforced by `check-docs-page-metadata`) |
+| New module size | Aim under ~1000 lines (`lib/iris/AGENTS.md`). F4 is why `_encode.py` exists rather than more of `_dataset.py` |
+| No network in tests | "**Remote stores** are covered by an in-memory store and a local directory store only. No test touches the network. The URL path is exercised by checking that the right store is constructed, not by fetching" (spec §6) |
+| Test skipping | "A single `pytest.importorskip("zarr")` fixture in the Zarr test package. Iris without zarr installed must show no new failures" (spec §6) |
+| Published prose | "must never cite an artefact of how the work was done — a review finding, a task or plan number, a phase" (`docs/AGENTS.md`). Cite the issue, the pull request or the design-document section instead |
+| Warnings | Every `UserWarning` subclass inherits from `IrisUserWarning`; the `check-categorised-warnings` hook enforces categorisation |
+| Before finishing | Run the repository's pre-commit hooks and fix everything they report (root `AGENTS.md`) |
+
+### Running anything
+
+The environment is the conda environment `iris-dev`. Either activate it, or prefix
+commands with `conda run -n iris-dev`. Do **not** install or remove packages
+without asking the user first — `xarray` is already installed there for this work
+(F5).
+
+Set `NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX"` before running the
+suite, per the root `AGENTS.md`.
+
+## Review Focus
+
+Five things the spec implies, that no task's own tests would have exercised,
+and that a person using this would meet in ordinary work. Each is pinned by a
+test in the task named — they were added to those tasks for this reason, not
+found there.
+
+1. **A cube with a mesh.** Iris supports unstructured meshes, PR 4 taught
+ `iris.mesh.load_meshes` about stores, and `CFSaver.write` writes mesh
+ variables through the same generic path as everything else. Nothing in the
+ Zarr work exercises that path: a mesh brings connectivity variables with
+ integer `start_index` attributes and dimensions no coordinate spans, which
+ is where a dimension-name or attribute-encoding mistake would surface. →
+ **Task 10, Step 1b**: `test_a_mesh_cube_round_trips` saves
+ `realistic_4d_w_everything(w_mesh=True)` and checks the mesh comes back.
+2. **Packed data.** `save(..., packing="i2")` narrows the dtype and writes
+ `scale_factor` and `add_offset`, which are numbers the JSON conversion
+ touches on the way out and the decode layer reads on the way back. Get the
+ conversion wrong and the values are quietly rescaled, not refused. →
+ **Task 10, Step 1b**: `test_packed_data_round_trips` compares the values
+ through the round trip and asserts the stored dtype.
+3. **`int64` data.** The one dtype the saver has narrowing logic for, because
+ the restricted netCDF formats cannot hold it. A Zarr store can, so
+ `supports_dtype` answers `True` and nothing is narrowed — a different
+ answer from netCDF's, taken by code written for netCDF. → **Task 10,
+ Step 1b**: `test_int64_data_survives`.
+4. **A non-ASCII attribute value.** A degree sign in a `title` is ordinary.
+ netCDF coerces ASCII to bytes and the Zarr path must not inherit that
+ instinct: JSON is text, and `json_native` passes a `str` through untouched
+ while refusing non-ASCII *bytes*. The failure would be a save that raises on
+ a perfectly saveable cube. → **Task 3, Step 1**:
+ `test_a_non_ascii_string_is_kept`, and **Task 10, Step 1b**:
+ `test_a_non_ascii_attribute_survives`.
+5. **A dataless cube.** `Cube` has a dataless form, and
+ `_create_generic_cf_array_var` has a branch that picks its own dtype and
+ fill value for it — written for netCDF, and reached here with a store whose
+ fill handling is new. → **Task 10, Step 1b**:
+ `test_a_dataless_cube_saves`.
+
+---
+
+## 1. Scope boundary
+
+**In scope** (spec §5, PR 6):
+
+- `zarr/_dataset.py` gains the write half of `CFDataset`: opening for writing,
+ `create_dimension`, `create_variable`, writable attributes and `finalise`.
+- `zarr/_encode.py`, new: the pure value conversions the write path needs (F4).
+- `zarr/saver.py`, new: `Saver` and `save`.
+- The save chain: a `zarr` entry in `iris.io._savers`, and
+ `iris.fileformats.zarr.saver` added to `_CUBELIST_SAVER_MODULES` (F8).
+- One fifth hook on `CFSaver` so that grid-mapping parameters reach a store
+ that has no netCDF4 object behind it (F2).
+- `netcdf/_attribute_handlers.py` moves to `cf/_attribute_handlers.py` (F3).
+- `xarray` added to the test section of `requirements/py3{12,13,14}.yml` (F5).
+- Documentation: saving in `docs/src/user_manual/explanation/zarr_io.rst` and
+ `docs/src/user_manual/tutorial/s3_io.rst`, both of which PR 4 created or
+ rewrote for loading.
+- The §6 write tests: the write-alignment invariant on a sharded target, the
+ existing-store refusals, the Zarr-native encoding keywords, the masked
+ round trip, the deferred save, and the bidirectional cross-reader
+ conformance test.
+
+**Out of scope**, and each has a reason:
+
+- **Version 2 writing.** Spec §4.5: "Writing is version 3 only." The guard is
+ one `NotImplementedError` in `ZarrDataset.__init__`.
+- **Appending to an array.** Spec §4.5 and §9. `mode="a"` adds a *new* array or
+ group and nothing else.
+- **Multi-process writes.** Spec §4.5 and §9. The provisions that keep it
+ reachable are already in place; this plan adds none and removes none.
+- **Regularising the netCDF grid-mapping attribute encoding.** Q8, spec §12.3:
+ "belongs in its own pull request". F2 is how this plan unblocks Zarr without
+ touching what lands in a netCDF file.
+- **The netCDF `compute=False` return type.** Q5, [#7291](https://github.com/SciTools/iris/issues/7291).
+ The Zarr path returns a real `Delayed`; the netCDF path is left as it is.
+- **The two-dimensional attribute check on the netCDF saver.**
+ [#7329](https://github.com/SciTools/iris/issues/7329). Spec §4.5 puts it
+ outside this programme.
+- **A Zarr `save_mesh`.** Spec §5 asks for `save`, not for the mesh-only entry
+ point `iris.mesh.save_mesh` offers for netCDF. A cube that *has* a mesh saves
+ its mesh, because that is `CFSaver.write`'s job and it is backend-agnostic
+ already; what is missing is only the standalone "save this mesh and nothing
+ else" call. Adding it is a later, small pull request.
+- **Real-world data and benchmarks.** PR 7.
+- **Changelog fragments and closing keywords.** The merge-back owns both.
+- **Regenerating `requirements/locks/*.lock`.** F5: the human developer does
+ this by hand, after PR 4 lands.
+
+---
+
+## 2. Git strategy: a branch over two unmerged pull requests
+
+PR 6 needs both of its predecessors in one tree. It edits `zarr/_dataset.py`,
+which exists only on PR 4, and it subclasses `CFSaver`, which exists only on
+PR 5. Neither has merged.
+
+So `zarr-io-pr-4-5-base` carries a merge of the two, and `zarr-io-pr-6`
+branches from it. The merge commit, `52c478bf9`, resolved five conflicts and
+recorded two things neither branch could have had on its own; read it before
+starting, because one of them is a gap that will reappear:
+
+> `ZarrDataset.supports_dtype`, the abstract method PR 5 adds to `CFDataset`
+> after PR 4 wrote `ZarrDataset`. Both Zarr versions hold every NumPy numeric
+> dtype, so it answers `True`. **Whichever of PR 4 and PR 5 rebases second must
+> carry this**, or `ZarrDataset` stays abstract and every Zarr test errors.
+
+That method is therefore *not* part of PR 6's diff, even though PR 6's branch
+contains it. The same goes for every other line of the merge commit.
+
+**When PR 4 and PR 5 have both landed in `brownfield`:**
+
+```bash
+git fetch upstream brownfield
+git rebase --onto upstream/brownfield zarr-io-pr-4-5-base zarr-io-pr-6
+```
+
+That replays PR 6's own commits and drops the merge, which is why the merge is
+a separate branch rather than the first commit of this one. Expect conflicts
+only where review changed PR 4's `ZarrDataset` or PR 5's `CFSaver` hooks under
+this work.
+
+**Opening the pull request before that rebase is possible** means GitHub would
+show PR 4's and PR 5's diffs inside PR 6. Open it as a draft against
+`brownfield` only once at least one predecessor has landed, or open it against
+whichever predecessor branch is still outstanding and retarget later. The
+pull-request body says which it is.
+
+---
+
+## 3. Findings that refine the spec
+
+Numbered so that tasks can cite them. Each is something checked against the
+code or against zarr-python 3.4.1 and xarray 2026.9.0, not something the spec
+already says.
+
+**F1. The base is green, and the suite is the proof.** `11188 passed, 59
+skipped` across `lib/iris/tests/unit` and `lib/iris/tests/integration` on
+`52c478bf9`. Any failure outside the files a task touches is a rebase artefact,
+not a bug this plan introduced.
+
+**F2. Grid mappings need a fifth hook, and the body of the method does not
+change.** `CFSaver._add_grid_mapping_to_dataset` binds
+`grid_variable = cf_var_grid.variable` and then makes **63** plain attribute
+assignments to it. `.variable` is netCDF4's own object; `ZarrDatasetVariable`
+has no such member, so a Zarr save of any cube with a coord system raises
+`AttributeError` today. Q8 refuses to regularise the encoding in this
+programme, because routing those 63 through `.attributes` changes what lands in
+a netCDF file — `crs_wkt` is `NC_STRING` while `grid_mapping_name` is
+`NC_CHAR`. The resolution is one hook, `_grid_mapping_writer(cf_var)`, and one
+changed line in the 230-line method. netCDF returns `cf_var.variable`, so its
+output is unchanged byte for byte. The base returns an adapter whose
+`__setattr__` writes into `cf_var.attributes`. Decided with the user,
+2026-10-15.
+
+**F3. `_attribute_handlers` moves to `cf/`, in this pull request.** Spec §12.4
+assigns it to PR 6: `cf/saver.py` imports `ATTRIBUTE_HANDLERS` from the netCDF
+package, which imports `cf.saver` back, and the cycle survives only because
+`_attribute_handlers` imports nothing from `iris.fileformats.cf`. Two source
+files and four test files name it. It is private — no deprecation alias is
+owed. It is the first task, and its own commit, so that the relocation and the
+behaviour work stay separately reviewable. Decided with the user, 2026-10-15.
+
+**F4. `zarr/_encode.py` is a new module the spec's §4.1 layout does not list.**
+The write side needs the same kind of pure value conversion the read side has
+in `_decode.py`: attributes to JSON-native types, `_FillValue` to its
+dtype-dependent encoding, masked arrays to filled ones. Putting them in
+`_dataset.py` would push a module that is already 780 lines before any write
+code well past the ~1000-line aim in `lib/iris/AGENTS.md`, and would mix pure
+functions in with the zarr-python calls, which is what makes `_decode.py`
+unit-testable without a store. Decided with the user, 2026-10-15.
+
+**F5. xarray goes in the requirement files now; the lock files are the human
+developer's.** Spec §6 puts xarray in with PR 6. PR 4 already regenerates all
+three lock files to add `zarr`, so PR 6 regenerating them guarantees a conflict
+and a second slow rebuild on rebase. So: `xarray` is added to the test section
+of `requirements/py3{12,13,14}.yml` in this pull request, and
+`requirements/locks/*.lock` is left alone for the human developer to regenerate
+by hand once PR 4 has landed. The consequence to state in the pull-request
+body: **until those locks are regenerated, CI has no xarray, so every
+cross-reader conformance test skips.** It is installed in `iris-dev` — one
+package, exactly as spec §6 measured — so the tests run locally while the work
+is being done. Decided with the user, 2026-10-15.
+
+**F6. Every zarr-python and xarray fact the write path leans on reproduces on
+today's versions.** Checked against zarr 3.4.1 and xarray 2026.9.0,
+2026-10-15:
+
+| Claim | Result |
+|---|---|
+| `mode="w-"` on an existing store raises | `FileExistsError: Cannot create '' with mode 'w-' because it already contains data. Use mode 'w' to overwrite.` |
+| Creating an array over an existing name raises | `zarr.errors.ContainsArrayError: An array exists in store ... at path 'data'.` |
+| A non-JSON attribute value poisons the mapping | `TypeError: Object of type float32 is not JSON serializable`, **and** `dict(group.attrs)` afterwards still contains the rejected value |
+| A floating `_FillValue` written as a plain JSON number | xarray's open fails for the **whole dataset**: `TypeError: Failed to decode fill_value: expected str or bytes for dtype float32, got float` |
+| The same value written as base64 of `struct.pack(" None: ...
+ def create_variable(
+ self,
+ name: str,
+ dtype,
+ dimensions=(),
+ *,
+ fill_value=None,
+ chunks=None, # Zarr's own names, from here down
+ shards=None,
+ compressors="auto",
+ filters="auto",
+ serializer="auto",
+ **unsupported, # anything left raises TypeError
+ ) -> "ZarrDatasetVariable": ...
+ def finalise(self) -> None: ... # consolidate_metadata, once
+
+class ZarrDatasetVariable(CFDatasetVariable):
+ def __setitem__(self, keys, values) -> None: ... # encodes, then writes
+ def write_handle(self) -> "ZarrDatasetVariable": ... # self; F7
+
+
+# iris/fileformats/zarr/_encode.py
+
+def json_native(value, *, name: str): ... # NumPy scalar -> Python, array -> list,
+ # bytes -> ASCII str; TypeError names `name`
+def encode_fill_value(value, dtype): ... # base64 str for floating, int for integer
+def filled(data, fill_value): ... # masked -> plain, lazy stays lazy
+
+
+# iris/fileformats/zarr/saver.py
+
+class Saver(CFSaver):
+ def __init__(
+ self,
+ filename,
+ compute: bool = True,
+ *,
+ mode: str = "w-",
+ group: str | None = None,
+ consolidate_metadata: bool = True,
+ ): ...
+ def _create_string_variable(self, cube_or_mesh, cf_name, element, element_dims, data): ...
+ def delayed_completion(self) -> Delayed: ... # a real Delayed, owning finalise()
+
+def save(
+ cube, # Cube or CubeList; F8
+ filename,
+ *,
+ mode: str = "w-",
+ group: str | None = None,
+ consolidate_metadata: bool = True,
+ local_keys=None,
+ fill_value=None,
+ packing=None,
+ compute: bool = True,
+ **encoding, # chunks, shards, compressors, filters, serializer
+): ...
+# `unlimited_dimensions` is deliberately absent: it lands in **encoding and
+# gets the TypeError that names why Zarr has no such thing.
+
+
+# iris/fileformats/cf/saver.py
+
+class CFSaver:
+ def _grid_mapping_writer(self, cf_var): # F2; the fifth hook
+ """Return the object the 63 grid-mapping parameters are assigned to."""
+```
+
+---
+
+## Task 1: `_attribute_handlers` moves to `cf/`
+
+Spec §12.4, the PR 5 entry: "Relocating `_attribute_handlers` is PR 6's, to keep
+this relocation's overlap with the parallel Zarr loading work small." The module
+serialises the three Iris-only attribute types that *every* CF store needs —
+STASH, GRIB_PARAM, `ukmo__process_flags` — so its home is the CF package, and
+moving it closes the one two-way import `cf/saver.py` still has. No behaviour
+change: the module imports nothing from `iris.fileformats.cf`, and nothing it
+exports is public API.
+
+It is first because it is the one part of this pull request reviewable without
+reading a line of Zarr.
+
+**Files:**
+- Move: `lib/iris/fileformats/netcdf/_attribute_handlers.py` → `lib/iris/fileformats/cf/_attribute_handlers.py`
+- Move: `lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/` → `lib/iris/tests/unit/fileformats/cf/attribute_handlers/`
+- Modify: `lib/iris/fileformats/cf/saver.py:32-40` (module docstring), `:83` (the import)
+- Modify: `lib/iris/fileformats/_nc_load_rules/actions.py:552`
+- Modify: `lib/iris/tests/unit/fileformats/cf/test__imports.py`
+
+**Interfaces:**
+- Consumes: nothing.
+- Produces: `iris.fileformats.cf._attribute_handlers`, exporting
+ `AttributeHandler`, `StashHandler`, `UkmoProcessFlagsHandler`,
+ `GribParamHandler` and `ATTRIBUTE_HANDLERS`, all unchanged.
+ `iris.fileformats.netcdf._attribute_handlers` stops existing; it is private,
+ so spec §4.8 gives it no alias.
+
+- [ ] **Step 1: Move the tests, and repoint their imports**
+
+```bash
+git mv lib/iris/tests/unit/fileformats/netcdf/attribute_handlers \
+ lib/iris/tests/unit/fileformats/cf/attribute_handlers
+```
+
+In all three test modules, change the import:
+
+```python
+from iris.fileformats.cf._attribute_handlers import ATTRIBUTE_HANDLERS
+```
+
+and the docstring reference that names the old module, in each of
+`test_StashHandler.py`, `test_UkmoProcessFlagsHandler.py` and
+`test_GribParamHandler.py`:
+
+```python
+"""Unit tests for :class:`iris.fileformats.cf._attribute_handlers.StashHandler`."""
+```
+
+`test_GribParamHandler.py`'s docstring names `StashHandler` and always has;
+leave that alone, or fix it in passing — it is one word and nobody is served by
+a separate pull request for it.
+
+The package `__init__.py` docstring currently reads "the
+:mod:`iris.fileformats._attribute_handlers` module", which names neither home
+correctly. Make it:
+
+```python
+"""Unit tests for the :mod:`iris.fileformats.cf._attribute_handlers` module."""
+```
+
+- [ ] **Step 2: Run them, and watch them fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/attribute_handlers -x
+```
+
+Expected: collection error, `ModuleNotFoundError: No module named
+'iris.fileformats.cf._attribute_handlers'`.
+
+- [ ] **Step 3: Move the module**
+
+```bash
+git mv lib/iris/fileformats/netcdf/_attribute_handlers.py \
+ lib/iris/fileformats/cf/_attribute_handlers.py
+```
+
+Replace the first line of the module docstring, which calls these translations
+netCDF's. Nothing else in the file changes — not the classes, not the registry,
+not `from iris.fileformats.pp import STASH`:
+
+```python
+"""Attribute translations for Iris attributes with special convenience types.
+
+These are things which are stored differently in an Iris cube attribute from how they
+are actually stored in a file. E.G. a STASH code is stored as a special object,
+but in a file it is just a string.
+```
+
+Leave the rest of that docstring, including the list of the three handlers,
+exactly as it is.
+
+- [ ] **Step 4: Repoint the library imports**
+
+`lib/iris/fileformats/cf/saver.py:83` becomes a relative import, because it is
+now a sibling:
+
+```python
+from ._attribute_handlers import ATTRIBUTE_HANDLERS
+```
+
+Place it with the other relative imports, where ruff's sorting wants it.
+
+`lib/iris/fileformats/_nc_load_rules/actions.py:552` keeps its deferred form —
+it is inside a function, and the comment above it says why — and changes only
+its path:
+
+```python
+ from iris.fileformats.cf._attribute_handlers import ATTRIBUTE_HANDLERS
+```
+
+- [ ] **Step 5: Rewrite the paragraph in `cf/saver.py` that the move makes untrue**
+
+Lines 32-40 of the module docstring describe a two-way import that no longer
+exists. Replace that whole paragraph with:
+
+```python
+``ATTRIBUTE_HANDLERS`` lives beside this module, in
+:mod:`iris.fileformats.cf._attribute_handlers`. The handlers serialise the
+Iris-only attribute types that every CF store needs - STASH, GRIB_PARAM,
+ukmo__process_flags - and the netCDF load rules read them from there too.
+```
+
+The surrounding paragraphs stay. The grid-mapping one goes in Task 2.
+
+- [ ] **Step 6: Add the new module to the import-order test**
+
+`lib/iris/tests/unit/fileformats/cf/test__imports.py` lists nine modules. Add a
+tenth, and a sentence saying what is new about this one:
+
+```python
+MODULES = [
+ "iris.fileformats.cf",
+ "iris.fileformats.cf.dataset",
+ "iris.fileformats.cf.loader",
+ "iris.fileformats.cf.saver",
+ "iris.fileformats.cf._attribute_handlers",
+ "iris.fileformats.cf._dask_locks",
+ "iris.fileformats.netcdf",
+ "iris.fileformats.netcdf._dataset",
+ "iris.fileformats.netcdf.loader",
+ "iris.fileformats.netcdf.saver",
+]
+```
+
+and append to the module docstring:
+
+```python
+``_attribute_handlers`` imports :mod:`iris.fileformats.pp`, so the CF package
+now reaches the PP package while the ``iris.fileformats`` package is itself
+part-way through importing. That was already true of the netCDF package, which
+is where this module used to live and which ``iris.fileformats.__init__``
+imports before ``pp``.
+```
+
+- [ ] **Step 7: Run everything that names the handlers**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf \
+ lib/iris/tests/unit/fileformats/netcdf \
+ lib/iris/tests/unit/fileformats/nc_load_rules \
+ lib/iris/tests/integration/netcdf \
+ lib/iris/tests/unit/fileformats/pp -q -n auto
+```
+
+Expected: all pass. `test_GribParamHandler.py` skips without `iris_grib`, as it
+does today.
+
+Then check nothing anywhere still names the old path:
+
+```bash
+grep -rn "netcdf._attribute_handlers\|netcdf import _attribute_handlers" lib/ docs/ benchmarks/
+```
+
+Expected: no output.
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add -A
+git commit -m "Move the CF attribute handlers into the CF package"
+```
+
+---
+
+## Task 2: Grid-mapping parameters reach a store that is not netCDF
+
+F2, and Q8 in spec §12.3. `CFSaver._add_grid_mapping_to_dataset` binds
+`grid_variable = cf_var_grid.variable` at `cf/saver.py:1883` and then makes 63
+plain attribute assignments to it. `.variable` is netCDF4's object; a Zarr
+store has none, so every cube with a coordinate system raises `AttributeError`
+there. Q8 refuses to regularise the encoding in this programme, because routing
+those 63 through `.attributes` changes what lands in a netCDF file.
+
+So: one hook, one changed line, and the netCDF path keeps writing to the very
+same object it writes to today.
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/saver.py:25-31` (docstring), `:1883`, and a new method beside the other hooks
+- Modify: `lib/iris/fileformats/netcdf/saver.py` (the override, beside `_patch_dataset`)
+- Create: `lib/iris/tests/unit/fileformats/cf/saver/__init__.py`
+- Create: `lib/iris/tests/unit/fileformats/cf/saver/test_grid_mapping_writer.py`
+
+**Interfaces:**
+- Consumes: nothing from Task 1.
+- Produces: `CFSaver._grid_mapping_writer(cf_var) -> object`, the fifth hook.
+ The base returns `_GridMappingAttributeWriter(cf_var)`, whose `__setattr__`
+ writes `cf_var.attributes[name] = value`. `netcdf.saver.Saver` overrides it to
+ return `cf_var.variable`, unchanged. Task 10's Zarr saver inherits the base
+ and implements nothing.
+
+- [ ] **Step 1: Write the failing tests**
+
+There is no `tests/unit/fileformats/cf/saver/` package yet: PR 5 left the
+saver's tests under `netcdf/saver/`, driving the generic machinery through the
+only subclass that existed, and said a shared body was worth factoring out when
+PR 6 added the second. This is the first test that belongs to `CFSaver`
+itself, so create the package marker first:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.cf.saver`."""
+```
+
+Then `lib/iris/tests/unit/fileformats/cf/saver/test_grid_mapping_writer.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :meth:`iris.fileformats.cf.saver.CFSaver._grid_mapping_writer`.
+
+The hook exists because the 63 grid-mapping parameters are written by plain
+attribute assignment, and netCDF4's own object is what receives them today.
+Changing where they land would change what is in a netCDF file, which Q8 of
+the design document rules out of this programme, so netCDF keeps that object
+and everything else gets a writer that uses the ``attributes`` mapping.
+"""
+
+import numpy as np
+import pytest
+
+import iris
+from iris.coord_systems import GeogCS
+from iris.cube import Cube
+from iris.fileformats.cf.saver import CFSaver, _GridMappingAttributeWriter
+from iris.fileformats.netcdf._thread_safe_nc import DatasetWrapper
+
+
+@pytest.fixture
+def cube():
+ cube = Cube(np.arange(6, dtype=np.float32).reshape(2, 3), var_name="air")
+ cube.add_dim_coord(
+ iris.coords.DimCoord(
+ np.array([0.0, 1.0]),
+ standard_name="latitude",
+ units="degrees",
+ coord_system=GeogCS(6371229.0),
+ ),
+ 0,
+ )
+ cube.add_dim_coord(
+ iris.coords.DimCoord(
+ np.array([0.0, 1.0, 2.0]),
+ standard_name="longitude",
+ units="degrees",
+ coord_system=GeogCS(6371229.0),
+ ),
+ 1,
+ )
+ return cube
+
+
+class TestTheWriterItself:
+ def test_it_writes_through_the_attributes_mapping(self):
+ recorded = {}
+
+ class FakeVariable:
+ attributes = recorded
+
+ writer = _GridMappingAttributeWriter(FakeVariable())
+ writer.earth_radius = 6371229.0
+ writer.horizontal_datum_name = "OSGB 1936"
+
+ assert recorded == {
+ "earth_radius": 6371229.0,
+ "horizontal_datum_name": "OSGB 1936",
+ }
+
+ def test_it_does_not_grow_members_of_its_own(self):
+ # Every assignment must reach the variable. An attribute that stuck to
+ # the writer instead would be silently dropped from the file.
+ class FakeVariable:
+ attributes: dict = {}
+
+ writer = _GridMappingAttributeWriter(FakeVariable())
+ writer.anything = 1
+ assert "anything" not in vars(writer)
+
+
+class TestTheNetCDFOverride:
+ def test_it_is_the_netcdf4_variable_itself(self, tmp_path, cube):
+ # Not "writes the same values": the same object, so the 63 assignments
+ # take the identical path they take today and the file cannot change.
+ from iris.fileformats.netcdf.saver import Saver
+
+ with Saver(tmp_path / "out.nc", "NETCDF4") as saver:
+ cf_var = saver._dataset.create_variable("probe", np.int32)
+ assert saver._grid_mapping_writer(cf_var) is cf_var.variable
+
+
+class TestTheFileIsUnchanged:
+ def test_grid_mapping_attributes_land_where_they_did(self, tmp_path, cube):
+ path = tmp_path / "out.nc"
+ iris.save(cube, path)
+
+ with DatasetWrapper(path) as dataset:
+ grid = dataset.variables["latitude_longitude"]
+ assert grid.getncattr("grid_mapping_name") == "latitude_longitude"
+ assert grid.getncattr("earth_radius") == pytest.approx(6371229.0)
+ assert grid.getncattr("longitude_of_prime_meridian") == pytest.approx(0.0)
+```
+
+- [ ] **Step 2: Run them, and watch them fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/saver/test_grid_mapping_writer.py -x
+```
+
+Expected: collection error, `ImportError: cannot import name
+'_GridMappingAttributeWriter' from 'iris.fileformats.cf.saver'`.
+
+- [ ] **Step 3: Add the writer and the hook**
+
+In `lib/iris/fileformats/cf/saver.py`, above `class CFSaver`:
+
+```python
+class _GridMappingAttributeWriter:
+ """An assignment target that writes into a variable's attributes.
+
+ :meth:`CFSaver._add_grid_mapping_to_dataset` sets sixty-three CF
+ grid-mapping parameters by plain attribute assignment. On netCDF those
+ assignments go to the netCDF4 variable itself and have done since long
+ before the CF layer existed, and that decides how each one is typed in
+ the file. Regularising it is its own piece of work -- see Q8 in §12.3 of
+ the native Zarr I/O design,
+ ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. Every other store
+ needs them to arrive in the ``attributes`` mapping instead, and this is the
+ object that puts them there.
+
+ """
+
+ def __init__(self, cf_var):
+ # Bypass this class's own __setattr__, which would otherwise write
+ # the variable into the variable.
+ object.__setattr__(self, "_cf_var", cf_var)
+
+ def __setattr__(self, name, value):
+ """Write one grid-mapping parameter into the variable's attributes."""
+ self._cf_var.attributes[name] = value
+```
+
+and, beside the other hooks on `CFSaver` — next to `_backend_write`, which is
+the nearest neighbour in kind:
+
+```python
+ def _grid_mapping_writer(self, cf_var):
+ """Return the object the grid-mapping parameters are assigned to.
+
+ Parameters
+ ----------
+ cf_var : :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+ The grid-mapping variable, newly created and otherwise empty.
+
+ Returns
+ -------
+ object
+ Something that accepts a plain attribute assignment per CF
+ grid-mapping parameter. Here, a writer onto ``cf_var.attributes``.
+ A store that types those attributes some other way overrides this.
+
+ """
+ return _GridMappingAttributeWriter(cf_var)
+```
+
+- [ ] **Step 4: Change the one line in the method**
+
+`cf/saver.py:1883`, and the comment above it that explains the netCDF4 reach:
+
+```python
+ # The 63 assignments below decide how each parameter is typed in the
+ # file, and on netCDF they go to the netCDF4 variable itself, exactly
+ # as they always have; see Q8 in the design document. Every other
+ # store gets a writer onto `.attributes`.
+ grid_variable = self._grid_mapping_writer(cf_var_grid)
+```
+
+Nothing else in the 230-line method changes.
+
+- [ ] **Step 5: Add the netCDF override**
+
+In `lib/iris/fileformats/netcdf/saver.py`, beside `_patch_dataset`:
+
+```python
+ def _grid_mapping_writer(self, cf_var):
+ """Return the netCDF4 variable itself, which is what has always received these.
+
+ Assignments to a netCDF4 variable bypass the ASCII-to-bytes coercion
+ that ``.attributes`` applies, so ``crs_wkt`` is written as
+ ``NC_STRING`` while ``grid_mapping_name`` is ``NC_CHAR``. Regularising
+ that is Q8 in §12.3 of the native Zarr I/O design,
+ ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``; until then,
+ this returns the same object the parameters have always been written
+ to, so the file is unchanged.
+
+ Parameters
+ ----------
+ cf_var : :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`
+ The grid-mapping variable.
+
+ Returns
+ -------
+ :class:`netCDF4.Variable`
+
+ """
+ return cf_var.variable
+```
+
+- [ ] **Step 6: Rewrite the docstring paragraph that said this was impossible**
+
+`cf/saver.py:25-31` currently tells a reader that a non-netCDF subclass fails on
+grid mappings. Replace that paragraph with:
+
+```python
+``_add_grid_mapping_to_dataset`` writes the CF grid-mapping parameters by plain
+attribute assignment, to whatever ``_grid_mapping_writer`` hands back. netCDF
+hands back its own variable, because what that does to the type of each
+attribute in the file is long-standing behaviour and not this module's to
+change -- see Q8 in §12.3 of the native Zarr I/O design,
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``. Every other store gets
+a writer onto the ``attributes`` mapping.
+```
+
+- [ ] **Step 7: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/saver/test_grid_mapping_writer.py -v
+pytest lib/iris/tests/unit/fileformats/netcdf \
+ lib/iris/tests/integration/netcdf -q -n auto
+```
+
+Expected: the new file passes, and the netCDF suites are unchanged — the
+grid-mapping save tests are the ones that matter, and there are many.
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add -A
+git commit -m "Let grid-mapping parameters reach a store that is not netCDF"
+```
+
+---
+
+## Task 3: `zarr/_encode.py`, the write-side value conversions
+
+Spec §4.5, "Attributes", "`_FillValue` and `missing_value`" and "Masked data".
+Three conversions, all pure functions of their arguments, all unit-testable
+without a store — which is why they are their own module (F4) and why they come
+before anything that calls them.
+
+**Files:**
+- Create: `lib/iris/fileformats/zarr/_encode.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/test__encode.py`
+
+**Interfaces:**
+- Consumes: nothing.
+- Produces:
+ - `json_native(value, *, name: str) -> Any` — a JSON-serialisable value, or
+ `TypeError` naming `name`.
+ - `encode_fill_value(value, dtype: np.dtype) -> str | int | float` — base64
+ for floating, a plain number otherwise.
+ - `encode_missing_values(value, dtype: np.dtype) -> float | int | list` — plain
+ numbers, scalar or vector (F10).
+ - `default_fill_value(dtype: np.dtype) -> np.generic` — netCDF's default fill
+ for that dtype (F11).
+ - `filled(data, fill_value)` — a plain array, lazily if `data` was lazy (F13).
+ - `DEFAULT_FILL_VALUES: dict[str, Any]`, keyed by `dtype.str[1:]`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `lib/iris/tests/unit/fileformats/zarr/test__encode.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.zarr._encode`."""
+
+import base64
+import struct
+
+import dask.array as da
+import numpy as np
+import pytest
+
+from iris.fileformats.zarr._encode import (
+ DEFAULT_FILL_VALUES,
+ default_fill_value,
+ encode_fill_value,
+ encode_missing_values,
+ filled,
+ json_native,
+)
+
+
+def unpack(encoded):
+ """Return the float in a base64 little-endian float64."""
+ return struct.unpack(" Any:
+ """Return `value` as something zarr's JSON encoder accepts.
+
+ Parameters
+ ----------
+ value : Any
+ The attribute value, as the cube holds it.
+ name : str
+ The attribute's name, for the error message. A failure here is a
+ value the user wrote, so the message has to say which one.
+
+ Returns
+ -------
+ Any
+ A JSON-serialisable value: NumPy scalars become Python ones, NumPy
+ arrays become lists, ``bytes`` become ``str`` via ASCII, everything
+ else is returned unchanged.
+
+ Raises
+ ------
+ TypeError
+ If the value cannot be stored. It is **not** converted to its string
+ form: a string that cannot be read back is a worse answer than a
+ refusal, and a warning would not stop the save.
+
+ """
+ if isinstance(value, bytes):
+ try:
+ converted = value.decode("ascii")
+ except UnicodeDecodeError as exception:
+ message = (
+ f"Cannot save attribute {name!r}: its bytes are not ascii, "
+ "and a Zarr attribute is text."
+ )
+ raise TypeError(message) from exception
+ elif isinstance(value, np.generic):
+ converted = value.item()
+ elif isinstance(value, np.ndarray):
+ converted = value.tolist()
+ else:
+ converted = value
+
+ try:
+ json.dumps(converted)
+ except TypeError as exception:
+ message = (
+ f"Cannot save attribute {name!r}: a value of type "
+ f"{type(value).__name__} has no JSON form. Convert it to a number, "
+ "a string, or a list of either."
+ )
+ raise TypeError(message) from exception
+
+ return converted
+
+
+def encode_fill_value(value: Any, dtype: np.dtype) -> Any:
+ """Return a ``_FillValue`` attribute in the encoding readers expect.
+
+ Parameters
+ ----------
+ value : Any
+ The sentinel, in any numeric form.
+ dtype : numpy.dtype
+ The array's dtype, which decides the encoding.
+
+ Returns
+ -------
+ str or int or float
+ For a floating array, the base64 of the little-endian ``float64``
+ form - including for ``float32``, which is what released xarray
+ reads. For anything else, a plain JSON value.
+
+ """
+ if dtype.kind == "f":
+ packed = struct.pack(_PACKED_DOUBLE, float(value))
+ return base64.b64encode(packed).decode("ascii")
+ return json_native(value, name="_FillValue")
+
+
+def encode_missing_values(value: Any, dtype: np.dtype) -> Any:
+ """Return a ``missing_value`` attribute as plain JSON numbers.
+
+ Never base64, whatever the dtype: xarray reads this attribute as an
+ ordinary value, so a packed string there opens cleanly and masks nothing.
+ CF section 2.5.1 allows a vector here, so a sequence stays a sequence.
+
+ Parameters
+ ----------
+ value : Any
+ One sentinel, or a sequence of them.
+ dtype : numpy.dtype
+ The array's dtype. Unused, and named so that the two encoders have
+ one signature: a caller choosing between them should not have to
+ learn which one wants the dtype.
+
+ Returns
+ -------
+ int or float or list
+
+ """
+ del dtype
+ return json_native(value, name="missing_value")
+
+
+def default_fill_value(dtype: np.dtype) -> np.generic:
+ """Return the sentinel to use for a masked source that declares none.
+
+ Parameters
+ ----------
+ dtype : numpy.dtype
+ The array's dtype.
+
+ Returns
+ -------
+ numpy.generic
+ The netCDF default fill for that dtype, as a value of it.
+
+ Raises
+ ------
+ ValueError
+ For a dtype with no entry - a complex or structured array, say,
+ which CF cannot express either.
+
+ """
+ key = dtype.str[1:]
+ if key not in DEFAULT_FILL_VALUES:
+ message = (
+ f"No default fill value for dtype {dtype}, so masked data of that "
+ "type cannot be saved to Zarr. Pass an explicit 'fill_value'."
+ )
+ raise ValueError(message)
+ return dtype.type(DEFAULT_FILL_VALUES[key])
+
+
+def filled(data, fill_value):
+ """Return `data` with any masked points replaced by `fill_value`.
+
+ A Zarr array stores the fill in the data, as spec §4.5 settles: no
+ separate mask array. Writing a masked array to one raises inside numpy,
+ so this is not optional.
+
+ Parameters
+ ----------
+ data : numpy.ndarray or dask.array.Array
+ Real or lazy, masked or not.
+ fill_value : Any
+ What replaces the masked points.
+
+ Returns
+ -------
+ numpy.ndarray or dask.array.Array
+ Lazy if `data` was lazy. An unmasked input is returned unchanged,
+ not copied.
+
+ """
+ if is_lazy_data(data):
+ return da.ma.filled(data, fill_value)
+ if np.ma.isMaskedArray(data):
+ return data.filled(fill_value)
+ return data
+```
+
+Note the `import json` that `json_native` needs: put it with the other imports
+at the top, not inside the function. The test imports it locally only because a
+test may.
+
+- [ ] **Step 4: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/test__encode.py -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 5: Check it against xarray, once, by hand**
+
+The encodings exist to satisfy a reader that is not Iris, so look at one with
+your own eyes before building on them:
+
+```python
+import base64, struct, numpy as np, xarray as xr, zarr
+from iris.fileformats.zarr._encode import encode_fill_value
+
+group = zarr.open_group(store="/tmp/check.zarr", mode="w", zarr_format=3)
+array = group.create_array(
+ "data", shape=(3,), dtype="f4", fill_value=np.float32(-99.0),
+ dimension_names=("x",),
+)
+array[:] = np.array([1.0, -99.0, 3.0], dtype="f4")
+array.attrs["_FillValue"] = encode_fill_value(np.float32(-99.0), np.dtype("f4"))
+print(xr.open_zarr("/tmp/check.zarr", consolidated=False)["data"].values)
+```
+
+Expected: `[ 1. nan 3.]`. If instead the open raises `TypeError: Failed to
+decode fill_value`, the encoder is writing a plain number and Task 14's
+conformance test would have caught it later, painfully.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add lib/iris/fileformats/zarr/_encode.py \
+ lib/iris/tests/unit/fileformats/zarr/test__encode.py
+git commit -m "Add the Zarr write-side value conversions"
+```
+
+---
+
+## Task 4: Open a Zarr store for writing
+
+Spec §4.5 and §4.6. PR 4's `ZarrDataset.__init__` accepts `mode="r"` and
+nothing else. This opens the other four, defends the default that makes a
+clobber deliberate, and refuses to write any version but 3 — in one place, so
+that adding version 2 writing later is that guard and the three version-specific
+points §4.5 lists.
+
+**Files:**
+- Modify: `lib/iris/fileformats/zarr/_dataset.py` — `_READ_MODES` and the
+ `__init__` guard, around `:592-648`
+- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py`
+
+**Interfaces:**
+- Consumes: nothing from Tasks 1-3.
+- Produces: `ZarrDataset(location, mode="r", *, group=None, zarr_format=3)`,
+ accepting `"r"`, `"r+"`, `"a"`, `"w"` and `"w-"`. `self._declared_dimensions:
+ dict[str, int]`, which Task 5 fills. The module constants `_WRITE_MODES`,
+ `_MODES` and `_WRITABLE_ZARR_FORMAT`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the write half of :class:`iris.fileformats.zarr._dataset.ZarrDataset`."""
+
+import numpy as np
+import pytest
+
+zarr = pytest.importorskip("zarr")
+
+from iris.fileformats.zarr._dataset import ZarrDataset
+
+
+@pytest.fixture
+def existing(tmp_path):
+ """Return the path of a store that already holds one array."""
+ path = tmp_path / "existing.zarr"
+ group = zarr.open_group(store=path, mode="w-", zarr_format=3)
+ array = group.create_array(
+ "already_here", shape=(2,), dtype="f4", dimension_names=("x",)
+ )
+ array[:] = np.array([1.0, 2.0], dtype="f4")
+ return path
+
+
+class TestOpening:
+ def test_a_new_store_is_created(self, tmp_path):
+ with ZarrDataset(tmp_path / "new.zarr", mode="w-") as dataset:
+ assert dataset.mode == "w-"
+ assert dataset.format == "ZARR_V3"
+ assert (tmp_path / "new.zarr" / "zarr.json").is_file()
+
+ def test_the_default_refuses_an_existing_store(self, existing):
+ # The default differs from the netCDF saver's deliberately: a Zarr
+ # clobber is many non-atomic deletes, so it is opted into.
+ with pytest.raises(FileExistsError, match="mode"):
+ ZarrDataset(existing, mode="w-")
+
+ def test_the_refusal_names_the_way_out(self, existing):
+ with pytest.raises(FileExistsError, match="'w'"):
+ ZarrDataset(existing, mode="w-")
+
+ def test_w_replaces_an_existing_store(self, existing):
+ with ZarrDataset(existing, mode="w") as dataset:
+ assert list(dataset.variables) == []
+
+ def test_a_adds_to_an_existing_store(self, existing):
+ with ZarrDataset(existing, mode="a") as dataset:
+ assert list(dataset.variables) == ["already_here"]
+
+ def test_a_opens_a_new_group_in_an_existing_store(self, existing):
+ with ZarrDataset(existing, mode="a", group="sub") as dataset:
+ assert dataset.location.endswith("existing.zarr/sub")
+ assert list(dataset.variables) == []
+ # The root is untouched.
+ with ZarrDataset(existing) as dataset:
+ assert list(dataset.variables) == ["already_here"]
+
+ def test_r_plus_needs_the_store_to_exist(self, tmp_path):
+ with pytest.raises(FileNotFoundError):
+ ZarrDataset(tmp_path / "missing.zarr", mode="r+")
+
+ def test_an_unknown_mode_names_itself(self, tmp_path):
+ with pytest.raises(ValueError, match="'a\\+'"):
+ ZarrDataset(tmp_path / "new.zarr", mode="a+")
+
+ def test_reading_is_still_the_default(self, existing):
+ with ZarrDataset(existing) as dataset:
+ assert dataset.mode == "r"
+
+
+class TestTheVersionGuard:
+ @pytest.mark.parametrize("mode", ["w-", "w", "a", "r+"])
+ def test_writing_version_2_is_refused(self, tmp_path, mode):
+ with pytest.raises(NotImplementedError, match="version 3"):
+ ZarrDataset(tmp_path / "new.zarr", mode=mode, zarr_format=2)
+
+ def test_reading_version_2_is_unaffected(self, tmp_path):
+ path = tmp_path / "v2.zarr"
+ group = zarr.open_group(store=path, mode="w-", zarr_format=2)
+ array = group.create_array("data", shape=(2,), dtype="f4")
+ array.attrs["_ARRAY_DIMENSIONS"] = ["x"]
+ with ZarrDataset(path, mode="r", zarr_format=2) as dataset:
+ assert dataset.format == "ZARR_V2"
+ assert list(dataset.variables) == ["data"]
+
+ def test_a_version_3_store_is_what_a_write_produces(self, tmp_path):
+ with ZarrDataset(tmp_path / "new.zarr", mode="w-") as dataset:
+ assert dataset.encoding["zarr_format"] == 3
+```
+
+- [ ] **Step 2: Run them, and watch them fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py -x
+```
+
+Expected: `NotImplementedError: Saving to Zarr is not supported. Cannot open
+with mode='w-'.`
+
+- [ ] **Step 3: Replace the mode constants**
+
+In `lib/iris/fileformats/zarr/_dataset.py`, the block above `class ZarrDataset`
+currently reads `_READ_MODES = ("r",)` and `_NO_SAVING`. Replace both with:
+
+```python
+#: The modes that only read. Writing through one of these is refused before
+#: zarr-python gets the chance, so the message names Iris's keyword.
+_READ_MODES = ("r",)
+
+#: The modes that may write. ``"r+"`` is here for the region write a future
+#: multi-process saver needs: open an existing store for update, creating
+#: nothing. See §4.5 of the design document.
+_WRITE_MODES = ("w-", "w", "a", "r+")
+
+#: Every mode, in the order the error message lists them.
+_MODES = _READ_MODES + _WRITE_MODES
+
+#: The only Zarr version Iris writes. Reading handles both. Adding version 2
+#: writing means the three version-specific points of §4.5 and deleting this.
+_WRITABLE_ZARR_FORMAT = 3
+```
+
+Delete `_NO_SAVING`; Tasks 5 and 6 implement every member that raised it.
+
+- [ ] **Step 4: Open for writing**
+
+Replace the guard and the open in `ZarrDataset.__init__`:
+
+```python
+ if mode not in _MODES:
+ message = f"Unknown mode {mode!r} for a Zarr store. Expected one of {_MODES}."
+ raise ValueError(message)
+ writing = mode in _WRITE_MODES
+ if writing and zarr_format != _WRITABLE_ZARR_FORMAT:
+ message = (
+ f"Iris writes Zarr version {_WRITABLE_ZARR_FORMAT} stores only, "
+ f"so mode={mode!r} cannot be used with zarr_format={zarr_format!r}."
+ )
+ raise NotImplementedError(message)
+
+ self._mode = mode
+ target = str(location)
+ self._location = target if group is None else f"{target.rstrip('/')}/{group}"
+
+ # `location` itself, not its stringified `target`, is handed to
+ # zarr-python: stringifying a store object such as a `MemoryStore`
+ # loses the store, leaving only a name zarr-python cannot reopen.
+ try:
+ self._group = zarr.open_group(
+ store=location,
+ mode=mode,
+ path=group,
+ zarr_format=zarr_format if writing else None,
+ )
+ except FileExistsError as exception:
+ message = (
+ f"Cannot create the Zarr store {self._location!r}: it already "
+ f"exists, and mode={mode!r} will not overwrite it. Pass "
+ "mode='w' to replace it, or mode='a' to add to it."
+ )
+ raise FileExistsError(message) from exception
+ self._format = _FORMAT_BY_ZARR_VERSION[self._group.metadata.zarr_format]
+```
+
+and add, with the other initialised state above the guard:
+
+```python
+ #: The dimensions `create_dimension` has declared, by name. Zarr has
+ #: no dimension object, so this dict is the whole of the concept on
+ #: the write side, and what lets a size conflict be caught.
+ self._declared_dimensions: dict[str, int] = {}
+```
+
+Extend the `mode` and `Raises` entries of the docstring:
+
+```python
+ mode : str, default="r"
+ One of ``"r"``, ``"r+"``, ``"a"``, ``"w"`` or ``"w-"``, with
+ zarr-python's meanings. ``"w-"`` creates and refuses an existing
+ target; ``"w"`` replaces one; ``"a"`` adds to one.
+ zarr_format : int, default=3
+ The version to create. Only version 3 can be written. Reading
+ handles either version, and ignores this.
+```
+
+```python
+ Raises
+ ------
+ ValueError
+ For an unknown `mode`.
+ NotImplementedError
+ For a write `mode` with any `zarr_format` but 3.
+ FileExistsError
+ For ``mode="w-"`` onto a target that already exists.
+ FileNotFoundError
+ If `location` does not exist and `mode` does not create it.
+ zarr.errors.GroupNotFoundError
+ If `location` is not a Zarr store, or holds no such `group`.
+```
+
+- [ ] **Step 5: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py -v
+pytest lib/iris/tests/unit/fileformats/zarr lib/iris/tests/integration/zarr -q -n auto
+```
+
+Expected: the new file passes and PR 4's Zarr tests are untouched — the read
+path takes the same `mode="r"` branch it always did.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add -A
+git commit -m "Open a Zarr store for writing"
+```
+
+---
+
+## Task 5: Declare dimensions, and create arrays
+
+Spec §4.5, "Dimensions", and the create-once invariant. This is the task that
+turns `CFSaver`'s two creation calls into Zarr arrays.
+
+**Files:**
+- Modify: `lib/iris/fileformats/zarr/_dataset.py` — `create_dimension`,
+ `create_variable`, `dimensions`, and the caches both invalidate
+- Modify: `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py`
+
+**Interfaces:**
+- Consumes: Task 3's `default_fill_value` and `encode_fill_value`; Task 4's
+ `_declared_dimensions` and `_WRITE_MODES`.
+- Produces: `ZarrDataset.create_dimension(name, size)` and
+ `ZarrDataset.create_variable(name, dtype, dimensions=(), *, fill_value=None,
+ **encoding) -> ZarrDatasetVariable`, with `_ZARR_ENCODING_KEYWORDS` naming
+ what `**encoding` accepts.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `test_ZarrDataset__write.py`:
+
+```python
+@pytest.fixture
+def writable(tmp_path):
+ """Return an open, empty, writable dataset."""
+ with ZarrDataset(tmp_path / "out.zarr", mode="w-") as dataset:
+ yield dataset
+
+
+class TestCreateDimension:
+ def test_a_dimension_is_declared(self, writable):
+ writable.create_dimension("time", 3)
+ assert dict(writable.dimensions) == {"time": 3}
+
+ def test_declaring_it_twice_at_one_size_is_harmless(self, writable):
+ writable.create_dimension("time", 3)
+ writable.create_dimension("time", 3)
+ assert dict(writable.dimensions) == {"time": 3}
+
+ def test_a_conflicting_size_raises_naming_both(self, writable):
+ writable.create_dimension("time", 3)
+ with pytest.raises(ValueError, match="time.*3.*4|time.*4.*3"):
+ writable.create_dimension("time", 4)
+
+ def test_unlimited_is_refused(self, writable):
+ # Spec §4.2: a store with no such concept raises NotImplementedError.
+ with pytest.raises(NotImplementedError, match="unlimited"):
+ writable.create_dimension("time", None)
+
+ def test_a_read_only_dataset_refuses(self, existing):
+ with ZarrDataset(existing) as dataset:
+ with pytest.raises(ValueError, match="mode='r'"):
+ dataset.create_dimension("time", 3)
+
+
+class TestCreateVariable:
+ def test_an_array_is_created_with_its_dimension_names(self, writable):
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert variable.name == "data"
+ assert variable.dimensions == ("x",)
+ assert variable.shape == (3,)
+ assert variable.dtype == np.dtype("f4")
+
+ def test_it_appears_in_the_dataset(self, writable):
+ writable.create_dimension("x", 3)
+ writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert list(writable.variables) == ["data"]
+
+ def test_a_scalar_variable_has_no_dimensions(self, writable):
+ variable = writable.create_variable("grid", np.dtype("i4"))
+ assert variable.shape == ()
+
+ def test_an_undeclared_dimension_raises_naming_it(self, writable):
+ with pytest.raises(ValueError, match="nowhere"):
+ writable.create_variable("data", np.dtype("f4"), ["nowhere"])
+
+ def test_an_existing_name_raises_naming_the_group(self, writable):
+ # Spec §4.5: "not an overwrite and not a merge".
+ writable.create_dimension("x", 3)
+ writable.create_variable("data", np.dtype("f4"), ["x"])
+ with pytest.raises(ValueError) as excinfo:
+ writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert "data" in str(excinfo.value)
+ assert writable.location in str(excinfo.value)
+
+ def test_a_read_only_dataset_refuses(self, existing):
+ with ZarrDataset(existing) as dataset:
+ with pytest.raises(ValueError, match="mode='r'"):
+ dataset.create_variable("data", np.dtype("f4"))
+
+
+class TestCreateVariableFillValue:
+ def test_an_undeclared_fill_is_netcdfs_default(self, writable):
+ # F11: zarr's own default is 0, which would make a masked point of a
+ # float array read back as a real zero.
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert variable.fill_value == np.float32(9.9692099683868690e36)
+
+ def test_an_undeclared_fill_writes_no_attribute(self, writable):
+ # Nothing is missing yet. The saver declares _FillValue when it finds
+ # the source is masked; see Task 11.
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert "_FillValue" not in variable.attributes
+
+ def test_a_declared_fill_reaches_the_array_and_the_attribute(self, writable):
+ import base64
+ import struct
+
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable(
+ "data", np.dtype("f4"), ["x"], fill_value=np.float32(-99.0)
+ )
+ assert variable.fill_value == np.float32(-99.0)
+ packed = base64.b64decode(variable.attributes["_FillValue"])
+ assert struct.unpack(" None:
+ """Declare a dimension of the given length.
+
+ Zarr has no dimension object: a dimension is a name an array carries
+ in its own metadata. So this records the name and size, and
+ :meth:`create_variable` writes the names onto each array. The record
+ is what lets a size conflict be caught here rather than becoming two
+ arrays that disagree.
+
+ Parameters
+ ----------
+ name : str
+ The dimension's name.
+ size : int or None
+ Its length. ``None`` requests an unlimited dimension, which Zarr
+ has no form for.
+
+ Raises
+ ------
+ NotImplementedError
+ For ``size=None``.
+ ValueError
+ If `name` is already declared at a different size, or if this
+ dataset was opened read-only.
+
+ """
+ self._check_writable("create_dimension")
+ if size is None:
+ message = (
+ f"Cannot create the unlimited dimension {name!r}: a Zarr array "
+ "has a fixed shape, so there is no unlimited dimension to "
+ "create."
+ )
+ raise NotImplementedError(message)
+
+ declared = self._declared_dimensions.get(name)
+ if declared is not None and declared != size:
+ message = (
+ f"Cannot declare dimension {name!r} with size {size}: it is "
+ f"already declared with size {declared}."
+ )
+ raise ValueError(message)
+ self._declared_dimensions[name] = size
+ self._dimensions = None
+```
+
+and the guard it uses, beside it:
+
+```python
+ def _check_writable(self, action: str) -> None:
+ """Raise unless this dataset was opened in a mode that writes."""
+ if self._mode not in _WRITE_MODES:
+ message = (
+ f"Cannot {action} on a Zarr store opened with mode={self._mode!r}. "
+ f"Open it with one of {_WRITE_MODES} to write."
+ )
+ raise ValueError(message)
+```
+
+- [ ] **Step 5: Report declared dimensions**
+
+`dimensions` assembles from the arrays present. A dimension declared but not
+yet written to any array has to appear too — `_get_dim_names` and
+`_create_string_variable` both ask this mapping what exists before choosing a
+name. Replace the property with:
+
+```python
+ @property
+ def dimensions(self) -> Mapping:
+ """The dimension sizes this group's arrays agree on, and any declared.
+
+ A dimension declared by :meth:`create_dimension` appears here before
+ any array carries it, because that is what a caller choosing a new
+ name needs to see.
+
+ """
+ if self._dimensions is None:
+ self._dimensions = {
+ **_assemble_dimensions(self.variables),
+ **self._declared_dimensions,
+ }
+ return self._dimensions
+```
+
+- [ ] **Step 6: Implement `create_variable`**
+
+```python
+ def create_variable(
+ self, name: str, dtype, dimensions=(), *, fill_value=None, **encoding
+ ) -> "ZarrDatasetVariable":
+ """Create and return a new array.
+
+ Parameters
+ ----------
+ name : str
+ The array's name within this group.
+ dtype : numpy.dtype or str or type
+ The array's dtype. ``str`` itself asks for a variable-length
+ string array, which is how this store holds text.
+ dimensions : iterable of str, optional
+ The names of its dimensions, each already declared by
+ :meth:`create_dimension`. Empty for a scalar.
+ fill_value : optional
+ The CF missing-value sentinel. When given it is written as a
+ ``_FillValue`` attribute as well as the array's own fill, because
+ on version 3 the attribute is the masking declaration and the
+ storage fill is not. When not given, the array takes the netCDF
+ default fill for its dtype, and declares nothing: the saver
+ declares one if it finds the data is masked.
+ **encoding
+ Zarr's own storage keywords: ``chunks``, ``shards``,
+ ``compressors``, ``filters``, ``serializer``, ``order``.
+
+ Returns
+ -------
+ :class:`ZarrDatasetVariable`
+
+ Raises
+ ------
+ ValueError
+ If the dataset is read-only, a dimension is undeclared, or `name`
+ already exists in this group.
+ TypeError
+ For a storage keyword this store does not have.
+
+ """
+ self._check_writable("create_variable")
+ check_encoding_keywords(encoding)
+
+ dtype = dtype if dtype is str else np.dtype(dtype)
+ dimensions = tuple(dimensions)
+ shape = tuple(self._dimension_size(dimension) for dimension in dimensions)
+
+ if name in self._group:
+ message = (
+ f"Cannot create {name!r} in the Zarr group {self._location!r}: "
+ "it already holds something of that name, and Iris neither "
+ "overwrites nor merges into one."
+ )
+ raise ValueError(message)
+
+ if fill_value is None:
+ stored_fill = _encode.default_fill_value(np.dtype(dtype))
+ else:
+ stored_fill = np.dtype(dtype).type(fill_value)
+
+ array = self._group.create_array(
+ name,
+ shape=shape,
+ dtype=dtype,
+ fill_value=stored_fill,
+ dimension_names=dimensions or None,
+ **encoding,
+ )
+ variable = ZarrDatasetVariable(array, self._location, self._format)
+ if fill_value is not None:
+ variable.attributes["_FillValue"] = fill_value
+
+ # Both caches describe a group that has just changed.
+ self._variables = None
+ self._dimensions = None
+ return variable
+```
+
+with the two helpers beside it:
+
+```python
+ def _dimension_size(self, name: str) -> int:
+ """Return a declared dimension's size, or raise naming it."""
+ size = self.dimensions.get(name)
+ if size is None:
+ message = (
+ f"Cannot create an array over dimension {name!r}: no dimension "
+ "of that name has been declared in this group."
+ )
+ raise ValueError(message)
+ return size
+```
+
+and, at module level beside the keyword tables rather than on the class,
+because :func:`iris.fileformats.zarr.saver.save` calls it too - before it
+creates a store, so that a mistyped keyword does not leave an empty one behind:
+
+```python
+def check_encoding_keywords(encoding) -> None:
+ """Raise for any storage keyword a Zarr store does not have.
+
+ Parameters
+ ----------
+ encoding : dict
+ The storage keywords a caller passed.
+
+ Raises
+ ------
+ TypeError
+ Naming each unknown keyword, and the Zarr replacement where there
+ is one.
+
+ """
+ unknown = set(encoding) - _ZARR_ENCODING_KEYWORDS
+ if not unknown:
+ return
+ lines = []
+ for keyword in sorted(unknown):
+ replacement = _NETCDF_KEYWORDS.get(keyword)
+ if replacement is None:
+ lines.append(f" {keyword}: not a Zarr storage keyword")
+ else:
+ lines.append(f" {keyword}: use {replacement}")
+ message = (
+ "Unknown Zarr storage keyword(s). A Zarr store takes zarr-python's "
+ "own keywords, which Iris does not translate netCDF's onto:\n"
+ + "\n".join(lines)
+ )
+ raise TypeError(message)
+```
+
+Add `from . import _encode` to the imports.
+
+**Note on `name in self._group`.** zarr-python raises `ContainsArrayError`
+itself, but only for an array; a *group* of that name takes a different path,
+and neither message names the store. One check, before anything is created, is
+both more uniform and the only form that can say "Iris neither overwrites nor
+merges", which is the rule being enforced.
+
+- [ ] **Step 7: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add -A
+git commit -m "Declare Zarr dimensions, and create Zarr arrays"
+```
+
+---
+
+## Task 6: Write values and attributes, and consolidate
+
+Spec §4.5, "Attributes", "Masked data" and "Consolidation is a separate,
+one-shot step". Three small changes in `_dataset.py`, each at the one place
+every write of its kind passes through.
+
+**Files:**
+- Modify: `lib/iris/fileformats/zarr/_dataset.py` — `_ZarrAttributes`,
+ `ZarrDatasetVariable.__setitem__`, `ZarrDataset.finalise`
+- Modify: `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py`
+
+**Interfaces:**
+- Consumes: Task 3's `json_native`, `encode_fill_value`,
+ `encode_missing_values` and `filled`; Task 5's `create_variable`.
+- Produces: attribute writes that convert before zarr sees them;
+ `ZarrDatasetVariable.__setitem__` that fills masked values;
+ `ZarrDataset.finalise()` that consolidates the whole store once.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `test_ZarrDataset__write.py`:
+
+```python
+class TestAttributeWriting:
+ def test_a_numpy_scalar_is_converted(self, writable):
+ # F6: zarr's encoder rejects numpy types, and a rejected value stays
+ # in the mapping - so converting afterwards is already too late.
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ variable.attributes["scale_factor"] = np.float32(0.5)
+ assert variable.attributes["scale_factor"] == 0.5
+ assert type(variable.attributes["scale_factor"]) is float
+
+ def test_a_numpy_array_becomes_a_list(self, writable):
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ variable.attributes["valid_range"] = np.array([0.0, 10.0], dtype="f4")
+ assert variable.attributes["valid_range"] == [0.0, 10.0]
+
+ def test_bytes_become_text(self, writable):
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ variable.attributes["units"] = b"K"
+ assert variable.attributes["units"] == "K"
+
+ def test_a_global_attribute_is_converted_too(self, writable):
+ writable.attributes["history"] = np.str_("made by a test")
+ assert type(writable.attributes["history"]) is str
+
+ def test_an_unserialisable_attribute_raises_and_writes_nothing(self, writable):
+ class Opaque:
+ pass
+
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ with pytest.raises(TypeError, match="odd"):
+ variable.attributes["odd"] = Opaque()
+ assert "odd" not in variable.attributes
+ # And the store itself is clean, not merely the mapping.
+ reopened = zarr.open_group(store=writable.location, mode="r")
+ assert "odd" not in dict(reopened["data"].attrs)
+
+ def test_missing_value_is_a_plain_number(self, writable):
+ # F10: base64 here opens in xarray and masks nothing.
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ variable.attributes["missing_value"] = np.float32(-99.0)
+ assert variable.attributes["missing_value"] == -99.0
+
+ def test_a_fill_value_attribute_is_packed(self, writable):
+ import base64
+ import struct
+
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ variable.attributes["_FillValue"] = np.float32(-99.0)
+ packed = base64.b64decode(variable.attributes["_FillValue"])
+ assert struct.unpack(" None:
+ """Set `key`'s value, here and in the store.
+
+ The value is converted to something JSON can hold *before* zarr sees
+ it. A value zarr's encoder rejects raises **and stays in the
+ mapping**, so converting on failure would mean converting a store
+ that is already wrong.
+
+ """
+ if key in self._withhold:
+ msg = f"Cannot set a storage attribute through `attributes`: {key}"
+ raise KeyError(msg)
+ encoded = self._encode(key, value)
+ self._node.attrs[key] = encoded
+ self._values[key] = encoded
+
+ def _encode(self, key: str, value: Any) -> Any:
+ """Return `value` in the form this attribute is stored in."""
+ if self._dtype is not None:
+ if key == "_FillValue":
+ return _encode.encode_fill_value(value, self._dtype)
+ if key == "missing_value":
+ return _encode.encode_missing_values(value, self._dtype)
+ return _encode.json_native(value, name=key)
+```
+
+In `ZarrDatasetVariable.__init__`, pass the dtype:
+
+```python
+ self._attributes = _ZarrAttributes(
+ array, withhold=frozenset({_ARRAY_DIMENSIONS}), dtype=array.dtype
+ )
+```
+
+A variable-length string array's `dtype` is zarr's own `StringDType`, not a
+NumPy dtype with a `kind`. `encode_fill_value` is reached only for
+``_FillValue``, which a string array does not get, but guard the lookup anyway:
+in `_encode`, use `getattr(self._dtype, "kind", None)` rather than
+`self._dtype.kind` if you touch it directly.
+
+- [ ] **Step 4: Fill masked values on the way in**
+
+```python
+ def __setitem__(self, keys, values) -> None:
+ """Write `values` into the indexed portion of the array.
+
+ Masked values are filled with the array's own fill value first. A
+ Zarr array has no mask of its own - spec §4.5 settles that the fill
+ goes in the data - and handing numpy.ma straight to zarr raises
+ inside the bytes codec, naming neither the variable nor the mask.
+
+ """
+ self._array[keys] = _encode.filled(values, self._array.fill_value)
+```
+
+- [ ] **Step 5: Implement `finalise`**
+
+```python
+ def finalise(self) -> None:
+ """Write consolidated metadata for the whole store.
+
+ One document at the root listing every array and group, which is what
+ makes a remote store usable: a reader fetches one object instead of
+ one per node. Separate from :meth:`close` because it is a write, and
+ a one-shot one - a worker that fills part of a store must not do it,
+ since several writers of one metadata object race.
+
+ Always consolidates from the root, even when this dataset is one
+ group of the store. A consolidated document that does not mention the
+ group just written is what a default reader would believe.
+
+ Raises
+ ------
+ ValueError
+ If this dataset was opened read-only.
+ RuntimeError
+ If the store cannot be consolidated. Raised rather than returned
+ quietly: a store left with stale consolidated metadata hides the
+ data just written from every default reader.
+
+ """
+ self._check_writable("finalise")
+ assert self._group is not None
+ with warnings.catch_warnings():
+ # zarr-python warns that consolidated metadata is not part of the
+ # version 3 specification. Iris asks for it deliberately, and the
+ # user cannot act on it.
+ warnings.filterwarnings(
+ "ignore",
+ message="Consolidated metadata is currently not part",
+ category=UserWarning,
+ )
+ try:
+ zarr.consolidate_metadata(self._group.store)
+ except Exception as exception:
+ message = (
+ f"Could not consolidate the metadata of the Zarr store "
+ f"{self._location!r}. The data written is in the store, but "
+ "a reader that uses consolidated metadata may not see it."
+ )
+ raise RuntimeError(message) from exception
+```
+
+Add `import warnings` to the module imports if it is not already there.
+
+`close()` keeps its one job and its docstring. `sync()` keeps its no-op.
+
+- [ ] **Step 6: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py -v
+pytest lib/iris/tests/unit/fileformats/zarr lib/iris/tests/integration/zarr -q -n auto
+```
+
+Expected: all pass, PR 4's read tests included — `_ZarrAttributes.__getitem__`
+is untouched, and nothing on the read path assigns an attribute.
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add -A
+git commit -m "Write Zarr values, attributes and consolidated metadata"
+```
+
+---
+
+## Task 7: `zarr/saver.py` — the saver, and `save`
+
+Spec §5 (PR 6), §4.5 and §4.6. The class is thin: PR 5's `CFSaver` does the CF
+work, and what is left is opening a store, saying how a string is stored, and
+the entry point users call.
+
+**Files:**
+- Create: `lib/iris/fileformats/zarr/saver.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/saver/__init__.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/saver/test_Saver.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/saver/test_save.py`
+- Modify: `lib/iris/fileformats/zarr/__init__.py`
+
+**Interfaces:**
+- Consumes: Tasks 4-6's `ZarrDataset`; Task 5's `check_encoding_keywords`;
+ PR 5's `CFSaver`.
+- Produces: `iris.fileformats.zarr.Saver(filename, compute=True, *, mode=None,
+ group=None, consolidate_metadata=True)` and
+ `iris.fileformats.zarr.save(cube, filename, *, mode=None, group=None,
+ consolidate_metadata=True, local_keys=None, fill_value=None, packing=None,
+ compute=True, **encoding)`. Task 8 adds `_backend_write` and
+ `delayed_completion` to the same class; Task 9 registers `save`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `lib/iris/tests/unit/fileformats/zarr/saver/__init__.py` with the
+copyright header and the one-line docstring `"""Unit tests for
+:mod:`iris.fileformats.zarr.saver`."""`, then
+`lib/iris/tests/unit/fileformats/zarr/saver/test_Saver.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.zarr.saver.Saver`."""
+
+import numpy as np
+import pytest
+
+zarr = pytest.importorskip("zarr")
+
+import iris
+from iris.coords import AuxCoord, DimCoord
+from iris.cube import Cube
+from iris.fileformats.cf.saver import CFSaver
+from iris.fileformats.zarr.saver import Saver
+
+
+@pytest.fixture
+def cube():
+ cube = Cube(
+ np.arange(6, dtype=np.float32).reshape(2, 3),
+ standard_name="air_temperature",
+ units="K",
+ var_name="air_temperature",
+ )
+ cube.add_dim_coord(
+ DimCoord([0.0, 1.0], standard_name="latitude", units="degrees"), 0
+ )
+ cube.add_dim_coord(
+ DimCoord([0.0, 1.0, 2.0], standard_name="longitude", units="degrees"), 1
+ )
+ return cube
+
+
+class TestTheClass:
+ def test_it_is_a_cf_saver(self):
+ assert issubclass(Saver, CFSaver)
+
+ def test_it_opens_a_store(self, tmp_path, cube):
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube)
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert sorted(group.array_keys()) == [
+ "air_temperature",
+ "latitude",
+ "longitude",
+ ]
+
+ def test_the_default_mode_refuses_an_existing_store(self, tmp_path, cube):
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube)
+ with pytest.raises(FileExistsError):
+ Saver(tmp_path / "out.zarr")
+
+ def test_a_group_implies_adding(self, tmp_path, cube):
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube)
+ with Saver(tmp_path / "out.zarr", group="later") as saver:
+ saver.write(cube)
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert "later" in list(group.group_keys())
+ assert "air_temperature" in list(group.array_keys())
+
+ def test_an_explicit_mode_beats_the_group_default(self, tmp_path, cube):
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube)
+ with pytest.raises(FileExistsError):
+ Saver(tmp_path / "out.zarr", mode="w-", group=None)
+
+
+class TestStringVariables:
+ def test_a_label_coordinate_is_a_string_array(self, tmp_path, cube):
+ # netCDF stores text as bytes over an extra dimension. Zarr has a
+ # string type, so the extra dimension would be an invention.
+ cube.add_aux_coord(
+ AuxCoord(np.array(["north", "south"]), long_name="place"), 0
+ )
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube)
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ place = group["place"]
+ assert list(place[:]) == ["north", "south"]
+ assert place.metadata.dimension_names == ("latitude",)
+
+ def test_no_string_dimension_is_invented(self, tmp_path, cube):
+ cube.add_aux_coord(
+ AuxCoord(np.array(["north", "south"]), long_name="place"), 0
+ )
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube)
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ names = {
+ name
+ for array in group.arrays()
+ for name in (array[1].metadata.dimension_names or ())
+ }
+ assert not {name for name in names if name.startswith("string")}
+
+
+class TestGridMappings:
+ def test_a_coordinate_system_is_written(self, tmp_path, cube):
+ # Task 2's hook, reached for real: without it this raises
+ # AttributeError on `.variable`.
+ from iris.coord_systems import GeogCS
+
+ for coord in cube.coords():
+ coord.coord_system = GeogCS(6371229.0)
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube)
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ grid = group["latitude_longitude"]
+ assert grid.attrs["grid_mapping_name"] == "latitude_longitude"
+ assert grid.attrs["earth_radius"] == pytest.approx(6371229.0)
+```
+
+and `lib/iris/tests/unit/fileformats/zarr/saver/test_save.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :func:`iris.fileformats.zarr.saver.save`."""
+
+import numpy as np
+import pytest
+
+zarr = pytest.importorskip("zarr")
+
+from iris.coords import DimCoord
+from iris.cube import Cube, CubeList
+from iris.fileformats.zarr.saver import save
+
+
+@pytest.fixture
+def cubes():
+ def one(name):
+ cube = Cube(
+ np.arange(4, dtype=np.float32).reshape(2, 2),
+ standard_name=name,
+ units="K",
+ var_name=name,
+ )
+ cube.add_dim_coord(
+ DimCoord([0.0, 1.0], standard_name="latitude", units="degrees"), 0
+ )
+ cube.add_dim_coord(
+ DimCoord([0.0, 1.0], standard_name="longitude", units="degrees"), 1
+ )
+ return cube
+
+ return CubeList([one("air_temperature"), one("air_pressure")])
+
+
+class TestSave:
+ def test_one_cube(self, tmp_path, cubes):
+ save(cubes[0], tmp_path / "out.zarr")
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert "air_temperature" in list(group.array_keys())
+
+ def test_a_cubelist_goes_into_one_store(self, tmp_path, cubes):
+ # One store, opened once: this is why the module joins
+ # _CUBELIST_SAVER_MODULES.
+ save(cubes, tmp_path / "out.zarr")
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert {"air_temperature", "air_pressure"} <= set(group.array_keys())
+
+ def test_the_conventions_attribute_is_written(self, tmp_path, cubes):
+ save(cubes[0], tmp_path / "out.zarr")
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert group.attrs["Conventions"].startswith("CF-")
+
+ def test_metadata_is_consolidated_by_default(self, tmp_path, cubes):
+ save(cubes[0], tmp_path / "out.zarr")
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert group.metadata.consolidated_metadata is not None
+
+ def test_consolidation_can_be_declined(self, tmp_path, cubes):
+ save(cubes[0], tmp_path / "out.zarr", consolidate_metadata=False)
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert group.metadata.consolidated_metadata is None
+
+
+class TestSaveRefusals:
+ def test_an_existing_store_is_refused(self, tmp_path, cubes):
+ save(cubes[0], tmp_path / "out.zarr")
+ with pytest.raises(FileExistsError, match="mode"):
+ save(cubes[1], tmp_path / "out.zarr")
+
+ def test_w_replaces(self, tmp_path, cubes):
+ save(cubes[0], tmp_path / "out.zarr")
+ save(cubes[1], tmp_path / "out.zarr", mode="w")
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert "air_temperature" not in list(group.array_keys())
+
+ def test_a_repeated_name_under_a_is_refused(self, tmp_path, cubes):
+ # Spec §4.5: not an overwrite, not a merge.
+ save(cubes[0], tmp_path / "out.zarr")
+ with pytest.raises(ValueError, match="air_temperature"):
+ save(cubes[0], tmp_path / "out.zarr", mode="a")
+
+ def test_a_netcdf_keyword_is_refused_before_anything_is_created(
+ self, tmp_path, cubes
+ ):
+ # The store must not exist afterwards: a half-made store is worse
+ # than no store, and nothing has been written yet.
+ with pytest.raises(TypeError, match="zlib"):
+ save(cubes[0], tmp_path / "out.zarr", zlib=True)
+ assert not (tmp_path / "out.zarr").exists()
+
+ def test_unlimited_dimensions_says_why_not(self, tmp_path, cubes):
+ with pytest.raises(TypeError, match="fixed shape"):
+ save(cubes[0], tmp_path / "out.zarr", unlimited_dimensions=["latitude"])
+
+
+class TestSaveIntoAGroup:
+ def test_a_group_is_created_and_the_root_is_kept(self, tmp_path, cubes):
+ save(cubes[0], tmp_path / "out.zarr")
+ save(cubes[1], tmp_path / "out.zarr", group="extra")
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert "air_temperature" in list(group.array_keys())
+ assert "air_pressure" in list(group["extra"].array_keys())
+
+ def test_the_root_is_re_consolidated(self, tmp_path, cubes):
+ # Spec §4.5: stale consolidated metadata hides what was just written.
+ save(cubes[0], tmp_path / "out.zarr")
+ save(cubes[1], tmp_path / "out.zarr", group="extra")
+ reader = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert "extra" in list(reader.group_keys())
+```
+
+- [ ] **Step 2: Run them, and watch them fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/saver -x
+```
+
+Expected: `ModuleNotFoundError: No module named 'iris.fileformats.zarr.saver'`.
+
+- [ ] **Step 3: Write the module**
+
+Create `lib/iris/fileformats/zarr/saver.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Save cubes to a Zarr store.
+
+.. z_reference:: iris.fileformats.zarr.saver
+ :tags: topic_load_save
+
+ API reference
+
+Writes Zarr version 3 stores. The CF work - dimension names, coordinate,
+bounds, mesh and grid-mapping variables, attribute compliance - belongs to
+:class:`iris.fileformats.cf.saver.CFSaver`, which this builds on. What is here
+is what a Zarr store does differently: it is created rather than opened, it
+holds text natively, and it finishes with a consolidated metadata document.
+
+**The storage keywords are zarr-python's own**: ``chunks``, ``shards``,
+``compressors``, ``filters``, ``serializer``. A netCDF keyword raises and names
+the Zarr replacement. Translating ``zlib=True`` onto a Blosc codec would sell a
+false equivalence, and would hand this new API the shape the netCDF saver is
+stuck with. See §4.5 of the native Zarr I/O design,
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+**The default mode is ``"w-"``**, which refuses an existing target, where the
+netCDF saver clobbers one. A Zarr clobber is many non-atomic deletes, so an
+interrupted one destroys the old store without completing the new.
+
+"""
+
+from pathlib import Path
+
+import iris
+from iris.cube import CubeList
+from iris.fileformats.cf import saver as cf_saver
+from iris.fileformats.cf.saver import CFSaver
+
+from ._dataset import ZarrDataset, check_encoding_keywords
+
+__all__ = ["Saver", "save"]
+
+
+class Saver(CFSaver):
+ """A manager for saving cubes to a Zarr store."""
+
+ def __init__(
+ self,
+ filename,
+ compute=True,
+ *,
+ mode=None,
+ group=None,
+ consolidate_metadata=True,
+ ):
+ """Manage saving cubes into a Zarr store.
+
+ Parameters
+ ----------
+ filename : str or pathlib.Path
+ The store's path or URL. A URL is handed to zarr-python
+ unaltered.
+ compute : bool, default=True
+ If ``True``, lazy data is written and the metadata consolidated
+ on exit from the saver context. If ``False``, those writes are
+ returned as a :class:`~dask.delayed.Delayed` by
+ :meth:`delayed_completion`, which also owns the consolidation.
+ mode : str, optional
+ One of ``"w-"``, ``"w"``, ``"a"`` or ``"r+"``. The default is
+ ``"w-"``, which refuses an existing target, or ``"a"`` when
+ `group` is given, since adding a group to a store means the store
+ may already be there.
+ group : str, optional
+ A ``/``-separated path to a group within the store. The root
+ group by default.
+ consolidate_metadata : bool, default=True
+ Whether to write a consolidated metadata document at the root
+ when the save completes. Consolidation is what makes a remote
+ store usable; declining it suits a store that will be added to
+ again before anyone reads it.
+
+ Examples
+ --------
+ >>> import iris # doctest: +SKIP
+ >>> from iris.fileformats.zarr.saver import Saver # doctest: +SKIP
+ >>> with Saver("out.zarr") as saver: # doctest: +SKIP
+ ... for cube in cubes: # doctest: +SKIP
+ ... saver.write(cube) # doctest: +SKIP
+
+ """
+ if mode is None:
+ mode = "a" if group is not None else "w-"
+ dataset = ZarrDataset(filename, mode=mode, group=group)
+ super().__init__(dataset, compute=compute)
+ #: Whether the completed store gets a consolidated metadata document.
+ self.consolidate_metadata = consolidate_metadata
+
+ def write(self, cube, *args, **kwargs):
+ """Write a cube into the store.
+
+ Takes what :meth:`iris.fileformats.cf.saver.CFSaver.write` takes,
+ with Zarr's storage keywords in ``**encoding``. ``chunks`` defaults to
+ the cube's own lazy chunking, so that the data is written in the shape
+ it is held in, and so that the dask source tiles the array's write grid
+ without being rechunked.
+
+ """
+ if "chunks" not in kwargs and cube.has_lazy_data():
+ kwargs["chunks"] = cube.lazy_data().chunksize
+ super().write(cube, *args, **kwargs)
+
+ def _create_string_variable(
+ self, cube_or_mesh, cf_name, element, element_dims, data
+ ):
+ """Create a variable-length string array.
+
+ Zarr has a string type, so unlike netCDF there is no trailing
+ character dimension to invent and `element_dims` is left alone.
+
+ Parameters
+ ----------
+ cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
+ The Cube or Mesh being saved. Unused here.
+ cf_name : str
+ The name the variable is created with.
+ element : :class:`iris.coords._DimensionalMetadata`
+ The element being saved. Unused here.
+ element_dims : list of str
+ The variable's dimension names. **Not** appended to.
+ data : :class:`numpy.ndarray`
+ The string values. Unused here: a Zarr string array has no width.
+
+ Returns
+ -------
+ :class:`~iris.fileformats.zarr._dataset.ZarrDatasetVariable`
+
+ """
+ return self._dataset.create_variable(cf_name, str, element_dims)
+
+
+def save(
+ cube,
+ filename,
+ *,
+ mode=None,
+ group=None,
+ consolidate_metadata=True,
+ local_keys=None,
+ fill_value=None,
+ packing=None,
+ compute=True,
+ **encoding,
+):
+ """Save cube(s) to a Zarr store.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube` or :class:`iris.cube.CubeList`
+ The cube, or cubes, to save. A whole :class:`~iris.cube.CubeList`
+ goes into one store, which is opened once.
+ filename : str or pathlib.Path
+ The store's path or URL.
+ mode : str, optional
+ See :class:`Saver`. ``"w-"`` by default, ``"a"`` when `group` is
+ given.
+ group : str, optional
+ A ``/``-separated path to a group within the store.
+ consolidate_metadata : bool, default=True
+ See :class:`Saver`.
+ local_keys : iterable of str, optional
+ Cube attribute names that become attributes of the data variable
+ rather than global attributes.
+ fill_value : optional
+ The value of the data variable's ``_FillValue`` attribute, and of the
+ array's own fill. One value, or one per cube.
+ packing : type or str or dict or list, optional
+ How to pack the data, if at all. See
+ :meth:`iris.fileformats.netcdf.saver.Saver.write`.
+ compute : bool, default=True
+ If ``False``, returns a :class:`~dask.delayed.Delayed` that completes
+ the store when computed, instead of writing lazy data here.
+ **encoding
+ Zarr's own storage keywords for the data variable: ``chunks``,
+ ``shards``, ``compressors``, ``filters``, ``serializer``, ``order``.
+ A netCDF keyword raises :class:`TypeError` naming the Zarr
+ equivalent.
+
+ Returns
+ -------
+ None or :class:`dask.delayed.Delayed`
+ A ``Delayed`` if `compute` is ``False``, otherwise ``None``.
+
+ Notes
+ -----
+ An interrupted save leaves a partial store. Iris offers no rollback: a
+ directory rename is atomic on a local filesystem but has no equivalent on
+ an object store, so the guarantee would evaporate where Zarr is most used.
+
+ """
+ # Validated before anything is created, so that a mistyped keyword does
+ # not leave an empty store behind.
+ check_encoding_keywords(encoding)
+
+ cubes = CubeList([cube]) if isinstance(cube, iris.cube.Cube) else CubeList(cube)
+ fill_values = _per_cube(fill_value, len(cubes), "fill_value")
+ packspecs = _per_cube(packing, len(cubes), "packing")
+
+ with Saver(
+ filename,
+ compute=compute,
+ mode=mode,
+ group=group,
+ consolidate_metadata=consolidate_metadata,
+ ) as saver:
+ for one, packspec, one_fill in zip(cubes, packspecs, fill_values):
+ saver.write(
+ one,
+ local_keys=local_keys,
+ packing=packspec,
+ fill_value=one_fill,
+ **encoding,
+ )
+
+ conventions = cf_saver.CF_CONVENTIONS_VERSION
+ if iris.FUTURE.save_split_attrs:
+ saver.update_global_attributes({"Conventions": conventions})
+ else:
+ saver.update_global_attributes(Conventions=conventions)
+
+ return None if compute else saver.delayed_completion()
+```
+
+with the small helper the two per-cube keywords share:
+
+```python
+def _per_cube(value, count, name):
+ """Return one `value` per cube, from one value or a sequence of them."""
+ if isinstance(value, (list, tuple)):
+ if len(value) != count:
+ message = (
+ f"If {name} is a sequence, it must have one element per cube: "
+ f"got {len(value)} for {count} cubes."
+ )
+ raise ValueError(message)
+ return list(value)
+ return [value] * count
+```
+
+**What this deliberately does not do.** The netCDF `save` carries
+`unlimited_dimensions`, the `cf_profile` conventions patch and
+`iris.config.netcdf.conventions_override`. The first has no Zarr meaning and is
+refused by name. The other two are a netCDF site-configuration hook; a new
+format with no users starts without them, and `CFSaver.write` still runs
+`cf_profile` itself, which is where the per-cube half of that hook lives.
+
+- [ ] **Step 4: Export the new names**
+
+In `lib/iris/fileformats/zarr/__init__.py`, extend `__all__` and the
+`__getattr__` forwarding. The deferred import is load-bearing — see the
+docstring there — so add to it rather than replacing it with a plain import:
+
+```python
+__all__ = ["Saver", "load_cubes", "save"]
+
+#: What `__getattr__` forwards, and the module it lives in. The import is
+#: deferred because `._dataset` imports `zarr` at module level, and this
+#: package promises that nothing does so until a Zarr target is used.
+_FORWARDED = {
+ "load_cubes": ".loader",
+ "save": ".saver",
+ "Saver": ".saver",
+}
+```
+
+```python
+def __getattr__(name):
+ """Forward the public names to the modules that define them.
+
+ ...
+
+ """
+ module_name = _FORWARDED.get(name)
+ if module_name is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ _check_zarr_version()
+
+ try:
+ module = importlib.import_module(module_name, __name__)
+ except ImportError as exception:
+ raise ImportError(_MISSING_MESSAGE) from exception
+
+ return getattr(module, name)
+```
+
+Add `import importlib` beside `import importlib.metadata`, and update the
+docstring's first paragraph, which says the package reads stores:
+
+```python
+Reads Zarr version 2 and version 3 stores, local or remote, and writes version
+3 ones. Not NCZarr: that is netCDF-c's own Zarr driver, reached through a URL
+fragment such as ``file:///path/store.zarr#mode=nczarr,file``, and it stays
+where it is in :mod:`iris.fileformats.netcdf`.
+```
+
+- [ ] **Step 5: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr -v -k "Saver or save"
+```
+
+Expected: every case in `test_Saver.py` and `test_save.py` passes except the
+deferred-save ones, which Task 8 adds. `test_dependency_errors.py` and
+`test_optional_dependency.py` from PR 4 must still pass: they are what hold the
+promise that importing `iris.fileformats.zarr` does not import `zarr`.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add -A
+git commit -m "Add the Zarr saver, and iris.fileformats.zarr.save"
+```
+
+---
+
+## Task 8: Deferred writes, aligned, and the mask declared
+
+Spec §4.5, "Deferred writes", "The write-alignment invariant" and "The write
+proxy". The task that makes `iris.save(cubes, "out.zarr", compute=False)` a
+real thing, and the one where getting it wrong loses data silently rather than
+raising.
+
+Three things happen in `_backend_write`, which `CFSaver._lazy_stream_data`
+calls with both the data and the variable — the one place where this saver sees
+them together:
+
+1. a masked source gets a `_FillValue` attribute declared, **before** the
+ metadata is consolidated;
+2. a lazy source is rechunked to the array's write grid, which is what licenses
+ `lock=False`;
+3. the aligned source is collected for one `da.store`.
+
+**Files:**
+- Modify: `lib/iris/fileformats/zarr/saver.py` — `_backend_write`,
+ `delayed_completion`, `_align_to_write_grid`, `_tiles_the_grid`
+- Modify: `lib/iris/fileformats/zarr/_dataset.py` — `write_grid` on the variable
+- Modify: `lib/iris/tests/unit/fileformats/zarr/saver/test_Saver.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/saver/test_alignment.py`
+
+**Interfaces:**
+- Consumes: Task 6's attribute encoding and filling; Task 7's `Saver`.
+- Produces: `ZarrDatasetVariable.write_grid -> tuple`, the shards if the array
+ is sharded and the chunks otherwise; `Saver._backend_write`;
+ `Saver.delayed_completion() -> Delayed`, a real one that owns `finalise()`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `lib/iris/tests/unit/fileformats/zarr/saver/test_alignment.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The write-alignment invariant.
+
+Spec §4.5: every Zarr array Iris writes has a write grid that the lazy
+source's dask chunking tiles exactly, where the write grid is the shards when
+the array is sharded and the chunks otherwise. Each concurrent task owns at
+least one whole cell; no cell is touched by two.
+
+The invariant is what licenses ``lock=False``. Stated over the chunk grid
+alone it is insufficient, and the gap is not theoretical: eight inner chunks
+sharing one shard means eight tasks read-modify-writing one stored object,
+which lost 72-96 of 128 values in 12 of 12 trials.
+"""
+
+import dask.array as da
+import numpy as np
+import pytest
+
+zarr = pytest.importorskip("zarr")
+
+from iris.coords import DimCoord
+from iris.cube import Cube
+from iris.fileformats.zarr.saver import Saver, _tiles_the_grid
+
+
+class TestTilesTheGrid:
+ def test_equal_chunks_tile(self):
+ assert _tiles_the_grid(((8, 8, 8),), (8,))
+
+ def test_a_whole_multiple_tiles(self):
+ # 128-value chunks over a 64-value grid: each task owns two cells.
+ assert _tiles_the_grid(((128, 128),), (64,))
+
+ def test_a_smaller_chunk_does_not(self):
+ # The corruption case: eight chunks per shard.
+ assert not _tiles_the_grid(((8,) * 16,), (64,))
+
+ def test_a_ragged_final_chunk_is_allowed(self):
+ # "A truncated final cell is permitted."
+ assert _tiles_the_grid(((64, 64, 7),), (64,))
+
+ def test_an_unaligned_boundary_does_not(self):
+ assert not _tiles_the_grid(((70, 58),), (64,))
+
+ def test_every_dimension_must_tile(self):
+ assert not _tiles_the_grid(((64, 64), (8, 8)), (64, 64))
+
+
+class TestTheInvariantHolds:
+ def test_a_sharded_target_keeps_every_value(self, tmp_path):
+ # The §4.5 reproduction, through the saver. Without the rechunk this
+ # loses 72-96 of 128 values, every time.
+ values = np.arange(128, dtype=np.float32)
+ cube = Cube(da.from_array(values, chunks=8), var_name="data")
+ cube.add_dim_coord(
+ DimCoord(np.arange(128, dtype=np.float32), var_name="x"), 0
+ )
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube, chunks=(8,), shards=(64,))
+ written = zarr.open_group(store=tmp_path / "out.zarr", mode="r")["data"]
+ np.testing.assert_array_equal(written[:], values)
+
+ def test_the_source_is_rechunked_to_the_shard(self, tmp_path):
+ cube = Cube(
+ da.from_array(np.arange(128, dtype=np.float32), chunks=8),
+ var_name="data",
+ )
+ cube.add_dim_coord(
+ DimCoord(np.arange(128, dtype=np.float32), var_name="x"), 0
+ )
+ with Saver(tmp_path / "out.zarr", compute=False) as saver:
+ saver.write(cube, chunks=(8,), shards=(64,))
+ sources = [source for source, _ in saver._delayed_writes]
+ data_source = [s for s in sources if s.shape == (128,)][0]
+ assert data_source.chunks == ((64, 64),)
+
+ def test_an_aligned_source_is_left_alone(self, tmp_path):
+ # A needless rechunk is a needless graph layer.
+ cube = Cube(
+ da.from_array(np.arange(128, dtype=np.float32), chunks=64),
+ var_name="data",
+ )
+ cube.add_dim_coord(
+ DimCoord(np.arange(128, dtype=np.float32), var_name="x"), 0
+ )
+ with Saver(tmp_path / "out.zarr", compute=False) as saver:
+ saver.write(cube, chunks=(8,), shards=(64,))
+ sources = [source for source, _ in saver._delayed_writes]
+ data_source = [s for s in sources if s.shape == (128,)][0]
+ assert data_source.chunks == ((64, 64),)
+
+ def test_an_unsharded_target_grids_on_chunks(self, tmp_path):
+ cube = Cube(
+ da.from_array(np.arange(64, dtype=np.float32), chunks=3),
+ var_name="data",
+ )
+ cube.add_dim_coord(DimCoord(np.arange(64, dtype=np.float32), var_name="x"), 0)
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube, chunks=(16,))
+ written = zarr.open_group(store=tmp_path / "out.zarr", mode="r")["data"]
+ np.testing.assert_array_equal(written[:], np.arange(64, dtype=np.float32))
+```
+
+and append to `test_Saver.py`:
+
+```python
+class TestDeferredSaving:
+ @pytest.fixture
+ def lazy_cube(self, cube):
+ import dask.array as da
+
+ lazy = cube.copy(da.from_array(cube.data, chunks=(1, 3)))
+ return lazy
+
+ def test_compute_false_returns_a_real_delayed(self, tmp_path, lazy_cube):
+ # Q5 and SciTools/iris#7291: the netCDF path returns a tuple here, and
+ # the documented `result.compute()` raises AttributeError on it.
+ from dask.delayed import Delayed
+
+ from iris.fileformats.zarr.saver import save
+
+ result = save(lazy_cube, tmp_path / "out.zarr", compute=False)
+ assert isinstance(result, Delayed)
+
+ def test_the_data_lands_when_it_is_computed(self, tmp_path, lazy_cube):
+ from iris.fileformats.zarr.saver import save
+
+ result = save(lazy_cube, tmp_path / "out.zarr", compute=False)
+ result.compute()
+ written = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ np.testing.assert_array_equal(
+ written["air_temperature"][:], lazy_cube.data
+ )
+
+ def test_consolidation_follows_the_last_chunk(self, tmp_path, lazy_cube):
+ # Spec §4.5: under compute=False the Delayed owns finalise(), so a
+ # store that is never completed never claims to be.
+ from iris.fileformats.zarr.saver import save
+
+ result = save(lazy_cube, tmp_path / "out.zarr", compute=False)
+ before = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert before.metadata.consolidated_metadata is None
+ result.compute()
+ after = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert after.metadata.consolidated_metadata is not None
+
+ def test_the_whole_graph_survives_a_pickle(self, tmp_path, lazy_cube):
+ # F7: `write_handle()` returns the ZarrDatasetVariable, not the
+ # zarr.Array the spec checked. A distributed scheduler pickles the
+ # graph, so the substitution has to be re-checked.
+ import pickle
+
+ from iris.fileformats.zarr.saver import save
+
+ result = save(lazy_cube, tmp_path / "out.zarr", compute=False)
+ revived = pickle.loads(pickle.dumps(result))
+ revived.compute()
+ written = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ np.testing.assert_array_equal(
+ written["air_temperature"][:], lazy_cube.data
+ )
+
+
+class TestMaskedSaving:
+ def test_a_masked_cube_declares_a_fill_value(self, tmp_path, cube):
+ # Spec §4.5: without the attribute, every masked point reads back
+ # unmasked - a round trip that silently loses the mask.
+ masked = cube.copy(np.ma.masked_greater(cube.data, 3.0))
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(masked)
+ array = zarr.open_group(store=tmp_path / "out.zarr", mode="r")[
+ "air_temperature"
+ ]
+ assert "_FillValue" in dict(array.attrs)
+
+ def test_a_lazy_masked_cube_declares_one_too(self, tmp_path, cube):
+ import dask.array as da
+
+ masked = cube.copy(da.ma.masked_greater(da.from_array(cube.data), 3.0))
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(masked)
+ array = zarr.open_group(store=tmp_path / "out.zarr", mode="r")[
+ "air_temperature"
+ ]
+ assert "_FillValue" in dict(array.attrs)
+
+ def test_the_mask_round_trips(self, tmp_path, cube):
+ masked = cube.copy(np.ma.masked_greater(cube.data, 3.0))
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(masked)
+ loaded = iris.load_cube(tmp_path / "out.zarr")
+ np.testing.assert_array_equal(loaded.data.mask, masked.data.mask)
+
+ def test_an_unmasked_cube_declares_nothing(self, tmp_path, cube):
+ with Saver(tmp_path / "out.zarr") as saver:
+ saver.write(cube)
+ array = zarr.open_group(store=tmp_path / "out.zarr", mode="r")[
+ "air_temperature"
+ ]
+ assert "_FillValue" not in dict(array.attrs)
+```
+
+- [ ] **Step 2: Run them, and watch them fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/saver -x
+```
+
+Expected: `ImportError: cannot import name '_tiles_the_grid'`, and the
+`TestDeferredSaving` cases failing on the tuple `da.store` returns.
+
+- [ ] **Step 3: Add `write_grid` to the variable**
+
+In `_dataset.py`, with the other Zarr-only members of `ZarrDatasetVariable`:
+
+```python
+ @property
+ def write_grid(self) -> tuple:
+ """The unit a concurrent writer must own whole cells of.
+
+ The shards when the array is sharded, the chunks when it is not. A
+ shard is one stored object holding many inner chunks, so two tasks
+ writing different chunks of one shard read-modify-write the same
+ object and one update is lost. Stating the invariant over the chunk
+ grid alone is what xarray's ``safe_chunks`` had to be corrected to;
+ see pydata/xarray#10831.
+
+ """
+ return tuple(self._array.shards or self._array.chunks)
+```
+
+- [ ] **Step 4: Add the alignment helpers to the saver module**
+
+```python
+def _tiles_the_grid(chunks, grid) -> bool:
+ """Whether every chunk boundary falls on a write-grid boundary.
+
+ Parameters
+ ----------
+ chunks : tuple of tuple of int
+ A dask array's ``chunks``: the lengths along each dimension.
+ grid : tuple of int
+ The write-grid cell shape.
+
+ Returns
+ -------
+ bool
+ True when no grid cell is touched by two chunks. A truncated final
+ cell is permitted, so the last boundary is not checked.
+
+ """
+ for lengths, cell in zip(chunks, grid):
+ offset = 0
+ for length in lengths[:-1]:
+ offset += length
+ if offset % cell:
+ return False
+ return True
+
+
+def _align_to_write_grid(data, cf_var):
+ """Return `data` chunked so that concurrent writes cannot collide.
+
+ Parameters
+ ----------
+ data : dask.array.Array
+ The lazy source about to be stored.
+ cf_var : :class:`~iris.fileformats.zarr._dataset.ZarrDatasetVariable`
+ The array it is going into.
+
+ Returns
+ -------
+ dask.array.Array
+ `data` itself when it already tiles the write grid - a needless
+ rechunk is a needless graph layer - and a rechunked view otherwise.
+
+ """
+ grid = cf_var.write_grid
+ if not grid or _tiles_the_grid(data.chunks, grid):
+ return data
+ return data.rechunk(grid)
+```
+
+- [ ] **Step 5: Implement `_backend_write`**
+
+On `Saver`:
+
+```python
+ def _backend_write(self, data, cf_var) -> bool:
+ """Declare a mask, align a lazy source, and collect it.
+
+ Parameters
+ ----------
+ data : array-like
+ The values to write.
+ cf_var : :class:`~iris.fileformats.zarr._dataset.ZarrDatasetVariable`
+ The variable to write them into.
+
+ Returns
+ -------
+ bool
+ True when the write was collected here, which it is for lazy
+ data. Real data is left to the base class, which writes it
+ straight in.
+
+ """
+ if (
+ is_lazy_masked_data(data) or np.ma.isMaskedArray(data)
+ ) and "_FillValue" not in cf_var.attributes:
+ # On version 3 the attribute is the masking declaration and the
+ # storage fill is not, so a store written with the fill alone
+ # reads back with every masked point unmasked. Declared here
+ # rather than when the data is written, because under
+ # compute=False that is after the metadata is consolidated.
+ cf_var.attributes["_FillValue"] = cf_var.fill_value
+
+ if not is_lazy_data(data):
+ return False
+
+ self._delayed_writes.append(
+ (_align_to_write_grid(data, cf_var), cf_var.write_handle())
+ )
+ return True
+```
+
+Import `is_lazy_data` and `is_lazy_masked_data` from `iris._lazy_data`, and
+`numpy as np`.
+
+- [ ] **Step 6: Implement `delayed_completion`**
+
+```python
+ def delayed_completion(self) -> Delayed:
+ """Return a :class:`~dask.delayed.Delayed` that completes the store.
+
+ The lazy writes first, then the metadata consolidation, which is the
+ store's last write: a store whose deferred save never runs should not
+ carry a metadata document claiming it did.
+
+ Returns
+ -------
+ :class:`dask.delayed.Delayed`
+ A real one. ``dask.array.store(..., compute=False)`` returns a
+ *tuple* when given a sequence of sources, which is what makes the
+ documented ``result.compute()`` fail on the netCDF path
+ (`SciTools/iris#7291 `_).
+ Wrapping it here is what keeps the promise this saver's docstring
+ makes.
+
+ """
+ # Bound locally, not through `self`: this closure is serialised to
+ # every worker of a distributed scheduler, and the saver holds cubes.
+ dataset = self._dataset
+ consolidating = self.consolidate_metadata
+
+ @dask.delayed
+ def complete(_written):
+ if consolidating:
+ dataset.finalise()
+
+ return complete(super().delayed_completion())
+```
+
+Import `dask` and `from dask.delayed import Delayed`.
+
+**Why `finalise()` is not wired into `CFSaver` instead.** It would put
+`self._dataset` into a delayed closure for every format, and a
+`NetCDFDataset` holds a `netCDF4.Dataset`, which does not pickle. The generic
+saver stays as PR 5 left it.
+
+- [ ] **Step 7: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr -v
+```
+
+Expected: all pass, including `test_alignment.py`.
+
+Then check the invariant the way the spec checked it — the failure is
+probabilistic without the fix, so run it under the threaded scheduler a few
+times:
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/saver/test_alignment.py \
+ -k sharded --count=5 2>/dev/null || \
+for i in 1 2 3 4 5; do \
+ pytest lib/iris/tests/unit/fileformats/zarr/saver/test_alignment.py -k sharded -q; \
+done
+```
+
+Expected: five passes. (`--count` needs `pytest-repeat`, which is not a
+dependency; the loop is the fallback.)
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add -A
+git commit -m "Defer Zarr writes, aligned to the write grid"
+```
+
+---
+
+## Task 9: Reach the saver from `iris.save`
+
+Spec §4.7, items 3 and 4. PR 4 already taught `find_saver` the `.zarr` suffix
+and added `_CUBELIST_SAVER_MODULES`; what is missing is the `_savers` entry
+that maps the suffix to something, and the module's place in that frozenset.
+
+**Files:**
+- Modify: `lib/iris/io/__init__.py` — `_check_init_savers`,
+ `_CUBELIST_SAVER_MODULES`
+- Modify: `lib/iris/fileformats/zarr/__init__.py` — the registration shim
+- Modify: `lib/iris/tests/results/file_load/known_loaders.txt` if it lists
+ savers too; check before editing
+- Create: `lib/iris/tests/unit/io/test_find_saver__zarr.py`
+
+**Interfaces:**
+- Consumes: Task 7's `save`.
+- Produces: `iris.save(cubes, "out.zarr")`; `iris.fileformats.zarr._save_cubes`,
+ the shim `_savers["zarr"]` holds.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `lib/iris/tests/unit/io/test_find_saver__zarr.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The Zarr saver is reachable from :func:`iris.save`."""
+
+import sys
+
+import numpy as np
+import pytest
+
+import iris
+from iris.cube import Cube, CubeList
+import iris.io
+
+
+@pytest.fixture
+def cubes():
+ return CubeList(
+ [
+ Cube(np.arange(4, dtype=np.float32), var_name=name)
+ for name in ("a", "b")
+ ]
+ )
+
+
+class TestFindSaver:
+ @pytest.mark.parametrize(
+ "target", ["out.zarr", "out.zarr/", "s3://bucket/out.zarr", "OUT.ZARR"]
+ )
+ def test_the_suffix_finds_a_saver(self, target):
+ assert iris.io.find_saver(target) is not None
+
+ def test_it_is_the_zarr_one(self):
+ saver = iris.io.find_saver("out.zarr")
+ assert saver.__module__ == "iris.fileformats.zarr"
+
+ def test_a_netcdf_target_is_unaffected(self):
+ saver = iris.io.find_saver("out.nc")
+ assert saver.__module__ == "iris.fileformats.netcdf.saver"
+
+
+class TestRegistrationIsLazy:
+ def test_finding_the_saver_does_not_import_zarr(self):
+ # The optional-dependency promise: `iris.save(cube, "out.pp")` must
+ # not import zarr on the way past.
+ code = (
+ "import sys, iris.io;"
+ "iris.io.find_saver('out.zarr');"
+ "print('zarr' in sys.modules)"
+ )
+ import subprocess
+
+ completed = subprocess.run(
+ [sys.executable, "-c", code], capture_output=True, text=True
+ )
+ assert completed.stdout.strip() == "False", completed.stderr
+
+
+class TestSavingACubeList:
+ def test_the_whole_list_goes_into_one_store(self, tmp_path, cubes):
+ # F8: without the module in _CUBELIST_SAVER_MODULES, iris.save loops
+ # over the cubes, and the second one meets a store that mode="w-"
+ # refuses.
+ pytest.importorskip("zarr")
+ import zarr
+
+ iris.save(cubes, tmp_path / "out.zarr")
+ group = zarr.open_group(store=tmp_path / "out.zarr", mode="r")
+ assert {"a", "b"} <= set(group.array_keys())
+
+ def test_keywords_reach_the_saver(self, tmp_path, cubes):
+ pytest.importorskip("zarr")
+
+ with pytest.raises(TypeError, match="chunks"):
+ iris.save(cubes, tmp_path / "out.zarr", chunksizes=(2,))
+```
+
+- [ ] **Step 2: Run them, and watch them fail**
+
+```bash
+pytest lib/iris/tests/unit/io/test_find_saver__zarr.py -x
+```
+
+Expected: `assert None is not None` — `find_saver` matches the suffix but
+`_savers` has no `zarr` key.
+
+- [ ] **Step 3: Add the registration shim**
+
+In `lib/iris/fileformats/zarr/__init__.py`, below `__getattr__`:
+
+```python
+def _save_cubes(cube, target, **kwargs):
+ """Save to a Zarr store, importing zarr-python only when called.
+
+ What :data:`iris.io._savers` holds for the ``zarr`` extension. A plain
+ reference to :func:`iris.fileformats.zarr.saver.save` cannot be
+ registered: reaching it imports ``zarr``, and registration happens the
+ first time *any* format is saved.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube` or :class:`iris.cube.CubeList`
+ What to save.
+ target : str or pathlib.Path
+ The store's path or URL.
+ **kwargs : dict
+ Passed to :func:`iris.fileformats.zarr.saver.save`.
+
+ Returns
+ -------
+ None or :class:`dask.delayed.Delayed`
+
+ """
+ from .saver import save
+
+ return save(cube, target, **kwargs)
+```
+
+It is private and stays out of `__all__`: `save` is the public name.
+
+- [ ] **Step 4: Register it**
+
+In `lib/iris/io/__init__.py`, `_check_init_savers`:
+
+```python
+def _check_init_savers():
+ from iris.fileformats import netcdf, pp, zarr
+
+ if "pp" not in _savers:
+ _savers.update(
+ {
+ "pp": pp.save,
+ "nc": netcdf.save,
+ "zarr": zarr._save_cubes,
+ "dot": _dot_save,
+ "dotpng": _dot_save_png,
+ "grib2": _grib_save,
+ }
+ )
+```
+
+`from iris.fileformats import zarr` imports the package, not zarr-python: the
+package's module body has no `import zarr` in it, which is the whole point of
+the `__getattr__` there.
+
+and extend the frozenset, which PR 4 created with one member:
+
+```python
+#: Modules whose `save` takes a CubeList directly, rather than one cube at a
+#: time with `append=True`. One line per format.
+_CUBELIST_SAVER_MODULES = frozenset(
+ {
+ "iris.fileformats.netcdf.saver",
+ "iris.fileformats.zarr",
+ }
+)
+```
+
+The Zarr entry names the package rather than `zarr.saver`, because the
+registered function is the shim above and `__module__` is what the dispatch
+reads. Say so in a comment, or the next person will "fix" it.
+
+- [ ] **Step 5: Check the format-listing test**
+
+```bash
+grep -rn "zarr" lib/iris/tests/results/file_load/known_loaders.txt
+pytest lib/iris/tests/unit/io lib/iris/tests/unit/fileformats/test_rules.py -q
+```
+
+PR 4 updated `known_loaders.txt` for the loader. If any test enumerates savers,
+it needs the new entry; if none does, there is nothing to change here.
+
+- [ ] **Step 6: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/io lib/iris/tests/unit/fileformats/zarr -q -n auto
+```
+
+Expected: all pass.
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add -A
+git commit -m "Reach the Zarr saver from iris.save"
+```
+
+---
+
+## Task 10: Round trips, and what the format does not promise
+
+Spec §6, "Integration", and §4.5, "Durability". Everything so far tests one
+piece. This tests the thing a user does — save a cube, load it back, get the
+same cube — and pins the two format behaviours Iris depends on, so that a
+zarr-python release changing either of them fails loudly here rather than
+quietly in someone's data.
+
+**Files:**
+- Create: `lib/iris/tests/integration/zarr/test_save.py`
+
+**Interfaces:**
+- Consumes: everything from Tasks 4-9.
+- Produces: nothing other tasks use.
+
+- [ ] **Step 1: Write the tests**
+
+Create `lib/iris/tests/integration/zarr/test_save.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Integration tests for saving cubes to a Zarr store."""
+
+import numpy as np
+import pytest
+
+zarr = pytest.importorskip("zarr")
+
+import iris
+from iris.cube import CubeList
+import iris.tests.stock as stock
+
+
+@pytest.fixture
+def cube():
+ # The signature is `realistic_4d_w_everything(w_mesh=False)`: there is no
+ # `lazy` keyword, and the cube it returns holds real data.
+ return stock.realistic_4d_w_everything()
+
+
+class TestRoundTrip:
+ def test_the_cube_comes_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ loaded = iris.load_cube(path)
+ assert loaded.standard_name == cube.standard_name
+ assert loaded.units == cube.units
+ assert loaded.shape == cube.shape
+ np.testing.assert_allclose(loaded.data, cube.data)
+
+ def test_the_coordinates_come_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ loaded = iris.load_cube(path)
+ saved_names = sorted(coord.name() for coord in cube.coords())
+ loaded_names = sorted(coord.name() for coord in loaded.coords())
+ assert loaded_names == saved_names
+
+ def test_the_bounds_come_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ loaded = iris.load_cube(path)
+ for coord in cube.coords():
+ if coord.has_bounds():
+ np.testing.assert_allclose(
+ loaded.coord(coord.name()).bounds, coord.bounds
+ )
+
+ def test_the_cell_methods_come_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ assert iris.load_cube(path).cell_methods == cube.cell_methods
+
+ def test_the_coordinate_system_comes_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ loaded = iris.load_cube(path)
+ assert loaded.coord_system() == cube.coord_system()
+
+ def test_attribute_values_come_back(self, tmp_path, cube):
+ # Values, not dtypes: spec §4.5 trades the NumPy dtype of an
+ # attribute for output every Zarr consumer can read.
+ path = tmp_path / "out.zarr"
+ cube.attributes["a_number"] = np.float32(1.5)
+ cube.attributes["a_list"] = np.array([1, 2], dtype=np.int16)
+ iris.save(cube, path)
+ loaded = iris.load_cube(path)
+ assert loaded.attributes["a_number"] == 1.5
+ assert list(loaded.attributes["a_list"]) == [1, 2]
+
+ def test_a_masked_cube_keeps_its_mask(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ masked = cube.copy(np.ma.masked_greater(cube.data, cube.data.mean()))
+ iris.save(masked, path)
+ loaded = iris.load_cube(path)
+ np.testing.assert_array_equal(loaded.data.mask, masked.data.mask)
+
+ def test_several_cubes_round_trip(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ other = cube[0].copy()
+ other.rename("air_pressure")
+ other.var_name = "air_pressure"
+ iris.save(CubeList([cube, other]), path)
+ assert len(iris.load(path)) == 2
+
+ def test_a_group_round_trips(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ iris.save(cube, path, group="again")
+ loaded = iris.load_cube(path, group="again")
+ assert loaded.shape == cube.shape
+
+
+class TestWhatTheFormatDoesNotPromise:
+ # Spec §4.5: neither of these is fixable from inside Iris. They are
+ # pinned so that a zarr-python release changing one fails here.
+ def test_a_missing_chunk_reads_back_filled(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ chunks = sorted((path / "air_potential_temperature" / "c").rglob("*"))
+ removed = [p for p in chunks if p.is_file()][0]
+ removed.unlink()
+ # Opens cleanly, and under-reports data, with no warning of any kind.
+ reloaded = iris.load_cube(path)
+ assert reloaded.data.shape == cube.shape
+
+ def test_consolidated_metadata_goes_stale(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ # Add a group behind Iris's back, without re-consolidating.
+ zarr.open_group(store=path, mode="a", path="sneaked").create_array(
+ "data", shape=(2,), dtype="f4", dimension_names=("x",)
+ )
+ reader = zarr.open_group(store=path, mode="r")
+ assert "sneaked" not in list(reader.group_keys())
+ # Which is why Iris re-consolidates when it adds a group itself.
+ iris.save(cube, path, group="properly")
+ reader = zarr.open_group(store=path, mode="r")
+ assert "properly" in list(reader.group_keys())
+
+
+class TestRefusals:
+ def test_an_existing_store_is_not_clobbered(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ with pytest.raises(FileExistsError, match="mode"):
+ iris.save(cube, path)
+ # And the first save is still intact.
+ assert iris.load_cube(path).shape == cube.shape
+
+ def test_adding_a_name_that_exists_is_refused(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ with pytest.raises(ValueError, match=cube.var_name):
+ iris.save(cube, path, mode="a")
+```
+
+- [ ] **Step 1b: Add the inputs the round trip above does not reach**
+
+Five input classes the design implies, that nothing so far exercises, and that
+a user meets in ordinary work. Append to the same file:
+
+```python
+class TestInputsTheDesignImplies:
+ def test_a_mesh_cube_round_trips(self, tmp_path):
+ # Iris supports unstructured meshes and PR 4 taught load_meshes about
+ # stores, so a mesh cube reaching the Zarr saver is ordinary. The mesh
+ # variables go through the same generic path as everything else, and
+ # nothing above has proved it.
+ path = tmp_path / "mesh.zarr"
+ mesh_cube = stock.realistic_4d_w_everything(w_mesh=True)
+ iris.save(mesh_cube, path)
+ loaded = iris.load_cube(path)
+ assert loaded.mesh is not None
+ assert loaded.location == mesh_cube.location
+ assert loaded.mesh.topology_dimension == mesh_cube.mesh.topology_dimension
+
+ def test_packed_data_round_trips(self, tmp_path, cube):
+ # `packing` writes scale_factor and add_offset and narrows the dtype.
+ # Both attributes are numbers the JSON conversion touches, and the
+ # read side unpacks with them.
+ path = tmp_path / "packed.zarr"
+ iris.save(cube, path, packing="i2")
+ stored = zarr.open_group(store=path, mode="r")["air_potential_temperature"]
+ assert stored.dtype == np.dtype("i2")
+ assert "scale_factor" in dict(stored.attrs)
+ loaded = iris.load_cube(path)
+ np.testing.assert_allclose(loaded.data, cube.data, rtol=1e-3)
+
+ def test_int64_data_survives(self, tmp_path, cube):
+ # The one dtype the saver has narrowing logic for. A Zarr store holds
+ # int64, so `supports_dtype` answers True and nothing is narrowed -
+ # which is a different answer from NETCDF3, and worth pinning.
+ path = tmp_path / "int64.zarr"
+ wide = cube.copy(cube.data.astype(np.int64))
+ iris.save(wide, path)
+ assert iris.load_cube(path).dtype == np.dtype("i8")
+
+ def test_a_non_ascii_attribute_survives(self, tmp_path, cube):
+ # A degree sign in a title is ordinary, and netCDF's ASCII-or-bytes
+ # coercion does not apply here: JSON is text.
+ path = tmp_path / "unicode.zarr"
+ cube.attributes["title"] = "Temperature in °C, naïve grid"
+ iris.save(cube, path)
+ assert iris.load_cube(path).attributes["title"] == (
+ "Temperature in °C, naïve grid"
+ )
+
+ def test_a_dataless_cube_saves(self, tmp_path, cube):
+ # `Cube` supports a dataless form, and the saver has a branch for it
+ # that chooses its own dtype and fill value. Nothing above reaches it.
+ path = tmp_path / "dataless.zarr"
+ dataless = iris.cube.Cube(
+ None, shape=cube.shape, standard_name=cube.standard_name,
+ units=cube.units, var_name=cube.var_name,
+ )
+ iris.save(dataless, path)
+ assert iris.load_cube(path).shape == cube.shape
+```
+
+- [ ] **Step 2: Run them**
+
+```bash
+pytest lib/iris/tests/integration/zarr/test_save.py -v
+```
+
+Expected: all pass. If `test_a_missing_chunk_reads_back_filled` cannot find a
+chunk object, print the store tree first — version 3 puts chunks under
+``c/``, but a sharded or differently keyed array would not, and the test
+should look wherever this store actually keeps them.
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add lib/iris/tests/integration/zarr/test_save.py
+git commit -m "Test the Zarr save round trip, and what the format does not promise"
+```
+
+---
+
+## Task 11: xarray reads what Iris writes
+
+Spec §6, "Cross-reader conformance": "bidirectional... xarray must open every
+store Iris writes". This is the check that justifies the JSON-native attribute
+decision and the base64 `_FillValue` encoding — if xarray cannot read Iris
+output, both were wrong.
+
+**Files:**
+- Modify: `requirements/py312.yml`, `py313.yml`, `py314.yml`
+- Create: `lib/iris/tests/integration/zarr/test_cross_reader.py`
+
+**Interfaces:**
+- Consumes: everything from Tasks 4-9.
+- Produces: nothing other tasks use.
+
+- [ ] **Step 1: Add the dependency**
+
+In each of `requirements/py312.yml`, `py313.yml` and `py314.yml`, add `xarray`
+to the **test** section, in alphabetical order — after `requests`, before
+nothing:
+
+```yaml
+# Test dependencies.
+ - asv_runner
+ - distributed
+ - filelock
+ - imagehash >=4.0
+ - pre-commit
+ - psutil
+ - pytest <9.1
+ - pytest-cov
+ - pytest-mock
+ - pytest-xdist
+ - requests
+ - xarray
+```
+
+**Do not regenerate `requirements/locks/*.lock`.** PR 4 regenerates all three
+to add `zarr`, and doing it again here guarantees a conflict and a second slow
+environment rebuild on rebase. The human developer regenerates them by hand
+once PR 4 has landed (F5). Until then CI has no xarray and the tests below
+skip, which the pull-request body must say.
+
+- [ ] **Step 2: Write the tests**
+
+Create `lib/iris/tests/integration/zarr/test_cross_reader.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Iris and xarray read each other's Zarr stores.
+
+The check that justifies two decisions in the design. Attributes are written
+as JSON-native values, losing their NumPy dtype, *so that* every other Zarr
+consumer can read them; and a floating ``_FillValue`` is written as base64,
+against CF section 2.5.1, because released xarray fails the whole dataset open
+on a plain JSON number.
+
+Both are only worth anything if someone checks, and xarray is the reader to
+check against: it is what most of the Zarr world reads with.
+"""
+
+import numpy as np
+import pytest
+
+zarr = pytest.importorskip("zarr")
+xr = pytest.importorskip("xarray")
+
+import iris
+from iris.coords import DimCoord
+from iris.cube import Cube
+
+
+def small_cube(dtype="f4", masked=False):
+ values = np.arange(6, dtype=dtype).reshape(2, 3)
+ if masked:
+ values = np.ma.masked_array(values, mask=[[False, True, False]] * 2)
+ cube = Cube(values, standard_name="air_temperature", units="K")
+ cube.var_name = "air_temperature"
+ cube.add_dim_coord(
+ DimCoord(np.array([0.0, 1.0], dtype="f4"), standard_name="latitude",
+ units="degrees", var_name="latitude"),
+ 0,
+ )
+ cube.add_dim_coord(
+ DimCoord(np.array([0.0, 1.0, 2.0], dtype="f4"),
+ standard_name="longitude", units="degrees",
+ var_name="longitude"),
+ 1,
+ )
+ return cube
+
+
+class TestXarrayReadsIris:
+ @pytest.mark.parametrize("dtype", ["f4", "f8", "i2", "i4"])
+ def test_a_store_opens(self, tmp_path, dtype):
+ # Asserted on the open, not on the values: the failure mode is a
+ # TypeError during open that takes the whole dataset with it.
+ path = tmp_path / f"out_{dtype}.zarr"
+ iris.save(small_cube(dtype=dtype), path)
+ dataset = xr.open_zarr(path)
+ assert "air_temperature" in dataset.data_vars
+
+ @pytest.mark.parametrize("dtype", ["f4", "f8", "i2"])
+ def test_a_masked_store_opens_and_masks(self, tmp_path, dtype):
+ path = tmp_path / f"masked_{dtype}.zarr"
+ iris.save(small_cube(dtype=dtype, masked=True), path)
+ values = xr.open_zarr(path)["air_temperature"].values
+ assert np.isnan(values[0, 1])
+ assert not np.isnan(values[0, 0])
+
+ def test_an_explicit_finite_sentinel_masks(self, tmp_path):
+ path = tmp_path / "sentinel.zarr"
+ iris.save(small_cube(masked=True), path, fill_value=np.float32(-99.0))
+ values = xr.open_zarr(path)["air_temperature"].values
+ assert np.isnan(values[0, 1])
+
+ def test_the_dimensions_and_coordinates_are_xarrays_too(self, tmp_path):
+ path = tmp_path / "dims.zarr"
+ iris.save(small_cube(), path)
+ dataset = xr.open_zarr(path)
+ assert dataset["air_temperature"].dims == ("latitude", "longitude")
+ np.testing.assert_allclose(dataset["latitude"].values, [0.0, 1.0])
+
+ def test_attributes_survive_as_values(self, tmp_path):
+ path = tmp_path / "attrs.zarr"
+ cube = small_cube()
+ cube.attributes["a_number"] = np.float32(1.5)
+ cube.attributes["a_list"] = np.array([1, 2], dtype=np.int16)
+ cube.attributes["text"] = "words"
+ iris.save(cube, path)
+ attributes = xr.open_zarr(path)["air_temperature"].attrs
+ assert attributes["a_number"] == 1.5
+ assert list(attributes["a_list"]) == [1, 2]
+ assert attributes["text"] == "words"
+
+ def test_a_consolidated_store_opens_consolidated(self, tmp_path):
+ path = tmp_path / "consolidated.zarr"
+ iris.save(small_cube(), path)
+ # The default: xarray uses the consolidated document when it is there.
+ assert "air_temperature" in xr.open_zarr(path).data_vars
+
+
+class TestIrisReadsXarray:
+ @pytest.mark.parametrize(
+ "fill, dtype",
+ [(np.nan, "f4"), (-99.0, "f4"), (-9, "i2")],
+ ids=["float32-nan", "float32-sentinel", "int16-sentinel"],
+ )
+ def test_a_store_xarray_wrote_loads_masked(self, tmp_path, fill, dtype):
+ path = tmp_path / f"xarray_{dtype}_{fill}.zarr"
+ values = np.array([[1, 2, 3], [4, 5, 6]], dtype=dtype)
+ values[0, 1] = fill if fill == fill else values[0, 1]
+ dataset = xr.Dataset(
+ {
+ "air_temperature": (
+ ("latitude", "longitude"),
+ np.ma.masked_equal(values, fill)
+ if fill == fill
+ else np.where(np.arange(6).reshape(2, 3) == 1, np.nan, values),
+ )
+ },
+ coords={"latitude": [0.0, 1.0], "longitude": [0.0, 1.0, 2.0]},
+ )
+ dataset["air_temperature"].encoding["_FillValue"] = fill
+ dataset.to_zarr(path, zarr_format=3)
+ cube = iris.load_cube(path)
+ assert cube.data.mask[0, 1]
+```
+
+- [ ] **Step 3: Run them**
+
+```bash
+pytest lib/iris/tests/integration/zarr/test_cross_reader.py -v
+```
+
+Expected: all pass, in the `iris-dev` environment where xarray is installed.
+Check they *skip* rather than error when it is not:
+
+```bash
+conda run -n iris-dev python -c "import xarray" && echo installed
+pytest lib/iris/tests/integration/zarr/test_cross_reader.py -q -p no:cacheprovider \
+ --co | tail -2
+```
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add requirements/py312.yml requirements/py313.yml requirements/py314.yml \
+ lib/iris/tests/integration/zarr/test_cross_reader.py
+git commit -m "Check that xarray reads what Iris writes, and Iris reads xarray's"
+```
+
+---
+
+## Task 12: Documentation
+
+Spec §5 (PR 6): "documents saving alongside the loading documentation from
+PR 4, including the S3 pages". PR 4 wrote a `Saving` section whose whole
+content is that Iris cannot. Replacing it is most of this task.
+
+**Files:**
+- Modify: `docs/src/user_manual/explanation/zarr_io.rst`
+- Modify: `docs/src/user_manual/tutorial/s3_io.rst`
+
+**Interfaces:**
+- Consumes: the finished behaviour of Tasks 4-9.
+- Produces: nothing other tasks use.
+
+- [ ] **Step 1: Replace the `Saving` section**
+
+In `docs/src/user_manual/explanation/zarr_io.rst`, the `Saving` section
+currently says Iris does not save to a native store. Replace its body with:
+
+```rst
+Saving
+------
+
+A store is a directory, or a URL, so pass it where you would pass a file::
+
+ iris.save(cubes, "/data/forecast.zarr")
+
+Iris writes Zarr **version 3** stores. A whole :class:`~iris.cube.CubeList`
+goes into one store, which is opened once.
+
+Unlike the netCDF saver, this one does **not** overwrite. ``iris.save`` raises
+if the target already exists, because replacing a store means deleting many
+objects one at a time, and an interrupted replacement destroys what was there
+without completing what replaces it. Say so explicitly to opt in::
+
+ iris.save(cubes, "/data/forecast.zarr", mode="w") # replace
+ iris.save(cubes, "/data/forecast.zarr", mode="a") # add to
+
+``mode="a"`` adds a *new* array or group to a store that already holds others.
+It never extends, rewrites, resizes or re-chunks an array that is already
+there, and a name that already exists in the target group is an error rather
+than an overwrite. Appending to data is not supported, for Zarr or for netCDF:
+deciding whether incoming data is compatible with what a store already holds is
+a data-model problem, and the storage format does not solve it.
+
+Storage keywords
+^^^^^^^^^^^^^^^^
+
+The Zarr saver takes **zarr-python's** vocabulary, not netCDF's::
+
+ iris.save(
+ cube,
+ "out.zarr",
+ chunks=(1, 50, 50),
+ shards=(4, 100, 100),
+ compressors=[zarr.codecs.BloscCodec(cname="zstd")],
+ )
+
+``chunks``, ``shards``, ``compressors``, ``filters`` and ``serializer`` are
+passed through as zarr-python defines them. A netCDF keyword such as
+``zlib=True`` raises :class:`TypeError` naming the Zarr replacement where there
+is one, and saying there is none where there is not. The two compression models
+differ, and a keyword that let you say ``complevel=4`` while quietly giving you
+Blosc would be misleading rather than convenient.
+
+``chunks`` defaults to the cube's own lazy chunking, so data is written in the
+shape it is held in.
+
+Deferred saving
+^^^^^^^^^^^^^^^
+
+``compute=False`` returns a :class:`dask.delayed.Delayed` that writes the lazy
+data when computed, and consolidates the store's metadata after the last chunk::
+
+ result = iris.save(cubes, "out.zarr", compute=False)
+ result.compute()
+
+Consolidated metadata
+^^^^^^^^^^^^^^^^^^^^^
+
+Iris writes a consolidated metadata document at the store's root when the save
+completes, because it is what makes a remote store usable: a reader fetches one
+object instead of one per array. Pass ``consolidate_metadata=False`` to decline.
+
+When Iris adds to a store that already carries one, it rewrites it. A
+consolidated document that does not mention what was just written is what a
+default reader believes, so stale consolidation hides data.
+
+What an interrupted save leaves behind
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A partial store, and Iris offers no rollback. Zarr has no multi-object
+transaction: a store's metadata and its chunks are separate objects, so a save
+interrupted between them leaves the two disagreeing, and a missing chunk is not
+an error — it reads back as the fill value, with no warning. The usual
+mitigation, writing to a temporary name and renaming on success, is not
+adopted: a directory rename is atomic on a local filesystem but object stores
+have no equivalent, so the guarantee would evaporate exactly where Zarr is most
+used. Callers who need transactional writes want `Icechunk
+`_.
+```
+
+- [ ] **Step 2: Extend the `Missing data` section**
+
+Append to it, after the paragraph about version 2 stores:
+
+```rst
+On save, Iris writes the missing-value declaration as well as the storage fill:
+a masked cube gets a ``_FillValue`` attribute even when it carried none of its
+own, because on version 3 the attribute is what declares missing data and the
+storage ``fill_value`` is not. Without it, every masked point would read back
+unmasked. Pass ``fill_value=`` to choose the sentinel; the default is the same
+one netCDF uses for that dtype.
+
+A floating ``_FillValue`` is written as the base64 of a little-endian
+``float64``, which is what xarray and the wider Zarr ecosystem read. This
+deviates from CF section 2.5.1, which asks for a value of the variable's own
+type. The deviation is deliberate and load-bearing: a plain JSON number there
+makes the whole store unreadable by default xarray, which is a worse outcome
+than a convention deviation that every reader understands.
+```
+
+- [ ] **Step 3: Update the S3 page**
+
+`docs/src/user_manual/tutorial/s3_io.rst` has a section on Zarr stores being
+accessed directly, which PR 4 wrote for loading. Add saving to it::
+
+```rst
+Saving works the same way::
+
+ iris.save(cubes, "s3://bucket/forecast.zarr")
+
+The same store library and credentials apply, and the same defaults: an
+existing store is not overwritten unless you pass ``mode="w"``, and the
+metadata is consolidated when the save completes, which is what makes the store
+cheap to open again over the network.
+```
+
+Check the page's "TL;DR" paragraph: PR 4 rewrote it for loading, and it may
+still say saving is unsupported.
+
+- [ ] **Step 4: Build the documentation**
+
+```bash
+cd docs && make html SPHINXOPTS="-W --keep-going" 2>&1 | tail -20
+```
+
+Expected: no warnings from the two pages. A broken reference is an error under
+`-W`; `:class:`dask.delayed.Delayed`` resolves through intersphinx, and
+`zarr.codecs.BloscCodec` in a literal block is not a reference at all, so it
+needs no target.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add docs/
+git commit -m "Document saving to Zarr"
+```
+
+---
+
+## Task 13: Record it, and open the pull request
+
+`docs/AGENTS.md` and spec §12. The design document is the live record for this
+programme, and an entry written now is worth more than one reconstructed later.
+
+**Files:**
+- Modify: `docs/superpowers/specs/2026-09-21-zarr-io-design.md` — §12.1, §12.3,
+ §12.4, §12.6
+
+**Interfaces:**
+- Consumes: the finished work of Tasks 1-12.
+- Produces: nothing code depends on.
+
+- [ ] **Step 1: Run everything**
+
+```bash
+export NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX"
+pytest lib/iris/tests/unit lib/iris/tests/integration -q -n auto
+```
+
+Expected: `11188 passed` plus this branch's new cases, and no failures. The
+baseline count is F1; a failure outside the files these tasks touched is a
+rebase artefact, not something to fix here.
+
+```bash
+pre-commit run --all-files
+```
+
+Expected: clean, after any auto-fixes are re-staged.
+
+- [ ] **Step 2: Record the decisions in §12.4**
+
+Append one entry, dated, saying what was decided and not how it was reached:
+
+```markdown
+**2026-10-15 — PR 6, Zarr saving**
+
+- Grid-mapping parameters reach a non-netCDF store through a fifth hook,
+ `_grid_mapping_writer`. netCDF returns its own variable, so what lands in a
+ netCDF file is unchanged; everything else gets a writer onto `.attributes`.
+ Q8's regularisation stays out of this programme.
+- `_attribute_handlers` moved to `cf/`, closing the last two-way import
+ between the CF and netCDF packages.
+- `zarr/_encode.py` added to §4.1's layout, holding the write-side value
+ conversions as pure functions, mirroring `_decode.py`.
+- `missing_value` is written as a plain JSON number, not base64. Only
+ `_FillValue` takes the packed encoding: xarray reads the first as an
+ ordinary attribute, where a packed string masks nothing, and the second
+ through zarr's fill-value parser, which fails the whole dataset open on a
+ plain number.
+- A masked source with no declared fill takes netCDF's default fill for its
+ dtype, for every dtype including floating, and the value is always written
+ as `_FillValue`. zarr-python's own default is 0 for every dtype, which would
+ make a masked point of a float array read back as a real zero.
+- The write-alignment invariant is enforced by rechunking the lazy source to
+ the array's write grid at write time, not by deriving the grid from the
+ source at create time: `create_variable` never sees the data. The data
+ variable's `chunks` defaults to the cube's own lazy chunking.
+- `finalise()` is wired into the Zarr saver's `delayed_completion`, not into
+ `CFSaver`'s: the generic version would put a `netCDF4.Dataset` into a
+ delayed closure, and it does not pickle.
+- `xarray` added to the test section of the requirement files; the lock files
+ are regenerated by hand once #7336 lands, so CI skips the cross-reader
+ tests until then.
+```
+
+- [ ] **Step 3: Close what this answers in §12.3**
+
+Q8 is *not* closed — the encoding regularisation it asks about is still
+undone. Update its "Blocks" column to say what changed:
+
+```markdown
+| Q8 | ... | ... | §4.5 | Nothing. PR 6 added the `_grid_mapping_writer` hook, so a non-netCDF store writes these through `.attributes` while netCDF keeps its own object and its own encoding |
+```
+
+Q5 stays open for netCDF and gains the Zarr half of its answer:
+
+```markdown
+| Q5 | ... | ... | [#7291](https://github.com/SciTools/iris/issues/7291) | Nothing. `zarr/saver.py` returns a real `Delayed` and tests `.compute()`; the netCDF path is unchanged |
+```
+
+- [ ] **Step 4: Update §12.1 and §12.6**
+
+§12.1: PR 6's row becomes `Drafted` with the branch, then `In review` with the
+number once the pull request is open. Record it when it is opened, not when it
+merges.
+
+§12.6: one row, in date order:
+
+```markdown
+| 2026-10-15 | PR 6 built, on a branch merging PR 4 and PR 5. Grid-mapping writing made store-agnostic through a fifth hook (Q8 unblocked, not closed); `_attribute_handlers` relocated to `cf/`; `zarr/_encode.py` added to §4.1; `missing_value` separated from `_FillValue` on the write side; the default fill made netCDF's for every dtype; the write-alignment invariant enforced by rechunking at write time. Seven decisions logged. |
+```
+
+- [ ] **Step 5: Commit the record**
+
+```bash
+git add docs/superpowers/specs/2026-09-21-zarr-io-design.md
+git commit -m "Record Zarr saving in the design document"
+```
+
+- [ ] **Step 6: Open the pull request**
+
+Push the branch, then open it **as a draft**, because its diff contains PR 4's
+and PR 5's work until at least one of them lands (§2):
+
+```bash
+git push -u origin zarr-io-pr-6
+```
+
+The body opens with `(posted by Claude Opus)`, before anything else — this
+replaces any trailing attribution the harness asks for. Never hard-wrap it.
+Fill in the repository's pull-request template, and say these five things:
+
+- **`Part of #6980`**, never a closing keyword. The issue closes at merge-back.
+- **The base.** What the branch is stacked on, which of PR 4 and PR 5 is still
+ outstanding, and that the branch will be rebased with
+ `git rebase --onto brownfield zarr-io-pr-4-5-base zarr-io-pr-6` once both
+ have landed.
+- **No changelog fragment**, by the programme's rule: fragments are written
+ once, at merge-back.
+- **The lock files are not regenerated**, so CI has no xarray and the
+ cross-reader conformance tests skip. They pass locally; name the command a
+ reviewer can run to see it.
+- **The one netCDF-facing change**, the `_grid_mapping_writer` hook, and why it
+ cannot change a netCDF file: the override returns the same object the
+ parameters have always been assigned to.
+
+Add the labels `Agentic` and `Type: Feature Branch`.
+
+---
+
+## Self-review notes
+
+Checked against the spec with fresh eyes, 2026-10-15.
+
+**Spec coverage.** Every §5 item for PR 6 has a task: `zarr/saver.py` (7), the
+`ZarrDataset` write path (4, 5, 6), JSON attribute conversion (3, 6),
+masked-data filling (3, 6, 8), deferred writes (8), the Zarr-native encoding
+keywords (5, 7), `consolidate_metadata` (6, 7), registering the saver (9),
+documentation (12), and each of §6's named write tests (8, 10, 11). Two things
+§5 assigns to PR 6 are in tasks of their own because they are not Zarr:
+`_attribute_handlers` (1) and the grid-mapping hook (2).
+
+**Not covered, deliberately**, each with its reason in §1: version 2 writing,
+appending, multi-process writes, the Q8 encoding regularisation, the netCDF
+`compute=False` return type, #7329's attribute check, PR 7's data and
+benchmarks, the changelog, and the lock files.
+
+**Where this plan departs from the spec**, all recorded as findings and all
+with the spec's own reasoning behind them: the `missing_value` encoding (F10),
+the default fill value (F11), where the write grid is decided (F12), and
+`write_handle` returning the wrapper rather than the array (F7). Each names the
+check that showed the spec's text would not do what it intends.
+
+**Type consistency.** `write_grid` is defined in Task 8 and used only there.
+`check_encoding_keywords` is module-level in Task 5 and called from Task 7's
+`save`. `_encode.json_native`, `encode_fill_value`, `encode_missing_values`,
+`default_fill_value` and `filled` are defined in Task 3 with the signatures
+Tasks 5, 6 and 8 call them with. `_grid_mapping_writer` is one name across
+Tasks 2 and 7. `Saver.consolidate_metadata` is set in Task 7 and read in
+Task 8.
+
+**One correction made during review.** Task 10's fixture called
+`stock.realistic_4d_w_everything(lazy=True)`; the function's only parameter is
+`w_mesh`. Fixed, and the mesh case now uses `w_mesh=True`.
+
+**What a reviewer should look at hardest**, in order: Task 8's
+`_backend_write`, because it is where a mistake loses data rather than raising;
+Task 6's attribute encoding, because a wrong answer there is readable by Iris
+and by nothing else; and Task 2, because it is the only change that can affect
+a netCDF file, and the argument that it cannot is an identity, not a
+comparison.
diff --git a/docs/superpowers/specs/2026-09-21-zarr-io-design.md b/docs/superpowers/specs/2026-09-21-zarr-io-design.md
index 298e80e62c..2a8287aa3c 100644
--- a/docs/superpowers/specs/2026-09-21-zarr-io-design.md
+++ b/docs/superpowers/specs/2026-09-21-zarr-io-design.md
@@ -2,19 +2,21 @@
> **This is a living document.** §12 is the single progress record for the
> programme — pull request states, open questions and the decision log. Update
-> §12 as work lands; do not track status anywhere else.
+> §12 as work lands; do not track status anywhere else. The table below is a
+> summary of §12.1 and nothing more, so it moves when §12.1 moves: a reader
+> who stops at the header must not be told something §12 contradicts.
| | |
|---|---|
| **Phase** | In delivery |
-| **Progress** | PRs 1, 2, 2.5, 3 and 5 merged into `brownfield`; PR 4 in review ([#7336](https://github.com/SciTools/iris/pull/7336)); merge-back not started — see §12.1 |
-| **Next action** | PR 4's review, then PR 6 — see §12.1 |
+| **Progress** | PRs 1, 2, 2.5, 3, 4 and 5 merged into `brownfield`; PR 6 in review ([#7346](https://github.com/SciTools/iris/pull/7346)); merge-back not started — see §12.1 |
+| **Next action** | PR 6's review, then PRs 7 and 8 — see §12.1 |
| **Blocked on** | Nothing |
| **Branch** | `zarr-io-design` on `bjlittle/iris`, targeting `SciTools/iris:brownfield` |
| **Delivery** | Nine pull requests into `brownfield`, then one merge-back into `main` — §5 |
| **Resolves** | SciTools/iris#6977, #6979, #6980 — at merge-back, not before |
-**Created:** 2026-09-21 · **Last updated:** 2026-10-20 (see §12.6)
+**Created:** 2026-09-21 · **Last updated:** 2026-10-23 (see §12.6)
**Baseline:** `brownfield` at `b0ab90b98`, Iris `3.17.0.dev4`
**Verified against:** zarr-python 3.4.0, numcodecs 0.16.5, Python 3.13,
dask 2026.7.1, xarray 2026.7.0
@@ -228,6 +230,7 @@ lib/iris/fileformats/
__init__.py # NEW public: load_cubes, save, Saver
_dataset.py # NEW ZarrDataset, ZarrDatasetVariable
_decode.py # NEW CF masking and unpacking that netCDF4 does for free
+ _encode.py # NEW CF packing and attribute conversion that netCDF4 does for free
loader.py # NEW
saver.py # NEW
```
@@ -367,6 +370,35 @@ this store hold this dtype?", so that is what the interface offers.
for every dtype Zarr has a data type for. The error message that follows
names `format`, so the user still reads `NETCDF3_CLASSIC`.
+**The question is narrower than it looks, so `ZarrDataset` answers `True`
+and stops.** `_ensure_valid_dtype` is the only generic caller, and it asks
+only about `int64` and the unsigned integers — the types a netCDF 3 file has
+no room for. Zarr has a data type for every one of them, so nothing is ever
+narrowed for a Zarr store. A dtype Zarr genuinely cannot hold is refused by
+zarr-python inside `create_variable`, with a `ValueError` naming the dtype,
+and the group is left as it was **[verified]**.
+
+Answering the wider question here as well would mean Iris keeping a second
+table of what zarr-python can store. That was tried, and it was wrong in
+both directions: it refused `datetime64`, an ordinary structured dtype, a
+zero-length `U0` and an eight-byte void, all of which zarr-python stores
+**[verified]**, while no saver ever asked about any of them. A table like
+that also goes stale as zarr-python gains data types, and nothing checks it.
+
+**Byte order is taken out of the question by `create_variable`.** A Zarr
+array is created in the platform's own byte order whatever order it was
+asked for, and NumPy converts on assignment, so the values stored are the
+caller's and only the layout is the store's. Zarr version 3 has no
+big-endian data type: endianness lives in the `bytes` codec, which
+zarr-python leaves at `little`, so asking for `>f4` writes the same store as
+asking for `f4` cube is `float32` in the file it writes **[verified]** — so
+the Zarr path is matching it, not inventing a second rule. A PP or GRIB load
+hands the saver big-endian arrays, so this is ordinary Iris data.
+
#### `format` and `storage_settings`
`format` names the concrete format an implementation provides — `"NETCDF4"`,
@@ -1028,9 +1060,16 @@ lives in `ZarrDataset`:
- which codecs are available;
- whether sharding is offered.
-`ZarrDataset.__init__` takes `zarr_format: int = 3`. Today it raises
-`NotImplementedError` for anything but 3, in one place. Adding version 2 write
-means implementing those three points and deleting that guard.
+`ZarrDataset.__init__` takes `zarr_format: int | None = None`, matching
+`NetCDFDataset.__init__`'s `netcdf_format=None`. A write resolves `None` to
+`_DEFAULT_ZARR_FORMAT` and raises `NotImplementedError` for a version outside
+`_WRITABLE_ZARR_FORMATS`, in one place. Adding version 2 write means
+implementing those three points and adding 2 to that tuple.
+
+A read drops the value rather than passing it on. zarr-python does not ignore
+`zarr_format` on a read, it constrains detection: a store whose version
+disagrees is reported as a missing group, which names the wrong problem.
+Verified against zarr-python 3.4.1.
#### Dimensions
@@ -1516,10 +1555,12 @@ Four changes make the chain reach that specification:
raising, and routes a local path to it when the path is a directory.
3. `iris.io.find_saver` matches `.zarr` with or without a trailing separator,
since `iris.save(cube, "out.zarr/")` is a natural thing to type.
-4. `iris.io.save` gets a module-level `_CUBELIST_SAVER_MODULES` frozenset
- naming the modules whose `save` accepts a `CubeList` directly, replacing the
- `"iris.fileformats.netcdf" not in saver.__module__` substring test at
- io/__init__.py:530. Explicit and greppable; one line per new format.
+4. `iris.io.save` decides which savers accept a `CubeList` directly from
+ `iris.io._savers.cubelist_savers`, a set of the saver functions
+ themselves, replacing the `"iris.fileformats.netcdf" not in
+ saver.__module__` substring test. Registered by identity, so a saver a
+ caller passes by hand is recognised as readily as one found by
+ extension.
**`load_files` is not taught about directories.** Do not have it skip its
`open(fn, "rb")` and pass `None` as the buffer when the path is a directory.
@@ -1723,8 +1764,8 @@ Also carries three changes to generic code that only Zarr can exercise:
its three overrides match on `(name, size)` pairs with an anonymous
dimension matching nothing (§4.4). netCDF behaviour is unchanged by
construction, and the tests say so on both backends.
-- `encoding` on `CFDataset` and `CFDatasetVariable` (§4.2), reporting what the
- opened store actually holds.
+- `storage_settings` on `CFDataset` and `CFDatasetVariable` (§4.2), reporting
+ what the opened store actually holds.
- `format` on `CFDatasetVariable` (§4.2), naming the concrete Zarr version so
the version-aware masking split (§4.4) can tell `ZARR_V2` from `ZARR_V3`.
`CFDataset.format` arrived with PR 5, which merged first; this pull request
@@ -1881,15 +1922,27 @@ The `CFDataset` contract gets one shared test body run against both
implementations, so a netCDF/Zarr divergence fails loudly. Stores are built
in-memory or in `tmp_path`.
-That shared body is also where `format`, `encoding` and `supports_dtype` are
-pinned, since the point of each is that it works the same way whatever is
-underneath: `format` names the concrete format; `encoding` reports the
-settings an opened store actually holds, and round-trips what
-`create_variable` was given; and `supports_dtype` answers `False` for `int64`
-on `NETCDF3_CLASSIC` and `True` on `NETCDF4` and on Zarr. A separate check
-greps the `cf` package for `isinstance` against an implementation class and
-asserts there are none — the rule in §4.2 is only worth writing down if
-something notices when it is broken.
+That shared body is also where `format`, `storage_settings` and
+`supports_dtype` are pinned, since the point of each is that it works the
+same way whatever is underneath: `format` names the concrete format;
+`storage_settings` reports the settings an opened store actually holds, and
+round-trips what `create_variable` was given; and `supports_dtype` answers
+`False` for `int64` on `NETCDF3_CLASSIC` and `True` on `NETCDF4` and on Zarr.
+A separate check greps the `cf` package for `isinstance` against an
+implementation class and asserts there are none — the rule in §4.2 is only
+worth writing down if something notices when it is broken.
+
+**A capability answer binds the implementation that gave it, over the
+dtypes a saver asks about.** Every dtype an Iris saver can reach
+`create_variable` with — each numeric width, fixed-length text, and the
+big-endian spellings a PP or GRIB load produces — is swept against
+`ZarrDataset`, asserting both that `supports_dtype` says `True` and that
+`create_variable` then creates it. The sweep does not extend to dtypes no
+saver produces: `supports_dtype` is the question asked before narrowing an
+integer (§4.2), not a general oracle, and a store that answers the wider
+question needs a table of the storage library's data types that nothing
+consults and nothing checks. What is pinned for those instead is that a
+dtype zarr-python refuses leaves the group as it was.
**Integration.** `lib/iris/tests/integration/zarr/` covers round trips through
`stock.realistic_4d_w_everything()`, following the shape of the existing
@@ -1897,18 +1950,27 @@ something notices when it is broken.
**Chunking and decoding** are unit-testable without a store, which is the main
practical dividend of dropping the proxy (§4.4). Each CF decode rule is a pure
-function of `(block, parameters)` and is tested on plain ndarrays. The
-alignment guard is tested on the `chunking` values alone:
+function of `(block, parameters)` and is tested on plain ndarrays. The sizing
+is tested on the `chunking` values alone:
- a store chunk smaller than the dask target expands to a whole multiple;
-- a store chunk larger than the dask target is **not** subdivided, and warns;
-- an explicit `CHUNK_CONTROL` setting that subdivides is honoured, and warns;
-- the resulting dask chunks tile the store chunk grid exactly, in every case;
+- a store chunk larger than the dask target **is** subdivided, because Iris
+ sizes with `_optimum_chunksize` and does not then correct for the store's
+ own layout (§4.4, 2026-10-20) — and the load warns, naming the variable and
+ `CHUNK_CONTROL`;
+- an explicit `CHUNK_CONTROL` setting is honoured whether it subdivides or
+ not, and does **not** warn: a caller who has pinned a dimension has already
+ taken that trade, and `AS_DASK` hands the whole decision to Dask;
- a **sharded** array chunks on `Array.chunks`, not `Array.shards`, and a
partial read of one inner chunk does not fetch the whole shard — asserted on
bytes fetched through an instrumented store, with the decode layer present,
since that layer is what defeats slice pushdown (§4.4).
+There is no read-side alignment requirement to test: the guard that would
+have made the dask chunking tile the store grid in every case was removed on
+review of #7336 (§4.4), and the warning is what replaced it. Write-side
+alignment is a different invariant, still binding, and tested below.
+
**Fill values and masking** (§4.4) get the cases that the earlier precedence
got wrong, each on a store built in `tmp_path`:
@@ -1941,10 +2003,10 @@ because there is no append.
**Encoding keywords** (§4.5) get one test each way: the Zarr-native names
reach the array (`compressors`, `chunks`, `shards` are readable back off
-`Array.metadata`, and through `CFDatasetVariable.encoding`), and a netCDF name
-such as `zlib=True` raises `TypeError` whose message names the Zarr
-replacement. The second is the one that stops the translation layer growing
-back by accident.
+`Array.metadata`, and through `CFDatasetVariable.storage_settings`), and a
+netCDF name such as `zlib=True` raises `TypeError` whose message names the
+Zarr replacement. The second is the one that stops the translation layer
+growing back by accident.
**Cross-reader conformance** (§4.5) is bidirectional and is the check that
would have caught the `_FillValue` encoding error: xarray opens every store
@@ -2010,8 +2072,13 @@ reader and the NCZarr writer agree.
store only. No test touches the network. The URL path is exercised by checking
that the right store is constructed, not by fetching.
-**Skipping.** A single `pytest.importorskip("zarr")` fixture in the Zarr test
-package. Iris without zarr installed must show no new failures.
+**Skipping.** A `pytest.importorskip("zarr", minversion="3.1.0")` in each Zarr
+test package's `__init__.py`, which covers everything written under it. A test
+written *outside* those packages that reaches the Zarr backend carries its own
+`importorskip` in the test body, rather than at the top of its module: the
+module it is in also holds netCDF tests, and skipping those where Zarr is
+missing would trade a new failure for a silent gap. Iris without zarr
+installed must show no new failures and no quietly skipped netCDF test.
---
@@ -2094,7 +2161,7 @@ preserved because that is what is being tested.
| dynamical.org closes before the subset is pulled | Pull within days; the design does not otherwise depend on it |
| PR 5 is a 3000-line diff | Pure `git mv` plus import edits, no behaviour change, full suite green |
| CF-Zarr conventions are still a moving target | Iris writes plain unprefixed CF attributes, which is what the draft and xarray both do; no `zarr_conventions` block is written until the specification settles |
-| zarr-python uses Effective Effort Versioning, not semantic versioning | Pin `>=3.0.8`, which is above the known data-loss bug, and rely on CI to catch drift |
+| zarr-python uses Effective Effort Versioning, not semantic versioning | Pin `>=3.1.0`, the oldest release this code actually runs on, and rely on CI to catch drift. `>=3.0.8` was chosen for being above the known data-loss bug, but was never executed: 3.0.x returns a `DataType` enum from `Array.metadata.to_dict()`, which `cache_key` cannot serialise, so every lazy read fails **[verified]** |
| Attribute dtype loss surprises someone | Documented; round-trip tests assert values not dtypes; xarray reads the output (§6); base64 remains available as a later opt-in |
| Designing against one publisher's files produces brittle code | Every behaviour in §4 is justified from the Zarr and CF specifications first; a real file is cited only as evidence that a path will be exercised. Where a file is non-conforming it is labelled a malformation and handled with a warning, never by bending the reader (§4.4) |
| Mistaking a de-facto convention for a publisher's mistake — the inverse risk, and the one that actually bit | Before calling a file non-conforming, check whether an independent reader *depends* on the form. The `_FillValue` encoding failed that check: it was xarray's, and the numeric alternative made Iris output unopenable (§4.4). The bidirectional cross-reader tests in §6 are the standing guard |
@@ -2207,9 +2274,9 @@ nothing else, and can land beside any of the others.
| 2 | `CFDataset`, and the CF variable classes rewritten against it | Merged | [#7303](https://github.com/SciTools/iris/pull/7303) |
| 2.5 | Deprecate `CFVariable.__getattr__` | Merged | [#7333](https://github.com/SciTools/iris/pull/7333) |
| 3 | Relocate the CF loader | Merged | [#7316](https://github.com/SciTools/iris/pull/7316) |
-| 4 | Zarr loading | In review | [#7336](https://github.com/SciTools/iris/pull/7336) |
+| 4 | Zarr loading | Merged | [#7336](https://github.com/SciTools/iris/pull/7336) |
| 5 | Relocate the CF saver | Merged | [#7338](https://github.com/SciTools/iris/pull/7338) |
-| 6 | Zarr saving | Not started | — |
+| 6 | Zarr saving | In review | [#7346](https://github.com/SciTools/iris/pull/7346) |
| 7 | Real-world test data and benchmarks | Not started | — |
| 8 | The terms-of-art sweep | Not started | — |
| — | **Merge back** `brownfield` → `main`: closing keywords, changelog fragments, docs | Not started | — |
@@ -2257,11 +2324,10 @@ closing keywords live (§5).
|---|---|---|---|---|
| Q1 | How should the CF attribute model map onto JSON, on write and on read? | JSON-native both ways (§4.4, §4.5) | #7288, §10 | Nothing. Changing it touches only the attribute conversion in `zarr/_dataset.py` |
| Q9 | Should a user be able to tell the loader that two same-named, same-length dimensions in different Zarr arrays are *not* the same dimension? The version 3 specification permits it, since a name guarantees nothing across arrays | No control. Treating that pair as one dimension is what xarray, NCZarr and netCDF all do, so a store meaning otherwise is relying on something no reader implements. Add one if a real store turns up that needs it | §4.4 | Nothing |
-| Q4 | The netCDF read path has the same sub-chunk re-read trap as Zarr, because `NetCDFDataProxy` reopens the `Dataset` on every `__getitem__` and so discards HDF5's chunk cache. Should it get the same alignment guard? | Out of scope here — the relocation pull requests must not change behaviour. Raise it separately once §4.4's Zarr guard has proven itself | §4.4 | Nothing |
-| Q5 | `iris.save(..., compute=False)` returns a tuple, not the documented `Delayed`, so `result.compute()` raises `AttributeError` **[verified]** | Caused by #6451 adapting to dask/dask#11844; the code is right and the docstrings were left behind. Not fixed here — PR 5 is behaviour-preserving. `zarr/saver.py` returns a real `Delayed` and tests it | [#7291](https://github.com/SciTools/iris/issues/7291) | Nothing |
+| Q5 | `iris.save(..., compute=False)` returns a tuple, not the documented `Delayed`, so `result.compute()` raises `AttributeError` **[verified]** | Caused by #6451 adapting to dask/dask#11844; the code is right and the docstrings were left behind. Not fixed here — PR 5 is behaviour-preserving. `zarr/saver.py` returns a real `Delayed` and tests it | [#7291](https://github.com/SciTools/iris/issues/7291) | Nothing. `zarr/saver.py` returns a real `Delayed` and tests `.compute()`; the netCDF path is unchanged |
| Q6 | The `as_lazy_data` cache is process-wide and keyed on metadata, so it cannot detect a store whose chunk contents changed under identical metadata. Should it be scoped to a load session instead? | Metadata identity closes the cases that were reproduced (§4.4) and matches the netCDF key's existing strength, so it ships. Session scoping is the durable fix and would cover both formats | §4.4 | Nothing |
| Q7 | Iris writes a floating `_FillValue` as base64 to stay readable by xarray, deviating from CF §2.5.1. Should the deviation be raised with the CF-Zarr conventions group rather than carried privately? | Carry it now, since the alternative is unreadable output; take it upstream to zarr-conventions/CF so the convention settles rather than each reader guessing | §4.4, §4.5 | Nothing |
-| Q8 | `_add_grid_mapping_to_dataset` sets sixty-three CF grid-mapping parameters by Python attribute assignment, bypassing the ASCII-to-bytes coercion every other saved attribute goes through — so `crs_wkt` is written as `NC_STRING` while `grid_mapping_name` is `NC_CHAR`. Should they be regularised? | Not in this programme. PR 2 kept them on a named netCDF4 handle (`grid_variable`) rather than change what lands in the file. Regularising is a one-line-per-parameter change with a real CDL diff, and belongs in its own pull request | §4.5 | PR 6's grid mappings. PR 5 relocated the method into `cf/saver.py`, where the netCDF4 handle is now the one thing in generic code that a non-netCDF store cannot supply |
+| Q8 | `_add_grid_mapping_to_dataset` sets its CF grid-mapping parameters by Python attribute assignment, bypassing the ASCII-to-bytes coercion every other saved attribute goes through — so `crs_wkt` is written as `NC_STRING` while `grid_mapping_name` is `NC_CHAR`. Should they be regularised? | Not in this programme. PR 2 kept them on a named netCDF4 handle (`grid_variable`) rather than change what lands in the file. Regularising is a one-line-per-parameter change with a real CDL diff, and belongs in its own pull request | §4.5 | Nothing. PR 6 added the `_grid_mapping_writer` hook, which hands back a function taking a parameter name and a value: netCDF's writes onto its own object and keeps its own encoding, every other store's writes into `.attributes`. Review asked for the function rather than an assignment proxy, so that the store-dependent step is visible at each call site; that also leaves the call sites in the shape regularisation needs, reducing it to deleting netCDF's override |
Close a question by moving it to §12.4 with the date and the answer. Do not
delete it.
@@ -2833,6 +2899,134 @@ All four findings reproduced and fixed; none disputed.
request, once PR 6 has shown where the seams between the two formats really
fall.
+**2026-10-15 — PR 6, Zarr saving**
+
+- Grid-mapping parameters reach a non-netCDF store through a fifth hook,
+ `_grid_mapping_writer`, which returns the function that writes one parameter.
+ netCDF's writes onto its own variable, so what lands in a netCDF file is
+ unchanged; everything else writes into `.attributes`. Q8's regularisation
+ stays out of this programme.
+- `_attribute_handlers` moved to `cf/`, closing the last two-way import
+ between the CF and netCDF packages.
+- `zarr/_encode.py` added to §4.1's layout, holding the write-side value
+ conversions as pure functions, mirroring `_decode.py`.
+- `missing_value` is written as a plain JSON number, not base64. Only
+ `_FillValue` takes the packed encoding: xarray reads the first as an
+ ordinary attribute, where a packed string masks nothing, and the second
+ through zarr's fill-value parser, which fails the whole dataset open on a
+ plain number.
+- A masked source with no declared fill takes netCDF's default fill for its
+ dtype, for every dtype that table covers, and the value is always written
+ as `_FillValue`. zarr-python's own default is 0 for every dtype, which would
+ make a masked point of a float array read back as a real zero — so a dtype
+ the table does not cover, such as `float16`, is refused when the data turns
+ out to be masked, and saves normally when it does not.
+- The write-alignment invariant is enforced by rechunking the lazy source to
+ the array's write grid at write time, not by deriving the grid from the
+ source at create time: `create_variable` never sees the data. The data
+ variable's `chunks` defaults to the cube's own lazy chunking, where the cube
+ has one and the caller passed no `shards`.
+- `finalise()` is wired into the Zarr saver's `delayed_completion`, not into
+ `_CFSaver`'s: the generic version would put a `netCDF4.Dataset` into a
+ delayed closure, and it does not pickle.
+- `xarray` added to the test section of the requirement files. The lock files
+ were deliberately left alone while #7336 was outstanding, since that pull
+ request regenerated all three and doing it twice guarantees a conflict and
+ a second slow rebuild; they were regenerated by hand once it landed, so CI
+ runs the cross-reader tests rather than skipping them.
+- `packing=` is applied by the Zarr write path itself, not inherited from
+ netCDF's. The generic saver sets `scale_factor` and `add_offset` as
+ attributes and streams the data unchanged, because netCDF4-python applies
+ the transform itself on assignment; zarr-python has no such behaviour. A
+ `pack` function in `zarr/_encode.py`, mirroring `_decode.unpack`, does the
+ arithmetic for Zarr, applied by `ZarrDatasetVariable.__setitem__` before the
+ masked fill. The Zarr saver refuses a non-integer packing dtype, matching
+ the netCDF saver's own check and its message.
+- `iris.save` reaches the Zarr saver through `iris.io._zarr_save`, a deferred-
+ import wrapper beside `_dot_save` and `_grib_save`, which is where review
+ asked for it. It imports `save` from the package, not from `.saver`, so the
+ package's `__getattr__` runs and with it the zarr version check: an
+ installation with no zarr, or too old a zarr, gets the same message from
+ `iris.save` as from `iris.load`. An earlier `_save_cubes` shim in the
+ package repeated those checks by hand, because importing from `.saver`
+ bypasses `__getattr__`; importing from the package makes it redundant.
+- Both routes to a Zarr save are registered as taking a `CubeList` whole: the
+ `_zarr_save` wrapper, registered in `iris.io._check_init_savers` beside the
+ netCDF saver, and `iris.fileformats.zarr.save` itself, which registers at
+ the foot of `zarr/saver.py`. A caller reaches the same save either way -
+ by extension, or by passing the saver as `saver=` - and only the first was
+ recognised, so the second looped over the cubes and the second cube met a
+ store `mode="w-"` refuses. The saver registers itself rather than being
+ named in `iris.io`, because `iris.io` must not import `zarr` to list it,
+ and by the time `zarr/saver.py` is imported the dependency is loaded
+ anyway. Review caught it; a test now covers both routes.
+- A cube with a mesh saves, carrying the full CF-UGRID structure. It was
+ written against a loader that could not rebuild a mesh from a store, so
+ the round trip was a strict `xfail`; the mesh loading added to #7336 on
+ 2026-10-22 closed that gap, and the `xfail` came off when this branch was
+ rebased onto it. The round trip is an ordinary passing test.
+- `mode="a"` writes new coordinate arrays beside the ones a store already
+ holds, rather than matching and reusing them — a second `latitude` is
+ written as `latitude_0`. Deciding when two coordinates are the same across
+ unequal values, differing units and different calendars is its own piece of
+ work. A data variable's own name is the one collision that is an error
+ rather than a rename.
+- A masked string variable is **refused**, not written. The only sentinel a
+ Zarr string array could carry is the empty string, which is also a
+ legitimate label — and `zarr/_decode.py` does not restore a mask from one:
+ it warns that the attribute is neither a number nor the base64 of a
+ float64, and reads the empty strings back as data. So writing it would lose
+ the mask in silence, which is the failure this design is least willing to
+ accept. The netCDF saver cannot store a masked string coordinate either, so
+ refusing is parity with it rather than a step back from it. The message
+ names the variable and offers no `fill_value` to pass, because there is
+ none that would work.
+- The deferred completion is a module-level function, with the dataset and
+ the consolidation flag bound as plain values rather than reached through
+ `self`, so the `Delayed` a deferred save returns survives a plain `pickle`
+ round trip and not only cloudpickle.
+
+**2026-10-15 — actioning the whole-branch review of PR 6**
+
+Every finding sat in the seam between the generic saver and the Zarr one:
+`cf/saver.py` had only ever been driven by a saver that creates its store
+from scratch, so the places where "the store already holds something"
+matters were not all swept. All reproduced; none disputed.
+
+- The dimensions a group already carries seed the generic saver's
+ `_existing_dim`, so that `_get_dim_names` disambiguates against them. Two
+ cubes with no dim coord both call their dimension `dim0`, and the generic
+ layer declares nothing for a name it already sees — so a second save sized
+ its array from the store's length rather than the cube's, and zarr-python
+ truncated the write in silence. §4.5.
+- `ZarrDatasetVariable.__setitem__` refuses a value whose shape does not
+ match the selection, naming the array. The second line of defence under
+ the same rule: Zarr neither resizes an array nor reports an oversized
+ write. §4.5.
+- The cube-attribute marshalling that decides which attributes a store holds
+ globally is now `cf.saver.marshal_attributes`, called by both savers. It
+ was written into `netcdf/saver.save` and the Zarr saver did not have it, so
+ every cube global attribute was dropped under split-attribute saving, and a
+ `CubeList` whose cubes disagreed collapsed to the last cube's value under
+ the legacy one. The netCDF path is unchanged except that the deprecation
+ message now names its format; the Zarr path now emits that deprecation too,
+ which it should, the legacy mode being deprecated for every format. §4.5.
+- A dtype with no entry in the netCDF default-fill table falls back to
+ zarr-python's own default rather than raising, so `float16` and `S1` save.
+ `supports_dtype` answers True for every NumPy dtype, and a store that
+ refused what it said it could hold contradicted it. The refusal moved to
+ where the mask is known. §4.5.
+- `json_native` recurses into a `list` or `tuple`, so a sequence of NumPy
+ scalars — `[data.min(), data.max()]` — is an attribute, as it is for
+ netCDF. §4.5.
+- `order` is refused rather than accepted and discarded: zarr-python takes it
+ for a version 2 array and Iris writes version 3, where it has no effect.
+ `append` gets a message naming `mode="a"`. §4.5.
+- `chunks` no longer defaults to the cube's own lazy chunking when `shards`
+ is given: a shard holds a whole number of inner chunks, and Dask's chunking
+ has no reason to divide one the user chose, so the pair failed over a
+ number the user never passed. §4.5.
+
**2026-10-16 — Q10 closed, at review of #7338 by @stephenworsley**
- #7337 is fixed in PR 5, not carried across it. Saving int64 data to
@@ -2853,6 +3047,129 @@ All four findings reproduced and fixed; none disputed.
format of the file that was opened, which is the honest answer and the same
rule every other format follows.
+**2026-10-23 — Q4 closed, and §12.3 reconciled, at the Codex review of #7346**
+
+- Q4 asked whether the netCDF read path should get the same read-side
+ alignment guard as Zarr. There is no such guard on either path: it was
+ removed on 2026-10-20, leaving both backends sizing with
+ `_optimum_chunksize` and both warning when a stored chunk is larger than
+ the dask target. The asymmetry the question existed to resolve cannot
+ arise, so there is nothing to raise separately, and `CHUNK_CONTROL` is the
+ answer on either format for a caller who wants the other trade.
+- Q10's row was still standing in §12.3 saying the `NETCDF3_64BIT` defect was
+ preserved rather than fixed, a week after it was closed into this log on
+ 2026-10-16. Removed. A question is closed by moving it, and half a move
+ leaves the table disagreeing with the log.
+- The `CFDataset` member renamed `storage_settings` on 2026-10-20 was renamed
+ in §4.2 only; §5 and §6 still called it `encoding`, which is not a member
+ of anything. Both corrected. §6's read-alignment bullets still described
+ the guard removed the same day, and its skipping rule still described one
+ `importorskip` per test package when a test outside those packages now
+ needs its own.
+
+**2026-10-23 — Actioned the Codex review of #7346**
+
+Five findings in this pull request's own work, all reproduced and fixed, and
+all in the write path's boundaries rather than its middle:
+
+- A save into a subgroup with `consolidate_metadata=False` left the root's
+ consolidated document in place, so a default reader saw no subgroup and the
+ data was in the store and invisible. The user guide already promised the
+ document was removed; nothing removed it. `ZarrDataset` gains
+ `drop_consolidated_metadata`, the other half of `finalise`, and the saver's
+ completion calls one or the other rather than one or nothing. The test that
+ passed throughout was a root-level save, where writing an array happens to
+ rewrite the root metadata without the document — a promise kept by side
+ effect is kept only where the side effect reaches.
+- `ZarrDataset.create_dimension` checked a new declaration against the
+ declarations made in that session and not against the arrays already in the
+ group, so `mode="a"` accepted a size that contradicted an array §4.5 says
+ is never resized. It now checks `dimensions`, which is both.
+- `create_variable` cast the storage fill to the array's dtype and wrote the
+ caller's original value as `_FillValue`, so an `int16` array asked for
+ `fill_value=1.5` stored `1` and declared `1.5`. Iris read it back correctly
+ by narrowing the sentinel, which is what hid it; xarray compares against
+ the attribute and would have masked nothing. A sentinel the dtype cannot
+ hold is now refused. The floating case is deliberately still allowed to
+ lose precision: every `float32` sentinel written as a Python float does.
+- `ZarrDataset.supports_dtype` answered `True` for every dtype, including
+ ones `create_variable` then refused. It was changed to answer from a table
+ of the dtypes every accepted zarr-python version holds, with
+ `create_variable` asking it rather than letting zarr-python raise. **That
+ change was reversed** when the version floor moved; see the end of this
+ log for why, and §4.2 for what the method does now.
+- Byte order was taken out of that question, and stays out. `create_variable`
+ creates in the platform's own byte order whatever order it was asked for:
+ the values are the caller's, the layout is the store's, and the netCDF
+ saver has always answered this data the same way **[verified]**.
+- The write half of the shared `CFDataset` contract had no Zarr subclass,
+ which §6 requires and which the module's own docstring said would arrive
+ with the saver. Added, passing unchanged.
+
+And one finding each side of this pull request:
+
+- The new `cf/saver/test_grid_mapping_writer.py` imported the Zarr saver
+ unguarded, so an Iris without zarr installed gained a test error — the one
+ thing §6 says adding Zarr must not do. The guard is per test rather than
+ per module, because the module's other tests are netCDF's.
+- `zarr.loader._check_has_data_variables` names only the immediate subgroups
+ of an empty root, where §4.6 promises the groups that contain data: for
+ data under `measurements/reflectance/r10m` it suggests `measurements`,
+ which has no data arrays either. Inherited from #7336, not introduced here,
+ and deferred by @trexfeathers to a whole-branch review of `brownfield` at
+ the end of the programme rather than fixed in passing.
+
+**The version floor moves to 3.1.0.** §8 pinned `>=3.0.8` for being above
+zarr-python's known data-loss bug, and nothing ever ran there: the code was
+written and executed against 3.4.1, and the audit above recorded actual
+3.0.8 compatibility as unverified. It was measured afterwards, by running
+the Zarr suite against every zarr-python release from 3.0.8 to 3.4.0.
+
+| zarr-python | Result |
+|---|---|
+| 3.0.8 | 43 failed |
+| 3.0.9, 3.0.10 | 42 failed |
+| 3.1.0 to 3.1.5 | 1 failed |
+| 3.1.6 and above | green, matching 3.4.1 |
+
+One defect accounts for every library failure. `cache_key` serialises
+`Array.metadata.to_dict()`, and on 3.0.x that mapping holds values `json`
+cannot encode: a `DataType` enum on a version 3 array, the codec object on a
+version 2 one, and a `numpy.float32` fill value. 3.1.0 replaced the fixed
+enum of data types with a registry and made `to_dict()` return plain JSON.
+`cache_key` is read on every lazy read and `_LAZYVAR_MIN_BYTES` is 5000, so
+every array worth loading failed on 3.0.x **[verified]**.
+
+The single remaining failure between 3.1.0 and 3.1.5 is a test helper, not
+the library. `CountingStore` subclasses `zarr.storage.WrapperStore`, which
+only grew `_with_store` and `with_read_only` in 3.1.6; the helper now
+supplies its own, so the floor is 3.1.0 rather than 3.1.6. The behaviour
+under test holds on 3.1.0 either way **[verified]**.
+
+**The `supports_dtype` table goes with it.** The table above was written to
+be the intersection across 3.0.8 and 3.4, and @stephenworsley objected in
+review that it reads as a change to backend behaviour nothing exercises.
+That is what it was. `_ensure_valid_dtype` is the only generic caller and it
+asks only about `int64` and the unsigned integers, every one of which Zarr
+holds on every version, so no Zarr save ever consults the rest of the table.
+Only the tests supplied `datetime64`, an object array or a `float128`.
+
+Removing it also removed four wrong answers. The table refused
+`datetime64`, an ordinary structured dtype, a zero-length `U0` and an
+eight-byte void; zarr-python stores all four, and `U0` it stores as a
+variable-width string **[verified]**. What is left is `return True`, with
+zarr-python refusing a dtype it has no data type for inside
+`create_variable` and the group left as it was.
+
+The byte-order coercion stays, for a reason that is not the one it was
+written for. 3.0.8 having no big-endian data type was the original
+justification; the lasting one is that Zarr version 3 has no big-endian data
+type either. Endianness lives in the `bytes` codec, zarr-python leaves it at
+`little`, and asking for `>f4` writes the same store as asking for ``_.
@@ -66,6 +65,7 @@
import numpy as np
from iris import FUTURE
+from iris._deprecation import warn_deprecated
from iris._lazy_data import _co_realise_lazy_arrays, is_lazy_data
from iris.aux_factory import (
AtmosphereSigmaFactory,
@@ -88,15 +88,11 @@
)
import iris.exceptions
from iris.fileformats.cf.dataset import CFDatasetVariable
-
-# TODO: remove when the handlers relocate here, with Zarr saving -
-# https://github.com/SciTools/iris/issues/6980
-# The one place the generic saver still reaches into the netCDF package. The
-# module docstring has the import cycle this closes, and why it is still here.
-from iris.fileformats.netcdf._attribute_handlers import ATTRIBUTE_HANDLERS
import iris.util
import iris.warnings
+from ._attribute_handlers import ATTRIBUTE_HANDLERS
+
if TYPE_CHECKING:
from iris.cube import Cube
from iris.mesh.components import (
@@ -302,6 +298,173 @@ class SaverFillValueWarning(iris.warnings.IrisSaverFillValueWarning):
pass
+def marshal_attributes(cubes, local_keys, format_name):
+ """Decide which cube attributes a store holds globally, and which locally.
+
+ A store holds one value of a global attribute, and a list of cubes may
+ carry several. Settling that is cube logic, not storage logic, so every
+ ``_CFSaver`` subclass's own ``save`` function calls this and nothing
+ does it twice.
+
+ Which half of this does the work depends on
+ :attr:`iris.FUTURE.save_split_attrs`.
+
+ * **Enabled.** ``_CFSaver.write`` writes no global attributes at all,
+ and the caller writes the returned `global_attributes` once, after the
+ last cube. An attribute the cubes disagree about cannot be global, so
+ it is demoted to a local attribute on each cube - on a *copy* of each
+ cube, so that the caller's own cubes are untouched.
+ * **Legacy.** ``_CFSaver.write`` writes each cube's global attributes
+ as it goes, so the returned `global_attributes` is empty and the work
+ is to extend `local_keys` with every attribute the cubes do not all
+ agree on. Without that, one store-wide value would silently become
+ whichever cube was written last.
+
+ Parameters
+ ----------
+ cubes : :class:`iris.cube.CubeList`
+ The cubes about to be saved.
+ local_keys : iterable of str or None
+ Attribute names the caller asked to be local to a data variable.
+ Ignored when split-attribute saving is enabled, which is what makes
+ the keyword deprecated.
+ format_name : str
+ How to name the target format in the legacy mode's deprecation
+ warning, e.g. ``"netcdf"``.
+
+ Returns
+ -------
+ cubes : :class:`iris.cube.CubeList`
+ The cubes to save, which are the ones passed in unless an attribute
+ had to be demoted.
+ local_keys : set of str
+ To pass to each ``_CFSaver.write`` call.
+ global_attributes : dict
+ To write once, after the last cube. Empty in the legacy mode.
+
+ Warns
+ -----
+ IrisSaveWarning
+ Naming the global attributes demoted to local ones, and any that a
+ local attribute of the same name blocked entirely.
+ IrisDeprecation
+ In the legacy mode, which is deprecated since Iris 3.8.
+
+ """
+ # TODO: when iris.FUTURE.save_split_attrs is removed, the whole legacy
+ # half of this function goes with it, along with `local_keys` and
+ # `format_name`.
+ global_attributes: dict = {}
+
+ if FUTURE.save_split_attrs:
+ # 'local_keys' is not used in this case.
+ local_keys = set()
+
+ # Find any collisions in the cube global attributes and "demote" all those
+ # to local attributes (where possible, else warn they are lost).
+ # N.B. "collision" includes when not all cubes *have* that attribute.
+ global_names = set()
+ for cube in cubes:
+ global_names |= set(cube.attributes.globals.keys())
+
+ # Find any global attributes which are not the same on *all* cubes.
+ def attr_values_equal(val1, val2):
+ # An equality test which also works when some values are numpy arrays (!)
+ return iris.util._attribute_equal(val1, val2)
+
+ cube0 = cubes[0]
+ invalid_globals = set(
+ [
+ attrname
+ for attrname in global_names
+ if not all(
+ attr_values_equal(
+ cube.attributes.globals.get(attrname),
+ cube0.attributes.globals.get(attrname),
+ )
+ for cube in cubes[1:]
+ )
+ ]
+ )
+
+ # Establish all the global attributes which we will write to the file (at end).
+ global_attributes = {
+ attr: cube0.attributes.globals.get(attr)
+ for attr in global_names - invalid_globals
+ }
+ if invalid_globals:
+ # Some cubes have different global attributes: modify cubes as required.
+ warnings.warn(
+ f"Saving the cube global attributes {sorted(invalid_globals)} as local "
+ "(i.e. data-variable) attributes, where possible, since they are not "
+ "the same on all input cubes.",
+ category=iris.warnings.IrisSaveWarning,
+ )
+ cubes = cubes.copy() # avoiding modifying the actual input arg.
+ for i_cube in range(len(cubes)):
+ # We iterate over cube *index*, so we can replace the list entries with
+ # with cube *copies* -- just to avoid changing our call args.
+ cube = cubes[i_cube]
+ demote_attrs = set(cube.attributes.globals) & invalid_globals
+ if any(demote_attrs):
+ # Catch any demoted attrs where there is already a local version
+ blocked_attrs = demote_attrs & set(cube.attributes.locals)
+ if blocked_attrs:
+ warnings.warn(
+ f"Global cube attributes {sorted(blocked_attrs)} "
+ f'of cube "{cube.name()}" were not saved, overlaid '
+ "by existing local attributes with the same names.",
+ category=iris.warnings.IrisSaveWarning,
+ )
+ demote_attrs -= blocked_attrs
+ if demote_attrs:
+ # This cube contains some 'demoted' global attributes.
+ # Replace input cube with a copy, so we can modify attributes.
+ cube = cube.copy()
+ cubes[i_cube] = cube
+ for attr in demote_attrs:
+ # move global to local
+ value = cube.attributes.globals.pop(attr)
+ cube.attributes.locals[attr] = value
+
+ else:
+ # Legacy mode: calculate "local_keys" to control which attributes are local
+ # and which global.
+ message = (
+ f"Saving to {format_name} with legacy-style attribute handling for "
+ "backwards compatibility.\n"
+ "This mode is deprecated since Iris 3.8, and will eventually be removed.\n"
+ "Please consider enabling the new split-attributes handling mode, by "
+ "setting 'iris.FUTURE.save_split_attrs = True'."
+ )
+ warn_deprecated(message)
+
+ if local_keys is None:
+ local_keys = set()
+ else:
+ local_keys = set(local_keys)
+
+ # Determine the attribute keys that are common across all cubes and
+ # thereby extend the collection of local_keys for attributes
+ # that should be attributes on data variables.
+ attributes = cubes[0].attributes
+ common_keys = set(attributes)
+ for cube in cubes[1:]:
+ keys = set(cube.attributes)
+ local_keys.update(keys.symmetric_difference(common_keys))
+ common_keys.intersection_update(keys)
+ different_value_keys = []
+ for key in common_keys:
+ if not iris.util._attribute_equal(
+ attributes[key], cube.attributes[key]
+ ):
+ different_value_keys.append(key)
+ common_keys.difference_update(different_value_keys)
+ local_keys.update(different_value_keys)
+
+ return cubes, local_keys, global_attributes
+
+
class _CFSaver:
"""A manager for saving cubes to a CF-conforming store.
@@ -1893,26 +2056,25 @@ def _add_grid_mapping_to_dataset(self, cs, extended_grid_mapping=False):
cf_var_grid = self._dataset.create_variable(cs.grid_mapping_name, np.int32)
cf_var_grid.attributes["grid_mapping_name"] = cs.grid_mapping_name
- # Assignments to properties of ``grid_variable`` bypass the ASCII-to-bytes
- # coercion that .attributes applies, so moving them there would change
- # the file. They need the netCDF4 variable itself. Not
- # deprecated_netcdf_member(), so a store with no netCDF4 object behind it
- # raises a bare AttributeError here.
- grid_variable = cf_var_grid.variable
+ # Where each parameter below lands decides how it is typed in the
+ # store, so the store supplies the function that puts it there. On
+ # netCDF that is the netCDF4 variable itself, exactly as it always
+ # has been; see Q8 in the design document.
+ gm_write_param = self._grid_mapping_writer(cf_var_grid)
def add_ellipsoid(ellipsoid):
- grid_variable.longitude_of_prime_meridian = (
- ellipsoid.longitude_of_prime_meridian
+ gm_write_param(
+ "longitude_of_prime_meridian", ellipsoid.longitude_of_prime_meridian
)
semi_major = ellipsoid.semi_major_axis
semi_minor = ellipsoid.semi_minor_axis
if semi_minor == semi_major:
- grid_variable.earth_radius = semi_major
+ gm_write_param("earth_radius", semi_major)
else:
- grid_variable.semi_major_axis = semi_major
- grid_variable.semi_minor_axis = semi_minor
+ gm_write_param("semi_major_axis", semi_major)
+ gm_write_param("semi_minor_axis", semi_minor)
if ellipsoid.datum is not None:
- grid_variable.horizontal_datum_name = ellipsoid.datum
+ gm_write_param("horizontal_datum_name", ellipsoid.datum)
# latlon
if isinstance(cs, iris.coord_systems.GeogCS):
@@ -1922,79 +2084,81 @@ def add_ellipsoid(ellipsoid):
elif isinstance(cs, iris.coord_systems.RotatedGeogCS):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.grid_north_pole_latitude = cs.grid_north_pole_latitude
- grid_variable.grid_north_pole_longitude = cs.grid_north_pole_longitude
- grid_variable.north_pole_grid_longitude = cs.north_pole_grid_longitude
+ gm_write_param("grid_north_pole_latitude", cs.grid_north_pole_latitude)
+ gm_write_param("grid_north_pole_longitude", cs.grid_north_pole_longitude)
+ gm_write_param("north_pole_grid_longitude", cs.north_pole_grid_longitude)
# tmerc
elif isinstance(cs, iris.coord_systems.TransverseMercator):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.longitude_of_central_meridian = (
- cs.longitude_of_central_meridian
+ gm_write_param(
+ "longitude_of_central_meridian", cs.longitude_of_central_meridian
)
- grid_variable.latitude_of_projection_origin = (
- cs.latitude_of_projection_origin
+ gm_write_param(
+ "latitude_of_projection_origin", cs.latitude_of_projection_origin
)
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
- grid_variable.scale_factor_at_central_meridian = (
- cs.scale_factor_at_central_meridian
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
+ gm_write_param(
+ "scale_factor_at_central_meridian", cs.scale_factor_at_central_meridian
)
# merc
elif isinstance(cs, iris.coord_systems.Mercator):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
+ gm_write_param(
+ "longitude_of_projection_origin", cs.longitude_of_projection_origin
)
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
# Only one of these should be set
if cs.standard_parallel is not None:
- grid_variable.standard_parallel = cs.standard_parallel
+ gm_write_param("standard_parallel", cs.standard_parallel)
elif cs.scale_factor_at_projection_origin is not None:
- grid_variable.scale_factor_at_projection_origin = (
- cs.scale_factor_at_projection_origin
+ gm_write_param(
+ "scale_factor_at_projection_origin",
+ cs.scale_factor_at_projection_origin,
)
# lcc
elif isinstance(cs, iris.coord_systems.LambertConformal):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.standard_parallel = cs.secant_latitudes
- grid_variable.latitude_of_projection_origin = cs.central_lat
- grid_variable.longitude_of_central_meridian = cs.central_lon
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
+ gm_write_param("standard_parallel", cs.secant_latitudes)
+ gm_write_param("latitude_of_projection_origin", cs.central_lat)
+ gm_write_param("longitude_of_central_meridian", cs.central_lon)
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
# polar stereo (have to do this before Stereographic because it subclasses it)
elif isinstance(cs, iris.coord_systems.PolarStereographic):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.latitude_of_projection_origin = cs.central_lat
- grid_variable.straight_vertical_longitude_from_pole = cs.central_lon
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
+ gm_write_param("latitude_of_projection_origin", cs.central_lat)
+ gm_write_param("straight_vertical_longitude_from_pole", cs.central_lon)
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
# Only one of these should be set
if cs.true_scale_lat is not None:
- grid_variable.true_scale_lat = cs.true_scale_lat
+ gm_write_param("true_scale_lat", cs.true_scale_lat)
elif cs.scale_factor_at_projection_origin is not None:
- grid_variable.scale_factor_at_projection_origin = (
- cs.scale_factor_at_projection_origin
+ gm_write_param(
+ "scale_factor_at_projection_origin",
+ cs.scale_factor_at_projection_origin,
)
else:
- grid_variable.scale_factor_at_projection_origin = 1.0
+ gm_write_param("scale_factor_at_projection_origin", 1.0)
# stereo
elif isinstance(cs, iris.coord_systems.Stereographic):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.longitude_of_projection_origin = cs.central_lon
- grid_variable.latitude_of_projection_origin = cs.central_lat
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
+ gm_write_param("longitude_of_projection_origin", cs.central_lon)
+ gm_write_param("latitude_of_projection_origin", cs.central_lat)
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
# Only one of these should be set
if cs.true_scale_lat is not None:
msg = (
@@ -2003,11 +2167,12 @@ def add_ellipsoid(ellipsoid):
)
raise ValueError(msg)
elif cs.scale_factor_at_projection_origin is not None:
- grid_variable.scale_factor_at_projection_origin = (
- cs.scale_factor_at_projection_origin
+ gm_write_param(
+ "scale_factor_at_projection_origin",
+ cs.scale_factor_at_projection_origin,
)
else:
- grid_variable.scale_factor_at_projection_origin = 1.0
+ gm_write_param("scale_factor_at_projection_origin", 1.0)
# osgb (a specific tmerc)
elif isinstance(cs, iris.coord_systems.OSGB):
@@ -2020,57 +2185,57 @@ def add_ellipsoid(ellipsoid):
elif isinstance(cs, iris.coord_systems.LambertAzimuthalEqualArea):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
+ gm_write_param(
+ "longitude_of_projection_origin", cs.longitude_of_projection_origin
)
- grid_variable.latitude_of_projection_origin = (
- cs.latitude_of_projection_origin
+ gm_write_param(
+ "latitude_of_projection_origin", cs.latitude_of_projection_origin
)
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
# albers conical equal area
elif isinstance(cs, iris.coord_systems.AlbersEqualArea):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.longitude_of_central_meridian = (
- cs.longitude_of_central_meridian
+ gm_write_param(
+ "longitude_of_central_meridian", cs.longitude_of_central_meridian
)
- grid_variable.latitude_of_projection_origin = (
- cs.latitude_of_projection_origin
+ gm_write_param(
+ "latitude_of_projection_origin", cs.latitude_of_projection_origin
)
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
- grid_variable.standard_parallel = cs.standard_parallels
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
+ gm_write_param("standard_parallel", cs.standard_parallels)
# vertical perspective
elif isinstance(cs, iris.coord_systems.VerticalPerspective):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
+ gm_write_param(
+ "longitude_of_projection_origin", cs.longitude_of_projection_origin
)
- grid_variable.latitude_of_projection_origin = (
- cs.latitude_of_projection_origin
+ gm_write_param(
+ "latitude_of_projection_origin", cs.latitude_of_projection_origin
)
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
- grid_variable.perspective_point_height = cs.perspective_point_height
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
+ gm_write_param("perspective_point_height", cs.perspective_point_height)
# geostationary
elif isinstance(cs, iris.coord_systems.Geostationary):
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
+ gm_write_param(
+ "longitude_of_projection_origin", cs.longitude_of_projection_origin
)
- grid_variable.latitude_of_projection_origin = (
- cs.latitude_of_projection_origin
+ gm_write_param(
+ "latitude_of_projection_origin", cs.latitude_of_projection_origin
)
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
- grid_variable.perspective_point_height = cs.perspective_point_height
- grid_variable.sweep_angle_axis = cs.sweep_angle_axis
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
+ gm_write_param("perspective_point_height", cs.perspective_point_height)
+ gm_write_param("sweep_angle_axis", cs.sweep_angle_axis)
# oblique mercator (and rotated variant)
# Use duck-typing over isinstance() - subclasses (i.e.
@@ -2082,17 +2247,18 @@ def add_ellipsoid(ellipsoid):
# all mention of RM.
if cs.ellipsoid:
add_ellipsoid(cs.ellipsoid)
- grid_variable.azimuth_of_central_line = cs.azimuth_of_central_line
- grid_variable.latitude_of_projection_origin = (
- cs.latitude_of_projection_origin
+ gm_write_param("azimuth_of_central_line", cs.azimuth_of_central_line)
+ gm_write_param(
+ "latitude_of_projection_origin", cs.latitude_of_projection_origin
)
- grid_variable.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
+ gm_write_param(
+ "longitude_of_projection_origin", cs.longitude_of_projection_origin
)
- grid_variable.false_easting = cs.false_easting
- grid_variable.false_northing = cs.false_northing
- grid_variable.scale_factor_at_projection_origin = (
- cs.scale_factor_at_projection_origin
+ gm_write_param("false_easting", cs.false_easting)
+ gm_write_param("false_northing", cs.false_northing)
+ gm_write_param(
+ "scale_factor_at_projection_origin",
+ cs.scale_factor_at_projection_origin,
)
# other
@@ -2106,7 +2272,7 @@ def add_ellipsoid(ellipsoid):
# add WKT string
if extended_grid_mapping:
- grid_variable.crs_wkt = cs.as_cartopy_crs().to_wkt()
+ gm_write_param("crs_wkt", cs.as_cartopy_crs().to_wkt())
def _create_cf_grid_mapping(self, cube, cf_var_cube):
"""Create CF-netCDF grid mapping and associated CF-netCDF variable.
@@ -2451,6 +2617,28 @@ def _backend_write(self, data, cf_var) -> bool:
"""
return False
+ def _grid_mapping_writer(self, cf_var):
+ """Return the function that writes one CF grid-mapping parameter.
+
+ Parameters
+ ----------
+ cf_var : :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+ The grid-mapping variable, newly created and otherwise empty.
+
+ Returns
+ -------
+ callable
+ Called with a parameter name and a value. Here it puts them in
+ ``cf_var.attributes``, which is where a CF store keeps them. A
+ store that types those attributes some other way overrides this.
+
+ """
+
+ def write_parameter(name, value):
+ cf_var.attributes[name] = value
+
+ return write_parameter
+
def _lazy_stream_data(
self,
data: np.typing.ArrayLike,
diff --git a/lib/iris/fileformats/netcdf/_dataset.py b/lib/iris/fileformats/netcdf/_dataset.py
index 925223aad7..cf0cddb255 100644
--- a/lib/iris/fileformats/netcdf/_dataset.py
+++ b/lib/iris/fileformats/netcdf/_dataset.py
@@ -466,9 +466,13 @@ def __init__(
location : str or :class:`pathlib.Path`
The file's path or URL.
mode : str, default="r"
- The netCDF4 open mode.
+ One of ``"r"``, ``"w"``, ``"x"``, ``"a"`` or ``"r+"``, with
+ netCDF4's meanings. ``"x"`` creates and refuses an existing file;
+ ``"w"`` replaces one; ``"a"`` and ``"r+"`` both open one for
+ update, and both create the file if it is not there.
netcdf_format : str, optional
- The netCDF format to create, when writing.
+ The netCDF format to create, when writing. netCDF4's own default
+ when not given.
warn_legacy_format : bool, default=False
Whether to warn that a netCDF3 file would load faster if
converted. Only loading asks for this; the loader is where the
diff --git a/lib/iris/fileformats/netcdf/saver.py b/lib/iris/fileformats/netcdf/saver.py
index ccb5f5f08e..e0d27aaa41 100644
--- a/lib/iris/fileformats/netcdf/saver.py
+++ b/lib/iris/fileformats/netcdf/saver.py
@@ -45,7 +45,6 @@
from iris.fileformats.netcdf import _bytecoding_datasets as bytecoding_datasets
from iris.fileformats.netcdf._bytecoding_datasets import VariableEncoder
from iris.fileformats.netcdf._dataset import NetCDFDataset, NetCDFDatasetVariable
-import iris.util
import iris.warnings
# Get the logger : shared logger for all in 'iris.fileformats.netcdf'.
@@ -414,6 +413,34 @@ def _patch_dataset(self, cf_patch, profile, cf_var_cube) -> None:
# 'variable.name = value': it defines no __setattr__.
cf_patch(profile, self._dataset.dataset, cf_var_cube.variable)
+ def _grid_mapping_writer(self, cf_var):
+ """Return a function writing onto the netCDF4 variable, as has always happened.
+
+ Setting an attribute on a netCDF4 variable bypasses the ASCII-to-bytes
+ coercion that ``.attributes`` applies, so ``crs_wkt`` is written as
+ ``NC_STRING`` while ``grid_mapping_name`` is ``NC_CHAR``. Regularising
+ that is Q8 in §12.3 of the native Zarr I/O design,
+ ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``; until then,
+ this writes each parameter exactly where netCDF has always written it,
+ so the file is unchanged.
+
+ Parameters
+ ----------
+ cf_var : :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`
+ The grid-mapping variable.
+
+ Returns
+ -------
+ callable
+ Called with a parameter name and a value.
+
+ """
+
+ def write_parameter(name, value):
+ setattr(cf_var.variable, name, value)
+
+ return write_parameter
+
def _create_string_variable(
self, cube_or_mesh, cf_name, element, element_dims, data
):
@@ -722,123 +749,12 @@ def save(
else:
cubes = cube
- # Decide which cube attributes will be saved as "global" attributes
- # NOTE: in 'legacy' mode, when iris.FUTURE.save_split_attrs == False, this code
- # section derives a common value for 'local_keys', which is passed to 'Saver.write'
- # when saving each input cube. The global attributes are then created by a call
- # to "Saver.update_global_attributes" within each 'Saver.write' call (which is
- # obviously a bit redundant!), plus an extra one to add 'Conventions'.
- # HOWEVER, in `split_attrs` mode (iris.FUTURE.save_split_attrs == False), this code
- # instead constructs a 'global_attributes' dictionary, and outputs that just once,
- # after writing all the input cubes.
- if iris.FUTURE.save_split_attrs:
- # We don't actually use 'local_keys' in this case.
- # TODO: can remove this when the iris.FUTURE.save_split_attrs is removed.
- local_keys = set()
-
- # Find any collisions in the cube global attributes and "demote" all those to
- # local attributes (where possible, else warn they are lost).
- # N.B. "collision" includes when not all cubes *have* that attribute.
- global_names = set()
- for cube in cubes:
- global_names |= set(cube.attributes.globals.keys())
-
- # Fnd any global attributes which are not the same on *all* cubes.
- def attr_values_equal(val1, val2):
- # An equality test which also works when some values are numpy arrays (!)
- return iris.util._attribute_equal(val1, val2)
-
- cube0 = cubes[0]
- invalid_globals = set(
- [
- attrname
- for attrname in global_names
- if not all(
- attr_values_equal(
- cube.attributes.globals.get(attrname),
- cube0.attributes.globals.get(attrname),
- )
- for cube in cubes[1:]
- )
- ]
- )
-
- # Establish all the global attributes which we will write to the file (at end).
- global_attributes = {
- attr: cube0.attributes.globals.get(attr)
- for attr in global_names - invalid_globals
- }
- if invalid_globals:
- # Some cubes have different global attributes: modify cubes as required.
- warnings.warn(
- f"Saving the cube global attributes {sorted(invalid_globals)} as local "
- "(i.e. data-variable) attributes, where possible, since they are not "
- "the same on all input cubes.",
- category=iris.warnings.IrisSaveWarning,
- )
- cubes = cubes.copy() # avoiding modifying the actual input arg.
- for i_cube in range(len(cubes)):
- # We iterate over cube *index*, so we can replace the list entries with
- # with cube *copies* -- just to avoid changing our call args.
- cube = cubes[i_cube]
- demote_attrs = set(cube.attributes.globals) & invalid_globals
- if any(demote_attrs):
- # Catch any demoted attrs where there is already a local version
- blocked_attrs = demote_attrs & set(cube.attributes.locals)
- if blocked_attrs:
- warnings.warn(
- f"Global cube attributes {sorted(blocked_attrs)} "
- f'of cube "{cube.name()}" were not saved, overlaid '
- "by existing local attributes with the same names.",
- category=iris.warnings.IrisSaveWarning,
- )
- demote_attrs -= blocked_attrs
- if demote_attrs:
- # This cube contains some 'demoted' global attributes.
- # Replace input cube with a copy, so we can modify attributes.
- cube = cube.copy()
- cubes[i_cube] = cube
- for attr in demote_attrs:
- # move global to local
- value = cube.attributes.globals.pop(attr)
- cube.attributes.locals[attr] = value
-
- else:
- # Legacy mode: calculate "local_keys" to control which attributes are local
- # and which global.
- # TODO: when iris.FUTURE.save_split_attrs is removed, this section can also be
- # removed
- message = (
- "Saving to netcdf with legacy-style attribute handling for backwards "
- "compatibility.\n"
- "This mode is deprecated since Iris 3.8, and will eventually be removed.\n"
- "Please consider enabling the new split-attributes handling mode, by "
- "setting 'iris.FUTURE.save_split_attrs = True'."
- )
- warn_deprecated(message)
-
- if local_keys is None:
- local_keys = set()
- else:
- local_keys = set(local_keys)
-
- # Determine the attribute keys that are common across all cubes and
- # thereby extend the collection of local_keys for attributes
- # that should be attributes on data variables.
- attributes = cubes[0].attributes
- common_keys = set(attributes)
- for cube in cubes[1:]:
- keys = set(cube.attributes)
- local_keys.update(keys.symmetric_difference(common_keys))
- common_keys.intersection_update(keys)
- different_value_keys = []
- for key in common_keys:
- if not iris.util._attribute_equal(
- attributes[key], cube.attributes[key]
- ):
- different_value_keys.append(key)
- common_keys.difference_update(different_value_keys)
- local_keys.update(different_value_keys)
+ # Decide which cube attributes will be saved as "global" attributes. This is
+ # cube logic rather than netCDF logic, so it lives in 'cf.saver' and every
+ # format's 'save' calls it; see its docstring for what each mode does.
+ cubes, local_keys, global_attributes = cf_saver.marshal_attributes(
+ cubes, local_keys, "netcdf"
+ )
def is_valid_packspec(p):
"""Only checks that the datatype is valid."""
diff --git a/lib/iris/fileformats/zarr/__init__.py b/lib/iris/fileformats/zarr/__init__.py
index 3b6306653e..473134fb9d 100644
--- a/lib/iris/fileformats/zarr/__init__.py
+++ b/lib/iris/fileformats/zarr/__init__.py
@@ -9,10 +9,10 @@
API reference
-Reads Zarr version 2 and version 3 stores, local or remote. Not NCZarr: that
-is netCDF-c's own Zarr driver, reached through a URL fragment such as
-``file:///path/store.zarr#mode=nczarr,file``, and it stays where it is in
-:mod:`iris.fileformats.netcdf`.
+Reads Zarr version 2 and version 3 stores, local or remote, and writes version
+3 ones. Not NCZarr: that is netCDF-c's own Zarr driver, reached through a URL
+fragment such as ``file:///path/store.zarr#mode=nczarr,file``, and it stays
+where it is in :mod:`iris.fileformats.netcdf`.
``zarr`` is an optional dependency. Nothing in :mod:`iris.fileformats` imports
this package until a Zarr target is actually loaded, so Iris installs and
@@ -28,25 +28,43 @@
"""
+import importlib
import importlib.metadata
from packaging.version import Version
-__all__ = ["load_cubes"]
+__all__ = ["Saver", "load_cubes", "save"]
+
+#: What `__getattr__` forwards, and the module it lives in. The import is
+#: deferred because `._dataset` imports `zarr` at module level, and this
+#: package promises that nothing does so until a Zarr target is used.
+_FORWARDED = {
+ "load_cubes": ".loader",
+ "save": ".saver",
+ "Saver": ".saver",
+}
#: The version floor ``docs/src/user_manual/explanation/zarr_io.rst`` and
#: ``requirements/py3*.yml`` both promise.
-_MINIMUM_VERSION = "3.0.8"
+#:
+#: 3.1.0 rather than the 3.0.8 that is merely above zarr-python's known
+#: data-loss bug: 3.1.0 replaced the fixed enum of data types with a
+#: registry, and with it made ``Array.metadata.to_dict()`` return plain JSON
+#: values. :attr:`~iris.fileformats.zarr._dataset.ZarrDatasetVariable.cache_key`
+#: serialises that mapping on every lazy read, so on 3.0.x every array large
+#: enough to read lazily fails with ``TypeError: Object of type DataType is
+#: not JSON serializable``.
+_MINIMUM_VERSION = "3.1.0"
#: Raised whether ``zarr`` is missing entirely or present below
#: :data:`_MINIMUM_VERSION`. Defined once here, not in
-#: :func:`iris.fileformats._load_zarr`, so that the two routes a caller can
-#: reach a Zarr load through - ``iris.load`` and calling
-#: :func:`load_cubes` directly - produce the same text rather than each
-#: carrying its own copy.
+#: :func:`iris.fileformats._load_zarr`, so that every route a caller can
+#: reach a Zarr store through - ``iris.load``, ``iris.save``, and calling
+#: :func:`load_cubes` directly - produces the same text rather than each
+#: carrying its own copy. "Use", not "load": ``iris.save`` raises this too.
_MISSING_MESSAGE = (
- "Unable to load a Zarr store - the 'zarr' package is not "
- "installed. Install zarr >=3.0.8, for example with "
+ "Unable to use a Zarr store - the 'zarr' package is not "
+ "installed. Install zarr >=3.1.0, for example with "
"'conda install -c conda-forge zarr'."
)
@@ -79,14 +97,15 @@ def _check_zarr_version() -> None:
def __getattr__(name):
- """Forward ``load_cubes`` to :mod:`iris.fileformats.zarr.loader`.
+ """Forward the public names to the modules that define them.
- PEP 562, not a plain ``from .loader import load_cubes``: ``loader``
- imports ``._dataset``, which imports ``zarr`` at module level, so an
- eager import here would make the dependency load with the package rather
- than with a Zarr load - breaking the promise this module's own docstring
- makes, that nothing imports ``zarr`` until a Zarr target is actually
- loaded. Resist "simplifying" this back to a plain import.
+ PEP 562, not a plain ``from .loader import load_cubes``: ``loader`` and
+ ``saver`` both import ``._dataset``, which imports ``zarr`` at module
+ level, so an eager import here would make the dependency load with the
+ package rather than with a Zarr load or save - breaking the promise this
+ module's own docstring makes, that nothing imports ``zarr`` until a Zarr
+ target is actually used. Resist "simplifying" this back to a plain
+ import.
Parameters
----------
@@ -97,7 +116,7 @@ def __getattr__(name):
Returns
-------
Any
- ``load_cubes``, the only name this forwards.
+ The object of that name, from the module :data:`_FORWARDED` names.
Raises
------
@@ -108,14 +127,15 @@ def __getattr__(name):
:data:`_MINIMUM_VERSION`.
"""
- if name != "load_cubes":
+ module_name = _FORWARDED.get(name)
+ if module_name is None:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
_check_zarr_version()
try:
- from .loader import load_cubes
+ module = importlib.import_module(module_name, __name__)
except ImportError as exception:
raise ImportError(_MISSING_MESSAGE) from exception
- return load_cubes
+ return getattr(module, name)
diff --git a/lib/iris/fileformats/zarr/_dataset.py b/lib/iris/fileformats/zarr/_dataset.py
index 8c9d5a5683..6397c0212f 100644
--- a/lib/iris/fileformats/zarr/_dataset.py
+++ b/lib/iris/fileformats/zarr/_dataset.py
@@ -30,6 +30,17 @@
:attr:`ZarrDatasetVariable.chunking` therefore returns ``Array.chunks``, never
``Array.shards``.
+**An array is created in the platform's own byte order**, whatever order it
+was asked for, and NumPy converts the values on assignment. Zarr version 3
+has no big-endian data type: endianness lives in the ``bytes`` codec, which
+zarr-python leaves at ``little``, so asking for ``>f4`` writes the same store
+as asking for ``f4``
+cube as ``float32``, and a PP or GRIB load hands the saver big-endian arrays
+routinely.
+
Chunking is sized by :func:`iris._lazy_data._optimum_chunksize`, the same way
the netCDF path sizes it, and is not then adjusted to the store's own
chunking. Iris warns when a stored chunk is larger than Dask's target, from
@@ -62,7 +73,7 @@
from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
import iris.warnings
-from . import _decode
+from . import _decode, _encode
#: What :attr:`CFDataset.format` answers, by the store's Zarr version.
_FORMAT_BY_ZARR_VERSION = {2: "ZARR_V2", 3: "ZARR_V3"}
@@ -143,6 +154,44 @@ def _dimension_names(array) -> tuple:
return tuple(names)
+def _selection_shape(shape: tuple, keys) -> tuple | None:
+ """Return the shape `keys` selects out of an array of `shape`.
+
+ Measured with numpy's own indexing rules, which are the rules
+ zarr-python implements for a basic selection, against a zero-strided
+ dummy so that measuring allocates nothing.
+
+ Parameters
+ ----------
+ shape : tuple of int
+ The array's full shape.
+ keys : Any
+ The selection, as it reached ``__setitem__``.
+
+ Returns
+ -------
+ tuple of int or None
+ ``None`` for anything but a basic selection - a boolean mask or an
+ index array, which zarr accepts and Iris never writes, and which
+ would make a copy to measure. A caller treats that as "nothing to
+ check".
+
+ """
+ parts = keys if isinstance(keys, tuple) else (keys,)
+ if not all(
+ isinstance(part, (slice, int, np.integer)) or part is Ellipsis or part is None
+ for part in parts
+ ):
+ return None
+ dummy = np.broadcast_to(np.empty((), dtype=np.int8), shape)
+ try:
+ return dummy[keys].shape
+ except IndexError:
+ # Too many indices for the rank, say. zarr-python raises for these
+ # too, with its own message, so there is nothing to add here.
+ return None
+
+
def _assemble_dimensions(variables) -> dict:
"""Return the dimension sizes the store's arrays agree on.
@@ -202,13 +251,28 @@ def __init__(
node: "zarr.Group | zarr.Array",
*,
withhold: frozenset = frozenset(),
+ dtype=None,
):
"""Materialise the attributes of `node`, a zarr array or group.
Writing through this mapping writes to `node`, which is the store.
+
+ Parameters
+ ----------
+ node : zarr.Array or zarr.Group
+ Whose attributes these are.
+ withhold : frozenset, optional
+ Names that are storage rather than CF, hidden here and refused
+ on assignment.
+ dtype : numpy.dtype, optional
+ The array's dtype, when `node` is an array. What decides the
+ encoding of ``_FillValue``; a group has neither.
+
+
"""
self._node = node
self._withhold = frozenset(withhold)
+ self._dtype = dtype
self._values = {
key: value for key, value in node.attrs.items() if key not in self._withhold
}
@@ -218,12 +282,29 @@ def __getitem__(self, key: str) -> Any:
return self._values[key]
def __setitem__(self, key: str, value: Any) -> None:
- """Set `key`'s value, here and in the store."""
+ """Set `key`'s value, here and in the store.
+
+ The value is converted to something JSON can hold *before* zarr sees
+ it. A value zarr's encoder rejects raises **and stays in the
+ mapping**, so converting on failure would mean converting a store
+ that is already wrong.
+
+ """
if key in self._withhold:
msg = f"Cannot set a storage attribute through `attributes`: {key}"
raise KeyError(msg)
- self._node.attrs[key] = value
- self._values[key] = value
+ encoded = self._encode(key, value)
+ self._node.attrs[key] = encoded
+ self._values[key] = encoded
+
+ def _encode(self, key: str, value: Any) -> Any:
+ """Return `value` in the form this attribute is stored in."""
+ if self._dtype is not None:
+ if key == "_FillValue":
+ return _encode.encode_fill_value(value, self._dtype)
+ if key == "missing_value":
+ return _encode.encode_missing_values(value, self._dtype)
+ return _encode.json_native(value, name=key)
def __delitem__(self, key: str) -> None:
"""Remove `key`, here and from the store."""
@@ -264,7 +345,7 @@ def __init__(self, array, location: str, format: str):
self._format = format
self._dimensions = _dimension_names(array)
self._attributes = _ZarrAttributes(
- array, withhold=frozenset({_ARRAY_DIMENSIONS})
+ array, withhold=frozenset({_ARRAY_DIMENSIONS}), dtype=array.dtype
)
@property
@@ -342,8 +423,60 @@ def __getitem__(self, keys) -> np.ndarray:
return self._array[keys]
def __setitem__(self, keys, values) -> None:
- """Write `values` into the indexed portion of the array."""
- self._array[keys] = values
+ """Write `values` into the indexed portion of the array.
+
+ Packed first, filled second. `values` arrives in physical units;
+ :func:`~iris.fileformats.zarr._encode.pack` applies this array's own
+ ``scale_factor``/``add_offset`` attributes, the transform
+ netCDF4-python would otherwise have applied automatically on
+ assignment, which a Zarr array has no equivalent of. Masked values are
+ filled with the array's own fill value afterwards, not before: the
+ fill lives in *stored* (packed) space, so filling first would send an
+ already-filled sentinel through the same transform as real data and
+ lose it. A Zarr array has no mask of its own either way - spec §4.5
+ settles that the fill goes in the data - and handing numpy.ma
+ straight to zarr raises inside the bytes codec, naming neither the
+ variable nor the mask.
+
+ A value whose shape does not match the selection is refused first.
+ zarr-python truncates an oversized assignment without a word, so a
+ caller that sized the array wrongly loses data in silence; too few
+ values raise, but from inside numpy, naming neither this variable
+ nor the array it belongs to.
+
+ Raises
+ ------
+ ValueError
+ If `values` neither matches the selection's shape nor is a
+ scalar to broadcast over it.
+
+ """
+ if self._array.ndim == 0 and keys == slice(None):
+ # `_CFSaver._lazy_stream_data` writes a whole variable as
+ # `cf_var[:] = data`, which is `slice(None)` - one axis too many
+ # for a 0-d array, which zarr indexes with `...` or `()` instead.
+ # A scalar cube is ordinary, and this is otherwise the only
+ # store that cannot hold one.
+ keys = Ellipsis
+
+ selected = _selection_shape(self.shape, keys)
+ given = np.shape(values)
+ if selected is not None and given not in ((), selected):
+ message = (
+ f"Cannot write values of shape {given} into {self.name!r}, "
+ f"whose selected region has shape {selected}. Zarr neither "
+ "resizes an array nor reports an oversized write, so this "
+ "would otherwise lose values in silence."
+ )
+ raise ValueError(message)
+
+ packed = _encode.pack(
+ values,
+ self._attributes.get("scale_factor"),
+ self._attributes.get("add_offset"),
+ self._array.dtype,
+ )
+ self._array[keys] = _encode.filled(packed, self._array.fill_value)
def read_data(self, chunking_policy: Callable[[], tuple]) -> Any:
"""Return this array's data, lazily where a lazy array is worth it.
@@ -476,6 +609,20 @@ def zarr_format(self) -> int:
"""2 or 3. The one Zarr fact that changes how a value is read."""
return self._array.metadata.zarr_format
+ @property
+ def write_grid(self) -> tuple:
+ """The unit a concurrent writer must own whole cells of.
+
+ The shards when the array is sharded, the chunks when it is not. A
+ shard is one stored object holding many inner chunks, so two tasks
+ writing different chunks of one shard read-modify-write the same
+ object and one update is lost. Stating the invariant over the chunk
+ grid alone is what xarray's ``safe_chunks`` had to be corrected to;
+ see pydata/xarray#10831.
+
+ """
+ return tuple(self._array.shards or self._array.chunks)
+
@property
def cache_key(self) -> str:
"""An identity for this array that goes stale when the array does.
@@ -494,17 +641,188 @@ def cache_key(self) -> str:
)
-#: What `mode` a dataset may be opened in. Writing is not implemented.
+#: The modes that only read.
_READ_MODES = ("r",)
-#: The one message every unimplemented write member raises.
-_NO_SAVING = "Saving to Zarr is not supported."
+#: The modes that may write. ``"r+"`` opens an existing store for update and
+#: creates nothing, which is what the region write of a future multi-process
+#: saver needs. See §4.5 of docs/superpowers/specs/2026-09-21-zarr-io-design.md.
+_WRITE_MODES = ("w-", "w", "a", "r+")
+
+#: Every mode, in the order the error message lists them.
+_MODES = _READ_MODES + _WRITE_MODES
+
+#: The Zarr versions Iris can write. This constrains writing only; reading
+#: handles either version. Adding version 2 here also means implementing the
+#: three version-specific points of §4.5 of
+#: docs/superpowers/specs/2026-09-21-zarr-io-design.md.
+_WRITABLE_ZARR_FORMATS = (3,)
+
+#: The version a write creates when the caller names none. Separate from
+#: `_WRITABLE_ZARR_FORMATS` because the order of that tuple says nothing
+#: about which member is the default.
+_DEFAULT_ZARR_FORMAT = 3
+
+#: The dtype kinds whose fill value must convert exactly, because they have
+#: no rounding to spend: `numpy.int16(1.5)` is `1`, and the `_FillValue`
+#: attribute would still say `1.5`.
+_EXACT_FILL_KINDS = frozenset(("b", "i", "u"))
+
+#: zarr-python's own storage keywords. `create_variable` passes them through
+#: directly. Anything else raises, with advice from `_KEYWORD_ADVICE` below
+#: where there is any to give. See §4.5 of
+#: docs/superpowers/specs/2026-09-21-zarr-io-design.md.
+#:
+#: `order` is **not** here, although zarr-python's signature takes it: it has
+#: no effect on a version 3 array, which is all Iris writes, and zarr-python
+#: warns and ignores it.
+_ZARR_ENCODING_KEYWORDS = frozenset(
+ {"chunks", "shards", "compressors", "filters", "serializer"}
+)
+
+#: Extra advice for the error message on certain refused keywords. Most are
+#: netCDF's, where the advice names a Zarr equivalent if one exists. The rest
+#: read as though they ought to work here, and are worth more than silence.
+_KEYWORD_ADVICE = {
+ "zlib": "use compressors, e.g. compressors=[zarr.codecs.GzipCodec()]",
+ "complevel": "use compressors, which carry their own level",
+ "shuffle": "use compressors, e.g. the shuffle option of BloscCodec",
+ "fletcher32": "no equivalent; Zarr checksums with a Crc32c codec",
+ "contiguous": "no equivalent; a Zarr array is always chunked",
+ "chunksizes": "use chunks",
+ "endian": "no equivalent; a Zarr codec decides byte order",
+ "least_significant_digit": "no equivalent",
+ "unlimited_dimensions": "no equivalent; a Zarr array has a fixed shape",
+ "order": (
+ "zarr-python takes order for a Zarr version 2 array, and Iris writes "
+ "version 3, where it has no effect"
+ ),
+ "append": (
+ "there is no append; use mode='a' to add new arrays to a store that "
+ "already exists"
+ ),
+}
+
+
+def check_encoding_keywords(encoding) -> None:
+ """Raise for any storage keyword a Zarr store does not have.
+
+ Parameters
+ ----------
+ encoding : dict
+ The storage keywords a caller passed.
+
+ Raises
+ ------
+ TypeError
+ Naming each unusable keyword, and what to do instead where there is
+ something to say.
+
+ """
+ unknown = set(encoding) - _ZARR_ENCODING_KEYWORDS
+ if not unknown:
+ return
+ lines = []
+ for keyword in sorted(unknown):
+ advice = _KEYWORD_ADVICE.get(keyword)
+ if advice is None:
+ lines.append(f" {keyword}: not a Zarr storage keyword")
+ else:
+ lines.append(f" {keyword}: {advice}")
+ message = (
+ "Unusable Zarr storage keyword(s). A Zarr store takes zarr-python's "
+ "own keywords, and Iris does not translate netCDF's onto them:\n"
+ + "\n".join(lines)
+ )
+ raise TypeError(message)
+
+
+def existing_name_message(name: str, location: str) -> str:
+ """Return the message for a name a Zarr group already holds.
+
+ One sentence, shared by :meth:`ZarrDataset.create_variable` and
+ :class:`iris.fileformats.zarr.saver.Saver`'s own earlier check of the
+ same fact, so the two call sites cannot drift into saying it two ways.
+
+ Parameters
+ ----------
+ name : str
+ The array name that already exists.
+ location : str
+ The group's path or URL, as :attr:`ZarrDataset.location` reports it.
+
+ Returns
+ -------
+ str
+
+ """
+ return (
+ f"Cannot create {name!r} in the Zarr group {location!r}: it "
+ "already holds something of that name, and Iris neither "
+ "overwrites nor merges into one."
+ )
+
+
+def _storage_fill_value(fill_value, dtype, name: str) -> Any:
+ """Return `fill_value` as a `dtype` scalar, or raise if it cannot be one.
+
+ An array's own fill value and its ``_FillValue`` attribute must be the
+ same number: the attribute declares a value missing, the fill says what
+ unwritten storage reads back as, and a reader that meets two numbers
+ masks the wrong points or none.
+
+ Parameters
+ ----------
+ fill_value :
+ The sentinel the caller gave.
+ dtype : numpy.dtype or type
+ The array's dtype. ``str`` itself is a variable-length string array,
+ whose sentinel is text and has nothing to round.
+ name : str
+ The array's name, for the message.
+
+ Returns
+ -------
+ Any
+ The sentinel as a scalar of `dtype`.
+
+ Raises
+ ------
+ ValueError
+ If `dtype` cannot hold `fill_value`.
+
+ """
+ if dtype is str:
+ return np.dtype(str).type(fill_value)
+
+ dtype = np.dtype(dtype)
+ refusal = (
+ f"Cannot use fill_value={fill_value!r} for the Zarr array {name!r}: "
+ f"{dtype} cannot hold it, so the array's own fill value and its "
+ "'_FillValue' attribute would be two different numbers. Pass a "
+ "sentinel of the array's own type."
+ )
+ try:
+ with np.errstate(over="raise"):
+ stored = dtype.type(fill_value)
+ except (OverflowError, ValueError, TypeError, FloatingPointError) as exception:
+ raise ValueError(refusal) from exception
+ if dtype.kind in _EXACT_FILL_KINDS and stored != fill_value:
+ raise ValueError(refusal)
+ return stored
class ZarrDataset(CFDataset):
"""A Zarr store, or one group of one, presented through the CF interface."""
- def __init__(self, location, mode: str = "r", *, group: str | None = None):
+ def __init__(
+ self,
+ location,
+ mode: str = "r",
+ *,
+ group: str | None = None,
+ zarr_format: int | None = None,
+ ):
"""Open a Zarr store.
Parameters
@@ -518,18 +836,31 @@ def __init__(self, location, mode: str = "r", *, group: str | None = None):
`location` then becomes zarr-python's own rendering of it, the
best name available for a store with no address.
mode : str, default="r"
- Only ``"r"`` is supported.
+ One of ``"r"``, ``"r+"``, ``"a"``, ``"w"`` or ``"w-"``, with
+ zarr-python's meanings. ``"w-"`` creates and refuses an existing
+ target; ``"w"`` replaces one; ``"a"`` adds to one; ``"r+"`` adds
+ to one and refuses to create it.
group : str, optional
A ``/``-separated path to a group within the store. The root group
by default. Iris reads one group per call and does not walk
nested groups.
+ zarr_format : int, optional
+ The version to create, when writing. Version 3 when not given,
+ and version 3 is the only one Iris writes. Reading handles either
+ version and ignores this: zarr-python reads the version from the
+ store, and a version named here that the store disagrees with
+ would be reported as a missing group.
Raises
------
+ ValueError
+ For an unknown `mode`.
NotImplementedError
- For any `mode` other than ``"r"``.
+ For a write `mode` with a `zarr_format` Iris does not write.
+ FileExistsError
+ For ``mode="w-"`` onto a target that already exists.
FileNotFoundError
- If `location` does not exist.
+ If `location` does not exist and `mode` does not create it.
zarr.errors.GroupNotFoundError
If `location` is not a Zarr store, or holds no such `group`.
@@ -541,8 +872,33 @@ def __init__(self, location, mode: str = "r", *, group: str | None = None):
self._attributes: _ZarrAttributes | None = None
self._dimensions: dict[str, int] | None = None
- if mode not in _READ_MODES:
- raise NotImplementedError(f"{_NO_SAVING} Cannot open with mode={mode!r}.")
+ #: A record of dimension names and sizes.
+ #: Zarr stores no dimension object of its own, so this is the only
+ #: record of a declared size, and the only way to catch two arrays
+ #: declaring one name at two sizes.
+ self._declared_dimensions: dict[str, int] = {}
+
+ if mode not in _MODES:
+ message = (
+ f"Unknown mode {mode!r} for a Zarr store. Expected one of {_MODES}."
+ )
+ raise ValueError(message)
+
+ writing = mode in _WRITE_MODES
+ if not writing:
+ # Reading detects the version from the store. Naming one here
+ # would constrain that detection, and a store of the other
+ # version would then be reported as a missing group.
+ zarr_format = None
+ elif zarr_format is None:
+ zarr_format = _DEFAULT_ZARR_FORMAT
+ elif zarr_format not in _WRITABLE_ZARR_FORMATS:
+ message = (
+ f"zarr_format={zarr_format} is not currently supported for "
+ "writing in Iris. Currently supported formats are: "
+ f"{_WRITABLE_ZARR_FORMATS}. "
+ )
+ raise NotImplementedError(message)
self._mode = mode
target = str(location)
@@ -551,7 +907,27 @@ def __init__(self, location, mode: str = "r", *, group: str | None = None):
# `location` itself, not its stringified `target`, is handed to
# zarr-python: stringifying a store object such as a `MemoryStore`
# loses the store, leaving only a name zarr-python cannot reopen.
- self._group = zarr.open_group(store=location, mode=mode, path=group)
+ try:
+ self._group = zarr.open_group(
+ store=location,
+ mode=mode,
+ path=group,
+ zarr_format=zarr_format,
+ # A write mode must see what the store actually holds, not a
+ # frozen consolidated snapshot: `mode="a"` onto a store this
+ # saver already consolidated would otherwise list only the
+ # arrays present at the last consolidation, and never the one
+ # this call is about to add. A read mode keeps the
+ # consolidated fast path, which is what it is for.
+ use_consolidated=False if writing else None,
+ )
+ except FileExistsError as exception:
+ message = (
+ f"Cannot create the Zarr store {self._location!r}: it already "
+ f"exists, and mode={mode!r} will not overwrite it. Pass "
+ "mode='w' to replace it, or mode='a' to add to it."
+ )
+ raise FileExistsError(message) from exception
self._format = _FORMAT_BY_ZARR_VERSION[self._group.metadata.zarr_format]
# Assembled here rather than on first access, because assembling is
@@ -600,11 +976,19 @@ def variables(self) -> Mapping:
@property
def dimensions(self) -> Mapping:
- """The dimension sizes this group's arrays agree on.
+ """The dimension sizes this group's arrays agree on, and any declared.
+
+ Assembled when the store is opened, and again after a write has
+ changed what the answer would be. A dimension declared by
+ :meth:`create_dimension` appears here before any array carries it,
+ because that is what a caller choosing a new name needs to see.
- Assembled when the store is opened, not on first access here.
"""
- assert self._dimensions is not None
+ if self._dimensions is None:
+ self._dimensions = {
+ **_assemble_dimensions(self.variables),
+ **self._declared_dimensions,
+ }
return self._dimensions
@property
@@ -630,25 +1014,197 @@ def attributes(self) -> MutableMapping:
return self._attributes
def create_dimension(self, name: str, size: int | None) -> None:
- """Raise: saving to Zarr is not supported."""
- raise NotImplementedError(_NO_SAVING)
+ """Declare a dimension of the given length.
+
+ Zarr has no dimension object: a dimension is a name an array carries
+ in its own metadata. So this records the name and size, and
+ :meth:`create_variable` writes the names onto each array. The record
+ is what lets a size conflict be caught here rather than becoming two
+ arrays that disagree.
+
+ A name an array in this group already carries counts as taken, at
+ that array's length. ``mode="a"`` is the case that needs it: an
+ existing array is never resized, so a declaration disagreeing with
+ one would leave :attr:`dimensions` reporting a single length for a
+ name two arrays hold at two.
+
+ Parameters
+ ----------
+ name : str
+ The dimension's name.
+ size : int or None
+ Its length. ``None`` requests an unlimited dimension, which Zarr
+ has no form for.
+
+ Raises
+ ------
+ NotImplementedError
+ For ``size=None``.
+ ValueError
+ If `name` is already in use at a different size, or if this
+ dataset was opened read-only.
+
+ """
+ self._check_writable("create_dimension")
+ if size is None:
+ message = (
+ f"Cannot create the unlimited dimension {name!r}: a Zarr array "
+ "has a fixed shape, so there is no unlimited dimension to "
+ "create."
+ )
+ raise NotImplementedError(message)
+
+ declared = self._declared_dimensions.get(name)
+ if declared is not None and declared != size:
+ message = (
+ f"Cannot declare dimension {name!r} with size {size}: it is "
+ f"already declared with size {declared}."
+ )
+ raise ValueError(message)
+ dimensions = dict(self.dimensions)
+ existing = dimensions.get(name)
+ if existing is not None and existing != size:
+ message = (
+ f"Cannot declare dimension {name!r} with size {size}: an array "
+ f"in the Zarr group {self._location!r} already uses that name "
+ f"with size {existing}, and an existing array is never resized."
+ )
+ raise ValueError(message)
+ self._declared_dimensions[name] = size
+ # Updated rather than dropped: dropping it would reassemble it from
+ # every array in the group on the next call, over a remote store.
+ dimensions[name] = size
+ self._dimensions = dimensions
+
+ def _check_writable(self, action: str) -> None:
+ """Raise unless this dataset was opened in a mode that writes."""
+ if self._mode not in _WRITE_MODES:
+ message = (
+ f"Cannot {action} on a Zarr store opened with mode={self._mode!r}. "
+ f"Open it with one of {_WRITE_MODES} to write."
+ )
+ raise ValueError(message)
def create_variable(
- self, name: str, dtype, dimensions=(), *, fill_value=None, **storage_settings
- ):
- """Raise: saving to Zarr is not supported."""
- raise NotImplementedError(_NO_SAVING)
+ self, name: str, dtype, dimensions=(), *, fill_value=None, **encoding
+ ) -> "ZarrDatasetVariable":
+ """Create and return a new array.
+
+ Parameters
+ ----------
+ name : str
+ The array's name within this group.
+ dtype : numpy.dtype or str or type
+ The array's dtype. ``str`` itself asks for a variable-length
+ string array, which is how this store holds text.
+ dimensions : iterable of str, optional
+ The names of its dimensions, each already declared by
+ :meth:`create_dimension`. Empty for a scalar.
+ fill_value : optional
+ The CF missing-value sentinel. When given it is written as a
+ ``_FillValue`` attribute as well as the array's own fill, because
+ on version 3 the attribute is the masking declaration and the
+ storage fill is not. When not given, the array takes the netCDF
+ default fill for its dtype, and declares nothing: the saver
+ declares one if it finds the data is masked.
+ **encoding
+ Zarr's own storage keywords: ``chunks``, ``shards``,
+ ``compressors``, ``filters``, ``serializer``.
+
+ Returns
+ -------
+ :class:`ZarrDatasetVariable`
+
+ Raises
+ ------
+ ValueError
+ If the dataset is read-only, a dimension is undeclared, `name`
+ already exists in this group, Zarr has no data type for `dtype`,
+ or `dtype` cannot hold `fill_value`.
+ TypeError
+ For a storage keyword this store does not have.
+
+ """
+ self._check_writable("create_variable")
+ check_encoding_keywords(encoding)
+ assert self._group is not None
+
+ if dtype is not str:
+ # Native byte order, whatever order was asked for: see the module
+ # docstring, and test_ZarrDataset__write.py's TestByteOrder.
+ dtype = np.dtype(dtype).newbyteorder("=")
+ dimensions = tuple(dimensions)
+ shape = tuple(self._dimension_size(dimension) for dimension in dimensions)
+
+ if name in self._group:
+ raise ValueError(existing_name_message(name, self._location))
+
+ if fill_value is None:
+ # None here means zarr-python's own default for the dtype
+ # applies, and the array declares no sentinel of its own. That
+ # covers a variable-length string array, which the netCDF
+ # default-fill table has no entry for, and every other dtype it
+ # has no entry for either - a Zarr store holds plenty netCDF
+ # does not. Whether a *missing-value* sentinel is needed is a
+ # question about the data, settled by the saver, which is the
+ # only thing that sees it.
+ stored_fill = (
+ None if dtype is str else _encode.default_fill_value(np.dtype(dtype))
+ )
+ else:
+ stored_fill = _storage_fill_value(fill_value, dtype, name)
+
+ create_kwargs = dict(encoding)
+ if stored_fill is not None:
+ create_kwargs["fill_value"] = stored_fill
+
+ array = self._group.create_array(
+ name,
+ shape=shape,
+ dtype=dtype,
+ dimension_names=dimensions or None,
+ **create_kwargs,
+ )
+ variable = ZarrDatasetVariable(array, self._location, self._format)
+ if fill_value is not None:
+ variable.attributes["_FillValue"] = fill_value
+
+ # Both caches describe a group that has just changed.
+ self._variables = None
+ self._dimensions = None
+ return variable
+
+ def _dimension_size(self, name: str) -> int:
+ """Return a declared dimension's size, or raise naming it."""
+ size = self.dimensions.get(name)
+ if size is None:
+ message = (
+ f"Cannot create an array over dimension {name!r}: no dimension "
+ "of that name has been declared in this group."
+ )
+ raise ValueError(message)
+ return size
def supports_dtype(self, dtype) -> bool:
"""Whether this store can hold a variable of this dtype.
- Always True. Zarr's data types cover the whole of NumPy's fixed-width
- numeric range, in both versions, so there is no dtype a caller could
- reach here with that the store would have to narrow. Contrast
+ Always True. :class:`~iris.fileformats.cf.dataset.CFDataset` asks this
+ before narrowing an integer type, and the only dtypes the saver asks
+ it about are ``int64`` and the unsigned integers - see
+ :meth:`iris.fileformats.cf.saver._CFSaver._ensure_valid_dtype`. Zarr
+ has a data type for every one of them, so nothing is ever narrowed for
+ a Zarr store. Contrast
:meth:`~iris.fileformats.netcdf._dataset.NetCDFDataset.supports_dtype`,
which answers False for the integer types the netCDF 3 formats have no
type for.
+ A dtype Zarr genuinely cannot hold - an object array, a zero-length
+ void, one of the platform's extended precisions - is refused by
+ zarr-python in :meth:`create_variable`, with a ``ValueError`` naming
+ the dtype. Answering that question here as well would mean keeping a
+ second table of what zarr-python can store, which no saver consults
+ and which goes stale as zarr-python gains data types.
+
Parameters
----------
dtype : :class:`numpy.dtype`
@@ -665,13 +1221,97 @@ def sync(self) -> None:
"""Do nothing: a Zarr write is not buffered."""
def finalise(self) -> None:
- """Raise: saving to Zarr is not supported.
+ """Write consolidated metadata for the whole store.
+
+ One document at the root listing every array and group, which is what
+ makes a remote store usable: a reader fetches one object instead of
+ one per node. Separate from :meth:`close` because it is a write, and
+ a one-shot one - a worker that fills part of a store must not do it,
+ since several writers of one metadata object race.
- A Zarr store's completion step is consolidating its metadata, which
- only a writer performs.
+ Always consolidates from the root, even when this dataset is one
+ group of the store. A consolidated document that does not mention the
+ group just written is what a default reader would believe.
+
+ Raises
+ ------
+ ValueError
+ If this dataset was opened read-only.
+ RuntimeError
+ If the store cannot be consolidated. Raised rather than returned
+ quietly: a store left with stale consolidated metadata hides the
+ data just written from every default reader.
+
+ """
+ self._check_writable("finalise")
+ assert self._group is not None
+ with warnings.catch_warnings():
+ # zarr-python warns that consolidated metadata is not part of the
+ # version 3 specification. Iris asks for it deliberately, and the
+ # user cannot act on it.
+ warnings.filterwarnings(
+ "ignore",
+ message="Consolidated metadata is currently not part",
+ category=UserWarning,
+ )
+ try:
+ zarr.consolidate_metadata(self._group.store)
+ except Exception as exception:
+ message = (
+ f"Could not consolidate the metadata of the Zarr store "
+ f"{self._location!r}. The data written is in the store, but "
+ "a reader that uses consolidated metadata may not see it."
+ )
+ raise RuntimeError(message) from exception
+
+ def drop_consolidated_metadata(self) -> None:
+ """Remove the store's consolidated metadata document, if it has one.
+
+ The other half of :meth:`finalise`, for a save that declines
+ consolidation. A document an earlier save wrote does not mention what
+ this one has just written, and a default reader believes the
+ document, so the data would be in the store and invisible.
+
+ Works from the whole store's root, like :meth:`finalise`, because
+ that is where the document lives however deep the group written was.
+ Does nothing when the store has no document, or no root group at all.
+
+ Raises
+ ------
+ ValueError
+ If this dataset was opened read-only.
+ RuntimeError
+ If the document cannot be removed, for the same reason
+ :meth:`finalise` raises: a store left carrying it hides the data
+ just written.
"""
- raise NotImplementedError(_NO_SAVING)
+ self._check_writable("drop consolidated metadata")
+ assert self._group is not None
+ store = self._group.store
+ try:
+ consolidated = zarr.open_group(store=store, mode="r+")
+ except (FileNotFoundError, KeyError):
+ # No root group, so nothing at the root to go stale.
+ # zarr-python's `GroupNotFoundError` is a `FileNotFoundError`.
+ return
+ if consolidated.metadata.consolidated_metadata is None:
+ return
+
+ try:
+ # Reopened ignoring the document, so the metadata written back
+ # carries no consolidated section. Rewriting the attributes is
+ # what writes it: zarr-python has no call that removes one.
+ root = zarr.open_group(store=store, mode="r+", use_consolidated=False)
+ root.update_attributes(dict(root.attrs))
+ except Exception as exception:
+ message = (
+ f"Could not remove the consolidated metadata of the Zarr store "
+ f"{self._location!r}. The data written is in the store, but the "
+ "store's consolidated metadata predates it, so a reader that "
+ "uses it may not see what was just written."
+ )
+ raise RuntimeError(message) from exception
def close(self) -> None:
"""Record that this dataset is finished with.
diff --git a/lib/iris/fileformats/zarr/_encode.py b/lib/iris/fileformats/zarr/_encode.py
new file mode 100644
index 0000000000..426c392d53
--- /dev/null
+++ b/lib/iris/fileformats/zarr/_encode.py
@@ -0,0 +1,317 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Value conversions a Zarr store needs on the way in.
+
+The mirror of :mod:`iris.fileformats.zarr._decode`: pure functions of their
+arguments, with no zarr-python call and no Iris object between them, so that
+every rule can be tested on a plain value.
+
+**Attribute values are converted before zarr sees them, never after.** A value
+zarr's JSON encoder rejects raises, *and stays in the attributes mapping*, so a
+caller that converts on failure has already corrupted the store's metadata.
+
+**``_FillValue`` and ``missing_value`` are encoded differently, and the
+difference is not Iris's.** On a floating array, released xarray reads
+``_FillValue`` through zarr's own fill-value parser, which wants the base64 of
+a little-endian ``float64`` and fails the *whole dataset open* on a plain JSON
+number. It reads ``missing_value`` as an ordinary attribute, where base64 is a
+string that masks nothing. So the first is packed and the second is not. See
+§4.5 of the native Zarr I/O design,
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+**Packing is applied here, not assumed from the backend.** netCDF4-python
+applies ``scale_factor``/``add_offset`` packing itself, on assignment, once
+those attributes are set on the variable - which is why the generic CF saver
+only ever writes the attributes and streams the unpacked values through
+unchanged. Zarr has no such behaviour, so :func:`pack` does the arithmetic
+that would otherwise silently not happen. It is the mirror of
+:func:`iris.fileformats.zarr._decode.unpack`, called from
+:meth:`iris.fileformats.zarr._dataset.ZarrDatasetVariable.__setitem__` before
+the fill described above, because the fill value lives in *packed* space and
+the data arrives in *unpacked* space - packing an already-filled sentinel
+would put it through the same transform as real data and lose it.
+
+"""
+
+import base64
+import json
+import struct
+from typing import Any
+
+import dask.array as da
+import numpy as np
+
+from iris._lazy_data import is_lazy_data
+
+#: The format of a base64 ``_FillValue``: one little-endian float64, whatever
+#: the array's own floating width.
+_PACKED_DOUBLE = " Any:
+ """Return `value` as something zarr's JSON encoder accepts.
+
+ Parameters
+ ----------
+ value : Any
+ The attribute value, as the cube holds it.
+ name : str
+ The attribute's name, for the error message. A failure here is a
+ value the user wrote, so the message has to say which one.
+
+ Returns
+ -------
+ Any
+ A JSON-serialisable value: NumPy scalars become Python ones, NumPy
+ arrays become lists, ``bytes`` become ``str`` via ASCII, a ``list``
+ or ``tuple`` becomes a list of converted elements, and everything
+ else is returned unchanged.
+
+ Raises
+ ------
+ TypeError
+ If the value cannot be stored. It is **not** converted to its string
+ form: a string that cannot be read back is a worse answer than a
+ refusal, and a warning would not stop the save.
+
+ """
+ converted: Any
+ if isinstance(value, bytes):
+ try:
+ converted = value.decode("ascii")
+ except UnicodeDecodeError as exception:
+ message = (
+ f"Cannot save attribute {name!r}: its bytes are not ascii, "
+ "and a Zarr attribute is text."
+ )
+ raise TypeError(message) from exception
+ elif isinstance(value, np.generic):
+ converted = value.item()
+ elif isinstance(value, np.ndarray):
+ converted = value.tolist()
+ elif isinstance(value, (list, tuple)):
+ # Element by element, because a sequence of NumPy scalars is an
+ # ordinary thing to build - `[data.min(), data.max()]`, say - and
+ # passing it through unexamined leaves the NumPy inside it for
+ # json.dumps to reject. A tuple comes back as a list because JSON
+ # has no tuple, which the rule above already settles for an array.
+ converted = [json_native(item, name=name) for item in value]
+ else:
+ converted = value
+
+ try:
+ json.dumps(converted)
+ except TypeError as exception:
+ message = (
+ f"Cannot save attribute {name!r}: a value of type "
+ f"{type(value).__name__} has no JSON form. Convert it to a number, "
+ "a string, or a list of either."
+ )
+ raise TypeError(message) from exception
+
+ return converted
+
+
+def encode_fill_value(value: Any, dtype: np.dtype) -> Any:
+ """Return a ``_FillValue`` attribute in the encoding readers expect.
+
+ Parameters
+ ----------
+ value : Any
+ The sentinel, in any numeric form.
+ dtype : numpy.dtype
+ The array's dtype, which decides the encoding.
+
+ Returns
+ -------
+ str or int or float
+ For a floating array, the base64 of the little-endian ``float64``
+ form - including for ``float32``, which is what released xarray
+ reads. For anything else, a plain JSON value.
+
+ """
+ if dtype.kind == "f":
+ packed = struct.pack(_PACKED_DOUBLE, float(value))
+ return base64.b64encode(packed).decode("ascii")
+ return json_native(value, name="_FillValue")
+
+
+def encode_missing_values(value: Any, dtype: np.dtype) -> Any:
+ """Return a ``missing_value`` attribute as plain JSON numbers.
+
+ Never base64, whatever the dtype: xarray reads this attribute as an
+ ordinary value, so a packed string there opens cleanly and masks nothing.
+ CF section 2.5.1 allows a vector here, so a sequence stays a sequence.
+
+ Parameters
+ ----------
+ value : Any
+ One sentinel, or a sequence of them.
+ dtype : numpy.dtype
+ The array's dtype. Unused, and named so that the two encoders have
+ one signature: a caller choosing between them should not have to
+ learn which one wants the dtype.
+
+ Returns
+ -------
+ int or float or list
+
+ """
+ del dtype
+ return json_native(value, name="missing_value")
+
+
+def default_fill_value(dtype: np.dtype) -> np.generic | None:
+ """Return the sentinel to use for a masked source that declares none.
+
+ Parameters
+ ----------
+ dtype : numpy.dtype
+ The array's dtype.
+
+ Returns
+ -------
+ numpy.generic or None
+ The netCDF default fill for that dtype, as a value of it, or
+ ``None`` for a dtype the table has no entry for.
+
+ Notes
+ -----
+ ``None`` rather than an error, because the table is netCDF's and a Zarr
+ store holds a good deal netCDF does not - ``float16``, ``complex64``, a
+ fixed-width byte string. Nothing about such a dtype stops it being
+ stored; what it lacks is a *missing-value* sentinel, which only matters
+ once the data turns out to be masked. That is known in
+ :meth:`iris.fileformats.zarr.saver.Saver._backend_write`, and refused
+ there, where the message can say so truthfully.
+
+ """
+ key = dtype.str[1:]
+ if key not in DEFAULT_FILL_VALUES:
+ return None
+ return dtype.type(DEFAULT_FILL_VALUES[key])
+
+
+def filled(data, fill_value):
+ """Return `data` with any masked points replaced by `fill_value`.
+
+ A Zarr array stores the fill in the data, as spec §4.5 settles: no
+ separate mask array. Writing a masked array to one raises inside numpy,
+ so this is not optional.
+
+ Parameters
+ ----------
+ data : numpy.ndarray or dask.array.Array
+ Real or lazy, masked or not.
+ fill_value : Any
+ What replaces the masked points.
+
+ Returns
+ -------
+ numpy.ndarray or dask.array.Array
+ Lazy if `data` was lazy. An unmasked input is returned unchanged,
+ not copied.
+
+ """
+ if is_lazy_data(data):
+ return da.ma.filled(data, fill_value)
+ if np.ma.isMaskedArray(data):
+ return data.filled(fill_value)
+ return data
+
+
+def pack(data, scale_factor, add_offset, dtype: np.dtype):
+ """Return `data` packed by CF's scale and offset, as `dtype`.
+
+ The inverse of :func:`iris.fileformats.zarr._decode.unpack`:
+ ``packed = round((unpacked - add_offset) / scale_factor)``. Rounding
+ applies only when `dtype` is an integer type - exactly netCDF4-python's
+ own transform, not a rounding version of it applied regardless of target -
+ so a non-integer `dtype` is merely narrowed, the same as :func:`unpack`
+ narrows an already-unpacked value. CF packing is round-to-nearest, not
+ truncation - ``int()`` would truncate toward zero and bias every negative
+ value by up to one count.
+
+ The arithmetic runs in float64 regardless of `dtype`: `scale_factor` and
+ `add_offset` reach here as plain Python floats, decoded from JSON, which
+ has no float32. A result can therefore differ from netCDF4's float32
+ computation of the same packing by one count - the float64 answer is the
+ more accurate one, but it is not byte-identical to netCDF4's.
+
+ Parameters
+ ----------
+ data : numpy.ndarray, numpy.ma.MaskedArray or dask.array.Array
+ The values to write, in physical units. A mask, if any, is carried
+ through the arithmetic unchanged and is still present in the result.
+ scale_factor, add_offset : optional
+ CF's packing parameters, read from the variable's own attributes.
+ Either absent behaves as its identity - no offset, unit scale.
+ dtype : numpy.dtype
+ The array's stored dtype, which the result is narrowed to.
+
+ Returns
+ -------
+ numpy.ndarray, numpy.ma.MaskedArray or dask.array.Array
+ `data` itself, unchanged, when neither parameter is given - a
+ variable with no packing attributes keeps its own dtype untouched,
+ rather than being rounded and recast to a dtype it already has.
+
+ Raises
+ ------
+ ValueError
+ If `scale_factor` is zero, naming it: every value would divide by
+ zero, silently becoming infinite or NaN rather than raising on its
+ own.
+
+ """
+ if scale_factor == 0:
+ message = "Cannot pack with scale_factor=0: every value would divide by zero."
+ raise ValueError(message)
+ if scale_factor is None and add_offset is None:
+ return data
+ result = data
+ if add_offset is not None:
+ result = result - add_offset
+ if scale_factor is not None:
+ result = result / scale_factor
+
+ def _round_and_narrow(block):
+ """Round (if `dtype` is an integer type) and cast one block."""
+ if dtype.kind in ("i", "u"):
+ block = np.round(block)
+ if np.ma.isMaskedArray(block):
+ # Not `block.astype(dtype)`: a masked array carries its own
+ # `fill_value`, inherited from `block`'s own (floating) dtype -
+ # typically a value like ``1e+20`` - and `MaskedArray.astype`
+ # casts that too, into a dtype it was never meant to fit. numpy
+ # warns "invalid value encountered in cast" for that, about a
+ # value this function never writes anywhere; reconstructing from
+ # the underlying data and mask lets the new dtype's own default
+ # apply instead, and casts only the real values.
+ return np.ma.masked_array(
+ block.data.astype(dtype, copy=False), mask=block.mask
+ )
+ return block.astype(dtype, copy=False)
+
+ if is_lazy_data(result):
+ return result.map_blocks(_round_and_narrow, dtype=dtype)
+ return _round_and_narrow(result)
diff --git a/lib/iris/fileformats/zarr/saver.py b/lib/iris/fileformats/zarr/saver.py
new file mode 100644
index 0000000000..73cd4798a3
--- /dev/null
+++ b/lib/iris/fileformats/zarr/saver.py
@@ -0,0 +1,548 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Save cubes to a Zarr store.
+
+.. z_reference:: iris.fileformats.zarr.saver
+ :tags: topic_load_save
+
+ API reference
+
+Writes Zarr version 3 stores. The CF work - dimension names, coordinate,
+bounds, mesh and grid-mapping variables, attribute compliance - belongs to
+``iris.fileformats.cf.saver._CFSaver``, which this builds on. What is here
+is what a Zarr store does differently: it is created rather than opened, it
+holds text natively, and it finishes with a consolidated metadata document.
+
+**The storage keywords are zarr-python's own**: ``chunks``, ``shards``,
+``compressors``, ``filters``, ``serializer``. A netCDF keyword raises and names
+the Zarr replacement. Translating ``zlib=True`` onto a Blosc codec would sell a
+false equivalence, and would hand this new API the shape the netCDF saver is
+stuck with. See §4.5 of the native Zarr I/O design,
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+**The default mode is ``"w-"``**, which refuses an existing target, where the
+netCDF saver clobbers one. A Zarr clobber is many non-atomic deletes, so an
+interrupted one destroys the old store without completing the new.
+
+"""
+
+import dask
+from dask.delayed import Delayed
+import numpy as np
+
+import iris
+from iris._lazy_data import is_lazy_data, is_lazy_masked_data
+from iris.cube import CubeList
+from iris.fileformats.cf import saver as cf_saver
+from iris.fileformats.cf.saver import _CFSaver
+import iris.io
+
+from . import _encode
+from ._dataset import ZarrDataset, check_encoding_keywords, existing_name_message
+
+__all__ = ["Saver", "save"]
+
+#: The :attr:`numpy.dtype.kind` codes of a text array: ``T`` for the
+#: variable-length string type a Zarr store holds text in, ``U`` and ``S``
+#: for the fixed-width unicode and bytes a cube may arrive with.
+_TEXT_KINDS = frozenset(("T", "U", "S"))
+
+
+def _tiles_the_grid(chunks, grid) -> bool:
+ """Whether every chunk boundary falls on a write-grid boundary.
+
+ Parameters
+ ----------
+ chunks : tuple of tuple of int
+ A dask array's ``chunks``: the lengths along each dimension.
+ grid : tuple of int
+ The write-grid cell shape.
+
+ Returns
+ -------
+ bool
+ True when no grid cell is touched by two chunks. A truncated final
+ cell is permitted, so the last boundary is not checked.
+
+ """
+ for lengths, cell in zip(chunks, grid, strict=True):
+ offset = 0
+ for length in lengths[:-1]:
+ offset += length
+ if offset % cell:
+ return False
+ return True
+
+
+def _align_to_write_grid(data, cf_var):
+ """Return `data` chunked so that concurrent writes cannot collide.
+
+ Parameters
+ ----------
+ data : dask.array.Array
+ The lazy source about to be stored.
+ cf_var : :class:`~iris.fileformats.zarr._dataset.ZarrDatasetVariable`
+ The array it is going into.
+
+ Returns
+ -------
+ dask.array.Array
+ `data` itself when it already tiles the write grid - a needless
+ rechunk is a needless graph layer - and a rechunked view otherwise.
+
+ """
+ grid = cf_var.write_grid
+ if not grid or _tiles_the_grid(data.chunks, grid):
+ return data
+ return data.rechunk(grid)
+
+
+def _complete(_written, dataset, consolidating) -> None:
+ """Settle `dataset`'s metadata document, once its delayed writes have run.
+
+ A module-level function, rather than a closure, so that the
+ :class:`~dask.delayed.Delayed` built from it survives a plain
+ ``pickle.dumps``/``pickle.loads`` round trip and not only a cloudpickle
+ one - what a :mod:`distributed` scheduler would use regardless.
+
+ Parameters
+ ----------
+ _written : Any
+ The result of the delayed writes this is chained after. Unused,
+ beyond making this task depend on them.
+ dataset : :class:`~iris.fileformats.zarr._dataset.ZarrDataset`
+ The store to consolidate.
+ consolidating : bool
+ Whether to consolidate at all. Declining removes any document the
+ store already carried, rather than leaving a stale one.
+
+ """
+ if consolidating:
+ dataset.finalise()
+ else:
+ dataset.drop_consolidated_metadata()
+
+
+class Saver(_CFSaver):
+ """A manager for saving cubes to a Zarr store."""
+
+ def __init__(
+ self,
+ filename,
+ compute=True,
+ *,
+ mode=None,
+ group=None,
+ consolidate_metadata=True,
+ ):
+ """Manage saving cubes into a Zarr store.
+
+ Parameters
+ ----------
+ filename : str or pathlib.Path
+ The store's path or URL. A URL is handed to zarr-python
+ unaltered.
+ compute : bool, default=True
+ If ``True``, lazy data is written and the metadata consolidated
+ on exit from the saver context. If ``False``, those writes are
+ returned as a :class:`~dask.delayed.Delayed` by
+ :meth:`delayed_completion`, which also owns the consolidation.
+ mode : str, optional
+ One of ``"w-"``, ``"w"``, ``"a"`` or ``"r+"``. The default is
+ ``"w-"``, which refuses an existing target, or ``"a"`` when
+ `group` is given, since adding a group to a store means the store
+ may already be there.
+ group : str, optional
+ A ``/``-separated path to a group within the store. The root
+ group by default.
+ consolidate_metadata : bool, default=True
+ Whether to write a consolidated metadata document at the root
+ when the save completes. Consolidation is what makes a remote
+ store usable; declining it suits a store that will be added to
+ again before anyone reads it. Declining it when adding to a
+ store that already carries one *removes* that document rather
+ than leaving it: the store is opened without consulting it, so
+ the root metadata is rewritten without one. Slower to read, and
+ true, where a stale document would hide what was just written.
+
+ Examples
+ --------
+ >>> import iris # doctest: +SKIP
+ >>> from iris.fileformats.zarr.saver import Saver # doctest: +SKIP
+ >>> with Saver("out.zarr") as saver: # doctest: +SKIP
+ ... for cube in cubes: # doctest: +SKIP
+ ... saver.write(cube) # doctest: +SKIP
+
+ """
+ if mode is None:
+ mode = "a" if group is not None else "w-"
+ dataset = ZarrDataset(filename, mode=mode, group=group)
+ super().__init__(dataset, compute=compute)
+ #: Whether the completed store gets a consolidated metadata document.
+ self.consolidate_metadata = consolidate_metadata
+ #: Names the group already held when this `Saver` opened it.
+ #: Snapshotted once, here, rather than read live from
+ #: `self._dataset.variables`: that mapping is invalidated on every
+ #: `create_variable` call, so a live read would also catch a name
+ #: this same call created moments before - right for something the
+ #: group held before this call, wrong for two cubes of one call
+ #: sharing a name.
+ self._pre_existing_names = frozenset(dataset.variables)
+ # The dimensions the group already carries are dimensions this save
+ # has "seen", so that `_CFSaver._get_dim_names` disambiguates against
+ # them: an anonymous cube dimension is named "dim0" whatever cube it
+ # came from, so a second save into one store routinely claims a name
+ # the store already uses. Without this the generic layer declares
+ # nothing - the name is already in `dataset.dimensions` - and
+ # `create_variable` sizes the new array from the store's length
+ # instead of the cube's, which zarr-python then truncates to in
+ # silence. Seeded here rather than in `_CFSaver` because only a store
+ # that can be added to has anything to seed; under "w-" and "w" the
+ # group is new and this is empty. The disambiguation loops then
+ # increment "dim0" to "dim0_0" for a conflicting length and reuse it
+ # for a matching one, which is what §4.5 of the design already
+ # promises for coordinates.
+ self._existing_dim.update(dataset.dimensions)
+
+ def write(self, cube, *args, **kwargs):
+ """Write a cube into the store.
+
+ Takes what ``iris.fileformats.cf.saver._CFSaver.write`` takes,
+ with Zarr's storage keywords in ``**encoding``. For a cube whose data
+ is lazy, ``chunks`` defaults to the cube's own lazy chunking, so that
+ the data is written in the shape it is held in, and so that the dask
+ source tiles the array's write grid without being rechunked. Real
+ data gets zarr-python's automatic chunking, which is what it has no
+ chunking of its own to offer.
+
+ The default is dropped when ``shards`` is given. A shard must be a
+ whole number of inner chunks, and Dask's chunking has no reason to
+ divide a shard the user chose for storage - so supplying both would
+ fail over a number the user never passed. With only ``shards``
+ given, zarr-python picks an inner chunk that fits it.
+
+ Raises
+ ------
+ ValueError
+ If the cube's data variable would collide with something the
+ group already held before this call, under a mode that adds to
+ an existing store. Left to the generic layer, such a collision
+ is disambiguated by renaming rather than raised: right for two
+ cubes of one call sharing a name, wrong for a store this call did
+ not just create, where adding is not a merge.
+
+ """
+ if self._dataset.mode in ("a", "r+"):
+ cf_name = self._get_element_variable_name(cube, cube)
+ if cf_name in self._pre_existing_names:
+ raise ValueError(existing_name_message(cf_name, self._dataset.location))
+ if "chunks" not in kwargs and "shards" not in kwargs and cube.has_lazy_data():
+ kwargs["chunks"] = cube.lazy_data().chunksize
+ super().write(cube, *args, **kwargs)
+
+ def _create_string_variable(
+ self, cube_or_mesh, cf_name, element, element_dims, data
+ ):
+ """Create a variable-length string array.
+
+ Zarr has a string type, so unlike netCDF there is no trailing
+ character dimension to invent and `element_dims` is left alone.
+
+ Parameters
+ ----------
+ cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
+ The Cube or Mesh being saved. Unused here.
+ cf_name : str
+ The name the variable is created with.
+ element : :class:`iris.coords._DimensionalMetadata`
+ The element being saved. Unused here.
+ element_dims : list of str
+ The variable's dimension names. **Not** appended to.
+ data : :class:`numpy.ndarray`
+ The string values. Unused here: a Zarr string array has no width.
+
+ Returns
+ -------
+ :class:`~iris.fileformats.zarr._dataset.ZarrDatasetVariable`
+
+ """
+ return self._dataset.create_variable(cf_name, str, element_dims)
+
+ def _backend_write(self, data, cf_var) -> bool:
+ """Declare a mask, align a lazy source, and collect it.
+
+ Parameters
+ ----------
+ data : array-like
+ The values to write.
+ cf_var : :class:`~iris.fileformats.zarr._dataset.ZarrDatasetVariable`
+ The variable to write them into.
+
+ Returns
+ -------
+ bool
+ True when the write was collected here, which it is for lazy
+ data. Real data is left to the base class, which writes it
+ straight in.
+
+ Raises
+ ------
+ ValueError
+ If the data is masked and its dtype has no netCDF default fill
+ value, so that there is no sentinel to declare. This is the one
+ place the mask and the dtype are both known. A text variable
+ gets its own message: a string has no sentinel at all, so there
+ is nothing the caller could pass instead.
+
+ """
+ # `np.ma.is_masked` is true only when something is actually masked,
+ # unlike `np.ma.isMaskedArray`, which is true of an np.ma array whose
+ # mask is all-False - and Iris produces those routinely, so that
+ # test would declare a missing-value sentinel on data the source
+ # never marked missing. Lazy data cannot be inspected without
+ # computing it, so `is_lazy_masked_data` is the safe answer there:
+ # declaring unnecessarily costs an attribute, where not declaring
+ # could lose a real mask silently.
+ if (
+ is_lazy_masked_data(data) or np.ma.is_masked(data)
+ ) and "_FillValue" not in cf_var.attributes:
+ if _encode.default_fill_value(cf_var.dtype) is None:
+ # No netCDF default for this dtype, so the array took
+ # zarr-python's own - which says what unwritten storage reads
+ # back as, not that a value is missing. Declaring that as
+ # `_FillValue` would mask every point that legitimately holds
+ # it: every zero of a numeric array, every empty string of a
+ # text one.
+ if cf_var.dtype.kind in _TEXT_KINDS:
+ # A text array's own fill is the empty string, and
+ # `iris.fileformats.zarr._decode` does not restore a mask
+ # from one - it warns and reads the empty strings back as
+ # data. So there is no sentinel to write and none the
+ # caller could supply instead, and writing one anyway
+ # would lose the mask in silence. The netCDF saver cannot
+ # store this either.
+ message = (
+ f"Cannot save the masked string variable "
+ f"{cf_var.name!r} to Zarr: a string has no "
+ "missing-value sentinel, so the mask could not be "
+ "read back."
+ )
+ else:
+ message = (
+ f"Cannot save masked data of dtype {cf_var.dtype} into "
+ f"the Zarr array {cf_var.name!r}: there is no default "
+ "missing-value sentinel for that type. Pass an "
+ "explicit 'fill_value'."
+ )
+ raise ValueError(message)
+ # On version 3 the attribute is the masking declaration and the
+ # storage fill is not, so a store written with the fill alone
+ # reads back with every masked point unmasked. Declared here
+ # rather than when the data is written, because under
+ # compute=False that is after the metadata is consolidated.
+ cf_var.attributes["_FillValue"] = cf_var.fill_value
+
+ if not is_lazy_data(data):
+ return False
+
+ self._delayed_writes.append(
+ (_align_to_write_grid(data, cf_var), cf_var.write_handle())
+ )
+ return True
+
+ def delayed_completion(self) -> Delayed:
+ """Return a :class:`~dask.delayed.Delayed` that completes the store.
+
+ The lazy writes first, then the metadata consolidation, which is the
+ store's last write: a store whose deferred save never runs should not
+ carry a metadata document claiming it did.
+
+ Returns
+ -------
+ :class:`dask.delayed.Delayed`
+ A real one. The inherited
+ ``_CFSaver.delayed_completion``
+ returns a :class:`dask.array.core.Array` for one delayed write,
+ or a tuple of them for several - never actually a
+ :class:`~dask.delayed.Delayed`, despite its own annotation -
+ which is what makes the documented ``result.compute()`` fail on
+ the netCDF path
+ (`SciTools/iris#7291 `_).
+ Wrapping it here is what keeps the promise this saver's docstring
+ makes. Plain ``pickle`` survives the round trip too, not only
+ cloudpickle: the completion is :func:`_complete`, a module-level
+ function with the dataset and the consolidation flag bound as
+ plain values, never reached through ``self`` - the saver holds
+ cubes, which must not be dragged into the graph.
+
+ """
+ complete = dask.delayed(_complete)
+ return complete(
+ super().delayed_completion(), self._dataset, self.consolidate_metadata
+ )
+
+
+def _per_cube(value, count, name):
+ """Return one `value` per cube, from one value or a sequence of them."""
+ if isinstance(value, (list, tuple)):
+ if len(value) != count:
+ message = (
+ f"If {name} is a sequence, it must have one element per cube: "
+ f"got {len(value)} for {count} cubes."
+ )
+ raise ValueError(message)
+ return list(value)
+ return [value] * count
+
+
+def _check_packing(packspecs) -> None:
+ """Raise for any packspec whose dtype is not a numpy integer type.
+
+ The same rule :meth:`iris.fileformats.netcdf.saver.Saver.write` enforces,
+ and the same message: CF packing's arithmetic, ``round((value -
+ add_offset) / scale_factor)``, only means what it says for an integer
+ target, so a non-integer dtype is refused before the store is created,
+ rather than accepted and silently rounded.
+
+ Parameters
+ ----------
+ packspecs : iterable
+ One packing specification per cube, as :func:`_per_cube` expands
+ them: each is ``None``, a dtype (a :class:`numpy.dtype`, a type, or a
+ string), or a dict with a ``"dtype"`` key.
+
+ Raises
+ ------
+ ValueError
+
+ """
+ for packspec in packspecs:
+ if packspec is None:
+ continue
+ dtype = packspec
+ if isinstance(packspec, dict):
+ if "dtype" not in packspec:
+ message = "The argument to packing must contain the key 'dtype'."
+ raise ValueError(message)
+ dtype = packspec["dtype"]
+ if np.dtype(dtype).kind not in ("i", "u"):
+ message = "The packing datatype must be a numpy integer type."
+ raise ValueError(message)
+
+
+def save(
+ cube,
+ filename,
+ *,
+ mode=None,
+ group=None,
+ consolidate_metadata=True,
+ local_keys=None,
+ fill_value=None,
+ packing=None,
+ compute=True,
+ **encoding,
+):
+ """Save cube(s) to a Zarr store.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube` or :class:`iris.cube.CubeList`
+ The cube, or cubes, to save. A whole :class:`~iris.cube.CubeList`
+ goes into one store, which is opened once.
+ filename : str or pathlib.Path
+ The store's path or URL.
+ mode : str, optional
+ See :class:`Saver`. ``"w-"`` by default, ``"a"`` when `group` is
+ given.
+ group : str, optional
+ A ``/``-separated path to a group within the store.
+ consolidate_metadata : bool, default=True
+ See :class:`Saver`.
+ local_keys : iterable of str, optional
+ Cube attribute names that become attributes of the data variable
+ rather than global attributes.
+ fill_value : optional
+ The value of the data variable's ``_FillValue`` attribute, and of the
+ array's own fill. One value, or one per cube.
+ packing : type or str or dict or list, optional
+ How to pack the data, if at all. See
+ :meth:`iris.fileformats.netcdf.saver.Saver.write`.
+ compute : bool, default=True
+ If ``False``, returns a :class:`~dask.delayed.Delayed` that completes
+ the store when computed, instead of writing lazy data here.
+ **encoding
+ Zarr's own storage keywords for the data variable: ``chunks``,
+ ``shards``, ``compressors``, ``filters``, ``serializer``. A netCDF
+ keyword raises :class:`TypeError` naming the Zarr equivalent, and so
+ does a keyword a version 3 store has no use for, such as ``order``.
+
+ Returns
+ -------
+ None or :class:`dask.delayed.Delayed`
+ A ``Delayed`` if `compute` is ``False``, otherwise ``None``.
+
+ Notes
+ -----
+ An interrupted save leaves a partial store. Iris offers no rollback: a
+ directory rename is atomic on a local filesystem but has no equivalent on
+ an object store, so the guarantee would evaporate where Zarr is most used.
+
+ """
+ # Validated before anything is created, so that a mistyped keyword does
+ # not leave an empty store behind.
+ check_encoding_keywords(encoding)
+
+ cubes = CubeList([cube]) if isinstance(cube, iris.cube.Cube) else CubeList(cube)
+ fill_values = _per_cube(fill_value, len(cubes), "fill_value")
+ packspecs = _per_cube(packing, len(cubes), "packing")
+ _check_packing(packspecs)
+
+ # A store holds one value of a global attribute and a list of cubes may
+ # carry several, so which of them can be global at all has to be settled
+ # first. That is cube logic rather than storage logic, which is why it is
+ # the same call the netCDF saver makes; done before the store is opened,
+ # so that a warning about it arrives before anything is created.
+ cubes, local_keys, global_attributes = cf_saver.marshal_attributes(
+ cubes, local_keys, "Zarr"
+ )
+
+ with Saver(
+ filename,
+ compute=compute,
+ mode=mode,
+ group=group,
+ consolidate_metadata=consolidate_metadata,
+ ) as saver:
+ for one, packspec, one_fill in zip(cubes, packspecs, fill_values):
+ saver.write(
+ one,
+ local_keys=local_keys,
+ packing=packspec,
+ fill_value=one_fill,
+ **encoding,
+ )
+
+ conventions = cf_saver.CF_CONVENTIONS_VERSION
+ if iris.FUTURE.save_split_attrs:
+ # Under split-attribute saving `_CFSaver.write` writes no global
+ # attributes at all, so this one call is the whole of them. In
+ # the legacy mode it has already written each cube's as it went,
+ # and `global_attributes` is empty.
+ global_attributes["Conventions"] = conventions
+ saver.update_global_attributes(global_attributes)
+ else:
+ saver.update_global_attributes(Conventions=conventions)
+
+ return None if compute else saver.delayed_completion()
+
+
+# `iris.save` decides whether a saver takes a whole CubeList by identity, and
+# registers the savers it knows about in `iris.io._check_init_savers`, which
+# must not import `zarr`. The extension route reaches `iris.io._zarr_save`,
+# registered there; a caller passing `iris.fileformats.zarr.save` directly
+# hands over this function instead, so it registers itself here - by which
+# point zarr is imported anyway, this module having needed it.
+iris.io._savers.cubelist_savers.add(save)
diff --git a/lib/iris/io/__init__.py b/lib/iris/io/__init__.py
index 9889aaf31b..08c3b27b03 100644
--- a/lib/iris/io/__init__.py
+++ b/lib/iris/io/__init__.py
@@ -442,6 +442,18 @@ def _dot_save_png(cube, target, **kwargs):
return save_png(cube, target, **kwargs)
+def _zarr_save(cube, target, **kwargs):
+ # A simple wrapper for `iris.fileformats.zarr.save` which allows the
+ # saver to be registered without triggering the import of `zarr`, an
+ # optional dependency. Importing the name from the package rather than
+ # from `iris.fileformats.zarr.saver` runs the package's `__getattr__`,
+ # which is what checks the installed zarr version and reports a missing
+ # or too-old one.
+ from iris.fileformats.zarr import save
+
+ return save(cube, target, **kwargs)
+
+
def _grib_save(cube, target, append=False, **kwargs):
# A simple wrapper for the grib save routine, which allows the saver to be
# registered without having the grib implementation installed.
@@ -463,6 +475,7 @@ def _check_init_savers():
{
"pp": pp.save,
"nc": netcdf.save,
+ "zarr": _zarr_save,
"dot": _dot_save,
"dotpng": _dot_save_png,
"grib2": _grib_save,
@@ -470,6 +483,7 @@ def _check_init_savers():
)
# One line per format that takes a CubeList whole.
_savers.cubelist_savers.add(netcdf.save)
+ _savers.cubelist_savers.add(_zarr_save)
def add_saver(file_extension, new_saver):
@@ -529,7 +543,7 @@ def find_saver(filespec):
def save(source, target, saver=None, **kwargs):
"""Save one or more Cubes to file (or other writeable).
- Iris currently supports four file formats for saving. Three are
+ Iris currently supports five file formats for saving. Four are
recognised by filename extension:
* **netCDF** - the Unidata network Common Data Format,
@@ -538,8 +552,11 @@ def save(source, target, saver=None, **kwargs):
see :func:`iris_grib.save_grib2`.
* **PP** - the Met Office UM Post Processing Format,
see :func:`iris.fileformats.pp.save`
+ * **Zarr** - a native Zarr version 3 store, written through zarr-python,
+ see :func:`iris.fileformats.zarr.saver.save`. See
+ :doc:`/user_manual/explanation/zarr_io` for more.
- The fourth format is recognised differently:
+ The fifth format is recognised differently:
* **Zarr (via NcZarr)** - "The open foundation for chunked, compressed,
N-dimensional
diff --git a/lib/iris/tests/integration/zarr/__init__.py b/lib/iris/tests/integration/zarr/__init__.py
index b7761f385a..eff4b52f0d 100644
--- a/lib/iris/tests/integration/zarr/__init__.py
+++ b/lib/iris/tests/integration/zarr/__init__.py
@@ -9,4 +9,4 @@
import pytest
-pytest.importorskip("zarr", minversion="3.0.8")
+pytest.importorskip("zarr", minversion="3.1.0")
diff --git a/lib/iris/tests/integration/zarr/test_cross_reader.py b/lib/iris/tests/integration/zarr/test_cross_reader.py
new file mode 100644
index 0000000000..1045ee875d
--- /dev/null
+++ b/lib/iris/tests/integration/zarr/test_cross_reader.py
@@ -0,0 +1,152 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Iris and xarray read each other's Zarr stores.
+
+The check that justifies two decisions in the design. Attributes are written
+as JSON-native values, losing their NumPy dtype, *so that* every other Zarr
+consumer can read them; and a floating ``_FillValue`` is written as base64,
+against CF section 2.5.1, because released xarray fails the whole dataset open
+on a plain JSON number. ``missing_value`` is the opposite case of the same
+pair: it is written as a plain number, never base64, because base64 there
+opens cleanly and masks nothing - a quieter failure than refusing to open at
+all, and one a unit test on the encoder alone cannot see.
+
+Both are only worth anything if someone checks, and xarray is the reader to
+check against: it is what most of the Zarr world reads with.
+"""
+
+import numpy as np
+import pytest
+
+zarr = pytest.importorskip("zarr")
+xr = pytest.importorskip("xarray")
+
+import iris
+from iris.coords import DimCoord
+from iris.cube import Cube
+
+
+def small_cube(dtype="f4", masked=False):
+ values = np.arange(6, dtype=dtype).reshape(2, 3)
+ if masked:
+ values = np.ma.masked_array(values, mask=[[False, True, False]] * 2)
+ cube = Cube(values, standard_name="air_temperature", units="K")
+ cube.var_name = "air_temperature"
+ cube.add_dim_coord(
+ DimCoord(
+ np.array([0.0, 1.0], dtype="f4"),
+ standard_name="latitude",
+ units="degrees",
+ var_name="latitude",
+ ),
+ 0,
+ )
+ cube.add_dim_coord(
+ DimCoord(
+ np.array([0.0, 1.0, 2.0], dtype="f4"),
+ standard_name="longitude",
+ units="degrees",
+ var_name="longitude",
+ ),
+ 1,
+ )
+ return cube
+
+
+class TestXarrayReadsIris:
+ @pytest.mark.parametrize("dtype", ["f4", "f8", "i2", "i4"])
+ def test_a_store_opens(self, tmp_path, dtype):
+ # Asserted on the open, not on the values: the failure mode is a
+ # TypeError during open that takes the whole dataset with it.
+ path = tmp_path / f"out_{dtype}.zarr"
+ iris.save(small_cube(dtype=dtype), path)
+ dataset = xr.open_zarr(path)
+ assert "air_temperature" in dataset.data_vars
+
+ @pytest.mark.parametrize("dtype", ["f4", "f8", "i2"])
+ def test_a_masked_store_opens_and_masks(self, tmp_path, dtype):
+ path = tmp_path / f"masked_{dtype}.zarr"
+ iris.save(small_cube(dtype=dtype, masked=True), path)
+ values = xr.open_zarr(path)["air_temperature"].values
+ assert np.isnan(values[0, 1])
+ assert not np.isnan(values[0, 0])
+
+ def test_an_explicit_finite_sentinel_masks(self, tmp_path):
+ path = tmp_path / "sentinel.zarr"
+ iris.save(small_cube(masked=True), path, fill_value=np.float32(-99.0))
+ values = xr.open_zarr(path)["air_temperature"].values
+ assert np.isnan(values[0, 1])
+
+ def test_a_missing_value_sentinel_masks(self, tmp_path):
+ # Not a masked array: a ``missing_value`` attribute is a CF
+ # declaration about ordinary stored values, so the sentinel is
+ # written as real data, and it is xarray's own read that has to
+ # recognise and mask it.
+ path = tmp_path / "missing_value.zarr"
+ cube = small_cube()
+ cube.data[0, 1] = -99.0
+ cube.attributes["missing_value"] = np.float32(-99.0)
+ iris.save(cube, path)
+ values = xr.open_zarr(path)["air_temperature"].values
+ assert np.isnan(values[0, 1])
+ assert not np.isnan(values[0, 0])
+
+ def test_the_dimensions_and_coordinates_are_xarrays_too(self, tmp_path):
+ path = tmp_path / "dims.zarr"
+ iris.save(small_cube(), path)
+ dataset = xr.open_zarr(path)
+ assert dataset["air_temperature"].dims == ("latitude", "longitude")
+ np.testing.assert_allclose(dataset["latitude"].values, [0.0, 1.0])
+
+ def test_attributes_survive_as_values(self, tmp_path):
+ # A cube attribute not reserved for a data variable is written as a
+ # Zarr group attribute, the same split the netCDF saver makes by
+ # default, so this reads the dataset's own attrs rather than the
+ # data variable's.
+ path = tmp_path / "attrs.zarr"
+ cube = small_cube()
+ cube.attributes["a_number"] = np.float32(1.5)
+ cube.attributes["a_list"] = np.array([1, 2], dtype=np.int16)
+ cube.attributes["text"] = "words"
+ iris.save(cube, path)
+ attributes = xr.open_zarr(path).attrs
+ assert attributes["a_number"] == 1.5
+ assert list(attributes["a_list"]) == [1, 2]
+ assert attributes["text"] == "words"
+
+ def test_a_consolidated_store_opens_consolidated(self, tmp_path):
+ path = tmp_path / "consolidated.zarr"
+ iris.save(small_cube(), path)
+ # The default: xarray uses the consolidated document when it is there.
+ assert "air_temperature" in xr.open_zarr(path).data_vars
+
+
+class TestIrisReadsXarray:
+ @pytest.mark.parametrize(
+ ("fill", "dtype"),
+ [(np.nan, "f4"), (-99.0, "f4"), (-9, "i2")],
+ ids=["float32-nan", "float32-sentinel", "int16-sentinel"],
+ )
+ def test_a_store_xarray_wrote_loads_masked(self, tmp_path, fill, dtype):
+ path = tmp_path / f"xarray_{dtype}_{fill}.zarr"
+ values = np.array([[1, 2, 3], [4, 5, 6]], dtype=dtype)
+ values[0, 1] = fill if fill == fill else values[0, 1]
+ dataset = xr.Dataset(
+ {
+ "air_temperature": (
+ ("latitude", "longitude"),
+ np.ma.masked_equal(values, fill)
+ if fill == fill
+ else np.where(np.arange(6).reshape(2, 3) == 1, np.nan, values),
+ )
+ },
+ coords={"latitude": [0.0, 1.0], "longitude": [0.0, 1.0, 2.0]},
+ )
+ dataset["air_temperature"].encoding["_FillValue"] = fill
+ # Consolidation is not what this test is about, and xarray's default
+ # warns that it is not yet part of the Zarr version 3 specification.
+ dataset.to_zarr(path, zarr_format=3, consolidated=False)
+ cube = iris.load_cube(path)
+ assert cube.data.mask[0, 1]
diff --git a/lib/iris/tests/integration/zarr/test_save.py b/lib/iris/tests/integration/zarr/test_save.py
new file mode 100644
index 0000000000..8e5238cc11
--- /dev/null
+++ b/lib/iris/tests/integration/zarr/test_save.py
@@ -0,0 +1,586 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Integration tests for saving cubes to a Zarr store."""
+
+import warnings
+
+import dask.array as da
+import dask.config
+import distributed
+import numpy as np
+import pytest
+
+zarr = pytest.importorskip("zarr")
+
+import iris
+from iris.cube import CubeList
+from iris.fileformats.cf.saver import _CFSaver
+import iris.fileformats.zarr
+import iris.tests.stock as stock
+import iris.warnings
+
+
+@pytest.fixture
+def cube():
+ # The signature is `realistic_4d_w_everything(w_mesh=False)`: there is no
+ # `lazy` keyword, and the cube it returns holds real data.
+ #
+ # Its hybrid-height factory depends on an orography coordinate, which the
+ # CF data model has no way to fold back into the main cube: it round
+ # trips as its own top-level cube, standard name "surface_altitude", the
+ # same as a plain netCDF round trip of this fixture does. So a store this
+ # fixture is saved into always holds two cubes, and a single-cube load
+ # names the one it wants.
+ return stock.realistic_4d_w_everything()
+
+
+class TestRoundTrip:
+ def test_the_cube_comes_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ loaded = iris.load_cube(path, cube.name())
+ assert loaded.standard_name == cube.standard_name
+ assert loaded.units == cube.units
+ assert loaded.shape == cube.shape
+ np.testing.assert_allclose(loaded.data, cube.data)
+
+ def test_the_coordinates_come_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ loaded = iris.load_cube(path, cube.name())
+ saved_names = sorted(coord.name() for coord in cube.coords())
+ loaded_names = sorted(coord.name() for coord in loaded.coords())
+ assert loaded_names == saved_names
+
+ def test_a_derived_coordinate_is_still_derived(self, tmp_path, cube):
+ # `test_the_coordinates_come_back` cannot show this on its own:
+ # `Cube.coords()` lists derived coordinates alongside the rest, so a
+ # load that rebuilt "altitude" as an ordinary AuxCoord would give the
+ # same list of names and pass. The factory is the thing to assert.
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ loaded = iris.load_cube(path, cube.name())
+ assert [factory.name() for factory in loaded.aux_factories] == [
+ factory.name() for factory in cube.aux_factories
+ ]
+ assert [coord.name() for coord in loaded.derived_coords] == ["altitude"]
+ np.testing.assert_allclose(
+ loaded.coord("altitude").points, cube.coord("altitude").points
+ )
+
+ def test_the_bounds_come_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ loaded = iris.load_cube(path, cube.name())
+ for coord in cube.coords():
+ if coord.has_bounds():
+ np.testing.assert_allclose(
+ loaded.coord(coord.name()).bounds, coord.bounds
+ )
+
+ def test_the_cell_methods_come_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ assert iris.load_cube(path, cube.name()).cell_methods == cube.cell_methods
+
+ def test_the_coordinate_system_comes_back(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ loaded = iris.load_cube(path, cube.name())
+ assert loaded.coord_system() == cube.coord_system()
+
+ def test_attribute_values_come_back(self, tmp_path, cube):
+ # Values, not dtypes: spec §4.5 trades the NumPy dtype of an
+ # attribute for output every Zarr consumer can read.
+ path = tmp_path / "out.zarr"
+ cube.attributes["a_number"] = np.float32(1.5)
+ cube.attributes["a_list"] = np.array([1, 2], dtype=np.int16)
+ iris.save(cube, path)
+ loaded = iris.load_cube(path, cube.name())
+ assert loaded.attributes["a_number"] == 1.5
+ assert list(loaded.attributes["a_list"]) == [1, 2]
+
+ def test_a_masked_cube_keeps_its_mask(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ masked = cube.copy(np.ma.masked_greater(cube.data, cube.data.mean()))
+ iris.save(masked, path)
+ loaded = iris.load_cube(path, masked.name())
+ np.testing.assert_array_equal(loaded.data.mask, masked.data.mask)
+
+ def test_several_cubes_round_trip(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ other = cube[0].copy()
+ other.rename("air_pressure")
+ other.var_name = "air_pressure"
+ iris.save(CubeList([cube, other]), path)
+ loaded = iris.load(path)
+ # Three, not two: `cube`'s hybrid-height factory always brings its
+ # orography cube along as well (see the `cube` fixture above), so
+ # both saved names are checked explicitly rather than trusting a
+ # bare count to mean the right two cubes came back.
+ assert len(loaded) == 3
+ assert {one.name() for one in loaded} >= {cube.name(), other.name()}
+
+ def test_a_group_round_trips(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ # A different name in the group than at the root: reading the
+ # group and finding this name only works if `group=` was honoured,
+ # where reading the root (which does not have this name) would not.
+ renamed = cube.copy()
+ renamed.rename("a_renamed_cube")
+ renamed.var_name = "a_renamed_cube"
+ iris.save(renamed, path, group="again")
+ # `iris.load`/`iris.load_cube` read the root group only - see
+ # "Groups" in `docs/src/user_manual/explanation/zarr_io.rst` - so a
+ # named group is read through the Zarr loader directly.
+ loaded = CubeList(
+ iris.fileformats.zarr.load_cubes(path, group="again")
+ ).extract_cube(renamed.name())
+ assert loaded.shape == renamed.shape
+
+
+class TestAddingToAnExistingStore:
+ def test_two_cubes_with_the_same_anonymous_dimension_both_come_back_whole(
+ self, tmp_path
+ ):
+ # A cube dimension with no dim coord is called "dim0", so two
+ # unrelated cubes routinely claim that name. Sizing the second
+ # cube's array from what the store already calls "dim0" silently
+ # truncated it: spec §4.5's create-once rule means the dimension the
+ # store holds is never resized, so the second cube needs its own.
+ path = tmp_path / "out.zarr"
+ short = iris.cube.Cube(
+ np.arange(2.0), long_name="short", units="1", var_name="short"
+ )
+ tall = iris.cube.Cube(
+ np.arange(3.0), long_name="tall", units="1", var_name="tall"
+ )
+ iris.save(short, path)
+ iris.save(tall, path, mode="a")
+ loaded = iris.load(path)
+ np.testing.assert_array_equal(loaded.extract_cube("short").data, [0.0, 1.0])
+ np.testing.assert_array_equal(loaded.extract_cube("tall").data, [0.0, 1.0, 2.0])
+
+ def test_the_shorter_cube_second_is_whole_too(self, tmp_path):
+ # The other order used to raise a bare numpy broadcasting message
+ # naming neither cube nor array.
+ path = tmp_path / "out.zarr"
+ tall = iris.cube.Cube(
+ np.arange(4.0), long_name="tall", units="1", var_name="tall"
+ )
+ short = iris.cube.Cube(
+ np.arange(2.0), long_name="short", units="1", var_name="short"
+ )
+ iris.save(tall, path)
+ iris.save(short, path, mode="a")
+ loaded = iris.load(path)
+ np.testing.assert_array_equal(
+ loaded.extract_cube("tall").data, [0.0, 1.0, 2.0, 3.0]
+ )
+ np.testing.assert_array_equal(loaded.extract_cube("short").data, [0.0, 1.0])
+
+
+class TestGlobalAttributes:
+ # `_CFSaver.write` deliberately leaves the global attributes to its
+ # caller under split-attribute saving, and computes `local_keys` for the
+ # legacy mode. Both are format-agnostic cube logic, shared with the
+ # netCDF saver - see §4.5 of the native Zarr I/O design.
+ def test_globals_are_written_when_split_attributes_are_enabled(self, tmp_path):
+ path = tmp_path / "out.zarr"
+ cube = iris.cube.Cube(np.arange(3.0), long_name="a", units="1", var_name="a")
+ cube.attributes.globals["title"] = "the title"
+ cube.attributes.globals["institution"] = "Met Office"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(cube, path)
+ loaded = iris.load_cube(path)
+ assert loaded.attributes.globals["title"] == "the title"
+ assert loaded.attributes.globals["institution"] == "Met Office"
+
+ def test_globals_are_written_in_the_legacy_mode_too(self, tmp_path):
+ path = tmp_path / "out.zarr"
+ cube = iris.cube.Cube(np.arange(3.0), long_name="a", units="1", var_name="a")
+ cube.attributes["title"] = "the title"
+ with iris.FUTURE.context(save_split_attrs=False):
+ iris.save(cube, path)
+ assert iris.load_cube(path).attributes["title"] == "the title"
+
+ def test_cubes_that_disagree_keep_their_own_values(self, tmp_path):
+ # One store cannot hold two values of one global attribute, so an
+ # attribute the cubes disagree about becomes a local one on each
+ # data variable, exactly as a netCDF save of the same list does.
+ path = tmp_path / "out.zarr"
+ first = iris.cube.Cube(
+ np.arange(3.0), long_name="first", units="1", var_name="first"
+ )
+ first.attributes["title"] = "first"
+ second = iris.cube.Cube(
+ np.arange(3.0), long_name="second", units="1", var_name="second"
+ )
+ second.attributes["title"] = "second"
+ with iris.FUTURE.context(save_split_attrs=False):
+ iris.save(CubeList([first, second]), path)
+ loaded = iris.load(path)
+ assert loaded.extract_cube("first").attributes["title"] == "first"
+ assert loaded.extract_cube("second").attributes["title"] == "second"
+
+ def test_disagreeing_globals_are_demoted_under_split_attributes(self, tmp_path):
+ path = tmp_path / "out.zarr"
+ first = iris.cube.Cube(
+ np.arange(3.0), long_name="first", units="1", var_name="first"
+ )
+ first.attributes.globals["title"] = "first"
+ second = iris.cube.Cube(
+ np.arange(3.0), long_name="second", units="1", var_name="second"
+ )
+ second.attributes.globals["title"] = "second"
+ with iris.FUTURE.context(save_split_attrs=True):
+ with pytest.warns(iris.warnings.IrisSaveWarning, match="title"):
+ iris.save(CubeList([first, second]), path)
+ loaded = iris.load(path)
+ assert loaded.extract_cube("first").attributes.locals["title"] == "first"
+ assert loaded.extract_cube("second").attributes.locals["title"] == "second"
+
+ def test_the_input_cubes_are_not_modified(self, tmp_path):
+ # The demotion above works on copies: `iris.save` must not rewrite
+ # the caller's own cubes.
+ path = tmp_path / "out.zarr"
+ first = iris.cube.Cube(
+ np.arange(3.0), long_name="first", units="1", var_name="first"
+ )
+ first.attributes.globals["title"] = "first"
+ second = iris.cube.Cube(
+ np.arange(3.0), long_name="second", units="1", var_name="second"
+ )
+ second.attributes.globals["title"] = "second"
+ with iris.FUTURE.context(save_split_attrs=True):
+ with pytest.warns(iris.warnings.IrisSaveWarning):
+ iris.save(CubeList([first, second]), path)
+ assert first.attributes.globals["title"] == "first"
+ assert second.attributes.globals["title"] == "second"
+
+
+class TestWhatTheFormatDoesNotPromise:
+ # Spec §4.5: neither of these is fixable from inside Iris. They are
+ # pinned so that a zarr-python release changing one fails here.
+ def test_a_missing_chunk_reads_back_filled(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ # The data variable's name is the cube's `var_name`, which this
+ # fixture sets to "air_temp" - not its standard name.
+ chunks = sorted((path / cube.var_name / "c").rglob("*"))
+ removed = [p for p in chunks if p.is_file()][0]
+ removed.unlink()
+ original_masked = np.ma.count_masked(cube.data)
+ # Opens cleanly, and under-reports data, with no warning of any kind:
+ # both halves of that are asserted below, not only the shape, which
+ # is unchanged whether the gap is filled in silence or not.
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ reloaded = iris.load_cube(path, cube.name())
+ data = reloaded.data
+ assert caught == []
+ assert data.shape == cube.shape
+ # The deleted chunk reads back as the array's own fill, which this
+ # masked cube's decode then masks - strictly more points than the
+ # cube ever declared missing, and nothing said so.
+ assert np.ma.count_masked(data) > original_masked
+
+ def test_consolidated_metadata_goes_stale(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ # Add a group behind Iris's back, without re-consolidating.
+ zarr.open_group(store=path, mode="a", path="sneaked").create_array(
+ "data", shape=(2,), dtype="f4", dimension_names=("x",)
+ )
+ reader = zarr.open_group(store=path, mode="r")
+ assert "sneaked" not in list(reader.group_keys())
+ # Which is why Iris re-consolidates when it adds a group itself.
+ iris.save(cube, path, group="properly")
+ reader = zarr.open_group(store=path, mode="r")
+ assert "properly" in list(reader.group_keys())
+
+
+class TestRefusals:
+ def test_an_existing_store_is_not_clobbered(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ with pytest.raises(FileExistsError, match="mode"):
+ iris.save(cube, path)
+ # And the first save is still intact.
+ assert iris.load_cube(path, cube.name()).shape == cube.shape
+
+ def test_adding_a_name_that_exists_is_refused(self, tmp_path, cube):
+ path = tmp_path / "out.zarr"
+ iris.save(cube, path)
+ with pytest.raises(ValueError, match=cube.var_name):
+ iris.save(cube, path, mode="a")
+
+
+class TestInputsTheDesignImplies:
+ def test_a_big_endian_cube_round_trips(self, tmp_path):
+ # A PP or GRIB load yields big-endian arrays, so this is an ordinary
+ # cube to save rather than a corner. Zarr version 3 has no big-endian
+ # data type, so the store is written in the platform's own order and
+ # the values are the cube's - the same answer the netCDF saver has
+ # always given this data.
+ from iris.cube import Cube
+
+ cube = Cube(
+ np.array([1.5, -2.5, 3.5], dtype=">f4"),
+ long_name="air_temperature",
+ var_name="air",
+ units="K",
+ )
+ path = tmp_path / "big_endian.zarr"
+ iris.save(cube, path)
+
+ loaded = iris.load_cube(path)
+ assert loaded.dtype == np.dtype("f4")
+ np.testing.assert_array_equal(loaded.data, cube.data)
+
+ def test_a_mesh_cube_writes_cf_ugrid_structure(self, tmp_path):
+ # Iris supports unstructured meshes, and a mesh cube reaching the
+ # Zarr saver is ordinary: the mesh variables go through the same
+ # generic path as everything else, and nothing above has proved it.
+ # This is the saver's own promise, independent of what a loader can
+ # rebuild from it - see the round trip below for that.
+ path = tmp_path / "mesh.zarr"
+ mesh_cube = stock.realistic_4d_w_everything(w_mesh=True)
+ iris.save(mesh_cube, path)
+ store = zarr.open_group(store=path, mode="r")
+
+ data_var = store[mesh_cube.var_name]
+ assert data_var.attrs["location"] == mesh_cube.location
+
+ topology = store[data_var.attrs["mesh"]]
+ assert topology.attrs["cf_role"] == "mesh_topology"
+ assert topology.attrs["topology_dimension"] == mesh_cube.mesh.topology_dimension
+
+ connectivity = store[topology.attrs["face_node_connectivity"]]
+ assert connectivity.attrs["cf_role"] == "face_node_connectivity"
+ assert "start_index" in connectivity.attrs
+
+ # The mesh's own dimensions carry no coordinate variable of their
+ # own name - that is how UGRID keeps a mesh dimension distinct from
+ # an ordinary CF one.
+ assert mesh_cube.mesh.face_dimension not in store.array_keys()
+ assert mesh_cube.mesh.node_dimension not in store.array_keys()
+
+ def test_a_mesh_cube_round_trips(self, tmp_path):
+ path = tmp_path / "mesh.zarr"
+ mesh_cube = stock.realistic_4d_w_everything(w_mesh=True)
+ iris.save(mesh_cube, path)
+ loaded = iris.load_cube(path, mesh_cube.name())
+ assert loaded.mesh is not None
+ assert loaded.location == mesh_cube.location
+ assert loaded.mesh.topology_dimension == mesh_cube.mesh.topology_dimension
+
+ def test_packed_data_round_trips(self, tmp_path, cube):
+ # `packing` writes scale_factor and add_offset and narrows the dtype.
+ # Both attributes are numbers the JSON conversion touches, and the
+ # read side unpacks with them.
+ path = tmp_path / "packed.zarr"
+ iris.save(cube, path, packing="i2")
+ # The data variable's name is the cube's `var_name`, which this
+ # fixture sets to "air_temp" - not its standard name.
+ stored = zarr.open_group(store=path, mode="r")[cube.var_name]
+ assert stored.dtype == np.dtype("i2")
+ assert "scale_factor" in dict(stored.attrs)
+ assert "add_offset" in dict(stored.attrs)
+ loaded = iris.load_cube(path, cube.name())
+ # A tolerance of one stored count: loose enough for the quantisation
+ # error correct packing is allowed to introduce (up to half a count),
+ # tight enough that a systematic one-count bias - truncation instead
+ # of rounding, say - would fail it, where the original `rtol=1e-3`
+ # would not have.
+ scale_factor = stored.attrs["scale_factor"]
+ np.testing.assert_allclose(loaded.data, cube.data, atol=scale_factor, rtol=0)
+
+ def test_int64_data_survives(self, tmp_path, cube):
+ # The one dtype the saver has narrowing logic for. A Zarr store holds
+ # int64, so `supports_dtype` answers True and nothing is narrowed -
+ # which is a different answer from NETCDF3, and worth pinning.
+ path = tmp_path / "int64.zarr"
+ wide = cube.copy(cube.data.astype(np.int64))
+ iris.save(wide, path)
+ assert iris.load_cube(path, wide.name()).dtype == np.dtype("i8")
+
+ def test_a_dtype_netcdf_has_no_fill_for_still_saves(self, tmp_path):
+ # `ZarrDataset.supports_dtype` answers True for every NumPy dtype, so
+ # nothing is narrowed on the way in. A dtype outside netCDF's
+ # default-fill table must therefore reach the store intact: the fill
+ # table is about missing values, and this data has none.
+ path = tmp_path / "float16.zarr"
+ narrow = iris.cube.Cube(
+ np.arange(4, dtype=np.float16),
+ long_name="narrow",
+ units="1",
+ var_name="narrow",
+ )
+ iris.save(narrow, path)
+ loaded = iris.load_cube(path, narrow.name())
+ assert loaded.dtype == np.dtype("f2")
+ np.testing.assert_allclose(loaded.data, narrow.data)
+
+ def test_a_list_of_numpy_scalars_is_an_attribute(self, tmp_path, cube):
+ # `[data.min(), data.max()]` is an ordinary way to build an
+ # attribute, and netCDF accepts the list it produces.
+ path = tmp_path / "sequence.zarr"
+ cube.attributes["probe"] = [np.float32(1.5), np.float32(2.5)]
+ iris.save(cube, path)
+ loaded = iris.load_cube(path, cube.name())
+ np.testing.assert_allclose(loaded.attributes["probe"], [1.5, 2.5])
+
+ def test_masked_data_with_no_default_fill_says_what_to_pass(self, tmp_path, cube):
+ # The other side of the same coin: with no netCDF default for the
+ # dtype there is no sentinel to declare, and the array's own fill
+ # value says only what unwritten storage reads back as - so a zero
+ # there would mask every valid zero. The refusal names the keyword.
+ path = tmp_path / "masked16.zarr"
+ narrow = cube.copy(np.ma.masked_greater(cube.data.astype(np.float16), 0.0))
+ with pytest.raises(ValueError, match="fill_value"):
+ iris.save(narrow, path)
+
+ def test_an_explicit_fill_value_saves_that_masked_data(self, tmp_path, cube):
+ path = tmp_path / "masked16.zarr"
+ narrow = cube.copy(np.ma.masked_greater(cube.data.astype(np.float16), 0.0))
+ iris.save(narrow, path, fill_value=np.float16(-99.0))
+ loaded = iris.load_cube(path, narrow.name())
+ np.testing.assert_array_equal(loaded.data.mask, narrow.data.mask)
+
+ def test_a_non_ascii_attribute_survives(self, tmp_path, cube):
+ # A degree sign in a title is ordinary, and netCDF's ASCII-or-bytes
+ # coercion does not apply here: JSON is text.
+ path = tmp_path / "unicode.zarr"
+ cube.attributes["title"] = "Temperature in °C, naïve grid"
+ iris.save(cube, path)
+ assert iris.load_cube(path, cube.name()).attributes["title"] == (
+ "Temperature in °C, naïve grid"
+ )
+
+ def test_byte_string_data_saves(self, tmp_path):
+ # Iris supports a cube of fixed-width bytes, and netCDF saves one as
+ # its own character type. A Zarr store holds it too, so nothing may
+ # refuse it on the way in.
+ path = tmp_path / "bytes.zarr"
+ letters = iris.cube.Cube(
+ np.array([b"a", b"b", b"c"], dtype="S1"),
+ long_name="letters",
+ var_name="letters",
+ )
+ iris.save(letters, path)
+ stored = zarr.open_group(store=path, mode="r")["letters"]
+ np.testing.assert_array_equal(stored[...], letters.data)
+
+
+class TestDataless:
+ # `Cube` supports a dataless form, and `_CFSaver` has a branch for it that
+ # chooses its own dtype and fill value and writes a marker attribute.
+ # Nothing else here reaches that branch. The marker is a string the Zarr
+ # attribute conversion has to carry intact, which is the half of this the
+ # netCDF saver does not share - compare
+ # `iris.tests.integration.netcdf.test_dataless`.
+ @pytest.fixture
+ def dataless(self):
+ return iris.cube.Cube(
+ None,
+ shape=(3, 4),
+ long_name="testdata",
+ var_name="testdata",
+ )
+
+ def test_the_store_holds_the_dataless_form(self, tmp_path, dataless):
+ path = tmp_path / "dataless.zarr"
+ iris.save(dataless, path)
+ # The marker belongs in the store, not on the caller's cube.
+ assert _CFSaver._DATALESS_ATTRNAME not in dataless.attributes
+ stored = zarr.open_group(store=path, mode="r")["testdata"]
+ assert stored.attrs[_CFSaver._DATALESS_ATTRNAME] == "true"
+ assert stored.dtype == _CFSaver._DATALESS_DTYPE
+ assert stored.fill_value == _CFSaver._DATALESS_FILLVALUE
+ # No chunk was ever written, so every point reads back as the fill.
+ np.testing.assert_array_equal(
+ stored[...], np.full(dataless.shape, _CFSaver._DATALESS_FILLVALUE)
+ )
+
+ def test_the_cube_comes_back_dataless(self, tmp_path, dataless):
+ path = tmp_path / "dataless.zarr"
+ iris.save(dataless, path)
+ # `load_raw`, because dataless merging is not supported yet - the
+ # same reason `test_dataless` uses it for netCDF.
+ (loaded,) = iris.load_raw(path)
+ assert loaded.is_dataless()
+ assert loaded.shape == dataless.shape
+ # The marker is an implementation detail of the file, and the load
+ # consumes it rather than handing it back as a cube attribute.
+ assert _CFSaver._DATALESS_ATTRNAME not in loaded.attributes
+
+ def test_a_dataless_cube_mixes_with_a_normal_one(self, tmp_path, dataless):
+ path = tmp_path / "mixture.zarr"
+ dataful = iris.cube.Cube(
+ np.ones(dataless.shape),
+ long_name="other",
+ units="1",
+ var_name="other",
+ )
+ iris.save(CubeList([dataless, dataful]), path)
+ loaded = iris.load_raw(path)
+ assert loaded.extract_cube("testdata").is_dataless()
+ assert not loaded.extract_cube("other").is_dataless()
+
+ def test_a_dataless_cube_takes_almost_no_space(self, tmp_path):
+ # A dataless array has no chunks to write, so the store is metadata
+ # and nothing else, however large the cube says it is.
+ path = tmp_path / "big.zarr"
+ shape = (10000, 10000)
+ iris.save(iris.cube.Cube(None, shape=shape, var_name="big"), path)
+ on_disk = sum(p.stat().st_size for p in path.rglob("*") if p.is_file())
+ # The dtype is one byte per point, so the array would be ~100 MB.
+ assert on_disk < 0.1 * np.prod(shape)
+
+
+class TestEveryScheduler:
+ # `Saver.write_handle` puts a `ZarrDatasetVariable` into the Dask graph
+ # rather than the `zarr.Array` itself, so whatever runs the graph has to
+ # revive one and write through it. `TestDeferredSaving` in the `Saver`
+ # unit tests pickles that graph and computes it in the same process; only
+ # a distributed client proves a worker in another process can finish the
+ # save. Modelled on `test_scheduler_types` in
+ # `iris.tests.integration.netcdf.test_delayed_save`.
+ @pytest.fixture(params=["threads", "single-threaded", "distributed"], scope="class")
+ def scheduler(self, request):
+ if request.param == "distributed":
+ client = distributed.Client()
+ with dask.config.set(scheduler="distributed"):
+ yield request.param
+ client.close()
+ else:
+ with dask.config.set(scheduler=request.param):
+ yield request.param
+
+ @pytest.fixture(params=[False, True], ids=["immediate", "delayed"])
+ def save_is_delayed(self, request):
+ return request.param
+
+ def test_the_data_lands_whatever_runs_the_graph(
+ self, tmp_path, scheduler, save_is_delayed
+ ):
+ assert dask.config.get("scheduler") == scheduler
+ path = tmp_path / "out.zarr"
+ expected = np.arange(24.0).reshape(4, 6)
+ cube = iris.cube.Cube(
+ da.from_array(expected, chunks=(2, 3)),
+ standard_name="air_temperature",
+ units="K",
+ var_name="air_temperature",
+ )
+ result = iris.save(cube, path, compute=not save_is_delayed)
+ assert save_is_delayed == (result is not None)
+ if save_is_delayed:
+ # Nothing is written until this runs - see
+ # `TestDeferredSaving.test_consolidation_follows_the_last_chunk`.
+ result.compute()
+ np.testing.assert_array_equal(iris.load_cube(path).data, expected)
diff --git a/lib/iris/tests/unit/fileformats/cf/attribute_handlers/__init__.py b/lib/iris/tests/unit/fileformats/cf/attribute_handlers/__init__.py
new file mode 100644
index 0000000000..78f4908fcd
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/attribute_handlers/__init__.py
@@ -0,0 +1,5 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the :mod:`iris.fileformats.cf._attribute_handlers` module."""
diff --git a/lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/test_GribParamHandler.py b/lib/iris/tests/unit/fileformats/cf/attribute_handlers/test_GribParamHandler.py
similarity index 96%
rename from lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/test_GribParamHandler.py
rename to lib/iris/tests/unit/fileformats/cf/attribute_handlers/test_GribParamHandler.py
index 9e9ad017e2..b80a761e63 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/test_GribParamHandler.py
+++ b/lib/iris/tests/unit/fileformats/cf/attribute_handlers/test_GribParamHandler.py
@@ -2,7 +2,7 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Unit tests for :class:`iris.fileformats.netcdf._attribute_handlers.StashHandler`."""
+"""Unit tests for :class:`iris.fileformats.cf._attribute_handlers.GribParamHandler`."""
import pytest
@@ -13,7 +13,7 @@
)
import numpy as np
-from iris.fileformats.netcdf._attribute_handlers import ATTRIBUTE_HANDLERS
+from iris.fileformats.cf._attribute_handlers import ATTRIBUTE_HANDLERS
GP_HANDLER = ATTRIBUTE_HANDLERS["GRIB_PARAM"]
diff --git a/lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/test_StashHandler.py b/lib/iris/tests/unit/fileformats/cf/attribute_handlers/test_StashHandler.py
similarity index 94%
rename from lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/test_StashHandler.py
rename to lib/iris/tests/unit/fileformats/cf/attribute_handlers/test_StashHandler.py
index 84cddf2ff9..913c14de03 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/test_StashHandler.py
+++ b/lib/iris/tests/unit/fileformats/cf/attribute_handlers/test_StashHandler.py
@@ -2,11 +2,11 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Unit tests for :class:`iris.fileformats.netcdf._attribute_handlers.StashHandler`."""
+"""Unit tests for :class:`iris.fileformats.cf._attribute_handlers.StashHandler`."""
import pytest
-from iris.fileformats.netcdf._attribute_handlers import ATTRIBUTE_HANDLERS
+from iris.fileformats.cf._attribute_handlers import ATTRIBUTE_HANDLERS
from iris.fileformats.pp import STASH
STASH_HANDLER = ATTRIBUTE_HANDLERS["STASH"]
diff --git a/lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/test_UkmoProcessFlagsHandler.py b/lib/iris/tests/unit/fileformats/cf/attribute_handlers/test_UkmoProcessFlagsHandler.py
similarity index 96%
rename from lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/test_UkmoProcessFlagsHandler.py
rename to lib/iris/tests/unit/fileformats/cf/attribute_handlers/test_UkmoProcessFlagsHandler.py
index 585d3d966f..50881c1ce8 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/test_UkmoProcessFlagsHandler.py
+++ b/lib/iris/tests/unit/fileformats/cf/attribute_handlers/test_UkmoProcessFlagsHandler.py
@@ -3,13 +3,13 @@
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for
-:class:`iris.fileformats.netcdf._attribute_handlers.UkmoProcessFlagsHandler`.
+:class:`iris.fileformats.cf._attribute_handlers.UkmoProcessFlagsHandler`.
"""
import numpy as np
import pytest
-from iris.fileformats.netcdf._attribute_handlers import ATTRIBUTE_HANDLERS
+from iris.fileformats.cf._attribute_handlers import ATTRIBUTE_HANDLERS
UPF_HANDLER = ATTRIBUTE_HANDLERS["ukmo__process_flags"]
diff --git a/lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/__init__.py b/lib/iris/tests/unit/fileformats/cf/saver/__init__.py
similarity index 68%
rename from lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/__init__.py
rename to lib/iris/tests/unit/fileformats/cf/saver/__init__.py
index 0a94c147eb..08ec6013d3 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/attribute_handlers/__init__.py
+++ b/lib/iris/tests/unit/fileformats/cf/saver/__init__.py
@@ -2,4 +2,4 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Unit tests for the :mod:`iris.fileformats._attribute_handlers` module."""
+"""Unit tests for :mod:`iris.fileformats.cf.saver`."""
diff --git a/lib/iris/tests/unit/fileformats/cf/saver/test_grid_mapping_writer.py b/lib/iris/tests/unit/fileformats/cf/saver/test_grid_mapping_writer.py
new file mode 100644
index 0000000000..4ef29392f2
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/saver/test_grid_mapping_writer.py
@@ -0,0 +1,108 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :meth:`iris.fileformats.cf.saver._CFSaver._grid_mapping_writer`.
+
+The hook exists because netCDF4's own object is what has always received the
+CF grid-mapping parameters. Changing where they land would change what is in a
+netCDF file, which Q8 of the design document rules out of this programme, so
+netCDF keeps writing onto that object and every other store writes into the
+``attributes`` mapping.
+
+The base writer is exercised through the Zarr saver, the only other store
+there is, so that one test skips where Zarr is not installed. The skip is per
+test rather than per module: §6 of the design says an Iris without Zarr
+acquires no new test failures, and it would be no better if it quietly
+stopped running the netCDF half of this module instead.
+"""
+
+import types
+
+import numpy as np
+import pytest
+
+import iris
+from iris.coord_systems import GeogCS
+from iris.cube import Cube
+from iris.fileformats.netcdf._thread_safe_nc import DatasetWrapper
+
+
+@pytest.fixture
+def cube():
+ cube = Cube(np.arange(6, dtype=np.float32).reshape(2, 3), var_name="air")
+ cube.add_dim_coord(
+ iris.coords.DimCoord(
+ np.array([0.0, 1.0]),
+ standard_name="latitude",
+ units="degrees",
+ coord_system=GeogCS(6371229.0),
+ ),
+ 0,
+ )
+ cube.add_dim_coord(
+ iris.coords.DimCoord(
+ np.array([0.0, 1.0, 2.0]),
+ standard_name="longitude",
+ units="degrees",
+ coord_system=GeogCS(6371229.0),
+ ),
+ 1,
+ )
+ return cube
+
+
+def fake_cf_var():
+ """Return a variable whose two write destinations can be told apart.
+
+ A real variable's ``attributes`` is a view onto the netCDF4 object, so
+ writing to either one shows up in both, and the tests below could not say
+ which route a parameter took.
+ """
+ return types.SimpleNamespace(attributes={}, variable=types.SimpleNamespace())
+
+
+class TestTheBaseWriter:
+ def test_it_writes_into_the_attributes_mapping(self, tmp_path):
+ pytest.importorskip("zarr", minversion="3.1.0")
+ from iris.fileformats.zarr.saver import Saver
+
+ cf_var = fake_cf_var()
+ with Saver(tmp_path / "out.zarr") as saver:
+ write_parameter = saver._grid_mapping_writer(cf_var)
+ write_parameter("earth_radius", 6371229.0)
+ write_parameter("horizontal_datum_name", "OSGB 1936")
+
+ assert cf_var.attributes == {
+ "earth_radius": 6371229.0,
+ "horizontal_datum_name": "OSGB 1936",
+ }
+
+
+class TestTheNetCDFOverride:
+ def test_it_writes_onto_the_netcdf4_variable(self, tmp_path):
+ # Not just "writes the same values": it must bypass `.attributes`,
+ # whose ASCII-to-bytes coercion would change how netCDF types each
+ # parameter in the file.
+ from iris.fileformats.netcdf.saver import Saver
+
+ cf_var = fake_cf_var()
+ with Saver(tmp_path / "out.nc", "NETCDF4") as saver:
+ write_parameter = saver._grid_mapping_writer(cf_var)
+ write_parameter("crs_wkt", "GEOGCRS[...]")
+
+ assert cf_var.variable.crs_wkt == "GEOGCRS[...]"
+ assert cf_var.attributes == {}
+
+
+class TestTheFileIsUnchanged:
+ def test_grid_mapping_attributes_land_where_they_did(self, tmp_path, cube):
+ path = tmp_path / "out.nc"
+ iris.save(cube, path)
+
+ dataset = DatasetWrapper(path)
+ grid = dataset.variables["latitude_longitude"]
+ assert grid.getncattr("grid_mapping_name") == "latitude_longitude"
+ assert grid.getncattr("earth_radius") == pytest.approx(6371229.0)
+ assert grid.getncattr("longitude_of_prime_meridian") == pytest.approx(0.0)
+ dataset.close()
diff --git a/lib/iris/tests/unit/fileformats/cf/test__imports.py b/lib/iris/tests/unit/fileformats/cf/test__imports.py
index 3284f07dfe..64dfbf9614 100644
--- a/lib/iris/tests/unit/fileformats/cf/test__imports.py
+++ b/lib/iris/tests/unit/fileformats/cf/test__imports.py
@@ -12,6 +12,12 @@
``iris.fileformats.__init__`` before any submodule body, and that imports the
netCDF package, which draws in the whole cascade.
+``_attribute_handlers`` imports :mod:`iris.fileformats.pp`, so the CF package
+now reaches the PP package while the ``iris.fileformats`` package is itself
+part-way through importing. That was already true of the netCDF package, which
+is where this module used to live and which ``iris.fileformats.__init__``
+imports before ``pp``.
+
The subprocess is what makes it a cold start. Inside the test process the
modules are already in ``sys.modules``, so an import there is a dictionary
lookup.
diff --git a/lib/iris/tests/unit/fileformats/zarr/__init__.py b/lib/iris/tests/unit/fileformats/zarr/__init__.py
index 2016f9f175..ce3b4fe5a2 100644
--- a/lib/iris/tests/unit/fileformats/zarr/__init__.py
+++ b/lib/iris/tests/unit/fileformats/zarr/__init__.py
@@ -11,4 +11,4 @@
import pytest
-pytest.importorskip("zarr", minversion="3.0.8")
+pytest.importorskip("zarr", minversion="3.1.0")
diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py
index a742e45e84..c6070fee21 100644
--- a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py
+++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py
@@ -147,24 +147,6 @@ def test_a_missing_group_raises(self, tmp_path):
ZarrDataset(location, group="nope")
-class Test_the_write_surface:
- def test_the_write_surface_refuses(self, tmp_path):
- # Saving to Zarr is not implemented. The members exist because the
- # interface declares them; refusing is the honest answer.
- dataset = ZarrDataset(stores.make_store(tmp_path / "s.zarr"))
- with pytest.raises(NotImplementedError, match="Saving"):
- dataset.create_dimension("t", 3)
- with pytest.raises(NotImplementedError, match="Saving"):
- dataset.create_variable("v", np.dtype("f4"), ("t",))
- with pytest.raises(NotImplementedError, match="Saving"):
- dataset.finalise()
-
- def test_a_write_mode_refuses(self, tmp_path):
- stores.make_store(tmp_path / "s.zarr")
- with pytest.raises(NotImplementedError, match="Saving"):
- ZarrDataset(str(tmp_path / "s.zarr"), mode="w")
-
-
class Test_supports_dtype:
@pytest.mark.parametrize(
"dtype", ["i1", "i2", "i4", "i8", "u1", "u2", "u4", "u8", "f4", "f8", "c8"]
diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py
new file mode 100644
index 0000000000..3f984387b4
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset__write.py
@@ -0,0 +1,821 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the write half of :class:`iris.fileformats.zarr._dataset.ZarrDataset`."""
+
+import numpy as np
+import pytest
+
+zarr = pytest.importorskip("zarr")
+
+from iris.fileformats.zarr._dataset import ZarrDataset
+
+
+@pytest.fixture
+def existing(tmp_path):
+ """Return the path of a store that already holds one array."""
+ path = tmp_path / "existing.zarr"
+ group = zarr.open_group(store=path, mode="w-", zarr_format=3)
+ array = group.create_array(
+ "already_here", shape=(2,), dtype="f4", dimension_names=("x",)
+ )
+ array[:] = np.array([1.0, 2.0], dtype="f4")
+ return path
+
+
+class TestOpening:
+ def test_a_new_store_is_created(self, tmp_path):
+ with ZarrDataset(tmp_path / "new.zarr", mode="w-") as dataset:
+ assert dataset.mode == "w-"
+ assert dataset.format == "ZARR_V3"
+ assert (tmp_path / "new.zarr" / "zarr.json").is_file()
+
+ def test_the_default_refuses_an_existing_store(self, existing):
+ # The default differs from the netCDF saver's deliberately: a Zarr
+ # clobber is many non-atomic deletes, so it is opted into.
+ with pytest.raises(FileExistsError, match="mode"):
+ ZarrDataset(existing, mode="w-")
+
+ def test_the_refusal_names_the_way_out(self, existing):
+ with pytest.raises(FileExistsError, match="'w'"):
+ ZarrDataset(existing, mode="w-")
+
+ def test_w_replaces_an_existing_store(self, existing):
+ with ZarrDataset(existing, mode="w") as dataset:
+ assert list(dataset.variables) == []
+
+ def test_a_adds_to_an_existing_store(self, existing):
+ with ZarrDataset(existing, mode="a") as dataset:
+ assert list(dataset.variables) == ["already_here"]
+
+ def test_a_opens_a_new_group_in_an_existing_store(self, existing):
+ with ZarrDataset(existing, mode="a", group="sub") as dataset:
+ assert dataset.location.endswith("existing.zarr/sub")
+ assert list(dataset.variables) == []
+ # The root is untouched.
+ with ZarrDataset(existing) as dataset:
+ assert list(dataset.variables) == ["already_here"]
+
+ def test_r_plus_needs_the_store_to_exist(self, tmp_path):
+ with pytest.raises(FileNotFoundError):
+ ZarrDataset(tmp_path / "missing.zarr", mode="r+")
+
+ def test_r_plus_adds_to_an_existing_store(self, existing):
+ # The one thing "r+" does that "a" does not is refuse to create the
+ # store, which the test above covers. Everything else about it is
+ # this: an array created through it lands alongside what was there.
+ with ZarrDataset(existing, mode="r+") as dataset:
+ dataset.create_dimension("y", 3)
+ variable = dataset.create_variable("added", "f4", ("y",))
+ variable[:] = np.array([1.0, 2.0, 3.0], dtype="f4")
+ with ZarrDataset(existing) as dataset:
+ assert sorted(dataset.variables) == ["added", "already_here"]
+ assert dataset.variables["added"][:].tolist() == [1.0, 2.0, 3.0]
+
+ def test_an_unknown_mode_names_itself(self, tmp_path):
+ with pytest.raises(ValueError, match="'a\\+'"):
+ ZarrDataset(tmp_path / "new.zarr", mode="a+")
+
+ def test_reading_is_still_the_default(self, existing):
+ with ZarrDataset(existing) as dataset:
+ assert dataset.mode == "r"
+
+
+@pytest.fixture
+def version_2(tmp_path):
+ """Return the path of a Zarr version 2 store holding one array."""
+ path = tmp_path / "v2.zarr"
+ group = zarr.open_group(store=path, mode="w-", zarr_format=2)
+ array = group.create_array("data", shape=(2,), dtype="f4")
+ array.attrs["_ARRAY_DIMENSIONS"] = ["x"]
+ return path
+
+
+class TestTheVersionGuard:
+ @pytest.mark.parametrize("mode", ["w-", "w", "a", "r+"])
+ def test_writing_version_2_is_refused(self, tmp_path, mode):
+ with pytest.raises(NotImplementedError, match="zarr_format=2"):
+ ZarrDataset(tmp_path / "new.zarr", mode=mode, zarr_format=2)
+
+ def test_the_refusal_names_the_versions_iris_writes(self, tmp_path):
+ with pytest.raises(NotImplementedError, match=r"\(3,\)"):
+ ZarrDataset(tmp_path / "new.zarr", mode="w-", zarr_format=2)
+
+ def test_reading_version_2_is_unaffected(self, version_2):
+ with ZarrDataset(version_2, mode="r") as dataset:
+ assert dataset.format == "ZARR_V2"
+ assert list(dataset.variables) == ["data"]
+
+ def test_reading_ignores_the_version_asked_for(self, version_2):
+ # zarr-python does not ignore `zarr_format` on a read, it constrains
+ # detection: asking for 3 here would report the store as a missing
+ # group. Iris drops the value instead, so the store still opens.
+ with ZarrDataset(version_2, mode="r", zarr_format=3) as dataset:
+ assert dataset.format == "ZARR_V2"
+
+ def test_a_version_3_store_is_what_a_write_produces(self, tmp_path):
+ with ZarrDataset(tmp_path / "new.zarr", mode="w-") as dataset:
+ assert dataset.storage_settings["zarr_format"] == 3
+
+
+@pytest.fixture
+def writable(tmp_path):
+ """Return an open, empty, writable dataset."""
+ with ZarrDataset(tmp_path / "out.zarr", mode="w-") as dataset:
+ yield dataset
+
+
+class TestCreateDimension:
+ def test_a_dimension_is_declared(self, writable):
+ writable.create_dimension("time", 3)
+ assert dict(writable.dimensions) == {"time": 3}
+
+ def test_declaring_it_twice_at_one_size_is_harmless(self, writable):
+ writable.create_dimension("time", 3)
+ writable.create_dimension("time", 3)
+ assert dict(writable.dimensions) == {"time": 3}
+
+ def test_a_conflicting_size_raises_naming_both(self, writable):
+ writable.create_dimension("time", 3)
+ with pytest.raises(ValueError, match="time.*3.*4|time.*4.*3"):
+ writable.create_dimension("time", 4)
+
+ def test_unlimited_is_refused(self, writable):
+ # Spec §4.2: a store with no such concept raises NotImplementedError.
+ with pytest.raises(NotImplementedError, match="unlimited"):
+ writable.create_dimension("time", None)
+
+ def test_a_read_only_dataset_refuses(self, existing):
+ with ZarrDataset(existing) as dataset:
+ with pytest.raises(ValueError, match="mode='r'"):
+ dataset.create_dimension("time", 3)
+
+ def test_an_existing_arrays_size_conflicts_too(self, existing):
+ # The conflict a declaration made in this session would hit, met by
+ # an array that was already in the store: `mode="a"` is the case,
+ # and §4.5's create-once rule means that array is never resized, so
+ # the two sizes cannot both be true of the name.
+ with ZarrDataset(existing, mode="a") as dataset:
+ assert dict(dataset.dimensions) == {"x": 2}
+ with pytest.raises(ValueError, match="'x'.*3.*2"):
+ dataset.create_dimension("x", 3)
+
+ def test_the_conflict_names_the_group(self, existing):
+ with ZarrDataset(existing, mode="a") as dataset:
+ with pytest.raises(ValueError, match="'x'") as excinfo:
+ dataset.create_dimension("x", 3)
+ assert dataset.location in str(excinfo.value)
+
+ def test_an_existing_arrays_size_agreeing_is_harmless(self, existing):
+ # The ordinary case: a second cube on the same dimension reuses the
+ # name, which is the whole point of recording it.
+ with ZarrDataset(existing, mode="a") as dataset:
+ dataset.create_dimension("x", 2)
+ variable = dataset.create_variable("alongside", np.dtype("f4"), ["x"])
+ assert variable.shape == (2,)
+
+
+class TestCreateVariable:
+ def test_an_array_is_created_with_its_dimension_names(self, writable):
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert variable.name == "data"
+ assert variable.dimensions == ("x",)
+ assert variable.shape == (3,)
+ assert variable.dtype == np.dtype("f4")
+
+ def test_it_appears_in_the_dataset(self, writable):
+ writable.create_dimension("x", 3)
+ writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert list(writable.variables) == ["data"]
+
+ def test_a_scalar_variable_has_no_dimensions(self, writable):
+ variable = writable.create_variable("grid", np.dtype("i4"))
+ assert variable.shape == ()
+
+ def test_an_undeclared_dimension_raises_naming_it(self, writable):
+ with pytest.raises(ValueError, match="nowhere"):
+ writable.create_variable("data", np.dtype("f4"), ["nowhere"])
+
+ def test_an_existing_name_raises_naming_the_group(self, writable):
+ # Spec §4.5: "not an overwrite and not a merge".
+ writable.create_dimension("x", 3)
+ writable.create_variable("data", np.dtype("f4"), ["x"])
+ with pytest.raises(ValueError, match="data") as excinfo:
+ writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert "data" in str(excinfo.value)
+ assert writable.location in str(excinfo.value)
+
+ def test_a_read_only_dataset_refuses(self, existing):
+ with ZarrDataset(existing) as dataset:
+ with pytest.raises(ValueError, match="mode='r'"):
+ dataset.create_variable("data", np.dtype("f4"))
+
+ def test_a_string_variable_with_no_fill_value_works(self, writable):
+ # dtype=str asks for a variable-length string array, which has no
+ # entry in the netCDF default-fill table: there is nothing to look
+ # up, so zarr-python's own default is what applies.
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("labels", str, ["x"])
+ variable[:] = np.array(["a", "bb", "ccc"], dtype=str)
+ assert list(variable[:]) == ["a", "bb", "ccc"]
+
+
+class TestCreateVariableFillValue:
+ def test_an_undeclared_fill_is_netcdfs_default(self, writable):
+ # Zarr's own default is 0, which would make a masked point of a
+ # float array read back as a real zero.
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert variable.fill_value == np.float32(9.9692099683868690e36)
+
+ def test_an_undeclared_fill_writes_no_attribute(self, writable):
+ # Nothing is missing yet. The saver declares _FillValue when it finds
+ # the source is masked.
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("data", np.dtype("f4"), ["x"])
+ assert "_FillValue" not in variable.attributes
+
+ def test_a_declared_fill_reaches_the_array_and_the_attribute(self, writable):
+ import base64
+ import struct
+
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable(
+ "data", np.dtype("f4"), ["x"], fill_value=np.float32(-99.0)
+ )
+ assert variable.fill_value == np.float32(-99.0)
+ packed = base64.b64decode(variable.attributes["_FillValue"])
+ assert struct.unpack("f8",
+ ">i8",
+]
+
+
+class TestSupportsDtype:
+ """`supports_dtype` says yes, and `create_variable` then creates.
+
+ Spec §4.2: the generic saver asks the store what it holds rather than
+ testing the format. For Zarr the answer is always True, so what needs
+ pinning is the other half - that each of those dtypes really does create.
+ """
+
+ def test_it_is_true_for_everything_a_saver_asks_about(self, writable):
+ # `_CFSaver._ensure_valid_dtype` only asks about int64 and the
+ # unsigned integers, but the answer is not qualified, so neither is
+ # this.
+ for spelling in _SAVER_DTYPES:
+ assert writable.supports_dtype(np.dtype(spelling)) is True
+
+ @pytest.mark.parametrize("spelling", _SAVER_DTYPES)
+ def test_create_variable_creates_it(self, writable, spelling):
+ dtype = np.dtype(spelling)
+ writable.create_dimension("x", 2)
+ variable = writable.create_variable("data", dtype, ["x"])
+ # Compared in the platform's own byte order, which is what the array
+ # is created in whatever it was asked for - see TestByteOrder.
+ assert variable.dtype == dtype.newbyteorder("=")
+
+ @pytest.mark.parametrize(
+ ("spelling", "message"),
+ [
+ ("O", "resolution from object failed"),
+ ([("x", "O")], "resolution from object failed"),
+ ("V0", "length must be >= 1"),
+ ("S0", "length must be >= 1"),
+ ],
+ ids=str,
+ )
+ def test_a_dtype_zarr_cannot_hold_creates_nothing(
+ self, writable, spelling, message
+ ):
+ # The message is zarr-python's, not Iris's: Iris does not refuse these
+ # itself, because doing so would mean keeping its own table of what
+ # zarr-python can store. The fragments matched here are the stable
+ # part, unchanged from 3.1.0 to 3.4.1. What Iris owns is the outcome -
+ # a refused create leaves the group as it was.
+ writable.create_dimension("x", 2)
+ with pytest.raises(ValueError, match=message):
+ writable.create_variable("data", np.dtype(spelling), ["x"])
+ assert list(writable.variables) == []
+
+ @pytest.mark.parametrize(
+ "spelling", ["datetime64[s]", "V8", "U0", [("a", "i4"), ("b", "f8")]], ids=str
+ )
+ def test_a_dtype_no_saver_reaches_is_left_to_zarr(self, writable, spelling):
+ """Zarr stores these, so Iris does not refuse them on Zarr's behalf.
+
+ None of them arrives from an Iris saver, which is why the answer does
+ not matter much. It is pinned because the alternative - a table of
+ what Zarr holds - got all four of these wrong, refusing them on the
+ oldest zarr-python Iris then accepted.
+ """
+ writable.create_dimension("x", 2)
+ writable.create_variable("data", np.dtype(spelling), ["x"])
+ assert list(writable.variables) == ["data"]
+
+
+class TestByteOrder:
+ """A big-endian dtype is stored in the platform's own order.
+
+ Zarr version 3 has no big-endian data type. Endianness lives in the
+ `bytes` codec, which zarr-python leaves at `little`, so asking for `>f4`
+ writes the same store as asking for `f4", ">f8", ">i8", ">i2", ">u4"])
+ def test_the_array_is_native(self, writable, spelling):
+ writable.create_dimension("x", 2)
+ variable = writable.create_variable("data", np.dtype(spelling), ["x"])
+ assert variable.dtype == np.dtype(spelling).newbyteorder("=")
+
+ def test_the_values_are_the_callers(self, writable):
+ # The point of the conversion: the bytes are laid out this store's
+ # way, and the numbers are the ones handed in.
+ source = np.array([1.5, -2.5], dtype=">f4")
+ writable.create_dimension("x", 2)
+ variable = writable.create_variable("data", source.dtype, ["x"])
+ variable[:] = source
+ np.testing.assert_array_equal(variable[:], source)
+
+ def test_a_native_dtype_is_untouched(self, writable):
+ writable.create_dimension("x", 2)
+ variable = writable.create_variable("data", np.dtype("=3.0.8
+ - zarr >=3.1.0
# Test dependencies.
- asv_runner
@@ -51,6 +51,7 @@ dependencies:
- pytest-mock
- pytest-xdist
- requests
+ - xarray
# Documentation dependencies.
- sphinx <8.2
diff --git a/requirements/py313.yml b/requirements/py313.yml
index dfb47e25b0..8af9a1d029 100644
--- a/requirements/py313.yml
+++ b/requirements/py313.yml
@@ -37,7 +37,7 @@ dependencies:
- python-stratify
- rasterio
- affine
- - zarr >=3.0.8
+ - zarr >=3.1.0
# Test dependencies.
- asv_runner
@@ -51,6 +51,7 @@ dependencies:
- pytest-mock
- pytest-xdist
- requests
+ - xarray
# Documentation dependencies.
- sphinx <8.2
diff --git a/requirements/py314.yml b/requirements/py314.yml
index 11187c6165..decd8f988d 100644
--- a/requirements/py314.yml
+++ b/requirements/py314.yml
@@ -37,7 +37,7 @@ dependencies:
- python-stratify
- rasterio
- affine
- - zarr >=3.0.8
+ - zarr >=3.1.0
# Test dependencies.
- asv_runner
@@ -51,6 +51,7 @@ dependencies:
- pytest-mock
- pytest-xdist
- requests
+ - xarray
# Documentation dependencies.
- sphinx <8.2