diff --git a/.github/WRITING.md b/.github/WRITING.md
new file mode 100644
index 0000000000..17b65a30d2
--- /dev/null
+++ b/.github/WRITING.md
@@ -0,0 +1,163 @@
+# Writing for a Reader in a Hurry
+
+Clear writing is familiar writing, and familiar writing is not jarring. A
+reader who is not stopped by the phrasing reaches the meaning, and a reader
+who understands you can decide whether to believe you. Unfamiliar prose fails
+before any of that. It gets re-read or skipped, and its content is never
+judged.
+
+One thing sits alongside. A reader who cannot tell what you verified from what
+you guessed has no way to check you, so they discount all of it rather than
+the part they caught.
+
+Most of what follows is prohibitions, because telling a writer to "be clear"
+changes nothing. Where a rule has an edge it does not cover, ask what a reader
+in a hurry would take away.
+
+Trust comes from being checkable, not from sounding careful. Say what you ran.
+Say what you did not. Neither needs more than a sentence.
+
+**Scope.** The sentence rules govern every piece of prose you write for a
+person to read: chat replies to the person you are working with, docstrings,
+comments, commit messages, pull request bodies, issues, review comments,
+specifications and plans. The GitHub rules govern only what is posted to
+GitHub.
+
+For **where** prose belongs in library code — comment, docstring, test or
+module docstring — see [`lib/iris/AGENTS.md`](../lib/iris/AGENTS.md). This
+file covers how the sentence is built, not where it lives.
+
+
+## Use Words That Are Already In Use
+
+Search for the repository's own term before coining one. A word invented for a
+single sentence reads as foreign even when it is accurate, and a reader who
+meets three of them stops trusting the register.
+
+- Grep first. If Iris calls it a *cell measure*, do not call it a *per-cell
+ weight*.
+- Do not introduce a term of art that appears exactly once in the repository.
+- Conventional verbs are fine: a loader *expects*, a test *pins*, a function
+ *assumes*. Novel metaphors are not. A method is not *half of a class*, and
+ reading a property is not a *write-path act*.
+
+
+## Sentences
+
+- **Put the conclusion in the first clause** of every sentence and every
+ paragraph. The reader stops partway through. What survives truncation should
+ still be the point.
+- **One sentence, one fact.** A qualifier that matters gets its own sentence.
+ A reader in a hurry takes the main clause and drops the rest, so anything in
+ a subordinate clause is not really there.
+- **Split, never compress.** When a sentence is too long, divide it or cut a
+ fact. Making it denser is the failure this file exists to prevent.
+- **An em dash rescuing a sentence means the sentence is overloaded.** Use a
+ full stop.
+- **Name what you did instead of hedging.** "This should work" tells the
+ reader nothing. "I ran it on both branches" tells them what to believe.
+ Write "I did not test this" when you did not.
+- **No rhetorical contrast unless the reader would assume the rejected side.**
+ "Returns a view rather than a copy" stays, because a reader would assume a
+ copy. "Passed through, not reached around" goes, because nobody proposed
+ reaching around.
+- **Do not answer an objection nobody made.** A sentence that would read
+ naturally as a reply in a review thread belongs in the review thread.
+- **Write the fact, not your side of the argument about it.** Three tells that
+ you have written a reply: a negation whose positive was never stated; a
+ "therefore" or "so" whose premise is absent; a term of art used once.
+
+
+## What Stays True
+
+Prose is not corrected by the thing it describes. A docstring gets fixed by
+the next change to the file. A commit message and a pull request body are
+never fixed at all.
+
+- **Past tense about a change is durable. Present tense about a state is
+ not.** "I changed the guard to a membership test" stays true. "The guard is
+ a membership test" is true until someone changes it. They cost the same.
+- **Never cite what decays silently**: a line number in another file, a count
+ of call sites, a time relative to writing ("currently", "this PR", "the new
+ behaviour"). Nothing checks any of them, and your "now" is never the
+ reader's.
+- **Never cite an artefact of how the work was done**: a review finding, a
+ task or plan number, a phase, a checklist item. These resolve to nothing for
+ a later reader. Cite an issue, a pull request, a design document, or a
+ section of the convention.
+- **Do not enumerate test names in a pull request body.** They get renamed and
+ deleted, and nothing flags the text that named them. Say the behaviour is
+ covered.
+
+
+## Posting to GitHub
+
+Formatting belongs to the destination. This section applies to pull request
+bodies, issues, and comments on either. It applies nowhere else.
+
+- **No hard line wrapping. One paragraph is one line.** GitHub wraps to the
+ reader's window. Breaks at 80 columns produce a ragged column that looks
+ twice as long as it is, and they follow anyone who quotes you.
+- **Write real characters.** `--` renders as two hyphens, not an em dash.
+- **Open with the model in parentheses**, as the root
+ [`AGENTS.md`](../AGENTS.md) requires.
+
+### Pull request bodies
+
+- **The first three sentences must be enough to decide whether to read on.**
+ Say what changed, what breaks if you are wrong, and what you want from the
+ reader.
+- **Everything above the first collapsed section fits on one screen.**
+- **Say where you would look first.** Name the two or three places you are
+ least confident about. It aims the reviewer's attention, and it is the
+ cheapest thing you can do for their trust.
+- **The body describes the branch, not the review.** Edit it as the branch
+ changes, and let the threads carry the history.
+- **Record what you rejected.** That has no home in the tree, and it is the
+ part of a body most worth reading later.
+
+### Collapsed sections
+
+- **Fold evidence. Never fold a claim the reviewer has to weigh.** Test
+ output, tracebacks, and enumerations behind a visible headline are evidence.
+ A breaking change, a known gap, or something you did not test is a claim. A
+ folded admission is a deniable one, and that costs more than verbosity.
+- **The summary states the conclusion, so that opening it is optional.** A
+ reader skimming every summary line should come away with the argument. "Why
+ the write lock moved" fails that. "The write lock is built on first use,
+ because Dask's process scheduler rejects it at construction" passes.
+- **Leave a blank line after ``**, or the Markdown inside will not
+ render.
+
+### Alerts
+
+- **An alert is for something the reader must act on, or must not miss.**
+ Never for emphasis. Their whole value is scarcity.
+- **Two per body at most**, both above the first collapsed section. Wanting a
+ third means none of them were alerts.
+- **`[!IMPORTANT]` for a decision you need. `[!WARNING]` for something that
+ will break for someone.** Do not use `[!NOTE]`, `[!TIP]` or `[!CAUTION]` in
+ text you write; they decay into decoration. Something merely worth knowing
+ is a sentence.
+
+### Permalinks
+
+Press y on a file view to pin the URL to a commit. A permalink
+never rots, which makes it the answer to "never cite a line number".
+
+- **On its own line it renders the code inline.** Use that where the code is
+ the point.
+- **Inside a sentence it stays a link.** Use that in a bullet list, where a
+ preview would wreck the scanning the list exists for.
+- **Three or four per body.** Linking every call site turns the page into a
+ code dump.
+
+
+## Everything That Is Not GitHub
+
+Specifications, plans, and anything under `docs/` are read in other renderers,
+by people with more patience. Keep them to plain Markdown: no alerts, no
+collapsed sections, no ``. Footnotes are fine here and nowhere else.
+
+Their length is not a fault. They are read once, in full, before work starts.
+Every sentence rule above still applies to every line of them.
diff --git a/.hooks/README.md b/.hooks/README.md
new file mode 100644
index 0000000000..35143ce7de
--- /dev/null
+++ b/.hooks/README.md
@@ -0,0 +1,5 @@
+# Local pre-commit hooks
+
+Each module in this directory is a local pre-commit hook for Iris, written in the format expected by the pre-commit framework. See [`.pre-commit-config.yaml`](../.pre-commit-config.yaml).
+
+**TODO: review this new framework fully.**
diff --git a/.hooks/check_categorised_warnings.py b/.hooks/check_categorised_warnings.py
new file mode 100644
index 0000000000..a3d90122a3
--- /dev/null
+++ b/.hooks/check_categorised_warnings.py
@@ -0,0 +1,110 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+
+"""Pre-commit hook: ensure all warnings are categorised.
+
+All UserWarnings raised by Iris must be categorised (not use the base
+UserWarning class), to allow users to filter them.
+
+Usage (pre-commit passes staged file paths as arguments)::
+
+ python .hooks/check_categorised_warnings.py [file ...]
+
+Exit codes:
+ 0 – all clear
+ 1 – one or more files have categorisation issues
+"""
+
+import ast
+from pathlib import Path
+import sys
+
+
+def _get_warnings_issues(path: Path) -> list[str]:
+ """Return a list of warning categorisation issues in *path*."""
+ try:
+ file_text = path.read_text()
+ except (OSError, UnicodeDecodeError):
+ # Let other tools handle file read errors
+ return []
+
+ try:
+ tree = ast.parse(source=file_text, filename=str(path))
+ except SyntaxError:
+ # Let other hooks (check-ast) handle syntax errors
+ return []
+
+ violations = []
+
+ # Find all .warn() calls
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Call):
+ continue
+
+ # Check if this is a .warn(...) call
+ if not (isinstance(node.func, ast.Attribute) and node.func.attr == "warn"):
+ continue
+
+ warn_ref = f"{path}:{node.lineno}"
+
+ # Check for 'category' keyword argument
+ category_kwargs = [k for k in node.keywords if k.arg == "category"]
+
+ if not category_kwargs:
+ violations.append(
+ f"{warn_ref}: warning raised without category= kwarg — "
+ "use a specific IrisWarning subclass"
+ )
+ else:
+ category_kwarg = category_kwargs[0]
+ # Check if it's using the base UserWarning
+ category_name = None
+ if isinstance(category_kwarg.value, ast.Name):
+ category_name = category_kwarg.value.id
+ elif isinstance(category_kwarg.value, ast.Attribute):
+ category_name = category_kwarg.value.attr
+
+ if category_name == "UserWarning":
+ violations.append(
+ f"{warn_ref}: warning uses base UserWarning — "
+ "use a specific IrisWarning subclass"
+ )
+
+ return violations
+
+
+def check_file(path: Path) -> list[str]:
+ """Return a list of violation strings for *path*, empty if clean."""
+ if not path.suffix == ".py":
+ return []
+
+ return _get_warnings_issues(path)
+
+
+def main(argv: list[str] | None = None) -> int:
+ if argv is None:
+ argv = sys.argv[1:]
+
+ paths = [Path(p) for p in argv]
+ all_violations: list[str] = []
+
+ for path in paths:
+ all_violations.extend(check_file(path))
+
+ if all_violations:
+ print(
+ "Warning categorisation check failed.\n"
+ "All warnings raised by Iris must be categorised with a specific "
+ "IrisWarning subclass.\n"
+ )
+ for v in all_violations:
+ print(v)
+ return 1
+
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/.hooks/check_docs_page_metadata.py b/.hooks/check_docs_page_metadata.py
new file mode 100644
index 0000000000..ad733275b8
--- /dev/null
+++ b/.hooks/check_docs_page_metadata.py
@@ -0,0 +1,318 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+
+"""Pre-commit hook: validate the metadata every documentation page must carry.
+
+Two requirements are checked:
+
+* every non-exempt page carries a ``.. readingtime::`` directive;
+* every page with a Diataxis type carries exactly one correctly-configured
+ sphinx-needs item.
+
+Every page's metadata is authored in a version-controlled file, so none of
+this needs a documentation build to answer. Three kinds of file carry it:
+
+* pages under ``docs/src`` carry theirs in their own reStructuredText;
+* gallery pages are built from the scripts in ``docs/gallery_code``, and
+ carry theirs in the script's module docstring;
+* API pages are built from the library itself, and carry theirs in the
+ module's own docstring.
+
+The last two are the reason this hook does not simply walk ``.rst`` files:
+their pages appear under ``docs/src/generated`` during a build, and the
+generated reStructuredText contains an ``automodule`` directive rather than
+the item, which is only resolved later. Checking the docstrings instead
+reaches the same conclusion from files that are always present.
+
+The same rules are enforced during the documentation build, by the Sphinx
+extension in ``docs/src/sphinxext/page_metadata_validator.py``, which imports
+this module rather than reimplementing it. That is the point of putting the
+logic here: a rule added below is enforced in both places at once, and the two
+can never disagree about whether a page is valid.
+
+Usage (the hook passes no filenames; the whole repository is walked)::
+
+ python .hooks/check_docs_page_metadata.py [repository-root]
+
+Exit codes:
+ 0 – all clear
+ 1 – one or more pages are missing required metadata
+"""
+
+import ast
+import fnmatch
+from pathlib import Path, PurePosixPath
+import re
+import sys
+
+REPO_ROOT = Path(__file__).parent.parent
+"""The repository root, relative to this file."""
+
+READINGTIME_EXCEPTIONS = {
+ "index",
+ "*/index",
+ "voted_issues",
+ "sg_execution_times",
+ "copyright",
+ "user_manual/section_indexes/*",
+ "user_manual/reference/glossary",
+ "developers_guide/contributing_getting_involved",
+ "developers_guide/contributing_changes",
+ "developers_guide/contributing_codebase_index",
+ "developers_guide/contributing_documentation",
+ "developers_guide/contributing_testing_index",
+ "developers_guide/release_do_nothing",
+ "developers_guide/gitwash/**",
+ "whatsnew/**",
+}
+"""
+Pages exempt from readingtime validation.
+
+Supports glob patterns using standard wildcards.
+"""
+
+EXPECTED_TYPE_BY_PARENT = (
+ ("user_manual/tutorial", "tutorial"),
+ ("user_manual/explanation", "explanation"),
+ ("user_manual/how_to", "how-to"),
+ ("user_manual/reference", "z_reference"),
+)
+"""
+The sphinx-needs item type each Diataxis directory is expected to contain.
+
+Ordered; the first matching parent wins. The type names are the values of
+``Diataxis`` in ``docs/src/sphinxext/user_manual_directives.py``, which is
+where the corresponding directives are configured.
+"""
+
+GALLERY_TYPE = "how-to"
+"""The item type every gallery script's docstring is expected to carry."""
+
+API_TYPE = "z_reference"
+"""The item type every documented library module's docstring is expected to carry."""
+
+MAX_NEED_LINE = 25
+"""Items must start within this many lines, so links land at the page top."""
+
+_READINGTIME_PATTERN = re.compile(r"^\.\.\s+readingtime::", re.MULTILINE)
+_NEED_PATTERN = re.compile(
+ r"^\.\.\s+(tutorial|explanation|how-to|z_reference)::\s*(?P.*)$"
+)
+_OPTION_PATTERN = re.compile(r"^\s+:(?P<name>[^:]+):\s*(?P<value>.*)$")
+_TAG_SEPARATORS = re.compile(r"[;,]")
+
+
+def is_readingtime_exception(docname: str) -> bool:
+ """Return True if *docname* is exempt from the readingtime requirement."""
+ return any(fnmatch.fnmatch(docname, pattern) for pattern in READINGTIME_EXCEPTIONS)
+
+
+def expected_diataxis_type(docname: str) -> str | None:
+ """Return the sphinx-needs item type *docname* must carry, if any."""
+ doc_path = PurePosixPath(docname)
+ for parent, diataxis in EXPECTED_TYPE_BY_PARENT:
+ if PurePosixPath(parent) in doc_path.parents:
+ return diataxis
+ return None
+
+
+def _parse_directive(lines: list[str], index: int) -> tuple[dict[str, str], list[str]]:
+ """Return the options and content of the directive starting at *index*."""
+ options: dict[str, str] = {}
+ content: list[str] = []
+ # Options must directly follow the directive; the first blank line ends
+ # them, and everything indented after that is content.
+ in_options = True
+ for line in lines[index + 1 :]:
+ if not line.strip():
+ in_options = False
+ continue
+ if not line.startswith((" ", "\t")):
+ # Dedent: the directive has ended.
+ break
+ option = _OPTION_PATTERN.match(line) if in_options else None
+ if option:
+ options[option["name"].strip()] = option["value"].strip()
+ else:
+ in_options = False
+ content.append(line.strip())
+ return options, content
+
+
+def check_item(text: str, expected_type: str) -> list[str]:
+ """Return the problems with the sphinx-needs item in *text*, empty if clean.
+
+ *text* is whatever will be rendered at the top of the page - a page's own
+ reStructuredText, or the module docstring a page is generated from.
+ """
+ problems: list[str] = []
+
+ lines = text.splitlines()
+ found = [
+ (number, match)
+ for number, line in enumerate(lines, start=1)
+ if (match := _NEED_PATTERN.match(line))
+ ]
+ if len(found) != 1:
+ problems.append(
+ f"Page expected to have exactly 1 sphinx-needs item; found {len(found)}."
+ )
+ return problems
+
+ line_number, match = found[0]
+ if (page_type := match.group(1)) != expected_type:
+ problems.append(
+ f"sphinx-needs item expected to have type '{expected_type}'; "
+ f"found type '{page_type}'."
+ )
+ if line_number > MAX_NEED_LINE:
+ problems.append(
+ "sphinx-needs item expected to be defined within first "
+ f"{MAX_NEED_LINE} lines; found at line {line_number}."
+ )
+
+ # Title is not validated as it is always populated.
+
+ options, content = _parse_directive(lines, line_number - 1)
+ if not content:
+ problems.append("sphinx-needs item must have non-empty content section.")
+
+ tags = _TAG_SEPARATORS.split(options.get("tags", ""))
+ if not any(tag.strip().startswith("topic_") for tag in tags):
+ problems.append(
+ "sphinx-needs item must have at least one 'topic_xxx' tag in its "
+ "'tags' field."
+ )
+
+ return problems
+
+
+def check_page(docname: str, text: str) -> list[str]:
+ """Return a list of metadata problems with a page, empty if it is clean."""
+ problems: list[str] = []
+
+ if not is_readingtime_exception(docname) and not _READINGTIME_PATTERN.search(text):
+ problems.append(
+ "Missing '.. readingtime::' directive. Documentation pages should "
+ "include a readingtime directive for consistency. If this page "
+ f"should be exempt, add '{docname}' to READINGTIME_EXCEPTIONS in "
+ ".hooks/check_docs_page_metadata.py"
+ )
+
+ expected_type = expected_diataxis_type(docname)
+ if expected_type is not None:
+ problems.extend(check_item(text, expected_type))
+
+ return problems
+
+
+def check_docstring(text: str, expected_type: str) -> list[str]:
+ """Return a list of metadata problems with a Python file's docstring.
+
+ The module docstring is rendered at the top of the page generated from
+ *text*, and is therefore where that page's item is authored.
+ """
+ try:
+ docstring = ast.get_docstring(ast.parse(text))
+ except SyntaxError as error:
+ return [f"Could not be parsed, so its docstring cannot be checked: {error}"]
+
+ if docstring is None:
+ return [
+ "Missing a module docstring, so the page generated from it has no "
+ "sphinx-needs item."
+ ]
+
+ return check_item(docstring, expected_type)
+
+
+def source_pages(docs_src: Path) -> list[tuple[str, Path]]:
+ """Return the ``(docname, path)`` of every authored documentation page.
+
+ Anything below ``generated/`` is a build artifact - present only after a
+ documentation build, and only until the next ``make clean``. Including it
+ would make the result depend on whether the caller happens to have built
+ the docs; the files those pages are generated from are checked instead, by
+ :func:`gallery_scripts` and :func:`library_modules`.
+ """
+ pages = []
+ for path in sorted(docs_src.rglob("*.rst")):
+ relative = path.relative_to(docs_src)
+ if relative.parts[0] == "generated":
+ continue
+ pages.append((relative.with_suffix("").as_posix(), path))
+ return pages
+
+
+def gallery_scripts(gallery_code: Path) -> list[Path]:
+ """Return every gallery script that becomes a documentation page.
+
+ sphinx-gallery builds a page from each script matching its
+ ``filename_pattern``, configured in ``docs/src/conf.py``.
+ """
+ return sorted(gallery_code.rglob("plot_*.py"))
+
+
+def library_modules(library: Path) -> list[Path]:
+ """Return every library module that becomes an API page.
+
+ ``sphinxcontrib.apidoc`` writes one page per module - see the ``apidoc_*``
+ settings in ``docs/src/conf.py`` - skipping the tests, and skipping
+ anything private. ``iris.experimental.raster`` is also skipped there, to
+ avoid a gdal conflict, but is deliberately not skipped here: it carries
+ its item like any other module, and an exception would be more to explain
+ than it saves.
+ """
+ modules = []
+ for path in sorted(library.rglob("*.py")):
+ parts = path.relative_to(library).with_suffix("").parts
+ if parts[1:2] == ("tests",):
+ continue
+ if any(part.startswith("_") and part != "__init__" for part in parts):
+ continue
+ modules.append(path)
+ return modules
+
+
+def check_tree(repo_root: Path) -> list[str]:
+ """Return a list of problem strings for every page *repo_root* produces."""
+ problems: list[str] = []
+
+ def record(path: Path, found: list[str]) -> None:
+ problems.extend(f"{path}: {problem}" for problem in found)
+
+ for docname, path in source_pages(repo_root / "docs" / "src"):
+ record(path, check_page(docname, path.read_text(encoding="utf-8")))
+
+ for path in gallery_scripts(repo_root / "docs" / "gallery_code"):
+ record(path, check_docstring(path.read_text(encoding="utf-8"), GALLERY_TYPE))
+
+ for path in library_modules(repo_root / "lib"):
+ record(path, check_docstring(path.read_text(encoding="utf-8"), API_TYPE))
+
+ return problems
+
+
+def main(argv: list[str] | None = None) -> int:
+ if argv is None:
+ argv = sys.argv[1:]
+
+ repo_root = Path(argv[0]) if argv else REPO_ROOT
+ problems = check_tree(repo_root)
+
+ if problems:
+ print(
+ "Documentation page metadata check failed.\n"
+ "These problems would otherwise only surface during a docs build.\n"
+ )
+ for problem in problems:
+ print(problem)
+ return 1
+
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/.hooks/check_license_headers.py b/.hooks/check_license_headers.py
new file mode 100644
index 0000000000..0b47271dd3
--- /dev/null
+++ b/.hooks/check_license_headers.py
@@ -0,0 +1,184 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+
+"""Pre-commit hook: ensure Python files have proper license headers.
+
+All Python files (with some exceptions) must start with the Iris license
+header.
+
+Usage (pre-commit passes staged file paths as arguments)::
+
+ python .hooks/check_license_headers.py [file ...]
+
+Exit codes:
+ 0 – all clear
+ 1 – one or more files lack the required license header
+"""
+
+from fnmatch import fnmatch
+from pathlib import Path
+import subprocess
+import sys
+
+LICENSE_TEMPLATE = """# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details."""
+
+# Patterns of files to exclude from license header check
+_EXCLUDE_PATTERNS = (
+ "setup.py",
+ "noxfile.py",
+ "build/*",
+ "dist/*",
+ "docs/gallery_code/*/*.py",
+ "docs/src/developers_guide/documenting/*.py",
+ "docs/src/user_manual/tutorial/plotting_examples/*.py",
+ "docs/src/user_manual/tutorial/regridding_plots/*.py",
+ "docs/src/_build/*",
+ "lib/iris/analysis/_scipy_interpolate.py",
+)
+
+
+def _find_repo_root(start_path: Path) -> Path | None:
+ """Find the repository root by looking for .git."""
+ current = start_path.resolve()
+ while current != current.parent:
+ if (current / ".git").exists():
+ return current
+ current = current.parent
+ return None
+
+
+def _should_check(rel_path: str) -> bool:
+ """Return True if relative path should be checked for a license header."""
+ if not rel_path.endswith(".py"):
+ return False
+
+ return not any(fnmatch(rel_path, pat) for pat in _EXCLUDE_PATTERNS)
+
+
+def _get_license_header_content(path: Path) -> str:
+ """Get the content to check for license header, accounting for shebang."""
+ content = path.read_text()
+ if content.startswith("#!"):
+ # Strip shebang line
+ content = "\n".join(content.split("\n")[1:])
+ return content
+
+
+def check_file(path: Path, repo_root: Path | None = None) -> list[str]:
+ """Return a list of violation strings for *path*, empty if clean.
+
+ Args:
+ path: Path to check (can be relative or absolute).
+ repo_root: Repository root. If None, will be inferred.
+
+ Returns
+ -------
+ List of violation strings.
+ """
+ # Ensure path is absolute for consistent comparison
+ path = path.resolve()
+
+ if repo_root is None:
+ repo_root = _find_repo_root(path)
+ if not repo_root:
+ repo_root = path.parents[2] if len(path.parents) > 2 else path.parent
+
+ repo_root = repo_root.resolve()
+
+ try:
+ rel_path = path.relative_to(repo_root).as_posix()
+ except ValueError:
+ # Path is not under repo root
+ return []
+
+ if not _should_check(rel_path):
+ return []
+
+ try:
+ content = _get_license_header_content(path)
+ except (OSError, UnicodeDecodeError):
+ # Let other tools handle file read errors
+ return []
+
+ if not content.startswith(LICENSE_TEMPLATE):
+ return [f"{path}: missing or incorrect license header"]
+
+ return []
+
+
+def _get_all_tracked_files(repo_root: Path) -> list[Path]:
+ """Get all Python files tracked by git using git ls-files.
+
+ This is much faster than recursive glob for checking entire repo.
+ """
+ try:
+ output = subprocess.check_output(
+ ["git", "ls-files", "*.py"],
+ cwd=repo_root,
+ text=True,
+ )
+ return [
+ repo_root / line.strip() for line in output.splitlines() if line.strip()
+ ]
+ except (subprocess.CalledProcessError, FileNotFoundError):
+ # git not available or not a git repo; fall back to glob
+ return list(repo_root.rglob("*.py"))
+
+
+def main(argv: list[str] | None = None) -> int:
+ if argv is None:
+ argv = sys.argv[1:]
+
+ all_violations: list[str] = []
+
+ # If files provided, check only those files (pre-commit mode)
+ if argv:
+ paths = [Path(p) for p in argv]
+ repo_root = _find_repo_root(paths[0])
+ if not repo_root:
+ repo_root = (
+ paths[0].parents[2] if len(paths[0].parents) > 2 else paths[0].parent
+ )
+
+ for path in paths:
+ all_violations.extend(check_file(path, repo_root))
+ else:
+ # No files provided; check entire repo (test mode)
+ # This requires finding the repo root first
+ cwd = Path.cwd()
+ repo_root = _find_repo_root(cwd)
+ if not repo_root:
+ # Try to find .git by going up from cwd
+ current = cwd.resolve()
+ while current != current.parent:
+ if (current / ".git").exists():
+ repo_root = current
+ break
+ current = current.parent
+
+ if not repo_root:
+ print("Error: Could not find git repository root")
+ return 1
+
+ for path in _get_all_tracked_files(repo_root):
+ all_violations.extend(check_file(path, repo_root))
+
+ if all_violations:
+ print(
+ "License header check failed.\n"
+ "All Python files must start with the Iris license header.\n"
+ )
+ for v in all_violations:
+ print(v)
+ return 1
+
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/.hooks/check_netcdf4_imports.py b/.hooks/check_netcdf4_imports.py
new file mode 100644
index 0000000000..222a70d184
--- /dev/null
+++ b/.hooks/check_netcdf4_imports.py
@@ -0,0 +1,113 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+
+"""Pre-commit hook: ensure netCDF4 is only imported via _thread_safe_nc.
+
+Direct imports of the netCDF4 library are forbidden throughout Iris (except in
+the permitted files listed below). All access must go via the thread-safe
+wrapper at ``iris.fileformats.netcdf._thread_safe_nc``.
+
+Usage (pre-commit passes staged file paths as arguments)::
+
+ python .hooks/check_netcdf4_imports.py [file ...]
+
+Exit codes:
+ 0 – all clear
+ 1 – one or more files contain a forbidden netCDF4 import
+"""
+
+import ast
+from pathlib import Path
+import sys
+
+# Files that are allowed to import netCDF4 directly.
+_PERMITTED_SUFFIXES = (
+ # The thread-safe wrapper itself.
+ "iris/fileformats/netcdf/_thread_safe_nc.py",
+ # The test for the wrapper.
+ "iris/tests/unit/fileformats/netcdf/_thread_safe_nc/test_NetCDFWriteProxy.py",
+ # The tests for the bytecoding dataset wrapper.
+ "iris/tests/unit/fileformats/netcdf/test_bytecoding_datasets.py",
+ # The test of NetCDFDataset.from_existing() wrapping a bare netCDF4.Dataset,
+ # as ncdata hands to iris.save / CFReader.
+ "iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py",
+ # The test that CFReader wraps a bare netCDF4.Dataset borrowed from a
+ # caller, decoding its character data - the same wrapping as above, but
+ # exercised through CFReader rather than NetCDFDataset directly.
+ "iris/tests/unit/fileformats/cf/test_CFReader__dataset.py",
+ # The system test that checks netCDF4 is importable.
+ "iris/tests/system_test.py",
+)
+
+
+def _is_permitted(path: Path) -> bool:
+ """Return True if *path* is one of the permitted files."""
+ as_posix = path.as_posix()
+ return any(as_posix.endswith(suffix) for suffix in _PERMITTED_SUFFIXES)
+
+
+def _has_netcdf4_import(node: ast.AST) -> bool:
+ """Return True if *node* is a direct netCDF4 import statement."""
+ if isinstance(node, ast.Import):
+ return any(alias.name == "netCDF4" for alias in node.names)
+ if isinstance(node, ast.ImportFrom):
+ return node.module is not None and node.module.startswith("netCDF4")
+ return False
+
+
+def check_file(path: Path) -> list[str]:
+ """Return a list of violation strings for *path*, empty if clean."""
+ if _is_permitted(path):
+ return []
+
+ file_text = path.read_text()
+
+ # Fast pre-filter: skip parsing if "netCDF4" doesn't appear at all.
+ if "netCDF4" not in file_text:
+ return []
+
+ try:
+ tree = ast.parse(source=file_text, filename=str(path))
+ except SyntaxError:
+ # Let other hooks (check-ast) handle syntax errors.
+ return []
+
+ violations = []
+ for imp in ast.walk(tree):
+ if not isinstance(imp, (ast.Import, ast.ImportFrom)):
+ continue
+ if _has_netcdf4_import(imp):
+ violations.append(
+ f"{path}:{getattr(imp, 'lineno', 0)}: direct netCDF4 import — "
+ "use iris.fileformats.netcdf._thread_safe_nc instead."
+ )
+
+ return violations
+
+
+def main(argv: list[str] | None = None) -> int:
+ if argv is None:
+ argv = sys.argv[1:]
+
+ paths = [Path(p) for p in argv]
+ all_violations: list[str] = []
+
+ for path in paths:
+ all_violations.extend(check_file(path))
+
+ if all_violations:
+ print(
+ "netCDF4 import check failed.\n"
+ "All netCDF4 imports must be via iris.fileformats.netcdf._thread_safe_nc.\n"
+ )
+ for v in all_violations:
+ print(v)
+ return 1
+
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/.hooks/check_python_versions.py b/.hooks/check_python_versions.py
new file mode 100644
index 0000000000..bd138c0169
--- /dev/null
+++ b/.hooks/check_python_versions.py
@@ -0,0 +1,198 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+
+"""Pre-commit hook: ensure Python versions are consistent across config files.
+
+Python versions must be updated consistently everywhere they appear in the
+repository (pyproject.toml, noxfile.py, CI workflows, requirements files, etc.).
+
+This hook runs when any version-related file is modified and validates that
+all versions are in sync across the entire repository.
+
+Usage (pre-commit passes staged file paths as arguments)::
+
+ python .hooks/check_python_versions.py [file ...]
+
+Exit codes:
+ 0 – all versions are consistent
+ 1 – version inconsistencies detected
+"""
+
+from pathlib import Path
+import sys
+
+from packaging.version import Version
+
+# Files that contain Python version information
+_VERSION_CONFIG_FILES = {
+ "pyproject.toml",
+ "noxfile.py",
+ ".github/workflows/ci-wheels.yml",
+ ".github/workflows/ci-tests.yml",
+ "benchmarks/bm_runner.py",
+}
+
+
+def _find_repo_root(start_path: Path) -> Path:
+ """Find the repository root by looking for .git."""
+ current = start_path.resolve()
+ while current != current.parent:
+ if (current / ".git").exists():
+ return current
+ current = current.parent
+ return start_path
+
+
+def _extract_supported_versions(root_dir: Path) -> list[str] | None:
+ """Extract supported Python versions from pyproject.toml.
+
+ Returns None if versions cannot be determined.
+ """
+ pyproject = root_dir / "pyproject.toml"
+ if not pyproject.exists():
+ return None
+
+ try:
+ content = pyproject.read_text()
+ # Look for "Programming Language :: Python :: 3.X" entries
+ import re
+
+ matches = re.findall(r'"Programming Language :: Python :: (3\.\d+)"', content)
+ if matches:
+ return sorted(set(matches))
+ except (OSError, UnicodeDecodeError):
+ pass
+
+ return None
+
+
+def check_consistency(root_dir: Path) -> list[str]:
+ """Check that Python versions are consistent across all config files.
+
+ Returns a list of violation strings, empty if all consistent.
+ """
+ # Get the authoritative version list from pyproject.toml
+ all_supported = _extract_supported_versions(root_dir)
+ if not all_supported:
+ # Can't validate without knowing what versions should be supported
+ return []
+
+ _parsed = [Version(v) for v in all_supported]
+ latest_supported = str(max(_parsed))
+
+ violations = []
+ checks = []
+
+ # pyproject.toml - version classifiers
+ pyproject_toml_file = root_dir / "pyproject.toml"
+ if pyproject_toml_file.exists():
+ expected = "\n ".join(
+ [f'"Programming Language :: Python :: {ver}",' for ver in all_supported]
+ )
+ checks.append(
+ (pyproject_toml_file, expected, "pyproject.toml version classifiers")
+ )
+
+ # noxfile.py - _PY_VERSIONS_ALL
+ nox_file = root_dir / "noxfile.py"
+ if nox_file.exists():
+ expected = "_PY_VERSIONS_ALL = [" + ", ".join(
+ [f'"{ver}"' for ver in all_supported]
+ )
+ checks.append((nox_file, expected, "noxfile.py _PY_VERSIONS_ALL"))
+
+ # CI workflows
+ ci_wheels_file = root_dir / ".github" / "workflows" / "ci-wheels.yml"
+ if ci_wheels_file.exists():
+ expected = "python-version: [" + ", ".join(
+ [f'"{ver}"' for ver in all_supported]
+ )
+ checks.append((ci_wheels_file, expected, "ci-wheels.yml python-version"))
+
+ ci_tests_file = root_dir / ".github" / "workflows" / "ci-tests.yml"
+ if ci_tests_file.exists():
+ expected = (
+ f'python-version: ["{latest_supported}"]\n'
+ f'{" " * 8}session: ["doctest", "gallery"]'
+ )
+ checks.append((ci_tests_file, expected, "ci-tests.yml doctest/gallery"))
+
+ benchmarks_dir = root_dir / "benchmarks"
+ benchmark_runner_file = benchmarks_dir / "bm_runner.py"
+ if benchmark_runner_file.exists():
+ expected = f'python_version = "{latest_supported}"'
+ checks.append((benchmark_runner_file, expected, "bm_runner.py python_version"))
+
+ # Requirements files
+ requirements_dir = root_dir / "requirements"
+ if requirements_dir.exists():
+ for ver in all_supported:
+ req_yaml = requirements_dir / f"py{ver.replace('.', '')}.yml"
+ if req_yaml.exists():
+ expected = f"- python ={ver}"
+ checks.append(
+ (req_yaml, expected, f"requirements/py{ver.replace('.', '')}.yml")
+ )
+
+ # CI tests file version entries
+ if ci_tests_file.exists():
+ for ver in all_supported:
+ expected = f'python-version: "{ver}"\n{" " * 12}session: "tests"'
+ checks.append((ci_tests_file, expected, f"ci-tests.yml py{ver} tests"))
+
+ # Run all checks
+ for path, search, description in checks:
+ try:
+ content = path.read_text()
+ if search not in content:
+ violations.append(
+ f"{path}: missing expected version entry for {description}"
+ )
+ except (OSError, UnicodeDecodeError):
+ pass
+
+ return violations
+
+
+def main(argv: list[str] | None = None) -> int:
+ if argv is None:
+ argv = sys.argv[1:]
+
+ # Find repository root
+ if not argv:
+ # No files provided, can't determine repo root
+ # In this case, do nothing (let normal test handle it)
+ return 0
+
+ paths = [Path(p) for p in argv]
+ repo_root = _find_repo_root(paths[0])
+
+ # Check if any of the version-related files are in the staged files
+ is_version_file_staged = any(
+ str(p).endswith(tuple(f"/{name}" for name in _VERSION_CONFIG_FILES))
+ or str(p).endswith(tuple(_VERSION_CONFIG_FILES))
+ for p in paths
+ )
+
+ if not is_version_file_staged:
+ # Only run check if a version-related file is being modified
+ return 0
+
+ violations = check_consistency(repo_root)
+
+ if violations:
+ print(
+ "Python version consistency check failed.\n"
+ "Python versions must be updated consistently across all config files.\n"
+ )
+ for v in violations:
+ print(v)
+ return 1
+
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 2148d94604..384fdac500 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -13,7 +13,9 @@ ci:
# Alphabetised, for lack of a better order.
files: |
(?x)(
+ AGENTS\.md|
.github\/workflows\/.*|
+ .hooks\/.+\.py|
benchmarks\/.+\.py|
docs\/.+\.py|
lib\/.+\.py|
@@ -25,6 +27,48 @@ files: |
minimum_pre_commit_version: 1.21.0
repos:
+- repo: local
+ hooks:
+ - id: check-netcdf4-imports
+ name: Check netCDF4 imports are via _thread_safe_nc
+ language: python
+ entry: python .hooks/check_netcdf4_imports.py
+ types: [python]
+ - id: check-license-headers
+ name: Check license headers
+ language: python
+ entry: python .hooks/check_license_headers.py
+ types: [python]
+ - id: check-categorised-warnings
+ name: Check warnings are categorised
+ language: python
+ entry: python .hooks/check_categorised_warnings.py
+ types: [python]
+ - id: check-docs-page-metadata
+ name: Check docs page metadata
+ language: python
+ entry: python .hooks/check_docs_page_metadata.py
+ # Every file a documentation page is built from is walked on every
+ # run: it takes milliseconds, and it catches pages broken by a change
+ # made elsewhere, such as a rename. Passing filenames would also
+ # require admitting .rst to the global `files` list above, which would
+ # newly expose every page to the formatting hooks.
+ pass_filenames: false
+ always_run: true
+ - id: check-python-versions
+ name: Check Python version consistency
+ language: python
+ entry: python .hooks/check_python_versions.py
+ additional_dependencies: [packaging]
+ files: |
+ (?x)(
+ pyproject\.toml|
+ noxfile\.py|
+ \.github/workflows/(ci-tests|ci-wheels)\.yml|
+ benchmarks/bm_runner\.py|
+ requirements/py\d+\.yml
+ )
+
# Hook for pre-commit's built-in checks
- repo: https://github.com/pre-commit/pre-commit-hooks
diff --git a/AGENTS.md b/AGENTS.md
index 164eac11e7..7ee2cb12ee 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -9,8 +9,16 @@ built around CF-compliant multi-dimensional arrays ("Cubes").
Subdirectory AGENTS.md files take precedence for their subtrees:
- [`changelog/AGENTS.md`](changelog/AGENTS.md) — documentation on changelog
- [`docs/AGENTS.md`](docs/AGENTS.md) — documentation-specific rules
+- [`lib/iris/AGENTS.md`](lib/iris/AGENTS.md) — how to write library source code
- [`lib/iris/tests/AGENTS.md`](lib/iris/tests/AGENTS.md) — test-specific rules
+For a map of where code lives and how data flows, see
+[`lib/iris/ARCHITECTURE.md`](lib/iris/ARCHITECTURE.md).
+
+Everything you write for a person to read — chat replies, commit messages,
+pull request bodies, issues, specs, plans, docstrings and comments — follows
+[`.github/WRITING.md`](.github/WRITING.md). Read it before writing prose.
+
## Project Overview
@@ -21,7 +29,7 @@ Subdirectory AGENTS.md files take precedence for their subtrees:
| **Distribution** | conda-forge (`iris`), PyPI (`scitools-iris`) |
| **Key dependencies** | NumPy, Dask, SciPy, Cartopy, CF-Python, NetCDF4 |
| **Linter / formatter** | Ruff (88-char line length) |
-| **Test runner** | pytest + pytest-xdist (`-n auto`) |
+| **Test runner** | pytest; pytest-xdist is installed but **not** in `addopts` — pass `-n auto` yourself |
| **Env management** | nox + conda |
@@ -88,25 +96,31 @@ export CARTOPY_CACHE_DIR=~/.local/share/cartopy
## Testing
-- [`lib/iris/tests/AGENTS.md`](lib/iris/tests/AGENTS.md) — test-specific
+- [`lib/iris/tests/AGENTS.md`](lib/iris/tests/AGENTS.md) — test-specific rules.
+- Before finishing a change, run the repository's pre-commit hooks and fix any reported issues. (This is expected to encode a growing number of preferred coding practices, to enable productive collaboration with non-deterministic LLM agents).
## Code Style
-```bash
-# Lint
-ruff check lib/iris
+Linting and formatting are driven by pre-commit, not by invoking tools
+directly.
-# Auto-fix safe lint issues
-ruff check --fix lib/iris
+```bash
+# Check and auto-fix the files you changed (preferred while iterating)
+pre-commit run --files <paths>
-# Format
-ruff format lib/iris
+# Check and auto-fix everything
+pre-commit run --all-files
-# Check formatting without writing
-ruff format --check lib/iris
+# Run a single hook, e.g. after a formatting-only change
+pre-commit run ruff-format --all-files
```
+Pre-commit hooks rewrite files in place, so re-stage and re-run until it
+passes cleanly. Only fall back to `ruff check` / `ruff format` directly when
+pre-commit is unavailable — the hook config is the source of truth for
+versions and arguments.
+
- **Line length**: 88 characters (Ruff default).
- **Docstrings**: NumPy style; strictly validated.
- **Copyright header**: Every new Python file must start with:
@@ -118,8 +132,12 @@ ruff format --check lib/iris
# See LICENSE in the root of the repository for full licensing details.
```
-- **Imports**: Ruff-managed ordering. No direct `import netCDF4` — always use
- `iris.fileformats.netcdf._thread_safe_nc` for thread safety.
+- **Imports**: Ruff-managed ordering, applied by pre-commit. No direct
+ `import netCDF4` — always use `iris.fileformats.netcdf._thread_safe_nc` for
+ thread safety.
+- **Readability for humans and agents**: see
+ [`lib/iris/AGENTS.md`](lib/iris/AGENTS.md) for the house style on locality,
+ greppability, explicitness and comment hygiene.
## Development Conventions
@@ -159,8 +177,8 @@ strings.
## Changelog
-Changelog fragments lives under `chngelog/` and is built with towncrier via
-sphinx. See [`changelog/AGENTS.md`](changelog/AGENTS.md) for full rules.
+Changelog fragments live under `changelog/` and are built with towncrier
+via sphinx. See [`changelog/AGENTS.md`](changelog/AGENTS.md) for full rules.
## Documentation
@@ -179,18 +197,53 @@ make lockfiles
```
+## Posting to GitHub
+
+Open every text you post to GitHub with your model in parentheses, before
+anything else — pull request descriptions, issues, comments, replies and
+inline review comments alike:
+
+```
+(posted by Claude Opus)
+```
+
+Name the model, not its version. This **replaces** any trailing attribution
+your harness tells you to add: Iris has many authors using many models, and
+one short uniform marker beats a different footer for each.
+
+Git commit messages are out of scope. They are not posts, and they keep their
+own conventions — `Co-Authored-By:` trailers and the like — which this rule
+neither overrides nor requires.
+
+**Never hard-wrap a GitHub post.** One paragraph is one line; GitHub wraps to
+the reader's window.
+
+Collapsed sections, alerts, permalinks and the shape of a pull request body
+are in [`.github/WRITING.md`](.github/WRITING.md).
+
+
## Pull Request Guidelines
- When creating a pull request a template is provided to ensure all checks are
considered.
-- This project is configured to use pre-commit tht will ensure some checks are
- performed automatically.
+- This project is configured to use pre-commit, which will ensure some checks
+ are performed automatically.
- Keep changes focused; avoid unrelated refactors in the same PR.
- Add or update tests for every change to production code.
- Ensure a whatsnew fragment is added, see
[`changelog/AGENTS.md`](changelog/AGENTS.md)
+## Responding to Review
+
+A reviewer has not read every `AGENTS.md`; you have. Before actioning a
+comment, check whether the thing objected to is something these files
+explicitly sanction. If it is, reply with the rule and the file it lives in,
+and let a human settle it — complying silently turns one reviewer's
+preference into a change against a written convention, and nothing catches it
+afterwards.
+
+
## Critical Development Gotchas
1. **xfail_strict Behavior**: Tests marked `@pytest.mark.xfail` that now PASS
@@ -233,13 +286,14 @@ make lockfiles
- **Code of Conduct**: [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)
-## ⚠️ Meta-Instruction: Auto-Update Rule
-- **Trigger**: You MUST automatically update this `AGENTS.md` file if a new
- structural pattern, critical bug fix, or persistent repository rule is
- established during this session.
-- **Constraint 1**: Keep the file under 200 lines. Delete obsolete rules to
- make room for new ones.
-- **Constraint 2**: Do not ask for user permission before updating. Perform the
- edit silently as your final step.
-- **Constraint 3**: Only log global, reusable lessons. Do not log temporary or
+## ⚠️ Meta-Instruction: Changing This File
+- **Trigger**: If your work establishes a durable, reusable rule, you MUST
+ propose it before your session ends.
+- **Constraint 1**: Propose, never self-apply. Say it in your closing message,
+ or raise it as its own pull request. NEVER edit an `AGENTS.md` silently, or
+ as a side effect of unrelated work.
+- **Constraint 2**: Keep this file under 300 lines — every agent loads it in
+ full. If an addition would break that, tighten your wording; do NOT delete
+ existing guidance to make room. Removing a rule is its own proposal.
+- **Constraint 3**: Only global, reusable lessons. Do not propose temporary or
component-specific fixes.
diff --git a/MANIFEST.in b/MANIFEST.in
index 7c3ddac806..c952daa2e0 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -73,6 +73,7 @@ exclude pixi.lock
# SECTION: generic exclusions
# (1) top-level directories to omit entirely
prune .github
+prune .hooks
prune .nox
prune .tox
prune .coverage
diff --git a/benchmarks/benchmarks/generate_data/__init__.py b/benchmarks/benchmarks/generate_data/__init__.py
index 9a3671389b..34a3e11417 100644
--- a/benchmarks/benchmarks/generate_data/__init__.py
+++ b/benchmarks/benchmarks/generate_data/__init__.py
@@ -24,7 +24,6 @@
from warnings import warn
from iris._lazy_data import as_concrete_data
-from iris.fileformats import netcdf
#: Python executable used by :func:`run_function_elsewhere`, set via env
#: variable of same name. Must be path of Python within an environment that
@@ -123,13 +122,14 @@ def load_realised():
arrays.
"""
from iris.fileformats._nc_load_rules import helpers
- from iris.fileformats.netcdf.loader import _get_cf_var_data as pre_patched
+ from iris.fileformats.cf import loader as cf_loader
+ from iris.fileformats.cf.loader import _get_cf_var_data as pre_patched
def patched(*args, **kwargs):
return as_concrete_data(pre_patched(*args, **kwargs))
- netcdf.loader._get_cf_var_data = patched
+ cf_loader._get_cf_var_data = patched
helpers._get_cf_var_data = patched
yield
- netcdf.loader._get_cf_var_data = pre_patched
+ cf_loader._get_cf_var_data = pre_patched
helpers._get_cf_var_data = pre_patched
diff --git a/changelog/7303.internal.rst b/changelog/7303.internal.rst
new file mode 100644
index 0000000000..0d7cfbccd0
--- /dev/null
+++ b/changelog/7303.internal.rst
@@ -0,0 +1 @@
+:user:`trexfeathers` removed ``iris.fileformats.netcdf.saver.VariableEmulator``.
diff --git a/changelog/7312.internal.rst b/changelog/7312.internal.rst
new file mode 100644
index 0000000000..b5f6280674
--- /dev/null
+++ b/changelog/7312.internal.rst
@@ -0,0 +1,5 @@
+:user:`trexfeathers` tightened the ``AGENTS.md`` agent instructions: comment
+and docstring rules are regrouped, lazy data has its own section, citations
+that decay silently - line numbers in other files, counts of call sites - are
+banned, and a new section covers a review comment that contradicts a written
+rule.
diff --git a/changelog/README.rst b/changelog/README.rst
index 77574bcb60..0711445ffd 100644
--- a/changelog/README.rst
+++ b/changelog/README.rst
@@ -59,12 +59,16 @@ For example, a file named ``7146.feature.rst`` might contain::
Notes
-----
+* Before recording a removal, check whether the name predates your branch:
+ ``git log -S<name> <base-branch> -- <path>``. Removing something your own
+ branch added needs no fragment; removing something already released is an
+ API change, and does.
* You do not need to include the PR number in the fragment as it is already in
the fragment filename.
* Multiple fragments may reference the same PR number if the PR makes
changes across different categories.
* For multiple PRs that reference the same change, simply create a separate
- changlog fragment file for each PR with **identical** contents.
+ changelog fragment file for each PR with **identical** contents.
* Use ``:issue:`NNNN``` for issue references, ``:pull:`NNNN``` for PR
references, and ``:user:`github-name``` for user references.
* The rendered changelog can be previewed with::
diff --git a/docs/AGENTS.md b/docs/AGENTS.md
index 31f506c2d7..8201806a86 100644
--- a/docs/AGENTS.md
+++ b/docs/AGENTS.md
@@ -55,7 +55,7 @@ Build output goes to `docs/src/_build/html/`.
## Whatsnew Entries
-- See [`AGENTS.md`](../changelog/AGENTS.md) in the `/changelog` directory.
+- See [`AGENTS.md`](../changelog/AGENTS.md) in the `/changelog` directory.
## RST / Sphinx Conventions
@@ -71,12 +71,39 @@ Build output goes to `docs/src/_build/html/`.
``:func:`iris.load` ``, etc.
- Gallery scripts live under `docs/gallery_code/` and must be valid
standalone Python files executable by `matplotlib` / `sphinx-gallery`.
+- Published prose must never cite an artefact of how the work was done — a
+ review finding, a task or plan number, a phase. These resolve to nothing
+ for a reader. Cite an issue, a pull request, a design document, or the
+ convention's section. The same applies to changelog fragments.
+
+
+## Page Metadata
+
+Every page's metadata is authored in a version-controlled file, never in
+build output:
+
+- pages under `docs/src/` state theirs in their own RST;
+- gallery pages state theirs in their script's module docstring;
+- API pages state theirs in the library module's docstring.
+
+Pages under `docs/src/` need a `.. readingtime::` directive. Every page with
+a Diataxis type — the `user_manual/` subdirectories, and all gallery and API
+pages — needs exactly one sphinx-needs item within its first 25 lines, with
+non-empty content and at least one `topic_*` tag (tags are separated by `;`).
+Copy the form from a sibling page.
+
+The rules live in `.hooks/check_docs_page_metadata.py`, and run both as the
+`check-docs-page-metadata` pre-commit hook and during the build. Prefer the
+hook: `pre-commit run check-docs-page-metadata` answers in milliseconds what
+a build takes minutes to tell you. Exemptions belong in that file's
+`READINGTIME_EXCEPTIONS`, and a new rule belongs there too — never in the
+Sphinx extension alone, or the two callers will disagree.
## Gallery Examples
- Each gallery script must have a module-level docstring that becomes its
- title and description.
+ title and description, and that carries the page's `how-to` item.
- Scripts are grouped by subdirectory: `general/`, `meteorology/`,
`oceanography/`.
- Gallery tests in `docs/gallery_tests/` verify examples execute without
@@ -87,7 +114,8 @@ Build output goes to `docs/src/_build/html/`.
## Doctest / Inline Code Examples
-- Doctests in RST files are run via `make doctest` (from `docs/src/`).
+- How to run them, and what not to substitute, is under Doctests in
+ [`lib/iris/tests/AGENTS.md`](../lib/iris/tests/AGENTS.md).
- Use `# doctest: +SKIP` sparingly and only when execution is genuinely
impossible (e.g., requires a display).
- Ensure all `>>>` examples produce the exact output shown, or use
@@ -104,15 +132,30 @@ Build output goes to `docs/src/_build/html/`.
will break in CI.
4. **API doc changes**: Moving or renaming public symbols requires updating
any manual cross-references in the RST files.
-
-
-## ⚠️ Meta-Instruction: Auto-Update Rule
-- **Trigger**: You MUST automatically update this `AGENTS.md` file if a new
- structural pattern, critical bug fix, or persistent repository rule is
- established during this session.
-- **Constraint 1**: Keep the file under 200 lines. Delete obsolete rules to
- make room for new ones.
-- **Constraint 2**: Do not ask for user permission before updating. Perform the
- edit silently as your final step.
-- **Constraint 3**: Only log global, reusable lessons. Do not log temporary or
+5. **Re-export packages (`__init__.py` importing from private modules)**:
+ run the docs build, and read the rendered page - the tests will not catch
+ these. Three failure modes, two of them silent:
+ - A public name missing from `__all__` is dropped from the page; autodoc
+ rejects it on `__module__`.
+ - Module-level **data** is documented only where the module's own source
+ assigns it. An imported name produces no entry, no warning and no
+ failure; `__all__` does not help. Re-state it with its `#:` comment.
+ - Never "fix" either by rewriting `__module__` onto the package.
+ `inspect` resolves a class's source through it, so that breaks
+ `getsource` and silently strips every `[source]` link. If a class is
+ also aliased as a class attribute, autodoc describes it twice and the
+ two collide - suppress the duplicate with `:meta private:` on the
+ alias instead.
+
+
+## ⚠️ Meta-Instruction: Changing This File
+- **Trigger**: If your work establishes a durable, reusable rule, you MUST
+ propose it before your session ends.
+- **Constraint 1**: Propose, never self-apply. Say it in your closing message,
+ or raise it as its own pull request. NEVER edit an `AGENTS.md` silently, or
+ as a side effect of unrelated work.
+- **Constraint 2**: Keep this file under 300 lines — every agent loads it in
+ full. If an addition would break that, tighten your wording; do NOT delete
+ existing guidance to make room. Removing a rule is its own proposal.
+- **Constraint 3**: Only global, reusable lessons. Do not propose temporary or
component-specific fixes.
diff --git a/docs/src/conf.py b/docs/src/conf.py
index b243d8d8ca..8b029498c2 100644
--- a/docs/src/conf.py
+++ b/docs/src/conf.py
@@ -154,8 +154,8 @@ def _dotv(version):
extensions = [
"matplotlib.sphinxext.mathmpl",
"matplotlib.sphinxext.plot_directive",
+ "page_metadata_validator",
"readingtime",
- "readingtime_validator",
"sphinx.ext.autodoc",
"sphinx.ext.autosummary",
"sphinx.ext.coverage",
diff --git a/docs/src/sphinxext/page_metadata_validator.py b/docs/src/sphinxext/page_metadata_validator.py
new file mode 100644
index 0000000000..5b385007d4
--- /dev/null
+++ b/docs/src/sphinxext/page_metadata_validator.py
@@ -0,0 +1,72 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Sphinx extension to validate the metadata every documentation page carries.
+
+The rules themselves live in ``.hooks/check_docs_page_metadata.py``, which is
+also a pre-commit hook. Keeping them there means an author gets told about a
+missing ``readingtime`` directive or sphinx-needs item in milliseconds, rather
+than several minutes into a documentation build - while this extension keeps
+the build itself authoritative, so nothing depends on a hook having been run.
+
+Every rule is answerable from version-controlled source, including the rules
+for pages generated from gallery scripts and library modules, so this
+extension does not inspect anything the build produces. It runs the same
+check over the same files that the hook does; only the trigger differs.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+import typing
+
+from sphinx.util import logging as sphinx_logging
+
+if typing.TYPE_CHECKING:
+ from sphinx.application import Sphinx
+
+logger = sphinx_logging.getLogger(__name__)
+
+REPO_ROOT = Path(__file__).parents[3]
+
+
+def _load_page_metadata_rules():
+ """Load the validation rules shared with the pre-commit hook.
+
+ The rules live outside the documentation tree, in ``.hooks/``, alongside
+ Iris' other pre-commit hooks; that directory is not importable, so the
+ module is loaded by path. ``lib/iris/tests/test_coding_standards.py``
+ loads the other hooks the same way.
+ """
+ hook_path = REPO_ROOT / ".hooks" / "check_docs_page_metadata.py"
+ spec = importlib.util.spec_from_file_location("check_docs_page_metadata", hook_path)
+ if spec is None or spec.loader is None:
+ message = f"Failed to load the page metadata rules from: {hook_path}"
+ raise RuntimeError(message)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+rules = _load_page_metadata_rules()
+
+
+def validate_pages(app: Sphinx) -> None:
+ """Report every page whose metadata is missing or malformed."""
+ for problem in rules.check_tree(REPO_ROOT):
+ logger.error(problem)
+
+
+def setup(app: Sphinx) -> dict:
+ # Connect at builder-inited: the check reads source files only, so there
+ # is nothing to wait for, and reporting before the build starts is
+ # quicker for the author than reporting at the end of it.
+ app.connect("builder-inited", validate_pages)
+
+ return {
+ "version": "0.1.0",
+ "parallel_read_safe": True,
+ "parallel_write_safe": True,
+ }
diff --git a/docs/src/sphinxext/readingtime_validator.py b/docs/src/sphinxext/readingtime_validator.py
deleted file mode 100644
index 210545bccc..0000000000
--- a/docs/src/sphinxext/readingtime_validator.py
+++ /dev/null
@@ -1,101 +0,0 @@
-# Copyright Iris contributors
-#
-# This file is part of Iris and is released under the BSD license.
-# See LICENSE in the root of the repository for full licensing details.
-"""Sphinx extension to validate readingtime directive presence in docs pages."""
-
-from __future__ import annotations
-
-import fnmatch
-from pathlib import Path
-from typing import TYPE_CHECKING
-
-from docutils import nodes # type: ignore[import-untyped]
-from sphinx.util import logging as sphinx_logging
-
-if TYPE_CHECKING:
- from sphinx.application import Sphinx
-
-logger = sphinx_logging.getLogger(__name__)
-
-READINGTIME_EXCEPTIONS = {
- "index",
- "*/index",
- "voted_issues",
- "sg_execution_times",
- "copyright",
- "user_manual/section_indexes/*",
- "user_manual/reference/glossary",
- "developers_guide/contributing_getting_involved",
- "developers_guide/contributing_changes",
- "developers_guide/contributing_codebase_index",
- "developers_guide/contributing_documentation",
- "developers_guide/contributing_testing_index",
- "developers_guide/release_do_nothing",
- "developers_guide/gitwash/**",
- "generated/**",
- "whatsnew/**",
-}
-"""
-Pages exempt from readingtime validation.
-
-Supports glob patterns using standard wildcards.
-"""
-
-
-def _is_exception(docname: str) -> bool:
- for pattern in READINGTIME_EXCEPTIONS:
- if fnmatch.fnmatch(docname, pattern):
- return True
- return False
-
-
-def _has_readingtime_directive(doctree: nodes.document) -> bool:
- # Look for nodes generated by the readingtime directive
- # The directive produces a raw HTML node with a specific class
- for node in doctree.findall(nodes.raw):
- if "reading-time" in str(node):
- return True
- return False
-
-
-def _validate_readingtime(
- app: Sphinx,
- doctree: nodes.document,
- docname: str,
-) -> None:
- """Validate ``readingtime`` directive presence in docs pages.
-
- Parameters
- ----------
- app : Sphinx
- The Sphinx application object.
- doctree : nodes.document
- The parsed document tree.
- docname : str
- The name of the document being processed.
-
- """
- # Check if this docname matches any exception pattern
- if _is_exception(docname):
- return
-
- # Validate readingtime presence
- if not _has_readingtime_directive(doctree):
- msg = (
- f"Missing '.. readingtime::' directive in '{docname}.rst'. "
- f"Documentation pages should include a readingtime directive for consistency. "
- f"If this page should be exempt, add '{docname}' to READINGTIME_EXCEPTIONS "
- f"in sphinxext/readingtime_validator.py"
- )
- logger.warning(msg)
-
-
-def setup(app: Sphinx) -> dict:
- app.connect("doctree-resolved", _validate_readingtime)
-
- return {
- "version": "0.1.0",
- "parallel_read_safe": True,
- "parallel_write_safe": True,
- }
diff --git a/docs/src/sphinxext/user_manual_directives.py b/docs/src/sphinxext/user_manual_directives.py
index fa90a8bccf..ad8c4573a4 100644
--- a/docs/src/sphinxext/user_manual_directives.py
+++ b/docs/src/sphinxext/user_manual_directives.py
@@ -8,24 +8,21 @@
from pathlib import Path
import re
from textwrap import indent
-import typing
from docutils import nodes # type: ignore[import-untyped]
from docutils.parsers.rst import Directive # type: ignore[import-untyped]
from docutils.statemachine import StringList # type: ignore[import-untyped]
from sphinx.application import Sphinx
-from sphinx.builders import Builder
-from sphinx.util import logging as sphinx_logging
-from sphinx_needs.api import get_needs_view
-
-if typing.TYPE_CHECKING:
- from sphinx_needs.api.need import NeedsInfoType
-
-logger = sphinx_logging.getLogger(__name__)
class Diataxis(enum.StrEnum):
- """The Diataxis-inspired sphinx-needs directives configured in conf.py."""
+ """The Diataxis-inspired sphinx-needs directives configured in conf.py.
+
+ Which pages are expected to carry which of these is recorded in
+ ``.hooks/check_docs_page_metadata.py`` - by directory in
+ ``EXPECTED_TYPE_BY_PARENT``, and in ``GALLERY_TYPE`` and ``API_TYPE`` for
+ the generated pages; renaming a member here means editing those too.
+ """
ALL = "all"
TUTORIAL = "tutorial"
@@ -152,100 +149,11 @@ def run(self):
return node.children
-def validate_items(app: Sphinx, builder: Builder) -> None:
- """Validate that each user manual page has a single correctly configured item."""
- env = app.env
- found_docs: typing.Iterable[str] = env.found_docs
-
- # Read-only iterable of all sphinx-needs items; only valid in the write phase.
- needs_view = get_needs_view(app)
- # Group needs by docname
- by_doc: dict[Path, list[NeedsInfoType]] = {}
- for need_id in needs_view:
- need = needs_view[need_id]
- doc_name = need.get("docname")
- if not doc_name:
- # External/imported needs may have no docname; skip page accounting
- continue
- by_doc.setdefault(Path(doc_name), []).append(need)
-
- def _get_expected_type(doc_path: Path) -> typing.Optional[Diataxis]:
- """Get the expected Diataxis type for the given document path."""
- parents_and_diataxis = [
- (Path("generated/api"), Diataxis.REFERENCE),
- (Path("generated/gallery"), Diataxis.HOW_TO),
- (Path("user_manual/tutorial"), Diataxis.TUTORIAL),
- (Path("user_manual/explanation"), Diataxis.EXPLANATION),
- (Path("user_manual/how_to"), Diataxis.HOW_TO),
- (Path("user_manual/reference"), Diataxis.REFERENCE),
- ]
- expected = None
- for parent, diataxis in parents_and_diataxis:
- if parent in doc_path.parents:
- expected = diataxis
- break
- if Path("generated/gallery") in doc_path.parents and doc_path.name == "index":
- expected = None
- if doc_path.name == "sg_execution_times":
- expected = None
- return expected
-
- for doc_name in found_docs:
- doc_path = Path(doc_name)
- expected_type = _get_expected_type(doc_path)
- if expected_type is not None:
- problem_prefix = "Page expected to have exactly 1 sphinx-needs item;"
- try:
- (page_need,) = by_doc[doc_path]
- except KeyError:
- problem = f"{problem_prefix} found 0."
- logger.error(problem, location=doc_name)
- continue
- except ValueError:
- count = len(by_doc[doc_path])
- problem = f"{problem_prefix} found {count}."
- logger.error(problem, location=doc_name)
- continue
-
- if (page_type := page_need["type"]) != expected_type:
- problem = (
- "sphinx-needs item expected to have type "
- f"'{expected_type}'; found type '{page_type}'."
- )
- logger.error(problem, location=doc_name)
-
- if (line_no := page_need.get("lineno")) > 25:
- # Ensures that links to the needs directive take reader to the
- # start of the page.
- problem = (
- "sphinx-needs item expected to be defined within "
- f"first 25 lines; found at line {line_no}."
- )
- logger.error(problem, location=doc_name)
-
- # Title is not validated as it is always populated.
-
- if page_need["content"] == "":
- problem = "sphinx-needs item must have non-empty content section."
- logger.error(problem, location=doc_name)
-
- tags = page_need.get("tags", [])
- if [tag for tag in tags if tag.startswith("topic_")] == []:
- problem = (
- "sphinx-needs item must have at least one 'topic_xxx' tag "
- "in its 'tags' field."
- )
- logger.error(problem, location=doc_name)
-
-
def setup(app: Sphinx):
"""Set up the Sphinx extension.
This function is expected by Sphinx to register the extension.
"""
- # Connect at write-started so needs are fully collected & resolved.
- app.connect("write-started", validate_items)
-
app.add_directive("diataxis-page-list", DiataxisDirective)
return {"version": "0.1"}
diff --git a/docs/src/user_manual/explanation/netcdf_io.rst b/docs/src/user_manual/explanation/netcdf_io.rst
index 35611c92e8..a643d61135 100644
--- a/docs/src/user_manual/explanation/netcdf_io.rst
+++ b/docs/src/user_manual/explanation/netcdf_io.rst
@@ -6,7 +6,7 @@
.. testsetup:: chunk_control
import iris
- from iris.fileformats.netcdf.loader import CHUNK_CONTROL
+ from iris.fileformats.cf.loader import CHUNK_CONTROL
from pathlib import Path
import dask
@@ -67,14 +67,24 @@ calculated based on a number of factors, including:
>>> print(cube.core_data().chunksize)
(60, 37, 49)
+A file chunk is read and decompressed whole, so a Dask chunk smaller than one
+costs a re-read per division. Iris does not adjust its chunking to avoid
+that, but it warns when a file chunk is larger than Dask's default chunksize,
+which is where the cost starts to be worth your attention.
+
For more user control, functionality was updated in :pull:`5588`, with the
-creation of the :data:`iris.fileformats.netcdf.loader.CHUNK_CONTROL` class.
+creation of the :data:`iris.fileformats.cf.loader.CHUNK_CONTROL` class.
+
+Chunking policy is not specific to NetCDF, so from Iris 3.17 it lives in
+:mod:`iris.fileformats.cf.loader`. The old name,
+``iris.fileformats.netcdf.loader.CHUNK_CONTROL``, still works but raises
+a deprecation warning.
Custom Chunking: Set
^^^^^^^^^^^^^^^^^^^^
-There are three context managers within :data:`~iris.fileformats.netcdf.loader.CHUNK_CONTROL`. The most basic is
-:meth:`~iris.fileformats.netcdf.loader.ChunkControl.set`. This allows you to specify the chunksize for each dimension,
+There are three context managers within :data:`~iris.fileformats.cf.loader.CHUNK_CONTROL`. The most basic is
+:meth:`~iris.fileformats.cf.loader.ChunkControl.set`. This allows you to specify the chunksize for each dimension,
and to specify a ``var_name`` specifically to change.
Using ``-1`` in place of a chunksize will ensure the chunksize stays the same
@@ -104,7 +114,7 @@ specify only one dimension, the rest will be optimised using Iris' default behav
Custom Chunking: From File
^^^^^^^^^^^^^^^^^^^^^^^^^^
-The second context manager is :meth:`~iris.fileformats.netcdf.loader.ChunkControl.from_file`.
+The second context manager is :meth:`~iris.fileformats.cf.loader.ChunkControl.from_file`.
This takes chunksizes as defined in the NetCDF file. Any dimensions without specified chunks
will default to Iris optimisation.
@@ -120,7 +130,7 @@ will default to Iris optimisation.
Custom Chunking: As Dask
^^^^^^^^^^^^^^^^^^^^^^^^
-The final context manager, :meth:`~iris.fileformats.netcdf.loader.ChunkControl.as_dask`, bypasses
+The final context manager, :meth:`~iris.fileformats.cf.loader.ChunkControl.as_dask`, bypasses
Iris' optimisation all together, and will take its chunksizes from Dask's behaviour.
.. doctest:: chunk_control
@@ -482,7 +492,7 @@ loader so it can be make a more informed decision on lazy loading:
.. doctest::
>>> import iris
- >>> from iris.fileformats.netcdf.loader import CHUNK_CONTROL
+ >>> from iris.fileformats.cf.loader import CHUNK_CONTROL
>>>
>>> sample_file = iris.sample_data_path("vlstr_type.nc")
>>> cube = iris.load_cube(sample_file)
diff --git a/docs/src/user_manual/explanation/zarr_io.rst b/docs/src/user_manual/explanation/zarr_io.rst
new file mode 100644
index 0000000000..76efe4374f
--- /dev/null
+++ b/docs/src/user_manual/explanation/zarr_io.rst
@@ -0,0 +1,190 @@
+.. explanation:: Zarr I/O Handling in Iris
+ :tags: topic_load_save
+
+ Read about how Iris loads Zarr stores.
+
+.. include:: ../../common_links.inc
+
+.. _zarr_io:
+
+=========================
+Zarr I/O Handling in Iris
+=========================
+
+.. readingtime::
+
+`Zarr <https://zarr.dev/>`_ stores an array as many independently compressed
+chunks under a prefix, rather than as one file. Iris reads Zarr version 2 and
+version 3 stores natively, through
+`zarr-python <https://zarr.readthedocs.io/>`_.
+
+``zarr`` is an **optional dependency**, and Iris requires at least version
+3.0.8. Install it to load Zarr stores::
+
+ conda install -c conda-forge "zarr>=3.0.8"
+
+Iris without it installs and runs normally, raising an ``ImportError`` that
+names the package and its minimum required version if you ask it to load a
+store.
+
+.. _zarr_io_loading:
+
+Loading
+-------
+
+A store is a directory, or a URL, so pass it where you would pass a file::
+
+ cubes = iris.load("/data/forecast.zarr")
+
+Iris recognises a local store by the metadata file at its root -- ``zarr.json``
+for version 3, ``.zgroup`` for version 2 -- not by the ``.zarr`` suffix, so a
+directory that merely has the suffix is not mistaken for one.
+
+A store holding UGRID mesh data loads as a mesh cube, the same as the
+equivalent netCDF file would. :func:`iris.mesh.load_meshes` reads a store
+too.
+
+Saving
+------
+
+Iris does not save to a native Zarr store. ``iris.save(cube, "out.zarr")``
+raises ``ValueError: Cannot save; no saver``, naming neither Zarr nor an
+alternative. Save to netCDF instead, or write Zarr through netCDF-c's NCZarr
+driver, covered below in :ref:`zarr_io_nczarr`.
+
+A cube loaded from Zarr may carry an attribute netCDF cannot hold, which
+fails the save. See :ref:`zarr_io_attributes`.
+
+Remote stores
+-------------
+
+A URL with one of the ``s3``, ``gs``, ``az`` or ``abfs`` schemes, ending in
+``.zarr``, is passed to zarr-python unaltered::
+
+ cubes = iris.load("s3://noaa-gfs-bdp-pds/forecast.zarr")
+
+Iris adds no store library of its own and handles no credentials. Install the
+one your scheme needs -- ``s3fs``, ``gcsfs``, ``adlfs`` -- and configure it as
+that package documents. zarr-python raises an ``ImportError`` naming the
+package if it is missing.
+
+Groups
+------
+
+A Zarr store may hold several groups, and they need not describe the same
+thing. Iris loads **one** group per call, like xarray, rather than merging
+them: ``iris.load`` reads the root group, and a store whose root holds no data
+variables raises a message naming the groups that do.
+
+To read another group, call the Zarr loader directly::
+
+ from iris.cube import CubeList
+ import iris.fileformats.zarr
+
+ cubes = CubeList(
+ iris.fileformats.zarr.load_cubes(
+ "/data/sentinel.zarr", group="measurements/reflectance/r10m"
+ )
+ )
+
+Dimension names
+---------------
+
+A netCDF dimension has a name, one length, and a meaning shared across the
+whole file. A Zarr dimension name promises less. An array need not name its
+dimensions at all, and the same name in two arrays guarantees nothing -- not
+even the same length. Both are permitted by the specification.
+
+So Iris treats two arrays as sharing a dimension only when they name it
+identically **and** it is the same length in both. Anything else loads as an
+**anonymous** dimension, which is an ordinary Iris cube dimension with no
+coordinate on it. That covers both a dimension the store never named, and a
+name the store used at two different lengths.
+
+A coordinate left with nothing to attach to is not discarded. It goes to
+:data:`iris.loading.LOAD_PROBLEMS`, and the load warns that not every object
+was parsed.
+
+Version 2 stores have no dimension-name field. They carry names in an
+``_ARRAY_DIMENSIONS`` attribute instead -- the convention xarray established
+-- which Iris reads, so dimensions are named on both versions. The attribute
+describes the store rather than the data, so it does not itself reach
+``cube.attributes``.
+
+Missing data
+------------
+
+Zarr returns a plain array and applies no masking, so Iris does it, from the
+CF attributes: ``_FillValue`` and ``missing_value`` for sentinels,
+``valid_min``/``valid_max``/``valid_range`` for a range, and
+``scale_factor``/``add_offset`` for packed data.
+
+A Zarr array also carries a ``fill_value`` in its metadata. That says what
+unwritten storage reads back as, not which values are missing, and
+zarr-python defaults it to zero. Iris reads it as missing data only on a
+version 2 store whose array declares no CF attribute -- which is what version
+2 writers meant by it, and what xarray does.
+
+That one case can mask a genuine zero, so Iris warns when it happens, naming
+the variable. Setting a ``_FillValue`` attribute on the array prevents both
+the masking and the warning.
+
+Chunking
+--------
+
+Iris chooses Dask chunks from the store's own chunking, as it does for
+netCDF, and :data:`~iris.fileformats.cf.loader.CHUNK_CONTROL` works the same
+way for both formats.
+
+A Zarr chunk is the atomic unit of storage: reading any part of one fetches
+and decompresses the whole object, so a Dask chunk smaller than a store chunk
+costs a re-read per division. Iris does not adjust its chunking to avoid
+that, any more than it does for netCDF, but it warns when a store chunk is
+larger than Dask's target chunk size -- where the cost starts to be worth
+your attention. Use ``CHUNK_CONTROL`` to choose the chunking yourself.
+
+A **sharded** array is chunked on its inner chunks, not its shards. Zarr's
+sharding codec is indexed, so a reader fetches only the inner chunks it
+needs.
+
+zarr-python fetches chunks concurrently, and Dask's worker count multiplies
+with that concurrency. If a remote store rate-limits you, lower either Dask's
+worker count or zarr-python's own ``zarr.config`` setting
+``async.concurrency``.
+
+.. _zarr_io_attributes:
+
+Attributes
+----------
+
+A Zarr attribute value may be any JSON literal, including many things CF
+cannot express, such as multidimensional arrays, dictionaries or lists of
+lists. Iris carries such a value onto the cube unchanged and does not
+interpret it. See :issue:`7288` for the wider question of how the CF
+attribute model should map onto JSON.
+
+.. note::
+
+ A cube carrying such an attribute **cannot be saved as netCDF**, because
+ netCDF attribute values are scalars, strings and one-dimensional arrays.
+ The netCDF library raises ``TypeError: illegal data type for attribute
+ b'stac_discovery'``, which does not name the cube. Delete the attribute
+ before saving::
+
+ del cube.attributes["stac_discovery"]
+
+ See :issue:`7329`.
+
+.. _zarr_io_nczarr:
+
+NCZarr
+------
+
+Iris also reads and writes Zarr through netCDF-c's NCZarr driver, reached by
+way of a URL fragment::
+
+ iris.load("file:///data/forecast.zarr#mode=nczarr,file")
+
+NCZarr implements Zarr version 2, and writes scalar variables as length-one
+arrays. The native reader described above handles both versions, and is
+what a plain path or URL -- one with no ``#mode=`` fragment -- reaches.
diff --git a/docs/src/user_manual/section_indexes/general.rst b/docs/src/user_manual/section_indexes/general.rst
index f3d28824a1..219cab679b 100644
--- a/docs/src/user_manual/section_indexes/general.rst
+++ b/docs/src/user_manual/section_indexes/general.rst
@@ -26,3 +26,4 @@ Below are any pages not belonging to any other User Manual section.
../explanation/um_files_loading
../explanation/ux_guide
../explanation/which_regridder_to_use
+ ../explanation/zarr_io
diff --git a/docs/src/user_manual/tutorial/s3_io.rst b/docs/src/user_manual/tutorial/s3_io.rst
index ed5e8dca5f..c03d78de7e 100644
--- a/docs/src/user_manual/tutorial/s3_io.rst
+++ b/docs/src/user_manual/tutorial/s3_io.rst
@@ -1,7 +1,8 @@
.. tutorial:: Loading From and Saving To S3 Buckets
:tags: topic_load_save
- A lesson on using s3-fuse with Iris to load/save data from/to S3 buckets.
+ A lesson on using s3-fuse with Iris to load/save netCDF, PP and GRIB
+ data from/to S3 buckets, and on reading a Zarr store from S3 directly.
.. _s3_io:
@@ -19,14 +20,34 @@ objects. S3 access URLs can also contain a nested
`'prefix string' <https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html>`_
structure, which naturally mirrors sub-directories in a file-system.
-While it would be possible for Iris to support S3 access directly, as it does the
-"OpenDAP" protocol for netCDF data, this approach has some serious limitations : most
-notably, each supported file format would have to be separately extended to support S3
-URLs in the place of file paths for loading and saving.
+Zarr stores are accessed directly
+---------------------------------
+
+Everything below concerns **files** -- netCDF, PP, GRIB -- which Iris reaches
+through a file-system path. A :ref:`Zarr store <zarr_io_loading>` is
+different: Iris passes an ``s3://`` URL straight to zarr-python, which reads
+the objects it needs over the network::
+
+ cubes = iris.load("s3://my-bucket/data.zarr")
-Instead, we have found that it is most practical to perform this access using a virtual
-file system approach. However, one drawback is that this is best controlled *outside*
-the Python code -- see details below.
+That needs ``s3fs`` installed and configured, and no mount. If your data is in
+Zarr, prefer this to the FUSE approach described below.
+
+Other filetypes via a virtual file system
+-----------------------------------------
+
+While it would be possible for Iris to support S3 access directly, as it does the
+"OpenDAP" protocol for netCDF data, this approach has some serious limitations for
+netCDF, PP and GRIB : most notably, each supported file format would have to be
+separately extended to support S3 URLs in the place of file paths for loading and
+saving. The Zarr route above is that direct approach, made practical because
+zarr-python is already a URL-aware store library; netCDF, PP and GRIB have no
+equivalent, so that limitation still applies to them.
+
+Instead, for netCDF, PP and GRIB, we have found that it is most practical to
+perform this access using a virtual file system approach. However, one
+drawback is that this is best controlled *outside* the Python code -- see
+details below.
TL;DR
diff --git a/docs/superpowers/plans/2026-09-23-zarr-io-pr1.md b/docs/superpowers/plans/2026-09-23-zarr-io-pr1.md
new file mode 100644
index 0000000000..f193612c04
--- /dev/null
+++ b/docs/superpowers/plans/2026-09-23-zarr-io-pr1.md
@@ -0,0 +1,1279 @@
+# PR 1 — `iris.fileformats.cf` becomes a package, with tests first
+
+> **This is a frozen document.** It records the sequence agreed before the work
+> started. Programme status lives in §12 of the
+> [design spec](../specs/2026-09-21-zarr-io-design.md); nothing here duplicates
+> it. If implementation proves a step wrong, correct it here on this branch —
+> a plan is frozen when its work merges, not while the work is under way.
+
+> **For agentic workers:** REQUIRED SUB-SKILL — `superpowers:executing-plans`.
+> Native execution was chosen: one worker implements §4's five commits in
+> order, then a fresh reviewer reads the whole branch. §4 *is* the task list;
+> each commit is one task, and each carries its own two-part green check
+> (§4.0). The checkbox steps live in §12.
+
+| | |
+|---|---|
+| **Spec** | [`../specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md) §4.1, §4.3, §5 (PR 1), §6 |
+| **Implements** | The first of the seven pull requests in spec §5 |
+| **Branch** | `zarr-io-pr-1`, targeting `SciTools/iris:brownfield` |
+| **Baseline** | `brownfield` at `40a988e0e`, Iris `3.17.0.dev16` |
+| **Delivers** | The `cf` package split, and the #7259 unit test suite that protects it |
+| **Behaviour change** | None |
+| **Public API change** | None |
+
+**Created:** 2026-09-23
+
+Line references are accurate at the baseline commit. Claims marked
+**[verified]** were checked by running code or `git` against this working tree;
+everything else is read from source.
+
+**Goal:** Turn the 1721-line `iris.fileformats.cf` module into a package with a
+re-exporting `__init__.py`, and land the unit test coverage that protects the
+rewrite PR 2 will do to it — without changing any behaviour.
+
+**Architecture:** `cf.py` is cut at the three seams it already has — variable
+classifiers, the `CFGroup` mapping, the `CFReader` that drives them — into
+`_variables.py`, `_group.py` and `_reader.py`, with a strictly one-way
+dependency chain and no name needed from a module below. `__init__.py` re-exports
+all sixteen public names verbatim into `__all__`, so every existing import in
+the tree and downstream keeps working unchanged and the private file layout is
+free to move in later pull requests.
+
+**Tech Stack:** Python 3.12–3.14, pytest + pytest-xdist, pre-commit (ruff,
+numpydoc, mypy, sort-all), Sphinx with `sphinxcontrib.apidoc`, `git mv`.
+
+**Spec:** [`../specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md)
+— §4.1 is the module layout this implements, §5's first entry is this pull
+request, §6 the testing rules. The plan argues from the spec; read both.
+
+## Global Constraints
+
+Project-wide requirements that every commit below implicitly carries. Values
+are copied from the spec and the repository's agent rules, not restated from
+memory.
+
+| Constraint | Value | Source |
+|---|---|---|
+| Target branch | `brownfield`. **Not** `main` | spec §5 |
+| Labels | `Agentic` and `Type: Feature Branch` | spec §5 |
+| Issue linkage | `Part of #6977` — never a closing keyword; those live on the merge-back | spec §5 |
+| Changelog | Fragments land with the merge-back; the body must say the omission is deliberate | spec §5 |
+| This pull request | "**No behaviour change and no API change**" | spec §5, PR 1 |
+| Public surface | "Every public name that `iris.fileformats.cf` exports today is re-exported from `__init__.py`" | spec §4.1 |
+| Re-export form | "verbatim and static … name for name, gathered into `__all__`" — no renaming, no conditional imports, no logic in `__init__.py` | spec §4.1 |
+| `_variables.py` size | Stays over the ~1000-line aim. Do **not** split the `CFUGrid*` classes out to get under it | spec §4.1 |
+| Private modules | `_variables.py`, `_group.py`, `_reader.py` are a file layout, not an API | spec §4.1 |
+| Tests | "pytest style, no network, no unittest classes in new files" | spec §6 |
+| Every commit | `pre-commit` then `pytest`, both **before** the commit. Never `--no-verify` | §4.0, `AGENTS.md` |
+| Line length | 88 characters, Ruff default | `AGENTS.md` |
+| Docstrings | NumPy style, numpydoc-validated. Module docstring mandatory outside tests (D100) | `AGENTS.md` |
+| New `.py` files | Four-line copyright header, checked by `check-license-headers` | `AGENTS.md` |
+| netCDF access | No direct `import netCDF4` — always `iris.fileformats.netcdf._thread_safe_nc` | `AGENTS.md` |
+| Python | 3.12 / 3.13 / 3.14 | `AGENTS.md` |
+| Attribution | Commits end `Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>`; the pull request body ends with the Claude Code line | repository convention |
+
+## Review Focus
+
+Five failure modes the spec implies that **nothing in this pull request's test
+suite or hook set exercises**, most likely first. Each names the condition and
+the behaviour a reasonable person expects; each has a test or gate added to the
+commit that owns the code, written out in §6.4.
+
+1. **A public name left out of `cf/__init__.py` or `__all__.`** Someone writes
+ `from iris.fileformats.cf import CFMeasureVariable`, or a docstring writes
+ `:class:`~iris.fileformats.cf.CFReader`` (there are two such in
+ `docs/src/whatsnew/` alone **[verified]**), and expects it to resolve as it
+ does today. Nothing checks this: ruff's whole `F` ruleset is off so F401
+ never fires, `sort-all` only sorts what is already there, no test walks the
+ surface, and the docs build has no `-W` **[verified]** — so the symptom is a
+ silently empty API page rather than an error. → §6.4 test 1, commit 4.
+
+2. **A name the moved code still needs, left behind in the module it came
+ from.** `_reader.py` calls a helper that stayed in `_variables.py`, on a
+ branch the suite never travels; the import succeeds and the `NameError`
+ waits for a user's file. The linter that exists for exactly this — Pyflakes
+ F821 — is in the project's ignore list **[verified]**. → §6.4 gate 2,
+ commit 4. `ruff check --select F lib/iris/fileformats/cf.py` reports
+ **`All checks passed`** today **[verified]**, so the baseline is zero and
+ any finding after the cut was caused by the cut.
+
+3. **An import left behind that the moved code no longer needs.** If
+ `_variables.py` keeps `from iris.fileformats.netcdf import _bytecoding_datasets,
+ _thread_safe_nc`, every test still passes and §5.3's claim that `_reader.py`
+ is the only module coupled to `netcdf` is quietly false — which is the exact
+ coupling spec §4.2 sends PR 2 to remove. F401 is off, so nothing says so.
+ → §6.4 test 3, commit 4.
+
+4. **`import iris.fileformats.cf` as the first import in a fresh
+ interpreter.** A user or a downstream library imports the package before
+ anything has imported `iris`, and expects it to work. The suite can never
+ see this: pytest imports `iris` — and so `iris.fileformats.netcdf` — long
+ before any cf test module is collected. The split adds a hop to the
+ pre-existing `cf` ↔ `netcdf` cycle (§9.2). → §6.4 test 4, commit 4.
+
+5. **Reaching a name that is now private.** `from iris.fileformats.cf import
+ _CFFormulaTermsVariable` works today and deliberately stops working (§5.5),
+ as do `cf._is_str_dtype`, `cf._getncattr` and `cf._CF_PARSE`. The decision
+ is right — spec §4.1 says third-party code must not reach past the package —
+ but it exists only as prose, so nothing stops a later pull request
+ re-widening the surface by adding one convenient re-export. → §6.4 test 5,
+ commit 4.
+
+Five more candidates were checked and **cleared**, so a reviewer need not
+re-derive them: the wheel excludes the new subpackage (no — `packages.find`
+uses `include = ["iris*"]`, automatic discovery); sphinx-apidoc needs
+registering (no — it walks `lib/` with two unrelated exclusions); `CFReader.__del__`
+breaks at interpreter shutdown (no — it calls `self._close()`, no module
+globals); a user's `warnings.filterwarnings(module=...)` stops matching (no —
+`module` is a prefix match, so `iris.fileformats.cf` still matches
+`…cf._variables`); and the incoming suite trips over `_shared_utils.py` drift
+since #7259 forked (no — that drift is two added cartopy/PROJ helpers, nothing
+removed or re-signed). All five **[verified]**.
+
+**Two more were not anticipated and were found by check 6, the docs build,
+after everything else was green** — a Sphinx duplicate-object warning that
+fails the Read the Docs build outright, and `reference_terms` silently
+dropping off the API page. Both are fixed, both have tests, and both are
+written up in §8.4 because they will recur in PRs 2 through 7. A reviewer
+should read that section: it is the only part of this work where the plan was
+wrong about something that mattered.
+
+---
+
+## 1. What this pull request is
+
+Two things, in this order:
+
+1. **Land the unit test coverage from [#7259](https://github.com/SciTools/iris/pull/7259)**
+ (branch `cf_reader_zarr`, by @trexfeathers and @pp-mo) against today's
+ `cf.py`, unchanged. Eighteen files, +2354/-643 **[verified]**.
+2. **Split `cf.py` into the `iris.fileformats.cf` package** along the three
+ seams already in the file, behind an `__init__.py` that re-exports every
+ public name.
+
+The order is the point. The tests are written against today's behaviour by
+people who were not doing the rewrite, so when PR 2 rewrites the CF variable
+classes against `CFDataset`, the net that catches a regression did not move
+with the thing it is testing. Landing the tests in the same pull request as the
+split, rather than a separate one, is what lets a reviewer see the suite pass
+before and after the move — see §4.
+
+Neither half changes behaviour. The whole existing suite passes untouched.
+
+## 2. Scope boundary
+
+Deliberately **not** in this pull request, each with the spec section that owns
+it:
+
+| Not here | Owned by |
+|---|---|
+| `CFDataset` / `CFDatasetVariable`, and `netcdf/_dataset.py` | PR 2, spec §4.2 |
+| Rewiring `CFVariable.__getattr__` onto `.attributes` | PR 2, spec §4.3 |
+| The `_NCZARR_SCALAR_DIMENSION` gap in the three overriding `spans` methods | PR 2 |
+| Relocating `netcdf/loader.py` and `netcdf/saver.py` into the package | PRs 3 and 5 |
+| `cf/dataset.py`, `cf/loader.py`, `cf/saver.py` — the three NEW modules in the spec §4.1 tree | PRs 2, 3, 5 |
+| Anything Zarr: the dependency, `fileformats/zarr/`, lock files | PRs 4 and 6 |
+| A changelog fragment | The merge-back, spec §5 |
+
+`cf/__init__.py`, `cf/_variables.py`, `cf/_group.py` and `cf/_reader.py` are
+the only four files the package has when this pull request merges. The other
+three modules in the spec §4.1 tree arrive with the code that fills them.
+
+One thing found while planning is also out of scope, and §10 explains why:
+the dead branch at `cf.py:1663`.
+
+## 3. Findings that refine the spec
+
+Five things checked against the working tree that change what §5 said PR 1
+would do. None of them changes the design; three of them make the pull request
+smaller than the spec implies.
+
+### 3.1 No import edits are needed across the tree
+
+Spec §5 says "Imports updated across the tree". **[verified]** — they are not.
+All 67 references to the module across `lib/`, `benchmarks/` and `docs/` take
+one of three forms, and the re-exporting `__init__.py` satisfies all three
+unchanged:
+
+```python
+import iris.fileformats.cf # netcdf/loader.py:43, saver.py:64, ...
+import iris.fileformats.cf as cf # _nc_load_rules/helpers.py:38
+from iris.fileformats.cf import CFDataVariable # _nc_load_rules/engine.py:24, ...
+```
+
+The one apparent module-global access, `cf._coord_system_mappings` at
+`netcdf/loader.py:94`, is an instance attribute on a local `CFReader` named
+`cf` (assigned at `cf.py:1333`), not a module attribute. `reference_terms` is
+reached as `iris.fileformats.cf.reference_terms` from
+`_nc_load_rules/actions.py:628`, which the re-export covers.
+
+What *does* need editing is four target strings in the incoming test suite —
+§6.3.
+
+### 3.2 `__all__` in `__init__.py` is load-bearing for the API docs
+
+Not a style preference. `sphinx.ext.autodoc`'s `ModuleDocumenter` has two
+branches **[verified against sphinx 8.1.3 in `iris-dev`]**:
+
+```python
+if self.__all__ is None:
+ # for implicit module members, check __module__ to avoid
+ # documenting imported objects
+ return True, list(members.values())
+```
+
+After the split, every class's `__module__` is `iris.fileformats.cf._variables`
+or `._reader`, not `iris.fileformats.cf`. Without `__all__`, autodoc takes the
+first branch, the `__module__` check rejects all fifteen classes, and the
+`iris.fileformats.cf` API page renders **empty**. With `__all__` present,
+autodoc takes the other branch and skips only members *absent* from it, so the
+re-exports are documented and no `imported-members` option is needed.
+
+`iris.mesh` is the existing precedent for exactly this and declares `__all__`
+for the same reason. §8 makes a docs build one of the verification steps, and
+§6.4's first test pins the same property without needing one.
+
+### 3.3 The seams are clean in one direction
+
+**[verified]** by mapping every module-level name to the lines that use it:
+
+| Name | Defined | Used at | Lands in |
+|---|---|---|---|
+| `_CF_PARSE` | 44 | 612, 872 | `_variables.py` |
+| `_NCZARR_SCALAR_DIMENSION` | 56 | 180 | `_variables.py` |
+| `_CF_ATTRS_IGNORE` | 60 | 229 | `_variables.py` |
+| `reference_terms` | 64 | 1672 | `_reader.py` — see §5.4 |
+| `_is_str_dtype` | 78 | 362, 547, 781, 950, 1027, 1102 | `_variables.py` |
+| `_getncattr` | 1715 | 1448 | `_reader.py` |
+
+and the same for the imports: `hh`, `_bytecoding_datasets`, `_thread_safe_nc`,
+`Path`, `urlparse`, `Optional`, `Iterable` and `iris.exceptions` are used
+**only** below line 1279, so only `_reader.py` imports them. `MutableMapping`
+is used only by `CFGroup`, so only `_group.py` imports it. `ABCMeta`,
+`abstractmethod`, `ClassVar`, `re`, `np`, `ma`, `iris.util`, `Connectivity`
+and the two named warning classes are used only above line 1116, so they stay
+in `_variables.py`. Only `warnings` and `iris.warnings` are needed by two
+modules, and each imports them.
+
+**[corrected during execution]** an earlier draft had `Connectivity` used by
+`_group.py` at line 1219 and imported by both. That was a substring match on
+`CFUGridConnectivityVariable`; a word-boundary grep puts its only two real
+uses inside `CFUGridConnectivityVariable` itself, in `_variables.py`.
+
+The resulting dependency chain is strictly one-way —
+`_variables` ← `_group` ← `_reader` — with no module needing a name from one
+below it. **[verified]** by class reference: `_group.py` needs thirteen names,
+the twelve its `_cf_getter` properties instantiate plus `CFVariable` for the
+`isinstance` guard at cf.py:1243, and *not* `_CFFormulaTermsVariable` — its
+`formula_terms` property filters on `has_formula_terms()` rather than on type.
+`_reader.py` needs fourteen: the ten of `_variable_types` (cf.py:1291-1302),
+`CFCoordinateVariable`, `CFDataVariable` and `_CFFormulaTermsVariable` — the
+three "special cases" that tuple excludes — and `CFGroup`, for the
+`CFGroup = CFGroup` class attribute at cf.py:1304.
+
+### 3.4 #7259 rebases onto `brownfield` with one trivial conflict
+
+**[verified]**: the branch forks at `cefe4b78b`; `brownfield` has taken seven
+commits since; the intersection of the two changed-file sets is exactly one
+file, `AGENTS.md`, and the hunk there is an unrelated six-line tip about
+running pytest under `conda run`. Every one of the eighteen files applies
+clean.
+
+Two things beyond the file-set check, because a clean apply is not the same as
+a meaningful one. **`cf.py` has not changed at all since the fork**
+**[verified]** — `git log cefe4b78b..upstream/brownfield -- lib/iris/fileformats/cf.py`
+is empty — so the incoming tests were written against byte-identical code and
+pin today's behaviour rather than June's. And the two shared test files the
+suite leans on, `tests/_shared_utils.py` and `tests/conftest.py`, did move
+(+50/-3), but **additively**: two new cartopy/PROJ skip helpers, nothing
+removed and no signature changed **[verified]**. So a failure after the apply
+is a real failure, not a rebase artefact.
+
+### 3.5 The incoming suite already pins what PR 2 will move
+
+Spec §4.3 requires PR 1 to pin `cf_attrs_unused()` before PR 2 rewires
+attribute tracking. `test_CFVariable.py::TestAttributeAccess` does exactly
+that, including the sharp case —
+`test_getattr_non_ncattr_value_is_cached_but_not_marked_used` — and
+`TestSpans::test_scalar_dimension_always_true` pins the base-class
+`_NCZARR_SCALAR_DIMENSION` behaviour that PR 2 extends to the three
+overriding `spans` methods. No additional tests need writing for this pull
+request.
+
+## 4. Commit sequence
+
+Five commits, each green on its own.
+
+### 4.0 What "green" means at every commit
+
+Two gates, both run **before** the commit is made, never after:
+
+```bash
+git add <paths>
+pre-commit run --files <paths> # re-stage, repeat until clean
+pytest -n auto <targets>
+```
+
+**`pre-commit` first, and in a loop.** The hooks rewrite files in place —
+`ruff-check --fix`, `ruff-format` and `sort-all` all do — so a first run that
+reports failures and a second that passes is the normal outcome, not a problem.
+Re-stage and re-run until a full pass is clean, then commit. `AGENTS.md`
+requires this, and the ISC001/COM812 interaction in its gotcha list is exactly
+the case where a single run is not enough.
+
+**Never `--no-verify`, and never defer the hooks to the end of the branch.** A
+formatting fix applied three commits later lands in a diff that claims to be a
+pure move, which destroys the one property that makes commit 3 reviewable
+(§9.1). The hooks belong to the commit that caused them.
+
+Commit 1 is markdown under `docs/superpowers/`, which is outside the global
+`files:` regex in `.pre-commit-config.yaml` **[verified]** — every Python hook
+reports "no files to check". Run it anyway: on a commit that should touch no
+Python at all, that output *is* the check.
+
+§8.2 lists which hooks have something to say at which commit, and why.
+
+### Commit 1 — the plan
+
+```
+docs: implementation plan for the cf package split
+```
+
+This file. First so that the rest of the pull request has something to be read
+against.
+
+**Green check:**
+
+```bash
+pre-commit run --files docs/superpowers/plans/2026-09-23-zarr-io-pr1.md
+```
+
+Expect `don't commit to branch` to pass and every other hook to report "no
+files to check" **[verified]**. No pytest target — nothing executable changed.
+
+### Commit 2 — the tests, against today's `cf.py`
+
+```
+Unit test coverage for iris.fileformats.cf
+```
+
+Squash the 21 commits of `origin/cf_reader_zarr` into one, applied to today's
+flat `cf.py`:
+
+```bash
+BASE=$(git merge-base origin/cf_reader_zarr upstream/brownfield)
+git diff $BASE origin/cf_reader_zarr -- lib/iris/tests lib/iris/fileformats \
+ | git apply
+```
+
+Note the explicit pathspec: it takes the tests and the one-line `cf.py` change,
+and drops the `AGENTS.md` hunk of §3.4 — that tip is unrelated to this
+programme and would conflict for no gain.
+
+The branch's own history is a refactoring conversation between two authors
+("Cease using the term catalog", "Use correct assertion order") that carries no
+value into `brownfield`, so it is squashed. Authorship is preserved in the
+trailers instead — **[verified]** 14 commits by Martin Yeo, 7 by Patrick
+Peglar:
+
+```
+Co-authored-by: Martin Yeo <martin.yeo@metoffice.gov.uk>
+Co-authored-by: Patrick Peglar <patrick.peglar@metoffice.gov.uk>
+```
+
+The commit body cites #7259 as the source branch. The pull request body credits
+it again in prose (§9).
+
+The one non-test line in this commit is the `# TODO` comment at `cf.py:1663`;
+§10 explains what happens to it.
+
+**Green check:**
+
+```bash
+pre-commit run --files lib/iris/tests/unit/fileformats/cf/*.py \
+ lib/iris/tests/integration/test_cf.py \
+ lib/iris/fileformats/cf.py
+pytest -n auto lib/iris/tests/unit/fileformats/cf \
+ lib/iris/tests/integration/test_cf.py
+```
+
+`check-license-headers` on the twelve new files is the hook most likely to bite
+here — an incoming file without the header fails the commit, and the fix is to
+add it, not to extend the hook's exclusion list. `numpydoc-validation` excludes
+`^lib/iris/tests/` and ruff relaxes D100/D106/D205/D401 under
+`lib/iris/tests/*.py` **[verified]**, so the test modules are not held to the
+library's docstring standard.
+
+Baseline before this commit is 47 passed, 19 skipped in 2.6s **[verified]**;
+afterwards the count rises and nothing that passed before may fail.
+
+### Commit 3 — the pure rename
+
+```
+Move cf.py into a package, unchanged
+```
+
+```bash
+git mv lib/iris/fileformats/cf.py lib/iris/fileformats/cf/_variables.py
+```
+
+Plus a new `lib/iris/fileformats/cf/__init__.py` holding the re-exports and
+`__all__` of §5.5, and carrying `cf.py`'s existing module docstring — text
+unchanged, one paragraph added per §7. It keeps the
+`.. z_reference:: iris.fileformats.cf` directive, which is the docs entry point
+for the module and must land where sphinx-apidoc will look for it.
+
+`_variables.py` still holds all three seams at this point. It gets a one-line
+docstring, rewritten properly in commit 4. The only edit to its body is
+deleting the module docstring that moved to `__init__.py`.
+
+Why this is a commit on its own: `git` records it as a rename of `cf.py`, so a
+reviewer can confirm the body is untouched without reading 1721 lines. Commit
+4 then shows only the cut. **[corrected during execution]** the similarity
+index is `R099`, not the `R100` an earlier draft predicted, because the module
+docstring moves out to `__init__.py` in this same commit. The docstring move
+is the entire difference; the 1700 lines below it are byte-identical.
+
+**Green check:**
+
+```bash
+pre-commit run --files lib/iris/fileformats/cf/__init__.py \
+ lib/iris/fileformats/cf/_variables.py
+pytest -n auto lib/iris/tests
+python -c "import iris.fileformats.cf; import iris.fileformats.netcdf"
+python -c "import iris.fileformats.netcdf; import iris.fileformats.cf"
+```
+
+The two import probes are §9.2. The hooks that bite here:
+`check-license-headers` on the new `__init__.py`; ruff **D100** on both files,
+which is *not* in the ignore list outside `lib/iris/tests/*.py`
+**[verified]** — so a module without a docstring fails the commit, which is why
+`_variables.py` gets a placeholder now rather than waiting for commit 4;
+`sort-all`, which will reorder `__all__` if §5.5's order was not already
+correct; and `mypy`.
+
+Note what does **not** protect the re-exports: ruff's entire `F` ruleset is in
+the project's ignore list **[verified]**, so F401 will not flag an
+`__init__.py` import that is missing from `__all__`. Nothing in pre-commit
+catches the §3.2 failure, at this commit or any other — until commit 4 adds
+§6.4's test, the docs build is the only thing that would.
+
+### Commit 4 — the cut
+
+```
+Split the cf package into _variables, _group and _reader
+```
+
+Cut `_group.py` (source lines 1116-1279) and `_reader.py` (1280-1723, plus
+`reference_terms` from 62-74) out of `_variables.py`; redistribute the imports
+per §3.3; write the three module docstrings of §7; move the three
+`CFReader`-bound test targets of §6.3 on from `._variables` to `._reader`; add
+`test___init__.py` per §6.4.
+
+Line numbers above are `cf.py`'s, as the plan was written. In the tree as
+commit 3 leaves it, subtract 15 — the module docstring that moved to
+`__init__.py` — so the banner above `class CFGroup` is at 1101, the banner
+above `class CFReader` at 1265, and `reference_terms` at 47-59.
+
+**Green check:**
+
+```bash
+ruff check --select F lib/iris/fileformats/cf/
+pre-commit run --files lib/iris/fileformats/cf/*.py \
+ lib/iris/tests/unit/fileformats/cf/test___init__.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFReader.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFGroup.py \
+ lib/iris/tests/unit/fileformats/cf/test__CFFormulaTermsVariable.py
+pytest -n auto lib/iris/tests
+```
+
+The `ruff --select F` line is the Review Focus 2 gate of §6.4, run first
+because a name left behind by the cut makes every later result noise. It is
+deliberately outside pre-commit.
+
+This is the commit where `ruff-check --fix` does the most: the redistributed
+imports of §3.3 are re-sorted by isort rules with
+`force-sort-within-sections = true` and `known-first-party = ["iris"]`
+**[verified]**, so expect the first run to rewrite all three modules and the
+second to pass. `mypy` sees three new modules for the first time — §9.4.
+
+Then, before committing:
+
+```bash
+git diff --find-renames --stat upstream/brownfield...HEAD
+```
+
+Additions and deletions must balance to within the size of the four docstrings
+and the import redistribution. Anything larger means content was edited, not
+moved — and since the hooks have already run, a surprise here is a real change,
+not a reformat.
+
+### Commit 5 — record the pull request in the spec
+
+```
+Record PR 1 in the Zarr design spec
+```
+
+Spec §12.1 says to record the number and state "as soon as it is opened, not
+when it merges", which cannot be done before the number exists. So: open the
+pull request after commit 4, then push this commit setting the PR 1 row to
+`In review` with its link, and the header block's **Progress** line to
+`1 of 7 pull requests raised`.
+
+**Green check:** `pre-commit run --files
+docs/superpowers/specs/2026-09-21-zarr-io-design.md` — same "no files to check"
+expectation as commit 1. No pytest target.
+
+## 5. The split in detail
+
+Source line numbers are `cf.py` at the baseline commit.
+
+### 5.1 `cf/_variables.py` — ~1115 lines
+
+Source lines 44-1114: the four module constants, `_is_str_dtype`, and all
+thirteen public `CFVariable` subclasses plus the private
+`_CFFormulaTermsVariable`.
+
+```python
+from abc import ABCMeta, abstractmethod
+import re
+from typing import ClassVar
+import warnings
+
+import numpy as np
+import numpy.ma as ma
+
+from iris.mesh.components import Connectivity
+import iris.util
+import iris.warnings
+from iris.warnings import IrisCfLabelVarWarning, IrisCfMissingVarWarning
+```
+
+It sits over the ~1000-line aim in `lib/iris/AGENTS.md` and **stays there**.
+Spec §4.1 argues this at length and the argument is not reopened here: the
+three `CFUGrid*` classes are `CFVariable` subclasses with the same shape doing
+the same job as the classic ones, `CFReader._variable_types` lists all ten
+together, and `_is_str_dtype` is called from both sets. Splitting them out
+would export a private helper across a module boundary to buy a line count.
+
+### 5.2 `cf/_group.py` — ~195 lines
+
+Source lines 1116-1279: `CFGroup` alone.
+
+```python
+from collections.abc import MutableMapping
+
+from iris.mesh.components import Connectivity
+
+from ._variables import (
+ CFAncillaryDataVariable, CFAuxiliaryCoordinateVariable, CFBoundaryVariable,
+ CFClimatologyVariable, CFCoordinateVariable, CFDataVariable,
+ CFGridMappingVariable, CFLabelVariable, CFMeasureVariable,
+ CFUGridAuxiliaryCoordinateVariable, CFUGridConnectivityVariable,
+ CFUGridMeshVariable, CFVariable,
+)
+```
+
+### 5.3 `cf/_reader.py` — ~505 lines
+
+Source lines 1279-1721: `CFReader` and `_getncattr`, plus `reference_terms`.
+
+```python
+from collections.abc import Iterable
+from pathlib import Path
+from typing import Optional
+from urllib.parse import urlparse
+import warnings
+
+import iris.exceptions
+import iris.fileformats._nc_load_rules.helpers as hh
+from iris.fileformats.netcdf import _bytecoding_datasets, _thread_safe_nc
+from iris.mesh.components import Connectivity
+import iris.warnings
+
+from ._group import CFGroup
+from ._variables import (
+ CFAncillaryDataVariable,
+ CFAuxiliaryCoordinateVariable,
+ CFBoundaryVariable,
+ CFClimatologyVariable,
+ CFCoordinateVariable,
+ CFDataVariable,
+ CFGridMappingVariable,
+ CFLabelVariable,
+ CFMeasureVariable,
+ CFUGridAuxiliaryCoordinateVariable,
+ CFUGridConnectivityVariable,
+ CFUGridMeshVariable,
+ _CFFormulaTermsVariable,
+)
+```
+
+Fourteen names: the ten of `_variable_types` (cf.py:1291-1302), plus
+`CFCoordinateVariable`, `CFDataVariable` and `_CFFormulaTermsVariable` — the
+three special cases that tuple excludes — and `CFGroup` from `._group`, for the
+`CFGroup = CFGroup` class attribute at cf.py:1304. `CFVariable` itself is
+**not** needed here: `_reader.py` never names the base class **[verified]**.
+
+This is the only module in the package that imports from
+`iris.fileformats.netcdf`. That is the coupling PR 2 removes.
+
+### 5.4 Where `reference_terms` goes, and why it matters
+
+The spec does not say. It is a public name — a CF Appendix D constant table
+carrying `#:` doc comments — with exactly two consumers: `CFReader` at
+cf.py:1672, and `_nc_load_rules/actions.py:628` through the package.
+
+It goes in **`_reader.py`**, with its only in-package consumer. Co-location is
+the whole argument: the constant is a detail of how the reader promotes
+formula-term variables, nobody else in the package reads it, and a name that
+lives next to its only caller is one fewer hop for the reader of the code.
+
+**[corrected during execution]** An earlier draft of this section claimed the
+deciding argument was `test_CFReader.py:815`, on the reasoning that patching
+rebinds a module global and so must name the module the reader resolves
+against. That call is `mocker.patch.dict(...)`, not `mocker.patch(...)`, and
+the distinction is the whole of it: `patch.dict` *mutates the existing dict in
+place* rather than rebinding a name, so it reaches `CFReader` through any
+alias, including the `iris.fileformats.cf` re-export. The line needs no change
+wherever `reference_terms` lands, and the row for it in §6.3 was spurious. The
+placement decision stands on co-location alone.
+
+### 5.5 `cf/__init__.py` — ~70 lines
+
+`cf.py`'s module docstring (§7), then verbatim static re-exports gathered into
+`__all__`. No renaming, no conditional imports, no logic —
+`lib/iris/AGENTS.md`, and spec §4.1.
+
+**[corrected during execution]** "no logic" survives, but only after a
+detour. The docs build first forced a loop setting `__module__` to the package
+on every re-exported class, which broke the rule; the independent review then
+showed that loop cost more than it bought, and §8.4 records both the reasoning
+and the replacement. What the file actually ends with is a single re-statement
+of `reference_terms` carrying its doc comment — an assignment, not logic. Spec
+§4.1's "verbatim and static" still describes the imports themselves, and rules
+out re-export *tricks*: lazy attribute hooks, conditional imports,
+`__getattr__` shims. There are none.
+
+Sixteen public names:
+
+```python
+__all__ = [
+ "CFAncillaryDataVariable",
+ "CFAuxiliaryCoordinateVariable",
+ "CFBoundaryVariable",
+ "CFClimatologyVariable",
+ "CFCoordinateVariable",
+ "CFDataVariable",
+ "CFGridMappingVariable",
+ "CFGroup",
+ "CFLabelVariable",
+ "CFMeasureVariable",
+ "CFReader",
+ "CFUGridAuxiliaryCoordinateVariable",
+ "CFUGridConnectivityVariable",
+ "CFUGridMeshVariable",
+ "CFVariable",
+ "reference_terms",
+]
+```
+
+Already in the order the `sort-all` pre-commit hook will impose.
+
+`_CFFormulaTermsVariable` is **not** re-exported and **not** in `__all__`. It
+is private, and the two test modules that use it import it from
+`iris.fileformats.cf._variables` instead — §6.3.
+
+## 6. Carrying #7259
+
+### 6.1 What arrives
+
+Eighteen files under `lib/iris/tests/`, +2354/-643, twelve of them new
+**[verified]**. The shape:
+
+- `identify_mixins.py` — shared `SpansMixin`, `IdentifyByAttributeMixin` and
+ `IdentifyByAttributeListMixin` bodies, plus the `_NetCDFVar` stubs, run
+ against each `CFVariable` subclass.
+- One `test_CF*Variable.py` per class, including
+ `test__CFFormulaTermsVariable.py` for the private one.
+- `test_CFVariable.py` — the base class, including the attribute-access and
+ `cf_attrs_*` cases of §3.5.
+- Rewrites of the three existing `test_CFUGrid*` modules onto the shared
+ mixins, which is where most of the 643 deletions are.
+- `test_CFGroup.py` and `test_CFReader.py`, substantially extended.
+
+### 6.2 `tests/test_cf.py` moves to `tests/integration/`
+
+The branch relocates it and adjusts 59 lines. It is already pytest-style with
+no `unittest` classes **[verified]**, it drives real files end to end, and
+`lib/iris/tests/AGENTS.md` puts cross-component behaviour in `integration/`.
+Carried as-is.
+
+### 6.3 The targets the split breaks
+
+These are the *only* references in the tree that must change when the module
+becomes a package, and all of them are in the incoming suite. Each resolves a
+dotted prefix or a private name through `iris.fileformats.cf`, which after the
+split no longer carries it.
+
+They move in two stages, because the split happens in two commits. Commit 3
+makes `cf` a package whose whole body is `_variables.py`, so every target
+retargets to `._variables`; commit 4 cuts `_group.py` and `_reader.py` out, so
+the three targets that follow `CFReader` move again to `._reader`. The "After
+commit 3" column is what the tree looks like mid-sequence, and is worth
+stating because commit 3 must be green on its own.
+
+| File:line | Today | After commit 3 | After commit 4 |
+|---|---|---|---|
+| `test_CFReader.py:546` | `"iris.fileformats.cf._thread_safe_nc.DatasetWrapper"` | `..cf._variables._thread_safe_nc...` | `..cf._reader._thread_safe_nc...` |
+| `test_CFReader.py:614` | `"iris.fileformats.cf.hh._parse_extended_grid_mapping"` | `..cf._variables.hh...` | `..cf._reader.hh...` |
+| `test_CFReader.py:909` | `"iris.fileformats.cf.CFBoundaryVariable"` | `..cf._variables.CFBoundaryVariable` | `..cf._reader.CFBoundaryVariable` |
+| `test_CFVariable.py:96` | `cf._NCZARR_SCALAR_DIMENSION` | `_variables._NCZARR_SCALAR_DIMENSION` | unchanged |
+| `test_CFGroup.py:16,114` | `cf._CFFormulaTermsVariable` | `_variables._CFFormulaTermsVariable` | unchanged |
+| `test__CFFormulaTermsVariable.py:11` | `from iris.fileformats.cf import _CFFormulaTermsVariable` | `from iris.fileformats.cf._variables import ...` | unchanged |
+
+**[corrected during execution]** An earlier draft listed four targets and had
+`test_CFReader.py:815` among them. It is `mocker.patch.dict`, which needs no
+change — see §5.4. In its place the full-suite run at commit 3 surfaced two
+that the draft had missed, `test_CFVariable.py:96` and `test_CFReader.py:909`.
+
+`test_CFReader.py:909` is the sharp one, and the reason §6.4 exists.
+`mocker.patch` *rebinds a module global*, so it only affects `CFReader` if it
+names the module `CFReader`'s own global lookup resolves against. Targeting
+the `iris.fileformats.cf` re-export leaves the reader's binding untouched —
+and the test still passes, because its assertion holds on the unpatched path
+too. A silently-defanged test, not a failure. It carries a comment saying so.
+
+The five `mocker.patch` calls in `test_CFReader.py` that target `CFReader`
+methods (lines 160, 161, 267, 404, 472) need **no** change: they patch
+attributes on the class object, which the re-export hands over unchanged.
+
+A test reaching into `cf._reader` is reaching into the file layout, which is
+what spec §4.1 says third-party code should not do. For tests that is correct
+and unavoidable — a patch must name where the lookup happens.
+
+### 6.4 The tests the Review Focus adds
+
+#7259 tests the CF classes thoroughly and tests the *package* not at all — it
+was written before the package existed. Four of the five Review Focus items get
+one new test module, written in commit 4 because the cut is what creates the
+thing they check; the fifth is a lint gate, not a test.
+
+New file, `lib/iris/tests/unit/fileformats/cf/test___init__.py` — named for the
+module under test, `cf/__init__.py`, following the one-module-per-test-module
+convention the rest of the directory uses:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the public surface of :mod:`iris.fileformats.cf`.
+
+The package is a re-export layer over a private file layout, so what needs
+testing is the layer itself: that it exposes everything it used to, exposes
+nothing it should not, and imports cleanly on its own.
+"""
+
+import subprocess
+import sys
+from types import ModuleType
+
+import pytest
+
+import iris.fileformats.cf as cf
+from iris.fileformats.cf import _group, _reader, _variables
+
+PRIVATE_MODULES = (_variables, _group, _reader)
+
+
+def _origin(value, fallback):
+ """Return the module path a name came from, for values and modules alike."""
+ if isinstance(value, ModuleType):
+ return value.__name__
+ return getattr(value, "__module__", fallback)
+
+
+def _defined_public_names(module):
+ """Return the public names a module defines itself, ignoring imports."""
+ return {
+ name
+ for name, value in vars(module).items()
+ if not name.startswith("_")
+ and not isinstance(value, ModuleType)
+ and _origin(value, module.__name__) == module.__name__
+ }
+
+
+@pytest.mark.parametrize("module", PRIVATE_MODULES, ids=lambda m: m.__name__)
+def test_public_names_are_re_exported(module):
+ # Review Focus 1. A name defined in a private module and left out of
+ # __all__ vanishes from the API docs without any warning, because autodoc
+ # rejects it on __module__ - see the plan, section 3.2.
+ assert _defined_public_names(module) <= set(cf.__all__)
+
+
+@pytest.mark.parametrize("name", cf.__all__)
+def test_all_entries_resolve(name):
+ # Review Focus 1, the other direction: a name in __all__ that the package
+ # cannot supply breaks "from iris.fileformats.cf import *" and the docs.
+ assert hasattr(cf, name)
+
+
+@pytest.mark.parametrize("module", (_variables, _group), ids=lambda m: m.__name__)
+def test_only_the_reader_is_coupled_to_netcdf(module):
+ # Review Focus 3. Confining the netCDF import to _reader is the whole
+ # point of the split; F401 is disabled here, so a stale import left
+ # behind by the cut would otherwise go unremarked.
+ coupled = {
+ name
+ for name, value in vars(module).items()
+ if _origin(value, "").startswith("iris.fileformats.netcdf")
+ }
+ assert coupled == set()
+
+
+@pytest.mark.parametrize(
+ "first_import",
+ ["iris.fileformats.cf", "iris.fileformats.netcdf"],
+)
+def test_imports_in_a_fresh_interpreter(first_import):
+ # Review Focus 4. cf and netcdf import each other, and the split adds a
+ # hop to that cycle. pytest has already imported iris by the time this
+ # module is collected, so a new process is the only way to see it.
+ code = f"import {first_import}\nimport iris.fileformats.cf"
+ completed = subprocess.run(
+ [sys.executable, "-c", code],
+ capture_output=True,
+ text=True,
+ )
+ assert completed.returncode == 0, completed.stderr
+
+
+@pytest.mark.parametrize(
+ "name",
+ ["_CFFormulaTermsVariable", "_is_str_dtype", "_getncattr", "_CF_PARSE"],
+)
+def test_private_names_are_not_re_exported(name):
+ # Review Focus 5. These were reachable on the flat module and are
+ # deliberately not reachable on the package - spec section 4.1.
+ assert not hasattr(cf, name)
+```
+
+`subprocess` in a unit test is unusual but has precedent in this tree
+(`tests/_shared_utils.py`, `tests/stock/netcdf.py`) **[verified]**, and two
+interpreter starts is the whole cost.
+
+**Review Focus 2 is a gate, not a test.** A name left behind by the cut cannot
+be caught by a test that does not happen to travel the line. Add to commit 4,
+before committing:
+
+```bash
+ruff check --select F lib/iris/fileformats/cf/
+```
+
+Expect `All checks passed`. That is the same result the command gives on
+today's `cf.py` **[verified]**, so the baseline is zero and any finding is the
+cut's doing: F821 for a name left behind, F401 for an import left behind. This
+is a one-off command, not a change to `.pre-commit-config.yaml` — re-enabling
+`F` across the tree is a separate argument and not this pull request's.
+
+## 7. Module docstrings — the only prose in this pull request
+
+Everything else is a move. `lib/iris/AGENTS.md` requires each module to explain
+its subsystem in its docstring and to be generous about it, so the three new
+private modules each get one, and this is the part of the diff a reviewer must
+read rather than skim.
+
+- **`__init__.py`** — `cf.py`'s existing docstring verbatim, carrying the
+ `z_reference` directive and the `[CF]`/`[NUG]` references. One added
+ paragraph: that this is a re-export surface over a private file layout, and
+ that the layout is free to change.
+- **`_variables.py`** — what a CF variable classifier *is*: the
+ `cf_identity` / `cf_identities` / `identify()` contract every subclass
+ implements, that `identify` is a classmethod returning a name-to-instance
+ mapping, and that a non-conforming reference warns (`IrisCfMissingVarWarning`)
+ rather than raising. Name the reason the UGRID classes live here.
+- **`_group.py`** — that `CFGroup` is a `MutableMapping` over the classified
+ variables with one `_cf_getter` property per type, and what `promoted` means.
+- **`_reader.py`** — the two-phase build (`_build_cf_groups` then
+ `_reset`), what `_coord_system_mappings` accumulates, the file-ownership rule
+ in `__init__` (`_own_file`, and the destructor that depends on it), and that
+ this is the one module in the package still coupled to
+ `iris.fileformats.netcdf`, with a pointer to spec §4.2 for what removes it.
+
+No docstring or comment anywhere else is touched. Per `lib/iris/AGENTS.md`, do
+not document code this pull request did not otherwise change. Note that a
+module docstring here is not only house style: ruff **D100** is enforced
+outside `lib/iris/tests/*.py` **[verified]**, so a module without one does not
+reach a commit.
+
+## 8. Verification
+
+Run from an activated `iris-dev`, not `conda run` — the incoming `AGENTS.md`
+tip of §3.4 is about a real NumPy re-import failure under `conda run` with
+`pytest-cov`.
+
+```bash
+source ~/miniforge3/etc/profile.d/conda.sh && conda activate iris-dev
+export NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX"
+```
+
+### 8.1 Per commit
+
+`pre-commit` and `pytest` at every commit, per §4.0. The checks below are the
+branch-level ones — run once, when the work is complete, in addition to and not
+instead of the per-commit gates.
+
+### 8.2 Which hooks bite where
+
+Worth knowing in advance, so a hook failure reads as expected rather than as a
+surprise:
+
+| Hook | Commit | What it will say |
+|---|---|---|
+| `check-license-headers` | 2, 3, 4 | Fifteen genuinely new `.py` files need the four-line header written — twelve test modules, plus `__init__.py`, `_group.py` and `_reader.py`. `_variables.py` inherits `cf.py`'s |
+| `ruff-check --fix` (isort, I001) | 3, 4 | Rewrites the redistributed imports of §3.3; first run fails, second passes |
+| `ruff-check` (D100) | 3, 4 | A package module without a docstring fails. Relaxed for `lib/iris/tests/*.py`, not for `lib/iris/fileformats/` |
+| `ruff-format` | 2, 3, 4 | Line-length 88 on anything the cut reflowed |
+| `sort-all` | 3 | Reorders `__all__` if §5.5's order was wrong |
+| `numpydoc-validation` | 3, 4 | Package modules only; `^lib/iris/tests/` is excluded |
+| `mypy` | 3, 4 | Three new modules seen for the first time — §9.4 |
+| `codespell` | 2, 3, 4 | The four new docstrings of §7 |
+| *(nothing)* | — | **No hook checks that `__all__` is complete.** Ruff's whole `F` ruleset is ignored, so a missing re-export is invisible to pre-commit entirely. From commit 4 §6.4's first test catches it; before that, only the docs build does — §3.2, and check 6 below |
+
+### 8.3 Branch-level checks
+
+| # | Check | Passes when |
+|---|---|---|
+| 1 | `pytest -n auto lib/iris/tests/unit/fileformats/cf lib/iris/tests/integration/test_cf.py` | Green at commits 2, 3 and 4 alike |
+| 2 | `pytest -n auto lib/iris/tests` | No new failures against the baseline |
+| 3 | `pre-commit run --all-files` | Clean, and — the point of running it at branch level — **modifies nothing**. Every per-commit run of §4.0 already left its files clean, so a fix appearing here means one was skipped, and it must be folded back into the commit that caused it rather than appended as a tidy-up |
+| 4 | Import order probes — §9.2 | Both orders import clean in a fresh interpreter |
+| 5 | `python -c "from iris.fileformats.cf import CFReader, reference_terms"` and `import iris.fileformats.cf as cf; cf.CFGroup` | The public surface is reachable exactly as before |
+| 6 | `make -C docs html-noplot`, then open `docs/src/generated/api/iris.fileformats.cf.html` | All fifteen classes and `reference_terms` are listed — the §3.2 failure mode renders an empty page, and a `-W` build will not catch it. `html-noplot` skips the gallery but still runs apidoc; **do not** use `html-noapi`, which skips the very step under test |
+| 7 | `git diff --find-renames --stat upstream/brownfield...HEAD` | `cf.py → cf/_variables.py` shows as a rename; the net line change outside the tests is roughly the four docstrings |
+| 8 | `ruff check --select F lib/iris/fileformats/cf/` | `All checks passed` — the Review Focus 2 gate of §6.4. Same result as on today's `cf.py` **[verified]**, so anything it reports was caused by the cut |
+
+Check 6 is the one that is easy to skip and expensive to miss: it fails
+silently, produces no warning, and — per §8.2 — is checked by no hook at all.
+It would ship an empty API reference page. §6.4's first test now covers the
+same ground from the other side, but only for names a private module *defines*;
+the docs build is still what proves the page renders.
+
+### 8.4 What check 6 actually caught **[added during execution]**
+
+It was right to insist on it. Everything else — the full suite, every hook,
+the `ruff --select F` gate, all seven of §6.4's tests — was green, and the
+docs build found two defects that none of them could see. Both are recorded
+here because both are properties of *any* flat-module-to-package split in this
+repository, not of this one, and PRs 2 through 7 will meet them again.
+
+**One is a build failure.** `.readthedocs.yaml` sets
+`sphinx: fail_on_warning: true` and `docs/src/Makefile`'s `html` target passes
+`-W --keep-going` **[verified]**, so a warning is a broken pull request. The
+split produces exactly one:
+
+```
+_group.py:docstring of iris.fileformats.cf._group.CFGroup:1: WARNING:
+duplicate object description of iris.fileformats.cf._group.CFGroup, other
+instance in generated/api/iris.fileformats.cf
+```
+
+`CFReader` carries a `CFGroup = CFGroup` class attribute (`cf.py:1304`), so
+autodoc documents `CFGroup` twice on the page — once at module level and once
+nested under `CFReader`. Both were always there. What the split adds is that
+`CFGroup.__module__` is no longer the module being documented, so Sphinx
+emits a `canonical` registration for each, and the two collide. Note that
+`iris.mesh` does *not* demonstrate this: it re-exports classes from
+`.components` and is warning-free, because nothing there aliases a re-exported
+class as a class attribute.
+
+**The other is silent.** `reference_terms` — public, in `__all__`, carrying
+`#:` doc comments — simply **stops appearing on the API page**. Autodoc
+documents module-level data only where the module's own source assigns it; for
+an imported name it emits nothing, warns about nothing, and fails nothing.
+`__all__` does not rescue it: §3.2's finding is about classes, whose
+`__module__` check `__all__` bypasses, and a `dict` has no `__module__` at
+all. This is the §3.2 failure mode with a different mechanism and no
+mitigation, and it is worse, because the first one at least renders an empty
+page that a human would notice.
+
+**[verified]** in isolation — a fifteen-line throwaway package reproduces both
+exactly, and a flat module with the same contents and the same `__all__`
+reproduces neither.
+
+The silent one is fixed in `cf/__init__.py`, by re-stating `reference_terms`
+there with its doc comment. Ugly, and the only thing that works short of
+`autodoc_default_options` `imported-members`, which would apply to every
+module in Iris.
+
+#### The first fix for the build failure was wrong **[corrected during execution]**
+
+The duplicate was first fixed by re-pointing `__module__` at the package for
+every re-exported class, on the reasoning that `__module__` is what `repr()`
+prints and what `pickle` records, so restoring it to its pre-split value was
+the change that *preserved* behaviour. That reasoning was sound as far as it
+went, and the fix did work: the warning went, and the nested
+`CFReader.CFGroup` entry still rendered.
+
+It was caught by the independent branch review, and the cost was larger than
+the benefit. `inspect` resolves a class's source through `__module__`, so
+rewriting it sends `inspect` looking for the class body in `__init__.py`,
+where it is not:
+
+```
+CFReader __module__=iris.fileformats.cf getsource=OSError: could not find
+ class definition
+MeshXY __module__=iris.mesh.components getsource=OK
+```
+
+That breaks `inspect.getsource`, `getsourcelines`, IPython's `??` and
+source-aware debuggers — and, worse, **silently drops the `[source]` link
+from all fifteen classes on the API page** **[verified]**, because
+`viewcode` asks the documenting module for the class's tags and
+`__init__.py` defines none. Method-level links survive, so the page ends up
+inconsistent rather than obviously broken, and nothing warns: a `-W` build
+passes. One silent docs defect had been traded for another.
+
+**The fix that holds** is to leave `__module__` alone — matching `iris.mesh`,
+which is the house precedent and does not rewrite — and instead suppress the
+*duplicate entry* at its source, with `:meta private:` on the
+`CFReader.CFGroup` alias in `_reader.py`. **[verified]** by A/B in the
+fifteen-line reproduction: removing the marker restores the warning exactly,
+and with it the build succeeds, all sixteen names render, every class keeps
+its `[source]` link, and `reference_terms` keeps its doc comment.
+
+What this costs: `repr(CFReader)` now names `iris.fileformats.cf._reader`
+rather than the package, and the nested `CFReader.CFGroup` documentation
+entry — a duplicate description of an alias to an already-documented class —
+is gone. Both are smaller than what the rewrite cost, and the first is
+already the accepted trade throughout `iris.mesh`. Pickle is unaffected:
+round-tripping returns the identical class object **[verified]**, because the
+class is importable at both paths.
+
+**The general lesson, for PRs 2–7:** when a re-export package trips Sphinx,
+fix the thing Sphinx is complaining about, not `__module__`. `__module__` is
+load-bearing for `inspect` and `viewcode`, and both fail silently.
+
+And two tests in §6.4's module hold this: one asserting `inspect.getsource`
+succeeds on every public class — which fails immediately if anyone reinstates
+the rewrite — and one asserting every non-class entry in `__all__` is assigned
+in `__init__.py`'s own source. The second needs to read the source rather than
+`vars()`, which also makes `_defined_public_names` catch data members — the
+kind of name that had already slipped through once.
+
+### 8.5 The independent review **[added during execution]**
+
+A fresh-context adversarial review of the whole branch ran once the five
+commits were in place. It confirmed the things the plan most needed
+confirming — the code motion is byte-faithful, the dependency chain really is
+one-way, no public name was lost, `pre-commit` and the `ruff --select F` gate
+both pass and change nothing, and the integration-to-unit test migration lost
+no coverage — and found four things worth acting on. All four are fixed:
+
+| Finding | Disposition |
+|---|---|
+| The `__module__` rewrite breaks `inspect.getsource` and silently drops every class's `[source]` link | Replaced with `:meta private:` — see §8.4 |
+| `_variables.py`'s docstring claims "nothing in this module reads array data"; `CFCoordinateVariable.identify(monotonic=True)` does | Docstring corrected to name the one path that does, and where it is reached from |
+| `_group.py`'s docstring says `promoted` records variables "referenced by nothing"; neither promotion path means that | Docstring corrected to describe both paths |
+| `test_derived_bounds_boundary_guard_continue_branch` asserted something true either way, so it did not in fact detect the patch-target drift §6.3 warns about | Parametrised over guard-fires/guard-does-not, asserting on promotion, which is the branch's only observable. **[verified]** by re-neutering the target: the test now fails, where before it passed |
+
+It also flagged `ARCHITECTURE.md`'s stale `fileformats/cf.py` reference, now
+fixed, and a missing `timeout=` on the `subprocess.run` in §6.4's import test —
+an import deadlock is precisely what that test looks for, and without a
+timeout it would hang the suite rather than fail it. Also fixed.
+
+**One finding is deliberately not acted on here.**
+`identify_mixins.py`'s `test_warn` binds `result` and never asserts on it,
+where the pre-#7259 per-class test asserted `{} == result`. The review read
+this as a weakening introduced by the test-suite move. It is not:
+`git diff pr7259 HEAD -- lib/iris/tests/unit/fileformats/cf/identify_mixins.py`
+is **empty** **[verified]** — the file is byte-identical to #7259's. Changing
+it here would silently edit someone else's in-flight pull request and destroy
+the one property that makes commit 2 reviewable, namely that the suite arrives
+unmodified. It belongs as a comment on #7259, and that is where it goes.
+
+## 9. Risks
+
+### 9.1 A move that is not a move
+
+**The risk:** an accidental edit inside 1721 relocated lines — a lost line
+during the cut, a mangled decorator, a constant redefined in two modules.
+
+**Why it is low:** commit 3 is a `git mv` that `git` records as `R099` — the
+only difference being the module docstring moving to `__init__.py` — so only
+commit 4's cut can introduce one, and that commit moves two contiguous blocks
+with a hard boundary each (`class CFGroup` at 1116, `class CFReader` at 1279).
+
+**How we would know:** check 7 in §8 — the additions and deletions must balance.
+A block lost in the cut shows as an unexplained deletion, and the suite from
+commit 2 covers all three seams.
+
+### 9.2 Import cycles
+
+`iris.fileformats.cf` and `iris.fileformats.netcdf` already import each other
+at module level today: `cf.py:35` takes `_bytecoding_datasets` and
+`_thread_safe_nc` from `netcdf`, while `netcdf/loader.py:43`, `saver.py:64` and
+`_nc_load_rules/helpers.py:38` import `cf`. It works because the consumers
+resolve the names at call time, not import time — the one import that would
+bite, `from iris.fileformats.cf import CFBoundaryVariable` in `helpers.py:49`,
+is under `TYPE_CHECKING` **[verified]**.
+
+**What the split changes:** one more hop. `cf/__init__.py` executes
+`._variables` (which imports neither `netcdf` nor `hh` — §3.3), then `._group`,
+then `._reader`, which is where the `netcdf` import now happens. So the cycle
+is entered *later* than today, not earlier. `_variables.py` does still pull
+`iris.mesh.components`, which executes `iris.mesh/__init__.py`, which imports
+`iris.fileformats.netcdf.saver` — the same cycle `cf.py` has today at the same
+point.
+
+**How we would know:** check 4 in §8 — both import orders in a fresh
+interpreter:
+
+```bash
+python -c "import iris.fileformats.cf"
+python -c "import iris.fileformats.netcdf"
+python -c "import iris.mesh"
+python -c "import iris"
+```
+
+A cycle failure is deterministic and loud, so this is cheap insurance rather
+than a real worry.
+
+### 9.3 Rebase drift on #7259
+
+`brownfield` moves while this is in review. The §3.4 conflict surface is one
+file today; if `brownfield` takes a commit touching
+`lib/iris/tests/unit/fileformats/cf/` before this merges, re-check the
+intersection before rebasing:
+
+```bash
+BASE=$(git merge-base origin/cf_reader_zarr upstream/brownfield)
+comm -12 <(git diff --name-only $BASE origin/cf_reader_zarr | sort) \
+ <(git diff --name-only $BASE upstream/brownfield | sort)
+```
+
+### 9.4 `mypy` on the new files
+
+The pre-commit `mypy` hook runs over `lib/`. `CFVariable.__getattr__` makes
+mypy stop checking every attribute on the class and its subclasses — noted in
+`lib/iris/AGENTS.md` — so the split could surface errors that the single-module
+layout happened to hide. Not expected, since no signature changes, but check 3
+in §8 is where it would appear. Resolve any that do appear by typing the new
+module-level names, not by widening the ignore list.
+
+## 10. The dead branch at `cf.py:1663`
+
+The #7259 branch adds one non-test line, a `# TODO` comment against:
+
+```python
+for cf_var in self.cf_group.formula_terms.values():
+ if iris.FUTURE.derived_bounds:
+ if self.cf_group[cf_var.cf_name] is CFBoundaryVariable:
+ continue
+```
+
+The TODO is right, and the finding is stronger than "is this supposed to be an
+`isinstance()`?" — `CFGroup.__getitem__` yields a `CFVariable` **instance**, so
+`is CFBoundaryVariable`, which compares it to the **class**, can never be true.
+The `continue` has never executed. The guard that `iris.FUTURE.derived_bounds`
+was meant to install is absent.
+
+**This pull request does not fix it.** Spec §5 declares PR 1 "no behaviour
+change and no API change", and changing `is` to `isinstance` alters what loads
+under `iris.FUTURE.derived_bounds` — it needs its own tests, its own reviewer
+attention and a `bugfix` changelog fragment, none of which belong in a move.
+
+**What it does instead:** raise a GitHub issue against `SciTools/iris`
+describing the finding, and carry the comment forward rewritten to cite it, per
+`lib/iris/AGENTS.md`'s rule that a non-obvious finding links to its source.
+The issue is
+[#7296](https://github.com/SciTools/iris/issues/7296), raised 2026-09-23, and
+it records the provenance: the branch was introduced by #6481
+(`Pp derived bounds`, 2025-06-02) **[verified]**.
+
+```python
+if iris.FUTURE.derived_bounds:
+ # Always False: cf_group[...] yields a CFVariable instance, not the
+ # class. See https://github.com/SciTools/iris/issues/7296
+ if self.cf_group[cf_var.cf_name] is CFBoundaryVariable:
+ continue
+```
+
+One comment line in an otherwise pure-move diff, which is cheaper than a
+finding that exists only on GitHub where nobody reading the code will meet it.
+
+## 11. Opening the pull request
+
+**Before opening:** post a courtesy comment on
+[#7259](https://github.com/SciTools/iris/pull/7259) explaining that its
+coverage is being carried into this programme and where — spec §12.2 asks for
+this, and the branch's authors should hear it from us rather than from a
+notification.
+
+**Title:** `[Agentic] iris.fileformats.cf becomes a package, with tests first`
+
+**Base:** `SciTools/iris:brownfield`. Not `main`.
+
+**Labels:** `Agentic`, `Type: Feature Branch`.
+
+**Body must contain:**
+
+- `Part of #6977` — **not** `Closes`. Spec §5: a closing keyword only fires on
+ merge into the default branch, so `Closes` here would link the issue and
+ never close it. The closing keywords live on the merge-back.
+- A line stating that the changelog fragment is deferred to the merge-back, so
+ its absence does not read as an oversight. Cite the precedent: #7285 and
+ #7287 merged into `brownfield` without one, #7267 into `main` with one.
+- Credit to @trexfeathers and @pp-mo for the test suite, naming #7259 and
+ `cf_reader_zarr`.
+- A pointer to this plan and to spec §4.1 and §5.
+- The attribution trailer for Claude, per the repository's convention.
+
+**Reviewer guidance to include in the body**, because it is what makes a
+1700-line diff reviewable:
+
+> Read commit-by-commit. Commit 3 is `git mv` plus the module docstring moving
+> to `__init__.py` — `git` records it `R099`, and that docstring is the whole
+> of the 1%. Commit 4 is the cut: two contiguous blocks moved out, imports
+> redistributed, test target strings updated, four module docstrings written.
+> The docstrings are the only prose in the pull request.
+
+**After opening:** commit 5 of §4 — record the number and `In review` in spec
+§12.1, and update the header **Progress** line.
+
+## 12. Definition of done
+
+- [ ] Every commit was gated on both `pre-commit` and `pytest` before it was
+ made — §4.0. No commit was made with `--no-verify`.
+- [ ] All eight checks in §8.3 pass, including the docs build of check 6, the
+ `pre-commit run --all-files` of check 3 **modifying nothing**, and the
+ `ruff --select F` gate of check 8.
+- [ ] `cf/` holds exactly four files: `__init__.py`, `_variables.py`,
+ `_group.py`, `_reader.py`.
+- [ ] `test___init__.py` exists and all five Review Focus items are covered:
+ four by its tests, the fifth by check 8. Plus the two §8.4 defects that
+ check 6 found: one test that `inspect.getsource` still works on every
+ public class, one that every non-class `__all__` entry is assigned in
+ `__init__.py`'s own source.
+- [ ] No file outside `docs/superpowers/` documents a claim the code
+ contradicts. The independent review of §8.5 found three — two module
+ docstrings and `ARCHITECTURE.md` — and all three are fixed.
+- [ ] The docs build reports **`build succeeded.`** with no warning count at
+ all — not "succeeded, N warnings". Read the Docs fails on any warning.
+- [ ] No file outside `lib/iris/fileformats/cf/` and `lib/iris/tests/` is
+ modified, except `docs/superpowers/` for this plan and the spec update,
+ and `lib/iris/ARCHITECTURE.md`, whose one-line description of the module
+ this pull request moves would otherwise be left pointing at a path that
+ no longer exists — §8.5.
+ One pre-existing exception: `zarr-io-pr-1` already carries `a9c666016`,
+ a one-line `AGENTS.md` correction about pre-commit environments that
+ predates this work. It is unrelated to the split and is left where it is
+ rather than rewritten out; the pull request body should say so in a line.
+- [ ] The targets of §6.3 are updated and nothing else was.
+- [x] The issue of §10 is raised — [#7296](https://github.com/SciTools/iris/issues/7296)
+ — and its number is in the comment.
+- [ ] The courtesy comment is on #7259.
+- [ ] Spec §12.1 records the pull request number and state.
diff --git a/docs/superpowers/plans/2026-09-24-zarr-io-pr2.md b/docs/superpowers/plans/2026-09-24-zarr-io-pr2.md
new file mode 100644
index 0000000000..4c469ed083
--- /dev/null
+++ b/docs/superpowers/plans/2026-09-24-zarr-io-pr2.md
@@ -0,0 +1,8274 @@
+# PR 2 — `CFDataset`, and the CF variable classes rewritten against it
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use
+> `superpowers:subagent-driven-development` (recommended) or
+> `superpowers:executing-plans` to implement this plan task-by-task. Steps use
+> checkbox (`- [ ]`) syntax for tracking.
+
+> **This plan ships with its implementation.** Plan and code are delivered on
+> the same branch and in the same pull request. If implementation proves a step
+> wrong, correct the step here on the branch — the document is frozen when the
+> work merges, not while it is under way. Programme status lives in §12 of the
+> [design spec](../specs/2026-09-21-zarr-io-design.md); nothing here duplicates
+> it.
+
+| | |
+|---|---|
+| **Spec** | [`../specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md) §4.2, §4.3, §4.4, §4.5, §5 (PR 2), §6 |
+| **Implements** | The second of the seven pull requests in spec §5 |
+| **Branch** | `zarr-io-pr-2`, targeting `SciTools/iris:brownfield` |
+| **Baseline** | `brownfield` at `2cc0b01d4`, Iris `3.17.0.dev16`, `iris-test-data` at `f4a6a05`. Full suite green: `11724 passed, 66 skipped, 0 failed, 0 errors` (§6.2) |
+| **Delivers** | `cf/dataset.py`, `netcdf/_dataset.py`, and every CF attribute and netCDF-API access in `cf/`, `_nc_load_rules/` and `netcdf/` routed through them |
+| **Behaviour change** | Two, both about names that collide with a Python member: such an attribute now reaches the cube on load (F1, F13) and reaches the file on save (F13). **Not** the `spans` fix spec §5 anticipated — that gap turned out to be unreachable, so Task 7 characterises it instead (F9). Plus one new deprecation warning, on a path library code no longer takes |
+| **Public API change** | Additive only: the new public module `iris.fileformats.cf.dataset` (`CFDataset`, `CFDatasetVariable`, `TrackedAttributes`) and `CFVariable.attributes` |
+
+**Created:** 2026-09-24
+
+Line references are accurate at the baseline commit. Claims marked
+**[verified]** were checked by running code or `git` against this working tree;
+everything else is read from source.
+
+**Goal:** Give Iris a narrow, format-agnostic description of a CF-conforming
+array store — `CFDataset` / `CFDatasetVariable` — implement it once for netCDF,
+and rewrite every CF attribute read and every netCDF-only API call in the `cf`
+package, `_nc_load_rules/` and `netcdf/` to go through it, so that PR 4 can add
+a Zarr implementation and reuse the whole machine.
+
+**Architecture:** Two abstract classes in `cf/dataset.py` declare what the CF
+layer needs from a store: typed properties (`dimensions`, `shape`, `dtype`,
+`size`, `fill_value`, `chunking`) and one open-world mapping (`attributes`) for
+the CF attributes themselves. `netcdf/_dataset.py` implements them over the
+existing `_thread_safe_nc` / `_bytecoding_datasets` wrappers, absorbing the
+`ncattrs`/`getncattr`/`setncattr`/`chunking()`/`file_format`/
+`set_auto_chartostring`/`createVariable`/`createDimension` calls that are
+scattered across six modules today. `CFVariable` then holds a
+`CFDatasetVariable` instead of a `netCDF4.Variable`, resolves `__getattr__`
+against a read-tracking view of `.attributes` instead of caching into
+`__dict__`, and keeps a one-release deprecating fallback to the backing netCDF4
+object so that `cf_var.getncattr("units")` still works for downstream code.
+
+**Tech Stack:** Python 3.12–3.14, `netCDF4`, NumPy, Dask, pytest +
+pytest-xdist, pre-commit (ruff, numpydoc, mypy, sort-all), `abc.ABC`,
+`collections.abc.MutableMapping`, `types.MappingProxyType`.
+
+**Spec:** [`../specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md)
+— §4.2 defines the two abstract classes, §4.3 the `__getattr__` rewiring and
+the attribute-tracking hazard, §4.4 the `spans` gap this fixes, §4.5 the write
+proxy and the `mode`/`finalise` provisions, §5's second entry is this pull
+request, §6 the testing rules. The plan argues from the spec; read both.
+
+---
+
+## Global Constraints
+
+Project-wide requirements every task below implicitly carries. Values are
+copied from the spec and the repository's agent rules, not restated from
+memory.
+
+| Constraint | Value | Source |
+|---|---|---|
+| Target branch | `brownfield`. **Not** `main` | spec §5 |
+| Labels | `Agentic` and `Type: Feature Branch` | spec §5 |
+| Issue linkage | `Part of #6977` — never a closing keyword; those live on the merge-back | spec §5 |
+| Changelog | Fragments land with the merge-back; the pull request body must say the omission is deliberate | spec §5 |
+| This pull request | "**No behaviour change other than the `spans` fix. The suite from PR 1 must pass untouched.**" | spec §5, PR 2 |
+| `create_variable` keywords | `**encoding` is deliberate, "against the general rule about keyword passthrough, because the accepted keys are backend-specific by nature". Each implementation documents its own accepted keys, and the saver never constructs them | spec §4.2 |
+| `__getattr__` | Resolves against `self.attributes`; **no `setattr(self, name, value)` caching**; a missing key raises plain `AttributeError` | spec §4.3 |
+| Attribute tracking | "`.attributes` is therefore a tracking mapping, not a plain `dict`" — `cf_attrs_unused()` decides which attributes reach the cube | spec §4.3 |
+| netCDF-only coercion | "`_bytes_if_ascii` and `_setncattr` … That coercion stays on the netCDF path only" | spec §4.5 |
+| `write_handle()` | "is needed today, by netCDF". `NetCDFDatasetVariable` returns the write proxy — in practice `EncodedNetCDFWriteProxy`; "The base class survives as public API and as the superclass. The relocation must preserve both" | spec §4.5 |
+| Not fixed here | The `da.store(..., compute=False)` tuple-return defect, [#7291](https://github.com/SciTools/iris/issues/7291). PR 2 relocates the call site unchanged | spec §4.5 |
+| Tests | "pytest style, no network, no unittest classes in new files" | spec §6 |
+| Contract test | "The `CFDataset` contract gets one shared test body run against both implementations, so a netCDF/Zarr divergence fails loudly" | spec §6 |
+| Every commit | `pre-commit run --files <paths>` then `pytest`, both **before** the commit. Never `--no-verify` | `AGENTS.md` |
+| Test tiers | `pytest lib/iris/tests/unit/<area>/` per edit (no `-n auto` on small runs); `-n auto` only for the parent-directory and full-suite runs | `lib/iris/tests/AGENTS.md` |
+| Test data | `OVERRIDE_TEST_DATA_REPOSITORY` must be set **and verified** before trusting any run; read the skip count, not just the failures | `lib/iris/tests/AGENTS.md` |
+| Line length | 88 characters, Ruff default | `AGENTS.md` |
+| Docstrings | NumPy style, numpydoc-validated. Module docstring mandatory outside tests (D100) | `AGENTS.md` |
+| New `.py` files | Four-line copyright header, checked by `check-license-headers` | `AGENTS.md` |
+| netCDF access | No direct `import netCDF4` — always `iris.fileformats.netcdf._thread_safe_nc` | `AGENTS.md` |
+| `__getattr__` | Permitted only for "open-world file data keys forwarding to a declared Mapping", and that Mapping must be the path library code takes | `lib/iris/AGENTS.md` |
+| No string dispatch | No `getattr(obj, computed_name)` to choose behaviour; the two computed-name sites in this diff become mapping lookups | `lib/iris/AGENTS.md` |
+| New modules | Aim under ~1000 lines; cohesion wins over line count | `lib/iris/AGENTS.md` |
+| Deprecation | `iris._deprecation.warn_deprecated()`; NEP 29 schedule; mark `.. deprecated:: 3.17` | `AGENTS.md` |
+| Attribution | Commits end `Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>`; the pull request body ends with the Claude Code line | repository convention |
+
+---
+
+## Review Focus
+
+Five failure modes the spec implies that **nothing in the existing suite
+exercises**, most likely first. Each names the condition and the behaviour a
+reasonable person expects; each has a test added to the task that owns the
+code, written out in that task's own step style.
+
+1. **A file attribute whose name collides with a `CFVariable` class member.**
+ A variable carrying an attribute called `filename`, `cf_name`, `spans`,
+ `attributes` or `cf_data` is silently shadowed by the class member of that
+ name — spec §4.3 documents this as a known limitation rather than fixing it,
+ because CF reserves no namespace. Nothing today checks that the *mapping*
+ still yields the file value, which is the whole practical argument for
+ making `.attributes` the library path. The shadowing must be one-way:
+ `cf_var.filename` is the file path, `cf_var.attributes["filename"]` is the
+ file attribute, and `cf_attrs_unused()` still reports it so it reaches the
+ cube. → **Task 6, Steps 9–10**.
+
+2. **`getattr(cf_var, name, default)` and `hasattr(cf_var, name)` after the
+ rewiring.** Spec §4.3 calls these "a deliberate idiom throughout the
+ loader" and requires a miss to raise "plain `AttributeError`". Two things
+ can silently break: a `KeyError` or `TypeError` escaping `__getattr__`
+ (which makes `getattr(..., None)` raise instead of returning the default),
+ and a change in whether `hasattr` marks the attribute *used* — today it
+ does, because `hasattr` goes through `__getattr__`. If `hasattr` stops
+ tracking, every attribute probed but not read leaks onto loaded cubes.
+ → **Task 6, Steps 11–12**.
+
+3. **An attribute read again after `cf_attrs_reset()`.** Today `__getattr__`
+ caches with `setattr(self, name, value)`, and `cf_attrs_reset()` clears the
+ `_cf_attrs` set but **not** the cached instance attributes **[verified:
+ `_variables.py:202-210` and `:250`]**. So an attribute touched during
+ `CFReader._translate` and read again by the loading rules hits `__dict__`,
+ never re-enters `__getattr__`, and is reported *unused* — landing on the
+ cube. Removing the cache makes the second read re-register as used, and the
+ attribute silently stops reaching the cube. This is the single most likely
+ way this pull request breaks user-visible behaviour, and it is invisible to
+ a unit test of `CFVariable` alone. → **Task 6, Steps 13–14** (unit) and
+ **Task 11, Step 1** (`TestAttributeTrackingAcrossReset`, a real file, end
+ to end).
+
+4. **`iris.save(cube, <an open netCDF4 dataset>)` — the Xarray bridge.**
+ `Saver.__init__` duck-types its argument with
+ `hasattr(filename, "createVariable")` (`saver.py:426`) and accepts an
+ emulating object carrying `_data_array` instead of file-backed storage
+ (`saver.py:2612`, `loader.py:242`, issue #4994). Routing the saver through
+ `CFDataset` puts a wrapper between that object and the code that probes it,
+ and the probe is a `hasattr` on a private name — exactly the kind of thing a
+ wrapper swallows. A user passing their own dataset must still get their
+ arrays populated, not an empty file. → **Task 12, Steps 1–3**
+ (`test_Saver__user_dataset.py`, written green and then proved to bite).
+
+5. **An attribute value that is not ASCII-encodable, written through the
+ relocated coercion.** `_bytes_if_ascii` (`saver.py:271`) returns non-ASCII
+ `str` values *unchanged* and only encodes what round-trips as ASCII; spec
+ §4.5 says this coercion "stays on the netCDF path only". Moving it behind
+ `attributes.__setitem__` is where a plausible simplification — "just encode
+ it" — silently corrupts a UTF-8 `long_name`, or raises where today it
+ succeeds. → **Task 4, Step 10**.
+
+---
+
+## 1. Scope boundary
+
+**In scope.** Every access, in `lib/iris/fileformats/cf/`,
+`lib/iris/fileformats/_nc_load_rules/` and `lib/iris/fileformats/netcdf/`,
+that either
+
+- reads a CF attribute off a `CFVariable` by attribute syntax
+ (`getattr(cf_var, ...)`, `hasattr(cf_var, ...)`, `cf_var.units`), or
+- calls a netCDF-only API on a variable or dataset (`ncattrs`, `getncattr`,
+ `setncattr`, `chunking()`, `file_format`, `set_auto_chartostring`,
+ `createVariable`, `createDimension`, `filepath()`, `isopen()`).
+
+Spec §2.1 counts "86 `getattr(cf_*` or `hasattr(cf_*` call sites and 57 direct
+uses of netCDF-only APIs". Those counts match this tree exactly **[verified]**,
+but they are shaped by the `cf_*` naming: `netcdf/ugrid_load.py` reaches CF
+attributes through locals named `mesh_var`, `coord_var` and `connectivity_var`
+(`ugrid_load.py:230, 322-406`), and those count too. **The working definition
+is the bullet list above, not the number.**
+
+**Out of scope, deliberately.**
+
+- Moving `netcdf/loader.py` or `netcdf/saver.py`. That is PRs 3 and 5, and
+ those are specified as pure `git mv`s — which is only achievable because
+ this pull request makes both modules backend-agnostic first.
+- Anything Zarr. No `fileformats/zarr/` directory, no new dependency, no
+ lock-file change.
+- The `da.store(..., compute=False)` tuple-return defect,
+ [#7291](https://github.com/SciTools/iris/issues/7291). Spec §4.5 is explicit
+ that this programme does not fix it.
+- `_thread_safe_nc.py` and `_bytecoding_datasets.py` keep their current
+ contents and public names. `netcdf/_dataset.py` is a layer *over* them, not
+ a replacement.
+- The sixty-three grid-mapping parameter assignments in
+ `_add_grid_mapping_to_dataset` (`saver.py:2067-2272`). They bypass
+ `_bytes_if_ascii` today, so routing them through `attributes` would change
+ the CDL Iris writes for every projected file. Task 12 moves them onto a
+ named netCDF4 handle and leaves the bytes alone. See **finding F8**; Task 13
+ records it as an open question in the spec.
+
+**netCDF-only members that stay netCDF-only.** A handful of things the loader
+and saver need have no place on a format-agnostic ABC: whether a variable is a
+netCDF VLEN type, whether an emulating object carries `_data_array`, the write
+lock. These become **named members on `NetCDFDatasetVariable` /
+`NetCDFDataset`**, reached from netCDF modules through `cf_var.cf_data`. They
+are named rather than probed, which is the point. Where they live after PRs 3
+and 5 split the loader and saver is those pull requests' decision; this one
+only has to make them nameable.
+
+---
+
+## 2. Findings that refine the spec
+
+Fifteen things the spec does not settle, which the implementation has to.
+F1–F6 refine the §4.2 sketch, and none is a discovery of an error: §4.2 calls
+its own listing "deliberately small: only what the CF variable classes,
+`cf/loader.py` and `cf/saver.py` actually need", and these are what they turn
+out to need. F7–F13 are things found by reading the code the pull request
+touches, two of which change what spec §5 said this pull request would do.
+F14 and F15 are things found by *running* the suite on the baseline commit.
+Neither is this pull request's code to fix, and both will waste your time if
+you meet them without warning. F15 has since been resolved — it was the
+environment, not the repository — and resolving it is what makes the
+baseline green and this pull request's negative claim checkable at all. **Task 13 records F1–F6, F8, F9 and F13 in the
+spec itself**, so the ABC in §4.2 and the ABC in the code do not drift, and so
+the programme's decision log carries the two behaviour changes.
+
+**F1 — Tracking lives on `CFVariable`, not on `CFDatasetVariable`.** §4.2
+annotates `CFDatasetVariable.attributes` "materialised once; tracks reads",
+while §4.3 has `CFVariable.__getattr__` resolve "against `self.attributes`".
+They cannot both own the tracking state, because `CFReader` builds a *second*
+`CFVariable` over the *same* backing variable when it promotes one
+(`_reader.py:505` `CFDataVariable(cf_var_name, cf_var.cf_data)`, and
+`:518`), and today those two have independent `_cf_attrs` sets. Tracking on the
+dataset variable would make them share one, changing which attributes reach
+each cube. **Decision:** `CFDatasetVariable.attributes` is a plain
+`MutableMapping`, materialised once; `CFVariable.attributes` is a
+`TrackedAttributes` view over it, one per `CFVariable`. Behaviour is then
+identical to today's.
+
+**F2 — `CFDatasetVariable` gains `location: str`.** `CFVariable.filename`
+(`_variables.py:107-111`) is `data.group().filepath()` today, and it is
+load-bearing twice: it is `NetCDFDataProxy.path`, which appears in
+`repr(proxy)` and is therefore part of the dask array cache key
+(`_thread_safe_nc.py:368-374`), and it is `LOAD_PROBLEMS.record(filename=...)`.
+A variable must be able to answer it without a back-reference to its dataset.
+§4.2 already gives `CFDataset.location` exactly this meaning — "path or URL,
+for messages and proxies" — so the member is repeated on the variable with the
+same meaning and the existing `"<unknown_filename>"` fallback.
+
+**F3 — `CFDataset` gains `closed: bool`.** `Saver.complete()` guards on
+`self._dataset.isopen()` (`saver.py:2694`) and must keep doing so. `isopen()`
+is netCDF vocabulary, and the saver has to be backend-agnostic before PR 5 can
+`git mv` it, so the state is declared on the ABC as a property.
+
+**F4 — `create_variable`'s `dimensions` defaults to `()`.** Grid-mapping
+variables are scalar: `self._dataset.createVariable(cs.grid_mapping_name,
+np.int32)` (`saver.py:2080`) passes no dimensions at all.
+
+**F5 — `CFDatasetVariable` declares `__len__`.** `len(cf_var)` is part of
+`CFVariable`'s surface (`_variables.py:214-215`, pinned by PR 1's
+`test_getitem_and_len_delegate_to_underlying_variable`). The ABC gives it a
+concrete default of `self.shape[0]`; `NetCDFDatasetVariable` overrides it to
+delegate, preserving netCDF4's exact behaviour including on scalars.
+
+**F6 — `finalise()` is defined but not called.** Spec §4.5 requires it on the
+ABC from the start and says "For netCDF, `finalise()` is a no-op". Wiring a
+no-op into `Saver.__exit__` would be untestable speculation about an ordering
+only Zarr can exercise. PR 6 wires it, in `cf/saver.py`, where it does
+something. PR 5's `git mv` is unaffected either way.
+
+**F7 — "The suite from PR 1 must pass untouched" holds with two exceptions,
+both in `lib/iris/tests/unit/fileformats/cf/`.** PR 1's
+`test_CFVariable.py::TestAttributeAccess::test_cached` and
+`::test_getattr_non_ncattr_value_is_cached_but_not_marked_used` assert the
+`setattr(self, name, value)` caching that spec §4.3 requires this pull request
+to delete — `assert "coordinates" in cf_var.__dict__` (`test_CFVariable.py:153`)
+is the cache, named. They are rewritten in Task 6.
+
+Separately, the mock-netCDF fixtures in `test_CFVariable.py` and
+`identify_mixins.py` construct `CFVariable`s directly from a `MagicMock`
+netCDF variable. They change twice, and the two changes are deliberately kept
+apart. **Task 6** gives them a working `getncattr`, because `CFVariable`
+starts reading attributes through the netCDF4 API rather than off
+`ncattrs()`-driven `setattr` caching; the doubles are still netCDF4 doubles.
+**Task 11** wraps them in a real `NetCDFDatasetVariable`, or gives them an
+`attributes` mapping, because `cf_data` stops being a netCDF4 object at all.
+Their assertions are untouched by either.
+
+**Every other test file in PR 1's suite — the other seventeen modules, plus
+`lib/iris/tests/unit/fileformats/nc_load_rules/` and the integration suite —
+must pass byte-identically.** Task 13 verifies that by diffing a full-suite run
+against the baseline commit.
+
+**F8 — sixty-three grid-mapping parameters keep a raw netCDF4 handle.**
+`_add_grid_mapping_to_dataset` (`saver.py:2067-2272`) writes every CF
+grid-mapping parameter by Python attribute assignment —
+`cf_var_grid.longitude_of_projection_origin = ...` — not through
+`_setncattr`. So they bypass `_bytes_if_ascii`, and `crs_wkt` (`:2272`) lands
+as `NC_STRING` where `grid_mapping_name` lands as `NC_CHAR`. Routing them
+through `attributes.__setitem__` would apply the coercion and change the CDL
+of every projected file Iris writes — a real output change, hiding inside a
+refactor that promises none. **Decision:** Task 12 Step 8 binds
+`grid_variable = cf_var_grid.variable` once and renames the sixty-three
+assignments onto it, leaving what reaches the file identical. Recorded as
+open question Q8 in the spec; regularising it is its own pull request.
+
+**F9 — the `spans` gap spec §4.4 calls a latent bug is unreachable
+[verified].** §4.4 says the three `CFVariable` subclass `spans` overrides
+lack the `len(dimensions) == 1` guard that `CFVariable.spans` has, so a
+variable with `_NCZARR_SCALAR_DIMENSION` *plus* a real dimension would be
+wrongly treated as scalar. But `_NCZARR_SCALAR_DIMENSION` is NCZarr's marker
+for a variable that has *no* dimensions; it never appears alongside another
+one, in any file Iris can read. The guard can therefore never fire, and
+"fixing" it would be a change no test can distinguish from a no-op.
+**Decision:** Task 7 unifies the four implementations onto one helper —
+which is the maintainability half of what §4.4 wanted — and pins the current
+behaviour with a characterisation test, explicitly including the
+two-dimension case §4.4 imagined. The "one behaviour change" §5 budgets is
+spent on F13 instead.
+
+**F10 — `build_raw_cube` may start showing a synthesised `bounds`.** Raw
+loading copies `cf_var.cf_attrs_unused()` onto the cube. `CFReader._build`
+sets `bounds` on a promoted variable's `CFVariable` in a couple of places
+(`_reader.py`), and with the `setattr` cache gone those writes have to land
+somewhere — they land in `attributes`, where `cf_attrs_unused()` can see
+them. Task 10 Step 3 therefore routes `CFReader`'s own writes through
+`attributes` **and** marks them read, so raw cubes carry exactly what they
+carry today. The `test_cf_raw_attrs` integration tests are the check.
+
+**F11 — `CFVariable.attributes` is a snapshot, not a live view.**
+`TrackedAttributes` wraps its source without copying, which is right for a
+`dict`. It is wrong over `_NetCDFAttributes`, whose `__setitem__` calls
+`setncattr` — a write to the file. `CFReader` assigns to a `CFVariable`'s
+attributes during loading (F10), and a read-mode load must never write.
+**Decision:** `CFVariable.__init__` does
+`TrackedAttributes(dict(data.attributes), ignored=_CF_ATTRS_IGNORED)`. The
+copy is a few dozen short strings per variable and is taken once.
+
+**F12 — `_NetCDFAttributes` must cache the value as given, not as coerced
+[verified].** `__setitem__` applies `_bytes_if_ascii` on the way to
+`setncattr`, so an ASCII `str` goes to the file as `bytes`. netCDF4 hands it
+back as `str`: `setncattr(nm, b"hello")` then `getncattr(nm)` returns
+`'hello'`. If the in-memory cache stores the coerced value, a just-written
+attribute reads back as `bytes` while the same attribute read from a file
+reads back as `str` — and `saver.py` compares exactly that:
+`cf_var.attributes["formula_terms"] != formula_terms` (`:1184`) and
+`" ".join(coords)` (`:1213`). **Decision:** coerce only on the way out; cache
+what the caller gave. Task 4 Step 8 has the two-line body.
+
+**F13 — attribute names that collide with a netCDF4 Python member now reach
+the file [verified].** `_set_cf_var_attributes` guards each write with
+`if not hasattr(cf_var, name)` (`saver.py:1802`), meaning "don't clobber a
+netCDF4 member". But `setncattr("shape", ...)` is perfectly legal — verified
+against netCDF4 for `shape`, `name`, `dimensions`, `size`, `dtype`,
+`datatype` and `mask` — so the guard silently *drops* a coordinate attribute
+with one of those names instead of protecting anything. Once the write goes
+through `attributes`, `hasattr` is the wrong question anyway: a
+`NetCDFDatasetVariable` has different members from a `netCDF4.Variable`, so
+keeping `hasattr` would change *which* attributes get dropped, which is worse
+than either answer. **Decision:** Task 12 Step 9 makes it
+`if name not in cf_var.attributes:` — "don't overwrite an attribute already
+set", which is what the line is for. This is the saver-side half of the same
+defect the `__getattr__` rewrite fixes on the load side, and it is the one
+behaviour change this pull request spends its budget on.
+
+**F14 — `integration/netcdf/test_coord_systems.py`'s module fixture errors
+depending on what else is running [verified].** Its `_setup`
+(`test_coord_systems.py:171-176`) is
+
+```python
+@pytest.fixture(autouse=True, scope="module")
+def _setup(tmp_path_factory):
+ if not hasattr(tlc, "TMP_DIR"):
+ tlc.TMP_DIR = tmp_path_factory.mktemp("temp")
+ yield
+ delattr(tlc, "TMP_DIR")
+```
+
+— the `yield` is *inside* the `if`. When `tlc.TMP_DIR` is already set, the
+generator returns without yielding and all ten tests in `TestCoordSystem`
+error with `ValueError: _setup did not yield a value`. (Corrected in Task 13,
+which reproduced it: this is **deterministic on module import order**, not on
+worker scheduling. `test_coord_systems.py:18` imports
+`unit/fileformats/netcdf/loader/test_load_cubes.py` as `tlc`, and that module
+sets `TMP_DIR` as a global at import — so any run that imports the unit module
+into the same process first errors all ten, serially, with no xdist at all,
+and the reverse order does not.) Under `-n auto` it therefore depends on which
+modules a worker is handed, which is why on one and the same commit, minutes
+apart, it appears in a `-n auto unit/fileformats/ integration/` run and does
+**not** appear in a `-n auto lib/iris/tests` run. It also does **not** appear
+when `integration/netcdf/` is run serially [all three verified].
+
+**This matters more now than it looks.** Since F15 was fixed the rest of the
+suite is clean, so these ten errors are the *entire* difference between the
+tier baseline and zero — the one piece of noise left, and the only thing that
+could be mistaken for damage you did. **Decision:** out of scope, and named
+here so it is recognised rather than debugged. Take the baseline for each
+tier you intend to compare (§6.1) and it cancels out; if you would rather not
+have to think about it, run that tier serially or drop
+`integration/netcdf/test_coord_systems.py` from the selection, and both
+sides go to zero.
+
+**F15 — a stale `iris-test-data` removed exactly the coverage Task 9 needs
+[verified, and fixed before this plan was finalised].** The checkout on this
+machine was at `1696ac3`, a hundred commits behind
+`SciTools/iris-test-data` at `f4a6a05`, and the gap contained almost the
+whole `NetCDF/unstructured_grid/` set — **one** file present where there
+should be thirteen. All thirteen tests in
+`integration/netcdf/test_ugrid_load.py` failed as a result, and
+`ugrid_load.py` is what Task 9 rewrites.
+
+This is the hazard `lib/iris/tests/AGENTS.md` describes as a silent skip,
+here wearing a failure's clothes instead of a skip's and therefore *more*
+dangerous: 38 red tests look like pre-existing noise you have already agreed
+to ignore, so a real regression landing on top of them is invisible.
+
+**Resolved.** The remote was already configured, so the fix was two commands:
+
+```bash
+git -C ../iris-test-data fetch upstream
+git -C ../iris-test-data merge --ff-only upstream/master
+# now at f4a6a05; NetCDF/unstructured_grid/ holds 13 files, was 1
+```
+
+With that done **the full suite is green — 0 failed, 0 errors** (§6.2), which
+is what makes this pull request's negative claim checkable at all. §6.1 keeps
+the one-line environment proof as a precondition of Task 1, because a fresh
+clone or a different machine can land back in the old state without saying so.
+
+---
+
+## 3. File Structure
+
+### Created
+
+| File | Responsibility | Task |
+|---|---|---|
+| `lib/iris/fileformats/cf/dataset.py` | `CFDataset`, `CFDatasetVariable`, `TrackedAttributes`. Pure interface plus the one concrete mapping type they share. No netCDF import. ~220 lines | 1 |
+| `lib/iris/fileformats/netcdf/_dataset.py` | `NetCDFDataset`, `NetCDFDatasetVariable`, `_NetCDFAttributes`. The single place `ncattrs`/`getncattr`/`setncattr`/`chunking()`/`file_format`/`set_auto_chartostring`/`createVariable`/`createDimension` are called. ~400 lines | 2, 3, 4 |
+| `lib/iris/tests/unit/fileformats/cf/dataset/__init__.py` | Package marker | 1 |
+| `lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py` | Tracking semantics: what records a read and what does not | 1 |
+| `lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py` | The ABCs are abstract; the concrete defaults (`__len__`, `deprecated_netcdf_member`) behave | 1 |
+| `lib/iris/tests/unit/fileformats/cf/dataset/contract.py` | `class CFDatasetContract` — the shared test body spec §6 requires, parametrised on a `dataset` fixture the backend supplies. Not collected itself (no `Test` prefix) | 5 |
+| `lib/iris/tests/unit/fileformats/netcdf/dataset/__init__.py` | Package marker | 2 |
+| `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py` | Read surface over a real file in `tmp_path` | 2 |
+| `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py` | Open / borrow / close, dimensions, global attributes | 3 |
+| `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py` | `create_dimension`, `create_variable`, attribute coercion, `write_handle()` | 4 |
+| `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py` | `class TestNetCDFDatasetContract(CFDatasetContract)` — supplies the fixture, inherits the body | 5 |
+| `lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py` | `CFReader` builds and owns a `NetCDFDataset`; attribute tracking across `cf_attrs_reset()`, end to end on a real file | 11 |
+| `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py` | Review Focus 4. Saving into a dataset the caller opened, real and emulated. Written green, and proved to bite | 12 |
+| `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dataset.py` | `Saver` owns or borrows a `NetCDFDataset`; writes go through it; `complete()` sees a dataset closed behind its back | 12 |
+
+### Modified
+
+`lib/iris/fileformats/cf/__init__.py` is **not** touched. Spec §4.1 names
+`dataset.py` without a leading underscore, alongside `loader.py` and
+`saver.py`, so it is a public module in its own right and the documented import
+is `from iris.fileformats.cf.dataset import CFDataset` — the same shape as
+§4.8's `iris.fileformats.cf.loader.CHUNK_CONTROL`. The package `__init__.py`
+re-export layer covers the three *private* modules and stays exactly as PR 1
+left it, which also leaves PR 1's `test___init__.py` untouched.
+
+| File | Change | Task |
+|---|---|---|
+| `lib/iris/fileformats/cf/_variables.py` | `CFVariable.attributes`, the `__getattr__` rewiring, typed properties, the `cf_attrs_*` helpers onto the mapping, the deprecating fallback (Task 6); `_NCZARR_SCALAR_DIMENSION` in three `spans` overrides (Task 7); `cf_data` becomes a `CFDatasetVariable` (Task 11) | 6, 7, 11 |
+| `lib/iris/fileformats/_nc_load_rules/helpers.py` | 63 attribute reads onto `.attributes`, including the computed-name site at `:567` and the deliberately-untracked flag probe at `:1213` | 8, 11 |
+| `lib/iris/fileformats/_nc_load_rules/actions.py` | 2 attribute reads (`:179`, `:626`) | 8 |
+| `lib/iris/fileformats/netcdf/loader.py` | Attribute reads (Task 9); `chunking()`, `cf_data.dimensions`, `_FillValue`, proxy construction (Task 11) | 9, 11 |
+| `lib/iris/fileformats/netcdf/ugrid_load.py` | Attribute reads on `mesh_var`/`coord_var`/`connectivity_var`, including the computed-name site at `:406` | 9 |
+| `lib/iris/fileformats/cf/_reader.py` | Attribute reads (Task 10); dataset construction, `file_format`, `set_auto_chartostring`, deletion of `_getncattr` (Task 11) | 10, 11 |
+| `lib/iris/fileformats/netcdf/saver.py` | `Saver` builds and uses a `NetCDFDataset`; `_bytes_if_ascii`/`_setncattr` relocate; `write_handle()` replaces the direct proxy construction | 12 |
+| `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py` | Two caching tests rewritten, Review Focus 1–3 added (Task 6); doubles rewritten (Task 11) | 6, 11 |
+| `lib/iris/tests/unit/fileformats/cf/identify_mixins.py` | Stub gains `getncattr` (Task 6); doubles rewritten (Task 11). Finding F7 | 6, 11 |
+| `lib/iris/tests/unit/fileformats/cf/test_CFReader.py` | `netcdf_variable()` drives `ncattrs`/`getncattr`; the patch targets move | 11 |
+| `lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py` | `_data_variable` helper replaces seven bare `MagicMock`s (Task 9), then changes backing once (Task 11). No assertion changes | 9, 11 |
+| `lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py` | `_make`'s `cf_data` becomes a `spec=`'d `NetCDFDatasetVariable`: `chunking` a property, `is_variable_length`, `is_emulated`/`emulated_data_array` | 11 |
+| `lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_dimension_coordinate.py` | Two `Mock(spec=[])` doubles gain `is_emulated=False`; two `delattr(..., "_data_array")` lines go | 11 |
+| `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py` | `test_zlib`'s patch target and expected call; three `createvar_spy` blocks; `_check_bounds_setting` | 12 |
+| `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__lazy_stream_data.py` | `mock_var` specs `NetCDFDatasetVariable`; `is_emulated` / `emulated_data_array` / `write_handle()` | 12 |
+| `docs/superpowers/specs/2026-09-21-zarr-io-design.md` | §4.2 ABC updated for F1–F6; §4.3 says what `CFVariable.attributes` is; §12.1 PR 2 row; §12.3 gains Q8 (F8); §12.4 gains thirteen decisions (five were foreseen here; Tasks 6 and 10-12 added eight more); §12.6 history row | 13 |
+
+### Sequencing, and why it is this way
+
+`CFVariable.cf_data` changing type breaks every consumer at once. So the
+consumers are rewritten **first**, against a `CFVariable.attributes` that is
+initially backed by today's netCDF4 object (Task 6), and the type swap happens
+once, late, in Task 11. Every task in between leaves the tree green.
+
+```
+1 cf/dataset.py (new, unwired)
+2 netcdf/_dataset.py variable (new, unwired)
+3 netcdf/_dataset.py dataset read (new, unwired)
+4 netcdf/_dataset.py dataset write (new, unwired)
+5 the shared contract test (new, unwired)
+ |
+6 CFVariable.attributes + __getattr__ rewiring <- still netCDF4-backed
+ <- behaviour change 1 of 2
+7 one definition of "scalar" across four spans <- characterisation; F9
+ |
+8 _nc_load_rules/ ---.
+9 netcdf/loader.py | consumers, in any order, each green
+ netcdf/ugrid_load.py |
+10 cf/_reader.py ---'
+ |
+11 the swap: CFReader builds a NetCDFDataset; cf_data becomes CFDatasetVariable
+12 netcdf/saver.py: Saver builds and uses a NetCDFDataset
+13 spec updates, full-suite diff, pull request
+```
+
+---
+
+## 4. The interface, as implemented
+
+The single reference for every task below. Task 1 writes the first block,
+Tasks 2–4 the second. Names here are what later tasks call.
+
+```python
+# lib/iris/fileformats/cf/dataset.py
+
+class TrackedAttributes(MutableMapping[str, Any]):
+ def __init__(self, source: MutableMapping[str, Any],
+ *, ignored: Iterable[str] = ()) -> None: ...
+
+ # Recording lookups.
+ def __getitem__(self, key: str) -> Any: ... # records `key` as read
+ def __contains__(self, key: object) -> bool: ... # records a *hit* as read
+ # (`get` is Mapping's, and therefore records too.)
+
+ # Not recording.
+ def __setitem__(self, key: str, value: Any) -> None: ...
+ def __delitem__(self, key: str) -> None: ...
+ def __iter__(self) -> Iterator[str]: ...
+ def __len__(self) -> int: ...
+ def keys(self) -> KeysView[str]: ...
+ def values(self) -> ValuesView[Any]: ...
+ def items(self) -> ItemsView[str, Any]: ...
+
+ @property
+ def untracked(self) -> Mapping[str, Any]: ... # read-only, records nothing
+ @property
+ def read(self) -> frozenset[str]: ...
+ @property
+ def unread(self) -> frozenset[str]: ...
+ def reset(self) -> None: ... # back to `ignored` ∩ keys
+
+
+class CFDatasetVariable(ABC):
+ name: str # abstract property
+ location: str # abstract property (finding F2)
+ dimensions: tuple[str, ...] # abstract property
+ shape: tuple[int, ...] # abstract property
+ dtype: np.dtype # abstract property
+ size: int # abstract property
+ fill_value: Any | None # abstract property
+ chunking: tuple[int, ...] | None # abstract property; None when unchunked
+ attributes: MutableMapping[str, Any] # abstract property; materialised once
+
+ def __getitem__(self, keys) -> np.ndarray: ... # abstract
+ def __setitem__(self, keys, values) -> None: ... # abstract
+ def write_handle(self) -> Any: ... # abstract
+ def __len__(self) -> int: ... # concrete: self.shape[0]
+ def deprecated_netcdf_member(self, name: str) -> Any: ...
+ # concrete: raises AttributeError
+
+
+class CFDataset(ABC):
+ location: str # abstract property
+ mode: str # abstract property; "r"|"r+"|"a"|"w"|"w-"
+ closed: bool # abstract property (finding F3)
+ variables: Mapping[str, CFDatasetVariable] # abstract property
+ dimensions: Mapping[str, int] # abstract property
+ attributes: MutableMapping[str, Any] # abstract property
+
+ def create_dimension(self, name: str, size: int | None) -> None: ... # abstract
+ # size=None requests an unlimited dimension:
+ # saver.py:829 passes it. A store with no such
+ # concept raises.
+ def create_variable(self, name: str, dtype, dimensions=(), *,
+ fill_value=None, **encoding) -> CFDatasetVariable: ...
+ # abstract
+ def sync(self) -> None: ... # abstract
+ def finalise(self) -> None: ... # abstract; no-op for netCDF (finding F6)
+ def close(self) -> None: ... # abstract
+ def __enter__(self) -> "CFDataset": ... # concrete
+ def __exit__(self, *exc_info) -> None: ...# concrete: calls self.close()
+```
+
+```python
+# lib/iris/fileformats/netcdf/_dataset.py
+
+class _NetCDFAttributes(MutableMapping[str, Any]):
+ """A netCDF object's attributes: materialised for reads, write-through."""
+ def __init__(self, target) -> None: ... # target: VariableWrapper | DatasetWrapper
+ # __setitem__ applies _bytes_if_ascii coercion, then target.setncattr
+
+
+class NetCDFDatasetVariable(CFDatasetVariable):
+ def __init__(self, variable, location: str, *, write_lock=None) -> None: ...
+ # variable: _thread_safe_nc.VariableWrapper | _bytecoding_datasets.EncodedVariable
+ # write_lock: passed down by NetCDFDataset so that every variable of one
+ # dataset shares ONE lock. _dask_locks.get_worker_lock() returns a fresh
+ # threading.Lock under the threaded scheduler, so calling it per variable
+ # would hand out locks that exclude nothing.
+
+ @property
+ def variable(self): ... # the backing wrapper, for netCDF-only use
+ @property
+ def is_variable_length(self) -> bool: ... # netCDF VLType
+ @property
+ def is_emulated(self) -> bool: ... # backed by an object carrying _data_array
+ @property
+ def emulated_data_array(self): ... # Xarray bridge, #4994
+ @emulated_data_array.setter
+ def emulated_data_array(self, value) -> None: ...
+ # is_emulated and emulated_data_array are a pair, because the thing they
+ # replace is `hasattr(cf_var, "_data_array")` - a PRESENCE test, which a
+ # None-returning property could not reproduce: an emulator is free to
+ # initialise _data_array to None and still be emulated.
+ def deprecated_netcdf_member(self, name: str) -> Any: ... # -> backing wrapper
+ def write_handle(self): ... # -> EncodedNetCDFWriteProxy
+
+
+class NetCDFDataset(CFDataset):
+ def __init__(self, location, mode: str = "r", *, netcdf_format=None,
+ warn_legacy_format: bool = False) -> None: ...
+
+ @classmethod
+ def from_existing(cls, dataset) -> "NetCDFDataset": ... # borrowed; never closed
+
+ @property
+ def dataset(self): ... # the backing DatasetWrapper, for netCDF-only use
+ @property
+ def write_lock(self): ... # _dask_locks.get_worker_lock(location), one per dataset
+```
+
+---
+
+## 5. Tasks
+
+### 5.0 The green check every task ends with
+
+Before **every** commit, in this order. No task below repeats it; it is assumed.
+
+```bash
+# 1. Confirm the test-data path took effect. A missing repository SKIPS tests
+# silently, so an unset variable removes coverage without failing anything;
+# a *stale* one fails them loudly and looks like noise (F15).
+echo "${OVERRIDE_TEST_DATA_REPOSITORY:?set OVERRIDE_TEST_DATA_REPOSITORY first}"
+ls "$OVERRIDE_TEST_DATA_REPOSITORY/NetCDF/unstructured_grid/" | wc -l # 13
+
+# 2. Hooks rewrite files in place: re-stage and re-run until clean.
+pre-commit run --files <the files this task touched>
+
+# 3. The tier the task's own steps name. Never --no-verify on the commit.
+```
+
+Read the **skip count**, not only the failure count. A skip is neither a pass
+nor a failure, so a `FAILED|ERROR` diff cannot see coverage that silently
+stopped running. §6.2 gives the baseline skip count for every selection the
+tasks below use; "green" means matching it, not beating it.
+
+Two pieces of expected noise, so you recognise them rather than debug them:
+ten `test_coord_systems.py` errors whenever `integration/` is run under
+`-n auto` (F14), and one skip apiece in `unit/fileformats/cf/` and
+`unit/fileformats/netcdf/` (§6.2). Everything else is green at baseline.
+
+---
+
+### Task 1: `cf/dataset.py` — the interface and its tracking mapping
+
+**Files:**
+- Create: `lib/iris/fileformats/cf/dataset.py`
+- Create: `lib/iris/tests/unit/fileformats/cf/dataset/__init__.py`
+- Test: `lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py`
+- Test: `lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py`
+
+**Interfaces:**
+- Consumes: nothing. This is the first task.
+- Produces: everything in §4's first code block. Later tasks rely on
+ `TrackedAttributes(source, *, ignored=())` with `.untracked`, `.read`,
+ `.unread`, `.reset()`; `CFDatasetVariable` with abstract `name`, `location`,
+ `dimensions`, `shape`, `dtype`, `size`, `fill_value`, `chunking`,
+ `attributes`, `__getitem__`, `__setitem__`, `write_handle()` and concrete
+ `__len__`, `deprecated_netcdf_member(name)`; `CFDataset` with abstract
+ `location`, `mode`, `closed`, `variables`, `dimensions`, `attributes`,
+ `create_dimension(name, size)`,
+ `create_variable(name, dtype, dimensions=(), *, fill_value=None, **encoding)`,
+ `sync()`, `finalise()`, `close()` and concrete `__enter__`/`__exit__`.
+
+- [ ] **Step 1: Create the test package marker**
+
+```bash
+mkdir -p lib/iris/tests/unit/fileformats/cf/dataset
+cat > lib/iris/tests/unit/fileformats/cf/dataset/__init__.py <<'EOF'
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.cf.dataset`."""
+EOF
+```
+
+- [ ] **Step 2: Write the failing tests for `TrackedAttributes`**
+
+Create `lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.dataset.TrackedAttributes`."""
+
+import pytest
+
+from iris.fileformats.cf.dataset import TrackedAttributes
+
+# The set CFVariable seeds tracking with: attributes netCDF4 handles itself,
+# and which are therefore "used" before anyone reads them.
+IGNORED = ("_FillValue", "add_offset", "missing_value", "scale_factor")
+
+
+@pytest.fixture
+def source():
+ return {"units": "K", "standard_name": "air_temperature", "_FillValue": -999}
+
+
+@pytest.fixture
+def tracked(source):
+ return TrackedAttributes(source, ignored=IGNORED)
+
+
+class TestWhatRecordsARead:
+ def test_starts_with_only_the_present_ignored_names(self, tracked):
+ # "scale_factor" is ignored but absent, so it is not seeded.
+ assert tracked.read == frozenset(["_FillValue"])
+ assert tracked.unread == frozenset(["units", "standard_name"])
+
+ def test_getitem_records(self, tracked):
+ assert tracked["units"] == "K"
+ assert tracked.read == frozenset(["_FillValue", "units"])
+ assert tracked.unread == frozenset(["standard_name"])
+
+ def test_get_records(self, tracked):
+ assert tracked.get("units") == "K"
+ assert "units" in tracked.read
+
+ def test_contains_records_a_hit(self, tracked):
+ # hasattr(cf_var, name) goes through __getattr__ today and marks the
+ # attribute used; the mapping form has to do the same.
+ assert "units" in tracked
+ assert "units" in tracked.read
+
+ def test_contains_does_not_record_a_miss(self, tracked):
+ assert "nonesuch" not in tracked
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_getitem_of_a_missing_key_raises_before_recording(self, tracked):
+ with pytest.raises(KeyError, match="nonesuch"):
+ tracked["nonesuch"]
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_get_of_a_missing_key_records_nothing(self, tracked):
+ assert tracked.get("nonesuch") is None
+ assert tracked.read == frozenset(["_FillValue"])
+
+
+class TestWhatDoesNotRecordARead:
+ def test_untracked_getitem(self, tracked):
+ assert tracked.untracked["units"] == "K"
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_untracked_contains(self, tracked):
+ # The form helpers.py needs for its deliberately-unmarked flag probe.
+ assert "units" in tracked.untracked
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_untracked_is_read_only(self, tracked):
+ with pytest.raises(TypeError):
+ tracked.untracked["units"] = "m"
+
+ def test_iteration(self, tracked):
+ assert sorted(tracked) == ["_FillValue", "standard_name", "units"]
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_keys_values_items(self, tracked):
+ assert sorted(tracked.keys()) == ["_FillValue", "standard_name", "units"]
+ assert sorted(tracked.values(), key=str) == [-999, "K", "air_temperature"]
+ assert dict(tracked.items())["units"] == "K"
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_len(self, tracked):
+ assert len(tracked) == 3
+ assert tracked.read == frozenset(["_FillValue"])
+
+
+class TestMutation:
+ def test_setitem_writes_through_and_does_not_record(self, tracked, source):
+ tracked["comment"] = "hello"
+ assert source["comment"] == "hello"
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_delitem_writes_through_and_forgets_the_read(self, tracked, source):
+ _ = tracked["units"]
+ del tracked["units"]
+ assert "units" not in source
+ assert tracked.read == frozenset(["_FillValue"])
+
+
+class TestReset:
+ def test_reset_returns_to_the_present_ignored_names(self, tracked):
+ _ = tracked["units"]
+ _ = tracked["standard_name"]
+ assert tracked.read == frozenset(["_FillValue", "units", "standard_name"])
+ tracked.reset()
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_reset_sees_attributes_added_since_construction(self, tracked):
+ tracked["scale_factor"] = 2.0
+ tracked.reset()
+ assert tracked.read == frozenset(["_FillValue", "scale_factor"])
+
+
+class TestEmptySource:
+ def test_a_variable_with_no_attributes_is_not_an_error(self):
+ tracked = TrackedAttributes({}, ignored=IGNORED)
+ assert tracked.read == frozenset()
+ assert tracked.unread == frozenset()
+ assert len(tracked) == 0
+ assert "units" not in tracked
+```
+
+- [ ] **Step 3: Run the tests to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py`
+Expected: collection error —
+`ModuleNotFoundError: No module named 'iris.fileformats.cf.dataset'`
+
+- [ ] **Step 4: Write `TrackedAttributes`**
+
+Create `lib/iris/fileformats/cf/dataset.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The format-agnostic description of a CF-conforming array store.
+
+:class:`CFDataset` and :class:`CFDatasetVariable` are what the rest of the CF
+layer is written against. They are deliberately small: only what the
+:class:`~iris.fileformats.cf.CFVariable` classes, the CF loader and the CF
+saver actually need, with one implementation per storage format
+(:mod:`iris.fileformats.netcdf._dataset` today, Zarr next).
+
+The split that matters is ``attributes`` versus everything else. A netCDF
+variable presents ``units`` - a CF attribute read from the file - and
+``dimensions`` - a property of the storage - through the same attribute syntax,
+and a caller cannot tell which is which. Here, CF attributes are only ever
+reached through ``attributes``, and storage properties are named, typed
+members.
+
+``attributes`` is an open-ended set of data keys read from a file, with no
+schema to enumerate, which is why :class:`TrackedAttributes` exists and why
+:meth:`~iris.fileformats.cf.CFVariable.__getattr__` is allowed to forward to
+it. Tracking is the load-bearing part: Iris decides which file attributes
+survive onto a loaded cube by asking which ones the loading rules did *not*
+read, so a read that goes unrecorded silently leaks a CF-reserved attribute
+onto the cube.
+
+See sections 4.2, 4.3 and 4.5 of
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+from abc import ABC, abstractmethod
+from collections.abc import (
+ ItemsView,
+ Iterable,
+ Iterator,
+ KeysView,
+ Mapping,
+ MutableMapping,
+ ValuesView,
+)
+from types import MappingProxyType
+from typing import Any
+
+import numpy as np
+
+
+class TrackedAttributes(MutableMapping):
+ """A variable's attributes, recording which of them have been looked up.
+
+ Single-key lookups record: :meth:`__getitem__`, :meth:`get` and
+ :meth:`__contains__` - the last because ``hasattr(cf_var, name)`` marks an
+ attribute used today, by reaching ``__getattr__``. Bulk access does not:
+ iteration, :meth:`keys`, :meth:`values`, :meth:`items` and :func:`len`
+ leave the record alone, and :attr:`untracked` is the explicit escape hatch
+ for a single-key lookup that must not count.
+
+ """
+
+ def __init__(self, source: MutableMapping, *, ignored: Iterable[str] = ()):
+ """Wrap ``source``, treating any ``ignored`` name it holds as already read."""
+ self._source = source
+ self._ignored = frozenset(ignored)
+ self._read: set[str] = set()
+ self.reset()
+
+ def __getitem__(self, key: str) -> Any:
+ """Return ``key``'s value, recording it as read."""
+ # Index first: a KeyError must escape without recording anything.
+ value = self._source[key]
+ self._read.add(key)
+ return value
+
+ def __contains__(self, key: object) -> bool:
+ """Return whether ``key`` is present, recording a hit as read."""
+ result = key in self._source
+ if result:
+ self._read.add(key) # type: ignore[arg-type]
+ return result
+
+ def __setitem__(self, key: str, value: Any) -> None:
+ """Set ``key``'s value, recording nothing."""
+ self._source[key] = value
+
+ def __delitem__(self, key: str) -> None:
+ """Remove ``key``, and any record of it having been read."""
+ del self._source[key]
+ self._read.discard(key)
+
+ def __iter__(self) -> Iterator[str]:
+ """Return an iterator over the attribute names, recording nothing."""
+ return iter(self._source)
+
+ def __len__(self) -> int:
+ """Return the number of attributes, recording nothing."""
+ return len(self._source)
+
+ def keys(self) -> KeysView:
+ """Return a view of the attribute names, recording nothing."""
+ return self._source.keys()
+
+ def values(self) -> ValuesView:
+ """Return a view of the attribute values, recording nothing."""
+ return self._source.values()
+
+ def items(self) -> ItemsView:
+ """Return a view of the attribute pairs, recording nothing."""
+ return self._source.items()
+
+ def __repr__(self) -> str:
+ """Return a string representation, recording nothing."""
+ return f"{self.__class__.__name__}({dict(self._source)!r})"
+
+ @property
+ def untracked(self) -> Mapping:
+ """A read-only view of the same attributes that records nothing."""
+ return MappingProxyType(dict(self._source))
+
+ @property
+ def read(self) -> frozenset:
+ """The names looked up since construction or the last :meth:`reset`."""
+ return frozenset(self._read)
+
+ @property
+ def unread(self) -> frozenset:
+ """The names present but not looked up since the last :meth:`reset`."""
+ return frozenset(self._source) - self._read
+
+ def reset(self) -> None:
+ """Forget every recorded lookup, re-seeding from the ignored names."""
+ self._read = set(self._ignored) & set(self._source)
+```
+
+Note `untracked` returns a proxy over a **copy**: `MappingProxyType` needs a
+real mapping, and the backing `_NetCDFAttributes` of Task 2 is not a `dict`.
+Callers use it for point lookups, never to observe later mutation.
+
+- [ ] **Step 5: Run the tests to verify they pass**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py`
+Expected: PASS — every test in the file, 0 failed, 0 skipped.
+
+- [ ] **Step 6: Write the failing tests for the two abstract classes**
+
+Create `lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.dataset.CFDataset` and friends."""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+
+
+class MinimalVariable(CFDatasetVariable):
+ """The smallest thing that satisfies CFDatasetVariable."""
+
+ name = "air_temperature"
+ location = "<memory>"
+ dimensions = ("time", "lat")
+ shape = (3, 4)
+ dtype = np.dtype("f4")
+ size = 12
+ fill_value = None
+ chunking = None
+ attributes: dict = {}
+
+ def __getitem__(self, keys):
+ return np.zeros(self.shape, dtype=self.dtype)[keys]
+
+ def __setitem__(self, keys, values):
+ raise NotImplementedError
+
+ def write_handle(self):
+ return self
+
+
+class MinimalDataset(CFDataset):
+ """The smallest thing that satisfies CFDataset."""
+
+ location = "<memory>"
+ mode = "r"
+ closed = False
+ variables: dict = {}
+ dimensions: dict = {}
+ attributes: dict = {}
+
+ def __init__(self):
+ self.closes = 0
+
+ def create_dimension(self, name, size):
+ raise NotImplementedError
+
+ def create_variable(self, name, dtype, dimensions=(), *, fill_value=None, **kw):
+ raise NotImplementedError
+
+ def sync(self):
+ pass
+
+ def finalise(self):
+ pass
+
+ def close(self):
+ self.closes += 1
+
+
+class TestAbstractness:
+ def test_variable_cannot_be_instantiated(self):
+ with pytest.raises(TypeError, match="abstract"):
+ CFDatasetVariable()
+
+ def test_dataset_cannot_be_instantiated(self):
+ with pytest.raises(TypeError, match="abstract"):
+ CFDataset()
+
+ @pytest.mark.parametrize(
+ "name",
+ [
+ "name",
+ "location",
+ "dimensions",
+ "shape",
+ "dtype",
+ "size",
+ "fill_value",
+ "chunking",
+ "attributes",
+ "ndim",
+ "__getitem__",
+ "__setitem__",
+ "write_handle",
+ ],
+ )
+ def test_variable_declares_member(self, name):
+ assert name in CFDatasetVariable.__abstractmethods__ or hasattr(
+ CFDatasetVariable, name
+ )
+
+ @pytest.mark.parametrize(
+ "name",
+ [
+ "location",
+ "mode",
+ "closed",
+ "variables",
+ "dimensions",
+ "attributes",
+ "create_dimension",
+ "create_variable",
+ "sync",
+ "finalise",
+ "close",
+ ],
+ )
+ def test_dataset_declares_member(self, name):
+ assert name in CFDataset.__abstractmethods__ or hasattr(CFDataset, name)
+
+
+class TestConcreteDefaults:
+ def test_ndim(self):
+ assert MinimalVariable().ndim == 2
+
+ def test_len_is_the_leading_dimension(self):
+ assert len(MinimalVariable()) == 3
+
+ def test_len_of_a_scalar_matches_netcdf4(self):
+ # netCDF4.Variable raises TypeError, and CFVariable.__len__ forwards
+ # to it today, so anything catching that keeps working.
+ class Scalar(MinimalVariable):
+ shape = ()
+
+ with pytest.raises(TypeError, match="unsized"):
+ len(Scalar())
+
+ def test_deprecated_netcdf_member_raises_attribute_error(self):
+ # A store with no backing netCDF4 object - Zarr, say - has no fallback
+ # to offer, so the compatibility route simply does not apply.
+ with pytest.raises(AttributeError, match="getncattr"):
+ MinimalVariable().deprecated_netcdf_member("getncattr")
+
+ def test_context_manager_closes(self):
+ dataset = MinimalDataset()
+ with dataset as entered:
+ assert entered is dataset
+ assert dataset.closes == 0
+ assert dataset.closes == 1
+
+ def test_context_manager_closes_on_exception(self):
+ dataset = MinimalDataset()
+ with pytest.raises(ValueError, match="boom"):
+ with dataset:
+ raise ValueError("boom")
+ assert dataset.closes == 1
+```
+
+- [ ] **Step 7: Run the tests to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py`
+Expected: collection error —
+`ImportError: cannot import name 'CFDataset' from 'iris.fileformats.cf.dataset'`
+
+- [ ] **Step 8: Write the two abstract classes**
+
+Append to `lib/iris/fileformats/cf/dataset.py`:
+
+```python
+class CFDatasetVariable(ABC):
+ """One named array in a CF-conforming dataset."""
+
+ @property
+ @abstractmethod
+ def name(self) -> str:
+ """The variable's name within its dataset."""
+
+ @property
+ @abstractmethod
+ def location(self) -> str:
+ """The path or URL of the dataset holding this variable.
+
+ Repeated from :attr:`CFDataset.location` with the same meaning, so that
+ a variable can label a message or a data proxy without a reference back
+ to its dataset.
+
+ """
+
+ @property
+ @abstractmethod
+ def dimensions(self) -> tuple:
+ """The names of the dimensions this variable spans, in order."""
+
+ @property
+ @abstractmethod
+ def shape(self) -> tuple:
+ """The variable's shape."""
+
+ @property
+ @abstractmethod
+ def dtype(self) -> np.dtype:
+ """The variable's stored data type.
+
+ Usually a :class:`numpy.dtype`. netCDF's variable-length string type
+ reports the builtin :class:`str` instead, and
+ :func:`iris.fileformats.netcdf.loader._get_cf_var_data` branches on
+ that, so it is passed through rather than normalised.
+
+ """
+
+ @property
+ @abstractmethod
+ def size(self) -> int:
+ """The total number of elements in the variable."""
+
+ @property
+ @abstractmethod
+ def fill_value(self) -> Any:
+ """The store's own fill value, or ``None`` if it has none."""
+
+ @property
+ @abstractmethod
+ def chunking(self) -> tuple | None:
+ """The variable's storage chunk shape, or ``None`` when unchunked."""
+
+ @property
+ @abstractmethod
+ def attributes(self) -> MutableMapping:
+ """The variable's CF attributes, materialised once at construction."""
+
+ @abstractmethod
+ def __getitem__(self, keys) -> np.ndarray:
+ """Return the indexed portion of the variable's data."""
+
+ @abstractmethod
+ def __setitem__(self, keys, values) -> None:
+ """Write ``values`` into the indexed portion of the variable."""
+
+ @abstractmethod
+ def write_handle(self) -> Any:
+ """Return a picklable object supporting ``__setitem__`` on this variable.
+
+ This is what a Dask worker receives as a ``da.store`` target, so it must
+ survive pickling and must remain usable after the dataset it came from
+ has been closed.
+
+ """
+
+ @property
+ def ndim(self) -> int:
+ """The number of dimensions the variable spans."""
+ return len(self.shape)
+
+ def __len__(self) -> int:
+ """Return the length of the variable's leading dimension."""
+ if not self.shape:
+ # netCDF4.Variable.__len__ raises exactly this for a scalar, and
+ # CFVariable.__len__ forwards to it today.
+ msg = "len() of unsized object"
+ raise TypeError(msg)
+ return self.shape[0]
+
+ def deprecated_netcdf_member(self, name: str) -> Any:
+ """Return a netCDF4-only member of the object backing this variable.
+
+ The one-release compatibility route for code that reached netCDF4 API
+ through a :class:`~iris.fileformats.cf.CFVariable`. A store with no
+ backing netCDF4 object has nothing to offer and raises, which is the
+ correct answer rather than a special case.
+
+ """
+ raise AttributeError(name)
+
+
+class CFDataset(ABC):
+ """A CF-conforming array store, open for reading or writing."""
+
+ @property
+ @abstractmethod
+ def location(self) -> str:
+ """The store's path or URL, for messages and data proxies."""
+
+ @property
+ @abstractmethod
+ def mode(self) -> str:
+ """The mode the store was opened in: ``r``, ``r+``, ``a``, ``w`` or ``w-``."""
+
+ @property
+ @abstractmethod
+ def closed(self) -> bool:
+ """Whether :meth:`close` has already released the store."""
+
+ @property
+ @abstractmethod
+ def variables(self) -> Mapping:
+ """The store's variables, by name."""
+
+ @property
+ @abstractmethod
+ def dimensions(self) -> Mapping:
+ """The store's dimension lengths, by name."""
+
+ @property
+ @abstractmethod
+ def attributes(self) -> MutableMapping:
+ """The store's global attributes."""
+
+ @abstractmethod
+ def create_dimension(self, name: str, size: int | None) -> None:
+ """Declare a dimension of the given length.
+
+ ``size=None`` requests an unlimited dimension. A store with no such
+ concept raises :class:`NotImplementedError`.
+
+ """
+
+ @abstractmethod
+ def create_variable(
+ self, name: str, dtype, dimensions=(), *, fill_value=None, **encoding
+ ) -> "CFDatasetVariable":
+ """Create and return a new variable.
+
+ ``**encoding`` is storage-specific by nature - ``zlib``, ``complevel``
+ and ``chunksizes`` for netCDF; ``compressors``, ``chunks`` and
+ ``shards`` for Zarr - so each implementation documents the keys it
+ accepts, and generic CF code never constructs them.
+
+ """
+
+ @abstractmethod
+ def sync(self) -> None:
+ """Flush buffered writes to the store."""
+
+ @abstractmethod
+ def finalise(self) -> None:
+ """Perform the store's one-shot completion step, after every write.
+
+ Deliberately **not** part of :meth:`close`: a worker that writes one
+ slab of a store must close its handle without performing a completion
+ step that only one process may perform.
+
+ """
+
+ @abstractmethod
+ def close(self) -> None:
+ """Release the store's resources."""
+
+ def __enter__(self) -> "CFDataset":
+ """Return this dataset, for use as a context manager."""
+ return self
+
+ def __exit__(self, *exc_info) -> None:
+ """Close this dataset on leaving the context, whatever happened in it."""
+ self.close()
+```
+
+- [ ] **Step 9: Run the tests to verify they pass**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/dataset/`
+Expected: PASS — every test in both files, 0 failed, 0 skipped.
+
+- [ ] **Step 10: Confirm the new module is importable standalone and adds no coupling**
+
+```bash
+python -c "
+import subprocess, sys
+src = open('lib/iris/fileformats/cf/dataset.py').read()
+assert 'netcdf' not in src and 'netCDF4' not in src, 'cf/dataset.py must not know netCDF'
+print('ok')
+"
+python -c "from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable, TrackedAttributes; print('ok')"
+```
+Expected: `ok` twice.
+
+- [ ] **Step 11: Green check and commit**
+
+```bash
+pre-commit run --files lib/iris/fileformats/cf/dataset.py \
+ lib/iris/tests/unit/fileformats/cf/dataset/__init__.py \
+ lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py \
+ lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py
+pytest lib/iris/tests/unit/fileformats/cf/
+git add lib/iris/fileformats/cf/dataset.py lib/iris/tests/unit/fileformats/cf/dataset/
+git commit -m "$(cat <<'EOF'
+Add the CFDataset interface and its tracking attributes mapping
+
+cf/dataset.py declares what the CF layer needs from a storage format:
+typed properties for the storage, and one open-world mapping for the CF
+attributes. TrackedAttributes records which attributes were looked up,
+which is how Iris decides what survives onto a loaded cube.
+
+Nothing is wired to it yet.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 2: `netcdf/_dataset.py` — the variable, read surface
+
+**Files:**
+- Create: `lib/iris/fileformats/netcdf/_dataset.py`
+- Create: `lib/iris/tests/unit/fileformats/netcdf/dataset/__init__.py`
+- Create: `lib/iris/tests/unit/fileformats/netcdf/dataset/conftest.py`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py`
+
+**Interfaces:**
+- Consumes: `CFDatasetVariable` from Task 1 — every abstract member listed
+ there, plus the concrete `__len__` and `deprecated_netcdf_member`.
+- Produces:
+ - `_NetCDFAttributes(target)` — a `MutableMapping`; `target` is anything with
+ `ncattrs()`, `getncattr()`, `setncattr()` and `delncattr()`, i.e. a
+ `VariableWrapper`, an `EncodedVariable` or a `DatasetWrapper`. Task 3 uses
+ it for global attributes; Task 4 adds the write-side coercion.
+ - `NetCDFDatasetVariable(variable, location, *, write_lock=None)` with, on
+ top of the ABC, `variable`, `is_variable_length`, `is_emulated`,
+ `emulated_data_array` (get/set). Tasks 11 and 12 consume all of these.
+ - the pytest fixtures `sample_path`, and the module constants
+ `SAMPLE_DIMENSIONS` / `SAMPLE_GLOBALS`, used by Tasks 3, 4 and 5.
+
+- [ ] **Step 1: Create the test package and its sample file**
+
+`lib/iris/tests/unit/fileformats/netcdf/dataset/__init__.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.netcdf._dataset`."""
+```
+
+`lib/iris/tests/unit/fileformats/netcdf/dataset/conftest.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""A real netCDF4 file to test the netCDF CFDataset implementation against.
+
+Real, rather than mocked, because the members being implemented are precisely
+the ones whose netCDF4 behaviour is easy to misremember - what ``chunking()``
+returns for a contiguous variable, what ``dimensions`` is for char data, what
+``ncattrs()`` includes once ``fill_value=`` has been passed.
+"""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.netcdf import _thread_safe_nc
+
+#: How the sample file's dimensions are created. None means unlimited.
+SAMPLE_DIMENSION_SIZES = {"time": 3, "lat": 4, "nchars": 8, "record": None}
+
+#: How they should then read back. An unlimited dimension reports the number
+#: of records actually written, which here is none.
+SAMPLE_DIMENSIONS = {"time": 3, "lat": 4, "nchars": 8, "record": 0}
+
+#: The sample file's global attributes.
+SAMPLE_GLOBALS = {"Conventions": "CF-1.7", "title": "sample"}
+
+#: The sample file's air_temperature payload.
+SAMPLE_AIR = np.arange(12, dtype="f4").reshape(3, 4)
+
+#: The sample file's label payload, as the strings it encodes.
+SAMPLE_LABELS = ["alpha", "beta", "gamma"]
+
+
+@pytest.fixture(scope="session")
+def sample_path(tmp_path_factory):
+ """Write, once per session, a small netCDF4 file and return its path."""
+ path = tmp_path_factory.mktemp("cf_dataset") / "sample.nc"
+ dataset = _thread_safe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ try:
+ for name, size in SAMPLE_DIMENSION_SIZES.items():
+ dataset.createDimension(name, size)
+ for name, value in SAMPLE_GLOBALS.items():
+ dataset.setncattr(name, value)
+
+ air = dataset.createVariable(
+ "air_temperature",
+ "f4",
+ ("time", "lat"),
+ fill_value=-999.0,
+ zlib=True,
+ chunksizes=(1, 4),
+ )
+ air.setncattr("units", "K")
+ air.setncattr("standard_name", "air_temperature")
+ air.setncattr("coordinates", "height")
+ air[:] = SAMPLE_AIR
+
+ # Scalar, and deliberately unchunked: chunking() answers differently.
+ height = dataset.createVariable("height", "f4", ())
+ height.setncattr("units", "m")
+ height[:] = 1.5
+
+ # Char data: the one case where EncodedVariable changes shape, dtype
+ # and dimensions out from under the wrapper.
+ label = dataset.createVariable("label", "S1", ("time", "nchars"))
+ label[:] = np.array(
+ [list(f"{text:<8}") for text in SAMPLE_LABELS], dtype="S1"
+ )
+ finally:
+ dataset.close()
+
+ return path
+
+
+@pytest.fixture
+def sample_location(sample_path):
+ """The sample file's path, as the string a CFDataset reports."""
+ return str(sample_path)
+```
+
+- [ ] **Step 2: Write the failing tests for the variable's read surface**
+
+`lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`."""
+
+from collections.abc import MutableMapping
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf.dataset import CFDatasetVariable
+from iris.fileformats.netcdf import _bytecoding_datasets, _dataset, _thread_safe_nc
+
+from .conftest import SAMPLE_AIR, SAMPLE_LABELS
+
+
+@pytest.fixture
+def raw(sample_path):
+ dataset = _thread_safe_nc.DatasetWrapper(sample_path, mode="r")
+ yield dataset
+ dataset.close()
+
+
+@pytest.fixture
+def encoded(sample_path):
+ dataset = _bytecoding_datasets.EncodedDataset(sample_path, mode="r")
+ yield dataset
+ dataset.close()
+
+
+def wrap(dataset, name, location, write_lock=None):
+ return _dataset.NetCDFDatasetVariable(
+ dataset.variables[name], location, write_lock=write_lock
+ )
+
+
+@pytest.fixture
+def air(raw, sample_location):
+ return wrap(raw, "air_temperature", sample_location)
+
+
+@pytest.fixture
+def height(raw, sample_location):
+ return wrap(raw, "height", sample_location)
+
+
+class TestStorageProperties:
+ def test_is_a_cf_dataset_variable(self, air):
+ assert isinstance(air, CFDatasetVariable)
+
+ def test_name(self, air):
+ assert air.name == "air_temperature"
+
+ def test_location(self, air, sample_location):
+ assert air.location == sample_location
+
+ def test_dimensions_are_a_tuple(self, air):
+ # netCDF4 answers with a tuple already, but the contract says tuple
+ # and CFVariable.spans does set arithmetic on it, so pin it.
+ assert air.dimensions == ("time", "lat")
+ assert isinstance(air.dimensions, tuple)
+
+ def test_shape_dtype_size_ndim(self, air):
+ assert air.shape == (3, 4)
+ assert air.dtype == np.dtype("f4")
+ assert air.size == 12
+ assert air.ndim == 2
+
+ def test_len(self, air):
+ assert len(air) == 3
+
+ def test_scalar_has_no_len(self, height):
+ assert height.shape == ()
+ with pytest.raises(TypeError, match="unsized"):
+ len(height)
+
+ def test_fill_value(self, air):
+ assert air.fill_value == -999.0
+
+ def test_no_fill_value_is_none(self, height):
+ assert height.fill_value is None
+
+ def test_chunking_is_a_tuple(self, air):
+ # netCDF4 answers with a list; the contract says a shape.
+ assert air.chunking == (1, 4)
+
+ def test_contiguous_chunking_is_none(self, height):
+ # netCDF4 answers "contiguous" here, and loader.py already treats
+ # that and None identically, so both collapse to None.
+ assert height.chunking is None
+
+
+class TestData:
+ def test_getitem(self, air):
+ np.testing.assert_array_equal(air[:], SAMPLE_AIR)
+
+ def test_getitem_indexed(self, air):
+ np.testing.assert_array_equal(air[0], SAMPLE_AIR[0])
+
+
+class TestAttributes:
+ def test_is_a_plain_mutable_mapping(self, air):
+ # Tracking belongs to CFVariable, not here: a CFDatasetVariable can
+ # have two CFVariables promoted over it - see finding F1.
+ assert isinstance(air.attributes, MutableMapping)
+ assert not hasattr(air.attributes, "read")
+
+ def test_contents(self, air):
+ assert dict(air.attributes) == {
+ "_FillValue": -999.0,
+ "units": "K",
+ "standard_name": "air_temperature",
+ "coordinates": "height",
+ }
+
+ def test_is_built_once(self, air):
+ assert air.attributes is air.attributes
+
+ def test_missing_key_raises_key_error(self, air):
+ with pytest.raises(KeyError, match="nonesuch"):
+ air.attributes["nonesuch"]
+
+ def test_unreadable_attribute_becomes_empty_string(self, mocker, raw, sample_location):
+ # ncattrs() can list a name that getncattr then refuses. _getncattr in
+ # cf/_reader.py tolerated exactly this with a "" default, and that
+ # tolerance has to survive the move.
+ variable = raw.variables["height"]
+ mocker.patch.object(variable, "ncattrs", return_value=["units", "broken"])
+ mocker.patch.object(
+ variable,
+ "getncattr",
+ side_effect=lambda name: (
+ "m" if name == "units" else _raise(AttributeError(name))
+ ),
+ )
+ wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location)
+ assert wrapped.attributes["broken"] == ""
+
+
+def _raise(exception):
+ raise exception
+
+
+class TestNetCDFOnlyMembers:
+ def test_variable_exposes_the_backing_wrapper(self, air, raw):
+ assert air.variable is raw.variables["air_temperature"]
+
+ def test_not_variable_length(self, air):
+ assert air.is_variable_length is False
+
+ def test_not_emulated(self, air):
+ assert air.is_emulated is False
+
+ def test_emulated_data_array_raises_when_not_emulated(self, air):
+ with pytest.raises(AttributeError, match="_data_array"):
+ air.emulated_data_array
+
+ def test_emulated_round_trip(self, air):
+ # The Xarray bridge hook, issue #4994: an emulating variable carries
+ # its own array instead of file storage.
+ air.variable._data_array = np.zeros(3)
+ assert air.is_emulated is True
+ np.testing.assert_array_equal(air.emulated_data_array, np.zeros(3))
+ air.emulated_data_array = np.ones(3)
+ np.testing.assert_array_equal(air.variable._data_array, np.ones(3))
+
+ def test_deprecated_netcdf_member_reaches_the_wrapper(self, air):
+ assert sorted(air.deprecated_netcdf_member("ncattrs")()) == [
+ "_FillValue",
+ "coordinates",
+ "standard_name",
+ "units",
+ ]
+
+ def test_deprecated_netcdf_member_of_a_missing_name_raises(self, air):
+ with pytest.raises(AttributeError):
+ air.deprecated_netcdf_member("no_such_netcdf_member")
+
+
+class TestCharacterData:
+ def test_raw_wrapper_sees_the_char_dimension(self, raw, sample_location):
+ label = wrap(raw, "label", sample_location)
+ assert label.dimensions == ("time", "nchars")
+ assert label.shape == (3, 8)
+ assert label.dtype == np.dtype("S1")
+
+ def test_encoded_wrapper_sees_strings(self, encoded, sample_location):
+ # EncodedVariable drops the trailing char dimension and reports a
+ # string dtype. The CFDatasetVariable must pass that through, not
+ # reach around it to the contained netCDF4 variable.
+ label = wrap(encoded, "label", sample_location)
+ assert label.dimensions == ("time",)
+ assert label.shape == (3,)
+ assert label.dtype == np.dtype("U8")
+ assert [text.strip() for text in label[:]] == SAMPLE_LABELS
+```
+
+- [ ] **Step 3: Run the tests to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/`
+Expected: collection error —
+`ModuleNotFoundError: No module named 'iris.fileformats.netcdf._dataset'`
+
+- [ ] **Step 4: Write the module and the variable's read surface**
+
+Create `lib/iris/fileformats/netcdf/_dataset.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The netCDF implementation of the :mod:`iris.fileformats.cf.dataset` interface.
+
+This is the only module that knows both the CF interface and the netCDF4 API.
+Everything netCDF-specific that the CF layer used to reach through a
+:class:`~iris.fileformats.cf.CFVariable` - ``ncattrs``, ``getncattr``,
+``setncattr``, ``chunking()``, ``file_format``, ``createVariable``,
+``createDimension``, ``filepath()``, ``isopen()`` - either has a named member
+on the interface or lives here as a netCDF-only member.
+
+See section 4.5 of ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+from collections.abc import Iterator, MutableMapping
+from typing import Any
+
+import numpy as np
+
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+
+from . import _bytecoding_datasets, _thread_safe_nc
+
+#: What ``netCDF4.Variable.chunking()`` answers for an unchunked variable.
+#: ``None`` means the same thing, and arrives from non-version-4 files.
+_CONTIGUOUS = "contiguous"
+
+#: The member an emulating variable carries instead of file storage. See
+#: https://github.com/SciTools/iris/issues/4994 "Xarray bridge".
+_EMULATED_DATA_ARRAY = "_data_array"
+
+
+class _NetCDFAttributes(MutableMapping):
+ """A netCDF object's attributes as a mapping: read once, written through.
+
+ Values are read once, at construction, because the interface promises a
+ mapping whose ``keys`` and ``items`` cost nothing - attribute tracking asks
+ "which of these went unread?" on every variable of every loaded file, and
+ a lazily-fetching mapping would make that question expensive.
+
+ """
+
+ def __init__(self, target):
+ """Materialise the attributes of ``target``, a netCDF variable or dataset."""
+ self._target = target
+ self._values: dict[str, Any] = {}
+ for name in target.ncattrs():
+ try:
+ value = target.getncattr(name)
+ except AttributeError:
+ # ncattrs() can list a name that getncattr then refuses. The
+ # netCDF4 library does this for some malformed files, and
+ # cf/_reader.py's _getncattr tolerated it with this default.
+ value = ""
+ self._values[name] = value
+
+ def __getitem__(self, key: str) -> Any:
+ """Return ``key``'s value."""
+ return self._values[key]
+
+ def __setitem__(self, key: str, value: Any) -> None:
+ """Set ``key``'s value, here and in the netCDF object."""
+ self._target.setncattr(key, value)
+ self._values[key] = value
+
+ def __delitem__(self, key: str) -> None:
+ """Remove ``key``, here and from the netCDF object."""
+ self._target.delncattr(key)
+ del self._values[key]
+
+ def __iter__(self) -> Iterator[str]:
+ """Return an iterator over the attribute names, in file order."""
+ return iter(self._values)
+
+ def __len__(self) -> int:
+ """Return the number of attributes."""
+ return len(self._values)
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self._values!r})"
+
+
+class NetCDFDatasetVariable(CFDatasetVariable):
+ """One variable of a netCDF file, presented through the CF interface."""
+
+ def __init__(self, variable, location: str, *, write_lock=None):
+ """Wrap ``variable``, a thread-safe netCDF variable wrapper.
+
+ Parameters
+ ----------
+ variable : :class:`~iris.fileformats.netcdf._thread_safe_nc.VariableWrapper`
+ The wrapped netCDF variable. May be an
+ :class:`~iris.fileformats.netcdf._bytecoding_datasets.EncodedVariable`,
+ whose shape, dimensions and dtype differ from the file's own for
+ character data; those are passed through, not reached around.
+ location : str
+ The path or URL of the dataset this variable belongs to.
+ write_lock : optional
+ The lock shared by every variable of one dataset, used by
+ :meth:`write_handle`. Supplied by :class:`NetCDFDataset`.
+
+ """
+ self._variable = variable
+ self._location = location
+ self._write_lock = write_lock
+ self._attributes = _NetCDFAttributes(variable)
+
+ @property
+ def name(self) -> str:
+ """The variable's name within its dataset."""
+ return self._variable.name
+
+ @property
+ def location(self) -> str:
+ """The path or URL of the dataset holding this variable."""
+ return self._location
+
+ @property
+ def dimensions(self) -> tuple:
+ """The names of the dimensions this variable spans, in order."""
+ return tuple(self._variable.dimensions)
+
+ @property
+ def shape(self) -> tuple:
+ """The variable's shape."""
+ return tuple(self._variable.shape)
+
+ @property
+ def dtype(self):
+ """The variable's stored data type, or ``str`` for a VLEN string type."""
+ return self._variable.dtype
+
+ @property
+ def size(self) -> int:
+ """The total number of elements in the variable."""
+ return self._variable.size
+
+ @property
+ def fill_value(self) -> Any:
+ """The variable's ``_FillValue`` attribute, or ``None`` if it has none."""
+ return self._attributes.get("_FillValue")
+
+ @property
+ def chunking(self) -> tuple | None:
+ """The variable's storage chunk shape, or ``None`` when unchunked.
+
+ netCDF answers ``None`` for a non-version-4 file and the string
+ ``"contiguous"`` for an unchunked version-4 variable. Both mean the
+ same thing to a caller choosing a Dask chunking, so both become
+ ``None``.
+
+ """
+ chunks = self._variable.chunking()
+ if chunks is None or chunks == _CONTIGUOUS:
+ return None
+ return tuple(chunks)
+
+ @property
+ def attributes(self) -> MutableMapping:
+ """The variable's CF attributes."""
+ return self._attributes
+
+ def __getitem__(self, keys) -> np.ndarray:
+ """Return the indexed portion of the variable's data."""
+ return self._variable[keys]
+
+ def __setitem__(self, keys, values) -> None:
+ """Write ``values`` into the indexed portion of the variable."""
+ self._variable[keys] = values
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self.name!r}, {self.location!r})"
+
+ # netCDF-only members below: named here rather than reached for through
+ # getattr, so that a Zarr caller fails to import them rather than failing
+ # at run time with an AttributeError from somewhere unrelated.
+
+ @property
+ def variable(self):
+ """The backing thread-safe netCDF variable wrapper."""
+ return self._variable
+
+ @property
+ def is_variable_length(self) -> bool:
+ """Whether this is a netCDF variable-length (VLEN) type.
+
+ Such a variable's total size cannot be known without reading it - see
+ https://github.com/Unidata/netcdf-c/issues/1893 - so the loader has to
+ guess whether it is worth making lazy.
+
+ """
+ datatype = getattr(self._variable, "datatype", None)
+ return isinstance(datatype, _thread_safe_nc.VLType)
+
+ @property
+ def is_emulated(self) -> bool:
+ """Whether an emulating object supplies this variable's data directly.
+
+ The Xarray bridge, https://github.com/SciTools/iris/issues/4994: the
+ "file" is an emulator and its variables carry their own arrays.
+
+ """
+ return hasattr(self._variable, _EMULATED_DATA_ARRAY)
+
+ @property
+ def emulated_data_array(self):
+ """The array an emulating variable carries instead of file storage."""
+ return getattr(self._variable, _EMULATED_DATA_ARRAY)
+
+ @emulated_data_array.setter
+ def emulated_data_array(self, value) -> None:
+ setattr(self._variable, _EMULATED_DATA_ARRAY, value)
+
+ def deprecated_netcdf_member(self, name: str) -> Any:
+ """Return a member of the backing netCDF variable wrapper."""
+ return getattr(self._variable, name)
+```
+
+`write_handle` is deliberately absent: it is a write concern and arrives in
+Task 4. `NetCDFDatasetVariable` is therefore still abstract at the end of this
+task — which is why every test above constructs it through `wrap`, and why
+`TestStorageProperties.test_is_a_cf_dataset_variable` would fail if it were
+not. Add the stub now, so the class is concrete:
+
+```python
+ def write_handle(self) -> Any:
+ """Return a picklable object supporting ``__setitem__``, for Dask stores."""
+ raise NotImplementedError
+```
+
+- [ ] **Step 5: Run the tests to verify they pass**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/`
+Expected: PASS — every test in the file, 0 failed, 0 skipped.
+
+- [ ] **Step 6: Check the module's docstrings satisfy numpydoc**
+
+Run: `pre-commit run numpydoc-validation --files lib/iris/fileformats/netcdf/_dataset.py`
+Expected: Passed. Property setters need no docstring; everything else does,
+one line, imperative, ending in a period. Fix anything it names.
+
+- [ ] **Step 7: Green check and commit**
+
+```bash
+pre-commit run --files lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/__init__.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/conftest.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/
+git add lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/
+git commit -m "$(cat <<'EOF'
+Add NetCDFDatasetVariable, the read surface
+
+Implements the CFDatasetVariable interface over a thread-safe netCDF
+variable wrapper, plus the netCDF-only members - the VLEN check, the
+Xarray-bridge data array, and the backing wrapper itself - that the CF
+layer used to reach for through getattr.
+
+Not wired to anything yet.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 3: `netcdf/_dataset.py` — the dataset, read surface
+
+**Files:**
+- Modify: `lib/iris/fileformats/netcdf/_dataset.py` (append `NetCDFDataset`)
+- Test: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`
+
+**Interfaces:**
+- Consumes: `CFDataset` from Task 1; `NetCDFDatasetVariable` and
+ `_NetCDFAttributes` from Task 2; `sample_path` / `sample_location` /
+ `SAMPLE_DIMENSIONS` / `SAMPLE_GLOBALS` from Task 2's conftest.
+- Produces:
+ - `NetCDFDataset(location, mode="r", *, netcdf_format=None,
+ warn_legacy_format=False)` — opens and owns a netCDF file.
+ - `NetCDFDataset.from_existing(dataset)` — borrows an already-open dataset
+ or emulator; `close()` on it is a no-op. Task 10 uses this from
+ `CFReader`; Task 12 uses it from `Saver`.
+ - `.dataset` (the backing `DatasetWrapper`) and `.write_lock`. Task 12 sets
+ `Saver.file_write_lock = self._dataset.write_lock`.
+
+- [ ] **Step 1: Write the failing tests**
+
+`lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.netcdf._dataset.NetCDFDataset`."""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf.dataset import CFDataset
+from iris.fileformats.netcdf import _bytecoding_datasets, _dataset, _thread_safe_nc
+from iris.warnings import IrisLoadWarning
+
+from .conftest import SAMPLE_DIMENSIONS, SAMPLE_GLOBALS
+
+
+@pytest.fixture
+def reader(sample_path):
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ yield dataset
+
+
+class TestOpening:
+ def test_is_a_cf_dataset(self, reader):
+ assert isinstance(reader, CFDataset)
+
+ def test_location_is_the_string_path(self, reader, sample_location):
+ assert reader.location == sample_location
+
+ def test_default_mode_is_read(self, reader):
+ assert reader.mode == "r"
+
+ def test_starts_open(self, reader):
+ assert reader.closed is False
+
+ def test_close_is_idempotent(self, sample_path):
+ dataset = _dataset.NetCDFDataset(sample_path)
+ dataset.close()
+ assert dataset.closed is True
+ dataset.close()
+ assert dataset.closed is True
+
+ def test_context_manager_closes(self, sample_path):
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ assert dataset.closed is False
+ assert dataset.closed is True
+
+ def test_missing_file_raises(self, tmp_path):
+ with pytest.raises((FileNotFoundError, OSError)):
+ _dataset.NetCDFDataset(tmp_path / "nope.nc")
+
+ def test_decoding_setting_chooses_the_wrapper(self, sample_path):
+ with _bytecoding_datasets.DECODE_TO_STRINGS_ON_READ.context(True):
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ assert isinstance(dataset.dataset, _bytecoding_datasets.EncodedDataset)
+ with _bytecoding_datasets.DECODE_TO_STRINGS_ON_READ.context(False):
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ assert not isinstance(
+ dataset.dataset, _bytecoding_datasets.EncodedDataset
+ )
+ assert isinstance(dataset.dataset, _thread_safe_nc.DatasetWrapper)
+
+
+class TestContents:
+ def test_variables(self, reader, sample_location):
+ assert sorted(reader.variables) == ["air_temperature", "height", "label"]
+ air = reader.variables["air_temperature"]
+ assert isinstance(air, _dataset.NetCDFDatasetVariable)
+ assert air.name == "air_temperature"
+ assert air.location == sample_location
+
+ def test_variables_are_built_once(self, reader):
+ assert (
+ reader.variables["air_temperature"]
+ is reader.variables["air_temperature"]
+ )
+
+ def test_variables_share_one_write_lock(self, reader):
+ # _dask_locks.get_worker_lock() returns a FRESH threading.Lock under
+ # the threaded scheduler, so a per-variable call would hand out locks
+ # that exclude nothing.
+ air = reader.variables["air_temperature"]
+ height = reader.variables["height"]
+ assert air._write_lock is height._write_lock
+ assert air._write_lock is reader.write_lock
+
+ def test_dimensions_are_names_and_lengths(self, reader):
+ assert dict(reader.dimensions) == SAMPLE_DIMENSIONS
+
+ def test_unlimited_dimension_reports_its_current_length(self, reader):
+ # Nothing was written along "record", so it is zero-length - which is
+ # what netCDF4 reports, and what saver.py's membership tests need.
+ assert reader.dimensions["record"] == 0
+ assert "record" in reader.dimensions
+
+ def test_global_attributes(self, reader):
+ assert dict(reader.attributes) == SAMPLE_GLOBALS
+
+ def test_global_attributes_are_built_once(self, reader):
+ assert reader.attributes is reader.attributes
+
+
+class TestBorrowing:
+ def test_from_existing_wraps_an_open_dataset(self, sample_path, sample_location):
+ raw = _bytecoding_datasets.EncodedDataset(sample_path, mode="r")
+ try:
+ dataset = _dataset.NetCDFDataset.from_existing(raw)
+ assert dataset.dataset is raw
+ assert dataset.location == sample_location
+ assert sorted(dataset.variables) == [
+ "air_temperature",
+ "height",
+ "label",
+ ]
+ finally:
+ raw.close()
+
+ def test_from_existing_does_not_close_what_it_borrowed(self, sample_path):
+ raw = _bytecoding_datasets.EncodedDataset(sample_path, mode="r")
+ try:
+ dataset = _dataset.NetCDFDataset.from_existing(raw)
+ dataset.close()
+ assert dataset.closed is True
+ # Still usable: the borrower released nothing.
+ assert raw.isopen()
+ finally:
+ raw.close()
+
+ def test_from_existing_wraps_a_bare_netcdf4_dataset(self, sample_path):
+ # What the Xarray bridge hands iris.save / CFReader: an object with
+ # the netCDF4 API but no thread-safe wrapper around it.
+ import netCDF4
+
+ raw = netCDF4.Dataset(sample_path, mode="r")
+ try:
+ dataset = _dataset.NetCDFDataset.from_existing(raw)
+ assert isinstance(
+ dataset.dataset, _bytecoding_datasets.EncodedDataset
+ )
+ assert dataset.dataset._contained_instance is raw
+ finally:
+ raw.close()
+
+
+class TestAutoChartostring:
+ """Iris decodes byte data itself, so netCDF4 must not do it first.
+
+ CFReader turned this off on every dataset it opened (_reader.py:182).
+ The dataset now does it, so no caller has to remember.
+ """
+
+ def test_turned_off_on_an_opened_dataset(self, sample_path, mocker):
+ spy = mocker.spy(_thread_safe_nc.DatasetWrapper, "set_auto_chartostring")
+ with _bytecoding_datasets.DECODE_TO_STRINGS_ON_READ.context(False):
+ with _dataset.NetCDFDataset(sample_path):
+ pass
+ spy.assert_called_once_with(mocker.ANY, False)
+
+ def test_turned_off_on_a_borrowed_dataset(self, sample_path, mocker):
+ raw = _thread_safe_nc.DatasetWrapper(sample_path, mode="r")
+ spy = mocker.spy(raw, "set_auto_chartostring")
+ try:
+ _dataset.NetCDFDataset.from_existing(raw)
+ spy.assert_called_once_with(False)
+ finally:
+ raw.close()
+
+ def test_an_encoded_dataset_blocks_it_rather_than_forwarding(self, sample_path):
+ # EncodedDataset does its own decoding, so the call is inert there -
+ # which is why making it unconditionally is safe.
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ assert isinstance(dataset.dataset, _bytecoding_datasets.EncodedDataset)
+ with pytest.raises(TypeError, match="not supported"):
+ dataset.dataset.set_auto_chartostring(True)
+
+
+class TestLegacyFormatWarning:
+ def test_silent_by_default(self, tmp_path, sample_path):
+ legacy = _write_netcdf3(tmp_path)
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ with _dataset.NetCDFDataset(legacy):
+ pass
+
+ def test_warns_when_asked(self, tmp_path):
+ legacy = _write_netcdf3(tmp_path)
+ with pytest.warns(IrisLoadWarning, match="nccopy"):
+ with _dataset.NetCDFDataset(legacy, warn_legacy_format=True):
+ pass
+
+ def test_does_not_warn_for_netcdf4(self, sample_path):
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ with _dataset.NetCDFDataset(sample_path, warn_legacy_format=True):
+ pass
+
+
+def _write_netcdf3(tmp_path):
+ path = tmp_path / "legacy.nc"
+ dataset = _thread_safe_nc.DatasetWrapper(path, mode="w", format="NETCDF3_CLASSIC")
+ dataset.createDimension("time", 2)
+ variable = dataset.createVariable("time", "f4", ("time",))
+ variable[:] = np.arange(2, dtype="f4")
+ dataset.close()
+ return path
+```
+
+Add `import warnings` at the top of the file, beside `numpy`.
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`
+Expected: collection error —
+`AttributeError: module 'iris.fileformats.netcdf._dataset' has no attribute 'NetCDFDataset'`
+
+- [ ] **Step 3: Write the dataset's read surface**
+
+Append to `lib/iris/fileformats/netcdf/_dataset.py`:
+
+```python
+#: The formats that make CF loading slow, and that the user can convert away
+#: from with "nccopy".
+_LEGACY_FORMATS = ("NETCDF3_CLASSIC", "NETCDF3_64BIT")
+
+
+class NetCDFDataset(CFDataset):
+ """A netCDF file, presented through the CF interface."""
+
+ def __init__(
+ self,
+ location,
+ mode: str = "r",
+ *,
+ netcdf_format=None,
+ warn_legacy_format: bool = False,
+ ):
+ """Open a netCDF file.
+
+ Parameters
+ ----------
+ location : str or :class:`pathlib.Path`
+ The file's path or URL.
+ mode : str, default="r"
+ The netCDF4 open mode.
+ netcdf_format : str, optional
+ The netCDF format to create, when writing.
+ warn_legacy_format : bool, default=False
+ Whether to warn that a netCDF3 file would load faster if
+ converted. Only loading asks for this; the loader is where the
+ user can act on it.
+
+ """
+ # Set first, so that __del__ and close() are safe if opening fails.
+ self._dataset = None
+ self._owned = True
+ self._closed = False
+ self._variables: dict[str, NetCDFDatasetVariable] | None = None
+ self._attributes: _NetCDFAttributes | None = None
+ self._write_lock = None
+
+ self._location = str(location)
+ self._mode = mode
+
+ if mode == "r" and not _bytecoding_datasets.DECODE_TO_STRINGS_ON_READ:
+ # The user has turned string decoding off for reads. Writing has
+ # no such switch: the saver always encodes.
+ dataset_class = _thread_safe_nc.DatasetWrapper
+ else:
+ dataset_class = _bytecoding_datasets.EncodedDataset
+
+ # netCDF4.Dataset validates "format" against a fixed list and rejects
+ # None, so omit it entirely rather than passing the default through.
+ extra = {} if netcdf_format is None else {"format": netcdf_format}
+ self._dataset = dataset_class(location, mode=mode, **extra)
+
+ if warn_legacy_format and self._dataset.file_format in _LEGACY_FORMATS:
+ warnings.warn(
+ "Optimise CF-netCDF loading by converting data from NetCDF3 "
+ 'to NetCDF4 file format using the "nccopy" command.',
+ category=iris.warnings.IrisLoadWarning,
+ )
+
+ # Turn off *any* automatic decoding by netCDF4 itself. Iris decodes
+ # byte data on its own terms, in _bytecoding_datasets. Inert on an
+ # EncodedDataset, which blocks the call; real on a DatasetWrapper.
+ self._dataset.set_auto_chartostring(False)
+
+ @classmethod
+ def from_existing(cls, dataset) -> "NetCDFDataset":
+ """Wrap an already-open netCDF dataset, without taking ownership of it.
+
+ ``dataset`` may be a thread-safe wrapper, a bare
+ :class:`netCDF4.Dataset`, or any object emulating one - the Xarray
+ bridge passes the last of these. :meth:`close` will not release it,
+ because whoever opened it is still responsible for it.
+
+ """
+ instance = cls.__new__(cls)
+ instance._owned = False
+ instance._closed = False
+ instance._variables = None
+ instance._attributes = None
+ instance._write_lock = None
+ instance._mode = "r+"
+
+ if not hasattr(dataset, "THREAD_SAFE_FLAG"):
+ # The wrappers forbid re-wrapping, so only wrap what is not one.
+ dataset = _bytecoding_datasets.EncodedDataset.from_existing(dataset)
+ instance._dataset = dataset
+ instance._dataset.set_auto_chartostring(False)
+ instance._location = str(dataset.filepath())
+ return instance
+
+ @property
+ def location(self) -> str:
+ """The file's path or URL."""
+ return self._location
+
+ @property
+ def mode(self) -> str:
+ """The mode the file was opened in."""
+ return self._mode
+
+ @property
+ def closed(self) -> bool:
+ """Whether :meth:`close` has already released the file."""
+ return self._closed
+
+ @property
+ def variables(self) -> Mapping:
+ """The file's variables, by name."""
+ if self._variables is None:
+ self._variables = {
+ name: NetCDFDatasetVariable(
+ variable, self._location, write_lock=self.write_lock
+ )
+ for name, variable in self._dataset.variables.items()
+ }
+ return self._variables
+
+ @property
+ def dimensions(self) -> Mapping:
+ """The file's dimension lengths, by name.
+
+ An unlimited dimension reports the number of records written so far,
+ which is what ``len()`` of a netCDF4 dimension gives and what every
+ caller in Iris - all of them membership tests - needs.
+
+ """
+ return {
+ name: len(dimension)
+ for name, dimension in self._dataset.dimensions.items()
+ }
+
+ @property
+ def attributes(self) -> MutableMapping:
+ """The file's global attributes."""
+ if self._attributes is None:
+ self._attributes = _NetCDFAttributes(self._dataset)
+ return self._attributes
+
+ def sync(self) -> None:
+ """Flush buffered writes to the file."""
+ self._dataset.sync()
+
+ def finalise(self) -> None:
+ """Do nothing: a netCDF file needs no completion step.
+
+ Kept so that callers can be written once. Zarr's consolidated metadata
+ is what this exists for - see finding F6.
+
+ """
+
+ def close(self) -> None:
+ """Close the file, if this object opened it."""
+ if self._owned and self._dataset is not None and not self._closed:
+ self._dataset.close()
+ self._closed = True
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self.location!r}, mode={self.mode!r})"
+
+ # netCDF-only members below.
+
+ @property
+ def dataset(self):
+ """The backing thread-safe netCDF dataset wrapper."""
+ return self._dataset
+
+ @property
+ def write_lock(self):
+ """The lock every worker writing to this file must hold.
+
+ One per dataset, because under the threaded scheduler
+ :func:`iris.fileformats.netcdf._dask_locks.get_worker_lock` returns a
+ new :class:`threading.Lock` on each call, and two such locks exclude
+ nothing.
+
+ """
+ if self._write_lock is None:
+ self._write_lock = _dask_locks.get_worker_lock(self._location)
+ return self._write_lock
+```
+
+Extend the module's imports to match:
+
+```python
+from collections.abc import Iterator, Mapping, MutableMapping
+import warnings
+
+import iris.warnings
+
+from . import _bytecoding_datasets, _dask_locks, _thread_safe_nc
+```
+
+`create_dimension` and `create_variable` are still abstract at this point, so
+the class cannot be instantiated. Add them as stubs now — Task 4 fills them
+in, driven by its own tests:
+
+```python
+ def create_dimension(self, name: str, size: int | None) -> None:
+ """Declare a dimension of the given length."""
+ raise NotImplementedError
+
+ def create_variable(
+ self, name: str, dtype, dimensions=(), *, fill_value=None, **encoding
+ ) -> NetCDFDatasetVariable:
+ """Create and return a new variable."""
+ raise NotImplementedError
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/`
+Expected: PASS — every test in both files, 0 failed, 0 skipped.
+
+- [ ] **Step 5: Confirm the interface is still one-way**
+
+```bash
+python - <<'PY'
+src = open('lib/iris/fileformats/cf/dataset.py').read()
+assert 'netcdf' not in src and 'netCDF4' not in src
+import iris.fileformats.netcdf._dataset as m
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+assert issubclass(m.NetCDFDataset, CFDataset)
+assert issubclass(m.NetCDFDatasetVariable, CFDatasetVariable)
+print('ok')
+PY
+```
+Expected: `ok`.
+
+- [ ] **Step 6: Green check and commit**
+
+```bash
+pre-commit run --files lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/
+git add lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py
+git commit -m "$(cat <<'EOF'
+Add NetCDFDataset, the read surface
+
+Opens a netCDF file, or borrows an already-open one, and presents its
+variables, dimensions and global attributes through the CF interface.
+
+The NetCDF3 "use nccopy" warning moves here from CFReader, behind an
+explicit warn_legacy_format flag: the file format is netCDF vocabulary,
+and CFReader is no longer allowed to know it.
+
+Not wired to anything yet.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 4: `netcdf/_dataset.py` — the write surface
+
+**Files:**
+- Modify: `lib/iris/fileformats/netcdf/_dataset.py`
+- Modify: `lib/iris/fileformats/netcdf/saver.py:271-296` (`_bytes_if_ascii` moves out)
+- Test: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py`
+
+**Interfaces:**
+- Consumes: `NetCDFDataset` and `NetCDFDatasetVariable` from Tasks 2 and 3.
+- Produces:
+ - `NetCDFDataset.create_dimension(name, size)` — `size=None` is unlimited.
+ - `NetCDFDataset.create_variable(name, dtype, dimensions=(), *,
+ fill_value=None, **encoding)` returning a `NetCDFDatasetVariable`.
+ `dimensions` defaults to `()` so that `saver.py:2080`'s dimensionless
+ grid-mapping variable needs no argument (finding F4). `**encoding` is
+ passed straight to `netCDF4.createVariable`: `zlib`, `complevel`,
+ `shuffle`, `chunksizes`, `compression`, and the rest.
+ - `NetCDFDatasetVariable.write_handle()` returning an
+ `EncodedNetCDFWriteProxy` (or a `NetCDFWriteProxy` for an unencoded
+ variable). Task 12 replaces `saver.py:2641`'s direct construction with it.
+ - `_dataset._bytes_if_ascii(value)`, moved here from `saver.py:271`.
+ `saver.py` imports it back for its own `_setncattr`, which Task 12 deletes.
+
+- [ ] **Step 1: Write the failing tests for dimensions, variables and writes**
+
+`lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the write surface of the netCDF CFDataset implementation."""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.netcdf import _bytecoding_datasets, _dataset, _thread_safe_nc
+
+
+@pytest.fixture
+def path(tmp_path):
+ return tmp_path / "written.nc"
+
+
+@pytest.fixture
+def writer(path):
+ with _dataset.NetCDFDataset(path, mode="w", netcdf_format="NETCDF4") as dataset:
+ yield dataset
+
+
+@pytest.fixture
+def grid(writer):
+ """A 2x3 grid, ready for variables to be created against."""
+ writer.create_dimension("y", 3)
+ writer.create_dimension("x", 2)
+ return writer
+
+
+class TestCreateDimension:
+ def test_appears_with_its_length(self, writer):
+ writer.create_dimension("x", 5)
+ assert writer.dimensions["x"] == 5
+
+ def test_none_means_unlimited(self, writer):
+ # saver.py:829 passes None for a dimension the user asked to make
+ # unlimited. It reads back as zero-length until records are written.
+ writer.create_dimension("t", None)
+ assert writer.dimensions["t"] == 0
+ assert writer.dataset.dimensions["t"].isunlimited()
+
+
+class TestCreateVariable:
+ def test_returns_a_dataset_variable(self, grid):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ assert isinstance(variable, _dataset.NetCDFDatasetVariable)
+ assert variable.name == "air"
+ assert variable.dimensions == ("y", "x")
+ assert variable.shape == (3, 2)
+ assert variable.dtype == np.dtype("f4")
+
+ def test_appears_in_the_dataset(self, grid):
+ created = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ assert grid.variables["air"].name == created.name
+
+ def test_dimensions_default_to_scalar(self, writer):
+ # saver.py:2080 creates a grid-mapping variable with no dimensions at
+ # all, and passes only a name and a dtype - see finding F4.
+ variable = writer.create_variable("grid", np.int32)
+ assert variable.dimensions == ()
+ assert variable.shape == ()
+
+ def test_dimensions_may_be_a_list(self, grid):
+ # saver.py:1938 and :1983 pass a list, not a tuple.
+ variable = grid.create_variable("air", np.dtype("f4"), ["y", "x"])
+ assert variable.dimensions == ("y", "x")
+
+ def test_fill_value_becomes_an_attribute(self, grid):
+ variable = grid.create_variable(
+ "air", np.dtype("f4"), ("y", "x"), fill_value=-1.0
+ )
+ assert variable.fill_value == -1.0
+ assert variable.attributes["_FillValue"] == -1.0
+
+ def test_encoding_is_passed_through(self, grid):
+ variable = grid.create_variable(
+ "air", np.dtype("f4"), ("y", "x"), zlib=True, chunksizes=(1, 2)
+ )
+ assert variable.chunking == (1, 2)
+
+ def test_shares_the_datasets_write_lock(self, grid):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ assert variable._write_lock is grid.write_lock
+
+
+class TestWritingData:
+ def test_setitem_round_trip(self, grid, path):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ payload = np.arange(6, dtype="f4").reshape(3, 2)
+ variable[:] = payload
+ grid.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ np.testing.assert_array_equal(reader.variables["air"][:], payload)
+
+ def test_write_handle_writes_after_the_dataset_is_closed(self, grid, path):
+ # This is what a Dask worker gets as a da.store target, long after the
+ # Saver's own handle on the file has gone.
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ handle = variable.write_handle()
+ grid.close()
+
+ payload = np.arange(6, dtype="f4").reshape(3, 2)
+ handle[:] = payload
+
+ with _dataset.NetCDFDataset(path) as reader:
+ np.testing.assert_array_equal(reader.variables["air"][:], payload)
+
+ def test_write_handle_matches_the_variable_encoding(self, grid):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ # The dataset was opened for writing, so its variables are encoded;
+ # an unencoded proxy here would silently skip string encoding.
+ assert isinstance(
+ variable.write_handle(), _bytecoding_datasets.EncodedNetCDFWriteProxy
+ )
+
+ def test_write_handle_of_an_unencoded_variable(self, path):
+ raw = _thread_safe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ try:
+ raw.createDimension("x", 2)
+ raw.createVariable("air", "f4", ("x",))
+ variable = _dataset.NetCDFDatasetVariable(
+ raw.variables["air"], str(path), write_lock=None
+ )
+ assert isinstance(
+ variable.write_handle(), _thread_safe_nc.NetCDFWriteProxy
+ )
+ finally:
+ raw.close()
+
+
+class TestSyncAndFinalise:
+ def test_sync_flushes(self, grid, path):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable[:] = np.zeros((3, 2), dtype="f4")
+ grid.sync()
+ # Readable while the writer is still open, because sync() flushed.
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.variables["air"].shape == (3, 2)
+
+ def test_finalise_is_a_no_op(self, grid):
+ assert grid.finalise() is None
+ assert grid.closed is False
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py`
+Expected: FAIL — `NotImplementedError` from the `create_dimension`,
+`create_variable` and `write_handle` stubs left by Tasks 2 and 3.
+
+- [ ] **Step 3: Implement the write surface**
+
+In `lib/iris/fileformats/netcdf/_dataset.py`, replace
+`NetCDFDatasetVariable.write_handle`'s stub:
+
+```python
+ def write_handle(self) -> Any:
+ """Return a picklable object supporting ``__setitem__``, for Dask stores.
+
+ It carries the file path and variable name rather than the open file,
+ reopening on each write, so that a worker can use it after the saver
+ that created it has closed its own handle.
+
+ """
+ if isinstance(self._variable, _bytecoding_datasets.EncodedVariable):
+ proxy_class = _bytecoding_datasets.EncodedNetCDFWriteProxy
+ else:
+ # Only reachable for a dataset opened without string encoding;
+ # the saver always encodes.
+ proxy_class = _thread_safe_nc.NetCDFWriteProxy
+ return proxy_class(self._location, self._variable, self._write_lock)
+```
+
+and replace `NetCDFDataset`'s two stubs:
+
+```python
+ def create_dimension(self, name: str, size: int | None) -> None:
+ """Declare a dimension of the given length, or unlimited for ``None``."""
+ self._dataset.createDimension(name, size)
+
+ def create_variable(
+ self, name: str, dtype, dimensions=(), *, fill_value=None, **encoding
+ ) -> NetCDFDatasetVariable:
+ """Create and return a new variable.
+
+ ``**encoding`` reaches :meth:`netCDF4.Dataset.createVariable`
+ unaltered: ``compression``, ``zlib``, ``complevel``, ``shuffle``,
+ ``chunksizes``, ``least_significant_digit`` and the rest.
+
+ """
+ variable = self._dataset.createVariable(
+ name, dtype, tuple(dimensions), fill_value=fill_value, **encoding
+ )
+ wrapped = NetCDFDatasetVariable(
+ variable, self._location, write_lock=self.write_lock
+ )
+ if self._variables is not None:
+ # Keep an already-materialised mapping in step, rather than
+ # leaving it stale for the rest of the dataset's life.
+ self._variables[name] = wrapped
+ return wrapped
+```
+
+netCDF4 rejects `fill_value=None` for some dtypes only when combined with
+other options; it is its own documented default, so passing it through
+unconditionally is correct and keeps one code path.
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/`
+Expected: PASS — every test in all three files, 0 failed, 0 skipped.
+
+- [ ] **Step 5: Commit the write surface**
+
+```bash
+pre-commit run --files lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/
+git add lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py
+git commit -m "$(cat <<'EOF'
+Add the netCDF CFDataset write surface
+
+create_dimension, create_variable and write_handle(). write_handle is
+the seam that lets a Dask worker write into a variable after the saver
+has closed the file, and is where the Zarr implementation will return a
+zarr.Array instead of a reopen-on-write proxy.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+- [ ] **Step 6: Write the failing test for attribute write-through**
+
+Append to `test_NetCDFDataset__write.py`:
+
+```python
+class TestAttributeWrites:
+ def test_variable_attribute_write_through(self, grid, path):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["units"] = "K"
+ assert variable.attributes["units"] == "K"
+ grid.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.variables["air"].attributes["units"] == "K"
+
+ def test_global_attribute_write_through(self, writer, path):
+ writer.attributes["Conventions"] = "CF-1.7"
+ assert writer.attributes["Conventions"] == "CF-1.7"
+ writer.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.attributes["Conventions"] == "CF-1.7"
+
+ def test_attribute_deletion(self, grid, path):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["units"] = "K"
+ del variable.attributes["units"]
+ assert "units" not in variable.attributes
+ grid.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert "units" not in reader.variables["air"].attributes
+
+ def test_ascii_value_is_written_as_bytes(self, grid):
+ # _bytes_if_ascii, moved here from saver.py. netCDF4 writes a bytes
+ # value as NC_CHAR; the coercion is what makes every string attribute
+ # Iris writes take that type, whatever the file format.
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["units"] = "K"
+ assert variable.variable.getncattr("units") == "K"
+ assert _dataset._bytes_if_ascii("K") == b"K"
+
+ def test_non_string_values_pass_through(self, grid):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["valid_min"] = np.float32(-3.5)
+ assert variable.attributes["valid_min"] == np.float32(-3.5)
+ assert _dataset._bytes_if_ascii(3) == 3
+```
+
+- [ ] **Step 7: Run the tests to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py -k Attribute`
+Expected: FAIL —
+`AttributeError: module 'iris.fileformats.netcdf._dataset' has no attribute '_bytes_if_ascii'`
+
+- [ ] **Step 8: Move `_bytes_if_ascii` and apply it on write**
+
+Add to `lib/iris/fileformats/netcdf/_dataset.py`, above `_NetCDFAttributes`:
+
+```python
+def _bytes_if_ascii(value):
+ """Return an ASCII string as bytes, and anything else unchanged.
+
+ netCDF4 stores a bytes attribute as NC_CHAR. Coercing on the way in is
+ what keeps Iris's string attributes to that type across file formats,
+ rather than leaving it to netCDF4's own str handling.
+
+ """
+ if isinstance(value, str):
+ try:
+ return value.encode(encoding="ascii")
+ except (AttributeError, UnicodeEncodeError):
+ pass
+ return value
+```
+
+and apply it in `_NetCDFAttributes.__setitem__`:
+
+```python
+ def __setitem__(self, key: str, value: Any) -> None:
+ """Set ``key``'s value, here and in the netCDF object."""
+ # Coerce on the way out only. Caching the coerced value would make a
+ # just-written attribute read back as bytes, where the same attribute
+ # read from a file reads back as str. See finding F12.
+ self._target.setncattr(key, _bytes_if_ascii(value))
+ self._values[key] = value
+```
+
+The comment is load-bearing, and the fact it rests on is worth seeing for
+yourself before you trust it — netCDF4 writes the `bytes` as `NC_CHAR` and
+hands back a `str` **[verified]**:
+
+```python
+>>> ds.setncattr("nm", b"hello"); ds.getncattr("nm")
+'hello'
+```
+
+So a cache holding the *coerced* value disagrees with the file it mirrors:
+`attributes["units"]` would be `b"K"` on the dataset that just wrote it and
+`"K"` on one that reads it back. Two saver comparisons need the `str` —
+`cf_var.attributes["formula_terms"] != formula_terms` (`saver.py:1184`) and
+`" ".join(coords)` (`:1213`) — and both fail silently, as a spurious
+inequality and a `TypeError` respectively, against `bytes`.
+
+Step 6 already wrote the test: `test_variable_attribute_write_through`'s
+`assert variable.attributes["units"] == "K"` is on the *writing* dataset,
+before the file is reopened. Write `value = _bytes_if_ascii(value)` on its
+own line above the two, as the obvious reading of Step 6 suggests, and that
+assertion is what goes red.
+
+Then, in `lib/iris/fileformats/netcdf/saver.py`, delete the `_bytes_if_ascii`
+definition at line 271 and import it instead, leaving `_setncattr` otherwise
+untouched:
+
+```python
+from ._dataset import _bytes_if_ascii
+```
+
+Task 12 removes `_setncattr` and this import together. Check nothing else
+imported it from `saver`:
+
+```bash
+grep -rn "_bytes_if_ascii" lib/ benchmarks/ docs/
+```
+
+- [ ] **Step 9: Run the tests to verify they pass**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/`
+Expected: PASS — 0 failed, 1 skipped. That one skip is
+`attribute_handlers/test_GribParamHandler.py` wanting `iris_grib`, and it is
+0 if you have `iris_grib` installed (§6.2). Any *other* skip is a test that
+stopped being collected; treat it as a failure.
+
+- [ ] **Step 10: Review Focus 5 — a non-ASCII attribute value**
+
+`_bytes_if_ascii` is a `try`/`except` whose failure path is the interesting
+one: a value that will not encode as ASCII is left as `str`. Every degree
+sign, every accented place name in a `long_name` takes that path, and the
+relocated coercion has to keep taking it. Append:
+
+```python
+class TestNonAsciiAttributes:
+ """Review Focus 5. The coercion's except branch, which is the common one
+ for real-world metadata: degree signs, accented names, superscripts."""
+
+ @pytest.mark.parametrize(
+ "value",
+ ["degC \N{DEGREE SIGN}", "na\N{LATIN SMALL LETTER I WITH DIAERESIS}ve",
+ "m s\N{SUPERSCRIPT MINUS}\N{SUPERSCRIPT ONE}"],
+ )
+ def test_round_trips_unchanged(self, grid, path, value):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["long_name"] = value
+ assert variable.attributes["long_name"] == value
+ grid.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.variables["air"].attributes["long_name"] == value
+
+ def test_is_not_coerced_to_bytes(self):
+ value = "degC \N{DEGREE SIGN}"
+ assert _dataset._bytes_if_ascii(value) is value
+
+ def test_global_non_ascii_round_trips(self, writer, path):
+ value = "Produced at M\N{LATIN SMALL LETTER E WITH ACUTE}t\N{LATIN SMALL LETTER E WITH ACUTE}o"
+ writer.attributes["institution"] = value
+ writer.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.attributes["institution"] == value
+```
+
+- [ ] **Step 11: Run the tests to verify they pass**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/`
+Expected: PASS. These should pass on the implementation as written; if any
+fails, the coercion is over-reaching and the `except` clause is what to look
+at, not the test.
+
+Then the saver's own suite, which is the largest consumer of `_bytes_if_ascii`:
+
+Run: `pytest -n auto lib/iris/tests/unit/fileformats/netcdf/ lib/iris/tests/integration/`
+Expected: 0 failed, and the same skip count as before this task.
+
+- [ ] **Step 12: Green check and commit**
+
+```bash
+pre-commit run --files lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py
+git add lib/iris/fileformats/netcdf/_dataset.py lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py
+git commit -m "$(cat <<'EOF'
+Write netCDF attributes through the CFDataset interface
+
+_NetCDFAttributes.__setitem__ applies the ASCII coercion that saver.py's
+_setncattr applied, so the coercion stays on the netCDF path - as the
+spec requires - instead of following CF attributes into generic code.
+
+_bytes_if_ascii moves from saver.py to _dataset.py; saver.py imports it
+back until its own _setncattr goes.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 5: the shared `CFDataset` contract test
+
+Spec section 6: *"The `CFDataset` contract gets one shared test body run
+against both implementations."* This is that body. PR 4 adds
+`TestZarrDatasetContract(CFDatasetContract)` and inherits every test below
+without writing one of them again.
+
+**Files:**
+- Create: `lib/iris/tests/unit/fileformats/cf/dataset/contract.py`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py`
+
+**Interfaces:**
+- Consumes: `CFDataset`, `CFDatasetVariable` (Task 1); `NetCDFDataset`
+ (Tasks 3 and 4).
+- Produces:
+ - `CFDatasetContract` — a test body with **no** `Test` prefix, so pytest
+ does not collect it where it is defined. A subclass must supply three
+ fixtures: `readable`, `writable`, and `reopen` (a zero-argument callable
+ returning a fresh read-mode dataset over what `writable` wrote).
+ - `populate(dataset)` — fills an empty writable dataset with the canonical
+ contents, using only the `CFDataset` API.
+ - the constants `CONTRACT_DIMENSIONS`, `CONTRACT_DATA`,
+ `CONTRACT_VARIABLE_ATTRS`, `CONTRACT_GLOBALS`.
+
+- [ ] **Step 1: Write the contract body**
+
+`lib/iris/tests/unit/fileformats/cf/dataset/contract.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""One test body, run against every :class:`CFDataset` implementation.
+
+Subclass :class:`CFDatasetContract` in a module named for the implementation,
+supply the three fixtures it declares, and every test here runs against it.
+The class deliberately has no ``Test`` prefix, so pytest does not collect it
+where it is written, only where it is subclassed.
+
+Spec section 6: the interface is only worth having if both sides of it agree,
+and agreement is cheapest to check by asking the same questions twice.
+"""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+
+#: The canonical dataset every implementation is checked against.
+CONTRACT_DIMENSIONS = {"y": 3, "x": 2}
+CONTRACT_DATA = np.arange(6, dtype="f8").reshape(3, 2)
+CONTRACT_VARIABLE_ATTRS = {"units": "K", "long_name": "surface temperature"}
+CONTRACT_GLOBALS = {"Conventions": "CF-1.7", "title": "contract"}
+
+
+def populate(dataset):
+ """Fill an empty writable dataset with the canonical contents.
+
+ Uses only the CFDataset API, so this doubles as the write half of the
+ contract: an implementation that cannot build this cannot be used to save.
+ """
+ for name, size in CONTRACT_DIMENSIONS.items():
+ dataset.create_dimension(name, size)
+ for name, value in CONTRACT_GLOBALS.items():
+ dataset.attributes[name] = value
+
+ variable = dataset.create_variable(
+ "air_temperature", np.dtype("f8"), tuple(CONTRACT_DIMENSIONS), fill_value=-1.0
+ )
+ for name, value in CONTRACT_VARIABLE_ATTRS.items():
+ variable.attributes[name] = value
+ variable[:] = CONTRACT_DATA
+
+ # A dimensionless variable: what a grid mapping is, and the one shape a
+ # CFDatasetVariable must handle without a leading dimension.
+ scalar = dataset.create_variable("grid", np.dtype("i4"))
+ scalar.attributes["grid_mapping_name"] = "latitude_longitude"
+ scalar[()] = 0
+ return dataset
+
+
+class CFDatasetContract:
+ """What every CFDataset implementation must do, whatever it stores into."""
+
+ @pytest.fixture
+ def readable(self):
+ """Return an open read-mode dataset holding what populate() writes."""
+ raise NotImplementedError("supply a 'readable' fixture")
+
+ @pytest.fixture
+ def writable(self):
+ """Return an open, empty, write-mode dataset."""
+ raise NotImplementedError("supply a 'writable' fixture")
+
+ @pytest.fixture
+ def reopen(self):
+ """Return a callable giving a fresh read-mode dataset over 'writable'."""
+ raise NotImplementedError("supply a 'reopen' fixture")
+
+ # -- The dataset itself ------------------------------------------------
+
+ def test_is_a_cf_dataset(self, readable):
+ assert isinstance(readable, CFDataset)
+
+ def test_location_is_a_non_empty_string(self, readable):
+ assert isinstance(readable.location, str)
+ assert readable.location
+
+ def test_mode_is_a_read_mode(self, readable):
+ assert readable.mode in ("r", "r+", "a")
+
+ def test_starts_open(self, readable):
+ assert readable.closed is False
+
+ def test_close_then_closed(self, writable):
+ writable.close()
+ assert writable.closed is True
+
+ def test_close_is_idempotent(self, writable):
+ writable.close()
+ writable.close()
+ assert writable.closed is True
+
+ def test_context_manager_returns_self_and_closes(self, writable):
+ with writable as entered:
+ assert entered is writable
+ assert writable.closed is True
+
+ def test_dimensions(self, readable):
+ assert dict(readable.dimensions) == CONTRACT_DIMENSIONS
+
+ def test_global_attributes(self, readable):
+ assert dict(readable.attributes) == CONTRACT_GLOBALS
+
+ def test_variable_names(self, readable):
+ assert sorted(readable.variables) == ["air_temperature", "grid"]
+
+ # -- A variable --------------------------------------------------------
+
+ @pytest.fixture
+ def variable(self, readable):
+ return readable.variables["air_temperature"]
+
+ @pytest.fixture
+ def scalar(self, readable):
+ return readable.variables["grid"]
+
+ def test_variable_is_a_cf_dataset_variable(self, variable):
+ assert isinstance(variable, CFDatasetVariable)
+
+ def test_variable_name(self, variable):
+ assert variable.name == "air_temperature"
+
+ def test_variable_location_matches_its_dataset(self, variable, readable):
+ assert variable.location == readable.location
+
+ def test_variable_dimensions(self, variable):
+ assert variable.dimensions == tuple(CONTRACT_DIMENSIONS)
+
+ def test_variable_shape(self, variable):
+ assert variable.shape == tuple(CONTRACT_DIMENSIONS.values())
+
+ def test_variable_dtype(self, variable):
+ assert variable.dtype == np.dtype("f8")
+
+ def test_variable_size(self, variable):
+ assert variable.size == CONTRACT_DATA.size
+
+ def test_variable_ndim(self, variable):
+ assert variable.ndim == CONTRACT_DATA.ndim
+
+ def test_variable_len(self, variable):
+ assert len(variable) == CONTRACT_DATA.shape[0]
+
+ def test_variable_data(self, variable):
+ np.testing.assert_array_equal(variable[:], CONTRACT_DATA)
+
+ def test_variable_data_indexed(self, variable):
+ np.testing.assert_array_equal(variable[1], CONTRACT_DATA[1])
+
+ def test_variable_attributes(self, variable):
+ for name, value in CONTRACT_VARIABLE_ATTRS.items():
+ assert variable.attributes[name] == value
+
+ def test_variable_attributes_omit_unset_names(self, variable):
+ assert "nonesuch" not in variable.attributes
+ with pytest.raises(KeyError):
+ variable.attributes["nonesuch"]
+
+ def test_variable_fill_value(self, variable):
+ assert variable.fill_value == -1.0
+
+ def test_variable_chunking_is_none_or_a_shape(self, variable):
+ chunking = variable.chunking
+ assert chunking is None or (
+ isinstance(chunking, tuple)
+ and len(chunking) == len(variable.shape)
+ and all(isinstance(size, int) for size in chunking)
+ )
+
+ def test_scalar_variable(self, scalar):
+ assert scalar.dimensions == ()
+ assert scalar.shape == ()
+ with pytest.raises(TypeError, match="unsized"):
+ len(scalar)
+
+ # -- Writing -----------------------------------------------------------
+
+ def test_populate_then_read_back(self, writable, reopen):
+ populate(writable)
+ writable.close()
+ with reopen() as reader:
+ np.testing.assert_array_equal(
+ reader.variables["air_temperature"][:], CONTRACT_DATA
+ )
+ assert dict(reader.dimensions) == CONTRACT_DIMENSIONS
+ assert dict(reader.attributes) == CONTRACT_GLOBALS
+
+ def test_created_variable_is_visible_on_the_dataset(self, writable):
+ writable.create_dimension("x", 2)
+ created = writable.create_variable("thing", np.dtype("f4"), ("x",))
+ assert writable.variables["thing"].name == created.name
+
+ def test_create_variable_without_dimensions(self, writable):
+ # Finding F4: saver.py:2080 supplies only a name and a dtype.
+ variable = writable.create_variable("grid", np.dtype("i4"))
+ assert variable.dimensions == ()
+
+ def test_setitem_then_getitem(self, writable):
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("thing", np.dtype("f4"), ("x",))
+ variable[:] = [1.0, 2.0, 3.0]
+ np.testing.assert_array_equal(variable[:], [1.0, 2.0, 3.0])
+
+ def test_attribute_write_through(self, writable, reopen):
+ populate(writable)
+ writable.variables["air_temperature"].attributes["comment"] = "added"
+ writable.close()
+ with reopen() as reader:
+ assert reader.variables["air_temperature"].attributes["comment"] == "added"
+
+ def test_write_handle_works_after_close(self, writable, reopen):
+ populate(writable)
+ handle = writable.variables["air_temperature"].write_handle()
+ writable.close()
+
+ payload = CONTRACT_DATA * 10
+ handle[:] = payload
+ with reopen() as reader:
+ np.testing.assert_array_equal(
+ reader.variables["air_temperature"][:], payload
+ )
+
+ def test_sync_and_finalise_are_callable(self, writable):
+ populate(writable)
+ assert writable.sync() is None
+ assert writable.finalise() is None
+ assert writable.closed is False
+```
+
+- [ ] **Step 2: Write the netCDF subclass**
+
+`lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Run the shared CFDataset contract against the netCDF implementation."""
+
+import pytest
+
+from iris.fileformats.netcdf import _dataset
+from iris.tests.unit.fileformats.cf.dataset.contract import (
+ CFDatasetContract,
+ populate,
+)
+
+
+class TestNetCDFDatasetContract(CFDatasetContract):
+ """The netCDF side of the contract. PR 4 adds the Zarr side."""
+
+ @pytest.fixture
+ def readable(self, tmp_path):
+ path = tmp_path / "contract.nc"
+ with _dataset.NetCDFDataset(
+ path, mode="w", netcdf_format="NETCDF4"
+ ) as dataset:
+ populate(dataset)
+ with _dataset.NetCDFDataset(path) as dataset:
+ yield dataset
+
+ @pytest.fixture
+ def writable(self, tmp_path):
+ dataset = _dataset.NetCDFDataset(
+ tmp_path / "written.nc", mode="w", netcdf_format="NETCDF4"
+ )
+ yield dataset
+ dataset.close()
+
+ @pytest.fixture
+ def reopen(self, tmp_path):
+ def _reopen():
+ return _dataset.NetCDFDataset(tmp_path / "written.nc")
+
+ return _reopen
+```
+
+The `writable` fixture closes unconditionally on teardown, which is why
+`CFDataset.close()` has to be idempotent — several contract tests close it
+themselves first.
+
+- [ ] **Step 3: Run the contract to verify it fails**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py`
+Expected: collection error —
+`ModuleNotFoundError: No module named 'iris.tests.unit.fileformats.cf.dataset.contract'`
+
+- [ ] **Step 4: Run the contract to verify it passes**
+
+Once both files exist:
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py -v`
+Expected: PASS — every test, 0 failed, 0 skipped.
+
+Anything that fails here is a real disagreement between Tasks 1–4 and the
+interface as written; fix the implementation, not the contract. The contract
+is what PR 4 inherits, so a test loosened here to fit netCDF stops being a
+check on Zarr.
+
+- [ ] **Step 5: Confirm the contract body is not collected where it lives**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/dataset/ --collect-only -q | grep -c contract
+```
+Expected: `0`. `contract.py` does not match `python_files`, and
+`CFDatasetContract` does not match `python_classes`; both must hold, because
+the fixtures raise `NotImplementedError` and would report as errors.
+
+- [ ] **Step 6: Green check and commit**
+
+```bash
+pre-commit run --files lib/iris/tests/unit/fileformats/cf/dataset/contract.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py
+pytest lib/iris/tests/unit/fileformats/cf/ lib/iris/tests/unit/fileformats/netcdf/dataset/
+git add lib/iris/tests/unit/fileformats/cf/dataset/contract.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py
+git commit -m "$(cat <<'EOF'
+Add the shared CFDataset contract test
+
+One test body, subclassed per implementation, as the spec's testing
+section requires. PR 4 supplies three fixtures and inherits the lot.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 6: `CFVariable.attributes`, and `__getattr__` rewired onto it
+
+The pivot of the PR. `cf_data` is still a netCDF4 variable at the end of this
+task — only where CF attributes are *read from* changes. Tasks 8, 9 and 10
+then rewrite the consumers against `.attributes`, and Task 11 swaps the type
+underneath them.
+
+Spec section 4.3, and its warning that this is "the sharpest edge in PR 2".
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/_variables.py:97-250`
+- Modify: `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py:137-180`
+- Modify: `lib/iris/tests/unit/fileformats/cf/identify_mixins.py:21-35`
+- Test: `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py`
+
+**Interfaces:**
+- Consumes: `TrackedAttributes` from Task 1.
+- Produces, on every `CFVariable` subclass:
+ - `attributes: TrackedAttributes` — the CF attributes, and the record of
+ which have been read. Tasks 8, 9, 10 and 12 all read through it.
+ - typed properties `dimensions: tuple`, `shape: tuple`, `ndim: int`,
+ `dtype`, `size: int`, all forwarding to `cf_data`.
+ - `__getattr__(name)` resolving against `attributes`, falling back to
+ `_deprecated_netcdf_member(name)`, and **not** caching onto the instance.
+ - `cf_attrs()`, `cf_attrs_ignored()`, `cf_attrs_used()`,
+ `cf_attrs_unused()`, `cf_attrs_reset()` — same signatures and same
+ `tuple[tuple[str, Any], ...]` returns as before, now computed from the
+ mapping.
+- Removes: `CFVariable._nc_attrs` and `CFVariable._cf_attrs`. Nothing outside
+ `_variables.py` and its own unit test reads either — confirmed by
+ `grep -rn "_nc_attrs\|_cf_attrs" lib/`.
+
+- [ ] **Step 1: Rewrite PR 1's two caching tests to describe the new contract**
+
+In `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py`, replace
+`TestAttributeAccess.test_cached` (line 138) and
+`test_getattr_non_ncattr_value_is_cached_but_not_marked_used` (line 166) —
+the only two tests in the PR 1 suite that name the instance cache, and so the
+only two this PR is entitled to change (finding F7):
+
+```python
+class TestAttributeAccess:
+ def test_reads_are_not_cached_on_the_instance(self, nc_var):
+ # The setattr cache is gone. It made a re-read after cf_attrs_reset()
+ # invisible to attribute tracking, which decided which attributes
+ # reached the cube - see TestReadAfterReset below.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert "coordinates" not in cf_var.__dict__
+ assert cf_var.coordinates == "x y"
+ assert "coordinates" not in cf_var.__dict__
+ assert cf_var.coordinates == "x y"
+ assert "coordinates" not in cf_var.__dict__
+
+ def test_attributes_are_read_from_the_file_once(self, nc_var):
+ # Materialised at construction, so repeated reads cost nothing and
+ # cf_attrs_unused() does not have to go back to the file to answer.
+ cf_var = CFVariableSub("foo", nc_var)
+ assert nc_var.ncattrs.call_count == 1
+
+ _ = cf_var.coordinates
+ _ = cf_var.standard_name
+ assert nc_var.ncattrs.call_count == 1
+ assert nc_var.getncattr.call_count == 3
+
+ def test_getattr_of_a_non_attribute_reaches_the_variable(self, nc_var):
+ # The one-cycle compatibility route: a netCDF4 member that is not a
+ # CF attribute still resolves, and is still not marked as used.
+ nc_var.not_an_ncattr = 42
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.not_an_ncattr == 42
+ assert "not_an_ncattr" not in cf_var.__dict__
+ assert "not_an_ncattr" not in dict(cf_var.cf_attrs())
+ assert "not_an_ncattr" not in dict(cf_var.cf_attrs_used())
+
+ def test_getattr_of_nothing_at_all_raises_attribute_error(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ # A MagicMock invents any member, so ask a real object instead.
+ cf_var.cf_data = object()
+ with pytest.raises(AttributeError, match="nonesuch"):
+ cf_var.nonesuch
+```
+
+- [ ] **Step 2: Add the tests for the mapping itself and the typed properties**
+
+Append to `test_CFVariable.py`:
+
+```python
+class TestAttributesMapping:
+ def test_attributes_contents(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert dict(cf_var.attributes) == {
+ "coordinates": "x y",
+ "standard_name": "air_temperature",
+ "_FillValue": -999,
+ }
+
+ def test_getattr_and_mapping_are_the_same_read(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ _ = cf_var.coordinates
+ assert cf_var.attributes.read == frozenset(["_FillValue", "coordinates"])
+
+ cf_var.cf_attrs_reset()
+ _ = cf_var.attributes["coordinates"]
+ assert cf_var.attributes.read == frozenset(["_FillValue", "coordinates"])
+
+ def test_untracked_read_is_not_recorded(self, nc_var):
+ # What helpers.py's flag-attribute probe needs: a look that does not
+ # count, so the attribute still reaches the cube.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.attributes.untracked["coordinates"] == "x y"
+ assert "coordinates" in dict(cf_var.cf_attrs_unused())
+
+
+class TestTypedProperties:
+ def test_dimensions_is_a_tuple(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ assert cf_var.dimensions == ("time", "lat")
+
+ def test_shape_ndim_dtype_size(self, mocker, nc_var):
+ nc_var.shape = (3, 4)
+ nc_var.dtype = np.dtype("f4")
+ nc_var.size = 12
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.shape == (3, 4)
+ assert cf_var.ndim == 2
+ assert cf_var.dtype == np.dtype("f4")
+ assert cf_var.size == 12
+
+ def test_a_file_attribute_does_not_reach_the_typed_properties(self, nc_var):
+ # The properties are looked up before __getattr__ runs, so a file
+ # attribute called "shape" cannot displace the variable's shape.
+ nc_var.ncattrs.return_value = ["shape"]
+ nc_var.getncattr.side_effect = {"shape": "not a shape"}.__getitem__
+ nc_var.shape = (2,)
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.shape == (2,)
+ assert cf_var.attributes["shape"] == "not a shape"
+```
+
+Add `import numpy as np` at the top of the file.
+
+- [ ] **Step 3: Give the identify() stub a `getncattr`**
+
+`CFVariable.__init__` now materialises attribute *values*, not just names, so
+every stub standing in for a netCDF variable needs the method that fetches
+them. In `lib/iris/tests/unit/fileformats/cf/identify_mixins.py`, add to
+`_NetCDFVar`:
+
+```python
+ def getncattr(self, name):
+ return getattr(self, name)
+```
+
+`test_CFCoordinateVariable.py`'s stub returns `[]` from `ncattrs()`, so it
+needs nothing. `test_CFReader.py` uses `MagicMock`, which supplies
+`getncattr` already.
+
+- [ ] **Step 4: Run the tests to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/test_CFVariable.py`
+Expected: FAIL — `AttributeError` on `cf_var.attributes` (resolved through the
+old `__getattr__` to the mock, so it will report as a wrong *value* rather
+than a missing member: `assert dict(<MagicMock ...>) == {...}` raising
+`TypeError`), and `assert "coordinates" not in cf_var.__dict__` failing on
+the second read.
+
+- [ ] **Step 5: Rewire `CFVariable`**
+
+In `lib/iris/fileformats/cf/_variables.py`, add below `_CF_ATTRS_IGNORE`:
+
+```python
+# Names __getattr__ must never resolve through self.attributes, because
+# reading self.attributes is how it resolves anything at all. Deliberately
+# not "every underscore-prefixed name": loader.py and saver.py both probe
+# for "_data_array", and that probe has to reach through.
+_GETATTR_RECURSION_GUARD = frozenset(["attributes", "cf_data"])
+
+
+def _attributes_source(data):
+ """Return a mapping of the CF attributes of a variable's backing store."""
+ # PR 2 transitional. Task 11 deletes this: once CFReader supplies
+ # CFDatasetVariables, __init__ reads data.attributes directly.
+ if isinstance(data, CFDatasetVariable):
+ return data.attributes
+ return {name: data.getncattr(name) for name in data.ncattrs()}
+```
+
+`isinstance`, not `hasattr(data, "attributes")`. Half the CF test suite hands
+this function a `MagicMock`, and a `MagicMock` has every attribute you ask it
+for — including `attributes`, which would send it down the wrong branch and
+put a mock where the mapping belongs.
+
+Add `CFDatasetVariable` to the imports too:
+
+```python
+from iris.fileformats.cf.dataset import CFDatasetVariable, TrackedAttributes
+```
+
+`cf/dataset.py` imports nothing from `iris.fileformats.netcdf` (Task 1,
+Step 10), so this does not breach PR 1's
+`test_only_the_reader_is_coupled_to_netcdf`.
+
+Replace `CFVariable.__init__`:
+
+```python
+ def __init__(self, name, data):
+ self.cf_name = name
+ """NetCDF variable name."""
+
+ self.cf_data = data
+ """The variable's storage: a CFDatasetVariable."""
+
+ self.attributes = TrackedAttributes(
+ _attributes_source(data), ignored=_CF_ATTRS_IGNORE
+ )
+ """The variable's CF attributes, and a record of which have been read.
+
+ The only place CF attributes live. ``__getattr__`` forwards here, so
+ ``cf_var.units`` and ``cf_var.attributes["units"]`` are the same read
+ and count once. What was read decides what survives onto the loaded
+ cube - see :func:`iris.fileformats.netcdf.loader._add_unused_attributes`.
+ """
+
+ """File source of the NetCDF content."""
+ try:
+ self.filename = data.group().filepath()
+ except AttributeError:
+ self.filename = "<unknown_filename>"
+
+ self.cf_group = None
+ """Collection of CF-netCDF variables associated with this variable."""
+
+ self.cf_terms_by_root = {}
+ """CF-netCDF formula terms that his variable participates in."""
+
+ self._to_be_promoted = False
+```
+
+The trailing `self.cf_attrs_reset()` goes: `TrackedAttributes` seeds itself
+from `ignored` at construction, which is what that call was for.
+
+Replace `__getattr__` (line 203) with:
+
+```python
+ @property
+ def dimensions(self) -> tuple:
+ """The names of the dimensions this variable spans, in order."""
+ return tuple(self.cf_data.dimensions)
+
+ @property
+ def shape(self) -> tuple:
+ """The variable's shape."""
+ return self.cf_data.shape
+
+ @property
+ def ndim(self) -> int:
+ """The number of dimensions this variable spans."""
+ return len(self.shape)
+
+ @property
+ def dtype(self):
+ """The variable's stored data type."""
+ return self.cf_data.dtype
+
+ @property
+ def size(self) -> int:
+ """The total number of elements in the variable."""
+ return self.cf_data.size
+
+ def __getattr__(self, name):
+ """Return the named CF attribute, as read from the file.
+
+ The open-ended half of this class. CF attribute names are data read
+ from a file, not API, so they cannot be declared - which is what
+ justifies ``__getattr__`` here at all. It resolves only against
+ :attr:`attributes`; everything structural is a declared property
+ above. Reading through it records the attribute as used, exactly as
+ ``cf_var.attributes[name]`` does.
+
+ """
+ if name.startswith("__") or name in _GETATTR_RECURSION_GUARD:
+ # Dunder probes - copy, pickle, numpy protocols - must not be
+ # answered from file data, and the two members this method reads
+ # must not be resolved by this method.
+ raise AttributeError(name)
+
+ try:
+ return self.attributes[name]
+ except KeyError:
+ pass
+ return self._deprecated_netcdf_member(name)
+
+ def _deprecated_netcdf_member(self, name):
+ """Return a netCDF4 member of the backing variable, as this class used to.
+
+ The one-cycle compatibility route for code that reached netCDF4 API
+ through a CFVariable. Records nothing: this is not a CF attribute.
+
+ """
+ # PR 2 transitional. Task 11 replaces the body with
+ # value = self.cf_data.deprecated_netcdf_member(name)
+ # and the deprecation warning, once no Iris code takes this path.
+ return getattr(self.cf_data, name)
+```
+
+Replace the five `cf_attrs_*` helpers (lines 226-250):
+
+```python
+ def cf_attrs(self):
+ """Return a list of all attribute name and value pairs of the CF-netCDF variable."""
+ return tuple(
+ (name, self.attributes.untracked[name]) for name in sorted(self.attributes)
+ )
+
+ def cf_attrs_ignored(self):
+ """Return a list of all ignored attribute name and value pairs of the CF-netCDF variable."""
+ names = set(self.attributes) & _CF_ATTRS_IGNORE
+ return tuple(
+ (name, self.attributes.untracked[name]) for name in sorted(names)
+ )
+
+ def cf_attrs_used(self):
+ """Return a list of all accessed attribute name and value pairs of the CF-netCDF variable."""
+ return tuple(
+ (name, self.attributes.untracked[name])
+ for name in sorted(self.attributes.read)
+ )
+
+ def cf_attrs_unused(self):
+ """Return a list of all non-accessed attribute name and value pairs of the CF-netCDF variable."""
+ return tuple(
+ (name, self.attributes.untracked[name])
+ for name in sorted(self.attributes.unread)
+ )
+
+ def cf_attrs_reset(self):
+ """Reset the history of accessed attribute names of the CF-netCDF variable."""
+ self.attributes.reset()
+```
+
+Every one of these reads through `.untracked`: a report on what was read must
+not itself count as reading.
+
+`__getitem__`, `__len__`, `__repr__`, `__eq__`, `__ne__`, `__hash__` and
+`spans` are unchanged. `spans` already reads `self.dimensions`, which is now
+the declared property rather than a `__getattr__` hop.
+
+- [ ] **Step 6: Run the tests to verify they pass**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/`
+Expected: PASS — 0 failed, 1 skipped (§6.2; the skip is `test_CFReader.py`'s
+FUTURE-context case and is expected). The passed count will be higher than
+§6.2's 306, by the tests Tasks 1–5 added under `cf/dataset/`. This includes
+every `test_CF*Variable.py` `identify()` test, which is the broadest check
+that the rewiring did not change what the CF layer sees.
+
+- [ ] **Step 7: Run the consumers, which still take the fallback**
+
+Nothing outside `cf/` has been touched yet, so every `getattr(cf_var, ...)`
+in `_nc_load_rules/`, `netcdf/loader.py` and `netcdf/ugrid_load.py` must still
+work — through `attributes` for CF attributes, through
+`_deprecated_netcdf_member` for the rest.
+
+Run: `pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/`
+Expected: `0 failed, 26 skipped`, plus F14's ten `test_coord_systems.py`
+errors and nothing else (§6.2). The skip count must be *identical* to the
+baseline, not merely small.
+
+If `lib/iris/tests/integration/test_cf.py::test_variable_attribute_touch_pass_0`
+fails, read it before changing anything: it pins the tracking semantics on a
+real file and is a PR 1 test that must pass untouched.
+
+- [ ] **Step 8: Commit the rewiring**
+
+```bash
+pre-commit run --files lib/iris/fileformats/cf/_variables.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFVariable.py \
+ lib/iris/tests/unit/fileformats/cf/identify_mixins.py
+git add lib/iris/fileformats/cf/_variables.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFVariable.py \
+ lib/iris/tests/unit/fileformats/cf/identify_mixins.py
+git commit -m "$(cat <<'EOF'
+Resolve CFVariable attributes against a declared mapping
+
+CFVariable.__getattr__ now reads self.attributes - a TrackedAttributes -
+instead of reaching for arbitrary members of the netCDF4 variable, and no
+longer caches what it finds onto the instance. dimensions, shape, ndim,
+dtype and size become declared properties, so structure and file data are
+no longer reached for through the same syntax.
+
+The instance cache had to go: it made a re-read after cf_attrs_reset()
+invisible to attribute tracking, and tracking is what decides which file
+attributes survive onto a loaded cube.
+
+cf_data is still a netCDF4 variable; that swap is later in this PR.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+- [ ] **Step 9: Review Focus 1 — write the failing test for a shadowing attribute**
+
+A file is free to carry an attribute called `filename`, `cf_name`, `spans`,
+`attributes` or `cf_data`. `__getattr__` runs only when normal lookup fails,
+so for those five names it never runs and the class member wins. The spec
+calls this a known limitation; what it must not be is a *silent* one, and
+what must never happen is the attribute vanishing from the cube because
+nothing recorded whether it was read.
+
+Append to `test_CFVariable.py`:
+
+```python
+#: CFVariable members that a CF attribute of the same name cannot displace.
+SHADOWED_NAMES = ["filename", "cf_name", "spans", "attributes", "cf_data"]
+
+
+class TestShadowedAttributeNames:
+ """Review Focus 1. Spec section 4.3's known limitation, pinned."""
+
+ @pytest.fixture
+ def shadowing(self, nc_var):
+ nc_var.ncattrs.return_value = SHADOWED_NAMES + ["units"]
+ nc_var.getncattr.side_effect = (
+ {name: f"file value of {name}" for name in SHADOWED_NAMES} | {"units": "K"}
+ ).__getitem__
+ return CFVariableSub("foo", nc_var)
+
+ @pytest.mark.parametrize("name", SHADOWED_NAMES)
+ def test_the_class_member_wins(self, shadowing, name):
+ assert getattr(shadowing, name) != f"file value of {name}"
+
+ @pytest.mark.parametrize("name", SHADOWED_NAMES)
+ def test_the_file_value_is_still_reachable(self, shadowing, name):
+ assert shadowing.attributes[name] == f"file value of {name}"
+
+ @pytest.mark.parametrize("name", SHADOWED_NAMES)
+ def test_reading_it_through_the_mapping_marks_it_used(self, shadowing, name):
+ assert name in dict(shadowing.cf_attrs_unused())
+ _ = shadowing.attributes[name]
+ assert name in dict(shadowing.cf_attrs_used())
+ assert name not in dict(shadowing.cf_attrs_unused())
+
+ @pytest.mark.parametrize("name", SHADOWED_NAMES)
+ def test_it_is_listed_among_the_variables_attributes(self, shadowing, name):
+ assert name in dict(shadowing.cf_attrs())
+
+ def test_a_shadowing_name_does_not_disturb_its_neighbours(self, shadowing):
+ assert shadowing.units == "K"
+ assert dict(shadowing.cf_attrs_used())["units"] == "K"
+```
+
+- [ ] **Step 10: Run them**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/test_CFVariable.py -k Shadowed -v`
+Expected: PASS on the implementation from Step 5. If
+`test_the_file_value_is_still_reachable` fails, `__init__` is dropping
+attributes whose names collide with members — which is the failure this whole
+class exists to catch.
+
+- [ ] **Step 11: Review Focus 2 — `getattr` with a default, and `hasattr`**
+
+`_nc_load_rules/helpers.py` alone makes 63 of these calls, almost all
+`getattr(cf_var, SOME_CF_ATTR, None)`. They rely on `__getattr__` raising
+`AttributeError` and nothing else: a `KeyError` escaping from the mapping, or
+a `TypeError` from an unhashable name, would not be caught by the default and
+would abort the load. And `hasattr` must keep marking the attribute used, or
+an attribute Iris consulted would go on to be copied onto the cube as well.
+
+Append to `test_CFVariable.py`:
+
+```python
+class TestGetattrAndHasattr:
+ """Review Focus 2. The two call shapes the loading rules actually use."""
+
+ def test_getattr_with_a_default_finds_the_attribute(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ assert getattr(cf_var, "coordinates", None) == "x y"
+
+ def test_getattr_with_a_default_returns_the_default(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+ assert getattr(cf_var, "nonesuch", None) is None
+ assert getattr(cf_var, "nonesuch", "fallback") == "fallback"
+
+ def test_a_missing_attribute_raises_attribute_error_not_key_error(self, nc_var):
+ # getattr(..., default) only swallows AttributeError. A KeyError from
+ # the mapping would escape and abort the load.
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+ with pytest.raises(AttributeError):
+ cf_var.nonesuch
+
+ def test_getattr_of_a_default_does_not_record_a_read(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+ _ = getattr(cf_var, "nonesuch", None)
+ assert cf_var.cf_attrs_used() == (("_FillValue", -999),)
+
+ def test_hasattr_true_marks_the_attribute_used(self, nc_var):
+ # Parity with the old behaviour: hasattr went through __getattr__,
+ # which added the name to the used set.
+ cf_var = CFVariableSub("foo", nc_var)
+ assert hasattr(cf_var, "coordinates")
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+
+ def test_hasattr_false_marks_nothing(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+ assert not hasattr(cf_var, "nonesuch")
+ assert cf_var.cf_attrs_used() == (("_FillValue", -999),)
+
+ def test_dunder_probes_do_not_reach_the_file(self, nc_var):
+ # copy, pickle and numpy all probe for dunders. Answering one from
+ # file data would make a CFVariable behave as whatever the file says.
+ nc_var.ncattrs.return_value = ["__array__"]
+ nc_var.getncattr.side_effect = {"__array__": "nonsense"}.__getitem__
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert not hasattr(cf_var, "__array_interface__")
+ assert cf_var.attributes.untracked["__array__"] == "nonsense"
+
+ @pytest.mark.parametrize("name", ["attributes", "cf_data"])
+ def test_the_recursion_guard_holds_before_init_completes(self, name):
+ # An instance whose __init__ never ran - what copy and pickle build -
+ # must raise, not recurse until the stack is gone.
+ cf_var = CFVariableSub.__new__(CFVariableSub)
+ with pytest.raises(AttributeError, match=name):
+ getattr(cf_var, name)
+```
+
+- [ ] **Step 12: Run them**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/test_CFVariable.py -k "GetattrAndHasattr" -v`
+Expected: PASS. A `RecursionError` on the last two means
+`_GETATTR_RECURSION_GUARD` is missing a name.
+
+- [ ] **Step 13: Review Focus 3 — a read repeated after `cf_attrs_reset()`**
+
+The behaviour the removed cache was hiding. `CFReader` resets every variable's
+touch history once it has finished parsing the file's structure, so any
+attribute the rules go on to read is one the rules used. With the cache, a
+name read during parsing and read again by the rules was served from
+`__dict__`, never re-recorded, and so was copied onto the cube as an "unused"
+attribute as well as being acted on. Removing the cache fixes that; this
+pins it so it cannot come back.
+
+Append to `test_CFVariable.py`:
+
+```python
+class TestReadAfterReset:
+ """Review Focus 3. The cache removal's whole point, at unit scale.
+
+ Task 11, Step 1 pins the same behaviour end to end, on a real file.
+ """
+
+ def test_a_re_read_after_reset_counts_again(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ # As CFReader does: read while parsing structure, then reset.
+ _ = cf_var.coordinates
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+ cf_var.cf_attrs_reset()
+ assert "coordinates" in dict(cf_var.cf_attrs_unused())
+
+ # As the loading rules then do: read it again.
+ _ = cf_var.coordinates
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+ assert "coordinates" not in dict(cf_var.cf_attrs_unused())
+
+ def test_the_same_holds_through_hasattr(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ _ = cf_var.coordinates
+ cf_var.cf_attrs_reset()
+
+ assert hasattr(cf_var, "coordinates")
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+
+ def test_reset_restores_the_ignored_names_only(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ _ = cf_var.coordinates
+ _ = cf_var.standard_name
+ cf_var.cf_attrs_reset()
+
+ assert cf_var.cf_attrs_used() == (("_FillValue", -999),)
+ assert cf_var.cf_attrs_unused() == (
+ ("coordinates", "x y"),
+ ("standard_name", "air_temperature"),
+ )
+
+ def test_values_are_unchanged_by_any_of_this(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ for _ in range(3):
+ assert cf_var.coordinates == "x y"
+ cf_var.cf_attrs_reset()
+```
+
+- [ ] **Step 14: Run them**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/test_CFVariable.py -k ReadAfterReset -v`
+Expected: PASS.
+
+- [ ] **Step 15: Green check and commit the Review Focus tests**
+
+```bash
+pre-commit run --files lib/iris/tests/unit/fileformats/cf/test_CFVariable.py
+pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/
+git add lib/iris/tests/unit/fileformats/cf/test_CFVariable.py
+git commit -m "$(cat <<'EOF'
+Pin the sharp edges of the CFVariable attribute rewiring
+
+Three classes of input the rewiring could plausibly break: a file
+attribute whose name collides with a CFVariable member; getattr with a
+default and hasattr, which the loading rules use 86 times between them
+and which depend on AttributeError and nothing else; and an attribute
+read again after cf_attrs_reset(), which is what removing the instance
+cache actually changes.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 7: One definition of "scalar" across the four `spans` methods
+
+Spec section 4.4 names this the programme's only `bugfix`: `CFVariable.spans`
+recognises NCZarr's `_scalar_` pseudo-dimension and the three subclasses that
+override `spans` do not. **Finding F9 corrects that** — measured against the
+real classes, the three overrides already return `True` for a source whose
+dimensions are exactly `("_scalar_",)`, because they open with
+`if self.dimensions:` and then compare `set(source[:-1])`, which for a
+one-element tuple is the empty set, a subset of everything. The gap is real
+in the source and unreachable in behaviour.
+
+So this task does not fix an outcome. It replaces four separate spellings of
+"is this variable scalar" with one, which matters from PR 4 on: `ZarrDataset`
+derives `dimensions` from `_ARRAY_DIMENSIONS` or NCZarr metadata (spec
+section 4.4), and the question "does `_scalar_` count as scalar here?" will
+be asked again by a backend that has never been through netCDF's rules.
+
+**Because the behaviour is already correct, this task inverts the usual
+cycle**: the tests are written first and expected to *pass*, recording what
+the code does today; the refactor must then leave them passing. Do not
+"correct" a step that says PASS where a step usually says FAIL.
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/_variables.py` — `CFVariable.spans`,
+ `CFBoundaryVariable.spans:420`, `CFClimatologyVariable.spans:496`,
+ `CFLabelVariable.spans:819`
+- Test: `lib/iris/tests/unit/fileformats/cf/identify_mixins.py` — `SpansMixin`,
+ which `test_CFBoundaryVariable.py`, `test_CFClimatologyVariable.py` and
+ `test_CFLabelVariable.py` all subclass, so one test there covers all three
+- Test: `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py` — `TestSpans`
+
+**Interfaces:**
+- Produces: `CFVariable._is_scalar() -> bool` — true for a variable with no
+ dimensions, and for one whose only dimension is
+ `_NCZARR_SCALAR_DIMENSION`. The single definition of scalar in the CF
+ layer; PR 4's Zarr work reads it rather than restating the rule.
+
+- [ ] **Step 1: Add the characterisation tests to the shared mixin**
+
+In `lib/iris/tests/unit/fileformats/cf/identify_mixins.py`, add to
+`SpansMixin`, after `test_empty_dimensions_spans`:
+
+```python
+ def test_nczarr_scalar_dimension_spans(self):
+ """An NCZarr scalar source variable always spans the target.
+
+ NCZarr spells a zero-dimensional variable as one dimension named
+ _scalar_, so this is the same case as test_empty_dimensions_spans
+ arriving from a different writer.
+ """
+ cf_source = self._make_cf_var("source_var", (_NCZARR_SCALAR_DIMENSION,))
+ cf_target = self._make_cf_var("target_var", ("x", "y"))
+ assert cf_source.spans(cf_target)
+
+ def test_nczarr_scalar_target_is_spanned_by_a_scalar(self):
+ cf_source = self._make_cf_var("source_var", (_NCZARR_SCALAR_DIMENSION,))
+ cf_target = self._make_cf_var("target_var", (_NCZARR_SCALAR_DIMENSION,))
+ assert cf_source.spans(cf_target)
+
+ def test_nczarr_scalar_is_not_a_dimension_name_to_match_on(self):
+ """_scalar_ marks a scalar; it is not a dimension two variables share.
+
+ A source that is not scalar has to justify itself dimension by
+ dimension, whether or not _scalar_ is among its names.
+ """
+ cf_source = self._make_cf_var("source_var", ("a", "b", "extra"))
+ cf_target = self._make_cf_var("target_var", (_NCZARR_SCALAR_DIMENSION,))
+ assert not cf_source.spans(cf_target)
+```
+
+Add the import at the top of the file:
+
+```python
+from iris.fileformats.cf._variables import _NCZARR_SCALAR_DIMENSION
+```
+
+- [ ] **Step 2: Add the matching tests for the base class**
+
+In `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py`, add to
+`TestSpans`:
+
+```python
+ def test_scalar_dimension_with_others_is_not_scalar(self, mocker, nc_var):
+ # _scalar_ means "this variable is scalar", which a variable with a
+ # second dimension is not. It is not a wildcard.
+ nc_var.dimensions = (_variables._NCZARR_SCALAR_DIMENSION, "bnds")
+ cf_var = CFVariableSub("not_scalar", nc_var)
+
+ other = mocker.MagicMock()
+ other.dimensions = ("time",)
+
+ assert not cf_var.spans(other)
+
+ def test_no_dimensions_always_spans(self, mocker, nc_var):
+ nc_var.dimensions = ()
+ cf_var = CFVariableSub("scalar", nc_var)
+
+ other = mocker.MagicMock()
+ other.dimensions = ("time",)
+
+ assert cf_var.spans(other)
+```
+
+- [ ] **Step 3: Run them, and expect PASS**
+
+Run:
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/test_CFVariable.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFBoundaryVariable.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFClimatologyVariable.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFLabelVariable.py -k "span or Span" -v
+```
+Expected: PASS, all of them, on unmodified code. That result **is** the
+finding — record the exact output in the pull request body, because it is the
+evidence for F9 and for softening the merge-back's `bugfix` wording.
+
+If any of them fails, F9 is wrong and the spec was right: stop, and say so in
+the pull request. The refactor below is then a genuine fix, and the merge-back
+owes the `bugfix` fragment as originally specified.
+
+- [ ] **Step 4: Commit the characterisation tests, before changing anything**
+
+```bash
+pre-commit run --files lib/iris/tests/unit/fileformats/cf/identify_mixins.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFVariable.py
+git add lib/iris/tests/unit/fileformats/cf/identify_mixins.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFVariable.py
+git commit -m "$(cat <<'EOF'
+Pin how spans() treats the NCZarr _scalar_ dimension
+
+Committed before the code changes, so the record shows these pass
+against unmodified spans() methods.
+
+The design spec calls the missing _NCZARR_SCALAR_DIMENSION check in the
+three overriding spans() methods a latent bug. It is latent in the
+strong sense: unreachable. Each override opens with "if self.dimensions"
+and then tests set(source[:-1]), which for the one-element tuple NCZarr
+produces is the empty set - a subset of every target. The overrides
+already answer True.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+- [ ] **Step 5: Give `CFVariable` one definition of scalar**
+
+In `lib/iris/fileformats/cf/_variables.py`, add to `CFVariable`, immediately
+above `spans`:
+
+```python
+ def _is_scalar(self) -> bool:
+ """Whether this variable is scalar, in either spelling of scalar.
+
+ NetCDF gives a zero-dimensional variable no dimensions at all.
+ NCZarr gives it one, named ``_scalar_``. The two mean the same
+ thing, and every ``spans`` implementation has to treat them alike.
+
+ """
+ return not self.dimensions or self.dimensions == (_NCZARR_SCALAR_DIMENSION,)
+```
+
+Rewrite the body of `CFVariable.spans` (keeping its docstring):
+
+```python
+ # Scalar variables always span the target variable.
+ if self._is_scalar():
+ return True
+
+ result = set(self.dimensions).issubset(cf_variable.dimensions)
+ return result
+```
+
+This drops the local `dimensions = tuple(self.dimensions)`: Task 6 made
+`CFVariable.dimensions` a property that already returns a tuple.
+
+- [ ] **Step 6: Use it in the three overrides**
+
+In each of `CFBoundaryVariable.spans`, `CFClimatologyVariable.spans` and
+`CFLabelVariable.spans`, replace `if self.dimensions:` with
+`if not self._is_scalar():`. Nothing else in those three bodies changes —
+including the comment above each subset test, which names a different extent
+dimension in each class and must keep doing so. The result in
+`CFBoundaryVariable`:
+
+```python
+ # Scalar variables always span the target variable.
+ result = True
+ if not self._is_scalar():
+ source = self.dimensions
+ target = cf_variable.dimensions
+ # Ignore the bounds extent dimension.
+ result = set(source[:-1]).issubset(target) or set(source[1:]).issubset(
+ target
+ )
+ return result
+```
+
+- [ ] **Step 7: Run the tests again, and expect the same PASS**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/`
+Expected: PASS, 0 failed, 1 skipped (the §6.2 FUTURE-context skip) — the
+same set of tests, the same results, over
+consolidated code. `test_CFReader.py::test_nczarr_scalar_grid_mapping_spans_data_var`
+is the one existing test that exercises `_scalar_` through the reader; it must
+still pass.
+
+- [ ] **Step 8: Run the loading path**
+
+Run: `pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/`
+Expected: `0 failed, 26 skipped`, plus F14's ten `test_coord_systems.py`
+errors and nothing else (§6.2). `spans` decides which coordinates attach to
+which cube, so a mistake here shows up as missing or duplicated coordinates
+rather than as an error.
+
+- [ ] **Step 9: Commit**
+
+```bash
+pre-commit run --files lib/iris/fileformats/cf/_variables.py
+git add lib/iris/fileformats/cf/_variables.py
+git commit -m "$(cat <<'EOF'
+Ask _is_scalar() rather than restate the rule four times
+
+CFVariable.spans and its three overrides each decided for themselves
+what a scalar variable is, and only one of the four knew about NCZarr's
+_scalar_ pseudo-dimension. The behaviour is unchanged - see the tests
+committed just before this - but the question is now asked in one place.
+
+That matters from PR 4: ZarrDataset derives dimension names from the
+store rather than from netCDF's rules, and will ask the same question of
+data that has never been through them.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 8: The loading rules read `.attributes`
+
+66 sites, 63 of them the same two lines apart. Mechanical, but it is the task
+that earns the `__getattr__` exemption in `lib/iris/AGENTS.md`: *"It must
+forward to a declared `Mapping`, that `Mapping` must be the path library code
+takes."* Until this task, the second half of that sentence is false.
+
+Nothing here changes behaviour. `getattr(cf_var, CF_ATTR_UNITS, None)` already
+resolves through `.attributes` after Task 6; this task makes the read say so.
+
+**Files:**
+- Modify: `lib/iris/fileformats/_nc_load_rules/helpers.py` — 63 sites between
+ line 567 and line 2077
+- Modify: `lib/iris/fileformats/_nc_load_rules/actions.py:179,560,626`
+- Test: `lib/iris/tests/unit/fileformats/nc_load_rules/helpers/` — the existing
+ modules, unchanged; they are the regression net for this task
+
+**Interfaces:**
+- Consumes: `CFVariable.attributes` from Task 6.
+- Produces: nothing new. This task removes the loading rules' dependence on
+ `CFVariable.__getattr__`, which is what lets Task 11 make the netCDF
+ fallback warn.
+
+**The rule, applied 63 times:**
+
+| Before | After |
+|---|---|
+| `getattr(cf_var, CF_ATTR_UNITS, None)` | `cf_var.attributes.get(CF_ATTR_UNITS)` |
+| `getattr(cf_grid_var, CF_ATTR_GRID_NORTH_POLE_LAT, 90.0)` | `cf_grid_var.attributes.get(CF_ATTR_GRID_NORTH_POLE_LAT, 90.0)` |
+| `getattr(cf_var, CF_ATTR_AXIS, "")` | `cf_var.attributes.get(CF_ATTR_AXIS, "")` |
+| `getattr(cf_grid_var, CF_ATTR_GRID_MAPPING_NAME)` | `cf_grid_var.attributes[CF_ATTR_GRID_MAPPING_NAME]` |
+
+A default of `None` is `Mapping.get`'s own default, so drop it; any other
+default is passed through. The constants keep their names — they are the
+greppable literals `lib/iris/AGENTS.md` asks for, and the point of this task
+is the object being asked, not the key.
+
+- [ ] **Step 1: Rewrite the 63 plain sites in `helpers.py`**
+
+Work top to bottom, applying the table above. The sites, by line number in the
+unmodified file:
+
+```
+567 616 625 745 774 775 776 784 786 789 832 833 840 863 864
+903 904 905 934 935 961 966 967 990 991 992 1020 1021 1045 1046
+1047 1075 1076 1104 1105 1106 1126 1136 1137 1149 1169 1220 1231 1232 1267
+1268 1836 1844 1850 1855 1865 1905 1915 1929 1931 1932 1948 1964 2012 2017
+2046 2077
+```
+
+Line 1905 and line 1915 each hold two `getattr` calls in one expression:
+
+```python
+ attr_name = getattr(cf_var, CF_ATTR_STD_NAME, None) or getattr(
+ cf_var, CF_ATTR_LONG_NAME, None
+ )
+```
+
+becomes
+
+```python
+ attr_name = cf_var.attributes.get(CF_ATTR_STD_NAME) or cf_var.attributes.get(
+ CF_ATTR_LONG_NAME
+ )
+```
+
+Line 1149 is the only site with no default. It reads `grid_mapping_name` from
+a variable that was identified as a grid mapping *because* it carries that
+attribute, so it cannot miss; make it an indexing read and let a miss be a
+`KeyError`:
+
+```python
+ # Handle the alternative form noted in CF: rotated mercator.
+ grid_mapping_name = cf_grid_var.attributes[CF_ATTR_GRID_MAPPING_NAME]
+```
+
+`_add_or_capture_load_problem` catches `Exception`, so a `KeyError` here is
+recorded exactly as the `AttributeError` was.
+
+- [ ] **Step 2: Rewrite `helpers.py:567`, the computed-name site**
+
+`attr_key` is a variable, not a literal — `lib/iris/AGENTS.md` bans `getattr`
+string dispatch, and the mapping is the sanctioned form:
+
+```python
+ if attr_key is not None:
+ captured_attr = None
+ with contextlib.suppress(KeyError):
+ captured_attr = cf_var.attributes[attr_key]
+ captured = {attr_key: captured_attr}
+```
+
+The suppressed exception changes with the call. Do not leave
+`suppress(AttributeError)` in place "just in case": it would silently stop
+suppressing, and this block runs while already handling a load failure, so a
+second exception escaping it loses the first.
+
+- [ ] **Step 3: Rewrite `helpers.py:1213`, the deliberately untracked probe**
+
+This one asks whether a flag attribute exists *without* marking it read, so
+that it still reaches the cube. Today it says so by reaching past the
+`CFVariable` to `cf_data`, which Task 11 would break.
+
+```python
+ if any(
+ # Deliberately untracked: asking whether these exist must not count as
+ # using them, or they would stop being copied onto the cube.
+ name in cf_var.attributes.untracked
+ for name in ("flag_values", "flag_masks", "flag_meanings")
+ ):
+ attr_units = cf_units._NO_UNIT_STRING
+```
+
+- [ ] **Step 4: Rewrite the three sites in `actions.py`**
+
+Line 179 and line 626 take the plain rule:
+
+```python
+ grid_mapping_type = cf_var.attributes.get(hh.CF_ATTR_GRID_MAPPING_NAME)
+```
+
+```python
+ # cf_var.standard_name is a formula type (or we should never get here).
+ formula_type = cf_var.attributes.get("standard_name")
+```
+
+Line 560 is a second computed-name site — `match_name` comes from
+`handler.netcdf_names`:
+
+```python
+ for match_name in handler.netcdf_names:
+ match_value = var.attributes.get(match_name)
+```
+
+- [ ] **Step 5: Check no site was missed**
+
+Run:
+```bash
+grep -n "getattr(cf_\|hasattr(cf_\|getattr(var,\|hasattr(var," \
+ lib/iris/fileformats/_nc_load_rules/helpers.py \
+ lib/iris/fileformats/_nc_load_rules/actions.py
+```
+Expected: no output. A match is a site still resolving through
+`CFVariable.__getattr__`, which Task 11 will make warn.
+
+- [ ] **Step 6: Run the loading rules' own tests**
+
+Run: `pytest lib/iris/tests/unit/fileformats/nc_load_rules/`
+Expected: PASS, 0 failed, 0 skipped. These tests were written against the
+`getattr` form and are not modified by this task — which is exactly what makes
+them evidence.
+
+- [ ] **Step 7: Run everything that loads a file**
+
+Run:
+```bash
+pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/ \
+ lib/iris/tests/test_netcdf.py
+```
+Expected: `0 failed`, the §6.2 skip count for this selection, and F14's ten
+`test_coord_systems.py` errors — nothing else.
+
+Watch for cube attributes appearing or disappearing rather than for errors:
+this task moves reads onto the tracking mapping, and a read that stopped
+counting shows up as an extra attribute on a loaded cube, not as a traceback.
+`lib/iris/tests/integration/test_netcdf__loadsaveattrs.py` is the densest
+check of that and must be clean.
+
+- [ ] **Step 8: Commit**
+
+```bash
+pre-commit run --files lib/iris/fileformats/_nc_load_rules/helpers.py \
+ lib/iris/fileformats/_nc_load_rules/actions.py
+git add lib/iris/fileformats/_nc_load_rules/helpers.py \
+ lib/iris/fileformats/_nc_load_rules/actions.py
+git commit -m "$(cat <<'EOF'
+Read CF attributes from cf_var.attributes in the loading rules
+
+66 sites, 63 of them the same rewrite: getattr(cf_var, CF_ATTR_X, None)
+becomes cf_var.attributes.get(CF_ATTR_X). Three are not:
+
+- helpers.py:567 and actions.py:560 look up a name held in a variable.
+ getattr string dispatch is banned by lib/iris/AGENTS.md; a mapping
+ lookup is the sanctioned form, and reads as one.
+- helpers.py:1213 asks whether a flag attribute exists without marking it
+ used, so that it still reaches the cube. It said so by reaching past
+ the CFVariable to the netCDF4 object; it now says so with .untracked.
+
+No behaviour change: __getattr__ already resolved these through
+.attributes. The rules now take that path openly, which is what lets
+the netCDF fallback start warning later in this PR.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 9: `netcdf/loader.py` and `netcdf/ugrid_load.py` read `.attributes`
+
+The same rewrite as Task 8 in the two netCDF loading modules — **and only the
+CF-attribute half of it**. Both modules also reach through `cf_var.cf_data`
+for netCDF storage API: `_data_array`, `datatype`, `chunking()`, the proxy
+class. Those cannot move until `cf_data` is a `CFDatasetVariable`, which is
+Task 11. Mixing the two here would leave the tree red in between.
+
+A `hasattr` in this task matters more than in Task 8. `_add_unused_attributes`
+copies onto the cube every attribute nothing read, so a probe that stops
+counting as a read adds an attribute to loaded cubes.
+`TrackedAttributes.__contains__` records a hit for exactly this reason, which
+makes `X in cf_var.attributes` the faithful replacement for
+`hasattr(cf_var, "X")`.
+
+**Files:**
+- Modify: `lib/iris/fileformats/netcdf/loader.py:214,216,306,350,418,651-653,731`
+- Modify: `lib/iris/fileformats/netcdf/ugrid_load.py:230,322,326,336,338,340,346,354,370,371,383,385,406,456`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py`
+ — its doubles change, its assertions do not (Step 7)
+- Test: the rest of `lib/iris/tests/unit/fileformats/netcdf/loader/` and
+ `lib/iris/tests/unit/fileformats/netcdf/test_ugrid_load.py` — unchanged
+
+**Interfaces:**
+- Consumes: `CFVariable.attributes`, `CFVariable.dimensions` from Task 6.
+- Produces: nothing new.
+
+**Deliberately left for Task 11**, because they are storage, not attributes:
+`loader.py:242` `_data_array`, `:254` `datatype`, `:316` the
+`EncodedVariable` proxy switch, `:321` the proxy construction, `:327`
+`chunking()`.
+
+- [ ] **Step 1: `_get_actual_dtype` — loader.py:214-216**
+
+```python
+def _get_actual_dtype(cf_var):
+ # Figure out what the eventual data type will be after any scale/offset
+ # transforms.
+ dummy_data = np.zeros(1, dtype=cf_var.dtype)
+ if "scale_factor" in cf_var.attributes:
+ dummy_data = cf_var.attributes["scale_factor"] * dummy_data
+ if "add_offset" in cf_var.attributes:
+ dummy_data = cf_var.attributes["add_offset"] + dummy_data
+ return dummy_data.dtype
+```
+
+`scale_factor` and `add_offset` are both in `_CF_ATTRS_IGNORE`, so they are
+seeded as read and this cannot change what reaches the cube either way.
+
+- [ ] **Step 2: The fill value — loader.py:306**
+
+```python
+ fill_dtype = "S1" if cf_var.dtype is str else cf_var.dtype.str[1:]
+ fill_value = cf_var.attributes.get(
+ "_FillValue", _thread_safe_nc.default_fillvals[fill_dtype]
+ )
+```
+
+- [ ] **Step 3: The chunking dimensions — loader.py:350**
+
+```python
+ dims = cf_var.dimensions
+```
+
+`CFVariable.dimensions` is a declared property as of Task 6 and returns the
+same tuple `cf_var.cf_data.dimensions` did.
+
+- [ ] **Step 4: The dataless-cube marker — loader.py:418**
+
+```python
+ if Saver._DATALESS_ATTRNAME in cf_var.attributes:
+```
+
+- [ ] **Step 5: The constraint fast path — loader.py:641-655**
+
+Three things at once here: the computed name, the read, and a comment that
+Task 6 made false.
+
+```python
+ def inner(cf_datavar):
+ match_any_constraint = False
+ for constraint in constraints:
+ match_this_constraint = True
+ for name in constraint._names:
+ expected = getattr(constraint, name)
+ if name != "STASH" and expected != "none":
+ if name == "var_name":
+ # Iris's name for it; not a file attribute at all.
+ actual = cf_datavar.cf_name
+ elif name in cf_datavar.attributes:
+ actual = cf_datavar.attributes[name]
+ else:
+ # Unlike NameConstraint, a variable that does not
+ # carry the attribute is not a mismatch here: the
+ # cube may still acquire the name later in the load.
+ continue
+ if actual != expected:
+ match_this_constraint = False
+ break
+ if match_this_constraint:
+ match_any_constraint = True
+ break
+ return match_any_constraint
+```
+
+Deleted with it: `# Fetch property : N.B. CFVariable caches the property
+values`. Task 6 removed that cache, and `lib/iris/AGENTS.md` requires a
+comment your change invalidates to be fixed or deleted. Also gone is
+`attr_name = "cf_name" if name == "var_name" else name` — the computed name.
+
+`getattr(constraint, name)` stays. `constraint` is an
+`iris._constraints.NameConstraint`, not a `CFVariable`; rewriting it is
+outside this pull request.
+
+- [ ] **Step 6: The mesh reference — loader.py:731**
+
+```python
+ mesh_name = cf_var.attributes.get("mesh")
+```
+
+- [ ] **Step 7: Give the constraint callback's doubles real CF attributes**
+
+Step 5's rewrite is the first consumer change with no fallback behind it, and
+it breaks the only test file that builds `CFDataVariable`s out of bare mocks.
+`lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py`
+does `CFDataVariable("var1", mocker.MagicMock(standard_name="x_wind"))`,
+which puts `standard_name` on the *object*. `MagicMock.ncattrs()` returns a
+`MagicMock`, and `list()` of one is empty, so after Task 6 the attribute
+mapping for every double here is `{}`. Until now that did not show, because
+`__getattr__` fell through to the netCDF4 member and found the mock's
+attribute; Step 5 stops asking `__getattr__` at all.
+
+Give them the interface a netCDF variable actually has. Add above `class
+Test`:
+
+```python
+def _data_variable(mocker, name, **attributes):
+ """Build a CFDataVariable whose CF attributes read as a file's would.
+
+ `MagicMock(standard_name=...)` puts the name on the object, which is no
+ longer where CFVariable looks for it. Drive `ncattrs`/`getncattr`
+ instead.
+
+ """
+ nc_var = mocker.MagicMock()
+ nc_var.ncattrs.return_value = list(attributes)
+ nc_var.getncattr.side_effect = attributes.__getitem__
+ return CFDataVariable(name, nc_var)
+```
+
+and use it in `_setup`:
+
+```python
+ @pytest.fixture(autouse=True)
+ def _setup(self, mocker):
+ self.data_variables = [
+ _data_variable(mocker, "var1", standard_name="x_wind"),
+ _data_variable(mocker, "var2", standard_name="y_wind"),
+ _data_variable(mocker, "var1", long_name="x component of wind"),
+ _data_variable(
+ mocker,
+ "var1",
+ standard_name="x_wind",
+ long_name="x component of wind",
+ ),
+ _data_variable(mocker, "var1"),
+ ]
+```
+
+and in `test_multiple_constraints__multiname`:
+
+```python
+ vars = self.data_variables + [
+ _data_variable(mocker, "var1", standard_name="x_wind"),
+ _data_variable(mocker, "var1", standard_name="air_pressure"),
+ ]
+```
+
+Every assertion in the file is untouched — the expected result lists stay
+exactly as they are. That is the point: the doubles change, the behaviour
+they pin does not. Task 11 Step 10 revisits this helper once `cf_data` stops
+being a netCDF4 object at all.
+
+- [ ] **Step 8: Run the loader tests**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/`
+Expected: PASS, 0 failed, 1 skipped (§6.2).
+
+If `test__translate_constraints_to_var_callback.py` is red here with every
+expected list reading `[False, False, ...]`, Step 7's helper was not applied
+to one of the two construction sites.
+
+- [ ] **Step 9: Commit the loader**
+
+```bash
+pre-commit run --files lib/iris/fileformats/netcdf/loader.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py
+git add lib/iris/fileformats/netcdf/loader.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py
+git commit -m "$(cat <<'EOF'
+Read CF attributes from cf_var.attributes in the netCDF loader
+
+Seven sites. The constraint fast path also loses a computed attribute
+name and a comment about a CFVariable property cache that no longer
+exists.
+
+Its test doubles were MagicMocks carrying standard_name as a Python
+attribute, which is not where CFVariable looks any more. They now drive
+ncattrs/getncattr, like the netCDF variable they stand in for. No
+assertion changed.
+
+The netCDF storage reaches in _get_cf_var_data - _data_array, datatype,
+chunking(), the proxy class - are deliberately untouched: they need
+cf_data to be a CFDatasetVariable, which is a later commit in this PR.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+- [ ] **Step 10: `ugrid_load.py` — the plain reads**
+
+Lines 230, 370, 371, 383, 385 and 456 take Task 8's rule:
+
+```python
+ attr_climatology = coord_var.attributes.get("climatology")
+```
+```python
+ edge_dimension = mesh_var.attributes.get("edge_dimension")
+ face_dimension = mesh_var.attributes.get("face_dimension")
+```
+```python
+ elif coord.var_name in mesh_var.attributes.get("edge_coordinates", "").split():
+```
+```python
+ elif coord.var_name in mesh_var.attributes.get("face_coordinates", "").split():
+```
+```python
+ location = cf_var.attributes.get("location", "<empty>")
+```
+
+- [ ] **Step 11: `ugrid_load.py:322-326` — the `cf_role` probe and read**
+
+```python
+ cf_role_message = None
+ if "cf_role" not in mesh_var.attributes:
+ cf_role_message = f"{mesh_var.cf_name} has no cf_role attribute."
+ cf_role = "mesh_topology"
+ else:
+ cf_role = mesh_var.attributes["cf_role"]
+```
+
+- [ ] **Step 12: `ugrid_load.py:336-354` — the topology probes**
+
+```python
+ if "volume_node_connectivity" in mesh_var.attributes:
+ topology_dimension = 3
+ elif "face_node_connectivity" in mesh_var.attributes:
+ topology_dimension = 2
+ elif "edge_node_connectivity" in mesh_var.attributes:
+ topology_dimension = 1
+ else:
+ # Nodes only. We aren't sure yet whether this is a valid option.
+ topology_dimension = 0
+
+ if "topology_dimension" not in mesh_var.attributes:
+ msg = (
+ f"MeshXY variable {mesh_var.cf_name} has no 'topology_dimension'"
+ f" : *Assuming* topology_dimension={topology_dimension}"
+ ", consistent with the attached connectivities."
+ )
+ warnings.warn(msg, category=_WarnComboCfDefaulting)
+ else:
+ quoted_topology_dimension = mesh_var.attributes["topology_dimension"]
+```
+
+These six probes are the reason `__contains__` records a read. Each names an
+attribute Iris then acts on, and an unrecorded probe would put
+`face_node_connectivity` onto every mesh cube as a stray attribute.
+
+- [ ] **Step 13: `ugrid_load.py:406` — the third computed-name site**
+
+`connectivity.cf_role` is a value, not a literal:
+
+```python
+ assert connectivity.var_name == mesh_var.attributes[connectivity.cf_role]
+```
+
+- [ ] **Step 14: Check no site was missed**
+
+Run:
+```bash
+grep -n "getattr(cf_var\|hasattr(cf_var\|getattr(mesh_var\|hasattr(mesh_var\|getattr(coord_var\|hasattr(coord_var" \
+ lib/iris/fileformats/netcdf/ugrid_load.py
+```
+Expected: no output.
+
+- [ ] **Step 15: Run the mesh tests**
+
+Run:
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/test_ugrid_load.py \
+ lib/iris/tests/unit/mesh/ lib/iris/tests/integration/mesh/
+```
+Expected: PASS, 0 failed, 0 skipped.
+
+Then the wider net, because this task's risk is stray cube attributes rather
+than errors:
+
+Run: `pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/`
+Expected: `0 failed, 26 skipped`, plus F14's ten `test_coord_systems.py`
+errors and nothing else (§6.2).
+
+- [ ] **Step 16: Commit the mesh loader**
+
+```bash
+pre-commit run --files lib/iris/fileformats/netcdf/ugrid_load.py
+git add lib/iris/fileformats/netcdf/ugrid_load.py
+git commit -m "$(cat <<'EOF'
+Read CF attributes from cf_var.attributes in the UGRID loader
+
+Fourteen sites, including the third and last computed attribute name in
+the loading path: getattr(mesh_var, connectivity.cf_role).
+
+The six hasattr probes around topology_dimension become "in
+mesh_var.attributes", which still records the attribute as read -
+otherwise face_node_connectivity and its neighbours would start
+appearing as stray attributes on every mesh cube.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 10: `cf/_reader.py` reads — and writes — `.attributes`
+
+Eight sites, and the only module in the programme that *writes* CF attributes
+during a load. The formula-terms code synthesises a `bounds` link onto a
+variable that did not carry one:
+
+```python
+cf_var.bounds = term_bounds_var.cf_name # _reader.py:324
+new_var.bounds = cf_var.bounds # _reader.py:341
+root_var.bounds = None # _reader.py:358
+```
+
+Today those land in the instance `__dict__`, where they shadow anything
+`__getattr__` would find — which is exactly how the matching
+`hasattr(cf_variable, "bounds")` reads see them. Move the *reads* to
+`.attributes` and leave the *writes* on the instance and the link becomes
+invisible. So both move, together, in one commit.
+
+**Two `None` traps.** `cf_root_coord` at line 304 and `root_bounds_var` at
+line 354 can each be `None`, and `getattr(None, "bounds", None)` answers
+`None` quite happily where `None.attributes` raises. The comment at line
+305 already warns about the first. A mechanical rewrite turns both into
+`AttributeError` during load.
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/_reader.py:304,309,324,339,341,350-358,439,460,497,499`
+- Test: `lib/iris/tests/unit/fileformats/cf/test_CFReader.py` — unchanged
+- Test: `lib/iris/tests/integration/test_cf.py` — unchanged
+
+**Interfaces:**
+- Consumes: `CFVariable.attributes` from Task 6.
+- Produces: nothing new. Leaves `_reader.py` free of `getattr`/`hasattr` over
+ `CFVariable`s, which Task 11 needs before the fallback can warn.
+
+**Deliberately left for Task 11**, because they read raw netCDF variables
+rather than `CFVariable`s: `:204` `hasattr(variable, "mesh")`, `:235`
+`getattr(nc_var, "grid_mapping", None)`, `:274` the global-attribute
+comprehension and `_getncattr` itself. `:373`
+`getattr(self.cf_group, "ugrid_coords", None)` stays as it is — `cf_group` is
+a `CFGroup`, not a variable.
+
+- [ ] **Step 1: Write the failing test for the synthesised bounds link**
+
+This is the behaviour the task could silently lose, and no existing test
+names it directly. In `lib/iris/tests/unit/fileformats/cf/test_CFReader.py`,
+add:
+
+```python
+class TestSynthesisedBoundsLink:
+ """The reader writes a bounds link that the file did not carry.
+
+ _reader.py:324 attaches a formula term's bounds variable to the term.
+ Whatever stores that link has to be the same place the reads look, or
+ the link is written and never seen.
+ """
+
+ def test_a_written_bounds_link_is_visible_to_a_reader(self, mocker):
+ nc_var = mocker.MagicMock()
+ nc_var.ncattrs.return_value = ["units"]
+ nc_var.getncattr.side_effect = {"units": "m"}.__getitem__
+ nc_var.dimensions = ("model_level_number",)
+ cf_var = CFAuxiliaryCoordinateVariable("a", nc_var)
+
+ assert "bounds" not in cf_var.attributes
+ cf_var.attributes["bounds"] = "a_bnds"
+
+ assert cf_var.attributes["bounds"] == "a_bnds"
+ assert "bounds" in cf_var.attributes
+ assert cf_var.bounds == "a_bnds"
+
+ def test_a_written_bounds_link_overrides_the_file(self, mocker):
+ nc_var = mocker.MagicMock()
+ nc_var.ncattrs.return_value = ["bounds"]
+ nc_var.getncattr.side_effect = {"bounds": "from_file"}.__getitem__
+ nc_var.dimensions = ("model_level_number",)
+ cf_var = CFAuxiliaryCoordinateVariable("a", nc_var)
+
+ cf_var.attributes["bounds"] = "synthesised"
+ assert cf_var.bounds == "synthesised"
+
+ def test_a_bounds_link_set_to_none_still_reads_as_present(self, mocker):
+ # _reader.py:358 invalidates a broken link by setting it to None
+ # rather than deleting it, and the reads downstream check presence
+ # before value.
+ nc_var = mocker.MagicMock()
+ nc_var.ncattrs.return_value = ["bounds"]
+ nc_var.getncattr.side_effect = {"bounds": "broken"}.__getitem__
+ nc_var.dimensions = ("model_level_number",)
+ cf_var = CFAuxiliaryCoordinateVariable("a", nc_var)
+
+ cf_var.attributes["bounds"] = None
+ assert "bounds" in cf_var.attributes
+ assert cf_var.attributes.get("bounds") is None
+```
+
+Add `CFAuxiliaryCoordinateVariable` to the module's imports from
+`iris.fileformats.cf` if it is not already there.
+
+- [ ] **Step 2: Run it**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/test_CFReader.py -k Synthesised -v`
+Expected: PASS on Task 6's implementation — `TrackedAttributes` is a
+`MutableMapping` and `__getattr__` reads it. This is a characterisation test,
+written now because the *next* step is what could break it.
+
+- [ ] **Step 3: Move the three writes onto the mapping**
+
+```python
+ if term_bounds_var != cf_var:
+ cf_var.attributes["bounds"] = term_bounds_var.cf_name
+```
+
+```python
+ if iris.FUTURE.derived_bounds and "bounds" in cf_var.attributes:
+ # Copy "old-style" derived bounds link
+ new_var.attributes["bounds"] = cf_var.attributes["bounds"]
+```
+
+and, in the `derived_bounds` invalidation block:
+
+```python
+ root_var.attributes["bounds"] = None
+```
+
+- [ ] **Step 4: Rewrite line 304, guarding the `None`**
+
+```python
+ # N.B. cf_root_coord may here be None, if the root var was not a
+ # coord - that is ok, it will not have a 'bounds', we will skip it.
+ root_bounds_name = None
+ if cf_root_coord is not None:
+ root_bounds_name = cf_root_coord.attributes.get("bounds")
+ if root_bounds_name in self.cf_group:
+ root_bounds_var = self.cf_group.get(root_bounds_name)
+ if "formula_terms" not in root_bounds_var.attributes:
+```
+
+The comment moves above the guard, where it now explains code rather than
+trailing it.
+
+- [ ] **Step 5: Rewrite the `derived_bounds` invalidation block, guarding the second `None`**
+
+`self.cf_group.get(...)` returns `None` for a name the group does not hold,
+and `getattr(None, "formula_terms", None)` answered `None` — which made the
+`not` true and invalidated the link. Spell that out:
+
+```python
+ if iris.FUTURE.derived_bounds:
+ for cf_root in all_roots:
+ # Invalidate "broken" bounds connections
+ root_var = self.cf_group[cf_root]
+ if root_var.attributes.get("formula_terms") and root_var.attributes.get(
+ "bounds"
+ ):
+ root_bounds_var = self.cf_group.get(root_var.attributes["bounds"])
+ if root_bounds_var is None or not root_bounds_var.attributes.get(
+ "formula_terms"
+ ):
+ # This means it is *not* a valid bounds var, according to CF, and so therefore we are
+ # invalidating the bounds.
+ root_var.attributes["bounds"] = None
+```
+
+- [ ] **Step 6: Rewrite the three remaining reads**
+
+Line 439, in `_build`:
+
+```python
+ if "bounds" in cf_variable.attributes:
+ bounds_name = cf_variable.attributes["bounds"]
+ if bounds_name not in cf_group:
+ bounds_var = self.cf_group.get(bounds_name)
+ if bounds_var:
+ # TODO: warning if span fails
+ if bounds_var.spans(cf_variable):
+ cf_group[bounds_name] = bounds_var
+```
+
+Line 460:
+
+```python
+ coordinates_attr = cf_variable.attributes.get("coordinates", "")
+```
+
+Lines 497-499:
+
+```python
+ if iris.FUTURE.derived_bounds:
+ if "standard_name" not in cf_root_var.attributes:
+ continue
+ name = cf_root_var.attributes["standard_name"] or cf_root_var.attributes.get(
+ "long_name"
+ )
+```
+
+Note the asymmetry, which is deliberate and matches today: `standard_name` is
+indexed, so a formula root without one raises — as it does today, with
+`AttributeError` rather than `KeyError` — while `long_name` is fetched with
+`get`, because it is only consulted when `standard_name` is falsy and a
+variable may legitimately lack it.
+
+- [ ] **Step 7: Check no site was missed**
+
+Run: `grep -n "getattr(cf_\|hasattr(cf_\|\.bounds = " lib/iris/fileformats/cf/_reader.py`
+Expected: only `getattr(self.cf_group, "ugrid_coords", None)` on line 373.
+
+- [ ] **Step 8: Run the reader tests**
+
+Run:
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/ lib/iris/tests/integration/test_cf.py
+```
+Expected: PASS, 0 failed, 1 skipped (§6.2).
+
+- [ ] **Step 9: Run the formula-terms and derived-bounds coverage**
+
+`iris.FUTURE.derived_bounds` gates three of the rewritten sites, so the
+default-off suite does not reach them.
+
+Run:
+```bash
+pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/ \
+ -k "derived or formula or hybrid or aux_factory"
+```
+Expected: 0 failed. Then the full area:
+
+Run: `pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/`
+Expected: `0 failed, 26 skipped`, plus F14's ten `test_coord_systems.py`
+errors and nothing else (§6.2).
+
+- [ ] **Step 10: Commit**
+
+```bash
+pre-commit run --files lib/iris/fileformats/cf/_reader.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFReader.py
+git add lib/iris/fileformats/cf/_reader.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFReader.py
+git commit -m "$(cat <<'EOF'
+Read and write CF attributes through cf_var.attributes in CFReader
+
+CFReader is the one module that writes CF attributes during a load: the
+formula-terms code synthesises a bounds link onto a variable that did not
+carry one. Those writes land in the mapping now, alongside the reads
+that look for them - moving one without the other would write a link
+nothing could see.
+
+Two sites needed a None guard rather than a rewrite. cf_root_coord and
+root_bounds_var can each be None, and getattr(None, "bounds", None)
+answers None where None.attributes raises. The first already had a
+comment saying so.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 11: The swap — `CFReader` builds a `NetCDFDataset`
+
+This is the task the previous five were clearing the way for. Every consumer
+now reads CF attributes through `cf_var.attributes`; nothing reads them off a
+netCDF4 object any more except `CFVariable.__init__` itself, the twelve
+`identify()` calls, and `CFReader`'s own bookkeeping. Those move here, in one
+step, and `cf_var.cf_data` stops being a `netCDF4.Variable` and becomes a
+`CFDatasetVariable`.
+
+**One long red stretch.** A type swap cannot be done a consumer at a time:
+the moment `CFReader` hands out `NetCDFDatasetVariable`s, everything that
+still expects a netCDF4 object is broken, and the fix for each is in a
+different file. Steps 3 to 10 therefore run with no test run between them,
+and the green check is Step 11. This is deliberate, and it is why the five
+preceding tasks exist — they shrink this stretch from "every consumer in the
+package" to "the eight sites below".
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/_variables.py:97-135` (`__init__`),
+ `:3319-3329` region (delete `_attributes_source`), `_deprecated_netcdf_member`,
+ and the twelve `identify()` attribute reads at
+ `:304, :353, :402, :478, :614, :687, :771, :874, :938, :1013, :1087, :1091`
+- Modify: `lib/iris/fileformats/cf/_reader.py:132-186` (`__init__`),
+ `:200-206` (`_has_meshes`), `:235`, `:272-276`, `:543-549` (delete `_getncattr`)
+- Modify: `lib/iris/fileformats/netcdf/_dataset.py` — `from_existing` gains
+ `warn_legacy_format`; `NetCDFDatasetVariable.deprecated_netcdf_member` starts
+ warning
+- Modify: `lib/iris/fileformats/netcdf/loader.py:242, :254, :316, :321, :327-330`
+- Test: `lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py` (create)
+- Test: `lib/iris/tests/unit/fileformats/cf/identify_mixins.py`,
+ `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py`,
+ `lib/iris/tests/unit/fileformats/cf/test_CFReader.py`,
+ `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`,
+ `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py`
+- Test: the doubles that stand in for a netCDF4 variable, all in Step 10 —
+ `lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py`,
+ `lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py`,
+ `lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_dimension_coordinate.py`
+
+**Interfaces:**
+- Consumes: `NetCDFDataset(location, mode="r", *, netcdf_format=None,
+ warn_legacy_format=False)`, `NetCDFDataset.from_existing(dataset)`,
+ `.location`, `.variables`, `.attributes`, `.close()` (Task 3);
+ `NetCDFDatasetVariable.name`, `.location`, `.dimensions`, `.shape`,
+ `.dtype`, `.size`, `.attributes`, `.chunking`, `.variable`,
+ `.is_variable_length`, `.is_emulated`, `.emulated_data_array`,
+ `.deprecated_netcdf_member(name)` (Task 2); `CFVariable.attributes`
+ (Task 6); every consumer rewritten in Tasks 8, 9 and 10.
+- Produces: `CFVariable.cf_data` is a `CFDatasetVariable`, for every
+ `CFVariable` any part of Iris builds. `CFReader._dataset` is a
+ `NetCDFDataset`. `CFVariable.filename` comes from
+ `cf_data.location`. Task 12 relies on all of it, and on
+ `NetCDFDatasetVariable.deprecated_netcdf_member` warning.
+
+- [ ] **Step 1: Write the failing tests for the swap**
+
+Create `lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Tests for :class:`iris.fileformats.cf.CFReader` reading a real file.
+
+The rest of the CFReader tests drive it with mocks, which is the right shape
+for its classification logic and the wrong shape for the question here: what
+type the reader actually hands out, and what a load does with it. These use a
+small file on disk instead.
+"""
+
+import numpy as np
+import pytest
+
+import iris
+from iris.fileformats.cf import CFReader
+from iris.fileformats.cf.dataset import CFDatasetVariable
+from iris.fileformats.netcdf import _dataset, _thread_safe_nc
+
+UNREAD_COMMENT = "an attribute nothing in Iris reads"
+
+
+@pytest.fixture
+def sample_path(tmp_path):
+ """Write a small CF file: a data variable, a coordinate and its bounds."""
+ path = tmp_path / "sample.nc"
+ dataset = _thread_safe_nc.DatasetWrapper(path, mode="w")
+ dataset.title = "a sample file"
+ dataset.createDimension("time", 3)
+ dataset.createDimension("bnds", 2)
+
+ time = dataset.createVariable("time", "f8", ("time",))
+ time.standard_name = "time"
+ time.units = "days since 1970-01-01"
+ time.bounds = "time_bnds"
+ time.comment = UNREAD_COMMENT
+ time[:] = np.arange(3, dtype="f8")
+
+ bounds = dataset.createVariable("time_bnds", "f8", ("time", "bnds"))
+ bounds[:] = np.zeros((3, 2))
+
+ air = dataset.createVariable("air", "f4", ("time",))
+ air.standard_name = "air_temperature"
+ air.units = "K"
+ air.coordinates = "time"
+ air.comment = UNREAD_COMMENT
+ air[:] = np.arange(3, dtype="f4")
+
+ dataset.close()
+ return path
+
+
+class TestTheSwap:
+ def test_the_reader_owns_a_netcdf_dataset(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ assert isinstance(reader._dataset, _dataset.NetCDFDataset)
+
+ def test_cf_data_is_a_cf_dataset_variable(self, sample_path):
+ # The whole point of the PR: nothing downstream of here needs to know
+ # the file is netCDF.
+ with CFReader(str(sample_path)) as reader:
+ assert isinstance(reader.cf_group["air"].cf_data, CFDatasetVariable)
+
+ def test_attributes_come_from_the_file(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ air = reader.cf_group["air"]
+ assert air.attributes["units"] == "K"
+ assert air.units == "K"
+ assert air.dimensions == ("time",)
+
+ def test_global_attributes_come_from_the_dataset(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ assert reader.cf_group.global_attributes["title"] == "a sample file"
+
+ def test_filename_is_the_variables_location(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ assert reader.cf_group["air"].filename == str(sample_path)
+
+ def test_the_bounds_variable_was_classified(self, sample_path):
+ # identify() now reads through .attributes, so a miss here means the
+ # classification pass lost sight of the file's attributes entirely.
+ with CFReader(str(sample_path)) as reader:
+ assert "time_bnds" in reader.cf_group.bounds
+
+ def test_a_borrowed_dataset_is_used_and_not_closed(self, sample_path):
+ raw = _thread_safe_nc.DatasetWrapper(sample_path, mode="r")
+ try:
+ with CFReader(raw) as reader:
+ assert reader.cf_group["air"].units == "K"
+ assert raw.isopen()
+ finally:
+ raw.close()
+
+
+class TestAttributesAreASnapshot:
+ """Finding F11.
+
+ ``NetCDFDatasetVariable.attributes`` writes through to the file, which is
+ right for the saver and wrong for the reader: CFReader synthesises a
+ "bounds" link during load, on a file it opened read-only. CFVariable
+ therefore takes a copy.
+ """
+
+ def test_a_write_does_not_reach_the_file(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ air = reader.cf_group["air"]
+ air.attributes["bounds"] = "invented"
+
+ assert "bounds" not in air.cf_data.attributes
+
+
+class TestAttributeTrackingAcrossReset:
+ """Review Focus 3.
+
+ CFReader reads attributes while classifying variables, then calls
+ ``cf_attrs_reset()`` so the rules start from a clean record. Until this
+ PR, ``__getattr__`` cached the value on the instance, so a read after the
+ reset found the cache and was never recorded - and an attribute Iris had
+ in fact consumed was still reported unused, and so was copied onto the
+ cube as if the file had volunteered it.
+ """
+
+ def test_the_reader_resets_what_it_read_while_classifying(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ time = reader.cf_group["time"]
+ # CFBoundaryVariable.identify() read "bounds" during __init__.
+ # Ask through .untracked, so that asking does not itself record.
+ assert "bounds" in time.attributes.untracked
+ assert dict(time.cf_attrs_used()) == dict(time.cf_attrs_ignored())
+
+ def test_a_read_after_the_reset_is_recorded(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ time = reader.cf_group["time"]
+
+ assert time.units == "days since 1970-01-01"
+ assert "units" in dict(time.cf_attrs_used())
+ assert "units" not in dict(time.cf_attrs_unused())
+
+ def test_only_unread_attributes_reach_the_cube(self, sample_path):
+ cube = iris.load_cube(str(sample_path))
+
+ assert cube.attributes == {"comment": UNREAD_COMMENT}
+ assert cube.coord("time").attributes == {"comment": UNREAD_COMMENT}
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py`
+Expected: `TestTheSwap::test_the_reader_owns_a_netcdf_dataset` and
+`test_cf_data_is_a_cf_dataset_variable` FAIL on `assert isinstance(...)` —
+the reader still opens an `EncodedDataset` directly. `TestAttributesAreASnapshot`
+fails with `AttributeError: 'Variable' object has no attribute 'attributes'`.
+The rest pass already: they describe behaviour this task must preserve.
+
+- [ ] **Step 3: Take `CFVariable.__init__` off the netCDF4 API**
+
+In `lib/iris/fileformats/cf/_variables.py`, delete `_attributes_source`
+entirely — the function, its docstring and its comment. Then in
+`CFVariable.__init__`, replace the two lines it fed:
+
+```python
+ self.attributes = TrackedAttributes(
+ dict(data.attributes), ignored=_CF_ATTRS_IGNORE
+ )
+```
+
+`dict(...)`, not the mapping itself. `NetCDFDatasetVariable.attributes` is a
+live view: `__setitem__` calls `setncattr` on the file. CFReader writes a
+synthesised `bounds` link through `cf_var.attributes` (Task 10, Step 3) on a
+file opened `mode="r"`, and a live view would turn that into
+`RuntimeError: NetCDF: Write to read only`. Copying also makes the mapping
+cheap to re-read, which is what `cf_attrs_unused()` asks of it once per
+variable per load. See finding F11.
+
+Replace the `filename` block immediately below it:
+
+```python
+ """File source of the NetCDF content."""
+ try:
+ self.filename = data.location
+ except AttributeError:
+ self.filename = "<unknown_filename>"
+```
+
+`CFDatasetVariable.location` is the path or URL of the dataset the variable
+belongs to — the same string `data.group().filepath()` used to return, and
+now supplied by the object itself rather than reached around it. The
+`try/except` stays because the identify() test stubs are not
+`CFDatasetVariable`s and have no `location`; they keep the same
+`"<unknown_filename>"` they get today.
+
+Finally, point the compatibility route at the storage object. Replace the
+body of `_deprecated_netcdf_member`, and its transitional comment:
+
+```python
+ def _deprecated_netcdf_member(self, name):
+ """Return a netCDF4 member of the backing variable, as this class used to.
+
+ The one-cycle compatibility route for code that reached netCDF4 API
+ through a CFVariable. Records nothing: this is not a CF attribute.
+ The storage object decides what this means, and warns - see
+ :meth:`iris.fileformats.netcdf._dataset.NetCDFDatasetVariable.deprecated_netcdf_member`.
+
+ """
+ return self.cf_data.deprecated_netcdf_member(name)
+```
+
+- [ ] **Step 4: Move the twelve `identify()` reads onto `.attributes`**
+
+Still in `_variables.py`. `identify()` is handed the dataset's variables,
+which are now `CFDatasetVariable`s, so `getattr(nc_var, ...)` no longer
+reaches a CF attribute at all — it reaches the storage class's own members,
+or nothing.
+
+At `:304, :353, :402, :478, :614, :687, :771` and `:874`, each reading:
+
+```python
+ nc_var_att = getattr(nc_var, cls.cf_identity, None)
+```
+
+becomes:
+
+```python
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
+```
+
+At `:938` and `:1013`, inside the loop over `cls.cf_identities`:
+
+```python
+ nc_var_att = getattr(nc_var, identity, None)
+```
+
+becomes:
+
+```python
+ nc_var_att = nc_var.attributes.get(identity)
+```
+
+At `:1087`:
+
+```python
+ is_mesh_var = nc_var.attributes.get("cf_role", "") == "mesh_topology"
+```
+
+At `:1091`:
+
+```python
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
+```
+
+These reads are deliberately **not** tracked. `CFDatasetVariable.attributes`
+is a plain mapping; tracking lives in `CFVariable.attributes`, and these run
+before any `CFVariable` exists. That matches what they do today — `getattr`
+on a raw netCDF variable recorded nothing — and it is why `CFReader._reset`
+still has work to do afterwards.
+
+- [ ] **Step 5: Make `CFReader` build a dataset**
+
+In `lib/iris/fileformats/cf/_reader.py`, replace `__init__` down to the line
+that sets `self._check_monotonic`:
+
+```python
+ def __init__(self, file_source, warn=False, monotonic=False):
+ # Ensure safe operation for destructor, should init fail.
+ self._own_file = False
+ if isinstance(file_source, str):
+ # Create from filepath : open it + own it (=close when we die).
+ if not urlparse(file_source).scheme:
+ self._filename = Path(file_source).expanduser()
+ else:
+ self._filename = file_source
+
+ self._dataset = NetCDFDataset(
+ self._filename, mode="r", warn_legacy_format=warn
+ )
+ self._own_file = True
+ else:
+ # We have been passed an open dataset.
+ # We use it but don't own it (don't close it).
+ self._dataset = NetCDFDataset.from_existing(
+ file_source, warn_legacy_format=warn
+ )
+ self._filename = self._dataset.location
+
+ #: Collection of CF-netCDF variables associated with this netCDF file
+ self.cf_group = self.CFGroup()
+
+ # Result of parsing "grid_mapping" attribute; mapping of coordinate_system => coordinates
+ self._coord_system_mappings = {}
+
+ self._check_monotonic = monotonic
+```
+
+Four things go, and all four went somewhere: the choice between
+`EncodedDataset` and `DatasetWrapper` on `DECODE_TO_STRINGS_ON_READ`, the
+`set_auto_chartostring(False)` call, the netCDF3 warning, and
+`self._dataset.filepath()`. `NetCDFDataset` does all of them, which is what
+§3's "single home for netCDF API" line means in practice.
+
+Leave the block that follows untouched apart from its two dead lines:
+
+```python
+ self._with_ugrid = True
+ if not self._has_meshes():
+ self._trim_ugrid_variable_types()
+ self._with_ugrid = False
+
+ # Read the variables in the dataset only once to reduce runtime.
+ # NetCDFDataset.variables caches, so this is the same dict - and the
+ # same CFDatasetVariable objects - that _has_meshes just walked.
+ variables = self._dataset.variables
+ self._translate(variables)
+ self._build_cf_groups(variables)
+ self._reset(variables)
+```
+
+`from_existing` needs the new keyword. In
+`lib/iris/fileformats/netcdf/_dataset.py`, give both constructors the same
+warning by factoring it out. Replace the warning block in `__init__` with a
+call:
+
+```python
+ if warn_legacy_format:
+ self._warn_if_legacy_format()
+```
+
+and add the method, beside `close`:
+
+```python
+ def _warn_if_legacy_format(self) -> None:
+ """Warn that this file would load faster in netCDF4 format, if it would."""
+ if self._dataset.file_format in _LEGACY_FORMATS:
+ warnings.warn(
+ "Optimise CF-netCDF loading by converting data from NetCDF3 "
+ 'to NetCDF4 file format using the "nccopy" command.',
+ category=iris.warnings.IrisLoadWarning,
+ )
+```
+
+Then extend `from_existing`:
+
+```python
+ @classmethod
+ def from_existing(cls, dataset, *, warn_legacy_format: bool = False):
+```
+
+and end its body, before the `return`:
+
+```python
+ if warn_legacy_format:
+ instance._warn_if_legacy_format()
+```
+
+CFReader has always offered this warning for a borrowed dataset as well as
+for one it opened, and the constraint on this PR is no behaviour change.
+Add the matching test to `TestLegacyFormatWarning` in
+`lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`:
+
+```python
+ def test_warns_for_a_borrowed_dataset_when_asked(self, tmp_path):
+ legacy = _write_netcdf3(tmp_path)
+ raw = _thread_safe_nc.DatasetWrapper(legacy, mode="r")
+ try:
+ with pytest.warns(IrisLoadWarning, match="nccopy"):
+ _dataset.NetCDFDataset.from_existing(raw, warn_legacy_format=True)
+ finally:
+ raw.close()
+```
+
+- [ ] **Step 6: Finish `_reader.py` — meshes, grid mapping, globals, `_getncattr`**
+
+`_has_meshes` walks raw variables today. Replace it:
+
+```python
+ def _has_meshes(self):
+ result = False
+ for variable in self._dataset.variables.values():
+ attributes = variable.attributes
+ if "mesh" in attributes or "node_coordinates" in attributes:
+ result = True
+ break
+ return result
+```
+
+At `:235`, in `_translate`:
+
+```python
+ if grid_mapping_attr := nc_var.attributes.get("grid_mapping"):
+```
+
+At `:272-276`, the global attributes. `_getncattr` existed to give
+`getncattr` a `getattr`-shaped default, and `_NetCDFAttributes` already
+absorbs the malformed-file case it was tolerating (Task 2):
+
+```python
+ # Identify global netCDF attributes.
+ self.cf_group.global_attributes.update(dict(self._dataset.attributes))
+```
+
+Delete `_getncattr` at the foot of the module, and its docstring. The
+`test___init__.py` parametrisation that names `_getncattr` stays as it is:
+it asserts the name is *not* on the package, which a deleted name satisfies,
+and it goes on guarding against the name coming back.
+
+Finally the imports. `_reader.py` no longer needs `_bytecoding_datasets` or
+`_thread_safe_nc`; it needs the dataset instead:
+
+```python
+from iris.fileformats.netcdf._dataset import NetCDFDataset
+```
+
+Check what is left: `warnings` is still used by the `grid_mapping` parse
+error, `iris.warnings` with it. Run
+`pre-commit run ruff --files lib/iris/fileformats/cf/_reader.py` at the end
+of the task and let F401 name anything that has gone cold.
+
+- [ ] **Step 7: Move the loader's five storage reads onto the interface**
+
+These are the sites Task 9 deliberately left alone: they ask about *storage*,
+not about CF attributes, and until now the only way to ask was to reach for
+netCDF4 API. In `lib/iris/fileformats/netcdf/loader.py`:
+
+`:242`, the Xarray bridge (#4994) —
+
+```python
+ if cf_var.cf_data.is_emulated:
+ # The variable is not an actual netCDF4 file variable, but an emulating
+ # object with an attached data array (either numpy or dask), which can be
+ # returned immediately as-is. This is used as a hook to translate data to/from
+ # netcdf data container objects in other packages, such as xarray.
+ # See https://github.com/SciTools/iris/issues/4994 "Xarray bridge".
+ result = cf_var.cf_data.emulated_data_array
+```
+
+`:254`, the VLEN check —
+
+```python
+ if cf_var.cf_data.is_variable_length:
+```
+
+`_thread_safe_nc.VLType` leaves the loader with it; the whole `isinstance`
+disappears.
+
+`:316` and `:321`, the data proxy. `cf_var.cf_data` is no longer the netCDF
+variable the proxy needs, so both ask the interface for it:
+
+```python
+ if isinstance(cf_var.cf_data.variable, _bytecoding_datasets.EncodedVariable):
+ proxy_class = _bytecoding_datasets.EncodedNetCDFDataProxy
+ else:
+ proxy_class = _thread_safe_nc.NetCDFDataProxy
+
+ proxy = proxy_class(
+ cf_var.cf_data.variable, dtype, cf_var.filename, fill_value
+ )
+```
+
+`.variable` is the one deliberate escape hatch in the interface, and the
+proxy is what it exists for: it is pickled and shipped to Dask workers, and
+it has to carry a netCDF handle to reopen. Task 12 uses it for the same
+reason. A Zarr backend will supply its own proxy class here in a later PR.
+
+`:327-330`, the chunking. `NetCDFDatasetVariable.chunking` is a property, and
+it has already turned netCDF's two ways of saying "unchunked" — `None` for a
+non-version-4 file, the string `"contiguous"` for a version-4 one — into one.
+That collapses the two-step dance the loader used to do:
+
+```python
+ chunks = cf_var.cf_data.chunking
+ if chunks is None:
+ # Unchunked : either a non-version-4 file, or a contiguous
+ # version-4 variable. Neither offers a chunking to adopt.
+ if (
+ CHUNK_CONTROL.mode is ChunkControl.Modes.FROM_FILE
+ and isinstance(cf_var, iris.fileformats.cf.CFDataVariable)
+ ):
+ raise KeyError(
+ f"{cf_var.cf_name} does not contain pre-existing chunk specifications."
+ f" Instead, you might wish to use CHUNK_CONTROL.set(), or just use default"
+ f" behaviour outside of a context manager. "
+ )
+ # Equivalent to chunks=None, but value required by chunking control
+ chunks = list(cf_var.shape)
+ else:
+ # The chunk-control block below assigns into this.
+ chunks = list(chunks)
+```
+
+Delete the two lines that set and then test `chunks = "contiguous"`; the
+`if chunks == "contiguous":` line and its comment go with them. `list(chunks)`
+matters: the property returns a tuple, and
+`chunks[i_dim] = dim_chunksize` a few lines further down needs a list.
+
+- [ ] **Step 8: Teach the identify() stubs the new shape**
+
+The stubs in the CF unit tests stand in for a variable that `identify()`
+reads and `CFVariable.__init__` copies from. Both now go through
+`attributes`.
+
+In `lib/iris/tests/unit/fileformats/cf/identify_mixins.py`, delete the
+`getncattr` method Task 6 added to `_NetCDFVar`, and add in its place:
+
+```python
+ @property
+ def attributes(self):
+ """The CF attributes, as a CFDatasetVariable presents them."""
+ return {name: getattr(self, name) for name in self.ncattrs()}
+```
+
+`ncattrs()` stays: it is what decides which of the instance's members count
+as file attributes, and the property is built from it. Neither `attributes`
+nor `ncattrs` appears in `self.__dict__`, so neither can list itself.
+
+In `lib/iris/tests/unit/fileformats/cf/test_CFCoordinateVariable.py`, add the
+same to `_CoordVariableStub`, whose `ncattrs()` returns `[]`:
+
+```python
+ @property
+ def attributes(self):
+ return {}
+```
+
+That stub had needed nothing until now, because `getattr` on it answered
+whatever the test had set. `dict(data.attributes)` will not invent a mapping.
+
+- [ ] **Step 9: Rewrite the `test_CFVariable.py` doubles**
+
+The module's `MagicMock` variables have to present the same surface. Replace
+`make_nc_var` and the fixture below it:
+
+```python
+def make_nc_var(mocker):
+ nc_var = mocker.MagicMock()
+ nc_var.attributes = {
+ "coordinates": "x y",
+ "standard_name": "air_temperature",
+ "_FillValue": -999,
+ }
+ nc_var.dimensions = ("time", "lat")
+ nc_var.location = "/tmp/file.nc"
+ nc_var.__len__.return_value = 4
+ nc_var.__getitem__.return_value = "payload"
+
+ return nc_var
+
+
+@pytest.fixture
+def nc_var(mocker):
+ return make_nc_var(mocker)
+
+
+@pytest.fixture
+def nc_var_without_location(nc_var):
+ del nc_var.location
+
+ return nc_var
+```
+
+`TestInit` follows the rename:
+
+```python
+class TestInit:
+ def test_records_filename_from_the_variables_location(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.filename == "/tmp/file.nc"
+ assert cf_var.cf_name == "foo"
+ assert cf_var.cf_data is nc_var
+ assert cf_var.cf_group is None
+ assert cf_var.cf_terms_by_root == {}
+ assert cf_var._to_be_promoted is False
+
+ def test_falls_back_to_unknown_filename_without_a_location(
+ self, nc_var_without_location
+ ):
+ cf_var = CFVariableSub("foo", nc_var_without_location)
+
+ assert cf_var.filename == "<unknown_filename>"
+```
+
+Then five tests that named `ncattrs`/`getncattr`. In `TestAttributeAccess`,
+replace `test_attributes_are_read_from_the_file_once` — there is no call to
+count any more, and the property it was really asserting is better said
+directly:
+
+```python
+ def test_attributes_are_a_snapshot_taken_at_construction(self, nc_var):
+ # Finding F11. Copied, not wrapped: the storage object's mapping
+ # writes through to the file, and CFReader synthesises a "bounds"
+ # link on a file it opened read-only.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ nc_var.attributes["units"] = "K"
+ assert "units" not in cf_var.attributes.untracked
+
+ cf_var.attributes["bounds"] = "foo_bnds"
+ assert "bounds" not in nc_var.attributes
+```
+
+Replace `test_getattr_of_a_non_attribute_reaches_the_variable`:
+
+```python
+ def test_getattr_of_a_non_attribute_reaches_the_variable(self, nc_var):
+ # The one-cycle compatibility route, now delegated to the storage
+ # object - which is also what warns. See Step 15.
+ nc_var.deprecated_netcdf_member.return_value = 42
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.not_an_ncattr == 42
+ nc_var.deprecated_netcdf_member.assert_called_once_with("not_an_ncattr")
+ assert "not_an_ncattr" not in dict(cf_var.cf_attrs())
+ assert "not_an_ncattr" not in dict(cf_var.cf_attrs_used())
+```
+
+and `test_getattr_of_nothing_at_all_raises_attribute_error`, which used to
+swap in a bare `object()` to get a real miss:
+
+```python
+ def test_getattr_of_nothing_at_all_raises_attribute_error(self, nc_var):
+ nc_var.deprecated_netcdf_member.side_effect = AttributeError("nonesuch")
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.raises(AttributeError, match="nonesuch"):
+ cf_var.nonesuch
+```
+
+In `TestTypedProperties`, the shadowing check:
+
+```python
+ def test_a_file_attribute_does_not_reach_the_typed_properties(self, nc_var):
+ # The properties are looked up before __getattr__ runs, so a file
+ # attribute called "shape" cannot displace the variable's shape.
+ nc_var.attributes = {"shape": "not a shape"}
+ nc_var.shape = (2,)
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.shape == (2,)
+ assert cf_var.attributes["shape"] == "not a shape"
+```
+
+In `TestShadowedAttributeNames`, the fixture:
+
+```python
+ @pytest.fixture
+ def shadowing(self, nc_var):
+ nc_var.attributes = {
+ name: f"file value of {name}" for name in SHADOWED_NAMES
+ } | {"units": "K"}
+ return CFVariableSub("foo", nc_var)
+```
+
+In `TestGetattrAndHasattr`, the dunder probe:
+
+```python
+ def test_dunder_probes_do_not_reach_the_file(self, nc_var):
+ # copy, pickle and numpy all probe for dunders. Answering one from
+ # file data would make a CFVariable behave as whatever the file says.
+ nc_var.attributes = {"__array__": "nonsense"}
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert not hasattr(cf_var, "__array_interface__")
+ assert cf_var.attributes.untracked["__array__"] == "nonsense"
+```
+
+`test_hasattr_false_marks_nothing`, which sets `cf_var.cf_data = object()`,
+needs nothing: `object()` has no `deprecated_netcdf_member` either, so the
+miss is still a miss.
+
+- [ ] **Step 10: Rewrite the `test_CFReader.py` and constraint-callback doubles**
+
+`netcdf_variable()` builds a mock whose CF attributes are ordinary members,
+because that is what `getattr`-based `identify()` read. The dataset now wraps
+each of these in a `NetCDFDatasetVariable`, which reads `ncattrs()` and
+`getncattr()`, so the two have to agree. In
+`lib/iris/tests/unit/fileformats/cf/test_CFReader.py`, replace the
+construction at the end of `netcdf_variable`:
+
+```python
+ members = dict(
+ ancillary_variables=ancillary_variables,
+ coordinates=coordinates,
+ bounds=bounds,
+ climatology=climatology,
+ formula_terms=formula_terms,
+ grid_mapping=grid_mapping,
+ cell_measures=cell_measures,
+ standard_name=standard_name,
+ **{name: None for name in ugrid_identities},
+ )
+ # A None member stood for "no such attribute" when identify() read these
+ # with getattr(..., None). Now that it reads a mapping, absent means
+ # absent - so a None must not be listed.
+ attributes = {key: value for key, value in members.items() if value is not None}
+ ncvar = mocker.Mock(
+ name=name,
+ dimensions=dimensions,
+ ndim=ndim,
+ dtype=dtype,
+ ncattrs=mocker.Mock(return_value=list(attributes)),
+ getncattr=mocker.Mock(side_effect=attributes.__getitem__),
+ **members,
+ )
+ return ncvar
+```
+
+`Test_init_and_lifecycle` patches the class `CFReader` used to name, which it
+no longer does. Two edits in `_setup` and one test:
+
+```python
+ self.encoded_ds = mocker.patch(
+ "iris.fileformats.netcdf._bytecoding_datasets.EncodedDataset",
+ return_value=self.dataset,
+ )
+```
+
+stays exactly as it is — `NetCDFDataset` reaches the class by the same path,
+so the patch still bites. But `test_init_uses_dataset_wrapper_when_string_decode_disabled`
+names `_reader`'s own import, which has gone:
+
+```python
+ wrapper_ds = mocker.patch(
+ "iris.fileformats.netcdf._thread_safe_nc.DatasetWrapper",
+ return_value=self.dataset,
+ )
+```
+
+and `self.dataset` needs the two members `NetCDFDataset` now asks of it,
+beside the three it already has:
+
+```python
+ self.dataset = mocker.Mock(
+ file_format="NetCDF4",
+ variables=self.variables,
+ ncattrs=mocker.Mock(return_value=[]),
+ filepath=mocker.Mock(return_value="in-memory.nc"),
+ set_auto_chartostring=mocker.Mock(),
+ isopen=mocker.Mock(return_value=True),
+ )
+```
+
+`test_init_with_no_meshes_trims_ugrid_variable_types` substitutes variables
+that `_has_meshes` walks, and those are wrapped now. Give both of them the
+netCDF surface a wrapper needs:
+
+```python
+ def test_init_with_no_meshes_trims_ugrid_variable_types(self, mocker):
+ self.dataset.variables = {
+ "a": netcdf_variable(mocker, "a", "x", np.float64),
+ "b": netcdf_variable(mocker, "b", "x", np.float64, mesh="my_mesh"),
+ }
+
+ reader = CFReader("dummy.nc")
+
+ assert reader._with_ugrid is True
+
+ mesh_free = mocker.Mock(
+ file_format="NetCDF4",
+ variables={"a": netcdf_variable(mocker, "a", "x", np.float64)},
+ ncattrs=mocker.Mock(return_value=[]),
+ filepath=mocker.Mock(return_value="in-memory.nc"),
+ set_auto_chartostring=mocker.Mock(),
+ )
+ self.encoded_ds.return_value = mesh_free
+ reader = CFReader("dummy.nc")
+
+ assert reader._with_ugrid is False
+ assert CFUGridMeshVariable not in reader._variable_types
+```
+
+`netcdf_variable` does not take a `mesh` keyword today; add one alongside
+`grid_mapping`, defaulting to `None`, and include it in `members` above.
+It is one of the UGRID identities, so it is already in the `ugrid_identities`
+sweep — the explicit keyword just lets a test set it.
+
+`TestSynthesisedBoundsLink`, added in Task 10, builds `CFVariable`s straight
+from `MagicMock`s. All three of its mocks change the same way — for example
+the first:
+
+```python
+ nc_var = mocker.MagicMock()
+ nc_var.attributes = {"units": "m"}
+ nc_var.dimensions = ("model_level_number",)
+```
+
+with `{"bounds": "from_file"}` and `{"bounds": "broken"}` in the other two.
+
+`lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py`
+next. Task 9 gave it a `_data_variable` helper when it rewrote the constraint
+fast path; the helper's body becomes:
+
+```python
+def _data_variable(mocker, name, **attributes):
+ """A CFDataVariable whose CF attributes are exactly the ones named."""
+ return CFDataVariable(name, mocker.MagicMock(attributes=attributes))
+```
+
+The call sites do not move — that is the whole reason Task 9 introduced a
+helper instead of editing seven mock constructions twice.
+
+Then `lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py`,
+which is the direct test of everything Step 7 rewrote. Its `_make` builds a
+`cf_data` that answers netCDF4's API, and all four of those answers changed.
+Replace the body (`:26-49`) with:
+
+```python
+ def _make(self, chunksizes=None, shape=None, dtype="i4", **extra_properties):
+ if shape is None:
+ shape = self.shape
+ cf_data = self.mocker.MagicMock(
+ spec=iris.fileformats.netcdf._dataset.NetCDFDatasetVariable,
+ fill_value=None,
+ dimensions=tuple("dim_" + str(x) for x in range(len(shape))),
+ shape=shape,
+ chunking=chunksizes,
+ is_variable_length=False,
+ is_emulated=False,
+ )
+ if dtype is not str: # for testing VLen str arrays (dtype=`class <str>`)
+ dtype = np.dtype(dtype)
+ cf_var = self.mocker.MagicMock(
+ spec=iris.fileformats.cf.CFVariable,
+ dtype=dtype,
+ cf_data=cf_data,
+ filename=self.filename,
+ cf_name="DUMMY_VAR",
+ shape=shape,
+ size=np.prod(shape),
+ **extra_properties,
+ )
+ cf_var.__getitem__.return_value = self.mocker.sentinel.real_data_accessed
+ return cf_var
+```
+
+Four changes, each matching one line of Step 7:
+
+- `chunking` is a plain value, not `MagicMock(return_value=...)`. It is a
+ property now. The tests that pass `"contiguous"` —
+ `test_cf_data_contiguous` — must pass `None` instead, because the property
+ has already collapsed netCDF's two spellings of "unchunked" into one. Keep
+ the test; rename it `test_cf_data_unchunked` and merge it into
+ `test_cf_data_no_chunks`'s expectation. Its comment, "Chunks 'contiguous'
+ is equivalent to no chunks", becomes a comment on
+ `NetCDFDatasetVariable.chunking` in `_dataset.py`, where the collapsing now
+ happens — that is where a reader needs it.
+- `is_variable_length` replaces `datatype`. The six `test_vltype__*` tests
+ pass `datatype=mock_vltype` as an `extra_properties` keyword, which lands on
+ `cf_var`, not `cf_data`. They become `_make(..., cf_data_is_variable_length=True)`
+ — or, less cleverly and better, set it directly:
+
+```python
+ def test_vltype__1000str_is_lazy(self):
+ cf_var = self._make(shape=(1000,), dtype=str)
+ cf_var.cf_data.is_variable_length = True
+ var_data = _get_cf_var_data(cf_var)
+ assert isinstance(var_data, da.Array)
+```
+
+ and the `VLType` import goes with the last of them.
+- `test_cf_data_emulation` passes `_data_array=emulated_data`, which likewise
+ landed on `cf_var`. It becomes:
+
+```python
+ def test_cf_data_emulation(self, mocker):
+ # Check that a variable emulation object passes its real data directly.
+ emulated_data = mocker.Mock()
+ cf_var = self._make(chunksizes=None)
+ cf_var.cf_data.is_emulated = True
+ cf_var.cf_data.emulated_data_array = emulated_data
+ result = _get_cf_var_data(cf_var)
+ # This should get directly returned.
+ assert emulated_data is result
+```
+
+- `spec=` on `cf_data` is the point of the whole edit. Without it, every one
+ of these reads returns a fresh `MagicMock` and every test passes whatever
+ Step 7 wrote. With it, a member Step 7 invented but did not implement is an
+ `AttributeError` here, in the test that exists to catch it.
+
+`_FillValue` becomes `fill_value`: the interface's name for it. It is never
+read in this file, but leaving a netCDF4 spelling on a spec'd mock is a
+`AttributeError` waiting for whoever adds the next test.
+
+Last, `lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_dimension_coordinate.py`.
+Two mocks there carry `cf_data=Mock(spec=[])` with a following
+`delattr(..., "_data_array")` (`:60` and `:98`, the `delattr`s at `:77` and
+`:111`), which was how a test said "this is a real file variable, not an
+emulated one". `spec=[]` means the mock has *no* attributes, so
+`cf_var.cf_data.is_emulated` raises. Both become:
+
+```python
+ cf_data = self.mocker.Mock(spec=["is_emulated"], is_emulated=False)
+```
+
+and both `delattr(..., "_data_array")` lines go. The `del cf_data.flag_values`
+trio above the first one stays and still works — `Mock.__delattr__` tolerates
+deleting a name the spec never had **[verified]**.
+
+- [ ] **Step 11: Run the swap's own tests, then the two packages**
+
+Run: `pytest lib/iris/tests/unit/fileformats/cf/ lib/iris/tests/unit/fileformats/netcdf/`
+Expected: PASS, 0 failed, **2 skipped** — the two pre-existing skips §6.2
+records, one in each directory. This is the first test run since Step 2, and
+the first point at which the tree is consistent again.
+
+If `test_CFReader.py` fails in bulk with `KeyError` from `getncattr`, the
+`netcdf_variable` rewrite and the mock's members have fallen out of step —
+`ncattrs()` is listing a name `getncattr` cannot supply. If a single
+`identify()` test fails, compare its stub against Step 8.
+
+- [ ] **Step 12: Run the load path end to end, then commit**
+
+Run: `pytest -n auto lib/iris/tests/unit/ lib/iris/tests/integration/`
+Expected: matches the baseline — §6.3's diff is empty and the skip count has
+not risen. Concretely: `lib/iris/tests/unit/` alone is `7540 passed,
+35 skipped, 0 failed` at baseline **[verified]** — the 35 are gdal,
+`iris_grib`, cartopy and FUTURE-context guards spread over seven modules, not
+a smell — and the integration half is `0 failed` with only F14's ten
+`test_coord_systems.py` errors. A new skip here means a test stopped running
+rather than started passing; that is the failure mode this step exists to
+catch, and it is the one the tail line will not shout about.
+
+```bash
+pre-commit run --files lib/iris/fileformats/cf/_variables.py \
+ lib/iris/fileformats/cf/_reader.py \
+ lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/fileformats/netcdf/loader.py \
+ lib/iris/tests/unit/fileformats/cf/identify_mixins.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFCoordinateVariable.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFVariable.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFReader.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py \
+ lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_dimension_coordinate.py
+git add lib/iris/fileformats/cf/ lib/iris/fileformats/netcdf/ \
+ lib/iris/tests/unit/fileformats/cf/ lib/iris/tests/unit/fileformats/netcdf/ \
+ lib/iris/tests/unit/fileformats/nc_load_rules/
+git commit -m "$(cat <<'EOF'
+Build a NetCDFDataset in CFReader, so cf_data is a CFDatasetVariable
+
+The swap this PR was for. CFReader opened a netCDF dataset and handed
+its variables straight out; every CFVariable therefore wrapped a
+netCDF4.Variable, and so did every consumer downstream. It now opens a
+NetCDFDataset, and cf_data is a CFDatasetVariable.
+
+Nothing outside iris.fileformats.netcdf calls netCDF4 API any more. The
+twelve identify() reads, CFVariable.__init__, CFReader's mesh probe,
+grid-mapping parse and global attributes, and the loader's five storage
+questions all go through the interface instead. _getncattr is gone:
+_NetCDFAttributes absorbs the malformed-file case it existed for.
+
+CFVariable copies the attribute mapping rather than wrapping it. The
+storage object's mapping writes through to the file, and CFReader
+synthesises a bounds link during load, on a file opened read-only.
+
+The test doubles that stood in for a netCDF4 variable stand in for a
+NetCDFDatasetVariable now, and the ones in test__get_cf_var_data.py gain
+a spec= so that an invented member is an error rather than a mock. No
+assertion changed.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+- [ ] **Step 13: Write the failing test for the deprecation warning**
+
+Nothing in Iris reaches netCDF4 API through a `CFVariable` any more, so the
+route can start telling third-party code that it is going away. It could not
+have warned before this point: Tasks 6 to 10 left consumers on it one at a
+time, and a warning would have fired thousands of times per load.
+
+In `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py`,
+replace the two tests Task 2 wrote for this method:
+
+```python
+class TestDeprecatedNetcdfMember:
+ def test_a_reach_through_warns(self, air):
+ with pytest.warns(IrisDeprecation, match="ncattrs"):
+ names = air.deprecated_netcdf_member("ncattrs")()
+ assert sorted(names) == ["long_name", "standard_name", "units"]
+
+ def test_a_missing_name_raises_and_does_not_warn(self, air):
+ # hasattr() probes land here, and a probe that comes back False is
+ # not a use of anything - warning about it would be noise.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ with pytest.raises(AttributeError, match="no_such_netcdf_member"):
+ air.deprecated_netcdf_member("no_such_netcdf_member")
+
+ def test_the_message_names_the_replacement(self, air):
+ with pytest.warns(IrisDeprecation, match=r"cf_data\.variable"):
+ air.deprecated_netcdf_member("ncattrs")
+```
+
+Add to the module's imports:
+
+```python
+import warnings
+
+from iris._deprecation import IrisDeprecation
+```
+
+Adjust `sorted(names)` to whatever the module's `air` fixture actually
+carries if it differs — Task 2 wrote that fixture, and this assertion is
+lifted from its own `test_deprecated_netcdf_member_reaches_the_wrapper`.
+
+- [ ] **Step 14: Run them to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py -k Deprecated -v`
+Expected: `test_a_reach_through_warns` and `test_the_message_names_the_replacement`
+FAIL with `DID NOT WARN`. `test_a_missing_name_raises_and_does_not_warn`
+passes already.
+
+- [ ] **Step 15: Make the reach-through warn**
+
+In `lib/iris/fileformats/netcdf/_dataset.py`, replace
+`NetCDFDatasetVariable.deprecated_netcdf_member`:
+
+```python
+ def deprecated_netcdf_member(self, name: str) -> Any:
+ """Return a member of the backing netCDF variable wrapper, with a warning."""
+ # Fetch before warning, so that a name the wrapper does not have is an
+ # ordinary AttributeError. hasattr() probes arrive here, and a probe
+ # that comes back False has not used anything.
+ value = getattr(self._variable, name)
+ warn_deprecated(
+ f"Reaching netCDF variable member {name!r} through a CFVariable is "
+ "deprecated and will be removed in a future release. Use the CF "
+ "dataset interface instead - iris.fileformats.cf.dataset - or, for "
+ "a netCDF-only need, cf_var.cf_data.variable."
+ )
+ return value
+```
+
+Add to the module's imports:
+
+```python
+from iris._deprecation import warn_deprecated
+```
+
+The base class's `CFDatasetVariable.deprecated_netcdf_member` (Task 1) keeps
+raising `AttributeError`: a backend with no netCDF underneath it has nothing
+to deprecate, and nothing to warn about.
+
+- [ ] **Step 16: Run them to verify they pass, and that nothing in Iris trips it**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py -k Deprecated -v`
+Expected: PASS.
+
+Then prove the route is unused, which is the claim that entitles it to warn:
+
+Run: `pytest -W "error:Reaching netCDF variable member" lib/iris/tests/unit/fileformats/cf/ lib/iris/tests/unit/fileformats/netcdf/ lib/iris/tests/integration/`
+Expected: PASS. Any failure here means a site in `lib/iris/` still reaches
+through — find it in the traceback and move it onto `.attributes` or onto a
+named member of the interface, in this task.
+
+(Corrected in Task 13. As first written this step said
+`-W error::iris._deprecation.IrisDeprecation`, which cannot run at all: it
+exits 4 during conftest import, because `lib/iris/tests/graphics/__init__.py`
+calls `warn_deprecated` for `GraphicsTestMixin` at import time. That file and
+`lib/iris/_deprecation.py` are byte-identical to the merge base, so it is
+pre-existing and unrelated to this PR — and it means the check as written was
+never runnable. Task 13 Step 7 carries the same correction; this copy was
+missed when that one was made. The message-targeted filter is narrower *and*
+stronger: it turns only this PR's own reach-through warning into an error, so
+an unrelated pre-existing deprecation can neither break the run nor mask the
+thing being checked — which is also why the old "or it is an unrelated Iris
+deprecation, note it and move on" escape hatch has gone.)
+
+- [ ] **Step 17: Green check and commit**
+
+```bash
+pre-commit run --files lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/
+git add lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py
+git commit -m "$(cat <<'EOF'
+Deprecate reaching netCDF4 API through a CFVariable
+
+CFVariable.__getattr__ falls back to the backing variable for any name
+the CF attributes do not supply. That was how the class worked, and
+third-party code will have relied on it, so it stays for one release
+cycle - but it now says so.
+
+The warning waits until here because it could not have been switched on
+earlier: consumers came off the fallback one task at a time, and a
+warning during that would have fired once per attribute per variable per
+load. Nothing in Iris takes the route now, which the -W error run over
+the netCDF, CF and integration suites demonstrates.
+
+It warns only on success. hasattr() probes reach this method, and a
+probe that comes back False has not used anything.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 12: `netcdf/saver.py` — `Saver` builds and uses a `NetCDFDataset`
+
+The last consumer, and the only one that writes. Everything the saver does
+to a file goes through four verbs — declare a dimension, create a variable,
+set an attribute, put data in — and this task routes all four through
+`CFDataset`. After it, `saver.py` names `netCDF4` in exactly one place: the
+`self._dataset.dataset` escape hatch, kept deliberately and **three times**
+only. (Corrected in Task 13, which counted them: the `file_format` read
+that decides whether int64 is available, the second `file_format` read in
+the error message it raises, and the third-party `cf_patch` hook,
+documented since Iris 1.3 as receiving netCDF4 objects. The step text below
+already says three; only this summary said two.)
+
+**Files:**
+- Modify: `lib/iris/fileformats/netcdf/saver.py` (throughout; sites listed per step)
+- Modify: `lib/iris/fileformats/netcdf/_dataset.py` (`NetCDFDataset.closed`)
+- Test: `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py` (create)
+- Test: `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dataset.py` (create)
+- Test: `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py` (mocks and patch targets)
+- Test: `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__lazy_stream_data.py` (mocks)
+
+**Interfaces:**
+- Consumes: `NetCDFDataset`, `NetCDFDatasetVariable` and `_bytes_if_ascii`
+ from Tasks 2–4; `NetCDFDataset.write_lock`; `NetCDFDatasetVariable.attributes`,
+ `.write_handle()`, `.is_emulated`, `.emulated_data_array`, `.variable`.
+- Produces:
+ - `Saver._dataset` is a `NetCDFDataset`, not an `EncodedDataset`. Anything
+ reaching for netCDF4 API on it must go via `Saver._dataset.dataset`.
+ - `Saver.file_write_lock` is `Saver._dataset.write_lock` — the *same*
+ object every `NetCDFDatasetVariable` of that dataset holds.
+ - `saver._setncattr` is **deleted**. `saver._bytes_if_ascii` was already
+ deleted in Task 4; its re-import goes too.
+ - `saver.CFVariable` (the type alias, not `cf.CFVariable`) becomes
+ `NetCDFDatasetVariable`.
+ - `NetCDFDataset.closed` reports a dataset closed behind its back.
+
+> **One long red stretch.** Steps 5–10 convert the saver. There is no useful
+> place to stop in the middle: the saver creates variables in one method and
+> writes attributes to them in another, so a half-converted saver cannot write
+> a file at all. Steps 5–10 therefore run with no test run between them, and
+> the green check is Step 11. This is the second and last such stretch in the
+> plan; Task 11's note explains why they are worth one commit each.
+
+- [ ] **Step 1: Write the Review Focus 4 test — saving into a dataset somebody else opened**
+
+This is a **characterisation** test: it passes against today's code, before
+anything in this task changes. `iris.fileformats.netcdf.save(cube, dataset)`
+— the Xarray bridge, [#4994](https://github.com/SciTools/iris/issues/4994),
+and the route [ncdata](https://github.com/pp-mo/ncdata) takes — has no test
+anywhere in the suite today, and this task rewrites every line it touches.
+Writing the guard first is the only way the conversion has anything to break.
+
+Create
+`lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Tests for saving into a dataset the caller opened, or is only pretending to.
+
+``iris.fileformats.netcdf.save(cube, dataset, compute=False)`` accepts an
+open dataset in place of a path. Two kinds of caller do this:
+
+* one holding a real, open netCDF4 dataset, who wants Iris to add to it;
+* one holding an object that merely *looks* like a netCDF4 dataset, and
+ collects what Iris writes rather than storing it - the "Xarray bridge",
+ https://github.com/SciTools/iris/issues/4994, which is how
+ https://github.com/pp-mo/ncdata translates between Iris and Xarray.
+
+The second is the reason ``Saver`` may not assume its dataset is netCDF4,
+and it is entirely untested elsewhere.
+
+"""
+
+import dask
+import numpy as np
+import pytest
+
+import iris
+from iris.coords import DimCoord
+from iris.cube import Cube
+import iris.fileformats.netcdf as inetcdf
+from iris.fileformats.netcdf import _thread_safe_nc as threadsafe_nc
+
+
+@pytest.fixture(autouse=True)
+def split_attrs():
+ # Avoid the legacy-attribute deprecation warning; irrelevant here.
+ with iris.FUTURE.context(save_split_attrs=True):
+ yield
+
+
+@pytest.fixture
+def cube():
+ cube = Cube(
+ np.arange(6.0).reshape(2, 3),
+ standard_name="air_temperature",
+ units="K",
+ var_name="air",
+ )
+ cube.add_dim_coord(
+ DimCoord(np.arange(2.0), standard_name="latitude", units="degrees"), 0
+ )
+ cube.add_dim_coord(
+ DimCoord(np.arange(3.0), standard_name="longitude", units="degrees"), 1
+ )
+ return cube
+
+
+class TestRealDataset:
+ """A dataset the caller opened, and closes themselves."""
+
+ def test_save_into_an_open_dataset(self, cube, tmp_path):
+ path = tmp_path / "user.nc"
+ dataset = threadsafe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ delayed = inetcdf.save(cube, dataset, compute=False)
+ # The caller owns the dataset, so the caller closes it - and the
+ # delayed writes only resolve once they have.
+ dataset.close()
+ dask.compute(delayed)
+
+ result = iris.load_cube(path)
+ assert result.standard_name == "air_temperature"
+ assert result.units == "K"
+ assert np.array_equal(result.data, cube.data)
+ assert [coord.name() for coord in result.coords()] == [
+ "latitude",
+ "longitude",
+ ]
+
+ def test_saver_does_not_close_what_it_was_given(self, cube, tmp_path):
+ path = tmp_path / "user.nc"
+ dataset = threadsafe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ inetcdf.save(cube, dataset, compute=False)
+ assert dataset.isopen()
+ dataset.close()
+
+ def test_compute_true_is_refused(self, cube, tmp_path):
+ path = tmp_path / "user.nc"
+ dataset = threadsafe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ with pytest.raises(ValueError, match="Cannot save to a user-provided dataset"):
+ inetcdf.save(cube, dataset, compute=True)
+ dataset.close()
+
+
+class _EmulatedDimension:
+ def __init__(self, size):
+ self._size = size
+
+ def __len__(self):
+ return self._size
+
+ def isunlimited(self):
+ return self._size is None
+
+
+class _EmulatedVariable:
+ """The least a netCDF4.Variable emulator can be and still be written to.
+
+ ``_data_array`` is the whole point: an emulator receives its data by
+ having this attribute set, never by ``__setitem__``. ``__setitem__``
+ raises here so that a regression shows up as a failure rather than as
+ data quietly going nowhere.
+
+ """
+
+ def __init__(self, name, datatype, dimensions, shape):
+ self.name = name
+ self.datatype = np.dtype(datatype)
+ self.dtype = self.datatype
+ self.dimensions = tuple(dimensions)
+ self.shape = shape
+ self.size = int(np.prod(shape)) if shape else 1
+ self._data_array = None
+ self._attrs = {}
+
+ def setncattr(self, name, value):
+ self._attrs[name] = value
+
+ def getncattr(self, name):
+ return self._attrs[name]
+
+ def ncattrs(self):
+ return list(self._attrs)
+
+ def chunking(self):
+ return "contiguous"
+
+ def __setitem__(self, keys, values):
+ raise AssertionError("An emulated variable must receive _data_array.")
+
+
+class _EmulatedDataset:
+ """A netCDF4.Dataset emulator, in the shape ncdata presents.
+
+ Deliberately *not* a ``_thread_safe_nc`` wrapper and deliberately without
+ ``THREAD_SAFE_FLAG``, so that the wrapping branch of
+ ``NetCDFDataset.from_existing`` is the one under test.
+
+ """
+
+ def __init__(self):
+ self.variables = {}
+ self.dimensions = {}
+ self.file_format = "NETCDF4"
+ self._attrs = {}
+ self._open = True
+
+ def createDimension(self, name, size):
+ self.dimensions[name] = _EmulatedDimension(size)
+ return self.dimensions[name]
+
+ def createVariable(self, name, datatype, dimensions=(), **kwargs):
+ shape = tuple(len(self.dimensions[name_]) for name_ in dimensions)
+ variable = _EmulatedVariable(name, datatype, dimensions, shape)
+ self.variables[name] = variable
+ return variable
+
+ def setncattr(self, name, value):
+ self._attrs[name] = value
+
+ def getncattr(self, name):
+ return self._attrs[name]
+
+ def ncattrs(self):
+ return list(self._attrs)
+
+ def sync(self):
+ pass
+
+ def close(self):
+ self._open = False
+
+ def isopen(self):
+ return self._open
+
+ def filepath(self):
+ return "<emulated>"
+
+ def set_auto_chartostring(self, onoff):
+ pass
+
+
+class TestEmulatedDataset:
+ """An object that only looks like a dataset - the Xarray bridge."""
+
+ @pytest.fixture
+ def written(self, cube):
+ dataset = _EmulatedDataset()
+ inetcdf.save(cube, dataset, compute=False)
+ return dataset
+
+ def test_variables_are_created(self, written):
+ assert sorted(written.variables) == ["air", "latitude", "longitude"]
+
+ def test_dimensions_are_created(self, written):
+ assert {name: len(dim) for name, dim in written.dimensions.items()} == {
+ "latitude": 2,
+ "longitude": 3,
+ }
+
+ def test_attributes_reach_the_emulator_as_bytes(self, written):
+ # _bytes_if_ascii: an ASCII string attribute is offered as bytes, so
+ # that netCDF4 gives it type NC_CHAR. An emulator sees the same.
+ assert written.variables["air"]._attrs == {
+ "standard_name": b"air_temperature",
+ "units": b"K",
+ }
+ assert written._attrs == {"Conventions": b"CF-1.7"}
+
+ def test_data_arrives_as_a_data_array(self, written, cube):
+ # Not via __setitem__, which the emulated variable refuses.
+ assert np.array_equal(written.variables["air"]._data_array, cube.data)
+ assert np.array_equal(
+ written.variables["latitude"]._data_array, np.arange(2.0)
+ )
+
+ def test_dimensions_are_recorded_on_the_variable(self, written):
+ assert written.variables["air"].dimensions == ("latitude", "longitude")
+```
+
+- [ ] **Step 2: Run the test to verify it passes**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py -v`
+Expected: PASS — all 8 tests, against unmodified `saver.py`.
+
+A green test at the start of a task is unusual, and is the point here: from
+Step 5 the saver stops touching netCDF4 directly, and this file is the only
+thing in the suite that would notice if the emulated path broke. Confirm it
+really bites before relying on it:
+
+```bash
+python - <<'PY'
+import pathlib
+p = pathlib.Path("lib/iris/fileformats/netcdf/saver.py")
+src = p.read_text()
+p.write_text(src.replace('if hasattr(cf_var, "_data_array"):', "if False:", 1))
+PY
+pytest lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py \
+ -k Emulated -q
+git checkout lib/iris/fileformats/netcdf/saver.py
+```
+Expected: the four `TestEmulatedDataset` data/attribute tests FAIL while the
+line is disabled, and the file is restored afterwards.
+
+- [ ] **Step 3: Commit the guard**
+
+```bash
+pre-commit run --files \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py
+git add lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py
+git commit -m "$(cat <<'EOF'
+Characterise saving into a caller-supplied dataset
+
+iris.fileformats.netcdf.save() accepts an open dataset in place of a
+path, and ncdata uses that to translate between Iris and Xarray by
+passing an object that only emulates one. Nothing in the suite tested
+it, and the next commit rewrites every line it runs through.
+
+Written against unmodified code, so it is green on arrival. That is
+what makes it a guard rather than a specification.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+- [ ] **Step 4: Write the failing tests for the dataset swap**
+
+Create
+`lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dataset.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Tests for the dataset :class:`iris.fileformats.netcdf.saver.Saver` owns."""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.netcdf import _thread_safe_nc as threadsafe_nc
+from iris.fileformats.netcdf._dataset import NetCDFDataset
+from iris.fileformats.netcdf.saver import Saver
+
+
+class TestOwnedDataset:
+ @pytest.fixture
+ def saver(self, tmp_path):
+ with Saver(tmp_path / "test.nc", "NETCDF4") as saver:
+ yield saver
+
+ def test_dataset_is_a_netcdf_cfdataset(self, saver):
+ assert isinstance(saver._dataset, NetCDFDataset)
+
+ def test_filepath_is_still_a_path(self, saver, tmp_path):
+ # Public API: Saver.filepath has always been a Path for a local file.
+ assert saver.filepath == (tmp_path / "test.nc").absolute()
+
+ def test_the_write_lock_is_the_datasets_own(self, saver):
+ # Every variable of the dataset holds this same lock. A second lock
+ # would exclude nothing - see NetCDFDataset.write_lock.
+ assert saver.file_write_lock is saver._dataset.write_lock
+
+ def test_format_reaches_the_file(self, tmp_path):
+ with Saver(tmp_path / "classic.nc", "NETCDF3_CLASSIC") as saver:
+ assert saver._dataset.dataset.file_format == "NETCDF3_CLASSIC"
+
+ def test_exit_closes_an_owned_dataset(self, tmp_path):
+ with Saver(tmp_path / "test.nc", "NETCDF4") as saver:
+ pass
+ assert saver._dataset.closed
+
+
+class TestBorrowedDataset:
+ @pytest.fixture
+ def raw(self, tmp_path):
+ dataset = threadsafe_nc.DatasetWrapper(
+ tmp_path / "user.nc", mode="w", format="NETCDF4"
+ )
+ yield dataset
+ if dataset.isopen():
+ dataset.close()
+
+ def test_dataset_is_a_netcdf_cfdataset(self, raw):
+ with Saver(raw, "NETCDF4", compute=False) as saver:
+ assert isinstance(saver._dataset, NetCDFDataset)
+
+ def test_exit_leaves_a_borrowed_dataset_open(self, raw):
+ with Saver(raw, "NETCDF4", compute=False) as saver:
+ pass
+ assert raw.isopen()
+ assert not saver._dataset.closed
+
+ def test_complete_refuses_while_the_file_is_open(self, raw):
+ with Saver(raw, "NETCDF4", compute=False) as saver:
+ pass
+ with pytest.raises(ValueError, match="until its dataset is closed"):
+ saver.complete()
+
+ def test_complete_sees_a_dataset_closed_behind_its_back(self, raw):
+ # The caller owns the dataset and closes it themselves, so the flag
+ # NetCDFDataset.close() sets is never set. complete() has to ask the
+ # file, not the flag.
+ with Saver(raw, "NETCDF4", compute=False) as saver:
+ pass
+ raw.close()
+ assert saver._dataset.closed
+ saver.complete() # must not raise
+
+
+class TestWritingThroughTheDataset:
+ """The four verbs, through CFDataset rather than netCDF4."""
+
+ @pytest.fixture
+ def saver(self, tmp_path):
+ with Saver(tmp_path / "test.nc", "NETCDF4") as saver:
+ yield saver
+
+ def test_create_dimension(self, saver):
+ saver._dataset.create_dimension("x", 3)
+ assert saver._dataset.dimensions["x"] == 3
+
+ def test_create_variable_returns_a_cf_dataset_variable(self, saver):
+ from iris.fileformats.netcdf._dataset import NetCDFDatasetVariable
+
+ saver._dataset.create_dimension("x", 3)
+ variable = saver._dataset.create_variable("a", np.dtype("f4"), ("x",))
+ assert isinstance(variable, NetCDFDatasetVariable)
+ assert saver._dataset.variables["a"] is variable
+```
+
+- [ ] **Step 5: Run the tests to verify they fail**
+
+Run: `pytest lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dataset.py -v`
+Expected: FAIL — `assert isinstance(saver._dataset, NetCDFDataset)` fails
+(it is an `EncodedDataset`), `saver._dataset.write_lock` raises
+`AttributeError`, and `saver._dataset.closed` raises `AttributeError`.
+
+- [ ] **Step 6: Build a `NetCDFDataset` in `Saver.__init__`, and use it in `__exit__` and `complete`**
+
+In `lib/iris/fileformats/netcdf/saver.py`, add the import alongside the
+existing netCDF imports at line 65:
+
+```python
+from iris.fileformats.netcdf._dataset import NetCDFDataset, NetCDFDatasetVariable
+```
+
+Replace lines 425-480 (`self._to_open_dataset = ...` down to
+`self.file_write_lock = ...`) with:
+
+```python
+ # Detect if we were passed a pre-opened dataset (or something like one)
+ self._to_open_dataset = hasattr(filename, "createVariable")
+ if self._to_open_dataset:
+ # We were passed a *dataset*, so we don't open (or close) one of our own.
+ if compute:
+ msg = (
+ "Cannot save to a user-provided dataset with 'compute=True'. "
+ "Please use 'compute=False' and complete delayed saving in the "
+ "calling code after the file is closed."
+ )
+ raise ValueError(msg)
+
+ # from_existing() handles the thread-safety wrapping, including the
+ # case of an object that only emulates a dataset and carries no
+ # wrapper of its own.
+ self._dataset = NetCDFDataset.from_existing(filename)
+
+ # In this case the dataset gives a filepath, not the other way around.
+ self.filepath = self._dataset.location
+
+ else:
+ # Given a filepath string/path : create a dataset from that
+ try:
+ # Lazy import to avoid circular import overhead at module import-time.
+ from iris.io import _is_nczarr_fragment
+
+ self._is_nczarr = _is_nczarr_fragment(
+ urlsplit(str(filename)).fragment or None
+ )
+ if self._is_nczarr:
+ # NCZarr URLs contain a #mode= fragment; Path() strips it.
+ # Keep as a plain string and pass directly to the dataset.
+ self.filepath = str(filename)
+ else:
+ filepath = Path(filename)
+ self.filepath = filepath.absolute()
+ self._dataset = NetCDFDataset(
+ self.filepath, mode="w", netcdf_format=netcdf_format
+ )
+ except RuntimeError:
+ if self._is_nczarr:
+ raise
+ dir_name = Path(self.filepath).parent
+ if not dir_name.is_dir():
+ msg = "No such file or directory: {}".format(dir_name)
+ raise IOError(msg)
+ if not os.access(dir_name, os.R_OK | os.W_OK):
+ msg = "Permission denied: {}".format(self.filepath)
+ raise IOError(msg)
+ else:
+ raise
+
+ # One lock for the whole file, shared with every variable of it. A
+ # second get_worker_lock() call would hand back a different
+ # threading.Lock under the threaded scheduler, excluding nothing.
+ self.file_write_lock = self._dataset.write_lock
+```
+
+`Saver.filepath` stays exactly what it was — a `Path` for a local file, the
+URL string for NCZarr, whatever the dataset reports for a borrowed one.
+`NetCDFDataset` keeps its own `str()` of it internally; the two are not the
+same object and are not required to be.
+
+Replace `__exit__` (lines 485-497):
+
+```python
+ def __exit__(self, type, value, traceback):
+ """Flush any buffered data to the CF-NetCDF/NcZarr file before closing."""
+ if self._nczarr_writes:
+ # NCZarr lazy writes must be computed while the file is still open;
+ # the deferred reopen-write pattern used for netCDF is not supported.
+ sources, targets = zip(*self._nczarr_writes)
+ da.store(list(sources), list(targets))
+ self._dataset.sync()
+ if not self._to_open_dataset:
+ # Only close if the Saver created it.
+ self._dataset.close()
+ # Complete after closing, if required
+ if self.compute:
+ self.complete()
+```
+
+`finalise()` is deliberately **not** called here. It exists on the ABC from
+the start because spec §4.5 requires it, and for netCDF it does nothing;
+wiring a no-op in would be a guess at an ordering only Zarr can exercise.
+PR 6 wires it, where it does something. See finding F6.
+
+And in `complete()` (line 2694), replace `if self._dataset.isopen():` with:
+
+```python
+ if not self._dataset.closed:
+```
+
+- [ ] **Step 7: Teach `NetCDFDataset.closed` about a dataset closed behind its back**
+
+`Saver.complete()` is the first caller for which the flag alone is not the
+answer: the documented workflow for a caller-supplied dataset is
+`compute=False`, *caller* closes, `complete()`. `NetCDFDataset.close()` never
+runs in that sequence, because the dataset is borrowed.
+
+In `lib/iris/fileformats/netcdf/_dataset.py`, replace the `closed` property:
+
+```python
+ @property
+ def closed(self) -> bool:
+ """Whether the file has been released - by this object or by its owner.
+
+ The flag :meth:`close` sets is not the whole answer for a borrowed
+ dataset, which whoever opened it closes themselves. Ask the backing
+ object as well, when it can say: an emulating object need not
+ implement ``isopen()``, and is then taken to be open.
+
+ """
+ if self._closed:
+ return True
+ isopen = getattr(self._dataset, "isopen", None)
+ if isopen is None:
+ return False
+ return not isopen()
+```
+
+- [ ] **Step 8: Move dimension and variable creation onto the dataset**
+
+Four `createDimension` calls become `create_dimension`, at lines 833, 929,
+1588 and 1930:
+
+```python
+ self._dataset.create_dimension(dim_name, size) # 833
+ self._dataset.create_dimension(last_dim, length) # 929
+ self._dataset.create_dimension( # 1588
+ bounds_dimension_name, n_bounds
+ )
+ self._dataset.create_dimension( # 1930
+ string_dimension_name, string_dimension_depth
+ )
+```
+
+Five `createVariable` calls become `create_variable`, at lines 1592, 1720,
+1938, 1983 and 2080:
+
+```python
+ cf_var_bounds = self._dataset.create_variable( # 1592
+ boundsvar_name,
+ bounds.dtype.newbyteorder("="),
+ cf_var.dimensions + (bounds_dimension_name,),
+ **compression_kwargs,
+ )
+
+ # Create the main variable
+ cf_mesh_var = self._dataset.create_variable( # 1720
+ cf_mesh_name,
+ np.dtype(np.int32),
+ )
+ # 'dimensions' defaults to (), so the empty list goes - finding F4.
+
+ cf_var = self._dataset.create_variable( # 1938
+ cf_name, "|S1", element_dims
+ )
+
+ cf_var = self._dataset.create_variable( # 1983
+ cf_name,
+ dtype,
+ element_dims,
+ fill_value=fill_value,
+ **compression_kwargs,
+ )
+
+ cf_var_grid = self._dataset.create_variable( # 2080
+ cs.grid_mapping_name, np.int32
+ )
+```
+
+`create_variable` normalises `dimensions` with `tuple()`, so the lists the
+saver passes (`element_dims`, `["dim0", "dim1"]`) reach netCDF4 as tuples.
+That is invisible to netCDF4 and visible to two mock-based tests, which
+Step 11 updates.
+
+Then the grid-mapping escape hatch. `_add_grid_mapping_to_dataset`
+(lines 2067-2272) sets sixty-three CF grid-mapping parameters by plain Python
+attribute assignment, one per CF parameter name. They bypass
+`_bytes_if_ascii` today — `crs_wkt` at line 2273 is written as `NC_STRING`,
+not `NC_CHAR` — so converting them to `attributes[...]` entries would change
+what lands in the file. That is finding F8, and it is not this PR's job.
+Keep them on the netCDF4 variable, named once:
+
+```python
+ cf_var_grid = self._dataset.create_variable(
+ cs.grid_mapping_name, np.int32
+ )
+ cf_var_grid.attributes["grid_mapping_name"] = cs.grid_mapping_name
+
+ # The sixty-three assignments below set CF grid-mapping parameters by
+ # Python attribute assignment. Unlike every other attribute the saver
+ # writes, they bypass the ASCII-to-bytes coercion, so moving them onto
+ # .attributes would change the file. See finding F8; until then they
+ # need the netCDF4 variable itself.
+ grid_variable = cf_var_grid.variable
+```
+
+and rename the rest mechanically:
+
+```bash
+python - <<'PY'
+import pathlib
+p = pathlib.Path("lib/iris/fileformats/netcdf/saver.py")
+lines = p.read_text().splitlines(keepends=True)
+start = next(
+ i for i, line in enumerate(lines) if "grid_variable = cf_var_grid.variable" in line
+)
+end = next(
+ i for i, line in enumerate(lines) if "def _create_cf_grid_mapping" in line
+)
+for i in range(start + 1, end):
+ lines[i] = lines[i].replace("cf_var_grid.", "grid_variable.")
+p.write_text("".join(lines))
+print("renamed in lines", start + 2, "to", end)
+PY
+```
+Expected: `renamed in lines <n> to <m>`, and
+`grep -c "cf_var_grid\." lib/iris/fileformats/netcdf/saver.py` reports `2`.
+
+- [ ] **Step 9: Move every attribute write onto `.attributes`**
+
+Thirty-three of the thirty-four `_setncattr` calls are a single line of the
+form `_setncattr(target, name, value)`. Rewrite them by splitting the
+argument list on top-level commas, so that values containing commas inside
+their own brackets survive:
+
+```bash
+python - <<'PY'
+import pathlib
+import re
+
+SRC = pathlib.Path("lib/iris/fileformats/netcdf/saver.py")
+lines = SRC.read_text().splitlines(keepends=True)
+
+
+def split_args(text):
+ """Split a call's argument text on top-level commas."""
+ args, depth, start = [], 0, 0
+ for i, ch in enumerate(text):
+ if ch in "([{":
+ depth += 1
+ elif ch in ")]}":
+ depth -= 1
+ elif ch == "," and depth == 0:
+ args.append(text[start:i].strip())
+ start = i + 1
+ args.append(text[start:].strip())
+ return args
+
+
+out, changed = [], 0
+for line in lines:
+ match = re.match(r"^(\s*)_setncattr\((.*)\)\s*$", line.rstrip("\n"))
+ if match and match.group(2):
+ indent, inner = match.groups()
+ args = split_args(inner)
+ if len(args) == 3:
+ target, name, value = args
+ out.append(f"{indent}{target}.attributes[{name}] = {value}\n")
+ changed += 1
+ continue
+ out.append(line)
+
+SRC.write_text("".join(out))
+print("rewritten:", changed)
+PY
+```
+Expected: `rewritten: 33`.
+
+Then the one multi-line call, at line 1728:
+
+```python
+ cf_mesh_var.attributes["topology_dimension"] = np.int32(
+ mesh.topology_dimension
+ )
+```
+
+Then delete `_setncattr` itself (lines 290-301) and the `_bytes_if_ascii`
+import Task 4 left behind. Nothing in Iris calls either any more: the
+coercion now happens inside `_NetCDFAttributes.__setitem__`, once, for every
+attribute the saver writes.
+
+Three attribute *reads* on netCDF variables also have to move, and they are
+the ones that would fail silently if missed. `NetCDFDatasetVariable` has no
+`__getattr__` fallback — deliberately, see `lib/iris/AGENTS.md` — so
+`hasattr(cf_var, "formula_terms")` would simply become permanently `False`
+and the dimensionless-vertical-coordinate branch would stop running.
+
+At line 1182, in `_add_aux_factories`:
+
+```python
+ if "formula_terms" in cf_var.attributes:
+ if (
+ cf_var.attributes["formula_terms"] != formula_terms
+ or cf_var.attributes["standard_name"] != std_name
+ ):
+```
+
+At lines 1226-1240, in the same method's `FUTURE.derived_bounds` block:
+
+```python
+ bounds_varname = cf_var.attributes.get("bounds")
+ cf_bounds_var = self._dataset.variables.get(bounds_varname, None)
+ if (
+ cf_bounds_var is not None
+ and cf_bounds_var.attributes.get("formula_terms") is None
+ ):
+```
+
+and, inside `boundsterm_varname`:
+
+```python
+ termvar = self._dataset.variables.get(term_varname)
+ boundsname = termvar.attributes.get("bounds")
+```
+
+`termvar` cannot be `None` here: `term_varname` comes from
+`self._name_coord_map`, which only holds names the saver has created.
+`getattr(None, "bounds", None)` would have returned `None` silently; if the
+assumption is ever wrong an `AttributeError` now says so.
+
+At line 1802, in `_set_cf_var_attributes`:
+
+```python
+ # Don't clobber existing attributes.
+ if name not in cf_var.attributes:
+ cf_var.attributes[name] = value
+```
+
+This one is a **deliberate behaviour change**, and the saver-side mirror of
+Review Focus 1. `hasattr(cf_var, name)` asked a netCDF4 variable whether it
+had a Python member of that name, so a coordinate carrying an attribute
+called `shape`, `size`, `name`, `dtype`, `dimensions` or `mask` was silently
+dropped: `hasattr` said yes, and the value never reached the file. netCDF
+itself has no such restriction — `setncattr("shape", ...)` and
+`getncattr("shape")` both work, and the array shape is still reported
+correctly — so the attribute now round-trips. Finding F13.
+
+Two netCDF-only reads stay, through the escape hatch. `_ensure_valid_dtype`
+needs the file format to know whether int64 is available, at lines 1516 and
+1536:
+
+```python
+ ) and self._dataset.dataset.file_format in (
+ msg = msg.format(src_name, src_object, self._dataset.dataset.file_format)
+```
+
+and `cf_patch`, at line 733, is a third-party hook documented as receiving a
+netCDF4 dataset:
+
+```python
+ cf_patch(profile, self._dataset.dataset, cf_var_cube)
+```
+
+`self._dataset.dataset` is deliberately ugly. Grepping for it finds every
+place `saver.py` still assumes netCDF4, and there are exactly three.
+
+- [ ] **Step 10: Route data writes through the variable**
+
+Replace `_lazy_stream_data` (lines 2598-2656):
+
+```python
+ def _lazy_stream_data(
+ self,
+ data: np.typing.ArrayLike,
+ cf_var: NetCDFDatasetVariable,
+ ) -> None:
+ if hasattr(data, "shape") and data.shape == (1,) + cf_var.shape:
+ # (Don't do this check for string data).
+ # Reduce dimensionality where the data array has an extra dimension
+ # versus the cf_var - to avoid a broadcasting ambiguity.
+ # Happens when bounds data is for a scalar point - array is 2D but
+ # contains just 1 row, so the cf_var is 1D.
+ data = data.squeeze(axis=0)
+
+ if cf_var.is_emulated:
+ # The variable is not an actual file variable, but an emulating
+ # object with an attached data array (either numpy or dask), which should
+ # be copied immediately to the target. This is used as a hook to translate
+ # data to/from netcdf data container objects in other packages, such as
+ # xarray.
+ # See https://github.com/SciTools/iris/issues/4994 "Xarray bridge".
+ cf_var.emulated_data_array = data
+
+ else:
+ doing_delayed_save = is_lazy_data(data)
+ if doing_delayed_save:
+ if self._is_nczarr:
+ # NCZarr cannot be reopened for deferred writes.
+ # Collect here; da.store() will be called in __exit__.
+ self._nczarr_writes.append((data, cf_var))
+ return
+
+ # save lazy data with a delayed operation. For now, we just record the
+ # necessary information -- a single, complete delayed action is
+ # constructed later by a call to delayed_completion().
+ def store(
+ data: np.typing.ArrayLike,
+ cf_var: NetCDFDatasetVariable,
+ ) -> None:
+ # Ask the variable for something a worker can stream into
+ # after this file is closed. What that is is the backend's
+ # business: netCDF reopens the file, Zarr will hand back the
+ # array itself.
+ self._delayed_writes.append((data, cf_var.write_handle()))
+
+ else:
+ # Real data is always written directly, i.e. not via lazy save.
+ def store(
+ data: np.typing.ArrayLike,
+ cf_var: NetCDFDatasetVariable,
+ ) -> None:
+ cf_var[:] = data # type: ignore[index]
+
+ # Store the data.
+ store(data, cf_var)
+```
+
+The write proxy was built here from `self.filepath`, `cf_var` and
+`self.file_write_lock`; the variable now supplies all three itself, and
+`file_write_lock` is the dataset's own lock, so the two are the same object
+rather than two locks that happened to agree.
+
+Then the type alias at line 326, and the import at line 67:
+
+```python
+# A saver variable is whatever the dataset hands back. For netCDF that is a
+# NetCDFDatasetVariable, whether it wraps a real file variable or an
+# emulating object - see VariableEmulator, above.
+CFVariable = NetCDFDatasetVariable
+```
+
+`from iris.fileformats.netcdf import _thread_safe_nc as threadsafe_nc` was
+used only by the three type hints just replaced: delete it.
+`bytecoding_datasets` stays — lines 1916 and 1919 still need
+`_identify_encoding` and `_ENCODING_WIDTH_TRANSLATIONS`.
+
+- [ ] **Step 11: Update the mocks that stood in for netCDF4 objects**
+
+Three existing test modules build mocks shaped like netCDF4 objects, and now
+need mocks shaped like `CFDataset` ones.
+
+`lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py`, `test_zlib`
+(lines 218-246). It patched the module `saver.py` opened its dataset from;
+`saver.py` no longer opens one, so patch the module that does, and expect a
+tuple of dimension names:
+
+```python
+ def test_zlib(self, mocker):
+ cube = self._simple_cube(">f4")
+ api = mocker.patch(
+ "iris.fileformats.netcdf._dataset._bytecoding_datasets"
+ )
+ # Mock the apparent dtype of mocked variables, to avoid an error.
+ dataset = api.EncodedDataset.return_value
+ dataset.createVariable.return_value.dtype = np.dtype(np.float32)
+ # NOTE: use compute=False as otherwise it gets in a pickle trying to construct
+ # a fill-value report on a non-compliant variable in a non-file (!)
+ with Saver("/dummy/path", "NETCDF4", compute=False) as saver:
+ saver.write(cube, zlib=True)
+ create_var_call = mocker.call(
+ "air_pressure_anomaly",
+ np.dtype("float32"),
+ ("dim0", "dim1"),
+ fill_value=None,
+ shuffle=True,
+ least_significant_digit=None,
+ contiguous=False,
+ zlib=True,
+ fletcher32=False,
+ endian="native",
+ complevel=4,
+ chunksizes=None,
+ )
+ assert create_var_call in dataset.createVariable.call_args_list
+```
+
+The `api.default_fillvals` line goes with it: `saver.py` never read
+`bytecoding_datasets.default_fillvals`, so the line has been inert since the
+netCDF modules were split, and it now patches a module the saver does not
+touch at all.
+
+The three `createvar_spy` blocks in the same file (lines 273-279, 313-319,
+353-359) keep their patch target — `NetCDFDataset.create_variable` calls
+`EncodedDataset.createVariable`, so the spy still sees every call — but
+`wraps=` must name the object that now has that method:
+
+```python
+ wraps=saver._dataset.dataset.createVariable,
+```
+
+`_check_bounds_setting` (lines 499-527) builds a variable mock and asserts on
+`setncattr`:
+
+```python
+ var = self.mocker.MagicMock(spec=NetCDFDatasetVariable)
+ var.attributes = {}
+ var.dimensions = ("time",)
+
+ # Make the main call.
+ Saver._create_cf_bounds(saver, coord, var, "time")
+
+ # Test the attribute written by _create_cf_bounds. The ASCII-to-bytes
+ # coercion now happens inside _NetCDFAttributes.__setitem__, and is
+ # tested there; this plain dict stands in for one, so the value
+ # arrives as given.
+ assert var.attributes[property_name] == boundsvar_name
+
+ # Test the call of create_variable in _create_cf_bounds.
+ dataset = saver._dataset
+ expected_dimensions = var.dimensions + ("bnds",)
+ create_var_call = self.mocker.call(
+ boundsvar_name, coord.bounds.dtype, expected_dimensions
+ )
+ assert create_var_call == dataset.create_variable.call_args
+```
+
+with `from iris.fileformats.netcdf._dataset import NetCDFDatasetVariable`
+added to the imports, and the now-unused `ds_wrappers.EncodedVariable`
+reference removed if nothing else in the file needs it.
+
+`lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__lazy_stream_data.py`,
+`mock_var` (lines 61-80) and the assertions that use it:
+
+```python
+ @staticmethod
+ def mock_var(shape, with_data_array, mocker):
+ # Create a test cf_var object.
+ # 'is_emulated' is now a declared property rather than the presence of
+ # a '_data_array' member, so it can simply be set.
+ mock_cfvar = mocker.MagicMock(
+ spec=dataset_module.NetCDFDatasetVariable,
+ shape=tuple(shape),
+ dtype=np.dtype(np.float32),
+ is_emulated=with_data_array,
+ )
+ mock_cfvar.write_handle.return_value = mocker.sentinel.write_handle
+ # Give the mock cf-var a name property, as required by '_lazy_stream_data'.
+ # This *can't* be an extra kwarg to MagicMock __init__, since that already
+ # defines a specific 'name' kwarg, with a different purpose.
+ mock_cfvar.name = "<mock_cfvar>"
+ return mock_cfvar
+```
+
+with `import iris.fileformats.netcdf._dataset as dataset_module` added, and
+in `test_data_save`:
+
+```python
+ if data_form == "lazydata":
+ result_data, result_writer = saver._delayed_writes[0]
+ assert result_data is data
+ # What kind of handle it is is the dataset's business, and is
+ # tested in test_NetCDFDataset__write.py.
+ assert result_writer is mocker.sentinel.write_handle
+ elif data_form == "realdata":
+ cf_var.__setitem__.assert_called_once_with(slice(None), data)
+ else:
+ assert data_form == "emulateddata"
+ assert cf_var.emulated_data_array is data
+```
+
+Note the last line: the original wrote `cf_var._data_array == ...` with no
+`assert`, so it asserted nothing at all.
+
+- [ ] **Step 12: Run the netCDF suites to verify they pass**
+
+Run:
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/ \
+ lib/iris/tests/unit/fileformats/cf/ \
+ lib/iris/tests/integration/netcdf/
+```
+Expected: PASS — `0 failed, 23 skipped, 0 errors`. That is 1 + 1 + 21 from
+§6.2's per-selection table; run serially like this, F14 does not fire, so
+there is nothing to excuse here. A new skip is a test that stopped running,
+not a test that passed.
+
+`test_Saver__user_dataset.py` passing here is the point of Step 1: the
+emulated-dataset path went through `NetCDFDataset.from_existing`,
+`create_dimension`, `create_variable`, `_NetCDFAttributes.__setitem__` and
+`emulated_data_array` on this run, and none of them existed when it was
+written.
+
+- [ ] **Step 13: Green check and commit**
+
+```bash
+pytest -n auto lib/iris/tests/unit/ lib/iris/tests/integration/
+pre-commit run --files lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__lazy_stream_data.py
+git add lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/
+git commit -m "$(cat <<'EOF'
+Save through CFDataset rather than through netCDF4
+
+Saver now owns a NetCDFDataset. Dimensions, variables, attributes and
+data writes all go through the interface, so the saver no longer knows
+what kind of file it is writing.
+
+Three netCDF-only reads remain, all through the explicit
+'self._dataset.dataset' escape hatch: the file-format check that decides
+whether int64 is available, and the third-party cf_patch hook, which is
+documented as receiving a netCDF4 dataset. Grep for it to find them.
+
+The grid-mapping variable keeps a netCDF4 handle too. Its sixty-three
+parameter assignments bypass the ASCII-to-bytes coercion every other
+attribute goes through, so converting them would change the file - a
+separate change, recorded as finding F8.
+
+One deliberate behaviour change: a coordinate attribute whose name
+collides with a netCDF4 Python member - 'shape', 'size', 'dtype' and the
+rest - used to be dropped silently, because the "don't clobber" check
+asked hasattr(). It asks the attribute mapping now, and the attribute
+reaches the file.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 13: Spec updates, the full-suite diff, and the pull request
+
+The interface in `cf/dataset.py` is now the real one, and the sketch in spec
+§4.2 is a year-zero guess that six findings have refined. They have to agree,
+because PRs 3–6 are written against the spec. Then the claim this whole plan
+rests on — "no behaviour change other than the two named ones" — gets
+demonstrated rather than asserted, and the pull request goes up.
+
+**Files:**
+- Modify: `docs/superpowers/specs/2026-09-21-zarr-io-design.md` (§4.2, §4.3, §12.1, §12.3, §12.4, §12.6)
+- Test: none. This task changes no code.
+
+**Interfaces:**
+- Consumes: the interface as built in Tasks 1–4, and findings F1–F13.
+- Produces: a spec whose §4.2 matches `cf/dataset.py`, and an open pull
+ request against `SciTools/iris:brownfield`.
+
+- [ ] **Step 1: Bring spec §4.2 into line with the code**
+
+In `docs/superpowers/specs/2026-09-21-zarr-io-design.md`, replace the code
+block in §4.2 with what Tasks 1–4 actually built:
+
+```python
+class CFDatasetVariable(ABC):
+ """One named array in a CF-conforming dataset."""
+ name: str
+ location: str # the dataset's path or URL; F2
+ dimensions: tuple[str, ...]
+ shape: tuple[int, ...]
+ dtype: np.dtype
+ size: int
+ fill_value: Any | None
+ chunking: tuple[int, ...] | None # None when the store is unchunked
+ attributes: MutableMapping[str, Any] # materialised once
+
+ def __getitem__(self, keys) -> np.ndarray: ...
+ def __setitem__(self, keys, values) -> None: ...
+ def __len__(self) -> int: ... # defaults to self.shape[0]; F5
+
+ def write_handle(self) -> Any: ... # picklable __setitem__ target; see 4.5
+ def deprecated_netcdf_member(self, name: str) -> Any: ... # see 4.3
+
+
+class CFDataset(ABC):
+ """A CF-conforming array store, open for reading or writing."""
+ location: str # path or URL, for messages and proxies
+ mode: str # "r" | "r+" | "a" | "w" | "w-"; 4.5
+ closed: bool # F3
+ variables: Mapping[str, CFDatasetVariable]
+ dimensions: Mapping[str, int]
+ attributes: MutableMapping[str, Any]
+
+ def create_dimension(self, name: str, size: int | None) -> None: ...
+ def create_variable(self, name, dtype, dimensions=(), *,
+ fill_value=None, **encoding) -> CFDatasetVariable: ...
+ def sync(self) -> None: ...
+ def finalise(self) -> None: ... # one-shot; NOT part of close()
+ def close(self) -> None: ...
+```
+
+and replace the paragraph beginning "Three members exist only to keep
+multi-process writing reachable later" with:
+
+```markdown
+Three members exist only to keep multi-process writing reachable later, and
+are explained in §4.5: `write_handle`, `mode` and `finalise`. They cost
+almost nothing now — `mode` is a string the implementations already track
+internally, `write_handle` returns `self` for Zarr, and `finalise` is a no-op
+for netCDF — and their absence is what would force a breaking interface
+change later. `finalise` is defined but not yet called: PR 2 left
+`Saver.__exit__` alone rather than guess at an ordering only Zarr can
+exercise, and PR 6 wires it.
+
+`attributes` is materialised once and does *not* track reads. Tracking is a
+`CFVariable` concern, because `CFReader` builds a second `CFVariable` over
+the same backing variable when it promotes one, and the two must keep
+independent read sets (§4.3). `CFDatasetVariable.attributes` is therefore a
+plain mapping, and `CFVariable.attributes` is a tracking view over it.
+
+`location` repeats `CFDataset.location` on the variable so that a variable
+can name its own file without a back-reference to its dataset. It is
+load-bearing: it is the `path` of a read proxy, and so part of the dask
+array cache key, and it is what `LOAD_PROBLEMS.record()` reports.
+
+`closed` is on the interface because `Saver.complete()` has to refuse to run
+until the file is released, and `isopen()` is netCDF vocabulary. A dataset
+the caller opened is closed by the caller, so an implementation answers from
+the store where it can, not only from its own flag.
+
+`create_dimension` accepts `size=None` to request an unlimited dimension; a
+store with no such concept raises. `create_variable`'s `dimensions` defaults
+to `()`, because grid-mapping variables are scalar.
+```
+
+- [ ] **Step 2: Note the `CFVariable` attribute view in spec §4.3**
+
+§4.3 says `CFVariable.__getattr__` resolves "against `self.attributes`" but
+does not say what `self.attributes` *is*. Add, at the end of §4.3:
+
+```markdown
+`CFVariable.attributes` is a `TrackedAttributes` view, constructed per
+`CFVariable` over a snapshot of `cf_data.attributes`. It records reads —
+which is how `cf_attrs_unused()` decides what reaches `cube.attributes` —
+and it is a snapshot rather than a live view because a backend's attribute
+mapping may write through to the file, and loading must never write.
+```
+
+- [ ] **Step 3: Record the deferred and the deliberate in §12.3 and §12.4**
+
+Add to §12.3, the open questions table:
+
+```markdown
+| Q8 | `_add_grid_mapping_to_dataset` sets sixty-three CF grid-mapping parameters by Python attribute assignment, bypassing the ASCII-to-bytes coercion every other saved attribute goes through — so `crs_wkt` is written as `NC_STRING` while `grid_mapping_name` is `NC_CHAR`. Should they be regularised? | Not in this programme. PR 2 kept them on a named netCDF4 handle (`grid_variable`) rather than change what lands in the file. Regularising is a one-line-per-parameter change with a real CDL diff, and belongs in its own pull request | §4.5 | Nothing |
+```
+
+Add to §12.4, the decision log. The five bullets below were the ones known
+when this plan was written; Tasks 6 and 10 to 12 added eight more, so the
+entry as landed has **thirteen**, and §12.6's count in Step 4 moves with
+it. The added eight are: the five declared properties on `CFVariable`
+(T6-5), the `ncattrs`/`getncattr` leniency (T6-1), `AttributeError` →
+`KeyError` on a missing CF attribute, the synthesised `bounds` link
+reaching `IRIS_RAW` (T10-3), a borrowed bare dataset now wrapped in an
+`EncodedDataset` (T11-6), the three members an emulator must now provide
+(T12-4, T12-7), `cf_patch` keeping netCDF4 objects (T12-5), and the two
+test modules added to `.hooks/check_netcdf4_imports.py`'s
+`_PERMITTED_SUFFIXES`.
+
+```markdown
+**2026-09-24 — during PR 2**
+
+- `CFDatasetVariable.attributes` does not track reads; `CFVariable` does.
+ Two `CFVariable`s can share one backing variable, and they must not share
+ a read set.
+- `CFVariable.attributes` is a snapshot. A backend attribute mapping writes
+ through to the file, and a read-mode load must not write.
+- The `spans` gap §5 called a latent bug is unreachable from Iris:
+ `_NCZARR_SCALAR_DIMENSION` only ever appears alone, so the `len == 1`
+ guard it lacks can never fire. PR 2 characterises the behaviour instead of
+ changing it, and the "one behaviour change" §5 allows is spent elsewhere.
+- Saving a coordinate attribute whose name collides with a netCDF4 Python
+ member — `shape`, `size`, `dtype`, `name`, `dimensions`, `mask` — now
+ writes it, where the `hasattr()` "don't clobber" check used to drop it
+ silently. This is the saver-side half of the same defect the `__getattr__`
+ rewrite fixes on the load side.
+- `finalise()` is on the interface but unwired until PR 6.
+```
+
+- [ ] **Step 4: Update §12.1 and §12.6**
+
+In §12.1, the pull request table, replace the PR 2 row (leaving the number
+blank until Step 8 has it):
+
+```markdown
+| 2 | `CFDataset`, and the CF variable classes rewritten against it | Drafted | — |
+```
+
+In §12.6, the document history table, append:
+
+```markdown
+| 2026-09-24 | PR 2 built. §4.2 reconciled with the implemented interface: `location`, `__len__` and `ndim` on the variable, `closed` on the dataset, `attributes` no longer tracking, `create_dimension(size=None)` and `create_variable(dimensions=())`. §4.3 says what `CFVariable.attributes` is. Q8 opened on the grid-mapping assignments; thirteen decisions logged. |
+```
+
+- [ ] **Step 5: Commit the spec**
+
+```bash
+pre-commit run --files docs/superpowers/specs/2026-09-21-zarr-io-design.md
+git add docs/superpowers/specs/2026-09-21-zarr-io-design.md
+git commit -m "$(cat <<'EOF'
+Reconcile the Zarr I/O spec with the interface PR 2 built
+
+Six things the spec's §4.2 sketch did not settle, which the
+implementation had to. None is a correction: §4.2 called its own listing
+"deliberately small: only what the CF variable classes actually need",
+and this is what they turned out to need.
+
+PRs 3 to 6 are written against §4.2, so it has to be the real interface.
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+- [ ] **Step 6: Diff the full suite against the baseline**
+
+The plan's central claim is that nothing changes except the documented
+behaviours — ten of them, not two; see the Definition of Done.
+Demonstrate it.
+
+```bash
+pytest -n auto lib/iris/tests -q > ~/pr2-suite-final.txt 2>&1
+sed -i 's/\x1b\[[0-9;]*m//g' ~/pr2-suite-final.txt; tail -3 ~/pr2-suite-final.txt
+diff <(grep -E "^(FAILED|ERROR)" ~/pr2-suite-baseline.txt | sed 's/ - .*//' | sort) \
+ <(grep -E "^(FAILED|ERROR)" ~/pr2-suite-final.txt | sed 's/ - .*//' | sort)
+grep -oE "[0-9]+ (passed|failed|skipped|xfailed|xpassed|error)" \
+ ~/pr2-suite-baseline.txt ~/pr2-suite-final.txt | tail -20
+```
+
+Expected: the `diff` is empty, and the counts differ only by the tests this
+pull request adds. The baseline here is `11724 passed, 66 skipped, 0 failed,
+0 errors` (§6.2), so the `diff` being empty means **both sides have no
+`FAILED` or `ERROR` lines at all** — which also means an empty `diff` proves
+nothing on its own if the colour codes are still in the file, since
+`grep -E "^(FAILED|ERROR)"` would match nothing either way. That is what the
+`sed` on line 2 is for.
+
+So the real check is the last command. **Read the skip count**, not only the
+failures: a test that stopped running is neither a pass nor a failure, so a
+`FAILED|ERROR` diff cannot see it. `66 skipped` must still read `66
+skipped` — a rise means coverage was lost, almost always a missing
+`OVERRIDE_TEST_DATA_REPOSITORY`, which must be set for both runs or neither.
+
+`~/pr2-suite-baseline.txt` is the run recorded in section 6, on the baseline
+commit, before Task 1. **It does exist** — Task 13 confirmed both baseline
+files by md5 and compared against them, and the human partner has stated
+the §6.2 baseline is not to be re-measured. Do not run the block below: a
+re-measured baseline cannot detect the behaviour change it exists to
+detect, and a stash-and-checkout dance across a thirteen-commit branch is
+not a step to improvise. If a baseline file is ever genuinely missing, stop
+and report it rather than regenerating it. Kept only as the record of how
+it was made:
+
+```bash
+git stash --include-untracked
+git checkout 2cc0b01d4
+pytest -n auto lib/iris/tests -q > ~/pr2-suite-baseline.txt 2>&1
+sed -i 's/\x1b\[[0-9;]*m//g' ~/pr2-suite-baseline.txt
+git checkout zarr-io-pr-2
+git stash pop
+tail -1 ~/pr2-suite-baseline.txt # must be 0 failed, 0 errors; if not, see §6.1
+```
+
+- [ ] **Step 7: Confirm the boundary held**
+
+All four checks as first written were wrong, and Task 13 replaced them.
+Each correction is explained under the block.
+
+```bash
+# 1. cf/dataset.py imports no backend, at module level or inside a function.
+grep -nE "^\s*(from|import)\s" lib/iris/fileformats/cf/dataset.py
+
+# 2. ...and names no backend in executable code; comments and docstrings
+# are prose about the implementations, and are not a breach.
+python -c "
+import re, tokenize
+hits = []
+with tokenize.open('lib/iris/fileformats/cf/dataset.py') as f:
+ for tok in tokenize.generate_tokens(f.readline):
+ if tok.type in (tokenize.COMMENT, tokenize.STRING):
+ continue
+ if re.search(r'netcdf|zarr|h5|dask', tok.string, re.I):
+ hits.append((tok.start[0], tok.string))
+print(hits or 'clean')
+"
+
+# 3. Every remaining netCDF name in the CF package is one of four known
+# kinds. Classify each line; do not expect no output.
+grep -rn "netcdf" lib/iris/fileformats/cf/*.py | grep -v "^lib/iris/fileformats/cf/_reader.py"
+
+# 4. saver.py assumes netCDF in exactly the places Task 12 named.
+grep -n "_dataset\.dataset\|cf_var_grid\.\|grid_variable\." \
+ lib/iris/fileformats/netcdf/saver.py | wc -l
+
+# 5. Nothing in Iris takes the deprecated netCDF4 fallback.
+pytest -W "error:Reaching netCDF variable member" \
+ lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/netcdf/ -q
+```
+
+Expected: five import lines from the first — `abc`, `collections.abc`,
+`types`, `typing`, `numpy`, and nothing else; exactly one hit from the
+second, `deprecated_netcdf_member`, which spec §4.2 mandates; a
+classifiable line for every hit of the third; a count of **68** from the
+fourth — three `_dataset.dataset`, two `cf_var_grid.` and sixty-three
+`grid_variable.`; and a clean pass from the fifth.
+
+Why each changed:
+
+- **Checks 1 and 2 replace `! grep -niE "netcdf|zarr|h5|dask" …`, which can
+ never print `clean`.** `cf/dataset.py` has matched on a dozen lines since
+ Task 1 built it, and every one is correct: the module docstring explains
+ what the netCDF and Zarr implementations do with `**encoding`, comments
+ name the concrete case a member exists for, and `deprecated_netcdf_member`
+ is a mandated ABC member. The claim worth checking is about imports and
+ executable code, not about a word appearing in prose.
+- **Check 3 was "expect no output", which it can never give either.** Every
+ line is one of four kinds: (A) `dataset.py`'s own prose and its mandated
+ member; (B) the `_deprecated_netcdf_member` reach-through in
+ `_variables.py`, its call site and its docstrings; (C) docstring
+ cross-references to `netcdf._dataset.NetCDFDatasetVariable` and
+ `loader._add_unused_attributes`; (D) prose this PR never touched, in
+ `_variables.py` and `_group.py`. A bare "expect no output" cannot tell a
+ site that was converted from a site that was missed, so make it an
+ allow-list: classify every line, and treat a line fitting none of the four
+ as the finding. The *count* is not a finding — Tasks 11 and 12
+ legitimately add and remove docstring references.
+- **Check 4's expected count was 66.** The arithmetic rested on sixty-one
+ grid-mapping assignments; there are sixty-three (Ruling T12-3), so the
+ total is 68.
+- **Check 5 replaces `-W error::iris._deprecation.IrisDeprecation`, which
+ cannot run at all.** It exits 4 during conftest import, because
+ `lib/iris/tests/graphics/__init__.py` calls `warn_deprecated` for
+ `GraphicsTestMixin` at import time. That file and `lib/iris/_deprecation.py`
+ are byte-identical to the merge base, so it is pre-existing and unrelated
+ to this PR — and it means the check as written was never runnable. The
+ message-targeted filter above is narrower *and* stronger: it turns only
+ this PR's own reach-through warning into an error, so an unrelated
+ pre-existing deprecation cannot mask it. Ruling T11-9 declined to add the
+ filter to `pyproject.toml`, so it is a local check, not a CI gate.
+
+- [ ] **Step 8: Open the pull request**
+
+```bash
+git push -u origin zarr-io-pr-2
+gh pr create \
+ --repo SciTools/iris \
+ --base brownfield \
+ --head trexfeathers:zarr-io-pr-2 \
+ --label "Agentic" \
+ --label "Type: Feature Branch" \
+ --title "Zarr I/O PR 2: CFDataset, and the CF variable classes rewritten against it" \
+ --body "$(cat <<'EOF'
+Part of #6977. Second of seven pull requests into `brownfield`; see
+`docs/superpowers/specs/2026-09-21-zarr-io-design.md` §5 for the programme
+and §12.1 for its state. The implementation plan is in
+`docs/superpowers/plans/2026-09-24-zarr-io-pr2.md`, and ships with it.
+
+## What this does
+
+Introduces `iris.fileformats.cf.dataset`: a pair of narrow abstract classes,
+`CFDataset` and `CFDatasetVariable`, describing what Iris needs from an array
+store. `iris.fileformats.netcdf._dataset` implements them over netCDF4, and
+everything that used to reach through `CFVariable` into a `netCDF4.Variable`
+now goes through the interface instead — the CF reader, the load rules, the
+netCDF loader, and the netCDF saver.
+
+The change that will be most visible to a reader is `CFVariable.__getattr__`.
+It resolved an attribute name against the file *and* against the backing
+netCDF4 object, and cached the answer with `setattr`. It now resolves against
+`self.attributes` only. A CF attribute called `shape`, `dtype` or `name` used
+to be shadowed by the netCDF4 member of that name and silently lost; it
+reaches the cube now.
+
+## Behaviour changes
+
+Ten, all deliberate. Items 1 to 7 and items 9 and 10 change what Iris
+**does**. Item 8 changes what Iris **requires of its inputs**, which is still
+a behaviour change from the caller's side — a save that used to work now
+raises — so it is in the list rather than in a compatibility footnote.
+
+This list has been declared complete three times and grown three times —
+two, then eight, then nine, now ten — each time because a review enumerated
+something the previous pass had walked past, not because the code changed
+underneath it; read "ten" as the number three reviews have reached, not as a
+number anyone has proved.
+
+Each item names the test that pins it. Every one of those tests was checked
+by reverting the behaviour in the source and confirming the test fails; where
+that check found no coverage, the item says so plainly instead of implying
+there is a pin.
+
+1. **A CF attribute whose name collides with a member name is no longer
+ lost.** One headline, two unrelated mechanisms, so they are listed
+ separately.
+
+ - *On save* (finding F13). The "don't clobber" guard in
+ `Saver._set_cf_var_attributes` was `hasattr(cf_var, name)` and is now
+ `name not in cf_var.attributes`. An attribute named `shape`, `size`,
+ `name`, `dtype`, `dimensions` or `mask` is written to the file, where the
+ guard used to see the netCDF4 member of that name and drop the attribute
+ silently.
+ *Pinned by* `lib/iris/tests/integration/netcdf/test_attributes.py::TestAttributesNamedLikeNetcdf4Members::test_attributes_named_after_netcdf4_members_are_saved`.
+ Restoring the `hasattr` form fails it.
+ - *On load*, for names that are **not** declared `CFVariable` properties.
+ `__getattr__` resolves against `self.attributes` instead of falling
+ through to the netCDF4 object, so the file's value wins and is recorded
+ as used rather than being consumed and discarded.
+ *Pinned by* `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py::TestShadowedAttributeNames`,
+ parametrised over `SHADOWED_NAMES = ["filename", "cf_name", "spans",
+ "attributes", "cf_data"]`. Filtering those names out of the attribute
+ mapping fails 15 of the class's 17 tests.
+
+2. **The five declared-property names behave differently again — a different
+ mechanism with the same headline.** `dimensions`, `shape`, `ndim`, `dtype`
+ and `size` are declared properties on `CFVariable` and still resolve to the
+ *storage* object: `cf_var.shape` returns the array shape, not a file
+ attribute called `shape`. What changed is that the read is no longer
+ *recorded*. The old `__getattr__` marked such a name as read before
+ forwarding to the netCDF4 variable, so `_add_unused_attributes` dropped it
+ and a legitimate user attribute vanished from the loaded cube — even though
+ the value returned was never the file's. A declared property short-circuits
+ `__getattr__` entirely, so nothing is recorded and the attribute survives
+ onto the cube. This is emphatically *not* "the shadowing was removed": the
+ member still wins.
+ *Pinned by* `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py::TestTypedProperties::test_a_file_attribute_colliding_with_a_typed_property`,
+ parametrised over all five. Making each property record its own name fails
+ all five parametrisations.
+
+3. **`CFVariable.attributes` is a snapshot taken at construction**, so nothing
+ on the load path can write to the file it is reading.
+ *Pinned by* `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py::TestAttributeAccess::test_attributes_are_a_snapshot_taken_at_construction`,
+ with `lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py::TestMutation`
+ underneath it for the write-through semantics that make the snapshot
+ necessary. Wrapping the backend mapping live instead of copying it fails
+ the first; making `TrackedAttributes.__setitem__` not write through fails
+ the second.
+
+4. **A malformed file that used to raise now loads.** Where `ncattrs()` lists
+ a name that `getncattr()` then refuses — which the netCDF4 library does for
+ some malformed files — `_NetCDFAttributes` substitutes `""` and the load
+ succeeds. `cf/_reader.py`'s old `_getncattr` already tolerated it with the
+ same default; the leniency now lives one layer down, so the same malformed
+ file gets one answer rather than two different ones depending on which
+ layer read it.
+ *Pinned by* `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py::TestAttributes::test_unreadable_attribute_becomes_empty_string`.
+ Removing the `except AttributeError` fails it.
+
+5. **The exception type on a missing CF attribute moved `AttributeError` →
+ `KeyError`.** Call sites that read a CF attribute by attribute access
+ (`cf_var.long_name`) now subscript the mapping
+ (`cf_var.attributes["long_name"]`), and a miss raises `KeyError` where it
+ used to raise `AttributeError`. Minor, but downstream code catching
+ `AttributeError` around a CF read will see a different exception. There
+ are six such sites — four on the load path, and two on the **save** path,
+ which is worth stating separately because a reader of this list would not
+ think to look there:
+
+ - *Load.* `cf/_reader.py:493-494` reads `standard_name` and then
+ `long_name`; with `FUTURE.derived_bounds` off — the default — **both**
+ subscripts are unguarded, and with it on, `standard_name` is guarded by
+ an `in` test three lines above while `long_name` is not.
+ `_nc_load_rules/helpers.py:1158` reads `grid_mapping_name`.
+ `netcdf/ugrid_load.py:380` reads `node_coordinates`, and `:406` reads
+ the connectivity's `cf_role`, inside an `assert`.
+ - *Save.* `netcdf/saver.py:1159` reads `standard_name` and `:1187` reads
+ `coordinates`, both while rewriting a dimensionless vertical
+ coordinate's `formula_terms`.
+
+ A seventh converted read, `_nc_load_rules/helpers.py:567`, is deliberately
+ not in that list: its `contextlib.suppress` was changed from
+ `AttributeError` to `KeyError` along with the read, which is exactly the
+ handling the six above did not get.
+
+ *Pinned by* `lib/iris/tests/unit/fileformats/cf/test_CFReader.py::Test_translate__formula_terms_derived_bounds::test_raises_when_standard_name_empty_and_long_name_absent`
+ — **for the `cf/_reader.py` site only**, and there for its `long_name`
+ subscript. The other five have no test that fails if reverted. That is
+ measured, not assumed, and measured site by site rather than extrapolated:
+ softening each subscript to `.get()` and running the whole of
+ `lib/iris/tests/unit/fileformats/` and `lib/iris/tests/integration/` left
+ the suite green every time — same collected total as the unmutated
+ control, no new failure at any of the five.
+
+ All five are malformed-input paths — a UGRID mesh variable with no
+ `node_coordinates` or no connectivity attribute, a grid-mapping variable
+ with no `grid_mapping_name`, a `formula_terms` variable with no
+ `standard_name`, a cube variable with no `coordinates` — and none is
+ reachable from a well-formed file. **They ship flagged rather than
+ fixed.** Closing them means writing new tests for five malformed-input
+ paths into the final commit of a branch whose whole claim is that it
+ changed nothing; disclosing them here is the better trade. A follow-up
+ issue covering all six sites — settling once whether a missing CF
+ attribute should raise, and which exception — would sit naturally with the
+ merge-back changelog; this pull request proposes one rather than opening
+ it.
+
+6. **`IRIS_RAW` differs on the error-recovery path.** `CFReader` now writes
+ its synthesised and invalidated `bounds` links into the attributes mapping
+ rather than onto the `CFVariable` object, and `build_raw_cube` reads that
+ mapping unfiltered. A formula-term or derived-bounds variable that falls
+ back to `build_raw_cube` gains a `bounds` key it did not have — and where
+ the file carried its own `bounds`, the invalidation writes `None` over it,
+ so `IRIS_RAW` shows `None` where it used to show the file's value. Narrow,
+ but real.
+
+ *The write is pinned; the behaviour change is not.* Deleting the
+ invalidating write fails
+ `lib/iris/tests/integration/netcdf/derived_bounds/test_bounds_files.py::test_load_legacy_hh[with_db]`
+ and `lib/iris/tests/unit/fileformats/cf/test_CFReader.py::Test_translate__formula_terms_derived_bounds::test_promotes_non_formula_root_bounds_to_data`.
+ But replacing it with the pre-PR form — `root_var.bounds = None`, a plain
+ Python attribute that never reaches the mapping and so never reaches
+ `IRIS_RAW` — leaves **every** test green, including
+ `test_CFReader.py::TestSynthesisedBoundsLink`, whose own docstring is
+ explicit that it characterises `TrackedAttributes` rather than exercising
+ `_reader.py`. So this item joins item 5 as declared-but-unpinned, and for
+ the same reason it is flagged rather than fixed here.
+
+7. **A borrowed *bare* `netCDF4.Dataset` now decodes character data.**
+ `CFReader` and `iris.load` both accept an already-open dataset. Where that
+ object is **not** already one of Iris's wrappers — what the ncdata /
+ Xarray bridge passes — `NetCDFDataset.from_existing` now wraps it in an
+ `EncodedDataset`, where `CFReader` previously stored it untouched. Checked
+ against a real pre-branch checkout: `cf_group["labels"][:]` goes
+ `|S1 (3, 4)` → `<U4 (3,)`, and `iris.load(bare_dataset)` on a file whose
+ `coordinates` names a `char(x, strlen)` variable goes `aux_coords=[]` →
+ `[('labels', '<U4')]`. So a char auxiliary coordinate that used to be
+ dropped now loads, and a bare borrow behaves like every other input. Two
+ qualifications: the wrapping is unconditional and does not consult
+ `DECODE_TO_STRINGS_ON_READ` (only `__init__` does), so the read-time
+ opt-out cannot be exercised on a borrowed dataset; and `NetCDFDataset.mode`
+ reports the stipulated `"r+"` for a read-only borrow, which is cosmetic —
+ nothing in the library reads it but `__repr__`.
+ *Pinned by* `lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py::TestABorrowedBareDataset::test_char_data_is_decoded`.
+ Removing the wrap fails it.
+
+8. **Saving into an *emulated* netCDF4 dataset now requires three members it
+ did not before.** This lands on emulators — ncdata and the Xarray bridge
+ (#4994) — not on real netCDF4 datasets. The module docstring of the test
+ that covers it says it best:
+
+ > Saving through a ``CFDataset`` asks an emulating object for three members
+ > that the saver never used to reach for. All three are public netCDF4 API,
+ > and all three are now required of an emulator:
+ >
+ > * ``Dimension.size`` - the CF layer reads dimension lengths through it.
+ > ``__len__`` is not an alternative, as ``_EmulatedDimension`` explains.
+ > * ``Dataset.ncattrs()`` - read once, as the dataset is wrapped.
+ > * ``Variable.ncattrs()`` - read once per variable, as each is wrapped.
+ >
+ > They are marked below where the emulators define them. This is an
+ > accepted consequence of the change, not an oversight: there is
+ > deliberately no fallback for an emulator that lacks them.
+
+ And, at the `_EmulatedDimension` definition site, on why `len()` will not
+ serve:
+
+ > netCDF4.Dimension.size, which is how the length is read back. len() is
+ > not an alternative: the emulator is put inside a _thread_safe_nc wrapper,
+ > whose __getattr__ forwards named members but is never consulted for the
+ > len() protocol.
+
+ The absence of a shim is deliberate: a speculative
+ `getattr(dim, "size", None)` fallback was declined as defensive wrapping
+ for an unconfirmed problem, which `lib/iris/AGENTS.md` bans.
+ *Pinned by* the emulator doubles and the saves through them in
+ `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py`.
+ Deleting any one of the three members makes all five emulator saves error
+ — confirmed for each of the three separately. This is the only one of the
+ ten whose pin is a **test double** rather than a behavioural assertion,
+ which is appropriate: the thing being pinned is an interface demand on
+ third-party code, and the double is Iris's only standing description of
+ what that code must provide. It is *not* a live ncdata test.
+
+ **A question for a maintainer before this lands:** none of this is
+ verified against real ncdata. ncdata is not installed in the development
+ environment used here, and installing it needs your say-so. Is an ncdata
+ round-trip check wanted first? If ncdata turns out to lack any of the
+ three members, the fix is one `getattr` in the dataset layer, not a
+ redesign.
+
+9. **Reaching a netCDF4 member through a `CFVariable` now warns.** The old
+ `__getattr__` answered any name the backing netCDF4 object carried —
+ `value = getattr(self.cf_data, name)`, silently. The route survives for
+ one deprecation cycle, but it now goes through
+ `NetCDFDatasetVariable.deprecated_netcdf_member`, which issues an
+ `IrisDeprecation` naming the member and pointing at
+ `cf_var.cf_data.variable`. `CFVariable` is exported in
+ `iris.fileformats.cf.__all__`, so this reaches callers outside Iris:
+ third-party code that read `cf_var.getncattr` or `cf_var.group` gets a
+ warning where it got silence, and under `-W error` a read that worked
+ before now raises. A `hasattr()` probe that comes back **True** warns too,
+ because the member was fetched; one that comes back False does not,
+ because nothing was used. Nothing in Iris itself takes the route — see
+ the reach-through run below, which measures that at zero.
+ *Pinned by* `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py::TestDeprecatedNetcdfMember::test_a_reach_through_warns`
+ and `…::TestDeprecatedNetcdfMember::test_the_message_names_the_replacement`,
+ both `pytest.warns(IrisDeprecation)`, with
+ `…::TestDeprecatedNetcdfMember::test_a_missing_name_raises_and_does_not_warn`
+ holding the probe boundary. Deleting the `warn_deprecated` call fails the
+ first two and leaves the third passing.
+
+10. **A dataset the caller hands to `iris.save` is now mutated.**
+ `NetCDFDataset.from_existing` calls `set_auto_chartostring(False)` on the
+ object it borrows, so saving into a caller's already-open dataset turns
+ auto-chartostring off on it, permanently, as a side effect of the save.
+ The merge base's **save** path never called it. Two qualifications, both
+ narrowing: the **load** path already did exactly this before this branch
+ (`cf/_reader.py` applied the same call to datasets it borrowed), so the
+ change is to the save path alone; and it is inert for anything
+ `from_existing` has to wrap — a bare `netCDF4.Dataset`, or an emulator —
+ because the wrapping `EncodedDataset` does its own decoding and swallows
+ the call rather than forwarding it. What is left is a save into an
+ object that is already an Iris thread-safe wrapper, where the call
+ reaches the real netCDF4 dataset underneath. Iris's saver reads no
+ character data, so nothing here does anything with the setting; it is
+ listed because a permanent change to an object the caller still owns is
+ caller-visible whether or not it matters.
+ *Pinned by* `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py::TestAutoChartostring::test_turned_off_on_a_borrowed_dataset`,
+ which spies on the class rather than the instance because
+ `DatasetWrapper.__setattr__` forwards instance-level patching to the
+ contained object. Removing the call fails it.
+
+An eleventh that is *not* a change: spec §5 allowed one behaviour fix, to
+`CFVariable.spans`. Reading the code showed the gap is unreachable —
+`_NCZARR_SCALAR_DIMENSION` only ever appears as a variable's sole dimension —
+so this pull request characterises the current behaviour rather than altering
+it, and spends the allowance on (1) instead.
+
+## Five things a reviewer will want stated
+
+**`cf_patch` still receives netCDF4 objects.** The public
+`iris.site_configuration["cf_patch"]` hook has been documented since Iris 1.3
+as being handed netCDF4 objects. Both arguments — the dataset and the
+variable — are unwrapped before the call, so a `CFVariable` never reaches it.
+An interim version of this branch passed a `NetCDFDatasetVariable`, which
+defines no `__setattr__` and so would have swallowed `variable.name = value`
+silently; that was caught and fixed, and is now pinned by
+`lib/iris/tests/integration/netcdf/test_attributes.py::TestCfPatch::test_cf_patch_receives_a_netcdf4_variable`.
+
+**`.hooks/check_netcdf4_imports.py` gained two entries.** The repository
+forbids `import netCDF4` outside `_thread_safe_nc`;
+`lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py` and
+`lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py` are added to
+its `_PERMITTED_SUFFIXES` because they must construct a real, bare
+`netCDF4.Dataset` to test the wrapping path that exists for exactly such an
+object. This widens a lint allow-list for two test modules; it is not a
+behaviour change.
+
+**Where a file attribute and an undeclared netCDF4 member share a name, the
+file wins now.** That is the load half of behaviour change 1, stated the
+other way round, and it is worth saying explicitly because it inverts a
+precedence. The stronger statement is that nothing inside Iris depends on
+that precedence any more, in either direction: after this branch no reader in
+the library reaches an undeclared netCDF4 member through a `CFVariable` at
+all. That is measured rather than asserted — the reach-through run below
+promotes this branch's own deprecation warning to an error and gets zero
+occurrences. (`ugrid_load.py`'s two error messages did read `cf_var.name`,
+which would have been exactly such a reach; this branch moved them to
+`cf_var.cf_name`, Iris's own name for the variable.) So the inversion is
+visible only to a caller outside Iris, reading a colliding name from a
+`CFVariable` — and only for a file perverse enough to carry a CF attribute
+named after a netCDF4 member.
+
+**Six assertions in
+`lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py`
+changed, and that is not a loosening.** They were only ever reachable through
+`MagicMock` auto-vivification — the double returned a `MagicMock` for any
+attribute, so the assertions described the mock, not a variable. Re-measured
+against a real backing variable across 36 cases, the old and new forms
+diverge in none of them; the new form describes what a real `CFVariable`
+does.
+
+**Eight grid-mapping parameter assignments gained coverage they never had** —
+Mercator's `scale_factor_at_projection_origin` and the seven in the
+PolarStereographic branch, in
+`lib/iris/tests/integration/netcdf/test_attributes.py::TestGridMappingAttributes`.
+These are *tests added*, not behaviour changed: a round-trip against the
+pre-branch tree confirmed the values were always reaching the file.
+
+## How the negative claim was checked
+
+- **Full suite against the merge base.** Baseline (recorded on `2cc0b01d4`
+ before any of this work): `11724 passed, 66 skipped, 6482 warnings`, with
+ zero `FAILED` and zero `ERROR` lines. This branch: `11992 passed, 66
+ skipped, 6483 warnings`, also zero and zero. The normalised diff of the
+ `FAILED`/`ERROR` lines is empty on both sides. **Read nothing into the
+ warning totals.** Five runs of one unchanging tree gave 6483, 6485, 6484,
+ 6483, 6483 — the summary names the same 13 warning sites every time, but
+ import-time module deprecations are counted once per `-n auto` worker that
+ imports the module, so the total moves with how the tests happened to be
+ distributed. An earlier draft of this section explained the 6482 → 6483
+ step as one extra `iris.save`; that was a cause invented for a difference
+ inside run-to-run noise, and it is withdrawn. The pass and skip counts are
+ stable across those runs and are the ones carrying the claim. The skip
+ count is read deliberately and is unchanged — a test that silently stopped
+ running is neither a pass nor a failure, so a failure diff cannot see it.
+- **+268 passed, accounted for exactly.** 200 tests in new modules, plus 67
+ added in modified modules (counting parametrisations), plus 6 from two
+ inherited mixin methods reaching three modules, minus 5 removed. No
+ `parametrize` decorator was removed and no existing parametrisation was
+ widened, so the arithmetic is not hiding a substitution.
+- **The boundary.** `cf/dataset.py` imports only `abc`, `collections.abc`,
+ `types`, `typing` and `numpy`; a token-level scan excluding comments and
+ docstrings finds exactly one backend name in executable code,
+ `deprecated_netcdf_member`, which spec §4.2 mandates. `saver.py` contains
+ no `import netCDF4` and no `_setncattr`, and names netCDF4 in exactly 68
+ places: three `self._dataset.dataset` escape hatches (two `file_format`
+ reads and the `cf_patch` hand-off), two `cf_var_grid.` and sixty-three
+ `grid_variable.` grid-mapping assignments.
+- **The deprecating reach-through fires nowhere in Iris's own paths.**
+ Running `lib/iris/tests/unit/fileformats/` and
+ `lib/iris/tests/integration/netcdf/` with the reach-through warning
+ promoted to an error produces zero occurrences.
+
+## Not here
+
+No changelog fragment: feature-branch pull requests defer those to the
+merge-back, along with the closing keywords. See §5 of the spec. Behaviour
+change 9 — the deprecation on the netCDF4 reach-through — is the kind of
+change a fragment would normally accompany, and a follow-up issue for the
+six `KeyError` sites in behaviour change 5 is proposed above; both belong
+with the merge-back, where the fragment is written.
+
+The sixty-three grid-mapping parameter assignments in `saver.py` keep a
+netCDF4 handle rather than move to `.attributes`, because they bypass the
+ASCII-to-bytes coercion and moving them would change the CDL — `crs_wkt`
+would stop being `NC_STRING`. Recorded as Q8 in the spec.
+
+One unrelated defect was found and is **not** fixed here, and is not a flake.
+Running `lib/iris/tests/unit/fileformats/netcdf/loader/test_load_cubes.py`
+before `lib/iris/tests/integration/netcdf/test_coord_systems.py` in the same
+process errors all ten tests in the latter. `test_coord_systems.py` imports
+the unit module as `tlc`, and its module-scoped autouse fixture only reaches
+its `yield` inside `if not hasattr(tlc, "TMP_DIR")`, while the unit module
+sets that global at import. Unit module first ⇒ the fixture returns without
+yielding ⇒ every test in the module errors. It is deterministic on module
+ordering, not on scheduling. All three files are byte-identical to the merge
+base, so it predates this branch — but it is a latent CI failure under an
+unlucky ordering and is worth an issue of its own.
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+EOF
+)"
+```
+
+- [ ] **Step 9: Record the pull request number**
+
+Put the number and link into §12.1 of the spec, and push:
+
+```bash
+git add docs/superpowers/specs/2026-09-21-zarr-io-design.md
+git commit -m "$(cat <<'EOF'
+Record PR 2 in the Zarr I/O progress table
+
+Part of #6977
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+git push
+```
+
+---
+
+## 6. Verification
+
+The pull request's central claim is a *negative* one: nothing changes except
+the two documented behaviours. A negative claim cannot be shown by a green
+run — only by a run that matches a run taken before any of this existed. So
+the baseline comes first, before Task 1, and "green" below always means
+**matches the baseline**.
+
+On this machine, after F15 was fixed, the baseline happens to *be* green:
+`11724 passed, 66 skipped, 0 failed, 0 errors`. That is a gift, not a
+licence to stop comparing. Two things still separate "green" from
+"unchanged", and both are invisible to a failure count:
+
+- **The skip count.** A test that stops being collected takes its coverage
+ with it and leaves no failure behind. 66 is the number; it must not rise.
+- **The passed count.** It should rise by exactly the number of tests this
+ pull request adds, and §6.4 makes you enumerate them.
+
+So read the whole tail line, not the colour.
+
+### 6.1 Before Task 1: prove the environment, then record the baseline
+
+```bash
+source "$(conda info --base)/etc/profile.d/conda.sh"
+conda activate iris-dev
+echo "${OVERRIDE_TEST_DATA_REPOSITORY:?set OVERRIDE_TEST_DATA_REPOSITORY first}"
+ls "$OVERRIDE_TEST_DATA_REPOSITORY/NetCDF/unstructured_grid/" | wc -l
+```
+
+That last line is not ceremony. **It must print 13. If it prints a number
+under 10, stop and update `iris-test-data` before going any further** —
+
+```bash
+git -C ../iris-test-data fetch upstream # add the remote first if absent:
+ # https://github.com/SciTools/iris-test-data.git
+git -C ../iris-test-data merge --ff-only upstream/master
+```
+
+— because the UGRID files live there, and
+`lib/iris/tests/integration/netcdf/test_ugrid_load.py` is the only end-to-end
+coverage Task 9's `ugrid_load.py` rewrite has. Without them those thirteen
+tests fail on arrival, look like pre-existing noise, and go on failing for
+the same reason after the rewrite has broken something real. That is exactly
+the trap `lib/iris/tests/AGENTS.md` warns about, wearing a failure's clothes
+instead of a skip's. It had already sprung on this machine when the plan was
+written; see finding F15.
+
+Then, on the baseline commit and with the working tree clean:
+
+```bash
+git status --short # must be empty; the plan file itself is untracked
+git rev-parse --short HEAD # must be 2cc0b01d4
+pytest -n auto lib/iris/tests -q > ~/pr2-suite-baseline.txt 2>&1
+sed -i 's/\x1b\[[0-9;]*m//g' ~/pr2-suite-baseline.txt
+tail -1 ~/pr2-suite-baseline.txt
+```
+
+`sed` strips the colour codes, without which `grep -E "^(FAILED|ERROR)"`
+matches nothing and every later diff comes back empty and meaningless
+**[verified: this caught a false clean while writing this plan]**.
+
+Record the last line somewhere you will still have it in a day. Then do the
+same for the tier most tasks below use, which is not a subset of the same
+numbers:
+
+```bash
+pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/ -q \
+ > ~/pr2-tier-baseline.txt 2>&1
+sed -i 's/\x1b\[[0-9;]*m//g' ~/pr2-tier-baseline.txt
+```
+
+### 6.2 What the baseline was on the day this plan was written
+
+Measured here, on `zarr-io-pr-2` at `2cc0b01d4`, four cores, `-n auto`, with
+`iris-test-data` at `f4a6a05` **[verified]**:
+
+| Run | Result |
+|---|---|
+| `lib/iris/tests` | `11724 passed, 66 skipped, 6482 warnings in 247.29s` — **0 failed, 0 errors** |
+| `unit/fileformats/` + `integration/` | `4465 passed, 26 skipped, 2425 warnings, 10 errors in 160.05s` |
+
+And the smaller selections the tasks below actually run, all serial, all
+**[verified]**:
+
+| Selection | Baseline |
+|---|---|
+| `unit/fileformats/cf/` | `306 passed, 1 skipped` |
+| `unit/fileformats/netcdf/` | `374 passed, 1 skipped` |
+| `unit/fileformats/nc_load_rules/` | `332 passed, 0 skipped` |
+| `unit/mesh/` + `integration/mesh/` | `779 passed, 0 skipped` |
+| `integration/netcdf/` | `206 passed, 21 skipped` |
+| `integration/test_cf.py` | `18 passed, 0 skipped` |
+
+Three things to know before you read your own numbers against these.
+
+**The one skip in `unit/fileformats/cf/` and the one in
+`unit/fileformats/netcdf/` are expected, and they are not the same kind.**
+The first is `test_CFReader.py`, *"Test only applicable when FUTURE context
+is enabled"* — unconditional in this configuration, so it will be there for
+you too. The second is `attribute_handlers/test_GribParamHandler.py`,
+*"could not import 'iris_grib'"* — environment-dependent, so if you have
+`iris_grib` installed you will see `375 passed, 0 skipped` instead. Either
+is fine. **A second skip appearing in either directory is not**, and is
+almost always a rename that left a stale import or a module now raising at
+import time and being swallowed.
+
+**The ten errors in the tier run are finding F14** — all in
+`integration/netcdf/test_coord_systems.py`, all
+`ValueError: _setup did not yield a value`, a pre-existing fixture defect
+whose firing depends on whether
+`unit/fileformats/netcdf/loader/test_load_cubes.py` was imported into the
+same process first. (Corrected in Task 13: that is module *import order*, not
+xdist scheduling — it reproduces serially with no xdist at all. Under
+`-n auto` it is which modules share a worker that decides the order.) They do
+not appear in the full-suite run, and they do not appear when
+`integration/netcdf/` is run serially. They are not yours and chasing them
+will cost you an afternoon. Confirm they are in your baseline too, and move
+on.
+
+**Everything else is green, and that is recent.** Before `iris-test-data`
+was brought up to date this table read `27 failed … 11 errors`, all missing
+data files (F15). If your full-suite run is not at `0 failed, 0 errors`,
+your test data is stale — go back to §6.1 rather than treating the red as
+background.
+
+### 6.3 During the work: what each task compares against
+
+Every task's own steps name the tier to run; §5.0 is the ritual around it.
+Two rules govern reading the result.
+
+**Unit directories must be literally clean — 0 failed, and the skip count
+§6.2 records for that exact selection.** `lib/iris/tests/unit/` needs no
+external data, so there is nothing to excuse a failure. The skip count is
+not always zero, though: `unit/fileformats/cf/` carries 1 and
+`unit/fileformats/netcdf/` carries 1, both named in §6.2, and a step below
+that says "0 failed, 1 skipped" means that pre-existing one and no other.
+A *new* skip in a unit directory is a test that stopped being
+collected — almost always a rename that left a stale import, or a module
+that now raises at import time and is being swallowed. Treat it as a
+failure, because that is what it is.
+
+**Anything touching `lib/iris/tests/integration/` is compared, not judged.**
+
+```bash
+pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/ -q \
+ > ~/pr2-tier-now.txt 2>&1
+sed -i 's/\x1b\[[0-9;]*m//g' ~/pr2-tier-now.txt
+diff <(grep -E "^(FAILED|ERROR)" ~/pr2-tier-baseline.txt | sed 's/ - .*//' | sort) \
+ <(grep -E "^(FAILED|ERROR)" ~/pr2-tier-now.txt | sed 's/ - .*//' | sort)
+```
+
+Empty diff, and a skip count no higher than 6.2's, is the pass condition.
+Sorting and dropping the message keeps xdist's nondeterministic ordering and
+varying tracebacks out of the comparison.
+
+### 6.4 At the end: the whole-suite diff
+
+Task 13 Step 6 runs it, and Task 13 Step 7 checks the architectural boundary
+held. The pass condition is the same shape as 6.3's, over
+`lib/iris/tests`, plus: the counts differ only by the tests this pull request
+adds. Those are enumerable, so enumerate them —
+
+```bash
+git diff --stat 2cc0b01d4 -- "lib/iris/tests/**/test_*.py"
+```
+
+— and the passed count should rise from 11724 by the number of test
+functions added, no more and no less, with `66 skipped` and `0 failed,
+0 errors` unchanged. A larger rise means a parametrisation was widened
+somewhere; a smaller one means something stopped being collected.
+
+Note `git diff --stat` counts *files*, not functions, so read the diff
+itself for the new `def test_` lines:
+
+```bash
+git diff 2cc0b01d4 -- "lib/iris/tests/**" | grep -cE "^\+\s*def test_"
+```
+
+### 6.5 The four things a full-suite diff still cannot see
+
+Each has its own check, listed here so none is assumed to be covered by the
+big run:
+
+1. **An attribute that now reaches a cube, or stops reaching one.** Nothing
+ fails; the cube is simply different. Covered by
+ `integration/test_netcdf__loadsaveattrs.py` and `integration/test_cf.py`,
+ which Tasks 6 and 10 run explicitly, and by Review Focus 1 and 3.
+2. **A CDL change in a saved file.** Covered by the `assert_CDL` comparisons
+ in `test_netcdf.py` and `integration/netcdf/`, which Task 12 runs, and
+ deliberately sidestepped for the grid-mapping parameters (F8).
+3. **A deprecation warning Iris trips on its own paths.** Covered by the
+ `-W "error:Reaching netCDF variable member"` run in Task 13 Step 7,
+ which replaced `-W error::iris._deprecation.IrisDeprecation`: that form
+ exits 4 during conftest import on a pre-existing, unrelated deprecation,
+ so it never ran. See Step 7 for the detail.
+4. **The boundary itself** — a netCDF name leaking into `cf/dataset.py`, or a
+ backend reach-through appearing somewhere other than the three named
+ escape hatches. Covered by Task 13 Step 7's greps, which are exact counts,
+ not spot checks.
+
+---
+
+## 7. Definition of done
+
+- [ ] All thirteen tasks complete, each committed separately, no commit made
+ with `--no-verify`.
+- [ ] `lib/iris/fileformats/cf/dataset.py` exists, imports nothing but the
+ standard library and numpy, names no backend in *executable* code
+ except the ABC's own mandated `deprecated_netcdf_member`, and its two
+ ABCs match spec §4.2 exactly. (Corrected in Task 13. The original
+ bullet said the file "contains no reference to netCDF, Zarr, HDF5 or
+ Dask", which has been false since Task 1 built the file this plan
+ specified: the module docstring explains what the netCDF and Zarr
+ implementations do with `**encoding`, several comments name the
+ concrete case a member exists for, and `deprecated_netcdf_member` is
+ required by §4.2. A word-level grep cannot distinguish prose from a
+ boundary breach; the import list and the executable-token check can.)
+- [ ] `lib/iris/fileformats/netcdf/_dataset.py` implements both, and the
+ shared contract test from Task 5 runs against it.
+- [ ] `CFVariable.__getattr__` resolves against `self.attributes` only, with
+ no `setattr` caching, and a miss raises `AttributeError` after a single
+ deprecating reach-through.
+- [ ] `grep -rn "getattr(cf_\|hasattr(cf_" lib/iris/fileformats/cf/
+ lib/iris/fileformats/_nc_load_rules/ lib/iris/fileformats/netcdf/`
+ returns only the sites §1 lists as out of scope.
+- [ ] `saver.py` contains no `import netCDF4`, no `_setncattr`, and exactly
+ three `self._dataset.dataset` escape hatches plus the grid-mapping
+ handle.
+- [ ] The full suite is `0 failed, 0 errors` — as it is at baseline — and
+ still reads `66 skipped`, with the passed count up by exactly the
+ number of tests this pull request adds (§6.4).
+- [ ] The **ten** deliberate behaviour changes are each pinned by a test
+ that fails if reverted. (Corrected in Task 13, counted from the
+ ledger's rulings rather than from this plan, which has been wrong
+ about the number twice: it said two, then seven. The ten are
+ member-name collisions on load and on save (F13), the five declared
+ properties no longer recording a read (T6-5), the `attributes`
+ snapshot, `""` for an unfetchable attribute (T6-1), `AttributeError`
+ → `KeyError` on a missing CF attribute, the synthesised `bounds` link
+ reaching `IRIS_RAW` (T10-3), a borrowed *bare* `netCDF4.Dataset` now
+ decoding character data (T11-6), the three members a netCDF4
+ *emulator* must now provide (T12-4, T12-7), and — added in Task 13's
+ first fix round, an omission from the enumeration rather than a
+ re-opening of the count — the netCDF4 reach-through through a
+ `CFVariable` now emitting an `IrisDeprecation` where it used to be
+ silent, which reaches external callers because `CFVariable` is in
+ `iris.fileformats.cf.__all__`; and — added in the final review's fix
+ round, again an omission from the enumeration rather than a change to
+ the code — `NetCDFDataset.from_existing` turning auto-chartostring off
+ on a dataset the caller handed to `iris.save`, which the merge base's
+ save path never did.)
+- [ ] Two of the ten are only partly pinned, and the pull request body
+ says so rather than implying coverage: `AttributeError` → `KeyError`
+ is pinned at the `cf/_reader.py` site and at none of the other five
+ (corrected in Task 13's first fix round, which found six sites, not
+ three — two of them on the *save* path, in `saver.py` — and measured
+ each of the three new ones separately); and T10-3's *consequence* for
+ `IRIS_RAW` is untested, though the write itself is. All were measured
+ by reversion, not assumed.
+- [ ] All five Review Focus items have a passing test in the task that owns
+ the code.
+- [ ] Spec §4.2, §4.3, §12.1, §12.3, §12.4 and §12.6 updated (Task 13).
+- [ ] No changelog fragment, and the pull request body says so.
+- [ ] Pull request open against `SciTools/iris:brownfield`, labelled `Agentic`
+ and `Type: Feature Branch`, body containing "Part of #6977" and no
+ closing keyword.
+- [ ] This plan is on the branch, with any step the implementation proved
+ wrong corrected in place.
diff --git a/docs/superpowers/plans/2026-10-01-zarr-io-pr3.md b/docs/superpowers/plans/2026-10-01-zarr-io-pr3.md
new file mode 100644
index 0000000000..f00f27ba13
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-01-zarr-io-pr3.md
@@ -0,0 +1,2733 @@
+# Zarr I/O PR 3 — Relocate the CF loader: Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+| | |
+|---|---|
+| **Spec** | [`docs/superpowers/specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md) §4.1, §4.4, §4.8, §5 (PR 3) |
+| **Implements** | Pull request 3 of 7. `Part of #6977` — never a closing keyword. |
+| **Branch** | `zarr-io-pr-3`, targeting `brownfield` |
+| **Baseline** | `26c4a73c8` (identical to the tip of `zarr-io-pr-2`, which is PR [#7303](https://github.com/SciTools/iris/pull/7303)); Iris `3.17.0.dev16`; dask `2026.8.0`; numpy `2.5.2`; `iris-test-data` at `f4a6a05` |
+| **Delivers** | `iris.fileformats.cf.loader`, holding every format-agnostic part of today's `iris.fileformats.netcdf.loader`; a single `read_data()` hook on `CFDatasetVariable` through which a backend supplies arrays; `as_lazy_data(cache_key=)`, which ends `iris._lazy_data`'s import of `iris.fileformats.netcdf` |
+| **Behaviour change** | None intended. Every public name still resolves, the dask cache keys are byte-for-byte the ones computed today, and the whole existing test suite passes unchanged. The only visible difference is a deprecation warning when `CHUNK_CONTROL` or `ChunkControl` is reached through the old module path. |
+| **Public API change** | `iris.fileformats.cf.loader.CHUNK_CONTROL` and `.ChunkControl` are the new homes of the two public chunk-control names. The old names still resolve, to the *same objects*, warning on use. Nothing is removed. |
+
+**Goal:** Move the format-agnostic half of the netCDF loader into `iris.fileformats.cf.loader`, leaving `iris.fileformats.netcdf.loader` holding only netCDF's entry point and warn-on-use aliases, so that PR 4 can add Zarr loading by implementing one method.
+
+**Architecture:** Today `netcdf/loader.py` mixes three jobs: driving the CF rules engine (generic), deciding chunking policy (generic), and building netCDF data proxies (netCDF-only). This plan separates them along one seam — a new abstract `CFDatasetVariable.read_data()` — then physically moves the generic side to `cf/loader.py` with `git mv`, so the diff reads as a move rather than a rewrite. The netCDF-only body lands on `NetCDFDatasetVariable.read_data()` next to the rest of the netCDF storage adapter. `iris._lazy_data.as_lazy_data` stops recognising `NetCDFDataProxy` by type and takes a caller-supplied `cache_key=` instead, which is what lets the two modules stop importing each other.
+
+**Tech Stack:** Python 3.12–3.14, NumPy, Dask, netCDF4 (via `iris.fileformats.netcdf._thread_safe_nc`), pytest + pytest-mock, pre-commit (ruff, numpydoc, mypy, sort-all, and the repository's local hooks), Sphinx.
+
+---
+
+## Global Constraints
+
+Every task's requirements implicitly include this section.
+
+| Constraint | Value, verbatim from its source |
+|---|---|
+| Target branch | `brownfield`, not `main` (spec §5) |
+| PR labels | `Agentic` and `Type: Feature Branch` (spec §5) |
+| Issue reference | `Part of #6977`. **Never** a closing keyword — the issue closes at merge-back (spec §5) |
+| Changelog | No `changelog/` fragment in this PR. Fragments are written once at merge-back; the PR body must say so (spec §5) |
+| GitHub posts | Every PR description, comment and review reply opens with `(posted by Claude Opus)`, before anything else. This **replaces** any trailing harness attribution (root `AGENTS.md`) |
+| Commit messages | Out of scope for the above; they keep `Co-Authored-By:` trailers |
+| Deprecation aliases | "The alias is the **same object**, not a copy." "Warnings are emitted on **use**, not on import." Mark `.. deprecated:: 3.17` (spec §4.8) |
+| Deprecation mechanism | `iris._deprecation.warn_deprecated()`, NEP29 schedule (root `AGENTS.md`) |
+| Names that must NOT move | `netcdf.save`, `netcdf.Saver`, `netcdf.load_cubes`, `netcdf.save_mesh`, `netcdf.NetCDFDataProxy`, `netcdf.DEBUG` (spec §4.8, verbatim). Of these, `load_cubes` and `DEBUG` are the two this plan touches; F7 records how `DEBUG` stays put while its only reader moves |
+| Laziness | "there is no `ZarrDataProxy`" (spec §4.4). Nothing in `cf/loader.py` may name a proxy class |
+| Cache key | "The netCDF loader passes `repr(proxy)`, which is byte-for-byte the key computed today, so the change is behaviour-preserving and testable as such" (spec §4.4) |
+| netCDF4 imports | No direct `import netCDF4`; always `iris.fileformats.netcdf._thread_safe_nc` (root `AGENTS.md`, enforced by the `check-netcdf4-imports` hook) |
+| Copyright header | Every new Python file opens with the four-line BSD header (root `AGENTS.md`) |
+| Line length | 88 characters (Ruff default) |
+| Docstrings | NumPy style, strictly validated by the `numpydoc-validation` hook |
+| Public module docstrings | Must open with a `z_reference` item within the first 25 lines (`lib/iris/AGENTS.md`, enforced by `check-docs-page-metadata`) |
+| New module size | Aim under ~1000 lines (`lib/iris/AGENTS.md`) |
+| Published prose | "must never cite an artefact of how the work was done — a review finding, a task or plan number, a phase" (`docs/AGENTS.md`). Cite the issue, the pull request or the design-document section instead |
+| Warnings | Every `UserWarning` subclass inherits from `IrisUserWarning`; the `check-categorised-warnings` hook enforces categorisation |
+| Before finishing | Run the repository's pre-commit hooks and fix everything they report (root `AGENTS.md`) |
+
+### Running anything
+
+The environment is the conda environment `iris-dev`. Either activate it, or prefix
+commands with `conda run -n iris-dev`. Do **not** install or remove packages
+without asking the user first.
+
+```bash
+conda activate iris-dev
+export NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX"
+export OVERRIDE_TEST_DATA_REPOSITORY=/data/home/martin.yeo/repos/iris-test-data/test_data
+```
+
+`pytest-xdist` is installed but is **not** in `addopts`; pass `-n auto` yourself for
+the full-suite runs this plan asks for.
+
+---
+
+## Review Focus
+
+Five things the spec implies, that no existing test exercises, and that would bite
+a real user. Each is pinned by a test in the task named.
+
+1. **A name that users *assign to* silently stops working.** A module-level
+ `__getattr__` forwards reads but cannot see writes — PEP 562 defines no
+ `__setattr__` hook — so forwarding `DEBUG` would leave
+ `iris.fileformats.netcdf.loader.DEBUG = True` succeeding and doing nothing.
+ `_LAZYVAR_MIN_BYTES` has the same shape and eight call sites, three of which
+ assign. → **Task 3, Steps 1–2 and 9**: `test_debug_is_not_relocated` pins
+ that `DEBUG` keeps a real attribute here and is absent from
+ `_RELOCATED_NAMES`, and `test_assignment_to_a_relocated_name_does_not_reach_the_new_home`
+ writes the forward's limitation down. Task 3 Step 11 repoints every
+ `_LAZYVAR_MIN_BYTES` site rather than aliasing it.
+2. **The dask array cache silently stops working.** A cache miss costs only time,
+ so an omitted or altered `cache_key=` would pass every existing test while
+ making every multi-cube file load slower. → **Task 1, Steps 1–4 and 7–9**:
+ `test_key_is_the_caller_s_string` asserts the exact key tuple,
+ `test_second_call_reuses_the_cached_array` asserts a second call does not
+ rebuild, and Step 7's test asserts the netCDF loader still supplies
+ `repr(proxy)`.
+3. **A CF attribute leaks onto the cube.** `read_data` reads `_FillValue`,
+ `scale_factor` and `add_offset` from the storage object's *untracked*
+ attributes, where today they are read from the `CFVariable`'s *tracked* copy.
+ An untracked read that was previously tracked would let a CF-reserved attribute
+ survive onto `cube.attributes`. → **Task 2, Steps 15–16**:
+ `test_reading_data_changes_no_attribute_tracking` compares the names
+ `cf_attrs_unused()` returns before and after a `_get_cf_var_data()` call, for
+ a variable carrying all four reserved names.
+4. **`CHUNK_CONTROL.from_file()` starts rejecting small cubes.** The `KeyError`
+ for an unchunked data variable sits *after* the "small enough to read whole"
+ shortcut. Hoisting the check into the CF layer, which is where the
+ "is this the data variable" fact lives, would make a small contiguous cube
+ raise where today it loads. → **Task 2, Steps 17–18**:
+ `test_from_file_still_loads_a_small_unchunked_cube` loads a sub-5000-byte
+ unchunked cube inside `from_file()`, with the module's threshold-zeroing
+ fixture put back for the duration.
+5. **ncdata's emulated variables and VLEN variables lose their special handling.**
+ Emulated datasets (issue #4994) and netCDF `VLType` variables each take an early
+ exit that never builds a proxy; both are exercised only indirectly today. →
+ **Task 2, Steps 11–14**: `test_emulated_array_is_returned_as_it_stands` and
+ `test_emulated_array_builds_no_proxy` cover emulation, and
+ `test_vltype_warns_that_the_size_is_unknown` covers the VLEN warning that
+ today's `_vl_hint` tests never assert.
+
+---
+
+## 1. Scope boundary
+
+**In scope** (spec §5, PR 3):
+
+- `git mv netcdf/loader.py cf/loader.py`, leaving a `netcdf/loader.py` that holds
+ the netCDF specifics and the deprecating aliases.
+- The `cache_key=` keyword on `as_lazy_data`, and deletion of the
+ `isinstance(data, NetCDFDataProxy)` special case in `iris/_lazy_data.py`.
+- Every in-repository reference repointed: library, tests, benchmarks, user
+ manual.
+- Still netCDF-only behaviour; the whole existing suite passes unchanged.
+
+**Out of scope** — do not do these, even if they look adjacent:
+
+- Zarr. No `iris/fileformats/zarr/` directory, no `zarr` import, no
+ `ZarrDatasetVariable`. That is PR 4.
+- The saver. `netcdf/saver.py` is not moved, not split and not reorganised.
+ That is PR 5. The two deferred imports this plan leaves pointing at it
+ (`_CF_ATTRS` and `Saver._DATALESS_ATTRNAME`) are expected to survive until then.
+- `load_cubes`. It stays in `netcdf/loader.py` as netCDF's entry point, with its
+ mesh imports from `netcdf/ugrid_load.py` (decided 2026-10-01, see §3).
+ Whether a format-agnostic driver is worth extracting is PR 4's question.
+- A changelog fragment. Deferred to merge-back (spec §5).
+- Historical `docs/src/whatsnew/*.rst` entries. 3.4 and 3.8 name
+ `iris.fileformats.netcdf.loader.CHUNK_CONTROL` because that is where it was
+ when they were written. They are a record, not documentation; leave them.
+- Any edit to an `AGENTS.md`. Propose; never self-apply (root `AGENTS.md`).
+
+---
+
+## 2. Git strategy: replaying past PR 2's squash-merge
+
+`zarr-io-pr-3` was branched from `zarr-io-pr-2` and currently sits at exactly its
+tip, so it carries all 72 of PR 2's commits. Once
+[#7303](https://github.com/SciTools/iris/pull/7303) is **squash-merged** into
+`brownfield`, those 72 commits will exist upstream as one commit with a different
+SHA. Left alone, GitHub would show PR 2's entire diff inside PR 3.
+
+Record this SHA now; it is the fence between PR 2's history and PR 3's:
+
+```
+PR2_TIP=26c4a73c8
+```
+
+After #7303 is merged, and **only** then:
+
+```bash
+cd /data/home/martin.yeo/repos/iris
+git fetch upstream
+git rebase --onto upstream/brownfield 26c4a73c8 zarr-io-pr-3
+# Everything before 26c4a73c8 is discarded; PR 3's own commits replay onto the
+# squashed brownfield. Expect no conflicts: the squashed commit has the same
+# tree as 26c4a73c8.
+git push --force-with-lease origin zarr-io-pr-3
+```
+
+Check it worked before opening the PR:
+
+```bash
+git log --oneline upstream/brownfield..zarr-io-pr-3 # PR 3's commits only
+git diff --stat upstream/brownfield...zarr-io-pr-3 # PR 3's files only
+```
+
+If the rebase does conflict, the cause is upstream work landing on `brownfield`
+between #7303 and the rebase, not PR 2 itself. Resolve normally.
+
+If PR 3 is already open at this point, finish by editing its description to
+delete the blockquote warning that #7303's commits are showing through. It has
+just stopped being true, and a stale warning sends reviewers looking for a
+problem that is no longer there.
+
+Do **not** merge `brownfield` into `zarr-io-pr-3` instead. A merge keeps PR 2's
+72 commits reachable from the branch tip, which is the thing this avoids.
+
+Until then, develop on `zarr-io-pr-3` as it stands.
+
+**Do not wait for #7303 to open PR 3.** Its human reviewers are the constraint,
+and the expectation is that PR 3 will be opened, and probably reviewed, while
+#7303 is still outstanding. So the realistic order is: build all six tasks, open
+the pull request, and rebase whenever #7303 eventually lands. Task 6 Step 1 says
+the same thing from the other end.
+
+The cost of opening first is cosmetic but startling, and the PR body warns about
+it: until the rebase happens, GitHub computes PR 3's diff against `brownfield`
+and therefore shows all 72 of PR 2's commits inside PR 3. Nothing is wrong, and
+the rebase removes them in one step. Reviewers should read PR 3's own commits —
+the ones after `26c4a73c8` — and nothing earlier.
+
+---
+
+## 3. Findings that refine the spec
+
+Read these before Task 1. They are what was learned from the code the spec
+describes; where one contradicts the spec, this plan follows the finding and
+Task 5 records it in the spec's decision log.
+
+**F1 — The import `_lazy_data` sheds is one module deeper than the spec says.**
+Spec §5 says PR 3 removes `iris/_lazy_data.py`'s "import of
+`iris.fileformats.netcdf`". The actual statement, deferred inside `as_lazy_data`
+at `lib/iris/_lazy_data.py:302`, is
+`from iris.fileformats.netcdf._thread_safe_nc import NetCDFDataProxy`. Removing
+it has exactly the effect the spec wants: after Task 1, `iris._lazy_data` imports
+nothing from `iris.fileformats`.
+
+**F2 — There are two proxy classes, and today's `isinstance` catches both.**
+`EncodedNetCDFDataProxy` (`netcdf/_bytecoding_datasets.py:462`) subclasses
+`_thread_safe_nc.NetCDFDataProxy`. The loader picks between them per variable.
+Both paths must pass `cache_key=`, or byte-decoded variables silently stop being
+cached.
+
+**F3 — The cache key is built from the *normalised* chunks.** In today's
+`as_lazy_data`, `key = (repr(data), chunks, asarray, meta.dtype, type(meta))` is
+computed *after* `chunks` has been through `_optimum_chunksize`. `cache_key=` must
+be consumed at that same point, not at the top of the function, or the keys change.
+
+**F4 — `_get_cf_var_data` is half generic and half netCDF, and the netCDF half
+has three early exits.** Generic: the 5000-byte threshold, `_get_actual_dtype`,
+and the chunk-control block. netCDF-only: the emulated-dataset shortcut, VLEN
+sizing and its warning, the `_thread_safe_nc.default_fillvals` default, and the
+proxy-class choice. The seam is one new abstract method, `read_data()`, agreed
+with the user on 2026-10-01 in preference to three narrower hooks.
+
+**F5 — `read_data` needs one CF-layer fact, at a point only the backend reaches.**
+The chunk-control block raises `KeyError` for an unchunked variable under
+`from_file()` *only* when the variable is a `CFDataVariable`, and only *after*
+the emulated and small-array early exits have been passed. The dataset variable
+cannot know whether it is a cube's phenomenon variable, and the check cannot be
+hoisted without changing behaviour (Review Focus 4). So `read_data` takes a
+single zero-argument callable, `chunking_policy`, that it invokes only once it
+has decided the result will be lazy. `cf/loader._get_cf_var_data` binds it with
+`partial(_chunks_from_chunk_control, cf_var)`.
+
+**F6 — `netcdf/_dataset.py` cannot import `cf.loader` at module level.**
+`cf/__init__.py` imports `cf/_reader.py`, which imports
+`iris.fileformats.netcdf._dataset`. A module-level import back would leave
+`netcdf._dataset` partially initialised. Use a function-level import inside
+`read_data`, and import the **module**, not its members:
+`from iris.fileformats.cf import loader as cf_loader`. Member imports would
+freeze `_LAZYVAR_MIN_BYTES` at call time and break the eight tests that patch it.
+
+**F7 — `DEBUG` stays put, and the moved code reaches back for it.** Spec §4.8
+lists `netcdf.DEBUG` among the names that do not move, but its only reader —
+`if DEBUG:` in `_load_cube_inner`, a single line — is generic code that does
+move. The obvious resolution, making `cf.loader.DEBUG` the live flag and
+leaving a warn-on-use forward behind, does not work: `DEBUG` is a bool, so the
+*only* way anyone uses it is to rebind it, and PEP 562 gives a module a
+`__getattr__` and no `__setattr__`. `iris.fileformats.netcdf.loader.DEBUG = True`
+would succeed and change nothing. Two test modules already depend on that
+assignment working — `nc_load_rules/actions/__init__.py` patches the name at
+lines 82 and 85. Replacing the module's `__class__` to intercept the write is
+the sort of cleverness `lib/iris/AGENTS.md` rules out.
+
+So `DEBUG` keeps its definition in `netcdf/loader.py`, and `_load_cube_inner`
+reads it through a deferred `from iris.fileformats.netcdf import loader as
+netcdf_loader`, by module attribute rather than by value, so that assignment
+and `mocker.patch` both still bite. `netcdf/__init__.py` needs no change: its
+eager `from .loader import DEBUG, NetCDFDataProxy, load_cubes` is still a
+plain import of three names that are still there, which is also what keeps
+spec §4.8's "warnings on use, not on import" true. Only `CHUNK_CONTROL` and
+`ChunkControl` go through `__getattr__`; they are objects, used through their
+context managers, never rebound.
+
+The direction of the resulting dependency is the opposite of this PR's: a CF
+module reaching into a netCDF one for a debug flag. It is one line, behind a
+deferred import, and PR 4's Zarr loader will honour the same flag — which is
+odd but harmless, and better than a flag that silently ignores being set.
+
+**F8 — Two imports have to point backwards from `cf` to `netcdf` until PR 5.**
+`_add_unused_attributes` uses `_CF_ATTRS` and `_load_cube_inner` uses
+`Saver._DATALESS_ATTRNAME`, both from `netcdf/saver.py`. In `cf/loader.py` these
+must be **function-level** imports: a module-level one risks a partially
+initialised `iris.fileformats.netcdf` whenever `netcdf/__init__.py` is the entry
+point. Each gets a comment naming spec §4.1, which moves the saver to
+`cf/saver.py` and dissolves both.
+
+**F9 — Moving the attribute reads off the tracked copy is safe, and has to be
+shown to be.** Today `_get_cf_var_data` reads `_FillValue`, `scale_factor` and
+`add_offset` through `cf_var.attributes`, a `TrackedAttributes`. After the move
+`read_data` reads them through `NetCDFDatasetVariable.attributes`, which tracks
+nothing. This is behaviour-preserving because all four of `_FillValue`,
+`add_offset`, `missing_value` and `scale_factor` appear both in
+`_variables._CF_ATTRS_IGNORE` — so `TrackedAttributes` pre-seeds them as already
+read — and in `saver._CF_ATTRS`, which `_add_unused_attributes` filters out
+regardless. Two independent reasons, neither obvious; hence Review Focus 3.
+
+**F10 — `cf_var.filename` and `variable.location` are the same string, with one
+fallback that no longer applies.** `CFVariable.__init__` sets
+`self.filename = data.location`, falling back to `"<unknown_filename>"` if that
+raises `AttributeError`. `location` is abstract on `CFDatasetVariable`, so for a
+real dataset it always exists. Mock-based tests must give their fake variable a
+`location`.
+
+**F11 — Collapsing the two `as_lazy_data` calls into one is identical.** The
+`AS_DASK` branch calls `as_lazy_data(proxy, meta=..., chunks="auto")` with no
+`dims_fixed`; the other calls it with `dims_fixed=tuple(...)`. Expressing
+`AS_DASK` as the pair `("auto", None)` and using a single call site is
+byte-identical, because `dims_fixed=None` is the parameter's default.
+
+**F12 — The two size constants part company.** `_MEAN_VL_ARRAY_LEN` exists for
+netCDF `VLType` variables and travels with `read_data` into
+`netcdf/_dataset.py`. `_LAZYVAR_MIN_BYTES` is generic policy and stays in
+`cf/loader.py`, read as `cf_loader._LAZYVAR_MIN_BYTES` so it remains patchable.
+
+**F13 — Task 1 must come first, for an import reason.** `netcdf/_dataset.py`
+needs `from iris._lazy_data import as_lazy_data` at module level, so that
+`test__chunk_control` can patch `iris.fileformats.netcdf._dataset.as_lazy_data`.
+That import is only safe once Task 1 has removed `_lazy_data`'s import of
+`netcdf._thread_safe_nc`.
+
+**F14 — Sphinx will not break, and will not need a new file.** API pages come
+from `sphinxcontrib.apidoc` (`docs/src/conf.py:279-292`), so
+`iris.fileformats.cf.loader` gets a page with no RST written by hand;
+`docs/src/generated/` is gitignored. `nitpicky` is not set in `conf.py`, so the
+two historical whatsnew cross-references to
+`iris.fileformats.netcdf.loader.CHUNK_CONTROL` degrade to plain text rather than
+failing the `-W` build.
+
+**F15 — The references to repoint are these, and no others.** Established by
+`grep -rn` over `lib/`, `docs/` and `benchmarks/` for every top-level name in
+`netcdf/loader.py`.
+
+*Library:* `_nc_load_rules/helpers.py:40` (the import; its call sites at 606,
+1578, 1583, 1691 and 1767 need no change); `netcdf/ugrid_load.py:213,218,270,275,437`;
+`netcdf/_dataset.py` (the deferred import Task 2 adds); `cf/_variables.py:124`,
+`_nc_load_rules/actions.py:655` and `_nc_load_rules/engine.py:18` — three prose
+cross-references.
+
+*Tests, for `_LAZYVAR_MIN_BYTES`:* `integration/test_cube.py:23`,
+`integration/netcdf/test_general.py:42`,
+`integration/netcdf/test_stringdata.py:33,34,36,406`,
+`integration/netcdf/test_delayed_save.py:47,48,50`,
+`integration/netcdf/test__dask_locks.py:130` (a docstring),
+`unit/aux_factory/test_AuxCoordFactory.py:147`,
+`unit/fileformats/nc_load_rules/helpers/test_build_and_add_auxiliary_coordinate.py:213`.
+Three of those *assign* rather than patch, which is why F7 matters.
+
+*Tests, other:* `integration/test_cf.py:14`,
+`unit/fileformats/nc_load_rules/actions/__init__.py:14,22,76,104`, the five
+`nc_load_rules/actions/test__*.py` module docstrings, and the five loader
+unit-test modules Task 4 relocates.
+
+*Benchmarks:* `benchmarks/benchmarks/generate_data/__init__.py:126,131,134`.
+
+*Documentation:* `docs/src/user_manual/explanation/netcdf_io.rst:9,71,76,77,107,123,485`.
+
+Three things found by the same grep must **not** change.
+`netcdf/ugrid_load.py:183` compares against `nc_loader.load_cubes`, which stays
+put. `unit/fileformats/nc_load_rules/actions/__init__.py:82,85` patch
+`netcdf.loader.DEBUG`, which also stays put (F7). And
+`docs/src/whatsnew/3.8.rst:27,105` and `3.9.rst:52` record what shipped in
+those releases under the name it had then (F14).
+
+---
+
+## 4. File Structure
+
+### Created
+
+| File | Responsibility | Size |
+|---|---|---|
+| `lib/iris/fileformats/cf/loader.py` | Everything format-agnostic in today's netCDF loader: the rules-engine driver, cube and aux-factory construction, attribute transfer, the constraint callback, chunk-control policy, and the generic `_get_cf_var_data` entry point. | ~700 lines, nearly all moved verbatim |
+| `lib/iris/tests/unit/fileformats/cf/loader/__init__.py` | Package marker for the relocated unit tests. | ~6 lines |
+| `lib/iris/tests/unit/fileformats/netcdf/loader/test__relocated_names.py` | The deprecation shim: same object, warns on use not on import, and the documented assignment limitation. | ~120 lines |
+
+Every other new test is appended to a file that already exists, so that nothing
+is created in one task only to be moved in the next:
+
+| File | What is added |
+|---|---|
+| `lib/iris/tests/unit/lazy_data/test_as_lazy_data.py` | `Test_as_lazy_data__cache_key`, and the layering assertion. |
+| `lib/iris/tests/unit/fileformats/cf/dataset/contract.py` | Two `read_data` cases, so PR 4's Zarr variable is checked by the same body. |
+| `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py` | `TestReadData`: emulated, VLEN, small-array and lazy paths, and the attribute-tracking equivalence of F9. |
+| `lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py` | `_chunks_from_chunk_control` in each of the three modes, and the small-unchunked-under-`from_file` case. Moves to `cf/loader/` in Task 4. |
+| `lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py` | The VLEN warning that today's `_vl_hint` tests never assert. Moves to `cf/loader/` in Task 4. |
+
+### Modified
+
+| File | Change |
+|---|---|
+| `lib/iris/_lazy_data.py:265-348` | `as_lazy_data` gains keyword-only `cache_key=None`; the `NetCDFDataProxy` import and `isinstance` branch go. |
+| `lib/iris/fileformats/cf/dataset.py:185-201` | `CFDatasetVariable` gains abstract `read_data(chunking_policy)`. |
+| `lib/iris/fileformats/netcdf/_dataset.py` | Gains `_MEAN_VL_ARRAY_LEN`, a module-level `from iris._lazy_data import as_lazy_data`, and `NetCDFDatasetVariable.read_data()`. 528 → ~640 lines. |
+| `lib/iris/fileformats/netcdf/loader.py` | Rewritten at the same path, holding `DEBUG`, `NetCDFDataProxy`, `load_cubes` and the two relocated-name forwards. 995 → ~210 lines. |
+| `lib/iris/fileformats/netcdf/__init__.py` | **No change.** All three names it eagerly imports from `.loader` — `DEBUG`, `NetCDFDataProxy`, `load_cubes` — stayed. Listed here because it is the file a reader will expect to see changed, and Task 3 Step 7 says why it is not. |
+| `lib/iris/fileformats/cf/_variables.py:124`, `_nc_load_rules/actions.py:655`, `_nc_load_rules/engine.py:18` | Three prose cross-references repointed to `iris.fileformats.cf.loader`. |
+| `lib/iris/fileformats/_nc_load_rules/helpers.py:40` | Import repointed to `iris.fileformats.cf.loader`. |
+| `lib/iris/fileformats/netcdf/ugrid_load.py:213,218,270,275,437` | Three CF-layer uses repointed at a new `cf_loader` import; `nc_loader.load_cubes` at 183 left alone. |
+| Eleven test modules | `_LAZYVAR_MIN_BYTES` and the other private names repointed. Listed in full in F15 and in Task 3 Steps 11 and 13. |
+| `benchmarks/benchmarks/generate_data/__init__.py:117-136` | `load_realised()` patches `iris.fileformats.cf.loader._get_cf_var_data`. |
+| `docs/src/user_manual/explanation/netcdf_io.rst` | Seven references repointed, including two doctests. |
+| `docs/superpowers/specs/2026-09-21-zarr-io-design.md` | §12.1 row, §12.4 entry, §12.6 row. |
+
+### Moved, with `git mv`, so the diff reads as a move
+
+| From | To |
+|---|---|
+| `lib/iris/fileformats/netcdf/loader.py` | `lib/iris/fileformats/cf/loader.py` (then the netCDF part is re-created at the old path) |
+| `tests/unit/fileformats/netcdf/loader/test__chunk_control.py` | `tests/unit/fileformats/cf/loader/test__chunk_control.py` |
+| `tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py` | `tests/unit/fileformats/cf/loader/test__get_cf_var_data.py` |
+| `tests/unit/fileformats/netcdf/loader/test__load_aux_factory.py` | `tests/unit/fileformats/cf/loader/test__load_aux_factory.py` |
+| `tests/unit/fileformats/netcdf/loader/test__load_cube.py` | `tests/unit/fileformats/cf/loader/test__load_cube.py` |
+| `tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py` | `tests/unit/fileformats/cf/loader/test__translate_constraints_to_var_callback.py` |
+
+Staying under `tests/unit/fileformats/netcdf/loader/`: `test_load_cubes.py` and
+the `ugrid_load/` subdirectory, because `load_cubes` stays.
+
+---
+
+## 5. The interface, as these tasks leave it
+
+Names and signatures every task must match. Later tasks consume these verbatim.
+
+```python
+# iris/_lazy_data.py
+def as_lazy_data(
+ data, chunks=None, asarray=False, meta=None, dims_fixed=None, *, cache_key=None
+): ...
+
+# iris/fileformats/cf/dataset.py
+class CFDatasetVariable(ABC):
+ @abstractmethod
+ def read_data(self, chunking_policy: Callable[[], tuple]) -> Any: ...
+
+# iris/fileformats/netcdf/_dataset.py
+_MEAN_VL_ARRAY_LEN = 10
+
+class NetCDFDatasetVariable(CFDatasetVariable):
+ def read_data(self, chunking_policy: Callable[[], tuple]) -> Any: ...
+
+# iris/fileformats/cf/loader.py
+_LAZYVAR_MIN_BYTES = 5000
+CHUNK_CONTROL: "ChunkControl"
+
+class ChunkControl(threading.local): ...
+class _OrderedAddableList(list): ...
+class _WarnComboIgnoringBoundsLoad(IrisIgnoringBoundsWarning, IrisLoadWarning): ...
+
+def _get_cf_var_data(cf_var): ...
+def _chunks_from_chunk_control(cf_var) -> tuple: ...
+def _get_actual_dtype(cf_var): ...
+def _set_attributes(attributes, key, value): ...
+def _add_unused_attributes(iris_object, cf_var): ...
+def _actions_engine(): ...
+def _assert_case_specific_facts(engine, cf, cf_group): ...
+def _actions_activation_stats(engine, cf_name): ...
+def _load_cube(engine, cf, cf_var, filename): ...
+def _load_cube_inner(engine, cf, cf_var, filename): ...
+def _load_aux_factory(engine, cube): ...
+def _translate_constraints_to_var_callback(constraints): ...
+
+# iris/fileformats/netcdf/loader.py
+DEBUG: bool = False # stays: users assign to it (F7)
+NetCDFDataProxy = _bytecoding_datasets.EncodedNetCDFDataProxy
+_RELOCATED_NAMES = frozenset({"CHUNK_CONTROL", "ChunkControl"})
+
+def load_cubes(file_sources, callback=None, constraints=None): ...
+def __getattr__(name): ...
+```
+
+`_chunks_from_chunk_control` returns the pair `(chunks, dims_fixed)`, which the
+backend forwards straight into `as_lazy_data`:
+
+- `CHUNK_CONTROL.mode is ChunkControl.Modes.AS_DASK` → `("auto", None)`
+- otherwise → `(list_of_ints, tuple_of_bools_or_(None,))`
+
+---
+
+## Task 1: `as_lazy_data` takes the cache key from its caller
+
+Spec §4.4: "PR 3 replaces it with a caller-supplied `cache_key=` keyword on
+`as_lazy_data`. The netCDF loader passes `repr(proxy)`, which is byte-for-byte
+the key computed today, so the change is behaviour-preserving and testable as
+such. `_lazy_data` then knows about no file format at all."
+
+This task comes first for an import reason: Task 2 needs
+`netcdf/_dataset.py` to import `iris._lazy_data` at module level, and that is
+only safe once `iris._lazy_data` has stopped importing back (F13).
+
+**Files:**
+- Modify: `lib/iris/_lazy_data.py:265-348`
+- Modify: `lib/iris/fileformats/netcdf/loader.py:336` and `:381-387` (the two
+ `as_lazy_data` calls)
+- Test: `lib/iris/tests/unit/lazy_data/test_as_lazy_data.py`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py`
+
+**Interfaces:**
+- Consumes: nothing from earlier tasks.
+- Produces: `as_lazy_data(..., *, cache_key=None)`. When `cache_key` is not
+ `None`, the LRU cache key is
+ `(cache_key, chunks, asarray, meta.dtype, type(meta))`, where `chunks` is the
+ value *after* `_optimum_chunksize` normalisation. When it is `None`, nothing is
+ cached. Passing `cache_key` without `meta` raises `ValueError`.
+
+- [ ] **Step 1: Write the failing test for the key tuple**
+
+Add to `lib/iris/tests/unit/lazy_data/test_as_lazy_data.py`. Put the new class
+after `class Test_as_lazy_data`, and add the imports it needs at the top of the
+file (`from types import SimpleNamespace`, `from iris._lazy_data import CACHE`,
+`from iris.fileformats.netcdf._thread_safe_nc import NetCDFDataProxy`).
+
+```python
+class Test_as_lazy_data__cache_key:
+ @staticmethod
+ def _proxy(shape=(2, 3, 4), dtype="f4"):
+ # NetCDFDataProxy reads only `shape` and `name` from what it is given,
+ # so it can be built without a file as long as it is never indexed.
+ return NetCDFDataProxy(
+ SimpleNamespace(shape=shape, name="air_temperature"),
+ np.dtype(dtype),
+ "/does/not/exist.nc",
+ -1.0,
+ )
+
+ def test_key_is_the_caller_s_string(self):
+ proxy = self._proxy()
+ CACHE._cache.clear()
+ as_lazy_data(proxy, meta=proxy.dask_meta, cache_key=repr(proxy))
+ (key,) = CACHE._cache
+ assert key == (
+ repr(proxy),
+ (2, 3, 4),
+ False,
+ proxy.dask_meta.dtype,
+ type(proxy.dask_meta),
+ )
+
+ def test_no_key_means_no_caching(self):
+ proxy = self._proxy()
+ CACHE._cache.clear()
+ as_lazy_data(proxy, meta=proxy.dask_meta)
+ assert len(CACHE._cache) == 0
+
+ def test_second_call_reuses_the_cached_array(self, mocker):
+ proxy = self._proxy()
+ CACHE._cache.clear()
+ from_array = mocker.patch(
+ "iris._lazy_data.da.from_array", wraps=da.from_array
+ )
+ as_lazy_data(proxy, meta=proxy.dask_meta, cache_key=repr(proxy))
+ as_lazy_data(proxy, meta=proxy.dask_meta, cache_key=repr(proxy))
+ assert from_array.call_count == 1
+
+ def test_key_without_meta_is_refused(self):
+ data = np.arange(24).reshape((2, 3, 4))
+ with pytest.raises(ValueError, match="cache_key"):
+ as_lazy_data(data, cache_key="anything")
+```
+
+- [ ] **Step 2: Run the new tests and watch them fail**
+
+```bash
+pytest lib/iris/tests/unit/lazy_data/test_as_lazy_data.py::Test_as_lazy_data__cache_key -v
+```
+
+Expected: every test fails with
+`TypeError: as_lazy_data() got an unexpected keyword argument 'cache_key'`.
+
+- [ ] **Step 3: Add the keyword and delete the format-specific branch**
+
+In `lib/iris/_lazy_data.py`, change the signature at line 265:
+
+```python
+def as_lazy_data(
+ data, chunks=None, asarray=False, meta=None, dims_fixed=None, *, cache_key=None
+):
+```
+
+Add this parameter to the numpydoc `Parameters` block, after `dims_fixed`:
+
+```
+ cache_key : str, optional
+ If given, a string identifying `data` well enough that two calls
+ presenting the same string, chunking and `meta` may share one Dask
+ array. Callers wrapping a file-backed proxy pass ``repr(proxy)``.
+ Requires `meta`.
+```
+
+Delete the deferred import at line 302:
+
+```python
+ from iris.fileformats.netcdf._thread_safe_nc import NetCDFDataProxy
+```
+
+Add the guard, directly after the existing `meta is None` check:
+
+```python
+ if cache_key is not None and meta is None:
+ msg = "`meta` is required when `cache_key` is given: it forms part of the key."
+ raise ValueError(msg)
+```
+
+Replace the `isinstance` block:
+
+```python
+ # Define a cache key for caching arrays created from data proxies. Creating
+ # new Dask arrays is relatively slow, therefore caching is beneficial if many
+ # cubes in the same file share coordinate arrays. The caller decides what
+ # identifies its data, so this module needs to know no file format.
+ if cache_key is None:
+ key = None
+ else:
+ key = (cache_key, chunks, asarray, meta.dtype, type(meta))
+```
+
+- [ ] **Step 4: Run the new tests and watch them pass**
+
+```bash
+pytest lib/iris/tests/unit/lazy_data/test_as_lazy_data.py -v
+```
+
+Expected: PASS, including the pre-existing tests in the file.
+
+- [ ] **Step 5: Write the failing test that `_lazy_data` has shed the dependency**
+
+Add to the same file, after `Test_as_lazy_data__cache_key`:
+
+```python
+def test_lazy_data_imports_no_file_format():
+ """Pin spec section 4.4: `_lazy_data` knows about no file format at all."""
+ import inspect
+ import re
+
+ import iris._lazy_data
+
+ source = inspect.getsource(iris._lazy_data)
+ offenders = [
+ line
+ for line in source.splitlines()
+ if re.match(r"\s*(from|import)\s", line) and "iris.fileformats" in line
+ ]
+ assert offenders == []
+```
+
+- [ ] **Step 6: Run it**
+
+```bash
+pytest lib/iris/tests/unit/lazy_data/test_as_lazy_data.py::test_lazy_data_imports_no_file_format -v
+```
+
+Expected: PASS, because Step 3 removed the only such line. If it fails, Step 3
+was incomplete — find and remove the remaining import.
+
+- [ ] **Step 7: Write the failing test that the netCDF loader supplies the old key**
+
+Add to `lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py`:
+
+```python
+def test_cache_key_is_the_proxy_repr(mocker):
+ """The key is byte-for-byte the one this module used to compute itself."""
+ as_lazy_data = mocker.patch("iris.fileformats.netcdf.loader.as_lazy_data")
+ cf_var = _make_cf_var(shape=(1000, 1000), dtype=np.dtype("f4"))
+ _get_cf_var_data(cf_var)
+ (proxy,) = as_lazy_data.call_args.args
+ assert as_lazy_data.call_args.kwargs["cache_key"] == repr(proxy)
+```
+
+`_make_cf_var` is the helper this module already uses to build its fake
+variables; read the top of the file and call it the way the neighbouring tests
+do. The shape must be large enough to clear `_LAZYVAR_MIN_BYTES` (5000 bytes),
+so that the lazy branch is taken.
+
+- [ ] **Step 8: Run it and watch it fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py::test_cache_key_is_the_proxy_repr -v
+```
+
+Expected: FAIL with `KeyError: 'cache_key'` — the loader does not pass it yet.
+
+- [ ] **Step 9: Pass the key from both `as_lazy_data` call sites**
+
+In `lib/iris/fileformats/netcdf/loader.py`, the `AS_DASK` call (line 336):
+
+```python
+ result = as_lazy_data(
+ proxy, meta=proxy.dask_meta, chunks="auto", cache_key=repr(proxy)
+ )
+```
+
+and the main call (lines 381-387):
+
+```python
+ result = as_lazy_data(
+ proxy,
+ meta=proxy.dask_meta,
+ chunks=chunks,
+ dims_fixed=tuple(dims_fixed),
+ cache_key=repr(proxy),
+ )
+```
+
+Both proxy classes go through these two sites, so both stay cached (F2).
+
+- [ ] **Step 10: Run the loader and lazy-data unit tests**
+
+```bash
+pytest lib/iris/tests/unit/lazy_data/ \
+ lib/iris/tests/unit/fileformats/netcdf/ -q
+```
+
+Expected: PASS.
+
+- [ ] **Step 11: Run the whole suite, because this touches every lazy array**
+
+```bash
+pytest lib/iris/tests -n auto -q
+```
+
+Expected: the same result as on the baseline commit. If anything fails, compare
+against `git stash && pytest ... && git stash pop` before assuming it is new.
+
+- [ ] **Step 12: Run pre-commit on the touched files**
+
+```bash
+pre-commit run --files lib/iris/_lazy_data.py \
+ lib/iris/fileformats/netcdf/loader.py \
+ lib/iris/tests/unit/lazy_data/test_as_lazy_data.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py
+```
+
+Hooks rewrite files in place; re-stage and re-run until clean.
+
+- [ ] **Step 13: Commit**
+
+```bash
+git add lib/iris/_lazy_data.py \
+ lib/iris/fileformats/netcdf/loader.py \
+ lib/iris/tests/unit/lazy_data/test_as_lazy_data.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py
+git commit -m "$(cat <<'EOF'
+Let as_lazy_data take its cache key from the caller
+
+Replaces the isinstance(data, NetCDFDataProxy) special case with a
+cache_key= keyword. The netCDF loader passes repr(proxy), so the keys
+are byte-for-byte what they were, and iris._lazy_data no longer imports
+anything from iris.fileformats.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+## Task 2: One hook — `CFDatasetVariable.read_data`
+
+The largest task, and the one the rest of the PR exists to enable. It splits
+`_get_cf_var_data` along the only seam that works (F4, F5): the backend builds
+the array, the CF layer supplies the chunking policy. Nothing moves files yet —
+everything stays where it is, so the `git mv` in Task 3 reads as a move.
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/dataset.py` — add abstract `read_data`
+- Modify: `lib/iris/fileformats/netcdf/_dataset.py` — add `_MEAN_VL_ARRAY_LEN`,
+ a module-level `as_lazy_data` import, and `NetCDFDatasetVariable.read_data`
+- Modify: `lib/iris/fileformats/netcdf/loader.py:234-388` — `_get_cf_var_data`
+ becomes a delegation; `_chunks_from_chunk_control` is extracted;
+ `_MEAN_VL_ARRAY_LEN` leaves
+- Test: `lib/iris/tests/unit/fileformats/cf/dataset/contract.py`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py`
+
+**Interfaces:**
+- Consumes: `as_lazy_data(..., *, cache_key=None)` from Task 1.
+- Produces:
+ - `CFDatasetVariable.read_data(chunking_policy)` — abstract. Returns
+ `numpy.ndarray` or `dask.array.Array`. `chunking_policy` is a zero-argument
+ callable returning `(chunks, dims_fixed)`, invoked **only** when the result
+ will be lazy.
+ - `iris.fileformats.netcdf.loader._chunks_from_chunk_control(cf_var)` →
+ `("auto", None)` in `AS_DASK` mode, else `(list[int], tuple[bool] | (None,))`.
+ Raises `KeyError` for an unchunked `CFDataVariable` under `from_file()`.
+ - `iris.fileformats.netcdf.loader._get_cf_var_data(cf_var)` — unchanged
+ signature and unchanged behaviour; a one-line delegation.
+ - `iris.fileformats.netcdf._dataset._MEAN_VL_ARRAY_LEN = 10`.
+
+- [ ] **Step 1: Write the failing contract tests**
+
+In `lib/iris/tests/unit/fileformats/cf/dataset/contract.py`, add two methods to
+`class CFDatasetContract`, after `test_variable_chunking_is_none_or_a_shape`.
+These run against every implementation, so PR 4's Zarr variable inherits them.
+
+```python
+ def test_read_data_returns_the_data(self, variable):
+ result = variable.read_data(lambda: (None, None))
+ np.testing.assert_array_equal(np.asarray(result), CONTRACT_DATA)
+
+ def test_read_data_leaves_the_policy_alone_for_a_small_variable(self, variable):
+ # The chunking policy raises under CHUNK_CONTROL.from_file() for an
+ # unchunked variable. A variable small enough to read whole must reach
+ # its data without ever asking, or `from_file` would start rejecting
+ # small cubes that load today.
+ def policy():
+ raise AssertionError("chunking policy consulted for a small variable")
+
+ variable.read_data(policy)
+```
+
+- [ ] **Step 2: Run them and watch them fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py -v -k read_data
+```
+
+Expected: FAIL with `AttributeError: 'NetCDFDatasetVariable' object has no
+attribute 'read_data'`.
+
+- [ ] **Step 3: Extract the chunking policy, still inside `netcdf/loader.py`**
+
+In `lib/iris/fileformats/netcdf/loader.py`, add this function immediately before
+`_get_cf_var_data`. The body is lifted verbatim from inside `_get_cf_var_data`;
+do not re-word the `KeyError` message, which users match on.
+
+```python
+def _chunks_from_chunk_control(cf_var):
+ """Return the ``(chunks, dims_fixed)`` pair for a lazy array of this variable.
+
+ This is the one piece of a lazy array that is CF's business rather than the
+ storage's: how big the dask chunks should be, given the store's own chunking
+ and whatever :data:`CHUNK_CONTROL` context is active.
+
+ Parameters
+ ----------
+ cf_var : :class:`iris.fileformats.cf.CFVariable`
+ The variable the array is being built for.
+
+ Returns
+ -------
+ tuple
+ ``("auto", None)`` in :attr:`ChunkControl.Modes.AS_DASK`, otherwise a
+ list of chunk sizes and a tuple of per-dimension "fixed" flags, both
+ shaped for :func:`iris._lazy_data.as_lazy_data`.
+
+ Raises
+ ------
+ KeyError
+ In :attr:`ChunkControl.Modes.FROM_FILE`, when ``cf_var`` is a cube's
+ data variable and the store offers no chunking to adopt.
+
+ """
+ if CHUNK_CONTROL.mode is ChunkControl.Modes.AS_DASK:
+ return "auto", None
+
+ # Get the chunking specified for the variable : this is either a shape, or
+ # None if the variable is unchunked.
+ chunks = cf_var.cf_data.chunking
+ if chunks is None:
+ # Unchunked : either a non-version-4 file, or a contiguous
+ # version-4 variable. Neither offers a chunking to adopt.
+ if CHUNK_CONTROL.mode is ChunkControl.Modes.FROM_FILE and isinstance(
+ cf_var, iris.fileformats.cf.CFDataVariable
+ ):
+ raise KeyError(
+ f"{cf_var.cf_name} does not contain pre-existing chunk specifications."
+ f" Instead, you might wish to use CHUNK_CONTROL.set(), or just use default"
+ f" behaviour outside of a context manager. "
+ )
+ # Equivalent to chunks=None, but value required by chunking control
+ chunks = list(cf_var.shape)
+ else:
+ # The chunk-control block below assigns into this.
+ chunks = list(chunks)
+
+ # Modify the chunking in the context of an active chunking control.
+ # N.B. settings specific to this named var override global ('*') ones.
+ dim_chunks = CHUNK_CONTROL.var_dim_chunksizes.get(
+ cf_var.cf_name
+ ) or CHUNK_CONTROL.var_dim_chunksizes.get("*")
+ dims = cf_var.dimensions
+ if CHUNK_CONTROL.mode is ChunkControl.Modes.FROM_FILE:
+ dims_fixed = np.ones(len(dims), dtype=bool)
+ elif not dim_chunks:
+ dims_fixed = None
+ else:
+ # Modify the chunks argument, and pass in a list of 'fixed' dims, for
+ # any of our dims which are controlled.
+ dims_fixed = np.zeros(len(dims), dtype=bool)
+ for i_dim, dim_name in enumerate(dims):
+ dim_chunksize = dim_chunks.get(dim_name)
+ if dim_chunksize:
+ if dim_chunksize == -1:
+ chunks[i_dim] = cf_var.shape[i_dim]
+ else:
+ chunks[i_dim] = dim_chunksize
+ dims_fixed[i_dim] = True
+ if dims_fixed is None:
+ dims_fixed = [dims_fixed]
+ return chunks, tuple(dims_fixed)
+```
+
+Now delete the same code from inside `_get_cf_var_data` and call the new
+function there. The lazy tail of `_get_cf_var_data` becomes:
+
+```python
+ proxy = proxy_class(
+ cf_var.cf_data.variable, dtype, cf_var.filename, fill_value
+ )
+ chunks, dims_fixed = _chunks_from_chunk_control(cf_var)
+ result = as_lazy_data(
+ proxy,
+ meta=proxy.dask_meta,
+ chunks=chunks,
+ dims_fixed=dims_fixed,
+ cache_key=repr(proxy),
+ )
+```
+
+This replaces **both** former `as_lazy_data` calls with one; see F11 for why
+that is identical.
+
+- [ ] **Step 4: Run the loader tests — this step must be green on its own**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/loader/ -q
+```
+
+Expected: PASS. The contract tests from Step 1 still fail; nothing else should.
+
+- [ ] **Step 5: Move the body onto `NetCDFDatasetVariable.read_data`**
+
+In `lib/iris/fileformats/netcdf/_dataset.py`, add the import and the constant
+near the top, after `from iris._deprecation import warn_deprecated`:
+
+```python
+from iris._lazy_data import as_lazy_data
+```
+
+and, after the module's existing constants:
+
+```python
+# A stab in the dark at the mean length of the "ragged dimension" for netCDF
+# "variable length arrays" (`NetCDF.VLType` type). Total array size is unknown
+# until the variable is read in. Making this number bigger makes it more likely
+# an array will be loaded lazily.
+_MEAN_VL_ARRAY_LEN = 10
+```
+
+Then add this method to `NetCDFDatasetVariable`, immediately after
+`write_handle`. It is a transplant: keep the comments, the warning text and the
+branch order exactly as they are in `_get_cf_var_data` today, so the reviewer
+can diff the two.
+
+One deliberate exception, so it is not mistaken for an accident: the comment on
+the emulation branch currently calls the facility the "Xarray bridge", after the
+2022 title of issue #4994. That name is retired — the facility is
+[ncdata](https://github.com/SciTools/ncdata), and `_dataset.py` already says
+"ncdata" everywhere else it mentions emulation. The comment below is reworded to
+match its new home. The issue link stays; only the dead name goes.
+
+```python
+ def read_data(self, chunking_policy):
+ """Return this variable's data, lazily where a lazy array is worth it.
+
+ Parameters
+ ----------
+ chunking_policy : callable
+ Called with no arguments, returning the ``(chunks, dims_fixed)``
+ pair to build a lazy array with. It is called only once this method
+ has decided the result will be lazy: under
+ :meth:`~iris.fileformats.cf.loader.ChunkControl.from_file` the
+ policy raises for a variable the store has not chunked, and a
+ variable small enough to read whole must not raise.
+
+ Returns
+ -------
+ numpy.ndarray or dask.array.Array
+ Real data for a small variable, lazy data otherwise.
+
+ """
+ # Deferred, and the module rather than its members: iris.fileformats.cf
+ # reaches this module through iris.fileformats.cf._reader, so a
+ # module-level import would be a cycle; and the tests patch
+ # _LAZYVAR_MIN_BYTES on the module object.
+ from iris.fileformats.netcdf import loader
+
+ if self.is_emulated:
+ # The variable is not an actual netCDF4 file variable, but an
+ # emulating object with an attached data array (either numpy or
+ # dask), which can be returned immediately as-is. This is the hook
+ # ncdata (https://github.com/SciTools/ncdata) uses to translate data
+ # to and from other packages' netCDF data containers.
+ # See https://github.com/SciTools/iris/issues/4994.
+ result = self.emulated_data_array
+ if result.dtype.kind == "S":
+ # We must also perform any byte-to-string decoding since, in
+ # ncdata, the emulating objects don't do this, and also don't
+ # support a 'set_auto_chartostring(True)'.
+ # Therefore, do here what an EncodedVariable.__getitem__ would
+ # do : ..
+ # .. get details based on the file (type 'char') variable ..
+ encoder = _bytecoding_datasets.VariableEncoder.from_var(
+ self.unencoded_variable
+ )
+ # .. convert byte array to strings.
+ result = encoder.decode_bytes_to_stringarray(result)
+ return result
+
+ # Determine size of data; however can't do this for variable length (VLEN)
+ # netCDF arrays as the size of the array can only be known by reading the
+ # data; see https://github.com/Unidata/netcdf-c/issues/1893.
+ # Note: "Variable length" netCDF types have a datatype of `nc.VLType`.
+ if self.is_variable_length:
+ msg = (
+ f"NetCDF variable `{self.name}` is a variable length type of kind "
+ f"{self.dtype} "
+ "thus the total data size cannot be known in advance. This may affect "
+ "the lazy loading of the data."
+ )
+ warnings.warn(msg, category=iris.warnings.IrisLoadWarning)
+
+ # Give user the chance to pass a hint of the average variable length array
+ # size via the chunk control context manager. This allows for better
+ # decisions to be made on whether the data should be lazy-loaded or not.
+ mean_vl_array_len = _MEAN_VL_ARRAY_LEN
+ chunk_control = loader.CHUNK_CONTROL
+ if chunk_control.mode is not chunk_control.Modes.AS_DASK:
+ if chunks := chunk_control.var_dim_chunksizes.get(self.name):
+ if vl_chunk_hint := chunks.get("_vl_hint"):
+ mean_vl_array_len = vl_chunk_hint
+
+ # Special handling for strings (`str` type) as these don't have an
+ # itemsize attribute; assume 4 bytes which is sufficient for unicode
+ # character storage
+ itemsize = 4 if self.dtype is str else self.dtype.itemsize
+
+ # For `VLType` self.size will just return the known dimension size.
+ total_bytes = self.size * mean_vl_array_len * itemsize
+ else:
+ # Normal NCVariable type:
+ total_bytes = self.size * self.dtype.itemsize
+
+ if total_bytes < loader._LAZYVAR_MIN_BYTES:
+ # Don't make a lazy array, as it will cost more memory AND more time
+ # to access.
+ result = self[:]
+
+ # Special handling of masked scalar value; this will be returned as
+ # an `np.ma.masked` instance which will lose the original dtype.
+ # Workaround for this it return a 1-element masked array of the
+ # correct dtype. Note: this is not an issue for masked arrays,
+ # only masked scalar values.
+ if result is np.ma.masked:
+ result = np.ma.masked_all(1, dtype=self.dtype)
+ return result
+
+ # Get lazy chunked data out of a cf variable.
+ # Creates Dask wrappers around data arrays for any cube components which
+ # can have lazy values, e.g. Cube, Coord, CellMeasure, AuxiliaryVariable.
+ dtype = loader._get_actual_dtype(self)
+
+ # Make a data-proxy that mimics array access and can fetch from the file.
+ # Note: Special handling needed for "variable length string" types which
+ # return a dtype of `str`, rather than a numpy type; use `S1` in this case.
+ if getattr(self.dtype, "kind", None) == "U":
+ # Special handling for "string variables".
+ fill_value = ""
+ else:
+ fill_dtype = "S1" if self.dtype is str else self.dtype.str[1:]
+ fill_value = self.attributes.get(
+ "_FillValue", _thread_safe_nc.default_fillvals[fill_dtype]
+ )
+
+ # Switch type of proxy, based on type of variable.
+ # It is done this way, instead of using an instance variable, because the
+ # limited nature of the wrappers makes a stateful choice awkward,
+ # e.g. especially, "variable.group()" is *not* the parent DatasetWrapper.
+ if isinstance(self.variable, _bytecoding_datasets.EncodedVariable):
+ proxy_class = _bytecoding_datasets.EncodedNetCDFDataProxy
+ else:
+ proxy_class = _thread_safe_nc.NetCDFDataProxy
+
+ proxy = proxy_class(self.variable, dtype, self.location, fill_value)
+
+ chunks, dims_fixed = chunking_policy()
+ return as_lazy_data(
+ proxy,
+ meta=proxy.dask_meta,
+ chunks=chunks,
+ dims_fixed=dims_fixed,
+ cache_key=repr(proxy),
+ )
+```
+
+Then replace the whole body of `_get_cf_var_data` in
+`lib/iris/fileformats/netcdf/loader.py` with the delegation, and delete
+`_MEAN_VL_ARRAY_LEN` from that module:
+
+```python
+def _get_cf_var_data(cf_var):
+ """Get an array representing the data of a CF variable.
+
+ This is typically a lazy array, but if the variable is "sufficiently small"
+ the storage fetches the data as a real (numpy) array instead. The latter is
+ especially valuable for scalar coordinates, which are otherwise
+ unnecessarily slow + wasteful of memory.
+
+ The store builds the array; this supplies the one decision that is CF's and
+ not the store's, namely what chunking a lazy result should have. See section
+ 4.4 of ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+ """
+ return cf_var.cf_data.read_data(partial(_chunks_from_chunk_control, cf_var))
+```
+
+`partial` is already imported at the top of the module. Ruff will now report
+`VariableEncoder`, `_thread_safe_nc` and possibly `as_lazy_data` as unused
+imports in `netcdf/loader.py`; remove the ones it names and leave the rest.
+
+- [ ] **Step 6: Run the contract tests and watch them pass**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/ -q
+```
+
+Expected: PASS, including the two from Step 1.
+
+- [ ] **Step 7: Repair the two test modules that reach past the seam**
+
+`test__get_cf_var_data.py` builds its `cf_data` as a `MagicMock` specced on
+`NetCDFDatasetVariable`, so it would now answer `read_data` with another mock.
+Give it the real method, and the members `read_data` reads. In `_make`, after
+the `cf_data = self.mocker.MagicMock(...)` call, add:
+
+```python
+ # `name` is a MagicMock constructor keyword, so it is set afterwards.
+ cf_data.name = "DUMMY_VAR"
+ cf_data.location = self.filename
+ cf_data.dtype = dtype
+ cf_data.size = np.prod(shape)
+ cf_data.attributes = {}
+ cf_data.__getitem__.return_value = self.mocker.sentinel.real_data_accessed
+ # The real method, bound to the mock: it reads only the members above,
+ # so the existing assertions keep testing the code they always did.
+ cf_data.read_data = partial(
+ iris.fileformats.netcdf._dataset.NetCDFDatasetVariable.read_data, cf_data
+ )
+```
+
+Move the `dtype = np.dtype(dtype)` normalisation above this block so `cf_data.dtype`
+gets the normalised value, and add `from functools import partial` to the
+module's imports. Leave
+`cf_var.__getitem__.return_value = self.mocker.sentinel.real_data_accessed`
+where it is: harmless, and it documents that the CF variable forwards.
+
+In `test__chunk_control.py`, line 197, the patch target moves with the call:
+
+```python
+ as_lazy_data = mocker.patch("iris.fileformats.netcdf._dataset.as_lazy_data")
+```
+
+- [ ] **Step 8: Run the netCDF and CF unit tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/ \
+ lib/iris/tests/unit/fileformats/cf/ -q
+```
+
+Expected: PASS.
+
+- [ ] **Step 9: Declare the hook on the abstract base**
+
+Only now, when every implementation has it. In
+`lib/iris/fileformats/cf/dataset.py`, add `Callable` to the
+`collections.abc` import list, and add this method to `CFDatasetVariable`
+immediately after `__setitem__`:
+
+```python
+ @abstractmethod
+ def read_data(self, chunking_policy: Callable[[], tuple]) -> Any:
+ """Return this variable's data, lazily where a lazy array is worth it.
+
+ The one route by which the CF loader obtains arrays, so that a new
+ storage format is a new implementation of this method and nothing else.
+
+ ``chunking_policy`` is called with no arguments and returns the
+ ``(chunks, dims_fixed)`` pair for a lazy result. Call it **only** once
+ the result is known to be lazy: under
+ :meth:`~iris.fileformats.cf.loader.ChunkControl.from_file` it raises for
+ a variable the store has not chunked, and a variable small enough to
+ read whole must reach its data without raising.
+
+ """
+```
+
+- [ ] **Step 10: Run the dataset tests again**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/dataset/ \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/ -q
+```
+
+Expected: PASS. A failure here means some other `CFDatasetVariable` subclass
+exists that Step 5 did not reach; `grep -rn "CFDatasetVariable)" lib/` will find
+it.
+
+- [ ] **Step 11: Pin the netCDF-specific paths — emulated data, and no proxy**
+
+Add to `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py`,
+after `class TestEmulation` (the class holding `test_emulated_round_trip`). Add
+`import dask.array as da` to the module imports.
+
+```python
+def _unreachable_policy():
+ raise AssertionError("chunking policy consulted when the result is not lazy")
+
+
+class TestReadData:
+ def test_small_variable_is_real(self, air):
+ result = air.read_data(_unreachable_policy)
+ assert isinstance(result, np.ndarray)
+ np.testing.assert_array_equal(result, SAMPLE_AIR)
+
+ def test_large_variable_is_lazy(self, air, mocker):
+ mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 0)
+ result = air.read_data(lambda: ([1, 4], (False, False)))
+ assert isinstance(result, da.Array)
+ np.testing.assert_array_equal(result.compute(), SAMPLE_AIR)
+
+ def test_cache_key_is_the_proxy_repr(self, air, mocker):
+ mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 0)
+ as_lazy_data = mocker.patch("iris.fileformats.netcdf._dataset.as_lazy_data")
+ air.read_data(lambda: ([1, 4], (False, False)))
+ (proxy,) = as_lazy_data.call_args.args
+ assert as_lazy_data.call_args.kwargs["cache_key"] == repr(proxy)
+
+ def test_emulated_array_is_returned_as_it_stands(self, sample_location):
+ # The ncdata emulation hook, issue #4994: the emulating object's own
+ # array is the answer, with no reading and no proxy.
+ variable = _EmulatedVariable()
+ variable._data_array = np.arange(6, dtype="f4").reshape(2, 3)
+ wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location)
+ result = wrapped.read_data(_unreachable_policy)
+ assert result is variable._data_array
+
+ def test_emulated_array_builds_no_proxy(self, sample_location, mocker):
+ proxy_class = mocker.patch.object(_thread_safe_nc, "NetCDFDataProxy")
+ variable = _EmulatedVariable()
+ variable._data_array = np.arange(6, dtype="f4").reshape(2, 3)
+ wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location)
+ wrapped.read_data(_unreachable_policy)
+ assert proxy_class.call_count == 0
+```
+
+- [ ] **Step 12: Run them**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py -v -k ReadData
+```
+
+Expected: PASS. These characterise transplanted code, so a failure means the
+transplant in Step 5 diverged from the original — diff it against
+`git show HEAD:lib/iris/fileformats/netcdf/loader.py`.
+
+- [ ] **Step 13: Pin the VLEN warning**
+
+Add to `class Test__get_cf_var_data` in
+`lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py`
+(the file's VLEN tests already cover `_vl_hint`, but none asserts the warning):
+
+```python
+ def test_vltype_warns_that_the_size_is_unknown(self):
+ cf_var = self._make(shape=(1000,), dtype="f8")
+ cf_var.cf_data.is_variable_length = True
+ with pytest.warns(
+ iris.warnings.IrisLoadWarning, match="variable length type of kind"
+ ):
+ _get_cf_var_data(cf_var)
+```
+
+Add `import iris.warnings` to the module's imports.
+
+- [ ] **Step 14: Run it**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py -v
+```
+
+Expected: PASS.
+
+- [ ] **Step 15: Pin the attribute-tracking equivalence (F9)**
+
+`read_data` now reads `_FillValue`, `scale_factor` and `add_offset` from the
+storage object's untracked attributes, where the old code read them from the
+`CFVariable`'s tracked copy. Add to `class TestReadData`:
+
+```python
+ def test_reading_data_changes_no_attribute_tracking(self, tmp_path, mocker):
+ # Which file attributes survive onto a cube is decided by which ones
+ # the loading rules did *not* read. read_data reads three CF-reserved
+ # names off the untracked storage mapping; this is only safe because
+ # those names are pre-marked as read. Pin that, because the failure is
+ # a silent leak onto cube.attributes.
+ from iris.fileformats.cf import CFReader
+
+ path = tmp_path / "packed.nc"
+ dataset = _thread_safe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ try:
+ dataset.createDimension("x", 4)
+ var = dataset.createVariable("thing", "i2", ("x",), fill_value=-1)
+ var.setncattr("scale_factor", np.float32(0.5))
+ var.setncattr("add_offset", np.float32(2.0))
+ var.setncattr("missing_value", np.int16(-1))
+ var.setncattr("units", "K")
+ var[:] = np.arange(4, dtype="i2")
+ finally:
+ dataset.close()
+
+ mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 0)
+ with CFReader(str(path)) as reader:
+ cf_var = reader.cf_group["thing"]
+ before = {name for name, _ in cf_var.cf_attrs_unused()}
+ assert before == {"units"}, (
+ "the four CF-reserved names should start out marked as read"
+ )
+ iris.fileformats.netcdf.loader._get_cf_var_data(cf_var)
+ after = {name for name, _ in cf_var.cf_attrs_unused()}
+ assert after == before
+```
+
+Add `import iris.fileformats.netcdf.loader` to the module's imports.
+
+- [ ] **Step 16: Run it**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py -v -k attribute_tracking
+```
+
+Expected: PASS. If `before` is not exactly `{"units"}`, the premise of F9 is
+wrong — stop and check `_CF_ATTRS_IGNORE` in `cf/_variables.py` before going on.
+
+- [ ] **Step 17: Pin the chunking policy, including the ordering invariant**
+
+Add to `lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py`.
+Note the module's autouse `remove_min_bytes` fixture sets the threshold to 0;
+the last test here is precisely about what that threshold guards, so it puts it
+back.
+
+```python
+class TestChunksFromChunkControl:
+ @staticmethod
+ def _cf_var(mocker, chunking, shape=(2, 3, 4), cls=None):
+ if cls is None:
+ cls = iris.fileformats.cf.CFDataVariable
+ dimensions = tuple(f"dim_{i}" for i in range(len(shape)))
+ cf_data = mocker.MagicMock(
+ spec=iris.fileformats.netcdf._dataset.NetCDFDatasetVariable,
+ chunking=chunking,
+ dimensions=dimensions,
+ )
+ return mocker.MagicMock(
+ spec=cls,
+ cf_data=cf_data,
+ cf_name="DUMMY_VAR",
+ shape=shape,
+ dimensions=dimensions,
+ )
+
+ def test_as_dask_defers_everything_to_dask(self, mocker):
+ cf_var = self._cf_var(mocker, chunking=None)
+ with CHUNK_CONTROL.as_dask():
+ assert loader._chunks_from_chunk_control(cf_var) == ("auto", None)
+
+ def test_default_unchunked_uses_the_shape(self, mocker):
+ cf_var = self._cf_var(mocker, chunking=None)
+ chunks, dims_fixed = loader._chunks_from_chunk_control(cf_var)
+ assert chunks == [2, 3, 4]
+ assert dims_fixed == (None,)
+
+ def test_from_file_adopts_and_fixes_the_store_chunking(self, mocker):
+ cf_var = self._cf_var(mocker, chunking=(1, 3, 4))
+ with CHUNK_CONTROL.from_file():
+ chunks, dims_fixed = loader._chunks_from_chunk_control(cf_var)
+ assert chunks == [1, 3, 4]
+ assert tuple(bool(flag) for flag in dims_fixed) == (True, True, True)
+
+ def test_from_file_refuses_an_unchunked_data_variable(self, mocker):
+ cf_var = self._cf_var(mocker, chunking=None)
+ with CHUNK_CONTROL.from_file():
+ with pytest.raises(KeyError, match="pre-existing chunk specifications"):
+ loader._chunks_from_chunk_control(cf_var)
+
+ def test_from_file_accepts_an_unchunked_coordinate(self, mocker):
+ # Only the cube's data variable is required to carry a chunking; an
+ # auxiliary coordinate that does not is normal.
+ cf_var = self._cf_var(
+ mocker,
+ chunking=None,
+ cls=iris.fileformats.cf.CFAuxiliaryCoordinateVariable,
+ )
+ with CHUNK_CONTROL.from_file():
+ chunks, _ = loader._chunks_from_chunk_control(cf_var)
+ assert chunks == [2, 3, 4]
+
+
+def test_from_file_still_loads_a_small_unchunked_cube(tmp_path, mocker):
+ # The KeyError sits behind the "small enough to read whole" shortcut, so a
+ # small contiguous cube loads under from_file() today. Hoisting the check
+ # into the CF layer, which is where the "is this the data variable" fact
+ # lives, would break this - so pin it.
+ mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 5000)
+ path = str(tmp_path / "small.nc")
+ iris.save(istk.simple_3d(), path)
+ with CHUNK_CONTROL.from_file():
+ cube = next(loader.load_cubes(path))
+ assert cube.shape == (2, 3, 4)
+ assert not cube.has_lazy_data()
+```
+
+Add `import iris.fileformats.cf` and `import iris.fileformats.netcdf._dataset`
+to the module's imports.
+
+- [ ] **Step 18: Run the chunk-control tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py -v
+```
+
+Expected: PASS.
+
+- [ ] **Step 19: Run the whole suite, then pre-commit**
+
+```bash
+pytest lib/iris/tests -n auto -q
+pre-commit run --files \
+ lib/iris/fileformats/cf/dataset.py \
+ lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/fileformats/netcdf/loader.py \
+ lib/iris/tests/unit/fileformats/cf/dataset/contract.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py
+```
+
+Expected: the same suite result as the baseline. `numpydoc-validation` will want
+`Parameters`, `Returns` and `Raises` sections on the new public-looking methods;
+the docstrings above already have them.
+
+- [ ] **Step 20: Commit**
+
+```bash
+git add lib/iris/fileformats/cf/dataset.py \
+ lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/fileformats/netcdf/loader.py \
+ lib/iris/tests/unit/fileformats/cf/dataset/contract.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py
+git commit -m "$(cat <<'EOF'
+Give CFDatasetVariable a read_data hook
+
+The loader asked a netCDF variable for its data by building a netCDF
+proxy itself. It now asks the storage object, through one new abstract
+method, and supplies only the part that is CF's business: what chunking
+a lazy result should have.
+
+The netCDF body moves verbatim, branch order included: the KeyError for
+an unchunked variable under from_file() still sits behind the "small
+enough to read whole" shortcut, so a small contiguous cube still loads.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 3: Move the loader to `iris.fileformats.cf.loader`
+
+This is the task the pull request is named after. Everything before it was
+making the move possible without the move itself changing behaviour.
+
+**Files:**
+- Move: `lib/iris/fileformats/netcdf/loader.py` →
+ `lib/iris/fileformats/cf/loader.py` (with `git mv`, so the history follows)
+- Create: `lib/iris/fileformats/netcdf/loader.py` (the netCDF part, written
+ fresh at the old path)
+- Create: `lib/iris/tests/unit/fileformats/netcdf/loader/test__relocated_names.py`
+- Modify: `lib/iris/fileformats/netcdf/__init__.py`
+- Modify: `lib/iris/fileformats/netcdf/_dataset.py` (one deferred import)
+- Modify: `lib/iris/fileformats/netcdf/ugrid_load.py`
+- Modify: `lib/iris/fileformats/_nc_load_rules/helpers.py`
+- Modify: `lib/iris/fileformats/_nc_load_rules/actions.py` (one comment)
+- Modify: `lib/iris/fileformats/_nc_load_rules/engine.py` (one docstring)
+- Modify: `lib/iris/fileformats/cf/_variables.py` (one docstring)
+- Modify: `benchmarks/benchmarks/generate_data/__init__.py`
+- Modify: eleven test modules, listed in Step 11 and Step 13
+
+**Interfaces:**
+- Consumes: `CFDatasetVariable.read_data` and `_chunks_from_chunk_control`
+ from Task 2.
+- Produces: `iris.fileformats.cf.loader`, holding `CHUNK_CONTROL`,
+ `ChunkControl`, `_LAZYVAR_MIN_BYTES`, `_get_cf_var_data`,
+ `_chunks_from_chunk_control`, `_load_cube`, `_load_cube_inner`,
+ `_load_aux_factory`, `_actions_engine`, `_actions_activation_stats`,
+ `_assert_case_specific_facts`, `_set_attributes`, `_add_unused_attributes`,
+ `_get_actual_dtype`, `_OrderedAddableList`, `_WarnComboIgnoringBoundsLoad`
+ and `_translate_constraints_to_var_callback`. And
+ `iris.fileformats.netcdf.loader`, holding `DEBUG`, `NetCDFDataProxy`,
+ `load_cubes`, `_RELOCATED_NAMES` and a module `__getattr__`.
+
+**What goes where, and why**
+
+Three names decide the shape of this task.
+
+`CHUNK_CONTROL` and `ChunkControl` are in the spec's relocation table
+(§4.8): they move to `iris.fileformats.cf.loader`, and the old names keep
+working for one deprecation cycle, warning on use.
+
+`DEBUG` is in the spec's *other* list — the names that explicitly do **not**
+move, because they are netCDF API (§4.8: "``netcdf.save``, ``netcdf.Saver``,
+``netcdf.load_cubes``, ``netcdf.save_mesh``, ``netcdf.NetCDFDataProxy``,
+``netcdf.DEBUG``"). Its one reader, `_load_cube_inner`, is generic code that
+is moving. So `DEBUG` stays defined where it is and the moved code reaches
+back for it, rather than the reverse. That is the right way round for a
+second reason: `DEBUG` is a flag users set by assignment, and a module
+`__getattr__` forward cannot intercept assignment (PEP 562 defines
+`__getattr__` for modules and nothing for `__setattr__`). A name that is only
+ever read can be forwarded; a name that is written cannot. `DEBUG` is
+written, so it does not get forwarded.
+
+`load_cubes` stays too, for the same reason: opening a list of file sources
+is netCDF's job, and `iris.fileformats.netcdf.load_cubes` is in the
+do-not-move list. It imports the four CF-layer functions it drives.
+
+Two files that might look like they need editing do not. `cf/__init__.py`
+re-exports the public names `iris.fileformats.cf` exports *today*, and
+`CHUNK_CONTROL` is not one of them: the spec's relocation table (§4.8) names
+`iris.fileformats.cf.loader.CHUNK_CONTROL`, with the submodule spelled out, so
+do not add it to `cf/__init__.py` or to its `__all__`. `netcdf/__init__.py`
+imports `DEBUG`, `NetCDFDataProxy` and `load_cubes` from `.loader` (line 34)
+and lists all three in `__all__` — all three stay, so that line is already
+correct and must not be touched.
+
+- [ ] **Step 1: Write the failing test for the relocated names**
+
+Create `lib/iris/tests/unit/fileformats/netcdf/loader/test__relocated_names.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the deprecated aliases left behind in the netCDF loader.
+
+``CHUNK_CONTROL`` and ``ChunkControl`` moved to
+:mod:`iris.fileformats.cf.loader`; the old names still answer, with a warning.
+"""
+
+import importlib
+import warnings
+
+import pytest
+
+from iris._deprecation import IrisDeprecation
+from iris.fileformats.cf import loader as cf_loader
+from iris.fileformats.netcdf import loader as netcdf_loader
+
+RELOCATED = ["CHUNK_CONTROL", "ChunkControl"]
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+def test_the_old_name_warns(name):
+ with pytest.warns(IrisDeprecation, match=f"netcdf.loader.{name}"):
+ getattr(netcdf_loader, name)
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+def test_the_message_names_the_new_location(name):
+ with pytest.warns(IrisDeprecation, match=r"iris\.fileformats\.cf\.loader"):
+ getattr(netcdf_loader, name)
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+def test_the_alias_is_the_same_object(name):
+ # Spec 4.8: "The alias is the same object, not a copy, so existing code
+ # keeps working including inside a with CHUNK_CONTROL.set(...)". A copy
+ # would give a with-block that silently chunked nothing.
+ with pytest.warns(IrisDeprecation):
+ assert getattr(netcdf_loader, name) is getattr(cf_loader, name)
+
+
+def test_importing_the_module_does_not_warn():
+ # Spec 4.8: "Warnings are emitted on use, not on import, so simply
+ # importing iris.fileformats.netcdf stays quiet."
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ importlib.reload(importlib.import_module("iris.fileformats.netcdf"))
+
+
+def test_an_unrelated_name_raises_without_warning():
+ # hasattr() probes land in __getattr__ too, and a probe that comes back
+ # False is not a use of anything.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ with pytest.raises(AttributeError, match="no_such_name"):
+ netcdf_loader.no_such_name
+
+
+def test_debug_is_not_relocated():
+ # Spec 4.8 lists netcdf.DEBUG among the names that do not move. It is a
+ # flag users assign to, and a module __getattr__ cannot intercept
+ # assignment - so forwarding it would make `netcdf.loader.DEBUG = True`
+ # silently stop working. Pin that it is a real attribute here.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ assert netcdf_loader.DEBUG is False
+ assert "DEBUG" not in netcdf_loader._RELOCATED_NAMES
+
+
+def test_assignment_to_a_relocated_name_does_not_reach_the_new_home():
+ # The known limitation of the PEP 562 forward, written down so that it is
+ # a documented edge and not a surprise. Assigning here shadows the
+ # forward; it does not change iris.fileformats.cf.loader. Nothing in Iris
+ # assigns to CHUNK_CONTROL - it is used through its context managers - but
+ # if that ever changes, this test says where to look.
+ original = cf_loader.CHUNK_CONTROL
+ netcdf_loader.CHUNK_CONTROL = "shadow"
+ try:
+ assert netcdf_loader.CHUNK_CONTROL == "shadow"
+ assert cf_loader.CHUNK_CONTROL is original
+ finally:
+ del netcdf_loader.CHUNK_CONTROL
+```
+
+- [ ] **Step 2: Run it to watch it fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/loader/test__relocated_names.py -v
+```
+
+Expected: collection error, `ModuleNotFoundError: No module named
+'iris.fileformats.cf.loader'`.
+
+- [ ] **Step 3: Move the file**
+
+```bash
+git mv lib/iris/fileformats/netcdf/loader.py lib/iris/fileformats/cf/loader.py
+```
+
+Nothing else yet. Doing the move as its own command, with `git mv` rather than
+a delete-and-create, is what makes `git log --follow` and `git blame` keep
+working across the relocation — and it is what lets a reviewer read the next
+step as a diff rather than as 1000 new lines.
+
+- [ ] **Step 4: Cut the netCDF part out of the moved file**
+
+In `lib/iris/fileformats/cf/loader.py`, delete these four things. They are
+given by name, not by line number, because Task 2 has already shifted the
+numbering.
+
+1. The `DEBUG = False` definition and its `# Show actions activation
+ statistics.` comment.
+2. The `# Get the logger ...` comment and `from . import logger`.
+3. The `NetCDFDataProxy = _bytecoding_datasets.EncodedNetCDFDataProxy`
+ assignment and the four-line comment block above it.
+4. The whole of `def load_cubes(...)`.
+
+Then replace the module docstring and import block with:
+
+```python
+"""Build Iris cubes from CF-conforming variables, whatever stores them.
+
+.. z_reference:: iris.fileformats.cf.loader
+ :tags: topic_load_save
+
+ API reference
+
+This module is the CF half of what used to be
+``iris.fileformats.netcdf.loader``. It takes the
+:class:`~iris.fileformats.cf.CFVariable` objects a
+:class:`~iris.fileformats.cf.CFReader` produced, runs the loading rules over
+them and returns :class:`~iris.cube.Cube` objects. None of that is specific to
+netCDF, and nothing here imports netCDF4.
+
+Where a storage format *does* show through - reading a variable's array - the
+work is delegated to :meth:`~iris.fileformats.cf.dataset.CFDatasetVariable.read_data`,
+and this module supplies only the chunking policy to apply if the result comes
+back lazy.
+
+Opening files remains the format's own business: see
+:func:`iris.fileformats.netcdf.load_cubes`.
+
+Also : `CF Conventions <https://cfconventions.org/>`_.
+
+"""
+
+from collections.abc import Iterable, Iterator, Mapping
+from contextlib import contextmanager
+from copy import deepcopy
+from enum import Enum, auto
+from functools import partial
+import threading
+import warnings
+
+import numpy as np
+
+from iris.aux_factory import (
+ AtmosphereSigmaFactory,
+ HybridHeightFactory,
+ HybridPressureFactory,
+ OceanSFactory,
+ OceanSg1Factory,
+ OceanSg2Factory,
+ OceanSigmaFactory,
+ OceanSigmaZFactory,
+)
+import iris.coords
+import iris.fileformats.cf
+import iris.warnings
+```
+
+Four imports go: `as_lazy_data`, `_bytecoding_datasets`/`_thread_safe_nc`,
+`VariableEncoder` and `_CF_ATTRS`. The first three left with the code Task 2
+moved. `_CF_ATTRS` has to become a deferred import, because
+`iris.fileformats.netcdf.saver` is not importable from here at module scope —
+see Step 5. Four more go because `load_cubes` took their only uses with it:
+`iris.config`, `iris.coord_systems`, `iris.io` and `iris.util`. If `ruff` in
+Step 14 reports an unused import that this list does not mention, delete it;
+if it reports one as *missing*, something was cut too enthusiastically.
+
+- [ ] **Step 5: Make the two netCDF reach-backs deferred**
+
+Still in `lib/iris/fileformats/cf/loader.py`.
+
+In `_add_unused_attributes`, the test `item[0] not in _CF_ATTRS` needs
+`_CF_ATTRS` from the saver, which cannot be imported at module scope: the
+saver imports this module's package, and PR 5 is where the saver itself moves.
+Add a deferred import at the top of the function, beside the comment
+explaining why:
+
+```python
+def _add_unused_attributes(iris_object, cf_var):
+ # Deferred import: iris.fileformats.netcdf.saver imports the CF layer, so
+ # importing it here at module scope would close the loop. PR 5 moves the
+ # saver and this becomes a plain import.
+ from iris.fileformats.netcdf.saver import _CF_ATTRS
+```
+
+In `_load_cube_inner`, replace `if DEBUG:` with a read through the netCDF
+loader module:
+
+```python
+ # DEBUG stays in iris.fileformats.netcdf (spec 4.8 lists it among the
+ # names that do not move). Read it through the module, not as a
+ # from-import, so that assigning to iris.fileformats.netcdf.loader.DEBUG -
+ # which is how it is used - still takes effect here.
+ from iris.fileformats.netcdf import loader as netcdf_loader
+
+ if netcdf_loader.DEBUG:
+ _actions_activation_stats(engine, cf_var.cf_name)
+```
+
+The existing `from iris.fileformats.netcdf.saver import Saver` at the top of
+`_load_cube_inner` is already deferred and needs no change.
+
+- [ ] **Step 6: Write the new netCDF loader**
+
+Create `lib/iris/fileformats/netcdf/loader.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Load Iris cubes from NetCDF files and OPeNDAP URLs.
+
+.. z_reference:: iris.fileformats.netcdf.loader
+ :tags: topic_load_save
+
+ API reference
+
+What is left here is what is genuinely netCDF's: opening the file sources, and
+the proxy class that reads from them. Interpreting what is inside is
+:mod:`iris.fileformats.cf.loader`'s job.
+
+.. deprecated:: 3.17
+ ``CHUNK_CONTROL`` and ``ChunkControl`` have moved to
+ :mod:`iris.fileformats.cf.loader`, because chunking policy is not specific
+ to netCDF. They can still be reached here, with a warning, for one
+ deprecation cycle.
+
+See : `NetCDF User's Guide <https://docs.unidata.ucar.edu/nug/current/>`_
+and `netCDF4 python module <https://github.com/Unidata/netcdf4-python>`_.
+
+Also : `CF Conventions <https://cfconventions.org/>`_.
+
+"""
+
+from collections.abc import Iterable
+from functools import partial
+import warnings
+
+from iris._deprecation import warn_deprecated
+from iris.fileformats.cf.loader import (
+ _actions_engine,
+ _load_aux_factory,
+ _load_cube,
+ _translate_constraints_to_var_callback,
+)
+import iris.warnings
+
+# Show actions activation statistics.
+# NOTE: read by iris.fileformats.cf.loader._load_cube_inner, through this
+# module rather than by value, so that assigning to it here still works.
+DEBUG = False
+
+# Get the logger : shared logger for all in 'iris.fileformats.netcdf'.
+from . import _bytecoding_datasets, logger
+
+# An expected part of the public loader API, but includes thread safety
+# concerns so is housed in _thread_safe_nc.
+# NOTE: this is the *default*, as required for public legacy api
+# - in practice, when creating our proxies we dynamically choose between this and
+# :class:`_thread_safe_nc.DatasetWrapper`, depending on
+# :data:`_bytecoding_datasets.DECODE_TO_STRINGS_ON_READ`
+NetCDFDataProxy = _bytecoding_datasets.EncodedNetCDFDataProxy
+
+#: Names that moved to :mod:`iris.fileformats.cf.loader` in Iris 3.17 and are
+#: still readable here, with a warning, for one deprecation cycle.
+#:
+#: Read-only: a module ``__getattr__`` cannot intercept assignment, so a name
+#: that users *set* must keep its definition here instead. ``DEBUG`` is the
+#: one such name, and is deliberately absent from this set.
+_RELOCATED_NAMES = frozenset({"CHUNK_CONTROL", "ChunkControl"})
+
+
+def __getattr__(name):
+ """Forward a relocated name to :mod:`iris.fileformats.cf.loader`.
+
+ Parameters
+ ----------
+ name : str
+ The attribute being looked up. Python calls this only for names that
+ are not already module attributes.
+
+ Returns
+ -------
+ Any
+ The object of that name in :mod:`iris.fileformats.cf.loader` - the
+ same object, not a copy.
+
+ Raises
+ ------
+ AttributeError
+ If ``name`` is not one of the relocated names. Raised without warning,
+ so that a ``hasattr`` probe that comes back False stays quiet.
+
+ """
+ if name not in _RELOCATED_NAMES:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ # Deferred so that importing this module stays quiet and cheap.
+ from iris.fileformats.cf import loader as cf_loader
+
+ warn_deprecated(
+ f"iris.fileformats.netcdf.loader.{name} has moved to "
+ f"iris.fileformats.cf.loader.{name}, because it is not specific to "
+ "netCDF. The name here will be removed in a future release."
+ )
+ return getattr(cf_loader, name)
+```
+
+Then paste `load_cubes` in, unchanged, from
+`git show HEAD:lib/iris/fileformats/netcdf/loader.py`.
+
+- [ ] **Step 7: Point `iris.fileformats.netcdf` at the moved names**
+
+In `lib/iris/fileformats/netcdf/__init__.py`, the eager
+`from .loader import DEBUG, NetCDFDataProxy, load_cubes` is still correct:
+all three stayed. `__all__` needs no change either.
+
+Confirm that by reading the file, then move on. If any of those three names
+has in fact moved, the import will fail in Step 8 and the right fix is to
+revisit Step 6, not to add a forward here: a package-level
+`from .loader import X` of a forwarded name would fire the deprecation warning
+at *import* time, which spec §4.8 forbids.
+
+- [ ] **Step 8: Check the import graph before anything else**
+
+```bash
+python -c "import iris.fileformats.cf; print('cf first: ok')"
+python -c "import iris.fileformats.netcdf; print('netcdf first: ok')"
+python -c "import iris; print('iris: ok')"
+```
+
+Expected: three `ok` lines. The cycle is real and worth checking in both
+orders: `netcdf/__init__` imports `_nc_load_rules.helpers`, which imports
+`cf.loader`, whose package imports `cf._reader`, which imports
+`netcdf._dataset`. It resolves because every edge into a partially-initialised
+package is `from package.submodule import name`, which does not need the
+package body to have finished. An `ImportError` here means some edit turned
+one of those into a `from package import name`.
+
+- [ ] **Step 9: Run the relocated-names test**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/loader/test__relocated_names.py -v
+```
+
+Expected: PASS, all nine.
+
+- [ ] **Step 10: Repoint the production references**
+
+Six files, each a one-line change except `ugrid_load.py`.
+
+`lib/iris/fileformats/_nc_load_rules/helpers.py:40`:
+
+```python
+from iris.fileformats.cf.loader import _get_cf_var_data
+```
+
+`lib/iris/fileformats/netcdf/_dataset.py` — the deferred import inside
+`NetCDFDatasetVariable.read_data` that Task 2 added:
+
+```python
+ from iris.fileformats.cf import loader
+```
+
+`lib/iris/fileformats/netcdf/ugrid_load.py` — three call sites. The module
+imports the loader as `nc_loader` inside four functions (around lines 144,
+213, 270 and 315). Leave `nc_loader` alone where it is used for
+`nc_loader.load_cubes` (around line 183): that one is still netCDF's. For the
+three CF-layer uses, add a second import in the two functions that need it:
+
+```python
+ from iris.fileformats.cf import loader as cf_loader
+```
+
+and change `nc_loader._get_cf_var_data` to `cf_loader._get_cf_var_data`
+(two sites) and `nc_loader._add_unused_attributes` to
+`cf_loader._add_unused_attributes` (one site). Where that leaves a function
+with an unused `nc_loader` import, delete it.
+
+`lib/iris/fileformats/cf/_variables.py:124` — a docstring reference:
+
+```
+ cube - see :func:`iris.fileformats.cf.loader._add_unused_attributes`.
+```
+
+`lib/iris/fileformats/_nc_load_rules/actions.py:655` — a comment:
+
+```python
+ # Set 'requires' info for iris.fileformats.cf.loader._load_aux_factory.
+```
+
+`lib/iris/fileformats/_nc_load_rules/engine.py:18` — a docstring:
+
+```
+used in :meth:`iris.fileformats.cf.loader._actions_activation_stats`.
+```
+
+- [ ] **Step 11: Repoint `_LAZYVAR_MIN_BYTES` wherever a test reaches for it**
+
+This is the private name with the widest reach, and the one whose breakage
+would be quietest: some of these sites *assign* rather than patch, and an
+assignment to a name the module no longer defines succeeds and does nothing.
+Change `iris.fileformats.netcdf.loader` to `iris.fileformats.cf.loader` in
+all of:
+
+| File | Lines |
+|---|---|
+| `lib/iris/tests/unit/aux_factory/test_AuxCoordFactory.py` | 147 |
+| `lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_auxiliary_coordinate.py` | 213 |
+| `lib/iris/tests/integration/test_cube.py` | 23 |
+| `lib/iris/tests/integration/netcdf/test_general.py` | 42 |
+| `lib/iris/tests/integration/netcdf/test_stringdata.py` | 33, 34, 36, 406 |
+| `lib/iris/tests/integration/netcdf/test_delayed_save.py` | 47, 48, 50 |
+| `lib/iris/tests/integration/netcdf/test__dask_locks.py` | 130 (a docstring) |
+| `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py` | the three `mocker.patch` calls Task 2 added |
+
+Then check nothing was missed:
+
+```bash
+grep -rn "netcdf\.loader\._LAZYVAR_MIN_BYTES\|netcdf import loader" lib/ benchmarks/
+```
+
+Expected: no hits for `_LAZYVAR_MIN_BYTES`.
+
+- [ ] **Step 12: Repoint the benchmark helper**
+
+`benchmarks/benchmarks/generate_data/__init__.py`, in `load_realised()`
+(lines 117-136). Line 126 imports `_get_cf_var_data` from the netCDF loader
+and lines 131 and 134 assign it back. All three move to the CF loader; the
+`helpers._get_cf_var_data` lines beside them are unaffected.
+
+```python
+ from iris.fileformats.cf.loader import _get_cf_var_data as pre_patched
+```
+
+```python
+ cf_loader._get_cf_var_data = patched
+```
+
+with `from iris.fileformats.cf import loader as cf_loader` replacing whatever
+`netcdf` import the two assignment lines were using. If that leaves the
+`netcdf` import unused, delete it.
+
+Benchmarks are not run by the test suite, so nothing here fails loudly. Check
+it by eye, and with:
+
+```bash
+python -c "import benchmarks.benchmarks.generate_data as g; print(g.load_realised)"
+```
+
+- [ ] **Step 13: Repoint the remaining test references**
+
+`lib/iris/tests/integration/test_cf.py:14`:
+
+```python
+from iris.fileformats.cf.loader import _get_cf_var_data
+```
+
+`lib/iris/tests/unit/fileformats/nc_load_rules/actions/__init__.py` — line 14
+(`from iris.fileformats.netcdf.loader import _load_cube`), line 76
+(`iris.fileformats.netcdf.loader._actions_engine()`) and the docstrings at
+lines 22 and 104. Lines 82 and 85 patch
+`"iris.fileformats.netcdf.loader.DEBUG"` and are **correct as they stand** —
+`DEBUG` did not move, and Step 5 made the CF loader read it through the module
+so that this patch still bites. Leave them.
+
+The four `nc_load_rules/actions/test__*.py` module docstrings that say
+`iris.fileformats.netcdf._load_cube` (`test__hybrid_formulae.py:6`,
+`test__miscellaneous.py:6`, `test__latlon_dimcoords.py:6`,
+`test__time_coords.py:6`, `test__grid_mappings.py:6` — five, in fact) become
+`iris.fileformats.cf.loader._load_cube`.
+
+The five test modules under `lib/iris/tests/unit/fileformats/netcdf/loader/`
+that import from the loader are **not** touched here. Task 4 moves them, and
+repointing their imports is part of that move.
+
+- [ ] **Step 14: Run everything, then pre-commit**
+
+```bash
+pytest lib/iris/tests -n auto -q
+```
+
+Expected failures, all of them inside
+`lib/iris/tests/unit/fileformats/netcdf/loader/`, and all of them Task 4's
+work:
+
+| Module | What happens |
+|---|---|
+| `test__get_cf_var_data.py` | ImportError at collection: `_get_cf_var_data` is not in the new `netcdf/loader.py` at all |
+| `test__chunk_control.py` | Imports (with a deprecation warning, because `CHUNK_CONTROL` *is* forwarded), then every test errors in the autouse `remove_min_bytes` fixture: `AttributeError: _LAZYVAR_MIN_BYTES` |
+| `test__load_cube.py` | Imports fine, then errors in `_setup`: the patch target `netcdf.loader._assert_case_specific_facts` no longer exists |
+| `test__load_aux_factory.py` | **Passes.** `_load_aux_factory` is one of the four names the new `netcdf/loader.py` imports for `load_cubes`, so the old path still resolves |
+| `test__translate_constraints_to_var_callback.py` | **Passes**, for the same reason |
+
+Anything that fails outside that directory belongs to this task, not Task 4.
+
+To get a clean signal meanwhile:
+
+```bash
+pytest lib/iris/tests -n auto -q \
+ --ignore=lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py \
+ --ignore=lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py \
+ --ignore=lib/iris/tests/unit/fileformats/netcdf/loader/test__load_cube.py
+```
+
+Expected: the same result as the baseline.
+
+```bash
+pre-commit run --files $(git diff --name-only HEAD) \
+ lib/iris/fileformats/cf/loader.py \
+ lib/iris/fileformats/netcdf/loader.py \
+ lib/iris/tests/unit/fileformats/netcdf/loader/test__relocated_names.py
+```
+
+`check-docs-page-metadata` will object if either module's `z_reference` is not
+within the first 25 lines; both docstrings above put it at line 7.
+`numpydoc-validation` wants the `Parameters`/`Returns`/`Raises` sections on
+`__getattr__`; they are in Step 6.
+
+- [ ] **Step 15: Commit**
+
+```bash
+git add -A
+git commit -m "$(cat <<'EOF'
+Move the CF loader to iris.fileformats.cf.loader
+
+Interpreting CF metadata and running the loading rules is not netCDF's
+job; it only lived in iris/fileformats/netcdf/loader.py because netCDF
+was the only store that did it. The file moves with git mv so the
+history follows, and what stays behind at the old path is what is
+genuinely netCDF's: load_cubes, NetCDFDataProxy and DEBUG.
+
+CHUNK_CONTROL and ChunkControl move with the loader and keep warn-on-use
+aliases at their old names, per the spec's deprecation table. DEBUG does
+not move: it is a flag users assign to, and a module __getattr__ cannot
+intercept assignment, so forwarding it would have made setting it
+silently stop working.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 4: Move the loader's tests to match
+
+Five of the seven test modules under
+`lib/iris/tests/unit/fileformats/netcdf/loader/` test functions that are now
+in `iris.fileformats.cf.loader`. Iris mirrors the source tree in the test
+tree, so they move. Two stay: `test_load_cubes.py`, because `load_cubes`
+stayed, and `test__relocated_names.py`, which Task 3 wrote about the aliases
+that live at the netCDF path. The `ugrid_load/` subdirectory stays too.
+
+**Files:**
+- Create: `lib/iris/tests/unit/fileformats/cf/loader/__init__.py`
+- Move (`git mv`), from `lib/iris/tests/unit/fileformats/netcdf/loader/` to
+ `lib/iris/tests/unit/fileformats/cf/loader/`:
+ `test__chunk_control.py`, `test__get_cf_var_data.py`,
+ `test__load_aux_factory.py`, `test__load_cube.py`,
+ `test__translate_constraints_to_var_callback.py`
+
+**Interfaces:**
+- Consumes: `iris.fileformats.cf.loader` as Task 3 leaves it.
+- Produces: nothing other tasks depend on.
+
+- [ ] **Step 1: Create the destination package**
+
+```bash
+mkdir lib/iris/tests/unit/fileformats/cf/loader
+```
+
+Create `lib/iris/tests/unit/fileformats/cf/loader/__init__.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the :mod:`iris.fileformats.cf.loader` module."""
+```
+
+- [ ] **Step 2: Move the five modules**
+
+```bash
+cd lib/iris/tests/unit/fileformats
+git mv netcdf/loader/test__chunk_control.py cf/loader/
+git mv netcdf/loader/test__get_cf_var_data.py cf/loader/
+git mv netcdf/loader/test__load_aux_factory.py cf/loader/
+git mv netcdf/loader/test__load_cube.py cf/loader/
+git mv netcdf/loader/test__translate_constraints_to_var_callback.py cf/loader/
+cd -
+```
+
+Again `git mv`, and again as its own step, so the next step reads as a diff.
+
+- [ ] **Step 3: Repoint `test__get_cf_var_data.py`**
+
+Docstring (line 5) and import (line 15):
+
+```python
+"""Unit tests for :func:`iris.fileformats.cf.loader._get_cf_var_data`.
+
+The function is format-agnostic, but the only ``CFDatasetVariable`` that
+exists today is the netCDF one, so the mocks here are netCDF-shaped. PR 6
+adds the Zarr implementation and these become the shared cases.
+"""
+```
+
+```python
+from iris.fileformats.cf.loader import CHUNK_CONTROL, _get_cf_var_data
+```
+
+- [ ] **Step 4: Repoint `test__load_aux_factory.py`**
+
+```python
+"""Unit tests for :func:`iris.fileformats.cf.loader._load_aux_factory`."""
+```
+
+```python
+from iris.fileformats.cf.loader import _load_aux_factory
+```
+
+- [ ] **Step 5: Repoint `test__translate_constraints_to_var_callback.py`**
+
+```python
+"""Unit tests for
+:func:`iris.fileformats.cf.loader._translate_constraints_to_var_callback`.
+
+"""
+```
+
+```python
+from iris.fileformats.cf.loader import _translate_constraints_to_var_callback
+```
+
+- [ ] **Step 6: Repoint `test__load_cube.py`**
+
+Docstring (line 5), import (line 12), and the patch target that appears twice
+(lines 33 and 130):
+
+```python
+"""Unit tests for :func:`iris.fileformats.cf.loader._load_cube`."""
+```
+
+```python
+from iris.fileformats.cf.loader import _load_cube
+```
+
+```python
+ this = "iris.fileformats.cf.loader._assert_case_specific_facts"
+```
+
+- [ ] **Step 7: Repoint `test__chunk_control.py`**
+
+This one reaches for both loaders: `load_cubes` is still netCDF's, while
+`_LAZYVAR_MIN_BYTES` and `_chunks_from_chunk_control` moved. A bare `loader`
+in a file that now lives under `cf/` would be a trap, so name both.
+
+Docstring (line 5) and the comment at line 36:
+
+```python
+"""Unit tests for :class:`iris.fileformats.cf.loader.ChunkControl`."""
+```
+
+```python
+ # iris.fileformats.cf.loader._get_cf_var_data()
+```
+
+Replace the two loader imports (lines 15-16) with three:
+
+```python
+from iris.fileformats.cf import loader as cf_loader
+from iris.fileformats.cf.loader import CHUNK_CONTROL
+from iris.fileformats.netcdf import loader as netcdf_loader
+```
+
+Then rewrite the uses:
+
+```bash
+cd lib/iris/tests/unit/fileformats/cf/loader
+sed -i \
+ -e 's/\bloader\.load_cubes\b/netcdf_loader.load_cubes/g' \
+ -e 's/\bloader\._LAZYVAR_MIN_BYTES\b/cf_loader._LAZYVAR_MIN_BYTES/g' \
+ -e 's/\bloader\._chunks_from_chunk_control\b/cf_loader._chunks_from_chunk_control/g' \
+ test__chunk_control.py
+grep -n '\bloader\.' test__chunk_control.py
+cd -
+```
+
+Expected from the `grep`: only `cf_loader.` and `netcdf_loader.` prefixed
+uses, plus the two docstring/comment lines just rewritten. A bare `loader.`
+surviving means a use this step did not anticipate — look at it rather than
+widening the `sed`.
+
+Two things in this file stay as they are. The patch target
+`"iris.fileformats.netcdf._dataset.as_lazy_data"` (around line 197) is right
+already: Task 2 put it there, and the call it watches is in the netCDF
+dataset. And `mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 5000)`
+inside `test_from_file_still_loads_a_small_unchunked_cube` was already
+repointed by Task 3 Step 11, so the `sed` above will not find it.
+
+- [ ] **Step 8: Run the moved tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/loader -v
+```
+
+Expected: PASS, and the count should match what these five files reported
+before Task 3 moved anything.
+
+- [ ] **Step 9: Run the whole suite**
+
+```bash
+pytest lib/iris/tests -n auto -q
+```
+
+Expected: the same result as the baseline, with no `--ignore` needed this
+time. This is the first point in the branch where that is true, so if it is
+not, the cause is in Task 3 and is worth finding now rather than in review.
+
+- [ ] **Step 10: Confirm nothing still points at the old home**
+
+```bash
+grep -rn "netcdf\.loader\._\|netcdf import loader" lib/ benchmarks/ docs/src
+```
+
+Expected hits, and only these: `test__relocated_names.py` (which is about the
+old path on purpose), `test__chunk_control.py`'s `netcdf_loader` import and
+its `_LAZYVAR_MIN_BYTES` patch, `nc_load_rules/actions/__init__.py`'s two
+`DEBUG` patches, and `ugrid_load.py`'s `nc_loader` import for `load_cubes`.
+Documentation is Task 5's and is expected to still show hits.
+
+- [ ] **Step 11: pre-commit, then commit**
+
+```bash
+pre-commit run --files \
+ lib/iris/tests/unit/fileformats/cf/loader/__init__.py \
+ lib/iris/tests/unit/fileformats/cf/loader/test__chunk_control.py \
+ lib/iris/tests/unit/fileformats/cf/loader/test__get_cf_var_data.py \
+ lib/iris/tests/unit/fileformats/cf/loader/test__load_aux_factory.py \
+ lib/iris/tests/unit/fileformats/cf/loader/test__load_cube.py \
+ lib/iris/tests/unit/fileformats/cf/loader/test__translate_constraints_to_var_callback.py
+git add -A
+git commit -m "$(cat <<'EOF'
+Move the CF loader's tests to match
+
+The test tree mirrors the source tree, so the five modules that test
+functions now in iris.fileformats.cf.loader move with them. test_load_cubes
+and test__relocated_names stay: they test what stayed.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 5: Documentation, and the spec's progress record
+
+The user manual tells people to reach `CHUNK_CONTROL` through the netCDF
+loader. That advice now produces a deprecation warning, so it has to change —
+including in two executable blocks, which is the part a reader will actually
+copy.
+
+**Files:**
+- Modify: `docs/src/user_manual/explanation/netcdf_io.rst` (seven references,
+ two of them executed)
+- Modify: `docs/superpowers/specs/2026-09-21-zarr-io-design.md` (§12.1, §12.4,
+ §12.6)
+
+**Interfaces:**
+- Consumes: the public names as Tasks 3 and 4 leave them.
+- Produces: nothing other tasks depend on.
+
+**Not changed here:** `docs/src/whatsnew/3.8.rst` and `3.9.rst` mention
+`iris.fileformats.netcdf.loader.CHUNK_CONTROL`. Those are historical records
+of what shipped in 3.8 and 3.9, where that *was* the name; rewriting them
+would make the record wrong. `nitpicky` is not set in `docs/src/conf.py`, so
+the stale targets render as plain text and do not fail the `-W` build.
+
+- [ ] **Step 1: Repoint the seven references in `netcdf_io.rst`**
+
+Line 9, inside `.. testsetup:: chunk_control`:
+
+```python
+ from iris.fileformats.cf.loader import CHUNK_CONTROL
+```
+
+Line 485, inside the `.. doctest::` block about `_vl_hint`:
+
+```
+ >>> from iris.fileformats.cf.loader import CHUNK_CONTROL
+```
+
+Lines 71, 76, 77, 107 and 123, which are cross-references in prose. Each
+substitutes `iris.fileformats.cf.loader` for `iris.fileformats.netcdf.loader`:
+
+```
+creation of the :data:`iris.fileformats.cf.loader.CHUNK_CONTROL` class.
+```
+
+```
+There are three context managers within :data:`~iris.fileformats.cf.loader.CHUNK_CONTROL`. The most basic is
+:meth:`~iris.fileformats.cf.loader.ChunkControl.set`. This allows you to specify the chunksize for each dimension,
+```
+
+```
+The second context manager is :meth:`~iris.fileformats.cf.loader.ChunkControl.from_file`.
+```
+
+```
+The final context manager, :meth:`~iris.fileformats.cf.loader.ChunkControl.as_dask`, bypasses
+```
+
+Then check:
+
+```bash
+grep -rn "netcdf\.loader" docs/src/user_manual/
+```
+
+Expected: nothing.
+
+- [ ] **Step 2: Add a sentence saying where it went**
+
+The page is explanation, not reference, and a reader who last saw this in 3.16
+will search for the old name. After the line about :pull:`5588` (line 71), add:
+
+```rst
+Chunking policy is not specific to NetCDF, so from Iris 3.17 it lives in
+:mod:`iris.fileformats.cf.loader`. The old name,
+``iris.fileformats.netcdf.loader.CHUNK_CONTROL``, still works and warns.
+```
+
+Keep it to that. The deprecation itself is documented in the module, which is
+where the API reference will render it.
+
+- [ ] **Step 3: Run the doctests**
+
+```bash
+cd docs && make doctest; cd -
+```
+
+Expected: PASS. If a `chunk_control` block fails on *output* rather than on
+import, the cause is in Task 2 or 3, not here — the chunk sizes printed in
+this page are the real ones.
+
+- [ ] **Step 4: Build the docs**
+
+```bash
+cd docs && make html; cd -
+```
+
+Expected: a clean build. `-W` is on, so a cross-reference typo in Step 1 fails
+it. The API pages are generated by `sphinxcontrib.apidoc`, so the new module
+gets a page without any new RST; check that
+`docs/src/_build/html/generated/api/iris.fileformats.cf.loader.html` exists
+and renders `CHUNK_CONTROL`, and that the netCDF loader page shows the
+`.. deprecated:: 3.17` note.
+
+This build takes minutes. `pre-commit run check-docs-page-metadata --all-files`
+answers the page-metadata half of it in milliseconds, and is worth running
+first.
+
+- [ ] **Step 5: Update the spec's progress record**
+
+In `docs/superpowers/specs/2026-09-21-zarr-io-design.md`.
+
+§12.1, the PR 3 row — fill in the state and the link once the pull request
+exists:
+
+```
+| 3 | Relocate the CF loader | In review | [#NNNN](https://github.com/SciTools/iris/pull/NNNN) |
+```
+
+§12.4, appended at the end of the decision log, after the
+`**2026-09-24 — during PR 2**` entry:
+
+```markdown
+**2026-10-01 — during PR 3**
+
+- `load_cubes` stays in `iris.fileformats.netcdf.loader`. Opening a list of
+ file sources is the format's own business, and §4.8 lists
+ `netcdf.load_cubes` among the names that do not move. It imports the four
+ CF-layer functions it drives.
+- `DEBUG` stays too, and the moved `_load_cube_inner` reaches back for it
+ through the module. §4.8 lists it as a name that does not move, and it is a
+ flag users set by assignment: a module `__getattr__` can forward a read but
+ cannot intercept a write (PEP 562 defines no `__setattr__` hook), so
+ forwarding it would have made `netcdf.loader.DEBUG = True` silently stop
+ working. Only `CHUNK_CONTROL` and `ChunkControl` are forwarded.
+- The storage seam is one method, `CFDatasetVariable.read_data(chunking_policy)`,
+ taking a zero-argument callable that the implementation invokes only once
+ the result is known to be lazy. The alternative — hoisting the
+ `CHUNK_CONTROL.from_file()` check into the CF layer, which is where the
+ "is this the cube's data variable" fact lives — would have made small
+ unchunked cubes raise `KeyError` where they load today, because that check
+ sits behind the "small enough to read whole" shortcut.
+- `as_lazy_data` takes its cache key from the caller rather than testing
+ `isinstance(data, NetCDFDataProxy)`. This is the `cache_key=` change §4.4
+ called for, landed here because the CF loader cannot import a netCDF proxy
+ class.
+- `_LAZYVAR_MIN_BYTES` moves with the loader and is repointed at all eight of
+ its call sites rather than aliased. It is private, so no deprecation is
+ owed — and several of those sites *assign* to it, which an alias could not
+ have supported.
+```
+
+§12.6, a new row at the end of the document history table:
+
+```
+| 2026-10-01 | PR 3 built. The CF loader relocated to `iris.fileformats.cf.loader`; `load_cubes`, `NetCDFDataProxy` and `DEBUG` stayed. `CFDatasetVariable.read_data` added as the storage seam; `as_lazy_data` gained `cache_key=`. Five decisions logged. |
+```
+
+- [ ] **Step 6: pre-commit, then commit**
+
+```bash
+pre-commit run --files \
+ docs/src/user_manual/explanation/netcdf_io.rst \
+ docs/superpowers/specs/2026-09-21-zarr-io-design.md
+git add docs/
+git commit -m "$(cat <<'EOF'
+Point the chunking documentation at its new home
+
+The user manual told people to import CHUNK_CONTROL from the netCDF
+loader, which now warns. Two of the seven references are executed, so
+this is a doctest change as well as a prose one.
+
+The 3.8 and 3.9 whatsnew pages keep the old name: they record what
+shipped in those releases, where it was correct.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### Task 6: Open the pull request
+
+**Files:** none. This task produces a pull request.
+
+- [ ] **Step 1: Check the branch is still clean against its target**
+
+```bash
+git fetch upstream
+git log --oneline upstream/brownfield..zarr-io-pr-3 --no-merges | wc -l
+```
+
+If PR 2 (#7303) has merged by now, do the rebase in §2 **before** opening this
+one, so that GitHub shows only this PR's commits. If it has not, open against
+`brownfield` anyway and do the rebase when #7303 lands — §2 covers both.
+
+- [ ] **Step 2: Run the full verification sweep**
+
+Everything in §Verification below, in one sitting, on the final branch state.
+Record the numbers; the PR body quotes them.
+
+- [ ] **Step 3: Open it**
+
+```bash
+gh pr create --base brownfield --head zarr-io-pr-3 \
+ --title "Relocate the CF loader to iris.fileformats.cf.loader" \
+ --label "Agentic" --label "Type: Feature Branch" \
+ --body-file /tmp/pr3-body.md
+```
+
+The body is below. It opens with the model marker, which under the root
+`AGENTS.md` **replaces** any trailing attribution — so there is no footer.
+
+````markdown
+(posted by Claude Opus)
+
+Part of #6977. Third of seven; follows #7298 and #7303, and targets
+`brownfield`, not `main`.
+
+> **If this PR appears to contain #7303's changes, it does not.** This
+> branch was taken from #7303 while that PR was still in review, so until
+> #7303 squash-merges, GitHub computes the diff against `brownfield` and
+> shows #7303's commits here too. This PR's own commits are the ones
+> after `26c4a73c8`. Once #7303 lands, a `rebase --onto` drops the rest
+> and this note comes down.
+
+## What this does
+
+`iris/fileformats/netcdf/loader.py` did three jobs. Two of them —
+driving the CF rules engine, and deciding dask chunk sizes — are not
+netCDF's. They only lived there because netCDF was the only format that
+did them.
+
+This moves those two to a new `iris.fileformats.cf.loader`, with
+`git mv`, so the history follows the code. What stays at the old path is
+what is genuinely netCDF's: `load_cubes`, `NetCDFDataProxy` and `DEBUG`.
+
+Reading a variable's array is the one place where the format does show
+through. That becomes a single abstract method on the storage interface
+#7303 introduced:
+
+```python
+class CFDatasetVariable(ABC):
+ @abstractmethod
+ def read_data(self, chunking_policy): ...
+```
+
+The CF loader supplies `chunking_policy` — a zero-argument callable that
+answers "what chunks, if this comes back lazy" — and the netCDF
+implementation decides everything else. That is the seam the Zarr loader
+plugs into in the next pull request; there is nothing else for it to
+implement.
+
+One supporting change made this possible. `iris._lazy_data.as_lazy_data`
+used to recognise `NetCDFDataProxy` by `isinstance` to decide whether a
+result was cacheable, which meant the bottom of the lazy-data stack
+imported a file format. It now takes a `cache_key=` keyword from its
+caller, and the netCDF loader passes `repr(proxy)` — byte-for-byte the
+key computed before.
+
+## Behaviour changes
+
+None intended, and the existing suite passes unchanged.
+
+The one visible difference: `iris.fileformats.netcdf.loader.CHUNK_CONTROL`
+and `.ChunkControl` now emit an `IrisDeprecation` when read. They still
+resolve, to the *same objects*, so `with CHUNK_CONTROL.set(...)` reached
+through the old path still chunks. The user manual points at the new
+path.
+
+## Five things a reviewer will want stated
+
+1. **`DEBUG` deliberately did not move**, even though its only reader
+ did. It is a flag you set by assignment, and a module `__getattr__`
+ can forward a read but cannot intercept a write — PEP 562 gives
+ modules no `__setattr__`. Forwarding it would have left
+ `netcdf.loader.DEBUG = True` succeeding and doing nothing. So it
+ stays defined where it is and `_load_cube_inner` reads it back
+ through the module. `CHUNK_CONTROL` and `ChunkControl` are objects,
+ used through context managers and never rebound, so forwarding them
+ is safe.
+
+2. **`CHUNK_CONTROL.from_file()` still loads small unchunked cubes.**
+ The `KeyError` it raises for an unchunked data variable sits *behind*
+ the "small enough to read whole" shortcut, so a small contiguous cube
+ loads today. The tidier-looking refactor — hoisting that check up
+ into the CF layer, which is where the "is this the cube's data
+ variable" fact lives — would have broken that. The policy callable
+ exists so the check can stay where it is, and there is now a test
+ that loads exactly such a cube.
+
+3. **Nothing changed about which attributes reach `cube.attributes`.**
+ `read_data` reads `_FillValue`, `scale_factor` and `add_offset` from
+ the storage object's attribute mapping, which does not track reads,
+ where the old code read them from the `CFVariable`'s mapping, which
+ does. Safe for two independent reasons — all four reserved names are
+ pre-seeded as read in `_CF_ATTRS_IGNORE`, *and* filtered by
+ `saver._CF_ATTRS` — and there is a test comparing `cf_attrs_unused()`
+ across a `read_data` call rather than relying on either.
+
+4. **The dask array cache still works.** A cache miss is invisible
+ except as slowness, so there are tests asserting the exact key tuple,
+ and that a second call with the same key does not rebuild the array.
+
+5. **ncdata's emulation hook still works.** An emulated variable (#4994)
+ returns its own array with no proxy built; there are now tests saying
+ so directly, where before it was covered only through a full load.
+
+## How the "no behaviour change" claim was checked
+
+- The full suite, unchanged, before and after: same pass count.
+- `_chunks_from_chunk_control` is tested in all three `ChunkControl`
+ modes, including the two `from_file` cases that differ by whether the
+ variable is the cube's data variable.
+- The cache key is asserted as a literal tuple, not just as "present".
+- The netCDF body moved verbatim, branch order included, and the
+ commits are separable: the move is its own commit, with `git mv`, so
+ `git log --follow` and `git blame` both still work.
+
+## Not here
+
+- No Zarr. The next pull request adds `iris/fileformats/zarr/` and
+ implements `read_data` there.
+- No saver changes. `iris.fileformats.netcdf.saver` moves in PR 5, which
+ is also what dissolves the two deferred imports `cf/loader.py` needs
+ to reach `_CF_ATTRS` and `Saver._DATALESS_ATTRNAME`.
+- No changelog fragment. The programme writes its fragments once, at the
+ merge-back of `brownfield` into `main`, so that users read one
+ coherent story rather than seven partial ones.
+````
+
+- [ ] **Step 4: Check the rendered body**
+
+```bash
+gh pr view --web
+```
+
+Confirm the model marker is the first line, the labels are both on, and the
+issue reference reads `Part of #6977` with no closing keyword.
+
+---
+
+## Verification
+
+Run all of this on the final branch state, before opening the pull request.
+
+```bash
+conda activate iris-dev
+export NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX"
+export OVERRIDE_TEST_DATA_REPOSITORY=/data/home/martin.yeo/repos/iris-test-data/test_data
+
+# 1. The whole suite, with no --ignore.
+pytest lib/iris/tests -n auto -q
+
+# 2. The same count as the baseline. Compare against:
+# git stash && pytest lib/iris/tests -n auto -q && git stash pop
+# or against the figure recorded when Task 1 started.
+
+# 3. Both import orders, because the cycle is real.
+python -c "import iris.fileformats.cf; print('ok')"
+python -c "import iris.fileformats.netcdf; print('ok')"
+
+# 4. _lazy_data no longer knows about any file format.
+grep -rn "fileformats" lib/iris/_lazy_data.py # expect: nothing
+
+# 5. cf/loader.py names no proxy class and imports no netCDF at module scope.
+grep -n "Proxy" lib/iris/fileformats/cf/loader.py # expect: nothing
+grep -n "^from iris.fileformats.netcdf\|^import iris.fileformats.netcdf" \
+ lib/iris/fileformats/cf/loader.py # expect: nothing
+
+# 6. Nothing still reaches for a moved private name at the old path.
+grep -rn "netcdf\.loader\._LAZYVAR_MIN_BYTES\|netcdf\.loader\._get_cf_var_data" \
+ lib/ benchmarks/ # expect: nothing
+
+# 7. Hooks.
+pre-commit run --all-files
+
+# 8. Docs, including the two executed blocks.
+cd docs && make doctest && make html; cd -
+```
+
+---
+
+## Definition of done
+
+- [ ] `iris.fileformats.cf.loader` exists, holds every format-agnostic part of
+ the old netCDF loader, and imports nothing from
+ `iris.fileformats.netcdf` at module scope.
+- [ ] `iris.fileformats.netcdf.loader` holds `DEBUG`, `NetCDFDataProxy`,
+ `load_cubes` and a warn-on-use `__getattr__` for `CHUNK_CONTROL` and
+ `ChunkControl` — and nothing else.
+- [ ] `CFDatasetVariable.read_data` is abstract, has exactly one
+ implementation, and is exercised by the shared contract suite so that
+ PR 4's Zarr variable inherits the cases.
+- [ ] `as_lazy_data` takes `cache_key=`; `iris/_lazy_data.py` contains no
+ reference to any file format.
+- [ ] The five Review Focus items each have a named, passing test.
+- [ ] The full suite passes with no `--ignore`, at the baseline count.
+- [ ] `make doctest` and `make html` both pass; the user manual points at the
+ new path and says where the old one went.
+- [ ] `pre-commit run --all-files` is clean.
+- [ ] The spec's §12.1, §12.4 and §12.6 record what happened.
+- [ ] No `changelog/` fragment, no Zarr, no saver changes, no `AGENTS.md`
+ edits.
+- [ ] The pull request targets `brownfield`, carries `Agentic` and
+ `Type: Feature Branch`, opens with `(posted by Claude Opus)`, and says
+ `Part of #6977` with no closing keyword.
diff --git a/docs/superpowers/plans/2026-10-12-zarr-io-pr2-5.md b/docs/superpowers/plans/2026-10-12-zarr-io-pr2-5.md
new file mode 100644
index 0000000000..cd6dd9185f
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-12-zarr-io-pr2-5.md
@@ -0,0 +1,814 @@
+# Zarr I/O PR 2.5 — Deprecate `CFVariable.__getattr__` Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make `CFVariable.__getattr__` warn under `IrisDeprecation` on every CF
+attribute it resolves, and move the last four reads inside Iris that would
+trigger it onto `CFVariable.attributes`, so a normal load stays silent.
+
+**Architecture:** `CFVariable.__getattr__` already resolves against
+`self.attributes` and falls back to the backing netCDF4 object. This adds a
+`warn_deprecated` call on the `self.attributes` hit path only, leaves the
+fallback's own warning alone, and leaves the miss path silent. Four call sites
+in `iris.fileformats` still read a CF attribute as an attribute; each becomes a
+mapping read, which records the attribute as used exactly as before.
+
+**Tech Stack:** Python, pytest, `iris._deprecation.warn_deprecated`,
+`iris.fileformats.cf.dataset.TrackedAttributes`.
+
+**Spec:** [`docs/superpowers/specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md)
+— §4.3 is the design, §5 "PR 2.5" is the scope, §12.1 is the live record.
+
+## Global Constraints
+
+- Base branch for the pull request is `brownfield` on `SciTools/iris`
+ (`upstream`), never `main`.
+- The branch starts from `zarr-spec-pp-mo-review`, because the spec text
+ describing PR 2.5 lives there and is still in review as
+ [#7330](https://github.com/SciTools/iris/pull/7330). Rebase onto `brownfield`
+ once #7330 merges, before opening the pull request.
+- No changelog fragment. Fragments land with the merge-back (spec §5).
+- The pull request body says `Part of #6977`, never `Closes`.
+- Pull request labels: `Agentic`, `Type: Feature Branch`.
+- Every GitHub post opens with `(posted by Claude Opus)` on its own line.
+- Deprecation directive is `.. deprecated:: 3.17`. Removal follows NEP 29, one
+ cycle later.
+- `IrisDeprecation` and `warn_deprecated` both live in `iris._deprecation`.
+ They are **not** re-exported from `iris.exceptions` or `iris.warnings`, so a
+ `-W` filter naming `iris.exceptions.IrisDeprecation` silently matches
+ nothing.
+- Ruff, 88-character lines, NumPy docstrings. `pre-commit run --files <paths>`
+ must pass with auto-fixes re-staged.
+- Do **not** edit `lib/iris/AGENTS.md`. Its `__getattr__` carve-out describes
+ shipped code and is revisited when the removal lands, not now (spec §4.3).
+- Nothing in this branch may change which attributes reach a loaded cube.
+ `TrackedAttributes` records a read from `__getitem__`, `get` and
+ `__contains__`, so each conversion below preserves tracking — but the tests
+ have to say so.
+
+## Review Focus
+
+- **A UGRID connectivity variable with no `start_index`.** CF makes it
+ optional and `iris.mesh.Connectivity` defaults it to 0, but
+ `_build_connectivity` reads it unconditionally. The mesh then fails to
+ build, the cube loads without one, and the failure is recorded in
+ `iris.loading.LOAD_PROBLEMS` behind an `IrisLoadWarning` — verified by
+ deleting the attribute from a saved file. This change alters the exception
+ recorded there from `AttributeError` to `KeyError` and nothing a user sees.
+ Task 2 pins the user-visible behaviour and opens an issue for the
+ underlying bug rather than fixing it here.
+- **`getattr(cf_var, "nonesuch", None)` and `hasattr(cf_var, "nonesuch")` from
+ outside Iris.** A miss must stay silent and keep returning the default, or
+ the deprecation turns a working idiom into noise on every probe. Task 1.
+- **Two reads of the same attribute.** The `setattr` cache was removed in
+ [#7303](https://github.com/SciTools/iris/pull/7303), so each read warns.
+ A test that reads twice and expects one warning would pass only because of
+ `__warningregistry__`. Task 1 asserts two.
+- **`copy`, `pickle` and numpy probing dunders on a `CFVariable`.** The dunder
+ guard runs before the mapping read, so none of them may warn. Task 1.
+- **The attributes that reach a loaded cube.** `cf_attrs_unused()` decides
+ this, and all four conversions in Task 2 move a read from `__getattr__` to
+ the mapping. Task 2 asserts that `bounds`, `climatology`, `cf_role` and
+ `start_index` still do not land on the loaded coordinate or connectivity.
+
+---
+
+## File Structure
+
+| File | Responsibility |
+|---|---|
+| `lib/iris/fileformats/cf/_variables.py` | `CFVariable.__getattr__` gains the warning and the `.. deprecated:: 3.17` directive. |
+| `lib/iris/fileformats/_nc_load_rules/helpers.py` | `get_cf_bounds_var` reads `bounds` and `climatology` from `.attributes`. |
+| `lib/iris/fileformats/netcdf/ugrid_load.py` | `_build_connectivity` reads `cf_role` and `start_index` from `.attributes`. |
+| `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py` | Unit coverage for when the warning fires, when it does not, and where it points. |
+| `lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_cf_bounds_var.py` | Rewritten to set `attributes` instead of setting attributes on a mock. |
+| `lib/iris/tests/integration/test_cf.py` | The end-to-end guarantee: a normal load emits no `IrisDeprecation`. |
+| `docs/superpowers/specs/2026-09-21-zarr-io-design.md` | §12.1 state and §12.6 history, updated when the pull request opens. |
+
+**Verified, so you do not have to rediscover it.** Instrumenting
+`CFVariable.__getattr__` to log its caller and running the full suite
+(`12013 passed, 66 skipped`) found exactly four call sites inside
+`lib/iris` that resolve a CF attribute through it:
+
+| Site | Attribute |
+|---|---|
+| `_nc_load_rules/helpers.py`, `get_cf_bounds_var` | `bounds` |
+| `_nc_load_rules/helpers.py`, `get_cf_bounds_var` | `climatology` |
+| `netcdf/ugrid_load.py`, `_build_connectivity` | `cf_role` |
+| `netcdf/ugrid_load.py`, `_build_connectivity` | `start_index` |
+
+Twenty-four further sites are in the test tree, all of them in
+`unit/fileformats/cf/` and `integration/test_cf.py`, and all of them
+deliberately exercising `__getattr__`. `filterwarnings = ["default"]` in
+`pyproject.toml` and no `-W error` anywhere in CI, so those tests warn and
+still pass. Leave them alone.
+
+---
+
+## Task 0: Branch
+
+**Files:** none.
+
+- [ ] **Step 1: Confirm the starting point**
+
+Run:
+
+```bash
+cd /data/home/martin.yeo/repos/iris
+git status --short
+git log --oneline -1
+```
+
+Expected: the working tree holds no changes to `lib/`, and `HEAD` is
+`Action the #7292 spec review by @pp-mo`. An untracked `codex-bug.txt` is
+pre-existing and is not yours to touch or commit.
+
+- [ ] **Step 2: Create the branch**
+
+```bash
+git switch -c zarr-io-pr-2-5
+```
+
+---
+
+## Task 1: `__getattr__` warns
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/_variables.py`
+- Test: `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py`
+
+**Interfaces:**
+- Consumes: `CFVariable.attributes`, a
+ `iris.fileformats.cf.dataset.TrackedAttributes` built in `CFVariable.__init__`;
+ `CFVariable._deprecated_netcdf_member`, already present.
+- Produces: no new names. `CFVariable.__getattr__` keeps its signature and
+ return value, and gains an `IrisDeprecation` on the CF attribute path.
+
+- [ ] **Step 1: Write the failing tests**
+
+Add this class to `lib/iris/tests/unit/fileformats/cf/test_CFVariable.py`,
+after `TestGetattrAndHasattr`. The module already defines the `nc_var` fixture
+and `CFVariableSub`; use them.
+
+Add to the module's imports, in Ruff's order:
+
+```python
+import copy
+import warnings
+
+import numpy as np
+import pytest
+
+from iris._deprecation import IrisDeprecation
+from iris.fileformats import cf as cf
+from iris.fileformats.cf import _variables
+from iris.fileformats.cf.dataset import TrackedAttributes
+```
+
+The new class:
+
+```python
+class TestDeprecatedAttributeAccess:
+ """Reading a CF attribute as an attribute warns; everything else is silent.
+
+ cf_var.units and cf_var.attributes["units"] are the same read. Only the
+ spelling that cannot be type-checked is deprecated. Spec section 4.3.
+ """
+
+ def test_reading_a_cf_attribute_warns(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(IrisDeprecation) as record:
+ assert cf_var.coordinates == "x y"
+
+ assert len(record) == 1
+
+ def test_the_warning_names_the_mapping_and_the_key(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(
+ IrisDeprecation, match=r"cf_var\.attributes\['coordinates'\]"
+ ):
+ _ = cf_var.coordinates
+
+ def test_the_warning_points_at_the_caller(self, nc_var):
+ # stacklevel: a warning reported against _variables.py tells the
+ # reader nothing about which of their lines to change.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(IrisDeprecation) as record:
+ _ = cf_var.coordinates
+
+ assert record[0].filename == __file__
+
+ def test_a_warned_read_is_still_recorded(self, nc_var):
+ # The warning is additional to the read, not instead of it. What was
+ # read decides what reaches the cube.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(IrisDeprecation):
+ _ = cf_var.coordinates
+
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+ assert "coordinates" not in dict(cf_var.cf_attrs_unused())
+
+ def test_every_read_warns(self, nc_var):
+ # There is no instance cache to suppress the second one, and a test
+ # expecting one warning would pass only through __warningregistry__.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(IrisDeprecation) as record:
+ _ = cf_var.coordinates
+ _ = cf_var.coordinates
+
+ assert len(record) == 2
+
+ def test_the_mapping_does_not_warn(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert cf_var.attributes["coordinates"] == "x y"
+
+ def test_a_declared_property_does_not_warn(self, nc_var):
+ # A declared property short-circuits __getattr__ entirely.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert cf_var.dimensions == ("time", "lat")
+ assert cf_var.cf_name == "foo"
+
+ def test_a_missing_attribute_does_not_warn(self, nc_var):
+ # getattr(cf_var, "bounds", None) is a deliberate idiom throughout the
+ # loading rules. A miss must stay silent and keep its default.
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert getattr(cf_var, "nonesuch", None) is None
+ assert not hasattr(cf_var, "nonesuch")
+
+ def test_hasattr_warns_on_a_hit(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(IrisDeprecation):
+ assert hasattr(cf_var, "coordinates")
+
+ def test_dunder_probes_do_not_warn(self, nc_var):
+ # copy, pickle and numpy all probe for dunders, and the file here
+ # really does carry one.
+ nc_var.attributes = {"__array__": "nonsense"}
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert not hasattr(cf_var, "__array_interface__")
+ assert not hasattr(cf_var, "__array__")
+ _ = copy.copy(cf_var)
+
+ def test_the_netcdf_fallback_keeps_its_own_message(self, nc_var):
+ # Reaching a netCDF4 member is a different deprecation, worded for a
+ # different fix, and issued by the storage object.
+ nc_var.deprecated_netcdf_member.return_value = 42
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert cf_var.not_an_ncattr == 42
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run:
+
+```bash
+NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX" \
+ pytest lib/iris/tests/unit/fileformats/cf/test_CFVariable.py \
+ -k TestDeprecatedAttributeAccess -v
+```
+
+Expected: the four tests that expect a warning fail with
+`DID NOT WARN`. The silence tests pass already — that is correct, they are
+guards against the implementation over-reaching, not drivers for it.
+
+- [ ] **Step 3: Add the warning**
+
+In `lib/iris/fileformats/cf/_variables.py`, add the import alongside the other
+`iris` imports:
+
+```python
+from iris._deprecation import warn_deprecated
+```
+
+Replace the body of `__getattr__` below the recursion guard:
+
+```python
+ try:
+ value = self.attributes[name]
+ except KeyError:
+ pass
+ else:
+ # stacklevel=3 reports the caller's own "cf_var.units" line:
+ # warn_deprecated's frame is 1 and this method's is 2.
+ warn_deprecated(
+ f"Reading the CF attribute {name!r} as an attribute of a "
+ "CFVariable is deprecated and will be removed in a future "
+ f"release. Use cf_var.attributes[{name!r}] instead.",
+ stacklevel=3,
+ )
+ return value
+ return self._deprecated_netcdf_member(name)
+```
+
+Add the directive to the `__getattr__` docstring, after the existing
+paragraph and before the closing quotes:
+
+```python
+ .. deprecated:: 3.17
+ Read CF attributes from :attr:`attributes` instead. Every CF
+ attribute resolved through this method warns, and the method is
+ removed one release cycle later under NEP 29. :attr:`attributes`
+ is also unambiguous: a file carrying an attribute named after a
+ class member cannot reach it through this method at all.
+
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run:
+
+```bash
+NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX" \
+ pytest lib/iris/tests/unit/fileformats/cf/ -v
+```
+
+Expected: PASS, no failures. Existing tests in this file read CF attributes as
+attributes and now warn; `filterwarnings = ["default"]` means they still pass.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add lib/iris/fileformats/cf/_variables.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFVariable.py
+git commit -m "$(cat <<'EOF'
+Deprecate reading CF attributes as CFVariable attributes
+
+cf_var.units and cf_var.attributes["units"] are the same read, with the
+same effect on which attributes reach the cube. __getattr__ now warns on
+the spelling that cannot be type-checked, and that any class member of the
+same name silently shadows.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+## Task 2: Move the last four reads inside Iris onto `.attributes`
+
+**Files:**
+- Modify: `lib/iris/fileformats/_nc_load_rules/helpers.py` (`get_cf_bounds_var`)
+- Modify: `lib/iris/fileformats/netcdf/ugrid_load.py` (`_build_connectivity`)
+- Test: `lib/iris/tests/integration/test_cf.py`
+- Test: `lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_cf_bounds_var.py`
+
+**Interfaces:**
+- Consumes: `CFVariable.attributes` (a `TrackedAttributes`), and the warning
+ added in Task 1.
+- Produces: no new names. `get_cf_bounds_var(cf_coord_var)` keeps returning
+ `(bounds_var, climatological)`. `_build_connectivity(connectivity_var,
+ element_dims)` keeps returning `(connectivity, first_dim_name)`.
+
+- [ ] **Step 1: Write the failing integration test**
+
+Add to the imports at the top of `lib/iris/tests/integration/test_cf.py`:
+
+```python
+from typing import Iterable
+import warnings
+
+import pytest
+
+import iris
+from iris._deprecation import IrisDeprecation
+import iris.fileformats.cf as cf
+from iris.fileformats.netcdf import DECODE_TO_STRINGS_ON_READ
+from iris.fileformats.netcdf.loader import _get_cf_var_data
+from iris.tests import _shared_utils
+import iris.tests.stock as stock
+from iris.tests.stock.mesh import sample_mesh_cube
+```
+
+Add this class at the end of the module. It is deliberately not decorated with
+`@_shared_utils.skip_data`: every file it reads it writes first, so it runs
+without `iris-test-data`.
+
+```python
+class TestLoadingIsSilent:
+ """A normal load must not reach CFVariable.__getattr__.
+
+ Every CF attribute read inside Iris goes through CFVariable.attributes,
+ so the deprecation on __getattr__ fires only for code outside Iris.
+ Spec section 4.3.
+
+ The three cubes between them cover the four attributes that reached
+ __getattr__ when the deprecation was added: "bounds" and "climatology"
+ from the loading rules, "cf_role" and "start_index" from the UGRID
+ loader.
+ """
+
+ CUBES = {
+ "bounded": lambda: stock.simple_2d(with_bounds=True),
+ "mesh": sample_mesh_cube,
+ "climatology": lambda: stock.climatology_3d(),
+ }
+
+ @pytest.fixture(params=sorted(CUBES))
+ def saved_file(self, request, tmp_path):
+ path = tmp_path / f"{request.param}.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(self.CUBES[request.param](), path)
+ return path
+
+ def test_load_emits_no_deprecation(self, saved_file):
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ cubes = iris.load(saved_file)
+
+ assert len(cubes) == 1
+
+ def test_a_cf_attribute_read_from_outside_iris_still_warns(self, saved_file):
+ # The other half of the deprecation: silence on a load must not mean
+ # the accessor has stopped working. All three cubes carry a long_name,
+ # and CFVariable declares no property of that name.
+ with cf.CFReader(str(saved_file)) as reader:
+ (data_var,) = reader.cf_group.data_variables.values()
+ with pytest.warns(IrisDeprecation):
+ assert data_var.long_name == data_var.attributes.untracked["long_name"]
+
+ @pytest.mark.parametrize(
+ "cube_name, coord_name",
+ [("bounded", "foo"), ("climatology", "time")],
+ )
+ def test_a_cf_reserved_attribute_does_not_reach_the_coord(
+ self, tmp_path, cube_name, coord_name
+ ):
+ # "bounds" and "climatology" are consumed by the loading rules.
+ # Attribute tracking is what keeps them off the coordinate, and this
+ # pins that reading them through the mapping records them as the
+ # attribute spelling did.
+ path = tmp_path / f"{cube_name}.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(self.CUBES[cube_name](), path)
+
+ cube = iris.load_cube(path)
+
+ assert cube.coord(coord_name).attributes == {}
+
+ def test_a_cf_reserved_attribute_does_not_reach_the_connectivity(
+ self, tmp_path
+ ):
+ # "cf_role" and "start_index" are consumed by the UGRID loader.
+ path = tmp_path / "mesh.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(sample_mesh_cube(), path)
+
+ cube = iris.load_cube(path)
+
+ assert cube.mesh.face_node_connectivity.attributes == {}
+
+ def test_a_connectivity_without_start_index_loses_its_mesh(self, tmp_path):
+ """A UGRID file omitting the optional "start_index" loads without a mesh.
+
+ CF-UGRID makes start_index optional with a default of 0, and
+ iris.mesh.Connectivity defaults it to 0, but the loader requires it.
+ The mesh build fails and is recorded in iris.loading.LOAD_PROBLEMS.
+ Pinned because this reads start_index through the attribute mapping
+ now - see the issue linked from this test's pull request.
+ """
+ path = tmp_path / "mesh.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(sample_mesh_cube(), path)
+
+ dataset = _thread_safe_nc.DatasetWrapper(str(path), "a")
+ try:
+ for variable in dataset.variables.values():
+ if "start_index" in variable.ncattrs():
+ variable.delncattr("start_index")
+ finally:
+ dataset.close()
+
+ with pytest.warns(IrisLoadWarning):
+ cube = iris.load_cube(path)
+
+ assert cube.mesh is None
+```
+
+That last test needs two more imports. `_thread_safe_nc` is the only
+sanctioned route to netCDF4 (root `AGENTS.md`):
+
+```python
+from iris.fileformats.netcdf import _thread_safe_nc
+from iris.warnings import IrisLoadWarning
+```
+
+It passes before the conversion as well as after, like the tracking tests
+above. It is a guard on behaviour a reviewer will ask about, not a driver.
+
+- [ ] **Step 2: Run the integration test to verify it fails**
+
+Run:
+
+```bash
+NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX" \
+ pytest lib/iris/tests/integration/test_cf.py -k TestLoadingIsSilent -v
+```
+
+Expected: `test_load_emits_no_deprecation` FAILS for all three parameters,
+raising `IrisDeprecation: Reading the CF attribute 'bounds' ...` for `bounded`
+and `climatology`, and `'cf_role'` for `mesh`. Every other test in the class
+passes already — they guard behaviour the conversion must not disturb.
+
+- [ ] **Step 3: Convert `get_cf_bounds_var`**
+
+In `lib/iris/fileformats/_nc_load_rules/helpers.py`, replace the first two
+lines of `get_cf_bounds_var`:
+
+```python
+def get_cf_bounds_var(cf_coord_var):
+ """Return the CF variable representing the bounds of a coordinate variable."""
+ attributes = cf_coord_var.attributes
+ attr_bounds = attributes.get(CF_ATTR_BOUNDS)
+ attr_climatology = attributes.get(CF_ATTR_CLIMATOLOGY)
+```
+
+`TrackedAttributes.get` records a hit and records nothing on a miss, which is
+what `getattr(..., None)` did.
+
+- [ ] **Step 4: Convert `_build_connectivity`**
+
+In `lib/iris/fileformats/netcdf/ugrid_load.py`, replace the two reads in
+`_build_connectivity`:
+
+```python
+ # "cf_role" is how a connectivity variable is identified in the first
+ # place, so it is always present. "start_index" is optional in CF and the
+ # loader has always required it.
+ cf_role = connectivity_var.attributes["cf_role"]
+ start_index = connectivity_var.attributes["start_index"]
+```
+
+- [ ] **Step 5: Fix `test_get_cf_bounds_var.py`**
+
+The existing test builds its subject with
+`setattr(cf_coord_var, attr, attr_val)` on a `MagicMock`. After Step 3 that
+sets nothing the function reads, and `mock.attributes.get(...)` returns a
+truthy `Mock`, so both branches take the bounds path. Replace `_generic_test`:
+
+```python
+ def _generic_test(self, test_climatological_bounds=False):
+ cf_coord_var = self.mocker.MagicMock()
+
+ cf_group_dict = {"TEST": self.mocker.sentinel.bounds_var}
+ if test_climatological_bounds:
+ cf_coord_var.cf_group.climatology = cf_group_dict
+ test_attr = CF_ATTR_CLIMATOLOGY
+ else:
+ cf_coord_var.cf_group.bounds = cf_group_dict
+ test_attr = CF_ATTR_BOUNDS
+
+ cf_coord_var.attributes = {test_attr: "TEST"}
+
+ bounds_var, climatological = get_cf_bounds_var(cf_coord_var)
+ assert bounds_var is self.mocker.sentinel.bounds_var
+ assert climatological == test_climatological_bounds
+```
+
+Add a third test to the same class, pinning the preference the function
+documents:
+
+```python
+ def test_bounds_wins_over_climatology(self):
+ cf_coord_var = self.mocker.MagicMock()
+ cf_coord_var.cf_group.bounds = {"B": self.mocker.sentinel.bounds_var}
+ cf_coord_var.cf_group.climatology = {"C": self.mocker.sentinel.clim_var}
+ cf_coord_var.attributes = {
+ CF_ATTR_BOUNDS: "B",
+ CF_ATTR_CLIMATOLOGY: "C",
+ }
+
+ bounds_var, climatological = get_cf_bounds_var(cf_coord_var)
+
+ assert bounds_var is self.mocker.sentinel.bounds_var
+ assert climatological is False
+```
+
+- [ ] **Step 6: Run the tests to verify they pass**
+
+Run:
+
+```bash
+NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX" \
+ pytest lib/iris/tests/integration/test_cf.py \
+ lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_cf_bounds_var.py -v
+```
+
+Expected: PASS.
+
+- [ ] **Step 7: Prove nothing else inside Iris reaches `__getattr__`**
+
+This is the measurement that makes the end-to-end claim, rather than
+re-asserting the three cubes above. Run the whole suite with the new
+deprecation promoted to an error, and read every failure.
+
+```bash
+NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX" \
+OVERRIDE_TEST_DATA_REPOSITORY=/data/home/martin.yeo/repos/iris-test-data/test_data \
+ pytest lib/iris/tests -q -n 4 \
+ -W "error:Reading the CF attribute:iris._deprecation.IrisDeprecation" \
+ 2>&1 | grep -E "^(FAILED|ERROR)|passed|failed|skipped"
+```
+
+Expected: every remaining failure is in `lib/iris/tests/`, in
+`unit/fileformats/cf/` or `integration/test_cf.py`, and its traceback bottoms
+out in the test file rather than in `lib/iris/fileformats/`. Those tests
+exercise `__getattr__` on purpose; leave them. **If a traceback bottoms out
+anywhere under `lib/iris/fileformats/`, that is a fifth call site: convert it
+to `.attributes` and re-run before continuing.**
+
+Note the skip count and compare it with a run on the merge base. A skip is
+neither a pass nor a failure, so a failure diff cannot see coverage that
+stopped running (`lib/iris/tests/AGENTS.md`).
+
+- [ ] **Step 8: Run the full suite normally**
+
+```bash
+NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX" \
+OVERRIDE_TEST_DATA_REPOSITORY=/data/home/martin.yeo/repos/iris-test-data/test_data \
+ pytest lib/iris/tests -q -n 4 2>&1 | tail -5
+```
+
+Expected: no failures. On the merge base this reported
+`12013 passed, 66 skipped`; the count rises by the tests this branch adds.
+
+- [ ] **Step 9: Commit**
+
+```bash
+git add lib/iris/fileformats/_nc_load_rules/helpers.py \
+ lib/iris/fileformats/netcdf/ugrid_load.py \
+ lib/iris/tests/integration/test_cf.py \
+ lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_cf_bounds_var.py
+git commit -m "$(cat <<'EOF'
+Read the last four CF attributes through CFVariable.attributes
+
+get_cf_bounds_var and _build_connectivity were the only code in Iris still
+resolving a CF attribute through CFVariable.__getattr__, so a normal load
+emitted the deprecation added for callers outside Iris. Attribute tracking
+is unchanged: TrackedAttributes records a read from get() and from
+__getitem__ exactly as it did from __getattr__.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+## Task 3: Finish and open the pull request
+
+**Files:**
+- Modify: `docs/superpowers/specs/2026-09-21-zarr-io-design.md`
+
+- [ ] **Step 1: Run pre-commit over everything you touched**
+
+```bash
+pre-commit run --files \
+ lib/iris/fileformats/cf/_variables.py \
+ lib/iris/fileformats/_nc_load_rules/helpers.py \
+ lib/iris/fileformats/netcdf/ugrid_load.py \
+ lib/iris/tests/unit/fileformats/cf/test_CFVariable.py \
+ lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_cf_bounds_var.py \
+ lib/iris/tests/integration/test_cf.py
+```
+
+Hooks rewrite in place. Re-stage and re-run until it passes clean, then amend
+the commit the change belongs to.
+
+- [ ] **Step 2: Build the docstring you changed**
+
+`__getattr__`'s docstring gained a `.. deprecated::` directive, and nothing in
+pytest renders it.
+
+```bash
+cd docs/src && make doctest
+```
+
+Expected: no new warnings naming `iris.fileformats.cf`. This takes minutes;
+run it once.
+
+- [ ] **Step 3: Raise the `_build_connectivity` issue**
+
+`_build_connectivity` requires two attributes it never checks for, and loses
+the mesh when either is absent. The guard test in Task 2 records the
+behaviour; it does not fix it, and fixing it here would make a behaviour-only
+deprecation into a bug fix.
+
+```bash
+gh issue create --repo SciTools/iris \
+ --title "UGRID load loses the mesh when a connectivity variable lacks cf_role or start_index" \
+ --label "Type: Bug"
+```
+
+Body, opening with the model marker and with no hard wrapping:
+
+```
+(posted by Claude Opus)
+
+`iris.fileformats.netcdf.ugrid_load._build_connectivity` reads `cf_role` and `start_index` straight off the connectivity variable, and a file missing either one loses its mesh: the mesh build raises, the failure lands in `iris.loading.LOAD_PROBLEMS`, and the user gets an `IrisLoadWarning` and a mesh-less cube.
+
+Neither read is guarded, and nothing upstream guarantees either attribute. `CFUGridConnectivityVariable.identify` finds the connectivity variable through the *mesh* variable's role attribute — `face_node_connectivity` and friends — so it never looks at the connectivity variable's own `cf_role`.
+
+`start_index` is the clearer bug of the two: CF-UGRID makes it optional with a default of 0, and `iris.mesh.Connectivity` already defaults it to 0, so the loader should fall back rather than raise. `cf_role` is required by UGRID, so raising is defensible, but the message a user gets does not say which attribute was missing.
+
+Reproduced by saving `iris.tests.stock.mesh.sample_mesh_cube`, deleting `start_index` from the connectivity variables, and reloading.
+
+Found while moving the last internal CF attribute reads off `CFVariable.__getattr__`. Not fixed there: that branch is behaviour-only apart from the deprecation it adds. The current behaviour is pinned by a test in `lib/iris/tests/integration/test_cf.py`, which will need inverting when this is fixed.
+```
+
+- [ ] **Step 4: Rebase onto `brownfield`**
+
+Only once [#7330](https://github.com/SciTools/iris/pull/7330) has merged, so
+that the spec text describing PR 2.5 is on the base branch and the diff is
+your two commits and nothing else.
+
+```bash
+git fetch upstream
+git rebase upstream/brownfield
+```
+
+Expected: no conflicts. `_variables.py`, `helpers.py` and `ugrid_load.py` are
+untouched by #7330. If #7330 has not merged, stop here and say so — opening
+early puts the spec commit in your diff.
+
+- [ ] **Step 5: Update the spec's live record**
+
+In §12.1, change PR 2.5's row to `Drafted` and fill in the link once the pull
+request number exists. Add a row to §12.6 dated the day you open it, naming
+what changed: the deprecation landed, the four remaining internal reads moved
+to `.attributes`, and the `start_index` bug was raised rather than fixed.
+
+Commit this separately:
+
+```bash
+git add docs/superpowers/specs/2026-09-21-zarr-io-design.md
+git commit -m "$(cat <<'EOF'
+Record PR 2.5 in the Zarr I/O design document
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+- [ ] **Step 6: Push and open the pull request**
+
+```bash
+git push -u origin zarr-io-pr-2-5
+```
+
+Open against `brownfield`, with the `Agentic` and `Type: Feature Branch`
+labels. The body opens with `(posted by Claude Opus)`, says `Part of #6977`,
+states that the changelog fragment is deferred to the merge-back, and names
+where you would look first: the `start_index` exception type changing from
+`AttributeError` to `KeyError`, and whether the four converted reads are
+really all of them. No hard wrapping. Do not list test names.
+
+---
+
+## Self-Review
+
+**Spec coverage.** §5's PR 2.5 paragraph asks for four things. `__getattr__`
+keeps resolving through `.attributes` and warns, naming `.attributes` and the
+key — Task 1. `IrisDeprecation` and `.. deprecated:: 3.17` — Task 1. A normal
+load stays silent — Task 2, which also removes the four reads that would
+otherwise break that claim. A test that reaches `cf_var.units` directly and
+does warn — Task 1 and Task 2's
+`test_a_cf_attribute_read_from_outside_iris_still_warns`. §4.3's other
+requirements were delivered by [#7303](https://github.com/SciTools/iris/pull/7303)
+and are not re-implemented here: one implementation on the base class, no
+`setattr` caching, `AttributeError` on a miss, the netCDF fallback. Each has an
+existing test, and Task 1 adds silence guards around the new warning so they
+stay true.
+
+**The spec is wrong in one place, and this plan does not follow it.** §5 gives
+the load filter as `-W error::iris.exceptions.IrisDeprecation`.
+`IrisDeprecation` lives in `iris._deprecation` and is re-exported from
+neither `iris.exceptions` nor `iris.warnings`, so that filter matches nothing
+and the test would pass against any implementation. Propose the correction to
+§5 when the branch is reviewed.
+
+**The spec is also optimistic in one place.** §5 says every read inside Iris
+already goes through `.attributes` after PR 2. Four did not, which is why
+Task 2 exists rather than being a one-line postscript to Task 1.
diff --git a/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md
new file mode 100644
index 0000000000..395922e229
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-13-zarr-io-pr4.md
@@ -0,0 +1,6723 @@
+# PR 4 — Zarr loading
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use
+> `superpowers:subagent-driven-development` (recommended) or
+> `superpowers:executing-plans` to implement this plan task-by-task. Steps use
+> checkbox (`- [ ]`) syntax for tracking.
+
+> **This plan ships with its implementation.** Plan and code are delivered on
+> the same branch and in the same pull request. If implementation proves a step
+> wrong, correct the step here on the branch — the document is frozen when the
+> work merges, not while it is under way. Programme status lives in §12 of the
+> [design spec](../specs/2026-09-21-zarr-io-design.md); nothing here duplicates
+> it.
+
+| | |
+|---|---|
+| **Spec** | [`../specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md) §2.4, §3, §4.1, §4.2, §4.4, §4.6, §4.7, §5 (PR 4), §6 |
+| **Implements** | The fourth of the eight pull requests in spec §5. The subject of [#6979](https://github.com/SciTools/iris/issues/6979) |
+| **Branch** | `zarr-io-pr-4`, cut from `zarr-io-pr-3`. A **preview** pull request opens on the fork, `trexfeathers/iris`, with `zarr-io-pr-3` as its base, so that its diff is PR 4's changes and nothing else. The real pull request opens against `SciTools/iris:brownfield` once PR 3 merges (Task 15, Step 6) |
+| **Builds on** | **PR 3** ([#7316](https://github.com/SciTools/iris/pull/7316)), which is this branch's **base**, not a prerequisite waited on: this plan is written against `iris.fileformats.cf.loader`, `CFDatasetVariable.read_data` and the `cache_key=` keyword on `as_lazy_data`, none of which are on `brownfield` yet |
+| **Baseline** | `zarr-io-pr-3` at `231cd019e`, which carries `brownfield` at `40d377d26` — where the spec review of [#7330](https://github.com/SciTools/iris/pull/7330) merged — plus PR 3's own ten commits; Iris `3.17.0.dev`, zarr-python `3.4.1`, dask `2025.x`, NumPy `2.5.2` |
+| **Delivers** | `iris.fileformats.zarr` — `_dataset.py`, `_decode.py`, `loader.py` — the `"Zarr"` format specification, `iris.io.load_stores`, and `iris.load("store.zarr")` working for version 2 and version 3 stores, local and remote |
+| **Behaviour change** | Three, all previously errors: a local `.zarr` directory and an `s3`/`gs`/`az`/`abfs` URL now load instead of raising; `CFVariable.spans` matches on `(name, size)` pairs rather than bare names, which is a widening no netCDF file can observe |
+| **Public API change** | Additive: the new public module `iris.fileformats.zarr` (`load_cubes`), `iris.io.load_stores`, `CFDataset.format`, `CFDataset.encoding` and `CFDatasetVariable.encoding` |
+
+**Created:** 2026-10-13
+
+Claims marked **[verified]** were checked by running code in the `iris-dev`
+environment against zarr-python 3.4.1 while this plan was written; everything
+else is read from source or from the spec.
+
+**Goal:** Make `iris.load("store.zarr")` and
+`iris.load("s3://bucket/store.zarr")` return cubes from a native Zarr version 2
+or version 3 store, by implementing the `CFDataset` interface over zarr-python
+and teaching Iris's load chain that a target can be a directory or a URL rather
+than a seekable file.
+
+**Architecture:** `iris/fileformats/zarr/_dataset.py` implements `CFDataset` and
+`CFDatasetVariable` over a `zarr.Group` and its `zarr.Array`s, which is all the
+CF layer from PR 2 and PR 3 needs in order to classify variables and build
+cubes. There is no data proxy: `zarr.Array` is picklable, so `read_data` hands
+it straight to `as_lazy_data` and adds CF decoding — masking, `valid_range`,
+`scale_factor`/`add_offset` — as explicit `map_blocks` layers over the raw
+array, implemented as pure functions in `_decode.py`. `iris/fileformats/zarr/
+loader.py` is a thin `load_cubes` that opens a `ZarrDataset`, hands it to
+`CFReader`, and runs the same `_load_cube` loop the netCDF loader runs.
+`iris.io.load_stores` is the new entry point for a target that is not a file,
+and a `ZarrStore` file element lets `FORMAT_AGENT` recognise one.
+
+**Tech Stack:** Python 3.12–3.14, zarr-python ≥3.0.8, NumPy, Dask, pytest +
+pytest-xdist, pre-commit (ruff, numpydoc, mypy, sort-all), `abc.ABC`,
+`collections.abc.MutableMapping`.
+
+**Spec:** [`../specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md)
+— §2.4 lists the Zarr facts that shape the design, §3 the constraints agreed
+before design, §4.1 the module layout, §4.2 the `CFDataset` interface, §4.4 the
+whole of reading, §4.6 groups, §4.7 format detection and the load chain, §5's
+fourth entry is this pull request, §6 the testing rules. The plan argues from
+the spec; read both.
+
+---
+
+## Global Constraints
+
+Project-wide requirements every task below implicitly carries. Values are
+copied from the spec and the repository's agent rules, not restated from
+memory.
+
+| Constraint | Value | Source |
+|---|---|---|
+| Target branch | `brownfield`. **Not** `main`. Reached by rebase once PR 3 merges; until then the branch sits on `zarr-io-pr-3` and the preview pull request is based there | spec §5 |
+| Labels | `Agentic` and `Type: Feature Branch` | spec §5 |
+| Issue linkage | `Part of #6979` — never a closing keyword; those live on the merge-back | spec §5 |
+| Changelog | Fragments land with the merge-back; the pull request body must say the omission is deliberate | spec §5 |
+| Dependency | `zarr` is **optional** and lazily imported. "Absent, Zarr tests skip and Zarr use raises a clear `ImportError`" | spec §3 |
+| Version pin | `zarr >=3.0.8`, "which is above the known data-loss bug" | spec §8 |
+| Read support | **Version 2 and version 3** | spec §3 |
+| Write support | Not here. The `ZarrDataset` write surface is PR 6 | spec §5 |
+| Remote stores | "URLs are passed through to zarr-python. Iris adds no dependency and handles no credentials" — no fsspec, s3fs or gcsfs in Iris's requirements | spec §3, §9 |
+| Groups | "Root group by default; a `group=` keyword selects another"; Iris loads **one** group per call and does not walk nested groups | spec §3, §4.6 |
+| No implementation-type tests | "No code above `cf/dataset.py` tests the implementation type. No `isinstance`, no checking a class name, no `hasattr` probe standing in for one" | spec §4.2 |
+| `format` is not a branch key | "Where the generic layer needs a capability, it asks for the capability, not the format. `format` is for messages, diagnostics and the user" | spec §4.2 |
+| Dimension contract | A dimension name may be absent, null, repeated within an array, or shared across arrays at different lengths. "All four are conforming Zarr. None may raise" | spec §4.4 |
+| Masking precedence | Version 3: CF attributes only. Version 2: `Array.fill_value` when no CF attribute is present | spec §4.4 |
+| Decode layers | "Decoding is applied as **explicit dask graph layers** over the raw array … not hidden inside an opaque `__getitem__`" | spec §4.4 |
+| No `ZarrDataProxy` | "Do not mirror `NetCDFDataProxy` on the Zarr side" | spec §4.4 |
+| Cache key | `(store location, array path, json.dumps(Array.metadata.to_dict(), sort_keys=True))`, treated as opaque | spec §4.4 |
+| Read unit | `chunking` returns `Array.chunks`, never `Array.shards`; alignment aggregates **upward** only | spec §4.4 |
+| Format detection | Local: a directory holding `zarr.json` or `.zgroup`. Remote: scheme in `s3`, `gs`, `az`, `abfs` **and** the path ends `.zarr`. "**Do not add a third rule matching any path that ends in `.zarr`**" | spec §4.7 |
+| `load_files` | "is not taught about directories" — a separate `iris.io.load_stores` answers "this target is not a file", once, for local and remote alike | spec §4.7 |
+| NCZarr | Unaffected. `_uri_is_nczarr` is left exactly as it is | spec §4.7, §9 |
+| Attribute policy | A nested JSON object or array is carried onto the cube unchanged and not interpreted | spec §4.4 |
+| Tests | "pytest style, no network, no unittest classes in new files" | spec §6 |
+| Skipping | "A single `pytest.importorskip(\"zarr\")` fixture in the Zarr test package. Iris without zarr installed must show no new failures" | spec §6 |
+| Remote tests | "covered by an in-memory store and a local directory store only. No test touches the network" | spec §6 |
+| Every commit | `pre-commit run --files <paths>` then `pytest`, both **before** the commit. Never `--no-verify` | `AGENTS.md` |
+| Test tiers | `pytest lib/iris/tests/unit/<area>/` per edit (no `-n auto` on small runs); `-n auto` only for parent-directory and full-suite runs | `lib/iris/tests/AGENTS.md` |
+| Test data | `OVERRIDE_TEST_DATA_REPOSITORY` must be set **and verified** before trusting any run; read the skip count, not just the failures | `lib/iris/tests/AGENTS.md` |
+| Line length | 88 characters, Ruff default | `AGENTS.md` |
+| Docstrings | NumPy style, numpydoc-validated. Module docstring mandatory outside tests (D100); a public module needs a `z_reference` item in its first 25 lines | `AGENTS.md`, `docs/AGENTS.md` |
+| New `.py` files | Four-line copyright header, checked by `check-license-headers` | `AGENTS.md` |
+| netCDF access | No direct `import netCDF4` — always `iris.fileformats.netcdf._thread_safe_nc` | `AGENTS.md` |
+| New modules | Aim under ~1000 lines; cohesion wins over line count | `lib/iris/AGENTS.md` |
+| Warning classes | Every `UserWarning` subclass descends from `IrisUserWarning`; load-time CF complaints use `IrisCfLoadWarning`, chunking complaints `IrisLoadWarning` | `AGENTS.md`, `iris.warnings` |
+| Prose | No citation of a plan number, task number or phase in code, tests or docs. Cite an issue, a pull request, the design document, or the convention's section | `lib/iris/tests/AGENTS.md`, `docs/AGENTS.md` |
+| Attribution | Commits end `Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>`; every GitHub post opens `(posted by Claude Opus)` | `AGENTS.md` |
+
+---
+
+## Review Focus
+
+Five failure modes the spec implies that **no task's own tests would otherwise
+exercise**, most likely first. Each names the input or condition and the
+behaviour a reasonable person expects; each has a test added to the task that
+owns the code, written out in that task's own step style.
+
+1. **A version 2 store's `_ARRAY_DIMENSIONS` attribute landing on the loaded
+ cube.** It is how a version 2 store records dimension names, so under the
+ §4.2 split it is a *property*, not a CF attribute — and every attribute a
+ `CFVariable` does not consume reaches `cube.attributes`
+ (`cf/loader.py::_add_unused_attributes`). Nothing in `_CF_ATTRS` excludes
+ it, so unless `ZarrDatasetVariable.attributes` withholds it, every cube
+ loaded from a version 2 store gains a stray `_ARRAY_DIMENSIONS` attribute
+ and fails to compare equal to the same cube loaded from version 3. The user
+ expects the two to be indistinguishable. → **Task 3, Step 9**, and
+ **Task 13, Step 4** end to end.
+
+2. **A store whose root group holds no arrays.** Spec §4.6 requires "a message
+ that names the groups that do", and §7 records a real store in that shape —
+ the ESA EOPF Sentinel-2 product, whose root holds only subgroups. The
+ failure to defend against is not the message but its absence: an empty
+ `CubeList` returned silently from `iris.load` of a store that plainly
+ contains data is the worst available answer, and the one that falls out of
+ writing the loop without thinking about it. → **Task 10, Step 2**.
+
+3. **`iris.load("data/*")` matching a mix of `.nc` files and `.zarr`
+ directories.** Spec §4.7 states the requirement and names the consequence —
+ the file-or-store decision has to happen after glob expansion and before
+ format dispatch. The quiet failure is partial: `_generate_cubes` partitions
+ what `expand_filespecs` returns, so a bug that mis-partitions raises
+ `IsADirectoryError` from `load_files` for the stores while the netCDF files
+ load fine, and a user who did not expect the glob to match a store reads it
+ as an Iris bug. → **Task 12, Step 2**.
+
+4. **A `scale_factor` or `add_offset` that changes the result dtype.**
+ `cf/loader.py::_get_actual_dtype` computes the post-unpacking dtype and the
+ netCDF path gets the arithmetic itself from netCDF4. On the Zarr path Iris
+ does the arithmetic, in `_decode.py`, so the dask array's declared dtype and
+ what the decode layer actually produces are two separately-written
+ statements that must agree. When they disagree dask does not complain: the
+ graph is built, and the mismatch surfaces later as a wrong dtype on the
+ cube, or as a `ValueError` from a downstream operation. A `uint16` band with
+ `scale_factor = 0.0001` — the EOPF reflectance case in spec §7 — must give a
+ `float64` cube whose lazy and realised dtypes match. → **Task 7, Step 11**.
+
+5. **A `zarr` that is installed but too old, or not installed at all.** Spec §3
+ requires "a clear `ImportError`" and §6 requires that "Iris without zarr
+ installed must show no new failures". Two distinct paths reach a user here —
+ `iris.load` of a `.zarr` target, and `import iris.fileformats.zarr` — and the
+ second is the one that breaks the first rule by accident, because
+ `iris.fileformats.__init__` imports every format module it registers. If the
+ Zarr import is not deferred, installing Iris without zarr stops `import iris`
+ working at all. → **Task 1, Steps 5–6**, and §6.3.
+
+---
+
+## 1. Scope boundary
+
+**In.**
+
+- `lib/iris/fileformats/zarr/` — `__init__.py`, `_dataset.py`, `_decode.py`,
+ `loader.py`.
+- `CFDataset.format` / `CFDatasetVariable.format`, and `CFDataset.encoding` /
+ `CFDatasetVariable.encoding`, on the interface and on both implementations.
+- `CFVariable.spans` and its three overrides matching on `(name, size)` pairs.
+- `CFReader` accepting an already-open `CFDataset`.
+- The `"Zarr"` `FormatSpecification` and its `ZarrStore` file element.
+- The four load-chain changes of spec §4.7, including `iris.io.load_stores`.
+- `zarr` added to the optional-dependency section of
+ `requirements/py3{12,13,14}.yml`, and the lock files regenerated once.
+- User-guide and API-reference documentation for Zarr loading, including the
+ remote URL forms and `group=`; and the S3 page rewritten.
+- Unit and integration tests for all of the above.
+
+**Out.**
+
+- **Saving.** No `zarr/saver.py`, no `ZarrDataset` write path, no
+ `consolidate_metadata`, no registration in `_savers`. `ZarrDataset`'s
+ write-surface members exist because the ABC declares them and raise
+ `NotImplementedError`. That is PR 6.
+- **The write half of the shared `CFDataset` contract.** It cannot run against
+ a dataset that refuses to create variables; §2 F10 explains the split.
+- **Real-world fixtures and benchmarks.** The NOAA GFS and ESA EOPF samples,
+ and the ASV coverage, are PR 7. Every fixture here is synthetic, built in
+ `tmp_path` or in memory.
+- **xarray as a test dependency.** Spec §6 lands it in PR 6, with the saver it
+ validates. Nothing here needs a second reader.
+- **The netCDF read path's own sub-chunk re-read trap** (spec §12.3 Q4). The
+ alignment guard is added on the Zarr path only.
+- **UGRID meshes from a Zarr store.** `iris.mesh.load_meshes` gains the ability
+ to *reach* a Zarr store (spec §4.7), because its scheme dispatch is a copy of
+ `loading.py`'s and the two move together. Nothing here builds or tests a mesh
+ from Zarr.
+- **Changing the NCZarr path**, beyond leaving it alone.
+
+---
+
+## 2. Findings that refine the spec
+
+Checked while writing this plan. Each is either a correction the implementer
+would otherwise hit as a failing test, or an answer the spec asked for.
+
+**F1 — `zarr.Array` has no `dimension_names` property, and the error spec §2.4
+quotes is a *write* error.** Version 3 names live at
+`Array.metadata.dimension_names`. `create_array(..., dimension_names=…)` on a
+version 2 group raises `ValueError: Zarr format 2 arrays do not support
+dimension names.`, which is the message the spec cites — but **reading**
+`ArrayV2Metadata.dimension_names` raises `AttributeError`, because the
+attribute simply is not there **[verified]**. So `ZarrDatasetVariable.dimensions`
+must branch on `metadata.zarr_format` and read `attrs["_ARRAY_DIMENSIONS"]` on
+version 2. A `try: … except ValueError` would not catch anything.
+
+**F2 — zarr-python rejects a wrong-length `dimension_names` on write and on
+read, so the case is reachable only through a version 2 store.**
+`create_array(..., shape=(2, 2), dimension_names=("y",))` raises
+``ValueError: `dimension_names` and `shape` need to have the same number of
+dimensions`` **[verified]** — and the identical check runs on **read**: a
+version 3 store whose `zarr.json` is hand-edited to the wrong length raises
+that same `ValueError` from `open_group(...)["v"]` itself, before Iris ever
+holds an array **[verified]**. Spec §4.4's warn-and-load-anonymously rule is
+therefore untestable, and unreachable, on version 3. The fixture for that
+test is a version 2 store instead, whose `_ARRAY_DIMENSIONS` carries no such
+validation. The length check in the reader stays format-agnostic and correct
+as defensive code; it simply never fires for a version 3 array that reached
+Iris at all.
+
+**F3 — spec §12.3 Q2 is answered: `<U7` reads fine.** A fixed-length unicode
+array creates, writes and reads back with its dtype intact on **both** version
+2 and version 3 **[verified]**: `dtype('<U7')`, values
+`array(['alpha', 'beta'], dtype='<U7')`. #6961's note that these are "not
+strictly Zarr-supported" describes the specification, not zarr-python's
+behaviour, and no special handling is needed. Close Q2 in spec §12.3 with this
+answer.
+
+**F4 — `Array.metadata.to_dict()` is JSON-serialisable as it stands.** No
+`default=` fallback is needed, on either version **[verified]**. The version 3
+key set is `attributes, chunk_grid, chunk_key_encoding, codecs, data_type,
+dimension_names, fill_value, node_type, shape, storage_transformers,
+zarr_format`; version 2's is `attributes, chunks, compressor,
+dimension_separator, dtype, fill_value, filters, order, shape, zarr_format`.
+A small array's key is ~460–520 characters, matching spec §4.4's estimate.
+
+**F5 — zarr-python copies a store when it opens it read-only.** `WrapperStore`
+implements `with_read_only` by constructing a new instance through
+`_with_store`, so after `zarr.open_group(store=my_wrapper, mode="r")` the
+array's store **is not** `my_wrapper` **[verified]**. An instrumented store
+that counts bytes must therefore keep its tally in an object the copy shares,
+and override `_with_store` to pass it on. A counter stored on the instance
+reads zero and the test passes vacuously.
+
+**F6 — the sharding read-unit result reproduces, but not its byte counts.** On
+`shape=(1024,)`, `chunks=(8,)`, `shards=(1024,)` int32, reading the first eight
+values **[verified]**: inner-chunk dask blocks fetch 2,052 bytes with or
+without a decode layer; shard-sized blocks fetch 2,052 without one and **7,300
+with** one. The shape of spec §4.4's table holds exactly — a `map_blocks` layer
+defeats slice pushdown, so a shard-sized block materialises the whole shard —
+but the absolute numbers differ from the spec's 2,084 / 6,148. They are
+zarr-version-dependent. **The test compares its own two measurements; it must
+not pin a literal byte count.**
+
+**F7 — `_optimum_chunksize` sub-chunks a large store chunk, exactly as spec
+§4.4 says.** `chunks=(8000, 8000)`, `shape=(16000, 16000)`, `f8` gives
+`(2000, 8000)`; `chunks=(6000, 6000)`, `shape=(6000, 6000)`, `f8` gives
+`(2000, 6000)` **[verified]**. Both are the real regression cases for the
+alignment guard, and neither needs a store to reproduce.
+
+**F8 — the errors zarr-python raises for the targets Iris will meet**
+**[verified]**:
+
+| target | exception |
+|---|---|
+| a path that does not exist | `FileNotFoundError` |
+| a directory that is not a store | `zarr.errors.GroupNotFoundError` |
+| `group=` naming a group that is not there | `zarr.errors.GroupNotFoundError` |
+| `s3://…` with no s3fs installed | `ImportError: Install s3fs to access S3` |
+
+The last is why spec §3's "Iris adds no dependency and handles no credentials"
+costs nothing to honour: zarr-python's own message already tells the user what
+to install.
+
+**F9 — a read-mode `zarr.Array` refuses attribute writes** with
+`ValueError: store was opened in read-only mode and does not support writing`
+**[verified]**. `CFVariable` copies `data.attributes` into a plain `dict` at
+construction (`cf/_variables.py::CFVariable.__init__`), so loading never writes
+through — but `ZarrDatasetVariable.attributes` must still be a `MutableMapping`
+to satisfy the interface, and its `__setitem__` will raise that `ValueError`
+from zarr-python on a read-mode store. That is the right answer and needs no
+wrapping.
+
+**F10 — the shared `CFDataset` contract has to be split in two.**
+`lib/iris/tests/unit/fileformats/cf/dataset/contract.py` holds one class whose
+`writable` and `reopen` fixtures drive `populate()` through `create_dimension`,
+`create_variable`, `__setitem__` and `write_handle`. Spec §5 puts the Zarr
+write path in PR 6, so the whole write half is unrunnable here. Split
+`CFDatasetContract` into `CFDatasetReadContract` and `CFDatasetWriteContract`;
+`test_NetCDFDataset.py` inherits both and its coverage is unchanged; the new
+Zarr test module inherits the read half, and PR 6 adds the other. Keeping one
+class and skipping half of it would hide, behind a skip, the fact that an
+implementation is incomplete — spec §6's whole argument for a shared body is
+that divergence "fails loudly".
+
+**F11 — `format` has to land here, not in PR 5.** Spec §5 assigns
+`CFDataset.format` and `CFDataset.supports_dtype` to PR 5, but PR 4 merges
+first and `ZarrDataset` cannot answer `ZARR_V2` versus `ZARR_V3` without it:
+the version-aware masking split (§4.4) and the warnings this pull request
+emits both need to name the format. `supports_dtype` is genuinely a saver
+concern and stays in PR 5. **Add `format` here; record the move in spec §12.4
+and amend §5's PR 4 and PR 5 entries.**
+
+**F12 — `CFReader` cannot yet be handed an open `CFDataset`.**
+`cf/_reader.py::CFReader.__init__` has two branches: a `str` opens a
+`NetCDFDataset`, and anything else goes to `NetCDFDataset.from_existing`. A
+`ZarrDataset` would be wrapped as a netCDF dataset and fail. PR 4 adds a third
+branch that takes an already-open `CFDataset` and borrows it. Note that
+`isinstance(file_source, CFDataset)` tests the **interface**, which is what
+spec §4.2 requires generic code to do; the rule it forbids is testing an
+implementation class.
+
+**F13 — the marker files are where spec §4.7 says they are** **[verified]**. A
+version 2 store's root directory holds `.zgroup` and `.zattrs`; a version 3
+store's holds `zarr.json`. One `Path.is_file()` each.
+
+**F14 — attribute values round-trip through `Array.attrs` untouched**, for a
+nested object (`{'min': 1, 'max': 2}`), a numeric vector (`[1.0, 2.0]`) and
+`float("nan")` **[verified]**. The read side of spec §4.4's nested-attribute
+rule therefore needs no code at all: the value arrives as a `dict` or `list`
+and is carried onto the cube. It needs a **test**, because "we do nothing" is
+indistinguishable from "we never tried it".
+
+**F15 — `Group.arrays()` yields only that group's own arrays**, and
+`Group.group_keys()` lists its subgroups **[verified]**. Together they are
+exactly spec §4.6: one group per call, and a root group with no arrays can name
+the groups that have them.
+
+**F16 — the `(name, size)` change does not have to reach
+`_nc_load_rules/helpers.py`.** Four sites there match dimensions by name —
+`helpers.py` lines near `build_auxiliary_coordinate`,
+`build_dimension_coordinate`, `_add_cell_measures` and
+`_add_ancillary_variables` — and `None in (None, "x")` is `True`, so an
+anonymous dimension reaching them would attach a coordinate to the wrong cube
+dimension. It cannot reach them: `CFReader._build_cf_groups` drops any variable
+whose `spans` is `False` and promotes it to a data variable of its own, and
+under the new rule a variable with any anonymous dimension spans nothing.
+Dimension coordinates are safe for a second reason — `CFCoordinateVariable`
+identifies on `nc_var_name in nc_var.dimensions`, which no anonymous dimension
+satisfies. **Task 7 pins both claims with tests** rather than leaving them as
+prose; without them this finding is the kind of comment that goes stale
+unnoticed.
+
+**F17 — `_CF_ATTRS_IGNORE` already covers the four attributes Iris consumes.**
+`cf/_variables.py` seeds attribute tracking with
+`{"_FillValue", "add_offset", "missing_value", "scale_factor"}`, described as
+"handled by the netCDF4 module". On the Zarr path they are handled by
+`zarr/_decode.py` instead, so the seeding stays correct and needs no change —
+but the comment naming netCDF4 does, and `valid_min`/`valid_max`/`valid_range`
+are deliberately **not** in that set, so they reach the cube on both paths.
+
+**F18 — this pull request creates the first way to get a nested attribute onto
+a cube, and the netCDF saver cannot write one.** Spec §4.4 says Iris carries a
+JSON object or array attribute onto the cube and "does not write them back
+through the netCDF saver"; spec §4.5 says checking attribute content before
+saving is [#7329](https://github.com/SciTools/iris/issues/7329), out of scope.
+Until then, `iris.load("store.zarr")` followed by `iris.save(cube, "out.nc")`
+fails inside netCDF4 with ``TypeError: illegal data type for attribute
+b'stac_discovery', ..., got O`` **[verified]** — it names the attribute, not
+the cube. Nothing in this pull request can fix that without changing the
+netCDF saver, which PR 5 relocates unchanged — so **Task 10 characterises it
+with a test and Task 14 documents it**, which is the difference between a
+known limitation and a surprise.
+
+---
+
+## What changed during implementation
+
+Every task's own tests pass against the code below; these are the places
+where that code is not quite what the task text above shows, and why.
+
+- **Task 7, reversed after review.** The chunk alignment this task builds --
+ `_align_chunks`, `_fixed_flags`, `_store_chunk_exceeds_dask_target` and
+ `test__align_chunks.py` -- is no longer in the branch. Review on #7336
+ asked whether the extra machinery earned its place, and it did not:
+ alignment only ever changed the answer where a stored chunk exceeds the
+ Dask target, which is a trade the netCDF path has always accepted, so
+ correcting it for Zarr alone would make two formats differ over physics
+ they share. The warning the task added is kept, and moved to
+ `iris.fileformats.cf.loader._warn_if_storage_chunk_binds`, where it covers
+ both backends; `CHUNK_CONTROL` is the answer for a caller who wants the
+ other trade. The measurements and the full reasoning are in spec §4.4 and
+ the 2026-10-20 decision-log entry. Everything else in Task 7 -- laziness,
+ the decoded-dtype sizing, the cache key -- stands as written.
+
+- **Task 3.** F2's test could not be built the way Step 2 shows it: a
+ version 3 store whose `dimension_names` is hand-edited to the wrong length
+ is rejected on **read**, not only on write, so `open_group(...)["v"]`
+ raises before `ZarrDatasetVariable` ever sees the array. The test is a
+ version 2 store instead, with `_ARRAY_DIMENSIONS` set by hand — version 2
+ carries no such validation. See F2 above, corrected.
+
+- **Task 5.** The shared contract's first cut put `close`, `closed` and the
+ context manager in `CFDatasetWriteContract`. They sit in
+ `CFDatasetReadContract` instead: spec §4.2 describes all three as
+ answerable by a dataset the caller merely opened, and leaving them on the
+ write side would have let a read-only implementation diverge on exactly
+ the question the split exists to catch.
+
+- **Task 6.** `decode_fill_value` also treats a decoded sentinel that
+ overflows the variable's own dtype — signed, negative or unsigned — as a
+ producer error: a warning naming the variable, and `None`, rather than
+ letting `OverflowError` propagate out of the whole load. The task text
+ only names "neither form" as the failure case; a value of the right form
+ and the wrong width is a third one.
+
+- **Task 7.** `_align_chunks` does not warn when rounding up is capped at
+ the variable's own extent. A block capped at the extent still reads each
+ store chunk exactly once, so nothing is subdivided; the first cut warned
+ on every load of an ordinary store whose chunking does not divide its
+ shape evenly, which is most stores. The two conditions that genuinely mean
+ something — a store chunk already larger than the Dask target, and an
+ explicit `CHUNK_CONTROL` setting that subdivides — are unchanged, and each
+ keeps its own test.
+
+- **Task 12.** Step calls to `iris.io.load_files(files, callback,
+ constraints)` from `_generate_cubes` are not what ships. `load_files` is
+ public API with a documented promise that its argument may be a glob;
+ narrowing it to expect an already-expanded list, as the step shows,
+ silently breaks that promise for any other caller holding a reference to
+ it. `load_files`'s body moves to a private `_load_files` that expects
+ pre-expanded paths — which is what the duplicate-glob-match fix actually
+ needs — and public `load_files` goes back to expanding once and
+ delegating. `_generate_cubes` calls `_load_files` directly, with the list
+ it has already expanded and partitioned, so a glob is still expanded
+ exactly once end to end.
+
+- **Task 13.** `ZarrDataset.attributes` withholds `_NCProperties` as well as
+ `_ARRAY_DIMENSIONS`. It is netCDF-c's own bookkeeping stamp, written onto
+ the root group of every store it produces; netCDF4-python already hides it
+ from `Dataset.ncattrs()`, so it has never reached a cube loaded through
+ netCDF, and the native reader must withhold it for the same reason it
+ withholds `_ARRAY_DIMENSIONS` — otherwise the same data would produce
+ cubes differing by a provenance string depending on which writer made the
+ store. Confirmed not to appear on an array or a subgroup, only the root.
+ Not anticipated by any task above.
+
+- **Task 4.** `ZarrDataset.__init__` accepts a store object, not only a
+ `str`: the interface sketch above still declares `location: str`, but
+ `zarr.open_group` is handed `location` itself rather than its stringified
+ form, so a store with no path of its own - a `zarr.storage.MemoryStore`,
+ say - is accepted and opened unaltered.
+ `test_an_in_memory_store_is_just_another_location` is built on exactly
+ this.
+
+- **Task 1.** `stores.VARIABLE_ATTRS` carries `"grid_mapping": "grid"`, not
+ only `units` and `long_name` as the task text shows, so that the shared
+ canonical store exercises the same grid-mapping path the loader's own
+ tests ask questions of.
+
+These are the substantive ones. A few test files were also split, renamed or
+reorganised without changing what they prove — Task 1's store-builder tests
+moved out of `test_optional_dependency.py` into their own `test_stores.py`,
+for one — and are not repeated here file by file.
+
+---
+## 3. File Structure
+
+### Created
+
+| Path | Responsibility | Rough size |
+|---|---|---|
+| `lib/iris/fileformats/zarr/__init__.py` | Public surface: `load_cubes`, re-exported, plus the package docstring that says `import zarr` inside this package still resolves to the third-party distribution (spec §4.1) | ~60 |
+| `lib/iris/fileformats/zarr/_dataset.py` | `ZarrDataset`, `ZarrDatasetVariable`, `_ZarrAttributes`, `_align_chunks`. The only module that knows both the CF interface and the zarr-python API | ~550 |
+| `lib/iris/fileformats/zarr/_decode.py` | The CF rules netCDF4 applies for free: missing-value masking, `valid_range` masking, `scale_factor`/`add_offset` unpacking, and base64 `_FillValue` decoding. Pure functions of `(block, parameters)` | ~250 |
+| `lib/iris/fileformats/zarr/loader.py` | `load_cubes(file_sources, callback=None, constraints=None, group=None)` | ~180 |
+| `lib/iris/tests/unit/fileformats/zarr/__init__.py` | The package-wide `pytest.importorskip("zarr")` and the store-building fixtures | ~120 |
+| `lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py` | Empty package marker | 5 |
+| `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py` | `ZarrDataset`'s own cases: dimensions, groups, errors, the refused write surface | ~230 |
+| `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py` | The shared read contract, run against both Zarr versions | ~40 |
+| `lib/iris/tests/unit/fileformats/zarr/test_CFReader.py` | `CFReader` classifying a Zarr store | ~30 |
+| `lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py` | No module in the `cf` package type-tests an implementation class | ~50 |
+| `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py` | The dimension contract, `fill_value`, `chunking`, `encoding`, attributes | ~320 |
+| `lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py` | The alignment guard, on `chunking` values alone | ~140 |
+| `lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py` | Laziness, the cache key's staleness and sensitivity cases, the sharded read unit | ~300 |
+| `lib/iris/tests/unit/fileformats/zarr/test__decode.py` | Each decode rule on plain ndarrays, and `_FillValue` in both encodings | ~280 |
+| `lib/iris/tests/unit/fileformats/zarr/loader/__init__.py` | Empty package marker | 5 |
+| `lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py` | One group per call, the empty-root message, `group=` | ~200 |
+| `lib/iris/tests/unit/fileformats/cf/test_spans.py` | The `(name, size)` suite, run against both backends | ~260 |
+| `lib/iris/tests/unit/io/test_load_stores.py` | Partitioning, scheme routing, and what `load_stores` refuses | ~180 |
+| `lib/iris/tests/unit/io/format_picker/test_ZarrStore.py` | The local marker-file test and the remote scheme test, and that neither swallows the other | ~150 |
+| `lib/iris/tests/integration/zarr/__init__.py` | Empty package marker | 5 |
+| `lib/iris/tests/integration/zarr/test_load.py` | `iris.load` end to end, version 2 and version 3, including an NCZarr-written version 2 store | ~260 |
+| `lib/iris/tests/integration/zarr/fixtures.py` | A version 3 store shaped like the NOAA GFS archive | ~110 |
+| `docs/src/user_manual/explanation/zarr_io.rst` | Loading a Zarr store: local and remote, groups, dimension names, missing data, chunking, attributes | ~170 |
+
+### Modified
+
+| Path | Change |
+|---|---|
+| `lib/iris/fileformats/cf/dataset.py` | Add `format` and `encoding` to both abstract classes (spec §4.2, F11) |
+| `lib/iris/fileformats/netcdf/_dataset.py` | Implement `format` from `file_format`, and `encoding` from `filters()`/`chunking()`/`createVariable`'s keywords |
+| `lib/iris/fileformats/cf/_variables.py` | `CFVariable.dimensions` admits `None` entries; `spans` and the three overrides match `(name, size)` pairs; `_is_scalar` unchanged |
+| `lib/iris/fileformats/cf/_reader.py` | A third `__init__` branch for an already-open `CFDataset`; module docstring updated — it is no longer "the one module still coupled to `iris.fileformats.netcdf`" in the way it says |
+| `lib/iris/fileformats/__init__.py` | Register the `"Zarr"` `FormatSpecification`; import the Zarr loader lazily |
+| `lib/iris/io/format_picker.py` | Add the `ZarrStore` file element |
+| `lib/iris/io/__init__.py` | `expand_filespecs` leaves a recognised remote scheme alone; new `load_stores`; `find_saver` tolerates a trailing separator; `_CUBELIST_SAVER_MODULES` replaces the substring test |
+| `lib/iris/loading.py` | `_generate_cubes` partitions files from stores and routes the remote schemes |
+| `lib/iris/fileformats/netcdf/ugrid_load.py` | The same scheme dispatch, updated in step |
+| `lib/iris/tests/unit/fileformats/cf/dataset/contract.py` | Split into `CFDatasetReadContract` and `CFDatasetWriteContract` (F10); add the `format` and `encoding` cases |
+| `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py` | Inherit both contract halves |
+| `requirements/py312.yml`, `py313.yml`, `py314.yml` | `zarr >=3.0.8` in the optional-dependency section |
+| `requirements/locks/*.lock` | Regenerated, once, by `make lockfiles` |
+| `docs/src/user_manual/tutorial/s3_io.rst` | Replace "S3 is unsupported, use a FUSE mount" with the Zarr route (spec §4.7) |
+| `docs/src/user_manual/section_indexes/general.rst` | List the new page in the toctree, beside `netcdf_io` |
+| `docs/superpowers/specs/2026-09-21-zarr-io-design.md` | §12.1 state, §12.3 Q2 closed, §12.4 entries for F10 and F11, §5's PR 4/PR 5 split amended |
+
+### Sequencing, and why it is this way
+
+The order is: make a Zarr store answerable as a `CFDataset`, then make the CF
+layer able to hold one, then make Iris able to find one.
+
+1. **Tasks 1–7 build the Zarr dataset bottom-up** — dependency, interface,
+ variable, group, contract, decoding, reading — and are testable without any
+ of Iris's loading machinery. Each one ends with a store built in `tmp_path`
+ and questions asked of it directly.
+2. **Tasks 8–9 are the two changes to generic code.** They are separated from
+ the Zarr work because they are the only parts a netCDF-only reviewer has to
+ check, and because `spans` is the one place in this pull request where a
+ mistake silently changes what loads from an existing netCDF file.
+3. **Task 10 is the loader**, which can only be written once 1–9 exist.
+4. **Tasks 11–12 are the load chain**, which is independent of everything above
+ and could be reviewed by someone who knows nothing about Zarr. They come
+ last among the code tasks because until Task 10 exists there is nothing for
+ the chain to dispatch *to*.
+5. **Tasks 13–15** are integration, documentation and the pull request.
+
+The dependency that matters in the other direction is **PR 3**. This plan calls
+`CFDatasetVariable.read_data`, `iris.fileformats.cf.loader._get_actual_dtype`,
+`_LAZYVAR_MIN_BYTES` and `as_lazy_data(..., cache_key=...)`, none of which are
+on `brownfield` until [#7316](https://github.com/SciTools/iris/pull/7316)
+merges. So this branch is cut from `zarr-io-pr-3` rather than waiting, and
+lands on `brownfield` later:
+
+```bash
+git switch -c zarr-io-pr-4 zarr-io-pr-3
+```
+
+Work proceeds on that base, and the preview pull request described in Task 15
+is raised against it on the fork. When #7316 merges, Task 15's Step 6 moves
+this branch's own commits across:
+
+```bash
+git fetch upstream
+git rebase --onto upstream/brownfield zarr-io-pr-3 zarr-io-pr-4
+```
+
+and re-runs Task 6 and Task 9's tests — the two that read PR 3's code most
+closely — before anything is trusted again. Nothing between here and there
+assumes `brownfield`; every instruction that names a base names
+`zarr-io-pr-3`.
+
+---
+
+## 4. The Zarr implementation, as designed
+
+Signatures the tasks below fill in. Read this before Task 1; it is the single
+place the names and types agree.
+
+```python
+# lib/iris/fileformats/cf/dataset.py (additions)
+
+class CFDatasetVariable(ABC):
+ @property
+ @abstractmethod
+ def format(self) -> str:
+ """The concrete format of the dataset holding this variable."""
+
+ @property
+ @abstractmethod
+ def encoding(self) -> Mapping[str, Any]:
+ """The storage settings this variable actually holds."""
+
+
+class CFDataset(ABC):
+ @property
+ @abstractmethod
+ def format(self) -> str:
+ """The concrete format: "NETCDF4", "NETCDF3_CLASSIC", "ZARR_V2", "ZARR_V3"."""
+
+ @property
+ @abstractmethod
+ def encoding(self) -> Mapping[str, Any]:
+ """The store-wide settings this store actually holds."""
+```
+
+```python
+# lib/iris/fileformats/zarr/_dataset.py
+
+#: What CFDataset.format answers, by the store's Zarr version.
+_FORMAT_BY_ZARR_VERSION = {2: "ZARR_V2", 3: "ZARR_V3"}
+
+#: The version 2 attribute that carries dimension names. It is storage, not a
+#: CF attribute, so it is withheld from `attributes` and read by `dimensions`.
+_ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS"
+
+
+def _dimension_names(array: "zarr.Array") -> tuple[str | None, ...]: ...
+def _assemble_dimensions(variables: Mapping) -> dict[str, int]: ...
+def _fixed_flags(dims_fixed, ndim: int) -> tuple[bool, ...]: ...
+def _store_chunk_exceeds_dask_target(store_chunks, dtype) -> bool: ...
+
+
+def _align_chunks(
+ chunks: tuple[int, ...],
+ store_chunks: tuple[int, ...] | None,
+ shape: tuple[int, ...],
+ dtype: np.dtype,
+ dims_fixed,
+ name: str,
+) -> tuple[int, ...]:
+ """Round `chunks` up to whole multiples of the store's own chunk shape."""
+
+
+class _ZarrAttributes(MutableMapping):
+ """A zarr node's attributes as a mapping, withholding the storage ones."""
+
+ def __init__(self, node, *, withhold: frozenset[str] = frozenset()): ...
+
+
+class ZarrDatasetVariable(CFDatasetVariable):
+ def __init__(self, array: "zarr.Array", location: str, format: str): ...
+
+ name: str # array.basename
+ location: str # the dataset's location, verbatim
+ format: str # "ZARR_V2" | "ZARR_V3"
+ dimensions: tuple[str | None, ...] # see the table in spec 4.4
+ shape: tuple[int, ...]
+ dtype: np.dtype
+ size: int
+ fill_value: Any | None # array.fill_value: STORAGE, not a mask
+ chunking: tuple[int, ...] | None # array.chunks, never array.shards
+ encoding: Mapping[str, Any] # chunks, shards, codecs, zarr_format
+ attributes: MutableMapping # array.attrs, less _ARRAY_DIMENSIONS
+
+ def __getitem__(self, keys) -> np.ndarray: ...
+ def __setitem__(self, keys, values) -> None: ...
+ def read_data(self, chunking_policy: Callable[[], tuple]) -> Any: ...
+ def write_handle(self) -> "ZarrDatasetVariable": ... # returns self
+
+ @property
+ def cache_key(self) -> str:
+ """`(location, array path, the metadata document)`, as one string."""
+
+ @property
+ def zarr_format(self) -> int:
+ """2 or 3. Zarr-only; the generic layer never asks which version."""
+
+ @property
+ def array(self) -> "zarr.Array":
+ """The backing zarr array. Zarr-only; nothing generic may read it."""
+
+
+class ZarrDataset(CFDataset):
+ def __init__(
+ self,
+ location: str,
+ mode: str = "r",
+ *,
+ group: str | None = None,
+ ): ...
+
+ location: str # the target as given, with the group path appended
+ format: str # "ZARR_V2" | "ZARR_V3"
+ mode: str
+ closed: bool
+ variables: Mapping[str, ZarrDatasetVariable]
+ dimensions: Mapping[str, int] # names used at one consistent size
+ encoding: Mapping[str, Any] # zarr_format, consolidated
+ attributes: MutableMapping # group.attrs
+
+ def create_dimension(self, name, size) -> None: ... # NotImplementedError
+ def create_variable(self, name, dtype, dimensions=(), *,
+ fill_value=None, **encoding): ... # NotImplementedError
+ def sync(self) -> None: ... # no-op; a Zarr write is not buffered
+ def finalise(self) -> None: ... # NotImplementedError until PR 6
+ def close(self) -> None: ... # records closure; a store is not a handle
+
+ @property
+ def subgroup_names(self) -> tuple[str, ...]:
+ """The names of this group's subgroups. Zarr-only; for the no-data message."""
+```
+
+```python
+# lib/iris/fileformats/zarr/_decode.py
+
+def decode_fill_value(value: Any, dtype: np.dtype, variable_name: str) -> Any | None:
+ """Return a CF `_FillValue`/`missing_value` as a value of `dtype`, or None.
+
+ Accepts a JSON number, and the base64 of a little-endian float64 that
+ released xarray writes for every floating dtype. Anything else warns and
+ returns None.
+ """
+
+
+def mask_and_scale(
+ block: np.ndarray,
+ *,
+ sentinels: tuple,
+ valid_min: Any | None,
+ valid_max: Any | None,
+ scale_factor: Any | None,
+ add_offset: Any | None,
+ dtype: np.dtype,
+) -> np.ndarray:
+ """Apply the CF decode rules netCDF4 applies for free, in CF order."""
+
+
+def mask_by_sentinels(block: np.ndarray, sentinels: tuple) -> np.ndarray: ...
+def mask_by_valid_range(block, valid_min, valid_max) -> np.ndarray: ...
+def unpack(block, scale_factor, add_offset, dtype) -> np.ndarray: ...
+
+
+def decode_parameters(
+ attributes: Mapping,
+ *,
+ dtype: np.dtype,
+ storage_fill_value: Any,
+ zarr_format: int,
+ variable_name: str,
+) -> dict:
+ """Read the CF decode parameters for one variable.
+
+ Plain arguments, not a variable, so that spec section 4.4's version-aware
+ masking precedence is testable without building a store for each case:
+ on version 3 only the CF attributes are a mask source; on version 2
+ `Array.fill_value` is one as well, when no CF attribute is present.
+
+ Always returns the same keys - the keyword arguments `mask_and_scale`
+ takes. `decoding_required` says whether they do anything.
+ """
+
+
+def decoding_required(parameters: dict) -> bool: ...
+def _unpacked_dtype(dtype, scale_factor, add_offset) -> np.dtype: ...
+```
+
+```python
+# lib/iris/fileformats/zarr/loader.py
+
+def load_cubes(file_sources, callback=None, constraints=None, group=None):
+ """Generate cubes from Zarr stores."""
+```
+
+```python
+# lib/iris/io/__init__.py (additions)
+
+#: Modules whose `save` takes a CubeList directly, rather than one cube at a
+#: time with `append=True`. One line per format.
+_CUBELIST_SAVER_MODULES = frozenset({"iris.fileformats.netcdf.saver"})
+
+#: URI schemes Iris passes through to a store library rather than resolving as
+#: a local path.
+REMOTE_STORE_SCHEMES = frozenset({"s3", "gs", "az", "abfs"})
+
+
+def _is_remote_store_uri(spec) -> bool: ...
+
+
+def load_stores(targets, callback, constraints=None):
+ """Create a generator of Cubes from targets that are stores, not files."""
+```
+
+```python
+# lib/iris/fileformats/__init__.py (additions)
+
+#: What a version 3 and a version 2 Zarr store put at the root of a store.
+_ZARR_MARKERS = ("zarr.json", ".zgroup")
+
+
+def _is_zarr_store(target) -> bool:
+ """Whether `target` names a native Zarr store."""
+
+
+def _load_zarr(*args, **kwargs):
+ """Load from Zarr stores, reporting clearly when zarr-python is absent."""
+```
+
+```python
+# lib/iris/io/format_picker.py (addition)
+
+class ZarrStore(FileElement):
+ """A :class:`FileElement` that returns a target that may be a Zarr store.
+
+ Carries the whole target, not just its basename, because recognising a
+ local store means reading a marker file inside it.
+ """
+
+ def __init__(self):
+ super().__init__(requires_fh=False)
+
+ def get_element(self, basename, file_handle): ... # returns basename
+```
+
+Three points about this shape are worth stating once, here, rather than
+repeating in every task.
+
+**`fill_value` means different things on the two backends, deliberately.**
+`NetCDFDatasetVariable.fill_value` returns the `_FillValue` *attribute*;
+`ZarrDatasetVariable.fill_value` returns `Array.fill_value`, the *storage*
+field. On netCDF the two coincide. On Zarr they are different things (spec
+§4.4), and conflating them masks every valid zero in an array that never
+declared a sentinel. Nothing generic reads `fill_value` to decide masking —
+`zarr/_decode.py::decode_parameters` does, and it reads the CF attributes
+first.
+
+**`encoding` is readable and never interpreted.** Generic code may print it,
+record it, or hand it back to a user; it must not branch on a key. Its keys are
+the backend's own spelling: `chunks`, `shards`, `codecs`, `zarr_format` for
+Zarr; `chunksizes`, `zlib`, `complevel` for netCDF. `chunking` is the typed,
+guaranteed member that the chunk calculation reads (spec §4.2).
+
+**`cache_key` is a member of `ZarrDatasetVariable`, not of the interface.**
+`read_data` is the only caller and both implementations build their own key —
+netCDF from `repr(proxy)`, Zarr from the metadata document. Putting it on the
+ABC would add a member no generic code reads, which is the indirection
+`lib/iris/AGENTS.md` bans for a single caller.
+
+---
+## 5. Tasks
+
+### 5.0 The green check every task ends with
+
+Every task's last step is a commit, and every commit is preceded by this,
+unchanged:
+
+```bash
+# 1. Confirm the test-data path took effect. A missing repository SKIPS tests
+# silently, so an unset variable removes coverage without failing anything.
+echo "$OVERRIDE_TEST_DATA_REPOSITORY"
+ls "$OVERRIDE_TEST_DATA_REPOSITORY" > /dev/null
+
+# 2. Hooks rewrite files in place: re-stage and re-run until clean.
+pre-commit run --files <the paths this task touched>
+git add <the paths this task touched>
+pre-commit run --files <the paths this task touched>
+
+# 3. The tier the task's own steps name. Never --no-verify on the commit.
+```
+
+Read the **skip count** as well as the failure count. A Zarr test that skips
+because `zarr` is missing from the environment is a test that did not run, and
+`pytest -q` reports it in the same line as the passes.
+
+---
+
+### Task 1: `zarr` as an optional dependency, and a package that does not need it
+
+**Files:**
+- Modify: `requirements/py312.yml`, `requirements/py313.yml`,
+ `requirements/py314.yml`
+- Modify: `requirements/locks/py312-linux-64.lock`, `py313-linux-64.lock`,
+ `py314-linux-64.lock`
+- Create: `lib/iris/fileformats/zarr/__init__.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/__init__.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/stores.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py`
+
+**Interfaces:**
+- Consumes: nothing.
+- Produces: the importable package `iris.fileformats.zarr`; the test package
+ whose `__init__.py` skips everything below it when `zarr` is absent; and
+ `stores.py`'s builders — `make_store(path, *, zarr_format=3, chunks="auto",
+ shards=None)`, `make_array(group, name, shape, dtype, zarr_format, *,
+ dimensions=None, chunks="auto", shards=None, fill_value=None,
+ attributes=None)`, and the constants `DIMENSIONS`, `DATA`, `VARIABLE_ATTRS`,
+ `GLOBALS` — which every later Zarr test builds its fixtures from.
+
+The package's `__init__.py` re-exports nothing yet; `loader.py` does not exist
+until Task 10, and a re-export of a name that is not there is an import error.
+Task 10 adds the `from .loader import load_cubes` line. The registration that
+gives a user a clear `ImportError` is Task 11, beside the format
+specification, which is the only place that needs it.
+
+- [ ] **Step 1: Add the dependency to all three requirement files**
+
+In each of `requirements/py312.yml`, `py313.yml` and `py314.yml`, under the
+`# Optional dependencies.` heading, after ` - affine`:
+
+```yaml
+ - zarr >=3.0.8
+```
+
+The floor is spec §8: "Pin `>=3.0.8`, which is above the known data-loss bug".
+
+- [ ] **Step 2: Regenerate the lock files**
+
+```bash
+make lockfiles
+git status --short requirements/locks/
+```
+
+Expected: the three `py3{12,13,14}-linux-64.lock` files change, each gaining
+`zarr`, `numcodecs`, `donfig` and their own dependencies.
+
+This needs network access and takes minutes. If it cannot run here, say so in
+the pull request body and ask a maintainer to regenerate — do **not** hand-edit
+a lock file.
+
+- [ ] **Step 3: Create the package**
+
+Create `lib/iris/fileformats/zarr/__init__.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Native Zarr support, built on zarr-python.
+
+.. z_reference:: iris.fileformats.zarr
+ :tags: topic_load_save
+
+ API reference
+
+Reads Zarr version 2 and version 3 stores, local or remote. Not NCZarr: that
+is netCDF-c's own Zarr driver, reached through a URL fragment such as
+``file:///path/store.zarr#mode=nczarr,file``, and it stays where it is in
+:mod:`iris.fileformats.netcdf`.
+
+``zarr`` is an optional dependency. Nothing in :mod:`iris.fileformats` imports
+this package until a Zarr target is actually loaded, so Iris installs and
+imports without it.
+
+This package shadows the third-party ``zarr`` distribution by name. Python 3
+resolves ``import zarr`` absolutely, so inside these modules it still finds the
+third-party package; xarray's ``xarray.backends.zarr`` has the same
+arrangement.
+
+See sections 4.1 and 4.4 of
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+```
+
+- [ ] **Step 4: Create the test package, with the skip that guards it**
+
+Create `lib/iris/tests/unit/fileformats/zarr/__init__.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.zarr`.
+
+``zarr`` is optional, so every module below this one is skipped when it is
+absent. The skip lives here rather than in a ``conftest.py``: the same call
+there raises during collection and is reported as an **error**, not a skip.
+"""
+
+import pytest
+
+pytest.importorskip("zarr", minversion="3.0.8")
+```
+
+- [ ] **Step 5: Write the failing test for the import guard**
+
+Create `lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Iris must import, and load netCDF, without zarr-python installed."""
+
+import subprocess
+import sys
+
+
+def _imports_zarr(statement):
+ """Run `statement` in a fresh interpreter; report whether zarr was imported."""
+ probe = f"{statement}\nimport sys\nprint('zarr' in sys.modules)\n"
+ completed = subprocess.run(
+ [sys.executable, "-c", probe],
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+ return completed.stdout.strip() == "True"
+
+
+def test_importing_iris_does_not_import_zarr():
+ assert not _imports_zarr("import iris")
+
+
+def test_importing_the_format_registry_does_not_import_zarr():
+ # iris.fileformats imports every format module it registers a handler
+ # from. If the Zarr loader joined that list, Iris would stop installing
+ # without zarr-python.
+ assert not _imports_zarr("import iris.fileformats")
+
+
+def test_importing_the_zarr_package_does_not_import_zarr():
+ # The package docstring is reachable without the dependency; only the
+ # modules under it need zarr itself.
+ assert not _imports_zarr("import iris.fileformats.zarr")
+```
+
+- [ ] **Step 6: Run the tests to see them pass, and know why**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py -v
+```
+
+Expected: 3 passed. These pass from the start — that is the point. They are a
+*regression* guard, not a red-green cycle: the way they fail is by someone
+later adding `from . import zarr` to `iris/fileformats/__init__.py`, or an
+`import zarr` to `iris/fileformats/zarr/__init__.py`. Prove they can fail
+before trusting them, by temporarily adding `import zarr` to the package
+`__init__.py` and re-running:
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py -v
+```
+
+Expected with the temporary line: `test_importing_the_zarr_package_does_not_import_zarr`
+FAILS. Remove the line.
+
+- [ ] **Step 7: Write the shared store builders**
+
+Create `lib/iris/tests/unit/fileformats/zarr/stores.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Builders for the small Zarr stores these tests ask questions of.
+
+One canonical store, deliberately the same contents as the shared CFDataset
+contract writes, so that a Zarr answer and a netCDF answer can be compared
+without translating between two sets of fixtures.
+"""
+
+import numpy as np
+import zarr
+
+#: The canonical store's dimensions, in order.
+DIMENSIONS = {"y": 3, "x": 2}
+DATA = np.arange(6, dtype="f8").reshape(3, 2)
+VARIABLE_ATTRS = {"units": "K", "long_name": "surface temperature"}
+GLOBALS = {"Conventions": "CF-1.7", "title": "contract"}
+
+#: The version 2 attribute carrying dimension names.
+ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS"
+
+
+def make_array(
+ group,
+ name,
+ shape,
+ dtype,
+ zarr_format,
+ *,
+ dimensions=None,
+ chunks="auto",
+ shards=None,
+ fill_value=None,
+ attributes=None,
+):
+ """Create one array, recording dimension names the way its version does.
+
+ zarr-python refuses `dimension_names` on a version 2 array, so version 2
+ names are written to the `_ARRAY_DIMENSIONS` attribute by hand - which is
+ what xarray and NCZarr both do.
+ """
+ extra = {}
+ if zarr_format == 3 and dimensions is not None:
+ extra["dimension_names"] = dimensions
+ array = group.create_array(
+ name,
+ shape=shape,
+ dtype=dtype,
+ chunks=chunks,
+ shards=shards,
+ fill_value=fill_value,
+ **extra,
+ )
+ if zarr_format == 2 and dimensions is not None:
+ array.attrs[ARRAY_DIMENSIONS] = list(dimensions)
+ for key, value in (attributes or {}).items():
+ array.attrs[key] = value
+ return array
+
+
+def make_store(path, *, zarr_format=3, chunks="auto", shards=None):
+ """Write the canonical store at `path`, and return `path` as a string."""
+ root = zarr.open_group(store=path, mode="w", zarr_format=zarr_format)
+ for name, value in GLOBALS.items():
+ root.attrs[name] = value
+
+ array = make_array(
+ root,
+ "air_temperature",
+ DATA.shape,
+ "f8",
+ zarr_format,
+ dimensions=tuple(DIMENSIONS),
+ chunks=chunks,
+ shards=shards,
+ fill_value=-1.0,
+ attributes=VARIABLE_ATTRS,
+ )
+ array[:] = DATA
+
+ # A dimensionless variable: what a grid mapping is, and the one shape a
+ # CFDatasetVariable must handle without a leading dimension.
+ scalar = make_array(
+ root,
+ "grid",
+ (),
+ "i4",
+ zarr_format,
+ dimensions=(),
+ attributes={"grid_mapping_name": "latitude_longitude"},
+ )
+ scalar[()] = 0
+
+ return str(path)
+```
+
+- [ ] **Step 8: Prove the builders build both versions**
+
+Append to `test_optional_dependency.py`:
+
+```python
+import zarr
+
+from . import stores
+
+
+def test_make_store_version_3(tmp_path):
+ location = stores.make_store(tmp_path / "v3.zarr")
+ group = zarr.open_group(store=location, mode="r")
+ assert group["air_temperature"].metadata.zarr_format == 3
+ assert group["air_temperature"].metadata.dimension_names == ("y", "x")
+
+
+def test_make_store_version_2(tmp_path):
+ location = stores.make_store(tmp_path / "v2.zarr", zarr_format=2)
+ group = zarr.open_group(store=location, mode="r")
+ array = group["air_temperature"]
+ assert array.metadata.zarr_format == 2
+ assert array.attrs[stores.ARRAY_DIMENSIONS] == ["y", "x"]
+```
+
+- [ ] **Step 9: Run them**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/ -v
+```
+
+Expected: 5 passed.
+
+- [ ] **Step 10: Commit**
+
+Run §5.0, then:
+
+```bash
+git add requirements lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr
+git commit -m "Add zarr as an optional dependency, and the package that uses it
+
+The package holds only its docstring for now: iris.fileformats does not
+import it, so Iris still installs and imports without zarr-python. The
+test package skips itself when the dependency is absent."
+```
+
+---
+
+### Task 2: `format` and `encoding` on the interface, and on netCDF
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/dataset.py`
+- Modify: `lib/iris/fileformats/netcdf/_dataset.py`
+- Modify: `lib/iris/tests/unit/fileformats/cf/dataset/contract.py`
+- Modify: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py`
+ — this is where `TestNetCDFDatasetContract` lives, not in
+ `test_NetCDFDataset.py`
+
+**Interfaces:**
+- Consumes: the existing `CFDataset` / `CFDatasetVariable` from PR 2.
+- Produces: `CFDataset.format`, `CFDataset.encoding`,
+ `CFDatasetVariable.format`, `CFDatasetVariable.encoding`; and the contract
+ split into `CFDatasetReadContract` and `CFDatasetWriteContract` (F10), which
+ Task 5 subclasses.
+
+Spec §5 gives `format` to PR 5. It has to be here instead, because PR 4 merges
+first and a `ZarrDataset` that cannot say which Zarr version it is cannot
+implement §4.4's masking rule or name itself in a warning (F11).
+`supports_dtype` is a saver question and stays in PR 5.
+
+- [ ] **Step 1: Write the failing contract tests**
+
+In `lib/iris/tests/unit/fileformats/cf/dataset/contract.py`, add to
+`CFDatasetContract`, under the `# -- The dataset itself ---` heading:
+
+```python
+ def test_format_is_a_non_empty_string(self, readable):
+ assert isinstance(readable.format, str)
+ assert readable.format
+
+ def test_encoding_is_a_readable_mapping(self, readable):
+ # Its keys are the backend's own spelling and generic code must never
+ # branch on one, but it must be possible to read them: otherwise a
+ # caller that wants to know how a store is laid out has to reach
+ # through to the netCDF4 or zarr object, which is the abstraction
+ # leaking at the point it exists to hold.
+ assert isinstance(readable.encoding, Mapping)
+ dict(readable.encoding)
+```
+
+and under `# -- A variable ---`:
+
+```python
+ def test_variable_format_matches_its_dataset(self, variable, readable):
+ assert variable.format == readable.format
+
+ def test_variable_encoding_reports_the_chunking(self, variable):
+ encoding = variable.encoding
+ assert isinstance(encoding, Mapping)
+ # Whatever a backend spells it, the chunk shape it reports through
+ # `chunking` has to be somewhere in `encoding` too, or `encoding` is
+ # not reporting what the store holds.
+ assert variable.chunking in (None, *encoding.values())
+```
+
+Add `from collections.abc import Mapping` to the module's imports.
+
+- [ ] **Step 2: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/ -v -k "format or encoding"
+```
+
+Expected: FAIL, `AttributeError: 'NetCDFDataset' object has no attribute 'format'`.
+
+- [ ] **Step 3: Add the two members to the interface**
+
+In `lib/iris/fileformats/cf/dataset.py`, in `CFDatasetVariable`, after
+`location`:
+
+```python
+ @property
+ @abstractmethod
+ def format(self) -> str:
+ """The concrete format of the dataset holding this variable.
+
+ Repeated here from the dataset so that a variable can answer for
+ itself, as :attr:`location` is. For messages and diagnostics only:
+ generic code asks for a capability, never for a format.
+
+ """
+```
+
+and after `chunking`:
+
+```python
+ @property
+ @abstractmethod
+ def encoding(self) -> Mapping:
+ """The storage settings this variable holds, in the backend's spelling.
+
+ ``zlib``, ``complevel``, ``chunksizes`` for netCDF; ``chunks``,
+ ``shards``, ``codecs``, ``zarr_format`` for Zarr. Readable, never
+ interpreted: no generic code may branch on a key. The typed,
+ guaranteed-present member is :attr:`chunking`.
+
+ """
+```
+
+In `CFDataset`, after `mode`:
+
+```python
+ @property
+ @abstractmethod
+ def format(self) -> str:
+ """The concrete format: ``NETCDF4``, ``NETCDF3_CLASSIC``, ``ZARR_V2``, ``ZARR_V3``.
+
+ What the loader, the saver and ``LOAD_PROBLEMS`` *say* when they have
+ to name the format. Reconstructing it from the implementation class
+ would be the type test this interface exists to forbid.
+
+ """
+
+ @property
+ @abstractmethod
+ def encoding(self) -> Mapping:
+ """The store-wide settings this store holds, in the backend's spelling."""
+```
+
+`Mapping` is already imported in that module.
+
+- [ ] **Step 4: Implement them for netCDF**
+
+In `lib/iris/fileformats/netcdf/_dataset.py`, in `NetCDFDatasetVariable`, after
+`location`:
+
+```python
+ @property
+ def format(self) -> str:
+ """The netCDF format of the file holding this variable."""
+ return self._format
+```
+
+taking a `format` argument in `__init__` — add it to the signature after
+`location`, store it as `self._format`, and pass
+`self._dataset.file_format` from both construction sites in `NetCDFDataset`
+(`_materialised_variables` and `create_variable`).
+
+After `chunking`:
+
+```python
+ @property
+ def encoding(self) -> Mapping:
+ """The netCDF storage settings this variable holds.
+
+ Read back off the file, so a variable created with `zlib=True` reports
+ the compression netCDF4 actually applied rather than the keyword it
+ was given.
+
+ """
+ filters = self._variable.filters() or {}
+ return MappingProxyType({**filters, "chunksizes": self.chunking})
+```
+
+Add `from types import MappingProxyType` to the module's imports.
+
+In `NetCDFDataset`, after `mode`:
+
+```python
+ @property
+ def format(self) -> str:
+ """The netCDF format of this file, as netCDF4 reports it."""
+ assert self._dataset is not None
+ return self._dataset.file_format
+
+ @property
+ def encoding(self) -> Mapping:
+ """The store-wide netCDF settings. Only the format is one."""
+ return MappingProxyType({"format": self.format})
+```
+
+- [ ] **Step 5: Run the contract**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/ -v
+```
+
+Expected: all pass, including the four new cases.
+
+- [ ] **Step 6: Split the contract into its read and write halves**
+
+Rename `CFDatasetContract` to `CFDatasetReadContract`, keeping everything
+above the `# -- Writing ---` heading and the `readable`, `variable` and
+`scalar` fixtures. Move everything below that heading, plus the `writable` and
+`reopen` fixtures and `populate()`, into a new class
+`CFDatasetWriteContract` in the same module. Update the module docstring:
+
+```python
+"""One test body, run against every :class:`CFDataset` implementation.
+
+Subclass :class:`CFDatasetReadContract` in a module named for the
+implementation, supply the fixtures it declares, and every test here runs
+against it. Both classes deliberately lack a ``Test`` prefix, so pytest does
+not collect them where they are written, only where they are subclassed.
+
+The split is read from write because an implementation may arrive in two
+pieces: the Zarr loader can answer every question in the read half before a
+Zarr saver exists. Skipping half of one class instead would hide an
+incomplete implementation behind a skip, and the point of a shared body is
+that divergence fails loudly.
+
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md`` section 6: the
+interface is only worth having if both sides of it agree, and agreement is
+cheapest to check by asking the same questions twice.
+
+Deliberately not exercised here: :meth:`CFDatasetVariable.deprecated_netcdf_member`.
+It is an explicitly netCDF4-specific compatibility escape hatch, so asserting
+either its presence or its absence in a shared body would itself be
+backend-specific - the omission is intentional, not a gap.
+"""
+```
+
+In `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py`,
+change the test class to inherit both, and update its import:
+
+```python
+from iris.tests.unit.fileformats.cf.dataset.contract import (
+ CFDatasetReadContract,
+ CFDatasetWriteContract,
+)
+
+
+class TestNetCDFDatasetContract(CFDatasetReadContract, CFDatasetWriteContract):
+```
+
+- [ ] **Step 7: Run the netCDF contract again and compare the count**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/ -q
+```
+
+Expected: the same number of tests as before the split, plus the four added in
+Step 1. The split must move tests, not lose them — count before and after.
+
+- [ ] **Step 8: Commit**
+
+Run §5.0, then:
+
+```bash
+git add lib/iris/fileformats/cf/dataset.py lib/iris/fileformats/netcdf/_dataset.py lib/iris/tests/unit/fileformats
+git commit -m "Add format and encoding to the CF dataset interface
+
+format names the concrete format for a message or a diagnostic; encoding
+reports the storage settings a store actually holds, readable but never
+interpreted. Splits the shared contract into its read and write halves, so
+that an implementation arriving in two pieces can be held to the first."
+```
+
+---
+
+### Task 3: `ZarrDatasetVariable` — the properties, and the dimension contract
+
+**Files:**
+- Create: `lib/iris/fileformats/zarr/_dataset.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py`
+
+**Interfaces:**
+- Consumes: `CFDatasetVariable` with `format` and `encoding` (Task 2);
+ `stores.make_array`, `stores.make_store` (Task 1).
+- Produces: `ZarrDatasetVariable(array, location, format)` with every read
+ member of the interface except `read_data`, which is Task 7; the constants
+ `_FORMAT_BY_ZARR_VERSION` and `_ARRAY_DIMENSIONS`; and the module docstring
+ that later tasks add to.
+
+Spec §4.4's dimension table is the whole of this task:
+
+| store | `dimensions` |
+|---|---|
+| version 3, `dimension_names` present | its entries, null becoming `None` |
+| version 3, `dimension_names` absent | `(None,) * ndim` |
+| version 2, `_ARRAY_DIMENSIONS` present | its entries |
+| version 2, `_ARRAY_DIMENSIONS` absent | `(None,) * ndim` |
+| any version, scalar array | `()` |
+| any version, wrong length | `(None,) * ndim`, with an `IrisCfLoadWarning` |
+
+- [ ] **Step 1: Write the failing tests for the dimension table**
+
+Create `lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py` with only the
+copyright header, then
+`lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDatasetVariable`."""
+
+import json
+
+import numpy as np
+import pytest
+import zarr
+
+from iris.fileformats.zarr._dataset import ZarrDatasetVariable
+from iris.warnings import IrisCfLoadWarning
+
+from .. import stores
+
+
+def wrap(array, location="store.zarr"):
+ """Return the ZarrDatasetVariable over `array`, naming its own format."""
+ zarr_format = array.metadata.zarr_format
+ return ZarrDatasetVariable(array, location, f"ZARR_V{zarr_format}")
+
+
+@pytest.fixture(params=[2, 3], ids=["v2", "v3"])
+def zarr_format(request):
+ return request.param
+
+
+@pytest.fixture
+def group(tmp_path, zarr_format):
+ return zarr.open_group(
+ store=tmp_path / "probe.zarr", mode="w", zarr_format=zarr_format
+ )
+
+
+def test_named_dimensions(group, zarr_format):
+ array = stores.make_array(
+ group, "v", (3, 2), "f8", zarr_format, dimensions=("y", "x")
+ )
+ assert wrap(array).dimensions == ("y", "x")
+
+
+def test_unnamed_dimensions_are_anonymous(group, zarr_format):
+ array = stores.make_array(group, "v", (3, 2), "f8", zarr_format)
+ assert wrap(array).dimensions == (None, None)
+
+
+def test_scalar_has_no_dimensions(group, zarr_format):
+ array = stores.make_array(group, "v", (), "i4", zarr_format)
+ assert wrap(array).dimensions == ()
+
+
+def test_partially_named_dimensions_version_3(tmp_path):
+ # The version 3 core specification: "an unnamed dimension is indicated by
+ # the null object". A store may name some dimensions and not others.
+ group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3)
+ array = group.create_array(
+ "v", shape=(3, 2), dtype="f8", dimension_names=("y", None)
+ )
+ assert wrap(array).dimensions == ("y", None)
+
+
+def test_repeated_name_within_one_array(tmp_path):
+ # "It is recommended but not required that all non-null dimension names
+ # are distinct." Recommended is not required, so this must not raise.
+ group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3)
+ array = group.create_array(
+ "v", shape=(2, 2), dtype="f8", dimension_names=("d", "d")
+ )
+ assert wrap(array).dimensions == ("d", "d")
+```
+
+- [ ] **Step 2: Add the malformed-store test**
+
+zarr-python rejects a wrong-length `dimension_names` at creation (F2), so this
+one writes the metadata by hand. Append:
+
+```python
+def test_wrong_length_dimension_names_warns_and_loads_anonymous(tmp_path):
+ """A `dimension_names` whose length is not the rank is a malformation.
+
+ The version 3 core specification says the field "must be an array of
+ strings or null objects with the same length as `shape`". zarr-python
+ refuses to write one, so the metadata is edited directly.
+ """
+ group = zarr.open_group(store=tmp_path / "bad.zarr", mode="w", zarr_format=3)
+ group.create_array("v", shape=(3, 2), dtype="f8", dimension_names=("y", "x"))
+
+ metadata_path = tmp_path / "bad.zarr" / "v" / "zarr.json"
+ document = json.loads(metadata_path.read_text())
+ document["dimension_names"] = ["y"]
+ metadata_path.write_text(json.dumps(document))
+
+ array = zarr.open_group(store=tmp_path / "bad.zarr", mode="r")["v"]
+ with pytest.warns(IrisCfLoadWarning, match="dimension_names"):
+ dimensions = wrap(array).dimensions
+ assert dimensions == (None, None)
+```
+
+- [ ] **Step 3: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/ -v
+```
+
+Expected: collection error,
+`ModuleNotFoundError: No module named 'iris.fileformats.zarr._dataset'`.
+
+- [ ] **Step 4: Write the module, and the variable's dimension logic**
+
+Create `lib/iris/fileformats/zarr/_dataset.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The Zarr implementation of the :mod:`iris.fileformats.cf.dataset` interface.
+
+This is the only module that knows both the CF interface and the zarr-python
+API. Everything Zarr-specific the CF layer could want - where dimension names
+are recorded, what a chunk is, what the store's own fill value is - either has
+a named member on the interface or lives here as a Zarr-only member.
+
+Three things about Zarr shape this module, and each is a trap if forgotten.
+
+**A dimension name guarantees less than a netCDF one.** The version 3 core
+specification makes ``dimension_names`` optional, allows a null entry, only
+*recommends* that names are distinct within an array, and places no
+restriction at all on reusing a name across arrays. All of that is conforming
+Zarr, so none of it may raise. A dimension with no name is **anonymous**, which
+Iris has always had: a cube dimension carrying no coordinate.
+
+**``Array.fill_value`` is not ``_FillValue``.** It says what unwritten storage
+reads back as; it does not say the value is missing. zarr-python defaults it to
+zero, so reading it as a missing-value sentinel masks every valid zero in an
+array that never declared one. :attr:`ZarrDatasetVariable.fill_value` reports
+it as the storage fact it is, and the masking decision is made in
+:mod:`iris.fileformats.zarr._decode` from the CF attributes.
+
+**The read unit is the inner chunk, even when the array is sharded.** Zarr's
+sharding codec is indexed, so a reader fetches only the inner chunks it needs.
+:attr:`ZarrDatasetVariable.chunking` therefore returns ``Array.chunks``, never
+``Array.shards``, and :func:`_align_chunks` aggregates upward into whole
+multiples of it.
+
+See sections 4.2, 4.4 and 4.6 of
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+from collections.abc import Iterator, Mapping, MutableMapping
+from types import MappingProxyType
+from typing import Any
+import warnings
+
+import numpy as np
+import zarr
+
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+import iris.warnings
+
+#: What :attr:`CFDataset.format` answers, by the store's Zarr version.
+_FORMAT_BY_ZARR_VERSION = {2: "ZARR_V2", 3: "ZARR_V3"}
+
+#: The version 2 attribute carrying dimension names, as xarray and NCZarr
+#: write it. Storage, not a CF attribute, so it is withheld from `attributes`.
+_ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS"
+
+
+def _dimension_names(array) -> tuple:
+ """Return one name or None per dimension of `array`.
+
+ Parameters
+ ----------
+ array : :class:`zarr.Array`
+ The array to read names from. Version 3 carries them in its metadata;
+ version 2 carries them in the ``_ARRAY_DIMENSIONS`` attribute, which
+ is a de-facto convention rather than part of the Zarr specification.
+
+ Returns
+ -------
+ tuple
+ Of length ``array.ndim``. ``None`` where the store names no dimension.
+
+ Warns
+ -----
+ IrisCfLoadWarning
+ If the store names a number of dimensions other than ``array.ndim``.
+ The version 3 specification says the lengths "must" match, so such a
+ store is malformed; its names are ignored and the array loads with
+ anonymous dimensions.
+
+ """
+ anonymous = (None,) * array.ndim
+ if not array.ndim:
+ return ()
+
+ if array.metadata.zarr_format == 3:
+ names = array.metadata.dimension_names
+ else:
+ names = array.attrs.get(_ARRAY_DIMENSIONS)
+
+ if names is None:
+ return anonymous
+
+ if len(names) != array.ndim:
+ message = (
+ "Ignoring malformed dimension_names, which names "
+ f"{len(names)} dimensions of a {array.ndim}-dimensional array: "
+ f"{array.basename}"
+ )
+ warnings.warn(message, category=iris.warnings.IrisCfLoadWarning)
+ return anonymous
+
+ return tuple(names)
+```
+
+- [ ] **Step 5: Write the class**
+
+Append to `_dataset.py`:
+
+```python
+class ZarrDatasetVariable(CFDatasetVariable):
+ """One array of a Zarr store, presented through the CF interface."""
+
+ def __init__(self, array, location: str, format: str):
+ """Wrap `array`.
+
+ Parameters
+ ----------
+ array : :class:`zarr.Array`
+ The array this variable presents.
+ location : str
+ The path or URL of the dataset this array belongs to.
+ format : str
+ ``ZARR_V2`` or ``ZARR_V3``, as its dataset reports.
+
+ """
+ self._array = array
+ self._location = location
+ self._format = format
+ self._dimensions = _dimension_names(array)
+ self._attributes = _ZarrAttributes(array, withhold={_ARRAY_DIMENSIONS})
+
+ @property
+ def name(self) -> str:
+ """The array's name within its group."""
+ return self._array.basename
+
+ @property
+ def location(self) -> str:
+ """The path or URL of the dataset holding this variable."""
+ return self._location
+
+ @property
+ def format(self) -> str:
+ """``ZARR_V2`` or ``ZARR_V3``."""
+ return self._format
+
+ @property
+ def dimensions(self) -> tuple:
+ """One name per dimension, ``None`` where the store names none."""
+ return self._dimensions
+
+ @property
+ def shape(self) -> tuple:
+ """The array's shape."""
+ return tuple(self._array.shape)
+
+ @property
+ def dtype(self) -> np.dtype:
+ """The array's stored data type."""
+ return self._array.dtype
+
+ @property
+ def size(self) -> int:
+ """The total number of elements in the array."""
+ return int(self._array.size)
+
+ @property
+ def fill_value(self) -> Any:
+ """What unwritten storage reads back as. **Not** a missing-value flag.
+
+ Version 3 requires this field, and zarr-python defaults it to zero, so
+ it says nothing about whether a value is missing. The CF
+ ``_FillValue`` and ``missing_value`` attributes say that, and
+ :mod:`iris.fileformats.zarr._decode` reads them.
+
+ """
+ return self._array.fill_value
+
+ @property
+ def chunking(self) -> tuple | None:
+ """The array's inner chunk shape, never its shard shape."""
+ chunks = self._array.chunks
+ return tuple(chunks) if chunks else None
+
+ @property
+ def encoding(self) -> Mapping:
+ """The Zarr storage settings this array holds."""
+ return MappingProxyType(
+ {
+ "chunks": self._array.chunks,
+ "shards": self._array.shards,
+ "codecs": self._array.metadata.to_dict().get("codecs"),
+ "zarr_format": self._array.metadata.zarr_format,
+ }
+ )
+
+ @property
+ def attributes(self) -> MutableMapping:
+ """The array's CF attributes, less the ones that are really storage."""
+ return self._attributes
+
+ def __getitem__(self, keys) -> np.ndarray:
+ """Return the indexed portion of the array's data."""
+ return self._array[keys]
+
+ def __setitem__(self, keys, values) -> None:
+ """Write `values` into the indexed portion of the array."""
+ self._array[keys] = values
+
+ def write_handle(self) -> "ZarrDatasetVariable":
+ """Return self: a Zarr write target needs no proxy.
+
+ ``zarr.Array`` is picklable and a chunk is an independent object in a
+ key-value store, so there is no open file handle to avoid holding and
+ no lock to acquire.
+
+ """
+ return self
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self.name!r}, {self.location!r})"
+
+ # Zarr-only members below.
+
+ @property
+ def array(self):
+ """The backing :class:`zarr.Array`. Nothing generic may read this."""
+ return self._array
+```
+
+Note `chunking` returns `None` for a scalar array, because `Array.chunks` is
+`()` there and an empty tuple is not a chunk shape.
+
+- [ ] **Step 6: Write the attributes mapping**
+
+Append to `_dataset.py`, above `ZarrDatasetVariable`:
+
+```python
+class _ZarrAttributes(MutableMapping):
+ """A Zarr node's attributes as a mapping, less the ones that are storage.
+
+ ``_ARRAY_DIMENSIONS`` is how a version 2 store records dimension names.
+ Under the interface's split that is a property, reported by
+ :attr:`ZarrDatasetVariable.dimensions`, not a CF attribute - so it is
+ withheld here. Leaving it in would put it on every cube loaded from a
+ version 2 store, since every attribute the loading rules do not consume
+ reaches ``cube.attributes``.
+ """
+
+ def __init__(self, node, *, withhold: frozenset = frozenset()):
+ """Materialise the attributes of `node`, a zarr array or group."""
+ self._node = node
+ self._withhold = frozenset(withhold)
+ self._values = {
+ key: value
+ for key, value in node.attrs.items()
+ if key not in self._withhold
+ }
+
+ def __getitem__(self, key: str) -> Any:
+ """Return `key`'s value."""
+ return self._values[key]
+
+ def __setitem__(self, key: str, value: Any) -> None:
+ """Set `key`'s value, here and in the store."""
+ if key in self._withhold:
+ msg = f"Cannot set a storage attribute through `attributes`: {key}"
+ raise KeyError(msg)
+ self._node.attrs[key] = value
+ self._values[key] = value
+
+ def __delitem__(self, key: str) -> None:
+ """Remove `key`, here and from the store."""
+ del self._node.attrs[key]
+ del self._values[key]
+
+ def __iter__(self) -> Iterator[str]:
+ """Return an iterator over the attribute names, in store order."""
+ return iter(self._values)
+
+ def __len__(self) -> int:
+ """Return the number of attributes."""
+ return len(self._values)
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self._values!r})"
+```
+
+- [ ] **Step 7: Run the dimension tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/ -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 8: Write the tests for the remaining properties**
+
+Append to `test_ZarrDatasetVariable.py`:
+
+```python
+def test_plain_properties(tmp_path, zarr_format):
+ location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format)
+ array = zarr.open_group(store=location, mode="r")["air_temperature"]
+ variable = wrap(array, location)
+
+ assert variable.name == "air_temperature"
+ assert variable.location == location
+ assert variable.format == f"ZARR_V{zarr_format}"
+ assert variable.shape == stores.DATA.shape
+ assert variable.dtype == np.dtype("f8")
+ assert variable.size == stores.DATA.size
+ assert variable.ndim == stores.DATA.ndim
+ assert len(variable) == stores.DATA.shape[0]
+ np.testing.assert_array_equal(variable[:], stores.DATA)
+ np.testing.assert_array_equal(variable[1], stores.DATA[1])
+
+
+def test_fill_value_is_the_storage_field(tmp_path, zarr_format):
+ """`fill_value` is what unwritten storage reads as, not `_FillValue`.
+
+ zarr-python defaults the field to zero, so an array that declares no CF
+ sentinel still reports one here. Reading it as a mask source is what
+ masks every valid zero.
+ """
+ group = zarr.open_group(
+ store=tmp_path / "f.zarr", mode="w", zarr_format=zarr_format
+ )
+ array = stores.make_array(group, "v", (3,), "i2", zarr_format)
+ assert wrap(array).fill_value == 0
+
+ declared = stores.make_array(
+ group, "w", (3,), "i2", zarr_format, fill_value=-999
+ )
+ assert wrap(declared).fill_value == -999
+
+
+def test_chunking_is_the_inner_chunk_not_the_shard(tmp_path):
+ """A sharded array reports `Array.chunks`.
+
+ Zarr's sharding codec is indexed: a reader fetches the shard index, then
+ byte ranges for only the inner chunks it needs. The inner chunk is the
+ independently decodable unit.
+ """
+ group = zarr.open_group(store=tmp_path / "sh.zarr", mode="w", zarr_format=3)
+ array = group.create_array(
+ "v", shape=(1024,), dtype="i4", chunks=(8,), shards=(1024,)
+ )
+ variable = wrap(array)
+ assert variable.chunking == (8,)
+ assert variable.encoding["shards"] == (1024,)
+
+
+def test_encoding_reports_the_store(tmp_path, zarr_format):
+ location = stores.make_store(tmp_path / "e.zarr", zarr_format=zarr_format)
+ array = zarr.open_group(store=location, mode="r")["air_temperature"]
+ encoding = wrap(array, location).encoding
+ assert encoding["chunks"] == array.chunks
+ assert encoding["zarr_format"] == zarr_format
+```
+
+- [ ] **Step 9: Write the test for Review Focus 1 — `_ARRAY_DIMENSIONS`**
+
+Append:
+
+```python
+def test_array_dimensions_is_not_a_cf_attribute(tmp_path):
+ """`_ARRAY_DIMENSIONS` is storage, so it must not reach `attributes`.
+
+ It is how a version 2 store records dimension names. Every attribute the
+ loading rules do not consume reaches `cube.attributes`, so leaving it in
+ would put it on every cube loaded from a version 2 store, and make that
+ cube unequal to the same data loaded from version 3.
+ """
+ group = zarr.open_group(store=tmp_path / "v2.zarr", mode="w", zarr_format=2)
+ array = stores.make_array(
+ group,
+ "v",
+ (3, 2),
+ "f8",
+ 2,
+ dimensions=("y", "x"),
+ attributes={"units": "K"},
+ )
+ variable = wrap(array)
+
+ assert variable.dimensions == ("y", "x")
+ assert stores.ARRAY_DIMENSIONS in array.attrs
+ assert stores.ARRAY_DIMENSIONS not in variable.attributes
+ assert dict(variable.attributes) == {"units": "K"}
+
+
+def test_version_2_and_version_3_present_the_same_attributes(tmp_path):
+ v2 = stores.make_store(tmp_path / "a2.zarr", zarr_format=2)
+ v3 = stores.make_store(tmp_path / "a3.zarr", zarr_format=3)
+ from_2 = zarr.open_group(store=v2, mode="r")["air_temperature"]
+ from_3 = zarr.open_group(store=v3, mode="r")["air_temperature"]
+ assert dict(wrap(from_2, v2).attributes) == dict(wrap(from_3, v3).attributes)
+```
+
+- [ ] **Step 10: Write the test for nested and vector attributes (F14)**
+
+Append:
+
+```python
+def test_structured_attribute_values_are_carried_unchanged(tmp_path):
+ """A JSON object or array attribute is carried, not interpreted.
+
+ The version 3 specification allows an attribute to be any JSON literal,
+ and CF cannot express a nested one at any rank. Two independent producers
+ write them - NOAA GFS gives `valid_time` a `statistics_approximate`
+ object, and the ESA EOPF product root carries `stac_discovery`.
+ A flat numeric array is an ordinary CF vector attribute and is not special.
+ """
+ group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ (2,),
+ "f8",
+ 3,
+ attributes={
+ "statistics_approximate": {"min": 0.0, "max": 1.0},
+ "valid_range": [0.0, 1.0],
+ "nested_list": [[1, 2], [3, 4]],
+ },
+ )
+ attributes = wrap(array).attributes
+ assert attributes["statistics_approximate"] == {"min": 0.0, "max": 1.0}
+ assert attributes["valid_range"] == [0.0, 1.0]
+ assert attributes["nested_list"] == [[1, 2], [3, 4]]
+```
+
+- [ ] **Step 11: Run the module**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/ -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 12: Commit**
+
+Run §5.0, then:
+
+```bash
+git add lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr
+git commit -m "Present a Zarr array through the CF dataset interface
+
+The dimension contract is the substance: a Zarr dimension may be unnamed,
+and an unnamed one is anonymous rather than an error. _ARRAY_DIMENSIONS is
+storage, so it is withheld from the CF attributes, and fill_value is
+reported as the storage fact it is rather than as a missing-value flag."
+```
+
+---
+### Task 4: `ZarrDataset` — the group, its variables and its dimensions
+
+**Files:**
+- Modify: `lib/iris/fileformats/zarr/_dataset.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py`
+
+**Interfaces:**
+- Consumes: `ZarrDatasetVariable`, `_ZarrAttributes`,
+ `_FORMAT_BY_ZARR_VERSION` (Task 3).
+- Produces: `ZarrDataset(location, mode="r", *, group=None)` with
+ `location`, `format`, `mode`, `closed`, `variables`, `dimensions`,
+ `encoding`, `attributes`, `close`, `sync`, and the Zarr-only
+ `subgroup_names`. The write members exist and raise.
+
+Spec §4.4's second rule is the substance here: **`CFDataset.dimensions` is the
+names that are used consistently.** A name that appears at two different sizes
+is not an error and is not silently merged — it is omitted from the mapping,
+with an `IrisCfLoadWarning` naming the dimension and the sizes. The variables
+that use it keep it in their own `dimensions`, where the `(name, size)` rule of
+Task 8 stops it linking them.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDataset`."""
+
+import numpy as np
+import pytest
+import zarr
+
+from iris.fileformats.zarr._dataset import ZarrDataset
+from iris.warnings import IrisCfLoadWarning
+
+from .. import stores
+
+
+@pytest.fixture(params=[2, 3], ids=["v2", "v3"])
+def zarr_format(request):
+ return request.param
+
+
+def test_opens_and_reports_itself(tmp_path, zarr_format):
+ location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format)
+ dataset = ZarrDataset(location)
+
+ assert dataset.location == location
+ assert dataset.format == f"ZARR_V{zarr_format}"
+ assert dataset.mode == "r"
+ assert dataset.closed is False
+ assert sorted(dataset.variables) == ["air_temperature", "grid"]
+ assert dict(dataset.attributes) == stores.GLOBALS
+ assert dataset.encoding["zarr_format"] == zarr_format
+
+
+def test_dimensions_are_assembled_from_the_arrays(tmp_path, zarr_format):
+ location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format)
+ assert dict(ZarrDataset(location).dimensions) == stores.DIMENSIONS
+
+
+def test_a_name_used_at_two_sizes_is_omitted_and_warns(tmp_path):
+ """Two arrays may share a dimension name at different lengths.
+
+ The version 3 specification "does not place any restrictions on the use of
+ the same dimension name across multiple arrays", so this is conforming
+ Zarr and must not raise. Merging the two would produce a wrong cube.
+ """
+ group = zarr.open_group(store=tmp_path / "c.zarr", mode="w", zarr_format=3)
+ stores.make_array(group, "short", (2,), "f8", 3, dimensions=("t",))
+ stores.make_array(group, "long", (5,), "f8", 3, dimensions=("t",))
+
+ with pytest.warns(IrisCfLoadWarning, match="t"):
+ dimensions = dict(ZarrDataset(str(tmp_path / "c.zarr")).dimensions)
+
+ assert "t" not in dimensions
+
+
+def test_anonymous_dimensions_are_not_in_the_mapping(tmp_path):
+ group = zarr.open_group(store=tmp_path / "a.zarr", mode="w", zarr_format=3)
+ stores.make_array(group, "named", (2,), "f8", 3, dimensions=("t",))
+ stores.make_array(group, "unnamed", (4,), "f8", 3)
+
+ dataset = ZarrDataset(str(tmp_path / "a.zarr"))
+ assert dict(dataset.dimensions) == {"t": 2}
+
+
+def test_close_is_idempotent(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ dataset = ZarrDataset(location)
+ dataset.close()
+ dataset.close()
+ assert dataset.closed is True
+
+
+def test_context_manager_closes(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ dataset = ZarrDataset(location)
+ with dataset as entered:
+ assert entered is dataset
+ assert dataset.closed is True
+
+
+def test_group_keyword_selects_a_subgroup(tmp_path):
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ sub = root.create_group("measurements/r10m")
+ stores.make_array(sub, "b02", (2,), "u2", 3, dimensions=("p",))
+
+ dataset = ZarrDataset(str(tmp_path / "n.zarr"), group="measurements/r10m")
+ assert list(dataset.variables) == ["b02"]
+ assert dataset.location.endswith("measurements/r10m")
+
+
+def test_subgroup_names_lists_only_the_groups(tmp_path):
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ root.create_group("conditions")
+ root.create_group("measurements")
+ stores.make_array(root, "v", (2,), "f8", 3)
+
+ dataset = ZarrDataset(str(tmp_path / "n.zarr"))
+ assert sorted(dataset.subgroup_names) == ["conditions", "measurements"]
+
+
+def test_nested_groups_are_not_walked(tmp_path):
+ """Iris loads one group per call, like xarray.
+
+ A Zarr store can legitimately hold unrelated datasets, and merging them
+ into one CubeList would be wrong.
+ """
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ stores.make_array(root, "top", (2,), "f8", 3)
+ stores.make_array(root.create_group("sub"), "inner", (2,), "f8", 3)
+
+ assert list(ZarrDataset(str(tmp_path / "n.zarr")).variables) == ["top"]
+
+
+def test_the_write_surface_refuses(tmp_path):
+ # Saving to Zarr is not implemented. The members exist because the
+ # interface declares them; refusing is the honest answer.
+ dataset = ZarrDataset(stores.make_store(tmp_path / "s.zarr"))
+ with pytest.raises(NotImplementedError, match="Saving"):
+ dataset.create_dimension("t", 3)
+ with pytest.raises(NotImplementedError, match="Saving"):
+ dataset.create_variable("v", np.dtype("f4"), ("t",))
+ with pytest.raises(NotImplementedError, match="Saving"):
+ dataset.finalise()
+
+
+def test_a_write_mode_refuses(tmp_path):
+ stores.make_store(tmp_path / "s.zarr")
+ with pytest.raises(NotImplementedError, match="Saving"):
+ ZarrDataset(str(tmp_path / "s.zarr"), mode="w")
+```
+
+- [ ] **Step 2: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py -v
+```
+
+Expected: collection error, `ImportError: cannot import name 'ZarrDataset'`.
+
+- [ ] **Step 3: Write the dimension assembly**
+
+Append to `lib/iris/fileformats/zarr/_dataset.py`, after `_dimension_names`:
+
+```python
+def _assemble_dimensions(variables) -> dict:
+ """Return the dimension sizes the store's arrays agree on.
+
+ Parameters
+ ----------
+ variables : mapping of str to :class:`ZarrDatasetVariable`
+ Every array in one group.
+
+ Returns
+ -------
+ dict
+ Name to size, for every name used at exactly one size. Anonymous
+ dimensions have no name and so are absent.
+
+ Warns
+ -----
+ IrisCfLoadWarning
+ For each name used at more than one size. Zarr places no restriction
+ on reusing a name across arrays, so this is conforming and must not
+ raise; merging the two would produce a wrong cube, so the name is
+ omitted instead and the arrays that use it keep it in their own
+ `dimensions`.
+
+ """
+ sizes: dict[str, int] = {}
+ conflicts: dict[str, set[int]] = {}
+ for variable in variables.values():
+ for name, size in zip(variable.dimensions, variable.shape):
+ if name is None:
+ continue
+ if name not in sizes:
+ sizes[name] = size
+ elif sizes[name] != size:
+ conflicts.setdefault(name, {sizes[name]}).add(size)
+
+ for name, found in conflicts.items():
+ message = (
+ "Ignoring a dimension name used at more than one size, "
+ f"{sorted(found)}: {name}"
+ )
+ warnings.warn(message, category=iris.warnings.IrisCfLoadWarning)
+ del sizes[name]
+
+ return sizes
+```
+
+- [ ] **Step 4: Write the class**
+
+Append to `_dataset.py`:
+
+```python
+#: What `mode` a dataset may be opened in. Writing is not implemented.
+_READ_MODES = ("r",)
+
+#: The one message every unimplemented write member raises.
+_NO_SAVING = "Saving to Zarr is not supported."
+
+
+class ZarrDataset(CFDataset):
+ """A Zarr store, or one group of one, presented through the CF interface."""
+
+ def __init__(self, location, mode: str = "r", *, group: str | None = None):
+ """Open a Zarr store.
+
+ Parameters
+ ----------
+ location : str or :class:`pathlib.Path`
+ The store's path or URL. A URL is handed to zarr-python
+ unaltered; Iris adds no store library and handles no credentials,
+ so an unreachable scheme fails with zarr-python's own message.
+ mode : str, default="r"
+ Only ``"r"`` is supported.
+ group : str, optional
+ A ``/``-separated path to a group within the store. The root group
+ by default. Iris reads one group per call and does not walk
+ nested groups.
+
+ Raises
+ ------
+ NotImplementedError
+ For any `mode` other than ``"r"``.
+ FileNotFoundError
+ If `location` does not exist.
+ zarr.errors.GroupNotFoundError
+ If `location` is not a Zarr store, or holds no such `group`.
+
+ """
+ # Set first, so that close() is safe if opening fails.
+ self._group = None
+ self._closed = False
+ self._variables: dict[str, ZarrDatasetVariable] | None = None
+ self._attributes: _ZarrAttributes | None = None
+ self._dimensions: dict[str, int] | None = None
+
+ if mode not in _READ_MODES:
+ raise NotImplementedError(f"{_NO_SAVING} Cannot open with mode={mode!r}.")
+
+ self._mode = mode
+ self._group_path = group
+ target = str(location)
+ self._location = target if group is None else f"{target.rstrip('/')}/{group}"
+
+ self._group = zarr.open_group(store=target, mode=mode, path=group)
+ self._format = _FORMAT_BY_ZARR_VERSION[self._group.metadata.zarr_format]
+
+ @property
+ def location(self) -> str:
+ """The store's path or URL, with the group path when there is one."""
+ return self._location
+
+ @property
+ def format(self) -> str:
+ """``ZARR_V2`` or ``ZARR_V3``."""
+ return self._format
+
+ @property
+ def mode(self) -> str:
+ """The mode the store was opened in."""
+ return self._mode
+
+ @property
+ def closed(self) -> bool:
+ """Whether :meth:`close` has been called.
+
+ A Zarr store is a namespace rather than an open file handle, so there
+ is nothing to release and nothing else to ask.
+
+ """
+ return self._closed
+
+ @property
+ def variables(self) -> Mapping:
+ """The group's arrays, by name. Arrays in subgroups are not included."""
+ if self._variables is None:
+ self._variables = {
+ name: ZarrDatasetVariable(array, self._location, self._format)
+ for name, array in self._group.arrays()
+ }
+ return self._variables
+
+ @property
+ def dimensions(self) -> Mapping:
+ """The dimension sizes this group's arrays agree on."""
+ if self._dimensions is None:
+ self._dimensions = _assemble_dimensions(self.variables)
+ return self._dimensions
+
+ @property
+ def encoding(self) -> Mapping:
+ """The store-wide Zarr settings."""
+ consolidated = self._group.metadata.consolidated_metadata
+ return MappingProxyType(
+ {
+ "zarr_format": self._group.metadata.zarr_format,
+ "consolidated": consolidated is not None,
+ }
+ )
+
+ @property
+ def attributes(self) -> MutableMapping:
+ """The group's attributes."""
+ if self._attributes is None:
+ self._attributes = _ZarrAttributes(self._group)
+ return self._attributes
+
+ def create_dimension(self, name: str, size: int | None) -> None:
+ """Raise: saving to Zarr is not supported."""
+ raise NotImplementedError(_NO_SAVING)
+
+ def create_variable(
+ self, name: str, dtype, dimensions=(), *, fill_value=None, **encoding
+ ):
+ """Raise: saving to Zarr is not supported."""
+ raise NotImplementedError(_NO_SAVING)
+
+ def sync(self) -> None:
+ """Do nothing: a Zarr write is not buffered."""
+
+ def finalise(self) -> None:
+ """Raise: saving to Zarr is not supported.
+
+ A Zarr store's completion step is consolidating its metadata, which
+ only a writer performs.
+
+ """
+ raise NotImplementedError(_NO_SAVING)
+
+ def close(self) -> None:
+ """Record that this dataset is finished with.
+
+ A store is a namespace, not an open handle, so nothing is released.
+ The flag exists because callers - :class:`iris.fileformats.cf.CFReader`
+ among them - must be able to ask.
+
+ """
+ self._closed = True
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self.location!r}, mode={self.mode!r})"
+
+ # Zarr-only members below.
+
+ @property
+ def subgroup_names(self) -> tuple:
+ """The names of this group's subgroups.
+
+ What the loader names when a group holds no data variables but the
+ store plainly holds data elsewhere.
+
+ """
+ return tuple(self._group.group_keys())
+```
+
+- [ ] **Step 5: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 6: Add the remote and error-path tests**
+
+Spec §6: "Remote stores are covered by an in-memory store and a local
+directory store only. No test touches the network." Append to
+`test_ZarrDataset.py`:
+
+```python
+def test_an_in_memory_store_is_just_another_location(tmp_path):
+ """A store need not be a directory. zarr-python decides what it is."""
+ memory = zarr.storage.MemoryStore()
+ group = zarr.open_group(store=memory, mode="w", zarr_format=3)
+ stores.make_array(group, "v", (2,), "f8", 3, dimensions=("t",))
+
+ dataset = ZarrDataset(memory)
+ assert list(dataset.variables) == ["v"]
+
+
+def test_a_missing_store_raises_zarrs_own_error(tmp_path):
+ with pytest.raises(FileNotFoundError):
+ ZarrDataset(str(tmp_path / "nothing.zarr"))
+
+
+def test_a_directory_that_is_not_a_store_raises(tmp_path):
+ (tmp_path / "plain").mkdir()
+ with pytest.raises(zarr.errors.GroupNotFoundError):
+ ZarrDataset(str(tmp_path / "plain"))
+
+
+def test_a_missing_group_raises(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ with pytest.raises(zarr.errors.GroupNotFoundError):
+ ZarrDataset(location, group="nope")
+```
+
+`ZarrDataset(memory)` passes a store object where the signature says "str or
+Path". That is deliberate and documented in the docstring's `location`
+paragraph as "handed to zarr-python": `str(store)` of a `MemoryStore` is not a
+path, so `location` becomes zarr-python's own rendering of it, which is the
+best name available for a store with no address. Add that sentence to the
+`location` docstring when this test is written.
+
+- [ ] **Step 7: Run and commit**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/ -v
+```
+
+Expected: all pass. Then run §5.0 and:
+
+```bash
+git add lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr
+git commit -m "Present a Zarr group through the CF dataset interface
+
+Dimensions are assembled from the arrays, and a name used at two sizes is
+omitted with a warning rather than merged: Zarr places no restriction on
+reusing a name across arrays, so a store meaning two things by one name is
+conforming, and merging them would produce a wrong cube."
+```
+
+---
+
+### Task 5: the shared read contract, run against Zarr
+
+**Files:**
+- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py`
+
+**Interfaces:**
+- Consumes: `CFDatasetReadContract` (Task 2); `ZarrDataset` (Task 4);
+ `stores.make_store` (Task 1).
+- Produces: nothing other than coverage. This task exists so that a
+ netCDF/Zarr divergence fails loudly, which is spec §6's whole argument for a
+ shared body.
+
+The contract's canonical store is the one `stores.make_store` writes, which is
+why Task 1 wrote it to match `contract.populate()` exactly — same dimensions,
+same data, same attributes, same scalar `grid` variable.
+
+- [ ] **Step 1: Write the test module**
+
+Create
+`lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The shared CFDataset read contract, asked of the Zarr implementation.
+
+Only the read half: the Zarr write path does not exist, and `ZarrDataset`
+refuses every member of the write surface. The write half of the contract
+arrives with the saver.
+"""
+
+import pytest
+
+from iris.fileformats.zarr._dataset import ZarrDataset
+from iris.tests.unit.fileformats.cf.dataset import contract
+
+from .. import stores
+
+
+class TestZarrDatasetVersion3(contract.CFDatasetReadContract):
+ @pytest.fixture
+ def readable(self, tmp_path):
+ return ZarrDataset(stores.make_store(tmp_path / "contract.zarr"))
+
+
+class TestZarrDatasetVersion2(contract.CFDatasetReadContract):
+ @pytest.fixture
+ def readable(self, tmp_path):
+ location = stores.make_store(tmp_path / "contract.zarr", zarr_format=2)
+ return ZarrDataset(location)
+```
+
+- [ ] **Step 2: Run it, and expect disagreements**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py -v
+```
+
+This is the step that earns the task. Expect failures, and resolve each one
+by deciding **which side is wrong**, not by loosening the contract:
+
+- `test_variable_fill_value` asserts `-1.0`, which `stores.make_store` sets as
+ the array's `fill_value`. Zarr answers from storage and netCDF from the
+ `_FillValue` attribute (§4, "`fill_value` means different things"), so both
+ give `-1.0` here and the test should pass. If it does not, the Zarr store
+ builder is not setting what it claims to.
+- `test_variable_chunking_is_none_or_a_shape` already admits `np.integer`,
+ which was written for Zarr.
+- `test_mode_is_a_read_mode` asserts `mode in ("r", "r+", "a")`; `ZarrDataset`
+ answers `"r"`.
+- `test_scalar_variable` expects `len(scalar)` to raise `TypeError` matching
+ "unsized". `CFDatasetVariable.__len__` supplies that for any implementation
+ with an empty `shape`, and `ZarrDatasetVariable` does not override it.
+
+Record each resolution in the commit message. A contract test that needs a
+Zarr-shaped exception is a design signal, not a test problem.
+
+- [ ] **Step 3: Confirm the netCDF side is untouched**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/dataset/ -q
+```
+
+Expected: the same count as at the end of Task 2.
+
+- [ ] **Step 4: Add the no-isinstance check**
+
+Spec §6: "A separate check greps the `cf` package for `isinstance` against an
+implementation class and asserts there are none — the rule in §4.2 is only
+worth writing down if something notices when it is broken."
+
+Create `lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The CF layer must not branch on which storage implementation it holds.
+
+`isinstance(thing, CFDataset)` tests the *interface*, which is how generic
+code is supposed to accept any backend, and `CFReader` does exactly that.
+What is banned is testing an *implementation*: that is what makes a third
+backend a rewrite rather than a new module. Constructing one is a third
+thing again, and legitimate - `CFReader` opens a `NetCDFDataset` for a bare
+path, which is a default, not a branch on something it was handed.
+"""
+
+from pathlib import Path
+
+import iris.fileformats.cf
+
+#: Implementation class names that may not appear in a type test.
+IMPLEMENTATION_NAMES = (
+ "NetCDFDataset",
+ "NetCDFDatasetVariable",
+ "ZarrDataset",
+ "ZarrDatasetVariable",
+)
+
+#: The spellings of a type test. `__class__` catches the hand-rolled form.
+TYPE_TESTS = ("isinstance", "issubclass", "__class__", "type(")
+
+
+def cf_package_sources():
+ """Yield `(path, text)` for every module in `iris.fileformats.cf`."""
+ package = Path(iris.fileformats.cf.__file__).parent
+ for path in sorted(package.glob("*.py")):
+ yield path, path.read_text(encoding="utf-8")
+
+
+def test_no_module_type_tests_an_implementation_class():
+ offenders = []
+ for path, text in cf_package_sources():
+ for line_number, line in enumerate(text.splitlines(), start=1):
+ if line.lstrip().startswith("#"):
+ continue
+ names_one = any(name in line for name in IMPLEMENTATION_NAMES)
+ tests_type = any(spelling in line for spelling in TYPE_TESTS)
+ if names_one and tests_type:
+ offenders.append(f"{path.name}:{line_number}: {line.strip()}")
+ assert offenders == []
+```
+
+This passes as written, and the way it fails is by someone adding the branch
+it forbids. Prove it can fail before trusting it: temporarily add
+`if isinstance(self._dataset, NetCDFDataset): pass` to `cf/_reader.py`, re-run,
+see one offender, and remove it.
+
+- [ ] **Step 5: Commit**
+
+Run §5.0, then:
+
+```bash
+git add lib/iris/tests/unit/fileformats
+git commit -m "Ask the shared CFDataset read contract of the Zarr store
+
+Both Zarr versions answer the same questions the netCDF implementation
+does, so a divergence between the two fails here rather than somewhere in
+the loader. Adds the check that no module in the cf package names an
+implementation class."
+```
+
+---
+### Task 6: `_decode.py` — the CF rules netCDF4 applies for free
+
+**Files:**
+- Create: `lib/iris/fileformats/zarr/_decode.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/test__decode.py`
+
+**Interfaces:**
+- Consumes: nothing. Every function here is pure and takes plain values, which
+ is the point: spec §6 wants decoding "unit-testable without a store at all".
+- Produces: `decode_fill_value(value, dtype, variable_name)`,
+ `decode_parameters(attributes, *, dtype, storage_fill_value, zarr_format,
+ variable_name)`, `decoding_required(parameters)`,
+ `mask_and_scale(block, *, sentinels, valid_min, valid_max, scale_factor,
+ add_offset, dtype)`, and the three rules it composes.
+
+Two rules from spec §4.4 are the substance:
+
+- **The CF attribute governs masking and the storage field does not** — except
+ on version 2, where `Array.fill_value` is the mask source when no CF
+ attribute is present, "because version 2 stores predate the convention and
+ this is what their writers meant". This matches released xarray's
+ `use_zarr_fill_value_as_mask`.
+- **A floating `_FillValue` is base64 on version 3, and that is a convention,
+ not a malformation.** `base64(struct.pack("<d", value))`, for *every*
+ floating dtype including `float32`. Released xarray requires that form and
+ raises `TypeError` for a plain JSON number. Iris reads both.
+
+`decode_parameters` takes plain arguments rather than a variable, so the
+version-aware precedence can be tested without building a store for each case.
+
+- [ ] **Step 1: Write the failing tests for `decode_fill_value`**
+
+Create `lib/iris/tests/unit/fileformats/zarr/test__decode.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.zarr._decode`."""
+
+import base64
+import struct
+
+import numpy as np
+import pytest
+
+from iris.fileformats.zarr import _decode
+from iris.warnings import IrisCfLoadWarning
+
+
+def base64_double(value):
+ """Return `value` in the encoding released xarray writes for a float."""
+ return base64.b64encode(struct.pack("<d", value)).decode("ascii")
+
+
+def test_fill_value_from_a_json_number():
+ result = _decode.decode_fill_value(-999, np.dtype("i2"), "v")
+ assert result == -999
+ assert result.dtype == np.dtype("i2")
+
+
+def test_fill_value_from_base64_nan():
+ """NOAA GFS writes 'AAAAAAAA+H8=' and it is not a CF violation.
+
+ base64(struct.pack("<d", float("nan"))) is that literal string, and
+ released xarray writes it for every floating dtype.
+ """
+ result = _decode.decode_fill_value(
+ "AAAAAAAA+H8=", np.dtype("f4"), "temperature_2m"
+ )
+ assert result.dtype == np.dtype("f4")
+ assert np.isnan(result)
+
+
+def test_fill_value_from_base64_finite_sentinel():
+ # Not only NaN: a finite sentinel must survive the narrowing too.
+ encoded = base64_double(-999.0)
+ result = _decode.decode_fill_value(encoded, np.dtype("f4"), "v")
+ assert result == np.float32(-999.0)
+
+
+def test_fill_value_from_a_json_number_on_a_float():
+ result = _decode.decode_fill_value(-999.0, np.dtype("f4"), "v")
+ assert result == np.float32(-999.0)
+
+
+def test_an_undecodable_fill_value_warns_and_is_ignored():
+ with pytest.warns(IrisCfLoadWarning, match="banana"):
+ result = _decode.decode_fill_value("banana", np.dtype("f4"), "v")
+ assert result is None
+
+
+def test_a_base64_string_of_the_wrong_width_warns():
+ encoded = base64.b64encode(b"abc").decode("ascii")
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ assert _decode.decode_fill_value(encoded, np.dtype("f4"), "v") is None
+
+
+def test_a_base64_string_on_an_integer_variable_warns():
+ # The convention is for floating dtypes. An integer variable's sentinel
+ # is a JSON number, and a string there is a producer error.
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ assert _decode.decode_fill_value(base64_double(1.0), np.dtype("i2"), "v") is None
+```
+
+- [ ] **Step 2: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/test__decode.py -v
+```
+
+Expected: collection error, `ImportError: cannot import name '_decode'`.
+
+- [ ] **Step 3: Write `decode_fill_value`**
+
+Create `lib/iris/fileformats/zarr/_decode.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The CF decoding netCDF4 does for free and Zarr does not.
+
+netCDF4 applies ``_FillValue``/``missing_value`` masking,
+``valid_min``/``valid_max``/``valid_range`` masking and
+``scale_factor``/``add_offset`` unpacking on every read. Zarr returns a plain
+``numpy.ndarray`` and applies none of it, so these rules are implemented here.
+They live on the Zarr side rather than in the shared CF layer because the
+netCDF path gets the behaviour from the C library and has nothing to reuse.
+
+Everything here is a pure function of values, with no store and no Dask in
+sight. :func:`mask_and_scale` is what the loader lays over a raw array as a
+``map_blocks`` layer - visible in the graph, fusable with its neighbours, and
+testable on a plain ndarray.
+
+Two traps, both from the Zarr specification rather than from CF.
+
+**A floating ``_FillValue`` is base64 on version 3.** Released xarray encodes
+it as base64 of a little-endian ``float64`` for *every* floating dtype,
+including ``float32``, and its reader requires that form. It is the de-facto
+encoding, not a producer's mistake, so Iris reads both it and a plain JSON
+number. CF section 2.5.1 asks for "the scalar attribute with the name
+``_FillValue`` and of the same type as its variable"; this is a deliberate,
+narrow deviation, taken because interoperability with the other Zarr readers
+is the reason for native Zarr support.
+
+**``Array.fill_value`` is not a missing-value declaration.** It says what
+unwritten storage reads back as, and zarr-python defaults it to zero. On
+version 3 it is never a mask source: an ``int16`` array of ``[0, 1, 2]`` with
+no CF attributes would otherwise load with its valid zero masked. On version 2
+it is one, but only when no CF attribute is present, because version 2 stores
+predate the convention - which is what released xarray's
+``use_zarr_fill_value_as_mask`` does.
+
+See section 4.4 of ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+import base64
+import binascii
+import struct
+from typing import Any
+import warnings
+
+import numpy as np
+
+import iris.warnings
+
+#: The width of the base64 `_FillValue` encoding: one little-endian float64.
+_PACKED_DOUBLE = "<d"
+_PACKED_DOUBLE_BYTES = 8
+
+#: The Zarr version whose stores predate the CF `_FillValue` convention, and
+#: whose `Array.fill_value` is therefore read as a missing-value sentinel.
+_FILL_VALUE_IS_A_MASK_BEFORE = 3
+
+
+def decode_fill_value(value: Any, dtype: np.dtype, variable_name: str) -> Any:
+ """Return a CF missing-value sentinel as a value of `dtype`.
+
+ Parameters
+ ----------
+ value : Any
+ The attribute's value, as the store holds it: a JSON number, or - for
+ a floating variable - the base64 of a little-endian ``float64``.
+ dtype : numpy.dtype
+ The variable's own dtype. The result is narrowed to it, because a
+ sentinel of a different width never equals anything in the array.
+ variable_name : str
+ Named in the warning, so the user can find the variable.
+
+ Returns
+ -------
+ numpy.generic or None
+ ``None`` when `value` is neither form, which is a producer error.
+
+ Warns
+ -----
+ IrisCfLoadWarning
+ When `value` cannot be read as a sentinel of `dtype`.
+
+ """
+ if isinstance(value, (bool, np.bool_)):
+ # A JSON true/false is a number to Python and nothing to CF.
+ decoded = None
+ elif isinstance(value, (int, float, np.number)):
+ decoded = value
+ elif isinstance(value, str) and dtype.kind == "f":
+ decoded = _decode_packed_double(value)
+ else:
+ decoded = None
+
+ if decoded is None:
+ message = (
+ "Ignoring a missing-value attribute that is neither a number nor "
+ f"the base64 of a float64, on variable {variable_name}: {value!r}"
+ )
+ warnings.warn(message, category=iris.warnings.IrisCfLoadWarning)
+ return None
+
+ return dtype.type(decoded)
+
+
+def _decode_packed_double(value: str) -> float | None:
+ """Return the float in a base64 little-endian float64, or None."""
+ try:
+ raw = base64.b64decode(value, validate=True)
+ except (binascii.Error, ValueError):
+ return None
+ if len(raw) != _PACKED_DOUBLE_BYTES:
+ return None
+ return struct.unpack(_PACKED_DOUBLE, raw)[0]
+```
+
+- [ ] **Step 4: Run the first group**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/test__decode.py -v
+```
+
+Expected: the seven `decode_fill_value` tests pass.
+
+- [ ] **Step 5: Write the failing tests for the masking precedence**
+
+Append to `test__decode.py`:
+
+```python
+def parameters(attributes, *, dtype="i2", storage_fill_value=0, zarr_format=3):
+ return _decode.decode_parameters(
+ attributes,
+ dtype=np.dtype(dtype),
+ storage_fill_value=storage_fill_value,
+ zarr_format=zarr_format,
+ variable_name="v",
+ )
+
+
+def test_version_3_ignores_the_storage_fill_value():
+ """An int16 [0, 1, 2] with no CF attributes loads with nothing masked.
+
+ zarr-python defaults `fill_value` to zero, so reading it as a sentinel
+ masks valid data in the ordinary case, not the exotic one.
+ """
+ assert parameters({}, storage_fill_value=0)["sentinels"] == ()
+
+
+def test_version_2_uses_the_storage_fill_value_when_there_is_no_attribute():
+ result = parameters({}, storage_fill_value=0, zarr_format=2)
+ assert result["sentinels"] == (0,)
+
+
+def test_a_cf_attribute_beats_the_storage_fill_value_on_version_2():
+ result = parameters(
+ {"_FillValue": -999}, storage_fill_value=0, zarr_format=2
+ )
+ assert result["sentinels"] == (np.int16(-999),)
+
+
+def test_both_cf_missing_value_attributes_are_sentinels():
+ result = parameters({"_FillValue": -999, "missing_value": -888})
+ assert set(result["sentinels"]) == {np.int16(-999), np.int16(-888)}
+
+
+def test_valid_range_sets_both_bounds():
+ result = parameters({"valid_range": [0, 10]})
+ assert result["valid_min"] == 0
+ assert result["valid_max"] == 10
+
+
+def test_valid_min_and_valid_max_are_read_separately():
+ result = parameters({"valid_min": 1, "valid_max": 9})
+ assert (result["valid_min"], result["valid_max"]) == (1, 9)
+
+
+def test_packing_parameters_and_the_resulting_dtype():
+ result = parameters({"scale_factor": 0.0001, "add_offset": 1.0}, dtype="u2")
+ assert result["scale_factor"] == 0.0001
+ assert result["add_offset"] == 1.0
+ assert result["dtype"] == np.dtype("f8")
+
+
+def test_decoding_required_is_false_for_a_plain_array():
+ assert _decode.decoding_required(parameters({})) is False
+
+
+def test_decoding_required_is_true_for_each_rule():
+ assert _decode.decoding_required(parameters({"_FillValue": -999})) is True
+ assert _decode.decoding_required(parameters({"valid_min": 1})) is True
+ assert _decode.decoding_required(parameters({"scale_factor": 2})) is True
+```
+
+- [ ] **Step 6: Write `decode_parameters` and `decoding_required`**
+
+Append to `_decode.py`:
+
+```python
+def decode_parameters(
+ attributes,
+ *,
+ dtype: np.dtype,
+ storage_fill_value: Any,
+ zarr_format: int,
+ variable_name: str,
+) -> dict:
+ """Return the CF decode parameters for one variable.
+
+ Parameters
+ ----------
+ attributes : mapping
+ The variable's CF attributes.
+ dtype : numpy.dtype
+ The variable's stored dtype, before any unpacking.
+ storage_fill_value : Any
+ ``zarr.Array.fill_value`` - what unwritten storage reads back as. A
+ mask source only on version 2, and only when the variable declares no
+ CF missing-value attribute.
+ zarr_format : int
+ 2 or 3. The one place the version changes what is decoded.
+ variable_name : str
+ Named in any warning.
+
+ Returns
+ -------
+ dict
+ The keyword arguments :func:`mask_and_scale` takes, always with the
+ same keys. :func:`decoding_required` says whether they do anything.
+
+ """
+ sentinels = []
+ for name in ("_FillValue", "missing_value"):
+ if name in attributes:
+ decoded = decode_fill_value(attributes[name], dtype, variable_name)
+ if decoded is not None:
+ sentinels.append(decoded)
+
+ if (
+ not sentinels
+ and zarr_format < _FILL_VALUE_IS_A_MASK_BEFORE
+ and storage_fill_value is not None
+ ):
+ sentinels.append(storage_fill_value)
+
+ valid_min = attributes.get("valid_min")
+ valid_max = attributes.get("valid_max")
+ valid_range = attributes.get("valid_range")
+ if valid_range is not None and len(valid_range) == 2:
+ valid_min, valid_max = valid_range
+
+ scale_factor = attributes.get("scale_factor")
+ add_offset = attributes.get("add_offset")
+
+ return {
+ "sentinels": tuple(sentinels),
+ "valid_min": valid_min,
+ "valid_max": valid_max,
+ "scale_factor": scale_factor,
+ "add_offset": add_offset,
+ "dtype": _unpacked_dtype(dtype, scale_factor, add_offset),
+ }
+
+
+def _unpacked_dtype(dtype, scale_factor, add_offset) -> np.dtype:
+ """Return the dtype the unpacked values will have.
+
+ The same calculation as
+ :func:`iris.fileformats.cf.loader._get_actual_dtype`, done here on plain
+ values so that the dtype the Dask graph declares and the dtype the decode
+ layer produces are one statement rather than two.
+ """
+ probe = np.zeros(1, dtype=dtype)
+ if scale_factor is not None:
+ probe = scale_factor * probe
+ if add_offset is not None:
+ probe = add_offset + probe
+ return probe.dtype
+
+
+def decoding_required(parameters: dict) -> bool:
+ """Whether these parameters would change anything.
+
+ A variable with nothing to decode gets no ``map_blocks`` layer and an
+ unmasked Dask meta. Adding a masked meta without a layer beneath it would
+ be a meta that lies about its own graph.
+ """
+ if parameters["sentinels"]:
+ return True
+ return any(
+ parameters[name] is not None
+ for name in ("valid_min", "valid_max", "scale_factor", "add_offset")
+ )
+```
+
+- [ ] **Step 7: Run, then write the failing tests for `mask_and_scale`**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/test__decode.py -v
+```
+
+Expected: all pass so far. Then append:
+
+```python
+def test_mask_by_sentinels():
+ block = np.array([0, 1, -999], dtype="i2")
+ result = _decode.mask_by_sentinels(block, (np.int16(-999),))
+ assert result.tolist() == [False, False, True]
+
+
+def test_mask_by_a_nan_sentinel():
+ """NaN never equals itself, so a NaN sentinel needs `isnan`, not `==`.
+
+ The base64 `_FillValue` nearly every xarray-written store carries is NaN,
+ so this is the common case rather than the exotic one.
+ """
+ block = np.array([1.0, np.nan], dtype="f4")
+ result = _decode.mask_by_sentinels(block, (np.float32("nan"),))
+ assert result.tolist() == [False, True]
+
+
+def test_mask_by_valid_range():
+ block = np.array([-1, 5, 11], dtype="i2")
+ result = _decode.mask_by_valid_range(block, 0, 10)
+ assert result.tolist() == [True, False, True]
+
+
+def test_unpack():
+ block = np.array([1, 2], dtype="u2")
+ result = _decode.unpack(block, 0.5, 1.0, np.dtype("f8"))
+ np.testing.assert_allclose(result, [1.5, 2.0])
+ assert result.dtype == np.dtype("f8")
+
+
+def test_mask_and_scale_masks_the_packed_values():
+ """Masking is applied before unpacking, as netCDF4 does it.
+
+ CF section 2.5.1: `_FillValue` is "of the same type as its variable", so
+ it is a packed value and comparing it with an unpacked one matches
+ nothing.
+ """
+ block = np.array([-999, 1, 2], dtype="i2")
+ result = _decode.mask_and_scale(
+ block,
+ sentinels=(np.int16(-999),),
+ valid_min=None,
+ valid_max=None,
+ scale_factor=0.5,
+ add_offset=0.0,
+ dtype=np.dtype("f8"),
+ )
+ assert result.mask.tolist() == [True, False, False]
+ np.testing.assert_allclose(result.compressed(), [0.5, 1.0])
+
+
+def test_mask_and_scale_always_returns_a_masked_array():
+ # The Dask meta says masked, so every block must be, mask or no mask.
+ block = np.array([1, 2], dtype="i2")
+ result = _decode.mask_and_scale(
+ block,
+ sentinels=(np.int16(-999),),
+ valid_min=None,
+ valid_max=None,
+ scale_factor=None,
+ add_offset=None,
+ dtype=np.dtype("i2"),
+ )
+ assert isinstance(result, np.ma.MaskedArray)
+ assert not result.mask.any()
+```
+
+- [ ] **Step 8: Write the masking and unpacking functions**
+
+Append to `_decode.py`:
+
+```python
+def mask_by_sentinels(block: np.ndarray, sentinels: tuple) -> np.ndarray:
+ """Return a boolean mask of the elements equal to any sentinel.
+
+ A NaN sentinel is matched with :func:`numpy.isnan`, because NaN does not
+ equal itself and NaN is what nearly every version 3 store's base64
+ ``_FillValue`` decodes to.
+ """
+ mask = np.zeros(block.shape, dtype=bool)
+ for sentinel in sentinels:
+ if isinstance(sentinel, (float, np.floating)) and np.isnan(sentinel):
+ mask |= np.isnan(block)
+ else:
+ mask |= block == sentinel
+ return mask
+
+
+def mask_by_valid_range(block: np.ndarray, valid_min, valid_max) -> np.ndarray:
+ """Return a boolean mask of the elements outside the valid range."""
+ mask = np.zeros(block.shape, dtype=bool)
+ if valid_min is not None:
+ mask |= block < valid_min
+ if valid_max is not None:
+ mask |= block > valid_max
+ return mask
+
+
+def unpack(block: np.ndarray, scale_factor, add_offset, dtype: np.dtype) -> np.ndarray:
+ """Return `block` unpacked by CF's scale and offset, as `dtype`."""
+ result = block
+ if scale_factor is not None:
+ result = scale_factor * result
+ if add_offset is not None:
+ result = add_offset + result
+ return result.astype(dtype, copy=False)
+
+
+def mask_and_scale(
+ block: np.ndarray,
+ *,
+ sentinels: tuple,
+ valid_min,
+ valid_max,
+ scale_factor,
+ add_offset,
+ dtype: np.dtype,
+) -> np.ma.MaskedArray:
+ """Apply the CF decode rules to one block, in CF's order.
+
+ Masking first, on the **packed** values, because CF declares a sentinel to
+ be "of the same type as its variable"; unpacking second. The result is
+ always a masked array, so that every block matches the Dask meta the
+ loader declares for the layer.
+
+ Parameters
+ ----------
+ block : numpy.ndarray
+ One Dask block of the raw stored values.
+ sentinels : tuple
+ Values meaning "missing", already narrowed to the stored dtype.
+ valid_min, valid_max : optional
+ CF's valid range. Values outside it are masked.
+ scale_factor, add_offset : optional
+ CF's packing. ``value * scale_factor + add_offset``.
+ dtype : numpy.dtype
+ The dtype of the result, which the Dask graph has already declared.
+
+ Returns
+ -------
+ numpy.ma.MaskedArray
+
+ """
+ mask = mask_by_sentinels(block, sentinels)
+ mask |= mask_by_valid_range(block, valid_min, valid_max)
+ return np.ma.masked_array(
+ unpack(block, scale_factor, add_offset, dtype), mask=mask
+ )
+```
+
+- [ ] **Step 9: Run the module**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/test__decode.py -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 10: Commit**
+
+Run §5.0, then:
+
+```bash
+git add lib/iris/fileformats/zarr/_decode.py lib/iris/tests/unit/fileformats/zarr/test__decode.py
+git commit -m "Implement the CF decoding Zarr does not do for itself
+
+Masking, valid-range masking and scale/offset unpacking, as pure functions
+of values. The two rules worth reading twice: a floating _FillValue is
+base64 on version 3, which is xarray's convention rather than a defect;
+and Array.fill_value is a mask source on version 2 only, and only when the
+variable declares no CF attribute of its own."
+```
+
+---
+
+### Task 7: `read_data` — laziness, chunk alignment and the cache key
+
+> The chunk-alignment half of this task was reversed on review. See "Task 7,
+> reversed after review" under *What changed during implementation*. The task
+> text below is left as it was executed.
+
+**Files:**
+- Modify: `lib/iris/fileformats/zarr/_dataset.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py`
+
+**Interfaces:**
+- Consumes: `_decode.decode_parameters`, `decoding_required`, `mask_and_scale`
+ (Task 6); `ZarrDatasetVariable` (Task 3);
+ `iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES` and `_get_actual_dtype`, and
+ `iris._lazy_data.as_lazy_data(..., cache_key=...)` (**PR 3**).
+- Produces: `ZarrDatasetVariable.read_data(chunking_policy)`,
+ `ZarrDatasetVariable.cache_key`, `ZarrDatasetVariable.zarr_format`, and the
+ module function `_align_chunks(chunks, store_chunks, shape, dtype,
+ dims_fixed, name)`.
+
+Three spec rules meet here.
+
+**No proxy.** `zarr.Array` is picklable, a chunk is an independent object in a
+key-value store, and a store is a namespace rather than an open handle, so
+every reason `NetCDFDataProxy` exists is absent. The array goes to
+`as_lazy_data` directly.
+
+**Alignment aggregates upward only.** `_optimum_chunksize` expands a small
+store chunk to a whole multiple, which is right; but when the store chunk is
+*larger* than dask's target it shrinks below it, and subdividing a Zarr chunk
+buys nothing — a read of any part of one fetches and decompresses the whole
+object (F7 has the two reproductions). So the computed chunking is rounded
+**up** to a whole multiple of the store chunk, never below, and the variable
+warns when the store's own chunking exceeds the dask target.
+
+**The key carries the metadata, not just the address.** `CACHE` is a
+process-wide `LRUCache(100)`, so an address goes stale the moment a store is
+rewritten in place.
+
+- [ ] **Step 1: Write the failing tests for `_align_chunks`**
+
+Create `lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the Zarr read-side chunk alignment.
+
+No store is built here: the guard is a function of chunk shapes, which is
+most of the point of having it as one.
+"""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.zarr._dataset import _align_chunks
+from iris.warnings import IrisLoadWarning
+
+F8 = np.dtype("f8")
+
+
+def test_a_whole_multiple_is_left_alone():
+ assert _align_chunks((4096, 4096), (512, 512), (4096, 4096), F8, None, "v") == (
+ 4096,
+ 4096,
+ )
+
+
+def test_a_sub_multiple_is_rounded_up():
+ """A dask chunk that subdivides a store chunk costs N re-reads.
+
+ A read of any part of a Zarr chunk fetches and decompresses the whole
+ object, so N dask blocks over one store chunk is N times the bytes and N
+ times the CPU - and, over a remote store, N HTTP GETs for one object.
+ """
+ with pytest.warns(IrisLoadWarning, match="v"):
+ result = _align_chunks((2000, 8000), (8000, 8000), (16000, 16000), F8, None, "v")
+ assert result == (8000, 8000)
+
+
+def test_rounding_up_is_capped_at_the_shape():
+ assert _align_chunks((2000, 6000), (6000, 6000), (6000, 6000), F8, None, "v") == (
+ 6000,
+ 6000,
+ )
+
+
+def test_an_unchunked_store_is_left_alone():
+ assert _align_chunks((10,), None, (100,), F8, None, "v") == (10,)
+
+
+def test_an_explicit_chunk_control_setting_wins_but_warns():
+ """A deliberate instruction is honoured, on the same terms.
+
+ CHUNK_CONTROL marks the dimensions it set as fixed. Overriding them would
+ make the keyword a suggestion.
+ """
+ with pytest.warns(IrisLoadWarning, match="v"):
+ result = _align_chunks(
+ (100, 8000), (8000, 8000), (16000, 16000), F8, (True, False), "v"
+ )
+ assert result[0] == 100
+
+
+def test_a_small_store_chunk_does_not_warn():
+ # Dask's target is the binding constraint here, which is the ordinary
+ # case and nothing for the user to act on.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ result = _align_chunks((1024, 1024), (512, 512), (4096, 4096), F8, None, "v")
+ assert result == (1024, 1024)
+
+
+@pytest.mark.parametrize(
+ "chunks, store_chunks, shape",
+ [
+ ((2000, 8000), (8000, 8000), (16000, 16000)),
+ ((1024, 1024), (512, 512), (4096, 4096)),
+ ((3000, 10000), (1000, 1000), (10000, 10000)),
+ ],
+)
+def test_the_result_tiles_the_store_grid_exactly(chunks, store_chunks, shape):
+ with warnings.catch_warnings():
+ warnings.simplefilter("ignore", IrisLoadWarning)
+ result = _align_chunks(chunks, store_chunks, shape, F8, None, "v")
+ for size, store_size, extent in zip(result, store_chunks, shape):
+ assert size % store_size == 0 or size == extent
+```
+
+`import warnings` goes at the top of the module with the others.
+
+- [ ] **Step 2: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py -v
+```
+
+Expected: `ImportError: cannot import name '_align_chunks'`.
+
+- [ ] **Step 3: Write `_align_chunks`**
+
+Append to `lib/iris/fileformats/zarr/_dataset.py`, after `_assemble_dimensions`:
+
+```python
+def _fixed_flags(dims_fixed, ndim: int) -> tuple:
+ """Return one "do not change this" flag per dimension.
+
+ `iris.fileformats.cf.loader._chunks_from_chunk_control` answers ``(None,)``
+ when nothing is fixed, whatever the rank, so a length that does not match
+ means nothing is fixed.
+ """
+ if dims_fixed is None or len(dims_fixed) != ndim:
+ return (False,) * ndim
+ return tuple(bool(flag) for flag in dims_fixed)
+
+
+def _align_chunks(
+ chunks: tuple,
+ store_chunks: tuple | None,
+ shape: tuple,
+ dtype: np.dtype,
+ dims_fixed,
+ name: str,
+) -> tuple:
+ """Round `chunks` up to whole multiples of the store's own chunk shape.
+
+ A Zarr chunk is the atomic unit of storage: reading any part of one
+ fetches and decompresses the whole object. So a Dask chunking that
+ subdivides the store's costs a re-read per subdivision - and over a remote
+ store, one HTTP GET per subdivision for the same object. Aggregating
+ upward costs nothing, because peak memory per task is a whole decompressed
+ store chunk either way.
+
+ Parameters
+ ----------
+ chunks : tuple of int
+ The chunking computed for this variable, already through
+ :func:`iris._lazy_data._optimum_chunksize`.
+ store_chunks : tuple of int, optional
+ The store's own chunk shape. ``None`` for an unchunked store, which
+ constrains nothing.
+ shape : tuple of int
+ The variable's full shape. A result is never larger than this.
+ dtype : numpy.dtype
+ The stored dtype, for deciding whether one store chunk already exceeds
+ Dask's target chunk size.
+ dims_fixed : sequence of bool, optional
+ Dimensions an explicit ``CHUNK_CONTROL`` setting pinned. These are
+ left exactly as given: the keyword is an instruction, not a
+ suggestion.
+ name : str
+ The variable's name, for the warning.
+
+ Returns
+ -------
+ tuple of int
+
+ Warns
+ -----
+ IrisLoadWarning
+ When the result cannot tile the store grid without subdividing it -
+ either because one store chunk already exceeds Dask's target size, so
+ the store's layout is the binding constraint, or because an explicit
+ setting subdivides it.
+
+ """
+ if store_chunks is None:
+ return tuple(chunks)
+
+ fixed = _fixed_flags(dims_fixed, len(shape))
+ aligned = []
+ subdivided = False
+ for size, store_size, extent, is_fixed in zip(chunks, store_chunks, shape, fixed):
+ if is_fixed:
+ aligned.append(size)
+ subdivided |= size < store_size
+ continue
+ whole = -(-size // store_size) * store_size
+ aligned.append(min(whole, extent))
+ subdivided |= whole > extent and extent % store_size != 0
+
+ if _store_chunk_exceeds_dask_target(store_chunks, dtype):
+ subdivided = True
+
+ if subdivided:
+ message = (
+ "A Zarr chunk is read whole, so this store's chunking is the "
+ "binding constraint on memory per task rather than Dask's target "
+ f"chunk size: {name}"
+ )
+ warnings.warn(message, category=iris.warnings.IrisLoadWarning)
+
+ return tuple(aligned)
+
+
+def _store_chunk_exceeds_dask_target(store_chunks: tuple, dtype: np.dtype) -> bool:
+ """Whether one whole store chunk is larger than Dask's target chunk size."""
+ target = dask.utils.parse_bytes(dask.config.get("array.chunk-size"))
+ return int(np.prod(store_chunks)) * dtype.itemsize > target
+```
+
+Add `import dask`, `import dask.config` and `import dask.utils` to the module
+imports.
+
+- [ ] **Step 4: Run the alignment tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test__align_chunks.py -v
+```
+
+Expected: all pass. The two regression cases are F7's: `(8000, 8000)` over
+`(16000, 16000)` and `(6000, 6000)` over `(6000, 6000)`, both of which
+`_optimum_chunksize` subdivides on its own.
+
+- [ ] **Step 5: Write the failing tests for `read_data`**
+
+Create `lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :meth:`ZarrDatasetVariable.read_data`."""
+
+import dask.array as da
+import numpy as np
+import pytest
+import zarr
+
+from iris.fileformats.cf import loader as cf_loader
+from iris.fileformats.zarr._dataset import ZarrDataset
+
+from .. import stores
+
+
+def variable(location, name="air_temperature"):
+ return ZarrDataset(location).variables[name]
+
+
+def policy():
+ """The chunking policy a loader outside any CHUNK_CONTROL supplies."""
+ return None, None
+
+
+def test_a_small_variable_is_read_whole(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ result = variable(location).read_data(policy)
+ assert not isinstance(result, da.Array)
+ np.testing.assert_array_equal(np.asarray(result), stores.DATA)
+
+
+def test_a_small_variable_never_consults_the_policy(tmp_path):
+ """Under CHUNK_CONTROL.from_file() the policy raises for an unchunked store.
+
+ A variable small enough to read whole must reach its data without asking,
+ or `from_file` starts rejecting small cubes that load today.
+ """
+
+ def raising_policy():
+ raise AssertionError("chunking policy consulted for a small variable")
+
+ location = stores.make_store(tmp_path / "s.zarr")
+ variable(location).read_data(raising_policy)
+
+
+def test_a_large_variable_is_lazy(tmp_path, monkeypatch):
+ monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ location = stores.make_store(tmp_path / "s.zarr")
+ result = variable(location).read_data(policy)
+ assert isinstance(result, da.Array)
+ np.testing.assert_array_equal(result.compute(), stores.DATA)
+
+
+def test_a_plain_array_gets_an_unmasked_meta(tmp_path, monkeypatch):
+ # No decode layer, so no masked meta: a masked meta with nothing beneath
+ # it is a meta that lies about its own graph.
+ monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ location = stores.make_store(tmp_path / "s.zarr")
+ result = variable(location).read_data(policy)
+ assert not isinstance(result._meta, np.ma.MaskedArray)
+
+
+def test_a_decoded_array_gets_a_masked_meta(tmp_path, monkeypatch):
+ monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ group = zarr.open_group(store=tmp_path / "m.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group, "v", (4,), "i2", 3, dimensions=("t",),
+ attributes={"_FillValue": -999},
+ )
+ array[:] = [-999, 1, 2, 3]
+
+ result = ZarrDataset(str(tmp_path / "m.zarr")).variables["v"].read_data(policy)
+ assert isinstance(result._meta, np.ma.MaskedArray)
+ realised = result.compute()
+ assert realised.mask.tolist() == [True, False, False, False]
+```
+
+- [ ] **Step 6: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py -v
+```
+
+Expected: FAIL,
+`TypeError: Can't instantiate abstract class ZarrDatasetVariable with abstract method read_data`
+— `read_data` is abstract on the interface, so until it is written nothing in
+the Zarr package instantiates at all. If Tasks 3–5 passed, the class already
+has a stub; delete it and watch this failure appear, so that the test is known
+to be able to fail.
+
+- [ ] **Step 7: Write `read_data`, `cache_key` and `zarr_format`**
+
+In `ZarrDatasetVariable`, add to the Zarr-only members:
+
+```python
+ @property
+ def zarr_format(self) -> int:
+ """2 or 3. The one Zarr fact that changes how a value is read."""
+ return self._array.metadata.zarr_format
+
+ @property
+ def cache_key(self) -> str:
+ """An identity for this array that goes stale when the array does.
+
+ `iris._lazy_data.CACHE` is process-wide, so an address alone is not
+ enough: a store rewritten in place keeps its address and changes its
+ contents. The metadata document covers shape, chunks, shards, dtype,
+ fill value, codecs, dimension names and attributes in one value, and
+ is treated as opaque - listing the parts Iris cares about would go
+ stale, silently, the first time zarr-python added a field that
+ mattered.
+ """
+ return json.dumps(
+ [self._location, self._array.path, self._array.metadata.to_dict()],
+ sort_keys=True,
+ )
+```
+
+and, among the interface members, after `__setitem__`:
+
+```python
+ def read_data(self, chunking_policy):
+ """Return this array's data, lazily where a lazy array is worth it.
+
+ Parameters
+ ----------
+ chunking_policy : callable
+ Called with no arguments, returning the ``(chunks, dims_fixed)``
+ pair to build a lazy array with. Called only once the result is
+ known to be lazy.
+
+ Returns
+ -------
+ numpy.ndarray, numpy.ma.MaskedArray or dask.array.Array
+ Real data for a small array, lazy data otherwise. Masked whenever
+ the variable declares anything to decode.
+
+ Notes
+ -----
+ There is no data proxy. ``zarr.Array`` is picklable, a chunk is an
+ independent object in a key-value store so concurrent reads need no
+ lock, and a store is a namespace rather than an open file handle. CF
+ decoding is a ``map_blocks`` layer over the raw array rather than
+ something hidden inside ``__getitem__``, so it is visible in the
+ graph and testable without a store.
+
+ """
+ # Deferred, and the module rather than its members: the CF package
+ # reaches this one through iris.fileformats.zarr.loader, so a
+ # module-level import would be a cycle; and the tests patch
+ # _LAZYVAR_MIN_BYTES on the module object.
+ from iris.fileformats.cf import loader
+
+ parameters = _decode.decode_parameters(
+ self._attributes,
+ dtype=self.dtype,
+ storage_fill_value=self.fill_value,
+ zarr_format=self.zarr_format,
+ variable_name=self.name,
+ )
+ decoding = _decode.decoding_required(parameters)
+
+ if self.size * self.dtype.itemsize < loader._LAZYVAR_MIN_BYTES:
+ # A lazy array would cost more memory AND more time to access.
+ block = np.asanyarray(self._array[...])
+ if decoding:
+ return _decode.mask_and_scale(block, **parameters)
+ return block
+
+ chunks, dims_fixed = chunking_policy()
+ raw_meta = np.empty((0,) * self.ndim, dtype=self.dtype)
+
+ if chunks == "auto":
+ # CHUNK_CONTROL.as_dask(): the user asked for Dask's own policy,
+ # so the store's layout is deliberately not consulted.
+ lazy = as_lazy_data(
+ self._array, chunks="auto", meta=raw_meta, cache_key=self.cache_key
+ )
+ else:
+ chunks = _optimum_chunksize(
+ chunks, shape=self.shape, dtype=self.dtype, dims_fixed=dims_fixed
+ )
+ chunks = _align_chunks(
+ chunks, self.chunking, self.shape, self.dtype, dims_fixed, self.name
+ )
+ # Fixed in every dimension, because the alignment above is the
+ # final word: as_lazy_data would otherwise re-run the
+ # optimisation and undo it.
+ lazy = as_lazy_data(
+ self._array,
+ chunks=chunks,
+ meta=raw_meta,
+ dims_fixed=(True,) * self.ndim,
+ cache_key=self.cache_key,
+ )
+
+ if not decoding:
+ return lazy
+
+ return da.map_blocks(
+ _decode.mask_and_scale,
+ lazy,
+ dtype=parameters["dtype"],
+ meta=np.ma.masked_array(np.empty((0,) * self.ndim, parameters["dtype"])),
+ **parameters,
+ )
+```
+
+Add to the module imports: `import json`, `import dask.array as da`,
+`from iris._lazy_data import _optimum_chunksize, as_lazy_data`, and
+`from . import _decode`.
+
+Note `chunks` arrives as a list from `_chunks_from_chunk_control`;
+`_optimum_chunksize` returns a tuple, and `_align_chunks` returns one, so what
+reaches `as_lazy_data` is hashable — which matters, because it goes into the
+cache key.
+
+- [ ] **Step 8: Run, then write the cache-key tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py -v
+```
+
+Expected: all pass. Then append the staleness and sensitivity cases from
+spec §6:
+
+```python
+def key_of(location, name="v"):
+ return ZarrDataset(location).variables[name].cache_key
+
+
+def test_the_key_is_stable_across_a_reopen(tmp_path):
+ group = zarr.open_group(store=tmp_path / "k.zarr", mode="w", zarr_format=3)
+ stores.make_array(group, "v", (4,), "i4", 3, dimensions=("t",))
+ location = str(tmp_path / "k.zarr")
+ assert key_of(location) == key_of(location)
+
+
+def test_the_key_changes_when_the_shape_does(tmp_path):
+ """A store rewritten in place keeps its address and changes its contents.
+
+ Keyed on the address alone, an array reopened after its shape changed
+ from (4,) to (6,) returns the cached four values, silently truncating -
+ and that is weaker than the netCDF key it replaces, since
+ repr(NetCDFDataProxy) already carries shape and dtype.
+ """
+ location = str(tmp_path / "k.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ stores.make_array(group, "v", (4,), "i4", 3, dimensions=("t",), chunks=(2,))
+ before = key_of(location)
+
+ group = zarr.open_group(store=location, mode="a", zarr_format=3)
+ group.create_array(
+ "v", shape=(6,), dtype="i4", chunks=(2,), dimension_names=("t",),
+ overwrite=True,
+ )
+ assert key_of(location) != before
+
+
+def test_the_key_changes_when_the_fill_value_does(tmp_path):
+ location = str(tmp_path / "k.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ group.create_array("v", shape=(4,), dtype="i4", fill_value=-1)
+ before = key_of(location)
+
+ group = zarr.open_group(store=location, mode="a", zarr_format=3)
+ group.create_array("v", shape=(4,), dtype="i4", fill_value=-2, overwrite=True)
+ assert key_of(location) != before
+
+
+@pytest.mark.parametrize(
+ "change",
+ [
+ {"shape": (8,)},
+ {"chunks": (4,)},
+ {"shards": (8,)},
+ {"dtype": "f8"},
+ {"fill_value": -7},
+ {"dimension_names": ("other",)},
+ ],
+)
+def test_the_key_is_sensitive_to_every_metadata_field(tmp_path, change):
+ """`Array.metadata.to_dict()` is zarr-python's rendering, not a guarantee.
+
+ Nothing in the Zarr specification defines what that method returns, so
+ these are what fail if a release stops carrying a field the key needs.
+ """
+ base = dict(
+ shape=(4,), dtype="i4", chunks=(2,), fill_value=-1,
+ dimension_names=("t",), shards=None,
+ )
+ location = str(tmp_path / "k.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ group.create_array("v", **base)
+ before = key_of(location)
+
+ group = zarr.open_group(store=location, mode="a", zarr_format=3)
+ group.create_array("v", **{**base, **change}, overwrite=True)
+ assert key_of(location) != before
+
+
+def test_the_key_is_sensitive_to_attributes(tmp_path):
+ location = str(tmp_path / "k.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ array = group.create_array("v", shape=(4,), dtype="i4")
+ before = key_of(location)
+ array.attrs["units"] = "K"
+ assert key_of(location) != before
+
+
+def test_two_cubes_sharing_an_array_share_one_dask_array(tmp_path, monkeypatch):
+ # What the cache is for: many cubes in one store sharing a coordinate.
+ monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ location = stores.make_store(tmp_path / "s.zarr")
+ first = variable(location).read_data(policy)
+ second = variable(location).read_data(policy)
+ assert first.name == second.name
+```
+
+The sensitivity test's `shards` case needs `chunks` to divide `shards`; keep
+`chunks=(2,)` and `shards=(8,)` together when writing it, and drop the case
+with a comment if zarr-python refuses the combination rather than loosening the
+assertion.
+
+- [ ] **Step 9: Write the sharded read-unit test**
+
+Append to `test_read_data.py`:
+
+```python
+class CountingStore(zarr.storage.WrapperStore):
+ """Wrap a store, adding every byte it returns to a caller-owned tally.
+
+ zarr-python copies a store when it opens it read-only, through
+ `with_read_only` and `_with_store`, so a tally kept on the instance is
+ never the one the live store writes to, and the test passes vacuously.
+ """
+
+ def __init__(self, store, tally):
+ super().__init__(store=store)
+ self.tally = tally
+
+ def _with_store(self, store):
+ return type(self)(store, self.tally)
+
+ async def get(self, key, prototype, byte_range=None):
+ buffer = await self._store.get(key, prototype, byte_range)
+ if buffer is not None:
+ self.tally.append(len(buffer))
+ return buffer
+
+ async def get_partial_values(self, prototype, key_ranges):
+ buffers = await self._store.get_partial_values(prototype, key_ranges)
+ self.tally.extend(len(b) for b in buffers if b is not None)
+ return buffers
+
+
+def bytes_to_read_first_chunk(location, dask_chunks):
+ """Return the bytes fetched reading the first eight values of "v"."""
+ tally = []
+ store = CountingStore(zarr.storage.LocalStore(location), tally)
+ array = zarr.open_group(store=store, mode="r")["v"]
+ tally.clear()
+ lazy = da.from_array(array, chunks=dask_chunks, meta=np.empty((0,), "i4"))
+ lazy = da.map_blocks(
+ lambda block: np.ma.masked_array(block), lazy, dtype="i4"
+ )
+ lazy[:8].compute()
+ return sum(tally)
+
+
+def test_a_sharded_array_is_read_by_the_inner_chunk(tmp_path):
+ """Shard-sized Dask blocks fetch the whole shard, because of the layer.
+
+ Bare Dask slicing can push a slice down into the array's own __getitem__,
+ but a map_blocks layer in between cannot, so the block is materialised in
+ full. The absolute byte counts are zarr-version-dependent; the comparison
+ is not.
+ """
+ location = str(tmp_path / "sh.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ array = group.create_array(
+ "v", shape=(1024,), dtype="i4", chunks=(8,), shards=(1024,)
+ )
+ array[:] = np.arange(1024, dtype="i4")
+
+ inner = bytes_to_read_first_chunk(location, 8)
+ whole_shard = bytes_to_read_first_chunk(location, 1024)
+ assert inner < whole_shard
+
+ # And the variable chooses the inner chunk for exactly this reason.
+ assert ZarrDataset(location).variables["v"].chunking == (8,)
+```
+
+- [ ] **Step 10: Run the module**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/ -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 11: Write the test for Review Focus 4 — the unpacked dtype**
+
+Append to `test_read_data.py`:
+
+```python
+def test_the_declared_dtype_matches_what_the_decode_layer_produces(tmp_path, monkeypatch):
+ """A packed variable's lazy dtype must equal its realised dtype.
+
+ On the netCDF path netCDF4 does the arithmetic and reports the dtype. On
+ the Zarr path Iris does both, in two separately written statements, and
+ Dask does not check them against each other: a mismatch builds a graph
+ and surfaces later as a wrong dtype on the cube.
+ """
+ monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "b02",
+ (4,),
+ "u2",
+ 3,
+ dimensions=("p",),
+ attributes={"scale_factor": 0.0001, "add_offset": -0.1},
+ )
+ array[:] = [1000, 2000, 3000, 4000]
+
+ lazy = ZarrDataset(str(tmp_path / "p.zarr")).variables["b02"].read_data(policy)
+ realised = lazy.compute()
+
+ assert lazy.dtype == realised.dtype
+ assert lazy.dtype == np.dtype("f8")
+ np.testing.assert_allclose(realised, [0.0, 0.1, 0.2, 0.3])
+
+
+def test_the_declared_dtype_matches_the_cf_loader_calculation(tmp_path):
+ """And it is the same answer `_get_actual_dtype` gives.
+
+ The cube's data variable and its coordinates must agree about what a
+ packed value becomes, and they reach that answer through different code.
+ """
+ group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3)
+ stores.make_array(
+ group, "v", (4,), "u2", 3,
+ attributes={"scale_factor": np.float32(0.01)},
+ )
+ variable = ZarrDataset(str(tmp_path / "p.zarr")).variables["v"]
+ from iris.fileformats.zarr import _decode
+
+ parameters = _decode.decode_parameters(
+ variable.attributes,
+ dtype=variable.dtype,
+ storage_fill_value=variable.fill_value,
+ zarr_format=variable.zarr_format,
+ variable_name=variable.name,
+ )
+ assert parameters["dtype"] == cf_loader._get_actual_dtype(variable)
+```
+
+- [ ] **Step 12: Run everything built so far, and commit**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/ lib/iris/tests/unit/fileformats/cf/ lib/iris/tests/unit/lazy_data/ -q
+```
+
+Expected: all pass. Then run §5.0 and:
+
+```bash
+git add lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr
+git commit -m "Read Zarr data lazily, aligned to the store's own chunks
+
+No proxy: a zarr.Array is picklable and a store is a namespace, so the
+array goes to as_lazy_data directly and CF decoding is a visible map_blocks
+layer. Chunking is rounded up to whole multiples of the store's own, never
+below, because reading any part of a Zarr chunk fetches the whole object.
+The Dask cache key carries the array's metadata document rather than its
+address, which alone goes stale when a store is rewritten in place."
+```
+
+---
+### Task 8: `spans` matches on `(name, size)` pairs
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/_variables.py`
+- Create: `lib/iris/tests/unit/fileformats/cf/test_spans.py`
+
+**Interfaces:**
+- Consumes: `ZarrDataset` (Task 4), `NetCDFDataset`, `stores.make_array`.
+- Produces: `CFVariable._dimension_identities()` and the module-private
+ `_AnonymousDimension`; `CFVariable.spans` and the three overrides in
+ `CFBoundaryVariable`, `CFClimatologyVariable` and `CFLabelVariable` matching
+ on pairs.
+
+This is the one change in this pull request that can alter what loads from an
+existing netCDF file, so it is on its own and its tests run on both backends.
+Spec §4.4:
+
+> **The identity that links two variables is the pair `(name, size)`, and an
+> anonymous dimension has no identity.**
+>
+> - Two variables share a dimension when they name it identically *and* it is
+> the same length in both. A name reused at two lengths links nothing.
+> - An anonymous dimension never matches anything — not a named dimension, and
+> not another anonymous dimension, including within the same variable.
+> - A variable may name a dimension the dataset does not list, and may name
+> some of its dimensions and not others. Neither is an error.
+
+For netCDF this changes nothing by construction: dimension names are unique
+within a group, so `(name, size)` and `name` are the same key, and no netCDF
+variable has an anonymous dimension.
+
+- [ ] **Step 1: Write the failing shared suite**
+
+Create `lib/iris/tests/unit/fileformats/cf/test_spans.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""`CFVariable.spans`, asked of both storage backends.
+
+Dimension matching is a pair, `(name, size)`, not a bare name - which is a
+widening to admit what Zarr allows, not a change to what netCDF does. These
+tests run against both so that the second half of that sentence is checked
+rather than asserted.
+"""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf import (
+ CFBoundaryVariable,
+ CFDataVariable,
+ CFLabelVariable,
+)
+
+zarr = pytest.importorskip("zarr")
+
+from iris.fileformats.netcdf._dataset import NetCDFDataset # noqa: E402
+from iris.fileformats.zarr._dataset import ZarrDataset # noqa: E402
+
+
+def build_netcdf(path, described):
+ """Write `described` as a netCDF file and return its open dataset."""
+ dataset = NetCDFDataset(path, mode="w")
+ sizes = {}
+ for dimensions, shape in described.values():
+ for name, size in zip(dimensions, shape):
+ sizes.setdefault(name, size)
+ for name, size in sizes.items():
+ dataset.create_dimension(name, size)
+ for name, (dimensions, shape) in described.items():
+ dataset.create_variable(name, np.dtype("f4"), dimensions)
+ return dataset
+
+
+def build_zarr(path, described):
+ """Write `described` as a Zarr store and return its open dataset."""
+ group = zarr.open_group(store=path, mode="w", zarr_format=3)
+ for name, (dimensions, shape) in described.items():
+ group.create_array(
+ name, shape=shape, dtype="f4", dimension_names=dimensions
+ )
+ return ZarrDataset(str(path))
+
+
+@pytest.fixture(params=[build_netcdf, build_zarr], ids=["netcdf", "zarr"])
+def build(request, tmp_path):
+ """Return a callable giving `{name: CFDataVariable}` for a description."""
+ suffix = ".nc" if request.param is build_netcdf else ".zarr"
+
+ def make(described):
+ dataset = request.param(tmp_path / f"spans{suffix}", described)
+ return {
+ name: CFDataVariable(name, dataset.variables[name])
+ for name in described
+ }
+
+ return make
+
+
+def test_identical_dimensions_span(build):
+ variables = build({"data": (("y", "x"), (3, 2)), "aux": (("y", "x"), (3, 2))})
+ assert variables["aux"].spans(variables["data"]) is True
+
+
+def test_a_subset_spans(build):
+ variables = build({"data": (("y", "x"), (3, 2)), "aux": (("y",), (3,))})
+ assert variables["aux"].spans(variables["data"]) is True
+
+
+def test_a_different_name_does_not_span(build):
+ variables = build({"data": (("y", "x"), (3, 2)), "aux": (("t",), (3,))})
+ assert variables["aux"].spans(variables["data"]) is False
+
+
+def test_a_scalar_always_spans(build):
+ variables = build({"data": (("y", "x"), (3, 2)), "grid": ((), ())})
+ assert variables["grid"].spans(variables["data"]) is True
+
+
+def test_a_bounds_variable_ignores_its_extent_dimension(build):
+ variables = build(
+ {"data": (("y",), (3,)), "bounds": (("y", "bnds"), (3, 2))}
+ )
+ bounds = CFBoundaryVariable("bounds", variables["bounds"].cf_data)
+ assert bounds.spans(variables["data"]) is True
+
+
+def test_a_label_variable_ignores_its_string_length_dimension(build):
+ variables = build(
+ {"data": (("y",), (3,)), "label": (("y", "strlen"), (3, 8))}
+ )
+ label = CFLabelVariable("label", variables["label"].cf_data)
+ assert label.spans(variables["data"]) is True
+```
+
+- [ ] **Step 2: Run it against today's code**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/test_spans.py -v
+```
+
+Expected: all pass on netCDF, all pass on Zarr. These are the cases that must
+be **unchanged** by this task — run them before the change so that the
+"netCDF behaviour is unchanged" claim has a before as well as an after.
+
+- [ ] **Step 3: Write the failing tests for the new rules**
+
+Append to `test_spans.py`:
+
+```python
+def test_a_name_at_two_sizes_does_not_link(tmp_path):
+ """Zarr-only: a name guarantees nothing across arrays.
+
+ The version 3 core specification "does not place any restrictions on the
+ use of the same dimension name across multiple arrays within the same Zarr
+ hierarchy", so two arrays may mean different things by one name. Matching
+ on the name alone would produce a cube whose coordinate is the wrong
+ length. netCDF cannot express this: a dimension name has one size per
+ group.
+ """
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array("data", shape=(3,), dtype="f4", dimension_names=("t",))
+ group.create_array("aux", shape=(5,), dtype="f4", dimension_names=("t",))
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ data = CFDataVariable("data", dataset.variables["data"])
+ aux = CFDataVariable("aux", dataset.variables["aux"])
+ assert aux.spans(data) is False
+
+
+def test_an_anonymous_dimension_matches_nothing(tmp_path):
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array("data", shape=(3,), dtype="f4", dimension_names=("t",))
+ group.create_array("aux", shape=(3,), dtype="f4")
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ data = CFDataVariable("data", dataset.variables["data"])
+ aux = CFDataVariable("aux", dataset.variables["aux"])
+ assert aux.spans(data) is False
+
+
+def test_two_anonymous_dimensions_of_one_size_do_not_match(tmp_path):
+ """Not even each other, and not within one variable.
+
+ There is no information to match on, so claiming a match would be
+ invention.
+ """
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array("data", shape=(3, 3), dtype="f4")
+ group.create_array("aux", shape=(3,), dtype="f4")
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ data = CFDataVariable("data", dataset.variables["data"])
+ aux = CFDataVariable("aux", dataset.variables["aux"])
+ assert aux.spans(data) is False
+
+
+def test_a_partly_named_variable_matches_on_the_named_part(tmp_path):
+ # "A variable may name some of its dimensions and not others." The named
+ # ones still have to be present in the target, and the anonymous one
+ # still matches nothing - so this does not span.
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array(
+ "data", shape=(3, 2), dtype="f4", dimension_names=("y", "x")
+ )
+ group.create_array(
+ "aux", shape=(3, 2), dtype="f4", dimension_names=("y", None)
+ )
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ data = CFDataVariable("data", dataset.variables["data"])
+ aux = CFDataVariable("aux", dataset.variables["aux"])
+ assert aux.spans(data) is False
+```
+
+- [ ] **Step 4: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/test_spans.py -v -k "two_sizes or anonymous or partly"
+```
+
+Expected: four FAILs — today's `set(self.dimensions).issubset(...)` matches a
+bare name, so a name at two sizes spans, and `None` matches `None`.
+
+- [ ] **Step 5: Add the identity helper**
+
+In `lib/iris/fileformats/cf/_variables.py`, after `_NCZARR_SCALAR_DIMENSION`:
+
+```python
+class _AnonymousDimension:
+ """One dimension the store gives no name.
+
+ A distinct instance per dimension, relying on object identity for
+ equality, so that an anonymous dimension matches nothing: not a named
+ dimension, not another anonymous one, and not itself as seen from a
+ second variable. There is nothing to match on, so claiming a match would
+ be invention.
+
+ Zarr's version 3 core specification makes ``dimension_names`` optional and
+ allows a null entry, so a conforming store can present this and it must
+ not be an error. No netCDF variable ever has one.
+ """
+
+ def __repr__(self) -> str:
+ return "<anonymous dimension>"
+```
+
+and on `CFVariable`, beside `spans`:
+
+```python
+ def _dimension_identities(self) -> tuple:
+ """Return one hashable identity per dimension this variable spans.
+
+ A named dimension's identity is the pair ``(name, size)``: two
+ variables share a dimension when they name it identically *and* it is
+ the same length in both. A name reused at two lengths links nothing,
+ which is the only reading that cannot produce a wrong cube.
+
+ An unnamed dimension gets a fresh :class:`_AnonymousDimension`, which
+ equals nothing but itself.
+
+ Relies on the :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+ invariant that ``dimensions`` and ``shape`` have the same length.
+ """
+ return tuple(
+ (name, size) if name is not None else _AnonymousDimension()
+ for name, size in zip(self.dimensions, self.shape)
+ )
+```
+
+- [ ] **Step 6: Rewrite the four `spans` methods**
+
+`CFVariable.spans` (keep the docstring, replace the comparison):
+
+```python
+ # Scalar variables always span the target variable.
+ if self._is_scalar():
+ return True
+
+ source = self._dimension_identities()
+ target = cf_variable._dimension_identities()
+ return set(source).issubset(target)
+```
+
+`CFBoundaryVariable.spans`, `CFClimatologyVariable.spans` and
+`CFLabelVariable.spans` keep their own comments about which extent dimension
+they ignore, and become:
+
+```python
+ # Scalar variables always span the target variable.
+ result = True
+ if not self._is_scalar():
+ source = self._dimension_identities()
+ target = cf_variable._dimension_identities()
+ # Ignore the bounds extent dimension.
+ result = set(source[:-1]).issubset(target) or set(source[1:]).issubset(
+ target
+ )
+ return result
+```
+
+with "bounds" replaced by "climatology" and "label string length" in the other
+two, exactly as they read today.
+
+- [ ] **Step 7: Run the whole suite**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/test_spans.py -v
+pytest -n auto lib/iris/tests/unit/fileformats/ lib/iris/tests/integration/netcdf/ -q
+```
+
+Expected: the new tests pass, and **nothing else changes**. This is the claim
+the task exists to support: a netCDF file cannot tell the difference.
+
+- [ ] **Step 8: Pin the two claims that keep this change contained (F16)**
+
+Append to `test_spans.py`:
+
+```python
+def test_a_non_spanning_variable_is_dropped_and_promoted(tmp_path):
+ """An anonymous-dimension variable becomes a cube of its own.
+
+ This is what keeps `None` out of `iris.fileformats._nc_load_rules.helpers`,
+ which matches dimensions by name and would read `None in (None, "x")` as a
+ match. `CFReader._build_cf_groups` drops any variable whose `spans` is
+ False from the referrer's group and promotes it.
+ """
+ from iris.fileformats.cf import CFReader
+
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ data = group.create_array(
+ "data", shape=(3,), dtype="f4", dimension_names=("t",)
+ )
+ data.attrs["coordinates"] = "aux"
+ group.create_array("aux", shape=(3,), dtype="f4")
+
+ with CFReader(ZarrDataset(str(tmp_path / "s.zarr"))) as reader:
+ assert "aux" not in reader.cf_group["data"].cf_group
+ assert "aux" in reader.cf_group.promoted
+
+
+def test_an_anonymous_dimension_is_not_a_coordinate_variable(tmp_path):
+ """A one-dimensional array is a coordinate variable by name coincidence.
+
+ `CFCoordinateVariable.identify` requires the variable's own name to be
+ among its dimension names, which no anonymous dimension satisfies - so
+ the second route into the name-matching helpers is closed too.
+ """
+ from iris.fileformats.cf import CFCoordinateVariable
+
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array("t", shape=(3,), dtype="f4")
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ assert CFCoordinateVariable.identify(dataset.variables) == {}
+```
+
+These two depend on Task 9, which teaches `CFReader` to take an open
+`CFDataset`. Write them here and run them at the end of Task 9.
+
+- [ ] **Step 9: Commit**
+
+Run §5.0, then:
+
+```bash
+git add lib/iris/fileformats/cf/_variables.py lib/iris/tests/unit/fileformats/cf/test_spans.py
+git commit -m "Match CF dimensions on (name, size), not on the name alone
+
+Zarr places no restriction on reusing a dimension name across arrays, so a
+name alone can link two variables that mean different things by it. An
+unnamed dimension is anonymous and matches nothing, including another
+anonymous dimension. netCDF names are unique within a group, so this is a
+widening that no netCDF file can observe."
+```
+
+---
+
+### Task 9: `CFReader` accepts an already-open `CFDataset`
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/_reader.py`
+- Modify: `lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py`
+
+**Interfaces:**
+- Consumes: `CFDataset` (the ABC), `ZarrDataset` (Task 4).
+- Produces: `CFReader(file_source)` where `file_source` may be a `CFDataset`,
+ which is how Task 10's loader hands a Zarr store to the CF layer.
+
+`CFReader.__init__` has two branches today: a `str` opens a `NetCDFDataset`,
+and anything else is passed to `NetCDFDataset.from_existing`. A `ZarrDataset`
+falls into the second and is wrapped as a netCDF dataset (F12). The new branch
+goes first and borrows rather than owns, because whoever opened the dataset is
+responsible for closing it.
+
+- [ ] **Step 1: Write the failing test**
+
+Append to `lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py`:
+
+```python
+def test_an_already_open_cf_dataset_is_used_as_is(tmp_path):
+ """A loader that has already opened a store hands it straight over.
+
+ The alternative - a path and a format keyword - would put the choice of
+ backend inside CFReader, which is the branch the dataset interface exists
+ to remove.
+ """
+ from iris.fileformats.netcdf._dataset import NetCDFDataset
+
+ path = str(tmp_path / "source.nc")
+ with NetCDFDataset(path, mode="w") as writer:
+ writer.create_dimension("x", 2)
+ writer.create_variable("v", np.dtype("f4"), ("x",))
+
+ dataset = NetCDFDataset(path, mode="r")
+ with CFReader(dataset) as reader:
+ assert reader.filename == dataset.location
+ assert "v" in reader.cf_group
+ # Borrowed, not owned: leaving the context must not close it.
+ assert dataset.closed is False
+ dataset.close()
+```
+
+and, in the Zarr unit tests, a matching one —
+`lib/iris/tests/unit/fileformats/zarr/test_CFReader.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The CF layer reads a Zarr store through the same reader netCDF uses."""
+
+from iris.fileformats.cf import CFReader
+from iris.fileformats.zarr._dataset import ZarrDataset
+
+from . import stores
+
+
+def test_the_cf_reader_classifies_a_zarr_store(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ with CFReader(ZarrDataset(location)) as reader:
+ assert "air_temperature" in reader.cf_group.data_variables
+ assert "grid" in reader.cf_group.grid_mappings
+ assert reader.cf_group.global_attributes["title"] == "contract"
+ assert reader.filename == location
+```
+
+- [ ] **Step 2: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/test_CFReader.py -v
+```
+
+Expected: FAIL. `NetCDFDataset.from_existing` is handed a `ZarrDataset`, which
+has no `THREAD_SAFE_FLAG`, so `EncodedDataset.from_existing` raises on it.
+
+- [ ] **Step 3: Add the branch**
+
+In `lib/iris/fileformats/cf/_reader.py`, add `from .dataset import CFDataset`
+to the imports, and make `__init__`'s dispatch read:
+
+```python
+ # Ensure safe operation for destructor, should init fail.
+ self._own_file = False
+ if isinstance(file_source, CFDataset):
+ # Already open, in whatever format: use it, don't own it. The
+ # caller that opened it is responsible for closing it.
+ self._dataset = file_source
+ self._filename = file_source.location
+ elif isinstance(file_source, str):
+ # Create from filepath : open it + own it (=close when we die).
+ if not urlparse(file_source).scheme:
+ self._filename = Path(file_source).expanduser()
+ else:
+ self._filename = file_source
+
+ self._dataset = NetCDFDataset(
+ self._filename, mode="r", warn_legacy_format=warn
+ )
+ self._own_file = True
+ else:
+ # We have been passed an open dataset.
+ # We use it but don't own it (don't close it).
+ self._dataset = NetCDFDataset.from_existing(
+ file_source, warn_legacy_format=warn
+ )
+ self._filename = self._dataset.location
+```
+
+- [ ] **Step 4: Update the module docstring**
+
+The paragraph beginning "This is the one module in :mod:`iris.fileformats.cf`
+still coupled to :mod:`iris.fileformats.netcdf`" is now only half true, and a
+stale docstring is worse than none. Replace it with:
+
+```python
+``CFReader`` takes one of three things. A path opens a netCDF file, because
+netCDF is the default a bare filename means. An open netCDF4 dataset, or an
+emulator of one such as ncdata passes, is wrapped. And a
+:class:`~iris.fileformats.cf.dataset.CFDataset` of any format is used as it
+stands, which is how a Zarr store reaches this machinery: its loader opens
+the store and hands it over, so nothing here has to know that Zarr exists.
+Only the first case mentions netCDF, and only because something has to be
+the default.
+```
+
+- [ ] **Step 5: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/ lib/iris/tests/unit/fileformats/zarr/ -v
+```
+
+Expected: all pass, including Task 8's two deferred tests and Task 5's
+no-type-test check.
+
+- [ ] **Step 6: Commit**
+
+Run §5.0, then:
+
+```bash
+git add lib/iris/fileformats/cf/_reader.py lib/iris/tests/unit/fileformats
+git commit -m "Let CFReader read any already-open CF dataset
+
+A loader that has opened a store hands it straight over, so the choice of
+backend stays with the loader rather than becoming a branch inside the
+reader. A borrowed dataset is not closed by the reader that borrowed it."
+```
+
+---
+
+### Task 10: `zarr/loader.py` — `load_cubes`
+
+**Files:**
+- Create: `lib/iris/fileformats/zarr/loader.py`
+- Modify: `lib/iris/fileformats/zarr/__init__.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/loader/__init__.py`
+- Create: `lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py`
+
+**Interfaces:**
+- Consumes: `ZarrDataset` (Task 4); `CFReader` taking one (Task 9);
+ `iris.fileformats.cf.loader._actions_engine`, `_load_cube`,
+ `_load_aux_factory`, `_translate_constraints_to_var_callback` (**PR 3**);
+ `iris.io.run_callback`.
+- Produces: `iris.fileformats.zarr.load_cubes(file_sources, callback=None,
+ constraints=None, group=None)`, re-exported from the package.
+
+The shape is `netcdf/loader.py::load_cubes` with the mesh handling removed:
+UGRID from Zarr is out of scope here (§1), and `_meshes_from_cf` lives in
+`netcdf/ugrid_load.py`, which this package must not import.
+
+Spec §4.6's rule is the one piece that is not a copy: "a store whose root
+group holds no data variables raises a message that names the groups that do".
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `lib/iris/tests/unit/fileformats/zarr/loader/__init__.py` with only the
+copyright header, then `.../loader/test_load_cubes.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :func:`iris.fileformats.zarr.load_cubes`."""
+
+import numpy as np
+import pytest
+import zarr
+
+from iris.cube import Cube
+from iris.exceptions import TranslationError
+from iris.fileformats.zarr import load_cubes
+
+from .. import stores
+
+
+def test_one_cube_per_data_variable(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ cubes = list(load_cubes(location))
+ assert len(cubes) == 1
+ cube = cubes[0]
+ assert isinstance(cube, Cube)
+ assert cube.var_name == "air_temperature"
+ assert cube.units == "K"
+ np.testing.assert_array_equal(cube.data, stores.DATA)
+
+
+def test_a_version_2_store_loads_the_same_cube(tmp_path):
+ v3 = list(load_cubes(stores.make_store(tmp_path / "v3.zarr")))[0]
+ v2 = list(load_cubes(stores.make_store(tmp_path / "v2.zarr", zarr_format=2)))[0]
+ assert v2 == v3
+
+
+def test_a_callback_can_reject_a_cube(tmp_path):
+ from iris.exceptions import IgnoreCubeException
+
+ def reject(cube, field, filename):
+ raise IgnoreCubeException
+
+ location = stores.make_store(tmp_path / "s.zarr")
+ assert list(load_cubes(location, callback=reject)) == []
+
+
+def test_the_group_keyword_selects_a_group(tmp_path):
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ sub = root.create_group("measurements")
+ band = stores.make_array(
+ sub, "b02", (3,), "u2", 3, dimensions=("p",), attributes={"units": "1"}
+ )
+ band[:] = [1, 2, 3]
+
+ cubes = list(load_cubes(str(tmp_path / "n.zarr"), group="measurements"))
+ assert [cube.var_name for cube in cubes] == ["b02"]
+
+
+def test_several_stores_in_one_call(tmp_path):
+ first = stores.make_store(tmp_path / "a.zarr")
+ second = stores.make_store(tmp_path / "b.zarr")
+ assert len(list(load_cubes([first, second]))) == 2
+```
+
+- [ ] **Step 2: Write the failing tests for Review Focus 2 — an empty root**
+
+Append:
+
+```python
+def test_a_group_with_no_arrays_names_the_groups_that_have_them(tmp_path):
+ """A real store in this shape exists, and silence is the worst answer.
+
+ The ESA EOPF Sentinel-2 products hold no arrays in the root group, only
+ subgroups - `conditions/geometry`, `measurements/reflectance/r10m` and
+ others. Iris loads one group per call, so loading the default root yields
+ nothing; returning an empty CubeList from a store that plainly holds data
+ leaves the user with no way to find out why.
+ """
+ root = zarr.open_group(store=tmp_path / "e.zarr", mode="w", zarr_format=3)
+ root.create_group("conditions")
+ measurements = root.create_group("measurements")
+ stores.make_array(measurements, "b02", (2,), "u2", 3, dimensions=("p",))
+
+ with pytest.raises(TranslationError, match="measurements"):
+ list(load_cubes(str(tmp_path / "e.zarr")))
+
+
+def test_the_message_says_so_when_there_are_no_groups_either(tmp_path):
+ zarr.open_group(store=tmp_path / "empty.zarr", mode="w", zarr_format=3)
+ with pytest.raises(TranslationError, match="no data variables"):
+ list(load_cubes(str(tmp_path / "empty.zarr")))
+
+
+def test_nested_groups_are_not_walked(tmp_path):
+ """Iris loads one group per call, like xarray.
+
+ A Zarr store can legitimately hold unrelated datasets, and merging them
+ into one CubeList would be wrong.
+ """
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ top = stores.make_array(
+ root, "top", (2,), "f8", 3, dimensions=("p",), attributes={"units": "1"}
+ )
+ top[:] = [1.0, 2.0]
+ inner = stores.make_array(
+ root.create_group("sub"), "inner", (2,), "f8", 3, dimensions=("p",)
+ )
+ inner[:] = [3.0, 4.0]
+
+ cubes = list(load_cubes(str(tmp_path / "n.zarr")))
+ assert [cube.var_name for cube in cubes] == ["top"]
+```
+
+- [ ] **Step 3: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/loader/ -v
+```
+
+Expected: `ImportError: cannot import name 'load_cubes'`.
+
+- [ ] **Step 4: Write the loader**
+
+Create `lib/iris/fileformats/zarr/loader.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Load Iris cubes from Zarr stores.
+
+.. z_reference:: iris.fileformats.zarr.loader
+ :tags: topic_load_save
+
+ API reference
+
+A thin loop: open the store as a
+:class:`~iris.fileformats.zarr._dataset.ZarrDataset`, hand it to
+:class:`~iris.fileformats.cf.CFReader`, and build one cube per CF data
+variable with :mod:`iris.fileformats.cf.loader` - the same machinery the
+netCDF loader runs. Everything Zarr-specific happens below this module, in
+:mod:`iris.fileformats.zarr._dataset`.
+
+Iris loads **one** group per call. A Zarr store can legitimately hold
+unrelated datasets, and merging them into one ``CubeList`` would be wrong, so
+nested groups are not walked and ``group=`` selects which one to read.
+
+UGRID meshes are not built here. :func:`iris.mesh.load_meshes` reaches a Zarr
+store through the same format dispatch, but the mesh machinery lives in
+:mod:`iris.fileformats.netcdf.ugrid_load` and this package does not import it.
+
+See sections 4.6 and 4.7 of
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+from collections.abc import Iterable
+import warnings
+
+from iris.exceptions import TranslationError
+from iris.fileformats.cf import CFReader
+from iris.fileformats.cf.loader import (
+ _actions_engine,
+ _load_aux_factory,
+ _load_cube,
+ _translate_constraints_to_var_callback,
+)
+import iris.warnings
+
+from ._dataset import ZarrDataset
+
+
+def _check_has_data_variables(reader, dataset) -> None:
+ """Raise if this group holds nothing to make a cube from.
+
+ Returning an empty ``CubeList`` from a store that plainly holds data
+ elsewhere leaves the user nothing to act on, and a store whose root group
+ holds only subgroups is a real shape - the ESA EOPF Sentinel products are
+ built that way.
+ """
+ if reader.cf_group.data_variables or reader.cf_group.promoted:
+ return
+
+ subgroups = dataset.subgroup_names
+ if subgroups:
+ message = (
+ f"No data variables in Zarr group {dataset.location!r}. Load one "
+ "of its groups instead, with the 'group' keyword: "
+ f"{', '.join(sorted(subgroups))}"
+ )
+ else:
+ message = f"No data variables in Zarr store {dataset.location!r}."
+ raise TranslationError(message)
+
+
+def load_cubes(file_sources, callback=None, constraints=None, group=None):
+ """Load cubes from one or more Zarr stores.
+
+ Parameters
+ ----------
+ file_sources : str or iterable of str
+ The stores to load. A local directory path, or a URL that
+ zarr-python can open - ``s3://``, ``gs://``, ``az://`` and ``abfs://``
+ need the matching store library installed, which Iris does not
+ depend on.
+ callback : callable, optional
+ Passed to :func:`iris.io.run_callback`.
+ constraints : optional
+ Used to skip data variables that cannot match, before they are built.
+ group : str, optional
+ A ``/``-separated path to a group within each store. The root group by
+ default.
+
+ Yields
+ ------
+ iris.cube.Cube
+
+ Raises
+ ------
+ iris.exceptions.TranslationError
+ If the selected group holds no data variables. The message names the
+ store's other groups, when it has any.
+
+ Notes
+ -----
+ This function maintains laziness; it does not realise data.
+
+ """
+ from iris.io import run_callback
+
+ var_callback = _translate_constraints_to_var_callback(constraints)
+ engine = _actions_engine()
+
+ if isinstance(file_sources, str) or not isinstance(file_sources, Iterable):
+ file_sources = [file_sources]
+
+ for file_source in file_sources:
+ with ZarrDataset(file_source, group=group) as dataset:
+ with CFReader(dataset) as cf:
+ _check_has_data_variables(cf, dataset)
+
+ data_variables = list(cf.cf_group.data_variables.values()) + list(
+ cf.cf_group.promoted.values()
+ )
+ for cf_var in data_variables:
+ if var_callback and not var_callback(cf_var):
+ continue
+
+ cube = _load_cube(engine, cf, cf_var, cf.filename)
+
+ try:
+ _load_aux_factory(engine, cube)
+ except ValueError as exception:
+ warnings.warn(
+ str(exception),
+ category=iris.warnings.IrisLoadWarning,
+ )
+
+ cube = run_callback(callback, cube, cf_var, file_source)
+ if cube is None:
+ # The callback rejected it.
+ continue
+
+ yield cube
+```
+
+- [ ] **Step 5: Re-export it**
+
+In `lib/iris/fileformats/zarr/__init__.py`, after the docstring:
+
+```python
+from .loader import load_cubes
+
+__all__ = ["load_cubes"]
+```
+
+This is the line Task 1 deferred. Re-run Task 1's import-guard tests
+immediately: `iris.fileformats.zarr` now imports `zarr` through `loader`, so
+`test_importing_the_zarr_package_does_not_import_zarr` **must now fail**, and
+that is correct — the test was about the package before it had a loader.
+Replace it with the one that still matters:
+
+```python
+def test_importing_the_zarr_package_imports_zarr():
+ # Once the package exports a loader it needs the dependency. What must
+ # stay true is that nothing in `iris` or `iris.fileformats` imports this
+ # package to find out that it exists.
+ assert _imports_zarr("import iris.fileformats.zarr")
+```
+
+- [ ] **Step 6: Run the loader tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/ -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 7: Check what a cube from a Zarr store actually carries**
+
+Append to `test_load_cubes.py`:
+
+```python
+def test_the_cube_is_lazy(tmp_path, monkeypatch):
+ from iris.fileformats.cf import loader as cf_loader
+
+ monkeypatch.setattr(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ cube = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))[0]
+ assert cube.has_lazy_data()
+
+
+def test_the_grid_mapping_variable_does_not_become_a_cube(tmp_path):
+ cubes = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))
+ assert "grid" not in [cube.var_name for cube in cubes]
+
+
+def test_cf_reserved_attributes_do_not_reach_the_cube(tmp_path):
+ """`units` becomes a property, so it must not also be an attribute.
+
+ Which attributes survive is decided by which ones the loading rules did
+ not read, so an attribute Iris consumes but fails to record as read
+ arrives on the cube as well.
+ """
+ cube = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))[0]
+ assert "units" not in cube.attributes
+ assert "long_name" not in cube.attributes
+ assert cube.attributes["Conventions"] == "CF-1.7"
+
+
+def test_a_structured_attribute_reaches_the_cube_unchanged(tmp_path):
+ """CF cannot express a nested attribute; Zarr allows one. Carry it.
+
+ Two independent producers write them - NOAA GFS gives `valid_time` a
+ `statistics_approximate` object, and the ESA EOPF product root carries
+ `stac_discovery`. Interpreting them needs the attribute-model discussion
+ in SciTools/iris#7288; dropping them loses the publisher's metadata.
+ """
+ group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ (2,),
+ "f8",
+ 3,
+ dimensions=("p",),
+ attributes={"stac_discovery": {"id": "abc"}},
+ )
+ array[:] = [1.0, 2.0]
+
+ cube = list(load_cubes(str(tmp_path / "n.zarr")))[0]
+ assert cube.attributes["stac_discovery"] == {"id": "abc"}
+
+
+def test_a_structured_attribute_cannot_then_be_saved_to_netcdf(tmp_path):
+ """Known limitation, characterised so that it is not a surprise.
+
+ Zarr allows any JSON literal as an attribute value and netCDF does not,
+ so a store carrying one produces a cube Iris cannot save as netCDF.
+ Checking attribute content before saving is SciTools/iris#7329, and
+ fixing it means changing the netCDF saver. Strip the attribute, or save
+ to Zarr once that is supported.
+ """
+ import iris
+
+ group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ (2,),
+ "f8",
+ 3,
+ dimensions=("p",),
+ attributes={"stac_discovery": {"id": "abc"}},
+ )
+ array[:] = [1.0, 2.0]
+ cube = list(load_cubes(str(tmp_path / "n.zarr")))[0]
+
+ with pytest.raises(Exception):
+ iris.save(cube, str(tmp_path / "out.nc"))
+
+ # And the documented workaround works.
+ del cube.attributes["stac_discovery"]
+ iris.save(cube, str(tmp_path / "out.nc"))
+
+
+def test_version_3_zeros_are_not_masked(tmp_path):
+ """An int16 [0, 1, 2] with no CF attributes loads with nothing masked.
+
+ zarr-python defaults `fill_value` to zero, so reading the storage field
+ as a sentinel masks valid data in the ordinary case.
+ """
+ group = zarr.open_group(store=tmp_path / "z3.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group, "v", (3,), "i2", 3, dimensions=("p",), attributes={"units": "1"}
+ )
+ array[:] = [0, 1, 2]
+
+ cube = list(load_cubes(str(tmp_path / "z3.zarr")))[0]
+ assert not np.ma.is_masked(cube.data)
+ np.testing.assert_array_equal(cube.data, [0, 1, 2])
+
+
+def test_version_2_zeros_are_masked(tmp_path):
+ """The same array as version 2 does use `fill_value` as the mask source.
+
+ Version 2 stores predate the CF `_FillValue` convention, and this is what
+ their writers meant - which is what released xarray's
+ `use_zarr_fill_value_as_mask` does.
+ """
+ group = zarr.open_group(store=tmp_path / "z2.zarr", mode="w", zarr_format=2)
+ array = stores.make_array(
+ group, "v", (3,), "i2", 2, dimensions=("p",), attributes={"units": "1"}
+ )
+ array[:] = [0, 1, 2]
+
+ cube = list(load_cubes(str(tmp_path / "z2.zarr")))[0]
+ assert cube.data.mask.tolist() == [True, False, False]
+
+
+def test_a_cf_fill_value_beats_the_storage_one(tmp_path):
+ group = zarr.open_group(store=tmp_path / "z.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ (3,),
+ "i2",
+ 3,
+ dimensions=("p",),
+ fill_value=0,
+ attributes={"units": "1", "_FillValue": -999},
+ )
+ array[:] = [0, -999, 2]
+
+ cube = list(load_cubes(str(tmp_path / "z.zarr")))[0]
+ assert cube.data.mask.tolist() == [False, True, False]
+```
+
+- [ ] **Step 8: Run and commit**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/zarr/ -v
+```
+
+Expected: all pass. Then run §5.0 and:
+
+```bash
+git add lib/iris/fileformats/zarr lib/iris/tests/unit/fileformats/zarr
+git commit -m "Build cubes from a Zarr store
+
+Open the store, hand it to CFReader, and run the same cube-building loop
+the netCDF loader runs. One group per call: a store can hold unrelated
+datasets, so a group with no data variables names the groups that have
+some rather than returning nothing."
+```
+
+---
+### Task 11: format detection — the `ZarrStore` element and the `"Zarr"` specification
+
+**Files:**
+- Modify: `lib/iris/io/format_picker.py`
+- Modify: `lib/iris/fileformats/__init__.py`
+- Create: `lib/iris/tests/unit/io/format_picker/test_ZarrStore.py`
+
+**Interfaces:**
+- Consumes: `iris.io.decode_uri`, `iris.io._is_nczarr_fragment`;
+ `iris.fileformats.zarr.load_cubes` (Task 10).
+- Produces: `ZarrStore(FileElement)`; `iris.io.REMOTE_STORE_SCHEMES`;
+ `iris.fileformats._is_zarr_store(target)` and `_load_zarr(*args, **kwargs)`;
+ and the registered `"Zarr"` `FormatSpecification`, which Task 12 dispatches
+ to.
+
+Spec §4.7 sets the matching rule and forbids the obvious simplification:
+
+> - **locally**, the target is a directory containing `zarr.json` (version 3)
+> or `.zgroup` (version 2);
+> - **remotely**, the URI scheme is one of `s3`, `gs`, `az`, `abfs` and the
+> path ends in `.zarr`.
+>
+> **Do not add a third rule matching any path that ends in `.zarr`.** Joined by
+> "or" it swallows the remote rule whole, leaving the scheme test unreachable.
+
+One case the spec's prose does not spell out and the code must handle: the
+format agent is asked about NCZarr URLs too, because `load_http` calls
+`get_spec(url, None)` with the fragment still attached. A local NCZarr target
+— `file:///path/store.zarr#mode=nczarr,file` — is a real Zarr store on disk, so
+the marker-file test says yes unless the fragment is checked first. Spec §4.7's
+claim that "the native reader and the NCZarr one are never candidates for the
+same target" is true of what the *user* asked for, and this is what makes the
+code agree with it.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `lib/iris/tests/unit/io/format_picker/test_ZarrStore.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Recognising a Zarr store, locally by its marker file and remotely by its URL."""
+
+import pytest
+
+from iris.fileformats import _is_zarr_store
+from iris.io.format_picker import ZarrStore
+
+
+def test_the_element_carries_the_whole_target():
+ # Not the basename: recognising a local store means reading a file inside
+ # it, so the element needs the path, not the last component of it.
+ element = ZarrStore()
+ assert element.requires_fh is False
+ assert element.get_element("/data/store.zarr", None) == "/data/store.zarr"
+
+
+def test_a_version_3_store_is_recognised(tmp_path):
+ store = tmp_path / "v3.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}')
+ assert _is_zarr_store(str(store)) is True
+
+
+def test_a_version_2_store_is_recognised(tmp_path):
+ store = tmp_path / "v2.zarr"
+ store.mkdir()
+ (store / ".zgroup").write_text('{"zarr_format": 2}')
+ assert _is_zarr_store(str(store)) is True
+
+
+def test_a_directory_named_zarr_with_no_marker_is_not_a_store(tmp_path):
+ """The suffix alone is not the local test, and must not become one.
+
+ Reading the marker file is what distinguishes a real store from a
+ directory someone happened to name `.zarr`, and locally that costs one
+ stat.
+ """
+ store = tmp_path / "pretend.zarr"
+ store.mkdir()
+ assert _is_zarr_store(str(store)) is False
+
+
+def test_a_plain_file_is_not_a_store(tmp_path):
+ path = tmp_path / "data.nc"
+ path.write_bytes(b"CDF\x01")
+ assert _is_zarr_store(str(path)) is False
+
+
+@pytest.mark.parametrize("scheme", ["s3", "gs", "az", "abfs"])
+def test_a_remote_store_is_recognised_by_scheme_and_suffix(scheme):
+ assert _is_zarr_store(f"{scheme}://bucket/path/data.zarr") is True
+ assert _is_zarr_store(f"{scheme}://bucket/path/data.zarr/") is True
+
+
+def test_a_remote_url_without_the_suffix_is_not_a_store():
+ # Remotely the suffix is all there is to go on: reading a marker file
+ # would cost a network round trip before Iris knows it wants the target.
+ assert _is_zarr_store("s3://bucket/path/data") is False
+
+
+def test_an_http_url_is_not_a_store():
+ # OPeNDAP, which the netCDF loader already claims.
+ assert _is_zarr_store("https://example.invalid/data.zarr") is False
+
+
+def test_an_nczarr_url_is_not_a_native_store(tmp_path):
+ """A user who writes `#mode=nczarr` has asked for NCZarr by name.
+
+ The store on disk is a real one, so the marker-file test says yes unless
+ the fragment is checked first - and `load_http` passes the fragment
+ through to the format agent.
+ """
+ store = tmp_path / "s.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}')
+ assert _is_zarr_store(f"file://{store}#mode=nczarr,file") is False
+
+
+def test_a_non_string_target_is_not_a_store():
+ # `load_data_objects` passes open datasets, not paths.
+ assert _is_zarr_store(object()) is False
+```
+
+- [ ] **Step 2: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/io/format_picker/test_ZarrStore.py -v
+```
+
+Expected: `ImportError: cannot import name 'ZarrStore'`.
+
+- [ ] **Step 3: Add the file element**
+
+In `lib/iris/io/format_picker.py`, after `UriProtocol`:
+
+```python
+class ZarrStore(FileElement):
+ """A :class:`FileElement` that returns a target that may be a Zarr store.
+
+ Carries the whole target rather than its basename, because recognising a
+ local store means reading a marker file inside it, and recognising a
+ remote one means reading its scheme.
+
+ """
+
+ def __init__(self):
+ FileElement.__init__(self, requires_fh=False)
+
+ def get_element(self, basename, file_handle):
+ """Return `basename`, which here is the whole path or URL."""
+ return basename
+```
+
+- [ ] **Step 4: Add the schemes and the matching rule**
+
+In `lib/iris/io/__init__.py`, beside `_is_nczarr_fragment`:
+
+```python
+#: URI schemes Iris hands to a store library rather than resolving as a local
+#: path. Iris depends on none of them: zarr-python raises its own ImportError
+#: naming what to install.
+REMOTE_STORE_SCHEMES = frozenset({"s3", "gs", "az", "abfs"})
+```
+
+In `lib/iris/fileformats/__init__.py`, after `_uri_is_nczarr`:
+
+```python
+#: What a version 3 and a version 2 Zarr store put at the root of a store.
+_ZARR_MARKERS = ("zarr.json", ".zgroup")
+
+
+def _is_zarr_store(target):
+ """Whether `target` names a native Zarr store.
+
+ Locally the test is a marker file, which costs one stat and is what tells
+ a real store from a directory someone named ``.zarr``. Remotely it is the
+ scheme and the suffix, because reading a marker file would cost a network
+ round trip before Iris knows it wants the target at all; a remote
+ ``.zarr`` that turns out not to be a store fails when zarr-python opens
+ it, with zarr-python's message.
+
+ These are two tests for two kinds of target, not two chances for one. A
+ third rule matching any path ending in ``.zarr`` would subsume the remote
+ one and leave the scheme test unreachable.
+ """
+ if not isinstance(target, str):
+ # iris.io.load_data_objects passes open datasets, not paths.
+ return False
+
+ scheme, part, fragment = decode_uri(target)
+ if _is_nczarr_fragment(fragment):
+ # The user asked for netCDF-c's Zarr driver by name. The store on
+ # disk is a real one, so the local test below would otherwise claim it.
+ return False
+
+ if scheme in REMOTE_STORE_SCHEMES:
+ return part.rstrip("/").endswith(".zarr")
+
+ directory = Path(part)
+ return any((directory / marker).is_file() for marker in _ZARR_MARKERS)
+
+
+def _load_zarr(*args, **kwargs):
+ """Load from Zarr stores, reporting clearly when zarr-python is absent."""
+ try:
+ from iris.fileformats.zarr import load_cubes
+ except ImportError as exception:
+ message = (
+ "Unable to load a Zarr store - the 'zarr' package is not "
+ "installed. Install zarr >=3.0.8, for example with "
+ "'conda install -c conda-forge zarr'."
+ )
+ raise ImportError(message) from exception
+
+ return load_cubes(*args, **kwargs)
+```
+
+Add `from pathlib import Path` and extend the existing
+`from iris.io import _is_nczarr_fragment` to
+`from iris.io import REMOTE_STORE_SCHEMES, _is_nczarr_fragment, decode_uri`,
+and add `ZarrStore` to the `iris.io.format_picker` import list.
+
+`_load_zarr` follows `_load_grib`, three functions above it: a module-level
+shim is how Iris already defers an optional format dependency, and it keeps
+`iris.fileformats` importable without zarr.
+
+- [ ] **Step 5: Register the specification**
+
+In `lib/iris/fileformats/__init__.py`, under the `# Zarr files.` heading,
+**before** the `NcZarr` specification so that the two read in version order:
+
+```python
+FORMAT_AGENT.add_spec(
+ FormatSpecification(
+ "Zarr",
+ ZarrStore(),
+ _is_zarr_store,
+ _load_zarr,
+ priority=5,
+ constraint_aware_handler=True,
+ )
+)
+```
+
+- [ ] **Step 6: Run the tests**
+
+```bash
+pytest lib/iris/tests/unit/io/ -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 7: Prove the registration does not disturb the other formats**
+
+Append to `test_ZarrStore.py`:
+
+```python
+def test_the_agent_still_picks_netcdf_for_a_netcdf_file(tmp_path):
+ """A `requires_fh=False` spec is consulted for every target with no buffer.
+
+ Adding one risks claiming a target another format already handles, and
+ the format agent returns the first match rather than the best.
+ """
+ import iris.tests as tests
+ from iris.fileformats import FORMAT_AGENT
+
+ path = tests.get_data_path(("NetCDF", "global", "xyt", "SMALL_hires.nc"))
+ with open(path, "rb") as handle:
+ spec = FORMAT_AGENT.get_spec("SMALL_hires.nc", handle)
+ assert spec.name.startswith("NetCDF")
+
+
+def test_the_agent_picks_zarr_for_a_store(tmp_path):
+ from iris.fileformats import FORMAT_AGENT
+
+ store = tmp_path / "s.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}')
+ assert FORMAT_AGENT.get_spec(str(store), None).name == "Zarr"
+
+
+def test_the_agent_still_picks_nczarr_for_an_nczarr_url(tmp_path):
+ from iris.fileformats import FORMAT_AGENT
+
+ store = tmp_path / "s.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}')
+ uri = f"file://{store}#mode=nczarr,file"
+ assert FORMAT_AGENT.get_spec(uri, None).name == "NcZarr"
+```
+
+The first needs `iris-test-data`; it is a skip if the repository is missing,
+which is why §5.0 checks the path.
+
+- [ ] **Step 8: Run, then commit**
+
+```bash
+pytest lib/iris/tests/unit/io/ lib/iris/tests/unit/fileformats/ -q
+```
+
+Expected: all pass. Then run §5.0 and:
+
+```bash
+git add lib/iris/io/format_picker.py lib/iris/fileformats/__init__.py lib/iris/tests/unit/io
+git commit -m "Teach the format agent to recognise a Zarr store
+
+Locally by the marker file a store puts at its root, remotely by the URI
+scheme and the .zarr suffix. Not by the suffix alone: that would claim any
+directory someone named .zarr, and joined with the remote rule it would
+leave the scheme test unreachable. An NCZarr URL is still NCZarr's, which
+needs saying in code because the store it names is a real one."
+```
+
+---
+
+### Task 12: the load chain — a target that is not a file
+
+**Files:**
+- Modify: `lib/iris/io/__init__.py`
+- Modify: `lib/iris/loading.py`
+- Modify: `lib/iris/fileformats/netcdf/ugrid_load.py`
+- Create: `lib/iris/tests/unit/io/test_load_stores.py`
+- Modify: `lib/iris/tests/unit/io/test_expand_filespecs.py` (or the module
+ holding those tests today)
+
+**Interfaces:**
+- Consumes: `REMOTE_STORE_SCHEMES`, the `"Zarr"` specification (Task 11).
+- Produces: `iris.io.load_stores(targets, callback, constraints=None)`;
+ `iris.io._is_remote_store_uri(spec)`; `iris.io._CUBELIST_SAVER_MODULES`; and
+ `iris.load("store.zarr")` and `iris.load("s3://bucket/store.zarr")` working.
+
+Spec §4.7's four changes, and the one consequence it names:
+
+> The one thing this costs is that the file-or-store decision has to happen
+> *after* glob expansion and *before* format dispatch, because `iris.load`
+> must still accept `"data/*"` matching a mix of `.nc` files and `.zarr`
+> stores.
+
+- [ ] **Step 1: Write the failing tests for `load_stores`**
+
+Create `lib/iris/tests/unit/io/test_load_stores.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :func:`iris.io.load_stores`."""
+
+import numpy as np
+import pytest
+
+import iris
+from iris.io import _is_remote_store_uri, load_stores
+
+zarr = pytest.importorskip("zarr", minversion="3.0.8")
+
+from iris.tests.unit.fileformats.zarr import stores # noqa: E402
+
+
+def test_a_local_store_loads(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ cubes = list(load_stores([location], None))
+ assert [cube.var_name for cube in cubes] == ["air_temperature"]
+
+
+def test_several_stores_in_one_call(tmp_path):
+ first = stores.make_store(tmp_path / "a.zarr")
+ second = stores.make_store(tmp_path / "b.zarr")
+ assert len(list(load_stores([first, second], None))) == 2
+
+
+def test_constraints_reach_the_handler(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ constraint = iris.Constraint(cube_func=lambda cube: False)
+ assert list(load_stores([location], None, constraint)) == []
+
+
+@pytest.mark.parametrize(
+ "uri, expected",
+ [
+ ("s3://bucket/data.zarr", True),
+ ("gs://bucket/data.zarr", True),
+ ("az://container/data.zarr", True),
+ ("abfs://container/data.zarr", True),
+ ("https://example.invalid/data.nc", False),
+ ("/local/path/data.zarr", False),
+ ("relative/data.zarr", False),
+ ],
+)
+def test_which_uris_are_remote_stores(uri, expected):
+ assert _is_remote_store_uri(uri) is expected
+```
+
+- [ ] **Step 2: Write the failing tests for the chain**
+
+Append:
+
+```python
+def test_iris_load_accepts_a_local_store(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ cubes = iris.load(location)
+ assert len(cubes) == 1
+ np.testing.assert_array_equal(cubes[0].data, stores.DATA)
+
+
+def test_iris_load_accepts_a_remote_url_and_does_not_absolutise_it():
+ """A remote URL must reach zarr-python intact.
+
+ `expand_filespecs` would otherwise turn `s3://bucket/data.zarr` into
+ `/<cwd>/s3:/bucket/data.zarr`. No network is touched: opening the store
+ fails, and what matters is that the message still names the URL.
+ """
+ with pytest.raises(Exception) as caught:
+ iris.load("s3://bucket/data.zarr")
+ assert "s3://bucket/data.zarr" in str(caught.value) or isinstance(
+ caught.value, ImportError
+ )
+
+
+def test_a_glob_matching_both_files_and_stores(tmp_path):
+ """`iris.load("data/*")` must accept a mix.
+
+ The file-or-store decision happens after glob expansion and before format
+ dispatch. Mis-partitioning raises IsADirectoryError from `load_files` for
+ the stores while the netCDF files load fine, which reads as an Iris bug
+ to a user who did not expect the glob to match a store.
+ """
+ import iris.tests as tests
+
+ source = iris.load_cube(
+ tests.get_data_path(("NetCDF", "global", "xyt", "SMALL_hires.nc"))
+ )
+ iris.save(source, str(tmp_path / "a.nc"))
+ stores.make_store(tmp_path / "b.zarr")
+
+ cubes = iris.load(str(tmp_path / "*"))
+ names = sorted(cube.var_name or cube.name() for cube in cubes)
+ assert "air_temperature" in names
+ assert len(cubes) == 2
+```
+
+- [ ] **Step 3: Run them to verify they fail**
+
+```bash
+pytest lib/iris/tests/unit/io/test_load_stores.py -v
+```
+
+Expected: `ImportError: cannot import name 'load_stores'`, and once that is
+written, `IsADirectoryError` from `iris.load`.
+
+- [ ] **Step 4: Teach `expand_filespecs` about remote schemes**
+
+In `lib/iris/io/__init__.py`, beside `REMOTE_STORE_SCHEMES`:
+
+```python
+def _is_remote_store_uri(spec):
+ """Whether `spec` names a store Iris hands to a store library untouched.
+
+ Such a URI must not be resolved as a local path: `.absolute()` turns
+ ``s3://bucket/data.zarr`` into ``/<cwd>/s3:/bucket/data.zarr``.
+ """
+ if not isinstance(spec, str):
+ return False
+ scheme, _part, _fragment = decode_uri(spec)
+ return scheme in REMOTE_STORE_SCHEMES
+```
+
+and in `expand_filespecs`, replace the two statements that build `filenames`
+and `glob_expanded`:
+
+```python
+ # Remove any hostname component - currently unused. A remote store URI is
+ # passed through: it is not a path, and resolving it mangles it.
+ filenames = [
+ fn
+ if _is_remote_store_uri(fn)
+ else str(pathlib.Path(fn.removeprefix("//")).expanduser().absolute())
+ for fn in file_specs
+ ]
+
+ if files_expected:
+ # Try to expand all filenames as globs. A remote URI stands for
+ # itself; Iris does not list a remote store's neighbours.
+ glob_expanded = OrderedDict(
+ [
+ [fn, [fn] if _is_remote_store_uri(fn) else sorted(glob.glob(fn))]
+ for fn in filenames
+ ]
+ )
+```
+
+- [ ] **Step 5: Write `load_stores`**
+
+In `lib/iris/io/__init__.py`, after `load_files`:
+
+```python
+def load_stores(targets, callback, constraints=None):
+ """Create a generator of Cubes from targets that are stores, not files.
+
+ A store is a directory or a URL: something with no single seekable file
+ to open, so :func:`load_files` cannot reach it and the format agent is
+ asked without a buffer.
+
+ Parameters
+ ----------
+ targets : iterable of str
+ Local directory paths or remote URLs, already expanded.
+ callback : callable
+ Passed to the format handler.
+ constraints : optional
+ Passed to a constraint-aware format handler.
+
+ Yields
+ ------
+ iris.cube.Cube
+
+ Notes
+ -----
+ Typically, this function should not be called directly; instead, the
+ intended interface for loading is :func:`iris.load`.
+
+ """
+ from iris.fileformats import FORMAT_AGENT
+
+ handler_map = collections.defaultdict(list)
+ for target in targets:
+ handling_format_spec = FORMAT_AGENT.get_spec(target, None)
+ handler_map[handling_format_spec].append(target)
+
+ for handling_format_spec in sorted(handler_map):
+ names = handler_map[handling_format_spec]
+ if handling_format_spec.constraint_aware_handler:
+ for cube in handling_format_spec.handler(names, callback, constraints):
+ yield cube
+ else:
+ for cube in handling_format_spec.handler(names, callback):
+ yield cube
+```
+
+- [ ] **Step 6: Partition in `_generate_cubes`**
+
+In `lib/iris/loading.py`, replace the `file` branch and the scheme list:
+
+```python
+ if category == "file":
+ part_names = [part for _cat, _scheme, part in groups]
+ # The file-or-store decision has to happen after glob expansion
+ # and before format dispatch, because a glob may match both.
+ # Neither routine below learns about the other's kind of target.
+ expanded = iris.io.expand_filespecs(part_names)
+ files = [name for name in expanded if not Path(name).is_dir()]
+ stores = [name for name in expanded if Path(name).is_dir()]
+ if files:
+ for cube in iris.io.load_files(files, callback, constraints):
+ yield cube
+ if stores:
+ for cube in iris.io.load_stores(stores, callback, constraints):
+ yield cube
+ elif category in ["http", "https", "nczarr"]:
+ urls = [f"{scheme}:{part}" for _cat, scheme, part in groups]
+ for cube in iris.io.load_http(urls, callback):
+ yield cube
+ elif category in iris.io.REMOTE_STORE_SCHEMES:
+ urls = [f"{scheme}:{part}" for _cat, scheme, part in groups]
+ for cube in iris.io.load_stores(urls, callback, constraints):
+ yield cube
+ elif category == "data":
+```
+
+Add `from pathlib import Path` to `loading.py`'s imports.
+
+- [ ] **Step 7: Run the chain tests**
+
+```bash
+pytest lib/iris/tests/unit/io/ -v
+```
+
+Expected: all pass.
+
+- [ ] **Step 8: Update the saver-side pair**
+
+Still in `lib/iris/io/__init__.py`. Beside `_savers`:
+
+```python
+#: Modules whose `save` takes a CubeList directly, rather than one cube at a
+#: time with `append=True`. One line per format.
+_CUBELIST_SAVER_MODULES = frozenset({"iris.fileformats.netcdf.saver"})
+```
+
+In `find_saver`:
+
+```python
+ _check_init_savers()
+ # A store is a directory, so "out.zarr/" is a natural thing to type.
+ candidate = filespec.rstrip("/\\").lower()
+ matches = [
+ ext for ext in _savers if candidate.endswith("." + ext) or candidate == ext
+ ]
+```
+
+In `save`, replace the substring test:
+
+```python
+ # Only allow cubelist saving for those fileformats that are capable.
+ if saver.__module__ not in _CUBELIST_SAVER_MODULES:
+```
+
+- [ ] **Step 9: Test the saver-side pair**
+
+Append to `test_load_stores.py`:
+
+```python
+def test_find_saver_tolerates_a_trailing_separator():
+ from iris.io import find_saver
+
+ assert find_saver("out.nc") is find_saver("out.nc/")
+
+
+def test_the_cubelist_saver_set_names_the_netcdf_saver():
+ """The substring test it replaces matched by accident of module naming.
+
+ `"iris.fileformats.netcdf" not in saver.__module__` is true of any module
+ whose path happens to contain that string, and says nothing a reader can
+ check. The set is one line per format.
+ """
+ from iris.fileformats.netcdf import save as netcdf_save
+ from iris.io import _CUBELIST_SAVER_MODULES
+
+ assert netcdf_save.__module__ in _CUBELIST_SAVER_MODULES
+
+
+def test_saving_a_cubelist_to_netcdf_still_takes_one_call(tmp_path):
+ import iris.tests as tests
+
+ cubes = iris.load(tests.get_data_path(("NetCDF", "global", "xyt", "SMALL_hires.nc")))
+ iris.save(cubes, str(tmp_path / "out.nc"))
+ assert len(iris.load(str(tmp_path / "out.nc"))) == len(cubes)
+```
+
+- [ ] **Step 10: Update the mesh loader's copy of the dispatch**
+
+In `lib/iris/fileformats/netcdf/ugrid_load.py`, the `load_meshes` scheme
+dispatch is a second copy of `_generate_cubes`'s and moves with it:
+
+```python
+ for category, groups in groupby(uri_tuples, key=lambda x: x[0]):
+ # Call each scheme handler with the appropriate URIs
+ if category == "file":
+ filenames = [part for _cat, _scheme, part in groups]
+ sources = expand_filespecs(filenames)
+ elif category in ["http", "https", "nczarr"]:
+ sources = [f"{scheme}:{part}" for _cat, scheme, part in groups]
+ elif category in REMOTE_STORE_SCHEMES:
+ sources = [f"{scheme}:{part}" for _cat, scheme, part in groups]
+ else:
+ message = f"Iris cannot handle the URI scheme: {category}"
+ raise ValueError(message)
+
+ for source in sources:
+ if category == "file" and not Path(source).is_dir():
+ with open(source, "rb") as fh:
+ handling_format_spec = FORMAT_AGENT.get_spec(Path(source).name, fh)
+ else:
+ handling_format_spec = FORMAT_AGENT.get_spec(source, None)
+
+ if handling_format_spec.handler == nc_loader.load_cubes:
+ valid_sources.append(source)
+ else:
+ message = f"Ignoring non-NetCDF file: {source}"
+ logger.info(msg=message, extra=dict(cls=None))
+```
+
+Add `REMOTE_STORE_SCHEMES` to its `iris.io` import.
+
+A Zarr store reaches `CFReader(source)` as a path, which opens a
+`NetCDFDataset` and fails — so `load_meshes` does not yet build a mesh from
+Zarr, and must not claim to. The handler check above rejects it: the `"Zarr"`
+specification's handler is `_load_zarr`, not `nc_loader.load_cubes`, so a Zarr
+store is logged and skipped exactly as a PP file is. That is the honest
+behaviour for this pull request, and it is what the test below pins.
+
+- [ ] **Step 11: Test the mesh dispatch**
+
+Append to `test_load_stores.py`:
+
+```python
+def test_load_meshes_skips_a_zarr_store_rather_than_raising(tmp_path):
+ """Reaching a store is this pull request's change; reading a mesh from
+ one is not. What must not happen is the ValueError the scheme dispatch
+ raised for anything that was not a file.
+ """
+ from iris.mesh import load_meshes
+
+ stores.make_store(tmp_path / "s.zarr")
+ assert load_meshes(str(tmp_path / "s.zarr")) == {}
+```
+
+- [ ] **Step 12: Run the neighbourhood**
+
+```bash
+pytest -n auto lib/iris/tests/unit/io/ lib/iris/tests/unit/fileformats/ lib/iris/tests/unit/test_load*.py -q
+pytest -n auto lib/iris/tests/integration/netcdf/ -q
+```
+
+Expected: all pass, with no change to the netCDF counts.
+
+- [ ] **Step 13: Commit**
+
+Run §5.0, then:
+
+```bash
+git add lib/iris/io lib/iris/loading.py lib/iris/fileformats/netcdf/ugrid_load.py lib/iris/tests/unit
+git commit -m "Let the load chain reach a target that is not a file
+
+A new iris.io.load_stores answers 'this target is not a file', once, for a
+local directory and a remote URL alike, rather than teaching load_files
+about directories in one place and the scheme dispatch in another. The
+partition happens after glob expansion, so iris.load('data/*') still
+accepts a mix of netCDF files and Zarr stores."
+```
+
+---
+### Task 13: integration — a realistic cube, both Zarr versions
+
+**Files:**
+- Create: `lib/iris/tests/integration/zarr/__init__.py`
+- Create: `lib/iris/tests/integration/zarr/fixtures.py`
+- Create: `lib/iris/tests/integration/zarr/test_load.py`
+
+**Interfaces:**
+- Consumes: everything above; `iris.tests.stock.realistic_4d_w_everything`;
+ the existing NCZarr save path.
+- Produces: coverage only.
+
+PR 4 has no saver, so there is no Iris round trip to make. Two substitutes do
+the work, and between them they are better evidence than a round trip would
+be, because neither is Iris marking its own homework:
+
+- **Version 2** comes from Iris's own NCZarr `mode=xarray` save path, which
+ already writes a version 2 store with `_ARRAY_DIMENSIONS` (spec §6). Save
+ `realistic_4d_w_everything()` through it, load it back natively, and compare
+ cubes. That is a direct check that the native reader and the netCDF-c writer
+ agree.
+- **Version 3** comes from a fixture written with zarr-python, carrying the CF
+ features that matter: dimension coordinates, a two-dimensional auxiliary
+ coordinate, bounds, a scalar grid mapping, packing, a base64 `_FillValue`
+ and sharding. Those are the properties spec §7 says make the real NOAA GFS
+ fixture worth having; PR 7 replaces this with the real thing.
+
+- [ ] **Step 1: Create the package and the version 3 fixture builder**
+
+Create `lib/iris/tests/integration/zarr/__init__.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Integration tests for native Zarr loading.
+
+Skipped when zarr-python is absent, which is also how the unit tests do it.
+"""
+
+import pytest
+
+pytest.importorskip("zarr", minversion="3.0.8")
+```
+
+Create `lib/iris/tests/integration/zarr/fixtures.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""A version 3 store carrying the CF features a real one does.
+
+Modelled on the NOAA GFS archive at dynamical.org, whose properties the Zarr
+I/O design names as the ones worth exercising: version 3 `dimension_names`, a
+sharded codec, a scalar grid-mapping variable, a two-dimensional auxiliary
+coordinate, and the base64 `_FillValue` that released xarray writes for every
+floating dtype. A cut-down copy of the real store replaces this.
+"""
+
+import base64
+import struct
+
+import numpy as np
+import zarr
+
+SHAPE = (2, 4, 6, 5)
+DIMENSIONS = ("time", "level", "latitude", "longitude")
+
+
+def float_fill_value(value):
+ """Return `value` in the encoding released xarray writes for a float."""
+ return base64.b64encode(struct.pack("<d", value)).decode("ascii")
+
+
+def make_gfs_like_store(path):
+ """Write the store and return its path as a string."""
+ root = zarr.open_group(store=path, mode="w", zarr_format=3)
+ root.attrs["Conventions"] = "CF-1.12"
+ root.attrs["title"] = "A store shaped like a real one"
+ root.attrs["license"] = "CC-BY-4.0"
+
+ sizes = dict(zip(DIMENSIONS, SHAPE))
+ for name, size in sizes.items():
+ coordinate = root.create_array(
+ name, shape=(size,), dtype="f8", dimension_names=(name,)
+ )
+ coordinate[:] = np.arange(size, dtype="f8")
+ root["time"].attrs.update(
+ {"units": "hours since 2000-01-01 00:00:00", "calendar": "standard"}
+ )
+ root["level"].attrs.update({"units": "hPa", "positive": "down"})
+ root["latitude"].attrs.update({"units": "degrees_north", "axis": "Y"})
+ root["longitude"].attrs.update({"units": "degrees_east", "axis": "X"})
+
+ # A scalar grid mapping, as the GFS store's `spatial_ref` is.
+ grid = root.create_array("spatial_ref", shape=(), dtype="i4")
+ grid[()] = 0
+ grid.attrs.update(
+ {"grid_mapping_name": "latitude_longitude", "earth_radius": 6371229.0}
+ )
+
+ # A two-dimensional auxiliary coordinate, as the GFS store's valid_time is.
+ valid_time = root.create_array(
+ "valid_time",
+ shape=SHAPE[:2],
+ dtype="f8",
+ dimension_names=DIMENSIONS[:2],
+ )
+ valid_time[:] = np.arange(np.prod(SHAPE[:2]), dtype="f8").reshape(SHAPE[:2])
+ valid_time.attrs.update(
+ {
+ "units": "hours since 2000-01-01 00:00:00",
+ "standard_name": "time",
+ "statistics_approximate": {"min": 0.0, "max": 7.0},
+ }
+ )
+
+ # The data variable: sharded, packed, with a base64 fill value.
+ data = root.create_array(
+ "temperature_2m",
+ shape=SHAPE,
+ dtype="i2",
+ chunks=(1, 1, 3, 5),
+ shards=(2, 4, 6, 5),
+ dimension_names=DIMENSIONS,
+ fill_value=-32768,
+ )
+ data[:] = np.arange(np.prod(SHAPE), dtype="i2").reshape(SHAPE)
+ data.attrs.update(
+ {
+ "standard_name": "air_temperature",
+ "units": "K",
+ "scale_factor": 0.01,
+ "add_offset": 273.15,
+ # An integer variable's sentinel is a JSON number. The base64
+ # encoding is for floating dtypes, and is on `pressure_surface`.
+ "_FillValue": -32768,
+ "coordinates": "valid_time",
+ "grid_mapping": "spatial_ref",
+ }
+ )
+
+ # A floating variable, carrying the base64 `_FillValue` that released
+ # xarray writes for every floating dtype.
+ flat_shape = SHAPE[:1] + SHAPE[2:]
+ pressure = root.create_array(
+ "pressure_surface",
+ shape=flat_shape,
+ dtype="f4",
+ dimension_names=DIMENSIONS[:1] + DIMENSIONS[2:],
+ )
+ pressure[:] = np.arange(int(np.prod(flat_shape)), dtype="f4").reshape(flat_shape)
+ pressure.attrs.update(
+ {
+ "standard_name": "surface_air_pressure",
+ "units": "Pa",
+ "_FillValue": float_fill_value(float("nan")),
+ "grid_mapping": "spatial_ref",
+ }
+ )
+
+ return str(path)
+```
+
+- [ ] **Step 2: Write the version 3 integration tests**
+
+Create `lib/iris/tests/integration/zarr/test_load.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Integration tests for loading native Zarr stores."""
+
+import numpy as np
+import pytest
+
+import iris
+from iris.coord_systems import GeogCS
+from iris.tests import stock
+
+from . import fixtures
+
+
+@pytest.fixture
+def gfs_like(tmp_path):
+ return fixtures.make_gfs_like_store(tmp_path / "gfs.zarr")
+
+
+def test_the_store_loads_two_cubes(gfs_like):
+ cubes = iris.load(gfs_like)
+ assert sorted(cube.name() for cube in cubes) == [
+ "air_temperature",
+ "surface_air_pressure",
+ ]
+
+
+def test_dimension_coordinates_attach(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ assert [coord.name() for coord in cube.coords(dim_coords=True)] == [
+ "time",
+ "level",
+ "latitude",
+ "longitude",
+ ]
+ assert cube.shape == fixtures.SHAPE
+
+
+def test_the_auxiliary_coordinate_attaches_to_its_own_dimensions(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ aux = cube.coord("valid_time")
+ assert cube.coord_dims(aux) == (0, 1)
+
+
+def test_the_grid_mapping_becomes_a_coordinate_system(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ assert isinstance(cube.coord("latitude").coord_system, GeogCS)
+
+
+def test_packing_is_applied(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ assert cube.dtype == np.dtype("f8")
+ # The first stored value is 0, so scale_factor * 0 + add_offset.
+ assert cube.data.flat[0] == pytest.approx(273.15)
+
+
+def test_the_base64_fill_value_masks_nothing_here(gfs_like):
+ """A NaN sentinel over data with no NaNs masks nothing.
+
+ The attribute is still read, and read wrongly it raises rather than
+ quietly doing nothing - which is why this asserts the unmasked result
+ rather than only that loading succeeded.
+ """
+ cube = iris.load_cube(gfs_like, "surface_air_pressure")
+ assert not np.ma.is_masked(cube.data)
+
+
+def test_the_cube_is_lazy_and_chunked_to_the_store(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ assert cube.has_lazy_data()
+ # Whatever the chunking works out to, it tiles the store's inner chunk
+ # grid: a Zarr chunk is read whole, so a dask block that subdivides one
+ # costs a re-read.
+ store_chunks = (1, 1, 3, 5)
+ for sizes, store_size in zip(cube.lazy_data().chunks, store_chunks):
+ assert all(size % store_size == 0 for size in sizes[:-1])
+
+
+def test_a_structured_attribute_survives(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ assert cube.coord("valid_time").attributes["statistics_approximate"] == {
+ "min": 0.0,
+ "max": 7.0,
+ }
+```
+
+- [ ] **Step 3: Run them**
+
+```bash
+pytest lib/iris/tests/integration/zarr/ -v
+```
+
+Expected: all pass. Where one does not, decide which side is wrong before
+changing anything: a fixture that is not CF-conforming is a fixture bug, and a
+loader that cannot read conforming CF is a loader bug.
+
+- [ ] **Step 4: Write the version 2 cross-writer test**
+
+Append to `test_load.py`:
+
+```python
+@pytest.fixture
+def realistic_4d_w_everything():
+ """Return a realistic 4D cube with all the bells and whistles."""
+ cube = stock.realistic_4d_w_everything()
+ # Precision loss for floating-point scalar variable attributes:
+ # earth_radius 6371229.0 round-trips as 6371230.0.
+ for coord_name in ("grid_latitude", "grid_longitude"):
+ cube.coord(coord_name).coord_system.ellipsoid = GeogCS(6371230.0)
+ return cube
+
+
+def test_the_native_reader_agrees_with_the_nczarr_writer(
+ tmp_path, realistic_4d_w_everything
+):
+ """A version 2 store Iris wrote through netCDF-c, read natively.
+
+ The NCZarr `xarray` mode writes `_ARRAY_DIMENSIONS`, which is how a
+ version 2 store records dimension names. The two code paths share no
+ code, so agreement between them is evidence rather than a tautology.
+ """
+ path = tmp_path / "nczarr.zarr"
+ uri = path.as_uri() + "#mode=xarray,file"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(realistic_4d_w_everything, uri)
+
+ assert (path / ".zgroup").is_file()
+ loaded = iris.load_cube(str(path))
+ assert loaded == realistic_4d_w_everything
+
+
+def test_array_dimensions_does_not_reach_the_cube(tmp_path, realistic_4d_w_everything):
+ """It is how version 2 records dimension names, so it is storage.
+
+ Left in the CF attributes it would land on every cube loaded from a
+ version 2 store, and make it unequal to the same data in version 3.
+ """
+ path = tmp_path / "nczarr.zarr"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(realistic_4d_w_everything, path.as_uri() + "#mode=xarray,file")
+
+ cube = iris.load_cube(str(path))
+ assert "_ARRAY_DIMENSIONS" not in cube.attributes
+ for coord in cube.coords():
+ assert "_ARRAY_DIMENSIONS" not in coord.attributes
+```
+
+- [ ] **Step 5: Run, and read the skip count**
+
+```bash
+pytest lib/iris/tests/integration/zarr/ -v
+```
+
+Expected: all pass, none skipped. A skip here means zarr is missing from the
+environment, which removes the whole task's coverage.
+
+- [ ] **Step 6: Commit**
+
+Run §5.0, then:
+
+```bash
+git add lib/iris/tests/integration/zarr
+git commit -m "Load realistic Zarr stores end to end, both versions
+
+Version 2 comes from Iris's own NCZarr writer, so the native reader and
+netCDF-c are checked against each other. Version 3 comes from a fixture
+shaped like the NOAA GFS archive: sharded, packed, with a scalar grid
+mapping, a two-dimensional auxiliary coordinate and a base64 fill value."
+```
+
+---
+
+### Task 14: documentation
+
+**Files:**
+- Create: `docs/src/user_manual/explanation/zarr_io.rst`
+- Modify: `docs/src/user_manual/section_indexes/general.rst`
+- Modify: `docs/src/user_manual/tutorial/s3_io.rst`
+
+**Interfaces:**
+- Consumes: the working loader.
+- Produces: user-facing documentation for Zarr loading, remote stores and
+ `group=`.
+
+The page is `explanation/zarr_io.rst`, beside `explanation/netcdf_io.rst`,
+because that is the page a user comparing the two formats will look for and
+because PR 6 adds saving to the same page.
+
+- [ ] **Step 1: Write the page**
+
+Create `docs/src/user_manual/explanation/zarr_io.rst`:
+
+```rst
+.. explanation:: Zarr I/O Handling in Iris
+ :tags: topic_load_save
+
+ Read about how Iris loads Zarr stores.
+
+.. include:: ../../common_links.inc
+
+.. _zarr_io:
+
+=========================
+Zarr I/O Handling in Iris
+=========================
+
+.. readingtime::
+
+`Zarr <https://zarr.dev/>`_ stores an array as many independently compressed
+chunks under a prefix, rather than as one file. Iris reads Zarr version 2 and
+version 3 stores natively, through
+`zarr-python <https://zarr.readthedocs.io/>`_.
+
+``zarr`` is an **optional dependency**. Install it to load Zarr stores::
+
+ conda install -c conda-forge zarr
+
+Iris without it behaves exactly as before, and raises an ``ImportError``
+naming the package if you ask it to load a store.
+
+Loading
+-------
+
+A store is a directory, or a URL, so pass it where you would pass a file::
+
+ cubes = iris.load("/data/forecast.zarr")
+
+Iris recognises a local store by the metadata file at its root - ``zarr.json``
+for version 3, ``.zgroup`` for version 2 - not by the ``.zarr`` suffix, so a
+directory that merely has the suffix is not mistaken for one.
+
+Remote stores
+-------------
+
+A URL with one of the ``s3``, ``gs``, ``az`` or ``abfs`` schemes, ending in
+``.zarr``, is passed to zarr-python unaltered::
+
+ cubes = iris.load("s3://noaa-gfs-bdp-pds/forecast.zarr")
+
+Iris adds no store library of its own and handles no credentials. Install the
+one your scheme needs -- ``s3fs``, ``gcsfs``, ``adlfs`` -- and configure it as
+that package documents. zarr-python raises an ``ImportError`` naming the
+package if it is missing.
+
+Groups
+------
+
+A Zarr store may hold several groups, and they need not describe the same
+thing. Iris loads **one** group per call, like xarray, rather than merging
+them: ``iris.load`` reads the root group, and a store whose root holds no data
+variables raises a message naming the groups that do.
+
+To read another group, call the Zarr loader directly::
+
+ from iris.cube import CubeList
+ import iris.fileformats.zarr
+
+ cubes = CubeList(
+ iris.fileformats.zarr.load_cubes(
+ "/data/sentinel.zarr", group="measurements/reflectance/r10m"
+ )
+ )
+
+Dimension names
+---------------
+
+A netCDF dimension has a name, one length, and a meaning shared across the
+whole file. A Zarr dimension name is weaker in three ways, and all three are
+permitted by the version 3 specification:
+
+* an array need not name its dimensions at all;
+* it may name some and not others;
+* the same name in two arrays guarantees nothing -- not even the same length.
+
+So Iris matches dimensions on the pair *(name, length)*. Two arrays share a
+dimension when they name it identically **and** it is the same length in both.
+A dimension with no name is **anonymous**, and never matches anything; the
+cube simply has a dimension with no coordinate on it, which is an ordinary
+Iris cube. A dimension name used at two different lengths is reported with a
+warning and links nothing.
+
+Version 2 stores have no dimension-name field, and carry names in an
+``_ARRAY_DIMENSIONS`` attribute instead -- the convention xarray established.
+Iris reads it, and does not pass it on to the cube.
+
+Missing data
+------------
+
+Zarr applies no masking of its own, so Iris does it, from the CF attributes:
+``_FillValue`` and ``missing_value`` for sentinels,
+``valid_min``/``valid_max``/``valid_range`` for a range, and
+``scale_factor``/``add_offset`` for packed data.
+
+A Zarr array also has a ``fill_value`` in its metadata. This is **not** a
+missing-value declaration: it says what a chunk that was never written reads
+back as, and zarr-python defaults it to zero. Iris therefore ignores it as a
+mask source on version 3, where it is a required field, and uses it on version
+2 only when the array declares no CF attribute -- which is what version 2
+writers meant by it, and what xarray does.
+
+Chunking
+--------
+
+Iris chooses Dask chunks from the store's own chunking, as it does for netCDF,
+and :data:`~iris.fileformats.cf.loader.CHUNK_CONTROL` works the same way for
+both formats.
+
+One rule is specific to Zarr. A Zarr chunk is the atomic unit of storage:
+reading any part of one fetches and decompresses the whole object. So Iris
+rounds the computed chunking **up** to a whole multiple of the store's
+chunking, never below it. When a single store chunk is already larger than
+Dask's target chunk size, the store's layout is the binding constraint on
+memory per task, and Iris warns to say so.
+
+A **sharded** array is chunked on its inner chunks, not its shards. Zarr's
+sharding codec is indexed, so a reader fetches only the inner chunks it needs.
+
+zarr-python fetches chunks concurrently, under its own
+``zarr.config`` setting ``async.concurrency``. Iris does not change it. Dask's
+worker count and that concurrency multiply, so a large Dask pool over a remote
+store can open far more connections than either number suggests; if you are
+rate-limited, lower one of them.
+
+Attributes
+----------
+
+A Zarr attribute value may be any JSON literal, including a nested object or
+an array of arrays, which CF cannot express at any rank. Iris carries such a
+value onto the cube unchanged and does not interpret it. See
+:issue:`7288` for the wider question of how the CF attribute model should map
+onto JSON.
+
+.. note::
+
+ A cube carrying such an attribute **cannot be saved as netCDF**, because
+ netCDF attribute values are scalars, strings and one-dimensional arrays.
+ The failure comes from the netCDF library and does not name the attribute;
+ see :issue:`7329`. Delete the attribute before saving::
+
+ del cube.attributes["stac_discovery"]
+
+NCZarr
+------
+
+Iris has long been able to read and write Zarr through netCDF-c's NCZarr
+driver, by way of a URL fragment::
+
+ iris.load("file:///data/forecast.zarr#mode=nczarr,file")
+
+That is unchanged, and is still how you reach NCZarr. NCZarr implements Zarr
+version 2 and writes scalar variables as length-one arrays; the native reader
+described above handles both versions and is what a plain path or URL now
+uses.
+```
+
+Check `:issue:` is the role `common_links.inc` provides; if it is not, write
+the full URL.
+
+- [ ] **Step 2: List it**
+
+In `docs/src/user_manual/section_indexes/general.rst`, add to the toctree,
+keeping the alphabetical order the list is in:
+
+```rst
+ ../explanation/zarr_io
+```
+
+- [ ] **Step 3: Rewrite the S3 page's premise**
+
+`docs/src/user_manual/tutorial/s3_io.rst` says S3 is unsupported and
+recommends a FUSE mount. That is still true for netCDF and no longer true for
+Zarr. Change the sphinx-needs summary and add a section near the top, before
+"TL;DR":
+
+```rst
+Zarr stores are accessed directly
+---------------------------------
+
+Everything below concerns **files** -- netCDF, PP, GRIB -- which Iris reaches
+through a file-system path. A :ref:`Zarr store <zarr_io>` is different: Iris
+passes an ``s3://`` URL straight to zarr-python, which reads the objects it
+needs over the network::
+
+ cubes = iris.load("s3://my-bucket/data.zarr")
+
+That needs ``s3fs`` installed and configured, and no mount. If your data is in
+Zarr, prefer this to the FUSE approach described below.
+```
+
+and amend the paragraph beginning "While it would be possible for Iris to
+support S3 access directly" to say that this is what the Zarr path now does,
+and that the limitation it describes applies to the file formats.
+
+- [ ] **Step 4: Check the metadata and build**
+
+```bash
+pre-commit run check-docs-page-metadata
+cd docs && make html-noplot
+```
+
+Expected: the hook passes, and the build is warning-free. Warnings are errors
+in the standard build, so a bad cross-reference fails CI.
+
+- [ ] **Step 5: Read the generated API pages**
+
+`sphinxcontrib-apidoc` writes one page per non-private module from its
+docstring, so `iris.fileformats.zarr` and `iris.fileformats.zarr.loader` get
+pages from what Tasks 1 and 10 wrote. Open them and read them:
+
+```bash
+cd docs && make html-noplot
+python -m webbrowser src/_build/html/generated/api/iris/fileformats/zarr.html
+```
+
+`docs/AGENTS.md` gotcha 5 is the reason this is a step rather than an
+assumption: a re-exporting `__init__.py` has three failure modes, two of them
+silent. Check that `load_cubes` appears on the package page — it is in
+`__all__`, which is what autodoc needs — and that it is described once, not
+twice.
+
+- [ ] **Step 6: Run the doctests**
+
+The page has no `>>>` examples — the blocks above are `::` literals, not
+doctests, deliberately, because none of them can run without a store or a
+network. If you add a `>>>` example, run:
+
+```bash
+cd docs/src && make doctest
+```
+
+- [ ] **Step 7: Commit**
+
+Run §5.0, then:
+
+```bash
+git add docs/src
+git commit -m "Document Zarr loading
+
+The page a user comparing formats will look for, beside the netCDF one:
+what a store is, how a remote one is reached, what a Zarr dimension name
+does and does not guarantee, why the array fill value is not a
+missing-value flag, and why chunking rounds up. The S3 page no longer
+begins by saying this is impossible."
+```
+
+---
+
+### Task 15: the spec, the full-suite diff, and the pull requests
+
+**Files:**
+- Modify: `docs/superpowers/specs/2026-09-21-zarr-io-design.md`
+- Modify: `docs/superpowers/plans/2026-10-13-zarr-io-pr4.md` (this file)
+
+**Interfaces:**
+- Consumes: everything.
+- Produces: a preview pull request on the fork now, the real one against
+ `brownfield` when PR 3 merges, and a spec that tells the next reader the
+ truth.
+
+Steps 1–5 run now, on the `zarr-io-pr-3` base. **Step 6 runs later**, when
+#7316 merges, and is the only step that touches `brownfield`.
+
+- [ ] **Step 1: Run the full suite, twice**
+
+```bash
+echo "$OVERRIDE_TEST_DATA_REPOSITORY"
+git stash list # expect empty
+
+git switch --detach zarr-io-pr-3
+pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/baseline.txt
+
+git switch -
+pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/branch.txt
+diff /tmp/baseline.txt /tmp/branch.txt
+```
+
+The baseline is `zarr-io-pr-3`, **not** `brownfield`: this branch's changes are
+the ones made on top of PR 3, and measuring against `brownfield` would fold
+PR 3's ten commits into the comparison and attribute their effects here. Step 6
+repeats the comparison against `brownfield` after the rebase, which is where
+that question gets its own answer.
+
+Expected: the branch adds passes and adds no failures. Compare **like with
+like** — a tier run says nothing about a full-suite one — and read the **skip**
+count as well as the failures. A skip that is new on the branch is a test that
+never ran.
+
+- [ ] **Step 2: Update the spec's progress record**
+
+In `docs/superpowers/specs/2026-09-21-zarr-io-design.md`:
+
+- §12.1: set PR 4's state to `Built, awaiting PR 3` and say that the work is
+ complete on a branch based on `zarr-io-pr-3`. The state becomes `In review`
+ and gains a number in Step 6, when the real pull request opens — a fork
+ preview has no number worth recording in a document that outlives it.
+- The header table's **Progress** and **Next action** rows.
+- §12.3: move **Q2** to §12.4 with today's date and the answer from F3 —
+ fixed-length unicode arrays read as themselves on both versions, verified
+ against zarr-python 3.4.1, so no special handling is needed.
+- §12.4: append two entries, dated, for the decisions this pull request took
+ that change what the spec says:
+
+ ```markdown
+ **2026-10-13 — during PR 4**
+
+ - `CFDataset.format` lands in PR 4, not PR 5. `ZarrDataset` cannot answer
+ `ZARR_V2` versus `ZARR_V3` without it, and PR 4 merges first.
+ `supports_dtype` is a saver question and stays in PR 5. §5's PR 4 and PR 5
+ entries are amended to match.
+ - The shared `CFDataset` contract is split into a read half and a write
+ half. PR 4 implements the Zarr read path only, so the write half is
+ unrunnable against it; one class with half of it skipped would hide an
+ incomplete implementation behind a skip, which is the opposite of what §6
+ asks a shared body to do. PR 6 adds the write half for Zarr.
+ ```
+
+- §5, PR 4 and PR 5 entries: move `CFDataset.format` from PR 5's sentence to
+ PR 4's list of generic-code changes.
+
+- [ ] **Step 3: Record what the implementation taught you**
+
+Add a section to this plan, below §2, titled **"What changed during
+implementation"**, listing every step whose code differed from what is written
+here and why. The plan ships with the pull request, so a reader comparing the
+two must not have to guess which one is true.
+
+- [ ] **Step 4: Open the preview pull request, on the fork**
+
+The base branch of a pull request has to live in the repository the pull
+request is raised against, and `zarr-io-pr-3` lives on the fork. Raising this
+against `SciTools/iris:brownfield` now would therefore show PR 3's ten commits
+as part of PR 4. Instead the preview is raised fork-internally, where
+`zarr-io-pr-3` *is* available as a base and the diff is exactly this pull
+request's work:
+
+```bash
+git push -u origin zarr-io-pr-4
+gh pr create --repo trexfeathers/iris --base zarr-io-pr-3 --head zarr-io-pr-4 \
+ --draft --title "Zarr I/O PR 4: Zarr loading" --body-file /tmp/pr4-body.md
+```
+
+No labels: `Agentic` and `Type: Feature Branch` are SciTools labels and may not
+exist on the fork. They go on in Step 6.
+
+Know what this preview is and is not. It **is** a rehearsal — the real diff,
+the real body, somewhere a reviewer can be sent by link, and a place to
+self-review before anyone else reads it. It is **not** the pull request: it
+does not appear in SciTools' list, a `Part of #6979` in it does not reach the
+issue, its CI runs under the fork without the upstream's secrets and may not
+run at all, and any review comment left on it stays behind when Step 6 opens
+the real one. Do not solicit review on it that you would mind re-soliciting.
+
+Write the body now and keep the file: Step 6 reuses it unchanged, which is the
+point of writing it this early.
+
+The body opens `(posted by Claude Opus)` before anything else, per
+`AGENTS.md`, which **replaces** any trailing attribution — except the
+`🤖 Generated with Claude Code` line the harness asks for, which stays at the
+end. Never hard-wrap it. It must cover:
+
+- **What this does.** `iris.load("store.zarr")` and
+ `iris.load("s3://bucket/store.zarr")` return cubes, for version 2 and
+ version 3 stores.
+- **Part of #6979.** Not `Closes` — a closing keyword only fires on a merge
+ into the default branch, and the real pull request targets `brownfield`. The
+ line is written the same way in the preview, where it is inert: a reference
+ from a fork does not reach the parent repository's issue.
+- **The changelog fragment is deferred to the merge-back**, deliberately, per
+ spec §5. Say so, so the omission does not read as an oversight.
+- **Behaviour changes**, all three, each with what used to happen: a local
+ `.zarr` directory raised `IsADirectoryError`; an `s3://` URL raised
+ `ValueError("Iris cannot handle the URI scheme")`; and `CFVariable.spans`
+ now matches `(name, size)` pairs, which no netCDF file can observe and which
+ the test suite asserts on both backends.
+- **Public API added**: `iris.fileformats.zarr`, `iris.io.load_stores`,
+ `CFDataset.format`, `CFDataset.encoding`, `CFDatasetVariable.encoding`.
+- **The spec amendments** from Step 2, each in one line, so a reviewer who
+ knows the spec is not surprised.
+- **What is not here**: saving, real-world fixtures, benchmarks, xarray as a
+ test dependency, UGRID meshes from Zarr.
+- **How to review it**: the task order in §3 of this plan is the reading
+ order, and Tasks 8 and 12 are the two a netCDF-only reviewer should check.
+- **The dependency on #7316.** This branch is cut from `zarr-io-pr-3` and
+ calls four names that #7316 introduces. Say, in the preview, that it will be
+ rebased onto `brownfield` before the real pull request opens; in the real
+ one, say that it was.
+
+- [ ] **Step 5: Commit the spec and plan updates**
+
+Run §5.0, then:
+
+```bash
+git add docs/superpowers
+git commit -m "Record PR 4 in the spec, and what implementation changed
+
+Closes open question Q2: fixed-length unicode arrays read as themselves on
+both Zarr versions, so #6961's note describes the specification rather than
+zarr-python. Logs the two decisions that move work between pull requests:
+format lands here rather than with the saver, and the shared dataset
+contract splits into read and write halves."
+```
+
+- [ ] **Step 6: When #7316 merges — rebase, re-verify, raise the real pull
+ request**
+
+This step is deferred by definition. Everything above is done and pushed; this
+is what turns a rehearsal into a pull request, and none of it can happen until
+PR 3 is on `brownfield`.
+
+```bash
+git fetch upstream
+git log --oneline -1 upstream/brownfield # confirm #7316 is actually in
+git rebase --onto upstream/brownfield zarr-io-pr-3 zarr-io-pr-4
+```
+
+The three-argument form moves **only** this branch's own commits: everything
+after `zarr-io-pr-3` is replayed onto `brownfield`, and PR 3's commits are left
+behind because they are already there. A plain `git rebase upstream/brownfield`
+would try to replay them too.
+
+Expect conflicts in `docs/superpowers/specs/2026-09-21-zarr-io-design.md`
+alone, and only if PR 3 changed §12 on its way through review. The code should
+not conflict: PR 3 moved the CF loader, and this branch calls it rather than
+touching it.
+
+Then re-verify, because a rebase is a rewrite and nothing on the old commits
+was tested in this arrangement:
+
+```bash
+# The two tasks that read PR 3's code most closely.
+pytest lib/iris/tests/unit/fileformats/zarr/test__decode.py \
+ lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py -q
+
+# Then the whole suite, now against brownfield - the comparison Step 1
+# deliberately could not make.
+git switch --detach upstream/brownfield
+pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/baseline-brownfield.txt
+git switch -
+pytest -n auto lib/iris/tests -q 2>&1 | tail -5 > /tmp/branch-final.txt
+diff /tmp/baseline-brownfield.txt /tmp/branch-final.txt
+```
+
+Update the spec's §12.1 state from `Built, awaiting PR 3` to `In review` with
+the number, once there is one. Then:
+
+```bash
+git push --force-with-lease origin zarr-io-pr-4
+gh pr create --repo SciTools/iris --base brownfield \
+ --title "Zarr I/O PR 4: Zarr loading" --body-file /tmp/pr4-body.md
+gh pr edit --add-label "Agentic" --add-label "Type: Feature Branch"
+gh pr close --repo trexfeathers/iris <preview number> \
+ --comment "(posted by Claude Opus)
+
+Superseded by SciTools/iris#<number>, now that #7316 has merged and this branch sits on brownfield."
+```
+
+The body is Step 4's file, with its #7316 paragraph changed from *will be
+rebased* to *was rebased*. Close the preview rather than leaving it: two pull
+requests showing the same work, one of them stale, is how a reviewer ends up
+commenting on the wrong one.
+
+---
+
+## 6. Verification
+
+What "done" is measured against, and how to measure it.
+
+### 6.1 The green check, per task
+
+Every task ends with §5.0: the test-data path confirmed, `pre-commit` clean on
+the paths touched, and the task's own test tier passing. No task's commit is
+made with `--no-verify`.
+
+### 6.2 The full suite, against the base
+
+Run once at the start of implementation and once at the end, on the same
+machine, with `OVERRIDE_TEST_DATA_REPOSITORY` set and verified:
+
+```bash
+pytest -n auto lib/iris/tests -q
+```
+
+The base is `zarr-io-pr-3` while the branch sits there, and `brownfield` after
+Task 15's Step 6 rebases it. Both comparisons get made, in that order: the
+first is the only one that isolates this pull request's effect, and the second
+is the one the eventual reviewer sees.
+
+Record `passed`, `skipped`, `failed` and `errors` for both. The branch must
+add passes, add no failures, and **add no skips**. A new skip is a test that
+never ran, and the usual causes — a missing optional dependency, a missing
+`iris-test-data` — are exactly the ones this pull request introduces a new
+instance of.
+
+`zarr` must be installed in the environment the final run uses, or every test
+this pull request adds is a skip and the comparison is meaningless.
+
+### 6.3 Without zarr
+
+The other half of spec §6's "Iris without zarr installed must show no new
+failures". In an environment with zarr removed:
+
+```bash
+python -c "import iris; print(iris.__version__)"
+pytest -n auto lib/iris/tests -q
+```
+
+Expected: Iris imports; the Zarr unit and integration packages skip as whole
+directories; nothing fails. If a clean environment is not available, state
+that in the pull request rather than claiming the check was made.
+
+### 6.4 The documentation build
+
+```bash
+pre-commit run check-docs-page-metadata --all-files
+cd docs && make html
+```
+
+Expected: no warnings. The standard build uses `-W --keep-going`, so any new
+warning fails CI.
+
+### 6.5 The claims this pull request makes, and what checks each
+
+| Claim | Checked by |
+|---|---|
+| A netCDF file loads exactly as it did | §6.2's diff, plus `test_spans.py` running the same suite on both backends (Task 8) |
+| `_ARRAY_DIMENSIONS` does not reach a cube | Task 3 Step 9, and Task 13 Step 4 end to end |
+| A version 3 array's zeros are not masked | Task 6 Step 5 (the rule), Task 10 Step 7 (a loaded cube) |
+| A version 2 array's fill value *is* a mask source | Task 6 Step 5, Task 10 Step 7 |
+| A CF `_FillValue` beats the storage one | Task 6 Step 5, Task 10 Step 7 |
+| A nested attribute reaches the cube, and saving it as netCDF fails loudly | Task 10 Step 7, documented in Task 14 |
+| Both `_FillValue` encodings read | Task 6 Steps 1–2, Task 13 Step 2 |
+| A sharded array is read by the inner chunk | Task 7 Step 9, on bytes fetched through an instrumented store |
+| The chunking tiles the store grid | Task 7 Steps 1–4, Task 13 Step 3 |
+| The dask cache key goes stale when the array does | Task 7 Step 8 |
+| An anonymous dimension matches nothing | Task 8 Steps 3–4 |
+| A group with no data variables says so | Task 10 Step 2 |
+| A `.zarr`-suffixed directory that is not a store is not claimed | Task 11 Step 1 |
+| NCZarr still reaches NCZarr | Task 11 Steps 1 and 7 |
+| A glob matching files and stores works | Task 12 Step 2 |
+| Iris imports without zarr | Task 1 Steps 5–6, §6.3 |
+| The native reader agrees with a second writer | Task 13 Step 4, against NCZarr |
+
+### 6.6 What is deliberately not checked here
+
+- **Against xarray.** Spec §6 adds xarray as a test dependency in PR 6, with
+ the saver it validates. Nothing in PR 4 produces a store for another reader
+ to open.
+- **Against a real published store.** PR 7.
+- **Over a network.** Spec §6: "No test touches the network. The URL path is
+ exercised by checking that the right store is constructed, not by fetching."
+- **Performance.** The ASV benchmarks are PR 7.
+- **A masked cube round-tripping its mask.** Spec §6 lists it among the fill
+ value cases; it is a save test, and it catches a saver that sets the storage
+ `fill_value` without also writing the `_FillValue` attribute. PR 6.
+- **Saving a nested attribute to netCDF.** Task 10 pins that it fails, not
+ what the message says: fixing the message means changing the netCDF saver
+ ([#7329](https://github.com/SciTools/iris/issues/7329)), and PR 5 relocates
+ that code unchanged.
+
+---
+
+## 7. Definition of done
+
+- [ ] Every task's checkboxes are ticked, and every commit was made with
+ `pre-commit` clean and its tier of tests passing.
+- [ ] `pytest -n auto lib/iris/tests` adds passes, adds no failures and adds
+ no skips against `zarr-io-pr-3`, with `zarr` installed and
+ `OVERRIDE_TEST_DATA_REPOSITORY` set (§6.2) — and again against
+ `brownfield` after the rebase.
+- [ ] Iris imports and its suite runs with `zarr` absent (§6.3), or the pull
+ request says the check could not be made.
+- [ ] `cd docs && make html` is warning-free (§6.4).
+- [ ] `iris.load("store.zarr")` works for a version 2 and a version 3 store,
+ and `iris.fileformats.zarr.load_cubes(store, group=...)` reads a
+ subgroup.
+- [ ] `requirements/py3{12,13,14}.yml` carry `zarr >=3.0.8` and the lock files
+ are regenerated — or the pull request asks a maintainer to regenerate
+ them.
+- [ ] The spec's §12.1 records PR 4 as `Built, awaiting PR 3`; §12.3 and §12.4
+ are updated, and §5's PR 4 and PR 5 entries reflect where `format`
+ landed (Task 15 Step 2).
+- [ ] This plan carries a "What changed during implementation" section, or
+ states that nothing did.
+- [ ] The preview pull request is open on `trexfeathers/iris`, based on
+ `zarr-io-pr-3`, in draft, with the body Step 6 will reuse.
+
+Then, once #7316 has merged (Task 15, Step 6 — the rest of this list can be
+complete without it):
+
+- [ ] The branch is rebased onto `brownfield` with
+ `git rebase --onto upstream/brownfield zarr-io-pr-3`, and Task 6's and
+ Task 9's tests pass on the rebased commits.
+- [ ] The pull request is open against `SciTools/iris:brownfield`, labelled
+ `Agentic` and `Type: Feature Branch`, says `Part of #6979`, opens
+ `(posted by Claude Opus)`, and states that its changelog fragment is
+ deferred to the merge-back.
+- [ ] The preview pull request is closed, pointing at the real one.
+- [ ] The spec's §12.1 records PR 4 as `In review`, with its number.
+
diff --git a/docs/superpowers/plans/2026-10-14-zarr-io-pr5.md b/docs/superpowers/plans/2026-10-14-zarr-io-pr5.md
new file mode 100644
index 0000000000..410346d182
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-14-zarr-io-pr5.md
@@ -0,0 +1,2932 @@
+# Zarr I/O PR 5 — Relocate the CF saver: Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+| | |
+|---|---|
+| **Spec** | [`docs/superpowers/specs/2026-09-21-zarr-io-design.md`](../specs/2026-09-21-zarr-io-design.md) §4.1, §4.2, §4.5, §4.8, §5 (PR 5) |
+| **Implements** | Pull request 5 of 8. `Part of #6977` — never a closing keyword. |
+| **Branch** | `zarr-io-pr-5`, branched from and targeting `brownfield` |
+| **Baseline** | `478c168be`, the tip of `brownfield` with PR 3 ([#7316](https://github.com/SciTools/iris/pull/7316)) and PR 2.5 ([#7333](https://github.com/SciTools/iris/pull/7333)) merged; Iris `3.17.0.dev16`; dask `2026.8.0`; numpy `2.5.2` |
+| **Delivers** | `iris.fileformats.cf.saver`, holding every format-agnostic part of today's `iris.fileformats.netcdf.saver`; `iris.fileformats.cf._dask_locks`; `CFDataset.format` and `CFDataset.supports_dtype`, which end the generic saver's last two reads of a netCDF format string |
+| **Behaviour change** | None intended. Every public name still resolves, every existing test passes unchanged. The only visible difference is a deprecation warning when one of four relocated names is reached through `iris.fileformats.netcdf`. |
+| **Public API change** | `CFNameCoordMap`, `CF_CONVENTIONS_VERSION`, `MESH_ELEMENTS` and `SPATIO_TEMPORAL_AXES` gain new homes in `iris.fileformats.cf.saver`. The old names still resolve, to the *same objects*, warning on use. Nothing is removed. `iris.fileformats.netcdf.Saver`, `.save` and `.save_mesh` do not move. |
+
+**Goal:** Move the format-agnostic half of the netCDF saver into `iris.fileformats.cf.saver`, leaving `iris.fileformats.netcdf.saver` holding netCDF's entry points, its keyword vocabulary and four netCDF-only hooks, so that PR 6 can add Zarr saving by subclassing one class.
+
+**Architecture:** Today `netcdf/saver.py` is one 3178-line module that mixes the CF data model (dimension naming, coordinate and mesh variables, grid mappings, cell methods, attribute compliance) with netCDF4's own vocabulary (`zlib`, `endian`, `|S1` char dimensions, NCZarr URLs, the ncdata bridge). PR 2 already pushed the storage calls behind `CFDataset`, so most of the class no longer names netCDF at all. This plan extracts the five places that still do, as named hooks on the class, then `git mv`s the file to `cf/saver.py` and renames the class to `CFSaver`. `iris.fileformats.netcdf.saver.Saver` becomes a subclass that supplies the netCDF hooks and keeps today's signature exactly.
+
+**Tech Stack:** Python 3.12–3.14, NumPy, Dask, netCDF4 (via `iris.fileformats.netcdf._thread_safe_nc`), pytest + pytest-mock, pre-commit (ruff, numpydoc, mypy, sort-all, and the repository's local hooks), Sphinx.
+
+---
+
+## Global Constraints
+
+Every task's requirements implicitly include this section.
+
+| Constraint | Value, verbatim from its source |
+|---|---|
+| Target branch | `brownfield`, not `main` (spec §5) |
+| PR labels | `Agentic` and `Type: Feature Branch` (spec §5) |
+| Issue reference | `Part of #6977`. **Never** a closing keyword — the issue closes at merge-back (spec §5) |
+| Changelog | No `changelog/` fragment in this PR. Fragments are written once at merge-back; the PR body must say so (spec §5) |
+| GitHub posts | Every PR description, comment and review reply opens with `(posted by Claude Opus)`, before anything else. This **replaces** any trailing harness attribution (root `AGENTS.md`) |
+| Commit messages | Out of scope for the above; they keep `Co-Authored-By:` trailers |
+| Deprecation aliases | "The alias is the **same object**, not a copy." "Warnings are emitted on **use**, not on import." Mark `.. deprecated:: 3.17` (spec §4.8) |
+| Deprecation mechanism | `iris._deprecation.warn_deprecated()`, NEP29 schedule (root `AGENTS.md`) |
+| Where `IrisDeprecation` lives | `iris._deprecation`. "Re-exported from neither `iris.exceptions` nor `iris.warnings`. A `-W` filter naming either of those matches nothing and fails open" (spec §5, PR 2.5) |
+| One exit where something must still happen | "Give a function one exit where it must still warn, record or close something on the way out. A second exit added later silently skips that work" (`lib/iris/AGENTS.md`) |
+| Names that move, with warn-on-use aliases | `netcdf.CFNameCoordMap`, `netcdf.CF_CONVENTIONS_VERSION`, `netcdf.MESH_ELEMENTS`, `netcdf.SPATIO_TEMPORAL_AXES` → `cf.saver.*` (spec §4.8, verbatim) |
+| Names that must NOT move | `netcdf.save`, `netcdf.Saver`, `netcdf.load_cubes`, `netcdf.save_mesh`, `netcdf.NetCDFDataProxy`, `netcdf.DEBUG` (spec §4.8, verbatim) |
+| No behaviour change | "The largest diff of the eight and the one with no behaviour change at all, which is precisely why it is on its own" (spec §5, PR 5) |
+| Capability, not format | "Where the generic layer needs a capability, it asks for the capability, not the format. `format` is for messages, diagnostics and the user; it is not a branch key" (spec §4.2) |
+| No type tests above the interface | "No code above `cf/dataset.py` tests the implementation type. No `isinstance`, no checking a class name, no `hasattr` probe standing in for one" (spec §4.2) |
+| Deferred, not fixed | The two-dimensional attribute check ([#7329](https://github.com/SciTools/iris/issues/7329)) and the `compute=False` return type ([#7291](https://github.com/SciTools/iris/issues/7291)) are both known netCDF defects. "PR 5 relocates that code without changing it" (spec §4.5) |
+| netCDF4 imports | No direct `import netCDF4`; always `iris.fileformats.netcdf._thread_safe_nc` (root `AGENTS.md`, enforced by the `check-netcdf4-imports` hook) |
+| Copyright header | Every new Python file opens with the four-line BSD header (root `AGENTS.md`) |
+| Line length | 88 characters (Ruff default) |
+| Docstrings | NumPy style, strictly validated by the `numpydoc-validation` hook |
+| Public module docstrings | Must open with a `z_reference` item within the first 25 lines (`lib/iris/AGENTS.md`, enforced by `check-docs-page-metadata`) |
+| New module size | Aim under ~1000 lines (`lib/iris/AGENTS.md`). `cf/saver.py` will not reach it; F12 records why that is accepted here |
+| Published prose | "must never cite an artefact of how the work was done — a review finding, a task or plan number, a phase" (`docs/AGENTS.md`). Cite the issue, the pull request or the design-document section instead |
+| Warnings | Every `UserWarning` subclass inherits from `IrisUserWarning`; the `check-categorised-warnings` hook enforces categorisation |
+| Before finishing | Run the repository's pre-commit hooks and fix everything they report (root `AGENTS.md`) |
+
+### Running anything
+
+The environment is the conda environment `iris-dev`. Either activate it, or prefix
+commands with `conda run -n iris-dev`. Do **not** install or remove packages
+without asking the user first.
+
+```bash
+conda activate iris-dev
+export NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX"
+export OVERRIDE_TEST_DATA_REPOSITORY=/data/home/martin.yeo/repos/iris-test-data/test_data
+```
+
+`pytest-xdist` is installed but is **not** in `addopts`; pass `-n auto` yourself for
+the full-suite runs this plan asks for.
+
+---
+
+## Review Focus
+
+Five things the spec implies, that no existing test exercises, and that would bite
+a real user. Each is pinned by a test in the task named.
+
+1. **`supports_dtype` changes which dtypes get cast down.** Today's guard fires only
+ for int64 and unsigned integers, and only when `file_format` is one of three
+ strings. Two ways to get this wrong, and both are silent. Answering `False` more
+ widely sends some other dtype into the `astype(np.int32)` branch below, which
+ truncates a float. Answering `False` *correctly* for `NETCDF3_64BIT` turns
+ today's crash on that format into a successful save, which is a behaviour change
+ this pull request may not make (F3). → **Task 1, Steps 2 and 6**: `TestInt64`
+ saves an int64 cube to each of the four netCDF formats and records what each
+ does today, the failure included, and `TestSupportsDtype` pins the matrix the
+ dataset answers, including the one answer that is deliberately wrong.
+2. **A method quietly loses its name in the move.** `Saver` is driven from outside
+ Iris: the emulated-variable branch exists so that ncdata can hand the saver its
+ own dataset-like objects (issue #4994, "Xarray bridge"), and the class is public
+ API that third parties subclass. Every method this plan moves to `CFSaver` keeps
+ its name, so an override keeps overriding. Nothing today would notice if one
+ were renamed or dropped along the way. → **Task 4, Steps 1–2**:
+ `TestSaverMethods` checks all twenty-nine method names, recorded from the class
+ before the move, and `test_write_keeps_its_signature` pins the twelve keywords
+ and three of their defaults.
+3. **Lazy data silently vanishes from an NCZarr save.** NCZarr cannot be reopened
+ for deferred writes, so `Saver.__exit__` flushes `_nczarr_writes` before closing.
+ A missing chunk in a Zarr store is not an error on read: it comes back as the
+ fill value with no warning (spec §4.5). If the flush is left behind in `CFSaver`
+ while `_is_nczarr` moves to the subclass, or the reverse, the save succeeds and
+ the data is gone. → **Task 3, Step 1**:
+ `test_lazy_data_reaches_an_nczarr_target` saves a lazy cube to an NCZarr URL and
+ reads the values back.
+4. **`cf_patch` receives the wrong objects.** `iris.site_configuration["cf_patch"]`
+ is a hook documented since Iris 1.3 as receiving netCDF4 objects. Handing it a
+ `CFDataset` would not raise: `CFVariable` defines no `__setattr__`, so
+ `variable.name = value` inside a user's hook would succeed and write nothing. →
+ **Task 3, Step 1**: `test_cf_patch_receives_raw_netcdf4_objects` asserts what
+ both arguments the hook is called with are, and are not.
+5. **Importing the saver from the loader at module scope closes a loop.**
+ `cf/loader.py` reaches into the saver twice, deferred, with a comment saying
+ they stay that way until this pull request. Making them module-scope puts
+ `cf.saver` inside a chain that already runs `cf/__init__` → `_reader` →
+ `netcdf/_dataset` → the netCDF package → `netcdf/saver` → back to `cf.saver`.
+ Whether that resolves depends on which module the interpreter is asked for
+ first, and a test suite only ever exercises one order. The failure is an
+ `ImportError` naming a name that is obviously there. → **Task 5, Steps 1–2 and
+ 5**: `test_importing_it_first_works` imports each of five modules first, each in
+ its own subprocess.
+
+---
+
+## 1. Scope boundary
+
+**In scope** (spec §5, PR 5):
+
+- `git mv netcdf/saver.py cf/saver.py`, leaving a `netcdf/saver.py` that holds the
+ netCDF specifics and the deprecating aliases.
+- `git mv netcdf/_dask_locks.py cf/_dask_locks.py`, and a docstring that stops
+ describing it as netCDF's.
+- `CFDataset.format` and `CFDataset.supports_dtype`, used at the one place where
+ relocated generic code would otherwise test netCDF format strings.
+- Every in-repository reference repointed: library, tests, benchmarks, user manual,
+ API reference pages.
+- Still netCDF-only behaviour; the whole existing suite passes unchanged.
+
+**Out of scope** — do not do these, even if they look adjacent:
+
+- Zarr saving. No `zarr/saver.py`, no `ZarrDataset` write path, no Zarr keyword
+ vocabulary. That is PR 6.
+- Zarr loading. `iris/fileformats/zarr/` does not exist on this branch and this
+ plan does not create it. That is PR 4, developed in parallel (see §2).
+- The two-dimensional attribute check (#7329). The netCDF saver owes it, and
+ adding it here would make this a behaviour change (spec §4.5).
+- The `compute=False` return type (#7291). `save()` keeps returning a tuple.
+- Splitting `save()` into a generic driver and a netCDF one. `netcdf.save` does not
+ move (spec §4.8), and what a generic save function would need is PR 6's question.
+- A changelog fragment. Deferred to merge-back (spec §5).
+- Historical `docs/src/whatsnew/*.rst` entries. 3.4, 3.8 and 3.14 name
+ `iris.fileformats.netcdf.saver` because that is where the code was when they were
+ written. They are a record, not documentation; leave them.
+- Any edit to an `AGENTS.md`. Propose; never self-apply (root `AGENTS.md`).
+
+---
+
+## 2. Git strategy: PR 4 and PR 5 are siblings
+
+Both branch from `brownfield` and neither needs the other's code. Spec §12.1 says
+the refactor chain is sequential, 1 → 2 → 3 → 5, and that the Zarr features depend
+on the relocation before them: PR 4 needs PR 3, PR 6 needs PR 5. Nothing says PR 5
+needs PR 4.
+
+```bash
+cd /data/home/martin.yeo/repos/iris
+git fetch upstream
+git checkout -b zarr-io-pr-5 upstream/brownfield
+```
+
+The two branches overlap in exactly three places, and all three are additions to
+the same files rather than edits to the same lines:
+
+| File | PR 4 adds | PR 5 adds |
+|---|---|---|
+| `cf/dataset.py` | `format` and `encoding`, on both classes | `format` on `CFDataset`, and `supports_dtype` |
+| `netcdf/_dataset.py` | `format` and `encoding`, on both classes | `format` on `NetCDFDataset`, and `supports_dtype` |
+| `tests/unit/fileformats/cf/dataset/` | Zarr's half of the shared contract body | the `format` and `supports_dtype` contract tests |
+
+Task 1 writes `format` with text **byte-identical** to PR 4's, so whichever lands
+second resolves the conflict by taking either side. Copy it from
+`git show zarr-io-pr-4:lib/iris/fileformats/cf/dataset.py`; do not paraphrase it.
+`supports_dtype` is PR 5's alone, and `encoding` is PR 4's alone.
+
+If PR 4 merges first, rebase before opening this one and delete the duplicate
+`format` in favour of whichever text is already there. If PR 5 merges first, PR 4
+does the same in the other direction, and additionally implements `supports_dtype`
+on `ZarrDataset` as `return True` for every dtype Zarr has a data type for
+(spec §4.2).
+
+Do **not** branch PR 5 from `zarr-io-pr-4`. PR 4 is a 12,500-line change still in
+progress, and a PR 5 stacked on it could not be reviewed or merged until it lands.
+
+One test spec §6 asks for is deliberately left to PR 4: the standing check that
+greps the `cf` package for an `isinstance` against an implementation class. It
+belongs with the second implementation, it would be a fourth place these two
+branches collide, and Task 4 Step 10 greps this pull request's own 2400 relocated
+lines for the same thing by hand.
+
+---
+
+## 3. Findings that refine the spec
+
+Read these before Task 1. They are what was learned from the code the spec
+describes; where one contradicts the spec, this plan follows the finding and
+Task 6 records it in the spec's decision log.
+
+**F1 — The saver already speaks `CFDataset` almost everywhere.** PR 2 did more of
+this work than the spec's "3000-line diff" risk row implies. Of the forty-odd
+places `Saver` touches storage, exactly five still name something netCDF-specific:
+the format string in `_ensure_valid_dtype`, the `|S1` char-dimension block in
+`_create_generic_cf_array_var`, the ncdata emulated-variable branch in
+`_lazy_stream_data`, the `cf_patch` call at the end of `write`, and `__init__`
+itself. The rest of the class moves verbatim.
+
+**F2 — The condition is compound, and only half of it is the dataset's business.**
+Today's guard reads "int64 or unsigned, *and* one of three classic formats".
+`supports_dtype` answers the second half. The first half stays in the generic
+saver, because it is not a question about the store: it says which dtypes are
+worth narrowing to int32 rather than giving up on. Keeping the split means
+`supports_dtype` can answer honestly about dtypes the narrowing branch never sees
+— strings on netCDF3, say — without changing what the saver does with them.
+
+```python
+is_narrowable_integer = np.issubdtype(values.dtype, np.int64) or np.issubdtype(
+ values.dtype, np.unsignedinteger
+)
+if is_narrowable_integer and not self._dataset.supports_dtype(values.dtype):
+```
+
+**F3 — `NETCDF3_64BIT` never reaches the guard, and this pull request must keep it
+that way.** netCDF4 reports `file_format` as `"NETCDF3_64BIT_OFFSET"` for a file
+created with `netcdf_format="NETCDF3_64BIT"`, and that string is not in the tuple
+the saver tests against **[verified: `NetCDFDataset(p, mode="w",
+netcdf_format="NETCDF3_64BIT").dataset.file_format`]**. So an int64 cube saved to
+that format is never cast down, and the save fails inside the C library:
+
+```
+RuntimeError: NetCDF: Not a valid data type or _FillValue type mismatch:
+(variable 'small', group '/')
+```
+
+**[verified: `iris.save(Cube(np.arange(4, dtype=np.int64)), p,
+netcdf_format="NETCDF3_64BIT")`]**. The other three formats behave as the code
+reads. Writing `supports_dtype` to answer honestly would turn that crash into a
+successful save, which is a behaviour change, and spec §5 says this pull request
+makes none. So the netCDF implementation keeps the same three-string list, with a
+comment naming the issue, and Task 1 pins the crash in a test so that the next
+person to see it knows it was left alone on purpose. Task 6 raises the issue.
+
+**F4 — The generic saver needs `format` only for one error message.** After
+`supports_dtype` takes the branch, `self._dataset.dataset.file_format` survives in
+exactly one place: the `ValueError` naming the format that refused the cast. Spec
+§4.2 says the user still reads `NETCDF3_CLASSIC`, so `CFDataset.format` supplies
+it. `CFDatasetVariable.format` is not needed by this plan; PR 4 adds it for the
+read side.
+
+**F5 — `_dask_locks` is already generic, and moving it is safe for the import
+graph.** It imports `threading` and four `dask` modules, and nothing from Iris
+**[verified: `grep -n "^import\|^from" lib/iris/fileformats/netcdf/_dask_locks.py`]**.
+`netcdf/_dataset.py` imports it as `from . import _dask_locks` and will import it as
+`from iris.fileformats.cf import _dask_locks`. That is the same shape as the
+`from iris.fileformats.cf.dataset import CFDataset` already on line 33 of that file,
+which works despite `cf/__init__.py` importing `_reader`, which imports
+`netcdf/_dataset.py`: the submodule has no import back into the package, so Python
+finishes it while the parent package is still partly initialised.
+
+**F6 — Four module constants move and four names need aliases, but they are not the
+same four.** Spec §4.8 lists `CFNameCoordMap`, `CF_CONVENTIONS_VERSION`,
+`MESH_ELEMENTS` and `SPATIO_TEMPORAL_AXES` as the aliased names. All four are
+defined in `netcdf/saver.py` today and re-exported from `netcdf/__init__.py`, which
+is where users reach them and so where the alias has to be. `_CF_ATTRS`,
+`_CF_DATA_ATTRS`, `_CF_GLOBAL_ATTRS`, `_UKMO_DATA_ATTRS`, `_FACTORY_DEFNS` and
+`SaverFillValueWarning` also move, and are private, so they get no alias. One
+exception: `iris/cube.py` imports `_CF_GLOBAL_ATTRS` from the saver, so its import
+is repointed in Task 5 rather than aliased.
+
+**F7 — Two deferred imports in `cf/loader.py` exist only because of where the saver
+lives.** They are commented
+`# TODO: make this a plain module-scope import once the saver is relocated`. After
+the move, `_CF_ATTRS` comes from `cf/saver.py` and the loader can import it at
+module scope. `Saver._DATALESS_ATTRNAME` becomes `CFSaver._DATALESS_ATTRNAME`,
+which means the dataless marker must live on the generic class, not the netCDF
+subclass. It already describes a CF-level convention rather than a netCDF one.
+
+**F8 — `netcdf/saver.py` imports the netCDF package logger, and `cf/saver.py`
+cannot.** `from . import logger` at the top of today's file resolves to
+`iris.fileformats.netcdf.logger`, re-exported for the module's users and otherwise
+unused. `cf/saver.py` neither needs it nor can have it: `iris.fileformats.cf`
+defines no logger. The import stays with `netcdf/saver.py`, where the bare `logger`
+statement below it keeps the linter quiet.
+
+**F9 — `write()` has to exist on both classes, with different signatures.** The
+generic body is unchanged apart from the keywords it reads. `CFSaver.write` takes
+`cube, local_keys=None, unlimited_dimensions=None, **encoding`, bundles nothing,
+and passes `encoding` through to `create_variable`. `Saver.write` keeps today's
+twelve explicit keywords, bundles the four compression ones exactly as today, and
+calls `super().write(...)`. Keeping the explicit keywords is not optional: they are
+documented API, they carry defaults users rely on (`complevel=4`, `shuffle=True`),
+and `save()` passes them positionally in places.
+
+**F10 — The saver's unit tests stay where they are, and that is not an oversight.**
+Every test in `tests/unit/fileformats/netcdf/saver/` drives the generic machinery
+by constructing a netCDF `Saver`, because a netCDF file is the only store there
+is on this branch. Moving them under `cf/` would leave tests in the CF package
+that open netCDF files, which is worse than leaving them. `cf/saver.py` is
+therefore covered the way it was covered before it moved: through the only
+subclass that exists. PR 6 adds a second one, and that is the point at which a
+shared body is worth factoring out, the way `cf/dataset/contract.py` already is.
+Say this in the pull request body, because a reviewer will reasonably ask where
+the new module's tests are.
+
+**F11 — `netcdf/saver.py` has an import nothing in it uses.** Line 64 is
+`import iris.fileformats.cf`, and no other line in the file names it
+**[verified: `grep -n "fileformats\.cf"`]**. Ruff does not flag it, because the
+name it binds is `iris`, which the file uses constantly. Leave it in
+`netcdf/saver.py` when the file splits. It may be load-bearing by preloading the
+package for someone else, proving it is not would take more than this pull
+request's budget, and deleting it is a change with no upside here.
+
+**F12 — `cf/saver.py` will be about 2400 lines, well over the ~1000-line aim.** So is
+the module it comes from, and splitting it is not this pull request's job. The aim
+in `lib/iris/AGENTS.md` governs new modules; this one is a relocation of an existing
+module whose size is the status quo. Splitting the CF saver into coherent pieces is
+worth doing and deserves its own pull request, with its own review, after the Zarr
+work has shown where the seams really are. Note it in the pull request body so a
+reviewer does not have to raise it.
+
+---
+
+## 4. File Structure
+
+### Moved, with `git mv`, so the diff reads as a move
+
+| From | To | What lands there |
+|---|---|---|
+| `lib/iris/fileformats/netcdf/saver.py` | `lib/iris/fileformats/cf/saver.py` | `CFSaver`, `CFNameCoordMap`, `SaverFillValueWarning`, and every module constant except the logger import |
+| `lib/iris/fileformats/netcdf/_dask_locks.py` | `lib/iris/fileformats/cf/_dask_locks.py` | unchanged code, rewritten docstring |
+| `lib/iris/tests/integration/netcdf/test__dask_locks.py` | `lib/iris/tests/integration/cf/test__dask_locks.py` | unchanged, with its import repointed |
+
+The saver's unit tests stay in `lib/iris/tests/unit/fileformats/netcdf/saver/`,
+all of them. F10 says why.
+
+### Created
+
+| Path | Responsibility |
+|---|---|
+| `lib/iris/tests/integration/cf/__init__.py` | package marker for the moved lock tests |
+| `lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py` | that the four relocated names still resolve, warn on use and are the same objects, and that every `Saver` method kept its name |
+| `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__hooks.py` | the four netCDF hooks, each called with what it expects |
+| `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py` | what each netCDF format does with a dtype it cannot hold |
+| `lib/iris/tests/unit/fileformats/cf/test__imports.py` | that each module of both packages can be the first one imported |
+
+### Modified
+
+| Path | Change |
+|---|---|
+| `lib/iris/fileformats/cf/dataset.py` | `CFDataset.format` and `CFDataset.supports_dtype`, both abstract |
+| `lib/iris/fileformats/netcdf/_dataset.py` | the netCDF implementations of both; `_dask_locks` imported from its new home |
+| `lib/iris/fileformats/netcdf/saver.py` | shrinks to `Saver(CFSaver)`, `save`, `save_mesh`, the logger import and four aliases |
+| `lib/iris/fileformats/netcdf/__init__.py` | the four relocated names come from `cf.saver` through a module `__getattr__` |
+| `lib/iris/fileformats/cf/loader.py` | two deferred imports become module-scope imports of `cf.saver` |
+| `lib/iris/cube.py` | `_CF_GLOBAL_ATTRS` comes from `cf.saver` |
+| `lib/iris/tests/unit/cube/test_CubeAttrsDict.py`, `lib/iris/tests/integration/test_netcdf__loadsaveattrs.py` | the private attribute lists come from `cf.saver` |
+| `lib/iris/tests/unit/fileformats/netcdf/saver/test_save.py`, `lib/iris/tests/integration/netcdf/test_attributes.py` | `CF_CONVENTIONS_VERSION` comes from `cf.saver`, so the suite does not trip its own deprecation |
+
+Three files look as though they should be on that list and are not.
+`lib/iris/fileformats/cf/__init__.py` re-exports the CF variable classes and does
+not mention `loader`; `saver` is reached as a submodule in the same way.
+`lib/iris/mesh/__init__.py` imports `save_mesh` and `lib/iris/io/__init__.py`
+imports `save`, both from `netcdf.saver`, and both names stay there. Task 5
+Step 7's grep confirms all three rather than assuming them.
+
+The API reference pages under `docs/src/generated/` are built by sphinx-apidoc and
+are gitignored **[verified: `git check-ignore -v`]**. `cf/saver.py` gets a page
+because its module docstring opens with a `z_reference` item, which is what the
+`check-docs-page-metadata` hook requires of a public module. Nothing to create by
+hand.
+
+---
+
+## 5. The interface, as these tasks leave it
+
+```python
+# iris/fileformats/cf/dataset.py
+class CFDataset(ABC):
+ @property
+ @abstractmethod
+ def format(self) -> str: ... # "NETCDF4", "NETCDF3_CLASSIC", ...
+
+ @abstractmethod
+ def supports_dtype(self, dtype: np.dtype) -> bool: ...
+
+# iris/fileformats/netcdf/_dataset.py
+class NetCDFDataset(CFDataset):
+ @property
+ def format(self) -> str: # self._dataset.file_format
+ def supports_dtype(self, dtype) -> bool: # False for i8/unsigned on classic
+
+# iris/fileformats/cf/saver.py
+CF_CONVENTIONS_VERSION: str
+MESH_ELEMENTS: tuple[str, str, str]
+SPATIO_TEMPORAL_AXES: list[str]
+class CFNameCoordMap: ...
+class SaverFillValueWarning(iris.warnings.IrisSaverFillValueWarning): ...
+
+class CFSaver:
+ _DATALESS_ATTRNAME: str
+ _DATALESS_DTYPE: np.dtype
+ _DATALESS_FILLVALUE: int
+
+ def __init__(self, dataset: CFDataset, compute: bool = True,
+ *, owns_dataset: bool = True): ...
+ def write(self, cube, local_keys=None, unlimited_dimensions=None,
+ packing=None, fill_value=None, compression_kwargs=None,
+ **encoding): ...
+ def complete(self) -> None: ...
+ def delayed_completion(self): ...
+
+ # The four hooks a format supplies beyond the dataset. Two of them have a
+ # generic answer and two raise, because storing a string and patching a
+ # netCDF4 object do not.
+ def _create_string_variable(self, cube_or_mesh, cf_name, element,
+ element_dims, data): ... # raises
+ def _backend_write(self, data, cf_var) -> bool: ... # False
+ def _flush_before_close(self) -> None: ... # no-op
+ def _patch_dataset(self, cf_patch, profile, cf_var_cube) -> None: ... # raises
+
+# iris/fileformats/netcdf/saver.py
+class Saver(CFSaver):
+ def __init__(self, filename, netcdf_format, compute=True): ...
+ def write(self, cube, local_keys=None, unlimited_dimensions=None,
+ zlib=False, complevel=4, shuffle=True, fletcher32=False,
+ contiguous=False, chunksizes=None, endian="native",
+ least_significant_digit=None, packing=None, fill_value=None): ...
+ # plus overrides of all four hooks
+
+def save(cube, filename, netcdf_format="NETCDF4", ...): ...
+def save_mesh(mesh, filename, netcdf_format="NETCDF4"): ...
+```
+
+---
+
+## Task 1: The dataset answers "can you hold this dtype?"
+
+Spec §4.2. The saver reads `self._dataset.dataset.file_format` in two places, both
+inside `_ensure_valid_dtype`, and both would become generic code naming netCDF
+format strings once the file moves. This task replaces them with a capability
+question and a name for messages, and leaves the saver's behaviour untouched on
+all four netCDF formats.
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/dataset.py`
+- Modify: `lib/iris/fileformats/netcdf/_dataset.py`
+- Modify: `lib/iris/fileformats/netcdf/saver.py:1473-1504`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py` (new)
+- Test: `lib/iris/tests/unit/fileformats/cf/dataset/contract.py`
+- Test: `lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`
+
+**Interfaces:**
+- Consumes: `CFDataset` as PR 2 left it; `NetCDFDataset._dataset`, the wrapped
+ `netCDF4.Dataset`.
+- Produces: `CFDataset.format -> str` and `CFDataset.supports_dtype(dtype) -> bool`,
+ both abstract; `NetCDFDataset.format` and `NetCDFDataset.supports_dtype`. Task 4
+ moves `_ensure_valid_dtype` onto `CFSaver` and it reads both there.
+
+- [ ] **Step 1: Raise the issue for the `NETCDF3_64BIT` crash**
+
+F3 found it and this pull request deliberately keeps it. The test written in
+Step 2 cites the number, so raise it first.
+
+Put the body in a scratch file first. It contains a code block, so passing it
+inline fights the shell for no reason.
+
+```bash
+gh issue create --repo SciTools/iris \
+ --title "netCDF saver fails on int64 data when saving NETCDF3_64BIT" \
+ --body-file /tmp/iris-64bit-issue.md
+```
+
+The body, one paragraph per line, opening with the model marker the root
+`AGENTS.md` requires:
+
+> (posted by Claude Opus)
+>
+> Saving a cube with int64 data to `netcdf_format="NETCDF3_64BIT"` fails with `RuntimeError: NetCDF: Not a valid data type or _FillValue type mismatch`. The same cube saves successfully to `NETCDF3_CLASSIC` and to `NETCDF4_CLASSIC`, where the saver narrows the data to int32 first.
+>
+> The saver decides whether to narrow by testing the open file against `("NETCDF3_CLASSIC", "NETCDF3_64BIT", "NETCDF4_CLASSIC")`. netCDF4 reports a 64-bit-offset file as `NETCDF3_64BIT_OFFSET`, so that test never matches and the narrowing never happens.
+>
+> ```python
+> import numpy as np
+> import iris
+> from iris.cube import Cube
+>
+> cube = Cube(np.arange(4, dtype=np.int64), var_name="payload")
+> iris.save(cube, "out.nc", netcdf_format="NETCDF3_64BIT")
+> ```
+>
+> Expected: the data is narrowed to int32 and the file is written, as it is for the other netCDF3 format. A value too large for int32 should raise `ValueError` naming the format, which is what the other formats do.
+>
+> Found while relocating the CF saver for Zarr support, SciTools/iris#6977. The relocation preserves this behaviour rather than fixing it, so that it stays a pure move.
+
+Note the number it prints. It is written `#NNNN` in Steps 2, 6 and 9.
+
+- [ ] **Step 2: Write the characterisation test for today's behaviour**
+
+This one passes before the change and after it. That is its job: it says what the
+four formats do today, so the refactor below cannot quietly alter any of them.
+
+Create `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""What the netCDF saver does with a dtype the target format cannot hold.
+
+Every test here passes both before and after the saver stops reading netCDF
+format strings and starts asking the dataset. Relocating the CF saver for Zarr
+support (SciTools/iris#6977) must not change any answer in this file.
+"""
+
+import numpy as np
+import pytest
+
+import iris
+from iris.coords import AuxCoord
+from iris.cube import Cube
+from iris.fileformats.netcdf._dataset import NetCDFDataset
+
+#: Every format Saver accepts.
+ALL_FORMATS = ("NETCDF4", "NETCDF4_CLASSIC", "NETCDF3_CLASSIC", "NETCDF3_64BIT")
+
+
+def saved_dtype(cube, path, netcdf_format):
+ """Save the cube, and return the dtype its data variable has in the file."""
+ iris.save(cube, path, netcdf_format=netcdf_format)
+ with NetCDFDataset(path) as dataset:
+ return dataset.variables[cube.var_name].dtype
+
+
+@pytest.fixture
+def int64_cube():
+ """Return an int64 cube whose values all fit in an int32."""
+ cube = Cube(np.arange(4, dtype=np.int64), var_name="payload")
+ cube.add_aux_coord(AuxCoord(np.arange(4, dtype=np.int64), var_name="index"), 0)
+ return cube
+
+
+class TestInt64:
+ """int64 is the dtype the restricted formats cannot hold."""
+
+ def test_netcdf4_keeps_it(self, int64_cube, tmp_path):
+ assert saved_dtype(int64_cube, tmp_path / "out.nc", "NETCDF4") == np.int64
+
+ @pytest.mark.parametrize(
+ "netcdf_format", ["NETCDF4_CLASSIC", "NETCDF3_CLASSIC"]
+ )
+ def test_the_restricted_formats_narrow_it(
+ self, int64_cube, tmp_path, netcdf_format
+ ):
+ assert saved_dtype(int64_cube, tmp_path / "out.nc", netcdf_format) == np.int32
+
+ def test_64bit_offset_fails_in_the_library(self, int64_cube, tmp_path):
+ # Known defect, left alone on purpose: netCDF4 reports this file as
+ # NETCDF3_64BIT_OFFSET, which is not one of the three strings the
+ # narrowing tests against, so no narrowing happens and netCDF4 refuses
+ # the variable. SciTools/iris#NNNN.
+ with pytest.raises(RuntimeError, match="Not a valid data type"):
+ iris.save(int64_cube, tmp_path / "out.nc", netcdf_format="NETCDF3_64BIT")
+
+ def test_a_value_too_large_to_narrow_names_the_format(self, tmp_path):
+ cube = Cube(np.array([2**40], dtype=np.int64), var_name="payload")
+ with pytest.raises(ValueError, match="NETCDF3_CLASSIC"):
+ iris.save(cube, tmp_path / "out.nc", netcdf_format="NETCDF3_CLASSIC")
+
+
+class TestDtypesEveryFormatHolds:
+ """A dtype the format holds is written as it stands, never narrowed."""
+
+ @pytest.mark.parametrize("dtype", ["f4", "f8", "i2", "i4"])
+ @pytest.mark.parametrize("netcdf_format", ALL_FORMATS)
+ def test_not_narrowed(self, tmp_path, dtype, netcdf_format):
+ cube = Cube(np.arange(4, dtype=dtype), var_name="payload")
+ assert saved_dtype(cube, tmp_path / "out.nc", netcdf_format) == np.dtype(dtype)
+```
+
+Replace `#NNNN` with the number Step 1 printed.
+
+- [ ] **Step 3: Run it, and watch it pass**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py -v
+```
+
+Expected: 21 passed. If any fails, stop: either the environment differs from the
+one this plan was written against, or F3 is wrong. Do not change the test to
+match. Find out which.
+
+- [ ] **Step 4: Write the failing contract tests**
+
+In `lib/iris/tests/unit/fileformats/cf/dataset/contract.py`, in the
+`-- The dataset itself --` section, after `test_mode_is_a_read_mode`:
+
+```python
+ def test_format_is_a_non_empty_string(self, readable):
+ assert isinstance(readable.format, str)
+ assert readable.format
+
+ def test_supports_the_contract_dtype(self, readable):
+ # Whatever a store is, it holds the dtype the contract writes into it.
+ assert readable.supports_dtype(CONTRACT_DATA.dtype) is True
+```
+
+The shared body stops there. "int64 is refused by NETCDF3_CLASSIC" is a fact about
+netCDF, so it is pinned in Step 6 against the netCDF implementation, not here.
+
+- [ ] **Step 5: Add both names to the abstractness test**
+
+In `lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py`, add `"format"`
+and `"supports_dtype"` to the parametrize list of `test_dataset_declares_member`,
+keeping the existing order of the others:
+
+```python
+ @pytest.mark.parametrize(
+ "name",
+ [
+ "location",
+ "mode",
+ "format",
+ "closed",
+ "variables",
+ "dimensions",
+ "attributes",
+ "create_dimension",
+ "create_variable",
+ "supports_dtype",
+ "sync",
+ "finalise",
+ "close",
+ ],
+ )
+ def test_dataset_declares_member(self, name):
+ assert name in CFDataset.__abstractmethods__ or hasattr(CFDataset, name)
+```
+
+- [ ] **Step 6: Write the failing netCDF tests**
+
+In `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`, at the
+end of the module:
+
+```python
+class TestFormat:
+ @pytest.mark.parametrize(
+ "netcdf_format, reported",
+ [
+ ("NETCDF4", "NETCDF4"),
+ ("NETCDF4_CLASSIC", "NETCDF4_CLASSIC"),
+ ("NETCDF3_CLASSIC", "NETCDF3_CLASSIC"),
+ # netCDF4 renames this one. The saver's restricted-format list does
+ # not know the reported name: SciTools/iris#NNNN.
+ ("NETCDF3_64BIT", "NETCDF3_64BIT_OFFSET"),
+ ],
+ )
+ def test_format_is_what_netcdf4_reports(self, tmp_path, netcdf_format, reported):
+ path = tmp_path / "formatted.nc"
+ with NetCDFDataset(path, mode="w", netcdf_format=netcdf_format) as dataset:
+ assert dataset.format == reported
+
+
+class TestSupportsDtype:
+ @pytest.fixture
+ def dataset(self, tmp_path, request):
+ path = tmp_path / "typed.nc"
+ with NetCDFDataset(path, mode="w", netcdf_format=request.param) as dataset:
+ yield dataset
+
+ @pytest.mark.parametrize("dataset", ["NETCDF4"], indirect=True)
+ @pytest.mark.parametrize("dtype", ["i8", "u1", "u4", "f4", "f8", "i2", "i4"])
+ def test_netcdf4_holds_everything(self, dataset, dtype):
+ assert dataset.supports_dtype(np.dtype(dtype)) is True
+
+ @pytest.mark.parametrize(
+ "dataset", ["NETCDF4_CLASSIC", "NETCDF3_CLASSIC"], indirect=True
+ )
+ @pytest.mark.parametrize("dtype", ["i8", "u1", "u4"])
+ def test_the_restricted_formats_refuse_wide_and_unsigned_integers(
+ self, dataset, dtype
+ ):
+ assert dataset.supports_dtype(np.dtype(dtype)) is False
+
+ @pytest.mark.parametrize(
+ "dataset",
+ ["NETCDF4", "NETCDF4_CLASSIC", "NETCDF3_CLASSIC", "NETCDF3_64BIT"],
+ indirect=True,
+ )
+ @pytest.mark.parametrize("dtype", ["f4", "f8", "i2", "i4"])
+ def test_every_format_holds_the_dtypes_the_saver_writes(self, dataset, dtype):
+ assert dataset.supports_dtype(np.dtype(dtype)) is True
+
+ @pytest.mark.parametrize("dataset", ["NETCDF3_64BIT"], indirect=True)
+ def test_64bit_offset_claims_to_hold_int64(self, dataset):
+ # The honest answer is False. Answering it would turn the failure
+ # recorded in SciTools/iris#NNNN into a successful save, which is a
+ # behaviour change this relocation may not make.
+ assert dataset.supports_dtype(np.dtype("i8")) is True
+```
+
+Check the module's existing imports cover `numpy as np`, `pytest` and
+`NetCDFDataset`; add whichever are missing.
+
+- [ ] **Step 7: Run the three test files, and watch them fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/dataset/ \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/ -v
+```
+
+Expected: failures with `AttributeError: 'NetCDFDataset' object has no attribute
+'format'` and `'supports_dtype'`, plus `test_dataset_declares_member[format]` and
+`[supports_dtype]` failing on the abstract-method list.
+
+- [ ] **Step 8: Declare both members on the abstract base**
+
+In `lib/iris/fileformats/cf/dataset.py`, in `class CFDataset`, immediately after
+the `mode` property. The `format` docstring is **byte-identical** to the one on
+`zarr-io-pr-4`; copy it from
+`git show zarr-io-pr-4:lib/iris/fileformats/cf/dataset.py` rather than retyping it,
+so that whichever pull request lands second has a conflict it can resolve by
+taking either side (§2).
+
+```python
+ @property
+ @abstractmethod
+ def format(self) -> str:
+ """The concrete format of this dataset.
+
+ One of ``NETCDF4``, ``NETCDF3_CLASSIC``, ``ZARR_V2`` or ``ZARR_V3``.
+
+ What the loader, the saver and ``LOAD_PROBLEMS`` *say* when they have
+ to name the format. Reconstructing it from the implementation class
+ would be the type test this interface exists to forbid.
+
+ """
+```
+
+And after `create_variable`:
+
+```python
+ @abstractmethod
+ def supports_dtype(self, dtype) -> bool:
+ """Whether this store can hold a variable of this dtype.
+
+ The capability question the saver asks before it narrows an integer
+ type. Asking it of the store, rather than testing the format, is what
+ keeps the saver free of format names.
+
+ Parameters
+ ----------
+ dtype : :class:`numpy.dtype`
+ The dtype a variable would be created with.
+
+ Returns
+ -------
+ bool
+
+ """
+```
+
+- [ ] **Step 9: Implement both on `NetCDFDataset`**
+
+In `lib/iris/fileformats/netcdf/_dataset.py`. Add the module constant beside
+`_LEGACY_FORMATS`:
+
+```python
+#: The formats that cannot hold every dtype netCDF4 itself accepts.
+#: What netCDF4 reports as 'file_format', not what Saver was asked for: a file
+#: created as "NETCDF3_64BIT" reports "NETCDF3_64BIT_OFFSET" and so is absent
+#: from this list, which is the defect in SciTools/iris#NNNN.
+_RESTRICTED_FORMATS = ("NETCDF3_CLASSIC", "NETCDF3_64BIT", "NETCDF4_CLASSIC")
+```
+
+Add the property after `mode`, again byte-identical to `zarr-io-pr-4`:
+
+```python
+ @property
+ def format(self) -> str:
+ """The netCDF format of this file, as netCDF4 reports it."""
+ assert self._dataset is not None
+ return self._dataset.file_format
+```
+
+And the method after `create_variable`:
+
+```python
+ def supports_dtype(self, dtype) -> bool:
+ """Whether this file can hold a variable of this dtype.
+
+ Parameters
+ ----------
+ dtype : :class:`numpy.dtype`
+ The dtype a variable would be created with.
+
+ Returns
+ -------
+ bool
+ False for 64-bit and unsigned integers on the restricted formats,
+ which have no such types. True otherwise.
+
+ """
+ if self.format not in _RESTRICTED_FORMATS:
+ return True
+ return not (
+ np.issubdtype(dtype, np.int64) or np.issubdtype(dtype, np.unsignedinteger)
+ )
+```
+
+- [ ] **Step 10: Give `MinimalDataset` the two new members**
+
+In `lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py`, `MinimalDataset`
+implements the whole interface, so it stops being instantiable without them. Add
+`format` beside `mode`, and the method beside `create_variable`:
+
+```python
+class MinimalDataset(CFDataset):
+ """The smallest thing that satisfies CFDataset."""
+
+ location = "<memory>"
+ mode = "r"
+ format = "MINIMAL"
+ closed = False
+```
+
+```python
+ def supports_dtype(self, dtype):
+ return True
+```
+
+- [ ] **Step 11: Run the dataset tests, and watch them pass**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/dataset/ \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/ -v
+```
+
+Expected: all pass, including the netCDF side of the shared contract.
+
+- [ ] **Step 12: Rewrite `_ensure_valid_dtype` to ask the dataset**
+
+In `lib/iris/fileformats/netcdf/saver.py`. Replace the format test and the message
+argument; leave every other line of the method as it is.
+
+```python
+ def _ensure_valid_dtype(self, values, src_name, src_object):
+ # Some formats have no 64-bit or unsigned integer types, so we check
+ # whether we can store them as int32 instead. Which dtypes are worth
+ # narrowing is the saver's question; whether this store holds them is
+ # the store's.
+ is_narrowable_integer = np.issubdtype(values.dtype, np.int64) or np.issubdtype(
+ values.dtype, np.unsignedinteger
+ )
+ if is_narrowable_integer and not self._dataset.supports_dtype(values.dtype):
+ val_min, val_max = (values.min(), values.max())
+ if is_lazy_data(values):
+ val_min, val_max = _co_realise_lazy_arrays([val_min, val_max])
+ # NumPy will inherit values.dtype even if the scalar numbers work
+ # with a smaller type.
+ min_dtype = np.promote_types(
+ *[np.min_scalar_type(m) for m in (val_min, val_max)]
+ )
+ # Cast to an integer type the format supports.
+ if not np.can_cast(min_dtype, np.int32):
+ msg = (
+ "The data type of {} {!r} is not supported by {} and"
+ " its values cannot be safely cast to a supported"
+ " integer type."
+ )
+ msg = msg.format(src_name, src_object, self._dataset.format)
+ raise ValueError(msg)
+ values = values.astype(np.int32)
+ return values
+```
+
+- [ ] **Step 13: Run the characterisation test again, then the saver suite**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py -v
+pytest lib/iris/tests/unit/fileformats/netcdf/ lib/iris/tests/unit/fileformats/cf/ -q
+```
+
+Expected: 21 passed, then the whole of both packages green. Step 2's file is the
+one that matters here. It passed before the rewrite and must pass after it,
+unchanged.
+
+- [ ] **Step 14: Check that no netCDF format string survives in the saver**
+
+```bash
+grep -n "NETCDF3\|NETCDF4\|file_format" lib/iris/fileformats/netcdf/saver.py
+```
+
+Expected: matches only in `Saver.__init__`'s validation list and in docstrings.
+No match inside `_ensure_valid_dtype`. Those remaining matches stay in
+`netcdf/saver.py` when the file moves in Task 4.
+
+- [ ] **Step 15: Run pre-commit on the touched files**
+
+```bash
+pre-commit run --files \
+ lib/iris/fileformats/cf/dataset.py \
+ lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/tests/unit/fileformats/cf/dataset/contract.py \
+ lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py
+```
+
+Hooks rewrite files in place. Re-stage and re-run until it passes cleanly.
+
+- [ ] **Step 16: Commit**
+
+```bash
+git add lib/iris/fileformats/cf/dataset.py \
+ lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/tests/unit/fileformats/cf/dataset/contract.py \
+ lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py
+git commit -m "Ask the dataset whether it holds a dtype
+
+The saver read the open file's netCDF format string to decide whether to
+narrow an integer to int32. That read would become generic code naming
+netCDF formats once the CF saver moves, so the question now goes to the
+dataset as a capability, and the format survives only in the message the
+user reads.
+
+A file created as NETCDF3_64BIT reports itself as NETCDF3_64BIT_OFFSET
+and so escapes the narrowing, which fails the save. That is today's
+behaviour and is kept, with a test saying so.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 2: `_dask_locks` moves to `cf/`
+
+Spec §4.5, "Multi-process writes: kept reachable, not built". The module imports
+`threading` and four `dask` modules and nothing from Iris, so this is a move and a
+docstring. It is a small task and it is on its own, because it is the one part of
+this pull request that can be reviewed without reading the saver at all.
+
+**Files:**
+- Move: `lib/iris/fileformats/netcdf/_dask_locks.py` → `lib/iris/fileformats/cf/_dask_locks.py`
+- Move: `lib/iris/tests/integration/netcdf/test__dask_locks.py` → `lib/iris/tests/integration/cf/test__dask_locks.py`
+- Create: `lib/iris/tests/integration/cf/__init__.py`
+- Modify: `lib/iris/fileformats/netcdf/_dataset.py:36`
+- Modify: `lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py:80`
+
+**Interfaces:**
+- Consumes: nothing from Task 1.
+- Produces: `iris.fileformats.cf._dask_locks`, exporting `DaskSchedulerTypeError`,
+ `dask_scheduler_is_distributed`, `get_dask_array_scheduler_type` and
+ `get_worker_lock`, all unchanged. `iris.fileformats.netcdf._dask_locks` stops
+ existing: it is private, so spec §4.8 gives it no alias.
+
+- [ ] **Step 1: Move the test, and repoint its import**
+
+```bash
+mkdir -p lib/iris/tests/integration/cf
+git mv lib/iris/tests/integration/netcdf/test__dask_locks.py \
+ lib/iris/tests/integration/cf/test__dask_locks.py
+```
+
+Create `lib/iris/tests/integration/cf/__init__.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Integration tests for the :mod:`iris.fileformats.cf` package."""
+```
+
+In the moved file, change the import and the first line of the module docstring:
+
+```python
+"""Unit tests for the :mod:`iris.fileformats.cf._dask_locks` package.
+```
+
+```python
+from iris.fileformats.cf._dask_locks import (
+ DaskSchedulerTypeError,
+ dask_scheduler_is_distributed,
+ get_dask_array_scheduler_type,
+ get_worker_lock,
+)
+```
+
+Leave the rest of that file alone, including the docstring of
+`test_load_does_not_need_a_worker_lock`, which quotes the error message. The
+message itself does not change: see Step 3.
+
+- [ ] **Step 2: Run it, and watch it fail**
+
+```bash
+pytest lib/iris/tests/integration/cf/test__dask_locks.py -x
+```
+
+Expected: collection error, `ModuleNotFoundError: No module named
+'iris.fileformats.cf._dask_locks'`.
+
+- [ ] **Step 3: Move the module, and rewrite its docstring**
+
+```bash
+git mv lib/iris/fileformats/netcdf/_dask_locks.py \
+ lib/iris/fileformats/cf/_dask_locks.py
+```
+
+Replace the module docstring, the whole of it, with this. Nothing below the
+docstring changes.
+
+```python
+"""Locks that let dask workers co-operate when writing to one target.
+
+A saver writing lazy data hands each chunk to a dask worker, and the workers
+must not write over each other. What that takes depends on the dask scheduler,
+so this module works out which scheduler is in play and returns a lock to
+match. The scheduler must therefore be configured before a lock is asked for.
+
+A dataset implementation asks for one lock per target, identified by its path
+or URL, and gives it to every write handle it builds. The netCDF write handle
+reopens the file, acquires the lock, writes its chunk, closes the file and
+releases the lock, which is the mirror image of what the read path does for a
+lazy chunk. A store whose chunks are separate objects needs no such
+coordination, and asks for no lock.
+
+For a threaded scheduler the lock is a plain :class:`threading.Lock`. The
+workers are threads sharing one address space, so they all hold the same
+object, and the identity string is never used.
+
+For a distributed scheduler the lock is a ``distributed.Lock`` named for the
+target. It reaches the workers by being serialised along with the task, and
+every copy carries the same name, so the scheduler can make them exclude one
+another. A worker behaves like a process, and may run on another machine.
+
+The "processes" scheduler is refused by :func:`get_worker_lock`. Supporting it
+would need a serialisable shared lock that does not require a distributed
+scheduler to function.
+
+Writing to one target from genuinely separate processes is a further step
+again. For netCDF it needs libnetcdf built with the parallel access option,
+which is uncommon: with an ordinary build, opening a file for writing that is
+already open for writing simply raises an access error. Iris does not support
+that mode at present.
+
+"""
+```
+
+The `DaskSchedulerTypeError` message still ends "which is not supported by the
+Iris netcdf saver". Leave it. It is text a user reads, so changing it is a
+behaviour change, and it stays true while netCDF is the only saver asking for a
+lock. A second saver that wants one is what should reword it.
+
+- [ ] **Step 4: Repoint the one library import**
+
+In `lib/iris/fileformats/netcdf/_dataset.py`, line 36 imports three siblings. Split
+the moved one out:
+
+```python
+from iris.fileformats.cf import _dask_locks
+
+from . import _bytecoding_datasets, _thread_safe_nc
+```
+
+Put the `iris.fileformats.cf` import with the other absolute `iris` imports above,
+where `from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable`
+already sits, and leave the relative import where it is. Ruff's import sorting
+will insist on this anyway; doing it by hand first keeps the diff readable.
+
+This import works despite `cf/__init__.py` importing `_reader`, which imports this
+very module. `cf/_dask_locks.py` imports nothing from Iris, so Python finishes it
+while `iris.fileformats.cf` is still partly initialised, exactly as the
+`cf.dataset` import on the line above already does (F5).
+
+- [ ] **Step 5: Run the moved tests, and watch them pass**
+
+```bash
+pytest lib/iris/tests/integration/cf/test__dask_locks.py -v
+```
+
+Expected: all pass. This file configures real dask schedulers, including
+distributed, so it is slower than a unit test and is not run with `-n auto`.
+
+- [ ] **Step 6: Find every stale reference**
+
+```bash
+grep -rn "netcdf._dask_locks\|netcdf import _dask_locks\|netcdf\.saver\.NetCDFWriteProxy" \
+ lib benchmarks docs
+```
+
+Expected: one match, a comment in
+`lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py`. Repoint it:
+
+```python
+ # _dask_locks.get_worker_lock() returns a FRESH threading.Lock under
+```
+
+becomes
+
+```python
+ # cf._dask_locks.get_worker_lock() returns a FRESH threading.Lock under
+```
+
+If the grep finds anything else, repoint it too. Nothing outside `lib/` should
+match: the module is private and has never been documented.
+
+- [ ] **Step 7: Run the netCDF and CF unit suites**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/ lib/iris/tests/unit/fileformats/cf/ -q
+```
+
+Expected: all pass. A circular import from Step 4 would show here as a collection
+error naming `iris.fileformats.cf`, not as a test failure.
+
+- [ ] **Step 8: Prove the import graph is still acyclic, from a cold interpreter**
+
+```bash
+python -c "import iris.fileformats.cf; print('cf first: ok')"
+python -c "import iris.fileformats.netcdf; print('netcdf first: ok')"
+python -c "import iris.fileformats.cf._dask_locks; print('locks first: ok')"
+```
+
+Expected: three lines of `ok`. Import order is the thing a circular import is
+sensitive to, and the test suite only ever exercises one of these orders.
+
+- [ ] **Step 9: Run pre-commit on the touched files**
+
+```bash
+pre-commit run --files \
+ lib/iris/fileformats/cf/_dask_locks.py \
+ lib/iris/fileformats/netcdf/_dataset.py \
+ lib/iris/tests/integration/cf/__init__.py \
+ lib/iris/tests/integration/cf/test__dask_locks.py \
+ lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py
+```
+
+- [ ] **Step 10: Commit**
+
+```bash
+git add -A lib/iris/fileformats lib/iris/tests
+git commit -m "Move the dask lock toolkit to iris.fileformats.cf
+
+The module coordinates dask workers writing to one target, and does it
+by asking dask which scheduler is in play. It imports nothing from Iris
+and nothing from netCDF4, so it was only ever netCDF's by where it sat.
+A second saver needing worker coordination should find it rather than
+grow its own.
+
+The code is unchanged. The docstring no longer describes the netCDF
+saver as the only caller.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 3: Name the four places the saver is netCDF
+
+The class stays where it is and keeps working. All this task does is lift four
+blocks of netCDF-specific code out of generic methods and into methods of their
+own, so that Task 4's split is a matter of deciding which class each method
+belongs to. F1 lists them; nothing else in the class names netCDF after Task 1.
+
+**Files:**
+- Modify: `lib/iris/fileformats/netcdf/saver.py`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__hooks.py` (new)
+
+**Interfaces:**
+- Consumes: `CFDataset.format` from Task 1, for one error message in Task 4.
+- Produces, all on `Saver`:
+ - `_create_string_variable(cube_or_mesh, cf_name, element, element_dims, data)
+ -> CFDatasetVariable`
+ - `_backend_write(data, cf_var) -> bool`
+ - `_patch_dataset(cf_patch, profile, cf_var_cube) -> None`
+ - `_flush_before_close() -> None`
+
+ Task 4 gives each of them a base version on `CFSaver` and leaves these four as
+ overrides.
+
+- [ ] **Step 1: Write the failing test for the hooks that have no test today**
+
+Create `lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__hooks.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The netCDF-specific parts of the save chain, named and tested.
+
+Relocating the CF saver (SciTools/iris#6977) moves the rest of the class to a
+format-agnostic home. Each test here covers one of the parts that stays
+behind, from the outside, so that the move cannot quietly change what a
+backend is asked to do.
+"""
+
+import dask.array as da
+import numpy as np
+import pytest
+
+import iris
+from iris.coords import AuxCoord
+from iris.cube import Cube
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+from iris.fileformats.netcdf._dataset import NetCDFDataset
+
+
+class TestStringVariables:
+ """Unicode data becomes a char variable over a string dimension."""
+
+ @pytest.fixture
+ def labelled_cube(self):
+ cube = Cube(np.arange(3.0), var_name="payload")
+ cube.add_aux_coord(
+ AuxCoord(np.array(["alpha", "beta", "gamma"]), var_name="label"), 0
+ )
+ return cube
+
+ def test_a_string_dimension_is_created_for_the_label(
+ self, labelled_cube, tmp_path
+ ):
+ path = tmp_path / "labelled.nc"
+ iris.save(labelled_cube, path)
+ with NetCDFDataset(path) as dataset:
+ # Five ascii characters, so five bytes. The dimension is the
+ # width, and exists only because netCDF has no unicode type.
+ assert dataset.dimensions["string5"] == 5
+
+ def test_the_label_round_trips(self, labelled_cube, tmp_path):
+ path = tmp_path / "labelled.nc"
+ iris.save(labelled_cube, path)
+ loaded = iris.load_cube(path)
+ assert list(loaded.coord("label").points) == ["alpha", "beta", "gamma"]
+
+
+class TestNCZarrWrites:
+ """NCZarr cannot be reopened, so lazy data is written before the close."""
+
+ def test_lazy_data_reaches_an_nczarr_target(self, tmp_path):
+ cube = Cube(da.arange(6.0, chunks=3).reshape(2, 3), long_name="payload")
+ assert cube.has_lazy_data()
+ uri = (tmp_path / "out.zarr").as_uri() + "#mode=nczarr,file"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(cube, uri)
+ loaded = iris.load_cube(uri)
+ assert loaded.data.ravel().tolist() == [0.0, 1.0, 2.0, 3.0, 4.0, 5.0]
+
+
+class TestCFPatch:
+ """The site-configuration hook is documented as receiving netCDF4 objects."""
+
+ def test_cf_patch_receives_raw_netcdf4_objects(self, mocker, tmp_path):
+ seen = {}
+
+ def record(profile, dataset, variable):
+ seen["dataset"] = dataset
+ seen["variable"] = variable
+
+ mocker.patch.dict(
+ iris.site_configuration,
+ {"cf_profile": lambda cube: "profiled", "cf_patch": record},
+ )
+ iris.save(Cube(np.arange(3.0), var_name="payload"), tmp_path / "patched.nc")
+
+ # Unwrapped, both of them. A CFVariable defines no __setattr__, so a
+ # hook doing 'variable.name = value' against one would succeed and
+ # write nothing at all.
+ assert not isinstance(seen["dataset"], CFDataset)
+ assert not isinstance(seen["variable"], CFDatasetVariable)
+ assert hasattr(seen["dataset"], "createVariable")
+ assert hasattr(seen["variable"], "setncattr")
+```
+
+- [ ] **Step 2: Run it, and watch it pass**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__hooks.py -v
+```
+
+Expected: 4 passed. Like Task 1's file, this one pins today's behaviour before the
+code is rearranged. If `test_cf_patch_receives_raw_netcdf4_objects` fails, check
+that `iris.site_configuration` is still a plain dict before changing anything else.
+
+- [ ] **Step 3: Extract `_create_string_variable`**
+
+In `_create_generic_cf_array_var`, replace the whole unicode branch — from
+`string_dimension_depth = data.dtype.itemsize` down to and including
+`cf_var = self._dataset.create_variable(cf_name, "|S1", element_dims)` — with one
+call:
+
+```python
+ if not is_dataless and data.dtype.kind == "U":
+ # Unicode string data: typically a CF label variable, sometimes an
+ # ancillary variable. How a store holds strings is the store's own
+ # business, so the backend creates the variable.
+ cf_var = self._create_string_variable(
+ cube_or_mesh, cf_name, element, element_dims, data
+ )
+ else:
+```
+
+Add the method next to `_create_cf_bounds`, holding the removed block unchanged:
+
+```python
+ def _create_string_variable(
+ self, cube_or_mesh, cf_name, element, element_dims, data
+ ):
+ """Create the char variable that will hold this unicode string data.
+
+ netCDF has no unicode type, so string data is stored as bytes over an
+ extra trailing dimension whose length is the width of the widest
+ value. The name of that dimension is appended to ``element_dims``.
+
+ Parameters
+ ----------
+ cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
+ The Cube or Mesh being saved. Named in the error message.
+ cf_name : str
+ The name the variable is created with.
+ element : :class:`iris.coords._DimensionalMetadata`
+ The element being saved. Supplies the ``_Encoding`` attribute.
+ element_dims : list of str
+ The variable's dimension names, appended to in place.
+ data : :class:`numpy.ndarray`
+ The string values, used for their dtype.
+
+ Returns
+ -------
+ :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+
+ """
+ # NOTE: all we are doing here is to calculate the byte dimension length,
+ # based on the dtype and any encoding attribute.
+ # The actual char --> byte data *translation* is done by the variable,
+ # being a _bytecoding_datasets.EncodedVariable.
+ string_dimension_depth = data.dtype.itemsize
+
+ # String content (U) instead of bytes (S).
+ # For numpy strings, itemsize is **always** a multiple of 4
+ if string_dimension_depth % 4 != 0:
+ msg = (
+ "Unexpected numpy string 'dtype.itemsize' for element "
+ f"{cube_or_mesh.name()!r}: "
+ f"'dtype.itemsize = {string_dimension_depth}, expected "
+ "a multiple of four (always)."
+ )
+ raise ValueError(msg)
+ nchars = string_dimension_depth // 4
+
+ encoding_attr = element.attributes.get("_Encoding", "ascii")
+ # Look this up + return a supported encoding name
+ # NB implements defaults and raises a warning if given not recognised.
+ encoding = bytecoding_datasets._identify_encoding(
+ encoding=encoding_attr, var_name=cf_name, writing=True
+ )
+ width_fns = bytecoding_datasets._ENCODING_WIDTH_TRANSLATIONS[encoding]
+ string_dimension_depth = width_fns.nchars_2_nbytes(nchars)
+
+ string_dimension_name = "string%d" % string_dimension_depth
+
+ # Determine whether to create the string length dimension.
+ if string_dimension_name not in self._dataset.dimensions:
+ while string_dimension_name in self._dataset.variables:
+ # Also avoid collision with variable names.
+ # See '_get_dim_names' for reason.
+ string_dimension_name = self._increment_name(string_dimension_name)
+ self._dataset.create_dimension(
+ string_dimension_name, string_dimension_depth
+ )
+
+ # Add the string length dimension to the variable dimensions.
+ element_dims.append(string_dimension_name)
+
+ # Create the label coordinate variable.
+ return self._dataset.create_variable(cf_name, "|S1", element_dims)
+```
+
+- [ ] **Step 4: Extract `_backend_write`**
+
+Two netCDF branches inside `_lazy_stream_data` become one question: does the
+backend take this write itself? The emulated case does, and so does NCZarr, which
+collects lazy writes for `__exit__` because it cannot reopen the store.
+
+This adds an early return, which `lib/iris/AGENTS.md` warns about. The rule is
+about a function that must still warn, record or close on the way out, and this
+one does none of those: after it stores, it returns. The emulated branch already
+reaches the end of the method without storing, and the NCZarr branch already
+returns early. Say so in the pull request body, because a reviewer reading the
+rule rather than the method will ask.
+
+Replace the body of `_lazy_stream_data` below the squeeze with:
+
+```python
+ if self._backend_write(data, cf_var):
+ return
+
+ doing_delayed_save = is_lazy_data(data)
+ if doing_delayed_save:
+ # save lazy data with a delayed operation. For now, we just record the
+ # necessary information -- a single, complete delayed action is
+ # constructed later by a call to delayed_completion().
+ def store(
+ data: np.typing.ArrayLike,
+ cf_var: CFDatasetVariable,
+ ) -> None:
+ # TODO: recheck once Zarr behaviour is settled.
+ # Ask the variable for something a worker can stream into
+ # after this file is closed. What that is is the backend's
+ # business: netCDF reopens the file, Zarr will hand back the
+ # array itself.
+ self._delayed_writes.append((data, cf_var.write_handle()))
+
+ else:
+ # Real data is always written directly, i.e. not via lazy save.
+ def store(
+ data: np.typing.ArrayLike,
+ cf_var: CFDatasetVariable,
+ ) -> None:
+ cf_var[:] = data # type: ignore[index]
+
+ # Store the data.
+ store(data, cf_var)
+```
+
+The type annotations change from `NetCDFDatasetVariable` to `CFDatasetVariable`,
+here and on `_lazy_stream_data` itself. Import `CFDatasetVariable` from
+`iris.fileformats.cf.dataset`; leave the `NetCDFDatasetVariable` import, which the
+new method below uses.
+
+Add the new method directly above `_lazy_stream_data`:
+
+```python
+ def _backend_write(
+ self,
+ data: np.typing.ArrayLike,
+ cf_var: NetCDFDatasetVariable,
+ ) -> bool:
+ """Write the data here and now, if this backend handles it itself.
+
+ Returns
+ -------
+ bool
+ Whether the data was written. False leaves the caller to write it
+ the ordinary way, directly or by a delayed store.
+
+ """
+ if cf_var.is_emulated:
+ # The variable is not an actual file variable, but an emulating
+ # object with an attached data array (either numpy or dask), which should
+ # be copied immediately to the target. This is used as a hook to translate
+ # data to/from netcdf data container objects in other packages, such as
+ # xarray.
+ # See https://github.com/SciTools/iris/issues/4994 "Xarray bridge".
+ if cf_var.dtype.kind == "U":
+ # We also need to perform string-to-bytes encoding since, in ncdata,
+ # the emulating objects don't do this, and also don't support a
+ # 'set_auto_chartostring(True)'.
+ # Therefore, do here what an EncodedVariable.__setitem__ would do : ..
+ # .. get details from the file (char) variable to be written ..
+ encoder = VariableEncoder.from_var(cf_var.unencoded_variable)
+ # .. apply encoding to get the bytes to write.
+ data = encoder.encode_strings_as_bytearray(data)
+
+ cf_var.emulated_data_array = data
+ return True
+
+ if is_lazy_data(data) and self._is_nczarr:
+ # NCZarr cannot be reopened for deferred writes.
+ # Collect here; da.store() will be called in __exit__.
+ self._nczarr_writes.append((data, cf_var))
+ return True
+
+ return False
+```
+
+- [ ] **Step 5: Extract `_flush_before_close`**
+
+`__exit__` becomes:
+
+```python
+ def __exit__(self, type, value, traceback):
+ """Flush any buffered data to the CF-NetCDF/NcZarr file before closing."""
+ self._flush_before_close()
+ self._dataset.sync()
+ if not self._to_open_dataset:
+ # Only close if the Saver created it.
+ self._dataset.close()
+ # Complete after closing, if required
+ if self.compute:
+ self.complete()
+```
+
+and the new method sits beside it:
+
+```python
+ def _flush_before_close(self) -> None:
+ """Write anything that cannot be written after the dataset closes."""
+ if self._nczarr_writes:
+ # NCZarr lazy writes must be computed while the file is still open;
+ # the deferred reopen-write pattern used for netCDF is not supported.
+ sources, targets = zip(*self._nczarr_writes)
+ da.store(list(sources), list(targets))
+```
+
+Do **not** add a `self._dataset.finalise()` call here. Spec §4.2 says `finalise`
+is defined and not yet called, and that PR 6 wires it, "rather than guess at an
+ordering only Zarr can exercise".
+
+- [ ] **Step 6: Extract `_patch_dataset`**
+
+At the end of `write`, the call becomes:
+
+```python
+ if cf_profile_available:
+ cf_patch = iris.site_configuration.get("cf_patch")
+ if cf_patch is not None:
+ self._patch_dataset(cf_patch, profile, cf_var_cube)
+ else:
+ msg = "cf_profile is available but no {} defined.".format("cf_patch")
+ warnings.warn(msg, category=iris.warnings.IrisCfSaveWarning)
+```
+
+and the method joins the others:
+
+```python
+ def _patch_dataset(self, cf_patch, profile, cf_var_cube) -> None:
+ """Hand the site-configured patch hook the objects it expects."""
+ # A reserved iris.site_configuration hook, documented since
+ # Iris 1.3 as receiving netCDF4 objects - so both arguments
+ # are unwrapped, not just the dataset. A CFVariable would
+ # raise on setncattr() and, worse, silently swallow
+ # 'variable.name = value': it defines no __setattr__.
+ cf_patch(profile, self._dataset.dataset, cf_var_cube.variable)
+```
+
+- [ ] **Step 7: Run the hook tests, then the whole netCDF saver suite**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/saver/ -v
+```
+
+Expected: all pass, including `test_Saver__lazy_stream_data.py`, which mocks the
+emulated and NCZarr paths. That file is the real net under Step 4: if the two
+branches changed order or condition, it fails.
+
+- [ ] **Step 8: Run the integration tests that exercise these four paths**
+
+```bash
+pytest lib/iris/tests/integration/netcdf/test_nczarr.py \
+ lib/iris/tests/integration/netcdf/test_stringdata.py \
+ lib/iris/tests/integration/netcdf/test_delayed_save.py -q
+```
+
+Expected: all pass. These are slower and are the ones that touch real NCZarr
+stores, real char variables and real delayed writes.
+
+- [ ] **Step 9: Check that the four methods are the only netCDF left**
+
+```bash
+grep -n "is_emulated\|_is_nczarr\|_nczarr_writes\|VariableEncoder\|bytecoding_datasets\|\.dataset\.\|unencoded_variable" \
+ lib/iris/fileformats/netcdf/saver.py
+```
+
+Expected: matches only inside `__init__`, `_create_string_variable`,
+`_backend_write`, `_flush_before_close`, `_patch_dataset`, `save` and
+`save_mesh`. Any match in another method is a fifth netCDF leak the plan did not
+find; extract it the same way and say so in the pull request body.
+
+- [ ] **Step 10: Run pre-commit, then commit**
+
+```bash
+pre-commit run --files \
+ lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__hooks.py
+git add lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__hooks.py
+git commit -m "Gather the saver's netCDF-specific code into four methods
+
+The class mixes the CF data model with netCDF4's own vocabulary. Four
+blocks are the whole of the second part: creating a char variable over a
+string dimension, writing to an emulating variable or an NCZarr target,
+flushing those writes before the file closes, and handing the
+site-configured patch hook its unwrapped netCDF4 objects.
+
+Each is now a method with a name, called from the generic code that used
+to contain it. Nothing moves modules and nothing changes behaviour.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 4: Move the saver to `iris.fileformats.cf.saver`
+
+The pull request's reason for existing. After Task 3 the class divides cleanly, so
+this task moves the file, renames the class to `CFSaver`, and leaves
+`iris.fileformats.netcdf.saver` holding a subclass, two functions and four
+deprecating aliases.
+
+**Files:**
+- Move: `lib/iris/fileformats/netcdf/saver.py` → `lib/iris/fileformats/cf/saver.py`
+- Create: `lib/iris/fileformats/netcdf/saver.py`, from what the move leaves behind
+- Modify: `lib/iris/fileformats/netcdf/__init__.py`
+- Test: `lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py` (new)
+
+**Interfaces:**
+- Consumes: the four hooks from Task 3; `CFDataset.format` from Task 1.
+- Produces:
+ - `iris.fileformats.cf.saver.CFSaver`, constructed from an open `CFDataset`
+ - `iris.fileformats.cf.saver.{CFNameCoordMap, CF_CONVENTIONS_VERSION,
+ MESH_ELEMENTS, SPATIO_TEMPORAL_AXES, SaverFillValueWarning, _CF_ATTRS,
+ _CF_DATA_ATTRS, _CF_GLOBAL_ATTRS, _UKMO_DATA_ATTRS, _FACTORY_DEFNS}`
+ - `iris.fileformats.netcdf.saver.Saver(CFSaver)`, with today's `__init__` and
+ `write` signatures unchanged, plus `save` and `save_mesh` unchanged
+
+ Task 5 repoints every reader of those names.
+
+- [ ] **Step 1: Record the method inventory, from the code as it stands**
+
+```bash
+python -c "
+import inspect
+from iris.fileformats.netcdf.saver import Saver
+print(sorted(n for n, v in inspect.getmembers(Saver) if callable(v) and not n.startswith('__')))
+"
+```
+
+Expected: 29 names. Compare them against the list written into the test in Step 2.
+If the two differ, the list in this plan is stale: use what the command prints,
+and say so in the pull request body.
+
+- [ ] **Step 2: Write the failing tests for the names that must survive**
+
+Create `lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Names that must survive the CF saver's move to iris.fileformats.cf.
+
+Two separate promises. Four public names moved and are still reachable from
+their old homes, warning when they are. Every method of the netCDF Saver kept
+its name, so that code subclassing it keeps overriding what it overrode.
+"""
+
+import inspect
+import warnings
+
+import numpy as np
+import pytest
+
+import iris
+from iris._deprecation import IrisDeprecation
+from iris.cube import Cube
+from iris.fileformats import netcdf
+from iris.fileformats.cf import saver as cf_saver
+from iris.fileformats.netcdf import saver as netcdf_saver
+from iris.loading import LOAD_PROBLEMS
+
+#: The four names spec section 4.8 relocates, with warn-on-use aliases left
+#: behind at both their old homes.
+RELOCATED = [
+ "CFNameCoordMap",
+ "CF_CONVENTIONS_VERSION",
+ "MESH_ELEMENTS",
+ "SPATIO_TEMPORAL_AXES",
+]
+
+#: Every method the netCDF Saver answered to before the generic half of it
+#: became CFSaver. A name missing from the class is an override, somewhere
+#: outside Iris, that has silently stopped overriding anything.
+SAVER_METHODS = [
+ "_add_ancillary_variables",
+ "_add_aux_coords",
+ "_add_aux_factories",
+ "_add_cell_measures",
+ "_add_dim_coords",
+ "_add_grid_mapping_to_dataset",
+ "_add_inner_related_vars",
+ "_add_mesh",
+ "_cf_coord_standardised_units",
+ "_create_cf_bounds",
+ "_create_cf_cell_methods",
+ "_create_cf_data_variable",
+ "_create_cf_dimensions",
+ "_create_cf_grid_mapping",
+ "_create_generic_cf_array_var",
+ "_create_mesh",
+ "_ensure_valid_dtype",
+ "_get_dim_names",
+ "_get_element_variable_name",
+ "_get_mesh_variable_name",
+ "_increment_name",
+ "_lazy_stream_data",
+ "_set_cf_var_attributes",
+ "cf_valid_var_name",
+ "check_attribute_compliance",
+ "complete",
+ "delayed_completion",
+ "update_global_attributes",
+ "write",
+]
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+class TestFromTheSaverModule:
+ def test_resolves_to_the_same_object(self, name):
+ with pytest.warns(IrisDeprecation):
+ assert getattr(netcdf_saver, name) is getattr(cf_saver, name)
+
+ def test_the_warning_names_the_new_home(self, name):
+ with pytest.warns(IrisDeprecation, match=f"iris.fileformats.cf.saver.{name}"):
+ getattr(netcdf_saver, name)
+
+ def test_is_discoverable(self, name):
+ # PEP 562 forwards the read but not dir(), so the module says so itself.
+ assert name in dir(netcdf_saver)
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+class TestFromThePackage:
+ def test_resolves_to_the_same_object(self, name):
+ with pytest.warns(IrisDeprecation):
+ assert getattr(netcdf, name) is getattr(cf_saver, name)
+
+ def test_is_discoverable(self, name):
+ assert name in dir(netcdf)
+
+
+class TestNamesThatDoNotMove:
+ @pytest.mark.parametrize("name", ["Saver", "save", "save_mesh"])
+ def test_the_saver_module_still_defines_it(self, name):
+ # Spec section 4.8: netCDF API, and it stays that way. Defined here,
+ # not forwarded, so reading it must not warn.
+ assert name in vars(netcdf_saver)
+
+ def test_an_unknown_name_raises_quietly(self, recwarn):
+ with pytest.raises(AttributeError, match="no attribute"):
+ netcdf_saver.not_a_real_name
+ assert recwarn.list == []
+
+
+class TestSaverMethods:
+ @pytest.mark.parametrize("name", SAVER_METHODS)
+ def test_the_netcdf_saver_still_answers_to_it(self, name):
+ assert callable(getattr(netcdf_saver.Saver, name, None))
+
+ def test_the_netcdf_saver_is_a_cf_saver(self):
+ assert issubclass(netcdf_saver.Saver, cf_saver.CFSaver)
+
+ def test_write_keeps_its_signature(self):
+ # Documented API with defaults users rely on, and save() passes some
+ # of them by position.
+ parameters = inspect.signature(netcdf_saver.Saver.write).parameters
+ assert list(parameters) == [
+ "self",
+ "cube",
+ "local_keys",
+ "unlimited_dimensions",
+ "zlib",
+ "complevel",
+ "shuffle",
+ "fletcher32",
+ "contiguous",
+ "chunksizes",
+ "endian",
+ "least_significant_digit",
+ "packing",
+ "fill_value",
+ ]
+ assert parameters["complevel"].default == 4
+ assert parameters["shuffle"].default is True
+ assert parameters["endian"].default == "native"
+
+
+class TestNothingInsideIrisUsesTheOldNames:
+ def test_a_save_emits_no_deprecation(self, tmp_path):
+ # The aliases are for code outside Iris. A save that trips one means a
+ # reader inside Iris was left pointing at the old home. Promoting to an
+ # error is safe here: iris.save has no equivalent of LOAD_PROBLEMS, so
+ # an exception reaches the test rather than being recorded.
+ #
+ # save_split_attrs is set because the legacy attribute mode is itself
+ # deprecated, and its warning would fail this test for a reason that
+ # has nothing to do with where the saver lives.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(
+ Cube(np.arange(3.0), var_name="payload"), tmp_path / "quiet.nc"
+ )
+
+ def test_a_load_emits_no_deprecation(self, tmp_path):
+ # Recorded rather than promoted to an error, for the reason
+ # SciTools/iris#7333 found: iris.load captures an exception raised
+ # while building a cube into iris.loading.LOAD_PROBLEMS, so a filter of
+ # "error" drops the offending coordinate and the test passes against an
+ # implementation that is still warning. The LOAD_PROBLEMS assertion is
+ # what keeps a silent load from being a degraded one.
+ path = tmp_path / "quiet.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(Cube(np.arange(3.0), var_name="payload"), path)
+
+ with warnings.catch_warnings(record=True) as record:
+ warnings.simplefilter("always")
+ cubes = iris.load(path)
+
+ deprecations = [
+ str(warning.message)
+ for warning in record
+ if issubclass(warning.category, IrisDeprecation)
+ ]
+ assert deprecations == []
+ assert LOAD_PROBLEMS.problems_by_file.get(str(path), []) == []
+ assert len(cubes) == 1
+```
+
+- [ ] **Step 3: Run it, and watch it fail**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py -x
+```
+
+Expected: a collection error, `ModuleNotFoundError: No module named
+'iris.fileformats.cf.saver'`.
+
+- [ ] **Step 4: Move the file, and bring a copy back**
+
+```bash
+git mv lib/iris/fileformats/netcdf/saver.py lib/iris/fileformats/cf/saver.py
+git checkout HEAD -- lib/iris/fileformats/netcdf/saver.py
+```
+
+Both files now hold the same 3178 lines. Steps 5 and 6 cut each down to its half.
+Doing it this way keeps git's rename detection pointed at `cf/saver.py`, which is
+where most of the content ends up, so the diff reads as a move with edits rather
+than as a delete and an add.
+
+- [ ] **Step 5: Cut `cf/saver.py` down to the generic saver**
+
+Delete, in this file only:
+
+- the four netCDF imports: `from iris.fileformats.netcdf import _bytecoding_datasets
+ as bytecoding_datasets`, `from iris.fileformats.netcdf._attribute_handlers import
+ ATTRIBUTE_HANDLERS`, `from iris.fileformats.netcdf._bytecoding_datasets import
+ VariableEncoder`, `from iris.fileformats.netcdf._dataset import NetCDFDataset,
+ NetCDFDatasetVariable`
+- the logger import `from . import logger`, the comment above it and the bare
+ `logger` statement below it (F8)
+- `import codecs`, `import os`, `from pathlib import Path` and
+ `from urllib.parse import urlsplit`, if nothing left in the file uses them.
+ Check each with grep rather than guessing; `codecs` and `os` are used by code
+ that stays, and `Path` and `urlsplit` are used by `Saver.__init__`, which does
+ not.
+- `Saver.__init__`, `Saver.write`, and the four hooks Task 3 named
+- `save` and `save_mesh`
+
+Add `from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable` if Task
+3 did not already, and replace the module docstring with:
+
+```python
+"""Save Iris cubes to a CF-conforming store.
+
+.. z_reference:: iris.fileformats.cf.saver
+ :tags: topic_load_save
+
+ API reference
+
+Everything about writing a cube that does not depend on what it is written
+into. The CF data model is here - dimension naming, coordinate and bounds
+variables, cell methods, grid mappings, meshes, attribute compliance - and the
+storage calls all go through :class:`~iris.fileformats.cf.dataset.CFDataset`.
+
+A format supplies a subclass of :class:`CFSaver` holding its own vocabulary of
+storage keywords, and whatever of the four hooks it needs:
+``_create_string_variable``, ``_backend_write``, ``_flush_before_close`` and
+``_patch_dataset``. :mod:`iris.fileformats.netcdf.saver` is one such subclass.
+
+See : `CF Conventions <https://cfconventions.org/>`_.
+
+"""
+```
+
+Rename the class, and give it the generic constructor:
+
+```python
+class CFSaver:
+ """A manager for saving cubes to a CF-conforming store."""
+
+ def __init__(self, dataset, compute=True, *, owns_dataset=True):
+ """Manage saving cubes into an open store.
+
+ Parameters
+ ----------
+ dataset : :class:`~iris.fileformats.cf.dataset.CFDataset`
+ An open, writable store. A subclass opens it and passes it here.
+ compute : bool, default=True
+ If ``True``, delayed variable saves will be completed on exit from
+ the saver context, after the store is closed, equivalent to
+ :meth:`complete`.
+
+ If ``False``, the store is created and closed without writing the
+ data of variables whose source data was lazy. Those writes can be
+ completed later, see :meth:`delayed_completion`.
+ owns_dataset : bool, default=True
+ Whether leaving the context should close the store. ``False`` when
+ the caller supplied an already-open store and will close it itself.
+
+ """
+ # All persistent variables
+ #: CF name mapping with iris coordinates
+ self._name_coord_map = CFNameCoordMap()
+ #: Map of dimensions to characteristic coordinates with which they are identified
+ self._dim_names_and_coords = CFNameCoordMap()
+ #: List of grid mappings added to the file
+ self._coord_systems = []
+ #: A dictionary, listing dimension names and corresponding length
+ self._existing_dim = {}
+ #: A map from meshes to their actual file dimensions (names).
+ # NB: might not match those of the mesh, if they were 'incremented'.
+ self._mesh_dims = {}
+ #: A dictionary, mapping formula terms to owner cf variable name
+ self._formula_terms_cache = {}
+ #: Whether to complete delayed saves on exit.
+ self.compute = compute
+ # A list of delayed writes for lazy saving
+ # a list of couples (source, target).
+ self._delayed_writes = []
+
+ self._dataset = dataset
+ self._owns_dataset = owns_dataset
+ #: Target filepath
+ self.filepath = dataset.location
+```
+
+`__exit__` tests ownership through the new attribute:
+
+```python
+ def __exit__(self, type, value, traceback):
+ """Flush any buffered data to the store before closing."""
+ self._flush_before_close()
+ self._dataset.sync()
+ if self._owns_dataset:
+ # Only close if the saver opened it.
+ self._dataset.close()
+ # Complete after closing, if required
+ if self.compute:
+ self.complete()
+```
+
+Give the four hooks their base versions, beside the methods that call them:
+
+```python
+ def _create_string_variable(
+ self, cube_or_mesh, cf_name, element, element_dims, data
+ ):
+ """Create the variable that will hold this unicode string data.
+
+ Parameters
+ ----------
+ cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
+ The Cube or Mesh being saved.
+ cf_name : str
+ The name the variable is created with.
+ element : :class:`iris.coords._DimensionalMetadata`
+ The element being saved.
+ element_dims : list of str
+ The variable's dimension names. A subclass may append to it.
+ data : :class:`numpy.ndarray`
+ The string values, used for their dtype.
+
+ Returns
+ -------
+ :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+
+ Raises
+ ------
+ NotImplementedError
+ Always, here. How a store holds a string differs so much between
+ formats that there is no generic answer to give.
+
+ """
+ msg = (
+ f"Cannot save the string variable {cf_name!r}: "
+ f"{type(self).__name__} does not say how {self._dataset.format} "
+ "stores strings."
+ )
+ raise NotImplementedError(msg)
+
+ def _backend_write(self, data, cf_var) -> bool:
+ """Write the data here and now, if this backend handles it itself.
+
+ Parameters
+ ----------
+ data : array-like
+ The values to write.
+ cf_var : :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+ The variable to write them into.
+
+ Returns
+ -------
+ bool
+ Whether the data was written. ``False``, here: the direct and
+ delayed writes below cover every store with no special case.
+
+ """
+ return False
+
+ def _flush_before_close(self) -> None:
+ """Write anything that cannot be written once the store is closed.
+
+ Nothing, here. A store that cannot be reopened for a deferred write
+ overrides this and writes its lazy data while it is still open.
+
+ """
+
+ def _patch_dataset(self, cf_patch, profile, cf_var_cube) -> None:
+ """Hand the site-configured patch hook the objects it expects.
+
+ Parameters
+ ----------
+ cf_patch : callable
+ The hook, from ``iris.site_configuration``.
+ profile : object
+ Whatever the configured ``cf_profile`` returned for this cube.
+ cf_var_cube : :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+ The cube's data variable.
+
+ Raises
+ ------
+ NotImplementedError
+ Always, here. The hook has been documented as receiving netCDF4
+ objects since Iris 1.3.
+
+ """
+ msg = (
+ "The 'cf_patch' site configuration hook is documented as receiving "
+ f"netCDF4 objects, and {self._dataset.format} has none to give it."
+ )
+ raise NotImplementedError(msg)
+```
+
+And give `write` the generic signature (F9). Only the signature, the
+`compression_kwargs` line and the `_create_cf_data_variable` call change; the rest
+of the body is untouched:
+
+```python
+ def write(
+ self,
+ cube,
+ local_keys=None,
+ unlimited_dimensions=None,
+ packing=None,
+ fill_value=None,
+ compression_kwargs=None,
+ **encoding,
+ ):
+ """Write a cube into the store.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube`
+ The cube to save.
+ local_keys : iterable of str, optional
+ Cube attribute keys that become attributes of the data variable
+ rather than global attributes.
+
+ .. note::
+
+ Has no effect if :attr:`iris.FUTURE.save_split_attrs` is ``True``.
+ unlimited_dimensions : iterable of str and/or :class:`iris.coords.Coord`, optional
+ Coordinate names, or coordinates, whose dimensions are saved with
+ unlimited length. By default, none are.
+ packing : type or str or dict or list, optional
+ How to pack the data, if at all. See
+ :meth:`iris.fileformats.netcdf.saver.Saver.write`.
+ fill_value : optional
+ The value for the data variable's ``_FillValue`` attribute.
+ compression_kwargs : dict, optional
+ Storage keywords applied to *every* variable this cube creates, not
+ only its data variable. The store's own vocabulary.
+ **encoding
+ Storage keywords applied to the data variable alone. The store's
+ own vocabulary.
+
+ Returns
+ -------
+ None
+
+ """
+```
+
+```python
+ # Group the storage keywords that apply to every variable, as they are
+ # applied to other cube metadata as well as the cube data payload.
+ compression_kwargs = dict(compression_kwargs or {})
+```
+
+```python
+ # Create the associated cube CF-netCDF data variable.
+ data_encoding = dict(encoding)
+ data_encoding.update(compression_kwargs)
+ cf_var_cube = self._create_cf_data_variable(
+ cube,
+ cube_dimensions,
+ local_keys=local_keys,
+ packing=packing,
+ fill_value=fill_value,
+ **data_encoding,
+ )
+```
+
+Leave every other mention of `compression_kwargs` in the file alone. It is the
+name eight methods already pass it by, and the keyword means the same thing it
+did.
+
+- [ ] **Step 6: Cut `netcdf/saver.py` down to the netCDF saver**
+
+Delete everything the moved file keeps. What is left is: the module docstring, the
+imports, the logger, `Saver`, `save`, `save_mesh`. Most of the imports then have
+no reader — `cf_units`, the aux-factory classes, most of `iris.coords`. Do not
+hunt for them by eye: `ruff check` names every one, and Step 11 runs it. Then:
+
+Replace the module docstring's body, keeping its `z_reference`:
+
+```python
+"""Module to support the saving of Iris cubes to a NetCDF file.
+
+.. z_reference:: iris.fileformats.netcdf.saver
+ :tags: topic_load_save
+
+ API reference
+
+netCDF's half of saving: the file to write into, the keywords netCDF4 accepts,
+and the four things netCDF does differently from a generic CF store. The CF
+data model lives in :mod:`iris.fileformats.cf.saver`, which this builds on.
+
+See : `NetCDF User's Guide <https://docs.unidata.ucar.edu/nug/current/>`_
+and `netCDF4 python module <https://github.com/Unidata/netcdf4-python>`_.
+
+Also : `CF Conventions <https://cfconventions.org/>`_.
+
+"""
+```
+
+Import the generic half as a module, **not** by name:
+
+```python
+from iris.fileformats.cf import saver as cf_saver
+from iris.fileformats.cf.saver import CFSaver
+```
+
+`CF_CONVENTIONS_VERSION` is read twice by `save` and once by `save_mesh`. Write
+those three reads as `cf_saver.CF_CONVENTIONS_VERSION`. Do **not** import the bare
+name: a module `__getattr__` fires only for names that are not already module
+globals, so importing it here would make
+`iris.fileformats.netcdf.saver.CF_CONVENTIONS_VERSION` resolve silently and the
+deprecation would never be seen.
+
+`Saver` becomes a subclass. `__init__` keeps today's body, with the two lines at
+the end replaced by a call to the base:
+
+```python
+class Saver(CFSaver):
+ """A manager for saving NetCDF/NcZarr files."""
+
+ def __init__(self, filename, netcdf_format, compute=True):
+ """<unchanged: today's docstring, including the doctest>"""
+ if netcdf_format not in [
+ "NETCDF4",
+ "NETCDF4_CLASSIC",
+ "NETCDF3_CLASSIC",
+ "NETCDF3_64BIT",
+ ]:
+ raise ValueError("Unknown netCDF file format, got %r" % netcdf_format)
+
+ #: Whether the save target is an NCZarr URL.
+ self._is_nczarr = False
+ # A list of pre-close writes for NCZarr lazy saving.
+ # NCZarr cannot be reopened for deferred writes, so lazy data is
+ # written before __exit__ closes the file.
+ self._nczarr_writes = []
+
+ # Detect if we were passed a pre-opened dataset (or something like one)
+ self._to_open_dataset = hasattr(filename, "createVariable")
+ if self._to_open_dataset:
+ # We were passed a *dataset*, so we don't open (or close) one of our own.
+ if compute:
+ msg = (
+ "Cannot save to a user-provided dataset with 'compute=True'. "
+ "Please use 'compute=False' and complete delayed saving in the "
+ "calling code after the file is closed."
+ )
+ raise ValueError(msg)
+
+ # from_existing() handles the thread-safety wrapping.
+ dataset = NetCDFDataset.from_existing(filename)
+
+ # In this case the dataset gives a filepath, not the other way around.
+ filepath = dataset.location
+
+ else:
+ # Given a filepath string/path : create a dataset from that
+ try:
+ # Lazy import to avoid circular import overhead at module import-time.
+ from iris.io import _is_nczarr_fragment
+
+ self._is_nczarr = _is_nczarr_fragment(
+ urlsplit(str(filename)).fragment or None
+ )
+ if self._is_nczarr:
+ # NCZarr URLs contain a #mode= fragment; Path() strips it.
+ # Keep as a plain string and pass directly to the dataset.
+ filepath = str(filename)
+ else:
+ filepath = Path(filename).absolute()
+ dataset = NetCDFDataset(
+ filepath, mode="w", netcdf_format=netcdf_format
+ )
+ except RuntimeError:
+ if self._is_nczarr:
+ raise
+ dir_name = Path(filepath).parent
+ if not dir_name.is_dir():
+ msg = "No such file or directory: {}".format(dir_name)
+ raise IOError(msg)
+ if not os.access(dir_name, os.R_OK | os.W_OK):
+ msg = "Permission denied: {}".format(filepath)
+ raise IOError(msg)
+ else:
+ raise
+
+ super().__init__(
+ dataset, compute=compute, owns_dataset=not self._to_open_dataset
+ )
+
+ #: Target filepath. A Path for a local file, which is public API:
+ #: the base class has only the dataset's string form of it.
+ self.filepath = filepath
+
+ # One lock for the whole file, shared with every variable of it.
+ #: A per-file write lock to prevent dask attempting overlapping writes.
+ # N.B. the file-write-lock *type* actually depends on the dask scheduler type.
+ self.file_write_lock = dataset.write_lock
+```
+
+Two details in that body are not today's, and both are needed. `self.filepath` is
+assigned after `super().__init__`, because the base sets it from
+`dataset.location` and a local file's is a `Path`. The exception handler reads the
+local `filepath` rather than `self.filepath`, which is not set yet when the open
+fails.
+
+`write` keeps today's twelve keywords and its whole docstring, and hands the
+bundle to the base:
+
+```python
+ def write(
+ self,
+ cube,
+ local_keys=None,
+ unlimited_dimensions=None,
+ zlib=False,
+ complevel=4,
+ shuffle=True,
+ fletcher32=False,
+ contiguous=False,
+ chunksizes=None,
+ endian="native",
+ least_significant_digit=None,
+ packing=None,
+ fill_value=None,
+ ):
+ """<unchanged: today's docstring, every parameter and the Notes>"""
+ super().write(
+ cube,
+ local_keys=local_keys,
+ unlimited_dimensions=unlimited_dimensions,
+ packing=packing,
+ fill_value=fill_value,
+ # Applied to every variable the cube creates, not just its data.
+ compression_kwargs={
+ "complevel": complevel,
+ "fletcher32": fletcher32,
+ "shuffle": shuffle,
+ "zlib": zlib,
+ },
+ contiguous=contiguous,
+ chunksizes=chunksizes,
+ endian=endian,
+ least_significant_digit=least_significant_digit,
+ )
+```
+
+Keep the four hook methods from Task 3 exactly as that task left them, and add the
+alias machinery at the end of the imports:
+
+```python
+#: Names that moved to iris.fileformats.cf.saver, and are still reachable
+#: here. Spec section 4.8.
+_RELOCATED_NAMES = frozenset(
+ {
+ "CFNameCoordMap",
+ "CF_CONVENTIONS_VERSION",
+ "MESH_ELEMENTS",
+ "SPATIO_TEMPORAL_AXES",
+ }
+)
+
+
+def __getattr__(name):
+ """Forward a relocated name to :mod:`iris.fileformats.cf.saver`.
+
+ Parameters
+ ----------
+ name : str
+ The attribute being looked up. Python calls this only for names that
+ are not already module attributes.
+
+ Returns
+ -------
+ Any
+ The object of that name in :mod:`iris.fileformats.cf.saver` - the same
+ object, not a copy.
+
+ Raises
+ ------
+ AttributeError
+ If ``name`` is not one of the relocated names. Raised without warning,
+ so that a ``hasattr`` probe that comes back False stays quiet.
+
+ """
+ if name not in _RELOCATED_NAMES:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ warn_deprecated(
+ f"iris.fileformats.netcdf.saver.{name} has moved to "
+ f"iris.fileformats.cf.saver.{name}, because it is not specific to "
+ "netCDF. The name here will be removed in a future release.",
+ # warn_deprecated's default stacklevel=2 points at the line below,
+ # inside this function - PEP 562 means that line is a genuine call
+ # frame, not the user's code. One level further out reaches the
+ # line that actually read the attribute.
+ stacklevel=3,
+ )
+ return getattr(cf_saver, name)
+
+
+def __dir__():
+ """List module attributes, including the relocated names.
+
+ A module ``__getattr__`` forwards reads of the relocated names, but does
+ not make them discoverable: PEP 562 does not extend to ``dir()``,
+ :func:`inspect.getmembers`, or tab-completion, so this fills the gap.
+
+ Returns
+ -------
+ list of str
+ The real module globals, unioned with the relocated names.
+
+ """
+ return sorted(set(globals()) | _RELOCATED_NAMES)
+```
+
+Mark the deprecation in the four docstrings? No. The objects themselves are
+unchanged and live in `cf/saver.py`; what is deprecated is reaching them through
+this module, which the warning says. Spec §4.8's `.. deprecated:: 3.17` marker
+goes on the `__getattr__` docstring, as PR 3 put it on the loader's.
+
+- [ ] **Step 7: Give the netCDF package the same forward**
+
+`lib/iris/fileformats/netcdf/__init__.py` imports all four names eagerly, so
+users reach them as `iris.fileformats.netcdf.CFNameCoordMap`. Drop them from the
+import, and forward them from here too:
+
+```python
+from .loader import DEBUG, NetCDFDataProxy, load_cubes
+from .saver import Saver, save
+```
+
+Leave `__all__` as it is: the names still resolve, and listing them is what keeps
+them visible to anyone reading the module. Add, below `__all__`:
+
+```python
+def __getattr__(name):
+ """Forward a relocated name to :mod:`iris.fileformats.cf.saver`.
+
+ .. deprecated:: 3.17
+ Four saver names moved to :mod:`iris.fileformats.cf.saver`, because
+ they are not specific to netCDF. Reading them here still works, and
+ warns.
+
+ Parameters
+ ----------
+ name : str
+ The attribute being looked up.
+
+ Returns
+ -------
+ Any
+ The object of that name - the same object, not a copy.
+
+ Raises
+ ------
+ AttributeError
+ If ``name`` is not one of the relocated names.
+
+ """
+ from .saver import _RELOCATED_NAMES
+
+ if name not in _RELOCATED_NAMES:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ from iris.fileformats.cf import saver as cf_saver
+
+ warn_deprecated(
+ f"iris.fileformats.netcdf.{name} has moved to "
+ f"iris.fileformats.cf.saver.{name}, because it is not specific to "
+ "netCDF. The name here will be removed in a future release.",
+ stacklevel=3,
+ )
+ return getattr(cf_saver, name)
+
+
+def __dir__():
+ """List module attributes, including the relocated names.
+
+ Returns
+ -------
+ list of str
+ The real module globals, unioned with the relocated names.
+
+ """
+ from .saver import _RELOCATED_NAMES
+
+ return sorted(set(globals()) | _RELOCATED_NAMES)
+```
+
+`warn_deprecated` needs importing here: `from iris._deprecation import
+warn_deprecated`.
+
+- [ ] **Step 8: Run the relocation tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py -v
+```
+
+Expected: all pass. `TestNothingInsideIrisUsesTheOldNames` will fail until Task 5
+repoints `iris/cube.py` and `cf/loader.py`; if it is the only failure, carry on
+and let Task 5 clear it.
+
+- [ ] **Step 9: Run every saver test, then the whole fileformats suite**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/netcdf/saver/ -v
+pytest lib/iris/tests/unit/fileformats/ -q -n auto
+```
+
+Expected: all pass. Failures here are most likely in `test_Saver.py`, which mocks
+methods by name on the class, and in `test_save.py`, which asserts the keywords
+`save` passes to `Saver.write`.
+
+- [ ] **Step 10: Check that no netCDF survives in the generic module**
+
+```bash
+grep -nE "netcdf|netCDF|NETCDF|NetCDF|nczarr|NCZarr|zlib|fletcher32|endian|least_significant" \
+ lib/iris/fileformats/cf/saver.py
+```
+
+Expected: matches in prose only — docstrings and comments that describe CF, name
+the netCDF saver as the subclass that exists, or quote a CF convention. No import,
+no attribute, no literal in executable code. Anything else is a fifth netCDF leak;
+extract it as Task 3 did and say so in the pull request body.
+
+```bash
+grep -c "" lib/iris/fileformats/cf/saver.py lib/iris/fileformats/netcdf/saver.py
+```
+
+Expected: roughly 2400 and 800. Wildly different numbers mean one of the two cuts
+went wrong.
+
+- [ ] **Step 11: Run pre-commit, then commit**
+
+```bash
+pre-commit run --files \
+ lib/iris/fileformats/cf/saver.py \
+ lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/fileformats/netcdf/__init__.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py
+```
+
+`numpydoc-validation` is the hook most likely to complain, because both modules
+gained methods with new docstrings. `check-docs-page-metadata` will fail if the
+`z_reference` in `cf/saver.py` is not within the first 25 lines.
+
+```bash
+git add lib/iris/fileformats/cf/saver.py \
+ lib/iris/fileformats/netcdf/saver.py \
+ lib/iris/fileformats/netcdf/__init__.py \
+ lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py
+git commit -m "Relocate the CF saver to iris.fileformats.cf.saver
+
+Everything about writing a cube that does not depend on what it is
+written into now lives in iris.fileformats.cf.saver.CFSaver: the CF data
+model, and storage calls that go through CFDataset.
+
+iris.fileformats.netcdf.saver keeps what is netCDF's. The file to write
+into, the keywords netCDF4 accepts, and the four things netCDF does
+differently from a generic store. Saver, save and save_mesh stay exactly
+where they are, and Saver.write keeps its signature.
+
+Four names that are not netCDF's - CFNameCoordMap,
+CF_CONVENTIONS_VERSION, MESH_ELEMENTS and SPATIO_TEMPORAL_AXES - are
+reachable from both old homes, warning on use.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 5: Repoint everything that read the saver's old home
+
+Six modules import names that moved. Two are library code, four are tests. The
+library ones matter most: a reader left pointing at the old home makes an
+ordinary save warn at a user who has done nothing wrong.
+
+**Files:**
+- Modify: `lib/iris/fileformats/cf/loader.py:18, 185-187, 367-369`
+- Modify: `lib/iris/cube.py:1081`
+- Modify: `lib/iris/tests/unit/cube/test_CubeAttrsDict.py:14`
+- Modify: `lib/iris/tests/integration/test_netcdf__loadsaveattrs.py:51, 69-70`
+- Modify: `lib/iris/tests/unit/fileformats/netcdf/saver/test_save.py:13`
+- Modify: `lib/iris/tests/integration/netcdf/test_attributes.py:17`
+- Test: `lib/iris/tests/unit/fileformats/cf/test__imports.py` (new)
+
+**Interfaces:**
+- Consumes: `iris.fileformats.cf.saver` from Task 4.
+- Produces: nothing new. This is the task that makes
+ `TestNothingInsideIrisUsesTheOldNames` from Task 4 pass.
+
+- [ ] **Step 1: Write the import-order test**
+
+Create `lib/iris/tests/unit/fileformats/cf/test__imports.py`:
+
+```python
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Every module in the CF and netCDF packages can be imported first.
+
+The two packages import each other: the CF reader builds a netCDF dataset, and
+the netCDF saver builds on the CF saver. Python resolves that as long as no
+module needs a name from a module that is still part-way through its own
+imports - which depends on where the interpreter starts. A test suite only
+ever starts in one place, so each case gets a subprocess of its own.
+"""
+
+import subprocess
+import sys
+
+import pytest
+
+MODULES = [
+ "iris.fileformats.cf",
+ "iris.fileformats.cf.dataset",
+ "iris.fileformats.cf.loader",
+ "iris.fileformats.cf.saver",
+ "iris.fileformats.cf._dask_locks",
+ "iris.fileformats.netcdf",
+ "iris.fileformats.netcdf._dataset",
+ "iris.fileformats.netcdf.loader",
+ "iris.fileformats.netcdf.saver",
+]
+
+
+@pytest.mark.parametrize("module", MODULES)
+def test_importing_it_first_works(module):
+ completed = subprocess.run(
+ [sys.executable, "-c", f"import {module}"],
+ capture_output=True,
+ text=True,
+ )
+ assert completed.returncode == 0, completed.stderr
+```
+
+- [ ] **Step 2: Run it, and watch it pass**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/test__imports.py -v
+```
+
+Expected: 9 passed. It passes before Step 3 and must still pass after it. Nine
+subprocesses each importing Iris takes a few seconds; that is the price of
+testing import order at all.
+
+- [ ] **Step 3: Make the loader's two imports module-scope**
+
+In `lib/iris/fileformats/cf/loader.py`, delete both deferred imports and their
+`TODO` comments, and import once at the top of the module with the other
+`iris.fileformats` imports:
+
+```python
+from iris.fileformats.cf.saver import _CF_ATTRS, CFSaver
+```
+
+At the first site, the body keeps using the bare name:
+
+```python
+ def attribute_predicate(item):
+ return item[0] not in _CF_ATTRS
+```
+
+At the second, `Saver._DATALESS_ATTRNAME` becomes `CFSaver._DATALESS_ATTRNAME`.
+The attribute is inherited by the netCDF subclass, so what is written to a file
+and what is read back are still the same string.
+
+The module docstring, around line 18, says the module reaches back into
+`iris.fileformats.netcdf` for `saver._CF_ATTRS` and `Saver._DATALESS_ATTRNAME`.
+That sentence stops being true here. Rewrite the paragraph it sits in to name
+only whatever reach-backs survive; if none do, delete it.
+
+- [ ] **Step 4: Repoint the one other library reader**
+
+In `lib/iris/cube.py`, the deferred import inside `CubeAttrsDict`:
+
+```python
+ from iris.fileformats.cf.saver import _CF_GLOBAL_ATTRS
+```
+
+Leave it deferred. It is deferred to keep `iris.cube` cheap to import, not
+because of where the saver lived.
+
+- [ ] **Step 5: Run the import-order test and the no-deprecation tests**
+
+```bash
+pytest lib/iris/tests/unit/fileformats/cf/test__imports.py -v
+pytest lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py -v
+```
+
+Expected: both green. `TestNothingInsideIrisUsesTheOldNames` was failing at the
+end of Task 4 and passes now. If an import-order case fails, read its `stderr`:
+an `ImportError` naming `CFSaver` or `_CF_ATTRS` means the module-scope import in
+Step 3 has to go back to being deferred, and F7 is wrong.
+
+- [ ] **Step 6: Repoint the four test modules**
+
+None of these change what is tested. They stop the suite tripping its own
+deprecation.
+
+`lib/iris/tests/unit/cube/test_CubeAttrsDict.py`:
+
+```python
+from iris.fileformats.cf.saver import _CF_GLOBAL_ATTRS
+```
+
+`lib/iris/tests/integration/test_netcdf__loadsaveattrs.py`, three places, where
+the module is currently reached as `iris.fileformats.netcdf.saver`:
+
+```python
+_GLOBAL_TEST_ATTRS = sorted(iris.fileformats.cf.saver._CF_GLOBAL_ATTRS)
+```
+
+```python
+ iris.fileformats.cf.saver._CF_DATA_ATTRS
+ + iris.fileformats.cf.saver._UKMO_DATA_ATTRS
+```
+
+Add `import iris.fileformats.cf.saver` to that module if it is not already there.
+
+`lib/iris/tests/unit/fileformats/netcdf/saver/test_save.py`:
+
+```python
+from iris.fileformats.cf.saver import CF_CONVENTIONS_VERSION
+from iris.fileformats.netcdf import Saver, _thread_safe_nc, save
+```
+
+`lib/iris/tests/integration/netcdf/test_attributes.py`:
+
+```python
+from iris.fileformats.cf.saver import CF_CONVENTIONS_VERSION
+```
+
+- [ ] **Step 7: Find anything the list missed**
+
+```bash
+grep -rn "netcdf\.saver\|netcdf import saver\|from iris.fileformats.netcdf import" \
+ lib benchmarks docs --include=*.py --include=*.rst \
+ | grep -v "docs/src/whatsnew"
+```
+
+Expected matches, all of which are correct and stay: `iris/io/__init__.py`
+importing `save`, `iris/mesh/__init__.py` importing `save_mesh`, the user-manual
+references to `netcdf.saver.save`, and the saver's own tests. Anything importing
+one of the four relocated names from the netCDF package is a miss: repoint it.
+
+The historical `docs/src/whatsnew/` entries are excluded on purpose. They are a
+record of what was true when they were written.
+
+- [ ] **Step 8: Run the full test suite**
+
+```bash
+pytest lib/iris -q -n auto
+```
+
+Expected: the same pass and skip counts as `brownfield`. Record the numbers; the
+pull request body quotes them. If anything fails, it fails here rather than in
+CI, which is the point of running it before Task 6.
+
+- [ ] **Step 9: Run pre-commit over everything**
+
+```bash
+pre-commit run --all-files
+```
+
+Expected: clean. Hooks rewrite in place, so re-stage and re-run until it is.
+
+- [ ] **Step 10: Commit**
+
+```bash
+git add -A lib
+git commit -m "Point every reader at the CF saver's new home
+
+The CF loader's two reaches into the netCDF saver were deferred with a
+comment saying they would become plain imports once the saver moved. It
+has, so they have.
+
+Four test modules read a relocated name through the netCDF package,
+which now warns. They read it from iris.fileformats.cf.saver instead, so
+that a full test run raises no deprecation of its own.
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 6: Record it, and open the pull request
+
+The code is done after Task 5. This task is the paperwork that makes it
+reviewable: the spec's own record of what was decided, and a pull request body
+that aims the reviewer at the three places worth their attention.
+
+**Files:**
+- Modify: `docs/superpowers/specs/2026-09-21-zarr-io-design.md` §5, §12.1, §12.3, §12.4, §12.6
+
+**Interfaces:**
+- Consumes: everything above.
+- Produces: the pull request.
+
+- [ ] **Step 1: Say what the relocation leaves behind, in §5**
+
+The PR 5 subsection says the saver moves "same shape as PR 3" and does not say
+what shape that turned out to be. PR 6 needs to know. Add, after the first
+paragraph:
+
+```markdown
+The generic half is `CFSaver`, built from an open `CFDataset`.
+`iris.fileformats.netcdf.saver.Saver` subclasses it, supplying the netCDF
+vocabulary of storage keywords and four hooks: `_create_string_variable`,
+because netCDF stores unicode as bytes over a string dimension;
+`_backend_write`, which claims the writes netCDF does itself, to an emulating
+variable or to an NCZarr target that cannot be reopened; `_flush_before_close`,
+which performs the second of those before the file closes; and `_patch_dataset`,
+which hands `iris.site_configuration`'s `cf_patch` hook the unwrapped netCDF4
+objects it has been documented as receiving since Iris 1.3. A Zarr saver
+implements the first and inherits the rest.
+```
+
+- [ ] **Step 2: Update the pull request table in §12.1**
+
+Set PR 5's state to `In review` and add its number and link, in the form the
+other rows use. Correct PR 2.5's row at the same time: it says `In review` and
+[#7333](https://github.com/SciTools/iris/pull/7333) merged on 2026-10-14. Leave
+every other row alone.
+
+- [ ] **Step 3: Add the open question to §12.3**
+
+Append a row, in the table's existing column order:
+
+```markdown
+| Q10 | Saving int64 data to `NETCDF3_64BIT` fails inside the C library, because netCDF4 reports that format as `NETCDF3_64BIT_OFFSET` and the saver's narrowing list does not have that string **[verified]** | Preserved, not fixed: PR 5 is a relocation and may not change behaviour. `NetCDFDataset.supports_dtype` answers from the same three strings the saver tested, with a test pinning the failure so it is not mistaken for a regression | #NNNN | Nothing |
+```
+
+Replace `#NNNN` with the issue Task 1 raised, and link it as the other rows link
+theirs.
+
+- [ ] **Step 4: Add the decision-log entry to §12.4**
+
+Append, at the end, in the log's existing shape:
+
+```markdown
+**2026-10-14 — PR 5, the CF saver relocation**
+
+- The generic saver is `CFSaver`, constructed from an open `CFDataset` rather
+ than from a filename. Each format's `Saver` opens its own store and passes it
+ down.
+- Four hooks are what a format supplies beyond the dataset:
+ `_create_string_variable`, `_backend_write`, `_flush_before_close` and
+ `_patch_dataset`. Two of them raise on the base class rather than guessing a
+ generic answer, because storing a string and patching a netCDF4 object have no
+ generic answer to guess.
+- `write()` is the one method whose signature differs between the two classes.
+ The netCDF one keeps its twelve documented keywords, which `save()` passes by
+ position; the generic one takes `**encoding` plus a `compression_kwargs` dict
+ for what applies to every variable rather than only the data variable.
+- The saver's unit tests stay under `tests/unit/fileformats/netcdf/`. They drive
+ the generic machinery through the only subclass that exists; a shared body is
+ worth factoring out when PR 6 adds the second one.
+- `supports_dtype` answers the capability question, and the saver keeps the
+ question of which dtypes are worth narrowing to int32. Splitting it that way
+ lets a backend answer honestly about dtypes the narrowing never sees.
+```
+
+- [ ] **Step 5: Add the document-history row to §12.6**
+
+```markdown
+| 2026-10-14 | PR 5 raised. §5 records the shape the saver relocation takes - `CFSaver` and the four hooks - and §12.3 records the `NETCDF3_64BIT` defect found while doing it. |
+```
+
+- [ ] **Step 6: Commit the spec, and push**
+
+```bash
+git add docs/superpowers/specs/2026-09-21-zarr-io-design.md
+git commit -m "Record the CF saver relocation in the design document
+
+Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>"
+git push -u origin zarr-io-pr-5
+```
+
+- [ ] **Step 7: Open the pull request**
+
+Labels `Agentic` and `Type: Feature Branch`, base `brownfield`. Write the body to
+a file first; it is long and contains code.
+
+```bash
+gh pr create --repo SciTools/iris --base brownfield \
+ --title "Relocate the CF saver to iris.fileformats.cf.saver" \
+ --label Agentic --label "Type: Feature Branch" \
+ --body-file /tmp/iris-pr5-body.md
+```
+
+The body, one paragraph per line, no hard wrapping:
+
+> (posted by Claude Opus)
+>
+> Moves the format-agnostic half of `iris.fileformats.netcdf.saver` into `iris.fileformats.cf.saver`, so that Zarr saving can be a subclass rather than a second copy. No behaviour changes: every public name still resolves, `Saver`, `save` and `save_mesh` stay exactly where they are, and the whole existing suite passes unchanged. If I have got this wrong, the failure mode is a method that quietly stopped being called, so that is what I would look at first.
+>
+> Part of #6977. The changelog fragment is deferred to the merge-back into `main`, with the rest of the programme's fragments, as agreed in the design document's section 5.
+>
+> **Where I would look first.** `Saver.write` is the one signature that differs between the two classes: netCDF keeps its twelve documented keywords and the generic one takes `**encoding`, so a keyword that stopped being forwarded would be invisible to the type checker and quiet at runtime. `Saver.__init__` now builds the dataset and hands it to the base class, which is where `filepath` and the file write lock are set, and `filepath` is public API that tests compare to a `Path`. The four hooks are the whole of the netCDF-specific behaviour, so anything netCDF-specific I failed to notice is now sitting in a module that claims to be format-agnostic.
+>
+> **What is in `cf/saver.py`.** The CF data model: dimension naming, coordinate, bounds and label variables, cell methods, grid mappings, meshes, attribute compliance, packing, and the delayed-write machinery. Every storage call goes through `CFDataset`.
+>
+> **What stays in `netcdf/saver.py`.** The file to open, the keywords netCDF4 accepts, and four things netCDF does differently: it stores unicode as bytes over a string dimension, it writes to emulating variables for the ncdata bridge, it cannot reopen an NCZarr target so it flushes lazy writes before closing, and it hands `cf_patch` the unwrapped netCDF4 objects that hook has been documented as receiving since Iris 1.3.
+>
+> **Two names on the dataset interface.** `CFDataset.supports_dtype` replaces the saver's reading of the open file's netCDF format string, which would otherwise have become generic code naming netCDF formats. `CFDataset.format` supplies the format name the resulting error message quotes. The Zarr loading pull request adds `format` as well, with identical text, so whichever of the two lands second resolves that conflict by taking either side.
+>
+> **The dask lock toolkit moves too.** `_dask_locks` imports `threading` and four `dask` modules and nothing else, so it was netCDF's only by where it sat. It is now `iris.fileformats.cf._dask_locks`, with a docstring that no longer describes the netCDF saver as its only caller.
+>
+> **Four names are reachable from two places, and warn from one.** `CFNameCoordMap`, `CF_CONVENTIONS_VERSION`, `MESH_ELEMENTS` and `SPATIO_TEMPORAL_AXES` are not specific to netCDF, so they live in `iris.fileformats.cf.saver` now. Reading them from `iris.fileformats.netcdf` or `iris.fileformats.netcdf.saver` still returns the same objects, and warns.
+>
+> **A defect found and deliberately kept.** Saving int64 data to `NETCDF3_64BIT` fails inside the netCDF C library, because netCDF4 reports that format as `NETCDF3_64BIT_OFFSET` and the saver's list of formats that cannot hold int64 does not contain that string. Fixing it would be a behaviour change in a pull request that promises none, so `supports_dtype` reproduces the same gap and a test pins the failure. Raised separately.
+>
+> **Where the new module's tests are.** Still under `tests/unit/fileformats/netcdf/saver/`. They drive the generic machinery by opening netCDF files, because a netCDF file is the only store there is until Zarr saving lands. Moving them would put tests that open netCDF files inside the CF package. A shared body, in the shape of the existing `cf/dataset/contract.py`, is worth factoring out when there are two savers to run it against.
+>
+> **What I rejected.** A pure `git mv` with a re-exporting shim, which would have left netCDF4-specific code inside a module named for the format-agnostic layer and made the Zarr saving pull request do the teasing-apart as a refactor of generic code. Splitting `cf/saver.py` further, which it deserves at about 2400 lines, but which is its own pull request with its own review rather than a rider on a relocation. Fixing #7291 and #7329, both of which this code touches and neither of which belongs in a move.
+
+- [ ] **Step 8: Record the number, and check the diff reads as a move**
+
+Put the pull request number into §12.1 as Step 2 describes, commit and push that
+edit.
+
+```bash
+git diff --stat upstream/brownfield...zarr-io-pr-5 | tail -20
+```
+
+Expected: `lib/iris/fileformats/{netcdf => cf}/saver.py` shown as a rename with
+modifications. If git shows a separate delete and add instead, the two halves
+have drifted far enough apart that rename detection gave up. That is cosmetic,
+but say so in the body so reviewers know to read the two files side by side.
+
+- [ ] **Step 9: Watch CI, and fix what it finds**
+
+```bash
+gh pr checks --watch
+```
+
+The jobs that most often disagree with a local run are the oldest and newest
+Python versions, and the documentation build, which is the only thing that
+renders the new module's docstrings.
+
+---
+
+## Self-review notes
+
+Three things a reviewer of this plan should know were checked rather than
+assumed.
+
+**Every spec requirement for PR 5 has a task.** `git mv` of the saver, Task 4.
+`git mv` of `_dask_locks` with a rewritten docstring, Task 2. `CFDataset.format`
+and `supports_dtype` used at the two format-string reads, Task 1. The four
+deprecation aliases of §4.8, Task 4. Every in-repository reference repointed,
+Task 5. No behaviour change, pinned by the characterisation tests in Tasks 1 and
+3 and by the full-suite run in Task 5 Step 8.
+
+**The names used across tasks are the ones the code will have.** `CFSaver`,
+`_create_string_variable`, `_backend_write`, `_flush_before_close`,
+`_patch_dataset`, `supports_dtype`, `format`, `owns_dataset`,
+`compression_kwargs`, `_RELOCATED_NAMES`. Task 3 creates the four hooks on the
+netCDF class and Task 4 gives each a base version with the same name and the same
+parameters.
+
+**Two things this plan asserts were run, not reasoned about.** The
+`NETCDF3_64BIT` failure, by saving an int64 cube to each of the four formats. The
+import entanglement between the two packages, by importing
+`iris.fileformats.cf` in a fresh interpreter and finding
+`iris.fileformats.netcdf.saver` already in `sys.modules`. Both are recorded in §3
+with the command that produced them.
diff --git a/docs/superpowers/specs/2026-09-21-zarr-io-design.md b/docs/superpowers/specs/2026-09-21-zarr-io-design.md
new file mode 100644
index 0000000000..298e80e62c
--- /dev/null
+++ b/docs/superpowers/specs/2026-09-21-zarr-io-design.md
@@ -0,0 +1,2902 @@
+# Native Zarr I/O for Iris
+
+> **This is a living document.** §12 is the single progress record for the
+> programme — pull request states, open questions and the decision log. Update
+> §12 as work lands; do not track status anywhere else.
+
+| | |
+|---|---|
+| **Phase** | In delivery |
+| **Progress** | PRs 1, 2, 2.5, 3 and 5 merged into `brownfield`; PR 4 in review ([#7336](https://github.com/SciTools/iris/pull/7336)); merge-back not started — see §12.1 |
+| **Next action** | PR 4's review, then PR 6 — see §12.1 |
+| **Blocked on** | Nothing |
+| **Branch** | `zarr-io-design` on `bjlittle/iris`, targeting `SciTools/iris:brownfield` |
+| **Delivery** | Nine pull requests into `brownfield`, then one merge-back into `main` — §5 |
+| **Resolves** | SciTools/iris#6977, #6979, #6980 — at merge-back, not before |
+
+**Created:** 2026-09-21 · **Last updated:** 2026-10-20 (see §12.6)
+**Baseline:** `brownfield` at `b0ab90b98`, Iris `3.17.0.dev4`
+**Verified against:** zarr-python 3.4.0, numcodecs 0.16.5, Python 3.13,
+dask 2026.7.1, xarray 2026.7.0
+
+Line references in this document are accurate as of the baseline commit.
+Claims marked **[verified]** were checked by running code against the versions
+above; everything else is read from source or specification.
+
+---
+
+## 1. Summary
+
+Iris can already read and write Zarr, but only through the netCDF-c NCZarr
+driver, which implements the **Zarr version 2** specification and writes scalar
+variables as length-one arrays. This document specifies **native Zarr support**
+built on zarr-python: reading Zarr version 2 and version 3 stores, and writing
+version 3 stores.
+
+The work is structured as nine pull requests against the `brownfield` branch.
+Four of them are behaviour-preserving refactors that promote
+`iris.fileformats.cf` to a package and extract the format-agnostic CF machinery
+out of `iris.fileformats.netcdf`; four add user-visible Zarr capability, or
+change behaviour, on top; the last changes prose only. No pull request both
+moves code and changes behaviour.
+
+The central design move is a `CFDataset` abstraction: a narrow, explicit
+interface describing what a CF-conforming array store must provide, with one
+implementation per backend. The CF variable classes, the loader and the saver
+are rewritten against that interface instead of against the netCDF4 Python API.
+
+Every rule in §4 is derived from the Zarr and CF specifications. Real published
+stores are cited as evidence that a code path will be exercised, and where such
+a store is non-conforming it is named as a malformation and handled with a
+warning — never by bending the reader to fit one publisher's files.
+
+Where the two specifications are silent and an established *de-facto*
+convention has filled the gap, that is a third case, distinct from both a
+specification rule and a publisher's mistake — and misreading it as the latter
+is its own failure mode (§4.4, `_FillValue`). Such a convention is adopted only
+when it is named, its source cited, and the cost of ignoring it stated. The
+test of one is whether an independent reader depends on it, not whether a file
+happens to contain it.
+
+---
+
+## 2. What exists today
+
+### 2.1 The CF layer reaches straight into netCDF4
+
+`lib/iris/fileformats/cf.py` (1721 lines) classifies the variables in a file
+into `CFDataVariable`, `CFCoordinateVariable`, `CFGridMappingVariable` and so
+on. It is nominally format-agnostic, but in practice it calls netCDF4 directly:
+
+- `CFVariable.__init__` (cf.py:92-118) calls `data.ncattrs()` and
+ `data.group().filepath()`.
+- `CFVariable.__getattr__` (cf.py:198-206) forwards every unknown attribute to
+ the underlying netCDF4 variable, conflating **properties** (`shape`, `dtype`,
+ `dimensions`) with **CF attributes** (`standard_name`, `units`). It caches the
+ result onto the instance, so the touch-tracking used by `cf_attrs_used()`
+ registers only the first access.
+- `CFReader.__init__` (cf.py:1336) branches on `self._dataset.file_format`, and
+ (cf.py:1356) calls `set_auto_chartostring(False)`.
+
+Across `cf.py`, `_nc_load_rules/` and `netcdf/`, there are 86 `getattr(cf_*`
+or `hasattr(cf_*` call sites and 57 direct uses of netCDF-only APIs
+(`ncattrs`, `getncattr`, `setncattr`, `chunking()`, `file_format`).
+
+### 2.2 The loader and saver are generic code in a netCDF-shaped box
+
+`netcdf/loader.py` (979 lines) and `netcdf/saver.py` (3190 lines) are mostly CF
+logic. The genuinely netCDF-specific parts are small and identifiable:
+
+| Module | netCDF-only content |
+|---|---|
+| `loader.py` | `_get_cf_var_data` chunking and proxy selection; VLEN handling; `_bytecoding_datasets` string machinery |
+| `saver.py` | dataset open/close; `_setncattr`; `_create_cf_dimensions`; `_create_generic_cf_array_var`; write proxies; `_dask_locks` |
+
+Everything else — `CFNameCoordMap`, `_get_dim_names`, `_add_aux_coords`,
+`_add_cell_measures`, `_add_aux_factories`, `_create_cf_grid_mapping`, the CF
+constant tables, `ChunkControl` — is format-agnostic and currently
+unreachable from any other format.
+
+### 2.3 The load and save chain assumes a single seekable file
+
+A Zarr store is a **directory**, or a URL. Both break the existing chain:
+
+- `iris.io.load_files` (io/__init__.py:241) does `open(fn, "rb")` on every path,
+ which raises `IsADirectoryError` for a store.
+- `iris.io.expand_filespecs` (io/__init__.py:196-199) calls
+ `Path(fn.removeprefix("//")).expanduser().absolute()` on every spec, which
+ mangles `s3://bucket/data.zarr` into `/<cwd>/s3:/bucket/data.zarr`.
+- `iris.loading._generate_cubes` (loading.py:110) raises
+ `ValueError("Iris cannot handle the URI scheme: %s")` for any scheme that is
+ not `file`, `http`, `https`, `nczarr` or `data`.
+- `iris.io.save` (io/__init__.py:530) sends a `CubeList` to a per-cube append
+ loop unless `"iris.fileformats.netcdf" in saver.__module__`.
+- `iris.mesh.load_meshes` carries a second copy of the same scheme dispatch in
+ `netcdf/ugrid_load.py:156-183`.
+
+### 2.4 Zarr facts that shape the design
+
+All **[verified]** against zarr-python 3.4.0:
+
+- Version 3 arrays carry `metadata.dimension_names` as a tuple. Version 2
+ arrays have no such field; zarr-python raises `ValueError("Zarr format 2
+ arrays do not support dimension names.")` if you try. Version 2 stores use
+ the xarray convention of an `_ARRAY_DIMENSIONS` attribute holding a list.
+- Scalar arrays have `shape == ()` and `dimension_names is None`.
+- Attributes are JSON. NumPy scalars, NumPy arrays, `bytes` and `numpy.int64`
+ all raise `TypeError: Object of type ... is not JSON serializable`. Python
+ floats, ints, strings, lists and `numpy.str_` are accepted. A failed
+ assignment leaves the in-memory attributes mapping dirty, so every later
+ write to that mapping fails with the **first** error, not its own. Iris never
+ reaches that state, because §4.5 converts every value *before* handing it to
+ zarr, so a value zarr would reject is caught by Iris and reported against its
+ own attribute name. The dirty mapping is a zarr-python defect worth reporting
+ upstream rather than a constraint Iris has to design around.
+- `float("nan")` is accepted and round-trips, written as the bare token `NaN`,
+ which is not valid strict JSON. Python's own `json` module writes and reads
+ that token by default — `allow_nan=True` on both `dump` and `load`, and
+ turning it off takes a deliberate argument — so the round trip holds for any
+ Python reader. It is a reader that is strict, or not Python, that would
+ reject it; §6's cross-reader conformance test is what covers that.
+- Reads return a plain `numpy.ndarray`. Zarr applies **no** masking and **no**
+ `scale_factor`/`add_offset` handling; netCDF4 does both automatically.
+- `zarr.Array` is picklable, and `dask.array.from_zarr` produces an unmasked
+ `numpy.ndarray` dask meta.
+- `zarr.consolidate_metadata` warns that consolidated metadata is not part of
+ the version 3 specification.
+
+Further facts come from the Zarr *specifications* rather than from
+zarr-python, and they set the boundary between what Iris must support and what
+Iris must merely defend against:
+
+- An attribute value is **an arbitrary JSON literal**. Nested objects and
+ arrays are legal Zarr. CF's attribute model is flat scalars, strings and
+ one-dimensional numeric vectors, so any conforming Zarr store may carry
+ attributes that CF cannot express.
+- **`dimension_names` is optional, its entries may be null, and a name carries
+ no guarantee outside its own array.** The version 3 core specification says
+ the field "must be an array of strings or null objects with the same length
+ as `shape`", that "an unnamed dimension is indicated by the null object",
+ that "it is recommended but not required that all non-null dimension names
+ are distinct" within one array, and that it "does not place any restrictions
+ on the use of the same dimension name across multiple arrays within the same
+ Zarr hierarchy". Every one of those is a case a conforming store may present,
+ so none of them may be an error. §4.4 sets out the contract that follows.
+- An array's `fill_value` is a **required** version 3 metadata field. For a
+ float the permitted encodings are a JSON number, `"NaN"`, `"Infinity"`,
+ `"-Infinity"`, or the hex string form `"0xYYYYYYYY"`; base64 is not among
+ them. CF's `_FillValue` is an *optional attribute*. These are two different
+ things, and the design reconciles them rather than conflating them.
+- **`fill_value` says what unwritten storage reads as. It does not say the
+ value is missing.** The two are easy to conflate and the conflation is
+ destructive: zarr-python defaults `fill_value` to **zero** — `np.int16(0)`
+ for `int16`, `np.float32(0.0)` for `float32`, on both version 2 and version
+ 3 **[verified]**. Treating it as a CF missing-value sentinel masks every
+ valid zero in an array that simply never specified one. And because the
+ field is *required* on version 3, there is no "array without a usable
+ `fill_value`" case for a CF attribute to fall back into. §4.4 sets the
+ precedence accordingly.
+
+---
+
+## 3. Constraints and decisions taken
+
+These were agreed before design and are not revisited here.
+
+| Decision | Choice |
+|---|---|
+| Dependency | `zarr` is **optional** and lazily imported. Absent, Zarr tests skip and Zarr use raises a clear `ImportError`. |
+| Module layout | Extract the format-agnostic CF machinery out of `fileformats/netcdf/` into generic siblings. |
+| Write support | **Version 3 only**, but the saver must be shaped so version 2 write is a later, small addition. |
+| Read support | **Version 2 and version 3.** |
+| Deprecation | Relocated public names keep working and warn on use. |
+| Remote stores | URLs are passed through to zarr-python. Iris adds no dependency and handles no credentials. |
+| Groups | Root group by default; a `group=` keyword selects another, on both load and save. |
+| Real-world data | NOAA GFS from dynamical.org, and ESA EOPF Sentinel Zarr samples. |
+
+One further constraint comes from the repository's own agent rules
+(`lib/iris/AGENTS.md`): identifiers are never constructed at runtime, because
+exact text search is how both agents and newcomers find code. The design
+honours this; where it cannot, it says so.
+
+---
+
+## 4. Design
+
+### 4.1 Module layout
+
+`iris.fileformats.cf` becomes a **package**. It is where all format-agnostic
+CF machinery ends up, and a flat module cannot hold a dataset abstraction, a
+loader and a saver alongside 1721 lines of variable classification.
+
+```
+lib/iris/fileformats/
+ cf/
+ __init__.py # public surface: re-exports + __all__, nothing else
+ dataset.py # NEW CFDataset / CFDatasetVariable
+ loader.py # NEW generic CF -> Cube (from netcdf/loader.py)
+ saver.py # NEW generic Cube -> CF (from netcdf/saver.py)
+ _variables.py # every CFVariable subclass, classic and UGRID (was cf.py)
+ _group.py # CFGroup (was cf.py)
+ _reader.py # CFReader (was cf.py)
+ netcdf/
+ _dataset.py # NEW NetCDFDataset, NetCDFDatasetVariable
+ loader.py # netCDF load_cubes + deprecating aliases
+ saver.py # netCDF Saver, save, save_mesh
+ ... # _thread_safe_nc, _bytecoding_datasets, _dask_locks unchanged
+ zarr/
+ __init__.py # NEW public: load_cubes, save, Saver
+ _dataset.py # NEW ZarrDataset, ZarrDatasetVariable
+ _decode.py # NEW CF masking and unpacking that netCDF4 does for free
+ loader.py # NEW
+ saver.py # NEW
+```
+
+Every public name that `iris.fileformats.cf` exports today is re-exported from
+`__init__.py`, so `from iris.fileformats.cf import CFReader` is unaffected and
+no deprecation is needed for the move itself. This is the shape `iris.mesh`
+already has: a 36-line `__init__.py` over `components.py` and `utils.py`.
+
+The private split cuts at the three seams already in `cf.py`, each of which
+is a different kind of thing: the variable classifiers (cf.py:78-1116), the
+`CFGroup` mapping that collects them (cf.py:1117-1280) and the `CFReader`
+that drives the whole thing (cf.py:1281-1721). That gives `_variables.py` at
+roughly 1120 lines, `_group.py` at 180 and `_reader.py` at 460.
+
+`_variables.py` sits just over the ~1000-line aim in `lib/iris/AGENTS.md`,
+and stays there deliberately. Do not split the three `CFUGrid*` classes into a
+fourth module to get under the number; that is a line count in search of a
+rationale. The UGRID classes are not a separate
+concern from the classic ones — they are `CFVariable` subclasses with the
+same shape (`cf_identity`, `cf_identities`, a classmethod `identify`), doing
+the same job of classifying a CF-netCDF variable by its attributes.
+`CFReader._variable_types` lists all ten in one tuple (cf.py:1291-1302);
+`CFGroup.non_data_variable_names` enumerates `connectivities`, `ugrid_coords`
+and `meshes` alongside `bounds`, `labels` and `cell_measures`; and the
+module-private `_is_str_dtype` (cf.py:78) is called from both sets, so the
+split would export a private helper across a module boundary to buy nothing.
+The `iris.mesh` analogy does not hold either: that package holds cube-level
+data-model objects, whereas these are variable classifiers doing the work of
+`CFAuxiliaryCoordinateVariable`. `lib/iris/AGENTS.md` settles it — *"Cohesion
+wins. Do not fragment into micro-modules purely to shrink files."*
+
+The re-export layer in `__init__.py` is the ordinary Iris pattern, not a
+concession. `iris.mesh` re-exports from four modules, two of them in a
+different subtree entirely (`iris.fileformats.netcdf.saver`,
+`iris.fileformats.netcdf.ugrid_load`), so that users import `MeshXY` and
+`save_mesh` from one place. Surfacing objects where users expect them is a
+genuine convenience, and it is what leaves the private file layout free to
+change. `lib/iris/AGENTS.md` says so explicitly; its "do not add indirection"
+rule governs implementation layers, not API surface.
+
+The re-exports are verbatim and static — `from ._reader import CFReader`, name
+for name, gathered into `__all__` — so `class CFReader` still finds the
+definition in one grep. No renaming, no conditional imports, no logic in
+`__init__.py`.
+
+`_variables.py`, `_group.py` and `_reader.py` are private because
+they are a file layout, not an API. Third-party code that reaches past
+`iris.fileformats.cf` into a submodule is reaching into the split itself, which
+is exactly the thing that should stay free to move.
+
+`iris.fileformats.zarr` shadows the third-party `zarr` distribution by name.
+Absolute imports mean `import zarr` inside that package still resolves to the
+third-party one; xarray has the same arrangement. The package docstring says so.
+
+### 4.2 The `CFDataset` interface
+
+`cf/dataset.py` defines two abstract classes. They are small — only what the
+CF variable classes, `cf/loader.py` and `cf/saver.py` actually need — but they
+are **complete in their own terms**. The interface describes what a
+CF-conforming array store is, not what netCDF and Zarr happen to have in
+common, and nothing above it may reason about which implementation it holds.
+
+```python
+class CFDatasetVariable(ABC):
+ """One named array in a CF-conforming dataset."""
+ name: str
+ location: str # the dataset's path or URL; F2
+ format: str # the concrete format; see below
+ dimensions: tuple[str | None, ...] # None where unnamed; 4.4
+ shape: tuple[int, ...]
+ dtype: np.dtype
+ size: int
+ fill_value: Any | None
+ chunking: tuple[int, ...] | None # None when the store is unchunked
+ storage_settings: Mapping[str, Any] # backend storage settings; see below
+ attributes: MutableMapping[str, Any] # materialised once
+
+ def __getitem__(self, keys) -> np.ndarray: ...
+ def __setitem__(self, keys, values) -> None: ...
+ def __len__(self) -> int: ... # defaults to self.shape[0]; F5
+ ndim: int # defaults to len(self.shape)
+
+ def write_handle(self) -> Any: ... # picklable __setitem__ target; see 4.5
+ def deprecated_netcdf_member(self, name: str) -> Any: ... # see 4.3
+
+
+class CFDataset(ABC):
+ """A CF-conforming array store, open for reading or writing."""
+ location: str # path or URL, for messages and proxies
+ format: str # the concrete format; see below
+ mode: str # "r" | "r+" | "a" | "w" | "w-"; 4.5
+ closed: bool # F3
+ variables: Mapping[str, CFDatasetVariable]
+ dimensions: Mapping[str, int]
+ storage_settings: Mapping[str, Any] # backend store settings; see below
+ attributes: MutableMapping[str, Any]
+
+ def create_dimension(self, name: str, size: int | None) -> None: ...
+ def create_variable(self, name, dtype, dimensions=(), *,
+ fill_value=None, **storage_settings) -> CFDatasetVariable: ...
+ def supports_dtype(self, dtype: np.dtype) -> bool: ... # see below
+ def sync(self) -> None: ...
+ def finalise(self) -> None: ... # one-shot; NOT part of close()
+ def close(self) -> None: ...
+
+ def __enter__(self) -> "CFDataset": ... # returns self; concrete
+ def __exit__(self, *exc_info) -> None: ... # calls close(); concrete
+```
+
+#### The interface is not a two-backend intersection
+
+A third backend is not planned, but the interface is written so that one would
+not require it to change shape, and so that a reader can understand it without
+knowing what implements it. Two rules follow, and they bind the loader and the
+saver as much as the dataset classes:
+
+- **No code above `cf/dataset.py` tests the implementation type.** No
+ `isinstance`, no checking a class name, no `hasattr` probe standing in for
+ one. Where generic code needs to behave differently, the difference is a
+ declared member that every implementation answers — which is a contract, so
+ a fourth implementation satisfies it by implementing it rather than by
+ inheriting from the right class.
+- **Where the generic layer needs a capability, it asks for the capability,
+ not the format.** `format` is for messages, diagnostics and the user; it is
+ not a branch key.
+
+`supports_dtype` is the first case and the reason the rule is written down.
+`saver.py:1479` and `:1500` today ask
+`self._dataset.dataset.file_format in ("NETCDF3_CLASSIC", "NETCDF3_64BIT",
+"NETCDF4_CLASSIC")` before deciding whether an `int64` or unsigned value must
+be cast down to `int32`. That is a netCDF question with a netCDF answer, and
+it sits in code PR 5 relocates to `cf/saver.py`, where it would become generic
+code naming netCDF format strings. The question it is really asking is "can
+this store hold this dtype?", so that is what the interface offers.
+`NetCDFDataset` answers it from `file_format`; `ZarrDataset` answers `True`
+for every dtype Zarr has a data type for. The error message that follows
+names `format`, so the user still reads `NETCDF3_CLASSIC`.
+
+#### `format` and `storage_settings`
+
+`format` names the concrete format an implementation provides — `"NETCDF4"`,
+`"NETCDF3_CLASSIC"`, `"ZARR_V2"`, `"ZARR_V3"`. Some parts of the loader and
+the saver legitimately need to *say* which format they are working with, in a
+warning, an error or a `LOAD_PROBLEMS` record, and reconstructing that from
+the class is exactly the type test the rule above forbids. On a variable it
+repeats its dataset's value, for the same reason `location` does: so a
+variable can answer for itself without a back-reference.
+
+`storage_settings` is the open-ended bag of backend storage settings: what
+`create_variable` was given on write, and what the store actually holds on
+read — `zlib`, `complevel`, `chunksizes` for netCDF; `compressors`, `chunks`,
+`shards`, `zarr_format` for Zarr. Its keys are backend-specific by nature and
+the generic layer never constructs or interprets them, but it must be
+*readable*, because a format-aware caller — a benchmark, a round trip that
+preserves chunking, a user asking why a load is slow — otherwise has to reach
+through to the underlying `zarr.Array` or `netCDF4.Variable` to find out, and
+that is the abstraction leaking at exactly the point it is supposed to hold.
+
+It overlaps `chunking` deliberately and they are not duplicates. `chunking` is
+load-bearing: it is typed, it is guaranteed present on every implementation,
+and §4.4's chunk calculation reads it. `storage_settings["chunks"]` is a backend's own
+spelling of its own settings, which generic code must never depend on.
+
+Three members exist only to keep multi-process writing reachable later, and
+are explained in §4.5: `write_handle`, `mode` and `finalise`. They cost
+almost nothing now — `mode` is a string the implementations already track
+internally, `write_handle` returns `self` for Zarr, and `finalise` is a no-op
+for netCDF — and their absence is what would force a breaking interface
+change later. `finalise` is defined but not yet called: PR 2 left
+`Saver.__exit__` alone rather than guess at an ordering only Zarr can
+exercise, and PR 6 wires it.
+
+`attributes` is materialised once and does *not* track reads. Tracking is a
+`CFVariable` concern, because `CFReader` builds a second `CFVariable` over
+the same backing variable when it promotes one, and the two must keep
+independent read sets (§4.3). `CFDatasetVariable.attributes` is therefore a
+plain mapping, and `CFVariable.attributes` is a tracking view over it. The
+view's class, `TrackedAttributes`, lives in `cf/dataset.py` beside the two
+abstract classes, because it is part of the same contract.
+
+`location` repeats `CFDataset.location` on the variable so that a variable
+can name its own file without a back-reference to its dataset. It is
+load-bearing: it is the `path` of a read proxy, and so part of the dask
+array cache key, and it is what `LOAD_PROBLEMS.record()` reports.
+
+`closed` is on the interface because `Saver.complete()` has to refuse to run
+until the file is released, and `isopen()` is netCDF vocabulary. A dataset
+the caller opened is closed by the caller, so an implementation answers from
+the store where it can, not only from its own flag.
+
+`create_dimension` accepts `size=None` to request an unlimited dimension; a
+store with no such concept raises. `create_variable`'s `dimensions` defaults
+to `()`, because grid-mapping variables are scalar.
+
+The split that matters is `attributes` versus everything else. Today
+`cf_var.units` might be a CF attribute or a netCDF property and the caller
+cannot tell. After this change, CF attributes are only ever reached through
+`.attributes`, and properties are named fields.
+
+`create_variable` takes `**storage_settings` against the general rule about keyword
+passthrough, because the accepted keys are backend-specific by nature
+(`zlib`/`complevel`/`chunksizes` for netCDF, `compressors`/`chunks`/`shards`
+for Zarr). Each implementation documents its own accepted keys, and
+`cf/saver.py` never constructs them. What it is given is retained and surfaced
+on `CFDatasetVariable.storage_settings`, so create-time configuration is readable
+afterwards rather than write-only.
+
+### 4.3 `CFVariable.__getattr__` and backwards compatibility
+
+`CFVariable.__getattr__` (cf.py:198) does two unrelated jobs today, and the
+refactor separates them. Counting what reaches it from inside Iris:
+
+| Reached via `__getattr__` | Count | Job |
+|---|---|---|
+| `.dimensions` | 28 | proxy the backing netCDF4 variable |
+| `.shape` | 11 | proxy |
+| `.dtype` | 11 | proxy |
+| `.getncattr` | 6 | proxy |
+| `.size`, `.group`, `.chunking` | 7 | proxy |
+| CF attribute names (`units`, `bounds`, `coordinates`, ...) | the rest | open-world data |
+
+`self.getncattr(attr)` at cf.py:223 is the clearest case: `CFVariable` does not
+define `getncattr`, so the call resolves through `__getattr__` onto the netCDF4
+object. Everything in the "proxy" rows is exactly what `CFDatasetVariable`
+(§4.2) declares explicitly and typed, so the refactor does not have to defend
+`__getattr__` - it shrinks it to the one job that justifies it.
+
+**The surviving job is legitimate but is not kept.** CF attributes are an
+open-ended set of data keys read from a file; there is no schema to enumerate,
+so forwarding them through `__getattr__` is defensible, on the condition that
+`.attributes` is the path library code takes and `__getattr__` is only a
+convenience accessor over it. The accessor is removed anyway (§12.4).
+
+That case is correctly argued and removing `__getattr__` is not a judgement on
+it. The accessor is removed because `.attributes` already does the whole job.
+`cf_var.units` and `cf_var.attributes["units"]` perform the *same* read with
+the *same* side effect: both record the attribute as used, and that record is
+what decides which attributes survive onto the loaded cube.
+So this is not a convenience layer over a plain getter. It is a second way of
+spelling one operation — the spelling that cannot be type-checked, and that
+silently loses to any class member of the same name (see the known limitation
+below).
+
+**It is deprecated, not deleted.** `.attributes` does not exist on released
+Iris, so until the merge-back ships there is no spelling a user can move to.
+Removing `__getattr__` in the same release that introduces its replacement
+would leave no version where both work, which is what a deprecation cycle is
+for. So:
+
+- `__getattr__` keeps working and warns on use, naming `.attributes` and the
+ key, under `IrisDeprecation` and marked `.. deprecated:: 3.17`. Removal
+ follows NEP 29, one cycle later.
+- The warning costs nothing in the ordinary case. Every read inside Iris goes
+ through `.attributes` directly after PR 2, so the warning fires only for
+ code outside Iris, and a normal load is silent.
+- `CFVariable` is documented public API — it is in `iris.fileformats.cf`'s
+ `__all__` and on the generated API reference page — but it is API reached
+ only by going to `iris.fileformats.cf` deliberately. Nothing a user does
+ through `iris.load` can hold one. That is what makes the eventual removal
+ proportionate; it is not a reason to skip the cycle.
+
+This is delivered by **PR 2.5** (§5), a small behaviour-only pull request,
+because PR 2 has already merged and the programme's rule is that no pull
+request both moves code and changes behaviour.
+
+So, on `CFVariable`, for as long as `__getattr__` survives its deprecation
+cycle:
+
+- `__getattr__` resolves against `self.attributes` - one implementation on the
+ base class, **identical for netCDF and Zarr**. It must not raise `TypeError`
+ for Zarr-backed variables: that would put a hole in the public accessor of
+ the very abstraction §4.2 exists to provide, and deprecating a thing is not
+ the same as letting it rot. A missing key raises plain `AttributeError`,
+ which is correct: `getattr(nc_var, "bounds", None)` is a deliberate idiom
+ throughout the loader and depends on `hasattr` semantics working.
+- **No `setattr(self, name, value)` caching.** Today's version caches by
+ mutating `self.__dict__`, so an instance's shape depends on its access
+ history - that genuinely is the "dynamically generated attributes"
+ anti-pattern. A read-through to a mapping materialised once at construction
+ is faster than today's *first* access and stays inspectable.
+- On a miss, a netCDF-backed variable falls back to the backing netCDF4 object
+ and issues a deprecation warning naming the `CFDatasetVariable` member that
+ replaces it, so `cf_var.getncattr("units")` and `cf_var.ndim` keep working
+ for one cycle. A Zarr-backed variable has no such object, so the fallback
+ simply does not apply - consistent, not a special case.
+
+**Known limitation, now fixed by removal.** `__getattr__` fires only when
+normal lookup fails, so a file carrying an attribute named `filename`,
+`cf_name` or `spans` is silently shadowed by the class member of that name. CF
+reserves no namespace, so this cannot be ruled out, and it cannot be fixed
+while the accessor exists — the shadowing *is* how Python attribute lookup
+works. `.attributes` has no such ambiguity. This was the practical argument
+for making the mapping the library path; it is now also the practical argument
+for the mapping being the only path.
+
+**Consequence for attribute tracking, the sharpest edge in PR 2.** The
+`_cf_attrs` "which attributes were read" set is currently a side effect of
+`__getattr__`, and `cf_attrs_unused()` has exactly one consumer:
+`netcdf/loader.py:190`, which decides which attributes survive onto the cube.
+Move the reads to `.attributes` and the tracking must move with them, or every
+CF-reserved attribute silently leaks onto loaded cubes. `.attributes` is
+therefore a tracking mapping, not a plain `dict`, and PR 1's tests must pin
+`cf_attrs_unused()` before PR 2 touches it.
+
+`CFVariable.attributes` is a `TrackedAttributes` view, constructed per
+`CFVariable` over a snapshot of `cf_data.attributes`. It records reads —
+which is how `cf_attrs_unused()` decides what reaches `cube.attributes` —
+and it is a snapshot rather than a live view because a backend's attribute
+mapping may write through to the file, and loading must never write.
+
+### 4.4 Reading
+
+#### Dimension names, and the dimension contract
+
+A Zarr dimension name is weaker than a netCDF one in three ways at once, and
+the design has to say what each means rather than treat any of them as a file
+defect. §2.4 quotes the specification: `dimension_names` is optional, an entry
+may be null, a name need not be unique within an array, and a name shared
+between two arrays guarantees nothing — not even that the two dimensions have
+the same length. All four are conforming Zarr. None may raise.
+
+**Do not raise when dimension names are absent, and do not synthesise them.**
+Raising is wrong because such a store is legal and is perfectly loadable.
+Synthesising is wrong because invented names collide with each other and
+produce silently mismatched cubes. The two look like the only options and they
+are not: the dimension is **anonymous**, and Iris has had anonymous dimensions
+all along. A cube whose dimensions carry no coordinates is an ordinary Iris
+cube.
+
+`ZarrDatasetVariable.dimensions` therefore returns a tuple of length `ndim`
+whose entries are names or `None`:
+
+| store | result |
+|---|---|
+| version 3, `dimension_names` present | its entries, null becoming `None` |
+| version 3, `dimension_names` absent | `(None,) * ndim` |
+| version 2, `_ARRAY_DIMENSIONS` present | its entries |
+| version 2, `_ARRAY_DIMENSIONS` absent | `(None,) * ndim` |
+| any version, scalar array | `()` |
+
+A `dimension_names` whose length is not `ndim` *is* a malformation — the
+specification says "must" — so it is ignored with an `IrisCfLoadWarning`
+naming the array, and the array loads with anonymous dimensions. zarr-python
+enforces the length match on **read** as well as on write, so this case
+reaches Iris only through a version 2 store: `_ARRAY_DIMENSIONS` is a plain
+attribute with no such check, where a version 3 array in the same shape is
+rejected before `open_group` ever returns it. The check in the reader is
+written for both versions regardless, as defensive code that costs nothing
+and happens to be exercised by only one of them.
+
+**Two attributes are storage conventions rather than CF metadata, and are
+withheld from `attributes` for that reason.** `_ARRAY_DIMENSIONS` is version
+2's own carrier for the dimension names above — `dimensions` already reads
+it, so leaving it in `attributes` too would put it on every cube loaded from
+a version 2 store. `_NCProperties` is netCDF-c's own bookkeeping stamp,
+written onto the root group of every store its NCZarr driver produces;
+netCDF4-python already hides it from `Dataset.ncattrs()`, so it has never
+reached a cube loaded through netCDF, and Zarr has no equivalent convention
+to hide it by. Left in, it would surface only on cubes loaded from an
+NCZarr-written store, so the same data would produce cubes that differ by a
+provenance string depending on which writer made the store. `_ARRAY_DIMENSIONS`
+is withheld from an array's own `attributes`, where it lives; `_NCProperties`
+is withheld from the dataset's, where it lives. Neither is withheld from the
+other kind of node, because neither ever appears there: `_ARRAY_DIMENSIONS` is
+not a group attribute and `_NCProperties` is not an array or subgroup
+attribute.
+
+**The identity that links two variables is the pair `(name, size)`, and an
+anonymous dimension has no identity.** One rule covers every case above:
+
+- Two variables share a dimension when they name it identically *and* it is
+ the same length in both. A name reused at two lengths links nothing, which
+ is the only reading that cannot produce a wrong cube.
+- An anonymous dimension never matches anything — not a named dimension, and
+ not another anonymous dimension, including within the same variable. There
+ is no information to match on, so claiming a match would be invention.
+- A variable may name a dimension the dataset does not list, and may name some
+ of its dimensions and not others. Neither is an error.
+
+For netCDF this changes nothing: dimension names are unique within a group, so
+`(name, size)` and `name` are the same key, and no netCDF variable ever has an
+anonymous dimension. It is a widening of the contract to admit what Zarr
+allows, not a change to existing behaviour — but it is a change to generic
+code. `CFVariable.spans` and its three overrides compare dimension *names*
+today (`_variables.py:221`, and `:530`, `:606`, `:929`); they must compare the
+pairs, and must treat `None` as unmatchable. PR 4 makes that change and tests
+it on both backends, since it is reachable only from Zarr but must not disturb
+netCDF.
+
+**The same rule decides which cube dimensions a coordinate is attached to.**
+`spans` is only the first half of the question. Once a coordinate is found to
+span the data variable, the loading rules have to say *which* of the data
+variable's dimensions it covers, and they did that by name alone —
+`cf_var.dimensions.index(name)`, in `_add_dimension_coordinate`,
+`_add_auxiliary_coordinate`, `_add_cell_measure` and `_add_ancil_var`. Given a
+data variable whose dimensions are `('d', 'd')` at sizes `(2, 3)` and a
+three-point coordinate on `d`, `spans` says yes and `.index('d')` answers 0,
+so the coordinate is rejected as the wrong length and lands in
+`LOAD_PROBLEMS` — with the axis it belongs on uniquely identifiable. The
+identity rule is therefore a function, `_dimension_identities(cf_variable)` in
+`cf/_variables.py`, used by `spans` and by one shared
+`helpers._common_data_dims`: one rule, not two that can drift. netCDF cannot
+observe the difference, its dimension names being unique within a group.
+
+**`CFDataset.dimensions` is the names that are used consistently.** It maps
+name to size, as it does for netCDF, and on Zarr it is assembled from the
+arrays. A name that appears at two different sizes is not an error and is not
+silently merged: it is omitted from the mapping, with an `IrisCfLoadWarning`
+naming the dimension and the conflicting sizes. The variables that use it keep
+it in their own `dimensions`, where the `(name, size)` rule stops it linking
+them. The saver's use of this mapping — "has this dimension been created
+yet?", `saver.py:787` and three others — is unaffected, because the saver
+creates every dimension it uses.
+
+What Iris does **not** offer is a control to force two same-named,
+same-length dimensions apart. Treating that pair as one dimension is what
+xarray, NCZarr and netCDF itself all do, so it is the only sane default; a
+store that means otherwise is relying on something no reader implements. It is
+recorded as Q9 (§12.3) in case a real store turns up that needs it.
+
+NCZarr-written stores use a `_scalar_` pseudo-dimension for scalar variables.
+`cf.py` already handles this in `CFVariable.spans` (cf.py:179-184) via
+`_NCZARR_SCALAR_DIMENSION`. The three subclasses that override `spans`
+(`CFBoundaryVariable` cf.py:415, `CFClimatologyVariable` cf.py:491,
+`CFLabelVariable` cf.py:814) do **not**, which is a latent bug on the existing
+NCZarr path. It is fixed in PR 2, and is the only `bugfix` fragment the
+merge-back owes: everything else in the programme is new capability, a
+relocation or a deprecation.
+
+#### Fill values: storage fact versus masking instruction
+
+Zarr's array-level `fill_value` and CF's `_FillValue` attribute are different
+things — see §2.4 — and the difference runs the opposite way to the intuitive
+one. `fill_value` is a *storage* fact: it is what a chunk that was never
+written reads back as. `_FillValue` is a *semantic* one: it declares a value to
+mean "missing". Only the second is a masking instruction, so **the CF attribute
+governs masking and the storage field does not.**
+
+The precedence is therefore version-aware, matching what released xarray does
+(`use_zarr_fill_value_as_mask`, `xarray/backends/zarr.py:1989-1995`):
+
+| | masking source |
+|---|---|
+| **Version 3** | the CF `_FillValue`/`missing_value` attributes only. `Array.fill_value` is *not* a mask source |
+| **Version 2** | `Array.fill_value` is used as the mask source when no CF attribute is present, because version 2 stores predate the convention and this is what their writers meant |
+
+Reading `Array.fill_value` as a sentinel on version 3 would mask valid data in
+the ordinary case, not the exotic one: zarr-python defaults the field to zero,
+so an `int16` array holding `[0, 1, 2]` with no CF attributes at all has its
+valid zero masked **[verified]**. And since the field is *required* on version
+3, an "only when the array carries no usable one" fallback would never fire, so
+a genuine CF sentinel that differs from the storage fill — `_FillValue = -999`
+over a default `fill_value = 0` — could never win. Both cases were reproduced
+against a store written by xarray **[verified]**.
+
+`Array.fill_value` is still read, and still used: it is what an unwritten chunk
+yields, it round-trips through `create_variable` on save (§4.5), and on version
+2 it is the mask source above. It is simply not a CF missing-value declaration.
+
+#### A floating `_FillValue` is base64 on version 3
+
+That is a convention, not a malformation. Released xarray encodes a
+floating-point `_FillValue` attribute as base64 of a little-endian `float64` —
+`struct.pack("<d", value)` — for *every* floating dtype including `float32`,
+and its reader *requires* that form, raising `TypeError` for a plain JSON number
+(`FillValueCoder.encode`/`decode`, `xarray/backends/zarr.py:144-149` and
+`185-191`).
+
+Do not treat a base64 `_FillValue` as a file malformation to defend against.
+The NOAA GFS store's `'AAAAAAAA+H8='` is the obvious candidate, and it is not a
+publisher's CF violation: `base64(struct.pack("<d", float("nan")))` **is**
+`'AAAAAAAA+H8='` **[verified]**, and writing a `float32` NaN with xarray
+produces that literal string **[verified]**. It is the de-facto encoding for
+Zarr version 3, and most version 3 data in the wild will carry it. Warning on
+it would mean warning on essentially every xarray-written store.
+
+So Iris **reads** both forms on version 3 — a base64 string decoded as
+`float64` and narrowed to the array dtype, or a JSON number — and writes the
+base64 form (§4.5). CF §2.5.1 does require "the scalar attribute with the name
+`_FillValue` and of the same type as its variable", and base64 is a deliberate,
+documented deviation from that, taken because interoperability with the other
+Zarr readers is the reason for native Zarr support in the first place. It is
+one narrow exception for one attribute whose dtype is load-bearing; it is not a
+general licence to reshape the reader around whatever a file happens to
+contain, and it does not reopen the base64-envelope rejection for general
+attributes (§4.5).
+
+A `_FillValue` that is neither a number nor a decodable base64 string of the
+right width is still a producer error: Iris ignores it and emits an
+`IrisCfLoadWarning` naming the variable and the value.
+
+The cut-down NOAA fixture keeps the attribute verbatim, now as the
+**cross-reader conformance case** rather than as a malformation sample.
+
+#### Attributes whose value is a nested JSON object or array
+
+Do not describe this set as "attributes that are not scalars or strings". That
+names the wrong set, and it would send the reader looking for a problem where
+there is none. A JSON string is an ordinary CF attribute. So is a flat JSON
+array of numbers: that is a CF vector attribute, which is what `valid_range`
+has always been, and it needs no special handling. The gap is narrower and
+stranger. It is values with *structure* — an object, or an array of arrays —
+which CF cannot express at any rank.
+
+This one *is* a property of Zarr: the version 3 specification says an attribute
+value can be an arbitrary JSON literal, and CF has no nested-attribute concept,
+so the gap exists for every conforming store. It is also an established habit
+rather than one team's slip — the GFS store gives `valid_time` a
+`statistics_approximate` attribute
+whose value is `{'min': ..., 'max': ...}`, and the ESA EOPF product root
+carries `stac_discovery`, `other_metadata` and `processing_history` as nested
+objects **[verified, two independent producers]**.
+
+Iris carries such values onto the cube unchanged, does not interpret them, and
+does not write them back through the netCDF saver. A cube carrying one
+therefore cannot be saved as netCDF: the attempt fails inside netCDF4 with a
+`TypeError` naming the attribute — `illegal data type for attribute
+b'stac_discovery', ..., got O` — but not the cube, since checking attribute
+content before saving and raising a message that does both is
+[#7329](https://github.com/SciTools/iris/issues/7329), out of scope here
+(§4.5, §9). This is the provisional answer; the durable one needs the
+attribute-model discussion in SciTools/iris#7288 (§10).
+
+#### Masking and unpacking
+
+netCDF4 applies `_FillValue`/`missing_value` masking,
+`valid_min`/`valid_max`/`valid_range` masking and
+`scale_factor`/`add_offset` unpacking for free. Zarr does none of it, so
+`zarr/_decode.py` implements the same CF rules. It is Zarr-side rather than
+shared because the netCDF path gets the behaviour from the C library and has
+nothing to reuse.
+
+`missing_value` may be a **vector**. CF section 2.5.1 makes it "a scalar or
+vector containing values indicating missing data", and netCDF4 masks on every
+element of one, so a store and a file carrying the same attribute have to mask
+the same values. `_FillValue` is not read that way: the same section makes it
+a scalar, and a vector there warns and is ignored. Read as a scalar, a vector
+`missing_value` fails twice over — the declared sentinels do not mask, and on
+version 2 the variable then looks as though it declared no CF attribute at
+all, so the storage fill value becomes the sentinel and every valid zero is
+masked.
+
+Decoding is applied as **explicit dask graph layers** over the raw array —
+`da.map_blocks(_mask_and_scale, raw, ...)` — not hidden inside an opaque
+`__getitem__`. Each rule is a small pure function of `(block, parameters)`,
+unit-testable on a plain ndarray with no store at all; the layers are visible
+in the graph, fuse with neighbours, and keep the lazy contract. A proxy whose
+`__getitem__` silently returned something other than what the store holds is
+exactly the "behaviour not derivable from the file in front of you" that
+`lib/iris/AGENTS.md` warns against.
+
+#### Laziness: there is no `ZarrDataProxy`
+
+Do not mirror `NetCDFDataProxy` on the Zarr side. It is the obvious move and it
+is the wrong one, because every reason `NetCDFDataProxy` exists is a
+netCDF4/HDF5 reason:
+
+- `netCDF4.Variable` is not picklable, so it cannot travel in a dask graph.
+- The netCDF4 Python package is not re-entrant, so every call into it
+ serialises on `_GLOBAL_NETCDF4_LOCK`. Note what that lock is and is not: one
+ process-wide lock guarding the netCDF4 API
+ (`_thread_safe_nc.py:19`, and `:355-357`, where `NetCDFDataProxy.__getitem__`
+ takes it directly). Iris has **no** per-file read lock and needs none — the
+ per-file locking in `_dask_locks.py` is a write-path mechanism.
+- A `Dataset` is an open OS file handle that must not be held for the lifetime
+ of a lazy graph — hence the open/read/close on every `__getitem__`
+ (`_thread_safe_nc.py:353-366`).
+
+None of the three holds for Zarr. `zarr.Array` **is** picklable and round-trips
+shape and chunks **[verified]**. A chunk is an independent object in a
+key-value store, so concurrent reads of distinct chunks need no lock. A store
+is a namespace, not an open handle, so there is nothing to leave open. And this
+is the mainstream path rather than a novelty: `dask.array.from_zarr` is a thin
+wrapper that calls `from_array` on the `zarr.Array` itself **[verified]**.
+
+So `zarr/loader.py` hands the `zarr.Array` to `as_lazy_data` directly and adds
+the decode layers on top. The three things a proxy might look necessary for
+all survive without one:
+
+1. **`CHUNK_CONTROL`.** Chunking is computed by the loader from
+ `CFDatasetVariable.chunking` *before* `as_lazy_data` is called. It never
+ depended on proxy-ness.
+2. **Masked dask meta.** `meta` is an argument to `as_lazy_data`, not something
+ a proxy supplies. With decoding as a graph layer the mask arrives with the
+ decode layer, so a masked meta is correct by construction. Note the honest
+ ordering — passing a masked meta with no decode layer beneath it would be a
+ meta that lies about its own graph.
+3. **The dask cache key.** This is the one real point, and it wants a hashable
+ identity, not a class. See below.
+
+#### The dask cache key
+
+**Replacing the `NetCDFDataProxy` special case.** `iris/_lazy_data.py` imports
+`NetCDFDataProxy` from `iris.fileformats.netcdf` (line 302) purely to
+recognise it (lines 333-337) and build a cache key from `repr(data)`, so that
+many cubes sharing a coordinate array share one dask array. That is a backwards
+import — the lazy-data layer reaching into a file format — and adding a second
+`isinstance` branch for Zarr would double it.
+
+PR 3 replaces it with a caller-supplied `cache_key=` keyword on
+`as_lazy_data`. The netCDF loader passes `repr(proxy)`, which is
+byte-for-byte the key computed today, so the change is behaviour-preserving and
+testable as such. `_lazy_data` then knows about no file format at all.
+
+**The Zarr key must carry the array's metadata, not just its address.** Do not
+key on `(store location, array path, zarr format)`. That is an *address*, and
+`CACHE` is a process-wide `LRUCache(100)` (`_lazy_data.py:262`),
+so the address goes stale the moment a store is rewritten in place — which
+`mode="w"` supports and which a notebook or a long-running service does
+routinely. Two failure modes were reproduced against the real key shape
+**[verified]**:
+
+- an array reopened after its shape changed from `(4,)` to `(6,)` under fixed
+ chunking returned the **cached four values**, silently truncating;
+- an array whose storage `fill_value` changed from `-1` to `-2`, with chunks
+ left unwritten, returned the **old fill**, while a fresh open returned the
+ new one. Under the rules above that feeds the version 2 mask, so stale
+ metadata becomes wrongly masked data.
+
+Note that an address-only key would be **weaker than the netCDF key it
+replaces**: `repr(NetCDFDataProxy)` already includes shape and dtype
+(`_thread_safe_nc.py:370-376`), so the first case above cannot happen on the
+netCDF path today. A relocation that quietly loosened that invariant would not
+be behaviour-preserving.
+
+The Zarr loader therefore passes `(store location, array path,
+json.dumps(Array.metadata.to_dict(), sort_keys=True))`. The metadata document
+covers shape, chunks, shards, dtype, `fill_value`, codecs, `dimension_names`
+and attributes in one value; it is stable across reopen and distinguishes both
+cases above **[verified]**, at a few hundred characters per key.
+
+**`to_dict()` is zarr-python's rendering, not a specification guarantee, and
+that is handled by a test rather than by enumerating our own key.** Nothing in
+the Zarr specification defines what that method returns, so Iris must not
+assume it keeps carrying what the key needs.
+
+Do not fix that by listing the components Iris cares about and building the key
+from those. It trades the risk for a worse one. The two failure modes are not
+symmetrical: a key that changes when nothing meaningful did costs a cache miss,
+while a key that *fails* to change when something meaningful did serves stale
+data, which is the bug this subsection exists to prevent. A hand-written list
+is exactly what goes stale when a zarr-python release adds a metadata field
+that matters, and nothing fails when it does.
+
+So the key stays complete-by-construction and is treated as opaque, and the
+assumption is made checkable instead. §6 pins it: the key must be stable
+across a reopen of an unchanged array, and must change when each of shape,
+chunks, shards, dtype, `fill_value`, codecs, `dimension_names` and attributes
+changes. A zarr-python release that stopped carrying one of those fails a named
+test rather than quietly weakening the cache.
+
+This narrows the window rather than closing it: metadata identity still cannot
+detect a store whose *chunk contents* changed under identical metadata. Neither
+can the netCDF key, so this is the existing standard and not a regression — but
+it is the reason the cache is an optimisation for array sharing *within* a
+load, and §12.3 Q6 records scoping it to a load session as the durable fix.
+
+#### Read-side chunk alignment
+
+§4.5 fixes a write-side invariant: the dask chunking must tile the Zarr chunk
+grid exactly. Reading needs the same invariant for a different reason, and it
+is *not* free.
+
+A Zarr chunk is the atomic unit of storage: a read of any part of it fetches
+and decompresses the whole object. So if the dask chunking subdivides a store
+chunk into N pieces, N tasks each pull and decompress that entire chunk — N
+times the bytes and N times the CPU, and over a remote store, N HTTP GETs for
+the same object.
+
+`_optimum_chunksize` does the right thing in the ordinary case, expanding to
+whole multiples of the store chunk **[verified]**:
+
+| store chunk | array shape | dask chunk | multiple |
+|---|---|---|---|
+| `(512, 512)` | `(4096, 4096)` | `(4096, 4096)` | 8 × 8 |
+| `(1000, 1000)` | `(10000, 10000)` | `(1000, 10000)` | 1 × 10 |
+
+But when the store chunk is *itself* larger than the `dask.array.chunk-size`
+target (128 MiB by default) it shrinks below it **[verified]**:
+
+| store chunk | size | array shape | dask chunk | result |
+|---|---|---|---|---|
+| `(8000, 8000)` | 488 MiB | `(16000, 16000)` | `(2000, 8000)` | 4 × re-read |
+| `(6000, 6000)` | 275 MiB | `(6000, 6000)` | `(2000, 6000)` | 3 × re-read |
+
+Subdividing buys nothing here. Peak memory per task is still a whole
+decompressed chunk, because that is what the store hands back; only the I/O
+and decompression multiply.
+
+**Iris does not correct for this, and warns instead.** The loader sizes
+chunks with `_optimum_chunksize` and stops there, for both storage backends.
+When a stored chunk is larger than the dask target it emits an
+`IrisLoadWarning` naming the variable, and `CHUNK_CONTROL` is how a caller
+who wants the other trade takes it.
+
+An earlier revision of this section had the Zarr path round the computed
+chunking **up** to a whole multiple of the store chunk. That was removed on
+review (#7336), on grounds of proportionality rather than mechanism:
+
+- The mechanism was never in doubt, and measurement sharpened it. Against a
+ 33.6 MiB store chunk subdivided into quarters, aligned chunking held lower
+ peak RSS at every worker count above one (305 MiB against 446 MiB at four
+ workers) and ran in roughly half the time, with 1.0 fetches per store chunk
+ against 3.2 **[verified]**.
+- But alignment only ever changes the answer in the shrink case above.
+ Wherever `_optimum_chunksize` expands, it already lands on a whole multiple
+ and alignment is a no-op **[verified]**. So the whole of it existed to
+ override one decision.
+- And that decision is one the netCDF path makes too. Both backends pass the
+ file's own chunking into the same `_optimum_chunksize`, and netCDF has
+ always accepted the subdivision — `NetCDFDataProxy` additionally reopens the
+ `Dataset` on every `__getitem__`, discarding HDF5's chunk cache between
+ tasks. Correcting it on one path only would make two formats behave
+ differently over physics they share, and would make `CHUNK_CONTROL` mean
+ different things depending on what was loaded.
+
+The house preference decides it: give the user the tools to adapt their own
+case rather than second-guess it. The warning is what makes that fair — it
+says the situation exists and names the tool — and it now covers both
+backends, from `iris.fileformats.cf.loader`. The remaining asymmetry is only
+that this costs far more over a network than over a page cache, which is an
+argument for the user reaching for `CHUNK_CONTROL` sooner, not for Iris
+choosing on their behalf.
+
+**Chunks are sized on the decoded dtype, not the stored one.** A packed
+variable's values are unpacked block by block, on the way out of the graph, so
+what a task holds is the decoded block: an `int8` array with a `scale_factor`
+that unpacks to `float64` is eight times its stored size the moment the decode
+layer runs. Sized on the stored dtype, a `(512, 512, 512)` `int8` array gets
+one chunk spanning the array — 128 MiB stored, 1 GiB decoded, against a 128
+MiB target. The netCDF path reaches the same answer from the other end: its
+data proxy declares the decoded dtype, and `as_lazy_data` sizes on the array
+it is given. `_optimum_chunksize` therefore takes the decoded dtype, which for
+an unpacked variable is the stored one.
+
+`CHUNK_CONTROL.as_dask()` is not exempt. That setting hands sizing to Dask,
+and `da.from_array` sizes on the array it is given — which is the stored one,
+so the same multiplier survives there. The Zarr path therefore calls
+`dask.array.core.normalize_chunks("auto", shape, dtype=decoded_dtype)`
+itself and passes the result on with every dimension fixed. The algorithm and
+the configured target are still Dask's; only the dtype it budgets against is
+supplied rather than inferred. Fixing the dimensions is what stops
+`as_lazy_data` re-sizing the result on the stored dtype and undoing it — and
+is the thing to watch in that branch, since Iris's own sizing or the store
+alignment creeping back in on top would make `as_dask()` mean something other
+than what it says.
+
+`CFDatasetVariable.chunking` returns the store's own chunk shape, which
+`cf/loader.py` uses exactly as it uses `chunking()` today.
+
+#### The read unit is the inner chunk, even when the array is sharded
+
+Do not have `chunking` return `Array.shards` when present, on the reasoning
+that "the shard is the unit a reader actually fetches". That is not how
+sharding works. Zarr's sharding codec is *indexed*: a reader fetches the
+shard index, then byte ranges for only the inner chunks it needs. The inner
+chunk is the independently decodable unit, and it stays the right unit for the
+chunk calculation.
+
+Requiring shard-sized dask blocks actively destroys that, and it does so
+*because* of the explicit decode layers above. Bare dask slicing can push a
+slice down into the array's own `__getitem__`, but a `map_blocks` layer in
+between cannot, so the block is materialised in full. On a `shape=(1024,)`,
+`chunks=(8,)`, `shards=(1024,)` int32 array, reading the first eight values
+**[verified]**:
+
+| dask blocks | decode layer | bytes fetched |
+|---|---|---|
+| inner-chunk `(8,)` | no | 2,084 |
+| inner-chunk `(8,)` | yes | 2,084 |
+| shard `(1024,)` | no | 2,084 |
+| shard `(1024,)` | **yes** | **6,148** — the whole shard |
+
+2,052 of those bytes are the shard index and 32 are the data actually wanted.
+So `chunking` returns `Array.chunks`, never a shard boundary — the read unit
+is the inner chunk, and that is what the chunking policy and the warning above
+are both sized against. This is not academic: the NOAA GFS fixture is sharded
+(§7).
+
+Shard-sized blocks remain a *write* requirement, for a different reason and in
+one direction only. §4.5 keeps them there.
+
+#### Async concurrency
+
+Zarr's own async concurrency is left at its default. Iris does not write to
+`zarr.config`; the interaction between `zarr_async_concurrency` and the dask
+worker count is documented instead.
+
+### 4.5 Writing
+
+Writing is version 3 only. The shape that keeps version 2 cheap to add later is
+that **`cf/saver.py` never mentions a Zarr version.** Everything version-specific
+lives in `ZarrDataset`:
+
+- how dimension names are recorded (`dimension_names` field versus
+ `_ARRAY_DIMENSIONS` attribute);
+- which codecs are available;
+- whether sharding is offered.
+
+`ZarrDataset.__init__` takes `zarr_format: int = 3`. Today it raises
+`NotImplementedError` for anything but 3, in one place. Adding version 2 write
+means implementing those three points and deleting that guard.
+
+#### Dimensions
+
+Zarr has no dimension objects. `ZarrDataset.create_dimension` records name and
+size in a plain dict, and `create_variable` writes the names onto each array.
+The dict is what lets the saver check for the size conflicts it checks for
+today.
+
+#### Attributes
+
+Values are converted to JSON-native types **before** being handed to zarr,
+never after, because a rejected value poisons the attributes mapping
+**[verified]**. The conversion is: NumPy scalar to its Python equivalent, NumPy
+array to a list, `bytes` to `str` via ASCII, everything else unchanged.
+Anything still unserialisable raises with the attribute name in the message.
+
+**An attribute the format cannot hold is an error, and is not quietly turned
+into a string.** Do not generalise the known-object conversions in
+`netcdf/_attribute_handlers.py` — STASH codes, cell methods — into "convert
+*anything* unrecognised to its string form and warn". The rule holds for both
+backends (@trexfeathers, §12.4):
+
+> I confidently see `_attribute_handlers` as a special case for things where
+> there is an obvious and understood way to represent something as a string.
+> This does not generalise to everything that might not be storable in the
+> file (e.g. someone assigns an object which has a `__str__` and `__repr__`,
+> but neither of which is adequate for reconstructing said object
+> downstream; that would be a very different decision we are making). In Iris
+> we are typically very careful about Cube-to-Cube roundtripping — can I save
+> a Cube to a file then load back a Cube that is the same as the original —
+> and falling back on converting all unsaveable content to a string seems to
+> deviate from that.
+
+The existing handlers convert where the string form is defined and reversible,
+and the reverse direction is implemented beside it. `str(some_object)` has
+neither property. A warning is also the wrong instrument: it does not stop the
+save, so the user gets a file, is told in passing that part of it is now prose,
+and finds out at load time.
+
+The netCDF saver owes the same policy a check it does not yet make. netCDF
+accepts only scalars, strings and one-dimensional arrays as attribute values,
+so a two-dimensional array attribute fails inside netCDF4 with a message
+naming neither the attribute nor the cube. Checking first, and raising, is the
+answer there too. It is a behaviour change to the netCDF saver, so it is
+**not** in this programme — PR 5 relocates that code without changing it.
+[#7329](https://github.com/SciTools/iris/issues/7329).
+
+This loses the NumPy dtype of attributes. That is the same trade xarray makes,
+and it is what makes the output readable by every other Zarr consumer. The
+alternative offered in #6961 — a base64 envelope carrying `dtype` and `raw` —
+is **rejected** for general use: it would make Iris output opaque to xarray and
+to the GeoZarr tooling, which is the whole point of writing Zarr rather than
+netCDF. Round-trip tests therefore compare attribute **values**, not dtypes.
+
+This is the provisional answer to the wider question of how the CF attribute
+model maps onto JSON in both directions — the same question the nested-object
+paragraph in §4.4 runs into from the read side. §10 and SciTools/iris#7288
+record it as parked for a dedicated discussion; nothing else in the design
+depends on how it is settled.
+
+#### `_FillValue` and `missing_value`: a dtype-dependent encoding
+
+These two attributes are exempt from the JSON-native conversion above, because
+their dtype is load-bearing. They are written in the encoding released xarray
+reads, which on version 3 is **dtype-dependent**:
+
+| array dtype | `_FillValue` attribute written |
+|---|---|
+| floating | base64 of `struct.pack("<d", value)` — a `float64` payload, including for `float32` |
+| integer | a plain JSON integer |
+
+The array's own `fill_value` is set to the same value in its native type, so an
+unwritten chunk still reads back as missing.
+
+The floating case is the deviation from CF §2.5.1 argued in §4.4, and it is
+load-bearing in the literal sense: writing a plain JSON number instead makes
+the store **unreadable by default xarray**, which raises
+
+```
+TypeError: Failed to decode fill_value: expected str or bytes for dtype float32, got float
+```
+
+and fails the whole dataset open, not merely the one variable **[verified]**.
+An Iris-written store that no other Zarr reader can open would defeat the
+purpose of writing Zarr rather than netCDF. Note the check that catches this
+already existed on paper: §6 promises that xarray must read what Iris writes,
+and says "if xarray cannot read Iris output, the decision was wrong". That test
+now has a specific case to carry.
+
+`_bytes_if_ascii` and `_setncattr` (saver.py:271, 290) coerce attribute values
+to ASCII `bytes` for netCDF. That coercion stays on the netCDF path only.
+
+#### Masked data
+
+Masked arrays are stored as `data.filled(fill_value)` with the array's
+`fill_value` set to match, as recommended in #6961. Writing a separate mask
+array is rejected as over-engineering for no CF benefit.
+
+The saver **must also write a `_FillValue` attribute** in this case, in the
+encoding above, even when the source cube carries no such attribute of its own.
+This follows directly from §4.4: on version 3 the CF attribute is the masking
+declaration and the storage `fill_value` is not, so a store written with the
+storage field alone would read back with **every masked point silently
+unmasked** — a round trip that loses the mask. Setting `fill_value` without
+`_FillValue` is the write-side form of the same conflation the read-side
+precedence made. §6 pins it with a masked round-trip test.
+
+#### Deferred writes, and what licenses `lock=False`
+
+The netCDF saver defers lazy writes by closing the file, returning a
+`Delayed`, and reopening per chunk under a whole-file lock. Zarr needs no
+reopen: the store stays valid, and writes to *distinct chunks* are independent
+objects. The Zarr saver builds one
+`da.store(..., compute=False, lock=False)` over all lazy sources and returns it
+from `save(..., compute=False)`, matching the netCDF signature. This is also
+why the NCZarr special case in `Saver.__exit__` (saver.py:485-498) is
+untouched: it is a different, older path.
+
+`lock=False` is licensed by an invariant, not by Zarr being inherently safe.
+Two dask tasks that touch the *same stored object* race: each reads it,
+modifies its slice and writes the whole object back, so one update is silently
+lost.
+
+#### The write-alignment invariant
+
+**The stored object is the shard, not the chunk, whenever sharding is
+enabled.** Stating this invariant over the chunk grid alone is insufficient,
+and the gap is not theoretical — it silently destroys data. With
+`da.arange(128, chunks=8)` stored into an array with `chunks=(8,)`
+and `shards=(64,)`, the dask chunking tiles the chunk grid *exactly*, yet
+threaded `da.store(..., lock=False)` lost 72–96 of 128 values in **12 of 12
+trials** **[verified]**. Eight inner chunks share one shard; eight tasks
+read-modify-write that one object. Therefore:
+
+> **Write-alignment invariant.** Every Zarr array Iris writes has a *write
+> grid* that the lazy source's dask chunking tiles exactly, where the write
+> grid is `Array.shards` when the array is sharded and `Array.chunks`
+> otherwise. Each concurrent task owns at least one whole write-grid cell; no
+> cell is touched by two tasks. A truncated final cell is permitted.
+
+The fix was checked on the same reproduction **[verified]**: aligning the
+source to the shard (`chunks=64`) or to a whole multiple of it (`chunks=128`)
+gives 0 of 12 corrupt trials, as does the unsharded case. Serialising with
+`lock=True` also fixes it, at the cost of the concurrency the design exists to
+get.
+
+`create_variable` derives the write grid from the source's dask chunking by
+default, so the invariant holds by construction. Where the caller forces
+`chunksizes=` or a `shards=` encoding that the source does not tile, the saver
+rechunks the dask source to match before storing. It never issues an unaligned
+concurrent write.
+
+This is the same guarantee xarray spells `safe_chunks`, and xarray states it
+over the same unit — `effective_write_chunks = encoding.get("shards") or
+encoding["chunks"]`, added for exactly this corruption
+(`xarray/backends/zarr.py:1304-1316`, pydata/xarray#10831). Citing
+`safe_chunks` as precedent without also matching its unit is the specific trap
+here.
+
+It is a correctness requirement today, not only a precondition for the future
+work below, and it binds even though the saver does not offer a `shards=`
+keyword yet: the invariant is what licenses `lock=False`, so it has to be
+stated over the unit that will exist when sharding is offered (the version 3
+list) or when Iris writes into a store someone else sharded.
+
+#### Consolidation is a separate, one-shot step
+
+`CFDataset.finalise()` writes consolidated metadata; `close()` only releases
+resources. They are split because a worker that writes one slab of a store
+must not consolidate — N workers consolidating the same single metadata object
+is a race. Under
+`compute=False` the returned `Delayed` owns the `finalise()` call, after every
+chunk write. For netCDF, `finalise()` is a no-op.
+
+#### The write proxy: why netCDF needs one, and Zarr does not
+
+`NetCDFWriteProxy` is easy to read as multi-process machinery, because it
+carries the file lock. That is its *second* job. Its first is more basic: to be
+a `__setitem__` target that survives the file being closed.
+
+The netCDF deferred save closes the dataset before the `Delayed` is computed —
+`Saver.delayed_completion` says so explicitly, and computing with the file
+still open hangs. A `netCDF4.Variable` cannot outlive its `Dataset`, and is not
+picklable, so it cannot be the target handed to `da.store`. The proxy replaces
+it with `(path, varname, lock)` and reopens per chunk, retrying up to five
+times because HDF5 sometimes refuses a file Python believes it has released
+(`_thread_safe_nc.py:412-433`). So the proxy is needed for a **single-threaded**
+deferred save too, not only a distributed one. The lock is what makes it also
+work across workers.
+
+Neither job exists for Zarr. There is no handle to close, so nothing has to
+outlive one; `zarr.Array` is picklable and keeps writing after a pickle
+round trip **[verified]**; and distinct chunks are distinct objects, so the
+alignment invariant above removes the need for a lock rather than deferring it.
+The end-to-end property was checked directly: a `da.store(..., compute=False,
+lock=False)` over a `zarr.Array` still lands correctly after every in-process
+reference to the group is dropped, **and the whole delayed graph survives
+`pickle.dumps`/`loads` before computing** **[verified]** — which is the
+property a `distributed` scheduler actually requires.
+
+**Native Zarr therefore restores a feature NCZarr had to give up.** The
+existing saver cannot defer NCZarr writes at all: `Saver.__exit__`
+(saver.py:485-498) computes them eagerly with `da.store` while the file is
+still open, commenting that "the deferred reopen-write pattern used for netCDF
+is not supported". That is a limitation of reaching Zarr through netCDF-c, not
+of Zarr. Going direct, `iris.save(cubes, "out.zarr", compute=False)` returns a
+deferred save like the netCDF path — this is a *gain* over the status quo for
+the distributed-write use case, and PR 6 must include a test asserting it.
+
+So `write_handle()` earns its place, but not as future-proofing: it is needed
+today, by netCDF, for exactly the reason above. `NetCDFDatasetVariable` returns
+the write proxy; `ZarrDatasetVariable` returns its `zarr.Array`. `cf/saver.py`
+asks for a handle and stores into it, knowing nothing about locks, reopening or
+schedulers.
+
+One correction to the accounting in §4.5 below: library code never instantiates
+`_thread_safe_nc.NetCDFWriteProxy` directly. `_lazy_stream_data` builds
+`_bytecoding_datasets.EncodedNetCDFWriteProxy`, a subclass, and says why
+(saver.py:2638-2643). The base class survives as public API and as the
+superclass. The relocation must preserve both.
+
+**A live defect in the machinery this inherits.** With `dask 2026.7.1`,
+`da.store(sources, targets, compute=False)` returns a **tuple** when `sources`
+is a sequence, not a `Delayed` **[verified]**. `Saver.delayed_completion` is
+annotated `-> Delayed` and returns that value unchanged, so
+`iris.save(cubes, path, compute=False)` returns a tuple and the documented
+idiom fails:
+
+```
+>>> result = iris.save(cubes, path, compute=False)
+>>> result.compute()
+AttributeError: 'tuple' object has no attribute 'compute'
+```
+
+`dask.compute(result)` works, which is why nothing caught it. The cause is
+known: #6451 adapted Iris to `dask/dask#11844` for dask 2025.4, correctly
+switching every internal caller to `dask.compute()`, and relaxed the test
+assertion to a helper accepting either shape. The public return type changed
+with it, but `Saver.delayed_completion`, `save()` and
+`docs/src/user_manual/explanation/real_and_lazy_data.rst` still promise a
+`Delayed` completed by `result.compute()`. Raised as
+[#7291](https://github.com/SciTools/iris/issues/7291).
+
+This is a netCDF-side defect, not a Zarr one, and it is **not** fixed by this
+programme — PR 5 relocates the saver without changing behaviour. But it is
+recorded here because the Zarr saver inherits the same shape, so `zarr/saver.py`
+wraps its `da.store` result in `dask.delayed` and returns a real `Delayed`, and
+its tests assert `.compute()` works. Tracked as Q5 in §12.3 and as #7291.
+
+#### Durability: what an interrupted save leaves behind
+
+Zarr has no multi-object transaction. A store's metadata and its chunks are
+separate objects, so an edit that touches both can be interrupted between them
+and leave the two disagreeing. Two consequences, both confirmed against
+`zarr 3.4.0` **[verified]**:
+
+- A missing chunk is **not an error**. Remove one chunk object from a complete
+ array and it reads back as `fill_value` with no warning. An interrupted save
+ therefore leaves a store that opens cleanly and silently under-reports data.
+- Consolidated metadata goes **silently stale**. Add an array to a consolidated
+ store without re-consolidating, and a default reader — which uses
+ consolidated metadata when it is present — does not see the new array at all.
+ An unconsolidated reader of the same store does.
+
+Neither is fixable from inside Iris; they are properties of the format. What
+Iris can do is never create the conditions for them.
+
+> **Create-once invariant.** Array metadata is written when the array is
+> created and never edited afterwards. Iris does not resize, rechunk, or change
+> the dtype, `fill_value`, codecs or dimension names of an array that already
+> exists.
+
+That is what makes `iris.save(cubes, "new.zarr")` desynchronisation-free by
+construction rather than by luck: the chain is create-arrays, write-chunks,
+consolidate, with metadata written once at the front. It is stated here as a
+constraint on future work, not merely as a description of today's.
+
+**Writing into a store that already exists** is the case the invariant does not
+cover, and `save(..., group=...)` reaches it within the current scope.
+
+Read the next two paragraphs before the rules. `mode="a"` is easily read as a
+promise of something this design does not offer and should not.
+
+**Adding to a store is not appending to data.** `mode="a"` lets a second
+`iris.save` call put a *new* array, or a new group, into a store that already
+holds others. It never extends, rewrites, resizes or re-chunks an array that
+is already there. That is what the create-once invariant above forbids, and
+the invariant binds every mode. There is no `mode="a+"` and no `append=`
+keyword.
+
+The restraint is deliberate. Iris does not support appending to a netCDF file
+either, and the reason is not that nobody built it: an earlier attempt ran
+into the problem of deciding whether incoming data is *compatible* with what
+the file already holds — matching dimensions, coordinate values, attributes,
+units. That problem lives in the data model, not in the storage format. Zarr
+makes the write mechanically safer and solves none of it. §9 lists appending
+as out of scope so that `mode="a"` is not mistaken for a half-built version of
+it.
+
+Rules:
+
+- `save` takes `mode`, using zarr's vocabulary. The default is **`"w-"`:
+ create, and raise if the target already exists.** This deliberately differs
+ from the netCDF saver, which clobbers, because a Zarr clobber is many
+ non-atomic deletes: an interrupt destroys the old store without completing
+ the new one. `mode="w"` opts back in to replacement; `mode="a"` adds to an
+ existing store and is what `group=` needs. The error message names `mode`.
+- Under `mode="a"`, a name that already exists in the target group is an
+ **error**, not an overwrite and not a merge. The message names the array and
+ the group.
+- When Iris adds to a store that already carries consolidated metadata, it
+ **must** re-consolidate as its final act. If it cannot — the root is not
+ writable, say — it raises rather than returning, because leaving stale
+ consolidated metadata hides the data just written.
+- `finalise()` is always the last write. Under `compute=False` the returned
+ `Delayed` owns it, so consolidation follows the last chunk rather than
+ preceding it.
+
+**What Iris does not promise.** An interrupted save leaves a partial store, and
+Iris offers no torn-write detection or rollback. The usual mitigation — write
+to a temporary name and rename on success — is not adopted: directory rename is
+atomic on POSIX but object stores have no equivalent, so the guarantee would
+evaporate precisely where Zarr is most used, which is worse than not offering
+it. Callers needing transactional writes want Icechunk, which is out of scope
+(§9). The Zarr documentation section says so plainly rather than leaving users
+to discover it.
+
+**Tests.** Deleting a chunk object from a saved store and asserting the read is
+silently filled; adding an array to a consolidated store and asserting the
+default reader misses it until re-consolidation; and `mode="w-"` raising on an
+existing target. These pin format behaviour Iris depends on, so they should
+fail loudly if a zarr-python release changes it.
+
+#### Multi-process writes: kept reachable, not built
+
+Coordinated writing to one target from many processes is an Iris
+differentiator for netCDF and is explicitly wanted for Zarr. It is out of
+scope here (§9) purely for size. These are the provisions that keep it a
+later addition rather than a redesign.
+
+**zarr-python will not do it for us.** Version 3.4.0 dropped the version 2
+synchroniser machinery: `zarr.create_array` has no `synchronizer` parameter at
+all, and `zarr.open_group(synchronizer=...)` is accepted but warns
+`"synchronizer is not yet implemented"` **[verified]**. `zarr.ProcessSynchronizer`
+and `zarr.sync` no longer exist. Any coordination Iris offers must therefore be
+Iris's own, exactly as `_dask_locks.py` is for netCDF today. §9's bullet is
+worded accordingly.
+
+1. **`_dask_locks.py` moves to `cf/`, in PR 5.** Despite its netCDF-flavoured
+ docstring it imports only `threading` and four `dask` modules — **no netCDF
+ whatsoever** **[verified]**. It is scheduler-aware, file-identity-based
+ locking that is generic already. Relocating it costs one `git mv` and an
+ updated docstring, and it is the difference between a future Zarr
+ implementation reaching for an existing toolkit and growing a parallel one.
+
+2. **`CFDatasetVariable.write_handle()` is the coordination seam.** It returns
+ a picklable object supporting `__setitem__`, which is what a dask worker
+ receives. netCDF returns today's write proxy — in practice
+ `EncodedNetCDFWriteProxy` — which carries the `distributed.Lock` keyed on
+ the file path. Zarr returns the `zarr.Array`
+ itself — verified picklable, round-tripping shape and chunks under
+ `zarr 3.4.0` **[verified]** — because no lock is needed while the alignment
+ invariant holds. If Zarr later needs coordination, the Zarr implementation
+ returns a lock-carrying proxy instead: **no change to the interface and no
+ change to `cf/saver.py`**. This is the precise sense in which the design
+ does not preclude the feature.
+
+3. **The lock lives in the dataset implementation, never in `cf/saver.py`.**
+ The generic saver asks for write handles and stores into them; it holds no
+ lock and knows no scheduler. That is what lets the two backends coordinate
+ differently — and lets Zarr coordinate *better*, since a whole-store lock
+ would throw away the parallel throughput that is the reason to write Zarr.
+
+4. **`mode` is on `CFDataset` from the start.** The Zarr shape of this feature
+ is a region write: open an existing store `"r+"` and have each process fill
+ its own slab, rather than serialise every worker behind one lock. That needs
+ a dataset that can be opened for update without creating anything, which an
+ interface whose only verbs are `create_dimension` and `create_variable`
+ cannot express. Adding `mode` afterwards would be a breaking change to a
+ published ABC with two implementations and deprecation aliases pointing at
+ it; adding it now is one attribute.
+
+Nothing here builds the feature. Together they mean building it later is new
+code in `zarr/_dataset.py` plus a keyword on `save`, with the generic layer
+and the netCDF path untouched.
+
+The durability hazards above get sharper under parallelism, and the
+create-once invariant is most of the answer there too: workers that only fill
+chunks of arrays created up front never touch metadata, so the one shared
+mutable object is the consolidated metadata, written once at the end by the
+process that owns `finalise()`. A design that let workers create or resize
+arrays would have no such property, which is a further reason the invariant is
+written down rather than assumed.
+
+#### Encoding keywords
+
+**The Zarr saver takes Zarr's vocabulary, not netCDF's.** `compressors=`,
+`chunks=`, `shards=`, `filters=`, `serializer=` — the names zarr-python uses —
+plus `consolidate_metadata=True` (default `True`, since consolidation is what
+makes remote stores usable, with the zarr-python warning suppressed and
+explained). A netCDF keyword passed to the Zarr saver raises `TypeError`
+naming the Zarr equivalent where there is one, and saying there is none where
+there is not.
+
+**Do not translate netCDF keywords onto Zarr codecs** — `zlib`/`complevel`
+onto Blosc or Gzip, `shuffle` onto Blosc shuffle, `chunksizes` onto `chunks`,
+`fletcher32` onto Crc32c. The translation would make the Zarr saver inherit a
+shape the netCDF saver is stuck with, and it is a shape worth not repeating:
+Iris names every netCDF `createVariable` keyword in its own save signature and
+tunnels each one through several call layers that have no interest in it.
+`CHUNK_CONTROL` shows the better pattern, a context manager setting
+per-variable controls. Reworking the netCDF side is a breaking change with a
+wide blast radius and is
+[#7328](https://github.com/SciTools/iris/issues/7328), not part of this
+programme — but the Zarr saver is new API with no users and no compatibility
+debt, so it starts clean.
+
+Translating would also sell a false equivalence. `zlib=True` and a Blosc codec
+are not the same decision, and a user who writes `complevel=4` expecting zlib
+semantics and gets Blosc has been misled by the keyword that let them say it.
+
+The cost is that `iris.save(cube, "out.nc", zlib=True)` and
+`iris.save(cube, "out.zarr", zlib=True)` do not both work, which is a real
+cost for someone switching format. It is paid once, at the point where the
+compression model genuinely differs, and the `TypeError` names the replacement.
+
+### 4.6 Groups
+
+`load_cubes(..., group=None)` and `save(..., group=None)` both default to the
+root group. A non-`None` `group` is a `/`-separated path within the store.
+
+Writing to a non-root group means opening a store that may already exist, so
+`save(..., group=...)` is governed by the existing-store rules in §4.5 — it
+implies `mode="a"`, and re-consolidation is mandatory.
+
+Iris loads **one** group per call, like xarray. Nested groups are not walked:
+a Zarr store can legitimately hold unrelated datasets, and merging them into
+one `CubeList` would be wrong. `iris.load("store.zarr")` therefore returns the
+cubes in the root group, and a store whose root group holds no data variables
+raises a message that names the groups that do.
+
+### 4.7 Format detection and the load/save chain
+
+A new `FormatSpecification` named `"Zarr"` is registered with a new
+`ZarrStore` file element that, like `UriProtocol`, sets `requires_fh = False`
+so it is consulted when no file handle is available. It matches when:
+
+- **locally**, the target is a directory containing `zarr.json` (version 3) or
+ `.zgroup` (version 2);
+- **remotely**, the URI scheme is one of `s3`, `gs`, `az`, `abfs` and the path
+ ends in `.zarr`.
+
+The split is local-versus-remote, and the two tests are not alternatives for
+the same target.
+
+**The element answers `None` when a file handle is supplied**, so the
+specification cannot match a target that opened as a file. `requires_fh =
+False` means "consult me without a handle", not "consult me with one". Two
+things go wrong otherwise. A store is a directory or a URL, so a target with a
+handle is settled already. And the caller that opened the file —
+`iris.io._load_files` — passes `pathlib.Path(fn).name`, a basename rather than
+a path, which resolves against the working directory: a NIMROD file named
+`weather` was classified as Zarr whenever the working directory happened to
+contain a `weather/zarr.json`, the priority-5 Zarr specification being
+consulted before priority-3 NIMROD.
+
+**Do not add a third rule matching any path that ends in `.zarr`.** Joined by
+"or" it swallows the remote rule whole, leaving the scheme test unreachable,
+and it is the wrong rule locally in any case. Reading the marker file is what
+distinguishes a real store from a directory someone happened to name `.zarr`,
+and locally that costs one `stat`. Remotely it costs a network round trip
+before Iris knows it wants the target at all, so there the suffix is all there
+is to go on, and a remote `.zarr` that turns out not to be a store fails when
+zarr-python opens it, with zarr-python's message.
+
+Four changes make the chain reach that specification:
+
+1. `iris.io.expand_filespecs` leaves a spec with a recognised remote scheme
+ alone instead of absolutising it.
+2. A new `iris.io.load_stores` handles every target that is a store rather
+ than a file — a local directory or a remote URL.
+ `iris.loading._generate_cubes` routes the remote schemes to it instead of
+ raising, and routes a local path to it when the path is a directory.
+3. `iris.io.find_saver` matches `.zarr` with or without a trailing separator,
+ since `iris.save(cube, "out.zarr/")` is a natural thing to type.
+4. `iris.io.save` gets a module-level `_CUBELIST_SAVER_MODULES` frozenset
+ naming the modules whose `save` accepts a `CubeList` directly, replacing the
+ `"iris.fileformats.netcdf" not in saver.__module__` substring test at
+ io/__init__.py:530. Explicit and greppable; one line per new format.
+
+**`load_files` is not taught about directories.** Do not have it skip its
+`open(fn, "rb")` and pass `None` as the buffer when the path is a directory.
+A separate routine is both safer and more obvious: `load_files` keeps the
+invariant that every path reaching it is a readable file, and "this target is
+not a file" gets answered in one place for local and remote stores alike,
+rather than once in `load_files` and again in the scheme dispatch.
+
+The one thing this costs is that the file-or-store decision has to happen
+*after* glob expansion and *before* format dispatch, because `iris.load` must
+still accept `"data/*"` matching a mix of `.nc` files and `.zarr` stores. And
+it must happen only **once**: `expand_filespecs` keys its own glob cache by
+the spec string, so expanding an already-expanded list collapses two
+identical paths matched by two different patterns into one. `load_files`
+cannot simply receive the partitioned, already-expanded list, because its
+public signature promises a glob-accepting `filenames` to every other caller.
+So the expansion stays behind that signature — `load_files` expands once and
+delegates to a private `_load_files`, which expects a list already expanded
+— and `_generate_cubes` calls `_load_files` directly, with the list it has
+already expanded for itself and partitioned between `_load_files` and
+`load_stores`. Neither routine learns about the other's kind of target.
+
+The duplicated dispatch in `netcdf/ugrid_load.py:156-183` is updated in step
+with `loading.py`, so `iris.mesh.load_meshes` accepts Zarr stores too. It also
+has to accept the Zarr *handler*, not only the Zarr scheme: the format spec it
+gets back for a store names `iris.fileformats._load_zarr`, and a test against
+`nc_loader.load_cubes` alone sends every store down the "ignore this" branch
+while the scheme dispatch above says the store was fine.
+
+**NCZarr is unaffected, and native Zarr is already the default.** No
+precedence rule is needed to make it so, because the native reader and the
+NCZarr one are never candidates for the same target **[verified]**:
+
+- NCZarr is reached only through an explicit URL fragment —
+ `file:///path/store.zarr#mode=nczarr,file`. `_is_nczarr_fragment`
+ (`io/__init__.py:24`) is what selects it, on load (`loading.py:88`) and on
+ save (`io/__init__.py:505`). A user who writes that has asked for NCZarr by
+ name.
+- A plain path has no fragment, so it never reaches NCZarr. At the baseline it
+ reaches nothing at all: `iris.load("store.zarr")` raises `IsADirectoryError`
+ and `iris.save(cube, "out.zarr")` raises
+ `ValueError("Cannot save; no saver")`, because `_savers` has no `.zarr`
+ entry. Both become native Zarr.
+
+So the native path takes only what was an error at the baseline. Nothing that
+worked changes meaning, nothing has to be deprecated to let the native reader
+win, and `_uri_is_nczarr` is left exactly as it is.
+
+`docs/src/user_manual/tutorial/s3_io.rst` currently says S3 is unsupported and
+recommends FUSE mounting. It is updated to describe the Zarr route.
+
+### 4.8 Deprecations
+
+These names move and gain warn-on-use aliases at their old locations, following
+the pattern in `iris/experimental/ugrid.py`:
+
+| Old name | New name |
+|---|---|
+| `iris.fileformats.netcdf.loader.CHUNK_CONTROL` | `iris.fileformats.cf.loader.CHUNK_CONTROL` |
+| `iris.fileformats.netcdf.loader.ChunkControl` | `iris.fileformats.cf.loader.ChunkControl` |
+| `iris.fileformats.netcdf.CFNameCoordMap` | `iris.fileformats.cf.saver.CFNameCoordMap` |
+| `iris.fileformats.netcdf.CF_CONVENTIONS_VERSION` | `iris.fileformats.cf.saver.CF_CONVENTIONS_VERSION` |
+| `iris.fileformats.netcdf.MESH_ELEMENTS` | `iris.fileformats.cf.saver.MESH_ELEMENTS` |
+| `iris.fileformats.netcdf.SPATIO_TEMPORAL_AXES` | `iris.fileformats.cf.saver.SPATIO_TEMPORAL_AXES` |
+
+`CHUNK_CONTROL` is the one that matters: it is genuinely shared, and users
+reach it by module path. The alias is the **same object**, not a copy, so
+existing code keeps working including inside a `with CHUNK_CONTROL.set(...)`.
+
+These names do **not** move, because they are netCDF API and should stay that
+way: `netcdf.save`, `netcdf.Saver`, `netcdf.load_cubes`, `netcdf.save_mesh`,
+`netcdf.NetCDFDataProxy`, `netcdf.DEBUG`.
+
+Warnings are emitted on **use**, not on import, so simply importing
+`iris.fileformats.netcdf` stays quiet. The deprecation is marked
+`.. deprecated:: 3.17` and follows NEP 29.
+
+---
+
+## 5. The programme
+
+Nine pull requests into `brownfield`, each opened separately so that every
+step gets its own CI signal and can be verified on its own, followed by a
+single merge-back into `main`. PR 2.5 was inserted after PR 2 had merged, so
+the later numbers are unchanged from the original seven. Each carries the `Agentic` and
+`Type: Feature Branch` labels and attributes the contribution to Claude. This
+is the delivery plan, not a rehearsal for one: the pull requests below are the
+work, and the merge-back is how it reaches users.
+
+**Issues resolve at merge-back, not before.** A GitHub closing keyword only
+fires when the pull request merges into the repository's *default* branch, so
+`Closes #6977` on a pull request based on `brownfield` would link the issue and
+never close it. The feature-branch pull requests therefore write
+`Part of #6977`, and the merge-back pull request carries the closing keywords
+for the three sub-issues and for any defect a feature-branch pull request
+fixes along the way. This is bookkeeping, not caution: #6977, #6979, #6980 and
+#7337 are genuinely resolved by this programme, just not one branch merge
+earlier than they should be. Issue *state* is not tracked in this document at
+all — that lives on GitHub (§12.2).
+
+**Changelog fragments land with the merge-back.** Fragments are named
+`<PR-number>.<type>.rst` (`changelog/AGENTS.md`), and a fragment numbered for a
+feature-branch pull request would advertise a change that `main` has not yet
+received. Established practice on this branch agrees: the two pull requests
+already merged into `brownfield`, #7285 and #7287, carry no fragment, while
+#7267 into `main` does **[verified]**. Each feature-branch pull request body
+therefore states that its fragment is deferred, so the omission does not read
+as an oversight, and §5's merge-back subsection lists the fragments owed.
+
+### PR 1 — `iris.fileformats.cf` becomes a package, with tests first
+
+`git mv cf.py cf/_variables.py` and split out `_group.py` and `_reader.py`
+along the boundaries in §4.1, with an `__init__.py` that
+re-exports the existing public names. Imports updated across the tree. **No
+behaviour change and no API change**: the diff is a move plus import edits, so
+a reviewer can skim it.
+
+Brings in the unit test coverage from SciTools/iris#7259 by Martin Yeo, rebased
+onto `brownfield` and credited in the pull request body.
+That branch is named `cf_reader_zarr` and was written for exactly this
+refactor; it adds roughly 2300 lines of coverage across nineteen test modules.
+Landing it *here*, against today's behaviour, is what makes PR 2 reviewable:
+the tests become a regression net that was written before the rewrite and does
+not move with it. A comment on #7259 explains the overlap before the pull
+request is opened.
+
+### PR 2 — `CFDataset`, and the CF variable classes rewritten against it
+
+The subject of **#6977**. Adds `cf/dataset.py` and `netcdf/_dataset.py`. Rewrites the
+86 `getattr`/`hasattr` sites and the 57 netCDF-API sites in the `cf` package,
+`_nc_load_rules/` and `netcdf/` to go through the new interface. Rewires
+`CFVariable.__getattr__` onto `.attributes`, with the deprecating
+fallback and the attribute-read tracking of §4.3. Fixes the
+`_NCZARR_SCALAR_DIMENSION` gap in the three overriding `spans` methods.
+
+No behaviour change other than the `spans` fix. The suite from PR 1 must pass
+untouched.
+
+### PR 2.5 — Deprecate `CFVariable.__getattr__`
+
+Inserted after PR 2 had merged (§12.4). `__getattr__` keeps resolving CF
+attributes through `.attributes`, and warns when it does, naming `.attributes`
+and the key. `IrisDeprecation`, `.. deprecated:: 3.17`, removal one cycle
+later under NEP 29. §4.3.
+
+Behaviour only, and deliberately tiny. It is its own pull request because PR 2
+has merged and because the programme's rule is that no pull request both moves
+code and changes behaviour — folding it into PR 3, a 1000-line relocation,
+would bury the one line a reviewer needs to look at.
+
+A normal load must stay silent, and four reads inside Iris stood in the way:
+`bounds` and `climatology` in `_nc_load_rules/helpers.py`, `cf_role` and
+`start_index` in `netcdf/ugrid_load.py`. The pull request moves them to
+`.attributes` before it can claim silence. Finding them took instrumenting
+`__getattr__` to log its caller and running the whole suite; grep found two of
+the four.
+
+The test that matters loads a file end to end, records the warnings, and
+asserts no `IrisDeprecation` among them. It must not promote the warning to an
+error instead. `iris.load` captures an exception raised while building a cube
+into `iris.loading.LOAD_PROBLEMS`, so a filter of `error` drops the offending
+coordinate and the load returns silent and degraded — a test written that way
+passes against an implementation that still warns. The test therefore also
+asserts the file contributed no `LOAD_PROBLEMS` entry. A second test reaches
+`cf_var.units` directly and does warn.
+
+`IrisDeprecation` lives in `iris._deprecation`, and is re-exported from
+neither `iris.exceptions` nor `iris.warnings`. A `-W` filter naming either of
+those matches nothing and fails open.
+
+### PR 3 — Relocate the CF loader
+
+`git mv netcdf/loader.py cf/loader.py`, leaving a `netcdf/loader.py` that holds
+the netCDF specifics and the deprecating aliases. Still netCDF-only behaviour;
+the whole existing suite passes unchanged. Large, mechanical, independently
+verifiable.
+
+Also adds the `cache_key=` keyword to `as_lazy_data` and deletes the
+`isinstance(data, NetCDFDataProxy)` special case in `iris/_lazy_data.py`
+(§4.4), removing that module's import of `iris.fileformats.netcdf`. The
+netCDF loader passes the same `repr(proxy)` key it produces today, so this
+too is behaviour-preserving — and a test asserts the key is unchanged.
+
+### PR 4 — Zarr loading
+
+The subject of **#6979**. Adds `zarr` to the optional dependencies section of
+`requirements/py3{12,13,14}.yml` and regenerates the lock files, once, here.
+Adds `fileformats/zarr/` with `_dataset.py`, `_decode.py` and `loader.py`, the
+format specification, and the four load-chain changes in §4.7 including the new
+`iris.io.load_stores`. Documents Zarr loading in the user guide and the
+`iris.fileformats` API reference, including the remote-store URL forms and the
+`group=` keyword.
+
+Carries the §6 read tests for the rules that are easiest to get wrong: the
+version-aware masking split, both `_FillValue` encodings, the inner-chunk read
+unit on a sharded array, and the cache-key staleness and sensitivity cases.
+
+Also carries three changes to generic code that only Zarr can exercise:
+
+- `dimensions` becomes `tuple[str | None, ...]`, and `CFVariable.spans` and
+ its three overrides match on `(name, size)` pairs with an anonymous
+ dimension matching nothing (§4.4). netCDF behaviour is unchanged by
+ construction, and the tests say so on both backends.
+- `encoding` on `CFDataset` and `CFDatasetVariable` (§4.2), reporting what the
+ opened store actually holds.
+- `format` on `CFDatasetVariable` (§4.2), naming the concrete Zarr version so
+ the version-aware masking split (§4.4) can tell `ZARR_V2` from `ZARR_V3`.
+ `CFDataset.format` arrived with PR 5, which merged first; this pull request
+ adds the variable's half and `ZarrDataset`'s answer to both it and
+ `supports_dtype` (§12.4).
+
+**UGRID meshes load too**, through
+`iris.fileformats.netcdf.ugrid_load`. Nothing in that module is netCDF-specific
+— it reads a `CFReader`, and `CFReader` over a `ZarrDataset` answers the same
+questions — so the Zarr loader calls into it rather than growing a second copy.
+The mesh block that was inline in `netcdf/loader.py` becomes
+`_prepare_mesh_coords` and `_attach_mesh_coords`, two functions because the
+MeshCoords have to be built *before* `_load_cube` and attached *after* it: built
+after, `helpers._add_unused_attributes` has already taken the data variable's
+`mesh` and `location` attributes and left them on the cube as ordinary
+attributes. One netCDF detail survives the move: the "mesh could not be found"
+message keeps being raised on the `iris.fileformats.netcdf` logger rather than
+`ugrid_load`'s own, which sits at `INFO` and would drop it.
+
+After this pull request, `iris.load("store.zarr")` and
+`iris.load("s3://bucket/store.zarr")` work for version 2 and version 3 stores,
+mesh data included.
+
+### PR 5 — Relocate the CF saver
+
+`git mv netcdf/saver.py cf/saver.py`, same shape as PR 3. The largest code diff of
+the nine, and the relocated module is a pure move, which is precisely why it
+is on its own. The branch carries one behaviour change beside the move, in a
+netCDF-only module the move does not touch: the #7337 fix below.
+
+The generic half is `_CFSaver`, built from an open `CFDataset`, and private.
+`iris.fileformats.netcdf.saver.Saver` subclasses it, supplying the netCDF
+vocabulary of storage keywords and four hooks: `_create_string_variable`,
+because netCDF stores unicode as bytes over a string dimension;
+`_backend_write`, which claims the writes netCDF does itself, to an emulating
+variable or to an NCZarr target that cannot be reopened; `_flush_before_close`,
+which performs the second of those before the file closes; and `_patch_dataset`,
+which hands `iris.site_configuration`'s `cf_patch` hook the unwrapped netCDF4
+objects it has been documented as receiving since Iris 1.3. A Zarr saver
+implements the first and inherits the rest.
+
+Also `git mv`s `netcdf/_dask_locks.py` to `cf/_dask_locks.py` and rewrites its
+netCDF-specific docstring. The code imports only `threading` and four `dask`
+modules **[verified]**, so this is a relocation with no behaviour change — it
+puts the scheduler-aware locking toolkit where a future Zarr implementation
+can reach it (§4.5).
+
+Adds `CFDataset.format` and `CFDataset.supports_dtype` (§4.2), and asks the
+latter wherever relocated generic code would otherwise test netCDF format
+strings. Gathering the question into one method is what makes #7337 a
+one-constant fix: `NetCDFDataset` answers from a list of the formats that
+cannot hold every dtype, and that list had been missing the name netCDF4
+reports for a 64-bit-offset file. Fixed here rather than carried across
+(§12.4, 2026-10-16).
+
+### PR 6 — Zarr saving
+
+The subject of **#6980**. Adds `zarr/saver.py`, the `ZarrDataset` write path, JSON
+attribute conversion, masked-data filling, deferred writes, the Zarr-native
+encoding keywords of §4.5 and `consolidate_metadata`. Registers the `zarr`
+saver, and documents saving alongside the loading documentation from PR 4,
+including the S3 pages.
+
+Includes a test that `iris.save(cubes, "out.zarr", compute=False)` returns a
+`Delayed` whose `.compute()` completes the store — the deferred save that the
+NCZarr path cannot offer (§4.5), and the contract the netCDF path is currently
+breaking (§12.3 Q5).
+
+Also carries the §6 write tests for the rules that silently lose data without
+them: the write-alignment invariant on a sharded target, the
+existing-store refusals, the Zarr-native encoding keywords, and the
+bidirectional cross-reader conformance test — xarray must open every store
+Iris writes.
+
+After this pull request, `iris.save(cubes, "out.zarr")` works.
+
+### PR 7 — Real-world test data and benchmarks
+
+Integration tests against the cut-down NOAA GFS and ESA EOPF samples, plus ASV
+coverage: `ZarrSave` beside `NetcdfSave` in `benchmarks/benchmarks/save.py`,
+and Zarr variants of `LoadAndRealise` in `benchmarks/benchmarks/load/`.
+Depends on a companion pull request to `SciTools/iris-test-data`.
+
+### PR 8 — The terms-of-art sweep
+
+`.github/WRITING.md` bans introducing a term of art that appears exactly once
+in the repository. This document and the plans under `docs/superpowers/plans/`
+were written before that rule existed, so they may carry coinages it would now
+reject. The cost is not that they read oddly. A later agent decides whether a
+term is safe by checking whether the repository already uses it, so a coinage
+that reaches `main` becomes indistinguishable from an established term and
+gets copied in good faith. Removing them before the merge is the last point at
+which that is cheap.
+
+The sweep covers every prose line the branch adds: run `git diff
+main...brownfield` over this document, the plans, the agent instruction files,
+docstrings, comments, changelog fragments and the Zarr user documentation
+written in PRs 4 and 6. For each candidate term, grep `main`. A term that
+appears only on lines this branch added is a coinage — replace it with words
+Iris already uses, or keep it and record why it earns its place.
+
+Last of the nine because it reads the finished branch. It changes prose only;
+a diff that touches behaviour belongs in the pull request that introduced the
+behaviour.
+
+### Merge back — `brownfield` into `main`
+
+Labelled `Type: Merge Back`. This is where the programme becomes a release, and
+where three obligations deferred by the feature-branch pull requests are met.
+
+**Closing keywords** for #6977, #6979, #6980 and #7337.
+
+**Changelog fragments**, numbered for the merge-back pull request. Per
+`changelog/AGENTS.md`, one pull request may own several fragments of different
+types, which is what this needs:
+
+| Type | Covers |
+|---|---|
+| `feature` | Native Zarr version 2 and version 3 reading, and version 3 writing |
+| `dependency` | `zarr` added as an optional dependency; xarray as a test dependency |
+| `deprecation` | The relocated public names in §4.8, and `CFVariable.__getattr__` (§4.3) — all warn on use |
+| `bugfix` | Attributes whose name collides with a netCDF4 Python member — `shape`, `size`, `dtype`, `name`, `dimensions`, `mask` — are no longer silently dropped on save, nor silently shadowed on load (§12.4, 2026-09-24). Saving int64 data to `NETCDF3_64BIT` narrows it to int32 instead of failing inside the netCDF C library (#7337) |
+| `internal` | The `cf` package split and the CF loader/saver relocation |
+
+Every fragment credits `` :user:`claude` `` per `changelog/AGENTS.md`.
+
+**Documentation** is written in PRs 4 and 6, alongside the code it describes,
+and becomes visible to users only here. The merge-back checks it builds clean
+against `main` with `-W`, since the two branches can drift.
+
+**Issues to raise, not to act on.** One deliberately deferred follow-up,
+raised *here* rather than when it was found, so that nobody starts on it while
+the programme's pull requests are still in flight and depending on each other:
+
+| What | Why it waits |
+|---|---|
+| Unify the cube-building loops in `netcdf/loader.py` and `zarr/loader.py`, and with them the format dispatch `load_meshes` duplicates (§2.3) | Raised by @pp-mo reviewing PR 4. Since PR 4 gave the Zarr loader the same mesh handling, the two bodies differ only in how the dataset is opened. The duplicated dispatch is the same defect wearing a different hat, and is the only reason `netcdf/ugrid_load.py` names Zarr: the mesh code it shares with the Zarr loader names no format at all. A `TODO` in that function points here. The refactor is internal — neither `load_cubes` signature moves — so it is safe to do later and disruptive to do now |
+
+**Ordering rationale.** Refactor and feature alternate so that no feature pull
+request is large, and every relocation is a separate, skimmable diff. Tests
+land before the rewrite they protect. Loading lands before saving because it is
+the higher-value half if the programme is cut short. PR 1 to PR 3 are
+independent of any Zarr decision and could merge before the rest is agreed.
+
+---
+
+## 6. Testing
+
+Per `lib/iris/tests/AGENTS.md`: pytest style, no network, no unittest classes
+in new files.
+
+**Unit.** `lib/iris/tests/unit/fileformats/zarr/` mirrors the module layout.
+The `CFDataset` contract gets one shared test body run against both
+implementations, so a netCDF/Zarr divergence fails loudly. Stores are built
+in-memory or in `tmp_path`.
+
+That shared body is also where `format`, `encoding` and `supports_dtype` are
+pinned, since the point of each is that it works the same way whatever is
+underneath: `format` names the concrete format; `encoding` reports the
+settings an opened store actually holds, and round-trips what
+`create_variable` was given; and `supports_dtype` answers `False` for `int64`
+on `NETCDF3_CLASSIC` and `True` on `NETCDF4` and on Zarr. A separate check
+greps the `cf` package for `isinstance` against an implementation class and
+asserts there are none — the rule in §4.2 is only worth writing down if
+something notices when it is broken.
+
+**Integration.** `lib/iris/tests/integration/zarr/` covers round trips through
+`stock.realistic_4d_w_everything()`, following the shape of the existing
+`integration/netcdf/test_nczarr.py`.
+
+**Chunking and decoding** are unit-testable without a store, which is the main
+practical dividend of dropping the proxy (§4.4). Each CF decode rule is a pure
+function of `(block, parameters)` and is tested on plain ndarrays. The
+alignment guard is tested on the `chunking` values alone:
+
+- a store chunk smaller than the dask target expands to a whole multiple;
+- a store chunk larger than the dask target is **not** subdivided, and warns;
+- an explicit `CHUNK_CONTROL` setting that subdivides is honoured, and warns;
+- the resulting dask chunks tile the store chunk grid exactly, in every case;
+- a **sharded** array chunks on `Array.chunks`, not `Array.shards`, and a
+ partial read of one inner chunk does not fetch the whole shard — asserted on
+ bytes fetched through an instrumented store, with the decode layer present,
+ since that layer is what defeats slice pushdown (§4.4).
+
+**Fill values and masking** (§4.4) get the cases that the earlier precedence
+got wrong, each on a store built in `tmp_path`:
+
+- an `int16` version 3 array of `[0, 1, 2]` with **no** CF attributes and the
+ default `fill_value=0` loads with **nothing masked**;
+- a CF `_FillValue` that differs from the storage fill masks the CF value and
+ not the storage one;
+- the same array written as version 2 **does** use `fill_value` as the mask
+ source, pinning the version split;
+- a floating `_FillValue` reads correctly from both the base64 form and a plain
+ JSON number, including a finite sentinel rather than only NaN;
+- a `_FillValue` that is neither warns, names the variable, and loads;
+- a **masked cube round-trips its mask** through save and load, which is the
+ test that catches a saver setting the storage `fill_value` without also
+ writing the `_FillValue` attribute (§4.5).
+
+**Write alignment** (§4.5) is tested where it bites: a sharded target whose
+shards are larger than its chunks, stored from a source aligned to the inner
+chunk, must either be rechunked to the shard first or refused — never written
+unaligned. The regression case is the reproduction in §4.5, which fails 12
+times in 12 without the fix, so a single-trial test is enough to catch it.
+
+**Writing into an existing store** (§4.5) is tested for what it refuses as
+much as for what it does: `mode="w-"` raises on an existing target;
+`mode="a"` with `group=` adds a group and leaves the existing one byte-for-byte
+intact; `mode="a"` onto a name that already exists in the target group raises
+and names it; and re-consolidation after an add is asserted by opening the
+store with a default reader and seeing the new group. There is no append test
+because there is no append.
+
+**Encoding keywords** (§4.5) get one test each way: the Zarr-native names
+reach the array (`compressors`, `chunks`, `shards` are readable back off
+`Array.metadata`, and through `CFDatasetVariable.encoding`), and a netCDF name
+such as `zlib=True` raises `TypeError` whose message names the Zarr
+replacement. The second is the one that stops the translation layer growing
+back by accident.
+
+**Cross-reader conformance** (§4.5) is bidirectional and is the check that
+would have caught the `_FillValue` encoding error: xarray opens every store
+Iris writes — asserted on the dataset open, since the failure mode is a
+`TypeError` during open — and Iris loads a store xarray wrote, for `float32`
+NaN, a finite floating sentinel and an integer sentinel.
+
+**The dimension contract** (§4.4) is tested as the contract, not as a list of
+stores, because every case below is conforming Zarr that a reader may meet:
+
+- an array with no `dimension_names` at all loads, with anonymous dimensions;
+- an array with *some* null entries loads, naming the dimensions that are
+ named and leaving the rest anonymous;
+- two arrays sharing a name at the same length share a dimension, and their
+ coordinates attach;
+- two arrays sharing a name at **different** lengths do not, and the dataset
+ warns rather than raising, and neither array gets the other's coordinates;
+- two anonymous dimensions of the same length do not match each other, within
+ one array or across two;
+- a `dimension_names` whose length is not the array's rank warns and loads
+ anonymous;
+- the same `spans` suite runs against netCDF, where it must give identical
+ answers, since `(name, size)` matching is a widening and not a change.
+
+**The `cache_key=` change** (§4.4, PR 3) is pinned by asserting that the netCDF
+loader produces the identical key to today's `repr(proxy)`, so the removal of
+the `isinstance` branch in `iris/_lazy_data.py` cannot silently lose the
+array-sharing it exists to provide. The Zarr key gets the staleness cases: a
+store rewritten in place with a changed shape, and one with a changed
+`fill_value` over unwritten chunks, must each miss the cache.
+
+It also gets the sensitivity cases that make `Array.metadata.to_dict()` safe to
+rely on without the Zarr specification defining it (§4.4): the key is stable
+across a reopen of an unchanged array, and changes when each of shape, chunks,
+shards, dtype, `fill_value`, codecs, `dimension_names` and attributes changes.
+These are the tests that would fail if a zarr-python release stopped carrying
+one of those, which is the whole reason for keying on an opaque document.
+
+Note that the `["nczarr", "xarray"]` parametrisation in that module is **not**
+about the xarray package: both are netCDF-c NCZarr URL modes, and `xarray` is
+the mode that writes `_ARRAY_DIMENSIONS`.
+
+**Cross-reader testing.** xarray is added as a **test-only** dependency, so
+that a handful of integration tests can assert that xarray reads what Iris
+writes. This is the check that justifies the JSON-native attribute decision in
+§4.5: if xarray cannot read Iris output, the decision was wrong. The cost has
+been measured — `conda install -n iris-dev -c conda-forge xarray` adds
+**exactly one package** and updates nothing **[verified]**, because xarray's
+runtime requirements (NumPy, pandas, packaging) are already Iris dependencies.
+No `skip_gdal`-style guard is therefore needed, and xarray goes into the test
+section of `requirements/py3{12,13,14}.yml` alongside the other test-only
+packages. It lands in PR 6, with the saver it validates.
+
+**Version 2 reading** is tested two ways. Synthetic fixtures are written by
+zarr-python with `zarr_format=2` and `_ARRAY_DIMENSIONS` set by hand, since
+zarr-python refuses to write version 2 dimension names itself **[verified]**.
+Separately, the existing NCZarr `mode=xarray` save path already produces a
+version 2 store with `_ARRAY_DIMENSIONS`, so it generates realistic version 2
+fixtures with no new dependency, and gives a direct check that the native
+reader and the NCZarr writer agree.
+
+**Remote stores** are covered by an in-memory store and a local directory
+store only. No test touches the network. The URL path is exercised by checking
+that the right store is constructed, not by fetching.
+
+**Skipping.** A single `pytest.importorskip("zarr")` fixture in the Zarr test
+package. Iris without zarr installed must show no new failures.
+
+---
+
+## 7. Real-world test data
+
+Two sources, both cut down before contribution to `SciTools/iris-test-data`
+(fork `bjlittle/iris-test-data`, default branch `master`).
+
+**NOAA GFS, dynamical.org** — the version 3 fixture. Confirmed by fetching
+`https://data.dynamical.org/noaa/gfs/forecast/latest.zarr/zarr.json`:
+`zarr_format: 3`, 33 consolidated members, CF attributes including
+`standard_name`, `units`, `axis`, `calendar` and `_FillValue`, version 3
+`dimension_names`, sharded Blosc/zstd codecs, and a scalar `spatial_ref`
+variable carrying `crs_wkt`. Licensed CC BY 4.0; the attribution string and
+DOI 10.5281/zenodo.18777399 travel with the data.
+
+**This endpoint is documented as closing on 30 September 2026.** The subset has
+therefore already been cut, ahead of the rest of the programme; §12.5 holds its
+current location. It is 1.4 MB, 35 files, 12
+members, 2 forecast initialisations x 8 lead times x 120 x 120 points, for
+`temperature_2m`, `wind_u_10m`, `wind_v_10m` and `precipitation_surface`. It
+keeps the sharding codec, the scalar `spatial_ref` grid mapping, the
+two-dimensional `valid_time` auxiliary coordinate, all source attributes
+verbatim including the base64 `_FillValue`, and the `CC-BY-4.0` licence and
+attribution in the root group.
+
+Two of those are load-bearing rather than incidental. The **sharding codec**
+makes this the fixture that exercises the read-unit rule in §4.4 and the
+write-alignment invariant in §4.5, the two rules in this design most easily got
+wrong; keeping it is what stops those regressions being caught only in
+synthetic tests. The **base64 `_FillValue`** is retained as the cross-reader conformance
+case — it is xarray's version 3 encoding, not a producer defect (§4.4).
+
+**ESA EOPF Sentinel samples** — the version 2 and deep-group-hierarchy
+fixture, exercising the `group=` keyword and the `_ARRAY_DIMENSIONS` path.
+
+**Confirmed version 2 by reading the store** **[verified]**. For a Sentinel-2
+L2A product under
+`https://data.eodc.eu/collections/EOPF_ZARR/products/cpm_v270/S02MSIL2A/`,
+`zarr.json` returns HTTP 404 while `.zgroup` returns HTTP 200 containing
+`{"zarr_format": 2}`; opening the root with zarr-python reports
+`zarr_format: 2`. Dimension names are carried on `_ARRAY_DIMENSIONS`
+throughout. The service is migrating to version 3; until it does, this is a
+more realistic version 2 sample than anything synthetic.
+
+Two properties of this store earn it a place beyond "it is version 2":
+
+- **The root group holds no arrays at all**, only subgroups:
+ `conditions/geometry`, `conditions/mask/detector_footprint/{r10m,r20m,r60m}`,
+ `conditions/meteorology/{cams,ecmwf}` and
+ `measurements/reflectance/{r10m,r20m,r60m}`. Loading the default root group
+ yields nothing, which makes the §4.6 design — raise a message naming the
+ groups that *do* hold data — a necessity rather than a nicety.
+- It uses `<U7` and `<U3` fixed-length unicode dtypes, which #6961 notes are
+ not strictly Zarr-supported. A real store already contains them, so the
+ reader needs a defined answer.
+
+The `measurements/reflectance/r10m` group holds `b02`, `b03`, `b04` and `b08`
+as `(10980, 10980)` `uint16` with `fill_value` 0, plus `x` and `y` coordinate
+arrays — so a cropped window of one or two bands is the natural cut-down.
+Each band carries `scale_factor`, `add_offset`, `units`, `long_name`,
+`valid_min` and `valid_max` as flat, correct CF attributes — so this fixture
+exercises the `zarr/_decode.py` unpacking path on real data — **and** a
+redundant `_eopf_attrs` object duplicating them one level down
+**[verified]**. The nested object is producer bookkeeping, not a relocation of
+the CF metadata. It is the §4.4 nested-attribute case in its mildest form:
+Iris reads the flat attributes and must simply carry the nested one through
+without choking on it.
+
+Target size is a few megabytes each: one or two variables, a handful of time
+steps, a cropped spatial window, with the group structure and codec variety
+preserved because that is what is being tested.
+
+---
+
+## 8. Risks
+
+| Risk | Mitigation |
+|---|---|
+| dynamical.org closes before the subset is pulled | Pull within days; the design does not otherwise depend on it |
+| PR 5 is a 3000-line diff | Pure `git mv` plus import edits, no behaviour change, full suite green |
+| CF-Zarr conventions are still a moving target | Iris writes plain unprefixed CF attributes, which is what the draft and xarray both do; no `zarr_conventions` block is written until the specification settles |
+| zarr-python uses Effective Effort Versioning, not semantic versioning | Pin `>=3.0.8`, which is above the known data-loss bug, and rely on CI to catch drift |
+| Attribute dtype loss surprises someone | Documented; round-trip tests assert values not dtypes; xarray reads the output (§6); base64 remains available as a later opt-in |
+| Designing against one publisher's files produces brittle code | Every behaviour in §4 is justified from the Zarr and CF specifications first; a real file is cited only as evidence that a path will be exercised. Where a file is non-conforming it is labelled a malformation and handled with a warning, never by bending the reader (§4.4) |
+| Mistaking a de-facto convention for a publisher's mistake — the inverse risk, and the one that actually bit | Before calling a file non-conforming, check whether an independent reader *depends* on the form. The `_FillValue` encoding failed that check: it was xarray's, and the numeric alternative made Iris output unopenable (§4.4). The bidirectional cross-reader tests in §6 are the standing guard |
+| Consolidated metadata is not in the version 3 specification | Written by default because remote stores are unusable without it; controllable by keyword; readers that ignore it still work |
+
+---
+
+## 9. Out of scope
+
+- Writing Zarr version 2.
+- **Appending to, or modifying, data that a store already holds.** Iris does
+ not do this for netCDF either, and the obstacle is not the storage format:
+ deciding whether incoming data is compatible with what a file already holds
+ — dimensions, coordinate values, units, attributes — is a data-model
+ problem that Zarr leaves exactly where it found it. `mode="a"` adds *new*
+ arrays and groups to an existing store and does nothing else; the
+ create-once invariant in §4.5 is what holds the line.
+- Reworking the netCDF saver's `createVariable` keyword API (§4.5,
+ [#7328](https://github.com/SciTools/iris/issues/7328)), and checking
+ attribute content against netCDF's constraints before saving (§4.5,
+ [#7329](https://github.com/SciTools/iris/issues/7329)). Both are breaking or
+ behaviour changes to the netCDF saver, which PR 5 relocates unchanged.
+- Walking or loading multiple groups in one call.
+- GeoZarr and the proposed Zarr-CS coordinate-system convention.
+- Icechunk, virtual Zarr and Kerchunk reference stores. Icechunk is the
+ ecosystem's answer to transactional Zarr writes; §4.5 says so where the
+ limitation bites.
+- **Building** multi-process write coordination for Zarr — see below.
+- Any change to the existing NCZarr path other than the `spans` bug fix.
+- Adding an fsspec, s3fs or gcsfs dependency to Iris.
+
+Multi-process write coordination is deferred for size only. It is an Iris
+differentiator for netCDF, it is asked for by name, and zarr-python offers
+nothing to build on — version 3.4.0 warns `"synchronizer is not yet
+implemented"` **[verified]**, so it will be Iris's own machinery when it comes.
+§4.5 lists the four provisions that keep it a later addition rather than a
+redesign, and the write-alignment invariant it depends on is a correctness
+requirement of the first release regardless. Anything in review that would
+break those provisions should be treated as in scope, not out of it.
+
+---
+
+## 10. The parked question: the CF attribute model on JSON
+
+Everything else raised at review is settled and recorded in the decision log
+at §12.4. This one is not, and it is left open deliberately.
+
+It is one question with two faces, and the design currently answers both
+provisionally:
+
+- *Writing* (§4.5): JSON-native conversion, losing the NumPy dtype of
+ attributes, versus a base64 envelope carrying `dtype` and `raw` as offered in
+ #6961.
+- *Reading* (§4.4): what to do with an attribute value that is a legal JSON
+ object or array and has no CF equivalent — carry it opaquely, flatten it,
+ drop it with a warning, or define a convention.
+
+The provisional answer in both cases is **JSON-native**: convert on write,
+carry through untouched on read. It is what xarray does and what makes Iris
+output readable by the wider Zarr ecosystem. The design is deliberately
+arranged so that changing this answer later touches only the attribute
+conversion in `zarr/_dataset.py` — nothing in the loader, the saver or the
+`CFDataset` interface depends on it.
+
+**SciTools/iris#7288** tracks the discussion, with the relevant specification
+citations (Zarr v3 "arbitrary JSON literal"; CF §2.5.1 "of the same type as its
+variable") and the two real-world examples from §4.4.
+
+---
+
+## 11. References
+
+- SciTools/iris#6961 — Zarr I/O, parent issue
+- SciTools/iris#6977, #6979, #6980 — the three sub-issues
+- SciTools/iris#7259 — `cf.py` unit test coverage, by Martin Yeo
+- SciTools/iris#7288 — the parked CF attribute model question (§10)
+- Zarr specifications: https://zarr-specs.readthedocs.io/
+- Zarr v3 core specification, the source for the `dimension_names` rules in
+ §2.4 and the contract in §4.4:
+ https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html
+- zarr-python: https://zarr.readthedocs.io/
+- CF conventions for Zarr: https://github.com/zarr-conventions/CF
+- NCZarr: https://docs.unidata.ucar.edu/nug/current/ncZarr_head.html
+- NOAA GFS archive: https://data.dynamical.org/noaa/gfs/forecast/
+- xarray's Zarr backend, the reference implementation for the fill-value and
+ write-alignment conventions in §4.4 and §4.5:
+ https://github.com/pydata/xarray/blob/v2026.07.0/xarray/backends/zarr.py
+- pydata/xarray#10831 — shard-aligned writes, the corruption §4.5 guards
+ against: https://github.com/pydata/xarray/issues/10831
+
+---
+
+## 12. Progress
+
+The live record for the programme. Everything with a state belongs here;
+the rest of this document describes the design and should change only when
+the design changes.
+
+### 12.1 Pull requests
+
+Scope for each is in §5. Dependencies are strictly sequential within the
+`cf`/`netcdf` refactor chain (1 → 2 → 3 → 5); the Zarr features (4, 6) depend
+on the relocation before them, and PRs 7 and 8 depend on everything. PR 2.5 is
+the exception: it touches only `cf/_variables.py`, so it needs PR 2 and
+nothing else, and can land beside any of the others.
+
+| # | Title | State | Link |
+|---|---|---|---|
+| 1 | `iris.fileformats.cf` becomes a package, with tests first | Merged | [#7298](https://github.com/SciTools/iris/pull/7298) |
+| 2 | `CFDataset`, and the CF variable classes rewritten against it | Merged | [#7303](https://github.com/SciTools/iris/pull/7303) |
+| 2.5 | Deprecate `CFVariable.__getattr__` | Merged | [#7333](https://github.com/SciTools/iris/pull/7333) |
+| 3 | Relocate the CF loader | Merged | [#7316](https://github.com/SciTools/iris/pull/7316) |
+| 4 | Zarr loading | In review | [#7336](https://github.com/SciTools/iris/pull/7336) |
+| 5 | Relocate the CF saver | Merged | [#7338](https://github.com/SciTools/iris/pull/7338) |
+| 6 | Zarr saving | Not started | — |
+| 7 | Real-world test data and benchmarks | Not started | — |
+| 8 | The terms-of-art sweep | Not started | — |
+| — | **Merge back** `brownfield` → `main`: closing keywords, changelog fragments, docs | Not started | — |
+
+Companion work outside `SciTools/iris`:
+
+| Repository | Purpose | State | Link |
+|---|---|---|---|
+| `SciTools/iris-test-data` | Cut-down NOAA GFS (v3) and ESA EOPF (v2) fixtures for PR 7 | Not started | — |
+
+States are `Not started`, `Drafted`, `Built, awaiting PR N` (work complete on
+a branch based on another pull request not yet merged, with no number of its
+own worth recording), `In review`, `Changes requested`, `Merged` or
+`Abandoned`. Record the pull request number and its state as soon as it is
+opened, not when it merges.
+
+There is deliberately no `Closes` column and no changelog-fragment column:
+both obligations belong to the merge-back, not to the feature-branch pull
+requests. See §5.
+
+### 12.2 Related work
+
+Issue state is tracked on GitHub, not here. This table says what each item is
+*for* in this programme, and nothing about whether it is open. The three
+sub-issues are resolved by the merge-back pull request, which is where the
+closing keywords live (§5).
+
+| Item | What it is to this programme |
+|---|---|
+| [#6961](https://github.com/SciTools/iris/issues/6961) | The requirement. Parent issue for Zarr I/O |
+| [#6977](https://github.com/SciTools/iris/issues/6977) | Implemented by PR 2; closed at merge-back |
+| [#6979](https://github.com/SciTools/iris/issues/6979) | Implemented by PR 4; closed at merge-back |
+| [#6980](https://github.com/SciTools/iris/issues/6980) | Implemented by PR 6; closed at merge-back |
+| [#7288](https://github.com/SciTools/iris/issues/7288) | The parked attribute-model question this design raised (§10) |
+| [#7291](https://github.com/SciTools/iris/issues/7291) | The deferred-save return type this design turned up (§4.5, Q5). Not fixed by this programme |
+| [#7328](https://github.com/SciTools/iris/issues/7328) | The netCDF `createVariable` keyword API (§12.4). Out of scope (§9); the Zarr saver avoids the pattern rather than inheriting it (§4.5) |
+| [#7329](https://github.com/SciTools/iris/issues/7329) | Checking attribute content against netCDF's constraints before saving (§12.4). Out of scope (§9); the Zarr write path already has the policy (§4.5) |
+| [#7332](https://github.com/SciTools/iris/issues/7332) | The UGRID loader's unguarded `cf_role` and `start_index` reads, turned up by PR 2.5 (§5). Not fixed by this programme |
+| [#7337](https://github.com/SciTools/iris/issues/7337) | The netCDF saver's int64 failure on `NETCDF3_64BIT`, turned up by PR 5. Fixed by PR 5; closed at merge-back |
+| [#7259](https://github.com/SciTools/iris/pull/7259) | @trexfeathers' unit test coverage for `cf.py` (`cf_reader_zarr`, +2361/-643), carried into PR 1 with credit. Post a courtesy comment before opening PR 1 |
+
+### 12.3 Open questions
+
+| # | Question | Provisional answer | Tracked | Blocks |
+|---|---|---|---|---|
+| Q1 | How should the CF attribute model map onto JSON, on write and on read? | JSON-native both ways (§4.4, §4.5) | #7288, §10 | Nothing. Changing it touches only the attribute conversion in `zarr/_dataset.py` |
+| Q9 | Should a user be able to tell the loader that two same-named, same-length dimensions in different Zarr arrays are *not* the same dimension? The version 3 specification permits it, since a name guarantees nothing across arrays | No control. Treating that pair as one dimension is what xarray, NCZarr and netCDF all do, so a store meaning otherwise is relying on something no reader implements. Add one if a real store turns up that needs it | §4.4 | Nothing |
+| Q4 | The netCDF read path has the same sub-chunk re-read trap as Zarr, because `NetCDFDataProxy` reopens the `Dataset` on every `__getitem__` and so discards HDF5's chunk cache. Should it get the same alignment guard? | Out of scope here — the relocation pull requests must not change behaviour. Raise it separately once §4.4's Zarr guard has proven itself | §4.4 | Nothing |
+| Q5 | `iris.save(..., compute=False)` returns a tuple, not the documented `Delayed`, so `result.compute()` raises `AttributeError` **[verified]** | Caused by #6451 adapting to dask/dask#11844; the code is right and the docstrings were left behind. Not fixed here — PR 5 is behaviour-preserving. `zarr/saver.py` returns a real `Delayed` and tests it | [#7291](https://github.com/SciTools/iris/issues/7291) | Nothing |
+| Q6 | The `as_lazy_data` cache is process-wide and keyed on metadata, so it cannot detect a store whose chunk contents changed under identical metadata. Should it be scoped to a load session instead? | Metadata identity closes the cases that were reproduced (§4.4) and matches the netCDF key's existing strength, so it ships. Session scoping is the durable fix and would cover both formats | §4.4 | Nothing |
+| Q7 | Iris writes a floating `_FillValue` as base64 to stay readable by xarray, deviating from CF §2.5.1. Should the deviation be raised with the CF-Zarr conventions group rather than carried privately? | Carry it now, since the alternative is unreadable output; take it upstream to zarr-conventions/CF so the convention settles rather than each reader guessing | §4.4, §4.5 | Nothing |
+| Q8 | `_add_grid_mapping_to_dataset` sets sixty-three CF grid-mapping parameters by Python attribute assignment, bypassing the ASCII-to-bytes coercion every other saved attribute goes through — so `crs_wkt` is written as `NC_STRING` while `grid_mapping_name` is `NC_CHAR`. Should they be regularised? | Not in this programme. PR 2 kept them on a named netCDF4 handle (`grid_variable`) rather than change what lands in the file. Regularising is a one-line-per-parameter change with a real CDL diff, and belongs in its own pull request | §4.5 | PR 6's grid mappings. PR 5 relocated the method into `cf/saver.py`, where the netCDF4 handle is now the one thing in generic code that a non-netCDF store cannot supply |
+
+Close a question by moving it to §12.4 with the date and the answer. Do not
+delete it.
+
+### 12.4 Decision log
+
+Append-only. Each entry is the decision, not the discussion.
+
+**2026-09-21 — before design, agreed with @trexfeathers**
+
+- `zarr` is an optional, lazily imported dependency.
+- Extract the format-agnostic CF machinery out of `iris.fileformats.netcdf`,
+ with warn-on-use deprecation aliases at the old public names.
+- Write Zarr version 3 only, shaped so version 2 write is a later small change.
+- Read both version 2 and version 3.
+- Pass remote URLs straight to zarr-python; add no fsspec/s3fs dependency and
+ handle no credentials.
+- Load and save the root group by default, with a `group=` keyword.
+- Source real-world test data from NOAA GFS (dynamical.org) and the ESA EOPF
+ Sentinel samples.
+- Carry the test modules from #7259 into the programme, with credit.
+
+**2026-09-21 — at review of the first draft, agreed with @trexfeathers**
+
+- **Module layout:** `iris.fileformats.cf` becomes a package, not three flat
+ sibling modules. §4.1. The move became PR 1, on its own, so the programme
+ went from six pull requests to seven.
+- **Cross-reader testing:** xarray is added as a test-only dependency. Measured
+ at exactly one extra conda package in `iris-dev`, so no `skip_gdal`-style
+ guard is needed. §6.
+- **EOPF's role:** confirmed Zarr version 2 by reading `.zgroup`, so it keeps
+ its role as the version 2 and deep-group fixture. §7.
+- **Attribute model:** parked rather than decided; JSON-native is provisional.
+ Q1 above, issue #7288, §10.
+- **Method:** behaviour is justified from the Zarr and CF specifications, and a
+ real file is cited only as evidence that a code path gets exercised. Two
+ earlier claims credited to "real data" were re-examined under this rule:
+ ~~the base64 `_FillValue` turned out to be a publisher's CF violation~~
+ **— wrong, corrected 2026-09-23 below; it is xarray's version 3 encoding —**
+ and nested JSON attributes turned out to be already in the Zarr
+ specification. Now recorded in `lib/iris/AGENTS.md`.
+
+**2026-09-22 — agreed with @trexfeathers**
+
+- ~~**This programme is a proof of concept**, with official pull requests to
+ follow.~~ **Superseded the same day** — see the next entry.
+- **Issue state is not tracked in this document.** It lives on GitHub. §12.2.
+ *(Survives the retraction above; it is a rule about what a spec is for.)*
+- **Seven separate pull requests**, not a collapsed branch. Per-step CI and
+ independent verification are worth the extra cycles.
+
+**2026-09-22 — retraction, agreed with @trexfeathers**
+
+- **This is the delivery plan, not a proof of concept.** @trexfeathers: "I was
+ mistaken to introduce talk of proof-of-concept. This spec is THE spec we will
+ be using, and I need to see your strategy for real, even if everything is
+ being isolated to the feature branch." The programme is unchanged in shape;
+ what changes is that it is expected to reach users.
+- **Issues resolve at merge-back.** A closing keyword only fires on merge into
+ the default branch, so the feature-branch pull requests write `Part of
+ #NNNN` and the merge-back carries `Closes`. §5.
+- **Changelog fragments land with the merge-back**, for the same structural
+ reason and matching established practice on this branch: #7285 and #7287
+ carry no fragment, #7267 into `main` does **[verified]**. The merge-back owes
+ five fragments; §5 lists them. This supersedes the earlier "changelog no"
+ reasoning, which was justified by the proof-of-concept framing rather than by
+ how the repository works.
+- **Tests are required on every pull request**, unchanged. `tests/AGENTS.md`.
+
+**2026-09-22 — cross-cutting design review, with @trexfeathers**
+
+- **The `CFUGrid*` classes stay with the other variable classes.** @trexfeathers
+ asked why they warranted a private module of their own; they did not. They
+ are peers of the classic classifiers, share the private `_is_str_dtype`
+ helper, and appear in the same tuples in `CFReader` and `CFGroup`. The
+ earlier four-way split was a line count in search of a rationale. `cf`
+ splits three ways instead, and `_variables.py` is allowed over the
+ ~1000-line aim. §4.1.
+- **`CFVariable.__getattr__` works identically for both backends.** @trexfeathers
+ asked whether `lib/iris/AGENTS.md` overreached in banning `__getattr__`. It
+ did, in one place, and the spec was worse: it had the accessor raise
+ `TypeError` for Zarr-backed variables, which holed the abstraction §4.2
+ exists to provide. `__getattr__` now reads through `.attributes` on the base
+ class for netCDF and Zarr alike, drops the `setattr` instance caching, and
+ keeps a one-cycle deprecating fallback to the netCDF4 object. §4.3.
+- **The `getattr` rules in `lib/iris/AGENTS.md` are narrowed, not lifted.**
+ `getattr(var, "cf_role", "")` was never banned — the rule says *computed*
+ names and *string dispatch*, and `iris.fileformats` has 42 constant-name
+ lookups and **zero** dispatch sites **[verified]**. That stands, with a
+ clarifying clause. The blanket `__getattr__` ban is replaced by a checkable
+ exception: open-world data keys read from a file, forwarding to a declared
+ `Mapping`, with that `Mapping` as the path library code takes.
+- **Re-exporting from `cf/__init__.py` needs no justification.** @trexfeathers
+ noted that `lib/iris/AGENTS.md` made surfacing low-level objects at a higher
+ level read as naughty, and that the spec had written a defensive paragraph in
+ response. Neither was right: `iris.mesh` already re-exports from four
+ modules across two subtrees, and the "do not add indirection" rule is about
+ implementation layers. `AGENTS.md` now states the pattern positively, with
+ the conditions that keep it greppable — verbatim, static, `__all__`, no
+ renaming, no conditional imports. §4.1.
+- **Multi-process writing stays out of scope, but the design must not
+ preclude it.** @trexfeathers: it is "SPECIFICALLY DESIRED by multiple users, and
+ is a USP of Iris when it comes to NetCDF". Four provisions added in §4.5:
+ `_dask_locks.py` relocates to `cf/` in PR 5, `CFDatasetVariable.write_handle`
+ becomes the coordination seam, the lock stays inside the dataset
+ implementation, and `CFDataset` carries `mode` from the start.
+- **The chunk-alignment invariant is a correctness requirement now.** Writing
+ `da.store(..., lock=False)` is only safe where the lazy source tiles the Zarr
+ chunk grid exactly; two tasks sharing a chunk silently lose one update. The
+ saver derives `chunks` from the source's dask chunking, and rechunks the
+ source when the caller forces a conflicting `chunksizes=`. §4.5.
+- **`finalise()` is separate from `close()`.** Consolidating metadata is
+ one-shot and belongs to the returned `Delayed` under `compute=False`; N
+ workers consolidating one metadata object would race. §4.2, §4.5.
+- **Store durability was an oversight; now covered in §4.5.** @trexfeathers asked
+ whether metadata/data desynchronisation is impossible in the Iris write
+ chain. It is for a fresh store — metadata is written once at array creation
+ and never edited — but not for `save(..., group=...)`, which writes into a
+ store that may already exist. Both hazards confirmed against zarr 3.4.0
+ **[verified]**: a missing chunk reads back as `fill_value` with no warning,
+ and consolidated metadata goes silently stale so a default reader cannot see
+ arrays added without re-consolidation.
+- **The create-once invariant is written down as a constraint**, not left as an
+ accident of the current chain: Iris never resizes, rechunks or changes the
+ dtype, `fill_value`, codecs or dimension names of an existing array. It is
+ also most of the safety argument for future parallel writes. §4.5.
+- **`save` defaults to `mode="w-"` — raise if the target exists.** Chosen by
+ @trexfeathers over matching the netCDF saver's clobber, because a Zarr clobber is
+ many non-atomic deletes and an interrupt destroys the old store without
+ completing the new one. `mode="w"` opts back in; `mode="a"` is what `group=`
+ needs. Re-consolidation is mandatory when adding to a consolidated store.
+- **No torn-write detection, and write-to-temp-then-rename is rejected.**
+ Directory rename is atomic on POSIX but object stores have no equivalent, so
+ the guarantee would evaporate where Zarr is most used. Icechunk is the
+ ecosystem's answer and stays out of scope; the documentation says so. §4.5.
+- **There is no `ZarrDataProxy`.** @trexfeathers asked whether the netCDF proxy
+ classes are actually necessary for Zarr, given that chunks are inherent to
+ the store. They are not. Every reason `NetCDFDataProxy` exists is a
+ netCDF4/HDF5 reason — an unpicklable `Variable`, a thread-unsafe library, an
+ open file handle — and none holds for Zarr, where `zarr.Array` is picklable
+ **[verified]** and a chunk is an independent object needing no lock. Two of
+ the draft's three justifications for a proxy were false on inspection; the
+ third, the dask cache key, wants a hashable identity rather than a class.
+ §4.4.
+- **CF decoding is an explicit dask graph layer**, not hidden inside a proxy's
+ `__getitem__`: lazy, inspectable, fuseable, and testable without a store.
+ §4.4.
+- **`as_lazy_data` gains `cache_key=`**, and `iris/_lazy_data.py` stops
+ importing `NetCDFDataProxy` from `iris.fileformats.netcdf`. The netCDF loader
+ passes the key it produces today, so the change is behaviour-preserving.
+ PR 3. §4.4.
+- **Read-side chunk alignment, mirroring the write-side invariant.** A Zarr
+ chunk is the atomic unit of storage, so dask chunking must never subdivide
+ it. `_optimum_chunksize` ordinarily expands to whole multiples, but **shrinks
+ below the store chunk when the store chunk already exceeds the 128 MiB dask
+ target** — `(8000, 8000)` becomes `(2000, 8000)`, so four tasks each fetch
+ and decompress the same 488 MiB object **[verified]**. The Zarr loader rounds
+ up instead, and warns when the store's layout is the binding constraint.
+ §4.4.
+- **No `ZarrWriteProxy` either, but `write_handle()` stays.** @trexfeathers asked
+ whether the write proxy is purely multi-process machinery and so purely
+ future scope. It is not: `NetCDFWriteProxy`'s first job is to be a
+ `__setitem__` target that outlives the closed `Dataset`, which a deferred
+ save needs even single-threaded. The lock is its second job. Zarr needs
+ neither — nothing to close, `zarr.Array` picklable, distinct chunks
+ independent — and the delayed graph was verified to survive
+ `pickle.dumps`/`loads` before computing. `write_handle()` is therefore
+ justified by present netCDF need, not by future-proofing. §4.5.
+- **Native Zarr restores deferred saving, which NCZarr gave up.**
+ `Saver.__exit__` computes NCZarr writes eagerly because netCDF-c cannot
+ reopen a Zarr store for deferred writes. Going direct removes that
+ limitation, so `compute=False` works for Zarr; PR 6 tests it. §4.5.
+- **Found in passing, not fixed:** `iris.save(..., compute=False)` returns a
+ tuple rather than the documented `Delayed`, so the documented
+ `result.compute()` raises **[verified]**. Untested because every internal
+ caller uses `dask.compute`. Q5 in §12.3; a netCDF issue of its own.
+
+**2026-09-23 — review of #7292 by OpenAI Codex, posted by @bjlittle**
+
+Five findings, all reproduced against `zarr 3.4.0` / `dask 2026.7.1` /
+`xarray 2026.7.0` and all accepted. Three were outright design errors, not
+omissions. The pattern behind four of the five is the same: a *storage*
+property was read as a *semantic* one.
+
+- **Shards, not chunks, are the write-alignment unit.** The invariant licensing
+ `lock=False` was stated over the chunk grid. Dask chunks that tile the chunk
+ grid exactly still lost 72–96 of 128 values in 12 of 12 trials when eight
+ inner chunks shared a shard **[verified]**. Restated over the write grid;
+ aligning to the shard fixes it. xarray already does this and the earlier
+ draft cited `safe_chunks` while under-specifying it. §4.5.
+- **Zarr `fill_value` does not mean CF missing.** The precedence had the
+ storage field beat the CF attribute. zarr-python defaults `fill_value` to
+ zero, so `[0, 1, 2]` with no CF attributes had its valid zero masked, and
+ since the field is required on version 3 the CF fallback was dead code
+ **[verified]**. Now version-aware, matching xarray's
+ `use_zarr_fill_value_as_mask`. §2.4, §4.4.
+- **The base64 `_FillValue` is xarray's convention, not a malformation.**
+ `base64(struct.pack("<d", nan))` is exactly `'AAAAAAAA+H8='` **[verified]**,
+ and writing the numeric form the spec specified makes the store
+ **unreadable** by default xarray, failing the whole dataset open with
+ `TypeError` **[verified]**. Iris reads both forms and writes base64 for
+ floats: a single deliberate exception for one attribute whose dtype is
+ load-bearing, taken because interoperability is the point of native Zarr.
+ It does not reopen the base64-envelope rejection for general attributes, and
+ it is not a licence to conform to whatever a file contains. Q7 takes it
+ upstream. This reverses the 2026-09-21 entry above, which had credited the
+ encoding to a publisher's CF violation — an inversion of the very rule that
+ entry established. §4.4, §4.5.
+- **Inner chunks, not shards, are the read unit.** Sharding is *indexed*: a
+ reader fetches the index and only the byte ranges it needs. Requiring
+ shard-sized dask blocks turned a 2,084-byte partial read into a 6,148-byte
+ whole-shard read — but only with the decode layer present, because that layer
+ is what defeats dask's slice pushdown **[verified]**. The read and write
+ units are now separate and stated separately. §4.4.
+- **The Zarr cache key needs metadata, not an address.** `(store, path,
+ format)` went stale across an in-place `mode="w"`, returning four cached
+ values for a six-element array and an old `fill_value` over unwritten chunks
+ **[verified]** — and was weaker than the netCDF key it replaced, whose
+ `repr` already carries shape and dtype, so the relocation would not have been
+ behaviour-preserving. Keyed on `Array.metadata` instead; Q6 records session
+ scoping as the durable fix. §4.4.
+
+Every one of these is now a named test in §6. The cross-reader test that
+catches the third was already promised there before the review; it had simply
+not been written yet.
+
+**2026-09-24 — during PR 2**
+
+- `CFDatasetVariable.attributes` does not track reads; `CFVariable` does.
+ Two `CFVariable`s can share one backing variable, and they must not share
+ a read set.
+- `CFVariable.attributes` is a snapshot. A backend attribute mapping writes
+ through to the file, and a read-mode load must not write.
+- The `spans` gap §5 called a latent bug is unreachable from Iris:
+ `_NCZARR_SCALAR_DIMENSION` only ever appears alone, so the `len == 1`
+ guard it lacks can never fire. PR 2 characterises the behaviour instead of
+ changing it, and the "one behaviour change" §5 allows is spent elsewhere.
+- Saving a coordinate attribute whose name collides with a netCDF4 Python
+ member — `shape`, `size`, `dtype`, `name`, `dimensions`, `mask` — now
+ writes it, where the `hasattr()` "don't clobber" check used to drop it
+ silently. This is the saver-side half of the same defect the `__getattr__`
+ rewrite fixes on the load side.
+- `finalise()` is on the interface but unwired until PR 6.
+- The five declared properties — `dimensions`, `shape`, `ndim`, `dtype` and
+ `size` — stay typed properties on `CFVariable` and keep resolving to the
+ storage object. What changed is that the read is no longer *recorded*, so a
+ file attribute of one of those names is no longer consumed and now reaches
+ `cube.attributes`. A different mechanism from the bullet above, with the
+ same headline; the two must not be described as one.
+- A name `ncattrs()` lists but `getncattr()` cannot fetch yields `""` rather
+ than raising, so a malformed file that used to fail now loads. The netCDF
+ dataset layer already shipped that leniency, and two answers for the same
+ malformed file depending on which layer read it is worse than one lenient
+ answer.
+- CF attribute reads move from attribute access to a `Mapping` subscript, so
+ an absent name raises `KeyError` where it used to raise `AttributeError`.
+ Accepted as the cost of making `.attributes` the path library code takes.
+- `CFReader` writes its synthesised `bounds` link into the attributes mapping,
+ so a formula-term or derived-bounds variable that falls back to
+ `build_raw_cube` carries that key into `IRIS_RAW`. Accepted rather than
+ filtering inside `build_raw_cube`, which reads the mapping unfiltered by
+ design.
+- A borrowed *bare* `netCDF4.Dataset` — one that is not already an Iris
+ wrapper — is now wrapped in an `EncodedDataset`, so a borrow behaves like
+ every other input and character data decodes. The wrapping is
+ unconditional and does not consult `DECODE_TO_STRINGS_ON_READ`.
+- A dataset that *emulates* netCDF4 must now expose three more members:
+ `Dimension.size`, `Dataset.ncattrs()` and `Variable.ncattrs()`. No
+ `getattr` fallback was added — all three are public `netCDF4` API, and
+ `lib/iris/AGENTS.md` bans defensive wrapping for an unconfirmed problem.
+ Unverified against ncdata, which is not installed in `iris-dev`.
+- `cf_patch` keeps receiving netCDF4 objects. Both the dataset and the
+ variable handed to the hook are the netCDF4 ones, not the CF dataset
+ wrappers, because the hook's documented contract is netCDF4 attribute
+ assignment.
+- Two test modules join `_PERMITTED_SUFFIXES` in
+ `.hooks/check_netcdf4_imports.py` — `test_NetCDFDataset.py` and
+ `test_CFReader__dataset.py`. Both need a genuinely bare, unwrapped
+ `netCDF4.Dataset`, which is precisely the input whose handling changed.
+
+**2026-10-01 — during PR 3**
+
+- `load_cubes` stays in `iris.fileformats.netcdf.loader`. Opening a list of
+ file sources is the format's own business, and §4.8 lists
+ `netcdf.load_cubes` among the names that do not move. It imports the four
+ CF-layer functions it drives.
+- `DEBUG` stays too, and the moved `_load_cube_inner` reaches back for it
+ through the module. §4.8 lists it as a name that does not move, and it is a
+ flag users set by assignment: a module `__getattr__` can forward a read but
+ cannot intercept a write (PEP 562 defines no `__setattr__` hook), so
+ forwarding it would have made `netcdf.loader.DEBUG = True` silently stop
+ working. Only `CHUNK_CONTROL` and `ChunkControl` are forwarded.
+- The storage seam is one method, `CFDatasetVariable.read_data(chunking_policy)`,
+ taking a zero-argument callable that the implementation invokes only once
+ the result is known to be lazy. The alternative — hoisting the
+ `CHUNK_CONTROL.from_file()` check into the CF layer, which is where the
+ "is this the cube's data variable" fact lives — would have made small
+ unchunked cubes raise `KeyError` where they load today, because that check
+ sits behind the "small enough to read whole" shortcut.
+- `as_lazy_data` takes its cache key from the caller rather than testing
+ `isinstance(data, NetCDFDataProxy)`. This is the `cache_key=` change §4.4
+ called for, landed here because the CF loader cannot import a netCDF proxy
+ class.
+- `_LAZYVAR_MIN_BYTES` moves with the loader and is repointed at all eight of
+ its call sites rather than aliased. It is private, so no deprecation is
+ owed — and several of those sites *assign* to it, which an alias could not
+ have supported.
+**2026-10-12 — review of #7292 by @pp-mo, decisions with @trexfeathers**
+
+Eighteen comments from several days' reading by a netCDF and Iris-I/O
+specialist, on a spec that had had one afternoon's proofing. Fourteen changed
+the design or the document, two were already covered, one was declined and one
+is a correction to the review. Nothing here touches PR 3, which was in review
+at the time.
+
+The pattern across the substantive ones is a single theme: the design had
+taken *netCDF's* guarantees as if they were properties of CF, and Zarr
+provides weaker ones. Dimension names are the clearest case, but the same
+shape appears in the format-string branch in the saver, and in the assumption
+that a store detected by suffix must be local.
+
+- **The interface must be complete in its own terms, and nothing above it may
+ test the implementation type.** §4.2 said the abstraction held "only what the
+ CF variable classes, `cf/loader.py` and `cf/saver.py` actually need", which
+ reads as an intersection of two backends. It now states the rule and the one
+ place that breaks it: `saver.py:1479`/`:1500` compare `file_format` against
+ netCDF format strings, inside code PR 5 relocates to `cf/saver.py`. Replaced
+ by `CFDataset.supports_dtype`, which asks the capability question the code is
+ really asking. §4.2, PR 5.
+- **`CFDataset` gains `format` and `encoding`.** `format` so that the loader
+ and saver can *name* a format in a message without reconstructing it from
+ the class, which is the type test the rule above forbids. `encoding` so that
+ create-time configuration — zarr sharding and compressors, netCDF
+ `chunksizes` — is readable rather than write-only, instead of a format-aware
+ caller having to reach through to the `zarr.Array`. §4.2.
+- **`CFVariable.__getattr__` is deprecated, for removal one cycle later.**
+ @pp-mo argued for dropping it outright; @trexfeathers chose a cycle. What
+ settled *whether* is that `.attributes` already does the whole job.
+ `cf_var.units` and `cf_var.attributes["units"]` perform the same read with
+ the same side effect, so one is an alias for the other, and it is the alias
+ that loses silently to a class member of the same name. What settled *how*
+ is that `.attributes` does not exist on released Iris, so removing
+ `__getattr__` in the same release as its replacement leaves no version where
+ both work. Reverses the 2026-09-22 entry above, which kept it and narrowed
+ `lib/iris/AGENTS.md` to state an exception for it. That exception now
+ describes code scheduled for removal and should be revisited when the
+ removal lands. New PR 2.5. §4.3.
+- **A Zarr store with no dimension names loads, with anonymous dimensions.**
+ The spec had it raise `ValueError`. The version 3 core specification makes
+ `dimension_names` optional, permits null entries for unnamed dimensions,
+ only *recommends* that names are distinct within an array, and explicitly
+ declines to restrict reuse of a name across arrays — so all four are
+ conforming stores and none may be an error. @pp-mo asked whether the AI had
+ read the specification rather than guessed; it had not read it closely
+ enough, and the quoted text is now in §2.4. **Closes Q3**, which asked
+ whether the error needed an escape hatch and never questioned the error.
+- **The dimension contract is `(name, size)`, and an anonymous dimension
+ matches nothing.** One rule covering every case above, including the one the
+ spec previously called an error: a name used at two lengths links nothing,
+ and `CFDataset.dimensions` omits it with a warning. `CFVariable.spans` and
+ its three overrides compare names today and must compare pairs. netCDF is
+ unaffected, because its names are unique within a group. §4.4, PR 4. The
+ question of a user control to split same-named same-length dimensions is
+ Q9.
+- **Attributes that CF cannot express are structured ones, not non-strings.**
+ The §4.4 heading said "attributes that are not scalars or strings", which
+ names the wrong set: a flat JSON array of numbers is an ordinary CF vector
+ attribute. §4.4.
+- **An attribute the format cannot hold is an error, not a warning.** @pp-mo
+ suggested generalising `netcdf/_attribute_handlers.py` into "stringify
+ anything unknown and warn". Declined by @trexfeathers, quoted in §4.5: the
+ existing handlers convert where the string form is defined and reversible,
+ `str(some_object)` is neither, and Iris is careful about cube-to-cube round
+ tripping. The other half of the comment — explicit checks when saving content
+ netCDF cannot hold — is accepted, raising rather than warning, and is a
+ netCDF behaviour change outside this programme.
+ [#7329](https://github.com/SciTools/iris/issues/7329).
+- **The Zarr saver takes Zarr's keyword vocabulary, not netCDF's.** @pp-mo
+ objects to Iris naming every netCDF `createVariable` keyword in its own save
+ signature and tunnelling each through several call layers, and holds up
+ `CHUNK_CONTROL` as the better pattern. Reworking the netCDF side is a
+ breaking change with a wide blast radius and gets
+ [#7328](https://github.com/SciTools/iris/issues/7328). But the
+ Zarr saver is new API with no compatibility debt, and the netCDF→codec
+ translation layer the spec described was the mechanism by which it would
+ have inherited the pattern — so the translation goes and the Zarr names
+ stand alone. §4.5.
+- **`mode="a"` adds new arrays and groups; it is not append.** @pp-mo read it
+ as promising append-to-existing-data and asked for that to be ruled out.
+ It was never offered — the create-once invariant forbids it — but the
+ writing invited the reading. §4.5 now says so before the rules rather than
+ after, `mode="a"` onto an existing name is an error, and §9 lists appending
+ as out of scope with the reason: the obstacle is compatibility between
+ existing and incoming data, which is a data-model problem Zarr does not
+ touch.
+- **The format-detection rules were local-versus-remote, written as three
+ alternatives.** The bare "path ends in `.zarr`" test subsumed the
+ scheme-qualified one, making it unreachable, and was the rule that should
+ never have applied locally — reading the marker file costs one `stat` there
+ and a network round trip remotely. §4.7.
+- **`load_files` is not taught about directories.** @pp-mo asked for a separate
+ routine rather than a buffer-is-`None` special case, and that is adopted:
+ `iris.io.load_stores` takes every target that is a store, local directory or
+ remote URL, and `load_files` keeps the invariant that every path reaching it
+ is a readable file. The cost is that the file-or-store partition has to
+ happen between glob expansion and format dispatch, so that a glob matching
+ both still works. §4.7.
+- **The dask cache key stays `Array.metadata.to_dict()`, and the assumption
+ becomes a test.** @pp-mo is right that nothing in the Zarr specification
+ defines that method's output. The suggested remedy — enumerate our own
+ components — trades a harmless failure for a harmful one, because a
+ hand-written list goes stale when zarr-python adds a field that matters and
+ nothing fails when it does. The key stays complete-by-construction and §6
+ now pins its stability and its sensitivity to each component. §4.4.
+- **Two factual corrections to §2.4 and §4.4.** There is no per-file read lock
+ in Iris: `_GLOBAL_NETCDF4_LOCK` is one process-wide lock guarding a
+ non-re-entrant netCDF4 package, and the per-file locking in `_dask_locks.py`
+ is a write-path mechanism. And zarr's dirty-attributes-mapping behaviour,
+ which @pp-mo called plain nasty, is never reached by Iris, because §4.5
+ converts values before handing them to zarr; it is a zarr-python defect worth
+ reporting upstream, not a constraint.
+- **`NaN` in JSON, confirmed and recorded.** @pp-mo confirmed that strict JSON
+ has no NaN and that Python implements it as an extension, round-tripping by
+ default via `allow_nan`. Added to §2.4, because it says precisely which
+ reader would break: a strict one, or one that is not Python. The cross-reader
+ test is what covers it.
+
+Two comments needed no change:
+
+- **`finalise()` really is a no-op for netCDF.** @pp-mo read it as a likely
+ typo for "for *Zarr*". It is not: `finalise()` writes consolidated metadata,
+ which is a Zarr concept with no netCDF equivalent, and §4.2 says the same.
+- **Native Zarr already takes precedence over NCZarr, with no rule needed.**
+ NCZarr is reachable only through an explicit `#mode=nczarr` URL fragment, on
+ load and on save. A plain `.zarr` path reaches neither at the baseline: load
+ raises `IsADirectoryError` and save raises
+ `ValueError("Cannot save; no saver")`, because `_savers` has no `.zarr`
+ entry **[verified]**. The native path takes only what was an error, so
+ nothing changes meaning. §4.7 says this rather than leaving it to be
+ inferred.
+
+Also corrected in passing, and not from the review: §5's merge-back table still
+owed a `bugfix` fragment for the `_NCZARR_SCALAR_DIMENSION` `spans` gap, which
+the 2026-09-24 entry above records as unreachable and characterised rather than
+fixed. The fragment is now the attribute-name collision that PR 2 did change.
+
+**2026-10-13 — during PR 4**
+
+- `CFDataset.format` lands in PR 4, not PR 5. `ZarrDataset` cannot answer
+ `ZARR_V2` versus `ZARR_V3` without it, and PR 4 merges first.
+ `supports_dtype` is a saver question and stays in PR 5. §5's PR 4 and PR 5
+ entries are amended to match.
+- The shared `CFDataset` contract is split into a read half and a write
+ half. PR 4 implements the Zarr read path only, so the write half is
+ unrunnable against it; one class with half of it skipped would hide an
+ incomplete implementation behind a skip, which is the opposite of what §6
+ asks a shared body to do. PR 6 adds the write half for Zarr.
+- **Closes Q2.** `<U7`-style fixed-length unicode arrays read as themselves
+ on both Zarr versions, verified against zarr-python 3.4.1. #6961's note
+ that they are "not strictly Zarr-supported" describes the specification
+ rather than zarr-python's behaviour, so no special handling is needed.
+- `iris.io.load_files` is a public wrapper over a private `_load_files`. The
+ load chain expands globs once, before partitioning files from stores;
+ expanding again inside `load_files` would collapse duplicate glob matches,
+ because the expansion is keyed by resolved path. The split keeps
+ `load_files`'s documented glob-accepting contract intact for external
+ callers, while the partition path in `iris.loading._generate_cubes` calls
+ the private form directly. §4.7.
+- `ZarrDataset.attributes` withholds `_NCProperties` alongside
+ `_ARRAY_DIMENSIONS`. §4.4.
+**2026-10-14 — review of #7316 by @stephenworsley, decision by @trexfeathers**
+
+- **The `getattr` anti-patterns leave `lib/iris/AGENTS.md`.** The rule never
+ caught anything. Twice it has been argued against and twice the code was
+ kept: `CFVariable.__getattr__` on 2026-09-22, which ended in the rule being
+ narrowed to admit it, and the PR 3 deprecation shim here, a module-level
+ `__getattr__` of the kind PEP 562 added for exactly that job, which opened
+ the pull request on a request for a ruling. Neither occasion changed a line
+ of code. @stephenworsley found the rule ambiguous on its own terms;
+ @trexfeathers judged that a rule whose whole effect is to ask for the
+ argument costs more than it earns, and removed all three bullets —
+ dynamically generated members, `eval`/`exec`/`getattr` string dispatch, and
+ the `__getattr__`/metaclass ban with its narrow exception. `eval` and `exec`
+ came straight back as their own bullet: they shared a sentence with `getattr`
+ dispatch but none of its ambiguity, and nothing in the programme has ever
+ had to argue about them. The greppability rule against constructing
+ identifiers at runtime is untouched and still carries the reasoning the
+ anti-patterns were standing in for. This reverses
+ the narrowing recorded on 2026-09-22 and the "revisit when the removal
+ lands" note of 2026-10-12; §3 and §4.3 no longer cite an exception that
+ exists. The design is unchanged: §4.3 removes `__getattr__` because
+ `.attributes` already does the whole job, which was never an argument from
+ the rule.
+
+**2026-10-14 — actioning the Codex review of PR 4**
+
+All four findings reproduced and fixed; none disputed.
+
+- The `(name, size)` identity rule carries through to coordinate attachment,
+ not only to `spans`. It becomes a function rather than a method, used by
+ both. §4.4.
+- `missing_value` may be a vector, per CF section 2.5.1; `_FillValue` may
+ not. §4.4.
+- Chunks are sized on the decoded dtype, which is what a task holds, on
+ the `CHUNK_CONTROL.as_dask()` path as well as the default one. §4.4.
+- The `ZarrStore` file element answers `None` when a file handle is supplied,
+ so a file's basename is never resolved against the working directory. §4.7.
+
+**2026-10-14 — PR 5, the CF saver relocation**
+
+- The generic saver is `_CFSaver`, constructed from an open `CFDataset` rather
+ than from a filename. Each format's `Saver` opens its own store and passes it
+ down.
+- `_CFSaver` is private, unlike `CFDataset`. The two look like peers and are
+ not. Every member of `CFDataset` is something a format must implement, so
+ its public surface and its contract are the same set of names, and a user
+ can hold one of its variables through `CFVariable.cf_data`. What a format
+ implements on the saver is four hooks, all private, and the Zarr saver
+ supplies one of them and inherits three; everything public on the class is
+ the caller's API, which each format's own public `Saver` inherits unchanged.
+ Making the class public would promise the shape of thirty-six methods on one
+ subclass's evidence, and only a deprecation cycle could take it back.
+ Private costs nothing and PR 6 can promote it. The `cf/saver.py` module
+ stays public either way - it holds the four relocated names of §4.8, and its
+ docstring is where a format author is told what to subclass. This matches
+ `cf/loader.py`, whose generic body is likewise private functions in a public
+ module.
+- Four hooks are what a format supplies beyond the dataset:
+ `_create_string_variable`, `_backend_write`, `_flush_before_close` and
+ `_patch_dataset`. Two of them raise on the base class rather than guessing a
+ generic answer, because storing a string and patching a netCDF4 object have no
+ generic answer to guess.
+- `write()` is the one method whose signature differs between the two classes.
+ The netCDF one keeps its twelve documented keywords, which `save()` passes by
+ position; the generic one takes `**encoding` plus a `compression_kwargs` dict
+ for what applies to every variable rather than only the data variable.
+- The saver's unit tests stay under `tests/unit/fileformats/netcdf/`. They drive
+ the generic machinery through the only subclass that exists; a shared body is
+ worth factoring out when PR 6 adds the second one.
+- `supports_dtype` answers the capability question, and the saver keeps the
+ question of which dtypes are worth narrowing to int32. Splitting it that way
+ lets a backend answer honestly about dtypes the narrowing never sees.
+- The grid-mapping handle stays netCDF's, and the work it needs is larger than a
+ fifth hook. `_add_grid_mapping_to_dataset` takes `cf_var_grid.variable` and
+ writes the CF grid-mapping parameters straight onto the netCDF4 variable,
+ bypassing the ASCII coercion `attributes` applies (Q8). It does not go through
+ `CFDatasetVariable.deprecated_netcdf_member()`, so a store with no netCDF4
+ object behind it raises a bare `AttributeError` rather than a diagnosable one.
+ The job is to make grid-mapping attribute writing store-agnostic, and it
+ blocks PR 6's grid mappings.
+- `cf/saver.py` keeps one import from the netCDF package:
+ `ATTRIBUTE_HANDLERS`, which `_create_cf_data_variable` uses. It is the
+ module's only runtime reach into `iris.fileformats.netcdf`, and it closes a
+ two-way import cycle, because `netcdf/saver.py` imports `cf.saver` and
+ `cf.saver` imports back. The cycle survives because `_attribute_handlers`
+ imports nothing from `iris.fileformats.cf`. Traced from five entry points,
+ `cf.saver`'s body runs while the netCDF package is itself part-way through
+ importing, so a netCDF submodule that reached back into `cf.saver` would find
+ it half-built. The eager `from . import name, netcdf, nimrod, pp, um` in
+ `lib/iris/fileformats/__init__.py` fixes which order that happens in, and is
+ not what makes it work: patched out in memory, with the module-level
+ `netcdf.load_cubes` references made lazy, all nine modules still import first
+ cleanly. Relocating
+ `_attribute_handlers` is PR 6's, to keep this relocation's overlap with the
+ parallel Zarr loading work small.
+- `cf/saver.py` ships at the size it already was, about 2500 lines against the
+ roughly 1000 that `lib/iris/AGENTS.md` asks of a new module. A relocation
+ inherits its subject's size. Splitting it is worth doing in its own pull
+ request, once PR 6 has shown where the seams between the two formats really
+ fall.
+
+**2026-10-16 — Q10 closed, at review of #7338 by @stephenworsley**
+
+- #7337 is fixed in PR 5, not carried across it. Saving int64 data to
+ `NETCDF3_64BIT` narrows to int32, as every other restricted format already
+ did, and a value too large to narrow raises `ValueError`. The fix is
+ `"NETCDF3_64BIT_OFFSET"` added to `_RESTRICTED_FORMATS` in
+ `netcdf/_dataset.py`, which netCDF4 reports for a file created under the
+ other spelling.
+- The relocated module is untouched by the fix. `cf/saver.py` asks
+ `supports_dtype` and is a pure move either way, so the one promise the
+ relocation's review rests on survives.
+- Fixing it on `main` instead would have conflicted at the merge-back. `main`
+ has no `netcdf/_dataset.py`; the same defect lives in an inline tuple in
+ `netcdf/saver.py` that this branch has already deleted, so the merge would
+ have had to be resolved by editing a third file.
+- The `ValueError` names `NETCDF3_64BIT_OFFSET`, which is not the string the
+ caller passed. Left as it is, and pinned by a test: the message names the
+ format of the file that was opened, which is the honest answer and the same
+ rule every other format follows.
+
+### 12.5 Artefacts
+
+| Artefact | Location | State |
+|---|---|---|
+| Cut-down NOAA GFS fixture, 1.4 MB, 12 members, Zarr v3 | `~/projects/iris-zarr-testdata/gfs_forecast_sample.zarr` | Cut, not yet contributed to `iris-test-data` |
+| Cut-down ESA EOPF fixture, Zarr v2 | — | Not started |
+
+The GFS subset was cut ahead of the rest of the programme because its source
+endpoint is documented as closing on **30 September 2026** (§8).
+
+### 12.6 Document history
+
+| Date | Change |
+|---|---|
+| 2026-09-21 | First draft: design, six-pull-request programme, four open decisions. |
+| 2026-09-21 | Grounded against the real NOAA GFS store. |
+| 2026-09-21 | Package layout adopted; programme grew to seven pull requests; the two "real data" findings re-framed as one specification fact and one publisher malformation; EOPF confirmed version 2; three decisions settled and the attribute-model question parked to #7288. |
+| 2026-09-21 | Added this progress record (§12). |
+| 2026-09-22 | Reframed as a proof of concept; removed issue-closure claims and issue-state tracking. |
+| 2026-09-22 | Confirmed seven separate pull requests; tests required, changelog fragments deferred. |
+| 2026-09-22 | Proof-of-concept framing retracted. Added the merge-back step (§5) that carries the closing keywords, the five changelog fragments and the documentation. |
+| 2026-09-22 | Dropped the proposed `cf/_ugrid.py`; the `cf` package splits three ways, not four. |
+| 2026-09-22 | Rewrote §4.3: `__getattr__` separated from the backend-proxy job, made backend-agnostic, and the attribute-tracking consequence for PR 2 called out. `lib/iris/AGENTS.md` narrowed to match. |
+| 2026-09-22 | Dropped the apologetic framing of the `cf/__init__.py` re-exports; `lib/iris/AGENTS.md` now endorses the pattern. |
+| 2026-09-22 | Multi-process writing kept reachable: chunk-alignment invariant, `write_handle` seam, `_dask_locks` relocation, `mode` and `finalise` on the ABC. |
+| 2026-09-22 | Added store durability (§4.5): the create-once invariant, `mode="w-"` by default, mandatory re-consolidation, and the two format hazards verified against zarr 3.4.0. |
+| 2026-09-22 | Dropped `ZarrDataProxy` (§4.4): decoding becomes an explicit dask graph layer, `as_lazy_data` gains `cache_key=`, and the read-side chunk-alignment guard was added after `_optimum_chunksize` was shown to subdivide large store chunks. |
+| 2026-09-22 | Covered the write proxy (§4.5): no Zarr equivalent needed, `write_handle()` justified by present netCDF need, native Zarr regains the deferred saving NCZarr gave up, and the `da.store` return-type defect recorded as Q5. |
+| 2026-09-23 | Accepted all five findings of the #7292 review, all reproduced. Write alignment restated over shards; fill-value masking made version-aware and taken off the storage field; the base64 `_FillValue` corrected from "malformation" to xarray's convention, and now written as well as read; the read unit separated from the write unit; the Zarr cache keyed on `Array.metadata`. Tests named in §6; Q6 and Q7 opened. |
+| 2026-09-23 | Structural pass for readability. §4.4 and §4.5 given `####` subheadings throughout — they were 617 lines navigated only by run-in bold lead-ins, and `Encoding` had been nested under multi-process writes by accident. Design history recast from "an earlier draft said X" into the rule it implies ("do not do X, because Y"): same guidance against re-deriving the rejected answer, without depending on knowledge of a draft the reader never saw. No normative content changed. |
+| 2026-09-24 | PR 2 built. §4.2 reconciled with the implemented interface: `location`, `__len__` and `ndim` on the variable, `closed`, `__enter__` and `__exit__` on the dataset, `attributes` no longer tracking, `create_dimension(size=None)` and `create_variable(dimensions=())`. §4.3 says what `CFVariable.attributes` is. Q8 opened on the grid-mapping assignments; thirteen decisions logged. |
+| 2026-10-01 | PR 3 built. The CF loader relocated to `iris.fileformats.cf.loader`; `load_cubes`, `NetCDFDataProxy` and `DEBUG` stayed. `CFDatasetVariable.read_data` added as the storage seam; `as_lazy_data` gained `cache_key=`. Five decisions logged. |
+| 2026-10-02 | PR 3 raised as #7316. §12.1 records it, and #7298 corrected to Merged. |
+| 2026-10-02 | #7303 merged; #7316 rebased onto `brownfield` so it carries only its own ten commits. |
+| 2026-10-12 | Actioned the #7292 review by @pp-mo. Dimension names made optional and the `(name, size)` contract written down; `format`, `encoding` and `supports_dtype` added to the interface with the rule that nothing above it tests the implementation type; `CFVariable.__getattr__` deprecated, adding PR 2.5; the Zarr saver's netCDF keyword translation dropped; `mode="a"` separated from append, and append declared out of scope; format detection split local from remote; `load_files` left alone in favour of `load_stores`; the cache key's assumption turned into a test. Q3 closed, Q9 opened, two issues raised. |
+| 2026-10-12 | PR 2.5 built, correcting §5's account of it. Four reads inside Iris still reached `__getattr__` after PR 2, not none; they are named, and so is the measurement that found them. The silent-load test records warnings rather than promoting them to errors, because `iris.load` captures a build exception into `LOAD_PROBLEMS` and an `error` filter degrades the load instead of failing the test. `IrisDeprecation`'s module corrected to `iris._deprecation`. |
+| 2026-10-13 | PR 4 built, on a branch based on PR 3. `format` moved to PR 4 (§5); the shared `CFDataset` contract split into read and write halves, so PR 4 runs only the read half; `load_files` split into a public glob-expanding wrapper over a private `_load_files` (§4.7); `_NCProperties` withheld from `ZarrDataset.attributes` alongside `_ARRAY_DIMENSIONS` (§4.4). Q2 closed. Four decisions logged. |
+| 2026-10-14 | The three `getattr` anti-patterns removed from `lib/iris/AGENTS.md`, and the passages in §3 and §4.3 that leaned on them rewritten to stand on their own. No design change. |
+| 2026-10-14 | Actioned the Codex review of PR 4. Four findings, all reproduced and fixed: the `(name, size)` rule extended through coordinate attachment (§4.4); vector `missing_value` decoded (§4.4); chunks sized on the decoded dtype, including under `CHUNK_CONTROL.as_dask()` (§4.4); the `ZarrStore` element ruled out when a file handle is supplied (§4.7). |
+| 2026-10-14 | PR 5 raised. §5 records the shape the saver relocation takes - `CFSaver` and the four hooks - and §12.3 records the `NETCDF3_64BIT` defect found while doing it. §12.4 records what the relocation leaves netCDF-specific inside `cf/saver.py`, and §12.1 corrected to Merged for #7316 and #7333. |
+| 2026-10-16 | The generic saver renamed `_CFSaver` and made private, following review of #7338 by @stephenworsley. §5 and §12.4 say why it is private where `CFDataset` is public, and what would have to be true for PR 6 to promote it. |
+| 2026-10-16 | #7337 fixed inside PR 5 rather than carried across it, following review of #7338 by @stephenworsley. Q10 closed into §12.4, and §12.1 now records the issue as closed by PR 5. |
+| 2026-10-16 | Added PR 8, the terms-of-art sweep: this document and the plans predate the rule in `.github/WRITING.md`, and a coinage that reaches `main` passes the test a later agent applies to decide whether a term is already in use. The programme goes from eight pull requests to nine. No design change. |
+| 2026-10-20 | Read-side chunk alignment removed (§4.4), reversing the 2026-09-22 entry that added it. The mechanism held up under measurement, but alignment only ever changed the answer where a stored chunk exceeds the dask target, and that is a trade the netCDF path has always accepted; correcting one backend would make two formats differ over shared physics. The warning it carried is kept and moved to `iris.fileformats.cf.loader`, where it covers both backends, and `CHUNK_CONTROL` is the answer for a caller who wants the other trade. Decided with @trexfeathers on #7336, following review by @pp-mo. |
+| 2026-10-20 | #7338 merged, and PR 4 merged `brownfield` back in. The three predicted collisions resolved as PR 5's plan said they would: one copy of `format` kept on each class, PR 4's `encoding` kept beside it, and `supports_dtype` implemented on `ZarrDataset` (§4.2). |
+| 2026-10-22 | UGRID meshes load from Zarr, closing a gap @pp-mo found reviewing #7336. §4.7 already said `load_meshes` would accept a store; it reached one and then rejected it, because the handler test named only `nc_loader.load_cubes`. Worse, `iris.load` on a UGRID store returned a cube that had silently dropped its mesh coordinates: `CFReader` classifies them as UGRID coords, and nothing then built the mesh. The fix is wiring, not machinery - `_meshes_from_cf` already built a correct `MeshXY` from a `ZarrDataset`. The inline mesh block in `netcdf/loader.py` became `_prepare_mesh_coords` and `_attach_mesh_coords`, shared by both loaders (§5). |
+| 2026-10-22 | Unifying the two cube-building loops deferred to an issue raised at merge-back rather than acted on now (§5), per @trexfeathers: the refactor is internal and safe to do late, and raising the issue while the programme's pull requests are interdependent and in flight would invite someone to start on it. |
diff --git a/lib/iris/AGENTS.md b/lib/iris/AGENTS.md
new file mode 100644
index 0000000000..422d487ff7
--- /dev/null
+++ b/lib/iris/AGENTS.md
@@ -0,0 +1,302 @@
+# AGENTS.md
+
+Agent instructions for `lib/iris/` — the Iris library source.
+
+These rules govern **how code is written** under this tree. Tests have their
+own rules in [`tests/AGENTS.md`](tests/AGENTS.md).
+
+For **where code lives and how data flows**, read
+[`ARCHITECTURE.md`](ARCHITECTURE.md) before exploring unfamiliar subsystems.
+It is an orientation map only; algorithm detail belongs in module docstrings.
+
+For **how to build the prose itself** — every comment, docstring and commit
+message here, as well as anything posted to GitHub — read
+[`WRITING.md`](../../.github/WRITING.md).
+
+
+## Why This File Exists
+
+Iris is maintained by humans and agents together. Agents read through a
+narrow context window using text search and cannot run the code; humans read
+with intuition but limited patience. Both want explicit names, local
+reasoning and small units. This file records that shared style, plus the few
+places the two diverge and which way to resolve them. Models are trained to
+please diff reviewers rather than to produce code that is cheap to navigate
+later, and drift towards long functions, redundant comments and defensive
+wrapping; correct for that deliberately.
+
+
+## Fast Rules
+
+1. Write code that can be understood from the file in front of you.
+2. Name things so they can be found by exact text search.
+3. Prefer explicit and slightly verbose over implicit and clever. This
+ governs code, never prose: comments and docstrings are terse.
+4. Prefer duplication over an abstraction you are not confident in.
+5. Match surrounding Iris idioms even where you would choose differently.
+6. Every comment and docstring must still be true after your edit.
+7. Explain subsystems in module docstrings, not inline or in side files.
+
+
+## Locality — Readable In One Place
+
+- Keep behaviour derivable from the function's own body, signature, type
+ hints and docstring. Avoid designs that require opening three other files.
+- Prefer composition and explicit delegation over deep inheritance and mixin
+ stacks. Resolving an MRO across modules costs an agent many tool calls and
+ is opaque to a newcomer. Where Iris already uses mixins (e.g.
+ `iris.common.mixin`, metadata managers), follow the existing pattern rather
+ than inventing a parallel one.
+- Do not add indirection (registry, dispatch dict, plugin hook, base class)
+ for a single caller. This governs implementation layers, not API surface;
+ a package `__init__.py` that re-exports is not indirection, see below.
+- Pass dependencies in as arguments rather than reaching for module-level
+ mutable state.
+
+
+## Greppability
+
+Exact text search is the primary retrieval tool. Code that cannot be found
+does not exist.
+
+- Never construct identifiers at runtime: no `setattr(obj, f"{name}_bounds",
+ ...)`, no `getattr(module, method_name)` dispatch where a dict or `if/elif`
+ would do. A constructed name is unfindable for every reader. This bans
+ *computed* names only — `getattr(var, "cf_role", "")` is a safe lookup with
+ a greppable literal and is the preferred idiom for optional CF attributes.
+- Give public names enough distinctiveness to search for. Local `data`,
+ `cube`, `result` are fine; a public helper called `_process` is not.
+- Declare `__all__` in modules with a public surface. No `import *`.
+- **Re-export freely from a package `__init__.py`.** Surfacing objects at
+ the level users import from is a real convenience, and it keeps the file
+ layout free to change later. `iris.mesh` is the model: explicit
+ `from .components import MeshXY`, gathered into `__all__`. Re-export
+ verbatim — never rename on the way out, never wrap in `try`/conditional
+ imports, never put logic in `__init__.py`. Each object stays defined in
+ one module, so `class MeshXY` still finds it in a single grep.
+ (`iris.common` uses `import *`; that is legacy, not a pattern to copy.)
+- Make error and warning messages distinctive and mostly static, so a
+ traceback maps to exactly one line. Put interpolated values at the end:
+ `f"Cannot collapse a coordinate with bounds: {coord.name()}"`.
+- Do not reuse one helper name across modules for differing behaviour.
+
+
+## Explicitness
+
+- Add type hints to new or modified public functions. Agents cannot execute
+ code to discover types, so hints are the cheapest reliable signal. Do not
+ retrofit hints to code you are not otherwise changing.
+- Avoid `**kwargs` passthrough on public API — spell out the parameters. If
+ passthrough is unavoidable, document the accepted keys.
+- Use keyword arguments at call sites for anything not obviously positional.
+ Booleans are always keyword.
+- Replace repeated magic strings with module-level constants.
+- Return one documented type. Do not write functions whose return type
+ depends on an argument's value. Iris has some legacy examples; add no more.
+
+
+## Laziness
+
+Realising data is the caller's decision, never a library function's.
+
+- Use the `core_*` accessors when you do not care which you have:
+ `core_data()` on a cube, `core_points()` / `core_bounds()` on a coord. They
+ return real or lazy without realising either. Reach for `.data` only to
+ return a result the caller asked to realise.
+- A lazy input gives a lazy output. State in the docstring whether the result
+ is lazy and whether the call realises data.
+- Never realise to inspect: `shape`, `dtype` and `ndim` are on the lazy array,
+ and `has_lazy_data()` / `has_lazy_points()` answer which you have.
+- Build graphs whole-array with `iris._lazy_data` — `map_complete_blocks`,
+ `lazy_elementwise`, `as_lazy_data`. Per-slice graph building is expensive.
+
+
+## Conform to the Specification, Not to the Sample File
+
+Iris implements published conventions — CF, UGRID, Zarr, netCDF. Sample files
+are evidence that a code path gets exercised, not authority for what it does.
+
+- Derive behaviour from the convention's text and cite the section. Before
+ claiming a file taught you something general, check the convention first.
+- When a real file disagrees with the convention, the file is wrong: warn
+ (`iris.warnings`) naming the variable and the offending value, and carry on.
+- Do not reshape the reader around one publisher's output. Raise only where
+ the data cannot be interpreted at all.
+
+
+## Comments and Docstrings
+
+A stale comment is worse than no comment — an agent treats it as evidence and
+a human treats it as documentation.
+
+- Comments explain **why**: the CF rule, the constraint, the bug worked
+ around. Never restate what the code plainly does.
+- Fix or delete any comment your change invalidates, even if you did not
+ write it.
+- A comment claiming a branch is unreachable is a testable claim. Pin it
+ with a test or an `assert`, or restructure so it is true by construction.
+ Left as prose it goes stale unnoticed.
+- Do not add docstrings or comments to code you did not otherwise change —
+ but see the module docstring exemption below.
+- NumPy-style docstrings are mandatory and validated. State the contract:
+ units, shapes, laziness, mutation, exceptions raised.
+- Link non-obvious workarounds to their source:
+ `# See https://github.com/SciTools/iris/issues/1234`.
+- Cite only what a later reader can still resolve: an issue, a pull request,
+ a design document, a section of the convention. Everything else decays
+ unchecked. [`WRITING.md`](../../.github/WRITING.md) lists what decays and
+ what to write instead.
+
+
+### Brevity
+
+- Minimise every section of every docstring, by subtraction. Sentence by
+ sentence, ask what breaks if it is gone; most rationale survives its own
+ deletion. Then ask who each surviving sentence is for — a **user** calling
+ the thing, a **maintainer** not breaking it, or a **reviewer** you are
+ answering. Only the first belongs here; the second goes to the module
+ docstring or a test, the third to the pull request.
+- Write notes, not an essay. Fragments are correct, articles are optional,
+ shorthand is welcome. Say what holds; do not restate the signature, defend
+ the design against an alternative nobody proposed, or narrate how you found
+ out. Contract first, caveat second, rationale last, so that anything
+ overlong is at least skippable.
+- A comment runs one to three lines. A fourth is the alarm bell: the fact has
+ outgrown the site and wants a better home.
+- One fact, one place — for prose. A comment you would paste at a second site
+ belongs in the class or module docstring instead; pasted at a third it is
+ certainly misplaced. Labels are the exception: a short fixed marker that
+ indexes rather than explains (`# netCDF-only members below.`) is meant to
+ repeat, and should read identically at every site. A repeated sentence of
+ reasoning is not a label.
+- Docstring a dunder only when it says something the signature does not.
+ `D105` is permanently ignored and numpydoc excludes `__repr__`, `__eq__`
+ and `__ne__` — "Return a string representation." is noise nothing asked for.
+
+
+### TODO Comments
+
+`# TODO: ` is house style, and Ruff's `TD` and `FIX` rules are in `ignore`
+(`pyproject.toml`) — use it.
+
+- Mark the deferral in the file, not only in the pull request body.
+- Name the event that removes it: `# TODO: drop once the saver is relocated`.
+- No such event and it is not a TODO: a permanent caveat is a plain comment,
+ schedulable work is an issue.
+
+
+## Facts Learned by Debugging
+
+Most over-long comments in Iris were written after the code already worked. A
+test failed, you investigated, you found something surprising and true, and
+you wrote it down at the line that taught you it. The fact is worth keeping.
+That line is the worst place to keep it, and you are, just then, the reader
+least able to judge how much of it belongs in the file. Assume too much.
+
+- A surprise a test can pin becomes **the test** — named for the trap, detail
+ in its docstring, the source keeping one line that names it. This is the
+ only home that fails loudly when it stops being true.
+- A surprise no test can pin — a library quirk, an invariant, a coupling —
+ goes in the module docstring's traps.
+- A surprise caused by an external bug keeps the URL, not a retelling.
+- Write the fact, not your side of the argument about it. The tells that you
+ have written a reply are in [`WRITING.md`](../../.github/WRITING.md).
+
+
+## Module Docstrings Carry the Explanation
+
+Prose explaining a subsystem belongs in the module docstring. Inline comments
+scatter one argument across dozens of sites; a companion Markdown file is
+validated by nothing and rots unseen. Only the docstring is read on every
+visit, published by Sphinx, checked by numpydoc, and reviewed in the diff.
+
+Be generous here, and only here: sixty lines of orientation above a
+two-thousand-line module is cheap, read once per visit rather than once per
+call site. Not a tutorial.
+
+Cover whichever apply:
+
+- what the module is for, and what it deliberately does not do;
+- the vocabulary — name the concepts the code assumes you already know;
+- invariants the code depends on but cannot assert;
+- why the design is as it is, including approaches tried and rejected;
+- known traps, and the modules this one is coupled to.
+
+Leave out anything a reader or Sphinx can already derive: API listings,
+signatures, call sequences, per-release history. Explanation scoped to a
+single class belongs in that class's docstring instead.
+
+A public module's docstring is also published as an API page, so it must open
+with a `z_reference` item — a title, `:tags:` carrying at least one `topic_*`,
+and a one-line summary — within its first 25 lines. Copy the form from a
+neighbouring module; `pre-commit run check-docs-page-metadata` checks it.
+
+**Exemption to "do not document code you did not change":** if you had to work
+out how a subsystem behaves in order to edit it, write that understanding into
+the module docstring — the comprehension is otherwise discarded when your
+context ends. `_concatenate.py`, `_merge.py` and `common/resolve.py` are the
+most commonly misread modules and carry barely a line each.
+
+
+## Size and Shape
+
+Agents pay for every line they read; humans lose the thread. Existing giants
+such as `cube.py` (~5600 lines) are legacy: do not grow them without cause,
+but do not opportunistically split them either — put genuinely separable new
+code in a private sibling module instead.
+
+- New modules: aim under ~1000 lines, one coherent concept each.
+- Functions: aim to fit one screen (~50 lines).
+- Use guard clauses and early returns instead of nested conditionals.
+- Give a function one exit where it must still warn, record or close something
+ on the way out. A second exit added later silently skips that work.
+- Nesting beyond three levels is a signal to extract a named helper.
+
+
+## Where Human and Agent Preferences Diverge
+
+| Tension | Resolution |
+|---|---|
+| Abstraction vs duplication | Duplicate up to ~3 occurrences; abstract only once the shape is proven. |
+| Clever idiom vs plain code | Plain. No nested comprehensions beyond two levels, no walrus inside complex expressions, no metaclass tricks. |
+| Small files vs cohesion | Cohesion wins. Do not fragment into micro-modules purely to shrink files. |
+| Type hints vs noise | Hint new and changed public code only. |
+| Consistency vs local improvement | Consistency wins. A uniform mediocre idiom beats a mix of good ones. |
+
+
+## Anti-Patterns — Do Not Introduce
+
+- `eval` or `exec` in library code.
+- Silent `except Exception: pass`.
+- Boolean flags that switch a function between two unrelated behaviours —
+ write two functions.
+- In-place mutation of input cubes or coordinates; return new objects.
+- Refactors, renames or "improvements" outside the scope of the change.
+
+
+## Pre-Finish Checklist
+
+- `pre-commit run --files <changed paths>` passes, with any auto-fixes
+ re-staged and the run repeated until clean.
+- New and changed public functions have type hints and NumPy docstrings.
+- No comment or docstring was left stale by the change.
+- If you debugged during this change, re-read the diff as someone who was not
+ there. Comments added after the first working version are where verbosity
+ collects — and where a comment turns into a reply to a conversation the
+ reader never had.
+- Laziness behaviour is preserved and documented.
+- Tests updated per [`tests/AGENTS.md`](tests/AGENTS.md); changelog fragment
+ added per [`../../changelog/AGENTS.md`](../../changelog/AGENTS.md).
+
+
+## ⚠️ Meta-Instruction: Changing This File
+- **Trigger**: If your work establishes a durable, reusable rule, you MUST
+ propose it before your session ends.
+- **Constraint 1**: Propose, never self-apply. Say it in your closing message,
+ or raise it as its own pull request. NEVER edit an `AGENTS.md` silently, or
+ as a side effect of unrelated work.
+- **Constraint 2**: Keep this file under 300 lines — every agent loads it in
+ full. If an addition would break that, tighten your wording; do NOT delete
+ existing guidance to make room. Removing a rule is its own proposal.
+- **Constraint 3**: Only global, reusable lessons. Do not propose temporary or
+ component-specific fixes.
diff --git a/lib/iris/ARCHITECTURE.md b/lib/iris/ARCHITECTURE.md
new file mode 100644
index 0000000000..6d204b7f3b
--- /dev/null
+++ b/lib/iris/ARCHITECTURE.md
@@ -0,0 +1,178 @@
+# ARCHITECTURE.md
+
+An orientation map for `lib/iris`: **where things live and how data flows**.
+
+## Scope of This Document
+
+This file answers "where do I look for X?" and "what talks to what?".
+
+It deliberately does **not** explain how any algorithm works. Subsystem
+detail belongs in the module docstring of the code it describes, where it
+is validated by numpydoc, rendered by Sphinx, and travels in the same diff
+as the code.
+
+The test for anything added here: **would it still be true after a competent
+refactor?** Module locations and data flow survive refactors. Call sequences
+and class member lists do not — do not record them here.
+
+
+## The One-Paragraph Version
+
+Iris turns data files into `Cube` objects and back again. Loading detects a
+file's format, hands it to a format-specific handler that yields many small
+"raw" cubes, then combines those into fewer, larger cubes. Everything in
+between — analysis, arithmetic, regridding, plotting — operates on `Cube`.
+Data inside a cube is normally **lazy** (a Dask array) and is only realised
+when something asks for it.
+
+
+## Layer Map
+
+| Layer | Location | Responsibility |
+|---|---|---|
+| Public API | `__init__.py`, `loading.py` | `load*`, `save`, `Constraint`, policy objects |
+| I/O dispatch | `io/` | URI decoding, format detection, handler dispatch |
+| Format handlers | `fileformats/` | Per-format read/write; CF interpretation |
+| Combination | `_merge.py`, `_concatenate.py`, `_combine.py` | Raw cubes into fewer, larger cubes |
+| Core data model | `cube.py`, `coords.py`, `aux_factory.py`, `mesh/` | `Cube`, coordinates, derived and mesh coords |
+| Metadata | `common/` | Metadata classes, CF property mixin, operand resolution |
+| Laziness | `_lazy_data.py`, `_data_manager.py` | Dask boundary; real-vs-lazy storage |
+| Analysis | `analysis/` | Aggregation, regridding, interpolation, maths |
+| Visualisation | `plot.py`, `quickplot.py`, `palette.py`, `symbols.py` | Matplotlib and Cartopy integration |
+
+
+## The Load Pipeline
+
+```
+iris.load / load_cube / load_cubes / load_raw loading.py
+ -> _load_collection -> _generate_cubes loading.py
+ -> iris.io.decode_uri, expand_filespecs io/__init__.py
+ -> FORMAT_AGENT.get_spec io/format_picker.py
+ matches on magic number, leading line,
+ file extension or URI protocol
+ -> format handler yields raw cubes fileformats/<format>.py
+ -> constraint filtering (_CubeFilterCollection) loading.py
+ -> combination _combine.py
+ -> CubeList.merge -> ProtoCube _merge.py
+ -> CubeList.concatenate _concatenate.py
+```
+
+Key points that are stable over time:
+
+- **`load_raw` skips the combination step.** That is its entire purpose, and
+ it is the fastest way to see what a format handler actually produced.
+- **Format handlers are registered, not hard-coded.** `FORMAT_AGENT` in
+ `fileformats/__init__.py` holds every `FormatSpecification`. To find out
+ which formats are supported, read that registry — do not rely on any list,
+ including this one, staying current.
+- **Handlers are generators** yielding one `Cube` per field or variable.
+ A single PP file commonly yields thousands.
+- **Combination behaviour is policy-driven**, not fixed: `iris.LOAD_POLICY` /
+ `iris.COMBINE_POLICY` (`CombineOptions` in `_combine.py`) decide whether
+ merge, concatenate, both, or neither is applied, and in which order.
+- **Failures are captured, not discarded.** `LOAD_PROBLEMS` in `loading.py`
+ collects objects that could not be loaded, for inspection after the fact.
+
+### Merge versus concatenate
+
+Both combine a `CubeList` into fewer cubes, and they are not interchangeable:
+
+- **merge** (`_merge.py`) builds a *new* dimension from cubes that differ only
+ by scalar coordinate values.
+- **concatenate** (`_concatenate.py`) extends an *existing* dimension by
+ joining cubes along it.
+
+### NetCDF specifics
+
+`fileformats/netcdf/loader.py` drives CF interpretation through a rules engine
+in `fileformats/_nc_load_rules/`, which translates CF metadata into Iris
+coordinates. `fileformats/cf/` provides the CF-level view of a file: it
+classifies variables (`_variables.py`), groups them (`_group.py`) and reads
+them from a dataset (`_reader.py`, the only part coupled to netCDF).
+Never `import netCDF4` directly — use
+`fileformats/netcdf/_thread_safe_nc.py`.
+
+
+## The Save Pipeline
+
+`iris.save` in `io/__init__.py` dispatches on file extension to a
+format-specific saver (`fileformats/netcdf/saver.py`, `pp_save_rules.py`,
+and so on). Saving is *not* a registry lookup — it is extension-based, which
+is a genuine asymmetry with loading and a common source of confusion.
+
+
+## Core Data Model
+
+- **`Cube`** (`cube.py`) — an n-dimensional data array plus CF metadata.
+- **`DimCoord`** — describes exactly one dimension; points must be monotonic.
+- **`AuxCoord`** — may span any number of dimensions, including none.
+- **`CellMeasure`**, **`AncillaryVariable`** — further CF payloads on a cube.
+- **`AuxCoordFactory`** (`aux_factory.py`) — manufactures a coordinate on
+ demand from a formula over other coordinates, e.g. hybrid height. This is
+ why a cube can report a coordinate that is not stored anywhere.
+- **`MeshXY`**, **`MeshCoord`** (`mesh/`) — unstructured UGRID support.
+
+Invariants that hold across the codebase:
+
+- Operations return **new** cubes; they do not mutate their inputs.
+- Metadata is carried by namedtuple-like classes in `common/metadata.py`;
+ `CFVariableMixin` (`common/mixin.py`) gives every object the same CF
+ property interface (`standard_name`, `units`, `attributes`, `rename()`).
+- Comparison can be **lenient or strict** (`common/lenient.py`). Lenient is
+ the default for most user-facing operations, which is why two objects may
+ compare equal despite differing metadata.
+- `Resolve` (`common/resolve.py`) decides the metadata and dimension mapping
+ of the result when two cubes are combined; cube arithmetic goes through it.
+
+
+## Laziness
+
+Laziness is a cross-cutting concern, not a feature of one module.
+
+- `DataManager` (`_data_manager.py`) holds *either* a real array or a lazy
+ one, never both, for cubes and coordinates alike.
+- `_lazy_data.py` is the Dask boundary: `as_lazy_data`, `as_concrete_data`,
+ `co_realise_cubes`, and chunking policy.
+- `cube.data` **realises** data. `cube.core_data()` and `cube.lazy_data()` do
+ not. Library code should avoid `.data` unless realisation is intended.
+
+
+## Structural Oddities Worth Knowing
+
+These are real and will otherwise cost you a confused half-hour:
+
+- **Several modules are very large** — `common/resolve.py` (~2600 lines),
+ `fileformats/pp.py` (~2400), `cube.py` (~5600). They are legacy. Do not
+ grow them unnecessarily, but do not opportunistically split them either.
+- **GRIB support is an external package** (`iris_grib`), loaded through a
+ registered handler; it is not vendored here.
+- **`iris/etc/`** holds colour palettes and a config template — not the CF
+ reference tables, which live in the repository-root `etc/`. CF standard
+ names reach the code as the generated module `std_names.py`.
+- **`experimental/`** is genuinely unstable API and exempt from the usual
+ deprecation guarantees.
+
+
+## Where Do I Look For...?
+
+| Question | Start here |
+|---|---|
+| Why did my file not load? | `LOAD_PROBLEMS` in `loading.py` |
+| Why was my format not recognised? | `FORMAT_AGENT` in `fileformats/__init__.py` |
+| Why did my cubes not merge? | `_merge.py`; compare against `load_raw` output |
+| Why is this cube's metadata wrong after arithmetic? | `common/resolve.py` |
+| Why do these two objects compare equal? | `common/lenient.py` |
+| Why did my data realise unexpectedly? | `_data_manager.py`, `_lazy_data.py` |
+| Where does this coordinate come from? | `aux_factory.py` |
+| How is a CF attribute interpreted? | `fileformats/_nc_load_rules/` |
+
+
+## Maintaining This File
+
+- Keep it under ~200 lines. It is an index, not a manual.
+- Record structure and data flow only. Push explanation down into module
+ docstrings.
+- Do not enumerate things the code already enumerates (format lists, class
+ members, function signatures). Point at the authoritative location instead.
+- If a change here would also need a change to a module docstring, change the
+ docstring and leave this file alone.
diff --git a/lib/iris/_lazy_data.py b/lib/iris/_lazy_data.py
index 1a4eb565c9..63b8e7eed4 100644
--- a/lib/iris/_lazy_data.py
+++ b/lib/iris/_lazy_data.py
@@ -262,7 +262,9 @@ def __repr__(self):
CACHE = LRUCache(MAX_CACHE_SIZE)
-def as_lazy_data(data, chunks=None, asarray=False, meta=None, dims_fixed=None):
+def as_lazy_data(
+ data, chunks=None, asarray=False, meta=None, dims_fixed=None, *, cache_key=None
+):
"""Convert the input array `data` to a :class:`dask.array.Array`.
Parameters
@@ -284,6 +286,11 @@ def as_lazy_data(data, chunks=None, asarray=False, meta=None, dims_fixed=None):
If set, a list of values equal in length to 'chunks' or data.ndim.
'True' values indicate a dimension which can not be changed, i.e. the
result for that index must equal the value in 'chunks' or data.shape.
+ cache_key : str, optional
+ If given, a string identifying `data` well enough that two calls
+ presenting the same string, chunking and `meta` may share one Dask
+ array. Callers wrapping a file-backed proxy pass ``repr(proxy)``.
+ Requires `meta`.
Returns
-------
@@ -299,8 +306,6 @@ def as_lazy_data(data, chunks=None, asarray=False, meta=None, dims_fixed=None):
but reduced by a factor if that exceeds the dask default chunksize.
"""
- from iris.fileformats.netcdf._thread_safe_nc import NetCDFDataProxy
-
if isinstance(data, ma.core.MaskedConstant):
data = ma.masked_array(data.data, mask=data.mask)
@@ -310,6 +315,10 @@ def as_lazy_data(data, chunks=None, asarray=False, meta=None, dims_fixed=None):
"anything other than a Numpy or Dask array."
)
+ if cache_key is not None and meta is None:
+ msg = "`meta` is required when `cache_key` is given: it forms part of the key."
+ raise ValueError(msg)
+
if chunks != "auto":
if chunks is None:
# No existing chunks : Make a chunk the shape of the entire input array
@@ -328,13 +337,14 @@ def as_lazy_data(data, chunks=None, asarray=False, meta=None, dims_fixed=None):
dims_fixed=dims_fixed,
)
- # Define a cache key for caching arrays created from NetCDFDataProxy objects.
- # Creating new Dask arrays is relatively slow, therefore caching is beneficial
- # if many cubes in the same file share coordinate arrays.
- if isinstance(data, NetCDFDataProxy):
- key = (repr(data), chunks, asarray, meta.dtype, type(meta))
- else:
+ # Define a cache key for caching arrays created from data proxies. Creating
+ # new Dask arrays is relatively slow, therefore caching is beneficial if many
+ # cubes in the same file share coordinate arrays. The caller decides what
+ # identifies its data, so this module needs to know no file format.
+ if cache_key is None:
key = None
+ else:
+ key = (cache_key, chunks, asarray, meta.dtype, type(meta))
if is_lazy_data(data):
result = data
diff --git a/lib/iris/cube.py b/lib/iris/cube.py
index 0823b7a847..1452d7f93b 100644
--- a/lib/iris/cube.py
+++ b/lib/iris/cube.py
@@ -1078,7 +1078,7 @@ def __setitem__(self, key, value):
store = self.globals
else:
# If NO existing attribute, create local unless it is a "known global" one.
- from iris.fileformats.netcdf.saver import _CF_GLOBAL_ATTRS
+ from iris.fileformats.cf.saver import _CF_GLOBAL_ATTRS
if key in _CF_GLOBAL_ATTRS:
store = self.globals
diff --git a/lib/iris/fileformats/__init__.py b/lib/iris/fileformats/__init__.py
index 873ca644ef..65df546c49 100644
--- a/lib/iris/fileformats/__init__.py
+++ b/lib/iris/fileformats/__init__.py
@@ -10,7 +10,9 @@
API reference
"""
-from iris.io import _is_nczarr_fragment
+from pathlib import Path
+
+from iris.io import _REMOTE_STORE_SCHEMES, _is_nczarr_fragment, decode_uri
from iris.io.format_picker import (
DataSourceObjectProtocol,
FileExtension,
@@ -19,6 +21,7 @@
LeadingLine,
MagicNumber,
UriProtocol,
+ ZarrStore,
)
from . import name, netcdf, nimrod, pp, um
@@ -44,6 +47,53 @@ def _uri_is_nczarr(uri_parts):
return _is_nczarr_fragment(fragment)
+#: What a version 3 and a version 2 Zarr store put at the root of a store.
+_ZARR_MARKERS = ("zarr.json", ".zgroup")
+
+
+def _is_zarr_store(target):
+ """Whether `target` names a native Zarr store.
+
+ Locally the test is a marker file, which costs one stat and is what tells
+ a real store from a directory someone named ``.zarr``. Remotely it is the
+ scheme and the suffix, because reading a marker file would cost a network
+ round trip before Iris knows it wants the target at all; a remote
+ ``.zarr`` that turns out not to be a store fails when zarr-python opens
+ it, with zarr-python's message.
+
+ These are two tests for two kinds of target, not two chances for one. A
+ third rule matching any path ending in ``.zarr`` would subsume the remote
+ one and leave the scheme test unreachable.
+ """
+ if not isinstance(target, str):
+ # iris.io.load_data_objects passes open datasets, not paths.
+ return False
+
+ scheme, part, fragment = decode_uri(target)
+ if _is_nczarr_fragment(fragment):
+ # The user asked for netCDF-c's Zarr driver by name. The store on
+ # disk is a real one, so the local test below would otherwise claim it.
+ return False
+
+ if scheme in _REMOTE_STORE_SCHEMES:
+ return part.rstrip("/").endswith(".zarr")
+
+ directory = Path(part)
+ return any((directory / marker).is_file() for marker in _ZARR_MARKERS)
+
+
+def _load_zarr(*args, **kwargs):
+ """Load from Zarr stores.
+
+ ``iris.fileformats.zarr``'s own ``__getattr__`` raises a clear
+ ``ImportError`` - naming the package and the version floor - if ``zarr``
+ is absent or too old; nothing here needs to repeat that.
+ """
+ from iris.fileformats.zarr import load_cubes
+
+ return load_cubes(*args, **kwargs)
+
+
#
# PP files.
#
@@ -169,6 +219,18 @@ def _load_grib(*args, **kwargs):
#
# Zarr files.
#
+FORMAT_AGENT.add_spec(
+ FormatSpecification(
+ "Zarr",
+ ZarrStore(),
+ _is_zarr_store,
+ _load_zarr,
+ priority=5,
+ constraint_aware_handler=True,
+ )
+)
+
+
FORMAT_AGENT.add_spec(
FormatSpecification(
"NcZarr",
diff --git a/lib/iris/fileformats/_nc_load_rules/actions.py b/lib/iris/fileformats/_nc_load_rules/actions.py
index 5f441c55bc..4d600c9420 100644
--- a/lib/iris/fileformats/_nc_load_rules/actions.py
+++ b/lib/iris/fileformats/_nc_load_rules/actions.py
@@ -176,7 +176,7 @@ def action_provides_grid_mapping(engine, gridmapping_fact):
(var_name,) = gridmapping_fact
rule_name = "fc_provides_grid_mapping"
cf_var = engine.cf_var.cf_group[var_name]
- grid_mapping_type = getattr(cf_var, hh.CF_ATTR_GRID_MAPPING_NAME, None)
+ grid_mapping_type = cf_var.attributes.get(hh.CF_ATTR_GRID_MAPPING_NAME)
succeed = True
if grid_mapping_type is None:
@@ -557,7 +557,7 @@ def action_all_managed_attributes(engine):
iris_name = handler.iris_name
matches = []
for match_name in handler.netcdf_names:
- match_value = getattr(var, match_name, None)
+ match_value = var.attributes.get(match_name)
if match_value is not None:
matches.append((match_name, match_value))
@@ -623,7 +623,7 @@ def action_formula_type(engine, formula_root_fact):
(var_name,) = formula_root_fact
cf_var = engine.cf_var.cf_group[var_name]
# cf_var.standard_name is a formula type (or we should never get here).
- formula_type = getattr(cf_var, "standard_name", None)
+ formula_type = cf_var.attributes.get("standard_name")
succeed = True
if formula_type not in iris.fileformats.cf.reference_terms:
succeed = False
@@ -652,7 +652,7 @@ def action_formula_type(engine, formula_root_fact):
category=_WarnComboLoadIgnoring,
)
rule_name += f"_{formula_type}"
- # Set 'requires' info for iris.fileformats.netcdf._load_aux_factory.
+ # Set 'requires' info for iris.fileformats.cf.loader._load_aux_factory.
engine.requires["formula_type"] = formula_type
return rule_name
diff --git a/lib/iris/fileformats/_nc_load_rules/engine.py b/lib/iris/fileformats/_nc_load_rules/engine.py
index 60b6de596e..b410506964 100644
--- a/lib/iris/fileformats/_nc_load_rules/engine.py
+++ b/lib/iris/fileformats/_nc_load_rules/engine.py
@@ -15,7 +15,7 @@
engine.get_kb() also returns a FactEntity object, which mimics *just enough*
API of a Pyke.knowlege_base, so that we can list its case-specific facts, as
-used in :meth:`iris.fileformats.netcdf._actions_activation_stats`.
+used in :meth:`iris.fileformats.cf.loader._actions_activation_stats`.
"""
diff --git a/lib/iris/fileformats/_nc_load_rules/helpers.py b/lib/iris/fileformats/_nc_load_rules/helpers.py
index 7c4810ffe7..54c4590cb1 100644
--- a/lib/iris/fileformats/_nc_load_rules/helpers.py
+++ b/lib/iris/fileformats/_nc_load_rules/helpers.py
@@ -36,8 +36,9 @@
from iris.cube import Cube
import iris.exceptions
import iris.fileformats.cf as cf
+from iris.fileformats.cf._variables import _dimension_identities
+from iris.fileformats.cf.loader import _get_cf_var_data
import iris.fileformats.netcdf
-from iris.fileformats.netcdf.loader import _get_cf_var_data
from iris.loading import LOAD_PROBLEMS, LoadProblems
import iris.std_names
import iris.util
@@ -563,8 +564,8 @@ def _add_or_capture(
# best to capture objects IF possible.
if attr_key is not None:
captured_attr = None
- with contextlib.suppress(AttributeError):
- captured_attr = getattr(cf_var, attr_key)
+ with contextlib.suppress(KeyError):
+ captured_attr = cf_var.attributes[attr_key]
captured = {attr_key: captured_attr}
else:
with contextlib.suppress(Exception):
@@ -613,7 +614,7 @@ def build_raw_cube(cf_var: cf.CFVariable) -> Cube:
################################################################################
def _build_name_standard(cf_var: cf.CFVariable) -> str | None:
- value = getattr(cf_var, CF_ATTR_STD_NAME, None)
+ value = cf_var.attributes.get(CF_ATTR_STD_NAME)
if value is not None:
standard_name = _get_valid_standard_name(value)
else:
@@ -622,7 +623,7 @@ def _build_name_standard(cf_var: cf.CFVariable) -> str | None:
def _build_name_long(cf_var: cf.CFVariable) -> str | None:
- return getattr(cf_var, CF_ATTR_LONG_NAME, None)
+ return cf_var.attributes.get(CF_ATTR_LONG_NAME)
def _build_name_var(cf_var: cf.CFVariable) -> str | None:
@@ -742,7 +743,7 @@ def build_and_add_units(engine: Engine):
################################################################################
def _build_cell_methods(cf_var: cf.CFDataVariable) -> List[iris.coords.CellMethod]:
- nc_att_cell_methods = getattr(cf_var, CF_ATTR_CELL_METHODS, None)
+ nc_att_cell_methods = cf_var.attributes.get(CF_ATTR_CELL_METHODS)
return parse_cell_methods(nc_att_cell_methods, cf_var.cf_name)
@@ -771,9 +772,9 @@ def _get_ellipsoid(cf_grid_var):
`cf_grid_var`. Returns None if no relevant properties are specified.
"""
- major = getattr(cf_grid_var, CF_ATTR_GRID_SEMI_MAJOR_AXIS, None)
- minor = getattr(cf_grid_var, CF_ATTR_GRID_SEMI_MINOR_AXIS, None)
- inverse_flattening = getattr(cf_grid_var, CF_ATTR_GRID_INVERSE_FLATTENING, None)
+ major = cf_grid_var.attributes.get(CF_ATTR_GRID_SEMI_MAJOR_AXIS)
+ minor = cf_grid_var.attributes.get(CF_ATTR_GRID_SEMI_MINOR_AXIS)
+ inverse_flattening = cf_grid_var.attributes.get(CF_ATTR_GRID_INVERSE_FLATTENING)
# Avoid over-specification exception.
if major is not None and minor is not None:
@@ -781,12 +782,12 @@ def _get_ellipsoid(cf_grid_var):
# Check for a default spherical earth.
if major is None and minor is None and inverse_flattening is None:
- major = getattr(cf_grid_var, CF_ATTR_GRID_EARTH_RADIUS, None)
+ major = cf_grid_var.attributes.get(CF_ATTR_GRID_EARTH_RADIUS)
- datum = getattr(cf_grid_var, CF_ATTR_GRID_DATUM, None)
+ datum = cf_grid_var.attributes.get(CF_ATTR_GRID_DATUM)
# Check crs_wkt if no datum
if datum is None:
- crs_wkt = getattr(cf_grid_var, CF_ATTR_GRID_CRS_WKT, None)
+ crs_wkt = cf_grid_var.attributes.get(CF_ATTR_GRID_CRS_WKT)
if crs_wkt is not None:
proj_crs = pyproj.crs.CRS.from_wkt(crs_wkt)
if proj_crs.datum is not None:
@@ -829,15 +830,17 @@ def build_rotated_coordinate_system(engine, cf_grid_var):
"""Create a rotated coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- north_pole_latitude = getattr(cf_grid_var, CF_ATTR_GRID_NORTH_POLE_LAT, 90.0)
- north_pole_longitude = getattr(cf_grid_var, CF_ATTR_GRID_NORTH_POLE_LON, 0.0)
+ north_pole_latitude = cf_grid_var.attributes.get(CF_ATTR_GRID_NORTH_POLE_LAT, 90.0)
+ north_pole_longitude = cf_grid_var.attributes.get(CF_ATTR_GRID_NORTH_POLE_LON, 0.0)
if north_pole_latitude is None or north_pole_longitude is None:
warnings.warn(
"Rotated pole position is not fully specified",
category=iris.warnings.IrisCfLoadWarning,
)
- north_pole_grid_lon = getattr(cf_grid_var, CF_ATTR_GRID_NORTH_POLE_GRID_LON, 0.0)
+ north_pole_grid_lon = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_NORTH_POLE_GRID_LON, 0.0
+ )
rcs = iris.coord_systems.RotatedGeogCS(
north_pole_latitude,
@@ -854,27 +857,27 @@ def build_transverse_mercator_coordinate_system(engine, cf_grid_var):
"""Create a transverse Mercator coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- longitude_of_central_meridian = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_CENT_MERIDIAN, None
+ longitude_of_central_meridian = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_CENT_MERIDIAN
)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
- scale_factor_at_central_meridian = getattr(
- cf_grid_var, CF_ATTR_GRID_SCALE_FACTOR_AT_CENT_MERIDIAN, None
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
+ scale_factor_at_central_meridian = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_SCALE_FACTOR_AT_CENT_MERIDIAN
)
# The following accounts for the inconsistency in the transverse
# mercator description within the CF spec.
if longitude_of_central_meridian is None:
- longitude_of_central_meridian = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_PROJ_ORIGIN, None
+ longitude_of_central_meridian = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_PROJ_ORIGIN
)
if scale_factor_at_central_meridian is None:
- scale_factor_at_central_meridian = getattr(
- cf_grid_var, CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN, None
+ scale_factor_at_central_meridian = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN
)
cs = iris.coord_systems.TransverseMercator(
@@ -894,15 +897,15 @@ def build_lambert_conformal_coordinate_system(engine, cf_grid_var):
"""Create a Lambert conformal conic coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- longitude_of_central_meridian = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_CENT_MERIDIAN, None
+ longitude_of_central_meridian = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_CENT_MERIDIAN
)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
- standard_parallel = getattr(cf_grid_var, CF_ATTR_GRID_STANDARD_PARALLEL, None)
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
+ standard_parallel = cf_grid_var.attributes.get(CF_ATTR_GRID_STANDARD_PARALLEL)
cs = iris.coord_systems.LambertConformal(
latitude_of_projection_origin,
@@ -921,18 +924,18 @@ def build_stereographic_coordinate_system(engine, cf_grid_var):
"""Create a stereographic coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- longitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_PROJ_ORIGIN, None
+ longitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_PROJ_ORIGIN
)
- scale_factor_at_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN, None
+ scale_factor_at_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN
)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
cs = iris.coord_systems.Stereographic(
latitude_of_projection_origin,
@@ -952,19 +955,19 @@ def build_polar_stereographic_coordinate_system(engine, cf_grid_var):
"""Create a polar stereographic coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- longitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_STRAIGHT_VERT_LON, None
+ longitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_STRAIGHT_VERT_LON
)
- true_scale_lat = getattr(cf_grid_var, CF_ATTR_GRID_STANDARD_PARALLEL, None)
- scale_factor_at_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN, None
+ true_scale_lat = cf_grid_var.attributes.get(CF_ATTR_GRID_STANDARD_PARALLEL)
+ scale_factor_at_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN
)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
cs = iris.coord_systems.PolarStereographic(
latitude_of_projection_origin,
@@ -984,14 +987,14 @@ def build_mercator_coordinate_system(engine, cf_grid_var):
"""Create a Mercator coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- longitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_PROJ_ORIGIN, None
+ longitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_PROJ_ORIGIN
)
- standard_parallel = getattr(cf_grid_var, CF_ATTR_GRID_STANDARD_PARALLEL, None)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
- scale_factor_at_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN, None
+ standard_parallel = cf_grid_var.attributes.get(CF_ATTR_GRID_STANDARD_PARALLEL)
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
+ scale_factor_at_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN
)
cs = iris.coord_systems.Mercator(
@@ -1011,14 +1014,14 @@ def build_lambert_azimuthal_equal_area_coordinate_system(engine, cf_grid_var):
"""Create a lambert azimuthal equal area coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- longitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_PROJ_ORIGIN, None
+ longitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_PROJ_ORIGIN
)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
cs = iris.coord_systems.LambertAzimuthalEqualArea(
latitude_of_projection_origin,
@@ -1036,15 +1039,15 @@ def build_albers_equal_area_coordinate_system(engine, cf_grid_var):
"""Create a albers conical equal area coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- longitude_of_central_meridian = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_CENT_MERIDIAN, None
+ longitude_of_central_meridian = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_CENT_MERIDIAN
)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
- standard_parallels = getattr(cf_grid_var, CF_ATTR_GRID_STANDARD_PARALLEL, None)
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
+ standard_parallels = cf_grid_var.attributes.get(CF_ATTR_GRID_STANDARD_PARALLEL)
cs = iris.coord_systems.AlbersEqualArea(
latitude_of_projection_origin,
@@ -1063,17 +1066,17 @@ def build_vertical_perspective_coordinate_system(engine, cf_grid_var):
"""Create a vertical perspective coordinate system from the CF-netCDF grid mapping variables."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- longitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_PROJ_ORIGIN, None
+ longitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_PROJ_ORIGIN
)
- perspective_point_height = getattr(
- cf_grid_var, CF_ATTR_GRID_PERSPECTIVE_HEIGHT, None
+ perspective_point_height = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_PERSPECTIVE_HEIGHT
)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
cs = iris.coord_systems.VerticalPerspective(
latitude_of_projection_origin,
@@ -1092,18 +1095,18 @@ def build_geostationary_coordinate_system(engine, cf_grid_var):
"""Create a geostationary coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- longitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_PROJ_ORIGIN, None
+ longitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_PROJ_ORIGIN
)
- perspective_point_height = getattr(
- cf_grid_var, CF_ATTR_GRID_PERSPECTIVE_HEIGHT, None
+ perspective_point_height = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_PERSPECTIVE_HEIGHT
)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
- sweep_angle_axis = getattr(cf_grid_var, CF_ATTR_GRID_SWEEP_ANGLE_AXIS, None)
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
+ sweep_angle_axis = cf_grid_var.attributes.get(CF_ATTR_GRID_SWEEP_ANGLE_AXIS)
cs = iris.coord_systems.Geostationary(
latitude_of_projection_origin,
@@ -1123,18 +1126,18 @@ def build_oblique_mercator_coordinate_system(engine, cf_grid_var):
"""Create an oblique mercator coordinate system from the CF-netCDF grid mapping variable."""
ellipsoid = _get_ellipsoid(cf_grid_var)
- azimuth_of_central_line = getattr(cf_grid_var, CF_ATTR_GRID_AZIMUTH_CENT_LINE, None)
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ azimuth_of_central_line = cf_grid_var.attributes.get(CF_ATTR_GRID_AZIMUTH_CENT_LINE)
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- longitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LON_OF_PROJ_ORIGIN, None
+ longitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LON_OF_PROJ_ORIGIN
)
- scale_factor_at_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN, None
+ scale_factor_at_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN
)
- false_easting = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_EASTING, None)
- false_northing = getattr(cf_grid_var, CF_ATTR_GRID_FALSE_NORTHING, None)
+ false_easting = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_EASTING)
+ false_northing = cf_grid_var.attributes.get(CF_ATTR_GRID_FALSE_NORTHING)
kwargs = dict(
azimuth_of_central_line=azimuth_of_central_line,
latitude_of_projection_origin=latitude_of_projection_origin,
@@ -1146,7 +1149,7 @@ def build_oblique_mercator_coordinate_system(engine, cf_grid_var):
)
# Handle the alternative form noted in CF: rotated mercator.
- grid_mapping_name = getattr(cf_grid_var, CF_ATTR_GRID_MAPPING_NAME)
+ grid_mapping_name = cf_grid_var.attributes[CF_ATTR_GRID_MAPPING_NAME]
candidate_systems = dict(
oblique_mercator=iris.coord_systems.ObliqueMercator,
rotated_mercator=iris.coord_systems.RotatedMercator,
@@ -1166,7 +1169,7 @@ def build_oblique_mercator_coordinate_system(engine, cf_grid_var):
################################################################################
def get_attr_units(cf_var, attributes, capture_invalid=False):
- attr_units = getattr(cf_var, CF_ATTR_UNITS, UNKNOWN_UNIT_STRING)
+ attr_units = cf_var.attributes.get(CF_ATTR_UNITS, UNKNOWN_UNIT_STRING)
if not attr_units:
attr_units = UNKNOWN_UNIT_STRING
@@ -1210,14 +1213,16 @@ def get_attr_units(cf_var, attributes, capture_invalid=False):
attr_units = UNKNOWN_UNIT_STRING
if any(
- hasattr(cf_var.cf_data, name)
+ # Deliberately untracked: asking whether these exist must not count as
+ # using them, or they would stop being copied onto the cube.
+ name in cf_var.attributes.untracked
for name in ("flag_values", "flag_masks", "flag_meanings")
):
attr_units = cf_units._NO_UNIT_STRING
# Get any associated calendar for a time reference coordinate.
if cf_units.as_unit(attr_units).is_time_reference():
- attr_calendar = getattr(cf_var, CF_ATTR_CALENDAR, None)
+ attr_calendar = cf_var.attributes.get(CF_ATTR_CALENDAR)
if attr_calendar:
attr_units = cf_units.Unit(attr_units, calendar=attr_calendar)
@@ -1228,8 +1233,8 @@ def get_attr_units(cf_var, attributes, capture_invalid=False):
################################################################################
def get_names(cf_coord_var, coord_name, attributes):
"""Determine the standard_name, long_name and var_name attributes."""
- standard_name = getattr(cf_coord_var, CF_ATTR_STD_NAME, None)
- long_name = getattr(cf_coord_var, CF_ATTR_LONG_NAME, None)
+ standard_name = cf_coord_var.attributes.get(CF_ATTR_STD_NAME)
+ long_name = cf_coord_var.attributes.get(CF_ATTR_LONG_NAME)
cf_name = str(cf_coord_var.cf_name)
if standard_name is not None:
@@ -1264,8 +1269,9 @@ def get_names(cf_coord_var, coord_name, attributes):
################################################################################
def get_cf_bounds_var(cf_coord_var):
"""Return the CF variable representing the bounds of a coordinate variable."""
- attr_bounds = getattr(cf_coord_var, CF_ATTR_BOUNDS, None)
- attr_climatology = getattr(cf_coord_var, CF_ATTR_CLIMATOLOGY, None)
+ attributes = cf_coord_var.attributes
+ attr_bounds = attributes.get(CF_ATTR_BOUNDS)
+ attr_climatology = attributes.get(CF_ATTR_CLIMATOLOGY)
# Determine bounds, preferring standard bounds over climatology.
# NB. No need to raise a warning if the bounds/climatology
@@ -1466,6 +1472,35 @@ def _build_dimension_coordinate(
return coord
+def _common_data_dims(
+ cf_var: cf.CFVariable, cf_data_var: cf.CFVariable
+) -> Optional[list[int]]:
+ """Return the dimensions of `cf_data_var` that `cf_var` shares with it.
+
+ Two variables share a dimension when they name it identically *and* it is
+ the same length in both - the ``(name, size)`` rule
+ :meth:`iris.fileformats.cf.CFVariable.spans` decides spanning by. The bare
+ name is not enough: a Zarr store may give one variable two dimensions of
+ the same name at two different lengths, and ``dimensions.index(name)``
+ would then answer with the first of them whichever one was meant. No
+ netCDF file can tell the difference, a netCDF dimension name being unique
+ within its group and carrying one size.
+
+ Returns ``None`` rather than an empty list when nothing is shared, because
+ that is what the ``Cube.add_*`` methods take for "no dimensions".
+ """
+ # `_dimension_identities` rather than a pairing written out here, so that
+ # the rule deciding a variable spans the data variable and the rule
+ # attaching it to its dimensions are one rule, not two that can drift.
+ data_identities = _dimension_identities(cf_data_var)
+ data_dims = [
+ data_identities.index(identity)
+ for identity in _dimension_identities(cf_var)
+ if identity in data_identities
+ ]
+ return data_dims or None
+
+
def _add_dimension_coordinate(
engine: Engine,
cf_coord_var: cf.CFCoordinateVariable,
@@ -1475,15 +1510,9 @@ def _add_dimension_coordinate(
assert engine.cube is not None
assert engine.cube_parts is not None
- # Determine the name of the dimension/s shared between the CF-netCDF
- # data variable and the coordinate being built.
- common_dims = [
- dim for dim in cf_coord_var.dimensions if dim in engine.cf_var.dimensions
- ]
- data_dims = None
- if common_dims:
- # Calculate the offset of each common dimension.
- data_dims = [int(engine.cf_var.dimensions.index(dim)) for dim in common_dims]
+ # Determine which dimension/s of the CF-netCDF data variable the
+ # coordinate being built covers.
+ data_dims = _common_data_dims(cf_coord_var, engine.cf_var)
if hasattr(coord, "circular") and data_dims is not None:
# Appease MyPy. The check itself uses duck typing to avoid any
@@ -1634,15 +1663,9 @@ def _add_auxiliary_coordinate(
assert engine.cube is not None
assert engine.cube_parts is not None
- # Determine the name of the dimension/s shared between the CF-netCDF data variable
- # and the coordinate being built.
- coord_dims = cf_coord_var.dimensions
- datavar_dims = engine.cf_var.dimensions
- common_dims = [dim for dim in coord_dims if dim in datavar_dims]
- data_dims = None
- if common_dims:
- # Calculate the offset of each common dimension.
- data_dims = [engine.cf_var.dimensions.index(dim) for dim in common_dims]
+ # Determine which dimension/s of the CF-netCDF data variable the
+ # coordinate being built covers.
+ data_dims = _common_data_dims(cf_coord_var, engine.cf_var)
engine.cube.add_aux_coord(coord, data_dims)
@@ -1718,13 +1741,9 @@ def _add_cell_measure(
cf_var = engine.cf_var
cube = engine.cube
- # Determine the name of the dimension/s shared between the CF-netCDF data
- # variable and the coordinate being built.
- common_dims = [dim for dim in cf_cm_var.dimensions if dim in cf_var.dimensions]
- data_dims = None
- if common_dims:
- # Calculate the offset of each common dimension.
- data_dims = [cf_var.dimensions.index(dim) for dim in common_dims]
+ # Determine which dimension/s of the CF-netCDF data variable the cell
+ # measure being built covers.
+ data_dims = _common_data_dims(cf_cm_var, cf_var)
# Add it to the cube
cube.add_cell_measure(cell_measure, data_dims)
@@ -1790,13 +1809,9 @@ def _add_ancil_var(
cf_var = engine.cf_var
cube = engine.cube
- # Determine the name of the dimension/s shared between the CF-netCDF data variable
- # and the AV being built.
- common_dims = [dim for dim in cf_av_var.dimensions if dim in cf_var.dimensions]
- data_dims = None
- if common_dims:
- # Calculate the offset of each common dimension.
- data_dims = [cf_var.dimensions.index(dim) for dim in common_dims]
+ # Determine which dimension/s of the CF-netCDF data variable the ancillary
+ # variable being built covers.
+ data_dims = _common_data_dims(cf_av_var, cf_var)
# Add it to the cube
cube.add_ancillary_variable(av, data_dims)
@@ -1833,7 +1848,7 @@ def _is_lat_lon(cf_var, ud_units, std_name, std_name_grid, axis_name, prefixes):
"""
is_valid = False
- attr_units = getattr(cf_var, CF_ATTR_UNITS, None)
+ attr_units = cf_var.attributes.get(CF_ATTR_UNITS)
if isinstance(attr_units, str):
attr_units = attr_units.lower()
@@ -1841,18 +1856,18 @@ def _is_lat_lon(cf_var, ud_units, std_name, std_name_grid, axis_name, prefixes):
# Special case - Check for rotated pole.
if attr_units == "degrees":
- attr_std_name = getattr(cf_var, CF_ATTR_STD_NAME, None)
+ attr_std_name = cf_var.attributes.get(CF_ATTR_STD_NAME)
if attr_std_name is not None:
is_valid = attr_std_name.lower() == std_name_grid
else:
is_valid = False
# TODO: check that this interpretation of axis is correct.
- attr_axis = getattr(cf_var, CF_ATTR_AXIS, None)
+ attr_axis = cf_var.attributes.get(CF_ATTR_AXIS)
if attr_axis is not None:
is_valid = attr_axis.lower() == axis_name
else:
# Alternative is to check standard_name or axis.
- attr_std_name = getattr(cf_var, CF_ATTR_STD_NAME, None)
+ attr_std_name = cf_var.attributes.get(CF_ATTR_STD_NAME)
if attr_std_name is not None:
attr_std_name = attr_std_name.lower()
@@ -1862,7 +1877,7 @@ def _is_lat_lon(cf_var, ud_units, std_name, std_name_grid, axis_name, prefixes):
[attr_std_name.startswith(prefix) for prefix in prefixes]
)
else:
- attr_axis = getattr(cf_var, CF_ATTR_AXIS, None)
+ attr_axis = cf_var.attributes.get(CF_ATTR_AXIS)
if attr_axis is not None:
is_valid = attr_axis.lower() == axis_name
@@ -1902,8 +1917,8 @@ def is_longitude(engine, cf_name):
def is_projection_x_coordinate(engine, cf_name):
"""Determine whether the CF coordinate variable is a projection_x_coordinate variable."""
cf_var = engine.cf_var.cf_group[cf_name]
- attr_name = getattr(cf_var, CF_ATTR_STD_NAME, None) or getattr(
- cf_var, CF_ATTR_LONG_NAME, None
+ attr_name = cf_var.attributes.get(CF_ATTR_STD_NAME) or cf_var.attributes.get(
+ CF_ATTR_LONG_NAME
)
return attr_name == CF_VALUE_STD_NAME_PROJ_X
@@ -1912,8 +1927,8 @@ def is_projection_x_coordinate(engine, cf_name):
def is_projection_y_coordinate(engine, cf_name):
"""Determine whether the CF coordinate variable is a projection_y_coordinate variable."""
cf_var = engine.cf_var.cf_group[cf_name]
- attr_name = getattr(cf_var, CF_ATTR_STD_NAME, None) or getattr(
- cf_var, CF_ATTR_LONG_NAME, None
+ attr_name = cf_var.attributes.get(CF_ATTR_STD_NAME) or cf_var.attributes.get(
+ CF_ATTR_LONG_NAME
)
return attr_name == CF_VALUE_STD_NAME_PROJ_Y
@@ -1926,10 +1941,10 @@ def is_time(engine, cf_name):
"""
cf_var = engine.cf_var.cf_group[cf_name]
- attr_units = getattr(cf_var, CF_ATTR_UNITS, None)
+ attr_units = cf_var.attributes.get(CF_ATTR_UNITS)
- attr_std_name = getattr(cf_var, CF_ATTR_STD_NAME, None)
- attr_axis = getattr(cf_var, CF_ATTR_AXIS, "")
+ attr_std_name = cf_var.attributes.get(CF_ATTR_STD_NAME)
+ attr_axis = cf_var.attributes.get(CF_ATTR_AXIS, "")
try:
is_time_reference = cf_units.Unit(attr_units or 1).is_time_reference()
except ValueError:
@@ -1945,7 +1960,7 @@ def is_time_period(engine, cf_name):
"""Determine whether the CF coordinate variable represents a time period."""
is_valid = False
cf_var = engine.cf_var.cf_group[cf_name]
- attr_units = getattr(cf_var, CF_ATTR_UNITS, None)
+ attr_units = cf_var.attributes.get(CF_ATTR_UNITS)
if attr_units is not None:
try:
@@ -1961,7 +1976,7 @@ def is_grid_mapping(engine, cf_name, grid_mapping):
"""Determine whether the CF grid mapping variable is of the appropriate type."""
is_valid = False
cf_var = engine.cf_var.cf_group[cf_name]
- attr_mapping_name = getattr(cf_var, CF_ATTR_GRID_MAPPING_NAME, None)
+ attr_mapping_name = cf_var.attributes.get(CF_ATTR_GRID_MAPPING_NAME)
if attr_mapping_name is not None:
is_valid = attr_mapping_name.lower() == grid_mapping
@@ -2009,12 +2024,12 @@ def _is_rotated(engine, cf_name, cf_attr_value):
"""Determine whether the CF coordinate variable is rotated."""
is_valid = False
cf_var = engine.cf_var.cf_group[cf_name]
- attr_std_name = getattr(cf_var, CF_ATTR_STD_NAME, None)
+ attr_std_name = cf_var.attributes.get(CF_ATTR_STD_NAME)
if attr_std_name is not None:
is_valid = attr_std_name.lower() == cf_attr_value
else:
- attr_units = getattr(cf_var, CF_ATTR_UNITS, None)
+ attr_units = cf_var.attributes.get(CF_ATTR_UNITS)
if attr_units is not None:
is_valid = attr_units.lower() == "degrees"
@@ -2043,9 +2058,9 @@ def has_supported_mercator_parameters(engine, cf_name):
is_valid = True
cf_grid_var = engine.cf_var.cf_group[cf_name]
- standard_parallel = getattr(cf_grid_var, CF_ATTR_GRID_STANDARD_PARALLEL, None)
- scale_factor_at_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN, None
+ standard_parallel = cf_grid_var.attributes.get(CF_ATTR_GRID_STANDARD_PARALLEL)
+ scale_factor_at_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN
)
if scale_factor_at_projection_origin is not None and standard_parallel is not None:
@@ -2070,13 +2085,13 @@ def has_supported_polar_stereographic_parameters(engine, cf_name):
is_valid = True
cf_grid_var = engine.cf_var.cf_group[cf_name]
- latitude_of_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN, None
+ latitude_of_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_LAT_OF_PROJ_ORIGIN
)
- standard_parallel = getattr(cf_grid_var, CF_ATTR_GRID_STANDARD_PARALLEL, None)
- scale_factor_at_projection_origin = getattr(
- cf_grid_var, CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN, None
+ standard_parallel = cf_grid_var.attributes.get(CF_ATTR_GRID_STANDARD_PARALLEL)
+ scale_factor_at_projection_origin = cf_grid_var.attributes.get(
+ CF_ATTR_GRID_SCALE_FACTOR_AT_PROJ_ORIGIN
)
if latitude_of_projection_origin != 90 and latitude_of_projection_origin != -90:
diff --git a/lib/iris/fileformats/cf/__init__.py b/lib/iris/fileformats/cf/__init__.py
new file mode 100644
index 0000000000..31b4747a75
--- /dev/null
+++ b/lib/iris/fileformats/cf/__init__.py
@@ -0,0 +1,78 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Provide capability to load netCDF files and interpret them.
+
+.. z_reference:: iris.fileformats.cf
+ :tags: topic_load_save
+
+ API reference
+
+Provides the capability to load netCDF files and interpret them
+according to the 'NetCDF Climate and Forecast (CF) Metadata Conventions'.
+
+This package is a re-export surface and holds no logic of its own. Every
+public name below is defined in exactly one private module and imported here
+verbatim, so ``class CFReader`` still finds its definition in a single grep.
+Import from :mod:`iris.fileformats.cf`, not from the private modules: the file
+layout is deliberately free to change, and it will, as the format-agnostic CF
+machinery grows a dataset abstraction, a loader and a saver.
+
+The re-exported classes keep the ``__module__`` their defining module gives
+them, so ``repr(CFReader)`` names ``iris.fileformats.cf._reader`` rather than
+this package. That is the same trade-off :mod:`iris.mesh` already makes, and
+it is the one that keeps ``inspect.getsource`` and the documentation's
+``[source]`` links working. See the plan, section 8.4.
+
+References
+----------
+ [CF] NetCDF Climate and Forecast (CF) Metadata conventions.
+ [NUG] NetCDF User's Guide, https://docs.unidata.ucar.edu/nug/current/
+
+"""
+
+from ._group import CFGroup
+from ._reader import CFReader, reference_terms
+from ._variables import (
+ CFAncillaryDataVariable,
+ CFAuxiliaryCoordinateVariable,
+ CFBoundaryVariable,
+ CFClimatologyVariable,
+ CFCoordinateVariable,
+ CFDataVariable,
+ CFGridMappingVariable,
+ CFLabelVariable,
+ CFMeasureVariable,
+ CFUGridAuxiliaryCoordinateVariable,
+ CFUGridConnectivityVariable,
+ CFUGridMeshVariable,
+ CFVariable,
+)
+
+__all__ = [
+ "CFAncillaryDataVariable",
+ "CFAuxiliaryCoordinateVariable",
+ "CFBoundaryVariable",
+ "CFClimatologyVariable",
+ "CFCoordinateVariable",
+ "CFDataVariable",
+ "CFGridMappingVariable",
+ "CFGroup",
+ "CFLabelVariable",
+ "CFMeasureVariable",
+ "CFReader",
+ "CFUGridAuxiliaryCoordinateVariable",
+ "CFUGridConnectivityVariable",
+ "CFUGridMeshVariable",
+ "CFVariable",
+ "reference_terms",
+]
+
+#: Supported dimensionless vertical coordinate reference surface/phemomenon
+#: formula terms. Ref: [CF] Appendix D.
+reference_terms = reference_terms
+# Re-stating the name here, with the doc comment that travels with it, is what
+# puts it on the API reference page. Sphinx autodoc documents module data only
+# where the module's own source assigns it; an imported name it silently drops,
+# with no warning and no build failure. See the plan, section 8.4.
diff --git a/lib/iris/fileformats/netcdf/_dask_locks.py b/lib/iris/fileformats/cf/_dask_locks.py
similarity index 56%
rename from lib/iris/fileformats/netcdf/_dask_locks.py
rename to lib/iris/fileformats/cf/_dask_locks.py
index b677f1af28..df04c95698 100644
--- a/lib/iris/fileformats/netcdf/_dask_locks.py
+++ b/lib/iris/fileformats/cf/_dask_locks.py
@@ -2,51 +2,38 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Module containing code to create locks enabling dask workers to co-operate.
-
-This matter is complicated by needing different solutions for different dask
-scheduler types, i.e. local 'threads' scheduler, local 'processes' or
-distributed.
-
-In any case, an "iris.fileformats.netcdf.saver.Saver" object contains a
-netCDF4.Dataset targeting an output file, and creates a Saver.file_write_lock
-object to serialise write-accesses to the file from dask tasks : All dask-task
-file writes go via a "iris.fileformats.netcdf.saver.NetCDFWriteProxy" object,
-which also contains a link to the Saver.file_write_lock, and uses it to prevent
-workers from fouling each other.
-
-For each chunk written, the NetCDFWriteProxy acquires the common per-file lock;
-opens a Dataset on the file; performs a write to the relevant variable; closes
-the Dataset and then releases the lock. This process is obviously very similar
-to what the NetCDFDataProxy does for reading lazy chunks.
-
-For a threaded scheduler, the Saver.lock is a simple threading.Lock(). The
-workers (threads) execute tasks which contain a NetCDFWriteProxy, as above.
-All of those contain the common lock, and this is simply **the same object**
-for all workers, since they share an address space.
-
-For a distributed scheduler, the Saver.lock is a `distributed.Lock()` which is
-identified with the output filepath. This is distributed to the workers by
-serialising the task function arguments, which will include the
-NetCDFWriteProxy. A worker behaves like a process, though it may execute on a
-remote machine. When a distributed.Lock is deserialised to reconstruct the
-worker task, this creates an object that communicates with the scheduler.
-These objects behave as a single common lock, as they all have the same string
-'identity', so the scheduler implements inter-process communication so that
-they can mutually exclude each other.
-
-It is also *conceivable* that multiple processes could write to the same file in
-parallel, if the operating system supports it. However, this also requires
-that the libnetcdf C library is built with parallel access option, which is
-not common. With the "ordinary" libnetcdf build, a process which attempts to
-open for writing a file which is _already_ open for writing simply raises an
-access error. In any case, Iris netcdf saver will not support this mode of
-operation, at present.
-
-We don't currently support a local "processes" type scheduler. If we did, the
-behaviour should be very similar to a distributed scheduler. It would need to
-use some other serialisable shared-lock solution in place of
-'distributed.Lock', which requires a distributed scheduler to function.
+"""Locks that let dask workers co-operate when writing to one target.
+
+A saver writing lazy data hands each chunk to a dask worker, and the workers
+must not write over each other. What that takes depends on the dask scheduler,
+so this module works out which scheduler is in play and returns a lock to
+match. The scheduler must therefore be configured before a lock is asked for.
+
+A dataset implementation asks for one lock per target, identified by its path
+or URL, and gives it to every write handle it builds. The netCDF write handle
+reopens the file, acquires the lock, writes its chunk, closes the file and
+releases the lock, which is the mirror image of what the read path does for a
+lazy chunk. A store whose chunks are separate objects needs no such
+coordination, and asks for no lock.
+
+For a threaded scheduler the lock is a plain :class:`threading.Lock`. The
+workers are threads sharing one address space, so they all hold the same
+object, and the identity string is never used.
+
+For a distributed scheduler the lock is a ``distributed.Lock`` named for the
+target. It reaches the workers by being serialised along with the task, and
+every copy carries the same name, so the scheduler can make them exclude one
+another. A worker behaves like a process, and may run on another machine.
+
+The "processes" scheduler is refused by :func:`get_worker_lock`. Supporting it
+would need a serialisable shared lock that does not require a distributed
+scheduler to function.
+
+Writing to one target from genuinely separate processes is a further step
+again. For netCDF it needs libnetcdf built with the parallel access option,
+which is uncommon: with an ordinary build, opening a file for writing that is
+already open for writing simply raises an access error. Iris does not support
+that mode at present.
"""
diff --git a/lib/iris/fileformats/cf/_group.py b/lib/iris/fileformats/cf/_group.py
new file mode 100644
index 0000000000..f38586a3a8
--- /dev/null
+++ b/lib/iris/fileformats/cf/_group.py
@@ -0,0 +1,215 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Collect classified CF-netCDF variables into a single addressable group.
+
+:class:`CFGroup` is the output of classification and the input to rule-based
+loading. It is a :class:`~collections.abc.MutableMapping` from CF-netCDF
+variable name to the :class:`~iris.fileformats.cf.CFVariable` instance that
+:mod:`iris.fileformats.cf._variables` classified it as, so a group holds every
+variable in a file exactly once, whatever its type.
+
+On top of that mapping sit one read-only property per variable type --
+:attr:`CFGroup.bounds`, :attr:`CFGroup.grid_mappings` and so on -- each of
+which filters the mapping by ``isinstance``. They are computed on every access
+rather than cached, so a group stays correct when
+:class:`~iris.fileformats.cf.CFReader` mutates it during its second pass. A
+variable therefore appears in every property whose type it satisfies, and
+:attr:`CFGroup.formula_terms` is the one exception to the ``isinstance``
+rule: it filters on ``has_formula_terms()`` instead, because participating in
+a formula term is a property of the variable rather than of its class.
+
+Two attributes sit alongside the variables and are populated by the reader,
+not by this module. ``global_attributes`` holds the file's netCDF global
+attributes. ``promoted`` records the variables that the reader re-classified
+as :class:`~iris.fileformats.cf.CFDataVariable`, which it does for two
+distinct reasons: a formula term that names a reference surface is a data
+pay-load in its own right, per [CF] Appendix D; and a variable that was
+referenced but whose dimensions are not a subset of the referrer's is dropped
+from that referrer with a warning, then promoted so that it is not lost
+altogether. Promotion is how Iris avoids silently discarding either.
+
+"""
+
+from collections.abc import MutableMapping
+
+from ._variables import (
+ CFAncillaryDataVariable,
+ CFAuxiliaryCoordinateVariable,
+ CFBoundaryVariable,
+ CFClimatologyVariable,
+ CFCoordinateVariable,
+ CFDataVariable,
+ CFGridMappingVariable,
+ CFLabelVariable,
+ CFMeasureVariable,
+ CFUGridAuxiliaryCoordinateVariable,
+ CFUGridConnectivityVariable,
+ CFUGridMeshVariable,
+ CFVariable,
+)
+
+
+################################################################################
+class CFGroup(MutableMapping):
+ """Collection of 'NetCDF CF Metadata Conventions variables and netCDF global attributes.
+
+ Represents a collection of 'NetCDF Climate and Forecast (CF) Metadata
+ Conventions' variables and netCDF global attributes.
+
+ """
+
+ def __init__(self):
+ #: Collection of CF-netCDF variables
+ self._cf_variables = {}
+ #: Collection of netCDF global attributes
+ self.global_attributes = {}
+ #: Collection of CF-netCDF variables promoted to a CFDataVariable.
+ self.promoted = {}
+
+ def _cf_getter(self, cls):
+ # Generate dictionary with dictionary comprehension.
+ return {
+ cf_name: cf_var
+ for cf_name, cf_var in self._cf_variables.items()
+ if isinstance(cf_var, cls)
+ }
+
+ @property
+ def ancillary_variables(self):
+ """Collection of CF-netCDF ancillary variables."""
+ return self._cf_getter(CFAncillaryDataVariable)
+
+ @property
+ def auxiliary_coordinates(self):
+ """Collection of CF-netCDF auxiliary coordinate variables."""
+ return self._cf_getter(CFAuxiliaryCoordinateVariable)
+
+ @property
+ def bounds(self):
+ """Collection of CF-netCDF boundary variables."""
+ return self._cf_getter(CFBoundaryVariable)
+
+ @property
+ def climatology(self):
+ """Collection of CF-netCDF climatology variables."""
+ return self._cf_getter(CFClimatologyVariable)
+
+ @property
+ def coordinates(self):
+ """Collection of CF-netCDF coordinate variables."""
+ return self._cf_getter(CFCoordinateVariable)
+
+ @property
+ def data_variables(self):
+ """Collection of CF-netCDF data pay-load variables."""
+ return self._cf_getter(CFDataVariable)
+
+ @property
+ def formula_terms(self):
+ """Collection of CF-netCDF variables that participate in a CF-netCDF formula term."""
+ return {
+ cf_name: cf_var
+ for cf_name, cf_var in self._cf_variables.items()
+ if cf_var.has_formula_terms()
+ }
+
+ @property
+ def grid_mappings(self):
+ """Collection of CF-netCDF grid mapping variables."""
+ return self._cf_getter(CFGridMappingVariable)
+
+ @property
+ def labels(self):
+ """Collection of CF-netCDF label variables."""
+ return self._cf_getter(CFLabelVariable)
+
+ @property
+ def cell_measures(self):
+ """Collection of CF-netCDF measure variables."""
+ return self._cf_getter(CFMeasureVariable)
+
+ @property
+ def non_data_variable_names(self):
+ """:class:`set` names of the CF-netCDF variables that are not the data pay-load."""
+ non_data_variables = (
+ self.ancillary_variables,
+ self.auxiliary_coordinates,
+ self.bounds,
+ self.climatology,
+ self.coordinates,
+ self.grid_mappings,
+ self.labels,
+ self.cell_measures,
+ self.connectivities,
+ self.ugrid_coords,
+ self.meshes,
+ )
+ result = set()
+ for variable in non_data_variables:
+ result |= set(variable)
+ return result
+
+ @property
+ def connectivities(self):
+ """Collection of CF-UGRID connectivity variables."""
+ return self._cf_getter(CFUGridConnectivityVariable)
+
+ @property
+ def ugrid_coords(self):
+ """Collection of CF-UGRID-relevant auxiliary coordinate variables."""
+ return self._cf_getter(CFUGridAuxiliaryCoordinateVariable)
+
+ @property
+ def meshes(self):
+ """Collection of CF-UGRID mesh variables."""
+ return self._cf_getter(CFUGridMeshVariable)
+
+ def keys(self):
+ """Return the names of all the CF-netCDF variables in the group."""
+ return self._cf_variables.keys()
+
+ def __len__(self):
+ return len(self._cf_variables)
+
+ def __iter__(self):
+ for item in self._cf_variables:
+ yield item
+
+ def __setitem__(self, name, variable):
+ if not isinstance(variable, CFVariable):
+ raise TypeError(
+ "Attempted to add an invalid CF-netCDF variable to the %s"
+ % self.__class__.__name__
+ )
+
+ if name != variable.cf_name:
+ raise ValueError(
+ "Mismatch between key name %r and CF-netCDF variable name %r"
+ % (str(name), variable.cf_name)
+ )
+
+ self._cf_variables[name] = variable
+
+ def __getitem__(self, name):
+ if name not in self._cf_variables:
+ raise KeyError("Cannot get unknown CF-netCDF variable name %r" % str(name))
+
+ return self._cf_variables[name]
+
+ def __delitem__(self, name):
+ if name not in self._cf_variables:
+ raise KeyError(
+ "Cannot delete unknown CF-netcdf variable name %r" % str(name)
+ )
+
+ del self._cf_variables[name]
+
+ def __repr__(self):
+ result = []
+ result.append("variables:%d" % len(self._cf_variables))
+ result.append("global_attributes:%d" % len(self.global_attributes))
+ result.append("promoted:%d" % len(self.promoted))
+
+ return "<%s of %s>" % (self.__class__.__name__, ", ".join(result))
diff --git a/lib/iris/fileformats/cf/_reader.py b/lib/iris/fileformats/cf/_reader.py
new file mode 100644
index 0000000000..2b9caceb43
--- /dev/null
+++ b/lib/iris/fileformats/cf/_reader.py
@@ -0,0 +1,537 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Read a netCDF dataset and interpret it as CF-netCDF variables and groups.
+
+:class:`CFReader` is the entry point to the CF layer: give it a file path or
+an already-open dataset and it returns an object whose ``cf_group`` is a fully
+populated :class:`~iris.fileformats.cf.CFGroup`. All of the work happens in
+:meth:`CFReader.__init__`, in two passes over the dataset's variables.
+
+The first pass, ``_translate``, is classification. It asks each class in
+``_variable_types`` to ``identify`` itself among the netCDF variables, then
+handles the three special cases that tuple deliberately excludes:
+:class:`~iris.fileformats.cf.CFCoordinateVariable` (identified by the netCDF
+dimension/variable name coincidence rather than by an attribute),
+``_CFFormulaTermsVariable`` (whose members are also something else) and
+:class:`~iris.fileformats.cf.CFDataVariable` (what is left over once every
+other classification has had its say). It also caches the parse of every
+``grid_mapping`` attribute in ``_coord_system_mappings``, mapping a variable
+name to the coordinate-system-to-coordinates relationships it declares, so
+that the loading rules do not re-parse the same attribute once per coordinate.
+
+The second pass, ``_build_cf_groups``, is association: for each variable it
+builds the sub-:class:`~iris.fileformats.cf.CFGroup` of the variables that
+variable references, and checks that a referenced variable's dimensions span
+the referrer's. One that does not is not an error -- it is dropped from the
+referrer's group with a warning and, if nothing else claims it, *promoted* to
+a :class:`~iris.fileformats.cf.CFDataVariable` of its own. A final
+``_reset`` clears the attribute touch history that classification dirtied, so
+that the loading rules start from a clean record of which netCDF attributes
+have been consumed.
+
+``CFReader`` may or may not own the file it reads, and ``_own_file`` records
+which. Constructed from a path, it opens the dataset and owns it; constructed
+from an open dataset, it borrows it. ``_close`` -- reached from ``__exit__``
+and from ``__del__`` -- only closes what it owns, so a borrowed dataset
+survives garbage collection of the reader. ``_own_file`` is set ``False``
+before anything that can raise, so the destructor is safe even if
+``__init__`` fails part way.
+
+``CFReader`` takes one of three things. A path opens a netCDF file, because
+netCDF is the default a bare filename means. An open netCDF4 dataset, or an
+emulator of one such as ncdata passes, is wrapped. And a
+:class:`~iris.fileformats.cf.dataset.CFDataset` of any format is used as it
+stands, which is how a Zarr store reaches this machinery: its loader opens
+the store and hands it over, so nothing here has to know that Zarr exists.
+Only the first case mentions netCDF, and only because something has to be
+the default.
+
+"""
+
+from collections.abc import Iterable
+from pathlib import Path
+from typing import Optional
+from urllib.parse import urlparse
+import warnings
+
+import iris.exceptions
+import iris.fileformats._nc_load_rules.helpers as hh
+from iris.fileformats.netcdf._dataset import NetCDFDataset
+import iris.warnings
+
+from ._group import CFGroup
+from ._variables import (
+ CFAncillaryDataVariable,
+ CFAuxiliaryCoordinateVariable,
+ CFBoundaryVariable,
+ CFClimatologyVariable,
+ CFCoordinateVariable,
+ CFDataVariable,
+ CFGridMappingVariable,
+ CFLabelVariable,
+ CFMeasureVariable,
+ CFUGridAuxiliaryCoordinateVariable,
+ CFUGridConnectivityVariable,
+ CFUGridMeshVariable,
+ _CFFormulaTermsVariable,
+)
+from .dataset import CFDataset
+
+#: Supported dimensionless vertical coordinate reference surface/phemomenon
+#: formula terms. Ref: [CF] Appendix D.
+reference_terms = dict(
+ atmosphere_sigma_coordinate=["ps"],
+ atmosphere_hybrid_sigma_pressure_coordinate=["ps"],
+ atmosphere_hybrid_height_coordinate=["orog"],
+ atmosphere_sleve_coordinate=["zsurf1", "zsurf2"],
+ ocean_sigma_coordinate=["eta", "depth"],
+ ocean_s_coordinate=["eta", "depth"],
+ ocean_sigma_z_coordinate=["eta", "depth"],
+ ocean_s_coordinate_g1=["eta", "depth"],
+ ocean_s_coordinate_g2=["eta", "depth"],
+)
+
+
+################################################################################
+class CFReader:
+ """Allows the contents of a netCDF file to be interpreted.
+
+ This class allows the contents of a netCDF file to be interpreted according
+ to the 'NetCDF Climate and Forecast (CF) Metadata Conventions'.
+
+ """
+
+ # All CF variable types EXCEPT for the "special cases" of
+ # CFDataVariable, CFCoordinateVariable and _CFFormulaTermsVariable.
+ _variable_types = (
+ CFAncillaryDataVariable,
+ CFAuxiliaryCoordinateVariable,
+ CFBoundaryVariable,
+ CFClimatologyVariable,
+ CFGridMappingVariable,
+ CFLabelVariable,
+ CFMeasureVariable,
+ CFUGridConnectivityVariable,
+ CFUGridAuxiliaryCoordinateVariable,
+ CFUGridMeshVariable,
+ )
+
+ #: Alias of :class:`~iris.fileformats.cf.CFGroup`, for callers that reach
+ #: it through the reader.
+ #:
+ #: Hidden from the documentation because autodoc would otherwise describe
+ #: ``CFGroup`` twice - once as a member of the package and once as this
+ #: attribute - and the two descriptions collide, which fails the build
+ #: outright under Read the Docs' ``fail_on_warning``. The alias itself is
+ #: unaffected; only its duplicate documentation entry goes.
+ #:
+ #: :meta private:
+ CFGroup = CFGroup
+
+ def __init__(self, file_source, warn=False, monotonic=False):
+ # Ensure safe operation for destructor, should init fail.
+ self._own_file = False
+ if isinstance(file_source, CFDataset):
+ # Already open, in whatever format: use it, don't own it. The
+ # caller that opened it is responsible for closing it.
+ self._dataset = file_source
+ self._filename = file_source.location
+ elif isinstance(file_source, str):
+ # Create from filepath : open it + own it (=close when we die).
+ if not urlparse(file_source).scheme:
+ self._filename = Path(file_source).expanduser()
+ else:
+ self._filename = file_source
+
+ self._dataset = NetCDFDataset(
+ self._filename, mode="r", warn_legacy_format=warn
+ )
+ self._own_file = True
+ else:
+ # We have been passed an open dataset.
+ # We use it but don't own it (don't close it).
+ self._dataset = NetCDFDataset.from_existing(
+ file_source, warn_legacy_format=warn
+ )
+ self._filename = self._dataset.location
+
+ #: Collection of CF-netCDF variables associated with this netCDF file
+ self.cf_group = self.CFGroup()
+
+ # Result of parsing "grid_mapping" attribute; mapping of coordinate_system => coordinates
+ self._coord_system_mappings = {}
+
+ self._check_monotonic = monotonic
+
+ self._with_ugrid = True
+ if not self._has_meshes():
+ self._trim_ugrid_variable_types()
+ self._with_ugrid = False
+
+ variables = self._dataset.variables
+ self._translate(variables)
+ self._build_cf_groups(variables)
+ self._reset(variables)
+
+ def __enter__(self):
+ # Enable use as a context manager
+ # N.B. this **guarantees* closure of the file, when the context is exited.
+ # Note: ideally, the class would not do so much work in the __init__ call, and
+ # would do all that here, after acquiring necessary permissions/locks.
+ # But for legacy reasons, we can't do that. So **effectively**, the context
+ # (in terms of access control) already started, when we created the object.
+ return self
+
+ def __exit__(self, exc_type, exc_value, traceback):
+ # When used as a context-manager, **always** close the file on exit.
+ self._close()
+
+ def _has_meshes(self):
+ result = False
+ for variable in self._dataset.variables.values():
+ attributes = variable.attributes
+ if "mesh" in attributes or "node_coordinates" in attributes:
+ result = True
+ break
+ return result
+
+ def _trim_ugrid_variable_types(self):
+ self._variable_types = (
+ CFAncillaryDataVariable,
+ CFAuxiliaryCoordinateVariable,
+ CFBoundaryVariable,
+ CFClimatologyVariable,
+ CFGridMappingVariable,
+ CFLabelVariable,
+ CFMeasureVariable,
+ )
+
+ @property
+ def filename(self):
+ """The file that the CFReader is reading."""
+ return self._filename
+
+ def __repr__(self):
+ return "%s(%r)" % (self.__class__.__name__, self._filename)
+
+ def _translate(self, variables):
+ """Classify the netCDF variables into CF-netCDF variables."""
+ netcdf_variable_names = list(variables.keys())
+
+ # Parse all instances of "grid_mapping" attributes and store in CFReader
+ # This avoids re-parsing the grid_mappings each time they are needed.
+ for nc_var in variables.values():
+ if grid_mapping_attr := nc_var.attributes.get("grid_mapping"):
+ try:
+ cs_mappings = hh._parse_extended_grid_mapping(grid_mapping_attr)
+ self._coord_system_mappings[nc_var.name] = cs_mappings
+ except iris.exceptions.CFParseError as e:
+ msg = f"Error parsing `grid_mapping` attribute for {nc_var.name}: {str(e)}"
+ warnings.warn(msg, category=iris.warnings.IrisCfWarning)
+ continue
+
+ # Identify all CF coordinate variables first. This must be done
+ # first as, by CF convention, the definition of a CF auxiliary
+ # coordinate variable may include a scalar CF coordinate variable,
+ # whereas we want these two types of variables to be mutually exclusive.
+ coords = CFCoordinateVariable.identify(
+ variables, monotonic=self._check_monotonic
+ )
+ self.cf_group.update(coords)
+ coordinate_names = list(self.cf_group.coordinates.keys())
+
+ # Identify all CF variables EXCEPT for the "special cases".
+ for variable_type in self._variable_types:
+ # Prevent grid mapping variables being mis-identified as CF coordinate variables.
+ ignore = (
+ None
+ if issubclass(variable_type, CFGridMappingVariable)
+ else coordinate_names
+ )
+ kwargs = (
+ {"coord_system_mappings": self._coord_system_mappings}
+ if issubclass(variable_type, CFGridMappingVariable)
+ else {}
+ )
+
+ self.cf_group.update(
+ variable_type.identify(variables, ignore=ignore, **kwargs)
+ )
+
+ # Identify global netCDF attributes.
+ self.cf_group.global_attributes.update(dict(self._dataset.attributes))
+
+ # Identify and register all CF formula terms.
+ formula_terms = _CFFormulaTermsVariable.identify(variables)
+
+ if iris.FUTURE.derived_bounds:
+ # Keep track of all the root vars so we can unpick invalid bounds vars
+ all_roots = set()
+
+ # cf_var = CFFormulaTermsVariable (loops through everything that appears in formula terms)
+ for cf_var in formula_terms.values():
+ # Example of a formula term:
+ # Suppose in the file eta:formula_terms contains "a: var_A"
+ # cf_var = var_A, cf_root = eta and cf_term = 'a'. cf_var.cf_terms_by_root = {eta: 'a'}
+ for cf_root, cf_term in cf_var.cf_terms_by_root.items():
+ if iris.FUTURE.derived_bounds:
+ # For the "newstyle" derived-bounds implementation, find vars which appear in derived bounds terms
+ # and turn them into bounds vars (though they don't appear in a "bounds" attribute)
+
+ # Adds each root only once
+ all_roots.add(cf_root)
+
+ # cf_root_coord = CFCoordinateVariable or CFAuxiliaryCoordinateVariable of the coordinate relating to the root
+ cf_root_coord = self.cf_group.coordinates.get(cf_root)
+ if cf_root_coord is None:
+ cf_root_coord = self.cf_group.auxiliary_coordinates.get(cf_root)
+
+ # N.B. cf_root_coord may here be None, if the root var was not a
+ # coord - that is ok, it will not have a 'bounds', we will skip it.
+ root_bounds_name = None
+ if cf_root_coord is not None:
+ root_bounds_name = cf_root_coord.attributes.get("bounds")
+ if root_bounds_name in self.cf_group:
+ root_bounds_var = self.cf_group.get(root_bounds_name)
+ if "formula_terms" not in root_bounds_var.attributes:
+ # this is an invalid root bounds, according to CF, and therefore should be promoted into a cube
+ root_bounds_var._to_be_promoted = True
+ else:
+ # Found a valid *root* bounds variable : search for a corresponding *term* bounds variable,
+ term_bounds_vars = [
+ # loop through all formula terms and add them if they have a cf_term_by_root
+ # where (bounds of cf_root): cf_term (same as before)
+ f
+ for f in formula_terms.values()
+ if f.cf_terms_by_root.get(root_bounds_name) == cf_term
+ ]
+ if len(term_bounds_vars) == 1:
+ (term_bounds_var,) = term_bounds_vars
+ # N.B. bounds==main-var is valid CF for *no* bounds
+ if term_bounds_var != cf_var:
+ cf_var.attributes["bounds"] = (
+ term_bounds_var.cf_name
+ )
+ new_var = CFBoundaryVariable(
+ term_bounds_var.cf_name, term_bounds_var.cf_data
+ )
+ new_var.add_formula_term(root_bounds_name, cf_term)
+ # "Reclassify" this var as a bounds variable
+ self.cf_group[term_bounds_var.cf_name] = new_var
+
+ if cf_root not in self.cf_group.bounds:
+ # This records all formula terms in the main cf_group that were previously only stored in the formula_terms dictionary.
+ cf_name = cf_var.cf_name
+ if cf_name not in self.cf_group:
+ # If the formula term variable is not already in the group, add it as a coordinate.
+ new_var = CFAuxiliaryCoordinateVariable(cf_name, cf_var.cf_data)
+ if iris.FUTURE.derived_bounds and "bounds" in cf_var.attributes:
+ # Copy "old-style" derived bounds link
+ new_var.attributes["bounds"] = cf_var.attributes["bounds"]
+ self.cf_group[cf_name] = new_var
+
+ self.cf_group[cf_name].add_formula_term(cf_root, cf_term)
+
+ if iris.FUTURE.derived_bounds:
+ for cf_root in all_roots:
+ # Invalidate "broken" bounds connections
+ root_var = self.cf_group[cf_root]
+ if root_var.attributes.get("formula_terms") and root_var.attributes.get(
+ "bounds"
+ ):
+ root_bounds_var = self.cf_group.get(root_var.attributes["bounds"])
+ if root_bounds_var is None or not root_bounds_var.attributes.get(
+ "formula_terms"
+ ):
+ # This means it is *not* a valid bounds var, according to CF, and so therefore we are
+ # invalidating the bounds.
+ root_var.attributes["bounds"] = None
+
+ # Determine the CF data variables.
+ data_variable_names = (
+ set(netcdf_variable_names) - self.cf_group.non_data_variable_names
+ )
+
+ for name in data_variable_names:
+ self.cf_group[name] = CFDataVariable(name, variables[name])
+
+ def _build_cf_groups(self, variables):
+ """Build the first order relationships between CF-netCDF variables."""
+
+ def _build(cf_variable):
+ is_mesh_var = isinstance(cf_variable, CFUGridMeshVariable)
+ ugrid_coord_names = []
+ ugrid_coords = getattr(self.cf_group, "ugrid_coords", None)
+ if ugrid_coords is not None:
+ ugrid_coord_names = list(ugrid_coords.keys())
+
+ coordinate_names = list(self.cf_group.coordinates.keys())
+ cf_group = self.CFGroup()
+
+ def _span_check(
+ var_name: str, via_formula_terms: Optional[str] = None
+ ) -> None:
+ """Sanity check dimensionality."""
+ var = self.cf_group[var_name]
+ # No span check is necessary if variable is attached to a mesh.
+ if (is_mesh_var or var.spans(cf_variable)) and not var._to_be_promoted:
+ cf_group[var_name] = var
+ else:
+ # Register the ignored variable.
+ # N.B. 'ignored' variable from enclosing scope.
+ ignored.add(var_name)
+
+ text_formula = text_via = ""
+ if via_formula_terms:
+ text_formula = " formula terms"
+ text_via = f" via variable {via_formula_terms}"
+
+ message = (
+ f"Ignoring{text_formula} variable {var_name} "
+ f"referenced by variable {cf_variable.cf_name}"
+ f"{text_via}: Dimensions {var.dimensions} do not span "
+ f"{cf_variable.dimensions}"
+ )
+ warnings.warn(
+ message,
+ category=iris.warnings.IrisCfNonSpanningVarWarning,
+ )
+
+ # Build CF variable relationships.
+ for variable_type in self._variable_types:
+ ignore = []
+ kwargs = {}
+ # Avoid UGridAuxiliaryCoordinateVariables also being
+ # processed as CFAuxiliaryCoordinateVariables.
+ if not is_mesh_var:
+ ignore += ugrid_coord_names
+ # Prevent grid mapping variables being mis-identified as CF coordinate variables.
+ if issubclass(variable_type, CFGridMappingVariable):
+ # pass parsed grid_mappings to CFGridMappingVariable types
+ kwargs.update(
+ {"coord_system_mappings": self._coord_system_mappings}
+ )
+ else:
+ ignore += coordinate_names
+
+ match = variable_type.identify(
+ variables,
+ ignore=ignore,
+ target=cf_variable.cf_name,
+ warn=False,
+ **kwargs,
+ )
+ # Sanity check dimensionality coverage.
+ for cf_name in match:
+ _span_check(cf_name)
+
+ if iris.FUTURE.derived_bounds:
+ # Include bounds of every variable, within cf_group attached to the variable.
+ if "bounds" in cf_variable.attributes:
+ bounds_name = cf_variable.attributes["bounds"]
+ if bounds_name not in cf_group:
+ bounds_var = self.cf_group.get(bounds_name)
+ if bounds_var:
+ # TODO: warning if span fails
+ if bounds_var.spans(cf_variable):
+ cf_group[bounds_name] = bounds_var
+
+ # Build CF data variable relationships.
+ if isinstance(cf_variable, CFDataVariable):
+ # Add global netCDF attributes.
+ cf_group.global_attributes.update(self.cf_group.global_attributes)
+ # Add appropriate "dimensioned" CF coordinate variables.
+ cf_group.update(
+ {
+ cf_name: self.cf_group[cf_name]
+ for cf_name in cf_variable.dimensions
+ if cf_name in self.cf_group.coordinates
+ }
+ )
+ # Add appropriate "dimensionless" CF coordinate variables.
+ coordinates_attr = cf_variable.attributes.get("coordinates", "")
+ cf_group.update(
+ {
+ cf_name: self.cf_group[cf_name]
+ for cf_name in coordinates_attr.split()
+ if cf_name in self.cf_group.coordinates
+ }
+ )
+ # Add appropriate formula terms.
+ for cf_var in self.cf_group.formula_terms.values():
+ for cf_root in cf_var.cf_terms_by_root:
+ if cf_root in cf_group and cf_var.cf_name not in cf_group:
+ _span_check(cf_var.cf_name, cf_root)
+
+ # Add the CF group to the variable.
+ cf_variable.cf_group = cf_group
+
+ # Ignored variables are those that cannot be attached to a
+ # data variable as the dimensionality of that variable is not
+ # a subset of the dimensionality of the data variable.
+ ignored = set()
+
+ for cf_variable in self.cf_group.values():
+ _build(cf_variable)
+
+ # Determine whether there are any formula terms that
+ # may be promoted to a CFDataVariable and restrict promotion to only
+ # those formula terms that are reference surface/phenomenon.
+ for cf_var in self.cf_group.formula_terms.values():
+ if iris.FUTURE.derived_bounds:
+ # Always False: cf_group[...] yields a CFVariable instance, not
+ # the class. See https://github.com/SciTools/iris/issues/7296
+ if self.cf_group[cf_var.cf_name] is CFBoundaryVariable:
+ continue
+ for cf_root, cf_term in cf_var.cf_terms_by_root.items():
+ cf_root_var = self.cf_group[cf_root]
+ if iris.FUTURE.derived_bounds:
+ if "standard_name" not in cf_root_var.attributes:
+ continue
+ name = (
+ cf_root_var.attributes["standard_name"]
+ or cf_root_var.attributes["long_name"]
+ )
+ terms = reference_terms.get(name, [])
+ if isinstance(terms, str) or not isinstance(terms, Iterable):
+ terms = [terms]
+ cf_var_name = cf_var.cf_name
+ if cf_term in terms and cf_var_name not in self.cf_group.promoted:
+ data_var = CFDataVariable(cf_var_name, cf_var.cf_data)
+ self.cf_group.promoted[cf_var_name] = data_var
+ _build(data_var)
+ break
+
+ # Promote any ignored variables.
+ promoted = set()
+ not_promoted = ignored.difference(promoted)
+ while not_promoted:
+ cf_name = not_promoted.pop()
+ if (
+ cf_name not in self.cf_group.data_variables
+ and cf_name not in self.cf_group.promoted
+ ):
+ data_var = CFDataVariable(cf_name, self.cf_group[cf_name].cf_data)
+ self.cf_group.promoted[cf_name] = data_var
+ _build(data_var)
+ # Determine whether there are still any ignored variables
+ # yet to be promoted.
+ promoted.add(cf_name)
+ not_promoted = ignored.difference(promoted)
+
+ def _reset(self, variables):
+ """Reset the attribute touch history of each variable."""
+ for nc_var_name in variables.keys():
+ self.cf_group[nc_var_name].cf_attrs_reset()
+
+ def _close(self):
+ # Explicitly close dataset to prevent file remaining open.
+ if self._own_file and self._dataset is not None:
+ self._dataset.close()
+ self._dataset = None
+
+ def __del__(self):
+ # Be sure to close dataset when CFReader is destroyed / garbage-collected.
+ self._close()
diff --git a/lib/iris/fileformats/cf.py b/lib/iris/fileformats/cf/_variables.py
similarity index 56%
rename from lib/iris/fileformats/cf.py
rename to lib/iris/fileformats/cf/_variables.py
index 2879e71596..91f229bd21 100644
--- a/lib/iris/fileformats/cf.py
+++ b/lib/iris/fileformats/cf/_variables.py
@@ -2,37 +2,58 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Provide capability to load netCDF files and interpret them.
-
-.. z_reference:: iris.fileformats.cf
- :tags: topic_load_save
-
- API reference
-
-Provides the capability to load netCDF files and interpret them
-according to the 'NetCDF Climate and Forecast (CF) Metadata Conventions'.
-
-References
-----------
- [CF] NetCDF Climate and Forecast (CF) Metadata conventions.
- [NUG] NetCDF User's Guide, https://docs.unidata.ucar.edu/nug/current/
+"""Classify netCDF variables by the role the CF conventions give them.
+
+A CF-netCDF file is a flat bag of variables; the conventions turn it into a
+structure by having variables name each other in their attributes. A
+``bounds`` attribute names a bounds variable, a ``grid_mapping`` attribute
+names a coordinate system variable, and so on. This module is where those
+attributes are read and each variable is given a class that says what it is.
+
+:class:`CFVariable` is the abstract base and defines the contract. A subclass
+declares which netCDF attribute names it -- ``cf_identity``, a single
+attribute name, or ``cf_identities``, a list of them for the classes that
+answer to several -- and implements ``identify``. ``identify`` is a
+classmethod, not an instance method: it is called on the class, is handed the
+whole ``{name: CFDatasetVariable}`` mapping of the file, and returns the
+subset of it that belongs to that class, as a ``{name: CFVariable instance}``
+mapping. ``ignore`` and ``target`` narrow what it looks at; ``warn`` gates
+whether it complains.
+
+Classification is deliberately forgiving. A variable that names a variable
+which is not in the file, or names one whose dimensions make no sense for the
+role, produces a warning -- :class:`~iris.warnings.IrisCfMissingVarWarning`
+and friends -- and is skipped. It does not raise. Iris is expected to load
+what it can from a file that is only mostly conformant, and a hard failure
+here would make a single bad attribute cost the user the whole file.
+
+The CF-UGRID classes for unstructured meshes live here rather than in
+:mod:`iris.mesh` because they are classifiers, not data: they answer the same
+``identify`` contract as every other class in this module and are driven by
+the same pass in :class:`~iris.fileformats.cf.CFReader`. What they identify is
+handed to :mod:`iris.mesh` to build the mesh proper.
+
+This module opens no files. It is given netCDF variables and returns
+classifications; :mod:`iris.fileformats.cf._group` collects them and
+:mod:`iris.fileformats.cf._reader` drives the whole thing. It does read array
+data in exactly one place: ``CFCoordinateVariable.identify`` fetches a
+candidate's values when called with ``monotonic=True``, because whether a
+coordinate is monotonic cannot be told from its metadata. That is the only
+path here that touches data, and it is reached only from
+``CFReader(..., monotonic=True)``.
"""
from abc import ABCMeta, abstractmethod
-from collections.abc import Iterable, MutableMapping
-from pathlib import Path
import re
-from typing import ClassVar, Optional
-from urllib.parse import urlparse
+from typing import ClassVar
import warnings
import numpy as np
import numpy.ma as ma
-import iris.exceptions
-import iris.fileformats._nc_load_rules.helpers as hh
-from iris.fileformats.netcdf import _bytecoding_datasets, _thread_safe_nc
+from iris._deprecation import warn_deprecated
+from iris.fileformats.cf.dataset import TrackedAttributes
from iris.mesh.components import Connectivity
import iris.util
import iris.warnings
@@ -55,23 +76,73 @@
# NCZarr stores scalar variables as size-1 arrays on this pseudo-dimension.
_NCZARR_SCALAR_DIMENSION = "_scalar_"
+
+class _DimensionIdentity:
+ """One dimension of a CF variable, as something to match on.
+
+ A named dimension compares on ``(name, size)``: two variables share it
+ when they name it identically *and* it is the same length in both. A name
+ reused at two lengths links nothing, which is the only reading that
+ cannot produce a wrong cube.
+
+ An unnamed dimension is **anonymous**, and matches nothing: not a named
+ dimension, not another anonymous one, and not itself as seen from a
+ second variable. There is nothing to match on, so claiming a match would
+ be invention.
+
+ Zarr's version 3 core specification makes ``dimension_names`` optional and
+ allows a null entry, so a conforming store can present this and it must
+ not be an error. No netCDF variable ever has one.
+ """
+
+ def __init__(self, name: str | None, size: int):
+ self.name = name
+ self.size = size
+
+ def __eq__(self, other) -> bool:
+ if not isinstance(other, _DimensionIdentity):
+ return NotImplemented
+ if self.name is None or other.name is None:
+ return self is other
+ return self.name == other.name and self.size == other.size
+
+ def __hash__(self) -> int:
+ if self.name is None:
+ return object.__hash__(self)
+ return hash((self.name, self.size))
+
+ def __repr__(self) -> str:
+ if self.name is None:
+ return f"<anonymous dimension of {self.size}>"
+ return f"<dimension {self.name!r} of {self.size}>"
+
+
+def _dimension_identities(cf_variable) -> tuple[_DimensionIdentity, ...]:
+ """Return one hashable identity per dimension `cf_variable` spans.
+
+ A function rather than a method, because the netCDF loading rules decide
+ which cube dimensions to attach a coordinate to, and have to decide it by
+ the rule that let the coordinate through :meth:`CFVariable.spans` in the
+ first place - see
+ :func:`iris.fileformats._nc_load_rules.helpers._common_data_dims`.
+
+ Relies on the :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+ invariant that ``dimensions`` and ``shape`` have the same length.
+ """
+ return tuple(
+ _DimensionIdentity(name, size)
+ for name, size in zip(cf_variable.dimensions, cf_variable.shape)
+ )
+
+
# NetCDF variable attributes handled by the netCDF4 module and
# therefore automatically classed as "used" attributes.
_CF_ATTRS_IGNORE = set(["_FillValue", "add_offset", "missing_value", "scale_factor"])
-#: Supported dimensionless vertical coordinate reference surface/phemomenon
-#: formula terms. Ref: [CF] Appendix D.
-reference_terms = dict(
- atmosphere_sigma_coordinate=["ps"],
- atmosphere_hybrid_sigma_pressure_coordinate=["ps"],
- atmosphere_hybrid_height_coordinate=["orog"],
- atmosphere_sleve_coordinate=["zsurf1", "zsurf2"],
- ocean_sigma_coordinate=["eta", "depth"],
- ocean_s_coordinate=["eta", "depth"],
- ocean_sigma_z_coordinate=["eta", "depth"],
- ocean_s_coordinate_g1=["eta", "depth"],
- ocean_s_coordinate_g2=["eta", "depth"],
-)
+# __getattr__ resolves a name by reading self.attributes, so these two names
+# cannot be resolved that way. Every other name reaching __getattr__ is a CF
+# attribute read from a file, "_Encoding" and "_FillValue" included.
+_GETATTR_RECURSION_GUARD = frozenset(["attributes", "cf_data"])
# NetCDF returns a different type for strings depending on Python version.
@@ -90,22 +161,42 @@ class CFVariable(metaclass=ABCMeta):
cf_identity: ClassVar[str | None] = None
def __init__(self, name, data):
- # Accessing the list of netCDF attributes is surprisingly slow.
- # Since it's used repeatedly, caching the list makes things
- # quite a bit faster.
- self._nc_attrs = data.ncattrs()
-
self.cf_name = name
"""NetCDF variable name."""
self.cf_data = data
- """NetCDF4 Variable data instance."""
+ """The variable's storage.
+
+ A :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`.
+ ``_deprecated_netcdf_member`` is the only netCDF4-specific route out
+ of this class; everything else goes through the format-agnostic
+ interface.
+ """
+
+ self.attributes = TrackedAttributes(
+ dict(data.attributes), ignored=_CF_ATTRS_IGNORE
+ )
+ """The variable's CF attributes, and a record of which have been read.
+
+ The only place CF attributes live. ``__getattr__`` forwards here, so
+ ``cf_var.units`` and ``cf_var.attributes["units"]`` perform the same
+ read and are recorded once. What was read decides what survives onto the
+ cube - see :func:`iris.fileformats.cf.loader._add_unused_attributes`.
+
+ Copied from ``data.attributes``, not a view onto it:
+ :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`'s
+ mapping writes through to the file, and a plain ``dict`` is what keeps
+ two instances of :class:`CFVariable` built over the same storage object
+ from sharing one mapping.
+ """
- """File source of the NetCDF content."""
try:
- self.filename = data.group().filepath()
+ location = data.location
except AttributeError:
- self.filename = "<unknown_filename>"
+ location = "<unknown_filename>"
+
+ self.filename = location
+ """File source of the NetCDF content."""
self.cf_group = None
"""Collection of CF-netCDF variables associated with this variable."""
@@ -115,8 +206,6 @@ def __init__(self, name, data):
self._to_be_promoted = False
- self.cf_attrs_reset()
-
@staticmethod
def _identify_common(variables, ignore, target):
if ignore is None:
@@ -142,7 +231,7 @@ def identify(self, variables, ignore=None, target=None, warn=True):
Parameters
----------
variables :
- Dictionary of netCDF4.Variable instance by variable name.
+ Dictionary of CFDatasetVariable instance by variable name.
ignore : optional
List of variable names to ignore.
target : optional
@@ -157,12 +246,26 @@ def identify(self, variables, ignore=None, target=None, warn=True):
"""
pass
+ def _is_scalar(self) -> bool:
+ """Whether this variable is scalar, in either spelling of scalar.
+
+ NetCDF gives a zero-dimensional variable no dimensions at all.
+ NCZarr gives it one, named ``_scalar_``. The two mean the same thing.
+
+ """
+ return not self.dimensions or self.dimensions == (_NCZARR_SCALAR_DIMENSION,)
+
def spans(self, cf_variable):
"""Determine dimensionality coverage.
Determine whether the dimensionality of this variable
is a subset of the specified target variable.
+ A dimension matches on the pair *(name, size)*, not on the bare
+ name: see :func:`_dimension_identities`. A name that is the same in
+ both variables but a different size links nothing, and a Zarr
+ store's anonymous dimensions never match anything, even each other.
+
Note that, by default scalar variables always span the
dimensionality of the target variable.
@@ -176,12 +279,13 @@ def spans(self, cf_variable):
bool
"""
- dimensions = tuple(self.dimensions)
- if dimensions == (_NCZARR_SCALAR_DIMENSION,):
+ # Scalar variables always span the target variable.
+ if self._is_scalar():
return True
- result = set(dimensions).issubset(cf_variable.dimensions)
- return result
+ source = _dimension_identities(self)
+ target = _dimension_identities(cf_variable)
+ return set(source).issubset(target)
def __eq__(self, other):
# CF variable names are unique.
@@ -195,16 +299,87 @@ def __hash__(self):
# CF variable names are unique.
return hash(self.cf_name)
+ @property
+ def dimensions(self) -> tuple:
+ """The names of the dimensions this variable spans, in order.
+
+ An entry is ``None`` where the store names no dimension - only a
+ Zarr store does this; no netCDF variable has an anonymous dimension.
+ """
+ return tuple(self.cf_data.dimensions)
+
+ @property
+ def shape(self) -> tuple:
+ """The variable's shape."""
+ return self.cf_data.shape
+
+ @property
+ def ndim(self) -> int:
+ """The number of dimensions this variable spans."""
+ return len(self.shape)
+
+ @property
+ def dtype(self):
+ """The variable's stored data type."""
+ return self.cf_data.dtype
+
+ @property
+ def size(self) -> int:
+ """The total number of elements in the variable."""
+ return self.cf_data.size
+
def __getattr__(self, name):
- # Accessing netCDF attributes is surprisingly slow. Since
- # they're often read repeatedly, caching the values makes things
- # quite a bit faster.
- if name in self._nc_attrs:
- self._cf_attrs.add(name)
- value = getattr(self.cf_data, name)
- setattr(self, name, value)
+ """Return the named CF attribute, as read from the file.
+
+ .. deprecated:: 3.17
+ Read CF attributes from :attr:`attributes` instead. Every CF
+ attribute resolved through this method warns, and the method is
+ removed one release cycle later under NEP 29. :attr:`attributes`
+ is also unambiguous: a file carrying an attribute named after a
+ class member cannot reach it through this method at all.
+
+ CF attribute names are data read from a file, not API, so they cannot
+ be declared as properties. This method resolves only against
+ :attr:`attributes`; everything structural is a declared property
+ above. Reading through this method records the attribute as used,
+ exactly as ``cf_var.attributes[name]`` does.
+
+ """
+ if name.startswith("__") or name in _GETATTR_RECURSION_GUARD:
+ # Dunder probes - copy, pickle, numpy protocols - must not be
+ # answered from file data, and the two members this method reads
+ # must not be resolved by this method.
+ raise AttributeError(name)
+
+ if name in self.attributes:
+ # Read before warning, as _deprecated_netcdf_member does.
+ value = self.attributes[name]
+ # stacklevel=3 reports the caller's own "cf_var.units" line:
+ # warn_deprecated's frame is 1 and this method's is 2.
+ warn_deprecated(
+ f"Reading the CF attribute {name!r} as an attribute of a "
+ "CFVariable is deprecated and will be removed in a future "
+ f"release. Use cf_var.attributes[{name!r}] instead.",
+ stacklevel=3,
+ )
+ else:
+ # Not in an except block: an AttributeError raised from here must
+ # not arrive chained to a KeyError the caller never caused.
+ value = self._deprecated_netcdf_member(name)
return value
+ def _deprecated_netcdf_member(self, name):
+ """Return a netCDF4 member of the backing variable, as this class used to.
+
+ The one-cycle compatibility route for code that reached netCDF4 API
+ through a CFVariable. Records nothing: this is not a CF attribute.
+ The storage object decides what this means, and warns - see
+ :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`'s
+ ``deprecated_netcdf_member``.
+
+ """
+ return self.cf_data.deprecated_netcdf_member(name)
+
def __getitem__(self, key):
return self.cf_data.__getitem__(key)
@@ -218,31 +393,37 @@ def __repr__(self):
self.cf_data,
)
+ # Every cf_attrs_* reader below takes its values from
+ # ``self.attributes.untracked``: a report on what was read must not itself
+ # count as reading. Each binds that view to a local first, because reading
+ # the property copies the whole mapping.
+
def cf_attrs(self):
"""Return a list of all attribute name and value pairs of the CF-netCDF variable."""
- return tuple((attr, self.getncattr(attr)) for attr in sorted(self._nc_attrs))
+ attributes = self.attributes.untracked
+ return tuple((name, attributes[name]) for name in sorted(attributes))
def cf_attrs_ignored(self):
"""Return a list of all ignored attribute name and value pairs of the CF-netCDF variable."""
- return tuple(
- (attr, self.getncattr(attr))
- for attr in sorted(set(self._nc_attrs) & _CF_ATTRS_IGNORE)
- )
+ attributes = self.attributes.untracked
+ names = set(attributes) & _CF_ATTRS_IGNORE
+ return tuple((name, attributes[name]) for name in sorted(names))
def cf_attrs_used(self):
"""Return a list of all accessed attribute name and value pairs of the CF-netCDF variable."""
- return tuple((attr, self.getncattr(attr)) for attr in sorted(self._cf_attrs))
+ attributes = self.attributes.untracked
+ return tuple((name, attributes[name]) for name in sorted(self.attributes.read))
def cf_attrs_unused(self):
"""Return a list of all non-accessed attribute name and value pairs of the CF-netCDF variable."""
+ attributes = self.attributes.untracked
return tuple(
- (attr, self.getncattr(attr))
- for attr in sorted(set(self._nc_attrs) - self._cf_attrs)
+ (name, attributes[name]) for name in sorted(self.attributes.unread)
)
def cf_attrs_reset(self):
"""Reset the history of accessed attribute names of the CF-netCDF variable."""
- self._cf_attrs = set([item[0] for item in self.cf_attrs_ignored()])
+ self.attributes.reset()
def add_formula_term(self, root, term):
"""Register the participation of this CF-netCDF variable in a CF-netCDF formula term.
@@ -296,7 +477,7 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
# Identify all CF ancillary data variables.
for nc_var_name, nc_var in target.items():
# Check for ancillary data variable references.
- nc_var_att = getattr(nc_var, cls.cf_identity, None)
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
if nc_var_att is not None:
for name in nc_var_att.split():
@@ -345,7 +526,7 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
# Identify all CF auxiliary coordinate variables.
for nc_var_name, nc_var in target.items():
# Check for auxiliary coordinate variable references.
- nc_var_att = getattr(nc_var, cls.cf_identity, None)
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
if nc_var_att is not None:
for name in nc_var_att.split():
@@ -394,7 +575,7 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
# Identify all CF boundary variables.
for nc_var_name, nc_var in target.items():
# Check for a boundary variable reference.
- nc_var_att = getattr(nc_var, cls.cf_identity, None)
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
if nc_var_att is not None:
name = nc_var_att.strip()
@@ -433,9 +614,9 @@ def spans(self, cf_variable):
"""
# Scalar variables always span the target variable.
result = True
- if self.dimensions:
- source = self.dimensions
- target = cf_variable.dimensions
+ if not self._is_scalar():
+ source = _dimension_identities(self)
+ target = _dimension_identities(cf_variable)
# Ignore the bounds extent dimension.
result = set(source[:-1]).issubset(target) or set(source[1:]).issubset(
target
@@ -470,7 +651,7 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
# Identify all CF climatology variables.
for nc_var_name, nc_var in target.items():
# Check for a climatology variable reference.
- nc_var_att = getattr(nc_var, cls.cf_identity, None)
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
if nc_var_att is not None:
name = nc_var_att.strip()
@@ -509,9 +690,9 @@ def spans(self, cf_variable):
"""
# Scalar variables always span the target variable.
result = True
- if self.dimensions:
- source = self.dimensions
- target = cf_variable.dimensions
+ if not self._is_scalar():
+ source = _dimension_identities(self)
+ target = _dimension_identities(cf_variable)
# Ignore the climatology extent dimension.
result = set(source[:-1]).issubset(target) or set(source[1:]).issubset(
target
@@ -606,7 +787,7 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
# Identify all CF formula terms variables.
for nc_var_name, nc_var in target.items():
# Check for formula terms variable references.
- nc_var_att = getattr(nc_var, cls.cf_identity, None)
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
if nc_var_att is not None:
for match_item in _CF_PARSE.finditer(nc_var_att):
@@ -679,7 +860,7 @@ def identify(
# Identify all grid mapping variables.
for nc_var_name, nc_var in target.items():
# Check for a grid mapping variable reference.
- nc_var_att = getattr(nc_var, cls.cf_identity, None)
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
if nc_var_att is not None:
# All `grid_mapping` attributes will already have been parsed prior
@@ -763,7 +944,7 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
# Identify all CF label variables.
for nc_var_name, nc_var in target.items():
# Check for label variable references.
- nc_var_att = getattr(nc_var, cls.cf_identity, None)
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
if nc_var_att is not None:
for name in nc_var_att.split():
@@ -832,9 +1013,9 @@ def spans(self, cf_variable):
"""
# Scalar variables always span the target variable.
result = True
- if self.dimensions:
- source = self.dimensions
- target = cf_variable.dimensions
+ if not self._is_scalar():
+ source = _dimension_identities(self)
+ target = _dimension_identities(cf_variable)
# Ignore label string length dimension.
result = set(source[:-1]).issubset(target) or set(source[1:]).issubset(
target
@@ -866,7 +1047,7 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
# Identify all CF measure variables.
for nc_var_name, nc_var in target.items():
# Check for measure variable references.
- nc_var_att = getattr(nc_var, cls.cf_identity, None)
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
if nc_var_att is not None:
for match_item in _CF_PARSE.finditer(nc_var_att):
@@ -930,7 +1111,7 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
# Check for connectivity variable references, iterating through
# the valid cf roles.
for identity in cls.cf_identities:
- nc_var_att = getattr(nc_var, identity, None)
+ nc_var_att = nc_var.attributes.get(identity)
if nc_var_att is not None:
# UGRID only allows for one of each connectivity cf role.
@@ -1005,7 +1186,7 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
for nc_var_name, nc_var in target.items():
# Check for UGRID auxiliary coordinate variable references.
for identity in cls.cf_identities:
- nc_var_att = getattr(nc_var, identity, None)
+ nc_var_att = nc_var.attributes.get(identity)
if nc_var_att is not None:
for name in nc_var_att.split():
@@ -1079,11 +1260,11 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
# SPECIAL BEHAVIOUR FOR MESH VARIABLES.
# We are looking for all mesh variables. Check if THIS variable
# is a mesh using its own attributes.
- if getattr(nc_var, "cf_role", "") == "mesh_topology":
+ if nc_var.attributes.get("cf_role", "") == "mesh_topology":
result[nc_var_name] = CFUGridMeshVariable(nc_var_name, nc_var)
# Check for mesh variable references.
- nc_var_att = getattr(nc_var, cls.cf_identity, None)
+ nc_var_att = nc_var.attributes.get(cls.cf_identity)
if nc_var_att is not None:
# UGRID only allows for 1 mesh per variable.
@@ -1111,611 +1292,3 @@ def identify(cls, variables, ignore=None, target=None, warn=True):
warnings.warn(message, category=IrisCfLabelVarWarning)
return result
-
-
-################################################################################
-class CFGroup(MutableMapping):
- """Collection of 'NetCDF CF Metadata Conventions variables and netCDF global attributes.
-
- Represents a collection of 'NetCDF Climate and Forecast (CF) Metadata
- Conventions' variables and netCDF global attributes.
-
- """
-
- def __init__(self):
- #: Collection of CF-netCDF variables
- self._cf_variables = {}
- #: Collection of netCDF global attributes
- self.global_attributes = {}
- #: Collection of CF-netCDF variables promoted to a CFDataVariable.
- self.promoted = {}
-
- def _cf_getter(self, cls):
- # Generate dictionary with dictionary comprehension.
- return {
- cf_name: cf_var
- for cf_name, cf_var in self._cf_variables.items()
- if isinstance(cf_var, cls)
- }
-
- @property
- def ancillary_variables(self):
- """Collection of CF-netCDF ancillary variables."""
- return self._cf_getter(CFAncillaryDataVariable)
-
- @property
- def auxiliary_coordinates(self):
- """Collection of CF-netCDF auxiliary coordinate variables."""
- return self._cf_getter(CFAuxiliaryCoordinateVariable)
-
- @property
- def bounds(self):
- """Collection of CF-netCDF boundary variables."""
- return self._cf_getter(CFBoundaryVariable)
-
- @property
- def climatology(self):
- """Collection of CF-netCDF climatology variables."""
- return self._cf_getter(CFClimatologyVariable)
-
- @property
- def coordinates(self):
- """Collection of CF-netCDF coordinate variables."""
- return self._cf_getter(CFCoordinateVariable)
-
- @property
- def data_variables(self):
- """Collection of CF-netCDF data pay-load variables."""
- return self._cf_getter(CFDataVariable)
-
- @property
- def formula_terms(self):
- """Collection of CF-netCDF variables that participate in a CF-netCDF formula term."""
- return {
- cf_name: cf_var
- for cf_name, cf_var in self._cf_variables.items()
- if cf_var.has_formula_terms()
- }
-
- @property
- def grid_mappings(self):
- """Collection of CF-netCDF grid mapping variables."""
- return self._cf_getter(CFGridMappingVariable)
-
- @property
- def labels(self):
- """Collection of CF-netCDF label variables."""
- return self._cf_getter(CFLabelVariable)
-
- @property
- def cell_measures(self):
- """Collection of CF-netCDF measure variables."""
- return self._cf_getter(CFMeasureVariable)
-
- @property
- def non_data_variable_names(self):
- """:class:`set` names of the CF-netCDF variables that are not the data pay-load."""
- non_data_variables = (
- self.ancillary_variables,
- self.auxiliary_coordinates,
- self.bounds,
- self.climatology,
- self.coordinates,
- self.grid_mappings,
- self.labels,
- self.cell_measures,
- self.connectivities,
- self.ugrid_coords,
- self.meshes,
- )
- result = set()
- for variable in non_data_variables:
- result |= set(variable)
- return result
-
- @property
- def connectivities(self):
- """Collection of CF-UGRID connectivity variables."""
- return self._cf_getter(CFUGridConnectivityVariable)
-
- @property
- def ugrid_coords(self):
- """Collection of CF-UGRID-relevant auxiliary coordinate variables."""
- return self._cf_getter(CFUGridAuxiliaryCoordinateVariable)
-
- @property
- def meshes(self):
- """Collection of CF-UGRID mesh variables."""
- return self._cf_getter(CFUGridMeshVariable)
-
- def keys(self):
- """Return the names of all the CF-netCDF variables in the group."""
- return self._cf_variables.keys()
-
- def __len__(self):
- return len(self._cf_variables)
-
- def __iter__(self):
- for item in self._cf_variables:
- yield item
-
- def __setitem__(self, name, variable):
- if not isinstance(variable, CFVariable):
- raise TypeError(
- "Attempted to add an invalid CF-netCDF variable to the %s"
- % self.__class__.__name__
- )
-
- if name != variable.cf_name:
- raise ValueError(
- "Mismatch between key name %r and CF-netCDF variable name %r"
- % (str(name), variable.cf_name)
- )
-
- self._cf_variables[name] = variable
-
- def __getitem__(self, name):
- if name not in self._cf_variables:
- raise KeyError("Cannot get unknown CF-netCDF variable name %r" % str(name))
-
- return self._cf_variables[name]
-
- def __delitem__(self, name):
- if name not in self._cf_variables:
- raise KeyError(
- "Cannot delete unknown CF-netcdf variable name %r" % str(name)
- )
-
- del self._cf_variables[name]
-
- def __repr__(self):
- result = []
- result.append("variables:%d" % len(self._cf_variables))
- result.append("global_attributes:%d" % len(self.global_attributes))
- result.append("promoted:%d" % len(self.promoted))
-
- return "<%s of %s>" % (self.__class__.__name__, ", ".join(result))
-
-
-################################################################################
-class CFReader:
- """Allows the contents of a netCDF file to be interpreted.
-
- This class allows the contents of a netCDF file to be interpreted according
- to the 'NetCDF Climate and Forecast (CF) Metadata Conventions'.
-
- """
-
- # All CF variable types EXCEPT for the "special cases" of
- # CFDataVariable, CFCoordinateVariable and _CFFormulaTermsVariable.
- _variable_types = (
- CFAncillaryDataVariable,
- CFAuxiliaryCoordinateVariable,
- CFBoundaryVariable,
- CFClimatologyVariable,
- CFGridMappingVariable,
- CFLabelVariable,
- CFMeasureVariable,
- CFUGridConnectivityVariable,
- CFUGridAuxiliaryCoordinateVariable,
- CFUGridMeshVariable,
- )
-
- CFGroup = CFGroup
-
- def __init__(self, file_source, warn=False, monotonic=False):
- # Ensure safe operation for destructor, should init fail.
- self._own_file = False
- if isinstance(file_source, str):
- # Create from filepath : open it + own it (=close when we die).
- if not urlparse(file_source).scheme:
- self._filename = Path(file_source).expanduser()
- else:
- self._filename = file_source
-
- if _bytecoding_datasets.DECODE_TO_STRINGS_ON_READ:
- ds_type = _bytecoding_datasets.EncodedDataset
- else:
- ds_type = _thread_safe_nc.DatasetWrapper
-
- self._dataset = ds_type(self._filename, mode="r")
- self._own_file = True
- else:
- # We have been passed an open dataset.
- # We use it but don't own it (don't close it).
- self._dataset = file_source
- self._filename = self._dataset.filepath()
-
- #: Collection of CF-netCDF variables associated with this netCDF file
- self.cf_group = self.CFGroup()
-
- # Result of parsing "grid_mapping" attribute; mapping of coordinate_system => coordinates
- self._coord_system_mappings = {}
-
- # Issue load optimisation warning.
- if warn and self._dataset.file_format in [
- "NETCDF3_CLASSIC",
- "NETCDF3_64BIT",
- ]:
- warnings.warn(
- "Optimise CF-netCDF loading by converting data from NetCDF3 "
- 'to NetCDF4 file format using the "nccopy" command.',
- category=iris.warnings.IrisLoadWarning,
- )
-
- self._check_monotonic = monotonic
-
- self._with_ugrid = True
- if not self._has_meshes():
- self._trim_ugrid_variable_types()
- self._with_ugrid = False
-
- # Read the variables in the dataset only once to reduce runtime.
- ds = self._dataset
- # Turn off *any* automatic decoding in the underlying netCDF4 dataset.
- ds.set_auto_chartostring(False)
- variables = self._dataset.variables
- self._translate(variables)
- self._build_cf_groups(variables)
- self._reset(variables)
-
- def __enter__(self):
- # Enable use as a context manager
- # N.B. this **guarantees* closure of the file, when the context is exited.
- # Note: ideally, the class would not do so much work in the __init__ call, and
- # would do all that here, after acquiring necessary permissions/locks.
- # But for legacy reasons, we can't do that. So **effectively**, the context
- # (in terms of access control) already started, when we created the object.
- return self
-
- def __exit__(self, exc_type, exc_value, traceback):
- # When used as a context-manager, **always** close the file on exit.
- self._close()
-
- def _has_meshes(self):
- result = False
- for variable in self._dataset.variables.values():
- if hasattr(variable, "mesh") or hasattr(variable, "node_coordinates"):
- result = True
- break
- return result
-
- def _trim_ugrid_variable_types(self):
- self._variable_types = (
- CFAncillaryDataVariable,
- CFAuxiliaryCoordinateVariable,
- CFBoundaryVariable,
- CFClimatologyVariable,
- CFGridMappingVariable,
- CFLabelVariable,
- CFMeasureVariable,
- )
-
- @property
- def filename(self):
- """The file that the CFReader is reading."""
- return self._filename
-
- def __repr__(self):
- return "%s(%r)" % (self.__class__.__name__, self._filename)
-
- def _translate(self, variables):
- """Classify the netCDF variables into CF-netCDF variables."""
- netcdf_variable_names = list(variables.keys())
-
- # Parse all instances of "grid_mapping" attributes and store in CFReader
- # This avoids re-parsing the grid_mappings each time they are needed.
- for nc_var in variables.values():
- if grid_mapping_attr := getattr(nc_var, "grid_mapping", None):
- try:
- cs_mappings = hh._parse_extended_grid_mapping(grid_mapping_attr)
- self._coord_system_mappings[nc_var.name] = cs_mappings
- except iris.exceptions.CFParseError as e:
- msg = f"Error parsing `grid_mapping` attribute for {nc_var.name}: {str(e)}"
- warnings.warn(msg, category=iris.warnings.IrisCfWarning)
- continue
-
- # Identify all CF coordinate variables first. This must be done
- # first as, by CF convention, the definition of a CF auxiliary
- # coordinate variable may include a scalar CF coordinate variable,
- # whereas we want these two types of variables to be mutually exclusive.
- coords = CFCoordinateVariable.identify(
- variables, monotonic=self._check_monotonic
- )
- self.cf_group.update(coords)
- coordinate_names = list(self.cf_group.coordinates.keys())
-
- # Identify all CF variables EXCEPT for the "special cases".
- for variable_type in self._variable_types:
- # Prevent grid mapping variables being mis-identified as CF coordinate variables.
- ignore = (
- None
- if issubclass(variable_type, CFGridMappingVariable)
- else coordinate_names
- )
- kwargs = (
- {"coord_system_mappings": self._coord_system_mappings}
- if issubclass(variable_type, CFGridMappingVariable)
- else {}
- )
-
- self.cf_group.update(
- variable_type.identify(variables, ignore=ignore, **kwargs)
- )
-
- # Identify global netCDF attributes.
- attr_dict = {
- attr_name: _getncattr(self._dataset, attr_name, "")
- for attr_name in self._dataset.ncattrs()
- }
- self.cf_group.global_attributes.update(attr_dict)
-
- # Identify and register all CF formula terms.
- formula_terms = _CFFormulaTermsVariable.identify(variables)
-
- if iris.FUTURE.derived_bounds:
- # Keep track of all the root vars so we can unpick invalid bounds vars
- all_roots = set()
-
- # cf_var = CFFormulaTermsVariable (loops through everything that appears in formula terms)
- for cf_var in formula_terms.values():
- # Example of a formula term:
- # Suppose in the file eta:formula_terms contains "a: var_A"
- # cf_var = var_A, cf_root = eta and cf_term = 'a'. cf_var.cf_terms_by_root = {eta: 'a'}
- for cf_root, cf_term in cf_var.cf_terms_by_root.items():
- if iris.FUTURE.derived_bounds:
- # For the "newstyle" derived-bounds implementation, find vars which appear in derived bounds terms
- # and turn them into bounds vars (though they don't appear in a "bounds" attribute)
-
- # Adds each root only once
- all_roots.add(cf_root)
-
- # cf_root_coord = CFCoordinateVariable or CFAuxiliaryCoordinateVariable of the coordinate relating to the root
- cf_root_coord = self.cf_group.coordinates.get(cf_root)
- if cf_root_coord is None:
- cf_root_coord = self.cf_group.auxiliary_coordinates.get(cf_root)
-
- root_bounds_name = getattr(cf_root_coord, "bounds", None)
- # N.B. cf_root_coord may here be None, if the root var was not a
- # coord - that is ok, it will not have a 'bounds', we will skip it.
- if root_bounds_name in self.cf_group:
- root_bounds_var = self.cf_group.get(root_bounds_name)
- if not hasattr(root_bounds_var, "formula_terms"):
- # this is an invalid root bounds, according to CF, and therefore should be promoted into a cube
- root_bounds_var._to_be_promoted = True
- else:
- # Found a valid *root* bounds variable : search for a corresponding *term* bounds variable,
- term_bounds_vars = [
- # loop through all formula terms and add them if they have a cf_term_by_root
- # where (bounds of cf_root): cf_term (same as before)
- f
- for f in formula_terms.values()
- if f.cf_terms_by_root.get(root_bounds_name) == cf_term
- ]
- if len(term_bounds_vars) == 1:
- (term_bounds_var,) = term_bounds_vars
- # N.B. bounds==main-var is valid CF for *no* bounds
- if term_bounds_var != cf_var:
- cf_var.bounds = term_bounds_var.cf_name
- new_var = CFBoundaryVariable(
- term_bounds_var.cf_name, term_bounds_var.cf_data
- )
- new_var.add_formula_term(root_bounds_name, cf_term)
- # "Reclassify" this var as a bounds variable
- self.cf_group[term_bounds_var.cf_name] = new_var
-
- if cf_root not in self.cf_group.bounds:
- # This records all formula terms in the main cf_group that were previously only stored in the formula_terms dictionary.
- cf_name = cf_var.cf_name
- if cf_name not in self.cf_group:
- # If the formula term variable is not already in the group, add it as a coordinate.
- new_var = CFAuxiliaryCoordinateVariable(cf_name, cf_var.cf_data)
- if iris.FUTURE.derived_bounds and hasattr(cf_var, "bounds"):
- # Copy "old-style" derived bounds link
- new_var.bounds = cf_var.bounds
- self.cf_group[cf_name] = new_var
-
- self.cf_group[cf_name].add_formula_term(cf_root, cf_term)
-
- if iris.FUTURE.derived_bounds:
- for cf_root in all_roots:
- # Invalidate "broken" bounds connections
- root_var = self.cf_group[cf_root]
- if getattr(root_var, "formula_terms", None) and getattr(
- root_var, "bounds", None
- ):
- root_bounds_var = self.cf_group.get(root_var.bounds)
- if not getattr(root_bounds_var, "formula_terms", None):
- # This means it is *not* a valid bounds var, according to CF, and so therefore we are
- # invalidating the bounds.
- root_var.bounds = None
-
- # Determine the CF data variables.
- data_variable_names = (
- set(netcdf_variable_names) - self.cf_group.non_data_variable_names
- )
-
- for name in data_variable_names:
- self.cf_group[name] = CFDataVariable(name, variables[name])
-
- def _build_cf_groups(self, variables):
- """Build the first order relationships between CF-netCDF variables."""
-
- def _build(cf_variable):
- is_mesh_var = isinstance(cf_variable, CFUGridMeshVariable)
- ugrid_coord_names = []
- ugrid_coords = getattr(self.cf_group, "ugrid_coords", None)
- if ugrid_coords is not None:
- ugrid_coord_names = list(ugrid_coords.keys())
-
- coordinate_names = list(self.cf_group.coordinates.keys())
- cf_group = self.CFGroup()
-
- def _span_check(
- var_name: str, via_formula_terms: Optional[str] = None
- ) -> None:
- """Sanity check dimensionality."""
- var = self.cf_group[var_name]
- # No span check is necessary if variable is attached to a mesh.
- if (is_mesh_var or var.spans(cf_variable)) and not var._to_be_promoted:
- cf_group[var_name] = var
- else:
- # Register the ignored variable.
- # N.B. 'ignored' variable from enclosing scope.
- ignored.add(var_name)
-
- text_formula = text_via = ""
- if via_formula_terms:
- text_formula = " formula terms"
- text_via = f" via variable {via_formula_terms}"
-
- message = (
- f"Ignoring{text_formula} variable {var_name} "
- f"referenced by variable {cf_variable.cf_name}"
- f"{text_via}: Dimensions {var.dimensions} do not span "
- f"{cf_variable.dimensions}"
- )
- warnings.warn(
- message,
- category=iris.warnings.IrisCfNonSpanningVarWarning,
- )
-
- # Build CF variable relationships.
- for variable_type in self._variable_types:
- ignore = []
- kwargs = {}
- # Avoid UGridAuxiliaryCoordinateVariables also being
- # processed as CFAuxiliaryCoordinateVariables.
- if not is_mesh_var:
- ignore += ugrid_coord_names
- # Prevent grid mapping variables being mis-identified as CF coordinate variables.
- if issubclass(variable_type, CFGridMappingVariable):
- # pass parsed grid_mappings to CFGridMappingVariable types
- kwargs.update(
- {"coord_system_mappings": self._coord_system_mappings}
- )
- else:
- ignore += coordinate_names
-
- match = variable_type.identify(
- variables,
- ignore=ignore,
- target=cf_variable.cf_name,
- warn=False,
- **kwargs,
- )
- # Sanity check dimensionality coverage.
- for cf_name in match:
- _span_check(cf_name)
-
- if iris.FUTURE.derived_bounds:
- # Include bounds of every variable, within cf_group attached to the variable.
- if hasattr(cf_variable, "bounds"):
- if cf_variable.bounds not in cf_group:
- bounds_var = self.cf_group.get(cf_variable.bounds)
- if bounds_var:
- # TODO: warning if span fails
- if bounds_var.spans(cf_variable):
- cf_group[cf_variable.bounds] = bounds_var
-
- # Build CF data variable relationships.
- if isinstance(cf_variable, CFDataVariable):
- # Add global netCDF attributes.
- cf_group.global_attributes.update(self.cf_group.global_attributes)
- # Add appropriate "dimensioned" CF coordinate variables.
- cf_group.update(
- {
- cf_name: self.cf_group[cf_name]
- for cf_name in cf_variable.dimensions
- if cf_name in self.cf_group.coordinates
- }
- )
- # Add appropriate "dimensionless" CF coordinate variables.
- coordinates_attr = getattr(cf_variable, "coordinates", "")
- cf_group.update(
- {
- cf_name: self.cf_group[cf_name]
- for cf_name in coordinates_attr.split()
- if cf_name in self.cf_group.coordinates
- }
- )
- # Add appropriate formula terms.
- for cf_var in self.cf_group.formula_terms.values():
- for cf_root in cf_var.cf_terms_by_root:
- if cf_root in cf_group and cf_var.cf_name not in cf_group:
- _span_check(cf_var.cf_name, cf_root)
-
- # Add the CF group to the variable.
- cf_variable.cf_group = cf_group
-
- # Ignored variables are those that cannot be attached to a
- # data variable as the dimensionality of that variable is not
- # a subset of the dimensionality of the data variable.
- ignored = set()
-
- for cf_variable in self.cf_group.values():
- _build(cf_variable)
-
- # Determine whether there are any formula terms that
- # may be promoted to a CFDataVariable and restrict promotion to only
- # those formula terms that are reference surface/phenomenon.
- for cf_var in self.cf_group.formula_terms.values():
- if iris.FUTURE.derived_bounds:
- if self.cf_group[cf_var.cf_name] is CFBoundaryVariable:
- continue
- for cf_root, cf_term in cf_var.cf_terms_by_root.items():
- cf_root_var = self.cf_group[cf_root]
- if iris.FUTURE.derived_bounds:
- if not hasattr(cf_root_var, "standard_name"):
- continue
- name = cf_root_var.standard_name or cf_root_var.long_name
- terms = reference_terms.get(name, [])
- if isinstance(terms, str) or not isinstance(terms, Iterable):
- terms = [terms]
- cf_var_name = cf_var.cf_name
- if cf_term in terms and cf_var_name not in self.cf_group.promoted:
- data_var = CFDataVariable(cf_var_name, cf_var.cf_data)
- self.cf_group.promoted[cf_var_name] = data_var
- _build(data_var)
- break
-
- # Promote any ignored variables.
- promoted = set()
- not_promoted = ignored.difference(promoted)
- while not_promoted:
- cf_name = not_promoted.pop()
- if (
- cf_name not in self.cf_group.data_variables
- and cf_name not in self.cf_group.promoted
- ):
- data_var = CFDataVariable(cf_name, self.cf_group[cf_name].cf_data)
- self.cf_group.promoted[cf_name] = data_var
- _build(data_var)
- # Determine whether there are still any ignored variables
- # yet to be promoted.
- promoted.add(cf_name)
- not_promoted = ignored.difference(promoted)
-
- def _reset(self, variables):
- """Reset the attribute touch history of each variable."""
- for nc_var_name in variables.keys():
- self.cf_group[nc_var_name].cf_attrs_reset()
-
- def _close(self):
- # Explicitly close dataset to prevent file remaining open.
- if self._own_file and self._dataset is not None:
- self._dataset.close()
- self._dataset = None
-
- def __del__(self):
- # Be sure to close dataset when CFReader is destroyed / garbage-collected.
- self._close()
-
-
-def _getncattr(dataset, attr, default=None):
- """Wrap `netCDF4.Dataset.getncattr` to make it behave more like `getattr`."""
- try:
- value = dataset.getncattr(attr)
- except AttributeError:
- value = default
- return value
diff --git a/lib/iris/fileformats/cf/dataset.py b/lib/iris/fileformats/cf/dataset.py
new file mode 100644
index 0000000000..2982708a2e
--- /dev/null
+++ b/lib/iris/fileformats/cf/dataset.py
@@ -0,0 +1,391 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The format-agnostic description of a CF-conforming array store.
+
+.. z_reference:: iris.fileformats.cf.dataset
+ :tags: topic_load_save
+
+ API reference
+
+:class:`CFDataset` and :class:`CFDatasetVariable` are what the rest of the CF
+layer is written against. They are deliberately small: only what the
+:class:`~iris.fileformats.cf.CFVariable` classes, the CF loader and the CF
+saver actually need, with one implementation per storage format.
+
+These classes are more restrictive than a netCDF variable about what can be
+reached through attribute syntax. netCDF presents ``units``, a CF attribute
+read from the file, and ``dimensions``, a property of the storage, the same
+way, and a caller cannot tell which is which. Here the property ``attributes``
+is the only route to a CF attribute, and every storage property is a named,
+typed member.
+
+The property ``attributes`` holds an open-ended set of data keys read from a
+file, with no schema to enumerate, which is why :class:`TrackedAttributes`
+exists and why :meth:`~iris.fileformats.cf.CFVariable.__getattr__` forwards to
+it. Tracking is the load-bearing part: Iris decides which file attributes
+survive onto a loaded cube by asking which ones the loading rules did *not*
+read, so a read that goes unrecorded silently leaks a CF-reserved attribute
+onto the cube.
+
+See sections 4.2, 4.3 and 4.5 of
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+from abc import ABC, abstractmethod
+from collections.abc import (
+ Callable,
+ ItemsView,
+ Iterable,
+ Iterator,
+ KeysView,
+ Mapping,
+ MutableMapping,
+ ValuesView,
+)
+from types import MappingProxyType
+from typing import Any
+
+import numpy as np
+
+
+class TrackedAttributes(MutableMapping):
+ """A variable's attributes, recording which of them have been looked up.
+
+ Single-key lookups record: :meth:`__getitem__`, :meth:`get` and
+ :meth:`__contains__`. Bulk access does not: iteration, :meth:`keys`,
+ :meth:`values`, :meth:`items` and :func:`len` leave the record alone.
+ :attr:`untracked` reads a single key without recording it.
+
+ """
+
+ def __init__(self, source: MutableMapping, *, ignored: Iterable[str] = ()):
+ """Wrap ``source``, treating any ``ignored`` name it holds as already read."""
+ self._source = source
+ self._ignored = frozenset(ignored)
+ self._read: set[str] = set()
+ self.reset()
+
+ def __getitem__(self, key: str) -> Any:
+ """Return ``key``'s value, recording it as read."""
+ # Index first: a KeyError must escape without recording anything.
+ value = self._source[key]
+ self._read.add(key)
+ return value
+
+ def __contains__(self, key: object) -> bool:
+ """Return whether ``key`` is present, recording a hit as read."""
+ result = key in self._source
+ if result:
+ self._read.add(key) # type: ignore[arg-type]
+ return result
+
+ def __setitem__(self, key: str, value: Any) -> None:
+ """Set ``key``'s value, recording nothing."""
+ self._source[key] = value
+
+ def __delitem__(self, key: str) -> None:
+ """Remove ``key``, and any record of it having been read."""
+ del self._source[key]
+ self._read.discard(key)
+
+ def __iter__(self) -> Iterator[str]:
+ """Return an iterator over the attribute names, recording nothing."""
+ return iter(self._source)
+
+ def __len__(self) -> int:
+ """Return the number of attributes, recording nothing."""
+ return len(self._source)
+
+ def keys(self) -> KeysView:
+ """Return a view of the attribute names, recording nothing."""
+ return self._source.keys()
+
+ def values(self) -> ValuesView:
+ """Return a view of the attribute values, recording nothing."""
+ return self._source.values()
+
+ def items(self) -> ItemsView:
+ """Return a view of the attribute pairs, recording nothing."""
+ return self._source.items()
+
+ def __repr__(self) -> str:
+ """Return a string representation, recording nothing."""
+ return f"{self.__class__.__name__}({dict(self._source)!r})"
+
+ @property
+ def untracked(self) -> Mapping:
+ """A read-only view of the same attributes that records nothing."""
+ return MappingProxyType(dict(self._source))
+
+ @property
+ def read(self) -> frozenset:
+ """The names looked up since construction or the last :meth:`reset`."""
+ return frozenset(self._read)
+
+ @property
+ def unread(self) -> frozenset:
+ """The names present but not looked up since the last :meth:`reset`."""
+ return frozenset(self._source) - self._read
+
+ def reset(self) -> None:
+ """Forget every recorded lookup, re-seeding from the ignored names."""
+ self._read = set(self._ignored) & set(self._source)
+
+
+class CFDatasetVariable(ABC):
+ """One named array in a CF-conforming dataset."""
+
+ @property
+ @abstractmethod
+ def name(self) -> str:
+ """The variable's name within its dataset."""
+
+ @property
+ @abstractmethod
+ def location(self) -> str:
+ """The path or URL of the dataset holding this variable."""
+
+ @property
+ @abstractmethod
+ def format(self) -> str:
+ """The concrete format of the dataset holding this variable.
+
+ Repeated here from the dataset so that a variable can answer for
+ itself, as :attr:`location` is. For messages and diagnostics only:
+ generic code asks for a capability, never for a format.
+
+ """
+
+ @property
+ @abstractmethod
+ def dimensions(self) -> tuple:
+ """The names of the dimensions this variable spans, in order."""
+
+ @property
+ @abstractmethod
+ def shape(self) -> tuple:
+ """The variable's shape."""
+
+ @property
+ @abstractmethod
+ def dtype(self) -> np.dtype:
+ """A :class:`numpy.dtype`, or :class:`str` for a variable-length string."""
+
+ @property
+ @abstractmethod
+ def size(self) -> int:
+ """The total number of elements in the variable."""
+
+ @property
+ @abstractmethod
+ def fill_value(self) -> Any:
+ """The store's own fill value, or ``None`` if it has none."""
+
+ @property
+ @abstractmethod
+ def chunking(self) -> tuple | None:
+ """The variable's storage chunk shape, or ``None`` when unchunked."""
+
+ @property
+ @abstractmethod
+ def storage_settings(self) -> Mapping:
+ """The storage settings this variable holds, in the backend's spelling.
+
+ ``zlib``, ``complevel``, ``chunksizes`` for netCDF; ``chunks``,
+ ``shards``, ``codecs``, ``zarr_format`` for Zarr. Readable, never
+ interpreted: no generic code may branch on a key. The typed,
+ guaranteed-present member is :attr:`chunking`.
+
+ """
+
+ @property
+ @abstractmethod
+ def attributes(self) -> MutableMapping:
+ """The variable's CF attributes, materialised once at construction."""
+
+ @abstractmethod
+ def __getitem__(self, keys) -> np.ndarray:
+ """Return the indexed portion of the variable's data."""
+
+ @abstractmethod
+ def __setitem__(self, keys, values) -> None:
+ """Write ``values`` into the indexed portion of the variable."""
+
+ @abstractmethod
+ def read_data(self, chunking_policy: Callable[[], tuple]) -> Any:
+ """Return this variable's data, lazily where appropriate.
+
+ The one route by which the CF loader obtains arrays, so a new storage
+ format mainly needs a new implementation of this method. An
+ implementation is expected to share the CF-layer chunking helpers in
+ :mod:`iris.fileformats.cf.loader` - ``_get_actual_dtype`` for a lazy
+ result's dtype and ``_LAZYVAR_MIN_BYTES`` for the small-array cutoff -
+ as :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`
+ does, rather than reinventing them. Read ``_LAZYVAR_MIN_BYTES`` through
+ the module object (``loader._LAZYVAR_MIN_BYTES``), not by
+ ``from ... import _LAZYVAR_MIN_BYTES``, so that tests can patch the
+ cutoff.
+
+ ``chunking_policy`` is called with no arguments and returns the
+ ``(chunks, dims_fixed)`` pair for a lazy result. Call it **only** once
+ the result is known to be lazy, otherwise it may raise. Under
+ :meth:`~iris.fileformats.cf.loader.ChunkControl.from_file` it raises for
+ a variable the store has not chunked, and we want to avoid this when a
+ variable is small enough to read whole.
+
+ """
+
+ @abstractmethod
+ def write_handle(self) -> Any:
+ """Return a picklable object supporting ``__setitem__`` on this variable.
+
+ This is what a Dask worker receives as a ``da.store`` target, so it must
+ survive pickling and must remain usable after the dataset it came from
+ has been closed.
+
+ """
+
+ @property
+ def ndim(self) -> int:
+ """The number of dimensions the variable spans."""
+ return len(self.shape)
+
+ def __len__(self) -> int:
+ """Return the length of the variable's leading dimension."""
+ if not self.shape:
+ # netCDF4.Variable.__len__ raises exactly this for a scalar, and
+ # CFVariable.__len__ forwards to it.
+ msg = "len() of unsized object"
+ raise TypeError(msg)
+ return self.shape[0]
+
+ def deprecated_netcdf_member(self, name: str) -> Any:
+ """Return a netCDF4-only member of the object backing this variable.
+
+ The one-release compatibility route for code that reached netCDF4 API
+ through a ``CFVariable``. A store with no backing netCDF4 object raises
+ :class:`AttributeError`.
+
+ """
+ raise AttributeError(name)
+
+
+class CFDataset(ABC):
+ """A CF-conforming array store, open for reading or writing."""
+
+ @property
+ @abstractmethod
+ def location(self) -> str:
+ """The store's path or URL, for messages and data proxies."""
+
+ @property
+ @abstractmethod
+ def mode(self) -> str:
+ """The mode the store was opened in: ``r``, ``r+``, ``a``, ``w`` or ``w-``."""
+
+ @property
+ @abstractmethod
+ def format(self) -> str:
+ """The concrete format of this dataset.
+
+ Examples: ``NETCDF4``, ``NETCDF3_CLASSIC``, ``ZARR_V2`` or ``ZARR_V3``.
+
+ What the loader, the saver and ``LOAD_PROBLEMS`` *say* when they have
+ to name the format. Reconstructing it from the implementation class
+ would be the type test this interface exists to forbid.
+
+ """
+
+ @property
+ @abstractmethod
+ def storage_settings(self) -> Mapping:
+ """The store-wide settings this store holds, in the backend's spelling."""
+
+ @property
+ @abstractmethod
+ def closed(self) -> bool:
+ """Whether :meth:`close` has already released the store."""
+
+ @property
+ @abstractmethod
+ def variables(self) -> Mapping:
+ """The store's variables, by name."""
+
+ @property
+ @abstractmethod
+ def dimensions(self) -> Mapping:
+ """The store's dimension lengths, by name."""
+
+ @property
+ @abstractmethod
+ def attributes(self) -> MutableMapping:
+ """The store's global attributes."""
+
+ @abstractmethod
+ def create_dimension(self, name: str, size: int | None) -> None:
+ """Declare a dimension of the given length.
+
+ ``size=None`` requests an unlimited dimension. A store with no such
+ concept raises :class:`NotImplementedError`.
+
+ """
+
+ @abstractmethod
+ def create_variable(
+ self, name: str, dtype, dimensions=(), *, fill_value=None, **storage_settings
+ ) -> "CFDatasetVariable":
+ """Create and return a new variable.
+
+ ``**storage_settings`` is storage-specific: ``zlib``, ``complevel`` and
+ ``chunksizes`` for netCDF; ``compressors``, ``chunks`` and ``shards``
+ for Zarr.
+
+ """
+
+ @abstractmethod
+ def supports_dtype(self, dtype) -> bool:
+ """Whether this store can hold a variable of this dtype.
+
+ The capability question the saver asks before it narrows an integer
+ type. Asking it of the store, rather than testing the format, is what
+ keeps the saver free of format names.
+
+ Parameters
+ ----------
+ dtype : :class:`numpy.dtype`
+ The dtype a variable would be created with.
+
+ Returns
+ -------
+ bool
+
+ """
+
+ @abstractmethod
+ def sync(self) -> None:
+ """Flush buffered writes to the store."""
+
+ @abstractmethod
+ def finalise(self) -> None:
+ """Perform the store's one-shot completion step, after every write.
+
+ Deliberately **not** part of :meth:`close`: a worker that writes one
+ slab of a store must close its handle without performing a completion
+ step that only one process may perform.
+
+ """
+
+ @abstractmethod
+ def close(self) -> None:
+ """Release the store's resources."""
+
+ def __enter__(self) -> "CFDataset":
+ """Return this dataset, for use as a context manager."""
+ return self
+
+ def __exit__(self, *exc_info) -> None:
+ """Close this dataset on leaving the context, whatever happened in it."""
+ self.close()
diff --git a/lib/iris/fileformats/cf/loader.py b/lib/iris/fileformats/cf/loader.py
new file mode 100644
index 0000000000..7133f10503
--- /dev/null
+++ b/lib/iris/fileformats/cf/loader.py
@@ -0,0 +1,827 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Build Iris cubes from CF-conforming variables, whatever stores them.
+
+.. z_reference:: iris.fileformats.cf.loader
+ :tags: topic_load_save
+
+ API reference
+
+This module is the CF half of what used to be
+``iris.fileformats.netcdf.loader``. It takes the
+:class:`~iris.fileformats.cf.CFVariable` objects a
+:class:`~iris.fileformats.cf.CFReader` produced, runs the loading rules over
+them and returns :class:`~iris.cube.Cube` objects. None of that is specific to
+netCDF, and nothing here imports netCDF4 - though one deferred import still
+reaches back into :mod:`iris.fileformats.netcdf` for ``loader.DEBUG``, which
+never moves (see below). It is commented at its own call site with why.
+
+Where a storage format *does* show through - reading a variable's array - the
+work is delegated to :meth:`~iris.fileformats.cf.dataset.CFDatasetVariable.read_data`,
+and this module supplies only the chunking policy to apply if the result comes
+back lazy.
+
+Opening files remains the format's own business: see
+:func:`iris.fileformats.netcdf.load_cubes`.
+
+Also : `CF Conventions <https://cfconventions.org/>`_.
+
+"""
+
+from collections.abc import Iterable, Iterator, Mapping
+from contextlib import contextmanager
+from copy import deepcopy
+from enum import Enum, auto
+from functools import partial
+import threading
+import warnings
+
+import dask.config
+import dask.utils
+import numpy as np
+
+from iris.aux_factory import (
+ AtmosphereSigmaFactory,
+ HybridHeightFactory,
+ HybridPressureFactory,
+ OceanSFactory,
+ OceanSg1Factory,
+ OceanSg2Factory,
+ OceanSigmaFactory,
+ OceanSigmaZFactory,
+)
+import iris.coords
+import iris.fileformats.cf
+from iris.fileformats.cf.saver import _CF_ATTRS, _CFSaver
+import iris.warnings
+
+
+class _WarnComboIgnoringBoundsLoad(
+ iris.warnings.IrisIgnoringBoundsWarning,
+ iris.warnings.IrisLoadWarning,
+):
+ """One-off combination of warning classes - enhances user filtering."""
+
+ pass
+
+
+def _actions_engine():
+ # Return an 'actions engine', which provides a pyke-rules-like interface to
+ # the core cf translation code.
+ # Deferred import to avoid circularity.
+ import iris.fileformats._nc_load_rules.engine as nc_actions_engine
+
+ engine = nc_actions_engine.Engine()
+ return engine
+
+
+def _assert_case_specific_facts(engine, cf, cf_group):
+ # Initialise a data store for built cube elements.
+ # This is used to patch element attributes *not* setup by the actions
+ # process, after the actions code has run.
+ engine.cube_parts["coordinates"] = []
+ engine.cube_parts["cell_measures"] = []
+ engine.cube_parts["ancillary_variables"] = []
+ engine.cube_parts["coordinate_systems"] = {}
+
+ # Add the parsed coordinate reference system mappings
+ engine.cube_parts["coordinate_system_mappings"] = cf._coord_system_mappings.get(
+ engine.cf_var.cf_name, None
+ )
+
+ # Assert facts for CF coordinates.
+ for cf_name in cf_group.coordinates.keys():
+ engine.add_case_specific_fact("coordinate", (cf_name,))
+
+ # Assert facts for CF auxiliary coordinates.
+ for cf_name in cf_group.auxiliary_coordinates.keys():
+ engine.add_case_specific_fact("auxiliary_coordinate", (cf_name,))
+
+ # Assert facts for CF cell measures.
+ for cf_name in cf_group.cell_measures.keys():
+ engine.add_case_specific_fact("cell_measure", (cf_name,))
+
+ # Assert facts for CF ancillary variables.
+ for cf_name in cf_group.ancillary_variables.keys():
+ engine.add_case_specific_fact("ancillary_variable", (cf_name,))
+
+ # Assert facts for CF grid_mappings.
+ for cf_name in cf_group.grid_mappings.keys():
+ engine.add_case_specific_fact("grid_mapping", (cf_name,))
+
+ # Assert facts for CF labels.
+ for cf_name in cf_group.labels.keys():
+ engine.add_case_specific_fact("label", (cf_name,))
+
+ # Assert facts for CF formula terms associated with the cf_group
+ # of the CF data variable.
+
+ # Collect varnames of formula-root variables as we go.
+ # NOTE: use dictionary keys as an 'OrderedSet'
+ # - see: https://stackoverflow.com/a/53657523/2615050
+ # This is to ensure that we can handle the resulting facts in a definite
+ # order, as using a 'set' led to indeterminate results.
+ formula_root = {}
+ for cf_var in cf.cf_group.formula_terms.values():
+ for cf_root, cf_term in cf_var.cf_terms_by_root.items():
+ # Only assert this fact if the formula root variable is
+ # defined in the CF group of the CF data variable.
+ if cf_root in cf_group:
+ formula_root[cf_root] = True
+ engine.add_case_specific_fact(
+ "formula_term",
+ (cf_var.cf_name, cf_root, cf_term),
+ )
+
+ for cf_root in formula_root.keys():
+ engine.add_case_specific_fact("formula_root", (cf_root,))
+
+
+def _actions_activation_stats(engine, cf_name):
+ print("-" * 80)
+ print("CF Data Variable: %r" % cf_name)
+
+ engine.print_stats()
+
+ print("Rules Triggered:")
+
+ for rule in sorted(list(engine.rules_triggered)):
+ print("\t%s" % rule)
+
+ print("Case Specific Facts:")
+ kb_facts = engine.get_kb()
+
+ for key in kb_facts.entity_lists.keys():
+ for arg in kb_facts.entity_lists[key].case_specific_facts:
+ print("\t%s%s" % (key, arg))
+
+
+def _set_attributes(attributes, key, value):
+ """Set attributes dictionary, converting unicode strings appropriately."""
+ if isinstance(value, str):
+ try:
+ attributes[str(key)] = str(value)
+ except UnicodeEncodeError:
+ attributes[str(key)] = value
+ else:
+ attributes[str(key)] = value
+
+
+def _add_unused_attributes(iris_object, cf_var):
+ """Populate the attributes of a cf element with the "unused" attributes.
+
+ Populate the attributes of a cf element with the "unused" attributes
+ from the associated CF-netCDF variable. That is, all those that aren't CF
+ reserved terms.
+
+ """
+ from iris.fileformats._nc_load_rules.helpers import _add_or_capture
+ from iris.loading import LoadProblems
+
+ def attribute_predicate(item):
+ return item[0] not in _CF_ATTRS
+
+ tmpvar = filter(attribute_predicate, cf_var.cf_attrs_unused())
+ attrs_dict = iris_object.attributes
+ if hasattr(attrs_dict, "locals"):
+ # Treat cube attributes (i.e. a CubeAttrsDict) as a special case.
+ # These attrs are "local" (i.e. on the variable), so record them as such.
+ attrs_dict = attrs_dict.locals
+
+ for attr_name, attr_value in tmpvar:
+ _ = _add_or_capture(
+ build_func=partial(lambda: attr_value),
+ add_method=partial(_set_attributes, attrs_dict, attr_name),
+ cf_var=cf_var,
+ attr_key=attr_name,
+ destination=LoadProblems.Problem.Destination(
+ iris_class=iris_object.__class__,
+ identifier=cf_var.cf_name,
+ ),
+ )
+
+
+def _get_actual_dtype(cf_var):
+ # Figure out what the eventual data type will be after any scale/offset
+ # transforms.
+ dummy_data = np.zeros(1, dtype=cf_var.dtype)
+ if "scale_factor" in cf_var.attributes:
+ dummy_data = cf_var.attributes["scale_factor"] * dummy_data
+ if "add_offset" in cf_var.attributes:
+ dummy_data = cf_var.attributes["add_offset"] + dummy_data
+ return dummy_data.dtype
+
+
+# An arbitrary variable array size, below which we will fetch real data from a variable
+# rather than making a lazy array for deferred access.
+# Set by experiment at roughly the point where it begins to save us memory, but actually
+# mostly done for speed improvement. See https://github.com/SciTools/iris/pull/5069
+_LAZYVAR_MIN_BYTES = 5000
+
+
+def _warn_if_storage_chunk_binds(cf_var) -> None:
+ """Warn when one stored chunk is larger than Dask's target chunk size.
+
+ Storage reads a chunk whole and decompresses it whole, for netCDF and for
+ Zarr alike. So when Dask's target is the smaller of the two, the stored
+ chunking is what sets peak memory per task, and the chunking Iris picks
+ divides stored chunks rather than tiling them - fetching each one once per
+ division.
+
+ Iris does not correct for this. Which trade is right depends on the case,
+ and :data:`CHUNK_CONTROL` is how a caller says so; this warning exists so
+ that they know there is something to say. See section 4.6 of
+ ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+ """
+ storage_chunks = cf_var.cf_data.chunking
+ if storage_chunks is None:
+ return
+
+ target = dask.utils.parse_bytes(dask.config.get("array.chunk-size"))
+ stored_bytes = int(np.prod(storage_chunks)) * _get_actual_dtype(cf_var).itemsize
+ if stored_bytes <= target:
+ return
+
+ message = (
+ "One stored chunk is larger than Dask's target chunk size, so the "
+ "stored chunking sets peak memory per task and Iris's chunking will "
+ "divide it. Use iris.fileformats.cf.loader.CHUNK_CONTROL to choose "
+ f"the chunking yourself: {cf_var.cf_name}"
+ )
+ warnings.warn(message, category=iris.warnings.IrisLoadWarning)
+
+
+def _chunks_from_chunk_control(cf_var):
+ """Return the ``(chunks, dims_fixed)`` pair for a lazy array of this variable.
+
+ This is the one aspect of a lazy array that is CF's business rather than the
+ storage's: how big the dask chunks should be, given the store's own chunking
+ and whatever :data:`CHUNK_CONTROL` context is active.
+
+ Parameters
+ ----------
+ cf_var : :class:`iris.fileformats.cf.CFVariable`
+ The variable the array is being built for.
+
+ Returns
+ -------
+ tuple
+ ``("auto", None)`` in :attr:`ChunkControl.Modes.AS_DASK`, otherwise a
+ list of chunk sizes and a tuple of per-dimension "fixed" flags, both
+ shaped for :func:`iris._lazy_data.as_lazy_data`.
+
+ Raises
+ ------
+ KeyError
+ In :attr:`ChunkControl.Modes.FROM_FILE`, when ``cf_var`` is a cube's
+ data variable and the store offers no chunking to adopt.
+
+ """
+ if CHUNK_CONTROL.mode is ChunkControl.Modes.AS_DASK:
+ return "auto", None
+
+ # Get the chunking specified for the variable : this is either a shape, or
+ # None if the variable is unchunked.
+ chunks = cf_var.cf_data.chunking
+ if chunks is None:
+ # Unchunked : either a non-version-4 file, or a contiguous
+ # version-4 variable. Neither offers a chunking to adopt.
+ if CHUNK_CONTROL.mode is ChunkControl.Modes.FROM_FILE and isinstance(
+ cf_var, iris.fileformats.cf.CFDataVariable
+ ):
+ raise KeyError(
+ f"{cf_var.cf_name} does not contain pre-existing chunk specifications."
+ f" Instead, you might wish to use CHUNK_CONTROL.set(), or just use default"
+ f" behaviour outside of a context manager. "
+ )
+ # Equivalent to chunks=None, but value required by chunking control
+ chunks = list(cf_var.shape)
+ else:
+ # The chunk-control block below assigns into this.
+ chunks = list(chunks)
+
+ # Modify the chunking in the context of an active chunking control.
+ # N.B. settings specific to this named var override global ('*') ones.
+ dim_chunks = CHUNK_CONTROL.var_dim_chunksizes.get(
+ cf_var.cf_name
+ ) or CHUNK_CONTROL.var_dim_chunksizes.get("*")
+ dims = cf_var.dimensions
+ if CHUNK_CONTROL.mode is ChunkControl.Modes.FROM_FILE:
+ dims_fixed = np.ones(len(dims), dtype=bool)
+ elif not dim_chunks:
+ dims_fixed = None
+ else:
+ # Modify the chunks argument, and pass in a list of 'fixed' dims, for
+ # any of our dims which are controlled.
+ dims_fixed = np.zeros(len(dims), dtype=bool)
+ for i_dim, dim_name in enumerate(dims):
+ dim_chunksize = dim_chunks.get(dim_name)
+ if dim_chunksize:
+ if dim_chunksize == -1:
+ chunks[i_dim] = cf_var.shape[i_dim]
+ else:
+ chunks[i_dim] = dim_chunksize
+ dims_fixed[i_dim] = True
+ if dims_fixed is None:
+ dims_fixed = [dims_fixed]
+ if not any(dims_fixed):
+ # Nothing pinned, so Iris is choosing and the caller may want to know
+ # that the store's own layout has a say. A caller who has pinned
+ # anything has already taken that decision.
+ _warn_if_storage_chunk_binds(cf_var)
+ return chunks, tuple(dims_fixed)
+
+
+def _get_cf_var_data(cf_var):
+ """Get an array representing the data of a CF variable.
+
+ This is typically a lazy array, but if the variable is "sufficiently small"
+ the storage fetches the data as a real (numpy) array instead. The latter is
+ especially valuable for scalar coordinates, which are otherwise
+ unnecessarily slow + wasteful of memory.
+
+ The store builds the array; this supplies the one decision that is CF's and
+ not the store's, namely what chunking a lazy result should have. See section
+ 4.4 of ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+ """
+ # We defer the calling of _chunks_from_chunk_control since there are cases
+ # where this would unnecessarily raise an error. Such cases are skipped
+ # within the logic of read_data.
+ chunking_policy = partial(_chunks_from_chunk_control, cf_var)
+ return cf_var.cf_data.read_data(chunking_policy)
+
+
+class _OrderedAddableList(list):
+ """A custom container object for actions recording.
+
+ Used purely in actions debugging, to accumulate a record of which actions
+ were activated.
+
+ It replaces a set, so as to preserve the ordering of operations, with
+ possible repeats, and it also numbers the entries.
+
+ The actions routines invoke an 'add' method, so this effectively replaces
+ a set.add with a list.append.
+
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self._n_add = 0
+
+ def add(self, msg):
+ self._n_add += 1
+ n_add = self._n_add
+ self.append(f"#{n_add:03d} : {msg}")
+
+
+def _load_cube(engine, cf, cf_var, filename):
+ # Translate dimension chunk-settings specific to this cube (i.e. named by
+ # it's data-var) into global ones, for the duration of this load.
+ # Thus, by default, we will create any AuxCoords, CellMeasures et al with
+ # any per-dimension chunksizes specified for the cube.
+ these_settings = CHUNK_CONTROL.var_dim_chunksizes.get(cf_var.cf_name, {})
+ with CHUNK_CONTROL.set(**these_settings):
+ return _load_cube_inner(engine, cf, cf_var, filename)
+
+
+def _load_cube_inner(engine, cf, cf_var, filename):
+ from iris.cube import Cube
+
+ """Create the cube associated with the CF-netCDF data variable."""
+ if _CFSaver._DATALESS_ATTRNAME in cf_var.attributes:
+ # This data-variable represents a dataless cube.
+ # The variable array content was never written (to take up no space).
+ data = None
+ shape = cf_var.shape
+ else:
+ data = _get_cf_var_data(cf_var)
+ shape = None
+ cube = Cube(data=data, shape=shape)
+
+ # Reset the actions engine.
+ engine.reset()
+
+ # Initialise engine rule processing hooks.
+ engine.cf_var = cf_var
+ engine.cube = cube
+ engine.cube_parts = {}
+ engine.requires = {}
+ engine.rules_triggered = _OrderedAddableList()
+ engine.filename = filename
+
+ # Assert all the case-specific facts.
+ # This extracts 'facts' specific to this data-variable (aka cube), from
+ # the info supplied in the CFGroup object.
+ _assert_case_specific_facts(engine, cf, cf_var.cf_group)
+
+ # Run the actions engine.
+ # This creates various cube elements and attaches them to the cube.
+ # It also records various other info on the engine, to be processed later.
+ engine.activate()
+
+ # Having run the rules, now add the "unused" attributes to each cf element.
+ def fix_attributes_all_elements(role_name):
+ elements_and_names = engine.cube_parts.get(role_name, [])
+
+ for iris_object, cf_var_name in elements_and_names:
+ _add_unused_attributes(iris_object, cf.cf_group[cf_var_name])
+
+ # Populate the attributes of all coordinates, cell-measures and ancillary-vars.
+ fix_attributes_all_elements("coordinates")
+ fix_attributes_all_elements("ancillary_variables")
+ fix_attributes_all_elements("cell_measures")
+
+ # Also populate attributes of the top-level cube itself.
+ _add_unused_attributes(cube, cf_var)
+
+ # Work out reference names for all the coords.
+ names = {
+ coord.var_name: coord.standard_name or coord.var_name or "unknown"
+ for coord in cube.coords()
+ }
+
+ # Add all the cube cell methods.
+ cube.cell_methods = [
+ iris.coords.CellMethod(
+ method=method.method,
+ intervals=method.intervals,
+ comments=method.comments,
+ coords=[
+ names[coord_name] if coord_name in names else coord_name
+ for coord_name in method.coord_names
+ ],
+ )
+ for method in cube.cell_methods
+ ]
+
+ # Set extended_grid_mapping property ONLY if extended grid_mapping was used.
+ # This avoids having an unnecessary `iris_extended_grid_mapping` attribute entry.
+ if cs_mappings := engine.cube_parts.get("coordinate_system_mappings", None):
+ # `None` as a mapping key implies simple mapping syntax (single coord system)
+ if None not in cs_mappings:
+ cube.extended_grid_mapping = True
+
+ # DEBUG stays in iris.fileformats.netcdf (spec 4.8 lists it among the
+ # names that do not move). Read it through the module, not as a
+ # from-import, so that assigning to iris.fileformats.netcdf.loader.DEBUG -
+ # which is how it is used - still takes effect here.
+ from iris.fileformats.netcdf import loader as netcdf_loader
+
+ if netcdf_loader.DEBUG:
+ # Show activation statistics for this data-var (i.e. cube).
+ _actions_activation_stats(engine, cf_var.cf_name)
+
+ return cube
+
+
+def _load_aux_factory(engine, cube):
+ """Convert any CF-netCDF dimensionless coordinate to an AuxCoordFactory."""
+ formula_type = engine.requires.get("formula_type")
+ if formula_type in [
+ "atmosphere_sigma_coordinate",
+ "atmosphere_hybrid_height_coordinate",
+ "atmosphere_hybrid_sigma_pressure_coordinate",
+ "ocean_sigma_z_coordinate",
+ "ocean_sigma_coordinate",
+ "ocean_s_coordinate",
+ "ocean_s_coordinate_g1",
+ "ocean_s_coordinate_g2",
+ ]:
+
+ def coord_from_term(term):
+ # Convert term names to coordinates (via netCDF variable names).
+ name = engine.requires["formula_terms"].get(term, None)
+ if name is not None:
+ for coord, cf_var_name in engine.cube_parts["coordinates"]:
+ if cf_var_name == name:
+ return coord
+ warnings.warn(
+ "Unable to find coordinate for variable {!r}".format(name),
+ category=iris.warnings.IrisFactoryCoordNotFoundWarning,
+ )
+
+ if formula_type == "atmosphere_sigma_coordinate":
+ pressure_at_top = coord_from_term("ptop")
+ sigma = coord_from_term("sigma")
+ surface_air_pressure = coord_from_term("ps")
+ factory = AtmosphereSigmaFactory(
+ pressure_at_top, sigma, surface_air_pressure
+ )
+ elif formula_type == "atmosphere_hybrid_height_coordinate":
+ delta = coord_from_term("a")
+ sigma = coord_from_term("b")
+ orography = coord_from_term("orog")
+ factory = HybridHeightFactory(delta, sigma, orography)
+ elif formula_type == "atmosphere_hybrid_sigma_pressure_coordinate":
+ # Hybrid pressure has two valid versions of its formula terms:
+ # "p0: var1 a: var2 b: var3 ps: var4" or
+ # "ap: var1 b: var2 ps: var3" where "ap = p0 * a"
+ # Attempt to get the "ap" term.
+ delta = coord_from_term("ap")
+ if delta is None:
+ # The "ap" term is unavailable, so try getting terms "p0"
+ # and "a" terms in order to derive an "ap" equivalent term.
+ coord_p0 = coord_from_term("p0")
+ if coord_p0 is not None:
+ if coord_p0.shape != (1,):
+ msg = (
+ "Expecting {!r} to be a scalar reference "
+ "pressure coordinate, got shape {!r}".format(
+ coord_p0.var_name, coord_p0.shape
+ )
+ )
+ raise ValueError(msg)
+ if coord_p0.has_bounds():
+ msg = (
+ "Ignoring atmosphere hybrid sigma pressure "
+ "scalar coordinate {!r} bounds.".format(coord_p0.name())
+ )
+ warnings.warn(
+ msg,
+ category=_WarnComboIgnoringBoundsLoad,
+ )
+ coord_a = coord_from_term("a")
+ if coord_a is not None:
+ if coord_a.units.is_unknown():
+ # Be graceful, and promote unknown to dimensionless units.
+ coord_a.units = "1"
+ delta = coord_a * coord_p0.points[0]
+ delta.units = coord_a.units * coord_p0.units
+ delta.rename("vertical pressure")
+ delta.var_name = "ap"
+ cube.add_aux_coord(delta, cube.coord_dims(coord_a))
+
+ sigma = coord_from_term("b")
+ surface_air_pressure = coord_from_term("ps")
+ factory = HybridPressureFactory(delta, sigma, surface_air_pressure)
+ elif formula_type == "ocean_sigma_z_coordinate":
+ sigma = coord_from_term("sigma")
+ eta = coord_from_term("eta")
+ depth = coord_from_term("depth")
+ depth_c = coord_from_term("depth_c")
+ nsigma = coord_from_term("nsigma")
+ zlev = coord_from_term("zlev")
+ factory = OceanSigmaZFactory(sigma, eta, depth, depth_c, nsigma, zlev)
+ elif formula_type == "ocean_sigma_coordinate":
+ sigma = coord_from_term("sigma")
+ eta = coord_from_term("eta")
+ depth = coord_from_term("depth")
+ factory = OceanSigmaFactory(sigma, eta, depth)
+ elif formula_type == "ocean_s_coordinate":
+ s = coord_from_term("s")
+ eta = coord_from_term("eta")
+ depth = coord_from_term("depth")
+ a = coord_from_term("a")
+ depth_c = coord_from_term("depth_c")
+ b = coord_from_term("b")
+ factory = OceanSFactory(s, eta, depth, a, b, depth_c)
+ elif formula_type == "ocean_s_coordinate_g1":
+ s = coord_from_term("s")
+ c = coord_from_term("c")
+ eta = coord_from_term("eta")
+ depth = coord_from_term("depth")
+ depth_c = coord_from_term("depth_c")
+ factory = OceanSg1Factory(s, c, eta, depth, depth_c)
+ elif formula_type == "ocean_s_coordinate_g2":
+ s = coord_from_term("s")
+ c = coord_from_term("c")
+ eta = coord_from_term("eta")
+ depth = coord_from_term("depth")
+ depth_c = coord_from_term("depth_c")
+ factory = OceanSg2Factory(s, c, eta, depth, depth_c)
+ cube.add_aux_factory(factory)
+
+
+def _translate_constraints_to_var_callback(constraints):
+ """Translate load constraints into a simple data-var filter function, if possible.
+
+ Returns
+ -------
+ bool or None
+
+ Notes
+ -----
+ For now, ONLY handles NameConstraints with no 'STASH' component.
+
+ """
+ import iris._constraints
+
+ constraints = iris._constraints.list_of_constraints(constraints)
+ if len(constraints) == 0 or not all(
+ isinstance(constraint, iris._constraints.NameConstraint)
+ and constraint.STASH == "none"
+ for constraint in constraints
+ ):
+ # We can define a var-filtering function to speedup the load, *ONLY* when we
+ # have some constraints, and all are simple NameConstraints with no STASH.
+ result = None
+ else:
+
+ def inner(cf_datavar):
+ match_any_constraint = False
+ for constraint in constraints:
+ match_this_constraint = True
+ for name in constraint._names:
+ expected = getattr(constraint, name)
+ if name != "STASH" and expected != "none":
+ if name == "var_name":
+ actual = cf_datavar.cf_name
+ elif name in cf_datavar.attributes:
+ actual = cf_datavar.attributes[name]
+ else:
+ # Unlike NameConstraint, a variable that does not
+ # carry the attribute is not a mismatch here: the
+ # cube may still acquire the name later in the load.
+ continue
+ if actual != expected:
+ match_this_constraint = False
+ break
+ if match_this_constraint:
+ match_any_constraint = True
+ break
+ return match_any_constraint
+
+ result = inner
+ return result
+
+
+class ChunkControl(threading.local):
+ """Provide user control of Chunk Control."""
+
+ class Modes(Enum):
+ """Modes Enums."""
+
+ DEFAULT = auto()
+ FROM_FILE = auto()
+ AS_DASK = auto()
+
+ def __init__(self, var_dim_chunksizes=None):
+ """Provide user control of Dask chunking.
+
+ The CF loader is controlled by the single instance of this: the
+ :data:`~iris.fileformats.cf.loader.CHUNK_CONTROL` object.
+
+ A chunk size can be set for a specific (named) file dimension, when
+ loading specific (named) variables, or for all variables.
+
+ When a selected variable is a CF data-variable, which loads as a
+ :class:`~iris.cube.Cube`, then the given dimension chunk size is *also*
+ fixed for all variables which are components of that :class:`~iris.cube.Cube`,
+ i.e. any :class:`~iris.coords.Coord`, :class:`~iris.coords.CellMeasure`,
+ :class:`~iris.coords.AncillaryVariable` etc.
+ This can be overridden, if required, by variable-specific settings.
+
+ For this purpose, :class:`~iris.mesh.MeshCoord` and
+ :class:`~iris.mesh.Connectivity` are not
+ :class:`~iris.cube.Cube` components, and chunk control on a
+ :class:`~iris.cube.Cube` data-variable will not affect them.
+
+ """
+ self.var_dim_chunksizes = var_dim_chunksizes or {}
+ self.mode = self.Modes.DEFAULT
+
+ @contextmanager
+ def set(
+ self,
+ var_names: str | Iterable[str] | None = None,
+ **dimension_chunksizes: Mapping[str, int],
+ ) -> Iterator[None]:
+ r"""Control the Dask chunk sizes applied to NetCDF variables during loading.
+
+ This function can also be used to provide a size hint for the unknown
+ array lengths when loading "variable-length" NetCDF data types.
+ See https://unidata.github.io/netcdf4-python/#netCDF4.Dataset.vltypes
+
+ Parameters
+ ----------
+ var_names : str or list of str, default=None
+ Apply the `dimension_chunksizes` controls only to these variables,
+ or when building :class:`~iris.cube.Cube` from these data variables.
+ If ``None``, settings apply to all loaded variables.
+ **dimension_chunksizes : dict of {str: int}
+ Kwargs specifying chunksizes for dimensions of file variables.
+ Each key-value pair defines a chunk size for a named file
+ dimension, e.g. ``{'time': 10, 'model_levels':1}``.
+ Values of ``-1`` will lock the chunk size to the full size of that
+ dimension. To specify a size hint for "variable-length" data types
+ use the special name `_vl_hint`.
+
+ Notes
+ -----
+ This function acts as a context manager, for use in a ``with`` block.
+
+ >>> import iris
+ >>> from iris.fileformats.cf.loader import CHUNK_CONTROL
+ >>> with CHUNK_CONTROL.set("air_temperature", time=180, latitude=-1):
+ ... cube = iris.load(iris.sample_data_path("E1_north_america.nc"))[0]
+
+ When `var_names` is present, the chunk size adjustments are applied
+ only to the selected variables. However, for a CF data variable, this
+ extends to all components of the (raw) :class:`~iris.cube.Cube` created
+ from it.
+
+ **Un**-adjusted dimensions have chunk sizes set in the 'usual' way.
+ That is, according to the normal behaviour of
+ :func:`iris._lazy_data.as_lazy_data`, which is: chunk size is based on
+ the file variable chunking, or full variable shape; this is scaled up
+ or down by integer factors to best match the Dask default chunk size,
+ i.e. the setting configured by
+ ``dask.config.set({'array.chunk-size': '250MiB'})``.
+
+ For variable-length data types the size of the variable (or "ragged")
+ dimension of the individual array elements cannot be known without
+ reading the data. This can make it difficult for Iris to determine
+ whether to load the data lazily or not. If the user has some apriori
+ knowledge of the mean variable array length this can be passed as
+ as a size hint via the special `_vl_hint` name. For example a hint
+ that variable-length string array that contains 4 character experiment
+ identifiers:
+ ``CHUNK_CONTROL.set("expver", _vl_hint=4)``
+
+ """
+ old_mode = self.mode
+ old_var_dim_chunksizes = deepcopy(self.var_dim_chunksizes)
+ if var_names is None:
+ var_names = ["*"]
+ elif isinstance(var_names, str):
+ var_names = [var_names]
+ try:
+ for var_name in var_names:
+ # Note: here we simply treat '*' as another name.
+ # A specific name match should override a '*' setting, but
+ # that is implemented elsewhere.
+ if not isinstance(var_name, str):
+ msg = ( # type: ignore[unreachable]
+ "'var_names' should be an iterable of strings, "
+ f"not {var_names!r}."
+ )
+ raise ValueError(msg)
+ dim_chunks = self.var_dim_chunksizes.setdefault(var_name, {})
+ for dim_name, chunksize in dimension_chunksizes.items():
+ if not (isinstance(dim_name, str) and isinstance(chunksize, int)): # type: ignore[redundant-expr]
+ msg = (
+ "'dimension_chunksizes' kwargs should be a dict "
+ f"of `str: int` pairs, not {dimension_chunksizes!r}."
+ )
+ raise ValueError(msg)
+ dim_chunks[dim_name] = chunksize
+ yield
+ finally:
+ self.var_dim_chunksizes = old_var_dim_chunksizes
+ self.mode = old_mode
+
+ @contextmanager
+ def from_file(self) -> Iterator[None]:
+ r"""Ensure the chunk sizes are loaded in from NetCDF file variables.
+
+ Raises
+ ------
+ KeyError
+ If any NetCDF data variables - those that become
+ :class:`~iris.cube.Cube` - do not specify chunk sizes.
+
+ Notes
+ -----
+ This function acts as a context manager, for use in a ``with`` block.
+ """
+ old_mode = self.mode
+ old_var_dim_chunksizes = deepcopy(self.var_dim_chunksizes)
+ try:
+ self.mode = self.Modes.FROM_FILE
+ yield
+ finally:
+ self.mode = old_mode
+ self.var_dim_chunksizes = old_var_dim_chunksizes
+
+ @contextmanager
+ def as_dask(self) -> Iterator[None]:
+ """Rely on Dask :external+dask:doc:`array` to control chunk sizes.
+
+ Notes
+ -----
+ This function acts as a context manager, for use in a ``with`` block.
+
+ """
+ old_mode = self.mode
+ old_var_dim_chunksizes = deepcopy(self.var_dim_chunksizes)
+ try:
+ self.mode = self.Modes.AS_DASK
+ yield
+ finally:
+ self.mode = old_mode
+ self.var_dim_chunksizes = old_var_dim_chunksizes
+
+
+# Note: the CHUNK_CONTROL object controls chunk sizing in the
+# :meth:`_get_cf_var_data` method.
+# N.B. :meth:`_load_cube` also modifies this when loading each cube,
+# introducing an additional context in which any cube-specific settings are
+# 'promoted' into being global ones.
+
+#: The global :class:`ChunkControl` object providing user-control of Dask chunking
+#: when Iris loads NetCDF files.
+CHUNK_CONTROL: ChunkControl = ChunkControl()
diff --git a/lib/iris/fileformats/cf/saver.py b/lib/iris/fileformats/cf/saver.py
new file mode 100644
index 0000000000..d738cc5062
--- /dev/null
+++ b/lib/iris/fileformats/cf/saver.py
@@ -0,0 +1,2541 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Save Iris cubes to a CF-conforming store.
+
+.. z_reference:: iris.fileformats.cf.saver
+ :tags: topic_load_save
+
+ API reference
+
+Everything about writing a cube that does not depend on what it is written
+into. The CF data model is here - dimension naming, coordinate and bounds
+variables, cell methods, grid mappings, meshes, attribute compliance. Every
+storage call goes through :class:`~iris.fileformats.cf.dataset.CFDataset`, with
+one exception: ``_add_grid_mapping_to_dataset``.
+
+A format supplies a subclass of ``_CFSaver`` holding its own vocabulary of
+storage keywords, and whatever of the four hooks it needs:
+``_create_string_variable``, ``_backend_write``, ``_flush_before_close`` and
+``_patch_dataset``. :class:`iris.fileformats.netcdf.saver.Saver` is one such
+subclass, and is where a user of the netCDF format meets all of this.
+
+``_CFSaver`` is private. What a format must supply is the four hooks, which
+are private too, and one subclass is not enough evidence to promise their
+shape; everything public on the class is the caller's API, inherited
+unchanged by each format's own public saver. Contrast
+:class:`~iris.fileformats.cf.dataset.CFDataset`, which is public because every
+member of it is something a format must implement, and because a user can
+reach one through :attr:`iris.fileformats.cf.CFVariable.cf_data`.
+
+Two things here are still netCDF-specific.
+
+``_add_grid_mapping_to_dataset`` writes the CF grid-mapping parameters onto a
+netCDF4 variable, not through the ``attributes`` mapping. A subclass backed by
+any other store fails there, for every cube that has a coordinate system.
+Making those writes store-agnostic is its own piece of work -- see Q8 in §12.3
+of the native Zarr I/O design,
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+``ATTRIBUTE_HANDLERS`` is imported from the netCDF package, which imports this
+module back. The two-way import survives because ``_attribute_handlers``
+imports nothing from :mod:`iris.fileformats.cf`. This module's body runs while
+the netCDF package is itself part-way through importing, so a netCDF submodule
+that reached back here would find this module half-built. The handlers
+serialise the Iris-only attribute types that every CF store needs - STASH,
+GRIB_PARAM, ukmo__process_flags - so they are relocating here, and this import
+goes when they do. The load rules read them too, which is what makes that a
+move of its own; it is made alongside Zarr saving,
+https://github.com/SciTools/iris/issues/6980.
+
+See : `CF Conventions <https://cfconventions.org/>`_.
+
+"""
+
+import collections
+import re
+import string
+from typing import TYPE_CHECKING
+import warnings
+
+import cf_units
+import dask
+import dask.array as da
+from dask.delayed import Delayed
+import numpy as np
+
+from iris import FUTURE
+from iris._lazy_data import _co_realise_lazy_arrays, is_lazy_data
+from iris.aux_factory import (
+ AtmosphereSigmaFactory,
+ HybridHeightFactory,
+ HybridPressureFactory,
+ OceanSFactory,
+ OceanSg1Factory,
+ OceanSg2Factory,
+ OceanSigmaFactory,
+ OceanSigmaZFactory,
+)
+import iris.coord_systems
+import iris.coords
+from iris.coords import (
+ AncillaryVariable,
+ AuxCoord,
+ CellMeasure,
+ DimCoord,
+ _DimensionalMetadata,
+)
+import iris.exceptions
+from iris.fileformats.cf.dataset import CFDatasetVariable
+
+# TODO: remove when the handlers relocate here, with Zarr saving -
+# https://github.com/SciTools/iris/issues/6980
+# The one place the generic saver still reaches into the netCDF package. The
+# module docstring has the import cycle this closes, and why it is still here.
+from iris.fileformats.netcdf._attribute_handlers import ATTRIBUTE_HANDLERS
+import iris.util
+import iris.warnings
+
+if TYPE_CHECKING:
+ from iris.cube import Cube
+ from iris.mesh.components import (
+ MeshEdgeCoords,
+ MeshFaceCoords,
+ MeshNodeCoords,
+ MeshXY,
+ )
+
+# Standard CML spatio-temporal axis names.
+SPATIO_TEMPORAL_AXES = ["t", "z", "y", "x"]
+
+# The CF-meaningful attributes which may appear on a data variable.
+_CF_ATTRS = [
+ "add_offset",
+ "ancillary_variables",
+ "axis",
+ "bounds",
+ "calendar",
+ "cell_measures",
+ "cell_methods",
+ "climatology",
+ "compress",
+ "coordinates",
+ "_FillValue",
+ "formula_terms",
+ "grid_mapping",
+ "leap_month",
+ "leap_year",
+ "long_name",
+ "missing_value",
+ "month_lengths",
+ "scale_factor",
+ "standard_error_multiplier",
+ "standard_name",
+ "units",
+]
+
+# CF attributes that should not be global.
+_CF_DATA_ATTRS = [
+ "flag_masks",
+ "flag_meanings",
+ "flag_values",
+ "instance_dimension",
+ "missing_value",
+ "sample_dimension",
+ "standard_error_multiplier",
+]
+
+# CF attributes that should only be global.
+_CF_GLOBAL_ATTRS = ["conventions", "featureType", "history", "title"]
+
+# UKMO specific attributes that should not be global.
+_UKMO_DATA_ATTRS = [
+ "STASH",
+ "um_stash_source",
+ "ukmo__process_flags",
+ "iris_extended_grid_mapping",
+]
+
+# TODO: whenever we advance to CF-1.11 we should then discuss a completion date
+# for the deprecation of Rotated Mercator in coord_systems.py and
+# _nc_load_rules/helpers.py .
+CF_CONVENTIONS_VERSION = "CF-1.7"
+
+_FactoryDefn = collections.namedtuple(
+ "_FactoryDefn", ("primary", "std_name", "formula_terms_format")
+)
+_FACTORY_DEFNS = {
+ AtmosphereSigmaFactory: _FactoryDefn(
+ primary="sigma",
+ std_name="atmosphere_sigma_coordinate",
+ formula_terms_format="ptop: {pressure_at_top} sigma: {sigma} "
+ "ps: {surface_air_pressure}",
+ ),
+ HybridHeightFactory: _FactoryDefn(
+ primary="delta",
+ std_name="atmosphere_hybrid_height_coordinate",
+ formula_terms_format="a: {delta} b: {sigma} orog: {orography}",
+ ),
+ HybridPressureFactory: _FactoryDefn(
+ primary="delta",
+ std_name="atmosphere_hybrid_sigma_pressure_coordinate",
+ formula_terms_format="ap: {delta} b: {sigma} ps: {surface_air_pressure}",
+ ),
+ OceanSigmaZFactory: _FactoryDefn(
+ primary="zlev",
+ std_name="ocean_sigma_z_coordinate",
+ formula_terms_format="sigma: {sigma} eta: {eta} depth: {depth} "
+ "depth_c: {depth_c} nsigma: {nsigma} zlev: {zlev}",
+ ),
+ OceanSigmaFactory: _FactoryDefn(
+ primary="sigma",
+ std_name="ocean_sigma_coordinate",
+ formula_terms_format="sigma: {sigma} eta: {eta} depth: {depth}",
+ ),
+ OceanSFactory: _FactoryDefn(
+ primary="s",
+ std_name="ocean_s_coordinate",
+ formula_terms_format="s: {s} eta: {eta} depth: {depth} a: {a} b: {b} "
+ "depth_c: {depth_c}",
+ ),
+ OceanSg1Factory: _FactoryDefn(
+ primary="s",
+ std_name="ocean_s_coordinate_g1",
+ formula_terms_format="s: {s} c: {c} eta: {eta} depth: {depth} "
+ "depth_c: {depth_c}",
+ ),
+ OceanSg2Factory: _FactoryDefn(
+ primary="s",
+ std_name="ocean_s_coordinate_g2",
+ formula_terms_format="s: {s} c: {c} eta: {eta} depth: {depth} "
+ "depth_c: {depth_c}",
+ ),
+}
+
+
+class CFNameCoordMap:
+ """Provide a simple CF name to CF coordinate mapping."""
+
+ _Map = collections.namedtuple("_Map", ["name", "coord"])
+
+ def __init__(self):
+ self._map = []
+
+ def append(self, name, coord):
+ """Append the given name and coordinate pair to the mapping.
+
+ Parameters
+ ----------
+ name :
+ CF name of the associated coordinate.
+ coord :
+ The coordinate of the associated CF name.
+
+ Returns
+ -------
+ None.
+
+ """
+ self._map.append(CFNameCoordMap._Map(name, coord))
+
+ @property
+ def names(self):
+ """Return all the CF names."""
+ return [pair.name for pair in self._map]
+
+ @property
+ def coords(self):
+ """Return all the coordinates."""
+ return [pair.coord for pair in self._map]
+
+ def name(self, coord):
+ """Return the CF name, given a coordinate, or None if not recognised.
+
+ Parameters
+ ----------
+ coord :
+ The coordinate of the associated CF name.
+
+ Returns
+ -------
+ Coordinate or None.
+
+ """
+ result = None
+ for pair in self._map:
+ if coord == pair.coord:
+ result = pair.name
+ break
+ return result
+
+ def coord(self, name):
+ """Return the coordinate, given a CF name, or None if not recognised.
+
+ Parameters
+ ----------
+ name :
+ CF name of the associated coordinate, or None if not recognised.
+
+ Returns
+ -------
+ CF name or None.
+
+ """
+ result = None
+ for pair in self._map:
+ if name == pair.name:
+ result = pair.coord
+ break
+ return result
+
+
+# NOTE : this matches :class:`iris.mesh.MeshXY.ELEMENTS`,
+# but in the preferred order for coord/connectivity variables in the file.
+MESH_ELEMENTS = ("node", "edge", "face")
+
+
+class SaverFillValueWarning(iris.warnings.IrisSaverFillValueWarning):
+ """Backwards compatible form of :class:`iris.warnings.IrisSaverFillValueWarning`."""
+
+ # TODO: remove at the next major release.
+ pass
+
+
+class _CFSaver:
+ """A manager for saving cubes to a CF-conforming store.
+
+ Private: a format's own saver subclasses this and is the public class.
+ The module docstring says why.
+ """
+
+ def __init__(self, dataset, compute=True, *, owns_dataset=True):
+ """Manage saving cubes into an open store.
+
+ Parameters
+ ----------
+ dataset : :class:`~iris.fileformats.cf.dataset.CFDataset`
+ An open, writable store. A subclass opens it and passes it here.
+ compute : bool, default=True
+ If ``True``, delayed variable saves will be completed on exit from
+ the saver context, after the store is closed, equivalent to
+ :meth:`complete`.
+
+ If ``False``, the store is created and closed without writing the
+ data of variables whose source data was lazy. Those writes can be
+ completed later, see :meth:`delayed_completion`.
+ owns_dataset : bool, default=True
+ Whether leaving the context should close the store. ``False`` when
+ the caller supplied an already-open store and will close it itself.
+
+ """
+ # All persistent variables
+ #: CF name mapping with iris coordinates
+ self._name_coord_map = CFNameCoordMap()
+ #: Map of dimensions to characteristic coordinates with which they are identified
+ self._dim_names_and_coords = CFNameCoordMap()
+ #: List of grid mappings added to the file
+ self._coord_systems = []
+ #: A dictionary, listing dimension names and corresponding length
+ self._existing_dim = {}
+ #: A map from meshes to their actual file dimensions (names).
+ # NB: might not match those of the mesh, if they were 'incremented'.
+ self._mesh_dims = {}
+ #: A dictionary, mapping formula terms to owner cf variable name
+ self._formula_terms_cache = {}
+ #: Whether to complete delayed saves on exit.
+ self.compute = compute
+ # A list of delayed writes for lazy saving
+ # a list of couples (source, target).
+ self._delayed_writes = []
+
+ self._dataset = dataset
+ self._owns_dataset = owns_dataset
+ #: Target filepath
+ self.filepath = dataset.location
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, type, value, traceback):
+ """Flush any buffered data to the store before closing."""
+ self._flush_before_close()
+ self._dataset.sync()
+ if self._owns_dataset:
+ # Only close if the saver opened it.
+ self._dataset.close()
+ # Complete after closing, if required
+ if self.compute:
+ self.complete()
+
+ def _flush_before_close(self) -> None:
+ """Write anything that cannot be written once the store is closed.
+
+ Nothing, here. A store that cannot be reopened for a deferred write
+ overrides this and writes its lazy data while it is still open.
+
+ """
+ pass
+
+ def write(
+ self,
+ cube,
+ local_keys=None,
+ unlimited_dimensions=None,
+ packing=None,
+ fill_value=None,
+ compression_kwargs=None,
+ **encoding,
+ ):
+ """Write a cube into the store.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube`
+ The cube to save.
+ local_keys : iterable of str, optional
+ Cube attribute keys that become attributes of the data variable
+ rather than global attributes.
+
+ .. note::
+
+ Has no effect if :attr:`iris.FUTURE.save_split_attrs` is ``True``.
+ unlimited_dimensions : iterable of str and/or :class:`iris.coords.Coord`, optional
+ Coordinate names, or coordinates, whose dimensions are saved with
+ unlimited length. By default, none are.
+ packing : type or str or dict or list, optional
+ How to pack the data, if at all. See
+ :meth:`iris.fileformats.netcdf.saver.Saver.write`.
+ fill_value : optional
+ The value for the data variable's ``_FillValue`` attribute.
+ compression_kwargs : dict, optional
+ Storage keywords applied to *every* variable this cube creates, not
+ only its data variable. The store's own vocabulary.
+ **encoding
+ Storage keywords applied to the data variable alone. The store's
+ own vocabulary.
+
+ Returns
+ -------
+ None
+
+ """
+ # TODO: when iris.FUTURE.save_split_attrs defaults to True, we can deprecate the
+ # "local_keys" arg, and finally remove it when we finally remove the
+ # save_split_attrs switch.
+ if unlimited_dimensions is None:
+ unlimited_dimensions = []
+
+ cf_profile_available = iris.site_configuration.get("cf_profile") not in [
+ None,
+ False,
+ ]
+ if cf_profile_available:
+ # Perform a CF profile of the cube. This may result in an exception
+ # being raised if mandatory requirements are not satisfied.
+ profile = iris.site_configuration["cf_profile"](cube)
+
+ # Ensure that attributes are CF compliant and if possible to make them
+ # compliant.
+ self.check_attribute_compliance(cube, cube.dtype)
+ for coord in cube.coords():
+ self.check_attribute_compliance(coord, coord.dtype)
+
+ # Get suitable dimension names.
+ mesh_dimensions, cube_dimensions = self._get_dim_names(cube)
+
+ # Create all the CF-netCDF data dimensions.
+ # Put mesh dims first, then non-mesh dims in cube-occurring order.
+ nonmesh_dimensions = [
+ dim for dim in cube_dimensions if dim not in mesh_dimensions
+ ]
+ all_dimensions = mesh_dimensions + nonmesh_dimensions
+ self._create_cf_dimensions(cube, all_dimensions, unlimited_dimensions)
+
+ # Group the storage keywords that apply to every variable, as they are
+ # applied to other cube metadata as well as the cube data payload.
+ compression_kwargs = dict(compression_kwargs or {})
+
+ # Create the mesh components, if there is a mesh.
+ # We do this before creating the data-var, so that mesh vars precede
+ # data-vars in the file.
+ cf_mesh_name = self._add_mesh(cube, compression_kwargs=compression_kwargs)
+
+ # Create the associated cube CF-netCDF data variable.
+ data_encoding = dict(encoding)
+ data_encoding.update(compression_kwargs)
+ cf_var_cube = self._create_cf_data_variable(
+ cube,
+ cube_dimensions,
+ local_keys=local_keys,
+ packing=packing,
+ fill_value=fill_value,
+ **data_encoding,
+ )
+
+ # Associate any mesh with the data-variable.
+ # N.B. _add_mesh cannot do this, as we want to put mesh variables
+ # before data-variables in the file.
+ if cf_mesh_name is not None:
+ cf_var_cube.attributes["mesh"] = cf_mesh_name
+ cf_var_cube.attributes["location"] = cube.location
+
+ # Add coordinate variables.
+ self._add_dim_coords(cube, cube_dimensions)
+
+ # Add the auxiliary coordinate variables and associate the data
+ # variable to them
+ self._add_aux_coords(
+ cube, cf_var_cube, cube_dimensions, compression_kwargs=compression_kwargs
+ )
+
+ # Add the cell_measures variables and associate the data
+ # variable to them
+ self._add_cell_measures(cube, cf_var_cube, cube_dimensions)
+
+ # Add the ancillary_variables variables and associate the data variable
+ # to them
+ self._add_ancillary_variables(
+ cube, cf_var_cube, cube_dimensions, compression_kwargs=compression_kwargs
+ )
+
+ # Add the formula terms to the appropriate cf variables for each
+ # aux factory in the cube.
+ self._add_aux_factories(cube, cf_var_cube, cube_dimensions)
+
+ if not iris.FUTURE.save_split_attrs:
+ # In the "old" way, we update global attributes as we go.
+ # Add data variable-only attribute names to local_keys.
+ if local_keys is None:
+ local_keys = set()
+ else:
+ local_keys = set(local_keys)
+ local_keys.update(_CF_DATA_ATTRS, _UKMO_DATA_ATTRS, ["GRIB_PARAM"])
+
+ # Add global attributes taking into account local_keys.
+ # NOTE: we never need to "convert" managed global attributes, as they are all
+ # data-local type, i.e. included in "local_keys" above.
+ cube_attributes = cube.attributes
+ global_attributes = {
+ k: v
+ for k, v in cube_attributes.items()
+ if (k not in local_keys and k.lower() != "conventions")
+ }
+ self.update_global_attributes(global_attributes)
+
+ if cf_profile_available:
+ cf_patch = iris.site_configuration.get("cf_patch")
+ if cf_patch is not None:
+ self._patch_dataset(cf_patch, profile, cf_var_cube)
+ else:
+ msg = "cf_profile is available but no {} defined.".format("cf_patch")
+ warnings.warn(msg, category=iris.warnings.IrisCfSaveWarning)
+
+ def _patch_dataset(self, cf_patch, profile, cf_var_cube) -> None:
+ """Hand the site-configured patch hook the objects it expects.
+
+ Parameters
+ ----------
+ cf_patch : callable
+ The hook, from ``iris.site_configuration``.
+ profile : object
+ Whatever the configured ``cf_profile`` returned for this cube.
+ cf_var_cube : :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+ The cube's data variable.
+
+ Raises
+ ------
+ NotImplementedError
+ Always, here. The hook has been documented as receiving netCDF4
+ objects since Iris 1.3.
+
+ """
+ msg = (
+ "The 'cf_patch' site configuration hook is documented as receiving "
+ f"netCDF4 objects, and {self._dataset.format} has none to give it."
+ )
+ raise NotImplementedError(msg)
+
+ def check_attribute_compliance(self, container, data_dtype):
+ """Check attributte complliance."""
+
+ def _coerce_value(val_attr, val_attr_value, data_dtype):
+ val_attr_tmp = np.array(val_attr_value, dtype=data_dtype)
+ if (val_attr_tmp != val_attr_value).any():
+ msg = '"{}" is not of a suitable value ({})'
+ raise ValueError(msg.format(val_attr, val_attr_value))
+ return val_attr_tmp
+
+ # Ensure that conflicting attributes are not provided.
+ if (
+ container.attributes.get("valid_min") is not None
+ or container.attributes.get("valid_max") is not None
+ ) and container.attributes.get("valid_range") is not None:
+ msg = (
+ 'Both "valid_range" and "valid_min" or "valid_max" attributes present.'
+ )
+ raise ValueError(msg)
+
+ # Ensure correct datatype
+ for val_attr in ["valid_range", "valid_min", "valid_max"]:
+ val_attr_value = container.attributes.get(val_attr)
+ if val_attr_value is not None:
+ val_attr_value = np.asarray(val_attr_value)
+ self._ensure_valid_dtype(val_attr_value, val_attr, val_attr_value)
+ if data_dtype.itemsize == 1:
+ # Allow signed integral type
+ if val_attr_value.dtype.kind == "i":
+ continue
+ new_val = _coerce_value(val_attr, val_attr_value, data_dtype)
+ container.attributes[val_attr] = new_val
+
+ def update_global_attributes(self, attributes=None, **kwargs):
+ """Update the CF global attributes.
+
+ Update the CF global attributes based on the provided
+ iterable/dictionary and/or keyword arguments.
+
+ Parameters
+ ----------
+ attributes : dict or iterable of key, value pairs, optional
+ CF global attributes to be updated.
+ """
+ # TODO: when when iris.FUTURE.save_split_attrs is removed, this routine will
+ # only be called once: it can reasonably be renamed "_set_global_attributes",
+ # and the 'kwargs' argument can be removed.
+ if attributes is not None:
+ # Handle sequence e.g. [('fruit', 'apple'), ...].
+ if not hasattr(attributes, "keys"):
+ attributes = dict(attributes)
+
+ for attr_name in sorted(attributes):
+ self._dataset.attributes[attr_name] = attributes[attr_name]
+
+ for attr_name in sorted(kwargs):
+ self._dataset.attributes[attr_name] = kwargs[attr_name]
+
+ def _create_cf_dimensions(self, cube, dimension_names, unlimited_dimensions=None):
+ """Create the CF-netCDF data dimensions.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube`
+ A :class:`iris.cube.Cube` in which to lookup coordinates.
+ dimension_names :
+ unlimited_dimensions : iterable of strings and/or :class:`iris.coords.Coord` objects):
+ List of coordinates to make unlimited (None by default).
+
+ Returns
+ -------
+ None.
+
+ """
+ unlimited_dim_names = []
+ if unlimited_dimensions is not None:
+ for coord in unlimited_dimensions:
+ try:
+ coord = cube.coord(name_or_coord=coord, dim_coords=True)
+ except iris.exceptions.CoordinateNotFoundError:
+ # coordinate isn't used for this cube, but it might be
+ # used for a different one
+ pass
+ else:
+ dim_name = self._get_element_variable_name(cube, coord)
+ unlimited_dim_names.append(dim_name)
+
+ for dim_name in dimension_names:
+ # NOTE: these dim-names have been chosen by _get_dim_names, and
+ # were already checked+fixed to avoid any name collisions.
+ if dim_name not in self._dataset.dimensions:
+ if dim_name in unlimited_dim_names:
+ size = None
+ else:
+ size = self._existing_dim[dim_name]
+ self._dataset.create_dimension(dim_name, size)
+
+ def _add_mesh(self, cube_or_mesh, /, *, compression_kwargs=None):
+ """Add the cube's mesh, and all related variables to the dataset.
+
+ Add the cube's mesh, and all related variables to the dataset.
+ Includes all the mesh-element coordinate and connectivity variables.
+
+ .. note::
+
+ Here, we do *not* add the relevant referencing attributes to the
+ data-variable, because we want to create the data-variable later.
+
+ Parameters
+ ----------
+ cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
+ The Cube or Mesh being saved to the netCDF file.
+ compression_kwargs : dict, optional
+ NetCDF data compression keyword arguments.
+
+ Returns
+ -------
+ str or None
+ The name of the mesh variable created, or None if the cube does not
+ have a mesh.
+
+ """
+ cf_mesh_name = None
+
+ # Do cube- or -mesh-based save
+ from iris.cube import Cube
+ from iris.mesh.components import _MeshIndexSet
+
+ if isinstance(cube_or_mesh, Cube):
+ mesh = cube_or_mesh.mesh
+ else:
+ mesh = cube_or_mesh
+
+ if isinstance(mesh, _MeshIndexSet):
+ msg = "_MeshIndexSet saving is not yet supported. See https://github.com/SciTools/iris/issues/6123"
+ raise ValueError(msg)
+
+ if mesh:
+ cf_mesh_name = self._name_coord_map.name(mesh)
+ if cf_mesh_name is None:
+ # Not already present : create it
+ cf_mesh_name = self._create_mesh(mesh)
+ self._name_coord_map.append(cf_mesh_name, mesh)
+
+ cf_mesh_var = self._dataset.variables[cf_mesh_name]
+
+ # Get the mesh-element dim names.
+ mesh_dims = self._mesh_dims[mesh]
+
+ # Add all the element coordinate variables.
+ for location in MESH_ELEMENTS:
+ coords_meshobj_attr = f"{location}_coords"
+ coords_file_attr = f"{location}_coordinates"
+ mesh_coords = getattr(mesh, coords_meshobj_attr, None)
+ if mesh_coords:
+ coord_names = []
+ for coord in mesh_coords:
+ if coord is None:
+ continue # an awkward thing that mesh.coords does
+ coord_name = self._create_generic_cf_array_var(
+ mesh,
+ [],
+ coord,
+ element_dims=(mesh_dims[location],),
+ compression_kwargs=compression_kwargs,
+ )
+ # Only created once per file, but need to fetch the
+ # name later in _add_inner_related_vars().
+ self._name_coord_map.append(coord_name, coord)
+ coord_names.append(coord_name)
+ # Record the coordinates (if any) on the mesh variable.
+ if coord_names:
+ coord_names = " ".join(coord_names)
+ cf_mesh_var.attributes[coords_file_attr] = coord_names
+
+ # Add all the connectivity variables.
+ # pre-fetch the set + ignore "None"s, which are empty slots.
+ conns = [conn for conn in mesh.all_connectivities if conn is not None]
+ for conn in conns:
+ # Get the connectivity role, = "{loc1}_{loc2}_connectivity".
+ cf_conn_attr_name = conn.cf_role
+ loc_from, loc_to, _ = cf_conn_attr_name.split("_")
+ # Construct a trailing dimension name.
+ last_dim = f"{cf_mesh_name}_{loc_from}_N_{loc_to}s"
+ # Create if it does not already exist.
+ if last_dim not in self._dataset.dimensions:
+ while last_dim in self._dataset.variables:
+ # Also avoid collision with variable names.
+ # See '_get_dim_names' for reason.
+ last_dim = self._increment_name(last_dim)
+ length = conn.shape[1 - conn.location_axis]
+ self._dataset.create_dimension(last_dim, length)
+
+ # Create variable.
+ # NOTE: for connectivities *with missing points*, this will use a
+ # fixed standard fill-value of -1. In that case, we create the
+ # variable with a '_FillValue' property, which can only be done
+ # when it is first created.
+ loc_dim_name = mesh_dims[loc_from]
+ conn_dims = (loc_dim_name, last_dim)
+ if conn.location_axis == 1:
+ # Has the 'other' dimension order, =reversed
+ conn_dims = conn_dims[::-1]
+ if iris.util.is_masked(conn.core_indices()):
+ # Flexible mesh.
+ fill_value = -1
+ else:
+ fill_value = None
+ cf_conn_name = self._create_generic_cf_array_var(
+ mesh,
+ [],
+ conn,
+ element_dims=conn_dims,
+ fill_value=fill_value,
+ compression_kwargs=compression_kwargs,
+ )
+ # Add essential attributes to the Connectivity variable.
+ cf_conn_var = self._dataset.variables[cf_conn_name]
+ cf_conn_var.attributes["cf_role"] = cf_conn_attr_name
+ cf_conn_var.attributes["start_index"] = conn.start_index
+
+ # Record the connectivity on the parent mesh var.
+ cf_mesh_var.attributes[cf_conn_attr_name] = cf_conn_name
+ # If the connectivity had the 'alternate' dimension order, add the
+ # relevant dimension property
+ if conn.location_axis == 1:
+ loc_dim_attr = f"{loc_from}_dimension"
+ # Should only get here once.
+ assert loc_dim_attr not in cf_mesh_var.attributes
+ cf_mesh_var.attributes[loc_dim_attr] = loc_dim_name
+
+ return cf_mesh_name
+
+ def _add_inner_related_vars(
+ self,
+ cube,
+ cf_var_cube,
+ dimension_names,
+ coordlike_elements,
+ /,
+ *,
+ compression_kwargs=None,
+ ) -> None:
+ """Create a set of variables for aux-coords, ancillaries or cell-measures.
+
+ Create a set of variables for aux-coords, ancillaries or cell-measures,
+ and attach them to the parent data variable.
+
+ """
+ if coordlike_elements:
+ # Choose the appropriate parent attribute
+ elem_type = type(coordlike_elements[0])
+ if elem_type in (AuxCoord, DimCoord):
+ role_attribute_name = "coordinates"
+ elif elem_type == AncillaryVariable:
+ role_attribute_name = "ancillary_variables"
+ else:
+ # We *only* handle aux-coords, cell-measures and ancillaries
+ assert elem_type == CellMeasure
+ role_attribute_name = "cell_measures"
+
+ # Add CF-netCDF variables for the given cube components.
+ element_names = []
+ for element in sorted(
+ coordlike_elements, key=lambda element: element.name()
+ ):
+ # Reuse, or create, the associated CF-netCDF variable.
+ cf_name = self._name_coord_map.name(element)
+ if cf_name is None:
+ # Not already present : create it
+ cf_name = self._create_generic_cf_array_var(
+ cube,
+ dimension_names,
+ element,
+ compression_kwargs=compression_kwargs,
+ )
+ self._name_coord_map.append(cf_name, element)
+
+ if role_attribute_name == "cell_measures":
+ # In the case of cell-measures, the attribute entries are not just
+ # a var_name, but each have the form "<measure>: <varname>".
+ cf_name = "{}: {}".format(element.measure, cf_name)
+
+ element_names.append(cf_name)
+
+ # Add CF-netCDF references to the primary data variable.
+ if element_names:
+ variable_names = " ".join(sorted(element_names))
+ cf_var_cube.attributes[role_attribute_name] = variable_names
+
+ def _add_aux_coords(
+ self, cube, cf_var_cube, dimension_names, /, *, compression_kwargs=None
+ ) -> None:
+ """Add aux. coordinate to the dataset and associate with the data variable.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube`
+ A :class:`iris.cube.Cube` to be saved to a netCDF file.
+ cf_var_cube : :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`
+ A cf variable cube representation.
+ dimension_names : list
+ Names associated with the dimensions of the cube.
+ compression_kwargs : dict, optional
+ NetCDF data compression keyword arguments.
+
+ """
+ # Exclude any mesh coords, which are bundled in with the aux-coords.
+ coords_to_add = [
+ coord for coord in cube.aux_coords if not hasattr(coord, "mesh")
+ ]
+
+ # Include any relevant mesh location coordinates.
+ mesh: MeshXY | None = getattr(cube, "mesh") # type: ignore[annotation-unchecked]
+ mesh_location: str | None = getattr(cube, "location") # type: ignore[annotation-unchecked]
+ if mesh and mesh_location:
+ location_coords: MeshNodeCoords | MeshEdgeCoords | MeshFaceCoords = getattr(
+ mesh, f"{mesh_location}_coords"
+ ) # type: ignore[annotation-unchecked]
+ coords_to_add.extend(list(location_coords))
+
+ return self._add_inner_related_vars(
+ cube,
+ cf_var_cube,
+ dimension_names,
+ coords_to_add,
+ compression_kwargs=compression_kwargs,
+ )
+
+ def _add_cell_measures(self, cube, cf_var_cube, dimension_names):
+ """Add cell measures to the dataset and associate with the data variable.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube`
+ A :class:`iris.cube.Cube` to be saved to a netCDF file.
+ cf_var_cube : :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`
+ A cf variable cube representation.
+ dimension_names : list
+ Names associated with the dimensions of the cube.
+ """
+ return self._add_inner_related_vars(
+ cube,
+ cf_var_cube,
+ dimension_names,
+ cube.cell_measures(),
+ )
+
+ def _add_ancillary_variables(
+ self, cube, cf_var_cube, dimension_names, /, *, compression_kwargs=None
+ ):
+ """Add ancillary variables measures to the dataset and associate with the data variable.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube`
+ A :class:`iris.cube.Cube` to be saved to a netCDF file.
+ cf_var_cube : :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`
+ A cf variable cube representation.
+ dimension_names : list
+ Names associated with the dimensions of the cube.
+ compression_kwargs : dict, optional
+ NetCDF data compression keyword arguments.
+
+ """
+ return self._add_inner_related_vars(
+ cube,
+ cf_var_cube,
+ dimension_names,
+ cube.ancillary_variables(),
+ compression_kwargs=compression_kwargs,
+ )
+
+ def _add_dim_coords(self, cube, dimension_names):
+ """Add coordinate variables to NetCDF dataset.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube`
+ A :class:`iris.cube.Cube` to be saved to a netCDF file.
+ dimension_names : list
+ Names associated with the dimensions of the cube.
+ """
+ # Ensure we create the netCDF coordinate variables first.
+ for coord in cube.dim_coords:
+ # Create the associated coordinate CF-netCDF variable.
+ if coord not in self._name_coord_map.coords:
+ # Not already present : create it
+ cf_name = self._create_generic_cf_array_var(
+ cube, dimension_names, coord
+ )
+ self._name_coord_map.append(cf_name, coord)
+
+ def _add_aux_factories(self, cube, cf_var_cube, dimension_names):
+ """Represent the presence of dimensionless vertical coordinates.
+
+ Modify the variables of the NetCDF dataset to represent
+ the presence of dimensionless vertical coordinates based on
+ the aux factories of the cube (if any).
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube`
+ A :class:`iris.cube.Cube` to be saved to a netCDF file.
+ cf_var_cube : :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`
+ CF variable cube representation.
+ dimension_names : list
+ Names associated with the dimensions of the cube.
+ """
+ primaries = []
+ for factory in cube.aux_factories:
+ factory_defn = _FACTORY_DEFNS.get(type(factory), None)
+ if factory_defn is None:
+ msg = "Unable to determine formula terms for AuxFactory: {!r}".format(
+ factory
+ )
+ warnings.warn(msg, category=iris.warnings.IrisSaveWarning)
+ else:
+ # Override `standard_name`, `long_name`, and `axis` of the
+ # primary coord that signals the presence of a dimensionless
+ # vertical coord, then set the `formula_terms` attribute.
+ primary_coord = factory.dependencies[factory_defn.primary]
+ if primary_coord in primaries:
+ msg = (
+ "Cube {!r} has multiple aux factories that share "
+ "a common primary coordinate {!r}. Unable to save "
+ "to netCDF as having multiple formula terms on a "
+ "single coordinate is not supported."
+ )
+ raise ValueError(msg.format(cube, primary_coord.name()))
+ primaries.append(primary_coord)
+
+ cf_name = self._name_coord_map.name(primary_coord)
+ cf_var = self._dataset.variables[cf_name]
+
+ term_varnames = {
+ term: self._name_coord_map.name(coord)
+ for term, coord in factory.dependencies.items()
+ }
+ formula_terms = factory_defn.formula_terms_format.format(
+ **term_varnames
+ )
+ std_name = factory_defn.std_name
+
+ if "formula_terms" in cf_var.attributes:
+ if (
+ cf_var.attributes["formula_terms"] != formula_terms
+ or cf_var.attributes["standard_name"] != std_name
+ ):
+ # TODO: We need to resolve this corner-case where
+ # the dimensionless vertical coordinate containing
+ # the formula_terms is a dimension coordinate of
+ # the associated cube and a new alternatively named
+ # dimensionless vertical coordinate is required
+ # with new formula_terms and a renamed dimension.
+ if cf_name in dimension_names:
+ msg = "Unable to create dimensonless vertical coordinate."
+ raise ValueError(msg)
+ key = (cf_name, std_name, formula_terms)
+ name = self._formula_terms_cache.get(key)
+ if name is None:
+ # Create a new variable
+ name = self._create_generic_cf_array_var(
+ cube, dimension_names, primary_coord
+ )
+ cf_var = self._dataset.variables[name]
+ cf_var.attributes["standard_name"] = std_name
+ cf_var.attributes["axis"] = "Z"
+ # Update the formula terms.
+ ft = formula_terms.split()
+ ft = [name if t == cf_name else t for t in ft]
+ cf_var.attributes["formula_terms"] = " ".join(ft)
+ # Update the cache.
+ self._formula_terms_cache[key] = name
+ # Update the associated cube variable.
+ coords = cf_var_cube.attributes["coordinates"].split()
+ coords = [name if c == cf_name else c for c in coords]
+ cf_var_cube.attributes["coordinates"] = " ".join(coords)
+ else:
+ cf_var.attributes["standard_name"] = std_name
+ cf_var.attributes["axis"] = "Z"
+ cf_var.attributes["formula_terms"] = formula_terms
+
+ if FUTURE.derived_bounds:
+ # ensure that the primary variable *bounds*, if any, obey the CF
+ # encoding rule : the bounds variable of a parametric coordinate
+ # must itself have a "formula_terms" attribute.
+ # See : https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#boundaries-and-formula-terms
+ bounds_varname = cf_var.attributes.get("bounds")
+ cf_bounds_var = self._dataset.variables.get(bounds_varname, None)
+ if (
+ cf_bounds_var is not None
+ and cf_bounds_var.attributes.get("formula_terms") is None
+ ):
+ # We need a bounds formula, and there is none already attached.
+ # Construct and add one, mirroring the main formula.
+ def boundsterm_varname(term_varname):
+ """Identify the boundsvar for a given term var."""
+ # First establish a fallback default, meaning 'no bounds'.
+ result = term_varname
+ # Follow links (if they exist) to find the bounds var.
+ termvar = self._dataset.variables.get(term_varname)
+ boundsname = (
+ None
+ if termvar is None
+ else termvar.attributes.get("bounds")
+ )
+ if boundsname in self._dataset.variables:
+ result = boundsname
+ return result
+
+ boundsterm_varnames = {
+ term: boundsterm_varname(term_varname)
+ for term, term_varname in term_varnames.items()
+ }
+ bounds_formula_terms = factory_defn.formula_terms_format.format(
+ **boundsterm_varnames
+ )
+ cf_bounds_var.attributes["formula_terms"] = bounds_formula_terms
+
+ def _get_dim_names(self, cube_or_mesh):
+ """Determine suitable CF-netCDF data dimension names.
+
+ Parameters
+ ----------
+ cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
+ The Cube or Mesh being saved to the netCDF file.
+
+ Returns
+ -------
+ mesh_dimensions : list of str
+ A list of the mesh dimensions of the attached mesh, if any.
+ cube_dimensions : list of str
+ A lists of dimension names for each dimension of the cube.
+
+ Notes
+ -----
+ The returned lists are in the preferred file creation order.
+ One of the mesh dimensions will typically also appear in the cube
+ dimensions.
+
+ """
+
+ def record_dimension(names_list, dim_name, length, matching_coords=None):
+ """Record a file dimension, its length and associated "coordinates".
+
+ Record a file dimension, its length and associated "coordinates"
+ (which may in fact also be connectivities).
+
+ If the dimension has been seen already, check that it's length
+ matches the earlier finding.
+
+ """
+ if matching_coords is None:
+ matching_coords = []
+ if dim_name not in self._existing_dim:
+ self._existing_dim[dim_name] = length
+ else:
+ # Make sure we never re-write one, though it's really not
+ # clear how/if this could ever actually happen.
+ existing_length = self._existing_dim[dim_name]
+ if length != existing_length:
+ msg = (
+ "Netcdf saving error : existing dimension "
+ f'"{dim_name}" has length {existing_length}, '
+ f"but occurrence in cube {cube} has length {length}"
+ )
+ raise ValueError(msg)
+
+ # Record given "coords" (sort-of, maybe connectivities) to be
+ # identified with this dimension: add to the already-seen list.
+ existing_coords = self._dim_names_and_coords.coords
+ for coord in matching_coords:
+ if coord not in existing_coords:
+ self._dim_names_and_coords.append(dim_name, coord)
+
+ # Add the latest name to the list passed in.
+ names_list.append(dim_name)
+
+ # Choose cube or mesh saving.
+ from iris.cube import Cube
+
+ if isinstance(cube_or_mesh, Cube):
+ # there is a mesh, only if the cube has one
+ cube = cube_or_mesh
+ mesh = cube.mesh
+ else:
+ # there is no cube, only a mesh
+ cube = None
+ mesh = cube_or_mesh
+
+ # Get info on mesh, first.
+ mesh_dimensions = []
+ if mesh is None:
+ cube_mesh_dim = None
+ else:
+ # Identify all the mesh dimensions.
+ mesh_location_dimnames = {}
+ # NOTE: one of these will be a cube dimension, but that one does not
+ # get any special handling. We *do* want to list/create them in a
+ # definite order (node,edge,face), and before non-mesh dimensions.
+ for location in MESH_ELEMENTS:
+ # Find if this location exists in the mesh, and a characteristic
+ # coordinate to identify it with.
+ # To use only _required_ UGRID components, we use a location
+ # coord for nodes, but a connectivity for faces/edges
+ if location == "node":
+ # For nodes, identify the dim with a coordinate variable.
+ dim_coords = mesh.coords(location="node")
+ else:
+ # For face/edge, use the relevant "optionally required"
+ # connectivity variable.
+ cf_role = f"{location}_node_connectivity"
+ dim_coords = mesh.connectivities(cf_role=cf_role)
+ if len(dim_coords) > 0:
+ # As the mesh contains this location, we want to include this
+ # dim in our returned mesh dims.
+ # We should have 1 identifying variable (of either type).
+ dim_element = dim_coords[0]
+ dim_name = self._dim_names_and_coords.name(dim_element)
+ if dim_name is None:
+ # No existing dim matches this, so assign a new name
+ if location == "node":
+ # always 1-d
+ (dim_length,) = dim_element.shape
+ else:
+ # extract source dim, respecting dim-ordering
+ dim_length = dim_element.shape[dim_element.location_axis]
+ # Name it for the relevant mesh dimension
+ location_dim_attr = f"{location}_dimension"
+ dim_name = getattr(mesh, location_dim_attr)
+ # NOTE: This cannot currently be empty, as a MeshXY
+ # "invents" dimension names which were not given.
+ assert dim_name is not None
+ # Ensure it is a valid variable name.
+ dim_name = self.cf_valid_var_name(dim_name)
+ # Disambiguate if it has the same name as an existing
+ # dimension.
+ # NOTE: *OR* if it matches the name of an existing file
+ # variable. Because there is a bug ...
+ # See https://github.com/Unidata/netcdf-c/issues/1772
+ # N.B. the workarounds here *ONLY* function because the
+ # caller (write) will not create any more variables
+ # in between choosing dim names (here), and creating
+ # the new dims (via '_create_cf_dimensions').
+ while (
+ dim_name in self._existing_dim
+ or dim_name in self._dataset.variables
+ ):
+ dim_name = self._increment_name(dim_name)
+
+ # Record the new dimension.
+ record_dimension(
+ mesh_dimensions, dim_name, dim_length, dim_coords
+ )
+
+ # Store the mesh dims indexed by location
+ mesh_location_dimnames[location] = dim_name
+
+ if cube is None:
+ cube_mesh_dim = None
+ else:
+ # Finally, identify the cube dimension which maps to the mesh: this
+ # is used below to recognise the mesh among the cube dimensions.
+ any_mesh_coord = cube.coords(mesh_coords=True)[0]
+ (cube_mesh_dim,) = any_mesh_coord.cube_dims(cube)
+
+ # Record actual file dimension names for each mesh saved.
+ self._mesh_dims[mesh] = mesh_location_dimnames
+
+ # Get the cube dimensions, in order.
+ cube_dimensions = []
+ if cube is not None:
+ for dim in range(cube.ndim):
+ if dim == cube_mesh_dim:
+ # Handle a mesh dimension: we already named this.
+ dim_coords = []
+ dim_name = self._mesh_dims[mesh][cube.location]
+ else:
+ # Get a name from the dim-coord (if any).
+ dim_coords = cube.coords(dimensions=dim, dim_coords=True)
+ if dim_coords:
+ # Derive a dim name from a coord.
+ coord = dim_coords[0] # always have at least one
+
+ # Add only dimensions that have not already been added.
+ dim_name = self._dim_names_and_coords.name(coord)
+ if dim_name is None:
+ # Not already present : create a unique dimension name
+ # from the coord.
+ dim_name = self._get_element_variable_name(cube, coord)
+ # Disambiguate if it has the same name as an
+ # existing dimension.
+ # OR if it matches an existing file variable name.
+ # NOTE: check against variable names is needed
+ # because of a netcdf bug ... see note in the
+ # mesh dimensions block above.
+ while (
+ dim_name in self._existing_dim
+ or dim_name in self._dataset.variables
+ ):
+ dim_name = self._increment_name(dim_name)
+
+ else:
+ # No CF-netCDF coordinates describe this data dimension.
+ # Make up a new, distinct dimension name
+ dim_name = f"dim{dim}"
+ # Increment name if conflicted with one already existing
+ # (or planned)
+ # NOTE: check against variable names is needed because
+ # of a netcdf bug ... see note in the mesh dimensions
+ # block above.
+ while (
+ dim_name in self._existing_dim
+ and (self._existing_dim[dim_name] != cube.shape[dim])
+ ) or dim_name in self._dataset.variables:
+ dim_name = self._increment_name(dim_name)
+
+ # Record the dimension.
+ record_dimension(cube_dimensions, dim_name, cube.shape[dim], dim_coords)
+
+ return mesh_dimensions, cube_dimensions
+
+ @staticmethod
+ def cf_valid_var_name(var_name):
+ """Return a valid CF var_name given a potentially invalid name.
+
+ Parameters
+ ----------
+ var_name : str
+ The var_name to normalise.
+
+ Returns
+ -------
+ str
+ The var_name suitable for passing through for variable creation.
+
+ """
+ # Replace invalid characters with an underscore ("_").
+ var_name = re.sub(r"[^a-zA-Z0-9]", "_", var_name)
+ # Ensure the variable name starts with a letter.
+ if re.match(r"^[^a-zA-Z]", var_name):
+ var_name = "var_{}".format(var_name)
+ return var_name
+
+ @staticmethod
+ def _cf_coord_standardised_units(coord):
+ """Determine a suitable units from a given coordinate.
+
+ Parameters
+ ----------
+ coord : :class:`iris.coords.Coord`
+ A coordinate of a cube.
+
+ Returns
+ -------
+ units
+ The (standard_name, long_name, unit) of the given
+ :class:`iris.coords.Coord` instance.
+ """
+ units = str(coord.units)
+ # Set the 'units' of 'latitude' and 'longitude' coordinates specified
+ # in 'degrees' to 'degrees_north' and 'degrees_east' respectively,
+ # as defined in the CF conventions for netCDF files: sections 4.1 and
+ # 4.2.
+ if (
+ isinstance(coord.coord_system, iris.coord_systems.GeogCS)
+ or coord.coord_system is None
+ ) and coord.units == "degrees":
+ if coord.standard_name == "latitude":
+ units = "degrees_north"
+ elif coord.standard_name == "longitude":
+ units = "degrees_east"
+
+ return units
+
+ def _ensure_valid_dtype(self, values, src_name, src_object):
+ # Some formats have no 64-bit or unsigned integer types, so we check
+ # whether we can store them as int32 instead. Which dtypes are worth
+ # narrowing is the saver's question; whether this store holds them is
+ # the store's.
+ is_narrowable_integer = np.issubdtype(values.dtype, np.int64) or np.issubdtype(
+ values.dtype, np.unsignedinteger
+ )
+ if is_narrowable_integer and not self._dataset.supports_dtype(values.dtype):
+ val_min, val_max = (values.min(), values.max())
+ if is_lazy_data(values):
+ val_min, val_max = _co_realise_lazy_arrays([val_min, val_max])
+ # NumPy will inherit values.dtype even if the scalar numbers work
+ # with a smaller type.
+ min_dtype = np.promote_types(
+ *[np.min_scalar_type(m) for m in (val_min, val_max)]
+ )
+ # Cast to an integer type the format supports.
+ if not np.can_cast(min_dtype, np.int32):
+ msg = (
+ "The data type of {} {!r} is not supported by {} and"
+ " its values cannot be safely cast to a supported"
+ " integer type."
+ )
+ msg = msg.format(src_name, src_object, self._dataset.format)
+ raise ValueError(msg)
+ values = values.astype(np.int32)
+ return values
+
+ def _create_cf_bounds(self, coord, cf_var, cf_name, /, *, compression_kwargs=None):
+ """Create the associated CF-netCDF bounds variable.
+
+ Parameters
+ ----------
+ coord : :class:`iris.coords.Coord`
+ A coordinate of a cube.
+ cf_var :
+ CF-netCDF variable.
+ cf_name : str
+ Name of the CF-NetCDF variable.
+ compression_kwargs : dict, optional
+ NetCDF data compression keyword arguments.
+
+ Returns
+ -------
+ None
+
+ """
+ if hasattr(coord, "has_bounds") and coord.has_bounds():
+ if compression_kwargs is None:
+ compression_kwargs = {}
+
+ # Get the values in a form which is valid for the file format.
+ bounds = self._ensure_valid_dtype(
+ coord.core_bounds(), "the bounds of coordinate", coord
+ )
+ n_bounds = bounds.shape[-1]
+
+ if n_bounds == 2:
+ bounds_dimension_name = "bnds"
+ else:
+ bounds_dimension_name = "bnds_%s" % n_bounds
+
+ if coord.climatological:
+ property_name = "climatology"
+ varname_extra = "climatology"
+ else:
+ property_name = "bounds"
+ varname_extra = "bnds"
+
+ if bounds_dimension_name not in self._dataset.dimensions:
+ # Create the bounds dimension with the appropriate extent.
+ while bounds_dimension_name in self._dataset.variables:
+ # Also avoid collision with variable names.
+ # See '_get_dim_names' for reason.
+ bounds_dimension_name = self._increment_name(bounds_dimension_name)
+ self._dataset.create_dimension(bounds_dimension_name, n_bounds)
+
+ boundsvar_name = "{}_{}".format(cf_name, varname_extra)
+ cf_var.attributes[property_name] = boundsvar_name
+ cf_var_bounds = self._dataset.create_variable(
+ boundsvar_name,
+ bounds.dtype.newbyteorder("="),
+ cf_var.dimensions + (bounds_dimension_name,),
+ **compression_kwargs,
+ )
+ self._lazy_stream_data(data=bounds, cf_var=cf_var_bounds)
+
+ def _create_string_variable(
+ self, cube_or_mesh, cf_name, element, element_dims, data
+ ):
+ """Create the variable that will hold this unicode string data.
+
+ Parameters
+ ----------
+ cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
+ The Cube or Mesh being saved.
+ cf_name : str
+ The name the variable is created with.
+ element : :class:`iris.coords._DimensionalMetadata`
+ The element being saved.
+ element_dims : list of str
+ The variable's dimension names. A subclass may append to it.
+ data : :class:`numpy.ndarray`
+ The string values, used for their dtype.
+
+ Returns
+ -------
+ :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+
+ Raises
+ ------
+ NotImplementedError
+ Always, here. How a store holds a string differs so much between
+ formats that there is no generic answer to give.
+
+ """
+ msg = (
+ f"Cannot save the string variable {cf_name!r}: "
+ f"{type(self).__name__} does not say how {self._dataset.format} "
+ "stores strings."
+ )
+ raise NotImplementedError(msg)
+
+ def _get_element_variable_name(
+ self, cube_or_mesh: "Cube|MeshXY", element: "_DimensionalMetadata|Cube"
+ ) -> str:
+ # keep it as _DimensionalMetadata
+ # Don't want it as CFVariableMixin - implies a cube can be passed in
+ # Could be Coord AncillaryVariable Cellmeasure
+ """Return a CF-netCDF variable name for a given coordinate-like element, or cube.
+
+ Parameters
+ ----------
+ cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
+ The Cube or Mesh being saved to the netCDF file.
+ element : :class:`iris.coords._DimensionalMetadata` | :class:``iris.cube.Cube``
+ An instance of a coordinate (or similar), for which a CF-netCDF
+ variable name is required.
+
+ Returns
+ -------
+ str
+ A CF-netCDF variable name as a string.
+
+ """
+ # Support cube or mesh save.
+ from iris.cube import Cube
+
+ if isinstance(cube_or_mesh, Cube):
+ cube = cube_or_mesh
+ mesh = cube.mesh
+ else:
+ cube = None
+ mesh = cube_or_mesh
+
+ if element.var_name is not None:
+ cf_name = element.var_name
+ elif isinstance(element, Cube):
+ # Make name for a Cube without a var_name.
+ cf_name = "_".join(element.name().lower().split())
+ else:
+ # Make name for a Coord-like element without a var_name
+ name = element.standard_name or element.long_name
+ if not name or set(name).intersection(string.whitespace):
+ # We need to invent a name, based on its associated dimensions.
+ if cube is not None and cube.coords(element):
+ # It is a regular cube coordinate.
+ # Auto-generate a name based on the dims.
+ name = ""
+ for dim in cube.coord_dims(element):
+ name += f"dim{dim}"
+ # Handle scalar coordinate (dims == ()).
+ if not name:
+ name = "unknown_scalar"
+ else:
+ # Not a cube coord, so must be a connectivity or
+ # element-coordinate of the mesh.
+ # Name it for it's first dim, i.e. mesh-dim of its location.
+
+ from iris.mesh import Connectivity
+
+ if isinstance(element, Connectivity):
+ location = element.cf_role.split("_")[0]
+ location_dim_attr = f"{location}_dimension"
+ name = getattr(mesh, location_dim_attr)
+ else:
+ # 'coord' is a nameless coordinate of a mesh,
+ # so we set it as unknown for saving
+ name = "unknown"
+
+ # Convert to lower case and replace whitespace by underscores.
+ cf_name = "_".join(name.lower().split())
+
+ cf_name = self.cf_valid_var_name(cf_name)
+ return cf_name
+
+ def _get_mesh_variable_name(self, mesh):
+ """Return a CF-netCDF variable name for the mesh.
+
+ Parameters
+ ----------
+ mesh : :class:`iris.mesh.MeshXY`
+ An instance of a Mesh for which a CF-netCDF variable name is
+ required.
+
+ Returns
+ -------
+ str
+ A CF-netCDF variable name as a string.
+
+ """
+ cf_name = mesh.var_name or mesh.long_name
+ # Prefer a var-name, but accept a long_name as an alternative.
+ # N.B. we believe it can't (shouldn't) have a standard name.
+ if not cf_name:
+ # Auto-generate a name based on mesh properties.
+ cf_name = f"Mesh_{mesh.topology_dimension}d"
+
+ # Ensure valid form for var-name.
+ cf_name = self.cf_valid_var_name(cf_name)
+ return cf_name
+
+ def _create_mesh(self, mesh):
+ """Create a mesh variable in the netCDF dataset.
+
+ Parameters
+ ----------
+ mesh : :class:`iris.mesh.MeshXY`
+ The Mesh to be saved to CF-netCDF file.
+
+ Returns
+ -------
+ str
+ The string name of the associated CF-netCDF variable saved.
+
+ """
+ # First choose a var-name for the mesh variable itself.
+ cf_mesh_name = self._get_mesh_variable_name(mesh)
+ # Disambiguate any possible clashes.
+ while cf_mesh_name in self._dataset.variables:
+ cf_mesh_name = self._increment_name(cf_mesh_name)
+
+ # Create the main variable
+ cf_mesh_var = self._dataset.create_variable(
+ cf_mesh_name,
+ np.dtype(np.int32),
+ )
+
+ # Add the basic essential attributes
+ cf_mesh_var.attributes["cf_role"] = "mesh_topology"
+ cf_mesh_var.attributes["topology_dimension"] = np.int32(mesh.topology_dimension)
+ # Add the usual names + units attributes
+ self._set_cf_var_attributes(cf_mesh_var, mesh)
+
+ return cf_mesh_name
+
+ def _set_cf_var_attributes(self, cf_var, element):
+ from iris.cube import Cube
+
+ # Deal with CF-netCDF units, and add the name+units properties.
+ if isinstance(element, iris.coords.Coord):
+ # Fix "degree" units if needed.
+ units_str = self._cf_coord_standardised_units(element)
+ else:
+ units_str = str(element.units)
+
+ # NB this bit is a nasty hack to preserve existing behaviour through a refactor:
+ # The attributes for Coords are created in the order units, standard_name,
+ # whereas for data-variables (aka Cubes) it is the other way around.
+ # Needed now that this routine is also called from _create_cf_data_variable.
+ # TODO: when we can break things, rationalise these to be the same.
+ def add_units_attr():
+ if cf_units.as_unit(units_str).is_udunits():
+ cf_var.attributes["units"] = units_str
+
+ def add_names_attrs():
+ standard_name = element.standard_name
+ if standard_name is not None:
+ cf_var.attributes["standard_name"] = standard_name
+
+ long_name = element.long_name
+ if long_name is not None:
+ cf_var.attributes["long_name"] = long_name
+
+ if isinstance(element, Cube):
+ add_names_attrs()
+ add_units_attr()
+ else:
+ add_units_attr()
+ add_names_attrs()
+
+ # Add the CF-netCDF calendar attribute.
+ if element.units.calendar:
+ cf_var.attributes["calendar"] = str(element.units.calendar)
+
+ # Take a copy so we can remove things
+ element_attrs = element.attributes.copy()
+
+ # Note: when writing UGRID, "element" can be a Mesh which has no "dtype",
+ # and for dataless cubes it will have a 'None' dtype.
+ if getattr(element, "dtype", None) is not None:
+ # Most attributes are dealt with later. But _Encoding needs to be defined
+ # *before* we can write to a character variable.
+ if element.dtype.kind in "SU" and "_Encoding" in element_attrs:
+ encoding = element_attrs.pop("_Encoding")
+ cf_var.attributes["_Encoding"] = encoding
+
+ if not isinstance(element, Cube):
+ # Add any other custom coordinate attributes.
+ # N.B. not Cube, which has specific handling in _create_cf_data_variable
+ for name in sorted(element_attrs):
+ value = element_attrs[name]
+
+ if name == "STASH":
+ # Adopting provisional Metadata Conventions for representing MO
+ # Scientific Data encoded in NetCDF Format.
+ name = "um_stash_source"
+ value = str(value)
+
+ # Don't clobber existing attributes.
+ if name not in cf_var.attributes:
+ cf_var.attributes[name] = value
+
+ def _create_generic_cf_array_var(
+ self,
+ cube_or_mesh,
+ cube_dim_names,
+ element,
+ /,
+ *,
+ element_dims=None,
+ fill_value=None,
+ compression_kwargs=None,
+ packing_controls: dict | None = None,
+ is_dataless=False,
+ ):
+ """Create theCF-netCDF variable given dimensional_metadata.
+
+ Create the associated CF-netCDF variable in the netCDF dataset for the
+ given dimensional_metadata.
+
+ .. note::
+ If the metadata element is a coord, it may also contain bounds.
+ In which case, an additional var is created and linked to it.
+
+ Parameters
+ ----------
+ cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
+ The Cube or Mesh being saved to the netCDF file.
+ cube_dim_names : list of str
+ The name of each dimension of the cube.
+ element : :class:`iris.coords._DimensionalMetadata`
+ An Iris :class:`iris.coords._DimensionalMetadata`, belonging to the
+ cube. Provides data, units and standard/long/var names.
+ Not used if 'element_dims' is not None.
+ element_dims : list of str, optional
+ If set, contains the variable dimension (names),
+ otherwise these are taken from `element.cube_dims[cube]`.
+ For Mesh components (element coordinates and connectivities), this
+ *must* be passed in, as "element.cube_dims" does not function.
+ fill_value : number, optional
+ If set, create the variable with this fill-value, and fill any
+ masked data points with this value.
+ If not set, standard netcdf4-python behaviour : the variable has no
+ '_FillValue' property, and uses the "standard" fill-value for its
+ type.
+ compression_kwargs : dict, optional
+ NetCDF data compression keyword arguments.
+
+ Returns
+ -------
+ str
+ The name of the CF-netCDF variable created.
+ """
+ # Support cube or mesh save.
+ from iris.cube import Cube
+
+ if isinstance(cube_or_mesh, Cube):
+ cube = cube_or_mesh
+ else:
+ cube = None
+
+ if compression_kwargs is None:
+ compression_kwargs = {}
+
+ # Work out the var-name to use.
+ # N.B. the only part of this routine that may use a mesh _or_ a cube.
+ cf_name = self._get_element_variable_name(cube_or_mesh, element)
+ while cf_name in self._dataset.variables:
+ cf_name = self._increment_name(cf_name)
+
+ if cube and element_dims is None:
+ # Get the list of file-dimensions (names), to create the variable.
+ element_dims = [
+ cube_dim_names[dim] for dim in element.cube_dims(cube)
+ ] # NB using 'cube_dims' as this works for any type of element
+
+ # Get the data values, in a way which works for any element type, as
+ # all are subclasses of _DimensionalMetadata.
+ # (e.g. =points if a coord, =data if an ancillary, etc)
+ if isinstance(element, Cube):
+ data = element.core_data()
+ else:
+ data = element._core_values()
+
+ # This compression contract is *not* applicable to a mesh.
+ if cube is not None and data is not None and cube.shape != data.shape:
+ compression_kwargs = {}
+
+ if not is_dataless and data.dtype.kind == "U":
+ # Unicode string data: typically a CF label variable, sometimes an
+ # ancillary variable. How a store holds strings is the store's own
+ # business, so the backend creates the variable.
+ cf_var = self._create_string_variable(
+ cube_or_mesh, cf_name, element, element_dims, data
+ )
+ else:
+ # A non-string variable.
+ # ensure a valid datatype for the file format.
+ if is_dataless:
+ dtype = self._DATALESS_DTYPE
+ fill_value = self._DATALESS_FILLVALUE
+ else:
+ # Normal non-string data.
+ # NOTE: this includes byte-arrays (S1 only) : however these must
+ # use an actual cube dimension for the 'string dimension', which
+ # seriously limits the utility of DECODE_TO_STRINGS_ON_READ.
+ # TODO: also support netCDF variable-length strings ("string" type).
+ # Currently hit a **write error here**, being numpy object dtype ("O").
+ if data.dtype.kind not in "iufSU" or (
+ data.dtype.kind == "S" and data.dtype.itemsize != 1
+ ):
+ # This is a type of data we don't "understand".
+ # NB this includes "Sxx" types other than "S1" : It seems that
+ # netCDF4 saves Sxx as variable-length strings.
+ # But we don't support that type in Iris.
+ msg = (
+ f"Variable {cf_name!r} has unexpected dtype, {data.dtype!r}."
+ f"Data content arrays must be numeric, or contain "
+ "single-bytes (dtype 'S1'), or unicode strings (dtype 'U<n>')."
+ )
+ raise ValueError(msg)
+
+ element_type = type(element).__name__
+ data = self._ensure_valid_dtype(data, element_type, element)
+ if not packing_controls:
+ dtype = data.dtype.newbyteorder("=")
+ else:
+ dtype = packing_controls["dtype"]
+
+ # Check if this is a dim-coord.
+ is_dimcoord = cube is not None and element in cube.dim_coords
+
+ if is_dimcoord:
+ # By definition of a CF-netCDF coordinate variable this
+ # coordinate must be 1-D and the name of the CF-netCDF variable
+ # must be the same as its dimension name.
+ cf_name = element_dims[0]
+
+ # Create the CF-netCDF variable.
+ cf_var = self._dataset.create_variable(
+ cf_name,
+ dtype,
+ element_dims,
+ fill_value=fill_value,
+ **compression_kwargs,
+ )
+
+ # Add the axis attribute for spatio-temporal CF-netCDF coordinates.
+ if is_dimcoord:
+ axis = iris.util.guess_coord_axis(element)
+ if axis is not None and axis.lower() in SPATIO_TEMPORAL_AXES:
+ cf_var.attributes["axis"] = axis.upper()
+
+ # Create the associated CF-netCDF bounds variable, if any.
+ self._create_cf_bounds(
+ element, cf_var, cf_name, compression_kwargs=compression_kwargs
+ )
+
+ # Add names + units
+ # NOTE: *must* now do first, as we may need '_Encoding' set to write it !
+ self._set_cf_var_attributes(cf_var, element)
+
+ # Add the data to the CF-netCDF variable.
+ if not is_dataless:
+ if packing_controls:
+ # We must set packing attributes (if any), before assigning values.
+ for key, value in packing_controls["attributes"]:
+ cf_var.attributes[key] = value
+ self._lazy_stream_data(data=data, cf_var=cf_var)
+
+ return cf_name
+
+ def _create_cf_cell_methods(self, cube, dimension_names):
+ """Create CF-netCDF string representation of a cube cell methods.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube` or :class:`iris.cube.CubeList`
+ A :class:`iris.cube.Cube`, :class:`iris.cube.CubeList` or list of
+ cubes to be saved to a netCDF file.
+ dimension_names : list
+ Names associated with the dimensions of the cube.
+
+ Returns
+ -------
+ str
+ CF-netCDF string representation of a cube cell methods.
+
+ """
+ cell_methods = []
+
+ # Identify the collection of coordinates that represent CF-netCDF
+ # coordinate variables.
+ cf_coordinates = cube.dim_coords
+
+ for cm in cube.cell_methods:
+ names = ""
+
+ for name in cm.coord_names:
+ coord = cube.coords(name)
+
+ if coord:
+ coord = coord[0]
+ if coord in cf_coordinates:
+ name = dimension_names[cube.coord_dims(coord)[0]]
+
+ names += "%s: " % name
+
+ interval = " ".join(
+ ["interval: %s" % interval for interval in cm.intervals or []]
+ )
+ comment = " ".join(
+ ["comment: %s" % comment for comment in cm.comments or []]
+ )
+ extra = " ".join([interval, comment]).strip()
+
+ if extra:
+ extra = " (%s)" % extra
+
+ cell_methods.append(names + cm.method + extra)
+
+ return " ".join(cell_methods)
+
+ def _add_grid_mapping_to_dataset(self, cs, extended_grid_mapping=False):
+ """Create a CF-netCDF grid mapping variable and add to the dataset.
+
+ Parameters
+ ----------
+ cs : :class:`iris.coord_system.CoordSystem`
+ The :class:`iris.coord_system.CoordSystem` used to generate the
+ new netCDF CF grid mapping variable.
+
+ Returns
+ -------
+ None
+ """
+ cf_var_grid = self._dataset.create_variable(cs.grid_mapping_name, np.int32)
+ cf_var_grid.attributes["grid_mapping_name"] = cs.grid_mapping_name
+
+ # Assignments to properties of ``grid_variable`` bypass the ASCII-to-bytes
+ # coercion that .attributes applies, so moving them there would change
+ # the file. They need the netCDF4 variable itself. Not
+ # deprecated_netcdf_member(), so a store with no netCDF4 object behind it
+ # raises a bare AttributeError here.
+ grid_variable = cf_var_grid.variable
+
+ def add_ellipsoid(ellipsoid):
+ grid_variable.longitude_of_prime_meridian = (
+ ellipsoid.longitude_of_prime_meridian
+ )
+ semi_major = ellipsoid.semi_major_axis
+ semi_minor = ellipsoid.semi_minor_axis
+ if semi_minor == semi_major:
+ grid_variable.earth_radius = semi_major
+ else:
+ grid_variable.semi_major_axis = semi_major
+ grid_variable.semi_minor_axis = semi_minor
+ if ellipsoid.datum is not None:
+ grid_variable.horizontal_datum_name = ellipsoid.datum
+
+ # latlon
+ if isinstance(cs, iris.coord_systems.GeogCS):
+ add_ellipsoid(cs)
+
+ # rotated latlon
+ elif isinstance(cs, iris.coord_systems.RotatedGeogCS):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.grid_north_pole_latitude = cs.grid_north_pole_latitude
+ grid_variable.grid_north_pole_longitude = cs.grid_north_pole_longitude
+ grid_variable.north_pole_grid_longitude = cs.north_pole_grid_longitude
+
+ # tmerc
+ elif isinstance(cs, iris.coord_systems.TransverseMercator):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.longitude_of_central_meridian = (
+ cs.longitude_of_central_meridian
+ )
+ grid_variable.latitude_of_projection_origin = (
+ cs.latitude_of_projection_origin
+ )
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+ grid_variable.scale_factor_at_central_meridian = (
+ cs.scale_factor_at_central_meridian
+ )
+
+ # merc
+ elif isinstance(cs, iris.coord_systems.Mercator):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.longitude_of_projection_origin = (
+ cs.longitude_of_projection_origin
+ )
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+ # Only one of these should be set
+ if cs.standard_parallel is not None:
+ grid_variable.standard_parallel = cs.standard_parallel
+ elif cs.scale_factor_at_projection_origin is not None:
+ grid_variable.scale_factor_at_projection_origin = (
+ cs.scale_factor_at_projection_origin
+ )
+
+ # lcc
+ elif isinstance(cs, iris.coord_systems.LambertConformal):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.standard_parallel = cs.secant_latitudes
+ grid_variable.latitude_of_projection_origin = cs.central_lat
+ grid_variable.longitude_of_central_meridian = cs.central_lon
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+
+ # polar stereo (have to do this before Stereographic because it subclasses it)
+ elif isinstance(cs, iris.coord_systems.PolarStereographic):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.latitude_of_projection_origin = cs.central_lat
+ grid_variable.straight_vertical_longitude_from_pole = cs.central_lon
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+ # Only one of these should be set
+ if cs.true_scale_lat is not None:
+ grid_variable.true_scale_lat = cs.true_scale_lat
+ elif cs.scale_factor_at_projection_origin is not None:
+ grid_variable.scale_factor_at_projection_origin = (
+ cs.scale_factor_at_projection_origin
+ )
+ else:
+ grid_variable.scale_factor_at_projection_origin = 1.0
+
+ # stereo
+ elif isinstance(cs, iris.coord_systems.Stereographic):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.longitude_of_projection_origin = cs.central_lon
+ grid_variable.latitude_of_projection_origin = cs.central_lat
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+ # Only one of these should be set
+ if cs.true_scale_lat is not None:
+ msg = (
+ "It is not valid CF to save a true_scale_lat for "
+ "a Stereographic grid mapping."
+ )
+ raise ValueError(msg)
+ elif cs.scale_factor_at_projection_origin is not None:
+ grid_variable.scale_factor_at_projection_origin = (
+ cs.scale_factor_at_projection_origin
+ )
+ else:
+ grid_variable.scale_factor_at_projection_origin = 1.0
+
+ # osgb (a specific tmerc)
+ elif isinstance(cs, iris.coord_systems.OSGB):
+ warnings.warn(
+ "OSGB coordinate system not yet handled",
+ category=iris.warnings.IrisSaveWarning,
+ )
+
+ # lambert azimuthal equal area
+ elif isinstance(cs, iris.coord_systems.LambertAzimuthalEqualArea):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.longitude_of_projection_origin = (
+ cs.longitude_of_projection_origin
+ )
+ grid_variable.latitude_of_projection_origin = (
+ cs.latitude_of_projection_origin
+ )
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+
+ # albers conical equal area
+ elif isinstance(cs, iris.coord_systems.AlbersEqualArea):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.longitude_of_central_meridian = (
+ cs.longitude_of_central_meridian
+ )
+ grid_variable.latitude_of_projection_origin = (
+ cs.latitude_of_projection_origin
+ )
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+ grid_variable.standard_parallel = cs.standard_parallels
+
+ # vertical perspective
+ elif isinstance(cs, iris.coord_systems.VerticalPerspective):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.longitude_of_projection_origin = (
+ cs.longitude_of_projection_origin
+ )
+ grid_variable.latitude_of_projection_origin = (
+ cs.latitude_of_projection_origin
+ )
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+ grid_variable.perspective_point_height = cs.perspective_point_height
+
+ # geostationary
+ elif isinstance(cs, iris.coord_systems.Geostationary):
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.longitude_of_projection_origin = (
+ cs.longitude_of_projection_origin
+ )
+ grid_variable.latitude_of_projection_origin = (
+ cs.latitude_of_projection_origin
+ )
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+ grid_variable.perspective_point_height = cs.perspective_point_height
+ grid_variable.sweep_angle_axis = cs.sweep_angle_axis
+
+ # oblique mercator (and rotated variant)
+ # Use duck-typing over isinstance() - subclasses (i.e.
+ # RotatedMercator) upset mock tests.
+ elif getattr(cs, "grid_mapping_name", None) == "oblique_mercator":
+ # RotatedMercator subclasses ObliqueMercator, and RM
+ # instances are implicitly saved as OM due to inherited
+ # properties. This is correct because CF 1.11 is removing
+ # all mention of RM.
+ if cs.ellipsoid:
+ add_ellipsoid(cs.ellipsoid)
+ grid_variable.azimuth_of_central_line = cs.azimuth_of_central_line
+ grid_variable.latitude_of_projection_origin = (
+ cs.latitude_of_projection_origin
+ )
+ grid_variable.longitude_of_projection_origin = (
+ cs.longitude_of_projection_origin
+ )
+ grid_variable.false_easting = cs.false_easting
+ grid_variable.false_northing = cs.false_northing
+ grid_variable.scale_factor_at_projection_origin = (
+ cs.scale_factor_at_projection_origin
+ )
+
+ # other
+ else:
+ warnings.warn(
+ "Unable to represent the horizontal "
+ "coordinate system. The coordinate system "
+ "type %r is not yet implemented." % type(cs),
+ category=iris.warnings.IrisSaveWarning,
+ )
+
+ # add WKT string
+ if extended_grid_mapping:
+ grid_variable.crs_wkt = cs.as_cartopy_crs().to_wkt()
+
+ def _create_cf_grid_mapping(self, cube, cf_var_cube):
+ """Create CF-netCDF grid mapping and associated CF-netCDF variable.
+
+ Create CF-netCDF grid mapping variable and associated CF-netCDF
+ data variable grid mapping attribute.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube` or :class:`iris.cube.CubeList`
+ A :class:`iris.cube.Cube`, :class:`iris.cube.CubeList` or list of
+ cubes to be saved to a netCDF file.
+ cf_var_cube : :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`
+ A cf variable cube representation.
+
+ Returns
+ -------
+ None
+
+ """
+ coord_systems = []
+ if cube.extended_grid_mapping:
+ # get unique list of all coord_systems on cube coords:
+ for coord in cube.coords():
+ if coord.coord_system and coord.coord_system not in coord_systems:
+ coord_systems.append(coord.coord_system)
+ else:
+ # will only return a single coord system (if one exists):
+ if cs := cube.coord_system("CoordSystem"):
+ coord_systems.append(cs)
+
+ grid_mappings = {}
+ matched_all_coords = True
+
+ for cs in coord_systems:
+ # Grid var not yet created?
+ if cs not in self._coord_systems:
+ # handle potential duplicate netCDF variable names:
+ while cs.grid_mapping_name in self._dataset.variables:
+ aname = self._increment_name(cs.grid_mapping_name)
+ cs.grid_mapping_name = aname
+
+ # create grid mapping variable on dataset for this coordinate system:
+ self._add_grid_mapping_to_dataset(
+ cs, extended_grid_mapping=cube.extended_grid_mapping
+ )
+ self._coord_systems.append(cs)
+
+ # create the `grid_mapping` attribute for the data variable:
+ if cube.extended_grid_mapping:
+ # Order the coordinates as per the order in the CRS/WKT string.
+ # (We should only ever have a coordinate system for horizontal
+ # spatial coords, so check for east/north directions)
+ ordered_coords = []
+ for ax_info in cs.as_cartopy_crs().axis_info:
+ try:
+ match ax_info.direction.lower():
+ case "east":
+ ordered_coords.append(
+ cube.coord(axis="X", coord_system=cs)
+ )
+ case "north":
+ ordered_coords.append(
+ cube.coord(axis="Y", coord_system=cs)
+ )
+ case _:
+ msg = (
+ f"Can't handle axis direction {ax_info.direction!r}"
+ )
+ raise NotImplementedError(msg)
+ except iris.exceptions.CoordinateNotFoundError as e:
+ msg = (
+ f"Failed to assign coordinate for {ax_info.name} axis of "
+ f"coordinate system {cs.grid_mapping_name}: {str(e)}"
+ )
+ warnings.warn(msg, category=iris.warnings.IrisSaveWarning)
+
+ # fall back to simple grid mapping (single crs entry for DimCoord only)
+ matched_all_coords = False
+
+ # Get variable names, and store them if necessary:
+ cfvar_names = []
+ for coord in ordered_coords:
+ cfvar = self._name_coord_map.name(coord)
+ if not cfvar:
+ # not found - create and store it:
+ cfvar = self._get_element_variable_name(cube, coord)
+ self._name_coord_map.append(
+ cfvar, self._get_element_variable_name(cube, coord)
+ )
+ cfvar_names.append(cfvar)
+
+ coord_string = " ".join(cfvar_names)
+ grid_mappings[cs.grid_mapping_name] = coord_string
+
+ # Refer to grid var
+ if len(coord_systems):
+ grid_mapping = None
+ if cube.extended_grid_mapping:
+ if matched_all_coords:
+ grid_mapping = " ".join(
+ f"{cs_name}: {cs_coords}"
+ for cs_name, cs_coords in grid_mappings.items()
+ )
+ else:
+ # We didn't match all coords required for grid mapping, default to
+ # single coordinate system for DimCoords, if set.
+ for ax in ["x", "y"]:
+ try:
+ if dim_cs := cube.coord(
+ axis=ax, dim_coords=True
+ ).coord_system:
+ grid_mapping = dim_cs.grid_mapping_name
+ break
+ except iris.exceptions.CoordinateNotFoundError:
+ pass
+ else:
+ # Not using extended_grid_mapping: only ever be on cs in this case:
+ grid_mapping = coord_systems[0].grid_mapping_name
+
+ if grid_mapping:
+ cf_var_cube.attributes["grid_mapping"] = grid_mapping
+
+ _DATALESS_ATTRNAME = "iris_dataless_cube"
+ _DATALESS_DTYPE = np.dtype("u1")
+ _DATALESS_FILLVALUE = 127
+
+ def _create_cf_data_variable(
+ self,
+ cube,
+ dimension_names,
+ local_keys=None,
+ packing=None,
+ fill_value=None,
+ **kwargs,
+ ):
+ """Create CF-netCDF data variable for the cube and any associated grid mapping.
+
+ # TODO: when iris.FUTURE.save_split_attrs is removed, the 'local_keys' arg can
+ # be removed.
+
+ Parameters
+ ----------
+ cube : :class:`iris.cube.Cube`
+ The associated cube being saved to CF-netCDF file.
+ dimension_names : list
+ String names for each dimension of the cube.
+ local_keys : iterable of str, optional
+ See :func:`iris.fileformats.netcdf.Saver.write`.
+ packing : type or str or dict or list, optional
+ See :func:`iris.fileformats.netcdf.Saver.write`.
+ fill_value : optional
+ See :func:`iris.fileformats.netcdf.Saver.write`.
+
+ Returns
+ -------
+ The newly created CF-netCDF data variable.
+
+ Notes
+ -----
+ All other keywords are passed through to the dataset's `createVariable`
+ method.
+
+ """
+ # TODO: when iris.FUTURE.save_split_attrs is removed, the 'local_keys' arg can
+ # be removed.
+ # Get the values in a form which is valid for the file format.
+ is_dataless = cube.is_dataless()
+
+ packing_controls = None
+ if packing and not is_dataless:
+ data = self._ensure_valid_dtype(cube.core_data(), "cube", cube)
+ if isinstance(packing, dict):
+ if "dtype" not in packing:
+ msg = "The dtype attribute is required for packing."
+ raise ValueError(msg)
+ dtype = np.dtype(packing["dtype"])
+ scale_factor = packing.get("scale_factor", None)
+ add_offset = packing.get("add_offset", None)
+ valid_keys = {"dtype", "scale_factor", "add_offset"}
+ invalid_keys = set(packing.keys()) - valid_keys
+ if invalid_keys:
+ msg = (
+ "Invalid packing key(s) found: '{}'. The valid "
+ "keys are '{}'.".format(
+ "', '".join(invalid_keys), "', '".join(valid_keys)
+ )
+ )
+ raise ValueError(msg)
+ else:
+ # We compute the scale_factor and add_offset based on the
+ # min/max of the data.
+ masked = iris.util.is_masked(data)
+ dtype = np.dtype(packing)
+ cmin, cmax = (data.min(), data.max())
+ if is_lazy_data(data):
+ cmin, cmax = _co_realise_lazy_arrays([cmin, cmax])
+ n = dtype.itemsize * 8
+ if masked:
+ scale_factor = (cmax - cmin) / (2**n - 2)
+ else:
+ scale_factor = (cmax - cmin) / (2**n - 1)
+ if dtype.kind == "u":
+ add_offset = cmin
+ elif dtype.kind == "i":
+ if masked:
+ add_offset = (cmax + cmin) / 2
+ else:
+ add_offset = cmin + 2 ** (n - 1) * scale_factor
+
+ packing_controls = {
+ "dtype": dtype,
+ "attributes": [
+ ("scale_factor", scale_factor),
+ ("add_offset", add_offset),
+ ],
+ }
+
+ # Create the cube CF-netCDF data variable with data payload.
+ cf_name = self._create_generic_cf_array_var(
+ cube,
+ dimension_names,
+ cube,
+ element_dims=dimension_names,
+ fill_value=fill_value,
+ compression_kwargs=kwargs,
+ packing_controls=packing_controls,
+ is_dataless=is_dataless,
+ )
+ cf_var = self._dataset.variables[cf_name]
+
+ # Set general attrs: NB this part is cube-specific (not the same for components)
+ # - so 'set_cf_var_attributes' *doesn't* set these, if element is a Cube
+ if iris.FUTURE.save_split_attrs:
+ attr_names = cube.attributes.locals.keys()
+ else:
+ # Add data variable-only attribute names to local_keys.
+ if local_keys is None:
+ local_keys = set()
+ else:
+ local_keys = set(local_keys)
+ local_keys.update(_CF_DATA_ATTRS, _UKMO_DATA_ATTRS, ["GRIB_PARAM"])
+
+ # Add any cube attributes whose keys are in local_keys as
+ # CF-netCDF data variable attributes.
+ attr_names = set(cube.attributes).intersection(local_keys)
+
+ # Write all the variable attributes
+ for attr_name in sorted(attr_names):
+ # Do not output 'conventions' or extended grid mapping attribute.
+ if attr_name.lower() in ["conventions", "iris_extended_grid_mapping"]:
+ continue
+
+ value = cube.attributes[attr_name]
+
+ # Convert any "managed" attributes. These convert between an internal
+ # convenience representation and what is actually stored in the file.
+ handler = ATTRIBUTE_HANDLERS.get(attr_name)
+ if handler is not None:
+ try:
+ attr_name, value = handler.encode_object(value)
+ except (TypeError, ValueError):
+ string_value = str(value)
+ msg = (
+ f"Invalid value in managed '{attr_name}' attribute: {value!r}. "
+ f"File attribute set to raw (string) value {string_value!r}."
+ )
+ warnings.warn(msg, category=iris.warnings.IrisSaveWarning)
+ # Resolve by setting the **Iris** attr-name to the "raw" string repr
+ # NB we expect the default written name to be the last listed
+ attr_name = handler.iris_name
+ value = string_value
+
+ if attr_name in _CF_GLOBAL_ATTRS:
+ msg = (
+ "{attr_name!r} is being added as CF data variable "
+ "attribute, but {attr_name!r} should only be a CF "
+ "global attribute.".format(attr_name=attr_name)
+ )
+ warnings.warn(msg, category=iris.warnings.IrisCfSaveWarning)
+
+ cf_var.attributes[attr_name] = value
+
+ # Add the 'dataless' marker if needed
+ if is_dataless:
+ cf_var.attributes[self._DATALESS_ATTRNAME] = "true"
+
+ # Create the CF-netCDF data variable cell method attribute.
+ cell_methods = self._create_cf_cell_methods(cube, dimension_names)
+
+ if cell_methods:
+ cf_var.attributes["cell_methods"] = cell_methods
+
+ # Create the CF-netCDF grid mapping.
+ self._create_cf_grid_mapping(cube, cf_var)
+
+ return cf_var
+
+ def _increment_name(self, varname):
+ """Increment string name or begin increment.
+
+ Avoidance of conflicts between variable names, where the name is
+ incremented to distinguish it from others.
+
+ Parameters
+ ----------
+ varname : str
+ Variable name to increment.
+
+ Returns
+ -------
+ Incremented varname.
+
+ """
+ num = 0
+ try:
+ name, endnum = varname.rsplit("_", 1)
+ if endnum.isdigit():
+ num = int(endnum) + 1
+ varname = name
+ except ValueError:
+ pass
+
+ return "{}_{}".format(varname, num)
+
+ def _backend_write(self, data, cf_var) -> bool:
+ """Write the data here and now, if this backend handles it itself.
+
+ Parameters
+ ----------
+ data : array-like
+ The values to write.
+ cf_var : :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
+ The variable to write them into.
+
+ Returns
+ -------
+ bool
+ Whether the data was written. ``False``, here: the direct and
+ delayed writes below cover every store with no special case.
+
+ """
+ return False
+
+ def _lazy_stream_data(
+ self,
+ data: np.typing.ArrayLike,
+ cf_var: CFDatasetVariable,
+ ) -> None:
+ if hasattr(data, "shape") and data.shape == (1,) + cf_var.shape:
+ # (Don't do this check for string data).
+ # Reduce dimensionality where the data array has an extra dimension
+ # versus the cf_var - to avoid a broadcasting ambiguity.
+ # Happens when bounds data is for a scalar point - array is 2D but
+ # contains just 1 row, so the cf_var is 1D.
+ data = data.squeeze(axis=0)
+
+ if self._backend_write(data, cf_var):
+ return
+
+ doing_delayed_save = is_lazy_data(data)
+ if doing_delayed_save:
+ # save lazy data with a delayed operation. For now, we just record the
+ # necessary information -- a single, complete delayed action is
+ # constructed later by a call to delayed_completion().
+ def store(
+ data: np.typing.ArrayLike,
+ cf_var: CFDatasetVariable,
+ ) -> None:
+ # TODO: recheck once Zarr behaviour is settled.
+ # Ask the variable for something a worker can stream into
+ # after this file is closed. What that is is the backend's
+ # business: netCDF reopens the file, Zarr will hand back the
+ # array itself.
+ self._delayed_writes.append((data, cf_var.write_handle()))
+
+ else:
+ # Real data is always written directly, i.e. not via lazy save.
+ def store(
+ data: np.typing.ArrayLike,
+ cf_var: CFDatasetVariable,
+ ) -> None:
+ cf_var[:] = data # type: ignore[index]
+
+ # Store the data.
+ store(data, cf_var)
+
+ def delayed_completion(self) -> Delayed:
+ """Perform file completion for delayed saves.
+
+ Create and return a :class:`dask.delayed.Delayed` to perform file
+ completion for delayed saves.
+
+ Returns
+ -------
+ :class:`dask.delayed.Delayed`
+
+ Notes
+ -----
+ The dataset *must* be closed (saver has exited its context) before the
+ result can be computed, otherwise computation will hang (never return).
+ """
+ if self._delayed_writes:
+ # Create a single delayed da.store operation to complete the file.
+ sources, targets = zip(*self._delayed_writes)
+ result = da.store(sources, targets, compute=False, lock=False)
+
+ else:
+ # Return a do-nothing delayed, for usage consistency.
+ @dask.delayed
+ def no_op():
+ return None
+
+ result = no_op()
+
+ return result
+
+ def complete(self) -> None:
+ """Complete file by computing any delayed variable saves.
+
+ This requires that the Saver has closed the dataset (exited its context).
+
+ """
+ if not self._dataset.closed:
+ msg = (
+ "Cannot call Saver.complete() until its dataset is closed, "
+ "i.e. the saver's context has exited."
+ )
+ raise ValueError(msg)
+
+ # Complete the saves now
+ dask.compute(self.delayed_completion())
diff --git a/lib/iris/fileformats/netcdf/__init__.py b/lib/iris/fileformats/netcdf/__init__.py
index 795a9c5af0..f1f49b6c76 100644
--- a/lib/iris/fileformats/netcdf/__init__.py
+++ b/lib/iris/fileformats/netcdf/__init__.py
@@ -18,6 +18,7 @@
import logging
+from iris._deprecation import warn_deprecated
import iris.config
# Note: *must* be done before importing from submodules, as they also use this !
@@ -32,14 +33,7 @@
SUPPORTED_ENCODINGS,
)
from .loader import DEBUG, NetCDFDataProxy, load_cubes
-from .saver import (
- CF_CONVENTIONS_VERSION,
- MESH_ELEMENTS,
- SPATIO_TEMPORAL_AXES,
- CFNameCoordMap,
- Saver,
- save,
-)
+from .saver import Saver, save
# Export all public elements from the loader and saver submodules.
# NOTE: the separation is purely for neatness and developer convenience; from
@@ -61,3 +55,57 @@
"parse_cell_methods",
"save",
)
+
+
+def __getattr__(name):
+ """Forward a relocated name to :mod:`iris.fileformats.cf.saver`.
+
+ .. deprecated:: 3.17
+ Four saver names moved to :mod:`iris.fileformats.cf.saver`, because
+ they are not specific to netCDF. Reading them here still works, and
+ warns.
+
+ Parameters
+ ----------
+ name : str
+ The attribute being looked up.
+
+ Returns
+ -------
+ Any
+ The object of that name - the same object, not a copy.
+
+ Raises
+ ------
+ AttributeError
+ If ``name`` is not one of the relocated names.
+
+ """
+ from .saver import _RELOCATED_NAMES
+
+ if name not in _RELOCATED_NAMES:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ from iris.fileformats.cf import saver as cf_saver
+
+ warn_deprecated(
+ f"iris.fileformats.netcdf.{name} has moved to "
+ f"iris.fileformats.cf.saver.{name}, because it is not specific to "
+ "netCDF. The name here will be removed in a future release.",
+ stacklevel=3,
+ )
+ return getattr(cf_saver, name)
+
+
+def __dir__():
+ """List module attributes, including the relocated names.
+
+ Returns
+ -------
+ list of str
+ The real module globals, unioned with the relocated names.
+
+ """
+ from .saver import _RELOCATED_NAMES
+
+ return sorted(set(globals()) | _RELOCATED_NAMES)
diff --git a/lib/iris/fileformats/netcdf/_dataset.py b/lib/iris/fileformats/netcdf/_dataset.py
new file mode 100644
index 0000000000..925223aad7
--- /dev/null
+++ b/lib/iris/fileformats/netcdf/_dataset.py
@@ -0,0 +1,735 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The netCDF implementation of the :mod:`iris.fileformats.cf.dataset` interface.
+
+This is the only module that knows both the CF interface and the netCDF4 API.
+Everything netCDF-specific that the CF layer used to reach through a
+:class:`~iris.fileformats.cf.CFVariable` - ``ncattrs``, ``getncattr``,
+``setncattr``, ``chunking()``, ``file_format``, ``createVariable``,
+``createDimension``, ``filepath()``, ``isopen()`` - either has a named member
+on the interface or lives here as a netCDF-only member.
+
+Reading and writing share one class, and the read path must not touch
+:attr:`NetCDFDataset.write_lock`: making the lock raises for Dask schedulers
+that only the saver cares about, so it is made on first use rather than at
+construction. :meth:`NetCDFDataset.from_existing` serves both paths and
+cannot tell them apart, so it stipulates an open mode rather than observing
+one, and always wraps for byte encoding.
+
+See section 4.5 of ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+from collections.abc import Iterator, Mapping, MutableMapping
+from types import MappingProxyType
+from typing import Any
+import warnings
+
+import numpy as np
+
+from iris._deprecation import warn_deprecated
+from iris._lazy_data import as_lazy_data
+from iris.fileformats.cf import _dask_locks
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+import iris.warnings
+
+from . import _bytecoding_datasets, _thread_safe_nc
+
+#: What ``netCDF4.Variable.chunking()`` answers for an unchunked variable.
+#: ``None`` means the same thing, and arrives from non-version-4 files.
+_CONTIGUOUS = "contiguous"
+
+#: The member an ncdata emulating variable carries instead of file storage.
+_EMULATED_DATA_ARRAY = "_data_array"
+
+# A stab in the dark at the mean length of the "ragged dimension" for netCDF
+# "variable length arrays" (`NetCDF.VLType` type). Total array size is unknown
+# until the variable is read in. Making this number bigger makes it more likely
+# an array will be loaded lazily.
+_MEAN_VL_ARRAY_LEN = 10
+
+
+def _bytes_if_ascii(value):
+ """Return an ASCII string as bytes, and anything else unchanged.
+
+ netCDF4 stores a bytes value as NC_CHAR and a str value as NC_STRING.
+ Iris writes its string attributes as NC_CHAR, so they are encoded here
+ before being set.
+
+ """
+ if isinstance(value, str):
+ try:
+ return value.encode(encoding="ascii")
+ except (AttributeError, UnicodeEncodeError):
+ pass
+ return value
+
+
+class _NetCDFAttributes(MutableMapping):
+ """A netCDF object's attributes as a mapping: read once, written through."""
+
+ def __init__(self, target):
+ """Materialise the attributes of ``target``, a netCDF variable or dataset."""
+ self._target = target
+ self._values: dict[str, Any] = {}
+ for name in target.ncattrs():
+ try:
+ value = target.getncattr(name)
+ except AttributeError:
+ # For some malformed files, netCDF4 lists an attribute name
+ # that it then refuses to return.
+ value = ""
+ self._values[name] = value
+
+ def __getitem__(self, key: str) -> Any:
+ """Return ``key``'s value."""
+ return self._values[key]
+
+ def __setitem__(self, key: str, value: Any) -> None:
+ """Set ``key``'s value, here and in the netCDF object."""
+ # Cache the original, not the coerced value: a just-written attribute
+ # must read back as str, exactly as one read from a file does.
+ self._target.setncattr(key, _bytes_if_ascii(value))
+ self._values[key] = value
+
+ def __delitem__(self, key: str) -> None:
+ """Remove ``key``, here and from the netCDF object."""
+ self._target.delncattr(key)
+ del self._values[key]
+
+ def __iter__(self) -> Iterator[str]:
+ """Return an iterator over the attribute names, in file order."""
+ return iter(self._values)
+
+ def __len__(self) -> int:
+ """Return the number of attributes."""
+ return len(self._values)
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self._values!r})"
+
+
+class NetCDFDatasetVariable(CFDatasetVariable):
+ """One variable of a netCDF file, presented through the CF interface."""
+
+ def __init__(
+ self, variable, location: str, format: str, *, write_lock_factory=None
+ ):
+ """Wrap ``variable``, a thread-safe netCDF variable wrapper.
+
+ Parameters
+ ----------
+ variable : :class:`~iris.fileformats.netcdf._thread_safe_nc.VariableWrapper`
+ The wrapped netCDF variable. For character data this may be an
+ :class:`~iris.fileformats.netcdf._bytecoding_datasets.EncodedVariable`,
+ which reports a different shape, dimensions and dtype from the
+ file's own. This class reports whatever the wrapper reports.
+ location : str
+ The path or URL of the dataset this variable belongs to.
+ format : str
+ The netCDF format of the dataset this variable belongs to.
+ write_lock_factory : callable, optional
+ Returns the lock shared by every variable of one dataset. Called
+ by :meth:`write_handle`. A callable rather than a lock, so that a
+ read never makes one.
+
+ """
+ self._variable = variable
+ self._location = location
+ self._format = format
+ self._write_lock_factory = write_lock_factory
+ self._attributes = _NetCDFAttributes(variable)
+
+ @property
+ def name(self) -> str:
+ """The variable's name within its dataset."""
+ return self._variable.name
+
+ @property
+ def location(self) -> str:
+ """The path or URL of the dataset holding this variable."""
+ return self._location
+
+ @property
+ def format(self) -> str:
+ """The netCDF format of the file holding this variable."""
+ return self._format
+
+ @property
+ def dimensions(self) -> tuple:
+ """The names of the dimensions this variable spans, in order."""
+ return tuple(self._variable.dimensions)
+
+ @property
+ def shape(self) -> tuple:
+ """The variable's shape."""
+ return tuple(self._variable.shape)
+
+ @property
+ def dtype(self):
+ """The variable's stored data type, or ``str`` for a VLEN string type."""
+ return self._variable.dtype
+
+ @property
+ def size(self) -> int:
+ """The total number of elements in the variable."""
+ return self._variable.size
+
+ @property
+ def fill_value(self) -> Any:
+ """The variable's ``_FillValue`` attribute, or ``None`` if it has none."""
+ return self._attributes.get("_FillValue")
+
+ @property
+ def chunking(self) -> tuple | None:
+ """The variable's storage chunk shape, or ``None`` when unchunked."""
+ chunks = self._variable.chunking()
+ if chunks is None or chunks == _CONTIGUOUS:
+ return None
+ return tuple(chunks)
+
+ @property
+ def storage_settings(self) -> Mapping:
+ """The netCDF storage settings this variable holds.
+
+ Read back off the file, so a variable created with `zlib=True` reports
+ the compression netCDF4 actually applied rather than the keyword it
+ was given.
+
+ """
+ filters = self._variable.filters() or {}
+ return MappingProxyType({**filters, "chunksizes": self.chunking})
+
+ @property
+ def attributes(self) -> MutableMapping:
+ """The variable's CF attributes."""
+ return self._attributes
+
+ def __getitem__(self, keys) -> np.ndarray:
+ """Return the indexed portion of the variable's data."""
+ return self._variable[keys]
+
+ def __setitem__(self, keys, values) -> None:
+ """Write ``values`` into the indexed portion of the variable."""
+ self._variable[keys] = values
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self.name!r}, {self.location!r})"
+
+ # netCDF-only members below.
+
+ @property
+ def variable(self):
+ """The backing thread-safe netCDF variable wrapper."""
+ return self._variable
+
+ @property
+ def unencoded_variable(self):
+ """The backing variable, from beneath any byte-encoding wrapper.
+
+ An :class:`~iris.fileformats.netcdf._bytecoding_datasets.EncodedVariable`
+ hides the on-disk ``dtype`` and character dimension; callers needing
+ those want this rather than :attr:`variable`. An unwrapped variable is
+ returned unchanged.
+
+ """
+ if isinstance(self._variable, _bytecoding_datasets.EncodedVariable):
+ return self._variable._contained_instance
+ return self._variable
+
+ @property
+ def is_variable_length(self) -> bool:
+ """Whether this is a netCDF variable-length (VLEN) type.
+
+ Such a variable's total size cannot be known without reading it.
+
+ """
+ datatype = getattr(self._variable, "datatype", None)
+ return isinstance(datatype, _thread_safe_nc.VLType)
+
+ @property
+ def is_emulated(self) -> bool:
+ """Whether an emulating object supplies this variable's data directly.
+
+ ncdata, which bridges Iris and Xarray, passes a netCDF4 emulator
+ whose variables carry their own arrays instead of file storage.
+
+ """
+ return hasattr(self._variable, _EMULATED_DATA_ARRAY)
+
+ @property
+ def emulated_data_array(self):
+ """The array an emulating variable carries instead of file storage."""
+ if not self.is_emulated:
+ raise AttributeError(_EMULATED_DATA_ARRAY)
+ return getattr(self._variable, _EMULATED_DATA_ARRAY)
+
+ @emulated_data_array.setter
+ def emulated_data_array(self, value) -> None:
+ if not self.is_emulated:
+ raise AttributeError(_EMULATED_DATA_ARRAY)
+ setattr(self._variable, _EMULATED_DATA_ARRAY, value)
+
+ def deprecated_netcdf_member(self, name: str) -> Any:
+ """Return a member of the backing netCDF variable wrapper, with a warning."""
+ # Fetch before warning: if getattr raises, no warning is issued. A
+ # failed hasattr() probe has not used the deprecated member.
+ value = getattr(self._variable, name)
+ warn_deprecated(
+ f"Reaching netCDF variable member {name!r} through a CFVariable is "
+ "deprecated and will be removed in a future release. Use the CF "
+ "dataset interface instead - iris.fileformats.cf.dataset - or, for "
+ "a netCDF-only need, cf_var.cf_data.variable."
+ )
+ return value
+
+ def write_handle(self) -> Any:
+ """Return a picklable object supporting ``__setitem__``, for Dask stores.
+
+ It carries the file path and variable name rather than the open file,
+ and always encodes string data, whatever wrapper this variable arrived
+ in.
+
+ """
+ write_lock = None
+ if self._write_lock_factory is not None:
+ write_lock = self._write_lock_factory()
+ return _bytecoding_datasets.EncodedNetCDFWriteProxy(
+ self._location, self._variable, write_lock
+ )
+
+ def read_data(self, chunking_policy):
+ """Return this variable's data, lazily where appropriate.
+
+ Parameters
+ ----------
+ chunking_policy : callable
+ Called with no arguments, returning the ``(chunks, dims_fixed)``
+ pair to build a lazy array with. Call it only once the result is
+ known to be lazy, otherwise it may raise. Under
+ :meth:`~iris.fileformats.cf.loader.ChunkControl.from_file` it raises
+ for a variable the store has not chunked, and we want to avoid this
+ when a variable is small enough to read whole.
+
+ Returns
+ -------
+ numpy.ndarray or dask.array.Array
+ Real data for a small variable, lazy data otherwise.
+
+ """
+ # Deferred, and the module rather than its members: iris.fileformats.cf
+ # reaches this module through iris.fileformats.cf._reader, so a
+ # module-level import would be a cycle; and the tests patch
+ # _LAZYVAR_MIN_BYTES on the module object.
+ from iris.fileformats.cf import loader
+
+ if self.is_emulated:
+ # The variable is not an actual netCDF4 file variable, but an
+ # emulating object with an attached data array (either numpy or
+ # dask), which can be returned immediately as-is. This is the hook
+ # ncdata (https://github.com/SciTools/ncdata) uses to translate data
+ # to and from other packages' netCDF data containers.
+ # See https://github.com/SciTools/iris/issues/4994.
+ result = self.emulated_data_array
+ if result.dtype.kind == "S":
+ # We must also perform any byte-to-string decoding since, in
+ # ncdata, the emulating objects don't do this, and also don't
+ # support a 'set_auto_chartostring(True)'. Therefore, do here
+ # what an EncodedVariable.__getitem__ would do: take the
+ # encoding details from the file (type 'char') variable, then
+ # use them to convert the byte array to strings.
+ encoder = _bytecoding_datasets.VariableEncoder.from_var(
+ self.unencoded_variable
+ )
+ result = encoder.decode_bytes_to_stringarray(result)
+ return result
+
+ # Determine size of data; however can't do this for variable length (VLEN)
+ # netCDF arrays as the size of the array can only be known by reading the
+ # data; see https://github.com/Unidata/netcdf-c/issues/1893.
+ # Note: "Variable length" netCDF types have a datatype of `nc.VLType`.
+ if self.is_variable_length:
+ msg = (
+ f"NetCDF variable `{self.name}` is a variable length type of kind "
+ f"{self.dtype} "
+ "thus the total data size cannot be known in advance. This may affect "
+ "the lazy loading of the data."
+ )
+ warnings.warn(msg, category=iris.warnings.IrisLoadWarning)
+
+ # Give user the chance to pass a hint of the average variable length array
+ # size via the chunk control context manager. This allows for better
+ # decisions to be made on whether the data should be lazy-loaded or not.
+ mean_vl_array_len = _MEAN_VL_ARRAY_LEN
+ chunk_control = loader.CHUNK_CONTROL
+ if chunk_control.mode is not chunk_control.Modes.AS_DASK:
+ if chunks := chunk_control.var_dim_chunksizes.get(self.name):
+ if vl_chunk_hint := chunks.get("_vl_hint"):
+ mean_vl_array_len = vl_chunk_hint
+
+ # Special handling for strings (`str` type) as these don't have an
+ # itemsize attribute; assume 4 bytes which is sufficient for unicode
+ # character storage
+ itemsize = 4 if self.dtype is str else self.dtype.itemsize
+
+ # For `VLType` self.size will just return the known dimension size.
+ total_bytes = self.size * mean_vl_array_len * itemsize
+ else:
+ # Normal NCVariable type:
+ total_bytes = self.size * self.dtype.itemsize
+
+ if total_bytes < loader._LAZYVAR_MIN_BYTES:
+ # Don't make a lazy array, as it will cost more memory AND more time
+ # to access.
+ result = self[:]
+
+ # Special handling of masked scalar value; this will be returned as
+ # an `np.ma.masked` instance which will lose the original dtype.
+ # Workaround for this: return a 1-element masked array of the
+ # correct dtype. Note: this is not an issue for masked arrays,
+ # only masked scalar values.
+ if result is np.ma.masked:
+ result = np.ma.masked_all(1, dtype=self.dtype)
+ return result
+
+ # Get lazy chunked data out of a cf variable.
+ # Creates Dask wrappers around data arrays for any cube components which
+ # can have lazy values, e.g. Cube, Coord, CellMeasure, AuxiliaryVariable.
+ dtype = loader._get_actual_dtype(self)
+
+ # Make a data-proxy that mimics array access and can fetch from the file.
+ # Note: Special handling needed for "variable length string" types which
+ # return a dtype of `str`, rather than a numpy type; use `S1` in this case.
+ if getattr(self.dtype, "kind", None) == "U":
+ # Special handling for "string variables".
+ fill_value = ""
+ else:
+ fill_dtype = "S1" if self.dtype is str else self.dtype.str[1:]
+ fill_value = self.attributes.get(
+ "_FillValue", _thread_safe_nc.default_fillvals[fill_dtype]
+ )
+
+ # Switch type of proxy, based on type of variable.
+ if isinstance(self.variable, _bytecoding_datasets.EncodedVariable):
+ proxy_class = _bytecoding_datasets.EncodedNetCDFDataProxy
+ else:
+ proxy_class = _thread_safe_nc.NetCDFDataProxy
+
+ proxy = proxy_class(self.variable, dtype, self.location, fill_value)
+
+ chunks, dims_fixed = chunking_policy()
+ return as_lazy_data(
+ proxy,
+ meta=proxy.dask_meta,
+ chunks=chunks,
+ dims_fixed=dims_fixed,
+ cache_key=repr(proxy),
+ )
+
+
+#: The formats that make CF loading slow, and that the user can convert away
+#: from with "nccopy".
+_LEGACY_FORMATS = ("NETCDF3_CLASSIC", "NETCDF3_64BIT")
+
+#: The formats that cannot hold every dtype netCDF4 itself accepts.
+#: These are what netCDF4 reports as 'file_format', not what Saver was asked
+#: for. A file created as "NETCDF3_64BIT" reports "NETCDF3_64BIT_OFFSET", so
+#: both spellings are listed; omitting the reported one was the defect in
+#: SciTools/iris#7337.
+_RESTRICTED_FORMATS = (
+ "NETCDF3_CLASSIC",
+ "NETCDF3_64BIT",
+ "NETCDF3_64BIT_OFFSET",
+ "NETCDF4_CLASSIC",
+)
+
+
+class NetCDFDataset(CFDataset):
+ """A netCDF file, presented through the CF interface."""
+
+ def __init__(
+ self,
+ location,
+ mode: str = "r",
+ *,
+ netcdf_format=None,
+ warn_legacy_format: bool = False,
+ ):
+ """Open a netCDF file.
+
+ Parameters
+ ----------
+ location : str or :class:`pathlib.Path`
+ The file's path or URL.
+ mode : str, default="r"
+ The netCDF4 open mode.
+ netcdf_format : str, optional
+ The netCDF format to create, when writing.
+ warn_legacy_format : bool, default=False
+ Whether to warn that a netCDF3 file would load faster if
+ converted. Only loading asks for this; the loader is where the
+ user can act on it.
+
+ """
+ # Set first, so that __del__ and close() are safe if opening fails.
+ self._dataset: _thread_safe_nc.DatasetWrapper | None = None
+ self._owned = True
+ self._closed = False
+ self._variables: dict[str, NetCDFDatasetVariable] | None = None
+ self._attributes: _NetCDFAttributes | None = None
+ self._write_lock = None
+
+ self._location = str(location)
+ self._mode = mode
+
+ if mode == "r" and not _bytecoding_datasets.DECODE_TO_STRINGS_ON_READ:
+ # The user has turned string decoding off for reads. Writing has
+ # no such switch: the saver always encodes.
+ dataset_class = _thread_safe_nc.DatasetWrapper
+ else:
+ dataset_class = _bytecoding_datasets.EncodedDataset
+
+ # netCDF4.Dataset validates "format" against a fixed list and rejects
+ # None, so omit it entirely rather than passing the default through.
+ extra = {} if netcdf_format is None else {"format": netcdf_format}
+ self._dataset = dataset_class(location, mode=mode, **extra)
+
+ if warn_legacy_format:
+ self._warn_if_legacy_format()
+
+ # Turn off *any* automatic decoding by netCDF4 itself. Iris decodes
+ # byte data on its own terms, in _bytecoding_datasets. Inert on an
+ # EncodedDataset, which blocks the call; real on a DatasetWrapper.
+ self._dataset.set_auto_chartostring(False)
+
+ def _warn_if_legacy_format(self) -> None:
+ """Warn that this file would load faster in netCDF4 format, if it would."""
+ assert self._dataset is not None
+ if self._dataset.file_format in _LEGACY_FORMATS:
+ warnings.warn(
+ "Optimise CF-netCDF loading by converting data from NetCDF3 "
+ 'to NetCDF4 file format using the "nccopy" command.',
+ category=iris.warnings.IrisLoadWarning,
+ )
+
+ @classmethod
+ def from_existing(
+ cls, dataset, *, warn_legacy_format: bool = False
+ ) -> "NetCDFDataset":
+ """Wrap an already-open netCDF dataset, without taking ownership of it.
+
+ ``dataset`` may be a thread-safe wrapper, a bare
+ :class:`netCDF4.Dataset`, or an emulator of one, as ncdata passes.
+ :meth:`close` will not release it, because whoever opened it is still
+ responsible for it.
+
+ Parameters
+ ----------
+ warn_legacy_format : bool, default=False
+ Whether to warn that a netCDF3 file would load faster if
+ converted. Only loading asks for this.
+
+ """
+ instance = cls.__new__(cls)
+ instance._owned = False
+ instance._closed = False
+ instance._variables = None
+ instance._attributes = None
+ instance._write_lock = None
+ # netCDF4 records no open mode, so this is stipulated, not observed.
+ # Only __repr__ reads it.
+ instance._mode = "r+"
+
+ if not hasattr(dataset, "THREAD_SAFE_FLAG"):
+ # The wrappers forbid re-wrapping, so only wrap what is not one.
+ # Always encoded: DECODE_TO_STRINGS_ON_READ is not consulted here,
+ # because from_existing cannot tell a read from a write.
+ dataset = _bytecoding_datasets.EncodedDataset.from_existing(dataset)
+ instance._dataset = dataset
+ instance._dataset.set_auto_chartostring(False)
+ instance._location = str(dataset.filepath())
+
+ if warn_legacy_format:
+ instance._warn_if_legacy_format()
+
+ return instance
+
+ @property
+ def location(self) -> str:
+ """The file's path or URL."""
+ return self._location
+
+ @property
+ def mode(self) -> str:
+ """The mode the file was opened in."""
+ return self._mode
+
+ @property
+ def format(self) -> str:
+ """The netCDF format of this file, as netCDF4 reports it."""
+ assert self._dataset is not None
+ return self._dataset.file_format
+
+ @property
+ def storage_settings(self) -> Mapping:
+ """The store-wide netCDF settings. Only the format is one."""
+ return MappingProxyType({"format": self.format})
+
+ @property
+ def closed(self) -> bool:
+ """Whether the file has been released, by this object or by its owner.
+
+ A borrowed dataset can be closed by its owner without this object
+ knowing, so the backing object is asked too. An emulator need not
+ implement ``isopen()``; one that does not is taken to be open.
+
+ """
+ if self._closed:
+ return True
+ isopen = getattr(self._dataset, "isopen", None)
+ if isopen is None:
+ return False
+ return not isopen()
+
+ def _materialised_variables(self) -> dict[str, "NetCDFDatasetVariable"]:
+ """Return the wrapper mapping, building it from the file if need be."""
+ if self._variables is None:
+ assert self._dataset is not None
+ self._variables = {
+ name: NetCDFDatasetVariable(
+ variable,
+ self._location,
+ self._dataset.file_format,
+ write_lock_factory=self._write_lock_factory,
+ )
+ for name, variable in self._dataset.variables.items()
+ }
+ return self._variables
+
+ @property
+ def variables(self) -> Mapping:
+ """The file's variables, by name."""
+ return self._materialised_variables()
+
+ @property
+ def dimensions(self) -> Mapping:
+ """The file's dimension lengths, by name.
+
+ An unlimited dimension reports the number of records written so far.
+
+ """
+ assert self._dataset is not None
+ return {
+ name: dimension.size for name, dimension in self._dataset.dimensions.items()
+ }
+
+ @property
+ def attributes(self) -> MutableMapping:
+ """The file's global attributes."""
+ if self._attributes is None:
+ self._attributes = _NetCDFAttributes(self._dataset)
+ return self._attributes
+
+ def create_dimension(self, name: str, size: int | None) -> None:
+ """Declare a dimension of the given length, or unlimited for ``None``."""
+ assert self._dataset is not None
+ self._dataset.createDimension(name, size)
+
+ def create_variable(
+ self, name: str, dtype, dimensions=(), *, fill_value=None, **storage_settings
+ ) -> NetCDFDatasetVariable:
+ """Create and return a new variable.
+
+ ``**storage_settings`` reaches :meth:`netCDF4.Dataset.createVariable`
+ unaltered: ``compression``, ``zlib``, ``complevel``, ``shuffle``,
+ ``chunksizes``, ``least_significant_digit`` and the rest.
+
+ """
+ assert self._dataset is not None
+ variable = self._dataset.createVariable(
+ name, dtype, tuple(dimensions), fill_value=fill_value, **storage_settings
+ )
+ wrapped = NetCDFDatasetVariable(
+ variable,
+ self._location,
+ self._dataset.file_format,
+ write_lock_factory=self._write_lock_factory,
+ )
+ # Register this wrapper, so that a later lookup by name returns it
+ # rather than a second wrapper with its own attribute cache.
+ self._materialised_variables()[name] = wrapped
+ return wrapped
+
+ def supports_dtype(self, dtype) -> bool:
+ """Whether this file can hold a variable of this dtype.
+
+ Parameters
+ ----------
+ dtype : :class:`numpy.dtype`
+ The dtype a variable would be created with.
+
+ Returns
+ -------
+ bool
+ False for 64-bit and unsigned integers on the restricted formats,
+ which have no such types. True otherwise.
+
+ """
+ if self.format not in _RESTRICTED_FORMATS:
+ return True
+ return not (
+ np.issubdtype(dtype, np.int64) or np.issubdtype(dtype, np.unsignedinteger)
+ )
+
+ def sync(self) -> None:
+ """Flush buffered writes to the file."""
+ assert self._dataset is not None
+ self._dataset.sync()
+
+ def finalise(self) -> None:
+ """Do nothing: a netCDF file needs no completion step.
+
+ Exists for stores that do, such as Zarr's consolidated metadata.
+
+ """
+
+ def close(self) -> None:
+ """Close the file, if this object opened it."""
+ if self._owned and self._dataset is not None and not self._closed:
+ self._dataset.close()
+ self._closed = True
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self.location!r}, mode={self.mode!r})"
+
+ # netCDF-only members below.
+
+ @property
+ def dataset(self):
+ """The backing thread-safe netCDF dataset wrapper."""
+ return self._dataset
+
+ @property
+ def write_lock(self):
+ """The lock every worker writing to this file must hold.
+
+ One per dataset. Made on first use, never at construction.
+
+ """
+ if self._write_lock is None:
+ self._write_lock = _dask_locks.get_worker_lock(self._location)
+ return self._write_lock
+
+ def _write_lock_factory(self):
+ """Return :attr:`write_lock`, making it on the first call.
+
+ Handed to every :class:`NetCDFDatasetVariable` so that each can find
+ the one shared lock at the moment it builds a write handle.
+
+ """
+ return self.write_lock
diff --git a/lib/iris/fileformats/netcdf/loader.py b/lib/iris/fileformats/netcdf/loader.py
index c12a3c6d44..2eb780d7ec 100644
--- a/lib/iris/fileformats/netcdf/loader.py
+++ b/lib/iris/fileformats/netcdf/loader.py
@@ -2,13 +2,23 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Support loading Iris cubes from NetCDF files using the CF conventions for metadata interpretation.
+"""Load Iris cubes from NetCDF files and OPeNDAP URLs.
+
+.. deprecated:: 3.17
+ ``CHUNK_CONTROL`` and ``ChunkControl`` have moved to
+ :mod:`iris.fileformats.cf.loader`, because chunking policy is not specific
+ to netCDF. They can still be reached here, with a warning, for one
+ deprecation cycle.
.. z_reference:: iris.fileformats.netcdf.loader
:tags: topic_load_save
API reference
+What is left here is what is genuinely netCDF's: opening the file sources, and
+the proxy class that reads from them. Interpreting what is inside is
+:mod:`iris.fileformats.cf.loader`'s job.
+
See : `NetCDF User's Guide <https://docs.unidata.ucar.edu/nug/current/>`_
and `netCDF4 python module <https://github.com/Unidata/netcdf4-python>`_.
@@ -16,47 +26,25 @@
"""
-from collections.abc import Iterable, Iterator, Mapping
-from contextlib import contextmanager
-from copy import deepcopy
-from enum import Enum, auto
-from functools import partial
-import threading
+from collections.abc import Iterable
import warnings
-import numpy as np
-
-from iris._lazy_data import as_lazy_data
-from iris.aux_factory import (
- AtmosphereSigmaFactory,
- HybridHeightFactory,
- HybridPressureFactory,
- OceanSFactory,
- OceanSg1Factory,
- OceanSg2Factory,
- OceanSigmaFactory,
- OceanSigmaZFactory,
-)
-import iris.config
-import iris.coord_systems
-import iris.coords
-import iris.fileformats.cf
-from iris.fileformats.cf import CFDataVariable
-from iris.fileformats.netcdf import _bytecoding_datasets, _thread_safe_nc
-from iris.fileformats.netcdf._bytecoding_datasets import (
- EncodedVariable,
- VariableEncoder,
+from iris._deprecation import warn_deprecated
+from iris.fileformats.cf.loader import (
+ _actions_engine,
+ _load_aux_factory,
+ _load_cube,
+ _translate_constraints_to_var_callback,
)
-from iris.fileformats.netcdf.saver import _CF_ATTRS
-import iris.io
-import iris.util
import iris.warnings
# Show actions activation statistics.
+# NOTE: read by iris.fileformats.cf.loader._load_cube_inner, through this
+# module rather than by value, so that assigning to it here still works.
DEBUG = False
-# Get the logger : shared logger for all in 'iris.fileformats.netcdf'.
-from . import logger
+# Get _bytecoding_datasets, needed by NetCDFDataProxy just below.
+from . import _bytecoding_datasets
# An expected part of the public loader API, but includes thread safety
# concerns so is housed in _thread_safe_nc.
@@ -66,615 +54,76 @@
# :data:`_bytecoding_datasets.DECODE_TO_STRINGS_ON_READ`
NetCDFDataProxy = _bytecoding_datasets.EncodedNetCDFDataProxy
+#: Names that moved to :mod:`iris.fileformats.cf.loader` in Iris 3.17 and are
+#: still readable here, with a warning, for one deprecation cycle.
+#:
+#: Read-only: a module ``__getattr__`` cannot intercept assignment, so a name
+#: that users *set* must keep its definition here instead. ``DEBUG`` is the
+#: one such name, and is deliberately absent from this set.
+#:
+#: The same limitation means ``mock.patch("iris.fileformats.netcdf.loader."
+#: "CHUNK_CONTROL", ...)`` succeeds and is a silent no-op: ``mock.patch``
+#: creates the attribute on this module, while production code reads
+#: ``iris.fileformats.cf.loader.CHUNK_CONTROL``. Patch
+#: :mod:`iris.fileformats.cf.loader` instead.
+_RELOCATED_NAMES = frozenset({"CHUNK_CONTROL", "ChunkControl"})
-class _WarnComboIgnoringBoundsLoad(
- iris.warnings.IrisIgnoringBoundsWarning,
- iris.warnings.IrisLoadWarning,
-):
- """One-off combination of warning classes - enhances user filtering."""
-
- pass
-
-
-def _actions_engine():
- # Return an 'actions engine', which provides a pyke-rules-like interface to
- # the core cf translation code.
- # Deferred import to avoid circularity.
- import iris.fileformats._nc_load_rules.engine as nc_actions_engine
-
- engine = nc_actions_engine.Engine()
- return engine
-
-
-def _assert_case_specific_facts(engine, cf, cf_group):
- # Initialise a data store for built cube elements.
- # This is used to patch element attributes *not* setup by the actions
- # process, after the actions code has run.
- engine.cube_parts["coordinates"] = []
- engine.cube_parts["cell_measures"] = []
- engine.cube_parts["ancillary_variables"] = []
- engine.cube_parts["coordinate_systems"] = {}
-
- # Add the parsed coordinate reference system mappings
- engine.cube_parts["coordinate_system_mappings"] = cf._coord_system_mappings.get(
- engine.cf_var.cf_name, None
- )
-
- # Assert facts for CF coordinates.
- for cf_name in cf_group.coordinates.keys():
- engine.add_case_specific_fact("coordinate", (cf_name,))
-
- # Assert facts for CF auxiliary coordinates.
- for cf_name in cf_group.auxiliary_coordinates.keys():
- engine.add_case_specific_fact("auxiliary_coordinate", (cf_name,))
-
- # Assert facts for CF cell measures.
- for cf_name in cf_group.cell_measures.keys():
- engine.add_case_specific_fact("cell_measure", (cf_name,))
-
- # Assert facts for CF ancillary variables.
- for cf_name in cf_group.ancillary_variables.keys():
- engine.add_case_specific_fact("ancillary_variable", (cf_name,))
-
- # Assert facts for CF grid_mappings.
- for cf_name in cf_group.grid_mappings.keys():
- engine.add_case_specific_fact("grid_mapping", (cf_name,))
-
- # Assert facts for CF labels.
- for cf_name in cf_group.labels.keys():
- engine.add_case_specific_fact("label", (cf_name,))
-
- # Assert facts for CF formula terms associated with the cf_group
- # of the CF data variable.
-
- # Collect varnames of formula-root variables as we go.
- # NOTE: use dictionary keys as an 'OrderedSet'
- # - see: https://stackoverflow.com/a/53657523/2615050
- # This is to ensure that we can handle the resulting facts in a definite
- # order, as using a 'set' led to indeterminate results.
- formula_root = {}
- for cf_var in cf.cf_group.formula_terms.values():
- for cf_root, cf_term in cf_var.cf_terms_by_root.items():
- # Only assert this fact if the formula root variable is
- # defined in the CF group of the CF data variable.
- if cf_root in cf_group:
- formula_root[cf_root] = True
- engine.add_case_specific_fact(
- "formula_term",
- (cf_var.cf_name, cf_root, cf_term),
- )
-
- for cf_root in formula_root.keys():
- engine.add_case_specific_fact("formula_root", (cf_root,))
-
-
-def _actions_activation_stats(engine, cf_name):
- print("-" * 80)
- print("CF Data Variable: %r" % cf_name)
-
- engine.print_stats()
- print("Rules Triggered:")
-
- for rule in sorted(list(engine.rules_triggered)):
- print("\t%s" % rule)
-
- print("Case Specific Facts:")
- kb_facts = engine.get_kb()
-
- for key in kb_facts.entity_lists.keys():
- for arg in kb_facts.entity_lists[key].case_specific_facts:
- print("\t%s%s" % (key, arg))
-
-
-def _set_attributes(attributes, key, value):
- """Set attributes dictionary, converting unicode strings appropriately."""
- if isinstance(value, str):
- try:
- attributes[str(key)] = str(value)
- except UnicodeEncodeError:
- attributes[str(key)] = value
- else:
- attributes[str(key)] = value
+def __getattr__(name):
+ """Forward a relocated name to :mod:`iris.fileformats.cf.loader`.
+ Parameters
+ ----------
+ name : str
+ The attribute being looked up. Python calls this only for names that
+ are not already module attributes.
-def _add_unused_attributes(iris_object, cf_var):
- """Populate the attributes of a cf element with the "unused" attributes.
+ Returns
+ -------
+ Any
+ The object of that name in :mod:`iris.fileformats.cf.loader` - the
+ same object, not a copy.
- Populate the attributes of a cf element with the "unused" attributes
- from the associated CF-netCDF variable. That is, all those that aren't CF
- reserved terms.
+ Raises
+ ------
+ AttributeError
+ If ``name`` is not one of the relocated names. Raised without warning,
+ so that a ``hasattr`` probe that comes back False stays quiet.
"""
- from iris.fileformats._nc_load_rules.helpers import _add_or_capture
- from iris.loading import LoadProblems
-
- def attribute_predicate(item):
- return item[0] not in _CF_ATTRS
-
- tmpvar = filter(attribute_predicate, cf_var.cf_attrs_unused())
- attrs_dict = iris_object.attributes
- if hasattr(attrs_dict, "locals"):
- # Treat cube attributes (i.e. a CubeAttrsDict) as a special case.
- # These attrs are "local" (i.e. on the variable), so record them as such.
- attrs_dict = attrs_dict.locals
-
- for attr_name, attr_value in tmpvar:
- _ = _add_or_capture(
- build_func=partial(lambda: attr_value),
- add_method=partial(_set_attributes, attrs_dict, attr_name),
- cf_var=cf_var,
- attr_key=attr_name,
- destination=LoadProblems.Problem.Destination(
- iris_class=iris_object.__class__,
- identifier=cf_var.cf_name,
- ),
- )
-
-
-def _get_actual_dtype(cf_var):
- # Figure out what the eventual data type will be after any scale/offset
- # transforms.
- dummy_data = np.zeros(1, dtype=cf_var.dtype)
- if hasattr(cf_var, "scale_factor"):
- dummy_data = cf_var.scale_factor * dummy_data
- if hasattr(cf_var, "add_offset"):
- dummy_data = cf_var.add_offset + dummy_data
- return dummy_data.dtype
-
-
-# An arbitrary variable array size, below which we will fetch real data from a variable
-# rather than making a lazy array for deferred access.
-# Set by experiment at roughly the point where it begins to save us memory, but actually
-# mostly done for speed improvement. See https://github.com/SciTools/iris/pull/5069
-_LAZYVAR_MIN_BYTES = 5000
-
-# A stab in the dark at the mean length of the "ragged dimension" for netCDF "variable
-# length arrays" (`NetCDF.VLType` type). Total array size is unknown until the variable is
-# read in. Making this number bigger makes it more likely an array will be loaded lazily.
-_MEAN_VL_ARRAY_LEN = 10
-
-
-def _get_cf_var_data(cf_var):
- """Get an array representing the data of a CF variable.
-
- This is typically a lazy array based around a NetCDFDataProxy, but if the variable
- is "sufficiently small", we instead fetch the data as a real (numpy) array.
- The latter is especially valuable for scalar coordinates, which are otherwise
- unnecessarily slow + wasteful of memory.
+ if name not in _RELOCATED_NAMES:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ # Deferred so that importing this module stays quiet and cheap.
+ from iris.fileformats.cf import loader as cf_loader
+
+ warn_deprecated(
+ f"iris.fileformats.netcdf.loader.{name} has moved to "
+ f"iris.fileformats.cf.loader.{name}, because it is not specific to "
+ "netCDF. The name here will be removed in a future release.",
+ # warn_deprecated's default stacklevel=2 points at the line below,
+ # inside this function - PEP 562 means that line is a genuine call
+ # frame, not the user's code. One level further out reaches the
+ # line that actually read the attribute.
+ stacklevel=3,
+ )
+ return getattr(cf_loader, name)
- """
- if hasattr(cf_var, "_data_array"):
- # The variable is not an actual netCDF4 file variable, but an emulating
- # object with an attached data array (either numpy or dask), which can be
- # returned immediately as-is. This is used as a hook to translate data to/from
- # netcdf data container objects in other packages, such as xarray.
- # See https://github.com/SciTools/iris/issues/4994 "Xarray bridge".
- result = cf_var._data_array
- if result.dtype.kind == "S":
- # We must also perform any byte-to-string decoding since, in ncdata, the
- # emulating objects don't do this, and also don't support a
- # 'set_auto_chartostring(True)'.
- # Therefore, do here what an EncodedVariable.__getitem__ would do : ..
- # .. get details based on the file (type 'char') variable ..
- encoder = VariableEncoder.from_var(cf_var.cf_data)
- # .. convert byte array to strings.
- result = encoder.decode_bytes_to_stringarray(result)
- else:
- # Determine size of data; however can't do this for variable length (VLEN)
- # netCDF arrays as the size of the array can only be known by reading the
- # data; see https://github.com/Unidata/netcdf-c/issues/1893.
- # Note: "Variable length" netCDF types have a datatype of `nc.VLType`.
- if isinstance(getattr(cf_var, "datatype", None), _thread_safe_nc.VLType):
- msg = (
- f"NetCDF variable `{cf_var.cf_name}` is a variable length type of kind {cf_var.dtype} "
- "thus the total data size cannot be known in advance. This may affect the lazy loading "
- "of the data."
- )
- warnings.warn(msg, category=iris.warnings.IrisLoadWarning)
-
- # Give user the chance to pass a hint of the average variable length array size via
- # the chunk control context manager. This allows for better decisions to be made on
- # whether the data should be lazy-loaded or not.
- mean_vl_array_len = _MEAN_VL_ARRAY_LEN
- if CHUNK_CONTROL.mode is not CHUNK_CONTROL.Modes.AS_DASK:
- if chunks := CHUNK_CONTROL.var_dim_chunksizes.get(cf_var.cf_name):
- if vl_chunk_hint := chunks.get("_vl_hint"):
- mean_vl_array_len = vl_chunk_hint
-
- # Special handling for strings (`str` type) as these don't have an itemsize attribute;
- # assume 4 bytes which is sufficient for unicode character storage
- itemsize = 4 if cf_var.dtype is str else cf_var.dtype.itemsize
-
- # For `VLType` cf_var.size will just return the known dimension size.
- total_bytes = cf_var.size * mean_vl_array_len * itemsize
- else:
- # Normal NCVariable type:
- total_bytes = cf_var.size * cf_var.dtype.itemsize
-
- if total_bytes < _LAZYVAR_MIN_BYTES:
- # Don't make a lazy array, as it will cost more memory AND more time to access.
- result = cf_var[:]
-
- # Special handling of masked scalar value; this will be returned as
- # an `np.ma.masked` instance which will lose the original dtype.
- # Workaround for this it return a 1-element masked array of the
- # correct dtype. Note: this is not an issue for masked arrays,
- # only masked scalar values.
- if result is np.ma.masked:
- result = np.ma.masked_all(1, dtype=cf_var.dtype)
- else:
- # Get lazy chunked data out of a cf variable.
- # Creates Dask wrappers around data arrays for any cube components which
- # can have lazy values, e.g. Cube, Coord, CellMeasure, AuxiliaryVariable.
- dtype = _get_actual_dtype(cf_var)
-
- # Make a data-proxy that mimics array access and can fetch from the file.
- # Note: Special handling needed for "variable length string" types which
- # return a dtype of `str`, rather than a numpy type; use `S1` in this case.
- if getattr(cf_var.dtype, "kind", None) == "U":
- # Special handling for "string variables".
- fill_value = ""
- else:
- fill_dtype = "S1" if cf_var.dtype is str else cf_var.dtype.str[1:]
- fill_value = getattr(
- cf_var.cf_data,
- "_FillValue",
- _thread_safe_nc.default_fillvals[fill_dtype],
- )
-
- # Switch type of proxy, based on type of variable.
- # It is done this way, instead of using an instance variable, because the
- # limited nature of the wrappers makes a stateful choice awkward,
- # e.g. especially, "variable.group()" is *not* the parent DatasetWrapper.
- if isinstance(cf_var.cf_data, _bytecoding_datasets.EncodedVariable):
- proxy_class = _bytecoding_datasets.EncodedNetCDFDataProxy
- else:
- proxy_class = _thread_safe_nc.NetCDFDataProxy
-
- proxy = proxy_class(cf_var.cf_data, dtype, cf_var.filename, fill_value)
- # Get the chunking specified for the variable : this is either a shape, or
- # maybe the string "contiguous".
- if CHUNK_CONTROL.mode is ChunkControl.Modes.AS_DASK:
- result = as_lazy_data(proxy, meta=proxy.dask_meta, chunks="auto")
- else:
- chunks = cf_var.cf_data.chunking()
- if chunks is None:
- # Occurs for non-version-4 netcdf
- chunks = "contiguous"
- # In the "contiguous" case, pass chunks=None to 'as_lazy_data'.
- if chunks == "contiguous":
- if (
- CHUNK_CONTROL.mode is ChunkControl.Modes.FROM_FILE
- and isinstance(cf_var, iris.fileformats.cf.CFDataVariable)
- ):
- raise KeyError(
- f"{cf_var.cf_name} does not contain pre-existing chunk specifications."
- f" Instead, you might wish to use CHUNK_CONTROL.set(), or just use default"
- f" behaviour outside of a context manager. "
- )
- # Equivalent to chunks=None, but value required by chunking control
- chunks = list(cf_var.shape)
-
- # Modify the chunking in the context of an active chunking control.
- # N.B. settings specific to this named var override global ('*') ones.
- dim_chunks = CHUNK_CONTROL.var_dim_chunksizes.get(
- cf_var.cf_name
- ) or CHUNK_CONTROL.var_dim_chunksizes.get("*")
- dims = cf_var.cf_data.dimensions
- if CHUNK_CONTROL.mode is ChunkControl.Modes.FROM_FILE:
- dims_fixed = np.ones(len(dims), dtype=bool)
- elif not dim_chunks:
- dims_fixed = None
- else:
- # Modify the chunks argument, and pass in a list of 'fixed' dims, for
- # any of our dims which are controlled.
- dims_fixed = np.zeros(len(dims), dtype=bool)
- for i_dim, dim_name in enumerate(dims):
- dim_chunksize = dim_chunks.get(dim_name)
- if dim_chunksize:
- if dim_chunksize == -1:
- chunks[i_dim] = cf_var.shape[i_dim]
- else:
- chunks[i_dim] = dim_chunksize
- dims_fixed[i_dim] = True
- if dims_fixed is None:
- dims_fixed = [dims_fixed]
- result = as_lazy_data(
- proxy,
- meta=proxy.dask_meta,
- chunks=chunks,
- dims_fixed=tuple(dims_fixed),
- )
- return result
-
-
-class _OrderedAddableList(list):
- """A custom container object for actions recording.
-
- Used purely in actions debugging, to accumulate a record of which actions
- were activated.
-
- It replaces a set, so as to preserve the ordering of operations, with
- possible repeats, and it also numbers the entries.
-
- The actions routines invoke an 'add' method, so this effectively replaces
- a set.add with a list.append.
- """
+def __dir__():
+ """List module attributes, including the relocated names.
- def __init__(self, *args, **kwargs):
- super().__init__(*args, **kwargs)
- self._n_add = 0
-
- def add(self, msg):
- self._n_add += 1
- n_add = self._n_add
- self.append(f"#{n_add:03d} : {msg}")
-
-
-def _load_cube(engine, cf, cf_var, filename):
- # Translate dimension chunk-settings specific to this cube (i.e. named by
- # it's data-var) into global ones, for the duration of this load.
- # Thus, by default, we will create any AuxCoords, CellMeasures et al with
- # any per-dimension chunksizes specified for the cube.
- these_settings = CHUNK_CONTROL.var_dim_chunksizes.get(cf_var.cf_name, {})
- with CHUNK_CONTROL.set(**these_settings):
- return _load_cube_inner(engine, cf, cf_var, filename)
-
-
-def _load_cube_inner(engine, cf, cf_var, filename):
- from iris.cube import Cube
-
- """Create the cube associated with the CF-netCDF data variable."""
- from iris.fileformats.netcdf.saver import Saver
-
- if hasattr(cf_var, Saver._DATALESS_ATTRNAME):
- # This data-variable represents a dataless cube.
- # The variable array content was never written (to take up no space).
- data = None
- shape = cf_var.shape
- else:
- data = _get_cf_var_data(cf_var)
- shape = None
- cube = Cube(data=data, shape=shape)
-
- # Reset the actions engine.
- engine.reset()
-
- # Initialise engine rule processing hooks.
- engine.cf_var = cf_var
- engine.cube = cube
- engine.cube_parts = {}
- engine.requires = {}
- engine.rules_triggered = _OrderedAddableList()
- engine.filename = filename
-
- # Assert all the case-specific facts.
- # This extracts 'facts' specific to this data-variable (aka cube), from
- # the info supplied in the CFGroup object.
- _assert_case_specific_facts(engine, cf, cf_var.cf_group)
-
- # Run the actions engine.
- # This creates various cube elements and attaches them to the cube.
- # It also records various other info on the engine, to be processed later.
- engine.activate()
-
- # Having run the rules, now add the "unused" attributes to each cf element.
- def fix_attributes_all_elements(role_name):
- elements_and_names = engine.cube_parts.get(role_name, [])
-
- for iris_object, cf_var_name in elements_and_names:
- _add_unused_attributes(iris_object, cf.cf_group[cf_var_name])
-
- # Populate the attributes of all coordinates, cell-measures and ancillary-vars.
- fix_attributes_all_elements("coordinates")
- fix_attributes_all_elements("ancillary_variables")
- fix_attributes_all_elements("cell_measures")
-
- # Also populate attributes of the top-level cube itself.
- _add_unused_attributes(cube, cf_var)
-
- # Work out reference names for all the coords.
- names = {
- coord.var_name: coord.standard_name or coord.var_name or "unknown"
- for coord in cube.coords()
- }
-
- # Add all the cube cell methods.
- cube.cell_methods = [
- iris.coords.CellMethod(
- method=method.method,
- intervals=method.intervals,
- comments=method.comments,
- coords=[
- names[coord_name] if coord_name in names else coord_name
- for coord_name in method.coord_names
- ],
- )
- for method in cube.cell_methods
- ]
-
- # Set extended_grid_mapping property ONLY if extended grid_mapping was used.
- # This avoids having an unnecessary `iris_extended_grid_mapping` attribute entry.
- if cs_mappings := engine.cube_parts.get("coordinate_system_mappings", None):
- # `None` as a mapping key implies simple mapping syntax (single coord system)
- if None not in cs_mappings:
- cube.extended_grid_mapping = True
-
- if DEBUG:
- # Show activation statistics for this data-var (i.e. cube).
- _actions_activation_stats(engine, cf_var.cf_name)
-
- return cube
-
-
-def _load_aux_factory(engine, cube):
- """Convert any CF-netCDF dimensionless coordinate to an AuxCoordFactory."""
- formula_type = engine.requires.get("formula_type")
- if formula_type in [
- "atmosphere_sigma_coordinate",
- "atmosphere_hybrid_height_coordinate",
- "atmosphere_hybrid_sigma_pressure_coordinate",
- "ocean_sigma_z_coordinate",
- "ocean_sigma_coordinate",
- "ocean_s_coordinate",
- "ocean_s_coordinate_g1",
- "ocean_s_coordinate_g2",
- ]:
-
- def coord_from_term(term):
- # Convert term names to coordinates (via netCDF variable names).
- name = engine.requires["formula_terms"].get(term, None)
- if name is not None:
- for coord, cf_var_name in engine.cube_parts["coordinates"]:
- if cf_var_name == name:
- return coord
- warnings.warn(
- "Unable to find coordinate for variable {!r}".format(name),
- category=iris.warnings.IrisFactoryCoordNotFoundWarning,
- )
-
- if formula_type == "atmosphere_sigma_coordinate":
- pressure_at_top = coord_from_term("ptop")
- sigma = coord_from_term("sigma")
- surface_air_pressure = coord_from_term("ps")
- factory = AtmosphereSigmaFactory(
- pressure_at_top, sigma, surface_air_pressure
- )
- elif formula_type == "atmosphere_hybrid_height_coordinate":
- delta = coord_from_term("a")
- sigma = coord_from_term("b")
- orography = coord_from_term("orog")
- factory = HybridHeightFactory(delta, sigma, orography)
- elif formula_type == "atmosphere_hybrid_sigma_pressure_coordinate":
- # Hybrid pressure has two valid versions of its formula terms:
- # "p0: var1 a: var2 b: var3 ps: var4" or
- # "ap: var1 b: var2 ps: var3" where "ap = p0 * a"
- # Attempt to get the "ap" term.
- delta = coord_from_term("ap")
- if delta is None:
- # The "ap" term is unavailable, so try getting terms "p0"
- # and "a" terms in order to derive an "ap" equivalent term.
- coord_p0 = coord_from_term("p0")
- if coord_p0 is not None:
- if coord_p0.shape != (1,):
- msg = (
- "Expecting {!r} to be a scalar reference "
- "pressure coordinate, got shape {!r}".format(
- coord_p0.var_name, coord_p0.shape
- )
- )
- raise ValueError(msg)
- if coord_p0.has_bounds():
- msg = (
- "Ignoring atmosphere hybrid sigma pressure "
- "scalar coordinate {!r} bounds.".format(coord_p0.name())
- )
- warnings.warn(
- msg,
- category=_WarnComboIgnoringBoundsLoad,
- )
- coord_a = coord_from_term("a")
- if coord_a is not None:
- if coord_a.units.is_unknown():
- # Be graceful, and promote unknown to dimensionless units.
- coord_a.units = "1"
- delta = coord_a * coord_p0.points[0]
- delta.units = coord_a.units * coord_p0.units
- delta.rename("vertical pressure")
- delta.var_name = "ap"
- cube.add_aux_coord(delta, cube.coord_dims(coord_a))
-
- sigma = coord_from_term("b")
- surface_air_pressure = coord_from_term("ps")
- factory = HybridPressureFactory(delta, sigma, surface_air_pressure)
- elif formula_type == "ocean_sigma_z_coordinate":
- sigma = coord_from_term("sigma")
- eta = coord_from_term("eta")
- depth = coord_from_term("depth")
- depth_c = coord_from_term("depth_c")
- nsigma = coord_from_term("nsigma")
- zlev = coord_from_term("zlev")
- factory = OceanSigmaZFactory(sigma, eta, depth, depth_c, nsigma, zlev)
- elif formula_type == "ocean_sigma_coordinate":
- sigma = coord_from_term("sigma")
- eta = coord_from_term("eta")
- depth = coord_from_term("depth")
- factory = OceanSigmaFactory(sigma, eta, depth)
- elif formula_type == "ocean_s_coordinate":
- s = coord_from_term("s")
- eta = coord_from_term("eta")
- depth = coord_from_term("depth")
- a = coord_from_term("a")
- depth_c = coord_from_term("depth_c")
- b = coord_from_term("b")
- factory = OceanSFactory(s, eta, depth, a, b, depth_c)
- elif formula_type == "ocean_s_coordinate_g1":
- s = coord_from_term("s")
- c = coord_from_term("c")
- eta = coord_from_term("eta")
- depth = coord_from_term("depth")
- depth_c = coord_from_term("depth_c")
- factory = OceanSg1Factory(s, c, eta, depth, depth_c)
- elif formula_type == "ocean_s_coordinate_g2":
- s = coord_from_term("s")
- c = coord_from_term("c")
- eta = coord_from_term("eta")
- depth = coord_from_term("depth")
- depth_c = coord_from_term("depth_c")
- factory = OceanSg2Factory(s, c, eta, depth, depth_c)
- cube.add_aux_factory(factory)
-
-
-def _translate_constraints_to_var_callback(constraints):
- """Translate load constraints into a simple data-var filter function, if possible.
+ A module ``__getattr__`` forwards reads of the relocated names, but does
+ not make them discoverable: PEP 562 does not extend to ``dir()``,
+ :func:`inspect.getmembers`, or tab-completion, so this fills the gap.
Returns
-------
- bool or None
-
- Notes
- -----
- For now, ONLY handles NameConstraints with no 'STASH' component.
+ list of str
+ The real module globals, unioned with the relocated names.
"""
- import iris._constraints
-
- constraints = iris._constraints.list_of_constraints(constraints)
- if len(constraints) == 0 or not all(
- isinstance(constraint, iris._constraints.NameConstraint)
- and constraint.STASH == "none"
- for constraint in constraints
- ):
- # We can define a var-filtering function to speedup the load, *ONLY* when we
- # have some constraints, and all are simple NameConstraints with no STASH.
- result = None
- else:
-
- def inner(cf_datavar):
- match_any_constraint = False
- for constraint in constraints:
- match_this_constraint = True
- for name in constraint._names:
- expected = getattr(constraint, name)
- if name != "STASH" and expected != "none":
- attr_name = "cf_name" if name == "var_name" else name
- # Fetch property : N.B. CFVariable caches the property values
- # The use of a default here is the only difference from the code in NameConstraint.
- if not hasattr(cf_datavar, attr_name):
- continue
- actual = getattr(cf_datavar, attr_name, "")
- if actual != expected:
- match_this_constraint = False
- break
- if match_this_constraint:
- match_any_constraint = True
- break
- return match_any_constraint
-
- result = inner
- return result
+ return sorted(set(globals()) | _RELOCATED_NAMES)
def load_cubes(file_sources, callback=None, constraints=None):
@@ -701,15 +150,13 @@ def load_cubes(file_sources, callback=None, constraints=None):
"""
# Deferred import to avoid circular imports.
- from iris.cube import Cube
- from iris.fileformats._nc_load_rules.helpers import _add_or_capture
from iris.fileformats.cf import CFReader
from iris.io import run_callback
- from iris.loading import LoadProblems
from .ugrid_load import (
- _build_mesh_coords,
+ _attach_mesh_coords,
_meshes_from_cf,
+ _prepare_mesh_coords,
)
# Create a low-level data-var filter from the original load constraints, if they are suitable.
@@ -735,65 +182,14 @@ def load_cubes(file_sources, callback=None, constraints=None):
# Deliver only selected results.
continue
- # cf_var-specific mesh handling, if a mesh is present.
- # Build the mesh_coords *before* loading the cube - avoids
- # mesh-related attributes being picked up by
- # _add_unused_attributes().
- mesh_name = None
- mesh = None
- mesh_coords, mesh_dim = [], None
- mesh_name = getattr(cf_var, "mesh", None)
- if mesh_name is not None:
- try:
- mesh = meshes[mesh_name]
- except KeyError:
- message = (
- f"Mesh '{mesh_name}' - "
- f"referenced by variable: '{cf_var.cf_name}' - "
- "could not be found in file."
- )
- logger.debug(message)
-
- if mesh is not None:
- # Unconventional 'split' usage of _add_or_capture -
- # attribute handling means MeshCoords need to be built
- # BEFORE loading the Cube.
- capture_kwargs = dict(
- cf_var=cf.cf_group.meshes[mesh_name],
- # MeshCoords are an Iris concept; the best fallback we
- # have is to capture the CF Mesh.
- destination=LoadProblems.Problem.Destination(
- iris_class=Cube,
- identifier=cf_var.cf_name,
- ),
- )
-
- def _build_mesh_coords_inner():
- nonlocal mesh_coords
- nonlocal mesh_dim
- mesh_coords, mesh_dim = _build_mesh_coords(mesh, cf_var)
-
- def _add_mesh_coords(coords_and_dim):
- coords, dim = coords_and_dim
- for coord in coords:
- cube.add_aux_coord(coord, dim)
-
- # MeshCoords part 1.
- _ = _add_or_capture(
- build_func=partial(_build_mesh_coords_inner),
- add_method=partial(lambda built: None),
- **capture_kwargs,
- )
+ # MeshCoords are built before the cube and attached after it:
+ # see _prepare_mesh_coords for why the two cannot be one step.
+ pending_mesh_coords = _prepare_mesh_coords(cf, meshes, cf_var)
cube = _load_cube(engine, cf, cf_var, cf.filename)
- if mesh is not None:
- # MeshCoords part 2.
- _ = _add_or_capture(
- build_func=partial(lambda: (mesh_coords, mesh_dim)),
- add_method=partial(_add_mesh_coords),
- **capture_kwargs,
- )
+ if pending_mesh_coords is not None:
+ _attach_mesh_coords(pending_mesh_coords, cube)
# Process any associated formula terms and attach
# the corresponding AuxCoordFactory.
@@ -813,181 +209,3 @@ def _add_mesh_coords(coords_and_dim):
continue
yield cube
-
-
-class ChunkControl(threading.local):
- """Provide user control of Chunk Control."""
-
- class Modes(Enum):
- """Modes Enums."""
-
- DEFAULT = auto()
- FROM_FILE = auto()
- AS_DASK = auto()
-
- def __init__(self, var_dim_chunksizes=None):
- """Provide user control of Dask chunking.
-
- The NetCDF loader is controlled by the single instance of this: the
- :data:`~iris.fileformats.netcdf.loader.CHUNK_CONTROL` object.
-
- A chunk size can be set for a specific (named) file dimension, when
- loading specific (named) variables, or for all variables.
-
- When a selected variable is a CF data-variable, which loads as a
- :class:`~iris.cube.Cube`, then the given dimension chunk size is *also*
- fixed for all variables which are components of that :class:`~iris.cube.Cube`,
- i.e. any :class:`~iris.coords.Coord`, :class:`~iris.coords.CellMeasure`,
- :class:`~iris.coords.AncillaryVariable` etc.
- This can be overridden, if required, by variable-specific settings.
-
- For this purpose, :class:`~iris.mesh.MeshCoord` and
- :class:`~iris.mesh.Connectivity` are not
- :class:`~iris.cube.Cube` components, and chunk control on a
- :class:`~iris.cube.Cube` data-variable will not affect them.
-
- """
- self.var_dim_chunksizes = var_dim_chunksizes or {}
- self.mode = self.Modes.DEFAULT
-
- @contextmanager
- def set(
- self,
- var_names: str | Iterable[str] | None = None,
- **dimension_chunksizes: Mapping[str, int],
- ) -> Iterator[None]:
- r"""Control the Dask chunk sizes applied to NetCDF variables during loading.
-
- This function can also be used to provide a size hint for the unknown
- array lengths when loading "variable-length" NetCDF data types.
- See https://unidata.github.io/netcdf4-python/#netCDF4.Dataset.vltypes
-
- Parameters
- ----------
- var_names : str or list of str, default=None
- Apply the `dimension_chunksizes` controls only to these variables,
- or when building :class:`~iris.cube.Cube` from these data variables.
- If ``None``, settings apply to all loaded variables.
- **dimension_chunksizes : dict of {str: int}
- Kwargs specifying chunksizes for dimensions of file variables.
- Each key-value pair defines a chunk size for a named file
- dimension, e.g. ``{'time': 10, 'model_levels':1}``.
- Values of ``-1`` will lock the chunk size to the full size of that
- dimension. To specify a size hint for "variable-length" data types
- use the special name `_vl_hint`.
-
- Notes
- -----
- This function acts as a context manager, for use in a ``with`` block.
-
- >>> import iris
- >>> from iris.fileformats.netcdf.loader import CHUNK_CONTROL
- >>> with CHUNK_CONTROL.set("air_temperature", time=180, latitude=-1):
- ... cube = iris.load(iris.sample_data_path("E1_north_america.nc"))[0]
-
- When `var_names` is present, the chunk size adjustments are applied
- only to the selected variables. However, for a CF data variable, this
- extends to all components of the (raw) :class:`~iris.cube.Cube` created
- from it.
-
- **Un**-adjusted dimensions have chunk sizes set in the 'usual' way.
- That is, according to the normal behaviour of
- :func:`iris._lazy_data.as_lazy_data`, which is: chunk size is based on
- the file variable chunking, or full variable shape; this is scaled up
- or down by integer factors to best match the Dask default chunk size,
- i.e. the setting configured by
- ``dask.config.set({'array.chunk-size': '250MiB'})``.
-
- For variable-length data types the size of the variable (or "ragged")
- dimension of the individual array elements cannot be known without
- reading the data. This can make it difficult for Iris to determine
- whether to load the data lazily or not. If the user has some apriori
- knowledge of the mean variable array length this can be passed as
- as a size hint via the special `_vl_hint` name. For example a hint
- that variable-length string array that contains 4 character experiment
- identifiers:
- ``CHUNK_CONTROL.set("expver", _vl_hint=4)``
-
- """
- old_mode = self.mode
- old_var_dim_chunksizes = deepcopy(self.var_dim_chunksizes)
- if var_names is None:
- var_names = ["*"]
- elif isinstance(var_names, str):
- var_names = [var_names]
- try:
- for var_name in var_names:
- # Note: here we simply treat '*' as another name.
- # A specific name match should override a '*' setting, but
- # that is implemented elsewhere.
- if not isinstance(var_name, str):
- msg = ( # type: ignore[unreachable]
- "'var_names' should be an iterable of strings, "
- f"not {var_names!r}."
- )
- raise ValueError(msg)
- dim_chunks = self.var_dim_chunksizes.setdefault(var_name, {})
- for dim_name, chunksize in dimension_chunksizes.items():
- if not (isinstance(dim_name, str) and isinstance(chunksize, int)): # type: ignore[redundant-expr]
- msg = (
- "'dimension_chunksizes' kwargs should be a dict "
- f"of `str: int` pairs, not {dimension_chunksizes!r}."
- )
- raise ValueError(msg)
- dim_chunks[dim_name] = chunksize
- yield
- finally:
- self.var_dim_chunksizes = old_var_dim_chunksizes
- self.mode = old_mode
-
- @contextmanager
- def from_file(self) -> Iterator[None]:
- r"""Ensure the chunk sizes are loaded in from NetCDF file variables.
-
- Raises
- ------
- KeyError
- If any NetCDF data variables - those that become
- :class:`~iris.cube.Cube` - do not specify chunk sizes.
-
- Notes
- -----
- This function acts as a context manager, for use in a ``with`` block.
- """
- old_mode = self.mode
- old_var_dim_chunksizes = deepcopy(self.var_dim_chunksizes)
- try:
- self.mode = self.Modes.FROM_FILE
- yield
- finally:
- self.mode = old_mode
- self.var_dim_chunksizes = old_var_dim_chunksizes
-
- @contextmanager
- def as_dask(self) -> Iterator[None]:
- """Rely on Dask :external+dask:doc:`array` to control chunk sizes.
-
- Notes
- -----
- This function acts as a context manager, for use in a ``with`` block.
-
- """
- old_mode = self.mode
- old_var_dim_chunksizes = deepcopy(self.var_dim_chunksizes)
- try:
- self.mode = self.Modes.AS_DASK
- yield
- finally:
- self.mode = old_mode
- self.var_dim_chunksizes = old_var_dim_chunksizes
-
-
-# Note: the CHUNK_CONTROL object controls chunk sizing in the
-# :meth:`_get_cf_var_data` method.
-# N.B. :meth:`_load_cube` also modifies this when loading each cube,
-# introducing an additional context in which any cube-specific settings are
-# 'promoted' into being global ones.
-
-#: The global :class:`ChunkControl` object providing user-control of Dask chunking
-#: when Iris loads NetCDF files.
-CHUNK_CONTROL: ChunkControl = ChunkControl()
diff --git a/lib/iris/fileformats/netcdf/saver.py b/lib/iris/fileformats/netcdf/saver.py
index 232bfce66a..ccb5f5f08e 100644
--- a/lib/iris/fileformats/netcdf/saver.py
+++ b/lib/iris/fileformats/netcdf/saver.py
@@ -9,8 +9,9 @@
API reference
-Module to support the saving of Iris cubes to a NetCDF file, also using the CF
-conventions for metadata interpretation.
+netCDF's half of saving: the file to write into, the keywords netCDF4 accepts,
+and the four things netCDF does differently from a generic CF store. The CF
+data model lives in :mod:`iris.fileformats.cf.saver`, which this builds on.
See : `NetCDF User's Guide <https://docs.unidata.ucar.edu/nug/current/>`_
and `netCDF4 python module <https://github.com/Unidata/netcdf4-python>`_.
@@ -19,318 +20,118 @@
"""
-import codecs
-import collections
from itertools import repeat, zip_longest
import os
from pathlib import Path
-import re
-import string
import typing
-from typing import TYPE_CHECKING
from urllib.parse import urlsplit
import warnings
-import cf_units
-import dask
import dask.array as da
-from dask.delayed import Delayed
import numpy as np
-from iris import FUTURE
from iris._deprecation import warn_deprecated
-from iris._lazy_data import _co_realise_lazy_arrays, is_lazy_data
-from iris.aux_factory import (
- AtmosphereSigmaFactory,
- HybridHeightFactory,
- HybridPressureFactory,
- OceanSFactory,
- OceanSg1Factory,
- OceanSg2Factory,
- OceanSigmaFactory,
- OceanSigmaZFactory,
-)
+from iris._lazy_data import is_lazy_data
import iris.config
-import iris.coord_systems
-import iris.coords
-from iris.coords import (
- AncillaryVariable,
- AuxCoord,
- CellMeasure,
- DimCoord,
- _DimensionalMetadata,
-)
-import iris.exceptions
import iris.fileformats.cf
+from iris.fileformats.cf import saver as cf_saver
+from iris.fileformats.cf.dataset import CFDatasetVariable
+
+# Beside _CFSaver, SaverFillValueWarning also moved to iris.fileformats.cf.saver,
+# and is imported back rather than forwarded, so that reading it here stays
+# silent. It is public, and spec section 4.8 does not relocate it, so
+# iris.fileformats.netcdf.saver.SaverFillValueWarning must keep resolving.
+from iris.fileformats.cf.saver import SaverFillValueWarning, _CFSaver
from iris.fileformats.netcdf import _bytecoding_datasets as bytecoding_datasets
-from iris.fileformats.netcdf import _dask_locks
-from iris.fileformats.netcdf import _thread_safe_nc as threadsafe_nc
-from iris.fileformats.netcdf._attribute_handlers import ATTRIBUTE_HANDLERS
-from iris.fileformats.netcdf._bytecoding_datasets import (
- EncodedVariable,
- VariableEncoder,
-)
+from iris.fileformats.netcdf._bytecoding_datasets import VariableEncoder
+from iris.fileformats.netcdf._dataset import NetCDFDataset, NetCDFDatasetVariable
import iris.util
import iris.warnings
-if TYPE_CHECKING:
- from iris.cube import Cube
- from iris.mesh.components import (
- MeshEdgeCoords,
- MeshFaceCoords,
- MeshNodeCoords,
- MeshXY,
- )
# Get the logger : shared logger for all in 'iris.fileformats.netcdf'.
from . import logger
-# Avoid warning about unused import.
+# Avoid warnings about unused imports.
# We could use an __all__, but we don't want to maintain one here
logger
-
-# Standard CML spatio-temporal axis names.
-SPATIO_TEMPORAL_AXES = ["t", "z", "y", "x"]
-
-# The CF-meaningful attributes which may appear on a data variable.
-_CF_ATTRS = [
- "add_offset",
- "ancillary_variables",
- "axis",
- "bounds",
- "calendar",
- "cell_measures",
- "cell_methods",
- "climatology",
- "compress",
- "coordinates",
- "_FillValue",
- "formula_terms",
- "grid_mapping",
- "leap_month",
- "leap_year",
- "long_name",
- "missing_value",
- "month_lengths",
- "scale_factor",
- "standard_error_multiplier",
- "standard_name",
- "units",
-]
-
-# CF attributes that should not be global.
-_CF_DATA_ATTRS = [
- "flag_masks",
- "flag_meanings",
- "flag_values",
- "instance_dimension",
- "missing_value",
- "sample_dimension",
- "standard_error_multiplier",
-]
-
-# CF attributes that should only be global.
-_CF_GLOBAL_ATTRS = ["conventions", "featureType", "history", "title"]
-
-# UKMO specific attributes that should not be global.
-_UKMO_DATA_ATTRS = [
- "STASH",
- "um_stash_source",
- "ukmo__process_flags",
- "iris_extended_grid_mapping",
-]
-
-# TODO: whenever we advance to CF-1.11 we should then discuss a completion date
-# for the deprecation of Rotated Mercator in coord_systems.py and
-# _nc_load_rules/helpers.py .
-CF_CONVENTIONS_VERSION = "CF-1.7"
-
-_FactoryDefn = collections.namedtuple(
- "_FactoryDefn", ("primary", "std_name", "formula_terms_format")
+SaverFillValueWarning
+
+#: Names that moved to iris.fileformats.cf.saver, and are still reachable
+#: here. Spec section 4.8.
+_RELOCATED_NAMES = frozenset(
+ {
+ "CFNameCoordMap",
+ "CF_CONVENTIONS_VERSION",
+ "MESH_ELEMENTS",
+ "SPATIO_TEMPORAL_AXES",
+ }
)
-_FACTORY_DEFNS = {
- AtmosphereSigmaFactory: _FactoryDefn(
- primary="sigma",
- std_name="atmosphere_sigma_coordinate",
- formula_terms_format="ptop: {pressure_at_top} sigma: {sigma} "
- "ps: {surface_air_pressure}",
- ),
- HybridHeightFactory: _FactoryDefn(
- primary="delta",
- std_name="atmosphere_hybrid_height_coordinate",
- formula_terms_format="a: {delta} b: {sigma} orog: {orography}",
- ),
- HybridPressureFactory: _FactoryDefn(
- primary="delta",
- std_name="atmosphere_hybrid_sigma_pressure_coordinate",
- formula_terms_format="ap: {delta} b: {sigma} ps: {surface_air_pressure}",
- ),
- OceanSigmaZFactory: _FactoryDefn(
- primary="zlev",
- std_name="ocean_sigma_z_coordinate",
- formula_terms_format="sigma: {sigma} eta: {eta} depth: {depth} "
- "depth_c: {depth_c} nsigma: {nsigma} zlev: {zlev}",
- ),
- OceanSigmaFactory: _FactoryDefn(
- primary="sigma",
- std_name="ocean_sigma_coordinate",
- formula_terms_format="sigma: {sigma} eta: {eta} depth: {depth}",
- ),
- OceanSFactory: _FactoryDefn(
- primary="s",
- std_name="ocean_s_coordinate",
- formula_terms_format="s: {s} eta: {eta} depth: {depth} a: {a} b: {b} "
- "depth_c: {depth_c}",
- ),
- OceanSg1Factory: _FactoryDefn(
- primary="s",
- std_name="ocean_s_coordinate_g1",
- formula_terms_format="s: {s} c: {c} eta: {eta} depth: {depth} "
- "depth_c: {depth_c}",
- ),
- OceanSg2Factory: _FactoryDefn(
- primary="s",
- std_name="ocean_s_coordinate_g2",
- formula_terms_format="s: {s} c: {c} eta: {eta} depth: {depth} "
- "depth_c: {depth_c}",
- ),
-}
-
-
-class CFNameCoordMap:
- """Provide a simple CF name to CF coordinate mapping."""
-
- _Map = collections.namedtuple("_Map", ["name", "coord"])
-
- def __init__(self):
- self._map = []
-
- def append(self, name, coord):
- """Append the given name and coordinate pair to the mapping.
-
- Parameters
- ----------
- name :
- CF name of the associated coordinate.
- coord :
- The coordinate of the associated CF name.
-
- Returns
- -------
- None.
-
- """
- self._map.append(CFNameCoordMap._Map(name, coord))
-
- @property
- def names(self):
- """Return all the CF names."""
- return [pair.name for pair in self._map]
- @property
- def coords(self):
- """Return all the coordinates."""
- return [pair.coord for pair in self._map]
- def name(self, coord):
- """Return the CF name, given a coordinate, or None if not recognised.
+def __getattr__(name):
+ """Forward a relocated name to :mod:`iris.fileformats.cf.saver`.
- Parameters
- ----------
- coord :
- The coordinate of the associated CF name.
-
- Returns
- -------
- Coordinate or None.
-
- """
- result = None
- for pair in self._map:
- if coord == pair.coord:
- result = pair.name
- break
- return result
+ .. deprecated:: 3.17
+ Four saver names moved to :mod:`iris.fileformats.cf.saver`, because
+ they are not specific to netCDF. Reading them here still works, and
+ warns.
- def coord(self, name):
- """Return the coordinate, given a CF name, or None if not recognised.
-
- Parameters
- ----------
- name :
- CF name of the associated coordinate, or None if not recognised.
-
- Returns
- -------
- CF name or None.
-
- """
- result = None
- for pair in self._map:
- if name == pair.name:
- result = pair.coord
- break
- return result
-
-
-def _bytes_if_ascii(string):
- """Convert string to a byte string (str in py2k, bytes in py3k).
-
- Convert the given string to a byte string (str in py2k, bytes in py3k)
- if the given string can be encoded to ascii, else maintain the type
- of the inputted string.
-
- Note: passing objects without an `encode` method (such as None) will
- be returned by the function unchanged.
-
- """
- if isinstance(string, str):
- try:
- return string.encode(encoding="ascii")
- except (AttributeError, UnicodeEncodeError):
- pass
- return string
-
-
-def _setncattr(variable, name, attribute):
- """Put the given attribute on the given netCDF4 Data type.
+ Parameters
+ ----------
+ name : str
+ The attribute being looked up. Python calls this only for names that
+ are not already module attributes.
- Put the given attribute on the given netCDF4 Data type, casting
- attributes as we go to bytes rather than unicode.
+ Returns
+ -------
+ Any
+ The object of that name in :mod:`iris.fileformats.cf.saver` - the same
+ object, not a copy.
- NOTE: variable needs to be a _thread_safe_nc._ThreadSafeWrapper subclass.
+ Raises
+ ------
+ AttributeError
+ If ``name`` is not one of the relocated names. Raised without warning,
+ so that a ``hasattr`` probe that comes back False stays quiet.
"""
- assert hasattr(variable, "THREAD_SAFE_FLAG")
- attribute = _bytes_if_ascii(attribute)
- return variable.setncattr(name, attribute)
-
-
-# NOTE : this matches :class:`iris.mesh.MeshXY.ELEMENTS`,
-# but in the preferred order for coord/connectivity variables in the file.
-MESH_ELEMENTS = ("node", "edge", "face")
-
+ if name not in _RELOCATED_NAMES:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ warn_deprecated(
+ f"iris.fileformats.netcdf.saver.{name} has moved to "
+ f"iris.fileformats.cf.saver.{name}, because it is not specific to "
+ "netCDF. The name here will be removed in a future release.",
+ # warn_deprecated's default stacklevel=2 points at the line below,
+ # inside this function - PEP 562 means that line is a genuine call
+ # frame, not the user's code. One level further out reaches the
+ # line that actually read the attribute.
+ stacklevel=3,
+ )
+ return getattr(cf_saver, name)
-class SaverFillValueWarning(iris.warnings.IrisSaverFillValueWarning):
- """Backwards compatible form of :class:`iris.warnings.IrisSaverFillValueWarning`."""
- # TODO: remove at the next major release.
- pass
+def __dir__():
+ """List module attributes, including the relocated names.
+ A module ``__getattr__`` forwards reads of the relocated names, but does
+ not make them discoverable: PEP 562 does not extend to ``dir()``,
+ :func:`inspect.getmembers`, or tab-completion, so this fills the gap.
-class VariableEmulator(typing.Protocol):
- """Duck-type-hinting for a ncdata object.
+ Returns
+ -------
+ list of str
+ The real module globals, unioned with the relocated names.
- https://github.com/pp-mo/ncdata
"""
+ return sorted(set(globals()) | _RELOCATED_NAMES)
- _data_array: np.typing.ArrayLike
- shape: tuple[int, ...]
+# TODO: becomes a Union once a Zarr CFDataset implementation exists.
+CFVariable = NetCDFDatasetVariable
-CFVariable = typing.Union[bytecoding_datasets.VariableWrapper, VariableEmulator]
-
-class Saver:
+class Saver(_CFSaver):
"""A manager for saving NetCDF/NcZarr files."""
def __init__(self, filename, netcdf_format, compute=True):
@@ -392,35 +193,8 @@ def __init__(self, filename, netcdf_format, compute=True):
]:
raise ValueError("Unknown netCDF file format, got %r" % netcdf_format)
- # All persistent variables
- #: CF name mapping with iris coordinates
- self._name_coord_map = CFNameCoordMap()
- #: Map of dimensions to characteristic coordinates with which they are identified
- self._dim_names_and_coords = CFNameCoordMap()
- #: List of grid mappings added to the file
- self._coord_systems = []
- #: A dictionary, listing dimension names and corresponding length
- self._existing_dim = {}
- #: A map from meshes to their actual file dimensions (names).
- # NB: might not match those of the mesh, if they were 'incremented'.
- self._mesh_dims = {}
- #: A dictionary, mapping formula terms to owner cf variable name
- self._formula_terms_cache = {}
- #: Target filepath
- self.filepath = None # this line just for the API page -- value is set later
- #: Whether to complete delayed saves on exit.
- self.compute = compute
- # N.B. the file-write-lock *type* actually depends on the dask scheduler type.
- #: A per-file write lock to prevent dask attempting overlapping writes.
- self.file_write_lock = (
- None # this line just for the API page -- value is set later
- )
-
#: Whether the save target is an NCZarr URL.
self._is_nczarr = False
- # A list of delayed writes for lazy saving
- # a list of couples (source, target).
- self._delayed_writes = []
# A list of pre-close writes for NCZarr lazy saving.
# NCZarr cannot be reopened for deferred writes, so lazy data is
# written before __exit__ closes the file.
@@ -430,7 +204,6 @@ def __init__(self, filename, netcdf_format, compute=True):
self._to_open_dataset = hasattr(filename, "createVariable")
if self._to_open_dataset:
# We were passed a *dataset*, so we don't open (or close) one of our own.
- self._dataset = filename
if compute:
msg = (
"Cannot save to a user-provided dataset with 'compute=True'. "
@@ -439,15 +212,11 @@ def __init__(self, filename, netcdf_format, compute=True):
)
raise ValueError(msg)
- # Put it inside a _thread_safe_nc wrapper to ensure thread-safety.
- # Except if it already is one, since they forbid "re-wrapping".
- if not hasattr(self._dataset, "THREAD_SAFE_FLAG"):
- self._dataset = bytecoding_datasets.EncodedDataset.from_existing(
- self._dataset
- )
+ # from_existing() handles the thread-safety wrapping.
+ dataset = NetCDFDataset.from_existing(filename)
# In this case the dataset gives a filepath, not the other way around.
- self.filepath = self._dataset.filepath()
+ filepath = dataset.location
else:
# Given a filepath string/path : create a dataset from that
@@ -460,46 +229,44 @@ def __init__(self, filename, netcdf_format, compute=True):
)
if self._is_nczarr:
# NCZarr URLs contain a #mode= fragment; Path() strips it.
- # Keep as a plain string and pass directly to DatasetWrapper.
- self.filepath = str(filename)
+ # Keep as a plain string and pass directly to the dataset.
+ filepath = str(filename)
else:
- filepath = Path(filename)
- self.filepath = filepath.absolute()
- self._dataset = bytecoding_datasets.EncodedDataset(
- self.filepath, mode="w", format=netcdf_format
- )
+ filepath = Path(filename).absolute()
+ dataset = NetCDFDataset(filepath, mode="w", netcdf_format=netcdf_format)
except RuntimeError:
if self._is_nczarr:
raise
- dir_name = Path(self.filepath).parent
+ dir_name = Path(filepath).parent
if not dir_name.is_dir():
msg = "No such file or directory: {}".format(dir_name)
raise IOError(msg)
if not os.access(dir_name, os.R_OK | os.W_OK):
- msg = "Permission denied: {}".format(self.filepath)
+ msg = "Permission denied: {}".format(filepath)
raise IOError(msg)
else:
raise
- self.file_write_lock = _dask_locks.get_worker_lock(self.filepath)
+ super().__init__(
+ dataset, compute=compute, owns_dataset=not self._to_open_dataset
+ )
- def __enter__(self):
- return self
+ #: Target filepath. A Path for a local file, which is public API:
+ #: the base class has only the dataset's string form of it.
+ self.filepath = filepath
+
+ # One lock for the whole file, shared with every variable of it.
+ # N.B. the file-write-lock *type* actually depends on the dask scheduler type.
+ #: A per-file write lock to prevent dask attempting overlapping writes.
+ self.file_write_lock = dataset.write_lock
- def __exit__(self, type, value, traceback):
- """Flush any buffered data to the CF-NetCDF/NcZarr file before closing."""
+ def _flush_before_close(self) -> None:
+ """Write anything that cannot be written after the dataset closes."""
if self._nczarr_writes:
# NCZarr lazy writes must be computed while the file is still open;
# the deferred reopen-write pattern used for netCDF is not supported.
sources, targets = zip(*self._nczarr_writes)
da.store(list(sources), list(targets))
- self._dataset.sync()
- if not self._to_open_dataset:
- # Only close if the Saver created it.
- self._dataset.close()
- # Complete after closing, if required
- if self.compute:
- self.complete()
def write(
self,
@@ -619,2004 +386,129 @@ def write(
3 files that do not use HDF5.
"""
- # TODO: when iris.FUTURE.save_split_attrs defaults to True, we can deprecate the
- # "local_keys" arg, and finally remove it when we finally remove the
- # save_split_attrs switch.
- if unlimited_dimensions is None:
- unlimited_dimensions = []
-
- cf_profile_available = iris.site_configuration.get("cf_profile") not in [
- None,
- False,
- ]
- if cf_profile_available:
- # Perform a CF profile of the cube. This may result in an exception
- # being raised if mandatory requirements are not satisfied.
- profile = iris.site_configuration["cf_profile"](cube)
-
- # Ensure that attributes are CF compliant and if possible to make them
- # compliant.
- self.check_attribute_compliance(cube, cube.dtype)
- for coord in cube.coords():
- self.check_attribute_compliance(coord, coord.dtype)
-
- # Get suitable dimension names.
- mesh_dimensions, cube_dimensions = self._get_dim_names(cube)
-
- # Create all the CF-netCDF data dimensions.
- # Put mesh dims first, then non-mesh dims in cube-occurring order.
- nonmesh_dimensions = [
- dim for dim in cube_dimensions if dim not in mesh_dimensions
- ]
- all_dimensions = mesh_dimensions + nonmesh_dimensions
- self._create_cf_dimensions(cube, all_dimensions, unlimited_dimensions)
-
- # Group the generic compression keyword arguments together for
- # convenience, as they will be applied to other cube metadata
- # as well as the cube data payload.
- compression_kwargs = {
- "complevel": complevel,
- "fletcher32": fletcher32,
- "shuffle": shuffle,
- "zlib": zlib,
- }
-
- # Create the mesh components, if there is a mesh.
- # We do this before creating the data-var, so that mesh vars precede
- # data-vars in the file.
- cf_mesh_name = self._add_mesh(cube, compression_kwargs=compression_kwargs)
-
- # Create the associated cube CF-netCDF data variable.
- cf_var_cube = self._create_cf_data_variable(
+ super().write(
cube,
- cube_dimensions,
local_keys=local_keys,
+ unlimited_dimensions=unlimited_dimensions,
packing=packing,
fill_value=fill_value,
+ # Applied to every variable the cube creates, not just its data.
+ compression_kwargs={
+ "complevel": complevel,
+ "fletcher32": fletcher32,
+ "shuffle": shuffle,
+ "zlib": zlib,
+ },
contiguous=contiguous,
chunksizes=chunksizes,
endian=endian,
least_significant_digit=least_significant_digit,
- **compression_kwargs,
- )
-
- # Associate any mesh with the data-variable.
- # N.B. _add_mesh cannot do this, as we want to put mesh variables
- # before data-variables in the file.
- if cf_mesh_name is not None:
- _setncattr(cf_var_cube, "mesh", cf_mesh_name)
- _setncattr(cf_var_cube, "location", cube.location)
-
- # Add coordinate variables.
- self._add_dim_coords(cube, cube_dimensions)
-
- # Add the auxiliary coordinate variables and associate the data
- # variable to them
- self._add_aux_coords(
- cube, cf_var_cube, cube_dimensions, compression_kwargs=compression_kwargs
- )
-
- # Add the cell_measures variables and associate the data
- # variable to them
- self._add_cell_measures(cube, cf_var_cube, cube_dimensions)
-
- # Add the ancillary_variables variables and associate the data variable
- # to them
- self._add_ancillary_variables(
- cube, cf_var_cube, cube_dimensions, compression_kwargs=compression_kwargs
)
- # Add the formula terms to the appropriate cf variables for each
- # aux factory in the cube.
- self._add_aux_factories(cube, cf_var_cube, cube_dimensions)
-
- if not iris.FUTURE.save_split_attrs:
- # In the "old" way, we update global attributes as we go.
- # Add data variable-only attribute names to local_keys.
- if local_keys is None:
- local_keys = set()
- else:
- local_keys = set(local_keys)
- local_keys.update(_CF_DATA_ATTRS, _UKMO_DATA_ATTRS, ["GRIB_PARAM"])
-
- # Add global attributes taking into account local_keys.
- # NOTE: we never need to "convert" managed global attributes, as they are all
- # data-local type, i.e. included in "local_keys" above.
- cube_attributes = cube.attributes
- global_attributes = {
- k: v
- for k, v in cube_attributes.items()
- if (k not in local_keys and k.lower() != "conventions")
- }
- self.update_global_attributes(global_attributes)
-
- if cf_profile_available:
- cf_patch = iris.site_configuration.get("cf_patch")
- if cf_patch is not None:
- # Perform a CF patch of the dataset.
- cf_patch(profile, self._dataset, cf_var_cube)
- else:
- msg = "cf_profile is available but no {} defined.".format("cf_patch")
- warnings.warn(msg, category=iris.warnings.IrisCfSaveWarning)
-
- def check_attribute_compliance(self, container, data_dtype):
- """Check attributte complliance."""
-
- def _coerce_value(val_attr, val_attr_value, data_dtype):
- val_attr_tmp = np.array(val_attr_value, dtype=data_dtype)
- if (val_attr_tmp != val_attr_value).any():
- msg = '"{}" is not of a suitable value ({})'
- raise ValueError(msg.format(val_attr, val_attr_value))
- return val_attr_tmp
-
- # Ensure that conflicting attributes are not provided.
- if (
- container.attributes.get("valid_min") is not None
- or container.attributes.get("valid_max") is not None
- ) and container.attributes.get("valid_range") is not None:
- msg = (
- 'Both "valid_range" and "valid_min" or "valid_max" attributes present.'
- )
- raise ValueError(msg)
-
- # Ensure correct datatype
- for val_attr in ["valid_range", "valid_min", "valid_max"]:
- val_attr_value = container.attributes.get(val_attr)
- if val_attr_value is not None:
- val_attr_value = np.asarray(val_attr_value)
- self._ensure_valid_dtype(val_attr_value, val_attr, val_attr_value)
- if data_dtype.itemsize == 1:
- # Allow signed integral type
- if val_attr_value.dtype.kind == "i":
- continue
- new_val = _coerce_value(val_attr, val_attr_value, data_dtype)
- container.attributes[val_attr] = new_val
-
- def update_global_attributes(self, attributes=None, **kwargs):
- """Update the CF global attributes.
-
- Update the CF global attributes based on the provided
- iterable/dictionary and/or keyword arguments.
-
- Parameters
- ----------
- attributes : dict or iterable of key, value pairs, optional
- CF global attributes to be updated.
- """
- # TODO: when when iris.FUTURE.save_split_attrs is removed, this routine will
- # only be called once: it can reasonably be renamed "_set_global_attributes",
- # and the 'kwargs' argument can be removed.
- if attributes is not None:
- # Handle sequence e.g. [('fruit', 'apple'), ...].
- if not hasattr(attributes, "keys"):
- attributes = dict(attributes)
-
- for attr_name in sorted(attributes):
- _setncattr(self._dataset, attr_name, attributes[attr_name])
-
- for attr_name in sorted(kwargs):
- _setncattr(self._dataset, attr_name, kwargs[attr_name])
-
- def _create_cf_dimensions(self, cube, dimension_names, unlimited_dimensions=None):
- """Create the CF-netCDF data dimensions.
-
- Parameters
- ----------
- cube : :class:`iris.cube.Cube`
- A :class:`iris.cube.Cube` in which to lookup coordinates.
- dimension_names :
- unlimited_dimensions : iterable of strings and/or :class:`iris.coords.Coord` objects):
- List of coordinates to make unlimited (None by default).
-
- Returns
- -------
- None.
-
- """
- unlimited_dim_names = []
- if unlimited_dimensions is not None:
- for coord in unlimited_dimensions:
- try:
- coord = cube.coord(name_or_coord=coord, dim_coords=True)
- except iris.exceptions.CoordinateNotFoundError:
- # coordinate isn't used for this cube, but it might be
- # used for a different one
- pass
- else:
- dim_name = self._get_element_variable_name(cube, coord)
- unlimited_dim_names.append(dim_name)
-
- for dim_name in dimension_names:
- # NOTE: these dim-names have been chosen by _get_dim_names, and
- # were already checked+fixed to avoid any name collisions.
- if dim_name not in self._dataset.dimensions:
- if dim_name in unlimited_dim_names:
- size = None
- else:
- size = self._existing_dim[dim_name]
- self._dataset.createDimension(dim_name, size)
-
- def _add_mesh(self, cube_or_mesh, /, *, compression_kwargs=None):
- """Add the cube's mesh, and all related variables to the dataset.
-
- Add the cube's mesh, and all related variables to the dataset.
- Includes all the mesh-element coordinate and connectivity variables.
-
- .. note::
-
- Here, we do *not* add the relevant referencing attributes to the
- data-variable, because we want to create the data-variable later.
-
- Parameters
- ----------
- cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
- The Cube or Mesh being saved to the netCDF file.
- compression_kwargs : dict, optional
- NetCDF data compression keyword arguments.
-
- Returns
- -------
- str or None
- The name of the mesh variable created, or None if the cube does not
- have a mesh.
-
- """
- cf_mesh_name = None
-
- # Do cube- or -mesh-based save
- from iris.cube import Cube
- from iris.mesh.components import _MeshIndexSet
-
- if isinstance(cube_or_mesh, Cube):
- mesh = cube_or_mesh.mesh
- else:
- mesh = cube_or_mesh
-
- if isinstance(mesh, _MeshIndexSet):
- msg = "_MeshIndexSet saving is not yet supported. See https://github.com/SciTools/iris/issues/6123"
- raise ValueError(msg)
-
- if mesh:
- cf_mesh_name = self._name_coord_map.name(mesh)
- if cf_mesh_name is None:
- # Not already present : create it
- cf_mesh_name = self._create_mesh(mesh)
- self._name_coord_map.append(cf_mesh_name, mesh)
-
- cf_mesh_var = self._dataset.variables[cf_mesh_name]
-
- # Get the mesh-element dim names.
- mesh_dims = self._mesh_dims[mesh]
-
- # Add all the element coordinate variables.
- for location in MESH_ELEMENTS:
- coords_meshobj_attr = f"{location}_coords"
- coords_file_attr = f"{location}_coordinates"
- mesh_coords = getattr(mesh, coords_meshobj_attr, None)
- if mesh_coords:
- coord_names = []
- for coord in mesh_coords:
- if coord is None:
- continue # an awkward thing that mesh.coords does
- coord_name = self._create_generic_cf_array_var(
- mesh,
- [],
- coord,
- element_dims=(mesh_dims[location],),
- compression_kwargs=compression_kwargs,
- )
- # Only created once per file, but need to fetch the
- # name later in _add_inner_related_vars().
- self._name_coord_map.append(coord_name, coord)
- coord_names.append(coord_name)
- # Record the coordinates (if any) on the mesh variable.
- if coord_names:
- coord_names = " ".join(coord_names)
- _setncattr(cf_mesh_var, coords_file_attr, coord_names)
-
- # Add all the connectivity variables.
- # pre-fetch the set + ignore "None"s, which are empty slots.
- conns = [conn for conn in mesh.all_connectivities if conn is not None]
- for conn in conns:
- # Get the connectivity role, = "{loc1}_{loc2}_connectivity".
- cf_conn_attr_name = conn.cf_role
- loc_from, loc_to, _ = cf_conn_attr_name.split("_")
- # Construct a trailing dimension name.
- last_dim = f"{cf_mesh_name}_{loc_from}_N_{loc_to}s"
- # Create if it does not already exist.
- if last_dim not in self._dataset.dimensions:
- while last_dim in self._dataset.variables:
- # Also avoid collision with variable names.
- # See '_get_dim_names' for reason.
- last_dim = self._increment_name(last_dim)
- length = conn.shape[1 - conn.location_axis]
- self._dataset.createDimension(last_dim, length)
-
- # Create variable.
- # NOTE: for connectivities *with missing points*, this will use a
- # fixed standard fill-value of -1. In that case, we create the
- # variable with a '_FillValue' property, which can only be done
- # when it is first created.
- loc_dim_name = mesh_dims[loc_from]
- conn_dims = (loc_dim_name, last_dim)
- if conn.location_axis == 1:
- # Has the 'other' dimension order, =reversed
- conn_dims = conn_dims[::-1]
- if iris.util.is_masked(conn.core_indices()):
- # Flexible mesh.
- fill_value = -1
- else:
- fill_value = None
- cf_conn_name = self._create_generic_cf_array_var(
- mesh,
- [],
- conn,
- element_dims=conn_dims,
- fill_value=fill_value,
- compression_kwargs=compression_kwargs,
- )
- # Add essential attributes to the Connectivity variable.
- cf_conn_var = self._dataset.variables[cf_conn_name]
- _setncattr(cf_conn_var, "cf_role", cf_conn_attr_name)
- _setncattr(cf_conn_var, "start_index", conn.start_index)
-
- # Record the connectivity on the parent mesh var.
- _setncattr(cf_mesh_var, cf_conn_attr_name, cf_conn_name)
- # If the connectivity had the 'alternate' dimension order, add the
- # relevant dimension property
- if conn.location_axis == 1:
- loc_dim_attr = f"{loc_from}_dimension"
- # Should only get here once.
- assert loc_dim_attr not in cf_mesh_var.ncattrs()
- _setncattr(cf_mesh_var, loc_dim_attr, loc_dim_name)
-
- return cf_mesh_name
-
- def _add_inner_related_vars(
- self,
- cube,
- cf_var_cube,
- dimension_names,
- coordlike_elements,
- /,
- *,
- compression_kwargs=None,
- ) -> None:
- """Create a set of variables for aux-coords, ancillaries or cell-measures.
-
- Create a set of variables for aux-coords, ancillaries or cell-measures,
- and attach them to the parent data variable.
-
- """
- if coordlike_elements:
- # Choose the appropriate parent attribute
- elem_type = type(coordlike_elements[0])
- if elem_type in (AuxCoord, DimCoord):
- role_attribute_name = "coordinates"
- elif elem_type == AncillaryVariable:
- role_attribute_name = "ancillary_variables"
- else:
- # We *only* handle aux-coords, cell-measures and ancillaries
- assert elem_type == CellMeasure
- role_attribute_name = "cell_measures"
-
- # Add CF-netCDF variables for the given cube components.
- element_names = []
- for element in sorted(
- coordlike_elements, key=lambda element: element.name()
- ):
- # Reuse, or create, the associated CF-netCDF variable.
- cf_name = self._name_coord_map.name(element)
- if cf_name is None:
- # Not already present : create it
- cf_name = self._create_generic_cf_array_var(
- cube,
- dimension_names,
- element,
- compression_kwargs=compression_kwargs,
- )
- self._name_coord_map.append(cf_name, element)
-
- if role_attribute_name == "cell_measures":
- # In the case of cell-measures, the attribute entries are not just
- # a var_name, but each have the form "<measure>: <varname>".
- cf_name = "{}: {}".format(element.measure, cf_name)
-
- element_names.append(cf_name)
-
- # Add CF-netCDF references to the primary data variable.
- if element_names:
- variable_names = " ".join(sorted(element_names))
- _setncattr(cf_var_cube, role_attribute_name, variable_names)
-
- def _add_aux_coords(
- self, cube, cf_var_cube, dimension_names, /, *, compression_kwargs=None
- ) -> None:
- """Add aux. coordinate to the dataset and associate with the data variable.
-
- Parameters
- ----------
- cube : :class:`iris.cube.Cube`
- A :class:`iris.cube.Cube` to be saved to a netCDF file.
- cf_var_cube : :class:`netcdf.netcdf_variable`
- A cf variable cube representation.
- dimension_names : list
- Names associated with the dimensions of the cube.
- compression_kwargs : dict, optional
- NetCDF data compression keyword arguments.
-
- """
- # Exclude any mesh coords, which are bundled in with the aux-coords.
- coords_to_add = [
- coord for coord in cube.aux_coords if not hasattr(coord, "mesh")
- ]
-
- # Include any relevant mesh location coordinates.
- mesh: MeshXY | None = getattr(cube, "mesh") # type: ignore[annotation-unchecked]
- mesh_location: str | None = getattr(cube, "location") # type: ignore[annotation-unchecked]
- if mesh and mesh_location:
- location_coords: MeshNodeCoords | MeshEdgeCoords | MeshFaceCoords = getattr(
- mesh, f"{mesh_location}_coords"
- ) # type: ignore[annotation-unchecked]
- coords_to_add.extend(list(location_coords))
-
- return self._add_inner_related_vars(
- cube,
- cf_var_cube,
- dimension_names,
- coords_to_add,
- compression_kwargs=compression_kwargs,
- )
-
- def _add_cell_measures(self, cube, cf_var_cube, dimension_names):
- """Add cell measures to the dataset and associate with the data variable.
-
- Parameters
- ----------
- cube : :class:`iris.cube.Cube`
- A :class:`iris.cube.Cube` to be saved to a netCDF file.
- cf_var_cube : :class:`netcdf.netcdf_variable`
- A cf variable cube representation.
- dimension_names : list
- Names associated with the dimensions of the cube.
- """
- return self._add_inner_related_vars(
- cube,
- cf_var_cube,
- dimension_names,
- cube.cell_measures(),
- )
-
- def _add_ancillary_variables(
- self, cube, cf_var_cube, dimension_names, /, *, compression_kwargs=None
+ def _patch_dataset(self, cf_patch, profile, cf_var_cube) -> None:
+ """Hand the site-configured patch hook the objects it expects."""
+ # A reserved iris.site_configuration hook, documented since
+ # Iris 1.3 as receiving netCDF4 objects - so both arguments
+ # are unwrapped, not just the dataset. A CFVariable would
+ # raise on setncattr() and, worse, silently swallow
+ # 'variable.name = value': it defines no __setattr__.
+ cf_patch(profile, self._dataset.dataset, cf_var_cube.variable)
+
+ def _create_string_variable(
+ self, cube_or_mesh, cf_name, element, element_dims, data
):
- """Add ancillary variables measures to the dataset and associate with the data variable.
-
- Parameters
- ----------
- cube : :class:`iris.cube.Cube`
- A :class:`iris.cube.Cube` to be saved to a netCDF file.
- cf_var_cube : :class:`netcdf.netcdf_variable`
- A cf variable cube representation.
- dimension_names : list
- Names associated with the dimensions of the cube.
- compression_kwargs : dict, optional
- NetCDF data compression keyword arguments.
-
- """
- return self._add_inner_related_vars(
- cube,
- cf_var_cube,
- dimension_names,
- cube.ancillary_variables(),
- compression_kwargs=compression_kwargs,
- )
-
- def _add_dim_coords(self, cube, dimension_names):
- """Add coordinate variables to NetCDF dataset.
-
- Parameters
- ----------
- cube : :class:`iris.cube.Cube`
- A :class:`iris.cube.Cube` to be saved to a netCDF file.
- dimension_names : list
- Names associated with the dimensions of the cube.
- """
- # Ensure we create the netCDF coordinate variables first.
- for coord in cube.dim_coords:
- # Create the associated coordinate CF-netCDF variable.
- if coord not in self._name_coord_map.coords:
- # Not already present : create it
- cf_name = self._create_generic_cf_array_var(
- cube, dimension_names, coord
- )
- self._name_coord_map.append(cf_name, coord)
-
- def _add_aux_factories(self, cube, cf_var_cube, dimension_names):
- """Represent the presence of dimensionless vertical coordinates.
-
- Modify the variables of the NetCDF dataset to represent
- the presence of dimensionless vertical coordinates based on
- the aux factories of the cube (if any).
-
- Parameters
- ----------
- cube : :class:`iris.cube.Cube`
- A :class:`iris.cube.Cube` to be saved to a netCDF file.
- cf_var_cube : :class:`netcdf.netcdf_variable`
- CF variable cube representation.
- dimension_names : list
- Names associated with the dimensions of the cube.
- """
- primaries = []
- for factory in cube.aux_factories:
- factory_defn = _FACTORY_DEFNS.get(type(factory), None)
- if factory_defn is None:
- msg = "Unable to determine formula terms for AuxFactory: {!r}".format(
- factory
- )
- warnings.warn(msg, category=iris.warnings.IrisSaveWarning)
- else:
- # Override `standard_name`, `long_name`, and `axis` of the
- # primary coord that signals the presence of a dimensionless
- # vertical coord, then set the `formula_terms` attribute.
- primary_coord = factory.dependencies[factory_defn.primary]
- if primary_coord in primaries:
- msg = (
- "Cube {!r} has multiple aux factories that share "
- "a common primary coordinate {!r}. Unable to save "
- "to netCDF as having multiple formula terms on a "
- "single coordinate is not supported."
- )
- raise ValueError(msg.format(cube, primary_coord.name()))
- primaries.append(primary_coord)
-
- cf_name = self._name_coord_map.name(primary_coord)
- cf_var = self._dataset.variables[cf_name]
-
- term_varnames = {
- term: self._name_coord_map.name(coord)
- for term, coord in factory.dependencies.items()
- }
- formula_terms = factory_defn.formula_terms_format.format(
- **term_varnames
- )
- std_name = factory_defn.std_name
-
- if hasattr(cf_var, "formula_terms"):
- if (
- cf_var.formula_terms != formula_terms
- or cf_var.standard_name != std_name
- ):
- # TODO: We need to resolve this corner-case where
- # the dimensionless vertical coordinate containing
- # the formula_terms is a dimension coordinate of
- # the associated cube and a new alternatively named
- # dimensionless vertical coordinate is required
- # with new formula_terms and a renamed dimension.
- if cf_name in dimension_names:
- msg = "Unable to create dimensonless vertical coordinate."
- raise ValueError(msg)
- key = (cf_name, std_name, formula_terms)
- name = self._formula_terms_cache.get(key)
- if name is None:
- # Create a new variable
- name = self._create_generic_cf_array_var(
- cube, dimension_names, primary_coord
- )
- cf_var = self._dataset.variables[name]
- _setncattr(cf_var, "standard_name", std_name)
- _setncattr(cf_var, "axis", "Z")
- # Update the formula terms.
- ft = formula_terms.split()
- ft = [name if t == cf_name else t for t in ft]
- _setncattr(cf_var, "formula_terms", " ".join(ft))
- # Update the cache.
- self._formula_terms_cache[key] = name
- # Update the associated cube variable.
- coords = cf_var_cube.coordinates.split()
- coords = [name if c == cf_name else c for c in coords]
- _setncattr(cf_var_cube, "coordinates", " ".join(coords))
- else:
- _setncattr(cf_var, "standard_name", std_name)
- _setncattr(cf_var, "axis", "Z")
- _setncattr(cf_var, "formula_terms", formula_terms)
-
- if FUTURE.derived_bounds:
- # ensure that the primary variable *bounds*, if any, obey the CF
- # encoding rule : the bounds variable of a parametric coordinate
- # must itself have a "formula_terms" attribute.
- # See : https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#boundaries-and-formula-terms
- bounds_varname = getattr(cf_var, "bounds", None)
- cf_bounds_var = self._dataset.variables.get(bounds_varname, None)
- if (
- cf_bounds_var is not None
- and getattr(cf_bounds_var, "formula_terms", None) is None
- ):
- # We need a bounds formula, and there is none already attached.
- # Construct and add one, mirroring the main formula.
- def boundsterm_varname(term_varname):
- """Identify the boundsvar for a given term var."""
- # First establish a fallback default, meaning 'no bounds'.
- result = term_varname
- # Follow links (if they exist) to find the bounds var.
- termvar = self._dataset.variables.get(term_varname)
- boundsname = getattr(termvar, "bounds", None)
- if boundsname in self._dataset.variables:
- result = boundsname
- return result
-
- boundsterm_varnames = {
- term: boundsterm_varname(term_varname)
- for term, term_varname in term_varnames.items()
- }
- bounds_formula_terms = factory_defn.formula_terms_format.format(
- **boundsterm_varnames
- )
- _setncattr(cf_bounds_var, "formula_terms", bounds_formula_terms)
+ """Create the char variable that will hold this unicode string data.
- def _get_dim_names(self, cube_or_mesh):
- """Determine suitable CF-netCDF data dimension names.
+ netCDF has no unicode type, so string data is stored as bytes over an
+ extra trailing dimension whose length is the width of the widest
+ value. The name of that dimension is appended to ``element_dims``.
Parameters
----------
cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
- The Cube or Mesh being saved to the netCDF file.
-
- Returns
- -------
- mesh_dimensions : list of str
- A list of the mesh dimensions of the attached mesh, if any.
- cube_dimensions : list of str
- A lists of dimension names for each dimension of the cube.
-
- Notes
- -----
- The returned lists are in the preferred file creation order.
- One of the mesh dimensions will typically also appear in the cube
- dimensions.
-
- """
-
- def record_dimension(names_list, dim_name, length, matching_coords=None):
- """Record a file dimension, its length and associated "coordinates".
-
- Record a file dimension, its length and associated "coordinates"
- (which may in fact also be connectivities).
-
- If the dimension has been seen already, check that it's length
- matches the earlier finding.
-
- """
- if matching_coords is None:
- matching_coords = []
- if dim_name not in self._existing_dim:
- self._existing_dim[dim_name] = length
- else:
- # Make sure we never re-write one, though it's really not
- # clear how/if this could ever actually happen.
- existing_length = self._existing_dim[dim_name]
- if length != existing_length:
- msg = (
- "Netcdf saving error : existing dimension "
- f'"{dim_name}" has length {existing_length}, '
- f"but occurrence in cube {cube} has length {length}"
- )
- raise ValueError(msg)
-
- # Record given "coords" (sort-of, maybe connectivities) to be
- # identified with this dimension: add to the already-seen list.
- existing_coords = self._dim_names_and_coords.coords
- for coord in matching_coords:
- if coord not in existing_coords:
- self._dim_names_and_coords.append(dim_name, coord)
-
- # Add the latest name to the list passed in.
- names_list.append(dim_name)
-
- # Choose cube or mesh saving.
- from iris.cube import Cube
-
- if isinstance(cube_or_mesh, Cube):
- # there is a mesh, only if the cube has one
- cube = cube_or_mesh
- mesh = cube.mesh
- else:
- # there is no cube, only a mesh
- cube = None
- mesh = cube_or_mesh
-
- # Get info on mesh, first.
- mesh_dimensions = []
- if mesh is None:
- cube_mesh_dim = None
- else:
- # Identify all the mesh dimensions.
- mesh_location_dimnames = {}
- # NOTE: one of these will be a cube dimension, but that one does not
- # get any special handling. We *do* want to list/create them in a
- # definite order (node,edge,face), and before non-mesh dimensions.
- for location in MESH_ELEMENTS:
- # Find if this location exists in the mesh, and a characteristic
- # coordinate to identify it with.
- # To use only _required_ UGRID components, we use a location
- # coord for nodes, but a connectivity for faces/edges
- if location == "node":
- # For nodes, identify the dim with a coordinate variable.
- dim_coords = mesh.coords(location="node")
- else:
- # For face/edge, use the relevant "optionally required"
- # connectivity variable.
- cf_role = f"{location}_node_connectivity"
- dim_coords = mesh.connectivities(cf_role=cf_role)
- if len(dim_coords) > 0:
- # As the mesh contains this location, we want to include this
- # dim in our returned mesh dims.
- # We should have 1 identifying variable (of either type).
- dim_element = dim_coords[0]
- dim_name = self._dim_names_and_coords.name(dim_element)
- if dim_name is None:
- # No existing dim matches this, so assign a new name
- if location == "node":
- # always 1-d
- (dim_length,) = dim_element.shape
- else:
- # extract source dim, respecting dim-ordering
- dim_length = dim_element.shape[dim_element.location_axis]
- # Name it for the relevant mesh dimension
- location_dim_attr = f"{location}_dimension"
- dim_name = getattr(mesh, location_dim_attr)
- # NOTE: This cannot currently be empty, as a MeshXY
- # "invents" dimension names which were not given.
- assert dim_name is not None
- # Ensure it is a valid variable name.
- dim_name = self.cf_valid_var_name(dim_name)
- # Disambiguate if it has the same name as an existing
- # dimension.
- # NOTE: *OR* if it matches the name of an existing file
- # variable. Because there is a bug ...
- # See https://github.com/Unidata/netcdf-c/issues/1772
- # N.B. the workarounds here *ONLY* function because the
- # caller (write) will not create any more variables
- # in between choosing dim names (here), and creating
- # the new dims (via '_create_cf_dimensions').
- while (
- dim_name in self._existing_dim
- or dim_name in self._dataset.variables
- ):
- dim_name = self._increment_name(dim_name)
-
- # Record the new dimension.
- record_dimension(
- mesh_dimensions, dim_name, dim_length, dim_coords
- )
-
- # Store the mesh dims indexed by location
- mesh_location_dimnames[location] = dim_name
-
- if cube is None:
- cube_mesh_dim = None
- else:
- # Finally, identify the cube dimension which maps to the mesh: this
- # is used below to recognise the mesh among the cube dimensions.
- any_mesh_coord = cube.coords(mesh_coords=True)[0]
- (cube_mesh_dim,) = any_mesh_coord.cube_dims(cube)
-
- # Record actual file dimension names for each mesh saved.
- self._mesh_dims[mesh] = mesh_location_dimnames
-
- # Get the cube dimensions, in order.
- cube_dimensions = []
- if cube is not None:
- for dim in range(cube.ndim):
- if dim == cube_mesh_dim:
- # Handle a mesh dimension: we already named this.
- dim_coords = []
- dim_name = self._mesh_dims[mesh][cube.location]
- else:
- # Get a name from the dim-coord (if any).
- dim_coords = cube.coords(dimensions=dim, dim_coords=True)
- if dim_coords:
- # Derive a dim name from a coord.
- coord = dim_coords[0] # always have at least one
-
- # Add only dimensions that have not already been added.
- dim_name = self._dim_names_and_coords.name(coord)
- if dim_name is None:
- # Not already present : create a unique dimension name
- # from the coord.
- dim_name = self._get_element_variable_name(cube, coord)
- # Disambiguate if it has the same name as an
- # existing dimension.
- # OR if it matches an existing file variable name.
- # NOTE: check against variable names is needed
- # because of a netcdf bug ... see note in the
- # mesh dimensions block above.
- while (
- dim_name in self._existing_dim
- or dim_name in self._dataset.variables
- ):
- dim_name = self._increment_name(dim_name)
-
- else:
- # No CF-netCDF coordinates describe this data dimension.
- # Make up a new, distinct dimension name
- dim_name = f"dim{dim}"
- # Increment name if conflicted with one already existing
- # (or planned)
- # NOTE: check against variable names is needed because
- # of a netcdf bug ... see note in the mesh dimensions
- # block above.
- while (
- dim_name in self._existing_dim
- and (self._existing_dim[dim_name] != cube.shape[dim])
- ) or dim_name in self._dataset.variables:
- dim_name = self._increment_name(dim_name)
-
- # Record the dimension.
- record_dimension(cube_dimensions, dim_name, cube.shape[dim], dim_coords)
-
- return mesh_dimensions, cube_dimensions
-
- @staticmethod
- def cf_valid_var_name(var_name):
- """Return a valid CF var_name given a potentially invalid name.
-
- Parameters
- ----------
- var_name : str
- The var_name to normalise.
-
- Returns
- -------
- str
- The var_name suitable for passing through for variable creation.
-
- """
- # Replace invalid characters with an underscore ("_").
- var_name = re.sub(r"[^a-zA-Z0-9]", "_", var_name)
- # Ensure the variable name starts with a letter.
- if re.match(r"^[^a-zA-Z]", var_name):
- var_name = "var_{}".format(var_name)
- return var_name
-
- @staticmethod
- def _cf_coord_standardised_units(coord):
- """Determine a suitable units from a given coordinate.
-
- Parameters
- ----------
- coord : :class:`iris.coords.Coord`
- A coordinate of a cube.
-
- Returns
- -------
- units
- The (standard_name, long_name, unit) of the given
- :class:`iris.coords.Coord` instance.
- """
- units = str(coord.units)
- # Set the 'units' of 'latitude' and 'longitude' coordinates specified
- # in 'degrees' to 'degrees_north' and 'degrees_east' respectively,
- # as defined in the CF conventions for netCDF files: sections 4.1 and
- # 4.2.
- if (
- isinstance(coord.coord_system, iris.coord_systems.GeogCS)
- or coord.coord_system is None
- ) and coord.units == "degrees":
- if coord.standard_name == "latitude":
- units = "degrees_north"
- elif coord.standard_name == "longitude":
- units = "degrees_east"
-
- return units
-
- def _ensure_valid_dtype(self, values, src_name, src_object):
- # NetCDF3 and NetCDF4 classic do not support int64 or unsigned ints,
- # so we check if we can store them as int32 instead.
- if (
- np.issubdtype(values.dtype, np.int64)
- or np.issubdtype(values.dtype, np.unsignedinteger)
- ) and self._dataset.file_format in (
- "NETCDF3_CLASSIC",
- "NETCDF3_64BIT",
- "NETCDF4_CLASSIC",
- ):
- val_min, val_max = (values.min(), values.max())
- if is_lazy_data(values):
- val_min, val_max = _co_realise_lazy_arrays([val_min, val_max])
- # NumPy will inherit values.dtype even if the scalar numbers work
- # with a smaller type.
- min_dtype = np.promote_types(
- *[np.min_scalar_type(m) for m in (val_min, val_max)]
- )
- # Cast to an integer type supported by netCDF3.
- if not np.can_cast(min_dtype, np.int32):
- msg = (
- "The data type of {} {!r} is not supported by {} and"
- " its values cannot be safely cast to a supported"
- " integer type."
- )
- msg = msg.format(src_name, src_object, self._dataset.file_format)
- raise ValueError(msg)
- values = values.astype(np.int32)
- return values
-
- def _create_cf_bounds(self, coord, cf_var, cf_name, /, *, compression_kwargs=None):
- """Create the associated CF-netCDF bounds variable.
-
- Parameters
- ----------
- coord : :class:`iris.coords.Coord`
- A coordinate of a cube.
- cf_var :
- CF-netCDF variable.
+ The Cube or Mesh being saved. Named in the error message.
cf_name : str
- Name of the CF-NetCDF variable.
- compression_kwargs : dict, optional
- NetCDF data compression keyword arguments.
-
- Returns
- -------
- None
-
- """
- if hasattr(coord, "has_bounds") and coord.has_bounds():
- if compression_kwargs is None:
- compression_kwargs = {}
-
- # Get the values in a form which is valid for the file format.
- bounds = self._ensure_valid_dtype(
- coord.core_bounds(), "the bounds of coordinate", coord
- )
- n_bounds = bounds.shape[-1]
-
- if n_bounds == 2:
- bounds_dimension_name = "bnds"
- else:
- bounds_dimension_name = "bnds_%s" % n_bounds
-
- if coord.climatological:
- property_name = "climatology"
- varname_extra = "climatology"
- else:
- property_name = "bounds"
- varname_extra = "bnds"
-
- if bounds_dimension_name not in self._dataset.dimensions:
- # Create the bounds dimension with the appropriate extent.
- while bounds_dimension_name in self._dataset.variables:
- # Also avoid collision with variable names.
- # See '_get_dim_names' for reason.
- bounds_dimension_name = self._increment_name(bounds_dimension_name)
- self._dataset.createDimension(bounds_dimension_name, n_bounds)
-
- boundsvar_name = "{}_{}".format(cf_name, varname_extra)
- _setncattr(cf_var, property_name, boundsvar_name)
- cf_var_bounds = self._dataset.createVariable(
- boundsvar_name,
- bounds.dtype.newbyteorder("="),
- cf_var.dimensions + (bounds_dimension_name,),
- **compression_kwargs,
- )
- self._lazy_stream_data(data=bounds, cf_var=cf_var_bounds)
-
- def _get_element_variable_name(
- self, cube_or_mesh: "Cube|MeshXY", element: "_DimensionalMetadata|Cube"
- ) -> str:
- # keep it as _DimensionalMetadata
- # Don't want it as CFVariableMixin - implies a cube can be passed in
- # Could be Coord AncillaryVariable Cellmeasure
- """Return a CF-netCDF variable name for a given coordinate-like element, or cube.
-
- Parameters
- ----------
- cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
- The Cube or Mesh being saved to the netCDF file.
- element : :class:`iris.coords._DimensionalMetadata` | :class:``iris.cube.Cube``
- An instance of a coordinate (or similar), for which a CF-netCDF
- variable name is required.
-
- Returns
- -------
- str
- A CF-netCDF variable name as a string.
-
- """
- # Support cube or mesh save.
- from iris.cube import Cube
-
- if isinstance(cube_or_mesh, Cube):
- cube = cube_or_mesh
- mesh = cube.mesh
- else:
- cube = None
- mesh = cube_or_mesh
-
- if element.var_name is not None:
- cf_name = element.var_name
- elif isinstance(element, Cube):
- # Make name for a Cube without a var_name.
- cf_name = "_".join(element.name().lower().split())
- else:
- # Make name for a Coord-like element without a var_name
- name = element.standard_name or element.long_name
- if not name or set(name).intersection(string.whitespace):
- # We need to invent a name, based on its associated dimensions.
- if cube is not None and cube.coords(element):
- # It is a regular cube coordinate.
- # Auto-generate a name based on the dims.
- name = ""
- for dim in cube.coord_dims(element):
- name += f"dim{dim}"
- # Handle scalar coordinate (dims == ()).
- if not name:
- name = "unknown_scalar"
- else:
- # Not a cube coord, so must be a connectivity or
- # element-coordinate of the mesh.
- # Name it for it's first dim, i.e. mesh-dim of its location.
-
- from iris.mesh import Connectivity
-
- if isinstance(element, Connectivity):
- location = element.cf_role.split("_")[0]
- location_dim_attr = f"{location}_dimension"
- name = getattr(mesh, location_dim_attr)
- else:
- # 'coord' is a nameless coordinate of a mesh,
- # so we set it as unknown for saving
- name = "unknown"
-
- # Convert to lower case and replace whitespace by underscores.
- cf_name = "_".join(name.lower().split())
-
- cf_name = self.cf_valid_var_name(cf_name)
- return cf_name
-
- def _get_mesh_variable_name(self, mesh):
- """Return a CF-netCDF variable name for the mesh.
-
- Parameters
- ----------
- mesh : :class:`iris.mesh.MeshXY`
- An instance of a Mesh for which a CF-netCDF variable name is
- required.
-
- Returns
- -------
- str
- A CF-netCDF variable name as a string.
-
- """
- cf_name = mesh.var_name or mesh.long_name
- # Prefer a var-name, but accept a long_name as an alternative.
- # N.B. we believe it can't (shouldn't) have a standard name.
- if not cf_name:
- # Auto-generate a name based on mesh properties.
- cf_name = f"Mesh_{mesh.topology_dimension}d"
-
- # Ensure valid form for var-name.
- cf_name = self.cf_valid_var_name(cf_name)
- return cf_name
-
- def _create_mesh(self, mesh):
- """Create a mesh variable in the netCDF dataset.
-
- Parameters
- ----------
- mesh : :class:`iris.mesh.MeshXY`
- The Mesh to be saved to CF-netCDF file.
-
- Returns
- -------
- str
- The string name of the associated CF-netCDF variable saved.
-
- """
- # First choose a var-name for the mesh variable itself.
- cf_mesh_name = self._get_mesh_variable_name(mesh)
- # Disambiguate any possible clashes.
- while cf_mesh_name in self._dataset.variables:
- cf_mesh_name = self._increment_name(cf_mesh_name)
-
- # Create the main variable
- cf_mesh_var = self._dataset.createVariable(
- cf_mesh_name,
- np.dtype(np.int32),
- [],
- )
-
- # Add the basic essential attributes
- _setncattr(cf_mesh_var, "cf_role", "mesh_topology")
- _setncattr(
- cf_mesh_var,
- "topology_dimension",
- np.int32(mesh.topology_dimension),
- )
- # Add the usual names + units attributes
- self._set_cf_var_attributes(cf_mesh_var, mesh)
-
- return cf_mesh_name
-
- def _set_cf_var_attributes(self, cf_var, element):
- from iris.cube import Cube
-
- # Deal with CF-netCDF units, and add the name+units properties.
- if isinstance(element, iris.coords.Coord):
- # Fix "degree" units if needed.
- units_str = self._cf_coord_standardised_units(element)
- else:
- units_str = str(element.units)
-
- # NB this bit is a nasty hack to preserve existing behaviour through a refactor:
- # The attributes for Coords are created in the order units, standard_name,
- # whereas for data-variables (aka Cubes) it is the other way around.
- # Needed now that this routine is also called from _create_cf_data_variable.
- # TODO: when we can break things, rationalise these to be the same.
- def add_units_attr():
- if cf_units.as_unit(units_str).is_udunits():
- _setncattr(cf_var, "units", units_str)
-
- def add_names_attrs():
- standard_name = element.standard_name
- if standard_name is not None:
- _setncattr(cf_var, "standard_name", standard_name)
-
- long_name = element.long_name
- if long_name is not None:
- _setncattr(cf_var, "long_name", long_name)
-
- if isinstance(element, Cube):
- add_names_attrs()
- add_units_attr()
- else:
- add_units_attr()
- add_names_attrs()
-
- # Add the CF-netCDF calendar attribute.
- if element.units.calendar:
- _setncattr(cf_var, "calendar", str(element.units.calendar))
-
- # Take a copy so we can remove things
- element_attrs = element.attributes.copy()
-
- # Note: when writing UGRID, "element" can be a Mesh which has no "dtype",
- # and for dataless cubes it will have a 'None' dtype.
- if getattr(element, "dtype", None) is not None:
- # Most attributes are dealt with later. But _Encoding needs to be defined
- # *before* we can write to a character variable.
- if element.dtype.kind in "SU" and "_Encoding" in element_attrs:
- encoding = element_attrs.pop("_Encoding")
- _setncattr(cf_var, "_Encoding", encoding)
-
- if not isinstance(element, Cube):
- # Add any other custom coordinate attributes.
- # N.B. not Cube, which has specific handling in _create_cf_data_variable
- for name in sorted(element_attrs):
- value = element_attrs[name]
-
- if name == "STASH":
- # Adopting provisional Metadata Conventions for representing MO
- # Scientific Data encoded in NetCDF Format.
- name = "um_stash_source"
- value = str(value)
-
- # Don't clobber existing attributes.
- if not hasattr(cf_var, name):
- _setncattr(cf_var, name, value)
-
- def _create_generic_cf_array_var(
- self,
- cube_or_mesh,
- cube_dim_names,
- element,
- /,
- *,
- element_dims=None,
- fill_value=None,
- compression_kwargs=None,
- packing_controls: dict | None = None,
- is_dataless=False,
- ):
- """Create theCF-netCDF variable given dimensional_metadata.
-
- Create the associated CF-netCDF variable in the netCDF dataset for the
- given dimensional_metadata.
-
- .. note::
- If the metadata element is a coord, it may also contain bounds.
- In which case, an additional var is created and linked to it.
-
- Parameters
- ----------
- cube_or_mesh : :class:`iris.cube.Cube` or :class:`iris.mesh.MeshXY`
- The Cube or Mesh being saved to the netCDF file.
- cube_dim_names : list of str
- The name of each dimension of the cube.
+ The name the variable is created with.
element : :class:`iris.coords._DimensionalMetadata`
- An Iris :class:`iris.coords._DimensionalMetadata`, belonging to the
- cube. Provides data, units and standard/long/var names.
- Not used if 'element_dims' is not None.
- element_dims : list of str, optional
- If set, contains the variable dimension (names),
- otherwise these are taken from `element.cube_dims[cube]`.
- For Mesh components (element coordinates and connectivities), this
- *must* be passed in, as "element.cube_dims" does not function.
- fill_value : number, optional
- If set, create the variable with this fill-value, and fill any
- masked data points with this value.
- If not set, standard netcdf4-python behaviour : the variable has no
- '_FillValue' property, and uses the "standard" fill-value for its
- type.
- compression_kwargs : dict, optional
- NetCDF data compression keyword arguments.
+ The element being saved. Supplies the ``_Encoding`` attribute.
+ element_dims : list of str
+ The variable's dimension names, appended to in place.
+ data : :class:`numpy.ndarray`
+ The string values, used for their dtype.
Returns
-------
- str
- The name of the CF-netCDF variable created.
- """
- # Support cube or mesh save.
- from iris.cube import Cube
-
- if isinstance(cube_or_mesh, Cube):
- cube = cube_or_mesh
- else:
- cube = None
-
- if compression_kwargs is None:
- compression_kwargs = {}
-
- # Work out the var-name to use.
- # N.B. the only part of this routine that may use a mesh _or_ a cube.
- cf_name = self._get_element_variable_name(cube_or_mesh, element)
- while cf_name in self._dataset.variables:
- cf_name = self._increment_name(cf_name)
-
- if cube and element_dims is None:
- # Get the list of file-dimensions (names), to create the variable.
- element_dims = [
- cube_dim_names[dim] for dim in element.cube_dims(cube)
- ] # NB using 'cube_dims' as this works for any type of element
-
- # Get the data values, in a way which works for any element type, as
- # all are subclasses of _DimensionalMetadata.
- # (e.g. =points if a coord, =data if an ancillary, etc)
- if isinstance(element, Cube):
- data = element.core_data()
- else:
- data = element._core_values()
-
- # This compression contract is *not* applicable to a mesh.
- if cube is not None and data is not None and cube.shape != data.shape:
- compression_kwargs = {}
-
- if not is_dataless and data.dtype.kind == "U":
- # Deal with unicode-string-type variables.
- # Typically CF label variables, but also possibly ancil-vars ?
-
- # NOTE: all we are doing here is to calculate the byte dimension length,
- # based on the dtype and any encoding attribute.
- # The actual char --> byte data *translation* is done by the variable,
- # being a _bytecoding_datasets.EncodedVariable.
- string_dimension_depth = data.dtype.itemsize
-
- # String content (U) instead of bytes (S).
- # For numpy strings, itemsize is **always** a multiple of 4
- if string_dimension_depth % 4 != 0:
- msg = (
- "Unexpected numpy string 'dtype.itemsize' for element "
- f"{cube_or_mesh.name()!r}: "
- f"'dtype.itemsize = {string_dimension_depth}, expected "
- "a multiple of four (always)."
- )
- raise ValueError(msg)
- nchars = string_dimension_depth // 4
-
- encoding_attr = element.attributes.get("_Encoding", "ascii")
- # Look this up + return a supported encoding name
- # NB implements defaults and raises a warning if given not recognised.
- encoding = bytecoding_datasets._identify_encoding(
- encoding=encoding_attr, var_name=cf_name, writing=True
- )
- width_fns = bytecoding_datasets._ENCODING_WIDTH_TRANSLATIONS[encoding]
- string_dimension_depth = width_fns.nchars_2_nbytes(nchars)
-
- string_dimension_name = "string%d" % string_dimension_depth
-
- # Determine whether to create the string length dimension.
- if string_dimension_name not in self._dataset.dimensions:
- while string_dimension_name in self._dataset.variables:
- # Also avoid collision with variable names.
- # See '_get_dim_names' for reason.
- string_dimension_name = self._increment_name(string_dimension_name)
- self._dataset.createDimension(
- string_dimension_name, string_dimension_depth
- )
-
- # Add the string length dimension to the variable dimensions.
- element_dims.append(string_dimension_name)
-
- # Create the label coordinate variable.
- cf_var = self._dataset.createVariable(cf_name, "|S1", element_dims)
- else:
- # A non-string variable.
- # ensure a valid datatype for the file format.
- if is_dataless:
- dtype = self._DATALESS_DTYPE
- fill_value = self._DATALESS_FILLVALUE
- else:
- # Normal non-string data.
- # NOTE: this includes byte-arrays (S1 only) : however these must
- # use an actual cube dimension for the 'string dimension', which
- # seriously limits the utility of DECODE_TO_STRINGS_ON_READ.
- # TODO: also support netCDF variable-length strings ("string" type).
- # Currently hit a **write error here**, being numpy object dtype ("O").
- if data.dtype.kind not in "iufSU" or (
- data.dtype.kind == "S" and data.dtype.itemsize != 1
- ):
- # This is a type of data we don't "understand".
- # NB this includes "Sxx" types other than "S1" : It seems that
- # netCDF4 saves Sxx as variable-length strings.
- # But we don't support that type in Iris.
- msg = (
- f"Variable {cf_name!r} has unexpected dtype, {data.dtype!r}."
- f"Data content arrays must be numeric, or contain "
- "single-bytes (dtype 'S1'), or unicode strings (dtype 'U<n>')."
- )
- raise ValueError(msg)
-
- element_type = type(element).__name__
- data = self._ensure_valid_dtype(data, element_type, element)
- if not packing_controls:
- dtype = data.dtype.newbyteorder("=")
- else:
- dtype = packing_controls["dtype"]
-
- # Check if this is a dim-coord.
- is_dimcoord = cube is not None and element in cube.dim_coords
-
- if is_dimcoord:
- # By definition of a CF-netCDF coordinate variable this
- # coordinate must be 1-D and the name of the CF-netCDF variable
- # must be the same as its dimension name.
- cf_name = element_dims[0]
-
- # Create the CF-netCDF variable.
- cf_var = self._dataset.createVariable(
- cf_name,
- dtype,
- element_dims,
- fill_value=fill_value,
- **compression_kwargs,
- )
-
- # Add the axis attribute for spatio-temporal CF-netCDF coordinates.
- if is_dimcoord:
- axis = iris.util.guess_coord_axis(element)
- if axis is not None and axis.lower() in SPATIO_TEMPORAL_AXES:
- _setncattr(cf_var, "axis", axis.upper())
-
- # Create the associated CF-netCDF bounds variable, if any.
- self._create_cf_bounds(
- element, cf_var, cf_name, compression_kwargs=compression_kwargs
- )
-
- # Add names + units
- # NOTE: *must* now do first, as we may need '_Encoding' set to write it !
- self._set_cf_var_attributes(cf_var, element)
-
- # Add the data to the CF-netCDF variable.
- if not is_dataless:
- if packing_controls:
- # We must set packing attributes (if any), before assigning values.
- for key, value in packing_controls["attributes"]:
- _setncattr(cf_var, key, value)
- self._lazy_stream_data(data=data, cf_var=cf_var)
-
- return cf_name
-
- def _create_cf_cell_methods(self, cube, dimension_names):
- """Create CF-netCDF string representation of a cube cell methods.
-
- Parameters
- ----------
- cube : :class:`iris.cube.Cube` or :class:`iris.cube.CubeList`
- A :class:`iris.cube.Cube`, :class:`iris.cube.CubeList` or list of
- cubes to be saved to a netCDF file.
- dimension_names : list
- Names associated with the dimensions of the cube.
-
- Returns
- -------
- str
- CF-netCDF string representation of a cube cell methods.
+ :class:`~iris.fileformats.cf.dataset.CFDatasetVariable`
"""
- cell_methods = []
-
- # Identify the collection of coordinates that represent CF-netCDF
- # coordinate variables.
- cf_coordinates = cube.dim_coords
-
- for cm in cube.cell_methods:
- names = ""
-
- for name in cm.coord_names:
- coord = cube.coords(name)
-
- if coord:
- coord = coord[0]
- if coord in cf_coordinates:
- name = dimension_names[cube.coord_dims(coord)[0]]
-
- names += "%s: " % name
-
- interval = " ".join(
- ["interval: %s" % interval for interval in cm.intervals or []]
- )
- comment = " ".join(
- ["comment: %s" % comment for comment in cm.comments or []]
- )
- extra = " ".join([interval, comment]).strip()
-
- if extra:
- extra = " (%s)" % extra
-
- cell_methods.append(names + cm.method + extra)
-
- return " ".join(cell_methods)
-
- def _add_grid_mapping_to_dataset(self, cs, extended_grid_mapping=False):
- """Create a CF-netCDF grid mapping variable and add to the dataset.
-
- Parameters
- ----------
- cs : :class:`iris.coord_system.CoordSystem`
- The :class:`iris.coord_system.CoordSystem` used to generate the
- new netCDF CF grid mapping variable.
-
- Returns
- -------
- None
- """
- cf_var_grid = self._dataset.createVariable(cs.grid_mapping_name, np.int32)
- _setncattr(cf_var_grid, "grid_mapping_name", cs.grid_mapping_name)
-
- def add_ellipsoid(ellipsoid):
- cf_var_grid.longitude_of_prime_meridian = (
- ellipsoid.longitude_of_prime_meridian
- )
- semi_major = ellipsoid.semi_major_axis
- semi_minor = ellipsoid.semi_minor_axis
- if semi_minor == semi_major:
- cf_var_grid.earth_radius = semi_major
- else:
- cf_var_grid.semi_major_axis = semi_major
- cf_var_grid.semi_minor_axis = semi_minor
- if ellipsoid.datum is not None:
- cf_var_grid.horizontal_datum_name = ellipsoid.datum
-
- # latlon
- if isinstance(cs, iris.coord_systems.GeogCS):
- add_ellipsoid(cs)
-
- # rotated latlon
- elif isinstance(cs, iris.coord_systems.RotatedGeogCS):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.grid_north_pole_latitude = cs.grid_north_pole_latitude
- cf_var_grid.grid_north_pole_longitude = cs.grid_north_pole_longitude
- cf_var_grid.north_pole_grid_longitude = cs.north_pole_grid_longitude
-
- # tmerc
- elif isinstance(cs, iris.coord_systems.TransverseMercator):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.longitude_of_central_meridian = cs.longitude_of_central_meridian
- cf_var_grid.latitude_of_projection_origin = cs.latitude_of_projection_origin
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
- cf_var_grid.scale_factor_at_central_meridian = (
- cs.scale_factor_at_central_meridian
- )
-
- # merc
- elif isinstance(cs, iris.coord_systems.Mercator):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
- )
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
- # Only one of these should be set
- if cs.standard_parallel is not None:
- cf_var_grid.standard_parallel = cs.standard_parallel
- elif cs.scale_factor_at_projection_origin is not None:
- cf_var_grid.scale_factor_at_projection_origin = (
- cs.scale_factor_at_projection_origin
- )
-
- # lcc
- elif isinstance(cs, iris.coord_systems.LambertConformal):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.standard_parallel = cs.secant_latitudes
- cf_var_grid.latitude_of_projection_origin = cs.central_lat
- cf_var_grid.longitude_of_central_meridian = cs.central_lon
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
-
- # polar stereo (have to do this before Stereographic because it subclasses it)
- elif isinstance(cs, iris.coord_systems.PolarStereographic):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.latitude_of_projection_origin = cs.central_lat
- cf_var_grid.straight_vertical_longitude_from_pole = cs.central_lon
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
- # Only one of these should be set
- if cs.true_scale_lat is not None:
- cf_var_grid.true_scale_lat = cs.true_scale_lat
- elif cs.scale_factor_at_projection_origin is not None:
- cf_var_grid.scale_factor_at_projection_origin = (
- cs.scale_factor_at_projection_origin
- )
- else:
- cf_var_grid.scale_factor_at_projection_origin = 1.0
-
- # stereo
- elif isinstance(cs, iris.coord_systems.Stereographic):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.longitude_of_projection_origin = cs.central_lon
- cf_var_grid.latitude_of_projection_origin = cs.central_lat
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
- # Only one of these should be set
- if cs.true_scale_lat is not None:
- msg = (
- "It is not valid CF to save a true_scale_lat for "
- "a Stereographic grid mapping."
- )
- raise ValueError(msg)
- elif cs.scale_factor_at_projection_origin is not None:
- cf_var_grid.scale_factor_at_projection_origin = (
- cs.scale_factor_at_projection_origin
- )
- else:
- cf_var_grid.scale_factor_at_projection_origin = 1.0
-
- # osgb (a specific tmerc)
- elif isinstance(cs, iris.coord_systems.OSGB):
- warnings.warn(
- "OSGB coordinate system not yet handled",
- category=iris.warnings.IrisSaveWarning,
- )
-
- # lambert azimuthal equal area
- elif isinstance(cs, iris.coord_systems.LambertAzimuthalEqualArea):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
- )
- cf_var_grid.latitude_of_projection_origin = cs.latitude_of_projection_origin
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
-
- # albers conical equal area
- elif isinstance(cs, iris.coord_systems.AlbersEqualArea):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.longitude_of_central_meridian = cs.longitude_of_central_meridian
- cf_var_grid.latitude_of_projection_origin = cs.latitude_of_projection_origin
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
- cf_var_grid.standard_parallel = cs.standard_parallels
-
- # vertical perspective
- elif isinstance(cs, iris.coord_systems.VerticalPerspective):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
- )
- cf_var_grid.latitude_of_projection_origin = cs.latitude_of_projection_origin
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
- cf_var_grid.perspective_point_height = cs.perspective_point_height
-
- # geostationary
- elif isinstance(cs, iris.coord_systems.Geostationary):
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
- )
- cf_var_grid.latitude_of_projection_origin = cs.latitude_of_projection_origin
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
- cf_var_grid.perspective_point_height = cs.perspective_point_height
- cf_var_grid.sweep_angle_axis = cs.sweep_angle_axis
-
- # oblique mercator (and rotated variant)
- # Use duck-typing over isinstance() - subclasses (i.e.
- # RotatedMercator) upset mock tests.
- elif getattr(cs, "grid_mapping_name", None) == "oblique_mercator":
- # RotatedMercator subclasses ObliqueMercator, and RM
- # instances are implicitly saved as OM due to inherited
- # properties. This is correct because CF 1.11 is removing
- # all mention of RM.
- if cs.ellipsoid:
- add_ellipsoid(cs.ellipsoid)
- cf_var_grid.azimuth_of_central_line = cs.azimuth_of_central_line
- cf_var_grid.latitude_of_projection_origin = cs.latitude_of_projection_origin
- cf_var_grid.longitude_of_projection_origin = (
- cs.longitude_of_projection_origin
- )
- cf_var_grid.false_easting = cs.false_easting
- cf_var_grid.false_northing = cs.false_northing
- cf_var_grid.scale_factor_at_projection_origin = (
- cs.scale_factor_at_projection_origin
+ # NOTE: all we are doing here is to calculate the byte dimension length,
+ # based on the dtype and any encoding attribute.
+ # The actual char --> byte data *translation* is done by the variable,
+ # being a _bytecoding_datasets.EncodedVariable.
+ string_dimension_depth = data.dtype.itemsize
+
+ # String content (U) instead of bytes (S).
+ # For numpy strings, itemsize is **always** a multiple of 4
+ if string_dimension_depth % 4 != 0:
+ msg = (
+ "Unexpected numpy string 'dtype.itemsize' for element "
+ f"{cube_or_mesh.name()!r}: "
+ f"'dtype.itemsize = {string_dimension_depth}, expected "
+ "a multiple of four (always)."
)
+ raise ValueError(msg)
+ nchars = string_dimension_depth // 4
- # other
- else:
- warnings.warn(
- "Unable to represent the horizontal "
- "coordinate system. The coordinate system "
- "type %r is not yet implemented." % type(cs),
- category=iris.warnings.IrisSaveWarning,
+ encoding_attr = element.attributes.get("_Encoding", "ascii")
+ # Look this up + return a supported encoding name
+ # NB implements defaults and raises a warning if given not recognised.
+ encoding = bytecoding_datasets._identify_encoding(
+ encoding=encoding_attr, var_name=cf_name, writing=True
+ )
+ width_fns = bytecoding_datasets._ENCODING_WIDTH_TRANSLATIONS[encoding]
+ string_dimension_depth = width_fns.nchars_2_nbytes(nchars)
+
+ string_dimension_name = "string%d" % string_dimension_depth
+
+ # Determine whether to create the string length dimension.
+ if string_dimension_name not in self._dataset.dimensions:
+ while string_dimension_name in self._dataset.variables:
+ # Also avoid collision with variable names.
+ # See '_get_dim_names' for reason.
+ string_dimension_name = self._increment_name(string_dimension_name)
+ self._dataset.create_dimension(
+ string_dimension_name, string_dimension_depth
)
- # add WKT string
- if extended_grid_mapping:
- cf_var_grid.crs_wkt = cs.as_cartopy_crs().to_wkt()
-
- def _create_cf_grid_mapping(self, cube, cf_var_cube):
- """Create CF-netCDF grid mapping and associated CF-netCDF variable.
-
- Create CF-netCDF grid mapping variable and associated CF-netCDF
- data variable grid mapping attribute.
+ # Add the string length dimension to the variable dimensions.
+ element_dims.append(string_dimension_name)
- Parameters
- ----------
- cube : :class:`iris.cube.Cube` or :class:`iris.cube.CubeList`
- A :class:`iris.cube.Cube`, :class:`iris.cube.CubeList` or list of
- cubes to be saved to a netCDF file.
- cf_var_cube : :class:`netcdf.netcdf_variable`
- A cf variable cube representation.
-
- Returns
- -------
- None
+ # Create the label coordinate variable.
+ return self._dataset.create_variable(cf_name, "|S1", element_dims)
- """
- coord_systems = []
- if cube.extended_grid_mapping:
- # get unique list of all coord_systems on cube coords:
- for coord in cube.coords():
- if coord.coord_system and coord.coord_system not in coord_systems:
- coord_systems.append(coord.coord_system)
- else:
- # will only return a single coord system (if one exists):
- if cs := cube.coord_system("CoordSystem"):
- coord_systems.append(cs)
-
- grid_mappings = {}
- matched_all_coords = True
-
- for cs in coord_systems:
- # Grid var not yet created?
- if cs not in self._coord_systems:
- # handle potential duplicate netCDF variable names:
- while cs.grid_mapping_name in self._dataset.variables:
- aname = self._increment_name(cs.grid_mapping_name)
- cs.grid_mapping_name = aname
-
- # create grid mapping variable on dataset for this coordinate system:
- self._add_grid_mapping_to_dataset(
- cs, extended_grid_mapping=cube.extended_grid_mapping
- )
- self._coord_systems.append(cs)
-
- # create the `grid_mapping` attribute for the data variable:
- if cube.extended_grid_mapping:
- # Order the coordinates as per the order in the CRS/WKT string.
- # (We should only ever have a coordinate system for horizontal
- # spatial coords, so check for east/north directions)
- ordered_coords = []
- for ax_info in cs.as_cartopy_crs().axis_info:
- try:
- match ax_info.direction.lower():
- case "east":
- ordered_coords.append(
- cube.coord(axis="X", coord_system=cs)
- )
- case "north":
- ordered_coords.append(
- cube.coord(axis="Y", coord_system=cs)
- )
- case _:
- msg = (
- f"Can't handle axis direction {ax_info.direction!r}"
- )
- raise NotImplementedError(msg)
- except iris.exceptions.CoordinateNotFoundError as e:
- msg = (
- f"Failed to assign coordinate for {ax_info.name} axis of "
- f"coordinate system {cs.grid_mapping_name}: {str(e)}"
- )
- warnings.warn(msg, category=iris.warnings.IrisSaveWarning)
-
- # fall back to simple grid mapping (single crs entry for DimCoord only)
- matched_all_coords = False
-
- # Get variable names, and store them if necessary:
- cfvar_names = []
- for coord in ordered_coords:
- cfvar = self._name_coord_map.name(coord)
- if not cfvar:
- # not found - create and store it:
- cfvar = self._get_element_variable_name(cube, coord)
- self._name_coord_map.append(
- cfvar, self._get_element_variable_name(cube, coord)
- )
- cfvar_names.append(cfvar)
-
- coord_string = " ".join(cfvar_names)
- grid_mappings[cs.grid_mapping_name] = coord_string
-
- # Refer to grid var
- if len(coord_systems):
- grid_mapping = None
- if cube.extended_grid_mapping:
- if matched_all_coords:
- grid_mapping = " ".join(
- f"{cs_name}: {cs_coords}"
- for cs_name, cs_coords in grid_mappings.items()
- )
- else:
- # We didn't match all coords required for grid mapping, default to
- # single coordinate system for DimCoords, if set.
- for ax in ["x", "y"]:
- try:
- if dim_cs := cube.coord(
- axis=ax, dim_coords=True
- ).coord_system:
- grid_mapping = dim_cs.grid_mapping_name
- break
- except iris.exceptions.CoordinateNotFoundError:
- pass
- else:
- # Not using extended_grid_mapping: only ever be on cs in this case:
- grid_mapping = coord_systems[0].grid_mapping_name
-
- if grid_mapping:
- _setncattr(cf_var_cube, "grid_mapping", grid_mapping)
-
- _DATALESS_ATTRNAME = "iris_dataless_cube"
- _DATALESS_DTYPE = np.dtype("u1")
- _DATALESS_FILLVALUE = 127
-
- def _create_cf_data_variable(
+ def _backend_write(
self,
- cube,
- dimension_names,
- local_keys=None,
- packing=None,
- fill_value=None,
- **kwargs,
- ):
- """Create CF-netCDF data variable for the cube and any associated grid mapping.
-
- # TODO: when iris.FUTURE.save_split_attrs is removed, the 'local_keys' arg can
- # be removed.
-
- Parameters
- ----------
- cube : :class:`iris.cube.Cube`
- The associated cube being saved to CF-netCDF file.
- dimension_names : list
- String names for each dimension of the cube.
- local_keys : iterable of str, optional
- See :func:`iris.fileformats.netcdf.Saver.write`.
- packing : type or str or dict or list, optional
- See :func:`iris.fileformats.netcdf.Saver.write`.
- fill_value : optional
- See :func:`iris.fileformats.netcdf.Saver.write`.
-
- Returns
- -------
- The newly created CF-netCDF data variable.
-
- Notes
- -----
- All other keywords are passed through to the dataset's `createVariable`
- method.
-
- """
- # TODO: when iris.FUTURE.save_split_attrs is removed, the 'local_keys' arg can
- # be removed.
- # Get the values in a form which is valid for the file format.
- is_dataless = cube.is_dataless()
-
- packing_controls = None
- if packing and not is_dataless:
- data = self._ensure_valid_dtype(cube.core_data(), "cube", cube)
- if isinstance(packing, dict):
- if "dtype" not in packing:
- msg = "The dtype attribute is required for packing."
- raise ValueError(msg)
- dtype = np.dtype(packing["dtype"])
- scale_factor = packing.get("scale_factor", None)
- add_offset = packing.get("add_offset", None)
- valid_keys = {"dtype", "scale_factor", "add_offset"}
- invalid_keys = set(packing.keys()) - valid_keys
- if invalid_keys:
- msg = (
- "Invalid packing key(s) found: '{}'. The valid "
- "keys are '{}'.".format(
- "', '".join(invalid_keys), "', '".join(valid_keys)
- )
- )
- raise ValueError(msg)
- else:
- # We compute the scale_factor and add_offset based on the
- # min/max of the data.
- masked = iris.util.is_masked(data)
- dtype = np.dtype(packing)
- cmin, cmax = (data.min(), data.max())
- if is_lazy_data(data):
- cmin, cmax = _co_realise_lazy_arrays([cmin, cmax])
- n = dtype.itemsize * 8
- if masked:
- scale_factor = (cmax - cmin) / (2**n - 2)
- else:
- scale_factor = (cmax - cmin) / (2**n - 1)
- if dtype.kind == "u":
- add_offset = cmin
- elif dtype.kind == "i":
- if masked:
- add_offset = (cmax + cmin) / 2
- else:
- add_offset = cmin + 2 ** (n - 1) * scale_factor
-
- packing_controls = {
- "dtype": dtype,
- "attributes": [
- ("scale_factor", scale_factor),
- ("add_offset", add_offset),
- ],
- }
-
- # Create the cube CF-netCDF data variable with data payload.
- cf_name = self._create_generic_cf_array_var(
- cube,
- dimension_names,
- cube,
- element_dims=dimension_names,
- fill_value=fill_value,
- compression_kwargs=kwargs,
- packing_controls=packing_controls,
- is_dataless=is_dataless,
- )
- cf_var = self._dataset.variables[cf_name]
-
- # Set general attrs: NB this part is cube-specific (not the same for components)
- # - so 'set_cf_var_attributes' *doesn't* set these, if element is a Cube
- if iris.FUTURE.save_split_attrs:
- attr_names = cube.attributes.locals.keys()
- else:
- # Add data variable-only attribute names to local_keys.
- if local_keys is None:
- local_keys = set()
- else:
- local_keys = set(local_keys)
- local_keys.update(_CF_DATA_ATTRS, _UKMO_DATA_ATTRS, ["GRIB_PARAM"])
-
- # Add any cube attributes whose keys are in local_keys as
- # CF-netCDF data variable attributes.
- attr_names = set(cube.attributes).intersection(local_keys)
-
- # Write all the variable attributes
- for attr_name in sorted(attr_names):
- # Do not output 'conventions' or extended grid mapping attribute.
- if attr_name.lower() in ["conventions", "iris_extended_grid_mapping"]:
- continue
-
- value = cube.attributes[attr_name]
-
- # Convert any "managed" attributes. These convert between an internal
- # convenience representation and what is actually stored in the file.
- handler = ATTRIBUTE_HANDLERS.get(attr_name)
- if handler is not None:
- try:
- attr_name, value = handler.encode_object(value)
- except (TypeError, ValueError):
- string_value = str(value)
- msg = (
- f"Invalid value in managed '{attr_name}' attribute: {value!r}. "
- f"File attribute set to raw (string) value {string_value!r}."
- )
- warnings.warn(msg, category=iris.warnings.IrisSaveWarning)
- # Resolve by setting the **Iris** attr-name to the "raw" string repr
- # NB we expect the default written name to be the last listed
- attr_name = handler.iris_name
- value = string_value
-
- if attr_name in _CF_GLOBAL_ATTRS:
- msg = (
- "{attr_name!r} is being added as CF data variable "
- "attribute, but {attr_name!r} should only be a CF "
- "global attribute.".format(attr_name=attr_name)
- )
- warnings.warn(msg, category=iris.warnings.IrisCfSaveWarning)
-
- _setncattr(cf_var, attr_name, value)
-
- # Add the 'dataless' marker if needed
- if is_dataless:
- _setncattr(cf_var, self._DATALESS_ATTRNAME, "true")
-
- # Create the CF-netCDF data variable cell method attribute.
- cell_methods = self._create_cf_cell_methods(cube, dimension_names)
-
- if cell_methods:
- _setncattr(cf_var, "cell_methods", cell_methods)
-
- # Create the CF-netCDF grid mapping.
- self._create_cf_grid_mapping(cube, cf_var)
-
- return cf_var
-
- def _increment_name(self, varname):
- """Increment string name or begin increment.
-
- Avoidance of conflicts between variable names, where the name is
- incremented to distinguish it from others.
-
- Parameters
- ----------
- varname : str
- Variable name to increment.
+ data: np.typing.ArrayLike,
+ cf_var: CFDatasetVariable,
+ ) -> bool:
+ """Write the data here and now, if this backend handles it itself.
Returns
-------
- Incremented varname.
+ bool
+ Whether the data was written. False leaves the caller to write it
+ the ordinary way, directly or by a delayed store.
"""
- num = 0
- try:
- name, endnum = varname.rsplit("_", 1)
- if endnum.isdigit():
- num = int(endnum) + 1
- varname = name
- except ValueError:
- pass
-
- return "{}_{}".format(varname, num)
-
- def _lazy_stream_data(
- self,
- data: np.typing.ArrayLike,
- cf_var: threadsafe_nc.VariableWrapper,
- ) -> None:
- if hasattr(data, "shape") and data.shape == (1,) + cf_var.shape:
- # (Don't do this check for string data).
- # Reduce dimensionality where the data array has an extra dimension
- # versus the cf_var - to avoid a broadcasting ambiguity.
- # Happens when bounds data is for a scalar point - array is 2D but
- # contains just 1 row, so the cf_var is 1D.
- data = data.squeeze(axis=0)
-
- if hasattr(cf_var, "_data_array"):
- # The variable is not an actual netCDF4 file variable, but an emulating
- # object with an attached data array (either numpy or dask), which should be
- # copied immediately to the target. This is used as a hook to translate
+ # The base method takes any CFDatasetVariable, and an override may not
+ # ask for less. In this module the only variable there is, is netCDF's,
+ # and the members reached below are netCDF's alone.
+ cf_var = typing.cast("NetCDFDatasetVariable", cf_var)
+
+ if cf_var.is_emulated:
+ # The variable is not an actual file variable, but an emulating
+ # object with an attached data array (either numpy or dask), which should
+ # be copied immediately to the target. This is used as a hook to translate
# data to/from netcdf data container objects in other packages, such as
# xarray.
# See https://github.com/SciTools/iris/issues/4994 "Xarray bridge".
@@ -2626,94 +518,20 @@ def _lazy_stream_data(
# 'set_auto_chartostring(True)'.
# Therefore, do here what an EncodedVariable.__setitem__ would do : ..
# .. get details from the file (char) variable to be written ..
- encoder = VariableEncoder.from_var(cf_var._contained_instance)
+ encoder = VariableEncoder.from_var(cf_var.unencoded_variable)
# .. apply encoding to get the bytes to write.
data = encoder.encode_strings_as_bytearray(data)
- cf_var._data_array = data
-
- else:
- doing_delayed_save = is_lazy_data(data)
- if doing_delayed_save:
- if self._is_nczarr:
- # NCZarr cannot be reopened for deferred writes.
- # Collect here; da.store() will be called in __exit__.
- self._nczarr_writes.append((data, cf_var))
- return
-
- # save lazy data with a delayed operation. For now, we just record the
- # necessary information -- a single, complete delayed action is constructed
- # later by a call to delayed_completion().
- def store(
- data: np.typing.ArrayLike,
- cf_var: threadsafe_nc.VariableWrapper,
- ) -> None:
- # Create a data-writeable object that we can stream into, which
- # encapsulates the file to be opened + variable to be written.
- # Note: we do *not* support selectable string encoding for writes,
- # so this never needs to be a _thread_safe_nc.NetCDFWriteProxy.
- write_wrapper = bytecoding_datasets.EncodedNetCDFWriteProxy(
- self.filepath, cf_var, self.file_write_lock
- )
- # Add to the list of delayed writes, used in delayed_completion().
- self._delayed_writes.append((data, write_wrapper))
-
- else:
- # Real data is always written directly, i.e. not via lazy save.
- def store(
- data: np.typing.ArrayLike,
- cf_var: threadsafe_nc.VariableWrapper,
- ) -> None:
- cf_var[:] = data # type: ignore[index]
-
- # Store the data.
- store(data, cf_var)
-
- def delayed_completion(self) -> Delayed:
- """Perform file completion for delayed saves.
-
- Create and return a :class:`dask.delayed.Delayed` to perform file
- completion for delayed saves.
-
- Returns
- -------
- :class:`dask.delayed.Delayed`
-
- Notes
- -----
- The dataset *must* be closed (saver has exited its context) before the
- result can be computed, otherwise computation will hang (never return).
- """
- if self._delayed_writes:
- # Create a single delayed da.store operation to complete the file.
- sources, targets = zip(*self._delayed_writes)
- result = da.store(sources, targets, compute=False, lock=False)
-
- else:
- # Return a do-nothing delayed, for usage consistency.
- @dask.delayed
- def no_op():
- return None
-
- result = no_op()
-
- return result
-
- def complete(self) -> None:
- """Complete file by computing any delayed variable saves.
-
- This requires that the Saver has closed the dataset (exited its context).
+ cf_var.emulated_data_array = data
+ return True
- """
- if self._dataset.isopen():
- msg = (
- "Cannot call Saver.complete() until its dataset is closed, "
- "i.e. the saver's context has exited."
- )
- raise ValueError(msg)
+ if is_lazy_data(data) and self._is_nczarr:
+ # NCZarr cannot be reopened for deferred writes.
+ # Collect here; da.store() will be called in __exit__.
+ self._nczarr_writes.append((data, cf_var))
+ return True
- # Complete the saves now
- dask.compute(self.delayed_completion())
+ return False
def save(
@@ -3099,9 +917,11 @@ def is_valid_packspec(p):
if iris.config.netcdf.conventions_override:
# Set to the default if custom conventions are not available.
- conventions = cube.attributes.get("Conventions", CF_CONVENTIONS_VERSION)
+ conventions = cube.attributes.get(
+ "Conventions", cf_saver.CF_CONVENTIONS_VERSION
+ )
else:
- conventions = CF_CONVENTIONS_VERSION
+ conventions = cf_saver.CF_CONVENTIONS_VERSION
# Perform a CF patch of the conventions attribute.
cf_profile_available = iris.site_configuration.get("cf_profile") not in [
@@ -3201,4 +1021,4 @@ def save_mesh(
# Add a conventions attribute.
# TODO: add 'UGRID' to conventions, when this is agreed with CF ?
- sman.update_global_attributes(Conventions=CF_CONVENTIONS_VERSION)
+ sman.update_global_attributes(Conventions=cf_saver.CF_CONVENTIONS_VERSION)
diff --git a/lib/iris/fileformats/netcdf/ugrid_load.py b/lib/iris/fileformats/netcdf/ugrid_load.py
index 04a784b4f4..d2f46bb779 100644
--- a/lib/iris/fileformats/netcdf/ugrid_load.py
+++ b/lib/iris/fileformats/netcdf/ugrid_load.py
@@ -20,6 +20,8 @@
"""
+from contextlib import ExitStack
+from dataclasses import dataclass
from functools import partial
from itertools import groupby
from pathlib import Path
@@ -29,7 +31,11 @@
from iris.common.mixin import CFVariableMixin
from iris.config import get_logger
from iris.coords import AuxCoord
-from iris.io import _is_nczarr_fragment, decode_uri, expand_filespecs
+from iris.io import (
+ _REMOTE_STORE_SCHEMES,
+ _categorise_uri,
+ expand_filespecs,
+)
from iris.loading import LoadProblems
from iris.mesh.components import Connectivity, MeshXY
from iris.util import guess_coord_axis
@@ -87,8 +93,119 @@ def _meshes_from_cf(cf_reader):
return meshes
+@dataclass
+class _PendingMeshCoords:
+ """MeshCoords built ahead of their cube, waiting to be attached to it."""
+
+ coords: list
+ dim: int | None
+ capture_kwargs: dict
+
+
+def _prepare_mesh_coords(cf_reader, meshes, cf_var):
+ """Build the MeshCoords a data variable needs, before its cube exists.
+
+ Unconventional 'split' usage of ``_add_or_capture``: this half builds and
+ captures, :func:`_attach_mesh_coords` adds. The split is forced by
+ attribute handling - built after the cube,
+ ``helpers._add_unused_attributes`` has already taken the variable's
+ ``mesh`` and ``location`` attributes for cube attributes.
+
+ Parameters
+ ----------
+ cf_reader : iris.fileformats.cf.CFReader
+ The reader the mesh variables were found in.
+ meshes : dict
+ Meshes by name, as returned by :func:`_meshes_from_cf`.
+ cf_var : iris.fileformats.cf.CFDataVariable
+ The data variable about to be built into a cube.
+
+ Returns
+ -------
+ _PendingMeshCoords or None
+ None when `cf_var` names no mesh, or names one this dataset does not
+ hold.
+
+ """
+ # NOTE: dynamic imports avoid circularity : see note with module imports
+ from iris.cube import Cube
+ from iris.fileformats._nc_load_rules.helpers import _add_or_capture
+
+ # Not this module's `logger`, which sits at INFO and would drop a debug
+ # message at source. `TestsMesh.test_missing_mesh` asserts the package
+ # logger, where this message has always been raised.
+ from iris.fileformats.netcdf import logger as netcdf_logger
+
+ mesh_name = cf_var.attributes.get("mesh")
+ if mesh_name is None:
+ return None
+
+ mesh = meshes.get(mesh_name)
+ if mesh is None:
+ message = (
+ f"Mesh '{mesh_name}' - "
+ f"referenced by variable: '{cf_var.cf_name}' - "
+ "could not be found in file."
+ )
+ netcdf_logger.debug(message)
+ return None
+
+ pending = _PendingMeshCoords(
+ coords=[],
+ dim=None,
+ capture_kwargs=dict(
+ cf_var=cf_reader.cf_group.meshes[mesh_name],
+ # MeshCoords are an Iris concept; the best fallback we have is to
+ # capture the CF Mesh.
+ destination=LoadProblems.Problem.Destination(
+ iris_class=Cube,
+ identifier=cf_var.cf_name,
+ ),
+ ),
+ )
+
+ def _stash(coords_and_dim):
+ pending.coords, pending.dim = coords_and_dim
+
+ _ = _add_or_capture(
+ build_func=partial(_build_mesh_coords, mesh, cf_var),
+ add_method=partial(_stash),
+ **pending.capture_kwargs,
+ )
+ return pending
+
+
+def _attach_mesh_coords(pending, cube):
+ """Attach MeshCoords prepared by :func:`_prepare_mesh_coords` to their cube.
+
+ Parameters
+ ----------
+ pending : _PendingMeshCoords
+ The return of :func:`_prepare_mesh_coords`. Its `coords` are empty if
+ the build failed, in which case the failure is already recorded and
+ this call adds nothing.
+ cube : iris.cube.Cube
+ The cube built from the data variable the MeshCoords were prepared
+ for.
+
+ """
+ # NOTE: dynamic imports avoid circularity : see note with module imports
+ from iris.fileformats._nc_load_rules.helpers import _add_or_capture
+
+ def _add(coords_and_dim):
+ coords, dim = coords_and_dim
+ for coord in coords:
+ cube.add_aux_coord(coord, dim)
+
+ _ = _add_or_capture(
+ build_func=partial(lambda: (pending.coords, pending.dim)),
+ add_method=partial(_add),
+ **pending.capture_kwargs,
+ )
+
+
def load_mesh(uris, var_name=None):
- """Load a single :class:`~iris.mesh.MeshXY` object from one or more NetCDF files.
+ """Load a single :class:`~iris.mesh.MeshXY` from one or more files or stores.
Raises an error if more/less than one
:class:`~iris.mesh.MeshXY` is found.
@@ -96,8 +213,9 @@ def load_mesh(uris, var_name=None):
Parameters
----------
uris : str or iterable of str
- One or more filenames/URI's. Filenames can include wildcards. Any URI's
- must support OpenDAP or NcZarr.
+ One or more filenames/URI's. Filenames can include wildcards. NetCDF
+ files and Zarr stores are read; anything else is ignored. Any URI's
+ must support OpenDAP, NcZarr, or a Zarr remote store scheme.
var_name : str, optional
Only return a :class:`~iris.mesh.MeshXY` if its
var_name matches this value.
@@ -117,13 +235,14 @@ def load_mesh(uris, var_name=None):
def load_meshes(uris, var_name=None):
- r"""Load :class:`~iris.mesh.MeshXY` objects from one or more NetCDF files.
+ r"""Load :class:`~iris.mesh.MeshXY` objects from files or Zarr stores.
Parameters
----------
uris : str or iterable of str
- One or more filenames/URI's. Filenames can include wildcards. Any URI's
- must support OpenDAP nor NcZarr.
+ One or more filenames/URI's. Filenames can include wildcards. NetCDF
+ files and Zarr stores are read; anything else is ignored. Any URI's
+ must support OpenDAP, NcZarr, or a Zarr remote store scheme.
var_name : str, optional
Only return :class:`~iris.mesh.MeshXY` that have
var_names matching this value.
@@ -139,7 +258,7 @@ def load_meshes(uris, var_name=None):
# NOTE: no constraints or callbacks supported - these assume they are operating
# on a Cube.
# NOTE: dynamic imports avoid circularity : see note with module imports
- from iris.fileformats import FORMAT_AGENT
+ from iris.fileformats import FORMAT_AGENT, _load_zarr
from iris.fileformats.cf import CFReader
import iris.fileformats.netcdf.loader as nc_loader
@@ -151,44 +270,61 @@ def load_meshes(uris, var_name=None):
# Make a string, Dataset, or other single item, into an iterable.
uris = [uris]
- def _categorise(decoded: tuple[str, str, str | None]) -> tuple[str, str, str]:
- scheme, part, fragment = decoded
- category = "nczarr" if _is_nczarr_fragment(fragment) else scheme
- if fragment:
- part = f"{part}#{fragment}"
- return category, scheme, part
-
# Group collections of uris by their iris handler
# Create list of tuples relating schemes to part names.
- uri_tuples = sorted(_categorise(decode_uri(uri)) for uri in uris)
+ uri_tuples = sorted(_categorise_uri(uri) for uri in uris)
valid_sources = []
for category, groups in groupby(uri_tuples, key=lambda x: x[0]):
# Call each scheme handler with the appropriate URIs
if category == "file":
filenames = [part for _cat, _scheme, part in groups]
sources = expand_filespecs(filenames)
- elif category in ["http", "https", "nczarr"]:
+ elif category in {"http", "https", "nczarr"} | _REMOTE_STORE_SCHEMES:
sources = [f"{scheme}:{part}" for _cat, scheme, part in groups]
else:
message = f"Iris cannot handle the URI scheme: {category}"
raise ValueError(message)
for source in sources:
- if category == "file":
+ # iris.loading._generate_cubes makes this same file-or-store
+ # distinction independently, for general cube loading - keep
+ # both in step.
+ if category == "file" and not Path(source).is_dir():
with open(source, "rb") as fh:
handling_format_spec = FORMAT_AGENT.get_spec(Path(source).name, fh)
else:
handling_format_spec = FORMAT_AGENT.get_spec(source, None)
+ # TODO: drop this format dispatch once the netCDF and Zarr
+ # cube-building loops are unified. This copy, not the mesh code
+ # below, is the only reason a netCDF-named module names Zarr at
+ # all. See the merge-back section of
+ # `docs/superpowers/specs/2026-09-21-zarr-io-design.md`.
+ #
+ # `_load_zarr` is compared, never `zarr.load_cubes`: a store's
+ # presence is what should reach zarr-python, not this comparison.
if handling_format_spec.handler == nc_loader.load_cubes:
- valid_sources.append(source)
+ valid_sources.append((source, False))
+ elif handling_format_spec.handler == _load_zarr:
+ valid_sources.append((source, True))
else:
- message = f"Ignoring non-NetCDF file: {source}"
+ message = f"Ignoring file of a format that holds no mesh: {source}"
logger.info(msg=message, extra=dict(cls=None))
result = {}
- for source in valid_sources:
- with CFReader(source) as cf_reader:
+ for source, is_zarr in valid_sources:
+ with ExitStack() as stack:
+ if is_zarr:
+ # Deferred: a caller that named no Zarr store must not reach
+ # zarr-python at all.
+ from iris.fileformats.zarr._dataset import ZarrDataset
+
+ # CFReader takes an open CFDataset or a filename, so the mesh
+ # machinery below never learns which it was given.
+ target = stack.enter_context(ZarrDataset(source))
+ else:
+ target = source
+ cf_reader = stack.enter_context(CFReader(target))
meshes_dict = _meshes_from_cf(cf_reader)
meshes = list(meshes_dict.values())
if var_name is not None:
@@ -210,12 +346,12 @@ def _build_aux_coord(coord_var):
# NOTE: dynamic imports avoid circularity : see note with module imports
from iris.fileformats._nc_load_rules.helpers import get_attr_units, get_names
from iris.fileformats.cf import CFUGridAuxiliaryCoordinateVariable
- from iris.fileformats.netcdf import loader as nc_loader
+ from iris.fileformats.cf import loader as cf_loader
assert isinstance(coord_var, CFUGridAuxiliaryCoordinateVariable)
attributes = {}
attr_units = get_attr_units(coord_var, attributes)
- points_data = nc_loader._get_cf_var_data(coord_var)
+ points_data = cf_loader._get_cf_var_data(coord_var)
# Bounds will not be loaded:
# Bounds may be present, but the UGRID conventions state this would
@@ -227,7 +363,7 @@ def _build_aux_coord(coord_var):
climatological = False
# TODO: use CF_ATTR_CLIMATOLOGY on re-integration, when no longer
# 'experimental'.
- attr_climatology = getattr(coord_var, "climatology", None)
+ attr_climatology = coord_var.attributes.get("climatology")
if attr_climatology is not None:
climatology_vars = coord_var.cf_group.climatology
climatological = attr_climatology in climatology_vars
@@ -267,15 +403,21 @@ def _build_connectivity(connectivity_var, element_dims):
# NOTE: dynamic imports avoid circularity : see note with module imports
from iris.fileformats._nc_load_rules.helpers import get_attr_units, get_names
from iris.fileformats.cf import CFUGridConnectivityVariable
- from iris.fileformats.netcdf import loader as nc_loader
+ from iris.fileformats.cf import loader as cf_loader
assert isinstance(connectivity_var, CFUGridConnectivityVariable)
attributes = {}
attr_units = get_attr_units(connectivity_var, attributes)
- indices_data = nc_loader._get_cf_var_data(connectivity_var)
+ indices_data = cf_loader._get_cf_var_data(connectivity_var)
- cf_role = connectivity_var.cf_role
- start_index = connectivity_var.start_index
+ # UGRID requires both of these on a connectivity variable and Iris
+ # validates neither. CFUGridConnectivityVariable.identify finds this
+ # variable through the mesh variable's role attribute, never through
+ # anything on the variable itself, so neither name is guaranteed to be
+ # here. A file omitting either raises, and the mesh build is captured into
+ # LOAD_PROBLEMS - the cube then loads without a mesh. SciTools/iris#7332.
+ cf_role = connectivity_var.attributes["cf_role"]
+ start_index = connectivity_var.attributes["start_index"]
dim_names = connectivity_var.dimensions
# Connectivity arrays must have two dimensions.
@@ -312,18 +454,18 @@ def _build_mesh(cf, mesh_var):
# NOTE: dynamic imports avoid circularity : see note with module imports
from iris.fileformats._nc_load_rules.helpers import get_attr_units, get_names
from iris.fileformats.cf import CFUGridMeshVariable
- from iris.fileformats.netcdf import loader as nc_loader
+ from iris.fileformats.cf import loader as cf_loader
assert isinstance(mesh_var, CFUGridMeshVariable)
attributes = {}
attr_units = get_attr_units(mesh_var, attributes)
cf_role_message = None
- if not hasattr(mesh_var, "cf_role"):
+ if "cf_role" not in mesh_var.attributes:
cf_role_message = f"{mesh_var.cf_name} has no cf_role attribute."
cf_role = "mesh_topology"
else:
- cf_role = getattr(mesh_var, "cf_role")
+ cf_role = mesh_var.attributes["cf_role"]
if cf_role != "mesh_topology":
cf_role_message = f"{mesh_var.cf_name} has an inappropriate cf_role: {cf_role}."
if cf_role_message:
@@ -333,17 +475,17 @@ def _build_mesh(cf, mesh_var):
category=_WarnComboCfDefaulting,
)
- if hasattr(mesh_var, "volume_node_connectivity"):
+ if "volume_node_connectivity" in mesh_var.attributes:
topology_dimension = 3
- elif hasattr(mesh_var, "face_node_connectivity"):
+ elif "face_node_connectivity" in mesh_var.attributes:
topology_dimension = 2
- elif hasattr(mesh_var, "edge_node_connectivity"):
+ elif "edge_node_connectivity" in mesh_var.attributes:
topology_dimension = 1
else:
# Nodes only. We aren't sure yet whether this is a valid option.
topology_dimension = 0
- if not hasattr(mesh_var, "topology_dimension"):
+ if "topology_dimension" not in mesh_var.attributes:
msg = (
f"MeshXY variable {mesh_var.cf_name} has no 'topology_dimension'"
f" : *Assuming* topology_dimension={topology_dimension}"
@@ -351,7 +493,7 @@ def _build_mesh(cf, mesh_var):
)
warnings.warn(msg, category=_WarnComboCfDefaulting)
else:
- quoted_topology_dimension = mesh_var.topology_dimension
+ quoted_topology_dimension = mesh_var.attributes["topology_dimension"]
if quoted_topology_dimension != topology_dimension:
msg = (
f"*Assuming* 'topology_dimension'={topology_dimension}"
@@ -367,8 +509,8 @@ def _build_mesh(cf, mesh_var):
)
node_dimension = None
- edge_dimension = getattr(mesh_var, "edge_dimension", None)
- face_dimension = getattr(mesh_var, "face_dimension", None)
+ edge_dimension = mesh_var.attributes.get("edge_dimension")
+ face_dimension = mesh_var.attributes.get("face_dimension")
node_coord_args = []
edge_coord_args = []
@@ -377,12 +519,12 @@ def _build_mesh(cf, mesh_var):
coord_and_axis = _build_aux_coord(coord_var)
coord = coord_and_axis[0]
- if coord.var_name in mesh_var.node_coordinates.split():
+ if coord.var_name in mesh_var.attributes["node_coordinates"].split():
node_coord_args.append(coord_and_axis)
node_dimension = coord_var.dimensions[0]
- elif coord.var_name in getattr(mesh_var, "edge_coordinates", "").split():
+ elif coord.var_name in mesh_var.attributes.get("edge_coordinates", "").split():
edge_coord_args.append(coord_and_axis)
- elif coord.var_name in getattr(mesh_var, "face_coordinates", "").split():
+ elif coord.var_name in mesh_var.attributes.get("face_coordinates", "").split():
face_coord_args.append(coord_and_axis)
# TODO: support volume_coordinates.
else:
@@ -403,7 +545,7 @@ def _build_mesh(cf, mesh_var):
connectivity, first_dim_name = _build_connectivity(
connectivity_var, element_dims
)
- assert connectivity.var_name == getattr(mesh_var, connectivity.cf_role)
+ assert connectivity.var_name == mesh_var.attributes[connectivity.cf_role]
connectivity_args.append(connectivity)
# If the mesh_var has not supplied the dimension name, it is safe to
@@ -434,7 +576,7 @@ def _build_mesh(cf, mesh_var):
mesh_elements = list(mesh.all_coords) + list(mesh.all_connectivities) + [mesh]
mesh_elements = filter(None, mesh_elements)
for iris_object in mesh_elements:
- nc_loader._add_unused_attributes(iris_object, cf.cf_group[iris_object.var_name])
+ cf_loader._add_unused_attributes(iris_object, cf.cf_group[iris_object.var_name])
return mesh
@@ -453,12 +595,13 @@ def _build_mesh_coords(mesh, cf_var):
"edge": mesh.edge_dimension,
"face": mesh.face_dimension,
}
- location = getattr(cf_var, "location", "<empty>")
+ location = cf_var.attributes.get("location", "<empty>")
if location is None or location not in element_dimensions:
# We should probably issue warnings and recover, but that is too much
# work. Raising a more intelligible error is easy to do though.
msg = (
- f"mesh data variable {cf_var.name!r} has an invalid location={location!r}."
+ f"mesh data variable {cf_var.cf_name!r} has an invalid "
+ f"location={location!r}."
)
raise ValueError(msg)
mesh_dim_name = element_dimensions.get(location)
@@ -469,10 +612,10 @@ def _build_mesh_coords(mesh, cf_var):
mesh_dim = cf_var.dimensions.index(mesh_dim_name)
else:
msg = (
- f"mesh data variable {cf_var.name!r} does not have the "
+ f"mesh data variable {cf_var.cf_name!r} does not have the "
f"{location} mesh dimension {mesh_dim_name!r}, in its dimensions."
)
raise ValueError(msg)
- mesh_coords = mesh.to_MeshCoords(location=cf_var.location)
+ mesh_coords = mesh.to_MeshCoords(location=location)
return mesh_coords, mesh_dim
diff --git a/lib/iris/fileformats/zarr/__init__.py b/lib/iris/fileformats/zarr/__init__.py
new file mode 100644
index 0000000000..3b6306653e
--- /dev/null
+++ b/lib/iris/fileformats/zarr/__init__.py
@@ -0,0 +1,121 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Native Zarr support, built on zarr-python.
+
+.. z_reference:: iris.fileformats.zarr
+ :tags: topic_load_save
+
+ API reference
+
+Reads Zarr version 2 and version 3 stores, local or remote. Not NCZarr: that
+is netCDF-c's own Zarr driver, reached through a URL fragment such as
+``file:///path/store.zarr#mode=nczarr,file``, and it stays where it is in
+:mod:`iris.fileformats.netcdf`.
+
+``zarr`` is an optional dependency. Nothing in :mod:`iris.fileformats` imports
+this package until a Zarr target is actually loaded, so Iris installs and
+imports without it.
+
+This package shadows the third-party ``zarr`` distribution by name. Python 3
+resolves ``import zarr`` absolutely, so inside these modules it still finds the
+third-party package; xarray's ``xarray.backends.zarr`` has the same
+arrangement.
+
+See sections 4.1 and 4.4 of
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+import importlib.metadata
+
+from packaging.version import Version
+
+__all__ = ["load_cubes"]
+
+#: The version floor ``docs/src/user_manual/explanation/zarr_io.rst`` and
+#: ``requirements/py3*.yml`` both promise.
+_MINIMUM_VERSION = "3.0.8"
+
+#: Raised whether ``zarr`` is missing entirely or present below
+#: :data:`_MINIMUM_VERSION`. Defined once here, not in
+#: :func:`iris.fileformats._load_zarr`, so that the two routes a caller can
+#: reach a Zarr load through - ``iris.load`` and calling
+#: :func:`load_cubes` directly - produce the same text rather than each
+#: carrying its own copy.
+_MISSING_MESSAGE = (
+ "Unable to load a Zarr store - the 'zarr' package is not "
+ "installed. Install zarr >=3.0.8, for example with "
+ "'conda install -c conda-forge zarr'."
+)
+
+
+def _check_zarr_version() -> None:
+ """Raise a clear ``ImportError`` if ``zarr`` is installed below the floor.
+
+ Reads installed-distribution metadata rather than importing ``zarr``
+ itself: a zarr-python 2.x ``Group`` has no ``.metadata``, so importing it
+ only to compare `__version__` would load far enough to reach
+ :class:`~iris.fileformats.zarr._dataset.ZarrDataset` and fail there
+ instead, with a bare ``AttributeError``.
+
+ Does nothing if ``zarr`` is not installed at all - :func:`__getattr__`
+ below reports that case itself, from the import it goes on to attempt.
+
+ Raises
+ ------
+ ImportError
+ If ``zarr`` is installed below :data:`_MINIMUM_VERSION`.
+
+ """
+ try:
+ installed = importlib.metadata.version("zarr")
+ except importlib.metadata.PackageNotFoundError:
+ return
+
+ if Version(installed) < Version(_MINIMUM_VERSION):
+ raise ImportError(_MISSING_MESSAGE)
+
+
+def __getattr__(name):
+ """Forward ``load_cubes`` to :mod:`iris.fileformats.zarr.loader`.
+
+ PEP 562, not a plain ``from .loader import load_cubes``: ``loader``
+ imports ``._dataset``, which imports ``zarr`` at module level, so an
+ eager import here would make the dependency load with the package rather
+ than with a Zarr load - breaking the promise this module's own docstring
+ makes, that nothing imports ``zarr`` until a Zarr target is actually
+ loaded. Resist "simplifying" this back to a plain import.
+
+ Parameters
+ ----------
+ name : str
+ The attribute being looked up. Python calls this only for names that
+ are not already module attributes.
+
+ Returns
+ -------
+ Any
+ ``load_cubes``, the only name this forwards.
+
+ Raises
+ ------
+ AttributeError
+ For any other name.
+ ImportError
+ If ``zarr`` is not installed, or is installed below
+ :data:`_MINIMUM_VERSION`.
+
+ """
+ if name != "load_cubes":
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ _check_zarr_version()
+
+ try:
+ from .loader import load_cubes
+ except ImportError as exception:
+ raise ImportError(_MISSING_MESSAGE) from exception
+
+ return load_cubes
diff --git a/lib/iris/fileformats/zarr/_dataset.py b/lib/iris/fileformats/zarr/_dataset.py
new file mode 100644
index 0000000000..8c9d5a5683
--- /dev/null
+++ b/lib/iris/fileformats/zarr/_dataset.py
@@ -0,0 +1,701 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The Zarr implementation of the :mod:`iris.fileformats.cf.dataset` interface.
+
+This is the only module that knows both the CF interface and the zarr-python
+API. Everything Zarr-specific the CF layer could want - where dimension names
+are recorded, what a chunk is, what the store's own fill value is - either has
+a named member on the interface or lives here as a Zarr-only member.
+
+Three things about Zarr shape this module, and each is a trap if forgotten.
+
+**A dimension name guarantees less than a netCDF one.** The version 3 core
+specification makes ``dimension_names`` optional, allows a null entry, only
+*recommends* that names are distinct within an array, and places no
+restriction at all on reusing a name across arrays. All of that is conforming
+Zarr, so none of it may raise. A dimension with no name is **anonymous**, which
+Iris has always had: a cube dimension carrying no coordinate.
+
+**``Array.fill_value`` is not ``_FillValue``.** It says what unwritten storage
+reads back as; it does not say the value is missing. zarr-python defaults it to
+zero, so reading it as a missing-value sentinel masks every valid zero in an
+array that never declared one. :attr:`ZarrDatasetVariable.fill_value` reports
+it as the storage fact it is, and the masking decision is made in
+:mod:`iris.fileformats.zarr._decode` from the CF attributes.
+
+**The read unit is the inner chunk, even when the array is sharded.** Zarr's
+sharding codec is indexed, so a reader fetches only the inner chunks it needs.
+:attr:`ZarrDatasetVariable.chunking` therefore returns ``Array.chunks``, never
+``Array.shards``.
+
+Chunking is sized by :func:`iris._lazy_data._optimum_chunksize`, the same way
+the netCDF path sizes it, and is not then adjusted to the store's own
+chunking. Iris warns when a stored chunk is larger than Dask's target, from
+:func:`iris.fileformats.cf.loader._chunks_from_chunk_control`, and leaves the
+choice to ``CHUNK_CONTROL``.
+
+See sections 4.2, 4.4 and 4.6 of
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+from collections.abc import Callable, Iterator, Mapping, MutableMapping
+from functools import partial
+import json
+from types import MappingProxyType
+from typing import Any
+import warnings
+
+import dask
+import dask.array as da
+from dask.array.core import normalize_chunks
+import dask.config
+import dask.utils
+import numpy as np
+import zarr
+
+from iris._lazy_data import _optimum_chunksize, as_lazy_data
+from iris.exceptions import TranslationError
+from iris.fileformats.cf import loader as cf_loader
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+import iris.warnings
+
+from . import _decode
+
+#: What :attr:`CFDataset.format` answers, by the store's Zarr version.
+_FORMAT_BY_ZARR_VERSION = {2: "ZARR_V2", 3: "ZARR_V3"}
+
+#: The version 2 attribute carrying dimension names, as xarray and NCZarr
+#: write it. Storage, not a CF attribute, so it is withheld from `attributes`.
+_ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS"
+
+#: netCDF-c's own bookkeeping attribute, stamped onto the root group of every
+#: store it writes (confirmed on NCZarr output in both its `nczarr` and
+#: `xarray` URL modes, and never on a subgroup or an array). netCDF4-python
+#: hides it from `Dataset.ncattrs()` by convention, so it has never reached a
+#: cube loaded through netCDF; Zarr has no equivalent convention, so it must
+#: be withheld here for the same reason as `_ARRAY_DIMENSIONS` above -
+#: provenance the writer left behind, not CF metadata the user wrote.
+_NC_PROPERTIES = "_NCProperties"
+
+
+def _dimension_names(array) -> tuple:
+ """Return one name or None per dimension of `array`.
+
+ Parameters
+ ----------
+ array : :class:`zarr.Array`
+ The array to read names from. Version 3 carries them in its metadata;
+ version 2 carries them in the ``_ARRAY_DIMENSIONS`` attribute, which
+ is a de-facto convention rather than part of the Zarr specification.
+
+ Returns
+ -------
+ tuple
+ Of length ``array.ndim``. ``None`` where the store names no dimension.
+
+ Raises
+ ------
+ iris.exceptions.TranslationError
+ If `array` declares any Zarr version but 2 or 3. zarr-python parses
+ ``zarr_format`` as a ``Literal`` and rejects such an array before
+ Iris sees it, so this guards against a future widening there.
+
+ Warns
+ -----
+ IrisCfLoadWarning
+ If the store names a number of dimensions other than ``array.ndim``.
+ The version 3 specification says the lengths "must" match, so such a
+ store is malformed; its names are ignored and the array loads with
+ anonymous dimensions.
+
+ """
+ anonymous = (None,) * array.ndim
+ if not array.ndim:
+ return ()
+
+ zarr_format = array.metadata.zarr_format
+ if zarr_format == 3:
+ names = array.metadata.dimension_names
+ elif zarr_format == 2:
+ names = array.attrs.get(_ARRAY_DIMENSIONS)
+ else:
+ message = (
+ "Iris reads Zarr version 2 and version 3 stores. This array "
+ f"declares Zarr version {zarr_format!r}: {array.basename}"
+ )
+ raise TranslationError(message)
+
+ if names is None:
+ return anonymous
+
+ if len(names) != array.ndim:
+ message = (
+ "Ignoring malformed dimension_names, which names "
+ f"{len(names)} dimensions of a {array.ndim}-dimensional array: "
+ f"{array.basename}"
+ )
+ warnings.warn(message, category=iris.warnings.IrisCfLoadWarning)
+ return anonymous
+
+ return tuple(names)
+
+
+def _assemble_dimensions(variables) -> dict:
+ """Return the dimension sizes the store's arrays agree on.
+
+ Parameters
+ ----------
+ variables : mapping of str to :class:`ZarrDatasetVariable`
+ Every array in one group.
+
+ Returns
+ -------
+ dict
+ Name to size, for every name used at exactly one size. Anonymous
+ dimensions have no name and so are absent.
+
+ Warns
+ -----
+ IrisCfLoadWarning
+ For each name used at more than one size. Zarr places no restriction
+ on reusing a name across arrays, so this is conforming and must not
+ raise; merging the two would produce a wrong cube, so the name is
+ omitted instead and the arrays that use it keep it in their own
+ `dimensions`.
+
+ """
+ sizes: dict[str, int] = {}
+ conflicts: dict[str, set[int]] = {}
+ for variable in variables.values():
+ for name, size in zip(variable.dimensions, variable.shape):
+ if name is None:
+ continue
+ if name not in sizes:
+ sizes[name] = size
+ elif sizes[name] != size:
+ conflicts.setdefault(name, {sizes[name]}).add(size)
+
+ for name, found in conflicts.items():
+ message = (
+ "Ignoring a dimension name used at more than one size, "
+ f"{sorted(found)}: {name}"
+ )
+ warnings.warn(message, category=iris.warnings.IrisCfLoadWarning)
+ del sizes[name]
+
+ return sizes
+
+
+class _ZarrAttributes(MutableMapping):
+ """A Zarr node's attributes as a mapping, less the ones `withhold` names.
+
+ Which attributes those are is the caller's decision, taken where the
+ mapping is built. Both the constants named there - `_ARRAY_DIMENSIONS`
+ and `_NC_PROPERTIES` - carry the reason with them.
+ """
+
+ def __init__(
+ self,
+ node: "zarr.Group | zarr.Array",
+ *,
+ withhold: frozenset = frozenset(),
+ ):
+ """Materialise the attributes of `node`, a zarr array or group.
+
+ Writing through this mapping writes to `node`, which is the store.
+ """
+ self._node = node
+ self._withhold = frozenset(withhold)
+ self._values = {
+ key: value for key, value in node.attrs.items() if key not in self._withhold
+ }
+
+ def __getitem__(self, key: str) -> Any:
+ """Return `key`'s value."""
+ return self._values[key]
+
+ def __setitem__(self, key: str, value: Any) -> None:
+ """Set `key`'s value, here and in the store."""
+ if key in self._withhold:
+ msg = f"Cannot set a storage attribute through `attributes`: {key}"
+ raise KeyError(msg)
+ self._node.attrs[key] = value
+ self._values[key] = value
+
+ def __delitem__(self, key: str) -> None:
+ """Remove `key`, here and from the store."""
+ del self._node.attrs[key]
+ del self._values[key]
+
+ def __iter__(self) -> Iterator[str]:
+ """Return an iterator over the attribute names, in store order."""
+ return iter(self._values)
+
+ def __len__(self) -> int:
+ """Return the number of attributes."""
+ return len(self._values)
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self._values!r})"
+
+
+class ZarrDatasetVariable(CFDatasetVariable):
+ """One array of a Zarr store, presented through the CF interface."""
+
+ def __init__(self, array, location: str, format: str):
+ """Wrap `array`.
+
+ Parameters
+ ----------
+ array : :class:`zarr.Array`
+ The array this variable presents.
+ location : str
+ The path or URL of the dataset this array belongs to.
+ format : str
+ ``ZARR_V2`` or ``ZARR_V3``, as its dataset reports.
+
+ """
+ self._array = array
+ self._location = location
+ self._format = format
+ self._dimensions = _dimension_names(array)
+ self._attributes = _ZarrAttributes(
+ array, withhold=frozenset({_ARRAY_DIMENSIONS})
+ )
+
+ @property
+ def name(self) -> str:
+ """The array's name within its group."""
+ return self._array.basename
+
+ @property
+ def location(self) -> str:
+ """The path or URL of the dataset holding this variable."""
+ return self._location
+
+ @property
+ def format(self) -> str:
+ """``ZARR_V2`` or ``ZARR_V3``."""
+ return self._format
+
+ @property
+ def dimensions(self) -> tuple:
+ """One name per dimension, ``None`` where the store names none."""
+ return self._dimensions
+
+ @property
+ def shape(self) -> tuple:
+ """The array's shape."""
+ return tuple(self._array.shape)
+
+ @property
+ def dtype(self) -> np.dtype:
+ """The array's stored data type."""
+ return self._array.dtype
+
+ @property
+ def size(self) -> int:
+ """The total number of elements in the array."""
+ return int(self._array.size)
+
+ @property
+ def fill_value(self) -> Any:
+ """What unwritten storage reads back as. **Not** a missing-value flag.
+
+ Version 3 requires this field, and zarr-python defaults it to zero, so
+ it says nothing about whether a value is missing. The CF
+ ``_FillValue`` and ``missing_value`` attributes say that, and
+ :mod:`iris.fileformats.zarr._decode` reads them.
+
+ """
+ return self._array.fill_value
+
+ @property
+ def chunking(self) -> tuple | None:
+ """The array's inner chunk shape, never its shard shape."""
+ chunks = self._array.chunks
+ return tuple(chunks) if chunks else None
+
+ @property
+ def storage_settings(self) -> Mapping:
+ """The Zarr storage settings this array holds."""
+ return MappingProxyType(
+ {
+ "chunks": self._array.chunks,
+ "shards": self._array.shards,
+ "codecs": self._array.metadata.to_dict().get("codecs"),
+ "zarr_format": self._array.metadata.zarr_format,
+ }
+ )
+
+ @property
+ def attributes(self) -> MutableMapping:
+ """The array's CF attributes, less the ones that are really storage."""
+ return self._attributes
+
+ def __getitem__(self, keys) -> np.ndarray:
+ """Return the indexed portion of the array's data."""
+ return self._array[keys]
+
+ def __setitem__(self, keys, values) -> None:
+ """Write `values` into the indexed portion of the array."""
+ self._array[keys] = values
+
+ def read_data(self, chunking_policy: Callable[[], tuple]) -> Any:
+ """Return this array's data, lazily where a lazy array is worth it.
+
+ Parameters
+ ----------
+ chunking_policy : callable
+ Called with no arguments, returning the ``(chunks, dims_fixed)``
+ pair to build a lazy array with. Called only once the result is
+ known to be lazy.
+
+ Returns
+ -------
+ numpy.ndarray, numpy.ma.MaskedArray or dask.array.Array
+ Real data for a small array, lazy data otherwise. Masked whenever
+ the variable declares anything to decode.
+
+ Notes
+ -----
+ There is no data proxy. ``zarr.Array`` is picklable, a chunk is an
+ independent object in a key-value store so concurrent reads need no
+ lock, and a store is a namespace rather than an open file handle. CF
+ decoding is a ``map_blocks`` layer over the raw array rather than
+ something hidden inside ``__getitem__``, so it is visible in the
+ graph and testable without a store.
+
+ """
+ parameters = _decode.decode_parameters(
+ self._attributes,
+ dtype=self.dtype,
+ storage_fill_value=self.fill_value,
+ zarr_format=self.zarr_format,
+ variable_name=self.name,
+ )
+ decoding = _decode.decoding_required(parameters)
+
+ if self.size * self.dtype.itemsize < cf_loader._LAZYVAR_MIN_BYTES:
+ # A lazy array would cost more memory AND more time to access.
+ result = np.asanyarray(self._array[...])
+ if decoding:
+ result = _decode.mask_and_scale(result, **parameters)
+ else:
+ chunks, dims_fixed = chunking_policy()
+ # Chunks are sized on the dtype the values will have once decoded,
+ # not on the one the store holds: a packed int8 that unpacks to
+ # float64 costs eight times its stored size the moment the decode
+ # layer runs, and Dask's target chunk size is a target for what one
+ # task holds. The netCDF path arrives at the same place from the
+ # other end - its data proxy declares the decoded dtype, and
+ # `as_lazy_data` sizes on the array it is given. `_unpacked_dtype`
+ # returns the stored dtype for a variable with no packing, so
+ # nothing moves in the usual case.
+ decoded_dtype = parameters["dtype"]
+
+ if chunks == "auto":
+ # CHUNK_CONTROL.as_dask(): the user asked for Dask's own
+ # policy, so the store's layout is deliberately not consulted.
+ # The policy is still Dask's - its algorithm, its configured
+ # target - but it is called here rather than left to
+ # `from_array`, which is the only way to hand it the decoded
+ # dtype. `from_array` would size on the array it is given, and
+ # the array it is given is the stored one.
+ chunks = tuple(
+ sizes[0]
+ for sizes in normalize_chunks(
+ "auto", shape=self.shape, dtype=decoded_dtype
+ )
+ )
+ else:
+ if chunks is None:
+ # What `as_lazy_data` makes of None, done here because this
+ # path reaches `_optimum_chunksize` first and that does not
+ # accept it: one chunk spanning the array, divided down
+ # below.
+ chunks = self.shape
+ chunks = _optimum_chunksize(
+ chunks, shape=self.shape, dtype=decoded_dtype, dims_fixed=dims_fixed
+ )
+
+ # Fixed in every dimension, because the sizing above is the final
+ # word whichever branch produced it: `as_lazy_data` would otherwise
+ # re-run its own sizing on the stored dtype and undo it.
+ result = as_lazy_data(
+ self._array,
+ chunks=chunks,
+ meta=np.empty((0,) * self.ndim, dtype=self.dtype),
+ dims_fixed=(True,) * self.ndim,
+ cache_key=self.cache_key,
+ )
+
+ if decoding:
+ # `partial` rather than `**parameters` on the `map_blocks`
+ # call: `dtype` is a keyword `map_blocks` consumes for itself
+ # rather than forwarding, and `mask_and_scale` needs it too.
+ # Binding it once here also makes the dtype Dask declares and
+ # the dtype the layer produces one value rather than two
+ # statements that could disagree.
+ result = da.map_blocks(
+ partial(_decode.mask_and_scale, **parameters),
+ result,
+ dtype=decoded_dtype,
+ meta=np.ma.masked_array(np.empty((0,) * self.ndim, decoded_dtype)),
+ )
+
+ return result
+
+ def write_handle(self) -> "ZarrDatasetVariable":
+ """Return self: a Zarr write target needs no proxy.
+
+ ``zarr.Array`` is picklable and a chunk is an independent object in a
+ key-value store, so there is no open file handle to avoid holding and
+ no lock to acquire.
+
+ """
+ return self
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self.name!r}, {self.location!r})"
+
+ # Zarr-only members below.
+
+ @property
+ def array(self):
+ """The backing :class:`zarr.Array`. Nothing generic may read this."""
+ return self._array
+
+ @property
+ def zarr_format(self) -> int:
+ """2 or 3. The one Zarr fact that changes how a value is read."""
+ return self._array.metadata.zarr_format
+
+ @property
+ def cache_key(self) -> str:
+ """An identity for this array that goes stale when the array does.
+
+ `iris._lazy_data.CACHE` is process-wide, so an address alone is not
+ enough: a store rewritten in place keeps its address and changes its
+ contents. The metadata document covers shape, chunks, shards, dtype,
+ fill value, codecs, dimension names and attributes in one value, and
+ is treated as opaque - listing the parts Iris cares about would go
+ stale, silently, the first time zarr-python added a field that
+ mattered.
+ """
+ return json.dumps(
+ [self._location, self._array.path, self._array.metadata.to_dict()],
+ sort_keys=True,
+ )
+
+
+#: What `mode` a dataset may be opened in. Writing is not implemented.
+_READ_MODES = ("r",)
+
+#: The one message every unimplemented write member raises.
+_NO_SAVING = "Saving to Zarr is not supported."
+
+
+class ZarrDataset(CFDataset):
+ """A Zarr store, or one group of one, presented through the CF interface."""
+
+ def __init__(self, location, mode: str = "r", *, group: str | None = None):
+ """Open a Zarr store.
+
+ Parameters
+ ----------
+ location : str or :class:`pathlib.Path`
+ The store's path or URL. A URL is handed to zarr-python
+ unaltered; Iris adds no store library and handles no credentials,
+ so an unreachable scheme fails with zarr-python's own message. A
+ store object - such as a :class:`zarr.storage.MemoryStore`, which
+ has no path - is also accepted and handed through unaltered;
+ `location` then becomes zarr-python's own rendering of it, the
+ best name available for a store with no address.
+ mode : str, default="r"
+ Only ``"r"`` is supported.
+ group : str, optional
+ A ``/``-separated path to a group within the store. The root group
+ by default. Iris reads one group per call and does not walk
+ nested groups.
+
+ Raises
+ ------
+ NotImplementedError
+ For any `mode` other than ``"r"``.
+ FileNotFoundError
+ If `location` does not exist.
+ zarr.errors.GroupNotFoundError
+ If `location` is not a Zarr store, or holds no such `group`.
+
+ """
+ # Set first, so that close() is safe if opening fails.
+ self._group: zarr.Group | None = None
+ self._closed = False
+ self._variables: dict[str, ZarrDatasetVariable] | None = None
+ self._attributes: _ZarrAttributes | None = None
+ self._dimensions: dict[str, int] | None = None
+
+ if mode not in _READ_MODES:
+ raise NotImplementedError(f"{_NO_SAVING} Cannot open with mode={mode!r}.")
+
+ self._mode = mode
+ target = str(location)
+ self._location = target if group is None else f"{target.rstrip('/')}/{group}"
+
+ # `location` itself, not its stringified `target`, is handed to
+ # zarr-python: stringifying a store object such as a `MemoryStore`
+ # loses the store, leaving only a name zarr-python cannot reopen.
+ self._group = zarr.open_group(store=location, mode=mode, path=group)
+ self._format = _FORMAT_BY_ZARR_VERSION[self._group.metadata.zarr_format]
+
+ # Assembled here rather than on first access, because assembling is
+ # what warns about a dimension name used at two different lengths and
+ # a load never asks for the mapping: the CF layer reads each
+ # variable's own `dimensions`. Left to first access, that warning
+ # reached the unit tests and no user. It costs no extra reads -
+ # `variables` is materialised by everything that opens a store.
+ self._dimensions = _assemble_dimensions(self.variables)
+
+ @property
+ def location(self) -> str:
+ """The store's path or URL, with the group path when there is one."""
+ return self._location
+
+ @property
+ def format(self) -> str:
+ """``ZARR_V2`` or ``ZARR_V3``."""
+ return self._format
+
+ @property
+ def mode(self) -> str:
+ """The mode the store was opened in."""
+ return self._mode
+
+ @property
+ def closed(self) -> bool:
+ """Whether :meth:`close` has been called.
+
+ A Zarr store is a namespace rather than an open file handle, so there
+ is nothing to release and nothing else to ask.
+
+ """
+ return self._closed
+
+ @property
+ def variables(self) -> Mapping:
+ """The group's arrays, by name. Arrays in subgroups are not included."""
+ if self._variables is None:
+ assert self._group is not None
+ self._variables = {
+ name: ZarrDatasetVariable(array, self._location, self._format)
+ for name, array in self._group.arrays()
+ }
+ return self._variables
+
+ @property
+ def dimensions(self) -> Mapping:
+ """The dimension sizes this group's arrays agree on.
+
+ Assembled when the store is opened, not on first access here.
+ """
+ assert self._dimensions is not None
+ return self._dimensions
+
+ @property
+ def storage_settings(self) -> Mapping:
+ """The store-wide Zarr settings."""
+ assert self._group is not None
+ consolidated = self._group.metadata.consolidated_metadata
+ return MappingProxyType(
+ {
+ "zarr_format": self._group.metadata.zarr_format,
+ "consolidated": consolidated is not None,
+ }
+ )
+
+ @property
+ def attributes(self) -> MutableMapping:
+ """The group's attributes, less netCDF-c's own bookkeeping stamp."""
+ if self._attributes is None:
+ assert self._group is not None
+ self._attributes = _ZarrAttributes(
+ self._group, withhold=frozenset({_NC_PROPERTIES})
+ )
+ return self._attributes
+
+ def create_dimension(self, name: str, size: int | None) -> None:
+ """Raise: saving to Zarr is not supported."""
+ raise NotImplementedError(_NO_SAVING)
+
+ def create_variable(
+ self, name: str, dtype, dimensions=(), *, fill_value=None, **storage_settings
+ ):
+ """Raise: saving to Zarr is not supported."""
+ raise NotImplementedError(_NO_SAVING)
+
+ def supports_dtype(self, dtype) -> bool:
+ """Whether this store can hold a variable of this dtype.
+
+ Always True. Zarr's data types cover the whole of NumPy's fixed-width
+ numeric range, in both versions, so there is no dtype a caller could
+ reach here with that the store would have to narrow. Contrast
+ :meth:`~iris.fileformats.netcdf._dataset.NetCDFDataset.supports_dtype`,
+ which answers False for the integer types the netCDF 3 formats have no
+ type for.
+
+ Parameters
+ ----------
+ dtype : :class:`numpy.dtype`
+ The dtype a variable would be created with.
+
+ Returns
+ -------
+ bool
+
+ """
+ return True
+
+ def sync(self) -> None:
+ """Do nothing: a Zarr write is not buffered."""
+
+ def finalise(self) -> None:
+ """Raise: saving to Zarr is not supported.
+
+ A Zarr store's completion step is consolidating its metadata, which
+ only a writer performs.
+
+ """
+ raise NotImplementedError(_NO_SAVING)
+
+ def close(self) -> None:
+ """Record that this dataset is finished with.
+
+ A store is a namespace, not an open handle, so nothing is released.
+ The flag exists because callers - :class:`iris.fileformats.cf.CFReader`
+ among them - must be able to ask.
+
+ """
+ self._closed = True
+
+ def __repr__(self) -> str:
+ """Return a string representation."""
+ return f"{self.__class__.__name__}({self.location!r}, mode={self.mode!r})"
+
+ # Zarr-only members below.
+
+ @property
+ def subgroup_names(self) -> tuple:
+ """The names of this group's subgroups.
+
+ What the loader names when a group holds no data variables but the
+ store plainly holds data elsewhere.
+
+ """
+ assert self._group is not None
+ return tuple(self._group.group_keys())
diff --git a/lib/iris/fileformats/zarr/_decode.py b/lib/iris/fileformats/zarr/_decode.py
new file mode 100644
index 0000000000..e2064a430a
--- /dev/null
+++ b/lib/iris/fileformats/zarr/_decode.py
@@ -0,0 +1,387 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The CF decoding netCDF4 does for free and Zarr does not.
+
+netCDF4 applies ``_FillValue``/``missing_value`` masking,
+``valid_min``/``valid_max``/``valid_range`` masking and
+``scale_factor``/``add_offset`` unpacking on every read. Zarr returns a plain
+``numpy.ndarray`` and applies none of it, so these rules are implemented here.
+They live on the Zarr side rather than in the shared CF layer because the
+netCDF path gets the behaviour from the C library and has nothing to reuse.
+
+Everything here is a pure function of values, with no store and no Dask in
+sight. :func:`mask_and_scale` is what the loader lays over a raw array as a
+``map_blocks`` layer - visible in the graph, fusable with its neighbours, and
+testable on a plain ndarray.
+
+Three names are this module's surface: :func:`decode_parameters`,
+:func:`decoding_required` and :func:`mask_and_scale`. Everything else is a
+step of one of those, and carries a leading underscore to say so - the module
+being private already does not make the distinction worthless, since
+``_dataset.py`` should not be reaching for a decode step on its own.
+
+Two traps, both from the Zarr specification rather than from CF.
+
+**A floating ``_FillValue`` is base64 on version 3.** Released xarray encodes
+it as base64 of a little-endian ``float64`` for *every* floating dtype,
+including ``float32``, and its reader requires that form. It is the de-facto
+encoding, not a producer's mistake, so Iris reads both it and a plain JSON
+number. CF section 2.5.1 asks for "the scalar attribute with the name
+``_FillValue`` and of the same type as its variable"; this is a deliberate,
+narrow deviation, taken because interoperability with the other Zarr readers
+is the reason for native Zarr support.
+
+**``Array.fill_value`` is not a missing-value declaration.** It says what
+unwritten storage reads back as, and zarr-python defaults it to zero. On
+version 3 it is never a mask source: an ``int16`` array of ``[0, 1, 2]`` with
+no CF attributes would otherwise load with its valid zero masked. On version 2
+it is one, but only when no CF attribute is present, because version 2 stores
+predate the convention - which is what released xarray's
+``use_zarr_fill_value_as_mask`` does. zarr-python 3 defaults the field to zero
+for a version 2 store too, so a store written today can lose a genuine zero
+this way; :func:`decode_parameters` warns for exactly that case - the storage
+fill value is zero - and not for a non-zero one, which is the writer's own
+deliberate, pre-convention sentinel and not something to warn about.
+
+See section 4.4 of ``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+import base64
+import binascii
+import struct
+from typing import Any
+import warnings
+
+import numpy as np
+
+import iris.warnings
+
+#: The width of the base64 `_FillValue` encoding: one little-endian float64.
+_PACKED_DOUBLE = "<d"
+_PACKED_DOUBLE_BYTES = 8
+
+#: The Zarr version whose stores predate the CF `_FillValue` convention, and
+#: whose `Array.fill_value` is therefore read as a missing-value sentinel.
+_FILL_VALUE_IS_A_MASK_BEFORE = 3
+
+
+def _decode_fill_value(value: Any, dtype: np.dtype, variable_name: str) -> Any:
+ """Return a CF missing-value sentinel as a value of `dtype`.
+
+ Parameters
+ ----------
+ value : Any
+ The attribute's value, as the store holds it: a JSON number, or - for
+ a floating variable - the base64 of a little-endian ``float64``.
+ dtype : numpy.dtype
+ The variable's own dtype. The result is narrowed to it, because a
+ sentinel of a different width never equals anything in the array.
+ variable_name : str
+ Named in the warning, so the user can find the variable.
+
+ Returns
+ -------
+ numpy.generic or None
+ ``None`` when `value` is neither form, or does not fit `dtype` -
+ both are producer errors.
+
+ Warns
+ -----
+ IrisCfLoadWarning
+ When `value` cannot be read as a sentinel of `dtype`.
+
+ """
+ if isinstance(value, (bool, np.bool_)):
+ # A JSON true/false is a number to Python and nothing to CF.
+ decoded = None
+ elif isinstance(value, (int, float, np.number)):
+ decoded = value
+ elif isinstance(value, str) and dtype.kind == "f":
+ decoded = _decode_packed_double(value)
+ else:
+ decoded = None
+
+ if decoded is None:
+ message = (
+ "Ignoring a missing-value attribute that is neither a number nor "
+ f"the base64 of a float64, on variable {variable_name}: {value!r}"
+ )
+ warnings.warn(message, category=iris.warnings.IrisCfLoadWarning)
+ return None
+
+ try:
+ return dtype.type(decoded)
+ except (OverflowError, ValueError):
+ # Out of range for dtype: a producer error, not a reason to fail
+ # the whole load. A float dtype never raises here - an overflowing
+ # float narrows to +/-inf instead, which is ordinary numpy
+ # behaviour and not something to guard against.
+ message = (
+ f"Ignoring a missing-value attribute that does not fit {dtype}, "
+ f"on variable {variable_name}: {value!r}"
+ )
+ warnings.warn(message, category=iris.warnings.IrisCfLoadWarning)
+ return None
+
+
+def _decode_missing_values(value: Any, dtype: np.dtype, variable_name: str) -> list:
+ """Return every sentinel a ``missing_value`` attribute declares.
+
+ CF section 2.5.1 makes ``missing_value`` "a scalar or vector containing
+ values indicating missing data", so a JSON array there is as valid as a
+ number and every element of it masks - which is what netCDF4 does with a
+ vector ``missing_value``, and what Iris has always done through it.
+ ``_FillValue`` is not read this way: section 2.5.1 makes that one a
+ scalar.
+
+ Parameters
+ ----------
+ value : Any
+ The attribute's value: one sentinel, or a sequence of them.
+ dtype : numpy.dtype
+ The variable's own dtype, which every sentinel is narrowed to.
+ variable_name : str
+ Named in any warning.
+
+ Returns
+ -------
+ list
+ One entry per element :func:`_decode_fill_value` could read. An
+ element it could not is warned about and dropped, leaving the rest:
+ one bad sentinel in a vector is no reason to ignore the good ones.
+
+ """
+ if isinstance(value, (list, tuple)) or (
+ isinstance(value, np.ndarray) and value.ndim > 0
+ ):
+ elements = list(value)
+ else:
+ elements = [value]
+ decoded = (
+ _decode_fill_value(element, dtype, variable_name) for element in elements
+ )
+ return [sentinel for sentinel in decoded if sentinel is not None]
+
+
+def _decode_packed_double(value: str) -> float | None:
+ """Return the float in a base64 little-endian float64, or None."""
+ try:
+ raw = base64.b64decode(value, validate=True)
+ except (binascii.Error, ValueError):
+ return None
+ if len(raw) != _PACKED_DOUBLE_BYTES:
+ return None
+ return struct.unpack(_PACKED_DOUBLE, raw)[0]
+
+
+def decode_parameters(
+ attributes,
+ *,
+ dtype: np.dtype,
+ storage_fill_value: Any,
+ zarr_format: int,
+ variable_name: str,
+) -> dict:
+ """Return the CF decode parameters for one variable.
+
+ Parameters
+ ----------
+ attributes : mapping
+ The variable's CF attributes.
+ dtype : numpy.dtype
+ The variable's stored dtype, before any unpacking.
+ storage_fill_value : Any
+ ``zarr.Array.fill_value`` - what unwritten storage reads back as. A
+ mask source only on version 2, and only when the variable declares no
+ CF missing-value attribute.
+ zarr_format : int
+ 2 or 3. The one place the version changes what is decoded.
+ variable_name : str
+ Named in any warning.
+
+ Returns
+ -------
+ dict
+ The keyword arguments :func:`mask_and_scale` takes, always with the
+ same keys. :func:`decoding_required` says whether they do anything.
+
+ Warns
+ -----
+ IrisCfLoadWarning
+ When a version 2 variable declares no CF missing-value attribute, and
+ the storage fill value about to be used as a sentinel is zero. Zero
+ is zarr-python's own default for a storage fill value a writer never
+ set, so it is the case design document section 4.4 calls "the
+ ordinary case, not the exotic one" when it rules the field out as a
+ mask source on version 3 - a plausible real value, not a sentinel a
+ writer chose. A non-zero storage fill value - the classic netCDF
+ sentinel ``9.96921e+36``, say - is the case section 4.4 keeps this
+ rule for, and does not warn: a writer old enough to predate the CF
+ convention chose it on purpose, which is the rule's own premise.
+
+ """
+ # The two attributes are read differently because CF section 2.5.1
+ # defines them differently: `_FillValue` is a scalar, `missing_value` is
+ # "a scalar or vector".
+ sentinels = []
+ if "_FillValue" in attributes:
+ decoded = _decode_fill_value(attributes["_FillValue"], dtype, variable_name)
+ if decoded is not None:
+ sentinels.append(decoded)
+ if "missing_value" in attributes:
+ sentinels.extend(
+ _decode_missing_values(attributes["missing_value"], dtype, variable_name)
+ )
+
+ if (
+ not sentinels
+ and zarr_format < _FILL_VALUE_IS_A_MASK_BEFORE
+ and storage_fill_value is not None
+ ):
+ # Warn only when the sentinel about to be adopted is zero - the
+ # value this field defaults to when a writer never set it (design
+ # document section 4.4). A non-zero storage fill value is a writer's
+ # deliberate choice, old enough to predate `_FillValue`, which is
+ # exactly the case this rule exists for; masking on it is
+ # unremarkable and does not warn. Masking itself is unchanged either
+ # way - only whether this warns.
+ if storage_fill_value == 0:
+ message = (
+ f"Variable {variable_name} declares no `_FillValue` or "
+ f"`missing_value` on a version 2 store, so its storage fill "
+ f"value {storage_fill_value!r} will be read as missing: values "
+ "equal to it are masked. Set a `_FillValue` attribute to "
+ "control this."
+ )
+ warnings.warn(message, category=iris.warnings.IrisCfLoadWarning)
+ sentinels.append(storage_fill_value)
+
+ valid_min = attributes.get("valid_min")
+ valid_max = attributes.get("valid_max")
+ valid_range = attributes.get("valid_range")
+ if valid_range is not None and len(valid_range) == 2:
+ valid_min, valid_max = valid_range
+
+ scale_factor = attributes.get("scale_factor")
+ add_offset = attributes.get("add_offset")
+
+ return {
+ "sentinels": tuple(sentinels),
+ "valid_min": valid_min,
+ "valid_max": valid_max,
+ "scale_factor": scale_factor,
+ "add_offset": add_offset,
+ "dtype": _unpacked_dtype(dtype, scale_factor, add_offset),
+ }
+
+
+def _unpacked_dtype(dtype, scale_factor, add_offset) -> np.dtype:
+ """Return the dtype the unpacked values will have.
+
+ The same calculation as
+ :func:`iris.fileformats.cf.loader._get_actual_dtype`, done here on plain
+ values so that the dtype the Dask graph declares and the dtype the decode
+ layer produces are one statement rather than two.
+ """
+ probe = np.zeros(1, dtype=dtype)
+ if scale_factor is not None:
+ probe = scale_factor * probe
+ if add_offset is not None:
+ probe = add_offset + probe
+ return probe.dtype
+
+
+def decoding_required(parameters: dict) -> bool:
+ """Whether these parameters would change anything.
+
+ A variable with nothing to decode gets no ``map_blocks`` layer and an
+ unmasked Dask meta. Adding a masked meta without a layer beneath it would
+ be a meta that lies about its own graph.
+ """
+ if parameters["sentinels"]:
+ return True
+ return any(
+ parameters[name] is not None
+ for name in ("valid_min", "valid_max", "scale_factor", "add_offset")
+ )
+
+
+def _mask_by_sentinels(block: np.ndarray, sentinels: tuple) -> np.ndarray:
+ """Return a boolean mask of the elements equal to any sentinel.
+
+ A NaN sentinel is matched with :func:`numpy.isnan`, because NaN does not
+ equal itself and NaN is what nearly every version 3 store's base64
+ ``_FillValue`` decodes to.
+ """
+ mask = np.zeros(block.shape, dtype=bool)
+ for sentinel in sentinels:
+ if isinstance(sentinel, (float, np.floating)) and np.isnan(sentinel):
+ mask |= np.isnan(block)
+ else:
+ mask |= block == sentinel
+ return mask
+
+
+def _mask_by_valid_range(block: np.ndarray, valid_min, valid_max) -> np.ndarray:
+ """Return a boolean mask of the elements outside the valid range."""
+ mask = np.zeros(block.shape, dtype=bool)
+ if valid_min is not None:
+ mask |= block < valid_min
+ if valid_max is not None:
+ mask |= block > valid_max
+ return mask
+
+
+def _unpack(block: np.ndarray, scale_factor, add_offset, dtype: np.dtype) -> np.ndarray:
+ """Return `block` unpacked by CF's scale and offset, as `dtype`."""
+ result = block
+ if scale_factor is not None:
+ result = scale_factor * result
+ if add_offset is not None:
+ result = add_offset + result
+ return result.astype(dtype, copy=False)
+
+
+def mask_and_scale(
+ block: np.ndarray,
+ *,
+ sentinels: tuple,
+ valid_min,
+ valid_max,
+ scale_factor,
+ add_offset,
+ dtype: np.dtype,
+) -> np.ma.MaskedArray:
+ """Apply the CF decode rules to one block, in CF's order.
+
+ Masking first, on the **packed** values, because CF declares a sentinel to
+ be "of the same type as its variable"; unpacking second. The result is
+ always a masked array, so that every block matches the Dask meta the
+ loader declares for the layer.
+
+ Parameters
+ ----------
+ block : numpy.ndarray
+ One Dask block of the raw stored values.
+ sentinels : tuple
+ Values meaning "missing", already narrowed to the stored dtype.
+ valid_min, valid_max : optional
+ CF's valid range. Values outside it are masked.
+ scale_factor, add_offset : optional
+ CF's packing. ``value * scale_factor + add_offset``.
+ dtype : numpy.dtype
+ The dtype of the result, which the Dask graph has already declared.
+
+ Returns
+ -------
+ numpy.ma.MaskedArray
+
+ """
+ mask = _mask_by_sentinels(block, sentinels)
+ mask |= _mask_by_valid_range(block, valid_min, valid_max)
+ return np.ma.masked_array(
+ _unpack(block, scale_factor, add_offset, dtype), mask=mask
+ )
diff --git a/lib/iris/fileformats/zarr/loader.py b/lib/iris/fileformats/zarr/loader.py
new file mode 100644
index 0000000000..e6178e96dc
--- /dev/null
+++ b/lib/iris/fileformats/zarr/loader.py
@@ -0,0 +1,158 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Load Iris cubes from Zarr stores.
+
+.. z_reference:: iris.fileformats.zarr.loader
+ :tags: topic_load_save
+
+ API reference
+
+A thin loop: open the store as a
+:class:`~iris.fileformats.zarr._dataset.ZarrDataset`, hand it to
+:class:`~iris.fileformats.cf.CFReader`, and build one cube per CF data
+variable with :mod:`iris.fileformats.cf.loader` - the same machinery the
+netCDF loader runs. Everything Zarr-specific happens below this module, in
+:mod:`iris.fileformats.zarr._dataset`.
+
+Iris loads **one** group per call. A Zarr store can legitimately hold
+unrelated datasets, and merging them into one ``CubeList`` would be wrong, so
+nested groups are not walked and ``group=`` selects which one to read.
+
+UGRID meshes load from a Zarr store exactly as they do from a netCDF file.
+Nothing in the mesh machinery is netCDF-specific - it reads a
+:class:`~iris.fileformats.cf.CFReader`, and a `CFReader` over a `ZarrDataset`
+answers the same questions - so this module calls into
+:mod:`iris.fileformats.netcdf.ugrid_load` rather than growing a second copy.
+That module's name is the historical home of the code, not a statement about
+what it can read.
+
+See sections 4.6 and 4.7 of
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``.
+
+"""
+
+from collections.abc import Iterable
+import warnings
+
+from iris.exceptions import TranslationError
+from iris.fileformats.cf import CFReader
+from iris.fileformats.cf.loader import (
+ _actions_engine,
+ _load_aux_factory,
+ _load_cube,
+ _translate_constraints_to_var_callback,
+)
+import iris.warnings
+
+from ._dataset import ZarrDataset
+
+
+def _check_has_data_variables(reader, dataset) -> None:
+ """Raise if this group holds nothing to make a cube from.
+
+ Returning an empty ``CubeList`` from a store that plainly holds data
+ elsewhere leaves the user nothing to act on, and a store whose root group
+ holds only subgroups is a real shape - the ESA EOPF Sentinel products are
+ built that way.
+ """
+ if reader.cf_group.data_variables or reader.cf_group.promoted:
+ return
+
+ subgroups = dataset.subgroup_names
+ if subgroups:
+ message = (
+ f"Zarr group {dataset.location!r} has no data variables. Load "
+ "one of its groups instead, with the 'group' keyword: "
+ f"{', '.join(sorted(subgroups))}"
+ )
+ else:
+ message = f"Zarr store {dataset.location!r} has no data variables."
+ raise TranslationError(message)
+
+
+def load_cubes(file_sources, callback=None, constraints=None, group=None):
+ """Load cubes from one or more Zarr stores.
+
+ Parameters
+ ----------
+ file_sources : str or iterable of str
+ The stores to load. A local directory path, or a URL that
+ zarr-python can open - ``s3://``, ``gs://``, ``az://`` and ``abfs://``
+ need the matching store library installed, which Iris does not
+ depend on.
+ callback : callable, optional
+ Passed to :func:`iris.io.run_callback`.
+ constraints : optional
+ Used to skip data variables that cannot match, before they are built.
+ group : str, optional
+ A ``/``-separated path to a group within each store. The root group by
+ default.
+
+ Yields
+ ------
+ iris.cube.Cube
+
+ Raises
+ ------
+ iris.exceptions.TranslationError
+ If the selected group holds no data variables. The message names the
+ store's other groups, when it has any.
+
+ Notes
+ -----
+ This function maintains laziness; it does not realise data.
+
+ """
+ from iris.fileformats.netcdf.ugrid_load import (
+ _attach_mesh_coords,
+ _meshes_from_cf,
+ _prepare_mesh_coords,
+ )
+ from iris.io import run_callback
+
+ var_callback = _translate_constraints_to_var_callback(constraints)
+ engine = _actions_engine()
+
+ if isinstance(file_sources, str) or not isinstance(file_sources, Iterable):
+ file_sources = [file_sources]
+
+ for file_source in file_sources:
+ with ZarrDataset(file_source, group=group) as dataset:
+ with CFReader(dataset) as cf:
+ _check_has_data_variables(cf, dataset)
+
+ meshes = _meshes_from_cf(cf)
+
+ data_variables = list(cf.cf_group.data_variables.values()) + list(
+ cf.cf_group.promoted.values()
+ )
+ for cf_var in data_variables:
+ if var_callback and not var_callback(cf_var):
+ continue
+
+ # MeshCoords are built before the cube and attached after
+ # it: see _prepare_mesh_coords for why the two cannot be
+ # one step.
+ pending_mesh_coords = _prepare_mesh_coords(cf, meshes, cf_var)
+
+ cube = _load_cube(engine, cf, cf_var, cf.filename)
+
+ if pending_mesh_coords is not None:
+ _attach_mesh_coords(pending_mesh_coords, cube)
+
+ try:
+ _load_aux_factory(engine, cube)
+ except ValueError as exception:
+ warnings.warn(
+ str(exception),
+ category=iris.warnings.IrisLoadWarning,
+ )
+
+ cube = run_callback(callback, cube, cf_var, file_source)
+ if cube is None:
+ # The callback rejected it.
+ continue
+
+ yield cube
diff --git a/lib/iris/io/__init__.py b/lib/iris/io/__init__.py
index ddeb6a7157..9889aaf31b 100644
--- a/lib/iris/io/__init__.py
+++ b/lib/iris/io/__init__.py
@@ -38,9 +38,38 @@ def _is_nczarr_fragment(fragment):
return False
+# URI schemes Iris hands to a store library rather than resolving as a local
+# path. Iris depends on none of them: zarr-python raises its own ImportError
+# naming what to install.
+_REMOTE_STORE_SCHEMES = frozenset({"s3", "gs", "az", "abfs"})
+
+
+def _is_remote_store_uri(spec):
+ """Whether `spec` names a store Iris hands to a store library untouched.
+
+ Such a URI must not be resolved as a local path: `.absolute()` turns
+ ``s3://bucket/data.zarr`` into ``/<cwd>/s3:/bucket/data.zarr``.
+ """
+ if not isinstance(spec, str):
+ return False
+ scheme, _part, _fragment = decode_uri(spec)
+ return scheme in _REMOTE_STORE_SCHEMES
+
+
# Saving routines, indexed by file extension.
class _SaversDict(dict):
- """A dictionary that can only have string keys with no overlap."""
+ """A dictionary that can only have string keys with no overlap.
+
+ :attr:`cubelist_savers` records which of the registered savers take a
+ ``CubeList`` directly, rather than one cube at a time with
+ ``append=True``. It holds the saver functions themselves, not extensions,
+ because :func:`save` is also reached with a saver it never looked up here
+ - passed by the caller, or chosen directly for an NCZarr target.
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.cubelist_savers = set()
def __setitem__(self, key, value):
if not isinstance(key, str):
@@ -170,6 +199,29 @@ def decode_uri(uri, default="file"):
return scheme, part, fragment
+def _categorise_uri(uri) -> tuple[str, str, str]:
+ """Return `uri` as ``(category, scheme, part)``, for grouping by handler.
+
+ The category is the scheme, except that an NCZarr fragment makes it
+ ``"nczarr"``. netCDF-c's Zarr driver is asked for by ``#mode=nczarr,...``
+ rather than by scheme, so ``file:///s.zarr`` and
+ ``file:///s.zarr#mode=nczarr,file`` need different handlers despite
+ sharing one scheme. The fragment is folded back into `part`, because a
+ handler is handed a URI to reopen rather than the decoded pieces.
+
+ A native Zarr store has no category of its own. Remotely it is known by
+ its scheme, and locally it is a directory that only a marker file
+ distinguishes from any other - see
+ :func:`iris.fileformats._is_zarr_store`. Deciding it here would put a
+ filesystem test inside a function that otherwise only reads a string.
+ """
+ scheme, part, fragment = decode_uri(uri)
+ category = "nczarr" if _is_nczarr_fragment(fragment) else scheme
+ if fragment:
+ part = f"{part}#{fragment}"
+ return category, scheme, part
+
+
def expand_filespecs(file_specs, files_expected=True):
"""Find all matching file paths from a list of file-specs.
@@ -190,15 +242,24 @@ def expand_filespecs(file_specs, files_expected=True):
If files_expected is ``False``:
A list of expanded file paths.
"""
- # Remove any hostname component - currently unused
+ # Remove any hostname component - currently unused. A remote store URI is
+ # passed through: it is not a path, and resolving it mangles it.
filenames = [
- str(pathlib.Path(fn.removeprefix("//")).expanduser().absolute())
+ fn
+ if _is_remote_store_uri(fn)
+ else str(pathlib.Path(fn.removeprefix("//")).expanduser().absolute())
for fn in file_specs
]
if files_expected:
- # Try to expand all filenames as globs
- glob_expanded = OrderedDict([[fn, sorted(glob.glob(fn))] for fn in filenames])
+ # Try to expand all filenames as globs. A remote URI stands for
+ # itself; Iris does not list a remote store's neighbours.
+ glob_expanded = OrderedDict(
+ [
+ [fn, [fn] if _is_remote_store_uri(fn) else sorted(glob.glob(fn))]
+ for fn in filenames
+ ]
+ )
# If any of the specs expanded to an empty list then raise an error
all_expanded = glob_expanded.values()
@@ -231,13 +292,27 @@ def load_files(filenames, callback, constraints=None):
intended interface for loading is :func:`iris.load`.
"""
- from iris.fileformats import FORMAT_AGENT
+ yield from _load_files(expand_filespecs(filenames), callback, constraints)
+
- all_file_paths = expand_filespecs(filenames)
+def _load_files(filenames, callback, constraints=None):
+ """Create a generator of Cubes from files already expanded by `expand_filespecs`.
+
+ `filenames` is not expanded here: a glob matching the same file twice, by
+ two different patterns, must reach the format handler as two entries, and
+ :func:`expand_filespecs` keys its own glob cache by the spec string, so
+ expanding an already-expanded list collapses two identical paths into
+ one. :func:`load_files` is the public entry point that expands once and
+ delegates here; :func:`iris.loading._generate_cubes` is the other caller,
+ expanding once for itself so it can partition files from stores before
+ either sees the list.
+
+ """
+ from iris.fileformats import FORMAT_AGENT
# Create default dict mapping iris format handler to its associated filenames
handler_map = collections.defaultdict(list)
- for fn in all_file_paths:
+ for fn in filenames:
with open(fn, "rb") as fh:
handling_format_spec = FORMAT_AGENT.get_spec(pathlib.Path(fn).name, fh)
handler_map[handling_format_spec].append(fn)
@@ -253,6 +328,49 @@ def load_files(filenames, callback, constraints=None):
yield cube
+def load_stores(targets, callback, constraints=None):
+ """Create a generator of Cubes from targets that are stores, not files.
+
+ A store is a directory or a URL: something with no single seekable file
+ to open, so :func:`load_files` cannot reach it and the format agent is
+ asked without a buffer.
+
+ Parameters
+ ----------
+ targets : iterable of str
+ Local directory paths or remote URLs, already expanded.
+ callback : callable
+ Passed to the format handler.
+ constraints : optional
+ Passed to a constraint-aware format handler.
+
+ Yields
+ ------
+ iris.cube.Cube
+
+ Notes
+ -----
+ Typically, this function should not be called directly; instead, the
+ intended interface for loading is :func:`iris.load`.
+
+ """
+ from iris.fileformats import FORMAT_AGENT
+
+ handler_map = collections.defaultdict(list)
+ for target in targets:
+ handling_format_spec = FORMAT_AGENT.get_spec(target, None)
+ handler_map[handling_format_spec].append(target)
+
+ for handling_format_spec in sorted(handler_map):
+ names = handler_map[handling_format_spec]
+ if handling_format_spec.constraint_aware_handler:
+ for cube in handling_format_spec.handler(names, callback, constraints):
+ yield cube
+ else:
+ for cube in handling_format_spec.handler(names, callback):
+ yield cube
+
+
def load_http(urls, callback):
"""Create generator of Cubes from the given OPeNDAP or NcZarr URLs.
@@ -350,6 +468,8 @@ def _check_init_savers():
"grib2": _grib_save,
}
)
+ # One line per format that takes a CubeList whole.
+ _savers.cubelist_savers.add(netcdf.save)
def add_saver(file_extension, new_saver):
@@ -392,10 +512,10 @@ def find_saver(filespec):
"""
_check_init_savers()
+ # A store is a directory, so "out.zarr/" is a natural thing to type.
+ candidate = filespec.rstrip("/\\").lower()
matches = [
- ext
- for ext in _savers
- if filespec.lower().endswith("." + ext) or filespec.lower() == ext
+ ext for ext in _savers if candidate.endswith("." + ext) or candidate == ext
]
# Multiple matches could occur if one of the savers included a '.':
# e.g. _savers = {'.dot.png': dot_png_saver, '.png': png_saver}
@@ -527,7 +647,8 @@ def save(source, target, saver=None, **kwargs):
isinstance(source, (list, tuple)) and all([isinstance(i, Cube) for i in source])
):
# Only allow cubelist saving for those fileformats that are capable.
- if "iris.fileformats.netcdf" not in saver.__module__:
+ _check_init_savers()
+ if saver not in _savers.cubelist_savers:
# Make sure the saver accepts an append keyword
if "append" not in saver.__code__.co_varnames:
raise ValueError(
diff --git a/lib/iris/io/format_picker.py b/lib/iris/io/format_picker.py
index ac80e03611..dc04fd549e 100644
--- a/lib/iris/io/format_picker.py
+++ b/lib/iris/io/format_picker.py
@@ -377,6 +377,33 @@ def get_element(self, basename, file_handle):
return decode_uri(basename)
+class ZarrStore(FileElement):
+ """A :class:`FileElement` that returns a target that may be a Zarr store.
+
+ Carries the whole target rather than its basename, because recognising a
+ local store means reading a marker file inside it, and recognising a
+ remote one means reading its scheme.
+
+ """
+
+ def __init__(self):
+ FileElement.__init__(self, requires_fh=False)
+
+ def get_element(self, basename, file_handle):
+ """Return the target, which here is the whole path or URL.
+
+ ``None`` when a file handle is supplied, which no store ever has: a
+ store is a directory or a URL, so the caller that opened a single
+ seekable file has already settled the question. That caller also
+ passes a basename rather than a path, which resolved against the
+ working directory could name an unrelated store that happens to share
+ the file's name.
+ """
+ if file_handle is not None:
+ return None
+ return basename
+
+
class DataSourceObjectProtocol(FileElement):
"""A :class:`FileElement` that simply returns the URI entry itself.
diff --git a/lib/iris/loading.py b/lib/iris/loading.py
index 89e2e43059..e4bb5e0b09 100644
--- a/lib/iris/loading.py
+++ b/lib/iris/loading.py
@@ -13,6 +13,7 @@
from contextlib import contextmanager
from dataclasses import dataclass
import itertools
+from pathlib import Path
import threading
from traceback import TracebackException
from typing import Any, Iterable, Type
@@ -83,26 +84,38 @@ def _generate_cubes(uris, callback, constraints):
# Make a string, Dataset, or other single item, into an iterable.
uris = [uris]
- def _categorise(decoded: tuple[str, str, str | None]) -> tuple[str, str, str]:
- scheme, part, fragment = decoded
- category = "nczarr" if iris.io._is_nczarr_fragment(fragment) else scheme
- if fragment:
- part = f"{part}#{fragment}"
- return category, scheme, part
-
# Group collections of uris by their iris handler
# Create list of tuples relating schemes to part names
- uri_tuples = sorted(_categorise(iris.io.decode_uri(uri)) for uri in uris)
+ uri_tuples = sorted(iris.io._categorise_uri(uri) for uri in uris)
for category, groups in itertools.groupby(uri_tuples, key=lambda x: x[0]):
# Call each scheme handler with the appropriate URIs
if category == "file":
part_names = [part for _cat, _scheme, part in groups]
- for cube in iris.io.load_files(part_names, callback, constraints):
- yield cube
+ # The file-or-store decision has to happen after glob expansion
+ # and before format dispatch, because a glob may match both.
+ # Neither routine below learns about the other's kind of target.
+ # iris.fileformats.netcdf.ugrid_load.load_meshes makes this same
+ # distinction independently, for mesh loading - keep both in step.
+ expanded = iris.io.expand_filespecs(part_names)
+ files = [name for name in expanded if not Path(name).is_dir()]
+ stores = [name for name in expanded if Path(name).is_dir()]
+ if files:
+ # `expanded` is already the result of `expand_filespecs`, so
+ # the private, non-expanding `_load_files` is used here - see
+ # its docstring for why expanding twice would be wrong.
+ for cube in iris.io._load_files(files, callback, constraints):
+ yield cube
+ if stores:
+ for cube in iris.io.load_stores(stores, callback, constraints):
+ yield cube
elif category in ["http", "https", "nczarr"]:
urls = [f"{scheme}:{part}" for _cat, scheme, part in groups]
for cube in iris.io.load_http(urls, callback):
yield cube
+ elif category in iris.io._REMOTE_STORE_SCHEMES:
+ urls = [f"{scheme}:{part}" for _cat, scheme, part in groups]
+ for cube in iris.io.load_stores(urls, callback, constraints):
+ yield cube
elif category == "data":
data_objects = [part for _cat, _scheme, part in groups]
for cube in iris.io.load_data_objects(data_objects, callback):
diff --git a/lib/iris/tests/AGENTS.md b/lib/iris/tests/AGENTS.md
index 36d18c6b04..f776960912 100644
--- a/lib/iris/tests/AGENTS.md
+++ b/lib/iris/tests/AGENTS.md
@@ -6,8 +6,8 @@ These rules apply to all test files under this directory tree.
## Purpose
-This test suite validates Iris behaviour, metadata handling, and regressionc
-overage. Keep changes focused, deterministic, and compatible with the
+This test suite validates Iris behaviour, metadata handling, and regression
+coverage. Keep changes focused, deterministic, and compatible with the
existing test style.
@@ -38,21 +38,74 @@ existing test style.
- Avoid over-mocking; prefer realistic Cube/Coord setup via shared helpers.
-## Running Tests
+## Docstrings in Tests
-Run targeted tests while iterating:
+A test docstring is notes, not published documentation. numpydoc does not run
+here and `D205`/`D401` are relaxed, so drop the ceremony — no imperative mood,
+no summary-line ritual; fragments and shorthand are correct.
-```bash
-pytest -n auto lib/iris/tests/unit/<path_or_file>.py
-```
+When a test pins a trap that library code points at (see
+[`../AGENTS.md`](../AGENTS.md)), its docstring carries the detail: what is
+surprising, the smallest concrete evidence, the cause. Not the investigation.
+Under ~6 lines.
-Run broader coverage before finishing:
+```python
+def test_write_handle_encodes_char_data():
+ """Unencoded proxy writes unicode at an NC_CHAR variable.
-```bash
-pytest -n auto lib/iris/tests
+ Deferred save of ["abc", "def"] returns ["aaa", "ddd"].
+ from_existing only wraps a dataset lacking THREAD_SAFE_FLAG.
+ """
```
+## Running Tests
+
+`-n auto` is **not** in `addopts`. Pass it yourself or the run is serial:
+roughly half speed on the full suite, and no worse than break-even on
+anything smaller (see below).
+
+Work in tiers. The full suite is minutes; the area you are changing is
+usually seconds, and is the right command for nearly all of an iteration.
+
+| Tier | When | Command |
+|---|---|---|
+| The module you changed | every edit | `pytest lib/iris/tests/unit/<area>/` |
+| Its immediate neighbours | before committing | `pytest -n auto lib/iris/tests/unit/<parent>/ lib/iris/tests/integration/<area>/` |
+| Everything | before finishing, and to compare against a base branch | `pytest -n auto lib/iris/tests` |
+
+Indicative on a 4-core machine: ~6s, ~1min, ~4min respectively. Running the
+full suite on every edit is the single biggest avoidable delay in this
+repository.
+
+- **Do not add `-n auto` to small runs.** Spawning workers costs a few
+ seconds, so it makes a sub-10-second selection slower, not faster.
+- **Do not raise `-n` above the core count.** Measured on the full suite,
+ `-n 8` on 4 cores bought 7% wall time for 60% more CPU.
+- Prefer `-o cache_dir=<scratch>/pytest_cache` over `-p no:cacheprovider`
+ when concurrent runs must not collide. Disabling the cache plugin also
+ disables `--lf` and `--ff`, which are the cheapest speed-ups available
+ while iterating on failures.
+
+
+### Doctests
+
+The pytest suite does not run them — nothing here covers the `>>>` examples
+in library docstrings (29 modules) or the user guide (30 files). Only Sphinx
+does:
+
+| Command | Cost |
+|---|---|
+| `nox -s doctest` | what CI runs: `make clean html`, then `make doctest` |
+| `cd docs/src && make doctest` | skips the gallery rebuild; faster, not identical to CI |
+
+Both take minutes, so run one once before finishing, and only if you touched
+a `>>>` example. Do not substitute `pytest --doctest-modules` or
+`python -m doctest`: 150 `testsetup::` / `testcode::` directives supply
+context only Sphinx applies, so plain doctest reports failures that are not
+real.
+
+
## Style and Lint in Tests
- Follow Ruff formatting and linting used by the repository.
@@ -64,7 +117,10 @@ pytest -n auto lib/iris/tests
- Some tests require external `iris-test-data`; Preferred env
var: `OVERRIDE_TEST_DATA_REPOSITORY=/path/to/iris-test-data/test_data`
-- Missing data will skip affected tests by design.
+- **Set it before trusting any run**, and check it took effect. Missing data
+ skips affected tests by design — silently, and without failing — so an
+ unset variable quietly removes coverage from whatever you are changing. Ask
+ for the path if you cannot find it; do not proceed and caveat the result.
- Keep new tests independent of local machine paths and optional system state.
- For optional dependencies, skip gracefully using existing patterns.
@@ -75,7 +131,16 @@ pytest -n auto lib/iris/tests
- Keep assertions specific and deterministic; avoid timing-sensitive checks.
- Avoid broad snapshot-style updates without explaining intent in the change.
- Do not add network access in tests.
-- Do not weaken existing checks just to make tests pass.
+- Never name an artefact of how the work was done — a review finding, a task
+ or plan number, a phase — in a test name, docstring or comment. Cite an
+ issue, a pull request, a design document, or the convention's section.
+- Never cite a line number in another file, a count of call sites, or a time
+ relative to writing ("this PR", "before this branch"). All decay silently,
+ and nothing checks them. Name the function, class, constant or pull request
+ number instead — a test outlives the branch that added it.
+- Do not weaken existing checks just to make tests pass — but a test can pin
+ a *bug*, and correcting one is strengthening. Show by archaeology that the
+ expectation was wrong, invert rather than delete, and comment why.
## Graphics and Integration Notes
@@ -90,6 +155,21 @@ pytest -n auto lib/iris/tests
- The changed tests pass locally.
- No unrelated tests were modified.
- Any required external data setup is documented in the change notes.
+- **Read the skip count, not just the failures.** Comparing failures before
+ and after a change — the standard way to show a refactor broke nothing —
+ cannot detect a test that never ran. A skip is neither a pass nor a
+ failure, so a `FAILED|ERROR` diff filters it out and the comparison comes
+ back clean whether coverage is intact or absent. Treat any skip in the area
+ you changed as a gap to close, usually missing `iris-test-data` or an
+ optional dependency, rather than as background noise.
+- **Never compare warning totals.** Under `-n auto` they are not
+ reproducible — import-time deprecations are counted once per worker that
+ imports the module, so the total tracks work distribution. One unchanging
+ tree gave 6483, 6485, 6484, 6483, 6483. A warning delta is not evidence.
+- **Compare like with like.** Collection order affects error counts:
+ `integration/netcdf/test_coord_systems.py` reports ten errors in a tier
+ run and none in the full suite. A tier baseline says nothing about a
+ full-suite one.
## Scope Boundaries
@@ -101,15 +181,14 @@ pytest -n auto lib/iris/tests
when requested.
-## ⚠️ Meta-Instruction: Auto-Update Rule
-- **Trigger**: You MUST automatically update this `AGENTS.md` file if a new
- structural pattern, critical bug fix, or persistent repository rule is
- established during this session.
-- **Constraint 1**: Keep the file under 200 lines. Delete obsolete rules to
- make room for new ones.
-- **Constraint 2**: Do not ask for user permission before updating. Perform the
- edit silently as your final step.
-- **Constraint 3**: Only log global, reusable lessons. Do not log temporary or
+## ⚠️ Meta-Instruction: Changing This File
+- **Trigger**: If your work establishes a durable, reusable rule, you MUST
+ propose it before your session ends.
+- **Constraint 1**: Propose, never self-apply. Say it in your closing message,
+ or raise it as its own pull request. NEVER edit an `AGENTS.md` silently, or
+ as a side effect of unrelated work.
+- **Constraint 2**: Keep this file under 300 lines — every agent loads it in
+ full. If an addition would break that, tighten your wording; do NOT delete
+ existing guidance to make room. Removing a rule is its own proposal.
+- **Constraint 3**: Only global, reusable lessons. Do not propose temporary or
component-specific fixes.
-
-
diff --git a/lib/iris/tests/integration/cf/__init__.py b/lib/iris/tests/integration/cf/__init__.py
new file mode 100644
index 0000000000..46413e6ea7
--- /dev/null
+++ b/lib/iris/tests/integration/cf/__init__.py
@@ -0,0 +1,5 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Integration tests for the :mod:`iris.fileformats.cf` package."""
diff --git a/lib/iris/tests/integration/netcdf/test__dask_locks.py b/lib/iris/tests/integration/cf/test__dask_locks.py
similarity index 69%
rename from lib/iris/tests/integration/netcdf/test__dask_locks.py
rename to lib/iris/tests/integration/cf/test__dask_locks.py
index 1aee902195..d26cb31f7f 100644
--- a/lib/iris/tests/integration/netcdf/test__dask_locks.py
+++ b/lib/iris/tests/integration/cf/test__dask_locks.py
@@ -2,7 +2,7 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Unit tests for the :mod:`iris.fileformats.netcdf._dask_locks` package.
+"""Unit tests for the :mod:`iris.fileformats.cf._dask_locks` package.
Note: these integration tests replace any unit testing of this module, due to its total
dependence on Dask, and even on Dask's implementation details rather than supported
@@ -15,9 +15,12 @@
import dask
import dask.config
import distributed
+import numpy as np
import pytest
-from iris.fileformats.netcdf._dask_locks import (
+import iris
+from iris.cube import Cube
+from iris.fileformats.cf._dask_locks import (
DaskSchedulerTypeError,
dask_scheduler_is_distributed,
get_dask_array_scheduler_type,
@@ -109,3 +112,31 @@ def test_get_worker_lock(dask_scheduler):
else:
# low-level object doesn't have a readily available class for isinstance
assert all(hasattr(result, att) for att in ("acquire", "release", "locked"))
+
+
+def test_load_does_not_need_a_worker_lock(tmp_path):
+ """A load must not demand a lock only the saver needs.
+
+ ``get_worker_lock`` refuses the process scheduler, because the *saver*
+ cannot use it - see this module's own ``test_get_worker_lock``. Loading
+ has no such limitation, so no read path may ask for the lock: doing so
+ made ``iris.load`` fail under ``scheduler="processes"``, reporting that
+ the scheduler "is not supported by the Iris netcdf saver" during a load.
+
+ The process scheduler is really configured rather than simulated, because
+ that is the configuration a user hits - and configuring it is the whole
+ trigger: the failure was at lock construction during the load, before any
+ compute. No worker process is in fact spawned, because this file is below
+ ``cf.loader._LAZYVAR_MIN_BYTES`` and so loads eagerly, which is what keeps
+ the test quick and deterministic under pytest.
+ """
+ path = tmp_path / "process_scheduler.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(Cube(np.arange(6.0).reshape(2, 3), long_name="x"), path)
+
+ with dask.config.set(scheduler="processes"):
+ loaded = iris.load(path)
+ data = loaded[0].data
+
+ assert len(loaded) == 1
+ np.testing.assert_array_equal(data, np.arange(6.0).reshape(2, 3))
diff --git a/lib/iris/tests/integration/netcdf/derived_bounds/test_bounds_files.py b/lib/iris/tests/integration/netcdf/derived_bounds/test_bounds_files.py
index c7d3564fd2..2f39608aa0 100644
--- a/lib/iris/tests/integration/netcdf/derived_bounds/test_bounds_files.py
+++ b/lib/iris/tests/integration/netcdf/derived_bounds/test_bounds_files.py
@@ -18,6 +18,10 @@
import iris
from iris import FUTURE, sample_data_path
+from iris.aux_factory import HybridHeightFactory
+from iris.coords import AuxCoord, DimCoord
+from iris.cube import Cube
+from iris.fileformats.netcdf._thread_safe_nc import DatasetWrapper
from iris.tests._shared_utils import assert_CDL
from iris.tests.stock.netcdf import ncgen_from_cdl
@@ -220,3 +224,56 @@ def test_save_primary_cf_style(
assert_CDL(
request=request, netcdf_filename=nc_filepath, reference_filename=cdl_filepath
)
+
+
+@pytest.fixture
+def cube_with_absent_orography():
+ """Return a hybrid-height cube whose factory has no orography.
+
+ :class:`~iris.aux_factory.HybridHeightFactory` documents ``orography`` as
+ optional, so a saved ``formula_terms`` can name a dependency that was
+ never written as a variable.
+ """
+ cube = Cube(np.zeros(3, dtype="f8"), standard_name="air_temperature", units="K")
+ delta = DimCoord(
+ np.arange(3.0),
+ long_name="level_height",
+ var_name="level_height",
+ units="m",
+ bounds=np.array([[0.0, 1.0], [1.0, 2.0], [2.0, 3.0]]),
+ )
+ sigma = AuxCoord(
+ np.linspace(1.0, 0.0, 3),
+ long_name="sigma",
+ var_name="sigma",
+ units="1",
+ bounds=np.array([[1.0, 0.75], [0.75, 0.25], [0.25, 0.0]]),
+ )
+ cube.add_dim_coord(delta, 0)
+ cube.add_aux_coord(sigma, 0)
+ cube.add_aux_factory(HybridHeightFactory(delta=delta, sigma=sigma, orography=None))
+ return cube
+
+
+def test_save_absent_factory_dependency(cube_with_absent_orography, tmp_ncdir):
+ """Save a factory with an absent dependency, with derived bounds enabled.
+
+ The bounds ``formula_terms`` is built by following each term variable's
+ ``bounds`` link, and an absent dependency has no term variable to follow.
+ Asserts on the file rather than on the save completing, because a change
+ that silently dropped ``orog`` would also not raise.
+ """
+ nc_filepath = tmp_ncdir / "test_save_absent_orography.nc"
+
+ with FUTURE.context(derived_bounds=True):
+ iris.save(cube_with_absent_orography, nc_filepath)
+
+ dataset = DatasetWrapper(nc_filepath)
+ try:
+ formula_terms = dataset.variables["level_height"].formula_terms
+ bounds_formula_terms = dataset.variables["level_height_bnds"].formula_terms
+ finally:
+ dataset.close()
+
+ assert formula_terms == "a: level_height b: sigma orog: None"
+ assert bounds_formula_terms == "a: level_height_bnds b: sigma_bnds orog: None"
diff --git a/lib/iris/tests/integration/netcdf/test_attributes.py b/lib/iris/tests/integration/netcdf/test_attributes.py
index b24538079f..2da89f95cf 100644
--- a/lib/iris/tests/integration/netcdf/test_attributes.py
+++ b/lib/iris/tests/integration/netcdf/test_attributes.py
@@ -7,11 +7,16 @@
from contextlib import contextmanager
from cf_units import Unit
+import numpy as np
import pytest
import iris
+import iris.coord_systems
+from iris.coords import DimCoord
from iris.cube import Cube, CubeList
-from iris.fileformats.netcdf import CF_CONVENTIONS_VERSION
+from iris.fileformats.cf.saver import CF_CONVENTIONS_VERSION
+from iris.fileformats.netcdf import _thread_safe_nc as threadsafe_nc
+from iris.fileformats.netcdf._dataset import NetCDFDatasetVariable
from iris.tests import _shared_utils
@@ -101,6 +106,177 @@ def test_patching_conventions_attribute(self, tmp_path, mocker):
)
+class TestCfPatch:
+ """What the ``cf_patch`` hook is handed is public API.
+
+ ``cf_patch`` is a reserved :data:`iris.site_configuration` key, documented
+ since Iris 1.3 as receiving netCDF4 objects. Handing it a
+ :class:`~iris.fileformats.netcdf._dataset.NetCDFDatasetVariable` instead
+ breaks it two ways: ``setncattr`` raises, and a plain
+ ``variable.name = value`` is discarded in silence, since a CF variable
+ defines no ``__setattr__``.
+
+ """
+
+ def test_cf_patch_receives_a_netcdf4_variable(self, tmp_path):
+ received = []
+
+ def cf_profile(cube):
+ return "a profile"
+
+ def cf_patch(profile, dataset, variable):
+ received.append(variable)
+ # Write through the netCDF4 API the hook is promised.
+ variable.setncattr("patched_by_cf_patch", "cf_patch was here")
+
+ # A real function, not a Mock: a Mock accepts setncattr() whatever it
+ # is given, and records a call that never reached a file.
+ orig_site_config = iris.site_configuration.copy()
+ iris.site_configuration["cf_profile"] = cf_profile
+ iris.site_configuration["cf_patch"] = cf_patch
+ nc_path = tmp_path / "cf_patch.nc"
+ try:
+ cube = Cube([1.0], standard_name="air_temperature", units="K")
+ iris.save(cube, nc_path)
+ finally:
+ iris.site_configuration = orig_site_config
+
+ (variable,) = received
+ assert not isinstance(variable, NetCDFDatasetVariable)
+
+ # The read-back is the point. Asserting that the hook ran, or that it
+ # did not raise, would miss an attribute going quietly nowhere.
+ result = iris.load_cube(nc_path)
+ assert result.attributes["patched_by_cf_patch"] == "cf_patch was here"
+
+
+class TestAttributesNamedLikeNetcdf4Members:
+ """A coordinate attribute may be named after a netCDF4 Python member.
+
+ ``shape``, ``size`` and the rest below are members of a netCDF4 Variable
+ object as well as plausible attribute names. The saver's "don't clobber"
+ check used to ask ``hasattr``, so every one of them was silently dropped
+ on the way to the file. It asks the CF attribute mapping now, which knows
+ the difference between an attribute of the data and a member of the
+ object holding it, and the six reach the file like any other.
+
+ """
+
+ NAMES = ("shape", "size", "name", "dtype", "dimensions", "mask")
+
+ def test_attributes_named_after_netcdf4_members_are_saved(self, tmp_path):
+ coord = DimCoord(
+ np.arange(3.0),
+ standard_name="longitude",
+ units="degrees",
+ attributes={name: f"value of {name}" for name in self.NAMES},
+ )
+ cube = Cube(np.arange(3.0), standard_name="air_temperature", units="K")
+ cube.add_dim_coord(coord, 0)
+
+ nc_path = tmp_path / "netcdf4_member_names.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(cube, nc_path)
+
+ # Reading back is the only way to see this: the dropped attributes
+ # were dropped in silence, with no warning and no error.
+ result = iris.load_cube(nc_path).coord("longitude")
+ assert {name: result.attributes.get(name) for name in self.NAMES} == {
+ name: f"value of {name}" for name in self.NAMES
+ }
+
+
+class TestGridMappingAttributes:
+ """Grid-mapping parameters are written by plain attribute assignment.
+
+ Every other attribute the saver writes goes through the CF attribute
+ mapping. These do not - they are set straight onto the netCDF4 variable,
+ because doing otherwise would change their type in the file.
+ That makes a mistake in one of them silent: assigning to the wrong object,
+ or misspelling the object, leaves a stray Python attribute and no
+ attribute in the file at all.
+
+ These two projections are here because they carry the eight parameter
+ assignments no other test reads back.
+
+ """
+
+ @staticmethod
+ def saved_grid_mapping(coord_system, tmp_path):
+ """Save a cube on this coord system; return the grid-mapping attributes."""
+ cube = Cube(
+ np.zeros((2, 3), dtype=np.float32),
+ standard_name="air_temperature",
+ units="K",
+ )
+ for index, axis in enumerate("yx"):
+ cube.add_dim_coord(
+ DimCoord(
+ np.arange(cube.shape[index], dtype=np.float64),
+ standard_name=f"projection_{axis}_coordinate",
+ units="m",
+ coord_system=coord_system,
+ ),
+ index,
+ )
+ nc_path = tmp_path / f"{coord_system.grid_mapping_name}.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(cube, nc_path)
+
+ # Read the file itself rather than a loaded cube: what is being
+ # pinned is that these values reach the variable, not what the
+ # loader is able to reconstruct from them.
+ dataset = threadsafe_nc.DatasetWrapper(nc_path)
+ try:
+ variable = dataset.variables[coord_system.grid_mapping_name]
+ return {name: variable.getncattr(name) for name in variable.ncattrs()}
+ finally:
+ dataset.close()
+
+ def test_mercator_scale_factor(self, tmp_path):
+ # Mercator takes a scale factor *or* a standard parallel. Only the
+ # standard-parallel branch is read back anywhere else.
+ coord_system = iris.coord_systems.Mercator(
+ longitude_of_projection_origin=90.0,
+ scale_factor_at_projection_origin=0.9,
+ )
+ attributes = self.saved_grid_mapping(coord_system, tmp_path)
+ assert attributes["scale_factor_at_projection_origin"] == 0.9
+ assert "standard_parallel" not in attributes
+
+ @pytest.mark.parametrize(
+ ("scale_kwargs", "expected_scale"),
+ [
+ ({"true_scale_lat": 71.0}, {"true_scale_lat": 71.0}),
+ (
+ {"scale_factor_at_projection_origin": 0.9},
+ {"scale_factor_at_projection_origin": 0.9},
+ ),
+ # Neither given: the saver writes a scale factor of 1.0 rather
+ # than leaving the projection unscaled.
+ ({}, {"scale_factor_at_projection_origin": 1.0}),
+ ],
+ ids=["true_scale_lat", "scale_factor", "neither"],
+ )
+ def test_polar_stereographic(self, scale_kwargs, expected_scale, tmp_path):
+ coord_system = iris.coord_systems.PolarStereographic(
+ central_lat=90.0,
+ central_lon=-150.0,
+ false_easting=13.0,
+ false_northing=17.0,
+ **scale_kwargs,
+ )
+ attributes = self.saved_grid_mapping(coord_system, tmp_path)
+ expected = {
+ "latitude_of_projection_origin": 90.0,
+ "straight_vertical_longitude_from_pole": -150.0,
+ "false_easting": 13.0,
+ "false_northing": 17.0,
+ **expected_scale,
+ }
+ assert {name: attributes.get(name) for name in expected} == expected
+
+
class TestStandardName:
def test_standard_name_roundtrip(self, tmp_path):
standard_name = "air_temperature detection_minimum"
diff --git a/lib/iris/tests/integration/netcdf/test_delayed_save.py b/lib/iris/tests/integration/netcdf/test_delayed_save.py
index 6473266e6f..4f348a2ea6 100644
--- a/lib/iris/tests/integration/netcdf/test_delayed_save.py
+++ b/lib/iris/tests/integration/netcdf/test_delayed_save.py
@@ -14,6 +14,7 @@
import pytest
import iris
+import iris.fileformats.cf.loader
from iris.fileformats.netcdf._thread_safe_nc import default_fillvals
from iris.tests import _shared_utils
from iris.tests.stock import realistic_4d
@@ -44,10 +45,10 @@ def output_path(self, tmp_path):
def all_vars_lazy(self):
# For the operation of these tests, we want to force all netcdf variables
# to load as lazy data, i.e. **don't** use real data for 'small' ones.
- old_value = iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES
- iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES = 0
+ old_value = iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES
+ iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES = 0
yield
- iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES = old_value
+ iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES = old_value
@staticmethod
@pytest.fixture(params=[False, True], ids=["SaveImmediate", "SaveDelayed"])
diff --git a/lib/iris/tests/integration/netcdf/test_general.py b/lib/iris/tests/integration/netcdf/test_general.py
index ed125bcb13..a35ec0d4c2 100644
--- a/lib/iris/tests/integration/netcdf/test_general.py
+++ b/lib/iris/tests/integration/netcdf/test_general.py
@@ -39,7 +39,7 @@ def test_lazy_preserved_save(self, mocker, tmp_path):
("NetCDF", "label_and_climate", "small_FC_167_mon_19601101.nc")
)
# While loading, "turn off" loading small variables as real data.
- mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 0)
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
acube = iris.load_cube(fpath, "air_temperature")
assert acube.has_lazy_data()
# Also check a coord with lazy points + bounds.
diff --git a/lib/iris/tests/integration/netcdf/test_stringdata.py b/lib/iris/tests/integration/netcdf/test_stringdata.py
index 80aabcc792..06b418fc80 100644
--- a/lib/iris/tests/integration/netcdf/test_stringdata.py
+++ b/lib/iris/tests/integration/netcdf/test_stringdata.py
@@ -20,6 +20,7 @@
from iris.coords import AuxCoord, DimCoord
from iris.cube import Cube
import iris.exceptions
+import iris.fileformats.cf.loader
from iris.fileformats.netcdf import (
DECODE_TO_STRINGS_ON_READ,
SUPPORTED_ENCODINGS,
@@ -31,10 +32,10 @@
@pytest.fixture(scope="module")
def all_lazy_auxcoords():
"""Ensure that *all* aux-coords are loaded lazily, even really small ones."""
- old_minlazybytes = iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES
- iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES = 0
+ old_minlazybytes = iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES
+ iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES = 0
yield
- iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES = old_minlazybytes
+ iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES = old_minlazybytes
N_XDIM = 3
@@ -404,7 +405,7 @@ class TestWriteEncodings:
def lazy_data(self, request, mocker):
is_lazy = request.param == "allLazy"
if is_lazy:
- mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 0)
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
return is_lazy
@pytest.fixture(params=["dataAsStrings", "dataAsBytes"])
diff --git a/lib/iris/tests/test_cf.py b/lib/iris/tests/integration/test_cf.py
similarity index 70%
rename from lib/iris/tests/test_cf.py
rename to lib/iris/tests/integration/test_cf.py
index ca066c6c7b..ba7d0680c5 100644
--- a/lib/iris/tests/test_cf.py
+++ b/lib/iris/tests/integration/test_cf.py
@@ -2,19 +2,23 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Test the cf module."""
+"""Integration tests for :mod:`iris.fileformats.cf`."""
-import contextlib
-import io
from typing import Iterable
+import warnings
import pytest
import iris
+from iris._deprecation import IrisDeprecation
import iris.fileformats.cf as cf
-from iris.fileformats.netcdf import DECODE_TO_STRINGS_ON_READ
-from iris.fileformats.netcdf.loader import _get_cf_var_data
+from iris.fileformats.cf.loader import _get_cf_var_data
+from iris.fileformats.netcdf import DECODE_TO_STRINGS_ON_READ, _thread_safe_nc
+from iris.loading import LOAD_PROBLEMS
from iris.tests import _shared_utils
+import iris.tests.stock as stock
+from iris.tests.stock.mesh import sample_mesh_cube
+from iris.warnings import IrisLoadWarning
def fetch_cfvar_data(var, indices=()):
@@ -30,37 +34,6 @@ def fetch_cfvar_data(var, indices=()):
return data
-class TestCaching:
- def test_cached(self, mocker):
- # Make sure attribute access to the underlying netCDF4.Variable
- # is cached.
- name = "foo"
- nc_var = mocker.MagicMock()
- cf_var = cf.CFAncillaryDataVariable(name, nc_var)
- assert nc_var.ncattrs.call_count == 1
-
- # Accessing a netCDF attribute should result in no further calls
- # to nc_var.ncattrs() and the creation of an attribute on the
- # cf_var.
- # NB. Can't use hasattr() because that triggers the attribute
- # to be created!
- assert "coordinates" not in cf_var.__dict__
- _ = cf_var.coordinates
- assert nc_var.ncattrs.call_count == 1
- assert "coordinates" in cf_var.__dict__
-
- # Trying again results in no change.
- _ = cf_var.coordinates
- assert nc_var.ncattrs.call_count == 1
- assert "coordinates" in cf_var.__dict__
-
- # Trying another attribute results in just a new attribute.
- assert "standard_name" not in cf_var.__dict__
- _ = cf_var.standard_name
- assert nc_var.ncattrs.call_count == 1
- assert "standard_name" in cf_var.__dict__
-
-
@_shared_utils.skip_data
class TestCFReader:
@pytest.fixture(autouse=True)
@@ -245,30 +218,6 @@ def test_variable_attribute_touch_pass_0(self):
("units", "degrees_north"),
)
- def test_destructor(self, tmp_path):
- """Test the destructor when reading the dataset fails.
- Related to issue #3312: previously, the `CFReader` would
- always call `close()` on its `_dataset` attribute, even if it
- didn't exist because opening the dataset had failed.
- """
- fn = tmp_path / "tmp.nc"
- with fn.open("wb+") as fh:
- fh.write(b"\x89HDF\r\n\x1a\nBroken file with correct signature")
- fh.flush()
-
- with io.StringIO() as buf:
- with contextlib.redirect_stderr(buf):
- try:
- _ = cf.CFReader(str(fn))
- except OSError:
- pass
- try:
- _ = iris.load_cubes(str(fn))
- except OSError:
- pass
- buf.seek(0)
- assert buf.read() == ""
-
@_shared_utils.skip_data
class TestLoad:
@@ -428,3 +377,120 @@ def test_label_dim_end(self):
expect = "IFS33R1/HOPE-E, Sys 1, Met 1, ENSEMBLES"
expect = expect.ljust(len(str(content))) # expand to full string width
assert content == expect
+
+
+class TestLoadingIsSilent:
+ """A normal load must not reach CFVariable.__getattr__.
+
+ Every CF attribute read inside Iris goes through CFVariable.attributes,
+ so the deprecation on __getattr__ fires only for code outside Iris.
+ Spec section 4.3.
+
+ The three cubes between them cover the four attributes that reached
+ __getattr__ when the deprecation was added: "bounds" and "climatology"
+ from the loading rules, "cf_role" and "start_index" from the UGRID
+ loader.
+ """
+
+ CUBES = {
+ "bounded": lambda: stock.simple_2d(with_bounds=True),
+ "mesh": sample_mesh_cube,
+ "climatology": lambda: stock.climatology_3d(),
+ }
+
+ @pytest.fixture(params=sorted(CUBES))
+ def saved_file(self, request, tmp_path):
+ path = tmp_path / f"{request.param}.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(self.CUBES[request.param](), path)
+ return path
+
+ def test_load_emits_no_deprecation(self, saved_file):
+ # Recorded rather than promoted to an error. iris.load captures an
+ # exception raised while building a cube into
+ # iris.loading.LOAD_PROBLEMS, so a filter of "error" drops the
+ # offending coordinate and lets the test pass while the warning is
+ # still being issued. The LOAD_PROBLEMS assertion below is what keeps
+ # a silent load from being a degraded one.
+ with warnings.catch_warnings(record=True) as record:
+ warnings.simplefilter("always")
+ cubes = iris.load(saved_file)
+
+ deprecations = [
+ str(warning.message)
+ for warning in record
+ if issubclass(warning.category, IrisDeprecation)
+ ]
+ assert deprecations == []
+ assert LOAD_PROBLEMS.problems_by_file.get(str(saved_file), []) == []
+ assert len(cubes) == 1
+
+ def test_a_cf_attribute_read_from_outside_iris_still_warns(self, saved_file):
+ # The other half of the deprecation: silence on a load must not mean
+ # the accessor has stopped working. All three cubes carry a long_name,
+ # and CFVariable declares no property of that name.
+ with cf.CFReader(str(saved_file)) as reader:
+ (data_var,) = reader.cf_group.data_variables.values()
+ expected = data_var.attributes.untracked["long_name"]
+ with pytest.warns(IrisDeprecation):
+ assert data_var.long_name == expected
+
+ @pytest.mark.parametrize(
+ ("cube_name", "coord_name"),
+ [("bounded", "foo"), ("climatology", "time")],
+ )
+ def test_a_cf_reserved_attribute_does_not_reach_the_coord(
+ self, tmp_path, cube_name, coord_name
+ ):
+ # Belt and braces, not a test of attribute tracking. "bounds" and
+ # "climatology" are both in iris.fileformats.cf.saver._CF_ATTRS,
+ # which _add_unused_attributes filters on whatever tracking recorded,
+ # so they stay off the coordinate either way. The connectivity test
+ # below is the one that pins tracking: "cf_role" and "start_index" are
+ # not in that list.
+ path = tmp_path / f"{cube_name}.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(self.CUBES[cube_name](), path)
+
+ cube = iris.load_cube(path)
+
+ assert cube.coord(coord_name).attributes == {}
+
+ def test_a_cf_reserved_attribute_does_not_reach_the_connectivity(self, tmp_path):
+ # "cf_role" and "start_index" are consumed by the UGRID loader and are
+ # not in cf.saver._CF_ATTRS, so attribute tracking is the only
+ # thing keeping them off the connectivity. Reading them through the
+ # mapping has to record them exactly as reading them as attributes did.
+ path = tmp_path / "mesh.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(sample_mesh_cube(), path)
+
+ cube = iris.load_cube(path)
+
+ assert cube.mesh.face_node_connectivity.attributes == {}
+
+ def test_a_connectivity_without_start_index_loses_its_mesh(self, tmp_path):
+ """A UGRID file omitting the optional "start_index" loads without a mesh.
+
+ CF-UGRID makes start_index optional with a default of 0, and
+ iris.mesh.Connectivity defaults it to 0, but the loader requires it.
+ The mesh build fails and is recorded in iris.loading.LOAD_PROBLEMS.
+ Pinned because the loader reads start_index through the attribute
+ mapping. Invert this when SciTools/iris#7332 is fixed.
+ """
+ path = tmp_path / "mesh.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(sample_mesh_cube(), path)
+
+ dataset = _thread_safe_nc.DatasetWrapper(str(path), "a")
+ try:
+ for variable in dataset.variables.values():
+ if "start_index" in variable.ncattrs():
+ variable.delncattr("start_index")
+ finally:
+ dataset.close()
+
+ with pytest.warns(IrisLoadWarning):
+ cube = iris.load_cube(path)
+
+ assert cube.mesh is None
diff --git a/lib/iris/tests/integration/test_cube.py b/lib/iris/tests/integration/test_cube.py
index beb0d87b57..6423d395ee 100644
--- a/lib/iris/tests/integration/test_cube.py
+++ b/lib/iris/tests/integration/test_cube.py
@@ -20,7 +20,7 @@ def test_agg_by_aux_coord(self, mocker):
("NetCDF", "testing", "small_theta_colpex.nc")
)
# While loading, "turn off" loading small variables as real data.
- mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 0)
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
cube = iris.load_cube(problem_test_file, "air_potential_temperature")
# Test aggregating by aux coord, notably the `forecast_period` aux
# coord on `cube`, whose `_points` attribute is a lazy array.
diff --git a/lib/iris/tests/integration/test_netcdf__loadsaveattrs.py b/lib/iris/tests/integration/test_netcdf__loadsaveattrs.py
index 9a48c1662c..ac5060ec43 100644
--- a/lib/iris/tests/integration/test_netcdf__loadsaveattrs.py
+++ b/lib/iris/tests/integration/test_netcdf__loadsaveattrs.py
@@ -34,6 +34,7 @@
import iris.coord_systems
from iris.coords import DimCoord
from iris.cube import Cube
+import iris.fileformats.cf.saver
import iris.fileformats.netcdf
import iris.fileformats.netcdf._thread_safe_nc as threadsafe_nc4
@@ -48,7 +49,7 @@
# A list of "global-style" attribute names : those which should be global attributes by
# default (i.e. file- or group-level, *not* attached to a variable).
-_GLOBAL_TEST_ATTRS = sorted(iris.fileformats.netcdf.saver._CF_GLOBAL_ATTRS)
+_GLOBAL_TEST_ATTRS = sorted(iris.fileformats.cf.saver._CF_GLOBAL_ATTRS)
# Remove this one, which has peculiar behaviour + is tested separately
# N.B. this is not the same as 'Conventions', but is caught in the crossfire when that
# one is processed.
@@ -66,8 +67,8 @@ def global_attr(request):
# A list of "local-style" attribute names : those which should be variable attributes
# by default (aka "local", "variable" or "data" attributes) .
_LOCAL_TEST_ATTRS = (
- iris.fileformats.netcdf.saver._CF_DATA_ATTRS
- + iris.fileformats.netcdf.saver._UKMO_DATA_ATTRS
+ iris.fileformats.cf.saver._CF_DATA_ATTRS
+ + iris.fileformats.cf.saver._UKMO_DATA_ATTRS
)
# For simplicity, we exclude several special cases that don't act with "standard"
# behaviour (because they translate in special ways):
diff --git a/lib/iris/tests/integration/zarr/__init__.py b/lib/iris/tests/integration/zarr/__init__.py
new file mode 100644
index 0000000000..b7761f385a
--- /dev/null
+++ b/lib/iris/tests/integration/zarr/__init__.py
@@ -0,0 +1,12 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Integration tests for native Zarr loading.
+
+Skipped when zarr-python is absent, which is also how the unit tests do it.
+"""
+
+import pytest
+
+pytest.importorskip("zarr", minversion="3.0.8")
diff --git a/lib/iris/tests/integration/zarr/fixtures.py b/lib/iris/tests/integration/zarr/fixtures.py
new file mode 100644
index 0000000000..ee300200b0
--- /dev/null
+++ b/lib/iris/tests/integration/zarr/fixtures.py
@@ -0,0 +1,140 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""A version 3 store carrying the CF features a real one does.
+
+Modelled on the NOAA GFS archive at dynamical.org, whose properties the Zarr
+I/O design names as the ones worth exercising: version 3 `dimension_names`, a
+sharded codec, a scalar grid-mapping variable, a two-dimensional auxiliary
+coordinate, and the base64 `_FillValue` that released xarray writes for every
+floating dtype. A cut-down copy of the real store replaces this.
+
+Bounds on `latitude` are not part of that real store, but earn their place
+anyway: `CFBoundaryVariable.spans` is one of the overrides this pull request
+rewrote against the dimension contract, so a bounded coordinate exercises
+code this branch actually changed.
+"""
+
+import base64
+import struct
+
+import numpy as np
+import zarr
+
+SHAPE = (2, 4, 6, 5)
+DIMENSIONS = ("time", "level", "latitude", "longitude")
+
+#: Contiguous bounds for `latitude`, half a cell either side of each centre.
+LATITUDE_BOUNDS = np.stack(
+ [np.arange(SHAPE[2], dtype="f8") - 0.5, np.arange(SHAPE[2], dtype="f8") + 0.5],
+ axis=1,
+)
+
+
+def float_fill_value(value):
+ """Return `value` in the encoding released xarray writes for a float."""
+ return base64.b64encode(struct.pack("<d", value)).decode("ascii")
+
+
+def make_gfs_like_store(path):
+ """Write the store and return its path as a string."""
+ root = zarr.open_group(store=path, mode="w", zarr_format=3)
+ root.attrs["Conventions"] = "CF-1.12"
+ root.attrs["title"] = "A store shaped like a real one"
+ root.attrs["license"] = "CC-BY-4.0"
+
+ sizes = dict(zip(DIMENSIONS, SHAPE))
+ for name, size in sizes.items():
+ coordinate = root.create_array(
+ name, shape=(size,), dtype="f8", dimension_names=(name,)
+ )
+ coordinate[:] = np.arange(size, dtype="f8")
+ root["time"].attrs.update(
+ {"units": "hours since 2000-01-01 00:00:00", "calendar": "standard"}
+ )
+ root["level"].attrs.update({"units": "hPa", "positive": "down"})
+ root["latitude"].attrs.update(
+ {"units": "degrees_north", "axis": "Y", "bounds": "latitude_bnds"}
+ )
+ root["longitude"].attrs.update({"units": "degrees_east", "axis": "X"})
+
+ # Bounds on a dimension coordinate, exercising `CFBoundaryVariable.spans`:
+ # one more dimension than `latitude`, named for a dimension `latitude`
+ # does not have, at the size CF bounds always are.
+ bounds = root.create_array(
+ "latitude_bnds",
+ shape=(SHAPE[2], 2),
+ dtype="f8",
+ dimension_names=("latitude", "bnds"),
+ )
+ bounds[:] = LATITUDE_BOUNDS
+
+ # A scalar grid mapping, as the GFS store's `spatial_ref` is.
+ grid = root.create_array("spatial_ref", shape=(), dtype="i4")
+ grid[()] = 0
+ grid.attrs.update(
+ {"grid_mapping_name": "latitude_longitude", "earth_radius": 6371229.0}
+ )
+
+ # A two-dimensional auxiliary coordinate, as the GFS store's valid_time is.
+ valid_time = root.create_array(
+ "valid_time",
+ shape=SHAPE[:2],
+ dtype="f8",
+ dimension_names=DIMENSIONS[:2],
+ )
+ valid_time[:] = np.arange(np.prod(SHAPE[:2]), dtype="f8").reshape(SHAPE[:2])
+ valid_time.attrs.update(
+ {
+ "units": "hours since 2000-01-01 00:00:00",
+ "standard_name": "time",
+ "statistics_approximate": {"min": 0.0, "max": 7.0},
+ }
+ )
+
+ # The data variable: sharded, packed, with a base64 fill value.
+ data = root.create_array(
+ "temperature_2m",
+ shape=SHAPE,
+ dtype="i2",
+ chunks=(1, 1, 3, 5),
+ shards=(2, 4, 6, 5),
+ dimension_names=DIMENSIONS,
+ fill_value=-32768,
+ )
+ data[:] = np.arange(np.prod(SHAPE), dtype="i2").reshape(SHAPE)
+ data.attrs.update(
+ {
+ "standard_name": "air_temperature",
+ "units": "K",
+ "scale_factor": 0.01,
+ "add_offset": 273.15,
+ # An integer variable's sentinel is a JSON number. The base64
+ # encoding is for floating dtypes, and is on `pressure_surface`.
+ "_FillValue": -32768,
+ "coordinates": "valid_time",
+ "grid_mapping": "spatial_ref",
+ }
+ )
+
+ # A floating variable, carrying the base64 `_FillValue` that released
+ # xarray writes for every floating dtype.
+ flat_shape = SHAPE[:1] + SHAPE[2:]
+ pressure = root.create_array(
+ "pressure_surface",
+ shape=flat_shape,
+ dtype="f4",
+ dimension_names=DIMENSIONS[:1] + DIMENSIONS[2:],
+ )
+ pressure[:] = np.arange(int(np.prod(flat_shape)), dtype="f4").reshape(flat_shape)
+ pressure.attrs.update(
+ {
+ "standard_name": "surface_air_pressure",
+ "units": "Pa",
+ "_FillValue": float_fill_value(float("nan")),
+ "grid_mapping": "spatial_ref",
+ }
+ )
+
+ return str(path)
diff --git a/lib/iris/tests/integration/zarr/test_dimension_name_conflict.py b/lib/iris/tests/integration/zarr/test_dimension_name_conflict.py
new file mode 100644
index 0000000000..89c6093207
--- /dev/null
+++ b/lib/iris/tests/integration/zarr/test_dimension_name_conflict.py
@@ -0,0 +1,66 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""A dimension name used at two different lengths, loaded end to end.
+
+Zarr "does not place any restrictions on the use of the same dimension name
+across multiple arrays", so a store may name two arrays' dimensions ``t`` and
+give them different lengths. Iris cannot link those, and the arrays load with
+an anonymous dimension instead.
+
+The warning is the only thing that explains why. A coordinate that could not
+attach reaches the user through `LOAD_PROBLEMS`, which names the coordinate
+and not the cause. These tests go through `iris.load` rather than building a
+`ZarrDataset`, because the warning comes from a dataset member that the CF
+layer has no other reason to touch - the unit tests pass either way.
+"""
+
+import numpy as np
+import pytest
+import zarr
+
+import iris
+from iris.loading import LOAD_PROBLEMS
+from iris.warnings import IrisCfLoadWarning
+
+
+@pytest.fixture(autouse=True)
+def _reset_load_problems():
+ """These loads record problems, so neither leaks into another test."""
+ LOAD_PROBLEMS.reset()
+ yield
+ LOAD_PROBLEMS.reset()
+
+
+@pytest.fixture
+def conflicting_store(tmp_path):
+ """A store naming ``t`` at two lengths, with a coordinate for the shorter."""
+ location = tmp_path / "conflict.zarr"
+ root = zarr.open_group(store=location, mode="w", zarr_format=3)
+ root.attrs["Conventions"] = "CF-1.12"
+
+ def array(name, size, **attributes):
+ written = root.create_array(
+ name, shape=(size,), dtype="f4", dimension_names=("t",)
+ )
+ written[:] = np.arange(size, dtype="f4")
+ written.attrs.update(attributes)
+
+ array("t", 4, standard_name="time", units="days since 2000-01-01")
+ array("air_temperature", 4, standard_name="air_temperature", units="K")
+ array("air_pressure", 5, standard_name="air_pressure", units="Pa")
+ return str(location)
+
+
+def test_the_conflict_reaches_a_load(conflicting_store):
+ with pytest.warns(IrisCfLoadWarning, match=r"more than one size, \[4, 5\]: t"):
+ iris.load(conflicting_store)
+
+
+def test_the_unlinkable_dimension_is_anonymous(conflicting_store):
+ cubes = iris.load(conflicting_store)
+
+ by_name = {cube.name(): cube for cube in cubes}
+ assert by_name["air_pressure"].coords() == []
+ assert [coord.name() for coord in by_name["air_temperature"].coords()] == ["time"]
diff --git a/lib/iris/tests/integration/zarr/test_load.py b/lib/iris/tests/integration/zarr/test_load.py
new file mode 100644
index 0000000000..cd9b35c638
--- /dev/null
+++ b/lib/iris/tests/integration/zarr/test_load.py
@@ -0,0 +1,220 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Integration tests for loading native Zarr stores."""
+
+import warnings
+
+import numpy as np
+import pytest
+
+import iris
+from iris.coord_systems import GeogCS
+from iris.fileformats.cf import loader as cf_loader
+from iris.loading import LOAD_PROBLEMS
+from iris.tests import stock
+from iris.tests.unit.fileformats.zarr import stores as zarr_stores
+from iris.warnings import IrisLoadWarning
+
+from . import fixtures
+
+
+@pytest.fixture(autouse=True)
+def _assert_clean_load():
+ """Every load in this module must be free of recorded problems.
+
+ A cube that loads "successfully" while quietly recording a load problem
+ is a weaker result than it looks, so `IrisLoadWarning` is escalated to an
+ error for the duration of each test, and `LOAD_PROBLEMS` is asserted
+ empty afterwards.
+ """
+ LOAD_PROBLEMS.reset()
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisLoadWarning)
+ yield
+ assert LOAD_PROBLEMS.problems == []
+
+
+@pytest.fixture
+def gfs_like(tmp_path):
+ return fixtures.make_gfs_like_store(tmp_path / "gfs.zarr")
+
+
+def test_the_store_loads_two_cubes(gfs_like):
+ cubes = iris.load(gfs_like)
+ assert sorted(cube.name() for cube in cubes) == [
+ "air_temperature",
+ "surface_air_pressure",
+ ]
+
+
+def test_dimension_coordinates_attach(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ assert [coord.name() for coord in cube.coords(dim_coords=True)] == [
+ "time",
+ "level",
+ "latitude",
+ "longitude",
+ ]
+ assert cube.shape == fixtures.SHAPE
+
+
+def test_bounds_attach_to_their_coordinate(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ latitude = cube.coord("latitude")
+ assert latitude.has_bounds()
+ np.testing.assert_array_equal(latitude.bounds, fixtures.LATITUDE_BOUNDS)
+
+
+def test_the_auxiliary_coordinate_attaches_to_its_own_dimensions(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ # `valid_time` carries `standard_name: "time"`, as the real GFS store's
+ # does, so it shares its Iris name with the "time" dimension coordinate;
+ # `var_name` is what still tells the two apart.
+ aux = cube.coord(var_name="valid_time")
+ assert cube.coord_dims(aux) == (0, 1)
+
+
+def test_the_grid_mapping_becomes_a_coordinate_system(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ assert isinstance(cube.coord("latitude").coord_system, GeogCS)
+
+
+def test_packing_is_applied(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ assert cube.dtype == np.dtype("f8")
+ # The first stored value is 0, so scale_factor * 0 + add_offset.
+ assert cube.data.flat[0] == pytest.approx(273.15)
+
+
+def test_the_base64_fill_value_masks_nothing_here(gfs_like):
+ """A NaN sentinel over data with no NaNs masks nothing.
+
+ The attribute is still read, and read wrongly it raises rather than
+ quietly doing nothing - which is why this asserts the unmasked result
+ rather than only that loading succeeded.
+ """
+ cube = iris.load_cube(gfs_like, "surface_air_pressure")
+ assert not np.ma.is_masked(cube.data)
+
+
+def test_the_cube_is_lazy_and_chunked_to_the_store(gfs_like, mocker):
+ # `temperature_2m` is a few hundred bytes, well under the small-array
+ # cutoff (`_LAZYVAR_MIN_BYTES`) below which the CF loader reads a
+ # variable whole rather than lazily. A real store is nowhere near that
+ # small, so the cutoff is lowered here to exercise the chunking this
+ # test is actually about.
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ assert cube.has_lazy_data()
+ # The chunking here tiles the store's inner chunk grid, which is what
+ # `_optimum_chunksize` gives wherever it expands rather than shrinks.
+ # Iris does not enforce it: a store chunk larger than Dask's target is
+ # divided, and warned about, rather than rounded back up.
+ store_chunks = (1, 1, 3, 5)
+ for sizes, store_size in zip(cube.lazy_data().chunks, store_chunks):
+ assert all(size % store_size == 0 for size in sizes[:-1])
+
+
+def test_a_structured_attribute_survives(gfs_like):
+ cube = iris.load_cube(gfs_like, "air_temperature")
+ aux = cube.coord(var_name="valid_time")
+ assert aux.attributes["statistics_approximate"] == {
+ "min": 0.0,
+ "max": 7.0,
+ }
+
+
+@pytest.fixture
+def realistic_4d_w_everything():
+ """Return a realistic 4D cube with all the bells and whistles."""
+ cube = stock.realistic_4d_w_everything()
+ # Precision loss for floating-point scalar variable attributes:
+ # earth_radius 6371229.0 round-trips as 6371230.0.
+ for coord_name in ("grid_latitude", "grid_longitude"):
+ cube.coord(coord_name).coord_system.ellipsoid = GeogCS(6371230.0)
+ return cube
+
+
+def test_the_native_reader_agrees_with_the_nczarr_writer(
+ tmp_path, realistic_4d_w_everything
+):
+ """A version 2 store Iris wrote through netCDF-c, read natively.
+
+ The NCZarr `xarray` mode writes `_ARRAY_DIMENSIONS`, which is how a
+ version 2 store records dimension names. The two code paths share no
+ code, so agreement between them is evidence rather than a tautology.
+
+ `realistic_4d_w_everything` carries a hybrid height factory, whose
+ orography dependency round-trips through CF as its own top-level cube
+ rather than folding back into the main one - true of a plain netCDF
+ round trip of this fixture as well, so a name constraint selects the
+ main cube deliberately rather than papering over a Zarr-specific gap.
+
+ None of this fixture's variables declare a CF `_FillValue`, but netCDF-c's
+ NCZarr writer still gives every one its own default storage fill value -
+ the classic netCDF sentinel `9.96921e+36`, not zero, so this does not
+ warn: see `lib/iris/fileformats/zarr/_decode.py` for why zero is what
+ distinguishes a writer's deliberate pre-convention sentinel from
+ zarr-python's own default.
+ """
+ path = tmp_path / "nczarr.zarr"
+ uri = path.as_uri() + "#mode=xarray,file"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(realistic_4d_w_everything, uri)
+
+ assert (path / ".zgroup").is_file()
+ loaded = iris.load_cube(str(path), realistic_4d_w_everything.name())
+ # `_NCProperties`, netCDF-c's own bookkeeping stamp on the root group of
+ # every store it writes, is withheld by the reader rather than compared
+ # away here - the same storage-not-CF treatment as `_ARRAY_DIMENSIONS`
+ # below, and covered by its own unit test in
+ # `lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py`.
+ assert loaded == realistic_4d_w_everything
+
+
+def test_array_dimensions_does_not_reach_the_cube(tmp_path, realistic_4d_w_everything):
+ """It is how version 2 records dimension names, so it is storage.
+
+ Left in the CF attributes it would land on every cube loaded from a
+ version 2 store, and make it unequal to the same data in version 3.
+ Checked across every cube the store yields, including the orography
+ cube the hybrid height factory depends on, not only the main one.
+ """
+ path = tmp_path / "nczarr.zarr"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(realistic_4d_w_everything, path.as_uri() + "#mode=xarray,file")
+
+ cubes = iris.load(str(path))
+ assert len(cubes) == 2
+ for cube in cubes:
+ assert "_ARRAY_DIMENSIONS" not in cube.attributes
+ for coord in cube.coords():
+ assert "_ARRAY_DIMENSIONS" not in coord.attributes
+
+
+def test_a_mesh_from_zarr_is_a_full_iris_mesh(tmp_path):
+ """The claim the Zarr mesh path rests on, taken to netCDF and back.
+
+ `iris.fileformats.netcdf.ugrid_load` reads a `CFReader`, and a `CFReader`
+ over a `ZarrDataset` answers the same questions. A mesh built from a Zarr
+ store that then survives a netCDF round trip unchanged is a mesh with
+ nothing Zarr-shaped left in it.
+ """
+ from_zarr = iris.load_cube(zarr_stores.make_mesh_store(tmp_path / "mesh.zarr"))
+ assert from_zarr.mesh is not None
+ assert from_zarr.mesh_dim() is not None
+
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(from_zarr, tmp_path / "roundtrip.nc")
+ reloaded = iris.load_cube(str(tmp_path / "roundtrip.nc"))
+
+ assert reloaded.mesh == from_zarr.mesh
+ assert reloaded.mesh_dim() == from_zarr.mesh_dim()
+ np.testing.assert_array_equal(reloaded.data, from_zarr.data)
+ # Not the whole cube: the netCDF saver writes its own `Conventions`, so
+ # `CF-1.7, UGRID-1.0` comes back as `CF-1.7`. Nothing mesh-shaped.
+ assert reloaded.metadata._replace(attributes=None) == from_zarr.metadata._replace(
+ attributes=None
+ )
diff --git a/lib/iris/tests/results/file_load/known_loaders.txt b/lib/iris/tests/results/file_load/known_loaders.txt
index 51824c5273..7c3ee89ec6 100644
--- a/lib/iris/tests/results/file_load/known_loaders.txt
+++ b/lib/iris/tests/results/file_load/known_loaders.txt
@@ -4,6 +4,7 @@
* NetCDF 64 bit offset format (priority 5)
* NetCDF_v4 (priority 5)
* UM Post Processing file (PP) (priority 5)
+ * Zarr (priority 5)
* NetCDF dataset (priority 4)
* UM Fieldsfile (FF) post v5.2 (priority 4)
* ABF (priority 3)
diff --git a/lib/iris/tests/test_coding_standards.py b/lib/iris/tests/test_coding_standards.py
index dbfd726640..78fb3be592 100644
--- a/lib/iris/tests/test_coding_standards.py
+++ b/lib/iris/tests/test_coding_standards.py
@@ -3,132 +3,86 @@
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-import ast
-from datetime import datetime
-from fnmatch import fnmatch
-from glob import glob
+import importlib.util
import os
from pathlib import Path
-import subprocess
-from typing import Iterator, List, Tuple, cast
-from packaging.version import Version
import pytest
import iris
-from iris.tests import system_test
-from iris.tests.unit.fileformats.netcdf import test_bytecoding_datasets
-
-LICENSE_TEMPLATE = """# Copyright Iris contributors
-#
-# This file is part of Iris and is released under the BSD license.
-# See LICENSE in the root of the repository for full licensing details."""
# Guess iris repo directory of Iris - realpath is used to mitigate against
# Python finding the iris package via a symlink.
-IRIS_DIR = Path(iris.__file__).parent.resolve()
-IRIS_INSTALL_DIR = Path(IRIS_DIR).parent.parent
-DOCS_DIR = Path(IRIS_INSTALL_DIR) / "docs" / "iris"
-DOCS_DIR = iris.config.get_option("Resources", "doc_dir", default=str(DOCS_DIR))
-exclusion = ["Makefile", "build"]
-DOCS_DIRS = glob(str(Path(DOCS_DIR) / "*"))
-DOCS_DIRS = [DOC_DIR for DOC_DIR in DOCS_DIRS if Path(DOC_DIR).name not in exclusion]
-
+IRIS_DIR = os.path.realpath(os.path.dirname(iris.__file__))
+IRIS_INSTALL_DIR = os.path.dirname(os.path.dirname(IRIS_DIR))
# Get a dirpath to the git repository : allow setting with an environment
# variable, so Travis can test for headers in the repo, not the installation.
IRIS_REPO_DIRPATH = os.environ.get("IRIS_REPO_DIR", IRIS_INSTALL_DIR)
+def _load_hook(hook_name: str):
+ """Load a hook module from .hooks/.
+
+ Args:
+ hook_name: Name of the hook module (without .py extension).
+
+ Returns
+ -------
+ The loaded module.
+
+ Raises
+ ------
+ RuntimeError: If the hook module cannot be loaded.
+ """
+ hook_path = Path(__file__).parents[3] / ".hooks" / f"{hook_name}.py"
+ spec = importlib.util.spec_from_file_location(hook_name, hook_path)
+ if spec is None or spec.loader is None:
+ raise RuntimeError(f"Failed to load hook: {hook_name}")
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
def test_netcdf4_import():
"""Use of netCDF4 must be via iris.fileformats.netcdf._thread_safe_nc ."""
- # Please avoid including these phrases in any comments/strings throughout
- # Iris (e.g. use "from the netCDF4 library" instead) - this allows the
- # below search to remain quick and simple.
- from iris.fileformats.netcdf import _thread_safe_nc
- from iris.tests.unit.fileformats.netcdf._thread_safe_nc import test_NetCDFWriteProxy
-
- import_strings = ("import netCDF4", "from netCDF4")
+ # Logic lives in .hooks/check_netcdf4_imports.py (also used as a pre-commit hook).
+ check_netcdf4_imports = _load_hook("check_netcdf4_imports")
+ check_file = check_netcdf4_imports.check_file
- files_including_import = []
+ all_violations = []
for file_path in Path(IRIS_DIR).rglob("*.py"):
- file_text = file_path.read_text()
+ all_violations.extend(check_file(file_path))
- if any([i in file_text for i in import_strings]):
- files_including_import.append(file_path)
-
- expected = [
- Path(_thread_safe_nc.__file__),
- Path(test_NetCDFWriteProxy.__file__),
- Path(system_test.__file__),
- Path(__file__),
- Path(test_bytecoding_datasets.__file__),
- ]
- assert set(files_including_import) == set(expected)
+ message = (
+ "The following files import netCDF4 directly, which is not allowed:\n"
+ + "\n".join(all_violations)
+ + "\nAll netCDF4 imports must be via iris.fileformats.netcdf._thread_safe_nc."
+ )
+ assert not all_violations, message
-def test_python_versions() -> None:
+def test_python_versions():
"""Test Python Versions.
Test is designed to fail whenever Iris' supported Python versions are
updated, insisting that versions are updated EVERYWHERE in-sync.
- """
- all_supported = ["3.12", "3.13", "3.14"]
- _parsed = [Version(v) for v in all_supported]
- latest_supported = str(max(_parsed))
-
- root_dir = Path(__file__).parents[3]
- workflows_dir = root_dir / ".github" / "workflows"
- benchmarks_dir = root_dir / "benchmarks"
-
- # Places that are checked:
- pyproject_toml_file = root_dir / "pyproject.toml"
- requirements_dir = root_dir / "requirements"
- nox_file = root_dir / "noxfile.py"
- ci_wheels_file = workflows_dir / "ci-wheels.yml"
- ci_tests_file = workflows_dir / "ci-tests.yml"
- benchmark_runner_file = benchmarks_dir / "bm_runner.py"
-
- text_searches: List[Tuple[Path, str]] = [
- (
- pyproject_toml_file,
- "\n ".join(
- [f'"Programming Language :: Python :: {ver}",' for ver in all_supported]
- ),
- ),
- (
- nox_file,
- "_PY_VERSIONS_ALL = [" + ", ".join([f'"{ver}"' for ver in all_supported]),
- ),
- (
- ci_wheels_file,
- "python-version: [" + ", ".join([f'"{ver}"' for ver in all_supported]),
- ),
- (
- ci_tests_file,
- (
- f'python-version: ["{latest_supported}"]\n'
- f'{" " * 8}session: ["doctest", "gallery"]'
- ),
- ),
- (benchmark_runner_file, f'python_version = "{latest_supported}"'),
- ]
- for ver in all_supported:
- req_yaml = requirements_dir / f"py{ver.replace('.', '')}.yml"
- text_searches.append((req_yaml, f"- python ={ver}"))
+ Logic lives in .hooks/check_python_versions.py (also used as a pre-commit hook).
+ """
+ check_python_versions = _load_hook("check_python_versions")
+ repo_root = Path(IRIS_REPO_DIRPATH)
- text_searches.append(
- (
- ci_tests_file,
- f'python-version: "{ver}"\n{" " * 12}session: "tests"',
- )
- )
+ violations = check_python_versions.check_consistency(repo_root)
- for path, search in text_searches:
- assert search in path.read_text()
+ message = (
+ "Python version consistency check failed.\n"
+ "Python versions must be updated consistently across all config files.\n"
+ + "\n".join(violations)
+ )
+ assert not violations, message
-def test_categorised_warnings() -> None:
+def test_categorised_warnings():
r"""To ensure that all UserWarnings raised by Iris are categorised, for ease of use.
No obvious category? Use the parent:
@@ -145,151 +99,233 @@ class _WarnComboCfDefaulting(IrisCfWarning, IrisDefaultingWarning):
\"\"\"
pass
+ Logic lives in .hooks/check_categorised_warnings.py (also used as a pre-commit hook).
"""
- warns_without_category = []
- warns_with_user_warning = []
- tmp_list = []
+ check_categorised_warnings = _load_hook("check_categorised_warnings")
+ check_file = check_categorised_warnings.check_file
+ all_violations = []
for file_path in Path(IRIS_DIR).rglob("*.py"):
- file_text = file_path.read_text()
- parsed = ast.parse(source=file_text)
- calls: Iterator[ast.Call] = cast(
- "Iterator[ast.Call]",
- filter(lambda node: hasattr(node, "func"), ast.walk(parsed)),
- )
- warn_calls: Iterator[ast.Call] = filter(
- lambda c: getattr(c.func, "attr", None) == "warn", calls
- )
-
- warn_call: ast.Call
- for warn_call in warn_calls:
- warn_ref = f"{file_path}:{warn_call.lineno}"
- tmp_list.append(warn_ref)
-
- category_kwargs = filter(lambda k: k.arg == "category", warn_call.keywords)
- category_kwarg: ast.keyword | None = next(category_kwargs, None)
-
- if category_kwarg is None:
- warns_without_category.append(warn_ref)
- # Work with Attribute or Name instances.
- elif (
- getattr(category_kwarg.value, "attr", None)
- or getattr(category_kwarg.value, "id", None)
- ) == "UserWarning":
- warns_with_user_warning.append(warn_ref)
-
- # This avoids UserWarnings being raised by unwritten default behaviour.
- assert warns_without_category == [], (
- "All warnings raised by Iris must be raised with the category kwarg."
+ all_violations.extend(check_file(file_path))
+
+ message = "The following files have warning categorisation issues:\n" + "\n".join(
+ all_violations
)
+ assert not all_violations, message
+
+
+def test_license_headers():
+ """Check that all Python files have the required license header.
+
+ Logic lives in .hooks/check_license_headers.py (also used as a pre-commit hook).
+ """
+ check_license_headers = _load_hook("check_license_headers")
+ repo_root = Path(IRIS_REPO_DIRPATH)
+
+ all_violations: list[str] = []
+ for file_path in check_license_headers._get_all_tracked_files(repo_root):
+ all_violations.extend(check_license_headers.check_file(file_path, repo_root))
+
+ message = (
+ "The following files are missing or have incorrect license headers:\n"
+ + "\n".join(all_violations)
+ )
+ assert not all_violations, message
+
+
+_DOCS_PAGE = """\
+.. z_reference:: Phrasebook
+ :tags: topic_interoperability
+
+ Information on terminology differences between Iris and similar packages.
- assert warns_with_user_warning == [], (
- "No warnings raised by Iris can be the base UserWarning class."
+.. _phrasebook:
+
+Package Phrasebook
+==================
+
+.. readingtime::
+
+Body text.
+"""
+"""A documentation page carrying every piece of metadata Iris requires."""
+
+
+def test_docs_page_metadata():
+ """Check that all documentation pages carry their required metadata.
+
+ Logic lives in .hooks/check_docs_page_metadata.py (also used as a
+ pre-commit hook, and by the docs build via the page_metadata_validator
+ Sphinx extension).
+ """
+ check_docs_page_metadata = _load_hook("check_docs_page_metadata")
+
+ all_violations = check_docs_page_metadata.check_tree(Path(IRIS_REPO_DIRPATH))
+
+ message = "The following documentation pages are missing metadata:\n" + "\n".join(
+ all_violations
)
+ assert not all_violations, message
-class TestLicenseHeaders:
- @staticmethod
- def whatchanged_parse(whatchanged_output):
- r"""Returns a generator of tuples of data parsed from
- "git whatchanged --pretty='TIME:%at". The tuples are of the form
- ``(filename, last_commit_datetime)``.
-
- Sample input::
-
- ['TIME:1366884020', '',
- ':000000 100644 0000000... 5862ced... A\tlib/iris/cube.py']
-
- """
- dt = None
- for line in whatchanged_output:
- if not line.strip():
- continue
- elif line.startswith("TIME:"):
- dt = datetime.fromtimestamp(int(line[5:]))
- else:
- # Non blank, non date, line -> must be the lines
- # containing the file info.
- fname = " ".join(line.split("\t")[1:])
- yield fname, dt
-
- @staticmethod
- def last_change_by_fname():
- """Return a dictionary of all the files under git which maps to
- the datetime of their last modification in the git history.
-
- .. note::
-
- This function raises a ValueError if the repo root does
- not have a ".git" folder. If git is not installed on the system,
- or cannot be found by subprocess, an IOError may also be raised.
-
- """
- # Check the ".git" folder exists at the repo dir.
- if not (Path(IRIS_REPO_DIRPATH) / ".git").is_dir():
- msg = "{} is not a git repository."
- raise ValueError(msg.format(IRIS_REPO_DIRPATH))
-
- # Call "git log" to get the details of all the files and when
- # they were last changed.
- output = subprocess.check_output(
- ["git", "log", "--name-status", "--pretty=TIME:%ct"],
- cwd=IRIS_REPO_DIRPATH,
- )
-
- output = output.decode().split("\n")
- res = {}
- for fname, dt in TestLicenseHeaders.whatchanged_parse(output):
- if fname not in res or dt > res[fname]:
- res[fname] = dt
-
- return res
-
- def test_license_headers(self):
- exclude_patterns = (
- "setup.py",
- "noxfile.py",
- "build/*",
- "dist/*",
- "docs/gallery_code/*/*.py",
- "docs/src/developers_guide/documenting/*.py",
- "docs/src/user_manual/tutorial/plotting_examples/*.py",
- "docs/src/user_manual/tutorial/regridding_plots/*.py",
- "docs/src/_build/*",
- "lib/iris/analysis/_scipy_interpolate.py",
- )
-
- try:
- last_change_by_fname = self.last_change_by_fname()
- except ValueError as err:
- # Caught the case where this is not a git repo.
- msg = "Iris installation did not look like a git repo?\nERR = {}\n\n"
- return pytest.skip(msg.format(str(err)))
-
- failed = False
- for fname, last_change in sorted(last_change_by_fname.items()):
- full_fname = Path(IRIS_REPO_DIRPATH) / fname
- is_file = full_fname.is_file()
- full_fname = str(full_fname)
-
- if (
- full_fname.endswith(".py")
- and is_file
- and not any(fnmatch(fname, pat) for pat in exclude_patterns)
- ):
- with open(full_fname) as fh:
- content = fh.read()
- if content.startswith("#!"):
- # account for files with leading shebang directives
- # i.e., first strip out the shebang line before
- # then performing license header compliance checking
- content = "\n".join(content.split("\n")[1:])
- if not content.startswith(LICENSE_TEMPLATE):
- print(
- "The file {} does not start with the required "
- "license header.".format(fname)
- )
- failed = True
-
- if failed:
- raise ValueError("There were license header failures. See stdout.")
+def test_docs_page_metadata_clean():
+ """A fully populated page must be accepted, or the checks below prove nothing."""
+ check_docs_page_metadata = _load_hook("check_docs_page_metadata")
+
+ problems = check_docs_page_metadata.check_page(
+ "user_manual/reference/phrasebook", _DOCS_PAGE
+ )
+
+ assert problems == []
+
+
+@pytest.mark.parametrize(
+ ("page", "expected"),
+ [
+ pytest.param(
+ _DOCS_PAGE.replace(".. readingtime::\n", ""),
+ "Missing '.. readingtime::' directive",
+ id="missing_readingtime",
+ ),
+ pytest.param(
+ _DOCS_PAGE.replace(".. z_reference:: Phrasebook", "Phrasebook"),
+ "found 0.",
+ id="missing_item",
+ ),
+ pytest.param(
+ _DOCS_PAGE + _DOCS_PAGE,
+ "found 2.",
+ id="duplicate_item",
+ ),
+ pytest.param(
+ _DOCS_PAGE.replace(".. z_reference::", ".. tutorial::"),
+ "found type 'tutorial'",
+ id="wrong_type",
+ ),
+ pytest.param(
+ "\n" * 30 + _DOCS_PAGE,
+ "within first 25 lines",
+ id="item_too_far_down",
+ ),
+ pytest.param(
+ _DOCS_PAGE.replace(
+ " Information on terminology differences between Iris and"
+ " similar packages.\n",
+ "",
+ ),
+ "non-empty content section",
+ id="empty_content",
+ ),
+ pytest.param(
+ _DOCS_PAGE.replace(":tags: topic_interoperability", ":tags: data-model"),
+ "'topic_xxx' tag",
+ id="missing_topic_tag",
+ ),
+ ],
+)
+def test_docs_page_metadata_detects(page, expected):
+ """Each way of breaking a page's metadata must be detected.
+
+ These are the faults that would otherwise only surface minutes into a
+ documentation build.
+ """
+ check_docs_page_metadata = _load_hook("check_docs_page_metadata")
+
+ problems = check_docs_page_metadata.check_page(
+ "user_manual/reference/phrasebook", page
+ )
+
+ assert any(expected in problem for problem in problems), (
+ f"Expected a problem containing {expected!r}; got: {problems}"
+ )
+
+
+_DOCS_MODULE = '''\
+"""Summary of what this module is for.
+
+.. z_reference:: Phrasebook
+ :tags: topic_interoperability
+
+ Information on terminology differences between Iris and similar packages.
+
+"""
+
+CONSTANT = 1
+'''
+"""A library module carrying the metadata for the page generated from it."""
+
+
+def test_docs_page_metadata_docstring_clean():
+ """A fully populated docstring must be accepted, or the checks below prove nothing."""
+ check_docs_page_metadata = _load_hook("check_docs_page_metadata")
+
+ problems = check_docs_page_metadata.check_docstring(
+ _DOCS_MODULE, check_docs_page_metadata.API_TYPE
+ )
+
+ assert problems == []
+
+
+@pytest.mark.parametrize(
+ ("source", "expected"),
+ [
+ pytest.param(
+ "CONSTANT = 1\n",
+ "Missing a module docstring",
+ id="missing_docstring",
+ ),
+ pytest.param(
+ '"""Summary of what this module is for."""\n',
+ "found 0.",
+ id="missing_item",
+ ),
+ pytest.param(
+ _DOCS_MODULE.replace(".. z_reference::", ".. how-to::"),
+ "expected to have type 'z_reference'",
+ id="wrong_type",
+ ),
+ ],
+)
+def test_docs_page_metadata_docstring_detects(source, expected):
+ """Pages generated from a module are only as good as that module's docstring.
+
+ Nothing else states their metadata, so a docstring that omits it produces
+ a page that is missing metadata.
+ """
+ check_docs_page_metadata = _load_hook("check_docs_page_metadata")
+
+ problems = check_docs_page_metadata.check_docstring(
+ source, check_docs_page_metadata.API_TYPE
+ )
+
+ assert any(expected in problem for problem in problems), (
+ f"Expected a problem containing {expected!r}; got: {problems}"
+ )
+
+
+def test_docs_page_metadata_sources():
+ """Only the files that become documentation pages must carry page metadata.
+
+ These select which files are checked, by reproducing what apidoc and
+ sphinx-gallery are configured to build pages from.
+ """
+ check_docs_page_metadata = _load_hook("check_docs_page_metadata")
+ repo_root = Path(IRIS_REPO_DIRPATH)
+
+ library = repo_root / "lib"
+ modules = {
+ path.relative_to(library).as_posix()
+ for path in check_docs_page_metadata.library_modules(library)
+ }
+ assert "iris/cube.py" in modules
+ assert "iris/__init__.py" in modules
+ # Neither of these gets an API page, so neither needs page metadata.
+ assert "iris/_lazy_data.py" not in modules
+ assert not any(module.startswith("iris/tests/") for module in modules)
+
+ gallery_code = repo_root / "docs" / "gallery_code"
+ scripts = {
+ path.relative_to(gallery_code).as_posix()
+ for path in check_docs_page_metadata.gallery_scripts(gallery_code)
+ }
+ assert "general/plot_coriolis.py" in scripts
diff --git a/lib/iris/tests/test_io_init.py b/lib/iris/tests/test_io_init.py
index 737926578e..45da6ece3c 100644
--- a/lib/iris/tests/test_io_init.py
+++ b/lib/iris/tests/test_io_init.py
@@ -84,6 +84,58 @@ def test_decode_uri__path(self, uri, expected):
assert iris.io.decode_uri(Path(uri)) == expected
+class TestCategoriseUri:
+ """`_categorise_uri` groups URIs by the handler that will take them.
+
+ `iris.loading._generate_cubes` and
+ `iris.fileformats.netcdf.ugrid_load.load_meshes` both group on the
+ category, so the two must agree about what an NCZarr target is.
+ """
+
+ @pytest.mark.parametrize(
+ ("uri", "expected"),
+ [
+ (
+ "/data/forecast.nc",
+ ("file", "file", "/data/forecast.nc"),
+ ),
+ (
+ "/data/forecast.zarr",
+ ("file", "file", "/data/forecast.zarr"),
+ ),
+ (
+ "s3://bucket/forecast.zarr",
+ ("s3", "s3", "//bucket/forecast.zarr"),
+ ),
+ (
+ "https://example.com/forecast.nc",
+ ("https", "https", "//example.com/forecast.nc"),
+ ),
+ (
+ "file:///data/forecast.zarr#mode=nczarr,file",
+ ("nczarr", "file", "///data/forecast.zarr#mode=nczarr,file"),
+ ),
+ ],
+ )
+ def test_categorise_uri(self, uri, expected):
+ assert iris.io._categorise_uri(uri) == expected
+
+ def test_an_nczarr_fragment_outranks_its_scheme(self):
+ # The whole reason the category is not simply the scheme: these two
+ # share one and need different handlers.
+ plain = iris.io._categorise_uri("file:///data/forecast.zarr")
+ nczarr = iris.io._categorise_uri("file:///data/forecast.zarr#mode=nczarr,file")
+ assert plain[1] == nczarr[1] == "file"
+ assert plain[0] != nczarr[0]
+
+ def test_the_fragment_is_folded_back_into_the_part(self):
+ # A handler is handed a URI to reopen, so it needs the fragment.
+ _category, _scheme, part = iris.io._categorise_uri(
+ "file:///data/forecast.zarr#mode=nczarr,file"
+ )
+ assert part.endswith("#mode=nczarr,file")
+
+
class TestFileFormatPicker:
def test_known_formats(self, request):
_shared_utils.assert_string(
diff --git a/lib/iris/tests/test_netcdf.py b/lib/iris/tests/test_netcdf.py
index 2605343c4d..3e743db2cf 100644
--- a/lib/iris/tests/test_netcdf.py
+++ b/lib/iris/tests/test_netcdf.py
@@ -24,6 +24,7 @@
from iris.tests import _shared_utils
import iris.tests.stock as stock
from iris.tests.stock.netcdf import ncgen_from_cdl
+from iris.tests.unit.fileformats.nc_load_rules.helpers import CFVariableStandIn
import iris.util
from iris.warnings import IrisCfSaveWarning
@@ -466,16 +467,13 @@ def test_units(self, request):
class TestNetCDFCRS:
@pytest.fixture(autouse=True)
def _setup(self):
- class Var:
- pass
-
- self.grid = Var()
+ self.grid = CFVariableStandIn()
def test_lat_lon_major_minor(self):
major = 63781370
minor = 63567523
- self.grid.semi_major_axis = major
- self.grid.semi_minor_axis = minor
+ self.grid.attributes["semi_major_axis"] = major
+ self.grid.attributes["semi_minor_axis"] = minor
# NB 'build_coordinate_system' has an extra (unused) 'engine' arg, just
# so that it has the same signature as other coord builder routines.
engine = None
@@ -484,7 +482,7 @@ def test_lat_lon_major_minor(self):
def test_lat_lon_earth_radius(self):
earth_radius = 63700000
- self.grid.earth_radius = earth_radius
+ self.grid.attributes["earth_radius"] = earth_radius
# NB 'build_coordinate_system' has an extra (unused) 'engine' arg, just
# so that it has the same signature as other coord builder routines.
engine = None
diff --git a/lib/iris/tests/unit/aux_factory/test_AuxCoordFactory.py b/lib/iris/tests/unit/aux_factory/test_AuxCoordFactory.py
index faf57d0e65..cc9f5eef4c 100644
--- a/lib/iris/tests/unit/aux_factory/test_AuxCoordFactory.py
+++ b/lib/iris/tests/unit/aux_factory/test_AuxCoordFactory.py
@@ -144,7 +144,7 @@ class Test_lazy_aux_coords:
def sample_cube(self, mocker):
path = get_data_path(["NetCDF", "testing", "small_theta_colpex.nc"])
# While loading, "turn off" loading small variables as real data.
- mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 0)
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
cube = iris.load_cube(path, "air_potential_temperature")
return cube
diff --git a/lib/iris/tests/unit/cube/test_CubeAttrsDict.py b/lib/iris/tests/unit/cube/test_CubeAttrsDict.py
index 6864780dee..7cb2303246 100644
--- a/lib/iris/tests/unit/cube/test_CubeAttrsDict.py
+++ b/lib/iris/tests/unit/cube/test_CubeAttrsDict.py
@@ -11,7 +11,7 @@
from iris.common.mixin import LimitedAttributeDict
from iris.cube import CubeAttrsDict
-from iris.fileformats.netcdf.saver import _CF_GLOBAL_ATTRS
+from iris.fileformats.cf.saver import _CF_GLOBAL_ATTRS
@pytest.fixture
diff --git a/lib/iris/tests/unit/fileformats/cf/dataset/__init__.py b/lib/iris/tests/unit/fileformats/cf/dataset/__init__.py
new file mode 100644
index 0000000000..f50199ed98
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/dataset/__init__.py
@@ -0,0 +1,5 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.cf.dataset`."""
diff --git a/lib/iris/tests/unit/fileformats/cf/dataset/contract.py b/lib/iris/tests/unit/fileformats/cf/dataset/contract.py
new file mode 100644
index 0000000000..ac797244a5
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/dataset/contract.py
@@ -0,0 +1,317 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""One test body, run against every :class:`CFDataset` implementation.
+
+Subclass :class:`CFDatasetReadContract` in a module named for the
+implementation, supply the fixtures it declares, and every test here runs
+against it. Both classes deliberately lack a ``Test`` prefix, so pytest does
+not collect them where they are written, only where they are subclassed.
+
+The split is read from write because an implementation may arrive in two
+pieces: the Zarr loader can answer every question in the read half before a
+Zarr saver exists. Skipping half of one class instead would hide an
+incomplete implementation behind a skip, and the point of a shared body is
+that divergence fails loudly.
+
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md`` section 6: the
+interface is only worth having if both sides of it agree, and agreement is
+cheapest to check by asking the same questions twice.
+
+Deliberately not exercised here: :meth:`CFDatasetVariable.deprecated_netcdf_member`.
+It is an explicitly netCDF4-specific compatibility escape hatch, so asserting
+either its presence or its absence in a shared body would itself be
+backend-specific - the omission is intentional, not a gap.
+"""
+
+from collections.abc import Mapping
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+
+#: The canonical dataset every implementation is checked against.
+CONTRACT_DIMENSIONS = {"y": 3, "x": 2}
+CONTRACT_DATA = np.arange(6, dtype="f8").reshape(3, 2)
+CONTRACT_VARIABLE_ATTRS = {"units": "K", "long_name": "surface temperature"}
+CONTRACT_GLOBALS = {"Conventions": "CF-1.7", "title": "contract"}
+
+
+class CFDatasetReadContract:
+ """What every CFDataset implementation must answer, whatever it reads from."""
+
+ #: The key this backend files its chunk shape under in `storage_settings`.
+ #:
+ #: `storage_settings` keys are each backend's own spelling - netCDF4 says
+ #: `chunksizes`, Zarr says `chunks` - so the contract cannot name one. It
+ #: can still insist that `chunking` and `storage_settings` agree, which is
+ #: what the subclass naming its key buys.
+ chunking_key: str
+
+ @pytest.fixture
+ def readable(self):
+ """Return an open read-mode dataset holding what populate() writes."""
+ raise NotImplementedError("supply a 'readable' fixture")
+
+ # -- The dataset itself ------------------------------------------------
+
+ def test_is_a_cf_dataset(self, readable):
+ assert isinstance(readable, CFDataset)
+
+ def test_location_is_a_non_empty_string(self, readable):
+ assert isinstance(readable.location, str)
+ assert readable.location
+
+ def test_mode_is_a_read_mode(self, readable):
+ assert readable.mode in ("r", "r+", "a")
+
+ def test_starts_open(self, readable):
+ assert readable.closed is False
+
+ def test_close_then_closed(self, readable):
+ readable.close()
+ assert readable.closed is True
+
+ def test_close_is_idempotent(self, readable):
+ readable.close()
+ readable.close()
+ assert readable.closed is True
+
+ def test_context_manager_returns_self_and_closes(self, readable):
+ with readable as entered:
+ assert entered is readable
+ assert readable.closed is True
+
+ def test_dimensions(self, readable):
+ assert dict(readable.dimensions) == CONTRACT_DIMENSIONS
+
+ def test_global_attributes(self, readable):
+ assert dict(readable.attributes) == CONTRACT_GLOBALS
+
+ def test_variable_names(self, readable):
+ assert sorted(readable.variables) == ["air_temperature", "grid"]
+
+ def test_format_is_a_non_empty_string(self, readable):
+ assert isinstance(readable.format, str)
+ assert readable.format
+
+ def test_supports_the_contract_dtype(self, readable):
+ # Whatever a store is, it holds the dtype the contract writes into it.
+ assert readable.supports_dtype(CONTRACT_DATA.dtype) is True
+
+ def test_storage_settings_is_a_readable_mapping(self, readable):
+ # Its keys are the backend's own spelling and generic code must never
+ # branch on one, but it must be possible to read them: otherwise a
+ # caller that wants to know how a store is laid out has to reach
+ # through to the netCDF4 or zarr object, which is the abstraction
+ # leaking at the point it exists to hold.
+ assert isinstance(readable.storage_settings, Mapping)
+ dict(readable.storage_settings)
+
+ # -- A variable --------------------------------------------------------
+
+ @pytest.fixture
+ def variable(self, readable):
+ return readable.variables["air_temperature"]
+
+ @pytest.fixture
+ def scalar(self, readable):
+ return readable.variables["grid"]
+
+ def test_variable_is_a_cf_dataset_variable(self, variable):
+ assert isinstance(variable, CFDatasetVariable)
+
+ def test_variable_name(self, variable):
+ assert variable.name == "air_temperature"
+
+ def test_variable_location_matches_its_dataset(self, variable, readable):
+ assert variable.location == readable.location
+
+ def test_variable_format_matches_its_dataset(self, variable, readable):
+ assert variable.format == readable.format
+
+ def test_variable_storage_settings_reports_the_chunking(self, variable):
+ storage_settings = variable.storage_settings
+ assert isinstance(storage_settings, Mapping)
+ assert variable.chunking == storage_settings[self.chunking_key]
+
+ def test_variable_dimensions(self, variable):
+ assert variable.dimensions == tuple(CONTRACT_DIMENSIONS)
+
+ def test_variable_shape(self, variable):
+ assert variable.shape == tuple(CONTRACT_DIMENSIONS.values())
+
+ def test_variable_dtype(self, variable):
+ assert variable.dtype == np.dtype("f8")
+
+ def test_variable_size(self, variable):
+ assert variable.size == CONTRACT_DATA.size
+
+ def test_variable_ndim(self, variable):
+ assert variable.ndim == CONTRACT_DATA.ndim
+
+ def test_variable_len(self, variable):
+ assert len(variable) == CONTRACT_DATA.shape[0]
+
+ def test_variable_data(self, variable):
+ np.testing.assert_array_equal(variable[:], CONTRACT_DATA)
+
+ def test_variable_data_indexed(self, variable):
+ np.testing.assert_array_equal(variable[1], CONTRACT_DATA[1])
+
+ def test_variable_attributes(self, variable):
+ for name, value in CONTRACT_VARIABLE_ATTRS.items():
+ assert variable.attributes[name] == value
+
+ def test_variable_attributes_omit_unset_names(self, variable):
+ assert "nonesuch" not in variable.attributes
+ with pytest.raises(KeyError):
+ variable.attributes["nonesuch"]
+
+ def test_variable_fill_value(self, variable):
+ assert variable.fill_value == -1.0
+
+ def test_variable_chunking_is_none_or_a_shape(self, variable):
+ chunking = variable.chunking
+ # np.integer alongside int: netCDF4 answers plain ints, but a Zarr
+ # chunk shape is plausibly numpy.int64, and both are equally a shape.
+ assert chunking is None or (
+ isinstance(chunking, tuple)
+ and len(chunking) == len(variable.shape)
+ and all(isinstance(size, int | np.integer) for size in chunking)
+ )
+
+ def test_read_data_returns_the_data_for_a_small_variable(self, variable):
+ # CONTRACT_DATA is small, so this exercises the read-whole path only,
+ # and the policy below is never consulted. Reading a variable large
+ # enough to come back lazy is a per-format test, not a contract one:
+ # see test_NetCDFDatasetVariable.py.
+ result = variable.read_data(lambda: (None, None))
+ np.testing.assert_array_equal(np.asarray(result), CONTRACT_DATA)
+
+ def test_read_data_leaves_the_policy_alone_for_a_small_variable(self, variable):
+ # The chunking policy raises under CHUNK_CONTROL.from_file() for an
+ # unchunked variable. A variable small enough to read whole must reach
+ # its data without ever asking, or `from_file` would start rejecting
+ # small cubes that load today.
+ def policy():
+ raise AssertionError("chunking policy consulted for a small variable")
+
+ variable.read_data(policy)
+
+ def test_scalar_variable(self, scalar):
+ assert scalar.dimensions == ()
+ assert scalar.shape == ()
+ with pytest.raises(TypeError, match="unsized"):
+ len(scalar)
+
+
+class CFDatasetWriteContract:
+ """What every CFDataset implementation must do, whatever it stores into."""
+
+ @pytest.fixture
+ def writable(self):
+ """Return an open, empty, write-mode dataset."""
+ raise NotImplementedError("supply a 'writable' fixture")
+
+ @pytest.fixture
+ def reopen(self):
+ """Return a callable giving a fresh read-mode dataset over 'writable'."""
+ raise NotImplementedError("supply a 'reopen' fixture")
+
+ def populate(self, dataset):
+ """Fill an empty writable dataset with the canonical contents.
+
+ Uses only the CFDataset API, so this doubles as the write half of the
+ contract: an implementation that cannot build this cannot be used to save.
+ """
+ for name, size in CONTRACT_DIMENSIONS.items():
+ dataset.create_dimension(name, size)
+ for name, value in CONTRACT_GLOBALS.items():
+ dataset.attributes[name] = value
+
+ variable = dataset.create_variable(
+ "air_temperature",
+ np.dtype("f8"),
+ tuple(CONTRACT_DIMENSIONS),
+ fill_value=-1.0,
+ )
+ for name, value in CONTRACT_VARIABLE_ATTRS.items():
+ variable.attributes[name] = value
+ variable[:] = CONTRACT_DATA
+
+ # A dimensionless variable: what a grid mapping is, and the one shape a
+ # CFDatasetVariable must handle without a leading dimension.
+ scalar = dataset.create_variable("grid", np.dtype("i4"))
+ scalar.attributes["grid_mapping_name"] = "latitude_longitude"
+ scalar[()] = 0
+ return dataset
+
+ def test_populate_then_read_back(self, writable, reopen):
+ self.populate(writable)
+ writable.close()
+ with reopen() as reader:
+ np.testing.assert_array_equal(
+ reader.variables["air_temperature"][:], CONTRACT_DATA
+ )
+ assert dict(reader.dimensions) == CONTRACT_DIMENSIONS
+ assert dict(reader.attributes) == CONTRACT_GLOBALS
+
+ def test_created_variable_is_visible_on_the_dataset(self, writable):
+ writable.create_dimension("x", 2)
+ created = writable.create_variable("thing", np.dtype("f4"), ("x",))
+ assert writable.variables["thing"].name == created.name
+
+ def test_create_variable_without_dimensions(self, writable):
+ # saver.py creates a grid variable with only a name and a dtype.
+ variable = writable.create_variable("grid", np.dtype("i4"))
+ assert variable.dimensions == ()
+
+ def test_unlimited_dimension_is_supported_or_refused(self, writable):
+ # The one member whose docstring sanctions two answers: a store with
+ # no unlimited concept - Zarr - refuses rather than inventing one.
+ # Either answer conforms; what does not conform is a third, such as
+ # quietly creating a fixed-length dimension instead.
+ try:
+ writable.create_dimension("t", None)
+ except NotImplementedError:
+ return
+ assert "t" in writable.dimensions
+
+ def test_setitem_then_getitem(self, writable):
+ writable.create_dimension("x", 3)
+ variable = writable.create_variable("thing", np.dtype("f4"), ("x",))
+ variable[:] = [1.0, 2.0, 3.0]
+ np.testing.assert_array_equal(variable[:], [1.0, 2.0, 3.0])
+
+ def test_attribute_write_through(self, writable, reopen):
+ self.populate(writable)
+ writable.variables["air_temperature"].attributes["comment"] = "added"
+ writable.close()
+ with reopen() as reader:
+ assert reader.variables["air_temperature"].attributes["comment"] == "added"
+
+ def test_write_handle_works_after_close(self, writable, reopen):
+ # Not asserted: that the handle survives a pickle round trip. Under
+ # the default threaded scheduler its lock is an unpicklable
+ # threading.Lock, and only the distributed scheduler - which supplies
+ # a picklable one - ever ships a handle to another process. Using the
+ # handle after close is the portable approximation.
+ self.populate(writable)
+ handle = writable.variables["air_temperature"].write_handle()
+ writable.close()
+
+ payload = CONTRACT_DATA * 10
+ handle[:] = payload
+ with reopen() as reader:
+ np.testing.assert_array_equal(
+ reader.variables["air_temperature"][:], payload
+ )
+
+ def test_sync_and_finalise_are_callable(self, writable):
+ self.populate(writable)
+ assert writable.sync() is None
+ assert writable.finalise() is None
+ assert writable.closed is False
diff --git a/lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py b/lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py
new file mode 100644
index 0000000000..bbff315fca
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/dataset/test_CFDataset.py
@@ -0,0 +1,167 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.dataset.CFDataset` and friends."""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+
+
+class MinimalVariable(CFDatasetVariable):
+ """The smallest thing that satisfies CFDatasetVariable."""
+
+ name = "air_temperature"
+ location = "<memory>"
+ format = "MINIMAL"
+ dimensions = ("time", "lat")
+ shape = (3, 4)
+ dtype = np.dtype("f4")
+ size = 12
+ fill_value = None
+ chunking = None
+ storage_settings: dict = {}
+ attributes: dict = {}
+
+ def __getitem__(self, keys):
+ return np.zeros(self.shape, dtype=self.dtype)[keys]
+
+ def __setitem__(self, keys, values):
+ raise NotImplementedError
+
+ def read_data(self, chunking_policy):
+ return self[:]
+
+ def write_handle(self):
+ return self
+
+
+class MinimalDataset(CFDataset):
+ """The smallest thing that satisfies CFDataset."""
+
+ location = "<memory>"
+ mode = "r"
+ format = "MINIMAL"
+ closed = False
+ variables: dict = {}
+ dimensions: dict = {}
+ attributes: dict = {}
+ storage_settings: dict = {}
+
+ def __init__(self):
+ self.closes = 0
+
+ def create_dimension(self, name, size):
+ raise NotImplementedError
+
+ def create_variable(self, name, dtype, dimensions=(), *, fill_value=None, **kw):
+ raise NotImplementedError
+
+ def supports_dtype(self, dtype):
+ return True
+
+ def sync(self):
+ pass
+
+ def finalise(self):
+ pass
+
+ def close(self):
+ self.closes += 1
+
+
+class TestAbstractness:
+ def test_variable_cannot_be_instantiated(self):
+ with pytest.raises(TypeError, match="abstract"):
+ CFDatasetVariable()
+
+ def test_dataset_cannot_be_instantiated(self):
+ with pytest.raises(TypeError, match="abstract"):
+ CFDataset()
+
+ @pytest.mark.parametrize(
+ "name",
+ [
+ "name",
+ "location",
+ "format",
+ "dimensions",
+ "shape",
+ "dtype",
+ "size",
+ "fill_value",
+ "chunking",
+ "storage_settings",
+ "attributes",
+ "ndim",
+ "__getitem__",
+ "__setitem__",
+ "read_data",
+ "write_handle",
+ ],
+ )
+ def test_variable_declares_member(self, name):
+ assert name in CFDatasetVariable.__abstractmethods__ or hasattr(
+ CFDatasetVariable, name
+ )
+
+ @pytest.mark.parametrize(
+ "name",
+ [
+ "location",
+ "mode",
+ "format",
+ "closed",
+ "variables",
+ "dimensions",
+ "attributes",
+ "storage_settings",
+ "create_dimension",
+ "create_variable",
+ "supports_dtype",
+ "sync",
+ "finalise",
+ "close",
+ ],
+ )
+ def test_dataset_declares_member(self, name):
+ assert name in CFDataset.__abstractmethods__ or hasattr(CFDataset, name)
+
+
+class TestConcreteDefaults:
+ def test_ndim(self):
+ assert MinimalVariable().ndim == 2
+
+ def test_len_is_the_leading_dimension(self):
+ assert len(MinimalVariable()) == 3
+
+ def test_len_of_a_scalar_matches_netcdf4(self):
+ # netCDF4.Variable raises TypeError, and CFVariable.__len__ forwards
+ # to it, so anything catching that keeps working.
+ class Scalar(MinimalVariable):
+ shape = ()
+
+ with pytest.raises(TypeError, match="unsized"):
+ len(Scalar())
+
+ def test_deprecated_netcdf_member_raises_attribute_error(self):
+ # A store with no backing netCDF4 object - Zarr, say - has no fallback
+ # to offer, so the compatibility route simply does not apply.
+ with pytest.raises(AttributeError, match="getncattr"):
+ MinimalVariable().deprecated_netcdf_member("getncattr")
+
+ def test_context_manager_closes(self):
+ dataset = MinimalDataset()
+ with dataset as entered:
+ assert entered is dataset
+ assert dataset.closes == 0
+ assert dataset.closes == 1
+
+ def test_context_manager_closes_on_exception(self):
+ dataset = MinimalDataset()
+ with pytest.raises(ValueError, match="boom"):
+ with dataset:
+ raise ValueError("boom")
+ assert dataset.closes == 1
diff --git a/lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py b/lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py
new file mode 100644
index 0000000000..ab5abc02a7
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py
@@ -0,0 +1,123 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.dataset.TrackedAttributes`."""
+
+import pytest
+
+from iris.fileformats.cf.dataset import TrackedAttributes
+
+# The set CFVariable seeds tracking with: attributes netCDF4 handles itself,
+# and which are therefore "used" before anyone reads them.
+IGNORED = ("_FillValue", "add_offset", "missing_value", "scale_factor")
+
+
+@pytest.fixture
+def source():
+ return {"units": "K", "standard_name": "air_temperature", "_FillValue": -999}
+
+
+@pytest.fixture
+def tracked(source):
+ return TrackedAttributes(source, ignored=IGNORED)
+
+
+class TestWhatRecordsARead:
+ def test_starts_with_only_the_present_ignored_names(self, tracked):
+ # "scale_factor" is ignored but absent, so it is not seeded.
+ assert tracked.read == frozenset(["_FillValue"])
+ assert tracked.unread == frozenset(["units", "standard_name"])
+
+ def test_getitem_records(self, tracked):
+ assert tracked["units"] == "K"
+ assert tracked.read == frozenset(["_FillValue", "units"])
+ assert tracked.unread == frozenset(["standard_name"])
+
+ def test_get_records(self, tracked):
+ assert tracked.get("units") == "K"
+ assert "units" in tracked.read
+
+ def test_contains_records_a_hit(self, tracked):
+ # hasattr(cf_var, name) goes through __getattr__ and marks the
+ # attribute used; the mapping form has to do the same.
+ assert "units" in tracked
+ assert "units" in tracked.read
+
+ def test_contains_does_not_record_a_miss(self, tracked):
+ assert "nonesuch" not in tracked
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_getitem_of_a_missing_key_raises_before_recording(self, tracked):
+ with pytest.raises(KeyError, match="nonesuch"):
+ tracked["nonesuch"]
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_get_of_a_missing_key_records_nothing(self, tracked):
+ assert tracked.get("nonesuch") is None
+ assert tracked.read == frozenset(["_FillValue"])
+
+
+class TestWhatDoesNotRecordARead:
+ def test_untracked_getitem(self, tracked):
+ assert tracked.untracked["units"] == "K"
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_untracked_contains(self, tracked):
+ # The form helpers.py needs for its deliberately-unmarked flag probe.
+ assert "units" in tracked.untracked
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_untracked_is_read_only(self, tracked):
+ with pytest.raises(TypeError):
+ tracked.untracked["units"] = "m"
+
+ def test_iteration(self, tracked):
+ assert sorted(tracked) == ["_FillValue", "standard_name", "units"]
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_keys_values_items(self, tracked):
+ assert sorted(tracked.keys()) == ["_FillValue", "standard_name", "units"]
+ assert sorted(tracked.values(), key=str) == [-999, "K", "air_temperature"]
+ assert dict(tracked.items())["units"] == "K"
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_len(self, tracked):
+ assert len(tracked) == 3
+ assert tracked.read == frozenset(["_FillValue"])
+
+
+class TestMutation:
+ def test_setitem_writes_through_and_does_not_record(self, tracked, source):
+ tracked["comment"] = "hello"
+ assert source["comment"] == "hello"
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_delitem_writes_through_and_forgets_the_read(self, tracked, source):
+ _ = tracked["units"]
+ del tracked["units"]
+ assert "units" not in source
+ assert tracked.read == frozenset(["_FillValue"])
+
+
+class TestReset:
+ def test_reset_returns_to_the_present_ignored_names(self, tracked):
+ _ = tracked["units"]
+ _ = tracked["standard_name"]
+ assert tracked.read == frozenset(["_FillValue", "units", "standard_name"])
+ tracked.reset()
+ assert tracked.read == frozenset(["_FillValue"])
+
+ def test_reset_sees_attributes_added_since_construction(self, tracked):
+ tracked["scale_factor"] = 2.0
+ tracked.reset()
+ assert tracked.read == frozenset(["_FillValue", "scale_factor"])
+
+
+class TestEmptySource:
+ def test_a_variable_with_no_attributes_is_not_an_error(self):
+ tracked = TrackedAttributes({}, ignored=IGNORED)
+ assert tracked.read == frozenset()
+ assert tracked.unread == frozenset()
+ assert len(tracked) == 0
+ assert "units" not in tracked
diff --git a/lib/iris/tests/unit/fileformats/cf/identify_mixins.py b/lib/iris/tests/unit/fileformats/cf/identify_mixins.py
new file mode 100644
index 0000000000..e23476e78a
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/identify_mixins.py
@@ -0,0 +1,414 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Shared test mixin classes for CF variable identify() behaviour.
+
+This module provides reusable pytest test classes which concrete test modules
+can subclass and configure via class attributes.
+"""
+
+from abc import ABC
+import warnings
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf import CFVariable
+from iris.fileformats.cf._variables import _NCZARR_SCALAR_DIMENSION
+import iris.warnings
+
+
+class _NetCDFVar:
+ """Stub NetCDF variable for identify() tests."""
+
+ ATTRS_NOT_RETURN: list[str] = ["name", "dtype"]
+
+ def __init__(self, name, dtype=int):
+ self.name = name
+ self.dtype = np.dtype(dtype)
+
+ def ncattrs(self):
+ return [
+ attr
+ for attr in self.__dict__
+ if not attr.startswith("_") and attr not in self.ATTRS_NOT_RETURN
+ ]
+
+ @property
+ def attributes(self):
+ """The CF attributes, as a CFDatasetVariable presents them."""
+ return {name: getattr(self, name) for name in self.ncattrs()}
+
+
+#: The size each dimension name has stood for, the first time a stub used it.
+#: `spans()` now checks `(name, size)` pairs, so two stubs that name a
+#: dimension identically must also agree on its length, or a test exercising
+#: that agreement would be testing nothing. Keyed by name and never cleared,
+#: so the same name means the same size for every stub built in the process -
+#: regardless of which test runs first. Pinned for
+#: ``_NCZARR_SCALAR_DIMENSION``, matching the size NCZarr itself gives it;
+#: every other name claims the next unused integer, once, the first time
+#: :func:`_stub_dimension_size` is asked for it.
+_STUB_DIMENSION_SIZES: dict[str, int] = {_NCZARR_SCALAR_DIMENSION: 1}
+
+
+def _stub_dimension_size(name: str) -> int:
+ """Return `name`'s fixed size, claiming the next unused one if new."""
+ if name not in _STUB_DIMENSION_SIZES:
+ _STUB_DIMENSION_SIZES[name] = len(_STUB_DIMENSION_SIZES) + 1
+ return _STUB_DIMENSION_SIZES[name]
+
+
+class _NetCDFVarWithDimensions(_NetCDFVar):
+ """Stub NetCDF variable with dimensions for spans() tests."""
+
+ ATTRS_NOT_RETURN = _NetCDFVar.ATTRS_NOT_RETURN + ["dimensions"]
+
+ def __init__(self, name, dimensions, dtype=int):
+ super().__init__(name, dtype=dtype)
+ self.dimensions = dimensions
+
+ @property
+ def shape(self) -> tuple:
+ """One size per dimension, agreeing with every other stub naming it.
+
+ A property, not an instance attribute set in ``__init__``: `ncattrs`
+ reads `self.__dict__`, so a stored value would leak onto
+ `attributes` exactly as an unguarded `dimensions` would - a property
+ is never in `self.__dict__` and cannot leak that way.
+ """
+ return tuple(_stub_dimension_size(name) for name in self.dimensions)
+
+
+def assert_warning_gated(operation, warning_category, warning_regex):
+ """Shared convenience for warning asserts."""
+ with pytest.warns(warning_category, match=warning_regex):
+ operation(warn=True)
+
+ try:
+ with pytest.warns(warning_category, match=warning_regex):
+ operation(warn=False)
+ except pytest.fail.Exception:
+ pass
+ else:
+ pytest.fail(
+ f"Operation {operation.__name__} raised {warning_category.__name__} "
+ "when warn=False"
+ )
+
+
+class SpansMixin(ABC):
+ """Shared spans() tests for CF variable wrappers."""
+
+ __test__ = False
+
+ CF_CLASS: type[CFVariable]
+
+ def _make_cf_var(self, name, dimensions):
+ stub = _NetCDFVarWithDimensions(name, dimensions, dtype=np.bytes_)
+ return self.CF_CLASS(name, stub)
+
+ def test_empty_dimensions_spans(self):
+ """Scalar source variable always spans the target."""
+ cf_source = self._make_cf_var("source_var", ())
+ cf_target = self._make_cf_var("target_var", ("x", "y"))
+ assert cf_source.spans(cf_target)
+
+ def test_source_trailing_subset_spans(self):
+ """source[:-1] is a subset of target dimensions => spans."""
+ cf_source = self._make_cf_var("source_var", ("x", "y", "extra"))
+ cf_target = self._make_cf_var("target_var", ("x", "y"))
+ assert cf_source.spans(cf_target)
+
+ def test_source_leading_subset_spans(self):
+ """source[1:] is a subset of target dimensions => spans."""
+ cf_source = self._make_cf_var("source_var", ("extra", "x", "y"))
+ cf_target = self._make_cf_var("target_var", ("x", "y"))
+ assert cf_source.spans(cf_target)
+
+ def test_non_spanning(self):
+ """Dimensions that don't fit either slice => does not span."""
+ cf_source = self._make_cf_var("source_var", ("x", "b", "c"))
+ cf_target = self._make_cf_var("target_var", ("x", "y"))
+ assert not cf_source.spans(cf_target)
+
+ def test_nczarr_scalar_dimension_spans(self):
+ """An NCZarr scalar source variable always spans the target.
+
+ NCZarr spells a zero-dimensional variable as one dimension named
+ _scalar_, so this is the same case as test_empty_dimensions_spans
+ arriving from a different writer.
+ """
+ cf_source = self._make_cf_var("source_var", (_NCZARR_SCALAR_DIMENSION,))
+ cf_target = self._make_cf_var("target_var", ("x", "y"))
+ assert cf_source.spans(cf_target)
+
+ def test_nczarr_scalar_is_not_a_dimension_name_to_match_on(self):
+ """_scalar_ marks a scalar; it is not a dimension two variables share.
+
+ A source that is not scalar has to justify itself dimension by
+ dimension, whether or not _scalar_ is among its names.
+ """
+ cf_source = self._make_cf_var("source_var", ("a", "b", "extra"))
+ cf_target = self._make_cf_var("target_var", (_NCZARR_SCALAR_DIMENSION,))
+ assert not cf_source.spans(cf_target)
+
+
+class IdentifyByAttributeMixin(ABC):
+ """Parent class for CF variable identify() tests.
+
+ Supports both single-attribute and multi-attribute identity patterns.
+ Subclasses should define CF_IDENTITIES as a list of attribute names.
+ Use CF_IDENTITIES[0] as the primary identity in tests.
+ """
+
+ __test__ = False
+
+ CF_CLASS: type[CFVariable]
+ CF_IDENTITIES: list[str] # Always a list; use [0] as primary identity
+ MISSING_WARN_REGEX: str
+ SUBJECT_DTYPE_DEFAULT = None
+ # Turn this off for identities that support only a single reference
+ IDENTITY_SUPPORTS_MULTIPLE_REFS = True
+
+ @classmethod
+ def _make_subject(cls, name):
+ if cls.SUBJECT_DTYPE_DEFAULT is None:
+ return _NetCDFVar(name)
+ else:
+ return _NetCDFVar(name, dtype=cls.SUBJECT_DTYPE_DEFAULT)
+
+ # Common test methods (work for both single and multi-identity classes)
+
+ def test_one_ref(self):
+ subject_name = "ref_subject"
+ ref_subject = self._make_subject(subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], subject_name)
+ vars_all = {
+ subject_name: ref_subject,
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ expected = {subject_name: self.CF_CLASS(subject_name, ref_subject)}
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == expected
+
+ def test_two_refs(self):
+ # Check that a single "source" var may refer to multiple "subject" vars.
+ subject_names = ("ref_subject_1", "ref_subject_2")
+ ref_subject_vars = {name: self._make_subject(name) for name in subject_names}
+
+ if self.IDENTITY_SUPPORTS_MULTIPLE_REFS:
+ # make a single source reference multiple subjects.
+ ref_source_name = "ref_source"
+ ref_source_var = _NetCDFVar(ref_source_name)
+ setattr(ref_source_var, self.CF_IDENTITIES[0], " ".join(subject_names))
+ ref_source_vars = {ref_source_name: ref_source_var}
+ else:
+ # make two source vars, each referencing a different subject.
+ ref_source_vars = {
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
+ }
+ for ix, var in enumerate(ref_source_vars.values()):
+ setattr(var, self.CF_IDENTITIES[0], subject_names[ix])
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ **ref_subject_vars,
+ **ref_source_vars,
+ }
+
+ expected = {
+ name: self.CF_CLASS(name, var) for name, var in ref_subject_vars.items()
+ }
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == expected
+
+ def test_duplicate_refs(self):
+ subject_name = "ref_subject"
+ ref_subject = self._make_subject(subject_name)
+ ref_source_vars = {
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
+ }
+ for var in ref_source_vars.values():
+ setattr(var, self.CF_IDENTITIES[0], subject_name)
+ vars_all = {
+ subject_name: ref_subject,
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ **ref_source_vars,
+ }
+
+ expected = {subject_name: self.CF_CLASS(subject_name, ref_subject)}
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == expected
+
+ def test_ignore(self):
+ subject_names = ("ref_subject_1", "ref_subject_2")
+ ref_subject_vars = {name: self._make_subject(name) for name in subject_names}
+
+ ref_source_vars = {
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
+ }
+ for ix, var in enumerate(ref_source_vars.values()):
+ setattr(var, self.CF_IDENTITIES[0], subject_names[ix])
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ **ref_subject_vars,
+ **ref_source_vars,
+ }
+
+ expected_name = subject_names[0]
+ expected = {
+ expected_name: self.CF_CLASS(expected_name, ref_subject_vars[expected_name])
+ }
+ result = self.CF_CLASS.identify(vars_all, ignore=subject_names[1])
+ assert result == expected
+
+ def test_target(self):
+ subject_names = ("ref_subject_1", "ref_subject_2")
+ ref_subject_vars = {name: self._make_subject(name) for name in subject_names}
+
+ source_names = ("ref_source_1", "ref_source_2")
+ ref_source_vars = {name: _NetCDFVar(name) for name in source_names}
+ for ix, var in enumerate(ref_source_vars.values()):
+ setattr(var, self.CF_IDENTITIES[0], subject_names[ix])
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ **ref_subject_vars,
+ **ref_source_vars,
+ }
+
+ expected_name = subject_names[0]
+ expected = {
+ expected_name: self.CF_CLASS(expected_name, ref_subject_vars[expected_name])
+ }
+ result = self.CF_CLASS.identify(vars_all, target=source_names[0])
+ assert result == expected
+
+ def test_target_unknown_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Cannot identify unknown target CF-netCDF variable 'unknown'"
+ with pytest.raises(ValueError, match=message):
+ self.CF_CLASS.identify(vars_all, target="unknown")
+
+ def test_target_wrong_type_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Expect a target CF-netCDF variable name"
+ with pytest.raises(TypeError, match=message):
+ self.CF_CLASS.identify(vars_all, target=object())
+
+ def test_warn(self):
+ subject_name = "ref_subject"
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], subject_name)
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ def operation(warn: bool):
+ warnings.warn(
+ "emit at least 1 warning",
+ category=iris.warnings.IrisUserWarning,
+ )
+ result = self.CF_CLASS.identify(vars_all, warn=warn)
+ # For multi-identity tests with string-typed vars, result may be empty
+ # For single-identity tests, result may be non-empty
+
+ assert_warning_gated(
+ operation,
+ iris.warnings.IrisCfMissingVarWarning,
+ self.MISSING_WARN_REGEX.format(subject=subject_name),
+ )
+
+
+class IdentifyByAttributeListMixin(IdentifyByAttributeMixin):
+ """Parent class for CF variables identified by multiple possible attributes.
+
+ This class adds test methods specific to cases where CF_IDENTITIES contains
+ multiple possible attribute names (typical for UGRID variables).
+ Inherits all common tests from IdentifyByAttributeMixin.
+ """
+
+ __test__ = False
+
+ def test_cf_identities(self):
+ """Test that all CF_IDENTITIES attributes are recognized."""
+ assert self.CF_IDENTITIES
+
+ for identity in self.CF_IDENTITIES:
+ subject_name = "ref_subject"
+ ref_subject = _NetCDFVar(subject_name)
+ vars_common = {
+ subject_name: ref_subject,
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ }
+ expected = {subject_name: self.CF_CLASS(subject_name, ref_subject)}
+
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, identity, subject_name)
+ vars_all = dict({"ref_source": ref_source}, **vars_common)
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == expected
+
+ def test_two_identities(self):
+ """Test using multiple different CF_IDENTITIES."""
+ subject_names = ("ref_subject_1", "ref_subject_2")
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
+
+ ref_source_vars = {
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
+ }
+ for ix, var in enumerate(ref_source_vars.values()):
+ setattr(var, self.CF_IDENTITIES[ix], subject_names[ix])
+ vars_all = dict(
+ {"ref_not_subject": _NetCDFVar("ref_not_subject")},
+ **ref_subject_vars,
+ **ref_source_vars,
+ )
+
+ expected = {
+ name: self.CF_CLASS(name, var) for name, var in ref_subject_vars.items()
+ }
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == expected
+
+ def test_string_type_ignored(self):
+ """Test that string-typed referenced variables are ignored."""
+ subject_name = "ref_subject"
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], subject_name)
+ vars_all = {
+ subject_name: _NetCDFVar(subject_name, dtype=np.bytes_),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == {}
+
+ def test_warn_string_type(self):
+ """Test warning when string-typed var referenced by identity attribute."""
+ subject_name = "ref_subject"
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], subject_name)
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ subject_name: _NetCDFVar(subject_name, dtype=np.bytes_),
+ }
+
+ def operation(warn: bool):
+ warnings.warn(
+ "emit at least 1 warning",
+ category=iris.warnings.IrisUserWarning,
+ )
+ result = self.CF_CLASS.identify(vars_all, warn=warn)
+ assert result == {}
+
+ warn_regex = r".*is a CF-netCDF label variable.*"
+ assert_warning_gated(operation, iris.warnings.IrisCfLabelVarWarning, warn_regex)
diff --git a/lib/iris/tests/unit/fileformats/cf/loader/__init__.py b/lib/iris/tests/unit/fileformats/cf/loader/__init__.py
new file mode 100644
index 0000000000..f8328cda1d
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/loader/__init__.py
@@ -0,0 +1,5 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the :mod:`iris.fileformats.cf.loader` module."""
diff --git a/lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py b/lib/iris/tests/unit/fileformats/cf/loader/test__chunk_control.py
similarity index 59%
rename from lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py
rename to lib/iris/tests/unit/fileformats/cf/loader/test__chunk_control.py
index fb53fa01f0..0fca593522 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/loader/test__chunk_control.py
+++ b/lib/iris/tests/unit/fileformats/cf/loader/test__chunk_control.py
@@ -2,7 +2,7 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Unit tests for :class:`iris.fileformats.netcdf.loader.ChunkControl`."""
+"""Unit tests for :class:`iris.fileformats.cf.loader.ChunkControl`."""
import re
@@ -12,8 +12,11 @@
import iris
from iris.cube import CubeList
-from iris.fileformats.netcdf import loader
-from iris.fileformats.netcdf.loader import CHUNK_CONTROL
+import iris.fileformats.cf
+from iris.fileformats.cf import loader as cf_loader
+from iris.fileformats.cf.loader import CHUNK_CONTROL
+from iris.fileformats.netcdf import loader as netcdf_loader
+import iris.fileformats.netcdf._dataset
import iris.tests.stock as istk
@@ -33,7 +36,7 @@ def save_cubelist_with_sigma(tmp_filepath):
def save_cube_with_chunksize(tmp_filepath):
cube = istk.simple_3d()
# adding an aux coord allows us to test that
- # iris.fileformats.netcdf.loader._get_cf_var_data()
+ # iris.fileformats.cf.loader._get_cf_var_data()
# will only throw an error if from_file mode is
# True when the entire cube has no specified chunking
aux = iris.coords.AuxCoord(
@@ -54,15 +57,15 @@ def tmp_filepath(tmp_path_factory):
@pytest.fixture(autouse=True)
def remove_min_bytes():
- old_min_bytes = loader._LAZYVAR_MIN_BYTES
- loader._LAZYVAR_MIN_BYTES = 0
+ old_min_bytes = cf_loader._LAZYVAR_MIN_BYTES
+ cf_loader._LAZYVAR_MIN_BYTES = 0
yield
- loader._LAZYVAR_MIN_BYTES = old_min_bytes
+ cf_loader._LAZYVAR_MIN_BYTES = old_min_bytes
def test_default(tmp_filepath, save_cubelist_with_sigma):
cube_varname, _ = save_cubelist_with_sigma
- cubes = CubeList(loader.load_cubes(tmp_filepath))
+ cubes = CubeList(netcdf_loader.load_cubes(tmp_filepath))
cube = cubes.extract_cube(cube_varname)
assert cube.shape == (3, 4, 5, 6)
assert cube.lazy_data().chunksize == (3, 4, 5, 6)
@@ -85,7 +88,7 @@ def test_netcdf_v3():
def test_control_global(tmp_filepath, save_cubelist_with_sigma):
cube_varname, _ = save_cubelist_with_sigma
with CHUNK_CONTROL.set(model_level_number=2):
- cubes = CubeList(loader.load_cubes(tmp_filepath))
+ cubes = CubeList(netcdf_loader.load_cubes(tmp_filepath))
cube = cubes.extract_cube(cube_varname)
assert cube.shape == (3, 4, 5, 6)
assert cube.lazy_data().chunksize == (3, 2, 5, 6)
@@ -99,7 +102,7 @@ def test_control_global(tmp_filepath, save_cubelist_with_sigma):
def test_control_sigma_only(tmp_filepath, save_cubelist_with_sigma):
cube_varname, sigma_varname = save_cubelist_with_sigma
with CHUNK_CONTROL.set(sigma_varname, model_level_number=2):
- cubes = CubeList(loader.load_cubes(tmp_filepath))
+ cubes = CubeList(netcdf_loader.load_cubes(tmp_filepath))
cube = cubes.extract_cube(cube_varname)
assert cube.shape == (3, 4, 5, 6)
assert cube.lazy_data().chunksize == (3, 4, 5, 6)
@@ -114,7 +117,7 @@ def test_control_sigma_only(tmp_filepath, save_cubelist_with_sigma):
def test_control_cube_var(tmp_filepath, save_cubelist_with_sigma):
cube_varname, _ = save_cubelist_with_sigma
with CHUNK_CONTROL.set(cube_varname, model_level_number=2):
- cubes = CubeList(loader.load_cubes(tmp_filepath))
+ cubes = CubeList(netcdf_loader.load_cubes(tmp_filepath))
cube = cubes.extract_cube(cube_varname)
assert cube.shape == (3, 4, 5, 6)
assert cube.lazy_data().chunksize == (3, 2, 5, 6)
@@ -129,14 +132,14 @@ def test_invalid_chunksize(tmp_filepath, save_cubelist_with_sigma):
msg = "'dimension_chunksizes' kwargs should be a dict of `str: int` pairs, not {'model_level_numer': '2'}."
with pytest.raises(ValueError, match=msg):
with CHUNK_CONTROL.set(model_level_numer="2"):
- CubeList(loader.load_cubes(tmp_filepath))
+ CubeList(netcdf_loader.load_cubes(tmp_filepath))
def test_invalid_var_name(tmp_filepath, save_cubelist_with_sigma):
msg = re.escape("'var_names' should be an iterable of strings, not [1, 2].")
with pytest.raises(ValueError, match=msg):
with CHUNK_CONTROL.set([1, 2], model_level_numer="2"):
- CubeList(loader.load_cubes(tmp_filepath))
+ CubeList(netcdf_loader.load_cubes(tmp_filepath))
def test_control_multiple(tmp_filepath, save_cubelist_with_sigma):
@@ -145,7 +148,7 @@ def test_control_multiple(tmp_filepath, save_cubelist_with_sigma):
CHUNK_CONTROL.set(cube_varname, model_level_number=2),
CHUNK_CONTROL.set(sigma_varname, model_level_number=3),
):
- cubes = CubeList(loader.load_cubes(tmp_filepath))
+ cubes = CubeList(netcdf_loader.load_cubes(tmp_filepath))
cube = cubes.extract_cube(cube_varname)
assert cube.shape == (3, 4, 5, 6)
assert cube.lazy_data().chunksize == (3, 2, 5, 6)
@@ -160,7 +163,7 @@ def test_neg_one(tmp_filepath, save_cubelist_with_sigma):
cube_varname, _ = save_cubelist_with_sigma
with dask.config.set({"array.chunk-size": "50B"}):
with CHUNK_CONTROL.set(model_level_number=-1):
- cubes = CubeList(loader.load_cubes(tmp_filepath))
+ cubes = CubeList(netcdf_loader.load_cubes(tmp_filepath))
cube = cubes.extract_cube(cube_varname)
assert cube.shape == (3, 4, 5, 6)
# uses known good output
@@ -174,7 +177,7 @@ def test_neg_one(tmp_filepath, save_cubelist_with_sigma):
def test_from_file(tmp_filepath, save_cube_with_chunksize):
with CHUNK_CONTROL.from_file():
- cube = next(loader.load_cubes(tmp_filepath))
+ cube = next(netcdf_loader.load_cubes(tmp_filepath))
assert cube.shape == (2, 3, 4)
assert cube.lazy_data().chunksize == (1, 3, 4)
@@ -183,7 +186,7 @@ def test_no_chunks_from_file(tmp_filepath, save_cubelist_with_sigma):
cube_varname, _ = save_cubelist_with_sigma
with pytest.raises(KeyError):
with CHUNK_CONTROL.from_file():
- CubeList(loader.load_cubes(tmp_filepath))
+ CubeList(netcdf_loader.load_cubes(tmp_filepath))
def test_as_dask(tmp_filepath, save_cubelist_with_sigma, mocker):
@@ -194,22 +197,28 @@ def test_as_dask(tmp_filepath, save_cubelist_with_sigma, mocker):
from our own chunking behaviour.
"""
message = "Mock called, rest of test unneeded"
- as_lazy_data = mocker.patch("iris.fileformats.netcdf.loader.as_lazy_data")
+ as_lazy_data = mocker.patch("iris.fileformats.netcdf._dataset.as_lazy_data")
as_lazy_data.side_effect = RuntimeError(message)
with CHUNK_CONTROL.as_dask():
try:
- CubeList(loader.load_cubes(tmp_filepath))
+ CubeList(netcdf_loader.load_cubes(tmp_filepath))
except RuntimeError as e:
if str(e) != message:
raise e
- as_lazy_data.assert_called_with(mocker.ANY, meta=mocker.ANY, chunks="auto")
+ as_lazy_data.assert_called_with(
+ mocker.ANY,
+ meta=mocker.ANY,
+ chunks="auto",
+ dims_fixed=None,
+ cache_key=mocker.ANY,
+ )
def test_pinned_optimisation(tmp_filepath, save_cubelist_with_sigma):
cube_varname, _ = save_cubelist_with_sigma
with dask.config.set({"array.chunk-size": "250B"}):
with CHUNK_CONTROL.set(model_level_number=2):
- cubes = CubeList(loader.load_cubes(tmp_filepath))
+ cubes = CubeList(netcdf_loader.load_cubes(tmp_filepath))
cube = cubes.extract_cube(cube_varname)
assert cube.shape == (3, 4, 5, 6)
# uses known good output
@@ -220,3 +229,73 @@ def test_pinned_optimisation(tmp_filepath, save_cubelist_with_sigma):
assert sigma.shape == (4,)
assert sigma.lazy_points().chunksize == (2,)
assert sigma.lazy_bounds().chunksize == (2, 2)
+
+
+class TestChunksFromChunkControl:
+ @staticmethod
+ def _cf_var(mocker, chunking, shape=(2, 3, 4), cls=None):
+ if cls is None:
+ cls = iris.fileformats.cf.CFDataVariable
+ dimensions = tuple(f"dim_{i}" for i in range(len(shape)))
+ cf_data = mocker.MagicMock(
+ spec=iris.fileformats.netcdf._dataset.NetCDFDatasetVariable,
+ chunking=chunking,
+ dimensions=dimensions,
+ )
+ return mocker.MagicMock(
+ spec=cls,
+ cf_data=cf_data,
+ cf_name="DUMMY_VAR",
+ shape=shape,
+ dimensions=dimensions,
+ )
+
+ def test_as_dask_defers_everything_to_dask(self, mocker):
+ cf_var = self._cf_var(mocker, chunking=None)
+ with CHUNK_CONTROL.as_dask():
+ assert cf_loader._chunks_from_chunk_control(cf_var) == ("auto", None)
+
+ def test_default_unchunked_uses_the_shape(self, mocker):
+ cf_var = self._cf_var(mocker, chunking=None)
+ chunks, dims_fixed = cf_loader._chunks_from_chunk_control(cf_var)
+ assert chunks == [2, 3, 4]
+ assert dims_fixed == (None,)
+
+ def test_from_file_adopts_and_fixes_the_store_chunking(self, mocker):
+ cf_var = self._cf_var(mocker, chunking=(1, 3, 4))
+ with CHUNK_CONTROL.from_file():
+ chunks, dims_fixed = cf_loader._chunks_from_chunk_control(cf_var)
+ assert chunks == [1, 3, 4]
+ assert tuple(bool(flag) for flag in dims_fixed) == (True, True, True)
+
+ def test_from_file_refuses_an_unchunked_data_variable(self, mocker):
+ cf_var = self._cf_var(mocker, chunking=None)
+ with CHUNK_CONTROL.from_file():
+ with pytest.raises(KeyError, match="pre-existing chunk specifications"):
+ cf_loader._chunks_from_chunk_control(cf_var)
+
+ def test_from_file_accepts_an_unchunked_coordinate(self, mocker):
+ # Only the cube's data variable is required to carry a chunking; an
+ # auxiliary coordinate that does not is normal.
+ cf_var = self._cf_var(
+ mocker,
+ chunking=None,
+ cls=iris.fileformats.cf.CFAuxiliaryCoordinateVariable,
+ )
+ with CHUNK_CONTROL.from_file():
+ chunks, _ = cf_loader._chunks_from_chunk_control(cf_var)
+ assert chunks == [2, 3, 4]
+
+
+def test_from_file_still_loads_a_small_unchunked_cube(tmp_path, mocker):
+ # The KeyError sits behind the "small enough to read whole" shortcut, so a
+ # small contiguous cube loads under from_file() today. Hoisting the check
+ # into the CF layer, which is where the "is this the data variable" fact
+ # lives, would break this - so pin it.
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 5000)
+ path = str(tmp_path / "small.nc")
+ iris.save(istk.simple_3d(), path)
+ with CHUNK_CONTROL.from_file():
+ cube = next(netcdf_loader.load_cubes(path))
+ assert cube.shape == (2, 3, 4)
+ assert not cube.has_lazy_data()
diff --git a/lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py b/lib/iris/tests/unit/fileformats/cf/loader/test__get_cf_var_data.py
similarity index 59%
rename from lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py
rename to lib/iris/tests/unit/fileformats/cf/loader/test__get_cf_var_data.py
index 94a8e3cfd5..abb6b58e41 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/loader/test__get_cf_var_data.py
+++ b/lib/iris/tests/unit/fileformats/cf/loader/test__get_cf_var_data.py
@@ -2,7 +2,16 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Unit tests for the `iris.fileformats.netcdf._get_cf_var_data` function."""
+"""Unit tests for :func:`iris.fileformats.cf.loader._get_cf_var_data`.
+
+The function is format-agnostic, but the only ``CFDatasetVariable`` that
+exists today is the netCDF one, so the mocks here are netCDF-shaped. Zarr
+loading (see section 5 of
+``docs/superpowers/specs/2026-09-21-zarr-io-design.md``) adds a second
+implementation and these become the shared cases.
+"""
+
+from functools import partial
import dask.array
import dask.array as da
@@ -11,10 +20,11 @@
from iris._lazy_data import _optimum_chunksize
import iris.fileformats.cf
-from iris.fileformats.netcdf._thread_safe_nc import VLType
-from iris.fileformats.netcdf.loader import CHUNK_CONTROL, _get_cf_var_data
+from iris.fileformats.cf.loader import CHUNK_CONTROL, _get_cf_var_data
+import iris.fileformats.netcdf._dataset
from iris.tests import _shared_utils
from iris.tests.unit.fileformats import MockerMixin
+import iris.warnings
class Test__get_cf_var_data(MockerMixin):
@@ -27,26 +37,32 @@ def _setup(self):
def _make(self, chunksizes=None, shape=None, dtype="i4", **extra_properties):
if shape is None:
shape = self.shape
- dimensions_dict = {
- "dim_" + str(x): self.mocker.Mock(size=x) for x in range(len(shape))
- }
- dimension_names = list(dimensions_dict.keys())
+ if dtype is not str: # for testing VLen str arrays (dtype=`class <str>`)
+ dtype = np.dtype(dtype)
cf_data = self.mocker.MagicMock(
- _FillValue=None,
- _Encoding="",
- __getitem__="<real-data>",
- dimensions=dimension_names,
- # A parent .group() with dimensions is needed to validate the var dims
- group=self.mocker.Mock(
- return_value=self.mocker.Mock(dimensions=dimensions_dict)
- ),
- dtype=dtype,
+ spec=iris.fileformats.netcdf._dataset.NetCDFDatasetVariable,
+ fill_value=None,
+ dimensions=tuple("dim_" + str(x) for x in range(len(shape))),
shape=shape,
- **extra_properties,
+ chunking=chunksizes,
+ is_variable_length=False,
+ is_emulated=False,
+ # NetCDFDataProxy reads .shape straight off this: the one
+ # deliberate escape hatch, and the one place a real shape matters.
+ variable=self.mocker.MagicMock(shape=shape),
+ )
+ # `name` is a MagicMock constructor keyword, so it is set afterwards.
+ cf_data.name = "DUMMY_VAR"
+ cf_data.location = self.filename
+ cf_data.dtype = dtype
+ cf_data.size = np.prod(shape)
+ cf_data.attributes = {}
+ cf_data.__getitem__.return_value = self.mocker.sentinel.real_data_accessed
+ # The real method, bound to the mock: it reads only the members above,
+ # so the existing assertions keep testing the code they always did.
+ cf_data.read_data = partial(
+ iris.fileformats.netcdf._dataset.NetCDFDatasetVariable.read_data, cf_data
)
- cf_data.chunking = self.mocker.MagicMock(return_value=chunksizes)
- if dtype is not str: # for testing VLen str arrays (dtype=`class <str>`)
- dtype = np.dtype(dtype)
cf_var = self.mocker.MagicMock(
spec=iris.fileformats.cf.CFVariable,
dtype=dtype,
@@ -55,6 +71,8 @@ def _make(self, chunksizes=None, shape=None, dtype="i4", **extra_properties):
cf_name="DUMMY_VAR",
shape=shape,
size=np.prod(shape),
+ attributes={},
+ dimensions=cf_data.dimensions,
**extra_properties,
)
cf_var.__getitem__.return_value = self.mocker.sentinel.real_data_accessed
@@ -87,21 +105,15 @@ def test_cf_data_chunk_control(self):
def test_cf_data_no_chunks(self):
# No chunks means chunks are calculated from the array's shape by
- # `iris._lazy_data._optimum_chunksize()`.
+ # `iris._lazy_data._optimum_chunksize()`. This also covers what used
+ # to be the separate "contiguous" case: NetCDFDatasetVariable.chunking
+ # already collapses that netCDF spelling of "unchunked" to None.
chunks = None
cf_var = self._make(chunks)
lazy_data = _get_cf_var_data(cf_var)
lazy_data_chunks = [c[0] for c in lazy_data.chunks]
_shared_utils.assert_array_equal(lazy_data_chunks, self.expected_chunks)
- def test_cf_data_contiguous(self):
- # Chunks 'contiguous' is equivalent to no chunks.
- chunks = "contiguous"
- cf_var = self._make(chunks)
- lazy_data = _get_cf_var_data(cf_var)
- lazy_data_chunks = [c[0] for c in lazy_data.chunks]
- _shared_utils.assert_array_equal(lazy_data_chunks, self.expected_chunks)
-
def test_type__1kf8_is_lazy(self):
cf_var = self._make(shape=(1000,), dtype="f8")
var_data = _get_cf_var_data(cf_var)
@@ -117,68 +129,83 @@ def test_arraytype__100f8_is_real(self, mocker):
var_data = _get_cf_var_data(cf_var)
assert var_data is mocker.sentinel.real_data_accessed
- def test_vltype__1000str_is_lazy(self, mocker):
- # Variable length string type
- mock_vltype = mocker.Mock(spec=VLType, dtype=str, name="varlen string type")
- cf_var = self._make(shape=(1000,), dtype=str, datatype=mock_vltype)
+ def test_vltype__1000str_is_lazy(self):
+ cf_var = self._make(shape=(1000,), dtype=str)
+ cf_var.cf_data.is_variable_length = True
var_data = _get_cf_var_data(cf_var)
assert isinstance(var_data, da.Array)
def test_vltype__1000str_is_real_with_hint(self, mocker):
- # Variable length string type with a hint on the array variable length size
- mock_vltype = mocker.Mock(spec=VLType, dtype=str, name="varlen string type")
- cf_var = self._make(shape=(100,), dtype=str, datatype=mock_vltype)
+ cf_var = self._make(shape=(100,), dtype=str)
+ cf_var.cf_data.is_variable_length = True
with CHUNK_CONTROL.set("DUMMY_VAR", _vl_hint=1):
var_data = _get_cf_var_data(cf_var)
assert var_data is mocker.sentinel.real_data_accessed
def test_vltype__100str_is_real(self, mocker):
- # Variable length string type
- mock_vltype = mocker.Mock(spec=VLType, dtype=str, name="varlen string type")
- cf_var = self._make(shape=(100,), dtype=str, datatype=mock_vltype)
+ cf_var = self._make(shape=(100,), dtype=str)
+ cf_var.cf_data.is_variable_length = True
var_data = _get_cf_var_data(cf_var)
assert var_data is mocker.sentinel.real_data_accessed
- def test_vltype__100str_is_lazy_with_hint(self, mocker):
- # Variable length string type with a hint on the array variable length size
- mock_vltype = mocker.Mock(spec=VLType, dtype=str, name="varlen string type")
- cf_var = self._make(shape=(100,), dtype=str, datatype=mock_vltype)
+ def test_vltype__100str_is_lazy_with_hint(self):
+ cf_var = self._make(shape=(100,), dtype=str)
+ cf_var.cf_data.is_variable_length = True
with CHUNK_CONTROL.set("DUMMY_VAR", _vl_hint=50):
var_data = _get_cf_var_data(cf_var)
assert isinstance(var_data, da.Array)
- def test_vltype__100f8_is_lazy(self, mocker):
- # Variable length float64 type
- mock_vltype = mocker.Mock(spec=VLType, dtype="f8", name="varlen float64 type")
- cf_var = self._make(shape=(1000,), dtype="f8", datatype=mock_vltype)
+ def test_vltype__100f8_is_lazy(self):
+ cf_var = self._make(shape=(1000,), dtype="f8")
+ cf_var.cf_data.is_variable_length = True
var_data = _get_cf_var_data(cf_var)
assert isinstance(var_data, da.Array)
def test_vltype__100f8_is_real_with_hint(self, mocker):
- # Variable length float64 type with a hint on the array variable length size
- mock_vltype = mocker.Mock(spec=VLType, dtype="f8", name="varlen float64 type")
- cf_var = self._make(shape=(100,), dtype="f8", datatype=mock_vltype)
+ cf_var = self._make(shape=(100,), dtype="f8")
+ cf_var.cf_data.is_variable_length = True
with CHUNK_CONTROL.set("DUMMY_VAR", _vl_hint=2):
var_data = _get_cf_var_data(cf_var)
assert var_data is mocker.sentinel.real_data_accessed
+ def test_vltype_warns_that_the_size_is_unknown(self):
+ cf_var = self._make(shape=(1000,), dtype="f8")
+ cf_var.cf_data.is_variable_length = True
+ with pytest.warns(
+ iris.warnings.IrisLoadWarning, match="variable length type of kind"
+ ):
+ _get_cf_var_data(cf_var)
+
@pytest.mark.parametrize("is_realdata", [True, False], ids=["realdata", "lazydata"])
@pytest.mark.parametrize("is_string", [True, False], ids=["string", "numeric"])
def test_cf_data_emulation(self, mocker, is_realdata, is_string):
# Check that a variable emulation object passes its real data directly.
# ... or for string data, that it converts it, either lazy or real.
shape = (20, 30, 30)
- spec = np.ndarray if is_realdata else da.Array
+ array_spec = np.ndarray if is_realdata else da.Array
dtype = np.dtype("S1") if is_string else np.dtype("f4")
- emulated_data = mocker.MagicMock(spec=spec, dtype=dtype, shape=shape)
- # Make a cf_var with a special extra '_data_array' property.
- cf_var = self._make(
- chunksizes=None,
- dtype=dtype,
- _data_array=emulated_data,
- shape=shape,
- )
+ emulated_data = mocker.MagicMock(spec=array_spec, dtype=dtype, shape=shape)
+ cf_var = self._make(chunksizes=None, dtype=dtype, shape=shape)
+ cf_var.cf_data.is_emulated = True
+ cf_var.cf_data.emulated_data_array = emulated_data
+
if is_string:
+ # The encoding is described by the file variable beneath the
+ # emulation, which is 'char' where the data read out is strings.
+ unencoded = mocker.Mock(
+ dtype=np.dtype("S1"),
+ dimensions=("dim_0", "dim_1", "string5"),
+ # None is what an absent _Encoding attribute yields: the
+ # default encoding, and no "unsupported encoding" warning.
+ _Encoding=None,
+ )
+ # .name is a Mock constructor keyword, so it has to be set after.
+ unencoded.name = "DUMMY_VAR"
+ unencoded.group.return_value = mocker.Mock(
+ dimensions={"string5": mocker.Mock(size=5)}
+ )
+ cf_var.cf_data.unencoded_variable = unencoded
+
# detect that we added a translation to the data
ifnbd = "iris.fileformats.netcdf._bytecoding_datasets."
mock_decode = mocker.patch(ifnbd + "decode_bytesarray_to_stringarray")
diff --git a/lib/iris/tests/unit/fileformats/netcdf/loader/test__load_aux_factory.py b/lib/iris/tests/unit/fileformats/cf/loader/test__load_aux_factory.py
similarity index 98%
rename from lib/iris/tests/unit/fileformats/netcdf/loader/test__load_aux_factory.py
rename to lib/iris/tests/unit/fileformats/cf/loader/test__load_aux_factory.py
index da3ceaf77a..d0d22a7819 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/loader/test__load_aux_factory.py
+++ b/lib/iris/tests/unit/fileformats/cf/loader/test__load_aux_factory.py
@@ -2,7 +2,7 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Unit tests for the `iris.fileformats.netcdf._load_aux_factory` function."""
+"""Unit tests for :func:`iris.fileformats.cf.loader._load_aux_factory`."""
import re
import warnings
@@ -12,7 +12,7 @@
from iris.coords import DimCoord
from iris.cube import Cube
-from iris.fileformats.netcdf.loader import _load_aux_factory
+from iris.fileformats.cf.loader import _load_aux_factory
from iris.tests.unit.fileformats import MockerMixin
from iris.warnings import IrisFactoryCoordNotFoundWarning
diff --git a/lib/iris/tests/unit/fileformats/netcdf/loader/test__load_cube.py b/lib/iris/tests/unit/fileformats/cf/loader/test__load_cube.py
similarity index 95%
rename from lib/iris/tests/unit/fileformats/netcdf/loader/test__load_cube.py
rename to lib/iris/tests/unit/fileformats/cf/loader/test__load_cube.py
index cc6e03a2cc..7faf51ebeb 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/loader/test__load_cube.py
+++ b/lib/iris/tests/unit/fileformats/cf/loader/test__load_cube.py
@@ -2,14 +2,14 @@
#
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
-"""Unit tests for the `iris.fileformats.netcdf._load_cube` function."""
+"""Unit tests for :func:`iris.fileformats.cf.loader._load_cube`."""
import numpy as np
import pytest
from iris.coords import DimCoord
import iris.fileformats.cf
-from iris.fileformats.netcdf.loader import _load_cube
+from iris.fileformats.cf.loader import _load_cube
from iris.loading import LOAD_PROBLEMS
from iris.tests.unit.fileformats import MockerMixin
@@ -30,7 +30,7 @@ def _patcher(engine, cf, cf_group):
@pytest.fixture(autouse=True)
def _setup(self, mocker):
- this = "iris.fileformats.netcdf.loader._assert_case_specific_facts"
+ this = "iris.fileformats.cf.loader._assert_case_specific_facts"
_ = mocker.patch(this, side_effect=self._patcher)
self.engine = mocker.Mock()
self.filename = "DUMMY"
@@ -51,7 +51,7 @@ def _make(self, names, attrs):
cf_group[name] = self.mocker.Mock(cf_attrs_unused=cf_attrs_unused)
cf = self.mocker.Mock(cf_group=cf_group)
- cf_data = self.mocker.Mock(_FillValue=None)
+ cf_data = self.mocker.Mock()
cf_data.chunking = self.mocker.MagicMock(return_value=shape)
cf_var = self.mocker.MagicMock(
spec=iris.fileformats.cf.CFVariable,
@@ -61,6 +61,7 @@ def _make(self, names, attrs):
cf_group=coords,
shape=shape,
size=np.prod(shape),
+ attributes={},
)
return cf, cf_var
@@ -126,7 +127,7 @@ def test_load_problems(self):
class TestCubeAttributes(MockerMixin):
@pytest.fixture(autouse=True)
def _setup(self, mocker):
- this = "iris.fileformats.netcdf.loader._assert_case_specific_facts"
+ this = "iris.fileformats.cf.loader._assert_case_specific_facts"
_ = mocker.patch(this)
self.engine = mocker.Mock()
self.cf = None
@@ -141,7 +142,7 @@ def _setup(self, mocker):
def _make(self, attrs):
shape = (1,)
cf_attrs_unused = self.mocker.Mock(return_value=attrs)
- cf_data = self.mocker.Mock(_FillValue=None)
+ cf_data = self.mocker.Mock()
cf_data.chunking = self.mocker.MagicMock(return_value=shape)
cf_var = self.mocker.MagicMock(
spec=iris.fileformats.cf.CFVariable,
@@ -153,6 +154,7 @@ def _make(self, attrs):
cf_attrs_unused=cf_attrs_unused,
shape=shape,
size=np.prod(shape),
+ attributes={},
)
return cf_var
diff --git a/lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py b/lib/iris/tests/unit/fileformats/cf/loader/test__translate_constraints_to_var_callback.py
similarity index 68%
rename from lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py
rename to lib/iris/tests/unit/fileformats/cf/loader/test__translate_constraints_to_var_callback.py
index 3f386238ea..ab8d36255d 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py
+++ b/lib/iris/tests/unit/fileformats/cf/loader/test__translate_constraints_to_var_callback.py
@@ -3,7 +3,7 @@
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for
-:func:`iris.fileformats.netcdf._translate_constraints_to_var_callback`.
+:func:`iris.fileformats.cf.loader._translate_constraints_to_var_callback`.
"""
@@ -11,24 +11,29 @@
import iris
from iris.fileformats.cf import CFDataVariable
-from iris.fileformats.netcdf.loader import _translate_constraints_to_var_callback
+from iris.fileformats.cf.loader import _translate_constraints_to_var_callback
from iris.tests import _shared_utils
+def _data_variable(mocker, name, **attributes):
+ """A CFDataVariable whose CF attributes are exactly the ones named."""
+ return CFDataVariable(name, mocker.MagicMock(attributes=attributes))
+
+
class Test:
@pytest.fixture(autouse=True)
def _setup(self, mocker):
self.data_variables = [
- CFDataVariable("var1", mocker.MagicMock(standard_name="x_wind")),
- CFDataVariable("var2", mocker.MagicMock(standard_name="y_wind")),
- CFDataVariable("var1", mocker.MagicMock(long_name="x component of wind")),
- CFDataVariable(
+ _data_variable(mocker, "var1", standard_name="x_wind"),
+ _data_variable(mocker, "var2", standard_name="y_wind"),
+ _data_variable(mocker, "var1", long_name="x component of wind"),
+ _data_variable(
+ mocker,
"var1",
- mocker.MagicMock(
- standard_name="x_wind", long_name="x component of wind"
- ),
+ standard_name="x_wind",
+ long_name="x component of wind",
),
- CFDataVariable("var1", mocker.MagicMock()),
+ _data_variable(mocker, "var1"),
]
def test_multiple_constraints(self):
@@ -38,7 +43,7 @@ def test_multiple_constraints(self):
]
callback = _translate_constraints_to_var_callback(constrs)
result = [callback(var) for var in self.data_variables]
- _shared_utils.assert_array_equal(result, [True, True, False, True, False])
+ _shared_utils.assert_array_equal(result, [True, True, True, True, True])
def test_multiple_constraints_invalid(self):
constrs = [
@@ -57,12 +62,12 @@ def test_multiple_constraints__multiname(self, mocker):
callback = _translate_constraints_to_var_callback(constrs)
# Add 2 extra vars: one passes both name checks, and the other does not
vars = self.data_variables + [
- CFDataVariable("var1", mocker.MagicMock(standard_name="x_wind")),
- CFDataVariable("var1", mocker.MagicMock(standard_name="air_pressure")),
+ _data_variable(mocker, "var1", standard_name="x_wind"),
+ _data_variable(mocker, "var1", standard_name="air_pressure"),
]
result = [callback(var) for var in vars]
_shared_utils.assert_array_equal(
- result, [True, True, False, True, False, True, False]
+ result, [True, True, True, True, True, True, False]
)
def test_non_name_constraint(self):
@@ -83,13 +88,19 @@ def test_name_constraint_standard_name(self):
constr = iris.NameConstraint(standard_name="x_wind")
callback = _translate_constraints_to_var_callback(constr)
result = [callback(var) for var in self.data_variables]
- _shared_utils.assert_array_equal(result, [True, False, False, True, False])
+ _shared_utils.assert_array_equal(result, [True, False, True, True, True])
- def test_name_constraint_long_name(self):
+ def test_name_constraint_long_name(self, mocker):
constr = iris.NameConstraint(long_name="x component of wind")
callback = _translate_constraints_to_var_callback(constr)
- result = [callback(var) for var in self.data_variables]
- _shared_utils.assert_array_equal(result, [False, False, True, True, False])
+ # The shared vars either match long_name or omit it, and omission is
+ # permissive - so without a var that CONTRADICTS, this test would pass
+ # even if the callback ignored long_name altogether.
+ vars = self.data_variables + [
+ _data_variable(mocker, "var1", long_name="y component of wind"),
+ ]
+ result = [callback(var) for var in vars]
+ _shared_utils.assert_array_equal(result, [True, True, True, True, True, False])
def test_name_constraint_var_name(self):
constr = iris.NameConstraint(var_name="var1")
@@ -101,7 +112,7 @@ def test_name_constraint_standard_name_var_name(self):
constr = iris.NameConstraint(standard_name="x_wind", var_name="var1")
callback = _translate_constraints_to_var_callback(constr)
result = [callback(var) for var in self.data_variables]
- _shared_utils.assert_array_equal(result, [True, False, False, True, False])
+ _shared_utils.assert_array_equal(result, [True, False, True, True, True])
def test_name_constraint_standard_name_long_name_var_name(self):
constr = iris.NameConstraint(
@@ -111,7 +122,7 @@ def test_name_constraint_standard_name_long_name_var_name(self):
)
callback = _translate_constraints_to_var_callback(constr)
result = [callback(var) for var in self.data_variables]
- _shared_utils.assert_array_equal(result, [False, False, False, True, False])
+ _shared_utils.assert_array_equal(result, [True, False, True, True, True])
def test_name_constraint_with_stash(self):
constr = iris.NameConstraint(standard_name="x_wind", STASH="m01s00i024")
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFAncillaryDataVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFAncillaryDataVariable.py
new file mode 100644
index 0000000000..3b3c96605f
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFAncillaryDataVariable.py
@@ -0,0 +1,17 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFAncillaryDataVariable`."""
+
+from iris.fileformats.cf import CFAncillaryDataVariable
+
+from .identify_mixins import IdentifyByAttributeMixin
+
+
+class TestIdentify(IdentifyByAttributeMixin):
+ __test__ = True
+
+ CF_CLASS = CFAncillaryDataVariable
+ CF_IDENTITIES = ["ancillary_variables"]
+ MISSING_WARN_REGEX = r"Missing CF-netCDF ancillary data variable {subject!r}.*"
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFAuxiliaryCoordinateVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFAuxiliaryCoordinateVariable.py
new file mode 100644
index 0000000000..967518b0a7
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFAuxiliaryCoordinateVariable.py
@@ -0,0 +1,35 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFAuxiliaryCoordinateVariable`."""
+
+import numpy as np
+
+from iris.fileformats.cf import CFAuxiliaryCoordinateVariable
+
+from .identify_mixins import IdentifyByAttributeMixin, _NetCDFVar
+
+
+class TestIdentify(IdentifyByAttributeMixin):
+ __test__ = True
+
+ CF_CLASS = CFAuxiliaryCoordinateVariable
+ CF_IDENTITIES = ["coordinates"]
+ MISSING_WARN_REGEX = (
+ r"Missing CF-netCDF auxiliary coordinate variable {subject!r}.*"
+ )
+
+ def test_string_type_ignored(self):
+ # Coordinate-variable identify should reject label/string subjects.
+ subject_name = "ref_subject"
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], subject_name)
+ vars_all = {
+ subject_name: _NetCDFVar(subject_name, dtype=np.bytes_),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == {}
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFBoundaryVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFBoundaryVariable.py
new file mode 100644
index 0000000000..2c22be123e
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFBoundaryVariable.py
@@ -0,0 +1,45 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFBoundaryVariable`."""
+
+from iris.fileformats.cf import CFBoundaryVariable
+
+from .identify_mixins import (
+ IdentifyByAttributeMixin,
+ SpansMixin,
+ _NetCDFVar,
+)
+
+
+class TestIdentify(IdentifyByAttributeMixin):
+ __test__ = True
+
+ CF_CLASS = CFBoundaryVariable
+ CF_IDENTITIES = ["bounds"]
+ IDENTITY_SUPPORTS_MULTIPLE_REFS = False
+ MISSING_WARN_REGEX = r"Missing CF-netCDF boundary variable {subject!r}.*"
+
+ def test_whitespace_padded_ref(self):
+ # CF boundary references accept surrounding whitespace.
+ subject_name = "ref_subject"
+ ref_subject = self._make_subject(subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], f" {subject_name} ")
+ vars_all = {
+ subject_name: ref_subject,
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ expected = {subject_name: self.CF_CLASS(subject_name, ref_subject)}
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == expected
+
+
+class TestSpans(SpansMixin):
+ """Tests for CFBoundaryVariable.spans()."""
+
+ __test__ = True
+ CF_CLASS = CFBoundaryVariable
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFClimatologyVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFClimatologyVariable.py
new file mode 100644
index 0000000000..ea4fddf744
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFClimatologyVariable.py
@@ -0,0 +1,45 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFClimatologyVariable`."""
+
+from iris.fileformats.cf import CFClimatologyVariable
+
+from .identify_mixins import (
+ IdentifyByAttributeMixin,
+ SpansMixin,
+ _NetCDFVar,
+)
+
+
+class TestIdentify(IdentifyByAttributeMixin):
+ __test__ = True
+
+ CF_CLASS = CFClimatologyVariable
+ CF_IDENTITIES = ["climatology"]
+ IDENTITY_SUPPORTS_MULTIPLE_REFS = False
+ MISSING_WARN_REGEX = r"Missing CF-netCDF climatology variable {subject!r}.*"
+
+ def test_whitespace_padded_ref(self):
+ # CF climatology references accept surrounding whitespace.
+ subject_name = "ref_subject"
+ ref_subject = self._make_subject(subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], f" {subject_name} ")
+ vars_all = {
+ subject_name: ref_subject,
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ expected = {subject_name: self.CF_CLASS(subject_name, ref_subject)}
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == expected
+
+
+class TestSpans(SpansMixin):
+ """Tests for CFClimatologyVariable.spans()."""
+
+ __test__ = True
+ CF_CLASS = CFClimatologyVariable
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFCoordinateVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFCoordinateVariable.py
new file mode 100644
index 0000000000..3acdf3a5ee
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFCoordinateVariable.py
@@ -0,0 +1,170 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFCoordinateVariable`."""
+
+import numpy as np
+import numpy.ma as ma
+import pytest
+
+from iris.fileformats.cf import CFCoordinateVariable
+
+
+class _CoordVariableStub:
+ """Stub for a 1D netCDF variable acting as a coordinate variable."""
+
+ def __init__(self, name, dimensions, data, dtype=float):
+ self.name = name
+ self.dtype = np.dtype(dtype)
+ self.dimensions = dimensions
+ self.ndim = len(dimensions)
+ data_arr = np.asarray(data)
+ self.shape = data_arr.shape or ()
+ self._data = data
+
+ def ncattrs(self):
+ return []
+
+ @property
+ def attributes(self):
+ return {}
+
+ def __getitem__(self, key):
+ if self._data.ndim == 0:
+ return self._data
+ return self._data[key]
+
+ def __len__(self):
+ return len(self._data)
+
+
+def _make_coord_var(name, data, dtype=float):
+ """Helper: create a valid 1D coord variable with name == dimension."""
+ return _CoordVariableStub(name=name, dimensions=(name,), data=data, dtype=dtype)
+
+
+class TestIdentify:
+ def test_valid_coordinate_identified(self):
+ nc_var = _make_coord_var("lat", [1.0, 2.0, 3.0])
+ vars_all = {"lat": nc_var}
+
+ result = CFCoordinateVariable.identify(vars_all)
+ assert "lat" in result
+ assert isinstance(result["lat"], CFCoordinateVariable)
+
+ def test_string_dtype_rejected(self):
+ nc_var = _CoordVariableStub(
+ name="lat", dimensions=("lat",), data=["a", "b"], dtype=np.bytes_
+ )
+ vars_all = {"lat": nc_var}
+
+ result = CFCoordinateVariable.identify(vars_all)
+ assert result == {}
+
+ def test_ndim_not_one_rejected(self):
+ stub = _CoordVariableStub(
+ name="lat", dimensions=("lat", "lon"), data=[[1.0, 2.0]], dtype=float
+ )
+ assert stub.ndim == 2
+ assert stub.shape == (1, 2)
+ vars_all = {"lat": stub}
+
+ result = CFCoordinateVariable.identify(vars_all)
+ assert result == {}
+
+ def test_name_not_in_dimensions_rejected(self):
+ stub = _CoordVariableStub(
+ name="lat", dimensions=("x",), data=[1.0, 2.0], dtype=float
+ )
+ vars_all = {"lat": stub}
+
+ result = CFCoordinateVariable.identify(vars_all)
+ assert result == {}
+
+ def test_ignored_name_excluded(self):
+ nc_var = _make_coord_var("lat", [1.0, 2.0, 3.0])
+ vars_all = {"lat": nc_var}
+
+ result = CFCoordinateVariable.identify(vars_all, ignore=["lat"])
+ assert result == {}
+
+ def test_target_filters_to_named_var(self):
+ lat = _make_coord_var("lat", [1.0, 2.0])
+ lon = _make_coord_var("lon", [10.0, 20.0])
+ vars_all = {"lat": lat, "lon": lon}
+
+ result = CFCoordinateVariable.identify(vars_all, target="lat")
+ assert "lat" in result
+ assert "lon" not in result
+
+ def test_target_unknown_raises(self):
+ vars_all = {"lat": _make_coord_var("lat", [1.0])}
+
+ message = "Cannot identify unknown target CF-netCDF variable 'unknown'"
+ with pytest.raises(ValueError, match=message):
+ CFCoordinateVariable.identify(vars_all, target="unknown")
+
+ def test_target_wrong_type_raises(self):
+ vars_all = {"lat": _make_coord_var("lat", [1.0])}
+
+ message = "Expect a target CF-netCDF variable name"
+ with pytest.raises(TypeError, match=message):
+ CFCoordinateVariable.identify(vars_all, target=object())
+
+
+class TestIdentifyMonotonic:
+ def test_monotonic_increasing_accepted(self):
+ nc_var = _make_coord_var("lat", np.array([1.0, 2.0, 3.0]))
+ vars_all = {"lat": nc_var}
+
+ result = CFCoordinateVariable.identify(vars_all, monotonic=True)
+ assert "lat" in result
+
+ def test_monotonic_decreasing_accepted(self):
+ nc_var = _make_coord_var("lat", np.array([3.0, 2.0, 1.0]))
+ vars_all = {"lat": nc_var}
+
+ result = CFCoordinateVariable.identify(vars_all, monotonic=True)
+ assert "lat" in result
+
+ def test_non_monotonic_rejected(self):
+ nc_var = _make_coord_var("lat", np.array([1.0, 3.0, 2.0]))
+ vars_all = {"lat": nc_var}
+
+ result = CFCoordinateVariable.identify(vars_all, monotonic=True)
+ assert result == {}
+
+ def test_scalar_shape_accepted(self):
+ """Shape () is always accepted under monotonic mode."""
+ stub = _make_coord_var("lat", np.float64(1.0))
+ vars_all = {"lat": stub}
+
+ result = CFCoordinateVariable.identify(vars_all, monotonic=True)
+ assert "lat" in result
+
+ def test_single_element_shape_accepted(self):
+ """Shape (1,) is always accepted under monotonic mode."""
+ nc_var = _make_coord_var("lat", np.array([42.0]))
+ vars_all = {"lat": nc_var}
+
+ result = CFCoordinateVariable.identify(vars_all, monotonic=True)
+ assert "lat" in result
+
+ def test_masked_array_accepted_when_monotonic(self):
+ """Masked arrays are filled before monotonic check."""
+ data = ma.masked_array([1.0, 2.0, 3.0], mask=[False, False, False])
+ nc_var = _make_coord_var("lat", data)
+ vars_all = {"lat": nc_var}
+
+ result = CFCoordinateVariable.identify(vars_all, monotonic=True)
+ assert "lat" in result
+
+ def test_masked_array_rejected_when_masked(self):
+ """Masked arrays are rejected under monotonic mode if any elements are masked."""
+ data = ma.masked_array([1.0, 2.0, 3.0], mask=[False, True, False])
+ nc_var = _make_coord_var("lat", data)
+ vars_all = {"lat": nc_var}
+
+ result = CFCoordinateVariable.identify(vars_all, monotonic=True)
+ assert result == {}
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFDataVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFDataVariable.py
new file mode 100644
index 0000000000..e72a653976
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFDataVariable.py
@@ -0,0 +1,26 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFDataVariable`."""
+
+import pytest
+
+from iris.fileformats.cf import CFDataVariable
+
+from .identify_mixins import _NetCDFVar
+
+
+class TestIdentify:
+ def test_identify_raises_not_implemented(self):
+ vars_all = {"data_var": _NetCDFVar("data_var")}
+ with pytest.raises(NotImplementedError):
+ CFDataVariable.identify(vars_all)
+
+
+class TestConstructor:
+ def test_cf_name_and_data_stored(self):
+ nc_var = _NetCDFVar("data_var")
+ cf_var = CFDataVariable("data_var", nc_var)
+ assert cf_var.cf_name == "data_var"
+ assert cf_var.cf_data is nc_var
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFGridMappingVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFGridMappingVariable.py
new file mode 100644
index 0000000000..436c7b0ce2
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFGridMappingVariable.py
@@ -0,0 +1,240 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFGridMappingVariable`."""
+
+import warnings
+
+import pytest
+
+from iris.fileformats.cf import CFGridMappingVariable
+import iris.warnings
+
+from .identify_mixins import _NetCDFVar, assert_warning_gated
+
+CF_IDENTITY = "grid_mapping"
+
+
+class TestIdentify:
+ def test_no_coord_system_mappings_returns_empty(self):
+ """When coord_system_mappings is absent, no results."""
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, "crs_var")
+ crs_var = _NetCDFVar("crs_var")
+ vars_all = {"crs_var": crs_var, "ref_source": ref_source}
+
+ result = CFGridMappingVariable.identify(vars_all, coord_system_mappings=None)
+ assert result == {}
+
+ def test_no_mapping_entry_for_source_returns_empty(self):
+ """Data var has grid_mapping attr but no entry in coord_system_mappings."""
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, "crs_var")
+ crs_var = _NetCDFVar("crs_var")
+ vars_all = {"crs_var": crs_var, "ref_source": ref_source}
+
+ # Mapping dict exists but has no entry for "ref_source".
+ result = CFGridMappingVariable.identify(
+ vars_all, coord_system_mappings={"other_var": {"crs_var": [None]}}
+ )
+ assert result == {}
+
+ def test_simple_mapping_none_coord_identified(self):
+ """A mapping with coord=None (simple grid_mapping style) is accepted."""
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, "crs_var")
+ crs_var = _NetCDFVar("crs_var")
+ vars_all = {"crs_var": crs_var, "ref_source": ref_source}
+
+ # {coord_name -> cs_name}; None coord means simple style.
+ cs_mappings = {"ref_source": {None: "crs_var"}}
+
+ result = CFGridMappingVariable.identify(
+ vars_all, coord_system_mappings=cs_mappings
+ )
+ assert "crs_var" in result
+ assert isinstance(result["crs_var"], CFGridMappingVariable)
+
+ def test_valid_coord_ref_identified(self):
+ """Mapping with a real coordinate reference that exists is accepted."""
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, "crs_var")
+ crs_var = _NetCDFVar("crs_var")
+ coord_var = _NetCDFVar("lat")
+ vars_all = {
+ "crs_var": crs_var,
+ "lat": coord_var,
+ "ref_source": ref_source,
+ }
+
+ cs_mappings = {"ref_source": {"lat": "crs_var"}}
+
+ result = CFGridMappingVariable.identify(
+ vars_all, coord_system_mappings=cs_mappings
+ )
+ assert "crs_var" in result
+
+ def test_missing_mapping_variable_warns(self):
+ """Missing grid mapping variable itself emits a warning."""
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, "crs_var")
+ # crs_var is NOT in vars_all
+ vars_all = {"ref_source": ref_source}
+
+ cs_mappings = {"ref_source": {None: "crs_var"}}
+
+ def operation(warn: bool):
+ warnings.warn(
+ "emit at least 1 warning",
+ category=iris.warnings.IrisUserWarning,
+ )
+ CFGridMappingVariable.identify(
+ vars_all, coord_system_mappings=cs_mappings, warn=warn
+ )
+
+ warn_regex = r"Missing CF-netCDF grid mapping variable 'crs_var'.*"
+ assert_warning_gated(
+ operation, iris.warnings.IrisCfMissingVarWarning, warn_regex
+ )
+
+ def test_missing_coord_ref_warns(self):
+ """Missing coordinate associated with a grid mapping emits a warning.
+
+ Note: this warning is not gated by the `warn` argument - it is always
+ emitted when a referenced coordinate variable is absent.
+ """
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, "crs_var")
+ crs_var = _NetCDFVar("crs_var")
+ # lat is NOT in vars_all
+ vars_all = {"crs_var": crs_var, "ref_source": ref_source}
+
+ cs_mappings = {"ref_source": {"lat": "crs_var"}}
+
+ warn_regex = r"Missing CF-netCDF coordinate variable 'lat'.*"
+ with pytest.warns(iris.warnings.IrisCfMissingVarWarning, match=warn_regex):
+ CFGridMappingVariable.identify(vars_all, coord_system_mappings=cs_mappings)
+
+ def test_ignore(self):
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, "crs_var")
+ crs_var = _NetCDFVar("crs_var")
+ vars_all = {"crs_var": crs_var, "ref_source": ref_source}
+
+ cs_mappings = {"ref_source": {None: "crs_var"}}
+
+ result = CFGridMappingVariable.identify(
+ vars_all, ignore=["crs_var"], coord_system_mappings=cs_mappings
+ )
+ assert result == {}
+
+ def test_target_unknown_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Cannot identify unknown target CF-netCDF variable 'unknown'"
+ with pytest.raises(ValueError, match=message):
+ CFGridMappingVariable.identify(vars_all, target="unknown")
+
+ def test_target_wrong_type_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Expect a target CF-netCDF variable name"
+ with pytest.raises(TypeError, match=message):
+ CFGridMappingVariable.identify(vars_all, target=object())
+
+
+class TestIdentifyGroupingByCRS:
+ """Tests exercising the cs_coord_mappings grouping (lines 698-708 of cf.py).
+
+ The grouping step inverts {coord -> cs_name} into {cs_name -> [coords]},
+ so that each unique coordinate system is iterated once.
+ """
+
+ def test_multiple_coords_one_cs(self):
+ """Two coordinates both mapping to the same CRS: one CRS identified."""
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, "grid_mapping", "crs_1")
+ crs_1 = _NetCDFVar("crs_1")
+ lat = _NetCDFVar("lat")
+ lon = _NetCDFVar("lon")
+ vars_all = {
+ "crs_1": crs_1,
+ "lat": lat,
+ "lon": lon,
+ "ref_source": ref_source,
+ }
+
+ # Both lat and lon reference the same coordinate system.
+ cs_mappings = {"ref_source": {"lat": "crs_1", "lon": "crs_1"}}
+
+ result = CFGridMappingVariable.identify(
+ vars_all, coord_system_mappings=cs_mappings
+ )
+ assert list(result.keys()) == ["crs_1"]
+
+ def test_multiple_cs_both_identified(self):
+ """Two coordinates each referencing a different CRS: both CRSs identified."""
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, "grid_mapping", "crs_1 crs_2")
+ crs_1 = _NetCDFVar("crs_1")
+ crs_2 = _NetCDFVar("crs_2")
+ lat = _NetCDFVar("lat")
+ height = _NetCDFVar("height")
+ vars_all = {
+ "crs_1": crs_1,
+ "crs_2": crs_2,
+ "lat": lat,
+ "height": height,
+ "ref_source": ref_source,
+ }
+
+ cs_mappings = {"ref_source": {"lat": "crs_1", "height": "crs_2"}}
+
+ result = CFGridMappingVariable.identify(
+ vars_all, coord_system_mappings=cs_mappings
+ )
+ assert set(result.keys()) == {"crs_1", "crs_2"}
+
+ def test_partial_coords_missing_cs_still_identified(self):
+ """One coord present, one missing for the same CRS: CRS still identified
+ (has_a_valid_coord is True from the present coord), but a warning is
+ issued for the missing one.
+ """
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, "grid_mapping", "crs_1")
+ crs_1 = _NetCDFVar("crs_1")
+ lat = _NetCDFVar("lat")
+ # lon is intentionally absent from vars_all
+ vars_all = {
+ "crs_1": crs_1,
+ "lat": lat,
+ "ref_source": ref_source,
+ }
+
+ cs_mappings = {"ref_source": {"lat": "crs_1", "lon": "crs_1"}}
+
+ warn_regex = r"Missing CF-netCDF coordinate variable 'lon'.*"
+ with pytest.warns(iris.warnings.IrisCfMissingVarWarning, match=warn_regex):
+ result = CFGridMappingVariable.identify(
+ vars_all, coord_system_mappings=cs_mappings
+ )
+ # CRS still in result because lat was valid.
+ assert "crs_1" in result
+
+ def test_all_coords_missing_cs_excluded(self):
+ """All coords missing for a CRS: has_a_valid_coord stays False, CRS excluded."""
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, "grid_mapping", "crs_1")
+ crs_1 = _NetCDFVar("crs_1")
+ # Both lat and lon absent from vars_all
+ vars_all = {"crs_1": crs_1, "ref_source": ref_source}
+
+ cs_mappings = {"ref_source": {"lat": "crs_1", "lon": "crs_1"}}
+
+ warn_regex = "Missing CF-netCDF coordinate variable"
+ with pytest.warns(iris.warnings.IrisCfMissingVarWarning, match=warn_regex):
+ result = CFGridMappingVariable.identify(
+ vars_all, coord_system_mappings=cs_mappings
+ )
+ assert result == {}
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFGroup.py b/lib/iris/tests/unit/fileformats/cf/test_CFGroup.py
index 3724a2f628..987936fe73 100644
--- a/lib/iris/tests/unit/fileformats/cf/test_CFGroup.py
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFGroup.py
@@ -4,108 +4,186 @@
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for the :class:`iris.fileformats.cf.CFGroup` class."""
+from typing import Callable
from unittest.mock import MagicMock
import pytest
-from iris.fileformats.cf import (
- CFAuxiliaryCoordinateVariable,
- CFCoordinateVariable,
- CFDataVariable,
- CFGroup,
- CFUGridAuxiliaryCoordinateVariable,
- CFUGridConnectivityVariable,
- CFUGridMeshVariable,
+from iris.fileformats import cf
+from iris.fileformats.cf import _variables
+
+VariableMap: dict[type[cf.CFVariable], Callable] = {
+ _variables._CFFormulaTermsVariable: cf.CFGroup.formula_terms,
+ cf.CFAncillaryDataVariable: cf.CFGroup.ancillary_variables,
+ cf.CFAuxiliaryCoordinateVariable: cf.CFGroup.auxiliary_coordinates,
+ cf.CFBoundaryVariable: cf.CFGroup.bounds,
+ cf.CFClimatologyVariable: cf.CFGroup.climatology,
+ cf.CFCoordinateVariable: cf.CFGroup.coordinates,
+ cf.CFDataVariable: cf.CFGroup.data_variables,
+ cf.CFGridMappingVariable: cf.CFGroup.grid_mappings,
+ cf.CFLabelVariable: cf.CFGroup.labels,
+ cf.CFMeasureVariable: cf.CFGroup.cell_measures,
+ cf.CFUGridAuxiliaryCoordinateVariable: cf.CFGroup.ugrid_coords,
+ cf.CFUGridConnectivityVariable: cf.CFGroup.connectivities,
+ cf.CFUGridMeshVariable: cf.CFGroup.meshes,
+}
+
+
+def get_mocked_var(class_: type[cf.CFVariable], mocker) -> MagicMock:
+ cf_name = f"{class_.__name__}_var"
+ mocked = mocker.MagicMock(spec=class_, cf_name=cf_name)
+ return mocked
+
+
+@pytest.fixture(
+ params=[
+ (class_, scenario)
+ for class_ in VariableMap
+ for scenario in ("single", "duplicates", "multiple")
+ ],
+ ids=lambda value: f"{value[0].__name__}-{value[1]}",
)
+def mock_variables(
+ request, mocker
+) -> tuple[type[cf.CFVariable], tuple[MagicMock, ...]]:
+ class_, scenario = request.param
+ mocked = get_mocked_var(class_, mocker)
+ variables = [mocked]
+
+ if scenario == "duplicates":
+ variables.append(get_mocked_var(class_, mocker))
+ elif scenario == "multiple":
+ mocked_2 = get_mocked_var(class_, mocker)
+ mocked_2.cf_name = f"{mocked.cf_name}_2"
+ variables.append(mocked_2)
+
+ return class_, tuple(variables)
+
+
+@pytest.fixture(params=["single", "duplicates", "multiple"])
+def mock_variables_all(request, mocker) -> tuple[MagicMock, ...]:
+ scenario = request.param
+ all_variables = []
+
+ for class_ in VariableMap:
+ mocked = get_mocked_var(class_, mocker)
+ variables = [mocked]
+
+ if scenario == "duplicates":
+ variables.append(get_mocked_var(class_, mocker))
+ elif scenario == "multiple":
+ mocked_2 = get_mocked_var(class_, mocker)
+ mocked_2.cf_name = f"{mocked.cf_name}_2"
+ variables.append(mocked_2)
+
+ all_variables.extend(variables)
+
+ return tuple(all_variables)
+
+
+@pytest.fixture
+def cf_group() -> cf.CFGroup:
+ return cf.CFGroup()
+
+
+@pytest.fixture
+def cf_group_populated(cf_group, mock_variables_all) -> cf.CFGroup:
+ for mocked in mock_variables_all:
+ cf_group[mocked.cf_name] = mocked
+ return cf_group
+
+
+class TestProperties:
+ def test_common(self, cf_group, mock_variables):
+ class_, variables = mock_variables
+ for mocked in variables:
+ cf_group[mocked.cf_name] = mocked
+
+ property_ = VariableMap[class_]
+ result = property_.fget(cf_group)
+ expected_names = {var.cf_name for var in variables}
+ assert len(result) == len(expected_names)
+ for expected_name in expected_names:
+ assert expected_name in result
+
+ def test_non_data_names(self, cf_group_populated, mock_variables_all):
+ expected = {
+ mocked.cf_name
+ for mocked in mock_variables_all
+ if not isinstance(
+ mocked, (cf.CFDataVariable, _variables._CFFormulaTermsVariable)
+ )
+ }
+ assert cf_group_populated.non_data_variable_names == expected
-class Tests:
- # TODO: unit tests for existing functionality pre 2021-03-11.
+class TestReturns:
@pytest.fixture(autouse=True)
- def _setup(self):
- self.cf_group = CFGroup()
+ def _setup(self, cf_group_populated, mock_variables_all):
+ self.variables = mock_variables_all
+ self.cf_group = cf_group_populated
+
+ def test_keys(self):
+ expected_names = {mocked.cf_name for mocked in self.variables}
+ assert set(self.cf_group.keys()) == expected_names
+
+ def test_len(self):
+ expected_names = {mocked.cf_name for mocked in self.variables}
+ assert len(self.cf_group) == len(expected_names)
+
+ def test_iter(self):
+ expected_names = {mocked.cf_name for mocked in self.variables}
+ actual_names = set()
+ for name in self.cf_group:
+ actual_names.add(name)
+
+ assert actual_names == expected_names
+
+ def test_getitem(self):
+ expected_by_name = {}
+ for mocked in self.variables:
+ expected_by_name[mocked.cf_name] = mocked
+
+ for expected_name, expected_variable in expected_by_name.items():
+ assert self.cf_group[expected_name] is expected_variable
+
+ with pytest.raises(KeyError, match="Cannot get unknown CF-netCDF variable"):
+ _ = self.cf_group["unknown_name"]
+
+ def test_repr(self):
+ self.cf_group.global_attributes["global_attr"] = "value"
+ self.cf_group.promoted["promoted_var"] = self.variables[0]
+
+ expected_names = {mocked.cf_name for mocked in self.variables}
+ expected = (
+ "<CFGroup of "
+ f"variables:{len(expected_names)}, "
+ "global_attributes:1, "
+ "promoted:1>"
+ )
+ assert repr(self.cf_group) == expected
- def test_non_data_names(self):
- data_var = MagicMock(spec=CFDataVariable, cf_name="data_var")
- aux_var = MagicMock(spec=CFAuxiliaryCoordinateVariable, cf_name="aux_var")
- coord_var = MagicMock(spec=CFCoordinateVariable, cf_name="coord_var")
- coord_var2 = MagicMock(spec=CFCoordinateVariable, cf_name="coord_var2")
- duplicate_name_var = MagicMock(spec=CFCoordinateVariable, cf_name="aux_var")
- for var in (
- data_var,
- aux_var,
- coord_var,
- coord_var2,
- duplicate_name_var,
- ):
- self.cf_group[var.cf_name] = var
+class TestMutations:
+ def test_setitem(self, cf_group, mock_variables_all):
+ mocked = mock_variables_all[0]
- expected_names = [var.cf_name for var in (aux_var, coord_var, coord_var2)]
- expected = set(expected_names)
- assert self.cf_group.non_data_variable_names == expected
+ cf_group[mocked.cf_name] = mocked
+ assert cf_group[mocked.cf_name] is mocked
+ with pytest.raises(TypeError, match="Attempted to add an invalid"):
+ cf_group[mocked.cf_name] = object()
-class TestUgrid:
- """Separate class to test UGRID functionality."""
+ with pytest.raises(ValueError, match="Mismatch between key name"):
+ cf_group[f"{mocked.cf_name}_mismatch"] = mocked
- @pytest.fixture(autouse=True)
- def _setup(self):
- self.cf_group = CFGroup()
-
- def test_inherited(self):
- coord_var = MagicMock(spec=CFCoordinateVariable, cf_name="coord_var")
- self.cf_group[coord_var.cf_name] = coord_var
- assert self.cf_group.coordinates[coord_var.cf_name] == coord_var
-
- def test_connectivities(self):
- conn_var = MagicMock(spec=CFUGridConnectivityVariable, cf_name="conn_var")
- self.cf_group[conn_var.cf_name] = conn_var
- assert self.cf_group.connectivities[conn_var.cf_name] == conn_var
-
- def test_ugrid_coords(self):
- coord_var = MagicMock(
- spec=CFUGridAuxiliaryCoordinateVariable, cf_name="coord_var"
- )
- self.cf_group[coord_var.cf_name] = coord_var
- assert self.cf_group.ugrid_coords[coord_var.cf_name] == coord_var
-
- def test_meshes(self):
- mesh_var = MagicMock(spec=CFUGridMeshVariable, cf_name="mesh_var")
- self.cf_group[mesh_var.cf_name] = mesh_var
- assert self.cf_group.meshes[mesh_var.cf_name] == mesh_var
-
- def test_non_data_names(self):
- data_var = MagicMock(spec=CFDataVariable, cf_name="data_var")
- coord_var = MagicMock(spec=CFCoordinateVariable, cf_name="coord_var")
- conn_var = MagicMock(spec=CFUGridConnectivityVariable, cf_name="conn_var")
- ugrid_coord_var = MagicMock(
- spec=CFUGridAuxiliaryCoordinateVariable, cf_name="ugrid_coord_var"
- )
- mesh_var = MagicMock(spec=CFUGridMeshVariable, cf_name="mesh_var")
- mesh_var2 = MagicMock(spec=CFUGridMeshVariable, cf_name="mesh_var2")
- duplicate_name_var = MagicMock(spec=CFUGridMeshVariable, cf_name="coord_var")
-
- for var in (
- data_var,
- coord_var,
- conn_var,
- ugrid_coord_var,
- mesh_var,
- mesh_var2,
- duplicate_name_var,
- ):
- self.cf_group[var.cf_name] = var
-
- expected_names = [
- var.cf_name
- for var in (
- coord_var,
- conn_var,
- ugrid_coord_var,
- mesh_var,
- mesh_var2,
- )
- ]
- expected = set(expected_names)
- assert self.cf_group.non_data_variable_names == expected
+ def test_delitem(self, cf_group_populated, mock_variables_all):
+ expected_names = {mocked.cf_name for mocked in mock_variables_all}
+ name_to_delete = next(iter(expected_names))
+
+ del cf_group_populated[name_to_delete]
+ assert name_to_delete not in cf_group_populated
+ assert len(cf_group_populated) == len(expected_names) - 1
+
+ with pytest.raises(KeyError, match="Cannot delete unknown CF-netcdf"):
+ del cf_group_populated["unknown_name"]
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFLabelVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFLabelVariable.py
new file mode 100644
index 0000000000..c59de8072f
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFLabelVariable.py
@@ -0,0 +1,75 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFLabelVariable`."""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf import CFDataVariable, CFLabelVariable
+
+from .identify_mixins import (
+ IdentifyByAttributeMixin,
+ SpansMixin,
+ _NetCDFVar,
+ _NetCDFVarWithDimensions,
+)
+
+
+class TestIdentify(IdentifyByAttributeMixin):
+ __test__ = True
+
+ CF_CLASS = CFLabelVariable
+ CF_IDENTITIES = ["coordinates"]
+ MISSING_WARN_REGEX = r"Missing CF-netCDF label variable {subject!r}.*"
+ SUBJECT_DTYPE_DEFAULT = np.bytes_
+
+ def test_non_string_ref_ignored(self):
+ # Label identify should reject non-string referenced variables.
+ subject_name = "ref_subject"
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], subject_name)
+ vars_all = {
+ subject_name: _NetCDFVar(subject_name, dtype=int),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == {}
+
+
+class TestCfLabelDimensions:
+ def _make_label_var(self, label_dims):
+ stub = _NetCDFVarWithDimensions("label_var", label_dims, dtype=np.bytes_)
+ return CFLabelVariable("label_var", stub)
+
+ def _make_data_var(self, data_dims):
+ stub = _NetCDFVarWithDimensions("data_var", data_dims)
+ return CFDataVariable("data_var", stub)
+
+ def test_raises_for_non_cfdata_var(self):
+ label_var = self._make_label_var(("x",))
+ message = "cf_data_var argument should be of type CFDataVariable"
+ with pytest.raises(TypeError, match=message):
+ label_var.cf_label_dimensions(object())
+
+ def test_returns_overlap_dimensions(self):
+ label_var = self._make_label_var(("x", "strlen"))
+ data_var = self._make_data_var(("x", "y"))
+ result = label_var.cf_label_dimensions(data_var)
+ assert result == ("x",)
+
+ def test_no_overlap_returns_empty(self):
+ label_var = self._make_label_var(("a", "b"))
+ data_var = self._make_data_var(("x", "y"))
+ result = label_var.cf_label_dimensions(data_var)
+ assert result == ()
+
+
+class TestSpans(SpansMixin):
+ """Tests for CFLabelVariable.spans()."""
+
+ __test__ = True
+ CF_CLASS = CFLabelVariable
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFMeasureVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFMeasureVariable.py
new file mode 100644
index 0000000000..68a920e861
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFMeasureVariable.py
@@ -0,0 +1,182 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFMeasureVariable`."""
+
+import warnings
+
+import pytest
+
+from iris.fileformats.cf import CFMeasureVariable
+import iris.warnings
+
+from .identify_mixins import _NetCDFVar, assert_warning_gated
+
+CF_IDENTITY = "cell_measures"
+
+
+class TestIdentify:
+ def test_one_measure_ref(self):
+ subject_name = "ref_subject"
+ ref_subject = _NetCDFVar(subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, f"area: {subject_name}")
+ vars_all = {
+ subject_name: ref_subject,
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ expected = {subject_name: CFMeasureVariable(subject_name, ref_subject, "area")}
+ result = CFMeasureVariable.identify(vars_all)
+ assert result == expected
+
+ def test_measure_stored_on_instance(self):
+ subject_name = "ref_subject"
+ ref_subject = _NetCDFVar(subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, f"volume: {subject_name}")
+ vars_all = {
+ subject_name: ref_subject,
+ "ref_source": ref_source,
+ }
+
+ result = CFMeasureVariable.identify(vars_all)
+ assert result[subject_name].cf_measure == "volume"
+
+ def test_multi_term(self):
+ subject_names = ("ref_area", "ref_volume")
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
+ ref_source = _NetCDFVar("ref_source")
+ setattr(
+ ref_source,
+ CF_IDENTITY,
+ f"area: {subject_names[0]} volume: {subject_names[1]}",
+ )
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ **ref_subject_vars,
+ }
+
+ result = CFMeasureVariable.identify(vars_all)
+ assert set(result.keys()) == set(subject_names)
+ assert result[subject_names[0]].cf_measure == "area"
+ assert result[subject_names[1]].cf_measure == "volume"
+
+ def test_two_refs(self):
+ """Two source variables each referencing a different measure variable."""
+ subject_names = ("ref_area", "ref_volume")
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
+
+ ref_source_vars = {
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
+ }
+ for ix, var in enumerate(ref_source_vars.values()):
+ setattr(var, CF_IDENTITY, f"area: {subject_names[ix]}")
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ **ref_subject_vars,
+ **ref_source_vars,
+ }
+
+ expected = {
+ name: CFMeasureVariable(name, var, "area")
+ for name, var in ref_subject_vars.items()
+ }
+ result = CFMeasureVariable.identify(vars_all)
+ assert result == expected
+
+ def test_self_reference_ignored(self):
+ """A variable cannot reference itself as a cell measure."""
+ nc_var = _NetCDFVar("self_ref")
+ setattr(nc_var, CF_IDENTITY, "area: self_ref")
+ vars_all = {
+ "self_ref": nc_var,
+ }
+
+ result = CFMeasureVariable.identify(vars_all)
+ assert result == {}
+
+ def test_ignore(self):
+ subject_names = ("ref_area", "ref_volume")
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
+
+ ref_source_vars = {
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
+ }
+ for ix, var in enumerate(ref_source_vars.values()):
+ setattr(var, CF_IDENTITY, f"area: {subject_names[ix]}")
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ **ref_subject_vars,
+ **ref_source_vars,
+ }
+
+ expected_name = subject_names[0]
+ expected = {
+ expected_name: CFMeasureVariable(
+ expected_name, ref_subject_vars[expected_name], "area"
+ )
+ }
+ result = CFMeasureVariable.identify(vars_all, ignore=subject_names[1])
+ assert result == expected
+
+ def test_target(self):
+ subject_names = ("ref_area", "ref_volume")
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
+
+ source_names = ("ref_source_1", "ref_source_2")
+ ref_source_vars = {name: _NetCDFVar(name) for name in source_names}
+ for ix, var in enumerate(ref_source_vars.values()):
+ setattr(var, CF_IDENTITY, f"area: {subject_names[ix]}")
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ **ref_subject_vars,
+ **ref_source_vars,
+ }
+
+ expected_name = subject_names[0]
+ expected = {
+ expected_name: CFMeasureVariable(
+ expected_name, ref_subject_vars[expected_name], "area"
+ )
+ }
+ result = CFMeasureVariable.identify(vars_all, target=source_names[0])
+ assert result == expected
+
+ def test_target_unknown_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Cannot identify unknown target CF-netCDF variable 'unknown'"
+ with pytest.raises(ValueError, match=message):
+ CFMeasureVariable.identify(vars_all, target="unknown")
+
+ def test_target_wrong_type_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Expect a target CF-netCDF variable name"
+ with pytest.raises(TypeError, match=message):
+ CFMeasureVariable.identify(vars_all, target=object())
+
+ def test_warn(self):
+ subject_name = "ref_subject"
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, f"area: {subject_name}")
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ def operation(warn: bool):
+ warnings.warn(
+ "emit at least 1 warning",
+ category=iris.warnings.IrisUserWarning,
+ )
+ CFMeasureVariable.identify(vars_all, warn=warn)
+
+ warn_regex = rf"Missing CF-netCDF measure variable {subject_name!r}.*"
+ assert_warning_gated(
+ operation, iris.warnings.IrisCfMissingVarWarning, warn_regex
+ )
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFReader.py b/lib/iris/tests/unit/fileformats/cf/test_CFReader.py
index c5d0e068bf..2aa0e4ee29 100644
--- a/lib/iris/tests/unit/fileformats/cf/test_CFReader.py
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFReader.py
@@ -4,12 +4,18 @@
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for the `iris.fileformats.cf.CFReader` class."""
-from unittest import mock
+import contextlib
+import io
import numpy as np
import pytest
+import iris
+import iris.exceptions
+from iris.fileformats import cf
from iris.fileformats.cf import (
+ CFAuxiliaryCoordinateVariable,
+ CFBoundaryVariable,
CFCoordinateVariable,
CFDataVariable,
CFGridMappingVariable,
@@ -19,9 +25,21 @@
CFUGridConnectivityVariable,
CFUGridMeshVariable,
)
+import iris.warnings
+# CFVariable.attributes, and now identify() too, are built from a variable's
+# ncattrs()/getncattr() - see CFVariable.__init__ and CFReader._translate.
+# `netcdf_variable` below wires ncattrs()/getncattr() to a presence-keyed
+# dict of the CF-attribute keywords the caller passed here - fixed at this
+# call. A test that mutates or deletes one of these named attributes on the
+# returned mock afterwards - as a few in this file do, to simulate a variable
+# missing an attribute - changes only the raw Mock attribute, which neither
+# identify() nor `.attributes` ever reads again. To vary what a variable's
+# attributes are, pass the value to a fresh `netcdf_variable(...)` call
+# instead - see e.g. `test_derived_bounds_promotes_reference_terms` below.
def netcdf_variable(
+ mocker,
name,
dimensions,
dtype,
@@ -33,6 +51,12 @@ def netcdf_variable(
grid_mapping=None,
cell_measures=None,
standard_name=None,
+ long_name=None,
+ mesh=None,
+ cf_role=None,
+ node_coordinates=None,
+ face_coordinates=None,
+ face_node_connectivity=None,
):
"""Return a mock NetCDF4 variable."""
ndim = 0
@@ -41,18 +65,17 @@ def netcdf_variable(
ndim = len(dimensions)
else:
dimensions = []
+ # Arbitrary but real: NetCDFDatasetVariable.shape/.ndim read straight
+ # through to these, once this mock is wrapped rather than read directly.
+ shape = tuple(1 for _ in range(ndim))
+ size = 1
ugrid_identities = (
CFUGridAuxiliaryCoordinateVariable.cf_identities
+ CFUGridConnectivityVariable.cf_identities
+ [CFUGridMeshVariable.cf_identity]
)
- ncvar = mock.Mock(
- name=name,
- dimensions=dimensions,
- ncattrs=mock.Mock(return_value=[]),
- ndim=ndim,
- dtype=dtype,
+ members = dict(
ancillary_variables=ancillary_variables,
coordinates=coordinates,
bounds=bounds,
@@ -61,18 +84,48 @@ def netcdf_variable(
grid_mapping=grid_mapping,
cell_measures=cell_measures,
standard_name=standard_name,
+ long_name=long_name,
+ cf_role=cf_role,
**{name: None for name in ugrid_identities},
)
+ # Each of these is one of the UGRID identities the sweep above defaults
+ # to None; the explicit keywords are what let a test set them.
+ members["mesh"] = mesh
+ members["node_coordinates"] = node_coordinates
+ members["face_coordinates"] = face_coordinates
+ members["face_node_connectivity"] = face_node_connectivity
+ # A None member stood for "no such attribute" when identify() read these
+ # with getattr(..., None). Now that it reads a mapping, absent means
+ # absent - so a None must not be listed.
+ attributes = {key: value for key, value in members.items() if value is not None}
+ ncvar = mocker.Mock(
+ name=name,
+ dimensions=dimensions,
+ ndim=ndim,
+ shape=shape,
+ size=size,
+ dtype=dtype,
+ ncattrs=mocker.Mock(return_value=list(attributes)),
+ getncattr=mocker.Mock(side_effect=attributes.__getitem__),
+ # A few tests in this file build a CFVariable straight from this
+ # mock, bypassing NetCDFDatasetVariable entirely; CFVariable.__init__
+ # reads `.attributes` directly, so it has to be present here too -
+ # harmless for the usual path, where NetCDFDatasetVariable computes
+ # its own `.attributes` from ncattrs()/getncattr() and never looks at
+ # this one.
+ attributes=attributes,
+ **members,
+ )
return ncvar
class Test_translate__global_attributes:
@pytest.fixture(autouse=True)
def _setup(self, mocker):
- ncvar = netcdf_variable("ncvar", "height", np.float64)
- ncattrs = mock.Mock(return_value=["dimensions"])
- getncattr = mock.Mock(return_value="something something_else")
- dataset = mock.Mock(
+ ncvar = netcdf_variable(mocker, "ncvar", "height", np.float64)
+ ncattrs = mocker.Mock(return_value=["dimensions"])
+ getncattr = mocker.Mock(return_value="something something_else")
+ dataset = mocker.Mock(
file_format="NetCDF4",
variables={"ncvar": ncvar},
ncattrs=ncattrs,
@@ -91,14 +144,23 @@ def test_create_global_attributes(self, mocker):
class Test_translate__formula_terms:
@pytest.fixture(autouse=True)
def _setup(self, mocker):
- self.delta = netcdf_variable("delta", "height", np.float64, bounds="delta_bnds")
- self.delta_bnds = netcdf_variable("delta_bnds", "height bnds", np.float64)
- self.sigma = netcdf_variable("sigma", "height", np.float64, bounds="sigma_bnds")
- self.sigma_bnds = netcdf_variable("sigma_bnds", "height bnds", np.float64)
- self.orography = netcdf_variable("orography", "lat lon", np.float64)
+ self.delta = netcdf_variable(
+ mocker, "delta", "height", np.float64, bounds="delta_bnds"
+ )
+ self.delta_bnds = netcdf_variable(
+ mocker, "delta_bnds", "height bnds", np.float64
+ )
+ self.sigma = netcdf_variable(
+ mocker, "sigma", "height", np.float64, bounds="sigma_bnds"
+ )
+ self.sigma_bnds = netcdf_variable(
+ mocker, "sigma_bnds", "height bnds", np.float64
+ )
+ self.orography = netcdf_variable(mocker, "orography", "lat lon", np.float64)
formula_terms = "a: delta b: sigma orog: orography"
standard_name = "atmosphere_hybrid_height_coordinate"
self.height = netcdf_variable(
+ mocker,
"height",
"height",
np.float64,
@@ -110,16 +172,17 @@ def _setup(self, mocker):
# which will be ignored by the cf loader.
formula_terms = "a: delta_bnds b: sigma_bnds orog: orography"
self.height_bnds = netcdf_variable(
+ mocker,
"height_bnds",
"height bnds",
np.float64,
formula_terms=formula_terms,
)
- self.lat = netcdf_variable("lat", "lat", np.float64)
- self.lon = netcdf_variable("lon", "lon", np.float64)
+ self.lat = netcdf_variable(mocker, "lat", "lat", np.float64)
+ self.lon = netcdf_variable(mocker, "lon", "lon", np.float64)
# Note that, only lat and lon are explicitly associated as coordinates.
self.temp = netcdf_variable(
- "temp", "height lat lon", np.float64, coordinates="lat lon"
+ mocker, "temp", "height lat lon", np.float64, coordinates="lat lon"
)
self.variables = dict(
@@ -134,8 +197,8 @@ def _setup(self, mocker):
sigma_bnds=self.sigma_bnds,
height_bnds=self.height_bnds,
)
- ncattrs = mock.Mock(return_value=[])
- self.dataset = mock.Mock(
+ ncattrs = mocker.Mock(return_value=[])
+ self.dataset = mocker.Mock(
file_format="NetCDF4", variables=self.variables, ncattrs=ncattrs
)
# Restrict the CFReader functionality to only performing translations.
@@ -153,21 +216,21 @@ def test_create_formula_terms(self, mocker):
group = cf_group.data_variables
assert len(group) == 1
assert list(group.keys()) == ["temp"]
- assert group["temp"].cf_data is self.temp
+ assert group["temp"].cf_data.variable is self.temp
# Check there are three coordinates.
group = cf_group.coordinates
assert len(group) == 3
coordinates = ["height", "lat", "lon"]
assert set(group.keys()) == set(coordinates)
for name in coordinates:
- assert group[name].cf_data is getattr(self, name)
+ assert group[name].cf_data.variable is getattr(self, name)
# Check there are three auxiliary coordinates.
group = cf_group.auxiliary_coordinates
assert len(group) == 3
aux_coordinates = ["delta", "sigma", "orography"]
assert set(group.keys()) == set(aux_coordinates)
for name in aux_coordinates:
- assert group[name].cf_data is getattr(self, name)
+ assert group[name].cf_data.variable is getattr(self, name)
# Check all the auxiliary coordinates are formula terms.
formula_terms = cf_group.formula_terms
assert set(group.items()) == set(formula_terms.items())
@@ -177,20 +240,29 @@ def test_create_formula_terms(self, mocker):
bounds = ["height_bnds", "delta_bnds", "sigma_bnds"]
assert set(group.keys()) == set(bounds)
for name in bounds:
- assert group[name].cf_data == getattr(self, name)
+ assert group[name].cf_data.variable is getattr(self, name)
class Test_build_cf_groups__formula_terms:
@pytest.fixture(autouse=True)
def _setup(self, mocker):
- self.delta = netcdf_variable("delta", "height", np.float64, bounds="delta_bnds")
- self.delta_bnds = netcdf_variable("delta_bnds", "height bnds", np.float64)
- self.sigma = netcdf_variable("sigma", "height", np.float64, bounds="sigma_bnds")
- self.sigma_bnds = netcdf_variable("sigma_bnds", "height bnds", np.float64)
- self.orography = netcdf_variable("orography", "lat lon", np.float64)
+ self.delta = netcdf_variable(
+ mocker, "delta", "height", np.float64, bounds="delta_bnds"
+ )
+ self.delta_bnds = netcdf_variable(
+ mocker, "delta_bnds", "height bnds", np.float64
+ )
+ self.sigma = netcdf_variable(
+ mocker, "sigma", "height", np.float64, bounds="sigma_bnds"
+ )
+ self.sigma_bnds = netcdf_variable(
+ mocker, "sigma_bnds", "height bnds", np.float64
+ )
+ self.orography = netcdf_variable(mocker, "orography", "lat lon", np.float64)
formula_terms = "a: delta b: sigma orog: orography"
standard_name = "atmosphere_hybrid_height_coordinate"
self.height = netcdf_variable(
+ mocker,
"height",
"height",
np.float64,
@@ -202,18 +274,19 @@ def _setup(self, mocker):
# which will be ignored by the cf loader.
formula_terms = "a: delta_bnds b: sigma_bnds orog: orography"
self.height_bnds = netcdf_variable(
+ mocker,
"height_bnds",
"height bnds",
np.float64,
formula_terms=formula_terms,
)
- self.lat = netcdf_variable("lat", "lat", np.float64)
- self.lon = netcdf_variable("lon", "lon", np.float64)
- self.x = netcdf_variable("x", "lat lon", np.float64)
- self.y = netcdf_variable("y", "lat lon", np.float64)
+ self.lat = netcdf_variable(mocker, "lat", "lat", np.float64)
+ self.lon = netcdf_variable(mocker, "lon", "lon", np.float64)
+ self.x = netcdf_variable(mocker, "x", "lat lon", np.float64)
+ self.y = netcdf_variable(mocker, "y", "lat lon", np.float64)
# Note that, only lat and lon are explicitly associated as coordinates.
self.temp = netcdf_variable(
- "temp", "height lat lon", np.float64, coordinates="x y"
+ mocker, "temp", "height lat lon", np.float64, coordinates="x y"
)
self.variables = dict(
@@ -230,8 +303,8 @@ def _setup(self, mocker):
x=self.x,
y=self.y,
)
- ncattrs = mock.Mock(return_value=[])
- self.dataset = mock.Mock(
+ ncattrs = mocker.Mock(return_value=[])
+ self.dataset = mocker.Mock(
file_format="NetCDF4", variables=self.variables, ncattrs=ncattrs
)
# Restrict the CFReader functionality to only performing translations
@@ -242,6 +315,8 @@ def _setup(self, mocker):
return_value=self.dataset,
)
+ self.wibble = netcdf_variable(mocker, "wibble", "lat wibble", np.float64)
+
def test_associate_formula_terms_with_data_variable(self, mocker):
cf_group = CFReader("dummy").cf_group
assert len(cf_group) == len(self.variables)
@@ -255,19 +330,19 @@ def test_associate_formula_terms_with_data_variable(self, mocker):
coordinates = ["height", "lat", "lon"]
assert set(group.keys()) == set(coordinates)
for name in coordinates:
- assert group[name].cf_data is getattr(self, name)
+ assert group[name].cf_data.variable is getattr(self, name)
# Check the height coordinate is bounded.
group = group["height"].cf_group
assert len(group.bounds) == 1
assert "height_bnds" in group.bounds
- assert group["height_bnds"].cf_data is self.height_bnds
+ assert group["height_bnds"].cf_data.variable is self.height_bnds
# Check there are five auxiliary coordinates.
group = temp_cf_group.auxiliary_coordinates
assert len(group) == 5
aux_coordinates = ["delta", "sigma", "orography", "x", "y"]
assert set(group.keys()) == set(aux_coordinates)
for name in aux_coordinates:
- assert group[name].cf_data is getattr(self, name)
+ assert group[name].cf_data.variable is getattr(self, name)
# Check all the auxiliary coordinates are formula terms.
formula_terms = cf_group.formula_terms
assert set(formula_terms.items()).issubset(list(group.items()))
@@ -279,7 +354,9 @@ def test_associate_formula_terms_with_data_variable(self, mocker):
aux_coord_group = group[name].cf_group
assert len(aux_coord_group.bounds) == 1
assert name_bnds in aux_coord_group.bounds
- assert aux_coord_group[name_bnds].cf_data is getattr(self, name_bnds)
+ assert aux_coord_group[name_bnds].cf_data.variable is getattr(
+ self, name_bnds
+ )
def test_promote_reference(self):
cf_group = CFReader("dummy").cf_group
@@ -296,7 +373,7 @@ def test_promote_reference(self):
coordinates = ("lat", "lon")
assert set(group.keys()) == set(coordinates)
for name in coordinates:
- assert group[name].cf_data == getattr(self, name)
+ assert group[name].cf_data.variable is getattr(self, name)
def test_formula_terms_ignore(self):
self.orography.dimensions = ["lat", "wibble"]
@@ -304,7 +381,7 @@ def test_formula_terms_ignore(self):
cf_group = CFReader("dummy").cf_group
group = cf_group.promoted
assert list(group.keys()) == ["orography"]
- assert group["orography"].cf_data == self.orography
+ assert group["orography"].cf_data.variable is self.orography
def test_auxiliary_ignore(self):
self.x.dimensions = ["lat", "wibble"]
@@ -314,12 +391,16 @@ def test_auxiliary_ignore(self):
group = cf_group.promoted
assert set(group.keys()) == set(promoted)
for name in promoted:
- assert group[name].cf_data == getattr(self, name)
+ assert group[name].cf_data.variable is getattr(self, name)
- def test_promoted_auxiliary_ignore(self):
- self.wibble = netcdf_variable("wibble", "lat wibble", np.float64)
+ def test_promoted_auxiliary_ignore(self, mocker):
self.variables["wibble"] = self.wibble
- self.orography.coordinates = "wibble"
+ # "coordinates" must be set at construction - see the module-level
+ # comment on netcdf_variable().
+ self.orography = netcdf_variable(
+ mocker, "orography", "lat lon", np.float64, coordinates="wibble"
+ )
+ self.variables["orography"] = self.orography
with pytest.warns(match="Ignoring variable wibble") as warns:
cf_group = CFReader("dummy").cf_group.promoted
@@ -327,7 +408,7 @@ def test_promoted_auxiliary_ignore(self):
promoted = ["wibble", "orography"]
assert set(cf_group.keys()) == set(promoted)
for name in promoted:
- assert cf_group[name].cf_data == getattr(self, name)
+ assert cf_group[name].cf_data.variable is getattr(self, name)
# we should have got 2 warnings
assert len(warns.list) == 2
@@ -336,25 +417,41 @@ class Test_build_cf_groups__ugrid:
@pytest.fixture(autouse=True)
def _setup_class(self, mocker):
# Replicating syntax from test_CFReader.Test_build_cf_groups__formula_terms.
- self.mesh = netcdf_variable("mesh", "", int)
- self.node_x = netcdf_variable("node_x", "node", float)
- self.node_y = netcdf_variable("node_y", "node", float)
- self.face_x = netcdf_variable("face_x", "face", float)
- self.face_y = netcdf_variable("face_y", "face", float)
- self.face_nodes = netcdf_variable("face_nodes", "face vertex", int)
- self.levels = netcdf_variable("levels", "levels", int)
+ # Mesh-recognition attributes are passed at construction, not set
+ # afterwards: identify() now reads through .attributes, which is
+ # fixed when netcdf_variable() builds ncattrs()/getncattr() - see the
+ # module-level comment above.
+ self.mesh = netcdf_variable(
+ mocker,
+ "mesh",
+ "",
+ int,
+ cf_role="mesh_topology",
+ node_coordinates="node_x node_y",
+ face_coordinates="face_x face_y",
+ face_node_connectivity="face_nodes",
+ )
+ self.node_x = netcdf_variable(mocker, "node_x", "node", float)
+ self.node_y = netcdf_variable(mocker, "node_y", "node", float)
+ self.face_x = netcdf_variable(mocker, "face_x", "face", float)
+ self.face_y = netcdf_variable(mocker, "face_y", "face", float)
+ self.face_nodes = netcdf_variable(
+ mocker,
+ "face_nodes",
+ "face vertex",
+ int,
+ cf_role="face_node_connectivity",
+ )
+ self.levels = netcdf_variable(mocker, "levels", "levels", int)
self.data = netcdf_variable(
- "data", "levels face", float, coordinates="face_x face_y"
+ mocker,
+ "data",
+ "levels face",
+ float,
+ coordinates="face_x face_y",
+ mesh="mesh",
)
- # Add necessary attributes for mesh recognition.
- self.mesh.cf_role = "mesh_topology"
- self.mesh.node_coordinates = "node_x node_y"
- self.mesh.face_coordinates = "face_x face_y"
- self.mesh.face_node_connectivity = "face_nodes"
- self.face_nodes.cf_role = "face_node_connectivity"
- self.data.mesh = "mesh"
-
self.variables = dict(
mesh=self.mesh,
node_x=self.node_x,
@@ -365,8 +462,8 @@ def _setup_class(self, mocker):
levels=self.levels,
data=self.data,
)
- ncattrs = mock.Mock(return_value=[])
- self.dataset = mock.Mock(
+ ncattrs = mocker.Mock(return_value=[])
+ self.dataset = mocker.Mock(
file_format="NetCDF4", variables=self.variables, ncattrs=ncattrs
)
@@ -413,11 +510,12 @@ def test_is_cf_ugrid_group(self):
class Test_build_cf_groups__nczarr_scalar_grid_mapping:
@pytest.fixture(autouse=True)
def _setup_class(self, mocker):
- self.lat = netcdf_variable("lat", "lat", np.float64)
- self.lon = netcdf_variable("lon", "lon", np.float64)
- self.crs = netcdf_variable("crs", "_scalar_", np.int32)
+ self.lat = netcdf_variable(mocker, "lat", "lat", np.float64)
+ self.lon = netcdf_variable(mocker, "lon", "lon", np.float64)
+ self.crs = netcdf_variable(mocker, "crs", "_scalar_", np.int32)
self.crs.grid_mapping_name = "latitude_longitude"
self.temp = netcdf_variable(
+ mocker,
"temp",
"lat lon",
np.float64,
@@ -435,8 +533,8 @@ def _setup_class(self, mocker):
"crs": self.crs,
"temp": self.temp,
}
- ncattrs = mock.Mock(return_value=[])
- self.dataset = mock.Mock(
+ ncattrs = mocker.Mock(return_value=[])
+ self.dataset = mocker.Mock(
file_format="NetCDF4", variables=self.variables, ncattrs=ncattrs
)
mocker.patch("iris.fileformats.cf.CFReader._reset")
@@ -454,3 +552,569 @@ def test_nczarr_scalar_grid_mapping_retains_type(self):
def test_nczarr_scalar_grid_mapping_spans_data_var(self):
temp_cf_group = self.cf_group["temp"].cf_group
assert "crs" in temp_cf_group.grid_mappings
+
+
+def test_destructor(tmp_path):
+ """Test the destructor when reading the dataset fails.
+ Related to issue #3312: previously, the `CFReader` would
+ always call `close()` on its `_dataset` attribute, even if it
+ didn't exist because opening the dataset had failed.
+ """
+ fn = tmp_path / "tmp.nc"
+ with fn.open("wb+") as fh:
+ fh.write(b"\x89HDF\r\n\x1a\nBroken file with correct signature")
+ fh.flush()
+
+ with io.StringIO() as buf:
+ with contextlib.redirect_stderr(buf):
+ try:
+ _ = cf.CFReader(str(fn))
+ except OSError:
+ pass
+ try:
+ _ = iris.load_cubes(str(fn))
+ except OSError:
+ pass
+ buf.seek(0)
+ assert buf.read() == ""
+
+
+class Test_init_and_lifecycle:
+ @pytest.fixture(autouse=True)
+ def _setup(self, mocker):
+ self.variables = {"x": netcdf_variable(mocker, "x", "x", np.float64)}
+ self.dataset = mocker.Mock(
+ file_format="NetCDF4",
+ variables=self.variables,
+ ncattrs=mocker.Mock(return_value=[]),
+ filepath=mocker.Mock(return_value="in-memory.nc"),
+ set_auto_chartostring=mocker.Mock(),
+ isopen=mocker.Mock(return_value=True),
+ )
+ self.encoded_ds = mocker.patch(
+ "iris.fileformats.netcdf._bytecoding_datasets.EncodedDataset",
+ return_value=self.dataset,
+ )
+ mocker.patch("iris.fileformats.cf.CFReader._translate")
+ mocker.patch("iris.fileformats.cf.CFReader._build_cf_groups")
+ mocker.patch("iris.fileformats.cf.CFReader._reset")
+
+ def test_init_with_url_source_preserves_filename_string(self):
+ url = "https://example.com/some/file.nc"
+ reader = CFReader(url)
+
+ self.encoded_ds.assert_called_once_with(url, mode="r")
+ assert reader.filename == url
+ assert repr(reader) == f"CFReader('{url}')"
+
+ def test_init_uses_dataset_wrapper_when_string_decode_disabled(self, mocker):
+ mocker.patch(
+ "iris.fileformats.netcdf._bytecoding_datasets.DECODE_TO_STRINGS_ON_READ",
+ False,
+ )
+ wrapper_ds = mocker.patch(
+ "iris.fileformats.netcdf._thread_safe_nc.DatasetWrapper",
+ return_value=self.dataset,
+ )
+
+ reader = CFReader("dummy.nc")
+
+ assert reader.filename.name == "dummy.nc"
+ wrapper_ds.assert_called_once()
+ self.encoded_ds.assert_not_called()
+
+ def test_init_with_open_dataset_does_not_close_on_context_exit(self):
+ with CFReader(self.dataset) as reader:
+ assert reader is not None
+ assert reader.filename == "in-memory.nc"
+
+ self.dataset.close.assert_not_called()
+
+ def test_init_warns_for_netcdf3_when_requested(self):
+ self.dataset.file_format = "NETCDF3_CLASSIC"
+
+ with pytest.warns(iris.warnings.IrisLoadWarning, match="Optimise CF-netCDF"):
+ CFReader("dummy.nc", warn=True)
+
+ def test_init_warns_for_netcdf3_when_requested_for_a_borrowed_dataset(self):
+ # The owned-path equivalent above pins CFReader passing warn= through
+ # to NetCDFDataset; this pins the same wiring on the from_existing
+ # branch, which is a separate call site in __init__.
+ self.dataset.file_format = "NETCDF3_CLASSIC"
+
+ with pytest.warns(iris.warnings.IrisLoadWarning, match="Optimise CF-netCDF"):
+ CFReader(self.dataset, warn=True)
+
+ def test_init_with_no_meshes_trims_ugrid_variable_types(self, mocker):
+ self.dataset.variables = {
+ "a": netcdf_variable(mocker, "a", "x", np.float64),
+ "b": netcdf_variable(mocker, "b", "x", np.float64, mesh="my_mesh"),
+ }
+
+ reader = CFReader("dummy.nc")
+
+ assert reader._with_ugrid is True
+
+ mesh_free = mocker.Mock(
+ file_format="NetCDF4",
+ variables={"a": netcdf_variable(mocker, "a", "x", np.float64)},
+ ncattrs=mocker.Mock(return_value=[]),
+ filepath=mocker.Mock(return_value="in-memory.nc"),
+ set_auto_chartostring=mocker.Mock(),
+ )
+ self.encoded_ds.return_value = mesh_free
+ reader = CFReader("dummy.nc")
+
+ assert reader._with_ugrid is False
+ assert CFUGridMeshVariable not in reader._variable_types
+
+
+class Test_translate__grid_mapping_parse_errors:
+ @pytest.fixture(autouse=True)
+ def _setup(self, mocker):
+ self.lat = netcdf_variable(mocker, "lat", "lat", np.float64)
+ self.lon = netcdf_variable(mocker, "lon", "lon", np.float64)
+ self.temp = netcdf_variable(
+ mocker,
+ "temp",
+ "lat lon",
+ np.float64,
+ coordinates="lat lon",
+ grid_mapping="bad syntax",
+ )
+ self.temp.name = "temp"
+ variables = {"lat": self.lat, "lon": self.lon, "temp": self.temp}
+ self.dataset = mocker.Mock(
+ file_format="NetCDF4",
+ variables=variables,
+ ncattrs=mocker.Mock(return_value=[]),
+ )
+ mocker.patch(
+ "iris.fileformats.netcdf._bytecoding_datasets.EncodedDataset",
+ return_value=self.dataset,
+ )
+ mocker.patch(
+ "iris.fileformats.cf._reader.hh._parse_extended_grid_mapping",
+ side_effect=iris.exceptions.CFParseError("failed to parse"),
+ )
+
+ def test_parse_failure_warns_and_reader_continues(self):
+ with pytest.warns(
+ iris.warnings.IrisCfWarning,
+ match=r"Error parsing `grid_mapping` attribute for temp: failed to parse",
+ ):
+ cf_group = CFReader("dummy.nc").cf_group
+
+ assert "temp" in cf_group.data_variables
+
+
+class Test_translate__formula_terms_derived_bounds:
+ @pytest.fixture(autouse=True)
+ def _setup(self, mocker):
+ self.term = netcdf_variable(mocker, "term", "z", np.float64)
+ self.root = netcdf_variable(
+ mocker,
+ "z",
+ "z",
+ np.float64,
+ formula_terms="a: term",
+ bounds="z_bnds",
+ standard_name="atmosphere_hybrid_height_coordinate",
+ )
+ self.root_bnds = netcdf_variable(
+ mocker,
+ "z_bnds",
+ "z bnds",
+ np.float64,
+ formula_terms="a: term",
+ )
+ self.data = netcdf_variable(mocker, "temp", "z", np.float64)
+ self.variables = {
+ "z": self.root,
+ "z_bnds": self.root_bnds,
+ "term": self.term,
+ "temp": self.data,
+ }
+
+ @staticmethod
+ def _make_dataset(mocker, variables):
+ return mocker.Mock(
+ file_format="NetCDF4",
+ variables=variables,
+ ncattrs=mocker.Mock(return_value=[]),
+ )
+
+ def _patch_encoded_dataset(self, mocker):
+ dataset = self._make_dataset(mocker, self.variables)
+ mocker.patch(
+ "iris.fileformats.netcdf._bytecoding_datasets.EncodedDataset",
+ return_value=dataset,
+ )
+
+ @pytest.fixture(params=[True, False], ids=["FUTURE", "not_FUTURE"])
+ def future_context(self, request):
+ if request.param:
+ result = iris.FUTURE.context(derived_bounds=True)
+ else:
+ result = contextlib.nullcontext()
+ return result
+
+ def test_derived_bounds_term(self, mocker, future_context):
+ self._patch_encoded_dataset(mocker)
+
+ with future_context:
+ cf_group = CFReader("dummy.nc").cf_group
+
+ assert "term" in cf_group.formula_terms
+ assert isinstance(cf_group["term"], CFAuxiliaryCoordinateVariable)
+
+ def test_derived_bounds_skips_when_term_missing(self, mocker, future_context):
+ # "formula_terms" must be set at construction - see the module-level
+ # comment on netcdf_variable().
+ self.root_bnds = netcdf_variable(
+ mocker, "z_bnds", "z bnds", np.float64, formula_terms="a: missing_term"
+ )
+ self.variables["z_bnds"] = self.root_bnds
+ self._patch_encoded_dataset(mocker)
+
+ with future_context:
+ cf_group = CFReader("dummy.nc").cf_group
+
+ assert "term" in cf_group.formula_terms
+ assert isinstance(cf_group["term"], CFAuxiliaryCoordinateVariable)
+
+ def test_promotes_non_formula_root_bounds_to_data(self, mocker):
+ self.root_bnds = netcdf_variable(mocker, "z_bnds", "_scalar_", np.float64)
+ # With valid formula terms, the variable would instead be recorded
+ # correctly as a bounds variable.
+ self.variables["z_bnds"] = self.root_bnds
+ self._patch_encoded_dataset(mocker)
+
+ with iris.FUTURE.context(derived_bounds=True):
+ cf_group = CFReader("dummy.nc").cf_group
+
+ assert "z_bnds" in cf_group.promoted
+ assert cf_group["z"].bounds is None
+
+ def test_reclassifies_formula_term_bounds_variable(self, mocker, future_context):
+ # If not referenced by any formula terms, the bounds variable is
+ # promoted to a data variable.
+ # Referenced
+ delta = netcdf_variable(
+ mocker, "delta", "height", np.float64, bounds="delta_bnds"
+ )
+ # Referenced
+ sigma = netcdf_variable(
+ mocker, "sigma", "height", np.float64, bounds="sigma_bnds"
+ )
+ formula_terms = "a: delta b: sigma"
+ height = netcdf_variable(
+ mocker,
+ "height",
+ "height",
+ np.float64,
+ formula_terms=formula_terms,
+ bounds="height_bnds",
+ standard_name="atmosphere_hybrid_height_coordinate",
+ )
+ # Referenced
+ formula_terms_bnds = "a: delta_bnds b: sigma_bnds"
+ height_bnds = netcdf_variable(
+ mocker,
+ "height_bnds",
+ "height bnds",
+ np.float64,
+ formula_terms=formula_terms_bnds,
+ )
+ delta_bnds = netcdf_variable(mocker, "delta_bnds", "height bnds", np.float64)
+ sigma_bnds = netcdf_variable(mocker, "sigma_bnds", "height bnds", np.float64)
+ data = netcdf_variable(mocker, "temp", "height", np.float64)
+ variables = {
+ "delta": delta,
+ "sigma": sigma,
+ "height": height,
+ "height_bnds": height_bnds,
+ "delta_bnds": delta_bnds,
+ "sigma_bnds": sigma_bnds,
+ "temp": data,
+ }
+ self.variables = variables
+ self._patch_encoded_dataset(mocker)
+
+ with future_context:
+ cf_group = CFReader("dummy.nc").cf_group
+
+ assert isinstance(cf_group["delta_bnds"], CFBoundaryVariable)
+ assert isinstance(cf_group["sigma_bnds"], CFBoundaryVariable)
+ assert cf_group["delta"].bounds == "delta_bnds"
+ assert cf_group["sigma"].bounds == "sigma_bnds"
+
+ def test_formula_term_already_in_group_uses_existing_variable(
+ self, mocker, future_context
+ ):
+ self.root = netcdf_variable(
+ mocker,
+ "z",
+ "z",
+ np.float64,
+ formula_terms="a: pressure",
+ standard_name="atmosphere_hybrid_height_coordinate",
+ )
+ self.pressure = netcdf_variable(mocker, "pressure", "pressure", np.float64)
+ self.variables = {"z": self.root, "pressure": self.pressure, "temp": self.data}
+ self._patch_encoded_dataset(mocker)
+
+ with future_context:
+ cf_group = CFReader("dummy.nc").cf_group
+
+ # When existing elsewhere, a variable referenced by a formula term is
+ # NOT created as an aux coord as it usually would. Instead the existing
+ # variable is re-used.
+ assert cf_group.formula_terms["pressure"] is cf_group.coordinates["pressure"]
+ assert "pressure" not in cf_group.auxiliary_coordinates
+
+ @pytest.mark.parametrize(
+ "standard_name", [False, True], ids=["no_standard_name", "with_standard_name"]
+ )
+ def test_derived_bounds_promotes_reference_terms(
+ self, mocker, future_context, standard_name
+ ):
+ if not standard_name and isinstance(future_context, contextlib.nullcontext):
+ pytest.skip("Test only applicable when FUTURE context is enabled.")
+ # CFVariable.attributes is a snapshot taken at construction, so the
+ # "absent" case is built without standard_name from the start rather
+ # than deleted from the mock afterwards - a post-construction `del`
+ # is invisible to a CFVariable built from this mock either way.
+ self.root = netcdf_variable(
+ mocker,
+ "z",
+ "z",
+ np.float64,
+ formula_terms="a: pressure",
+ standard_name="custom_reference" if standard_name else None,
+ )
+ self.pressure = netcdf_variable(mocker, "pressure", "z", np.float64)
+ self.variables = {"z": self.root, "pressure": self.pressure, "temp": self.data}
+ self._patch_encoded_dataset(mocker)
+ # reference_terms supports hybrid heights. Variables that are both
+ # named in formula_terms and referenced in reference_terms - "a" in
+ # this case - are always promoted.
+ mocker.patch.dict(
+ "iris.fileformats.cf.reference_terms",
+ # Real world example: {"atmosphere_sigma_coordinate": ["ps"]}
+ {"custom_reference": "a"},
+ clear=False,
+ )
+
+ with future_context:
+ cf_group = CFReader("dummy.nc").cf_group
+
+ if standard_name:
+ assert "pressure" in cf_group.promoted
+ else:
+ # Promotion step is skipped if standard_name is absent.
+ assert "pressure" not in cf_group.promoted
+
+ def test_promotes_using_long_name_when_standard_name_empty(self, mocker):
+ # A present-but-empty standard_name is falsy, so the "or" in
+ # _reader.py's formula_terms promotion loop falls through to long_name.
+ self.root = netcdf_variable(
+ mocker,
+ "z",
+ "z",
+ np.float64,
+ formula_terms="a: pressure",
+ standard_name="",
+ long_name="custom_reference",
+ )
+ self.pressure = netcdf_variable(mocker, "pressure", "z", np.float64)
+ self.variables = {"z": self.root, "pressure": self.pressure, "temp": self.data}
+ self._patch_encoded_dataset(mocker)
+ mocker.patch.dict(
+ "iris.fileformats.cf.reference_terms",
+ {"custom_reference": "a"},
+ clear=False,
+ )
+
+ cf_group = CFReader("dummy.nc").cf_group
+
+ assert "pressure" in cf_group.promoted
+
+ def test_raises_when_standard_name_empty_and_long_name_absent(self, mocker):
+ # The one case ``.get("long_name")`` would survive: standard_name
+ # present-but-falsy forces the "or" to evaluate long_name, and with
+ # no long_name at all the subscript must raise KeyError.
+ self.root = netcdf_variable(
+ mocker,
+ "z",
+ "z",
+ np.float64,
+ formula_terms="a: pressure",
+ standard_name="",
+ )
+ self.pressure = netcdf_variable(mocker, "pressure", "z", np.float64)
+ self.variables = {"z": self.root, "pressure": self.pressure, "temp": self.data}
+ self._patch_encoded_dataset(mocker)
+
+ with pytest.raises(KeyError, match="long_name"):
+ CFReader("dummy.nc")
+
+
+class Test_translate__global_attributes_missing:
+ @pytest.fixture(autouse=True)
+ def _setup(self, mocker):
+ self.var = netcdf_variable(mocker, "x", "x", np.float64)
+ self.dataset = mocker.Mock(
+ file_format="NetCDF4",
+ variables={"x": self.var},
+ ncattrs=mocker.Mock(return_value=["history"]),
+ getncattr=mocker.Mock(side_effect=AttributeError),
+ )
+ mocker.patch(
+ "iris.fileformats.netcdf._bytecoding_datasets.EncodedDataset",
+ return_value=self.dataset,
+ )
+ mocker.patch("iris.fileformats.cf.CFReader._build_cf_groups")
+ mocker.patch("iris.fileformats.cf.CFReader._reset")
+
+ def test_translate_global_attr_missing_falls_back_to_default(self):
+ cf_group = CFReader("dummy.nc").cf_group
+
+ assert cf_group.global_attributes["history"] == ""
+
+
+class Test_build_cf_groups__private_edge_cases:
+ # Achieving full coverage requires a limited amount of internal state
+ # manipulation (to be avoided where possible as it hurts future refactoring).
+
+ @pytest.fixture(autouse=True)
+ def _setup(self):
+ self.reader = object.__new__(CFReader)
+ self.reader._own_file = False
+ self.reader._dataset = None
+ self.reader._variable_types = ()
+ self.reader._coord_system_mappings = {}
+ self.reader.cf_group = CFGroup()
+
+ def test_derived_bounds_relinks_spanning_bounds(self, mocker):
+ data = netcdf_variable(
+ mocker,
+ "temp",
+ "z",
+ np.float64,
+ bounds="temp_bnds",
+ standard_name="atmosphere_hybrid_height_coordinate",
+ )
+ data.__len__ = mocker.Mock(return_value=1)
+ data_var = CFDataVariable(
+ "temp",
+ data,
+ )
+ bnds_data = netcdf_variable(mocker, "temp_bnds", "z bnds", np.float64)
+ bnds_data.__len__ = mocker.Mock(return_value=1)
+ bnds_var = CFBoundaryVariable(
+ "temp_bnds",
+ bnds_data,
+ )
+ self.reader.cf_group["temp"] = data_var
+ self.reader.cf_group["temp_bnds"] = bnds_var
+
+ with iris.FUTURE.context(derived_bounds=True):
+ self.reader._build_cf_groups({})
+
+ assert "temp_bnds" in self.reader.cf_group["temp"].cf_group.bounds
+
+ @pytest.mark.parametrize("guard_fires", [False, True])
+ def test_derived_bounds_boundary_guard_continue_branch(self, mocker, guard_fires):
+ # The branch under test is dead in production - see
+ # https://github.com/SciTools/iris/issues/7296 - so it can only be
+ # reached by patching the name the identity check compares against.
+ #
+ # Both cases are run because the guard's only effect is to skip the
+ # promotion below it, so promotion is the sole observable that
+ # distinguishes them. Asserting anything else - that the term is still
+ # in cf_group.formula_terms, say - holds either way, and would leave
+ # the test passing while the branch went unexercised.
+ root = CFCoordinateVariable(
+ "z",
+ netcdf_variable(
+ mocker,
+ "z",
+ "z",
+ np.float64,
+ standard_name="atmosphere_hybrid_height_coordinate",
+ ),
+ )
+ term = CFAuxiliaryCoordinateVariable(
+ "orog", netcdf_variable(mocker, "orog", "z", np.float64)
+ )
+ # "orog" is the reference surface term for this standard name, so
+ # without the guard it qualifies for promotion to a CFDataVariable.
+ term.add_formula_term("z", "orog")
+ self.reader.cf_group["z"] = root
+ self.reader.cf_group["orog"] = term
+
+ if guard_fires:
+ # Force the exact identity check in CFReader._build_cf_groups. The
+ # target must name the module whose global CFReader reads, not the
+ # iris.fileformats.cf re-export, or the patch has no effect at all
+ # and the guard never fires - which is what the False case detects.
+ mocker.patch("iris.fileformats.cf._reader.CFBoundaryVariable", term)
+
+ with iris.FUTURE.context(derived_bounds=True):
+ self.reader._build_cf_groups({})
+
+ if guard_fires:
+ assert "orog" not in self.reader.cf_group.promoted
+ else:
+ assert "orog" in self.reader.cf_group.promoted
+
+
+class TestSynthesisedBoundsLink:
+ """Pin the write-and-read-the-same-place invariant at unit level.
+
+ This class does not exercise `_reader.py` - it builds a
+ `CFAuxiliaryCoordinateVariable` directly and characterises
+ `TrackedAttributes` plus `CFVariable.__getattr__`: a write to
+ ``cf_var.attributes`` is a write to the one place
+ ``cf_var.<attribute>`` reads from, including when the written value is
+ `None`. `_reader.py`'s own synthesised-bounds writes at :327 and :345
+ are exercised by
+ `lib/iris/tests/integration/netcdf/derived_bounds/test_bounds_files.py`,
+ which fails three ways if either write is removed.
+ """
+
+ def test_a_written_bounds_link_is_visible_to_a_reader(self, mocker):
+ nc_var = mocker.MagicMock()
+ nc_var.attributes = {"units": "m"}
+ nc_var.dimensions = ("model_level_number",)
+ cf_var = CFAuxiliaryCoordinateVariable("a", nc_var)
+
+ assert "bounds" not in cf_var.attributes
+ cf_var.attributes["bounds"] = "a_bnds"
+
+ assert cf_var.attributes["bounds"] == "a_bnds"
+ assert "bounds" in cf_var.attributes
+ assert cf_var.bounds == "a_bnds"
+
+ def test_a_written_bounds_link_overrides_the_file(self, mocker):
+ nc_var = mocker.MagicMock()
+ nc_var.attributes = {"bounds": "from_file"}
+ nc_var.dimensions = ("model_level_number",)
+ cf_var = CFAuxiliaryCoordinateVariable("a", nc_var)
+
+ cf_var.attributes["bounds"] = "synthesised"
+ assert cf_var.bounds == "synthesised"
+
+ def test_a_bounds_link_set_to_none_still_reads_as_present(self, mocker):
+ # CFReader._translate invalidates a broken link by setting it to None
+ # rather than deleting it, and the reads downstream check presence
+ # before value.
+ nc_var = mocker.MagicMock()
+ nc_var.attributes = {"bounds": "broken"}
+ nc_var.dimensions = ("model_level_number",)
+ cf_var = CFAuxiliaryCoordinateVariable("a", nc_var)
+
+ cf_var.attributes["bounds"] = None
+ assert "bounds" in cf_var.attributes
+ assert cf_var.attributes.get("bounds") is None
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py b/lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py
new file mode 100644
index 0000000000..2d91747616
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py
@@ -0,0 +1,230 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Tests for :class:`iris.fileformats.cf.CFReader` reading a real file.
+
+The rest of the CFReader tests drive it with mocks, which is the right shape
+for its classification logic and the wrong shape for the question here: what
+type the reader actually hands out, and what a load does with it. These use a
+small file on disk instead.
+"""
+
+import numpy as np
+import pytest
+
+import iris
+from iris.fileformats.cf import CFReader
+from iris.fileformats.cf.dataset import CFDatasetVariable
+from iris.fileformats.netcdf import _dataset, _thread_safe_nc
+
+UNREAD_COMMENT = "an attribute nothing in Iris reads"
+
+
+@pytest.fixture
+def sample_path(tmp_path):
+ """Write a small CF file: a data variable, a coordinate and its bounds."""
+ path = tmp_path / "sample.nc"
+ dataset = _thread_safe_nc.DatasetWrapper(path, mode="w")
+ dataset.title = "a sample file"
+ dataset.createDimension("time", 3)
+ dataset.createDimension("bnds", 2)
+
+ time = dataset.createVariable("time", "f8", ("time",))
+ time.standard_name = "time"
+ time.units = "days since 1970-01-01"
+ time.bounds = "time_bnds"
+ time.comment = UNREAD_COMMENT
+ time[:] = np.arange(3, dtype="f8")
+
+ bounds = dataset.createVariable("time_bnds", "f8", ("time", "bnds"))
+ bounds[:] = np.zeros((3, 2))
+
+ air = dataset.createVariable("air", "f4", ("time",))
+ air.standard_name = "air_temperature"
+ air.units = "K"
+ air.coordinates = "time"
+ air.comment = UNREAD_COMMENT
+ air[:] = np.arange(3, dtype="f4")
+
+ dataset.close()
+ return path
+
+
+@pytest.fixture
+def char_path(tmp_path):
+ """Write a small CF file with a character auxiliary coordinate.
+
+ A separate fixture from ``sample_path``: adding a char variable there
+ would change what ``test_only_unread_attributes_reach_the_cube`` and its
+ neighbours load.
+ """
+ path = tmp_path / "char.nc"
+ dataset = _thread_safe_nc.DatasetWrapper(path, mode="w")
+ dataset.createDimension("x", 3)
+ dataset.createDimension("strlen", 3)
+
+ labels = dataset.createVariable("labels", "S1", ("x", "strlen"))
+ labels._Encoding = "utf-8"
+ labels[:] = np.array([list(s) for s in ("aaa", "bbb", "ccc")], dtype="S1")
+
+ temp = dataset.createVariable("temp", "f4", ("x",))
+ temp.standard_name = "air_temperature"
+ temp.units = "K"
+ temp.coordinates = "labels"
+ temp[:] = np.arange(3, dtype="f4")
+
+ dataset.close()
+ return path
+
+
+class TestABorrowedBareDataset:
+ """A behaviour change made in #7303.
+
+ Before #7303, ``CFReader`` stored a borrowed dataset untouched
+ (``self._dataset = file_source``), so a bare ``netCDF4.Dataset`` - what
+ ncdata hands to ``iris.load``, per ``loader.py``'s docstring -
+ kept its character data as raw bytes. ``NetCDFDataset.from_existing`` now
+ wraps anything lacking ``THREAD_SAFE_FLAG`` in an ``EncodedDataset``
+ regardless of the caller's own decoding preference, so the same bare
+ dataset's character data is decoded to Python strings instead. This is
+ deliberate - the new behaviour is better, and a char auxiliary coordinate
+ that used to be silently dropped by ``iris.load`` now loads - but it is a
+ genuine, previously-undocumented change on a public entry point, so this
+ pins it.
+ """
+
+ def test_char_data_is_decoded(self, char_path):
+ # A genuinely bare, unwrapped dataset - not a DatasetWrapper, not an
+ # EncodedDataset - is the point of this test.
+ import netCDF4
+
+ raw = netCDF4.Dataset(char_path, mode="r")
+ try:
+ with CFReader(raw) as reader:
+ labels = reader.cf_group["labels"]
+ assert labels.dtype.kind == "U"
+ assert labels.shape == (3,)
+ np.testing.assert_array_equal(labels[:], ["aaa", "bbb", "ccc"])
+ finally:
+ raw.close()
+
+
+class TestTheSwap:
+ def test_the_reader_owns_a_netcdf_dataset(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ assert isinstance(reader._dataset, _dataset.NetCDFDataset)
+
+ def test_cf_data_is_a_cf_dataset_variable(self, sample_path):
+ # Nothing downstream of here needs to know the file is netCDF.
+ with CFReader(str(sample_path)) as reader:
+ assert isinstance(reader.cf_group["air"].cf_data, CFDatasetVariable)
+
+ def test_attributes_come_from_the_file(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ air = reader.cf_group["air"]
+ assert air.attributes["units"] == "K"
+ assert air.units == "K"
+ assert air.dimensions == ("time",)
+
+ def test_global_attributes_come_from_the_dataset(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ assert reader.cf_group.global_attributes["title"] == "a sample file"
+
+ def test_filename_is_the_variables_location(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ assert reader.cf_group["air"].filename == str(sample_path)
+
+ def test_the_bounds_variable_was_classified(self, sample_path):
+ # identify() now reads through .attributes, so a miss here means the
+ # classification pass lost sight of the file's attributes entirely.
+ with CFReader(str(sample_path)) as reader:
+ assert "time_bnds" in reader.cf_group.bounds
+
+ def test_a_borrowed_dataset_is_used_and_not_closed(self, sample_path):
+ raw = _thread_safe_nc.DatasetWrapper(sample_path, mode="r")
+ try:
+ with CFReader(raw) as reader:
+ assert reader.cf_group["air"].units == "K"
+ assert raw.isopen()
+ finally:
+ raw.close()
+
+
+class TestAttributesAreASnapshot:
+ """``NetCDFDatasetVariable.attributes`` writes through to the file, which is
+ right for the saver and wrong for the reader: CFReader synthesises a
+ "bounds" link during load, on a file it opened read-only. CFVariable
+ therefore takes a copy.
+ """
+
+ def test_a_write_does_not_reach_the_file(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ air = reader.cf_group["air"]
+ air.attributes["bounds"] = "invented"
+
+ assert "bounds" not in air.cf_data.attributes
+
+
+class TestAttributeTrackingAcrossReset:
+ """CFReader reads attributes while classifying variables, then calls
+ ``cf_attrs_reset()`` so the rules start from a clean record. Until #7303,
+ ``__getattr__`` cached the value on the instance, so a read after the
+ reset found the cache and was never recorded - and an attribute Iris had
+ in fact consumed was still reported unused, and so was copied onto the
+ cube as if the file had volunteered it.
+ """
+
+ def test_the_reader_resets_what_it_read_while_classifying(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ time = reader.cf_group["time"]
+ # CFBoundaryVariable.identify() read "bounds" during __init__.
+ # Ask through .untracked, so that asking does not itself record.
+ assert "bounds" in time.attributes.untracked
+ assert dict(time.cf_attrs_used()) == dict(time.cf_attrs_ignored())
+
+ def test_a_read_after_the_reset_is_recorded(self, sample_path):
+ with CFReader(str(sample_path)) as reader:
+ time = reader.cf_group["time"]
+
+ assert time.units == "days since 1970-01-01"
+ assert "units" in dict(time.cf_attrs_used())
+ assert "units" not in dict(time.cf_attrs_unused())
+
+ def test_an_unread_attribute_reaches_the_cube(self, sample_path):
+ cube = iris.load_cube(str(sample_path))
+
+ # "title" is a global attribute: build_and_add_global_attributes
+ # copies cf_group.global_attributes onto the cube unconditionally,
+ # untouched by the per-variable used/unused tracking this test
+ # otherwise pins - so it is expected here alongside the coordinate
+ # variable's genuinely-unread "comment".
+ assert cube.attributes == {"title": "a sample file", "comment": UNREAD_COMMENT}
+ assert cube.coord("time").attributes == {"comment": UNREAD_COMMENT}
+
+ # This does not pin the reverse: that no attribute Iris *read* reaches
+ # the cube. Every attribute read here is in saver.py's _CF_ATTRS, and
+ # _add_unused_attributes drops those before tracking is consulted.
+
+
+def test_an_already_open_cf_dataset_is_used_as_is(tmp_path):
+ """A loader that has already opened a store hands it straight over.
+
+ The alternative - a path and a format keyword - would put the choice of
+ backend inside CFReader, which is the branch the dataset interface exists
+ to remove.
+ """
+ from iris.fileformats.netcdf._dataset import NetCDFDataset
+
+ path = str(tmp_path / "source.nc")
+ with NetCDFDataset(path, mode="w") as writer:
+ writer.create_dimension("x", 2)
+ writer.create_variable("v", np.dtype("f4"), ("x",))
+
+ dataset = NetCDFDataset(path, mode="r")
+ with CFReader(dataset) as reader:
+ assert reader.filename == dataset.location
+ assert "v" in reader.cf_group
+ # Borrowed, not owned: leaving the context must not close it.
+ assert dataset.closed is False
+ dataset.close()
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFUGridAuxiliaryCoordinateVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFUGridAuxiliaryCoordinateVariable.py
index 630359f681..444b6a4594 100644
--- a/lib/iris/tests/unit/fileformats/cf/test_CFUGridAuxiliaryCoordinateVariable.py
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFUGridAuxiliaryCoordinateVariable.py
@@ -4,216 +4,38 @@
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for :class:`iris.fileformats.cf.CFUGridAuxiliaryCoordinateVariable`."""
-import re
-import warnings
-
-import numpy as np
-import pytest
-
from iris.fileformats.cf import CFUGridAuxiliaryCoordinateVariable
-from iris.tests.unit.fileformats.cf.test_CFReader import netcdf_variable
-import iris.warnings
-
-
-def named_variable(name):
- # Don't need to worry about dimensions or dtype for these tests.
- return netcdf_variable(name, "", int)
-
-
-class TestIdentify:
- @pytest.fixture(autouse=True)
- def _setup(self):
- self.cf_identities = [
- "node_coordinates",
- "edge_coordinates",
- "face_coordinates",
- "volume_coordinates",
- ]
-
- def test_cf_identities(self):
- subject_name = "ref_subject"
- ref_subject = named_variable(subject_name)
- vars_common = {
- subject_name: ref_subject,
- "ref_not_subject": named_variable("ref_not_subject"),
- }
- # ONLY expecting ref_subject, excluding ref_not_subject.
- expected = {
- subject_name: CFUGridAuxiliaryCoordinateVariable(subject_name, ref_subject)
- }
-
- for identity in self.cf_identities:
- ref_source = named_variable("ref_source")
- setattr(ref_source, identity, subject_name)
- vars_all = dict({"ref_source": ref_source}, **vars_common)
- result = CFUGridAuxiliaryCoordinateVariable.identify(vars_all)
- assert expected == result
-
- def test_duplicate_refs(self):
- subject_name = "ref_subject"
- ref_subject = named_variable(subject_name)
- ref_source_vars = {
- name: named_variable(name) for name in ("ref_source_1", "ref_source_2")
- }
- for var in ref_source_vars.values():
- setattr(var, self.cf_identities[0], subject_name)
- vars_all = dict(
- {
- subject_name: ref_subject,
- "ref_not_subject": named_variable("ref_not_subject"),
- },
- **ref_source_vars,
- )
- # ONLY expecting ref_subject, excluding ref_not_subject.
- expected = {
- subject_name: CFUGridAuxiliaryCoordinateVariable(subject_name, ref_subject)
- }
- result = CFUGridAuxiliaryCoordinateVariable.identify(vars_all)
- assert expected == result
+from .identify_mixins import IdentifyByAttributeListMixin, _NetCDFVar
- def test_two_coords(self):
- subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
- ref_source_vars = {
- name: named_variable(name) for name in ("ref_source_1", "ref_source_2")
- }
- for ix, var in enumerate(ref_source_vars.values()):
- setattr(var, self.cf_identities[ix], subject_names[ix])
- vars_all = dict(
- {"ref_not_subject": named_variable("ref_not_subject")},
- **ref_subject_vars,
- **ref_source_vars,
- )
+class TestIdentify(IdentifyByAttributeListMixin):
+ __test__ = True
- # Not expecting ref_not_subject.
- expected = {
- name: CFUGridAuxiliaryCoordinateVariable(name, var)
- for name, var in ref_subject_vars.items()
- }
- result = CFUGridAuxiliaryCoordinateVariable.identify(vars_all)
- assert expected == result
+ CF_CLASS = CFUGridAuxiliaryCoordinateVariable
+ CF_IDENTITIES = [
+ "node_coordinates",
+ "edge_coordinates",
+ "face_coordinates",
+ "volume_coordinates",
+ ]
+ MISSING_WARN_REGEX = r"Missing CF-netCDF auxiliary coordinate variable {subject}.*"
def test_two_part_ref(self):
+ # UGRID auxiliary coordinate attributes can contain multiple refs.
subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
- ref_source = named_variable("ref_source")
- setattr(ref_source, self.cf_identities[0], " ".join(subject_names))
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], " ".join(subject_names))
vars_all = {
- "ref_not_subject": named_variable("ref_not_subject"),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
"ref_source": ref_source,
**ref_subject_vars,
}
expected = {
- name: CFUGridAuxiliaryCoordinateVariable(name, var)
- for name, var in ref_subject_vars.items()
- }
- result = CFUGridAuxiliaryCoordinateVariable.identify(vars_all)
- assert expected == result
-
- def test_string_type_ignored(self):
- subject_name = "ref_subject"
- ref_source = named_variable("ref_source")
- setattr(ref_source, self.cf_identities[0], subject_name)
- vars_all = {
- subject_name: netcdf_variable(subject_name, "", np.bytes_),
- "ref_not_subject": named_variable("ref_not_subject"),
- "ref_source": ref_source,
+ name: self.CF_CLASS(name, var) for name, var in ref_subject_vars.items()
}
-
- result = CFUGridAuxiliaryCoordinateVariable.identify(vars_all)
- assert {} == result
-
- def test_ignore(self):
- subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
-
- ref_source_vars = {
- name: named_variable(name) for name in ("ref_source_1", "ref_source_2")
- }
- for ix, var in enumerate(ref_source_vars.values()):
- setattr(var, self.cf_identities[0], subject_names[ix])
- vars_all = dict(
- {"ref_not_subject": named_variable("ref_not_subject")},
- **ref_subject_vars,
- **ref_source_vars,
- )
-
- # ONLY expect the subject variable that hasn't been ignored.
- expected_name = subject_names[0]
- expected = {
- expected_name: CFUGridAuxiliaryCoordinateVariable(
- expected_name, ref_subject_vars[expected_name]
- )
- }
- result = CFUGridAuxiliaryCoordinateVariable.identify(
- vars_all, ignore=subject_names[1]
- )
- assert expected == result
-
- def test_target(self):
- subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
-
- source_names = ("ref_source_1", "ref_source_2")
- ref_source_vars = {name: named_variable(name) for name in source_names}
- for ix, var in enumerate(ref_source_vars.values()):
- setattr(var, self.cf_identities[0], subject_names[ix])
- vars_all = dict(
- {"ref_not_subject": named_variable("ref_not_subject")},
- **ref_subject_vars,
- **ref_source_vars,
- )
-
- # ONLY expect the variable referenced by the named ref_source_var.
- expected_name = subject_names[0]
- expected = {
- expected_name: CFUGridAuxiliaryCoordinateVariable(
- expected_name, ref_subject_vars[expected_name]
- )
- }
- result = CFUGridAuxiliaryCoordinateVariable.identify(
- vars_all, target=source_names[0]
- )
- assert expected == result
-
- def test_warn(self):
- subject_name = "ref_subject"
- ref_source = named_variable("ref_source")
- setattr(ref_source, self.cf_identities[0], subject_name)
- vars_all = {
- "ref_not_subject": named_variable("ref_not_subject"),
- "ref_source": ref_source,
- }
-
- def operation(warn: bool):
- warnings.warn(
- "emit at least 1 warning",
- category=iris.warnings.IrisUserWarning,
- )
- result = CFUGridAuxiliaryCoordinateVariable.identify(vars_all, warn=warn)
- assert {} == result
-
- # Missing warning.
- warn_regex = (
- rf"Missing CF-netCDF auxiliary coordinate variable {subject_name}.*"
- )
- with pytest.warns(iris.warnings.IrisCfMissingVarWarning, match=warn_regex):
- operation(warn=True)
- with pytest.warns() as record:
- operation(warn=False)
- warn_list = [str(w.message) for w in record]
- assert list(filter(re.compile(warn_regex).match, warn_list)) == []
-
- # String variable warning.
- warn_regex = r".*is a CF-netCDF label variable.*"
- vars_all[subject_name] = netcdf_variable(subject_name, "", np.bytes_)
- with pytest.warns(iris.warnings.IrisCfLabelVarWarning, match=warn_regex):
- operation(warn=True)
- with pytest.warns() as record:
- operation(warn=False)
- warn_list = [str(w.message) for w in record]
- assert list(filter(re.compile(warn_regex).match, warn_list)) == []
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == expected
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFUGridConnectivityVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFUGridConnectivityVariable.py
index d319f3660f..7a8e198919 100644
--- a/lib/iris/tests/unit/fileformats/cf/test_CFUGridConnectivityVariable.py
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFUGridConnectivityVariable.py
@@ -4,198 +4,32 @@
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for :class:`iris.fileformats.cf.CFUGridConnectivityVariable`."""
-import re
-import warnings
-
-import numpy as np
-import pytest
-
from iris.fileformats.cf import CFUGridConnectivityVariable
from iris.mesh import Connectivity
-from iris.tests.unit.fileformats.cf.test_CFReader import netcdf_variable
-import iris.warnings
-
-
-def named_variable(name):
- # Don't need to worry about dimensions or dtype for these tests.
- return netcdf_variable(name, "", int)
-
-class TestIdentify:
- def test_cf_identities(self):
- subject_name = "ref_subject"
- ref_subject = named_variable(subject_name)
- vars_common = {
- subject_name: ref_subject,
- "ref_not_subject": named_variable("ref_not_subject"),
- }
- # ONLY expecting ref_subject, excluding ref_not_subject.
- expected = {
- subject_name: CFUGridConnectivityVariable(subject_name, ref_subject)
- }
+from .identify_mixins import IdentifyByAttributeListMixin, _NetCDFVar
- for identity in Connectivity.UGRID_CF_ROLES:
- ref_source = named_variable("ref_source")
- setattr(ref_source, identity, subject_name)
- vars_all = dict({"ref_source": ref_source}, **vars_common)
- result = CFUGridConnectivityVariable.identify(vars_all)
- assert expected == result
- def test_duplicate_refs(self):
- subject_name = "ref_subject"
- ref_subject = named_variable(subject_name)
- ref_source_vars = {
- name: named_variable(name) for name in ("ref_source_1", "ref_source_2")
- }
- for var in ref_source_vars.values():
- setattr(var, Connectivity.UGRID_CF_ROLES[0], subject_name)
- vars_all = dict(
- {
- subject_name: ref_subject,
- "ref_not_subject": named_variable("ref_not_subject"),
- },
- **ref_source_vars,
- )
+class TestIdentify(IdentifyByAttributeListMixin):
+ __test__ = True
- # ONLY expecting ref_subject, excluding ref_not_subject.
- expected = {
- subject_name: CFUGridConnectivityVariable(subject_name, ref_subject)
- }
- result = CFUGridConnectivityVariable.identify(vars_all)
- assert expected == result
+ CF_CLASS = CFUGridConnectivityVariable
+ CF_IDENTITIES = Connectivity.UGRID_CF_ROLES
+ IDENTITY_SUPPORTS_MULTIPLE_REFS = False
+ MISSING_WARN_REGEX = r"Missing CF-UGRID connectivity variable {subject}.*"
- def test_two_cf_roles(self):
+ def test_two_part_ref_forbidden(self):
+ """Test that space-separated refs in a single attribute are rejected."""
subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
-
- ref_source_vars = {
- name: named_variable(name) for name in ("ref_source_1", "ref_source_2")
- }
- for ix, var in enumerate(ref_source_vars.values()):
- setattr(var, Connectivity.UGRID_CF_ROLES[ix], subject_names[ix])
- vars_all = dict(
- {"ref_not_subject": named_variable("ref_not_subject")},
- **ref_subject_vars,
- **ref_source_vars,
- )
-
- # Not expecting ref_not_subject.
- expected = {
- name: CFUGridConnectivityVariable(name, var)
- for name, var in ref_subject_vars.items()
- }
- result = CFUGridConnectivityVariable.identify(vars_all)
- assert expected == result
-
- def test_two_part_ref_ignored(self):
- # Not expected to handle more than one variable for a connectivity
- # cf role - invalid UGRID.
- subject_name = "ref_subject"
- ref_source = named_variable("ref_source")
- setattr(ref_source, Connectivity.UGRID_CF_ROLES[0], subject_name + " foo")
- vars_all = {
- subject_name: named_variable(subject_name),
- "ref_not_subject": named_variable("ref_not_subject"),
- "ref_source": ref_source,
- }
-
- result = CFUGridConnectivityVariable.identify(vars_all)
- assert {} == result
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
- def test_string_type_ignored(self):
- subject_name = "ref_subject"
- ref_source = named_variable("ref_source")
- setattr(ref_source, Connectivity.UGRID_CF_ROLES[0], subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, self.CF_IDENTITIES[0], " ".join(subject_names))
vars_all = {
- subject_name: netcdf_variable(subject_name, "", np.bytes_),
- "ref_not_subject": named_variable("ref_not_subject"),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
"ref_source": ref_source,
- }
-
- result = CFUGridConnectivityVariable.identify(vars_all)
- assert {} == result
-
- def test_ignore(self):
- subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
-
- ref_source_vars = {
- name: named_variable(name) for name in ("ref_source_1", "ref_source_2")
- }
- for ix, var in enumerate(ref_source_vars.values()):
- setattr(var, Connectivity.UGRID_CF_ROLES[0], subject_names[ix])
- vars_all = dict(
- {"ref_not_subject": named_variable("ref_not_subject")},
**ref_subject_vars,
- **ref_source_vars,
- )
-
- # ONLY expect the subject variable that hasn't been ignored.
- expected_name = subject_names[0]
- expected = {
- expected_name: CFUGridConnectivityVariable(
- expected_name, ref_subject_vars[expected_name]
- )
}
- result = CFUGridConnectivityVariable.identify(vars_all, ignore=subject_names[1])
- assert expected == result
-
- def test_target(self):
- subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
-
- source_names = ("ref_source_1", "ref_source_2")
- ref_source_vars = {name: named_variable(name) for name in source_names}
- for ix, var in enumerate(ref_source_vars.values()):
- setattr(var, Connectivity.UGRID_CF_ROLES[0], subject_names[ix])
- vars_all = dict(
- {"ref_not_subject": named_variable("ref_not_subject")},
- **ref_subject_vars,
- **ref_source_vars,
- )
-
- # ONLY expect the variable referenced by the named ref_source_var.
- expected_name = subject_names[0]
- expected = {
- expected_name: CFUGridConnectivityVariable(
- expected_name, ref_subject_vars[expected_name]
- )
- }
- result = CFUGridConnectivityVariable.identify(vars_all, target=source_names[0])
- assert expected == result
-
- def test_warn(self):
- subject_name = "ref_subject"
- ref_source = named_variable("ref_source")
- setattr(ref_source, Connectivity.UGRID_CF_ROLES[0], subject_name)
- vars_all = {
- "ref_not_subject": named_variable("ref_not_subject"),
- "ref_source": ref_source,
- }
-
- def operation(warn: bool):
- warnings.warn(
- "emit at least 1 warning",
- category=iris.warnings.IrisUserWarning,
- )
- result = CFUGridConnectivityVariable.identify(vars_all, warn=warn)
- assert {} == result
-
- # Missing warning.
- warn_regex = rf"Missing CF-UGRID connectivity variable {subject_name}.*"
- with pytest.warns(iris.warnings.IrisCfMissingVarWarning, match=warn_regex):
- operation(warn=True)
- with pytest.warns() as record:
- operation(warn=False)
- warn_list = [str(w.message) for w in record]
- assert list(filter(re.compile(warn_regex).match, warn_list)) == []
- # String variable warning.
- warn_regex = r".*is a CF-netCDF label variable.*"
- vars_all[subject_name] = netcdf_variable(subject_name, "", np.bytes_)
- with pytest.warns(iris.warnings.IrisCfLabelVarWarning, match=warn_regex):
- operation(warn=True)
- with pytest.warns() as record:
- operation(warn=False)
- warn_list = [str(w.message) for w in record]
- assert list(filter(re.compile(warn_regex).match, warn_list)) == []
+ result = self.CF_CLASS.identify(vars_all)
+ assert result == {}
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFUGridMeshVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFUGridMeshVariable.py
index e6e2db58af..25710e568c 100644
--- a/lib/iris/tests/unit/fileformats/cf/test_CFUGridMeshVariable.py
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFUGridMeshVariable.py
@@ -4,35 +4,28 @@
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for :class:`iris.fileformats.cf.CFUGridMeshVariable`."""
-import re
import warnings
import numpy as np
import pytest
from iris.fileformats.cf import CFUGridMeshVariable
-from iris.tests.unit.fileformats.cf.test_CFReader import netcdf_variable
import iris.warnings
+from .identify_mixins import _NetCDFVar, assert_warning_gated
-def named_variable(name):
- # Don't need to worry about dimensions or dtype for these tests.
- return netcdf_variable(name, "", int)
+CF_IDENTITY = "mesh"
class TestIdentify:
- @pytest.fixture(autouse=True)
- def _setup(self):
- self.cf_identity = "mesh"
-
def test_cf_role(self):
# Test that mesh variables can be identified by having `cf_role="mesh_topology"`.
match_name = "match"
- match = named_variable(match_name)
+ match = _NetCDFVar(match_name)
setattr(match, "cf_role", "mesh_topology")
not_match_name = f"not_{match_name}"
- not_match = named_variable(not_match_name)
+ not_match = _NetCDFVar(not_match_name)
setattr(not_match, "cf_role", "foo")
vars_all = {match_name: match, not_match_name: not_match}
@@ -40,45 +33,45 @@ def test_cf_role(self):
# ONLY expecting match, excluding not_match.
expected = {match_name: CFUGridMeshVariable(match_name, match)}
result = CFUGridMeshVariable.identify(vars_all)
- assert expected == result
+ assert result == expected
def test_cf_identity(self):
# Test that mesh variables can be identified by being another variable's
# `mesh` attribute.
subject_name = "ref_subject"
- ref_subject = named_variable(subject_name)
- ref_source = named_variable("ref_source")
- setattr(ref_source, self.cf_identity, subject_name)
+ ref_subject = _NetCDFVar(subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, subject_name)
vars_all = {
subject_name: ref_subject,
- "ref_not_subject": named_variable("ref_not_subject"),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
"ref_source": ref_source,
}
# ONLY expecting ref_subject, excluding ref_not_subject.
expected = {subject_name: CFUGridMeshVariable(subject_name, ref_subject)}
result = CFUGridMeshVariable.identify(vars_all)
- assert expected == result
+ assert result == expected
def test_cf_role_and_identity(self):
# Test that identification can successfully handle a combination of
# mesh variables having `cf_role="mesh_topology"` AND being referenced as
# another variable's `mesh` attribute.
role_match_name = "match"
- role_match = named_variable(role_match_name)
+ role_match = _NetCDFVar(role_match_name)
setattr(role_match, "cf_role", "mesh_topology")
- ref_source_1 = named_variable("ref_source_1")
- setattr(ref_source_1, self.cf_identity, role_match_name)
+ ref_source_1 = _NetCDFVar("ref_source_1")
+ setattr(ref_source_1, CF_IDENTITY, role_match_name)
subject_name = "ref_subject"
- ref_subject = named_variable(subject_name)
- ref_source_2 = named_variable("ref_source_2")
- setattr(ref_source_2, self.cf_identity, subject_name)
+ ref_subject = _NetCDFVar(subject_name)
+ ref_source_2 = _NetCDFVar("ref_source_2")
+ setattr(ref_source_2, CF_IDENTITY, subject_name)
vars_all = {
role_match_name: role_match,
subject_name: ref_subject,
- "ref_not_subject": named_variable("ref_not_subject"),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
"ref_source_1": ref_source_1,
"ref_source_2": ref_source_2,
}
@@ -89,20 +82,20 @@ def test_cf_role_and_identity(self):
subject_name: CFUGridMeshVariable(subject_name, ref_subject),
}
result = CFUGridMeshVariable.identify(vars_all)
- assert expected == result
+ assert result == expected
def test_duplicate_refs(self):
subject_name = "ref_subject"
- ref_subject = named_variable(subject_name)
+ ref_subject = _NetCDFVar(subject_name)
ref_source_vars = {
- name: named_variable(name) for name in ("ref_source_1", "ref_source_2")
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
}
for var in ref_source_vars.values():
- setattr(var, self.cf_identity, subject_name)
+ setattr(var, CF_IDENTITY, subject_name)
vars_all = dict(
{
subject_name: ref_subject,
- "ref_not_subject": named_variable("ref_not_subject"),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
},
**ref_source_vars,
)
@@ -110,19 +103,19 @@ def test_duplicate_refs(self):
# ONLY expecting ref_subject, excluding ref_not_subject.
expected = {subject_name: CFUGridMeshVariable(subject_name, ref_subject)}
result = CFUGridMeshVariable.identify(vars_all)
- assert expected == result
+ assert result == expected
def test_two_refs(self):
subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
ref_source_vars = {
- name: named_variable(name) for name in ("ref_source_1", "ref_source_2")
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
}
for ix, var in enumerate(ref_source_vars.values()):
- setattr(var, self.cf_identity, subject_names[ix])
+ setattr(var, CF_IDENTITY, subject_names[ix])
vars_all = dict(
- {"ref_not_subject": named_variable("ref_not_subject")},
+ {"ref_not_subject": _NetCDFVar("ref_not_subject")},
**ref_subject_vars,
**ref_source_vars,
)
@@ -133,47 +126,47 @@ def test_two_refs(self):
for name, var in ref_subject_vars.items()
}
result = CFUGridMeshVariable.identify(vars_all)
- assert expected == result
+ assert result == expected
def test_two_part_ref_ignored(self):
# Not expected to handle more than one variable for a mesh
# cf role - invalid UGRID.
subject_name = "ref_subject"
- ref_source = named_variable("ref_source")
- setattr(ref_source, self.cf_identity, subject_name + " foo")
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, subject_name + " foo")
vars_all = {
- subject_name: named_variable(subject_name),
- "ref_not_subject": named_variable("ref_not_subject"),
+ subject_name: _NetCDFVar(subject_name),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
"ref_source": ref_source,
}
result = CFUGridMeshVariable.identify(vars_all)
- assert {} == result
+ assert result == {}
def test_string_type_ignored(self):
subject_name = "ref_subject"
- ref_source = named_variable("ref_source")
- setattr(ref_source, self.cf_identity, subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, subject_name)
vars_all = {
- subject_name: netcdf_variable(subject_name, "", np.bytes_),
- "ref_not_subject": named_variable("ref_not_subject"),
+ subject_name: _NetCDFVar(subject_name, dtype=np.bytes_),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
"ref_source": ref_source,
}
result = CFUGridMeshVariable.identify(vars_all)
- assert {} == result
+ assert result == {}
def test_ignore(self):
subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
ref_source_vars = {
- name: named_variable(name) for name in ("ref_source_1", "ref_source_2")
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
}
for ix, var in enumerate(ref_source_vars.values()):
- setattr(var, self.cf_identity, subject_names[ix])
+ setattr(var, CF_IDENTITY, subject_names[ix])
vars_all = dict(
- {"ref_not_subject": named_variable("ref_not_subject")},
+ {"ref_not_subject": _NetCDFVar("ref_not_subject")},
**ref_subject_vars,
**ref_source_vars,
)
@@ -186,18 +179,18 @@ def test_ignore(self):
)
}
result = CFUGridMeshVariable.identify(vars_all, ignore=subject_names[1])
- assert expected == result
+ assert result == expected
def test_target(self):
subject_names = ("ref_subject_1", "ref_subject_2")
- ref_subject_vars = {name: named_variable(name) for name in subject_names}
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
source_names = ("ref_source_1", "ref_source_2")
- ref_source_vars = {name: named_variable(name) for name in source_names}
+ ref_source_vars = {name: _NetCDFVar(name) for name in source_names}
for ix, var in enumerate(ref_source_vars.values()):
- setattr(var, self.cf_identity, subject_names[ix])
+ setattr(var, CF_IDENTITY, subject_names[ix])
vars_all = dict(
- {"ref_not_subject": named_variable("ref_not_subject")},
+ {"ref_not_subject": _NetCDFVar("ref_not_subject")},
**ref_subject_vars,
**ref_source_vars,
)
@@ -210,14 +203,28 @@ def test_target(self):
)
}
result = CFUGridMeshVariable.identify(vars_all, target=source_names[0])
- assert expected == result
+ assert result == expected
+
+ def test_target_unknown_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Cannot identify unknown target CF-netCDF variable 'unknown'"
+ with pytest.raises(ValueError, match=message):
+ CFUGridMeshVariable.identify(vars_all, target="unknown")
+
+ def test_target_wrong_type_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Expect a target CF-netCDF variable name"
+ with pytest.raises(TypeError, match=message):
+ CFUGridMeshVariable.identify(vars_all, target=object())
def test_warn(self):
subject_name = "ref_subject"
- ref_source = named_variable("ref_source")
- setattr(ref_source, self.cf_identity, subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, subject_name)
vars_all = {
- "ref_not_subject": named_variable("ref_not_subject"),
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
"ref_source": ref_source,
}
@@ -227,23 +234,15 @@ def operation(warn: bool):
category=iris.warnings.IrisUserWarning,
)
result = CFUGridMeshVariable.identify(vars_all, warn=warn)
- assert {} == result
+ assert result == {}
# Missing warning.
warn_regex = rf"Missing CF-UGRID mesh variable {subject_name}.*"
- with pytest.warns(iris.warnings.IrisCfMissingVarWarning, match=warn_regex):
- operation(warn=True)
- with pytest.warns() as record:
- operation(warn=False)
- warn_list = [str(w.message) for w in record]
- assert list(filter(re.compile(warn_regex).match, warn_list)) == []
+ assert_warning_gated(
+ operation, iris.warnings.IrisCfMissingVarWarning, warn_regex
+ )
# String variable warning.
warn_regex = r".*is a CF-netCDF label variable.*"
- vars_all[subject_name] = netcdf_variable(subject_name, "", np.bytes_)
- with pytest.warns(iris.warnings.IrisCfLabelVarWarning, match=warn_regex):
- operation(warn=True)
- with pytest.warns() as record:
- operation(warn=False)
- warn_list = [str(w.message) for w in record]
- assert list(filter(re.compile(warn_regex).match, warn_list)) == []
+ vars_all[subject_name] = _NetCDFVar(subject_name, dtype=np.bytes_)
+ assert_warning_gated(operation, iris.warnings.IrisCfLabelVarWarning, warn_regex)
diff --git a/lib/iris/tests/unit/fileformats/cf/test_CFVariable.py b/lib/iris/tests/unit/fileformats/cf/test_CFVariable.py
new file mode 100644
index 0000000000..c830c1cc67
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_CFVariable.py
@@ -0,0 +1,661 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf.CFVariable`."""
+
+import copy
+import warnings
+
+import numpy as np
+import pytest
+
+from iris._deprecation import IrisDeprecation
+from iris.fileformats import cf as cf
+from iris.fileformats.cf import _variables
+from iris.fileformats.cf.dataset import TrackedAttributes
+
+from .identify_mixins import _NetCDFVarWithDimensions
+
+
+class CFVariableSub(cf.CFVariable):
+ """A subclass of CFVariable for testing purposes."""
+
+ def identify(self, variables, ignore=None, target=None, warn=True):
+ return super().identify(variables, ignore=ignore, target=target, warn=warn)
+
+
+def make_nc_var(mocker):
+ nc_var = mocker.MagicMock()
+ nc_var.attributes = {
+ "coordinates": "x y",
+ "standard_name": "air_temperature",
+ "_FillValue": -999,
+ }
+ nc_var.dimensions = ("time", "lat")
+ nc_var.location = "/tmp/file.nc"
+ nc_var.__len__.return_value = 4
+ nc_var.__getitem__.return_value = "payload"
+
+ return nc_var
+
+
+@pytest.fixture
+def nc_var(mocker):
+ return make_nc_var(mocker)
+
+
+@pytest.fixture
+def nc_var_without_location(nc_var):
+ del nc_var.location
+
+ return nc_var
+
+
+@pytest.fixture
+def nc_vars(mocker):
+ # Three is the maximum number of independent mock variables needed in one test.
+ return tuple(make_nc_var(mocker) for _ in range(3))
+
+
+class TestInit:
+ def test_records_filename_from_the_variables_location(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.filename == "/tmp/file.nc"
+ assert cf_var.cf_name == "foo"
+ assert cf_var.cf_data is nc_var
+ assert cf_var.cf_group is None
+ assert cf_var.cf_terms_by_root == {}
+ assert cf_var._to_be_promoted is False
+
+ def test_falls_back_to_unknown_filename_without_a_location(
+ self, nc_var_without_location
+ ):
+ cf_var = CFVariableSub("foo", nc_var_without_location)
+
+ assert cf_var.filename == "<unknown_filename>"
+
+
+class TestIdentifyCommon:
+ def test_handles_defaults_and_target_selection(self):
+ variables = {"a": object(), "b": object()}
+
+ ignore, target = CFVariableSub._identify_common(variables, None, None)
+ assert ignore == []
+ assert target is variables
+
+ ignore, target = CFVariableSub._identify_common(variables, ["a"], "b")
+ assert ignore == ["a"]
+ assert target == {"b": variables["b"]}
+
+ def test_raises_for_unknown_target(self):
+ with pytest.raises(ValueError, match="Cannot identify unknown target"):
+ CFVariableSub._identify_common({"a": object()}, None, "missing")
+
+ def test_raises_for_invalid_target_type(self):
+ with pytest.raises(TypeError, match="Expect a target CF-netCDF variable name"):
+ CFVariableSub._identify_common({"a": object()}, None, object())
+
+
+class TestSpans:
+ def test_scalar_dimension_always_true(self, mocker, nc_var):
+ nc_var.dimensions = (_variables._NCZARR_SCALAR_DIMENSION,)
+ cf_var = CFVariableSub("scalar", nc_var)
+
+ other = mocker.MagicMock()
+ other.dimensions = ("time",)
+
+ assert cf_var.spans(other)
+
+ def test_is_subset_check(self):
+ # `spans()` now reads `shape` as well as `dimensions` on both sides of
+ # the comparison, so a bare `mocker.MagicMock()` collaborator will not
+ # do here: `_dimension_identities()` reads both off an unconfigured
+ # mock and gets something that iterates as empty, making `spans()`
+ # trivially "true" instead of exercising the real comparison - see
+ # the Zarr I/O design doc, section 4.4, and this task's own history.
+ lhs = CFVariableSub("lhs", _NetCDFVarWithDimensions("lhs_var", ("time",)))
+ rhs = CFVariableSub("rhs", _NetCDFVarWithDimensions("rhs_var", ("time", "lat")))
+ assert lhs.spans(rhs)
+
+ other = CFVariableSub(
+ "other", _NetCDFVarWithDimensions("other_var", ("height",))
+ )
+ assert not other.spans(rhs)
+
+ def test_scalar_dimension_with_others_is_not_scalar(self):
+ # _scalar_ means "this variable is scalar", which a variable with a
+ # second dimension is not. It is not a wildcard.
+ cf_var = CFVariableSub(
+ "not_scalar",
+ _NetCDFVarWithDimensions(
+ "not_scalar_var", (_variables._NCZARR_SCALAR_DIMENSION, "bnds")
+ ),
+ )
+ other = CFVariableSub("other", _NetCDFVarWithDimensions("other_var", ("time",)))
+
+ assert not cf_var.spans(other)
+
+ def test_no_dimensions_always_spans(self, mocker, nc_var):
+ nc_var.dimensions = ()
+ cf_var = CFVariableSub("scalar", nc_var)
+
+ other = mocker.MagicMock()
+ other.dimensions = ("time",)
+
+ assert cf_var.spans(other)
+
+
+class TestIsScalar:
+ """Direct tests of the single definition of "scalar" that spans() shares.
+
+ A Zarr backend reads _is_scalar() rather than restating the rule, so it
+ is tested directly and not only through spans().
+ """
+
+ def test_no_dimensions_is_scalar(self, nc_var):
+ nc_var.dimensions = ()
+ cf_var = CFVariableSub("scalar", nc_var)
+
+ assert cf_var._is_scalar()
+
+ def test_nczarr_scalar_dimension_is_scalar(self, nc_var):
+ nc_var.dimensions = (_variables._NCZARR_SCALAR_DIMENSION,)
+ cf_var = CFVariableSub("scalar", nc_var)
+
+ assert cf_var._is_scalar()
+
+ def test_nczarr_scalar_dimension_with_another_is_not_scalar(self, nc_var):
+ nc_var.dimensions = (_variables._NCZARR_SCALAR_DIMENSION, "bnds")
+ cf_var = CFVariableSub("not_scalar", nc_var)
+
+ assert not cf_var._is_scalar()
+
+ def test_ordinary_dimensions_are_not_scalar(self, nc_var):
+ nc_var.dimensions = ("time", "lat")
+ cf_var = CFVariableSub("not_scalar", nc_var)
+
+ assert not cf_var._is_scalar()
+
+
+class TestComparisonAndRepresentation:
+ def test_equality_inequality_and_hash_by_name(self, nc_vars):
+ first, second, third = nc_vars
+ one = CFVariableSub("same", first)
+ two = CFVariableSub("same", second)
+ other = CFVariableSub("different", third)
+
+ assert one == two
+ assert one != other
+ assert hash(one) == hash(two)
+ assert hash(one) != hash(other)
+
+ def test_repr_contains_class_name_name_and_data_repr(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert repr(cf_var) == f"CFVariableSub('foo', {nc_var!r})"
+
+
+class TestAttributeAccess:
+ def test_reads_are_not_cached_on_the_instance(self, nc_var):
+ # The setattr cache is gone. It made a re-read after cf_attrs_reset()
+ # invisible to attribute tracking, which decided which attributes
+ # reached the cube - see TestReadAfterReset below.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert "coordinates" not in cf_var.__dict__
+ assert cf_var.coordinates == "x y"
+ assert "coordinates" not in cf_var.__dict__
+ assert cf_var.coordinates == "x y"
+ assert "coordinates" not in cf_var.__dict__
+
+ def test_attributes_are_a_snapshot_taken_at_construction(self, nc_var):
+ # Copied, not wrapped: the storage object's mapping writes through
+ # to the file, and CFReader synthesises a "bounds" link on a file it
+ # opened read-only.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ nc_var.attributes["units"] = "K"
+ assert "units" not in cf_var.attributes.untracked
+
+ cf_var.attributes["bounds"] = "foo_bnds"
+ assert "bounds" not in nc_var.attributes
+
+ def test_getattr_of_a_non_attribute_reaches_the_variable(self, nc_var):
+ # The one-cycle compatibility route, now delegated to the storage
+ # object - which is also what warns.
+ nc_var.deprecated_netcdf_member.return_value = 42
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.not_an_ncattr == 42
+ nc_var.deprecated_netcdf_member.assert_called_once_with("not_an_ncattr")
+ # __getattr__ must not cache the value onto the instance - a second
+ # access has to reach the storage object again, not a stale copy.
+ assert "not_an_ncattr" not in cf_var.__dict__
+ assert "not_an_ncattr" not in dict(cf_var.cf_attrs())
+ assert "not_an_ncattr" not in dict(cf_var.cf_attrs_used())
+
+ def test_getattr_of_nothing_at_all_raises_attribute_error(self, nc_var):
+ nc_var.deprecated_netcdf_member.side_effect = AttributeError("nonesuch")
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.raises(AttributeError, match="nonesuch"):
+ cf_var.nonesuch
+
+ def test_getattr_of_nothing_at_all_raises_unchained(self, nc_var):
+ """A missed attribute lookup leaves no KeyError on the traceback.
+
+ Resolving the fallback inside `except KeyError` would prefix every
+ mistyped attribute with "During handling of the above exception".
+ """
+ nc_var.deprecated_netcdf_member.side_effect = AttributeError("nonesuch")
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.raises(AttributeError) as excinfo:
+ cf_var.nonesuch
+
+ assert excinfo.value.__cause__ is None
+ assert excinfo.value.__context__ is None
+
+ def test_getitem_and_len_delegate_to_underlying_variable(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert len(cf_var) == 4
+ assert cf_var[0] == "payload"
+ nc_var.__len__.assert_called_once_with()
+ nc_var.__getitem__.assert_called_once_with(0)
+
+ def test_cf_attrs_access_helpers_and_reset(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.cf_attrs() == (
+ ("_FillValue", -999),
+ ("coordinates", "x y"),
+ ("standard_name", "air_temperature"),
+ )
+ assert cf_var.cf_attrs_ignored() == (("_FillValue", -999),)
+ assert cf_var.cf_attrs_used() == (("_FillValue", -999),)
+ assert cf_var.cf_attrs_unused() == (
+ ("coordinates", "x y"),
+ ("standard_name", "air_temperature"),
+ )
+
+ _ = cf_var.coordinates
+ assert cf_var.cf_attrs_used() == (("_FillValue", -999), ("coordinates", "x y"))
+
+ cf_var.cf_attrs_reset()
+ assert cf_var.cf_attrs_used() == (("_FillValue", -999),)
+
+
+class TestFormulaTerms:
+ def test_registration_and_presence(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert not cf_var.has_formula_terms()
+ cf_var.add_formula_term("root", "a")
+ assert cf_var.has_formula_terms()
+ assert cf_var.cf_terms_by_root == {"root": "a"}
+
+
+class TestIdentify:
+ def test_subclass_stub_returns_none(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.identify({}) is None
+
+
+class TestAttributesMapping:
+ def test_attributes_contents(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert dict(cf_var.attributes) == {
+ "coordinates": "x y",
+ "standard_name": "air_temperature",
+ "_FillValue": -999,
+ }
+
+ def test_getattr_and_mapping_are_the_same_read(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ _ = cf_var.coordinates
+ assert cf_var.attributes.read == frozenset(["_FillValue", "coordinates"])
+
+ cf_var.cf_attrs_reset()
+ _ = cf_var.attributes["coordinates"]
+ assert cf_var.attributes.read == frozenset(["_FillValue", "coordinates"])
+
+ def test_untracked_read_is_not_recorded(self, nc_var):
+ # What helpers.py's flag-attribute probe needs: a look that does not
+ # count, so the attribute still reaches the cube.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.attributes.untracked["coordinates"] == "x y"
+ assert "coordinates" in dict(cf_var.cf_attrs_unused())
+
+
+class TestTypedProperties:
+ def test_dimensions_is_a_tuple(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ assert cf_var.dimensions == ("time", "lat")
+
+ def test_shape_ndim_dtype_size(self, mocker, nc_var):
+ nc_var.shape = (3, 4)
+ nc_var.dtype = np.dtype("f4")
+ nc_var.size = 12
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert cf_var.shape == (3, 4)
+ assert cf_var.ndim == 2
+ assert cf_var.dtype == np.dtype("f4")
+ assert cf_var.size == 12
+
+ #: Each typed property, against the value its netCDF4 variable supplies.
+ TYPED_PROPERTIES = {
+ "dimensions": ("time", "lat"),
+ "shape": (3, 4),
+ "ndim": 2,
+ "dtype": np.dtype("f4"),
+ "size": 12,
+ }
+
+ @pytest.mark.parametrize("name", list(TYPED_PROPERTIES))
+ def test_a_file_attribute_colliding_with_a_typed_property(self, nc_var, name):
+ # A deliberate change in #7303. Before it, __getattr__ recorded the
+ # name as read, so _add_unused_attributes dropped a legitimate user
+ # attribute from the cube - though the value returned was never the
+ # file's. A declared property now short-circuits __getattr__.
+ nc_var.attributes = {name: f"file value of {name}"}
+ nc_var.shape = (3, 4)
+ nc_var.dtype = np.dtype("f4")
+ nc_var.size = 12
+ cf_var = CFVariableSub("foo", nc_var)
+
+ # The netCDF4 value wins, not the colliding file attribute.
+ assert getattr(cf_var, name) == self.TYPED_PROPERTIES[name]
+
+ # Reading the property recorded nothing, so the file attribute is
+ # still unused - and so still reaches the cube - with its own value
+ # intact. Deliberately asserted through cf_attrs_unused() rather than
+ # cf_var.attributes[name], which would itself record the read this
+ # test exists to detect.
+ assert dict(cf_var.cf_attrs_unused())[name] == f"file value of {name}"
+
+
+#: CFVariable members that a CF attribute of the same name cannot displace.
+SHADOWED_NAMES = ["filename", "cf_name", "spans", "attributes", "cf_data"]
+
+
+class TestShadowedAttributeNames:
+ """A CF attribute cannot displace a CFVariable member of the same name.
+
+ Spec section 4.3's known limitation, pinned.
+ """
+
+ @pytest.fixture
+ def shadowing(self, nc_var):
+ nc_var.attributes = {
+ name: f"file value of {name}" for name in SHADOWED_NAMES
+ } | {"units": "K"}
+ return CFVariableSub("foo", nc_var)
+
+ def test_the_class_member_wins(self, nc_var, shadowing):
+ # Each member's own value, not merely "not the file value": that
+ # weaker form passes against a member returning None or another
+ # member's value, and is trivially true of attributes and cf_data,
+ # neither of which can ever equal a string.
+ assert shadowing.filename == "/tmp/file.nc"
+ assert shadowing.cf_name == "foo"
+ assert shadowing.spans.__func__ is CFVariableSub.spans
+ assert isinstance(shadowing.attributes, TrackedAttributes)
+ assert shadowing.cf_data is nc_var
+
+ # Fails if SHADOWED_NAMES gains a member this test does not assert.
+ assert set(SHADOWED_NAMES) == {
+ "filename",
+ "cf_name",
+ "spans",
+ "attributes",
+ "cf_data",
+ }
+
+ @pytest.mark.parametrize("name", SHADOWED_NAMES)
+ def test_the_file_value_is_still_reachable(self, shadowing, name):
+ assert shadowing.attributes[name] == f"file value of {name}"
+
+ @pytest.mark.parametrize("name", SHADOWED_NAMES)
+ def test_reading_it_through_the_mapping_marks_it_used(self, shadowing, name):
+ assert name in dict(shadowing.cf_attrs_unused())
+ _ = shadowing.attributes[name]
+ assert name in dict(shadowing.cf_attrs_used())
+ assert name not in dict(shadowing.cf_attrs_unused())
+
+ @pytest.mark.parametrize("name", SHADOWED_NAMES)
+ def test_it_is_listed_among_the_variables_attributes(self, shadowing, name):
+ assert name in dict(shadowing.cf_attrs())
+
+ def test_a_shadowing_name_does_not_disturb_its_neighbours(self, shadowing):
+ assert shadowing.units == "K"
+ assert dict(shadowing.cf_attrs_used())["units"] == "K"
+
+
+class TestGetattrAndHasattr:
+ """The two call shapes the loading rules actually use."""
+
+ def test_getattr_with_a_default_finds_the_attribute(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ assert getattr(cf_var, "coordinates", None) == "x y"
+
+ def test_getattr_with_a_default_returns_the_default(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+ assert getattr(cf_var, "nonesuch", None) is None
+ assert getattr(cf_var, "nonesuch", "fallback") == "fallback"
+
+ def test_a_missing_attribute_raises_attribute_error_not_key_error(self, nc_var):
+ # getattr(..., default) only swallows AttributeError. A KeyError from
+ # the mapping would escape and abort the load.
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+ with pytest.raises(AttributeError):
+ cf_var.nonesuch
+
+ def test_getattr_of_a_default_does_not_record_a_read(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+ _ = getattr(cf_var, "nonesuch", None)
+ assert cf_var.cf_attrs_used() == (("_FillValue", -999),)
+
+ def test_hasattr_true_marks_the_attribute_used(self, nc_var):
+ # Parity with the old behaviour: hasattr went through __getattr__,
+ # which added the name to the used set.
+ cf_var = CFVariableSub("foo", nc_var)
+ assert hasattr(cf_var, "coordinates")
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+
+ def test_hasattr_false_marks_nothing(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+ assert not hasattr(cf_var, "nonesuch")
+ assert cf_var.cf_attrs_used() == (("_FillValue", -999),)
+
+ def test_dunder_probes_do_not_reach_the_file(self, nc_var):
+ # copy, pickle and numpy all probe for dunders. Answering one from
+ # file data would make a CFVariable behave as whatever the file says.
+ nc_var.attributes = {"__array__": "nonsense"}
+ cf_var = CFVariableSub("foo", nc_var)
+
+ assert not hasattr(cf_var, "__array_interface__")
+ # The file really does carry "__array__", and __getattr__ still
+ # refuses it - without that refusal numpy would believe the file.
+ with pytest.raises(AttributeError, match="__array__"):
+ cf_var.__array__
+ assert cf_var.attributes.untracked["__array__"] == "nonsense"
+
+ @pytest.mark.parametrize("name", ["attributes", "cf_data"])
+ def test_the_recursion_guard_holds_before_init_completes(self, name):
+ # An instance whose __init__ never ran - what copy and pickle build -
+ # must raise, not recurse until the stack is gone.
+ cf_var = CFVariableSub.__new__(CFVariableSub)
+ with pytest.raises(AttributeError, match=name):
+ getattr(cf_var, name)
+
+
+class TestDeprecatedAttributeAccess:
+ """Reading a CF attribute as an attribute warns; everything else is silent.
+
+ cf_var.units and cf_var.attributes["units"] are the same read. Only the
+ spelling that cannot be type-checked is deprecated. Spec section 4.3.
+ """
+
+ def test_reading_a_cf_attribute_warns(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(IrisDeprecation) as record:
+ assert cf_var.coordinates == "x y"
+
+ assert len(record) == 1
+
+ def test_the_warning_names_the_mapping_and_the_key(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(
+ IrisDeprecation, match=r"cf_var\.attributes\['coordinates'\]"
+ ):
+ _ = cf_var.coordinates
+
+ def test_the_warning_points_at_the_caller(self, nc_var):
+ # stacklevel: a warning reported against _variables.py tells the
+ # reader nothing about which of their lines to change.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(IrisDeprecation) as record:
+ _ = cf_var.coordinates
+
+ assert record[0].filename == __file__
+
+ def test_a_warned_read_is_still_recorded(self, nc_var):
+ # The warning is additional to the read, not instead of it. What was
+ # read decides what reaches the cube.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(IrisDeprecation):
+ _ = cf_var.coordinates
+
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+ assert "coordinates" not in dict(cf_var.cf_attrs_unused())
+
+ def test_every_read_warns(self, nc_var):
+ # There is no instance cache to suppress the second one, and a test
+ # expecting one warning would pass only through __warningregistry__.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with warnings.catch_warnings(record=True) as record:
+ warnings.simplefilter("always", IrisDeprecation)
+ _ = cf_var.coordinates
+ _ = cf_var.coordinates
+
+ deprecations = [w for w in record if issubclass(w.category, IrisDeprecation)]
+ assert len(deprecations) == 2
+
+ def test_the_mapping_does_not_warn(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert cf_var.attributes["coordinates"] == "x y"
+
+ def test_a_declared_property_does_not_warn(self, nc_var):
+ # A declared property short-circuits __getattr__ entirely.
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert cf_var.dimensions == ("time", "lat")
+ assert cf_var.cf_name == "foo"
+
+ def test_a_missing_attribute_does_not_warn(self, nc_var):
+ # getattr(cf_var, "bounds", None) is a deliberate idiom throughout the
+ # loading rules. A miss must stay silent and keep its default.
+ cf_var = CFVariableSub("foo", nc_var)
+ cf_var.cf_data = object()
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert getattr(cf_var, "nonesuch", None) is None
+ assert not hasattr(cf_var, "nonesuch")
+
+ def test_hasattr_warns_on_a_hit(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with pytest.warns(IrisDeprecation):
+ assert hasattr(cf_var, "coordinates")
+
+ def test_dunder_probes_do_not_warn(self, nc_var):
+ # copy, pickle and numpy all probe for dunders, and the file here
+ # really does carry one.
+ nc_var.attributes = {"__array__": "nonsense"}
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert not hasattr(cf_var, "__array_interface__")
+ assert not hasattr(cf_var, "__array__")
+ _ = copy.copy(cf_var)
+
+ def test_the_netcdf_fallback_keeps_its_own_message(self, nc_var):
+ # Reaching a netCDF4 member is a different deprecation, worded for a
+ # different fix, and issued by the storage object.
+ nc_var.deprecated_netcdf_member.return_value = 42
+ cf_var = CFVariableSub("foo", nc_var)
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ assert cf_var.not_an_ncattr == 42
+
+
+class TestReadAfterReset:
+ """The cache removal's whole point, at unit scale.
+
+ test_CFReader__dataset.py pins the same behaviour on a real file.
+ """
+
+ def test_a_re_read_after_reset_counts_again(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+
+ # As CFReader does: read while parsing structure, then reset.
+ _ = cf_var.coordinates
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+ cf_var.cf_attrs_reset()
+ assert "coordinates" in dict(cf_var.cf_attrs_unused())
+
+ # As the loading rules then do: read it again.
+ _ = cf_var.coordinates
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+ assert "coordinates" not in dict(cf_var.cf_attrs_unused())
+
+ def test_the_same_holds_through_hasattr(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ _ = cf_var.coordinates
+ cf_var.cf_attrs_reset()
+
+ assert hasattr(cf_var, "coordinates")
+ assert "coordinates" in dict(cf_var.cf_attrs_used())
+
+ def test_reset_restores_the_ignored_names_only(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ _ = cf_var.coordinates
+ _ = cf_var.standard_name
+ cf_var.cf_attrs_reset()
+
+ assert cf_var.cf_attrs_used() == (("_FillValue", -999),)
+ assert cf_var.cf_attrs_unused() == (
+ ("coordinates", "x y"),
+ ("standard_name", "air_temperature"),
+ )
+
+ def test_values_are_unchanged_by_any_of_this(self, nc_var):
+ cf_var = CFVariableSub("foo", nc_var)
+ for _ in range(3):
+ assert cf_var.coordinates == "x y"
+ cf_var.cf_attrs_reset()
diff --git a/lib/iris/tests/unit/fileformats/cf/test__CFFormulaTermsVariable.py b/lib/iris/tests/unit/fileformats/cf/test__CFFormulaTermsVariable.py
new file mode 100644
index 0000000000..3601de9ef7
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test__CFFormulaTermsVariable.py
@@ -0,0 +1,179 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.cf._CFFormulaTermsVariable`."""
+
+import warnings
+
+import pytest
+
+from iris.fileformats.cf._variables import _CFFormulaTermsVariable
+import iris.warnings
+
+from .identify_mixins import _NetCDFVar, assert_warning_gated
+
+CF_IDENTITY = "formula_terms"
+
+
+class TestIdentify:
+ def test_single_formula_term(self):
+ subject_name = "ref_sigma"
+ ref_subject = _NetCDFVar(subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, f"sigma: {subject_name}")
+ vars_all = {
+ subject_name: ref_subject,
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ result = _CFFormulaTermsVariable.identify(vars_all)
+ assert subject_name in result
+ assert result[subject_name].cf_terms_by_root == {"ref_source": "sigma"}
+
+ def test_multiple_terms_one_source(self):
+ subject_names = ("ref_sigma", "ref_ps")
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
+ ref_source = _NetCDFVar("ref_source")
+ setattr(
+ ref_source,
+ CF_IDENTITY,
+ f"sigma: {subject_names[0]} ps: {subject_names[1]}",
+ )
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ **ref_subject_vars,
+ }
+
+ result = _CFFormulaTermsVariable.identify(vars_all)
+ assert set(result.keys()) == set(subject_names)
+ assert result[subject_names[0]].cf_terms_by_root == {"ref_source": "sigma"}
+ assert result[subject_names[1]].cf_terms_by_root == {"ref_source": "ps"}
+
+ def test_term_name_lowercased(self):
+ """Formula term names must be normalised to lowercase."""
+ subject_name = "ref_sigma"
+ ref_subject = _NetCDFVar(subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, f"SIGMA: {subject_name}")
+ vars_all = {
+ subject_name: ref_subject,
+ "ref_source": ref_source,
+ }
+
+ result = _CFFormulaTermsVariable.identify(vars_all)
+ assert result[subject_name].cf_terms_by_root == {"ref_source": "sigma"}
+
+ def test_same_variable_multiple_roots_aggregates(self):
+ """Same variable referenced by two roots accumulates both terms."""
+ subject_name = "ref_sigma"
+ ref_subject = _NetCDFVar(subject_name)
+
+ source_vars = {
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
+ }
+ source_vars["ref_source_1"].formula_terms = f"sigma: {subject_name}"
+ setattr(source_vars["ref_source_1"], CF_IDENTITY, f"sigma: {subject_name}")
+ setattr(source_vars["ref_source_2"], CF_IDENTITY, f"eta: {subject_name}")
+
+ vars_all = {
+ subject_name: ref_subject,
+ **source_vars,
+ }
+
+ result = _CFFormulaTermsVariable.identify(vars_all)
+ assert subject_name in result
+ terms = result[subject_name].cf_terms_by_root
+ assert terms.get("ref_source_1") == "sigma"
+ assert terms.get("ref_source_2") == "eta"
+
+ def test_ignore(self):
+ subject_names = ("ref_sigma", "ref_ps")
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
+
+ ref_source_vars = {
+ name: _NetCDFVar(name) for name in ("ref_source_1", "ref_source_2")
+ }
+ for ix, var in enumerate(ref_source_vars.values()):
+ setattr(var, CF_IDENTITY, f"sigma: {subject_names[ix]}")
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ **ref_subject_vars,
+ **ref_source_vars,
+ }
+
+ result = _CFFormulaTermsVariable.identify(vars_all, ignore=subject_names[1])
+ assert subject_names[0] in result
+ assert subject_names[1] not in result
+
+ def test_target(self):
+ subject_names = ("ref_sigma", "ref_ps")
+ ref_subject_vars = {name: _NetCDFVar(name) for name in subject_names}
+
+ source_names = ("ref_source_1", "ref_source_2")
+ ref_source_vars = {name: _NetCDFVar(name) for name in source_names}
+ for ix, var in enumerate(ref_source_vars.values()):
+ setattr(var, CF_IDENTITY, f"sigma: {subject_names[ix]}")
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ **ref_subject_vars,
+ **ref_source_vars,
+ }
+
+ result = _CFFormulaTermsVariable.identify(vars_all, target=source_names[0])
+ assert subject_names[0] in result
+ assert subject_names[1] not in result
+
+ def test_target_unknown_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Cannot identify unknown target CF-netCDF variable 'unknown'"
+ with pytest.raises(ValueError, match=message):
+ _CFFormulaTermsVariable.identify(vars_all, target="unknown")
+
+ def test_target_wrong_type_raises(self):
+ vars_all = {"ref_source": _NetCDFVar("ref_source")}
+
+ message = "Expect a target CF-netCDF variable name"
+ with pytest.raises(TypeError, match=message):
+ _CFFormulaTermsVariable.identify(vars_all, target=object())
+
+ def test_warn(self):
+ subject_name = "ref_sigma"
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, f"sigma: {subject_name}")
+ vars_all = {
+ "ref_not_subject": _NetCDFVar("ref_not_subject"),
+ "ref_source": ref_source,
+ }
+
+ def operation(warn: bool):
+ warnings.warn(
+ "emit at least 1 warning",
+ category=iris.warnings.IrisUserWarning,
+ )
+ _CFFormulaTermsVariable.identify(vars_all, warn=warn)
+
+ warn_regex = rf"Missing CF-netCDF formula term variable {subject_name!r}.*"
+ assert_warning_gated(
+ operation, iris.warnings.IrisCfMissingVarWarning, warn_regex
+ )
+
+
+class TestRepr:
+ def test_repr_contains_terms_by_root(self):
+ subject_name = "ref_sigma"
+ ref_subject = _NetCDFVar(subject_name)
+ ref_source = _NetCDFVar("ref_source")
+ setattr(ref_source, CF_IDENTITY, f"sigma: {subject_name}")
+ vars_all = {
+ subject_name: ref_subject,
+ "ref_source": ref_source,
+ }
+
+ result = _CFFormulaTermsVariable.identify(vars_all)
+ repr_str = repr(result[subject_name])
+ assert "sigma" in repr_str
+ assert "ref_source" in repr_str
diff --git a/lib/iris/tests/unit/fileformats/cf/test__DimensionIdentity.py b/lib/iris/tests/unit/fileformats/cf/test__DimensionIdentity.py
new file mode 100644
index 0000000000..4ee34c1878
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test__DimensionIdentity.py
@@ -0,0 +1,69 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Tests for `_DimensionIdentity`, which decides what is the same dimension.
+
+Equality and hashing are hand-written, and `CFVariable.spans` and
+`iris.fileformats._nc_load_rules.helpers._common_data_dims` both lean on
+them through `set.issubset`, `in` and `tuple.index`. The pairs below are the
+whole contract.
+"""
+
+from iris.fileformats.cf._variables import _DimensionIdentity
+
+
+class Test___eq__:
+ def test_same_name_and_size_are_equal(self):
+ assert _DimensionIdentity("time", 4) == _DimensionIdentity("time", 4)
+
+ def test_same_name_at_two_sizes_is_not_equal(self):
+ # The reason matching is a pair: Zarr lets one name mean two lengths.
+ assert _DimensionIdentity("time", 4) != _DimensionIdentity("time", 5)
+
+ def test_different_names_at_one_size_are_not_equal(self):
+ assert _DimensionIdentity("time", 4) != _DimensionIdentity("level", 4)
+
+ def test_an_anonymous_dimension_equals_itself(self):
+ identity = _DimensionIdentity(None, 4)
+ assert identity == identity
+
+ def test_two_anonymous_dimensions_are_not_equal(self):
+ # Not even at the same size. There is nothing to match on.
+ assert _DimensionIdentity(None, 4) != _DimensionIdentity(None, 4)
+
+ def test_an_anonymous_dimension_does_not_equal_a_named_one(self):
+ assert _DimensionIdentity(None, 4) != _DimensionIdentity("time", 4)
+ assert _DimensionIdentity("time", 4) != _DimensionIdentity(None, 4)
+
+ def test_a_foreign_object_is_not_equal(self):
+ # `__eq__` returns NotImplemented, so Python falls back to identity.
+ assert _DimensionIdentity("time", 4) != ("time", 4)
+
+
+class Test___hash__:
+ def test_same_name_and_size_hash_alike(self):
+ # Equal members must land in the same bucket, or `set.issubset` misses.
+ assert hash(_DimensionIdentity("time", 4)) == hash(
+ _DimensionIdentity("time", 4)
+ )
+
+
+class Test_set_membership:
+ """How `spans` and `_common_data_dims` actually consume the class.
+
+ `__eq__` and `__hash__` imply these, so they are not independent
+ behaviour. They are here because set membership - not the dunders - is
+ the contract those two callers depend on, and an identity that stopped
+ satisfying it would break loading rather than fail a dunder test.
+ """
+
+ def test_named_identities_work_as_a_subset(self):
+ source = {_DimensionIdentity("time", 4)}
+ target = {_DimensionIdentity("time", 4), _DimensionIdentity("level", 3)}
+ assert source.issubset(target)
+
+ def test_an_anonymous_dimension_is_never_a_subset(self):
+ source = {_DimensionIdentity(None, 4)}
+ target = {_DimensionIdentity(None, 4), _DimensionIdentity("level", 3)}
+ assert not source.issubset(target)
diff --git a/lib/iris/tests/unit/fileformats/cf/test___init__.py b/lib/iris/tests/unit/fileformats/cf/test___init__.py
new file mode 100644
index 0000000000..62b9b96ce7
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test___init__.py
@@ -0,0 +1,134 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the public surface of :mod:`iris.fileformats.cf`.
+
+The package is a re-export layer over a private file layout, so what needs
+testing is the layer itself: that it exposes everything it used to, exposes
+nothing it should not, and imports cleanly on its own.
+"""
+
+import ast
+import inspect
+from pathlib import Path
+import subprocess
+import sys
+from types import ModuleType
+
+import pytest
+
+import iris.fileformats.cf as cf
+from iris.fileformats.cf import _group, _reader, _variables
+
+PRIVATE_MODULES = (_variables, _group, _reader)
+
+
+def _origin(value, fallback):
+ """Return the module path a name came from, for values and modules alike."""
+ if isinstance(value, ModuleType):
+ return value.__name__
+ return getattr(value, "__module__", fallback)
+
+
+def _defined_public_names(module):
+ """Return the public names a module's own source binds at the top level.
+
+ Read from the source rather than from ``vars()``, because ``__module__``
+ does not exist at all on data members such as ``reference_terms``. A
+ ``vars()`` walk would therefore have to skip exactly the kind of name that
+ has already gone astray once - see
+ ``test_data_members_are_assigned_in_the_package_source``.
+ """
+ tree = ast.parse(Path(inspect.getsourcefile(module)).read_text())
+ names = set()
+ for node in tree.body:
+ if isinstance(node, (ast.ClassDef, ast.FunctionDef)):
+ names.add(node.name)
+ elif isinstance(node, ast.Assign):
+ names.update(t.id for t in node.targets if isinstance(t, ast.Name))
+ elif isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name):
+ names.add(node.target.id)
+ return {name for name in names if not name.startswith("_")}
+
+
+@pytest.mark.parametrize("module", PRIVATE_MODULES, ids=lambda m: m.__name__)
+def test_public_names_are_re_exported(module):
+ # Review Focus 1. A name defined in a private module and left out of
+ # __all__ vanishes from the API docs without any warning, because autodoc
+ # rejects it on __module__ - see the plan, section 3.2.
+ assert _defined_public_names(module) <= set(cf.__all__)
+
+
+@pytest.mark.parametrize("name", cf.__all__)
+def test_re_exported_classes_keep_their_source(name):
+ # Rewriting __module__ to the package - a tempting way to hide the split
+ # from repr() - makes inspect look for the class body in __init__.py,
+ # where it is not. That breaks getsource, IPython's "??", debuggers, and
+ # silently drops the "[source]" link from every class on the API page,
+ # with no warning in the docs build to say so. See the plan, section 8.4.
+ value = getattr(cf, name)
+ if isinstance(value, type):
+ assert inspect.getsource(value).lstrip().startswith("class ")
+
+
+def test_data_members_are_assigned_in_the_package_source():
+ # Sphinx autodoc documents module data only where the module's own source
+ # assigns it, and drops an imported name silently. Anything in __all__
+ # that is not a class therefore has to be re-stated in cf/__init__.py, or
+ # it disappears from the API reference with no warning at all.
+ assigned = _defined_public_names(cf)
+ data_names = {
+ name for name in cf.__all__ if not isinstance(getattr(cf, name), type)
+ }
+ assert data_names <= assigned
+
+
+@pytest.mark.parametrize("name", cf.__all__)
+def test_all_entries_resolve(name):
+ # Review Focus 1, the other direction: a name in __all__ that the package
+ # cannot supply breaks "from iris.fileformats.cf import *" and the docs.
+ assert hasattr(cf, name)
+
+
+@pytest.mark.parametrize("module", [_variables, _group], ids=lambda m: m.__name__)
+def test_only_the_reader_is_coupled_to_netcdf(module):
+ # Review Focus 3. Confining the netCDF import to _reader is the whole
+ # point of the split; F401 is disabled here, so a stale import left
+ # behind by the cut would otherwise go unremarked.
+ coupled = {
+ name
+ for name, value in vars(module).items()
+ if _origin(value, "").startswith("iris.fileformats.netcdf")
+ }
+ assert coupled == set()
+
+
+@pytest.mark.parametrize(
+ "first_import",
+ ["iris.fileformats.cf", "iris.fileformats.netcdf"],
+)
+def test_imports_in_a_fresh_interpreter(first_import):
+ # Review Focus 4. cf and netcdf import each other, and the split adds a
+ # hop to that cycle. pytest has already imported iris by the time this
+ # module is collected, so a new process is the only way to see it.
+ code = f"import {first_import}\nimport iris.fileformats.cf"
+ completed = subprocess.run(
+ [sys.executable, "-c", code],
+ capture_output=True,
+ text=True,
+ # An import deadlock is one of the failures this is looking for, and
+ # without a timeout it would hang the suite instead of failing it.
+ timeout=120,
+ )
+ assert completed.returncode == 0, completed.stderr
+
+
+@pytest.mark.parametrize(
+ "name",
+ ["_CFFormulaTermsVariable", "_is_str_dtype", "_getncattr", "_CF_PARSE"],
+)
+def test_private_names_are_not_re_exported(name):
+ # Review Focus 5. These were reachable on the flat module and are
+ # deliberately not reachable on the package - spec section 4.1.
+ assert not hasattr(cf, name)
diff --git a/lib/iris/tests/unit/fileformats/cf/test__imports.py b/lib/iris/tests/unit/fileformats/cf/test__imports.py
new file mode 100644
index 0000000000..3284f07dfe
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test__imports.py
@@ -0,0 +1,30 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The CF and netCDF packages import from cold.
+
+The two packages import each other: the CF reader builds a netCDF dataset, and
+the netCDF saver builds on the CF saver. Nothing in the source states the order
+the module bodies must run in.
+
+One import covers every module in both packages. Python runs
+``iris.fileformats.__init__`` before any submodule body, and that imports the
+netCDF package, which draws in the whole cascade.
+
+The subprocess is what makes it a cold start. Inside the test process the
+modules are already in ``sys.modules``, so an import there is a dictionary
+lookup.
+"""
+
+import subprocess
+import sys
+
+
+def test_importing_from_cold_works():
+ completed = subprocess.run(
+ [sys.executable, "-c", "import iris.fileformats"],
+ capture_output=True,
+ text=True,
+ )
+ assert completed.returncode == 0, completed.stderr
diff --git a/lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py b/lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py
new file mode 100644
index 0000000000..65eaa1c7e9
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_no_implementation_coupling.py
@@ -0,0 +1,48 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The CF layer must not branch on which storage implementation it holds.
+
+`isinstance(thing, CFDataset)` tests the *interface*, which is how generic
+code is supposed to accept any backend, and `CFReader` does exactly that.
+What is banned is testing an *implementation*: that is what makes a third
+backend a rewrite rather than a new module. Constructing one is a third
+thing again, and legitimate - `CFReader` opens a `NetCDFDataset` for a bare
+path, which is a default, not a branch on something it was handed.
+"""
+
+from pathlib import Path
+
+import iris.fileformats.cf
+
+#: Implementation class names that may not appear in a type test.
+IMPLEMENTATION_NAMES = (
+ "NetCDFDataset",
+ "NetCDFDatasetVariable",
+ "ZarrDataset",
+ "ZarrDatasetVariable",
+)
+
+#: The spellings of a type test. `__class__` catches the hand-rolled form.
+TYPE_TESTS = ("isinstance", "issubclass", "__class__", "type(")
+
+
+def cf_package_sources():
+ """Yield `(path, text)` for every module in `iris.fileformats.cf`."""
+ package = Path(iris.fileformats.cf.__file__).parent
+ for path in sorted(package.glob("*.py")):
+ yield path, path.read_text(encoding="utf-8")
+
+
+def test_no_module_type_tests_an_implementation_class():
+ offenders = []
+ for path, text in cf_package_sources():
+ for line_number, line in enumerate(text.splitlines(), start=1):
+ if line.lstrip().startswith("#"):
+ continue
+ names_one = any(name in line for name in IMPLEMENTATION_NAMES)
+ tests_type = any(spelling in line for spelling in TYPE_TESTS)
+ if names_one and tests_type:
+ offenders.append(f"{path.name}:{line_number}: {line.strip()}")
+ assert offenders == []
diff --git a/lib/iris/tests/unit/fileformats/cf/test_spans.py b/lib/iris/tests/unit/fileformats/cf/test_spans.py
new file mode 100644
index 0000000000..2bcfcc93d0
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_spans.py
@@ -0,0 +1,203 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""`CFVariable.spans`, asked of both storage backends.
+
+Dimension matching is a pair, `(name, size)`, not a bare name - which is a
+widening to admit what Zarr allows, not a change to what netCDF does. These
+tests run against both so that the second half of that sentence is checked
+rather than asserted.
+"""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf import (
+ CFBoundaryVariable,
+ CFDataVariable,
+ CFLabelVariable,
+)
+
+zarr = pytest.importorskip("zarr")
+
+from iris.fileformats.netcdf._dataset import NetCDFDataset # noqa: E402
+from iris.fileformats.zarr._dataset import ZarrDataset # noqa: E402
+
+
+def build_netcdf(path, described):
+ """Write `described` as a netCDF file and return its open dataset."""
+ dataset = NetCDFDataset(path, mode="w")
+ sizes = {}
+ for dimensions, shape in described.values():
+ for name, size in zip(dimensions, shape):
+ sizes.setdefault(name, size)
+ for name, size in sizes.items():
+ dataset.create_dimension(name, size)
+ for name, (dimensions, shape) in described.items():
+ dataset.create_variable(name, np.dtype("f4"), dimensions)
+ return dataset
+
+
+def build_zarr(path, described):
+ """Write `described` as a Zarr store and return its open dataset."""
+ group = zarr.open_group(store=path, mode="w", zarr_format=3)
+ for name, (dimensions, shape) in described.items():
+ group.create_array(name, shape=shape, dtype="f4", dimension_names=dimensions)
+ return ZarrDataset(str(path))
+
+
+@pytest.fixture(params=[build_netcdf, build_zarr], ids=["netcdf", "zarr"])
+def build(request, tmp_path):
+ """Return a callable giving `{name: CFDataVariable}` for a description."""
+ suffix = ".nc" if request.param is build_netcdf else ".zarr"
+
+ def make(described):
+ dataset = request.param(tmp_path / f"spans{suffix}", described)
+ return {
+ name: CFDataVariable(name, dataset.variables[name]) for name in described
+ }
+
+ return make
+
+
+def test_identical_dimensions_span(build):
+ variables = build({"data": (("y", "x"), (3, 2)), "aux": (("y", "x"), (3, 2))})
+ assert variables["aux"].spans(variables["data"]) is True
+
+
+def test_a_subset_spans(build):
+ variables = build({"data": (("y", "x"), (3, 2)), "aux": (("y",), (3,))})
+ assert variables["aux"].spans(variables["data"]) is True
+
+
+def test_a_different_name_does_not_span(build):
+ variables = build({"data": (("y", "x"), (3, 2)), "aux": (("t",), (3,))})
+ assert variables["aux"].spans(variables["data"]) is False
+
+
+def test_a_scalar_always_spans(build):
+ variables = build({"data": (("y", "x"), (3, 2)), "grid": ((), ())})
+ assert variables["grid"].spans(variables["data"]) is True
+
+
+def test_a_bounds_variable_ignores_its_extent_dimension(build):
+ variables = build({"data": (("y",), (3,)), "bounds": (("y", "bnds"), (3, 2))})
+ bounds = CFBoundaryVariable("bounds", variables["bounds"].cf_data)
+ assert bounds.spans(variables["data"]) is True
+
+
+def test_a_label_variable_ignores_its_string_length_dimension(build):
+ variables = build({"data": (("y",), (3,)), "label": (("y", "strlen"), (3, 8))})
+ label = CFLabelVariable("label", variables["label"].cf_data)
+ assert label.spans(variables["data"]) is True
+
+
+def test_a_name_at_two_sizes_does_not_link(tmp_path):
+ """Zarr-only: a name guarantees nothing across arrays.
+
+ The version 3 core specification "does not place any restrictions on the
+ use of the same dimension name across multiple arrays within the same Zarr
+ hierarchy", so two arrays may mean different things by one name. Matching
+ on the name alone would produce a cube whose coordinate is the wrong
+ length. netCDF cannot express this: a dimension name has one size per
+ group.
+ """
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array("data", shape=(3,), dtype="f4", dimension_names=("t",))
+ group.create_array("aux", shape=(5,), dtype="f4", dimension_names=("t",))
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ data = CFDataVariable("data", dataset.variables["data"])
+ aux = CFDataVariable("aux", dataset.variables["aux"])
+ assert aux.spans(data) is False
+
+
+def test_an_anonymous_dimension_matches_nothing(tmp_path):
+ """Not even another anonymous dimension, of the same size, elsewhere.
+
+ Bare-name matching treats `None == None` as a match: `aux` would span
+ `data` under the old rule, for a dimension neither variable actually
+ named. There is nothing shared between two differently-sourced "no
+ name"s, so claiming a match would be invention.
+ """
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array("data", shape=(3,), dtype="f4")
+ group.create_array("aux", shape=(3,), dtype="f4")
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ data = CFDataVariable("data", dataset.variables["data"])
+ aux = CFDataVariable("aux", dataset.variables["aux"])
+ assert aux.spans(data) is False
+
+
+def test_two_anonymous_dimensions_of_one_size_do_not_match(tmp_path):
+ """Not even each other, and not within one variable.
+
+ There is no information to match on, so claiming a match would be
+ invention.
+ """
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array("data", shape=(3, 3), dtype="f4")
+ group.create_array("aux", shape=(3,), dtype="f4")
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ data = CFDataVariable("data", dataset.variables["data"])
+ aux = CFDataVariable("aux", dataset.variables["aux"])
+ assert aux.spans(data) is False
+
+
+def test_an_anonymous_dimension_is_not_a_coordinate_variable(tmp_path):
+ """A one-dimensional array is a coordinate variable by name coincidence.
+
+ `CFCoordinateVariable.identify` requires the variable's own name to be
+ among its dimension names, which no anonymous dimension satisfies - so an
+ anonymous one-dimensional array is never mistaken for a coordinate
+ variable, whatever it is named.
+ """
+ from iris.fileformats.cf import CFCoordinateVariable
+
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array("t", shape=(3,), dtype="f4")
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ assert CFCoordinateVariable.identify(dataset.variables) == {}
+
+
+def test_a_partly_named_variable_matches_on_the_named_part(tmp_path):
+ # "A variable may name some of its dimensions and not others." The named
+ # part ("y") is present in the target and would be enough to span under
+ # the old rule once the anonymous part is set aside - so `data` carries
+ # an anonymous dimension of its own, of the same size as `aux`'s, to show
+ # that one does not rescue the match either: bare-name matching would
+ # read both `None`s as the same dimension and wrongly call this a span.
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ group.create_array(
+ "data", shape=(3, 2, 2), dtype="f4", dimension_names=("y", "x", None)
+ )
+ group.create_array("aux", shape=(3, 2), dtype="f4", dimension_names=("y", None))
+
+ dataset = ZarrDataset(str(tmp_path / "s.zarr"))
+ data = CFDataVariable("data", dataset.variables["data"])
+ aux = CFDataVariable("aux", dataset.variables["aux"])
+ assert aux.spans(data) is False
+
+
+def test_a_non_spanning_variable_is_dropped_and_promoted(tmp_path):
+ """An anonymous-dimension variable becomes a cube of its own.
+
+ This is what keeps `None` out of `iris.fileformats._nc_load_rules.helpers`,
+ which matches dimensions by name and would read `None in (None, "x")` as a
+ match. `CFReader._build_cf_groups` drops any variable whose `spans` is
+ False from the referrer's group and promotes it.
+ """
+ from iris.fileformats.cf import CFReader
+
+ group = zarr.open_group(store=tmp_path / "s.zarr", mode="w", zarr_format=3)
+ data = group.create_array("data", shape=(3,), dtype="f4", dimension_names=("t",))
+ data.attrs["coordinates"] = "aux"
+ group.create_array("aux", shape=(3,), dtype="f4")
+
+ with CFReader(ZarrDataset(str(tmp_path / "s.zarr"))) as reader:
+ assert "aux" not in reader.cf_group["data"].cf_group
+ assert "aux" in reader.cf_group.promoted
diff --git a/lib/iris/tests/unit/fileformats/cf/test_storage_chunk_warning.py b/lib/iris/tests/unit/fileformats/cf/test_storage_chunk_warning.py
new file mode 100644
index 0000000000..c6cf546d55
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/cf/test_storage_chunk_warning.py
@@ -0,0 +1,96 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The warning raised when a stored chunk is larger than Dask's target.
+
+Iris sizes Dask chunks to Dask's target and does not round back up to the
+store's own chunking, so a large stored chunk is divided and then fetched once
+per division. The warning is what tells a caller there is a `CHUNK_CONTROL`
+decision to make. It has to reach both storage backends, or netCDF and Zarr
+would behave differently over physics they share.
+"""
+
+import warnings
+
+import dask.config
+import numpy as np
+import pytest
+
+from iris.fileformats.cf import CFDataVariable
+from iris.fileformats.cf.loader import CHUNK_CONTROL, _chunks_from_chunk_control
+from iris.warnings import IrisLoadWarning
+
+zarr = pytest.importorskip("zarr")
+
+from iris.fileformats.netcdf._dataset import NetCDFDataset # noqa: E402
+from iris.fileformats.zarr._dataset import ZarrDataset # noqa: E402
+
+SHAPE = (64,)
+#: Stored chunk of 32 float64 values, which is 256 bytes.
+BIG_CHUNK = (32,)
+#: Stored chunk of 4 float64 values, which is 32 bytes.
+SMALL_CHUNK = (4,)
+#: Sits between the two, so one choice of stored chunk crosses it and the
+#: other does not.
+TARGET = "128B"
+
+
+def build_netcdf(path, chunks):
+ dataset = NetCDFDataset(str(path), mode="w")
+ dataset.create_dimension("t", SHAPE[0])
+ dataset.create_variable("v", np.dtype("f8"), ("t",), chunksizes=chunks)
+ dataset.close()
+ return NetCDFDataset(str(path))
+
+
+def build_zarr(path, chunks):
+ group = zarr.open_group(store=path, mode="w", zarr_format=3)
+ group.create_array(
+ "v", shape=SHAPE, dtype="f8", chunks=chunks, dimension_names=("t",)
+ )
+ return ZarrDataset(str(path))
+
+
+@pytest.fixture(params=[build_netcdf, build_zarr], ids=["netcdf", "zarr"])
+def build(request, tmp_path):
+ """Return a callable giving the `CFDataVariable` for a stored chunking."""
+ suffix = ".nc" if request.param is build_netcdf else ".zarr"
+
+ def make(chunks):
+ dataset = request.param(tmp_path / f"chunked{suffix}", chunks)
+ return CFDataVariable("v", dataset.variables["v"])
+
+ return make
+
+
+def test_a_stored_chunk_over_the_target_warns(build):
+ cf_var = build(BIG_CHUNK)
+ with dask.config.set({"array.chunk-size": TARGET}):
+ with pytest.warns(IrisLoadWarning, match="CHUNK_CONTROL"):
+ _chunks_from_chunk_control(cf_var)
+
+
+def test_the_warning_names_the_variable(build):
+ cf_var = build(BIG_CHUNK)
+ with dask.config.set({"array.chunk-size": TARGET}):
+ with pytest.warns(IrisLoadWarning, match="v$"):
+ _chunks_from_chunk_control(cf_var)
+
+
+def test_a_stored_chunk_under_the_target_is_quiet(build):
+ cf_var = build(SMALL_CHUNK)
+ with dask.config.set({"array.chunk-size": TARGET}):
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisLoadWarning)
+ _chunks_from_chunk_control(cf_var)
+
+
+def test_a_pinned_chunking_is_not_second_guessed(build):
+ """A caller who has set the chunking has already made this decision."""
+ cf_var = build(BIG_CHUNK)
+ with dask.config.set({"array.chunk-size": TARGET}):
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisLoadWarning)
+ with CHUNK_CONTROL.set("v", t=8):
+ _chunks_from_chunk_control(cf_var)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/__init__.py b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/__init__.py
index 0b9caabdf7..d6423c0c4b 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/__init__.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/__init__.py
@@ -10,8 +10,7 @@
import iris.fileformats._nc_load_rules.engine
from iris.fileformats.cf import CFReader
-import iris.fileformats.netcdf
-from iris.fileformats.netcdf.loader import _load_cube
+from iris.fileformats.cf.loader import _load_cube
from iris.tests import _shared_utils
from iris.tests.stock.netcdf import ncgen_from_cdl
from iris.warnings import IrisLoadWarning
@@ -73,7 +72,7 @@ def load_cube_from_cdl(self, cdl_string, cdl_path, nc_path, mocker=None):
# Grab a data variable : FOR NOW always grab the 'phenom' variable.
cf_var = cf.cf_group.data_variables["phenom"]
- engine = iris.fileformats.netcdf.loader._actions_engine()
+ engine = iris.fileformats.cf.loader._actions_engine()
# If debug enabled, switch on the activation summary debug output.
# Use 'patch' so it is restored after the test.
@@ -101,7 +100,7 @@ def load_cube_from_cdl(self, cdl_string, cdl_path, nc_path, mocker=None):
# by the rules operation.
# Unlike the other translations, _load_cube does *not* convert this
# information into actual cube elements. That is instead done by
- # `iris.fileformats.netcdf._load_aux_factory`.
+ # `iris.fileformats.cf.loader._load_aux_factory`.
# For rules testing, it is anyway more convenient to deal with the raw
# data, as each factory type has different validity requirements to
# build it, and none of that is relevant to the rules operation.
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__grid_mappings.py b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__grid_mappings.py
index 41d5c33623..063d9f5ef9 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__grid_mappings.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__grid_mappings.py
@@ -3,7 +3,7 @@
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for the engine.activate() call within the
-`iris.fileformats.netcdf._load_cube` function.
+`iris.fileformats.cf.loader._load_cube` function.
Here, *specifically* testcases relating to grid-mappings and dim-coords.
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__hybrid_formulae.py b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__hybrid_formulae.py
index 838cb8b9c7..ed0f8bf991 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__hybrid_formulae.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__hybrid_formulae.py
@@ -3,7 +3,7 @@
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for the engine.activate() call within the
-`iris.fileformats.netcdf._load_cube` function.
+`iris.fileformats.cf.loader._load_cube` function.
Test rules activation relating to hybrid vertical coordinates.
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__latlon_dimcoords.py b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__latlon_dimcoords.py
index 9e534521f1..2f1823c941 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__latlon_dimcoords.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__latlon_dimcoords.py
@@ -3,7 +3,7 @@
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for the engine.activate() call within the
-`iris.fileformats.netcdf._load_cube` function.
+`iris.fileformats.cf.loader._load_cube` function.
Tests for rules behaviour in identifying latitude/longitude dim-coords, both
rotated and non-rotated.
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__miscellaneous.py b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__miscellaneous.py
index 283f01920c..5f907cb9b9 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__miscellaneous.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__miscellaneous.py
@@ -3,7 +3,7 @@
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for the engine.activate() call within the
-`iris.fileformats.netcdf._load_cube` function.
+`iris.fileformats.cf.loader._load_cube` function.
Tests for rules activation relating to some isolated aspects :
* UKMO um-specific metadata
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__time_coords.py b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__time_coords.py
index 6c13c35144..970d5eef73 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__time_coords.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/actions/test__time_coords.py
@@ -3,7 +3,7 @@
# This file is part of Iris and is released under the BSD license.
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for the engine.activate() call within the
-`iris.fileformats.netcdf._load_cube` function.
+`iris.fileformats.cf.loader._load_cube` function.
Tests for rules activation relating to 'time' and 'time_period' coords.
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/__init__.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/__init__.py
index 36ea8c9953..762457522a 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/__init__.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/__init__.py
@@ -10,6 +10,10 @@
import pytest
from pytest_mock import MockerFixture
+from iris.fileformats.cf import CFDataVariable
+from iris.fileformats.cf._variables import _CF_ATTRS_IGNORE
+from iris.fileformats.cf.dataset import TrackedAttributes
+
class MockerMixin:
mocker: MockerFixture
@@ -17,3 +21,129 @@ class MockerMixin:
@pytest.fixture(autouse=True)
def _mocker_mixin_setup(self, mocker):
self.mocker = mocker
+
+
+class CFVariableStandIn:
+ """A minimal stand-in for :class:`iris.fileformats.cf.CFVariable`.
+
+ The loading rules read a CF variable's file attributes through exactly
+ two things: a real :class:`~iris.fileformats.cf.dataset.TrackedAttributes`
+ mapping at ``.attributes``, and ``CFVariable.__getattr__`` routing unknown
+ names to it. This class models both, so a test can read
+ ``stand_in.some_name`` to build its own expected value and the production
+ code's ``stand_in.attributes.get("some_name")`` resolves the identical
+ value from the same underlying mapping - unlike a flat
+ ``Mock(some_name=...)``, which has nothing behind ``.attributes`` at all.
+
+ Every keyword given becomes a CF attribute: present in ``.attributes``
+ and returned via plain attribute access (through ``__getattr__``). A name
+ not given raises ``AttributeError`` on plain access and, through
+ ``.attributes.get``, returns the caller's default - exactly as a real
+ missing file attribute would. Structural members that production
+ ``CFVariable`` exposes as plain instance attributes (``cf_name``,
+ ``cf_group``, ``cf_data``, ``filename``, ...) are not modelled here;
+ set them directly on the instance after construction, as real
+ ``CFVariable`` does outside ``.attributes``.
+ """
+
+ def __init__(self, **attributes):
+ # Seed with the same ignored names production CFVariable.__init__
+ # does: scale_factor, add_offset and friends must start already-read
+ # here too, or a test that puts one into a stand-in sees it as unread
+ # (and so, e.g., surviving onto a built object's attributes) when a
+ # real load never would.
+ self.attributes = TrackedAttributes(dict(attributes), ignored=_CF_ATTRS_IGNORE)
+
+ def __getattr__(self, name):
+ # Mirrors production CFVariable.__getattr__'s recursion guard, keyed
+ # on _GETATTR_RECURSION_GUARD: "attributes" is what this method reads
+ # to resolve anything else, so it must never be resolved by this method
+ # itself. Without this, an instance with no "attributes" yet in its
+ # __dict__ - e.g. `cls.__new__(cls)` during copy/deepcopy/unpickling,
+ # then a `__setstate__` probe - recurses infinitely instead of
+ # raising AttributeError.
+ if name.startswith("__") or name == "attributes":
+ raise AttributeError(name)
+ try:
+ return self.attributes[name]
+ except KeyError:
+ raise AttributeError(name) from None
+
+ def __getitem__(self, key):
+ """Index the stand-in's data, exactly as ``CFVariable.__getitem__`` does.
+
+ Real ``CFVariable.__getitem__`` returns ``self.cf_data[key]``; a test
+ that needs indexing sets ``.cf_data`` to an indexable data array, the
+ same way it sets any other structural member.
+ """
+ return self.cf_data[key]
+
+ def cf_attrs(self):
+ """Return all attribute name/value pairs, exactly as ``CFVariable`` does.
+
+ Used by the "last resort" raw-cube fallback (``build_raw_cube``), so a
+ stand-in that hits that path still works without further setup.
+ """
+ attributes = self.attributes.untracked
+ return tuple((name, attributes[name]) for name in sorted(attributes))
+
+
+class RealArrayCfData:
+ """Wrap a real (non-lazy) array as a :class:`CFVariableStandIn`'s ``cf_data``.
+
+ ``read_data`` reads ``is_emulated`` and ``is_variable_length`` before it
+ ever looks at size, so a stand-in backed directly by a plain array - as
+ most of these tests are, since the array given is the coordinate's real
+ data rather than something read from a file - needs this much of the
+ storage interface even though the array is always far too small to reach
+ ``.chunking``/``.variable``, the two members only the lazy-loading branch
+ reads.
+ """
+
+ is_emulated = False
+ is_variable_length = False
+
+ def __init__(self, array):
+ self._array = array
+
+ def __getitem__(self, key):
+ return self._array[key]
+
+ def read_data(self, chunking_policy):
+ """Return the real array, exactly as the small-variable path would.
+
+ Always far too small to reach the lazy branch, so ``chunking_policy``
+ is never consulted.
+ """
+ return self._array[:]
+
+
+class _MinimalStorage:
+ """The least a :class:`~iris.fileformats.cf.CFVariable` can be built over.
+
+ ``CFVariable.__init__`` reads ``attributes`` (copied into a fresh
+ :class:`~iris.fileformats.cf.dataset.TrackedAttributes`) and ``location``
+ (which becomes ``filename``); ``dtype`` is read later, by the read-only
+ ``CFVariable.dtype`` property. Nothing else here is needed to build one.
+ """
+
+ def __init__(self, attributes, location, dtype):
+ self.attributes = attributes
+ self.location = location
+ self.dtype = dtype
+
+
+def real_cf_data_variable(name="wibble", location="DUMMY", dtype=float, **attributes):
+ """Build a genuine :class:`~iris.fileformats.cf.CFDataVariable`.
+
+ ``get_attr_units`` asserts ``isinstance(cf_var, cf.CFDataVariable)`` on the
+ ``capture_invalid=True`` branch it takes when building a Cube's own units.
+ :class:`CFVariableStandIn` cannot satisfy that assert - it deliberately does
+ not inherit from ``CFDataVariable`` - so a test reaching that branch builds
+ a real one here instead, over storage minimal enough to construct inline.
+
+ Prefer :class:`CFVariableStandIn` everywhere else. The real class takes its
+ ``dtype``, ``shape``, ``ndim`` and ``size`` from storage through read-only
+ properties, so a test cannot simply assign them.
+ """
+ return CFDataVariable(name, _MinimalStorage(dict(attributes), location, dtype))
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__add_or_capture.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__add_or_capture.py
index cadd2efa62..65efdef6f0 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__add_or_capture.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__add_or_capture.py
@@ -9,8 +9,8 @@
from iris.cube import Cube
from iris.fileformats._nc_load_rules import helpers
-from iris.fileformats.cf import CFVariable
from iris.loading import LOAD_PROBLEMS, LoadProblems
+from iris.tests.unit.fileformats.nc_load_rules.helpers import CFVariableStandIn
class Mixin:
@@ -30,9 +30,8 @@ def make_args(self, mocker):
self.build_func = mocker.MagicMock()
self.build_func.return_value = "BUILT"
self.add_method = mocker.MagicMock()
- self.cf_var = mocker.MagicMock(spec=CFVariable)
+ self.cf_var = CFVariableStandIn(**{self.attr_key: self.attr_value})
self.cf_var.filename = self.filename
- setattr(self.cf_var, self.attr_key, self.attr_value)
def call(
self,
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__common_data_dims.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__common_data_dims.py
new file mode 100644
index 0000000000..5ba346b721
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__common_data_dims.py
@@ -0,0 +1,75 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Test function :func:`iris.fileformats._nc_load_rules.helpers._common_data_dims`."""
+
+from dataclasses import dataclass
+
+from iris.fileformats._nc_load_rules.helpers import _common_data_dims
+
+
+@dataclass
+class Variable:
+ """The two members the dimension rule reads off a CF variable."""
+
+ dimensions: tuple
+ shape: tuple
+
+
+def test_the_one_shared_dimension():
+ coord = Variable(("x",), (3,))
+ data = Variable(("y", "x"), (2, 3))
+ assert _common_data_dims(coord, data) == [1]
+
+
+def test_several_shared_dimensions_keep_the_coordinates_order():
+ coord = Variable(("x", "y"), (3, 2))
+ data = Variable(("y", "x"), (2, 3))
+ assert _common_data_dims(coord, data) == [1, 0]
+
+
+def test_a_scalar_variable_shares_nothing():
+ # `None`, not `[]`: it is what `Cube.add_aux_coord` takes for "no
+ # dimensions", and an empty list is not.
+ assert _common_data_dims(Variable((), ()), Variable(("x",), (3,))) is None
+
+
+def test_a_dimension_the_data_variable_does_not_have():
+ assert _common_data_dims(Variable(("t",), (4,)), Variable(("x",), (3,))) is None
+
+
+def test_the_same_name_at_a_different_length_shares_nothing():
+ """A name is not an identity once a Zarr store is in scope.
+
+ `CFVariable.spans` decided this the same way before the coordinate got
+ this far, so answering differently here would attach a coordinate that
+ was never found to span the data variable.
+ """
+ assert _common_data_dims(Variable(("x",), (3,)), Variable(("x",), (4,))) is None
+
+
+def test_a_repeated_dimension_name_is_resolved_by_length():
+ """The case a Zarr store can present and no netCDF file can.
+
+ A netCDF dimension name is unique within its group and carries one size;
+ a Zarr dimension name is a label with no such guarantee, so one variable
+ may name both of its dimensions alike at two lengths. Matched on the bare
+ name, the coordinate lands on dimension 0 and is rejected as the wrong
+ length.
+ """
+ coord = Variable(("d",), (3,))
+ data = Variable(("d", "d"), (2, 3))
+ assert _common_data_dims(coord, data) == [1]
+
+
+def test_an_anonymous_dimension_matches_nothing():
+ """Not even another anonymous dimension: there is nothing to match on.
+
+ A Zarr version 3 store may leave a dimension unnamed, and
+ `CFVariable.spans` rejects a coordinate that has one, so this is the same
+ answer arrived at twice rather than two rules disagreeing.
+ """
+ coord = Variable((None,), (3,))
+ data = Variable((None,), (3,))
+ assert _common_data_dims(coord, data) is None
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__normalise_bounds_units.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__normalise_bounds_units.py
index 4d85ccacd6..23fbfdb908 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__normalise_bounds_units.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test__normalise_bounds_units.py
@@ -4,18 +4,20 @@
# See LICENSE in the root of the repository for full licensing details.
"""Test function :func:`iris.fileformats._nc_load_rules.helpers._normalise_bounds_units`."""
-from typing import Optional
+from typing import Any
import numpy as np
import pytest
-from pytest_mock import MockType
from iris.fileformats._nc_load_rules.helpers import (
_normalise_bounds_units,
_WarnComboIgnoringCfLoad,
)
from iris.tests import _shared_utils
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
from iris.warnings import IrisCfLoadWarning
CF_NAME = "dummy_bnds"
@@ -27,38 +29,40 @@ def _setup(self):
self.bounds = self.mocker.sentinel.bounds
def _make_cf_bounds_var(
- self,
- units: Optional[str] = None,
- unitless: bool = False,
- ) -> MockType:
- """Construct a mock CF bounds variable."""
+ self, units: str | None = None, unitless: bool = False
+ ) -> CFVariableStandIn:
+ """Construct a stand-in CF bounds variable.
+
+ Deliberately no ``flag_values``/``flag_masks``/``flag_meanings``: their
+ absence from ``.attributes`` is what tells ``helpers.get_attr_units``
+ this is not a flag variable.
+ """
if units is None:
units = "days since 1970-01-01"
- cf_data = self.mocker.Mock(spec=[])
- # we want to mock the absence of flag attributes to helpers.get_attr_units
- # see https://docs.python.org/3/library/unittest.mock.html#deleting-attributes
- del cf_data.flag_values
- del cf_data.flag_masks
- del cf_data.flag_meanings
-
- cf_var = self.mocker.MagicMock(
- cf_name=CF_NAME,
- cf_data=cf_data,
- units=units,
- calendar=None,
- dtype=float,
- )
-
- if unitless:
- del cf_var.units
+ attrs: dict[str, Any] = {"calendar": None}
+ if not unitless:
+ attrs["units"] = units
+ cf_var = CFVariableStandIn(**attrs)
+ # cf_name/dtype are structural members of a real CFVariable, not CF
+ # attributes, so CFVariableStandIn doesn't declare them - mypy can't
+ # see that setting them here is exactly what the docstring prescribes.
+ cf_var.cf_name = CF_NAME # type: ignore[attr-defined]
+ cf_var.dtype = float # type: ignore[attr-defined]
return cf_var
def test_unitless(self) -> None:
"""Test bounds variable with no units."""
cf_bounds_var = self._make_cf_bounds_var(unitless=True)
- result = _normalise_bounds_units(None, cf_bounds_var, self.bounds)
+ # cf_bounds_var is deliberately a CFVariableStandIn, not a real
+ # CFBoundaryVariable - see the class docstring. Repeated below at
+ # every other call for the same reason.
+ result = _normalise_bounds_units(
+ None,
+ cf_bounds_var, # type: ignore[arg-type]
+ self.bounds,
+ )
assert result == self.bounds
def test_invalid_units__pass_through(self) -> None:
@@ -67,7 +71,11 @@ def test_invalid_units__pass_through(self) -> None:
cf_bounds_var = self._make_cf_bounds_var(units=units)
wmsg = f"Ignoring invalid units {units!r} on netCDF variable {CF_NAME!r}"
with pytest.warns(_WarnComboIgnoringCfLoad, match=wmsg):
- result = _normalise_bounds_units(None, cf_bounds_var, self.bounds)
+ result = _normalise_bounds_units(
+ None,
+ cf_bounds_var, # type: ignore[arg-type]
+ self.bounds,
+ )
assert result == self.bounds
@pytest.mark.parametrize("units", ["unknown", "no_unit", "1", "kelvin"])
@@ -80,7 +88,11 @@ def test_ignore_bounds(self, units) -> None:
f"Expected units compatible with {points_units!r}"
)
with pytest.warns(IrisCfLoadWarning, match=wmsg):
- result = _normalise_bounds_units(points_units, cf_bounds_var, self.bounds)
+ result = _normalise_bounds_units(
+ points_units,
+ cf_bounds_var, # type: ignore[arg-type]
+ self.bounds,
+ )
assert result is None
def test_compatible(self) -> None:
@@ -88,7 +100,11 @@ def test_compatible(self) -> None:
points_units, bounds_units = "days since 1970-01-01", "hours since 1970-01-01"
cf_bounds_var = self._make_cf_bounds_var(units=bounds_units)
bounds = np.arange(10, dtype=float) * 24
- result = _normalise_bounds_units(points_units, cf_bounds_var, bounds)
+ result = _normalise_bounds_units(
+ points_units,
+ cf_bounds_var, # type: ignore[arg-type]
+ bounds,
+ )
expected = bounds / 24
_shared_utils.assert_array_equal(result, expected)
@@ -97,6 +113,10 @@ def test_same_units(self) -> None:
units = "days since 1970-01-01"
cf_bounds_var = self._make_cf_bounds_var(units=units)
bounds = np.arange(10, dtype=float)
- result = _normalise_bounds_units(units, cf_bounds_var, bounds)
+ result = _normalise_bounds_units(
+ units,
+ cf_bounds_var, # type: ignore[arg-type]
+ bounds,
+ )
_shared_utils.assert_array_equal(result, bounds)
assert result is bounds
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_albers_equal_area_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_albers_equal_area_coordinate_system.py
index d9a33dd948..69aeed1930 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_albers_equal_area_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_albers_equal_area_coordinate_system.py
@@ -12,7 +12,10 @@
from iris.fileformats._nc_load_rules.helpers import (
build_albers_equal_area_coordinate_system,
)
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class TestBuildAlbersEqualAreaCoordinateSystem(MockerMixin):
@@ -52,7 +55,7 @@ def _test(self, inverse_flattening=False, no_optionals=False):
gridvar_props["semi_minor_axis"] = 6356256.909
expected_ellipsoid = iris.coord_systems.GeogCS(6377563.396, 6356256.909)
- cf_grid_var = self.mocker.Mock(spec=[], **gridvar_props)
+ cf_grid_var = CFVariableStandIn(**gridvar_props)
cs = build_albers_equal_area_coordinate_system(None, cf_grid_var)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_ancil_var.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_ancil_var.py
index ce2fd7bf8b..0fd7f3fb89 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_ancil_var.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_ancil_var.py
@@ -11,15 +11,21 @@
from iris.cube import Cube
from iris.exceptions import CannotAddError
from iris.fileformats._nc_load_rules.helpers import build_and_add_ancil_var
-from iris.fileformats.cf import CFAncillaryDataVariable
from iris.loading import LOAD_PROBLEMS
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ RealArrayCfData,
+)
@pytest.fixture
def mock_engine(mocker):
return mocker.Mock(
cube=mocker.Mock(),
- cf_var=mocker.Mock(dimensions=("foo", "bar")),
+ # A data variable has a shape as well as dimension names, because an
+ # ancillary variable is attached to the dimensions it matches on
+ # `(name, size)` - see `helpers._common_data_dims`.
+ cf_var=mocker.Mock(dimensions=("foo", "bar"), shape=(6, 2)),
filename="DUMMY",
cube_parts=dict(ancillary_variables=[]),
)
@@ -28,22 +34,16 @@ def mock_engine(mocker):
@pytest.fixture
def mock_cf_av_var(mocker, monkeypatch, mock_engine):
data = np.arange(6)
- output = mocker.Mock(
- spec=CFAncillaryDataVariable,
- dimensions=("foo",),
- scale_factor=1,
- add_offset=0,
- cf_name="wibble",
- cf_data=mocker.MagicMock(chunking=mocker.Mock(return_value=None), spec=[]),
- filename=mock_engine.filename,
- standard_name=None,
- long_name="wibble",
- units="m2",
- shape=data.shape,
- size=np.prod(data.shape),
- dtype=data.dtype,
- __getitem__=lambda self, key: data[key],
- )
+ output = CFVariableStandIn(standard_name=None, long_name="wibble", units="m2")
+ output.dimensions = ("foo",)
+ output.scale_factor = 1
+ output.add_offset = 0
+ output.cf_name = "wibble"
+ output.cf_data = RealArrayCfData(data)
+ output.filename = mock_engine.filename
+ output.shape = data.shape
+ output.size = np.prod(data.shape)
+ output.dtype = data.dtype
# Create patch for deferred loading that prevents attempted
# file access. This assumes that output is defined in the test case.
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_auxiliary_coordinate.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_auxiliary_coordinate.py
index 0d4b16e8da..1d33aed985 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_auxiliary_coordinate.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_auxiliary_coordinate.py
@@ -8,6 +8,7 @@
"""
import contextlib
+from functools import partial
import numpy as np
import pytest
@@ -17,9 +18,13 @@
from iris.cube import Cube
from iris.exceptions import CannotAddError
from iris.fileformats._nc_load_rules.helpers import build_and_add_auxiliary_coordinate
-from iris.fileformats.cf import CFVariable
+import iris.fileformats.netcdf._dataset
from iris.loading import LOAD_PROBLEMS
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+ RealArrayCfData,
+)
class TestBoundsVertexDim(MockerMixin):
@@ -41,27 +46,29 @@ def _setup(self, mocker):
self.engine = mocker.Mock(
cube=mocker.Mock(),
+ # A data variable has a shape as well as dimension names, because
+ # a coordinate is attached to the dimensions it matches on
+ # `(name, size)` - see `helpers._common_data_dims`.
cf_var=mocker.Mock(
- dimensions=("foo", "bar"), cf_data=cf_data, dtype=np.int32
+ dimensions=("foo", "bar"),
+ shape=(self.dim_names_lens["foo"], self.dim_names_lens["bar"]),
+ cf_data=cf_data,
+ dtype=np.int32,
),
filename="DUMMY",
cube_parts=dict(coordinates=[]),
)
- self.cf_coord_var = mocker.Mock(
- spec=CFVariable,
- dimensions=dimension_names,
- cf_name="wibble",
- cf_data=cf_data,
- filename=self.engine.filename,
- standard_name=None,
- long_name="wibble",
- units="km",
- shape=points.shape,
- size=np.prod(points.shape),
- dtype=points.dtype,
- __getitem__=lambda self, key: points[key],
+ self.cf_coord_var = CFVariableStandIn(
+ standard_name=None, long_name="wibble", units="km"
)
+ self.cf_coord_var.dimensions = dimension_names
+ self.cf_coord_var.cf_name = "wibble"
+ self.cf_coord_var.cf_data = RealArrayCfData(points)
+ self.cf_coord_var.filename = self.engine.filename
+ self.cf_coord_var.shape = points.shape
+ self.cf_coord_var.size = np.prod(points.shape)
+ self.cf_coord_var.dtype = points.dtype
expected_bounds, _ = self._make_array_and_cf_data(
mocker, dimension_names=("foo", "bar", "nv")
@@ -119,18 +126,14 @@ def _make_cf_bounds_var(self, mocker, dimension_names, rollaxis=False):
mocker, dimension_names, rollaxis=rollaxis
)
bounds *= 1000 # Convert to metres.
- cf_bounds_var = self.mocker.Mock(
- spec=CFVariable,
- dimensions=dimension_names,
- cf_name="wibble_bnds",
- cf_data=cf_data,
- filename=self.engine.filename,
- units="m",
- shape=bounds.shape,
- size=np.prod(bounds.shape),
- dtype=bounds.dtype,
- __getitem__=lambda self, key: bounds[key],
- )
+ cf_bounds_var = CFVariableStandIn(units="m")
+ cf_bounds_var.dimensions = dimension_names
+ cf_bounds_var.cf_name = "wibble_bnds"
+ cf_bounds_var.cf_data = RealArrayCfData(bounds)
+ cf_bounds_var.filename = self.engine.filename
+ cf_bounds_var.shape = bounds.shape
+ cf_bounds_var.size = np.prod(bounds.shape)
+ cf_bounds_var.dtype = bounds.dtype
return cf_bounds_var
@@ -169,29 +172,50 @@ class TestDtype(MockerMixin):
def _setup(self, mocker):
# Create coordinate cf variables and pyke engine.
points = np.arange(6).reshape(2, 3)
- cf_data = mocker.MagicMock(_FillValue=None, shape=points.shape)
- cf_data.chunking = mocker.MagicMock(return_value=points.shape)
+ # `cf_data` must support both `.chunking` (the lazy-loading path,
+ # forced on by `deferred_load_patch` below, which also needs
+ # `.variable` for the proxy it builds) and real indexing (via
+ # `CFVariableStandIn.__getitem__`, which reads `self.cf_data[key]`
+ # exactly as production `CFVariable.__getitem__` does).
+ cf_data = mocker.MagicMock(
+ _FillValue=None,
+ shape=points.shape,
+ size=points.size,
+ dtype=points.dtype,
+ is_emulated=False,
+ is_variable_length=False,
+ chunking=points.shape,
+ variable=mocker.MagicMock(shape=points.shape),
+ )
+ cf_data.__getitem__ = mocker.MagicMock(side_effect=lambda key: points[key])
self.engine = mocker.Mock(
cube=mocker.Mock(),
- cf_var=mocker.Mock(dimensions=("foo", "bar"), dtype=np.int32),
+ cf_var=mocker.Mock(
+ dimensions=("foo", "bar"), shape=points.shape, dtype=np.int32
+ ),
filename="DUMMY",
cube_parts=dict(coordinates=[]),
)
- self.cf_coord_var = mocker.Mock(
- spec=CFVariable,
- dimensions=("foo", "bar"),
- cf_name="wibble",
- cf_data=cf_data,
- filename=self.engine.filename,
- standard_name=None,
- long_name="wibble",
- units="m",
- shape=points.shape,
- size=np.prod(points.shape),
- dtype=points.dtype,
- __getitem__=lambda self, key: points[key],
+ self.cf_coord_var = CFVariableStandIn(
+ standard_name=None, long_name="wibble", units="m"
+ )
+ self.cf_coord_var.dimensions = ("foo", "bar")
+ self.cf_coord_var.cf_name = "wibble"
+ self.cf_coord_var.cf_data = cf_data
+ self.cf_coord_var.filename = self.engine.filename
+ self.cf_coord_var.shape = points.shape
+ self.cf_coord_var.size = np.prod(points.shape)
+ self.cf_coord_var.dtype = points.dtype
+
+ # read_data reads scale_factor/add_offset off the storage's own
+ # attributes, so cf_data shares the coord's TrackedAttributes rather
+ # than carrying a second, independent copy.
+ cf_data.location = self.cf_coord_var.filename
+ cf_data.attributes = self.cf_coord_var.attributes
+ cf_data.read_data = partial(
+ iris.fileformats.netcdf._dataset.NetCDFDatasetVariable.read_data, cf_data
)
@contextlib.contextmanager
@@ -207,12 +231,12 @@ def patched__getitem__(proxy_self, keys):
new=patched__getitem__,
)
# While loading, "turn off" loading small variables as real data.
- self.mocker.patch("iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES", 0)
+ self.mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
yield
def test_scale_factor_add_offset_int(self):
- self.cf_coord_var.scale_factor = 3
- self.cf_coord_var.add_offset = 5
+ self.cf_coord_var.attributes["scale_factor"] = 3
+ self.cf_coord_var.attributes["add_offset"] = 5
build_and_add_auxiliary_coordinate(self.engine, self.cf_coord_var)
@@ -220,7 +244,7 @@ def test_scale_factor_add_offset_int(self):
assert coord.dtype.kind == "i"
def test_scale_factor_float(self):
- self.cf_coord_var.scale_factor = 3.0
+ self.cf_coord_var.attributes["scale_factor"] = 3.0
with self.deferred_load_patch():
build_and_add_auxiliary_coordinate(self.engine, self.cf_coord_var)
@@ -229,7 +253,7 @@ def test_scale_factor_float(self):
assert coord.dtype.kind == "f"
def test_add_offset_float(self):
- self.cf_coord_var.add_offset = 5.0
+ self.cf_coord_var.attributes["add_offset"] = 5.0
with self.deferred_load_patch():
build_and_add_auxiliary_coordinate(self.engine, self.cf_coord_var)
@@ -244,51 +268,44 @@ def _setup(self, mocker):
# Create dummy pyke engine.
self.engine = mocker.Mock(
cube=mocker.Mock(),
- cf_var=mocker.Mock(dimensions=("foo", "bar"), dtype=np.float32),
+ cf_var=mocker.Mock(
+ dimensions=("foo", "bar"), shape=(6, 2), dtype=np.float32
+ ),
filename="DUMMY",
cube_parts=dict(coordinates=[]),
)
points = np.arange(6)
units = "days since 1970-01-01"
- self.cf_coord_var = mocker.Mock(
- spec=CFVariable,
- dimensions=("foo",),
- scale_factor=1,
- add_offset=0,
- cf_name="wibble",
- cf_data=mocker.MagicMock(chunking=mocker.Mock(return_value=None), spec=[]),
- filename=self.engine.filename,
+ self.cf_coord_var = CFVariableStandIn(
standard_name=None,
long_name="wibble",
units=units,
calendar=None,
- shape=points.shape,
- size=np.prod(points.shape),
- dtype=points.dtype,
- __getitem__=lambda self, key: points[key],
)
+ self.cf_coord_var.dimensions = ("foo",)
+ self.cf_coord_var.scale_factor = 1
+ self.cf_coord_var.add_offset = 0
+ self.cf_coord_var.cf_name = "wibble"
+ self.cf_coord_var.cf_data = RealArrayCfData(points)
+ self.cf_coord_var.filename = self.engine.filename
+ self.cf_coord_var.shape = points.shape
+ self.cf_coord_var.size = np.prod(points.shape)
+ self.cf_coord_var.dtype = points.dtype
bounds = np.arange(12).reshape(6, 2)
- cf_data = mocker.MagicMock(chunking=mocker.Mock(return_value=None))
- # we want to mock the absence of flag attributes to helpers.get_attr_units
- # see https://docs.python.org/3/library/unittest.mock.html#deleting-attributes
- del cf_data.flag_values
- del cf_data.flag_masks
- del cf_data.flag_meanings
- self.cf_bounds_var = mocker.Mock(
- spec=CFVariable,
- dimensions=("x", "nv"),
- scale_factor=1,
- add_offset=0,
- cf_name="wibble_bnds",
- cf_data=cf_data,
- units=units,
- shape=bounds.shape,
- size=np.prod(bounds.shape),
- dtype=bounds.dtype,
- __getitem__=lambda self, key: bounds[key],
- )
+ # No flag_values/flag_masks/flag_meanings: their absence from
+ # `.attributes` is what tells helpers.get_attr_units this is not a
+ # flag variable.
+ self.cf_bounds_var = CFVariableStandIn(units=units)
+ self.cf_bounds_var.dimensions = ("x", "nv")
+ self.cf_bounds_var.scale_factor = 1
+ self.cf_bounds_var.add_offset = 0
+ self.cf_bounds_var.cf_name = "wibble_bnds"
+ self.cf_bounds_var.cf_data = RealArrayCfData(bounds)
+ self.cf_bounds_var.shape = bounds.shape
+ self.cf_bounds_var.size = np.prod(bounds.shape)
+ self.cf_bounds_var.dtype = bounds.dtype
self.bounds = bounds
# Create patch for deferred loading that prevents attempted
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_cell_measure.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_cell_measure.py
index aea5061f1e..2252eaf7e4 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_cell_measure.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_cell_measure.py
@@ -11,15 +11,21 @@
from iris.cube import Cube
from iris.exceptions import CannotAddError
from iris.fileformats._nc_load_rules.helpers import build_and_add_cell_measure
-from iris.fileformats.cf import CFMeasureVariable
from iris.loading import LOAD_PROBLEMS
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ RealArrayCfData,
+)
@pytest.fixture
def mock_engine(mocker):
return mocker.Mock(
cube=mocker.Mock(),
- cf_var=mocker.Mock(dimensions=("foo", "bar")),
+ # A data variable has a shape as well as dimension names, because a
+ # cell measure is attached to the dimensions it matches on
+ # `(name, size)` - see `helpers._common_data_dims`.
+ cf_var=mocker.Mock(dimensions=("foo", "bar"), shape=(6, 2)),
filename="DUMMY",
cube_parts=dict(cell_measures=[]),
)
@@ -28,23 +34,17 @@ def mock_engine(mocker):
@pytest.fixture
def mock_cf_cm_var(monkeypatch, mock_engine, mocker):
data = np.arange(6)
- output = mocker.Mock(
- spec=CFMeasureVariable,
- dimensions=("foo",),
- scale_factor=1,
- add_offset=0,
- cf_name="wibble",
- cf_data=mocker.MagicMock(chunking=mocker.Mock(return_value=None), spec=[]),
- filename=mock_engine.filename,
- standard_name=None,
- long_name="wibble",
- units="m2",
- shape=data.shape,
- size=np.prod(data.shape),
- dtype=data.dtype,
- __getitem__=lambda self, key: data[key],
- cf_measure="area",
- )
+ output = CFVariableStandIn(standard_name=None, long_name="wibble", units="m2")
+ output.dimensions = ("foo",)
+ output.scale_factor = 1
+ output.add_offset = 0
+ output.cf_name = "wibble"
+ output.cf_data = RealArrayCfData(data)
+ output.filename = mock_engine.filename
+ output.shape = data.shape
+ output.size = np.prod(data.shape)
+ output.dtype = data.dtype
+ output.cf_measure = "area"
# Create patch for deferred loading that prevents attempted
# file access. This assumes that output is defined in the test case.
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_cell_methods.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_cell_methods.py
index 31c063a90d..3aa171b880 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_cell_methods.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_cell_methods.py
@@ -9,18 +9,16 @@
from iris.coords import CellMethod
from iris.cube import Cube
from iris.fileformats._nc_load_rules import helpers
-from iris.fileformats.cf import CFDataVariable
from iris.loading import LOAD_PROBLEMS
+from iris.tests.unit.fileformats.nc_load_rules.helpers import CFVariableStandIn
@pytest.fixture
def mock_cf_data_var(mocker):
- return mocker.Mock(
- spec=CFDataVariable,
- cell_methods="time: mean",
- cf_name="wibble",
- filename="DUMMY",
- )
+ stand_in = CFVariableStandIn(cell_methods="time: mean")
+ stand_in.cf_name = "wibble"
+ stand_in.filename = "DUMMY"
+ return stand_in
@pytest.fixture
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_dimension_coordinate.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_dimension_coordinate.py
index 7e3155ad68..a032a78b5b 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_dimension_coordinate.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_dimension_coordinate.py
@@ -15,7 +15,11 @@
from iris.exceptions import CannotAddError
from iris.fileformats._nc_load_rules.helpers import build_and_add_dimension_coordinate
from iris.loading import LOAD_PROBLEMS
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+ RealArrayCfData,
+)
class RulesTestMixin(MockerMixin):
@@ -24,7 +28,7 @@ def _mixin_setup(self, mocker):
# Create dummy pyke engine.
self.engine = mocker.Mock(
cube=mocker.Mock(),
- cf_var=mocker.Mock(dimensions=("foo", "bar"), dtype=np.int32),
+ cf_var=mocker.Mock(dimensions=("foo", "bar"), shape=(6, 2), dtype=np.int32),
filename="DUMMY",
cube_parts=dict(coordinates=[]),
)
@@ -55,26 +59,28 @@ def get_cf_bounds_var(coord_var):
new=get_cf_bounds_var,
)
+ def _set_data_var_length(self, length):
+ """Keep the mock data variable's "foo" dimension the coordinate's length.
+
+ A coordinate is attached to the dimensions it matches on
+ `(name, size)` - see `helpers._common_data_dims` - so a data variable
+ naming "foo" at some other length shares nothing with the coordinate
+ and the coordinate reaches the cube with no dimensions at all.
+ """
+ self.engine.cf_var.shape = (length, 2)
+
def _make_bounds_var(self, bounds, dimensions, units):
bounds = np.array(bounds)
- cf_data = self.mocker.Mock(spec=[])
- # we want to mock the absence of flag attributes to helpers.get_attr_units
- # see https://docs.python.org/3/library/unittest.mock.html#deleting-attributes
- del cf_data.flag_values
- del cf_data.flag_masks
- del cf_data.flag_meanings
- result = self.mocker.Mock(
- dimensions=dimensions,
- cf_name="wibble_bnds",
- cf_data=cf_data,
- units=units,
- calendar=None,
- shape=bounds.shape,
- size=np.prod(bounds.shape),
- dtype=bounds.dtype,
- __getitem__=lambda self, key: bounds[key],
- )
- delattr(result, "_data_array")
+ # No flag_values/flag_masks/flag_meanings: their absence from
+ # `.attributes` is what tells helpers.get_attr_units this is not a
+ # flag variable.
+ result = CFVariableStandIn(units=units, calendar=None)
+ result.dimensions = dimensions
+ result.cf_name = "wibble_bnds"
+ result.cf_data = RealArrayCfData(bounds)
+ result.shape = bounds.shape
+ result.size = np.prod(bounds.shape)
+ result.dtype = bounds.dtype
return result
@@ -94,21 +100,20 @@ def _setup(self):
self.monkeypatch = pytest.MonkeyPatch()
def _set_cf_coord_var(self, points):
- self.cf_coord_var = self.mocker.Mock(
- dimensions=("foo",),
- cf_name="wibble",
- cf_data=self.mocker.Mock(spec=[]),
+ self.cf_coord_var = CFVariableStandIn(
standard_name=None,
long_name="wibble",
units="days since 1970-01-01",
calendar=None,
- shape=points.shape,
- size=np.prod(points.shape),
- dtype=points.dtype,
- __getitem__=lambda self, key: points[key],
- cf_attrs=lambda: [("foo", "a"), ("bar", "b")],
)
- delattr(self.cf_coord_var, "_data_array")
+ self.cf_coord_var.dimensions = ("foo",)
+ self.cf_coord_var.cf_name = "wibble"
+ self.cf_coord_var.cf_data = RealArrayCfData(points)
+ self.cf_coord_var.shape = points.shape
+ self.cf_coord_var.size = np.prod(points.shape)
+ self.cf_coord_var.dtype = points.dtype
+ self.cf_coord_var.filename = "DUMMY"
+ self._set_data_var_length(points.shape[0])
def check_case_dim_coord_construction(self, climatology=False):
# Test a generic dimension coordinate, with or without
@@ -346,17 +351,15 @@ class TestBoundsVertexDim(RulesTestMixin):
def _setup(self, mocker):
# Create test coordinate cf variable.
points = np.arange(6)
- self.cf_coord_var = mocker.Mock(
- dimensions=("foo",),
- cf_name="wibble",
- standard_name=None,
- long_name="wibble",
- cf_data=mocker.Mock(spec=[]),
- units="km",
- shape=points.shape,
- dtype=points.dtype,
- __getitem__=lambda self, key: points[key],
+ self.cf_coord_var = CFVariableStandIn(
+ standard_name=None, long_name="wibble", units="km"
)
+ self.cf_coord_var.dimensions = ("foo",)
+ self.cf_coord_var.cf_name = "wibble"
+ self.cf_coord_var.cf_data = RealArrayCfData(points)
+ self.cf_coord_var.shape = points.shape
+ self.cf_coord_var.dtype = points.dtype
+ self._set_data_var_length(points.shape[0])
def test_slowest_varying_vertex_dim__normalise_bounds(self):
# Create the bounds cf variable.
@@ -442,17 +445,15 @@ def _setup(self):
def _make_vars(self, points, bounds=None, units="degrees"):
points = np.array(points)
- self.cf_coord_var = self.mocker.MagicMock(
- dimensions=("foo",),
- cf_name="wibble",
- standard_name=None,
- long_name="wibble",
- cf_data=self.mocker.Mock(spec=[]),
- units=units,
- shape=points.shape,
- dtype=points.dtype,
- __getitem__=lambda self, key: points[key],
+ self.cf_coord_var = CFVariableStandIn(
+ standard_name=None, long_name="wibble", units=units
)
+ self.cf_coord_var.dimensions = ("foo",)
+ self.cf_coord_var.cf_name = "wibble"
+ self.cf_coord_var.cf_data = RealArrayCfData(points)
+ self.cf_coord_var.shape = points.shape
+ self.cf_coord_var.dtype = points.dtype
+ self._set_data_var_length(points.shape[0])
if bounds:
bounds = np.array(bounds).reshape(self.cf_coord_var.shape + (2,))
dimensions = ("x", "nv")
@@ -535,17 +536,14 @@ def _make_vars(self, bounds):
# the cf var is (), rather than (1,).
points = np.array([0.0])
units = "degrees"
- self.cf_coord_var = self.mocker.Mock(
- dimensions=(),
- cf_name="wibble",
- standard_name=None,
- long_name="wibble",
- units=units,
- cf_data=self.mocker.Mock(spec=[]),
- shape=(),
- dtype=points.dtype,
- __getitem__=lambda self, key: points[key],
+ self.cf_coord_var = CFVariableStandIn(
+ standard_name=None, long_name="wibble", units=units
)
+ self.cf_coord_var.dimensions = ()
+ self.cf_coord_var.cf_name = "wibble"
+ self.cf_coord_var.cf_data = RealArrayCfData(points)
+ self.cf_coord_var.shape = ()
+ self.cf_coord_var.dtype = points.dtype
bounds = np.array(bounds)
dimensions = ("bnds",)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_names.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_names.py
index ba6a289a08..46417fdb45 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_names.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_names.py
@@ -10,6 +10,7 @@
from iris.cube import Cube
from iris.fileformats._nc_load_rules.helpers import build_and_add_names
from iris.loading import LOAD_PROBLEMS
+from iris.tests.unit.fileformats.nc_load_rules.helpers import CFVariableStandIn
@pytest.fixture
@@ -19,15 +20,16 @@ def mock_engine(mocker):
"comment": "Mocked test object",
}
cf_group = mocker.Mock(global_attributes=global_attributes)
- cf_var = mocker.MagicMock(
- cf_name="wibble",
+ cf_var = CFVariableStandIn(
standard_name=None,
long_name=None,
units="m",
- dtype=np.float64,
cell_methods=None,
- cf_group=cf_group,
)
+ cf_var.cf_name = "wibble"
+ cf_var.dtype = np.float64
+ cf_var.cf_group = cf_group
+ cf_var.filename = "foo.nc"
engine = mocker.Mock(cube=Cube([23]), cf_var=cf_var, filename="foo.nc")
return engine
@@ -46,8 +48,8 @@ def check_cube_names(self, inputs, expected):
# Expected - The expected cube attributes.
exp_standard_name, exp_long_name = expected
- self.engine.cf_var.standard_name = standard_name
- self.engine.cf_var.long_name = long_name
+ self.engine.cf_var.attributes["standard_name"] = standard_name
+ self.engine.cf_var.attributes["long_name"] = long_name
# engine = _make_engine(standard_name=standard_name, long_name=long_name)
build_and_add_names(self.engine)
self.cf_name = self.engine.cf_var.cf_name
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_units.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_units.py
index ef2931963f..d696cecb73 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_units.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_and_add_units.py
@@ -9,20 +9,16 @@
from iris.cube import Cube
from iris.fileformats._nc_load_rules import helpers
-from iris.fileformats.cf import CFDataVariable
from iris.loading import LOAD_PROBLEMS
+from iris.tests.unit.fileformats.nc_load_rules.helpers import real_cf_data_variable
@pytest.fixture
-def mock_cf_data_var(mocker):
- return mocker.Mock(
- spec=CFDataVariable,
- units="kelvin",
- cf_name="wibble",
- filename="DUMMY",
- dtype=float,
- cf_data=mocker.Mock(spec=[]),
- )
+def mock_cf_data_var():
+ # A real CFDataVariable, not a CFVariableStandIn: build_and_add_units calls
+ # get_attr_units with capture_invalid=True, whose first act is to assert
+ # isinstance(cf_var, cf.CFDataVariable).
+ return real_cf_data_variable(units="kelvin")
@pytest.fixture
@@ -41,7 +37,7 @@ def test_construction(mock_engine):
def test_invalid_units(mock_engine, mock_cf_data_var):
- mock_cf_data_var.units = "not_built"
+ mock_cf_data_var.attributes["units"] = "not_built"
helpers.build_and_add_units(mock_engine)
assert mock_engine.cube.attributes["invalid_units"] == "not_built"
load_problem = LOAD_PROBLEMS.problems[-1]
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_geostationary_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_geostationary_coordinate_system.py
index d810152196..593ae20208 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_geostationary_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_geostationary_coordinate_system.py
@@ -12,7 +12,10 @@
from iris.fileformats._nc_load_rules.helpers import (
build_geostationary_coordinate_system,
)
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class TestBuildGeostationaryCoordinateSystem(MockerMixin):
@@ -49,7 +52,7 @@ def _test(self, inverse_flattening=False, replace_props=None, remove_props=None)
cf_grid_var_kwargs = non_ellipsoid_kwargs.copy()
cf_grid_var_kwargs.update(ellipsoid_kwargs)
- cf_grid_var = self.mocker.Mock(spec=[], **cf_grid_var_kwargs)
+ cf_grid_var = CFVariableStandIn(**cf_grid_var_kwargs)
cs = build_geostationary_coordinate_system(None, cf_grid_var)
ellipsoid = iris.coord_systems.GeogCS(**ellipsoid_kwargs)
expected = Geostationary(ellipsoid=ellipsoid, **non_ellipsoid_kwargs)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_lambert_azimuthal_equal_area_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_lambert_azimuthal_equal_area_coordinate_system.py
index 0d4af82889..479fbcc8a9 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_lambert_azimuthal_equal_area_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_lambert_azimuthal_equal_area_coordinate_system.py
@@ -12,7 +12,10 @@
from iris.fileformats._nc_load_rules.helpers import (
build_lambert_azimuthal_equal_area_coordinate_system,
)
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class TestBuildLambertAzimuthalEqualAreaCoordinateSystem(MockerMixin):
@@ -49,7 +52,7 @@ def _test(self, inverse_flattening=False, no_optionals=False):
gridvar_props["semi_minor_axis"] = 6356256.909
expected_ellipsoid = iris.coord_systems.GeogCS(6377563.396, 6356256.909)
- cf_grid_var = self.mocker.Mock(spec=[], **gridvar_props)
+ cf_grid_var = CFVariableStandIn(**gridvar_props)
cs = build_lambert_azimuthal_equal_area_coordinate_system(None, cf_grid_var)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_lambert_conformal_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_lambert_conformal_coordinate_system.py
index 9d91c2a3f7..f5255aa18d 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_lambert_conformal_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_lambert_conformal_coordinate_system.py
@@ -12,7 +12,10 @@
from iris.fileformats._nc_load_rules.helpers import (
build_lambert_conformal_coordinate_system,
)
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class TestBuildLambertConformalCoordinateSystem(MockerMixin):
@@ -52,7 +55,7 @@ def _test(self, inverse_flattening=False, no_optionals=False):
gridvar_props["semi_minor_axis"] = 6356256.909
expected_ellipsoid = iris.coord_systems.GeogCS(6377563.396, 6356256.909)
- cf_grid_var = self.mocker.Mock(spec=[], **gridvar_props)
+ cf_grid_var = CFVariableStandIn(**gridvar_props)
cs = build_lambert_conformal_coordinate_system(None, cf_grid_var)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_mercator_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_mercator_coordinate_system.py
index 3752337ea9..5bdaa029a1 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_mercator_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_mercator_coordinate_system.py
@@ -10,12 +10,12 @@
import iris
from iris.coord_systems import Mercator
from iris.fileformats._nc_load_rules.helpers import build_mercator_coordinate_system
+from iris.tests.unit.fileformats.nc_load_rules.helpers import CFVariableStandIn
class TestBuildMercatorCoordinateSystem:
def test_valid(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
longitude_of_projection_origin=-90,
semi_major_axis=6377563.396,
semi_minor_axis=6356256.909,
@@ -34,8 +34,7 @@ def test_valid(self, mocker):
assert cs == expected
def test_inverse_flattening(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
longitude_of_projection_origin=-90,
semi_major_axis=6377563.396,
inverse_flattening=299.3249646,
@@ -55,8 +54,7 @@ def test_inverse_flattening(self, mocker):
assert cs == expected
def test_longitude_missing(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
semi_major_axis=6377563.396,
inverse_flattening=299.3249646,
standard_parallel=10,
@@ -74,8 +72,7 @@ def test_longitude_missing(self, mocker):
assert cs == expected
def test_standard_parallel_missing(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
longitude_of_projection_origin=-90,
semi_major_axis=6377563.396,
semi_minor_axis=6356256.909,
@@ -92,8 +89,7 @@ def test_standard_parallel_missing(self, mocker):
assert cs == expected
def test_scale_factor_at_projection_origin(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
longitude_of_projection_origin=-90,
semi_major_axis=6377563.396,
semi_minor_axis=6356256.909,
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_oblique_mercator_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_oblique_mercator_coordinate_system.py
index ed4395fffa..7ce9036162 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_oblique_mercator_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_oblique_mercator_coordinate_system.py
@@ -14,6 +14,7 @@
from iris.fileformats._nc_load_rules.helpers import (
build_oblique_mercator_coordinate_system,
)
+from iris.tests.unit.fileformats.nc_load_rules.helpers import CFVariableStandIn
class ParamTuple(NamedTuple):
@@ -148,7 +149,7 @@ def make_variant_inputs(self, request) -> None:
self.coord_system_args_expected = list(coord_system_kwargs_expected.values())
def test_attributes(self, mocker):
- cf_var_mock = mocker.Mock(spec=[], **self.nc_attributes)
+ cf_var_mock = CFVariableStandIn(**self.nc_attributes)
coord_system_mock = mocker.Mock(spec=self.expected_class)
setattr(coord_systems, self.expected_class.__name__, coord_system_mock)
@@ -163,6 +164,6 @@ def test_deprecation(mocker):
longitude_of_projection_origin=0.0,
scale_factor_at_projection_origin=1.0,
)
- cf_var_mock = mocker.Mock(spec=[], **nc_attributes)
+ cf_var_mock = CFVariableStandIn(**nc_attributes)
with pytest.warns(IrisDeprecation, match="azimuth_of_central_line = 90"):
_ = build_oblique_mercator_coordinate_system(None, cf_var_mock)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_polar_stereographic_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_polar_stereographic_coordinate_system.py
index 3e9396cca4..bdba212bfd 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_polar_stereographic_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_polar_stereographic_coordinate_system.py
@@ -12,12 +12,12 @@
from iris.fileformats._nc_load_rules.helpers import (
build_polar_stereographic_coordinate_system,
)
+from iris.tests.unit.fileformats.nc_load_rules.helpers import CFVariableStandIn
class TestBuildPolarStereographicCoordinateSystem:
def test_valid_north(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=90,
scale_factor_at_projection_origin=1,
@@ -40,8 +40,7 @@ def test_valid_north(self, mocker):
assert cs == expected
def test_valid_south(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=-90,
scale_factor_at_projection_origin=1,
@@ -64,8 +63,7 @@ def test_valid_south(self, mocker):
assert cs == expected
def test_valid_with_standard_parallel(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=90,
standard_parallel=30,
@@ -86,8 +84,7 @@ def test_valid_with_standard_parallel(self, mocker):
assert cs == expected
def test_valid_with_false_easting_northing(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=90,
scale_factor_at_projection_origin=1,
@@ -114,8 +111,7 @@ def test_valid_with_false_easting_northing(self, mocker):
assert cs == expected
def test_valid_nonzero_veritcal_lon(self, mocker):
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=30,
latitude_of_projection_origin=90,
scale_factor_at_projection_origin=1,
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_raw_cube.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_raw_cube.py
index 88b3909592..60bf8f05ae 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_raw_cube.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_raw_cube.py
@@ -15,10 +15,17 @@
def _make_array_and_cf_data(mocker, dim_lens: dict[str, int]):
shape = list(dim_lens.values())
- cf_data = mocker.MagicMock(_FillValue=None, spec=[])
- cf_data.chunking = mocker.MagicMock(return_value=shape)
data = np.arange(np.prod(shape), dtype=float)
data = data.reshape(shape)
+ # This data is small enough that read_data never reaches .chunking or
+ # .variable - it simply returns the real array, as it would for any
+ # small variable.
+ cf_data = mocker.MagicMock(
+ spec=["is_emulated", "is_variable_length", "read_data"],
+ is_emulated=False,
+ is_variable_length=False,
+ )
+ cf_data.read_data.return_value = data
return data, cf_data
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_stereographic_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_stereographic_coordinate_system.py
index 481d4441f8..d0524daa56 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_stereographic_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_stereographic_coordinate_system.py
@@ -12,7 +12,10 @@
from iris.fileformats._nc_load_rules.helpers import (
build_stereographic_coordinate_system,
)
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class TestBuildStereographicCoordinateSystem(MockerMixin):
@@ -44,7 +47,7 @@ def _test(self, inverse_flattening=False, no_offsets=False):
test_easting = 0
test_northing = 0
- cf_grid_var = self.mocker.Mock(spec=[], **gridvar_props)
+ cf_grid_var = CFVariableStandIn(**gridvar_props)
cs = build_stereographic_coordinate_system(None, cf_grid_var)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_transverse_mercator_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_transverse_mercator_coordinate_system.py
index f63402dcc2..05cbe558b7 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_transverse_mercator_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_transverse_mercator_coordinate_system.py
@@ -12,7 +12,10 @@
from iris.fileformats._nc_load_rules.helpers import (
build_transverse_mercator_coordinate_system,
)
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class TestBuildTransverseMercatorCoordinateSystem(MockerMixin):
@@ -46,7 +49,7 @@ def _test(self, inverse_flattening=False, no_options=False):
test_northing = 0
test_scale_factor = 1.0
- cf_grid_var = self.mocker.Mock(spec=[], **gridvar_props)
+ cf_grid_var = CFVariableStandIn(**gridvar_props)
cs = build_transverse_mercator_coordinate_system(None, cf_grid_var)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_verticalp_coordinate_system.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_verticalp_coordinate_system.py
index 932e1d085d..69d201245f 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_verticalp_coordinate_system.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_build_verticalp_coordinate_system.py
@@ -12,7 +12,10 @@
from iris.fileformats._nc_load_rules.helpers import (
build_vertical_perspective_coordinate_system,
)
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class TestBuildVerticalPerspectiveCoordinateSystem(MockerMixin):
@@ -23,7 +26,6 @@ def _test(self, inverse_flattening=False, no_offsets=False):
test_easting = 100.0
test_northing = 200.0
cf_grid_var_kwargs = {
- "spec": [],
"latitude_of_projection_origin": 1.0,
"longitude_of_projection_origin": 2.0,
"perspective_point_height": 2000000.0,
@@ -45,7 +47,7 @@ def _test(self, inverse_flattening=False, no_offsets=False):
test_easting = 0
test_northing = 0
- cf_grid_var = self.mocker.Mock(**cf_grid_var_kwargs)
+ cf_grid_var = CFVariableStandIn(**cf_grid_var_kwargs)
ellipsoid = iris.coord_systems.GeogCS(**ellipsoid_kwargs)
cs = build_vertical_perspective_coordinate_system(None, cf_grid_var)
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_attr_units.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_attr_units.py
index 50698e72f8..d25e93b5a5 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_attr_units.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_attr_units.py
@@ -7,14 +7,18 @@
"""
+import cf_units
import numpy as np
import pytest
from iris.fileformats._nc_load_rules.helpers import get_attr_units
-from iris.fileformats.cf import CFDataVariable
from iris.loading import LOAD_PROBLEMS
from iris.tests import _shared_utils
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+ real_cf_data_variable,
+)
from iris.warnings import IrisCfLoadWarning
@@ -25,18 +29,20 @@ def _make_cf_var(self, global_attributes=None):
cf_group = self.mocker.Mock(global_attributes=global_attributes)
- cf_var = self.mocker.MagicMock(
- spec=CFDataVariable,
- cf_name="sound_frequency",
- cf_data=self.mocker.Mock(spec=[]),
- filename="DUMMY",
+ # A real CFDataVariable, not a CFVariableStandIn, because test_capture
+ # below passes capture_invalid=True - and that branch of
+ # get_attr_units opens by asserting isinstance(cf_var,
+ # cf.CFDataVariable). The other tests here share it for want of a
+ # reason to differ.
+ cf_var = real_cf_data_variable(
+ name="sound_frequency",
+ dtype=np.float64,
standard_name=None,
long_name=None,
units="\u266b",
- dtype=np.float64,
cell_methods=None,
- cf_group=cf_group,
)
+ cf_var.cf_group = cf_group
return cf_var
def test_unicode_character(self):
@@ -67,3 +73,22 @@ def test_capture(self):
load_problem = LOAD_PROBLEMS.problems[-1]
assert load_problem.loaded == {"units": "\u266b"}
+
+ def test_flag_values_untracked(self):
+ """Presence of flag_values must force NO_UNIT_STRING without recording a read.
+
+ This is the load-bearing assertion for the untracked probe in
+ `get_attr_units` (`name in cf_var.attributes.untracked`): if that
+ probe is ever simplified to `name in cf_var.attributes` (a tracked
+ read), `flag_values`/`flag_masks`/`flag_meanings` would stop being
+ copied onto loaded cubes by `_add_unused_attributes`, silently. The
+ second assertion below is the one that catches that regression - the
+ first alone would still pass.
+ """
+ cf_var = CFVariableStandIn(units="1", flag_values="1, 2, 3")
+ cf_var.cf_name = "flag_var"
+
+ attr_units = get_attr_units(cf_var, {})
+
+ assert attr_units == cf_units._NO_UNIT_STRING
+ assert "flag_values" not in cf_var.attributes.read
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_cf_bounds_var.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_cf_bounds_var.py
index 39a2e6caa7..44719ccdea 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_cf_bounds_var.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_cf_bounds_var.py
@@ -29,9 +29,7 @@ def _generic_test(self, test_climatological_bounds=False):
cf_coord_var.cf_group.bounds = cf_group_dict
test_attr = CF_ATTR_BOUNDS
- for attr in (CF_ATTR_BOUNDS, CF_ATTR_CLIMATOLOGY):
- attr_val = "TEST" if attr == test_attr else None
- setattr(cf_coord_var, attr, attr_val)
+ cf_coord_var.attributes = {test_attr: "TEST"}
bounds_var, climatological = get_cf_bounds_var(cf_coord_var)
assert bounds_var is self.mocker.sentinel.bounds_var
@@ -42,3 +40,17 @@ def test_bounds_normal(self):
def test_bounds_climatological(self):
self._generic_test(test_climatological_bounds=True)
+
+ def test_bounds_wins_over_climatology(self):
+ cf_coord_var = self.mocker.MagicMock()
+ cf_coord_var.cf_group.bounds = {"B": self.mocker.sentinel.bounds_var}
+ cf_coord_var.cf_group.climatology = {"C": self.mocker.sentinel.clim_var}
+ cf_coord_var.attributes = {
+ CF_ATTR_BOUNDS: "B",
+ CF_ATTR_CLIMATOLOGY: "C",
+ }
+
+ bounds_var, climatological = get_cf_bounds_var(cf_coord_var)
+
+ assert bounds_var is self.mocker.sentinel.bounds_var
+ assert climatological is False
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_names.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_names.py
index d54aec0692..0168288eeb 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_names.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_get_names.py
@@ -10,7 +10,10 @@
import numpy as np
from iris.fileformats._nc_load_rules.helpers import get_names
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class TestGetNames(MockerMixin):
@@ -28,15 +31,15 @@ class TestGetNames(MockerMixin):
"""
def _make_cf_var(self, standard_name, long_name, cf_name):
- cf_var = self.mocker.Mock(
- cf_name=cf_name,
+ cf_var = CFVariableStandIn(
standard_name=standard_name,
long_name=long_name,
units="degrees",
- dtype=np.float64,
cell_methods=None,
- cf_group=self.mocker.Mock(global_attributes={}),
)
+ cf_var.cf_name = cf_name
+ cf_var.dtype = np.float64
+ cf_var.cf_group = self.mocker.Mock(global_attributes={})
return cf_var
def check_names(self, inputs, expected):
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_has_supported_mercator_parameters.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_has_supported_mercator_parameters.py
index 8ac08c330c..c0941ad8ca 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_has_supported_mercator_parameters.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_has_supported_mercator_parameters.py
@@ -11,7 +11,10 @@
import warnings
from iris.fileformats._nc_load_rules.helpers import has_supported_mercator_parameters
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class _EngineMixin(MockerMixin):
@@ -24,8 +27,7 @@ def engine(self, cf_grid_var, cf_name):
class TestHasSupportedMercatorParameters(_EngineMixin):
def test_valid_base(self, mocker):
cf_name = "mercator"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
longitude_of_projection_origin=-90,
false_easting=0,
false_northing=0,
@@ -41,8 +43,7 @@ def test_valid_base(self, mocker):
def test_valid_false_easting_northing(self, mocker):
cf_name = "mercator"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
longitude_of_projection_origin=-90,
false_easting=15,
false_northing=10,
@@ -58,8 +59,7 @@ def test_valid_false_easting_northing(self, mocker):
def test_valid_standard_parallel(self, mocker):
cf_name = "mercator"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
longitude_of_projection_origin=-90,
false_easting=0,
false_northing=0,
@@ -75,8 +75,7 @@ def test_valid_standard_parallel(self, mocker):
def test_valid_scale_factor(self, mocker):
cf_name = "mercator"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
longitude_of_projection_origin=0,
false_easting=0,
false_northing=0,
@@ -94,8 +93,7 @@ def test_invalid_scale_factor_and_standard_parallel(self, mocker):
# Scale factor and standard parallel cannot both be specified for
# Mercator projections
cf_name = "mercator"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
longitude_of_projection_origin=0,
false_easting=0,
false_northing=0,
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_has_supported_polar_stereographic_parameters.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_has_supported_polar_stereographic_parameters.py
index 2bfc801af2..ab1bd6b671 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_has_supported_polar_stereographic_parameters.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_has_supported_polar_stereographic_parameters.py
@@ -13,7 +13,10 @@
from iris.fileformats._nc_load_rules.helpers import (
has_supported_polar_stereographic_parameters,
)
-from iris.tests.unit.fileformats.nc_load_rules.helpers import MockerMixin
+from iris.tests.unit.fileformats.nc_load_rules.helpers import (
+ CFVariableStandIn,
+ MockerMixin,
+)
class _EngineMixin(MockerMixin):
@@ -26,8 +29,7 @@ def engine(self, cf_grid_var, cf_name):
class TestHasSupportedPolarStereographicParameters(_EngineMixin):
def test_valid_base_north(self, mocker):
cf_name = "polar_stereographic"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=90,
false_easting=0,
@@ -44,8 +46,7 @@ def test_valid_base_north(self, mocker):
def test_valid_base_south(self, mocker):
cf_name = "polar_stereographic"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=-90,
false_easting=0,
@@ -62,8 +63,7 @@ def test_valid_base_south(self, mocker):
def test_valid_straight_vertical_longitude(self, mocker):
cf_name = "polar_stereographic"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=30,
latitude_of_projection_origin=90,
false_easting=0,
@@ -80,8 +80,7 @@ def test_valid_straight_vertical_longitude(self, mocker):
def test_valid_false_easting_northing(self, mocker):
cf_name = "polar_stereographic"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=90,
false_easting=15,
@@ -98,8 +97,7 @@ def test_valid_false_easting_northing(self, mocker):
def test_valid_standard_parallel(self, mocker):
cf_name = "polar_stereographic"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=90,
false_easting=0,
@@ -116,8 +114,7 @@ def test_valid_standard_parallel(self, mocker):
def test_valid_scale_factor(self, mocker):
cf_name = "polar_stereographic"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=90,
false_easting=0,
@@ -136,8 +133,7 @@ def test_invalid_scale_factor_and_standard_parallel(self, mocker):
# Scale factor and standard parallel cannot both be specified for
# Polar Stereographic projections
cf_name = "polar_stereographic"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=90,
false_easting=0,
@@ -165,8 +161,7 @@ def test_absent_scale_factor_and_standard_parallel(self, mocker):
# Scale factor and standard parallel cannot both be specified for
# Polar Stereographic projections
cf_name = "polar_stereographic"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=90,
false_easting=0,
@@ -192,8 +187,7 @@ def test_invalid_latitude_of_projection_origin(self, mocker):
# Scale factor and standard parallel cannot both be specified for
# Polar Stereographic projections
cf_name = "polar_stereographic"
- cf_grid_var = mocker.Mock(
- spec=[],
+ cf_grid_var = CFVariableStandIn(
straight_vertical_longitude_from_pole=0,
latitude_of_projection_origin=45,
false_easting=0,
diff --git a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_is_lat_lon.py b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_is_lat_lon.py
index b4a4b0d2d8..5f05621d29 100644
--- a/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_is_lat_lon.py
+++ b/lib/iris/tests/unit/fileformats/nc_load_rules/helpers/test_is_lat_lon.py
@@ -5,10 +5,10 @@
"""Test function :func:`iris.fileformats._nc_load_rules.helpers._is_lat_lon`."""
from iris.fileformats._nc_load_rules import helpers
-from iris.fileformats.cf import CFCoordinateVariable
+from iris.tests.unit.fileformats.nc_load_rules.helpers import CFVariableStandIn
-def test_non_string_units(mocker):
- cf_var = mocker.MagicMock(spec=CFCoordinateVariable, units=1.0)
+def test_non_string_units():
+ cf_var = CFVariableStandIn(units=1.0)
is_lat_lon = helpers._is_lat_lon(cf_var, [], "latitude", "grid_latitude", "y", [])
assert is_lat_lon is False
diff --git a/lib/iris/tests/unit/fileformats/netcdf/dataset/__init__.py b/lib/iris/tests/unit/fileformats/netcdf/dataset/__init__.py
new file mode 100644
index 0000000000..938797a364
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/dataset/__init__.py
@@ -0,0 +1,5 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.netcdf._dataset`."""
diff --git a/lib/iris/tests/unit/fileformats/netcdf/dataset/conftest.py b/lib/iris/tests/unit/fileformats/netcdf/dataset/conftest.py
new file mode 100644
index 0000000000..cb4beec456
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/dataset/conftest.py
@@ -0,0 +1,77 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""A real netCDF4 file to test the netCDF CFDataset implementation against.
+
+Real, rather than mocked, because the members being implemented are precisely
+the ones whose netCDF4 behaviour is easy to misremember - what ``chunking()``
+returns for a contiguous variable, what ``dimensions`` is for char data, what
+``ncattrs()`` includes once ``fill_value=`` has been passed.
+"""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.netcdf import _thread_safe_nc
+
+#: How the sample file's dimensions are created. None means unlimited.
+SAMPLE_DIMENSION_SIZES = {"time": 3, "lat": 4, "nchars": 8, "record": None}
+
+#: How they should then read back. An unlimited dimension reports the number
+#: of records actually written, which here is none.
+SAMPLE_DIMENSIONS = {"time": 3, "lat": 4, "nchars": 8, "record": 0}
+
+#: The sample file's global attributes.
+SAMPLE_GLOBALS = {"Conventions": "CF-1.7", "title": "sample"}
+
+#: The sample file's air_temperature payload.
+SAMPLE_AIR = np.arange(12, dtype="f4").reshape(3, 4)
+
+#: The sample file's label payload, as the strings it encodes.
+SAMPLE_LABELS = ["alpha", "beta", "gamma"]
+
+
+@pytest.fixture(scope="session")
+def sample_path(tmp_path_factory):
+ """Write, once per session, a small netCDF4 file and return its path."""
+ path = tmp_path_factory.mktemp("cf_dataset") / "sample.nc"
+ dataset = _thread_safe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ try:
+ for name, size in SAMPLE_DIMENSION_SIZES.items():
+ dataset.createDimension(name, size)
+ for name, value in SAMPLE_GLOBALS.items():
+ dataset.setncattr(name, value)
+
+ air = dataset.createVariable(
+ "air_temperature",
+ "f4",
+ ("time", "lat"),
+ fill_value=-999.0,
+ zlib=True,
+ chunksizes=(1, 4),
+ )
+ air.setncattr("units", "K")
+ air.setncattr("standard_name", "air_temperature")
+ air.setncattr("coordinates", "height")
+ air[:] = SAMPLE_AIR
+
+ # Scalar, and deliberately unchunked: chunking() answers differently.
+ height = dataset.createVariable("height", "f4", ())
+ height.setncattr("units", "m")
+ height[:] = 1.5
+
+ # Char data: the one case where EncodedVariable changes shape, dtype
+ # and dimensions out from under the wrapper.
+ label = dataset.createVariable("label", "S1", ("time", "nchars"))
+ label[:] = np.array([list(f"{text:<8}") for text in SAMPLE_LABELS], dtype="S1")
+ finally:
+ dataset.close()
+
+ return path
+
+
+@pytest.fixture
+def sample_location(sample_path):
+ """The sample file's path, as the string a CFDataset reports."""
+ return str(sample_path)
diff --git a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py
new file mode 100644
index 0000000000..81276a10cd
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py
@@ -0,0 +1,322 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.netcdf._dataset.NetCDFDataset`."""
+
+import warnings
+
+import numpy as np
+import pytest
+
+from iris.fileformats.cf.dataset import CFDataset
+from iris.fileformats.netcdf import _bytecoding_datasets, _dataset, _thread_safe_nc
+from iris.fileformats.netcdf._dataset import NetCDFDataset
+from iris.warnings import IrisLoadWarning
+
+from .conftest import SAMPLE_DIMENSIONS, SAMPLE_GLOBALS
+
+
+@pytest.fixture
+def reader(sample_path):
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ yield dataset
+
+
+class TestOpening:
+ def test_is_a_cf_dataset(self, reader):
+ assert isinstance(reader, CFDataset)
+
+ def test_location_is_the_string_path(self, reader, sample_location):
+ assert reader.location == sample_location
+
+ def test_default_mode_is_read(self, reader):
+ assert reader.mode == "r"
+
+ def test_starts_open(self, reader):
+ assert reader.closed is False
+
+ def test_close_is_idempotent(self, sample_path):
+ dataset = _dataset.NetCDFDataset(sample_path)
+ dataset.close()
+ assert dataset.closed is True
+ dataset.close()
+ assert dataset.closed is True
+
+ def test_context_manager_closes(self, sample_path):
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ assert dataset.closed is False
+ assert dataset.closed is True
+
+ def test_missing_file_raises(self, tmp_path):
+ with pytest.raises((FileNotFoundError, OSError)):
+ _dataset.NetCDFDataset(tmp_path / "nope.nc")
+
+ def test_decoding_setting_chooses_the_wrapper(self, sample_path):
+ with _bytecoding_datasets.DECODE_TO_STRINGS_ON_READ.context(True):
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ assert isinstance(dataset.dataset, _bytecoding_datasets.EncodedDataset)
+ with _bytecoding_datasets.DECODE_TO_STRINGS_ON_READ.context(False):
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ assert not isinstance(
+ dataset.dataset, _bytecoding_datasets.EncodedDataset
+ )
+ assert isinstance(dataset.dataset, _thread_safe_nc.DatasetWrapper)
+
+
+class TestContents:
+ def test_variables(self, reader, sample_location):
+ assert sorted(reader.variables) == ["air_temperature", "height", "label"]
+ air = reader.variables["air_temperature"]
+ assert isinstance(air, _dataset.NetCDFDatasetVariable)
+ assert air.name == "air_temperature"
+ assert air.location == sample_location
+
+ def test_variables_are_built_once(self, reader):
+ assert (
+ reader.variables["air_temperature"] is reader.variables["air_temperature"]
+ )
+
+ def test_variables_share_one_write_lock(self, reader):
+ # cf._dask_locks.get_worker_lock() returns a FRESH threading.Lock under
+ # the threaded scheduler, so a per-variable call would hand out locks
+ # that exclude nothing. Observed through the write handles, because
+ # the lock is only made when one is asked for - see
+ # test_materialising_variables_makes_no_write_lock.
+ air = reader.variables["air_temperature"]
+ height = reader.variables["height"]
+ assert air.write_handle().lock is height.write_handle().lock
+ assert air.write_handle().lock is reader.write_lock
+
+ def test_materialising_variables_makes_no_write_lock(self, reader):
+ assert sorted(reader.variables) == ["air_temperature", "height", "label"]
+ assert dict(reader.attributes) == SAMPLE_GLOBALS
+ assert reader.variables["air_temperature"].attributes is not None
+ assert reader._write_lock is None
+
+ def test_dimensions_are_names_and_lengths(self, reader):
+ assert dict(reader.dimensions) == SAMPLE_DIMENSIONS
+
+ def test_unlimited_dimension_reports_its_current_length(self, reader):
+ # Nothing was written along "record", so it is zero-length - which is
+ # what netCDF4 reports, and what saver.py's membership tests need.
+ assert reader.dimensions["record"] == 0
+ assert "record" in reader.dimensions
+
+ def test_global_attributes(self, reader):
+ assert dict(reader.attributes) == SAMPLE_GLOBALS
+
+ def test_global_attributes_are_built_once(self, reader):
+ assert reader.attributes is reader.attributes
+
+
+class _EmulatorWithoutIsopen:
+ """The least a borrowed emulator can be, and no ``isopen()``.
+
+ ``createVariable`` and ``close`` are what
+ :class:`~iris.fileformats.netcdf._thread_safe_nc.DatasetWrapper` checks
+ for before agreeing to wrap an object; ``filepath`` is what
+ ``from_existing`` reads. Nothing here is exercised - the members exist so
+ that the wrap succeeds.
+ """
+
+ def createVariable(self, *args, **kwargs):
+ raise NotImplementedError
+
+ def close(self):
+ raise NotImplementedError
+
+ def filepath(self):
+ return "<emulated>"
+
+
+class TestBorrowing:
+ def test_from_existing_wraps_an_open_dataset(self, sample_path, sample_location):
+ raw = _bytecoding_datasets.EncodedDataset(sample_path, mode="r")
+ try:
+ dataset = _dataset.NetCDFDataset.from_existing(raw)
+ assert dataset.dataset is raw
+ assert dataset.location == sample_location
+ assert sorted(dataset.variables) == [
+ "air_temperature",
+ "height",
+ "label",
+ ]
+ finally:
+ raw.close()
+
+ def test_from_existing_does_not_close_what_it_borrowed(self, sample_path):
+ raw = _bytecoding_datasets.EncodedDataset(sample_path, mode="r")
+ try:
+ dataset = _dataset.NetCDFDataset.from_existing(raw)
+ dataset.close()
+ assert dataset.closed is True
+ # Still usable: the borrower released nothing.
+ assert raw.isopen()
+ finally:
+ raw.close()
+
+ def test_from_existing_wraps_a_bare_netcdf4_dataset(self, sample_path):
+ # What ncdata hands iris.save / CFReader: an object with the netCDF4
+ # API but no thread-safe wrapper around it.
+ import netCDF4
+
+ raw = netCDF4.Dataset(sample_path, mode="r")
+ try:
+ dataset = _dataset.NetCDFDataset.from_existing(raw)
+ assert isinstance(dataset.dataset, _bytecoding_datasets.EncodedDataset)
+ assert dataset.dataset._contained_instance is raw
+ finally:
+ raw.close()
+
+ def test_a_borrowed_emulator_without_isopen_reads_as_open(self):
+ # NetCDFDataset.closed asks the backing object whether it is open,
+ # and an emulator need not implement isopen() - ncdata's does, so
+ # nothing else in the suite reaches this branch. Taken to be open:
+ # the borrower has no better answer, and close() still overrides it.
+ dataset = _dataset.NetCDFDataset.from_existing(_EmulatorWithoutIsopen())
+ assert dataset.closed is False
+ dataset.close()
+ assert dataset.closed is True
+
+ def test_from_existing_mode_is_stipulated_not_observed(self, sample_path):
+ # netCDF4 exposes no public attribute recording a dataset's open
+ # mode, so from_existing cannot read the mode a borrowed dataset was
+ # actually opened in - it *always* reports "r+", regardless of the
+ # mode below. Pins that stipulated value; it is not a description of
+ # `raw`'s real mode.
+ raw = _bytecoding_datasets.EncodedDataset(sample_path, mode="r")
+ try:
+ dataset = _dataset.NetCDFDataset.from_existing(raw)
+ assert dataset.mode == "r+"
+ finally:
+ raw.close()
+
+
+class TestAutoChartostring:
+ """Iris decodes byte data itself, so netCDF4 must not do it first.
+
+ ``CFReader`` used to call ``set_auto_chartostring(False)`` on every
+ dataset it opened. The dataset now does it, so no caller has to remember.
+ """
+
+ def test_turned_off_on_an_opened_dataset(self, sample_path, mocker):
+ spy = mocker.spy(_thread_safe_nc.DatasetWrapper, "set_auto_chartostring")
+ with _bytecoding_datasets.DECODE_TO_STRINGS_ON_READ.context(False):
+ with _dataset.NetCDFDataset(sample_path):
+ pass
+ spy.assert_called_once_with(mocker.ANY, False)
+
+ def test_turned_off_on_a_borrowed_dataset(self, sample_path, mocker):
+ # Spied on the class, not the instance: DatasetWrapper's __setattr__
+ # forwards instance-level patching to the contained netCDF4 object
+ # instead of shadowing the method, so mocker.spy(raw, ...) cannot
+ # observe the call.
+ raw = _thread_safe_nc.DatasetWrapper(sample_path, mode="r")
+ spy = mocker.spy(_thread_safe_nc.DatasetWrapper, "set_auto_chartostring")
+ try:
+ _dataset.NetCDFDataset.from_existing(raw)
+ spy.assert_called_once_with(raw, False)
+ finally:
+ raw.close()
+
+ def test_an_encoded_dataset_blocks_it_rather_than_forwarding(self, sample_path):
+ # EncodedDataset does its own decoding, so the call is inert there -
+ # which is why making it unconditionally is safe.
+ with _dataset.NetCDFDataset(sample_path) as dataset:
+ assert isinstance(dataset.dataset, _bytecoding_datasets.EncodedDataset)
+ with pytest.raises(TypeError, match="not supported"):
+ dataset.dataset.set_auto_chartostring(True)
+
+
+class TestLegacyFormatWarning:
+ def test_silent_by_default(self, tmp_path, sample_path):
+ legacy = _write_netcdf3(tmp_path)
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ with _dataset.NetCDFDataset(legacy):
+ pass
+
+ def test_warns_when_asked(self, tmp_path):
+ legacy = _write_netcdf3(tmp_path)
+ with pytest.warns(IrisLoadWarning, match="nccopy"):
+ with _dataset.NetCDFDataset(legacy, warn_legacy_format=True):
+ pass
+
+ def test_does_not_warn_for_netcdf4(self, sample_path):
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ with _dataset.NetCDFDataset(sample_path, warn_legacy_format=True):
+ pass
+
+ def test_warns_for_a_borrowed_dataset_when_asked(self, tmp_path):
+ legacy = _write_netcdf3(tmp_path)
+ raw = _thread_safe_nc.DatasetWrapper(legacy, mode="r")
+ try:
+ with pytest.warns(IrisLoadWarning, match="nccopy"):
+ _dataset.NetCDFDataset.from_existing(raw, warn_legacy_format=True)
+ finally:
+ raw.close()
+
+
+def _write_netcdf3(tmp_path):
+ path = tmp_path / "legacy.nc"
+ dataset = _thread_safe_nc.DatasetWrapper(path, mode="w", format="NETCDF3_CLASSIC")
+ dataset.createDimension("time", 2)
+ variable = dataset.createVariable("time", "f4", ("time",))
+ variable[:] = np.arange(2, dtype="f4")
+ dataset.close()
+ return path
+
+
+class TestFormat:
+ @pytest.mark.parametrize(
+ ("netcdf_format", "reported"),
+ [
+ ("NETCDF4", "NETCDF4"),
+ ("NETCDF4_CLASSIC", "NETCDF4_CLASSIC"),
+ ("NETCDF3_CLASSIC", "NETCDF3_CLASSIC"),
+ # netCDF4 renames this one, so anything matching on the reported
+ # name has to know both spellings: SciTools/iris#7337.
+ ("NETCDF3_64BIT", "NETCDF3_64BIT_OFFSET"),
+ ],
+ )
+ def test_format_is_what_netcdf4_reports(self, tmp_path, netcdf_format, reported):
+ path = tmp_path / "formatted.nc"
+ with NetCDFDataset(path, mode="w", netcdf_format=netcdf_format) as dataset:
+ assert dataset.format == reported
+
+
+class TestSupportsDtype:
+ @pytest.fixture
+ def dataset(self, tmp_path, request):
+ path = tmp_path / "typed.nc"
+ with NetCDFDataset(path, mode="w", netcdf_format=request.param) as dataset:
+ yield dataset
+
+ @pytest.mark.parametrize("dataset", ["NETCDF4"], indirect=True)
+ @pytest.mark.parametrize("dtype", ["i8", "u1", "u4", "f4", "f8", "i2", "i4"])
+ def test_netcdf4_holds_everything(self, dataset, dtype):
+ assert dataset.supports_dtype(np.dtype(dtype)) is True
+
+ @pytest.mark.parametrize(
+ "dataset",
+ ["NETCDF4_CLASSIC", "NETCDF3_CLASSIC", "NETCDF3_64BIT"],
+ indirect=True,
+ )
+ @pytest.mark.parametrize("dtype", ["i8", "u1", "u4"])
+ def test_the_restricted_formats_refuse_wide_and_unsigned_integers(
+ self, dataset, dtype
+ ):
+ # "NETCDF3_64BIT" is here because netCDF4 reports it under a second
+ # name, which the format list used to miss: SciTools/iris#7337.
+ assert dataset.supports_dtype(np.dtype(dtype)) is False
+
+ @pytest.mark.parametrize(
+ "dataset",
+ ["NETCDF4", "NETCDF4_CLASSIC", "NETCDF3_CLASSIC", "NETCDF3_64BIT"],
+ indirect=True,
+ )
+ @pytest.mark.parametrize("dtype", ["f4", "f8", "i2", "i4"])
+ def test_every_format_holds_the_dtypes_the_saver_writes(self, dataset, dtype):
+ assert dataset.supports_dtype(np.dtype(dtype)) is True
diff --git a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py
new file mode 100644
index 0000000000..131714e278
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py
@@ -0,0 +1,343 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.netcdf._dataset.NetCDFDatasetVariable`."""
+
+from collections.abc import MutableMapping
+import warnings
+
+import dask.array as da
+import numpy as np
+import pytest
+
+from iris._deprecation import IrisDeprecation
+from iris.fileformats.cf.dataset import CFDatasetVariable
+import iris.fileformats.cf.loader
+from iris.fileformats.netcdf import _bytecoding_datasets, _dataset, _thread_safe_nc
+
+from .conftest import SAMPLE_AIR, SAMPLE_LABELS
+
+
+@pytest.fixture
+def raw(sample_path):
+ dataset = _thread_safe_nc.DatasetWrapper(sample_path, mode="r")
+ yield dataset
+ dataset.close()
+
+
+@pytest.fixture
+def encoded(sample_path):
+ dataset = _bytecoding_datasets.EncodedDataset(sample_path, mode="r")
+ yield dataset
+ dataset.close()
+
+
+def wrap(dataset, name, location, write_lock_factory=None):
+ return _dataset.NetCDFDatasetVariable(
+ dataset.variables[name],
+ location,
+ dataset.file_format,
+ write_lock_factory=write_lock_factory,
+ )
+
+
+@pytest.fixture
+def air(raw, sample_location):
+ return wrap(raw, "air_temperature", sample_location)
+
+
+@pytest.fixture
+def height(raw, sample_location):
+ return wrap(raw, "height", sample_location)
+
+
+class TestStorageProperties:
+ def test_is_a_cf_dataset_variable(self, air):
+ assert isinstance(air, CFDatasetVariable)
+
+ def test_name(self, air):
+ assert air.name == "air_temperature"
+
+ def test_location(self, air, sample_location):
+ assert air.location == sample_location
+
+ def test_dimensions_are_a_tuple(self, air):
+ # netCDF4 answers with a tuple already, but the contract says tuple
+ # and CFVariable.spans does set arithmetic on it, so pin it.
+ assert air.dimensions == ("time", "lat")
+ assert isinstance(air.dimensions, tuple)
+
+ def test_shape_dtype_size_ndim(self, air):
+ assert air.shape == (3, 4)
+ assert air.dtype == np.dtype("f4")
+ assert air.size == 12
+ assert air.ndim == 2
+
+ def test_len(self, air):
+ assert len(air) == 3
+
+ def test_scalar_has_no_len(self, height):
+ assert height.shape == ()
+ with pytest.raises(TypeError, match="unsized"):
+ len(height)
+
+ def test_fill_value(self, air):
+ assert air.fill_value == -999.0
+
+ def test_no_fill_value_is_none(self, height):
+ assert height.fill_value is None
+
+ def test_chunking_is_a_tuple(self, air):
+ # netCDF4 answers with a list; the contract says a shape.
+ assert air.chunking == (1, 4)
+
+ def test_contiguous_chunking_is_none(self, height):
+ # netCDF4 answers "contiguous" here, and loader.py already treats
+ # that and None identically, so both collapse to None.
+ assert height.chunking is None
+
+
+class TestData:
+ def test_getitem(self, air):
+ np.testing.assert_array_equal(air[:], SAMPLE_AIR)
+
+ def test_getitem_indexed(self, air):
+ np.testing.assert_array_equal(air[0], SAMPLE_AIR[0])
+
+
+class TestAttributes:
+ def test_is_a_plain_mutable_mapping(self, air):
+ # Tracking belongs to CFVariable, not here: a CFDatasetVariable can
+ # have two CFVariables promoted over it.
+ assert isinstance(air.attributes, MutableMapping)
+ assert not hasattr(air.attributes, "read")
+
+ def test_contents(self, air):
+ assert dict(air.attributes) == {
+ "_FillValue": -999.0,
+ "units": "K",
+ "standard_name": "air_temperature",
+ "coordinates": "height",
+ }
+
+ def test_is_built_once(self, air):
+ assert air.attributes is air.attributes
+
+ def test_values_are_read_once(self, mocker, sample_location):
+ # Pins the contract _NetCDFAttributes's own docstring states: values
+ # are read once, at construction, because the interface promises a
+ # mapping whose keys/items cost nothing. test_is_built_once above
+ # only pins that the mapping object is cached, not that its values
+ # are materialised rather than fetched again on every read - call
+ # counts are the only way to see that.
+ variable = mocker.Mock()
+ variable.ncattrs.return_value = ["units"]
+ variable.getncattr.return_value = "K"
+ wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4")
+ _ = wrapped.attributes["units"]
+ _ = wrapped.attributes["units"]
+ assert variable.ncattrs.call_count == 1
+ assert variable.getncattr.call_count == 1
+
+ def test_missing_key_raises_key_error(self, air):
+ with pytest.raises(KeyError, match="nonesuch"):
+ air.attributes["nonesuch"]
+
+ def test_unreadable_attribute_becomes_empty_string(self, mocker, sample_location):
+ # ncattrs() can list a name that getncattr then refuses. _getncattr in
+ # cf/_reader.py tolerated exactly this with a "" default, and that
+ # tolerance has to survive the move.
+ #
+ # A real VariableWrapper cannot be mocked this way: its __setattr__
+ # forwards every set to the contained netCDF4 object, so patching
+ # one of its methods would write a same-named attribute into the
+ # file instead. A bare stand-in with the two methods
+ # _NetCDFAttributes actually calls is enough.
+ variable = mocker.Mock()
+ variable.ncattrs.return_value = ["units", "broken"]
+ variable.getncattr.side_effect = lambda name: (
+ "m" if name == "units" else _raise(AttributeError(name))
+ )
+ wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4")
+ assert wrapped.attributes["broken"] == ""
+
+
+def _raise(exception):
+ raise exception
+
+
+class _EmulatedVariable:
+ """A stand-in for ncdata's own variable object.
+
+ Never a real, file-backed
+ :class:`~iris.fileformats.netcdf._thread_safe_nc.VariableWrapper`: that
+ class's ``__setattr__`` forwards every set to the actual netCDF4 object,
+ so it cannot carry an ad-hoc ``_data_array`` of its own.
+ """
+
+ def ncattrs(self):
+ return []
+
+
+class TestNetCDFOnlyMembers:
+ def test_variable_exposes_the_backing_wrapper(self, raw, sample_location):
+ # raw.variables constructs a fresh VariableWrapper on every access,
+ # so the check has to compare against the one instance actually
+ # passed in, not a second, independently fetched wrapper.
+ variable = raw.variables["air_temperature"]
+ wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4")
+ assert wrapped.variable is variable
+
+ def test_not_variable_length(self, air):
+ assert air.is_variable_length is False
+
+ def test_not_emulated(self, air):
+ assert air.is_emulated is False
+
+ def test_emulated_data_array_raises_when_not_emulated(self, air):
+ with pytest.raises(AttributeError, match="_data_array"):
+ air.emulated_data_array
+
+ def test_setting_emulated_data_array_raises_when_not_emulated(self, air):
+ # The setter refuses for the same reason the getter does, and the
+ # consequence of not refusing is worse: VariableWrapper.__setattr__
+ # forwards to the contained object, so the set would write a file
+ # attribute named "_data_array".
+ with pytest.raises(AttributeError, match="_data_array"):
+ air.emulated_data_array = np.ones(3)
+ assert "_data_array" not in air.variable.ncattrs()
+
+ def test_emulated_round_trip(self, sample_location):
+ # The emulation hook ncdata uses (issue #4994): an emulating variable
+ # carries its own array instead of file storage.
+ variable = _EmulatedVariable()
+ wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4")
+ variable._data_array = np.zeros(3)
+ assert wrapped.is_emulated is True
+ np.testing.assert_array_equal(wrapped.emulated_data_array, np.zeros(3))
+ wrapped.emulated_data_array = np.ones(3)
+ np.testing.assert_array_equal(variable._data_array, np.ones(3))
+
+
+def _unreachable_policy():
+ raise AssertionError("chunking policy consulted when the result is not lazy")
+
+
+class TestReadData:
+ def test_small_variable_is_real(self, air):
+ result = air.read_data(_unreachable_policy)
+ assert isinstance(result, np.ndarray)
+ np.testing.assert_array_equal(result, SAMPLE_AIR)
+
+ def test_large_variable_is_lazy(self, air, mocker):
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
+ result = air.read_data(lambda: ([1, 4], (False, False)))
+ assert isinstance(result, da.Array)
+ np.testing.assert_array_equal(result.compute(), SAMPLE_AIR)
+
+ def test_cache_key_is_the_proxy_repr(self, air, mocker):
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
+ as_lazy_data = mocker.patch("iris.fileformats.netcdf._dataset.as_lazy_data")
+ air.read_data(lambda: ([1, 4], (False, False)))
+ (proxy,) = as_lazy_data.call_args.args
+ assert as_lazy_data.call_args.kwargs["cache_key"] == repr(proxy)
+
+ def test_emulated_array_is_returned_as_it_stands(self, sample_location, mocker):
+ # The ncdata emulation hook, issue #4994: the emulating object's own
+ # array is the answer, with no reading and no proxy.
+ # The cutoff goes to zero so that the policy's unreachability is the
+ # emulation branch's doing, not the array merely being small.
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
+ variable = _EmulatedVariable()
+ variable._data_array = np.arange(6, dtype="f4").reshape(2, 3)
+ wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4")
+ result = wrapped.read_data(_unreachable_policy)
+ assert result is variable._data_array
+
+ def test_emulated_array_builds_no_proxy(self, sample_location, mocker):
+ # As above, the cutoff goes to zero so that reaching no proxy is the
+ # emulation branch's doing, not the array merely being small.
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
+ proxy_class = mocker.patch.object(_thread_safe_nc, "NetCDFDataProxy")
+ variable = _EmulatedVariable()
+ variable._data_array = np.arange(6, dtype="f4").reshape(2, 3)
+ wrapped = _dataset.NetCDFDatasetVariable(variable, sample_location, "NETCDF4")
+ wrapped.read_data(_unreachable_policy)
+ assert proxy_class.call_count == 0
+
+ def test_reading_data_changes_no_attribute_tracking(self, tmp_path, mocker):
+ # Which file attributes survive onto a cube is decided by which ones
+ # the loading rules did *not* read. read_data reads three CF-reserved
+ # names off the untracked storage mapping; this is only safe because
+ # those names are pre-marked as read. Pin that, because the failure is
+ # a silent leak onto cube.attributes.
+ from iris.fileformats.cf import CFReader
+
+ path = tmp_path / "packed.nc"
+ dataset = _thread_safe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ try:
+ dataset.createDimension("x", 4)
+ var = dataset.createVariable("thing", "i2", ("x",), fill_value=-1)
+ var.setncattr("scale_factor", np.float32(0.5))
+ var.setncattr("add_offset", np.float32(2.0))
+ var.setncattr("missing_value", np.int16(-1))
+ var.setncattr("units", "K")
+ var[:] = np.arange(4, dtype="i2")
+ finally:
+ dataset.close()
+
+ mocker.patch("iris.fileformats.cf.loader._LAZYVAR_MIN_BYTES", 0)
+ with CFReader(str(path)) as reader:
+ cf_var = reader.cf_group["thing"]
+ before = {name for name, _ in cf_var.cf_attrs_unused()}
+ assert before == {"units"}, (
+ "the four CF-reserved names should start out marked as read"
+ )
+ iris.fileformats.cf.loader._get_cf_var_data(cf_var)
+ after = {name for name, _ in cf_var.cf_attrs_unused()}
+ assert after == before
+
+
+class TestDeprecatedNetcdfMember:
+ def test_a_reach_through_warns(self, air):
+ with pytest.warns(IrisDeprecation, match="ncattrs"):
+ names = air.deprecated_netcdf_member("ncattrs")()
+ assert sorted(names) == [
+ "_FillValue",
+ "coordinates",
+ "standard_name",
+ "units",
+ ]
+
+ def test_a_missing_name_raises_and_does_not_warn(self, air):
+ # hasattr() probes land here, and a probe that comes back False is
+ # not a use of anything - warning about it would be noise. The
+ # underlying netCDF4 variable raises its own AttributeError, which
+ # does not name the attribute - only the type matters here.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ with pytest.raises(AttributeError):
+ air.deprecated_netcdf_member("no_such_netcdf_member")
+
+ def test_the_message_names_the_replacement(self, air):
+ with pytest.warns(IrisDeprecation, match=r"cf_data\.variable"):
+ air.deprecated_netcdf_member("ncattrs")
+
+
+class TestCharacterData:
+ def test_raw_wrapper_sees_the_char_dimension(self, raw, sample_location):
+ label = wrap(raw, "label", sample_location)
+ assert label.dimensions == ("time", "nchars")
+ assert label.shape == (3, 8)
+ assert label.dtype == np.dtype("S1")
+
+ def test_encoded_wrapper_sees_strings(self, encoded, sample_location):
+ # EncodedVariable drops the trailing char dimension and reports a
+ # string dtype. The CFDatasetVariable must pass that through, not
+ # reach around it to the contained netCDF4 variable.
+ label = wrap(encoded, "label", sample_location)
+ assert label.dimensions == ("time",)
+ assert label.shape == (3,)
+ assert label.dtype == np.dtype("U8")
+ assert [text.strip() for text in label[:]] == SAMPLE_LABELS
diff --git a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py
new file mode 100644
index 0000000000..e72c15fa45
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__contract.py
@@ -0,0 +1,42 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Run the shared CFDataset contract against the netCDF implementation."""
+
+import pytest
+
+from iris.fileformats.netcdf import _dataset
+from iris.tests.unit.fileformats.cf.dataset.contract import (
+ CFDatasetReadContract,
+ CFDatasetWriteContract,
+)
+
+
+class TestNetCDFDatasetContract(CFDatasetReadContract, CFDatasetWriteContract):
+ """The netCDF side of the contract."""
+
+ chunking_key = "chunksizes"
+
+ @pytest.fixture
+ def readable(self, tmp_path):
+ path = tmp_path / "contract.nc"
+ with _dataset.NetCDFDataset(path, mode="w", netcdf_format="NETCDF4") as dataset:
+ self.populate(dataset)
+ with _dataset.NetCDFDataset(path) as dataset:
+ yield dataset
+
+ @pytest.fixture
+ def writable(self, tmp_path):
+ dataset = _dataset.NetCDFDataset(
+ tmp_path / "written.nc", mode="w", netcdf_format="NETCDF4"
+ )
+ yield dataset
+ dataset.close()
+
+ @pytest.fixture
+ def reopen(self, tmp_path):
+ def _reopen():
+ return _dataset.NetCDFDataset(tmp_path / "written.nc")
+
+ return _reopen
diff --git a/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py
new file mode 100644
index 0000000000..458efdfde7
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset__write.py
@@ -0,0 +1,256 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the write surface of the netCDF CFDataset implementation."""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.netcdf import _bytecoding_datasets, _dataset, _thread_safe_nc
+
+
+@pytest.fixture
+def path(tmp_path):
+ return tmp_path / "written.nc"
+
+
+@pytest.fixture
+def writer(path):
+ with _dataset.NetCDFDataset(path, mode="w", netcdf_format="NETCDF4") as dataset:
+ yield dataset
+
+
+@pytest.fixture
+def grid(writer):
+ """A 2x3 grid, ready for variables to be created against."""
+ writer.create_dimension("y", 3)
+ writer.create_dimension("x", 2)
+ return writer
+
+
+class TestCreateDimension:
+ def test_appears_with_its_length(self, writer):
+ writer.create_dimension("x", 5)
+ assert writer.dimensions["x"] == 5
+
+ def test_none_means_unlimited(self, writer):
+ # Saver._create_cf_dimensions passes None for a dimension the user
+ # asked to make unlimited. It reads back as zero-length until records
+ # are written.
+ writer.create_dimension("t", None)
+ assert writer.dimensions["t"] == 0
+ assert writer.dataset.dimensions["t"].isunlimited()
+
+
+class TestCreateVariable:
+ def test_returns_a_dataset_variable(self, grid):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ assert isinstance(variable, _dataset.NetCDFDatasetVariable)
+ assert variable.name == "air"
+ assert variable.dimensions == ("y", "x")
+ assert variable.shape == (3, 2)
+ assert variable.dtype == np.dtype("f4")
+
+ def test_appears_in_the_dataset(self, grid):
+ created = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ assert grid.variables["air"].name == created.name
+
+ def test_updates_an_already_materialised_variables_cache(self, grid):
+ # Pins that create_variable keeps an already-materialised .variables
+ # mapping current, rather than leaving it stale. saver.py's
+ # variable-naming collision loops - `while <name> in
+ # self._dataset.variables: <name> = self._increment_name(<name>)` -
+ # read `.variables` repeatedly while creating variables, so a stale
+ # cache here would let a name collision go undetected.
+ assert "air" not in grid.variables # materialise the cache first
+ grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ assert "air" in grid.variables
+
+ def test_dimensions_default_to_scalar(self, writer):
+ # saver.py creates a grid-mapping variable with no dimensions at all,
+ # passing only a name and a dtype.
+ variable = writer.create_variable("grid", np.int32)
+ assert variable.dimensions == ()
+ assert variable.shape == ()
+
+ def test_dimensions_may_be_a_list(self, grid):
+ # Saver._create_generic_cf_array_var passes element_dims, a list.
+ variable = grid.create_variable("air", np.dtype("f4"), ["y", "x"])
+ assert variable.dimensions == ("y", "x")
+
+ def test_fill_value_becomes_an_attribute(self, grid):
+ variable = grid.create_variable(
+ "air", np.dtype("f4"), ("y", "x"), fill_value=-1.0
+ )
+ assert variable.fill_value == -1.0
+ assert variable.attributes["_FillValue"] == -1.0
+
+ def test_encoding_is_passed_through(self, grid):
+ variable = grid.create_variable(
+ "air", np.dtype("f4"), ("y", "x"), zlib=True, chunksizes=(1, 2)
+ )
+ assert variable.chunking == (1, 2)
+
+ def test_shares_the_datasets_write_lock(self, grid):
+ # Observed at write time, not at construction time: the variable is
+ # handed a factory, and the lock is only made when a write handle
+ # actually asks for it - see NetCDFDataset.write_lock for why a read
+ # must not provoke one.
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ assert grid._write_lock is None
+ assert variable.write_handle().lock is grid.write_lock
+
+
+class TestWritingData:
+ def test_setitem_round_trip(self, grid, path):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ payload = np.arange(6, dtype="f4").reshape(3, 2)
+ variable[:] = payload
+ grid.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ np.testing.assert_array_equal(reader.variables["air"][:], payload)
+
+ def test_write_handle_writes_after_the_dataset_is_closed(self, grid, path):
+ # This is what a Dask worker gets as a da.store target, long after the
+ # Saver's own handle on the file has gone.
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ handle = variable.write_handle()
+ grid.close()
+
+ payload = np.arange(6, dtype="f4").reshape(3, 2)
+ handle[:] = payload
+
+ with _dataset.NetCDFDataset(path) as reader:
+ np.testing.assert_array_equal(reader.variables["air"][:], payload)
+
+ def test_write_handle_of_an_encoded_variable_encodes(self, grid):
+ # The handle always encodes, whatever the variable's own encoding.
+ # This is the encoded half - the dataset was opened for writing, so
+ # its variables are encoded; the test below is the unencoded half.
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ assert isinstance(variable.variable, _bytecoding_datasets.EncodedVariable)
+ assert isinstance(
+ variable.write_handle(), _bytecoding_datasets.EncodedNetCDFWriteProxy
+ )
+
+ def test_write_handle_of_an_unencoded_variable_still_encodes(self, path):
+ # The saver does meet unencoded variables: from_existing only wraps a
+ # dataset lacking THREAD_SAFE_FLAG, so a borrowed DatasetWrapper keeps
+ # plain variables like the one built here. Writes have no selectable
+ # string encoding, so the handle must encode regardless.
+ raw = _thread_safe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ try:
+ raw.createDimension("x", 2)
+ raw.createVariable("air", "f4", ("x",))
+ variable = _dataset.NetCDFDatasetVariable(
+ raw.variables["air"],
+ str(path),
+ raw.file_format,
+ write_lock_factory=None,
+ )
+ assert not isinstance(
+ variable.variable, _bytecoding_datasets.EncodedVariable
+ )
+ assert isinstance(
+ variable.write_handle(), _bytecoding_datasets.EncodedNetCDFWriteProxy
+ )
+ finally:
+ raw.close()
+
+
+class TestSyncAndFinalise:
+ def test_sync_flushes(self, grid, path):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable[:] = np.zeros((3, 2), dtype="f4")
+ grid.sync()
+ # Readable while the writer is still open, because sync() flushed.
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.variables["air"].shape == (3, 2)
+
+ def test_finalise_is_a_no_op(self, grid):
+ assert grid.finalise() is None
+ assert grid.closed is False
+
+
+class TestAttributeWrites:
+ def test_variable_attribute_write_through(self, grid, path):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["units"] = "K"
+ assert variable.attributes["units"] == "K"
+ grid.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.variables["air"].attributes["units"] == "K"
+
+ def test_global_attribute_write_through(self, writer, path):
+ writer.attributes["Conventions"] = "CF-1.7"
+ assert writer.attributes["Conventions"] == "CF-1.7"
+ writer.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.attributes["Conventions"] == "CF-1.7"
+
+ def test_attribute_deletion(self, grid, path):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["units"] = "K"
+ del variable.attributes["units"]
+ assert "units" not in variable.attributes
+ grid.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert "units" not in reader.variables["air"].attributes
+
+ def test_ascii_value_is_written_as_bytes(self, grid):
+ # _bytes_if_ascii, moved here from saver.py. netCDF4 writes a bytes
+ # value as NC_CHAR; the coercion is what makes every string attribute
+ # Iris writes take that type, whatever the file format.
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["units"] = "K"
+ assert variable.variable.getncattr("units") == "K"
+ assert _dataset._bytes_if_ascii("K") == b"K"
+
+ def test_non_string_values_pass_through(self, grid):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["valid_min"] = np.float32(-3.5)
+ assert variable.attributes["valid_min"] == np.float32(-3.5)
+ assert _dataset._bytes_if_ascii(3) == 3
+
+
+class TestNonAsciiAttributes:
+ """The coercion's except branch, which is the common one for real-world
+ metadata: degree signs, accented names, superscripts.
+ """
+
+ @pytest.mark.parametrize(
+ "value",
+ [
+ "degC \N{DEGREE SIGN}",
+ "na\N{LATIN SMALL LETTER I WITH DIAERESIS}ve",
+ "m s\N{SUPERSCRIPT MINUS}\N{SUPERSCRIPT ONE}",
+ ],
+ )
+ def test_round_trips_unchanged(self, grid, path, value):
+ variable = grid.create_variable("air", np.dtype("f4"), ("y", "x"))
+ variable.attributes["long_name"] = value
+ assert variable.attributes["long_name"] == value
+ grid.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.variables["air"].attributes["long_name"] == value
+
+ def test_is_not_coerced_to_bytes(self):
+ value = "degC \N{DEGREE SIGN}"
+ assert _dataset._bytes_if_ascii(value) is value
+
+ def test_global_non_ascii_round_trips(self, writer, path):
+ value = (
+ "Produced at M\N{LATIN SMALL LETTER E WITH ACUTE}"
+ "t\N{LATIN SMALL LETTER E WITH ACUTE}o"
+ )
+ writer.attributes["institution"] = value
+ writer.close()
+
+ with _dataset.NetCDFDataset(path) as reader:
+ assert reader.attributes["institution"] == value
diff --git a/lib/iris/tests/unit/fileformats/netcdf/loader/test__relocated_names.py b/lib/iris/tests/unit/fileformats/netcdf/loader/test__relocated_names.py
new file mode 100644
index 0000000000..f5c0353587
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/loader/test__relocated_names.py
@@ -0,0 +1,148 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for the deprecated aliases left behind in the netCDF loader.
+
+``CHUNK_CONTROL`` and ``ChunkControl`` moved to
+:mod:`iris.fileformats.cf.loader`; the old names still answer, with a warning.
+"""
+
+import importlib
+from unittest import mock
+import warnings
+
+import pytest
+
+from iris._deprecation import IrisDeprecation
+from iris.fileformats.cf import loader as cf_loader
+from iris.fileformats.netcdf import loader as netcdf_loader
+
+RELOCATED = ["CHUNK_CONTROL", "ChunkControl"]
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+def test_the_old_name_warns(name):
+ with pytest.warns(IrisDeprecation, match=f"netcdf.loader.{name}"):
+ getattr(netcdf_loader, name)
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+def test_the_message_names_the_new_location(name):
+ with pytest.warns(IrisDeprecation, match=r"iris\.fileformats\.cf\.loader"):
+ getattr(netcdf_loader, name)
+
+
+def test_the_warning_points_at_the_caller():
+ # warn_deprecated's default stacklevel=2 would name this module's own
+ # __getattr__ - an extra frame PEP 562 inserts between the caller and
+ # warn_deprecated - rather than the line below. Pin the fix: the warning
+ # must be attributed to *this* file, not to netcdf/loader.py.
+ with pytest.warns(IrisDeprecation) as record:
+ netcdf_loader.CHUNK_CONTROL
+ assert record[0].filename == __file__
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+def test_the_alias_is_the_same_object(name):
+ # Spec 4.8: "The alias is the same object, not a copy, so existing code
+ # keeps working including inside a with CHUNK_CONTROL.set(...)". A copy
+ # would give a with-block that silently chunked nothing.
+ with pytest.warns(IrisDeprecation):
+ assert getattr(netcdf_loader, name) is getattr(cf_loader, name)
+
+
+def test_importing_the_module_does_not_warn():
+ # Spec 4.8: "Warnings are emitted on use, not on import, so simply
+ # importing iris.fileformats.netcdf stays quiet."
+ #
+ # Reloads this module, not the whole iris.fileformats.netcdf package: the
+ # package's own body only *fetches* DEBUG/NetCDFDataProxy/load_cubes as
+ # already-bound names from this (unreloaded) submodule, so reloading the
+ # package would never re-execute the code this test means to exercise.
+ # Confirmed: reloading just this module leaves the shared
+ # "iris.fileformats.netcdf" logger's handler list untouched (the
+ # StreamHandler-per-call side effect lives in the package's __init__, in
+ # iris.config.get_logger, not here), so there is no handler-leak risk to
+ # guard against on this path.
+ #
+ # Reloading *does* rebuild `load_cubes` as a brand new function object,
+ # though, and that is consequential: iris.fileformats.netcdf.load_cubes -
+ # bound once, at the package's own first import, and the handler
+ # iris.fileformats.FORMAT_AGENT registered at Iris startup - keeps
+ # pointing at the *old* one, while iris.fileformats.netcdf.ugrid_load's
+ # load_meshes recognises netCDF sources by `==` identity against this
+ # module's (new, post-reload) one. Left unrestored, that silently breaks
+ # mesh loading for the rest of the process - confirmed by reproducing the
+ # failure directly before adding this restore.
+ original_load_cubes = netcdf_loader.load_cubes
+ try:
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ importlib.reload(netcdf_loader)
+ finally:
+ netcdf_loader.load_cubes = original_load_cubes
+
+
+def test_an_unrelated_name_raises_without_warning():
+ # hasattr() probes land in __getattr__ too, and a probe that comes back
+ # False is not a use of anything.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ with pytest.raises(AttributeError, match="no_such_name"):
+ netcdf_loader.no_such_name
+
+
+def test_debug_is_not_relocated():
+ # Spec 4.8 lists netcdf.DEBUG among the names that do not move. It is a
+ # flag users assign to, and a module __getattr__ cannot intercept
+ # assignment - so forwarding it would make `netcdf.loader.DEBUG = True`
+ # silently stop working. Pin that it is a real attribute here.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ assert netcdf_loader.DEBUG is False
+ assert "DEBUG" not in netcdf_loader._RELOCATED_NAMES
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+def test_dir_includes_the_relocated_name(name):
+ # A module __getattr__ forwards reads but is invisible to dir(), so
+ # __dir__ has to union the relocated names in by hand.
+ assert name in dir(netcdf_loader)
+
+
+def test_dir_still_includes_a_name_that_genuinely_lives_here():
+ # Pin that __dir__ adds the relocated names rather than replacing the
+ # real module listing.
+ assert "load_cubes" in dir(netcdf_loader)
+ assert "DEBUG" in dir(netcdf_loader)
+
+
+def test_mock_patch_on_the_old_path_is_a_silent_no_op():
+ # mock.patch("iris.fileformats.netcdf.loader.CHUNK_CONTROL", ...)
+ # succeeds, because patch simply sets an attribute here - the same
+ # mechanism test_assignment_to_a_relocated_name_does_not_reach_the_new_home
+ # pins for a plain assignment. Production code reads
+ # iris.fileformats.cf.loader.CHUNK_CONTROL, so the patch never reaches it.
+ # A downstream test suite patching the old path would go green while
+ # testing nothing - this test states that limitation rather than leaving
+ # a future reader to discover it the hard way.
+ original = cf_loader.CHUNK_CONTROL
+ with mock.patch("iris.fileformats.netcdf.loader.CHUNK_CONTROL", "patched"):
+ assert netcdf_loader.CHUNK_CONTROL == "patched"
+ assert cf_loader.CHUNK_CONTROL is original
+
+
+def test_assignment_to_a_relocated_name_does_not_reach_the_new_home():
+ # The known limitation of the PEP 562 forward, written down so that it is
+ # a documented edge and not a surprise. Assigning here shadows the
+ # forward; it does not change iris.fileformats.cf.loader. Nothing in Iris
+ # assigns to CHUNK_CONTROL - it is used through its context managers - but
+ # if that ever changes, this test says where to look.
+ original = cf_loader.CHUNK_CONTROL
+ netcdf_loader.CHUNK_CONTROL = "shadow"
+ try:
+ assert netcdf_loader.CHUNK_CONTROL == "shadow"
+ assert cf_loader.CHUNK_CONTROL is original
+ finally:
+ del netcdf_loader.CHUNK_CONTROL
diff --git a/lib/iris/tests/unit/fileformats/netcdf/loader/ugrid_load/test_load_meshes.py b/lib/iris/tests/unit/fileformats/netcdf/loader/ugrid_load/test_load_meshes.py
index 538554a428..1111034e53 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/loader/ugrid_load/test_load_meshes.py
+++ b/lib/iris/tests/unit/fileformats/netcdf/loader/ugrid_load/test_load_meshes.py
@@ -168,7 +168,7 @@ def test_invalid_scheme(self):
@_shared_utils.skip_data
def test_non_nc(self, caplog):
- log_regex = r"Ignoring non-NetCDF file:.*"
+ log_regex = r"Ignoring file of a format that holds no mesh:.*"
with _shared_utils.assert_logs(
caplog, logger, level="INFO", msg_regex=log_regex
):
diff --git a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py
index 1451bfdae9..d5dd726066 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py
+++ b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py
@@ -4,7 +4,6 @@
# See LICENSE in the root of the repository for full licensing details.
"""Unit tests for the :class:`iris.fileformats.netcdf.Saver` class."""
-import collections
from contextlib import contextmanager
from pathlib import Path
from types import ModuleType
@@ -32,6 +31,7 @@
from iris.cube import Cube
from iris.fileformats.netcdf import Saver, _thread_safe_nc
from iris.fileformats.netcdf import _bytecoding_datasets as ds_wrappers
+from iris.fileformats.netcdf._dataset import NetCDFDatasetVariable
from iris.tests import _shared_utils
from iris.tests._shared_utils import assert_CDL
import iris.tests.stock as stock
@@ -217,22 +217,18 @@ def test_big_endian(self, request, tmp_path):
def test_zlib(self, mocker):
cube = self._simple_cube(">f4")
- api = mocker.patch("iris.fileformats.netcdf.saver.bytecoding_datasets")
- # Define mocked default fill values to prevent deprecation warning (#4374).
- api.default_fillvals = collections.defaultdict(lambda: -99.0)
+ api = mocker.patch("iris.fileformats.netcdf._dataset._bytecoding_datasets")
# Mock the apparent dtype of mocked variables, to avoid an error.
- ref = api.DatasetWrapper.return_value
- ref = ref.createVariable.return_value
- ref.dtype = np.dtype(np.float32)
+ dataset = api.EncodedDataset.return_value
+ dataset.createVariable.return_value.dtype = np.dtype(np.float32)
# NOTE: use compute=False as otherwise it gets in a pickle trying to construct
# a fill-value report on a non-compliant variable in a non-file (!)
with Saver("/dummy/path", "NETCDF4", compute=False) as saver:
saver.write(cube, zlib=True)
- dataset = api.EncodedDataset.return_value
create_var_call = mocker.call(
"air_pressure_anomaly",
np.dtype("float32"),
- ["dim0", "dim1"],
+ ("dim0", "dim1"),
fill_value=None,
shuffle=True,
least_significant_digit=None,
@@ -276,7 +272,7 @@ def test_compression(self, mocker, tmp_path):
tgt,
# Use 'wraps' to allow the patched methods to function as normal
# - the patch object just acts as a 'spy' on its calls.
- wraps=saver._dataset.createVariable,
+ wraps=saver._dataset.dataset.createVariable,
)
saver.write(cube, **compression_kwargs)
@@ -316,7 +312,7 @@ def test_non_compression__shape(self, mocker, tmp_path):
tgt,
# Use 'wraps' to allow the patched methods to function as normal
# - the patch object just acts as a 'spy' on its calls.
- wraps=saver._dataset.createVariable,
+ wraps=saver._dataset.dataset.createVariable,
)
saver.write(cube, **compression_kwargs)
@@ -356,7 +352,7 @@ def test_non_compression__dtype(self, mocker, tmp_path):
tgt,
# Use 'wraps' to allow the patched methods to function as normal
# - the patch object just acts as a 'spy' on its calls.
- wraps=saver._dataset.createVariable,
+ wraps=saver._dataset.dataset.createVariable,
)
saver.write(cube, **compression_kwargs)
@@ -506,24 +502,26 @@ def _check_bounds_setting(self, climatological=False):
saver._ensure_valid_dtype.return_value = self.mocker.Mock(
shape=coord.bounds.shape, dtype=coord.bounds.dtype
)
- var = self.mocker.MagicMock(spec=ds_wrappers.EncodedVariable)
+ var = self.mocker.MagicMock(spec=NetCDFDatasetVariable)
+ var.attributes = {}
+ var.dimensions = ("time",)
# Make the main call.
Saver._create_cf_bounds(saver, coord, var, "time")
- # Test the call of _setncattr in _create_cf_bounds.
- setncattr_call = self.mocker.call(
- property_name, boundsvar_name.encode(encoding="ascii")
- )
- assert setncattr_call == var.setncattr.call_args
+ # Test the attribute written by _create_cf_bounds. The ASCII-to-bytes
+ # coercion now happens inside _NetCDFAttributes.__setitem__, and is
+ # tested there; this plain dict stands in for one, so the value
+ # arrives as given.
+ assert var.attributes[property_name] == boundsvar_name
- # Test the call of createVariable in _create_cf_bounds.
+ # Test the call of create_variable in _create_cf_bounds.
dataset = saver._dataset
expected_dimensions = var.dimensions + ("bnds",)
create_var_call = self.mocker.call(
boundsvar_name, coord.bounds.dtype, expected_dimensions
)
- assert create_var_call == dataset.createVariable.call_args
+ assert create_var_call == dataset.create_variable.call_args
def test_set_bounds_default(self):
self._check_bounds_setting(climatological=False)
@@ -742,8 +740,10 @@ def assert_attribute(self, value):
def check_attribute_compliance_call(self, value, file_type="NETCDF4"):
self.set_attribute(value)
with Saver("nonexistent test file", file_type) as saver:
- # Get the Mock to work properly.
- saver._dataset.file_format = file_type
+ # Get the Mock to work properly. The format is read from the
+ # netCDF dataset itself, through Saver's escape hatch; setting it
+ # on the NetCDFDataset would bind an attribute nothing consults.
+ saver._dataset.dataset.file_format = file_type
saver.check_attribute_compliance(self.container, self.data_dtype)
@@ -1050,12 +1050,19 @@ class NCMock(self.mocker.Mock):
def setncattr(self, name, attr):
setattr(self, name, attr)
+ def ncattrs(self):
+ # A variable the dataset wraps is asked for its attributes as
+ # it is wrapped; this one starts with none.
+ return []
+
# Calls the actual NetCDF saver with appropriate mocking, returning
# the grid variable that gets created.
grid_variable = NCMock(name="NetCDFVariable")
create_var_fn = self.mocker.Mock(side_effect=[grid_variable])
- dataset = self.mocker.Mock(variables=[], createVariable=create_var_fn)
- variable = NCMock()
+ # 'variables' is a mapping, as netCDF4 presents it: the saver looks
+ # names up in it to avoid a grid-mapping variable name collision.
+ dataset = self.mocker.Mock(variables={}, createVariable=create_var_fn)
+ variable = NCMock(attributes={})
saver = Saver(dataset, "NETCDF4", compute=False)
@@ -1063,7 +1070,13 @@ def setncattr(self, name, attr):
saver._create_cf_grid_mapping(cube, variable)
assert create_var_fn.call_count == 1
- assert variable.grid_mapping, grid_variable.grid_mapping_name
+ # The data variable refers to the grid variable by name. Under
+ # extended grid mapping the reference is followed by the coordinates
+ # it applies to, so compare the name alone. The grid variable's own
+ # name comes back as bytes, from the ASCII coercion that every
+ # attribute write goes through.
+ referenced_name = variable.attributes["grid_mapping"].split(":")[0]
+ assert referenced_name == grid_variable.grid_mapping_name.decode()
return grid_variable
def _variable_attributes(self, coord_system):
diff --git a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dataset.py b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dataset.py
new file mode 100644
index 0000000000..66b8cb3176
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dataset.py
@@ -0,0 +1,104 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Tests for the dataset :class:`iris.fileformats.netcdf.saver.Saver` owns."""
+
+import numpy as np
+import pytest
+
+from iris.fileformats.netcdf import _thread_safe_nc as threadsafe_nc
+from iris.fileformats.netcdf._dataset import NetCDFDataset
+from iris.fileformats.netcdf.saver import Saver
+
+
+class TestOwnedDataset:
+ @pytest.fixture
+ def saver(self, tmp_path):
+ with Saver(tmp_path / "test.nc", "NETCDF4") as saver:
+ yield saver
+
+ def test_dataset_is_a_netcdf_cfdataset(self, saver):
+ assert isinstance(saver._dataset, NetCDFDataset)
+
+ def test_filepath_is_still_a_path(self, saver, tmp_path):
+ # Public API: Saver.filepath has always been a Path for a local file.
+ assert saver.filepath == (tmp_path / "test.nc").absolute()
+
+ def test_the_write_lock_is_the_datasets_own(self, saver):
+ # Every variable of the dataset holds this same lock. A second lock
+ # would exclude nothing - see NetCDFDataset.write_lock.
+ assert saver.file_write_lock is saver._dataset.write_lock
+
+ def test_format_reaches_the_file(self, tmp_path):
+ # Not through the context manager: __exit__ syncs, and a netCDF3 file
+ # with nothing written to it is still in define mode, where sync()
+ # raises. That is as true of the saver before this change as after.
+ saver = Saver(tmp_path / "classic.nc", "NETCDF3_CLASSIC")
+ try:
+ assert saver._dataset.dataset.file_format == "NETCDF3_CLASSIC"
+ finally:
+ saver._dataset.close()
+
+ def test_exit_closes_an_owned_dataset(self, tmp_path):
+ with Saver(tmp_path / "test.nc", "NETCDF4") as saver:
+ pass
+ assert saver._dataset.closed
+
+
+class TestBorrowedDataset:
+ @pytest.fixture
+ def raw(self, tmp_path):
+ dataset = threadsafe_nc.DatasetWrapper(
+ tmp_path / "user.nc", mode="w", format="NETCDF4"
+ )
+ yield dataset
+ if dataset.isopen():
+ dataset.close()
+
+ def test_dataset_is_a_netcdf_cfdataset(self, raw):
+ with Saver(raw, "NETCDF4", compute=False) as saver:
+ assert isinstance(saver._dataset, NetCDFDataset)
+
+ def test_exit_leaves_a_borrowed_dataset_open(self, raw):
+ with Saver(raw, "NETCDF4", compute=False) as saver:
+ pass
+ assert raw.isopen()
+ assert not saver._dataset.closed
+
+ def test_complete_refuses_while_the_file_is_open(self, raw):
+ with Saver(raw, "NETCDF4", compute=False) as saver:
+ pass
+ with pytest.raises(ValueError, match="until its dataset is closed"):
+ saver.complete()
+
+ def test_complete_sees_a_dataset_closed_behind_its_back(self, raw):
+ # The caller owns the dataset and closes it themselves, so the flag
+ # NetCDFDataset.close() sets is never set. complete() has to ask the
+ # file, not the flag.
+ with Saver(raw, "NETCDF4", compute=False) as saver:
+ pass
+ raw.close()
+ assert saver._dataset.closed
+ saver.complete() # must not raise
+
+
+class TestWritingThroughTheDataset:
+ """The four verbs, through CFDataset rather than netCDF4."""
+
+ @pytest.fixture
+ def saver(self, tmp_path):
+ with Saver(tmp_path / "test.nc", "NETCDF4") as saver:
+ yield saver
+
+ def test_create_dimension(self, saver):
+ saver._dataset.create_dimension("x", 3)
+ assert saver._dataset.dimensions["x"] == 3
+
+ def test_create_variable_returns_a_cf_dataset_variable(self, saver):
+ from iris.fileformats.netcdf._dataset import NetCDFDatasetVariable
+
+ saver._dataset.create_dimension("x", 3)
+ variable = saver._dataset.create_variable("a", np.dtype("f4"), ("x",))
+ assert isinstance(variable, NetCDFDatasetVariable)
+ assert saver._dataset.variables["a"] is variable
diff --git a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py
new file mode 100644
index 0000000000..973ae597c3
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__dtypes.py
@@ -0,0 +1,80 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""What the netCDF saver does with a dtype the target format cannot hold.
+
+Relocating the CF saver for Zarr support (SciTools/iris#6977) moved the
+narrowing decision out of the saver, which now asks the dataset instead of
+reading netCDF format strings. Every answer here survived that move except
+one: SciTools/iris#7337 was fixed in the same change, so "NETCDF3_64BIT"
+narrows now where it used to fail inside the netCDF C library.
+"""
+
+import numpy as np
+import pytest
+
+import iris
+from iris.coords import AuxCoord
+from iris.cube import Cube
+from iris.fileformats.netcdf._dataset import NetCDFDataset
+
+#: Every format Saver accepts.
+ALL_FORMATS = ("NETCDF4", "NETCDF4_CLASSIC", "NETCDF3_CLASSIC", "NETCDF3_64BIT")
+
+
+def saved_dtype(cube, path, netcdf_format):
+ """Save the cube, and return the dtype its data variable has in the file."""
+ iris.save(cube, path, netcdf_format=netcdf_format)
+ with NetCDFDataset(path) as dataset:
+ return dataset.variables[cube.var_name].dtype
+
+
+@pytest.fixture
+def int64_cube():
+ """Return an int64 cube whose values all fit in an int32."""
+ cube = Cube(np.arange(4, dtype=np.int64), var_name="payload")
+ cube.add_aux_coord(AuxCoord(np.arange(4, dtype=np.int64), var_name="index"), 0)
+ return cube
+
+
+class TestInt64:
+ """int64 is the dtype the restricted formats cannot hold."""
+
+ def test_netcdf4_keeps_it(self, int64_cube, tmp_path):
+ assert saved_dtype(int64_cube, tmp_path / "out.nc", "NETCDF4") == np.int64
+
+ @pytest.mark.parametrize(
+ "netcdf_format", ["NETCDF4_CLASSIC", "NETCDF3_CLASSIC", "NETCDF3_64BIT"]
+ )
+ def test_the_restricted_formats_narrow_it(
+ self, int64_cube, tmp_path, netcdf_format
+ ):
+ assert saved_dtype(int64_cube, tmp_path / "out.nc", netcdf_format) == np.int32
+
+ @pytest.mark.parametrize(
+ ("netcdf_format", "named"),
+ [
+ ("NETCDF4_CLASSIC", "NETCDF4_CLASSIC"),
+ ("NETCDF3_CLASSIC", "NETCDF3_CLASSIC"),
+ # The message names the format netCDF4 reports, which for this one
+ # is not the format the caller asked for: SciTools/iris#7337.
+ ("NETCDF3_64BIT", "NETCDF3_64BIT_OFFSET"),
+ ],
+ )
+ def test_a_value_too_large_to_narrow_names_the_format(
+ self, tmp_path, netcdf_format, named
+ ):
+ cube = Cube(np.array([2**40], dtype=np.int64), var_name="payload")
+ with pytest.raises(ValueError, match=named):
+ iris.save(cube, tmp_path / "out.nc", netcdf_format=netcdf_format)
+
+
+class TestDtypesEveryFormatHolds:
+ """A dtype the format holds is written as it stands, never narrowed."""
+
+ @pytest.mark.parametrize("dtype", ["f4", "f8", "i2", "i4"])
+ @pytest.mark.parametrize("netcdf_format", ALL_FORMATS)
+ def test_not_narrowed(self, tmp_path, dtype, netcdf_format):
+ cube = Cube(np.arange(4, dtype=dtype), var_name="payload")
+ assert saved_dtype(cube, tmp_path / "out.nc", netcdf_format) == np.dtype(dtype)
diff --git a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__hooks.py b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__hooks.py
new file mode 100644
index 0000000000..438a1809e8
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__hooks.py
@@ -0,0 +1,85 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The netCDF-specific parts of the save chain, named and tested.
+
+Relocating the CF saver (SciTools/iris#6977) moved the rest of the class to a
+format-agnostic home. Each test here covers one of the parts that stayed
+behind, from the outside, so that the move could not quietly change what a
+backend is asked to do.
+"""
+
+import dask.array as da
+import numpy as np
+import pytest
+
+import iris
+from iris.coords import AuxCoord
+from iris.cube import Cube
+from iris.fileformats.cf.dataset import CFDataset, CFDatasetVariable
+from iris.fileformats.netcdf._dataset import NetCDFDataset
+
+
+class TestStringVariables:
+ """Unicode data becomes a char variable over a string dimension."""
+
+ @pytest.fixture
+ def labelled_cube(self):
+ cube = Cube(np.arange(3.0), var_name="payload")
+ cube.add_aux_coord(
+ AuxCoord(np.array(["alpha", "beta", "gamma"]), var_name="label"), 0
+ )
+ return cube
+
+ def test_a_string_dimension_is_created_for_the_label(self, labelled_cube, tmp_path):
+ path = tmp_path / "labelled.nc"
+ iris.save(labelled_cube, path)
+ with NetCDFDataset(path) as dataset:
+ # Five ascii characters, so five bytes. The dimension is the
+ # width, and exists only because netCDF has no unicode type.
+ assert dataset.dimensions["string5"] == 5
+
+ def test_the_label_round_trips(self, labelled_cube, tmp_path):
+ path = tmp_path / "labelled.nc"
+ iris.save(labelled_cube, path)
+ loaded = iris.load_cube(path)
+ assert list(loaded.coord("label").points) == ["alpha", "beta", "gamma"]
+
+
+class TestNCZarrWrites:
+ """NCZarr cannot be reopened, so lazy data is written before the close."""
+
+ def test_lazy_data_reaches_an_nczarr_target(self, tmp_path):
+ cube = Cube(da.arange(6.0, chunks=3).reshape(2, 3), long_name="payload")
+ assert cube.has_lazy_data()
+ uri = (tmp_path / "out.zarr").as_uri() + "#mode=nczarr,file"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(cube, uri)
+ loaded = iris.load_cube(uri)
+ assert loaded.data.ravel().tolist() == [0.0, 1.0, 2.0, 3.0, 4.0, 5.0]
+
+
+class TestCFPatch:
+ """The site-configuration hook is documented as receiving netCDF4 objects."""
+
+ def test_cf_patch_receives_raw_netcdf4_objects(self, mocker, tmp_path):
+ seen = {}
+
+ def record(profile, dataset, variable):
+ seen["dataset"] = dataset
+ seen["variable"] = variable
+
+ mocker.patch.dict(
+ iris.site_configuration,
+ {"cf_profile": lambda cube: "profiled", "cf_patch": record},
+ )
+ iris.save(Cube(np.arange(3.0), var_name="payload"), tmp_path / "patched.nc")
+
+ # Unwrapped, both of them. A CFVariable defines no __setattr__, so a
+ # hook doing 'variable.name = value' against one would succeed and
+ # write nothing at all.
+ assert not isinstance(seen["dataset"], CFDataset)
+ assert not isinstance(seen["variable"], CFDatasetVariable)
+ assert hasattr(seen["dataset"], "createVariable")
+ assert hasattr(seen["variable"], "setncattr")
diff --git a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__lazy_stream_data.py b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__lazy_stream_data.py
index 3f1c38661b..167ef0a8a8 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__lazy_stream_data.py
+++ b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__lazy_stream_data.py
@@ -16,8 +16,7 @@
import numpy as np
import pytest
-import iris.fileformats.netcdf._bytecoding_datasets as bytecoding_datasets
-import iris.fileformats.netcdf._thread_safe_nc as threadsafe_nc
+import iris.fileformats.netcdf._dataset as dataset_module
from iris.fileformats.netcdf.saver import Saver
@@ -61,18 +60,15 @@ def saver(compute, data_form, tmp_path) -> Saver:
@staticmethod
def mock_var(shape, with_data_array, mocker, dtype=np.dtype(np.float32)):
# Create a test cf_var object.
- # N.B. using 'spec=' so we can control whether it has a '_data_array' property.
- if with_data_array:
- extra_properties = {"_data_array": mocker.sentinel.initial_data_array}
- else:
- extra_properties = {}
+ # 'is_emulated' is now a declared property rather than the presence of
+ # a '_data_array' member, so it can simply be set.
mock_cfvar = mocker.MagicMock(
- spec=threadsafe_nc.VariableWrapper,
+ spec=dataset_module.NetCDFDatasetVariable,
shape=tuple(shape),
dtype=dtype,
- _contained_instance=mocker.Mock(dtype="f4"),
- **extra_properties,
+ is_emulated=with_data_array,
)
+ mock_cfvar.write_handle.return_value = mocker.sentinel.write_handle
# Give the mock cf-var a name property, as required by '_lazy_stream_data'.
# This *can't* be an extra kwarg to MagicMock __init__, since that already
# defines a specific 'name' kwarg, with a different purpose.
@@ -108,12 +104,14 @@ def test_data_save(self, compute, data_form, mocker, tmp_path):
if data_form == "lazydata":
result_data, result_writer = saver._delayed_writes[0]
assert result_data is data
- assert isinstance(result_writer, threadsafe_nc.NetCDFWriteProxy)
+ # What kind of handle it is is the dataset's business, and is
+ # tested in test_NetCDFDataset__write.py.
+ assert result_writer is mocker.sentinel.write_handle
elif data_form == "realdata":
cf_var.__setitem__.assert_called_once_with(slice(None), data)
else:
assert data_form == "emulateddata"
- assert cf_var._data_array is data
+ assert cf_var.emulated_data_array is data
@pytest.mark.parametrize("is_realdata", [True, False], ids=["realdata", "lazydata"])
@pytest.mark.parametrize("is_string", [True, False], ids=["string", "numeric"])
@@ -135,13 +133,21 @@ def test_data_save_emulated_data_array(
)
if is_string:
- contained = cf_var._contained_instance
- contained.name = "<mock_contained>"
- contained.dtype = np.dtype("S1")
- contained.dimensions = ("x", "strlen")
- mock_group = mocker.Mock()
- mock_group.dimensions = {"strlen": mocker.Mock(size=5)}
- contained.group.return_value = mock_group
+ # The encoding is described by the file variable beneath the
+ # emulation, which is 'char' where the data written is strings.
+ unencoded = mocker.Mock(
+ dtype=np.dtype("S1"),
+ dimensions=("x", "strlen"),
+ # None is what an absent _Encoding attribute yields: the
+ # default encoding, and no "unsupported encoding" warning.
+ _Encoding=None,
+ )
+ # .name is a Mock constructor keyword, so it has to be set after.
+ unencoded.name = "<mock_unencoded>"
+ unencoded.group.return_value = mocker.Mock(
+ dimensions={"strlen": mocker.Mock(size=5)}
+ )
+ cf_var.unencoded_variable = unencoded
ifnbd = "iris.fileformats.netcdf._bytecoding_datasets."
mock_encode = mocker.patch(ifnbd + "encode_stringarray_as_bytearray")
@@ -154,16 +160,16 @@ def test_data_save_emulated_data_array(
assert len(saver._nczarr_writes) == 0
if not is_string:
- assert cf_var._data_array is data
+ assert cf_var.emulated_data_array is data
else:
if is_realdata:
assert mock_encode.call_count == 1
- assert cf_var._data_array is mock_encode.return_value
+ assert cf_var.emulated_data_array is mock_encode.return_value
call_args = mock_encode.call_args_list[0][0]
assert call_args[0] is data
else:
assert mock_mapblocks.call_count == 1
- assert cf_var._data_array is mock_mapblocks.return_value
+ assert cf_var.emulated_data_array is mock_mapblocks.return_value
call_args = mock_mapblocks.call_args_list[0][0]
assert call_args[0] is mock_encode
assert call_args[1] is data
@@ -201,5 +207,7 @@ def test_exit_flushes_nczarr_writes(self, data_form, mocker, tmp_path):
saver.__exit__(None, None, None)
store_patch.assert_called_once_with([source], [target])
- saver._dataset.sync.assert_called_once_with()
- saver._dataset.close.assert_called_once_with()
+ # Saver._dataset is a NetCDFDataset now, so the mock that records the
+ # calls is the netCDF dataset it opened, one level down.
+ saver._dataset.dataset.sync.assert_called_once_with()
+ saver._dataset.dataset.close.assert_called_once_with()
diff --git a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py
new file mode 100644
index 0000000000..25bf3cb6f5
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py
@@ -0,0 +1,273 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Tests for saving into a dataset the caller opened, or is only pretending to.
+
+``iris.fileformats.netcdf.save(cube, dataset, compute=False)`` accepts an
+open dataset in place of a path. Two kinds of caller do this:
+
+* one holding a real, open netCDF4 dataset, who wants Iris to add to it;
+* one holding an object that merely *looks* like a netCDF4 dataset, and
+ collects what Iris writes rather than storing it. This is how
+ https://github.com/SciTools/ncdata translates between Iris and Xarray, added
+ for https://github.com/SciTools/iris/issues/4994.
+
+The second is the reason ``Saver`` may not assume its dataset is netCDF4,
+and it is entirely untested elsewhere.
+
+Saving through a ``CFDataset`` asks an emulating object for three members
+that the saver never used to reach for. All three are public netCDF4 API,
+and all three are now required of an emulator:
+
+* ``Dimension.size`` - the CF layer reads dimension lengths through it.
+ ``__len__`` is not an alternative, as ``_EmulatedDimension`` explains.
+* ``Dataset.ncattrs()`` - read once, as the dataset is wrapped.
+* ``Variable.ncattrs()`` - read once per variable, as each is wrapped.
+
+They are marked below where the emulators define them. This is an accepted
+consequence of the change, not an oversight: there is deliberately no
+fallback for an emulator that lacks them.
+
+"""
+
+import dask
+import dask.array as da
+import numpy as np
+import pytest
+
+import iris
+from iris.coords import DimCoord
+from iris.cube import Cube
+import iris.fileformats.netcdf as inetcdf
+from iris.fileformats.netcdf import _thread_safe_nc as threadsafe_nc
+
+
+@pytest.fixture(autouse=True)
+def split_attrs():
+ # Avoid the legacy-attribute deprecation warning; irrelevant here.
+ with iris.FUTURE.context(save_split_attrs=True):
+ yield
+
+
+@pytest.fixture
+def cube():
+ cube = Cube(
+ np.arange(6.0).reshape(2, 3),
+ standard_name="air_temperature",
+ units="K",
+ var_name="air",
+ )
+ cube.add_dim_coord(
+ DimCoord(np.arange(2.0), standard_name="latitude", units="degrees"), 0
+ )
+ cube.add_dim_coord(
+ DimCoord(np.arange(3.0), standard_name="longitude", units="degrees"), 1
+ )
+ return cube
+
+
+class TestRealDataset:
+ """A dataset the caller opened, and closes themselves."""
+
+ def test_save_into_an_open_dataset(self, cube, tmp_path):
+ path = tmp_path / "user.nc"
+ dataset = threadsafe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ delayed = inetcdf.save(cube, dataset, compute=False)
+ # The caller owns the dataset, so the caller closes it - and the
+ # delayed writes only resolve once they have.
+ dataset.close()
+ dask.compute(delayed)
+
+ result = iris.load_cube(path)
+ assert result.standard_name == "air_temperature"
+ assert result.units == "K"
+ assert np.array_equal(result.data, cube.data)
+ assert [coord.name() for coord in result.coords()] == [
+ "latitude",
+ "longitude",
+ ]
+
+ def test_saver_does_not_close_what_it_was_given(self, cube, tmp_path):
+ path = tmp_path / "user.nc"
+ dataset = threadsafe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ inetcdf.save(cube, dataset, compute=False)
+ assert dataset.isopen()
+ dataset.close()
+
+ def test_deferred_string_writes_are_encoded(self, tmp_path):
+ """A deferred unicode write into a borrowed dataset must reach the file intact.
+
+ ``NetCDFDataset.from_existing`` only wraps a dataset that lacks
+ ``THREAD_SAFE_FLAG``, so a borrowed ``DatasetWrapper`` keeps plain,
+ unencoded variables - and a write handle taken from one of those must
+ encode all the same, because writes have no selectable string
+ encoding. Asserting on the *file*, not on the handle's class: a handle
+ of the right class that still wrote the wrong bytes would pass a
+ type assertion. Unfixed, ``["abc", "def"]`` read back as
+ ``["aaa", "ddd"]``.
+ """
+ path = tmp_path / "strings.nc"
+ strings = np.array(["abc", "def"], dtype="U3")
+ # Lazy, and chunked, so that the save really is deferred to da.store.
+ cube = Cube(da.from_array(strings, chunks=1), long_name="strings")
+
+ dataset = threadsafe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ delayed = inetcdf.save(cube, dataset, compute=False)
+ dataset.close()
+ dask.compute(delayed)
+
+ result = iris.load_cube(path)
+ assert result.dtype.kind == "U"
+ np.testing.assert_array_equal(result.data, strings)
+
+ def test_compute_true_is_refused(self, cube, tmp_path):
+ path = tmp_path / "user.nc"
+ dataset = threadsafe_nc.DatasetWrapper(path, mode="w", format="NETCDF4")
+ with pytest.raises(ValueError, match="Cannot save to a user-provided dataset"):
+ inetcdf.save(cube, dataset, compute=True)
+ dataset.close()
+
+
+class _EmulatedDimension:
+ def __init__(self, size):
+ self._size = size
+ # NEWLY REQUIRED OF AN EMULATOR, 1 of 3 - see the module docstring.
+ # netCDF4.Dimension.size, which is how the length is read back. len()
+ # is not an alternative: the emulator is put inside a _thread_safe_nc
+ # wrapper, whose __getattr__ forwards named members but is never
+ # consulted for the len() protocol.
+ self.size = size
+
+ def __len__(self):
+ return self._size
+
+ def isunlimited(self):
+ return self._size is None
+
+
+class _EmulatedVariable:
+ """The least a netCDF4.Variable emulator can be and still be written to.
+
+ ``_data_array`` is the whole point: an emulator receives its data by
+ having this attribute set, never by ``__setitem__``. ``__setitem__``
+ raises here so that a regression shows up as a failure rather than as
+ data quietly going nowhere.
+
+ """
+
+ def __init__(self, name, datatype, dimensions, shape):
+ self.name = name
+ self.datatype = np.dtype(datatype)
+ self.dtype = self.datatype
+ self.dimensions = tuple(dimensions)
+ self.shape = shape
+ self.size = int(np.prod(shape)) if shape else 1
+ self._data_array = None
+ self._attrs = {}
+
+ def setncattr(self, name, value):
+ self._attrs[name] = value
+
+ def getncattr(self, name):
+ return self._attrs[name]
+
+ def ncattrs(self):
+ # NEWLY REQUIRED OF AN EMULATOR, 2 of 3 - see the module docstring.
+ # Read once, as the variable is wrapped for the attribute mapping.
+ return list(self._attrs)
+
+ def chunking(self):
+ return "contiguous"
+
+ def __setitem__(self, keys, values):
+ raise AssertionError("An emulated variable must receive _data_array.")
+
+
+class _EmulatedDataset:
+ """A netCDF4.Dataset emulator, in the shape ncdata presents.
+
+ Deliberately *not* a ``_thread_safe_nc`` wrapper and deliberately without
+ ``THREAD_SAFE_FLAG``, so that the wrapping branch of
+ ``NetCDFDataset.from_existing`` is the one under test.
+
+ """
+
+ def __init__(self):
+ self.variables = {}
+ self.dimensions = {}
+ self.file_format = "NETCDF4"
+ self._attrs = {}
+ self._open = True
+
+ def createDimension(self, name, size):
+ self.dimensions[name] = _EmulatedDimension(size)
+ return self.dimensions[name]
+
+ def createVariable(self, name, datatype, dimensions=(), **kwargs):
+ shape = tuple(len(self.dimensions[name_]) for name_ in dimensions)
+ variable = _EmulatedVariable(name, datatype, dimensions, shape)
+ self.variables[name] = variable
+ return variable
+
+ def setncattr(self, name, value):
+ self._attrs[name] = value
+
+ def getncattr(self, name):
+ return self._attrs[name]
+
+ def ncattrs(self):
+ # NEWLY REQUIRED OF AN EMULATOR, 3 of 3 - see the module docstring.
+ # Read once, as the dataset is wrapped for the attribute mapping.
+ return list(self._attrs)
+
+ def sync(self):
+ pass
+
+ def close(self):
+ self._open = False
+
+ def isopen(self):
+ return self._open
+
+ def filepath(self):
+ return "<emulated>"
+
+ def set_auto_chartostring(self, onoff):
+ pass
+
+
+class TestEmulatedDataset:
+ """An object that only looks like a dataset, as ncdata supplies."""
+
+ @pytest.fixture
+ def written(self, cube):
+ dataset = _EmulatedDataset()
+ inetcdf.save(cube, dataset, compute=False)
+ return dataset
+
+ def test_variables_are_created(self, written):
+ assert sorted(written.variables) == ["air", "latitude", "longitude"]
+
+ def test_dimensions_are_created(self, written):
+ assert {name: len(dim) for name, dim in written.dimensions.items()} == {
+ "latitude": 2,
+ "longitude": 3,
+ }
+
+ def test_attributes_reach_the_emulator_as_bytes(self, written):
+ # _bytes_if_ascii: an ASCII string attribute is offered as bytes, so
+ # that netCDF4 gives it type NC_CHAR. An emulator sees the same.
+ assert written.variables["air"]._attrs == {
+ "standard_name": b"air_temperature",
+ "units": b"K",
+ }
+ assert written._attrs == {"Conventions": b"CF-1.7"}
+
+ def test_data_arrives_as_a_data_array(self, written, cube):
+ # Not via __setitem__, which the emulated variable refuses.
+ assert np.array_equal(written.variables["air"]._data_array, cube.data)
+ assert np.array_equal(written.variables["latitude"]._data_array, np.arange(2.0))
+
+ def test_dimensions_are_recorded_on_the_variable(self, written):
+ assert written.variables["air"].dimensions == ("latitude", "longitude")
diff --git a/lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py b/lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py
new file mode 100644
index 0000000000..aef910f8bc
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/netcdf/saver/test__relocated_names.py
@@ -0,0 +1,193 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Names that must survive the CF saver's move to iris.fileformats.cf.
+
+Two separate promises. Four public names moved and are still reachable from
+their old homes, warning when they are. Every method of the netCDF Saver kept
+its name, so that code subclassing it keeps overriding what it overrode.
+"""
+
+import inspect
+import warnings
+
+import numpy as np
+import pytest
+
+import iris
+from iris._deprecation import IrisDeprecation
+from iris.cube import Cube
+from iris.fileformats import netcdf
+from iris.fileformats.cf import saver as cf_saver
+from iris.fileformats.netcdf import saver as netcdf_saver
+from iris.loading import LOAD_PROBLEMS
+
+#: The four names spec section 4.8 relocates, with warn-on-use aliases left
+#: behind at both their old homes.
+RELOCATED = [
+ "CFNameCoordMap",
+ "CF_CONVENTIONS_VERSION",
+ "MESH_ELEMENTS",
+ "SPATIO_TEMPORAL_AXES",
+]
+
+#: Method names the netCDF Saver must answer to: those it carried before the
+#: generic half of it became _CFSaver, plus the hooks _CFSaver calls on a
+#: subclass. A name missing from the class is an override, somewhere outside
+#: Iris, that has silently stopped overriding anything.
+SAVER_METHODS = [
+ "_add_ancillary_variables",
+ "_add_aux_coords",
+ "_add_aux_factories",
+ "_add_cell_measures",
+ "_add_dim_coords",
+ "_add_grid_mapping_to_dataset",
+ "_add_inner_related_vars",
+ "_add_mesh",
+ "_backend_write",
+ "_cf_coord_standardised_units",
+ "_create_cf_bounds",
+ "_create_cf_cell_methods",
+ "_create_cf_data_variable",
+ "_create_cf_dimensions",
+ "_create_cf_grid_mapping",
+ "_create_generic_cf_array_var",
+ "_create_mesh",
+ "_create_string_variable",
+ "_ensure_valid_dtype",
+ "_flush_before_close",
+ "_get_dim_names",
+ "_get_element_variable_name",
+ "_get_mesh_variable_name",
+ "_increment_name",
+ "_lazy_stream_data",
+ "_patch_dataset",
+ "_set_cf_var_attributes",
+ "cf_valid_var_name",
+ "check_attribute_compliance",
+ "complete",
+ "delayed_completion",
+ "update_global_attributes",
+ "write",
+]
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+class TestFromTheSaverModule:
+ def test_resolves_to_the_same_object(self, name):
+ with pytest.warns(IrisDeprecation):
+ assert getattr(netcdf_saver, name) is getattr(cf_saver, name)
+
+ def test_the_warning_names_the_new_home(self, name):
+ with pytest.warns(IrisDeprecation, match=f"iris.fileformats.cf.saver.{name}"):
+ getattr(netcdf_saver, name)
+
+ def test_is_discoverable(self, name):
+ # PEP 562 forwards the read but not dir(), so the module says so itself.
+ assert name in dir(netcdf_saver)
+
+
+@pytest.mark.parametrize("name", RELOCATED)
+class TestFromThePackage:
+ def test_resolves_to_the_same_object(self, name):
+ with pytest.warns(IrisDeprecation):
+ assert getattr(netcdf, name) is getattr(cf_saver, name)
+
+ def test_is_discoverable(self, name):
+ assert name in dir(netcdf)
+
+
+class TestNamesThatDoNotMove:
+ @pytest.mark.parametrize("name", ["Saver", "save", "save_mesh"])
+ def test_the_saver_module_still_defines_it(self, name):
+ # Spec section 4.8: netCDF API, and it stays that way. Defined here,
+ # not forwarded, so reading it must not warn.
+ assert name in vars(netcdf_saver)
+
+ def test_an_unknown_name_raises_quietly(self, recwarn):
+ with pytest.raises(AttributeError, match="no attribute"):
+ netcdf_saver.not_a_real_name
+ assert recwarn.list == []
+
+
+class TestSaverMethods:
+ @pytest.mark.parametrize("name", SAVER_METHODS)
+ def test_the_netcdf_saver_still_answers_to_it(self, name):
+ assert callable(getattr(netcdf_saver.Saver, name, None))
+
+ def test_the_netcdf_saver_is_a_cf_saver(self):
+ assert issubclass(netcdf_saver.Saver, cf_saver._CFSaver)
+
+ def test_write_keeps_its_signature(self):
+ # Documented API with defaults users rely on, and save() passes some
+ # of them by position.
+ parameters = inspect.signature(netcdf_saver.Saver.write).parameters
+ assert list(parameters) == [
+ "self",
+ "cube",
+ "local_keys",
+ "unlimited_dimensions",
+ "zlib",
+ "complevel",
+ "shuffle",
+ "fletcher32",
+ "contiguous",
+ "chunksizes",
+ "endian",
+ "least_significant_digit",
+ "packing",
+ "fill_value",
+ ]
+ assert parameters["complevel"].default == 4
+ assert parameters["shuffle"].default is True
+ assert parameters["endian"].default == "native"
+
+
+class TestNothingInsideIrisUsesTheOldNames:
+ """Cover the save and load paths, which is less than the name claims.
+
+ A reader of a relocated name that neither a save nor a load reaches goes
+ unnoticed here. Take a pass as saying those two paths are clean, not the
+ repository.
+ """
+
+ def test_a_save_emits_no_deprecation(self, tmp_path):
+ # The aliases are for code outside Iris. A save that trips one means a
+ # reader inside Iris was left pointing at the old home. Promoting to an
+ # error is safe here: iris.save has no equivalent of LOAD_PROBLEMS, so
+ # an exception reaches the test rather than being recorded.
+ #
+ # save_split_attrs is set because the legacy attribute mode is itself
+ # deprecated, and its warning would fail this test for a reason that
+ # has nothing to do with where the saver lives.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", IrisDeprecation)
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(
+ Cube(np.arange(3.0), var_name="payload"), tmp_path / "quiet.nc"
+ )
+
+ def test_a_load_emits_no_deprecation(self, tmp_path):
+ # Recorded rather than promoted to an error, for the reason
+ # SciTools/iris#7333 found: iris.load captures an exception raised
+ # while building a cube into iris.loading.LOAD_PROBLEMS, so a filter of
+ # "error" drops the offending coordinate and the test passes against an
+ # implementation that is still warning. The LOAD_PROBLEMS assertion is
+ # what keeps a silent load from being a degraded one.
+ path = tmp_path / "quiet.nc"
+ with iris.FUTURE.context(save_split_attrs=True):
+ iris.save(Cube(np.arange(3.0), var_name="payload"), path)
+
+ with warnings.catch_warnings(record=True) as record:
+ warnings.simplefilter("always")
+ cubes = iris.load(path)
+
+ deprecations = [
+ str(warning.message)
+ for warning in record
+ if issubclass(warning.category, IrisDeprecation)
+ ]
+ assert deprecations == []
+ assert LOAD_PROBLEMS.problems_by_file.get(str(path), []) == []
+ assert len(cubes) == 1
diff --git a/lib/iris/tests/unit/fileformats/netcdf/saver/test_save.py b/lib/iris/tests/unit/fileformats/netcdf/saver/test_save.py
index b4ae37f29b..5f8ae73331 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/saver/test_save.py
+++ b/lib/iris/tests/unit/fileformats/netcdf/saver/test_save.py
@@ -10,7 +10,8 @@
import iris
from iris.coords import AuxCoord, DimCoord
from iris.cube import Cube, CubeList
-from iris.fileformats.netcdf import CF_CONVENTIONS_VERSION, Saver, _thread_safe_nc, save
+from iris.fileformats.cf.saver import CF_CONVENTIONS_VERSION
+from iris.fileformats.netcdf import Saver, _thread_safe_nc, save
from iris.tests import _shared_utils
from iris.tests.stock import lat_lon_cube
from iris.tests.stock.mesh import sample_mesh_cube
diff --git a/lib/iris/tests/unit/fileformats/zarr/__init__.py b/lib/iris/tests/unit/fileformats/zarr/__init__.py
new file mode 100644
index 0000000000..2016f9f175
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/__init__.py
@@ -0,0 +1,14 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.zarr`.
+
+``zarr`` is optional, so every module below this one is skipped when it is
+absent. The skip lives here rather than in a ``conftest.py``: the same call
+there raises during collection and is reported as an **error**, not a skip.
+"""
+
+import pytest
+
+pytest.importorskip("zarr", minversion="3.0.8")
diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py b/lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py
new file mode 100644
index 0000000000..66f11b11ae
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/dataset/__init__.py
@@ -0,0 +1,5 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDatasetVariable`."""
diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py
new file mode 100644
index 0000000000..a742e45e84
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset.py
@@ -0,0 +1,199 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDataset`."""
+
+import numpy as np
+import pytest
+import zarr
+
+from iris.fileformats.zarr._dataset import ZarrDataset
+from iris.warnings import IrisCfLoadWarning
+
+from .. import stores
+
+
+@pytest.fixture(params=[2, 3], ids=["v2", "v3"])
+def zarr_format(request):
+ return request.param
+
+
+class Test_opening:
+ def test_opens_and_reports_itself(self, tmp_path, zarr_format):
+ location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format)
+ dataset = ZarrDataset(location)
+
+ assert dataset.location == location
+ assert dataset.format == f"ZARR_V{zarr_format}"
+ assert dataset.mode == "r"
+ assert dataset.closed is False
+ assert sorted(dataset.variables) == ["air_temperature", "grid"]
+ assert dict(dataset.attributes) == stores.GLOBALS
+ assert dataset.storage_settings["zarr_format"] == zarr_format
+
+
+class Test_attributes:
+ def test_nc_properties_is_not_a_cf_attribute(self, tmp_path):
+ """`_NCProperties` is netCDF-c's own bookkeeping, not CF metadata.
+
+ It is the stamp netCDF-c writes onto the root group of every store it
+ creates (confirmed on NCZarr output), reading `version=...,netcdf=...`.
+ netCDF4-python hides it from `Dataset.ncattrs()` by convention, so no
+ cube loaded through netCDF has ever carried it; a cube loaded from the
+ same store through the native Zarr reader must not gain it either, or
+ the two would stop comparing equal over nothing more than provenance.
+ """
+ group = zarr.open_group(store=tmp_path / "nc.zarr", mode="w", zarr_format=3)
+ group.attrs["Conventions"] = "CF-1.7"
+ group.attrs["_NCProperties"] = "version=2,netcdf=4.10.1,nczarr=2.0.0"
+ stores.make_array(group, "v", (2,), "f8", 3, dimensions=("t",))
+
+ dataset = ZarrDataset(str(tmp_path / "nc.zarr"))
+ assert "_NCProperties" in group.attrs
+ assert "_NCProperties" not in dataset.attributes
+ assert dict(dataset.attributes) == {"Conventions": "CF-1.7"}
+
+
+class Test_dimensions:
+ def test_dimensions_are_assembled_from_the_arrays(self, tmp_path, zarr_format):
+ location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format)
+ assert dict(ZarrDataset(location).dimensions) == stores.DIMENSIONS
+
+ def test_a_name_used_at_two_sizes_is_omitted_and_warns(self, tmp_path):
+ """Two arrays may share a dimension name at different lengths.
+
+ The version 3 specification "does not place any restrictions on the use of
+ the same dimension name across multiple arrays", so this is conforming
+ Zarr and must not raise. Merging the two would produce a wrong cube.
+
+ Opening the store is what reports it. The mapping is assembled in
+ `__init__` so that the warning reaches a load, which asks each variable
+ for its own `dimensions` and never asks the dataset for this mapping.
+ """
+ group = zarr.open_group(store=tmp_path / "c.zarr", mode="w", zarr_format=3)
+ stores.make_array(group, "short", (2,), "f8", 3, dimensions=("t",))
+ stores.make_array(group, "long", (5,), "f8", 3, dimensions=("t",))
+
+ with pytest.warns(IrisCfLoadWarning, match="t"):
+ dataset = ZarrDataset(str(tmp_path / "c.zarr"))
+
+ dimensions = dict(dataset.dimensions)
+ assert "t" not in dimensions
+ # The arrays keep the name in their own dimensions: nothing is renamed or
+ # dropped, only the dataset-wide mapping declines to link them.
+ assert dataset.variables["short"].dimensions == ("t",)
+ assert dataset.variables["long"].dimensions == ("t",)
+
+ def test_anonymous_dimensions_are_not_in_the_mapping(self, tmp_path):
+ group = zarr.open_group(store=tmp_path / "a.zarr", mode="w", zarr_format=3)
+ stores.make_array(group, "named", (2,), "f8", 3, dimensions=("t",))
+ stores.make_array(group, "unnamed", (4,), "f8", 3)
+
+ dataset = ZarrDataset(str(tmp_path / "a.zarr"))
+ assert dict(dataset.dimensions) == {"t": 2}
+
+
+class Test_close:
+ def test_close_is_idempotent(self, tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ dataset = ZarrDataset(location)
+ dataset.close()
+ dataset.close()
+ assert dataset.closed is True
+
+ def test_context_manager_closes(self, tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ dataset = ZarrDataset(location)
+ with dataset as entered:
+ assert entered is dataset
+ assert dataset.closed is True
+
+
+class Test_groups:
+ def test_group_keyword_selects_a_subgroup(self, tmp_path):
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ sub = root.create_group("measurements/r10m")
+ stores.make_array(sub, "b02", (2,), "u2", 3, dimensions=("p",))
+
+ dataset = ZarrDataset(str(tmp_path / "n.zarr"), group="measurements/r10m")
+ assert list(dataset.variables) == ["b02"]
+ assert dataset.location.endswith("measurements/r10m")
+
+ def test_subgroup_names_lists_only_the_groups(self, tmp_path):
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ root.create_group("conditions")
+ root.create_group("measurements")
+ stores.make_array(root, "v", (2,), "f8", 3)
+
+ dataset = ZarrDataset(str(tmp_path / "n.zarr"))
+ assert sorted(dataset.subgroup_names) == ["conditions", "measurements"]
+
+ def test_nested_groups_are_not_walked(self, tmp_path):
+ """Iris loads one group per call, like xarray.
+
+ A Zarr store can legitimately hold unrelated datasets, and merging them
+ into one CubeList would be wrong.
+ """
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ stores.make_array(root, "top", (2,), "f8", 3)
+ stores.make_array(root.create_group("sub"), "inner", (2,), "f8", 3)
+
+ assert list(ZarrDataset(str(tmp_path / "n.zarr")).variables) == ["top"]
+
+ def test_a_missing_group_raises(self, tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ with pytest.raises(zarr.errors.GroupNotFoundError):
+ ZarrDataset(location, group="nope")
+
+
+class Test_the_write_surface:
+ def test_the_write_surface_refuses(self, tmp_path):
+ # Saving to Zarr is not implemented. The members exist because the
+ # interface declares them; refusing is the honest answer.
+ dataset = ZarrDataset(stores.make_store(tmp_path / "s.zarr"))
+ with pytest.raises(NotImplementedError, match="Saving"):
+ dataset.create_dimension("t", 3)
+ with pytest.raises(NotImplementedError, match="Saving"):
+ dataset.create_variable("v", np.dtype("f4"), ("t",))
+ with pytest.raises(NotImplementedError, match="Saving"):
+ dataset.finalise()
+
+ def test_a_write_mode_refuses(self, tmp_path):
+ stores.make_store(tmp_path / "s.zarr")
+ with pytest.raises(NotImplementedError, match="Saving"):
+ ZarrDataset(str(tmp_path / "s.zarr"), mode="w")
+
+
+class Test_supports_dtype:
+ @pytest.mark.parametrize(
+ "dtype", ["i1", "i2", "i4", "i8", "u1", "u2", "u4", "u8", "f4", "f8", "c8"]
+ )
+ def test_supports_every_dtype(self, tmp_path, zarr_format, dtype):
+ # The netCDF 3 formats have no 64-bit or unsigned integer type, which is
+ # what makes supports_dtype worth asking. Zarr has every one of them, in
+ # both versions, so the Zarr answer carries no format test.
+ dataset = ZarrDataset(
+ stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format)
+ )
+ assert dataset.supports_dtype(np.dtype(dtype)) is True
+
+
+class Test_locations:
+ def test_an_in_memory_store_is_just_another_location(self, tmp_path):
+ """A store need not be a directory. zarr-python decides what it is."""
+ memory = zarr.storage.MemoryStore()
+ group = zarr.open_group(store=memory, mode="w", zarr_format=3)
+ stores.make_array(group, "v", (2,), "f8", 3, dimensions=("t",))
+
+ dataset = ZarrDataset(memory)
+ assert list(dataset.variables) == ["v"]
+
+ def test_a_missing_store_raises_zarrs_own_error(self, tmp_path):
+ with pytest.raises(FileNotFoundError):
+ ZarrDataset(str(tmp_path / "nothing.zarr"))
+
+ def test_a_directory_that_is_not_a_store_raises(self, tmp_path):
+ (tmp_path / "plain").mkdir()
+ with pytest.raises(zarr.errors.GroupNotFoundError):
+ ZarrDataset(str(tmp_path / "plain"))
diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py
new file mode 100644
index 0000000000..268273bb28
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDatasetVariable.py
@@ -0,0 +1,247 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :class:`iris.fileformats.zarr._dataset.ZarrDatasetVariable`."""
+
+import json
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+import zarr
+
+from iris.exceptions import TranslationError
+from iris.fileformats.zarr._dataset import ZarrDatasetVariable
+from iris.warnings import IrisCfLoadWarning
+
+from .. import stores
+
+
+def wrap(array, location="store.zarr"):
+ """Return the ZarrDatasetVariable over `array`, naming its own format."""
+ zarr_format = array.metadata.zarr_format
+ return ZarrDatasetVariable(array, location, f"ZARR_V{zarr_format}")
+
+
+@pytest.fixture(params=[2, 3], ids=["v2", "v3"])
+def zarr_format(request):
+ return request.param
+
+
+@pytest.fixture
+def group(tmp_path, zarr_format):
+ return zarr.open_group(
+ store=tmp_path / "probe.zarr", mode="w", zarr_format=zarr_format
+ )
+
+
+def test_named_dimensions(group, zarr_format):
+ array = stores.make_array(
+ group, "v", (3, 2), "f8", zarr_format, dimensions=("y", "x")
+ )
+ assert wrap(array).dimensions == ("y", "x")
+
+
+def test_unnamed_dimensions_are_anonymous(group, zarr_format):
+ array = stores.make_array(group, "v", (3, 2), "f8", zarr_format)
+ assert wrap(array).dimensions == (None, None)
+
+
+def test_scalar_has_no_dimensions(group, zarr_format):
+ array = stores.make_array(group, "v", (), "i4", zarr_format)
+ assert wrap(array).dimensions == ()
+
+
+def test_partially_named_dimensions_version_3(tmp_path):
+ # The version 3 core specification: "an unnamed dimension is indicated by
+ # the null object". A store may name some dimensions and not others.
+ group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3)
+ array = group.create_array(
+ "v", shape=(3, 2), dtype="f8", dimension_names=("y", None)
+ )
+ assert wrap(array).dimensions == ("y", None)
+
+
+def test_repeated_name_within_one_array(tmp_path):
+ # "It is recommended but not required that all non-null dimension names
+ # are distinct." Recommended is not required, so this must not raise.
+ group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3)
+ array = group.create_array(
+ "v", shape=(2, 2), dtype="f8", dimension_names=("d", "d")
+ )
+ assert wrap(array).dimensions == ("d", "d")
+
+
+def test_an_unsupported_zarr_format_raises():
+ """Iris names the Zarr versions it reads, and refuses others.
+
+ The array is a stand-in because zarr-python cannot produce the input,
+ which is the point of the companion test below.
+ """
+ array = SimpleNamespace(
+ ndim=1,
+ basename="v",
+ attrs={},
+ metadata=SimpleNamespace(zarr_format=4, dimension_names=("t",)),
+ )
+ with pytest.raises(TranslationError, match="declares Zarr version 4"):
+ wrap(array)
+
+
+def test_zarr_python_rejects_an_unsupported_zarr_format(tmp_path):
+ """zarr-python is what makes the branch above unreachable in practice.
+
+ It parses `zarr_format` as a `Literal` and raises on anything else -
+ `expected Literal[3], got 4` for an array in a version 3 group. If it
+ ever widens that, the branch above starts earning its keep.
+ """
+ group = zarr.open_group(store=tmp_path / "v4.zarr", mode="w", zarr_format=3)
+ stores.make_array(group, "v", (2,), "f8", 3, dimensions=("t",))
+
+ metadata_path = tmp_path / "v4.zarr" / "v" / "zarr.json"
+ metadata = json.loads(metadata_path.read_text())
+ metadata["zarr_format"] = 4
+ metadata_path.write_text(json.dumps(metadata))
+
+ with pytest.raises(Exception, match="zarr_format"):
+ zarr.open_group(store=tmp_path / "v4.zarr", mode="r")["v"]
+
+
+def test_wrong_length_dimension_names_warns_and_loads_anonymous(tmp_path):
+ """A dimension-name count that disagrees with `ndim` is a malformation.
+
+ Version 2 only: the version 3 core specification says `dimension_names`
+ "must" be the same length as `shape`, and zarr-python enforces that on
+ every read, not just on write - a hand-edited `zarr.json` with the wrong
+ length raises `ValueError` from `Array.open` itself, before Iris sees an
+ array at all. `_ARRAY_DIMENSIONS` carries no such validation, so this is
+ reachable only through a version 2 store.
+ """
+ group = zarr.open_group(store=tmp_path / "bad.zarr", mode="w", zarr_format=2)
+ array = stores.make_array(group, "v", (3, 2), "f8", 2, dimensions=("y", "x"))
+ array.attrs[stores.ARRAY_DIMENSIONS] = ["y"]
+
+ with pytest.warns(IrisCfLoadWarning, match="dimension_names"):
+ dimensions = wrap(array).dimensions
+ assert dimensions == (None, None)
+
+
+def test_plain_properties(tmp_path, zarr_format):
+ location = stores.make_store(tmp_path / "s.zarr", zarr_format=zarr_format)
+ array = zarr.open_group(store=location, mode="r")["air_temperature"]
+ variable = wrap(array, location)
+
+ assert variable.name == "air_temperature"
+ assert variable.location == location
+ assert variable.format == f"ZARR_V{zarr_format}"
+ assert variable.shape == stores.DATA.shape
+ assert variable.dtype == np.dtype("f8")
+ assert variable.size == stores.DATA.size
+ assert variable.ndim == stores.DATA.ndim
+ assert len(variable) == stores.DATA.shape[0]
+ np.testing.assert_array_equal(variable[:], stores.DATA)
+ np.testing.assert_array_equal(variable[1], stores.DATA[1])
+
+
+def test_fill_value_is_the_storage_field(tmp_path, zarr_format):
+ """`fill_value` is what unwritten storage reads as, not `_FillValue`.
+
+ zarr-python defaults the field to zero, so an array that declares no CF
+ sentinel still reports one here. Reading it as a mask source is what
+ masks every valid zero.
+ """
+ group = zarr.open_group(
+ store=tmp_path / "f.zarr", mode="w", zarr_format=zarr_format
+ )
+ array = stores.make_array(group, "v", (3,), "i2", zarr_format)
+ assert wrap(array).fill_value == 0
+
+ declared = stores.make_array(group, "w", (3,), "i2", zarr_format, fill_value=-999)
+ assert wrap(declared).fill_value == -999
+
+
+def test_chunking_is_the_inner_chunk_not_the_shard(tmp_path):
+ """A sharded array reports `Array.chunks`.
+
+ Zarr's sharding codec is indexed: a reader fetches the shard index, then
+ byte ranges for only the inner chunks it needs. The inner chunk is the
+ independently decodable unit.
+ """
+ group = zarr.open_group(store=tmp_path / "sh.zarr", mode="w", zarr_format=3)
+ array = group.create_array(
+ "v", shape=(1024,), dtype="i4", chunks=(8,), shards=(1024,)
+ )
+ variable = wrap(array)
+ assert variable.chunking == (8,)
+ assert variable.storage_settings["shards"] == (1024,)
+
+
+def test_storage_settings_reports_the_store(tmp_path, zarr_format):
+ location = stores.make_store(tmp_path / "e.zarr", zarr_format=zarr_format)
+ array = zarr.open_group(store=location, mode="r")["air_temperature"]
+ storage_settings = wrap(array, location).storage_settings
+ assert storage_settings["chunks"] == array.chunks
+ assert storage_settings["zarr_format"] == zarr_format
+
+
+def test_array_dimensions_is_not_a_cf_attribute(tmp_path):
+ """`_ARRAY_DIMENSIONS` is storage, so it must not reach `attributes`.
+
+ It is how a version 2 store records dimension names. Every attribute the
+ loading rules do not consume reaches `cube.attributes`, so leaving it in
+ would put it on every cube loaded from a version 2 store, and make that
+ cube unequal to the same data loaded from version 3.
+ """
+ group = zarr.open_group(store=tmp_path / "v2.zarr", mode="w", zarr_format=2)
+ array = stores.make_array(
+ group,
+ "v",
+ (3, 2),
+ "f8",
+ 2,
+ dimensions=("y", "x"),
+ attributes={"units": "K"},
+ )
+ variable = wrap(array)
+
+ assert variable.dimensions == ("y", "x")
+ assert stores.ARRAY_DIMENSIONS in array.attrs
+ assert stores.ARRAY_DIMENSIONS not in variable.attributes
+ assert dict(variable.attributes) == {"units": "K"}
+
+
+def test_version_2_and_version_3_present_the_same_attributes(tmp_path):
+ v2 = stores.make_store(tmp_path / "a2.zarr", zarr_format=2)
+ v3 = stores.make_store(tmp_path / "a3.zarr", zarr_format=3)
+ from_2 = zarr.open_group(store=v2, mode="r")["air_temperature"]
+ from_3 = zarr.open_group(store=v3, mode="r")["air_temperature"]
+ assert dict(wrap(from_2, v2).attributes) == dict(wrap(from_3, v3).attributes)
+
+
+def test_structured_attribute_values_are_carried_unchanged(tmp_path):
+ """A JSON object or array attribute is carried, not interpreted.
+
+ The version 3 specification allows an attribute to be any JSON literal,
+ and CF cannot express a nested one at any rank. Two independent producers
+ write them - NOAA GFS gives `valid_time` a `statistics_approximate`
+ object, and the ESA EOPF product root carries `stac_discovery`.
+ A flat numeric array is an ordinary CF vector attribute and is not special.
+ """
+ group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ (2,),
+ "f8",
+ 3,
+ attributes={
+ "statistics_approximate": {"min": 0.0, "max": 1.0},
+ "valid_range": [0.0, 1.0],
+ "nested_list": [[1, 2], [3, 4]],
+ },
+ )
+ attributes = wrap(array).attributes
+ assert attributes["statistics_approximate"] == {"min": 0.0, "max": 1.0}
+ assert attributes["valid_range"] == [0.0, 1.0]
+ assert attributes["nested_list"] == [[1, 2], [3, 4]]
diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py
new file mode 100644
index 0000000000..54a9ae8493
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_ZarrDataset_contract.py
@@ -0,0 +1,34 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The shared CFDataset read contract, asked of the Zarr implementation.
+
+Only the read half: the Zarr write path does not exist, and `ZarrDataset`
+refuses every member of the write surface. The write half of the contract
+arrives with the saver.
+"""
+
+import pytest
+
+from iris.fileformats.zarr._dataset import ZarrDataset
+from iris.tests.unit.fileformats.cf.dataset import contract
+
+from .. import stores
+
+
+class TestZarrDatasetVersion3(contract.CFDatasetReadContract):
+ chunking_key = "chunks"
+
+ @pytest.fixture
+ def readable(self, tmp_path):
+ return ZarrDataset(stores.make_store(tmp_path / "contract.zarr"))
+
+
+class TestZarrDatasetVersion2(contract.CFDatasetReadContract):
+ chunking_key = "chunks"
+
+ @pytest.fixture
+ def readable(self, tmp_path):
+ location = stores.make_store(tmp_path / "contract.zarr", zarr_format=2)
+ return ZarrDataset(location)
diff --git a/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py b/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py
new file mode 100644
index 0000000000..d3c9f6990a
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/dataset/test_read_data.py
@@ -0,0 +1,424 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :meth:`ZarrDatasetVariable.read_data`."""
+
+import dask
+import dask.array as da
+from dask.array.core import normalize_chunks
+import dask.config
+import numpy as np
+import pytest
+import zarr
+
+from iris import _lazy_data
+from iris.fileformats.cf import loader as cf_loader
+from iris.fileformats.zarr._dataset import ZarrDataset
+
+from .. import stores
+
+
+def variable(location, name="air_temperature"):
+ return ZarrDataset(location).variables[name]
+
+
+def policy():
+ """The chunking policy a loader outside any CHUNK_CONTROL supplies."""
+ return None, None
+
+
+def dask_policy():
+ """The chunking policy CHUNK_CONTROL.as_dask() supplies."""
+ return "auto", None
+
+
+def key_of(location, name="v"):
+ return ZarrDataset(location).variables[name].cache_key
+
+
+class CountingStore(zarr.storage.WrapperStore):
+ """Wrap a store, adding every byte it returns to a caller-owned tally.
+
+ zarr-python copies a store when it opens it read-only, through
+ `with_read_only` and `_with_store`, so a tally kept on the instance is
+ never the one the live store writes to, and the test passes vacuously.
+ """
+
+ def __init__(self, store, tally):
+ super().__init__(store=store)
+ self.tally = tally
+
+ def _with_store(self, store):
+ return type(self)(store, self.tally)
+
+ async def get(self, key, prototype, byte_range=None):
+ buffer = await self._store.get(key, prototype, byte_range)
+ if buffer is not None:
+ self.tally.append(len(buffer))
+ return buffer
+
+ async def get_partial_values(self, prototype, key_ranges):
+ buffers = await self._store.get_partial_values(prototype, key_ranges)
+ self.tally.extend(len(b) for b in buffers if b is not None)
+ return buffers
+
+
+def bytes_to_read_first_chunk(location, dask_chunks):
+ """Return the bytes fetched reading the first eight values of "v"."""
+ tally = []
+ store = CountingStore(zarr.storage.LocalStore(location), tally)
+ array = zarr.open_group(store=store, mode="r")["v"]
+ tally.clear()
+ lazy = da.from_array(array, chunks=dask_chunks, meta=np.empty((0,), "i4"))
+ lazy = da.map_blocks(lambda block: np.ma.masked_array(block), lazy, dtype="i4")
+ lazy[:8].compute()
+ return sum(tally)
+
+
+def packed_store(tmp_path, shape, chunks):
+ """Write an int8 array that unpacks to float64, and return its location."""
+ location = str(tmp_path / "p.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ shape,
+ "i1",
+ 3,
+ dimensions=tuple("yx"[: len(shape)]),
+ chunks=chunks,
+ attributes={"scale_factor": 0.5},
+ )
+ array[:] = 1
+ return location
+
+
+class Test_laziness:
+ def test_a_small_variable_is_read_whole(self, tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ result = variable(location).read_data(policy)
+ assert not isinstance(result, da.Array)
+ np.testing.assert_array_equal(np.asarray(result), stores.DATA)
+
+ def test_a_small_variable_never_consults_the_policy(self, tmp_path):
+ """Under CHUNK_CONTROL.from_file() the policy raises for an unchunked store.
+
+ A variable small enough to read whole must reach its data without asking,
+ or `from_file` starts rejecting small cubes that load today.
+ """
+
+ def raising_policy():
+ raise AssertionError("chunking policy consulted for a small variable")
+
+ location = stores.make_store(tmp_path / "s.zarr")
+ variable(location).read_data(raising_policy)
+
+ def test_a_large_variable_is_lazy(self, tmp_path, mocker):
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ location = stores.make_store(tmp_path / "s.zarr")
+ result = variable(location).read_data(policy)
+ assert isinstance(result, da.Array)
+ np.testing.assert_array_equal(result.compute(), stores.DATA)
+
+ def test_a_plain_array_gets_an_unmasked_meta(self, tmp_path, mocker):
+ # No decode layer, so no masked meta: a masked meta with nothing beneath
+ # it is a meta that lies about its own graph.
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ location = stores.make_store(tmp_path / "s.zarr")
+ result = variable(location).read_data(policy)
+ assert not isinstance(result._meta, np.ma.MaskedArray)
+
+ def test_a_decoded_array_gets_a_masked_meta(self, tmp_path, mocker):
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ group = zarr.open_group(store=tmp_path / "m.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ (4,),
+ "i2",
+ 3,
+ dimensions=("t",),
+ attributes={"_FillValue": -999},
+ )
+ array[:] = [-999, 1, 2, 3]
+
+ result = ZarrDataset(str(tmp_path / "m.zarr")).variables["v"].read_data(policy)
+ assert isinstance(result._meta, np.ma.MaskedArray)
+ realised = result.compute()
+ assert realised.mask.tolist() == [True, False, False, False]
+
+
+class Test_cache_key:
+ def test_the_key_is_stable_across_a_reopen(self, tmp_path):
+ group = zarr.open_group(store=tmp_path / "k.zarr", mode="w", zarr_format=3)
+ stores.make_array(group, "v", (4,), "i4", 3, dimensions=("t",))
+ location = str(tmp_path / "k.zarr")
+ assert key_of(location) == key_of(location)
+
+ def test_the_key_changes_when_the_shape_does(self, tmp_path):
+ """A store rewritten in place keeps its address and changes its contents.
+
+ Keyed on the address alone, an array reopened after its shape changed
+ from (4,) to (6,) returns the cached four values, silently truncating -
+ and that is weaker than the netCDF key it replaces, since
+ repr(NetCDFDataProxy) already carries shape and dtype.
+ """
+ location = str(tmp_path / "k.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ stores.make_array(group, "v", (4,), "i4", 3, dimensions=("t",), chunks=(2,))
+ before = key_of(location)
+
+ group = zarr.open_group(store=location, mode="a", zarr_format=3)
+ group.create_array(
+ "v",
+ shape=(6,),
+ dtype="i4",
+ chunks=(2,),
+ dimension_names=("t",),
+ overwrite=True,
+ )
+ assert key_of(location) != before
+
+ def test_the_key_changes_when_the_fill_value_does(self, tmp_path):
+ location = str(tmp_path / "k.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ group.create_array("v", shape=(4,), dtype="i4", fill_value=-1)
+ before = key_of(location)
+
+ group = zarr.open_group(store=location, mode="a", zarr_format=3)
+ group.create_array("v", shape=(4,), dtype="i4", fill_value=-2, overwrite=True)
+ assert key_of(location) != before
+
+ @pytest.mark.parametrize(
+ "change",
+ [
+ {"shape": (8,)},
+ {"chunks": (4,)},
+ {"shards": (8,)},
+ {"dtype": "f8"},
+ {"fill_value": -7},
+ {"dimension_names": ("other",)},
+ ],
+ )
+ def test_the_key_is_sensitive_to_every_metadata_field(self, tmp_path, change):
+ """`Array.metadata.to_dict()` is zarr-python's rendering, not a guarantee.
+
+ Nothing in the Zarr specification defines what that method returns, so
+ these are what fail if a release stops carrying a field the key needs.
+ """
+ base = dict(
+ shape=(4,),
+ dtype="i4",
+ chunks=(2,),
+ fill_value=-1,
+ dimension_names=("t",),
+ shards=None,
+ )
+ location = str(tmp_path / "k.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ group.create_array("v", **base)
+ before = key_of(location)
+
+ group = zarr.open_group(store=location, mode="a", zarr_format=3)
+ group.create_array("v", **{**base, **change}, overwrite=True)
+ assert key_of(location) != before
+
+ def test_the_key_is_sensitive_to_attributes(self, tmp_path):
+ location = str(tmp_path / "k.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ array = group.create_array("v", shape=(4,), dtype="i4")
+ before = key_of(location)
+ array.attrs["units"] = "K"
+ assert key_of(location) != before
+
+ def test_two_cubes_sharing_an_array_share_one_dask_array(self, tmp_path, mocker):
+ """What the cache is for: many cubes in one store sharing a coordinate.
+
+ Comparing the two Dask names proves nothing on its own: Dask tokenises a
+ zarr.Array deterministically, so two independently opened arrays get the
+ same name whether or not `read_data` passes a cache key at all. What has
+ to be shown is that the first read puts an entry in the process-wide
+ cache and the second read comes back out of it.
+
+ The cache is replaced for the test rather than measured in place, because
+ it is process-wide and holds only its last hundred entries - a test that
+ counted the real one would depend on what ran before it.
+ """
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ cache = _lazy_data.LRUCache(100)
+ mocker.patch.object(_lazy_data, "CACHE", cache)
+ location = stores.make_store(tmp_path / "s.zarr")
+
+ first = variable(location).read_data(policy)
+ assert len(cache._cache) == 1, "the first read did not populate the cache"
+
+ second = variable(location).read_data(policy)
+ assert len(cache._cache) == 1, "the second read built a second array"
+ assert first.name == second.name
+
+
+class Test_sharding:
+ def test_a_sharded_array_is_read_by_the_inner_chunk(self, tmp_path):
+ """Shard-sized Dask blocks fetch the whole shard, because of the layer.
+
+ Bare Dask slicing can push a slice down into the array's own __getitem__,
+ but a map_blocks layer in between cannot, so the block is materialised in
+ full. The absolute byte counts are zarr-version-dependent; the comparison
+ is not.
+ """
+ location = str(tmp_path / "sh.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ array = group.create_array(
+ "v", shape=(1024,), dtype="i4", chunks=(8,), shards=(1024,)
+ )
+ array[:] = np.arange(1024, dtype="i4")
+
+ inner = bytes_to_read_first_chunk(location, 8)
+ whole_shard = bytes_to_read_first_chunk(location, 1024)
+ assert inner < whole_shard
+
+ # And the variable chooses the inner chunk for exactly this reason.
+ assert ZarrDataset(location).variables["v"].chunking == (8,)
+
+
+class Test_declared_dtype:
+ def test_the_declared_dtype_matches_what_the_decode_layer_produces(
+ self, tmp_path, mocker
+ ):
+ """A packed variable's lazy dtype must equal its realised dtype.
+
+ On the netCDF path netCDF4 does the arithmetic and reports the dtype. On
+ the Zarr path Iris does both, in two separately written statements, and
+ Dask does not check them against each other: a mismatch builds a graph
+ and surfaces later as a wrong dtype on the cube.
+ """
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "b02",
+ (4,),
+ "u2",
+ 3,
+ dimensions=("p",),
+ attributes={"scale_factor": 0.0001, "add_offset": -0.1},
+ )
+ array[:] = [1000, 2000, 3000, 4000]
+
+ lazy = ZarrDataset(str(tmp_path / "p.zarr")).variables["b02"].read_data(policy)
+ realised = lazy.compute()
+
+ assert lazy.dtype == realised.dtype
+ assert lazy.dtype == np.dtype("f8")
+ np.testing.assert_allclose(realised, [0.0, 0.1, 0.2, 0.3])
+
+ @pytest.mark.parametrize(
+ ("scale_factor", "expected"),
+ # A Zarr attribute is JSON, so a scale factor reaches Iris as a Python
+ # float or a Python int and never as a NumPy scalar - zarr-python refuses
+ # to write one. Both forms are here because they promote differently:
+ # a weak Python int leaves a uint16 variable uint16.
+ [(0.01, "f8"), (1, "u2")],
+ )
+ def test_the_declared_dtype_matches_the_cf_loader_calculation(
+ self, tmp_path, scale_factor, expected
+ ):
+ """And it is the same answer `_get_actual_dtype` gives.
+
+ The cube's data variable and its coordinates must agree about what a
+ packed value becomes, and they reach that answer through different code.
+ """
+ group = zarr.open_group(store=tmp_path / "p.zarr", mode="w", zarr_format=3)
+ stores.make_array(
+ group,
+ "v",
+ (4,),
+ "u2",
+ 3,
+ attributes={"scale_factor": scale_factor},
+ )
+ variable = ZarrDataset(str(tmp_path / "p.zarr")).variables["v"]
+ from iris.fileformats.zarr import _decode
+
+ parameters = _decode.decode_parameters(
+ variable.attributes,
+ dtype=variable.dtype,
+ storage_fill_value=variable.fill_value,
+ zarr_format=variable.zarr_format,
+ variable_name=variable.name,
+ )
+ assert parameters["dtype"] == cf_loader._get_actual_dtype(variable)
+ assert parameters["dtype"] == np.dtype(expected)
+
+
+class Test_chunking:
+ @pytest.mark.parametrize("chunking", ["default", "as_dask"])
+ def test_chunks_are_sized_on_the_decoded_dtype(self, tmp_path, mocker, chunking):
+ """What a task holds is the unpacked block, not the stored one.
+
+ An int8 variable unpacking to float64 is eight times its stored size the
+ moment the decode layer runs, so chunks sized on the stored dtype hand a
+ task eight times the memory Dask was asked to target. The netCDF path
+ sizes on the decoded dtype, its data proxy declaring it, and this one has
+ to agree - under `CHUNK_CONTROL.as_dask()` as well as by default, since
+ `da.from_array` would otherwise size on the array it is given, which is
+ the stored one.
+ """
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ location = packed_store(tmp_path, (64, 64), (4, 64))
+ chunking_policy = {"default": policy, "as_dask": dask_policy}[chunking]
+
+ target = 2048
+ with dask.config.set({"array.chunk-size": f"{target}B"}):
+ lazy = ZarrDataset(location).variables["v"].read_data(chunking_policy)
+
+ assert lazy.dtype == np.dtype("f8")
+ block_bytes = int(np.prod(lazy.chunksize)) * lazy.dtype.itemsize
+ assert block_bytes <= target
+
+ def test_as_dask_is_still_dasks_own_answer(self, tmp_path, mocker):
+ """Calling Dask's policy by hand must not turn into replacing it.
+
+ Sizing on the decoded dtype means calling `normalize_chunks` here rather
+ than leaving it to `from_array`, and the risk in doing that is Iris's own
+ sizing or store alignment creeping back in on top. So: the result is what
+ Dask's policy gives for the decoded dtype, and the store's chunking of 7
+ leaves no mark on it.
+ """
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ location = packed_store(tmp_path, (64, 64), (7, 64))
+
+ with dask.config.set({"array.chunk-size": "2048B"}):
+ lazy = ZarrDataset(location).variables["v"].read_data(dask_policy)
+ # Inside the context: Dask's policy reads the configured target.
+ expected = normalize_chunks("auto", shape=(64, 64), dtype=np.dtype("f8"))
+
+ assert lazy.chunksize == tuple(sizes[0] for sizes in expected)
+ assert lazy.chunksize[0] % 7 != 0
+
+ def test_a_lazy_read_divides_a_store_chunk_to_meet_dasks_target(
+ self, tmp_path, mocker
+ ):
+ """Chunking follows Dask's target, even where that divides a store chunk.
+
+ Iris used to round back up to whole store chunks here. Dividing one costs
+ a re-read per division, because a chunk is fetched and decompressed
+ whole - but netCDF has always made the same trade, and which side of it a
+ caller wants depends on their case. `CHUNK_CONTROL` is how they say, and
+ `iris.fileformats.cf.loader` warns so that they know to.
+ """
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ location = str(tmp_path / "b.zarr")
+ group = zarr.open_group(store=location, mode="w", zarr_format=3)
+ array = group.create_array(
+ "v", shape=(64,), dtype="i4", chunks=(16,), dimension_names=("t",)
+ )
+ array[:] = np.arange(64, dtype="i4")
+
+ with dask.config.set({"array.chunk-size": "32B"}):
+ lazy = ZarrDataset(location).variables["v"].read_data(policy)
+
+ # 32 bytes of int32 is 8 elements, half the store's chunk of 16.
+ assert lazy.chunksize[0] == 8
+ np.testing.assert_array_equal(lazy.compute(), np.arange(64))
diff --git a/lib/iris/tests/unit/fileformats/zarr/loader/__init__.py b/lib/iris/tests/unit/fileformats/zarr/loader/__init__.py
new file mode 100644
index 0000000000..dc37340f8b
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/loader/__init__.py
@@ -0,0 +1,5 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :func:`iris.fileformats.zarr.load_cubes`."""
diff --git a/lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py b/lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py
new file mode 100644
index 0000000000..09a56862cf
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/loader/test_load_cubes.py
@@ -0,0 +1,407 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :func:`iris.fileformats.zarr.load_cubes`."""
+
+import numpy as np
+import pytest
+import zarr
+
+import iris
+from iris.cube import Cube
+from iris.exceptions import TranslationError
+from iris.fileformats.zarr import load_cubes
+from iris.loading import LOAD_PROBLEMS
+from iris.warnings import IrisCfLoadWarning
+
+from .. import stores
+
+
+@pytest.fixture(autouse=True)
+def _assert_clean_load():
+ """Every load in this file must leave `LOAD_PROBLEMS` empty.
+
+ A malformed grid mapping on the shared `stores.make_store` fixture once
+ made every test that loaded through it record a load problem and raise
+ the standing `IrisLoadWarning` that announces one - silently covering the
+ one signal that would reveal a genuine Zarr load problem. This guards
+ against that recurring, for every test in this file, not only the ones
+ that use the canonical fixture.
+ """
+ LOAD_PROBLEMS.reset()
+ yield
+ assert LOAD_PROBLEMS.problems == []
+
+
+def test_the_canonical_store_loads_its_one_data_variable(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ cubes = list(load_cubes(location))
+ assert len(cubes) == 1
+ cube = cubes[0]
+ assert isinstance(cube, Cube)
+ assert cube.var_name == "air_temperature"
+ assert cube.units == "K"
+ np.testing.assert_array_equal(cube.data, stores.DATA)
+
+
+def test_one_cube_per_data_variable(tmp_path):
+ root = zarr.open_group(store=tmp_path / "two.zarr", mode="w", zarr_format=3)
+ for name in ("first", "second", "third"):
+ array = stores.make_array(
+ root, name, (2,), "f8", 3, dimensions=("p",), attributes={"units": "1"}
+ )
+ array[:] = [1.0, 2.0]
+
+ cubes = list(load_cubes(str(tmp_path / "two.zarr")))
+ assert sorted(cube.var_name for cube in cubes) == ["first", "second", "third"]
+
+
+def test_a_version_2_store_loads_the_same_cube(tmp_path):
+ v3 = list(load_cubes(stores.make_store(tmp_path / "v3.zarr")))[0]
+ v2 = list(load_cubes(stores.make_store(tmp_path / "v2.zarr", zarr_format=2)))[0]
+ assert v2 == v3
+
+
+def test_a_name_constraint_skips_variables_before_they_are_built(tmp_path, mocker):
+ """`constraints` is a pre-build filter, not a post-build one.
+
+ `_translate_constraints_to_var_callback` only answers for a bare
+ `NameConstraint`; any other constraint returns None and every variable is
+ built. The spy is what distinguishes "filtered early" from "built then
+ discarded" - the cube list alone cannot.
+ """
+ from iris.fileformats.zarr import loader as zarr_loader
+
+ root = zarr.open_group(store=tmp_path / "two.zarr", mode="w", zarr_format=3)
+ for name in ("wanted", "unwanted"):
+ array = stores.make_array(
+ root, name, (2,), "f8", 3, dimensions=("p",), attributes={"units": "1"}
+ )
+ array[:] = [1.0, 2.0]
+
+ # The loader binds `_load_cube` at import, so the spy must go on its own
+ # module reference rather than on `cf.loader`'s.
+ spy = mocker.spy(zarr_loader, "_load_cube")
+ constraint = iris.NameConstraint(var_name="wanted")
+ cubes = list(load_cubes(str(tmp_path / "two.zarr"), constraints=constraint))
+
+ assert [cube.var_name for cube in cubes] == ["wanted"]
+ assert spy.call_count == 1
+
+
+def test_a_callback_can_reject_a_cube(tmp_path):
+ from iris.exceptions import IgnoreCubeException
+
+ def reject(cube, field, filename):
+ raise IgnoreCubeException
+
+ location = stores.make_store(tmp_path / "s.zarr")
+ assert list(load_cubes(location, callback=reject)) == []
+
+
+def test_the_group_keyword_selects_a_group(tmp_path):
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ sub = root.create_group("measurements")
+ band = stores.make_array(
+ sub, "b02", (3,), "u2", 3, dimensions=("p",), attributes={"units": "1"}
+ )
+ band[:] = [1, 2, 3]
+
+ cubes = list(load_cubes(str(tmp_path / "n.zarr"), group="measurements"))
+ assert [cube.var_name for cube in cubes] == ["b02"]
+
+
+def test_several_stores_in_one_call(tmp_path):
+ first = stores.make_store(tmp_path / "a.zarr")
+ second = stores.make_store(tmp_path / "b.zarr")
+ assert len(list(load_cubes([first, second]))) == 2
+
+
+def test_a_group_with_no_arrays_names_the_groups_that_have_them(tmp_path):
+ """A real store in this shape exists, and silence is the worst answer.
+
+ The ESA EOPF Sentinel-2 products hold no arrays in the root group, only
+ subgroups - `conditions/geometry`, `measurements/reflectance/r10m` and
+ others. Iris loads one group per call, so loading the default root yields
+ nothing; returning an empty CubeList from a store that plainly holds data
+ leaves the user with no way to find out why.
+ """
+ root = zarr.open_group(store=tmp_path / "e.zarr", mode="w", zarr_format=3)
+ root.create_group("conditions")
+ measurements = root.create_group("measurements")
+ stores.make_array(measurements, "b02", (2,), "u2", 3, dimensions=("p",))
+
+ with pytest.raises(TranslationError, match="measurements"):
+ list(load_cubes(str(tmp_path / "e.zarr")))
+
+
+def test_the_message_says_so_when_there_are_no_groups_either(tmp_path):
+ zarr.open_group(store=tmp_path / "empty.zarr", mode="w", zarr_format=3)
+ with pytest.raises(TranslationError, match="no data variables"):
+ list(load_cubes(str(tmp_path / "empty.zarr")))
+
+
+def test_nested_groups_are_not_walked(tmp_path):
+ """Iris loads one group per call, like xarray.
+
+ A Zarr store can legitimately hold unrelated datasets, and merging them
+ into one CubeList would be wrong.
+ """
+ root = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ top = stores.make_array(
+ root, "top", (2,), "f8", 3, dimensions=("p",), attributes={"units": "1"}
+ )
+ top[:] = [1.0, 2.0]
+ inner = stores.make_array(
+ root.create_group("sub"), "inner", (2,), "f8", 3, dimensions=("p",)
+ )
+ inner[:] = [3.0, 4.0]
+
+ cubes = list(load_cubes(str(tmp_path / "n.zarr")))
+ assert [cube.var_name for cube in cubes] == ["top"]
+
+
+def test_the_cube_is_lazy(tmp_path, mocker):
+ from iris.fileformats.cf import loader as cf_loader
+
+ mocker.patch.object(cf_loader, "_LAZYVAR_MIN_BYTES", 1)
+ cube = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))[0]
+ assert cube.has_lazy_data()
+
+
+def test_the_grid_mapping_variable_does_not_become_a_cube(tmp_path):
+ cubes = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))
+ assert "grid" not in [cube.var_name for cube in cubes]
+
+
+def test_cf_reserved_attributes_do_not_reach_the_cube(tmp_path):
+ """`units` becomes a property, so it must not also be an attribute.
+
+ Which attributes survive is decided by which ones the loading rules did
+ not read, so an attribute Iris consumes but fails to record as read
+ arrives on the cube as well.
+ """
+ cube = list(load_cubes(stores.make_store(tmp_path / "s.zarr")))[0]
+ assert "units" not in cube.attributes
+ assert "long_name" not in cube.attributes
+ assert cube.attributes["Conventions"] == "CF-1.7"
+
+
+def test_a_structured_attribute_reaches_the_cube_unchanged(tmp_path):
+ """CF cannot express a nested attribute; Zarr allows one. Carry it.
+
+ Two independent producers write them - NOAA GFS gives `valid_time` a
+ `statistics_approximate` object, and the ESA EOPF product root carries
+ `stac_discovery`. Interpreting them needs the attribute-model discussion
+ in SciTools/iris#7288; dropping them loses the publisher's metadata.
+ """
+ group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ (2,),
+ "f8",
+ 3,
+ dimensions=("p",),
+ attributes={"stac_discovery": {"id": "abc"}},
+ )
+ array[:] = [1.0, 2.0]
+
+ cube = list(load_cubes(str(tmp_path / "n.zarr")))[0]
+ assert cube.attributes["stac_discovery"] == {"id": "abc"}
+
+
+def test_a_structured_attribute_cannot_then_be_saved_to_netcdf(tmp_path):
+ """Known limitation, characterised so that it is not a surprise.
+
+ Zarr allows any JSON literal as an attribute value and netCDF does not,
+ so a store carrying one produces a cube Iris cannot save as netCDF.
+ Checking attribute content before saving is SciTools/iris#7329, and
+ fixing it means changing the netCDF saver. Strip the attribute, or save
+ to Zarr once that is supported.
+ """
+ import iris
+
+ group = zarr.open_group(store=tmp_path / "n.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ (2,),
+ "f8",
+ 3,
+ dimensions=("p",),
+ attributes={"stac_discovery": {"id": "abc"}},
+ )
+ array[:] = [1.0, 2.0]
+ cube = list(load_cubes(str(tmp_path / "n.zarr")))[0]
+
+ with pytest.raises(TypeError, match="stac_discovery"):
+ iris.save(cube, str(tmp_path / "out.nc"))
+
+ # And the documented workaround works.
+ del cube.attributes["stac_discovery"]
+ iris.save(cube, str(tmp_path / "out.nc"))
+
+
+def test_a_dimension_name_reused_at_two_lengths_still_places_a_coordinate(tmp_path):
+ """Zarr dimension names carry none of netCDF's uniqueness guarantee.
+
+ A data variable may name both of its dimensions `d` at two lengths, which
+ makes the name alone useless for choosing an axis. `(name, size)` names
+ one axis here, and the coordinate belongs on it; matching on the bare name
+ picks the first `d` and the coordinate is dropped as the wrong length.
+ """
+ group = zarr.open_group(store=tmp_path / "r.zarr", mode="w", zarr_format=3)
+ data = stores.make_array(
+ group,
+ "v",
+ (2, 3),
+ "f8",
+ 3,
+ dimensions=("d", "d"),
+ attributes={"units": "1", "coordinates": "lat"},
+ )
+ data[:] = np.zeros((2, 3))
+ lat = stores.make_array(
+ group,
+ "lat",
+ (3,),
+ "f8",
+ 3,
+ dimensions=("d",),
+ attributes={"units": "degrees_north", "standard_name": "latitude"},
+ )
+ lat[:] = [10.0, 20.0, 30.0]
+
+ cube = list(load_cubes(str(tmp_path / "r.zarr")))[0]
+ assert cube.coord_dims("latitude") == (1,)
+
+
+def test_version_3_zeros_are_not_masked(tmp_path):
+ """An int16 [0, 1, 2] with no CF attributes loads with nothing masked.
+
+ zarr-python defaults `fill_value` to zero, so reading the storage field
+ as a sentinel masks valid data in the ordinary case.
+ """
+ group = zarr.open_group(store=tmp_path / "z3.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group, "v", (3,), "i2", 3, dimensions=("p",), attributes={"units": "1"}
+ )
+ array[:] = [0, 1, 2]
+
+ cube = list(load_cubes(str(tmp_path / "z3.zarr")))[0]
+ assert not np.ma.is_masked(cube.data)
+ np.testing.assert_array_equal(cube.data, [0, 1, 2])
+
+
+def test_version_2_zeros_are_masked(tmp_path):
+ """The same array as version 2 does use `fill_value` as the mask source.
+
+ Version 2 stores predate the CF `_FillValue` convention, and this is what
+ their writers meant - which is what released xarray's
+ `use_zarr_fill_value_as_mask` does. Iris warns about exactly this, since
+ zarr-python 3 defaults a version 2 store's `fill_value` to zero too, and a
+ store written today can lose a genuine zero this way.
+ """
+ group = zarr.open_group(store=tmp_path / "z2.zarr", mode="w", zarr_format=2)
+ array = stores.make_array(
+ group, "v", (3,), "i2", 2, dimensions=("p",), attributes={"units": "1"}
+ )
+ array[:] = [0, 1, 2]
+
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ cube = list(load_cubes(str(tmp_path / "z2.zarr")))[0]
+ assert cube.data.mask.tolist() == [True, False, False]
+
+
+def test_a_cf_fill_value_beats_the_storage_one(tmp_path):
+ group = zarr.open_group(store=tmp_path / "z.zarr", mode="w", zarr_format=3)
+ array = stores.make_array(
+ group,
+ "v",
+ (3,),
+ "i2",
+ 3,
+ dimensions=("p",),
+ fill_value=0,
+ attributes={"units": "1", "_FillValue": -999},
+ )
+ array[:] = [0, -999, 2]
+
+ cube = list(load_cubes(str(tmp_path / "z.zarr")))[0]
+ assert cube.data.mask.tolist() == [False, True, False]
+
+
+class TestMesh:
+ """UGRID data, which a Zarr store holds exactly as a netCDF file does.
+
+ Nothing in `iris.fileformats.netcdf.ugrid_load` is netCDF-specific: it
+ reads a `CFReader`, and a `CFReader` over a `ZarrDataset` answers the
+ same questions. These assert that, rather than re-testing the mesh
+ machinery itself.
+ """
+
+ @pytest.fixture
+ def cube(self, tmp_path):
+ location = stores.make_mesh_store(tmp_path / "mesh.zarr")
+ return list(load_cubes(location))[0]
+
+ def test_the_cube_carries_a_mesh(self, cube):
+ assert cube.mesh is not None
+ assert cube.mesh.var_name == "mesh"
+ assert cube.mesh.topology_dimension == 2
+
+ def test_the_mesh_is_on_the_right_dimension(self, cube):
+ assert cube.mesh_dim() == cube.coord_dims("time")[0] + 1
+
+ def test_the_mesh_coords_are_attached(self, cube):
+ names = sorted(coord.name() for coord in cube.coords(mesh_coords=True))
+ assert names == ["latitude", "longitude"]
+
+ def test_the_connectivity_is_read(self, cube):
+ connectivity = cube.mesh.face_node_connectivity
+ np.testing.assert_array_equal(
+ connectivity.indices, stores.MESH_ARRAYS["face_nodes"][0]
+ )
+
+ def test_mesh_attributes_do_not_also_reach_the_cube(self, cube):
+ """`mesh` and `location` are consumed by the mesh build, not carried.
+
+ They reach the cube as stray attributes if the MeshCoords are built
+ after it rather than before - which is why `_prepare_mesh_coords` and
+ `_attach_mesh_coords` are two calls with `_load_cube` between them.
+ """
+ assert "mesh" not in cube.attributes
+ assert "location" not in cube.attributes
+
+ def test_a_version_2_store_gives_the_same_mesh(self, tmp_path):
+ v3 = list(load_cubes(stores.make_mesh_store(tmp_path / "v3.zarr")))[0]
+ v2 = list(
+ load_cubes(stores.make_mesh_store(tmp_path / "v2.zarr", zarr_format=2))
+ )[0]
+ assert v2.mesh == v3.mesh
+ assert v2 == v3
+
+ def test_load_meshes_reads_a_zarr_store(self, tmp_path):
+ """`iris.mesh.load_meshes` dispatches on format, so it reaches Zarr too.
+
+ Its dispatch is a second copy of the main loader's; the two are meant
+ to move together.
+ """
+ location = stores.make_mesh_store(tmp_path / "mesh.zarr")
+ meshes = iris.mesh.load_meshes(location)
+ assert list(meshes) == [location]
+ assert [mesh.var_name for mesh in meshes[location]] == ["mesh"]
+
+ def test_a_store_naming_an_absent_mesh_still_loads(self, tmp_path):
+ """A dangling `mesh` reference is logged, not raised.
+
+ Same treatment as netCDF: the cube is worth having without its mesh.
+ """
+ location = stores.make_mesh_store(tmp_path / "mesh.zarr")
+ group = zarr.open_group(store=location, mode="r+")
+ group["temp"].attrs["mesh"] = "absent"
+
+ cube = list(load_cubes(location))[0]
+ assert cube.mesh is None
diff --git a/lib/iris/tests/unit/fileformats/zarr/stores.py b/lib/iris/tests/unit/fileformats/zarr/stores.py
new file mode 100644
index 0000000000..7986a06821
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/stores.py
@@ -0,0 +1,224 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Builders for the small Zarr stores these tests ask questions of.
+
+One canonical store, built from the constants the shared CFDataset contract
+writes, so that a Zarr answer and a netCDF answer can be compared without
+translating between two sets of fixtures - and so that the two cannot drift
+apart.
+"""
+
+import numpy as np
+import zarr
+
+from iris.tests.unit.fileformats.cf.dataset.contract import (
+ CONTRACT_DATA as DATA,
+)
+from iris.tests.unit.fileformats.cf.dataset.contract import (
+ CONTRACT_DIMENSIONS as DIMENSIONS,
+)
+from iris.tests.unit.fileformats.cf.dataset.contract import (
+ CONTRACT_GLOBALS as GLOBALS,
+)
+from iris.tests.unit.fileformats.cf.dataset.contract import (
+ CONTRACT_VARIABLE_ATTRS,
+)
+
+#: The canonical store's variable attributes.
+#:
+#: The contract's own, plus the `grid_mapping` that makes the `grid` array
+#: below mean something. The contract checks its attributes are present
+#: rather than that they are all there is, so the extra is legitimate.
+VARIABLE_ATTRS = {**CONTRACT_VARIABLE_ATTRS, "grid_mapping": "grid"}
+
+#: The version 2 attribute carrying dimension names.
+ARRAY_DIMENSIONS = "_ARRAY_DIMENSIONS"
+
+#: `make_array`'s own default for `fill_value`, distinct from `None`.
+#:
+#: `Group.create_array` treats an explicit `fill_value=None` as "this array
+#: has no fill value" on a version 2 store, which is not the same thing as
+#: "the caller did not ask for one". Passing this sentinel through instead
+#: means `make_array`'s own default lets zarr-python choose its own default
+#: fill value, on both versions, the way omitting the argument would.
+_DEFAULT_FILL_VALUE = object()
+
+
+def make_array(
+ group,
+ name,
+ shape,
+ dtype,
+ zarr_format,
+ *,
+ dimensions=None,
+ chunks="auto",
+ shards=None,
+ fill_value=_DEFAULT_FILL_VALUE,
+ attributes=None,
+):
+ """Create one array, recording dimension names the way its version does.
+
+ zarr-python refuses `dimension_names` on a version 2 array, so version 2
+ names are written to the `_ARRAY_DIMENSIONS` attribute by hand - which is
+ what xarray and NCZarr both do.
+ """
+ extra = {}
+ if zarr_format == 3 and dimensions is not None:
+ extra["dimension_names"] = dimensions
+ if fill_value is not _DEFAULT_FILL_VALUE:
+ extra["fill_value"] = fill_value
+ array = group.create_array(
+ name,
+ shape=shape,
+ dtype=dtype,
+ chunks=chunks,
+ shards=shards,
+ **extra,
+ )
+ if zarr_format == 2 and dimensions is not None:
+ array.attrs[ARRAY_DIMENSIONS] = list(dimensions)
+ for key, value in (attributes or {}).items():
+ array.attrs[key] = value
+ return array
+
+
+def make_store(path, *, zarr_format=3, chunks="auto", shards=None):
+ """Write the canonical store at `path`, and return `path` as a string."""
+ root = zarr.open_group(store=path, mode="w", zarr_format=zarr_format)
+ for name, value in GLOBALS.items():
+ root.attrs[name] = value
+
+ array = make_array(
+ root,
+ "air_temperature",
+ DATA.shape,
+ "f8",
+ zarr_format,
+ dimensions=tuple(DIMENSIONS),
+ chunks=chunks,
+ shards=shards,
+ fill_value=-1.0,
+ attributes=VARIABLE_ATTRS,
+ )
+ array[:] = DATA
+
+ # A dimensionless variable: what a grid mapping is, and the one shape a
+ # CFDatasetVariable must handle without a leading dimension.
+ scalar = make_array(
+ root,
+ "grid",
+ (),
+ "i4",
+ zarr_format,
+ dimensions=(),
+ attributes={
+ "grid_mapping_name": "latitude_longitude",
+ "earth_radius": 6371229.0,
+ },
+ )
+ scalar[()] = 0
+
+ return str(path)
+
+
+#: The UGRID store's mesh topology variable, as CF attributes.
+MESH_ATTRS = {
+ "cf_role": "mesh_topology",
+ "topology_dimension": 2,
+ "node_coordinates": "node_x node_y",
+ "face_node_connectivity": "face_nodes",
+ "face_coordinates": "face_x face_y",
+ "long_name": "topology of a 2D mesh",
+}
+
+#: The UGRID store's arrays: name -> (values, dimension names, attributes).
+#:
+#: Two quadrilateral faces over six nodes, with a time dimension on the data
+#: so that the mesh dimension is not the only one. Every dimension is a
+#: different size, so a coordinate cannot be attached to the right axis by
+#: accident - see `_DimensionIdentity`.
+MESH_ARRAYS = {
+ "node_x": (
+ np.array([0.0, 1.0, 2.0, 0.0, 1.0, 2.0]),
+ ("node",),
+ {"standard_name": "longitude", "units": "degrees_east"},
+ ),
+ "node_y": (
+ np.array([0.0, 0.0, 0.0, 1.0, 1.0, 1.0]),
+ ("node",),
+ {"standard_name": "latitude", "units": "degrees_north"},
+ ),
+ "face_x": (
+ np.array([0.5, 1.5]),
+ ("face",),
+ {"standard_name": "longitude", "units": "degrees_east"},
+ ),
+ "face_y": (
+ np.array([0.5, 0.5]),
+ ("face",),
+ {"standard_name": "latitude", "units": "degrees_north"},
+ ),
+ "face_nodes": (
+ np.array([[0, 1, 4, 3], [1, 2, 5, 4]], dtype="i4"),
+ ("face", "vertex"),
+ {"cf_role": "face_node_connectivity", "start_index": 0},
+ ),
+ "time": (
+ np.array([0.0, 1.0, 2.0]),
+ ("time",),
+ {"standard_name": "time", "units": "days since 2000-01-01"},
+ ),
+ "temp": (
+ np.arange(6, dtype="f8").reshape(3, 2),
+ ("time", "face"),
+ {
+ "standard_name": "air_temperature",
+ "units": "K",
+ "mesh": "mesh",
+ "location": "face",
+ "coordinates": "face_x face_y",
+ },
+ ),
+}
+
+
+def make_mesh_store(path, *, zarr_format=3):
+ """Write a UGRID store at `path`, and return `path` as a string.
+
+ The same topology this module's netCDF counterpart writes, so that the
+ two backends' meshes can be compared directly.
+
+ Every array is written with no fill value. A mesh is full of legitimate
+ zeros - node coordinates on an axis, and connectivity indices under
+ `start_index = 0` - and on a version 2 store zarr-python's default fill
+ value is zero, which Iris reads as "mask the zeros". That rule has its
+ own test; a mesh fixture that tripped it would be testing it twice and
+ reading its own topology back full of masks.
+ """
+ root = zarr.open_group(store=path, mode="w", zarr_format=zarr_format)
+ root.attrs["Conventions"] = "CF-1.7, UGRID-1.0"
+
+ mesh = make_array(
+ root, "mesh", (), "i4", zarr_format, dimensions=(), fill_value=None
+ )
+ for key, value in MESH_ATTRS.items():
+ mesh.attrs[key] = value
+ mesh[()] = 0
+
+ for name, (values, dimensions, attributes) in MESH_ARRAYS.items():
+ array = make_array(
+ root,
+ name,
+ values.shape,
+ values.dtype,
+ zarr_format,
+ dimensions=dimensions,
+ fill_value=None,
+ attributes=attributes,
+ )
+ array[:] = values
+
+ return str(path)
diff --git a/lib/iris/tests/unit/fileformats/zarr/test_CFReader.py b/lib/iris/tests/unit/fileformats/zarr/test_CFReader.py
new file mode 100644
index 0000000000..e318d9ec05
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/test_CFReader.py
@@ -0,0 +1,19 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The CF layer reads a Zarr store through the same reader netCDF uses."""
+
+from iris.fileformats.cf import CFReader
+from iris.fileformats.zarr._dataset import ZarrDataset
+
+from . import stores
+
+
+def test_the_cf_reader_classifies_a_zarr_store(tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ with CFReader(ZarrDataset(location)) as reader:
+ assert "air_temperature" in reader.cf_group.data_variables
+ assert "grid" in reader.cf_group.grid_mappings
+ assert reader.cf_group.global_attributes["title"] == "contract"
+ assert reader.filename == location
diff --git a/lib/iris/tests/unit/fileformats/zarr/test__decode.py b/lib/iris/tests/unit/fileformats/zarr/test__decode.py
new file mode 100644
index 0000000000..30dcd3608f
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/test__decode.py
@@ -0,0 +1,266 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :mod:`iris.fileformats.zarr._decode`."""
+
+import base64
+import struct
+
+import numpy as np
+import pytest
+
+from iris.fileformats.zarr import _decode
+from iris.warnings import IrisCfLoadWarning
+
+
+def base64_double(value):
+ """Return `value` in the encoding released xarray writes for a float."""
+ return base64.b64encode(struct.pack("<d", value)).decode("ascii")
+
+
+def parameters(attributes, *, dtype="i2", storage_fill_value=0, zarr_format=3):
+ return _decode.decode_parameters(
+ attributes,
+ dtype=np.dtype(dtype),
+ storage_fill_value=storage_fill_value,
+ zarr_format=zarr_format,
+ variable_name="v",
+ )
+
+
+class Test__decode_fill_value:
+ def test_fill_value_from_a_json_number(self):
+ result = _decode._decode_fill_value(-999, np.dtype("i2"), "v")
+ assert result == -999
+ assert result.dtype == np.dtype("i2")
+
+ def test_fill_value_from_base64_nan(self):
+ """NOAA GFS writes 'AAAAAAAA+H8=' and it is not a CF violation.
+
+ base64(struct.pack("<d", float("nan"))) is that literal string, and
+ released xarray writes it for every floating dtype.
+ """
+ result = _decode._decode_fill_value(
+ "AAAAAAAA+H8=", np.dtype("f4"), "temperature_2m"
+ )
+ assert result.dtype == np.dtype("f4")
+ assert np.isnan(result)
+
+ def test_fill_value_from_base64_finite_sentinel(self):
+ # Not only NaN: a finite sentinel must survive the narrowing too.
+ encoded = base64_double(-999.0)
+ result = _decode._decode_fill_value(encoded, np.dtype("f4"), "v")
+ assert result == np.float32(-999.0)
+
+ def test_fill_value_from_a_json_number_on_a_float(self):
+ result = _decode._decode_fill_value(-999.0, np.dtype("f4"), "v")
+ assert result == np.float32(-999.0)
+
+ def test_an_undecodable_fill_value_warns_and_is_ignored(self):
+ with pytest.warns(IrisCfLoadWarning, match="banana"):
+ result = _decode._decode_fill_value("banana", np.dtype("f4"), "v")
+ assert result is None
+
+ def test_a_base64_string_of_the_wrong_width_warns(self):
+ encoded = base64.b64encode(b"abc").decode("ascii")
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ assert _decode._decode_fill_value(encoded, np.dtype("f4"), "v") is None
+
+ def test_a_base64_string_on_an_integer_variable_warns(self):
+ # The convention is for floating dtypes. An integer variable's sentinel
+ # is a JSON number, and a string there is a producer error.
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ result = _decode._decode_fill_value(base64_double(1.0), np.dtype("i2"), "v")
+ assert result is None
+
+ def test_an_out_of_range_integer_warns_and_is_ignored(self):
+ # 70000 does not fit an int16, however a producer came to write it.
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ result = _decode._decode_fill_value(70000, np.dtype("i2"), "v")
+ assert result is None
+
+ def test_a_negative_out_of_range_integer_warns_and_is_ignored(self):
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ result = _decode._decode_fill_value(-70000, np.dtype("i2"), "v")
+ assert result is None
+
+ def test_an_out_of_range_unsigned_integer_warns_and_is_ignored(self):
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ result = _decode._decode_fill_value(300, np.dtype("u1"), "v")
+ assert result is None
+
+
+class Test_decode_parameters:
+ """Includes `_decode_missing_values`, which only this calls."""
+
+ def test_version_3_ignores_the_storage_fill_value(self):
+ """An int16 [0, 1, 2] with no CF attributes loads with nothing masked.
+
+ zarr-python defaults `fill_value` to zero, so reading it as a sentinel
+ masks valid data in the ordinary case, not the exotic one.
+ """
+ assert parameters({}, storage_fill_value=0)["sentinels"] == ()
+
+ def test_version_3_zeros_survive_masking_and_scaling_unchanged(self):
+ """The property the rule above exists for, demonstrated end to end.
+
+ An int16 array of zeros, no CF attributes, storage fill value zero:
+ pushed through the actual masking function with the actual parameters
+ `decode_parameters` produces, not hand-written keyword arguments.
+ """
+ params = parameters({}, storage_fill_value=0, zarr_format=3)
+ block = np.array([0, 0, 1], dtype="i2")
+ result = _decode.mask_and_scale(block, **params)
+ assert not result.mask.any()
+ assert result.tolist() == [0, 0, 1]
+
+ def test_version_2_uses_the_storage_fill_value_when_there_is_no_attribute(self):
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ result = parameters({}, storage_fill_value=0, zarr_format=2)
+ assert result["sentinels"] == (0,)
+
+ def test_a_cf_attribute_beats_the_storage_fill_value_on_version_2(self):
+ result = parameters({"_FillValue": -999}, storage_fill_value=0, zarr_format=2)
+ assert result["sentinels"] == (np.int16(-999),)
+
+ def test_both_cf_missing_value_attributes_are_sentinels(self):
+ result = parameters({"_FillValue": -999, "missing_value": -888})
+ assert set(result["sentinels"]) == {np.int16(-999), np.int16(-888)}
+
+ def test_a_vector_missing_value_declares_every_sentinel(self):
+ """CF section 2.5.1: "a scalar or vector containing values".
+
+ netCDF4 masks on every element of a vector `missing_value`, so a store
+ and a file carrying the same attribute have to mask the same values.
+ """
+ result = parameters({"missing_value": [-999, -888]})
+ assert set(result["sentinels"]) == {np.int16(-999), np.int16(-888)}
+
+ def test_a_vector_missing_value_masks_every_sentinel(self):
+ params = parameters({"missing_value": [-999, -888]})
+ block = np.array([-999, 1, -888, 2], dtype="i2")
+ result = _decode.mask_and_scale(block, **params)
+ assert result.mask.tolist() == [True, False, True, False]
+
+ def test_a_vector_missing_value_keeps_the_version_2_fill_value_out(self):
+ """The second failure the scalar-only reading caused, and the worse one.
+
+ A rejected `missing_value` left a version 2 variable looking as though it
+ declared no CF attribute at all, so the storage fill value became the
+ sentinel instead - masking every valid zero in the array.
+ """
+ result = parameters(
+ {"missing_value": [-999, -888]}, storage_fill_value=0, zarr_format=2
+ )
+ assert 0 not in result["sentinels"]
+
+ def test_one_unreadable_element_does_not_discard_the_others(self):
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ result = parameters({"missing_value": [-999, "nonsense"]})
+ assert result["sentinels"] == (np.int16(-999),)
+
+ def test_a_vector_fill_value_is_not_read_as_sentinels(self):
+ """CF section 2.5.1 makes `_FillValue` a scalar, unlike `missing_value`.
+
+ A producer writing a vector there has written something the convention
+ does not define, and a warning naming the variable is more use than a
+ guess at what was meant.
+ """
+ with pytest.warns(IrisCfLoadWarning, match="v"):
+ result = parameters({"_FillValue": [-999, -888]})
+ assert result["sentinels"] == ()
+
+ def test_valid_range_sets_both_bounds(self):
+ result = parameters({"valid_range": [0, 10]})
+ assert result["valid_min"] == 0
+ assert result["valid_max"] == 10
+
+ def test_valid_min_and_valid_max_are_read_separately(self):
+ result = parameters({"valid_min": 1, "valid_max": 9})
+ assert (result["valid_min"], result["valid_max"]) == (1, 9)
+
+ def test_packing_parameters_and_the_resulting_dtype(self):
+ result = parameters({"scale_factor": 0.0001, "add_offset": 1.0}, dtype="u2")
+ assert result["scale_factor"] == 0.0001
+ assert result["add_offset"] == 1.0
+ assert result["dtype"] == np.dtype("f8")
+
+
+class Test_decoding_required:
+ def test_decoding_required_is_false_for_a_plain_array(self):
+ assert _decode.decoding_required(parameters({})) is False
+
+ def test_decoding_required_is_true_for_each_rule(self):
+ assert _decode.decoding_required(parameters({"_FillValue": -999})) is True
+ assert _decode.decoding_required(parameters({"valid_min": 1})) is True
+ assert _decode.decoding_required(parameters({"scale_factor": 2})) is True
+
+
+class Test__mask_by_sentinels:
+ def test_a_sentinel_is_masked(self):
+ block = np.array([0, 1, -999], dtype="i2")
+ result = _decode._mask_by_sentinels(block, (np.int16(-999),))
+ assert result.tolist() == [False, False, True]
+
+ def test_mask_by_a_nan_sentinel(self):
+ """NaN never equals itself, so a NaN sentinel needs `isnan`, not `==`.
+
+ The base64 `_FillValue` nearly every xarray-written store carries is NaN,
+ so this is the common case rather than the exotic one.
+ """
+ block = np.array([1.0, np.nan], dtype="f4")
+ result = _decode._mask_by_sentinels(block, (np.float32("nan"),))
+ assert result.tolist() == [False, True]
+
+
+class Test__mask_by_valid_range:
+ def test_values_outside_the_range_are_masked(self):
+ block = np.array([-1, 5, 11], dtype="i2")
+ result = _decode._mask_by_valid_range(block, 0, 10)
+ assert result.tolist() == [True, False, True]
+
+
+class Test__unpack:
+ def test_scale_and_offset_are_applied(self):
+ block = np.array([1, 2], dtype="u2")
+ result = _decode._unpack(block, 0.5, 1.0, np.dtype("f8"))
+ np.testing.assert_allclose(result, [1.5, 2.0])
+ assert result.dtype == np.dtype("f8")
+
+
+class Test_mask_and_scale:
+ def test_mask_and_scale_masks_the_packed_values(self):
+ """Masking is applied before unpacking, as netCDF4 does it.
+
+ CF section 2.5.1: `_FillValue` is "of the same type as its variable", so
+ it is a packed value and comparing it with an unpacked one matches
+ nothing.
+ """
+ block = np.array([-999, 1, 2], dtype="i2")
+ result = _decode.mask_and_scale(
+ block,
+ sentinels=(np.int16(-999),),
+ valid_min=None,
+ valid_max=None,
+ scale_factor=0.5,
+ add_offset=0.0,
+ dtype=np.dtype("f8"),
+ )
+ assert result.mask.tolist() == [True, False, False]
+ np.testing.assert_allclose(result.compressed(), [0.5, 1.0])
+
+ def test_mask_and_scale_always_returns_a_masked_array(self):
+ # The Dask meta says masked, so every block must be, mask or no mask.
+ block = np.array([1, 2], dtype="i2")
+ result = _decode.mask_and_scale(
+ block,
+ sentinels=(np.int16(-999),),
+ valid_min=None,
+ valid_max=None,
+ scale_factor=None,
+ add_offset=None,
+ dtype=np.dtype("i2"),
+ )
+ assert isinstance(result, np.ma.MaskedArray)
+ assert not result.mask.any()
diff --git a/lib/iris/tests/unit/fileformats/zarr/test_dependency_errors.py b/lib/iris/tests/unit/fileformats/zarr/test_dependency_errors.py
new file mode 100644
index 0000000000..f4de235473
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/test_dependency_errors.py
@@ -0,0 +1,129 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Both Zarr load routes report a clear error when ``zarr`` cannot be used.
+
+``iris.load`` reaches :mod:`iris.fileformats.zarr` through
+:func:`iris.fileformats._load_zarr`; the Zarr page of the user manual also
+tells a reader to call :func:`iris.fileformats.zarr.load_cubes` directly, for
+the ``group=`` keyword. Both must raise the same ``ImportError``, naming the
+package and the version floor, whether ``zarr`` is missing entirely or merely
+too old.
+"""
+
+import subprocess
+import sys
+
+import pytest
+
+import iris
+import iris.fileformats.zarr as zarr_package
+from iris.fileformats.zarr import _MISSING_MESSAGE
+
+#: Blocks `import zarr` in the interpreter this is run in, without touching
+#: the installed distribution's metadata - the only way to simulate "not
+#: installed" for the import system while leaving `importlib.metadata`
+#: answering as it does on a real machine with zarr present.
+_BLOCK_ZARR = """
+import sys
+
+
+class _BlockZarr:
+ def find_spec(self, name, path, target=None):
+ if name == "zarr" or name.startswith("zarr."):
+ raise ImportError(f"No module named {name!r}")
+ return None
+
+
+sys.meta_path.insert(0, _BlockZarr())
+"""
+
+
+def _run(statement):
+ """Run `statement` in a fresh interpreter; return what it printed."""
+ completed = subprocess.run(
+ [sys.executable, "-c", statement], capture_output=True, text=True
+ )
+ assert completed.returncode == 0, completed.stderr
+ return completed.stdout.strip()
+
+
+def _report_old_zarr(name):
+ """Stand in for `importlib.metadata.version`, answering below the floor."""
+ return "2.18.0"
+
+
+#: What the probe subprocess runs, once `_BLOCK_ZARR` has taken zarr away.
+#:
+#: Kept at module level, and so at column zero, because it is source for
+#: another interpreter: indenting it to sit inside a test body would indent
+#: the code the subprocess has to compile.
+_LOAD_PROBE = """
+import iris
+try:
+ iris.load({location!r})
+except ImportError as exception:
+ print(exception)
+"""
+
+_LOAD_CUBES_PROBE = """
+import iris.fileformats.zarr
+try:
+ iris.fileformats.zarr.load_cubes({location!r})
+except ImportError as exception:
+ print(exception)
+"""
+
+
+class Test__MISSING_MESSAGE:
+ def test_it_names_the_package_and_the_version_floor(self):
+ assert "zarr" in _MISSING_MESSAGE
+ assert "3.0.8" in _MISSING_MESSAGE
+
+
+class Test_zarr_is_missing:
+ def test_iris_load_reports_it(self, tmp_path):
+ store = tmp_path / "s.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text("{}")
+
+ probe = _BLOCK_ZARR + _LOAD_PROBE.format(location=str(store))
+ assert _run(probe) == _MISSING_MESSAGE
+
+ def test_load_cubes_reports_it(self, tmp_path):
+ store = tmp_path / "s.zarr"
+ store.mkdir()
+
+ probe = _BLOCK_ZARR + _LOAD_CUBES_PROBE.format(location=str(store))
+ assert _run(probe) == _MISSING_MESSAGE
+
+
+class Test_zarr_is_too_old:
+ def test_iris_load_reports_it(self, mocker, tmp_path):
+ """Zarr 2.x imports fine, so only a version check catches it.
+
+ Without this, `ZarrDataset.__init__` reaches
+ `self._group.metadata.zarr_format` against a zarr 2 `Group`, which
+ has no `.metadata`, and the user sees a bare `AttributeError`
+ instead.
+ """
+ mocker.patch.object(
+ zarr_package.importlib.metadata, "version", _report_old_zarr
+ )
+ store = tmp_path / "s.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text("{}")
+
+ with pytest.raises(ImportError) as excinfo:
+ iris.load(str(store))
+ assert str(excinfo.value) == _MISSING_MESSAGE
+
+ def test_load_cubes_reports_it(self, mocker):
+ mocker.patch.object(
+ zarr_package.importlib.metadata, "version", _report_old_zarr
+ )
+
+ with pytest.raises(ImportError) as excinfo:
+ zarr_package.load_cubes("irrelevant")
+ assert str(excinfo.value) == _MISSING_MESSAGE
diff --git a/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py b/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py
new file mode 100644
index 0000000000..e993e2021c
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/test_optional_dependency.py
@@ -0,0 +1,37 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Iris must import, and load netCDF, without zarr-python installed."""
+
+import subprocess
+import sys
+
+
+def _imports_zarr(statement):
+ """Run `statement` in a fresh interpreter; report whether zarr was imported."""
+ probe = f"{statement}\nimport sys\nprint('zarr' in sys.modules)\n"
+ completed = subprocess.run(
+ [sys.executable, "-c", probe],
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+ return completed.stdout.strip() == "True"
+
+
+def test_importing_iris_does_not_import_zarr():
+ assert not _imports_zarr("import iris")
+
+
+def test_importing_the_format_registry_does_not_import_zarr():
+ # iris.fileformats imports every format module it registers a handler
+ # from. If the Zarr loader joined that list, Iris would stop installing
+ # without zarr-python.
+ assert not _imports_zarr("import iris.fileformats")
+
+
+def test_importing_the_zarr_package_does_not_import_zarr():
+ # The package docstring is reachable without the dependency; only the
+ # modules under it need zarr itself.
+ assert not _imports_zarr("import iris.fileformats.zarr")
diff --git a/lib/iris/tests/unit/fileformats/zarr/test_stores.py b/lib/iris/tests/unit/fileformats/zarr/test_stores.py
new file mode 100644
index 0000000000..2cc6200f01
--- /dev/null
+++ b/lib/iris/tests/unit/fileformats/zarr/test_stores.py
@@ -0,0 +1,26 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""The builders in :mod:`stores` produce the stores the rest of these tests
+ask questions of; this checks they build both Zarr versions correctly.
+"""
+
+import zarr
+
+from . import stores
+
+
+def test_make_store_version_3(tmp_path):
+ location = stores.make_store(tmp_path / "v3.zarr")
+ group = zarr.open_group(store=location, mode="r")
+ assert group["air_temperature"].metadata.zarr_format == 3
+ assert group["air_temperature"].metadata.dimension_names == ("y", "x")
+
+
+def test_make_store_version_2(tmp_path):
+ location = stores.make_store(tmp_path / "v2.zarr", zarr_format=2)
+ group = zarr.open_group(store=location, mode="r")
+ array = group["air_temperature"]
+ assert array.metadata.zarr_format == 2
+ assert array.attrs[stores.ARRAY_DIMENSIONS] == ["y", "x"]
diff --git a/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py b/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py
new file mode 100644
index 0000000000..1cb6cc7ad4
--- /dev/null
+++ b/lib/iris/tests/unit/io/format_picker/test_ZarrStore.py
@@ -0,0 +1,168 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Recognising a Zarr store, locally by its marker file and remotely by its URL."""
+
+import pytest
+
+from iris.fileformats import _is_zarr_store
+from iris.io.format_picker import ZarrStore
+from iris.tests import _shared_utils
+
+
+def test_the_element_carries_the_whole_target():
+ # Not the basename: recognising a local store means reading a file inside
+ # it, so the element needs the path, not the last component of it.
+ element = ZarrStore()
+ assert element.requires_fh is False
+ assert element.get_element("/data/store.zarr", None) == "/data/store.zarr"
+
+
+def test_a_file_handle_rules_the_element_out(tmp_path):
+ """No store is a single seekable file, so a handle settles the question.
+
+ It also says the caller is `iris.io._load_files`, which passes a basename
+ rather than a path - and a basename resolves against the working
+ directory, which is not where the file came from.
+ """
+ path = tmp_path / "data.nc"
+ path.write_bytes(b"CDF\x01")
+ element = ZarrStore()
+ with open(path, "rb") as handle:
+ assert element.get_element(path.name, handle) is None
+
+
+def test_the_agent_ignores_a_store_sharing_a_files_name(tmp_path, monkeypatch):
+ """A regression test for the working directory deciding a file's format.
+
+ Given a directory named `weather` in the working directory that happens
+ to be a Zarr store, a real NIMROD file called `weather` somewhere else was
+ classified as Zarr, because file dispatch supplies only the basename.
+
+ NIMROD rather than netCDF because the Zarr specification has to be
+ consulted for the bug to bite: specifications are tried by descending
+ priority and then by name, so priority 5 "NetCDF" is answered before
+ priority 5 "Zarr" either way, while priority 3 "NIMROD" is not.
+ """
+ import struct
+
+ from iris.fileformats import FORMAT_AGENT
+
+ working = tmp_path / "working"
+ decoy = working / "weather"
+ decoy.mkdir(parents=True)
+ (decoy / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}')
+ monkeypatch.chdir(working)
+
+ elsewhere = tmp_path / "elsewhere"
+ elsewhere.mkdir()
+ path = elsewhere / "weather"
+ path.write_bytes(struct.pack(">l", 512) + b"\x00" * 600)
+
+ with open(path, "rb") as handle:
+ spec = FORMAT_AGENT.get_spec(path.name, handle)
+ assert spec.name == "NIMROD"
+
+
+def test_a_version_3_store_is_recognised(tmp_path):
+ store = tmp_path / "v3.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}')
+ assert _is_zarr_store(str(store)) is True
+
+
+def test_a_version_2_store_is_recognised(tmp_path):
+ store = tmp_path / "v2.zarr"
+ store.mkdir()
+ (store / ".zgroup").write_text('{"zarr_format": 2}')
+ assert _is_zarr_store(str(store)) is True
+
+
+def test_a_directory_named_zarr_with_no_marker_is_not_a_store(tmp_path):
+ """The suffix alone is not the local test, and must not become one.
+
+ Reading the marker file is what distinguishes a real store from a
+ directory someone happened to name `.zarr`, and locally that costs one
+ stat.
+ """
+ store = tmp_path / "pretend.zarr"
+ store.mkdir()
+ assert _is_zarr_store(str(store)) is False
+
+
+def test_a_plain_file_is_not_a_store(tmp_path):
+ path = tmp_path / "data.nc"
+ path.write_bytes(b"CDF\x01")
+ assert _is_zarr_store(str(path)) is False
+
+
+@pytest.mark.parametrize("scheme", ["s3", "gs", "az", "abfs"])
+def test_a_remote_store_is_recognised_by_scheme_and_suffix(scheme):
+ assert _is_zarr_store(f"{scheme}://bucket/path/data.zarr") is True
+ assert _is_zarr_store(f"{scheme}://bucket/path/data.zarr/") is True
+
+
+def test_a_remote_url_without_the_suffix_is_not_a_store():
+ # Remotely the suffix is all there is to go on: reading a marker file
+ # would cost a network round trip before Iris knows it wants the target.
+ assert _is_zarr_store("s3://bucket/path/data") is False
+
+
+def test_an_http_url_is_not_a_store():
+ # OPeNDAP, which the netCDF loader already claims.
+ assert _is_zarr_store("https://example.invalid/data.zarr") is False
+
+
+def test_an_nczarr_url_is_not_a_native_store(tmp_path):
+ """A user who writes `#mode=nczarr` has asked for NCZarr by name.
+
+ The store on disk is a real one, so the marker-file test says yes unless
+ the fragment is checked first - and `load_http` passes the fragment
+ through to the format agent.
+ """
+ store = tmp_path / "s.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}')
+ assert _is_zarr_store(f"file://{store}#mode=nczarr,file") is False
+
+
+def test_a_non_string_target_is_not_a_store():
+ # `load_data_objects` passes open datasets, not paths.
+ assert _is_zarr_store(object()) is False
+
+
+@_shared_utils.skip_data
+def test_the_agent_still_picks_netcdf_for_a_netcdf_file():
+ """A `requires_fh=False` spec is consulted for every target with no buffer.
+
+ Adding one risks claiming a target another format already handles, and
+ the format agent returns the first match rather than the best.
+ """
+ from iris.fileformats import FORMAT_AGENT
+
+ path = _shared_utils.get_data_path(
+ ("NetCDF", "global", "xyt", "SMALL_hires_wind_u_for_ipcc4.nc")
+ )
+ with open(path, "rb") as handle:
+ spec = FORMAT_AGENT.get_spec("SMALL_hires_wind_u_for_ipcc4.nc", handle)
+ assert spec.name.startswith("NetCDF")
+
+
+def test_the_agent_picks_zarr_for_a_store(tmp_path):
+ from iris.fileformats import FORMAT_AGENT
+
+ store = tmp_path / "s.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}')
+ assert FORMAT_AGENT.get_spec(str(store), None).name == "Zarr"
+
+
+def test_the_agent_still_picks_nczarr_for_an_nczarr_url(tmp_path):
+ from iris.fileformats import FORMAT_AGENT
+
+ store = tmp_path / "s.zarr"
+ store.mkdir()
+ (store / "zarr.json").write_text('{"zarr_format": 3, "node_type": "group"}')
+ uri = f"file://{store}#mode=nczarr,file"
+ assert FORMAT_AGENT.get_spec(uri, None).name == "NcZarr"
diff --git a/lib/iris/tests/unit/io/test__generate_cubes.py b/lib/iris/tests/unit/io/test__generate_cubes.py
index 406ea2fa32..b207955a83 100644
--- a/lib/iris/tests/unit/io/test__generate_cubes.py
+++ b/lib/iris/tests/unit/io/test__generate_cubes.py
@@ -22,7 +22,10 @@ def test_pathlib_paths(self, mocker):
decode_uri_mock = mocker.patch(
"iris.iris.io.decode_uri", return_value=("file", None, None)
)
- mocker.patch("iris.iris.io.load_files")
+ # `decode_uri` always reporting a `None` part is not a real path, so
+ # the file-or-store partition has nothing to resolve or classify.
+ mocker.patch("iris.iris.io.expand_filespecs", return_value=[])
+ mocker.patch("iris.iris.io._load_files")
for gc_arg, du_arg in test_variants:
decode_uri_mock.reset_mock()
@@ -38,7 +41,7 @@ def test_pathlib_paths(self, mocker):
],
)
def test_nczarr_uri(self, uri, mocker):
- load_files_mock = mocker.patch("iris.iris.io.load_files", return_value=[])
+ load_files_mock = mocker.patch("iris.iris.io._load_files", return_value=[])
load_http_mock = mocker.patch("iris.iris.io.load_http", return_value=[])
list(_generate_cubes(uri, None, None))
@@ -48,7 +51,12 @@ def test_nczarr_uri(self, uri, mocker):
def test_non_nczarr_uri(self, mocker):
uri = "file:///foo#bar=baz"
- load_files_mock = mocker.patch("iris.iris.io.load_files", return_value=[])
+ # The partition needs a real-looking part name to classify, not a
+ # file-system lookup against a path that was never written.
+ mocker.patch("iris.iris.io.expand_filespecs", side_effect=lambda names: names)
+ # The "file" branch calls the private, non-expanding `_load_files`
+ # directly, since it has already expanded - see its docstring.
+ load_files_mock = mocker.patch("iris.iris.io._load_files", return_value=[])
load_http_mock = mocker.patch("iris.iris.io.load_http", return_value=[])
list(_generate_cubes(uri, None, None))
diff --git a/lib/iris/tests/unit/io/test_expand_filespecs.py b/lib/iris/tests/unit/io/test_expand_filespecs.py
index 0a33f87c15..9ff879867e 100644
--- a/lib/iris/tests/unit/io/test_expand_filespecs.py
+++ b/lib/iris/tests/unit/io/test_expand_filespecs.py
@@ -105,3 +105,14 @@ def test_false_bool_relative(self):
assert item_out == item_in
finally:
os.chdir(cwd)
+
+ def test_remote_store_uri_is_passed_through_unresolved(self):
+ # `.absolute()` would otherwise turn this into
+ # "/<cwd>/s3:/bucket/data.zarr", which zarr-python cannot open.
+ uri = "s3://bucket/data.zarr"
+ assert iio.expand_filespecs([uri]) == [uri]
+
+ def test_remote_store_uri_is_not_globbed(self):
+ # A remote URI stands for itself: Iris does not list its neighbours.
+ uri = "s3://bucket/data.zarr"
+ assert iio.expand_filespecs([uri], False) == [uri]
diff --git a/lib/iris/tests/unit/io/test_load_stores.py b/lib/iris/tests/unit/io/test_load_stores.py
new file mode 100644
index 0000000000..8d2196e07d
--- /dev/null
+++ b/lib/iris/tests/unit/io/test_load_stores.py
@@ -0,0 +1,155 @@
+# Copyright Iris contributors
+#
+# This file is part of Iris and is released under the BSD license.
+# See LICENSE in the root of the repository for full licensing details.
+"""Unit tests for :func:`iris.io.load_stores`, and the load chain it joins."""
+
+import numpy as np
+import pytest
+
+import iris
+from iris.io import _is_remote_store_uri, load_stores
+from iris.tests import _shared_utils
+
+zarr = pytest.importorskip("zarr", minversion="3.0.8")
+
+from iris.tests.unit.fileformats.zarr import stores # noqa: E402
+
+
+class Test_load_stores:
+ def test_a_local_store_loads(self, tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ cubes = list(load_stores([location], None))
+ assert [cube.var_name for cube in cubes] == ["air_temperature"]
+
+ def test_several_stores_in_one_call(self, tmp_path):
+ first = stores.make_store(tmp_path / "a.zarr")
+ second = stores.make_store(tmp_path / "b.zarr")
+ assert len(list(load_stores([first, second], None))) == 2
+
+ def test_constraints_reach_the_handler(self, tmp_path):
+ """A constraint the handler can pre-filter on rules out a data variable.
+
+ `load_stores` forwards `constraints` to a constraint-aware handler
+ rather than applying it itself - the general `cube_func` form is only
+ enforced later, by `_CubeFilter.add` - so the constraint used here has
+ to be one `_translate_constraints_to_var_callback` can act on: a
+ `NameConstraint` with no `STASH` component.
+ """
+ location = stores.make_store(tmp_path / "s.zarr")
+ constraint = iris.NameConstraint(var_name="not_air_temperature")
+ assert list(load_stores([location], None, constraint)) == []
+
+
+class Test__is_remote_store_uri:
+ @pytest.mark.parametrize(
+ ("uri", "expected"),
+ [
+ ("s3://bucket/data.zarr", True),
+ ("gs://bucket/data.zarr", True),
+ ("az://container/data.zarr", True),
+ ("abfs://container/data.zarr", True),
+ ("https://example.invalid/data.nc", False),
+ ("/local/path/data.zarr", False),
+ ("relative/data.zarr", False),
+ ],
+ )
+ def test_which_uris_are_remote_stores(self, uri, expected):
+ assert _is_remote_store_uri(uri) is expected
+
+
+class Test_iris_load:
+ def test_accepts_a_local_store(self, tmp_path):
+ location = stores.make_store(tmp_path / "s.zarr")
+ cubes = iris.load(location)
+ assert len(cubes) == 1
+ np.testing.assert_array_equal(cubes[0].data, stores.DATA)
+
+ def test_accepts_a_remote_url_and_does_not_absolutise_it(self, mocker):
+ """A remote URL must reach `load_stores` unmangled, not as a local path.
+
+ `expand_filespecs` would otherwise turn `s3://bucket/data.zarr` into
+ `/<cwd>/s3:/bucket/data.zarr`. The dispatch is what is under test, so
+ `load_stores` itself is mocked out: no store library is invoked, and
+ no network is touched, whatever is installed alongside zarr-python.
+ """
+ mock_load_stores = mocker.patch("iris.io.load_stores", return_value=iter(()))
+ iris.load("s3://bucket/data.zarr")
+ mock_load_stores.assert_called_once_with(["s3://bucket/data.zarr"], None, None)
+
+ @_shared_utils.skip_data
+ def test_a_glob_matching_both_files_and_stores(self, tmp_path):
+ """`iris.load("data/*")` must accept a mix.
+
+ The file-or-store decision happens after glob expansion and before
+ format dispatch. Mis-partitioning raises IsADirectoryError from
+ `load_files` for the stores while the netCDF files load fine, which
+ reads as an Iris bug to a user who did not expect the glob to match a
+ store.
+ """
+ source = iris.load_cube(
+ _shared_utils.get_data_path(
+ ("NetCDF", "global", "xyt", "SMALL_hires_wind_u_for_ipcc4.nc")
+ )
+ )
+ iris.save(source, str(tmp_path / "a.nc"))
+ stores.make_store(tmp_path / "b.zarr")
+
+ cubes = iris.load(str(tmp_path / "*"))
+ names = sorted(cube.var_name or cube.name() for cube in cubes)
+ assert "air_temperature" in names
+ assert len(cubes) == 2
+
+
+class Test_load_files:
+ def test_still_expands_a_glob(self, tmp_path, mocker):
+ """Public `load_files` keeps its promise: filenames may be globs.
+
+ The duplicate-collapsing bug fixed alongside `load_stores` lives in
+ calling `expand_filespecs` twice, once in a caller and again inside
+ `load_files` - not in `load_files` expanding at all. Pinning this
+ guards against quietly narrowing the public contract while fixing
+ that.
+ """
+ from iris.io import load_files
+
+ (tmp_path / "a.nc").write_bytes(b"")
+ (tmp_path / "b.nc").write_bytes(b"")
+ mock_load = mocker.patch("iris.io._load_files", return_value=iter(()))
+
+ list(load_files([str(tmp_path / "*.nc")], None))
+
+ mock_load.assert_called_once_with(
+ [str(tmp_path / "a.nc"), str(tmp_path / "b.nc")], None, None
+ )
+
+
+class Test_saving:
+ def test_find_saver_tolerates_a_trailing_separator(self):
+ from iris.io import find_saver
+
+ assert find_saver("out.nc") is find_saver("out.nc/")
+
+ @_shared_utils.skip_data
+ def test_saving_a_cubelist_to_netcdf_still_takes_one_call(self, tmp_path):
+ cubes = iris.load(
+ _shared_utils.get_data_path(
+ ("NetCDF", "global", "xyt", "SMALL_hires_wind_u_for_ipcc4.nc")
+ )
+ )
+ iris.save(cubes, str(tmp_path / "out.nc"))
+ assert len(iris.load(str(tmp_path / "out.nc"))) == len(cubes)
+
+
+class Test_load_meshes:
+ def test_a_store_holding_no_mesh_yields_no_meshes(self, tmp_path):
+ """Reaching a Zarr store is what was new here.
+
+ What must not happen is the `ValueError` the scheme dispatch raised
+ for anything that was not a file. Reading a mesh *out* of a store is
+ covered by `unit/fileformats/zarr/loader/test_load_cubes.py::TestMesh`.
+ """
+ from iris.mesh import load_meshes
+
+ stores.make_store(tmp_path / "s.zarr")
+ assert load_meshes(str(tmp_path / "s.zarr")) == {}
diff --git a/lib/iris/tests/unit/io/test_save.py b/lib/iris/tests/unit/io/test_save.py
index 7010d4b67f..3662c8d3c6 100644
--- a/lib/iris/tests/unit/io/test_save.py
+++ b/lib/iris/tests/unit/io/test_save.py
@@ -6,8 +6,11 @@
from pathlib import Path
+import pytest
+
import iris
from iris.cube import Cube
+import iris.fileformats.netcdf.saver
class TestSave:
@@ -39,3 +42,49 @@ def replace_expand(file_specs, files_expected=True):
print("ValueError")
pass
find_saver_mock.assert_called_with(fs_val)
+
+
+class TestCubeListSavers:
+ """Which savers take a `CubeList` whole, rather than one cube at a time.
+
+ `iris.io._savers.cubelist_savers` holds saver functions, not extensions,
+ because `save` is reached with savers it never looked up by extension:
+ one passed by the caller, and the one chosen directly for an NCZarr
+ target. Keyed by extension, both would fall to the `append=True` path,
+ and the netCDF saver has no `append` parameter to fall to.
+ """
+
+ def test_the_netcdf_saver_is_registered(self):
+ """Registered by identity, so no name has to be matched.
+
+ Two earlier forms matched on the module name: a substring test, true
+ of any module whose path happened to contain `iris.fileformats.netcdf`,
+ and then an exact set of module strings. Neither could be checked by a
+ reader against the function actually being called.
+ """
+ from iris.fileformats.netcdf.saver import save as netcdf_save
+
+ iris.io._check_init_savers()
+ assert netcdf_save in iris.io._savers.cubelist_savers
+
+ def test_a_saver_passed_directly_is_recognised(self, tmp_path):
+ """The path that never reaches `find_saver`.
+
+ `netcdf.save` has no `append` parameter, so if the registry were
+ keyed by extension and missed this saver, the fallback would raise
+ rather than quietly saving one cube at a time.
+ """
+ from iris.fileformats.netcdf.saver import save as netcdf_save
+
+ cubes = iris.cube.CubeList([Cube([1], long_name="a"), Cube([2], long_name="b")])
+ target = tmp_path / "out.nc"
+ iris.save(cubes, str(target), saver=netcdf_save)
+ assert len(iris.load(str(target))) == 2
+
+ def test_an_unregistered_saver_still_needs_append(self, tmp_path):
+ def saver_without_append(cube, target):
+ pass
+
+ cubes = iris.cube.CubeList([Cube([1], long_name="a"), Cube([2], long_name="b")])
+ with pytest.raises(ValueError, match="Cannot append cubes"):
+ iris.save(cubes, str(tmp_path / "out.xyz"), saver=saver_without_append)
diff --git a/lib/iris/tests/unit/lazy_data/test_as_lazy_data.py b/lib/iris/tests/unit/lazy_data/test_as_lazy_data.py
index a0c13208ee..19a96767c1 100644
--- a/lib/iris/tests/unit/lazy_data/test_as_lazy_data.py
+++ b/lib/iris/tests/unit/lazy_data/test_as_lazy_data.py
@@ -4,6 +4,7 @@
# See LICENSE in the root of the repository for full licensing details.
"""Test the function :func:`iris._lazy data.as_lazy_data`."""
+from types import SimpleNamespace
from unittest import mock
import dask.array as da
@@ -12,7 +13,8 @@
import numpy.ma as ma
import pytest
-from iris._lazy_data import _optimum_chunksize, as_lazy_data
+from iris._lazy_data import CACHE, _optimum_chunksize, as_lazy_data
+from iris.fileformats.netcdf._thread_safe_nc import NetCDFDataProxy
class Test_as_lazy_data:
@@ -66,6 +68,67 @@ class MyProxy:
as_lazy_data(data)
+class Test_as_lazy_data__cache_key:
+ @staticmethod
+ def _proxy(shape=(2, 3, 4), dtype="f4"):
+ # NetCDFDataProxy reads only `shape` and `name` from what it is given,
+ # so it can be built without a file as long as it is never indexed.
+ return NetCDFDataProxy(
+ SimpleNamespace(shape=shape, name="air_temperature"),
+ np.dtype(dtype),
+ "/does/not/exist.nc",
+ -1.0,
+ )
+
+ def test_key_is_the_caller_s_string(self):
+ proxy = self._proxy()
+ CACHE._cache.clear()
+ as_lazy_data(proxy, meta=proxy.dask_meta, cache_key=repr(proxy))
+ (key,) = CACHE._cache
+ assert key == (
+ repr(proxy),
+ (2, 3, 4),
+ False,
+ proxy.dask_meta.dtype,
+ type(proxy.dask_meta),
+ )
+
+ def test_no_key_means_no_caching(self):
+ proxy = self._proxy()
+ CACHE._cache.clear()
+ as_lazy_data(proxy, meta=proxy.dask_meta)
+ assert len(CACHE._cache) == 0
+
+ def test_second_call_reuses_the_cached_array(self, mocker):
+ proxy = self._proxy()
+ CACHE._cache.clear()
+ from_array = mocker.patch("iris._lazy_data.da.from_array", wraps=da.from_array)
+ as_lazy_data(proxy, meta=proxy.dask_meta, cache_key=repr(proxy))
+ as_lazy_data(proxy, meta=proxy.dask_meta, cache_key=repr(proxy))
+ assert from_array.call_count == 1
+
+ def test_key_without_meta_is_refused(self):
+ data = np.arange(24).reshape((2, 3, 4))
+ with pytest.raises(ValueError, match="cache_key"):
+ as_lazy_data(data, cache_key="anything")
+
+
+def test_lazy_data_imports_no_file_format():
+ """Pin spec section 4.4: `_lazy_data` knows about no file format at all."""
+ import inspect
+ import re
+
+ import iris._lazy_data
+
+ source = inspect.getsource(iris._lazy_data)
+ offenders = [
+ line
+ for line in source.splitlines()
+ if re.match(r"\s*(from|import)\s", line) and "iris.fileformats" in line
+ ]
+ assert offenders == []
+
+
class Test__optimised_chunks:
# Stable, known chunksize for testing.
FIXED_CHUNKSIZE_LIMIT = 1024 * 1024 * 64
diff --git a/requirements/locks/py312-linux-64.lock b/requirements/locks/py312-linux-64.lock
index ad5e7ac1db..f77bde657f 100644
--- a/requirements/locks/py312-linux-64.lock
+++ b/requirements/locks/py312-linux-64.lock
@@ -1,6 +1,6 @@
# Generated by conda-lock.
# platform: linux-64
-# input_hash: f0e9da59603dbd6c5dc489438e9e4859a14009bcd2e087d8aece9a1e35d35255
+# input_hash: 882352f1d7a8b3e3b415f119530f26065dfc71c10354593ad1ecc6a92f1282cf
@EXPLICIT
https://conda.anaconda.org/conda-forge/linux-64/eigen-abi-5.0.1.80-hf414acd_3.conda#56a1cca9a4a7bed2999630e5644c8d0e
https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab24e00_0.tar.bz2#0c96522c6bdaed4b1566d11387caaf45
@@ -23,10 +23,10 @@ https://conda.anaconda.org/conda-forge/linux-64/libegl-1.7.0-ha4b6fd6_5.conda#3a
https://conda.anaconda.org/conda-forge/linux-64/libopengl-1.7.0-ha4b6fd6_5.conda#d207a2e9bae9da476bc0a603215d5167
https://conda.anaconda.org/conda-forge/linux-64/zlib-1.3.2-h25fd6f3_3.conda#6acb86426229f96f93e5468d1df3a5e8
https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.7-hb78ec9c_7.conda#aa459086047c0e5e27023ab19f8cb86a
-https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.46.1-default_hbd61a6d_102.conda#449500f2c089da11c40f5c21312e3e07
+https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.46.1-default_hbd61a6d_103.conda#2362c70e7e79827c32db35547d9c179b
https://conda.anaconda.org/conda-forge/linux-64/libgcc-16.2.0-ha9f2e26_7.conda#735629b400b70eeab21a2ba8f5b1c2ce
https://conda.anaconda.org/conda-forge/linux-64/alsa-lib-1.2.16.1-h7cc23a3_1.conda#7094e0d8d14de0eff6d83f0d2f1f661e
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-common-0.14.3-hb03c661_0.conda#ce8664d7c01d4a598ad38107b289b5ec
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-common-1.0.3-hebe6cf0_0.conda#cdb5797968bfe1737f1b71f525fc6518
https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-hda65f42_10.conda#e675fabcf81499adc7edf58124fb1e01
https://conda.anaconda.org/conda-forge/linux-64/c-ares-1.34.8-hebe6cf0_2.conda#3ad2b403be8fc5612b9159e2ce621f69
https://conda.anaconda.org/conda-forge/linux-64/fribidi-1.0.17-h7cc23a3_0.conda#f9d5302bca196ee87e62f393f2626562
@@ -47,14 +47,15 @@ https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.1-hb9d3cd8_1.conda#d8
https://conda.anaconda.org/conda-forge/linux-64/libntlm-1.8-hb9d3cd8_0.conda#7c7927b404672409d9917d49bff5f2d6
https://conda.anaconda.org/conda-forge/linux-64/libogg-1.3.5-hd0c01bc_1.conda#68e52064ed3897463c0e958ab5c8f91b
https://conda.anaconda.org/conda-forge/linux-64/libpciaccess-0.19-hb03c661_1.conda#35fa2b34bbced424e6976d30f5fde576
-https://conda.anaconda.org/conda-forge/linux-64/libpng-1.6.58-h922cc85_1.conda#fcf71c8d979148873f6f8ad4cfc73d86
+https://conda.anaconda.org/conda-forge/linux-64/libpng-1.6.59-h922cc85_0.conda#8c9caf742a485ac688ffa88bbadbf0be
+https://conda.anaconda.org/conda-forge/linux-64/libpython-3.12.15-ha57cdbe_2_cpython.conda#fe256863f652549334a81a8e49fdf097
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-16.2.0-h934c35e_7.conda#9b6707e0caca917fa6ea9ee473f37a5d
https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.42.4-hcfc3c73_0.conda#54a948337e66706380ff24d6b5fde63f
https://conda.anaconda.org/conda-forge/linux-64/libwebp-base-1.6.0-hd42ef1d_1.conda#9332b53d0ea93c5d39e33be03a0c611a
https://conda.anaconda.org/conda-forge/linux-64/libxcrypt-4.4.38-h280c20c_0.conda#f7a7ff5a6ab331e037abd34f379a631d
https://conda.anaconda.org/conda-forge/linux-64/lzo-2.10-hebe6cf0_1003.conda#10e0aa58ed390ba598b728badf9eb1b7
https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.6-hdb14827_1.conda#ee6c0cd80a60961a1f48aa3e0b91f986
-https://conda.anaconda.org/conda-forge/linux-64/openssl-3.6.4-h781a0a9_0.conda#16e3034a330cc625f2c685edec60cf4e
+https://conda.anaconda.org/conda-forge/linux-64/openssl-3.6.5-h781a0a9_0.conda#5a51a2ea653f120240b1a9bf76e0aae9
https://conda.anaconda.org/conda-forge/linux-64/pthread-stubs-0.4-h7cc23a3_1004.conda#bb66b610707b811e3c65181a6431e7a8
https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_h1df4ec4_4.conda#676a2f9b1c4fcf57b70aa5c65f6c60d0
https://conda.anaconda.org/conda-forge/linux-64/xorg-libice-1.1.2-h280c20c_0.conda#85c9442aec283b4e464fa9ecc484a2f3
@@ -63,10 +64,10 @@ https://conda.anaconda.org/conda-forge/linux-64/xorg-libxdmcp-1.1.5-h7cc23a3_2.c
https://conda.anaconda.org/conda-forge/linux-64/xorg-xorgproto-2025.1-hebe6cf0_3.conda#c3d0a6f03ba3aa312777b41b20a71ec0
https://conda.anaconda.org/conda-forge/linux-64/xxhash-0.8.3-hec8ed23_1.conda#72ab4ea51529ec63ff010ae9fb62b02f
https://conda.anaconda.org/conda-forge/linux-64/yaml-0.2.5-hebe6cf0_3.conda#e741576fb8f89821ac7c1c537322a33d
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-cal-0.9.15-h6bbde05_1.conda#8f2b374c79908f271be3d0e9f1c462a8
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-compression-0.3.2-h01ee8f8_5.conda#ae862abdbb3160f97478d3d4749164c0
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-sdkutils-0.2.7-h01ee8f8_3.conda#412f0cf18cc4c4945655dba954ca2d76
-https://conda.anaconda.org/conda-forge/linux-64/aws-checksums-0.2.10-h01ee8f8_5.conda#4d70caf5260e0787675c3bf61ae5b64c
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-cal-1.0.0-hcf7459e_1.conda#2009b71b358163738f339c76640b512c
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-compression-1.0.0-ha404f16_1.conda#da0c7b86d5379eb44acfa084897e8b98
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-sdkutils-1.0.0-ha404f16_1.conda#cbbd999c74a030951365225909e376c7
+https://conda.anaconda.org/conda-forge/linux-64/aws-checksums-1.0.1-ha404f16_0.conda#10e6e2f11271e66dfad43eb002359f22
https://conda.anaconda.org/conda-forge/linux-64/cli11-2.7.2-h54a6638_1.conda#42761b55062088f6f7fb58d0633e0769
https://conda.anaconda.org/conda-forge/linux-64/double-conversion-3.4.0-hecca717_0.conda#dbe3ec0f120af456b3477743ffd99b74
https://conda.anaconda.org/conda-forge/linux-64/fmt-12.1.0-h76c4fd7_1.conda#a70bb6d42ad121aef456e0007e92bed6
@@ -79,11 +80,11 @@ https://conda.anaconda.org/conda-forge/linux-64/libaec-1.1.5-h088129d_0.conda#86
https://conda.anaconda.org/conda-forge/linux-64/libbrotlidec-1.2.0-ha411449_4.conda#0cfd601eb03cca403dcc248844787486
https://conda.anaconda.org/conda-forge/linux-64/libbrotlienc-1.2.0-h018ffa1_4.conda#4136f6e4ef4835cc7df39a707cbd511c
https://conda.anaconda.org/conda-forge/linux-64/libdrm-2.4.129-h7cc23a3_0.conda#64cc91512b6278c315349dc13c53f680
-https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20250104-pl5321h373387f_1.conda#50708d3b951d0f8e2d7f2df5b5edc040
+https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20260512-pl5321h373387f_0.conda#27dbd7e2984b8fe48e069379636ccfdd
https://conda.anaconda.org/conda-forge/linux-64/libfreetype6-2.14.3-h5e6c136_2.conda#b72a266a9317036fd0464cb98b027cd0
https://conda.anaconda.org/conda-forge/linux-64/libgfortran-16.2.0-h69a702a_7.conda#3a8d285f5f22138ef826ad54cf267e73
https://conda.anaconda.org/conda-forge/linux-64/libglu-9.0.3-h5888daf_1.conda#8422fcc9e5e172c91e99aef703b3ce65
-https://conda.anaconda.org/conda-forge/linux-64/libhwy-1.4.0-h57c4cff_1.conda#d58bfbaaf51ed85771eae92c2e7f5816
+https://conda.anaconda.org/conda-forge/linux-64/libhwy-1.4.0-h57c4cff_2.conda#145497b38fe581a7760dd88b7377c96b
https://conda.anaconda.org/conda-forge/linux-64/libnghttp2-1.68.1-h74cf4be_1.conda#75d211f04ac87506f1f43420712a69fe
https://conda.anaconda.org/conda-forge/linux-64/libpython-3.12.14-h0c77377_3_cpython.conda#d247b7632f09324c11f24b5270361385
https://conda.anaconda.org/conda-forge/linux-64/libssh2-1.11.1-h6154650_1.conda#5c526561e1d2a2e071941174eae84557
@@ -99,29 +100,30 @@ https://conda.anaconda.org/conda-forge/linux-64/pcre2-10.47-h8b3dc9c_1.conda#06f
https://conda.anaconda.org/conda-forge/linux-64/pixman-0.46.4-h54a6638_3.conda#0ee5bb30034b081a1386c1e2c98ab0a7
https://conda.anaconda.org/conda-forge/linux-64/pugixml-1.15-h3f63f65_0.conda#b11a4c6bf6f6f44e5e143f759ffa2087
https://conda.anaconda.org/conda-forge/linux-64/readline-8.3-hd6e31c0_1.conda#69c01c781e7c8190bbdaf79e3848c5ca
-https://conda.anaconda.org/conda-forge/linux-64/s2n-1.7.6-h7e3ee7f_0.conda#e112cadcd79412d81496e784884bddf0
+https://conda.anaconda.org/conda-forge/linux-64/s2n-1.7.7-ha5dc4ae_0.conda#4928f1e1964a6f38f0da349b9d0e1c6a
https://conda.anaconda.org/conda-forge/linux-64/snappy-1.2.2-h34e00bb_2.conda#2b7e31ffbd476eecf33b3b833692339c
https://conda.anaconda.org/conda-forge/linux-64/wayland-1.26.0-hc1c935e_2.conda#88a47e22b53e50865b1cabe2eb648352
https://conda.anaconda.org/conda-forge/linux-64/xorg-libsm-1.2.6-h0d788c3_1.conda#aa7459ed9ad086ba11dda843d764b33d
https://conda.anaconda.org/conda-forge/linux-64/zfp-1.0.1-h79e284d_6.conda#9d7141ec747a9ede5da83e864fcc8213
https://conda.anaconda.org/conda-forge/linux-64/zlib-ng-2.3.3-hce19668_1.conda#1726acec19beeed1c764385cd8622405
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-io-0.27.5-h4f381ba_1.conda#948f5a4fddac779b7942626e5807caba
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-io-1.0.0-h07ba609_0.conda#72593f68772d8c7e72c8a09681b0fe42
https://conda.anaconda.org/conda-forge/linux-64/blosc-1.21.6-he440d0b_1.conda#2c2fae981fd2afd00812c92ac47d023d
https://conda.anaconda.org/conda-forge/linux-64/brotli-bin-1.2.0-h9908984_4.conda#c48c09444f3736800ea0b9550c365baf
https://conda.anaconda.org/conda-forge/linux-64/freexl-2.0.0-h1f12c17_3.conda#4a0a8fd87ee4767db791047546f56501
https://conda.anaconda.org/conda-forge/linux-64/hdf4-4.2.15-h2a13503_7.conda#bd77f8da987968ec3927990495dc22e4
https://conda.anaconda.org/conda-forge/linux-64/krb5-1.22.2-hbc21106_2.conda#53318d715316929a574f83591308b1f8
+https://conda.anaconda.org/conda-forge/linux-64/libcrc32c-1.1.2-h9c3ff4c_0.tar.bz2#c965a5aa0d5c1c37ffc62dff36e28400
https://conda.anaconda.org/conda-forge/linux-64/libfreetype-2.14.3-ha770c72_2.conda#f8054e759d0ddddaf34a5c8fedc900a1
https://conda.anaconda.org/conda-forge/linux-64/libglib-2.90.0-h569388d_0.conda#c85e9539551f1fd21104e518d940c244
-https://conda.anaconda.org/conda-forge/linux-64/libjxl-0.12.0-heb2dce7_2.conda#485788d339785bac87fe86315b4a0627
+https://conda.anaconda.org/conda-forge/linux-64/libjxl-0.12.0-heb2dce7_4.conda#8770a39fd4c5559357efc7cdb8f0c742
https://conda.anaconda.org/conda-forge/linux-64/libmo_unpack-3.1.2-hf484d3e_1001.tar.bz2#95f32a6a5a666d33886ca5627239f03d
https://conda.anaconda.org/conda-forge/linux-64/libopenblas-0.3.34-pthreads_hf13c14d_2.conda#7e19fcd73d7f0997d3cb3326d2586a86
https://conda.anaconda.org/conda-forge/linux-64/libpsl-0.23.1-hd9e3e90_1.conda#77be60417aa572afcb48cbe0f967401d
https://conda.anaconda.org/conda-forge/linux-64/librttopo-1.1.0-h55b0958_21.conda#6c639697369e9669032ca08d0a566282
-https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.53.4-h13e7031_1.conda#e72bbec309c2b0f37823ee7d4fabfcd3
+https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.54.0-h13e7031_1.conda#bb145547395d3329f3a1d930de9848b2
https://conda.anaconda.org/conda-forge/linux-64/libtheora-1.2.0-h85c0a6d_0.conda#177ef038f292390abe69137711fc0cbf
https://conda.anaconda.org/conda-forge/linux-64/libtiff-4.7.2-hcc2c06a_1.conda#0907d876f460ebc941ae590338b6102f
-https://conda.anaconda.org/conda-forge/linux-64/libxml2-16-2.15.4-hf3af7cc_0.conda#b7545c57a73a51d72aa6eee5695cf051
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-16-2.15.4-hf3af7cc_1.conda#046d9eada488e1a1657c80b55c23da52
https://conda.anaconda.org/conda-forge/linux-64/qhull-2020.2-h434a139_5.conda#353823361b1d27eb3960efb076dfcaf6
https://conda.anaconda.org/conda-forge/linux-64/udunits2-2.2.28-h40f5838_3.conda#6bb8deb138f87c9d48320ac21b87e7a1
https://conda.anaconda.org/conda-forge/linux-64/uriparser-0.9.8-hac33072_0.conda#d71d3a66528853c0a1ac2c02d79a0284
@@ -132,7 +134,7 @@ https://conda.anaconda.org/conda-forge/linux-64/xcb-util-wm-0.4.2-hb711507_0.con
https://conda.anaconda.org/conda-forge/linux-64/xerces-c-3.3.0-h78fed68_2.conda#f2ff0de6c209e90c53b3a195c2708f5f
https://conda.anaconda.org/conda-forge/linux-64/xorg-libx11-1.8.13-he1eb515_1.conda#8c282bbe4808a3cc80a5c98e9aec1cfc
https://conda.anaconda.org/conda-forge/linux-64/atk-1.0-2.38.0-h04ea711_2.conda#f730d54ba9cd543666d7220c9f7ed563
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-http-0.11.0-hbe094ff_5.conda#c09f7ec7327b8bd52fc2d6eacaa18d3b
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-http-1.0.0-h392d1a3_1.conda#6701f6e712f9513e722ba57cde99f023
https://conda.anaconda.org/conda-forge/linux-64/brotli-1.2.0-h505cf86_4.conda#746aa619a1bcaf4da541101179d7c60e
https://conda.anaconda.org/conda-forge/linux-64/cyrus-sasl-2.1.28-hac629b4_1.conda#af491aae930edc096b58466c51c4126c
https://conda.anaconda.org/conda-forge/linux-64/dbus-1.16.2-he8c428d_2.conda#a6eb37b51be1a9a654dc3a8aca039b1a
@@ -148,10 +150,10 @@ https://conda.anaconda.org/conda-forge/linux-64/libcurl-8.22.0-ha042cf0_0.conda#
https://conda.anaconda.org/conda-forge/linux-64/libglx-1.7.0-ha4b6fd6_5.conda#614e10aae180f87b84f0d1ca8ceb7d9e
https://conda.anaconda.org/conda-forge/linux-64/libkml-1.3.0-haa4a5bd_1023.conda#953b7cca897e21215302dbfe2af5cd0c
https://conda.anaconda.org/conda-forge/linux-64/libxkbfile-1.2.0-hb03c661_0.conda#f330a1b107cb5861f26dbb72a26f33e8
-https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.15.4-h7df9aa5_0.conda#ea192d1ef556bf1d55de4acb8c0f5b65
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.15.4-h7df9aa5_1.conda#51e5162f95bd6840f6b2b3169125fff5
https://conda.anaconda.org/conda-forge/linux-64/openjpeg-2.5.4-heb1ab33_2.conda#a3f3ac7a68d01c198e276dbb99e62032
-https://conda.anaconda.org/conda-forge/linux-64/python-3.12.14-h5f976f7_3_cpython.conda#98be3cf76eca2e8871f907a03aed3b84
-https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.53.4-h9ffa6c4_1.conda#d42b24574925bfa3167efb3571c905ad
+https://conda.anaconda.org/conda-forge/linux-64/python-3.12.15-h5f976f7_2_cpython.conda#13cef25ee4055b1fec19d35d20aadda2
+https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.54.0-h9ffa6c4_1.conda#1060a8b69c6774ec85d81878eb299038
https://conda.anaconda.org/conda-forge/linux-64/xcb-util-image-0.4.0-hb711507_2.conda#a0901183f08b6c7107aab109733a3c91
https://conda.anaconda.org/conda-forge/linux-64/xkeyboard-config-2.48-h280c20c_0.conda#b233b41be0bf210989d57160ed39b394
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxext-1.3.7-h7cc23a3_1.conda#e5b6b28536b81b3f4cb20db4668a4642
@@ -160,34 +162,34 @@ https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrender-0.9.12-hb03c661_
https://conda.anaconda.org/conda-forge/noarch/aiohappyeyeballs-2.7.1-pyhd8ed1ab_0.conda#e51e09bf3b91c62b7461a9f94e92c2c9
https://conda.anaconda.org/conda-forge/noarch/alabaster-1.0.0-pyhd8ed1ab_1.conda#1fd9696649f65fd6611fcdb4ffec738a
https://conda.anaconda.org/conda-forge/noarch/attrs-26.1.0-pyhcf101f3_0.conda#c6b0543676ecb1fb2d7643941fe375f2
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-auth-0.10.4-h4610da3_2.conda#cefa84dbf94066e7a6902af288a2fedd
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-auth-1.0.0-h7648d90_2.conda#f4f169a23884de1bdb9c25ded8f2aec1
https://conda.anaconda.org/conda-forge/noarch/babel-2.18.0-pyhcf101f3_1.conda#f1976ce927373500cc19d3c0b2c85177
-https://conda.anaconda.org/conda-forge/linux-64/backports.zstd-1.7.0-py312h3f22e6b_1.conda#b0e9b44b494bb001c4d35b206022eba2
+https://conda.anaconda.org/conda-forge/linux-64/backports.zstd-1.8.0-py312h3f22e6b_0.conda#0f5fe32a47138697efdd6d8ee40a00d0
https://conda.anaconda.org/conda-forge/linux-64/brotli-python-1.2.0-py312he9c40d5_4.conda#edb667e7ce56106424e8afab2dc0f0cb
https://conda.anaconda.org/conda-forge/linux-64/cairo-1.18.6-h3c89d7e_0.conda#7920701f9abcb488411495742cf365a7
https://conda.anaconda.org/conda-forge/noarch/certifi-2026.7.22-pyhd8ed1ab_0.conda#37e13edbe3b48f1095a9d085ef9cd83b
https://conda.anaconda.org/conda-forge/noarch/cfgv-3.5.0-pyhd8ed1ab_0.conda#381bd45fb7aa032691f3063aff47e3a1
-https://conda.anaconda.org/conda-forge/noarch/charset-normalizer-3.5.1-pyhd8ed1ab_0.conda#e0ac3accc64e23e40969d660e5f58ac8
https://conda.anaconda.org/conda-forge/noarch/click-8.5.0-pyh5ded981_0.conda#dea5b13a211bbf99876deb980b408a66
https://conda.anaconda.org/conda-forge/noarch/cloudpickle-3.1.2-pyhcf101f3_1.conda#61b8078a0905b12529abc622406cb62c
https://conda.anaconda.org/conda-forge/noarch/colorama-0.4.6-pyhd8ed1ab_1.conda#962b9857ee8e7018c22f2776ffa0b2d7
https://conda.anaconda.org/conda-forge/noarch/colorcet-3.2.1-pyhd8ed1ab_0.conda#0b6419b9a27c540bd96e66e12a44c379
https://conda.anaconda.org/conda-forge/linux-64/coverage-7.16.2-py312h8b7ec3f_0.conda#1d1ef312ddf97e5f5218d3a3d56dd7a0
-https://conda.anaconda.org/conda-forge/noarch/cpython-3.12.14-py312hd8ed1ab_3.conda#adee9213d9a98d7f8b5627209c8b4ae0
+https://conda.anaconda.org/conda-forge/noarch/cpython-3.12.15-py312hd8ed1ab_2.conda#34ff082ed834c6a62f4b11f877820c1d
https://conda.anaconda.org/conda-forge/noarch/cycler-0.12.1-pyhcf101f3_2.conda#4c2a8fef270f6c69591889b93f9f55c1
https://conda.anaconda.org/conda-forge/linux-64/cython-3.3.0-py312h03570ca_0.conda#b2498aa25d1654210448beb48fdce604
https://conda.anaconda.org/conda-forge/noarch/distlib-0.4.3-pyhcf101f3_0.conda#58638f77697c4f6726753eb8be34818b
https://conda.anaconda.org/conda-forge/noarch/docstring_parser-0.18.0-pyhd8ed1ab_0.conda#b0c7c29a60c82d57c5b4c4c38f0642c8
https://conda.anaconda.org/conda-forge/noarch/docutils-0.21.2-pyhd8ed1ab_1.conda#24c1ca34138ee57de72a943237cde4cc
https://conda.anaconda.org/conda-forge/noarch/execnet-2.1.2-pyhd8ed1ab_0.conda#a57b4be42619213a94f31d2c69c5dda7
-https://conda.anaconda.org/conda-forge/noarch/filelock-4.0.6-pyhd8ed1ab_0.conda#2a68742ffb325d6b25dcb3effff4234c
+https://conda.anaconda.org/conda-forge/noarch/filelock-4.1.1-pyhd8ed1ab_0.conda#fbff0cf6c30bdd9b38d0448fdadfe72d
https://conda.anaconda.org/conda-forge/linux-64/frozenlist-1.8.0-py312h447239a_0.conda#6a42923f35087cc88a9fac31ef096ce6
https://conda.anaconda.org/conda-forge/noarch/fsspec-2026.9.0-pyhd8ed1ab_0.conda#42de5cd170647599a84f468a42fabb89
+https://conda.anaconda.org/conda-forge/linux-64/google-crc32c-1.8.0-py312hda0ef95_2.conda#c5d2e1c00832cf3d97018829efd3d480
https://conda.anaconda.org/conda-forge/noarch/hpack-4.2.0-pyhd8ed1ab_0.conda#b395909221b9bd1df066e5930e18855b
https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda#8e6923fc12f1fe8f8c4e5c9f343256ac
https://conda.anaconda.org/conda-forge/noarch/idna-3.20-pyh5ded981_0.conda#ee73be987f9de9ea5c2fdaeefa7b6491
https://conda.anaconda.org/conda-forge/noarch/imagesize-2.0.1-pyhd8ed1ab_0.conda#ebc56022e4ef7e74a829be94c53797ca
-https://conda.anaconda.org/conda-forge/noarch/iniconfig-2.3.0-pyhd8ed1ab_0.conda#9614359868482abba1bd15ce465e3c42
+https://conda.anaconda.org/conda-forge/noarch/iniconfig-2.3.1-pyhd8ed1ab_0.conda#c0a803d97dd09da7e2c8b0fdfaa5739b
https://conda.anaconda.org/conda-forge/noarch/iris-sample-data-2.5.2-pyhd8ed1ab_0.conda#895f6625dd8a246fece9279fcc12c1de
https://conda.anaconda.org/conda-forge/linux-64/jsonschema-rs-0.37.4-py312h121d7ae_0.conda#1a2a2624770f712a536b8c5758c3387d
https://conda.anaconda.org/conda-forge/linux-64/kiwisolver-1.5.1-py312h9be0db6_3.conda#52258f682691a6838dd40316265d9cee
@@ -198,24 +200,24 @@ https://conda.anaconda.org/conda-forge/linux-64/libgl-1.7.0-ha4b6fd6_5.conda#811
https://conda.anaconda.org/conda-forge/linux-64/libglx-devel-1.7.0-ha4b6fd6_5.conda#fb00b4fb800f2d431303a5c1625ef9d0
https://conda.anaconda.org/conda-forge/linux-64/libhwloc-2.15.0-default_hd7c4d0c_1001.conda#66c755b3bd9ec82308fa6ec7674f473f
https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.11.0-11_h47877c9_openblas.conda#5622a13917855ce4b953b9e655cf9262
-https://conda.anaconda.org/conda-forge/linux-64/libllvm21-21.1.8-h474f4eb_1.conda#580bc512b273f4236bb37031620c091f
https://conda.anaconda.org/conda-forge/linux-64/libxkbcommon-1.13.2-h51789e4_1.conda#f888787e0eab7a8076a67d197e155ffc
-https://conda.anaconda.org/conda-forge/linux-64/libxml2-devel-2.15.4-h7df9aa5_0.conda#9a414200ffe436e47d72c9f4c11c3658
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-devel-2.15.4-h7df9aa5_1.conda#c6f14f767cbfe20fa03a324beb6cd7eb
https://conda.anaconda.org/conda-forge/noarch/locket-1.0.0-pyhd8ed1ab_0.tar.bz2#91e27ef3d05cc772ce627e51cff111c4
-https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.3-py312h8a5da7c_1.conda#93a4752d42b12943a355b682ee43285b
+https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.4-py312h8b7ec3f_1.conda#ee4135fc1ce77066d2be65475bcfbbff
https://conda.anaconda.org/conda-forge/noarch/mdurl-0.1.2-pyhd8ed1ab_1.conda#592132998493b3ff25fd7479396e8351
-https://conda.anaconda.org/conda-forge/linux-64/msgpack-python-1.2.2-py312h9be0db6_2.conda#e9600e8080bf57f25ad2e9f78564c895
+https://conda.anaconda.org/conda-forge/linux-64/msgpack-python-1.2.3-py312h9be0db6_0.conda#9f6cd0c3ce48c45d9a9c1645ecb186d1
+https://conda.anaconda.org/conda-forge/linux-64/msgspec-0.22.0-py312h5cc1888_0.conda#c762212af9018465d4bd165bd2c9938d
https://conda.anaconda.org/conda-forge/linux-64/multidict-6.7.1-py312h8a5da7c_0.conda#17c77acc59407701b54404cfd3639cac
https://conda.anaconda.org/conda-forge/noarch/munkres-1.1.4-pyhd8ed1ab_1.conda#37293a85a0f4f77bbd9cf7aaefc62609
https://conda.anaconda.org/conda-forge/linux-64/openldap-2.6.13-hbde042b_0.conda#680608784722880fbfe1745067570b00
https://conda.anaconda.org/conda-forge/noarch/packaging-26.3-pyhc364b38_0.conda#936687ed80f295a1f5dbcf8bd34c252c
https://conda.anaconda.org/conda-forge/linux-64/pillow-12.3.0-py312h38079b3_4.conda#87d8e4053a968e8b20937e6cdd945a18
-https://conda.anaconda.org/conda-forge/noarch/platformdirs-4.12.1-pyh5ded981_0.conda#c2718a0d7dc28faf28e714807604a60d
+https://conda.anaconda.org/conda-forge/noarch/platformdirs-4.13.0-pyh5ded981_0.conda#35cc2a176fc6f886d6efa41ecb5f1de0
https://conda.anaconda.org/conda-forge/noarch/pluggy-1.6.0-pyhf9edf01_1.conda#d7585b6550ad04c8c5e21097ada2888e
https://conda.anaconda.org/conda-forge/linux-64/proj-9.8.1-he0df7b0_0.conda#b23619e5e9009eaa070ead0342034027
-https://conda.anaconda.org/conda-forge/linux-64/propcache-0.5.2-py312h8a5da7c_0.conda#d681d6ad9fa2ca3c8cacb7f3b23d54f3
+https://conda.anaconda.org/conda-forge/linux-64/propcache-0.5.4-py312hfd18d64_0.conda#a1010a8328530ad72302e24a99104fe8
https://conda.anaconda.org/conda-forge/linux-64/psutil-7.2.2-py312h1b36aeb_3.conda#2f8faed6c111c9ed0c1d217bc1be75f2
-https://conda.anaconda.org/conda-forge/noarch/pycparser-3.0-pyhcf101f3_0.conda#9c5491066224083c41b6d5635ed7107b
+https://conda.anaconda.org/conda-forge/noarch/pycparser-3.11-pyh5ded981_0.conda#3cd164745122ab557cc30879390089e8
https://conda.anaconda.org/conda-forge/noarch/pygments-2.21.0-pyhcf101f3_0.conda#2882dee445dfa45b0c5afce3ffb7730a
https://conda.anaconda.org/conda-forge/noarch/pyparsing-3.3.3-pyh5ded981_0.conda#8bf43ae1312e9b966063459e04c20fa7
https://conda.anaconda.org/conda-forge/noarch/pyshp-3.1.6-pyhcf101f3_0.conda#6778d276eb7b4230d622bdd2a732c75e
@@ -227,12 +229,12 @@ https://conda.anaconda.org/conda-forge/noarch/setuptools-84.0.0-pyh332efcf_0.con
https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda#3339e3b65d58accf4ca4fb8748ab16b3
https://conda.anaconda.org/conda-forge/noarch/snowballstemmer-3.1.1-pyhd8ed1ab_0.conda#46b6abe31482f6bca064b965696ae807
https://conda.anaconda.org/conda-forge/noarch/sortedcontainers-2.4.0-pyhd8ed1ab_1.conda#0401a17ae845fa72c7210e206ec5647d
-https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.10-pyhd8ed1ab_0.conda#6c2382c11f0c0486f3300a761e5c1205
+https://conda.anaconda.org/conda-forge/noarch/soupsieve-3.0.3-pyhd8ed1ab_0.conda#bd2ea090567b9d0a53240bd1367c87ad
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-jsmath-1.0.1-pyhd8ed1ab_1.conda#fa839b5ff59e192f411ccc7dae6588bb
https://conda.anaconda.org/conda-forge/noarch/tabulate-0.10.0-pyhcf101f3_0.conda#3b887b7b3468b0f494b4fad40178b043
https://conda.anaconda.org/conda-forge/noarch/tblib-3.2.2-pyhcf101f3_0.conda#f88bb644823094f436792f80fba3207e
-https://conda.anaconda.org/conda-forge/noarch/tomli-2.4.1-pyhcf101f3_0.conda#b5325cf06a000c5b14970462ff5e4d58
-https://conda.anaconda.org/conda-forge/noarch/toolz-1.1.0-pyhd8ed1ab_1.conda#c07a6153f8306e45794774cf9b13bd32
+https://conda.anaconda.org/conda-forge/noarch/tomli-2.5.0-pyh5ded981_0.conda#36d6a162326c52d6339c8b606f43932d
+https://conda.anaconda.org/conda-forge/noarch/toolz-1.2.0-pyh5ded981_0.conda#17f8db9088267a3cb38d56cc22d268ae
https://conda.anaconda.org/conda-forge/linux-64/tornado-6.5.10-py312h5cc1888_0.conda#b0414832500d750321eddce3822bea7b
https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.16.0-pyhcf101f3_0.conda#c70ad746c22219b9700931707482992c
https://conda.anaconda.org/conda-forge/linux-64/unicodedata2-18.0.0-py312h5cc1888_0.conda#070e15c0fbb80b2f2fe405419b763f68
@@ -245,46 +247,46 @@ https://conda.anaconda.org/conda-forge/linux-64/xorg-libxinerama-1.1.6-hecca717_
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrandr-1.5.5-h7cc23a3_1.conda#798a8c9d171859a022e1cb89e7d0eb10
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxxf86vm-1.1.7-hb03c661_0.conda#665d152b9c6e78da404086088077c844
https://conda.anaconda.org/conda-forge/noarch/zict-3.0.0-pyhd8ed1ab_1.conda#e52c2ef711ccf31bb7f70ca87d144b9e
-https://conda.anaconda.org/conda-forge/noarch/zipp-4.1.0-pyhcf101f3_0.conda#ba3dcdc8584155c97c648ae9c044b7a3
+https://conda.anaconda.org/conda-forge/noarch/zipp-4.1.1-pyh5ded981_0.conda#6cad4d2cec9b7b64538c225ac91d2e8a
https://conda.anaconda.org/conda-forge/noarch/accessible-pygments-0.0.5-pyhd8ed1ab_1.conda#74ac5069774cdbc53910ec4d631a3999
https://conda.anaconda.org/conda-forge/noarch/affine-3.0.1-pyhecae5ae_0.conda#aeb404cab9e719a49264ab40928f80c8
https://conda.anaconda.org/conda-forge/noarch/aiosignal-1.4.0-pyhd8ed1ab_0.conda#421a865222cd0c9d83ff08bc78bf3a61
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-s3-0.13.1-h7508075_1.conda#2daf54d3eef87b757f68d7597534dd2f
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-s3-1.0.0-hfe03eda_1.conda#7a8a507033f2e478670a617bb4ffb0b8
https://conda.anaconda.org/conda-forge/linux-64/cffi-2.1.1-py312h703531f_3.conda#41f019d067f8c52c6d9283d8afb28889
https://conda.anaconda.org/conda-forge/noarch/click-default-group-1.2.4-pyhd8ed1ab_1.conda#7cd83dd6831b61ad9624a694e4afd7dc
https://conda.anaconda.org/conda-forge/noarch/click-plugins-1.1.1.2-pyhd8ed1ab_0.conda#e9b05deb91c013e5224672a4ba9cf8d1
https://conda.anaconda.org/conda-forge/noarch/cligj-0.7.2-pyhd8ed1ab_2.conda#55c7804f428719241a90b152016085a1
-https://conda.anaconda.org/conda-forge/linux-64/cytoolz-1.1.0-py312h5cc1888_3.conda#0e1919f4d37e009e1457df1c267fbc66
+https://conda.anaconda.org/conda-forge/linux-64/cytoolz-1.2.0-py312h1b36aeb_0.conda#20f3a25d978a201f5dc592380781a7e6
+https://conda.anaconda.org/conda-forge/noarch/donfig-0.8.1.post1-pyhd8ed1ab_1.conda#c56a7fa5597ad78b62e1f5d21f7f8b8f
https://conda.anaconda.org/conda-forge/noarch/exceptiongroup-1.3.1-pyhd8ed1ab_0.conda#8e662bd460bda79b1ea39194e3c4c9ab
-https://conda.anaconda.org/conda-forge/linux-64/fonttools-4.66.0-py312hfd18d64_0.conda#abfa8a0cafbbcbe90428fa2e5d4dbdb5
+https://conda.anaconda.org/conda-forge/linux-64/fonttools-4.66.1-py312hfd18d64_0.conda#3a646243ead357a2184be86b64be5203
https://conda.anaconda.org/conda-forge/linux-64/gl2ps-1.4.2-h36e74d4_2.conda#787c780ff43f9f79d78d01e476b81a7c
https://conda.anaconda.org/conda-forge/linux-64/glew-2.3.0-h71661d4_0.conda#7eefecda1c71c380bfc406d16e78bbee
https://conda.anaconda.org/conda-forge/noarch/h2-4.4.1-pyhcf101f3_0.conda#e652ac7756069c456d0da2a922cd7df5
https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.1-pyhcf101f3_0.conda#77ec68e0aa61c3fff9ecbf840f93d64e
-https://conda.anaconda.org/conda-forge/noarch/importlib_resources-7.1.0-pyhd8ed1ab_0.conda#0ba6225c279baf7ea9473a62ea0ec9ae
https://conda.anaconda.org/conda-forge/noarch/jinja2-3.1.6-pyhcf101f3_1.conda#04558c96691bed63104678757beb4f8d
https://conda.anaconda.org/conda-forge/noarch/lazy-loader-0.6-pyhd8ed1ab_0.conda#4ed11ec126e6da490282edfd0673fea0
https://conda.anaconda.org/conda-forge/linux-64/libgl-devel-1.7.0-ha4b6fd6_5.conda#f5f7fc038c611a25b22758c0a40d25c9
-https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-14.5.0-h23af247_0.conda#4c0d9662a054c51e69b66a427e7ead03
-https://conda.anaconda.org/conda-forge/linux-64/libpq-18.6-h3883810_2.conda#bf4a620772888bef1da4c33f87fd1f72
+https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-14.6.0-h23af247_2.conda#6a93147cab011f36d1659745c3f579e2
+https://conda.anaconda.org/conda-forge/linux-64/libpq-18.6-h3883810_3.conda#cc998047391971a452fed680158329f3
https://conda.anaconda.org/conda-forge/linux-64/libspatialite-5.1.0-gpl_h03b4e02_121.conda#affb2721df7014b4abcd75cc539c6cef
https://conda.anaconda.org/conda-forge/linux-64/libvulkan-loader-1.4.357.0-h0e34353_2.conda#c5f5f159b66332152197893427071695
https://conda.anaconda.org/conda-forge/noarch/markdown-it-py-4.2.0-pyhd8ed1ab_0.conda#6d03368f2b2b0a5fb6839df53b2eb5e0
https://conda.anaconda.org/conda-forge/noarch/nodeenv-1.10.0-pyhd8ed1ab_0.conda#eb52d14a901e23c39e9e7b4a1a5c015f
-https://conda.anaconda.org/conda-forge/linux-64/numpy-2.5.3-py312he827f4e_0.conda#5abf073f2cf066f2f3d28229b3cbe94d
+https://conda.anaconda.org/conda-forge/linux-64/numpy-2.5.4-py312he827f4e_0.conda#759b701f4b548ff6d06794d99788e6bb
https://conda.anaconda.org/conda-forge/noarch/partd-1.4.2-pyhd8ed1ab_0.conda#0badf9c54e24cecfb0ad2f99d680c163
https://conda.anaconda.org/conda-forge/linux-64/pyproj-3.8.0-py312hb99b6cc_1.conda#838b75fbf68ce1cf79970f30835ab334
https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.9.0.post0-pyhe01879c_2.conda#5b8d21249ff20967101ffa321cab24e8
-https://conda.anaconda.org/conda-forge/noarch/python-discovery-1.6.1-pyh5ded981_0.conda#ae0feb6a3f297952bd4f4270fe77f530
-https://conda.anaconda.org/conda-forge/noarch/python-gil-3.12.14-hd8ed1ab_3.conda#7fafcd204183cfa7c1ae4ad9c2d8d414
+https://conda.anaconda.org/conda-forge/noarch/python-discovery-1.6.2-pyh5ded981_0.conda#17e6c8b2fc2d021a217d317fe4f89fc4
+https://conda.anaconda.org/conda-forge/noarch/python-gil-3.12.15-hd8ed1ab_2.conda#deec3cfe011e962fe2eb872f2eae716c
https://conda.anaconda.org/conda-forge/linux-64/tbb-2023.1.0-haa8c3ee_2.conda#f805909f478c179e230baf7bd5878e26
https://conda.anaconda.org/conda-forge/noarch/typing-extensions-4.16.0-h69aa097_0.conda#c680b5747e8c4c8f23dca0bb7042a8fc
-https://conda.anaconda.org/conda-forge/noarch/vcs_versioning-2.5.0-pyh5ded981_0.conda#137d3106e230ae55fe7b8ee09c55aaf0
+https://conda.anaconda.org/conda-forge/noarch/vcs_versioning-2.6.0-pyh5ded981_0.conda#958f3e46d5025cdd9aa9c2417adceec8
https://conda.anaconda.org/conda-forge/noarch/wheel-0.48.0-pyhd8ed1ab_0.conda#9a2124517bfd5d792e0f7b86eefdfeb8
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxtst-1.2.5-h7cc23a3_4.conda#752a5ac9322e0fbbc24146c1ce3ae44e
-https://conda.anaconda.org/conda-forge/linux-64/yarl-1.24.5-py312h8a5da7c_0.conda#ef4788bdfaa84f3ae3a9c1390eaf3e73
+https://conda.anaconda.org/conda-forge/linux-64/yarl-1.25.1-py312hfd18d64_0.conda#91bb35918653d559dbd514c9064e512b
https://conda.anaconda.org/conda-forge/noarch/_python_abi3_support-1.0-hd8ed1ab_3.conda#3845f3d75991bae0fb90884662f4327c
-https://conda.anaconda.org/conda-forge/linux-64/aiohttp-3.14.3-py312h5d8c7f2_0.conda#fe9eb34a1c0e51dfc5d98f41c5ddc8b8
+https://conda.anaconda.org/conda-forge/linux-64/aiohttp-3.14.5-py312h8b7ec3f_0.conda#e5e9629c0a1c75dc79ec1ac35b0560da
https://conda.anaconda.org/conda-forge/noarch/asv_runner-0.3.1-pyhd8ed1ab_0.conda#960a0f64467ce624941ad7c67b80a2e2
https://conda.anaconda.org/conda-forge/linux-64/at-spi2-core-2.40.3-h0630a04_0.tar.bz2#8cb2fc4cd6cc63f1369cfa318f581cc3
https://conda.anaconda.org/conda-forge/noarch/beautifulsoup4-4.15.0-pyha770c72_0.conda#3b261da3fe9b4168738712832410b022
@@ -292,14 +294,14 @@ https://conda.anaconda.org/conda-forge/linux-64/cftime-1.6.5-py312h0ec8046_2.con
https://conda.anaconda.org/conda-forge/noarch/colorspacious-1.1.2-pyhecae5ae_1.conda#04151bb8e351c6209caad045e4b1f4bd
https://conda.anaconda.org/conda-forge/linux-64/contourpy-1.4.0-py312h9be0db6_1.conda#64e62dfa49dd4688301b02d8cfea3687
https://conda.anaconda.org/conda-forge/noarch/dask-core-2026.8.0-pyhc364b38_0.conda#2ccdf45d2372e15c7edcb7a72d573e76
-https://conda.anaconda.org/conda-forge/linux-64/hdf5-2.2.0-nompi_hc529623_100.conda#336f7fb9af0c49cc246e2cff0992c3c4
-https://conda.anaconda.org/conda-forge/noarch/importlib-resources-7.1.0-pyhd8ed1ab_0.conda#e3bffa82b874f8b9a2631bddb3869529
+https://conda.anaconda.org/conda-forge/linux-64/hdf5-2.2.0-nompi_hdbb200c_101.conda#66762e6067bf7d52586ff21c78c970db
https://conda.anaconda.org/conda-forge/noarch/lazy_loader-0.6-pyhd8ed1ab_0.conda#baa2be54811ef381364ae730a4613d0f
https://conda.anaconda.org/conda-forge/linux-64/libegl-devel-1.7.0-ha4b6fd6_5.conda#577800fc8c9d8b7d9727246bbfde704e
https://conda.anaconda.org/conda-forge/linux-64/libgdal-core-3.13.3-h31aba09_1.conda#44c6780e3082468d8676aebdcd5068a6
-https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-devel-14.5.0-h23af247_0.conda#82f4063122b5ebc18cc34bb269e941f3
+https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-devel-14.6.0-h23af247_2.conda#3cf459e930e6ea2e57bbf04be7fcb7c6
https://conda.anaconda.org/conda-forge/linux-64/libraqm-0.11.0-h6406941_0.conda#3ac89a48d224409739dbf6200e524373
https://conda.anaconda.org/conda-forge/linux-64/mo_pack-0.3.1-py312h0ec8046_3.conda#eb270473659371f1291b774377eef4b5
+https://conda.anaconda.org/conda-forge/linux-64/numcodecs-0.17.0-py312h0ec8046_1.conda#f21c0e5e878142ca05450f56019f99da
https://conda.anaconda.org/conda-forge/linux-64/pandas-3.0.6-np2py312h91ec553_0.conda#2143c2cef115b49a934d526f7da91696
https://conda.anaconda.org/conda-forge/linux-64/pango-1.58.2-h7bb47b9_1.conda#0b889b1c6b472d215d6332c6b195589c
https://conda.anaconda.org/conda-forge/noarch/pip-26.2.1-pyh8b19718_0.conda#77b6cf3b0689d9fc68561df0db9b856d
@@ -310,51 +312,54 @@ https://conda.anaconda.org/conda-forge/linux-64/pywavelets-1.10.0-py312h1b36aeb_
https://conda.anaconda.org/conda-forge/noarch/rich-15.0.0-pyhcf101f3_0.conda#0242025a3c804966bf71aa04eee82f66
https://conda.anaconda.org/conda-forge/linux-64/scipy-1.18.1-py312h106c528_1.conda#73cc9629383ee4e1f2166b314527b324
https://conda.anaconda.org/conda-forge/noarch/setuptools-scm-10.3.4-pyh5ded981_0.conda#437557778d99cc6f311f3c0ce023ad85
-https://conda.anaconda.org/conda-forge/linux-64/shapely-2.1.2-py312ha95667c_3.conda#f9bf71d5368b7255ada82596d02a4b2f
+https://conda.anaconda.org/conda-forge/linux-64/shapely-2.2.0-np2py312h224e744_2.conda#c89c76fe2aa0b35631ea0255902f27fd
https://conda.anaconda.org/conda-forge/noarch/snuggs-1.4.7-pyhd8ed1ab_2.conda#9aa358575bbd4be126eaa5e0039f835c
+https://conda.anaconda.org/conda-forge/noarch/towncrier-26.9.0-pyhd8ed1ab_0.conda#d4167f473b71694bbca708347abee99e
https://conda.anaconda.org/conda-forge/linux-64/ukkonen-1.1.0-py312hd9148b4_0.conda#55fd03988b1b1bc6faabbfb5b481ecd7
https://conda.anaconda.org/conda-forge/noarch/urllib3-2.8.0-pyhd8ed1ab_0.conda#407a3d3778570bae8e8fcf554e803e43
-https://conda.anaconda.org/conda-forge/noarch/virtualenv-21.13.0-pyh5ded981_0.conda#e553e26b7aa15efa4640ec55859dfc53
+https://conda.anaconda.org/conda-forge/noarch/virtualenv-21.14.7-pyh5ded981_0.conda#1635534aa129eb9ab6ce5e01929d86ff
https://conda.anaconda.org/conda-forge/linux-64/at-spi2-atk-2.38.0-h0630a04_3.tar.bz2#6b889f174df1e0f816276ae69281af4d
-https://conda.anaconda.org/conda-forge/linux-64/cf-units-3.3.1-np2hc8c11d4_1.conda#49647e9c24724a189e0e334f4f3223e7
+https://conda.anaconda.org/conda-forge/linux-64/cf-units-3.3.1-np2hc8c11d4_2.conda#9b777f085506bff796833be0ae292db1
+https://conda.anaconda.org/conda-forge/linux-64/charset-normalizer-3.5.2-h1114479_0.conda#78d5c6112c9320a1a797935cecfec7c2
https://conda.anaconda.org/conda-forge/noarch/distributed-2026.8.0-pyhc364b38_0.conda#f382c4eed5376ee372a6e67a0f6cd4d9
https://conda.anaconda.org/conda-forge/linux-64/epoxy-1.5.10-hb03c661_2.conda#057083b06ccf1c2778344b6dabace38b
-https://conda.anaconda.org/conda-forge/linux-64/harfbuzz-14.5.0-ha770c72_0.conda#ef253035e60216967c117ab9a279eea2
+https://conda.anaconda.org/conda-forge/linux-64/harfbuzz-14.6.0-ha770c72_2.conda#0709eacc38906e8a237d9ef172af6688
https://conda.anaconda.org/conda-forge/noarch/identify-2.6.20-pyhd8ed1ab_0.conda#be9ea4f030b4b2bc121b2a548b066c0b
https://conda.anaconda.org/conda-forge/noarch/imagehash-4.3.2-pyhd8ed1ab_0.conda#efbc812363856906a80344b496389d2e
https://conda.anaconda.org/conda-forge/linux-64/libnetcdf-4.10.1-nompi_he3e3c8e_201.conda#912aefa694a66897afeb7a2330b17d9f
https://conda.anaconda.org/conda-forge/linux-64/librsvg-2.62.4-ha427ee3_0.conda#f9bda2728debbde4c641bd1af103c623
https://conda.anaconda.org/conda-forge/linux-64/matplotlib-base-3.11.2-py312hfe9aac7_1.conda#0745f1af620efcb1095d8ddabcb2ed8b
-https://conda.anaconda.org/conda-forge/linux-64/minijinja-2.24.0-py310h300b7de_0.conda#1be3473b8f1e33cb92ce3401fab8e8a6
-https://conda.anaconda.org/conda-forge/noarch/pbr-7.1.1-pyhd8ed1ab_0.conda#51b000c0a6d35e244019f9fc8ab1c9f6
+https://conda.anaconda.org/conda-forge/noarch/pbr-7.1.3-pyhd8ed1ab_0.conda#66415d5d2894f1d10928005a264c2717
https://conda.anaconda.org/conda-forge/noarch/pytest-cov-7.1.0-pyhcf101f3_0.conda#67d1790eefa81ed305b89d8e314c7923
https://conda.anaconda.org/conda-forge/noarch/pytest-mock-3.16.0-pyhd8ed1ab_0.conda#b038267c191ebae9d6374caf30cc2cca
https://conda.anaconda.org/conda-forge/noarch/pytest-xdist-3.8.0-pyhd8ed1ab_0.conda#8375cfbda7c57fbceeda18229be10417
+https://conda.anaconda.org/conda-forge/noarch/python-abi3-3.12-he1d15d4_3.conda#ea0c63b97585ce9587b3a33316b2995b
https://conda.anaconda.org/conda-forge/linux-64/python-stratify-0.4.0-py312h5cc1888_4.conda#bb80e017cd2afba2950e328ee2709e82
-https://conda.anaconda.org/conda-forge/linux-64/qt6-main-6.11.2-pl5321h9df5c37_0.conda#2ff98660281d1c6bfebda5bffa52cc89
-https://conda.anaconda.org/conda-forge/linux-64/rasterio-1.5.1-py312h5d657d5_1.conda#c0d4e6e0b2f6a3e73ab81630deab839d
-https://conda.anaconda.org/conda-forge/noarch/requests-2.34.2-pyhcf101f3_0.conda#4a85203c1d80c1059086ae860836ffb9
+https://conda.anaconda.org/conda-forge/linux-64/qt6-main-6.12.0-pl5321hcf69127_1.conda#db3e81b8f192e5c9ee8e1e8c215c40b0
+https://conda.anaconda.org/conda-forge/linux-64/rasterio-1.5.2-py312h5613827_1.conda#7af5ba4cf7007c202f4a15bd273047fc
https://conda.anaconda.org/conda-forge/noarch/rich-rst-2.2.0-pyhd8ed1ab_0.conda#d91900e108041fa4d06ff329970027a1
-https://conda.anaconda.org/conda-forge/noarch/towncrier-25.8.0-pyhd8ed1ab_0.conda#3e0e8e44292bdac62f7bcbf0450b5cc7
https://conda.anaconda.org/conda-forge/linux-64/viskores-1.1.1-cpu_ha53dc59_1.conda#228549cd2f01bb53664bd2b5b78d7389
-https://conda.anaconda.org/conda-forge/noarch/wslink-2.5.7-pyhd8ed1ab_0.conda#410dee7cbee308f33b01645f16f11efc
+https://conda.anaconda.org/conda-forge/noarch/wslink-2.6.0-pyhd8ed1ab_0.conda#c3bcfd3bb22061d48cb833925795105f
+https://conda.anaconda.org/conda-forge/noarch/zarr-3.4.1-pyhc364b38_0.conda#052cab2fa318f6ffeefc149addd9e6fb
https://conda.anaconda.org/conda-forge/noarch/adwaita-icon-theme-51.0-unix_0.conda#6f62266fb2bfc35a5071cfa1d6ffdcc9
-https://conda.anaconda.org/conda-forge/linux-64/cartopy-0.26.0-np2py312h748b537_1.conda#b8d5a8220d88a90fe30a2e120a36bea3
+https://conda.anaconda.org/conda-forge/linux-64/cartopy-0.25.0-np2py312h0f77346_3.conda#612c6dd5655f10e16f5d693c57259131
https://conda.anaconda.org/conda-forge/noarch/cmocean-4.0.3-pyhd8ed1ab_1.conda#dd71e4ec2fbffe38c0359976505f816e
-https://conda.anaconda.org/conda-forge/noarch/cyclopts-5.0.0-pyh5ded981_0.conda#3a90e8cb67cb4fc314a2696bcf894ed8
+https://conda.anaconda.org/conda-forge/noarch/cyclopts-5.2.0-pyh5ded981_0.conda#fae753a13c1df14b3aeaf2f8e5b5b314
https://conda.anaconda.org/conda-forge/linux-64/gtk3-3.24.52-ha5ea40c_0.conda#bcaea22d85999a4f17918acfab877e61
+https://conda.anaconda.org/conda-forge/linux-64/minijinja-3.0.0-py311h1180b86_0.conda#ebba9c2cc80a43f736c13772bee155d8
https://conda.anaconda.org/conda-forge/noarch/nc-time-axis-1.4.1-pyhd8ed1ab_1.conda#9a2be7d0089f5934b550933ca0d9fe85
https://conda.anaconda.org/conda-forge/linux-64/netcdf-fortran-4.6.4-nompi_h52816da_101.conda#2f18bb969e8acd8a1f45293e9782374e
https://conda.anaconda.org/conda-forge/linux-64/netcdf4-1.7.4-nompi_py311hd34d2e2_111.conda#d697ebf773f33945402fe3e1fda13635
-https://conda.anaconda.org/conda-forge/noarch/pooch-1.9.0-pyhd8ed1ab_0.conda#dd4b6337bf8886855db6905b336db3c8
-https://conda.anaconda.org/conda-forge/noarch/pre-commit-4.6.2-pyha770c72_0.conda#c36b88db560b4e74f294380613f1a239
-https://conda.anaconda.org/conda-forge/noarch/requests-file-2.1.0-pyhd8ed1ab_1.conda#69be0472744969b97324d5d3b21845a9
+https://conda.anaconda.org/conda-forge/noarch/pre-commit-4.7.0-pyha770c72_0.conda#167169a52ede77767f9fa66b52983a20
+https://conda.anaconda.org/conda-forge/noarch/requests-2.34.2-pyhcf101f3_0.conda#4a85203c1d80c1059086ae860836ffb9
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-apidoc-0.3.0-py_1.tar.bz2#855b087883443abb10f5faf6eef40860
https://conda.anaconda.org/conda-forge/linux-64/vtk-base-9.7.0-py312hdc8235f_0.conda#f71b016e14a4df7e22c04b564afc21ec
https://conda.anaconda.org/conda-forge/linux-64/esmf-8.9.1-nompi_h39d0119_4.conda#074ca9356f9e916ec51ad6ab47fd55ba
https://conda.anaconda.org/conda-forge/linux-64/graphviz-14.1.2-h8b86629_0.conda#341fc61cfe8efa5c72d24db56c776f44
-https://conda.anaconda.org/conda-forge/noarch/pyvista-0.49.0-pyha804496_1.conda#ce10d5ff463de9dd9da958aa49a61064
+https://conda.anaconda.org/conda-forge/noarch/pooch-1.9.0-pyhd8ed1ab_0.conda#dd4b6337bf8886855db6905b336db3c8
+https://conda.anaconda.org/conda-forge/noarch/requests-file-2.1.0-pyhd8ed1ab_1.conda#69be0472744969b97324d5d3b21845a9
https://conda.anaconda.org/conda-forge/noarch/esmpy-8.9.1-pyhdfbf58e_0.conda#0155d4a55415cda7f7f09bc0b7760f51
+https://conda.anaconda.org/conda-forge/noarch/pyvista-0.49.1-pyha804496_0.conda#ea32ac5d548e14030818673f68e6a756
https://conda.anaconda.org/conda-forge/noarch/geovista-0.5.3-pyhd8ed1ab_1.conda#64348d05eedb1b1b5676f63101d004f2
https://conda.anaconda.org/conda-forge/noarch/pydata-sphinx-theme-0.19.0-pyhcf101f3_0.conda#87921f66a4dc56ce92e4ff13be5f63dc
https://conda.anaconda.org/conda-forge/noarch/sphinx-copybutton-0.5.2-pyhd8ed1ab_1.conda#bf22cb9c439572760316ce0748af3713
diff --git a/requirements/locks/py313-linux-64.lock b/requirements/locks/py313-linux-64.lock
index e79f6cd8eb..a706454f8b 100644
--- a/requirements/locks/py313-linux-64.lock
+++ b/requirements/locks/py313-linux-64.lock
@@ -1,6 +1,6 @@
# Generated by conda-lock.
# platform: linux-64
-# input_hash: ff80e901817a2b45bb0436081c24276d938204f7539fa7cadd8d3bddaec56550
+# input_hash: d36df080c40dbcaada47fdd3f5ee101a9b9ed5a3261d5620e155496bacf93c7c
@EXPLICIT
https://conda.anaconda.org/conda-forge/linux-64/eigen-abi-5.0.1.80-hf414acd_3.conda#56a1cca9a4a7bed2999630e5644c8d0e
https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab24e00_0.tar.bz2#0c96522c6bdaed4b1566d11387caaf45
@@ -23,10 +23,10 @@ https://conda.anaconda.org/conda-forge/linux-64/libegl-1.7.0-ha4b6fd6_5.conda#3a
https://conda.anaconda.org/conda-forge/linux-64/libopengl-1.7.0-ha4b6fd6_5.conda#d207a2e9bae9da476bc0a603215d5167
https://conda.anaconda.org/conda-forge/linux-64/zlib-1.3.2-h25fd6f3_3.conda#6acb86426229f96f93e5468d1df3a5e8
https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.7-hb78ec9c_7.conda#aa459086047c0e5e27023ab19f8cb86a
-https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.46.1-default_hbd61a6d_102.conda#449500f2c089da11c40f5c21312e3e07
+https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.46.1-default_hbd61a6d_103.conda#2362c70e7e79827c32db35547d9c179b
https://conda.anaconda.org/conda-forge/linux-64/libgcc-16.2.0-ha9f2e26_7.conda#735629b400b70eeab21a2ba8f5b1c2ce
https://conda.anaconda.org/conda-forge/linux-64/alsa-lib-1.2.16.1-h7cc23a3_1.conda#7094e0d8d14de0eff6d83f0d2f1f661e
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-common-0.14.3-hb03c661_0.conda#ce8664d7c01d4a598ad38107b289b5ec
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-common-1.0.3-hebe6cf0_0.conda#cdb5797968bfe1737f1b71f525fc6518
https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-hda65f42_10.conda#e675fabcf81499adc7edf58124fb1e01
https://conda.anaconda.org/conda-forge/linux-64/c-ares-1.34.8-hebe6cf0_2.conda#3ad2b403be8fc5612b9159e2ce621f69
https://conda.anaconda.org/conda-forge/linux-64/fribidi-1.0.17-h7cc23a3_0.conda#f9d5302bca196ee87e62f393f2626562
@@ -48,14 +48,15 @@ https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.1-hb9d3cd8_1.conda#d8
https://conda.anaconda.org/conda-forge/linux-64/libntlm-1.8-hb9d3cd8_0.conda#7c7927b404672409d9917d49bff5f2d6
https://conda.anaconda.org/conda-forge/linux-64/libogg-1.3.5-hd0c01bc_1.conda#68e52064ed3897463c0e958ab5c8f91b
https://conda.anaconda.org/conda-forge/linux-64/libpciaccess-0.19-hb03c661_1.conda#35fa2b34bbced424e6976d30f5fde576
-https://conda.anaconda.org/conda-forge/linux-64/libpng-1.6.58-h922cc85_1.conda#fcf71c8d979148873f6f8ad4cfc73d86
+https://conda.anaconda.org/conda-forge/linux-64/libpng-1.6.59-h922cc85_0.conda#8c9caf742a485ac688ffa88bbadbf0be
+https://conda.anaconda.org/conda-forge/linux-64/libpython-3.13.16-hd51c5da_102_cp313.conda#de5467b4cae23d95c711f84b2033341d
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-16.2.0-h934c35e_7.conda#9b6707e0caca917fa6ea9ee473f37a5d
https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.42.4-hcfc3c73_0.conda#54a948337e66706380ff24d6b5fde63f
https://conda.anaconda.org/conda-forge/linux-64/libwebp-base-1.6.0-hd42ef1d_1.conda#9332b53d0ea93c5d39e33be03a0c611a
https://conda.anaconda.org/conda-forge/linux-64/libxcrypt-4.4.38-h280c20c_0.conda#f7a7ff5a6ab331e037abd34f379a631d
https://conda.anaconda.org/conda-forge/linux-64/lzo-2.10-hebe6cf0_1003.conda#10e0aa58ed390ba598b728badf9eb1b7
https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.6-hdb14827_1.conda#ee6c0cd80a60961a1f48aa3e0b91f986
-https://conda.anaconda.org/conda-forge/linux-64/openssl-3.6.4-h781a0a9_0.conda#16e3034a330cc625f2c685edec60cf4e
+https://conda.anaconda.org/conda-forge/linux-64/openssl-3.6.5-h781a0a9_0.conda#5a51a2ea653f120240b1a9bf76e0aae9
https://conda.anaconda.org/conda-forge/linux-64/pthread-stubs-0.4-h7cc23a3_1004.conda#bb66b610707b811e3c65181a6431e7a8
https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_h1df4ec4_4.conda#676a2f9b1c4fcf57b70aa5c65f6c60d0
https://conda.anaconda.org/conda-forge/linux-64/xorg-libice-1.1.2-h280c20c_0.conda#85c9442aec283b4e464fa9ecc484a2f3
@@ -64,10 +65,10 @@ https://conda.anaconda.org/conda-forge/linux-64/xorg-libxdmcp-1.1.5-h7cc23a3_2.c
https://conda.anaconda.org/conda-forge/linux-64/xorg-xorgproto-2025.1-hebe6cf0_3.conda#c3d0a6f03ba3aa312777b41b20a71ec0
https://conda.anaconda.org/conda-forge/linux-64/xxhash-0.8.3-hec8ed23_1.conda#72ab4ea51529ec63ff010ae9fb62b02f
https://conda.anaconda.org/conda-forge/linux-64/yaml-0.2.5-hebe6cf0_3.conda#e741576fb8f89821ac7c1c537322a33d
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-cal-0.9.15-h6bbde05_1.conda#8f2b374c79908f271be3d0e9f1c462a8
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-compression-0.3.2-h01ee8f8_5.conda#ae862abdbb3160f97478d3d4749164c0
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-sdkutils-0.2.7-h01ee8f8_3.conda#412f0cf18cc4c4945655dba954ca2d76
-https://conda.anaconda.org/conda-forge/linux-64/aws-checksums-0.2.10-h01ee8f8_5.conda#4d70caf5260e0787675c3bf61ae5b64c
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-cal-1.0.0-hcf7459e_1.conda#2009b71b358163738f339c76640b512c
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-compression-1.0.0-ha404f16_1.conda#da0c7b86d5379eb44acfa084897e8b98
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-sdkutils-1.0.0-ha404f16_1.conda#cbbd999c74a030951365225909e376c7
+https://conda.anaconda.org/conda-forge/linux-64/aws-checksums-1.0.1-ha404f16_0.conda#10e6e2f11271e66dfad43eb002359f22
https://conda.anaconda.org/conda-forge/linux-64/cli11-2.7.2-h54a6638_1.conda#42761b55062088f6f7fb58d0633e0769
https://conda.anaconda.org/conda-forge/linux-64/double-conversion-3.4.0-hecca717_0.conda#dbe3ec0f120af456b3477743ffd99b74
https://conda.anaconda.org/conda-forge/linux-64/fmt-12.1.0-h76c4fd7_1.conda#a70bb6d42ad121aef456e0007e92bed6
@@ -80,11 +81,11 @@ https://conda.anaconda.org/conda-forge/linux-64/libaec-1.1.5-h088129d_0.conda#86
https://conda.anaconda.org/conda-forge/linux-64/libbrotlidec-1.2.0-ha411449_4.conda#0cfd601eb03cca403dcc248844787486
https://conda.anaconda.org/conda-forge/linux-64/libbrotlienc-1.2.0-h018ffa1_4.conda#4136f6e4ef4835cc7df39a707cbd511c
https://conda.anaconda.org/conda-forge/linux-64/libdrm-2.4.129-h7cc23a3_0.conda#64cc91512b6278c315349dc13c53f680
-https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20250104-pl5321h373387f_1.conda#50708d3b951d0f8e2d7f2df5b5edc040
+https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20260512-pl5321h373387f_0.conda#27dbd7e2984b8fe48e069379636ccfdd
https://conda.anaconda.org/conda-forge/linux-64/libfreetype6-2.14.3-h5e6c136_2.conda#b72a266a9317036fd0464cb98b027cd0
https://conda.anaconda.org/conda-forge/linux-64/libgfortran-16.2.0-h69a702a_7.conda#3a8d285f5f22138ef826ad54cf267e73
https://conda.anaconda.org/conda-forge/linux-64/libglu-9.0.3-h5888daf_1.conda#8422fcc9e5e172c91e99aef703b3ce65
-https://conda.anaconda.org/conda-forge/linux-64/libhwy-1.4.0-h57c4cff_1.conda#d58bfbaaf51ed85771eae92c2e7f5816
+https://conda.anaconda.org/conda-forge/linux-64/libhwy-1.4.0-h57c4cff_2.conda#145497b38fe581a7760dd88b7377c96b
https://conda.anaconda.org/conda-forge/linux-64/libnghttp2-1.68.1-h74cf4be_1.conda#75d211f04ac87506f1f43420712a69fe
https://conda.anaconda.org/conda-forge/linux-64/libpython-3.13.15-hdc7f604_103_cp313.conda#e64cd43bbd07afafbc458138e979c851
https://conda.anaconda.org/conda-forge/linux-64/libssh2-1.11.1-h6154650_1.conda#5c526561e1d2a2e071941174eae84557
@@ -100,29 +101,30 @@ https://conda.anaconda.org/conda-forge/linux-64/pcre2-10.47-h8b3dc9c_1.conda#06f
https://conda.anaconda.org/conda-forge/linux-64/pixman-0.46.4-h54a6638_3.conda#0ee5bb30034b081a1386c1e2c98ab0a7
https://conda.anaconda.org/conda-forge/linux-64/pugixml-1.15-h3f63f65_0.conda#b11a4c6bf6f6f44e5e143f759ffa2087
https://conda.anaconda.org/conda-forge/linux-64/readline-8.3-hd6e31c0_1.conda#69c01c781e7c8190bbdaf79e3848c5ca
-https://conda.anaconda.org/conda-forge/linux-64/s2n-1.7.6-h7e3ee7f_0.conda#e112cadcd79412d81496e784884bddf0
+https://conda.anaconda.org/conda-forge/linux-64/s2n-1.7.7-ha5dc4ae_0.conda#4928f1e1964a6f38f0da349b9d0e1c6a
https://conda.anaconda.org/conda-forge/linux-64/snappy-1.2.2-h34e00bb_2.conda#2b7e31ffbd476eecf33b3b833692339c
https://conda.anaconda.org/conda-forge/linux-64/wayland-1.26.0-hc1c935e_2.conda#88a47e22b53e50865b1cabe2eb648352
https://conda.anaconda.org/conda-forge/linux-64/xorg-libsm-1.2.6-h0d788c3_1.conda#aa7459ed9ad086ba11dda843d764b33d
https://conda.anaconda.org/conda-forge/linux-64/zfp-1.0.1-h79e284d_6.conda#9d7141ec747a9ede5da83e864fcc8213
https://conda.anaconda.org/conda-forge/linux-64/zlib-ng-2.3.3-hce19668_1.conda#1726acec19beeed1c764385cd8622405
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-io-0.27.5-h4f381ba_1.conda#948f5a4fddac779b7942626e5807caba
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-io-1.0.0-h07ba609_0.conda#72593f68772d8c7e72c8a09681b0fe42
https://conda.anaconda.org/conda-forge/linux-64/blosc-1.21.6-he440d0b_1.conda#2c2fae981fd2afd00812c92ac47d023d
https://conda.anaconda.org/conda-forge/linux-64/brotli-bin-1.2.0-h9908984_4.conda#c48c09444f3736800ea0b9550c365baf
https://conda.anaconda.org/conda-forge/linux-64/freexl-2.0.0-h1f12c17_3.conda#4a0a8fd87ee4767db791047546f56501
https://conda.anaconda.org/conda-forge/linux-64/hdf4-4.2.15-h2a13503_7.conda#bd77f8da987968ec3927990495dc22e4
https://conda.anaconda.org/conda-forge/linux-64/krb5-1.22.2-hbc21106_2.conda#53318d715316929a574f83591308b1f8
+https://conda.anaconda.org/conda-forge/linux-64/libcrc32c-1.1.2-h9c3ff4c_0.tar.bz2#c965a5aa0d5c1c37ffc62dff36e28400
https://conda.anaconda.org/conda-forge/linux-64/libfreetype-2.14.3-ha770c72_2.conda#f8054e759d0ddddaf34a5c8fedc900a1
https://conda.anaconda.org/conda-forge/linux-64/libglib-2.90.0-h569388d_0.conda#c85e9539551f1fd21104e518d940c244
-https://conda.anaconda.org/conda-forge/linux-64/libjxl-0.12.0-heb2dce7_2.conda#485788d339785bac87fe86315b4a0627
+https://conda.anaconda.org/conda-forge/linux-64/libjxl-0.12.0-heb2dce7_4.conda#8770a39fd4c5559357efc7cdb8f0c742
https://conda.anaconda.org/conda-forge/linux-64/libmo_unpack-3.1.2-hf484d3e_1001.tar.bz2#95f32a6a5a666d33886ca5627239f03d
https://conda.anaconda.org/conda-forge/linux-64/libopenblas-0.3.34-pthreads_hf13c14d_2.conda#7e19fcd73d7f0997d3cb3326d2586a86
https://conda.anaconda.org/conda-forge/linux-64/libpsl-0.23.1-hd9e3e90_1.conda#77be60417aa572afcb48cbe0f967401d
https://conda.anaconda.org/conda-forge/linux-64/librttopo-1.1.0-h55b0958_21.conda#6c639697369e9669032ca08d0a566282
-https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.53.4-h13e7031_1.conda#e72bbec309c2b0f37823ee7d4fabfcd3
+https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.54.0-h13e7031_1.conda#bb145547395d3329f3a1d930de9848b2
https://conda.anaconda.org/conda-forge/linux-64/libtheora-1.2.0-h85c0a6d_0.conda#177ef038f292390abe69137711fc0cbf
https://conda.anaconda.org/conda-forge/linux-64/libtiff-4.7.2-hcc2c06a_1.conda#0907d876f460ebc941ae590338b6102f
-https://conda.anaconda.org/conda-forge/linux-64/libxml2-16-2.15.4-hf3af7cc_0.conda#b7545c57a73a51d72aa6eee5695cf051
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-16-2.15.4-hf3af7cc_1.conda#046d9eada488e1a1657c80b55c23da52
https://conda.anaconda.org/conda-forge/linux-64/qhull-2020.2-h434a139_5.conda#353823361b1d27eb3960efb076dfcaf6
https://conda.anaconda.org/conda-forge/linux-64/udunits2-2.2.28-h40f5838_3.conda#6bb8deb138f87c9d48320ac21b87e7a1
https://conda.anaconda.org/conda-forge/linux-64/uriparser-0.9.8-hac33072_0.conda#d71d3a66528853c0a1ac2c02d79a0284
@@ -133,7 +135,7 @@ https://conda.anaconda.org/conda-forge/linux-64/xcb-util-wm-0.4.2-hb711507_0.con
https://conda.anaconda.org/conda-forge/linux-64/xerces-c-3.3.0-h78fed68_2.conda#f2ff0de6c209e90c53b3a195c2708f5f
https://conda.anaconda.org/conda-forge/linux-64/xorg-libx11-1.8.13-he1eb515_1.conda#8c282bbe4808a3cc80a5c98e9aec1cfc
https://conda.anaconda.org/conda-forge/linux-64/atk-1.0-2.38.0-h04ea711_2.conda#f730d54ba9cd543666d7220c9f7ed563
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-http-0.11.0-hbe094ff_5.conda#c09f7ec7327b8bd52fc2d6eacaa18d3b
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-http-1.0.0-h392d1a3_1.conda#6701f6e712f9513e722ba57cde99f023
https://conda.anaconda.org/conda-forge/linux-64/brotli-1.2.0-h505cf86_4.conda#746aa619a1bcaf4da541101179d7c60e
https://conda.anaconda.org/conda-forge/linux-64/cyrus-sasl-2.1.28-hac629b4_1.conda#af491aae930edc096b58466c51c4126c
https://conda.anaconda.org/conda-forge/linux-64/dbus-1.16.2-he8c428d_2.conda#a6eb37b51be1a9a654dc3a8aca039b1a
@@ -149,10 +151,10 @@ https://conda.anaconda.org/conda-forge/linux-64/libcurl-8.22.0-ha042cf0_0.conda#
https://conda.anaconda.org/conda-forge/linux-64/libglx-1.7.0-ha4b6fd6_5.conda#614e10aae180f87b84f0d1ca8ceb7d9e
https://conda.anaconda.org/conda-forge/linux-64/libkml-1.3.0-haa4a5bd_1023.conda#953b7cca897e21215302dbfe2af5cd0c
https://conda.anaconda.org/conda-forge/linux-64/libxkbfile-1.2.0-hb03c661_0.conda#f330a1b107cb5861f26dbb72a26f33e8
-https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.15.4-h7df9aa5_0.conda#ea192d1ef556bf1d55de4acb8c0f5b65
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.15.4-h7df9aa5_1.conda#51e5162f95bd6840f6b2b3169125fff5
https://conda.anaconda.org/conda-forge/linux-64/openjpeg-2.5.4-heb1ab33_2.conda#a3f3ac7a68d01c198e276dbb99e62032
-https://conda.anaconda.org/conda-forge/linux-64/python-3.13.15-hf47f18c_103_cp313.conda#641613f2d89da811bc29fa4cde88ef20
-https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.53.4-h9ffa6c4_1.conda#d42b24574925bfa3167efb3571c905ad
+https://conda.anaconda.org/conda-forge/linux-64/python-3.13.16-hf47f18c_102_cp313.conda#d406e245f44390c9c3adcd86cda21899
+https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.54.0-h9ffa6c4_1.conda#1060a8b69c6774ec85d81878eb299038
https://conda.anaconda.org/conda-forge/linux-64/xcb-util-image-0.4.0-hb711507_2.conda#a0901183f08b6c7107aab109733a3c91
https://conda.anaconda.org/conda-forge/linux-64/xkeyboard-config-2.48-h280c20c_0.conda#b233b41be0bf210989d57160ed39b394
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxext-1.3.7-h7cc23a3_1.conda#e5b6b28536b81b3f4cb20db4668a4642
@@ -161,34 +163,34 @@ https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrender-0.9.12-hb03c661_
https://conda.anaconda.org/conda-forge/noarch/aiohappyeyeballs-2.7.1-pyhd8ed1ab_0.conda#e51e09bf3b91c62b7461a9f94e92c2c9
https://conda.anaconda.org/conda-forge/noarch/alabaster-1.0.0-pyhd8ed1ab_1.conda#1fd9696649f65fd6611fcdb4ffec738a
https://conda.anaconda.org/conda-forge/noarch/attrs-26.1.0-pyhcf101f3_0.conda#c6b0543676ecb1fb2d7643941fe375f2
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-auth-0.10.4-h4610da3_2.conda#cefa84dbf94066e7a6902af288a2fedd
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-auth-1.0.0-h7648d90_2.conda#f4f169a23884de1bdb9c25ded8f2aec1
https://conda.anaconda.org/conda-forge/noarch/babel-2.18.0-pyhcf101f3_1.conda#f1976ce927373500cc19d3c0b2c85177
-https://conda.anaconda.org/conda-forge/linux-64/backports.zstd-1.7.0-py313h8f72f4d_1.conda#7ce840e6cf17fdba00e3d662f1d89e36
+https://conda.anaconda.org/conda-forge/linux-64/backports.zstd-1.8.0-py313h8f72f4d_0.conda#b2b55be09675e65d54fb1709208bd250
https://conda.anaconda.org/conda-forge/linux-64/brotli-python-1.2.0-py313h2fc2bef_4.conda#8beb0b96d9c701006ff5511a3a92329f
https://conda.anaconda.org/conda-forge/linux-64/cairo-1.18.6-h3c89d7e_0.conda#7920701f9abcb488411495742cf365a7
https://conda.anaconda.org/conda-forge/noarch/certifi-2026.7.22-pyhd8ed1ab_0.conda#37e13edbe3b48f1095a9d085ef9cd83b
https://conda.anaconda.org/conda-forge/noarch/cfgv-3.5.0-pyhd8ed1ab_0.conda#381bd45fb7aa032691f3063aff47e3a1
-https://conda.anaconda.org/conda-forge/noarch/charset-normalizer-3.5.1-pyhd8ed1ab_0.conda#e0ac3accc64e23e40969d660e5f58ac8
https://conda.anaconda.org/conda-forge/noarch/click-8.5.0-pyh5ded981_0.conda#dea5b13a211bbf99876deb980b408a66
https://conda.anaconda.org/conda-forge/noarch/cloudpickle-3.1.2-pyhcf101f3_1.conda#61b8078a0905b12529abc622406cb62c
https://conda.anaconda.org/conda-forge/noarch/colorama-0.4.6-pyhd8ed1ab_1.conda#962b9857ee8e7018c22f2776ffa0b2d7
https://conda.anaconda.org/conda-forge/noarch/colorcet-3.2.1-pyhd8ed1ab_0.conda#0b6419b9a27c540bd96e66e12a44c379
https://conda.anaconda.org/conda-forge/linux-64/coverage-7.16.2-py313h2af15c8_0.conda#97d81e36eb4c8fff723a1ff8a6c7d6fb
-https://conda.anaconda.org/conda-forge/noarch/cpython-3.13.15-py313hd8ed1ab_103.conda#23f5019df9a1d2287cb99af4c81ca8d2
+https://conda.anaconda.org/conda-forge/noarch/cpython-3.13.16-py313hd8ed1ab_102.conda#f06b4e55916af895989a48ea4c2732c1
https://conda.anaconda.org/conda-forge/noarch/cycler-0.12.1-pyhcf101f3_2.conda#4c2a8fef270f6c69591889b93f9f55c1
https://conda.anaconda.org/conda-forge/linux-64/cython-3.3.0-py313hb646302_0.conda#b7ddbe402c405d950a56bcb4ddb992a1
https://conda.anaconda.org/conda-forge/noarch/distlib-0.4.3-pyhcf101f3_0.conda#58638f77697c4f6726753eb8be34818b
https://conda.anaconda.org/conda-forge/noarch/docstring_parser-0.18.0-pyhd8ed1ab_0.conda#b0c7c29a60c82d57c5b4c4c38f0642c8
https://conda.anaconda.org/conda-forge/noarch/docutils-0.21.2-pyhd8ed1ab_1.conda#24c1ca34138ee57de72a943237cde4cc
https://conda.anaconda.org/conda-forge/noarch/execnet-2.1.2-pyhd8ed1ab_0.conda#a57b4be42619213a94f31d2c69c5dda7
-https://conda.anaconda.org/conda-forge/noarch/filelock-4.0.6-pyhd8ed1ab_0.conda#2a68742ffb325d6b25dcb3effff4234c
+https://conda.anaconda.org/conda-forge/noarch/filelock-4.1.1-pyhd8ed1ab_0.conda#fbff0cf6c30bdd9b38d0448fdadfe72d
https://conda.anaconda.org/conda-forge/linux-64/frozenlist-1.8.0-py313h6b9daa2_0.conda#a51a4bc5d02deee4354781783c132b7f
https://conda.anaconda.org/conda-forge/noarch/fsspec-2026.9.0-pyhd8ed1ab_0.conda#42de5cd170647599a84f468a42fabb89
+https://conda.anaconda.org/conda-forge/linux-64/google-crc32c-1.8.0-py313h5cbdff7_2.conda#c70fc04cb55616e53e2a2a835ffd1c69
https://conda.anaconda.org/conda-forge/noarch/hpack-4.2.0-pyhd8ed1ab_0.conda#b395909221b9bd1df066e5930e18855b
https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda#8e6923fc12f1fe8f8c4e5c9f343256ac
https://conda.anaconda.org/conda-forge/noarch/idna-3.20-pyh5ded981_0.conda#ee73be987f9de9ea5c2fdaeefa7b6491
https://conda.anaconda.org/conda-forge/noarch/imagesize-2.0.1-pyhd8ed1ab_0.conda#ebc56022e4ef7e74a829be94c53797ca
-https://conda.anaconda.org/conda-forge/noarch/iniconfig-2.3.0-pyhd8ed1ab_0.conda#9614359868482abba1bd15ce465e3c42
+https://conda.anaconda.org/conda-forge/noarch/iniconfig-2.3.1-pyhd8ed1ab_0.conda#c0a803d97dd09da7e2c8b0fdfaa5739b
https://conda.anaconda.org/conda-forge/noarch/iris-sample-data-2.5.2-pyhd8ed1ab_0.conda#895f6625dd8a246fece9279fcc12c1de
https://conda.anaconda.org/conda-forge/linux-64/jsonschema-rs-0.37.4-py313hdeb11d6_0.conda#e736b02c4a0905ad7636d798d48f3bed
https://conda.anaconda.org/conda-forge/linux-64/kiwisolver-1.5.1-py313h2551ef2_3.conda#45c8698dd577ba504513001219cfd0fd
@@ -199,25 +201,25 @@ https://conda.anaconda.org/conda-forge/linux-64/libgl-1.7.0-ha4b6fd6_5.conda#811
https://conda.anaconda.org/conda-forge/linux-64/libglx-devel-1.7.0-ha4b6fd6_5.conda#fb00b4fb800f2d431303a5c1625ef9d0
https://conda.anaconda.org/conda-forge/linux-64/libhwloc-2.15.0-default_hd7c4d0c_1001.conda#66c755b3bd9ec82308fa6ec7674f473f
https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.11.0-11_h47877c9_openblas.conda#5622a13917855ce4b953b9e655cf9262
-https://conda.anaconda.org/conda-forge/linux-64/libllvm21-21.1.8-h474f4eb_1.conda#580bc512b273f4236bb37031620c091f
https://conda.anaconda.org/conda-forge/linux-64/libxkbcommon-1.13.2-h51789e4_1.conda#f888787e0eab7a8076a67d197e155ffc
-https://conda.anaconda.org/conda-forge/linux-64/libxml2-devel-2.15.4-h7df9aa5_0.conda#9a414200ffe436e47d72c9f4c11c3658
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-devel-2.15.4-h7df9aa5_1.conda#c6f14f767cbfe20fa03a324beb6cd7eb
https://conda.anaconda.org/conda-forge/noarch/locket-1.0.0-pyhd8ed1ab_0.tar.bz2#91e27ef3d05cc772ce627e51cff111c4
-https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.3-py313h3dea7bd_1.conda#aeb9b9da79fd0258b3db091d1fefcd71
+https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.4-py313h2af15c8_1.conda#d15255d7f81d9188edecde1e024a1649
https://conda.anaconda.org/conda-forge/noarch/mdurl-0.1.2-pyhd8ed1ab_1.conda#592132998493b3ff25fd7479396e8351
-https://conda.anaconda.org/conda-forge/linux-64/msgpack-python-1.2.2-py313h2551ef2_2.conda#10eabac2cb17d77ce66a2d46890127bf
+https://conda.anaconda.org/conda-forge/linux-64/msgpack-python-1.2.3-py313h2551ef2_0.conda#276a99d974564361ffeef499749393e0
+https://conda.anaconda.org/conda-forge/linux-64/msgspec-0.22.0-py313h995f894_0.conda#fe6b5c45d2e8fff6cb46cbfa5cd8cff2
https://conda.anaconda.org/conda-forge/linux-64/multidict-6.7.1-py313h3dea7bd_0.conda#4f3e7bf5a9fc60a7d39047ba9e84c84c
https://conda.anaconda.org/conda-forge/noarch/munkres-1.1.4-pyhd8ed1ab_1.conda#37293a85a0f4f77bbd9cf7aaefc62609
https://conda.anaconda.org/conda-forge/linux-64/openldap-2.6.13-hbde042b_0.conda#680608784722880fbfe1745067570b00
https://conda.anaconda.org/conda-forge/noarch/packaging-26.3-pyhc364b38_0.conda#936687ed80f295a1f5dbcf8bd34c252c
https://conda.anaconda.org/conda-forge/linux-64/pillow-12.3.0-py313h3b26573_4.conda#6739325f3022c709753d26fff83ae20e
https://conda.anaconda.org/conda-forge/noarch/pip-26.2.1-pyh145f28c_0.conda#573cb3ed111004230ed3de650653cd4d
-https://conda.anaconda.org/conda-forge/noarch/platformdirs-4.12.1-pyh5ded981_0.conda#c2718a0d7dc28faf28e714807604a60d
+https://conda.anaconda.org/conda-forge/noarch/platformdirs-4.13.0-pyh5ded981_0.conda#35cc2a176fc6f886d6efa41ecb5f1de0
https://conda.anaconda.org/conda-forge/noarch/pluggy-1.6.0-pyhf9edf01_1.conda#d7585b6550ad04c8c5e21097ada2888e
https://conda.anaconda.org/conda-forge/linux-64/proj-9.8.1-he0df7b0_0.conda#b23619e5e9009eaa070ead0342034027
-https://conda.anaconda.org/conda-forge/linux-64/propcache-0.5.2-py313h3dea7bd_0.conda#9f8899a9482c1737bf402d8218780df1
+https://conda.anaconda.org/conda-forge/linux-64/propcache-0.5.4-py313h6cd9899_0.conda#c38cae05bdbe3d317baf350ebd9de3bb
https://conda.anaconda.org/conda-forge/linux-64/psutil-7.2.2-py313hd42f317_3.conda#0bc97f7c2f4a6993798c004fd557e3a1
-https://conda.anaconda.org/conda-forge/noarch/pycparser-3.0-pyhcf101f3_0.conda#9c5491066224083c41b6d5635ed7107b
+https://conda.anaconda.org/conda-forge/noarch/pycparser-3.11-pyh5ded981_0.conda#3cd164745122ab557cc30879390089e8
https://conda.anaconda.org/conda-forge/noarch/pygments-2.21.0-pyhcf101f3_0.conda#2882dee445dfa45b0c5afce3ffb7730a
https://conda.anaconda.org/conda-forge/noarch/pyparsing-3.3.3-pyh5ded981_0.conda#8bf43ae1312e9b966063459e04c20fa7
https://conda.anaconda.org/conda-forge/noarch/pyshp-3.1.6-pyhcf101f3_0.conda#6778d276eb7b4230d622bdd2a732c75e
@@ -229,12 +231,12 @@ https://conda.anaconda.org/conda-forge/noarch/setuptools-84.0.0-pyh332efcf_0.con
https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda#3339e3b65d58accf4ca4fb8748ab16b3
https://conda.anaconda.org/conda-forge/noarch/snowballstemmer-3.1.1-pyhd8ed1ab_0.conda#46b6abe31482f6bca064b965696ae807
https://conda.anaconda.org/conda-forge/noarch/sortedcontainers-2.4.0-pyhd8ed1ab_1.conda#0401a17ae845fa72c7210e206ec5647d
-https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.10-pyhd8ed1ab_0.conda#6c2382c11f0c0486f3300a761e5c1205
+https://conda.anaconda.org/conda-forge/noarch/soupsieve-3.0.3-pyhd8ed1ab_0.conda#bd2ea090567b9d0a53240bd1367c87ad
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-jsmath-1.0.1-pyhd8ed1ab_1.conda#fa839b5ff59e192f411ccc7dae6588bb
https://conda.anaconda.org/conda-forge/noarch/tabulate-0.10.0-pyhcf101f3_0.conda#3b887b7b3468b0f494b4fad40178b043
https://conda.anaconda.org/conda-forge/noarch/tblib-3.2.2-pyhcf101f3_0.conda#f88bb644823094f436792f80fba3207e
-https://conda.anaconda.org/conda-forge/noarch/tomli-2.4.1-pyhcf101f3_0.conda#b5325cf06a000c5b14970462ff5e4d58
-https://conda.anaconda.org/conda-forge/noarch/toolz-1.1.0-pyhd8ed1ab_1.conda#c07a6153f8306e45794774cf9b13bd32
+https://conda.anaconda.org/conda-forge/noarch/tomli-2.5.0-pyh5ded981_0.conda#36d6a162326c52d6339c8b606f43932d
+https://conda.anaconda.org/conda-forge/noarch/toolz-1.2.0-pyh5ded981_0.conda#17f8db9088267a3cb38d56cc22d268ae
https://conda.anaconda.org/conda-forge/linux-64/tornado-6.5.10-py313h995f894_0.conda#94985bd39e52fedc2270adb00e4779e6
https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.16.0-pyhcf101f3_0.conda#c70ad746c22219b9700931707482992c
https://conda.anaconda.org/conda-forge/linux-64/xcb-util-cursor-0.1.6-hb03c661_0.conda#4d1fc190b99912ed557a8236e958c559
@@ -246,46 +248,46 @@ https://conda.anaconda.org/conda-forge/linux-64/xorg-libxinerama-1.1.6-hecca717_
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrandr-1.5.5-h7cc23a3_1.conda#798a8c9d171859a022e1cb89e7d0eb10
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxxf86vm-1.1.7-hb03c661_0.conda#665d152b9c6e78da404086088077c844
https://conda.anaconda.org/conda-forge/noarch/zict-3.0.0-pyhd8ed1ab_1.conda#e52c2ef711ccf31bb7f70ca87d144b9e
-https://conda.anaconda.org/conda-forge/noarch/zipp-4.1.0-pyhcf101f3_0.conda#ba3dcdc8584155c97c648ae9c044b7a3
+https://conda.anaconda.org/conda-forge/noarch/zipp-4.1.1-pyh5ded981_0.conda#6cad4d2cec9b7b64538c225ac91d2e8a
https://conda.anaconda.org/conda-forge/noarch/accessible-pygments-0.0.5-pyhd8ed1ab_1.conda#74ac5069774cdbc53910ec4d631a3999
https://conda.anaconda.org/conda-forge/noarch/affine-3.0.1-pyhecae5ae_0.conda#aeb404cab9e719a49264ab40928f80c8
https://conda.anaconda.org/conda-forge/noarch/aiosignal-1.4.0-pyhd8ed1ab_0.conda#421a865222cd0c9d83ff08bc78bf3a61
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-s3-0.13.1-h7508075_1.conda#2daf54d3eef87b757f68d7597534dd2f
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-s3-1.0.0-hfe03eda_1.conda#7a8a507033f2e478670a617bb4ffb0b8
https://conda.anaconda.org/conda-forge/linux-64/cffi-2.1.1-py313ha52865f_3.conda#2b17c17965f71b59158030dce83832e1
https://conda.anaconda.org/conda-forge/noarch/click-default-group-1.2.4-pyhd8ed1ab_1.conda#7cd83dd6831b61ad9624a694e4afd7dc
https://conda.anaconda.org/conda-forge/noarch/click-plugins-1.1.1.2-pyhd8ed1ab_0.conda#e9b05deb91c013e5224672a4ba9cf8d1
https://conda.anaconda.org/conda-forge/noarch/cligj-0.7.2-pyhd8ed1ab_2.conda#55c7804f428719241a90b152016085a1
-https://conda.anaconda.org/conda-forge/linux-64/cytoolz-1.1.0-py313h995f894_3.conda#53f596d8ef2fdb4efbbeb4d50d936c5a
+https://conda.anaconda.org/conda-forge/linux-64/cytoolz-1.2.0-py313hd42f317_0.conda#4667e0f1ee7d0f9dba663c070323f9a1
+https://conda.anaconda.org/conda-forge/noarch/donfig-0.8.1.post1-pyhd8ed1ab_1.conda#c56a7fa5597ad78b62e1f5d21f7f8b8f
https://conda.anaconda.org/conda-forge/noarch/exceptiongroup-1.3.1-pyhd8ed1ab_0.conda#8e662bd460bda79b1ea39194e3c4c9ab
-https://conda.anaconda.org/conda-forge/linux-64/fonttools-4.66.0-py313h6cd9899_0.conda#4dbb35b5d7bd8bd0d08e1579ab5c4be9
+https://conda.anaconda.org/conda-forge/linux-64/fonttools-4.66.1-py313h6cd9899_0.conda#bc5e99327ffd810f4e3574315ed2d954
https://conda.anaconda.org/conda-forge/linux-64/gl2ps-1.4.2-h36e74d4_2.conda#787c780ff43f9f79d78d01e476b81a7c
https://conda.anaconda.org/conda-forge/linux-64/glew-2.3.0-h71661d4_0.conda#7eefecda1c71c380bfc406d16e78bbee
https://conda.anaconda.org/conda-forge/noarch/h2-4.4.1-pyhcf101f3_0.conda#e652ac7756069c456d0da2a922cd7df5
https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.1-pyhcf101f3_0.conda#77ec68e0aa61c3fff9ecbf840f93d64e
-https://conda.anaconda.org/conda-forge/noarch/importlib_resources-7.1.0-pyhd8ed1ab_0.conda#0ba6225c279baf7ea9473a62ea0ec9ae
https://conda.anaconda.org/conda-forge/noarch/jinja2-3.1.6-pyhcf101f3_1.conda#04558c96691bed63104678757beb4f8d
https://conda.anaconda.org/conda-forge/noarch/lazy-loader-0.6-pyhd8ed1ab_0.conda#4ed11ec126e6da490282edfd0673fea0
https://conda.anaconda.org/conda-forge/linux-64/libgl-devel-1.7.0-ha4b6fd6_5.conda#f5f7fc038c611a25b22758c0a40d25c9
-https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-14.5.0-h23af247_0.conda#4c0d9662a054c51e69b66a427e7ead03
-https://conda.anaconda.org/conda-forge/linux-64/libpq-18.6-h3883810_2.conda#bf4a620772888bef1da4c33f87fd1f72
+https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-14.6.0-h23af247_2.conda#6a93147cab011f36d1659745c3f579e2
+https://conda.anaconda.org/conda-forge/linux-64/libpq-18.6-h3883810_3.conda#cc998047391971a452fed680158329f3
https://conda.anaconda.org/conda-forge/linux-64/libspatialite-5.1.0-gpl_h03b4e02_121.conda#affb2721df7014b4abcd75cc539c6cef
https://conda.anaconda.org/conda-forge/linux-64/libvulkan-loader-1.4.357.0-h0e34353_2.conda#c5f5f159b66332152197893427071695
https://conda.anaconda.org/conda-forge/noarch/markdown-it-py-4.2.0-pyhd8ed1ab_0.conda#6d03368f2b2b0a5fb6839df53b2eb5e0
https://conda.anaconda.org/conda-forge/noarch/nodeenv-1.10.0-pyhd8ed1ab_0.conda#eb52d14a901e23c39e9e7b4a1a5c015f
-https://conda.anaconda.org/conda-forge/linux-64/numpy-2.5.3-py313hb5f73ae_0.conda#efea38e4c681d5c33aade39330bea01c
+https://conda.anaconda.org/conda-forge/linux-64/numpy-2.5.4-py313hb5f73ae_0.conda#1ddee4b1dc2f784b4137a43789be3816
https://conda.anaconda.org/conda-forge/noarch/partd-1.4.2-pyhd8ed1ab_0.conda#0badf9c54e24cecfb0ad2f99d680c163
-https://conda.anaconda.org/conda-forge/noarch/pbr-7.1.1-pyhd8ed1ab_0.conda#51b000c0a6d35e244019f9fc8ab1c9f6
+https://conda.anaconda.org/conda-forge/noarch/pbr-7.1.3-pyhd8ed1ab_0.conda#66415d5d2894f1d10928005a264c2717
https://conda.anaconda.org/conda-forge/linux-64/pyproj-3.8.0-py313hd11bf8e_1.conda#4fcc7972635bb83fcc9376559c338e4f
https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.9.0.post0-pyhe01879c_2.conda#5b8d21249ff20967101ffa321cab24e8
-https://conda.anaconda.org/conda-forge/noarch/python-discovery-1.6.1-pyh5ded981_0.conda#ae0feb6a3f297952bd4f4270fe77f530
-https://conda.anaconda.org/conda-forge/noarch/python-gil-3.13.15-h4df99d1_103.conda#86d251460e192368d2982e5b0dcad798
+https://conda.anaconda.org/conda-forge/noarch/python-discovery-1.6.2-pyh5ded981_0.conda#17e6c8b2fc2d021a217d317fe4f89fc4
+https://conda.anaconda.org/conda-forge/noarch/python-gil-3.13.16-h4df99d1_102.conda#6577480ed084a42b5f000087df16b894
https://conda.anaconda.org/conda-forge/linux-64/tbb-2023.1.0-haa8c3ee_2.conda#f805909f478c179e230baf7bd5878e26
https://conda.anaconda.org/conda-forge/noarch/typing-extensions-4.16.0-h69aa097_0.conda#c680b5747e8c4c8f23dca0bb7042a8fc
-https://conda.anaconda.org/conda-forge/noarch/vcs_versioning-2.5.0-pyh5ded981_0.conda#137d3106e230ae55fe7b8ee09c55aaf0
+https://conda.anaconda.org/conda-forge/noarch/vcs_versioning-2.6.0-pyh5ded981_0.conda#958f3e46d5025cdd9aa9c2417adceec8
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxtst-1.2.5-h7cc23a3_4.conda#752a5ac9322e0fbbc24146c1ce3ae44e
-https://conda.anaconda.org/conda-forge/linux-64/yarl-1.24.5-py313h3dea7bd_0.conda#abf83ec0fc49a445929114ed51133d7b
+https://conda.anaconda.org/conda-forge/linux-64/yarl-1.25.1-py313h6cd9899_0.conda#3e0ea10bd102b49c5e5b9d61f450673c
https://conda.anaconda.org/conda-forge/noarch/_python_abi3_support-1.0-hd8ed1ab_3.conda#3845f3d75991bae0fb90884662f4327c
-https://conda.anaconda.org/conda-forge/linux-64/aiohttp-3.14.3-py313hd6074c6_0.conda#9a084d4234ccd2e19fee2579414d773b
+https://conda.anaconda.org/conda-forge/linux-64/aiohttp-3.14.5-py313h2af15c8_0.conda#76e62dcb31698f72375b91a2c0882ae1
https://conda.anaconda.org/conda-forge/noarch/asv_runner-0.3.1-pyhd8ed1ab_0.conda#960a0f64467ce624941ad7c67b80a2e2
https://conda.anaconda.org/conda-forge/linux-64/at-spi2-core-2.40.3-h0630a04_0.tar.bz2#8cb2fc4cd6cc63f1369cfa318f581cc3
https://conda.anaconda.org/conda-forge/noarch/beautifulsoup4-4.15.0-pyha770c72_0.conda#3b261da3fe9b4168738712832410b022
@@ -293,14 +295,14 @@ https://conda.anaconda.org/conda-forge/linux-64/cftime-1.6.5-py313h12b3e8d_2.con
https://conda.anaconda.org/conda-forge/noarch/colorspacious-1.1.2-pyhecae5ae_1.conda#04151bb8e351c6209caad045e4b1f4bd
https://conda.anaconda.org/conda-forge/linux-64/contourpy-1.4.0-py313h2551ef2_1.conda#1883f23b947ffaa13f9511ea817fc4e6
https://conda.anaconda.org/conda-forge/noarch/dask-core-2026.8.0-pyhc364b38_0.conda#2ccdf45d2372e15c7edcb7a72d573e76
-https://conda.anaconda.org/conda-forge/linux-64/hdf5-2.2.0-nompi_hc529623_100.conda#336f7fb9af0c49cc246e2cff0992c3c4
-https://conda.anaconda.org/conda-forge/noarch/importlib-resources-7.1.0-pyhd8ed1ab_0.conda#e3bffa82b874f8b9a2631bddb3869529
+https://conda.anaconda.org/conda-forge/linux-64/hdf5-2.2.0-nompi_hdbb200c_101.conda#66762e6067bf7d52586ff21c78c970db
https://conda.anaconda.org/conda-forge/noarch/lazy_loader-0.6-pyhd8ed1ab_0.conda#baa2be54811ef381364ae730a4613d0f
https://conda.anaconda.org/conda-forge/linux-64/libegl-devel-1.7.0-ha4b6fd6_5.conda#577800fc8c9d8b7d9727246bbfde704e
https://conda.anaconda.org/conda-forge/linux-64/libgdal-core-3.13.3-h31aba09_1.conda#44c6780e3082468d8676aebdcd5068a6
-https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-devel-14.5.0-h23af247_0.conda#82f4063122b5ebc18cc34bb269e941f3
+https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-devel-14.6.0-h23af247_2.conda#3cf459e930e6ea2e57bbf04be7fcb7c6
https://conda.anaconda.org/conda-forge/linux-64/libraqm-0.11.0-h6406941_0.conda#3ac89a48d224409739dbf6200e524373
https://conda.anaconda.org/conda-forge/linux-64/mo_pack-0.3.1-py313h12b3e8d_3.conda#5a7f561522fe020e1ff4b31522ec37b0
+https://conda.anaconda.org/conda-forge/linux-64/numcodecs-0.17.0-py313h12b3e8d_1.conda#a9a5095b73124de28d231040ea871de1
https://conda.anaconda.org/conda-forge/linux-64/pandas-3.0.6-np2py313h4900a6c_0.conda#871596bf7cdadf8ccff03437e9b92173
https://conda.anaconda.org/conda-forge/linux-64/pango-1.58.2-h7bb47b9_1.conda#0b889b1c6b472d215d6332c6b195589c
https://conda.anaconda.org/conda-forge/linux-64/pykdtree-1.4.3-py313h12b3e8d_3.conda#3b2d1978d69a506ef1487feddd12f7b9
@@ -310,50 +312,53 @@ https://conda.anaconda.org/conda-forge/linux-64/pywavelets-1.10.0-py313hd42f317_
https://conda.anaconda.org/conda-forge/noarch/rich-15.0.0-pyhcf101f3_0.conda#0242025a3c804966bf71aa04eee82f66
https://conda.anaconda.org/conda-forge/linux-64/scipy-1.18.1-py313he2a5f1a_1.conda#70c6a7ea500089cf1543378c69f94fac
https://conda.anaconda.org/conda-forge/noarch/setuptools-scm-10.3.4-pyh5ded981_0.conda#437557778d99cc6f311f3c0ce023ad85
-https://conda.anaconda.org/conda-forge/linux-64/shapely-2.1.2-py313h14b1fe9_3.conda#1e704b28345f0404b56d63a8978293a2
+https://conda.anaconda.org/conda-forge/linux-64/shapely-2.2.0-np2py313ha973fa7_2.conda#9f38bb516b77ea8c97348d514b2abdbe
https://conda.anaconda.org/conda-forge/noarch/snuggs-1.4.7-pyhd8ed1ab_2.conda#9aa358575bbd4be126eaa5e0039f835c
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-apidoc-0.3.0-py_1.tar.bz2#855b087883443abb10f5faf6eef40860
+https://conda.anaconda.org/conda-forge/noarch/towncrier-26.9.0-pyhd8ed1ab_0.conda#d4167f473b71694bbca708347abee99e
https://conda.anaconda.org/conda-forge/linux-64/ukkonen-1.1.0-py313h7037e92_0.conda#cb423e0853b3dde2b3738db4dedf5ba2
https://conda.anaconda.org/conda-forge/noarch/urllib3-2.8.0-pyhd8ed1ab_0.conda#407a3d3778570bae8e8fcf554e803e43
-https://conda.anaconda.org/conda-forge/noarch/virtualenv-21.13.0-pyh5ded981_0.conda#e553e26b7aa15efa4640ec55859dfc53
+https://conda.anaconda.org/conda-forge/noarch/virtualenv-21.14.7-pyh5ded981_0.conda#1635534aa129eb9ab6ce5e01929d86ff
https://conda.anaconda.org/conda-forge/linux-64/at-spi2-atk-2.38.0-h0630a04_3.tar.bz2#6b889f174df1e0f816276ae69281af4d
-https://conda.anaconda.org/conda-forge/linux-64/cf-units-3.3.1-np2hc8c11d4_1.conda#49647e9c24724a189e0e334f4f3223e7
+https://conda.anaconda.org/conda-forge/linux-64/cf-units-3.3.1-np2hc8c11d4_2.conda#9b777f085506bff796833be0ae292db1
+https://conda.anaconda.org/conda-forge/linux-64/charset-normalizer-3.5.2-h1114479_0.conda#78d5c6112c9320a1a797935cecfec7c2
https://conda.anaconda.org/conda-forge/noarch/distributed-2026.8.0-pyhc364b38_0.conda#f382c4eed5376ee372a6e67a0f6cd4d9
https://conda.anaconda.org/conda-forge/linux-64/epoxy-1.5.10-hb03c661_2.conda#057083b06ccf1c2778344b6dabace38b
-https://conda.anaconda.org/conda-forge/linux-64/harfbuzz-14.5.0-ha770c72_0.conda#ef253035e60216967c117ab9a279eea2
+https://conda.anaconda.org/conda-forge/linux-64/harfbuzz-14.6.0-ha770c72_2.conda#0709eacc38906e8a237d9ef172af6688
https://conda.anaconda.org/conda-forge/noarch/identify-2.6.20-pyhd8ed1ab_0.conda#be9ea4f030b4b2bc121b2a548b066c0b
https://conda.anaconda.org/conda-forge/noarch/imagehash-4.3.2-pyhd8ed1ab_0.conda#efbc812363856906a80344b496389d2e
https://conda.anaconda.org/conda-forge/linux-64/libnetcdf-4.10.1-nompi_he3e3c8e_201.conda#912aefa694a66897afeb7a2330b17d9f
https://conda.anaconda.org/conda-forge/linux-64/librsvg-2.62.4-ha427ee3_0.conda#f9bda2728debbde4c641bd1af103c623
https://conda.anaconda.org/conda-forge/linux-64/matplotlib-base-3.11.2-py313he85fb2c_1.conda#85e795dd9e44b84961fafb4e7dd589aa
-https://conda.anaconda.org/conda-forge/linux-64/minijinja-2.24.0-py310h300b7de_0.conda#1be3473b8f1e33cb92ce3401fab8e8a6
https://conda.anaconda.org/conda-forge/noarch/pytest-cov-7.1.0-pyhcf101f3_0.conda#67d1790eefa81ed305b89d8e314c7923
https://conda.anaconda.org/conda-forge/noarch/pytest-mock-3.16.0-pyhd8ed1ab_0.conda#b038267c191ebae9d6374caf30cc2cca
https://conda.anaconda.org/conda-forge/noarch/pytest-xdist-3.8.0-pyhd8ed1ab_0.conda#8375cfbda7c57fbceeda18229be10417
+https://conda.anaconda.org/conda-forge/noarch/python-abi3-3.13-h9bebd56_3.conda#ba55acfc336780a55ee658d8608aa2d4
https://conda.anaconda.org/conda-forge/linux-64/python-stratify-0.4.0-py313h995f894_4.conda#7082bb91cc3fe449f0ba89bff1fc8ea4
-https://conda.anaconda.org/conda-forge/linux-64/qt6-main-6.11.2-pl5321h9df5c37_0.conda#2ff98660281d1c6bfebda5bffa52cc89
-https://conda.anaconda.org/conda-forge/linux-64/rasterio-1.5.1-py313ha8e1118_1.conda#5d82c0250bf4a2cdb02ebf9b7d3cf5f9
-https://conda.anaconda.org/conda-forge/noarch/requests-2.34.2-pyhcf101f3_0.conda#4a85203c1d80c1059086ae860836ffb9
+https://conda.anaconda.org/conda-forge/linux-64/qt6-main-6.12.0-pl5321hcf69127_1.conda#db3e81b8f192e5c9ee8e1e8c215c40b0
+https://conda.anaconda.org/conda-forge/linux-64/rasterio-1.5.2-py313h548c9e1_1.conda#697e246bb0b7eab5d417561af1dd291b
https://conda.anaconda.org/conda-forge/noarch/rich-rst-2.2.0-pyhd8ed1ab_0.conda#d91900e108041fa4d06ff329970027a1
-https://conda.anaconda.org/conda-forge/noarch/towncrier-25.8.0-pyhd8ed1ab_0.conda#3e0e8e44292bdac62f7bcbf0450b5cc7
https://conda.anaconda.org/conda-forge/linux-64/viskores-1.1.1-cpu_ha53dc59_1.conda#228549cd2f01bb53664bd2b5b78d7389
-https://conda.anaconda.org/conda-forge/noarch/wslink-2.5.7-pyhd8ed1ab_0.conda#410dee7cbee308f33b01645f16f11efc
+https://conda.anaconda.org/conda-forge/noarch/wslink-2.6.0-pyhd8ed1ab_0.conda#c3bcfd3bb22061d48cb833925795105f
+https://conda.anaconda.org/conda-forge/noarch/zarr-3.4.1-pyhc364b38_0.conda#052cab2fa318f6ffeefc149addd9e6fb
https://conda.anaconda.org/conda-forge/noarch/adwaita-icon-theme-51.0-unix_0.conda#6f62266fb2bfc35a5071cfa1d6ffdcc9
-https://conda.anaconda.org/conda-forge/linux-64/cartopy-0.26.0-np2py313h38606b7_1.conda#313aeba76e49560ecc5349605b13a364
+https://conda.anaconda.org/conda-forge/linux-64/cartopy-0.25.0-np2py313h73dcb5b_3.conda#85793522f0ed1e2d31e7dfc53fd66035
https://conda.anaconda.org/conda-forge/noarch/cmocean-4.0.3-pyhd8ed1ab_1.conda#dd71e4ec2fbffe38c0359976505f816e
-https://conda.anaconda.org/conda-forge/noarch/cyclopts-5.0.0-pyh5ded981_0.conda#3a90e8cb67cb4fc314a2696bcf894ed8
+https://conda.anaconda.org/conda-forge/noarch/cyclopts-5.2.0-pyh5ded981_0.conda#fae753a13c1df14b3aeaf2f8e5b5b314
https://conda.anaconda.org/conda-forge/linux-64/gtk3-3.24.52-ha5ea40c_0.conda#bcaea22d85999a4f17918acfab877e61
+https://conda.anaconda.org/conda-forge/linux-64/minijinja-3.0.0-py311h1180b86_0.conda#ebba9c2cc80a43f736c13772bee155d8
https://conda.anaconda.org/conda-forge/noarch/nc-time-axis-1.4.1-pyhd8ed1ab_1.conda#9a2be7d0089f5934b550933ca0d9fe85
https://conda.anaconda.org/conda-forge/linux-64/netcdf-fortran-4.6.4-nompi_h52816da_101.conda#2f18bb969e8acd8a1f45293e9782374e
https://conda.anaconda.org/conda-forge/linux-64/netcdf4-1.7.4-nompi_py311hd34d2e2_111.conda#d697ebf773f33945402fe3e1fda13635
-https://conda.anaconda.org/conda-forge/noarch/pooch-1.9.0-pyhd8ed1ab_0.conda#dd4b6337bf8886855db6905b336db3c8
-https://conda.anaconda.org/conda-forge/noarch/pre-commit-4.6.2-pyha770c72_0.conda#c36b88db560b4e74f294380613f1a239
-https://conda.anaconda.org/conda-forge/noarch/requests-file-2.1.0-pyhd8ed1ab_1.conda#69be0472744969b97324d5d3b21845a9
+https://conda.anaconda.org/conda-forge/noarch/pre-commit-4.7.0-pyha770c72_0.conda#167169a52ede77767f9fa66b52983a20
+https://conda.anaconda.org/conda-forge/noarch/requests-2.34.2-pyhcf101f3_0.conda#4a85203c1d80c1059086ae860836ffb9
https://conda.anaconda.org/conda-forge/linux-64/vtk-base-9.7.0-py313h6adec9e_0.conda#f8db7b1c571bc85ccacb28a1a59ef355
https://conda.anaconda.org/conda-forge/linux-64/esmf-8.9.1-nompi_h39d0119_4.conda#074ca9356f9e916ec51ad6ab47fd55ba
https://conda.anaconda.org/conda-forge/linux-64/graphviz-14.1.2-h8b86629_0.conda#341fc61cfe8efa5c72d24db56c776f44
-https://conda.anaconda.org/conda-forge/noarch/pyvista-0.49.0-pyha804496_1.conda#ce10d5ff463de9dd9da958aa49a61064
+https://conda.anaconda.org/conda-forge/noarch/pooch-1.9.0-pyhd8ed1ab_0.conda#dd4b6337bf8886855db6905b336db3c8
+https://conda.anaconda.org/conda-forge/noarch/requests-file-2.1.0-pyhd8ed1ab_1.conda#69be0472744969b97324d5d3b21845a9
https://conda.anaconda.org/conda-forge/noarch/esmpy-8.9.1-pyhdfbf58e_0.conda#0155d4a55415cda7f7f09bc0b7760f51
+https://conda.anaconda.org/conda-forge/noarch/pyvista-0.49.1-pyha804496_0.conda#ea32ac5d548e14030818673f68e6a756
https://conda.anaconda.org/conda-forge/noarch/geovista-0.5.3-pyhd8ed1ab_1.conda#64348d05eedb1b1b5676f63101d004f2
https://conda.anaconda.org/conda-forge/noarch/pydata-sphinx-theme-0.19.0-pyhcf101f3_0.conda#87921f66a4dc56ce92e4ff13be5f63dc
https://conda.anaconda.org/conda-forge/noarch/sphinx-copybutton-0.5.2-pyhd8ed1ab_1.conda#bf22cb9c439572760316ce0748af3713
diff --git a/requirements/locks/py314-linux-64.lock b/requirements/locks/py314-linux-64.lock
index 7ba2672472..2b99ea97de 100644
--- a/requirements/locks/py314-linux-64.lock
+++ b/requirements/locks/py314-linux-64.lock
@@ -1,6 +1,6 @@
# Generated by conda-lock.
# platform: linux-64
-# input_hash: bc630aeac74f823732c3b1426795e276f8b84a88bd9febcc1befaa2561603add
+# input_hash: 5229ccb2d47adc9bc4e1cf97d02301218610f0857e80d366efae609fed73833c
@EXPLICIT
https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab24e00_0.tar.bz2#0c96522c6bdaed4b1566d11387caaf45
https://conda.anaconda.org/conda-forge/noarch/font-ttf-inconsolata-3.000-h77eed37_0.tar.bz2#34893075a5c9e55cdafac56607368fc6
@@ -19,9 +19,9 @@ https://conda.anaconda.org/conda-forge/noarch/fonts-conda-ecosystem-1-0.tar.bz2#
https://conda.anaconda.org/conda-forge/linux-64/libegl-1.7.0-ha4b6fd6_5.conda#3a1b41c4591a0a1734382af3e2b8bc08
https://conda.anaconda.org/conda-forge/linux-64/zlib-1.3.2-h25fd6f3_3.conda#6acb86426229f96f93e5468d1df3a5e8
https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.7-hb78ec9c_7.conda#aa459086047c0e5e27023ab19f8cb86a
-https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.46.1-default_hbd61a6d_102.conda#449500f2c089da11c40f5c21312e3e07
+https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.46.1-default_hbd61a6d_103.conda#2362c70e7e79827c32db35547d9c179b
https://conda.anaconda.org/conda-forge/linux-64/libgcc-16.2.0-ha9f2e26_7.conda#735629b400b70eeab21a2ba8f5b1c2ce
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-common-0.14.3-hb03c661_0.conda#ce8664d7c01d4a598ad38107b289b5ec
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-common-1.0.3-hebe6cf0_0.conda#cdb5797968bfe1737f1b71f525fc6518
https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-hda65f42_10.conda#e675fabcf81499adc7edf58124fb1e01
https://conda.anaconda.org/conda-forge/linux-64/c-ares-1.34.8-hebe6cf0_2.conda#3ad2b403be8fc5612b9159e2ce621f69
https://conda.anaconda.org/conda-forge/linux-64/fribidi-1.0.17-h7cc23a3_0.conda#f9d5302bca196ee87e62f393f2626562
@@ -41,13 +41,14 @@ https://conda.anaconda.org/conda-forge/linux-64/liblzma-5.8.3-hb03c661_1.conda#1
https://conda.anaconda.org/conda-forge/linux-64/libmpdec-4.0.0-hb03c661_2.conda#fcfed1dc5053eb1901b66e7b1fc32588
https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.1-hb9d3cd8_1.conda#d864d34357c3b65a4b731f78c0801dc4
https://conda.anaconda.org/conda-forge/linux-64/libpciaccess-0.19-hb03c661_1.conda#35fa2b34bbced424e6976d30f5fde576
-https://conda.anaconda.org/conda-forge/linux-64/libpng-1.6.58-h922cc85_1.conda#fcf71c8d979148873f6f8ad4cfc73d86
+https://conda.anaconda.org/conda-forge/linux-64/libpng-1.6.59-h922cc85_0.conda#8c9caf742a485ac688ffa88bbadbf0be
+https://conda.anaconda.org/conda-forge/linux-64/libpython-3.14.8-hd51c5da_102_cp314.conda#5bc9fb686950e9ef2543adc230476238
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-16.2.0-h934c35e_7.conda#9b6707e0caca917fa6ea9ee473f37a5d
https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.42.4-hcfc3c73_0.conda#54a948337e66706380ff24d6b5fde63f
https://conda.anaconda.org/conda-forge/linux-64/libwebp-base-1.6.0-hd42ef1d_1.conda#9332b53d0ea93c5d39e33be03a0c611a
https://conda.anaconda.org/conda-forge/linux-64/lzo-2.10-hebe6cf0_1003.conda#10e0aa58ed390ba598b728badf9eb1b7
https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.6-hdb14827_1.conda#ee6c0cd80a60961a1f48aa3e0b91f986
-https://conda.anaconda.org/conda-forge/linux-64/openssl-3.6.4-h781a0a9_0.conda#16e3034a330cc625f2c685edec60cf4e
+https://conda.anaconda.org/conda-forge/linux-64/openssl-3.6.5-h781a0a9_0.conda#5a51a2ea653f120240b1a9bf76e0aae9
https://conda.anaconda.org/conda-forge/linux-64/pthread-stubs-0.4-h7cc23a3_1004.conda#bb66b610707b811e3c65181a6431e7a8
https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_h1df4ec4_4.conda#676a2f9b1c4fcf57b70aa5c65f6c60d0
https://conda.anaconda.org/conda-forge/linux-64/xorg-libice-1.1.2-h280c20c_0.conda#85c9442aec283b4e464fa9ecc484a2f3
@@ -56,10 +57,10 @@ https://conda.anaconda.org/conda-forge/linux-64/xorg-libxdmcp-1.1.5-h7cc23a3_2.c
https://conda.anaconda.org/conda-forge/linux-64/xorg-xorgproto-2025.1-hebe6cf0_3.conda#c3d0a6f03ba3aa312777b41b20a71ec0
https://conda.anaconda.org/conda-forge/linux-64/xxhash-0.8.3-hec8ed23_1.conda#72ab4ea51529ec63ff010ae9fb62b02f
https://conda.anaconda.org/conda-forge/linux-64/yaml-0.2.5-hebe6cf0_3.conda#e741576fb8f89821ac7c1c537322a33d
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-cal-0.9.15-h6bbde05_1.conda#8f2b374c79908f271be3d0e9f1c462a8
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-compression-0.3.2-h01ee8f8_5.conda#ae862abdbb3160f97478d3d4749164c0
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-sdkutils-0.2.7-h01ee8f8_3.conda#412f0cf18cc4c4945655dba954ca2d76
-https://conda.anaconda.org/conda-forge/linux-64/aws-checksums-0.2.10-h01ee8f8_5.conda#4d70caf5260e0787675c3bf61ae5b64c
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-cal-1.0.0-hcf7459e_1.conda#2009b71b358163738f339c76640b512c
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-compression-1.0.0-ha404f16_1.conda#da0c7b86d5379eb44acfa084897e8b98
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-sdkutils-1.0.0-ha404f16_1.conda#cbbd999c74a030951365225909e376c7
+https://conda.anaconda.org/conda-forge/linux-64/aws-checksums-1.0.1-ha404f16_0.conda#10e6e2f11271e66dfad43eb002359f22
https://conda.anaconda.org/conda-forge/linux-64/geos-3.14.1-h55b0958_0.conda#5f856b72ca9cd5b5b4a1c43551804622
https://conda.anaconda.org/conda-forge/linux-64/graphite2-1.3.15-h54a6638_1.conda#f9fe2984587fa8235a6af6004760cd18
https://conda.anaconda.org/conda-forge/linux-64/icu-78.3-py310h44b86e0_2.conda#72a381cbad04f24b1c2a43ef707f45b4
@@ -68,12 +69,11 @@ https://conda.anaconda.org/conda-forge/linux-64/libaec-1.1.5-h088129d_0.conda#86
https://conda.anaconda.org/conda-forge/linux-64/libbrotlidec-1.2.0-ha411449_4.conda#0cfd601eb03cca403dcc248844787486
https://conda.anaconda.org/conda-forge/linux-64/libbrotlienc-1.2.0-h018ffa1_4.conda#4136f6e4ef4835cc7df39a707cbd511c
https://conda.anaconda.org/conda-forge/linux-64/libdrm-2.4.129-h7cc23a3_0.conda#64cc91512b6278c315349dc13c53f680
-https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20250104-pl5321h373387f_1.conda#50708d3b951d0f8e2d7f2df5b5edc040
+https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20260512-pl5321h373387f_0.conda#27dbd7e2984b8fe48e069379636ccfdd
https://conda.anaconda.org/conda-forge/linux-64/libfreetype6-2.14.3-h5e6c136_2.conda#b72a266a9317036fd0464cb98b027cd0
https://conda.anaconda.org/conda-forge/linux-64/libgfortran-16.2.0-h69a702a_7.conda#3a8d285f5f22138ef826ad54cf267e73
-https://conda.anaconda.org/conda-forge/linux-64/libhwy-1.4.0-h57c4cff_1.conda#d58bfbaaf51ed85771eae92c2e7f5816
+https://conda.anaconda.org/conda-forge/linux-64/libhwy-1.4.0-h57c4cff_2.conda#145497b38fe581a7760dd88b7377c96b
https://conda.anaconda.org/conda-forge/linux-64/libnghttp2-1.68.1-h74cf4be_1.conda#75d211f04ac87506f1f43420712a69fe
-https://conda.anaconda.org/conda-forge/linux-64/libpython-3.14.7-hdc7f604_106_cp314.conda#28af2158bb8dbb62e55b0af3b44f9d4a
https://conda.anaconda.org/conda-forge/linux-64/libssh2-1.11.1-h6154650_1.conda#5c526561e1d2a2e071941174eae84557
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-16.2.0-hdf11a46_7.conda#25a23bde5e26d548374f3e02b4669670
https://conda.anaconda.org/conda-forge/linux-64/libudunits2-2.2.28-h40f5838_3.conda#4bdace082e911a3e1f1f0b721bed5b56
@@ -85,34 +85,35 @@ https://conda.anaconda.org/conda-forge/linux-64/muparser-2.3.5-hee9eb32_1.conda#
https://conda.anaconda.org/conda-forge/linux-64/pcre2-10.47-h8b3dc9c_1.conda#06f6113cf1ff4a54b65f87ead132a5f1
https://conda.anaconda.org/conda-forge/linux-64/pixman-0.46.4-h54a6638_3.conda#0ee5bb30034b081a1386c1e2c98ab0a7
https://conda.anaconda.org/conda-forge/linux-64/readline-8.3-hd6e31c0_1.conda#69c01c781e7c8190bbdaf79e3848c5ca
-https://conda.anaconda.org/conda-forge/linux-64/s2n-1.7.6-h7e3ee7f_0.conda#e112cadcd79412d81496e784884bddf0
+https://conda.anaconda.org/conda-forge/linux-64/s2n-1.7.7-ha5dc4ae_0.conda#4928f1e1964a6f38f0da349b9d0e1c6a
https://conda.anaconda.org/conda-forge/linux-64/snappy-1.2.2-h34e00bb_2.conda#2b7e31ffbd476eecf33b3b833692339c
https://conda.anaconda.org/conda-forge/linux-64/wayland-1.26.0-hc1c935e_2.conda#88a47e22b53e50865b1cabe2eb648352
https://conda.anaconda.org/conda-forge/linux-64/xorg-libsm-1.2.6-h0d788c3_1.conda#aa7459ed9ad086ba11dda843d764b33d
https://conda.anaconda.org/conda-forge/linux-64/zlib-ng-2.3.3-hce19668_1.conda#1726acec19beeed1c764385cd8622405
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-io-0.27.5-h4f381ba_1.conda#948f5a4fddac779b7942626e5807caba
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-io-1.0.0-h07ba609_0.conda#72593f68772d8c7e72c8a09681b0fe42
https://conda.anaconda.org/conda-forge/linux-64/blosc-1.21.6-he440d0b_1.conda#2c2fae981fd2afd00812c92ac47d023d
https://conda.anaconda.org/conda-forge/linux-64/brotli-bin-1.2.0-h9908984_4.conda#c48c09444f3736800ea0b9550c365baf
https://conda.anaconda.org/conda-forge/linux-64/freexl-2.0.0-h1f12c17_3.conda#4a0a8fd87ee4767db791047546f56501
https://conda.anaconda.org/conda-forge/linux-64/hdf4-4.2.15-h2a13503_7.conda#bd77f8da987968ec3927990495dc22e4
https://conda.anaconda.org/conda-forge/linux-64/krb5-1.22.2-hbc21106_2.conda#53318d715316929a574f83591308b1f8
+https://conda.anaconda.org/conda-forge/linux-64/libcrc32c-1.1.2-h9c3ff4c_0.tar.bz2#c965a5aa0d5c1c37ffc62dff36e28400
https://conda.anaconda.org/conda-forge/linux-64/libfreetype-2.14.3-ha770c72_2.conda#f8054e759d0ddddaf34a5c8fedc900a1
https://conda.anaconda.org/conda-forge/linux-64/libglib-2.90.0-h569388d_0.conda#c85e9539551f1fd21104e518d940c244
-https://conda.anaconda.org/conda-forge/linux-64/libjxl-0.12.0-heb2dce7_2.conda#485788d339785bac87fe86315b4a0627
+https://conda.anaconda.org/conda-forge/linux-64/libjxl-0.12.0-heb2dce7_4.conda#8770a39fd4c5559357efc7cdb8f0c742
https://conda.anaconda.org/conda-forge/linux-64/libmo_unpack-3.1.2-hf484d3e_1001.tar.bz2#95f32a6a5a666d33886ca5627239f03d
https://conda.anaconda.org/conda-forge/linux-64/libopenblas-0.3.34-pthreads_hf13c14d_2.conda#7e19fcd73d7f0997d3cb3326d2586a86
https://conda.anaconda.org/conda-forge/linux-64/libpsl-0.23.1-hd9e3e90_1.conda#77be60417aa572afcb48cbe0f967401d
https://conda.anaconda.org/conda-forge/linux-64/librttopo-1.1.0-h55b0958_21.conda#6c639697369e9669032ca08d0a566282
-https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.53.4-h13e7031_1.conda#e72bbec309c2b0f37823ee7d4fabfcd3
+https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.54.0-h13e7031_1.conda#bb145547395d3329f3a1d930de9848b2
https://conda.anaconda.org/conda-forge/linux-64/libtiff-4.7.2-hcc2c06a_1.conda#0907d876f460ebc941ae590338b6102f
-https://conda.anaconda.org/conda-forge/linux-64/libxml2-16-2.15.4-hf3af7cc_0.conda#b7545c57a73a51d72aa6eee5695cf051
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-16-2.15.4-hf3af7cc_1.conda#046d9eada488e1a1657c80b55c23da52
https://conda.anaconda.org/conda-forge/linux-64/qhull-2020.2-h434a139_5.conda#353823361b1d27eb3960efb076dfcaf6
https://conda.anaconda.org/conda-forge/linux-64/udunits2-2.2.28-h40f5838_3.conda#6bb8deb138f87c9d48320ac21b87e7a1
https://conda.anaconda.org/conda-forge/linux-64/uriparser-0.9.8-hac33072_0.conda#d71d3a66528853c0a1ac2c02d79a0284
https://conda.anaconda.org/conda-forge/linux-64/xerces-c-3.3.0-h78fed68_2.conda#f2ff0de6c209e90c53b3a195c2708f5f
https://conda.anaconda.org/conda-forge/linux-64/xorg-libx11-1.8.13-he1eb515_1.conda#8c282bbe4808a3cc80a5c98e9aec1cfc
https://conda.anaconda.org/conda-forge/linux-64/atk-1.0-2.38.0-h04ea711_2.conda#f730d54ba9cd543666d7220c9f7ed563
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-http-0.11.0-hbe094ff_5.conda#c09f7ec7327b8bd52fc2d6eacaa18d3b
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-http-1.0.0-h392d1a3_1.conda#6701f6e712f9513e722ba57cde99f023
https://conda.anaconda.org/conda-forge/linux-64/brotli-1.2.0-h505cf86_4.conda#746aa619a1bcaf4da541101179d7c60e
https://conda.anaconda.org/conda-forge/linux-64/dbus-1.16.2-he8c428d_2.conda#a6eb37b51be1a9a654dc3a8aca039b1a
https://conda.anaconda.org/conda-forge/linux-64/fontconfig-2.18.3-h4db4eae_1.conda#922776b528a470ab5afa81fd42abfa1d
@@ -126,41 +127,41 @@ https://conda.anaconda.org/conda-forge/linux-64/libcups-2.3.3-h7a8fb5f_6.conda#4
https://conda.anaconda.org/conda-forge/linux-64/libcurl-8.22.0-ha042cf0_0.conda#e617deee7af8eb0c04f6296d12b54157
https://conda.anaconda.org/conda-forge/linux-64/libglx-1.7.0-ha4b6fd6_5.conda#614e10aae180f87b84f0d1ca8ceb7d9e
https://conda.anaconda.org/conda-forge/linux-64/libkml-1.3.0-haa4a5bd_1023.conda#953b7cca897e21215302dbfe2af5cd0c
-https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.15.4-h7df9aa5_0.conda#ea192d1ef556bf1d55de4acb8c0f5b65
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.15.4-h7df9aa5_1.conda#51e5162f95bd6840f6b2b3169125fff5
https://conda.anaconda.org/conda-forge/linux-64/openjpeg-2.5.4-heb1ab33_2.conda#a3f3ac7a68d01c198e276dbb99e62032
-https://conda.anaconda.org/conda-forge/linux-64/python-3.14.7-hcd007b5_106_cp314.conda#56ee91e118243e46bfc0c41e4030d354
-https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.53.4-h9ffa6c4_1.conda#d42b24574925bfa3167efb3571c905ad
+https://conda.anaconda.org/conda-forge/linux-64/python-3.14.8-hcd007b5_102_cp314.conda#bcfadea0ff38555e26872aab0e974dbd
+https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.54.0-h9ffa6c4_1.conda#1060a8b69c6774ec85d81878eb299038
https://conda.anaconda.org/conda-forge/linux-64/xkeyboard-config-2.48-h280c20c_0.conda#b233b41be0bf210989d57160ed39b394
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxext-1.3.7-h7cc23a3_1.conda#e5b6b28536b81b3f4cb20db4668a4642
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxfixes-6.0.2-h7cc23a3_1.conda#09132e874fe0e1f5e4492b0b6b904b6e
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrender-0.9.12-hb03c661_1.conda#e470d224a7a5be1b1d021bded7abb536
https://conda.anaconda.org/conda-forge/noarch/alabaster-1.0.0-pyhd8ed1ab_1.conda#1fd9696649f65fd6611fcdb4ffec738a
https://conda.anaconda.org/conda-forge/noarch/attrs-26.1.0-pyhcf101f3_0.conda#c6b0543676ecb1fb2d7643941fe375f2
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-auth-0.10.4-h4610da3_2.conda#cefa84dbf94066e7a6902af288a2fedd
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-auth-1.0.0-h7648d90_2.conda#f4f169a23884de1bdb9c25ded8f2aec1
https://conda.anaconda.org/conda-forge/noarch/babel-2.18.0-pyhcf101f3_1.conda#f1976ce927373500cc19d3c0b2c85177
-https://conda.anaconda.org/conda-forge/noarch/backports.zstd-1.7.0-py314h680f03e_1.conda#67924260a851b5273e39f9b28b6b5e24
+https://conda.anaconda.org/conda-forge/noarch/backports.zstd-1.8.0-py314h680f03e_0.conda#5d5df669855f081b1c05f547f77925c6
https://conda.anaconda.org/conda-forge/linux-64/brotli-python-1.2.0-py314hcd2bdb6_4.conda#d9aabceecc99b3b8b0eb7090c5776af4
https://conda.anaconda.org/conda-forge/linux-64/cairo-1.18.6-h3c89d7e_0.conda#7920701f9abcb488411495742cf365a7
https://conda.anaconda.org/conda-forge/noarch/certifi-2026.7.22-pyhd8ed1ab_0.conda#37e13edbe3b48f1095a9d085ef9cd83b
https://conda.anaconda.org/conda-forge/noarch/cfgv-3.5.0-pyhd8ed1ab_0.conda#381bd45fb7aa032691f3063aff47e3a1
-https://conda.anaconda.org/conda-forge/noarch/charset-normalizer-3.5.1-pyhd8ed1ab_0.conda#e0ac3accc64e23e40969d660e5f58ac8
https://conda.anaconda.org/conda-forge/noarch/click-8.5.0-pyh5ded981_0.conda#dea5b13a211bbf99876deb980b408a66
https://conda.anaconda.org/conda-forge/noarch/cloudpickle-3.1.2-pyhcf101f3_1.conda#61b8078a0905b12529abc622406cb62c
https://conda.anaconda.org/conda-forge/noarch/colorama-0.4.6-pyhd8ed1ab_1.conda#962b9857ee8e7018c22f2776ffa0b2d7
https://conda.anaconda.org/conda-forge/linux-64/coverage-7.16.2-py314h93e9acf_0.conda#1101ca33d2a2af94aabe7438f9a7c9e1
-https://conda.anaconda.org/conda-forge/noarch/cpython-3.14.7-py314hd8ed1ab_106.conda#3b52b830ac00367685c08f98f50963bf
+https://conda.anaconda.org/conda-forge/noarch/cpython-3.14.8-py314hd8ed1ab_102.conda#54ecbe8d03ac833af969b3543bf116ef
https://conda.anaconda.org/conda-forge/noarch/cycler-0.12.1-pyhcf101f3_2.conda#4c2a8fef270f6c69591889b93f9f55c1
https://conda.anaconda.org/conda-forge/linux-64/cython-3.3.0-py314he5485c0_0.conda#c5d05ce7243daca0ff0ebac26d3d4e19
https://conda.anaconda.org/conda-forge/noarch/distlib-0.4.3-pyhcf101f3_0.conda#58638f77697c4f6726753eb8be34818b
https://conda.anaconda.org/conda-forge/noarch/docutils-0.21.2-pyhd8ed1ab_1.conda#24c1ca34138ee57de72a943237cde4cc
https://conda.anaconda.org/conda-forge/noarch/execnet-2.1.2-pyhd8ed1ab_0.conda#a57b4be42619213a94f31d2c69c5dda7
-https://conda.anaconda.org/conda-forge/noarch/filelock-4.0.6-pyhd8ed1ab_0.conda#2a68742ffb325d6b25dcb3effff4234c
+https://conda.anaconda.org/conda-forge/noarch/filelock-4.1.1-pyhd8ed1ab_0.conda#fbff0cf6c30bdd9b38d0448fdadfe72d
https://conda.anaconda.org/conda-forge/noarch/fsspec-2026.9.0-pyhd8ed1ab_0.conda#42de5cd170647599a84f468a42fabb89
+https://conda.anaconda.org/conda-forge/linux-64/google-crc32c-1.8.0-py314h34eed7f_2.conda#66938009746e41cdbccfeee9bb8a8db5
https://conda.anaconda.org/conda-forge/noarch/hpack-4.2.0-pyhd8ed1ab_0.conda#b395909221b9bd1df066e5930e18855b
https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda#8e6923fc12f1fe8f8c4e5c9f343256ac
https://conda.anaconda.org/conda-forge/noarch/idna-3.20-pyh5ded981_0.conda#ee73be987f9de9ea5c2fdaeefa7b6491
https://conda.anaconda.org/conda-forge/noarch/imagesize-2.0.1-pyhd8ed1ab_0.conda#ebc56022e4ef7e74a829be94c53797ca
-https://conda.anaconda.org/conda-forge/noarch/iniconfig-2.3.0-pyhd8ed1ab_0.conda#9614359868482abba1bd15ce465e3c42
+https://conda.anaconda.org/conda-forge/noarch/iniconfig-2.3.1-pyhd8ed1ab_0.conda#c0a803d97dd09da7e2c8b0fdfaa5739b
https://conda.anaconda.org/conda-forge/noarch/iris-sample-data-2.5.2-pyhd8ed1ab_0.conda#895f6625dd8a246fece9279fcc12c1de
https://conda.anaconda.org/conda-forge/linux-64/jsonschema-rs-0.37.4-py314h5059d10_0.conda#bbdf9e1de411fd55c05751bea6e2361c
https://conda.anaconda.org/conda-forge/linux-64/kiwisolver-1.5.1-py314h5383ef5_3.conda#f8b25504f794b11a671868f7262aeab0
@@ -171,19 +172,20 @@ https://conda.anaconda.org/conda-forge/linux-64/libgl-1.7.0-ha4b6fd6_5.conda#811
https://conda.anaconda.org/conda-forge/linux-64/libglx-devel-1.7.0-ha4b6fd6_5.conda#fb00b4fb800f2d431303a5c1625ef9d0
https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.11.0-11_h47877c9_openblas.conda#5622a13917855ce4b953b9e655cf9262
https://conda.anaconda.org/conda-forge/linux-64/libxkbcommon-1.13.2-h51789e4_1.conda#f888787e0eab7a8076a67d197e155ffc
-https://conda.anaconda.org/conda-forge/linux-64/libxml2-devel-2.15.4-h7df9aa5_0.conda#9a414200ffe436e47d72c9f4c11c3658
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-devel-2.15.4-h7df9aa5_1.conda#c6f14f767cbfe20fa03a324beb6cd7eb
https://conda.anaconda.org/conda-forge/noarch/locket-1.0.0-pyhd8ed1ab_0.tar.bz2#91e27ef3d05cc772ce627e51cff111c4
-https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.3-py314h67df5f8_1.conda#9a17c4307d23318476d7fbf0fedc0cde
-https://conda.anaconda.org/conda-forge/linux-64/msgpack-python-1.2.2-py314h5383ef5_2.conda#12730234204506e89076693b125e3c61
+https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.4-py314h93e9acf_1.conda#2ea393741e0fd221c98087478aeed284
+https://conda.anaconda.org/conda-forge/linux-64/msgpack-python-1.2.3-py314h5383ef5_0.conda#d91ed0c0fe2df67f0a80fdc47a9842e4
+https://conda.anaconda.org/conda-forge/linux-64/msgspec-0.22.0-py314h89acca1_0.conda#d0b947875606aa09cb40ee9900202b3b
https://conda.anaconda.org/conda-forge/noarch/munkres-1.1.4-pyhd8ed1ab_1.conda#37293a85a0f4f77bbd9cf7aaefc62609
https://conda.anaconda.org/conda-forge/noarch/packaging-26.3-pyhc364b38_0.conda#936687ed80f295a1f5dbcf8bd34c252c
https://conda.anaconda.org/conda-forge/linux-64/pillow-12.3.0-py314h50bfbbb_4.conda#2180323126daa9971a80cc679cf12f46
https://conda.anaconda.org/conda-forge/noarch/pip-26.2.1-pyh145f28c_0.conda#573cb3ed111004230ed3de650653cd4d
-https://conda.anaconda.org/conda-forge/noarch/platformdirs-4.12.1-pyh5ded981_0.conda#c2718a0d7dc28faf28e714807604a60d
+https://conda.anaconda.org/conda-forge/noarch/platformdirs-4.13.0-pyh5ded981_0.conda#35cc2a176fc6f886d6efa41ecb5f1de0
https://conda.anaconda.org/conda-forge/noarch/pluggy-1.6.0-pyhf9edf01_1.conda#d7585b6550ad04c8c5e21097ada2888e
https://conda.anaconda.org/conda-forge/linux-64/proj-9.8.1-he0df7b0_0.conda#b23619e5e9009eaa070ead0342034027
https://conda.anaconda.org/conda-forge/linux-64/psutil-7.2.2-py314hfe1a184_3.conda#23189243badc4f63a0f05c429aba161d
-https://conda.anaconda.org/conda-forge/noarch/pycparser-3.0-pyhcf101f3_0.conda#9c5491066224083c41b6d5635ed7107b
+https://conda.anaconda.org/conda-forge/noarch/pycparser-3.11-pyh5ded981_0.conda#3cd164745122ab557cc30879390089e8
https://conda.anaconda.org/conda-forge/noarch/pygments-2.21.0-pyhcf101f3_0.conda#2882dee445dfa45b0c5afce3ffb7730a
https://conda.anaconda.org/conda-forge/noarch/pyparsing-3.3.3-pyh5ded981_0.conda#8bf43ae1312e9b966063459e04c20fa7
https://conda.anaconda.org/conda-forge/noarch/pyshp-3.1.6-pyhcf101f3_0.conda#6778d276eb7b4230d622bdd2a732c75e
@@ -194,12 +196,12 @@ https://conda.anaconda.org/conda-forge/noarch/setuptools-84.0.0-pyh332efcf_0.con
https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda#3339e3b65d58accf4ca4fb8748ab16b3
https://conda.anaconda.org/conda-forge/noarch/snowballstemmer-3.1.1-pyhd8ed1ab_0.conda#46b6abe31482f6bca064b965696ae807
https://conda.anaconda.org/conda-forge/noarch/sortedcontainers-2.4.0-pyhd8ed1ab_1.conda#0401a17ae845fa72c7210e206ec5647d
-https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.10-pyhd8ed1ab_0.conda#6c2382c11f0c0486f3300a761e5c1205
+https://conda.anaconda.org/conda-forge/noarch/soupsieve-3.0.3-pyhd8ed1ab_0.conda#bd2ea090567b9d0a53240bd1367c87ad
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-jsmath-1.0.1-pyhd8ed1ab_1.conda#fa839b5ff59e192f411ccc7dae6588bb
https://conda.anaconda.org/conda-forge/noarch/tabulate-0.10.0-pyhcf101f3_0.conda#3b887b7b3468b0f494b4fad40178b043
https://conda.anaconda.org/conda-forge/noarch/tblib-3.2.2-pyhcf101f3_0.conda#f88bb644823094f436792f80fba3207e
-https://conda.anaconda.org/conda-forge/noarch/tomli-2.4.1-pyhcf101f3_0.conda#b5325cf06a000c5b14970462ff5e4d58
-https://conda.anaconda.org/conda-forge/noarch/toolz-1.1.0-pyhd8ed1ab_1.conda#c07a6153f8306e45794774cf9b13bd32
+https://conda.anaconda.org/conda-forge/noarch/tomli-2.5.0-pyh5ded981_0.conda#36d6a162326c52d6339c8b606f43932d
+https://conda.anaconda.org/conda-forge/noarch/toolz-1.2.0-pyh5ded981_0.conda#17f8db9088267a3cb38d56cc22d268ae
https://conda.anaconda.org/conda-forge/linux-64/tornado-6.5.10-py314h89acca1_0.conda#d2bd0918d9c10df4fe2355e33fe7bc63
https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.16.0-pyhcf101f3_0.conda#c70ad746c22219b9700931707482992c
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxcomposite-0.4.7-hb03c661_0.conda#f2ba4192d38b6cef2bb2c25029071d90
@@ -210,33 +212,33 @@ https://conda.anaconda.org/conda-forge/linux-64/xorg-libxinerama-1.1.6-hecca717_
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrandr-1.5.5-h7cc23a3_1.conda#798a8c9d171859a022e1cb89e7d0eb10
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxxf86vm-1.1.7-hb03c661_0.conda#665d152b9c6e78da404086088077c844
https://conda.anaconda.org/conda-forge/noarch/zict-3.0.0-pyhd8ed1ab_1.conda#e52c2ef711ccf31bb7f70ca87d144b9e
-https://conda.anaconda.org/conda-forge/noarch/zipp-4.1.0-pyhcf101f3_0.conda#ba3dcdc8584155c97c648ae9c044b7a3
+https://conda.anaconda.org/conda-forge/noarch/zipp-4.1.1-pyh5ded981_0.conda#6cad4d2cec9b7b64538c225ac91d2e8a
https://conda.anaconda.org/conda-forge/noarch/accessible-pygments-0.0.5-pyhd8ed1ab_1.conda#74ac5069774cdbc53910ec4d631a3999
https://conda.anaconda.org/conda-forge/noarch/affine-3.0.1-pyhecae5ae_0.conda#aeb404cab9e719a49264ab40928f80c8
-https://conda.anaconda.org/conda-forge/linux-64/aws-c-s3-0.13.1-h7508075_1.conda#2daf54d3eef87b757f68d7597534dd2f
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-s3-1.0.0-hfe03eda_1.conda#7a8a507033f2e478670a617bb4ffb0b8
https://conda.anaconda.org/conda-forge/linux-64/cffi-2.1.1-py314h8d76f0c_3.conda#ee66d46f7693fdcbaa1761fb8a4f6c3d
https://conda.anaconda.org/conda-forge/noarch/click-plugins-1.1.1.2-pyhd8ed1ab_0.conda#e9b05deb91c013e5224672a4ba9cf8d1
https://conda.anaconda.org/conda-forge/noarch/cligj-0.7.2-pyhd8ed1ab_2.conda#55c7804f428719241a90b152016085a1
-https://conda.anaconda.org/conda-forge/linux-64/cytoolz-1.1.0-py314h89acca1_3.conda#bec5d9071edb7fc8c600330cc55d2d49
+https://conda.anaconda.org/conda-forge/linux-64/cytoolz-1.2.0-py314hfe1a184_0.conda#f5c6a287281329672c700ffde2f0f4a2
+https://conda.anaconda.org/conda-forge/noarch/donfig-0.8.1.post1-pyhd8ed1ab_1.conda#c56a7fa5597ad78b62e1f5d21f7f8b8f
https://conda.anaconda.org/conda-forge/noarch/exceptiongroup-1.3.1-pyhd8ed1ab_0.conda#8e662bd460bda79b1ea39194e3c4c9ab
-https://conda.anaconda.org/conda-forge/linux-64/fonttools-4.66.0-py314hd42368d_0.conda#d4fd2f32eb93190b0c9e6dc594ca521b
+https://conda.anaconda.org/conda-forge/linux-64/fonttools-4.66.1-py314hd42368d_0.conda#3265ea810d2c81a2b37f0d5047179418
https://conda.anaconda.org/conda-forge/noarch/h2-4.4.1-pyhcf101f3_0.conda#e652ac7756069c456d0da2a922cd7df5
https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.1-pyhcf101f3_0.conda#77ec68e0aa61c3fff9ecbf840f93d64e
-https://conda.anaconda.org/conda-forge/noarch/importlib_resources-7.1.0-pyhd8ed1ab_0.conda#0ba6225c279baf7ea9473a62ea0ec9ae
https://conda.anaconda.org/conda-forge/noarch/jinja2-3.1.6-pyhcf101f3_1.conda#04558c96691bed63104678757beb4f8d
https://conda.anaconda.org/conda-forge/linux-64/libgl-devel-1.7.0-ha4b6fd6_5.conda#f5f7fc038c611a25b22758c0a40d25c9
-https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-14.5.0-h23af247_0.conda#4c0d9662a054c51e69b66a427e7ead03
+https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-14.6.0-h23af247_2.conda#6a93147cab011f36d1659745c3f579e2
https://conda.anaconda.org/conda-forge/linux-64/libspatialite-5.1.0-gpl_h03b4e02_121.conda#affb2721df7014b4abcd75cc539c6cef
https://conda.anaconda.org/conda-forge/noarch/nodeenv-1.10.0-pyhd8ed1ab_0.conda#eb52d14a901e23c39e9e7b4a1a5c015f
-https://conda.anaconda.org/conda-forge/linux-64/numpy-2.5.3-py314hd3a7d6b_0.conda#d1ae35d69e57534c8713ff0368b9fdb9
+https://conda.anaconda.org/conda-forge/linux-64/numpy-2.5.4-py314hd3a7d6b_0.conda#533d72030f9dbe5f9a439c7f9673940b
https://conda.anaconda.org/conda-forge/noarch/partd-1.4.2-pyhd8ed1ab_0.conda#0badf9c54e24cecfb0ad2f99d680c163
-https://conda.anaconda.org/conda-forge/noarch/pbr-7.1.1-pyhd8ed1ab_0.conda#51b000c0a6d35e244019f9fc8ab1c9f6
+https://conda.anaconda.org/conda-forge/noarch/pbr-7.1.3-pyhd8ed1ab_0.conda#66415d5d2894f1d10928005a264c2717
https://conda.anaconda.org/conda-forge/linux-64/pyproj-3.8.0-py314hf1f854a_1.conda#f38427e4851bc63bb993b9f038ee97d7
https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.9.0.post0-pyhe01879c_2.conda#5b8d21249ff20967101ffa321cab24e8
-https://conda.anaconda.org/conda-forge/noarch/python-discovery-1.6.1-pyh5ded981_0.conda#ae0feb6a3f297952bd4f4270fe77f530
-https://conda.anaconda.org/conda-forge/noarch/python-gil-3.14.7-h4df99d1_106.conda#b672ecb44198ade19ef249bf99f4b1ff
+https://conda.anaconda.org/conda-forge/noarch/python-discovery-1.6.2-pyh5ded981_0.conda#17e6c8b2fc2d021a217d317fe4f89fc4
+https://conda.anaconda.org/conda-forge/noarch/python-gil-3.14.8-h4df99d1_102.conda#556734568e49dfd7cab8c0341ca1e67b
https://conda.anaconda.org/conda-forge/noarch/typing-extensions-4.16.0-h69aa097_0.conda#c680b5747e8c4c8f23dca0bb7042a8fc
-https://conda.anaconda.org/conda-forge/noarch/vcs_versioning-2.5.0-pyh5ded981_0.conda#137d3106e230ae55fe7b8ee09c55aaf0
+https://conda.anaconda.org/conda-forge/noarch/vcs_versioning-2.6.0-pyh5ded981_0.conda#958f3e46d5025cdd9aa9c2417adceec8
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxtst-1.2.5-h7cc23a3_4.conda#752a5ac9322e0fbbc24146c1ce3ae44e
https://conda.anaconda.org/conda-forge/noarch/_python_abi3_support-1.0-hd8ed1ab_3.conda#3845f3d75991bae0fb90884662f4327c
https://conda.anaconda.org/conda-forge/noarch/asv_runner-0.3.1-pyhd8ed1ab_0.conda#960a0f64467ce624941ad7c67b80a2e2
@@ -245,53 +247,56 @@ https://conda.anaconda.org/conda-forge/noarch/beautifulsoup4-4.15.0-pyha770c72_0
https://conda.anaconda.org/conda-forge/linux-64/cftime-1.6.5-py314h12ceea2_2.conda#6b75e66981d66ebc1709356e7c8fd6a7
https://conda.anaconda.org/conda-forge/linux-64/contourpy-1.4.0-py314h5383ef5_1.conda#2848b95cbf026062b30f3f1f504a2f61
https://conda.anaconda.org/conda-forge/noarch/dask-core-2026.8.0-pyhc364b38_0.conda#2ccdf45d2372e15c7edcb7a72d573e76
-https://conda.anaconda.org/conda-forge/linux-64/hdf5-2.2.0-nompi_hc529623_100.conda#336f7fb9af0c49cc246e2cff0992c3c4
-https://conda.anaconda.org/conda-forge/noarch/importlib-resources-7.1.0-pyhd8ed1ab_0.conda#e3bffa82b874f8b9a2631bddb3869529
+https://conda.anaconda.org/conda-forge/linux-64/hdf5-2.2.0-nompi_hdbb200c_101.conda#66762e6067bf7d52586ff21c78c970db
https://conda.anaconda.org/conda-forge/linux-64/libegl-devel-1.7.0-ha4b6fd6_5.conda#577800fc8c9d8b7d9727246bbfde704e
https://conda.anaconda.org/conda-forge/linux-64/libgdal-core-3.13.3-h31aba09_1.conda#44c6780e3082468d8676aebdcd5068a6
-https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-devel-14.5.0-h23af247_0.conda#82f4063122b5ebc18cc34bb269e941f3
+https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-devel-14.6.0-h23af247_2.conda#3cf459e930e6ea2e57bbf04be7fcb7c6
https://conda.anaconda.org/conda-forge/linux-64/libraqm-0.11.0-h6406941_0.conda#3ac89a48d224409739dbf6200e524373
https://conda.anaconda.org/conda-forge/linux-64/mo_pack-0.3.1-py314h12ceea2_3.conda#06aa38d74d46235f8bdb7743e1921848
+https://conda.anaconda.org/conda-forge/linux-64/numcodecs-0.17.0-py314h12ceea2_1.conda#565a3b1cd503bc0953f6c64fb99cd525
https://conda.anaconda.org/conda-forge/linux-64/pandas-3.0.6-np2py314h9e1d7c2_0.conda#e0f55ab2a87a7f0483a633ac39557098
https://conda.anaconda.org/conda-forge/linux-64/pango-1.58.2-h7bb47b9_1.conda#0b889b1c6b472d215d6332c6b195589c
https://conda.anaconda.org/conda-forge/noarch/pytest-9.0.3-pyhc364b38_1.conda#6a991452eadf2771952f39d43615bb3e
https://conda.anaconda.org/conda-forge/linux-64/pywavelets-1.10.0-py314hfe1a184_1.conda#4eb9dcea543fa478c7ad2f0ef9f4ee13
https://conda.anaconda.org/conda-forge/linux-64/scipy-1.18.1-py314h36f0660_1.conda#2c5a3b238546edf4e3ea3bd9c1b9fa03
https://conda.anaconda.org/conda-forge/noarch/setuptools-scm-10.3.4-pyh5ded981_0.conda#437557778d99cc6f311f3c0ce023ad85
-https://conda.anaconda.org/conda-forge/linux-64/shapely-2.1.2-py314he501186_3.conda#047df7fa5fee2bfc538e185f6268a61b
+https://conda.anaconda.org/conda-forge/linux-64/shapely-2.2.0-np2py314hefd98a1_2.conda#43ae47a48fc57bbeae0651227fda3d83
https://conda.anaconda.org/conda-forge/noarch/snuggs-1.4.7-pyhd8ed1ab_2.conda#9aa358575bbd4be126eaa5e0039f835c
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-apidoc-0.3.0-py_1.tar.bz2#855b087883443abb10f5faf6eef40860
+https://conda.anaconda.org/conda-forge/noarch/towncrier-26.9.0-pyhd8ed1ab_0.conda#d4167f473b71694bbca708347abee99e
https://conda.anaconda.org/conda-forge/linux-64/ukkonen-1.1.0-py314h9891dd4_0.conda#5d3c008e54c7f49592fca9c32896a76f
https://conda.anaconda.org/conda-forge/noarch/urllib3-2.8.0-pyhd8ed1ab_0.conda#407a3d3778570bae8e8fcf554e803e43
-https://conda.anaconda.org/conda-forge/noarch/virtualenv-21.13.0-pyh5ded981_0.conda#e553e26b7aa15efa4640ec55859dfc53
+https://conda.anaconda.org/conda-forge/noarch/virtualenv-21.14.7-pyh5ded981_0.conda#1635534aa129eb9ab6ce5e01929d86ff
https://conda.anaconda.org/conda-forge/linux-64/at-spi2-atk-2.38.0-h0630a04_3.tar.bz2#6b889f174df1e0f816276ae69281af4d
-https://conda.anaconda.org/conda-forge/linux-64/cf-units-3.3.1-np2hc8c11d4_1.conda#49647e9c24724a189e0e334f4f3223e7
+https://conda.anaconda.org/conda-forge/linux-64/cf-units-3.3.1-np2hc8c11d4_2.conda#9b777f085506bff796833be0ae292db1
+https://conda.anaconda.org/conda-forge/linux-64/charset-normalizer-3.5.2-h1114479_0.conda#78d5c6112c9320a1a797935cecfec7c2
https://conda.anaconda.org/conda-forge/noarch/distributed-2026.8.0-pyhc364b38_0.conda#f382c4eed5376ee372a6e67a0f6cd4d9
https://conda.anaconda.org/conda-forge/linux-64/epoxy-1.5.10-hb03c661_2.conda#057083b06ccf1c2778344b6dabace38b
-https://conda.anaconda.org/conda-forge/linux-64/harfbuzz-14.5.0-ha770c72_0.conda#ef253035e60216967c117ab9a279eea2
+https://conda.anaconda.org/conda-forge/linux-64/harfbuzz-14.6.0-ha770c72_2.conda#0709eacc38906e8a237d9ef172af6688
https://conda.anaconda.org/conda-forge/noarch/identify-2.6.20-pyhd8ed1ab_0.conda#be9ea4f030b4b2bc121b2a548b066c0b
https://conda.anaconda.org/conda-forge/noarch/imagehash-4.3.2-pyhd8ed1ab_0.conda#efbc812363856906a80344b496389d2e
https://conda.anaconda.org/conda-forge/linux-64/libnetcdf-4.10.1-nompi_he3e3c8e_201.conda#912aefa694a66897afeb7a2330b17d9f
https://conda.anaconda.org/conda-forge/linux-64/librsvg-2.62.4-ha427ee3_0.conda#f9bda2728debbde4c641bd1af103c623
https://conda.anaconda.org/conda-forge/linux-64/matplotlib-base-3.11.2-py314h9124352_1.conda#55d364956c0d3a01f6e39db3f6f6cc9b
-https://conda.anaconda.org/conda-forge/linux-64/minijinja-2.24.0-py310h300b7de_0.conda#1be3473b8f1e33cb92ce3401fab8e8a6
https://conda.anaconda.org/conda-forge/noarch/pytest-cov-7.1.0-pyhcf101f3_0.conda#67d1790eefa81ed305b89d8e314c7923
https://conda.anaconda.org/conda-forge/noarch/pytest-mock-3.16.0-pyhd8ed1ab_0.conda#b038267c191ebae9d6374caf30cc2cca
https://conda.anaconda.org/conda-forge/noarch/pytest-xdist-3.8.0-pyhd8ed1ab_0.conda#8375cfbda7c57fbceeda18229be10417
+https://conda.anaconda.org/conda-forge/noarch/python-abi3-3.14-hab9af60_3.conda#ad28909b755a16c4f0d3fc8dfbff6fc7
https://conda.anaconda.org/conda-forge/linux-64/python-stratify-0.4.0-py314h89acca1_4.conda#6cbce4f0cd522372910ccc8869964abf
-https://conda.anaconda.org/conda-forge/linux-64/rasterio-1.5.1-py314h523d43f_1.conda#16272efd0bc234eaa6739cc9e446ab24
-https://conda.anaconda.org/conda-forge/noarch/requests-2.34.2-pyhcf101f3_0.conda#4a85203c1d80c1059086ae860836ffb9
-https://conda.anaconda.org/conda-forge/noarch/towncrier-25.8.0-pyhd8ed1ab_0.conda#3e0e8e44292bdac62f7bcbf0450b5cc7
+https://conda.anaconda.org/conda-forge/linux-64/rasterio-1.5.2-py314h2895243_1.conda#2fb192d4cea99fdaf9d446e475e85cef
+https://conda.anaconda.org/conda-forge/noarch/zarr-3.4.1-pyhc364b38_0.conda#052cab2fa318f6ffeefc149addd9e6fb
https://conda.anaconda.org/conda-forge/noarch/adwaita-icon-theme-51.0-unix_0.conda#6f62266fb2bfc35a5071cfa1d6ffdcc9
-https://conda.anaconda.org/conda-forge/linux-64/cartopy-0.26.0-np2py314hbf8ec49_1.conda#3b100bda6d915515389155b12b93fabb
+https://conda.anaconda.org/conda-forge/linux-64/cartopy-0.25.0-np2py314h6477eea_3.conda#1c2bf9dbf9be09fad080f505ffd6314c
https://conda.anaconda.org/conda-forge/linux-64/gtk3-3.24.52-ha5ea40c_0.conda#bcaea22d85999a4f17918acfab877e61
+https://conda.anaconda.org/conda-forge/linux-64/minijinja-3.0.0-py311h1180b86_0.conda#ebba9c2cc80a43f736c13772bee155d8
https://conda.anaconda.org/conda-forge/noarch/nc-time-axis-1.4.1-pyhd8ed1ab_1.conda#9a2be7d0089f5934b550933ca0d9fe85
https://conda.anaconda.org/conda-forge/linux-64/netcdf-fortran-4.6.4-nompi_h52816da_101.conda#2f18bb969e8acd8a1f45293e9782374e
https://conda.anaconda.org/conda-forge/linux-64/netcdf4-1.7.4-nompi_py311hd34d2e2_111.conda#d697ebf773f33945402fe3e1fda13635
-https://conda.anaconda.org/conda-forge/noarch/pre-commit-4.6.2-pyha770c72_0.conda#c36b88db560b4e74f294380613f1a239
-https://conda.anaconda.org/conda-forge/noarch/requests-file-2.1.0-pyhd8ed1ab_1.conda#69be0472744969b97324d5d3b21845a9
+https://conda.anaconda.org/conda-forge/noarch/pre-commit-4.7.0-pyha770c72_0.conda#167169a52ede77767f9fa66b52983a20
+https://conda.anaconda.org/conda-forge/noarch/requests-2.34.2-pyhcf101f3_0.conda#4a85203c1d80c1059086ae860836ffb9
https://conda.anaconda.org/conda-forge/linux-64/esmf-8.9.1-nompi_h39d0119_4.conda#074ca9356f9e916ec51ad6ab47fd55ba
https://conda.anaconda.org/conda-forge/linux-64/graphviz-14.1.2-h8b86629_0.conda#341fc61cfe8efa5c72d24db56c776f44
+https://conda.anaconda.org/conda-forge/noarch/requests-file-2.1.0-pyhd8ed1ab_1.conda#69be0472744969b97324d5d3b21845a9
https://conda.anaconda.org/conda-forge/noarch/esmpy-8.9.1-pyhdfbf58e_0.conda#0155d4a55415cda7f7f09bc0b7760f51
https://conda.anaconda.org/conda-forge/noarch/pydata-sphinx-theme-0.19.0-pyhcf101f3_0.conda#87921f66a4dc56ce92e4ff13be5f63dc
https://conda.anaconda.org/conda-forge/noarch/sphinx-copybutton-0.5.2-pyhd8ed1ab_1.conda#bf22cb9c439572760316ce0748af3713
diff --git a/requirements/py312.yml b/requirements/py312.yml
index 311a91f39d..d37a734226 100644
--- a/requirements/py312.yml
+++ b/requirements/py312.yml
@@ -37,6 +37,7 @@ dependencies:
- python-stratify
- rasterio
- affine
+ - zarr >=3.0.8
# Test dependencies.
- asv_runner
diff --git a/requirements/py313.yml b/requirements/py313.yml
index 4f27ae73e6..dfb47e25b0 100644
--- a/requirements/py313.yml
+++ b/requirements/py313.yml
@@ -37,6 +37,7 @@ dependencies:
- python-stratify
- rasterio
- affine
+ - zarr >=3.0.8
# Test dependencies.
- asv_runner
diff --git a/requirements/py314.yml b/requirements/py314.yml
index 79965dcdde..11187c6165 100644
--- a/requirements/py314.yml
+++ b/requirements/py314.yml
@@ -37,6 +37,7 @@ dependencies:
- python-stratify
- rasterio
- affine
+ - zarr >=3.0.8
# Test dependencies.
- asv_runner