AGENTS.md

AGENTS.md

imap-processing is the science data processing pipeline for NASA's IMAP mission, run by the Science Operations Center (SOC) at LASP. It turns raw CCSDS telemetry packets into ISTP-compliant CDF science products, one instrument and one data level at a time.

Big picture

• A high level overview of the mission lives in docs/source/mission-overview.rst. This document is not needed for software development, but provides a helpful reference for connecting together instrument-spanning questions or ideas.
• Every product is produced by a single invocation of imap_cli for one (instrument, data-level, descriptor, start-date) combination. See imap_processing/cli.py.
• imap_processing/cli.py is the only entry point. It defines an abstract ProcessInstrument base class and one subclass per instrument (Codice, Glows, Hi, Hit, Idex, Lo, Mag, Spacecraft, Swapi, Swe, Ultra). Each subclass implements do_processing(dependencies) -> list[xr.Dataset]; the base class handles downloading dependencies and writing the returned datasets to CDF. Adding a new data level means updating both PROCESSING_LEVELS in imap_processing/init.py and the relevant do_processing branch.
• Instrument subpackages live at imap_processing/<instrument>/. Two layouts are in use — flat modules (hi/hi_l1a.py, codice/codice_l1a.py) and level subpackages (mag/l1a/mag_l1a.py, swe/l1b/swe_l1b.py). Follow whichever the instrument already uses; do not restructure.
• Data flows as xarray.Dataset objects throughout. Processing functions take dependencies (an imap_data_access.ProcessingInputCollection or file paths) and return xr.Dataset / list[xr.Dataset]. They should not write files themselves — the CLI does that.

Algorithms

• Algorithm behavior is documented per-instrument in docs/source/algorithm-code-documentation/. Check there before changing science logic.

  Some instruments have a full working reference distilled from their algorithm document — a product inventory, the algorithms with equations, and an implementation-status page listing deviations and gaps. Read the instrument's page set before proposing or estimating work on it.

  • CoDICE: docs/source/algorithm-code-documentation/codice/index.rst
  • GLOWS: docs/source/algorithm-code-documentation/glows/index.rst
  • HIT: docs/source/algorithm-code-documentation/hit/index.rst
  • IDEX: docs/source/algorithm-code-documentation/idex/index.rst
  • MAG: docs/source/algorithm-code-documentation/mag/index.rst
  • SWAPI: docs/source/algorithm-code-documentation/swapi/index.rst
  • SWE: docs/source/algorithm-code-documentation/swe/index.rst

  Many of these pages are unreviewed AI-generated drafts. They contain the line .. include:: /algorithm-code-documentation/_ai_generated_notice.inc just below the page title (this renders as a warning banner). Treat a page with that line as a strong starting point, not as an authority:

  • When a marked page disagrees with the code, do not assume the code is wrong. The algorithm document or CMAD may be out of date, or the deviation may be intentional. Point out the discrepancy and let a human decide; don't "fix" code to match the page.
  • When you cite a marked page in a PR, review, or answer, say that it is unverified.
  • Only a person who has verified the page against the code or with the instrument team removes the include line. Never remove it yourself unless that person asks you to. When you edit a marked page, keep the marker.
  • As the pages are reviewed by humans to correct discrepancies, they will be added into sphinx docs. Currently, the sphinx documentation only points to autogenerated docs from the code.
  • The ENAs (Lo, Hi, and Ultra) will be included in more detail at a later date to this documentation, when their algorithm documents are finalized.

Use the shared infrastructure, don't reinvent it

Need	Use
Decommutate CCSDS packets	packet_file_to_datasets() in imap_processing/utils.py
Raw DN → engineering units	convert_raw_to_eu() in imap_processing/utils.py
CDF read/write	load_cdf() / write_cdf() in imap_processing/cdf/utils.py
CDF global/variable attributes	ImapCdfAttributes in imap_processing/cdf/imap_cdf_manager.py
Time conversion (MET / ET / TT-J2000 ns)	imap_processing/spice/time.py — never hand-roll epoch math
Spin, repointing, pointing frames, geometry	imap_processing/spice/
Data quality bitflags	imap_processing/quality_flags.py
ENA sky maps / pointing sets	imap_processing/ena_maps/ena_maps.py
Combining time-varying ancillary files	imap_processing/ancillary/ancillary_dataset_combiner.py

Packet definitions (XTCE)

Packet structures are XTCE XML files in imap_processing/<instrument>/packet_definitions/. They are generated as a first pass from the instrument team's telemetry spreadsheet via imap_xtce <spreadsheet.xlsx> --output <file.xml> (imap_processing/ccsds/excel_to_xtce.py) and then hand-refined. Decommutation is always done through packet_file_to_datasets(), which returns dict[apid, xr.Dataset]; instrument code dispatches on APID enums defined in the instrument's constants.py.

CDF products (required for every new data product)

1. Add/extend the YAML attribute configs in imap_processing/cdf/config/: imap_<instrument>_global_cdf_attrs.yaml and imap_<instrument>_<level>_variable_attrs.yaml.
2. Load them with ImapCdfAttributes.add_instrument_global_attrs() and .add_instrument_variable_attrs() and apply them to dataset.attrs and each variable's .attrs.
3. Set Logical_source = "imap_<instrument>_<level>_<descriptor>" and Data_version. write_cdf() derives the output filename from these, so a typo here produces a wrongly-named file rather than an error.
4. Missing ISTP attributes surface as cdflib ISTPError at write time — test by actually calling write_cdf() in a test, as imap_processing/tests/hi/test_hi_l1a.py does.

Environment and commands

Poetry 2.x with the dynamic-versioning plugin. Full setup instructions, including the required system libraries (libnetcdf, libhdf5, and openblas/gfortran on 3.14+), are in docs/source/development/getting-started.rst.

poetry install --extras "test,dev,doc"        # or --all-extras
pre-commit install                            # do this once

poetry run pytest -vvv -n auto -m "not external_kernel and not external_test_data"
poetry run pytest imap_processing/tests/hi/test_hi_l1a.py::test_sci_de_decom -vvv

pre-commit run --all-files                    # ruff check+format, mypy, numpydoc, codespell
make -C docs html SPHINXOPTS="-W --keep-going"

• Default to the -m "not external_kernel and not external_test_data" selection. Those markers trigger multi-hundred-MB downloads of SPICE kernels from NAIF and test data from the SDC; assume the sandbox has no network access unless told otherwise.
• Tests live in imap_processing/tests/, mirroring the instrument packages. Global fixtures (SPICE kernels, metakernels, fake spin/repoint data, temp DATA_DIR) are in imap_processing/tests/conftest.py; instrument-specific fixtures in each subdirectory's conftest.py. Prefer existing fixtures over new ad-hoc setup.
• Codecov requires ~90% patch coverage, so new code needs tests.

Conventions

• Ruff (ruff check / ruff format) and mypy (strict = true, tests excluded) are enforced in pre-commit and CI. Configuration lives in pyproject.toml.
• numpydoc docstrings are validated (numpydoc-validation hook) on all non-test code: summary, Parameters, Returns with types are effectively mandatory. Tests are exempt from docstring rules (D is ignored under */tests/*).
• Modern type hints everywhere: str | Path, list[xr.Dataset], npt.NDArray. Target version is py310.
• logger = logging.getLogger(__name__) — never print().
• Style details (naming, imports, PR checklist, review standards) are documented in docs/source/development/git-workflow-and-style-guide/index.rst — follow it rather than inventing conventions.

Gotchas

• Pre-commit blocks direct commits to main and dev, and files over 1000 KB. Work on a feature branch; test data is downloaded at runtime, never committed (no git-lfs).
• CI runs the suite on Linux/macOS/Windows × Python 3.10–3.14, so avoid platform-specific paths and version-specific syntax.