Skip to content

Fix xarray compression handling for SPDF epoch-like time variables - #332

Merged
bryan-harter merged 1 commit into
lasp:mainfrom
davidt0x:fix/istp-time-variable-compression
May 21, 2026
Merged

bryan-harter merged 1 commit into
lasp:mainfrom
davidt0x:fix/istp-time-variable-compression

Conversation

@davidt0x

Copy link
Copy Markdown
Contributor

Summary

This PR broadens cdflib's automatic compression-skip logic for time variables so it better matches the SPDF Java ISTP checker.

Previously, xarray_to_cdf() only disabled compression for variables named epoch or epoch_N. That was too narrow for ISTP validation. In particular, time-typed support variables such as IDEX on_off_times were still being written with gzip compression even though the SPDF validator treats them as epoch-like variables that should not be compressed.

What Changed

The xarray writer now treats a variable as SPDF epoch-like for compression purposes when it:

  • has a time CDF datatype:
    • CDF_EPOCH
    • CDF_EPOCH16
    • CDF_TIME_TT2000
  • is not range_epoch
  • and either:
    • is scalar in final CDF terms and record-varying, or
    • is an xarray time coordinate over its own dimension

The previous epoch / epoch_N behavior is preserved, so existing ISTP epoch variables continue to be written without compression.

This change is intentionally scoped to the xarray writer’s compression policy. It does not change the lower-level CDF writer behavior more broadly.

Why

The SPDF Java checker does not key this rule only off the variable name epoch. Its epoch compliance logic identifies the file’s ISTP epoch variable based on time datatype and variable shape/variance, then enforces that it not be compressed.

This PR brings cdflib closer to that behavior so local CDF generation aligns better with SPDF validation.

Testing

Added regression coverage to verify that:

  • epoch is still written uncompressed by default
  • epoch_1 is still written uncompressed by default
  • an on_off_times-style TT2000 variable is written uncompressed
  • range_epoch remains compressible
  • compression_skip_vars still works as expected

Validated locally with:

cd /mnt/c/Dropbox/princeton/imap/cdflib
/home/david/.virtualenvs/imap/bin/python -m pytest -q tests/test_xarray_reader_writer.py -k 'compression'

Expand the xarray writer's automatic compression skip logic so it better
matches the SPDF Java checker’s notion of an ISTP epoch variable.

Previously, xarray_to_cdf only disabled compression for variables named
`epoch` or `epoch_N`. That was too narrow for ISTP validation. In
particular, time-typed support variables such as IDEX `on_off_times`
were still being written with gzip compression even though the SPDF
validator treats them as epoch-like time variables that should not be
compressed.

Update the compression predicate to treat a variable as SPDF epoch-like
when it:
- has a CDF time datatype (`CDF_EPOCH`, `CDF_EPOCH16`, or
  `CDF_TIME_TT2000`)
- is not `range_epoch`
- and either:
  - is scalar in final CDF terms and record-varying, or
  - is an xarray time coordinate over its own dimension

Keep the previous `epoch` / `epoch_N` behavior intact so existing ISTP
epoch variables continue to be written without compression.

Also add focused regression coverage to verify that:
- `epoch` and `epoch_1` are still written uncompressed by default
- an `on_off_times`-style TT2000 variable is now written uncompressed
- `range_epoch` remains compressible
- explicit `compression_skip_vars` still works as expected

Validated locally with:
- `/home/david/.virtualenvs/imap/bin/python -m pytest -q tests/test_xarray_reader_writer.py -k 'compression'`
@bryan-harter
bryan-harter merged commit 3a43f17 into lasp:main May 21, 2026
12 of 13 checks passed
@bryan-harter

Copy link
Copy Markdown
Member

Thanks for that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants