Explore integration with Icechunk data engine

[aufdenkampe]

My vision for this package is that would work seamlessly in cooperation with a local and/or remote high performance data catalog and store (i.e. data engine). Presently, the Icechunk cloud-native transactional tensor storage engine is the most promising option, as it was recently open-sourced by EarthMover as the source code behind their ArrayLake services.

An ideal work flow would be to:

• User requests a dataset from a well-known data repository for a specific area of interest.
  • These well-known data repos will be cataloged here in a yaml file, and optionally referenced with Kerchunk or VirtualiZarr.
• This package first checks if the specific dataset has already been fetched and saved to a local Icechunk instance.
• If not, it fetches the specific dataset from the source repository, saving it locally in it's native format.
• If the user expects to reuse the data, they can choose to convert the dataset into a cloud-optimized, analysis-ready (ARCO) zarr3 dataset within Icechunk.

[aufdenkampe]

UPDATE: After a little reading, it appears that Icechunk's Virtual Datasets are superior to Kerchunk references or VirtualiZarr datasets if a dataset will get updated, because Icechunk has "transactional updates, version controlled history, and faster access speeds."

For VirtualiZarr, "you should not change or add to any of the files comprising the store once created." However, "VirtualiZarr allows you to ingest data as virtual references and write those references into an Icechunk Store." So you can get started in VirtualiZarr then hand it over to Icechunk before making updates.

References:

• https://virtualizarr.readthedocs.io/en/latest/faq.html#is-this-compatible-with-icechunk
• https://icechunk.io/en/latest/icechunk-python/virtual/#creating-a-virtual-dataset-with-virtualizarr

[aufdenkampe]

Icechunk presently only supports HDF5, netcdf4, and netcdf3 files for use in virtual references with VirtualiZarr.
https://icechunk.io/en/latest/icechunk-python/virtual/#virtual-reference-file-format-support

VirtualiZarr leverages Kerchunk, as an optional dependency, to create references to COG, FITS, and HDF4 file types, although COG and GRIB support are in the works.

Kerchunk has a wider array of supported file types, including GRIB2, Zarr2, etc.

All this might improve very soon, as the main issue was with the following, which just got merged!

• zarr-python v3 compatibility fsspec/kerchunk#516