Skip to content

Read legacy Exposures hdf5 metadata with affine 3 - #1328

Open
rmz-oz wants to merge 1 commit into
CLIMADA-project:developfrom
rmz-oz:fix/1311-affine3-legacy-hdf5
Open

rmz-oz wants to merge 1 commit into
CLIMADA-project:developfrom
rmz-oz:fix/1311-affine3-legacy-hdf5

Conversation

@rmz-oz

@rmz-oz rmz-oz commented Sep 28, 2026

Copy link
Copy Markdown

Changes proposed in this PR:

  • Exposures.from_hdf5 can read files written by CLIMADA < 6.1 again when affine 3.x is installed. If PyTables returns the metadata as raw pickle bytes, the bytes are unpickled a second time with a small pickle.Unpickler subclass that rebuilds the old Affine objects.
  • Two tests in TestIO: one calls the helper on a legacy metadata pickle, and one writes a minimal hdf5 file with that metadata and reads it back with Exposures.from_hdf5.

This PR fixes #1311

What goes wrong: PyTables pickles the metadata dict with protocol 0. In older files the dict contains the raster meta with an affine.Affine transform. Up to affine 2.4, Affine was a tuple subclass, so the pickle restores it with copyreg._reconstructor(Affine, tuple, state). In affine 3.0 Affine is an attrs class, and this call raises TypeError: tuple.__new__(Affine): Affine is not a subtype of tuple. PyTables catches the exception and returns the pickle string as it is, and metadata.get(...) then fails. The new unpickler only replaces copyreg._reconstructor. When the class is an Affine and the base is tuple, it builds Affine(*state[:6]). Every other object, such as the pyproj CRS in the same dict, still goes through the standard reconstructor. When PyTables returns the metadata without errors (all new files, and all files with affine 2.4), nothing changes.

I checked this against a real file from the data API, LitPop_150arcsec_ABW.hdf5 (v3), using the same script and the same venv. Only the affine version and the code change:

affine 3.0.1, develop:  AttributeError: 'numpy.bytes_' object has no attribute 'get'
affine 3.0.1, this PR:  ok: LitPop Exposure for ['ABW'] at 150 as, year: 2018, ... 2018 USD <WGS 84> 8 points
affine 2.4.0, develop:  same result as the line above

Tests (pytest climada/entity/exposures/test/test_base.py):

  • affine 3.0.1: develop 29 passed; this PR 31 passed. test_read_legacy_hdf5_pass fails on develop with the error above.
  • affine 2.4.0: develop 29 passed; this PR 31 passed.

Formatting is done with the pre-commit versions (black 24.4.2, isort 5.13.2). pylint --rcfile=.pylintrc on base.py gives the same 27 messages as before.

I did not touch the affine<3.0 pin in requirements/env_climada.yml. With this change the pin should no longer be needed for reading files, but you know better if anything else depends on it.

Note on my local setup: I installed the dependencies with pip, not conda, so GDAL was missing. I used a local stub for osgeo (only litpop/nightlight.py imports it at module level). The stub is not part of the PR.

PR Author Checklist

PR Reviewer Checklist

PyTables stores the Exposures metadata dict as a protocol 0 pickle.
Files written by CLIMADA < 6.1 have an affine.Affine transform in that
dict. With affine <= 2.4, Affine was a subclass of tuple and the pickle
restores it through copyreg._reconstructor(Affine, tuple, state). Since
affine 3.0, Affine is no longer a tuple, so unpickling raises a
TypeError. PyTables swallows the error and returns the raw pickle bytes,
and from_hdf5 fails with "'numpy.bytes_' object has no attribute 'get'".

When the metadata comes back as bytes, unpickle it again with an
Unpickler that builds such Affine objects from their first six values
and hands every other _reconstructor call to the standard one.

Fixes CLIMADA-project#1311
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant