Conversation
PyTables stores the Exposures metadata dict as a protocol 0 pickle. Files written by CLIMADA < 6.1 have an affine.Affine transform in that dict. With affine <= 2.4, Affine was a subclass of tuple and the pickle restores it through copyreg._reconstructor(Affine, tuple, state). Since affine 3.0, Affine is no longer a tuple, so unpickling raises a TypeError. PyTables swallows the error and returns the raw pickle bytes, and from_hdf5 fails with "'numpy.bytes_' object has no attribute 'get'". When the metadata comes back as bytes, unpickle it again with an Unpickler that builds such Affine objects from their first six values and hands every other _reconstructor call to the standard one. Fixes CLIMADA-project#1311
rmz-oz
requested review from
chahank,
emanuel-schmid and
peanutfun
as code owners
September 28, 2026 04:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes proposed in this PR:
Exposures.from_hdf5can read files written by CLIMADA < 6.1 again when affine 3.x is installed. If PyTables returns the metadata as raw pickle bytes, the bytes are unpickled a second time with a smallpickle.Unpicklersubclass that rebuilds the oldAffineobjects.TestIO: one calls the helper on a legacy metadata pickle, and one writes a minimal hdf5 file with that metadata and reads it back withExposures.from_hdf5.This PR fixes #1311
What goes wrong: PyTables pickles the metadata dict with protocol 0. In older files the dict contains the raster
metawith anaffine.Affinetransform. Up to affine 2.4,Affinewas a tuple subclass, so the pickle restores it withcopyreg._reconstructor(Affine, tuple, state). In affine 3.0Affineis an attrs class, and this call raisesTypeError: tuple.__new__(Affine): Affine is not a subtype of tuple. PyTables catches the exception and returns the pickle string as it is, andmetadata.get(...)then fails. The new unpickler only replacescopyreg._reconstructor. When the class is anAffineand the base istuple, it buildsAffine(*state[:6]). Every other object, such as the pyproj CRS in the same dict, still goes through the standard reconstructor. When PyTables returns the metadata without errors (all new files, and all files with affine 2.4), nothing changes.I checked this against a real file from the data API,
LitPop_150arcsec_ABW.hdf5(v3), using the same script and the same venv. Only the affine version and the code change:Tests (
pytest climada/entity/exposures/test/test_base.py):test_read_legacy_hdf5_passfails on develop with the error above.Formatting is done with the pre-commit versions (black 24.4.2, isort 5.13.2).
pylint --rcfile=.pylintrconbase.pygives the same 27 messages as before.I did not touch the
affine<3.0pin inrequirements/env_climada.yml. With this change the pin should no longer be needed for reading files, but you know better if anything else depends on it.Note on my local setup: I installed the dependencies with pip, not conda, so GDAL was missing. I used a local stub for
osgeo(onlylitpop/nightlight.pyimports it at module level). The stub is not part of the PR.PR Author Checklist
develop)PR Reviewer Checklist