Skip to content

utils: immutable source downloads are shared through SCRATCH_ROOT - #38

Open
danohu wants to merge 2 commits into
mainfrom
source-cache
Open

danohu wants to merge 2 commits into
mainfrom
source-cache

Conversation

@danohu

@danohu danohu commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Large source files are downloaded again by every process that needs them. Upstream's zarr cache stores each dataset's ingested output, and an archive is fetched only once per process, but a second machine ingesting other tiles of the same dataset still goes back to the publisher. The GAcED30 archives and GPW livestock rasters are large, and Zenodo throttles.

This shares those downloads through SCRATCH_ROOT, which every machine already shares for the zarr cache. There's no new configuration.

Behaviour

  • utils.save_remote_url_to_local_path takes immutable: bool = False. When it is true:
    • if {SCRATCH_ROOT}/sources/{host}{path} exists (the URL's percent-escapes decoded), the file is copied from there instead of asking the publisher;
    • otherwise it is downloaded as before, and a copy is left there for the next process.
  • Nothing ever refreshes the copy, so only URLs whose content never changes are marked: the fixed Zenodo record downloads in gpw_livestock, liao_gaced30, descals_oil_palm and ipcc_climate_zones. All other downloads are unchanged.
  • The copy is found by the URL's host and path alone, so an immutable download with params or a query string raises ValueError rather than risk two different requests sharing one copy. ipcc_climate_zones therefore moves from .../records/7303808/files/...tif?download=1 to the /api/records/.../content form the other Zenodo datasets use; both serve the same file.
  • storage.get_file is the read counterpart of put_file.

Testing

  • jdluc/__tests__/source_cache_test.py stubs the publisher and checks that an immutable download is fetched once and then reused, that an unmarked download is fetched every time and leaves nothing in SCRATCH_ROOT, and that an immutable download with parameters is refused before any request.
  • uv run pytest jdluc and uv run pre-commit run --all-files pass.

Replaces #23, which GitHub closed because its head repo is no longer in this repo's fork network.

A download marked immutable, with a copy under {SCRATCH_ROOT}/sources/{host}{path}, is
copied from there rather than fetched; otherwise it is fetched as before and a copy left
there, so other processes and machines skip the publisher. Only the Zenodo records behind
gpw_livestock and liao_gaced30 are marked: nothing refreshes the copy.
…ds too

Both are fixed Zenodo records. ipcc_climate_zones moves to the /api/.../content form of
its URL, as the other Zenodo datasets use, since an immutable download now refuses
parameters and query strings: the shared copy is found by host and path alone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant