Repository navigation
Conversation
A download marked immutable, with a copy under {SCRATCH_ROOT}/sources/{host}{path}, is
copied from there rather than fetched; otherwise it is fetched as before and a copy left
there, so other processes and machines skip the publisher. Only the Zenodo records behind
gpw_livestock and liao_gaced30 are marked: nothing refreshes the copy.
…ds too Both are fixed Zenodo records. ipcc_climate_zones moves to the /api/.../content form of its URL, as the other Zenodo datasets use, since an immutable download now refuses parameters and query strings: the shared copy is found by host and path alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large source files are downloaded again by every process that needs them. Upstream's zarr cache stores each dataset's ingested output, and an archive is fetched only once per process, but a second machine ingesting other tiles of the same dataset still goes back to the publisher. The GAcED30 archives and GPW livestock rasters are large, and Zenodo throttles.
This shares those downloads through
SCRATCH_ROOT, which every machine already shares for the zarr cache. There's no new configuration.Behaviour
utils.save_remote_url_to_local_pathtakesimmutable: bool = False. When it is true:{SCRATCH_ROOT}/sources/{host}{path}exists (the URL's percent-escapes decoded), the file is copied from there instead of asking the publisher;gpw_livestock,liao_gaced30,descals_oil_palmandipcc_climate_zones. All other downloads are unchanged.paramsor a query string raisesValueErrorrather than risk two different requests sharing one copy.ipcc_climate_zonestherefore moves from.../records/7303808/files/...tif?download=1to the/api/records/.../contentform the other Zenodo datasets use; both serve the same file.storage.get_fileis the read counterpart ofput_file.Testing
jdluc/__tests__/source_cache_test.pystubs the publisher and checks that an immutable download is fetched once and then reused, that an unmarked download is fetched every time and leaves nothing inSCRATCH_ROOT, and that an immutable download with parameters is refused before any request.uv run pytest jdlucanduv run pre-commit run --all-filespass.Replaces #23, which GitHub closed because its head repo is no longer in this repo's fork network.