Repository navigation
Add extract to download_dataset and download_sample - #155
Merged
Merged
Conversation
The platform serves a dataset's sample from POST /datasets/{id}/sample-download,
which the SDK had no call for. download_sample mirrors download_dataset: same
arguments, resolves a slug to the dataset id, skips the request when the sample
is already on disk, and reports its own download source.
Two differences follow from the endpoint. The sample's filename is only known
from the response, so samples are kept in samples/<dataset id>/ under the
download directory, and a finished file there means no request is made. The
response carries no checksum, so an interrupted sample download starts again
rather than resuming.
The overwrite, resume, download and rename steps move into _run_download_plan,
and response parsing into _build_download_plan, so both downloads share them.
A refused sample request, such as one for a dataset whose terms have not been agreed to, left an empty samples/<dataset id>/ directory behind.
Extraction was only reachable through load_dataset, which needs the dataset in the schema registry. With extract=True, download_dataset and download_sample also unpack the archive into a directory beside it and return that directory. It is the directory load_dataset extracts to, so the two never unpack the same archive twice. An existing directory is reused unless overwrite_existing=True, which re-downloads and so re-extracts. _extract_archive now extracts under a temporary name and renames once complete. Before, it created the target directory first, so an interrupted extraction left a directory that the next call took as finished and skipped. load_dataset gets the fix too. Replacing a sample also removes the directory the old one was extracted into.
…ive-python into download-extract # Conflicts: # src/datacollective/datasets.py # src/datacollective/download.py # tests/test_datasets.py # tests/test_download.py
Kostis-S-Z
approved these changes
Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extraction was only reachable through load_dataset, which needs the dataset in the schema registry. With extract=True, download_dataset and download_sample also unpack the archive into a directory beside it and return that directory. It is the directory load_dataset extracts to, so the two never unpack the same archive twice. An existing directory is reused unless overwrite_existing=True, which re-downloads and so re-extracts.
_extract_archive now extracts under a temporary name and renames once complete. Before, it created the target directory first, so an interrupted extraction left a directory that the next call took as finished and skipped. load_dataset gets the fix too.
Replacing a sample also removes the directory the old one was extracted into.