Skip to content

Add download_sample for a dataset's public sample - #154

Merged
Kostis-S-Z merged 9 commits into
mainfrom
download-sample
Oct 19, 2026
Merged

Kostis-S-Z merged 9 commits into
mainfrom
download-sample

Conversation

@moz-tello-wharton

Copy link
Copy Markdown
Contributor

The platform serves a dataset's sample from POST /datasets/{id}/sample-download, which the SDK had no call for. download_sample mirrors download_dataset: same arguments, resolves a slug to the dataset id, skips the request when the sample is already on disk, and reports its own download source.

Two differences follow from the endpoint. The sample's filename is only known from the response, so samples are kept in samples// under the download directory, and a finished file there means no request is made. The response carries no checksum, so an interrupted sample download starts again rather than resuming.

The overwrite, resume, download and rename steps move into _run_download_plan, and response parsing into _build_download_plan, so both downloads share them.

The platform serves a dataset's sample from POST /datasets/{id}/sample-download,
which the SDK had no call for. download_sample mirrors download_dataset: same
arguments, resolves a slug to the dataset id, skips the request when the sample
is already on disk, and reports its own download source.

Two differences follow from the endpoint. The sample's filename is only known
from the response, so samples are kept in samples/<dataset id>/ under the
download directory, and a finished file there means no request is made. The
response carries no checksum, so an interrupted sample download starts again
rather than resuming.

The overwrite, resume, download and rename steps move into _run_download_plan,
and response parsing into _build_download_plan, so both downloads share them.
A refused sample request, such as one for a dataset whose terms have not been agreed to, left an empty samples/<dataset id>/ directory behind.
@Kostis-S-Z

Copy link
Copy Markdown
Contributor

The platform serves a dataset's sample from POST /datasets/{id}/sample-download, which the SDK had no call for. download_sample mirrors download_dataset: same arguments, resolves a slug to the dataset id, skips the request when the sample is already on disk, and reports its own download source.

Two differences follow from the endpoint. The sample's filename is only known from the response, so samples are kept in samples// under the download directory, and a finished file there means no request is made. The response carries no checksum, so an interrupted sample download starts again rather than resuming.

The overwrite, resume, download and rename steps move into _run_download_plan, and response parsing into _build_download_plan, so both downloads share them.

Thanks @moz-tello-wharton for adding this functionality!

I would slightly adjust the implementation of some logic. The download plan from the API does return the filename so we should use that instead of the heuristic _find_downloaded_sample of treating any file under <download_dir>/samples/<id> as the sample file which seems a bit sketchy to me. Because that means that if the user ever puts any other files in that directory the SDK will not know which is the sample file or not. So 1) the cleanup function with overwrite=True will delete any user data in that directory (line ~152-160) 2) it might skip new downloads mistakenly.

I ll push the new implementation, but thank you for doing all the ground work first!

@Kostis-S-Z
Kostis-S-Z merged commit 6d2e07f into main Oct 19, 2026
3 checks passed
@Kostis-S-Z
Kostis-S-Z deleted the download-sample branch October 19, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants