Repository navigation
Add download_sample for a dataset's public sample - #154
Conversation
The platform serves a dataset's sample from POST /datasets/{id}/sample-download,
which the SDK had no call for. download_sample mirrors download_dataset: same
arguments, resolves a slug to the dataset id, skips the request when the sample
is already on disk, and reports its own download source.
Two differences follow from the endpoint. The sample's filename is only known
from the response, so samples are kept in samples/<dataset id>/ under the
download directory, and a finished file there means no request is made. The
response carries no checksum, so an interrupted sample download starts again
rather than resuming.
The overwrite, resume, download and rename steps move into _run_download_plan,
and response parsing into _build_download_plan, so both downloads share them.
A refused sample request, such as one for a dataset whose terms have not been agreed to, left an empty samples/<dataset id>/ directory behind.
Thanks @moz-tello-wharton for adding this functionality! I would slightly adjust the implementation of some logic. The download plan from the API does return the filename so we should use that instead of the heuristic I ll push the new implementation, but thank you for doing all the ground work first! |
The platform serves a dataset's sample from POST /datasets/{id}/sample-download, which the SDK had no call for. download_sample mirrors download_dataset: same arguments, resolves a slug to the dataset id, skips the request when the sample is already on disk, and reports its own download source.
Two differences follow from the endpoint. The sample's filename is only known from the response, so samples are kept in samples// under the download directory, and a finished file there means no request is made. The response carries no checksum, so an interrupted sample download starts again rather than resuming.
The overwrite, resume, download and rename steps move into _run_download_plan, and response parsing into _build_download_plan, so both downloads share them.