fix(hub): link ranged weight downloads into the HF snapshot dir - #56
Open
matts-path wants to merge 1 commit into
Open
fix(hub): link ranged weight downloads into the HF snapshot dir#56matts-path wants to merge 1 commit into
matts-path wants to merge 1 commit into
Conversation
HfFetcher::get_file's ranged/resumable branch (used for any file >=32MiB, i.e. the weight shards) returns a raw blobs/<hash> path and never creates the snapshots/<rev>/<filename> symlink that hf-hub's own single-stream download creates for small files. download_source discarded that return value and assumed every wanted file landed beside config.json, so a convert-on-pull of a raw HF repo (org/model, not the basecompute catalog) reliably downloaded the weight blob in full but failed conversion with "No such file or directory" opening it from the snapshot dir — and retrying couldn't self-heal, since the "already downloaded" check only compares blob size. download_source now links each fetched file into the snapshot dir itself when the fetcher didn't already place it there, which also makes a retry over a partially-broken cache self-healing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
matts-path
marked this pull request as ready for review
August 20, 2026 19:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #54.
Problem
basert pull <org>/<model>(raw HF convert-on-pull, not thebasecompute/<name>catalog path) reliably downloads weight shards in full but fails conversion withNo such file or directoryopening a.safetensorsfile — and a retry can't self-heal.Root cause
HfFetcher::get_file's ranged/resumable branch (base-convert/crates/base-hub/src/fetch.rs), used for any file ≥RANGED_MIN_BYTES(32MiB — i.e. the weight shards), returns a rawblobs/<hash>path. Unlike the small-file branch, which delegates to hf-hub's own single-stream download and gets itssnapshots/<rev>/<filename>symlink for free, the ranged branch never creates one.download_source(base-convert/crates/base-convert/src/hub.rs) trusted this: it anchors the snapshot dir onconfig.json's parent, then callsfetcher.get_filefor every other wanted file and discards the returned path, assuming it landed besideconfig.json. For a weight shard that assumption is false — the blob sits inblobs/, never linked into the snapshot dirdownload_sourcejust promised was complete. Conversion then reads from that (missing) snapshot path and fails.Retrying doesn't help either: the "already downloaded" check for ranged files only compares blob byte-length, so
pull(with or without--force) just reconfirms the blob is present and proceeds straight to the same failure.Full repro details, byte-level verification, and file:line references are in #54.
Fix
download_sourcenow checks, for every fetched file, whether it already resolves where the snapshot dir expects it — and if not (the ranged-download case), links it in itself (link_into_snapshot). This is a no-op for anything that was already placed correctly (small files, and every existing test double), and it also makes a partially-broken cache self-healing on the nextpull, since it runs unconditionally rather than depending on a fresh download happening.Testing
cargo build/cargo test/cargo clippyclean forbase-convertandbase-hub(45 + 44 tests passing).download_source_links_ranged_weight_into_snapshot: a newRangedFetchertest double that reproduces the actual split (weight file returns a raw blob path; metadata files already symlinked) — the existingStagedFetcherdouble pre-stages symlinks for every file and so never exercised this path, which is how the bug shipped uncaught.Qwen/Qwen3-0.6Bcache entry and re-ranbasert pull Qwen/Qwen3-0.6Bfrom scratch with the patched binary — succeeded in one shot with no manual intervention, where it reliably failed before this change.Opening as a draft for review — happy to adjust naming/placement of
link_into_snapshotor add coverage elsewhere if there's a preferred spot.