sync: open-core source for 0.2.3 - #53
Merged
Merged
Conversation
Generated by tools/sync_public.py from the internal tree at efda25f9. Only allow-listed paths are included; tuned quant profiles and internal tooling are excluded by rule.
Hand-written to accompany the generated sync; none of these paths are in the allow-list, so `tools/sync_public.py` cannot carry them. - Rust 1.85+ in both places that state the floor. hf-hub 1.0 is edition 2024, so 1.80 is now a hard build failure and these two pages are what a new user follows. - The five `BASERT_HF_*` download knobs, in the table that already documents the rest of the environment. - A "How downloads behave" section in the models guide, because the transport is now user-visible. The resume paragraph says exactly what was measured, which is not the tidier claim: a ranged download resumes from its chunk map, a Xet download restarts from the beginning. Killed at 2246 MiB, a Xet pull came back 12s into its next attempt with 328 MiB — a restart, and hf-hub 1.0 exposes no way to hand hf-xet a chunk cache that would change it. Readers planning an overnight pull on a bad link need that difference, and `FORCE_RANGED` is the lever it implies.
prabod
marked this pull request as draft
August 19, 2026 01:00
Contributor
Author
|
Holding as draft: the internal 0.2.3 tag is on hold while |
Regenerated from the internal tree at e8a1fa11, which carries two changes
that landed after this branch was first written:
* downloads are resumable by DEFAULT — the parallel ranged path is now
what every large file takes, and Xet is opt-in behind BASERT_HF_XET,
because Xet cannot resume;
* the catalog is generated by scanning the published org rather than
hand-written: 69 rows -> 105, adding every Whisper bundle and
Qwen3.8-27B, and correcting eight rows whose size/sha256 no longer
matched the Hub (those four models could not be pulled at all).
New file: base-hub/src/scan.rs, plus the `catalog-scan` command.
The hand-written docs follow the same change: the environment table now
documents 24 x 16MB defaults and BASERT_HF_XET in place of the removed
BASERT_HF_FORCE_RANGED, and the models guide states plainly that a pull
resumes, with Xet described as the deduplicating alternative that does
not.
Verified in this tree: cargo build --locked --workspace clean,
cargo test --workspace 262 passed, clippy clean under -D warnings.
prabod
marked this pull request as ready for review
August 19, 2026 03:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Source sync for 0.2.3, generated by
tools/sync_public.pyfrom the engine treeat
e8a1fa11, plus hand-written docs the allow-list does not cover.Commits
sync:— generated. Allow-listed paths only; tuned quantization profiles andinternal tooling are excluded by rule.
docs:— hand-written.docs/is outside the allow-list.sync:(refresh) — regenerated after the resumable-downloads andcatalog-scanner changes landed internally.
Contents
base-convert/crates/base-hub: Xet support and a parallel, resumabledownloader (
download.rs), the catalog scanner (scan.rs), andfetch.rsported to hf-hub 1.0.
base-hub/Cargo.tomladdsindicatifandreqwest.catalog.json: regenerated from the published organization — 69 rows → 105,adding every Whisper bundle and Qwen3.8-27B, and correcting eight rows whose
size/sha256 no longer matched the Hub.
package.json,setup.py,__init__.py.BASERT_HF_*variables, and a "How downloadsbehave" section in the models guide.
Notes
Cargo.tomlfiles are newly allow-listed. Without them this PR wouldpublish
fetch.rs/download.rsagainst a manifest declaring neitherdependency, and the tree would not compile.
BASERT_HF_XET; itdeduplicates across models but cannot resume an interrupted transfer.
Verified in this branch
cargo build --locked --workspacecargo test --workspacecargo clippy --all-targets -- -D warnings