Skip to content

Avoid redundant scans when extracting stage0 archives - #165819

Open
TaKO8Ki wants to merge 1 commit into
rust-lang:mainfrom
TaKO8Ki:avoid-stage0-archive-rescans
Open

TaKO8Ki wants to merge 1 commit into
rust-lang:mainfrom
TaKO8Ki:avoid-stage0-archive-rescans

Conversation

@TaKO8Ki

@TaKO8Ki TaKO8Ki commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

getnames() calls getmembers(), which can scan the entire archive if its members have not already been loaded. Iterating over tar directly (https://github.com/python/cpython/blob/v3.10.12/Lib/tarfile.py#L2715-L2747) allows archive entries to be handled one at a time. Passing each TarInfo directly to extract() also avoids name-based lookup, removing the preliminary scan and its associated repeated decompression.

With cached stage0 archives, RustBuild.download_toolchain() improved from 6.44 s to 3.50 s (medians of two runs per version). The rustc, rust-std, and cargo archives were extracted in parallel into fresh directories. Output hashes were verified separately.

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Nov 11, 2026
@rustbot

rustbot commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: bootstrap
  • bootstrap expanded to 6 candidates
  • Random selection from Mark-Simulacrum, clubby789, jieyouxu

@Mark-Simulacrum

Copy link
Copy Markdown
Member

@bors r+ rollup

@rust-bors

rust-bors Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

📌 Commit ce09ca6 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@rust-bors rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 12, 2026
Zalathar added a commit to Zalathar/rust that referenced this pull request Nov 12, 2026
…ns, r=Mark-Simulacrum

Avoid redundant scans when extracting stage0 archives

[`getnames()`](https://github.com/python/cpython/blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/Lib/tarfile.py#L1991-L1995) calls [`getmembers()`](https://github.com/python/cpython/blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/Lib/tarfile.py#L1981-L1989), which can scan the entire archive if its members have not already been loaded. Iterating over tar directly (https://github.com/python/cpython/blob/v3.10.12/Lib/tarfile.py#L2715-L2747) allows archive entries to be handled one at a time. Passing each `TarInfo` directly to [`extract()`](https://docs.python.org/3.10/library/tarfile.html#tarfile.TarFile.extract) also avoids name-based lookup, removing the preliminary scan and its associated repeated decompression.

With cached stage0 archives, `RustBuild.download_toolchain()` improved from 6.44 s to 3.50 s (medians of two runs per version). The rustc, rust-std, and cargo archives were extracted in parallel into fresh directories. Output hashes were verified separately.
rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026
Rollup of 6 pull requests

Successful merges:

 - #165778 (Bump stage0 to 1.101.0-beta.1)
 - #163077 (feat: support nested functions with `#[offload_kernel]`)
 - #165578 (Avoid `Hash::hash` when stable-hashing various types)
 - #165729 (tests: Add regression test for infinite recursive dereference)
 - #165819 (Avoid redundant scans when extracting stage0 archives)
 - #165822 (Remove unused compiler modules and types)
Zalathar added a commit to Zalathar/rust that referenced this pull request Nov 12, 2026
…ns, r=Mark-Simulacrum

Avoid redundant scans when extracting stage0 archives

[`getnames()`](https://github.com/python/cpython/blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/Lib/tarfile.py#L1991-L1995) calls [`getmembers()`](https://github.com/python/cpython/blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/Lib/tarfile.py#L1981-L1989), which can scan the entire archive if its members have not already been loaded. Iterating over tar directly (https://github.com/python/cpython/blob/v3.10.12/Lib/tarfile.py#L2715-L2747) allows archive entries to be handled one at a time. Passing each `TarInfo` directly to [`extract()`](https://docs.python.org/3.10/library/tarfile.html#tarfile.TarFile.extract) also avoids name-based lookup, removing the preliminary scan and its associated repeated decompression.

With cached stage0 archives, `RustBuild.download_toolchain()` improved from 6.44 s to 3.50 s (medians of two runs per version). The rustc, rust-std, and cargo archives were extracted in parallel into fresh directories. Output hashes were verified separately.
rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026
Rollup of 12 pull requests

Successful merges:

 - #165804 (cg_llvm: Use an invariant lifetime brand for LLVM values)
 - #165778 (Bump stage0 to 1.101.0-beta.1)
 - #165412 (Implement `core::ops::FnAddr`)
 - #165487 (Migrate built-in macros `pattern_type` & `view_type` to macros 2.0 that expand to internal syntax)
 - #165818 (do not lint `redundant_lifetimes` on items from external macros)
 - #163077 (feat: support nested functions with `#[offload_kernel]`)
 - #164795 (Extend support of lint `explicit_outlives_requirements` to checked type aliases)
 - #165578 (Avoid `Hash::hash` when stable-hashing various types)
 - #165729 (tests: Add regression test for infinite recursive dereference)
 - #165819 (Avoid redundant scans when extracting stage0 archives)
 - #165822 (Remove unused compiler modules and types)
 - #165824 (Add regression test for HRTB projection bound with TryFrom error)
@Zalathar

Copy link
Copy Markdown
Member

@bors r-
@bors try jobs=dist-x86_64-msvc

@rust-bors rust-bors Bot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 12, 2026
@rust-bors

rust-bors Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

This pull request was unapproved.

This PR was contained in a rollup (#165843), which was unapproved.

View changes since this unapproval

@rust-bors

rust-bors Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

⌛ Trying commit ce09ca6 with merge 61389fc…

To cancel the try build, run the command @bors try cancel.

Workflow: https://github.com/rust-lang/rust/actions/runs/43155951269

rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026
Avoid redundant scans when extracting stage0 archives


try-job: dist-x86_64-msvc
@Zalathar

Copy link
Copy Markdown
Member

Looks like the CI failure was probably unrelated flakiness, though I'm marking this as iffy in case this PR is the cause of the flakiness.

@bors r=Mark-Simulacrum rollup=iffy

@rust-bors

rust-bors Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

📌 Commit ce09ca6 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@rust-bors rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants