ESP32-S3 first-time install misses 4 min at 10 Mbit (5.81 min, 399 MB) #1944

Activity

1. fastled-project-sync

   added this to FastLED Trackeron Nov 7, 2026
2. zackees commented on Nov 7, 2026

   

   zackees

   on Nov 7, 2026

   MemberAuthor

   More actions

   Measured the first option on real hardware/host, and PR #1945 takes it:

   • pioarduino publishes the same esptool release as the 617 KiB source archive that fbuild already installs for Windows (esptoolpy-v5.3.0.zip → v5.3.0/esptool.zip). The standalone bundle is 82,508,446 B; the archive is 617,641 B.
   • Images are byte-identical (same elf2image argv, sha256=3ef75dd1…), and the source path is faster warm: 0.10 s vs 0.29–0.31 s per image, because the PyInstaller one-file bundle unpacks ~217 MB on every invocation.
   • Cold install of the source path end-to-end: 26.8 s, then 4.2 s warm. It needs uv (present for every uv tool install fbuild) and a Python 3.10 interpreter, so the fallback to the standalone binary stays and engages only when that environment cannot be built.

   That removes ~70 MiB: ~331 MB / ~4.4 min at 10 Mbit, which matches your estimate for dropping the bundle. The remaining gap to under 4 min is the 168 MB toolchain, i.e. the mirror-side "single-chip, zstd-compressed toolchain" option — this PR does not touch it, so the issue stays open for that.
3. zackees commented on Nov 7, 2026

   

   zackees

   on Nov 7, 2026

   MemberAuthor

   More actions

   Measured the remaining option on the real toolchain fbuild installs, so it stops being a guess.

   Source: the extracted xtensa-esp-elf 13.2.0 unified toolchain already in the local fbuild
   prod cache (660 MB extracted, linux x86_64), re-packed with tar + zstd. "Single-chip" excludes
   xtensa-esp-elf/lib/esp32 and lib/esp32s2, keeping esp32s3.

   archive	size	vs the 168 MB vendor download
   whole, zstd -3	173.6 MB	+3%
   whole, zstd -19	125.7 MB	−42 MB (25%)
   single-chip esp32s3, zstd -3	100.6 MB	−67 MB (40%)
   single-chip esp32s3, zstd -19	73.1 MB	−95 MB (56%)

   Why slicing pays: the per-chip SDK libs are lib/esp32 204 MB + lib/esp32s2 103 MB +
   lib/esp32s3 102 MB of 660 MB. Compiler/cc1/ld binaries are chip-shared (bin 80 MB,
   libexec 80 MB) and the top-level multilib archives (~66 MB) stay, so the chip halves are
   exactly the removable bulk.

   Compression alone is not enough — that is the load-bearing detail. zstd -19 on the whole
   tree saves only 42 MB (125.7 MB); the slice is what buys the other 53 MB. Doing "compress the
   toolchain" without slicing would leave you at 275 MB / 4.0 min, i.e. still on the line.

   Completeness check, not just a smaller tarball: the zstd -19 slice was extracted to a clean dir
   and bin/xtensa-esp32s3-elf-gcc reports 13.2.0, -print-libgcc-file-name resolves inside the
   slice (<slice>/lib/gcc/xtensa-esp-elf/13.2.0/esp32s3/libgcc.a), and compiling a C++ TU with
   -std=gnu++17 -Os -c exits 0.

   Projection at this issue's own measured rate (399 MB / 5.81 min = 68.7 MB/min ≈ 9.2 Mbit/s):

   • after feat(esptool): provision the 617 KiB source archive instead of the 82 MiB bundle #1945: 317 MB → 4.62 min
   • plus the single-chip zstd toolchain: 222 MB → 3.24 min — clears the under-4-min target

   Caveat: I measured 13.2.0 because 14.2.0 (xtensa-esp-elf-14.2.0_20260121, the one in your
   399 MB trace) is not in this host's cache. SDK lib content differs, so expect ~73–85 MB rather
   than a byte-exact 73.1; re-measure on 14.2.0 at publish time.

   Not implemented here: the server half is a slicer + publisher modelled on the existing
   ci/slice_esp_libs.py → ci/publish_esp_slices.py → @esp-libs shard path (the mirror
   credentials are present on this host), and the client half needs a manifest contract mapping
   toolchain URL + MCU → chip slice plus install-layout/caching keys, with the whole archive kept
   as the fallback. That is a multi-run change; this comment is the sizing, with real bytes.
4. zackees commented on Nov 7, 2026

   

   zackees

   on Nov 7, 2026

   MemberAuthor

   More actions

   Server half of the last option is now implemented rather than sized: #1947 adds
   ci/slice_toolchain.py, which emits one deterministic per-chip tar.zst plus a
   slices.json index carrying the same manifest_slices shape the ESP libs slicer
   already produces.

   Measured on the real cached xtensa-esp-elf 13.2.0 tree (--chips esp32s3 --verify, 4m50s including verification):

   	value
   slice	75,299,480 B (75.3 MB)
   payload	423,807,794 B, 2,030 entries
   vendor unified download	168 MB
   sha256	00a6ca50…c66c

   The slice is verified by using it: extracting it and running its own
   xtensa-esp32s3-elf-gcc (13.2.0), -print-libgcc-file-name resolving to
   lib/gcc/xtensa-esp-elf/13.2.0/esp32s3/libgcc.a inside the extraction, and a
   -std=gnu++17 -Os -c probe compile. Selection is by path component, so
   bin/libexec/include and the top-level multilib archives are never dropped;
   bin+libexec are chip-shared and are the bulk of what remains.

   Two things the implementation had to get right that the sizing did not show:
   the real tree ships an in-tree .. symlink (lib/bfd-plugins/liblto_plugin.so -> ../../libexec/...), so the guard tests containment rather than rejecting ..;
   and .install_complete is never archived, because fbuild's install check is
   directory-existence based and a stale marker would let a partial extraction pass.

   On the issue's own measured rate (68.7 MB/min) this puts the install at
   ~222 MB / ~3.24 min after #1945, inside the under-4-min target.

   Still open on this issue: publishing the slices to the mirror (key layout +
   shard registration, modelled on publish_esp_slices.py) and the client half
   (resolve toolchain URL + MCU → slice with the whole archive as fallback). I
   deliberately did not upload archives that nothing resolves yet; the client PR is
   where the end-to-end ESP32-S3 build proves this path.
5. zackees commented on Nov 7, 2026

   

   zackees

   on Nov 7, 2026

   MemberAuthor

   More actions

   Update after #1945, which replaced the 82 MB esptool bundle with a 617 KiB source archive. The published nightly (run 42359356709, ab98de1c) shows ESP32-S3 fbuild cold at 4.51 min / 310.8 MB (PlatformIO 10.83 min / 676 MB, Arduino CLI 26.64 min / 1.92 GB).

   Getting under 4 min needs about 30 MB more off the wire. The 168 MB toolchain is the remaining big item.
6. zackees commented on Nov 7, 2026

   

   zackees

   on Nov 7, 2026

   MemberAuthor

   More actions

   Client half is merged: #1949 (53277a80). fbuild now resolves per-chip toolchain slices; it no longer needs the whole 168 MB unified archive when a slice is published.

   What landed:

   • @esp-toolchain shard for toolchain-xtensa-esp-elf, alongside the existing @esp-libs one. The RISC-V toolchain is deliberately not slice-capable (its multilibs are ISA variants, not chips).
   • A slice installs into its own chip-scoped cache dir (<version>+<chip>), because a slice holds ONE chip's SDK libs; a whole archive keeps the shared dir and is preferred when already installed, so existing hosts never re-download and two chips still get two correct trees.
   • A listed slice is authoritative: a corrupt slice fails the install instead of quietly fetching the full archive.
   • Esp32Toolchain::with_slice_variant + the orchestrator pass build.mcu on both the build and install --check paths.

   Verified: 265 lib tests (5 new drive the real staged_install against a local server and assert the vendor archive is never requested), 222 fbuild-build-esp tests, workspace clippy -D warnings, dylint for both touched crates, and 3 mutations RED.

   Still open on this issue: publishing the slices (publish_esp_slices.py-shaped uploader + the @esp-toolchain shard entry, keyed so slices.json from ci/slice_toolchain.py maps onto the manifest's asset.slices), then the end-to-end ESP32-S3 install measurement — the ~222 MB / ~3.24 min projection is unproven until a slice is actually served. Until then every client takes the whole-archive fallback, i.e. today's behaviour.
7. zackees commented on Nov 8, 2026

   

   zackees

   on Nov 8, 2026

   MemberAuthor

   More actions

   Publishing is done and the path is now measured end-to-end on this host.

   Server (#1951, e729c666): @esp-toolchain is a live shard. linux-amd64 /
   14.2.0_20260121 slices, each --verifyd and re-downloaded + hashed against its
   pin after upload: esp32s3 106,462,349 B, esp32s2 106,655,161 B, esp32
   141,361,956 B, against the 176,559,292 B vendor archive (whole bundle mirrored
   bucket-first too, pinned sha da31f36d…). ci/publish_toolchain_slices.py is
   the reproducible uploader; --check keeps the Index descriptor honest.

   Client (#1953, 6e5be796) — both published slices were inert before it, and
   that is why the earlier numbers never moved:

   1. slice_shard matched only toolchain-xtensa-esp-elf, while Esp32Toolchain
      names its package esp32-xtensa-gcc: the shard was never even fetched.
   2. The slice was then re-verified against the WHOLE archive's checksum
      (expected da31f36d…, got 36a9f0e7…), so every slice install aborted.

   Measured after both fixes (rebuilt CLI + daemon, EMPTY isolated cache,
   minimal esp32-s3-devkitc-1 project): toolchain 106,462,349 B slice instead of
   the 168 MB archive; install dir …-linux-gnu+esp32s3 = 603 MB instead of 1.2 GB;
   two shards fetched (esp-libs + @esp-toolchain); no whole archive on disk;
   build succeeded in 65.1s with flash 355899 / ram 74919 — identical firmware
   sizes to the whole-archive build, so the slice is not a degraded toolchain.

   Still open on this issue:

   • Other host packages (linux-arm64, macos*, win32) need their extracted
     tree and are one publish_toolchain_slices.py invocation each; those hosts keep
     the whole archive until then (supported fallback, not a failure).
   • The benchmark's first-install measurement should now show the ~222 MB /
     under-4-min projection; that needs a nightly run with the released tool.
   • Note for whoever reads Pages: fastled.github.io snapshots main on
     github-pages deployments that do not fire on every push, so until the next
     one the primary manifest answers NotListed for @esp-toolchain and clients
     fall back to whole archives (raw.githubusercontent is already current).