Skip to content

execution-*.yml install jax unpinned — the daily linux run has been red since 2026-08-18 #619

Description

@mmcky

execution-linux.yml, execution-osx.yml and execution-win.yml each install jax with no version constraint, and the daily linux run has been failing since 2026-08-18 for exactly the reason the pinned workflows were pinned.

Found while independently validating the 2026-08-19 settle week for QuantEcon/workspace-lectures#51. The jax pin that landed in #617 covers three of the six workflows in this repo that install jax; these are the other three.

What is pinned and what is not

workflow line install pinned
cache.yml 30 pip install "jax[cuda13]==0.11.0" yes (#617)
ci.yml 35 pip install "jax[cuda13]==0.11.0" yes (#617)
publish.yml 47 pip install "jax[cuda13]==0.11.0" yes (#617)
execution-linux.yml 34 pip install "jax[CPU]" no
execution-osx.yml 34 pip install "jax[CPU]" no
execution-win.yml 38 pip install "jax[CPU]" no

Re-derived by parsing all 26 workflow files across this repo and the three translations at origin/main (915bfd9f). The translations have no execution-*.yml, so this is English only.

The daily run is red

execution-linux.yml runs on 0 15 * * * and builds with jb build lectures --path-output=./ -W --keep-going, which executes every lecture, so a CellTimeoutError fails the job.

date run outcome
2026-08-16 31954622977 success — Successfully installed jax-0.11.0 jaxlib-0.11.0, then numpy_vs_numba_vs_jax.md: Executed notebook in 4.35 seconds
2026-08-17 32041361630 failure, unrelated — a 429 downloading setup-miniconda, jax never installed
2026-08-18 32152771492 failureSuccessfully installed jax-0.11.1 jaxlib-0.11.1, then CellTimeoutError
2026-08-19 32268416676 failure — same, at head_sha 915bfd9f, i.e. after #617 merged

The 08-19 log shows jax_intro.md executing fine in 22.95 s, then numpy_vs_numba_vs_jax.md: Executing notebook using local CWD at 15:13:33 and WARNING: Executing notebook failed: CellTimeoutError at 15:24:47, hitting the 600 s myst-nb timeout.

Incidentally, this is a cleaner controlled experiment than the one the incident record says it lacks: same workflow, same runner image, only the jax version differing — 0.11.0 executes the lecture in 4.35 s, 0.11.1 times out.

Why it matters beyond the red badge

The exposure map on QuantEcon/workspace-lectures#49 lists three workflows and describes English as "one rebuild away" from the problem, pointing at the Monday cache cron. That understates it: English has been demonstrably broken on a daily linux schedule since 2026-08-18, and nobody is watching that check because it is not on the map.

Suggested change

Pin the three execution-*.yml files to jax[CPU]==0.11.0 to match the other three, or record explicitly why they are exempt. Pinning jax alone is sufficient — jax 0.11.0's metadata carries Requires-Dist: jaxlib<=0.11.0,>=0.11.0, and the pairing is additionally enforced at import in both directions, so jaxlib cannot drift.

Two related things worth deciding at the same time, though neither needs to block this:

  • lectures/jax_intro.md:36 is a second unpinned !pip install jax quantecon and also uses lax.fori_loop; it executed in 22.95 s under 0.11.1, so it does not hit the problem today.
  • lectures/numpy_vs_numba_vs_jax.md:48 is !pip install quantecon jax, unpinned, which is what Colab readers get. That one is already parked for discussion on QuantEcon/workspace-lectures#49.

Reproduction

The regression reproduces on linux x86_64, linux aarch64 and macOS arm64 with the lecture's own cells (jax.device_put to the CPU device, @partial(jax.jit, static_argnames=("n",)), lax.fori_loop, n=10_000_000). Under jax==0.11.0 it completes in about 1.8 s; under jax==0.11.1 it does not complete within 300 s.

It is worth noting it is not a deadlock and not specific to fori_loop — it is a quadratic blow-up in XLA:CPU execution that hits lax.scan identically. Measured on linux x86_64 at n=100k/200k/400k: fori 6.27 s / 28.49 s / 103.26 s and scan 3.83 s / 15.45 s / 60.43 s under 0.11.1, against a flat ~0.05 s under 0.11.0 for both.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions