[veenious]

Adds three ppc64le microarchitecture variant targets, mirroring the
existing x86_64_v2/v3/v4 pattern:

• ppc64le_power9-unknown-linux-gnu (-mcpu=pwr9)
• ppc64le_power10-unknown-linux-gnu (-mcpu=pwr10)
• ppc64le_power11-unknown-linux-gnu (-mcpu=pwr11)

The variants reuse the generic ppc64le LLVM + Stretch sysroot build
(build.cross image) and differ only in -mcpu. glibc floor stays 2.17.

Changes:

• cpython-unix/targets.yml: add the three variant targets
• ci-targets.yaml: add CI entries
• cpython-unix/build.py: normalise variant triples for autoconf
• cpython-unix/build-cpython.sh: use cleaned installed config vars for
  PYTHON.json
• cpython-unix/build-expat.sh: --with-dev-urandom for ppc64le
• cpython-unix/build-libX11.sh, build-libXau.sh, build-libxcb.sh: keep
  Clang cross flags in libtool
• cpython-unix/extension-modules.yml: _decimal CONFIG_64 pattern matches
  the variant triples
• docs/running.rst: document the variant targets
• src/release.rs, src/validation.rs: register the variant triples

Closes #1215

[jjhelmus]

As mentioned in #1215 before considering adding these targets I would like to see benchmarks which should that there is a benefit to building this microarchitecture specific distributions.

Additional the change to bookworm likely raises the minimum glibc version required to consume these distributions from 2.24 (Debian 9 Stretch) to 2.36 (Debian 12 Bookworm).

A better approach that is more maintainable in the long term to to use the LLVM toolchain to cross-compile these targets using a sysroot. This was mentioned in #1215.

[veenious]

Hardware validation + apples-to-apples benchmark — Python 3.13.15 on real P9/P10/P11 hardware

All three binaries were cross-compiled from x86 (commit cbcc593) and tested on RHEL 9.8 machines inside registry.access.redhat.com/ubi10/ubi (glibc 2.39) containers.

Smoke tests — ALL PASSED

All three binaries passed: Python version check, ssl, sqlite3, zlib, json, math, hashlib, ctypes imports, and pip invocation.

---

Performance benchmark — generic vs tuned, same machine

To isolate the tuning benefit from hardware differences, the same generic ppc64le-unknown-linux-gnu release binary was benchmarked against our tuned binary on the exact same machine.

Workload: Pi to 10,000 digits via Python's decimal stdlib module (pure CPU arithmetic, no external dependencies), 3 runs each.

POWER10

Binary	Run 1	Run 2	Run 3	Avg
ppc64le-unknown-linux-gnu (generic baseline)	14.282 s	14.298 s	14.522 s	14.367 s
ppc64le_power10-unknown-linux-gnu	8.770 s	8.802 s	8.842 s	8.805 s

→ 38.7% faster on the same P10 machine with -mcpu=power10

POWER11

Binary	Run 1	Run 2	Run 3	Avg
ppc64le-unknown-linux-gnu (generic baseline)	13.039 s	13.009 s	13.029 s	13.026 s
ppc64le_power11-unknown-linux-gnu	8.046 s	8.036 s	8.019 s	8.034 s

→ 38.3% faster on the same P11 machine with -mcpu=power11

---

glibc requirement

Power9 requires glibc ≥ 2.17 (same as the generic baseline). Power10 and Power11 require glibc ≥ 2.36 due to the Bookworm build base. This is enforced in validation.rs.

[jjhelmus]

The snapshot release should not be removed. We want reproducible images when possible

[jjhelmus]

A good amount of work is done to keep the x86-64 microarchitecture variants based on the same glibc version. The same should be done here. There is limited functionality in uv to select a build based on the host glibc version so supporting older versions is important.

I would appreciate more details benchmarks, at minimum using fastmark but if possible pyperformance. These would give a more complete picture. The benchmarks should be based on the same compiler version as these difference can be substantial. If upgrading the compiler (and glibc version) give the bulk of the performance gain there is no reason to expand the build matrix.

Is there a reason why cross-compiling using LLVM was not been tried discussed in #1215 and my earlier comment?

[jjhelmus]

This would need to have a different author

[clnperez]

@veenious sorry I missed that author fix when I sent over my initial suggestions. lmk if you want me to re-push anything or if your'e fine with fixing my commits.

@jjhelmus thanks for the reviews again. for the sake of this PR -- we'd assumed that since the project hadn't gone that route yet we could add these in the way they're already built and continue to discuss in #1100 . No other reason.

[veenious]

Hi @jjhelmus — updated the PR to address all review feedback. Summary below.

---

Changes from previous revision

Approach: LLVM 23 + Debian Stretch sysroot (cross-compiled on x86)

The LLVM cross-compilation approach in this PR follows #1259 (riscv64 LLVM PR) and #1241 (aarch64 Stretch sysroot PR) as the reference patterns.

• Replaced the three separate per-variant Dockerfiles with a single unified build.cross-ppc64le.Dockerfile using Debian Trixie base + LLVM 23 from apt.llvm.org + Debian Stretch ppc64el sysroot (same sysroot approach as the existing build.cross.Dockerfile for plain ppc64le-unknown-linux-gnu)
• Snapshot URL pinned to snapshot.debian.org/archive/debian/20221105T150728Z/ for reproducible builds
• Dockerfile LABEL updated (not Gregory Szorc)

Shell script changes — each one fixed a specific bug required for LLVM cross-compilation:

File	Change	Why
build-cpython.sh	4 fixes	(1) --no-tls-get-addr-optimize is GNU ld only — lld rejects it causing configure to fail; (2) clang cross-flags (--sysroot, --gcc-install-dir, --target) leaked into sysconfig, polluting downstream extension builds; (3) glibc_version.txt regex was broken with lld's objdump output format; (4) installed_config_vars used for correct PYTHON.json after sysconfig patch
build-expat.sh	1 line	expat detected getentropy() (glibc ≥ 2.25 symbol) — same --with-dev-urandom fix already applied for x86, extended to ppc64le
build-libX11.sh	4 lines	libtool's shared-library link filter silently stripped --target= and --gcc-install-dir= — wrong runtime objects were linked
build-libXau.sh	4 lines	same libtool bug
build-libxcb.sh	4 lines	same libtool bug

All changes are guarded to clang or ppc64le targets and do not affect any existing x86/aarch64/musl builds.

---

glibc

All three new variants now match the existing ppc64le-unknown-linux-gnu and x86 variant floor of glibc 2.17. The Stretch sysroot (glibc 2.24) combined with configure suppressions keeps the runtime minimum at 2.17.

validation.rs entries for all three variants set to 2.17.

---

pyperformance benchmarks

Ran pyperformance 1.14.0 (--fast, 85 benchmarks) on bare-metal RHEL 9.8. Same compiler (clang-23) on all three — only -mcpu differs.

Variant	Compiler	-mcpu	Geometric mean vs power9
power9 (baseline)	clang-23	pwr9	—
power10	clang-23	pwr10	1.35x faster (+35%)
power11	clang-23	pwr11	1.36x faster (+36%)

Selected results:

Benchmark	P10 vs P9	P11 vs P9
scimark_monte_carlo	1.64x	1.74x
scimark_sparse_mat_mult	1.66x	1.70x
scimark_sor	1.59x	1.67x
float	1.59x	1.60x
mdp	1.89x	1.91x
asyncio_tcp	2.17x	1.88x
xml_etree_parse	1.50x	1.56x
python_startup	1.58x	1.48x
unpickle_pure_python	1.57x	1.66x
2to3	1.31x	1.32x
regex_compile	1.29x	1.39x
nbody	1.23x	1.30x

The gains are from POWER10/11 ISA features (MMA, new vector/crypto instructions) enabled by -mcpu=pwr10/pwr11, not from compiler version differences — clang-23 was used identically for all three.

[jjhelmus]

Thanks for the update. I'm out this week but will review this when I am back.
There is also a build of LLVM 23.1.2 that includes lld packaged in the same manner as the existing llvm compilers used in this project. That will likely be preferred for reproducibility over the apt.llvm.org packages. #1303 has the start of this upgrade. I have some work use this for cross-compiling but not in a PR yet.

[veenious]

Hi @jjhelmus — happy to switch to the astral-sh/llvm-build tarball. We checked the existing llvm-x86_64-linux (gnu_only) tarball and it does not include lld, which we need for ppc64le cross-linking. Once #1303 has a tarball with lld included and your cross-compilation pattern is ready, we will rebase and update the Dockerfile. Let us know if there is anything else to address in the meantime.

[veenious]

Hi @jjhelmus
just a gentle reminder on this PR. I understand you’re out this week, so please take a look when you’re back. Thanks!

[veenious]

hi @jjhelmus
Could you please re-review when you have time? Thanks.

Approach

All four ppc64le targets (these three and the existing generic one) are cross-compiled from x86_64 with the packaged LLVM bundle (astral-sh/llvm-build, clang + lld) and a Debian Stretch ppc64el sysroot, following #1334.

glibc

All variants keep the same floor as generic ppc64le and the x86_64 variants: glibc 2.17. validation.rs enforces it, and the highest symbol version in the built binaries is GLIBC_2.17.

Testing

Python 3.12.15, 3.13.17 and 3.14.9 (noopt) were built for power9, power10 and power11. Each of the 9 builds:

• passes pythonbuild validate-distribution
• imports decimal, ssl, sqlite3, zlib, bz2, lzma, ctypes on the matching real hardware (POWER9, POWER10, Power11; RHEL 9.8)

[veenious]

Update: Upstream merged #1349, which adds LLVM cross-compilation support for ppc64le. Since this happened while the PR was under review, I rebased onto it (head 5ee6df0) and simplified the changes. Please ignore the earlier comments about a separate image and libgcc_s.so.1.

The three variants now reuse the upstream build.cross image and generic ppc64le settings, adding only -mcpu=pwr9/10/11. No new Dockerfile is needed.

I also removed the libgcc_s.so.1 allowlist, since upstream's -static-libgcc handles it, and dropped the Verify-Peer=false workaround. The glibc minimum remains unchanged at 2.17.

I've rebuilt and tested this commit with Python 3.12.15, 3.13.17, and 3.14.9 for POWER9, POWER10, and POWER11. All nine builds pass validate-distribution and successfully import decimal, ssl, sqlite3, zlib, bz2, lzma, and ctypes on the corresponding hardware.

@jjhelmus Could you please re-review the updated PR when you get a chance? Thanks!

[prb112]

Thanks for rebasing. I'm looking forward to the review. Thanks in advance Paul

[jjhelmus]

This is independent from adding new targets and should be done in a separate PR. This would improve all cross-builds.

[jjhelmus]

This is no longer needed given the embedding of these flags into the LIBTOOL_CC environment variable.

[jjhelmus]

Same as libX11

[jjhelmus]

Same as libX11

[jjhelmus]

These should be in the markdown file. The rst file only drive the older Sphinx documentation.

[jjhelmus]

These should not be changed.

[jjhelmus]

I think adding these new CI targets will exceed the number of Linux builds in the second shard of the matrix. This will need to be expanded to support the additional jobs.

[veenious]

Thanks for the review, @jjhelmus. I've rebased and addressed review comments.

Each point is addressed below:

• build-cpython.sh (PYTHON.json config vars): Dropped. The file is now identical to main. I'll open a separate PR if the change is still wanted.
• build-libX11.sh, build-libXau.sh, build-libxcb.sh (libtool sed): Removed. LIBTOOL_CC already carries --target, --sysroot and --gcc-install-dir, so the sed isn't needed. All three files are identical to main.
• docs/running.rst: Reverted to main. The docs are now in docs/running.md, and the flag text matches what the targets pass.
• pythonbuild/downloads.json: Reverted. The mirror-URL edits were a local network workaround I committed by accident. The file is identical to main.
• CI shard overflow: You were right. A push to main goes from 490 to 574 Linux jobs with these targets (+30 power9, +27 power10, +27 power11), which exceeds 2 × 256. I changed linux.yml to --max-shards 3, added the python-build-matrix-2 and has-builds-2 outputs, and added a build-2 job that reuses the same steps. I ran the workflow's own jq expressions on the generated matrix. The shards come out as 256 / 256 / 62, and every variant job appears exactly once. stdlib.yml is unchanged, because it only selects run: true entries and every ppc64le entry is run: false. I can't run the workflow locally, so build-2 is untested until CI runs.