Skip to content

BLD: add option to build future-compatible abi3 wheels - #1600

Draft
neutrinoceros wants to merge 11 commits into
pyproj4:mainfrom
neutrinoceros:bld/abi3
Draft

neutrinoceros wants to merge 11 commits into
pyproj4:mainfrom
neutrinoceros:bld/abi3

Conversation

@neutrinoceros

@neutrinoceros neutrinoceros commented Jun 16, 2026 •

Copy link
Copy Markdown
Contributor

This allows targeting CPython's LimitedAPI (and stable ABI) by setting PYPROJ_LIMITED_API=1 at build time. Resulting wheels are future compatible with yet-unreleased Python versions.
The trade-off is that runtime performance might be impacted, though I suggest measuring it against real load-bearing applications, as I found that in many numpy-based extensions the impact to be negligible or even non-measurable.
I could successfully build the package and run the test suite with this locally.

I suggest a couple possible approaches to leverage this:

  • append a cp314-abi3 target to the existing build matrix. Version-specific wheels will still be prefered by installers when available but will default to these otherwise, instead of having to, and often failing to build pyproj from source.
  • replace version-specific wheels with a single cp311-abi3 wheel per platform. If there are no visible performance regressions this is definitely the way to go.

Blocked by

Comment thread setup.py

# restrict LIMITED_API usage:
# - require an env var PYPROJ_LIMITED_API=1
# - LIMITED_API is not compatible with free-threading (as of CPython 3.14)

@neutrinoceros neutrinoceros Jun 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note this comment is going to age like milk because there's a new version of the Limited API coming up in Python 3.15 that's compatible with both GIL-enabled and free-threaded builds. However there are a couple other blockers right now (Cython and setuptools both have work-in-progress patches to support this). I'd be happy to revisit this later this year to add cp315-abi3.abi3t as a target

Comment thread setup.py Resolved
Comment thread setup.py Resolved
@snowman2 snowman2 added this to the 4.0.0 milestone Oct 10, 2026
@codecov

codecov Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.81%. Comparing base (cce793d) to head (3599d2d).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1600   +/-   ##
=======================================
  Coverage   95.81%   95.81%           
=======================================
  Files          20       20           
  Lines        1886     1886           
=======================================
  Hits         1807     1807           
  Misses         79       79           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@snowman2

Copy link
Copy Markdown
Member

Mind enabling this in the wheel builds to test?

@neutrinoceros

Copy link
Copy Markdown
Contributor Author

enabling this in CI revealed that it is currently blocked by cython

pyproj/_geod.c:1412:6: This module uses CPython specific internals of 'array.array', which are not available in PyPy or the limited API

and indeed, compilation then predictibly fails because the generated module isn't limited-api-compliant

@snowman2

Copy link
Copy Markdown
Member

You could test it by adding a flag to use the slower alternative:

IF CTE_PYTHON_IMPLEMENTATION == "CPython":

@neutrinoceros
neutrinoceros force-pushed the bld/abi3 branch 3 times, most recently from 8721db0 to 0ef9dbf Compare October 13, 2026 19:47
@neutrinoceros

Copy link
Copy Markdown
Contributor Author

thanks for the pointer. I now expect the current CI run to succeed. Should I drop the second commit before undrafting ?

@snowman2

Copy link
Copy Markdown
Member

I appreciate your work on this and have learned a few things in the process.

I hesitate to add this in. It doesn't significantly reduce the wheel count and makes the code slower. Someone could enable this without being aware of the consequences.

I would probably want a more performant alternative when this is enabled before merging.

@neutrinoceros

Copy link
Copy Markdown
Contributor Author

Note that the branch currently implements my second suggested approach, but I actually would favor the first one if you like it best

@snowman2

Copy link
Copy Markdown
Member

I should have read your comment. The first approach sounds like a good strategy.

@neutrinoceros

Copy link
Copy Markdown
Contributor Author

I think I won't be able to avoid duplication between jobs, but it should be possible to mitigate it by moving the static part of your cibw configuration to pyproject.toml first. It could absolutely be a separate, preliminary PR, though I do not know yet when I'll have time for it. Does it seem like an acceptable plan to you ?

@snowman2

Copy link
Copy Markdown
Member

Feel free to try it and see what it looks like.

@neutrinoceros

neutrinoceros commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author

see gh-1661

@neutrinoceros

Copy link
Copy Markdown
Contributor Author

I'm planning to rebase this atop #1663 once stable

@neutrinoceros

neutrinoceros commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants