From cf6b748a59fd5feb0b6b9129c73767584fda26f6 Mon Sep 17 00:00:00 2001 From: Viet Le Date: Tue, 3 Nov 2026 10:52:46 +0000 Subject: [PATCH 1/2] Create Alpha-Releases-Plan.md --- docs/Alpha-Releases-Plan.md | 118 ++++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 docs/Alpha-Releases-Plan.md diff --git a/docs/Alpha-Releases-Plan.md b/docs/Alpha-Releases-Plan.md new file mode 100644 index 0000000..b17c79d --- /dev/null +++ b/docs/Alpha-Releases-Plan.md @@ -0,0 +1,118 @@ +# Plan: daily alpha releases next to the quarterly beta release + +Goal: Python users can install the latest alpha and always get what is on `develop`. The quarterly beta (`release.yml`, see "Making a release" in +[Workflows.md](Workflows.md)) stays the stable, frozen release. + +## How C# does it (BuroHappold_Installer) + +- `ReleaseType` defaults to `alpha`. An alpha gets the patch number `yyMMdd` (the build date), so the installer is `X.Y.0.`, built daily from `master`/develop of every repository. +- A `beta` gets `X.Y.1.0` and is built from the tag of that version (`BuildSolution.ps1` checks out the latest matching tag). +- Same pipeline, two inputs: nothing is hand-edited for an alpha. + +## Mapping to Python + +| | Alpha | Beta (exists today) | +|---|---|---| +| Trigger | daily schedule, and by hand | by hand, end of a quarter's milestone | +| Source | tip of `develop`, as committed (no C# clone/build) | regenerated from the C#, then frozen on branch `vX.Y` | +| Version | `X.Y.0aYYMMDDHHmm`, UTC (PEP 440 pre-release) | `X.Y.0` | +| Install | `pip install --pre bhom-schema` | `pip install bhom-schema` | +| Published to | TestPyPI (PyPI later, same as beta) | TestPyPI (PyPI later) | +| Git | one moving tag `alpha-latest`, no GitHub Release | branch `vX.Y`, tag `vX.Y.0`, GitHub Release | + +pip has no `--alpha` option: `--pre` is the only built-in switch for pre-releases, so that is what the docs tell users to run. A specifier that names a pre-release (`"bhom-schema>=0a0"`, or an exact `bhom-schema==10.1.0a2605030600`) also works without `--pre`. A separate `bhom-schema-alpha` distribution was rejected: it would install the same `bhom_schema` package as `bhom-schema` and the two would overwrite each other. + +Why this shape: + +- **Reflect `develop`, not regenerate.** `develop` is already regenerated from the C# every Saturday by `python-schema-generation.yml` and takes human PRs in between. The alpha packages `develop` as it is, so it needs no C# build and runs in about a minute on ubuntu. +- **`aYYMMDDHHmm` extends the installer's date stamp with the time.** The installer's `yyMMdd` allows one build a day; here several alphas a day are needed, so the UTC time is added. The number after `a` is a plain integer in PEP 440, so it orders by time. pip ignores pre-releases by default, so stable users stay on the beta and alpha users opt in with `--pre`. Order is correct: `10.1.0a2601010600` < `10.1.0a2601021430` < `10.1.0`. +- **No tag per alpha.** `release.yml` finds the previous release with `git tag --list "v*.0"`; a tag per day would clutter tags and branches for nothing. One moving tag records the last published commit. + +## Version ordering around a beta release + +PEP 440 puts `10.0.0aYYMMDDHHmm` **below** `10.0.0`. Once `10.0.0` is released, alphas must be of the **next** version (`10.1.0aYYMMDDHHmm`), or `--pre` users would be +served the old beta. `__version__` comes from the BHoM version read from `BHoM.dll` (`_version.py`), so this holds once the BHoM version on `develop` has been +bumped after the milestone. The alpha workflow therefore refuses to publish, with a clear message, when `v{__bhom_version__}.0` already exists as a tag. + +Decided: a forgotten version bump fails the alpha (no fallback to `X.Y.0.post1.devYYMMDD`). The bump is a release checklist step (below). + +## The workflow: `.github/workflows/alpha-release.yml` + +Triggers: `schedule` daily (06:00 UTC, after Saturday's 04:00 generation has pushed) and `workflow_dispatch` with `dry_run` (default true) and `force` (publish even when nothing changed). + +Concurrency: one group shared with `python-schema-generation.yml` and `release.yml`, `cancel-in-progress: false`, so an alpha never packages a half-pushed `develop` or races a beta. + +One job on ubuntu, then a publish job (same split as `release.yml`): + +1. Checkout `develop`, `fetch-depth: 0`. +2. **Gate.** Skip (green, with a summary line) when `HEAD` equals the commit `alpha-latest` points at, unless `force`. Quiet days publish nothing. +3. **Guard.** Read `__bhom_version__` from `src/bhom_schema/_version.py`; fail when tag `v{version}.0` exists (see above). +4. **Stamp.** Compute `{bhom_version}.0a{yyMMddHHmm}` from the current UTC time and rewrite `__version__` in `_version.py` in the runner's checkout only. It is never committed, so `develop` keeps `X.Y.0.dev0`. +5. **Check.** `pip install -e ".[dev]" build`, `pytest`, `pyright`. These are the same fast checks as `pr-checks.yml` and need no C# assemblies (tests marked `realtypes` are skipped, as on the unit-test job). +6. **Build** the wheel; assert the wheel filename carries the stamped version. +7. Upload the wheel as an artifact. Stop here on `dry_run`. +8. **Publish job** (needs build, not dry run): environment `testpypi-alpha`, trusted publishing with `skip-existing: true` so a re-run of the same job is harmless (the stamp is computed once, in the build job, and passed on in the wheel); then move tag `alpha-latest` to `HEAD` (`git tag -f` + `git push -f` of that one tag, `contents: write` on this step only). + +Several alphas on the same day are allowed: each run has its own time in the stamp, and the gate only stops a run when `develop` has not moved since the last alpha. A manual run after a merge therefore publishes a new alpha at once, and the scheduled run that follows skips. + +## One-time setup (admin) + +### On the GitHub repository + +- Create the environment `testpypi` (used by `release.yml`; it can require a reviewer, since a beta is by hand) and the environment `testpypi-alpha` (used by `alpha-release.yml`; no reviewer, it runs unattended). Settings, Environments, New environment. +- Allow the workflow's token to push the `alpha-latest` tag (it is a tag, so it does not touch `develop` protection). + +### On test.pypi.org (once, for both releases) + +TestPyPI is a separate site with its own accounts: a pypi.org login does not work there. Trusted publishing means no token or password is stored in GitHub. + +1. Sign in at test.pypi.org with the account that will own the project (create it if needed), and turn on two-factor authentication, which is required to publish. +2. Open *Your account*, then *Publishing*. The project `bhom-schema` does not exist yet, so add **pending publishers** (they create the project on the first upload). Use *Add a new pending publisher*, GitHub tab, and enter, for the **beta**: + - PyPI project name: `bhom-schema` + - Owner: `BHoM` + - Repository name: `BHoM_PythonSchema` + - Workflow name: `release.yml` + - Environment name: `testpypi` +3. Add a second pending publisher for the **alpha**, identical except: + - Workflow name: `alpha-release.yml` + - Environment name: `testpypi-alpha` + + Two entries are needed: a publisher is bound to one workflow file and one environment, so the beta entry does not cover the alpha. +4. After the first successful upload the project exists and the pending publishers turn into normal ones. Under the project's *Manage*, then *Publishing*, check that both are listed. Under *Collaborators*, add the other maintainers as owners so the project does not depend on one person. +5. Check the first upload on the project page: the version, and that the wheel is there. Install it with `pip install --pre --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ bhom-schema` (the extra index is for pydantic). + +Real PyPI is set up the same way on pypi.org when it is time (the same two entries, with the same environment names or new ones), and the `repository-url` lines in both workflows are removed. + +### Scheduled runs + +- GitHub disables scheduled workflows after 60 days without repository activity; `develop` receives the weekly generation commit, which keeps it alive. If that ever stops, re-enable by hand. +- Allow the workflow's token to push the `alpha-latest` tag (it is a tag, so it does not touch `develop` protection). + +## Quarterly cycle + +| When | What | +|---|---| +| Daily | `alpha-release.yml` publishes `X.Y.0aYYMMDDHHmm` if `develop` changed (and by hand whenever a merge should be available at once) | +| Saturday | `python-schema-generation.yml` commits regenerated classes to `develop`; Sunday's alpha carries them | +| End of quarter (by hand) | Run `release.yml` dry run, then live: `vX.Y` branch, `vX.Y.0` tag, wheel on TestPyPI | +| Right after the beta | Bump the BHoM version on `develop` to the next `Major.Minor`; the next Saturday generation then writes `__bhom_version__ = "X.(Y+1)"` and alphas continue as `X.(Y+1).0aYYMMDDHHmm`. Until then the alpha guard fails by design | +| Between | Fixes to a released beta go on branch `vX.Y` and are released as `X.Y.N` (out of scope here; the beta workflow today only makes `X.Y.0`) | + +## Changes to make + +1. Add `.github/workflows/alpha-release.yml` as above. Put the stamp and guard logic in a small `py2cs` command (for example `python -m py2cs alpha-version --package src/bhom_schema --tags `) rather than inline PowerShell, so it can be unit tested under `.ci/unit-tests/python`. +2. Add the workflow to the shared concurrency group in `python-schema-generation.yml` and `release.yml`. +3. Update [Workflows.md](Workflows.md): a row in the table, a box in the diagram, a "Making an alpha release" section, and a line in the release section saying the BHoM version is bumped on `develop` afterwards. Document `pip install --pre bhom-schema` for users. +4. Add the unit test for the version stamp and guard (pre-release ordering against a released tag, two stamps on the same day ordering by time, no-tag-yet cases). + +## Verification (includes the test build) + +- Build and tests: run `run-all.ps1` (build, NUnit, pytest, Pyright) after adding the `py2cs` command and its tests. +- Dry run on GitHub: *Alpha release* with `dry_run` ticked on `develop`; check the wheel artifact's filename and that nothing is tagged or published. +- Ordering: `pip install --pre --dry-run --index-url https://test.pypi.org/simple/ bhom-schema` picks the newest alpha, and a plain install picks the beta. +- Live: untick `dry_run` once; confirm the TestPyPI project page, then re-run and confirm the second run skips. +- Guard: run against a state where `v{version}.0` exists and confirm it fails with the bump message. + +## Later (not part of this plan): Databricks + +The main consumers work in Databricks notebooks (`%pip install --pre bhom-schema`, which needs a runtime with Python 3.12 or later) and keep their data in Unity Catalog tables. Decided: their tables must follow the BHoM field names, so a row validates straight into a class. There are no real tables yet, so the mechanism can go the other way too: generate compliant tables from the Python schema (a Spark `StructType` or `CREATE TABLE` DDL derived from each model, nested types as `STRUCT`, interface-typed fields carrying `_t`), plus helpers to read a table into a list of objects and write objects back. This is a separate plan, to be written once the alpha/beta workflow above is in place. From 345e33176d5c80548a8ca9101758317da641f213 Mon Sep 17 00:00:00 2001 From: Viet Le Date: Tue, 3 Nov 2026 11:24:52 +0000 Subject: [PATCH 2/2] Add alpha release script --- .ci/generation/py2cs/__main__.py | 16 +- .ci/generation/py2cs/alpha.py | 49 ++++++ .../python/test_py2cs/test_alpha.py | 119 ++++++++++++++ .../python/test_py2cs/test_workflows.py | 36 +++++ .github/workflows/alpha-release.yml | 150 ++++++++++++++++++ .../workflows/python-schema-generation.yml | 4 +- .github/workflows/release.yml | 5 + docs/Workflows.md | 30 ++++ pyproject.toml | 2 +- 9 files changed, 407 insertions(+), 4 deletions(-) create mode 100644 .ci/generation/py2cs/alpha.py create mode 100644 .ci/unit-tests/python/test_py2cs/test_alpha.py create mode 100644 .github/workflows/alpha-release.yml diff --git a/.ci/generation/py2cs/__main__.py b/.ci/generation/py2cs/__main__.py index b19d0af..6602d11 100644 --- a/.ci/generation/py2cs/__main__.py +++ b/.ci/generation/py2cs/__main__.py @@ -1,4 +1,4 @@ -"""Command line: python -m py2cs check | manifest | diff | align | repos | plan | pr | close | verify | drafts | resolve | notes""" +"""Command line: python -m py2cs check | manifest | diff | align | repos | plan | pr | close | verify | drafts | resolve | notes | alpha-version""" from __future__ import annotations @@ -10,6 +10,7 @@ import tempfile from pathlib import Path +from py2cs.alpha import alpha_version, read_bhom_version, stamp_version_file from py2cs.align import AlignError, align, changed_types, generate from py2cs.check import check_package from py2cs.ci import affected_repos, load_allowed, make_client, verify_package @@ -108,6 +109,11 @@ def main(argv: list[str] | None = None) -> int: p.add_argument("--previous", help="the previous release's tag, named in the notes") p.add_argument("--out", required=True, type=Path) + p = sub.add_parser("alpha-version", help="the version of an alpha release, built from the BHoM version of a bhom_schema folder; fails when that BHoM version is already released") + p.add_argument("--package", required=True, type=Path) + p.add_argument("--tags", nargs="*", default=[], help="the release tags that exist, for example v10.0.0") + p.add_argument("--stamp", action="store_true", help="also write the version into the package's _version.py") + args = parser.parse_args(argv) try: return run(args) @@ -152,6 +158,14 @@ def run(args: argparse.Namespace) -> int: print(f"release notes for {len(diff.changes)} changed types") return 0 + if args.command == "alpha-version": + version_file = args.package / "_version.py" + version = alpha_version(read_bhom_version(version_file), args.tags) + if args.stamp: + stamp_version_file(version_file, version) + print(version) + return 0 + if args.command == "align": base, head = read_package(args.base_root), read_package(args.head_root) changed, removed = changed_types(base, head) diff --git a/.ci/generation/py2cs/alpha.py b/.ci/generation/py2cs/alpha.py new file mode 100644 index 0000000..7f181ef --- /dev/null +++ b/.ci/generation/py2cs/alpha.py @@ -0,0 +1,49 @@ +"""The version of an alpha release: the BHoM version of develop with a UTC time stamp, as a PEP 440 pre-release.""" + +from __future__ import annotations +import re +from datetime import datetime, timezone +from pathlib import Path + +BHOM_VERSION = re.compile(r'^__bhom_version__ = "(\d+\.\d+)"$', re.MULTILINE) +PACKAGE_VERSION = re.compile(r'^__version__ = ".*"$', re.MULTILINE) + +def read_bhom_version(version_file: Path) -> str: + """Returns the BHoM version (Major.Minor) that the generated _version.py holds. + + `alpha_version` builds the alpha from it. + """ + match = BHOM_VERSION.search(version_file.read_text(encoding="utf-8")) + if match is None: + raise ValueError(f"{version_file} has no __bhom_version__ of the form Major.Minor") + + return match.group(1) + +def alpha_version(bhom_version: str, released_tags: list[str], now: datetime | None = None) -> str: + """Returns `Major.Minor.0aYYMMDDHHmm` for the BHoM version, with the UTC time of `now` (the current time when left out). + + Fails when the tag `vMajor.Minor.0` is among the released tags: the alpha would sort below that release, so develop must first move to the next BHoM version. + The alpha workflow calls it. + """ + if f"v{bhom_version}.0" in released_tags: + raise ValueError( + f"develop still reports BHoM {bhom_version}, which is already released as v{bhom_version}.0, so an alpha would sort below that release. " + "Bump the BHoM version on develop to the next Major.Minor and let the next generation write it, then run the alpha again." + ) + + moment = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) + return f"{bhom_version}.0a{moment:%y%m%d%H%M}" + +def stamp_version_file(version_file: Path, version: str) -> None: + """Writes the version as `__version__` into _version.py, replacing the development version. + + The alpha workflow does it in its own checkout only, so develop keeps its development version. + """ + text = version_file.read_text(encoding="utf-8") + stamped, count = PACKAGE_VERSION.subn(f'__version__ = "{version}"', text) + if count != 1: + raise ValueError(f"{version_file} must have exactly one __version__ line") + + # newline="" keeps the LF endings of the generated file + with version_file.open("w", encoding="utf-8", newline="") as stream: + stream.write(stamped) diff --git a/.ci/unit-tests/python/test_py2cs/test_alpha.py b/.ci/unit-tests/python/test_py2cs/test_alpha.py new file mode 100644 index 0000000..2a44e2a --- /dev/null +++ b/.ci/unit-tests/python/test_py2cs/test_alpha.py @@ -0,0 +1,119 @@ +from __future__ import annotations +from datetime import datetime, timedelta, timezone +from pathlib import Path +import pytest +from packaging.version import Version +from py2cs.__main__ import main +from py2cs.alpha import alpha_version, read_bhom_version, stamp_version_file + +def write_version_file(folder: Path, bhom_version: str = "10.1") -> Path: + """Writes a _version.py in the form the generator writes it and returns its path. + + The tests of the alpha version use it. + """ + folder.mkdir(parents=True, exist_ok=True) + version_file = folder / "_version.py" + version_file.write_text( + "# Generated by BHoM_PythonSchema from the BHoM assemblies. Do not edit by hand.\n" + "# __bhom_version__ is the BHoM version (Major.Minor) of the classes; __version__ is the version of this package.\n\n" + f'__bhom_version__ = "{bhom_version}"\n' + f'__version__ = "{bhom_version}.0.dev0"\n', + encoding="utf-8", + newline="\n", + ) + return version_file + +def at(day: int, hour: int, minute: int) -> datetime: + """Returns a UTC time in November 2026. + + The tests of the alpha version use it as the build time. + """ + return datetime(2026, 11, day, hour, minute, tzinfo=timezone.utc) + +def test_the_alpha_version_is_the_bhom_version_with_the_utc_time(): + """Checks that the alpha version is Major.Minor.0a followed by the two-digit year, month, day, hour and minute in UTC. + + Protects the format that users and the workflow rely on. + """ + assert alpha_version("10.1", [], at(3, 14, 30)) == "10.1.0a2611031430" + +def test_the_time_is_taken_in_utc_whatever_the_zone_of_the_clock(): + """Checks that a time with another zone offset is converted to UTC before it is stamped. + + Protects the order of alphas from a runner with a local time zone. + """ + plus_two = datetime(2026, 11, 3, 14, 30, tzinfo=timezone.utc).astimezone(timezone(timedelta(hours=2))) + assert alpha_version("10.1", [], plus_two) == "10.1.0a2611031430" + +def test_alphas_of_the_same_day_order_by_time_and_stay_below_the_beta(): + """Checks, with the PEP 440 ordering pip uses, that two alphas of one day order by time, that a later day orders after them, and that all are below the beta of the same version and above the previous beta. + + Protects the rule that stable users stay on the beta and alpha users get the newest alpha. + """ + morning = Version(alpha_version("10.1", ["v10.0.0"], at(3, 6, 0))) + afternoon = Version(alpha_version("10.1", ["v10.0.0"], at(3, 14, 30))) + next_day = Version(alpha_version("10.1", ["v10.0.0"], at(4, 6, 0))) + + assert Version("10.0.0") < morning < afternoon < next_day < Version("10.1.0") + assert all(version.is_prerelease for version in (morning, afternoon, next_day)) + +def test_an_alpha_of_a_released_bhom_version_fails_and_says_to_bump(): + """Checks that the alpha fails, naming the released tag and the bump, when the tag of its BHoM version exists. + + Protects --pre users from being served an old beta because develop's BHoM version was not bumped after the release. + """ + with pytest.raises(ValueError, match=r"already released as v10\.0\.0.*Bump the BHoM version"): + alpha_version("10.0", ["v9.0.0", "v10.0.0"], at(3, 14, 30)) + +def test_an_older_release_does_not_stop_the_alpha_of_the_next_version(): + """Checks that tags of other versions, and no tags at all, do not stop an alpha. + + Protects the first alpha, and the alpha right after the bump. + """ + assert alpha_version("10.1", ["v9.0.0", "v10.0.0"], at(3, 6, 0)).startswith("10.1.0a") + assert alpha_version("10.0", [], at(3, 6, 0)).startswith("10.0.0a") + +def test_the_bhom_version_is_read_from_the_version_file(tmp_path: Path): + """Checks that the BHoM version is read from the generated _version.py and that a file without one is an error. + + Protects the alpha from stamping a version it did not read. + """ + assert read_bhom_version(write_version_file(tmp_path / "ok", "10.1")) == "10.1" + + empty = tmp_path / "empty" + empty.mkdir() + (empty / "_version.py").write_text("__version__ = '1'\n", encoding="utf-8") + with pytest.raises(ValueError, match="no __bhom_version__"): + read_bhom_version(empty / "_version.py") + +def test_stamping_replaces_only_the_package_version_and_keeps_line_endings(tmp_path: Path): + """Checks that stamping rewrites the __version__ line, leaves the BHoM version and the comments alone, and keeps the LF line endings. + + Protects the generated file in the alpha's own checkout from other edits. + """ + version_file = write_version_file(tmp_path) + stamp_version_file(version_file, "10.1.0a2611031430") + + text = version_file.read_bytes().decode("utf-8") + assert '__version__ = "10.1.0a2611031430"\n' in text + assert '__bhom_version__ = "10.1"\n' in text + assert "dev0" not in text + assert "\r" not in text + +def test_the_command_prints_the_version_and_stamps_only_when_asked(tmp_path: Path, capsys: pytest.CaptureFixture[str]): + """Checks that the alpha-version command prints the version and leaves the file alone without --stamp, writes it with --stamp, and fails with exit code 2 for a released BHoM version. + + Protects the workflow step that uses the command. + """ + version_file = write_version_file(tmp_path, "10.1") + + assert main(["alpha-version", "--package", str(tmp_path), "--tags", "v10.0.0"]) == 0 + printed = capsys.readouterr().out.strip() + assert Version(printed).is_prerelease and printed.startswith("10.1.0a") + assert "dev0" in version_file.read_text(encoding="utf-8") + + assert main(["alpha-version", "--package", str(tmp_path), "--tags", "v10.0.0", "--stamp"]) == 0 + stamped = capsys.readouterr().out.strip() + assert f'__version__ = "{stamped}"' in version_file.read_text(encoding="utf-8") + + assert main(["alpha-version", "--package", str(tmp_path), "--tags", "v10.1.0"]) == 2 diff --git a/.ci/unit-tests/python/test_py2cs/test_workflows.py b/.ci/unit-tests/python/test_py2cs/test_workflows.py index d684c1f..61e6ca7 100644 --- a/.ci/unit-tests/python/test_py2cs/test_workflows.py +++ b/.ci/unit-tests/python/test_py2cs/test_workflows.py @@ -277,3 +277,39 @@ def test_the_included_repositories_file_is_tidy(): assert text.endswith("\n") assert "dKop" not in text, "the repository is spelt dKoP_Toolkit" assert all(name == name.strip() for name in text.splitlines()) + +def test_the_alpha_workflow_publishes_only_when_asked_and_only_the_publish_job_gets_an_identity_token(): + """Checks that the alpha workflow starts on a schedule or by hand, is a dry run by default, publishes only when develop moved and not a dry run, gives an identity token to the publish job only, and stores no secret. + + Protects TestPyPI from an unplanned alpha and the identity token from the build job. + """ + workflow = load("alpha-release.yml") + assert set(workflow["on"]) == {"schedule", "workflow_dispatch"} + assert workflow["on"]["workflow_dispatch"]["inputs"]["dry_run"]["default"] is True + assert workflow["permissions"] == {"contents": "read"} + assert list(workflow["jobs"]) == ["build", "publish-testpypi", "mark"] + assert "permissions" not in workflow["jobs"]["build"] + publish = workflow["jobs"]["publish-testpypi"] + assert publish["permissions"] == {"id-token": "write"} and publish["environment"] == "testpypi-alpha" + assert publish["if"] == "${{ needs.build.outputs.publish == 'true' && !inputs.dry_run }}" + assert workflow["jobs"]["mark"]["needs"] == ["build", "publish-testpypi"], "the tag moves only after a publish" + assert "secrets." not in (WORKFLOWS / "alpha-release.yml").read_text(encoding="utf-8") + +def test_the_alpha_stamps_the_version_in_its_own_checkout_and_never_commits_it(): + """Checks that the alpha workflow reads the released tags, stamps with py2cs alpha-version, runs no commit or push of develop, and moves only the tag alpha-latest. + + Protects develop from a commit with a stamped version, and the weekly generation from a conflict with it. + """ + text = (WORKFLOWS / "alpha-release.yml").read_text(encoding="utf-8") + assert 'git tag --list "v*.0"' in text and "py2cs alpha-version" in text and "--stamp" in text + assert "git commit" not in text and "HEAD:develop" not in text + pushes = [line.strip() for line in text.splitlines() if "git push" in line] + assert pushes == ["git push --force origin alpha-latest"] + +def test_the_workflows_that_write_to_develop_or_its_tags_share_one_concurrency_group(): + """Checks that the generation, the release and the alpha share one concurrency group that never cancels a running one. + + Protects an alpha from packaging a half-pushed develop and from racing a release. + """ + for name in ("python-schema-generation.yml", "release.yml", "alpha-release.yml"): + assert load(name)["concurrency"] == {"group": "develop-publishing", "cancel-in-progress": False}, name diff --git a/.github/workflows/alpha-release.yml b/.github/workflows/alpha-release.yml new file mode 100644 index 0000000..1978ccd --- /dev/null +++ b/.github/workflows/alpha-release.yml @@ -0,0 +1,150 @@ +name: Alpha release + +# Publishes develop as it is, as a pre-release, so Python users can run pip install --pre bhom-schema and get what is on develop. It does not +# regenerate and needs no C# build: develop is regenerated by python-schema-generation.yml and takes pull requests in between. The version is +# the BHoM version with the UTC time (X.Y.0aYYMMDDHHmm); it is stamped in this checkout only, never committed. A run is skipped when develop has not +# moved since the last alpha. Fails when develop still reports a BHoM version that is already released: bump it first. + +on: + schedule: + - cron: '0 6 * * *' # every day, after the Saturday generation (4 AM) has pushed + workflow_dispatch: + inputs: + dry_run: + description: Check and build the wheel, but do not publish + type: boolean + default: true + force: + description: Publish even when develop has not changed since the last alpha + type: boolean + default: false + +# The generation and the release write to develop and its tags, so none of them runs while another does +concurrency: + group: develop-publishing + cancel-in-progress: false + +permissions: + contents: read + +jobs: + build: + runs-on: ubuntu-latest + outputs: + publish: ${{ steps.gate.outputs.publish }} + version: ${{ steps.stamp.outputs.version }} + sha: ${{ steps.gate.outputs.sha }} + env: + PYTHONPATH: ${{ github.workspace }}/.ci/generation + + steps: + - name: Checkout develop + uses: actions/checkout@v4 + with: + ref: develop + fetch-depth: 0 + + # alpha-latest is the commit of the last published alpha + - name: Has develop moved since the last alpha? + id: gate + shell: bash + run: | + sha=$(git rev-parse HEAD) + last=$(git ls-remote --tags origin alpha-latest | cut -f1) + echo "sha=$sha" >> "$GITHUB_OUTPUT" + if [ "$last" = "$sha" ] && [ "${{ inputs.force }}" != "true" ]; then + echo "publish=false" >> "$GITHUB_OUTPUT" + echo "develop ($sha) is already published as the last alpha: nothing to do." | tee -a "$GITHUB_STEP_SUMMARY" + else + echo "publish=true" >> "$GITHUB_OUTPUT" + fi + + - uses: actions/setup-python@v5 + if: steps.gate.outputs.publish == 'true' + with: + python-version: '3.12' + cache: pip + cache-dependency-path: pyproject.toml + + - name: Install the Python tools + if: steps.gate.outputs.publish == 'true' + run: python -m pip install -e ".[dev]" build + + # Fails when the BHoM version of develop is already released (tag vX.Y.0); the stamp is written into this checkout only + - name: Stamp the alpha version + id: stamp + if: steps.gate.outputs.publish == 'true' + shell: bash + run: | + tags=$(git tag --list "v*.0") + version=$(python -m py2cs alpha-version --package src/bhom_schema --tags $tags --stamp) + echo "version=$version" >> "$GITHUB_OUTPUT" + echo "Alpha version: $version" | tee -a "$GITHUB_STEP_SUMMARY" + + - name: Every module imports + if: steps.gate.outputs.publish == 'true' + run: python -m py2cs verify --package src/bhom_schema + + - name: Tests that need no C# assemblies + if: steps.gate.outputs.publish == 'true' + run: python -m pytest .ci/unit-tests/python/test_py2cs .ci/unit-tests/python/test_support -q + + - name: Pyright + if: steps.gate.outputs.publish == 'true' + run: python -m pyright + + - name: Build the wheel + if: steps.gate.outputs.publish == 'true' + shell: bash + run: | + python -m build --wheel --outdir dist + ls dist + ls dist | grep -F "${{ steps.stamp.outputs.version }}" || { echo "The wheel does not carry the version ${{ steps.stamp.outputs.version }}"; exit 1; } + + - uses: actions/upload-artifact@v4 + if: steps.gate.outputs.publish == 'true' + with: + name: alpha + path: dist + + # Trusted publishing: TestPyPI trusts this workflow and environment, so there is no token or password to store + publish-testpypi: + needs: build + if: ${{ needs.build.outputs.publish == 'true' && !inputs.dry_run }} + runs-on: ubuntu-latest + environment: testpypi-alpha + permissions: + id-token: write + + steps: + - name: Download the wheel + uses: actions/download-artifact@v4 + with: + name: alpha + path: dist + + - name: Publish to TestPyPI + uses: pypa/gh-action-pypi-publish@release/v1 + with: + repository-url: https://test.pypi.org/legacy/ + skip-existing: true + + # Records the commit that was published, so the next run can tell whether develop has moved + mark: + needs: [build, publish-testpypi] + runs-on: ubuntu-latest + permissions: + contents: write + + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ needs.build.outputs.sha }} + + - name: Move the tag alpha-latest + run: | + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + git tag -f alpha-latest + git push --force origin alpha-latest + echo "alpha-latest now points at ${{ needs.build.outputs.sha }} (${{ needs.build.outputs.version }})" >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/python-schema-generation.yml b/.github/workflows/python-schema-generation.yml index c75d4c4..05164a1 100644 --- a/.github/workflows/python-schema-generation.yml +++ b/.github/workflows/python-schema-generation.yml @@ -13,9 +13,9 @@ on: type: boolean default: false -# Only one generation at a time +# The generation, the release and the alpha write to develop and its tags, so none of them runs while another does concurrency: - group: python-schema-generation + group: develop-publishing cancel-in-progress: false # Raised for the commit step only diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index cafe81c..18ef80a 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -15,6 +15,11 @@ on: type: boolean default: true +# The generation, the release and the alpha write to develop and its tags, so none of them runs while another does +concurrency: + group: develop-publishing + cancel-in-progress: false + permissions: contents: read diff --git a/docs/Workflows.md b/docs/Workflows.md index f008ffc..c34e311 100644 --- a/docs/Workflows.md +++ b/docs/Workflows.md @@ -12,6 +12,7 @@ written to.** BHoM, BHoM_Engine and JSONSchema_Toolkit are only cloned and built | [linked-csharp-pr.yml](../.github/workflows/linked-csharp-pr.yml) | A pull request that changes `src/bhom_schema`, and by hand | One job, **Raise or update draft**: finds the C# repositories the change touches, raises or updates a linked **draft** pull request in each, and comments on the Python PR with its link. The shared action `.github/actions/linked-pr-context` does the finding | | [linked-csharp-pr-close.yml](../.github/workflows/linked-csharp-pr-close.yml) | A Python pull request that changes `src/bhom_schema` is closed, and by hand | Closes the linked C# draft with a comment when the Python PR was closed unmerged, or only comments when it was merged. A separate workflow so that nothing of it shows on an open pull request | | [release.yml](../.github/workflows/release.yml) | By hand only (dry run by default) | Regenerates with `--release X.Y`, checks, builds the wheel and the release notes; when not a dry run, pushes branch `vX.Y` and tag `vX.Y.0`, creates the GitHub Release and publishes to TestPyPI. See "Making a release" | +| [alpha-release.yml](../.github/workflows/alpha-release.yml) | Every day at 6 AM UTC, and by hand (dry run by default) | Publishes `develop` as it is, as the pre-release `X.Y.0aYYMMDDHHmm`, to TestPyPI. No C# build. Skipped when `develop` has not moved since the last alpha. See "Making an alpha release" | ## How they fit together @@ -53,6 +54,12 @@ written to.** BHoM, BHoM_Engine and JSONSchema_Toolkit are only cloned and built │ every PR can read it │ └──────────────────────────────┘ + Every day: +┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ +│ alpha-release.yml develop moved since tag alpha-latest? ─► stamp X.Y.0aYYMMDDHHmm ─► pytest, │ +│ Pyright ─► wheel ─► TestPyPI (trusted publishing) ─► move tag alpha-latest │ +└──────────────────────────────────────────────────────────────────────────────────────────────────────┘ + End of a milestone, by hand: ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ release.yml (input X.Y, dry run by default) │ @@ -134,6 +141,29 @@ TestPyPI is reached with trusted publishing, so no password or token is stored. Real PyPI is added the same way when BHoM/BHoM is in scope. The branch `v10.0` is left alone afterwards. +**Right after a release, bump the BHoM version on `develop`** to the next `Major.Minor`. The next weekly generation then writes it into `_version.py`. Until then +the alpha workflow fails on purpose: an alpha of `10.0` would sort below the release `10.0.0`, so `--pre` users would be served the old release. + +## Making an alpha release + +Alphas are made by `alpha-release.yml` every day at 6 AM UTC (after the Saturday generation has pushed), so nothing is done by hand. Python users install the +latest alpha with `pip install --pre bhom-schema`; without `--pre` they get the release. pip has no `--alpha` option. + +- **Source.** `develop` as it is. Nothing is regenerated and no C# is built; `develop` is regenerated weekly and takes pull requests in between. +- **Version.** `python -m py2cs alpha-version` takes the BHoM version from `_version.py` and adds the UTC time: `10.1.0a2611031430`. Several alphas a day are fine, they order by time. + The version is stamped in the runner's checkout only and never committed, so `develop` keeps `10.1.0.dev0`. It fails when the tag `v10.1.0` already exists (see the bump above). +- **Skip.** The tag `alpha-latest` points at the commit of the last published alpha. When `develop` is still there, the run does nothing, unless *force* is ticked. +- **Checks.** Every module imports, the tests that need no C# assemblies, Pyright, and the wheel carries the stamped version. +- **Publish.** The wheel goes to TestPyPI with trusted publishing, through the environment `testpypi-alpha`; then `alpha-latest` moves to the published commit. There is no GitHub Release and no tag per alpha. + +Run it by hand from the Actions tab with `dry_run` ticked (the default) to check and build without publishing, or unticked to publish a merge at once. +It shares one concurrency group with the weekly generation and the release, so it never packages a half-pushed `develop`. + +TestPyPI is set up once, by an administrator, for both the release and the alpha. On the repository, create the environments `testpypi` and `testpypi-alpha` +(Settings, Environments). On test.pypi.org, sign in with two-factor authentication, open *Your account*, *Publishing*, and add two pending publishers for the project +`bhom-schema`, owner `BHoM`, repository `BHoM_PythonSchema`: workflow `release.yml` with environment `testpypi`, and workflow `alpha-release.yml` with environment `testpypi-alpha`. +A publisher is bound to one workflow file, so both are needed. After the first upload, check that both are listed under the project's *Publishing* and add the other maintainers as owners. + ## When is a Python pull request ready? A Python PR is a proposal: it is ready once its linked C# pull request is merged. The **C# alignment** job of `pr-checks.yml` is the check for diff --git a/pyproject.toml b/pyproject.toml index e0e2f13..94aad35 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,7 +11,7 @@ license = { file = "LICENSE" } dependencies = ["pydantic>=2.13,<3"] [project.optional-dependencies] -dev = ["pytest>=8", "pyright==1.1.414", "pyyaml>=6", "types-PyYAML", "jsonschema>=4.18", "types-jsonschema"] +dev = ["pytest>=8", "pyright==1.1.414", "pyyaml>=6", "types-PyYAML", "jsonschema>=4.18", "types-jsonschema", "packaging>=23"] [tool.setuptools.dynamic] version = { attr = "bhom_schema._version.__version__" }