Add PyPI release workflow and Dependabot config by mwaisaida-13 · Pull Request #2 · aws/developer-toolkit-elasticache

mwaisaida-13 · 2026-08-18T16:58:26Z

What

.github/workflows/python-release.yml — publishes the Python package to PyPI when a python-v* GitHub Release is published.
- Auth via PyPI Trusted Publishing (OIDC) + PEP 740 attestations — no stored API token.
- test → build → publish. A full 3.10–3.14 matrix gates the release (build/publish needs: test); re-runs the CI tests as a release gate.
- build verifies tag == pyproject.toml version and refuses while the Private :: Do Not Upload classifier is set.
- All actions SHA-pinned (verified against their version tags); build/twine version-pinned and installed with --isolated; twine check --strict.
- Post-publish smoke test installs from PyPI and runs the CLI.
.github/dependabot.yml — weekly grouped updates for github-actions SHAs (+ version comments) and the pip dependencies.

⚠️ Operational prerequisites (must be done before a release can publish)

Create a pypi GitHub Environment with a required reviewer.
Register a Trusted Publisher on PyPI referencing repo aws/developer-toolkit-elasticache, workflow filename python-release.yml, environment pypi — and reserve the project name.
Enable Dependabot security updates (Settings → Code security) so botocore CVEs get fix PRs.

Note

The Private :: Do Not Upload classifier is still in pyproject.toml, so publishing is intentionally blocked until the open-source review clears — the build verify step enforces this. This PR adds the pipeline; it does not enable publishing.

python-release.yml publishes to PyPI on python-v* GitHub Releases via Trusted Publishing (OIDC) + PEP 740 attestations. A full 3.10-3.14 test matrix gates build/publish; the build job verifies tag==version and refuses while the 'Private :: Do Not Upload' guard is set. All actions are SHA-pinned, build/twine pinned and installed with --isolated, and a post-publish smoke test installs from PyPI and runs the CLI. dependabot.yml adds weekly grouped updates for github-actions SHAs and the Python (pip) dependencies.

ChristophKoerner

Looking through this change, I sense that we need a way to track "build" dependencies (to pin versions of build tools), a better mechanism to share data among automated CI jobs (e.g. supported Python versions), and a better mechanism to manage build scripts (e.g. as separate scripts in the python directory).

…e versions Per Christoph's review on PR #2: - Pin build tooling in python/requirements-build.txt, installed via build_dist.py. Dependabot's pip ecosystem now tracks and bumps these. - Centralize supported Python versions: pyproject.toml classifiers are the single source, read by scripts/supported_python_versions.py and consumed by both python.yml and python-release.yml matrices via a 'versions' job. - Move shell out of the release YAML into python/scripts/: verify_release.py, build_dist.py, smoke_test.py.

Address review feedback: - Extract all shell logic from workflow YAML into python/scripts/ (build.sh, lint.sh, test.sh, verify_release.sh, smoke_test.sh, load_python_versions.sh). Zero embedded shell in workflows. - Centralize CI Python version matrix in .github/python-versions.json, shared by both workflows via load_python_versions.sh. - Pin build tools (build==1.5.0, twine==7.0.0) in a dedicated python/requirements/build.txt tracked by Dependabot. - Update dependabot.yml with pip ecosystem entry for build tools. - Use actions/setup-python directly (SHA-pinned) in all jobs.

ChristophKoerner · 2026-08-20T10:51:37Z

@@ -0,0 +1 @@
+["3.10", "3.11", "3.12", "3.13", "3.14"]


Could these be embedded in pyproject.toml?

ChristophKoerner · 2026-08-20T10:52:06Z

+set -euo pipefail
+
+python3 -m pip install -e ".[dev]"
+python3 -m ruff check .


is ruff missing in the build dependencies?

It's a dev dependency, added a pin in the new commit

ChristophKoerner · 2026-08-20T10:53:52Z

not a big fan of this! The purpose of having a separate build file is that build / test / lint / release should not depend on Github CI runner to be executable. However, this script doesn't add any value for manual execution.

ChristophKoerner · 2026-08-20T10:54:46Z

+# Retries because the index can lag the upload.
+set -euo pipefail
+
+version="$(python3 -c "import pathlib; print(next(pathlib.Path('dist').glob('*.whl')).name.split('-')[1])")"


I am not a fan of inline python snippets like this. 2 options:

simpler bash native extraction

build script itself is written in Python

ChristophKoerner · 2026-08-20T10:55:57Z

+/tmp/smoke/bin/developer-toolkit-elasticache --help
+/tmp/smoke/bin/python -c "import developer_toolkit_elasticache"


Should we add any asserts here - for example on the version? Or are we just expecting the package to not stop with an exit code != 0?

Yeah that was the case. Modified the smoke test in dev.py to now asserts it: after pip install {pkg}=={version} it reads back importlib.metadata.version(...) and fails on mismatch

ChristophKoerner · 2026-08-20T10:57:13Z

+# Called from the release workflow's build job.
+set -euo pipefail
+
+tag_version="${GITHUB_REF_NAME#python-v}"


how can we modify this to also run outside of Github CI runner?

Replaced the shell scripts with a single dev.py task runner in next commit.

Supersede the shell-script approach with a single Python task runner that runs the same locally and in the workflows. - python/dev.py: one entry point for lint/test/build/verify/versions/smoke/ci. It creates and reuses a local .venv and shells out to the pinned tools, so contributors run `./dev.py ci` with no manual venv setup. - Supported Python versions live once in [tool.python-support] (pyproject); `dev.py versions` generates the range both workflow matrices consume. Removes .github/python-versions.json. - Dev/build/publish tooling pinned in python/requirements/dev.txt (ruff, pytest, build, twine), tracked by Dependabot. Removes requirements/build.txt. - Workflow steps are now one-line `python python/dev.py <cmd>` calls (no embedded shell); verify checks tag==version and the Do-Not-Upload guard; smoke asserts the installed version.

ChristophKoerner · 2026-08-21T22:17:16Z

+"""Developer/CI task runner — one entry point for every CI/CD step, run locally or
+from the workflows. Requires Python 3.11+ (tomllib) for `verify` and `versions`


Let's think about how the interface and ergonomics of this command runner a bit more. Ideally, we use this runner (I call it ./run below) for all supported languages, for example:

./run python lint

./run java test

In the above case, python and java would refer to the Devkit source code of the corresponding language - which would correspond to folders in this repo. This model could even extend to example code snippets, for example via ./run java/examples test.

I'd recommend to use a library (for example click to hide the undifferentiated complexity of the command definition, parameterization, and invocation.

Excellent idea, I will look into this expansion as a follow-up when we introduce new languages

ChristophKoerner · 2026-08-21T22:17:38Z

+    ./dev.py test             # editable install + pytest
+    ./dev.py build            # build wheel + sdist, twine check --strict
+    ./dev.py verify [TAG]     # tag == pyproject version + publish guard
+    ./dev.py versions         # print the supported-version matrix as JSON


nit: list-supported-versions

ChristophKoerner · 2026-08-21T22:19:00Z

+
+    ./dev.py lint             # ruff check + format --check
+    ./dev.py test             # editable install + pytest
+    ./dev.py build            # build wheel + sdist, twine check --strict


nit: package

ChristophKoerner · 2026-08-21T22:20:12Z

+# dev.py is the task runner (not shipped code), so exempt the rules that fight its job:
+#   T201    — it prints command output and ::error:: annotations
+#   S603    — it shells out to the build tools (subprocess) with fixed, trusted args
+#   PLC0415 — it imports tomllib lazily so lint/test/build still run on Python 3.10
+"dev.py" = ["T201", "S603", "PLC0415"]


Let's move this script to a separate folder, maybe scripts/ in the workspace root.

ChristophKoerner · 2026-08-21T22:23:26Z

@@ -0,0 +1,13 @@
+# Dev/CI tooling, pinned so CI is reproducible and a new release of any of these
+# can't silently break an unrelated PR. Dependabot bumps them (pip entry in


We need a mechanism to install the required dev tooling:

Python 3.12 + PIP

Dev tooling pip dependencies

Then we need dependencies for each of the supported languages (that should work for each of the supported versions for each language):

Build -> dependencies required to build, test and package the lib

Runtime -> runtime dependencies of the lib

I think this would also be a follow-up for multiple language support. Currently the different scripts runs run(py, "-m", "pip", "install", "-q", "-e", ".", "-r", "requirements/dev.txt") to install the dependencies. will add an additional setup script for local development

affonsov

A few comments

affonsov · 2026-08-21T23:03:39Z

-    "ruff>=0.6",
-]
+# Supported Python range — the source for the CI/release test matrices.
+[tool.python-support]


[tool.python-support] is now the source of truth for the test/release matrix, but the supported range is still declared in two other hand-maintained places: requires-python = ">=3.10" (line 17) and the Programming Language :: Python :: 3.10…3.14 trove classifiers (lines 29-33). Bumping max here to 3.15 won't add the classifier or move the floor, so the matrix and the package metadata will silently disagree. Since the whole point of this table is one source of truth, consider deriving the classifiers (and validating the requires-python floor against min) from [tool.python-support] — e.g. generate/verify them in dev.py — or at least add a check that the three stay in sync.

I see, will add a check to verify there's no drift

affonsov · 2026-08-21T23:03:39Z

+def versions(_args: argparse.Namespace) -> int:
+    support = _support()
+    low, high = int(support["min"].split(".")[1]), int(support["max"].split(".")[1])
+    vers = [f"3.{minor}" for minor in range(low, high + 1)]


vers = [f"3.{minor}" for minor in range(low, high + 1)] hardcodes the 3. major and derives the range only from .split(".")[1], so the major components of min/max are parsed and then discarded. This is the one place that turns [tool.python-support] into the actual release gate, and it will emit wrong versions the moment the range ever crosses a major boundary (e.g. max = "4.0" would still produce 3.x), while looking like it respected the config. Parse both parts and assert the majors match (or build the list from the real major), rather than assuming 3..

Good point, I will address this as follow-up in future PRs

affonsov · 2026-08-21T23:03:39Z

+    return 0
+
+
+def versions(_args: argparse.Namespace) -> int:


verify (tag == pyproject version, and the Private :: Do Not Upload guard) and versions (matrix generation) are what gate an actual PyPI publish, but there's no test covering them — the repo already has a tests/ dir. A regression in verify is exactly the kind of thing that would let a mistagged release or a still-private package slip through, and it wouldn't be caught until a real release. Worth a small unit test for both: verify rejects a tag mismatch and the do-not-upload classifier, and versions emits the expected JSON for a given [tool.python-support].

affonsov · 2026-08-21T23:03:39Z

+  test:
+    name: test (py${{ matrix.python-version }})
+    needs: versions
+    runs-on: ubuntu-latest


The release test gate runs only on ubuntu-latest, whereas regular CI (python.yml) additionally tests Windows and macOS at the ends of the range — precisely where interpreter/platform differences surface. As written, a Windows- or macOS-only regression at the range boundary passes the release gate and ships. If that's a deliberate cost/time tradeoff, a one-line comment saying so would help; otherwise mirror CI's include matrix (min/max on windows-latest + macos-14) here.

Sure, added a comment in next commit.

- Rename python/dev.py -> scripts/python_dev.py: a repo-level runner decoupled from the Python package (not shipped in the wheel or sdist). - Add `setup` command that installs the package + pinned dev tooling; every task self-installs on run (pip is a no-op when satisfied), matching GLIDE's pattern. - Rename commands: `versions` -> `list-supported-versions`, `build` -> `package`; fold the [tool.python-support] drift check into `verify`. - Pin pip in requirements/dev.txt; drop the now-unneeded dev.py lint ignore (the runner no longer lives under python/). - Workflows call `python scripts/python_dev.py <task>`; CI path filter adds scripts/**.

ChristophKoerner · 2026-08-26T12:05:59Z

+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        prog="dev.py", description="Developer/CI task runner."


nit: filename changed to python_dev.py

good catch, will update

ChristophKoerner · 2026-08-26T12:07:44Z

-defaults:
-  run:
-    working-directory: python


Can you verify that this was intentionally removed?

Yes it's intentional. The runner now lives at scripts/python_dev.py and is invoked from the repo root, so a working-directory: python default would break the path to the script.

ChristophKoerner · 2026-08-26T12:09:20Z

+  # python-v* tags only — test/build/publish inherit this via `needs`.
+  versions:
+    name: python versions
+    if: startsWith(github.ref_name, 'python-v')


Do we have a plan for generating these tags? Would this be done from a command in python_dev.py file?

Right now the python-v* tag is created manually via the GitHub Release UI. A python_dev.py release/tag command that bumps the pyproject version and cuts the tag would be a good follow-up addition

mwaisaida-13 requested review from ChristophKoerner and ogasarah August 18, 2026 16:59

ChristophKoerner reviewed Aug 19, 2026

View reviewed changes

mwaisaida-13 force-pushed the add-pypi-release-workflow branch from efcd182 to 2ddf66b Compare August 19, 2026 14:19

ChristophKoerner reviewed Aug 20, 2026

View reviewed changes

ChristophKoerner reviewed Aug 21, 2026

View reviewed changes

affonsov reviewed Aug 21, 2026

View reviewed changes

ChristophKoerner approved these changes Aug 26, 2026

View reviewed changes

mwaisaida-13 merged commit 39baf39 into main Aug 26, 2026
13 checks passed

mwaisaida-13 deleted the add-pypi-release-workflow branch October 13, 2026 15:55

		/tmp/smoke/bin/developer-toolkit-elasticache --help
		/tmp/smoke/bin/python -c "import developer_toolkit_elasticache"

		"""Developer/CI task runner — one entry point for every CI/CD step, run locally or
		from the workflows. Requires Python 3.11+ (tomllib) for `verify` and `versions`

		@@ -0,0 +1,13 @@
		# Dev/CI tooling, pinned so CI is reproducible and a new release of any of these
		# can't silently break an unrelated PR. Dependabot bumps them (pip entry in

Conversation

mwaisaida-13 commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

⚠️ Operational prerequisites (must be done before a release can publish)

Note

Uh oh!

ChristophKoerner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

affonsov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

mwaisaida-13 commented Aug 18, 2026 •

edited

Loading