Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,9 @@ just scan # out-of-band secret + SAST scan: semgrep (no-eval) + gitleaks (full

`just scan` runs semgrep's `no-eval` rule and a gitleaks **full-history** secret scan (`.gitleaks.toml` = default ruleset). It is out-of-band (chained into no recipe), but CI enforces it: the `scan` job in `.github/workflows/test-template.yml` is a blocking PR gate. gitleaks is pinned in `mise.toml` (`gitleaks = "8.30.1"`) and installed in CI via `jdx/mise-action` + `mise exec`; semgrep runs via `uvx semgrep@1.167.0` (no dep, like zizmor). **semgrep scans non-test Python only** — its built-in `.semgrepignore` excludes `tests/`, and there is no `src/`, so on this repo it currently scans **0 files** (a forward guard that mirrors the shipped gate and fires the moment any non-test Python is added at root); gitleaks scans the whole tree + full history regardless of language and is the substantive gate here. Never pass semgrep `--config auto` (it drops the pinned rule and needs metrics on); never hardcode the gitleaks version in CI (install via `mise exec`).

Deliberate divergences from the template's `scan.yml` (`template/{% if not in_existing_repo %}.github{% endif %}/workflows/…scan.yml….jinja`): the maintainer folds scanning into the existing `test-template.yml` as a sibling `scan` job (the template consolidates into a standalone `scan.yml`), matching the one-workflow / per-tool layout and letting the existing zizmor job audit it; zizmor stays its own job here rather than a step in `scan` (already dogfooded standalone). The CI `mise-action` comment drops the template's "kept fresh by Renovate" note — **the maintainer has no Renovate**, so the pins are static.
Deliberate divergences from the template's `scan.yml` (`template/{% if not in_existing_repo %}.github{% endif %}/workflows/…scan.yml….jinja`): the maintainer folds scanning into the existing `test-template.yml` as a sibling `scan` job (the template consolidates into a standalone `scan.yml`), matching the one-workflow / per-tool layout and letting the existing zizmor job audit it; zizmor stays its own job here rather than a step in `scan` (already dogfooded standalone). The CI `mise-action` comment drops the template's "kept fresh by Renovate" note; the maintainer's `mise.toml` pin is Renovate-managed too (see "Renovate"), but the comment names only the single-source-of-truth property.

Because nothing here re-derives the pins (no Renovate; the generation drift test reads only the *rendered* downstream), **bump every literal site by hand, against the template.** gitleaks (`8.30.1`) has two maintainer sites — `mise.toml` and the prose above — synced to `template/mise.toml.jinja` (CI installs via `mise exec`, so there is no third gitleaks literal). semgrep (`1.167.0`) has three — the `just scan` recipe, the `scan` job in `test-template.yml`, and the prose above — synced to `template/justfile.jinja` and the template `scan.yml`. (Mirrors the pre-commit "bump both `rev:` pins together" obligation.)
Renovate re-derives only the `mise.toml` gitleaks pin (see "Renovate"); every other literal — the prose here, the template copies, and the uvx-run semgrep — is hand-synced (the generation drift test reads only the *rendered* downstream), so **bump every literal site by hand, against the template.** gitleaks (`8.30.1`) has two maintainer sites — `mise.toml` and the prose above — synced to `template/mise.toml.jinja` (CI installs via `mise exec`, so there is no third gitleaks literal); `just policy` asserts all three agree, so a Renovate gitleaks PR stays red until the template copy and this prose are bumped on the same branch. semgrep (`1.167.0`) has three — the `just scan` recipe, the `scan` job in `test-template.yml`, and the prose above — synced to `template/justfile.jinja` and the template `scan.yml`. (Mirrors the pre-commit "bump both `rev:` pins together" obligation.)

## Dependency audit

Expand All @@ -78,17 +78,25 @@ just audit # dependency vulnerability audit: pip-audit over the full locked gr

Deliberate divergences from the template's dependency-audit layer: `--no-dev` is dropped (above); pip-audit runs via `uvx pip-audit@2.10.1` in both the recipe and CI with **no** pyproject dep (the template adds `pip-audit>=2.10` to its dev group and runs `uv run pip-audit` locally); it is folded into `test-template.yml`'s `scan` job as a step (the template ships it in a standalone `scan.yml`); and, like the template, `audit` is chained into `just ci` (see "Run every gate") while additionally enforced in CI as the `pip-audit` step in the `scan` job.

Because nothing here re-derives the pin (no Renovate; the generation drift test reads only the *rendered* downstream), **bump every literal by hand, against the template.** pip-audit (`2.10.1`) has three maintainer sites — the `just audit` recipe, the `pip-audit` step in `test-template.yml`, and the prose above — synced to `template/{% if not in_existing_repo %}.github{% endif %}/workflows/…scan.yml….jinja` (the only exact-version template site; the template justfile uses unpinned `uv run pip-audit` and template pyproject floors `pip-audit>=2.10`). No `mise.toml` or `pyproject.toml` pip-audit literal exists (uvx-run, unlike gitleaks). **Sync only the pin *value* — never the export flags:** the template's `uv export` keeps `--no-dev`, but the maintainer must not (it exports 0 packages here — see above), so a mechanical sync against the template would silently neuter the gate. (Mirrors the semgrep/gitleaks pin-sync note and the pre-commit "bump both `rev:` pins together" rule.)
Because nothing here re-derives the pin (Renovate has no manager for `uvx` run-steps — see "Renovate" — and the generation drift test reads only the *rendered* downstream), **bump every literal by hand, against the template.** pip-audit (`2.10.1`) has three maintainer sites — the `just audit` recipe, the `pip-audit` step in `test-template.yml`, and the prose above — synced to `template/{% if not in_existing_repo %}.github{% endif %}/workflows/…scan.yml….jinja` (the only exact-version template site; the template justfile uses unpinned `uv run pip-audit` and template pyproject floors `pip-audit>=2.10`). No `mise.toml` or `pyproject.toml` pip-audit literal exists (uvx-run, unlike gitleaks). **Sync only the pin *value* — never the export flags:** the template's `uv export` keeps `--no-dev`, but the maintainer must not (it exports 0 packages here — see above), so a mechanical sync against the template would silently neuter the gate. (Mirrors the semgrep/gitleaks pin-sync note and the pre-commit "bump both `rev:` pins together" rule.)

## Renovate

Root `renovate.json` is the maintainer's freshness layer, consumed by the Mend Renovate GitHub App (operator-installed on the repository; the JSON is inert without it). Like the template layer it has no recipe and no CI gate. Scoped per `docs/superpowers/specs/2026-06-25-dependency-inventory-design.md`: `config:recommended` + `helpers:pinGitHubActionDigests`; weekly `lockFileMaintenance` — the load-bearing line, since every dev dep is a capped range and `pep621` opens PRs only on cap-crossing releases, so in-range refreshes of `uv.lock` come from it alone; the `pre-commit` manager **off** (the `rev:` SHA must move together with the template's copy, which no manager can see); and `astral-sh/uv` **disabled** (multi-site pin: `mise.toml` plus every `setup-uv` `version:` input — bump by hand across all sites).

Renovate sees only the maintainer's own files, never `template/*.jinja`. A gitleaks bump PR turns `just policy` red until `template/mise.toml.jinja` and the AGENTS.md literal are bumped on the same branch (a template change, so it gets a CHANGELOG entry). Action digest bumps land on the maintainer's workflows alone; the template's separately zizmor-audited pins may lag, which is accepted.

Deliberate divergences from the template's `renovate.json`: the `pre-commit` manager is off (above); the `uvx` regex `customManager` is omitted — the only maintainer `uvx` pins are parity-locked to the rendered template by the generation suite, so a one-sided bump would fail `just test`; and the `astral-sh/uv` rule is maintainer-only (the template ships no such rule; its rendered `uv` pin is equally multi-site, not addressed here).

## Policy gate (`just policy`)

```bash
just policy # pins the gate config literals so they cannot be silently weakened (stdlib-only)
```

`just policy` runs `uv run pytest tests/policy` — a stdlib-only (`re`/`tomllib`/`pathlib`) suite pinning this repo's gate **config literals** so they cannot be silently weakened: basedpyright `typeCheckingMode == "recommended"` **and** `failOnWarnings is True`; `"ALL" in ruff select`; the first `just <recipe>` of each **fenced** AGENTS.md example is a real justfile recipe (docs-can't-lie); and every third-party `uses:` is `@<40-hex> # v<ver>` (`test_actions_are_sha_pinned` — this **closes the offline half of the SHA-pin policy**, gap #8). It is a `ci` member (`ci: … test policy audit`) and is also auto-collected by bare `just test` (`uv run pytest`, `testpaths=["tests"]`), so CI enforces it inside the existing `test` job with **no** workflow change.
`just policy` runs `uv run pytest tests/policy` — a stdlib-only (`re`/`tomllib`/`pathlib`) suite pinning this repo's gate **config literals** so they cannot be silently weakened: basedpyright `typeCheckingMode == "recommended"` **and** `failOnWarnings is True`; `"ALL" in ruff select`; the first `just <recipe>` of each **fenced** AGENTS.md example is a real justfile recipe (docs-can't-lie); every third-party `uses:` is `@<40-hex> # v<ver>` (`test_actions_are_sha_pinned` — this **closes the offline half of the SHA-pin policy**, gap #8); and the `mise.toml` gitleaks pin equals `template/mise.toml.jinja`'s and the "Scanning" prose literal (`test_gitleaks_pin_matches_template` — the one Renovate-managed literal that must stay hand-synced; see "Renovate"). It is a `ci` member (`ci: … test policy audit`) and is also auto-collected by bare `just test` (`uv run pytest`, `testpaths=["tests"]`), so CI enforces it inside the existing `test` job with **no** workflow change.

Changing any of those gates (ruff select, type mode, `failOnWarnings`, an Action pin) requires editing the matching test in `tests/policy/test_gates.py` in the **same commit** — that is the point of the suite. **Scope:** it pins those config *literals* — not the ruff `ignore` / `per-file-ignores` lists (each is separately audited; see "Lint & format this repo"), not the recipe *bodies*, and not the CI workflow that runs the gates (candidate F — see "Run every gate").
Changing any of those gates (ruff select, type mode, `failOnWarnings`, an Action pin) requires editing the matching test in `tests/policy/test_gates.py` in the **same commit** — that is the point of the suite. (A gitleaks bump needs no test edit — bump all three sites in the same commit instead.) **Scope:** it pins those config *literals* — not the ruff `ignore` / `per-file-ignores` lists (each is separately audited; see "Lint & format this repo"), not the recipe *bodies*, and not the CI workflow that runs the gates (candidate F — see "Run every gate").

Deliberate divergences from `template/tests/…/test_gates.py.jinja`: the **coverage-floor** assertion is dropped (the maintainer has no `[tool.coverage]` / no runtime package — nothing to protect; the `policy` recipe likewise drops `--no-cov`, which is invalid with no `pytest-cov`); a **`failOnWarnings is True`** assertion is added (the template ships none — the maintainer pins the type-gate's teeth explicitly, so the suite guards that pin); and the **docs-can't-lie regex is scoped to fenced code blocks** rather than the whole file. The template's naive whole-doc scan still passes here *today*, but it flips to a hard fail the moment this file names a template-only recipe (`fuzz`/`mutate`) in prose — which gap #9 will do. Fenced-only gates the examples you are told to **run** and lets prose name any recipe freely.

Expand Down
2 changes: 2 additions & 0 deletions docs/superpowers/plans/2026-07-01-dogfood-gap-audit.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Dogfooding-gap audit — python-kickstarter (2026-07-01)

> **Status (2026-09-08).** Phase 1 landed: gap 1 (PR #6), gap 2 (#7), gap 5 (#8), gaps 7 + 3 (#9), gaps 4 + 8 (#10), gap 6 (`chore/dogfood-renovate`; the Mend app install is the operator step). **Gap 9 deferred:** copier's validators are prompt-time Jinja, not importable, so fuzzing them costs one copier render per example, and the TOML escaper already has round-trip tests. **Candidate F deferred:** the `scan` CI job bundles pip-audit + semgrep + gitleaks as three steps that do not map onto the `audit`/`scan` recipe split (see AGENTS.md "Run every gate"). Phases 2 and 3 not started; first Phase 2 layer is the CHANGELOG-`[Unreleased]` PR check.

## 1. Executive summary

python-kickstarter is a Copier template whose entire value proposition is the guardrail suite it forces on every downstream project — yet several of those gates never run on the maintainer repo that ships them. The substantive code gates are fine: ruff (`select=["ALL"]`), basedpyright (`recommended`, `failOnWarnings`), pytest (OS matrix), and zizmor (workflow audit) all already run on this tree, and mise pins the toolchain identically to what it ships. What is missing is the rest of the shipped stack: **pre-commit (the flagged priority — the only always-on, non-toggleable gate besides lint/type/test, and it runs in zero surfaces here)**, secret + SAST scanning (gitleaks/semgrep), the dependency-advisory audit, the policy "gates-can't-be-weakened" suite, Renovate, an aggregate `just ci`, plus the trivial `.editorconfig` and half of the SHA-pin policy. Nine gaps are confirmed (seven fully missing, two partial); only one (gitleaks/semgrep) is a genuine security hole, so most of the remainder is hygiene-and-credibility rather than correctness risk. The governing principle throughout: dogfooding should **mirror** what the template already ships, not invent a divergent maintainer-only variant.
Expand Down
13 changes: 13 additions & 0 deletions renovate.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended", "helpers:pinGitHubActionDigests"],
"lockFileMaintenance": { "enabled": true, "schedule": ["before 4am on monday"] },
"pre-commit": { "enabled": false },
"packageRules": [
{
"description": "uv is pinned in mise.toml AND every setup-uv version: input in test-template.yml, with no maintainer-side test asserting they agree. Renovate sees both (mise depName 'uv'; github-actions depName 'astral-sh/uv') but would bump them in separate PRs. Match on the shared packageName so BOTH are disabled — matchDepNames: ['uv'] would silence only the mise side. Bump uv manually across all six sites instead.",
"matchPackageNames": ["astral-sh/uv"],
"enabled": false
}
]
}
15 changes: 15 additions & 0 deletions tests/policy/test_gates.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,21 @@ def test_agents_md_recipes_exist_in_justfile() -> None:
assert referenced <= recipes, f"AGENTS.md names missing recipes: {referenced - recipes}"


def test_gitleaks_pin_matches_template() -> None:
"""The one Renovate-managed literal AGENTS.md requires hand-synced to the template.

Renovate bumps only the maintainer's mise.toml; it cannot see the template's
.jinja copy or the AGENTS.md prose, so a bump PR stays red until both follow.
"""
pin = re.compile(r'^gitleaks = "(\d+\.\d+\.\d+)"$', re.MULTILINE)
ours = pin.search((ROOT / "mise.toml").read_text())
theirs = pin.search((ROOT / "template" / "mise.toml.jinja").read_text())
assert ours is not None
assert theirs is not None
assert ours.group(1) == theirs.group(1)
assert f"gitleaks (`{ours.group(1)}`)" in (ROOT / "AGENTS.md").read_text()


def test_actions_are_sha_pinned() -> None:
"""Every third-party `uses:` is a 40-char SHA + a v-prefixed version comment."""
# SHA + a version comment (v<major>[.minor[.patch]]); our pins carry the action's
Expand Down