From 19298b0f32e4f314b6fae2f5e0c613b7b4c5ecc2 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Fri, 16 Oct 2026 03:26:09 +0200 Subject: [PATCH 01/41] :page_facing_up: docs(spec): re-measure developer-toolchain-pins and apply the pin-ownership decisions Re-measured against main after cross-file-consistency-gates (#463): govulncheck v1.9.0 at Taskfile.yml:204, gitleaks env at ci.yaml:164, golangci-lint v2.14.0, a second dead Renovate manager (shellcheck), and git-cliff publishing only per-asset .sha512 checksums. Pin ownership A: golangci-lint and govulncheck floors and the Makefile/.custom-gcl.yml agreement move out of this change; CFC-2 owns them. Guard shape C: the checks live in consistency_toolchain_pins_test.sh, enrolled by CFC-6, and dev_mise_pin_drift_test.sh is folded into it mutant for mutant. Every behaviour task now follows the task that can fail on it, and manager liveness is defined only after the probe. --- .../developer-toolchain-pins/proposal.md | 109 +++++++++++------ .../specs/developer-toolchain/spec.md | 100 ++++++++++----- .../changes/developer-toolchain-pins/tasks.md | 115 +++++++++++++++--- 3 files changed, 238 insertions(+), 86 deletions(-) diff --git a/openspec/changes/developer-toolchain-pins/proposal.md b/openspec/changes/developer-toolchain-pins/proposal.md index 0cf8378c..14f78d67 100644 --- a/openspec/changes/developer-toolchain-pins/proposal.md +++ b/openspec/changes/developer-toolchain-pins/proposal.md @@ -2,40 +2,62 @@ ## Why -The operator wants the same tool versions across their repositories. Probed on this tree: +The operator wants the same tool versions across their repositories. Re-measured on 2026-10-16 +against `main` at 966e17e10 (change 4, cross-file-consistency-gates, merged): | Tool | Now | Where it is stated | | --- | --- | --- | -| Task | 3.51.1 | 18 `version:` inputs (17 in `.github/workflows/*`, 1 in `.github/actions/kind-e2e-setup/action.yml`), plus `mise.toml` (`task = "3.51.1"`) | -| govulncheck | v1.1.4 | inline in `Taskfile.yml:199`, no variable, no Renovate annotation | -| gitleaks | 8.30.1 | `ci.yaml` env, `hack/install-gitleaks.sh:14`, `.pre-commit-config.yaml:5` | +| Task | 3.51.1 | 18 `version:` inputs (17 in `.github/workflows/*`, 1 in `.github/actions/kind-e2e-setup/action.yml`), plus `mise.toml:54` (`task = "3.51.1"`) | +| govulncheck | v1.9.0 | inline in `Taskfile.yml:204`, no variable, no Renovate annotation | +| gitleaks | 8.30.1 | `ci.yaml:164` env, `hack/install-gitleaks.sh:14`, `.pre-commit-config.yaml:5` | | git-cliff | v2.13.1 | `Taskfile.yml:22` (annotated for Renovate) | - -The Renovate side is partly dead: `renovate.json:40-46` matches a gitleaks `releases/download` URL -that `ci.yaml` does not contain (the real pins are the env at `ci.yaml:152`, -`hack/install-gitleaks.sh:14` and `.pre-commit-config.yaml:5`), nothing matches `Makefile:184` or -`hack/tooling/.custom-gcl.yml:6`, and Renovate's `pre-commit` manager is off by default. A manager -that matches nothing is silent and looks like "nothing to update". `mise.toml` carries a comment -naming helm v3.21.4 while `Taskfile.yml:31` pins v3.22.0. - -Two installers download a binary and verify no checksum: `hack/install-git-cliff.sh:39` says so, -and `hack/install-helm-docs.sh` has no checksum step. `changelog-sync.yaml` runs git-cliff in a job -that holds a ruleset-bypass token, so an unverified download there is a supply-chain hole that -the auto-merge change must not widen. +| golangci-lint | v2.14.0 | `Makefile:184`, `hack/tooling/.custom-gcl.yml:6` | + +The Renovate side is partly dead. Two custom managers match a `releases/download` URL that +`ci.yaml` does not contain: the gitleaks one (`renovate.json:37-47`; the real pins are the env at +`ci.yaml:164`, `hack/install-gitleaks.sh:14` and `.pre-commit-config.yaml:5`) and the shellcheck +one (`renovate.json:48-58`; CI pins shellcheck through the env at `ci.yaml:299`). Nothing matches +`Makefile:184` or `hack/tooling/.custom-gcl.yml:6`, and Renovate's `pre-commit` manager is off by +default. A manager that matches nothing is silent and looks like "nothing to update". + +`mise.toml` comments restate values that live elsewhere: helm v3.21.4 at `mise.toml:28` while +`Taskfile.yml:31` pins v3.22.0, git-cliff and helm-docs versions at `:24`, `go 1.27.2` at `:39`, +and "all 17 occurrences in .github/workflows/" at `:52`, a count that goes stale the first time a +workflow adds or drops a setup-task step. + +Two installers download a binary and verify no checksum: `hack/install-git-cliff.sh:40-43` says so, +and `hack/install-helm-docs.sh:40` points at that note. `hack/test/ci_fetch_lib_hardening_test.sh` +(~766) records both in `UNVERIFIED_BY_DESIGN`. `changelog-sync.yaml` runs git-cliff in a job that +holds a ruleset-bypass token, so an unverified download there is a supply-chain hole that the +auto-merge change must not widen. ## What Changes - Task to 3.52.0 at every site; govulncheck behind one Taskfile variable `GOVULNCHECK_VERSION` - (Renovate-annotated, at least v1.6.0, exact pin chosen at implementation); golangci-lint, - gitleaks and git-cliff kept in agreement at their floors. -- The drift test asserts that all sites of a tool agree AND are at least the floor, never exact + (Renovate-annotated, value unchanged unless a bot moves it); gitleaks kept in agreement at its + floor, git-cliff at its floor. +- **Pin ownership (decision A, 2026-10-16, pending operator validation):** golangci-lint and + govulncheck get no floor and no agreement check here. Change 4's CFC-2 + (`hack/test/consistency_tool_pins_test.sh`) already smoke-runs each analysis tool under + `go.mod`'s Go and checks `.custom-gcl.yml` against `Makefile`; it owns whether those pins are + good enough. This change only moves govulncheck into a variable and re-anchors CFC-2's mutants + that edited the inline literal. +- The drift checks assert that all sites of a tool agree AND are at least the floor, never exact values (bot bumps that move every site together stay green). -- Renovate managers: fix or replace the dead gitleaks one; add managers for `Makefile`, - `.custom-gcl.yml`, `.pre-commit-config.yaml` and `hack/install-gitleaks.sh`; a check fails when - a manager matches no file or a pin site is matched by none. -- `mise.toml`: Go stays out of `[tools]` (read from `go.mod` through - `idiomatic_version_file_enable_tools`), and its version comments shrink to pointers. -- `install-git-cliff.sh` and `install-helm-docs.sh` verify a published checksum. +- Renovate managers: replace the dead gitleaks and shellcheck managers with ones that match the + real pins; add managers for `Makefile`, `.custom-gcl.yml`, `.pre-commit-config.yaml` and + `hack/install-gitleaks.sh`; a check fails when a manager matches no file or a pin site is matched + by none. +- `mise.toml`: Go stays out of `[tools]` (already true and already checked); its comments stop + restating versions and site counts. +- `install-git-cliff.sh` verifies the per-asset `.sha512` with `sha512sum`; `install-helm-docs.sh` + verifies against `checksums.txt`. Both leave `UNVERIFIED_BY_DESIGN`, which only shrinks. +- **Guard shape (decision C, 2026-10-16, pending operator validation):** the checks live in a new + `hack/test/consistency_toolchain_pins_test.sh` (plain and `--self-test`). CFC-6 in + `hack/test/ci_workflow_security_test.sh` enrols every `consistency_*_test.sh` and requires both + modes in a required job, so the wiring is enforced without a self-wiring check. + `hack/test/dev_mise_pin_drift_test.sh` (DEV-MISE-01) is folded into it, mutant for mutant, and + deleted together with its `ci.yaml` step. ## Capabilities @@ -51,28 +73,39 @@ None. ## Impact - Entry points: `Taskfile.yml`, `mise.toml`, `.github/workflows/*`, `.github/actions/*`, - `renovate.json`, `hack/install-*.sh`, and the drift tests run in the `lint` job and `task check`. + `renovate.json`, `hack/install-git-cliff.sh`, `hack/install-helm-docs.sh`, the new guard run in + the `lint` job, and `task check`. +- Test changes on guards this change does not own: CFC-2's govulncheck mutants in + `hack/test/consistency_tool_pins_test.sh` (change 4), the checksum scan and + `UNVERIFIED_BY_DESIGN` in `hack/test/ci_fetch_lib_hardening_test.sh`, and the CWS-6 mutant at + `hack/test/ci_workflow_security_test.sh:1145` that names the deleted guard. ## Dependencies Landing order across the ten proposed changes: (1) ci-workflow-hardening, (2) task-check-entrypoint, (3) golangci-lint-bump, (4) cross-file-consistency-gates, (5) developer-toolchain-pins, (6) dependency-update-automation, (7) ci-failure-reporting, (8) test-depth-signals, -(9) mutation-testing-signal, (10) public-agent-contract. Needs 3 (the golangci-lint floor is the -bumped version). The installer checksums are a hard prerequisite of change 6. +(9) mutation-testing-signal, (10) public-agent-contract. + +Change 4 is merged (#463) and this branch starts from it. This change must not land before it: +both review legs raised that decision A leaves golangci-lint and govulncheck to CFC-2 and decision C +relies on CFC-6's enrolment, so without change 4 both tools would be unguarded and the new guard +unwired. Changes 2 and 3 are archived. The installer checksums are a hard prerequisite of change 6. ## Non-goals -- Choosing the dependency bot: an operator question recorded in the change that would move off - Renovate; nothing here depends on it except that the managers are Renovate regex managers. -- The Go version (change 4), the linter bump (change 3), the `task check` entry point (change 2). -- Tools not listed (helm, shellcheck, polaris, kubeaudit, kind keep their pins). +- The dependency bot: it is Renovate (operator ruling 2026-10-05, recorded in + `agent-context/decisions.md`); the managers here are Renovate regex managers. +- Floors or agreement for golangci-lint and govulncheck (CFC-2 owns them), the Go version + (change 4), the `task check` entry point (change 2). +- Tool versions not listed: helm, shellcheck, polaris, kubeaudit and kind keep their pins + (shellcheck only gets a live manager). ## Assumptions -- Task 3.52.0, golangci-lint v2.13.1, gitleaks 8.30.1 and govulncheck v1.6.0 exist (operator - baseline). A newer govulncheck may be chosen at implementation. -- Upstream publishes checksum files for git-cliff and helm-docs releases; probe in task 4.1 and - stop if one is missing. -- `renovate-config-validator --strict` and a local dry run can show which files a custom manager - matches. Probe in task 3.1. +- Task 3.52.0 and gitleaks 8.30.1 exist (operator baseline). +- Checksums probed 2026-10-16 for v2.13.1 and v1.14.2: git-cliff publishes + `.sha512` (HTTP 200) and no `.sha256` (HTTP 404); helm-docs publishes + `checksums.txt` (HTTP 200). Task 4.1 re-probes every platform the installers support. +- `renovate-config-validator --strict` and a local dry run can show which files and which + dependencies a custom manager matches. Probe in task 3.1. diff --git a/openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md b/openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md index 9b228db6..c36816c5 100644 --- a/openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md +++ b/openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md @@ -3,91 +3,118 @@ ## Purpose Defines the pinned tool floors, where each version lives, how repeated sites are kept equal, and -that downloaded tool binaries are verified. +that downloaded tool binaries are verified. Whether golangci-lint and govulncheck pins work with +the repository's Go is owned by change 4's CFC-2, not by this capability. ## ADDED Requirements ### Requirement: DTP-1 Task sites agree and meet the floor -Every `go-task/setup-task` `version:` input and `mise.toml` `task` SHALL be equal to each other and -at least `3.52.0`. +Every `go-task/setup-task` `version:` input under `.github/workflows/` and `.github/actions/`, and +`mise.toml` `task`, SHALL be equal to each other and at least `3.52.0`. #### Scenario: One site missed - **WHEN** all sites but one say the floor and one says `3.51.1` -- **THEN** the drift test SHALL fail and name the site +- **THEN** the guard SHALL fail and name the site #### Scenario: Whole set moves to a newer version - **WHEN** a bot PR moves every site to a newer release -- **THEN** the drift test passes +- **THEN** the guard passes #### Scenario: Whole set below the floor - **WHEN** every site says `3.51.1` -- **THEN** the drift test SHALL fail +- **THEN** the guard SHALL fail ### Requirement: DTP-2 govulncheck is pinned once -`Taskfile.yml` SHALL define `GOVULNCHECK_VERSION` (Renovate-annotated, at least `v1.6.0`) and -`task vulncheck` SHALL use it. +`Taskfile.yml` SHALL define `GOVULNCHECK_VERSION` with a Renovate annotation and `task vulncheck` +SHALL use it. No floor is set here: CFC-2 (change 4) checks that the pinned govulncheck builds and +runs under `go.mod`'s Go. #### Scenario: Inline version returns - **WHEN** `task vulncheck` contains a literal `govulncheck@v...` -- **THEN** the drift test SHALL fail +- **THEN** the guard SHALL fail -### Requirement: DTP-3 Other pins agree and meet their floors +#### Scenario: Annotation dropped + +- **WHEN** `GOVULNCHECK_VERSION` has no `# renovate:` line above it +- **THEN** the guard SHALL fail + +### Requirement: DTP-3 gitleaks and git-cliff meet their floors -golangci-lint (`Makefile`, `hack/tooling/.custom-gcl.yml`) SHALL be equal and at least `v2.13.1`; gitleaks (`hack/install-gitleaks.sh`, the CI environment, `.pre-commit-config.yaml`) SHALL be equal -and at least `8.30.1`; git-cliff (`Taskfile.yml`) SHALL be at least `v2.13.1`. +and at least `8.30.1`; git-cliff (`Taskfile.yml`) SHALL be at least `v2.13.1`. golangci-lint is not +checked here: CFC-2 checks `.custom-gcl.yml` against `Makefile` and smoke-runs the tool. #### Scenario: Pre-commit rev drifts - **WHEN** `.pre-commit-config.yaml` names a different gitleaks release than the others -- **THEN** the drift test SHALL fail +- **THEN** the guard SHALL fail #### Scenario: Empty scan - **WHEN** the scan for any tool finds no site -- **THEN** the drift test SHALL fail; an empty scan never passes +- **THEN** the guard SHALL fail; an empty scan never passes ### Requirement: DTP-4 The update bot can find every pin -Every `customManagers` entry in `renovate.json` SHALL match at least one tracked file, and every pin -site of DTP-1 to DTP-3 SHALL be covered by a manager of one group per tool. +Every `customManagers` entry in `renovate.json` SHALL match at least one tracked file, where +"matches" is the definition settled by the task 3.1 probe (a file pattern alone is not enough when +the match strings find nothing in that file), and every pin site of DTP-1 to DTP-3, plus +golangci-lint in `Makefile` and `hack/tooling/.custom-gcl.yml` and shellcheck in the CI +environment, SHALL be covered by a manager of one group per tool. #### Scenario: Dead manager -- **WHEN** a manager matches no file (as the gitleaks download-URL manager does today) -- **THEN** the test SHALL fail and name the manager +- **WHEN** a manager matches no file, as the gitleaks (`renovate.json:37-47`) and shellcheck + (`renovate.json:48-58`) download-URL managers do today +- **THEN** the guard SHALL fail and name the manager #### Scenario: Uncovered pin site - **WHEN** a tool's pin (for example `Makefile` `GOLANGCI_LINT_VERSION`) is matched by no manager -- **THEN** the test SHALL fail and name the file +- **THEN** the guard SHALL fail and name the file -### Requirement: DTP-5 mise reads Go from go.mod and states no stale versions +### Requirement: DTP-5 mise mirrors CI, reads Go from go.mod and states no stale versions `mise.toml` SHALL NOT list `go` under `[tools]` and SHALL enable `idiomatic_version_file_enable_tools -= ["go"]`; its comments and those in `Taskfile.yml` SHALL NOT restate a tool version other than -the pin they sit on. += ["go"]`, with `go.mod` carrying a `go` line; this half is already satisfied and checked by +DEV-MISE-01 and is carried over unchanged. `mise.toml` `node` and `python` SHALL equal the CI +`node-version` and `python-version` inputs, which SHALL agree with each other. Comments in +`mise.toml` and `Taskfile.yml` SHALL NOT restate a tool version other than the pin they sit on, nor +a count of pin sites. #### Scenario: Go listed in mise - **WHEN** `go` appears under `[tools]` -- **THEN** the drift test SHALL fail +- **THEN** the guard SHALL fail + +#### Scenario: Node drifts from CI -#### Scenario: Stale comment +- **WHEN** `mise.toml` says `node = "20"` while CI says `22` +- **THEN** the guard SHALL fail and name both values + +#### Scenario: Stale version comment - **WHEN** a comment says `helm (v3.21.4)` while `Taskfile.yml` pins another version -- **THEN** the test SHALL fail +- **THEN** the guard SHALL fail + +#### Scenario: Restated site count + +- **WHEN** a comment says "all 17 occurrences in .github/workflows/" +- **THEN** the guard SHALL fail ### Requirement: DTP-6 Downloaded tool binaries are checksum-verified -Every `hack/install-*.sh` that downloads a binary SHALL verify a published checksum before -installing it, or be listed in an exceptions file with a reason. +Every `hack/install-*.sh` that downloads a binary SHALL verify a published checksum before installing +it. git-cliff SHALL be verified against its per-asset `.sha512` with `sha512sum`; helm-docs against +its `checksums.txt`. The only exceptions list is `UNVERIFIED_BY_DESIGN` in +`hack/test/ci_fetch_lib_hardening_test.sh`; it SHALL only shrink, and this change removes both of +its entries. #### Scenario: Installer without verification @@ -104,12 +131,25 @@ installing it, or be listed in an exceptions file with a reason. - **WHEN** the checksum file cannot be fetched - **THEN** the installer SHALL fail; it never installs unverified +#### Scenario: Exception added + +- **WHEN** an installer name is added to `UNVERIFIED_BY_DESIGN` +- **THEN** the guard SHALL fail + ### Requirement: DTP-7 The guards can fail -The drift tests SHALL carry throwaway-copy mutants for each of DTP-1 to DTP-6 and a no-op copy -that passes, and SHALL run in a required job, plain and `--self-test` as separate steps. +The checks of DTP-1 to DTP-6 SHALL live in `hack/test/consistency_toolchain_pins_test.sh`, which +SHALL carry throwaway-copy mutants for each requirement, every mutant of the folded +`hack/test/dev_mise_pin_drift_test.sh`, and a no-op copy that passes. It SHALL run in a required +job, plain and `--self-test` as separate steps, which CFC-6 enforces for every +`consistency_*_test.sh`. #### Scenario: Mutant survives -- **WHEN** editing one site in a copy does not make the test fail +- **WHEN** editing one site in a copy does not make the guard fail - **THEN** the self-test SHALL fail + +#### Scenario: Guard unwired + +- **WHEN** either step of the new guard is removed from the required job +- **THEN** CFC-6 SHALL fail and name the guard diff --git a/openspec/changes/developer-toolchain-pins/tasks.md b/openspec/changes/developer-toolchain-pins/tasks.md index 5740f08f..987b4c6c 100644 --- a/openspec/changes/developer-toolchain-pins/tasks.md +++ b/openspec/changes/developer-toolchain-pins/tasks.md @@ -1,38 +1,117 @@ # Tasks -## 1. Guard first (DTP-1 to DTP-7) +Each behaviour task runs after the test or guard task that can fail on it. A check that is red on +today's tree is shown red in that task's evidence and lands in the same commit as the behaviour +task that turns it green, so `main` stays green at every commit. -- [ ] 1.1 Extend `hack/test/dev_mise_pin_drift_test.sh` (or add `dev_toolchain_pin_drift_test.sh`) with floors, agreement, empty-scan, inline-govulncheck, stale-comment and manager-liveness checks and throwaway-tree mutants; watch it fail on the 3.51.1 / inline govulncheck / dead gitleaks manager assertions +## 1. Guard first (DTP-1, DTP-2, DTP-3, DTP-5, DTP-6, DTP-7) -## 2. Pins (DTP-1 to DTP-3, DTP-5) +- [ ] 1.1 Write `hack/test/consistency_toolchain_pins_test.sh` (plain and `--self-test`) with every + check except manager liveness (that is 3.2), and add its two steps to the `lint` job. Self-test + cases on throwaway copies, each with the failure it exposes: + - DTP-1 given 18 setup-task sites and `mise.toml` at 3.52.0, when one site says 3.51.1, then red + naming that file (a missed site); when all say 3.51.1, then red (set below floor); when all say + a newer release, then green (exact-value check that would block bot bumps). + - DTP-2 given `GOVULNCHECK_VERSION` with a `# renovate:` line, when `task vulncheck` carries a + literal `govulncheck@v1.9.0`, then red (inline pin returns); when the annotation is deleted, + then red (Renovate stops seeing it). + - DTP-3 given gitleaks 8.30.1 at `ci.yaml`, `hack/install-gitleaks.sh` and + `.pre-commit-config.yaml`, when the pre-commit `rev` says v8.29.0, then red (a site drifts); + when all three say 8.29.0, then red (below floor); when `GIT_CLIFF_VERSION` says v2.12.0, then + red; when the gitleaks scan finds no site, then red (empty scan). + - DTP-5 when a `mise.toml` comment says `helm (v3.21.4)`, `go 1.27.2` or "all 17 occurrences", + then red (stale restated version or count); when a comment names the pin it sits on, then + green (false positive). + - DTP-6 when `UNVERIFIED_BY_DESIGN` in `hack/test/ci_fetch_lib_hardening_test.sh` gains an + entry, then red (the exceptions list grows). + - DTP-7 a no-op copy of the real tree passes (a guard that reds on correct content). + Run it on today's tree and record it red on the 18 + 1 Task 3.51.1 sites, the inline + govulncheck, and the `mise.toml:24`, `:28`, `:39` and `:52` comments. Show CFC-6 red when either + new step is removed from `ci.yaml`. +- [ ] 1.2 Fold `hack/test/dev_mise_pin_drift_test.sh`: port every existing inline mutant one-to-one + into `--self-test`, show each red against the new guard, list the mapping in evidence, then + delete the old guard and its `ci.yaml` step. The mutants to port (old label -> expected + failure): `honest` (green), `task-drift` (mise task 3.50.0 vs CI), `node-drift` (node 20 vs 22), + `python-drift` (python 3.11 vs 3.12), `go-pinned` (`go` under `[tools]`), `go-setting-gone` + (`idiomatic_version_file_enable_tools` removed), `gomod-gone` (no `go` line in `go.mod`), + `ci-self-drift` (workflow and composite action disagree on task), `no-ci-pins` (empty CI scan), + `empty-tools` (empty `[tools]`), `no-tools-table` (no `[tools]` table), and the missing + `mise.toml` case. Its self-wiring check (`grep` for the step in `ci.yaml`) is replaced by CFC-6 + enrolment, shown red in 1.1. Delete `ci.yaml:411-419` (comment and step). Update every live + reference to the old name: `renovate.json:74` description, `mise.toml:8`, `.gitignore:26`, + `docs/development/setup.md:53`, the CWS-6 composite-action mutant at + `hack/test/ci_workflow_security_test.sh:1145-1146` (retarget it to the new guard and show it + still reds), and the comments at `hack/test/ci_fetch_lib_hardening_test.sh:844` and + `hack/test/ci_install_helm_hardening_test.sh:687`. Change 4's evidence and review files keep the + old name as history. -- [ ] 2.1 Task at every site including `mise.toml` -- [ ] 2.2 `GOVULNCHECK_VERSION` variable with Renovate annotation; `task vulncheck` uses it -- [ ] 2.3 Replace the stale helm comment in `mise.toml` with a pointer; confirm Go absent from `[tools]` +## 2. Pins (DTP-1, DTP-2, DTP-5) + +- [ ] 2.1 Task 3.52.0 at all 18 setup-task sites and `mise.toml:54`; 1.1's DTP-1 check turns green. +- [ ] 2.2 Test change on a guard owned by change 4, first: run + `bash hack/test/consistency_tool_pins_test.sh --self-test` green, then move govulncheck to + `GOVULNCHECK_VERSION` (Renovate-annotated, value v1.9.0) and show the CFC-2 mutants that `sed` + the literal `govulncheck@v…` in `Taskfile.yml` (~499-501 "unresolvable pin", ~523-525 "too old + for go.mod's Go") fail to apply. Re-anchor both onto the `GOVULNCHECK_VERSION:` line, and confirm + the two that match `govulncheck@` without a version (~503-505 "package path typo", ~527-529 "pin + missing") still apply and red for their stated reason. Confirm the plain run resolves + `{{.GOVULNCHECK_VERSION}}` (the variable lookup at ~162) rather than reporting + ``. Record a real `task vulncheck` run. 1.1's DTP-2 check turns green. +- [ ] 2.3 Replace the restated versions and count in `mise.toml:24`, `:28`, `:39` and `:52` with + pointers to the owning file; 1.1's DTP-5 check turns green. The Go half of DTP-5 needs no + change: it is the existing DEV-MISE-01 check, carried over by 1.2. ## 3. Managers (DTP-4) -- [ ] 3.1 Probe which files each custom manager matches (validator plus dry run) -- [ ] 3.2 Fix or replace the dead gitleaks manager; add managers for `Makefile`, `hack/tooling/.custom-gcl.yml`, `.pre-commit-config.yaml`, `hack/install-gitleaks.sh`; liveness guard passes +- [ ] 3.1 Probe which files and dependencies each custom manager matches + (`renovate-config-validator --strict` plus a local dry run). Settle and record what "a manager + matches a file" means: at least one `managerFilePatterns` hit on a tracked file AND at least one + `matchStrings` dependency extracted from it. The gitleaks and shellcheck managers pass the first + half and fail the second, which is why a file-pattern-only check would call them alive. +- [ ] 3.2 Add the manager-liveness and site-coverage checks to the guard, using 3.1's definition, + with self-test cases: given today's `renovate.json`, then red naming the gitleaks + (`renovate.json:37-47`) and shellcheck (`renovate.json:48-58`) managers (dead manager); given a + manager whose file pattern matches no tracked file, then red (pattern rot); given no manager + covering `Makefile` `GOLANGCI_LINT_VERSION`, `hack/tooling/.custom-gcl.yml`, + `.pre-commit-config.yaml`, `hack/install-gitleaks.sh`, `ci.yaml:164` or `ci.yaml:299`, then red + naming the file (uncovered site); given two golangci-lint sites in different groups, then red + (sites can move apart). Record it red on today's tree. +- [ ] 3.3 Replace the dead gitleaks manager with one that matches `ci.yaml:164`, + `hack/install-gitleaks.sh:14` and `.pre-commit-config.yaml:5`; replace the dead shellcheck + manager with one that matches `ci.yaml:299`; add managers for `Makefile:184` and + `hack/tooling/.custom-gcl.yml:6` in one group. Re-run 3.1's probe; 3.2's checks turn green. ## 4. Installers (DTP-6) -- [ ] 4.1 Probe the upstream checksum files for git-cliff and helm-docs -- [ ] 4.2 Write a test with a tampered tarball and a missing checksum for each installer (fake `fetch_to`); watch it fail; add verification through `hack/lib/fetch.sh` and update `hack/test/ci_fetch_lib_hardening_test.sh`, which records the git-cliff gap +- [ ] 4.1 Probe the upstream checksum files for every platform each installer supports: git-cliff + per-asset `.sha512` (the `.sha256` is 404), helm-docs `checksums.txt`. Stop if one is + missing. +- [ ] 4.2 Tests first, with a fake `fetch_to`, per installer: given a tarball whose digest differs + from the published one, then the installer exits non-zero and the output path does not exist + (tampered download installed); given a checksum fetch that fails, then the same (silent + fallback to unverified). Widen the checksum scan in `hack/test/ci_fetch_lib_hardening_test.sh` + (`verify_sha256|sha256sum`, ~779 and ~793) to accept `sha512sum`, with a mutant that deletes the + `sha512sum` call from a copy of `install-git-cliff.sh` and must red once the entry leaves + `UNVERIFIED_BY_DESIGN`. Watch all red. +- [ ] 4.3 Verify git-cliff with `sha512sum` against `.sha512` and helm-docs against + `checksums.txt` through `hack/lib/fetch.sh`; remove the NOTE at `install-git-cliff.sh:40-42` and + `install-helm-docs.sh:40`; remove both entries from `UNVERIFIED_BY_DESIGN` (~766). 4.2's tests + and the stale-exemption check turn green. ## 5. Land -- [ ] 5.1 Archive the change as the last commit of the PR, after review and green CI; the review record names the reviewed and the archive revision +- [ ] 5.1 Archive the change as the last commit of the PR, after review and green CI; the review + record names the reviewed and the archive revision ## Verification | Req | Check | Expected | Status | Evidence | | --- | --- | --- | --- | --- | -| DTP-1 | drift test: one-site and below-floor mutants red; newer set passes | as stated | not-run | | -| DTP-2 | inline-govulncheck mutant; `task vulncheck` real run | red; run result recorded | not-run | | -| DTP-3 | pre-commit-rev and empty-scan mutants | red | not-run | | -| DTP-4 | dead-manager and uncovered-site fixtures | red | not-run | | -| DTP-5 | go-in-tools and stale-comment mutants | red | not-run | | -| DTP-6 | tampered and missing-checksum cases per installer | installer fails, nothing installed | not-run | | -| DTP-7 | mutants plus no-op, by exit status | mutants red, no-op green | not-run | | +| DTP-1 | guard: one-site and below-floor mutants red; newer set passes | as stated | not-run | | +| DTP-2 | inline-govulncheck and dropped-annotation mutants; CFC-2 self-test after re-anchoring; `task vulncheck` real run | red; CFC-2 green; run result recorded | not-run | | +| DTP-3 | pre-commit-rev, below-floor and empty-scan mutants | red | not-run | | +| DTP-4 | dead-manager, rotted-pattern, uncovered-site and split-group fixtures | red | not-run | | +| DTP-5 | ported go/node/python/task mutants; stale-version and site-count comment mutants | red | not-run | | +| DTP-6 | tampered and missing-checksum cases per installer; grown-exceptions mutant | installer fails, nothing installed; red | not-run | | +| DTP-7 | all mutants plus no-op by exit status; old-to-new mutant mapping; CFC-6 with a step removed | mutants red, no-op green, mapping complete, CFC-6 red | not-run | | | all | independent review; CI on the PR head | APPROVE, green; both revisions recorded | not-run | | From 5490677b02fa3ae0dfc76d43945df12282d835c4 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Fri, 16 Oct 2026 03:27:18 +0200 Subject: [PATCH 02/41] :page_facing_up: docs(spec): start the developer-toolchain-pins loop --- .../changes/developer-toolchain-pins/loop.md | 94 +++++++++++++++++++ .../developer-toolchain-pins/task-prompt.md | 55 +++++++++++ 2 files changed, 149 insertions(+) create mode 100644 openspec/changes/developer-toolchain-pins/loop.md create mode 100644 openspec/changes/developer-toolchain-pins/task-prompt.md diff --git a/openspec/changes/developer-toolchain-pins/loop.md b/openspec/changes/developer-toolchain-pins/loop.md new file mode 100644 index 00000000..dd292fe6 --- /dev/null +++ b/openspec/changes/developer-toolchain-pins/loop.md @@ -0,0 +1,94 @@ +# Loop — developer-toolchain-pins (spec-loop-duo run state) + +Repo: `/home/koni/Projects/PlatformRelay/worktrees/kollect/developer-toolchain-pins` · Branch: `feat/developer-toolchain-pins` · Base: `origin/main` @ `966e17e10` (change 4 merged) · Started: 2026-10-16 · Reviewers: astra + claude (duo) +Next: R — spec-set review (astra:spec + claude/opus:spec) on 649e5811c +Budget: Claude review legs uncapped (operator goal 2026-10-16) · run under GOAL-2026-10-16-announcement-ready (decisions by model alignment, logged in agent-context/decisions.md) + +Change 5 of 10 (issue #442). Spec set: proposal.md, specs/developer-toolchain/spec.md, tasks.md. + +## Stages + +- [x] 0 orient — re-measured against change 4's tip (2026-10-16): 6 mechanical edits + 2 model-aligned decisions +- [x] P plan + tasks — present; spec set updated 649e5811c (re-measure, decisions A pin ownership and C guard shape) +- [ ] R spec-set review +- [ ] L task loop — see *Tasks* +- [ ] B branch review +- [ ] Hand-off + +## Fitness functions + +| Characteristic | Command | Kind | Baseline | Per task / at B | +|---|---|---|---|---| +| CI guard meta-tests | `bash hack/test/_test.sh` and `--self-test` | triggered | green | per task | +| CI wiring (CFC-6) | `hack/test/ci_workflow_security_test.sh` both modes | triggered | green | per task touching ci.yaml or guards | +| Shell hygiene | `task lint:shell` | triggered | no warnings | per task | +| Markdown | `task lint:markdown` (tracked files only) | triggered | 0 issues | per task + every orchestrator state edit | +| Tool pins (CFC-2) | `hack/test/consistency_tool_pins_test.sh` | triggered | green | task 2.2 | +| OpenSpec strict | `task spec:validate` | triggered | green | per spec-set edit | +| Evidence is Markdown | `hack/test/openspec_evidence_markdown_only_test.sh` | triggered | green | per task | +| Full local gate | `task check` | holistic | green on base | at B | + +## Triage + +| Stage | Finding (one line) | Sev | Models | Disposition | Where it went / why | Needs user? | +|---|---|---|---|---|---|---| +| 0 | DTP-2/DTP-3 floors + golangci agreement duplicate CFC-2 | decision | astra+opus | fix (A) | dropped; CFC-2 owns analysis-tool pins | logged, pending validation | +| 0 | guard shape: extend dev_mise vs new consistency_* guard | decision | astra+opus | fix (C) | new consistency_toolchain_pins_test.sh; dev_mise folded (task 1.2) | logged, pending validation | + +## Tasks + +| Task | Verdict | Review (legs, rounds, Claude?) | Gaps / decision request | +|---|---|---|---| + +## Known red + +| Test (file:name) | Story | Written in | Cleared in | +|---|---|---|---| + +## Test changes + +| Test (file:name) | Written in | Changed in | Evidence it was wrong | +|---|---|---|---| + +## Lessons + +Carried over from change 4 (cross-file-consistency-gates): + +- `task lint:markdown` lints only git-tracked files: a probe file needs `git add -N` or it is skipped with exit 0 (G8) +- Orchestrator: run `task lint:markdown` on every state-file edit before committing (B2 regression from 800a33e69) +- A variable image tag can contain `/` and `:` and name another repo: fail on any `$` in an image, never parse around it (G7) +- The Bash tool runs zsh here: `${PIPESTATUS[0]}` is empty — use `set -o pipefail` or bash -c (G7) +- A wiring rule must take its required modes from the spec, not from the guard's own code; anchor sed mutants on stable prefixes, not exact config lines (G6) +- Sensors: match whole tokens of the target grammar (Prometheus `[A-Za-z0-9_:]`), and make a parser fail on leftover input it does not recognise (G5) +- The block-ai-mentions hook checks the whole Bash command string: keep model names in evidence out of any command that also runs `git commit` (G5) +- Validate numbers as bounded + canonical before bash `(( ))`: inside `if` an arithmetic error is false and big values wrap → false green in a fail-closed sensor (G4) +- Under pipefail, `grep -v | grep -q` can SIGPIPE and read as no-match; use `grep >/dev/null` (G4) +- A fresh worktree lacks bin/operator-sdk: run `make operator-sdk` before the dist_* guards (G4) +- RBAC coverage must respect role kind and scope: never pool a namespaced Role with a ClusterRole; a Role never covers a cluster-scoped resource or subresource (G3) +- An exception class must be a closed list, or "any other X fails" is undone by one line (G3) +- A negative mutant must pass every check except the clause it names; one gate mutant per clause proves it; a file-deleting mutant needs an absence probe (G3) +- Write variable rm targets as "${R:?}"/... — the safety hook refuses `rm -f $R/*` (G3) +- Check a spec's exception list against the product docs' "by design" sections before cutting tasks (G3) +- `helm template --show-only` errors when the template renders nothing (clusterrole.yaml in tenant mode): render all, filter on `# Source:` (G3) +- `python3 -I` loses user-site PyYAML on this host; use plain `python3` as the repo guards do (G3) +- Wait on PIDs (`wait $pid`, `while kill -0 $pid`), NEVER `pgrep -f ` where the pattern appears in your own command line — it matches itself and never ends (G2, ~80 min lost) +- A bash helper called as `$(helper)` runs in a subshell and loses global state: register cleanup/temp roots in the parent shell (G2) +- `go mod download` under an exact GOTOOLCHAIN refuses a module needing a newer Go: parse "requires go >=" (G2) +- Guard smoke builds must build each pin at its own version and assert the built version; never reuse bin/ (G2) +- Clean up temp dirs in an EXIT trap; a leaking guard left 298 dirs in /tmp (G2) +- A Go minor bump breaks every pinned tool embedding golang.org/x/tools (golangci-lint, go-arch-lint, govulncheck at 1.27.2); 2.1 must cover all `go run`/`go install` tools; re-run `task lint` after each tool fix — the next broken step only shows once the earlier passes (G1) +- Run `task test` with `umask 022` on this host; umask 002 reds two internal/sink/git mirror tests at base too (G1) +- mise shims for gitleaks/jq fail in some dirs: use ~/.local/share/mise/installs/gitleaks/8.30.1/gitleaks; keep mise.toml out of scratch copies (G1) +- Renovate: the LAST matching rule wins, so a group rule must be the last groupName-setting rule; prove grouping with a local `renovate --platform=local` lookup dry-run (~1 min) (G1) + +## Where a human should look first + +## Proposed harness changes + +| Lesson | Seen in | Proposed change | +|---|---|---| + +## Owner tasks (skipped by the loop) + +| Task | Command sheet | +|---|---| diff --git a/openspec/changes/developer-toolchain-pins/task-prompt.md b/openspec/changes/developer-toolchain-pins/task-prompt.md new file mode 100644 index 00000000..dba0374f --- /dev/null +++ b/openspec/changes/developer-toolchain-pins/task-prompt.md @@ -0,0 +1,55 @@ +You are closing ONE Spec Kit task in a spec-loop run. State lives in files only; you carry nothing in from any earlier session. + +Repo: /home/koni/Projects/PlatformRelay/worktrees/kollect/developer-toolchain-pins (branch `feat/developer-toolchain-pins`, must be clean when you start; check with `git status`). +Feature directory: openspec/changes/developer-toolchain-pins (OpenSpec change: proposal.md, specs/*/spec.md = the spec, tasks.md; no plan.md/constitution — repo rules are AGENTS.md and docs/development/spec-workflow.md). +Records directory for review registers: openspec/changes/developer-toolchain-pins + +Do this: invoke the `close-task` skill (Skill tool, name `close-task`, args `__TID__`) and follow it end to end for task __TID__. Its helper scripts are at ~/.claude/skills/close-task/scripts/task.sh and sensors.sh; run `task.sh show __TID__` from the repo root first and read only the slice it gives you plus the ids it cites. + +Per-task review (this replaces the reviewer dispatch in close-task's REVIEW step). Record the pre-task sha before you change anything. Size the review to the diff: + +- evidence or documentation only → `--legs "GLM-5.3:diff"`; +- anything else → `--legs "GLM-5.3:diff,DeepSeek-V4.1-Flash:diff"`; +- append `,claude/opus:diff` (with `--claude-budget 3`) ONLY when Claude review legs left for this task (see dispatch note) is above 0 and: the free register says BLOCK; fewer than two free legs produced a verdict (`legs.log` shows `exit=truncated` or non-zero) on a non-doc diff; or the diff touches secrets handling, hieradata, Terraform, manifests, CI or a protected path. +Command: + ~/.claude/skills/fanout-review/scripts/fanout.sh --repo /home/koni/Projects/PlatformRelay/worktrees/kollect/developer-toolchain-pins --target "task __TID__: ; verification matrix in openspec/changes/developer-toolchain-pins/evidence/__TID__.md; diff ..HEAD" --legs "" --out openspec/changes/developer-toolchain-pins/reviews/L-tasks/__TID__[/round-N] +Without opencode, the orchestrator names the legs instead (e.g. `codex exec --sandbox read-only - < > leg-codex-diff.md` plus `fanout.sh --legs "claude/sonnet:diff" --no-unify`); write the brief with the Write tool and pass it on stdin. +Read only register.md; verify every CRITICAL and every single-model WARNING against the files yourself before acting; map CLEAN (or CONCERNS whose findings did not survive verification) → APPROVE, a verified finding → REQUEST_CHANGES, BLOCK → BLOCK. Record every gate run (command, sha, exit, result) in the evidence BEFORE writing the review brief, so reviewers do not spend round 1 on missing records. A second round happens only if round 1 found a verified BEHAVIOUR defect (evidence or wording findings: fix, no new round); it goes in a fresh `round-2/` directory and is the last: after it, fix only verified behavioural defects (no further review), record everything else as nits or gaps, each rejected with a reason or deferred to a named place, and close. If fanout.sh exits non-zero or register.md has no `## Unified verdict` line, do not wait: read the leg files yourself (each has its own `## Verdict`), say so in the state file, and continue. If fanout.sh cannot run at all, say so in the state file and use a fresh general-purpose subagent with ~/.claude/skills/close-task/reference/review-brief.md instead. Say in the state file which legs ran and whether a Claude leg ran and why. + +Fitness functions: the loop state file (openspec/changes/developer-toolchain-pins/loop.md, section *Fitness functions*) lists the repository's architectural fitness functions; the rows marked per-task run in your `task` tier on the final tree. If your diff adds an import edge between components, a dependency, a package or component, a function over the complexity threshold, or touches determinism-sensitive code, either the existing functions pass unchanged or you extend one in this task and say why in the evidence. Never widen an allow-list, exclusion list or coverage baseline to get green: that is an architecture decision, report BLOCKED with the decision request. + +Test budget: order the steps as focused tests, the fast check, the mutation gate, the cold review, fixes, the mutation gate on the fix lines only, then ONE full suite on the final code tree (red, or a fix beyond wording, goes back through one review round) (review findings otherwise force a second full run: two ~7-minute runs on anvil 017 P2); after an evidence- or docs-only commit, the fast check only. Do not create or edit files in the repo while the full suite runs (untracked-file guards fail it). Mutation proof through the repository's changed-lines mutation gate when it has one; hand-written mutants only where it cannot reach, as a representative sample — and then restore from the FIRST saved copy, drop stale `__pycache__`, and judge only the target test's own failure line. If your change touches code that only a deselected test tier or a scheduled CI job runs (golden, e2e, live markers), run that tier locally before closing; if it cannot run here, close CLOSED-WITH-GAPS naming the tier (the orchestrator dispatches CI for it). + +Shell: if the session is worktree-isolated, keep each Bash command plain — no loops over hosts, no `$(…)` or variables feeding git/gh/codex, no heredoc whose text mentions git, no long jq/python filters inline; write scripts and briefs with the Write tool and run them as files. Refused commands cost a round trip each. + +Lessons from earlier tasks: see loop.md *Lessons* (render at dispatch) + +If a trigger fires but no Claude review legs are left, run no Claude leg and close CLOSED-WITH-GAPS naming the review you skipped. + +Test tasks (the task text says it writes tests, or sits under "Tests for User Story N"): follow close-task's *Test tasks* rule. Write the cases the task names from spec.md and the task text, not from plan.md's design rationale. Close only on a verified red: each new test fails for its stated reason, command and failure line in the evidence; list them under `## Known red`. A compile error or missing symbol is not that red: a stub with the signature plan.md fixes (body: not implemented) is in scope; report BLOCKED only if plan.md does not fix the signature. A test that already passes is a finding against the test unless you established that the outcome already holds; then say so and close. Never bypass a commit hook that refuses red tests: report BLOCKED. +Known red for this feature (excluded from "green" at close, nothing else is): see loop.md *Known red* +Implementation tasks: tests written by an earlier test task of this feature are fixed points. Change one only through close-task's TEST class with evidence citing spec.md or the constitution (a spec-versus-plan conflict is SPECIFICATION: BLOCKED), list each change under `## Test changes` in the evidence, and name it in your report. Report every known-red test you turned green. + +Task kinds: if the task text says "Owner session" or "Owner gate", or needs a host, account, push, pull request or decision you do not have, do NOT attempt that part. Whole task → stop at SELECT and report `OWNER` with the command sheet you would hand the owner written to openspec/changes/developer-toolchain-pins/evidence/__TID__.md. Part → do the agent part, write the owner part as a command sheet in the same file, close CLOSED-WITH-GAPS listing it. + +Hard rules: + +- Never read, print, decrypt or copy a secret value; reference entries and secret names only. No .env, kubeconfig or token files; Renovate/App secrets are referenced by name only. +- Do not push, add or change any git remote, or open a pull request. Local commits on `feat/developer-toolchain-pins` only, one logical commit including the state file, Conventional Commit style with the repo convention `:gitmoji: (): ` (ASCII shortcode first, e.g. `:white_check_mark: test(ci): ...`, `:wrench: build: ...`, `:arrow_up: chore(deps): ...`); NO Co-Authored-By or any agent/model/tool attribution anywhere in the message. +- Do not edit CLAUDE.md, AGENTS.md, the constitution, .specify/ or .claude/; propose instead. +- Terse, plain English. Tag every claim in evidence as verified (you ran or read it, with date) or believed. +- Evidence: the close-task state file is openspec/changes/developer-toolchain-pins/evidence/__TID__.md; if the task names another evidence file, write that too and link it. No key, token, kubeconfig, whole log or address that is not needed. +- Commit the review `register.md` and `leg-*.md` files with the task when records live in the repo; never the `*.err` transcripts. +- Tick the task in tasks.md exactly as close-task says. + +Final report, at most 20 lines: task · verdict (CLOSED / CLOSED-WITH-GAPS / BLOCKED / OWNER) · matrix summary · sensors run · corrections · review verdict, legs and register path, Claude legs used · findings rejected or deferred (one line each: finding · reject/defer · reason or destination) · known red added or cleared · test changes · learnings (class, plus each reusable lesson in one line, for the next task's prompt) · commit sha · next action or decision request. Nothing else. + +Repo specifics (kollect): + +- REVIEW SIZING (spec-loop-duo, operator choice 2026-10-15; replaces every sizing instruction above): ONE leg per task: astra = `codex exec --sandbox read-only - < > /leg-astra-diff.md` (run from the repo root; brief written with the Write tool from ~/.claude/skills/close-task/reference/review-brief.md plus the diff range). Add a second leg `~/.claude/skills/fanout-review/scripts/fanout.sh --repo --target "..." --legs "claude/opus:diff" --no-unify --out ` ONLY if the diff touches secrets, CI (.github/ workflows or the CI wiring), infra or manifests, or security posture, or astra says BLOCK. Never pass an opencode model to fanout.sh; always --no-unify; read the leg files' `## Verdict` yourself. Codex quota error → claude/sonnet stands in for astra and say so in the evidence; codex "at capacity" → stand-in for this round only. A second round only for a verified BEHAVIOUR defect; at most two. +- Task ids in tasks.md are `N.M` numbers; you close the whole group named in the dispatch, ticking each line. One logical commit per sub-task where the task says "its own commit" (e.g. the go.mod bump is version-only, real drift fixes are separate). +- Guards follow `hack/test/dev_mise_pin_drift_test.sh` and `hack/test/ci_workflow_security_test.sh`: plain mode checks the real tree, `--self-test` corrupts a throwaway copy (mktemp) per mutant and asserts non-zero, plus a no-op copy that must pass. Run `task lint:shell` on new scripts. +- `task check` runs every hack/test guard by glob; do not run it per task (it is the stage-B holistic row). Do run the guards you touched, both modes. +- Gotchas: helm-unittest `contains` compares whole elements; "no tests to run" exits 0; `format:check`/`docs:verify` can be red on a clean main in a working copy (verify with stash before blaming your change). +- Nothing private ships: before committing run `git diff --cached | rg -i 'rewe|mkurator|titiseven|prod165|atlassian'` (must be empty) and `gitleaks protect --staged`. +- Evidence file per group: /evidence/G.md; review records in /reviews/L-tasks/G/. From d92e187ff45227a897dbf25bc064a7e2bfcf8d4c Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Fri, 16 Oct 2026 03:31:57 +0200 Subject: [PATCH 03/41] :page_facing_up: docs(spec): record the spec-set review of developer-toolchain-pins --- .../reviews/R-spec-set/leg-astra-spec.md | 46 +++++++++++++++++++ .../R-spec-set/leg-claude-opus-spec.md | 36 +++++++++++++++ .../reviews/R-spec-set/verification.md | 28 +++++++++++ 3 files changed, 110 insertions(+) create mode 100644 openspec/changes/developer-toolchain-pins/reviews/R-spec-set/leg-astra-spec.md create mode 100644 openspec/changes/developer-toolchain-pins/reviews/R-spec-set/leg-claude-opus-spec.md create mode 100644 openspec/changes/developer-toolchain-pins/reviews/R-spec-set/verification.md diff --git a/openspec/changes/developer-toolchain-pins/reviews/R-spec-set/leg-astra-spec.md b/openspec/changes/developer-toolchain-pins/reviews/R-spec-set/leg-astra-spec.md new file mode 100644 index 00000000..a6b84f98 --- /dev/null +++ b/openspec/changes/developer-toolchain-pins/reviews/R-spec-set/leg-astra-spec.md @@ -0,0 +1,46 @@ +Verdict: REQUEST CHANGES + +### DTP-R1 · WARNING · Installer tests lack a success control and an explicit execution home + +- Where: `openspec/changes/developer-toolchain-pins/tasks.md:89-99`; `openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md:141-145`; `hack/install-git-cliff.sh:43-45`; `hack/install-helm-docs.sh:41-43`; `hack/test/ci_fetch_lib_hardening_test.sh:778-782`. +- Evidence: Task 4.2 specifies only a mismatched digest and a failed checksum fetch, both expecting failure. It names the existing fetch guard for the static scan but gives no file or invocation for the installer execution tests. DTP-7 says the DTP-6 checks “SHALL live in `hack/test/consistency_toolchain_pins_test.sh`.” The existing checksum scan merely counts `verify_sha256|sha256sum` tokens. +- Why it matters: An installer that always fails—for example, because checksum verification uses the upstream asset filename while the downloaded archive is named `git-cliff.tgz`—passes both negative cases. The tasks also leave unclear how the behavioral tests become persistent required checks. +- Proposed fix: Add to task 4.2: “Put both installers’ executable fixture tests in the new guard’s self-test path. For each installer, first prove that a valid archive and matching published-format checksum install the expected executable successfully. Derive the tampered and missing-checksum cases from that passing fixture, assert the intended diagnostic and absent output, and prove that removing verification makes the tampered case fail its assertion.” Explicitly state whether the new guard implements or delegates the static installer scan. + +### DTP-R2 · WARNING · The comment rule has no defined attachment or matching grammar + +- Where: `openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md:88-89`; `openspec/changes/developer-toolchain-pins/tasks.md:22-24`; `mise.toml:51-63`. +- Evidence: The requirement exempts “the pin they sit on,” and the positive fixture says only “when a comment names the pin it sits on, then green.” Existing comments include `node-version: "22"` at `mise.toml:57`, followed by a continuation comment before `node = "22"` at line 59; Python similarly has a two-line comment block. +- Why it matters: Neither adjacency, tool identity nor value equality is defined. A checker could reject the existing Node comment because it is not immediately adjacent, or accept stale text merely because it is near a pin. Taskfile comments are also in scope, but the specified comment mutants exercise only mise. +- Proposed fix: Replace the exception with: “A version-bearing comment is permitted only in the contiguous comment block immediately preceding, or inline with, its owning pin; it must identify that tool and equal the pin’s normalized value. Other tool-version restatements and numeric pin-site counts are forbidden.” Define recognized tool/version and count forms. Add positive fixtures for the existing Node/Python blocks and negative fixtures for a mismatched attached version, an unrelated tool version, and a Taskfile comment. + +### DTP-R3 · WARNING · DTP-2 tests do not prove that vulncheck uses the named variable + +- Where: `openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md:33-35`; `openspec/changes/developer-toolchain-pins/tasks.md:15-17`; `Taskfile.yml:204`; `hack/test/consistency_tool_pins_test.sh:159-162`. +- Evidence: DTP-2 requires that `task vulncheck` “SHALL use” `GOVULNCHECK_VERSION`. Task 1.1 tests an inline literal and a missing annotation only. CFC-2 resolves any Taskfile variable matching `{{.([A-Za-z0-9_]+)}}`; it does not enforce this variable’s identity. +- Why it matters: Leaving an annotated `GOVULNCHECK_VERSION` unused while running govulncheck through another variable containing the same release violates DTP-2, yet neither prescribed mutant exposes that failure. CFC-2 correctly does not own this structural requirement. +- Proposed fix: Add to task 1.1: “With an annotated `GOVULNCHECK_VERSION` still present, change the vulncheck command to reference another variable containing the same version; require failure naming the missing `GOVULNCHECK_VERSION` reference. Also remove the variable definition and require a specific missing-definition failure.” + +### DTP-R4 · WARNING · Single-step removal normally trips CWS-6 before CFC-6 + +- Where: `openspec/changes/developer-toolchain-pins/tasks.md:29-30`; `openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md:152-155`; `hack/test/ci_workflow_security_test.sh:586-608`. +- Evidence: The task says, “Show CFC-6 red when either new step is removed.” The existing guard first iterates scripts still invoked by CI and fails with `CWS-6: ... mode is not invoked by any step of a required job` at line 590. CFC-6’s enrolment loop follows at line 600. +- Why it matters: With one new step remaining and both modes recognized, the earlier CWS-6 check terminates execution. Requiring a CFC-6 diagnostic would make correct wiring protection appear to fail the acceptance test and could prompt an unnecessary change to change 4. +- Proposed fix: Change the scenario and task to require “the workflow-security guard fails, naming the guard and missing mode.” Add a separate fixture deleting **both** steps and require the CFC-6 enrolment diagnostic. Keep decision C unchanged. + +The old-name reference inventory in task 1.2 is complete, including hidden files and the old guard’s own self-wiring references. The CFC-2 mutant re-anchoring plan matches the current implementation. No task explicitly requires provisioning operator accounts, secrets or a GitHub App. + +Verified file:line claims: + +- `mise.toml:8`, `:24`, `:28`, `:39`, `:52`, `:54`. +- `Taskfile.yml:22`, `:31`, `:204`. +- `.github/workflows/ci.yaml:164`, `:299`, `:411-419`. +- `renovate.json:37-47`, `:48-58`, `:74`. +- `Makefile:184`; `hack/tooling/.custom-gcl.yml:6`; `.pre-commit-config.yaml:5`; `hack/install-gitleaks.sh:14`. +- `hack/install-git-cliff.sh:40-43`; `hack/install-helm-docs.sh:40`. +- `hack/test/ci_fetch_lib_hardening_test.sh:766`, `:779`, `:793`, `:844`. +- `hack/test/ci_install_helm_hardening_test.sh:687`. +- `hack/test/ci_workflow_security_test.sh:1145-1146`. +- `hack/test/consistency_tool_pins_test.sh:159-162`, `:499-505`, `:523-529`. +- `.gitignore:26`; `docs/development/setup.md:53`. +- The **18 setup-task sites** count: 17 workflow inputs and one composite-action input, all currently `3.51.1`, plus `mise.toml:54`. diff --git a/openspec/changes/developer-toolchain-pins/reviews/R-spec-set/leg-claude-opus-spec.md b/openspec/changes/developer-toolchain-pins/reviews/R-spec-set/leg-claude-opus-spec.md new file mode 100644 index 00000000..03d07d07 --- /dev/null +++ b/openspec/changes/developer-toolchain-pins/reviews/R-spec-set/leg-claude-opus-spec.md @@ -0,0 +1,36 @@ +## Verdict: CONCERNS + +Requirement map, checked against the tree. DTP-1 holds: I counted 18 setup-task sites (17 workflows plus `kind-e2e-setup/action.yml:45`) and `mise.toml:54`. DTP-2 holds: the CFC-2 mutants are at `hack/test/consistency_tool_pins_test.sh:499-529` and the variable lookup at `:159-162`. DTP-3 holds. DTP-4 is partial. DTP-5 holds, and 1.2's list of references to the old guard matches a repo-wide grep. DTP-6 and DTP-7 are partial. Overlap with change 4 is clean: CFC-6 picks up the new guard through its `consistency_*_test.sh` glob (`hack/test/ci_workflow_security_test.sh:600-611`), and nothing re-checks golangci-lint or govulncheck floors. + +## Findings +- [WARNING] Task 3.3 turns on a shellcheck manager whose bumps can never pass CI — `hack/install-shellcheck.sh:42-68` + Failure: Renovate bumps `ci.yaml:299` to 0.11.1. The installer only has pinned digests for 0.11.0 and exits "no pinned sha256", so `lint` goes red. All custom.regex deps share the "pinned build tools" group (`renovate.json:127-128`), so that one red PR also holds back Task, gitleaks, golangci-lint and git-cliff. The default at `install-shellcheck.sh:15` and the pre-commit `shellcheck-py` rev (`.pre-commit-config.yaml:56`) are not in DTP-4's site list either, so they drift. + Fix: put shellcheck in its own group, and make the bump update the digest table (or fail the guard when the table has no entry for the pinned version). Add the default and the pre-commit rev to DTP-4's sites. + Confidence: 80 +- [WARNING] The commit-ordering rule cannot hold for 1.1 — `openspec/changes/developer-toolchain-pins/tasks.md:3-5,28-29` + Failure: 1.1 adds both lint steps and is red on today's tree for three separate later tasks (2.1, 2.2, 2.3). "Lands in the same commit as the behaviour task" then means either a red commit on `main` (each commit lands under rebase-merge) or one commit holding four logical changes. + Fix: state that each requirement's check lands with its 2.x task, with the steps wired in the first of them. + Confidence: 70 +- [WARNING] DTP-7's "checks of DTP-1 to DTP-6 SHALL live in" the new guard conflicts with how DTP-6 is built — `spec.md:141`, `tasks.md:89-95` + Failure: the "installer without verification" scenario is enforced by `ci_fetch_lib_hardening_test.sh:778-782`, not by the new guard. 4.2 never says which file the tampered and missing-checksum tests go in. + Fix: change DTP-7 to name the fetch-lib gate as DTP-6's home, or move the checks. Name the file for 4.2's tests. + Confidence: 75 +- [WARNING] The "only shrinks" rule for `UNVERIFIED_BY_DESIGN` has no ratchet after 4.3 — `tasks.md:25-26,96-99` + Failure: 1.1's grown-list check has to accept today's two entries, since they are not in 1.1's expected reds. 4.3 empties the list but never tightens that baseline. Later, someone can strip git-cliff's verification and re-add it to the list: the new guard passes, and the stale-exemption check (`ci_fetch_lib_hardening_test.sh:789-796`) passes too. + Fix: have 4.3 set the guard's baseline to empty, so any entry turns it red. + Confidence: 65 +- [WARNING] DTP-4's checks have no stated implementation — `tasks.md:66-78` + Failure: "matchStrings extracts a dep" and "sites in different groups" need either Renovate itself in `lint` (node, network, unpinned) or a bash copy of JS regex and packageRules order (a later groupName wins). A bash copy drifts silently, for example on `\n` in `renovate.json:80`. + Fix: make 3.1 record which way the guard goes, and pin the tool if it runs Renovate. + Confidence: 60 +- [NOTE] The govulncheck annotation's datasource is unspecified. A wrong one (e.g. github-releases instead of `go`/`golang.org/x/vuln`) still "extracts", so DTP-4 passes while lookups fail — `tasks.md:53`. Fix: name the datasource in 2.2. Confidence: 55 +- [NOTE] Any new per-tool group rule must sit before the "go toolchain" rule, which CFC-1 requires to stay last (`renovate.json:142`). Tasks 3.3 does not say so, though CFC-1's guard would catch it. Confidence: 50 +- [NOTE] DTP-6 hard-codes `sha512sum`, but `install-git-cliff.sh:22` supports Darwin, where macOS does not ship it by default. The repo's fallback pattern is at `hack/lab/report.sh:174-178`. The existing `verify-sha256.sh:21` has the same gap. Confidence: 50 +- [NOTE] `task-prompt.md:51` points later tasks at `dev_mise_pin_drift_test.sh` as the template, but 1.2 deletes it. Confidence: 85 +- [NOTE] The spec relies on CFC-2 and CFC-6, which exist only in the unarchived `openspec/changes/cross-file-consistency-gates/`. `openspec/specs/` has no cross-file-consistency capability yet. Confidence: 60 + +## Could not check +- Whether Task 3.52.0 exists, and whether the git-cliff `.sha512` and helm-docs `checksums.txt` files exist upstream for every platform (no network). +- That `lint` is in the required set: `elig_names`' source file is not at the path I tried. I inferred it from the CFC-6 mutants that target `jobs.lint`. +- How Renovate actually behaves (extraction, grouping); I only read `renovate.json`. +- The other reviewers' outputs in `reviews/R-spec-set/`, which I deliberately did not read. diff --git a/openspec/changes/developer-toolchain-pins/reviews/R-spec-set/verification.md b/openspec/changes/developer-toolchain-pins/reviews/R-spec-set/verification.md new file mode 100644 index 00000000..c2fdcaa4 --- /dev/null +++ b/openspec/changes/developer-toolchain-pins/reviews/R-spec-set/verification.md @@ -0,0 +1,28 @@ +# R spec-set review: verification + +Reviewed revision: 9621c98aa (spec set from 649e5811c). Legs: astra spec (REQUEST CHANGES), +claude spec (CONCERNS). Every WARNING below was checked against the tree by the orchestrator. + +| ID | Sev | Legs | Verified | Finding | Disposition | +| --- | --- | --- | --- | --- | --- | +| R1 | WARNING | claude | yes | A live shellcheck manager bumps `ci.yaml:299` only; `hack/install-shellcheck.sh:42-68` has digests for 0.11.0 alone, so every bump is red and holds the shared "pinned build tools" group; `install-shellcheck.sh:15` and `.pre-commit-config.yaml:56` drift | fix | +| R2 | WARNING | claude | yes | Header rule "red check lands with the behaviour task" cannot hold for 1.1, which is red for 2.1, 2.2 and 2.3 | fix | +| R3 | WARNING | claude, astra | yes | DTP-7 puts DTP-6's checks in the new guard, but the installer scan lives in `ci_fetch_lib_hardening_test.sh:778-782` and 4.2 names no home for the tamper tests | fix | +| R4 | WARNING | claude | yes | The "only shrinks" ceiling for `UNVERIFIED_BY_DESIGN` is never tightened to empty in 4.3 | fix | +| R5 | WARNING | claude | yes | DTP-4 extraction and grouping checks have no stated mechanism (Renovate in lint vs a bash copy of JS regex and packageRules order) | fix | +| R6 | WARNING | astra | yes | 4.2 has no success control: an installer that always fails passes both negative cases | fix | +| R7 | WARNING | astra | yes | DTP-5 "the pin they sit on" has no attachment or matching grammar | fix | +| R8 | WARNING | astra | yes | Removing one step trips CWS-6 at `ci_workflow_security_test.sh:590` (`fail` exits) before CFC-6 at :608 | fix | +| R9 | WARNING | astra | yes | No mutant proves `task vulncheck` references `GOVULNCHECK_VERSION` rather than another variable | fix | +| R10 | WARNING | orchestrator | yes | git-cliff has three pins: `Taskfile.yml:22`, `release.yaml:420` (git-cliff-action `version:`), `hack/verify-changelog.sh:11`; DTP-3/DTP-4 cover only the first | fix | +| R11 | WARNING | orchestrator | yes | `docs/development/setup.md:39` restates Task 3.51.1; 2.1 leaves it stale | fix | +| R12 | NOTE | claude | yes | govulncheck annotation datasource unspecified | fix | +| R13 | NOTE | claude | yes | A new group rule must precede the "go toolchain" rule (`renovate.json:142`) | fix | +| R14 | NOTE | claude | yes | `sha512sum` is absent on stock macOS; `verify-sha256.sh:21` has the same gap | defer | +| R15 | NOTE | claude | yes | `task-prompt.md:51` names the guard 1.2 deletes as the template | fix | +| R16 | NOTE | claude | yes | CFC-2/CFC-6 live only in the unarchived change 4 | reject | +| R17 | NOTE | orchestrator | yes | gitleaks sites differ in `v` prefix (`.pre-commit-config.yaml:5`); 3.1 should prove each form is rewritten correctly | fix | + +All other file:line claims in the spec set were re-checked and hold, including the 18 setup-task +sites, the CFC-2 mutant lines and the complete list of `dev_mise_pin_drift` references in 1.2. No +task needs an operator account, a secret or a GitHub App. From 42f0581ec3d4385532df6a3d4bbb568cfa777148 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Fri, 16 Oct 2026 03:37:43 +0200 Subject: [PATCH 04/41] :page_facing_up: docs(spec): apply the spec-set review to developer-toolchain-pins Fixes R1-R13, R15 and R17 from reviews/R-spec-set/verification.md: shellcheck gets its own Renovate group and a three-site agreement check with a digest-table case, git-cliff gains its release.yaml and verify-changelog.sh sites, each red check moves into the task that turns it green, DTP-5 defines comment attachment, DTP-7 splits the unwired scenario into CWS-6 and CFC-6 cases, and the installer tests gain a valid-download control and a ratchet to an empty exception list. --- .../developer-toolchain-pins/proposal.md | 43 ++-- .../specs/developer-toolchain/spec.md | 112 ++++++++-- .../developer-toolchain-pins/task-prompt.md | 2 +- .../changes/developer-toolchain-pins/tasks.md | 207 ++++++++++++------ 4 files changed, 256 insertions(+), 108 deletions(-) diff --git a/openspec/changes/developer-toolchain-pins/proposal.md b/openspec/changes/developer-toolchain-pins/proposal.md index 14f78d67..2240bc5e 100644 --- a/openspec/changes/developer-toolchain-pins/proposal.md +++ b/openspec/changes/developer-toolchain-pins/proposal.md @@ -10,15 +10,20 @@ against `main` at 966e17e10 (change 4, cross-file-consistency-gates, merged): | Task | 3.51.1 | 18 `version:` inputs (17 in `.github/workflows/*`, 1 in `.github/actions/kind-e2e-setup/action.yml`), plus `mise.toml:54` (`task = "3.51.1"`) | | govulncheck | v1.9.0 | inline in `Taskfile.yml:204`, no variable, no Renovate annotation | | gitleaks | 8.30.1 | `ci.yaml:164` env, `hack/install-gitleaks.sh:14`, `.pre-commit-config.yaml:5` | -| git-cliff | v2.13.1 | `Taskfile.yml:22` (annotated for Renovate) | +| git-cliff | v2.13.1 | `Taskfile.yml:22` (annotated for Renovate), `release.yaml:420` (git-cliff-action `version:`), `hack/verify-changelog.sh:11` (fallback install) | | golangci-lint | v2.14.0 | `Makefile:184`, `hack/tooling/.custom-gcl.yml:6` | +| shellcheck | 0.11.0 | `ci.yaml:299` env, `hack/install-shellcheck.sh:15` default (digest table at `:42-60` holds 0.11.0 only), `.pre-commit-config.yaml:56` (shellcheck-py `v0.11.0.1`) | The Renovate side is partly dead. Two custom managers match a `releases/download` URL that `ci.yaml` does not contain: the gitleaks one (`renovate.json:37-47`; the real pins are the env at `ci.yaml:164`, `hack/install-gitleaks.sh:14` and `.pre-commit-config.yaml:5`) and the shellcheck one (`renovate.json:48-58`; CI pins shellcheck through the env at `ci.yaml:299`). Nothing matches -`Makefile:184` or `hack/tooling/.custom-gcl.yml:6`, and Renovate's `pre-commit` manager is off by -default. A manager that matches nothing is silent and looks like "nothing to update". +`Makefile:184`, `hack/tooling/.custom-gcl.yml:6`, `release.yaml:420` or `hack/verify-changelog.sh:11`, +and Renovate's `pre-commit` manager is off by default. A manager that matches nothing is silent and +looks like "nothing to update". Every `custom.regex` dependency shares the "pinned build tools" +group (`renovate.json:126-129`), so a live shellcheck manager that bumps `ci.yaml:299` alone would +open a PR that is red (`install-shellcheck.sh` has no digest for the new version) and holds back +every other tool in that group. `mise.toml` comments restate values that live elsewhere: helm v3.21.4 at `mise.toml:28` while `Taskfile.yml:31` pins v3.22.0, git-cliff and helm-docs versions at `:24`, `go 1.27.2` at `:39`, @@ -34,8 +39,10 @@ auto-merge change must not widen. ## What Changes - Task to 3.52.0 at every site; govulncheck behind one Taskfile variable `GOVULNCHECK_VERSION` - (Renovate-annotated, value unchanged unless a bot moves it); gitleaks kept in agreement at its - floor, git-cliff at its floor. + (Renovate-annotated on the `go` datasource, value unchanged unless a bot moves it); gitleaks and + git-cliff kept in agreement across their three sites each, at their floors; shellcheck's three + sites kept in agreement, with the guard red when the installer's digest table lacks the pinned + version. - **Pin ownership (decision A, 2026-10-16, pending operator validation):** golangci-lint and govulncheck get no floor and no agreement check here. Change 4's CFC-2 (`hack/test/consistency_tool_pins_test.sh`) already smoke-runs each analysis tool under @@ -45,17 +52,26 @@ auto-merge change must not widen. - The drift checks assert that all sites of a tool agree AND are at least the floor, never exact values (bot bumps that move every site together stay green). - Renovate managers: replace the dead gitleaks and shellcheck managers with ones that match the - real pins; add managers for `Makefile`, `.custom-gcl.yml`, `.pre-commit-config.yaml` and - `hack/install-gitleaks.sh`; a check fails when a manager matches no file or a pin site is matched - by none. + real pins; add managers for `Makefile`, `.custom-gcl.yml`, `.pre-commit-config.yaml`, + `hack/install-gitleaks.sh`, `hack/install-shellcheck.sh`, the `release.yaml` git-cliff input and + `hack/verify-changelog.sh`; shellcheck gets a group of its own, so its digest-gated bumps hold + back no other tool; a check fails when a manager matches no file, a pin site is matched by none, + or one tool's sites land in different groups. +- git-cliff's two extra sites stay literal and join the agreement check rather than reading the + Taskfile variable: `release.yaml:420` is an action input that cannot read a Taskfile variable, so + a three-site agreement check and a manager are needed anyway, and `verify-changelog.sh` also runs + outside `task`, where reading the variable would need a Taskfile parse in bash. - `mise.toml`: Go stays out of `[tools]` (already true and already checked); its comments stop restating versions and site counts. - `install-git-cliff.sh` verifies the per-asset `.sha512` with `sha512sum`; `install-helm-docs.sh` - verifies against `checksums.txt`. Both leave `UNVERIFIED_BY_DESIGN`, which only shrinks. + verifies against `checksums.txt`. Both leave `UNVERIFIED_BY_DESIGN`, which only shrinks; once it + is empty, the new guard reds on any entry. - **Guard shape (decision C, 2026-10-16, pending operator validation):** the checks live in a new `hack/test/consistency_toolchain_pins_test.sh` (plain and `--self-test`). CFC-6 in `hack/test/ci_workflow_security_test.sh` enrols every `consistency_*_test.sh` and requires both - modes in a required job, so the wiring is enforced without a self-wiring check. + modes in a required job, so the wiring is enforced without a self-wiring check. The static + installer-checksum scan stays in `hack/test/ci_fetch_lib_hardening_test.sh`, which owns it; the + installer execution tests go in the new guard's `--self-test`. `hack/test/dev_mise_pin_drift_test.sh` (DEV-MISE-01) is folded into it, mutant for mutant, and deleted together with its `ci.yaml` step. @@ -73,8 +89,9 @@ None. ## Impact - Entry points: `Taskfile.yml`, `mise.toml`, `.github/workflows/*`, `.github/actions/*`, - `renovate.json`, `hack/install-git-cliff.sh`, `hack/install-helm-docs.sh`, the new guard run in - the `lint` job, and `task check`. + `renovate.json`, `hack/install-git-cliff.sh`, `hack/install-helm-docs.sh`, + `hack/verify-changelog.sh`, `docs/development/setup.md`, the new guard run in the `lint` job, and + `task check`. - Test changes on guards this change does not own: CFC-2's govulncheck mutants in `hack/test/consistency_tool_pins_test.sh` (change 4), the checksum scan and `UNVERIFIED_BY_DESIGN` in `hack/test/ci_fetch_lib_hardening_test.sh`, and the CWS-6 mutant at @@ -99,7 +116,7 @@ unwired. Changes 2 and 3 are archived. The installer checksums are a hard prereq - Floors or agreement for golangci-lint and govulncheck (CFC-2 owns them), the Go version (change 4), the `task check` entry point (change 2). - Tool versions not listed: helm, shellcheck, polaris, kubeaudit and kind keep their pins - (shellcheck only gets a live manager). + (shellcheck gets a live manager, its own group and a site-agreement check, but no floor). ## Assumptions diff --git a/openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md b/openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md index c36816c5..4376e1f4 100644 --- a/openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md +++ b/openspec/changes/developer-toolchain-pins/specs/developer-toolchain/spec.md @@ -30,8 +30,8 @@ Every `go-task/setup-task` `version:` input under `.github/workflows/` and `.git ### Requirement: DTP-2 govulncheck is pinned once -`Taskfile.yml` SHALL define `GOVULNCHECK_VERSION` with a Renovate annotation and `task vulncheck` -SHALL use it. No floor is set here: CFC-2 (change 4) checks that the pinned govulncheck builds and +`Taskfile.yml` SHALL define `GOVULNCHECK_VERSION` with a Renovate annotation on the Go module +datasource (`datasource=go depName=golang.org/x/vuln`) and `task vulncheck` SHALL use it. No floor is set here: CFC-2 (change 4) checks that the pinned govulncheck builds and runs under `go.mod`'s Go. #### Scenario: Inline version returns @@ -44,17 +44,46 @@ runs under `go.mod`'s Go. - **WHEN** `GOVULNCHECK_VERSION` has no `# renovate:` line above it - **THEN** the guard SHALL fail -### Requirement: DTP-3 gitleaks and git-cliff meet their floors +#### Scenario: vulncheck reads another variable -gitleaks (`hack/install-gitleaks.sh`, the CI environment, `.pre-commit-config.yaml`) SHALL be equal -and at least `8.30.1`; git-cliff (`Taskfile.yml`) SHALL be at least `v2.13.1`. golangci-lint is not -checked here: CFC-2 checks `.custom-gcl.yml` against `Makefile` and smoke-runs the tool. +- **WHEN** `task vulncheck` references a different variable holding the same version while the + annotated `GOVULNCHECK_VERSION` stays defined +- **THEN** the guard SHALL fail and name the missing `GOVULNCHECK_VERSION` reference + +#### Scenario: Variable definition removed + +- **WHEN** `GOVULNCHECK_VERSION` is no longer defined +- **THEN** the guard SHALL fail with a missing-definition message + +### Requirement: DTP-3 gitleaks, git-cliff and shellcheck sites agree + +gitleaks (`ci.yaml:164` env, `hack/install-gitleaks.sh:14`, `.pre-commit-config.yaml:5` `rev`) +SHALL be equal and at least `8.30.1`. git-cliff (`Taskfile.yml:22` `GIT_CLIFF_VERSION`, +`.github/workflows/release.yaml:420` git-cliff-action `version:`, `hack/verify-changelog.sh:11` +fallback install) SHALL be equal and at least `v2.13.1`. shellcheck (`ci.yaml:299` env, +`hack/install-shellcheck.sh:15` default, `.pre-commit-config.yaml:56` shellcheck-py `rev`, compared +on its first three components) SHALL be equal, and `hack/install-shellcheck.sh`'s digest table SHALL +have an entry for the pinned version on every platform it supports. Versions compare with any +leading `v` stripped. golangci-lint is not checked here: CFC-2 checks `.custom-gcl.yml` against +`Makefile` and smoke-runs the tool. #### Scenario: Pre-commit rev drifts - **WHEN** `.pre-commit-config.yaml` names a different gitleaks release than the others - **THEN** the guard SHALL fail +#### Scenario: git-cliff release pin drifts + +- **WHEN** `release.yaml` or `hack/verify-changelog.sh` names a different git-cliff release than + `Taskfile.yml` +- **THEN** the guard SHALL fail and name the file + +#### Scenario: shellcheck bumped without a digest + +- **WHEN** every shellcheck site says a release that `hack/install-shellcheck.sh`'s digest table + has no entry for +- **THEN** the guard SHALL fail and name the missing table entry + #### Scenario: Empty scan - **WHEN** the scan for any tool finds no site @@ -64,9 +93,13 @@ checked here: CFC-2 checks `.custom-gcl.yml` against `Makefile` and smoke-runs t Every `customManagers` entry in `renovate.json` SHALL match at least one tracked file, where "matches" is the definition settled by the task 3.1 probe (a file pattern alone is not enough when -the match strings find nothing in that file), and every pin site of DTP-1 to DTP-3, plus -golangci-lint in `Makefile` and `hack/tooling/.custom-gcl.yml` and shellcheck in the CI -environment, SHALL be covered by a manager of one group per tool. +the match strings find nothing in that file), and every pin site of DTP-1 to DTP-3 (including the +three git-cliff and three shellcheck sites), plus golangci-lint in `Makefile` and +`hack/tooling/.custom-gcl.yml`, SHALL be covered by a manager. All sites of one tool SHALL land in +one group, and shellcheck SHALL have a group of its own, so a shellcheck bump that is red until its +digests are added holds back no other tool. The check's mechanism (Renovate run in `lint` at a +pinned version, or a bash copy of the managers' regexes and `packageRules` order) is the one task +3.1 records. #### Scenario: Dead manager @@ -79,14 +112,24 @@ environment, SHALL be covered by a manager of one group per tool. - **WHEN** a tool's pin (for example `Makefile` `GOLANGCI_LINT_VERSION`) is matched by no manager - **THEN** the guard SHALL fail and name the file +#### Scenario: shellcheck shares a group + +- **WHEN** a shellcheck site resolves to the same group as another tool +- **THEN** the guard SHALL fail and name the group + ### Requirement: DTP-5 mise mirrors CI, reads Go from go.mod and states no stale versions `mise.toml` SHALL NOT list `go` under `[tools]` and SHALL enable `idiomatic_version_file_enable_tools = ["go"]`, with `go.mod` carrying a `go` line; this half is already satisfied and checked by DEV-MISE-01 and is carried over unchanged. `mise.toml` `node` and `python` SHALL equal the CI -`node-version` and `python-version` inputs, which SHALL agree with each other. Comments in -`mise.toml` and `Taskfile.yml` SHALL NOT restate a tool version other than the pin they sit on, nor -a count of pin sites. +`node-version` and `python-version` inputs, which SHALL agree with each other. + +A version-bearing comment in `mise.toml` or `Taskfile.yml` is allowed only in the contiguous +comment block directly above its pin (no blank line between), or inline on the pin's line, and only +when it names that pin's tool (the pin key or its CI input name, such as `node-version` for `node` +or setup-task `version:` for `task`) at a value equal to the pin's, with any leading `v` stripped. +Any other tool-version restatement in a comment, and any count of pin sites (a number followed by +`occurrences`, `sites`, `places` or `workflows`), SHALL fail the guard. #### Scenario: Go listed in mise @@ -103,6 +146,17 @@ a count of pin sites. - **WHEN** a comment says `helm (v3.21.4)` while `Taskfile.yml` pins another version - **THEN** the guard SHALL fail +#### Scenario: Attached comment names another value + +- **WHEN** the comment block directly above `task = "3.52.0"` says `version: 3.51.1` +- **THEN** the guard SHALL fail + +#### Scenario: Attached comment names the pin + +- **WHEN** the comment block directly above `node = "22"` says `node-version: "22"`, or the one + above `python = "3.12"` says `python-version: "3.12"` +- **THEN** the guard passes + #### Scenario: Restated site count - **WHEN** a comment says "all 17 occurrences in .github/workflows/" @@ -113,13 +167,18 @@ a count of pin sites. Every `hack/install-*.sh` that downloads a binary SHALL verify a published checksum before installing it. git-cliff SHALL be verified against its per-asset `.sha512` with `sha512sum`; helm-docs against its `checksums.txt`. The only exceptions list is `UNVERIFIED_BY_DESIGN` in -`hack/test/ci_fetch_lib_hardening_test.sh`; it SHALL only shrink, and this change removes both of -its entries. +`hack/test/ci_fetch_lib_hardening_test.sh`; it SHALL only shrink, this change removes both of its +entries, and once they are gone any entry SHALL fail the guard. #### Scenario: Installer without verification - **WHEN** `install-git-cliff.sh` downloads a tarball and does not compare a checksum -- **THEN** the guard SHALL fail and name the script +- **THEN** `hack/test/ci_fetch_lib_hardening_test.sh` SHALL fail and name the script + +#### Scenario: Valid download + +- **WHEN** the downloaded archive matches the published checksum +- **THEN** the installer SHALL succeed and the expected executable SHALL exist at the output path #### Scenario: Tampered download @@ -133,23 +192,32 @@ its entries. #### Scenario: Exception added -- **WHEN** an installer name is added to `UNVERIFIED_BY_DESIGN` +- **WHEN** an installer name is added to `UNVERIFIED_BY_DESIGN`, including after the list is empty - **THEN** the guard SHALL fail ### Requirement: DTP-7 The guards can fail -The checks of DTP-1 to DTP-6 SHALL live in `hack/test/consistency_toolchain_pins_test.sh`, which -SHALL carry throwaway-copy mutants for each requirement, every mutant of the folded -`hack/test/dev_mise_pin_drift_test.sh`, and a no-op copy that passes. It SHALL run in a required -job, plain and `--self-test` as separate steps, which CFC-6 enforces for every -`consistency_*_test.sh`. +The checks of DTP-1 to DTP-6 SHALL live in `hack/test/consistency_toolchain_pins_test.sh`, with +one exception: the static installer-checksum scan of DTP-6 stays in +`hack/test/ci_fetch_lib_hardening_test.sh`, which owns it and which this change widens to accept +`sha512sum`. The new guard SHALL carry throwaway-copy mutants for each requirement, every mutant of +the folded `hack/test/dev_mise_pin_drift_test.sh`, the installer execution tests of DTP-6 (valid, +tampered and missing-checksum downloads) in its `--self-test`, and a no-op copy that passes. It SHALL +run in a required job, plain and `--self-test` as separate steps, which CWS-6 and CFC-6 in +`hack/test/ci_workflow_security_test.sh` enforce. #### Scenario: Mutant survives - **WHEN** editing one site in a copy does not make the guard fail - **THEN** the self-test SHALL fail +#### Scenario: One step unwired + +- **WHEN** one of the new guard's two steps is removed from the required job +- **THEN** `hack/test/ci_workflow_security_test.sh` SHALL fail, naming the guard and the mode that + no required step runs (CWS-6 fires first, at `ci_workflow_security_test.sh:590`) + #### Scenario: Guard unwired -- **WHEN** either step of the new guard is removed from the required job +- **WHEN** both steps of the new guard are removed from the required job - **THEN** CFC-6 SHALL fail and name the guard diff --git a/openspec/changes/developer-toolchain-pins/task-prompt.md b/openspec/changes/developer-toolchain-pins/task-prompt.md index dba0374f..dbe0b899 100644 --- a/openspec/changes/developer-toolchain-pins/task-prompt.md +++ b/openspec/changes/developer-toolchain-pins/task-prompt.md @@ -48,7 +48,7 @@ Repo specifics (kollect): - REVIEW SIZING (spec-loop-duo, operator choice 2026-10-15; replaces every sizing instruction above): ONE leg per task: astra = `codex exec --sandbox read-only - < > /leg-astra-diff.md` (run from the repo root; brief written with the Write tool from ~/.claude/skills/close-task/reference/review-brief.md plus the diff range). Add a second leg `~/.claude/skills/fanout-review/scripts/fanout.sh --repo --target "..." --legs "claude/opus:diff" --no-unify --out ` ONLY if the diff touches secrets, CI (.github/ workflows or the CI wiring), infra or manifests, or security posture, or astra says BLOCK. Never pass an opencode model to fanout.sh; always --no-unify; read the leg files' `## Verdict` yourself. Codex quota error → claude/sonnet stands in for astra and say so in the evidence; codex "at capacity" → stand-in for this round only. A second round only for a verified BEHAVIOUR defect; at most two. - Task ids in tasks.md are `N.M` numbers; you close the whole group named in the dispatch, ticking each line. One logical commit per sub-task where the task says "its own commit" (e.g. the go.mod bump is version-only, real drift fixes are separate). -- Guards follow `hack/test/dev_mise_pin_drift_test.sh` and `hack/test/ci_workflow_security_test.sh`: plain mode checks the real tree, `--self-test` corrupts a throwaway copy (mktemp) per mutant and asserts non-zero, plus a no-op copy that must pass. Run `task lint:shell` on new scripts. +- Guards follow `hack/test/consistency_tool_pins_test.sh` and `hack/test/ci_workflow_security_test.sh`: plain mode checks the real tree, `--self-test` corrupts a throwaway copy (mktemp) per mutant and asserts non-zero, plus a no-op copy that must pass. Run `task lint:shell` on new scripts. - `task check` runs every hack/test guard by glob; do not run it per task (it is the stage-B holistic row). Do run the guards you touched, both modes. - Gotchas: helm-unittest `contains` compares whole elements; "no tests to run" exits 0; `format:check`/`docs:verify` can be red on a clean main in a working copy (verify with stash before blaming your change). - Nothing private ships: before committing run `git diff --cached | rg -i 'rewe|mkurator|titiseven|prod165|atlassian'` (must be empty) and `gitleaks protect --staged`. diff --git a/openspec/changes/developer-toolchain-pins/tasks.md b/openspec/changes/developer-toolchain-pins/tasks.md index 987b4c6c..b37975c9 100644 --- a/openspec/changes/developer-toolchain-pins/tasks.md +++ b/openspec/changes/developer-toolchain-pins/tasks.md @@ -1,33 +1,41 @@ # Tasks -Each behaviour task runs after the test or guard task that can fail on it. A check that is red on -today's tree is shown red in that task's evidence and lands in the same commit as the behaviour -task that turns it green, so `main` stays green at every commit. +Each check lands in the commit of the task that turns it green, so the `lint` job is green at every +commit. 1.1 creates the guard with only the checks that are green on today's tree and wires its two +`lint` steps in the same commit (CFC-6 reds on any unwired `consistency_*_test.sh`, so the guard +file and its wiring cannot land apart). Every check that is red on today's tree is written in the +task that fixes what it checks, shown red in that task's evidence before the fix, and committed with +the fix: DTP-1 in 2.1, DTP-2 in 2.2, the DTP-5 comment rule in 2.3, 3.2's checks with 3.3, 4.2's +tests with 4.3. -## 1. Guard first (DTP-1, DTP-2, DTP-3, DTP-5, DTP-6, DTP-7) +## 1. Guard first (DTP-3, DTP-6, DTP-7) -- [ ] 1.1 Write `hack/test/consistency_toolchain_pins_test.sh` (plain and `--self-test`) with every - check except manager liveness (that is 3.2), and add its two steps to the `lint` job. Self-test - cases on throwaway copies, each with the failure it exposes: - - DTP-1 given 18 setup-task sites and `mise.toml` at 3.52.0, when one site says 3.51.1, then red - naming that file (a missed site); when all say 3.51.1, then red (set below floor); when all say - a newer release, then green (exact-value check that would block bot bumps). - - DTP-2 given `GOVULNCHECK_VERSION` with a `# renovate:` line, when `task vulncheck` carries a - literal `govulncheck@v1.9.0`, then red (inline pin returns); when the annotation is deleted, - then red (Renovate stops seeing it). - - DTP-3 given gitleaks 8.30.1 at `ci.yaml`, `hack/install-gitleaks.sh` and - `.pre-commit-config.yaml`, when the pre-commit `rev` says v8.29.0, then red (a site drifts); - when all three say 8.29.0, then red (below floor); when `GIT_CLIFF_VERSION` says v2.12.0, then - red; when the gitleaks scan finds no site, then red (empty scan). - - DTP-5 when a `mise.toml` comment says `helm (v3.21.4)`, `go 1.27.2` or "all 17 occurrences", - then red (stale restated version or count); when a comment names the pin it sits on, then - green (false positive). - - DTP-6 when `UNVERIFIED_BY_DESIGN` in `hack/test/ci_fetch_lib_hardening_test.sh` gains an - entry, then red (the exceptions list grows). +- [ ] 1.1 Write `hack/test/consistency_toolchain_pins_test.sh` (plain and `--self-test`) with the + checks that are green today, and add its two steps to the `lint` job. Self-test cases on + throwaway copies, each with the failure it exposes: + - DTP-3 gitleaks, given 8.30.1 at `ci.yaml:164`, `hack/install-gitleaks.sh:14` and + `.pre-commit-config.yaml:5` (`v8.30.1`): when the pre-commit `rev` says v8.29.0, then red naming + that file (a site drifts); when all three say 8.29.0, then red (below floor); when the gitleaks + scan finds no site, then red (empty scan). + - DTP-3 git-cliff, given v2.13.1 at `Taskfile.yml:22`, `.github/workflows/release.yaml:420` and + `hack/verify-changelog.sh:11`: when `release.yaml` says v2.12.0, then red naming it (the release + notes are built by another git-cliff); when `verify-changelog.sh` says v2.12.0, then red naming + it (the changelog check runs another git-cliff); when all three say v2.12.0, then red (below + floor). + - DTP-3 shellcheck, given 0.11.0 at `ci.yaml:299` and `hack/install-shellcheck.sh:15` and + `v0.11.0.1` at `.pre-commit-config.yaml:56`: when `ci.yaml` says 0.11.1, then red naming it (a + bot moved one site); when all three say 0.11.1 (pre-commit `v0.11.1.0`), then red naming the + missing `pinned_sha256` entry in `install-shellcheck.sh` (a bump that cannot install); when the + pre-commit `rev` says `v0.11.0.2`, then green (the fourth component is shellcheck-py packaging, + not a shellcheck release). + - DTP-6 given `UNVERIFIED_BY_DESIGN` in `hack/test/ci_fetch_lib_hardening_test.sh` with today's + two entries as the guard's limit, when it gains a third, then red (the exceptions list grows). - DTP-7 a no-op copy of the real tree passes (a guard that reds on correct content). - Run it on today's tree and record it red on the 18 + 1 Task 3.51.1 sites, the inline - govulncheck, and the `mise.toml:24`, `:28`, `:39` and `:52` comments. Show CFC-6 red when either - new step is removed from `ci.yaml`. + - DTP-7 wiring, on a copy: when one of the two new steps is removed from `ci.yaml`, then + `hack/test/ci_workflow_security_test.sh` reds naming the guard and the missing mode (CWS-6, at + `:590`, fires before CFC-6); when both are removed, then it reds with the CFC-6 enrolment + diagnostic naming the guard (`:608`). + Run the plain mode on today's tree and record it green. - [ ] 1.2 Fold `hack/test/dev_mise_pin_drift_test.sh`: port every existing inline mutant one-to-one into `--self-test`, show each red against the new guard, list the mapping in evidence, then delete the old guard and its `ci.yaml` step. The mutants to port (old label -> expected @@ -36,10 +44,10 @@ task that turns it green, so `main` stays green at every commit. (`idiomatic_version_file_enable_tools` removed), `gomod-gone` (no `go` line in `go.mod`), `ci-self-drift` (workflow and composite action disagree on task), `no-ci-pins` (empty CI scan), `empty-tools` (empty `[tools]`), `no-tools-table` (no `[tools]` table), and the missing - `mise.toml` case. Its self-wiring check (`grep` for the step in `ci.yaml`) is replaced by CFC-6 - enrolment, shown red in 1.1. Delete `ci.yaml:411-419` (comment and step). Update every live - reference to the old name: `renovate.json:74` description, `mise.toml:8`, `.gitignore:26`, - `docs/development/setup.md:53`, the CWS-6 composite-action mutant at + `mise.toml` case. All are green on today's tree. Its self-wiring check (`grep` for the step in + `ci.yaml`) is replaced by CWS-6 and CFC-6, shown red in 1.1. Delete `ci.yaml:411-419` (comment and + step). Update every live reference to the old name: `renovate.json:74` description, `mise.toml:8`, + `.gitignore:26`, `docs/development/setup.md:53`, the CWS-6 composite-action mutant at `hack/test/ci_workflow_security_test.sh:1145-1146` (retarget it to the new guard and show it still reds), and the comments at `hack/test/ci_fetch_lib_hardening_test.sh:844` and `hack/test/ci_install_helm_hardening_test.sh:687`. Change 4's evidence and review files keep the @@ -47,56 +55,111 @@ task that turns it green, so `main` stays green at every commit. ## 2. Pins (DTP-1, DTP-2, DTP-5) -- [ ] 2.1 Task 3.52.0 at all 18 setup-task sites and `mise.toml:54`; 1.1's DTP-1 check turns green. -- [ ] 2.2 Test change on a guard owned by change 4, first: run - `bash hack/test/consistency_tool_pins_test.sh --self-test` green, then move govulncheck to - `GOVULNCHECK_VERSION` (Renovate-annotated, value v1.9.0) and show the CFC-2 mutants that `sed` - the literal `govulncheck@v…` in `Taskfile.yml` (~499-501 "unresolvable pin", ~523-525 "too old - for go.mod's Go") fail to apply. Re-anchor both onto the `GOVULNCHECK_VERSION:` line, and confirm - the two that match `govulncheck@` without a version (~503-505 "package path typo", ~527-529 "pin - missing") still apply and red for their stated reason. Confirm the plain run resolves - `{{.GOVULNCHECK_VERSION}}` (the variable lookup at ~162) rather than reporting - ``. Record a real `task vulncheck` run. 1.1's DTP-2 check turns green. -- [ ] 2.3 Replace the restated versions and count in `mise.toml:24`, `:28`, `:39` and `:52` with - pointers to the owning file; 1.1's DTP-5 check turns green. The Go half of DTP-5 needs no +- [ ] 2.1 Add the DTP-1 check, with self-test cases given 18 setup-task sites and `mise.toml` at + 3.52.0: when one site says 3.51.1, then red naming that file (a missed site); when all say 3.51.1, + then red (set below floor); when all say a newer release, then green (an exact-value check would + block bot bumps). Record the plain mode red on today's 18 + 1 Task 3.51.1 sites. Then set Task + 3.52.0 at all 18 sites and `mise.toml:54`, and rewrite `docs/development/setup.md:39` so the row + names the `go-task/setup-task` `version:` input without a version. Plain mode green; one commit. +- [ ] 2.2 Add the DTP-2 check, then move govulncheck, in one commit. Self-test cases given an + annotated `GOVULNCHECK_VERSION` used by `task vulncheck`: when `task vulncheck` carries a literal + `govulncheck@v1.9.0`, then red (inline pin returns); when the annotation is deleted, then red + (Renovate stops seeing it); when `task vulncheck` reads another variable holding v1.9.0 while the + annotated `GOVULNCHECK_VERSION` stays defined, then red naming the missing `GOVULNCHECK_VERSION` + reference (the annotated pin is not the one that runs); when the `GOVULNCHECK_VERSION` definition + is removed, then red with a missing-definition message (the reference resolves to nothing). + Record the plain mode red on today's inline `Taskfile.yml:204`. Test change on a guard owned by + change 4, first: run `bash hack/test/consistency_tool_pins_test.sh --self-test` green. Then define + `GOVULNCHECK_VERSION: v1.9.0` under `# renovate: datasource=go depName=golang.org/x/vuln` (the Go + module datasource: a `github-releases` annotation would still extract, so DTP-4 passes while + lookups fail) and use it in `task vulncheck`. Show the CFC-2 mutants that `sed` the literal + `govulncheck@v…` in `Taskfile.yml` (~499-501 "unresolvable pin", ~523-525 "too old for go.mod's + Go") fail to apply, re-anchor both onto the `GOVULNCHECK_VERSION:` line, and confirm the two that + match `govulncheck@` without a version (~503-505 "package path typo", ~527-529 "pin missing") + still apply and red for their stated reason. Confirm the plain run resolves + `{{.GOVULNCHECK_VERSION}}` (the variable lookup at ~162) rather than reporting ``. + Record a real `task vulncheck` run. Both guards green, both modes. +- [ ] 2.3 Add the DTP-5 comment rule, then fix the comments, in one commit. Self-test cases, each + with the failure it exposes: + - when a `mise.toml` comment says `helm (v3.21.4)`, `go 1.27.2` or "all 17 occurrences", then red + (a restated version or count away from its pin); + - when the block directly above `task = "3.52.0"` says `version: 3.51.1`, then red (an attached + comment gone stale); + - when a comment naming `node-version: "22"` is separated from `node = "22"` by a blank line, then + red (not attached); + - when the block above `node = "22"` names `gitleaks 8.30.1`, then red (another tool's version); + - the existing Node block (`mise.toml:56-58`, `node-version: "22"` above `node = "22"`) and Python + block (`:61-62`, `python-version: "3.12"` above `python = "3.12"`), then green (false positive + on a correct attached comment); + - in `Taskfile.yml`, a comment saying `helm v3.22.0` directly above `HELM_VERSION: v3.22.0` + (`:31`), then green; a comment saying `helm v3.21.4` anywhere else, then red (Taskfile comments + are in scope). + Record the plain mode red on today's `mise.toml:24`, `:28`, `:39` and `:52`. Replace those + restated versions and the count with pointers to the owning file. The Go half of DTP-5 needs no change: it is the existing DEV-MISE-01 check, carried over by 1.2. ## 3. Managers (DTP-4) - [ ] 3.1 Probe which files and dependencies each custom manager matches - (`renovate-config-validator --strict` plus a local dry run). Settle and record what "a manager - matches a file" means: at least one `managerFilePatterns` hit on a tracked file AND at least one - `matchStrings` dependency extracted from it. The gitleaks and shellcheck managers pass the first - half and fail the second, which is why a file-pattern-only check would call them alive. -- [ ] 3.2 Add the manager-liveness and site-coverage checks to the guard, using 3.1's definition, - with self-test cases: given today's `renovate.json`, then red naming the gitleaks - (`renovate.json:37-47`) and shellcheck (`renovate.json:48-58`) managers (dead manager); given a - manager whose file pattern matches no tracked file, then red (pattern rot); given no manager - covering `Makefile` `GOLANGCI_LINT_VERSION`, `hack/tooling/.custom-gcl.yml`, - `.pre-commit-config.yaml`, `hack/install-gitleaks.sh`, `ci.yaml:164` or `ci.yaml:299`, then red - naming the file (uncovered site); given two golangci-lint sites in different groups, then red - (sites can move apart). Record it red on today's tree. + (`renovate-config-validator --strict` plus a local dry run). Settle and record: + - what "a manager matches a file" means: at least one `managerFilePatterns` hit on a tracked file + AND at least one `matchStrings` dependency extracted from it. The gitleaks and shellcheck + managers pass the first half and fail the second, which is why a file-pattern-only check would + call them alive; + - the guard's mechanism for 3.2, with the reason: Renovate itself run in `lint` (record the pinned + Renovate version and how it runs without network), or a bash copy of the managers' regexes and + the `packageRules` order (a later `groupName` wins), with a self-test that reds when the copy + and `renovate.json` disagree (for example on the `\n` escapes at `renovate.json:80`); + - that each gitleaks pin form is rewritten correctly by a dry-run bump: `8.30.1` without `v` at + `ci.yaml:164` and `hack/install-gitleaks.sh:14`, `v8.30.1` at `.pre-commit-config.yaml:5` (no + `vv`, no dropped `v`); likewise shellcheck's `0.11.0` and the shellcheck-py `v0.11.0.1` rev. +- [ ] 3.2 Add the manager-liveness, site-coverage and grouping checks to the guard, using 3.1's + definition and mechanism, with self-test cases: given today's `renovate.json`, then red naming + the gitleaks (`renovate.json:37-47`) and shellcheck (`renovate.json:48-58`) managers (dead + manager); given a manager whose file pattern matches no tracked file, then red (pattern rot); + given no manager covering any one of `Makefile:184`, `hack/tooling/.custom-gcl.yml:6`, + `.pre-commit-config.yaml:5`, `.pre-commit-config.yaml:56`, `hack/install-gitleaks.sh:14`, + `hack/install-shellcheck.sh:15`, `ci.yaml:164`, `ci.yaml:299`, `release.yaml:420` or + `hack/verify-changelog.sh:11`, then red naming the file (uncovered site); given two golangci-lint + sites in different groups, then red (sites can move apart); given a shellcheck site in the + "pinned build tools" group, then red naming the group (a red shellcheck bump holds other tools). + Record it red on today's tree; it commits with 3.3. - [ ] 3.3 Replace the dead gitleaks manager with one that matches `ci.yaml:164`, - `hack/install-gitleaks.sh:14` and `.pre-commit-config.yaml:5`; replace the dead shellcheck - manager with one that matches `ci.yaml:299`; add managers for `Makefile:184` and - `hack/tooling/.custom-gcl.yml:6` in one group. Re-run 3.1's probe; 3.2's checks turn green. + `hack/install-gitleaks.sh:14` and `.pre-commit-config.yaml:5`; replace the dead shellcheck manager + with one that matches `ci.yaml:299`, `hack/install-shellcheck.sh:15` and + `.pre-commit-config.yaml:56`, in a `shellcheck` group of its own; add managers for `Makefile:184` + and `hack/tooling/.custom-gcl.yml:6` in one group, and for `release.yaml:420` and + `hack/verify-changelog.sh:11` in the group of `Taskfile.yml:22`. Any new group rule goes after the + "pinned build tools" rule (`renovate.json:126-129`, a later `groupName` wins) and before the "go + toolchain" rule (`renovate.json:142`), which must stay last; run + `hack/test/consistency_go_toolchain_test.sh` both modes. Re-run 3.1's probe; 3.2's checks turn + green. A shellcheck bump PR stays red until its digests are added to `install-shellcheck.sh`'s + `pinned_sha256` table: that is the intended manual step, and the own group keeps it from holding + other tools. ## 4. Installers (DTP-6) - [ ] 4.1 Probe the upstream checksum files for every platform each installer supports: git-cliff per-asset `.sha512` (the `.sha256` is 404), helm-docs `checksums.txt`. Stop if one is missing. -- [ ] 4.2 Tests first, with a fake `fetch_to`, per installer: given a tarball whose digest differs - from the published one, then the installer exits non-zero and the output path does not exist - (tampered download installed); given a checksum fetch that fails, then the same (silent - fallback to unverified). Widen the checksum scan in `hack/test/ci_fetch_lib_hardening_test.sh` - (`verify_sha256|sha256sum`, ~779 and ~793) to accept `sha512sum`, with a mutant that deletes the - `sha512sum` call from a copy of `install-git-cliff.sh` and must red once the entry leaves - `UNVERIFIED_BY_DESIGN`. Watch all red. +- [ ] 4.2 Tests first, in the new guard's `--self-test`, with a fake `fetch_to`, per installer: + given a valid archive and a checksum file in the published format, then the installer exits 0 and + the expected executable exists at the output path (an installer that always fails would pass + every negative case); from that passing fixture, flip one byte of the archive, then the installer + exits non-zero with its checksum diagnostic and the output path does not exist (tampered download + installed); make only the checksum fetch fail, then the same (silent fallback to unverified); on a + copy of the installer with its verification removed, the tampered case fails its assertion (the + test can see a missing check). Widen the checksum scan in + `hack/test/ci_fetch_lib_hardening_test.sh` (`verify_sha256|sha256sum`, ~779 and ~793) to accept + `sha512sum`, with a mutant that deletes the `sha512sum` call from a copy of + `install-git-cliff.sh` and must red once the entry leaves `UNVERIFIED_BY_DESIGN`. Watch all red; + the tests commit with 4.3. - [ ] 4.3 Verify git-cliff with `sha512sum` against `.sha512` and helm-docs against `checksums.txt` through `hack/lib/fetch.sh`; remove the NOTE at `install-git-cliff.sh:40-42` and - `install-helm-docs.sh:40`; remove both entries from `UNVERIFIED_BY_DESIGN` (~766). 4.2's tests - and the stale-exemption check turn green. + `install-helm-docs.sh:40`; remove both entries from `UNVERIFIED_BY_DESIGN` (~766). Set the new + guard's limit for `UNVERIFIED_BY_DESIGN` to empty, with a self-test case: when + `install-git-cliff.sh` is re-added, then red (the ratchet holds after the list empties). 4.2's + tests and the stale-exemption check turn green. ## 5. Land @@ -107,11 +170,11 @@ task that turns it green, so `main` stays green at every commit. | Req | Check | Expected | Status | Evidence | | --- | --- | --- | --- | --- | -| DTP-1 | guard: one-site and below-floor mutants red; newer set passes | as stated | not-run | | -| DTP-2 | inline-govulncheck and dropped-annotation mutants; CFC-2 self-test after re-anchoring; `task vulncheck` real run | red; CFC-2 green; run result recorded | not-run | | -| DTP-3 | pre-commit-rev, below-floor and empty-scan mutants | red | not-run | | -| DTP-4 | dead-manager, rotted-pattern, uncovered-site and split-group fixtures | red | not-run | | -| DTP-5 | ported go/node/python/task mutants; stale-version and site-count comment mutants | red | not-run | | -| DTP-6 | tampered and missing-checksum cases per installer; grown-exceptions mutant | installer fails, nothing installed; red | not-run | | -| DTP-7 | all mutants plus no-op by exit status; old-to-new mutant mapping; CFC-6 with a step removed | mutants red, no-op green, mapping complete, CFC-6 red | not-run | | +| DTP-1 | guard: one-site and below-floor mutants red; newer set passes; today's tree red before 2.1 | as stated | not-run | | +| DTP-2 | inline-govulncheck, dropped-annotation, other-variable and missing-definition mutants; CFC-2 self-test after re-anchoring; `task vulncheck` real run | red; CFC-2 green; run result recorded | not-run | | +| DTP-3 | gitleaks, git-cliff and shellcheck drift, below-floor and empty-scan mutants; shellcheck missing-digest mutant; four-component rev passes | red; rev case green | not-run | | +| DTP-4 | dead-manager, rotted-pattern, uncovered-site, split-group and shared-shellcheck-group fixtures; gitleaks pin forms rewritten correctly | red; forms recorded | not-run | | +| DTP-5 | ported go/node/python/task mutants; stale, detached, other-tool and site-count comment mutants; Node, Python and Taskfile attached comments pass | red; positives green | not-run | | +| DTP-6 | valid, tampered and missing-checksum cases per installer; verification-removed copy; grown-exceptions mutant before and after the list empties | valid installs; others fail, nothing installed; red | not-run | | +| DTP-7 | all mutants plus no-op by exit status; old-to-new mutant mapping; one step removed; both steps removed | mutants red, no-op green, mapping complete, CWS-6 red, CFC-6 red | not-run | | | all | independent review; CI on the PR head | APPROVE, green; both revisions recorded | not-run | | From f94b1550cd6d32b6230856ca35fed18577c57b3d Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Fri, 16 Oct 2026 03:38:20 +0200 Subject: [PATCH 05/41] :page_facing_up: docs(spec): record the spec-set triage for developer-toolchain-pins --- .../changes/developer-toolchain-pins/loop.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/openspec/changes/developer-toolchain-pins/loop.md b/openspec/changes/developer-toolchain-pins/loop.md index dd292fe6..c5410a47 100644 --- a/openspec/changes/developer-toolchain-pins/loop.md +++ b/openspec/changes/developer-toolchain-pins/loop.md @@ -1,7 +1,7 @@ # Loop — developer-toolchain-pins (spec-loop-duo run state) Repo: `/home/koni/Projects/PlatformRelay/worktrees/kollect/developer-toolchain-pins` · Branch: `feat/developer-toolchain-pins` · Base: `origin/main` @ `966e17e10` (change 4 merged) · Started: 2026-10-16 · Reviewers: astra + claude (duo) -Next: R — spec-set review (astra:spec + claude/opus:spec) on 649e5811c +Next: L — dispatch 1.1 (new consistency_toolchain_pins_test.sh + lint wiring) Budget: Claude review legs uncapped (operator goal 2026-10-16) · run under GOAL-2026-10-16-announcement-ready (decisions by model alignment, logged in agent-context/decisions.md) Change 5 of 10 (issue #442). Spec set: proposal.md, specs/developer-toolchain/spec.md, tasks.md. @@ -10,7 +10,7 @@ Change 5 of 10 (issue #442). Spec set: proposal.md, specs/developer-toolchain/sp - [x] 0 orient — re-measured against change 4's tip (2026-10-16): 6 mechanical edits + 2 model-aligned decisions - [x] P plan + tasks — present; spec set updated 649e5811c (re-measure, decisions A pin ownership and C guard shape) -- [ ] R spec-set review +- [x] R spec-set review — `reviews/R-spec-set/` (b3fd2d944) on 9621c98aa: astra REQUEST CHANGES, claude/opus CONCERNS; no CRITICAL; 11 WARNING + 6 NOTE all verified; fixes applied faa6754fd (R2 placement deviation: lint wiring lands with 1.1 because CFC-6 reds an unwired consistency_* guard; R9 mutants in 2.2) - [ ] L task loop — see *Tasks* - [ ] B branch review - [ ] Hand-off @@ -32,6 +32,20 @@ Change 5 of 10 (issue #442). Spec set: proposal.md, specs/developer-toolchain/sp | Stage | Finding (one line) | Sev | Models | Disposition | Where it went / why | Needs user? | |---|---|---|---|---|---|---| +| R | R1 shellcheck manager bump would always be red (digest table has only 0.11.0); shared group blocks | WARNING | opus | fix | own group, all 3 sites, missing-digest guard case | no | +| R | R2 1.1 left red until 2.x | WARNING | opus | fix | each check lands with its fix task | no | +| R | R3 DTP-7 vs installer-scan owner | WARNING | opus+astra | fix | fetch-lib gate owns the scan; tamper tests in new guard --self-test | no | +| R | R4 UNVERIFIED_BY_DESIGN shrink-only never tightened | WARNING | opus | fix | 4.3 limit = empty | no | +| R | R5 liveness mechanism unspecified | WARNING | opus | fix | 3.1 records mechanism + Renovate version | no | +| R | R6 tamper tests only negative | WARNING | astra | fix | valid-archive positive case first | no | +| R | R7 comment-to-pin rule undefined | WARNING | astra | fix | block-above/inline rule + Node/Python/Taskfile cases | no | +| R | R8 CWS-6 exits before CFC-6 | WARNING | astra | fix | scenario + both-steps-removed case | no | +| R | R9 vulncheck var not proven | WARNING | astra | fix | other-var + missing-def mutants (2.2) | no | +| R | R10 git-cliff pins at release.yaml:420, verify-changelog.sh:11 missed | WARNING | reviewer-verifier | fix | DTP-3/DTP-4 cover both (literal; action input can't read a Taskfile var) | no | +| R | R11 setup.md:39 restates Task version | WARNING | reviewer-verifier | fix | 2.1 rewrites without version | no | +| R | R12/R13/R15/R17 datasource, rule order, template pointer, gitleaks v-prefix | NOTE | mixed | fix | spec set | no | +| R | R14 sha512sum missing on stock macOS | NOTE | opus | defer | BACKLOG DTP-SHA512-MACOS | no | +| R | R16 change 4 archive state | NOTE | opus | reject | CFC-2/6 are merged code; archive state irrelevant | no | | 0 | DTP-2/DTP-3 floors + golangci agreement duplicate CFC-2 | decision | astra+opus | fix (A) | dropped; CFC-2 owns analysis-tool pins | logged, pending validation | | 0 | guard shape: extend dev_mise vs new consistency_* guard | decision | astra+opus | fix (C) | new consistency_toolchain_pins_test.sh; dev_mise folded (task 1.2) | logged, pending validation | From 61ebeb671c19527545c9ad1ff87f20f66201c2bb Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Fri, 16 Oct 2026 03:53:10 +0200 Subject: [PATCH 06/41] :white_check_mark: test(ci): add the toolchain-pins consistency guard New hack/test/consistency_toolchain_pins_test.sh checks that gitleaks, git-cliff and shellcheck agree across their pin sites and meet their floors, that shellcheck has digests for its pin, and that UNVERIFIED_BY_DESIGN only shrinks (DTP-3, DTP-6). Its --self-test breaks one clause per tree copy and also proves CWS-6/CFC-6 red when its lint steps are removed (DTP-7). Wired into the lint job, plain and --self-test, in the same commit because CFC-6 reds an unwired consistency guard. guard_modes in ci_workflow_security_test.sh now greps to /dev/null instead of grep -q: under pipefail the upstream grep died of SIGPIPE and the --self-test mode was dropped at random, which made the new wiring case flaky. --- .github/workflows/ci.yaml | 8 + hack/test/ci_workflow_security_test.sh | 4 +- hack/test/consistency_toolchain_pins_test.sh | 713 ++++++++++++++++++ .../developer-toolchain-pins/evidence/T1.1.md | 249 ++++++ .../reviews/L-tasks/T1.1/leg-astra-diff.md | 13 + .../L-tasks/T1.1/leg-claude-opus-diff.md | 35 + .../L-tasks/T1.1/round-2/leg-astra-diff.md | 13 + .../T1.1/round-2/leg-claude-opus-diff.md | 43 ++ .../changes/developer-toolchain-pins/tasks.md | 5 +- 9 files changed, 1080 insertions(+), 3 deletions(-) create mode 100755 hack/test/consistency_toolchain_pins_test.sh create mode 100644 openspec/changes/developer-toolchain-pins/evidence/T1.1.md create mode 100644 openspec/changes/developer-toolchain-pins/reviews/L-tasks/T1.1/leg-astra-diff.md create mode 100644 openspec/changes/developer-toolchain-pins/reviews/L-tasks/T1.1/leg-claude-opus-diff.md create mode 100644 openspec/changes/developer-toolchain-pins/reviews/L-tasks/T1.1/round-2/leg-astra-diff.md create mode 100644 openspec/changes/developer-toolchain-pins/reviews/L-tasks/T1.1/round-2/leg-claude-opus-diff.md diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 55bd156c..7ec134c6 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -562,6 +562,14 @@ jobs: run: bash hack/test/consistency_metrics_test.sh - name: Verify the metrics gate can still fail (CFC-6) run: bash hack/test/consistency_metrics_test.sh --self-test + # DTP-3/DTP-6/DTP-7 (openspec change developer-toolchain-pins): gitleaks, git-cliff and + # shellcheck agree across their pin sites and meet their floors, shellcheck has digests for + # its pin, and the installer-checksum exemptions only shrink. The self-test also removes + # these two steps from a copy and expects the CWS-6/CFC-6 meta-test to red. Needs yq. + - name: Verify gitleaks, git-cliff and shellcheck pins agree (DTP-3, DTP-6) + run: bash hack/test/consistency_toolchain_pins_test.sh + - name: Verify the toolchain-pins gate can still fail (DTP-7) + run: bash hack/test/consistency_toolchain_pins_test.sh --self-test - name: Verify the go-patch-lag sensor verdicts and workflow shape (CFC-5) run: bash hack/test/go_patch_lag_test.sh - name: Verify the go-patch-lag test can still fail (CFC-6) diff --git a/hack/test/ci_workflow_security_test.sh b/hack/test/ci_workflow_security_test.sh index b0632302..b9375dd1 100755 --- a/hack/test/ci_workflow_security_test.sh +++ b/hack/test/ci_workflow_security_test.sh @@ -454,7 +454,9 @@ cws6_required_set() { # with `read -r mode`. guard_modes() { echo bare - if grep -v '^[[:space:]]*#' "${1}" 2>/dev/null | grep -q -- '--self-test'; then + # Not `grep -q`: under pipefail it exits at the first match and the upstream grep dies of + # SIGPIPE, so the pipeline reads as no-match at random (15% of runs on a 28 KB guard). + if grep -v '^[[:space:]]*#' "${1}" 2>/dev/null | grep -- '--self-test' >/dev/null; then echo --self-test fi } diff --git a/hack/test/consistency_toolchain_pins_test.sh b/hack/test/consistency_toolchain_pins_test.sh new file mode 100755 index 00000000..c3fe9780 --- /dev/null +++ b/hack/test/consistency_toolchain_pins_test.sh @@ -0,0 +1,713 @@ +#!/usr/bin/env bash +# DTP-3, DTP-6 and DTP-7 (openspec change developer-toolchain-pins): the developer and CI tools +# that are pinned in more than one file agree, meet their floor, and can be installed. +# +# gitleaks, git-cliff and shellcheck are each pinned in three files, and nothing kept the three +# equal: a bot bump or a hand edit moves one site and the others silently run another release. +# What it locks: +# * gitleaks: every `GITLEAKS_VERSION:` under .github/, every `${GITLEAKS_VERSION:-}` default +# in hack/ (not hack/test) and the gitleaks `rev:` in .pre-commit-config.yaml agree and are at +# least GITLEAKS_FLOOR; ci.yaml, hack/install-gitleaks.sh and .pre-commit-config.yaml must each +# carry one; +# * git-cliff: Taskfile.yml `GIT_CLIFF_VERSION`, the `with: version:` input of every +# orhun/git-cliff-action step under .github/ (a step without one installs the latest +# release; a `version:` under `env:` or another map is not the input) and every literal `install-git-cliff.sh ` call agree with Taskfile.yml and are +# at least GIT_CLIFF_FLOOR; Taskfile.yml, release.yaml and hack/verify-changelog.sh must each +# carry one; +# * shellcheck: every `SHELLCHECK_VERSION:` under .github/, every `${SHELLCHECK_VERSION:-}` +# default in hack/ and the shellcheck-py `rev:` (first three components: the fourth is +# the packaging revision of the pre-commit wrapper, not a shellcheck release) agree, and +# hack/install-shellcheck.sh's pinned_sha256 table has an entry for that version on every +# OS and architecture the installer maps (a bump without digests cannot install); +# * versions compare with a leading `v` stripped and must be plain releases (N.N.N); a +# variable, `latest` or an empty value is a failure, never skipped; +# * an empty scan for any tool fails, never passes; +# * DTP-6: UNVERIFIED_BY_DESIGN in hack/test/ci_fetch_lib_hardening_test.sh (installers exempt +# from its checksum scan) is defined once, on one line, and otherwise only read (a closed list +# of read forms; any other mention is a write), and lists nothing outside +# UNVERIFIED_LIMIT: the list may only shrink. +# Lines whose first non-blank character is `#` are comments and never count as a site. +# +# --self-test (DTP-7) copies the tracked tree once per case, breaks one clause, and asserts the +# gate reds with that clause's message; one unmutated copy and the four-component shellcheck-py +# rev must pass; removing this guard's lint steps must red hack/test/ci_workflow_security_test.sh +# (CWS-6 for one step, CFC-6 for both). +# +# Usage: +# hack/test/consistency_toolchain_pins_test.sh # check the repository tree +# hack/test/consistency_toolchain_pins_test.sh --self-test # mutate copies, assert each check fails +# hack/test/consistency_toolchain_pins_test.sh # check another tree root +set -euo pipefail + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +pass() { + echo "ok - $*" +} + +SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")" + +MODE="check" +if [[ "${1:-}" == "--self-test" ]]; then + MODE="self-test" + shift +fi + +ROOT="${1:-}" +if [[ -z "${ROOT}" ]]; then + ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +fi +ROOT="${ROOT%/}" + +GITLEAKS_FLOOR="8.30.1" +GIT_CLIFF_FLOOR="2.13.1" +# The exemptions UNVERIFIED_BY_DESIGN may hold. Shrink-only: removing an entry here and from the +# hardening gate is the only allowed edit. +UNVERIFIED_LIMIT=("install-git-cliff.sh" "install-helm-docs.sh") +FETCH_GATE="hack/test/ci_fetch_lib_hardening_test.sh" + +WORKROOT="$(mktemp -d)" +cleanup() { + rm -rf "${WORKROOT:?}" 2>/dev/null || true +} +trap cleanup EXIT + +BROKEN=0 +err() { + echo "FAIL: $*" >&2 + BROKEN=1 +} + +# yaml_files : workflow and composite-action files, relative to root. +yaml_files() { + (cd "$1" && [[ -d .github ]] && find .github -type f \( -name '*.yml' -o -name '*.yaml' \) | sed 's|^\./||' | sort) || true +} + +# hack_scripts : shell scripts under hack/, guards (hack/test) excluded: they carry +# pin-shaped fixtures. +hack_scripts() { + (cd "$1" && [[ -d hack ]] && find hack -path hack/test -prune -o -type f -name '*.sh' -print | sort) || true +} + +# Every site line below is ":\t\t". + +# yaml_env_sites : `: ` keys in .github YAML. +yaml_env_sites() { + local root="$1" var="$2" f + while IFS= read -r f; do + awk -v f="${f}" -v var="${var}" ' + $0 ~ "^[[:space:]]*" var ":" { + v = $0 + sub("^[[:space:]]*" var ":[[:space:]]*", "", v) + sub(/[[:space:]]+#.*$/, "", v) + gsub(/^["\047]|["\047][[:space:]]*$/, "", v) + print f ":" NR "\t" v + }' "${root}/${f}" + done < <(yaml_files "${root}") +} + +# default_sites : `${:-}` defaults in hack/ scripts, comments excluded. +default_sites() { + local root="$1" var="$2" f + while IFS= read -r f; do + awk -v f="${f}" -v var="${var}" ' + /^[[:space:]]*#/ { next } + { + line = $0 + key = "${" var ":-" + while ((i = index(line, key)) > 0) { + rest = substr(line, i + length(key)) + j = index(rest, "}") + v = (j > 0) ? substr(rest, 1, j - 1) : rest + print f ":" NR "\t" v + line = rest + } + }' "${root}/${f}" + done < <(hack_scripts "${root}") +} + +# precommit_rev_sites : the `rev:` of every .pre-commit-config.yaml repo whose +# URL matches. +precommit_rev_sites() { + local root="$1" repo="$2" f=".pre-commit-config.yaml" + [[ -f "${root}/${f}" ]] || return 0 + awk -v f="${f}" -v repo="${repo}" ' + /^[[:space:]]*#/ { next } + /^[[:space:]]*-[[:space:]]*repo:/ { + cur = $0 + sub(/^[^:]*:[[:space:]]*/, "", cur) + sub(/[[:space:]]+#.*$/, "", cur) + gsub(/^["\047]|["\047][[:space:]]*$/, "", cur) + next + } + /^[[:space:]]*rev:/ && cur ~ repo { + v = $0 + sub(/^[[:space:]]*rev:[[:space:]]*/, "", v) + sub(/[[:space:]]+#.*$/, "", v) + gsub(/^["\047]|["\047][[:space:]]*$/, "", v) + print f ":" NR "\t" v + }' "${root}/${f}" +} + +# cliff_action_sites : the `with: version:` input of every orhun/git-cliff-action step, or +# "" when the step has none. A step is read whole (key order does not matter): +# it opens at a `- ` line and ends at the next `- ` line at its indent or a line left of it. Only +# a direct child of the step's `with:` is the input; a `version:` under `env:` or another map is +# not, and must not stand in for it. +cliff_action_sites() { + local root="$1" f + while IFS= read -r f; do + awk -v f="${f}" ' + function indent(s) { match(s, /^[[:space:]]*/); return RLENGTH } + function flush() { + if (open && cliff) print f ":" (ver == "" ? at "\t" : vat "\t" ver) + open = 0; cliff = 0; ver = ""; w = -1; wc = -1 + } + /^[[:space:]]*#/ { next } + !/[^[:space:]]/ { next } + { + ind = indent($0) + isdash = ($0 ~ /^[[:space:]]*-[[:space:]]/) + if (open && (ind < step || (ind == step && isdash))) flush() + if (!open && isdash) { + open = 1; step = ind; at = NR; w = -1; wc = -1 + match($0, /^[[:space:]]*-[[:space:]]+/); kl = RLENGTH + } + if (!open) next + lvl = (isdash && ind == step) ? kl : ind + k = $0 + sub(/^[[:space:]]*(-[[:space:]]+)?/, "", k) + sub(/[[:space:]]+#.*$/, "", k) + if (lvl == kl) { + w = -1 + if (k ~ /^uses:[[:space:]]*["\047]?orhun\/git-cliff-action@/) { cliff = 1; at = NR } + if (k ~ /^with:[[:space:]]*$/) { w = kl; wc = -1 } + next + } + if (w >= 0 && lvl > w) { + if (wc < 0) wc = lvl + if (lvl == wc && k ~ /^version:/) { + v = k + sub(/^version:[[:space:]]*/, "", v) + gsub(/^["\047]|["\047][[:space:]]*$/, "", v) + ver = (v == "") ? "" : v + vat = NR + } + } + } + END { flush() }' "${root}/${f}" + done < <(yaml_files "${root}") +} + +# cliff_install_sites : literal `install-git-cliff.sh ` calls in hack/ scripts, +# Makefile, Taskfile.yml and .github YAML. `{{.GIT_CLIFF_VERSION}}` reads the Taskfile site and +# is not a site of its own; any other non-literal argument is reported, never skipped. +cliff_install_sites() { + local root="$1" f + while IFS= read -r f; do + awk -v f="${f}" ' + /^[[:space:]]*#/ { next } + { + line = $0 + while (match(line, /install-git-cliff\.sh[[:space:]]+[^[:space:]]+/)) { + v = substr(line, RSTART, RLENGTH) + sub(/^install-git-cliff\.sh[[:space:]]+/, "", v) + gsub(/["\047]/, "", v) + if (v != "{{.GIT_CLIFF_VERSION}}") print f ":" NR "\t" v + line = substr(line, RSTART + RLENGTH) + } + }' "${root}/${f}" + done < <( + hack_scripts "${root}" + [[ -f "${root}/Makefile" ]] && echo Makefile + [[ -f "${root}/Taskfile.yml" ]] && echo Taskfile.yml + yaml_files "${root}" + ) +} + +taskfile_sites() { + local root="$1" var="$2" f="Taskfile.yml" + [[ -f "${root}/${f}" ]] || return 0 + awk -v f="${f}" -v var="${var}" ' + $0 ~ "^[[:space:]]*" var ":" { + v = $0 + sub("^[[:space:]]*" var ":[[:space:]]*", "", v) + sub(/[[:space:]]+#.*$/, "", v) + gsub(/^["\047]|["\047][[:space:]]*$/, "", v) + print f ":" NR "\t" v + }' "${root}/${f}" +} + +# with_compare : appends the compared form. `plain` strips a leading v; `first3` also +# drops a fourth numeric component (shellcheck-py's packaging revision). +with_compare() { + local mode="$1" loc v c + while IFS=$'\t' read -r loc v; do + [[ -n "${loc}" ]] || continue + c="${v#v}" + if [[ "${mode}" == "first3" && "${c}" =~ ^([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+$ ]]; then + c="${BASH_REMATCH[1]}" + fi + printf '%s\t%s\t%s\n' "${loc}" "${v}" "${c}" + done +} + +# version_lt : a < b for validated N.N.N strings. +version_lt() { + [[ "$1" != "$2" && "$(printf '%s\n%s\n' "$1" "$2" | sort -V | head -1)" == "$1" ]] +} + +# check_tool ... +# Sets AGREED to the shared version when every site is plain and equal, else to "". +AGREED="" +check_tool() { + local tool="$1" floor="$2" ref_file="$3" sites="$4" + shift 4 + local f loc v c ref_loc="" ref_v="" ref_c="" values="" ok=1 + AGREED="" + if [[ -z "${sites}" ]]; then + err "${tool}: the scan found no pin site (looked in $*) -- an empty scan never passes" + return 0 + fi + echo "${tool} sites:" + while IFS=$'\t' read -r loc v c; do echo " ${loc} ${v}"; done <<<"${sites}" + for f in "$@"; do + awk -v f="${f}" 'index($0, f ":") == 1 { found = 1 } END { exit !found }' <<<"${sites}" || + { err "${tool}: no pin found in ${f} -- a named site lost its pin or the scan can no longer read it"; ok=0; } + done + while IFS=$'\t' read -r loc v c; do + if [[ ! "${c}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + err "${tool}: ${loc} says '${v}', not a plain release version -- a variable, a range or 'latest' cannot be compared" + ok=0 + continue + fi + if [[ -z "${ref_loc}" && "${loc%%:*}" == "${ref_file}" ]]; then + ref_loc="${loc}" ref_v="${v}" ref_c="${c}" + fi + values="${values} ${c}" + done <<<"${sites}" + [[ -n "${values}" ]] || return 0 + if [[ -z "${ref_loc}" ]]; then + # The reference file has no plain site (reported above); compare against the first plain one. + IFS=$'\t' read -r ref_loc ref_v ref_c < <(awk -F'\t' '$3 ~ /^[0-9]+\.[0-9]+\.[0-9]+$/' <<<"${sites}" | head -1) + fi + while IFS=$'\t' read -r loc v c; do + [[ "${c}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || continue + if [[ "${c}" != "${ref_c}" ]]; then + err "${tool}: ${loc} says ${v}, but the reference site ${ref_loc} says ${ref_v} -- one tool, one release" + ok=0 + fi + done <<<"${sites}" + ((ok == 1)) || return 0 + if [[ "${floor}" != "-" ]] && version_lt "${ref_c}" "${floor}"; then + err "${tool}: every site says ${ref_c}, below the floor ${floor}" + return 0 + fi + AGREED="${ref_c}" +} + +# check_shellcheck_digests : pinned_sha256 has a 64-hex digest for every +# OS/architecture pair the installer maps. +check_shellcheck_digests() { + local root="$1" ver="$2" f="hack/install-shellcheck.sh" oses archs os arch asset + oses="$(sed -n -E 's/^[[:space:]]*[^#)]+\)[[:space:]]*OS="?([a-z0-9_]+)"?[[:space:]]*;;.*$/\1/p' "${root}/${f}" | sort -u)" + archs="$(sed -n -E 's/^[[:space:]]*[^#)]+\)[[:space:]]*ARCH="?([a-z0-9_]+)"?[[:space:]]*;;.*$/\1/p' "${root}/${f}" | sort -u)" + if [[ -z "${oses}" || -z "${archs}" ]]; then + err "shellcheck: cannot read the supported platforms of ${f} (no OS=/ARCH= case arms) -- the digest check would pass on nothing" + return 0 + fi + for os in ${oses}; do + for arch in ${archs}; do + asset="shellcheck-v${ver}.${os}.${arch}.tar.xz" + awk -v want="${asset})" ' + /^[[:space:]]*#/ { next } + found { if ($0 ~ /^[[:space:]]*echo[[:space:]]+"[0-9a-f]{64}"[[:space:]]*$/) ok = 1; exit } + { t = $0; gsub(/^[[:space:]]+|[[:space:]]+$/, "", t); if (t == want) found = 1 } + END { exit !ok }' "${root}/${f}" || + err "shellcheck: ${f} pinned_sha256 has no entry for ${asset} -- every site says ${ver}, which cannot install until its digests are added" + done + done +} + +# unverified_other_uses : line numbers of every non-comment mention of UNVERIFIED_BY_DESIGN +# that is not the definition `UNVERIFIED_BY_DESIGN=( ... )`, a read `${UNVERIFIED_BY_DESIGN[@]}`, +# `${UNVERIFIED_BY_DESIGN[*]}` or `${#UNVERIFIED_BY_DESIGN[@]}`, or the bare word between spaces +# inside a double-quoted string (a message). A closed list: `+=`, `[i]=`, `${...:=}`, mapfile, +# `read -a`, `printf -v`, a nameref and any other form are writes. +unverified_other_uses() { + awk ' + /^[[:space:]]*#/ { next } + /^[[:space:]]*UNVERIFIED_BY_DESIGN=\(.*\)[[:space:]]*$/ { next } + { + line = $0 + gsub(/\$\{#?UNVERIFIED_BY_DESIGN\[[@*]\]\}/, "", line) + name = "UNVERIFIED_BY_DESIGN" + done = 0 + while (!done && (i = index(line, name)) > 0) { + pre = substr(line, 1, i - 1) + q = gsub(/"/, "\"", pre) - gsub(/\\"/, "&", pre) + before = substr(line, i - 1, 1) + after = substr(line, i + length(name), 1) + if (!(q % 2 == 1 && before == " " && after == " ")) { print NR; done = 1 } + line = substr(line, i + length(name)) + } + }' "$1" +} + +check_unverified_limit() { + local root="$1" f="${FETCH_GATE}" defs n others body entry known ok entries=() + if [[ ! -f "${root}/${f}" ]]; then + err "DTP-6: ${f} not found -- the exemptions list cannot be read" + return 0 + fi + defs="$(grep -E '^[[:space:]]*UNVERIFIED_BY_DESIGN=\(.*\)[[:space:]]*$' "${root}/${f}" || true)" + others="$(unverified_other_uses "${root}/${f}" | tr '\n' ' ')" + if [[ -z "${defs}" ]]; then + err "DTP-6: ${f} has no single-line UNVERIFIED_BY_DESIGN=( ... ) definition -- the exemptions list cannot be read" + return 0 + fi + n="$(grep -c . <<<"${defs}" || true)" + if [[ "${n}" != "1" || -n "${others}" ]]; then + err "DTP-6: UNVERIFIED_BY_DESIGN is written $((n + $(wc -w <<<"${others}"))) times in ${f} (definitions: ${n}; other writes or unrecognised uses at lines: ${others:-none}) -- one definition and plain reads only, so the list cannot grow behind it" + return 0 + fi + [[ "${defs}" =~ ^[[:space:]]*UNVERIFIED_BY_DESIGN=\((.*)\)[[:space:]]*$ ]] + body="${BASH_REMATCH[1]//\"/}" + body="${body//\'/}" + read -r -a entries <<<"${body}" + for entry in ${entries[@]+"${entries[@]}"}; do + ok=0 + for known in ${UNVERIFIED_LIMIT[@]+"${UNVERIFIED_LIMIT[@]}"}; do + [[ "${entry}" == "${known}" ]] && ok=1 + done + ((ok == 1)) || + err "DTP-6: UNVERIFIED_BY_DESIGN lists ${entry}, which is not in this guard's limit (${UNVERIFIED_LIMIT[*]:-empty}) -- the exceptions list only shrinks" + done + echo "UNVERIFIED_BY_DESIGN: ${body:-}" +} + +check_all() { + local root="$1" sites + BROKEN=0 + + sites="$({ + yaml_env_sites "${root}" GITLEAKS_VERSION + default_sites "${root}" GITLEAKS_VERSION + precommit_rev_sites "${root}" 'github\.com/gitleaks/gitleaks(\.git)?$' + } | with_compare plain)" + check_tool gitleaks "${GITLEAKS_FLOOR}" hack/install-gitleaks.sh "${sites}" \ + .github/workflows/ci.yaml hack/install-gitleaks.sh .pre-commit-config.yaml + + sites="$({ + taskfile_sites "${root}" GIT_CLIFF_VERSION + cliff_action_sites "${root}" + cliff_install_sites "${root}" + } | with_compare plain)" + check_tool git-cliff "${GIT_CLIFF_FLOOR}" Taskfile.yml "${sites}" \ + Taskfile.yml .github/workflows/release.yaml hack/verify-changelog.sh + + sites="$({ + yaml_env_sites "${root}" SHELLCHECK_VERSION | with_compare plain + default_sites "${root}" SHELLCHECK_VERSION | with_compare plain + precommit_rev_sites "${root}" 'github\.com/shellcheck-py/shellcheck-py(\.git)?$' | with_compare first3 + })" + check_tool shellcheck - hack/install-shellcheck.sh "${sites}" \ + .github/workflows/ci.yaml hack/install-shellcheck.sh .pre-commit-config.yaml + if [[ -n "${AGREED}" ]]; then + check_shellcheck_digests "${root}" "${AGREED}" + fi + + check_unverified_limit "${root}" + return "${BROKEN}" +} + +if [[ "${MODE}" == "check" ]]; then + if check_all "${ROOT}"; then + pass "gitleaks, git-cliff and shellcheck sites agree and meet their floors, shellcheck has digests for its pin, and UNVERIFIED_BY_DESIGN is within its limit" + exit 0 + fi + fail "toolchain pins disagree (listed above)" +fi + +# --------------------------------------------------------------------------- +# Self-test (DTP-7): copy the tracked tree, break one clause per copy, and assert the gate reds +# WITH THE MESSAGE of that clause -- a non-zero exit alone is not evidence. Sentinels are EREs. +# --------------------------------------------------------------------------- +command -v git >/dev/null 2>&1 || fail "git not found; the self-test copies the tracked tree" +command -v yq >/dev/null 2>&1 || fail "yq not found; the wiring cases run hack/test/ci_workflow_security_test.sh, which needs it" +git -C "${ROOT}" rev-parse --is-inside-work-tree >/dev/null 2>&1 || + fail "self-test: ${ROOT} is not a git work tree; the self-test copies its tracked files" + +# Called directly (never as $(new_copy)), so COPY is set in this shell; WORKROOT's trap removes it. +COPY="" +new_copy() { + COPY="$(mktemp -d "${WORKROOT}/XXXXXXXX")" + (cd "${ROOT}" && git ls-files -z | tar --null --ignore-failed-read -T - -cf - 2>/dev/null) | + tar -xf - -C "${COPY}" +} + +# probe