Repository navigation
🔧 build: pin and verify the developer toolchain in one place - #465
Merged
Merged
Conversation
…wnership decisions Re-measured against main after cross-file-consistency-gates (#463): govulncheck v1.9.0 at Taskfile.yml:204, gitleaks env at ci.yaml:164, golangci-lint v2.14.0, a second dead Renovate manager (shellcheck), and git-cliff publishing only per-asset .sha512 checksums. Pin ownership A: golangci-lint and govulncheck floors and the Makefile/.custom-gcl.yml agreement move out of this change; CFC-2 owns them. Guard shape C: the checks live in consistency_toolchain_pins_test.sh, enrolled by CFC-6, and dev_mise_pin_drift_test.sh is folded into it mutant for mutant. Every behaviour task now follows the task that can fail on it, and manager liveness is defined only after the probe.
Fixes R1-R13, R15 and R17 from reviews/R-spec-set/verification.md: shellcheck gets its own Renovate group and a three-site agreement check with a digest-table case, git-cliff gains its release.yaml and verify-changelog.sh sites, each red check moves into the task that turns it green, DTP-5 defines comment attachment, DTP-7 splits the unwired scenario into CWS-6 and CFC-6 cases, and the installer tests gain a valid-download control and a ratchet to an empty exception list.
New hack/test/consistency_toolchain_pins_test.sh checks that gitleaks, git-cliff and shellcheck agree across their pin sites and meet their floors, that shellcheck has digests for its pin, and that UNVERIFIED_BY_DESIGN only shrinks (DTP-3, DTP-6). Its --self-test breaks one clause per tree copy and also proves CWS-6/CFC-6 red when its lint steps are removed (DTP-7). Wired into the lint job, plain and --self-test, in the same commit because CFC-6 reds an unwired consistency guard. guard_modes in ci_workflow_security_test.sh now greps to /dev/null instead of grep -q: under pipefail the upstream grep died of SIGPIPE and the --self-test mode was dropped at random, which made the new wiring case flaky.
DEV-MISE-01 now runs inside hack/test/consistency_toolchain_pins_test.sh. Each of the old guard's twelve cases is a --self-test mutant on a copy of the real tree, red for its own clause only; a thirteenth pins the [tools] scope, which no old case covered. The old guard, its ci.yaml step and its self-wiring grep go: CWS-6 and CFC-6 hold the new guard in the lint job in both modes. Live references, including the CWS-6 composite-action mutant and the renovate.json description, now name the new guard. Task 1.2 of developer-toolchain-pins; evidence in evidence/T1.2.md.
hack/test/consistency_toolchain_pins_test.sh now checks Task: the with: version: input of every go-task/setup-task step under .github and mise.toml's task must agree and be at least 3.52.0. A step without the input now fails; it used to be skipped, which let Task install unpinned. The step reader git-cliff already used is shared, and DEV-MISE-01 keeps only the presence of mise's task entry, so each mutant reds once. Self-test cases cover a missed site, a set below the floor, a bot bump to a newer release (green), a step without version:, a lost named site and an empty scan. The 18 setup-task inputs and mise.toml move from 3.51.1 to 3.52.0, and setup.md names the input instead of restating the version. Task 2.1 of developer-toolchain-pins; evidence in evidence/T2.1.md.
task vulncheck carried govulncheck@v1.9.0 inline, where no Renovate manager sees it. Move it to GOVULNCHECK_VERSION in Taskfile.yml under a datasource=go annotation (the Go module datasource; a github-releases annotation would extract but never look up) and have vulncheck read it. The toolchain-pins guard now fails when vulncheck pins inline, reads another variable, sets its own task-level GOVULNCHECK_VERSION, or when the definition is missing, outside vars:, not a plain vN.N.N, or not directly under the exact annotation the Taskfile manager extracts. Change 4's two CFC-2 mutants that rewrote the literal govulncheck@v... no longer landed; they now rewrite the GOVULNCHECK_VERSION line, with unchanged sentinels.
Add the DTP-5 comment rule to the toolchain-pins guard: a version in a mise.toml or Taskfile.yml comment is allowed only in the comment run directly above (or inline on) the pin it names, at that pin's value, and any count of pin sites fails. Rewrite the four stale mise.toml comments (git-cliff/helm-docs, helm, go, the task site count) as pointers to the file that owns each pin; the task comment drops its version because Renovate bumps the pin but not the comment.
The gitleaks and shellcheck custom managers matched ci.yaml but extracted nothing since CI moved to the install scripts, so no bot bump reached the three gitleaks, three shellcheck, two golangci-lint or two git-cliff sites outside Taskfile.yml. The toolchain-pins guard now reads renovate.json with a python model of Renovate's regex managers and packageRules order (the last matching groupName wins): every manager must hit a tracked file and extract a dependency from it, no file pattern may match nothing, every gitleaks, git-cliff, shellcheck, Task, govulncheck and golangci-lint site must be extracted at its line and value, each tool's sites must share one group, and shellcheck's group must hold nothing else. Shapes the model does not implement (RE2-foreign regex syntax included) fail. The model's extraction was checked identical to Renovate 44.115.4's on this tree; Renovate itself is not run in lint (its group lookup needs the GitHub API and a token). renovate.json: the dead managers are replaced (the version captured without the v so the pre-commit rev keeps it), shellcheck-py's four-part rev is its own github-tags dependency, golangci-lint and the remaining git-cliff sites get managers, and a shellcheck group rule sits before the go toolchain rule: a shellcheck bump stays red until its digests are added, and now holds back no other tool.
Probe, mechanism choice, red-on-base output, review records and gate runs for the DTP-4 manager checks.
install-git-cliff.sh now checks the tarball with sha512sum against the per-asset .sha512 upstream publishes (there is no .sha256), and install-helm-docs.sh checks it with verify_sha256 against checksums.txt. Both fetch the checksum first and install nothing on a mismatch, a missing checksum file or a file with no line for the asset. The fetch-lib hardening gate's checksum scan accepts sha512sum, and UNVERIFIED_BY_DESIGN is empty, as is the toolchain-pins guard's limit for it. The guard's self-test runs each installer against a fake fetch_to on valid, tampered, missing-checksum and other-asset downloads, and on a copy with its verification removed (DTP-6).
The checksum comes from the same release as the tarball, so it catches a corrupted or swapped download, not a compromised release. The comments no longer promise more.
The git-cliff sentinels, probes and env-mutant values hard-coded v2.13.1, so a correct git-cliff bump redded --self-test and held the shared Renovate group back (stage-B B1). They now read Taskfile.yml, and a new control bumps every pin the Renovate model extracts to its next patch release (shellcheck digests added by hand, as its PR would) and requires plain mode and a nested --self-test to pass on that copy. Task 6.1.
The self-test takes about 8.5 minutes with the bump control; lint runs it no more, following the operator ruling (2026-10-16) that keeps slow self-tests out of lint. It joins the CFC-2 self-test in the existing required tool-pins-selftest job (timeout 45 min kept; the two took about 16 minutes locally). CFC-6 now pins both self-tests to that job, with a mutant that moves the toolchain one back into lint; the tool-pins move mutant moves only its own step, and the not-required mutant names either self-test. Task 6.2.
Stock macOS has no sha512sum or sha256sum, so the git-cliff installer and verify_sha256 (helm-docs) failed closed there with 'command not found'. Both now fall back to shasum -a 512 / -a 256 and refuse, saying why, only when neither tool exists. helm-docs also requires its digest to be 64 hex before comparing (git-cliff already required 128). The toolchain-pins self-test runs both installers with sha*sum hidden from PATH (shasum only: valid installs, tampered refused; no hash tool: refused, nothing installed) and on a checksum line one hex digit short. Tasks 6.3 and 6.4.
A {{.VAR}} pin resolved through Taskfile.yml was used as-is, so GOVULNCHECK_VERSION: latest resolved through go mod download and passed while a literal @latest is skipped as no pin. A resolved value that is not a version is now reported as unresolved. Test change on change 4's guard: one new self-test mutant. Task 6.5.
GitHub resolves owner/repo case-insensitively, so a step using Go-Task/Setup-Task@ installs the same Task, but the DTP-1 and git-cliff readers matched the lower-case spelling only and skipped such a site, drift included. They now match case-insensitively; renovate.json's managers still match lower case only, so such a site also reds as uncovered (DTP-4). New self-test mutants: a drifted mixed-case setup-task and git-cliff-action step, and task = "latest" in mise.toml. Task 6.6.
golangci-lint shared 'pinned build tools', so a golangci-lint bump that is red on new findings, or on a Go it needs, held Task, gitleaks, git-cliff and govulncheck back. It now has its own group, after 'pinned build tools' and before 'go toolchain'. The toolchain-pins guard's alone-group check (shellcheck until now) covers golangci-lint too, with a mutant that removes the rule; the split-group mutant's sentinel names the new group. Task 6.7.
Drop the CWS-6 line number from DTP-7 and state where the guard is stronger than the spec: DTP-5 reads any dotted or quoted comment number as a version, DTP-4 fails a manager on one rotted file pattern. Record the G6 behaviour: golangci-lint's own group (DTP-4), the shasum fallback, no-hash-tool refusal and full-hex digest (DTP-6), the self-test's job and the all-pins bump control (DTP-7). The Renovate model's comment names its extends-presets limit. Task 6.8 and the spec side of 6.1, 6.3, 6.4 and 6.7.
A partial version such as v1 passed the v-digit check, and go mod download resolves it as a query to the newest v1 release, so a literal or Taskfile-variable pin of v1 floated. Literal and Taskfile-variable pins must now be full semantic versions (pre-release and pseudo-versions allowed), with a self-test mutant for a v1 variable. Review follow-up to task 6.5.
The inline and env: govulncheck mutants restated v1.9.0, which stops being today's pin after a Renovate bump; they now read GOVULNCHECK_VERSION like the git-cliff cases. The nested run says it skips the bump control, the Renovate model's extends comment says its dry-runs predate the golangci-lint group, and verify-sha256_test.sh covers the shasum fallback and the no-hash-tool refusal of the shared helper directly. Review follow-up to tasks 6.1, 6.3 and 6.8.
Evidence for the stage-B round-1 fixes (tasks 6.1 to 6.8): red runs, hand mutants, gate runs on the final tree, two review rounds and their dispositions. Ticks 6.1 to 6.8.
…ns group Renovate's github-actions manager extracts the node-version and python-version inputs of actions/setup-node and actions/setup-python (depType uses-with), so a bump would land in the "github actions" group without mise.toml and the DEV-MISE-01 check would red that whole group PR. A last packageRule disables those updates; they stay hand-bumped with mise.toml. The toolchain-pins guard's Renovate model now reads those inputs and fails if any stays enabled, with mutants for the rule removed and re-enabled by a later rule. The ci.yaml comment that said nothing else manages these inputs is corrected.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
This was referenced Oct 16, 2026
konih
added a commit
that referenced
this pull request
Oct 16, 2026
Fill the Verification rows from the evidence files (DTP-1..DTP-7) and the PR row (PR #465 head 6f5dc8e: 39 checks green, 3 skipped; cold review REQUEST_CHANGES with 2 non-blocking P2 findings deferred to the backlog; main post-merge CI by headSha 7ced03a: all workflows success), tick 5.1, close the loop hand-off, and archive the change. The living spec lands in openspec/specs/developer-toolchain/. Refs #442
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
GOVULNCHECK_VERSIONvariable, a new guardhack/test/consistency_toolchain_pins_test.shthat replacesdev_mise_pin_drift_test.sh, live Renovate managers for every pin site, and mandatory checksums in the git-cliff and helm-docs installers.lint, so a guard bug or a Renovate bump it misreads blocks every PR until fixed.renovate.json(managers and groups) and the installer checksum paths; approving also confirms the two model-aligned decisions below.Closes #442.
The same tool was pinned in up to four files with nothing keeping them in step, and the Renovate managers meant to move them were partly dead, so "no update PR" looked the same as "up to date". This PR makes every tool's sites agree, gives every site a live Renovate manager, and makes the guard red when they drift.
Most important things
dev_mise_pin_drift_test.sh.hack/test/consistency_toolchain_pins_test.shabsorbs the old guard's 12 cases 1:1 and adds a 13th. It checks that Task, gitleaks, git-cliff and shellcheck sites agree and meet their floors (never exact values, so a bot bump that moves every site stays green); that shellcheck's installer has a digest for its pin; thattask vulncheckruns the annotatedGOVULNCHECK_VERSION; thatmise.tomlandTaskfile.ymlcomments restate no version away from its pin; and that every Renovate custom manager is alive, covers every pin site and keeps each tool in one group. 146 self-test cases, including an all-pins bump control so a correct Renovate bump cannot red the guard.Taskfile.yml,release.yaml,hack/verify-changelog.sh) and golangci-lint (Makefile,.custom-gcl.yml). shellcheck and golangci-lint each get their own group: a shellcheck bump is red until its digest is added, and a red golangci-lint bump must not hold back Task, gitleaks, git-cliff or govulncheck. ApackageRulestops the github-actions manager bumpingnode-version/python-versioninputs withoutmise.toml..sha512; helm-docs verifies againstchecksums.txt(64-hex enforced).UNVERIFIED_BY_DESIGNinci_fetch_lib_hardening_test.shis now empty and limited to empty.hack/lib/verify-sha256.shand the git-cliff path fall back toshasum -aon macOS and refuse when neither tool exists. That fallback has had no real Darwin run; it is covered by tests on Linux only.hack/test/consistency_tool_pins_test.shnow requires a full semver for literal and Taskfile-variable pins, solatestor a go query such asv1no longer passes; its mutants were re-anchored because govulncheck moved into a variable. This is a test change to another change's guard.lint. The new guard's--self-testruns in the existing required jobtool-pins-selftest, next to CFC-2's. The job now takes about 15 min (45 min timeout). No new job, so no ruleset change. CFC-6 (ci_workflow_security_test.sh) now pins both self-tests to that job.proposal.md): A, pin ownership: golangci-lint and govulncheck get no floor or agreement check here, because CFC-2 already owns those pins. C, guard shape: a newconsistency_*guard instead of extendingdev_mise_pin_drift_test.sh, which is folded in and deleted.Risk
lintand with--self-testintool-pins-selftest. A Renovate bump that changes a file shape the guard's Renovate model does not know (presets, minimatch edge cases) could red the bot PR. The macOS checksum path could fail on a real Mac.changelog-sync.yamlruns git-cliff). No product code changes.go-install-toolvar or literal@lateststill floats past CFC-2;ENVTEST_VERSION's branch pin needs its own decision), DTP-SHA512-MACOS, the shasum fallback for install-helm/polaris, and case-sensitivesetup-gomatching in change 4's go-toolchain guard. The guard's Renovate model is a Python reading ofrenovate.json, checked against Renovate 44.115.4 extract dry-runs (41 deps), not Renovate itself.Files to look at first
hack/test/consistency_toolchain_pins_test.sh: the new guard; start at the Renovate model and the all-pins bump control.hack/test/consistency_tool_pins_test.sh: CFC-2's full-semver rule and re-anchored mutants.hack/test/ci_workflow_security_test.sh: CFC-6 now pins both self-tests totool-pins-selftest; also fixes agrep -v | grep -qSIGPIPE flake under pipefail..github/workflows/ci.yaml: self-test step moved totool-pins-selftest, olddev_misestep removed.renovate.json: new managers, own groups for shellcheck and golangci-lint, node/python uses-with disabled.hack/install-git-cliff.sh,hack/install-helm-docs.sh,hack/lib/verify-sha256.sh: supply-chain change..github/workflows/*,.github/actions/kind-e2e-setup/action.yml,mise.toml,Taskfile.yml: Task 3.52.0 at every site andGOVULNCHECK_VERSION.Review this commit-by-commit, not as one flattened diff. Each check landed with the fix it gates, and the review fixes are separate commits; most of the file count is OpenSpec evidence under
openspec/changes/.Testing protocol
--self-test, CFC-2 and CFC-6 both modes,task lint:shell,task lint:markdown,task spec:validate,task vulncheck, evidence-markdown guard;task checkgreen at both branch-review rounds.gitleaks detectoverorigin/main..HEADclean.tool-pins-selftestshows both self-tests (15-20 min expected).gh pr merge <n> --rebase --delete-branch.bash hack/install-git-cliff.shandbash hack/install-helm-docs.shcomplete; a tampered archive fails withchecksum mismatch.Optional reading — the long version
Nothing above depends on this. Read it if you want the full story, the road not taken, or the deployment fine print — skip it if the top half already told you what you needed.
Where a human should look first (from the loop state)
hack/test/consistency_toolchain_pins_test.sh: new guard (absorbsdev_mise_pin_drift_test.sh; 146 self-test cases incl. all-pins bump control and a Python Renovate model).hack/test/consistency_tool_pins_test.sh: change 4's CFC-2 tightened; re-anchored mutants.hack/test/ci_workflow_security_test.sh: CFC-6 pins both self-tests totool-pins-selftest; guard_modes SIGPIPE flake fix..github/workflows/ci.yaml: toolchain self-test intool-pins-selftest; old dev_mise step removed.renovate.json: new managers (gitleaks, shellcheck, shellcheck-py, git-cliff, golangci-lint), own groups for shellcheck and golangci-lint, uses-with node/python disabled.hack/install-git-cliff.sh,hack/install-helm-docs.sh,hack/lib/verify-sha256.sh: checksums mandatory, macOSshasumfallback..github/workflows/*+mise.toml: Task 3.52.0 at 19 sites; govulncheck viaGOVULNCHECK_VERSION.Why the self-test left
lintThe guard's
--self-testtook about 233 s locally. The operator ruled on 2026-10-16 that slow self-tests stay out oflint, so it runs as a second step intool-pins-selftest, which is already required. A new job would have needed a ruleset change after merge.Why golangci-lint got its own group
All
custom.regexdependencies shared the "pinned build tools" group. A golangci-lint bump often needs code fixes, and a red golangci-lint PR would have held back every other tool in that group.Review history
Spec-set review, a per-task loop with independent reviewers, and two branch-review rounds; every reproduced finding was fixed with a mutant, and the rest are recorded with dispositions in
openspec/changes/developer-toolchain-pins/loop.md(Triage table). The G6 fixes were split into ten commits from one tested tree; all eleven passed a per-commit gate in round 2.