Skip to content

ci: adopt check-supply-chain as a non-blocking step in the audit job - #77

Merged
sgort merged 1 commit into
accfrom
ci/adopt-check-supply-chain
Sep 8, 2026
Merged

sgort merged 1 commit into
accfrom
ci/adopt-check-supply-chain

Conversation

@sgort

@sgort sgort commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Closes #73.

Adopts scripts/check-supply-chain.mjs from ttl-editor, adds npm run check-supply-chain, and wires it into the existing audit job as a non-blocking step.

What it checks that zizmor cannot

zizmor validates pin format — that a uses: names a 40-character SHA. It cannot say the SHA is the right one, so a wrong or hostile digest carrying a plausible # v4.4.0 comment passes zizmor, Prettier and human review alike. This resolves every digest against the GitHub API and compares SECURITY-PIPELINE.md's register with the workflows: digests, versions, and (×N) multiplicities.

Copied verbatim from ttl-editor acc after ttl-editor#86 — confirmed by checking that the last commit touching the file is that fix. Taking it earlier would have imported the two false positives this repository is uniquely shaped to trigger.

Left unformatted deliberately: Prettier doesn't cover scripts/, and reformatting would diverge from upstream for no benefit. zizmor.yml likewise already fails prettier --check identically on acc, so I haven't reformatted that either.

Acceptance

# Box Result
1 npm run check-supply-chain exits 0 on acc ✅ 23 refs / 5 actions, register agrees
2 Planted wrong digest fails, naming every digest the register records ✅ named both recorded digests
3 Digest changed without updating the register fails
4 A real Renovate PR still passes see below

Box 2, since it is the one the upstream fix was about:

[register] actions/checkout: workflow pins deadbeef1234… (v4.4.0) but
           SECURITY-PIPELINE.md records only 11d5960a3267… (v4.4.0), a37ce9120846… (v3.7.0)
[pin-truth] actions/checkout: comment claims v4.4.0, which resolves to
            11d5960a3267… — but the workflow pins deadbeef1234…

Both recorded digests are named. Before ttl-editor#86 the second row overwrote the first and every workflow on the other digest read as disagreeing.

Box 4 is why this is non-blocking

Tested against the real thing, not reasoned about. Fetching the tree of #66 (actions/checkout → v7.0.1) and running the check against it:

[register] actions/checkout: workflow pins 3d3c42e5aac5… (v7.0.1) but
           SECURITY-PIPELINE.md records only 11d5960a3267… (v4.4.0), a37ce9120846… (v3.7.0)

Pin truth passes. Renovate's digest and its rewritten comment agree with each other and with GitHub. The failure is purely register drift — Renovate maintains workflow pins and never touches SECURITY-PIPELINE.md. The check is right and the register is stale, which is exactly what it exists to say.

Blocking on that would fail a required check on routine dependency updates, which is how gates get resented and then bypassed.

The policy this establishes

Recorded in SECURITY-PIPELINE.md under Keeping this register true: when a Renovate pull request bumps an action, the register is updated on that pull request's branch, before merging.

Not afterwards — and the distinction is load-bearing. The check runs on the pull request, so a register fixed after the merge leaves the check red for that pull request's entire life, and makes this step impossible to promote to blocking, because no Renovate bump could ever present a green result to merge on.

Promote to blocking once that habit holds.

Expected immediately after merge

#66 and #67 will show the register finding. That is the check working, not a regression. Under the policy above, each needs a SECURITY-PIPELINE.md commit pushed to its branch before merging. #66 is the interesting one: it converges all seven checkout references to v7.0.1, so the two existing rows collapse into one.

Placement

A step in the existing audit job rather than a new job: the ruleset requires the check named audit, so a step is covered automatically, while a new job would silently not block until someone added it to the ruleset. if: always() for the same reason as the config-validator step above it — one run should report on every half of the policy rather than stopping at the first failure.

Adds no uses: line, so the register's own totals stay accurate at 23 references across 5 actions.

Also verified

semgrep clean across 487 rules over the new script and changed files; the workflow parses with 5 audit steps and the new one carrying continue-on-error: true and if: always(); GITHUB_TOKEN: ${{ github.token }} with the job's existing contents: read, which suffices for public refs.

zizmor validates pin FORMAT — that a `uses:` names a 40-character SHA. It
cannot say the SHA is the right one, so a wrong or hostile digest carrying
a plausible "# v4.4.0" comment passes zizmor, Prettier and review alike.
check-supply-chain.mjs closes that gap: it resolves every digest against
the GitHub API and compares SECURITY-PIPELINE.md's register with the
workflows — digests, versions and multiplicities.

Copied verbatim from ttl-editor's acc, after sgort/ttl-editor#86. Taking it
before that fix would have imported two false positives this repository is
uniquely shaped to trigger: actions/checkout is pinned at two digests here
(v4.4.0 in three workflows, v3.7.0 in four, mid-upgrade) and the register
correctly records both rows, while Azure/static-web-apps-deploy is recorded
as "v1 (branch head)" — more informative than the bare "v1" the workflow
comment carries. Rows are now matched by digest and versions compared on
the leading token. Left unformatted deliberately: Prettier does not cover
scripts/, and reformatting would diverge from upstream for no benefit.

── Why non-blocking, and what it is waiting on ──

Not caution alone. Renovate rewrites workflow pins and their version
comments together, correctly, and never touches SECURITY-PIPELINE.md. So
every action-bump pull request fails the register half until the register
is updated by hand.

Verified against the real thing rather than assumed. Run against the tree
of #66 (actions/checkout to v7.0.1):

  [register] actions/checkout: workflow pins 3d3c42e5aac5… (v7.0.1) but
             SECURITY-PIPELINE.md records only 11d5960a3267… (v4.4.0),
             a37ce9120846… (v3.7.0)

Pin truth passes there — Renovate's digest and comment agree with each
other and with GitHub. The check is right and the register is stale, which
is exactly what it exists to say. Blocking on that would fail a required
check on routine dependency updates, which is how gates get resented and
then bypassed.

The agreed policy, recorded in SECURITY-PIPELINE.md: the register is
updated ON the Renovate pull request's branch, before merging. Not
afterwards — the check runs on the pull request, so a register fixed after
the merge leaves it red for that pull request's whole life and makes this
step impossible to promote, because no bump could ever show a green result
to merge on.

A second reason to wait: this puts a network call inside a job the ruleset
requires, so an API outage or rate limit could fail a gate unrelated to the
change under review. --offline checks format and register agreement without
resolving pins if that proves to be a problem.

── Placement ──

A step in the existing audit job, not a new job: the ruleset requires the
check named `audit`, so a step is covered automatically, while a new job
would silently not block until someone added it to the ruleset. if:
always() for the same reason as the config-validator step above it — one
run should report on every half of the policy rather than stopping at the
first failure. Adds no `uses:` line, so the register's own totals stay
accurate at 23 references across 5 actions.

── Verification ──

Exits 0 on acc. A planted wrong digest with a plausible version comment
fails and names every digest the register records for that action, which is
the #86 fix working on this repository's split pin. A digest changed
without updating the register fails. The Renovate case is the one that does
not pass, documented above and the reason for continue-on-error.

semgrep clean over 487 rules; the workflow parses and the step carries
continue-on-error: true with if: always().
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adopt check-supply-chain: verify pin truth and register agreement

1 participant