check-supply-chain was adopted in #77 as a non-blocking step in the audit job. That was the right way in, and it should not stay that way.
Today it is a log entry, not a signal
continue-on-error: true does more than keep the job green. It rewrites the step's reported conclusion too, and the honest result (outcome: failure) is not exposed by the REST API at all. Observed on #66 before its register was updated — the audit run for that pull request:
job: audit
job conclusion: success
step: Verify pin truth and register agreement -> success
while that very step's log read:
1 finding(s):
[register] actions/checkout: workflow pins 3d3c42e5aac5… (v7.0.1) but
SECURITY-PIPELINE.md records only 11d5960a3267… (v4.4.0), a37ce9120846… (v3.7.0)
So a real finding was present, and every surface a reviewer would look at said success — the PR checks list, the job, and the step. Only opening the step's log distinguished them.
That is the problem. A check nobody sees fail is not protecting anything; it is a check that has to be remembered. The register drifted in the first place because nothing was watching, and a non-blocking check restores that condition with extra steps.
Why it was non-blocking, and what has changed
Two reasons were recorded in #73 and in the workflow comment.
1. Renovate would fail it on every action bump. Renovate maintains workflow pins and never touches SECURITY-PIPELINE.md, so every bump fails the register half until the register is updated by hand. Blocking on that would fail a required check on routine dependency updates — how gates get resented and then bypassed.
This is now addressed. The agreed habit, recorded in SECURITY-PIPELINE.md under Keeping this register true, is that the register is updated on the bump's branch, before merging. #66 is the first exercise of it: the register was moved to v7.0.1 on that branch and the check went green there, before the merge rather than after.
2. A network call inside a required job. Resolving digests against the GitHub API means an outage or rate limit could fail a gate unrelated to the change under review. Still true, and the reason to gather a little evidence rather than flip immediately.
Proposed
Let the habit prove itself over the next few action bumps, then remove continue-on-error: true.
A smaller alternative, if blocking still feels early
Keep it non-blocking but make a finding visible — have the step write to $GITHUB_STEP_SUMMARY on failure, so the finding appears on the run page instead of only in the log. That fixes the invisibility without touching the gate. It is strictly worse than blocking, and better than the current state.
Related
check-supply-chainwas adopted in #77 as a non-blocking step in theauditjob. That was the right way in, and it should not stay that way.Today it is a log entry, not a signal
continue-on-error: truedoes more than keep the job green. It rewrites the step's reported conclusion too, and the honest result (outcome: failure) is not exposed by the REST API at all. Observed on #66 before its register was updated — the audit run for that pull request:while that very step's log read:
So a real finding was present, and every surface a reviewer would look at said success — the PR checks list, the job, and the step. Only opening the step's log distinguished them.
That is the problem. A check nobody sees fail is not protecting anything; it is a check that has to be remembered. The register drifted in the first place because nothing was watching, and a non-blocking check restores that condition with extra steps.
Why it was non-blocking, and what has changed
Two reasons were recorded in #73 and in the workflow comment.
1. Renovate would fail it on every action bump. Renovate maintains workflow pins and never touches
SECURITY-PIPELINE.md, so every bump fails the register half until the register is updated by hand. Blocking on that would fail a required check on routine dependency updates — how gates get resented and then bypassed.This is now addressed. The agreed habit, recorded in
SECURITY-PIPELINE.mdunder Keeping this register true, is that the register is updated on the bump's branch, before merging. #66 is the first exercise of it: the register was moved to v7.0.1 on that branch and the check went green there, before the merge rather than after.2. A network call inside a required job. Resolving digests against the GitHub API means an outage or rate limit could fail a gate unrelated to the change under review. Still true, and the reason to gather a little evidence rather than flip immediately.
Proposed
Let the habit prove itself over the next few action bumps, then remove
continue-on-error: true.auditgreen before merge, no one blocked. chore(deps): update actions/checkout action to v7 #66 and chore(deps): update actions/setup-node action to v7 #67 are the first two.--offline, which checks format and register agreement without resolving pins — it keeps the register half blocking and drops only the network-dependent half.continue-on-error: true. No ruleset change needed: the step lives in the existingauditjob, which is already the required check.SECURITY-PIPELINE.mdto say it blocks, and why it did not at first.A smaller alternative, if blocking still feels early
Keep it non-blocking but make a finding visible — have the step write to
$GITHUB_STEP_SUMMARYon failure, so the finding appears on the run page instead of only in the log. That fixes the invisibility without touching the gate. It is strictly worse than blocking, and better than the current state.Related
sgort/ronl-business-apihas the same adoption issue open and will land in the same non-blocking state; the same promotion applies there.