Skip to content

fix(release): the tag phase waits for release.yml instead of racing it - #896

Merged
defenwycke merged 1 commit into
mainfrom
fix/tag-phase-waits-for-the-release-workflow
Sep 12, 2026
Merged

defenwycke merged 1 commit into
mainfrom
fix/tag-phase-waits-for-the-release-workflow

Conversation

@defenwycke

Copy link
Copy Markdown
Contributor

The bug

Pushing the tag triggers release.yml, and that workflow is what creates the release carrying the tarballs and the signed SHA256SUMS.txt. On a fresh tag there is a window — seconds to minutes — in which the tag exists and the release does not.

phase_tag looked exactly once and refused:

rel_id=$(gh api "repos/$GH_REPO/releases?per_page=30" --jq '...')
[ -n "$rel_id" ] || die "no release found for $TAG to publish"

v1.11.42 died there at 22:17:26 BST with REFUSED: no release found for v1.11.42 to publish, while the workflow created its release at 22:17:23. A three-second miss, reported as a failure — after all eight nodes were already deployed and serving.

Why it survived four releases

It had only ever worked because the tag was pushed by an earlier aborted run, so the release already existed by the time the phase ran. v1.11.40 is the clearest case: its tag went up during a run that died in gates, the workflow built the draft overnight, and the next day's run found it waiting and published it in one step.

The happy path was an accident of a previous failure. The first genuinely clean, uninterrupted release is the one that exposed it.

The fix

Poll for the workflow's release (TAG_RELEASE_WAIT_SECS, default 600s) and only mint our own if none ever appears — which is what the original gh release create was for, and still covers a repo with no release workflow.

Waiting for the workflow's release rather than racing it also matters because its release is the one with the assets attached.

Tests, and a false pass worth reporting

Two cases added to test-release-orchestrator.sh, with the gh stub extended to model asynchronous creation (the release is invisible for the first N lookups):

  • a release that appears only after several lookups is waited for and published;
  • the wait is bounded, and a repo whose workflow never produces one still gets a release rather than hanging.

A first version of these tests passed against the un-waited code. With the wait removed, the phase falls through, mints its own release, and still exits 0. That is not a pass — it is precisely how a release gets published with no tarballs and no signature while the workflow is still building the real one, which is the state v1.11.42 is in right now.

So the test now also pins that no rival release is created while the workflow's is pending:

[FAIL] minted a rival release while release.yml was still creating one —
       the published release would carry no assets

That assertion does fail against the un-waited code (9 passed / 1 failed), and the suite is 10/10 with the fix.

Scope

scripts/release.sh and its self-test only. No change to any phase ordering, no change to what is built or rolled. The two new knobs are environment-overridable so the self-test can run in seconds.

Pushing the tag TRIGGERS release.yml, and that workflow is what creates the
release carrying the tarballs and the signed `SHA256SUMS.txt`. On a fresh tag
there is a window -- seconds to minutes -- in which the tag exists and the
release does not.

`phase_tag` looked exactly once and refused. v1.11.42 died there at 22:17:26 BST
with "no release found for v1.11.42 to publish" while the workflow created its
release at 22:17:23 -- a three-second miss, reported as a failure AFTER all eight
nodes were already deployed and serving.

It had only ever worked because the tag was pushed by an EARLIER aborted run, so
the release already existed by the time the phase ran. The happy path was an
accident of a previous failure, which is why this survived v1.11.38 through
v1.11.41.

The phase now polls for the workflow's release (`TAG_RELEASE_WAIT_SECS`, default
600s) and only mints its own if none ever appears -- which is what the original
`gh release create` was for, and still covers a repo with no release workflow.

Tests, in `test-release-orchestrator.sh`:
- a release that appears only after several lookups is waited for and published;
- the wait is bounded, and a repo whose workflow never produces one still gets a
  release rather than hanging.

The second assertion matters more than it looks. A first version of these tests
PASSED against the un-waited code, because with the wait removed the phase falls
through and mints its own release and still exits 0. That is not a pass: it is
how a release gets published with no tarballs and no signature while the workflow
is still building the real one -- which is the state v1.11.42 is in. So the test
now also pins that no rival release is created while the workflow's is pending,
and that assertion does fail against the un-waited code.
@defenwycke
defenwycke merged commit 6f801a8 into main Sep 12, 2026
13 checks passed
@defenwycke
defenwycke deleted the fix/tag-phase-waits-for-the-release-workflow branch September 12, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant