fix(release): the tag phase waits for release.yml instead of racing it - #896
Merged
Merged
Conversation
Pushing the tag TRIGGERS release.yml, and that workflow is what creates the release carrying the tarballs and the signed `SHA256SUMS.txt`. On a fresh tag there is a window -- seconds to minutes -- in which the tag exists and the release does not. `phase_tag` looked exactly once and refused. v1.11.42 died there at 22:17:26 BST with "no release found for v1.11.42 to publish" while the workflow created its release at 22:17:23 -- a three-second miss, reported as a failure AFTER all eight nodes were already deployed and serving. It had only ever worked because the tag was pushed by an EARLIER aborted run, so the release already existed by the time the phase ran. The happy path was an accident of a previous failure, which is why this survived v1.11.38 through v1.11.41. The phase now polls for the workflow's release (`TAG_RELEASE_WAIT_SECS`, default 600s) and only mints its own if none ever appears -- which is what the original `gh release create` was for, and still covers a repo with no release workflow. Tests, in `test-release-orchestrator.sh`: - a release that appears only after several lookups is waited for and published; - the wait is bounded, and a repo whose workflow never produces one still gets a release rather than hanging. The second assertion matters more than it looks. A first version of these tests PASSED against the un-waited code, because with the wait removed the phase falls through and mints its own release and still exits 0. That is not a pass: it is how a release gets published with no tarballs and no signature while the workflow is still building the real one -- which is the state v1.11.42 is in. So the test now also pins that no rival release is created while the workflow's is pending, and that assertion does fail against the un-waited code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Pushing the tag triggers
release.yml, and that workflow is what creates the release carrying the tarballs and the signedSHA256SUMS.txt. On a fresh tag there is a window — seconds to minutes — in which the tag exists and the release does not.phase_taglooked exactly once and refused:v1.11.42 died there at 22:17:26 BST with
REFUSED: no release found for v1.11.42 to publish, while the workflow created its release at 22:17:23. A three-second miss, reported as a failure — after all eight nodes were already deployed and serving.Why it survived four releases
It had only ever worked because the tag was pushed by an earlier aborted run, so the release already existed by the time the phase ran. v1.11.40 is the clearest case: its tag went up during a run that died in gates, the workflow built the draft overnight, and the next day's run found it waiting and published it in one step.
The happy path was an accident of a previous failure. The first genuinely clean, uninterrupted release is the one that exposed it.
The fix
Poll for the workflow's release (
TAG_RELEASE_WAIT_SECS, default 600s) and only mint our own if none ever appears — which is what the originalgh release createwas for, and still covers a repo with no release workflow.Waiting for the workflow's release rather than racing it also matters because its release is the one with the assets attached.
Tests, and a false pass worth reporting
Two cases added to
test-release-orchestrator.sh, with theghstub extended to model asynchronous creation (the release is invisible for the first N lookups):A first version of these tests passed against the un-waited code. With the wait removed, the phase falls through, mints its own release, and still exits 0. That is not a pass — it is precisely how a release gets published with no tarballs and no signature while the workflow is still building the real one, which is the state v1.11.42 is in right now.
So the test now also pins that no rival release is created while the workflow's is pending:
That assertion does fail against the un-waited code (9 passed / 1 failed), and the suite is 10/10 with the fix.
Scope
scripts/release.shand its self-test only. No change to any phase ordering, no change to what is built or rolled. The two new knobs are environment-overridable so the self-test can run in seconds.