ci(deploy): dump the instance console when the health gate fails [skip release] - #231
Merged
Merged
Conversation
…p release] When deploy-server's /health poll fails the box was replaced but never served, and the only place that failure shows is the instance console: user-data runs under cloud-init and nothing reaches the /vegify/server log group until the server is up. On 2026-09-15 three deploys died at that poll and the cause (dnf OOM-killed on a new AL2023 AMI) was visible only there. - deploy.yml: a failure-only step after the poll prints the last 200 console lines of the VegifyServer instance from cdk-out.json (assign-then-print so a CLI failure is reported, not hidden). - ci-stack.ts: the GitHub deploy role gets ec2:GetConsoleOutput, read-only, scoped to instances in the deploy region. deploy-ci.yml ships it on merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #230. When
deploy-server's/healthpoll fails, the instance was replaced but never served, and the only place that shows is the instance console: user-data runs under cloud-init, and nothing reaches the/vegify/serverlog group until the server is up. Today's three failed deploys (v1.0.50–v1.0.52) had their cause, dnf OOM-killed on a new AL2023 AMI, visible only inaws ec2 get-console-output, which nothing in the run could show.Changes
deploy.yml: a failure-only step after the health poll prints the last 200 console lines of theVegifyServerinstance (id fromcdk-out.json). Assign-then-print, so a CLI failure is reported rather than hidden by a pipe. Skips cleanly when the deploy step itself failed and there is nocdk-out.json.ci-stack.ts: the GitHub deploy role gainsec2:GetConsoleOutput, read-only, scoped toarn:aws:ec2:us-east-1:<account>:instance/*.deploy-ci.ymlpath-triggers on this file, so the merge deploys VegifyCi itself.[skip release]: neither file is a server- or web-relevant path, so the deploy run this merge triggers deploys nothing and cuts no version.Verification
pnpm --filter @vegify/infra buildclean;cdk --app 'tsx bin/ci.ts' synth VegifyCirenders the new statement asReadInstanceConsoleOnFailedHealthGate.deploy-serverwithif: failure().just checkgreen.