Skip to content

Harden GHCR publish step against transient 5xx failures - #52

Draft
callenflynn with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-github-actions-job
Draft

callenflynn with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-github-actions-job

Conversation

Copilot AI commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

The scheduled container build was failing in the Build nvidia (amd64) job during image publish, despite successful build/rechunk/tag stages. Root cause was an unhandled transient 500 Internal Server Error from GHCR during podman push.

  • Failure mode addressed

    • Push To GHCR previously failed fast on the first registry-side error, aborting the entire workflow run.
  • Workflow change

    • Added bounded retry logic to the GHCR push loop in .github/workflows/build.yml.
    • Each tag push now retries up to 3 times with incremental backoff before failing hard.
    • Preserved existing behavior for successful first-attempt pushes and digest output handling.
  • Resulting behavior

    • Transient GHCR failures no longer fail the run immediately.
    • Persistent failures still fail deterministically with explicit error output.
max_attempts=3
for tag in ${ALIAS_TAGS}; do
  for attempt in $(seq 1 "${max_attempts}"); do
    if sudo -E podman push --digestfile=/tmp/digestfile \
      ${IMAGE_NAME}:${tag} ${IMAGE_REGISTRY}/${IMAGE_NAME}:${tag}; then
      break
    fi
    [[ "${attempt}" -eq "${max_attempts}" ]] && exit 1
    sleep $((attempt * 10))
  done
done

Copilot AI balanced review requested due to automatic review settings October 9, 2026 10:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.

Co-authored-by: callenflynn <170468817+callenflynn@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 10:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI changed the title [WIP] Fix failing GitHub Actions job Harden GHCR publish step against transient 5xx failures Oct 9, 2026
Copilot AI requested a review from callenflynn October 9, 2026 10:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants