Skip to content

Poll for block progress and retry toolchain installs - #6

Open
nikolap-dat wants to merge 2 commits into
mainfrom
ci/robust-health-and-toolchain
Open

nikolap-dat wants to merge 2 commits into
mainfrom
ci/robust-health-and-toolchain

Conversation

@nikolap-dat

@nikolap-dat nikolap-dat commented Oct 8, 2026 •

Copy link
Copy Markdown

Summary

Two transient conditions failed real program deploys today with nothing wrong on the chain:

  1. Health probe. arch-kit health sampled the block height twice, 2 s apart, and failed when it had not moved. A healthy chain still has multi-second gaps between consecutive blocks: 3.5 s between heights 48004 and 48005 on internal-b while it averaged ~9 blocks/s. prop-amm's deploy (run 37778455694) landed its single probe in that gap: blocks are not progressing … height changed from 48005 to 48005 over 2s.
    The check now polls the height once a second for up to 20 s and succeeds as soon as it moves; observation_window reports the time that took, and a sampling error still ends the check immediately. src/client.rs, with unit tests for the polling helper; README updated.

  2. Toolchain install. The Anza installer and the platform-tools download behind cargo build-sbf both go through the GitHub release API, which intermittently returns a body the installer cannot parse. ausd-bridge's testnet deploy (run 37786560414) died on Failed to install platform-tools: Failed to parse Github response: error decoding response body.
    The reusable deploy-program.yml now installs the toolchain and prefetches platform-tools (cargo-build-sbf --install-only) with three attempts, installs arch-kit with three attempts, and probes health up to five times (covers a consumer pinned to an older arch-kit, or a node mid-restart).

Version bumped to 0.1.9 so the CLI change can be published; consumers pinned to the reusable workflow pick up the retries by moving their ref to the merge commit. The per-repo adapters (ausd-bridge, btc-wrapper, CLAMM, arch-vaults, arch-lend) already retry the health probe but install the toolchain in one attempt; that's a separate small change in each repo.

Test plan

  • cargo fmt --all -- --check, cargo test --locked --all-targets (109 passed), cargo clippy --locked --all-targets -- -D warnings
  • Workflow YAML parses
  • Publish 0.1.9 and bump an adapter to it: arch-kit health against internal-b reports a sub-second observation window
  • prop-amm pin moved to the merge commit: a real deploy passes the install and health steps

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Nikola Petkovic and others added 2 commits October 8, 2026 15:52
Two transient conditions failed real deploys today without anything being
wrong with the chain or the program:

- `arch-kit health` sampled the block height twice, 2 s apart, and failed
  when it had not moved. A healthy chain still has multi-second gaps between
  consecutive blocks (3.5 s between heights 48004 and 48005 on a chain
  averaging nine blocks per second), so one probe that lands in such a gap
  killed prop-amm's deploy. The check now polls once a second for up to 20 s
  and succeeds as soon as the height moves; `observation_window` reports the
  time that took. A sampling error still ends the check at once.

- The Anza installer and the platform-tools download behind `cargo
  build-sbf` go through the GitHub release API, which intermittently returns
  a body the installer cannot parse ("Failed to install platform-tools:
  Failed to parse Github response"). The reusable workflow now installs the
  toolchain and prefetches platform-tools with three attempts, installs
  arch-kit with three attempts, and probes health up to five times so a
  pinned older arch-kit or a restarting node does not fail the deploy.

Version 0.1.9 so the CLI change can be published.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
publish-crate.yml refuses a release unless the workflow installs the same
version Cargo.toml declares; the bump left ARCH_KIT_VERSION at 0.1.8. The
README example moves to the matching tag.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@itsarchnw

Copy link
Copy Markdown

ARCH_KIT_VERSION in deploy-program.yml is still 0.1.8. The publish workflow checks that against Cargo.toml and exits 1 on mismatch, so publishing 0.1.9 will fail with "deploy-program.yml installs 0.1.8, expected 0.1.9".

If it merges as-is the reusable workflow also keeps installing 0.1.8, so consumers pick up the retries but not the health polling change.

Needs the env bumped to 0.1.9.

@itsarchnw itsarchnw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Versions line up now and the publish check will pass. LGTM.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants