Skip to content

DEN-3829: prove GitHub-only recovery when Zed control plane is down - #388

Draft
ORESoftware wants to merge 24 commits into
mainfrom
DEN-3829-github-control-plane-fallback
Draft

ORESoftware wants to merge 24 commits into
mainfrom
DEN-3829-github-control-plane-fallback

Conversation

@ORESoftware

@ORESoftware ORESoftware commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Goal

Make zed provably able to resolve, reconstruct, install, and frozen-restore public packages when the entire Zed-operated network control plane is unavailable: no registry.zpkg.net, no cdn.zpkg.net/R2, no Cloudflare edge dependency, and no Zed application/API server.

Runtime behavior

The configured registry remains the normal first choice. Once that HTTP registry path fails, zed-cli now explicitly orders source recovery as:

  1. GitHub Release assets;
  2. GitHub Packages / GHCR;
  3. a GitHub tag archive, deterministically repacked with Zed's publish rules; then
  4. optional R2 only after the GitHub-owned recovery surfaces fail.

This local ordering deliberately overrides the generic zed-interfaces::artifact_locators() order for the outage-recovery path, because the generic locator list currently places R2 before GitHub. A focused Rust test locks the GitHub-before-R2 invariant.

Current main already contains the broader source-fallback implementation merged in #379: GitHub repository identity is admitted through the committed .zpkg.toml, versions come from tags/releases, GitHub-hosted artifacts are supported, and tag archives can be deterministically repacked. This PR does not replace that implementation with the stale overlapping code from #380.

Ten outage gates

The exact-head workflow proves:

  1. the configured registry is unreachable;
  2. the configured R2/CDN endpoint is a fail-if-used sentinel;
  3. no GitHub token is available for the public-package test;
  4. Go, Python, Dart, Node, and Gleam fixture packages resolve through GitHub fallback;
  5. packages are actually materialized into zed_modules;
  6. a second clean home/store produces the identical lockfile;
  7. --frozen replay succeeds from another clean store;
  8. disabling source fallback makes the outage fail instead of accidentally passing from cache;
  9. a tagged GitHub repository with a mismatched package identity is rejected fail-closed; and
  10. successful recovery performs no R2 request and depends on no successful zpkg.net or Cloudflare response.

The workflow also runs the focused Rust source_fallback tests on the exact PR head.

Recovery model

GitHub is an independent recovery/content plane: repository API + committed manifest + tags/releases/GHCR/tag archives. GitHub fallback does not become a looser authority model; identity ambiguity, manifest mismatch, unsafe archives, and provenance/integrity failures remain hard errors.

Local file:// and loopback registries remain hermetic by default. Control-plane mutations such as publish/yank/namespace claim are outside the availability guarantee.

See docs/github-control-plane-outage.md for the full contract.

@linear-code

linear-code Bot commented Sep 14, 2026

Copy link
Copy Markdown

DEN-3829

Copy link
Copy Markdown
Contributor Author

Exact-head execution update for 7af44a25a0316c57bea89dd8c8da5a782d09a7f3: workflow run 34908684610 (GitHub control-plane outage fallback) completed successfully with a real GitHub-hosted runner (runner_id=1000087017) and stepful evidence, not a zero-step admission artifact. The job successfully checked out the exact candidate, installed the pinned Rust toolchain, built with locked dependencies, ran focused fallback/security/archive tests, installed a five-language consumer with the registry down + R2 forbidden + no GitHub token, proved clean-store lock determinism, proved frozen replay from another empty store, proved successful recovery did not touch R2/Cloudflare, proved source fallback was required rather than accidental cache reuse, and rejected a tagged repository with a mismatched package identity. Other workflows on this head still include unrelated failures (cli-portability, fp-conformance, repository hardening, zed tip dependency canary, ci), so this comment certifies the outage contract only and does not claim the whole PR is globally green.

Copy link
Copy Markdown
Contributor Author

Continuation-batch acceptance note: preserve the outage order as GitHub Release -> GHCR/Packages -> GitHub tag archive -> optional R2, with registry first only during normal operation. The next promotion evidence should explicitly bind the exact zed-lock/source-fallback revisions and prove a clean-home install plus frozen replay while both zpkg.net and the R2 sentinel are unavailable; no successful Cloudflare request should be necessary.

Copy link
Copy Markdown
Contributor Author

Exact-head recheck on 7944b5cd58a7597e7e07b7f312601f1804e7cb23: the dedicated Release graph, GitHub-only fallback contract, and GitHub fallback outage contract workflows are green, but the broader Repository hardening, CI, Private and partner consumer lane, and Deterministic run-contract smoke workflows are genuinely stepful failures on this same head. This is therefore source/policy regression evidence rather than runner-admission noise. Keeping the PR draft/merge-blocked and preserving the previously proved outage-fallback contract; no blind rerun of deterministic failures.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant