Skip to content

fix(fallback): install from github.com tag archives when zpkg.net is down - #379

Merged
ORESoftware merged 3 commits into
mainfrom
fix/source-fallback-github-archive-digest
Sep 13, 2026
Merged

ORESoftware merged 3 commits into
mainfrom
fix/source-fallback-github-archive-digest

Conversation

@ORESoftware

@ORESoftware ORESoftware commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Part of #378. zed-cli has to install from github.com when both registry.zpkg.net and the Cloudflare edge (cdn.zpkg.net) are down.

Result

Verified live on 2026-09-12/13, while registry.zpkg.net returned 503 and cdn.zpkg.net returned 404, in a fresh clone of ores-wasm-loaders/owls-runtime.rs:

Command Before After
zed install --frozen --adapter none --install-mode copy (the repo's own deps task) failed: 16 × 404, tag archive never tried installs owls-interfaces@0.1.1 from github.com with the locked digest; cargo check --locked passes
zed fetch --frozen --output … (Nix / offline bundle) failed: lock names a vanished file:///tmp registry publishes a bundle with sha256 fea1f81d… and commit 291406191b…

What changed

src/source_fallback.rs

  1. The tag archive stays in the chain even when a digest is pinned. It is the last locator. It used to be skipped whenever a digest was known, including the digest the fallback had just derived from that same archive. Each prefetch worker opens its own registry, so no per-instance state could fix this. Callers still verify bytes against the pin.

  2. Tag archives are rebuilt exactly as zed publish packs them. The fallback:

    • extracts the tree; links, specials, escaping paths and oversize input fail closed
    • checks the declared org/name@version against the request
    • applies the same manifest hardening zed pack uses (pack_guard and pack_inputs)
    • runs pack::pack_all and keeps the root package

    The result is byte-identical to the published artifact:

    • owls-interfaces v0.1.0 tag → 3ed5a04a…, the published Release asset
    • v0.1.1 tag → fea1f81d…, the digest locked in owls-runtime.rs

    A pinned lock can therefore be satisfied from github.com alone.

  3. Fallback metadata records the tag's commit, from the GitHub commits API; only a full 40-hex id is accepted. Frozen restores compare VCS provenance exactly.

  4. Doubled mirror URLs. R2 locators collapse to the public base (they used to become <object>.tar.gz/artifacts/<sha>), and registry locators are dropped.

  5. Each failed attempt now keeps its full error chain.

src/fetch.rs + docs/frozen-fetch.md

  1. When a lock's well-formed file:// source directory does not exist on this machine, frozen fetch reads the pinned version from the configured registry (and its GitHub fallback). All lock fields are still verified, and a degraded note is emitted without echoing the path. immutable-nix-store-input sources never fall back.

Also

  • Cargo.lock: uses the zed-lock rev Cargo.toml already pins (1db0da0), the same change carried by 17 open branches. main's stale lock fails every --locked job, including this PR's proposed-head rust against dependency tip.
  • Removes the duplicate registry_for import in src/install_graph/hardening_tests.rs that broke cargo test --lib on main (same change as DEN-567: close remaining secret argv residue (#350) #377).

Tests

  • cargo test --locked --lib: 607 passed. cargo test --locked --all-targets --no-run builds.
  • New tests:
    • tag_archives_repack_into_the_published_root_artifact
    • tag_archive_identity_must_match_the_request
    • tag_archive_links_fail_closed
    • tag_commits_must_be_full_lowercase_git_ids
    • pinned_downloads_try_the_tag_archive_last
    • fallback_mirrors_use_object_store_bases_not_object_urls
    • missing_local_lock_source_reads_the_pin_from_the_configured_registry
  • Hosted CI: every other failure on the previous head (ci, repository hardening, Exact-head graph authority, GitHub Actions fail-closed policy, source-policy-lint, …) fails identically on main @ a80dad7, with real steps executed.
  • cargo clippy -- -D warnings reports nothing new in non-test code. The remaining hits in these files are on main already, plus the repo's usual unwrap() in test fixtures.

Not in this PR (tracked in #378)

  • Local-disk resolution: salvage of feat/local-project-registry, in a separate PR.
  • Publishing that requires a GitHub Release asset, plus a backfill command (less urgent now that tags reproduce published digests).
  • A zed-pkg-test end-to-end test that blocks zpkg.net.

🤖 Generated with Claude Code

ORESoftware and others added 2 commits September 12, 2026 17:13
The registry-down fallback could not install a package that has a git tag
but no published zpkg Release asset (#378):

- download_locators skipped the GitHub tag archive whenever a digest was
  known, including the digest the fallback had just derived from that
  archive, so every install failed after metadata resolved.
- A raw `git archive` tarball has a `<repo>-<tag>/` root and a
  pax_global_header entry, which the store extractor rejects. It is now
  repacked into the deterministic `pkg/` artifact layout (sorted entries,
  zeroed headers), so the derived digest names the tree, not GitHub's
  compressor. Links and specials still fail closed.
- Fallback metadata turned full R2 object URLs into object-store mirror
  bases, producing doubled URLs such as `<object>.tar.gz/artifacts/<sha>`.
  R2 locators now collapse to the public base and registry locators are
  dropped.

Also drops the duplicate `registry_for` import that stopped the lib test
target from compiling on main.

Verified live against the 2026-09-12 registry.zpkg.net 503 outage:
ores-wasm-loaders/owls-interfaces@0.1.1 now downloads and unpacks from
GitHub. cargo test --lib: 604 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s them

Follow-up to a983843 (#378). A raw tag archive still could
not install: its polyglot manifest keeps `[targets]`, so a consumer that
infers `rust` was refused, and its digest could never match a lockfile.

The fallback now extracts the tag tree (links, specials, escaping paths
and oversize input fail closed), checks the declared org/name/version,
applies the same manifest hardening `zed pack` uses, and packs it with
`pack::pack_all`, keeping the root package. Packing is deterministic, so
the fallback reproduces the published artifact byte for byte:

- owls-interfaces v0.1.0 tag -> 3ed5a04a…, the published Release asset
- owls-interfaces v0.1.1 tag -> fea1f81d…, the digest pinned in
  ores-wasm-loaders/owls-runtime.rs/.zpkg.lock

Verified live while registry.zpkg.net returns 503 and cdn.zpkg.net 404:
`zed install --frozen --adapter none --install-mode copy` in a fresh
owls-runtime.rs clone installs from github.com, and `cargo check --locked`
passes. cargo test --lib: 605 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ORESoftware ORESoftware changed the title fix(fallback): install from GitHub tag archives when zpkg.net is down fix(fallback): install from github.com tag archives when zpkg.net is down Sep 13, 2026
@ORESoftware
ORESoftware marked this pull request as ready for review September 13, 2026 02:32
`zed fetch --frozen` (the Nix/offline bundle path) still could not restore
a lock during the registry outage (#378):

- A committed lock can name a machine-local `file://` registry that does
  not exist here (owls-runtime.rs pins `file:///tmp/ores-wasm-registry`).
  When that directory is absent, fetch now reads the pinned version from
  the configured registry, and through its fallback from GitHub. It
  records a degraded note that does not echo the path.
  `immutable-nix-store-input` sources never fall back. Documented in
  docs/frozen-fetch.md.
- Fallback metadata carried no VCS commit, so the exact provenance check
  refused a digest-identical artifact. The fallback now records the
  commit the tag points at, from the GitHub commits API; a full 40-hex id
  only, otherwise the field stays empty and the check fails closed.
- The repacked tag archive was copied into a store cache directory that a
  fresh isolated fetch store has not created yet. Create it first, and
  keep full error chains in the fallback attempt list.
- Cargo.lock: take the zed-lock rev Cargo.toml already pins (1db0da0), as
  17 open branches do. main's stale lock fails every `--locked` job.

Verified live (registry.zpkg.net 503, cdn.zpkg.net 404) in a fresh
owls-runtime.rs clone:
- `zed fetch --frozen` publishes a bundle with sha256 fea1f81d…, commit
  291406191b…
- `zed install --frozen --adapter none --install-mode copy` succeeds.
cargo test --locked --lib: 607 passed; --all-targets builds under --locked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Baseline correction before promotion: please do not carry the branch-local Cargo.lock downgrade from zed-lock@1a95bb87272ac7b08895cea1f61743cf40321213 to 1db0da00d30fcf2e0762f50eedb1f88458020b52. #303 previously admitted 1a95bb87… specifically to preserve both Windows ERROR_SHARING_VIOLATION (32) and ERROR_LOCK_VIOLATION (33), including the added libc dependency. Sep-12 main retained that newer lock entry but regressed Cargo.toml/provenance assertions to 1db0…. The correct baseline repair is forward: restore Cargo.toml and stale source assertions to 1a95…; keep this PR's fallback changes independently attributable. Please remove the lock downgrade from the effective feature diff rather than making the feature depend on a semantic regression.

@ORESoftware
ORESoftware merged commit a1c5b7a into main Sep 13, 2026
57 of 93 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant