Skip to content

fix(release-mirror): publish a release in one run, and on demand - #80

Merged
yaacovcorcos merged 4 commits into
mainfrom
fix/release-mirror-speed-20261121
Nov 21, 2026
Merged

yaacovcorcos merged 4 commits into
mainfrom
fix/release-mirror-speed-20261121

Conversation

@yaacovcorcos

Copy link
Copy Markdown
Contributor

A new desktop release reached the update feeds 20–30 minutes after it was published. With this change it should take a few minutes, and the desktop release workflow can start the mirror the moment it publishes (scient-desktop PR to follow).

Why it was slow

From the worker's own logs:

Release Published Feed switched Runs
Stable 0.6.31 21:54 22:21 22:00 exceededCpu (32 s), 22:10 GitHub 429, 22:20 ok (30.6 s CPU)
Beta 0.6.32-beta.20261121.1 17:18 17:41 17:20 and 17:30 exceededCpu, 17:40 ok
Stable 0.6.32 19:26 ~19:41 19:30 copied, cut short; next run finished
  • CPU limit. The download-page step copied the four Stable installers checked only by SHA-256. The Stable feed step then read each one back from R2 to check the manifest's SHA-512 (mirror.ts verifyExisting). That re-read alone used about 28 s of the default 30 s.
  • 429. github.com download URLs refused some runs. Those runs fire on the round ten minutes, when Cron Triggers everywhere hit GitHub from Cloudflare's shared addresses.

What changed

  • limits.cpu_ms: 300000 (Workers Paid). Hashing one channel's release takes about 40 s.
  • Order: the Stable feed now runs before the download-page mirror. It writes every Stable installer with both digests, so the mirror step's verifyExisting passes without reading anything back. Each installer is hashed once.
  • Retries: every GitHub fetch goes through fetchFromGitHub. It retries 429/502/503/504 up to three attempts, honoring Retry-After up to 30 s, and gives each attempt its own timeout.
  • Schedule: 3-59/10 * * * * instead of */10.
  • Trigger: POST /sync?channel=stable|beta on the workers.dev address (workers_dev: true; preview_urls stays off).
    • Auth: bearer token compared in constant time against the MIRROR_SYNC_TOKEN secret. It answers 404 while the secret is unset, 401 for a wrong token, 405 for other methods and 400 for an unknown channel.
    • It streams NDJSON: each step's result, a heartbeat every 15 s, then {"message":"Release mirror finished","failures":n}.
    • An HTTP invocation has no wall-time limit while the caller stays connected. A caller that leaves only cuts the run short, and the next scheduled run continues from the bucket. Concurrent runs were already safe (immutable If-None-Match: * writes and compare-and-swap on the feeds and pointer).
  • The scheduled run behaves as before, apart from the order. It fails the invocation when any step fails.

Not done: a GitHub token

The 429s come from github.com download URLs, where a token doesn't help. Authenticated downloads would mean moving to the API's asset endpoints and creating a new credential. The retries and the schedule offset address the refusals that were observed.

Tests

  • New github.test.ts: honours Retry-After, caps it at 30 s, backs off without it, returns the third refusal, and doesn't retry a 404.
  • New index.test.ts: step order (Stable feed, then mirror, then Beta), a failed step doesn't stop the others, single-channel runs, the scheduled run fails on a failed step, and /sync streaming plus its 404/401/405/400 answers.
  • mirror.test.ts: the 429 test now expects three attempts, and a new test covers recovery after one 429.
  • bun run check passes locally. wrangler deploy --dry-run bundles the worker.

Deploy (after merge)

bun run mirror:deploy, then set the token with README step 3. The Cron schedule changes on deploy.

Documentation impact

Updated: README.md (runbook: schedule, CPU limit, token setup, stop procedure) and docs/architecture/desktop-downloads.md (order, retries, new "Run on publish" section).

Review and merge

Quality Review is by the author, and Integration Readiness Review is by Codex (read-only), recorded below. The product owner explicitly asked for this to be merged when Codex approves. There's no user-visible page change, so no preview evidence applies.

🤖 Generated with Claude Code

A release reached the update feeds two or three runs (20-30 minutes) after
it was published. The worker logs show why:

- Runs were stopped at the 30 s CPU limit. The Stable feed step read back
  every installer the download-page step had just copied, to hash it with
  the manifest's SHA-512 (30.6 s of CPU for 1.3 GB on 11-19).
- GitHub answered some runs with 429 at the round ten minutes.

Changes:

- limits.cpu_ms = 300000; a channel's release needs about 40 s of CPU.
- The Stable feed runs before the download-page mirror, so the mirror step
  finds every installer already verified and hashes nothing again.
- GitHub fetches retry 429 and gateway errors, up to three attempts,
  honoring Retry-After up to 30 s.
- The schedule moves to three minutes past every ten.
- POST /sync?channel=stable|beta (workers.dev, bearer token in the
  MIRROR_SYNC_TOKEN secret, 404 while unset) runs the mirror now and
  streams one JSON line per step, for the desktop release workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Deploying scientfactory-website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 5974f89
Status: ✅  Deploy successful!
Preview URL: https://4fd884ed.scientfactory-website.pages.dev
Branch Preview URL: https://fix-release-mirror-speed-202.scientfactory-website.pages.dev

View logs

yaacovcorcos and others added 3 commits November 21, 2026 22:03
…ries

From Codex's review of #80:
- The pointer step refuses a handoff older than the pointer it read: a
  stale GitHub answer after the trigger promoted a newer release could
  otherwise swap the pointer back, since the pointer had not changed.
- Retry-After is also read as an HTTP date (capped at 30 s), and cancelling
  a refused response's body can no longer stop the retry.
- Tests: a whole Stable run reads no installer back; when the feed step
  could not run first, the next run reads them back once (documented).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…guard

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From Codex's recheck of #80: a failed read of the pointer's body was
treated like a malformed pointer, so the run kept its ETag and a stale
GitHub handoff could still replace a newer pointer. Now only bytes read
and found malformed may be replaced; a failed read rejects the run.
Also qualifies when installers are read back again (until a feed run
publishes the release).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yaacovcorcos

Copy link
Copy Markdown
Contributor Author

Integration Readiness Review: Codex (read-only, gpt-6.2-sol high), three rounds. Round 1 raised a pointer rollback on a stale GitHub answer, overstated hash-once wording, and retry edge cases. Round 2 raised a failed pointer read still permitting rollback. All were fixed. Round 3 on 5974f89 returned APPROVE: 52 mirror tests and 10 Pages release-resolution tests passed, and mirror:typecheck passed. The product owner authorized merging once Codex approved.

@yaacovcorcos
yaacovcorcos merged commit 068f04b into main Nov 21, 2026
2 checks passed
@yaacovcorcos
yaacovcorcos deleted the fix/release-mirror-speed-20261121 branch November 21, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant