diff --git a/.github/workflows/workers.yml b/.github/workflows/workers.yml index 8057e67bf..4075e25de 100644 --- a/.github/workflows/workers.yml +++ b/.github/workflows/workers.yml @@ -120,6 +120,17 @@ jobs: if: inputs.operation == 'deploy' && inputs.environment == 'canary' && (inputs.worker == 'both' || inputs.worker == 'documentation') run: npx wrangler deploy --env canary --config workers/docs/wrangler.jsonc + - name: Verify production documentation + if: inputs.operation == 'deploy' && inputs.environment == 'production' && (inputs.worker == 'both' || inputs.worker == 'documentation') + run: | + version=$(node -p "JSON.parse(require('fs').readFileSync('workers/docs/wrangler.jsonc', 'utf8')).env.production.vars.LATEST_DOC_VERSION") + for attempt in 1 2 3 4 5 6 7 8 9 10 11 12; do + if node scripts/docs/smoke-worker.mjs https://www.gecode.dev "$version"; then exit 0; fi + if [ "$attempt" -eq 12 ]; then exit 1; fi + echo "Waiting for the production deployment to propagate (attempt $attempt)." + sleep 30 + done + - name: Verify documentation canary if: inputs.operation == 'deploy' && inputs.environment == 'canary' && inputs.worker == 'documentation' run: | @@ -133,7 +144,7 @@ jobs: - name: Remove documentation canary if: inputs.operation == 'remove-canary' - run: npx wrangler delete --env canary --config workers/docs/wrangler.jsonc --force + run: node scripts/docs/remove-canary.mjs - name: Deploy redirect Worker to staging if: inputs.operation == 'deploy' && inputs.environment == 'staging' && (inputs.worker == 'both' || inputs.worker == 'redirects') diff --git a/docs/deployment-runbook.md b/docs/deployment-runbook.md index 97fcd3701..8e2a06d03 100644 --- a/docs/deployment-runbook.md +++ b/docs/deployment-runbook.md @@ -15,15 +15,16 @@ small Pages fallback files. As checked on 5 September 2026, Cloudflare is authoritative for `gecode.dev`. Cloudflare proxies the apex GitHub Pages A records and the `www` CNAME to `gecode.github.io` with Full (strict) TLS and Always Use HTTPS. GitHub Pages -still serves the production website and documentation; the Astro artifact has -not been deployed. +still serves the active website. The 6.4.0 documentation canary now uses R2; +other versions and aliases await production rollout. Astro has not been deployed. Cloudflare Email Routing is ready. Its managed MX, SPF, and DKIM records are authoritative, both forwarding destinations are verified, and the catch-all rule sends mail through the checked-in `gecode-email-routing` Worker. The private R2 documentation archive and `docs-staging.gecode.dev` Worker custom -domain are live and pass the phase 4 smoke tests. No canary or production -documentation Worker routes exist yet. +domain are live and pass the phase 4 smoke tests. Full R2 verification passed +for all nine versions (52,385 files / 1,138,898,740 bytes). The 6.4.0 canary +passes live checks and its two routes are fail-closed. The `cloudflare-staging`, `cloudflare-canary`, and `cloudflare-production` GitHub environments contain the stable Cloudflare account ID and the dedicated @@ -36,7 +37,8 @@ The [5 September readiness review](migration-readiness-review-2026-09-05.md) records the local fixes, rollback build, verification limits and remaining work. Phases 2–4 below describe the original migration sequence; DNS delegation and infrastructure setup are largely complete. Staging deployment through -GitHub now passes. Continue with clean-checkout website CI and phase 5 checks. +GitHub and clean-checkout website CI pass. The website migration is merged +in PR #7; Pages remains manual. Continue with phase 5 production rollout. ## Cutover overview @@ -116,12 +118,14 @@ The successful [staging run](https://github.com/Gecode/gecode.github.io/actions/ used commit `5ec53f3b111a46d92c42f8b996d79b86dc39cd80` on `codex/verify-docs-staging-20260905`. That branch contains a limited Worker snapshot with a staging-only push workflow and suppresses the old Pages -workflow for its branch. Land the complete website migration separately. +workflow for its branch. The complete website migration was subsequently +merged in [PR #7](https://github.com/Gecode/gecode.github.io/pull/7). Worker tests, dry-run builds, deployment and all HTTP smoke checks passed. The deployed `gecode-documentation-staging` version is `f9295ce3-1641-4727-b3d9-8f0c7f31b016`. The pre-rehearsal version was -`8bfa5cfb-1c96-4cb6-8f57-d5d2bb85080c`. Production and canary zone routes remain -absent, and the classic production pages still return 200. +`8bfa5cfb-1c96-4cb6-8f57-d5d2bb85080c`. The later +[canary run](https://github.com/Gecode/gecode.github.io/actions/runs/33967976199) +passed from merged `main`. Classic production pages still return 200. To replace the token, create a new token with those permissions, set `CLOUDFLARE_API_TOKEN` in all three environments, verify a staging deployment, @@ -231,14 +235,21 @@ Worker route remains sufficient rollback. documentation Worker. 5. Deploy the production documentation routes and rerun the smoke tests. Confirm `/doc/sitemap.xml` serves the selected version's index and that its - shards contain immutable versioned URLs. + shards contain only canonical `/doc/latest/...` URLs. Confirm versioned + content and `/doc-latest/...` return `X-Robots-Tag: noindex`, including PDFs. + `/robots.txt` must allow documentation crawling so search engines can read + those headers; it continues to exclude the users archive. 6. Use the workflow's `remove-canary` operation with the `canary` environment. The narrow canary route is more specific than `/doc/*` and would otherwise keep intercepting that version. -The production documentation routes are `/doc`, `/doc/*`, `/doc-latest`, and -`/doc-latest/*`. The Worker selects aliases through `LATEST_DOC_VERSION`; it -does not copy alias objects. +The production documentation routes are `/doc`, `/doc/*`, `/doc-latest`, +`/doc-latest/*` and `/robots.txt`. The Worker serves the same checked-in robots +file as Astro so the indexing policy takes effect before the website cutover. +It selects aliases through `LATEST_DOC_VERSION`; it does not copy alias objects. +Only `https://www.gecode.dev/doc/latest/...` is indexable. Versioned URLs remain +available for citations and downloads. Stored versioned sitemaps stay immutable; +the Worker rewrites the selected sitemap responses to latest URLs. Set the documentation routes to fail closed: after documentation leaves the Pages artifact, fail-open traffic would reach a missing origin path. Set the @@ -359,7 +370,8 @@ Run `python3 scripts/build-classic-rollback.py` from the website checkout with the existing Bundler dependencies installed. It creates `dist/classic-rollback/` and `dist/classic-rollback.tar.gz`, refusing to replace an existing output directory. The 5 September rehearsal produced 91,168,795 -uncompressed bytes. Keep the tarball outside this checkout before cutover; +uncompressed bytes. A verified copy is retained at +`/Users/zayenz/gecode/website-rollback/classic-site-2026-07-15.tar.gz`; restore its contents as the Pages artifact when needed. Before Astro, removing the documentation routes restores the existing origin diff --git a/docs/gecode-release-pipeline.md b/docs/gecode-release-pipeline.md index a1bd48cec..a9e540e2a 100644 --- a/docs/gecode-release-pipeline.md +++ b/docs/gecode-release-pipeline.md @@ -85,6 +85,11 @@ Record the website-tool commit and final manifest digest in release state. The final manifest's SHA-256 values identify the approved local release tree; do not describe them as hashes recomputed by R2. +Stored sitemap artifacts may contain immutable version URLs. The Worker +rewrites the selected release's published sitemap index and shards to +`/doc/latest/...`; submit only `/doc/sitemap.xml`. Do not rewrite completed R2 +objects to change indexing policy. + The ordinary release must not invoke this repository's historical Doxygen patcher, staging publisher, staging verifier, or promotion scripts. Those scripts may remain useful for the one-time migration of historical content, @@ -160,8 +165,12 @@ content release, the candidate updates: - the production `LATEST_DOC_VERSION` selection. Keep generated documentation and R2 credentials out of the candidate. Use -`latest` aliases for human entry points, but use immutable version URLs in -release news and citations. +`/doc/latest/...` for human entry points and immutable version URLs in release +news and citations. Only the production latest URLs are indexable and +canonical. Immutable version URLs, including PDFs, and the HTTP 200 +`/doc-latest/...` compatibility alias carry `X-Robots-Tag: noindex`; staging +documentation is also `noindex`. Producer HTML must not contain conflicting +versioned canonical links; the Worker owns the served canonical selection. Run the configured website quality command and validate the release, download, and documentation pages. Before changing either alias, smoke-test @@ -197,6 +206,12 @@ caches may serve the previous version for up to five minutes. Roll back an alias failure by redeploying the previous `LATEST_DOC_VERSION`; immutable versions remain unchanged. +Check indexing headers on HTML and PDFs: only production `/doc/latest/...` +may be indexed. Verify latest HTML canonicals and that the published sitemap +index and every shard contain only latest URLs, including after alias +promotion. Keep immutable and compatibility paths crawlable so crawlers can +observe their `noindex` headers. + Staging and canary Worker deployments belong to the initial migration or to a Worker-code change. They are not required for an ordinary content-only release. diff --git a/docs/migration-readiness-review-2026-09-05.md b/docs/migration-readiness-review-2026-09-05.md index 6a71ce1db..c2d22cfe8 100644 --- a/docs/migration-readiness-review-2026-09-05.md +++ b/docs/migration-readiness-review-2026-09-05.md @@ -1,19 +1,17 @@ # Website migration and release readiness -Reviewed and updated on 5 September 2026 in `explore/astro-rework`, based on -`72760fbcf`. Three subagents reviewed deployment, documentation serving and -publication, and the Gecode/MPG release integration. Fixes are in the local -website, MPG and release-support working trees. Existing uncommitted work was -preserved. A limited documentation Worker snapshot was subsequently committed -and pushed on an isolated staging branch, then deployed through GitHub Actions. -The migration branch and production website remain unchanged by that deployment. - -The DNS move is complete. Production still serves Jekyll, and documentation -still comes from the old origin. The immediate code defects have been fixed -locally. Deployment access is now provisioned. The remaining cutover needs -clean-checkout website CI and staged production verification. Future coordinated -releases have a working local preparation slice; their publication coordinator -is unfinished. +Reviewed and updated on 5 September 2026. Three subagents reviewed deployment, +documentation serving and publication, and Gecode/MPG release integration. +The website migration was merged in [PR #7](https://github.com/Gecode/gecode.github.io/pull/7) +as `30ccd9636fb7eabed4b14ed5fea09602652ac0be` after clean-checkout CI passed. +The MPG and release-support producer changes remain local. Existing working +trees were preserved. + +DNS migration, staging verification, full historical R2 verification and the +6.4.0 documentation canary are complete. Jekyll still serves the active website. +The remaining traffic changes are the full documentation routes, a one-day +soak, and the separate Astro cutover. Future coordinated releases have a +working local preparation slice; their publication coordinator is unfinished. The hosting arrangement is Cloudflare DNS/proxy, Workers and private R2, with **GitHub Pages as the Astro origin**. Moving Astro itself to Cloudflare Pages @@ -26,14 +24,14 @@ or Workers is a separate project and is unnecessary for this cutover. | DNS | Cloudflare nameservers `milan` and `tegan` are authoritative; the zone is active. MX and SPF use Cloudflare Email Routing. | | TLS | Cloudflare uses Full (strict) and Always Use HTTPS. Public HTTP redirects to HTTPS and the apex redirects to `www`. GitHub reports `https_enforced: false`; reconcile the origin setting separately. | | Production website | `/download.html` returns 200; `/download/` returns 404. Jekyll remains live. | -| Production documentation | Responses lack the documentation Worker's version header; `/doc/sitemap.xml` returns 404. There are no production or canary zone routes. | +| Production documentation | The 6.4.0 canary passes live HTTP and browser checks. Both narrow routes are fail-closed. Other versions and aliases still use the old origin pending production rollout. | | Staging documentation | The reviewed Worker was deployed through GitHub. Its tests and live HTTP smoke checks pass for immutable 6.4.0, both aliases, canonical links, redirects, static assets and PDF range behavior. | -| R2 | Public `r2.dev` access is disabled. The 14-day lifecycle applies only to staging. Nine local historical manifests exist; the complete bucket was not rehashed during this review. | +| R2 | Public `r2.dev` access is disabled. The 14-day lifecycle applies only to staging. All nine archives were fully verified: 52,385 objects / 1,138,898,740 bytes, with no missing objects or hash failures. The 326 historical image-map MIME declarations are documented below. | | Email | Email Routing destinations and catch-all Worker configuration are present. Real delivery and delivery alerts were not exercised. | -| GitHub | The latest production Pages run remains the [15 July deployment](https://github.com/Gecode/gecode.github.io/actions/runs/29430643821). The [5 September staging Worker run](https://github.com/Gecode/gecode.github.io/actions/runs/33964652752) passed. The migration is not on the remote default branch. | +| GitHub | The latest production Pages run remains the [15 July deployment](https://github.com/Gecode/gecode.github.io/actions/runs/29430643821). The [5 September staging Worker run](https://github.com/Gecode/gecode.github.io/actions/runs/33964652752) passed. The migration is merged on `main`; clean-checkout [CI passed](https://github.com/Gecode/gecode.github.io/actions/runs/33967188164), with Pages deployment skipped. | | Deployment credentials | All three Cloudflare environments now contain the account ID and dedicated `gecode-github-workers` token. Production accepts only `main` and `release/*-website` branches; the existing `zayenz` reviewer and approval settings were preserved. | -## Fixes completed locally +## Fixes completed | Area | Result | | --- | --- | @@ -73,8 +71,8 @@ exceed the published-site size limit. The new mail archive, dereferences symlinks and checks required pages and size. A real build succeeded: **91,168,795 bytes** uncompressed. Its archive is -[dist/classic-rollback.tar.gz](../dist/classic-rollback.tar.gz). Retain it outside -this checkout and ephemeral CI storage before cutover. It requires the +[dist/classic-rollback.tar.gz](../dist/classic-rollback.tar.gz). A verified copy is retained outside +this checkout at `/Users/zayenz/gecode/website-rollback/classic-site-2026-07-15.tar.gz`. It requires the production documentation Worker and R2; remove active-site redirects when restoring classic pages. The build was verified locally, not deployed. @@ -85,10 +83,10 @@ Do not delete immutable R2 objects during rollback. ## Remaining cutover sequence -1. **Land the reviewed code.** Review and commit the current website, Worker, - archive and producer changes, including untracked dependencies. Run CI from - clean checkouts. Keep the first Pages deployment manual and retain the - classic rollback artifact. +1. **Website code landed.** The migration, archive and Worker changes are on + `main`, and clean-checkout CI passed. Pages deployment remains manual and + the classic rollback archive is retained outside the checkout. MPG and + release-support changes still need separate commits and producer rehearsal. 2. **Deployment access verified.** The dedicated Worker token is installed in `cloudflare-staging`, `cloudflare-canary` and `cloudflare-production`. Production branch restrictions and the existing reviewer were verified. @@ -98,10 +96,9 @@ Do not delete immutable R2 objects during rollback. 3. **Check operational records.** Confirm historical upload verification, fail-closed documentation routes, fail-open redirect routes, real mail delivery and useful error/delivery alerts. Keep the current DNS delegation. -4. **Canary documentation.** Deploy documentation only on the configured 6.4.0 - canary routes. Check HTML, source view, changelog fragment, CSS/JS/image, PDF - ranges, missing paths, canonical links and sitemap. Ordinary Jekyll pages - must continue to work. +4. **Canary documentation verified.** The 6.4.0 deployment passed HTML, source + folding, changelog fragment, CSS/JS/image, PDF ranges, missing paths, + canonical links and sitemap checks. Ordinary Jekyll pages remain unchanged. 5. **Move documentation.** Deploy production documentation routes, verify historical versions and both aliases, then remove the narrower canary. Allow at least one day with the Jekyll origin still available as fallback. @@ -135,7 +132,53 @@ used commit `5ec53f3b111a46d92c42f8b996d79b86dc39cd80` on live checks in `scripts/docs/smoke-worker.mjs` passed. A bounded retry allows for propagation immediately after deployment; the first attempt observed the old redirect behavior before the new Worker reached that request. Production -routes remain absent and production still serves the classic origin. +routes were absent at that stage. The subsequent canary is recorded below. + + +## Historical archive and canary verification + +Every historical R2 object was downloaded and compared with its manifest using +SHA-256, byte count, response length and MIME type. Complete key sets and sizes +were checked before and after downloading. All 52,385 files passed, across +1.3.1, 2.2.0, 3.7.3, 4.4.0, 5.1.0, 6.0.1, 6.1.1, 6.2.0 and 6.4.0. +Verification records are retained beside the rollback artifact in +`/Users/zayenz/gecode/website-rollback/verification-2026-09-05/`. + +The old manifests wrongly declared 326 Graphviz `.map` files as JSON. R2 serves +them as `application/octet-stream`. Their bytes are intact. Verification accepts +only the known image-map formats (`/...` URLs as canonical. HTML responses carry an -HTTP `Link` canonical for their versioned URL, including responses reached -through either alias. Allow crawlers to index versioned documentation, exclude -the aliases in `robots.txt`, and publish only versioned URLs in documentation -sitemaps. +Only production `/doc/latest/...` documentation is indexable. Its HTML uses +the corresponding latest URL as its canonical. Every immutable +`/doc//...` response, including PDFs, carries `X-Robots-Tag: noindex`. +The `/doc-latest/...` compatibility alias continues to serve content with HTTP +200 and `noindex`; all staging documentation also carries `noindex`. +Keep these paths crawlable so search engines can read the indexing headers. + +Submit `/doc/sitemap.xml`, which exposes the selected release using only +`/doc/latest/...` sitemap and page URLs. The Worker rewrites stored sitemap +URLs in its responses; immutable R2 sitemap artifacts remain unchanged and +are never submitted as versioned sitemaps. ## Worker behavior @@ -125,7 +131,8 @@ The Worker should implement only the behavior object storage lacks: 8. Return a small branded 404 page without trying extension fallbacks. 9. Add `X-Content-Type-Options: nosniff` and a conservative referrer policy. 10. Emit structured logs for misses, range failures, and unexpected methods. -11. Add an HTTP canonical link to versioned HTML responses. +11. Apply latest-only canonical and indexing headers to HTML, PDFs, and other + documentation responses; rewrite published sitemap URLs to `/doc/latest/`. Versioned objects can use a one-year shared cache because their keys never change. Alias responses should use a five-minute cache and expose the resolved @@ -175,7 +182,8 @@ The local implementation now includes the tested Worker in `workers/docs/`, manifest, inventory, and sitemap generators in `scripts/docs/`, an immutable staged `rclone` publisher with SHA-256 verification, and CI validation. Generated sitemap files are part of each immutable release, and the Worker -serves the selected sitemap at `/doc/sitemap.xml`. Provisioning buckets and +serves the selected sitemap at `/doc/sitemap.xml`, rewriting its URLs to +`/doc/latest/...` without modifying stored objects. Provisioning buckets and credentials, uploading objects, and changing DNS remain operator actions because they affect external infrastructure. @@ -232,8 +240,9 @@ while ordinary website URLs still come from GitHub Pages. commit. Keep a tagged pre-migration commit for provenance. - Consider a separate, carefully announced history rewrite only if clone size remains a problem; it is not required for serving the site. -- Update website links to versioned documentation URLs. Keep `latest` for human - entry points, not for citations or release notes. +- Use `/doc/latest/...` for website documentation entry points. Keep immutable + version URLs for citations and release notes; those URLs remain available + but are not indexable. Exit criterion: the GitHub Pages artifact contains only the Astro site and small first-party downloads. @@ -262,8 +271,11 @@ Automate these checks before changing DNS: - directory index behavior; - `GET`, `HEAD`, conditional requests, and byte ranges; - cache headers for versioned and alias paths; +- latest-only indexing and canonical URLs, with `noindex` on immutable URLs, + PDFs under those URLs, the compatibility alias, and staging; - branded 404s with no accidental bucket listing; -- sitemap size and URL-count limits; +- sitemap size and URL-count limits, with only latest URLs in the submitted + sitemap index and shards; - ordinary `www.gecode.dev` pages bypassing the documentation Worker. Set documentation routes to fail closed after the Pages archive is removed; diff --git a/documentation.html b/documentation.html index 788fb8b17..e51824dc8 100644 --- a/documentation.html +++ b/documentation.html @@ -10,7 +10,7 @@

Modeling and Programming with Gecode

- Modeling and Programming with Gecode + Modeling and Programming with Gecode provides comprehensive documentation of how to model and program with Gecode.

@@ -36,7 +36,7 @@

Modeling and Programming with Gecode