Index only latest documentation and complete Cloudflare cutover checks

.github/workflows/workers.yml

@@ -120,6 +120,17 @@ jobs:
         if: inputs.operation == 'deploy' && inputs.environment == 'canary' && (inputs.worker == 'both' || inputs.worker == 'documentation')
         run: npx wrangler deploy --env canary --config workers/docs/wrangler.jsonc
 
+      - name: Verify production documentation
+        if: inputs.operation == 'deploy' && inputs.environment == 'production' && (inputs.worker == 'both' || inputs.worker == 'documentation')
+        run: |
+          version=$(node -p "JSON.parse(require('fs').readFileSync('workers/docs/wrangler.jsonc', 'utf8')).env.production.vars.LATEST_DOC_VERSION")
+          for attempt in 1 2 3 4 5 6 7 8 9 10 11 12; do
+            if node scripts/docs/smoke-worker.mjs https://www.gecode.dev "$version"; then exit 0; fi
+            if [ "$attempt" -eq 12 ]; then exit 1; fi
+            echo "Waiting for the production deployment to propagate (attempt $attempt)."
+            sleep 30
+          done
+
       - name: Verify documentation canary
         if: inputs.operation == 'deploy' && inputs.environment == 'canary' && inputs.worker == 'documentation'
         run: |
@@ -133,7 +144,7 @@ jobs:
 
       - name: Remove documentation canary
         if: inputs.operation == 'remove-canary'
-        run: npx wrangler delete --env canary --config workers/docs/wrangler.jsonc --force
+        run: node scripts/docs/remove-canary.mjs
 
       - name: Deploy redirect Worker to staging
         if: inputs.operation == 'deploy' && inputs.environment == 'staging' && (inputs.worker == 'both' || inputs.worker == 'redirects')

docs/deployment-runbook.md

@@ -15,15 +15,16 @@ small Pages fallback files.
 As checked on 5 September 2026, Cloudflare is authoritative for `gecode.dev`.
 Cloudflare proxies the apex GitHub Pages A records and the `www` CNAME to
 `gecode.github.io` with Full (strict) TLS and Always Use HTTPS. GitHub Pages
-still serves the production website and documentation; the Astro artifact has
-not been deployed.
+still serves the active website. The 6.4.0 documentation canary now uses R2;
+other versions and aliases await production rollout. Astro has not been deployed.
 
 Cloudflare Email Routing is ready. Its managed MX, SPF, and DKIM records are
 authoritative, both forwarding destinations are verified, and the catch-all
 rule sends mail through the checked-in `gecode-email-routing` Worker. The
 private R2 documentation archive and `docs-staging.gecode.dev` Worker custom
-domain are live and pass the phase 4 smoke tests. No canary or production
-documentation Worker routes exist yet.
+domain are live and pass the phase 4 smoke tests. Full R2 verification passed
+for all nine versions (52,385 files / 1,138,898,740 bytes). The 6.4.0 canary
+passes live checks and its two routes are fail-closed.
 
 The `cloudflare-staging`, `cloudflare-canary`, and `cloudflare-production`
 GitHub environments contain the stable Cloudflare account ID and the dedicated
@@ -36,7 +37,8 @@ The [5 September readiness review](migration-readiness-review-2026-09-05.md)
 records the local fixes, rollback build, verification limits and remaining
 work. Phases 2–4 below describe the original migration sequence; DNS delegation
 and infrastructure setup are largely complete. Staging deployment through
-GitHub now passes. Continue with clean-checkout website CI and phase 5 checks.
+GitHub and clean-checkout website CI pass. The website migration is merged
+in PR #7; Pages remains manual. Continue with phase 5 production rollout.
 
 ## Cutover overview
 
@@ -116,12 +118,14 @@ The successful [staging run](https://github.com/Gecode/gecode.github.io/actions/
 used commit `5ec53f3b111a46d92c42f8b996d79b86dc39cd80` on
 `codex/verify-docs-staging-20260905`. That branch contains a limited Worker
 snapshot with a staging-only push workflow and suppresses the old Pages
-workflow for its branch. Land the complete website migration separately.
+workflow for its branch. The complete website migration was subsequently
+merged in [PR #7](https://github.com/Gecode/gecode.github.io/pull/7).
 Worker tests, dry-run builds, deployment and all HTTP smoke checks passed.
 The deployed `gecode-documentation-staging` version is
 `f9295ce3-1641-4727-b3d9-8f0c7f31b016`. The pre-rehearsal version was
-`8bfa5cfb-1c96-4cb6-8f57-d5d2bb85080c`. Production and canary zone routes remain
-absent, and the classic production pages still return 200.
+`8bfa5cfb-1c96-4cb6-8f57-d5d2bb85080c`. The later
+[canary run](https://github.com/Gecode/gecode.github.io/actions/runs/33967976199)
+passed from merged `main`. Classic production pages still return 200.
 
 To replace the token, create a new token with those permissions, set
 `CLOUDFLARE_API_TOKEN` in all three environments, verify a staging deployment,
@@ -231,14 +235,21 @@ Worker route remains sufficient rollback.
    documentation Worker.
 5. Deploy the production documentation routes and rerun the smoke tests.
    Confirm `/doc/sitemap.xml` serves the selected version's index and that its
-   shards contain immutable versioned URLs.
+   shards contain only canonical `/doc/latest/...` URLs. Confirm versioned
+   content and `/doc-latest/...` return `X-Robots-Tag: noindex`, including PDFs.
+   `/robots.txt` must allow documentation crawling so search engines can read
+   those headers; it continues to exclude the users archive.
 6. Use the workflow's `remove-canary` operation with the `canary` environment.
    The narrow canary route is more specific than `/doc/*` and would otherwise
    keep intercepting that version.
 
-The production documentation routes are `/doc`, `/doc/*`, `/doc-latest`, and
-`/doc-latest/*`. The Worker selects aliases through `LATEST_DOC_VERSION`; it
-does not copy alias objects.
+The production documentation routes are `/doc`, `/doc/*`, `/doc-latest`,
+`/doc-latest/*` and `/robots.txt`. The Worker serves the same checked-in robots
+file as Astro so the indexing policy takes effect before the website cutover.
+It selects aliases through `LATEST_DOC_VERSION`; it does not copy alias objects.
+Only `https://www.gecode.dev/doc/latest/...` is indexable. Versioned URLs remain
+available for citations and downloads. Stored versioned sitemaps stay immutable;
+the Worker rewrites the selected sitemap responses to latest URLs.
 
 Set the documentation routes to fail closed: after documentation leaves the
 Pages artifact, fail-open traffic would reach a missing origin path. Set the
@@ -359,7 +370,8 @@ Run `python3 scripts/build-classic-rollback.py` from the website checkout with
 the existing Bundler dependencies installed. It creates
 `dist/classic-rollback/` and `dist/classic-rollback.tar.gz`, refusing to replace
 an existing output directory. The 5 September rehearsal produced 91,168,795
-uncompressed bytes. Keep the tarball outside this checkout before cutover;
+uncompressed bytes. A verified copy is retained at
+`/Users/zayenz/gecode/website-rollback/classic-site-2026-07-15.tar.gz`;
 restore its contents as the Pages artifact when needed.
 
 Before Astro, removing the documentation routes restores the existing origin

docs/gecode-release-pipeline.md

@@ -85,6 +85,11 @@ Record the website-tool commit and final manifest digest in release state.
 The final manifest's SHA-256 values identify the approved local release tree;
 do not describe them as hashes recomputed by R2.
 
+Stored sitemap artifacts may contain immutable version URLs. The Worker
+rewrites the selected release's published sitemap index and shards to
+`/doc/latest/...`; submit only `/doc/sitemap.xml`. Do not rewrite completed R2
+objects to change indexing policy.
+
 The ordinary release must not invoke this repository's historical Doxygen
 patcher, staging publisher, staging verifier, or promotion scripts. Those
 scripts may remain useful for the one-time migration of historical content,
@@ -160,8 +165,12 @@ content release, the candidate updates:
 - the production `LATEST_DOC_VERSION` selection.
 
 Keep generated documentation and R2 credentials out of the candidate. Use
-`latest` aliases for human entry points, but use immutable version URLs in
-release news and citations.
+`/doc/latest/...` for human entry points and immutable version URLs in release
+news and citations. Only the production latest URLs are indexable and
+canonical. Immutable version URLs, including PDFs, and the HTTP 200
+`/doc-latest/...` compatibility alias carry `X-Robots-Tag: noindex`; staging
+documentation is also `noindex`. Producer HTML must not contain conflicting
+versioned canonical links; the Worker owns the served canonical selection.
 
 Run the configured website quality command and validate the release,
 download, and documentation pages. Before changing either alias, smoke-test
@@ -197,6 +206,12 @@ caches may serve the previous version for up to five minutes. Roll back an
 alias failure by redeploying the previous `LATEST_DOC_VERSION`; immutable
 versions remain unchanged.
 
+Check indexing headers on HTML and PDFs: only production `/doc/latest/...`
+may be indexed. Verify latest HTML canonicals and that the published sitemap
+index and every shard contain only latest URLs, including after alias
+promotion. Keep immutable and compatibility paths crawlable so crawlers can
+observe their `noindex` headers.
+
 Staging and canary Worker deployments belong to the initial migration or to a
 Worker-code change. They are not required for an ordinary content-only
 release.