Anchor the docs redirect patterns, so they match one path only

[jpmckinney]

RedirectMatch patterns in docs.conf.include end with /? and no $, so they match at any depth. Every retired path therefore has an infinite URL space beneath it that answers 302 instead of 404:

/latest/en/implementation/hosting/            302 -> /latest/en/guidance/build/hosting/
/latest/en/implementation/hosting/a/b/c/d/e/  302 -> /latest/en/guidance/build/hosting/
/latest/en/getting_started/a/b/c/             302 -> /latest/en/primer/

A crawler that resolves the target page's relative links against the requested URL appends a segment, is redirected again, and never terminates. One client has produced 550+ distinct paths up to 24 segments deep, with merging, hosting and serialization repeating.

Scale

15,569,216   requests to standard.open-contracting.org in the retained logs
 7,388,812   recursive                                                  47%

Between 333,000 and 689,000 a day, and the share has grown from 45% to 52% over the last week.

07/Sep  total=1,103,715  recursive=503,834  46%
17/Sep  total=  754,550  recursive=368,507  49%
19/Sep  total=1,174,808  recursive=653,384  56%
20/Sep  total=1,328,537  recursive=688,808  52%

Rate limiting cannot solve this. The crawler averages about 75 requests per 10 seconds — below any threshold a WordPress block-editor page load stays under, so the abusive client is slower than the legitimate one. The traffic is proxied (every peer IP is a Cloudflare edge), so this isn't origin circumvention either.

What changes

60 template patterns, expanding to 324 rendered rules. /? becomes /?$, and nothing else:

-RedirectMatch ^/{{ version }}/{{ lang }}/extensions/community/?                 https://…
+RedirectMatch ^/{{ version }}/{{ lang }}/extensions/community/?$                https://…

• The optional trailing slash is unchanged. It was already there and is load-bearing: dirhtml makes /path/ canonical while old links use both forms. Verified on rules that are already anchored — /latest and /latest/ both redirect correctly.
• Query strings are unaffected, because the anchor is on the URL-path. /latest?foo=bar → /latest/en/?foo=bar.

Nothing regresses

Of the deeper requests, 137 are not loop artefacts. Checking each against open-contracting/standard's history, none is a URL that ever existed:

what.md               only ever docs/primer/what.md
design.md             docs/guidance/phases/  →  docs/guidance/
pre-qualification.md  docs/guidance/model/   →  docs/guidance/map/
consortia.md          docs/guidance/model/   →  docs/guidance/map/
project, upgrade, planning-for-ocds, ocds_for_developers    never existed, anywhere

None ever lived under getting_started/ or implementation/, where they were requested — they are the same loop one level down, manufactured from sibling links on the redirect target.

No rules were deleted. Three deletion criteria were tested and none was met: no rule has a missing target (the 15 flagged are $1 backreferences and two-hop chains), none is unreachable (the "decreasing order of specificity" ordering holds across all 364), and 216 are never requested in the retained window — which is a link-rot judgement, not a technical one.

Also here

manage.py cloudflare events prints firewall events as JSON lines. The Free plan retains Security Events for 24 hours and samples the dashboard's logs, so a longer window has to be collected; this is what produced the evidence above. It pages by walking the upper bound backwards, since the dataset has no cursor.

Related

Bears on #266, which asks whether the redirects are in fact accessed: 216 of 364 rules have never been requested, and the 148 that have serve 2,726 requests — 0.017% of traffic. Of those, 124 rules are in-docs page moves that Sphinx or ReadTheDocs redirects could own, and 24 need hosting-level config.

🤖 Generated with Claude Code

https://claude.ai/code/session_0148VmvbJ2LcshwztmwuKRoU