Repository navigation
Add robots.txt - #436
Merged
Merged
Add robots.txt#436
Conversation
/robots.txt returns 404 today. A missing robots.txt is not neutral — crawlers
read it as "everything permitted" — so the site's policy was in force by
omission rather than stated. This writes it down and declares the sitemap.
Guides are untouched: every major stays open to every crawler. People still run
6.x and 7.x and must find their documentation, in search and through an
assistant they ask. The guides already ship one tree per major (/ for 8.x,
/7.6.14/, /6.28.11/).
The API reference is a separate site on the same host — Sphinx, not Docusaurus,
built and deployed by another pipeline. robots.txt is per-host, so this one file
governs both. Unlike the guides it ships a tree per MINOR version: six versioned
trees are live alongside the unversioned one. Each was verified by fetching ONE
real page from it (index.html is only a redirect stub, so probing it proves
nothing): that page is byte-identical at 38,575 bytes between the unversioned
tree and /8.6/, is ~38 KB in each of the frozen 8.x minors /8.5/ /8.4/ /8.3/,
and ~35 / ~32 KB in /7.6/ and /6.28/. Those are one-page measurements — no claim
is made about tree completeness. The current version always lives at the
unversioned path, so /8.6/ is a pure duplicate.
The rule is default-deny and self-maintaining rather than a version list that
goes stale:
Allow: /7.6/data-capture-sdk/
Allow: /6.28/data-capture-sdk/
Disallow: /*/data-capture-sdk/
A new minor is excluded with no edit, and its content is already served at the
unversioned path. A new major is also excluded with no edit — the Disallow's `*`
matches whatever the first path segment is, so /9.0/, /9.1/, … are each caught
individually as they appear; there is no "/9.x/" pattern involved. Keeping the
outgoing major's tree then costs one Allow line, decided once at the release.
The Allow lines are placed FIRST deliberately. Under RFC 9309 longest-match
precedence the order does not matter, because the longer Allow beats the shorter
Disallow either way. But this group deliberately excludes Googlebot and
Bingbot, and none of the six agents in it publishes a conformance statement, so
no RFC 9309 guarantee applies to them. A first-match-wins parser (historical
Nutch / crawler-commons behaviour, which is CCBot's lineage) would read a
leading Disallow and deny /7.6/ and /6.28/ outright, silently breaking the
requirement that every major stays reachable. Evaluated under both models, the
six cases that matter now agree:
path longest-match first-match-wins
/data-capture-sdk/ allow allow
/8.6/… deny deny
/7.6/… allow allow
/6.28/… allow allow
/9.0/… (future) deny deny
/sdks/… (guides) allow allow
With the Disallow first, first-match-wins denied /7.6/ and /6.28/ — the only two
rows that changed.
Bulk crawlers only, and the usual objection does not apply here. Blocking crawl
on an indexed URL can strand it, so the standard advice is to use
canonical/noindex instead — but the Sphinx pages carry no <meta name="robots">
and no canonical link at all, and adding them means changing the Sphinx build,
not this repository. robots.txt is the only lever reachable from here. It is
applied to bulk crawlers, where stranding is irrelevant because nothing is
ranking them; search-engine de-duplication is left to the Sphinx side rather
than half-done here.
No Crawl-delay. No requirement asks for rate limiting, only two of the six named
agents document support for it, and Amazonbot's own documentation says it does
not support the directive — so the directive would have thrown an inaccurate
claim into the file while throttling ClaudeBot and CCBot across the whole site,
including the guides the "reachable by assistants" requirement wants open.
The file states no page count for the API reference. The "~3,200" and "~3,900"
figures elsewhere in this repo both count pages that dropped out of the SEARCH
INDEX during a tag rename; neither is a census of sitemap-absent pages. What is
recorded instead is measured: the sitemap carries 1,508 guide URLs (618 root,
498 /7.6.14/, 392 /6.28.11/) and zero /data-capture-sdk/ URLs, so the API
reference is in no sitemap and is link-discoverable only, with three trees
link-reachable. Also noted: 21 retired version trees are still live and
unsitemapped — /7.6.3/–/7.6.13/ and /6.28.1/–/6.28.10/, roughly 450-500 pages
each — while /8.5.3/, /8.5.2/, /8.4.1/ and /8.3.1/ return 404.
Docusaurus copies static/ to the build root — verified live, where
static/img/logo-light.svg serves as /img/logo-light.svg. Deployment and cache
invalidation happen outside this repository, and a second pipeline writes the
Sphinx trees to the same host, so confirm https://docs.scandit.com/robots.txt
returns 200 after deploy and is not later clobbered by that pipeline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
moritzhartmeier
left a comment
moritzhartmeier
left a comment
Member
There was a problem hiding this comment.
Verified the factual claims against the live site and the repo before reviewing, and they hold up: /robots.txt is 404 today; the sitemap has exactly 1,508 URLs (618 root, 498 /7.6.14/, 392 /6.28.11/) and zero /data-capture-sdk/ entries; the Allow: paths match the real link targets in versioned_docs; the retired trees are still serving 200; and the preview build confirms static/robots.txt reaches the build root.
The rule is correct and I found no functional bug. Disallow: /*/data-capture-sdk/ does not catch the unversioned tree (the pattern needs a second occurrence of the literal segment), and the Allow-first ordering genuinely matters for a first-match-wins parser.
Three comments below, all about the commentary rather than the directives.
Member
|
Not sure if this is the right file, but is there a place we could make the agents browsing our docs + website aware that we have some published skills? |
Three review comments, plus two problems found while addressing them.
Generated by a plugin instead of committed to static/, which copies a file
verbatim. Two things a copy could not do:
- Track the versions. The Allow lines were hand-written, and the file's own
note said a major release "needs one line - a deliberate decision made
once". A decision made once is a decision forgotten once: retire 6.28 and
the Allow outlives the tree; freeze a new major and its readers lose their
API reference to the catch-all Disallow. Nothing would fail. They are now
derived from the docs versions in the build, through the same
linksToOwnApiLine the search tags use - so a version still linking the
unversioned tree does not get a line for a tree nothing points at.
- Tell a preview from production. Insurance, not a fix: previews are served
under a base_url subpath, so their robots.txt is inert today. Four lines to
cover a preview ever served at a host root.
Review comment 1: the published file advertised 21 retired guide trees.
Nothing links them and no sitemap carries them, so printing their addresses in
the most-fetched file on the host is a discovery list - the opposite of the
de-duplication the rest of the file argues for. Removed from the output, along
with the deploy postmortem, the search-index tag-rename history, the byte
counts and the build date. All of it now lives in the plugin source, which is
not served.
Review comment 2: inverted priority. Agreed, and it cannot be fixed here. The
~9,500 stale guide pages are the duplicate content that actually costs
something, and robots.txt is the wrong lever - blocking crawl on an indexed URL
strands it, unable to read the noindex that would resolve it. The fix is a
deploy that 410s them. The gap, its size and its owner are recorded in the
plugin source; it needs a ticket against the deploy pipeline.
Review comment 3: "the guides ship exactly one tree per major" contradicted the
census below it. Now "this build publishes one CURRENT tree per major", which
is accurate and derived. The contradiction also disappears with the census.
Found while doing it. The current tree had no Allow line: correct under RFC
9309, since `/*/data-capture-sdk/` needs a segment before the literal one and
the unversioned tree has none - but this group exists precisely for parsers
that publish no conformance statement, and the one failure that must not happen
is a sloppy `*` taking out the canonical tree while leaving the frozen ones
reachable. It is named explicitly now, for the same reason llms.txt is.
Also: deriving the sitemap counts by reading build/sitemap.xml was wrong. That
file is written by another plugin's postBuild, and plugin order put this one
first, so the read failed every build and the text degraded to "this build
emitted no sitemap" - a fallback that quietly said nothing. The counts are gone
from the published file anyway, per comment 1.
llms.txt and llms-full.txt are named in the file and allowed explicitly. The
naming is a comment: robots.txt has no registered field for them and nothing
discovers them from there - an agent finds /llms.txt by convention, the same
way it finds /robots.txt. The Allow lines are the functional part, so a future
blanket Disallow cannot remove them by accident.
Verified by building: production emits the rules above with no internals, a
preview emits Disallow: /, and the longest line is 80 characters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Agent Skills were already in llms.txt - ten entries, one per SDK - but only
nested inside each SDK's section, so nothing said "this site publishes Agent
Skills" until you had read the same line ten times in ten places. An agent
skimming for what Scandit offers had no top-level signal and no single URL to
fetch.
Two additions, both generated from the pages themselves:
- The blockquote at the top of llms.txt and llms-full.txt now says the skills
exist and where the index is. That blockquote is the one part of the
llmstxt.org layout an agent reads before the table of contents.
- /llms-agent-skills.txt, built from docs/sdks/**/agent-skills.mdx. An agent
asking "does Scandit ship skills for my host" should not have to pull
llms.txt (80 KB) or llms-full.txt (1.9 MB) and pick ten entries out of four
hundred. A new SDK appears here by having the page, not by anyone
remembering. Both are allowed explicitly in robots.txt.
scripts/test-robots.cjs asserts the generated output, which is what review
asked for when it said the measurements should be dropped or checked in
scripts/. Generating the file removed the hand-typed numbers; this removes the
rest of the assumption. It reads build/ and versions.json rather than
re-deriving from the config that produced the artifact - the same reason
test-search-facets.cjs reads the build - so it cannot agree with itself.
What it pins: the current tree is allowed explicitly; exactly the frozen
majors in versions.json get an Allow and no others, so a retired or newly
frozen major fails here instead of going quiet; the catch-all Disallow
survives; the llms indexes stay allowed; Allow precedes Disallow, which is
load-bearing for the first-match-wins parsers this group targets; search
engines stay unrestricted; and no internal narrative - retired trees, dates,
byte counts - leaks back into a file served on a public URL.
Each was mutation-checked: removing the current-tree Allow, removing a frozen
major's Allow, moving the Allow lines below the Disallow, and appending a
retired-tree census each turn the suite red.
Not addressed here, and not addressable here: the 21 retired guide trees. See
the note in docusaurus.config.ts and the PR discussion - that needs a deploy
that 410s them, not a rule in this file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Good question, and the answer is half yes — so let me split it, because the docs half is now done in this PR and the website half is not. robots.txt is not that place. It has no registered field for this, and crawlers ignore unknown directives, so anything we wrote there would look like it worked without doing anything. robots.txt controls permission (may you fetch this path), never advertising (here is what we publish). llms.txt is that place, and the skills were already in it — just not findably. All ten Agent Skills pages were listed, but only nested inside each SDK's own section. So nothing said "this site publishes Agent Skills" until you had read the same line ten times in ten places, and there was no single URL to fetch. Two changes in this PR:
Both are named explicitly in robots.txt so a future blanket How an agent actually finds these: by convention, not by us pointing. The website is not covered, and I can't fix it from this repo.
Happy to write that up as a ticket if you tell me who owns it. |
PR 431 merged while this branch was open, so both had appended to the scripts block in package.json and to the test steps in build-docs.yml. Both conflicts are two additions to the same list and neither side loses anything. The CI steps are ordered cheapest-first: test:robots reads two files already on disk, while the API-reference suites spawn child processes and stub origins. Review findings on the test, all three of them mine: The expectation disagreed with the rule it was checking. It derived the allowed API lines from versions.json while robotsTxtPlugin derives them from linksToOwnApiLine - a content scan that deliberately skips a frozen version whose snapshot still links the UNVERSIONED tree. Those diverge in the normal release shape: 8.6.0-beta.1 added 8.5.2 to versions.json, and until the freeze rewrites that snapshot's links it correctly gets no Allow line, because it sends readers somewhere already open. The test would have demanded one and failed the build on main, blaming the generated file for being right. It now reads the snapshots in versioned_docs/ directly - the same ground truth, a different implementation, so it is a second opinion rather than the same derivation twice. Verified both ways against a simulated freeze: a snapshot still linking the unversioned tree expects no Allow, and one linking its own line demands it. The table-of-contents guard could not fire. `indexOf` returns -1 when the heading is missing and `slice(0, -1)` is then the whole document, so the check silently widened from the blockquote to all 400 entries - where "agent skills" appears anyway, making it pass while the thing it protects was gone. The heading is a hardcoded literal in docusaurus-plugin-llms, so an upgrade can move it. Guarded before slicing. The per-SDK count ignored llmsIgnoreFiles. docs/sdks/titanium is on that list, so adding an agent-skills page there would have failed CI claiming the index was wrong when it was right. The ignore globs are now read out of the config - scoped to the ignore declarations, because scraping the whole file swept up this index's own includePatterns, whose literal prefix is "docs/sdks/" and which then matched every page and made the expectation zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every link this PR publishes was a 404.
docusaurus-plugin-llms hardcodes a "docs" segment into every URL it emits, and
this site sets routeBasePath: "/" on the docs plugin, so pages are served at
/sdks/... and not /docs/sdks/... . Checked against the live site:
/docs/sdks/android/agent-skills is 404, /sdks/android/agent-skills is 200.
llms.txt and llms-full.txt have carried that prefix since they were first
generated - roughly 400 dead links each - and it was survivable while nothing
pointed at them. This PR is what made it matter: llms-agent-skills.txt is a
file whose entire payload is ten of those links, announced in the blockquote
and allowed by name in robots.txt, so an agent following the new pointer would
have found ten 404s. Exactly the opposite of the point.
`pathTransformation: { ignorePaths: ["docs"] }` repairs all three. Verified
after the build: zero remaining /docs/ prefixes across the three files, and all
ten agent-skills URLs return 200 live.
crawlableApiLines no longer skips the served version. The skip read "served
unversioned", but whether a version's pages link the unversioned tree is
exactly what linksToOwnApiLine already decides from its content - redundant
where the two agreed and wrong where they did not.
scripts/update-version.py rewrites DOCS_LAST_VERSION from "current" to a
numbered version during a beta window, so the served version becomes a real
entry here and a directory in versioned_docs/. If the freeze has rewritten that
snapshot's links to its own line, the root-served guides point at
/8.5/data-capture-sdk/ while the catch-all Disallow blocks it and the Allow
covers a tree nothing links - backwards, and silent. The test's ground truth
grew to match: it scans the current docs/ tree too, taking the served number
from build/search-tags.json rather than from the config that wrote robots.txt.
And the ignore-block scrape in the test is now anchored on its own terminator.
`indexOf("\n]")` looked scoped and was not - llmsIgnoredSdkTrees is written on
one line, so its block ran forward into whatever array came next, and declaring
another one in between would have changed the answer silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five findings, and the two that matter are both in the function added last round to fix last round's finding. docs/ was paired with the wrong version number. It is ALWAYS the `current` version, but the test looked it up through `lastVersionTag`, which follows DOCS_LAST_VERSION - and update-version.py rewrites that from "current" to a number during a beta window, which is the window this check exists for. In that state the tag names the frozen release while docs/ still holds the unreleased one, so the test scanned the beta tree for the previous release's line and disagreed with the generator. Demonstrated: with lastVersionTag "docs-default-8.6.0" the old lookup pairs docs/ with 8.6.0 while docs/ holds 8.7.0. It now reads `docs-default-current`, which is what docs/ is by definition. The two assertions contradicted each other. The equality deliberately lets the current tree contribute an Allow line, and the guard below it was built from versions.json, which never contains the current version - so the moment the current tree did contribute, deepStrictEqual passed and the guard hard-failed on the same line. Dropped: the equality is already exact, and every line in the expectation came from a tree on disk. The test enumerated versioned_docs/ while the generator enumerates docsVersions, so a snapshot directory left on disk but dropped from versions.json - the config records version-8.5.3 in exactly that state - made the test demand an Allow the generator will never emit. It now takes the frozen trees from versions.json, which also removes an ENOTDIR crash on a stray file in that directory. Verified: a stale version-8.5.3 no longer votes. Two comments were wrong rather than the code. The pathTransformation note described stripping a prefix, but the plugin applies it to the whole doc-relative path, so a future route containing a segment named "docs" would lose it too - latent today, and now written down where it would break. And the llms-agent-skills.txt existence check now says why it is not a formality: generateCustomLLMFiles only warns and writes nothing when its patterns match no docs, and the plugin swallows that, so a rename would ship a robots.txt and an llms.txt both pointing at a 404 with a green build. That assertion is the only thing that catches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… guard
Two findings, both the same shape: a guard whose comment claimed more than the
code delivered.
The ignore-pattern scrape read two of the four lists. llmsIgnoreFiles assembles
llmsLinuxPartialIgnore and the llmsNonWebSdkRoots x llmsSharedPartialPageNames
cross-product by spreading identifiers rather than literals, so their patterns
were never seen - while the comment above it said "adding one does not silently
loosen this", which was true only for the titanium list. The regex also
required a trailing `*`, so a non-glob entry would have been invisible for the
same reason.
Rather than patch the scrape a third time, it is gone. llms.txt is produced by
the same plugin from the same ignore set, so any agent-skills page it lists is
one that was not ignored; comparing the two build outputs needs no knowledge of
how the ignore lists are written and cannot drift when someone adds a fifth.
The check also confirms every indexed URL maps to a page this repo builds, so
the index cannot advertise something nothing produces. Verified: dropping a
page from the index while it remains in llms.txt fails the suite.
The banned-number guard was /\b\d{2},\d{3}\b/, which caught the byte counts and
none of the other measurements its own comment named as the thing to keep out -
the sitemap census was 1,508 / 618 / 498 and the stale-tree figure ~9,500.
Someone re-adding "the sitemap carries 1,508 guide URLs" at a release would
have passed and shipped the number to the most-fetched URL on the host. Now any
comma-grouped number, plus a bare count with a unit. Deliberately not any long
number: RFC 9309 is cited in the file and is not a measurement. Verified all
three previously-missed figures are caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…them
VERIFICATION STATUS, stated up front: the glob change is verified directly
against minimatch; the three test changes are NOT yet exercised against a
build. The build that would have confirmed them was killed by the system for
low memory, which cleared build/, and it has not been re-run. Treat the test
changes as reviewed but unproven until `yarn build && yarn test:robots` has
run.
The index glob matched only .mdx. All ten skills pages are .mdx today so
nothing is broken now, but docusaurus-plugin-llms indexes .md and .mdx alike
and .md is the dominant convention in this tree - 448 files against 123. An SDK
adding docs/sdks/<new>/agent-skills.md would therefore land in llms.txt and not
in llms-agent-skills.txt: the file robots.txt advertises would quietly omit that
SDK, and the gate would fail the build blaming the index rather than the
pattern. Now `agent-skills.{md,mdx}`, checked against the installed minimatch
for both extensions and the two-level sdks/net/* paths.
Three guards were narrower than their own stated purpose:
The existence probe hardcoded `.mdx`, which was unreachable only because the
include pattern was .mdx-only as well - widening one without the other would
have failed a correct, building page. It now probes .md, .mdx and the index.*
forms.
The banned-narrative guard named /7.6.x/ and /6.28.x/ literally, and those are
exactly the lines that stop being the retired ones. After the next major
transition a discovery list for the then-frozen major would have passed every
pattern in the check written to stop precisely that. Derived now: any
three-part tree path. Two-part lines like /8.7/ stay legal, because the file
writes them when explaining the rule.
And llms.txt was read without the existence guard its two siblings get, so the
failure this suite exists to diagnose - the llms plugin not producing its files,
which it logs rather than failing on - arrived as a raw ENOENT stack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The commit before this said its three test changes were reviewed but not
exercised: the build that would have confirmed them was killed by the system
for low memory, which cleared build/. The build has now been re-run at the
user's request and the changes are confirmed, so the caveat no longer stands.
Build clean. Suites: test:robots 10, test:search-facets 10,
test:api-reference-seo 44, test:api-reference-seo-verdicts 15.
Each change exercised rather than inferred from a green run:
- The derived banned-pattern catches a discovery list for a major that is not
retired yet - "/8.6.1/ through /8.6.9/" - which the previous hardcoded
/7.6.x/ and /6.28.x/ list would have passed straight through. That is the
regression the check exists to stop, one major transition from now.
- Removing build/llms.txt now fails with "build/llms.txt is missing -
docusaurus-plugin-llms did not produce it", not a raw ENOENT stack.
- The widened existence probe still passes against the real index, so
accepting .md and the index.* forms did not loosen it into a no-op.
Still verified by construction rather than end to end: a new SDK authoring
agent-skills.md. minimatch matches the {md,mdx} glob for both extensions and
the plugin indexes both (utils.js:98), but no build has been run with such a
page present - that would have cost a second build while the machine was short
of memory.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
moritzhartmeier
left a comment
moritzhartmeier
left a comment
Member
There was a problem hiding this comment.
All three of my earlier comments are addressed, and two of them went further than I asked.
The narrative is out of the published file and into docusaurus.config.ts, which is not served — and the banned-pattern list is derived (/\d+\.\d+\.\d+\//) rather than listing /7.6.x/ and /6.28.x/, so it survives the next major transition instead of quietly going stale. The version contradiction is fixed. And the Allow lines are now derived from which trees actually link their own API line, which closes the staleness hole the original file described as "a deliberate decision made once".
I built the branch locally rather than trusting green CI. Generated directives are exactly right, 10/10 tests pass. Since a 351-line test file is only worth it if it bites, I broke the output five ways — Disallow reordered above the Allows, Allow: /6.28/data-capture-sdk/ dropped, # the sitemap carries 1,508 guide URLs re-added, catch-all flipped to Disallow: /, and a stale Allow: /8.5/data-capture-sdk/ appended. All five failed the suite. These assertions are load-bearing.
I also confirmed the llms fix is real, not theoretical: live llms.txt today emits https://docs.scandit.com/docs/sdks/android/agent-skills, which 404s, while /sdks/android/agent-skills returns 200. After this branch, zero /docs/ links in either llms.txt or llms-full.txt, and 10 entries in the new index.
One thing I checked and am NOT flagging: the 448 files against 123 count scoped to docs/sdks is exact.
Three remaining comments, none of them blocking.
pathTransformation is gone from this branch. It repairs ~400 dead links in llms.txt and llms-full.txt that have nothing to do with robots.txt, and holding it here made an independent public-facing fix wait on this review. It ships separately, ahead of this branch. A note at the config site records that llms-agent-skills.txt depends on it, so the ordering is not lost. The test file's comments explained themselves by narrating earlier commits on this branch - what a pattern used to be, which review round found what. git already holds that, and one of those passages re-printed 38,575, the exact measurement the check exists to keep out of the published file. The forward-looking half is kept intact and in some places sharpened: why the version guard is derived rather than listed by version, why any comma width is banned rather than one, why the index is compared against llms.txt rather than against the source tree. A future editor needs those to avoid reintroducing the bug; none of them needs to know the order the fixes landed in. No change to what any test does. Same assertions, same coverage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review asked for a ticket ID here. There isn't one and shouldn't be: the cleanup is already in progress on the publishing side, which is where the 410s have to come from anyway. The note now says who owns it and when that was true, and states why this file still declines to enumerate the 21 trees either way - if the cleanup lands the enumeration was never needed, and if it stalls, publishing a discovery list in the most-fetched file on the host is still the wrong fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…heck crawlableApiLines iterated docsVersions, which is the docs plugin's per-version OVERRIDE map - a frozen version that needs no overrides can be absent from it and still build out of versioned_docs/. So it answered "which versions did someone configure?" where the question is "which versions does this build contain?". Freeze a major, add it to versions.json and versioned_docs/, forget the override entry, and no Allow line is emitted while the catch-all Disallow: /*/data-capture-sdk/ blocks that major's API tree - the exact failure this function exists to prevent. It also put the generator on a different source of truth from its own test: linesThatLinkThemselves() in test-robots.cjs enumerates versions.json. The two disagreed and the generator was on the wrong side, so the build would have gone red on a correct tree rather than shipping something wrong - better than silent, still a cost. Both now read versions.json, with `current` added explicitly because it is not in that file by construction and is the one entry whose version number only docsVersions knows. Output is unchanged today: docsVersions holds current, 7.6.14 and 6.28.11, and versions.json holds the same two, so the set is identical and the result is sorted regardless. Separately, this branch broke `yarn typecheck`. crawlableApiLines returned [...lines] over a Set, which the repo's tsconfig target does not allow, so `tsc` exited 2 on an otherwise clean tree. No workflow runs typecheck - tsconfig.json says it is there "just for a nice editor experience" - so it would have surfaced as a red squiggle for whoever next opened the config. Array.from() instead; tsc is clean again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
@eugenia-scandit The website is owned by Rhys but I don't know how work is scheduled there. Can you check with Armida? I assume she knows more Please also note that the CI is currently still red |
Collaborator
Author
|
Correcting the premise of this before it gets acted on: It is a genuinely good file — proper llmstxt.org shape, blockquote summary, company profile, per-product sections with links, customer proof, differentiators, use cases. So "write one" is not the ask, and a ticket asking for that would be closed as already-done. Two things are wrong with it, and both are narrower and more actionable than a missing file. 1. It never links to the documentation. Going through every URL in it: So the estate is one-directional in the wrong direction: an agent that lands on the marketing site learns exactly what Scandit sells and has no route to how to build with it. Everything in this PR and in #418 sits behind a door that file does not point at. Your cross-link item was right — it is just the whole ask rather than half of it. 2. Its SDK list is factually stale, which is worse than incomplete, because an agent repeats it as fact:
Checked against this repo:
An agent asked "does Scandit support Kotlin Multiplatform?" answers no, and "does Scandit support Xamarin?" answers yes. Both wrong, both in the direction that costs us an evaluation. On the Agent Skills line: still worth adding, and it depends on #442 landing first — the URLs in Revised spec, if you are still willing to write it up:
I am checking with Armida who schedules work on that site. |
Collaborator
Author
|
There is a dependency of this PR on the PR442, so the CI will fail until that one is merged |
#442 is on main, so pathTransformation: { ignorePaths: ["docs"] } is now a real option rather than a dependency this branch had to describe. The placeholder note recording that dependency is dropped with it - it existed only to keep the ordering from being lost, and the ordering has happened. That is what test-robots.cjs was failing on: it resolves every URL in llms-agent-skills.txt back to a source file, and without the path fix the index emitted /docs/sdks/... so the probe looked for docs/docs/sdks/... . The check was enforcing the dependency rather than merely noting it, which is why the red was expected and why it clears here rather than needing a change to the test. package.json took the union of both sides' scripts: this branch's test:robots alongside main's api-reference-seo set from #431. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eugenia-scandit
added a commit
that referenced
this pull request
Sep 24, 2026
Review flagged the module count as stale. Measuring the rest of the header against a real build found it was not alone - five figures had drifted, not one: modules 4,386 -> 4,228 index size 10.41 -> 10.09 MiB graph size 11.62 -> 11.29 MiB docs_excerpt avg 376.5 -> 379.9 assistant avg 521.2 -> 525.2 Only llms-full.txt at 2.28 MiB and 410 llms.txt links were still right. Re-pinning them would set up the fourth round of the same correction, so the paragraph now argues in magnitudes - around 10 MiB, around 11, roughly 6 more, against about 2.3 for the whole prose corpus - which is all the argument ever needed and which survives a chunking change. Where to get exact values is named instead: baseline.json for pages and modules, the built artifacts for sizes, and the extractor's own postBuild line, all of which are derived rather than typed. This is the same conclusion #436 reached about measurements in a published file, applied to a source comment: nothing checks a number in a comment. Dropped the "423 sections" figure with them. I could not reproduce it from the built llms-full.txt under any reading I tried, and the sentence it supports - that the two artifacts cut the corpus differently - is better made structurally: llms.txt indexes whole PAGES, this indexes CHUNKS, several per page, so there is no per-module correspondence either way. And the DIVISION OF LABOUR paragraph still said the two plugins share `customFields.assistantIgnoreFiles`. Neither reads it. Both take their list as a plugin option, which this branch changed deliberately - a customFields key is a second name that has to be kept in step, and it had already drifted once. The curation decision is still made once; only the transport changed. The PR description carried the same stale sentence and is fixed too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
moritzhartmeier
approved these changes
Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/robots.txt returns 404 today. A missing robots.txt is not neutral — crawlers read it as "everything permitted" — so the site's policy was in force by omission rather than stated. This writes it down and declares the sitemap.
Guides are untouched: every major stays open to every crawler. People still run 6.x and 7.x and must find their documentation, in search and through an assistant they ask. The guides already ship one tree per major (/ for 8.x, /7.6.14/, /6.28.11/).
The API reference is a separate site on the same host — Sphinx, not Docusaurus, built and deployed by another pipeline. robots.txt is per-host, so this one file governs both. Unlike the guides it ships a tree per MINOR version: six versioned trees are live alongside the unversioned one. Each was verified by fetching ONE real page from it (index.html is only a redirect stub, so probing it proves nothing): that page is byte-identical at 38,575 bytes between the unversioned tree and /8.6/, is ~38 KB in each of the frozen 8.x minors /8.5/ /8.4/ /8.3/, and ~35 / ~32 KB in /7.6/ and /6.28/. Those are one-page measurements — no claim is made about tree completeness. The current version always lives at the unversioned path, so /8.6/ is a pure duplicate.
The rule is default-deny and self-maintaining rather than a version list that goes stale:
A new minor is excluded with no edit, and its content is already served at the unversioned path. A new major is also excluded with no edit — the Disallow's
*matches whatever the first path segment is, so /9.0/, /9.1/, … are each caught individually as they appear; there is no "/9.x/" pattern involved. Keeping the outgoing major's tree then costs one Allow line, decided once at the release.The Allow lines are placed FIRST deliberately. Under RFC 9309 longest-match precedence the order does not matter, because the longer Allow beats the shorter Disallow either way. But this group deliberately excludes Googlebot and Bingbot, and none of the six agents in it publishes a conformance statement, so no RFC 9309 guarantee applies to them. A first-match-wins parser (historical Nutch / crawler-commons behaviour, which is CCBot's lineage) would read a leading Disallow and deny /7.6/ and /6.28/ outright, silently breaking the requirement that every major stays reachable. Evaluated under both models, the six cases that matter now agree:
With the Disallow first, first-match-wins denied /7.6/ and /6.28/ — the only two rows that changed.
Bulk crawlers only, and the usual objection does not apply here. Blocking crawl on an indexed URL can strand it, so the standard advice is to use canonical/noindex instead — but the Sphinx pages carry no and no canonical link at all, and adding them means changing the Sphinx build, not this repository. robots.txt is the only lever reachable from here. It is applied to bulk crawlers, where stranding is irrelevant because nothing is ranking them; search-engine de-duplication is left to the Sphinx side rather than half-done here.
No Crawl-delay. No requirement asks for rate limiting, only two of the six named agents document support for it, and Amazonbot's own documentation says it does not support the directive — so the directive would have thrown an inaccurate claim into the file while throttling ClaudeBot and CCBot across the whole site, including the guides the "reachable by assistants" requirement wants open.
The file states no page count for the API reference. The "~3,200" and "~3,900" figures elsewhere in this repo both count pages that dropped out of the SEARCH INDEX during a tag rename; neither is a census of sitemap-absent pages. What is recorded instead is measured: the sitemap carries 1,508 guide URLs (618 root, 498 /7.6.14/, 392 /6.28.11/) and zero /data-capture-sdk/ URLs, so the API reference is in no sitemap and is link-discoverable only, with three trees link-reachable. Also noted: 21 retired version trees are still live and unsitemapped — /7.6.3/–/7.6.13/ and /6.28.1/–/6.28.10/, roughly 450-500 pages each — while /8.5.3/, /8.5.2/, /8.4.1/ and /8.3.1/ return 404.
Docusaurus copies static/ to the build root — verified live, where static/img/logo-light.svg serves as /img/logo-light.svg. Deployment and cache invalidation happen outside this repository, and a second pipeline writes the Sphinx trees to the same host, so confirm https://docs.scandit.com/robots.txt returns 200 after deploy and is not later clobbered by that pipeline.