Add a curated llms.txt generated from the docs.json nav - #47
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
| @@ -0,0 +1,151 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
This makes sense ✅ My one real concern is that this needs a human to run a script, and I don't think we've priced that in.
The CI check finds drift, it doesn't fix it. That's fine when a person is editing docs by hand. It's less fine for sync-man-pages.yml, which runs on cron every morning and opens a bot PR. Descriptions come from the ## NAME line, and those get synced from flox/flox. So the day someone upstream rewords a NAME line, FloxBot's PR goes red and whoever's on rotation has to know the fix is "run the generator and push to the bot's branch." We'd be adding a manual step to a job that currently needs nobody.
Worth noting the check isn't enforced either. I get a 404 on branch protection for main, so a red check doesn't block anything right now (might just be my token lacking admin scope, worth a look in settings). check-man-nav.yml has the same gap so this isn't new, but "CI fails until regenerated" only does work if failing means something.
I've added a build script for the one on the website maybe we do something similar?
There was a problem hiding this comment.
Good catch, and you're right that it wasn't priced in. Fixed in 835d4f8: the sync job now regenerates llms.txt after sync-man-pages.sh, with llms.txt added to add-paths (without that, create-pull-request drops it from the commit). The job that causes the drift is the one that can fix it, so FloxBot's PR arrives correct and nobody on rotation needs to know the incantation. The step sits outside flox activate on purpose — that environment ships node, vale and pandoc, and the generator needs python3 from the runner.
Verified by rewording a ## NAME line locally: it propagates to exactly one llms.txt line, which the new step now carries into the bot's commit.
On the website build script — that pattern doesn't port, unfortunately. floxwebsite generates llms-full.txt in postbuild because Vercel runs its build. Mintlify builds this site itself from repo contents and gives us no arbitrary build hook, so the file has to be committed and the automation has to happen at commit time. Regenerating in the producing job is the closest equivalent we have.
On branch protection: it wasn't your token. main is protected by a ruleset, not classic protection, and /branches/main/protection 404s for everyone in that case. The active "Main" ruleset has deletion, non_fast_forward, merge_queue and pull_request (1 approval) — and no required_status_checks rule at all, so you're right that red blocks nothing. Two traps before we add one, both of which bite this repo specifically:
- Both checks use
paths:filters. A required check that gets skipped sits "expected" forever and blocks the PR. - The ruleset has a merge queue, and neither workflow has a
merge_group:trigger — a required check that never reports there times out at 60 minutes and ejects the entry.
So it's drop the path filters (both jobs run in seconds) plus add merge_group:, then require both checks. Happy to do that as its own PR since it affects check-man-nav too — it's a settings change and shouldn't ride along here.
Worth flagging: check-man-nav has the identical shape on the bot PR — a new command's page lands with no nav entry and goes red. That one I'd leave manual, since picking the group is a curation call, but we could auto-insert into "Other" so the reviewer re-groups rather than unblocks.
If hand-edited drift ever shows up in practice, the next step is a post-merge self-heal on main (regenerate, open a FloxBot PR on drift), which covers forks and web edits too. Not worth it until we see it.
billlevine
left a comment
billlevine
left a comment
There was a problem hiding this comment.
Peer panel review — 4 seats, 2 rounds, unanimous REQUEST-CHANGES
Four independent reviewers examined this diff blind (bare Codex; a Codex specialist wearing a senior CI/CD & generated-artifact-tooling role; Claude under a senior-code-reviewer harness; Claude under the Forge lens pipeline), then exchanged findings and reconciled. All four seats say the architecture is right — deriving the index from the nav rather than hand-curating it is the correct call, and I confirmed the generator is deterministic and idempotent, with the committed llms.txt byte-identical to its output on this head. Every blocker below is in the enforcement layer or in branch state, not in the idea.
First, a note on the existing review: @Kaysahni's blocking concern is genuinely addressed by 835d4f8. The panel re-derived that independently — the sync job now regenerates llms.txt and carries it in add-paths, so FloxBot's PR arrives correct with no human incantation. The formal "changes requested" state just has not been re-requested, which is a human step. The finding below about that same step is a different failure mode, not a reopening.
The two things that block merge.
The branch is 22 days stale, and the drift gate this PR introduces does not pass on this PR's own merge result. git merge-tree --write-tree origin/main HEAD merges cleanly; regenerating on that tree yields 103 pages in 21 sections against the committed file's 99 in 20 — adding concepts/authentication, concepts/service-accounts, concepts/personal-access-tokens and a whole ## Changelog section, and dropping the two retired man pages. Production corroborates it: of the 105 URLs in the committed llms.txt, exactly two 404, and they are precisely man/nix-builds.toml.md and man/flox-build-update-catalogs.md. The green check-llms-txt on this PR was created 2026-07-28 and predates every relevant base commit; actions/checkout does test the merge ref, but GitHub does not re-run a PR's checks when the base moves, so that verdict is stale rather than absent. Refresh the branch and regenerate — rebase-or-merge is your call, so I have not touched it.
The other is the new regenerate step in sync-man-pages.yml, and main's own history is the evidence. 14de796 (2026-07-29) deleted those same two man pages, their nav entries remained, both sidebar links 404ed, and 28a1a9a (2026-08-06, "remove retired man pages from nav, catch dangling entries in CI") is a human cleaning it up. That is main's deliberate design: the bot opens the PR and the PR's CI fails so a person fixes the nav — note sync-man-pages.sh:94-96 calls check-man-nav.sh with || echo warning specifically to keep it non-blocking inside the sync. The new unconditional step inverts it: the next retirement makes the job exit 1 before create-pull-request, so no PR appears at all and the man pages quietly stop updating.
What the panel converged on beyond those: nothing enforces the "0 bare links" property the PR is sold on (the description-less branch is a normal successful outcome, and described is printed but never compared to total); the generator writes its output before it validates and prints a success line on the failing path; walk() recognizes three container keys where its own sibling check-man-nav.sh recurses every value; the stated invariant "a page cannot reach the site without also reaching llms.txt" is already false in the tree (concepts/flox-vs-containers-faq.mdx is live, linked from an indexed page, in no nav, in no index, and no check fires); and four accuracy problems in the preamble, which is the one file whose entire purpose is to give agents a correct mental model.
Fixups pushed. I have pushed the low-risk, converged half as separate commits and replied on the threads they touch — preamble accuracy (flox edit, the flox activate mechanism, WSL2, the generations glossary, the Codex install path), generator robustness (validate-before-write, explicit UTF-8), two title-echo descriptions, an AGENTS.md section for the new generated artifact, and the bot PR body wording. I deliberately left alone everything with a real design tradeoff, everything touching CI trigger semantics — you scoped the path-filter and required-checks work to its own PR on this thread already — and the branch refresh.
Beyond this PR. update-flox-version.yml seds **/*.mdx and opens a PR with no regenerate step, so it is the same drift source as its sibling, violating the principle this PR's own comment states. Latent today — no X.Y.Z appears in any title: or description:. Separately, llms.txt.header sits outside Vale (.vale.ini scopes to [*.mdx]), so the most agent-facing prose in the repo is the only prose not linted. Both are worth issues rather than changes here.
The ## OpenAPI Specs gap is correctly excluded — there is no openapi reference anywhere in the repo, so a nav-derived generator cannot produce it. DEV-203 is the right home.
(Peer panel: codex · codex-specialist · claude · forge. 4 seats, 2 rounds plus an evidence round. Weigh the two Codex seats as one vendor. Full artifacts: ~/.peer-panel/runs/docs-pr47.)
| # causes the drift is the one that can fix it without a human. | ||
| # Deliberately outside `flox activate` — this environment ships node, | ||
| # vale and pandoc, and the generator needs python3 from the runner. | ||
| - name: Regenerate llms.txt |
There was a problem hiding this comment.
blocking: The next time a command is retired upstream, the daily man-page sync will stop producing PRs altogether instead of surfacing the retirement. sync-man-pages.sh:33 deletes man/*.mdx before regenerating, so a retired command leaves the hand-maintained docs.json pointing at nothing; this step then exits 1 at generate-llms-txt.sh:146-150 with no continue-on-error, before create-pull-request ever runs. This is not hypothetical — 14de796 did exactly that on 2026-07-29, and 28a1a9a ("catch dangling entries in CI") is a human cleaning it up a week later, which is the workflow this replaces.
Suggested: add continue-on-error: true to this step, so a nav that has gone dangling degrades to a red check on a PR a human can fix rather than to no PR at all. Note the remedy cannot be "let the bot fix the nav": add-paths is man + llms.txt and does not include docs.json. The neighbouring sync-man-pages.sh:94-96 already makes the same call the other way, invoking check-man-nav.sh with || echo warning.
(Peer panel: all four seats converged; the strongest finding on the panel.)
|
Fixups from the peer panel are pushed — three commits on top of 835d4f8, each replied to on the thread it addresses. 30c4a32 — preamble accuracy. Four claims in 2631e35 — generator robustness. Validate before writing, so a run that exits 1 no longer leaves a truncated 742b57c — documentation. An The generator is still deterministic and idempotent after all of this, and still emits 99 pages / 20 sections / 99 descriptions. Deliberately not touched, all of it explained on the relevant thread: the branch refresh (rebase-vs-merge is yours), the sync-job abort on an upstream retirement (the remedy is constrained by (Peer panel fixup pass. Review: #pullrequestreview-4975071687) |
Mintlify auto-hosts an llms.txt, but it is a single flat alphabetical list with no preamble, and 40 of its 100 links carry no description at all because man/*.mdx has no `description:` frontmatter. Committing an llms.txt at the repo root overrides it. Rather than hand-maintain a curated file, generate it from the nav. docs.json is already where curation happens, and check-man-nav.sh already fails PRs that add a page without a nav entry, so a page cannot reach the site without also reaching llms.txt. Sections come from nav groups; descriptions come from frontmatter, falling back to the man page's `## NAME` line, which is the canonical one-liner shipped with each command. Result: 99 pages, 20 sections, 0 missing descriptions. The preamble in llms.txt.header restores agent guardrails that were lost when the docs moved off MkDocs — the non-interactive `flox activate` forms, `flox install` over the system package manager, editing manifest.toml directly, and the negation for the hallucinated `install.flox.dev` / `flox.dev/install` URLs. It sits ahead of the first H2 on purpose: strict llms.txt parsers extract only link lists from H2 sections, so prose under a heading gets dropped. Also drops the flox MCP server, which is deprecated in favour of the skill library, and repoints flox-agentic at flox-skills. The generated file omits Mintlify's `## OpenAPI Specs` section, which links a spec that currently serves Mintlify's plant-store sample (DEV-203). Restore it here once that is fixed. Refs AI-500
llms.txt takes each man page's description from its `## NAME` line, and those lines are synced from flox/flox. So the morning a command's NAME is reworded upstream, FloxBot's sync PR goes red on check-llms-txt and whoever is on rotation has to know the fix is "run the generator and push to the bot's branch" — a manual step added to a job that needs nobody today. Regenerate in the sync job instead. The job that causes the drift is the one that can fix it without a human, so the bot's PR arrives correct. `llms.txt` joins `add-paths`, without which create-pull-request drops the regenerated file from the commit. The step sits outside `flox activate` deliberately: that environment ships node, vale and pandoc, and the generator needs python3, which comes from the runner. check-llms-txt.yml stays as the net for hand-edited PRs, where the author is in a local checkout and the error names the command to run. Refs AI-500 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Peer review found four claims in the preamble that the docs it indexes contradict. This file exists to stop agents guessing, so a wrong mental model in it is worse than no entry. - `flox edit`: the old bullet told agents to edit `.flox/env/manifest.toml` directly, which per man/flox-edit.mdx blocks `flox install`, `uninstall`, `edit` and `upgrade` on any FloxHub-linked environment until `--sync` or `--reset`. Name `flox edit -f`, the documented non-interactive form, and state the sync caveat. - `flox activate`: the mode is selected by the invoking shell, not by the absence of a command argument. From a non-interactive shell with stdout redirected, bare `flox activate` is in-place mode, not a hang. The prescriptive half (`-c` / `--`) was already right and is unchanged. - WSL 2: install-flox/install.mdx marks support experimental and notes installation fails on WSL 1; carry both across. - Generations: there is no generations diff command — `ls man/ | grep generations` returns only history, list, rollback, switch. Also drop the clone requirement from the Codex install path. `codex plugin marketplace add` accepts `owner/repo`, so the remote form works and mirrors the Claude Code path; neither the header nor ide-extensions.mdx ever gave the `git clone` step, and in llms.txt an agent would run `marketplace add .` in whatever directory it happened to be standing in. Two descriptions restated their own link text; replace them with the same class of fix this PR already applied to flox-vs-containers and organizations. Regenerate llms.txt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two generator robustness fixes, both converged across all four reviewers. Validate first. The output was written at the top of the file and the `missing` check ran afterwards, so a run that exited 1 still replaced the committed llms.txt with a version missing the unresolvable pages — and printed `wrote ...: N pages` on its way out. Since the file is committed, a distracted `git add -A` could stage the truncated result, which would then pass the drift check on the next run. Moving the check above the write costs nothing and makes the failure leave no residue. Verified: with man/flox-gc.mdx removed the run now exits 1 with llms.txt byte-identical. The page count no longer subtracts `missing`, because reaching the write means there is nothing to subtract. Explicit encoding. All four `open()` calls used the ambient locale while llms.txt.header contains em dashes, so `LC_ALL=C ./scripts/generate-llms-txt.sh` died with UnicodeDecodeError. Worse than the crash: a latin-1 locale would have succeeded while writing different bytes, producing drift a contributor could not reproduce. CI is unaffected — GitHub runners are UTF-8 — so this is purely a local-run fix. Verified under LC_ALL=C LANG=C PYTHONUTF8=0. The two reads become `with` blocks, which also closes the handles they leaked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AGENTS.md already has a "Man pages are generated — do not edit directly" section naming the script, its dependency and the enforcing check. This PR adds a second generated-and-committed artifact with the same shape and left that documented nowhere: `grep -i llms AGENTS.md README.md CONTRIBUTING.md` returned nothing. A contributor's first contact with the new gate would have been a red check with no in-repo guidance, and anyone hand-editing llms.txt would have lost the work silently. Add the parallel section. The sync bot's PR body said any llms.txt change there "follows from a reworded `## NAME` line". Added, removed and renamed pages move it too — the link text comes from the frontmatter title the sync derives from the filename — so a reviewer told to expect only NAME rewordings would under-inspect a page-set change. Widen it. Add llms.txt.header to .mintignore. It is a build input rather than a page. Whether Mintlify would serve it is genuinely unresolved — it is not deployed yet, and Mintlify's llms.txt override is a named feature rather than evidence that arbitrary root files are served — but the line costs nothing and forecloses the question either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
742b57c to
4e1a1f4
Compare
Three things the peer panel found, all of the same shape: a property the PR claims held for today's content and would have stopped holding silently. "0 bare links" — the reason this file overrides Mintlify's, which leaves 40 of 100 bare — had no failure branch. A page with neither `description:` frontmatter nor a `## NAME` line emitted `- [Title](url)` as a normal successful outcome, and the drift check could not catch it, since it only compares the committed file against a fresh run. Bare links are now collected like `missing` already is and fail the run before it writes. A `description: >` block scalar is folded into the same condition: this one-line frontmatter parser cannot read the continuation lines, and was emitting the literal `>` as the description while counting the page as described. walk() descended `tabs`, `groups` and `pages` and skipped every other Mintlify container — `anchors`, `dropdowns`, `versions`, `languages` — so a nav restructure could shrink the published index with exit 0, while check-man-nav.sh (which recurses every dict value) kept printing `ok` against the same docs.json. Reproduced by injecting an `anchors` node: the page under it vanished with no error. It now refuses a nav it does not fully understand, rather than guessing at section semantics for containers this repo does not use. And the header comment's guarantee stops at man/: check-man-nav.sh iterates `man/*.mdx` only, so an ordinary page can still ship out of the nav and out of llms.txt. concepts/flox-vs-containers-faq.mdx is live, linked from concepts/flox-vs-containers.mdx:18, and in neither. llms.txt is byte-identical after all three — none of this changes today's output.
A PR that DELETES llms.txt passed green — the one outcome this workflow exists to prevent. The regenerate step recreates the file as untracked, `git diff` cannot see untracked files, so the drift came back empty and the merge would hand /docs/llms.txt back to Mintlify's generated version. Reproduced end to end: commit the file, `git rm` it, regenerate, run the old condition — exit 0. Staging first and diffing the index catches it. Verified across all three cases: in-sync exits 0, a stale committed file exits 1, a deleted one exits 1. `'**/*.mdx'` does not match the two root-level nav pages. GitHub documents `**` as matching zero or more of any character including `/`, which makes the `/` in `'**/*.mdx'` a literal one, and its own idiom for "any .js file anywhere" is `'**.js'`. index.mdx and flox-5-minutes.mdx are nav pages emitted into llms.txt, so editing either one's title or description moves the generated file — and in exactly that case llms.txt is what the PR did not touch, so the sibling `'llms.txt'` entry does not rescue it. Neither of these is the path-filter and required-checks rework scoped to its own PR; both are this check's own logic.
3 participants
Publishes a curated
llms.txtat/docs/llms.txt, generated from thedocs.jsonnavigation tree so it cannot drift.Part of the llms.txt reconciliation tracked in AI-500. A companion PR updates the site-level file in
flox/floxwebsite.Why
Mintlify auto-hosts an
llms.txt. It is one flat alphabetical list with no preamble, and 40 of its 100 links carry no description at all — every man page, plusflox-vs-containersandorganizations— becauseman/*.mdxhas nodescription:frontmatter. Committing anllms.txtat the repo root overrides it.Separately, the MkDocs→Mintlify migration dropped a set of agent guardrails that used to ship in floxdocs'
llms.txtpreamble and now exist nowhere on the docs site.Approach: generate, don't hand-maintain
A hand-curated file drifts the moment someone adds a page. This derives it from the nav instead.
docs.jsonis already where curation happens — the CLI reference is already grouped into Overview / Setup / Environments / Packages / Sharing / Build & Publish / Services / Generations / Other — andcheck-man-nav.shalready fails PRs that add a page without a nav entry. So a page cannot reach the site without also reachingllms.txt. Curation happens once, in the place it already happens.Docs,OpenAPI Specs)/llms.txtWhat ships
scripts/generate-llms-txt.sh— walks thedocs.jsonnav, emits one H2 per group in nav order. Link text is the page's frontmattertitle; the description is its frontmatterdescription, falling back to the man page's## NAMEline, which is the canonical one-liner shipped with each command. Exits non-zero if the nav references a page with no.mdx.llms.txt.header— the hand-written preamble, copied verbatim ahead of the generated sections.llms.txt— generated and committed, since Mintlify serves it from the repo rather than building it..github/workflows/check-llms-txt.yml— regenerates and fails on drift. Mirrorscheck-man-nav.yml.concepts/flox-vs-containers.mdxandconcepts/organizations.mdx, the only two non-man nav pages that lacked one. These also improve search and the sidebar.Restored agent guardrails
These shipped in floxdocs and were lost in the migration:
flox activateforms. Bareflox activatelaunches an interactive sub-shell that hangs a non-interactive harness;-cruns with hooks and profile scripts,--is exec mode and skips profile scripts.flox install, notapt/yum/dnf/brew..flox/env/manifest.tomldirectly rather thanflox edit, which opens\$EDITOR.flox search/flox show.curl | bashinstaller;install.flox.devandflox.dev/installdo not exist. This one had its own bugfix commit in floxdocs (e3c2bf6), so it was a response to observed hallucination.Note the old floxdocs rule was a blanket "NEVER run `flox activate` interactively." Reading
flox-activate.mdx, interactive is one of four documented, legitimate modes — so this scopes the rule to non-interactive harnesses instead of contradicting the man page or the site-levelllms.txt.The preamble sits ahead of the first H2 deliberately. Strict llms.txt parsers extract only link lists from H2 sections, so prose under a heading gets dropped. Same choice floxwebsite made.
MCP server removed
The flox MCP server is deprecated in favour of the skill library, so it is gone from the preamble and from
install-flox/ide-extensions.mdx. That page's "Skills and MCP" tab is now "Agent skills", repointed fromflox/flox-agenticto flox/flox-skills (pushed 2026-07-27 vs flox-agentic's 2026-07-02), covering thefloxandfloxifyskills with Claude Code, Codex, and skills.sh install paths.No references to
mcporflox-agenticremain anywhere in the repo.Known gap
The generated file omits Mintlify's
## OpenAPI Specssection. That link resolves to"title": "OpenAPI Plant Store"served againstsandbox.mintlify.com— Mintlify's sample spec, tracked in DEV-203. Restore the section here once that is fixed.Verification
llms.txtreturn 200grep -ri 'mcp\|flox-agentic'over the repo returns nothing🤖 Generated with Claude Code