[eseidel]

Step 1 of making shorebird.dev pass isitagentready.com (plan: private Notion doc on the "AI-friendly website and docs" arc). Today shorebird.dev scans at Level 1; docs.shorebird.dev is at Level 4. This PR holds only static files and headers, so it stays on Firebase Hosting.

• robots.txt: Content-Signal: search=yes, ai-input=yes, ai-train=yes and explicit rules for the major AI crawlers, the same as docs. This is what Level 2 needs.
• firebase.json: a Link header on / pointing to llms.txt, the API catalog, the OpenAPI spec, the docs, the OAuth server metadata on auth.shorebird.dev, agent skills, the AI catalog and the sitemap. application/linkset+json on /.well-known/api-catalog, and CORS * on /.well-known/** and /llms.txt.
• /.well-known/api-catalog (RFC 9727), /.well-known/agent-skills/index.json and /.well-known/ai-catalog.json: copies of the docs site's files, which already use absolute URLs into docs and api.shorebird.dev. The AI catalog adds one entry for this site's llms.txt. As on docs (docs#747), it is linked as rel="ai-catalog", served as application/ai-catalog+json, with specVersion: "1.0", which is what the AI Catalog spec registers and what the scanner looks for.
• llms.txt: removes the empty ## Links heading and the stray -- lines, adds a "Developers and agents" section and an ## Optional section, and drops "Notes for AI Systems". That section was prose that no llms.txt consumer reads, and its one pointer, to the docs llms.txt, now sits in the intro.

The catalogs are copies, so they can drift from docs. The skills list changes rarely, and a stale entry still points at a live doc page.

Still failing after this PR: Markdown content negotiation, which Firebase Hosting can't do (step 2783 on the arc, Eric's call), and the checks that would describe things we don't run publicly (MCP card, A2A, OAuth metadata on this domain).

Verified locally: npm run build, format:check, cspell, check:links. I'll scan the preview channel once it deploys.

Screenshots

No visible change: only robots.txt, llms.txt, .well-known files and response headers.

Landing

As soon as approved. After it deploys, re-run the scan on shorebird.dev and record the level on the arc.

🤖 Generated with Claude Code

[github-actions]

🐦 Preview of 2c09f08: https://shorebird-website--pr-426-dxm0p4sh.web.app (expires in 7 days)

[eseidel]

Adversarial review

This repo has no review bot, so here is the self-review. I scanned the preview channel with isitagentready.com: Level 2, up from Level 1. Link headers, AI bot rules, Content Signals, API Catalog, Agent Skills and ARD pass. The remaining failures are the ones the plan expected: Markdown negotiation (step 2783), DNS-AID, and OAuth/MCP/A2A/WebMCP/PAP metadata for things we don't publish here.

Checked:

• Every URL in llms.txt and every agent-skills url returns 200. That includes the docs .md pages and auth.shorebird.dev/.well-known/oauth-authorization-server.
• /.well-known/api-catalog (no extension) returns 200 with application/linkset+json and is not caught by the trailingSlash redirect. CORS * is applied.
• Content-Signal is an unknown directive to existing robots parsers, which ignore it. User-agent: * still allows everything, as before.

Findings:

1. Fixed in 18c894e. ai-catalog.json claimed host did:web:shorebird.dev, but https://shorebird.dev/.well-known/did.json is a 404. That asserts an identity we don't publish, so I removed it. docs.shorebird.dev has the same claim; I'll file a follow-up there.
2. Fixed in 18c894e. The Link header gave the AI catalog rel="service-desc". Agents read that rel as "the API's description" and could fetch the catalog expecting OpenAPI. It is now rel="describedby", and the OpenAPI link stays the only service-desc.
3. Accepted. The three .well-known catalogs copy the docs site's files, so they can drift. A stale entry still points at a live docs page, and the skills list changes rarely.
4. Accepted. The Link header is only on /. That is all the scanner and RFC 8288 discovery need, and per-page alternate links come with per-page Markdown (plan PR 2).

[eseidel]

I'm sure this will need more tweaking over time, in particular the suggested queries.

[eseidel]

lgtm

[eseidel]

Note before landing: the AI Catalog spec (Agent-Card/ai-catalog, "Link Relation Discovery") registers rel="ai-catalog" with type application/ai-catalog+json. isitagentready.com's ARD check looks for that relation, not describedby. shorebirdtech/docs#747 switched docs to rel="ai-catalog", specVersion: "1.0" (the spec requires Major.Minor) and that Content-Type. The firebase.json Link header and ai-catalog.json here want the same change, in this PR or a follow-up.

[easymac]

LGTM

[eseidel]

Done in 2c09f08: Link header now rel="ai-catalog"; type="application/ai-catalog+json", /.well-known/ai-catalog.json is served with that Content-Type, and specVersion is "1.0", matching docs#747.

[AbhishekDoshi26]

Reviewed against the preview channel. No blocking issues.

• / serves the Link header. /.well-known/api-catalog is served as application/linkset+json, ai-catalog.json as application/ai-catalog+json, and all /.well-known/** files and /llms.txt carry CORS *. trailingSlash doesn't redirect the extensionless file.
• All JSON parses. api-catalog and agent-skills/index.json are identical to the docs copies. Every referenced URL resolves, except api.shorebird.dev/api/v1, which returns 401 but is only the linkset anchor.
• robots.txt has no Disallow lines, so nothing that was crawlable before is blocked.
• Minor: "source": "/" doesn't match /index.html, so that URL gets no Link header. / is canonical and is what the scanner checks, so this is fine.
• Policy note: ai-train=yes applies to every listed crawler, including Bytespider and meta-externalagent. This matches docs.