[akash-kr]

What this adds

Everything is opt-in. A call that doesn't ask for the new extras takes the pre-existing code path, so existing users see no change.

Feature	How to turn it on	What you get
State JSON	scrape(url, state=True) · --state · "state": true	The JSON the page embeds for its frontend: Next.js __NEXT_DATA__, Nuxt, window.__X__ globals, JSON-LD
Appendix	appendix=True · --appendix	Nav/header/footer/sidebars as separate markdown regions. They're still stripped from the main content, as before.
Query highlights	query="…" · --query	The passages that answer the question. Ranked by local BM25 by default, or by TypeSafe Jev with SCRAPER_HIGHLIGHTS_GRADER=jev + TYPESAFE_API_KEY
Citations	comes with highlights	Each highlight has a link: a text-fragment URL into the page, or the video timestamp for YouTube
MCP server	pip install "switchback[mcp]" → switchback-mcp	scrape + search tools for Claude Code / Cursor / any MCP client
YouTube transcripts	pip install "switchback[youtube]"	tier_1 returns title, channel and a timestamped transcript. A cloud IP that YouTube blocks is retried through SCRAPER_EGRESS_PROXY.
Office / ebook files	pip install "switchback[documents]"	.docx/.pptx/.xlsx/.xls/.epub converted to markdown in tier_2 (via markitdown)

Deferred to the README Roadmap: layout-aware markdown, OCR for scanned pages and images, audio transcription, hosted service.

Why existing users are safe

• Default path unchanged. scrape(url) with no extras never creates a capture box, still reads the content cache, and returns the same fields. The new dataclass fields default to None and come after the existing ones, so positional construction still works.
• JSON shapes unchanged. CLI and HTTP output keep exactly url, source_method, markdown|html unless an extra is requested. Tests assert the key lists.
• No new required deps. New deps live only in [mcp], [youtube] and [documents], and are not added to [all]. Without them, YouTube and office URLs behave exactly as before. A test forces the "markitdown missing" path.
• Extras can't fail a scrape. State, appendix and highlights errors are caught. Jev errors fall back to BM25.
• Tiers keep their contracts. Page HTML is captured inside render(). tier_3 and tier_7 re-apply the capture on their worker threads. Only tier_1 (YouTube branch) and tier_2 (docs branch) gained new code paths, and both fall through to the old behavior.

Testing

• Offline: 71 tests pass on Python 3.12 and 3.10 with all extras installed. On a core-only install, 68 pass and the 3 tests that need optional add-ons skip cleanly. The existing 31 tests are unchanged.
• CI replica: core install, import smoke test, python -m build, twine check: all pass. The wheel includes the new modules and the switchback-mcp entry point.
• Live, main vs this branch, default path (27 URLs, Firecrawl disabled): identical markdown on every URL that succeeded on both. Small diffs on The Verge, Medium and Morning Brew come from page content changing between runs.
  • Stack Overflow failed once on this branch (Cloudflare served a partial page to Camoufox), then passed 3/3 on re-runs.
  • NYTimes, Indian Express and a Harvard-hosted .pptx fail on both main and this branch (blocked without the paid tier).
  • On main, .docx/.xlsx/.epub "succeed" but return the raw zip bytes (PK\x03\x04…) as markdown. With [documents] installed they now return real markdown; without it they behave as before.
• Live, extras: state, appendix and highlights work on every reachable site: Wikipedia, arXiv, IRS PDF, GitHub, Next.js docs, Python docs, Stack Overflow, BBC, The Verge, The Hindu, Moneycontrol ×2, Medium, Substack ×2, Morning Brew, Hindustan Times, Livemint, YouTube ×2, docx, xlsx, epub, Shopify.
  • Jev took 0.4–2.7 s per page.
  • Jev found answers where BM25 found none, e.g. a Shopify price question.
• Citation links checked in Chromium: 40/50 links matched visible page text, up from 36 before fixing link/title/autolink stripping. Misses come from text that isn't visible (<title>, alt text, hidden elements); those links just open the page at the top.
• MCP: exercised end to end over stdio with the MCP client SDK: tool listing, a scrape with highlights and state, and a failed URL returning {"ok": false, …}.

Rollback

git revert <merge-commit>. Nothing persistent changes: no state-file format changes and no migrations.