feat: opt-in agent extras (state, appendix, highlights), MCP server, YouTube transcripts, office docs

.github/workflows/ci.yml

@@ -35,6 +35,14 @@ jobs:
           python -c "from switchback import scrape, scrape_detailed, search; print('import OK')"
           python -m switchback --help || true
 
+      # Offline tests (fake tiers, no network). Includes the backward-compatibility
+      # tests that pin the result fields and CLI/HTTP JSON keys; fastapi + httpx
+      # let the HTTP shape test run too.
+      - name: Offline tests
+        run: |
+          pip install pytest fastapi httpx
+          pytest -q tests
+
       - name: Build distribution
         run: |
           pip install build

CHANGELOG.md

@@ -6,6 +6,36 @@ versioning while pre-1.0.
 
 ## [Unreleased]
 
+### Added
+- **Agent-ready extras, all opt-in** (`scrape(..., state=, appendix=, query=)`,
+  the same flags on `scrape_detailed`, `--state` / `--appendix` / `--query` on the
+  CLI, and `state` / `appendix` / `query` in the HTTP `/scrape` body). Without them
+  results, CLI JSON and HTTP JSON are unchanged: `ScrapeResult` and `ScrapeOutcome`
+  keep exactly their fields. With them, results are the new subclasses
+  `ScrapeResultWithExtras` / `ScrapeOutcomeWithExtras`.
+  - `state` — the page's embedded JSON (Next.js `__NEXT_DATA__`, Nuxt,
+    `window.__X__` globals, JSON-LD).
+  - `appendix` — nav/header/footer/aside as separate markdown regions.
+  - `query` → `highlights` — the passages that answer the question, each with a
+    text-fragment citation link. Graded by local BM25 by default, or by TypeSafe's
+    Jev model with `SCRAPER_HIGHLIGHTS_GRADER=jev` + `TYPESAFE_API_KEY` (falls back
+    to BM25 on any Jev error).
+- **MCP server** — `switchback-mcp` (stdio) exposes `scrape` and `search` as tools.
+  Install with `switchback[mcp]`.
+- **YouTube transcripts** in tier_1 (title, channel, timestamped transcript linked
+  to the moment in the video). Opt in with `SCRAPER_YOUTUBE=1` plus
+  `switchback[youtube]`; otherwise, or for videos with no captions, YouTube URLs
+  go through the cascade as before.
+  YouTube blocks most cloud IPs; a blocked request is retried once through
+  `SCRAPER_EGRESS_PROXY` when one is configured. Transcript highlights cite the
+  timestamp in the video.
+- **Office and ebook files** (.docx, .pptx, .xlsx, .xls, .epub) converted to
+  markdown in tier_2 via markitdown. Opt in with `SCRAPER_DOCUMENTS=1` plus
+  `switchback[documents]`; otherwise these URLs are handled exactly as before.
+
+The new extras are deliberately not added to `[all]`, so upgrading an existing
+`[all]` install pulls no new dependencies.
+
 ## [0.5.0] - 2026-06-30
 
 ### Changed

CONTRIBUTING.md

@@ -25,8 +25,40 @@ policy (`switchback/policy/`). Start with the cascade runner in
 - **Don't commit secrets or run artifacts.** `.env`, `state/`, and `*.csv` are
   gitignored; keep it that way.
 
+## Backward compatibility comes first
+Upgrading must never change what an existing call returns. Breaking a user's app
+is worse than any feature is good.
+
+What went wrong once: the agent extras added `state` / `appendix` / `highlights`
+as defaulted fields on `ScrapeResult` and `ScrapeOutcome`. That looked additive,
+but `asdict(r)` (how people serialise results) gained three `null` keys and
+`astuple(r)` grew from 4 values to 7. The CLI and HTTP JSON were fine; the Python
+objects were not. It was caught before merge and fixed with opt-in subclasses.
+
+Rules for any new feature:
+- **Never add, remove, rename or reorder fields on a public result type**
+  (`ScrapeResult`, `ScrapeOutcome`, `TierAttempt`, `SearchResult`). Put new data
+  on a subclass that is returned only when the caller opts in (see
+  `ScrapeResultWithExtras` in `switchback/orchestrator.py`).
+- **New parameters are keyword-only and default to the old behaviour.** Existing
+  parameters keep their names, order and defaults.
+- **CLI and HTTP JSON gain a key only when the request asked for it** (see
+  `result_dict` in `switchback/api.py`).
+- **A new dependency gets its own extra and stays out of `[all]`**, and a missing
+  dependency means the old behaviour. Installing a package is not opt-in: users
+  may already have it for something else. Anything that changes what an existing
+  call returns stays off until a `SCRAPER_*` variable turns it on (see
+  `SCRAPER_YOUTUBE`, `SCRAPER_DOCUMENTS`).
+- **The pinned tests are the contract.** The "backward compatibility" tests in
+  `tests/test_extras.py` pin the result fields and the CLI/HTTP JSON keys, and CI
+  runs them. If one fails, the change is breaking: redesign the change, don't
+  edit the pin.
+- **Before opening the PR, compare against `main` live:** scrape a few pages
+  with the default call on both; the output should be byte-identical.
+
 ## Tests
-`tests/test_suite.py` exercises the cascade across the anti-bot difficulty
+`pytest tests` runs the offline tests (fake tiers, no network, a few seconds);
+CI runs them on every PR. `tests/test_suite.py` exercises the cascade across the anti-bot difficulty
 spectrum (needs network + browser tiers): `python tests/test_suite.py --quick`
 for a fast tier-0/1 pass.
 

README.md

@@ -66,8 +66,8 @@ That's the whole loop. Add tiers as you need them (see [Install](#install)).
 
 | Tier | Strategy | Cost |
 |---|---|---|
-| tier_1 | Direct APIs / mirrors (arxiv, wikipedia, EuropePMC; extend: job boards) | free, cleanest |
-| tier_2 | Plain HTTP + TLS impersonation (`curl_cffi`), incl. PDFs | cheap |
+| tier_1 | Direct APIs / mirrors (arxiv, wikipedia, EuropePMC, YouTube transcripts; extend: job boards) | free, cleanest |
+| tier_2 | Plain HTTP + TLS impersonation (`curl_cffi`), incl. PDFs and office/ebook files | cheap |
 | tier_3 | Cloudflare / anti-bot solver (`cloudscraper`, install `.[cloudflare]`) | cheap-ish (~5s/host) |
 | tier_4 | Stealth headless browser (`patchright`, Chromium) | heavy |
 | tier_5 | Camoufox (Firefox stealth) — **on by default** (opt out: `SCRAPER_DISABLE_CAMOUFOX`) | heavy + slow (~40s on hard CF) |
@@ -94,9 +94,18 @@ pip install "switchback[browser]" && patchright install chromium   # + tier_4 st
 pip install "switchback[camoufox]" && camoufox fetch               # + tier_5 Firefox stealth
 pip install "switchback[firecrawl]"    # + tier_7 paid API (needs FIRECRAWL_API_KEY)
 pip install "switchback[tracing]"      # + OpenTelemetry -> any OTLP backend
-pip install "switchback[all]"          # everything
+pip install "switchback[all]"          # all of the above
+pip install "switchback[mcp]"          # + MCP server for agents (switchback-mcp)
+pip install "switchback[youtube]"      # + YouTube transcripts in tier_1 (and set SCRAPER_YOUTUBE=1)
+pip install "switchback[documents]"    # + .docx/.pptx/.xlsx/.xls/.epub → markdown in tier_2 (and set SCRAPER_DOCUMENTS=1)
 ```
 
+`mcp`, `youtube` and `documents` are not part of `[all]`, so upgrading an existing
+`[all]` install doesn't pull new dependencies (`documents` brings markitdown and
+onnxruntime). Add them by name. YouTube transcripts and office files also need
+their switch (`SCRAPER_YOUTUBE=1`, `SCRAPER_DOCUMENTS=1`), because they change what
+those URLs return; without it they behave exactly as before, installed or not.
+
 For tier_3's **full** v3 JS-VM + Turnstile + stealth, install the Enhanced Edition
 3.x fork (PyPI's `cloudscraper` is the older v1/v2 — PyPI forbids pinning a
 git-URL dep inside a published package, so install it alongside):
@@ -172,6 +181,61 @@ Non-Python callers: see [clients/node_bridge.md](clients/node_bridge.md). Python
 callers that want HTTP-with-CLI-fallback can drop in
 [clients/python_client.py](clients/python_client.py).
 
+**MCP server** (for agents: Claude Code, Cursor, any MCP client)
+```bash
+pip install "switchback[mcp]"
+claude mcp add switchback -- switchback-mcp           # stdio; tools: scrape, search
+```
+
+## Agent-ready extras
+
+Off by default: without them every result is exactly what it was before, the same
+`ScrapeResult` / `ScrapeOutcome` with the same fields, and the same CLI and HTTP
+JSON. Ask for any combination per call; results then come back as
+`ScrapeResultWithExtras` / `ScrapeOutcomeWithExtras` (subclasses, so existing
+`isinstance` checks still hold) with the extras you asked for filled in and the
+rest `None`. The CLI and HTTP JSON add only the keys you asked for.
+
+```python
+from switchback import scrape
+[r] = scrape("https://example.com/product", state=True, appendix=True,
+             query="what does it cost?")
+r.state       # {"__NEXT_DATA__": {...}, "json_ld": [...], "__INITIAL_STATE__": {...}}
+r.appendix    # [{"region": "nav", "markdown": "..."}, {"region": "footer", ...}]
+r.highlights  # [{"text": "...", "score": 7.1, "grader": "bm25",
+              #   "link": "https://example.com/product#:~:text=Price%20from,per%20month"}]
+```
+
+```bash
+switchback --state --appendix --query "what does it cost?" https://example.com/product
+curl -s localhost:8799/scrape -d '{"urls":["https://example.com"],"query":"pricing","state":true}'
+```
+
+| extra | what you get |
+| --- | --- |
+| `state` | The JSON the page embeds for its own frontend: Next.js `__NEXT_DATA__`, Nuxt, `window.__X__` globals (incl. `JSON.parse("…")`), schema.org JSON-LD. Often cleaner than the text (prices, ids, dates). Only valid JSON is returned. `{}` when the winning tier had no HTML (API mirrors, PDFs, docs, Firecrawl markdown). |
+| `appendix` | The menus, header, footer and sidebars that are stripped from the main content, as separate markdown, one entry per region. |
+| `query` → `highlights` | The passages that answer the question, best first, each with a citation `link`: a [text fragment](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Fragment/Text_fragments) URL that opens the page scrolled to, and highlighting, that passage (for a YouTube transcript, the video at that moment). If the text isn't visible on the page (e.g. it came from the `<title>` or image alt text) the browser simply opens the page at the top. |
+
+`state` and `appendix` need the fetched HTML, so they skip the content cache
+(`SCRAPER_CONTENT_TTL_S`) and fetch; `highlights` work on cached content too.
+
+**Grading highlights.** The default grader is local BM25 keyword scoring: free,
+offline, no data leaves the box. For better relevance, opt into
+[TypeSafe](https://typesafe.ai)'s Jev model, which answers "does this passage
+answer the question?" for every passage and returns a calibrated probability
+(passages under `SCRAPER_HIGHLIGHTS_MIN_SCORE`, default 0.5, are dropped, so an
+off-topic page honestly returns `[]`):
+
+```bash
+export SCRAPER_HIGHLIGHTS_GRADER=jev
+export TYPESAFE_API_KEY=...        # sends the page passages + query to api.typesafe.ai
+```
+
+Jev is billed per input token (about $0.0004 for a 10k-token page). If the call
+fails, that request falls back to BM25 (`"grader": "bm25"`); the scrape never fails
+because of highlights.
+
 ## Cost-scoped residential egress
 
 The dominant reason hard hosts wall you is the **datacenter IP**, not the
@@ -258,6 +322,13 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 - `SCRAPER_TIER_RETRY_ON` — failure classes eligible for a same-tier retry (default `timeout,rate_limited,connection`; widen to include `botwall,http_block` behind a rotating residential proxy so each retry gets a fresh IP). Retries are bounded by `SCRAPER_DEADLINE_S`; enabling them on the paid Firecrawl tier bills per attempt
 - `SCRAPER_LOGIN_HOOK` — `pkg.module:func` returning `{cookie: value}` for a host (see [Logged-in sessions](#logged-in-sessions))
 - `SCRAPER_EXTRACTION_FILE` — per-domain extraction prefs JSON (default `config/extraction.json`)
+- `SCRAPER_HIGHLIGHTS_GRADER` — `bm25` (default) or `jev` (needs `TYPESAFE_API_KEY`; see [Agent-ready extras](#agent-ready-extras))
+- `SCRAPER_HIGHLIGHTS_TOP_K` — highlights returned per URL (5)
+- `SCRAPER_HIGHLIGHTS_MIN_SCORE` — Jev probability below which a passage is dropped (0.5)
+- `SCRAPER_JEV_MODEL` / `SCRAPER_JEV_TIMEOUT_S` — Jev model alias (`jev-latest`) and per-request timeout (20s)
+- `SCRAPER_YOUTUBE` — opt-in: `1` returns YouTube transcripts from tier_1 (needs `switchback[youtube]`)
+- `SCRAPER_DOCUMENTS` — opt-in: `1` converts .docx/.pptx/.xlsx/.xls/.epub to markdown in tier_2 (needs `switchback[documents]`)
+- `SCRAPER_YOUTUBE_LANGS` — transcript languages to try, in order (`en`; e.g. `en,hi`). YouTube blocks most cloud IPs; when it does, the transcript is retried once through `SCRAPER_EGRESS_PROXY` if set
 - `SCRAPER_TRACE_SESSION` — opt-in: capture a Playwright trace (screenshots + DOM + network) per browser-tier attempt, written to `state/traces/`
 - `BENCH_FIRECRAWL_USD` / `BENCH_FIRECRAWL_HARD_MULT` — cost model for the savings report
 </details>
@@ -309,6 +380,20 @@ strip site-specific noise, declare prefs per host in `config/extraction.json`
 (see [config/extraction.example.json](config/extraction.example.json)); every
 tier's normalize step picks them up automatically.
 
+## Roadmap
+
+Not built yet:
+
+- **Layout-aware markdown** — keep pricing grids and card layouts built from
+  `<div>`s (not `<table>`s) as tables, using the browser tiers' rendered layout.
+- **Scanned pages and images** — OCR / vision-model text for PDFs with no text
+  layer and for images.
+- **Audio and caption-less video** — speech-to-text transcription.
+- **Hosted service** — managed browsers at high concurrency, stored results and
+  shareable result links. switchback stays a library you run yourself.
+- Smaller gaps: Next.js App Router flight data (`self.__next_f`) in `state`, and
+  YouTube chapters in transcripts.
+
 ## Contributing
 
 Issues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Start with the

pyproject.toml

@@ -53,6 +53,9 @@ browser = ["patchright"]                 # tier3: stealth headless Chromium
 camoufox = ["camoufox[geoip]"]           # tier3b: Firefox stealth (geoip: match locale/tz to proxy IP)
 firecrawl = ["firecrawl-py"]             # tier4: paid last-resort API
 server = ["fastapi", "uvicorn"]          # HTTP service (switchback.server) + /metrics + /traces
+mcp = ["mcp"]                            # MCP server for agents (switchback-mcp)
+youtube = ["youtube-transcript-api>=1.0"]   # tier_1: YouTube transcripts
+documents = ["markitdown[docx,pptx,xlsx,xls]"]  # tier_2: office/ebook files → markdown
 tracing = [                              # OTel → any OTLP backend
     "opentelemetry-sdk",
     "opentelemetry-exporter-otlp-proto-grpc",
@@ -62,11 +65,15 @@ all = [
     "fastapi", "uvicorn",
     "opentelemetry-sdk", "opentelemetry-exporter-otlp-proto-grpc",
 ]
+# Not in `all` on purpose: added in 0.6, kept out so existing `[all]` installs
+# don't suddenly pull markitdown's onnxruntime (via magika) or the MCP SDK.
+# Install them by name: "switchback[mcp,youtube,documents]".
 dev = ["pytest"]
 
 [project.scripts]
 switchback = "switchback.api:_main"
 switchback-server = "switchback.server:main"
+switchback-mcp = "switchback.mcp_server:main"
 
 [project.urls]
 Homepage = "https://github.com/akash-kr/switchback"

switchback/__init__.py

@@ -5,8 +5,10 @@
     results = scrape(["https://example.com/article"])
     hits = search("web scraping")            # query → URLs (SearXNG)
 """
-from .api import scrape, scrape_detailed, ScrapeOutcome, ScrapeResult, TierAttempt
+from .api import (scrape, scrape_detailed, ScrapeOutcome, ScrapeResult, TierAttempt,
+                  ScrapeOutcomeWithExtras, ScrapeResultWithExtras)
 from .search import search, SearchResult
 
 __all__ = ["scrape", "scrape_detailed", "ScrapeOutcome", "ScrapeResult",
-           "TierAttempt", "search", "SearchResult"]
+           "TierAttempt", "search", "SearchResult",
+           "ScrapeOutcomeWithExtras", "ScrapeResultWithExtras"]