[akash-kr]

TL;DR

Two new opt-in environment variables for deployments that scrape URLs submitted by people they don't fully trust (a "save this link" box, a feed reader, a public API):

Variable	What it does when set	When unset (default)
SCRAPER_DISABLE_CLOUDSCRAPER=1	Skips tier_3 (cloudscraper); the cascade moves on to the browser tiers	tier_3 runs exactly as today
SCRAPER_BROWSER_BLOCK_PRIVATE=1	Tiers 4 and 5 refuse any request or WebSocket the page makes to a loopback, private, link-local (incl. cloud metadata) or CGNAT address	No routing is installed; the browser behaves exactly as today

Nothing changes for existing users unless they set a variable. That's pinned by tests (see Proof the default is unchanged).

---

Background: why this is needed

Switchback is often pointed at URLs that someone else chose. Two parts of the cascade then trust the page more than they should.

1. tier_3 runs site JavaScript outside a sandbox.
To clear a Cloudflare JS challenge, cloudscraper evaluates JavaScript the site sent. It uses either js2py (pure Python) or, when Node is installed, Node's built-in vm module. Neither is designed as a security boundary: js2py has a published vulnerability advisory with no fixed release (and has had sandbox escapes reported), and Node's own docs say vm is not a security mechanism. So a hostile page posing as a challenge could, in principle, run code on the scraping machine.
The browser tiers (patchright/Chromium, Camoufox/Firefox) also run page JavaScript, but inside a real browser sandbox. That's the right place for untrusted code.

2. The browser tiers will fetch anything the page asks for.
A scraped page can redirect, frame, embed or fetch() an address like http://127.0.0.1:<port>/ or a cloud metadata endpoint. Whatever a service on the scraping machine (or its private network) answers then lands in the DOM, and so in the scrape result. Callers can check the URL they pass in, but they can't see what the page does once it's loaded.

Both are reasonable defaults for scraping your own list of public sites, which is why they stay the defaults. They're the wrong defaults when the URLs come from users.

---

What changed

File	Change	Lines
switchback/tiers/tier_3.py	disabled() hook reading SCRAPER_DISABLE_CLOUDSCRAPER (same pattern as SCRAPER_DISABLE_CAMOUFOX on tier_5); docstring explains when to use it	+11
switchback/tiers/_browser.py	The guard: is_private_url(), context_options() and block_private_networks(context), all no-ops unless SCRAPER_BROWSER_BLOCK_PRIVATE is set	+92
switchback/tiers/tier_4.py	new_context(**_browser.context_options()) + _browser.block_private_networks(ctx)	2 lines
switchback/tiers/tier_5.py	Same two calls on the Camoufox page/context	2 lines
switchback/doctor.py	Shows tier_3 as off (SCRAPER_DISABLE_CLOUDSCRAPER set); a tier you turned off on purpose doesn't fail the healthcheck	+4
tests/test_untrusted_urls.py	New: pins the default path, tests each switch, plus 4 live browser checks	new
README.md, CHANGELOG.md	Both variables documented, plus a short "Scraping URLs other people submit" section	docs

How the guard works when on:

• context.route("**/*") checks each request's host. A literal IP is checked directly; a hostname is resolved once per host per scrape (cached). Non-public addresses get route.abort("blockedbyclient") and everything else continues.
• context.route_web_socket() does the same for WebSockets (Playwright ≥ 1.48; skipped if the API is missing).
• Contexts get service_workers="block", because Playwright routing doesn't see requests made by a service worker.
• data:, blob: and about: URLs, and hosts that don't resolve, aren't judged. The browser handles those exactly as before.

How tier_3 being off flows through the cascade: the orchestrator already honours a tier's disabled() (as it does for tier_5 and tier_7). The attempt is recorded as disabled and the next tier runs. That holds even for a host whose learned winning_tier is tier_3, so no host gets stranded.

---

Proof the default is unchanged

• No setting → identical code path. context_options() returns {}, so new_context()/new_page() get no extra arguments. block_private_networks() returns before touching the context, and tier_3.disabled() is False.
• Tests that pin the default (tests/test_untrusted_urls.py):
  • test_by_default_cloudscraper_runs_and_browsers_are_untouched: no route or WebSocket handler is installed, and no context options are added.
  • test_by_default_doctor_reports_and_exits_exactly_as_before: same row and same exit codes, including exit 1 when cloudscraper is genuinely missing.
  • test_cloudscraper_still_runs_when_not_opted_out: the cascade still uses tier_3.
  • test_live_…[False] (real Chromium): a page can still read a local service with the guard off, i.e. today's behaviour is preserved.
• Smoke suite, before vs after: python tests/test_suite.py --quick gives the same result on main and on this branch: 6 passed, 1 failed (Ars Technica homepage, which fails identically on main from this server).
• Existing tests: all 31 still pass, unchanged.

---

What correct output looks like

switchback --doctor with the defaults (unchanged):

  [OK  ] tier_3 (cloudscraper)          cloudscraper 3.0.0
  [OK  ] tier_4 (browser)               patchright + Chromium ready
...
Capable tiers ready.            (exit 0)

switchback --doctor with SCRAPER_DISABLE_CLOUDSCRAPER=1:

  [MISS] tier_3 (cloudscraper)          off (SCRAPER_DISABLE_CLOUDSCRAPER set)
  [OK  ] tier_4 (browser)               patchright + Chromium ready
...
Capable tiers ready.            (exit 0 — off on purpose isn't a failure)

A scrape with SCRAPER_DISABLE_CLOUDSCRAPER=1: the outcome's attempts start with tier_3 → disabled and the result comes from a later tier. The log shows tier_3 disabled; skipping <url> (the orchestrator's existing message).

A scrape with SCRAPER_BROWSER_BLOCK_PRIVATE=1 where the page tries to reach a local service:

• The scrape still succeeds with the page's own content.
• The local service is never contacted, and its data isn't in the result.
• The log shows browser: blocked a request to a non-public address (127.0.0.1). Only the hostname is logged, never the full URL.

What would be wrong: tier_3 being skipped with the variable unset; any change to doctor's output with nothing set; a public page failing to load with the guard on. Any of these is a bug in this PR.

---

Tests run

Suite	Result
pytest tests --ignore=tests/test_suite.py	54 passed, 4 skipped (the live checks, gated)
Live checks: SWITCHBACK_BROWSER_TESTS=1 pytest tests/test_untrusted_urls.py -k live (real Chromium, two local HTTP servers)	4 passed: guard off → page reads the local service; guard on → blocked, local service received 0 requests; same through tier_4.fetch() end to end
tests/test_suite.py --quick on branch vs main	6/7 on both, same pre-existing failure

To re-run the live checks: patchright install chromium-headless-shell, then SWITCHBACK_BROWSER_TESTS=1 pytest tests/test_untrusted_urls.py -k live.

---

How to review (about 10 minutes)

1. switchback/tiers/tier_3.py: the disabled() hook (5 lines).
2. switchback/tiers/tier_4.py and tier_5.py: the two-line call sites. Confirm both are no-ops when unset.
3. switchback/tiers/_browser.py: the guard (bottom of the file).
4. switchback/doctor.py: the display and exit-code tweak.
5. tests/test_untrusted_urls.py: the first section pins the defaults.

Known limits (documented in the code and README)

• DNS rebinding: hosts are resolved by the guard; a host whose DNS answer changes between that check and the browser's own lookup isn't caught. Same for a host only a proxy resolves differently.
• WebRTC isn't intercepted. It can reach local addresses but can't read HTTP responses.
• HTTP tiers (1–3) aren't covered by the browser guard. Callers should still validate the URL they pass in.
• tier_6 (remote CDP browser) is excluded: its network is the remote machine's, not this one.

Not in this PR

• No change to any default, and no version bump (CHANGELOG entries sit under Unreleased).
• Whether these should become the defaults in a future major release is a separate decision.

Rollout for OwnFeed (after merge)

1. Install tier_4's browser on the server (patchright install chromium). With tier_3 off, that's what clears Cloudflare.
2. Set SCRAPER_DISABLE_CLOUDSCRAPER=1 and SCRAPER_BROWSER_BLOCK_PRIVATE=1 in OwnFeed's environment.
3. Bump OwnFeed's locked switchback commit, restart, and check --doctor plus a few real saves.