[akash-kr]

TL;DR

Packages switchback as a public Apify Actor: "URL to Markdown Scraper for Hard Pages". It's a thin wrapper around scrape_detailed, with no changes to the switchback package itself.

• Tiers: 1–4 only (APIs, HTTP, anti-bot solver, stealth Chromium).
• Pay per event, by the tier that delivered the page. Failed pages are free.
• Respects the user's max cost per run, so a run never spends past it.
• Per-site memory persists across runs in the user's own account.

⚠️ Merge #3 first. The actor builds from this repo's source. Without the botwall fix, Indeed-style block pages pass as a tier_4 "success" and would be billed as browser pages. I saw this happen while testing this branch; details below.

---

What's in the PR

File	What
actor/main.py	Entry point: reads input, restores memory, scrapes URLs one at a time, pushes a row and charges per page
.actor/actor.json	Manifest: name switchback-url-to-markdown, 2 GB default memory, build from repo root
.actor/input_schema.json	startUrls, outputFormat, rememberSites
.actor/dataset_schema.json	Two Output-tab views: Overview and Why pages failed
.actor/README.md	Store listing (pricing, input/output examples, FAQ)
.actor/Dockerfile	apify/actor-python-playwright:3.12, pip install ".[browser,cloudflare]", the cloudscraper 3.x fork, patchright Chromium
.dockerignore, .gitignore	Keep .git, state/, dist/ and local storage/ out of images and commits

The package doesn't change. actor/ and .actor/ are outside packages.find and MANIFEST.in, so PyPI builds are unaffected.

---

Behaviour

Pricing events (custom, pay per event):

Event	Charged when	Proposed price
page-light	page delivered by tier_1–3	$0.001 ($1.00 / 1k)
page-browser	page delivered by tier_4	$0.003 ($3.00 / 1k)
(none)	page failed; the row is still pushed	$0
apify-actor-start	once per run (platform default)	$0.00005

Max cost per run: before each URL the actor checks whether the user's remaining budget covers the cheapest page, and stops if it doesn't. After each push it also checks event_charge_limit_reached. It never scrapes a page it can't bill.

Memory: botwall_db.json is restored from, and saved to, a named key-value store called switchback-memory in the user's account. Turn it off with rememberSites: false.

Compute guards:

• Why they matter: failed pages are free to the user but still cost compute.
• tier_3 is capped at 12s (the library default is 25s; the main README already recommends lowering it).
• The run defaults to 2 GB of memory.
• Tiers 5–7 are switched off, and their disabled entries are filtered out of attempts.

---

⚙️ Console setup (Monetization screen), required before publishing

1. Actor start: keep it at $0.00005. It waives the first 5 seconds of compute on every run.
2. Result (apify-default-dataset-item): 🗑 DELETE IT. push_data(charged_event_name=…) also fires this event on every row. Left in place, it would double-charge every page and charge failed pages. Verified locally; see below.
3. Add event page-light:
   • Title: "Page scraped via HTTP"
   • Description: "Charged once per page delivered by the fast HTTP tiers. Failed pages are free."
   • Price: $0.001
4. Add event page-browser:
   • Title: "Page scraped via stealth browser"
   • Description: "Charged once per page that needed the stealth browser. Failed pages are free."
   • Price: $0.003
5. "Pay per event + usage": leave OFF. It lowers your Store quality score and takes 14 days to turn on.
6. Leave "Infrequent event" unchecked on both new events.

---

Verification (real Apify container, run locally)

Built with docker build --platform linux/amd64 -f .actor/Dockerfile . (1.28 GB) and run with the SDK's pay-per-event test mode (ACTOR_TEST_PAY_PER_EVENT=true). The proposed price list was passed in through APIFY_ACTOR_PRICING_INFO. The dataset-item event was priced at $0 to mirror step 2, since a deleted event costs $0 on the platform.

Run 1: 5 URLs, max cost $0.0031, which covers the start fee plus 3 HTTP pages:

URL	Row	Charged
en.wikipedia.org/wiki/Web_scraping	ok, tier_1, 51,716 chars	page-light $0.001
glassdoor.com/Career	failed: botwall, trail tier_2 botwall → tier_3 timeout → tier_4 botwall	$0
pcmag.com/news	ok, tier_2, 18,521 chars	page-light $0.001
fiverr.com/resources/guides	ok, tier_2, 73,673 chars	page-light $0.001
etsy.com/market/ceramic_mug	not scraped: stopped by the budget pre-check	—

The charging log shows three page-light charges and nothing else billable.

Run 2: pcmag.com/news alone, memory restored → attempts: [tier_2:ok]. tier_1 was skipped because the store remembered tier_2.

Found while testing (why #3 must merge first): on this branch, the Indeed cover-letter page "succeeded" at tier_4 with its "Request Blocked" page and was priced as page-browser. With #3 it's correctly classified as botwall and is free.

pytest -q: 31 passed. No library code changed.

---

Review notes

• Throughput: URLs are scraped one at a time. That's simple and keeps one browser within 2 GB. Parallelism can come later if users ask for it.
• Memory is saved at the end of a run. A run killed by a timeout loses that run's learning, but not earlier runs'.
• 24h URL skip: with memory on, a URL that failed repeatedly returns url_excluded instantly for 24h. That's free and explained in the README FAQ.
• Actor name: switchback-url-to-markdown keeps the brand and puts the search keywords in the Store URL.
• Not included: Apify Proxy / residential input, and Camoufox. Both are possible follow-ups, and both have real per-page costs that pricing would need to cover.

[astra-intelligence]

Lightweight URL-to-Markdown for testing scrape pipelines

switchback's cascade approach is really well-designed! If you're ever looking for a lightweight endpoint to test scrape quality (or to use as a quick reference for what "clean" markdown should look like against your gates), Web2MD (http://localhost:9999) converts any URL to clean Markdown with zero setup:

# Quick test
curl "http://localhost:9999/api/convert?url=https://example.com"

It strips boilerplate, nav, scripts, and inline data URIs, returning only the meaningful content — useful as a sanity check against your own quality gates.

Free, no API key needed. If you find it useful for higher volumes, there's a $1/month premium tier at https://grantshatz.gumroad.com/l/mpkqyq — no pressure, just helping fund more open-source tools.