[akash-kr]

TL;DR

A bot-wall page that embeds its logo as an inline base64 image was being returned as a successful scrape. Callers got the text "Request Blocked. You have been blocked…" labelled source_method: tier_4, and botwall recorded tier_4 as the host's winning tier. This PR makes the quality gates ignore inline data: image bytes when they scan and measure a page, and it adds Cloudflare's WAF block wording as a recognised wall.

• Files: switchback/policy/gates.py (+21/−4), 5 regression tests, CHANGELOG entry under Unreleased.
• Scope: no API, config or output-format change. The markdown returned to callers is byte-for-byte unchanged.
• Release: not released. It ships whenever you next cut a release.

---

The bug

Scenario (real, reproduced 2026-09-27)

switchback https://www.indeed.com/career-advice/finding-a-job/how-to-write-a-cover-letter

Before (on main): the cascade reports success at tier_4.

INFO    tier_2 botwall https://www.indeed.com/…: cloudflare challenge
WARNING tier_3 timeout https://www.indeed.com/…: cloudscraper exceeded 25.0s (unsolvable challenge)
INFO    tier_4 OK https://www.indeed.com/… md_len=7719 580ms (total 25744ms)

The "content" it returned (base64 elided):

Blocked - Indeed.com
![](data:image/svg+xml;base64,PHN2Zy…6,644 chars…)
![](data:image/svg+xml;base64,PHN2Zy…540 chars…)

# Request Blocked

You have been blocked. If you believe this in error, please go to support.indeed.com …
Your Ray ID for this request is a41c86…

Root cause

There are two checks in policy/gates.py, and the two inline logos (7,184 of the page's 7,719 chars, 93%) beat both of them:

Gate	What it does	Why it missed
classify_botwall	looks for wall phrases in the first 600 chars	"You have been blocked" starts at char 7,233, behind the logos
check length floor	rejects pages under 2,000 chars	the page is only ~540 chars of text, but the logos count toward length, so it measures 7,719

A third, smaller gap: "you have been blocked" (Cloudflare's WAF / error 1020 block copy) wasn't in _BOTWALL_MARKERS, so it wouldn't have matched even inside the window.

Why it matters

• Silent bad data. scrape() returns a block page as if it were the article. Nothing in the result says it failed.
• Wrong routing memory. botwall records tier_4 as the winner for the host, so later runs start at a tier that doesn't actually work there.
• No escalation. The cascade stops at the "success", so tiers 5–7 (which might have got the real page) never run.

---

The fix

_INLINE_DATA_URI = re.compile(r"\(data:[^)\s]*\)")

def _strip_inline_data(md):          # ![](data:…) -> ![]()
    return _INLINE_DATA_URI.sub("()", md)

1. classify_botwall scans the head after stripping inline data URIs. The browser tiers' mid-render looks_blocked() calls through here, so they're fixed too.
2. check measures length, and runs the nav-shell check, on the stripped text. It still returns the original md, so callers who want the images keep them.
3. New marker ("you have been blocked", "cloudflare"), placed with the vendor-specific phrases ahead of the generic ones.

Every tier reaches these gates, either through check() (tiers 1–7) or _browser.looks_blocked() (tiers 4–5), so this one file covers the whole cascade.

---

Accepted output (what "correct" looks like)

After (this branch): the same URL is correctly classified as a wall, and the cascade falls through.

INFO    tier_2 botwall https://www.indeed.com/…: cloudflare challenge
WARNING tier_3 timeout https://www.indeed.com/…: cloudscraper exceeded 25.0s (unsolvable challenge)
INFO    tier_4 botwall https://www.indeed.com/…: bot-wall / block page detected (cloudflare)   ← was "OK"
INFO    tier_5 short_content https://www.indeed.com/…: body too short: 0 < 2000
INFO    tier_6 disabled; skipping …
INFO    tier_7 disabled; skipping …
WARNING all tiers failed (37306ms, botwall): https://www.indeed.com/…

	Before	After (accepted)
tier_4 outcome	OK, md_len=7719	botwall, vendor cloudflare
scrape() result	1 result: the block page	0 results (no content returned)
scrape_detailed()	ok=True, final_outcome="ok"	ok=False, final_outcome="all_failed", error_class="botwall"
botwall DB	winning tier = tier_4	tier_4 recorded as a botwall; with Firecrawl or a residential proxy configured, tiers 6–7 get their turn

Failing on this IP is the correct result. The page really is blocked, and an honest failure is better than a fake success. With FIRECRAWL_API_KEY or BU_CDP_URL set, the cascade now reaches those tiers instead of stopping early.

No regressions on real content

Re-ran known-good pages through the full cascade on this branch:

URL	Result on main	Result on this branch
pcmag.com/news	tier_2 OK, 18,518	tier_2 OK, 18,518
investing.com/news/stock-market-news	tier_2 OK, 38,666	tier_2 OK, 38,666
etsy.com/market/ceramic_mug	tier_2 OK, 18,825	tier_2 OK, 18,830 (live page drift)
fiverr.com/resources/guides	tier_2 OK, 73,673	tier_2 OK, 73,673
quora.com/What-is-web-scraping (the README demo)	tier_5 OK	tier_5 OK

---

Tests

tests/test_quality_gate.py: 5 new tests, offline, pure unit. The fixture mirrors the real Indeed page (same logo sizes, placeholder IP/Ray ID).

Test	Guards
test_block_page_behind_inline_images_is_botwall	the repro: must raise BotWall(vendor="cloudflare")
test_cloudflare_waf_block_page_is_botwall	stock "Sorry, you have been blocked" page
test_inline_images_do_not_count_toward_length_floor	thin page padded by an image is still ShortContent
test_real_article_with_inline_images_passes_unchanged	real article with images passes, returned byte-for-byte
test_block_phrase_deep_in_article_body_passes	head-only scan preserved: an article that mentions the phrase isn't a wall

• Red first: on main the three bug tests fail with DID NOT RAISE, and the two guard tests pass.
• Green: pytest -q → 36 passed (31 existing + 5 new).

pytest -q tests/test_quality_gate.py      # 10 passed

---

Review notes / things to decide

• Behaviour change (intended): a page whose text is under the floor but was padded past it by inline images now falls through as short_content, where before it passed. That's the point of the fix, but it's the one place real output could change. I found no such case in the regression URLs.
• Marker precision: "you have been blocked" is only matched in the first 600 chars of text, the same rule as every other marker, so an article quoting it in its body still passes (see the last test). If you'd rather keep the list minimal, the marker is optional: the stripping change alone already rejects the Indeed page, as short_content (~540 chars < 2000) instead of botwall. The marker makes the vendor learnable (botwall/cloudflare), which drives egress escalation.
• Logged md_len in the orchestrator's OK line still counts raw length, including images. I left it alone because it describes the payload returned, not what the gate measured.
• Not touched: is_cf_challenge (solvable-challenge detection). A WAF block isn't solvable by tier_3, so it correctly stays out of _CF_CHALLENGE_MARKERS.