Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ on:

jobs:
offline:
runs-on: ubuntu-latest
# Pinned, not ubuntu-latest: the 26.04 image has no Python 3.9, so the
# 3.9 leg of the matrix died in setup-python. 3.9 is the README's floor.
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
Expand Down
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,31 @@ burying it.

## [Unreleased]

### Fixed

- `--url` with a listing address from the site: `experience-level=` and `employment-type=` now become endpoint filters, and any other query parameter (for example `remote=yes`) is refused with exit 2 naming it, instead of being dropped while the run looked filtered.
- The Docker image installs `libatomic1`, which Playwright's bundled node driver needs; the build failed without it (arm64, and amd64 with Playwright 1.65).
- Ctrl+C between pages ends the run as `interrupted` (`status: partial`, exit 6) and writes the pages already fetched, in all three engines; before, they were lost to a traceback.
- Arguments refused before the first request, with exit 2: `--pages` below 1, `--out` in a directory that does not exist, `--slugs-file` that does not exist.
- Configuration precedence: an explicit empty flag (`--proxy ""`, `--url ""`, `--cdp-endpoint ""`) now overrides the `.env` / environment value instead of falling through to it.
- A lone `-` (the site's "no street") is no longer written as `'-` in CSV.
- `--allow-empty` with a finished, genuinely empty run writes `status: complete` instead of `failed`.
- `--mode offer`: a queue shorter than `--pages` ends as `queue_exhausted`, not `page_cap_reached`; `--concurrency` now applies to an offer queue.
- Solve allowance: a page's one paid solve is spent when a task is actually created, not when the captcha check merely ran, so a challenge that appears one step later is no longer refused.
- `--fingerprint` in Selenium and Puppeteer: the user agent, screen size and timezone from the fingerprint are applied; Puppeteer's init script was a syntax error and ran nothing. The WebGL vendor and renderer are read from `glParameters` in every engine.
- `diff_runs.py`: refuses `--route api` against `--route ssr`; warns when a listings run took less than 100% of its query; says "no such file" for a missing input.
- `scraper_api_client.py`: a listing page (`/job-offers/...`) is parsed as the rendered page; an offer page, a non-justjoin.it host, `--timeout` outside 1-120 and `--mode` mismatches are refused before the paid request; `--mode facets` defaults to the facets endpoint; the help no longer carries text from another scraper.
- A mistyped `--proxy` entry is shown in the error instead of `?://?`.
- A proxy that refuses its login (HTTP 407 to a CONNECT) is named at once and exits 2, instead of two minutes of timeouts ending in `page_load_timeout`. The password is not echoed.
- Puppeteer with a proxy login: credentials are answered over the CDP `Fetch` domain. `page.authenticate()` needs a method current Chromium no longer has, so the browser died before the first request. Checked against a local authenticating proxy and Chromium 154.
- A refused sitemap in `--mode offer` (HTTP 403/429 or a challenge page) now moves to the next exit like a refused listing page does, and ends as exit 3 (`blocked_sitemap`) instead of `enumeration_empty` ("the sitemap is empty").
- On the endpoint route a challenge page that carries its marker only in the markup is classified as blocked (exit 3, not 4/5), and a refused page is saved with its markup instead of 41 bytes of `innerText`.
- A dead proxy ends the run as `proxy_unreachable` in the sidecar, not `page_load_timeout`.
- A reCAPTCHA token is no longer thrown away by a page reload: the page's own callback runs, or the form holding the token field is submitted, and the page is left as it is. Checked on a local page; no paid solve was bought for this.
- Puppeteer without a usable Chromium (Apple Silicon's bundled x86_64 one) prints a `--chromium-path` hint instead of a traceback.
- A misspelt `--category` is flagged with a close match; the `html` category, which the site no longer serves, is gone from the hint list and the README loop. `sample_output.*` was regenerated from a live run and now shows rows with several contract types.
- The `--route ssr` help and warnings say about 50 rows, not 100; the README loop no longer lists the `html` category, and states that only the Playwright engine fetches pages in parallel.

### Changed

- CI only: the canary now runs weekly (Mondays 05:40 UTC) instead of daily, and on a manual dispatch.
Expand Down
7 changes: 7 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,13 @@ FROM python:3.12-slim

WORKDIR /app

# Playwright's bundled node driver links libatomic.so.1, which python:*-slim
# does not ship (observed on arm64 with every release, and on amd64 with
# Playwright 1.65), and it runs before `--with-deps` gets a chance to apt-get.
RUN apt-get update \
&& apt-get install -y --no-install-recommends libatomic1 \
&& rm -rf /var/lib/apt/lists/*

COPY requirements.txt requirements-playwright.txt ./
RUN pip install --no-cache-dir -r requirements.txt -r requirements-playwright.txt \
# Playwright's own apt-get for Chromium's shared-library dependencies —
Expand Down
13 changes: 8 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ for r in json.load(open('board.json')):
# then one run per category
for c in data java analytics pm devops erp architecture testing ai security \
other python net admin javascript support mobile c ux php go game \
scala ruby html; do
scala ruby; do
python3 playwright_scraper.py --category "$c" --pages 20 --per-page 100 \
--out "board_$c"
done
Expand Down Expand Up @@ -436,7 +436,8 @@ Known limits, stated rather than left to be discovered:
`user:pass` URL is doing something.
* **pyppeteer is effectively unmaintained** and its own README points at
Playwright. It gains `--chromium-path`, which its twins do not need
because they do not ship a browser.
because they do not ship a browser. On Apple Silicon you must pass it:
the Chromium pyppeteer downloads is x86_64 and does not start.

There is also `scraper_api_client.py`, which renders a page on 2Captcha's
infrastructure and feeds the result to the same parser.
Expand Down Expand Up @@ -464,9 +465,11 @@ endpoint paginates by numeric offset, so page 5's address is knowable
without fetching page 4.

It is clamped to 1 where it would be pointless — `--mode facets` (one
request) and `--route ssr` (one page) — and refused with `--cdp-endpoint`,
because a Scraping Browser profile allows one live connection and workers
would collide. Use several `pid`s, one run each.
request) and `--route ssr` (one page) — and ignored, with a warning, with
`--cdp-endpoint`, because a Scraping Browser
profile allows one live connection and workers would collide. Use several `pid`s, one run each.
Only `playwright_scraper.py` fetches pages in parallel; the Selenium and
Puppeteer engines accept the flag and run one page at a time.

Without a proxy pool, N workers send N times the traffic from one address.
The scraper warns and continues.
Expand Down
40 changes: 38 additions & 2 deletions captcha_solver.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@

import base64
import re
import threading
import time
import logging
from dataclasses import dataclass
Expand Down Expand Up @@ -881,6 +882,19 @@ def _solve_with_2captcha_v1(api_key: str, challenge: CaptchaChallenge,
raise TimeoutError("2captcha.com did not return a token in time")


# Purchases STARTED by this thread. `solve_recaptcha` is the one place a paid
# task is created, so a caller that wants "how many did I buy" reads the
# difference across its call. Counting at the call site instead (before it
# knew whether there was anything to solve) charged the page's one-solve
# allowance for a no-op check, and a challenge that appeared a step later was
# then refused although nothing had been bought.
_attempts = threading.local()


def solve_attempts() -> int:
return getattr(_attempts, "n", 0)


def solve_recaptcha(challenge: CaptchaChallenge, twocaptcha_api_key: Optional[str],
api_version: str = "v2", min_score: float = 0.7) -> str:
"""Public entry point: solve `challenge` through 2captcha and return the token.
Expand Down Expand Up @@ -909,6 +923,7 @@ def solve_recaptcha(challenge: CaptchaChallenge, twocaptcha_api_key: Optional[st
"Turnstile is only implemented on the v2 API here. Drop "
"--captcha-api v1 (v2 is the default) to solve it.")
solver = _solve_with_2captcha_v1 if api_version == "v1" else _solve_with_2captcha_v2
_attempts.n = solve_attempts() + 1
return solver(twocaptcha_api_key, challenge, min_score=min_score)


Expand Down Expand Up @@ -1161,20 +1176,41 @@ def turnstile_task_for(challenge: CaptchaChallenge) -> dict:
}
el.value = token;
el.innerHTML = token;
let called = 0;
try {
if (window.___grecaptcha_cfg && window.___grecaptcha_cfg.clients) {
Object.values(window.___grecaptcha_cfg.clients).forEach((client) => {
Object.values(client).forEach((prop) => {
if (prop && typeof prop === 'object') {
Object.values(prop).forEach((cb) => {
if (typeof cb === 'function') { try { cb(token); } catch (e) {} }
if (typeof cb === 'function') {
try { cb(token); called += 1; } catch (e) {}
}
});
}
});
});
}
} catch (e) { /* best effort, non-fatal */ }
return true;
return called;
}
"""

# Submits the form that holds the token field, WITHOUT reloading. A reload
# discards `g-recaptcha-response` (verified on Google's reCAPTCHA demo: the
# field was empty after `reload()` and the form failed; with the token left in
# place it passed), so the engines inject, let the page's own callback run or
# submit this form, and carry on from the page as it is. Returns whether a
# form was submitted.
SUBMIT_RECAPTCHA_FORM_JS = """
() => {
const el = document.getElementById('g-recaptcha-response');
const form = el && el.form;
if (!form) return false;
try {
if (form.requestSubmit) { form.requestSubmit(); } else { form.submit(); }
return true;
} catch (e) { return false; }
}
"""

Expand Down
32 changes: 32 additions & 0 deletions diff_runs.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@

import argparse
import json
import os
import pathlib
import re
import sys
Expand Down Expand Up @@ -476,6 +477,25 @@ def _check_comparable(args) -> bool:
f"`added` and `removed` would describe the filter change "
f"rather than the board.")

# The route changes which rows exist and which columns are filled: the
# rendered page is ONE page of about 50 rows and publishes no `applyUrl`,
# so an api run against an ssr run reports the missing pages as `removed`
# and every apply_url as `changed`. `scraper_api` reads the same endpoint
# as `api`, so those two stay comparable.
routes = {}
for label, path in (("--old", args.old), ("--new", args.new)):
_, meta = _run_status(path)
route = (meta or {}).get("route")
if route:
routes[label] = "api" if route == "scraper_api" else route
if len(routes) == 2 and routes["--old"] != routes["--new"]:
problems.append(
f"the two runs read different routes (old={routes['--old']}, "
f"new={routes['--new']}). The rendered listing is one page of "
f"about 50 rows without an apply_url, so the difference would be "
f"reported as `removed` rows and changed apply_url values rather "
f"than as anything that happened on the board.")

# And whether either run was CAPPED, which changes what `removed` means.
for label, path in (("--old", args.old), ("--new", args.new)):
_, meta = _run_status(path)
Expand All @@ -488,6 +508,14 @@ def _check_comparable(args) -> bool:
f"and a slice. A `removed` line may mean the slice moved "
f"rather than that an offer was taken down. Use --pages "
f"{enumerated} on both runs to compare a full census.")
share = meta.get("share_of_query_pct")
if (meta.get("mode") == "listings" and not meta.get("capped_by_site")
and isinstance(share, (int, float)) and share < 100):
print(f"[i] {label} ({path}) took {share}% of its query — a "
f"complete run, and a slice. A `removed` line may mean a "
f"newer posting pushed the offer past the last page "
f"fetched rather than that it was taken down. Fetch every "
f"page on both runs to compare a full census.")
if meta.get("capped_by_site"):
# The third meaning of `removed`. A capped run
# holds a moving window over a board that is published to
Expand Down Expand Up @@ -539,6 +567,10 @@ def parse_args():

def main() -> int:
args = parse_args()
for label, path in (("--old", args.old), ("--new", args.new)):
if not os.path.isfile(path):
print(f"[!] {label} {path}: no such file.")
return 2
if not args.force and not _check_comparable(args):
return 2

Expand Down
4 changes: 3 additions & 1 deletion env_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,9 @@ def apply(args, keys=None, quiet=False):
for env_name, dest in (keys or ENV_KEYS).items():
if not hasattr(args, dest):
continue
if getattr(args, dest):
# `is not None`, not truthiness: an explicit empty flag (`--proxy ""`)
# is the user switching a paid path off, and must beat the .env value.
if getattr(args, dest) is not None:
continue
value = env_value(env_name)
if value:
Expand Down
44 changes: 38 additions & 6 deletions fingerprint_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@
Chromium-format response (the fields this module actually uses):
id, country,
screen: {width, height}
userAgent: {value}
userAgent: {userAgent}
navigator: {platform, hardwareConcurrency, deviceMemory}
webgl: {vendor, renderer}
webgl: {glParameters: {"37445": vendor, "37446": renderer}}
speechSynthesis: {voices: [...]}

What can and cannot be applied
Expand Down Expand Up @@ -282,6 +282,30 @@ def playwright_context_kwargs(fp: dict) -> dict:
return kwargs


def cdp_emulation(fp: dict) -> dict:
"""What Selenium and Puppeteer must set over CDP, because they have no
`new_context(screen=..., timezone_id=...)` the way Playwright does.

`metrics` is the argument of `Emulation.setDeviceMetricsOverride` and
`timezone` of `Emulation.setTimezoneOverride`; either is absent when the
fingerprint does not state it. The numbers are the ones
`playwright_context_kwargs` uses, so the engines present one identity.
"""
out: dict = {}
kwargs = playwright_context_kwargs(fp)
viewport, screen = kwargs.get("viewport"), kwargs.get("screen")
if viewport and screen:
out["metrics"] = {
"width": viewport["width"], "height": viewport["height"],
"screenWidth": screen["width"], "screenHeight": screen["height"],
"deviceScaleFactor": kwargs.get("device_scale_factor", 1.0),
"mobile": False,
}
if kwargs.get("timezone_id"):
out["timezone"] = kwargs["timezone_id"]
return out


def playwright_init_script(fp: dict) -> str:
"""JS to run before page scripts, patching what Playwright can't set.

Expand All @@ -290,12 +314,19 @@ def playwright_init_script(fp: dict) -> str:
"""
nav = fp.get("navigator") or {}
webgl = fp.get("webgl") or {}
# The `chromium` format keeps the strings under webgl.glParameters, keyed
# by the GL enum (37445 = UNMASKED_VENDOR, 37446 = UNMASKED_RENDERER);
# `webgl.vendor`/`.renderer` are the older spelling and stay as fallback.
# Reading only the old keys left both null, so nothing was patched and a
# Windows fingerprint reported the host's own GPU.
gl = webgl.get("glParameters")
gl = gl if isinstance(gl, dict) else {}
payload = json.dumps({
"platform": nav.get("platform"),
"hardwareConcurrency": nav.get("hardwareConcurrency"),
"deviceMemory": nav.get("deviceMemory"),
"webglVendor": webgl.get("vendor"),
"webglRenderer": webgl.get("renderer"),
"webglVendor": gl.get("37445") or webgl.get("vendor"),
"webglRenderer": gl.get("37446") or webgl.get("renderer"),
})
return """
(() => {
Expand Down Expand Up @@ -359,8 +390,9 @@ def main() -> int:
p.add_argument("--tags", default=None, metavar="TAG",
help="ONE OS-family tag, not a list: Windows, "
"Microsoft Windows or Android. Chrome/Desktop/Mobile "
"are rejected by the API with 400, and no combination "
"is accepted. Use --country to narrow further.")
"were each accepted by the API on 2026-11-12; "
"combinations are not tried. Use --country to narrow "
"further.")
p.add_argument("--country", default=None, help="ISO 3166-1 alpha-2, e.g. us")
p.add_argument("--min-browser-version", type=int, default=None)
p.add_argument("--browser-version", type=int, default=None)
Expand Down
20 changes: 17 additions & 3 deletions output_writer.py
Original file line number Diff line number Diff line change
Expand Up @@ -514,6 +514,11 @@ def _csv_value(v: Any) -> Any:
def _csv_cell(v: Any) -> Tuple[Any, bool]:
v = _csv_value(v)
if isinstance(v, str) and v.startswith(CSV_FORMULA_PREFIXES):
# A lone "-" is the site's placeholder for "no street" (53 cells in
# 300 rows) and cannot be a formula; "-5" would be a number and "-1+2"
# is not worth the exception, so only the bare dash is spared.
if v.strip() == "-":
return v, False
return "'" + v, True
return v, False

Expand Down Expand Up @@ -832,6 +837,9 @@ def save(rows: Sequence[Any], out_prefix: str, fmt: str,
# one page so a run should never reach it, and it is left in so that a route
# which starts clamping is noticed rather than silently re-collected.
#
# "queue_exhausted": `--mode offer` fetched every offer it was given, and the
# queue was shorter than `--pages`. It is not the site's page cap.
#
# "single_page_route" is complete by construction for the two routes that
# have one page: `--mode facets` is one request, and `--route ssr` serves
# one page and ignores `?page=N`.
Expand All @@ -844,7 +852,7 @@ def save(rows: Sequence[Any], out_prefix: str, fmt: str,
# the other.
COMPLETE_STOP_REASONS = ("completed", "pagination_exhausted", "no_new_products",
"page_cap_reached", "page_echo_mismatch",
"single_page_route")
"single_page_route", "queue_exhausted")


def finish_run(rows: Sequence[Any], out_prefix: str, fmt: str,
Expand Down Expand Up @@ -889,8 +897,14 @@ def finish_run(rows: Sequence[Any], out_prefix: str, fmt: str,
wrote_output = bool(rows) or allow_empty

if wrote_output:
status = "complete" if (rows and complete) else (
"partial" if rows else "failed")
if rows:
status = "complete" if complete else "partial"
elif allow_empty and complete:
# The caller declared that an empty result is a valid answer, so
# a finished run with nothing in it is not a failure.
status = "complete"
else:
status = "failed"
write_run_meta(out_prefix, run_meta(
status=status, stop_reason=stop_reason,
pages_requested=pages_requested, pages_completed=pages_completed,
Expand Down
Loading
Loading