Skip to content

Stop at the site's count, keep proxy credentials out of every message, flag mixed currencies - #16

Open
jehrr wants to merge 1 commit into
mainfrom
fix/audit-2026-11-09
Open

jehrr wants to merge 1 commit into
mainfrom
fix/audit-2026-11-09

Conversation

@jehrr

@jehrr jehrr commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Fixes from a manual test of 2026-11-02 and 2026-11-05, re-checked against main @ b176941 on 2026-11-09.

Behaviour changes

Pagination and waits

  • No page past the end of the listing is requested. When Flippa's own match count says the listing ends, the run stops there as listing_exhausted. This applies to all three browser engines and to Scraper API mode, where each extra page would be a billed task. A 34-match search is now one request.
  • The 15 s readiness wait ends as soon as a short page holds the number of cards its own count implies (the last page, a 4-match search, "Show 0 results", a page past the end).
  • addressable is null instead of false when page 2 lies past the end: there was no page 2 to check.
  • Playwright: --proxy-rotate per-page now rotates before page 2 (it used to start at page 3), and --delay now applies before page 2 (page 2 used to go out in the same millisecond page 1 was parsed). Selenium and pyppeteer already behaved this way.

Proxies and credentials

  • A proxy may be given as HOST:PORT:LOGIN:PASSWORD, with or without a scheme. That is the form proxy dashboards list exits in. As http://host:port:login:password it used to crash with a traceback quoting the password (exit 1). Without a scheme it was rejected with exit 2, but the message also quoted the password.
  • Every ProxyError message masks the value, and mask() can no longer raise. redact_secret_patterns also removes the colon form from free text.
  • python3 env_config.py printed a host:port:login:password proxy in full, and printed a CDP endpoint whose secret is a query parameter. It now shows only the proxy's scheme, host and port, and hides keys and endpoints.
  • Preflight drops an exit that fails it and tries the next one; exit 2 only when no exit works. A dead first line used to end the run while working exits sat behind it, so with --proxy-shuffle the outcome depended on which line came first. Dropped exits are removed from the pool, so neither rotation nor --concurrency workers can land on them.
  • When the target times out through an exit, preflight sends one request to a neutral address (https://www.gstatic.com/generate_204). If that also gets no answer, the exit is dead (exit 2). This replaces a warning that cost about 2 minutes of browser timeouts and ended in exit 5.

Data and metadata

  • finish_run records currencies_by_page and currency_mixed, and logs a warning when pages carry different currencies. On 2026-11-02 a /fr/ run came back EUR on page 1 and USD on pages 2–3. On 2026-11-09 the same URL was USD on every page. The locale message no longer claims prices were converted.
  • .meta.json records route: direct, proxy, proxy_pool, cdp or scraper_api.
  • Duplicates dropped across pages are logged as a warning instead of info.
  • A 429 response says it is a rate limit and what helps. The README now documents the limit: about 20 short runs in 5 minutes from one address on 2026-11-02, lifted after about 10 minutes.

diff_runs.py

  • Refuses to compare runs of different searches (exit 2; --force overrides). Two runs count as the same search when host (ignoring www.), path and query (ignoring page) match.
  • A sku whose currency changed between runs goes to "not comparable on price" and no longer counts as a price change, so it does not trip --fail-on-change.

CLI

  • An --out that cannot be written is exit 2 before any browser starts. It used to crash after every page had been fetched.

Tests and docs

  • Offline suite: the fingerprint-refusal check assumed ~/.cache/ms-playwright and failed on every clean Mac. It now reads the browser root from Playwright's own executable path.
  • README: the proxy figure "2 pages, 50 listings" was measured at 25 per page, before the 2026-09-29 rebuild, and now says so.

Evidence

  • Offline suite on Linux, Python 3.13: Playwright venv 563 passed, Selenium venv 530 passed, pyppeteer venv 529 passed, 0 failed in each. ci_checks.py --all, pytest -q and compileall pass.
  • New group check_audit_2026_11_09 runs each engine's scrape() on stubs: it checks which pages are fetched, the stop reason and status, rotations before pages 2 and 3, --delay before page 2, route, the diff refusals, the currency warning, --out, the proxy forms and env_config.py output.
  • Every guard was confirmed against its fault: 16 single-line mutations in a copy of the tree (plus one in Selenium) each turn a named check red without crashing the suite.
  • Live runs, Playwright, no proxy, 2026-11-09:
    • golf (34 matches), --pages 3: 1 request, 2 s, complete, listing_exhausted, addressable: null.
    • chess (4 matches): 2 s, down from 16 s.
    • saas --pages 2 --delay 3: 100 rows, a 3 s gap before page 2, currencies_by_page USD/USD.
    • --out into an unwritable directory: exit 2 before launch.
  • The proxy path was checked against a local CONNECT proxy with Basic auth, not a paid exit:
    • --proxy 127.0.0.1:PORT:LOGIN:PASSWORD: 50 rows, exit 0.
    • A pool of dead, wrong-password and good exits: the first two were dropped with warnings, then 100 rows, complete.
    • The password occurred 0 times in either log.

Not changed

  • A non-flippa.com --url still only warns. That is deliberate, to allow mirrors and locally served fixtures.
  • A 429 is still exit 3 without an automatic wait. The measured block lasted about 10 minutes, longer than a reasonable in-run retry.

🤖 Generated with Claude Code

…, flag mixed currencies

Fixes from a manual test of 2026-11-02/05, re-checked on b176941:

- No page past the end of the listing is requested, and a short page ends
  the 15 s readiness wait as soon as it holds what its own count implies
  (a 4-match search: 16 s -> 2 s). addressable is null, not false, when
  page 2 lies past the end.
- Playwright: per-page rotation starts at page 2, and --delay applies
  before page 2.
- Proxies: HOST:PORT:LOGIN:PASSWORD is accepted; a malformed value is
  exit 2 with credentials masked; mask() never raises; env_config.py no
  longer prints proxy credentials or a CDP endpoint's query token. A pool
  drops exits that fail the preflight instead of ending the run on the
  first one, and an exit that answers neither the target nor a neutral
  address is dead, not slow.
- Rows from pages in different currencies are flagged (log warning,
  currency_mixed, currencies_by_page); .meta.json records the route.
- diff_runs.py refuses different searches and treats a currency change as
  not comparable on price.
- An unwritable --out is exit 2 before a browser starts.
- 429 says it is a rate limit; duplicates across pages log a warning.
- Offline suite: the fingerprint check reads Playwright's browser root
  from its executable path (it failed on macOS).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr

jehrr commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

CI on 300cc79: audit, offline (3.12), docker and all three engine-smoke jobs pass. offline (3.9) fails before running anything:

##[error]The version '3.9' with architecture 'x64' was not found for Ubuntu 26.04.

ubuntu-latest now resolves to 26.04, and setup-python has no Python 3.9 build for it. The failure is not caused by this PR: main last ran tests on 2026-11-05, before the image change, and will hit the same error on its next run. The fix is one line in .github/workflows/tests.yml, which this branch could not push because the token used has no workflow scope:

   offline:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04

With that change locally, the offline suite passes (563 passed) and ci_checks.py --all passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant