Repository navigation
Conversation
…, flag mixed currencies Fixes from a manual test of 2026-11-02/05, re-checked on b176941: - No page past the end of the listing is requested, and a short page ends the 15 s readiness wait as soon as it holds what its own count implies (a 4-match search: 16 s -> 2 s). addressable is null, not false, when page 2 lies past the end. - Playwright: per-page rotation starts at page 2, and --delay applies before page 2. - Proxies: HOST:PORT:LOGIN:PASSWORD is accepted; a malformed value is exit 2 with credentials masked; mask() never raises; env_config.py no longer prints proxy credentials or a CDP endpoint's query token. A pool drops exits that fail the preflight instead of ending the run on the first one, and an exit that answers neither the target nor a neutral address is dead, not slow. - Rows from pages in different currencies are flagged (log warning, currency_mixed, currencies_by_page); .meta.json records the route. - diff_runs.py refuses different searches and treats a currency change as not comparable on price. - An unwritable --out is exit 2 before a browser starts. - 429 says it is a rate limit; duplicates across pages log a warning. - Offline suite: the fingerprint check reads Playwright's browser root from its executable path (it failed on macOS). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
CI on 300cc79:
offline:
- runs-on: ubuntu-latest
+ runs-on: ubuntu-24.04With that change locally, the offline suite passes (563 passed) and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from a manual test of 2026-11-02 and 2026-11-05, re-checked against
main@ b176941 on 2026-11-09.Behaviour changes
Pagination and waits
listing_exhausted. This applies to all three browser engines and to Scraper API mode, where each extra page would be a billed task. A 34-match search is now one request.addressableisnullinstead offalsewhen page 2 lies past the end: there was no page 2 to check.--proxy-rotate per-pagenow rotates before page 2 (it used to start at page 3), and--delaynow applies before page 2 (page 2 used to go out in the same millisecond page 1 was parsed). Selenium and pyppeteer already behaved this way.Proxies and credentials
HOST:PORT:LOGIN:PASSWORD, with or without a scheme. That is the form proxy dashboards list exits in. Ashttp://host:port:login:passwordit used to crash with a traceback quoting the password (exit 1). Without a scheme it was rejected with exit 2, but the message also quoted the password.ProxyErrormessage masks the value, andmask()can no longer raise.redact_secret_patternsalso removes the colon form from free text.python3 env_config.pyprinted ahost:port:login:passwordproxy in full, and printed a CDP endpoint whose secret is a query parameter. It now shows only the proxy's scheme, host and port, and hides keys and endpoints.--proxy-shufflethe outcome depended on which line came first. Dropped exits are removed from the pool, so neither rotation nor--concurrencyworkers can land on them.https://www.gstatic.com/generate_204). If that also gets no answer, the exit is dead (exit 2). This replaces a warning that cost about 2 minutes of browser timeouts and ended in exit 5.Data and metadata
finish_runrecordscurrencies_by_pageandcurrency_mixed, and logs a warning when pages carry different currencies. On 2026-11-02 a/fr/run came back EUR on page 1 and USD on pages 2–3. On 2026-11-09 the same URL was USD on every page. The locale message no longer claims prices were converted..meta.jsonrecordsroute:direct,proxy,proxy_pool,cdporscraper_api.diff_runs.py--forceoverrides). Two runs count as the same search when host (ignoringwww.), path and query (ignoringpage) match.--fail-on-change.CLI
--outthat cannot be written is exit 2 before any browser starts. It used to crash after every page had been fetched.Tests and docs
~/.cache/ms-playwrightand failed on every clean Mac. It now reads the browser root from Playwright's own executable path.Evidence
ci_checks.py --all,pytest -qandcompileallpass.check_audit_2026_11_09runs each engine'sscrape()on stubs: it checks which pages are fetched, the stop reason and status, rotations before pages 2 and 3,--delaybefore page 2,route, the diff refusals, the currency warning,--out, the proxy forms andenv_config.pyoutput.golf(34 matches),--pages 3: 1 request, 2 s,complete,listing_exhausted,addressable: null.chess(4 matches): 2 s, down from 16 s.saas --pages 2 --delay 3: 100 rows, a 3 s gap before page 2,currencies_by_pageUSD/USD.--outinto an unwritable directory: exit 2 before launch.--proxy 127.0.0.1:PORT:LOGIN:PASSWORD: 50 rows, exit 0.complete.Not changed
--urlstill only warns. That is deliberate, to allow mirrors and locally served fixtures.🤖 Generated with Claude Code