Repository navigation
Add AutoScout24 site adapter - #1
Merged
Merged
Conversation
Browserless: replays the pasted search URL with a page cursor and reads the listings from the page's __NEXT_DATA__ JSON. Sponsored slots (Nfm) repeat an organic result on the same or a later page, so ids are deduplicated within the run — otherwise the first-run seeding fails on the UNIQUE constraint.
AutoScout24 renamed /lst/bmw/3-series to /lst/bmw/serie-3. The old path redirects to the new one keeping only price, country and sort, so a saved search silently widened to every fuel, body and year and notified on cars outside it. Compare the param names asked for with the ones the site landed on and raise if any went missing.
a69c1bc makes run_query fold listings by id before seeding, so the adapter no longer needs its own pass for the sponsored slots that repeat an organic result. SPEC already says adapters never dedup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I've been using subitoo to watch used-car listings, and many of them are on AutoScout24 rather than Subito, so I wrote an adapter for it. It's been running on my instance for about two weeks with six hourly queries, and it has been stable. The only failed runs were a handful of network timeouts, and each was retried within the same hour.
How it works
curl_cffi, addspage=Nto it, and reads the listings from the page's__NEXT_DATA__JSON.__NEXT_DATA__is a Next.js internal, not a stable API, so the parser fails loudly when the shape changes. An empty result would look to the core like every listing had disappeared.Things worth knowing
sort=age&desc=1to keep new listings on page 1. The README says so./lst/bmw/3-seriesto/lst/bmw/serie-3. The old path redirects to the new one but keeps only price, country and sort, so a saved search silently became much wider and started notifying on diesels and cars from any year. The adapter now compares the parameters it asked for with the ones on the page it landed on, and fails the run if any are missing (the fix is to paste the URL again).run_query, so I removed it to keep the adapter in line with the SPEC (adapters never dedup).Tests
tests/test_autoscout24.pyreplays a real search page saved as a fixture (trimmed to five cars, seller contacts and addresses removed), with no network access. It covers parsing, price-on-request listings, URL paging,max_pages, blocks (non-200) and the redirect check. The whole suite passes (107 tests).Happy to change anything to fit how you'd like adapters to look.