Rakuten scraper — an open-source (MIT) Python tool that reads Rakuten Ichiba (rakuten.co.jp) keyword searches, genre listings and item pages, and writes products, prices, Rakuten points, shops and review scores to JSON or CSV. It needs no key, no proxy and no account: measured on 2026-09-21 from a datacentre address, a 3-page keyword search returned 135 rows, complete, exit 0. Use it to build a Japanese e-commerce price dataset, or to compare two runs of the same search with the bundled diff_runs.py.
Scrapes Rakuten Ichiba — Japan's main domestic marketplace — to JSON or CSV: products, prices, Rakuten points, shipping, shops, review scores and per-variant pricing, with four interchangeable browser back ends and one row schema shared with the rest of the 2scraper family.
Two records cut from sample_output.json, a real run
of the keyword search below on 2026-09-21. 13 of the 42 columns are shown
(titles shortened); every column is listed under
What comes out.
[
{
"url": "https://item.rakuten.co.jp/ajinomoto/r4901111371248/",
"sku": "ajinomoto:r4901111371248",
"title": "【9/24まで P10倍!】味の素AGF ブレンディ スティックブラック100本…",
"price": 2447.0,
"currency": "JPY",
"points": 220,
"point_rate": 10,
"rating": 4.76,
"review_count": 475,
"shop_name": "味の素グループ公式ショップ",
"category": "水・ソフトドリンク > コーヒー > インスタントコーヒー",
"genre_path": "/0/100316/100356/303017",
"scraped_at": "2026-09-21T09:22:22.033307+00:00"
},
{
"url": "https://item.rakuten.co.jp/sawaicoffee-tea/ac-sale-1960/",
"sku": "sawaicoffee-tea:ac-sale-1960",
"title": "\先着300円OFFクーポン配布/ 送料無料 コーヒー コーヒー豆 コーヒー粉…",
"price": 2788.0,
"currency": "JPY",
"points": 50,
"point_rate": 2,
"rating": 4.7,
"review_count": 50175,
"shop_name": "澤井珈琲Beans&Leaf",
"category": "水・ソフトドリンク > コーヒー > コーヒー豆",
"genre_path": "/0/100316/100356/201402",
"scraped_at": "2026-09-21T09:22:22.033563+00:00"
}
]Full sample: sample_output.json · 9 rows, collected
2026-09-21 (listing mode, the keyword search from the Quick start).
price is in yen and points is what the purchase earns on top of it — see
the FAQ on Rakuten points.
genre_path is Rakuten's genre ids; category is the same path in
Rakuten's own (Japanese) genre names.
| Runtime | Python 3.9 or newer (the offline suite runs on 3.9 and 3.13 in CI) |
| Engine | Playwright with its Chromium (recommended); Selenium or pyppeteer as alternatives, one per virtualenv |
| Coverage | rakuten.co.jp (Rakuten Ichiba, Japan, prices in JPY): keyword searches, genre listings and item pages. Merchant storefronts and ranking.rakuten.co.jp are not read — see Coverage and limitations |
| Required services | Nothing. No key, no proxy, no account |
| Measured | 2026-11-17 at commit fe0d03d, from a datacentre address with no key, proxy or account (see Verification); first on 2026-09-21 |
| Optional: a proxy | For long runs. Rakuten answers a client going too fast with HTTP 503, which is a throttle, not a block; raise --delay first, then spread the run across exits. See 503 is a throttle, not a block |
| Optional: 2Captcha Scraping Browser API | Not needed on this site. What it adds: no local browser, a chosen exit country and persistent cookies per profile |
| Captcha solving | Not needed: no captcha is configured on the routes this scraper reads. See There is no captcha on this site |
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt -r requirements-playwright.txt
playwright install chromium
python3 playwright_scraper.py \
--url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 3That is the whole setup. No key, no proxy, no account, no .env.
It writes three files to the current directory (choose the prefix with
--out):
rakuten_products.jsonandrakuten_products.csv— one row per product, 45 per page;rakuten_products.meta.json— the run's status (completeor not), stop reason, failed pages and the site's own result cap.
If the run does not end with exit 0:
- exit 3, refused — remove anything you added to the client's identity (a
custom user agent, or
--fingerprinttogether with--cdp-endpoint); see The one thing to know first. - exit 4, zero products — the page was served but matched nothing; check the keyword and any filters in the URL. Nothing is written over earlier output.
- exit 5 or 6, throttled or partial — raise
--delay;pages_failedin the.meta.jsonnames the pages that were not read. See 503 is a throttle, not a block.
All exit codes: Exit codes. Setup, .env and exit codes for
the whole family: How to run the 2scraper scrapers.
What the scraper reads, what it does not, and what the site itself caps. The short version: listings and item pages, never storefronts or rankings; at most 6,750 results per query; no was-price on listing rows; merchant text stays in Japanese.
Three routes carry products and one looks as though it should:
| URL | Reads |
|---|---|
search.rakuten.co.jp/search/mall/{keyword}/ |
✅ --mode listing |
search.rakuten.co.jp/search/mall/-/{genreId}/ |
✅ --mode listing |
www.rakuten.co.jp/category/{genreId}/ |
✅ --mode listing |
item.rakuten.co.jp/{shop}/{code}/ |
✅ --mode product |
www.rakuten.co.jp/{shopCode}/ |
❌ refused, with the reason |
ranking.rakuten.co.jp/… |
❌ not implemented |
A merchant storefront is refused rather than attempted because its
__INITIAL_STATE__.state.data is empty — it carries no product payload at
all, so reading it would need a second parser written against markup nobody
has captured. A mode that ships untested is worse than a mode that is absent.
ranking.rakuten.co.jp is a different application: no payload, one JSON-LD
BreadcrumbList, 80 product links in hand-rolled markup. It is also the one
route on this site that cares about the window — 403 to plain HTTP and to
headless Chromium (3 of 3 each), 200 to headful Chromium (3 of 3). Not
unreachable, just not implemented.
Rakuten serves at most 6,750 results per query — 150 pages of 45 — however many it matched, and it states both numbers itself. One measured query reported:
numFound: 3,053,682 subset: 6,750 pageSize: 45
So a full 150-page run of that query is genuinely status: complete — it
fetched everything the site will serve — and it is a 0.2% sample. A
sidecar that said only "complete" would be lying by omission, so every run
records total_results, reachable_max, page_size, capped_by_site and
pages_beyond_cap beside the status, and the engine logs the percentage.
It also gives diff_runs.py a third meaning for removed: not delisted, not
un-fetched, but outside this run's slice of a capped result set.
To reach more, narrow the query with the site's own filters — genre, price band, shop, tag — and run each slice.
ratingandreview_countnull on some rows. Nobody has reviewed that product. Rakuten writes that as{score: 0, numReviews: 0}rather than as a null, on 28 of 405 rows, and a zero written through as a rating drags every average a consumer computes — so both columns are nulled together, keyed on the count.original_priceanddiscount_pctnull on every listing row. The listing payload publishes no was-price of any kind: zero occurrences ofdoublePrice,listPrice,referencePriceorstrikeacross 405 rows. The nearest thing it has issale, on 32 of 405, which is a campaign window with no prices in it — read as a discount it would invent one. A verified was-price does exist on the detail page, gated on Rakuten's owndoublePrice.referencePriceVerifiedflag, which is the outcome of Japan's double-pricing substantiation rule. Verified on 2 of 8 item pages (¥10,398 → ¥5,199, and ¥3,476 → ¥1,738).in_stocktrue on every row. The search excludes sold-out products rather than marking them — checked specifically on deep pages (80, 120, 149, 150 of a 142,000-hit genre), where stale rows would live: 0 sold out. So afalseon the listing route is unproven; the detail route reads the page's ownavailabilitymicrodata, which is the source that can say otherwise.- 45 rows from a payload with 52 entries. Seven were sponsored
placements. Rakuten injects paid slots into its own result list; they carry
a click-tracking redirect instead of a product URL, so there is nothing to
key a row on. The count varies between fetches of the same URL — two
engines seconds apart saw 45 and 52 — which is why
positioncounts the rows emitted rather than the payload's slots. --locale endoes not translate anything you care about.?lang=encomes back withlocale: "en"in the payload and an English navigation bar, and product names, shop names and tag values are merchant-authored Japanese and come back byte-identical.?lang=zh-twand?lang=koare accepted with HTTP 200 and silently fall back toja.genre_rankis not a position in your results. It is the item's standing in its genre's ranking listing, which this run never fetched. Sibling repos in this family use published ranks to prove that cards went missing; that arithmetic does not work here.- A readiness wait that times out costs nothing. Every column comes out of the payload, which is in the first response with no JavaScript run at all — the same URL fetched by plain HTTP and by a real browser parses to the same 45 rows with the same skus and prices.
- Never wait on
networkidle. Rakuten's ad and tracking beacons keep firing indefinitely — 46 fetches and 13 pings still going after load — so that wait always runs to its full timeout. All three engines wait fordomcontentloaded.
More in TROUBLESHOOTING.md.
Step by step: How to scrape Rakuten · the Rakuten scraper page on 2scraper.com.
Each command is a complete run once the Quick start setup is done.
# A keyword search, three pages, JSON and CSV
python3 playwright_scraper.py \
--url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 3
# A genre listing, by Rakuten's own genre id
python3 playwright_scraper.py \
--url "https://www.rakuten.co.jp/category/100356/" --pages 5 --format csv
# One product: per-variant prices and a variant count (and a was-price while
# the shop's own sale is verified: see the note under "Measured")
python3 playwright_scraper.py --mode product \
--url "https://item.rakuten.co.jp/sawaicoffee-tea/solandluna/"
# The site's own filters are part of the URL and survive pagination
python3 playwright_scraper.py \
--url "https://search.rakuten.co.jp/search/mall/コーヒー/?min=1000&max=3000" \
--pages 3
# Spread a long run across exits (and see the note on 503 below). Put one
# proxy URL per line in the list; exits*.txt, proxies*.txt and proxylist*.txt
# are git-ignored, because that file holds logins and passwords.
python3 playwright_scraper.py \
--url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 50 \
--proxy-file exits.txt --proxy-rotate per-page --delay 4Last checked on 2026-11-17 at commit fe0d03d. The scraper's code is
unchanged since ab9babd, where the full three-page Quick start was run on
2026-11-09.
| Date, commit | Check | Conditions | Result |
|---|---|---|---|
2026-11-17, fe0d03d |
Quick start command with --pages 1 (one page of 45; kept short on purpose) |
a fresh virtualenv, Python 3.13, requirements.txt + requirements-playwright.txt (Playwright 1.65.2), headless Chromium, no .env, one datacentre address |
45 rows, 45 unique sku, status: complete, exit 0; price and currency (JPY) on 45 of 45; same 42 columns in the same order as sample_output.json |
2026-11-17, fe0d03d |
Site cap reported | the same run | total_results 3,233,987, reachable_max 6,750 — a complete run is a sample, as described above |
2026-11-09, ab9babd |
Quick start, as written above (--pages 3) |
a fresh virtualenv, Python 3.13, requirements.txt + requirements-playwright.txt, headless Chromium, no .env, one Hetzner datacentre address |
135 rows, 135 unique sku, status: complete, exit 0; price and currency (JPY) on 135 of 135; total_results 3,164,855 |
2026-11-17, fe0d03d |
Offline suite | python3 smoke_test.py and python3 .github/ci_checks.py --all, core requirements only |
passed (engine groups skipped, as designed, without their libraries) |
| 2026-10-01 to 2026-11-16 | Canary (daily, weekly since 2026-10-16) | a 3-page genre listing from a GitHub runner with no secrets | green on every run, the last one on 2026-11-16 at fe0d03d |
You do not need any of the paid products to read this site's listings, and the reason is not what you would guess.
Rakuten is fronted by Akamai, and what Akamai refuses here is not a datacentre address — it is a client whose claimed identity and TLS fingerprint disagree. Measured 2026-09-21 from one Hetzner datacentre address in Helsinki, nothing else changed between the rows:
| Client | Result |
|---|---|
curl, curl's own User-Agent |
200, 919 KB, 45 products |
curl, a Chrome User-Agent |
200, 42-43 bytes, Reference #… |
| headless Chromium (real UA, real TLS) | 200, 855 KB, 45 products |
| headful Chromium | 200, 969 KB, 45 products |
The two curl rows were repeated on 2026-10-29 from a Hetzner datacentre
address and gave the same answer (919,197 bytes and 45 parsed products; 42
bytes, the length varying with the reference id), which also replaced the
size this table first carried for the first row. The two Chromium rows are
from 2026-09-21 and were not repeated.
Read it by column: the address is identical in all four and the consistency of the client decides everything. curl claiming to be curl is served; curl claiming to be Chrome is refused. So the practical advice is the opposite of most sites in this family — if you are being refused, look for a disguise you added before you buy an exit:
- do not pass
--fingerprintor a custom user agent together with--cdp-endpoint, because the remote browser already brings its own identity and stacking a second one manufactures exactly this mismatch; - do not put a browser user agent on an HTTP client.
Two more things follow from that table, and both are easy to get wrong:
- A refusal arrives under HTTP 200. The status code is not the signal.
This scraper decides "was this served at all" from the reference-id shape
and from the page being built out of Rakuten's own
r10s.jpasset hosts (90–343 references on every served page, 0–1 on every refusal). - Headless is fine. It is the default here. The browser even announces
itself — a tracking beacon from the headless run carried
HeadlessChrome/153.0.8010.12— and Akamai served it anyway.
The canary in this repository is that claim under test: a real three-page scrape, weekly (Mondays), from a bare GitHub runner, with no secrets. If Rakuten ever puts the listing routes behind something, the badge above goes red on the next weekly run: a blocked, partial or empty run fails the job rather than passing with a warning.
One engine, not all three. They declare mutually unsatisfiable pins —
playwright and pyppeteer disagree on pyee, pyppeteer and selenium on
urllib3 — so use a virtualenv per engine if you need more than one.
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt -r requirements-playwright.txt
playwright install chromium # the recommended enginepip install -r requirements.txt -r requirements-selenium.txt # or
pip install -r requirements.txt -r requirements-puppeteer.txt # orpip install .[playwright] is equivalent.
pyppeteer downloads a Chromium of its own, and that build does not start on
every machine (seen on an Apple Silicon Mac, where it is an x86_64 build and
ends with Browser closed unexpectedly). Pass --chromium-path with a Chrome
or Chromium that you installed, e.g. --chromium-path "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome". Without it the run now ends with
that advice and exit 5 rather than a traceback; the Apple Silicon case itself
was reported by an audit and was not reproduced here (Linux, a browser that
exits at once, gave the same error and the same handling).
42 columns. The first 18 are the family prefix, byte-identical and in the same order as every other repo in this family, so one consumer reads a rakuten run and a mediamarkt run with the same code.
{
"source": "rakuten.co.jp",
"scraped_at": "2026-09-21T09:22:22+00:00",
"url": "https://item.rakuten.co.jp/ajinomoto/r4901111371248/",
"sku": "ajinomoto:r4901111371248",
"title": "【9/24まで P10倍!】味の素AGF ブレンディ スティックブラック100本…",
"brand": "味の素AGF",
"price": 2447.0,
"currency": "JPY",
"original_price": null,
"discount_pct": null,
"rating": 4.76,
"review_count": 475,
"in_stock": true,
"image_url": "https://thumbnail.image.rakuten.co.jp/@0_mall/ajinomoto/…",
"category": "水・ソフトドリンク > コーヒー > インスタントコーヒー",
"price_source": "state",
"page": 1,
"position": 1,
"variant_id": "r4901111371248",
"item_number": "10000338",
"price_max": 6975.0,
"has_price_range": true,
"subscription_price": 2324.0,
"points": 220,
"point_rate": 10,
"shipping_fee": 0.0,
"free_shipping": true,
"delivery_estimate": "最短9/23お届け",
"shop_name": "味の素グループ公式ショップ",
"shop_code": "ajinomoto",
"shop_id": 383220,
"shop_url": "https://www.rakuten.co.jp/ajinomoto/",
"genre_id": 303017,
"genre_path": "/0/100316/100356/303017",
"genre_rank": 2,
"is_super_deal": false,
"is_39shop": true,
"is_official_shop": true,
"variant_count": null,
"original_price_label": null,
"tags": {"ブランド": ["味の素AGF"], "単品重量": ["~ 99g"]},
"variants": null
}sample_output.json and sample_output.csv are cut from a real run, not
written by hand.
Three columns are worth explaining because this site is unusual:
skuis{shopCode}:{manageNumber}— Rakuten's own item code, exactly as a detail page states it in<meta itemprop="sku">. It is the same id on both routes, so you can join a listing run to a product run on it. Deliberately not the payload'svariantId, which names the pre-selected SKU inside the item and matches the URL's own code on only 39 of 180 measured rows.pointsis half the price story. Rakuten is a points marketplace: a 10× point campaign is effectively a 10% discount that never touches the price column, so a monitor watchingpricealone would call a product unchanged through an entire Super Sale.diff_runs.pytracks both.shop_*is not run metadata. This is a mall — 74 distinct merchants across 405 measured rows — so the same physical article is sold by many shops at many prices under many ids.
| Code | Means |
|---|---|
| 0 | products written |
| 1 | crash |
| 2 | bad usage |
| 3 | blocked before parsing |
| 4 | served a page, zero products |
| 5 | the content was never obtained: a remote API error, a dead exit, a load that never completed, a page that stayed throttled after every wait |
| 6 | partial run |
A run that finds nothing writes nothing — --allow-empty is the opt-out
— so a failed run cannot replace last night's good output with []. With
the opt-out the empty file IS written, but the exit code does not change: a
run that found nothing still exits 4 and its sidecar says status: failed.
--out may name a directory that does not exist yet: it is created before
the first request, and one that cannot be created is a usage error (exit 2)
instead of a traceback after the whole run.
A page that stays HTTP 503 after every wait is a failed page
(stop_reason: throttled, pages_failed names it): exit 6 if other pages
produced rows, 5 if none did. It is never read as a page with no products.
All three files are replaced atomically. Each is written to a temporary
file beside it and renamed over the old one, so a crash, a kill or a full
disk leaves the previous run's output exactly as it was rather than a
half-written file where a complete one had been. Measured before the fix: a
2,084-byte good out.json came back 0 bytes and invalid JSON after an
interrupted write, because open(path, "w") truncates before writing a
single byte. That matters most for the sidecar, which is the file a consumer
branches on.
Every run also writes <out>.meta.json, and on this site it carries the
site's own arithmetic as well as the status. See
complete is not exhaustive.
A CSV cell cannot be a formula. A spreadsheet executes a cell beginning
=, +, - or @, and every string here was written by a merchant, not by
this code. Such a cell is prefixed with an apostrophe in the CSV only —
the JSON keeps the site's bytes exactly as served — and csv_cells_escaped
in the sidecar declares how many cells the two files therefore differ in.
Only strings are touched: escaping a number would turn -5 into text and
break every sum written over the column.
This does not fire on today's data and is not claimed to. Measured
2026-09-30: 0 of 42,592 string cells across 2,662 stored rows begin with
any of those characters. It is here because a Rakuten title routinely opens
with a discount badge, and -16%OFF ... is one merchant away from being a
leading minus.
The JSON-LD on a listing page is a trap. A search page has exactly one
application/ld+json block, it is an ItemList, and it holds ten items
whose every url carries ?scid=seo-carousel-search — it is the SEO
recommendation carousel — while the page itself holds 45 products. A
JSON-LD-primary parser returns ten rows of the wrong products and reports
success.
The real sources:
| Route | Source |
|---|---|
| listing | window.__INITIAL_STATE__.state.data.ichibaSearch — items (45/page) + pagination |
| product | <script type="application/json" id="item-page-app-data"> → newApi.itemInfoSku |
The listing JSON-LD is read for exactly one thing: the currency. The
listing payload states none at all — zero occurrences of priceCurrency,
currency or the string JPY across 405 rows — and Rakuten publishes the
block that does state it on page 1 only (1 block on page 1, 0 on pages 2
and 3 of the same query). So the engines read it once from page 1 and carry
it forward, and a run that never got page 1 leaves currency null rather
than defaulting.
A detail page publishes a different structure again, and its only JSON-LD
is a BreadcrumbList — so the listing parser finds nothing on it, which the
test suite asserts directly.
Detail pages are EUC-JP. Content-Type: text/html;charset=EUC-JP on 8 of
8, while every listing page is UTF-8. A browser decodes it for you, so the
three engines never meet this; an HTTP client does, and those bytes decoded
as UTF-8 raise on the first Japanese character — or, with
errors="replace", turn every title on the page into replacement characters
while the numbers still parse.
?p=N, 45 products a page, and two routes that lie past the end.
Rakuten does not error past a listing's last page. ?p=151 of a 150-page
query answers 301 to page 1 (on 2026-10-29 the target was the same query
with ?pref=tokyo added, and its payload said start: 0 with the same
numFound) and then serves it with HTTP 200 and 45 real products; and www.rakuten.co.jp/category/{genreId}/?p=2 answers 200 with
page 1 without even redirecting. Both look like success from the client
side, and a run that trusted its own request would re-collect page 1 for as
long as it was asked to and report a complete, entirely duplicate file.
What settles it is that the payload states the offset the server actually
used. Asking for page 151 and being handed pagination.start: 0 is Rakuten
saying outright that it served page 1, so every fetched page is checked
against it and a mismatch ends the listing (stop_reason: end_of_listing,
which counts as a complete run).
The site also publishes no <link rel="next"> anywhere — only
rel="canonical", on 3 of 3 captures — so the durable layer the family's
pagination rules lead with simply does not exist here. The ?p= convention
is verified against the site's own numbered anchors instead, including the
cross-host case: a genre landing page advertises
search.rakuten.co.jp/search/mall/-/{genreId}/?p=2, which is exactly what
this scraper builds for it.
The site's own result cap, and what a complete run therefore means, is
described under Complete is not exhaustive.
| Engine | Status |
|---|---|
playwright_scraper.py |
primary, recommended |
selenium_scraper.py |
parity |
puppeteer_scraper.py |
parity (pyppeteer is effectively unmaintained) |
scraper_api_client.py |
HTTP-only, billed per page (the x-debug header of each response reports the price) |
All three browser engines take the same flags, produce the same columns in
the same order, and map the same exit codes. They are not guaranteed to
return the same rows. On 2026-09-21 one genre listing, two pages, gave 90
rows each with identical skus in identical order and every column
identical on all three. On 2026-10-29 one keyword listing with a price
filter (?min=1000&max=3000), two pages, did not: Playwright and Selenium
each returned 90 rows, 44 of the 45 skus on each page were common to both,
and every column of the common rows matched — but one slot per page held a
different product, the same one in each of two repeats, while Playwright
against itself differed in nothing. Rakuten injects paid placements into the
result list (see "Sponsored placements"), and the two clients were shown
different ones; that is a likely reading and it is not established. pyppeteer
was not repeated on 2026-10-29. Do not rely on two engines writing
byte-identical files.
Known limits, stated here rather than left to be discovered:
- Selenium cannot use an authenticated remote CDP endpoint. Playwright's
connect_over_cdpand pyppeteer'sbrowserWSEndpointtake a fullws://user:pass@host:port; chromedriver'sdebuggerAddresstakes a barehost:portwith nowhere to put a password. A credentialled--cdp-endpointis refused there with that explanation rather than silently dropping the credentials. - Selenium's
--proxy-servercannot authenticate at all. Credentials are stripped and a warning printed. A proxy that wants them answers HTTP 407, and Chromium does not raise for that: the page is blank. The engine reads the navigation's own status (responseStatus, measured 2026-10-29 on Chrome 154: 407 for that proxy, 200 for a served page), reports a failed load naming the 407, and rotates if the pool has another exit. It is not reported as a block by the site, which is what it used to say. - The pyppeteer engine deliberately sets no user agent, where the other two set one. That is not a parity gap to close — it is required for that engine to work on this site. See the next section.
Setting a user agent through pyppeteer's CDP override gets that engine refused from its second navigation onward. Measured over four consecutive navigations per variant, same browser, same address:
| pyppeteer variant | Result |
|---|---|
| no override | 4 of 4 served |
| a Windows UA override | nav 1 served, navs 2–4 denied |
| a Linux UA override, matching the real platform | nav 1 served, navs 2–4 denied |
The third row is what settles the cause: a UA that agreed with the platform
was refused just as hard. It is not Windows-on-Linux, it is the override —
most likely because the CDP override changes navigator.userAgent and leaves
the User-Agent Client Hints reporting the real browser, so the page claims
two clients at once. (That explanation is an inference and it is not
sufficient on its own: selenium_scraper.py issues the same
Network.setUserAgentOverride and was served 3 of 3 pages.)
The transferable part is the SHAPE. A fingerprint problem on this site looks
like success on page 1. --help worked, compileall passed, the offline
suite was green, and only a live two-page run found it. If you add a user
agent or a fingerprint to any engine here, test it over two pages.
For listings and product pages: no. Measured 2026-09-21 with no key, no proxy and no account, from a datacentre address:
- a keyword search returned 45 products a page;
- a three-page run returned 135 rows, 100% with a price,
status: complete, exit 0; - the same on the genre route, and with a headful browser.
Each was run against this site on 2026-09-21. The point of the table is that none of them is needed — and that each does something specific when you do want it.
| Path | Result | What it buys |
|---|---|---|
Scraping Browser API (--cdp-endpoint, US exit) |
90 rows over 2 pages, complete |
no browser infrastructure, a chosen exit country, persistent cookies per profile |
Proxy (--proxy, EU residential exit) |
90 rows over 2 pages, complete |
volume — spreading a long run so you stop meeting the 503 |
Fingerprint (--fingerprint) |
applied and served; 90 rows over 2 pages | a consistent device identity |
| Scraper API | HTTP 200, 1.09 MB, 45 products; priced per page (0.0005 in that 2026-09-21 run, 0.0003 reported by a 2026-10-29 audit run: it is the vendor's price, so read it from x-debug or the dashboard) |
no browser at all, one request a page |
| Captcha solving | nothing to solve — see below | — |
Four notes, because each cost a measurement:
- A fingerprint is NOT a hazard here, but half of one is. This README said otherwise before the path had been run, and the measurement is narrower than the warning was. A fingerprint applied completely — user agent, client hints, timezone, languages and platform together — is served. The run without it hit the same 503 on page 1 that the run with it did, so that throttle was request volume rather than the fingerprint. What Akamai refuses is a client that contradicts itself, which is why a bare user-agent override is refused and a complete identity is not.
--concurrencyis refused with--cdp-endpoint, and so is a--proxyor--proxy-filetyped on the command line (exit 2, with the reason). ARAKUTEN_PROXYthat only arrives through.envis ignored with a warning instead, so one.envcan serve both paths.--fingerprintis ignored there. A Scraping Browser API profile allows one live connection, it already has an exit, and it already has an identity. Each refusal prints its reason rather than quietly doing nothing.- Proxies buy volume, which is the thing actually worth buying here. Rakuten answers a client going too fast with HTTP 503 and its own アクセスが集中しております page — met three times during testing, always cleared by the wait. A Japanese exit is also what to try if a consistent client is somehow still refused.
- Credentials never reach a log. A proxied run prints
http://***:***@eu.proxy.2captcha.com:2334— host and port kept, because which exit a run used is the useful half and is not the secret.
Not "we did not meet one" — none is configured. Across 13 captures,
served and refused alike: zero reCAPTCHA, hCaptcha, Turnstile, DataDome,
PerimeterX, Incapsula, Kasada or AWS WAF markers; no challenge iframe; no
data-sitekey; no *_SITE_KEY in any page config; no <captcha-*> mount
point. And Akamai's own refusal here is 43 bytes with no widget on it —
which is the narrow, honest use of the word unsolvable: a property of that
page, not of any vendor.
The solver is still wired up and bounded to one solve per page, because a bot manager can be switched on between deploys and a scraper that cannot name what stopped it is much harder to fix. This repo implements reCAPTCHA v2/v3; if Rakuten ever renders an enterprise reCAPTCHA or a Turnstile, 2Captcha solves both and the work would be to add the task type here.
Rakuten serves the identical branded page under 403 (a route it refuses outright) and under 503 (a client going too fast). Only the status separates them, so:
- a 503 costs a wait at the same exit — 6s, then 15s, then 30s — and does not count towards exit 3;
- a 403 is a refusal of that route.
Measured: a URL that had just 503'd twice under no-delay hammering came back
200 with a full payload on 4 of 4 tries once 6 seconds were put between
requests. Raise --delay before you reach for a proxy.
Note that selenium_scraper.py cannot give you a status, so it reads a
statusless branded page as the recoverable one — calling a throttle a block
spends the block budget on a page that was about to come back, while calling
a block a throttle costs one wait.
# credentials in .env, never on a command line. Put the value ON the
# RAKUTEN_PROXY line of the .env you copied from .env.example (a second line
# further down works too: the last one is used, and a warning names the key).
# Replace {user} and {password} with the dashboard's login and password.
RAKUTEN_PROXY=http://{user}-zone-custom-region-jp:{password}@ap.proxy.2captcha.com:2334
python3 playwright_scraper.py \
--url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 20 \
--proxy-rotate per-page --delay 4The host is a gateway (ap, na or eu), not a country: the country
is the -region-jp segment of the login that the dashboard hands out.
Checked 2026-10-29 with a credential-free CONNECT to port 2334: ap, na
and eu answer 407 Proxy Authentication Required, i.e. a real gateway;
jp, as and ae resolve to the same Cloudflare addresses as the 2captcha
website and the connection times out. An earlier version of this example
used jp.proxy.2captcha.com, which cannot work.
A rotation is a fresh browser: cookies issued against one exit and replayed from another are a stronger signal than either address alone, so the engine tears the browser down and relaunches rather than swapping the proxy under a live session.
Everything in this section is a figure from a real run, with the date. Where a column is empty, it says whether that is the site or the parser.
2026-09-21, one Hetzner datacentre exit in Helsinki, no credentials:
| Run | Result |
|---|---|
| keyword search, 3 pages | 135 rows, 100% priced, complete, exit 0 |
| genre listing, 2 pages | 90 rows, identical across all three engines (a keyword listing on 2026-10-29 differed in one slot a page: see the engines section) |
| one item page | 1 row, 4 variants, a verified 50% was-price. On 2026-10-29 the same item still had 4 variants and no was-price: its page said referencePriceVerified: false, the sale having ended, and the parser correctly emitted none |
| 11 listing pages (405 rows) | price 405/405, points 405/405, shop_code 405/405 |
Field coverage across those 405 rows:
| Column | Populated | Note |
|---|---|---|
price |
405/405 | every listing names a price |
points |
405/405 | 27 of them are 0, which is a real zero |
rating / review_count |
377/405 | the other 28 are unreviewed — see Traps |
brand |
240/405 (59%) | from Rakuten's own ブランド tag |
shipping_fee == 0 |
317/405 (78%) | free shipping is the common case |
price_max |
95/405 | items whose variants differ in price |
subscription_price |
54/405 | 定期購入, always below price |
genre_rank |
20/405 | the item's rank in its genre ranking |
original_price |
0/405 | the listing payload publishes none — see Traps |
in_stock == true |
405/405 | the search excludes sold-out items |
# the two files are the outputs of your own earlier runs, e.g. made with
# `--out monday` and `--out tuesday`; both must be complete runs of one mode
python3 diff_runs.py --old monday.json --new tuesday.json --fail-on-changeJoins on sku and reports added / removed / changed. Two things it does that
matter on this site:
- It tracks
pointsandpoint_rate, because a point campaign is a discount that never touches the price column. - It refuses to compare runs it cannot compare — a partial run against a
complete one, or two different modes — rather than producing a diff whose
every line is an artefact. A price change that comes with a
price_sourcechange is reported assource_changed, notchanged, and--fail-on-changeignores it: it says something about our own two snapshots, not about the shop.
python3 smoke_test.py # the offline suite: no network, no engine needed
pip install .[dev] # pytest is only in this extra, not in requirements*.txt
pytest # the same suite, as one test
python3 .github/ci_checks.py --allThe suite passes with no engine library installed at all, and reports which groups it skipped — a suite that silently skips part of itself and still says "all passed" is the same defect as code that reports success without checking. CI installs each engine in its own virtualenv and fails if any group reports a skip.
Fixtures are cut from real captures by make_fixtures.py, which verifies
that each trim parses identically to the untrimmed original — every
column, not a sample — and scrubs credential-shaped and personal material
first: two 32-char front-end API keys baked into every item page, a real
customer's review nickname and text, a per-impression session id on each
sponsored slot, and a 32-hex catalogue id that is public and harmless and
reads as a credential to every scanner including this repo's own.
Every check added while building this repo was controlled — the code was broken deliberately, the suite confirmed red and the expected check confirmed to be the one that named it — because a control that silently did not apply prints a green suite and proves nothing.
- How to scrape Rakuten: step-by-step guide
- Rakuten scraper on 2scraper.com — what it collects, requirements and related scrapers on one page.
- How to run the 2scraper scrapers: setup, proxies, exit codes and access modes — installation,
.env,.meta.jsonfor the whole family.
This repository reads the public pages; it does not use or require Rakuten's developer API. It reads the same JSON payload the search and item pages are built from, so its rows carry what a shopper sees on those pages, Rakuten points and shop names included, with sponsored placements left out.
No. On 2026-11-09 the Quick start command returned 135 rows, complete, exit 0,
from a datacentre address with no key, no proxy and no account (see
Verification). Rakuten's HTTP 503 is a throttle for a client
going too fast, not a block: the scraper waits and retries at the same exit.
A proxy helps only for long runs, to spread the traffic across exits; raise
--delay first.
On the routes this scraper reads, it does not. Headless Chromium from a datacentre address is how the Quick start was run (see Verification). What Rakuten's Akamai front refuses is a client whose claimed identity disagrees with its TLS fingerprint, such as an HTTP client sending a Chrome user agent, and that refusal can arrive as a short HTTP 200 page rather than an error status — see The one thing to know first. A 503 is a throttle for going too fast: the scraper waits and retries.
Exactly as in the Quick start: headless is the default, so no
display is needed, and no key, proxy or .env is required. The checks in
Verification ran that way on a Linux datacentre server, and
the canary runs it weekly on a GitHub runner with no secrets. For a scheduled
job, branch on the exit code and the status in <out>.meta.json rather
than on whether a file exists.
42 columns per product, in the same order in JSON and CSV, all listed under
What comes out: the family's shared 18 (url, sku,
title, price, currency, rating, review_count, in_stock,
category, …) followed by Rakuten's own — points and point_rate,
shipping, the shop, genre ids and per-variant prices.
Yes. Put the Japanese keyword straight into the search URL, as in the Quick
start (/search/mall/コーヒー/). title, brand, shop_name, category
and tags come back in Japanese, as the merchants wrote them; genre_path is
numeric genre ids. --locale en (?lang=en) only translates the site's
navigation: product and shop names stay byte-identical.
points is the number of Rakuten points a purchase earns, and point_rate
is the item's point multiplier when the listing states one (null when it
does not). A 10× point campaign is effectively a 10% discount that never
touches the price column — in the example above, 220 points on a ¥2,447
item at point_rate 10. A monitor watching price alone misses it, which is
why diff_runs.py compares points and point_rate as well.
This is not legal advice. The scraper reads public pages only: it does not log in, touch a cart or attempt to defeat a challenge. Whether a particular use is allowed depends on your jurisdiction, Rakuten's terms of use, copyright in product texts and images, and data-protection law, which differ between countries. Keep the request rate low — the scraper waits between pages and backs off on a 503 — and do not collect personal data without a lawful basis.
A canary runs a real three-page scrape every Monday from a GitHub runner with no secrets, and a blocked, partial or empty run turns the canary badge at the top of this page red. Every canary run from 2026-10-01 to 2026-11-16 was green; four of the first ten, in late September 2026, were not. Changes made because the site changed are recorded in CHANGELOG.md.
Same CLI conventions, same output contract (one row per item, JSON or CSV, a .meta.json sidecar and shared exit codes):
- SHEIN scraper — us.shein.com search, category and product pages…
- Sleep Number scraper — Smart beds, mattresses and bedding from sleepnumber.com
- StockX scraper — Search results, category listings and product pages
- TikTok Shop scraper — Product pages to 39 columns
- Tokopedia scraper — IDR prices, discounts, ratings, sold counts and shops…
- Tractor Supply scraper — Categories, departments and searches
All scrapers: github.com/2scraper · Need another site? Open a request
CONTRIBUTING.md · SECURITY.md · TROUBLESHOOTING.md · CHANGELOG.md · MIT
This repo reads public pages. It respects the site's own pagination limits, defaults to a delay between pages, and treats a rate-limit response as a signal to wait rather than to push harder. It does not log in, does not touch a cart or a checkout, and does not attempt to defeat a challenge it was not given.