Meter every route, and sell a pass to whoever goes over - #215
Merged
Conversation
The limiter here watched three API prefixes and two page paths. That is the shape every site in the fleet had, and it is the shape that failed on coinpayportal on 2026-09-08: a headless browser found /explorer fifteen hours after it shipped and walked 19,000 of its URLs a day for two days. It declared nothing, so no crawler list matched it. It never touched a listed path, so no limiter saw it. The expensive routes were never the ones at risk -- the unlisted ones were. So the tuned numbers stay (30/min on /api/search/, /api/dht/ and /api/torrent-search, 60/min on /search and /dht) and everything NOT in that list is now metered too, at 100/min per caller, via @profullstack/throttle. Going over is answered 402 with the gate's own offer rather than 429, so a scraper that declares nothing is sold the same pass GPTBot buys. The good-bot and bad-bot tiers are untouched and still run first; this is the floor under them, not a replacement. A signed-in reader and an API bearer get the 600/min credentialed budget keyed on the credential itself -- the gate exempts both, because it is deciding whether to charge a crawler and a session is good evidence of a person, but this is deciding whether anyone at all reads 19,000 pages an hour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDGCxTmEPs3ecwjjLJDQXh
Keeps this on the same two versions as the rest of the rollout, and drops the second nested copy of the gateway: a caret range on a 0.x version only matches patches, so the throttle's old ^0.5.0 floor could not resolve 0.6.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDGCxTmEPs3ecwjjLJDQXh
The only conflict was pnpm-lock.yaml, where master's dependency bumps (next 16.3.4, @types/node 26.4.1) landed on the same importer lines that this branch added @profullstack/throttle to. Taking either side alone drops an entry, so both were kept and the lockfile re-resolved against the merged package.json. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ts1a6Lzj5DXySTCbeG8C
ThreatCrush Security Scan99 finding(s) HIGH/CRITICAL: 11 | MEDIUM: 29 | LOW: 59
…and 49 more. Full results in the Security tab. Snippets are redacted; ThreatCrush never prints matched credential material. |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The limiter here watched three API prefixes and two page paths. That is the shape every site in the fleet had, and it is the shape that failed on coinpayportal on 2026-09-08.
A headless browser found
/explorerfifteen hours after it shipped and walked 19,000 of its URLs a day for two days — 97% of all traffic to that site. It declared nothing, so no crawler list matched it. It never touched a listed path, so no limiter saw it.The expensive routes were never the ones at risk. The unlisted ones were.
What changes
The tuned numbers stay:
/api/search/,/api/dht/,/api/torrent-search/search,/dht/api/auth/Everything not in that list is now metered too, via
@profullstack/throttle. Going over is answered402with the gate's own x402 offer rather than429, so a scraper that declares nothing is sold the same pass GPTBot buys.What is untouched
denyCidrsOVH fleet block andchargeSpoofedBrowsersare unchanged.exempts both — it is deciding whether to charge a crawler, and a session is good evidence of a person. The throttle still meters them, because it is deciding whether anyone at all reads 19,000 pages an hour.Verified
vitest run— 2788 passed, 9 skipped, across 208 files. Three new tests: an unlisted route (/torrent/abc123) is metered at 100,/api/torrent-searchstays at 30, and callers get separate allowances.tsc --noEmitclean,next buildclean with the proxy intact.Note
The 402 only sells once
COINPAY_X402_KEYandCRAWL_PAY_TOare set — the gateway already reads them here. Without them the throttle falls back to429, which is the honest answer when there is nothing to buy.🤖 Generated with Claude Code