A house crawler gets its own rung on the two profile routes - #195
Merged
Conversation
nichedb.dev pulls every author's OpenProfile.md into its people collection and names itself in its user agent. On the free rung (120 a minute, 600 an hour) a backfill of two hundred thousand authors takes days. A request that names a house crawler AND asks for /api/openprofiles or an author's openprofile.md is placed on a `house` rung: 600 a minute, 36,000 an hour. Every other route and every other caller is unchanged, and a session or a key still places the caller by what it carries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S7yeJUHGxA4P5N74xnsRPQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
nichedb's
openprofilesadapter (UAniche-db/0.1 (+https://nichedb.dev)) was throttled after ~100 documents a minute (57 of 1,443 refused in one run), which stretched the 194k backfill to days.apps/web/src/lib/tiers.js:TIERS.house(burst 600/min viaTIER_HOUSE_BURST, 36,000/hour viaTIER_HOUSE_HOURLY),HOUSE_CRAWLERS = ['niche-db/'],isHouseCrawl(request)matching only/api/openprofilesand/authors/<slug>/openprofile.md;tierForreturns it below the session and key checks.anonas before.apps/web/test/tiers.test.js(21 pass); full web suite green.🤖 Generated with Claude Code
https://claude.ai/code/session_01S7yeJUHGxA4P5N74xnsRPQ