0.26.0: the dht collection, read from bittorrented.com's own database - #93
Merged
Conversation
Torrents as the bittorrented.com crawler observes them on the BitTorrent DHT. The site has no submit or export endpoint, but the function its own /dht page reads through, browse_dht_torrents, is callable over PostgREST with the publishable key the browser bundle already ships, so the adapter reads the same rows the site does, sorted by date from a cursor (the one shape the function has an index for; by seeders it hits its 15s timeout), 500 a page, paging by offset inside a shared timestamp because the crawler writes in batches. Adult material stays out: a torrent the crawler classes xxx is skipped, as is one whose name says so in plain words. nichedb.dev is read through family DNS filters, and the site this reads from went members-only for the same reason. Needs BITTORRENTED_SUPABASE_URL and BITTORRENTED_SUPABASE_KEY on the deployment (never the service role key); the source seeds paused without them. Feeds: newest, seeded, and one per content class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UpE2YNWeoYxtu2B8oL2LPS
ralyodio
force-pushed
the
feat/dht-collection
branch
from
September 13, 2026 13:46
8f93089 to
4d64530
Compare
ralyodio
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A
dhtcollection at nichedb.dev/c/dht: torrents as the bittorrented.com crawler (bitmagnet) observes them on the BitTorrent DHT, read from the site's own Supabase.bittorrented-dhtcallsrpc/browse_dht_torrentsover PostgREST with the site's publishable key (the same one its browser bundle ships; never the service role key). Sorted by date from a cursor, 500 rows a page,pagesPerRun(default 10) andminSeeders(default 0) as source config. The cursor is a timestamp plus how many rows at it were read, because the crawler writes in batches that share a timestamp anddate_fromis inclusive. Sorted by seeders the function hits its 15s statement timeout on a table this size; by date from a cursor a page returns in ~0.4s (measured).xxxis skipped, and so is a name that says so in plain words when the classifier has not caught up.torrent, keyed by infohash, linking tobittorrented.com/dht/<infohash>, tagsdht,type:<class>,seeded/unseeded; data carries the magnet (built with a URL-encoded display name), size, file count, swarm as crawled and the class.dht-latest,dht-seeded,dht-movies,dht-tv,dht-music,dht-software,dht-books.BITTORRENTED_SUPABASE_URL,BITTORRENTED_SUPABASE_KEY(needsEnv, so the source seeds paused without them). Set on the Railway services and in thenichedb--prodvault.Checks
bun test: 1364 pass (6 new).biome checkclean on the changed files (the repo's one remaining error is a pre-existing CSS specificity warning).Volume
The crawl can add a great many rows a day;
pagesPerRun× 500 per 15-minute cadence caps growth (default ≈ 480k rows/day at most). Lower it on the source page if the collection grows faster than wanted.🤖 Generated with Claude Code
https://claude.ai/code/session_01UpE2YNWeoYxtu2B8oL2LPS