0.24.0: dumps, not pages: MusicBrainz, Open Library, Podcast Index and Discogs whole - #94
Merged
Conversation
A source that reads a multi-gigabyte dump (MusicBrainz tar.xz, Discogs
XML gz, Open Library TSV gz, Podcast Index SQLite in a tgz) cannot return
one array: the file does not fit in memory and the walk does not fit in
one run. `pull` may now hand back an async iterable of `{ items, cursor }`
batches (or be an `async *` generator); ingest drains it one batch at a
time through the same normalise/dedupe/upsert path (extracted into
writeBatch), saves each batch's cursor as soon as it is written (new
queries.saveCursor) so a container that dies mid-walk resumes at the last
batch, and takes the run's `{ cursor, note, nextInMinutes }` from the
iterator's return value. The array form is unchanged. A batch written
past the deadline ends the run and reschedules in a minute, closing the
generator so its files and processes go with it.
http.download streams a file to disk with bounded memory and resumes
with Range from whatever is on disk (append on 206, restart on 200, 416
is whole), verifies Content-Length, sends the user agent (Podcast Index
403s without one) and returns { path, bytes, complete }.
core/dump.js: dumpDir under INGEST_DATA_DIR (default tmpdir), gzipLines
via node:zlib with a final-line flush (MusicBrainz ends without a
newline), xzLines by spawning `tar --to-stdout` for a member (or xz -dc),
chosen over extracting because the inflated member is ten times the
archive on an ephemeral disk shared with the web role, tsvJsonLines for
Open Library's five-column rows, lineOffsetReader for O(1) byte-offset
resume in a plain file, untar and sqliteRows for Podcast Index. Every
reader yields every line so `skip + lines read` is the file position;
skipping into a compressed file re-inflates from the top. xz-utils goes
into the image because Bun.Archive cannot read xz.
Reaper hazard: a run longer than its cadence (or one that asked for
nextInMinutes: 1) came due again while still writing and the tick
enqueued a second one beside it. dueSources now skips a source with a
run still running inside the reaper's window, force included, so an
adapter's budgetMs widens the guard along with the reaper and the lock.
runSource takes an optional resolveAdapter so a test can run a fake
adapter through the real loop: mocking the adapters registry hangs Bun's
loader, because the registry and core import each other.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`rs.pipe(gz)` left the file stream open when the consumer returned before the end, which every dump run does when it hits its deadline mid-file: 200 early returns measured as 202 leaked descriptors, and a worker that walks a dump hourly would run out of them. `stream.pipeline` takes the read stream down with the gunzip side, and the generator's finally destroys both regardless. Measured at 0 leaked across gzipLines, tsvJsonLines (plain and gz), lineOffsetReader and xzLines. Review tests added for the streaming ingest loop (an upsert that throws mid-walk, strict pull/upsert/save interleaving over 50 batches, a return without a cursor, no return value, dedupe per batch, an iterator with no return()), the download edges (416 with a longer local file, a 206 with the wrong start, no Content-Length, a body past Content-Length, 200 MB streamed with flat rss), the readers (1 MB multibyte line, corrupt and truncated gzip and xz, fd counts, escaped tabs and quotes in Open Library JSON, sqliteRows), and the reaper window at a 60 minute budget (50 and 69 minutes in: untouched; 71 minutes in: reaped then due). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
A dump-fed adapter for the music collection on the streaming contract:
resolves LATEST, downloads artist.tar.xz and release-group.tar.xz into
the dump directory with resume, streams the mbdump member through
xzLines and yields batches of 500 with a { dir, entity, line } cursor
after each. budgetMs 55 minutes; a run stops itself a minute short of
the deadline and resumes in ten, as does a download still in progress
or one that failed three times in a row. CC0 core fields only: tags,
genres, ratings, annotations and relations are dropped before storage.
Fixtures are real bytes: instrument.tar.xz as served and the first 200
rows of the artist and release-group members, cut from a Range GET of
each archive and re-packed as tar.xz with the same member names.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
…d once more A full-length archive on disk that xz rejects (a bad write, a volume that outlived a different build of the file) threw `tar exited 2` on every run and was never removed, and since http.download sees a whole file and answers nothing, the source errored once a cadence for good. The walk now drops the file, fetches it again in the same run and resumes from the last batch it yielded, so no row is lost or repeated; a second unreadable copy throws with the file gone. Review: two consecutive pulls resume at the exact next row, a near-deadline run yields nothing and asks for ten minutes, a new LATEST restarts at artist line 0 and prunes the old archives; rss stays at 70-74 MB across a 320 MB synthetic member; every fixture item has a prefixed id, a kind, a title, an absolute url, lowercase tags and the CC0 credit, and none carries tags, genres, rating or annotation; the live probe reads today's real bytes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
A dump-fed source for the books collection. One HEAD on the _latest URL
resolves the dump date, the two dated gzip files (authors 780 MB, works
4 GB) are downloaded with resume into dumpDir('openlibrary-catalog'), and
the walk yields 500-row batches with a { version, file, line } cursor so
the next run re-inflates to that line and carries on; budgetMs 55 min,
an unfinished walk or download resumes in 10 min, a complete pass keeps
the newest last_modified as a watermark and the next dump skips the rows
older than it. Works keep the /works/OL...W external id openlibrary-new
writes, so the two sources merge; authors are /authors/OL...A.
Fixtures are 100 real rows of each dump from a Range GET of the first
4 MB (dump 2026-08-31), chosen to cover boxed descriptions and bios,
every first_publish_date shape, -1 covers, nameless authors, links and
remote ids. The test drives the generator by hand against a fake
archive.org and a temp dump directory: item shape, a walk across two
runs resuming from the cursor, the watermark skip, a download still in
progress, bad rows skipped, an unreadable file removed, three failures
stopping without losing the place, every request failing throwing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
…ast-Modified Review fixes. A run whose deadline is already near yielded a batch and started a download with the 30 s floor timeout; it now returns before the download with the cursor unchanged and asks for ten minutes when less than DOWNLOAD_MIN_MS is left. resolveVersion took a date off Last-Modified when the final URL carried none, but that date is the upload (Sep 2 for the Aug 31 dump), so the dated archive.org URL it produced would have 404ed on every run; an unnamed answer is now a failed request, and a walk in progress carries on with its cursor's date. The test drives the deadline with a spied Date.now (stopAfterBatches) instead of deadline: 0, and covers both cases. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
A dump-fed source for the podcasts collection: HEAD the 1.8 GB tgz at
public.podcastindex.org with a descriptive user agent (it 403s without
one), version the walk on the ETag, download with Range resume into
dumpDir('podcastindex'), extract the 5.1 GB database with tar, discover
the table and columns from sqlite_master and pragma table_info (the
published file is a camelCase `podcasts` table, not the repository's
snake_case `newsfeeds` DDL), then walk `where id > ? order by id limit
500` yielding a batch and a cursor { version, lastModified, afterId }
at a time under a 55 minute budget, weekly.
Dead, untitled and unparseable rows are skipped; a row that throws is
one bad row; three consecutive failures keep the place and resume in
ten minutes; a run in which every request failed throws. Each item
carries the normalised feed URL the other podcast sources carry, the
Podcast Index attribution, and the licence in the description.
Fixtures are real bytes: the CREATE TABLE cut from the first 4 MB of
the archive, the real HEAD and Range responses, the tar listing and the
repository DDL. The test builds a tiny SQLite from that DDL and drives
the generator through a fake http and a temp dump directory.
Not registered in packages/adapters/src/index.js; that is the
integrator's step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
The walk read and yielded a batch before it looked at the clock, so a run that started near its deadline still did one batch, and one arriving with nothing extracted opened a 60 s download attempt only to abort it. The clock is now checked before the download and before every batch read, the first included; a run past its stop point keeps its place and asks for ten minutes. The test that asserted the old behaviour now asserts this one, drives the mid-walk stop with a patched clock, and checks that a generator the core closes after one batch resumes at that batch's cursor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
A dump-fed source for the music collection. Two gzipped XML files a month
from data.discogs.com (CC0), walked in batches of 500 under a 55 minute
budget with a { month, entity, recordIndex } cursor the core saves after
every batch.
The server ignores Range and rate-limits hard, so a run makes at most one
download attempt plus one checksum request through fetch directly (the
core's http.request retries a 429 after a minute, which re-arms the
window), schedules past retry-after on a 429, never asks for a file whose
complete copy is on disk, and verifies each download against the month's
CHECKSUM.txt before walking it. Records are read by a small streaming
tag scanner and a tolerant element parser, no dependency, one record in
memory at a time; a bad record is counted and skipped.
Fixtures: the first forty artist records are real bytes from the head of
discogs_20260901_artists.xml.gz (one Range-limited GET, cut at 512 KB);
the masters fixture follows the documented <master id="..."> shape.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
…g yielded past the deadline Review fixes on the dump adapter: - A run makes at most ONE request to data.discogs.com: the file or the checksum list, never both. A download this run defers the checksum list to the next run (ten minutes), and a file walked this run never triggers the next file's download. A month costs three requests over three runs. - A 404 (the month not published yet) returns the run's place instead of `prev`: when the artists file was walked in the same run, `prev` was behind batches the core had already saved, so the cursor stepped back and the next run re-walked them. - A run whose deadline has already passed before a walk yields nothing and asks for ten minutes, instead of reading one batch past it. Tests follow the new flow (28 tests), with a fake clock for the stop-after-a-batch case and a regression test for the 404 rewind. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
…d Discogs whole The four catalogues too big for a paged API arrive as files of one to eleven gigabytes, so the core learns to ingest a dump: a pull may hand back an async iterable of batches, each written and its cursor saved before the next is read, so a container that dies mid-file resumes at the last written batch; http.download streams to disk and resumes with Range; dump.js reads gzip, xz (tar member), Open Library TSV and SQLite with bounded memory; budgetMs on an adapter widens the reaper and lock windows, and a source whose run is still inside its budget is not re-enqueued. xz-utils joins the image. On top of it: every artist and release group in MusicBrainz (CC0 core fields only; tags, genres and ratings are CC BY-NC-SA and are dropped), every work and author in Open Library (works keep the same external id as openlibrary-new so the rows merge), every podcast in the Podcast Index (from the SQLite dump, never the API), and every artist and master in Discogs (one request a run, retry-after honoured, checksum verified). Each was built on its own branch and adversarially reviewed: a gzip reader leaked a file descriptor per early return, three sources yielded a batch past their deadline, a 404 rewound a cursor the core had already saved, Open Library's version fell back to an upload time, and a tar archive that could not be read was never removed. All fixed before integration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last four catalogues from the arr survey are files of 1 to 11 GB, not paged APIs. This PR gives the core a streaming-dump contract and adds the four sources on top of it. Everything was built one agent per piece on its own branch and adversarially reviewed before integration.
Core (
feat/dump-core): apullmay return an async iterable of{ items, cursor }batches;ingest.jswrites each batch through the existing normalise, dedupe and upsert path and saves the cursor before reading the next, so a container that dies mid-file resumes at the last written batch (at-least-once is safe: upserts are idempotent on(source_id, external_id)and skip unchanged content).http.downloadstreams to disk and resumes withRange(206 appends, 200 restarts, 416 means whole, Content-Length verified).dump.jshasdumpDir,gzipLines,xzLines(tar member via xz),tsvJsonLines(Open Library),lineOffsetReader,untar,sqliteRows.budgetMson an adapter widens the reaper and BullMQ lock windows, anddueSourcesno longer re-enqueues a source whose run is still inside its budget.xz-utilsadded to the image. Review fixed a file-descriptor leak ingzipLineson early return.Sources
musicbrainz-catalogopenlibrary-catalogpodcastindex-catalogdiscogs-catalogReview findings fixed before merge: three sources yielded a batch past their deadline; Discogs could make two requests a run and a 404 rewound a cursor the core had already saved; Open Library's version fell back to the upload time; a MusicBrainz archive tar could not read was never removed.
Known and documented: the Railway service has no volume, so
tmpdir()downloads do not survive a deploy (setINGEST_DATA_DIRon a volume to keep them);finishRunstill recounts items per source and should become incremental before a 30M-row source lands; every MusicBrainz release group links a Cover Art Archive image that may 404.Version 0.24.0.
bun test1528 pass; lint unchanged.🤖 Generated with Claude Code
https://claude.ai/code/session_01CDEiDss9RWYibtmxSk5Gr2