Fill the journal dimension's publication years from NLM's serial catalog

[gaurav]

journal.start_year / end_year / active have been NULL columns since the schema was written, because J_Entrez.txt does not carry them. NLM's serial catalog (serfile) does, so load_journals now reads both sources: J_Entrez defines the row set and the titles, the catalog fills the three year columns. A real load takes 40,384 of 42,056 journals (96.0%) from NULL to a publication year.

No issues are closed by this. It grew out of pubmed-downloader#16, where upstream's position is that J_Entrez.txt is the wrong source and the package reads the serfiles instead — so the first job was working out whether to switch sources or add one.

Why two sources rather than one

Switching wholesale would have been a regression, and the measurement is the reason to keep both. J_Entrez.JournalTitle is byte-for-byte identical to esummary.fulljournalname. On the 8,259 journals (20% of the 41k overlap) where the two sources disagree, sampling 50 at random and fetching one article each:

PubMed API	field	matches J_Entrez	matches serfile
esummary	fulljournalname	27 / 27	0 / 27
efetch	<Journal><Title>	22 / 23	0 / 23

The disagreements are systematic, not noise — 3,747 diacritics (NCBI ASCII-folds), 1,554 corporate suffixes (Academy of management review. Academy of Management), ~2,958 qualifier disambiguators (Ethos (Berkeley, Calif.)). Rebuilding the Entrez rendering from the catalog tops out at 99.22%; the residual needs NCBI's transliteration table, ~250-character truncation and its punctuation rules.

docs/journal-catalog.md is the full write-up, written to be shareable outside this repo since it is also the answer to the upstream PR. It covers the CatfilePlus evaluation too (5.08 GB, MARCXML-only since 2024, same catalog titles, ~1% marginal coverage) so nobody redoes it.

What it produces

uv run pubmed2db --data-dir data journals   # 69 s, ~890 MB
Loaded 42056 journals.

non-NULL start_year   40,384 / 42,056  (96.0%, was 0)
active t/f/NULL       28,035 / 13,012 / 1,009
Ethos (9877005)       'Ethos (Berkeley, Calif.)'  1973  NULL  true

That last row is the design in one assertion: J_Entrez's title with the Entrez disambiguator intact, next to the catalog's years.

Notes for review

• We enumerate NLM's listing ourselves rather than calling upstream's ensure_serfile_catalog(), which passes skip_prefix="serfilebase" and takes all 83 monthly deltas instead. That is a cost argument, not a coverage one — measured, its 83 files carry 151,974 distinct records against the baseline's 150,942, because NLM re-releases the whole catalog through the deltas over time. Anchoring on the baseline is 872 MiB across 10 files rather than 2.52 GiB across 83, and avoids the 80% duplicate records (761,624 yielded for 151,974 distinct journals, one appearing 24×) that a caller would otherwise have to dedupe. Fetching the deltas as well as the baseline is not incidental either: they supply 429 of the 902 journals the baseline lacks.
• Three things that are easy to get wrong, each pinned by a test: ~1,600 catalog years are MARC wildcards (19uu, uuuu) that int() raises on; 9999 is the "still publishing" sentinel and a missing end year means unknown rather than ceased (so active stays NULL, not false); later files must win over the baseline.
• Two properties of NLM's own files are handled explicitly, both found by measurement rather than guessed. It publishes zero-byte files — serfile.20240903.xml is served with Content-Length: 0, and iterparse raises on it even with recover=True — so _parse_serfile skips a file it cannot parse rather than letting one bad file cost every journal its years. And anchoring on the newest baseline drops 369 records NLM has since removed, of which 11 are J_Entrez journals (0.03%) that lose their years; accepted rather than spending 2.52 GiB and a dedupe pass on eleven journals.
• Presence and usability differ, which matters when checking a run: 99.9% of overlapping journals have a PublicationFirstYear element, but only 96.0% yield a parseable year. The gap is the wildcards.
• No schema migration — those three columns already existed and were always NULL.
• No export change. journal_name / journal_abbrev still come from J_Entrez, which is the point of not switching.
• The years are an enrichment. A ~450 MB download failing must not cost us the dimension, so the fetch degrades to NULL years with a warning, matching the posture cli.update already takes toward the journal step. conftest defaults every test to "no catalog files" so the suite stays offline.
• journals now pulls ~890 MB on a first run. NLM posts occasional bulk re-releases as monthly deltas — 2026's May and June files are 169 MB and 249 MB against the usual ~1 MB. Streaming them costs ~5 s at ~50 MiB RSS, so 02-journals.sbatch keeps its 8G header; only its host and scratch notes changed.
• Catalog files are re-fetched only when the server's copy moves. pystow's ensure() skips by file name, which would have let a republished file keep stale bytes indefinitely — the hazard AGENTS.md already records for upstream's ensure() — and re-fetching ~890 MB every run is not an option. NLM publishes no .md5 sidecars for serfile (checked: 302, and none in the listing), so download.py's checksum machinery does not transfer; the server's stable ETag does the same job for a few hundred bytes. Each cached file carries an .etag sidecar and one HEAD per run decides. A failed HEAD leaves the cached copy alone. Verified live: a warm run re-downloads none of the 10 files.

What it deliberately does not do

• Enrich journal_issn. Filtered to ValidYN="Y", serfile's ISSNs are 99.3% identical to J_Entrez's (+143 / −156) — an earlier count of "+4,300" was an artifact of including ValidYN="N" records. The genuinely new content is the 4,763 cancelled ISSNs, useful as resolver aliases but needing a valid column first so a consumer cannot mistake them for current.
• Drop journal.abbreviation_iso. IsoAbbr equals MedAbbr in 42,056 / 42,056 records, so the COALESCE in export.py is a distinction without a difference — but deleting a column is a Parquet schema change for downstream consumers, so it is its own decision. Filed as Consider dropping journal.abbreviation_iso, which duplicates abbreviation_medline #48.
• Remove our own _parse_journal_overview. It stays; we depend on nothing upstream. This work is nonetheless the argument for pubmed-downloader#16 rather than against it: process_journal_overview() cannot return a single record (Journal.start_year/end_year are annotated int | None with no default, so pydantic makes them required, and J_Entrez.txt has no such key), and since fulljournalname/IsoAbbr exist nowhere in CatalogRecord, that function is the only library path to them and its input can never supply years. If the PR lands we delete ~35 lines; if it does not, nothing here changes. FUTURE.md records both, plus the caveat that Journal.active defaults to True — wrong for the 13,012 journals serfile marks ceased.
• Issue Report pubmed-downloader's ReferenceList-scope bugs upstream (or decide to keep the workarounds local) #34 is unrelated and unaffected — that is the .//ReferenceList scope bug pair in upstream's _extract_article.

Before merging

Nothing is blocking. 247 tests pass, and the pipeline was run end to end against live NLM data (numbers above), including a warm re-run to confirm the ETag check prevents a refetch. Deferred work is recorded in FUTURE.md and listed under "deliberately does not do" above; #48 is filed for the abbreviation_iso question.

How the conclusion changed during the work — kept for anyone tracing why the docs argue what they argue; the durable versions are in docs/journal-catalog.md and AGENTS.md.

The branch is named switch-catalog-to-serfile because a wholesale switch is what was originally asked for. The esummary comparison is what redirected it to a two-source design; the branch name was left alone rather than rewritten.

Two numbers moved after the first full run against live data, and both are corrected above and in the doc:

• The serfile coverage gap is 902 against the baseline, but the monthly deltas recover 429 of them, so the real figure is 473 (1.1% of the dimension). That also prorated the article-volume estimate and cut CatfilePlus's marginal value roughly in half.
• A year of serfile is ~890 MB, not the ~460 MB estimated from the baseline plus twelve nominal ~1 MB deltas, because of NLM's bulk re-releases.

An intermediate commit removed FUTURE.md's pubmed-downloader#16 entry on the grounds that upstream's "the package reads the serfiles instead" made the PR moot. That read the measurement backwards and was reverted in 0a79557: the finding that J_Entrez.txt is not a subset of the catalog is precisely what justifies making Journal's year fields optional.

An earlier draft of the analysis also claimed serfile added ~4,300 ISSNs. That was an artifact of not filtering on ValidYN, and the doc now says so explicitly so the mistake is not repeated.