Germline reference provenance, reproducible re-download, and comparison - #7
Merged
Merged
Conversation
A download now writes IMGT.yaml and AIRRC.yaml inside reference_base, and `reference build` writes sourcerer_build.yaml and carries any source sidecars into the igblast_base, so a reference keeps its provenance wherever it is moved. `download --from <reference>` re-downloads the versions a reference was built from: OGRDB sets through the versioned API, and an IMGT release from the genedb-releases archive, since GENElect serves only the current build. When the archive lacks the release asked for, the nearest is used and recorded as a substitute, so a later --from cannot quietly pin the neighbour as though it were the original. `download --compare` and `sourcerer reference diff` compare two references allele by allele. `reference build --map` takes a manifest naming the species and chain of files whose own names do not say, rather than inferring them: a chain guessed wrongly does not fail, it files alleles under the wrong locus. `sourcerer reference show` reports what a folder is and where it came from. IgBLAST .ndm and .aux generation is deliberately not here; it belongs in nf-core/airrflow. One consequence to know about: the mirrored NCBI aux does not cover OGRDB's mouse J names, so a mouse igblast_base built here cannot call CDR3 until airrflow regenerates it. Verified against live IMGT, OGRDB and genedb-releases: an ogrdb --from round trip reproduces its baseline allele for allele, and airrc-imgt --from builds the 24-chain blend without the bulk archive overwriting the OGRDB sets it exists to use.
makeblastdb -parse_seqids refuses an identifier over 50 characters, taking the whole database with it. A name that long is now shortened instead: the head is kept and a digest of the whole name replaces the tail, so the same allele shortens identically wherever it is built, which matters because IgBLAST's auxiliary and delineation files are keyed by name. The shortened name is what lands in a v_call, so the mapping back to the original is written to shortened_alleles.tsv beside the databases. Dropping -parse_seqids would lift the limit, but igblastn then reports gnl|BL_ORD_ID|0 instead of the gene name, which MakeDb cannot parse. Only VDJbase-style novel allele names reach the limit in practice. A build now also compares the reference's J alleles against the mirrored NCBI auxiliary file and reports any it does not name. IgBLAST looks a J germline up there by name, unlike the V delineations, which it transfers by alignment, so an allele with no row gets no CDR3 and no productivity call and nothing says so. OGRDB's mouse sets hit this: they name their J alleles IGKJ0-4JXG*00 and NCBI's mouse_gl.aux lists none of the 22. The names are recorded under aux_not_covered in sourcerer_build.yaml and the warning recommends building an auxiliary file for them; sourcerer does not build one, that belongs with the pipeline running IgBLAST. Because it is a real comparison rather than a rule about mouse, it also covers custom references built with `reference build`.
A download fetches one species but a reference_base holds as many as were downloaded into it, and the sidecars were overwriting rather than merging: downloading mouse into a folder that already held human erased human from IMGT.yaml and AIRRC.yaml while leaving its FASTAs in the tree. The reference then described only mouse, and `download --from` would have re-fetched only mouse, silently building something else. Both sidecars now merge, IMGT.yaml by species and AIRRC.yaml on (species, set), so re-downloading a set updates its entry instead of adding a second one. The IMGT release moves from a single top-level field to one per species. Human and mouse can be downloaded weeks apart from different GENE-DB builds, and one field would misname one of them. applyPins reads only the species being downloaded, so a mouse --from against a mixed reference cannot pin human's release. Added `download all`, which fetches every species into one reference_base described by a single IMGT.yaml and AIRRC.yaml. It is offered only for the germline sources, and only for download: OAS's collections are paired and unpaired, which have different fields and different outputs, and searching two species at once would merge two unrelated hit lists. `all` means every species sourcerer supports, not every species the source publishes, and the help says so -- OGRDB also carries rhesus macaque, deer mouse and rainbow trout, and IMGT many more.
The default request delay is cautious because IMGT and OGRDB are small academic servers. The IgBLAST support mirror runs against NCBI's file server and makes over a hundred requests, where that delay spends most of the build asleep. The build now uses a low-delay client for the mirror alone; the germline downloads keep the polite default.
download all fetched each species through a fetchSpecies helper. The helper was only ever called from the one loop, so fold it back in: the per-species work reads inline and its docstring goes away. download all is now the species list plus the loop, nothing more.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on the merged IMGT/OGRDB/airrc-imgt sources (#3).
and AIRRC.yaml (each OGRDB set's version + release date, and Zenodo DOI under
--resolve-doi). Sidecars merge across species rather than overwrite.
built from — OGRDB via the versioned API, IMGT via the genedb-releases archive
(nearest release, with a warning, when the exact one is not archived).
(identical / added / removed / changed).
sidecars into the igblast_base.
logged) and reports J alleles the NCBI auxiliary file does not cover.