Conversation
Some OAS studies record no Subject in their own metadata; buildSamplesheet now passes that sentinel through raw (rather than silently falling back to the study name, which would falsely pool unrelated individuals into one subject) so there is something real for a human to resolve later. Ncbi.py closes that gap deterministically: it pulls the SRA run or GEO sample accession out of a samplesheet row's sample_name, resolves it to a BioSample record via E-utilities, and suggests a subject id only for the handful of generic patterns (trailing locus/visit suffix) that are safe to strip regardless of study -- everything else is left for a human, and a pooled/multi-donor BioSample is flagged rather than guessed. `sourcerer oas verify SAMPLESHEET` is the CLI surface: it writes an evidence report that carries every column the input samplesheet already had, with biosample_id/biosample_id_suggested/status/... appended, so the report can stand in as airrflow input once any remaining sentinel subject_id values are filled in by hand. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found reviewing oas verify's output against real OAS studies:
- A GSM whose SRA experiment is shared with sibling GSMs (a common 10x
cellranger submission pattern) named none of them individually, so
resolveAccessions' SRA-only lookup reported 'not_found' for a GSM that
plainly exists (McIntire_2024/GSM6504709). Even when SRA did resolve,
its linked BioSample text sometimes dropped the subject that GEO's own
title states plainly (GSM6504685: BioSample text has no subject at all,
GEO's title has "subject P04"). resolveGsmTitles now queries GEO's own
'gds' database directly for every GSM, and its title wins whenever GEO
has one -- verified live against both examples.
- poolCodes only recognized a pooled/hashed sample described with a
'hashed'/'pooled'/'multiplexed' keyword plus a parenthesized code list.
Ferreira_2024 pools donors with neither ("BCR-Seq, BNT/BNT d7, donor 31,
32 and 33"), so those rows were silently reported 'ok' with the entire
multi-donor sentence as if it were one subject -- worse than not_found,
exactly what AMBIGUOUS_POOLED exists to prevent. A second, independent
check for a plain-prose donor list now catches this, without flagging a
genuine single donor ("donor 31" alone still resolves normally).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… table, verify column names, one-pass conversion
- addFilterArgs shows the presence tokens (--subject {*,defined,undefined})
in usage instead of a generic VALUE, so --help alone tells you what to pass.
- Field.match accepts a unique case-insensitive vocabulary match (--species
Human -> human), logging the substitution; an ambiguous fold or a
presence-only field still errors as before.
- handleSearch and download --dry-run print an aligned table (unit_id, count,
Species, Disease, Subject, BSource) instead of just unit_id and count, so
units can be picked without --out.
- oas verify's biosample_id / biosample_id_suggested columns are renamed to
ncbi_sample_name / ncbi_subject_suggested, which read correctly (they hold
a sample name, not a biosample identifier); cheap to do now since the
command is still unpushed.
- --ncbi-api-key's help now says the NCBI_API_KEY environment variable is
preferred, since a key on the command line ends up in shell history.
- AirrWriter/FastaWriter replace writeAirr/writeFasta's internals with a
push-style API; handleDownload now converts each unit once and feeds both
writers from the same chunk stream when --format airr --format fasta are
both requested, instead of reading and normalizing the unit twice.
…kflow Pin the downloaded-file format alongside the form and catalog snapshots: data_contracts.yaml records one ranged-probe unit per paired path layout plus one unpaired, with column sets, gzip member count, metadata key types and heavy/light stem pairing. catalog_fingerprint.json and provenance.json complete the snapshot. Drift.py diffs a snapshot against git:HEAD or a directory, classifies every finding on the additive/anomaly/removed/structural ladder, and reports the maximum as the exit code behind `sourcerer schema check`. Contracts.py holds the file-format probe logic; OasSource.harvestArtifacts wires both into `schema refresh`, which now writes a file only when its content changed so a quiet month leaves the worktree clean. schema_drift.yml runs monthly, retries the flaky host, opens a PR only on real drift, and files an issue on a failed harvest. A KNOWN_FIELDS test turns an unmapped upstream field into a red CI run on that PR. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add a longitudinal column populated from the OAS Longitudinal field. Like Age, OAS records this as a presence flag rather than a real value, so both the "no" sentinel and an absent field collapse to the samplesheet's NA placeholder; only a genuine value passes through unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
reStructuredText was parsing ncbi_ as a link-target reference because of the trailing underscore before whitespace
…an match Detail pages escape commas in a value the same way the search form does, but parseDetailPage copied the raw text straight into the catalog without undoing that escaping. 56 paired units ended up with a BType value --btype could never match, since the flag validates against the unescaped form vocabulary while the catalog did an exact-value comparison, so the query silently returned zero hits. Fixes the parser, repairs the 56 affected rows in the checked-in catalog, and adds a unit test plus a schema check regression guard against the paired collection.
…every source schema refresh called harvestCatalog/enrichCatalog unconditionally, but only OasSource defined them, so refreshing imgt, ogrdb or airrc-imgt crashed with AttributeError right after the schema harvest. SourceBase now defaults harvestCatalog to None, and the refresh loop skips merging, enrichment and catalog writing for any collection that returns it. Adds a stub-source test confirmed to reproduce the original crash when the fix is reverted.
The code that flags upstream changes always tagged them "unpaired". Now it reads the correct label from the data instead of assuming it.
…load provenance During a download, the tool checks each file for problems, like rows missing key data, but it used to throw that information away right after checking. Now it saves a running total of those problems for the whole download, so you can tell if something went wrong. It also now records exactly which version of the source data it used, instead of just an approximate date.
…ctions Several modules meant to work for any data source only actually worked for OAS: to add a second dataset source, you'd have had to either shape its metadata like OAS's or edit code in four shared files. This moves every OAS-specific piece behind hooks a source can implement on its own: - Converting a downloaded file used to import OAS's report-counter function directly. SourceBase now has a newReport() method every source can override; the default is empty. - Building an airrflow samplesheet used to read OAS's own metadata field names (Species, BSource, Subject, ...) and hardcode its collection names. That mapping now lives in OasSource.samplesheetRow(); the shared Airrflow.buildSamplesheet code only knows the samplesheet's column names, not any one source's metadata. - The catalog file format and the "search" command's output columns were a single hardcoded list shaped for OAS. Each source now declares its own catalog_columns, search_columns and enrichment_columns. One visible effect: `imgt search -o` and its search table now show imgt's own (currently empty) columns instead of always-blank OAS ones; OAS's own output is unchanged. - Cli.py had an `if name == 'oas'` special case to add the `verify` subcommand, and a hardcoded dispatch for it. Sources can now add their own subcommands via SourceBase.addActions(), which registers a handler Cli.py looks up generically. The whole verify feature (parser, handler, NCBI evidence report) moved into Oas.py alongside the rest of OAS's code. Tests moved with the code they cover: OAS-specific samplesheet and verify tests are now in test_Oas.py, and test_Airrflow.py keeps only the merge logic that's genuinely shared. Germline source files (imgt, ogrdb, airrc-imgt) are untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.