Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
a3e7951
test(summary): an uncalled deletion must pair with nothing
GeiserX Nov 17, 2026
2a1427d
test(family): a pair somalier could not measure must not read ok
GeiserX Nov 17, 2026
690cc49
fix(family): a pair somalier gives no relatedness for reads not check…
GeiserX Nov 17, 2026
8ece222
fix(summary): pair only a deletion with a called ALT allele
GeiserX Nov 17, 2026
9d0f111
fix(clinvar): fail step 06 when the first bcftools of a container pip…
GeiserX Nov 17, 2026
da1e8bd
fix(clinvar): build step 06's shared normalised ClinVar once under a …
GeiserX Nov 17, 2026
dbd6c4c
test(clinvar): fake-docker case for step 06 pipeline failures and con…
GeiserX Nov 17, 2026
36e3205
docs(clinvar): say how step 06 builds its shared normalised ClinVar
GeiserX Nov 17, 2026
a95f854
fix(family): pair each ALT of a 1/2 call in compound_het.tsv
GeiserX Nov 17, 2026
4c20d16
fix(family): report a son's X and Y de novo calls and check parental …
GeiserX Nov 17, 2026
48febd8
docs(family): say which calls make a de novo pass and that 1/2 pairs …
GeiserX Nov 17, 2026
15c8fd4
fix(clinvar): stop step 06 when the stale normalised ClinVar's folder…
GeiserX Nov 17, 2026
d973ef6
fix(family): keep the too-few-sites reason under 1,000 sites
GeiserX Nov 17, 2026
2a7fbde
fix(family): pair the two ALTs of one 1/2 call in trans whatever the …
GeiserX Nov 17, 2026
b48fdb0
test(parity): golden family report outputs for the son's Y row and pa…
GeiserX Nov 17, 2026
9388727
fix(clinvar): take step 06's normalised ClinVar lock with flock when …
GeiserX Nov 17, 2026
817555a
test(parity): take the 3gen and bench family goldens from the runner'…
GeiserX Nov 17, 2026
2846ecf
Merge remote-tracking branch 'origin/main' into pgp/rv-two-hits-uncal…
GeiserX Nov 18, 2026
a45ecaf
Merge remote-tracking branch 'origin/main' into pgp/rv-family-check-nan
GeiserX Nov 18, 2026
8680815
Merge remote-tracking branch 'origin/main' into pgp/rv-family-trio-calls
GeiserX Nov 18, 2026
9c9b220
Merge remote-tracking branch 'origin/main' into pgp/rv-step06-screen-…
GeiserX Nov 18, 2026
cc99fa2
fix(family): more identical genotypes than sites is no count either
GeiserX Nov 18, 2026
7f6e39b
test(family): a declared parent and child with nan relatedness is not…
GeiserX Nov 18, 2026
d3a226e
fix(clinvar): open step 06's normalised ClinVar lock read-only so ano…
GeiserX Nov 18, 2026
e620a13
fix(clinvar): make every step 06 build of the normalised ClinVar take…
GeiserX Nov 18, 2026
7872c27
fix(clinvar): do not stop step 06 when another user creates its lock …
GeiserX Nov 18, 2026
6797001
Merge origin/main into the batch g tip
GeiserX Nov 18, 2026
85ea2ab
Merge #475 rv-family-check-nan into the batch g tip
GeiserX Nov 18, 2026
f3c8266
Merge #476 rv-two-hits-uncalled-deletion into the batch g tip
GeiserX Nov 18, 2026
e480eec
Merge #478 rv-family-trio-calls into the batch g tip
GeiserX Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,16 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- Step 06 (`scripts/06-clinvar-screen.sh`) no longer ships a ClinVar screen that is silently incomplete.
- Its three two-command pipelines in the bcftools container (ClinVar's normalisation, the sample's PASS filter and the copy of ClinVar's fields onto the hits) ran under `sh`, which has no `pipefail`. A failed first `bcftools` went unnoticed when the second one exited 0, and the step finished with part of the records. They now run in `bash -o pipefail`, so the step stops with an error and writes no hits file.
- Two samples screened at the same time after a new install or a ClinVar refresh both rebuilt the shared `clinvar/clinvar_pathogenic_chr.norm.vcf.gz` into one `.part` file. One run could fail, or publish a `.tbi` of another write than the VCF. Now one run builds it while the others wait and then reuse it. Every run that builds holds the folder `clinvar/.norm.lock.d`, with or without `flock`; where `flock` is installed it first takes `flock` on `clinvar/.norm.lock`, so a run with `flock` and a run without it still exclude each other. The old VCF is removed first and the new index goes in place before the new VCF, so a run that stops halfway leaves a copy the next run rebuilds. A stale copy in a `clinvar/` folder you cannot write stops the step with an error, and so does a `clinvar/.norm.lock` you cannot read; the step makes that file readable by every user.
No output file or option changes. The Nextflow module's code does not change. It reuses the copy step 06 publishes when that VCF and its `.tbi` sit beside the source and the VCF is newer; otherwise it normalises in its own task folder, with `pipefail`.
- `og family check` (step 38 and the Nextflow `FAMILY_CHECK` process) no longer reports `ok` for a pair somalier could not measure. somalier writes `nan` as the relatedness when it had nothing to divide by. Every comparison with NaN is false, so the degree bins read it as unrelated, and a pair declared unrelated or distant passed as "unrelated, as declared". Such a pair, one with `inf`, and one with counts that cannot be counts (negative, or more opposite homozygotes or more identical genotypes than sites) now get the verdict `not_checked`, a reason that names the value ("somalier gave relatedness 'nan' for this pair: not measured") and a `WARNING:` line. The step exits 0 for them. Under 1,000 sites the reason stays "too few to tell", unless a count is negative. Before, `inf`, and `nan` for a pair declared first or second degree, stopped the step as a mismatch. The columns of `relationships.tsv` do not change; `not_checked` was already one of its verdicts.
- A deletion the sample was never called for no longer makes a homozygous variant look biallelic. The two-hit rule of `og summary collect` dropped an AnnotSV deletion only when its `GT` was `0/0`. A row with `GT` `./.`, `0/.` or no `GT` still counted as a copy loss, so a `1/1` variant inside its span read as hemizygous, "biallelic" with the deletion, in the carrier list and the secondary-findings card. Now only a deletion with a called ALT allele in the sample pairs. A table without FORMAT and sample columns has no `GT`, so its deletions pair with nothing. No field, column or file name changes.
- Step 38's trio report (`og family report`) finds three kinds of call it missed or trusted too much. Each output keeps its name and columns; only which rows appear and their `quality` change.
- A child call with two different ALTs (`1/2`, as GLnexus writes a site with two ALTs) now counts as heterozygous for `compound_het.tsv`. Before, only a call with the reference allele did, so a `1/2` from a `0/1` father and a `0/2` mother, with both ALTs candidates in one gene, gave no row. Each ALT that is a candidate is now paired with its own origin. The two ALTs of one `1/2` call are on different copies, so they are one pair in trans even when the parents do not tell which ALT came from whom. A `0/1` call gives the same pairs as before.
- A son's de novo candidate on chrX or chrY outside the pseudoautosomal regions needs only the parent who passes that chromosome on: the mother for his X, the father for his Y. Before, both parents needed a call, so a son's Y with the mother not called (the usual case) and his X with the father not called gave no row. The depth and GQ floors of `pass` now apply to the child and those parents only.
- A de novo candidate is `low`, not `pass`, when a parent who passes the site on has 2 or more reads, or 5% or more of its reads, on the new allele (from its `AD`). Before, only the child's reads were read, so a parent called `0/0` with reads on the allele, as a mosaic parent has, still gave `pass`.
- A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again.
- VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37).
- The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2).
Expand Down
2 changes: 1 addition & 1 deletion docs/05-annotsv.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ AnnotSV gives class 4 or 5 to some calls that are not a rare deletion: a call on
| `AnnotSV_class` | AnnotSV's own class (4 or 5). `ACMG_class` is the class after the checks below: 3 when one of them moved the call, else AnnotSV's |
| `Depth_support` | for a deletion or duplication, whether its read depth shows the change (below); `.` for other types |
| `Rule`, `Rule_detail` | each check that fired (`no_depth_support`, `tandem_repeat`, `ig_tr_locus`, `mhc`, `gene_alias`) and why, in words |
| `FORMAT`, `<sample>` | the last two columns, as AnnotSV's input has them: the caller's FORMAT keys and the sample's values, with `GT` and, when the caller phased the call, `PS`. The reports pair a deletion with a small variant of the same gene by that phase (in trans when both sit in one phase set on different copies), and a deletion genotyped `0/0` pairs with nothing. An input without FORMAT, or a table with no class 4 or 5 SV, writes neither. A sample named like a column of this table stops the step with an error, since its `GT` and `PS` would be lost |
| `FORMAT`, `<sample>` | the last two columns, as AnnotSV's input has them: the caller's FORMAT keys and the sample's values, with `GT` and, when the caller phased the call, `PS`. The reports pair a deletion with a small variant of the same gene by that phase (in trans when both sit in one phase set on different copies), and a deletion with no called ALT in the sample (`0/0`, `./.`, `0/.` or no `GT`) pairs with nothing. An input without FORMAT, or a table with no class 4 or 5 SV, writes neither. A sample named like a column of this table stops the step with an error, since its `GT` and `PS` would be lost |

### Calls the table moves to class 3

Expand Down
2 changes: 1 addition & 1 deletion docs/06-clinvar-screen.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ In the Nextflow pipeline the same floor is `ext.min_gq` and `ext.min_dp` of `CLI
### What the Script Does

1. Lists the contigs each file holds records on (`bcftools index -s`) and keeps those the reference has too: ClinVar's contigs that are in the reference, and the sample's contigs among those. `bcftools norm` stops at the first record whose contig the reference lacks, such as a provider's unplaced scaffold or one of the `NT_` contigs the full ClinVar file has, and a record on a contig the other file lacks cannot match anyway. The step prints how many of the sample's records it leaves out and on which contigs. When no contig is shared it stops with an error instead of reporting zero hits, and says which file is named the other way (`1, 2, ...` against `chr1, chr2, ...`).
2. Builds `clinvar/clinvar_pathogenic_chr.norm.vcf.gz` beside the ClinVar file the first time, and again whenever the source file is newer: ClinVar's records on the reference's contigs, multiallelic records split (`bcftools norm -m -any`) and indels left-aligned against the reference. Every sample then reuses it. The Nextflow module reuses this file when it sits beside the `--clinvar` file and is newer; otherwise it normalises ClinVar inside the task.
2. Builds `clinvar/clinvar_pathogenic_chr.norm.vcf.gz` beside the ClinVar file the first time, and again whenever the source file is newer: ClinVar's records on the reference's contigs, multiallelic records split (`bcftools norm -m -any`) and indels left-aligned against the reference. Every sample then reuses it. It is rebuilt as well when its `.tbi` is missing. Two samples screened at the same time build it once: the first takes the lock, the others wait, then find it built. The lock is the folder `clinvar/.norm.lock.d` holding the process id, which every run takes, with or without `flock`. Where `flock` is installed, a run first takes `flock` on `clinvar/.norm.lock`, so only one run at a time makes the folder. The build goes to a `.part` file. The old VCF is removed first, and the new index goes in place before the new VCF, so a VCF newer than its source has the `.tbi` of the same build beside it, even after a run that stopped halfway. When the copy must be rebuilt and `clinvar/` is not writable by you, the step stops with an error. It also stops when you cannot read `clinvar/.norm.lock` (another user's file); the step makes that file readable by everyone. The Nextflow module reuses this file when it sits beside the `--clinvar` file with its `.tbi` and is newer than it; otherwise, a `.tbi` missing included, it normalises ClinVar inside the task.
3. Filters the sample VCF to PASS records. If the VCF has no PASS record at all (callers that leave FILTER as `.`), it uses `-f .,PASS` instead and prints a notice saying so. If no record is left, the step stops with an error.
4. Splits and left-aligns the sample the same way. `bcftools isec` matches only identical REF/ALT, so a pathogenic allele inside a multiallelic record (genotype `1/2`) is found only after this split. The split writes the other alt allele as missing, not as the reference (`bcftools norm --multi-overlaps .`): a `1/2` record gives `1/.` and `./1`, where the default would give `1/0` and `0/1`. Both copies of such a call carry an alt allele, so neither part may read as het. A poly-T tract is where this shows most: a call of 5 T on one copy and 9 T on the other, against a reference of 7, is one `1/2` record, and its 5 T allele is a ClinVar hit. The reports word such a hit "one copy, other copy not called ref (1/.)", never het; the ACMG and carrier lists count it as one copy of two. This step does not size the tract: a short-read call of a homopolymer length can be off by one or two bases, so check the reads. A sample record whose REF is not the reference's base at that position is kept as written (`bcftools norm -c w`), with a `REF_MISMATCH` warning in the log. Such a record cannot match a ClinVar allele, so the summary of item 11 counts these records, and prints a notice to check that the VCF was called on this reference when there are any.
5. Builds the combined alleles of neighbouring records. A caller can write one complex allele, a delins, as two records a few bases apart on the same haplotype, for example a deletion and an insertion. Matched one by one, the parts can hit two single-submitter ClinVar records and miss the better reviewed record ClinVar holds for the whole allele. `bcftools norm -m +any` joins only records at the same position, so it cannot build that allele. `og clinvar join-adjacent` pairs two records when the second starts at most 10 bases after the first ends (records that overlap are never paired) and their genotypes allow both ALT alleles on one haplotype: phased with the same genotype in the same phase set (`FORMAT/PS`), or unphased with the same genotype (`0/1` and `1/0` count as the same het call; an unphased call with a missing allele such as `./1` is never paired). A phased call with a missing allele, such as `1|.`, has its ALT on a known haplotype: it pairs with a record in the same phase set whose ALT sits on that haplotype (`1|0` or `1|.`), and the combined record keeps `.` on the haplotype where either part is missing. That `.` means the copy was not called there; when the part comes from a split multiallelic record, it holds the other alt allele. Two unphased het calls can also sit on different haplotypes, and the genotype cannot tell which, so the notes of a combined allele and its parts built from them end with "unphased, so the parts may sit on different haplotypes". The combined record runs from the first part's start to the second part's end; `og clinvar join-adjacent` takes the bases between the parts from the reference (an uncompressed FASTA with its `.fai`), and `bcftools norm` trims and left-aligns it the way ClinVar was. The gene is not checked here, because the sample VCF has no gene annotation yet: a combined allele only counts when ClinVar holds a record for it. The step prints how many combined alleles it built. Only pairs are built: an allele split into three or more records is matched by its pairs only.
Expand Down
Loading
Loading