Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
f453318
fix(mocha): send every autosome to the phase gate, panel file or not
GeiserX Nov 17, 2026
5e02dc2
test(e2e): check that MoChA's unpanelled autosomes are named, not dro…
GeiserX Nov 17, 2026
a2703b0
fix(lookup): read the whole VCF when its index is older than the file
GeiserX Nov 17, 2026
0a8c5c6
docs(lookup): say an index older than its file is not used
GeiserX Nov 17, 2026
f9c6165
fix(setup): list what failed to install and exit 3 instead of claimin…
GeiserX Nov 17, 2026
f83b336
docs(setup): document setup.sh's exit statuses and the missing list
GeiserX Nov 17, 2026
06cf3df
fix(io): create og's temporary output files new and owner-only
GeiserX Nov 17, 2026
80fd494
docs: say og writes owner-only files and keep --outdir private
GeiserX Nov 17, 2026
fd3e225
test(e2e): count MoChA's chr20 calls only on HG002, now that it reads…
GeiserX Nov 17, 2026
e2af95e
test(setup): expect the partial status from fake setups whose downloa…
GeiserX Nov 17, 2026
2817f59
docs: say og's atomic writer, not every og write, is owner-only
GeiserX Nov 17, 2026
5ed22f2
fix(lookup): stream past an older index only when it lacks the chromo…
GeiserX Nov 17, 2026
a89415b
Merge remote-tracking branch 'origin/main' into pgp/i411-mocha-unpane…
GeiserX Nov 18, 2026
5b78194
Merge remote-tracking branch 'origin/main' into pgp/rv-lookup-stale-i…
GeiserX Nov 18, 2026
ce6a424
Merge remote-tracking branch 'origin/main' into pgp/rv-io-owner-only-…
GeiserX Nov 18, 2026
eee0ee9
Merge remote-tracking branch 'origin/main' into pgp/rv-setup-partial-…
GeiserX Nov 18, 2026
8cef07b
Merge #479 rv-lookup-stale-index into the batch d tip
GeiserX Nov 18, 2026
a283ae5
Merge #486 rv-io-owner-only-writes into the batch d tip
GeiserX Nov 18, 2026
a9019ec
Merge #465 i411-mocha-unpanelled-autosomes into the batch d tip
GeiserX Nov 18, 2026
23dd4d2
fix(io): give og's owner-only files to the work folder's owner when o…
GeiserX Nov 18, 2026
6dbf474
Merge #486 rv-io-owner-only-writes (fixed) into the batch d tip
GeiserX Nov 18, 2026
c51194d
Merge origin/main into the batch d tip
GeiserX Nov 18, 2026
5fa9969
Keep one CHANGELOG entry for #486 in the batch d tip
GeiserX Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,14 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- `setup.sh` no longer says "Setup complete!" after a download failed. Before, a failed or skipped install (AnnotSV's data, a VEP cache, the CPSR bundle, somalier's sites, the STRchive catalog, the tandem-repeat map, gnomAD's chrM sites, a pinned data file, the sequence dictionary or og) printed a warning, then the same banner and exit status 0 as a full setup. Now setup keeps a list of what it could not install. When the list is empty it prints "Setup complete!" and exits 0 as before. Otherwise it prints "Setup finished: core ready, N items missing", lists each item with the command that retries it (`--annotsv-data`, `--vep-cache`, `--clinical-data`, `--sample-qc-data`, `--strchive-data`, `--og`, or setup again), and exits 3. A script that runs `setup.sh` and checks its status now sees a partial setup as one. `--no-clinical-data` still ends complete: skipping the clinical data is a choice. [Getting started](docs/getting-started.md) lists the three exit statuses.
- `og lookup` no longer trusts an index older than its VCF for a chromosome the index does not list. A gVCF or annotated VCF replaced by hand after its `.tbi` or `.csi` was built kept the old index, and a chromosome the new file added was missing from it, so a variant called there read as "no call". The lookup now compares the two modification times, as the blood-type step does. When the index is older and does not list the queried chromosome, it reads the whole file and prints `NOTE: the index ... is older than ... and does not list chrN; reading the whole file`. An older index that lists the chromosome is still read: in a Nextflow output directory the small index is often copied before its large VCF, so it carries the earlier time. A fresh index is used as before, and no answer or output file changes.
- The files `og` writes through its atomic writer (every analysis output) are owner-only (mode 0600) whatever the umask. Before, `og` wrote each output under a temporary name with the umask's mode, so under the common umask 022 an output was readable by every user of the machine, and replacing a 0600 file made it 0644. The step scripts set umask 077, so a run through them was already owner-only; `og` run by hand or by the Nextflow pipeline was not. Now the temporary file is created mode 0600, or with the mode of the file it replaces when that is stricter (a 0400 file stays 0400). It is always a new file: a file or link already at its name is left as it is and the next name is tried, and two outputs written at once to one name in one process get two temporary files. The temporary name gains a number, `.<name>.tmp-<pid>-<n>`. When `og` runs as root, each of these files goes to the owner and group of the nearest folder above it that root does not own; when every folder up to `/` is root's, nothing changes. The docker profile of the Nextflow pipeline runs its containers as root over a work folder the host user owns; Nextflow's `docker.fixOwnership` does not reach these files, because the biocontainers images ship a `chown` without `--from`. Without the hand-over the host user could not read an output `og` wrote there, and Nextflow could not publish it. No output name or content changes. [Nextflow Execution](docs/nextflow.md#run-the-pipeline) now says to keep `--outdir` private.
- A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again.
- VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37).
- The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2).
- The live site's access log no longer names the finding a page opens. A finding's id is built from its gene and position, and the finding route wrote that id into the log's section column, though `docs/live-site.md` says the log never holds a result. The row now holds `-` there, as the other result routes do. The log keeps its columns and one row per page open. An og-live cargo test opens a finding page on a made-up run and reads the log; its control puts the old line back and must fail.
- Step 51 (MoChA) no longer drops an autosome its phasing panel has no file for. With a panel cut to regions (`MOCHA_REGIONS`), such an autosome was cut out before MoChA ran, so it had no row in the sites table, the phase check could not see it, and the sample could read "assessed" with that autosome never looked at and no word about it. Now its sites go to MoChA unphased and keep their row in `_mocha_sites.tsv`, so the phase check marks the sample not assessed and the message names the autosome with its 0 phased sites. The step also prints a warning that names it. This holds for `scripts/51-mocha.sh` and the Nextflow `MOCHA_PHASE` and `MOCHA_CALL` processes. No file, column or summary row changes; the `chromosomes` row of such a sample now lists the unpanelled autosomes too. Part of #411.
- The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes.
- `og report render` now writes the sections of the opt-in steps and of two small default ones in Rust: clonal haematopoiesis with its detection limit (step 49), viral DNA (step 47), mobile element insertions (step 50), MoChA's mosaic alterations (step 51), VNtyper (step 52), Locityper (step 53), the Exomiser ranking (step 48), telomere content and runs of homozygosity. Both reports match `bin/render_report.py` byte for byte on the parity corpus, except for these changes:
- When the BAM has no unplaced unmapped reads, step 47 looked at no viral genome. The viral section now says the genomes were not assessed. The Python said "No read on any of the 24 viral genomes", which reads as a negative result.
Expand Down Expand Up @@ -53,7 +57,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n
- The step's stdout line lists each below-threshold genome with its count, and says "no reads above the threshold at this depth" when nothing counts.
- The docs say how many reads the 10% takes on each genome size, and what a negative leaves open.
- The `viral-report` parity cases are retired: `bin/viral_report.py` is gone, and every case's output changes on purpose here. The Rust tests in `rust/crates/og-reports` and `tests/test_viral_report.py` hold the same cases.
- Step 51 (MoChA) now reads "not assessed" when too few heterozygous sites are phased to see a mosaic event. It read "not assessed" only when no site at all was phased, so a sample with one phased site in the whole genome, or with one autosome left mostly unphased by a panel cut to regions, read "assessed" with no call, and a real 6% CN-LOH above the floor went unreported. The sample is now not assessed when an autosome has under 0.80 of its heterozygous sites phased, and the message names each such autosome. An autosome the panel has no file for is still left out of the step before this check, so it is not assessed and the report does not say so. The minimum comes from two public genomes, HG002 and HG005, where every autosome keeps 0.933 to 0.975 of its heterozygous sites phased by the full panel; `docs/51-mocha.md` has the table. The page also says that MoChA's call rate is always 1 on a WGS VCF, so the call-rate half of its sample filter never fires there. The summary file keeps its rows; only the status and message of such a sample change. Part of #411.
- Step 51 (MoChA) now reads "not assessed" when too few heterozygous sites are phased to see a mosaic event. It read "not assessed" only when no site at all was phased, so a sample with one phased site in the whole genome, or with one autosome left mostly unphased by a panel cut to regions, read "assessed" with no call, and a real 6% CN-LOH above the floor went unreported. The sample is now not assessed when an autosome has under 0.80 of its heterozygous sites phased, and the message names each such autosome. The minimum comes from two public genomes, HG002 and HG005, where every autosome keeps 0.933 to 0.975 of its heterozygous sites phased by the full panel; `docs/51-mocha.md` has the table. The page also says that MoChA's call rate is always 1 on a WGS VCF, so the call-rate half of its sample filter never fires there. The summary file keeps its rows; only the status and message of such a sample change. Part of #411.
- ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format.
- A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals.
- The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup.
Expand Down
8 changes: 4 additions & 4 deletions docs/51-mocha.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ With `run-all.sh`: `TOOLS=...,mocha`; a run lists step 51 only when `TOOLS` name
## How it works

1. The VCF's biallelic PASS SNVs on chr1 to chr22. A genotype under 10 reads or with GQ under 20 is set missing, as the MoChA README does for WGS. `mochatools` adds the GC content around each site from the reference.
2. Beagle phases the genotypes against the panel, one chromosome at a time (`impute=false`, `seed=1`). A chromosome the panel lacks is left out.
3. The phased genotypes go back onto the sites of step 1, and MoChA runs in its WGS mode (allele depths from AD) with `--genome GRCh38` and `cnps.bed` as `--cnp`. The MHC and KIR regions are left out of its model.
2. Beagle phases the genotypes against the panel, one chromosome at a time (`impute=false`, `seed=1`). A chromosome the panel has no file for is not phased, and the step prints a warning that names it.
3. The phased genotypes go back onto the sites of step 1, on every chromosome, and MoChA runs in its WGS mode (allele depths from AD) with `--genome GRCh38` and `cnps.bed` as `--cnp`. The MHC and KIR regions are left out of its model.
4. `og mocha report` reads MoChA's tables and decides which calls are reported.

A call is reported when all of these hold:
Expand All @@ -71,7 +71,7 @@ The sample is **not assessed** when MoChA's sample filter of the same README fai

**The call rate on WGS.** MoChA reads a WGS VCF by its allele depths (AD). In that mode it drops a site with a missing genotype before it counts the sites, so the call rate in its stats is 1 on every WGS VCF, whatever step 1's depth and GQ filter set missing. The call-rate half of the sample filter therefore never fires on this step's input. It applies only to a stats table that carries a real call rate. On WGS the phased BAF autocorrelation, the phased share below and the floor are what gate the sample.

**The phased share.** MoChA follows a haplotype only across phased heterozygous sites. An unphased site adds nothing to its phased model and adds 0 to its phased BAF autocorrelation, so a chromosome that is mostly unphased passes that filter and shows no event, even one above the floor. Beagle phases only the sites the panel holds, so the share of an autosome's heterozygous sites that end up phased is the share the panel holds. A panel cut with `MOCHA_REGIONS`, or a `MOCHA_PHASED_VCF` that is only partly phased, leaves whole stretches unphased. One gap stays: with Beagle, an autosome the panel has no file for (one a `MOCHA_REGIONS` file does not name) is left out of the step before MoChA runs. That autosome has no row in the sites table, so this check cannot see it, and the step does not assess it and does not say so.
**The phased share.** MoChA follows a haplotype only across phased heterozygous sites. An unphased site adds nothing to its phased model and adds 0 to its phased BAF autocorrelation, so a chromosome that is mostly unphased passes that filter and shows no event, even one above the floor. Beagle phases only the sites the panel holds, so the share of an autosome's heterozygous sites that end up phased is the share the panel holds. A panel cut with `MOCHA_REGIONS`, or a `MOCHA_PHASED_VCF` that is only partly phased, leaves whole stretches unphased. An autosome the panel has no file for (one a `MOCHA_REGIONS` file does not name) keeps its sites unphased and its row in the sites table, so this check names it with 0 phased sites and the sample is not assessed.

We measured that share on two public genomes, with step 1's filters (biallelic PASS SNVs, at least 10 reads and GQ 20) and the panel's sites (biallelic SNVs seen twice or more, the same position and alleles):

Expand Down Expand Up @@ -130,4 +130,4 @@ Not measured on a whole genome. Beagle's phasing of the 22 chromosomes takes mos
- A constitutional event (one every cell carries, such as a germline deletion) has a cell fraction near 1 and is left out by MoChA's filter; this step is not a test for those.
- A germline copy-number polymorphism is left out by type: MoChA types a call over a region of `cnps.bed` as `CNP_Loss` or `CNP_Gain`. A germline CNV missing from that list (rare, or under 10 kb) gets no such type. Short events (under a few Mb) are hard to tell from these and are reported only when MoChA's filter keeps them.
- Calls are labelled by chromosome arm, not by gene: an mCA spans millions of bases.
- The e2e case (`tests/e2e/mocha-1.sh`) runs the step on HG002's chr20 slice (MoChA makes no call; with about 500 heterozygous sites the sample filter can mark the slice not assessed, since the BAF autocorrelation it reads is meant for a whole genome), and on a synthetic phased VCF with a 20 Mb CN-LOH on 20p in 15% of the cells (one CN-LOH call, cell fraction 0.10 to 0.20) beside the same VCF with no shift (no call). Nothing in the case is a finding about the fixture's sample.
- The e2e case (`tests/e2e/mocha-1.sh`) runs the step on HG002's slices with the panel cut to the chr20 slice (MoChA makes no call on chr20; the fixture's other autosomes have no panel file, so they stay unphased and the sample is not assessed, with each of them named; a call MoChA makes on one of those slices, such as a germline CNV like the chr4 UGT2B17 deletion, is untyped because the cut panel's `cnps.bed` holds chr20 only, and is reported "no" because the sample is not assessed), and on a synthetic phased VCF with a 20 Mb CN-LOH on 20p in 15% of the cells (one CN-LOH call, cell fraction 0.10 to 0.20) beside the same VCF with no shift (no call). Nothing in the case is a finding about the fixture's sample.
10 changes: 10 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,16 @@ Use `./scripts/setup.sh --no-clinical-data "$GENOME_DIR"` to opt out.
Use `./scripts/setup.sh --clinical-data "$GENOME_DIR"` to add or retry only these data later.
`run-all.sh` skips steps whose data are still missing; inspect its plan and setup warnings.

Setup ends in one of three ways:

| Exit status | Last banner | What it means |
| --- | --- | --- |
| 0 | `Setup complete!` | Everything is installed. `--no-clinical-data` skips the clinical data on purpose, so that run can still end here. |
| 3 | `Setup finished: core ready, N items missing` | The reference, ClinVar and the images are in place, but at least one download, check or build failed or was skipped. Under `Missing` the summary lists each item and the command that retries it, such as `./scripts/setup.sh --annotsv-data "$GENOME_DIR"`. The steps that need an item say so when they run. |
| 1 | an error message, no banner | A core part failed (the reference, ClinVar or an image pull), or the arguments were wrong. Fix the error and run setup again. |

The PGS Catalog scores, pgsc_calc and its Nextflow plugin are not on the list: step 25 fetches what is missing when it runs.

## Platform Notes

### Linux (Recommended)
Expand Down
2 changes: 1 addition & 1 deletion docs/lookup.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Two ClinVar lines can appear. `ClinVar (pipeline's ClinVar file)` comes from the

## Runtime

One container starts for the whole command, whatever the number of queries. With indexed files a site or a gene takes well under a second inside it: only the region is read. An rsID that is in neither the site table nor the pipeline tables makes the command read the whole annotated VCF once, for all such rsIDs of the command line together, on two cores. A file without a `.tbi` or `.csi` index is streamed whole for each region, with a note: the answer is the same and only the records in the region are kept in memory, but on a whole genome it takes minutes. `bcftools index -t` makes it fast.
One container starts for the whole command, whatever the number of queries. With indexed files a site or a gene takes well under a second inside it: only the region is read. An rsID that is in neither the site table nor the pipeline tables makes the command read the whole annotated VCF once, for all such rsIDs of the command line together, on two cores. A file without a `.tbi` or `.csi` index is streamed whole for each region, with a note: the answer is the same and only the records in the region are kept in memory, but on a whole genome it takes minutes. `bcftools index -t` makes it fast. An index older than its file, as when the file was replaced after the index was built, may lack a chromosome the new file has. So when such an index does not list the queried chromosome, the file is streamed whole, with a note; `bcftools index -f -t` rebuilds the index. An older index that lists the chromosome is still used: a Nextflow output directory often holds an index dated a little before its file, because the small index finishes copying first.

## Notes

Expand Down
Loading
Loading