From f453318b80d6dbdc47ec56b9051bec737104fee3 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:18:01 +0100 Subject: [PATCH 01/14] fix(mocha): send every autosome to the phase gate, panel file or not With a panel cut to regions, an autosome the panel had no file for was cut out by `-r` before MoChA ran. It had no row in the sites table, so the phase gate of og mocha report could not see it, and the sample read assessed with that autosome never looked at. The bash step and the Nextflow MOCHA_CALL process now put the phased genotypes onto every prepared site, so such an autosome reaches MoChA unphased, keeps its sites row, and the gate marks the sample not assessed and names it. Both paths print a warning that names it. Bead: i411-mocha-unpanelled-autosomes --- CHANGELOG.md | 3 ++- docs/51-mocha.md | 8 ++++---- modules/local/mocha/main.nf | 9 +++++---- scripts/51-mocha.sh | 21 ++++++++++++--------- 4 files changed, 23 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05c3388e1..4c2759ad5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- Step 51 (MoChA) no longer drops an autosome its phasing panel has no file for. With a panel cut to regions (`MOCHA_REGIONS`), such an autosome was cut out before MoChA ran, so it had no row in the sites table, the phase check could not see it, and the sample could read "assessed" with that autosome never looked at and no word about it. Now its sites go to MoChA unphased and keep their row in `_mocha_sites.tsv`, so the phase check marks the sample not assessed and the message names the autosome with its 0 phased sites. The step also prints a warning that names it. This holds for `scripts/51-mocha.sh` and the Nextflow `MOCHA_PHASE` and `MOCHA_CALL` processes. No file, column or summary row changes; the `chromosomes` row of such a sample now lists the unpanelled autosomes too. Part of #411. - The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes. - The phased family VCF is indexed by the Rust `og` binary. WhatsHap's step (step 38 and the Nextflow WHATSHAP_PED process) wrote `family.phased.vcf.gz.tbi` with a Python one-liner, `pysam.tabix_index`. It now runs `og vcf index -f`. The `.tbi` keeps its name and format, and its bytes, once inflated, are the ones htslib 1.24 writes (the htslib of the pysam and bcftools the pipeline pins): the same bins in the same order, the same chunks and the same linear index. Only the deflate stream can differ, as two deflate libraries differ. On a 5-million-record three-sample VCF it took 0.75 to 0.91 s and 31 MB at peak, where htslib's own `tabix` took 0.81 to 0.93 s and 19 MB. - As in htslib 1.24, a record ends at the largest of INFO/END, the REF length, the SVLEN of a ``, ``, `` or `` ALT, and a sample's LEN on a gVCF block. A line with no position to read, unsorted records, a sequence that starts again, a record past 2^29 and an empty file stop the index with a message, and nothing is written. Without `-f` it never writes over an index, even one that appears while it runs. @@ -43,7 +44,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n - The step's stdout line lists each below-threshold genome with its count, and says "no reads above the threshold at this depth" when nothing counts. - The docs say how many reads the 10% takes on each genome size, and what a negative leaves open. - The `viral-report` parity cases are retired: `bin/viral_report.py` is gone, and every case's output changes on purpose here. The Rust tests in `rust/crates/og-reports` and `tests/test_viral_report.py` hold the same cases. -- Step 51 (MoChA) now reads "not assessed" when too few heterozygous sites are phased to see a mosaic event. It read "not assessed" only when no site at all was phased, so a sample with one phased site in the whole genome, or with one autosome left mostly unphased by a panel cut to regions, read "assessed" with no call, and a real 6% CN-LOH above the floor went unreported. The sample is now not assessed when an autosome has under 0.80 of its heterozygous sites phased, and the message names each such autosome. An autosome the panel has no file for is still left out of the step before this check, so it is not assessed and the report does not say so. The minimum comes from two public genomes, HG002 and HG005, where every autosome keeps 0.933 to 0.975 of its heterozygous sites phased by the full panel; `docs/51-mocha.md` has the table. The page also says that MoChA's call rate is always 1 on a WGS VCF, so the call-rate half of its sample filter never fires there. The summary file keeps its rows; only the status and message of such a sample change. Part of #411. +- Step 51 (MoChA) now reads "not assessed" when too few heterozygous sites are phased to see a mosaic event. It read "not assessed" only when no site at all was phased, so a sample with one phased site in the whole genome, or with one autosome left mostly unphased by a panel cut to regions, read "assessed" with no call, and a real 6% CN-LOH above the floor went unreported. The sample is now not assessed when an autosome has under 0.80 of its heterozygous sites phased, and the message names each such autosome. The minimum comes from two public genomes, HG002 and HG005, where every autosome keeps 0.933 to 0.975 of its heterozygous sites phased by the full panel; `docs/51-mocha.md` has the table. The page also says that MoChA's call rate is always 1 on a WGS VCF, so the call-rate half of its sample filter never fires there. The summary file keeps its rows; only the status and message of such a sample change. Part of #411. - ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format. - A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals. - The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup. diff --git a/docs/51-mocha.md b/docs/51-mocha.md index bb351a4aa..e15744040 100644 --- a/docs/51-mocha.md +++ b/docs/51-mocha.md @@ -50,8 +50,8 @@ With `run-all.sh`: `TOOLS=...,mocha`; a run lists step 51 only when `TOOLS` name ## How it works 1. The VCF's biallelic PASS SNVs on chr1 to chr22. A genotype under 10 reads or with GQ under 20 is set missing, as the MoChA README does for WGS. `mochatools` adds the GC content around each site from the reference. -2. Beagle phases the genotypes against the panel, one chromosome at a time (`impute=false`, `seed=1`). A chromosome the panel lacks is left out. -3. The phased genotypes go back onto the sites of step 1, and MoChA runs in its WGS mode (allele depths from AD) with `--genome GRCh38` and `cnps.bed` as `--cnp`. The MHC and KIR regions are left out of its model. +2. Beagle phases the genotypes against the panel, one chromosome at a time (`impute=false`, `seed=1`). A chromosome the panel has no file for is not phased, and the step prints a warning that names it. +3. The phased genotypes go back onto the sites of step 1, on every chromosome, and MoChA runs in its WGS mode (allele depths from AD) with `--genome GRCh38` and `cnps.bed` as `--cnp`. The MHC and KIR regions are left out of its model. 4. `og mocha report` reads MoChA's tables and decides which calls are reported. A call is reported when all of these hold: @@ -71,7 +71,7 @@ The sample is **not assessed** when MoChA's sample filter of the same README fai **The call rate on WGS.** MoChA reads a WGS VCF by its allele depths (AD). In that mode it drops a site with a missing genotype before it counts the sites, so the call rate in its stats is 1 on every WGS VCF, whatever step 1's depth and GQ filter set missing. The call-rate half of the sample filter therefore never fires on this step's input. It applies only to a stats table that carries a real call rate. On WGS the phased BAF autocorrelation, the phased share below and the floor are what gate the sample. -**The phased share.** MoChA follows a haplotype only across phased heterozygous sites. An unphased site adds nothing to its phased model and adds 0 to its phased BAF autocorrelation, so a chromosome that is mostly unphased passes that filter and shows no event, even one above the floor. Beagle phases only the sites the panel holds, so the share of an autosome's heterozygous sites that end up phased is the share the panel holds. A panel cut with `MOCHA_REGIONS`, or a `MOCHA_PHASED_VCF` that is only partly phased, leaves whole stretches unphased. One gap stays: with Beagle, an autosome the panel has no file for (one a `MOCHA_REGIONS` file does not name) is left out of the step before MoChA runs. That autosome has no row in the sites table, so this check cannot see it, and the step does not assess it and does not say so. +**The phased share.** MoChA follows a haplotype only across phased heterozygous sites. An unphased site adds nothing to its phased model and adds 0 to its phased BAF autocorrelation, so a chromosome that is mostly unphased passes that filter and shows no event, even one above the floor. Beagle phases only the sites the panel holds, so the share of an autosome's heterozygous sites that end up phased is the share the panel holds. A panel cut with `MOCHA_REGIONS`, or a `MOCHA_PHASED_VCF` that is only partly phased, leaves whole stretches unphased. An autosome the panel has no file for (one a `MOCHA_REGIONS` file does not name) keeps its sites unphased and its row in the sites table, so this check names it with 0 phased sites and the sample is not assessed. We measured that share on two public genomes, with step 1's filters (biallelic PASS SNVs, at least 10 reads and GQ 20) and the panel's sites (biallelic SNVs seen twice or more, the same position and alleles): @@ -130,4 +130,4 @@ Not measured on a whole genome. Beagle's phasing of the 22 chromosomes takes mos - A constitutional event (one every cell carries, such as a germline deletion) has a cell fraction near 1 and is left out by MoChA's filter; this step is not a test for those. - A germline copy-number polymorphism is left out by type: MoChA types a call over a region of `cnps.bed` as `CNP_Loss` or `CNP_Gain`. A germline CNV missing from that list (rare, or under 10 kb) gets no such type. Short events (under a few Mb) are hard to tell from these and are reported only when MoChA's filter keeps them. - Calls are labelled by chromosome arm, not by gene: an mCA spans millions of bases. -- The e2e case (`tests/e2e/mocha-1.sh`) runs the step on HG002's chr20 slice (MoChA makes no call; with about 500 heterozygous sites the sample filter can mark the slice not assessed, since the BAF autocorrelation it reads is meant for a whole genome), and on a synthetic phased VCF with a 20 Mb CN-LOH on 20p in 15% of the cells (one CN-LOH call, cell fraction 0.10 to 0.20) beside the same VCF with no shift (no call). Nothing in the case is a finding about the fixture's sample. +- The e2e case (`tests/e2e/mocha-1.sh`) runs the step on HG002's slices with the panel cut to the chr20 slice (MoChA makes no call; the fixture's other autosomes have no panel file, so they stay unphased and the sample is not assessed, with each of them named), and on a synthetic phased VCF with a 20 Mb CN-LOH on 20p in 15% of the cells (one CN-LOH call, cell fraction 0.10 to 0.20) beside the same VCF with no shift (no call). Nothing in the case is a finding about the fixture's sample. diff --git a/modules/local/mocha/main.nf b/modules/local/mocha/main.nf index 05bdf1b98..c02c38c38 100644 --- a/modules/local/mocha/main.nf +++ b/modules/local/mocha/main.nf @@ -9,7 +9,9 @@ from the reference (bcftools +mochatools) MOCHA_PHASE Beagle against the panel, one chromosome at a time (those the panel has), with the panel's cnps.bed alongside - MOCHA_CALL the phased genotypes onto the prepared sites, MoChA + MOCHA_CALL the phased genotypes onto the prepared sites of every + chromosome (one the panel lacks stays unphased, so the + phase check of og mocha report names it), MoChA (bcftools +mocha, the common germline CNVs as --cnp), and the per-chromosome site counts MOCHA_REPORT og mocha report (bin/og): the calls above the floor @@ -91,7 +93,7 @@ process MOCHA_PHASE { [ -f ${mocha_data}/cnps.bed ] || { echo "ERROR: ${mocha_data}/cnps.bed not found: the panel is older than its CNV list (rerun setup.sh --mocha-data)" >&2; exit 1; } cp ${mocha_data}/cnps.bed phase/cnps.bed for c in \$(cat ${chroms}); do - [ -s ${mocha_data}/panel/\$c.vcf.gz ] || continue + [ -s ${mocha_data}/panel/\$c.vcf.gz ] || { echo "[WARN] The panel has no file for \$c: not phased, so a heterozygous site there makes the sample not assessed"; continue; } beagle -Xmx${heap}g gt=${phase_in} chrom=\$c ref=${mocha_data}/panel/\$c.vcf.gz map=${mocha_data}/maps/\$c.map \\ out=phase/\$c impute=false seed=1 nthreads=${task.cpus} > phase/\$c.out 2>&1 \\ || { tail -n 20 phase/\$c.out; exit 1; } @@ -140,7 +142,6 @@ process MOCHA_CALL { def p = meta.id """ set -euo pipefail - chroms=\$(paste -sd, ${phase}/chroms.txt) # Beagle writes no ##contig lines and concat refuses records on an # undeclared contig: give each Beagle VCF the contig lines of prep.bcf. bcftools view -h ${prep} | grep '^##contig=' > contigs.hdr @@ -152,7 +153,7 @@ process MOCHA_CALL { done bcftools concat --no-version -Ob -o phased_gt.bcf "\${f[@]}" bcftools index -f phased_gt.bcf - bcftools annotate --no-version -a phased_gt.bcf -c FMT/GT -r "\$chroms" -Ob -o phased.bcf ${prep} + bcftools annotate --no-version -a phased_gt.bcf -c FMT/GT -Ob -o phased.bcf ${prep} bcftools index -f phased.bcf bcftools +mocha --genome GRCh38 --no-version --mhc chr6:27518932-33480487 --kir chr19:54071493-54992731 --cnp ${phase}/cnps.bed \\ --calls ${p}_mocha_calls_raw.tsv --stats ${p}_mocha_stats.tsv --log ${p}_mocha.log phased.bcf diff --git a/scripts/51-mocha.sh b/scripts/51-mocha.sh index 3063c8cbc..d26b4f52f 100755 --- a/scripts/51-mocha.sh +++ b/scripts/51-mocha.sh @@ -86,12 +86,14 @@ run_in --cpus 2 --memory 4g "$MOCHA_IMAGE" bash -euo pipefail -c ' CHROMS=$(tr '\n' ' ' < "${W}/chroms.txt") [ -n "${CHROMS// /}" ] || { echo "ERROR: no biallelic PASS SNV on chr1 to chr22 in ${VCF}" >&2; exit 1; } -# 2. Beagle against the panel, one chromosome at a time; a chromosome the panel -# lacks (a panel cut to regions) is left out. +# 2. Beagle against the panel, one chromosome at a time. A chromosome the panel +# lacks (a panel cut to regions) is not phased, but it still goes on to MoChA +# and the sites table, so og mocha report's phase check names it. if [ "$PHASING" = beagle ]; then - PHASE="" - for c in $CHROMS; do [ -s "${DATA}/panel/${c}.vcf.gz" ] && PHASE+="${c} "; done + PHASE="" NOPANEL="" + for c in $CHROMS; do if [ -s "${DATA}/panel/${c}.vcf.gz" ]; then PHASE+="${c} "; else NOPANEL+="${c} "; fi; done [ -n "$PHASE" ] || { echo "ERROR: the panel has none of the VCF's chromosomes (${CHROMS})" >&2; exit 1; } + [ -z "$NOPANEL" ] || echo "[WARN] The panel has no file for ${NOPANEL% }: not phased, so a heterozygous site there makes the sample not assessed" echo "[2/4] Phasing with Beagle against the panel (${PHASE% })..." # shellcheck disable=SC2016 # $1 to $4 belong to the inner bash run_in --cpus "$THREADS" --memory 16g "$BEAGLE_IMAGE" bash -euo pipefail -c ' @@ -104,10 +106,11 @@ if [ "$PHASING" = beagle ]; then tr ' ' '\n' <<< "${PHASE% }" > "${W}/phased_chroms.txt" else echo "[2/4] The VCF is already phased: no Beagle run." - cp "${W}/chroms.txt" "${W}/phased_chroms.txt" fi -# 3. The phased genotypes onto the prepared sites, then MoChA (WGS mode: AD). +# 3. The phased genotypes onto the prepared sites of every chromosome, then +# MoChA (WGS mode: AD). A chromosome Beagle did not phase keeps its unphased +# genotypes, so its row in the sites table shows its phased share. # SITES_AWK sums each chromosome: first and last position, sites, hets, phased hets. # shellcheck disable=SC2016 # awk's fields, not shell variables SITES_AWK='$1 != c { if (c != "") print c, a, b, n, h, ph; c = $1; a = $2; n = h = ph = 0 } @@ -116,7 +119,7 @@ SITES_AWK='$1 != c { if (c != "") print c, a, b, n, h, ph; c = $1; a = $2; n = h echo "[3/4] MoChA (bcftools +mocha, ${MOCHA_VERSION})..." # shellcheck disable=SC2016 # $1 to $7 belong to the inner bash run_in --cpus 2 --memory 8g "$MOCHA_IMAGE" bash -euo pipefail -c ' - w=$1; chroms=$(paste -sd, "$w/phased_chroms.txt") + w=$1 if [ "$2" = beagle ]; then # Beagle writes no ##contig lines and concat refuses records on an # undeclared contig: give each Beagle VCF the contig lines of prep.bcf. @@ -129,9 +132,9 @@ run_in --cpus 2 --memory 8g "$MOCHA_IMAGE" bash -euo pipefail -c ' done bcftools concat --no-version -Ob -o "$w/phased_gt.bcf" "${f[@]}" bcftools index -f "$w/phased_gt.bcf" - bcftools annotate --no-version -a "$w/phased_gt.bcf" -c FMT/GT -r "$chroms" -Ob -o "$w/phased.bcf" "$w/prep.bcf" + bcftools annotate --no-version -a "$w/phased_gt.bcf" -c FMT/GT -Ob -o "$w/phased.bcf" "$w/prep.bcf" else - bcftools view --no-version -r "$chroms" -Ob -o "$w/phased.bcf" "$w/prep.bcf" + bcftools view --no-version -Ob -o "$w/phased.bcf" "$w/prep.bcf" fi bcftools index -f "$w/phased.bcf" cnp=(); [ -z "$7" ] || cnp=(--cnp "$7") From 5e02dc2ad504640c831bf2efefd10bdf513f6ebe Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:18:01 +0100 Subject: [PATCH 02/14] test(e2e): check that MoChA's unpanelled autosomes are named, not dropped HG002's fixture holds slices on several autosomes, and the e2e panel is cut to the chr20 slice. The case asserted that MoChA read chr20 only, which is the bug. It now checks that the other autosomes reach the sites table with 0 phased sites, that the sample is not assessed, and that the message names each of them; a red control runs the same check on the sites table cut to chr20, which is all the step kept before. The Nextflow mirror also compares the summary message. Bead: i411-mocha-unpanelled-autosomes --- tests/e2e/mocha-1.sh | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/tests/e2e/mocha-1.sh b/tests/e2e/mocha-1.sh index 31bfdc235..2253cf8c8 100755 --- a/tests/e2e/mocha-1.sh +++ b/tests/e2e/mocha-1.sh @@ -11,7 +11,10 @@ # (the panel's common CNVs) written, in MoChA's --cnp format when not empty; # - HG002 (case 21's VCF) phased by Beagle on that slice: phased hets, MoChA # makes no call and none is reported, and MoChA's log names cnps.bed as -# its --cnp list; +# its --cnp list. The fixture's other autosomes, which the cut panel has +# no file for, still reach the sites table, unphased, so the sample is not +# assessed and the message names each of them (red control: the same check +# fails on the sites table cut to chr20, all the step kept before); # - MOSAIC, a synthetic phased VCF on chr16, chr19, chr20 and chr22 at 30x # (REF bases from the fixture reference) with a 20 Mb CN-LOH on 20p from # the telomere in 15% of the cells (the haplotype balance 0.575 instead of @@ -42,6 +45,18 @@ not() { ! "$@"; } cnp_logged() { grep -Eq '^Regions to genotype: .*cnps\.bed$' "$1" 2>/dev/null; } cnp_format() { awk -F'\t' 'NF != 4 || $2 !~ /^[0-9]+$/ || $3 !~ /^[0-9]+$/ || $3 <= $2 || $4 !~ /^(CNV|DEL|DUP)$/ {exit 1}' "$1"; } status_ok() { [ "$1" = assessed ] || { [ "$1" = "not assessed" ] && [ -n "$2" ]; }; } +# unpanelled SITES MESSAGE: the sites table has an autosome other than chr20, +# the cut panel's one chromosome, with heterozygous sites; each such autosome +# has none phased, and the message names it as "chrN (0 of H)". +unpanelled() { + local c h p n=0 + while IFS=$'\t' read -r c _ _ _ h p; do + [ "$c" != chr20 ] && [ "${h:-0}" != 0 ] || continue + [ "$p" = 0 ] && [[ "$2" == *"${c} (0 of ${h})"* ]] || return 1 + n=$((n + 1)) + done < "$1" + [ "$n" -gt 0 ] +} retry() { local i; for i in 1 2 3 4; do "$@" && return 0; [ "$i" -lt 4 ] && sleep $((i * 15)); done; return 1; } show() { echo "--- $1"; cat "${G}/$1/mosaic/$1_mocha_summary.tsv" "${G}/$1/mosaic/$1_mocha.tsv" \ "${G}/$1/mosaic/$1_mocha_calls_raw.tsv" "${G}/$1/mosaic/$1_mocha_stats.tsv" "${G}/$1/mosaic/$1_mocha_sites.tsv" 2>/dev/null; } @@ -90,11 +105,18 @@ run_step 51-mocha.sh "$SAMPLE" check_step_exit 51-mocha.sh show "$SAMPLE" check_eq "HG002: phased by Beagle" "$(summ "$SAMPLE" phasing)" beagle -check_eq "HG002: MoChA read chr20, the panel's one chromosome" "$(summ "$SAMPLE" chromosomes)" chr20 +check "HG002: MoChA read chr20, the panel's one chromosome, and more ($(summ "$SAMPLE" chromosomes))" \ + has '^(chr[0-9]+,)+chr20(,chr[0-9]+)*$|^chr20(,chr[0-9]+)+$' "$(summ "$SAMPLE" chromosomes)" +SITES="${G}/${SAMPLE}/mosaic/${SAMPLE}_mocha_sites.tsv" +check_eq "HG002: the autosomes the panel has no file for make the sample not assessed" "$(summ "$SAMPLE" status)" "not assessed" +check "HG002: the message names each of them with 0 phased sites ($(summ "$SAMPLE" message))" \ + unpanelled "$SITES" "$(summ "$SAMPLE" message)" +awk -F'\t' '$1 == "chr20"' "$SITES" > "${T}/sites_chr20.tsv" +check "red control: that check fails on the sites table cut to chr20 (all the step kept before)" \ + not unpanelled "${T}/sites_chr20.tsv" "$(summ "$SAMPLE" message)" check_ge "HG002: phased heterozygous sites" "$(summ "$SAMPLE" phased_het_sites)" 100 -# 500 kb hold about 500 hets, too few for the genome-wide BAF autocorrelation -# MoChA's sample filter reads, so the status may be either state (with a -# reason); MoChA's own call count is what must be zero. +# The status is not assessed (the autosomes above), with its reason; MoChA's +# own call count is what must be zero. check "HG002: a status with its reason ($(summ "$SAMPLE" status), baf_auto $(summ "$SAMPLE" baf_auto))" \ status_ok "$(summ "$SAMPLE" status)" "$(summ "$SAMPLE" message)" check_eq "HG002 chr20: MoChA made no call" "$(summ "$SAMPLE" calls_mocha)" 0 @@ -155,7 +177,7 @@ A="${G}/${SAMPLE}/mosaic/${SAMPLE}_mocha.tsv" B="${NF_OUT}/${SAMPLE}/mosaic/${SA check "Nextflow wrote ${SAMPLE}_mocha.tsv" test -s "$B" check "Nextflow's ${SAMPLE}_mocha.tsv is the bash step's" cmp -s "$A" "$B" cmp -s "$A" "$B" || diff "$A" "$B" | head -20 -for k in status chromosomes het_sites depth het_per_10mb calls_mocha calls_reported floor_min phasing; do +for k in status message chromosomes het_sites depth het_per_10mb calls_mocha calls_reported floor_min phasing; do check_eq "Nextflow's summary: ${k} as the bash step's" \ "$(awk -F'\t' -v k="$k" '$1 == k {print $2}' "${NF_OUT}/${SAMPLE}/mosaic/${SAMPLE}_mocha_summary.tsv" 2>/dev/null)" "$(summ "$SAMPLE" "$k")" done From a2703b02d2d303f3ce1afcccb26875ac761f2591 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:45:53 +0100 Subject: [PATCH 03/14] fix(lookup): read the whole VCF when its index is older than the file An index built before the VCF was replaced can lack a contig the new file has; the indexed read then returned no records and the lookup answered no call. Compare the modification times, as og-bloodtype does, and stream the file with a note when the index is older. Bead: C15 --- rust/crates/og-lookup/src/vcf.rs | 35 +++-- .../og-lookup/tests/rv_lookup_stale_index.rs | 125 ++++++++++++++++++ 2 files changed, 150 insertions(+), 10 deletions(-) create mode 100644 rust/crates/og-lookup/tests/rv_lookup_stale_index.rs diff --git a/rust/crates/og-lookup/src/vcf.rs b/rust/crates/og-lookup/src/vcf.rs index 0177b9bae..67959cfee 100644 --- a/rust/crates/og-lookup/src/vcf.rs +++ b/rust/crates/og-lookup/src/vcf.rs @@ -287,20 +287,23 @@ pub struct Span<'a> { /// The records of `path` that overlap `span` (a reference block by its /// END, a deletion by its REF), in file order: through the `.tbi` or `.csi` -/// beside the file when there is one, else by streaming the whole file, -/// which `note` is told. +/// beside the file when there is one and it is not older than the file, +/// else by streaming the whole file, which `note` is told. pub fn read(path: &str, span: Span, note: &mut dyn Write) -> Result { { - let has_index = [".tbi", ".csi"] + let index = [".tbi", ".csi"] .iter() - .any(|x| Path::new(&format!("{path}{x}")).is_file()); - if !has_index { + .map(|x| format!("{path}{x}")) + .find(|p| Path::new(p).is_file()); + if let Some(index) = index.as_deref().filter(|i| older(i, path)) { + // A contig the file gained after its index was built is not in + // the index, so an indexed read would answer no call there. let _ = writeln!( note, - "NOTE: {path} has no .tbi index; reading the whole file (bcftools index -t makes this fast)." + "NOTE: the index {index} is older than {path}; reading the whole file (bcftools index -f -t rebuilds it)." ); - } else { - match read_indexed(path, span) { + } else if let Some(index) = index.as_deref() { + match read_indexed(path, index, span) { Ok(r) => return Ok(r), Err(e) => { let _ = writeln!( @@ -313,11 +316,23 @@ pub fn read(path: &str, span: Span, note: &mut dyn Write) -> Result bool { + let mtime = |p: &str| std::fs::metadata(p).and_then(|m| m.modified()).ok(); + matches!((mtime(index), mtime(path)), (Some(i), Some(v)) if i < v) +} + /// Keeps `r` when it overlaps `span`. fn overlaps(r: &Rec, span: Span) -> bool { r.chrom == span.chrom && r.pos <= span.end && r.end >= span.start @@ -325,7 +340,7 @@ fn overlaps(r: &Rec, span: Span) -> bool { /// The header through the top of the file, then each index sequence that /// names the region's chromosome, through the index. -fn read_indexed(path: &str, span: Span) -> Result { +fn read_indexed(path: &str, index: &str, span: Span) -> Result { let file = og_io::open(path).map_err(|e| Error::io(&e, path))?; let mut header = Vec::new(); for line in Lines::new(file, Utf8::Replace) { @@ -338,7 +353,7 @@ fn read_indexed(path: &str, span: Span) -> Result { } } let csq = Csq::from_header(header.iter().map(String::as_str)); - let mut reader = tabix::Reader::open(path).map_err(|e| Error::io(&e, path))?; + let mut reader = tabix::Reader::with_index(path, index).map_err(|e| Error::io(&e, path))?; let names: Vec = reader .index() .names() diff --git a/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs b/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs new file mode 100644 index 000000000..ab6e56c56 --- /dev/null +++ b/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs @@ -0,0 +1,125 @@ +//! An index older than its VCF is not trusted: a contig the VCF gained after +//! the index was built is not in the index, so an indexed read would answer +//! "no call" for a variant the file holds. The lookup reads the whole file +//! instead and says so; a fresh index is still used. + +use std::fs::{self, File}; +use std::io::Write; +use std::num::NonZero; +use std::path::{Path, PathBuf}; +use std::time::{Duration, SystemTime}; + +fn input(name: &str) -> String { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../../../tests/parity/inputs/lookup") + .join(name) + .to_string_lossy() + .into_owned() +} + +/// Runs `og lookup query chrM:73` over `gvcf`: stdout, stderr. +fn chrm_73(gvcf: &str) -> (String, String) { + let args: Vec = [ + "query", + "--sites", + &input("sites.tsv.gz"), + "--genes", + &input("gencode.v50.basic.genes.gtf"), + "--table", + &input("traits.tsv"), + "--gvcf", + gvcf, + "--", + "chrM:73", + ] + .map(String::from) + .to_vec(); + let (mut out, mut err) = (Vec::new(), Vec::new()); + let rc = og_lookup::cli(&args, &mut out, &mut err); + let (out, err) = ( + String::from_utf8(out).unwrap(), + String::from_utf8(err).unwrap(), + ); + assert_eq!(rc, 0, "{out}{err}"); + (out, err) +} + +/// Writes `text` to `path` as BGZF. +fn bgzip(path: &Path, text: &str) { + let mut w = + og_io::bgzf::Writer::new(File::create(path).unwrap(), NonZero::::MIN).unwrap(); + w.write_all(text.as_bytes()).unwrap(); + w.finish().unwrap(); +} + +/// Builds the `.tbi` of the BGZF VCF `vcf`. +fn index(vcf: &Path, tbi: &Path) { + let built = og_io::tabix_write::build(File::open(vcf).unwrap()).unwrap(); + built.write_tbi(tbi, true).unwrap(); +} + +/// Sets the modification time of `path` a minute after (or before) that of `of`. +fn touch(path: &Path, of: &Path, later: bool) { + let base = fs::metadata(of).unwrap().modified().unwrap(); + let minute = Duration::from_secs(60); + let t: SystemTime = if later { base + minute } else { base - minute }; + File::options() + .write(true) + .open(path) + .unwrap() + .set_modified(t) + .unwrap(); +} + +#[test] +fn an_index_older_than_its_vcf_is_not_trusted() { + let dir: PathBuf = Path::new(env!("CARGO_TARGET_TMPDIR")).join("rv_lookup_stale_index"); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let (gz, tbi) = (dir.join("g.vcf.gz"), dir.join("g.vcf.gz.tbi")); + let gz_s = gz.to_string_lossy().into_owned(); + let full = fs::read_to_string(input("g.vcf")).unwrap(); + assert!(full.contains("\nchrM\t73\t"), "the fixture calls chrM:73"); + + // The answer from the whole plain file. + let (want, _) = chrm_73(&input("g.vcf")); + assert!(want.contains("genotype: G/G"), "{want}"); + + // The index of an earlier VCF without chrM, then the VCF rewritten with it. + let old: String = full + .split_inclusive('\n') + .filter(|l| !l.starts_with("chrM\t")) + .collect(); + bgzip(&gz, &old); + index(&gz, &tbi); + let stale = fs::read(&tbi).unwrap(); + bgzip(&gz, &full); + touch(&tbi, &gz, false); + + let (out, err) = chrm_73(&gz_s); + assert_eq!(out, want, "the stale index hid the chrM call"); + assert!( + err.contains(&format!( + "NOTE: the index {} is older than {gz_s}; reading the whole file", + tbi.display() + )), + "{err}" + ); + + // The same index with a newer time is read, which shows the indexed path + // is the one taken: it lacks chrM, so the answer is not the call. + touch(&tbi, &gz, true); + let (out, err) = chrm_73(&gz_s); + assert_ne!(out, want, "a fresh index is read, not the whole file"); + assert!(err.is_empty(), "{err}"); + + // A fresh index of the current VCF answers the call, with no note. + index(&gz, &tbi); + assert_ne!(fs::read(&tbi).unwrap(), stale); + touch(&tbi, &gz, true); + let (out, err) = chrm_73(&gz_s); + assert_eq!(out, want); + assert!(err.is_empty(), "{err}"); + + fs::remove_dir_all(&dir).unwrap(); +} From 0a8c5c60958474f7c5edd23943970301b358ab81 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:45:53 +0100 Subject: [PATCH 04/14] docs(lookup): say an index older than its file is not used Bead: C15 --- CHANGELOG.md | 1 + docs/lookup.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05c3388e1..e424d9034 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- `og lookup` no longer trusts an index older than its VCF. A gVCF or annotated VCF replaced by hand after its `.tbi` or `.csi` was built kept the old index, and a chromosome the new file added was missing from it, so a variant called there read as "no call". The lookup now compares the two modification times, as the blood-type step already does. When the index is older it reads the whole file and prints `NOTE: the index ... is older than ...; reading the whole file`. A fresh index is used as before, and no answer or output file changes. - The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes. - The phased family VCF is indexed by the Rust `og` binary. WhatsHap's step (step 38 and the Nextflow WHATSHAP_PED process) wrote `family.phased.vcf.gz.tbi` with a Python one-liner, `pysam.tabix_index`. It now runs `og vcf index -f`. The `.tbi` keeps its name and format, and its bytes, once inflated, are the ones htslib 1.24 writes (the htslib of the pysam and bcftools the pipeline pins): the same bins in the same order, the same chunks and the same linear index. Only the deflate stream can differ, as two deflate libraries differ. On a 5-million-record three-sample VCF it took 0.75 to 0.91 s and 31 MB at peak, where htslib's own `tabix` took 0.81 to 0.93 s and 19 MB. - As in htslib 1.24, a record ends at the largest of INFO/END, the REF length, the SVLEN of a ``, ``, `` or `` ALT, and a sample's LEN on a gVCF block. A line with no position to read, unsorted records, a sequence that starts again, a record past 2^29 and an empty file stop the index with a message, and nothing is written. Without `-f` it never writes over an index, even one that appears while it runs. diff --git a/docs/lookup.md b/docs/lookup.md index d67587dc3..83b5e3d5d 100644 --- a/docs/lookup.md +++ b/docs/lookup.md @@ -87,7 +87,7 @@ Two ClinVar lines can appear. `ClinVar (pipeline's ClinVar file)` comes from the ## Runtime -One container starts for the whole command, whatever the number of queries. With indexed files a site or a gene takes well under a second inside it: only the region is read. An rsID that is in neither the site table nor the pipeline tables makes the command read the whole annotated VCF once, for all such rsIDs of the command line together, on two cores. A file without a `.tbi` or `.csi` index is streamed whole for each region, with a note: the answer is the same and only the records in the region are kept in memory, but on a whole genome it takes minutes. `bcftools index -t` makes it fast. +One container starts for the whole command, whatever the number of queries. With indexed files a site or a gene takes well under a second inside it: only the region is read. An rsID that is in neither the site table nor the pipeline tables makes the command read the whole annotated VCF once, for all such rsIDs of the command line together, on two cores. A file without a `.tbi` or `.csi` index is streamed whole for each region, with a note: the answer is the same and only the records in the region are kept in memory, but on a whole genome it takes minutes. `bcftools index -t` makes it fast. An index older than its file, as when the file was replaced after the index was built, is not used: it may lack a chromosome the new file has, so the file is streamed whole, with a note. `bcftools index -f -t` rebuilds the index. ## Notes From f9c6165ba0095ef9227f7429cbaf5ae6b233b195 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:50:20 +0100 Subject: [PATCH 05/14] fix(setup): list what failed to install and exit 3 instead of claiming setup is complete setup.sh swallowed the failures of its sample QC, STRchive, tandem-repeat, AnnotSV and clinical installers and of the phase 4 downloads, then printed "Setup complete!" and exited 0. A caller could not tell a partial setup from a full one. Each failure or skip now goes on a MISSING list with the command that retries it. The summary prints "Setup complete!" and exits 0 only when the list is empty; otherwise "Setup finished: core ready, N items missing", one line per item with its retry command, and exit status 3. tests/fake-docker/rv-setup-partial-status.sh covers a complete setup, a failed AnnotSV download, two VEP caches skipped for disk space, the --no-clinical-data opt-out and a plain fake setup. Bead: F04 --- scripts/setup.sh | 68 ++++++++-- tests/fake-docker/rv-setup-partial-status.sh | 133 +++++++++++++++++++ 2 files changed, 192 insertions(+), 9 deletions(-) create mode 100755 tests/fake-docker/rv-setup-partial-status.sh diff --git a/scripts/setup.sh b/scripts/setup.sh index 026e16acc..38336acfb 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -88,6 +88,11 @@ # 7. VEP 116, PCGR/CPSR data and VEP 115 (~57 GB download, ~65 GB unpacked). # --no-clinical-data skips them; --clinical-data installs only these three. # +# Exit status of a full setup: 0 when everything is installed ("Setup +# complete!"); 3 when the core is ready but an item of 4 to 7 failed or was +# skipped ("Setup finished: core ready, N items missing", each listed with the +# command that retries it); 1 when the reference, ClinVar or an image failed. +# # Every download goes through fetch (scripts/lib/common.sh): it is written to # .part, checked, and only then renamed, so a file that exists is whole. # A failed image pull is tried FETCH_TRIES times too, FETCH_WAIT apart, and @@ -229,6 +234,12 @@ export GENOME_DIR . "$(dirname "$0")/lib/common.sh" if [ -n "$PCGR_SHA_OVERRIDE" ]; then PCGR_DATA_SHA256=$PCGR_SHA_OVERRIDE; fi +# What a full setup could not install, one "what|retry command" entry each. +# The summary lists them, and a run with any left exits SETUP_PARTIAL_EXIT. +MISSING=() +SETUP_PARTIAL_EXIT=3 +note_missing() { MISSING+=("$1|$2"); } + # NCBI's GRCh38 analysis set without ALT contigs: chr1-22, X, Y, M, the # unplaced and unlocalized scaffolds and chrEBV, 195 sequences. NCBI lists the # md5 of every file of the directory in md5checksums.txt, which is read at @@ -2133,6 +2144,8 @@ install_clinical_data() { else missing=1 fi + [ -d "${GENOME_DIR}/pcgr_data/${PCGR_DATA_BUNDLE}/data" ] \ + || note_missing "PCGR/CPSR bundle ${PCGR_DATA_BUNDLE} (step 17)" "$0 --clinical-data \"${GENOME_DIR}\"" else [ ! -f "${dir}/homo_sapiens/${rel}_GRCh38/info.txt" ] || { echo "[OK] VEP ${rel} cache already present."; continue; } peak=56 @@ -2148,6 +2161,13 @@ install_clinical_data() { else missing=1 fi + if [ ! -f "${dir}/homo_sapiens/${rel}_GRCh38/info.txt" ]; then + if [ "$rel" = "$VEP_CACHE_RELEASE" ]; then + note_missing "VEP ${rel} cache (steps 13, 30, 23 and 31)" "$0 --vep-cache \"${GENOME_DIR}\"" + else + note_missing "VEP ${rel} cache (step 17, CPSR)" "$0 --clinical-data \"${GENOME_DIR}\"" + fi + fi fi done return "$missing" @@ -2461,6 +2481,7 @@ elif "${PGP_ROOT}/rust/og.sh" install; then echo "[OK] og (bin/.og) built." else echo "[WARN] Could not build og. Steps 04, 05, 07, 25 and 33 stop without it; run: $0 --og" + note_missing "og, OwnGenome's own binary (steps 04, 05, 07, 25 and 33)" "$0 --og" fi ############################################################################### @@ -2485,6 +2506,7 @@ else else rm -f "$DICT_PART" echo "[WARN] Could not create ${REF_DICT}. Steps 03a, 20, 29 and chip-to-vcf need it; re-run setup.sh." + note_missing "the sequence dictionary ${REF_DICT##*/} (steps 03a, 20, 29 and chip-to-vcf)" "$0 \"${GENOME_DIR}\"" fi fi @@ -2506,6 +2528,7 @@ for name in $DATA_FILES; do echo "[OK] ${what}: ${dest}" else echo "[WARN] Could not install the ${what}. Re-run setup.sh to try again." + note_missing "the ${what}" "$0 \"${GENOME_DIR}\"" fi done # gnomAD v3.1's chrM sites (step 20 annotates the calls with them), checked @@ -2521,12 +2544,17 @@ else echo "[OK] gnomAD v3.1 mitochondrial sites (step 20): ${GNOMAD_MITO}" else echo "[WARN] Could not install gnomAD's mitochondrial sites. Step 20 runs without its gnomAD columns; re-run setup.sh to try again." + note_missing "gnomAD v3.1 mitochondrial sites (step 20)" "$0 \"${GENOME_DIR}\"" fi fi -# A failed download does not stop setup here either. -install_sample_qc_data || true -install_strchive_data || true -install_tandem_repeats || true +# A failed download does not stop setup here either: it goes on the list of +# what is missing, which the summary prints with the command that retries it. +install_sample_qc_data \ + || note_missing "somalier's sites and VerifyBamID2's panel (step 33)" "$0 --sample-qc-data \"${GENOME_DIR}\"" +install_strchive_data \ + || note_missing "STRchive ${STRCHIVE_VERSION} catalog (steps 09 and 9b)" "$0 --strchive-data \"${GENOME_DIR}\"" +install_tandem_repeats \ + || note_missing "GIAB tandem-repeat map (step 04c)" "$0 \"${GENOME_DIR}\"" # Step 25: its scores, pgsc_calc and its plugin (warnings only). install_prs @@ -2534,11 +2562,13 @@ install_prs # Phase 5: AnnotSV annotation data (step 5, a default step) ############################################################################### # The bundle, gnomAD-SV's common SVs and the MANE Select transcripts -# (install_annotsv_data above). A failure is a warning: the rest of setup goes on. +# (install_annotsv_data above). A failure goes on the list of what is missing: +# the rest of setup goes on. echo "" echo "=== Phase 5: AnnotSV annotation data (~5.3 GB + ~1.7 GB download, ~20 GB unpacked) ===" -install_annotsv_data || true +install_annotsv_data \ + || note_missing "AnnotSV data: the bundle, gnomAD-SV ${GNOMAD_SV_VERSION} and MANE ${MANE_RELEASE} (step 05)" "$0 --annotsv-data \"${GENOME_DIR}\"" ############################################################################### # Phase 6: Clinical reference data (default) @@ -2546,6 +2576,7 @@ install_annotsv_data || true echo "" echo "=== Phase 6: Clinical reference data ===" if $CLINICAL_DATA; then + # Each item it could not install is on the list already (note_missing). install_clinical_data || true else echo "[SKIP] Clinical data (--no-clinical-data): steps 13, 30, 23, 31 and 17 need the caches or bundle." @@ -2555,19 +2586,38 @@ fi ############################################################################### # Summary ############################################################################### +# "Setup complete!" only when nothing is missing. Otherwise each missing item +# with the command that retries it, and the exit status SETUP_PARTIAL_EXIT (3), +# so a caller can tell a partial setup from a full one. echo "" echo "============================================" -echo " Setup complete!" +if [ "${#MISSING[@]}" -eq 0 ]; then + echo " Setup complete!" +else + n=${#MISSING[@]} + echo " Setup finished: core ready, ${n} item$([ "$n" -eq 1 ] || echo s) missing" +fi echo "" echo " Reference genome: ${REFDIR}/" echo " ClinVar database: ${CLINVARDIR}/" echo " Docker images: ${PULLED} pulled, ${SKIPPED} cached" echo "============================================" +if [ "${#MISSING[@]}" -gt 0 ]; then + echo "" + echo "Missing (the steps named run without it or say it is missing; the warnings above say why):" + for item in "${MISSING[@]}"; do + echo " - ${item%%|*}" + echo " retry: ${item#*|}" + done +fi echo "" echo "Next steps:" echo " 1. Place your FASTQ/BAM/VCF in: ${GENOME_DIR}//" echo " 2. Run: export GENOME_DIR=${GENOME_DIR}" echo " 3. Run: ./scripts/validate-setup.sh " echo " 4. Run: ./scripts/run-all.sh " -echo "" -echo "Review any warnings above; use --clinical-data to retry missing clinical downloads." +if [ "${#MISSING[@]}" -gt 0 ]; then + echo "" + echo "Setup exits with status ${SETUP_PARTIAL_EXIT}: a partial setup. Run the retry commands above, or this setup again." + exit "$SETUP_PARTIAL_EXIT" +fi diff --git a/tests/fake-docker/rv-setup-partial-status.sh b/tests/fake-docker/rv-setup-partial-status.sh new file mode 100755 index 000000000..9558030ec --- /dev/null +++ b/tests/fake-docker/rv-setup-partial-status.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash +# setup.sh says "Setup complete!" and exits 0 only when it installed +# everything. When an install fails or is skipped, it prints "Setup finished: +# core ready, N items missing", lists each item with the command that retries +# it, and exits 3. +# +# The fake downloads fail their pinned checksums and og is not built for both +# Linux arches here, so a plain fake setup always misses something. To reach +# the complete case, setup.sh runs from a copy of the checkout (not a git +# checkout, so og.sh has no tree to compare) with two placeholder og binaries, +# and GENOME_DIR holds what the pinned downloads would have installed. A fake +# sha256sum gives the two files setup.sh checks by digest their pinned values. +# shellcheck source=../../scripts/ci/fake-docker/lib.sh +. "${REPO_ROOT:?}/scripts/ci/fake-docker/lib.sh" +# shellcheck source=../../versions.env +. "${REPO_ROOT}/versions.env" +seed_clinical_downloads +export FETCH_TRIES=1 FETCH_WAIT=0 + +COPY="${CASE_WORK}/checkout" +mkdir -p "${COPY}/rust" "${COPY}/bin/.og" +cp -R "${REPO_ROOT}/scripts" "${REPO_ROOT}/assets" "${REPO_ROOT}/versions.env" "$COPY/" +cp "${REPO_ROOT}/rust/og.sh" "${COPY}/rust/" +for f in "${REPO_ROOT}"/bin/*; do + [ -f "$f" ] && cp "$f" "${COPY}/bin/" +done +for a in x86_64 aarch64; do + printf '#!/bin/sh\nexit 0\n' > "${COPY}/bin/.og/og-linux-${a}" + chmod +x "${COPY}/bin/.og/og-linux-${a}" +done +SETUP="${COPY}/scripts/setup.sh" + +# A fake sha256sum: the pinned digest for the tandem-repeat map and gnomAD's +# chrM sites, the real one for every other file. +real_sha=$(command -v sha256sum || true) +mkdir -p "${CASE_WORK}/sha-bin" +cat > "${CASE_WORK}/sha-bin/sha256sum" < "${r}/GRCh38_no_alt_analysis_set.dict" + # The pinned data files, by the paths common.sh's data_file gives them. + # shellcheck disable=SC2016 # expanded by the inner bash + while read -r u; do + mkdir -p "${u%/*}" + printf 'x\n' > "$u" + done < <(GENOME_DIR=$g bash -c '. "$1"; for n in $DATA_FILES; do data_file "$n" || true; done' _ "${COPY}/scripts/lib/common.sh") + for u in somalier/sites.hg38.vcf.gz \ + verifybamid2/1000g.phase3.100k.b38.vcf.gz.dat.UD verifybamid2/1000g.phase3.100k.b38.vcf.gz.dat.mu \ + verifybamid2/1000g.phase3.100k.b38.vcf.gz.dat.bed "${TANDEM_REPEATS_URL##*/}" \ + "$(basename "${TANDEM_REPEATS_URL}" .gz)" "strchive-${STRCHIVE_VERSION}/catalog.json" \ + "strchive-${STRCHIVE_VERSION}/str_ranges.tsv"; do + printf 'x\n' > "${r}/${u}" + done + printf 'x\n' > "${g}/gnomad_mito/${GNOMAD_MITO_URL##*/}" + printf 'release=%s eh=%s loci=%s converter=og-str-catalog-1 extra=off\n' \ + "$STRCHIVE_VERSION" "$STRCHIVE_EH_SHA256" "$STRCHIVE_LOCI_SHA256" > "${r}/strchive-${STRCHIVE_VERSION}/catalog.stamp" + [ "${2:-}" = no-annotsv ] && return 0 + mkdir -p "${a}/Annotations_Human/Genes/GRCh38" "${a}/Annotations_Human/SVincludedInFt/BenignSV/GRCh38" + printf 'gnomAD-SV %s sha256 %s\n' "$GNOMAD_SV_VERSION" "$GNOMAD_SV_SHA256" \ + > "${a}/Annotations_Human/SVincludedInFt/BenignSV/GRCh38/gnomad-sv.installed" + printf 'NM_000001.1\n' > "${a}/mane_select_refseq.txt" + printf 'MANE %s sha256 %s\n' "$MANE_RELEASE" "$MANE_SUMMARY_SHA256" > "${a}/mane_select_refseq.release" +} + +# 1. Everything installs: the banner, no missing list, exit 0. +G="${CASE_WORK}/complete" +seed_installed "$G" +run_expect 0 complete "$SETUP" "$G" +output_has complete '^ Setup complete!$' +output_lacks complete 'Setup finished|Missing \(|exits with status 3' + +# 2. The AnnotSV download fails: no banner, AnnotSV and its retry, exit 3. +G="${CASE_WORK}/annotsv" +seed_installed "$G" no-annotsv +FAKE_DOWNLOAD_FAIL='lbgi\.fr' run_expect 3 annotsv "$SETUP" "$G" +output_lacks annotsv 'Setup complete' +output_has annotsv '^ Setup finished: core ready, 1 item missing$' +output_has annotsv '^ - AnnotSV data: the bundle, gnomAD-SV .* \(step 05\)$' +output_has annotsv "^ retry: .*/scripts/setup\\.sh --annotsv-data \"${G}\"$" +output_has annotsv 'Setup exits with status 3: a partial setup' +[ "$(grep -c '^ - ' "${CASE_WORK}/annotsv.out")" -eq 1 ] || fail 'annotsv lists more than the AnnotSV data' + +# 3. Too little disk for the two VEP caches: each skipped cache is listed +# with its own retry, the bundle still installs, exit 3. +mkdir -p "${CASE_WORK}/low-bin" +cat > "${CASE_WORK}/low-bin/df" <<'DF' +#!/usr/bin/env bash +echo 'Filesystem 1024-blocks Used Available Capacity Mounted on' +echo 'fakefs 41943040 0 41943040 0% /' +DF +chmod +x "${CASE_WORK}/low-bin/df" +G="${CASE_WORK}/low" +seed_installed "$G" +PATH="${CASE_WORK}/low-bin:${PATH}" run_expect 3 low "$SETUP" "$G" +output_lacks low 'Setup complete' +output_has low '^ Setup finished: core ready, 2 items missing$' +output_has low "^ - VEP ${VEP_CACHE_RELEASE} cache \\(steps 13, 30, 23 and 31\\)$" +output_has low "^ retry: .*/scripts/setup\\.sh --vep-cache \"${G}\"$" +output_has low "^ - VEP ${PCGR_VEP_CACHE_RELEASE} cache \\(step 17, CPSR\\)$" +output_has low "^ retry: .*/scripts/setup\\.sh --clinical-data \"${G}\"$" +output_lacks low '^ - PCGR' +[ -d "${G}/pcgr_data/${PCGR_DATA_BUNDLE}/data" ] || fail 'the bundle was not installed beside the skipped caches' + +# 4. --no-clinical-data is a choice, not a gap: complete, exit 0. +G="${CASE_WORK}/opt-out" +seed_installed "$G" +run_expect 0 opt_out "$SETUP" --no-clinical-data "$G" +output_has opt_out '^ Setup complete!$' + +# 5. A plain fake setup misses several items: each has a retry line. +G="${CASE_WORK}/plain" +mkdir -p "$G" +run_expect 3 plain "$SETUP" "$G" +output_lacks plain 'Setup complete' +items=$(grep -c '^ - ' "${CASE_WORK}/plain.out") +retries=$(grep -c '^ retry: ' "${CASE_WORK}/plain.out") +[ "$items" -ge 5 ] && [ "$items" -eq "$retries" ] || fail "plain lists ${items} items and ${retries} retry lines" +output_has plain "^ Setup finished: core ready, ${items} items missing$" +output_has plain "retry: .*/scripts/setup\\.sh --sample-qc-data \"${G}\"$" +output_has plain "retry: .*/scripts/setup\\.sh --strchive-data \"${G}\"$" From f83b3363bec4caca53f09b5b9c458d50fda28cc3 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:50:20 +0100 Subject: [PATCH 06/14] docs(setup): document setup.sh's exit statuses and the missing list Bead: F04 --- CHANGELOG.md | 1 + docs/getting-started.md | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05c3388e1..9ab8a455d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- `setup.sh` no longer says "Setup complete!" after a download failed. Before, a failed or skipped install (AnnotSV's data, a VEP cache, the CPSR bundle, somalier's sites, the STRchive catalog, the tandem-repeat map, gnomAD's chrM sites, a pinned data file, the sequence dictionary or og) printed a warning, then the same banner and exit status 0 as a full setup. Now setup keeps a list of what it could not install. When the list is empty it prints "Setup complete!" and exits 0 as before. Otherwise it prints "Setup finished: core ready, N items missing", lists each item with the command that retries it (`--annotsv-data`, `--vep-cache`, `--clinical-data`, `--sample-qc-data`, `--strchive-data`, `--og`, or setup again), and exits 3. A script that runs `setup.sh` and checks its status now sees a partial setup as one. `--no-clinical-data` still ends complete: skipping the clinical data is a choice. [Getting started](docs/getting-started.md) lists the three exit statuses. - The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes. - The phased family VCF is indexed by the Rust `og` binary. WhatsHap's step (step 38 and the Nextflow WHATSHAP_PED process) wrote `family.phased.vcf.gz.tbi` with a Python one-liner, `pysam.tabix_index`. It now runs `og vcf index -f`. The `.tbi` keeps its name and format, and its bytes, once inflated, are the ones htslib 1.24 writes (the htslib of the pysam and bcftools the pipeline pins): the same bins in the same order, the same chunks and the same linear index. Only the deflate stream can differ, as two deflate libraries differ. On a 5-million-record three-sample VCF it took 0.75 to 0.91 s and 31 MB at peak, where htslib's own `tabix` took 0.81 to 0.93 s and 19 MB. - As in htslib 1.24, a record ends at the largest of INFO/END, the REF length, the SVLEN of a ``, ``, `` or `` ALT, and a sample's LEN on a gVCF block. A line with no position to read, unsorted records, a sequence that starts again, a record past 2^29 and an empty file stop the index with a message, and nothing is written. Without `-f` it never writes over an index, even one that appears while it runs. diff --git a/docs/getting-started.md b/docs/getting-started.md index e72239787..c528c9cd2 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -57,6 +57,16 @@ Use `./scripts/setup.sh --no-clinical-data "$GENOME_DIR"` to opt out. Use `./scripts/setup.sh --clinical-data "$GENOME_DIR"` to add or retry only these data later. `run-all.sh` skips steps whose data are still missing; inspect its plan and setup warnings. +Setup ends in one of three ways: + +| Exit status | Last banner | What it means | +| --- | --- | --- | +| 0 | `Setup complete!` | Everything is installed. `--no-clinical-data` skips the clinical data on purpose, so that run can still end here. | +| 3 | `Setup finished: core ready, N items missing` | The reference, ClinVar and the images are in place, but at least one download, check or build failed or was skipped. Under `Missing` the summary lists each item and the command that retries it, such as `./scripts/setup.sh --annotsv-data "$GENOME_DIR"`. The steps that need an item say so when they run. | +| 1 | an error message, no banner | A core part failed (the reference, ClinVar or an image pull), or the arguments were wrong. Fix the error and run setup again. | + +The PGS Catalog scores, pgsc_calc and its Nextflow plugin are not on the list: step 25 fetches what is missing when it runs. + ## Platform Notes ### Linux (Recommended) From 06cf3df3bd5c161f69d1070489f74d00b015d011 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 18:04:10 +0100 Subject: [PATCH 07/14] fix(io): create og's temporary output files new and owner-only AtomicFile created its temporary file with File::create on a name made of the target and the pid. The mode followed the umask, so under umask 022 a new output was 0644 and a replaced 0600 output became 0644. File::create also opened whatever was already at that name, and two AtomicFiles on one target in one process shared one temporary file. The temporary file is now opened with create_new (O_CREAT | O_EXCL) and mode 0600, or the replaced target's mode when that is stricter. Its name carries a process-wide counter, and a taken name moves on to the next. Bead: rv-io-owner-only-writes --- rust/crates/og-io/src/lib.rs | 76 +++++++++-- rust/crates/og-io/tests/atomic_modes.rs | 118 ++++++++++++++++++ .../crates/og-io/tests/atomic_planted_temp.rs | 55 ++++++++ 3 files changed, 238 insertions(+), 11 deletions(-) create mode 100644 rust/crates/og-io/tests/atomic_modes.rs create mode 100644 rust/crates/og-io/tests/atomic_planted_temp.rs diff --git a/rust/crates/og-io/src/lib.rs b/rust/crates/og-io/src/lib.rs index 18d0652b4..4901ed3a3 100644 --- a/rust/crates/og-io/src/lib.rs +++ b/rust/crates/og-io/src/lib.rs @@ -6,7 +6,8 @@ //! - [`Lines`]: Python's text mode with `newline=None`: `\n`, `\r\n` and a //! lone `\r` each end a line and read as `\n`; UTF-8 strict or `replace`. //! - [`AtomicFile`]: written beside the target and renamed over it on -//! [`AtomicFile::commit`], so a reader never sees half a file. +//! [`AtomicFile::commit`], so a reader never sees half a file. What it +//! writes is owner-only (0600) whatever the umask. //! - [`bgzf`]: the same reading with BGZF inflated on many cores, and a //! BGZF writer that compresses on many cores. //! - [`tabix`]: the lines of a region of a bgzipped file, through its @@ -22,6 +23,7 @@ use flate2::bufread::GzDecoder; use std::fs::{self, File}; use std::io::{self, BufRead, BufReader, BufWriter, Read, Write}; use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; const GZIP_MAGIC: [u8; 2] = [0x1f, 0x8b]; @@ -198,28 +200,52 @@ impl Iterator for Lines { /// A file written under a temporary name and renamed into place by /// [`commit`](AtomicFile::commit). Dropped without a commit, the temporary /// file is removed and the target is untouched. +/// +/// The temporary file is always a new file: a file or link already at its +/// name is left alone and the next name is tried. On Unix it is created +/// mode 0600, or with the mode of the target it replaces when that is +/// stricter, so the umask never makes an output readable by others. pub struct AtomicFile { tmp: PathBuf, target: PathBuf, out: Option>, } +/// The number in the next temporary name, so two files open in one process +/// never share one. +static NEXT_TMP: AtomicU64 = AtomicU64::new(0); + +/// How many temporary names are tried before giving up. +const TMP_TRIES: usize = 64; + impl AtomicFile { pub fn create(target: impl AsRef) -> io::Result { let target = target.as_ref().to_path_buf(); let name = target .file_name() .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "no file name"))?; - let mut tmp_name = std::ffi::OsString::from("."); - tmp_name.push(name); - tmp_name.push(format!(".tmp-{}", std::process::id())); - let tmp = target.with_file_name(tmp_name); - let out = BufWriter::new(File::create(&tmp)?); - Ok(AtomicFile { - tmp, - target, - out: Some(out), - }) + let mode = temp_mode(&target); + let pid = std::process::id(); + let mut taken = None; + for _ in 0..TMP_TRIES { + let n = NEXT_TMP.fetch_add(1, Ordering::Relaxed); + let mut tmp_name = std::ffi::OsString::from("."); + tmp_name.push(name); + tmp_name.push(format!(".tmp-{pid}-{n}")); + let tmp = target.with_file_name(tmp_name); + match new_file(&tmp, mode) { + Ok(f) => { + return Ok(AtomicFile { + tmp, + target, + out: Some(BufWriter::new(f)), + }); + } + Err(e) if e.kind() == io::ErrorKind::AlreadyExists => taken = Some(e), + Err(e) => return Err(e), + } + } + Err(taken.expect("at least one name was tried")) } /// Flushes and renames over the target. On an error the temporary file @@ -240,6 +266,34 @@ impl AtomicFile { } } +/// The mode of a new temporary file: 0600, less any owner bit the target +/// it replaces lacks (a 0400 target stays 0400). +#[cfg(unix)] +fn temp_mode(target: &Path) -> u32 { + use std::os::unix::fs::PermissionsExt; + match fs::metadata(target) { + Ok(m) if m.is_file() => 0o600 & m.permissions().mode(), + _ => 0o600, + } +} + +#[cfg(not(unix))] +fn temp_mode(_target: &Path) -> u32 { + 0 +} + +/// Creates `path`, failing with `AlreadyExists` when anything is there, +/// a link included (`O_CREAT | O_EXCL`). +fn new_file(path: &Path, mode: u32) -> io::Result { + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + std::os::unix::fs::OpenOptionsExt::mode(&mut options, mode); + #[cfg(not(unix))] + let _ = mode; + options.open(path) +} + impl Write for AtomicFile { fn write(&mut self, buf: &[u8]) -> io::Result { self.out.as_mut().expect("not committed").write(buf) diff --git a/rust/crates/og-io/tests/atomic_modes.rs b/rust/crates/og-io/tests/atomic_modes.rs new file mode 100644 index 000000000..1947c518c --- /dev/null +++ b/rust/crates/og-io/tests/atomic_modes.rs @@ -0,0 +1,118 @@ +//! The modes and temporary names of [`og_io::AtomicFile`]: what it writes is +//! owner-only whatever the umask, and two writers in one process never share +//! a temporary file. + +use og_io::AtomicFile; +use std::fs; +use std::io::Write; +use std::os::unix::fs::PermissionsExt; +use std::path::{Path, PathBuf}; +use std::process::Command; + +const CHILD: &str = "OG_IO_UMASK_022_CHILD"; + +fn scratch(name: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!("og-io-{name}-{}", std::process::id())); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + dir +} + +/// The permission bits in octal, so a failure reads `644`, not `420`. +fn mode(path: &Path) -> String { + format!( + "{:o}", + fs::metadata(path).unwrap().permissions().mode() & 0o7777 + ) +} + +fn write(target: &Path, text: &str) { + let mut f = AtomicFile::create(target).unwrap(); + f.write_all(text.as_bytes()).unwrap(); + f.commit().unwrap(); +} + +/// Runs itself again in a child under `umask 022`, the umask most shells +/// start with, and checks that the child ran this test and passed it. +#[test] +fn owner_only_under_umask_022() { + if std::env::var_os(CHILD).is_none() { + let exe = std::env::current_exe().unwrap(); + let out = Command::new("/bin/sh") + .arg("-c") + .arg("umask 022 && exec \"$0\" --exact owner_only_under_umask_022 --nocapture --test-threads=1") + .arg(&exe) + .env(CHILD, "1") + .output() + .unwrap(); + let stdout = String::from_utf8_lossy(&out.stdout); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + out.status.success() && stdout.contains("1 passed"), + "the umask 022 child failed or ran no test:\n{stdout}\n{stderr}" + ); + return; + } + let dir = scratch("umask"); + // The control: under this umask a plain new file is 0644. + let control = dir.join("control"); + fs::File::create(&control).unwrap(); + assert_eq!( + mode(&control), + "644", + "the child does not run under umask 022" + ); + + // A new target. + let new = dir.join("new.tsv"); + write(&new, "new\n"); + assert_eq!(mode(&new), "600", "a new target"); + + // A 0600 target that is replaced stays 0600. + let private = dir.join("private.tsv"); + fs::write(&private, "old\n").unwrap(); + fs::set_permissions(&private, fs::Permissions::from_mode(0o600)).unwrap(); + write(&private, "new\n"); + assert_eq!(mode(&private), "600", "a replaced 0600 target"); + assert_eq!(fs::read_to_string(&private).unwrap(), "new\n"); + + // A 0644 target becomes owner-only. + let open = dir.join("open.tsv"); + fs::write(&open, "old\n").unwrap(); + fs::set_permissions(&open, fs::Permissions::from_mode(0o644)).unwrap(); + write(&open, "new\n"); + assert_eq!(mode(&open), "600", "a replaced 0644 target"); + + // A stricter target keeps its mode. + let read_only = dir.join("read-only.tsv"); + fs::write(&read_only, "old\n").unwrap(); + fs::set_permissions(&read_only, fs::Permissions::from_mode(0o400)).unwrap(); + write(&read_only, "new\n"); + assert_eq!(mode(&read_only), "400", "a replaced 0400 target"); + assert_eq!(fs::read_to_string(&read_only).unwrap(), "new\n"); + + fs::remove_dir_all(&dir).unwrap(); +} + +/// Two files open on one target in one process each write their own +/// temporary file: both commits succeed and the last one wins whole. +#[test] +fn two_writers_on_one_target() { + let dir = scratch("two"); + let target = dir.join("out.tsv"); + let mut a = AtomicFile::create(&target).unwrap(); + let mut b = AtomicFile::create(&target).unwrap(); + a.write_all(b"from a\n").unwrap(); + b.write_all(b"from b\n").unwrap(); + a.commit().unwrap(); + assert_eq!(fs::read_to_string(&target).unwrap(), "from a\n"); + b.commit().unwrap(); + assert_eq!(fs::read_to_string(&target).unwrap(), "from b\n"); + // Nothing but the target is left. + let names: Vec<_> = fs::read_dir(&dir) + .unwrap() + .map(|e| e.unwrap().file_name()) + .collect(); + assert_eq!(names, ["out.tsv"]); + fs::remove_dir_all(&dir).unwrap(); +} diff --git a/rust/crates/og-io/tests/atomic_planted_temp.rs b/rust/crates/og-io/tests/atomic_planted_temp.rs new file mode 100644 index 000000000..e6d3bd7c5 --- /dev/null +++ b/rust/crates/og-io/tests/atomic_planted_temp.rs @@ -0,0 +1,55 @@ +//! [`og_io::AtomicFile`] creates its temporary file and never opens one that +//! is already there. This is the only test in its binary, so the first +//! temporary name this process tries is known: `..tmp--0`. + +use og_io::AtomicFile; +use std::fs; +use std::io::Write; +use std::os::unix::fs::symlink; + +#[test] +fn a_link_at_the_temporary_name_is_left_alone() { + let pid = std::process::id(); + let dir = std::env::temp_dir().join(format!("og-io-planted-{pid}")); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let target = dir.join("out.tsv"); + let other = dir.join("other.tsv"); + fs::write(&other, "keep\n").unwrap(); + let missing = dir.join("missing.tsv"); + // Links at the first two names this process tries, and at the name + // releases before 1.0.0 used (`..tmp-`). + let planted = [ + (format!(".out.tsv.tmp-{pid}"), &other), + (format!(".out.tsv.tmp-{pid}-0"), &other), + (format!(".out.tsv.tmp-{pid}-1"), &missing), + ]; + for (name, to) in &planted { + symlink(to, dir.join(name)).unwrap(); + } + + let mut f = AtomicFile::create(&target).unwrap(); + f.write_all(b"new\n").unwrap(); + f.commit().unwrap(); + + assert_eq!(fs::read_to_string(&other).unwrap(), "keep\n"); + assert!( + !missing.exists(), + "a file was created through a dangling link" + ); + assert!( + !fs::symlink_metadata(&target) + .unwrap() + .file_type() + .is_symlink() + ); + assert_eq!(fs::read_to_string(&target).unwrap(), "new\n"); + for (name, to) in &planted { + assert_eq!( + &fs::read_link(dir.join(name)).unwrap(), + *to, + "{name} changed" + ); + } + fs::remove_dir_all(&dir).unwrap(); +} From 80fd49444eea3553d768469a74e660a571d19ba6 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 18:04:10 +0100 Subject: [PATCH 08/14] docs: say og writes owner-only files and keep --outdir private Bead: rv-io-owner-only-writes --- CHANGELOG.md | 1 + docs/nextflow.md | 2 ++ 2 files changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05c3388e1..46c499382 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- The files the `og` binary writes are owner-only (mode 0600) whatever the umask. Before, `og` wrote each output under a temporary name with the umask's mode, so under the common umask 022 an output was readable by every user of the machine, and replacing a 0600 file made it 0644. The step scripts set umask 077, so a run through them was already owner-only; `og` run by hand or by the Nextflow pipeline was not. Now the temporary file is created mode 0600, or with the mode of the file it replaces when that is stricter (a 0400 file stays 0400). It is always a new file: a file or link already at its name is left as it is and the next name is tried, and two outputs written at once to one name in one process get two temporary files. The temporary name gains a number, `..tmp--`. No output name or content changes. [Nextflow Execution](docs/nextflow.md#run-the-pipeline) now says to keep `--outdir` private. - The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes. - The phased family VCF is indexed by the Rust `og` binary. WhatsHap's step (step 38 and the Nextflow WHATSHAP_PED process) wrote `family.phased.vcf.gz.tbi` with a Python one-liner, `pysam.tabix_index`. It now runs `og vcf index -f`. The `.tbi` keeps its name and format, and its bytes, once inflated, are the ones htslib 1.24 writes (the htslib of the pysam and bcftools the pipeline pins): the same bins in the same order, the same chunks and the same linear index. Only the deflate stream can differ, as two deflate libraries differ. On a 5-million-record three-sample VCF it took 0.75 to 0.91 s and 31 MB at peak, where htslib's own `tabix` took 0.81 to 0.93 s and 19 MB. - As in htslib 1.24, a record ends at the largest of INFO/END, the REF length, the SVLEN of a ``, ``, `` or `` ALT, and a sample's LEN on a gVCF block. A line with no position to read, unsorted records, a sequence that starts again, a record past 2^29 and an empty file stop the index with a message, and nothing is written. Without `-f` it never writes over an index, even one that appears while it runs. diff --git a/docs/nextflow.md b/docs/nextflow.md index 6332f369e..8e6d79fca 100644 --- a/docs/nextflow.md +++ b/docs/nextflow.md @@ -91,6 +91,8 @@ nextflow run main.nf \ # it names (two or more, each with a gVCF; docs/multi-sample.md) ``` +**Keep `--outdir` private.** The results hold genome data. Create the folder before the run with `mkdir -m 700 results`, or put it in a folder only you can open. The `og` tools write their files owner-only (mode 0600), but Nextflow creates the folders, and the other tools write their files, with the modes your umask gives. + **Family workflow (opt-in).** `--pedigree` (tab separated: sample, sex, father, mother, partner; [`assets/pedigree.example.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/pedigree.example.tsv)) runs `workflows/family.nf` over the samplesheet rows it names. `--pedigree` may be a families file (a leading family column, [`assets/families.example.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/families.example.tsv)). `--family_name` names the family to run, which a file with more than one family needs, and the folder its results go to (`//` instead of `/family/`). The workflow does the same work as step 38: `FAMILY_PREPARE` checks the pedigree and that each member's gVCF names its own sample, `GLNEXUS` and `GLNEXUS_VCF` joint-call the gVCFs (`GLNEXUS` asks for 16 GB and gives GLnexus half of its memory as `--mem-gbytes`), `FAMILY_RELATE` and `FAMILY_CHECK` stop the run when somalier's relatedness of a pair disagrees with the pedigree (a sample swap), `WHATSHAP_PED` phases the children of complete trios by inheritance, and `FAMILY_REPORT` writes each child's allele origin, de novo candidates when both parents are sequenced, and compound heterozygotes among the clinical filter's candidates. With `--ancestry_fine_panel` (the panel folder of step 26b, `setup.sh --ancestry-fine-panel`), four more processes find the DNA each pair shares by descent, once `FAMILY_CHECK` has passed: `FAMILY_IBD_SITES` keeps the joint call's genotypes at the panel's SNVs on each autosome (a chromosome with fewer than 100 is left out), `FAMILY_IBD_PHASE` phases them with Beagle against the panel, `FAMILY_IBD_SEGMENTS` runs hap-ibd (its default 2 cM minimum) with the panel's genetic maps, and `FAMILY_IBD_REPORT` writes `ibd_pairs.tsv`, `ibd_segments.tsv` and `ibd_report.md`. They feed only the report, so a failure of one is logged and the run goes on. Without the panel none of them runs. Each member needs a gVCF: DEEPVARIANT's for a FASTQ or BAM row, the `gvcf` column for a VCF row (the run stops on a VCF row without one). Fewer than two named samples, or no `--somalier_sites`, stops the run at the start. When the pedigree marks two of those samples as each other's partner, `COUPLE_CARRIER` runs the couple carrier screen of step 39 on them ([Couple Carrier Screen](39-couple.md)): it reads each partner's ClinVar hits, so it needs `clinvar` in `--tools` (without it the run logs a warning and skips the screen), uses deletions over an exon only with `--gencode_exons` (the GENCODE exons BED `setup.sh` installs as `reference/gencode.v50.basic.exons.bed`), reads Delly's calls, ExpansionHunter's, and Parascopy's SMN1, interval and CYP21A2 tables when those steps ran, and, when `--annotsv_annotations` holds AnnotSV's benign loss set (`benign_Loss_SV_GRCh38.sorted.bed`, with gnomAD-SV's common deletions merged in by `setup.sh --annotsv-data`), sets aside a Delly deletion that matches a common one there. Without `--pedigree` none of these processes exists and a run is unchanged. What it checks and what the files hold: [Multi-sample comparison](multi-sample.md#family-analysis-step-38-opt-in). `--reference` is the FASTA the reads are aligned to and the BAMs were aligned to, the GRCh38 no-ALT analysis set that `setup.sh` installs as `reference/GRCh38_no_alt_analysis_set.fasta`. The Nextflow pipeline does not compare a given BAM's header with it; `./scripts/validate-setup.sh ` does, and [Realigning after a reference change](realignment.md) covers a BAM aligned to another reference. From fd3e22522be8e1291748c4317de40f7c795f2292 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 18:39:36 +0100 Subject: [PATCH 09/14] test(e2e): count MoChA's chr20 calls only on HG002, now that it reads every slice With the unpanelled autosomes kept, MoChA reads HG002's other fixture slices unphased and may call a germline CNV there (such as the chr4 UGT2B17 deletion, untyped because the cut panel's cnps.bed holds chr20 only). The zero-call check now counts chr20 only; the nothing-reported check stays. The red control on MOSAIC counts chr20 too, and the docs and comments say a call on another slice is reported no. --- docs/51-mocha.md | 2 +- tests/e2e/mocha-1.sh | 44 ++++++++++++++++++++++++++++---------------- 2 files changed, 29 insertions(+), 17 deletions(-) diff --git a/docs/51-mocha.md b/docs/51-mocha.md index e15744040..8e902eb30 100644 --- a/docs/51-mocha.md +++ b/docs/51-mocha.md @@ -130,4 +130,4 @@ Not measured on a whole genome. Beagle's phasing of the 22 chromosomes takes mos - A constitutional event (one every cell carries, such as a germline deletion) has a cell fraction near 1 and is left out by MoChA's filter; this step is not a test for those. - A germline copy-number polymorphism is left out by type: MoChA types a call over a region of `cnps.bed` as `CNP_Loss` or `CNP_Gain`. A germline CNV missing from that list (rare, or under 10 kb) gets no such type. Short events (under a few Mb) are hard to tell from these and are reported only when MoChA's filter keeps them. - Calls are labelled by chromosome arm, not by gene: an mCA spans millions of bases. -- The e2e case (`tests/e2e/mocha-1.sh`) runs the step on HG002's slices with the panel cut to the chr20 slice (MoChA makes no call; the fixture's other autosomes have no panel file, so they stay unphased and the sample is not assessed, with each of them named), and on a synthetic phased VCF with a 20 Mb CN-LOH on 20p in 15% of the cells (one CN-LOH call, cell fraction 0.10 to 0.20) beside the same VCF with no shift (no call). Nothing in the case is a finding about the fixture's sample. +- The e2e case (`tests/e2e/mocha-1.sh`) runs the step on HG002's slices with the panel cut to the chr20 slice (MoChA makes no call on chr20; the fixture's other autosomes have no panel file, so they stay unphased and the sample is not assessed, with each of them named; a call MoChA makes on one of those slices, such as a germline CNV like the chr4 UGT2B17 deletion, is untyped because the cut panel's `cnps.bed` holds chr20 only, and is reported "no" because the sample is not assessed), and on a synthetic phased VCF with a 20 Mb CN-LOH on 20p in 15% of the cells (one CN-LOH call, cell fraction 0.10 to 0.20) beside the same VCF with no shift (no call). Nothing in the case is a finding about the fixture's sample. diff --git a/tests/e2e/mocha-1.sh b/tests/e2e/mocha-1.sh index 2253cf8c8..e6220858a 100755 --- a/tests/e2e/mocha-1.sh +++ b/tests/e2e/mocha-1.sh @@ -10,11 +10,14 @@ # the genetic map of chr20 unpacked, the folder under 200 MB, and cnps.bed # (the panel's common CNVs) written, in MoChA's --cnp format when not empty; # - HG002 (case 21's VCF) phased by Beagle on that slice: phased hets, MoChA -# makes no call and none is reported, and MoChA's log names cnps.bed as -# its --cnp list. The fixture's other autosomes, which the cut panel has -# no file for, still reach the sites table, unphased, so the sample is not -# assessed and the message names each of them (red control: the same check -# fails on the sites table cut to chr20, all the step kept before); +# makes no call on chr20, none of its calls is reported, and MoChA's log +# names cnps.bed as its --cnp list. The fixture's other autosomes, which +# the cut panel has no file for, still reach the sites table, unphased, so +# the sample is not assessed and the message names each of them (red +# control: the same check fails on the sites table cut to chr20, all the +# step kept before). MoChA may call on those other slices (a germline CNV +# such as HG002's chr4 UGT2B17 deletion, untyped since the cut panel's +# cnps.bed has chr20 only); such a call is reported "no"; # - MOSAIC, a synthetic phased VCF on chr16, chr19, chr20 and chr22 at 30x # (REF bases from the fixture reference) with a 20 Mb CN-LOH on 20p from # the telomere in 15% of the cells (the haplotype balance 0.575 instead of @@ -26,10 +29,11 @@ # check fails on its log (red control of that check); # - nextflow run main.nf --tools mocha,html_report on HG002 writes the # bash step's call table byte for byte, the same summary values and a -# MoChA log naming cnps.bed, and the HTML report has the card. MoChA makes -# no call on HG002, so the byte check covers a header-only table: the -# summary values (sites, depth, hets per 10 Mb, chromosomes) are the mirror -# evidence. Control: MOSAIC's table differs. +# MoChA log naming cnps.bed, and the HTML report has the card. The byte +# check covers every row MoChA wrote on HG002, its calls on the other +# slices included, and the summary values (sites, depth, hets per 10 Mb, +# chromosomes, calls) are mirror evidence too. Control: MOSAIC's table +# differs. . "$(dirname "$0")/lib.sh" G="$GENOME_DIR" T="$CASE_TMP" @@ -37,6 +41,12 @@ MD="${G}/reference/mocha-${MOCHA_PANEL_VERSION}" summ() { awk -F'\t' -v k="$2" '$1 == k {print $2}' "${G}/$1/mosaic/$1_mocha_summary.tsv" 2>/dev/null; } reported() { awk -F'\t' 'NR > 1 && $13 == "yes"' "${G}/$1/mosaic/$1_mocha.tsv" 2>/dev/null; } nrep() { reported "$1" | grep -c . || true; } +# chr20_calls SAMPLE: MoChA's calls on chr20, reported or not ("missing" with no table). +chr20_calls() { + local f="${G}/$1/mosaic/$1_mocha.tsv" + [ -s "$f" ] || { echo missing; return; } + awk -F'\t' 'NR > 1 && $1 == "chr20"' "$f" | grep -c . || true +} between() { awk -v x="${1:-}" -v lo="$2" -v hi="$3" 'BEGIN {exit !(x ~ /^[0-9.]+$/ && x + 0 >= lo && x + 0 <= hi)}'; } differs() { [ -n "$1" ] && [ "$1" != "$2" ]; } not_same() { ! cmp -s "$1" "$2"; } @@ -115,15 +125,17 @@ awk -F'\t' '$1 == "chr20"' "$SITES" > "${T}/sites_chr20.tsv" check "red control: that check fails on the sites table cut to chr20 (all the step kept before)" \ not unpanelled "${T}/sites_chr20.tsv" "$(summ "$SAMPLE" message)" check_ge "HG002: phased heterozygous sites" "$(summ "$SAMPLE" phased_het_sites)" 100 -# The status is not assessed (the autosomes above), with its reason; MoChA's -# own call count is what must be zero. +# The status is not assessed (the autosomes above), with its reason. MoChA may +# call on the unphased slices (a germline CNV, untyped because the cut panel's +# cnps.bed has chr20 only), so the zero is its call count on chr20, and none of +# its calls may be reported. check "HG002: a status with its reason ($(summ "$SAMPLE" status), baf_auto $(summ "$SAMPLE" baf_auto))" \ status_ok "$(summ "$SAMPLE" status)" "$(summ "$SAMPLE" message)" -check_eq "HG002 chr20: MoChA made no call" "$(summ "$SAMPLE" calls_mocha)" 0 -check_eq "HG002 chr20: no mosaic call reported" "$(nrep "$SAMPLE")" 0 +check_eq "HG002: MoChA made no call on chr20 ($(summ "$SAMPLE" calls_mocha) on all slices)" "$(chr20_calls "$SAMPLE")" 0 +check_eq "HG002: no mosaic call reported" "$(nrep "$SAMPLE")" 0 check "HG002: MoChA read cnps.bed as its --cnp list (its log)" cnp_logged "${G}/${SAMPLE}/mosaic/${SAMPLE}_mocha.log" -echo "mocha: HG002 chr20 slice: $(summ "$SAMPLE" het_sites) hets ($(summ "$SAMPLE" phased_het_sites) phased)," \ - "depth $(summ "$SAMPLE" depth)x, floor $(summ "$SAMPLE" floor), $(summ "$SAMPLE" calls_mocha) MoChA calls" >> "$E2E_NOTES" +echo "mocha: HG002, panel cut to chr20: $(summ "$SAMPLE" het_sites) hets ($(summ "$SAMPLE" phased_het_sites) phased)," \ + "depth $(summ "$SAMPLE" depth)x, floor $(summ "$SAMPLE" floor), $(summ "$SAMPLE" calls_mocha) MoChA calls ($(chr20_calls "$SAMPLE") on chr20)" >> "$E2E_NOTES" # --- MOSAIC and MOSAIC0: a planted CN-LOH, and none --------------------------------- for s in MOSAIC MOSAIC0; do rm -rf "${G:?}/${s}"; mkdir -p "${G}/${s}/vcf"; done @@ -151,7 +163,7 @@ check "MOSAIC: and ends at 19 to 21 Mb ($(cut -f3 <<< "$ROW"))" between "$(cut - check "MOSAIC: cell fraction 0.10 to 0.20 ($(cut -f7 <<< "$ROW"))" between "$(cut -f7 <<< "$ROW")" 0.10 0.20 check_eq "MOSAIC0 (no shift): no call reported" "$(nrep MOSAIC0)" 0 check "red control: the one-call check fails on MOSAIC0" test "$(nrep MOSAIC0)" != 1 -check "red control: the no-MoChA-call check on HG002 fails on MOSAIC ($(summ MOSAIC calls_mocha))" differs "$(summ MOSAIC calls_mocha)" 0 +check "red control: the no-chr20-call check on HG002 fails on MOSAIC ($(chr20_calls MOSAIC))" differs "$(chr20_calls MOSAIC)" 0 check "MOSAIC0, no panel folder: the step warns that there is no --cnp list" has 'no --cnp list' "$(cat "$STEP_LOG")" check "red control: the --cnp log check fails on MOSAIC0's log" not cnp_logged "${G}/MOSAIC0/mosaic/MOSAIC0_mocha.log" echo "mocha: MOSAIC planted CN-LOH chr20:1-20000000 at 0.15: $(cut -f1-7 <<< "$ROW" | tr '\t' ' ');" \ From e2af95e7b0201b18e25525ff76a76dae310c12f1 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 18:58:39 +0100 Subject: [PATCH 10/14] test(setup): expect the partial status from fake setups whose downloads fail their checksums The fake downloads fail their pinned checksums, so a plain fake setup is partial: it now exits 3 and prints the 'Setup finished: core ready' banner. The troubleshooting page names the new banner and the Missing list, and the retry commands quote the script path so a path with a space pastes as is. --- docs/troubleshooting.md | 2 +- scripts/setup.sh | 25 +++++++++++-------- .../base-setup-clinvar-raw-only.sh | 2 +- tests/fake-docker/base-setup-no-wget.sh | 2 +- tests/fake-docker/base-setup.sh | 6 ++--- tests/fake-docker/bash-step-setup-data.sh | 2 +- tests/fake-docker/reference-setup.sh | 2 +- tests/fake-docker/setup-clinical-default.sh | 4 +-- tests/fake-docker/setup-gencode-one-fetch.sh | 4 +-- tests/fake-docker/single-source-downloads.sh | 2 +- 10 files changed, 27 insertions(+), 24 deletions(-) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 7d0c7f37a..84ff24c47 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -738,7 +738,7 @@ ExpansionHunter \ ### Clinical data skipped during setup Setup installs VEP 116, the PCGR/CPSR bundle and VEP 115 by default. -If one is missing, read the warning above `Setup complete!`. +If one is missing, setup ends with `Setup finished: core ready, N items missing` and lists it under `Missing` with its retry command. The warning above it says why. The disk checks require 56 GB, 15 GB and 52 GB peak respectively, each plus a 5 GB margin. The disk space occupied by a partial download counts toward that peak. Sparse file holes do not count. Setup removes stale extraction files before checking space and checks a retained PCGR archive against its pinned SHA-256 before reuse. diff --git a/scripts/setup.sh b/scripts/setup.sh index 38336acfb..85deff657 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -238,6 +238,9 @@ if [ -n "$PCGR_SHA_OVERRIDE" ]; then PCGR_DATA_SHA256=$PCGR_SHA_OVERRIDE; fi # The summary lists them, and a run with any left exits SETUP_PARTIAL_EXIT. MISSING=() SETUP_PARTIAL_EXIT=3 +# The retry commands quote the script path, so a checkout path with a space +# still copies and pastes as one word. +SETUP_CMD=$(printf '%q' "$0") note_missing() { MISSING+=("$1|$2"); } # NCBI's GRCh38 analysis set without ALT contigs: chr1-22, X, Y, M, the @@ -2145,7 +2148,7 @@ install_clinical_data() { missing=1 fi [ -d "${GENOME_DIR}/pcgr_data/${PCGR_DATA_BUNDLE}/data" ] \ - || note_missing "PCGR/CPSR bundle ${PCGR_DATA_BUNDLE} (step 17)" "$0 --clinical-data \"${GENOME_DIR}\"" + || note_missing "PCGR/CPSR bundle ${PCGR_DATA_BUNDLE} (step 17)" "${SETUP_CMD} --clinical-data \"${GENOME_DIR}\"" else [ ! -f "${dir}/homo_sapiens/${rel}_GRCh38/info.txt" ] || { echo "[OK] VEP ${rel} cache already present."; continue; } peak=56 @@ -2163,9 +2166,9 @@ install_clinical_data() { fi if [ ! -f "${dir}/homo_sapiens/${rel}_GRCh38/info.txt" ]; then if [ "$rel" = "$VEP_CACHE_RELEASE" ]; then - note_missing "VEP ${rel} cache (steps 13, 30, 23 and 31)" "$0 --vep-cache \"${GENOME_DIR}\"" + note_missing "VEP ${rel} cache (steps 13, 30, 23 and 31)" "${SETUP_CMD} --vep-cache \"${GENOME_DIR}\"" else - note_missing "VEP ${rel} cache (step 17, CPSR)" "$0 --clinical-data \"${GENOME_DIR}\"" + note_missing "VEP ${rel} cache (step 17, CPSR)" "${SETUP_CMD} --clinical-data \"${GENOME_DIR}\"" fi fi fi @@ -2481,7 +2484,7 @@ elif "${PGP_ROOT}/rust/og.sh" install; then echo "[OK] og (bin/.og) built." else echo "[WARN] Could not build og. Steps 04, 05, 07, 25 and 33 stop without it; run: $0 --og" - note_missing "og, OwnGenome's own binary (steps 04, 05, 07, 25 and 33)" "$0 --og" + note_missing "og, OwnGenome's own binary (steps 04, 05, 07, 25 and 33)" "${SETUP_CMD} --og" fi ############################################################################### @@ -2506,7 +2509,7 @@ else else rm -f "$DICT_PART" echo "[WARN] Could not create ${REF_DICT}. Steps 03a, 20, 29 and chip-to-vcf need it; re-run setup.sh." - note_missing "the sequence dictionary ${REF_DICT##*/} (steps 03a, 20, 29 and chip-to-vcf)" "$0 \"${GENOME_DIR}\"" + note_missing "the sequence dictionary ${REF_DICT##*/} (steps 03a, 20, 29 and chip-to-vcf)" "${SETUP_CMD} \"${GENOME_DIR}\"" fi fi @@ -2528,7 +2531,7 @@ for name in $DATA_FILES; do echo "[OK] ${what}: ${dest}" else echo "[WARN] Could not install the ${what}. Re-run setup.sh to try again." - note_missing "the ${what}" "$0 \"${GENOME_DIR}\"" + note_missing "the ${what}" "${SETUP_CMD} \"${GENOME_DIR}\"" fi done # gnomAD v3.1's chrM sites (step 20 annotates the calls with them), checked @@ -2544,17 +2547,17 @@ else echo "[OK] gnomAD v3.1 mitochondrial sites (step 20): ${GNOMAD_MITO}" else echo "[WARN] Could not install gnomAD's mitochondrial sites. Step 20 runs without its gnomAD columns; re-run setup.sh to try again." - note_missing "gnomAD v3.1 mitochondrial sites (step 20)" "$0 \"${GENOME_DIR}\"" + note_missing "gnomAD v3.1 mitochondrial sites (step 20)" "${SETUP_CMD} \"${GENOME_DIR}\"" fi fi # A failed download does not stop setup here either: it goes on the list of # what is missing, which the summary prints with the command that retries it. install_sample_qc_data \ - || note_missing "somalier's sites and VerifyBamID2's panel (step 33)" "$0 --sample-qc-data \"${GENOME_DIR}\"" + || note_missing "somalier's sites and VerifyBamID2's panel (step 33)" "${SETUP_CMD} --sample-qc-data \"${GENOME_DIR}\"" install_strchive_data \ - || note_missing "STRchive ${STRCHIVE_VERSION} catalog (steps 09 and 9b)" "$0 --strchive-data \"${GENOME_DIR}\"" + || note_missing "STRchive ${STRCHIVE_VERSION} catalog (steps 09 and 9b)" "${SETUP_CMD} --strchive-data \"${GENOME_DIR}\"" install_tandem_repeats \ - || note_missing "GIAB tandem-repeat map (step 04c)" "$0 \"${GENOME_DIR}\"" + || note_missing "GIAB tandem-repeat map (step 04c)" "${SETUP_CMD} \"${GENOME_DIR}\"" # Step 25: its scores, pgsc_calc and its plugin (warnings only). install_prs @@ -2568,7 +2571,7 @@ echo "" echo "=== Phase 5: AnnotSV annotation data (~5.3 GB + ~1.7 GB download, ~20 GB unpacked) ===" install_annotsv_data \ - || note_missing "AnnotSV data: the bundle, gnomAD-SV ${GNOMAD_SV_VERSION} and MANE ${MANE_RELEASE} (step 05)" "$0 --annotsv-data \"${GENOME_DIR}\"" + || note_missing "AnnotSV data: the bundle, gnomAD-SV ${GNOMAD_SV_VERSION} and MANE ${MANE_RELEASE} (step 05)" "${SETUP_CMD} --annotsv-data \"${GENOME_DIR}\"" ############################################################################### # Phase 6: Clinical reference data (default) diff --git a/tests/fake-docker/base-setup-clinvar-raw-only.sh b/tests/fake-docker/base-setup-clinvar-raw-only.sh index d66ea427a..7953e208b 100755 --- a/tests/fake-docker/base-setup-clinvar-raw-only.sh +++ b/tests/fake-docker/base-setup-clinvar-raw-only.sh @@ -18,4 +18,4 @@ awk '/^run / && /clinvar_chr\.vcf\.gz/ && !/clinvar_pathogenic/ { found = 1 } EN docker_log_has '^run .*clinvar_pathogenic_chr\.vcf\.gz' \ "setup.sh did not build clinvar_pathogenic_chr.vcf.gz" output_lacks setup 'unbound variable' -expect_rc setup 0 +expect_rc setup 3 diff --git a/tests/fake-docker/base-setup-no-wget.sh b/tests/fake-docker/base-setup-no-wget.sh index 18eda13a9..686a8cb5c 100755 --- a/tests/fake-docker/base-setup-no-wget.sh +++ b/tests/fake-docker/base-setup-no-wget.sh @@ -13,4 +13,4 @@ docker_log_has '^curl [^ ]*\.fa' "setup.sh did not download the reference with c docker_log_has '^curl [^ ]*clinvar\.vcf\.gz ' "setup.sh did not download ClinVar with curl" output_lacks setup 'wget: command not found' output_lacks setup 'unbound variable' -expect_rc setup 0 +expect_rc setup 3 diff --git a/tests/fake-docker/base-setup.sh b/tests/fake-docker/base-setup.sh index 82c7ee9dd..6c2315c86 100755 --- a/tests/fake-docker/base-setup.sh +++ b/tests/fake-docker/base-setup.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # setup.sh on an empty GENOME_DIR: the reference and ClinVar downloads go # through the fake wget, the image checks through the fake docker. It must -# finish with exit 0. +# finish with exit 3, a partial setup: the fake downloads fail their pinned checksums. # shellcheck source=../../scripts/ci/fake-docker/lib.sh . "${REPO_ROOT:?}/scripts/ci/fake-docker/lib.sh" seed_clinical_downloads @@ -9,9 +9,9 @@ seed_clinical_downloads G="${CASE_WORK}/genome" mkdir -p "$G" -run_expect 0 setup "${SCRIPTS}/setup.sh" "$G" +run_expect 3 setup "${SCRIPTS}/setup.sh" "$G" output_lacks setup 'unbound variable' -output_has setup 'Setup complete!' +output_has setup '^ Setup finished: core ready, [0-9]+ items missing$' for f in reference/GRCh38_no_alt_analysis_set.fasta reference/GRCh38_no_alt_analysis_set.fasta.fai \ clinvar/clinvar.vcf.gz clinvar/clinvar.vcf.gz.tbi \ diff --git a/tests/fake-docker/bash-step-setup-data.sh b/tests/fake-docker/bash-step-setup-data.sh index 7282da89b..1849987eb 100755 --- a/tests/fake-docker/bash-step-setup-data.sh +++ b/tests/fake-docker/bash-step-setup-data.sh @@ -41,7 +41,7 @@ run_expect 1 chip-no-dict "${SCRIPTS}/chip-to-vcf.sh" chip1 output_has chip-no-dict 'Required file not found: .*GRCh38_no_alt_analysis_set\.dict' if grep -q '^run ' "$FAKE_DOCKER_LOG"; then fail "chip-to-vcf.sh started a container without the sequence dictionary"; fi -run_expect 0 setup "${SCRIPTS}/setup.sh" "$GENOME_DIR" +run_expect 3 setup "${SCRIPTS}/setup.sh" "$GENOME_DIR" docker_log_has '^run image=[^ ]*gatk.* gatk CreateSequenceDictionary -R /genome/reference/GRCh38_no_alt_analysis_set\.fasta ' \ "setup.sh did not create the sequence dictionary" [ -f "${GENOME_DIR}/reference/GRCh38_no_alt_analysis_set.dict" ] || fail "setup.sh left no GRCh38_no_alt_analysis_set.dict" diff --git a/tests/fake-docker/reference-setup.sh b/tests/fake-docker/reference-setup.sh index 67cbba447..455d6c2fc 100755 --- a/tests/fake-docker/reference-setup.sh +++ b/tests/fake-docker/reference-setup.sh @@ -36,7 +36,7 @@ serve() { G="${CASE_WORK}/genome" mkdir -p "$G" serve "${CASE_WORK}/served" -FAKE_DOWNLOAD_DIR="${CASE_WORK}/served" run_expect 0 setup "${SCRIPTS}/setup.sh" "$G" +FAKE_DOWNLOAD_DIR="${CASE_WORK}/served" run_expect 3 setup "${SCRIPTS}/setup.sh" "$G" docker_log_has "^curl ${NCBI//./\\.}/${NAME//./\\.}\\.gz -> " "setup.sh did not download the no-ALT analysis set from NCBI" docker_log_has "^curl ${NCBI//./\\.}/md5checksums\\.txt -> " "setup.sh did not read NCBI's md5checksums.txt" docker_log_has "^curl ${NCBI//./\\.}/${NAME//./\\.}\\.fai -> " "setup.sh did not download NCBI's .fai" diff --git a/tests/fake-docker/setup-clinical-default.sh b/tests/fake-docker/setup-clinical-default.sh index 4c6b49e46..618309cfc 100644 --- a/tests/fake-docker/setup-clinical-default.sh +++ b/tests/fake-docker/setup-clinical-default.sh @@ -5,8 +5,8 @@ seed_clinical_downloads export FETCH_TRIES=1 FETCH_WAIT=0 G="${CASE_WORK}/without" -run_expect 0 without "${SCRIPTS}/setup.sh" --no-clinical-data "$G" -output_has without 'Setup complete!' +run_expect 3 without "${SCRIPTS}/setup.sh" --no-clinical-data "$G" +output_has without 'Setup finished: core ready' if grep -E '^(wget|curl).*homo_sapiens_vep_|^(wget|curl).*pcgr_ref_data' "$FAKE_DOCKER_LOG"; then fail '--no-clinical-data downloaded clinical data' fi diff --git a/tests/fake-docker/setup-gencode-one-fetch.sh b/tests/fake-docker/setup-gencode-one-fetch.sh index 824bdbdc4..0a3c1cf9f 100755 --- a/tests/fake-docker/setup-gencode-one-fetch.sh +++ b/tests/fake-docker/setup-gencode-one-fetch.sh @@ -42,7 +42,7 @@ fetches() { grep -cE "^(curl|wget) [^ ]*/${GTF//./\\.} " "$FAKE_DOCKER_LOG" || t # --- 1. one fetch, both files --------------------------------------------------------- : > "$FAKE_DOCKER_LOG" -FAKE_DOWNLOAD_DIR="$SERVED" run_expect 0 setup "${SCRIPTS}/setup.sh" "$GENOME_DIR" +FAKE_DOWNLOAD_DIR="$SERVED" run_expect 3 setup "${SCRIPTS}/setup.sh" "$GENOME_DIR" [ "$(fetches)" -eq 1 ] || fail "setup.sh downloaded ${GTF} $(fetches) times, expected once" [ -s "$GENES" ] || fail "setup.sh wrote no ${GENES}" [ -s "$EXONS" ] || fail "setup.sh wrote no ${EXONS}" @@ -65,7 +65,7 @@ touch -t 202001010000 "$GENES" run_rc validate-missing "${SCRIPTS}/validate-setup.sh" sample1 output_has validate-missing "GENCODE ${REL} exons \(steps 06 and 17 skip their cDNA read check without them\) not found at: ${EXONS//./\\.}" : > "$FAKE_DOCKER_LOG" -FAKE_DOWNLOAD_DIR="$SERVED" run_expect 0 setup-again "${SCRIPTS}/setup.sh" "$GENOME_DIR" +FAKE_DOWNLOAD_DIR="$SERVED" run_expect 3 setup-again "${SCRIPTS}/setup.sh" "$GENOME_DIR" [ "$(fetches)" -eq 1 ] || fail "the second setup.sh downloaded ${GTF} $(fetches) times, expected once" [ "$(cat "$EXONS" 2>/dev/null)" = "$want" ] || fail "the second setup.sh did not rebuild the exon BED" [ -z "$(find "$GENES" -newer "${SERVED}/MD5SUMS")" ] || fail "the second setup.sh rewrote the gene file it already had" diff --git a/tests/fake-docker/single-source-downloads.sh b/tests/fake-docker/single-source-downloads.sh index 42ec1bf6f..c26bbef41 100755 --- a/tests/fake-docker/single-source-downloads.sh +++ b/tests/fake-docker/single-source-downloads.sh @@ -25,7 +25,7 @@ FAKE_DOWNLOAD_PARTIAL='clinvar\.vcf\.gz$' run_rc setup-dropped "${SCRIPTS}/setup [ ! -e "${G}/clinvar/clinvar.vcf.gz" ] || fail "a cut download was stored as clinvar.vcf.gz" : > "$FAKE_DOCKER_LOG" -run_expect 0 setup-again "${SCRIPTS}/setup.sh" "$G" +run_expect 3 setup-again "${SCRIPTS}/setup.sh" "$G" docker_log_has '^curl [^ ]*/clinvar\.vcf\.gz -> ' "the second run did not download ClinVar again" [ -s "${G}/clinvar/clinvar.vcf.gz" ] || fail "the second run left no clinvar.vcf.gz" [ ! -e "${G}/clinvar/clinvar.vcf.gz.part" ] || fail "clinvar.vcf.gz.part is still there after a complete download" From 2817f5980c8ff112b49b357a7d27fec122e5644d Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 22:12:02 +0100 Subject: [PATCH 11/14] docs: say og's atomic writer, not every og write, is owner-only --- CHANGELOG.md | 2 +- docs/nextflow.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 46c499382..1e09bc81a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased -- The files the `og` binary writes are owner-only (mode 0600) whatever the umask. Before, `og` wrote each output under a temporary name with the umask's mode, so under the common umask 022 an output was readable by every user of the machine, and replacing a 0600 file made it 0644. The step scripts set umask 077, so a run through them was already owner-only; `og` run by hand or by the Nextflow pipeline was not. Now the temporary file is created mode 0600, or with the mode of the file it replaces when that is stricter (a 0400 file stays 0400). It is always a new file: a file or link already at its name is left as it is and the next name is tried, and two outputs written at once to one name in one process get two temporary files. The temporary name gains a number, `..tmp--`. No output name or content changes. [Nextflow Execution](docs/nextflow.md#run-the-pipeline) now says to keep `--outdir` private. +- The files `og` writes through its atomic writer (every analysis output) are owner-only (mode 0600) whatever the umask. Before, `og` wrote each output under a temporary name with the umask's mode, so under the common umask 022 an output was readable by every user of the machine, and replacing a 0600 file made it 0644. The step scripts set umask 077, so a run through them was already owner-only; `og` run by hand or by the Nextflow pipeline was not. Now the temporary file is created mode 0600, or with the mode of the file it replaces when that is stricter (a 0400 file stays 0400). It is always a new file: a file or link already at its name is left as it is and the next name is tried, and two outputs written at once to one name in one process get two temporary files. The temporary name gains a number, `..tmp--`. No output name or content changes. [Nextflow Execution](docs/nextflow.md#run-the-pipeline) now says to keep `--outdir` private. - The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes. - The phased family VCF is indexed by the Rust `og` binary. WhatsHap's step (step 38 and the Nextflow WHATSHAP_PED process) wrote `family.phased.vcf.gz.tbi` with a Python one-liner, `pysam.tabix_index`. It now runs `og vcf index -f`. The `.tbi` keeps its name and format, and its bytes, once inflated, are the ones htslib 1.24 writes (the htslib of the pysam and bcftools the pipeline pins): the same bins in the same order, the same chunks and the same linear index. Only the deflate stream can differ, as two deflate libraries differ. On a 5-million-record three-sample VCF it took 0.75 to 0.91 s and 31 MB at peak, where htslib's own `tabix` took 0.81 to 0.93 s and 19 MB. - As in htslib 1.24, a record ends at the largest of INFO/END, the REF length, the SVLEN of a ``, ``, `` or `` ALT, and a sample's LEN on a gVCF block. A line with no position to read, unsorted records, a sequence that starts again, a record past 2^29 and an empty file stop the index with a message, and nothing is written. Without `-f` it never writes over an index, even one that appears while it runs. diff --git a/docs/nextflow.md b/docs/nextflow.md index 8e6d79fca..ef5ddefad 100644 --- a/docs/nextflow.md +++ b/docs/nextflow.md @@ -91,7 +91,7 @@ nextflow run main.nf \ # it names (two or more, each with a gVCF; docs/multi-sample.md) ``` -**Keep `--outdir` private.** The results hold genome data. Create the folder before the run with `mkdir -m 700 results`, or put it in a folder only you can open. The `og` tools write their files owner-only (mode 0600), but Nextflow creates the folders, and the other tools write their files, with the modes your umask gives. +**Keep `--outdir` private.** The results hold genome data. Create the folder before the run with `mkdir -m 700 results`, or put it in a folder only you can open. The `og` tools write their analysis outputs owner-only (mode 0600), but Nextflow creates the folders, and the other tools write their files, with the modes your umask gives. **Family workflow (opt-in).** `--pedigree` (tab separated: sample, sex, father, mother, partner; [`assets/pedigree.example.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/pedigree.example.tsv)) runs `workflows/family.nf` over the samplesheet rows it names. `--pedigree` may be a families file (a leading family column, [`assets/families.example.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/families.example.tsv)). `--family_name` names the family to run, which a file with more than one family needs, and the folder its results go to (`//` instead of `/family/`). The workflow does the same work as step 38: `FAMILY_PREPARE` checks the pedigree and that each member's gVCF names its own sample, `GLNEXUS` and `GLNEXUS_VCF` joint-call the gVCFs (`GLNEXUS` asks for 16 GB and gives GLnexus half of its memory as `--mem-gbytes`), `FAMILY_RELATE` and `FAMILY_CHECK` stop the run when somalier's relatedness of a pair disagrees with the pedigree (a sample swap), `WHATSHAP_PED` phases the children of complete trios by inheritance, and `FAMILY_REPORT` writes each child's allele origin, de novo candidates when both parents are sequenced, and compound heterozygotes among the clinical filter's candidates. With `--ancestry_fine_panel` (the panel folder of step 26b, `setup.sh --ancestry-fine-panel`), four more processes find the DNA each pair shares by descent, once `FAMILY_CHECK` has passed: `FAMILY_IBD_SITES` keeps the joint call's genotypes at the panel's SNVs on each autosome (a chromosome with fewer than 100 is left out), `FAMILY_IBD_PHASE` phases them with Beagle against the panel, `FAMILY_IBD_SEGMENTS` runs hap-ibd (its default 2 cM minimum) with the panel's genetic maps, and `FAMILY_IBD_REPORT` writes `ibd_pairs.tsv`, `ibd_segments.tsv` and `ibd_report.md`. They feed only the report, so a failure of one is logged and the run goes on. Without the panel none of them runs. Each member needs a gVCF: DEEPVARIANT's for a FASTQ or BAM row, the `gvcf` column for a VCF row (the run stops on a VCF row without one). Fewer than two named samples, or no `--somalier_sites`, stops the run at the start. When the pedigree marks two of those samples as each other's partner, `COUPLE_CARRIER` runs the couple carrier screen of step 39 on them ([Couple Carrier Screen](39-couple.md)): it reads each partner's ClinVar hits, so it needs `clinvar` in `--tools` (without it the run logs a warning and skips the screen), uses deletions over an exon only with `--gencode_exons` (the GENCODE exons BED `setup.sh` installs as `reference/gencode.v50.basic.exons.bed`), reads Delly's calls, ExpansionHunter's, and Parascopy's SMN1, interval and CYP21A2 tables when those steps ran, and, when `--annotsv_annotations` holds AnnotSV's benign loss set (`benign_Loss_SV_GRCh38.sorted.bed`, with gnomAD-SV's common deletions merged in by `setup.sh --annotsv-data`), sets aside a Delly deletion that matches a common one there. Without `--pedigree` none of these processes exists and a run is unchanged. What it checks and what the files hold: [Multi-sample comparison](multi-sample.md#family-analysis-step-38-opt-in). From 5ed22f2374d0f6001b088cc11b466dee0c3e44b8 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 22:19:04 +0100 Subject: [PATCH 12/14] fix(lookup): stream past an older index only when it lacks the chromosome A Nextflow outdir copies the small .tbi before the large VCF, so a fresh index often carries the earlier time. Treating every older index as stale streamed whole genomes for every region there. An older index is now used when it lists the queried chromosome; only one that does not is skipped, with a note. --- CHANGELOG.md | 2 +- docs/lookup.md | 2 +- rust/crates/og-lookup/src/vcf.rs | 40 ++++++++++++------- .../og-lookup/tests/rv_lookup_stale_index.rs | 23 ++++++++--- 4 files changed, 45 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e424d9034..e099a779e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased -- `og lookup` no longer trusts an index older than its VCF. A gVCF or annotated VCF replaced by hand after its `.tbi` or `.csi` was built kept the old index, and a chromosome the new file added was missing from it, so a variant called there read as "no call". The lookup now compares the two modification times, as the blood-type step already does. When the index is older it reads the whole file and prints `NOTE: the index ... is older than ...; reading the whole file`. A fresh index is used as before, and no answer or output file changes. +- `og lookup` no longer trusts an index older than its VCF for a chromosome the index does not list. A gVCF or annotated VCF replaced by hand after its `.tbi` or `.csi` was built kept the old index, and a chromosome the new file added was missing from it, so a variant called there read as "no call". The lookup now compares the two modification times, as the blood-type step does. When the index is older and does not list the queried chromosome, it reads the whole file and prints `NOTE: the index ... is older than ... and does not list chrN; reading the whole file`. An older index that lists the chromosome is still read: in a Nextflow output directory the small index is often copied before its large VCF, so it carries the earlier time. A fresh index is used as before, and no answer or output file changes. - The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes. - The phased family VCF is indexed by the Rust `og` binary. WhatsHap's step (step 38 and the Nextflow WHATSHAP_PED process) wrote `family.phased.vcf.gz.tbi` with a Python one-liner, `pysam.tabix_index`. It now runs `og vcf index -f`. The `.tbi` keeps its name and format, and its bytes, once inflated, are the ones htslib 1.24 writes (the htslib of the pysam and bcftools the pipeline pins): the same bins in the same order, the same chunks and the same linear index. Only the deflate stream can differ, as two deflate libraries differ. On a 5-million-record three-sample VCF it took 0.75 to 0.91 s and 31 MB at peak, where htslib's own `tabix` took 0.81 to 0.93 s and 19 MB. - As in htslib 1.24, a record ends at the largest of INFO/END, the REF length, the SVLEN of a ``, ``, `` or `` ALT, and a sample's LEN on a gVCF block. A line with no position to read, unsorted records, a sequence that starts again, a record past 2^29 and an empty file stop the index with a message, and nothing is written. Without `-f` it never writes over an index, even one that appears while it runs. diff --git a/docs/lookup.md b/docs/lookup.md index 83b5e3d5d..136cea1d6 100644 --- a/docs/lookup.md +++ b/docs/lookup.md @@ -87,7 +87,7 @@ Two ClinVar lines can appear. `ClinVar (pipeline's ClinVar file)` comes from the ## Runtime -One container starts for the whole command, whatever the number of queries. With indexed files a site or a gene takes well under a second inside it: only the region is read. An rsID that is in neither the site table nor the pipeline tables makes the command read the whole annotated VCF once, for all such rsIDs of the command line together, on two cores. A file without a `.tbi` or `.csi` index is streamed whole for each region, with a note: the answer is the same and only the records in the region are kept in memory, but on a whole genome it takes minutes. `bcftools index -t` makes it fast. An index older than its file, as when the file was replaced after the index was built, is not used: it may lack a chromosome the new file has, so the file is streamed whole, with a note. `bcftools index -f -t` rebuilds the index. +One container starts for the whole command, whatever the number of queries. With indexed files a site or a gene takes well under a second inside it: only the region is read. An rsID that is in neither the site table nor the pipeline tables makes the command read the whole annotated VCF once, for all such rsIDs of the command line together, on two cores. A file without a `.tbi` or `.csi` index is streamed whole for each region, with a note: the answer is the same and only the records in the region are kept in memory, but on a whole genome it takes minutes. `bcftools index -t` makes it fast. An index older than its file, as when the file was replaced after the index was built, may lack a chromosome the new file has. So when such an index does not list the queried chromosome, the file is streamed whole, with a note; `bcftools index -f -t` rebuilds the index. An older index that lists the chromosome is still used: a Nextflow output directory often holds an index dated a little before its file, because the small index finishes copying first. ## Notes diff --git a/rust/crates/og-lookup/src/vcf.rs b/rust/crates/og-lookup/src/vcf.rs index 67959cfee..e21b280ad 100644 --- a/rust/crates/og-lookup/src/vcf.rs +++ b/rust/crates/og-lookup/src/vcf.rs @@ -287,24 +287,26 @@ pub struct Span<'a> { /// The records of `path` that overlap `span` (a reference block by its /// END, a deletion by its REF), in file order: through the `.tbi` or `.csi` -/// beside the file when there is one and it is not older than the file, -/// else by streaming the whole file, which `note` is told. +/// beside the file when there is one, else by streaming the whole file, +/// which `note` is told. An index older than the file that does not list +/// the region's chromosome is not trusted either: the file may have gained +/// that chromosome after the index was built. pub fn read(path: &str, span: Span, note: &mut dyn Write) -> Result { { let index = [".tbi", ".csi"] .iter() .map(|x| format!("{path}{x}")) .find(|p| Path::new(p).is_file()); - if let Some(index) = index.as_deref().filter(|i| older(i, path)) { - // A contig the file gained after its index was built is not in - // the index, so an indexed read would answer no call there. - let _ = writeln!( - note, - "NOTE: the index {index} is older than {path}; reading the whole file (bcftools index -f -t rebuilds it)." - ); - } else if let Some(index) = index.as_deref() { - match read_indexed(path, index, span) { - Ok(r) => return Ok(r), + if let Some(index) = index.as_deref() { + match read_indexed(path, index, span, older(index, path)) { + Ok(Some(r)) => return Ok(r), + Ok(None) => { + let _ = writeln!( + note, + "NOTE: the index {index} is older than {path} and does not list {}; reading the whole file (bcftools index -f -t rebuilds it).", + crate::query::contig_name(span.chrom) + ); + } Err(e) => { let _ = writeln!( note, @@ -339,8 +341,13 @@ fn overlaps(r: &Rec, span: Span) -> bool { } /// The header through the top of the file, then each index sequence that -/// names the region's chromosome, through the index. -fn read_indexed(path: &str, index: &str, span: Span) -> Result { +/// names the region's chromosome, through the index. `None` when the index +/// is `stale` (older than the file) and names no such sequence. +/// +/// Age alone does not condemn an index: Nextflow publishes a VCF and its +/// index as separate copies, and the small index finishes first, so a fresh +/// index often has the earlier time. +fn read_indexed(path: &str, index: &str, span: Span, stale: bool) -> Result, Error> { let file = og_io::open(path).map_err(|e| Error::io(&e, path))?; let mut header = Vec::new(); for line in Lines::new(file, Utf8::Replace) { @@ -361,6 +368,9 @@ fn read_indexed(path: &str, index: &str, span: Span) -> Result { .filter(|n| norm_chrom(n) == span.chrom) .cloned() .collect(); + if stale && names.is_empty() { + return Ok(None); + } let mut records = Vec::new(); for name in names { for line in reader.query(&name, span.start.saturating_sub(1), span.end) { @@ -374,7 +384,7 @@ fn read_indexed(path: &str, index: &str, span: Span) -> Result { } } } - Ok(Region { csq, records }) + Ok(Some(Region { csq, records })) } /// A data line of the region, or `None` for one outside it (or too short diff --git a/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs b/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs index ab6e56c56..ea42108a6 100644 --- a/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs +++ b/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs @@ -1,7 +1,9 @@ -//! An index older than its VCF is not trusted: a contig the VCF gained after -//! the index was built is not in the index, so an indexed read would answer -//! "no call" for a variant the file holds. The lookup reads the whole file -//! instead and says so; a fresh index is still used. +//! An index older than its VCF is not trusted for a contig it does not list: +//! the VCF may have gained that contig after the index was built, so an +//! indexed read would answer "no call" for a variant the file holds. The +//! lookup reads the whole file instead and says so. A fresh index, or an +//! older one that lists the contig (a Nextflow outdir copies the small index +//! before the large VCF), is still used. use std::fs::{self, File}; use std::io::Write; @@ -100,7 +102,7 @@ fn an_index_older_than_its_vcf_is_not_trusted() { assert_eq!(out, want, "the stale index hid the chrM call"); assert!( err.contains(&format!( - "NOTE: the index {} is older than {gz_s}; reading the whole file", + "NOTE: the index {} is older than {gz_s} and does not list chrM; reading the whole file", tbi.display() )), "{err}" @@ -121,5 +123,16 @@ fn an_index_older_than_its_vcf_is_not_trusted() { assert_eq!(out, want); assert!(err.is_empty(), "{err}"); + // The same full index dated before the file, as Nextflow leaves it when + // it copies the small index before the large VCF, is still read: it + // lists chrM, so the indexed path answers and nothing is streamed. + touch(&tbi, &gz, false); + let (out, err) = chrm_73(&gz_s); + assert_eq!(out, want); + assert!( + err.is_empty(), + "an older index that lists the contig is used: {err}" + ); + fs::remove_dir_all(&dir).unwrap(); } From 23dd4d278305a5b9f00470709d16e66c2b6c03d4 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Wed, 18 Nov 2026 16:09:19 +0100 Subject: [PATCH 13/14] fix(io): give og's owner-only files to the work folder's owner when og runs as root The docker profile runs containers as root over a work folder the host user owns. Nextflow's fixOwnership never hands the files back there: the biocontainers images ship a busybox chown without --from. A 0600 file og wrote as root was then unreadable to the user, and Nextflow could not publish it. The atomic writer now gives its temporary file to the owner and group of the nearest folder above it that root does not own. --- CHANGELOG.md | 2 +- docs/nextflow.md | 2 +- rust/crates/og-io/src/lib.rs | 82 ++++++++++++++++++++ rust/crates/og-io/tests/atomic_root_owner.rs | 70 +++++++++++++++++ 4 files changed, 154 insertions(+), 2 deletions(-) create mode 100644 rust/crates/og-io/tests/atomic_root_owner.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index ef7c25418..6255249b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased -- The files `og` writes through its atomic writer (every analysis output) are owner-only (mode 0600) whatever the umask. Before, `og` wrote each output under a temporary name with the umask's mode, so under the common umask 022 an output was readable by every user of the machine, and replacing a 0600 file made it 0644. The step scripts set umask 077, so a run through them was already owner-only; `og` run by hand or by the Nextflow pipeline was not. Now the temporary file is created mode 0600, or with the mode of the file it replaces when that is stricter (a 0400 file stays 0400). It is always a new file: a file or link already at its name is left as it is and the next name is tried, and two outputs written at once to one name in one process get two temporary files. The temporary name gains a number, `..tmp--`. No output name or content changes. [Nextflow Execution](docs/nextflow.md#run-the-pipeline) now says to keep `--outdir` private. +- The files `og` writes through its atomic writer (every analysis output) are owner-only (mode 0600) whatever the umask. Before, `og` wrote each output under a temporary name with the umask's mode, so under the common umask 022 an output was readable by every user of the machine, and replacing a 0600 file made it 0644. The step scripts set umask 077, so a run through them was already owner-only; `og` run by hand or by the Nextflow pipeline was not. Now the temporary file is created mode 0600, or with the mode of the file it replaces when that is stricter (a 0400 file stays 0400). It is always a new file: a file or link already at its name is left as it is and the next name is tried, and two outputs written at once to one name in one process get two temporary files. The temporary name gains a number, `..tmp--`. When `og` runs as root, each of these files goes to the owner and group of the nearest folder above it that root does not own; when every folder up to `/` is root's, nothing changes. The docker profile of the Nextflow pipeline runs its containers as root over a work folder the host user owns; Nextflow's `docker.fixOwnership` does not reach these files, because the biocontainers images ship a `chown` without `--from`. Without the hand-over the host user could not read an output `og` wrote there, and Nextflow could not publish it. No output name or content changes. [Nextflow Execution](docs/nextflow.md#run-the-pipeline) now says to keep `--outdir` private. - A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again. - VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37). - The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2). diff --git a/docs/nextflow.md b/docs/nextflow.md index ef5ddefad..56e6fa044 100644 --- a/docs/nextflow.md +++ b/docs/nextflow.md @@ -91,7 +91,7 @@ nextflow run main.nf \ # it names (two or more, each with a gVCF; docs/multi-sample.md) ``` -**Keep `--outdir` private.** The results hold genome data. Create the folder before the run with `mkdir -m 700 results`, or put it in a folder only you can open. The `og` tools write their analysis outputs owner-only (mode 0600), but Nextflow creates the folders, and the other tools write their files, with the modes your umask gives. +**Keep `--outdir` private.** The results hold genome data. Create the folder before the run with `mkdir -m 700 results`, or put it in a folder only you can open. The `og` tools write their analysis outputs owner-only (mode 0600; run as root in a container, they give each file to the owner of the work folder it sits in, so you can read it), but Nextflow creates the folders, and the other tools write their files, with the modes your umask gives. **Family workflow (opt-in).** `--pedigree` (tab separated: sample, sex, father, mother, partner; [`assets/pedigree.example.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/pedigree.example.tsv)) runs `workflows/family.nf` over the samplesheet rows it names. `--pedigree` may be a families file (a leading family column, [`assets/families.example.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/families.example.tsv)). `--family_name` names the family to run, which a file with more than one family needs, and the folder its results go to (`//` instead of `/family/`). The workflow does the same work as step 38: `FAMILY_PREPARE` checks the pedigree and that each member's gVCF names its own sample, `GLNEXUS` and `GLNEXUS_VCF` joint-call the gVCFs (`GLNEXUS` asks for 16 GB and gives GLnexus half of its memory as `--mem-gbytes`), `FAMILY_RELATE` and `FAMILY_CHECK` stop the run when somalier's relatedness of a pair disagrees with the pedigree (a sample swap), `WHATSHAP_PED` phases the children of complete trios by inheritance, and `FAMILY_REPORT` writes each child's allele origin, de novo candidates when both parents are sequenced, and compound heterozygotes among the clinical filter's candidates. With `--ancestry_fine_panel` (the panel folder of step 26b, `setup.sh --ancestry-fine-panel`), four more processes find the DNA each pair shares by descent, once `FAMILY_CHECK` has passed: `FAMILY_IBD_SITES` keeps the joint call's genotypes at the panel's SNVs on each autosome (a chromosome with fewer than 100 is left out), `FAMILY_IBD_PHASE` phases them with Beagle against the panel, `FAMILY_IBD_SEGMENTS` runs hap-ibd (its default 2 cM minimum) with the panel's genetic maps, and `FAMILY_IBD_REPORT` writes `ibd_pairs.tsv`, `ibd_segments.tsv` and `ibd_report.md`. They feed only the report, so a failure of one is logged and the run goes on. Without the panel none of them runs. Each member needs a gVCF: DEEPVARIANT's for a FASTQ or BAM row, the `gvcf` column for a VCF row (the run stops on a VCF row without one). Fewer than two named samples, or no `--somalier_sites`, stops the run at the start. When the pedigree marks two of those samples as each other's partner, `COUPLE_CARRIER` runs the couple carrier screen of step 39 on them ([Couple Carrier Screen](39-couple.md)): it reads each partner's ClinVar hits, so it needs `clinvar` in `--tools` (without it the run logs a warning and skips the screen), uses deletions over an exon only with `--gencode_exons` (the GENCODE exons BED `setup.sh` installs as `reference/gencode.v50.basic.exons.bed`), reads Delly's calls, ExpansionHunter's, and Parascopy's SMN1, interval and CYP21A2 tables when those steps ran, and, when `--annotsv_annotations` holds AnnotSV's benign loss set (`benign_Loss_SV_GRCh38.sorted.bed`, with gnomAD-SV's common deletions merged in by `setup.sh --annotsv-data`), sets aside a Delly deletion that matches a common one there. Without `--pedigree` none of these processes exists and a run is unchanged. What it checks and what the files hold: [Multi-sample comparison](multi-sample.md#family-analysis-step-38-opt-in). diff --git a/rust/crates/og-io/src/lib.rs b/rust/crates/og-io/src/lib.rs index 4901ed3a3..8c8d73b49 100644 --- a/rust/crates/og-io/src/lib.rs +++ b/rust/crates/og-io/src/lib.rs @@ -205,6 +205,12 @@ impl Iterator for Lines { /// name is left alone and the next name is tried. On Unix it is created /// mode 0600, or with the mode of the target it replaces when that is /// stricter, so the umask never makes an output readable by others. +/// +/// When og runs as root, the temporary file is handed to the owner and +/// group of the nearest folder above it that root does not own. That is a +/// container run as root over a work folder bind-mounted from the host, as +/// the pipeline's docker profile does: without it the host user could not +/// read an owner-only output og wrote there. pub struct AtomicFile { tmp: PathBuf, target: PathBuf, @@ -235,6 +241,7 @@ impl AtomicFile { let tmp = target.with_file_name(tmp_name); match new_file(&tmp, mode) { Ok(f) => { + give_to_folder_owner(&f, &tmp); return Ok(AtomicFile { tmp, target, @@ -282,6 +289,54 @@ fn temp_mode(_target: &Path) -> u32 { 0 } +/// Hands a file og created as root to the owner and group of the nearest +/// folder above it that root does not own: the work folder a container run +/// as root got from the host, even through a subfolder made as root inside +/// it. Nothing changes when og is not root or every folder up to `/` is +/// root's. Best effort: where root may not change owners, the file stays +/// root's. +#[cfg(unix)] +fn give_to_folder_owner(file: &File, path: &Path) { + use std::os::unix::fs::MetadataExt; + let Ok(f) = file.metadata() else { + return; + }; + if f.uid() != 0 { + return; + } + let dir = match path.parent() { + Some(d) if !d.as_os_str().is_empty() => d, + _ => Path::new("."), + }; + let Ok(dir) = fs::canonicalize(dir) else { + return; + }; + let owners = dir + .ancestors() + .map_while(|d| fs::metadata(d).ok()) + .map(|m| (m.uid(), m.gid())); + if let Some((uid, gid)) = hand_over_to(f.uid(), owners) { + let _ = std::os::unix::fs::fchown(file, Some(uid), Some(gid)); + } +} + +#[cfg(not(unix))] +fn give_to_folder_owner(_file: &File, _path: &Path) {} + +/// The owner and group a new file goes to, given the owners of its folders +/// from the nearest up: the first folder not owned by root, when the file +/// is root's (og runs as root); otherwise none. +#[cfg(unix)] +fn hand_over_to( + file_uid: u32, + mut folders: impl Iterator, +) -> Option<(u32, u32)> { + if file_uid != 0 { + return None; + } + folders.find(|&(uid, _)| uid != 0) +} + /// Creates `path`, failing with `AlreadyExists` when anything is there, /// a link included (`O_CREAT | O_EXCL`). fn new_file(path: &Path, mode: u32) -> io::Result { @@ -418,6 +473,33 @@ mod tests { } } +#[cfg(all(test, unix))] +mod owner_tests { + use super::hand_over_to; + + #[test] + fn root_in_a_user_folder_hands_the_file_over() { + assert_eq!( + hand_over_to(0, [(1001, 118), (0, 0)].into_iter()), + Some((1001, 118)) + ); + // Through a subfolder made as root inside the user's folder. + assert_eq!( + hand_over_to(0, [(0, 0), (1001, 118), (0, 0)].into_iter()), + Some((1001, 118)) + ); + } + + #[test] + fn otherwise_the_owner_stays() { + // og as root where every folder is root's, and og as a user. + assert_eq!(hand_over_to(0, [(0, 0), (0, 0)].into_iter()), None); + assert_eq!(hand_over_to(0, std::iter::empty()), None); + assert_eq!(hand_over_to(1001, [(1001, 118)].into_iter()), None); + assert_eq!(hand_over_to(1001, [(0, 0), (4321, 1)].into_iter()), None); + } +} + #[cfg(test)] mod bounded_line_tests { use super::*; diff --git a/rust/crates/og-io/tests/atomic_root_owner.rs b/rust/crates/og-io/tests/atomic_root_owner.rs new file mode 100644 index 000000000..7853bb477 --- /dev/null +++ b/rust/crates/og-io/tests/atomic_root_owner.rs @@ -0,0 +1,70 @@ +//! Run as root, [`og_io::AtomicFile`] hands what it writes in another user's +//! folder, or in a subfolder root made inside it, to that user, as in a +//! container run as root over a work folder bind-mounted from the host. The output stays owner-only, and the user who +//! owns the folder can read it. +//! +//! The test needs root, so it runs in a container (`docker run --user root`). +//! Run as any other user it prints that it was skipped, unless +//! `OG_IO_ROOT_TEST=1` is set: then it fails, so a run meant to check this +//! cannot pass without checking it. + +use og_io::AtomicFile; +use std::fs; +use std::io::Write; +use std::os::unix::fs::{MetadataExt, PermissionsExt, chown}; +use std::path::Path; + +fn write(target: &Path) { + let mut f = AtomicFile::create(target).unwrap(); + f.write_all(b"new\n").unwrap(); + f.commit().unwrap(); +} + +#[test] +fn root_hands_the_file_to_the_folder_owner() { + let dir = std::env::temp_dir().join(format!("og-io-root-{}", std::process::id())); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let probe = dir.join("probe"); + fs::write(&probe, "").unwrap(); + if fs::metadata(&probe).unwrap().uid() != 0 { + fs::remove_dir_all(&dir).unwrap(); + assert!( + std::env::var_os("OG_IO_ROOT_TEST").is_none(), + "OG_IO_ROOT_TEST is set but the test does not run as root" + ); + eprintln!("skipped: needs root"); + return; + } + + // A folder another user owns: the file goes to that user, still 0600. + let user = dir.join("user"); + fs::create_dir(&user).unwrap(); + chown(&user, Some(4321), Some(4322)).unwrap(); + let out = user.join("out.tsv"); + write(&out); + let m = fs::metadata(&out).unwrap(); + assert_eq!((m.uid(), m.gid()), (4321, 4322), "the folder owner's file"); + assert_eq!(m.permissions().mode() & 0o7777, 0o600); + assert_eq!(fs::read_to_string(&out).unwrap(), "new\n"); + + // A subfolder made as root inside the user's folder: still the user's. + let sub = user.join("sub"); + fs::create_dir(&sub).unwrap(); + let deep = sub.join("deep.tsv"); + write(&deep); + let m = fs::metadata(&deep).unwrap(); + assert_eq!( + (m.uid(), m.gid()), + (4321, 4322), + "a file in a root subfolder" + ); + + // Root's own folder (the temp folder, all root's up to `/` in a + // container): the file stays root's. + let own = dir.join("own.tsv"); + write(&own); + assert_eq!(fs::metadata(&own).unwrap().uid(), 0, "root's file"); + + fs::remove_dir_all(&dir).unwrap(); +} From 5fa99693faeff002d36716f0389ff4d04696c0d1 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Wed, 18 Nov 2026 17:40:17 +0100 Subject: [PATCH 14/14] Keep one CHANGELOG entry for #486 in the batch d tip --- CHANGELOG.md | 1 - 1 file changed, 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c90cc931d..ff5cccdbb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,6 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n - `setup.sh` no longer says "Setup complete!" after a download failed. Before, a failed or skipped install (AnnotSV's data, a VEP cache, the CPSR bundle, somalier's sites, the STRchive catalog, the tandem-repeat map, gnomAD's chrM sites, a pinned data file, the sequence dictionary or og) printed a warning, then the same banner and exit status 0 as a full setup. Now setup keeps a list of what it could not install. When the list is empty it prints "Setup complete!" and exits 0 as before. Otherwise it prints "Setup finished: core ready, N items missing", lists each item with the command that retries it (`--annotsv-data`, `--vep-cache`, `--clinical-data`, `--sample-qc-data`, `--strchive-data`, `--og`, or setup again), and exits 3. A script that runs `setup.sh` and checks its status now sees a partial setup as one. `--no-clinical-data` still ends complete: skipping the clinical data is a choice. [Getting started](docs/getting-started.md) lists the three exit statuses. - `og lookup` no longer trusts an index older than its VCF for a chromosome the index does not list. A gVCF or annotated VCF replaced by hand after its `.tbi` or `.csi` was built kept the old index, and a chromosome the new file added was missing from it, so a variant called there read as "no call". The lookup now compares the two modification times, as the blood-type step does. When the index is older and does not list the queried chromosome, it reads the whole file and prints `NOTE: the index ... is older than ... and does not list chrN; reading the whole file`. An older index that lists the chromosome is still read: in a Nextflow output directory the small index is often copied before its large VCF, so it carries the earlier time. A fresh index is used as before, and no answer or output file changes. -- The files `og` writes through its atomic writer (every analysis output) are owner-only (mode 0600) whatever the umask. Before, `og` wrote each output under a temporary name with the umask's mode, so under the common umask 022 an output was readable by every user of the machine, and replacing a 0600 file made it 0644. The step scripts set umask 077, so a run through them was already owner-only; `og` run by hand or by the Nextflow pipeline was not. Now the temporary file is created mode 0600, or with the mode of the file it replaces when that is stricter (a 0400 file stays 0400). It is always a new file: a file or link already at its name is left as it is and the next name is tried, and two outputs written at once to one name in one process get two temporary files. The temporary name gains a number, `..tmp--`. No output name or content changes. [Nextflow Execution](docs/nextflow.md#run-the-pipeline) now says to keep `--outdir` private. - The files `og` writes through its atomic writer (every analysis output) are owner-only (mode 0600) whatever the umask. Before, `og` wrote each output under a temporary name with the umask's mode, so under the common umask 022 an output was readable by every user of the machine, and replacing a 0600 file made it 0644. The step scripts set umask 077, so a run through them was already owner-only; `og` run by hand or by the Nextflow pipeline was not. Now the temporary file is created mode 0600, or with the mode of the file it replaces when that is stricter (a 0400 file stays 0400). It is always a new file: a file or link already at its name is left as it is and the next name is tried, and two outputs written at once to one name in one process get two temporary files. The temporary name gains a number, `..tmp--`. When `og` runs as root, each of these files goes to the owner and group of the nearest folder above it that root does not own; when every folder up to `/` is root's, nothing changes. The docker profile of the Nextflow pipeline runs its containers as root over a work folder the host user owns; Nextflow's `docker.fixOwnership` does not reach these files, because the biocontainers images ship a `chown` without `--from`. Without the hand-over the host user could not read an output `og` wrote there, and Nextflow could not publish it. No output name or content changes. [Nextflow Execution](docs/nextflow.md#run-the-pipeline) now says to keep `--outdir` private. - A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again. - VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37).