diff --git a/CHANGELOG.md b/CHANGELOG.md index c9dcf0c9a..ff5cccdbb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,10 +4,14 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- `setup.sh` no longer says "Setup complete!" after a download failed. Before, a failed or skipped install (AnnotSV's data, a VEP cache, the CPSR bundle, somalier's sites, the STRchive catalog, the tandem-repeat map, gnomAD's chrM sites, a pinned data file, the sequence dictionary or og) printed a warning, then the same banner and exit status 0 as a full setup. Now setup keeps a list of what it could not install. When the list is empty it prints "Setup complete!" and exits 0 as before. Otherwise it prints "Setup finished: core ready, N items missing", lists each item with the command that retries it (`--annotsv-data`, `--vep-cache`, `--clinical-data`, `--sample-qc-data`, `--strchive-data`, `--og`, or setup again), and exits 3. A script that runs `setup.sh` and checks its status now sees a partial setup as one. `--no-clinical-data` still ends complete: skipping the clinical data is a choice. [Getting started](docs/getting-started.md) lists the three exit statuses. +- `og lookup` no longer trusts an index older than its VCF for a chromosome the index does not list. A gVCF or annotated VCF replaced by hand after its `.tbi` or `.csi` was built kept the old index, and a chromosome the new file added was missing from it, so a variant called there read as "no call". The lookup now compares the two modification times, as the blood-type step does. When the index is older and does not list the queried chromosome, it reads the whole file and prints `NOTE: the index ... is older than ... and does not list chrN; reading the whole file`. An older index that lists the chromosome is still read: in a Nextflow output directory the small index is often copied before its large VCF, so it carries the earlier time. A fresh index is used as before, and no answer or output file changes. +- The files `og` writes through its atomic writer (every analysis output) are owner-only (mode 0600) whatever the umask. Before, `og` wrote each output under a temporary name with the umask's mode, so under the common umask 022 an output was readable by every user of the machine, and replacing a 0600 file made it 0644. The step scripts set umask 077, so a run through them was already owner-only; `og` run by hand or by the Nextflow pipeline was not. Now the temporary file is created mode 0600, or with the mode of the file it replaces when that is stricter (a 0400 file stays 0400). It is always a new file: a file or link already at its name is left as it is and the next name is tried, and two outputs written at once to one name in one process get two temporary files. The temporary name gains a number, `..tmp--`. When `og` runs as root, each of these files goes to the owner and group of the nearest folder above it that root does not own; when every folder up to `/` is root's, nothing changes. The docker profile of the Nextflow pipeline runs its containers as root over a work folder the host user owns; Nextflow's `docker.fixOwnership` does not reach these files, because the biocontainers images ship a `chown` without `--from`. Without the hand-over the host user could not read an output `og` wrote there, and Nextflow could not publish it. No output name or content changes. [Nextflow Execution](docs/nextflow.md#run-the-pipeline) now says to keep `--outdir` private. - A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again. - VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37). - The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2). - The live site's access log no longer names the finding a page opens. A finding's id is built from its gene and position, and the finding route wrote that id into the log's section column, though `docs/live-site.md` says the log never holds a result. The row now holds `-` there, as the other result routes do. The log keeps its columns and one row per page open. An og-live cargo test opens a finding page on a made-up run and reads the log; its control puts the old line back and must fail. +- Step 51 (MoChA) no longer drops an autosome its phasing panel has no file for. With a panel cut to regions (`MOCHA_REGIONS`), such an autosome was cut out before MoChA ran, so it had no row in the sites table, the phase check could not see it, and the sample could read "assessed" with that autosome never looked at and no word about it. Now its sites go to MoChA unphased and keep their row in `_mocha_sites.tsv`, so the phase check marks the sample not assessed and the message names the autosome with its 0 phased sites. The step also prints a warning that names it. This holds for `scripts/51-mocha.sh` and the Nextflow `MOCHA_PHASE` and `MOCHA_CALL` processes. No file, column or summary row changes; the `chromosomes` row of such a sample now lists the unpanelled autosomes too. Part of #411. - The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes. - `og report render` now writes the sections of the opt-in steps and of two small default ones in Rust: clonal haematopoiesis with its detection limit (step 49), viral DNA (step 47), mobile element insertions (step 50), MoChA's mosaic alterations (step 51), VNtyper (step 52), Locityper (step 53), the Exomiser ranking (step 48), telomere content and runs of homozygosity. Both reports match `bin/render_report.py` byte for byte on the parity corpus, except for these changes: - When the BAM has no unplaced unmapped reads, step 47 looked at no viral genome. The viral section now says the genomes were not assessed. The Python said "No read on any of the 24 viral genomes", which reads as a negative result. @@ -53,7 +57,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n - The step's stdout line lists each below-threshold genome with its count, and says "no reads above the threshold at this depth" when nothing counts. - The docs say how many reads the 10% takes on each genome size, and what a negative leaves open. - The `viral-report` parity cases are retired: `bin/viral_report.py` is gone, and every case's output changes on purpose here. The Rust tests in `rust/crates/og-reports` and `tests/test_viral_report.py` hold the same cases. -- Step 51 (MoChA) now reads "not assessed" when too few heterozygous sites are phased to see a mosaic event. It read "not assessed" only when no site at all was phased, so a sample with one phased site in the whole genome, or with one autosome left mostly unphased by a panel cut to regions, read "assessed" with no call, and a real 6% CN-LOH above the floor went unreported. The sample is now not assessed when an autosome has under 0.80 of its heterozygous sites phased, and the message names each such autosome. An autosome the panel has no file for is still left out of the step before this check, so it is not assessed and the report does not say so. The minimum comes from two public genomes, HG002 and HG005, where every autosome keeps 0.933 to 0.975 of its heterozygous sites phased by the full panel; `docs/51-mocha.md` has the table. The page also says that MoChA's call rate is always 1 on a WGS VCF, so the call-rate half of its sample filter never fires there. The summary file keeps its rows; only the status and message of such a sample change. Part of #411. +- Step 51 (MoChA) now reads "not assessed" when too few heterozygous sites are phased to see a mosaic event. It read "not assessed" only when no site at all was phased, so a sample with one phased site in the whole genome, or with one autosome left mostly unphased by a panel cut to regions, read "assessed" with no call, and a real 6% CN-LOH above the floor went unreported. The sample is now not assessed when an autosome has under 0.80 of its heterozygous sites phased, and the message names each such autosome. The minimum comes from two public genomes, HG002 and HG005, where every autosome keeps 0.933 to 0.975 of its heterozygous sites phased by the full panel; `docs/51-mocha.md` has the table. The page also says that MoChA's call rate is always 1 on a WGS VCF, so the call-rate half of its sample filter never fires there. The summary file keeps its rows; only the status and message of such a sample change. Part of #411. - ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format. - A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals. - The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup. diff --git a/docs/51-mocha.md b/docs/51-mocha.md index bb351a4aa..8e902eb30 100644 --- a/docs/51-mocha.md +++ b/docs/51-mocha.md @@ -50,8 +50,8 @@ With `run-all.sh`: `TOOLS=...,mocha`; a run lists step 51 only when `TOOLS` name ## How it works 1. The VCF's biallelic PASS SNVs on chr1 to chr22. A genotype under 10 reads or with GQ under 20 is set missing, as the MoChA README does for WGS. `mochatools` adds the GC content around each site from the reference. -2. Beagle phases the genotypes against the panel, one chromosome at a time (`impute=false`, `seed=1`). A chromosome the panel lacks is left out. -3. The phased genotypes go back onto the sites of step 1, and MoChA runs in its WGS mode (allele depths from AD) with `--genome GRCh38` and `cnps.bed` as `--cnp`. The MHC and KIR regions are left out of its model. +2. Beagle phases the genotypes against the panel, one chromosome at a time (`impute=false`, `seed=1`). A chromosome the panel has no file for is not phased, and the step prints a warning that names it. +3. The phased genotypes go back onto the sites of step 1, on every chromosome, and MoChA runs in its WGS mode (allele depths from AD) with `--genome GRCh38` and `cnps.bed` as `--cnp`. The MHC and KIR regions are left out of its model. 4. `og mocha report` reads MoChA's tables and decides which calls are reported. A call is reported when all of these hold: @@ -71,7 +71,7 @@ The sample is **not assessed** when MoChA's sample filter of the same README fai **The call rate on WGS.** MoChA reads a WGS VCF by its allele depths (AD). In that mode it drops a site with a missing genotype before it counts the sites, so the call rate in its stats is 1 on every WGS VCF, whatever step 1's depth and GQ filter set missing. The call-rate half of the sample filter therefore never fires on this step's input. It applies only to a stats table that carries a real call rate. On WGS the phased BAF autocorrelation, the phased share below and the floor are what gate the sample. -**The phased share.** MoChA follows a haplotype only across phased heterozygous sites. An unphased site adds nothing to its phased model and adds 0 to its phased BAF autocorrelation, so a chromosome that is mostly unphased passes that filter and shows no event, even one above the floor. Beagle phases only the sites the panel holds, so the share of an autosome's heterozygous sites that end up phased is the share the panel holds. A panel cut with `MOCHA_REGIONS`, or a `MOCHA_PHASED_VCF` that is only partly phased, leaves whole stretches unphased. One gap stays: with Beagle, an autosome the panel has no file for (one a `MOCHA_REGIONS` file does not name) is left out of the step before MoChA runs. That autosome has no row in the sites table, so this check cannot see it, and the step does not assess it and does not say so. +**The phased share.** MoChA follows a haplotype only across phased heterozygous sites. An unphased site adds nothing to its phased model and adds 0 to its phased BAF autocorrelation, so a chromosome that is mostly unphased passes that filter and shows no event, even one above the floor. Beagle phases only the sites the panel holds, so the share of an autosome's heterozygous sites that end up phased is the share the panel holds. A panel cut with `MOCHA_REGIONS`, or a `MOCHA_PHASED_VCF` that is only partly phased, leaves whole stretches unphased. An autosome the panel has no file for (one a `MOCHA_REGIONS` file does not name) keeps its sites unphased and its row in the sites table, so this check names it with 0 phased sites and the sample is not assessed. We measured that share on two public genomes, with step 1's filters (biallelic PASS SNVs, at least 10 reads and GQ 20) and the panel's sites (biallelic SNVs seen twice or more, the same position and alleles): @@ -130,4 +130,4 @@ Not measured on a whole genome. Beagle's phasing of the 22 chromosomes takes mos - A constitutional event (one every cell carries, such as a germline deletion) has a cell fraction near 1 and is left out by MoChA's filter; this step is not a test for those. - A germline copy-number polymorphism is left out by type: MoChA types a call over a region of `cnps.bed` as `CNP_Loss` or `CNP_Gain`. A germline CNV missing from that list (rare, or under 10 kb) gets no such type. Short events (under a few Mb) are hard to tell from these and are reported only when MoChA's filter keeps them. - Calls are labelled by chromosome arm, not by gene: an mCA spans millions of bases. -- The e2e case (`tests/e2e/mocha-1.sh`) runs the step on HG002's chr20 slice (MoChA makes no call; with about 500 heterozygous sites the sample filter can mark the slice not assessed, since the BAF autocorrelation it reads is meant for a whole genome), and on a synthetic phased VCF with a 20 Mb CN-LOH on 20p in 15% of the cells (one CN-LOH call, cell fraction 0.10 to 0.20) beside the same VCF with no shift (no call). Nothing in the case is a finding about the fixture's sample. +- The e2e case (`tests/e2e/mocha-1.sh`) runs the step on HG002's slices with the panel cut to the chr20 slice (MoChA makes no call on chr20; the fixture's other autosomes have no panel file, so they stay unphased and the sample is not assessed, with each of them named; a call MoChA makes on one of those slices, such as a germline CNV like the chr4 UGT2B17 deletion, is untyped because the cut panel's `cnps.bed` holds chr20 only, and is reported "no" because the sample is not assessed), and on a synthetic phased VCF with a 20 Mb CN-LOH on 20p in 15% of the cells (one CN-LOH call, cell fraction 0.10 to 0.20) beside the same VCF with no shift (no call). Nothing in the case is a finding about the fixture's sample. diff --git a/docs/getting-started.md b/docs/getting-started.md index e72239787..c528c9cd2 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -57,6 +57,16 @@ Use `./scripts/setup.sh --no-clinical-data "$GENOME_DIR"` to opt out. Use `./scripts/setup.sh --clinical-data "$GENOME_DIR"` to add or retry only these data later. `run-all.sh` skips steps whose data are still missing; inspect its plan and setup warnings. +Setup ends in one of three ways: + +| Exit status | Last banner | What it means | +| --- | --- | --- | +| 0 | `Setup complete!` | Everything is installed. `--no-clinical-data` skips the clinical data on purpose, so that run can still end here. | +| 3 | `Setup finished: core ready, N items missing` | The reference, ClinVar and the images are in place, but at least one download, check or build failed or was skipped. Under `Missing` the summary lists each item and the command that retries it, such as `./scripts/setup.sh --annotsv-data "$GENOME_DIR"`. The steps that need an item say so when they run. | +| 1 | an error message, no banner | A core part failed (the reference, ClinVar or an image pull), or the arguments were wrong. Fix the error and run setup again. | + +The PGS Catalog scores, pgsc_calc and its Nextflow plugin are not on the list: step 25 fetches what is missing when it runs. + ## Platform Notes ### Linux (Recommended) diff --git a/docs/lookup.md b/docs/lookup.md index d67587dc3..136cea1d6 100644 --- a/docs/lookup.md +++ b/docs/lookup.md @@ -87,7 +87,7 @@ Two ClinVar lines can appear. `ClinVar (pipeline's ClinVar file)` comes from the ## Runtime -One container starts for the whole command, whatever the number of queries. With indexed files a site or a gene takes well under a second inside it: only the region is read. An rsID that is in neither the site table nor the pipeline tables makes the command read the whole annotated VCF once, for all such rsIDs of the command line together, on two cores. A file without a `.tbi` or `.csi` index is streamed whole for each region, with a note: the answer is the same and only the records in the region are kept in memory, but on a whole genome it takes minutes. `bcftools index -t` makes it fast. +One container starts for the whole command, whatever the number of queries. With indexed files a site or a gene takes well under a second inside it: only the region is read. An rsID that is in neither the site table nor the pipeline tables makes the command read the whole annotated VCF once, for all such rsIDs of the command line together, on two cores. A file without a `.tbi` or `.csi` index is streamed whole for each region, with a note: the answer is the same and only the records in the region are kept in memory, but on a whole genome it takes minutes. `bcftools index -t` makes it fast. An index older than its file, as when the file was replaced after the index was built, may lack a chromosome the new file has. So when such an index does not list the queried chromosome, the file is streamed whole, with a note; `bcftools index -f -t` rebuilds the index. An older index that lists the chromosome is still used: a Nextflow output directory often holds an index dated a little before its file, because the small index finishes copying first. ## Notes diff --git a/docs/nextflow.md b/docs/nextflow.md index 6332f369e..56e6fa044 100644 --- a/docs/nextflow.md +++ b/docs/nextflow.md @@ -91,6 +91,8 @@ nextflow run main.nf \ # it names (two or more, each with a gVCF; docs/multi-sample.md) ``` +**Keep `--outdir` private.** The results hold genome data. Create the folder before the run with `mkdir -m 700 results`, or put it in a folder only you can open. The `og` tools write their analysis outputs owner-only (mode 0600; run as root in a container, they give each file to the owner of the work folder it sits in, so you can read it), but Nextflow creates the folders, and the other tools write their files, with the modes your umask gives. + **Family workflow (opt-in).** `--pedigree` (tab separated: sample, sex, father, mother, partner; [`assets/pedigree.example.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/pedigree.example.tsv)) runs `workflows/family.nf` over the samplesheet rows it names. `--pedigree` may be a families file (a leading family column, [`assets/families.example.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/families.example.tsv)). `--family_name` names the family to run, which a file with more than one family needs, and the folder its results go to (`//` instead of `/family/`). The workflow does the same work as step 38: `FAMILY_PREPARE` checks the pedigree and that each member's gVCF names its own sample, `GLNEXUS` and `GLNEXUS_VCF` joint-call the gVCFs (`GLNEXUS` asks for 16 GB and gives GLnexus half of its memory as `--mem-gbytes`), `FAMILY_RELATE` and `FAMILY_CHECK` stop the run when somalier's relatedness of a pair disagrees with the pedigree (a sample swap), `WHATSHAP_PED` phases the children of complete trios by inheritance, and `FAMILY_REPORT` writes each child's allele origin, de novo candidates when both parents are sequenced, and compound heterozygotes among the clinical filter's candidates. With `--ancestry_fine_panel` (the panel folder of step 26b, `setup.sh --ancestry-fine-panel`), four more processes find the DNA each pair shares by descent, once `FAMILY_CHECK` has passed: `FAMILY_IBD_SITES` keeps the joint call's genotypes at the panel's SNVs on each autosome (a chromosome with fewer than 100 is left out), `FAMILY_IBD_PHASE` phases them with Beagle against the panel, `FAMILY_IBD_SEGMENTS` runs hap-ibd (its default 2 cM minimum) with the panel's genetic maps, and `FAMILY_IBD_REPORT` writes `ibd_pairs.tsv`, `ibd_segments.tsv` and `ibd_report.md`. They feed only the report, so a failure of one is logged and the run goes on. Without the panel none of them runs. Each member needs a gVCF: DEEPVARIANT's for a FASTQ or BAM row, the `gvcf` column for a VCF row (the run stops on a VCF row without one). Fewer than two named samples, or no `--somalier_sites`, stops the run at the start. When the pedigree marks two of those samples as each other's partner, `COUPLE_CARRIER` runs the couple carrier screen of step 39 on them ([Couple Carrier Screen](39-couple.md)): it reads each partner's ClinVar hits, so it needs `clinvar` in `--tools` (without it the run logs a warning and skips the screen), uses deletions over an exon only with `--gencode_exons` (the GENCODE exons BED `setup.sh` installs as `reference/gencode.v50.basic.exons.bed`), reads Delly's calls, ExpansionHunter's, and Parascopy's SMN1, interval and CYP21A2 tables when those steps ran, and, when `--annotsv_annotations` holds AnnotSV's benign loss set (`benign_Loss_SV_GRCh38.sorted.bed`, with gnomAD-SV's common deletions merged in by `setup.sh --annotsv-data`), sets aside a Delly deletion that matches a common one there. Without `--pedigree` none of these processes exists and a run is unchanged. What it checks and what the files hold: [Multi-sample comparison](multi-sample.md#family-analysis-step-38-opt-in). `--reference` is the FASTA the reads are aligned to and the BAMs were aligned to, the GRCh38 no-ALT analysis set that `setup.sh` installs as `reference/GRCh38_no_alt_analysis_set.fasta`. The Nextflow pipeline does not compare a given BAM's header with it; `./scripts/validate-setup.sh ` does, and [Realigning after a reference change](realignment.md) covers a BAM aligned to another reference. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 7d0c7f37a..84ff24c47 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -738,7 +738,7 @@ ExpansionHunter \ ### Clinical data skipped during setup Setup installs VEP 116, the PCGR/CPSR bundle and VEP 115 by default. -If one is missing, read the warning above `Setup complete!`. +If one is missing, setup ends with `Setup finished: core ready, N items missing` and lists it under `Missing` with its retry command. The warning above it says why. The disk checks require 56 GB, 15 GB and 52 GB peak respectively, each plus a 5 GB margin. The disk space occupied by a partial download counts toward that peak. Sparse file holes do not count. Setup removes stale extraction files before checking space and checks a retained PCGR archive against its pinned SHA-256 before reuse. diff --git a/modules/local/mocha/main.nf b/modules/local/mocha/main.nf index 05bdf1b98..c02c38c38 100644 --- a/modules/local/mocha/main.nf +++ b/modules/local/mocha/main.nf @@ -9,7 +9,9 @@ from the reference (bcftools +mochatools) MOCHA_PHASE Beagle against the panel, one chromosome at a time (those the panel has), with the panel's cnps.bed alongside - MOCHA_CALL the phased genotypes onto the prepared sites, MoChA + MOCHA_CALL the phased genotypes onto the prepared sites of every + chromosome (one the panel lacks stays unphased, so the + phase check of og mocha report names it), MoChA (bcftools +mocha, the common germline CNVs as --cnp), and the per-chromosome site counts MOCHA_REPORT og mocha report (bin/og): the calls above the floor @@ -91,7 +93,7 @@ process MOCHA_PHASE { [ -f ${mocha_data}/cnps.bed ] || { echo "ERROR: ${mocha_data}/cnps.bed not found: the panel is older than its CNV list (rerun setup.sh --mocha-data)" >&2; exit 1; } cp ${mocha_data}/cnps.bed phase/cnps.bed for c in \$(cat ${chroms}); do - [ -s ${mocha_data}/panel/\$c.vcf.gz ] || continue + [ -s ${mocha_data}/panel/\$c.vcf.gz ] || { echo "[WARN] The panel has no file for \$c: not phased, so a heterozygous site there makes the sample not assessed"; continue; } beagle -Xmx${heap}g gt=${phase_in} chrom=\$c ref=${mocha_data}/panel/\$c.vcf.gz map=${mocha_data}/maps/\$c.map \\ out=phase/\$c impute=false seed=1 nthreads=${task.cpus} > phase/\$c.out 2>&1 \\ || { tail -n 20 phase/\$c.out; exit 1; } @@ -140,7 +142,6 @@ process MOCHA_CALL { def p = meta.id """ set -euo pipefail - chroms=\$(paste -sd, ${phase}/chroms.txt) # Beagle writes no ##contig lines and concat refuses records on an # undeclared contig: give each Beagle VCF the contig lines of prep.bcf. bcftools view -h ${prep} | grep '^##contig=' > contigs.hdr @@ -152,7 +153,7 @@ process MOCHA_CALL { done bcftools concat --no-version -Ob -o phased_gt.bcf "\${f[@]}" bcftools index -f phased_gt.bcf - bcftools annotate --no-version -a phased_gt.bcf -c FMT/GT -r "\$chroms" -Ob -o phased.bcf ${prep} + bcftools annotate --no-version -a phased_gt.bcf -c FMT/GT -Ob -o phased.bcf ${prep} bcftools index -f phased.bcf bcftools +mocha --genome GRCh38 --no-version --mhc chr6:27518932-33480487 --kir chr19:54071493-54992731 --cnp ${phase}/cnps.bed \\ --calls ${p}_mocha_calls_raw.tsv --stats ${p}_mocha_stats.tsv --log ${p}_mocha.log phased.bcf diff --git a/rust/crates/og-io/src/lib.rs b/rust/crates/og-io/src/lib.rs index 18d0652b4..8c8d73b49 100644 --- a/rust/crates/og-io/src/lib.rs +++ b/rust/crates/og-io/src/lib.rs @@ -6,7 +6,8 @@ //! - [`Lines`]: Python's text mode with `newline=None`: `\n`, `\r\n` and a //! lone `\r` each end a line and read as `\n`; UTF-8 strict or `replace`. //! - [`AtomicFile`]: written beside the target and renamed over it on -//! [`AtomicFile::commit`], so a reader never sees half a file. +//! [`AtomicFile::commit`], so a reader never sees half a file. What it +//! writes is owner-only (0600) whatever the umask. //! - [`bgzf`]: the same reading with BGZF inflated on many cores, and a //! BGZF writer that compresses on many cores. //! - [`tabix`]: the lines of a region of a bgzipped file, through its @@ -22,6 +23,7 @@ use flate2::bufread::GzDecoder; use std::fs::{self, File}; use std::io::{self, BufRead, BufReader, BufWriter, Read, Write}; use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; const GZIP_MAGIC: [u8; 2] = [0x1f, 0x8b]; @@ -198,28 +200,59 @@ impl Iterator for Lines { /// A file written under a temporary name and renamed into place by /// [`commit`](AtomicFile::commit). Dropped without a commit, the temporary /// file is removed and the target is untouched. +/// +/// The temporary file is always a new file: a file or link already at its +/// name is left alone and the next name is tried. On Unix it is created +/// mode 0600, or with the mode of the target it replaces when that is +/// stricter, so the umask never makes an output readable by others. +/// +/// When og runs as root, the temporary file is handed to the owner and +/// group of the nearest folder above it that root does not own. That is a +/// container run as root over a work folder bind-mounted from the host, as +/// the pipeline's docker profile does: without it the host user could not +/// read an owner-only output og wrote there. pub struct AtomicFile { tmp: PathBuf, target: PathBuf, out: Option>, } +/// The number in the next temporary name, so two files open in one process +/// never share one. +static NEXT_TMP: AtomicU64 = AtomicU64::new(0); + +/// How many temporary names are tried before giving up. +const TMP_TRIES: usize = 64; + impl AtomicFile { pub fn create(target: impl AsRef) -> io::Result { let target = target.as_ref().to_path_buf(); let name = target .file_name() .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "no file name"))?; - let mut tmp_name = std::ffi::OsString::from("."); - tmp_name.push(name); - tmp_name.push(format!(".tmp-{}", std::process::id())); - let tmp = target.with_file_name(tmp_name); - let out = BufWriter::new(File::create(&tmp)?); - Ok(AtomicFile { - tmp, - target, - out: Some(out), - }) + let mode = temp_mode(&target); + let pid = std::process::id(); + let mut taken = None; + for _ in 0..TMP_TRIES { + let n = NEXT_TMP.fetch_add(1, Ordering::Relaxed); + let mut tmp_name = std::ffi::OsString::from("."); + tmp_name.push(name); + tmp_name.push(format!(".tmp-{pid}-{n}")); + let tmp = target.with_file_name(tmp_name); + match new_file(&tmp, mode) { + Ok(f) => { + give_to_folder_owner(&f, &tmp); + return Ok(AtomicFile { + tmp, + target, + out: Some(BufWriter::new(f)), + }); + } + Err(e) if e.kind() == io::ErrorKind::AlreadyExists => taken = Some(e), + Err(e) => return Err(e), + } + } + Err(taken.expect("at least one name was tried")) } /// Flushes and renames over the target. On an error the temporary file @@ -240,6 +273,82 @@ impl AtomicFile { } } +/// The mode of a new temporary file: 0600, less any owner bit the target +/// it replaces lacks (a 0400 target stays 0400). +#[cfg(unix)] +fn temp_mode(target: &Path) -> u32 { + use std::os::unix::fs::PermissionsExt; + match fs::metadata(target) { + Ok(m) if m.is_file() => 0o600 & m.permissions().mode(), + _ => 0o600, + } +} + +#[cfg(not(unix))] +fn temp_mode(_target: &Path) -> u32 { + 0 +} + +/// Hands a file og created as root to the owner and group of the nearest +/// folder above it that root does not own: the work folder a container run +/// as root got from the host, even through a subfolder made as root inside +/// it. Nothing changes when og is not root or every folder up to `/` is +/// root's. Best effort: where root may not change owners, the file stays +/// root's. +#[cfg(unix)] +fn give_to_folder_owner(file: &File, path: &Path) { + use std::os::unix::fs::MetadataExt; + let Ok(f) = file.metadata() else { + return; + }; + if f.uid() != 0 { + return; + } + let dir = match path.parent() { + Some(d) if !d.as_os_str().is_empty() => d, + _ => Path::new("."), + }; + let Ok(dir) = fs::canonicalize(dir) else { + return; + }; + let owners = dir + .ancestors() + .map_while(|d| fs::metadata(d).ok()) + .map(|m| (m.uid(), m.gid())); + if let Some((uid, gid)) = hand_over_to(f.uid(), owners) { + let _ = std::os::unix::fs::fchown(file, Some(uid), Some(gid)); + } +} + +#[cfg(not(unix))] +fn give_to_folder_owner(_file: &File, _path: &Path) {} + +/// The owner and group a new file goes to, given the owners of its folders +/// from the nearest up: the first folder not owned by root, when the file +/// is root's (og runs as root); otherwise none. +#[cfg(unix)] +fn hand_over_to( + file_uid: u32, + mut folders: impl Iterator, +) -> Option<(u32, u32)> { + if file_uid != 0 { + return None; + } + folders.find(|&(uid, _)| uid != 0) +} + +/// Creates `path`, failing with `AlreadyExists` when anything is there, +/// a link included (`O_CREAT | O_EXCL`). +fn new_file(path: &Path, mode: u32) -> io::Result { + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + std::os::unix::fs::OpenOptionsExt::mode(&mut options, mode); + #[cfg(not(unix))] + let _ = mode; + options.open(path) +} + impl Write for AtomicFile { fn write(&mut self, buf: &[u8]) -> io::Result { self.out.as_mut().expect("not committed").write(buf) @@ -364,6 +473,33 @@ mod tests { } } +#[cfg(all(test, unix))] +mod owner_tests { + use super::hand_over_to; + + #[test] + fn root_in_a_user_folder_hands_the_file_over() { + assert_eq!( + hand_over_to(0, [(1001, 118), (0, 0)].into_iter()), + Some((1001, 118)) + ); + // Through a subfolder made as root inside the user's folder. + assert_eq!( + hand_over_to(0, [(0, 0), (1001, 118), (0, 0)].into_iter()), + Some((1001, 118)) + ); + } + + #[test] + fn otherwise_the_owner_stays() { + // og as root where every folder is root's, and og as a user. + assert_eq!(hand_over_to(0, [(0, 0), (0, 0)].into_iter()), None); + assert_eq!(hand_over_to(0, std::iter::empty()), None); + assert_eq!(hand_over_to(1001, [(1001, 118)].into_iter()), None); + assert_eq!(hand_over_to(1001, [(0, 0), (4321, 1)].into_iter()), None); + } +} + #[cfg(test)] mod bounded_line_tests { use super::*; diff --git a/rust/crates/og-io/tests/atomic_modes.rs b/rust/crates/og-io/tests/atomic_modes.rs new file mode 100644 index 000000000..1947c518c --- /dev/null +++ b/rust/crates/og-io/tests/atomic_modes.rs @@ -0,0 +1,118 @@ +//! The modes and temporary names of [`og_io::AtomicFile`]: what it writes is +//! owner-only whatever the umask, and two writers in one process never share +//! a temporary file. + +use og_io::AtomicFile; +use std::fs; +use std::io::Write; +use std::os::unix::fs::PermissionsExt; +use std::path::{Path, PathBuf}; +use std::process::Command; + +const CHILD: &str = "OG_IO_UMASK_022_CHILD"; + +fn scratch(name: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!("og-io-{name}-{}", std::process::id())); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + dir +} + +/// The permission bits in octal, so a failure reads `644`, not `420`. +fn mode(path: &Path) -> String { + format!( + "{:o}", + fs::metadata(path).unwrap().permissions().mode() & 0o7777 + ) +} + +fn write(target: &Path, text: &str) { + let mut f = AtomicFile::create(target).unwrap(); + f.write_all(text.as_bytes()).unwrap(); + f.commit().unwrap(); +} + +/// Runs itself again in a child under `umask 022`, the umask most shells +/// start with, and checks that the child ran this test and passed it. +#[test] +fn owner_only_under_umask_022() { + if std::env::var_os(CHILD).is_none() { + let exe = std::env::current_exe().unwrap(); + let out = Command::new("/bin/sh") + .arg("-c") + .arg("umask 022 && exec \"$0\" --exact owner_only_under_umask_022 --nocapture --test-threads=1") + .arg(&exe) + .env(CHILD, "1") + .output() + .unwrap(); + let stdout = String::from_utf8_lossy(&out.stdout); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + out.status.success() && stdout.contains("1 passed"), + "the umask 022 child failed or ran no test:\n{stdout}\n{stderr}" + ); + return; + } + let dir = scratch("umask"); + // The control: under this umask a plain new file is 0644. + let control = dir.join("control"); + fs::File::create(&control).unwrap(); + assert_eq!( + mode(&control), + "644", + "the child does not run under umask 022" + ); + + // A new target. + let new = dir.join("new.tsv"); + write(&new, "new\n"); + assert_eq!(mode(&new), "600", "a new target"); + + // A 0600 target that is replaced stays 0600. + let private = dir.join("private.tsv"); + fs::write(&private, "old\n").unwrap(); + fs::set_permissions(&private, fs::Permissions::from_mode(0o600)).unwrap(); + write(&private, "new\n"); + assert_eq!(mode(&private), "600", "a replaced 0600 target"); + assert_eq!(fs::read_to_string(&private).unwrap(), "new\n"); + + // A 0644 target becomes owner-only. + let open = dir.join("open.tsv"); + fs::write(&open, "old\n").unwrap(); + fs::set_permissions(&open, fs::Permissions::from_mode(0o644)).unwrap(); + write(&open, "new\n"); + assert_eq!(mode(&open), "600", "a replaced 0644 target"); + + // A stricter target keeps its mode. + let read_only = dir.join("read-only.tsv"); + fs::write(&read_only, "old\n").unwrap(); + fs::set_permissions(&read_only, fs::Permissions::from_mode(0o400)).unwrap(); + write(&read_only, "new\n"); + assert_eq!(mode(&read_only), "400", "a replaced 0400 target"); + assert_eq!(fs::read_to_string(&read_only).unwrap(), "new\n"); + + fs::remove_dir_all(&dir).unwrap(); +} + +/// Two files open on one target in one process each write their own +/// temporary file: both commits succeed and the last one wins whole. +#[test] +fn two_writers_on_one_target() { + let dir = scratch("two"); + let target = dir.join("out.tsv"); + let mut a = AtomicFile::create(&target).unwrap(); + let mut b = AtomicFile::create(&target).unwrap(); + a.write_all(b"from a\n").unwrap(); + b.write_all(b"from b\n").unwrap(); + a.commit().unwrap(); + assert_eq!(fs::read_to_string(&target).unwrap(), "from a\n"); + b.commit().unwrap(); + assert_eq!(fs::read_to_string(&target).unwrap(), "from b\n"); + // Nothing but the target is left. + let names: Vec<_> = fs::read_dir(&dir) + .unwrap() + .map(|e| e.unwrap().file_name()) + .collect(); + assert_eq!(names, ["out.tsv"]); + fs::remove_dir_all(&dir).unwrap(); +} diff --git a/rust/crates/og-io/tests/atomic_planted_temp.rs b/rust/crates/og-io/tests/atomic_planted_temp.rs new file mode 100644 index 000000000..e6d3bd7c5 --- /dev/null +++ b/rust/crates/og-io/tests/atomic_planted_temp.rs @@ -0,0 +1,55 @@ +//! [`og_io::AtomicFile`] creates its temporary file and never opens one that +//! is already there. This is the only test in its binary, so the first +//! temporary name this process tries is known: `..tmp--0`. + +use og_io::AtomicFile; +use std::fs; +use std::io::Write; +use std::os::unix::fs::symlink; + +#[test] +fn a_link_at_the_temporary_name_is_left_alone() { + let pid = std::process::id(); + let dir = std::env::temp_dir().join(format!("og-io-planted-{pid}")); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let target = dir.join("out.tsv"); + let other = dir.join("other.tsv"); + fs::write(&other, "keep\n").unwrap(); + let missing = dir.join("missing.tsv"); + // Links at the first two names this process tries, and at the name + // releases before 1.0.0 used (`..tmp-`). + let planted = [ + (format!(".out.tsv.tmp-{pid}"), &other), + (format!(".out.tsv.tmp-{pid}-0"), &other), + (format!(".out.tsv.tmp-{pid}-1"), &missing), + ]; + for (name, to) in &planted { + symlink(to, dir.join(name)).unwrap(); + } + + let mut f = AtomicFile::create(&target).unwrap(); + f.write_all(b"new\n").unwrap(); + f.commit().unwrap(); + + assert_eq!(fs::read_to_string(&other).unwrap(), "keep\n"); + assert!( + !missing.exists(), + "a file was created through a dangling link" + ); + assert!( + !fs::symlink_metadata(&target) + .unwrap() + .file_type() + .is_symlink() + ); + assert_eq!(fs::read_to_string(&target).unwrap(), "new\n"); + for (name, to) in &planted { + assert_eq!( + &fs::read_link(dir.join(name)).unwrap(), + *to, + "{name} changed" + ); + } + fs::remove_dir_all(&dir).unwrap(); +} diff --git a/rust/crates/og-io/tests/atomic_root_owner.rs b/rust/crates/og-io/tests/atomic_root_owner.rs new file mode 100644 index 000000000..7853bb477 --- /dev/null +++ b/rust/crates/og-io/tests/atomic_root_owner.rs @@ -0,0 +1,70 @@ +//! Run as root, [`og_io::AtomicFile`] hands what it writes in another user's +//! folder, or in a subfolder root made inside it, to that user, as in a +//! container run as root over a work folder bind-mounted from the host. The output stays owner-only, and the user who +//! owns the folder can read it. +//! +//! The test needs root, so it runs in a container (`docker run --user root`). +//! Run as any other user it prints that it was skipped, unless +//! `OG_IO_ROOT_TEST=1` is set: then it fails, so a run meant to check this +//! cannot pass without checking it. + +use og_io::AtomicFile; +use std::fs; +use std::io::Write; +use std::os::unix::fs::{MetadataExt, PermissionsExt, chown}; +use std::path::Path; + +fn write(target: &Path) { + let mut f = AtomicFile::create(target).unwrap(); + f.write_all(b"new\n").unwrap(); + f.commit().unwrap(); +} + +#[test] +fn root_hands_the_file_to_the_folder_owner() { + let dir = std::env::temp_dir().join(format!("og-io-root-{}", std::process::id())); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let probe = dir.join("probe"); + fs::write(&probe, "").unwrap(); + if fs::metadata(&probe).unwrap().uid() != 0 { + fs::remove_dir_all(&dir).unwrap(); + assert!( + std::env::var_os("OG_IO_ROOT_TEST").is_none(), + "OG_IO_ROOT_TEST is set but the test does not run as root" + ); + eprintln!("skipped: needs root"); + return; + } + + // A folder another user owns: the file goes to that user, still 0600. + let user = dir.join("user"); + fs::create_dir(&user).unwrap(); + chown(&user, Some(4321), Some(4322)).unwrap(); + let out = user.join("out.tsv"); + write(&out); + let m = fs::metadata(&out).unwrap(); + assert_eq!((m.uid(), m.gid()), (4321, 4322), "the folder owner's file"); + assert_eq!(m.permissions().mode() & 0o7777, 0o600); + assert_eq!(fs::read_to_string(&out).unwrap(), "new\n"); + + // A subfolder made as root inside the user's folder: still the user's. + let sub = user.join("sub"); + fs::create_dir(&sub).unwrap(); + let deep = sub.join("deep.tsv"); + write(&deep); + let m = fs::metadata(&deep).unwrap(); + assert_eq!( + (m.uid(), m.gid()), + (4321, 4322), + "a file in a root subfolder" + ); + + // Root's own folder (the temp folder, all root's up to `/` in a + // container): the file stays root's. + let own = dir.join("own.tsv"); + write(&own); + assert_eq!(fs::metadata(&own).unwrap().uid(), 0, "root's file"); + + fs::remove_dir_all(&dir).unwrap(); +} diff --git a/rust/crates/og-lookup/src/vcf.rs b/rust/crates/og-lookup/src/vcf.rs index 0177b9bae..e21b280ad 100644 --- a/rust/crates/og-lookup/src/vcf.rs +++ b/rust/crates/og-lookup/src/vcf.rs @@ -288,20 +288,25 @@ pub struct Span<'a> { /// The records of `path` that overlap `span` (a reference block by its /// END, a deletion by its REF), in file order: through the `.tbi` or `.csi` /// beside the file when there is one, else by streaming the whole file, -/// which `note` is told. +/// which `note` is told. An index older than the file that does not list +/// the region's chromosome is not trusted either: the file may have gained +/// that chromosome after the index was built. pub fn read(path: &str, span: Span, note: &mut dyn Write) -> Result { { - let has_index = [".tbi", ".csi"] + let index = [".tbi", ".csi"] .iter() - .any(|x| Path::new(&format!("{path}{x}")).is_file()); - if !has_index { - let _ = writeln!( - note, - "NOTE: {path} has no .tbi index; reading the whole file (bcftools index -t makes this fast)." - ); - } else { - match read_indexed(path, span) { - Ok(r) => return Ok(r), + .map(|x| format!("{path}{x}")) + .find(|p| Path::new(p).is_file()); + if let Some(index) = index.as_deref() { + match read_indexed(path, index, span, older(index, path)) { + Ok(Some(r)) => return Ok(r), + Ok(None) => { + let _ = writeln!( + note, + "NOTE: the index {index} is older than {path} and does not list {}; reading the whole file (bcftools index -f -t rebuilds it).", + crate::query::contig_name(span.chrom) + ); + } Err(e) => { let _ = writeln!( note, @@ -313,19 +318,36 @@ pub fn read(path: &str, span: Span, note: &mut dyn Write) -> Result bool { + let mtime = |p: &str| std::fs::metadata(p).and_then(|m| m.modified()).ok(); + matches!((mtime(index), mtime(path)), (Some(i), Some(v)) if i < v) +} + /// Keeps `r` when it overlaps `span`. fn overlaps(r: &Rec, span: Span) -> bool { r.chrom == span.chrom && r.pos <= span.end && r.end >= span.start } /// The header through the top of the file, then each index sequence that -/// names the region's chromosome, through the index. -fn read_indexed(path: &str, span: Span) -> Result { +/// names the region's chromosome, through the index. `None` when the index +/// is `stale` (older than the file) and names no such sequence. +/// +/// Age alone does not condemn an index: Nextflow publishes a VCF and its +/// index as separate copies, and the small index finishes first, so a fresh +/// index often has the earlier time. +fn read_indexed(path: &str, index: &str, span: Span, stale: bool) -> Result, Error> { let file = og_io::open(path).map_err(|e| Error::io(&e, path))?; let mut header = Vec::new(); for line in Lines::new(file, Utf8::Replace) { @@ -338,7 +360,7 @@ fn read_indexed(path: &str, span: Span) -> Result { } } let csq = Csq::from_header(header.iter().map(String::as_str)); - let mut reader = tabix::Reader::open(path).map_err(|e| Error::io(&e, path))?; + let mut reader = tabix::Reader::with_index(path, index).map_err(|e| Error::io(&e, path))?; let names: Vec = reader .index() .names() @@ -346,6 +368,9 @@ fn read_indexed(path: &str, span: Span) -> Result { .filter(|n| norm_chrom(n) == span.chrom) .cloned() .collect(); + if stale && names.is_empty() { + return Ok(None); + } let mut records = Vec::new(); for name in names { for line in reader.query(&name, span.start.saturating_sub(1), span.end) { @@ -359,7 +384,7 @@ fn read_indexed(path: &str, span: Span) -> Result { } } } - Ok(Region { csq, records }) + Ok(Some(Region { csq, records })) } /// A data line of the region, or `None` for one outside it (or too short diff --git a/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs b/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs new file mode 100644 index 000000000..ea42108a6 --- /dev/null +++ b/rust/crates/og-lookup/tests/rv_lookup_stale_index.rs @@ -0,0 +1,138 @@ +//! An index older than its VCF is not trusted for a contig it does not list: +//! the VCF may have gained that contig after the index was built, so an +//! indexed read would answer "no call" for a variant the file holds. The +//! lookup reads the whole file instead and says so. A fresh index, or an +//! older one that lists the contig (a Nextflow outdir copies the small index +//! before the large VCF), is still used. + +use std::fs::{self, File}; +use std::io::Write; +use std::num::NonZero; +use std::path::{Path, PathBuf}; +use std::time::{Duration, SystemTime}; + +fn input(name: &str) -> String { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../../../tests/parity/inputs/lookup") + .join(name) + .to_string_lossy() + .into_owned() +} + +/// Runs `og lookup query chrM:73` over `gvcf`: stdout, stderr. +fn chrm_73(gvcf: &str) -> (String, String) { + let args: Vec = [ + "query", + "--sites", + &input("sites.tsv.gz"), + "--genes", + &input("gencode.v50.basic.genes.gtf"), + "--table", + &input("traits.tsv"), + "--gvcf", + gvcf, + "--", + "chrM:73", + ] + .map(String::from) + .to_vec(); + let (mut out, mut err) = (Vec::new(), Vec::new()); + let rc = og_lookup::cli(&args, &mut out, &mut err); + let (out, err) = ( + String::from_utf8(out).unwrap(), + String::from_utf8(err).unwrap(), + ); + assert_eq!(rc, 0, "{out}{err}"); + (out, err) +} + +/// Writes `text` to `path` as BGZF. +fn bgzip(path: &Path, text: &str) { + let mut w = + og_io::bgzf::Writer::new(File::create(path).unwrap(), NonZero::::MIN).unwrap(); + w.write_all(text.as_bytes()).unwrap(); + w.finish().unwrap(); +} + +/// Builds the `.tbi` of the BGZF VCF `vcf`. +fn index(vcf: &Path, tbi: &Path) { + let built = og_io::tabix_write::build(File::open(vcf).unwrap()).unwrap(); + built.write_tbi(tbi, true).unwrap(); +} + +/// Sets the modification time of `path` a minute after (or before) that of `of`. +fn touch(path: &Path, of: &Path, later: bool) { + let base = fs::metadata(of).unwrap().modified().unwrap(); + let minute = Duration::from_secs(60); + let t: SystemTime = if later { base + minute } else { base - minute }; + File::options() + .write(true) + .open(path) + .unwrap() + .set_modified(t) + .unwrap(); +} + +#[test] +fn an_index_older_than_its_vcf_is_not_trusted() { + let dir: PathBuf = Path::new(env!("CARGO_TARGET_TMPDIR")).join("rv_lookup_stale_index"); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let (gz, tbi) = (dir.join("g.vcf.gz"), dir.join("g.vcf.gz.tbi")); + let gz_s = gz.to_string_lossy().into_owned(); + let full = fs::read_to_string(input("g.vcf")).unwrap(); + assert!(full.contains("\nchrM\t73\t"), "the fixture calls chrM:73"); + + // The answer from the whole plain file. + let (want, _) = chrm_73(&input("g.vcf")); + assert!(want.contains("genotype: G/G"), "{want}"); + + // The index of an earlier VCF without chrM, then the VCF rewritten with it. + let old: String = full + .split_inclusive('\n') + .filter(|l| !l.starts_with("chrM\t")) + .collect(); + bgzip(&gz, &old); + index(&gz, &tbi); + let stale = fs::read(&tbi).unwrap(); + bgzip(&gz, &full); + touch(&tbi, &gz, false); + + let (out, err) = chrm_73(&gz_s); + assert_eq!(out, want, "the stale index hid the chrM call"); + assert!( + err.contains(&format!( + "NOTE: the index {} is older than {gz_s} and does not list chrM; reading the whole file", + tbi.display() + )), + "{err}" + ); + + // The same index with a newer time is read, which shows the indexed path + // is the one taken: it lacks chrM, so the answer is not the call. + touch(&tbi, &gz, true); + let (out, err) = chrm_73(&gz_s); + assert_ne!(out, want, "a fresh index is read, not the whole file"); + assert!(err.is_empty(), "{err}"); + + // A fresh index of the current VCF answers the call, with no note. + index(&gz, &tbi); + assert_ne!(fs::read(&tbi).unwrap(), stale); + touch(&tbi, &gz, true); + let (out, err) = chrm_73(&gz_s); + assert_eq!(out, want); + assert!(err.is_empty(), "{err}"); + + // The same full index dated before the file, as Nextflow leaves it when + // it copies the small index before the large VCF, is still read: it + // lists chrM, so the indexed path answers and nothing is streamed. + touch(&tbi, &gz, false); + let (out, err) = chrm_73(&gz_s); + assert_eq!(out, want); + assert!( + err.is_empty(), + "an older index that lists the contig is used: {err}" + ); + + fs::remove_dir_all(&dir).unwrap(); +} diff --git a/scripts/51-mocha.sh b/scripts/51-mocha.sh index 3063c8cbc..d26b4f52f 100755 --- a/scripts/51-mocha.sh +++ b/scripts/51-mocha.sh @@ -86,12 +86,14 @@ run_in --cpus 2 --memory 4g "$MOCHA_IMAGE" bash -euo pipefail -c ' CHROMS=$(tr '\n' ' ' < "${W}/chroms.txt") [ -n "${CHROMS// /}" ] || { echo "ERROR: no biallelic PASS SNV on chr1 to chr22 in ${VCF}" >&2; exit 1; } -# 2. Beagle against the panel, one chromosome at a time; a chromosome the panel -# lacks (a panel cut to regions) is left out. +# 2. Beagle against the panel, one chromosome at a time. A chromosome the panel +# lacks (a panel cut to regions) is not phased, but it still goes on to MoChA +# and the sites table, so og mocha report's phase check names it. if [ "$PHASING" = beagle ]; then - PHASE="" - for c in $CHROMS; do [ -s "${DATA}/panel/${c}.vcf.gz" ] && PHASE+="${c} "; done + PHASE="" NOPANEL="" + for c in $CHROMS; do if [ -s "${DATA}/panel/${c}.vcf.gz" ]; then PHASE+="${c} "; else NOPANEL+="${c} "; fi; done [ -n "$PHASE" ] || { echo "ERROR: the panel has none of the VCF's chromosomes (${CHROMS})" >&2; exit 1; } + [ -z "$NOPANEL" ] || echo "[WARN] The panel has no file for ${NOPANEL% }: not phased, so a heterozygous site there makes the sample not assessed" echo "[2/4] Phasing with Beagle against the panel (${PHASE% })..." # shellcheck disable=SC2016 # $1 to $4 belong to the inner bash run_in --cpus "$THREADS" --memory 16g "$BEAGLE_IMAGE" bash -euo pipefail -c ' @@ -104,10 +106,11 @@ if [ "$PHASING" = beagle ]; then tr ' ' '\n' <<< "${PHASE% }" > "${W}/phased_chroms.txt" else echo "[2/4] The VCF is already phased: no Beagle run." - cp "${W}/chroms.txt" "${W}/phased_chroms.txt" fi -# 3. The phased genotypes onto the prepared sites, then MoChA (WGS mode: AD). +# 3. The phased genotypes onto the prepared sites of every chromosome, then +# MoChA (WGS mode: AD). A chromosome Beagle did not phase keeps its unphased +# genotypes, so its row in the sites table shows its phased share. # SITES_AWK sums each chromosome: first and last position, sites, hets, phased hets. # shellcheck disable=SC2016 # awk's fields, not shell variables SITES_AWK='$1 != c { if (c != "") print c, a, b, n, h, ph; c = $1; a = $2; n = h = ph = 0 } @@ -116,7 +119,7 @@ SITES_AWK='$1 != c { if (c != "") print c, a, b, n, h, ph; c = $1; a = $2; n = h echo "[3/4] MoChA (bcftools +mocha, ${MOCHA_VERSION})..." # shellcheck disable=SC2016 # $1 to $7 belong to the inner bash run_in --cpus 2 --memory 8g "$MOCHA_IMAGE" bash -euo pipefail -c ' - w=$1; chroms=$(paste -sd, "$w/phased_chroms.txt") + w=$1 if [ "$2" = beagle ]; then # Beagle writes no ##contig lines and concat refuses records on an # undeclared contig: give each Beagle VCF the contig lines of prep.bcf. @@ -129,9 +132,9 @@ run_in --cpus 2 --memory 8g "$MOCHA_IMAGE" bash -euo pipefail -c ' done bcftools concat --no-version -Ob -o "$w/phased_gt.bcf" "${f[@]}" bcftools index -f "$w/phased_gt.bcf" - bcftools annotate --no-version -a "$w/phased_gt.bcf" -c FMT/GT -r "$chroms" -Ob -o "$w/phased.bcf" "$w/prep.bcf" + bcftools annotate --no-version -a "$w/phased_gt.bcf" -c FMT/GT -Ob -o "$w/phased.bcf" "$w/prep.bcf" else - bcftools view --no-version -r "$chroms" -Ob -o "$w/phased.bcf" "$w/prep.bcf" + bcftools view --no-version -Ob -o "$w/phased.bcf" "$w/prep.bcf" fi bcftools index -f "$w/phased.bcf" cnp=(); [ -z "$7" ] || cnp=(--cnp "$7") diff --git a/scripts/setup.sh b/scripts/setup.sh index 026e16acc..85deff657 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -88,6 +88,11 @@ # 7. VEP 116, PCGR/CPSR data and VEP 115 (~57 GB download, ~65 GB unpacked). # --no-clinical-data skips them; --clinical-data installs only these three. # +# Exit status of a full setup: 0 when everything is installed ("Setup +# complete!"); 3 when the core is ready but an item of 4 to 7 failed or was +# skipped ("Setup finished: core ready, N items missing", each listed with the +# command that retries it); 1 when the reference, ClinVar or an image failed. +# # Every download goes through fetch (scripts/lib/common.sh): it is written to # .part, checked, and only then renamed, so a file that exists is whole. # A failed image pull is tried FETCH_TRIES times too, FETCH_WAIT apart, and @@ -229,6 +234,15 @@ export GENOME_DIR . "$(dirname "$0")/lib/common.sh" if [ -n "$PCGR_SHA_OVERRIDE" ]; then PCGR_DATA_SHA256=$PCGR_SHA_OVERRIDE; fi +# What a full setup could not install, one "what|retry command" entry each. +# The summary lists them, and a run with any left exits SETUP_PARTIAL_EXIT. +MISSING=() +SETUP_PARTIAL_EXIT=3 +# The retry commands quote the script path, so a checkout path with a space +# still copies and pastes as one word. +SETUP_CMD=$(printf '%q' "$0") +note_missing() { MISSING+=("$1|$2"); } + # NCBI's GRCh38 analysis set without ALT contigs: chr1-22, X, Y, M, the # unplaced and unlocalized scaffolds and chrEBV, 195 sequences. NCBI lists the # md5 of every file of the directory in md5checksums.txt, which is read at @@ -2133,6 +2147,8 @@ install_clinical_data() { else missing=1 fi + [ -d "${GENOME_DIR}/pcgr_data/${PCGR_DATA_BUNDLE}/data" ] \ + || note_missing "PCGR/CPSR bundle ${PCGR_DATA_BUNDLE} (step 17)" "${SETUP_CMD} --clinical-data \"${GENOME_DIR}\"" else [ ! -f "${dir}/homo_sapiens/${rel}_GRCh38/info.txt" ] || { echo "[OK] VEP ${rel} cache already present."; continue; } peak=56 @@ -2148,6 +2164,13 @@ install_clinical_data() { else missing=1 fi + if [ ! -f "${dir}/homo_sapiens/${rel}_GRCh38/info.txt" ]; then + if [ "$rel" = "$VEP_CACHE_RELEASE" ]; then + note_missing "VEP ${rel} cache (steps 13, 30, 23 and 31)" "${SETUP_CMD} --vep-cache \"${GENOME_DIR}\"" + else + note_missing "VEP ${rel} cache (step 17, CPSR)" "${SETUP_CMD} --clinical-data \"${GENOME_DIR}\"" + fi + fi fi done return "$missing" @@ -2461,6 +2484,7 @@ elif "${PGP_ROOT}/rust/og.sh" install; then echo "[OK] og (bin/.og) built." else echo "[WARN] Could not build og. Steps 04, 05, 07, 25 and 33 stop without it; run: $0 --og" + note_missing "og, OwnGenome's own binary (steps 04, 05, 07, 25 and 33)" "${SETUP_CMD} --og" fi ############################################################################### @@ -2485,6 +2509,7 @@ else else rm -f "$DICT_PART" echo "[WARN] Could not create ${REF_DICT}. Steps 03a, 20, 29 and chip-to-vcf need it; re-run setup.sh." + note_missing "the sequence dictionary ${REF_DICT##*/} (steps 03a, 20, 29 and chip-to-vcf)" "${SETUP_CMD} \"${GENOME_DIR}\"" fi fi @@ -2506,6 +2531,7 @@ for name in $DATA_FILES; do echo "[OK] ${what}: ${dest}" else echo "[WARN] Could not install the ${what}. Re-run setup.sh to try again." + note_missing "the ${what}" "${SETUP_CMD} \"${GENOME_DIR}\"" fi done # gnomAD v3.1's chrM sites (step 20 annotates the calls with them), checked @@ -2521,12 +2547,17 @@ else echo "[OK] gnomAD v3.1 mitochondrial sites (step 20): ${GNOMAD_MITO}" else echo "[WARN] Could not install gnomAD's mitochondrial sites. Step 20 runs without its gnomAD columns; re-run setup.sh to try again." + note_missing "gnomAD v3.1 mitochondrial sites (step 20)" "${SETUP_CMD} \"${GENOME_DIR}\"" fi fi -# A failed download does not stop setup here either. -install_sample_qc_data || true -install_strchive_data || true -install_tandem_repeats || true +# A failed download does not stop setup here either: it goes on the list of +# what is missing, which the summary prints with the command that retries it. +install_sample_qc_data \ + || note_missing "somalier's sites and VerifyBamID2's panel (step 33)" "${SETUP_CMD} --sample-qc-data \"${GENOME_DIR}\"" +install_strchive_data \ + || note_missing "STRchive ${STRCHIVE_VERSION} catalog (steps 09 and 9b)" "${SETUP_CMD} --strchive-data \"${GENOME_DIR}\"" +install_tandem_repeats \ + || note_missing "GIAB tandem-repeat map (step 04c)" "${SETUP_CMD} \"${GENOME_DIR}\"" # Step 25: its scores, pgsc_calc and its plugin (warnings only). install_prs @@ -2534,11 +2565,13 @@ install_prs # Phase 5: AnnotSV annotation data (step 5, a default step) ############################################################################### # The bundle, gnomAD-SV's common SVs and the MANE Select transcripts -# (install_annotsv_data above). A failure is a warning: the rest of setup goes on. +# (install_annotsv_data above). A failure goes on the list of what is missing: +# the rest of setup goes on. echo "" echo "=== Phase 5: AnnotSV annotation data (~5.3 GB + ~1.7 GB download, ~20 GB unpacked) ===" -install_annotsv_data || true +install_annotsv_data \ + || note_missing "AnnotSV data: the bundle, gnomAD-SV ${GNOMAD_SV_VERSION} and MANE ${MANE_RELEASE} (step 05)" "${SETUP_CMD} --annotsv-data \"${GENOME_DIR}\"" ############################################################################### # Phase 6: Clinical reference data (default) @@ -2546,6 +2579,7 @@ install_annotsv_data || true echo "" echo "=== Phase 6: Clinical reference data ===" if $CLINICAL_DATA; then + # Each item it could not install is on the list already (note_missing). install_clinical_data || true else echo "[SKIP] Clinical data (--no-clinical-data): steps 13, 30, 23, 31 and 17 need the caches or bundle." @@ -2555,19 +2589,38 @@ fi ############################################################################### # Summary ############################################################################### +# "Setup complete!" only when nothing is missing. Otherwise each missing item +# with the command that retries it, and the exit status SETUP_PARTIAL_EXIT (3), +# so a caller can tell a partial setup from a full one. echo "" echo "============================================" -echo " Setup complete!" +if [ "${#MISSING[@]}" -eq 0 ]; then + echo " Setup complete!" +else + n=${#MISSING[@]} + echo " Setup finished: core ready, ${n} item$([ "$n" -eq 1 ] || echo s) missing" +fi echo "" echo " Reference genome: ${REFDIR}/" echo " ClinVar database: ${CLINVARDIR}/" echo " Docker images: ${PULLED} pulled, ${SKIPPED} cached" echo "============================================" +if [ "${#MISSING[@]}" -gt 0 ]; then + echo "" + echo "Missing (the steps named run without it or say it is missing; the warnings above say why):" + for item in "${MISSING[@]}"; do + echo " - ${item%%|*}" + echo " retry: ${item#*|}" + done +fi echo "" echo "Next steps:" echo " 1. Place your FASTQ/BAM/VCF in: ${GENOME_DIR}//" echo " 2. Run: export GENOME_DIR=${GENOME_DIR}" echo " 3. Run: ./scripts/validate-setup.sh " echo " 4. Run: ./scripts/run-all.sh " -echo "" -echo "Review any warnings above; use --clinical-data to retry missing clinical downloads." +if [ "${#MISSING[@]}" -gt 0 ]; then + echo "" + echo "Setup exits with status ${SETUP_PARTIAL_EXIT}: a partial setup. Run the retry commands above, or this setup again." + exit "$SETUP_PARTIAL_EXIT" +fi diff --git a/tests/e2e/mocha-1.sh b/tests/e2e/mocha-1.sh index 31bfdc235..e6220858a 100755 --- a/tests/e2e/mocha-1.sh +++ b/tests/e2e/mocha-1.sh @@ -10,8 +10,14 @@ # the genetic map of chr20 unpacked, the folder under 200 MB, and cnps.bed # (the panel's common CNVs) written, in MoChA's --cnp format when not empty; # - HG002 (case 21's VCF) phased by Beagle on that slice: phased hets, MoChA -# makes no call and none is reported, and MoChA's log names cnps.bed as -# its --cnp list; +# makes no call on chr20, none of its calls is reported, and MoChA's log +# names cnps.bed as its --cnp list. The fixture's other autosomes, which +# the cut panel has no file for, still reach the sites table, unphased, so +# the sample is not assessed and the message names each of them (red +# control: the same check fails on the sites table cut to chr20, all the +# step kept before). MoChA may call on those other slices (a germline CNV +# such as HG002's chr4 UGT2B17 deletion, untyped since the cut panel's +# cnps.bed has chr20 only); such a call is reported "no"; # - MOSAIC, a synthetic phased VCF on chr16, chr19, chr20 and chr22 at 30x # (REF bases from the fixture reference) with a 20 Mb CN-LOH on 20p from # the telomere in 15% of the cells (the haplotype balance 0.575 instead of @@ -23,10 +29,11 @@ # check fails on its log (red control of that check); # - nextflow run main.nf --tools mocha,html_report on HG002 writes the # bash step's call table byte for byte, the same summary values and a -# MoChA log naming cnps.bed, and the HTML report has the card. MoChA makes -# no call on HG002, so the byte check covers a header-only table: the -# summary values (sites, depth, hets per 10 Mb, chromosomes) are the mirror -# evidence. Control: MOSAIC's table differs. +# MoChA log naming cnps.bed, and the HTML report has the card. The byte +# check covers every row MoChA wrote on HG002, its calls on the other +# slices included, and the summary values (sites, depth, hets per 10 Mb, +# chromosomes, calls) are mirror evidence too. Control: MOSAIC's table +# differs. . "$(dirname "$0")/lib.sh" G="$GENOME_DIR" T="$CASE_TMP" @@ -34,6 +41,12 @@ MD="${G}/reference/mocha-${MOCHA_PANEL_VERSION}" summ() { awk -F'\t' -v k="$2" '$1 == k {print $2}' "${G}/$1/mosaic/$1_mocha_summary.tsv" 2>/dev/null; } reported() { awk -F'\t' 'NR > 1 && $13 == "yes"' "${G}/$1/mosaic/$1_mocha.tsv" 2>/dev/null; } nrep() { reported "$1" | grep -c . || true; } +# chr20_calls SAMPLE: MoChA's calls on chr20, reported or not ("missing" with no table). +chr20_calls() { + local f="${G}/$1/mosaic/$1_mocha.tsv" + [ -s "$f" ] || { echo missing; return; } + awk -F'\t' 'NR > 1 && $1 == "chr20"' "$f" | grep -c . || true +} between() { awk -v x="${1:-}" -v lo="$2" -v hi="$3" 'BEGIN {exit !(x ~ /^[0-9.]+$/ && x + 0 >= lo && x + 0 <= hi)}'; } differs() { [ -n "$1" ] && [ "$1" != "$2" ]; } not_same() { ! cmp -s "$1" "$2"; } @@ -42,6 +55,18 @@ not() { ! "$@"; } cnp_logged() { grep -Eq '^Regions to genotype: .*cnps\.bed$' "$1" 2>/dev/null; } cnp_format() { awk -F'\t' 'NF != 4 || $2 !~ /^[0-9]+$/ || $3 !~ /^[0-9]+$/ || $3 <= $2 || $4 !~ /^(CNV|DEL|DUP)$/ {exit 1}' "$1"; } status_ok() { [ "$1" = assessed ] || { [ "$1" = "not assessed" ] && [ -n "$2" ]; }; } +# unpanelled SITES MESSAGE: the sites table has an autosome other than chr20, +# the cut panel's one chromosome, with heterozygous sites; each such autosome +# has none phased, and the message names it as "chrN (0 of H)". +unpanelled() { + local c h p n=0 + while IFS=$'\t' read -r c _ _ _ h p; do + [ "$c" != chr20 ] && [ "${h:-0}" != 0 ] || continue + [ "$p" = 0 ] && [[ "$2" == *"${c} (0 of ${h})"* ]] || return 1 + n=$((n + 1)) + done < "$1" + [ "$n" -gt 0 ] +} retry() { local i; for i in 1 2 3 4; do "$@" && return 0; [ "$i" -lt 4 ] && sleep $((i * 15)); done; return 1; } show() { echo "--- $1"; cat "${G}/$1/mosaic/$1_mocha_summary.tsv" "${G}/$1/mosaic/$1_mocha.tsv" \ "${G}/$1/mosaic/$1_mocha_calls_raw.tsv" "${G}/$1/mosaic/$1_mocha_stats.tsv" "${G}/$1/mosaic/$1_mocha_sites.tsv" 2>/dev/null; } @@ -90,18 +115,27 @@ run_step 51-mocha.sh "$SAMPLE" check_step_exit 51-mocha.sh show "$SAMPLE" check_eq "HG002: phased by Beagle" "$(summ "$SAMPLE" phasing)" beagle -check_eq "HG002: MoChA read chr20, the panel's one chromosome" "$(summ "$SAMPLE" chromosomes)" chr20 +check "HG002: MoChA read chr20, the panel's one chromosome, and more ($(summ "$SAMPLE" chromosomes))" \ + has '^(chr[0-9]+,)+chr20(,chr[0-9]+)*$|^chr20(,chr[0-9]+)+$' "$(summ "$SAMPLE" chromosomes)" +SITES="${G}/${SAMPLE}/mosaic/${SAMPLE}_mocha_sites.tsv" +check_eq "HG002: the autosomes the panel has no file for make the sample not assessed" "$(summ "$SAMPLE" status)" "not assessed" +check "HG002: the message names each of them with 0 phased sites ($(summ "$SAMPLE" message))" \ + unpanelled "$SITES" "$(summ "$SAMPLE" message)" +awk -F'\t' '$1 == "chr20"' "$SITES" > "${T}/sites_chr20.tsv" +check "red control: that check fails on the sites table cut to chr20 (all the step kept before)" \ + not unpanelled "${T}/sites_chr20.tsv" "$(summ "$SAMPLE" message)" check_ge "HG002: phased heterozygous sites" "$(summ "$SAMPLE" phased_het_sites)" 100 -# 500 kb hold about 500 hets, too few for the genome-wide BAF autocorrelation -# MoChA's sample filter reads, so the status may be either state (with a -# reason); MoChA's own call count is what must be zero. +# The status is not assessed (the autosomes above), with its reason. MoChA may +# call on the unphased slices (a germline CNV, untyped because the cut panel's +# cnps.bed has chr20 only), so the zero is its call count on chr20, and none of +# its calls may be reported. check "HG002: a status with its reason ($(summ "$SAMPLE" status), baf_auto $(summ "$SAMPLE" baf_auto))" \ status_ok "$(summ "$SAMPLE" status)" "$(summ "$SAMPLE" message)" -check_eq "HG002 chr20: MoChA made no call" "$(summ "$SAMPLE" calls_mocha)" 0 -check_eq "HG002 chr20: no mosaic call reported" "$(nrep "$SAMPLE")" 0 +check_eq "HG002: MoChA made no call on chr20 ($(summ "$SAMPLE" calls_mocha) on all slices)" "$(chr20_calls "$SAMPLE")" 0 +check_eq "HG002: no mosaic call reported" "$(nrep "$SAMPLE")" 0 check "HG002: MoChA read cnps.bed as its --cnp list (its log)" cnp_logged "${G}/${SAMPLE}/mosaic/${SAMPLE}_mocha.log" -echo "mocha: HG002 chr20 slice: $(summ "$SAMPLE" het_sites) hets ($(summ "$SAMPLE" phased_het_sites) phased)," \ - "depth $(summ "$SAMPLE" depth)x, floor $(summ "$SAMPLE" floor), $(summ "$SAMPLE" calls_mocha) MoChA calls" >> "$E2E_NOTES" +echo "mocha: HG002, panel cut to chr20: $(summ "$SAMPLE" het_sites) hets ($(summ "$SAMPLE" phased_het_sites) phased)," \ + "depth $(summ "$SAMPLE" depth)x, floor $(summ "$SAMPLE" floor), $(summ "$SAMPLE" calls_mocha) MoChA calls ($(chr20_calls "$SAMPLE") on chr20)" >> "$E2E_NOTES" # --- MOSAIC and MOSAIC0: a planted CN-LOH, and none --------------------------------- for s in MOSAIC MOSAIC0; do rm -rf "${G:?}/${s}"; mkdir -p "${G}/${s}/vcf"; done @@ -129,7 +163,7 @@ check "MOSAIC: and ends at 19 to 21 Mb ($(cut -f3 <<< "$ROW"))" between "$(cut - check "MOSAIC: cell fraction 0.10 to 0.20 ($(cut -f7 <<< "$ROW"))" between "$(cut -f7 <<< "$ROW")" 0.10 0.20 check_eq "MOSAIC0 (no shift): no call reported" "$(nrep MOSAIC0)" 0 check "red control: the one-call check fails on MOSAIC0" test "$(nrep MOSAIC0)" != 1 -check "red control: the no-MoChA-call check on HG002 fails on MOSAIC ($(summ MOSAIC calls_mocha))" differs "$(summ MOSAIC calls_mocha)" 0 +check "red control: the no-chr20-call check on HG002 fails on MOSAIC ($(chr20_calls MOSAIC))" differs "$(chr20_calls MOSAIC)" 0 check "MOSAIC0, no panel folder: the step warns that there is no --cnp list" has 'no --cnp list' "$(cat "$STEP_LOG")" check "red control: the --cnp log check fails on MOSAIC0's log" not cnp_logged "${G}/MOSAIC0/mosaic/MOSAIC0_mocha.log" echo "mocha: MOSAIC planted CN-LOH chr20:1-20000000 at 0.15: $(cut -f1-7 <<< "$ROW" | tr '\t' ' ');" \ @@ -155,7 +189,7 @@ A="${G}/${SAMPLE}/mosaic/${SAMPLE}_mocha.tsv" B="${NF_OUT}/${SAMPLE}/mosaic/${SA check "Nextflow wrote ${SAMPLE}_mocha.tsv" test -s "$B" check "Nextflow's ${SAMPLE}_mocha.tsv is the bash step's" cmp -s "$A" "$B" cmp -s "$A" "$B" || diff "$A" "$B" | head -20 -for k in status chromosomes het_sites depth het_per_10mb calls_mocha calls_reported floor_min phasing; do +for k in status message chromosomes het_sites depth het_per_10mb calls_mocha calls_reported floor_min phasing; do check_eq "Nextflow's summary: ${k} as the bash step's" \ "$(awk -F'\t' -v k="$k" '$1 == k {print $2}' "${NF_OUT}/${SAMPLE}/mosaic/${SAMPLE}_mocha_summary.tsv" 2>/dev/null)" "$(summ "$SAMPLE" "$k")" done diff --git a/tests/fake-docker/base-setup-clinvar-raw-only.sh b/tests/fake-docker/base-setup-clinvar-raw-only.sh index d66ea427a..7953e208b 100755 --- a/tests/fake-docker/base-setup-clinvar-raw-only.sh +++ b/tests/fake-docker/base-setup-clinvar-raw-only.sh @@ -18,4 +18,4 @@ awk '/^run / && /clinvar_chr\.vcf\.gz/ && !/clinvar_pathogenic/ { found = 1 } EN docker_log_has '^run .*clinvar_pathogenic_chr\.vcf\.gz' \ "setup.sh did not build clinvar_pathogenic_chr.vcf.gz" output_lacks setup 'unbound variable' -expect_rc setup 0 +expect_rc setup 3 diff --git a/tests/fake-docker/base-setup-no-wget.sh b/tests/fake-docker/base-setup-no-wget.sh index 18eda13a9..686a8cb5c 100755 --- a/tests/fake-docker/base-setup-no-wget.sh +++ b/tests/fake-docker/base-setup-no-wget.sh @@ -13,4 +13,4 @@ docker_log_has '^curl [^ ]*\.fa' "setup.sh did not download the reference with c docker_log_has '^curl [^ ]*clinvar\.vcf\.gz ' "setup.sh did not download ClinVar with curl" output_lacks setup 'wget: command not found' output_lacks setup 'unbound variable' -expect_rc setup 0 +expect_rc setup 3 diff --git a/tests/fake-docker/base-setup.sh b/tests/fake-docker/base-setup.sh index 82c7ee9dd..6c2315c86 100755 --- a/tests/fake-docker/base-setup.sh +++ b/tests/fake-docker/base-setup.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # setup.sh on an empty GENOME_DIR: the reference and ClinVar downloads go # through the fake wget, the image checks through the fake docker. It must -# finish with exit 0. +# finish with exit 3, a partial setup: the fake downloads fail their pinned checksums. # shellcheck source=../../scripts/ci/fake-docker/lib.sh . "${REPO_ROOT:?}/scripts/ci/fake-docker/lib.sh" seed_clinical_downloads @@ -9,9 +9,9 @@ seed_clinical_downloads G="${CASE_WORK}/genome" mkdir -p "$G" -run_expect 0 setup "${SCRIPTS}/setup.sh" "$G" +run_expect 3 setup "${SCRIPTS}/setup.sh" "$G" output_lacks setup 'unbound variable' -output_has setup 'Setup complete!' +output_has setup '^ Setup finished: core ready, [0-9]+ items missing$' for f in reference/GRCh38_no_alt_analysis_set.fasta reference/GRCh38_no_alt_analysis_set.fasta.fai \ clinvar/clinvar.vcf.gz clinvar/clinvar.vcf.gz.tbi \ diff --git a/tests/fake-docker/bash-step-setup-data.sh b/tests/fake-docker/bash-step-setup-data.sh index 7282da89b..1849987eb 100755 --- a/tests/fake-docker/bash-step-setup-data.sh +++ b/tests/fake-docker/bash-step-setup-data.sh @@ -41,7 +41,7 @@ run_expect 1 chip-no-dict "${SCRIPTS}/chip-to-vcf.sh" chip1 output_has chip-no-dict 'Required file not found: .*GRCh38_no_alt_analysis_set\.dict' if grep -q '^run ' "$FAKE_DOCKER_LOG"; then fail "chip-to-vcf.sh started a container without the sequence dictionary"; fi -run_expect 0 setup "${SCRIPTS}/setup.sh" "$GENOME_DIR" +run_expect 3 setup "${SCRIPTS}/setup.sh" "$GENOME_DIR" docker_log_has '^run image=[^ ]*gatk.* gatk CreateSequenceDictionary -R /genome/reference/GRCh38_no_alt_analysis_set\.fasta ' \ "setup.sh did not create the sequence dictionary" [ -f "${GENOME_DIR}/reference/GRCh38_no_alt_analysis_set.dict" ] || fail "setup.sh left no GRCh38_no_alt_analysis_set.dict" diff --git a/tests/fake-docker/reference-setup.sh b/tests/fake-docker/reference-setup.sh index 67cbba447..455d6c2fc 100755 --- a/tests/fake-docker/reference-setup.sh +++ b/tests/fake-docker/reference-setup.sh @@ -36,7 +36,7 @@ serve() { G="${CASE_WORK}/genome" mkdir -p "$G" serve "${CASE_WORK}/served" -FAKE_DOWNLOAD_DIR="${CASE_WORK}/served" run_expect 0 setup "${SCRIPTS}/setup.sh" "$G" +FAKE_DOWNLOAD_DIR="${CASE_WORK}/served" run_expect 3 setup "${SCRIPTS}/setup.sh" "$G" docker_log_has "^curl ${NCBI//./\\.}/${NAME//./\\.}\\.gz -> " "setup.sh did not download the no-ALT analysis set from NCBI" docker_log_has "^curl ${NCBI//./\\.}/md5checksums\\.txt -> " "setup.sh did not read NCBI's md5checksums.txt" docker_log_has "^curl ${NCBI//./\\.}/${NAME//./\\.}\\.fai -> " "setup.sh did not download NCBI's .fai" diff --git a/tests/fake-docker/rv-setup-partial-status.sh b/tests/fake-docker/rv-setup-partial-status.sh new file mode 100755 index 000000000..9558030ec --- /dev/null +++ b/tests/fake-docker/rv-setup-partial-status.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash +# setup.sh says "Setup complete!" and exits 0 only when it installed +# everything. When an install fails or is skipped, it prints "Setup finished: +# core ready, N items missing", lists each item with the command that retries +# it, and exits 3. +# +# The fake downloads fail their pinned checksums and og is not built for both +# Linux arches here, so a plain fake setup always misses something. To reach +# the complete case, setup.sh runs from a copy of the checkout (not a git +# checkout, so og.sh has no tree to compare) with two placeholder og binaries, +# and GENOME_DIR holds what the pinned downloads would have installed. A fake +# sha256sum gives the two files setup.sh checks by digest their pinned values. +# shellcheck source=../../scripts/ci/fake-docker/lib.sh +. "${REPO_ROOT:?}/scripts/ci/fake-docker/lib.sh" +# shellcheck source=../../versions.env +. "${REPO_ROOT}/versions.env" +seed_clinical_downloads +export FETCH_TRIES=1 FETCH_WAIT=0 + +COPY="${CASE_WORK}/checkout" +mkdir -p "${COPY}/rust" "${COPY}/bin/.og" +cp -R "${REPO_ROOT}/scripts" "${REPO_ROOT}/assets" "${REPO_ROOT}/versions.env" "$COPY/" +cp "${REPO_ROOT}/rust/og.sh" "${COPY}/rust/" +for f in "${REPO_ROOT}"/bin/*; do + [ -f "$f" ] && cp "$f" "${COPY}/bin/" +done +for a in x86_64 aarch64; do + printf '#!/bin/sh\nexit 0\n' > "${COPY}/bin/.og/og-linux-${a}" + chmod +x "${COPY}/bin/.og/og-linux-${a}" +done +SETUP="${COPY}/scripts/setup.sh" + +# A fake sha256sum: the pinned digest for the tandem-repeat map and gnomAD's +# chrM sites, the real one for every other file. +real_sha=$(command -v sha256sum || true) +mkdir -p "${CASE_WORK}/sha-bin" +cat > "${CASE_WORK}/sha-bin/sha256sum" < "${r}/GRCh38_no_alt_analysis_set.dict" + # The pinned data files, by the paths common.sh's data_file gives them. + # shellcheck disable=SC2016 # expanded by the inner bash + while read -r u; do + mkdir -p "${u%/*}" + printf 'x\n' > "$u" + done < <(GENOME_DIR=$g bash -c '. "$1"; for n in $DATA_FILES; do data_file "$n" || true; done' _ "${COPY}/scripts/lib/common.sh") + for u in somalier/sites.hg38.vcf.gz \ + verifybamid2/1000g.phase3.100k.b38.vcf.gz.dat.UD verifybamid2/1000g.phase3.100k.b38.vcf.gz.dat.mu \ + verifybamid2/1000g.phase3.100k.b38.vcf.gz.dat.bed "${TANDEM_REPEATS_URL##*/}" \ + "$(basename "${TANDEM_REPEATS_URL}" .gz)" "strchive-${STRCHIVE_VERSION}/catalog.json" \ + "strchive-${STRCHIVE_VERSION}/str_ranges.tsv"; do + printf 'x\n' > "${r}/${u}" + done + printf 'x\n' > "${g}/gnomad_mito/${GNOMAD_MITO_URL##*/}" + printf 'release=%s eh=%s loci=%s converter=og-str-catalog-1 extra=off\n' \ + "$STRCHIVE_VERSION" "$STRCHIVE_EH_SHA256" "$STRCHIVE_LOCI_SHA256" > "${r}/strchive-${STRCHIVE_VERSION}/catalog.stamp" + [ "${2:-}" = no-annotsv ] && return 0 + mkdir -p "${a}/Annotations_Human/Genes/GRCh38" "${a}/Annotations_Human/SVincludedInFt/BenignSV/GRCh38" + printf 'gnomAD-SV %s sha256 %s\n' "$GNOMAD_SV_VERSION" "$GNOMAD_SV_SHA256" \ + > "${a}/Annotations_Human/SVincludedInFt/BenignSV/GRCh38/gnomad-sv.installed" + printf 'NM_000001.1\n' > "${a}/mane_select_refseq.txt" + printf 'MANE %s sha256 %s\n' "$MANE_RELEASE" "$MANE_SUMMARY_SHA256" > "${a}/mane_select_refseq.release" +} + +# 1. Everything installs: the banner, no missing list, exit 0. +G="${CASE_WORK}/complete" +seed_installed "$G" +run_expect 0 complete "$SETUP" "$G" +output_has complete '^ Setup complete!$' +output_lacks complete 'Setup finished|Missing \(|exits with status 3' + +# 2. The AnnotSV download fails: no banner, AnnotSV and its retry, exit 3. +G="${CASE_WORK}/annotsv" +seed_installed "$G" no-annotsv +FAKE_DOWNLOAD_FAIL='lbgi\.fr' run_expect 3 annotsv "$SETUP" "$G" +output_lacks annotsv 'Setup complete' +output_has annotsv '^ Setup finished: core ready, 1 item missing$' +output_has annotsv '^ - AnnotSV data: the bundle, gnomAD-SV .* \(step 05\)$' +output_has annotsv "^ retry: .*/scripts/setup\\.sh --annotsv-data \"${G}\"$" +output_has annotsv 'Setup exits with status 3: a partial setup' +[ "$(grep -c '^ - ' "${CASE_WORK}/annotsv.out")" -eq 1 ] || fail 'annotsv lists more than the AnnotSV data' + +# 3. Too little disk for the two VEP caches: each skipped cache is listed +# with its own retry, the bundle still installs, exit 3. +mkdir -p "${CASE_WORK}/low-bin" +cat > "${CASE_WORK}/low-bin/df" <<'DF' +#!/usr/bin/env bash +echo 'Filesystem 1024-blocks Used Available Capacity Mounted on' +echo 'fakefs 41943040 0 41943040 0% /' +DF +chmod +x "${CASE_WORK}/low-bin/df" +G="${CASE_WORK}/low" +seed_installed "$G" +PATH="${CASE_WORK}/low-bin:${PATH}" run_expect 3 low "$SETUP" "$G" +output_lacks low 'Setup complete' +output_has low '^ Setup finished: core ready, 2 items missing$' +output_has low "^ - VEP ${VEP_CACHE_RELEASE} cache \\(steps 13, 30, 23 and 31\\)$" +output_has low "^ retry: .*/scripts/setup\\.sh --vep-cache \"${G}\"$" +output_has low "^ - VEP ${PCGR_VEP_CACHE_RELEASE} cache \\(step 17, CPSR\\)$" +output_has low "^ retry: .*/scripts/setup\\.sh --clinical-data \"${G}\"$" +output_lacks low '^ - PCGR' +[ -d "${G}/pcgr_data/${PCGR_DATA_BUNDLE}/data" ] || fail 'the bundle was not installed beside the skipped caches' + +# 4. --no-clinical-data is a choice, not a gap: complete, exit 0. +G="${CASE_WORK}/opt-out" +seed_installed "$G" +run_expect 0 opt_out "$SETUP" --no-clinical-data "$G" +output_has opt_out '^ Setup complete!$' + +# 5. A plain fake setup misses several items: each has a retry line. +G="${CASE_WORK}/plain" +mkdir -p "$G" +run_expect 3 plain "$SETUP" "$G" +output_lacks plain 'Setup complete' +items=$(grep -c '^ - ' "${CASE_WORK}/plain.out") +retries=$(grep -c '^ retry: ' "${CASE_WORK}/plain.out") +[ "$items" -ge 5 ] && [ "$items" -eq "$retries" ] || fail "plain lists ${items} items and ${retries} retry lines" +output_has plain "^ Setup finished: core ready, ${items} items missing$" +output_has plain "retry: .*/scripts/setup\\.sh --sample-qc-data \"${G}\"$" +output_has plain "retry: .*/scripts/setup\\.sh --strchive-data \"${G}\"$" diff --git a/tests/fake-docker/setup-clinical-default.sh b/tests/fake-docker/setup-clinical-default.sh index 4c6b49e46..618309cfc 100644 --- a/tests/fake-docker/setup-clinical-default.sh +++ b/tests/fake-docker/setup-clinical-default.sh @@ -5,8 +5,8 @@ seed_clinical_downloads export FETCH_TRIES=1 FETCH_WAIT=0 G="${CASE_WORK}/without" -run_expect 0 without "${SCRIPTS}/setup.sh" --no-clinical-data "$G" -output_has without 'Setup complete!' +run_expect 3 without "${SCRIPTS}/setup.sh" --no-clinical-data "$G" +output_has without 'Setup finished: core ready' if grep -E '^(wget|curl).*homo_sapiens_vep_|^(wget|curl).*pcgr_ref_data' "$FAKE_DOCKER_LOG"; then fail '--no-clinical-data downloaded clinical data' fi diff --git a/tests/fake-docker/setup-gencode-one-fetch.sh b/tests/fake-docker/setup-gencode-one-fetch.sh index 824bdbdc4..0a3c1cf9f 100755 --- a/tests/fake-docker/setup-gencode-one-fetch.sh +++ b/tests/fake-docker/setup-gencode-one-fetch.sh @@ -42,7 +42,7 @@ fetches() { grep -cE "^(curl|wget) [^ ]*/${GTF//./\\.} " "$FAKE_DOCKER_LOG" || t # --- 1. one fetch, both files --------------------------------------------------------- : > "$FAKE_DOCKER_LOG" -FAKE_DOWNLOAD_DIR="$SERVED" run_expect 0 setup "${SCRIPTS}/setup.sh" "$GENOME_DIR" +FAKE_DOWNLOAD_DIR="$SERVED" run_expect 3 setup "${SCRIPTS}/setup.sh" "$GENOME_DIR" [ "$(fetches)" -eq 1 ] || fail "setup.sh downloaded ${GTF} $(fetches) times, expected once" [ -s "$GENES" ] || fail "setup.sh wrote no ${GENES}" [ -s "$EXONS" ] || fail "setup.sh wrote no ${EXONS}" @@ -65,7 +65,7 @@ touch -t 202001010000 "$GENES" run_rc validate-missing "${SCRIPTS}/validate-setup.sh" sample1 output_has validate-missing "GENCODE ${REL} exons \(steps 06 and 17 skip their cDNA read check without them\) not found at: ${EXONS//./\\.}" : > "$FAKE_DOCKER_LOG" -FAKE_DOWNLOAD_DIR="$SERVED" run_expect 0 setup-again "${SCRIPTS}/setup.sh" "$GENOME_DIR" +FAKE_DOWNLOAD_DIR="$SERVED" run_expect 3 setup-again "${SCRIPTS}/setup.sh" "$GENOME_DIR" [ "$(fetches)" -eq 1 ] || fail "the second setup.sh downloaded ${GTF} $(fetches) times, expected once" [ "$(cat "$EXONS" 2>/dev/null)" = "$want" ] || fail "the second setup.sh did not rebuild the exon BED" [ -z "$(find "$GENES" -newer "${SERVED}/MD5SUMS")" ] || fail "the second setup.sh rewrote the gene file it already had" diff --git a/tests/fake-docker/single-source-downloads.sh b/tests/fake-docker/single-source-downloads.sh index 42ec1bf6f..c26bbef41 100755 --- a/tests/fake-docker/single-source-downloads.sh +++ b/tests/fake-docker/single-source-downloads.sh @@ -25,7 +25,7 @@ FAKE_DOWNLOAD_PARTIAL='clinvar\.vcf\.gz$' run_rc setup-dropped "${SCRIPTS}/setup [ ! -e "${G}/clinvar/clinvar.vcf.gz" ] || fail "a cut download was stored as clinvar.vcf.gz" : > "$FAKE_DOCKER_LOG" -run_expect 0 setup-again "${SCRIPTS}/setup.sh" "$G" +run_expect 3 setup-again "${SCRIPTS}/setup.sh" "$G" docker_log_has '^curl [^ ]*/clinvar\.vcf\.gz -> ' "the second run did not download ClinVar again" [ -s "${G}/clinvar/clinvar.vcf.gz" ] || fail "the second run left no clinvar.vcf.gz" [ ! -e "${G}/clinvar/clinvar.vcf.gz.part" ] || fail "clinvar.vcf.gz.part is still there after a complete download"