Skip to content

Commit 2207350

Browse files
authored
Merge landing batch b: 3 pull requests (#489 #474 #468)
One merge for a batch of independent pull requests. Each was reviewed in its own lane. The tip merges every head below, and one end-to-end run plus the fast checks passed on that exact tree. - #489 - #474 refactor(docs): check the docs site with og dev instead of a Python script - #468 refactor(steps): find vendor data paths with sh instead of inline Python E2E evidence: https://github.com/geiresx/OwnGenome/actions/runs/44289882413
2 parents 85652cb + 7d203bb commit 2207350

53 files changed

Lines changed: 10423 additions & 2838 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/scripts/site-check.py‎

Lines changed: 0 additions & 464 deletions
This file was deleted.

‎.github/workflows/docs.yml‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@ on:
2121
- 'docs/**'
2222
- 'zensical.toml'
2323
- '.github/workflows/docs.yml'
24-
- '.github/scripts/site-check.py'
2524
# Add every file a page includes with pymdownx.snippets (for example
2625
# docker-compose.yml, .env.example): a change to it must rebuild the
2726
# site, and a rename must fail the build.
@@ -86,12 +85,6 @@ jobs:
8685
- name: Build
8786
run: python -m zensical build --clean --strict
8887

89-
# What the strict build does not check: a page in no nav entry, a nav
90-
# entry for a missing file, a missing image, a /rooted link, a raw <title>
91-
# in a page. Each one was proven to turn this step red (validate-zensical.sh).
92-
- name: Check the built site
93-
run: python -I .github/scripts/site-check.py site
94-
9588
# og (rust/, docs/rust.md): the demo sample builder runs step 27's
9689
# parser, og pgx cpic-report (tests/og_binary.py reads OG), writes the
9790
# genome map's documents, og map build, and runs step 33's og qc sex-loss
@@ -113,6 +106,14 @@ jobs:
113106
OG_TREE_HASH=$(./og.sh tree-hash) cargo build --release --locked -p og
114107
echo "OG=${PWD}/target/release/og" >> "$GITHUB_ENV"
115108
109+
# What the strict build does not check: a page in no nav entry, a nav
110+
# entry for a missing file, a missing image, a /rooted link, a raw <title>
111+
# in a page (og dev site-check, rust/crates/og-dev/src/site_check.rs; its
112+
# tests plant each fault in rust/crates/og-dev/tests/site_check.rs).
113+
- name: Check the built site
114+
run: |
115+
"$OG" dev site-check site
116+
116117
# /demo/: the live site replaying an invented run of DEMO-001, the invented
117118
# sample of the docs pictures, through the real page and plugin code
118119
# (tests/demo/build_live_demo.py; standard library only). It is a static

‎CHANGELOG.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,12 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n
99
- The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2).
1010
- The live site's access log no longer names the finding a page opens. A finding's id is built from its gene and position, and the finding route wrote that id into the log's section column, though `docs/live-site.md` says the log never holds a result. The row now holds `-` there, as the other result routes do. The log keeps its columns and one row per page open. An og-live cargo test opens a finding page on a made-up run and reads the log; its control puts the old line back and must fail.
1111
- The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes.
12+
- `og report render` now writes the sections of the opt-in steps and of two small default ones in Rust: clonal haematopoiesis with its detection limit (step 49), viral DNA (step 47), mobile element insertions (step 50), MoChA's mosaic alterations (step 51), VNtyper (step 52), Locityper (step 53), the Exomiser ranking (step 48), telomere content and runs of homozygosity. Both reports match `bin/render_report.py` byte for byte on the parity corpus, except for these changes:
13+
- When the BAM has no unplaced unmapped reads, step 47 looked at no viral genome. The viral section now says the genomes were not assessed. The Python said "No read on any of the 24 viral genomes", which reads as a negative result.
14+
- A stat value from the summary is escaped in the HTML card. The Python put it into the page as it was.
15+
- The genome chart of the ROH card is not drawn yet: it comes with the port of the charts.
16+
- The lines of the SV callers and of the cDNA scan are ready as parts for the structural-variant and ClinVar sections, which come with the clinical sections.
17+
- The checks of `tests/test_{chip,viral,mei,mocha,vntyper,locityper,exomiser}_report.py` moved to cargo tests in og-chip, og-reports, og-summary and og-render, and run-all.sh's plan and the DONE guards of step 53 to the fake-docker case `port-report-optin-locityper`. The seven Python files are gone. Part of #411.
1218
- The phased family VCF is indexed by the Rust `og` binary. WhatsHap's step (step 38 and the Nextflow WHATSHAP_PED process) wrote `family.phased.vcf.gz.tbi` with a Python one-liner, `pysam.tabix_index`. It now runs `og vcf index -f`. The `.tbi` keeps its name and format, and its bytes, once inflated, are the ones htslib 1.24 writes (the htslib of the pysam and bcftools the pipeline pins): the same bins in the same order, the same chunks and the same linear index. Only the deflate stream can differ, as two deflate libraries differ. On a 5-million-record three-sample VCF it took 0.75 to 0.91 s and 31 MB at peak, where htslib's own `tabix` took 0.81 to 0.93 s and 19 MB.
1319
- As in htslib 1.24, a record ends at the largest of INFO/END, the REF length, the SVLEN of a `<DEL>`, `<DUP>`, `<INV>` or `<CNV>` ALT, and a sample's LEN on a gVCF block. A line with no position to read, unsorted records, a sequence that starts again, a record past 2^29 and an empty file stop the index with a message, and nothing is written. Without `-f` it never writes over an index, even one that appears while it runs.
1420
- Steps 02b and the variant benchmark resolve paths with `realpath`, then `readlink -f`, instead of a `python3` one-liner.

‎assets/chip_genes.tsv‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
# residue (R882: any change of that residue), one exact change (V617F), a codon range (102-292)
99
# or exons:A-B.
1010
# A change outside these classes is listed as other_change, not counted.
11-
# og chip check-table checks the schema; tests/test_chip_report.py runs it.
11+
# og chip check-table checks the schema; rust/crates/og-chip/tests/optin_chip.rs runs it.
1212
gene truncating missense note pmids
1313
DNMT3A all R882 The most common CHIP gene. R882 is the hotspot of DNMT3A in acute myeloid leukaemia; other missense changes are listed, not counted. 25426837;21067377
1414
TET2 all none Second most common CHIP gene. Truncating changes only; missense changes are listed, not counted. 25426837;19474426

‎docs/50-mei.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ On HG002's chr20 at 30x (13 million mapped reads) on a 4-CPU GitHub runner, MEGA
8484

8585
## Validation
8686

87-
The e2e case `tests/e2e/mei-1.sh` runs the step on HG002's chr20: the reads of chr20 in GIAB's 60x novoalign BAM, half the pairs (30x), realigned with minimap2 to chr20 alone, the unmapped reads dropped (so the case fits a runner's time), and the k-mer set of that chromosome. Its Alu calls are scored against the GIAB v5.0q insertions of HG002 on chr20 (RepeatMasker SINE/Alu, interspersed, 250 to 400 bp, inside the benchmark regions): recall must be at least 0.70 and precision at least 0.80, within 100 bp. On 2026-10-27 it found 12 of the 15 insertions (0.80), and 13 of its 15 Alu calls inside the benchmark regions matched a GIAB insertion (0.87); with the unmapped reads kept, the same 12 of 15 and 13 of 15. With every read pair within 1 kb of a GIAB Alu insertion removed, it found 0 of 15 and the recall check failed. The same case then runs AnnotSV on the calls with made-up AnnotSV data (the bundle of `tests/fixtures/annotsv/class45.sh` plus one made-up gene over all of chr20) and checks that every call comes back inside that gene, with its location there and class `NA`; the same VCF read without AnnotSV's table fails that check. It also runs the Nextflow pipeline with `--tools mei` and `--annotsv_annotations` on the same BAM and checks that its calls table is the bash step's, byte for byte. `tests/test_mei_report.py` plants an SVA in the FKTN window of a synthetic MEGAnE VCF and checks it is seen, and not seen without it, and joins a table in AnnotSV's columns back to the calls.
87+
The e2e case `tests/e2e/mei-1.sh` runs the step on HG002's chr20: the reads of chr20 in GIAB's 60x novoalign BAM, half the pairs (30x), realigned with minimap2 to chr20 alone, the unmapped reads dropped (so the case fits a runner's time), and the k-mer set of that chromosome. Its Alu calls are scored against the GIAB v5.0q insertions of HG002 on chr20 (RepeatMasker SINE/Alu, interspersed, 250 to 400 bp, inside the benchmark regions): recall must be at least 0.70 and precision at least 0.80, within 100 bp. On 2026-10-27 it found 12 of the 15 insertions (0.80), and 13 of its 15 Alu calls inside the benchmark regions matched a GIAB insertion (0.87); with the unmapped reads kept, the same 12 of 15 and 13 of 15. With every read pair within 1 kb of a GIAB Alu insertion removed, it found 0 of 15 and the recall check failed. The same case then runs AnnotSV on the calls with made-up AnnotSV data (the bundle of `tests/fixtures/annotsv/class45.sh` plus one made-up gene over all of chr20) and checks that every call comes back inside that gene, with its location there and class `NA`; the same VCF read without AnnotSV's table fails that check. It also runs the Nextflow pipeline with `--tools mei` and `--annotsv_annotations` on the same BAM and checks that its calls table is the bash step's, byte for byte. `rust/crates/og-reports/tests/optin_mei.rs` plants an SVA in the FKTN window of a synthetic MEGAnE VCF and checks it is seen, and not seen without it, and joins a table in AnnotSV's columns back to the calls.
8888

8989
## Notes
9090

‎modules/local/cnvpytor/main.nf‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,14 @@ process CNVPYTOR {
4747
# the image's site-packages, so this step needs a writable container FS —
4848
# run with the docker profile (the shell script bind-mounts instead, and
4949
# Singularity's read-only image is not supported for this step).
50-
DATADIR=\$(python -c 'import cnvpytor, os; print(os.path.dirname(cnvpytor.__file__) + "/data")')
50+
# The package sits under the lib directory of the prefix the cnvpytor
51+
# command is installed in; exactly one data directory, or the step stops.
52+
CNVPYTOR_BIN=\$(command -v cnvpytor)
53+
DATADIR=\$(find "\${CNVPYTOR_BIN%/bin/*}/lib" -type d -path '*/site-packages/cnvpytor/data')
54+
if [ ! -d "\$DATADIR" ]; then
55+
echo "ERROR: no single cnvpytor data directory in the image (\${DATADIR:-not found})" >&2
56+
exit 1
57+
fi
5158
cp -f ${cnvpytor_resources}/*.pytor "\$DATADIR"/
5259
5360
cnvpytor -root ${meta.id}.pytor -rd ${bam} -chrom ${canonical_chroms} -j ${task.cpus}

‎modules/local/stranger/main.nf‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,10 @@ process STRANGER {
4343
# the bundled GRCh38 catalog.
4444
CATALOG="${cat}"
4545
if [ -z "\$CATALOG" ] || ! grep -q '"NormalMax"' "\$CATALOG"; then
46-
CATALOG=\$(python -c 'import os, stranger; print(os.path.join(os.path.dirname(stranger.__file__), "resources", "variant_catalog_grch38.json"))')
46+
# The package sits under the lib directory of the prefix the stranger
47+
# command is installed in. Exactly one match, or the -f test stops.
48+
STRANGER_BIN=\$(command -v stranger)
49+
CATALOG=\$(find "\${STRANGER_BIN%/bin/*}/lib" -path '*/site-packages/stranger/resources/variant_catalog_grch38.json')
4750
fi
4851
echo "Repeat catalog: \$CATALOG"
4952
if [ ! -f "\$CATALOG" ]; then

0 commit comments

Comments
 (0)