From 34117e7631da90c9dc2ab016855ce7540e9d5ad8 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:18:42 +0100 Subject: [PATCH 01/16] ci(guard): fail when MITO_HAPLOGROUP leaves its task directory A cd or a readlink -f in the MITO_HAPLOGROUP process would bring back the haplogrep3 run in the binary's directory, which can be read-only. The new mito-workdir job checks only that process, so the readlink -f of the reference in MITO_EXTRACT_CHRM stays allowed. Its self-test plants a cd, a readlink -f and a renamed process, which must each fail, and a cd in MITO_EXTRACT_CHRM, which must not. Bead: followups-v18-docs-scope --- .github/workflows/guard.yml | 61 ++++++++++++++++++++++++++++++++++++- 1 file changed, 60 insertions(+), 1 deletion(-) diff --git a/.github/workflows/guard.yml b/.github/workflows/guard.yml index d203d4668..eaddc7558 100644 --- a/.github/workflows/guard.yml +++ b/.github/workflows/guard.yml @@ -75,6 +75,65 @@ jobs: if: ${{ !cancelled() }} # runs even when a step above failed, so each result shows run: scripts/ci/check-coupled-versions.sh + # MITO_HAPLOGROUP runs haplogrep3 in its Nextflow task directory: the + # pinned build finds its config and trees beside its jar, and the report + # must land where Nextflow collects it. A `cd` or a `readlink -f` in that + # process would bring back the run in the binary's directory, which can be + # read-only. Only that process is checked: the readlink -f of the + # reference in MITO_EXTRACT_CHRM, in the same file, is right. + mito-workdir: + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + # The self-test plants its faults first: a cd or readlink -f in + # MITO_HAPLOGROUP, or a renamed process, must be caught; a cd in + # MITO_EXTRACT_CHRM must not. Then the module itself. + - name: MITO_HAPLOGROUP has no cd and no readlink -f (self-test first) + run: | + set -euo pipefail + # check FILE: exit 1 and name each line when the MITO_HAPLOGROUP + # block (from its process line to the closing brace at column 1) + # has a cd or a readlink -f outside a comment, or when the file has + # no such block. + check() { + awk ' + /^process MITO_HAPLOGROUP[[:space:]]*\{/ { p = 1; seen = 1; next } + p && /^\}/ { p = 0 } + p && !/^[[:space:]]*#/ && (/(^|[^[:alnum:]_.-])cd([[:space:]]|$)/ || /readlink[[:space:]]+-f/) { + print FILENAME ":" FNR ": " $0; bad = 1 + } + END { + if (!seen) { print FILENAME ": no process MITO_HAPLOGROUP"; exit 1 } + exit bad + }' "$1" + } + M=modules/local/mito_haplogroup/main.nf + T=$(mktemp -d) + # plant NAME SED: a copy of the module with one edit; the edit must + # change the file, or the control would test the module itself. + plant() { + sed "$2" "$M" > "${T}/$1.nf" + if cmp -s "$M" "${T}/$1.nf"; then echo "FAIL: the plant $1 changed nothing"; exit 1; fi + } + # shellcheck disable=SC2016 # the planted lines are text, never expanded here + plant cd '/^process MITO_HAPLOGROUP/,/^}/ s|^ haplogrep3 classify| cd "$(dirname "$(command -v haplogrep3)")"\n haplogrep3 classify|' + # shellcheck disable=SC2016 # as above + plant readlink '/^process MITO_HAPLOGROUP/,/^}/ s|^ haplogrep3 classify| BIN=$(readlink -f "$(command -v haplogrep3)")\n haplogrep3 classify|' + plant renamed 's/^process MITO_HAPLOGROUP /process MITO_HAPLOGROUP_X /' + plant chrm-cd '/^process MITO_EXTRACT_CHRM/,/^}/ s|^ { echo ">chrM"| cd .\n { echo ">chrM"|' + for c in cd readlink renamed; do + if out=$(check "${T}/${c}.nf"); then + echo "FAIL: the planted ${c} was not caught"; exit 1 + fi + echo "caught ${c}: ${out}" + done + out=$(check "${T}/chrm-cd.nf") || { echo "FAIL: a cd in MITO_EXTRACT_CHRM was taken for MITO_HAPLOGROUP, or the module already fails:"; echo "$out"; exit 1; } + echo "OK: every plant in MITO_HAPLOGROUP is caught, and MITO_EXTRACT_CHRM is left alone" + check "$M" + echo "OK: MITO_HAPLOGROUP runs haplogrep3 in its task directory" + fake-docker: runs-on: ubuntu-latest timeout-minutes: 20 @@ -887,7 +946,7 @@ jobs: # job above failed or was skipped, and fails unless every one succeeded. ci-ok: if: always() - needs: [image-vars, images, coupled-versions, fake-docker, helper-binaries, og, unit-tests, rust, parity] + needs: [image-vars, images, coupled-versions, mito-workdir, fake-docker, helper-binaries, og, unit-tests, rust, parity] runs-on: ubuntu-latest timeout-minutes: 5 steps: From 270435576a5024a79d743603ae81ca2b18b9db56 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:18:42 +0100 Subject: [PATCH 02/16] docs: say steps 21, 32 and 9b run og and list the tandem-repeat map The 1.0.0 notes said those steps still run the Python CYP2D6 depth check and RFC1 motif count; both Python files are gone. The pinned-data table gets the GIAB tandem-repeat map setup.sh installs for step 04c. Bead: followups-v18-docs-scope --- docs/00-reference-setup.md | 1 + docs/releases/v1.0.0.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/00-reference-setup.md b/docs/00-reference-setup.md index b6baa03d2..74160adf1 100644 --- a/docs/00-reference-setup.md +++ b/docs/00-reference-setup.md @@ -141,6 +141,7 @@ It downloads the new file under `clinvar/.refresh/`, checks the md5, builds both | `reference/gencode.v50.basic.genes.gtf` | the gene lines of GENCODE 50's basic annotation (md5 checked) | step 8 (gene positions for T1K) | step 8 is skipped | | `reference/gencode.v50.basic.exons.bed` | the exon intervals of the same GENCODE 50 basic annotation, written from the same download as the gene lines (md5 checked) | steps 6 and 17 and Nextflow `--gencode_exons` with `clinvar` (cDNA read check); step 39 and `--gencode_exons` with `--pedigree` (deletions over an exon) | steps 6 and 17 and `CLINVAR_CDNA_CHECK` write every hit as `not_checked`; step 39 does not use deletions | | `gnomad_mito/gnomad.genomes.v3.1.sites.chrM.vcf.bgz` | gnomAD v3.1's mitochondrial sites, about 5 MB (`GNOMAD_MITO_URL`, sha256 `GNOMAD_MITO_SHA256` in `versions.env`, checked by `setup.sh` and `validate-setup.sh`) | step 20 (gnomAD AF_hom, AF_het and max_hl beside each chrM call) | step 20 writes its annotation table with the gnomAD columns empty, and `validate-setup.sh` warns (it fails on a file whose sha256 is not the pinned one) | +| `reference/GRCh38_AllTandemRepeats.bed.gz` and its unpacked copy `reference/GRCh38_AllTandemRepeats.bed` | every GRCh38 tandem repeat of GIAB's genome stratifications v3.6 (TRF and RepeatMasker simple repeats, 5 bp added on each side), about 9 MB, NIST data free to reuse (`TANDEM_REPEATS_URL`, sha256 `TANDEM_REPEATS_SHA256` in `versions.env`, checked by `setup.sh`). Sniffles2 reads a plain BED only, so setup keeps the unpacked copy beside the download | step 04c (`sniffles --tandem-repeats`, long reads only) | step 04c prints a note and runs Sniffles2 without it: SVs inside repeats get less precise breakpoints and merging | | `reference/somalier/sites.hg38.vcf.gz` | somalier's GRCh38 sites, 17,766 SNPs (sha256 checked) | step 33 (`somalier extract`), Nextflow `--somalier_sites` | step 33 stops and says to install it | | `reference/verifybamid2/1000g.phase3.100k.b38.vcf.gz.dat.{UD,mu,bed}` | VerifyBamID2's 1000 Genomes panel of 100,000 markers, from the VerifyBamID v2.0.3 release (sha256 checked) | step 33 (`verifybamid2`), Nextflow `--verifybamid2_panel` | step 33 stops and says to install it | diff --git a/docs/releases/v1.0.0.md b/docs/releases/v1.0.0.md index 2ee06b4b1..d210cd32b 100644 --- a/docs/releases/v1.0.0.md +++ b/docs/releases/v1.0.0.md @@ -123,7 +123,7 @@ OwnGenome 1.0.0 is the first stable release. It is one Nextflow pipeline that ta - Malformed input to any of the four filters (too few columns, a `POS` that is not a whole number) stops with exit 2 and names the file and line, instead of a Python traceback. - A run refuses an `og` built from other code than the checkout, and `run_manifest.tsv` records which Rust code was present. - The shared base for the ports that follow: number formatting and CSV and text output exactly as Python writes them, BGZF read and written on many cores, VCF records parsed in place without copying, region lookups through a `.tbi`, `.csi` or `.fai` index, and interval sets. -- The CYP2D6 depth check and the RFC1 motif count have Rust ports (`og pgx cyp2d6-*`, `og str motif`) that read BAMs without pysam and write what the Python writes. Steps 21, 32 and 9b still run the Python in 1.0.0. +- The CYP2D6 depth check and the RFC1 motif count have Rust ports (`og pgx cyp2d6-*`, `og str motif`) that read BAMs without pysam and write what the Python writes. Steps 21, 32 and 9b run them, in the bash scripts and the Nextflow modules; `bin/cyp2d6_depth_check.py` and `bin/rfc1_motif.py` are gone. - `og map build` writes the first layers of a per-person genome map document: chromosome bands, runs of homozygosity and every finding placed on the genome, with a child's closed results kept closed there too. `og map genes` names every gene and its exons, offline, from the GENCODE files setup installs. The live site serves the map and gene files behind its login; its page does not draw the map yet. - `setup.sh --masked-reference` masks the reference with `og` in one pass. It refuses a FASTA with old Mac line endings instead of passing it through unmasked, and counts overlapping rows once. From 83ef1f6d6e1013a6eb516dcd6261400bf7b2ec72 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:18:43 +0100 Subject: [PATCH 03/16] docs(changelog): note the mito-workdir guard and the doc fixes Bead: followups-v18-docs-scope --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05c3388e1..007a8025d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- Guard's new `mito-workdir` job fails when the Nextflow `MITO_HAPLOGROUP` process has a `cd` or a `readlink -f`: haplogrep3 must run in the task directory, where it finds its bundled config and where Nextflow collects the report. The check reads only that process, so the `readlink -f` of the reference in `MITO_EXTRACT_CHRM` stays allowed. Its self-test plants a `cd`, a `readlink -f` and a renamed process, which must each fail, and a `cd` in `MITO_EXTRACT_CHRM`, which must not. No output changes. +- The 1.0.0 release notes said steps 21, 32 and 9b still run Python for the CYP2D6 depth check and the RFC1 motif count. They run `og`, and the Python files are gone; the notes now say so. `docs/00-reference-setup.md` lists the GIAB tandem-repeat map `setup.sh` installs for step 04c, with its source, its checksum and what the step does without it. - The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes. - The phased family VCF is indexed by the Rust `og` binary. WhatsHap's step (step 38 and the Nextflow WHATSHAP_PED process) wrote `family.phased.vcf.gz.tbi` with a Python one-liner, `pysam.tabix_index`. It now runs `og vcf index -f`. The `.tbi` keeps its name and format, and its bytes, once inflated, are the ones htslib 1.24 writes (the htslib of the pysam and bcftools the pipeline pins): the same bins in the same order, the same chunks and the same linear index. Only the deflate stream can differ, as two deflate libraries differ. On a 5-million-record three-sample VCF it took 0.75 to 0.91 s and 31 MB at peak, where htslib's own `tabix` took 0.81 to 0.93 s and 19 MB. - As in htslib 1.24, a record ends at the largest of INFO/END, the REF length, the SVLEN of a ``, ``, `` or `` ALT, and a sample's LEN on a gVCF block. A line with no position to read, unsorted records, a sequence that starts again, a record past 2^29 and an empty file stop the index with a message, and nothing is written. Without `-f` it never writes over an index, even one that appears while it runs. From 2bb89215278d69558bbad16031459d866beca44f Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:20:44 +0100 Subject: [PATCH 04/16] fix(live): keep a gated person's calls out of their children's trio tables on the family page A person a family lists as a child, with a result to act on still unopened, can be a parent in a family with three generations. Their children's de novo and compound-het rows hold their calls. The family view now leaves out both tables of a child while the child or either parent is gated, in one rule in core.js, and the page shows one fixed line instead. Bead: live-family-hide-closed --- assets/live/core.js | 19 +++++++++++++++---- assets/live/family-page.js | 9 +++++++++ 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/assets/live/core.js b/assets/live/core.js index ed73d8a2d..a74043c2b 100644 --- a/assets/live/core.js +++ b/assets/live/core.js @@ -457,11 +457,22 @@ layout: raw.layout ? { width: raw.layout.width, height: raw.layout.height, size: raw.layout.size, lines: copy(raw.layout.lines), people: (raw.layout.people || []).map(function (m) { return { sample: m.sample, sex: m.sex, sequenced: m.sequenced, x: m.x, y: m.y }; }) } : null, couple_note: raw.couple_note, chrom_lengths: copy(raw.chrom_lengths), gated: closed, runs: {} }; + // A child's trio tables (the de novo candidates and the compound-het pairs) are read against the child and its + // parents: a row names the child, and the parent each call or variant comes from. Both tables of a child stay + // out while any of the three is gated, whole, so nothing the page says of them depends on which rows there are. + // The one rule for these rows: the family view and the genome map's family mode get them only through here. + function parentsOf(s) { return (raw.state.members || []).filter(function (m) { return m.sample === s; }).map(function (m) { return [m.father, m.mother]; })[0] || []; } + function trioShut(s) { return gated(s) || parentsOf(s).some(function (p) { return p && gated(p); }); } // Each person keeps the role this family gives them: its children are the ones it lists with a father or a // mother. A gated person it lists as a parent or partner (a child in another family) is in gated alone, and the - // view gives them their buttons beside the children. + // view gives them their buttons beside the children. An open child with a gated parent keeps its entry and + // names that parent in trio_closed: the page says its tables stay closed, the same line whatever rows there are. f.state.children = (raw.state.members || []).filter(function (m) { return m.father || m.mother; }).map(function (m) { - return gated(m.sample) ? { child: m.sample, gated: true } : copy((raw.state.children || []).filter(function (c) { return c.child === m.sample; })[0]); + if (gated(m.sample)) { return { child: m.sample, gated: true }; } + var c = copy((raw.state.children || []).filter(function (x) { return x.child === m.sample; })[0]); + var by = [m.father, m.mother].filter(function (p) { return p && gated(p); }); + if (c && by.length) { c.trio_closed = by; } + return c; }).filter(Boolean); // A row belongs to a gated child when it names that child as one of its members, by exact name. function names(v) { return String(v || '').split(/[\s,;]+/); } @@ -483,9 +494,9 @@ ['segments', 'trio_status', 'trio_files', 'trio_file_state'].forEach(function (key) { if (!raw[key] || typeof raw[key] !== 'object') { return; } f[key] = {}; - Object.keys(raw[key]).forEach(function (k) { if (!gated(k)) { f[key][k] = copy(raw[key][k]); } }); + Object.keys(raw[key]).forEach(function (k) { if (!(key === 'trio_status' ? trioShut(k) : gated(k))) { f[key][k] = copy(raw[key][k]); } }); }); - ['de_novo', 'compound_het'].forEach(function (key) { f[key] = copy((raw[key] || []).filter(function (r) { return !gated(r.child); })); }); + ['de_novo', 'compound_het'].forEach(function (key) { f[key] = copy((raw[key] || []).filter(function (r) { return !trioShut(r.child); })); }); Object.keys(raw.runs || {}).forEach(function (k) { f.runs[k] = gated(k) ? Object.assign(progressOf(raw.runs[k]), { url: raw.runs[k].url, gated: true }) : copy(raw.runs[k]); }); diff --git a/assets/live/family-page.js b/assets/live/family-page.js index b02455086..4781493a8 100644 --- a/assets/live/family-page.js +++ b/assets/live/family-page.js @@ -389,6 +389,15 @@ } else { add(d, el('p', { 'class': 'dim' }, 'No chromosome map yet: step 38 draws one for a child with a sequenced parent.')); } + // A parent whose report is closed (a child in another family) closes this child's trio tables too: they are + // read against that parent's genome. One fixed line, whatever rows there are; the tables come once it opens. + if (c.trio_closed) { + add(d, el('p', { 'class': 'dim', 'data-trio': 'closed' }, 'The de novo candidates and compound-het pairs stay closed while ' + + c.trio_closed.join(' and ') + (c.trio_closed.length > 1 ? '’s reports are closed: they are read against those genomes.' + : '’s report is closed: they are read against that genome.'))); + add(s, d); + return; + } var dn = (f.de_novo || []).filter(function (r) { return r.child === c.child; }); add(d, el('h3', {}, 'De novo candidates')); if (dn.length) { From a53f6dedf5367c28334965de81c10b4435814340 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:20:44 +0100 Subject: [PATCH 05/16] test(live): check the family page keeps a child's closed trio rows off, under node Runs core.js and family-page.js in a vm with a small DOM on an invented family, with controls that cut the trio rule out and must fail. Red on the previous main: the grandchild's rows reached the page. Bead: live-family-hide-closed --- tests/live/family_hide_closed.js | 203 ++++++++++++++++++++++++++ tests/test_live_family_hide_closed.sh | 14 ++ 2 files changed, 217 insertions(+) create mode 100644 tests/live/family_hide_closed.js create mode 100644 tests/test_live_family_hide_closed.sh diff --git a/tests/live/family_hide_closed.js b/tests/live/family_hide_closed.js new file mode 100644 index 000000000..70899e5db --- /dev/null +++ b/tests/live/family_hide_closed.js @@ -0,0 +1,203 @@ +// The family page keeps a child's closed de novo and compound-het rows off the page (assets/live/core.js +// projectFamily, assets/live/family-page.js). Node only: core.js and family-page.js run in a vm context that +// stands in for the browser window, with a small DOM, on an invented family (DEMO-FAMILY names, no real data). +// +// The family: DEMO-001 and DEMO-002, their son DEMO-003 (one result to act on, in KCNH2), his partner DEMO-004 +// and their daughter DEMO-005 (no result to act on). Checks: +// 1. While DEMO-003's result is closed, neither his trio rows nor DEMO-005's (read against his genome) reach +// the family view or the page, and the page is the same with or without those rows: no count, no heading. +// 2. Opened on the page with its button, both children's rows show. +// 3. With no result to act on, every row shows. +// Controls: the same checks on a core.js with the trio rule cut out must fail. +// +// Run: node tests/live/family_hide_closed.js [DIR], DIR holding assets/live (default: this repo). +'use strict'; +const fs = require('fs'); +const path = require('path'); +const vm = require('vm'); + +const REPO = path.resolve(process.argv[2] || path.join(__dirname, '..', '..')); +const CORE = fs.readFileSync(path.join(REPO, 'assets/live/core.js'), 'utf8'); +const PAGE = fs.readFileSync(path.join(REPO, 'assets/live/family-page.js'), 'utf8'); +let fails = 0; +function check(desc, ok, detail) { + console.log((ok ? 'PASS ' : 'FAIL ') + desc + (ok || !detail ? '' : ' -- ' + String(detail).slice(0, 600))); + if (!ok) { fails++; } +} + +// --- a small DOM: enough for el/add/svg, textContent, attributes, details and buttons ------------------------ +class Node { + constructor(tag) { + this.tag = tag; this.attrs = {}; this.kids = []; this.text = ''; this.listeners = {}; this.parent = null; + this.style = { setProperty() {} }; + this.classList = { add: (c) => { this.attrs.class = ((this.attrs.class || '') + ' ' + c).trim(); }, contains: () => false }; + } + setAttribute(k, v) { this.attrs[k] = String(v); } + getAttribute(k) { return k in this.attrs ? this.attrs[k] : null; } + removeAttribute(k) { delete this.attrs[k]; } + get attributes() { return Object.keys(this.attrs).map((name) => ({ name, value: this.attrs[name] })); } + appendChild(c) { if (c.parent) { c.remove(); } c.parent = this; this.kids.push(c); return c; } + remove() { if (this.parent) { this.parent.kids = this.parent.kids.filter((k) => k !== this); this.parent = null; } } + replaceWith(o) { const p = this.parent, i = p.kids.indexOf(this); o.parent = p; p.kids[i] = o; this.parent = null; } + addEventListener(k, f) { (this.listeners[k] = this.listeners[k] || []).push(f); } + click() { (this.listeners.click || []).forEach((f) => f({ preventDefault() {} })); } + get textContent() { return this.tag === '#text' ? this.text : this.kids.map((k) => k.textContent).join(''); } + set textContent(t) { this.kids = []; if (this.tag === '#text') { this.text = String(t); } else if (t !== '') { const n = new Node('#text'); n.text = String(t); this.appendChild(n); } } + all() { return [this].concat(...this.kids.map((k) => k.all())); } + // tag[attr] or [attr] or [attr=value]: what the views ask for. + querySelectorAll(sel) { + const m = /^([a-z0-9]*)(?:\[([a-z-]+)(?:=([^\]]+))?\])?$/.exec(sel); + if (!m) { return []; } + return this.all().slice(1).filter((n) => (!m[1] || n.tag === m[1]) && (!m[2] || (m[2] in n.attrs && (m[3] === undefined || n.attrs[m[2]] === m[3])))); + } + querySelector(sel) { return this.querySelectorAll(sel)[0] || null; } +} + +// --- one page load: the window, the server's answers, core.js then family-page.js -------------------------- +function load(core, files) { + const html = new Node('html'), body = new Node('body'), main = new Node('main'); + main.attrs.id = 'family'; + body.classList = { contains: (c) => c === 'page-family', add() {} }; + body.appendChild(main); html.appendChild(body); + const location = { href: 'http://demo.invalid/runs/DEMO-FAMILY/', pathname: '/runs/DEMO-FAMILY/', hash: '', search: '', replace() {} }; + const store = {}; + const win = { + document: { documentElement: html, body, readyState: 'complete', hidden: false, title: '', addEventListener() {}, + createElement: (t) => new Node(t), createElementNS: (ns, t) => new Node(t), + createTextNode: (t) => { const n = new Node('#text'); n.text = String(t); return n; }, + getElementById: (id) => html.all().filter((n) => n.attrs.id === id)[0] || null }, + location, name: '', scrollY: 0, scrollTo() {}, setInterval() { return 0; }, addEventListener() {}, + history: { replaceState() {} }, + sessionStorage: { getItem: (k) => store[k] || null, setItem: (k, v) => { store[k] = String(v); }, removeItem: (k) => { delete store[k]; } }, + EventSource: function () {}, URL, Response, TextEncoder, crypto: globalThis.crypto, setTimeout, clearTimeout, console, + fetch: (u) => { + const p = new URL(typeof u === 'string' ? u : u.url, location.href).pathname; + const body = files[p]; + return Promise.resolve(body === undefined ? new Response('not found', { status: 404 }) : new Response(JSON.stringify(body), { status: 200 })); + } + }; + win.window = win; win.self = win; + vm.createContext(win); + vm.runInContext(core, win, { filename: 'core.js' }); + vm.runInContext(PAGE, win, { filename: 'family-page.js' }); + return win; +} +function tick() { return new Promise((r) => setTimeout(r, 5)); } +async function settle(win, want) { + for (let i = 0; i < 400; i++) { + if (win.document.documentElement.attrs['data-pgp-status'] === 'ready' && (!want || want())) { return true; } + await tick(); + } + return false; +} +function pageOf(win) { + const main = win.document.getElementById('family'); + return { text: main.textContent, headings: main.querySelectorAll('h3').map((h) => h.textContent), main }; +} +// What the family view itself receives (core.js projects family.json for every reader of it). +async function viewData(win) { + const r = await win.fetch('/runs/DEMO-FAMILY/family.json'); + return r.json(); +} + +// --- the invented family --------------------------------------------------------------------------------- +const DN = (child, pos) => ({ child, chrom: 'chr3', pos: String(pos), ref: 'C', alt: 'T', child_gt: '0/1', father_gt: '0/0', mother_gt: '0/0', + child_dp: '34', child_gq: '52', child_ab: '0.47', quality: 'high' }); +const CH = (child, gene, a, b) => ({ child, gene, variant_a: a, variant_b: b, a_from: 'father', b_from: 'mother', phase: 'in trans' }); +function family(rows) { + const members = [ + { sample: 'DEMO-001', sex: 'male', father: '', mother: '', partner: 'DEMO-002', sequenced: true }, + { sample: 'DEMO-002', sex: 'female', father: '', mother: '', partner: 'DEMO-001', sequenced: true }, + { sample: 'DEMO-003', sex: 'male', father: 'DEMO-001', mother: 'DEMO-002', partner: 'DEMO-004', sequenced: true }, + { sample: 'DEMO-004', sex: 'female', father: '', mother: '', partner: 'DEMO-003', sequenced: true }, + { sample: 'DEMO-005', sex: 'female', father: 'DEMO-003', mother: 'DEMO-004', partner: '', sequenced: true }]; + const assessed = { de_novo: { assessed: true, why: 'both parents are sequenced' }, compound_het: { assessed: true, why: 'a parent is sequenced' } }; + const run = (s, child) => ({ url: '/runs/' + s + '/', state: 'finished', planned: 10, done: 10, attention: child ? null : 1, results: true, child }); + return { + state: { kind: 'family', name: 'DEMO-FAMILY', pedigree: null, members, + children: [{ child: 'DEMO-003', father: 'DEMO-001', mother: 'DEMO-002' }, { child: 'DEMO-005', father: 'DEMO-003', mother: 'DEMO-004' }], + couples: [], plan: [], reports: [] }, + layout: null, relationships: [], ibd_pairs: [], couples: {}, segments: {}, + de_novo: rows.de_novo, compound_het: rows.compound_het, + trio_files: { 'de_novo.tsv': true, 'compound_het.tsv': true }, trio_file_state: { 'de_novo.tsv': 'here', 'compound_het.tsv': 'here' }, + trio_status: { 'DEMO-003': assessed, 'DEMO-005': assessed }, side_by_side: null, + runs: { 'DEMO-001': run('DEMO-001', false), 'DEMO-002': run('DEMO-002', false), 'DEMO-003': run('DEMO-003', true), + 'DEMO-004': run('DEMO-004', false), 'DEMO-005': run('DEMO-005', true) } }; +} +const ROWS = { + de_novo: [DN('DEMO-003', 48211904), DN('DEMO-005', 51122233)], + compound_het: [CH('DEMO-003', 'KCNH2', 'chr7:150960000:C>T', 'chr7:150962400:G>A'), CH('DEMO-005', 'G0187', 'chr7:94371020:G>A', 'chr7:94398114:T>C')] +}; +// The values a row of each child would put on the page. +const SEEN = { 'DEMO-003': ['48211904', 'KCNH2', '150960000'], 'DEMO-005': ['51122233', 'G0187', '94371020'] }; +const ACT = { id: 'act-kcnh2', level: 'act', gene: 'KCNH2', title: 'A change in KCNH2', detail: 'chr7:150960000 C>T', text: 'Invented.', plan: ['Invented.'] }; +function files(rows, act) { + const out = { '/runs/DEMO-FAMILY/family.json': family(rows) }; + out['/runs/DEMO-003/results.json'] = { findings: (act ? [ACT] : []).concat([{ id: 'look-1', level: 'review', gene: 'G0001', title: 'Invented' }]) }; + out['/runs/DEMO-005/results.json'] = { findings: [{ id: 'look-2', level: 'review', gene: 'G0002', title: 'Invented' }] }; + return out; +} +const NONE = { de_novo: [], compound_het: [] }; +function rowsOf(d, child) { return (d.de_novo || []).concat(d.compound_het || []).filter((r) => r.child === child).length; } +function shows(text, child) { return SEEN[child].filter((v) => text.indexOf(v) >= 0); } + +// --- the checks, on one core.js ------------------------------------------------------------------------------ +async function closedChecks(core, label) { + const ok = []; + const win = load(core, files(ROWS, true)); + if (!(await settle(win))) { return [false]; } + const p = pageOf(win), d = await viewData(win); + ok.push(['closed: no row of DEMO-003 or DEMO-005 reaches the family view (' + label + ')', + rowsOf(d, 'DEMO-003') === 0 && rowsOf(d, 'DEMO-005') === 0, JSON.stringify([d.de_novo, d.compound_het])]); + ok.push(['closed: no value of their rows is on the page (' + label + ')', + !shows(p.text, 'DEMO-003').length && !shows(p.text, 'DEMO-005').length, shows(p.text, 'DEMO-003').concat(shows(p.text, 'DEMO-005'))]); + ok.push(['closed: no trio heading, count or row button (' + label + ')', + !p.headings.some((h) => /De novo|Compound-het/.test(h)) && !/Show all|more rows/.test(p.text), p.headings]); + // The page with the rows and the page without any row are the same: nothing on it says a row is there. + const empty = load(core, files(NONE, true)); + await settle(empty); + ok.push(['closed: the page is the same with or without their rows (' + label + ')', pageOf(empty).text === p.text, + pageOf(empty).text.length + ' vs ' + p.text.length]); + ok.push(['closed: DEMO-005 keeps her entry and one fixed closed line (' + label + ')', + p.main.querySelectorAll('[data-trio=closed]').length === 1 && /DEMO-003’s report is closed/.test(p.text), p.text.slice(-400)]); + return ok; +} +async function openedChecks(core) { + const ok = []; + const win = load(core, files(ROWS, true)); + await settle(win); + const buttons = pageOf(win).main.querySelectorAll('[data-opening]'); + ok.push(['opened: DEMO-003 has one button', buttons.length === 1, buttons.length]); + buttons.forEach((b) => b.click()); + const done = await settle(win, () => shows(pageOf(win).text, 'DEMO-003').length === 3); + const p = pageOf(win); + ok.push(['opened: both children\'s rows show once his result is opened', done && shows(p.text, 'DEMO-005').length === 3, + shows(p.text, 'DEMO-003').concat(shows(p.text, 'DEMO-005'))]); + ok.push(['opened: the closed line is gone', !p.main.querySelectorAll('[data-trio=closed]').length]); + return ok; +} +async function noActChecks(core) { + const win = load(core, files(ROWS, false)); + await settle(win); + const p = pageOf(win); + return [['no result to act on: every row shows', shows(p.text, 'DEMO-003').length === 3 && shows(p.text, 'DEMO-005').length === 3 && + p.headings.filter((h) => /De novo|Compound-het/.test(h)).length === 4, p.headings]]; +} + +(async () => { + for (const c of await closedChecks(CORE, 'core.js')) { check(c[0], c[1], c[2]); } + for (const c of await openedChecks(CORE)) { check(c[0], c[1], c[2]); } + for (const c of await noActChecks(CORE)) { check(c[0], c[1], c[2]); } + // Controls: the trio rule cut out of core.js (rows kept whoever is gated) must fail the closed checks. + const RULE = 'function trioShut(s) { return gated(s) || parentsOf(s).some(function (p) { return p && gated(p); }); }'; + check('control: core.js holds the trio rule the control cuts out', CORE.indexOf(RULE) >= 0); + if (CORE.indexOf(RULE) >= 0) { + for (const [name, cut] of [['rows of a gated parent kept', 'function trioShut(s) { return gated(s); }'], + ['rows of any child kept', 'function trioShut(s) { return false; }']]) { + const res = await closedChecks(CORE.replace(RULE, cut), 'control'); + check('control: ' + name + ' fails the closed checks', res.some((c) => !c[1])); + } + } + console.log(fails ? fails + ' check(s) failed' : 'All checks passed.'); + process.exit(fails ? 1 : 0); +})().catch((e) => { console.log('FAIL ' + (e && e.stack || e)); process.exit(1); }); diff --git a/tests/test_live_family_hide_closed.sh b/tests/test_live_family_hide_closed.sh new file mode 100644 index 000000000..62826188f --- /dev/null +++ b/tests/test_live_family_hide_closed.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# The family page keeps a child's closed de novo and compound-het rows off the +# page, the rows of a grandchild read against that child's genome included: +# tests/live/family_hide_closed.js runs assets/live/core.js and +# family-page.js under node on an invented family, with its own controls (the +# trio rule cut out of core.js must fail). No node is a failure, never a skip. +set -euo pipefail + +REPO=$(cd "$(dirname "$0")/.." && pwd) +if ! command -v node >/dev/null 2>&1; then + echo "FAIL: node is not on PATH; this test needs node 20 or newer" + exit 1 +fi +exec node "${REPO}/tests/live/family_hide_closed.js" "$REPO" From 658daf5448a61f719499068a7addd6eae9981e65 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:20:45 +0100 Subject: [PATCH 06/16] docs(live): say when a child's trio tables stay closed on the family page Bead: live-family-hide-closed --- CHANGELOG.md | 1 + docs/live-site.md | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05c3388e1..15b610613 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- The live family page keeps a child's closed results out of the de novo and compound-het tables of that child's own children. A person whom one family lists as a child, with a result to act on still unopened, can be a parent in a family with three generations. Their children's trio rows hold their calls: the parent each variant came from, and their genotype at each de novo candidate. The page left out the gated person's own rows but drew those of their children, so the grandparents' family page showed them. Now both tables of a child stay out of the family view while the child or either parent is gated, and chapter IV says so in one fixed line that does not depend on which rows there are. The tables come back once that parent's results are opened on the page. `family.json` does not change; only what the page draws from it. A node test (`tests/test_live_family_hide_closed.sh`) runs `core.js` and `family-page.js` on an invented family, with controls that cut the rule out. - The self-test of `scripts/ci/check-images.sh` copies the whole tree again once the list of tracked paths passes 128 KiB. It piped that list through `xargs` to `tar`, and GNU `xargs` split a list that long over two `tar` runs; the reading `tar` stopped after the first archive, so Guard's `images` job failed on a broken pipe (exit 123). One `tar` now reads the list itself (`tar --null -T -`). Nothing the check reports changes. - The phased family VCF is indexed by the Rust `og` binary. WhatsHap's step (step 38 and the Nextflow WHATSHAP_PED process) wrote `family.phased.vcf.gz.tbi` with a Python one-liner, `pysam.tabix_index`. It now runs `og vcf index -f`. The `.tbi` keeps its name and format, and its bytes, once inflated, are the ones htslib 1.24 writes (the htslib of the pysam and bcftools the pipeline pins): the same bins in the same order, the same chunks and the same linear index. Only the deflate stream can differ, as two deflate libraries differ. On a 5-million-record three-sample VCF it took 0.75 to 0.91 s and 31 MB at peak, where htslib's own `tabix` took 0.81 to 0.93 s and 19 MB. - As in htslib 1.24, a record ends at the largest of INFO/END, the REF length, the SVLEN of a ``, ``, `` or `` ALT, and a sample's LEN on a gVCF block. A line with no position to read, unsorted records, a sequence that starts again, a record past 2^29 and an empty file stop the index with a message, and nothing is written. Without `-f` it never writes over an index, even one that appears while it runs. diff --git a/docs/live-site.md b/docs/live-site.md index d49ac0161..e757307d1 100644 --- a/docs/live-site.md +++ b/docs/live-site.md @@ -82,6 +82,8 @@ The home page decides from `/runs.json` alone. A gated child is a child (the rul On the family page, chapter IV shows a gated child's buttons only, and each child opens on its own. A gated person whom this family lists as a parent or partner, because another family lists them as a child, gets their buttons under the members, after the member list. The family view gets the family without that child's report data: no de novo or compound-het rows, stretches, trio status or trio files of the child, no relationship or shared-segment row with the child, and no couple screen of a couple the child is part of (the couple keeps its names and whether it was screened; a screened couple's line says its screen stays closed). The side by side keeps a row only when an open member has a value in it, in a fixed order, and the child's cell reads "closed". Plan rows that name the child are left out. The parents' other plan rows and the couple reports of couples with no gated partner stay. Other reports of the family folder are left out while a child is gated, because they can hold the child's data. A member with no run here has no report, so the gate does not close anything of theirs. The pedigree and the members' progress stay. +The de novo candidates and the compound-het pairs of a child are read against the child and both parents: a row names the child and the parent each call or variant comes from. So both tables of a child stay out of the family view while the child or either parent is gated. That covers a family with three generations: while a person's report is closed, their children's tables in this family are closed too, because those rows hold that person's calls. Such a child keeps their entry and their stretches, and chapter IV shows one fixed line in place of the two tables: "The de novo candidates and compound-het pairs stay closed while 's report is closed: they are read against that genome." The line is the same whatever rows there are, so it gives no count and no heading of rows. Once every result to act on of that parent is opened on the page, the tables come back, as they do for a child whose own results are opened. A child with no result to act on, and no gated parent, shows every row. + The gate holds the page's `fetch` and `EventSource`. A view or plugin reads a run's files through them. `XMLHttpRequest`, an ``, an `