Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
a8f3b4e
feat(report): write the ancestry, haplogroup and PRS sections in Rust
GeiserX Nov 17, 2026
835905f
feat(report): write the traits, copy-number and repeat sections in Rust
GeiserX Nov 17, 2026
c401572
test(parity): compare the five new report sections with the Python, e…
GeiserX Nov 17, 2026
4506b43
feat(report): write the clinical sections of both reports in Rust
GeiserX Nov 17, 2026
d7b484d
fix(summary): refuse a sample folder og summary collect cannot list
GeiserX Nov 17, 2026
a786d08
docs(changelog): og summary collect refuses a sample folder it cannot…
GeiserX Nov 17, 2026
997d5e7
feat(report): render the PGx, HLA and mitochondrial sections with og
GeiserX Nov 17, 2026
5597569
test(report): move the HLA and mitochondrial report checks to cargo
GeiserX Nov 17, 2026
af9ab5c
test(report): move the clinical Python tests into cargo tests
GeiserX Nov 17, 2026
e7e69a3
fix(report): a PRS row with no counts reads "not available"
GeiserX Nov 17, 2026
09ba83b
test(report): move the ancestry chapter's tests from Python to cargo
GeiserX Nov 17, 2026
4bb952f
docs(changelog): the ancestry chapter and the PRS card in Rust
GeiserX Nov 17, 2026
918c829
test(cn): move the copy-number and Lp(a) checks from Python to cargo …
GeiserX Nov 17, 2026
ba24672
test(traits): move the traits checks to cargo tests, with the label c…
GeiserX Nov 17, 2026
8ea5dfb
test(str): move the repeat range checks from Python to cargo tests
GeiserX Nov 17, 2026
78a4e8e
docs(changelog): note the five report sections and the moved tests
GeiserX Nov 17, 2026
91947ae
test: collect a real folder where a test relied on a missing one
GeiserX Nov 17, 2026
8d14772
test(parity): keep the invented repeat counts off the personal-data p…
GeiserX Nov 17, 2026
165aef7
fix(report): say Line::Html is the one raw-markup line, and make the …
GeiserX Nov 17, 2026
fdd105a
test(report): check the HLA candidate cap agrees with og summary collect
GeiserX Nov 17, 2026
9a34ae0
test(report): remove the ancestry tests' scratch folders when each te…
GeiserX Nov 17, 2026
37e85d2
Merge remote-tracking branch 'origin/main' into pgp/port-report-ancestry
GeiserX Nov 18, 2026
cacaa6e
Merge remote-tracking branch 'origin/main' into pgp/port-report-genome
GeiserX Nov 18, 2026
7a7a172
Merge remote-tracking branch 'origin/main' into pgp/rv-summary-collec…
GeiserX Nov 18, 2026
3ba99c3
test(render): check the whole text chapter list, not only the first
GeiserX Nov 18, 2026
db55bfe
Merge remote-tracking branch 'origin/main' into pgp/port-report-pgx
GeiserX Nov 18, 2026
fda40e9
fix(report): show an explore address that is not a web address as tex…
GeiserX Nov 18, 2026
e25e624
test(parity): render interval changes and both SMN1 verdicts in the g…
GeiserX Nov 18, 2026
c17b6fb
fix(report): keep the legend CSS, explore links and test folders safe
GeiserX Nov 18, 2026
7c244d1
Merge origin/main into the report convergence tip
GeiserX Nov 18, 2026
03902fd
Merge #487 pgp/port-report-clinical into the report convergence tip
GeiserX Nov 18, 2026
d1902c2
Merge #488 pgp/port-report-ancestry into the report convergence tip
GeiserX Nov 18, 2026
dc1c844
Merge #490 pgp/port-report-genome into the report convergence tip
GeiserX Nov 18, 2026
0137a0d
refactor(report): build the clinical tables with the shared Table model
GeiserX Nov 18, 2026
bbc5eee
refactor(report): build the ancestry and genome tables with the Table…
GeiserX Nov 18, 2026
217a7e6
test(parity): show the newly ported sections in the viral not-assesse…
GeiserX Nov 18, 2026
2918247
test(report): check the AnnotSV tables and every card's Viewer sentence
GeiserX Nov 18, 2026
3a9993c
Merge #480 rv-summary-collect-missing-dir into the report convergence…
GeiserX Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 18 additions & 37 deletions .github/workflows/guard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,24 @@ jobs:
grep -E "^test ${test} .*FAILED" "$log" || { cat "$log"; echo "ERROR: ${fault} did not fail ${test}"; exit 1; }
echo "Control: ${fault} failed ${test}, as it must."
done
# The reference-block labels of the multi-SNP rules are written in the
# test, not read from the rules table: with WRONG_REFERENCE planted as
# the label of APOL1 G0/G0 and CAD9p21 nonrisk/nonrisk, exactly those
# two tests must fail.
- name: Control (a changed reference-block label fails its independent expectation)
run: |
log="${RUNNER_TEMP}/wrong-reference.log"
rc=0
cargo test --locked --no-fail-fast -p og-steps --features fault-wrong-reference --test genome_traits \
> "$log" 2>&1 || rc=$?
cat "$log"
[ "$rc" -ne 0 ] || { echo "ERROR: the planted WRONG_REFERENCE labels passed"; exit 1; }
for test in reference_blocks_apol1 reference_blocks_cad9p21; do
grep -E "^test ${test} \.\.\. FAILED$" "$log" || { echo "ERROR: ${test} did not catch WRONG_REFERENCE"; exit 1; }
done
grep -E 'test result: FAILED\. [0-9]+ passed; 2 failed;' "$log" \
|| { echo "ERROR: a test other than the two planted ones failed"; exit 1; }
echo "Control: WRONG_REFERENCE failed reference_blocks_apol1 and reference_blocks_cad9p21."
- name: Traits controls (diplotype order, REF check and missing phase rows)
env:
CARGO_BUILD_JOBS: 3
Expand Down Expand Up @@ -755,43 +773,6 @@ jobs:
echo "${#tests[@]} tests carry the og-leg marker"
og_leg "${tests[@]}"

- name: Traits independent expectations reject changed reference labels and marker metadata
working-directory: .
env:
OG: ${{ github.workspace }}/bin/og
OG_REQUIRED: 1
run: |
python3 - <<'PY'
import contextlib, io, pathlib, runpy, tempfile
test = runpy.run_path("tests/test_traits_report.py")["main"].__globals__
original = pathlib.Path(test["RULES"]).read_text()
with tempfile.TemporaryDirectory() as work:
for rid, key, label in [("APOL1", "G0/G0", "G0/G0"), ("CAD9p21", "nonrisk/nonrisk", "no risk copy")]:
path = pathlib.Path(work) / "rules.tsv"
old = f"combination\t{rid}\t{key}\t{label}\t"
assert old in original
path.write_text(original.replace(old, f"combination\t{rid}\t{key}\tWRONG_REFERENCE\t"))
test["RULES"], test["FAILS"] = str(path), 0
log = io.StringIO()
with contextlib.redirect_stdout(log):
rc = test["main"]()
print(log.getvalue())
assert rc == 1 and test["FAILS"] == 1
assert f"[FAIL] {rid}, reference blocks only: {label}" in log.getvalue()
print(f"Control: {rid} WRONG_REFERENCE failed its independent expectation")
marker = test["MARKERS"]["SMN1_27134"]
rs, chrom, pos, ref, alt = marker["dbsnp"]
marker["dbsnp"] = rs, chrom, pos + 1, ref, alt
test["FAILS"] = 0
log = io.StringIO()
with contextlib.redirect_stdout(log):
test["marker_checks"]()
print(log.getvalue())
assert test["FAILS"] == 1
assert "[FAIL] marker SMN1_27134: the pinned dbSNP record agrees" in log.getvalue()
print("Control: retained SMN1 off-copy metadata failed its independent expectation")
PY

# The fake-docker job builds no og, so its cn-engine-calls case says it
# is skipped there; here bin/og is built and the case must run.
- name: The fake-docker case cn-engine-calls (steps 35, 41 and 46 through og cn) with og
Expand Down
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,19 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- `og report render` now writes the pharmacogenomics, HLA and mitochondrial sections of both reports: PharmCAT, the CPIC drug-gene recommendations, pypgx with the PGx copy number, CYP2D6 across the callers with step 36's verdict, HLA typing with its derived lines (coeliac DQ types, the DQB1*02 dose, the drug and disease alleles) and the KIR genes, and the mitochondrial variants with the confirmed list, the large deletions and the mtDNA copy number. No step calls the verb yet, so nothing a run writes changes. On 110 summaries (the 99 report-render parity cases and 11 new changes of one summary that reach each state of these sections) its output is byte for byte the Python's for those parts. The mitochondrial card's chart is left out until the charts are ported. The checks of `tests/test_hla_report.py` and `tests/test_mito_extended.py` are cargo tests in `rust/crates/og-render/tests/port_report_pgx.rs`, and both Python files are gone. What og does on purpose that the Python did not:
- A summary without the CYP2D6 block (`"cyp2d6"` missing or `null`) shows no CYP2D6 across the callers. The Python stopped with a KeyError or a TypeError and wrote no report.
- `og report render` writes most of the ancestry chapter and the polygenic score card in Rust: fine-grained ancestry with the painting and its legend, the mitochondrial haplogroup, the Y chromosome (Yleaf, the placement on the current public tree, the Y-STR markers in FamilyTreeDNA's order), archaic and ancient ancestry, and the PRS table with its ancestry and sex notes. Runs of homozygosity are not written yet, so both Rust reports leave them out even when the summary has them. Both reports read as `bin/render_report.py` writes them, except:
- A PRS row whose score, matched or total the summary has as null (a short row of pgsc_calc's table) reads "not available" in both reports, and in the HTML result cell when there is no score either. The Python wrote "None" in the text report and stopped the whole HTML report on that row.
- The painting and the ancient PCA plot are escaped in their `src` attribute. A real picture is base64 and does not change; a value that is not text renders instead of stopping the report.
- A haplogroup's explore address that is not a web address with a host (`http://` or `https://`), such as a `javascript:` one, shows as text in the HTML report. The Python made it a link, which would run it when clicked. Only the two docs pages the collector links get the plain "opens a website" note; any other address gets the note that it carries the haplogroup.
- A painting legend colour that is not `#rrggbb` shows as a fixed grey, so a summary cannot add CSS to the legend. The collector always writes `#rrggbb`.
- The charts are not part of these cards yet; `og report charts` will write them.
The tests of these sections moved from Python to cargo (`rust/crates/og-render/tests/ancestry_*.rs`): `tests/test_ancient_report.py`, `test_archaic_report.py`, `test_haplogroup_explore.py` and `test_ystr_report.py` are gone, and `test_ancestry_fine_report.py` no longer renders the reports. The IBDmix runs of the archaic test need docker and amd64; they run in CI.
- `og report render` now writes five more sections of both reports: the traits panel with its genotype rules, the blood groups, Lp(a), the paralog genes (Parascopy and step 35's depth counts for HBA, CYP21A2, the dosage loci and the copy-number intervals) and the repeat expansions. No step calls the verb yet, so nothing a run writes changes. On 106 summaries (the 99 report-render parity cases before, and 7 new ones that drive each branch of these sections in each state) its output is byte for byte the Python's for the sections it writes.
- The tests of these sections and of the tools behind them are now cargo tests: `tests/test_traits_report.py` in og-steps (`tests/genome_traits.rs`), `tests/test_copy_number.py` and `tests/test_lpa_kiv2.py` in og-cn (`tests/genome_cn.rs`, `genome_lpa.rs`, `genome_markers.rs`), `tests/test_str_ranges.py` in og-setup (`tests/genome_str_ranges.rs`), and the checks that cross the steps, the summary and the report in `rust/og/tests/genome_steps.rs` and `genome_traits_report.rs`. The four Python files are gone, and `tests/test_collect_summary.py` and `tests/test_render_report.py` no longer import `tests/test_traits_report.py`.
- The guard's check that a changed reference-block label in `assets/genotype_rules.tsv` fails its expectation is now a cargo feature, `og-steps/fault-wrong-reference`. The guard runs the tests with it and needs exactly the two planted blocks to fail.
- `og summary collect` refuses a sample folder it cannot list. A mistyped `--sample-dir`, a file, or a folder without read permission read as a folder where no step ran, so the collect wrote a summary.json with every section missing and exited 0, over the good summary when `--out` named it. It now exits 1 with `ERROR: <folder>: not a readable folder: <reason>` and writes nothing. An empty folder that exists is still collected, and a step folder inside it that cannot be read still counts as empty. No output file or format changes.
- A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again.
- VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37).
- The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2).
Expand Down Expand Up @@ -37,6 +50,9 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n
- Each report is written under a temporary name of its own and renamed into place.
- An `-o` name must end in `.txt` or `.html`, or the verb exits 2 and writes nothing. The Python wrote the HTML report for any name that did not end in `.txt`.
- A summary with no section for a step reads that section as not run. The Python stopped with a KeyError when the `variants` section was absent.
- `og report render` writes the clinical chapters of both reports in Rust, one module per section under `rust/crates/og-render/src/sections/`: gene callability, the ClinVar screen (the hits, the call floor, the frequency marks, the cDNA and construct marks, the known deep intronic sites and the cDNA scan), CPSR, the ACMG secondary findings and the clinical filter counts, the carrier list with its two-hit lines, slivar's compound-het phase and the couple section. The bytes are the Python's on every parity case. AnnotSV's rows are in `sections/annotsv.rs`, but the structural variants section that shows them is not switched on yet: half of it is the SV caller support, which another change ports.
- A number in a clinical table cell is escaped in the HTML. The Python put it into the page as it was, so a text value there could break the page.
- `tests/test_acmg_sf_rules.py`, `tests/test_cross_engine_biallelic.py` and `tests/test_callability.py` are gone. Their checks are cargo tests: `rust/crates/og-render/tests/clinical.rs`, with the couple screen checks in `rust/crates/og-couple/tests/port_report_clinical.rs` and the ALT contig refusal in `rust/crates/og-cn/tests/port_report_clinical.rs`. `bin/collect_summary.py` loses `parse_alleles`, `allele_key` and `SEVERITY_KINDS`, which only the ACMG test used.
- Step 04c (Sniffles2) now refuses a short-read BAM. Sniffles2 finds no SV in short reads and still exits 0, so the step reported success with an empty VCF. It now reads the mean length of the first 10,000 primary reads with the pinned samtools image and stops with exit 1 below 1,000 bp, naming the long-read BAM to use. Part of #411.
- `setup.sh` installs GIAB's GRCh38 tandem-repeat stratification (v3.6, `TANDEM_REPEATS_URL` and `TANDEM_REPEATS_SHA256`) and keeps it unpacked as `reference/GRCh38_AllTandemRepeats.bed`. Step 04c passes it to Sniffles2 as `--tandem-repeats`, which improves breakpoints and merging inside repeats, and says so when it is missing. On a public HG002 HiFi slice of chr20 it found one more benchmark SV (50 of 64 instead of 49) at the same precision. Sniffles' own GRCh38 file comes from PacBio's pbsv under a licence limited to PacBio data, so we use GIAB's.
- The step's closing hint no longer says duphold adds a GQ to long-read SVs. Sniffles2 writes its own QUAL and GQ, and step 15 reads Manta's calls only. The long-read guide's duphold row and its uncited accuracy lines are corrected.
Expand Down
28 changes: 0 additions & 28 deletions bin/collect_summary.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,9 +147,6 @@ def collect(sample, sample_dir, declared_sex=None, out=None):
"42", "43", "44", "45", "46", "47", "48", "49", "50", "51", "52", "53"}


SEVERITY_KINDS = ("hypomorph", "pseudodeficiency")


def open_text(path):
if path.endswith((".gz", ".bgz")):
return gzip.open(path, "rt", errors="replace")
Expand Down Expand Up @@ -214,28 +211,3 @@ def get(v, k):
freq = "none" if not fields else "unknown" if not afs else "rare" if max(afs) < COMMON_AF else "common"
return cls, [{"impact": get(v, "IMPACT"), "gene": get(v, "SYMBOL"),
"consequence": get(v, "Consequence").replace("&", ",")} for v in chosen], freq


def allele_key(locus):
"""'chr6:26092913:G:A' -> ('chr6', 26092913, 'G', 'A')."""
c, pos, ref, alt = locus.split(":")
return (c, int(pos), ref, alt)


def parse_alleles(text, kinds=False):
"""The variants or severity cell of assets/acmg_sf.tsv and
assets/carrier_genes.tsv: '.' or entries separated by ';', each
'[kind ]chrom:pos:ref:alt name'. {key: {'kind', 'name'}}."""
out = {}
for e in (text or ".").split(";"):
e = e.strip()
if e in ("", "."):
continue
parts = e.split(" ", 2 if kinds else 1)
if kinds and len(parts) < 2:
raise ValueError(f"severity entry needs a kind and a chrom:pos:ref:alt: {e}")
kind = parts.pop(0) if kinds else None
if kinds and kind not in SEVERITY_KINDS:
raise ValueError(f"severity '{kind}' is not one of {', '.join(SEVERITY_KINDS)}: {e}")
out[allele_key(parts[0])] = {"kind": kind, "name": parts[1] if len(parts) > 1 else parts[0]}
return out
2 changes: 1 addition & 1 deletion bin/hla_alleles.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# allele: an allele group at a field boundary: B*27 matches B*27:05:02, B*57:01 matches B*57:01:01.
# kind: drug (a drug reaction) or disease (a condition). association: what the study in pmids reports.
# pmids: the primary papers, comma-separated PubMed ids. citation: first author, year, journal.
# tests/test_hla_report.py checks the schema.
# rust/crates/og-render/tests/port_report_pgx.rs checks the schema.
allele kind association pmids citation
A*31:01 drug carbamazepine hypersensitivity reactions in Europeans 21428769 McCormack 2011, N Engl J Med
B*15:02 drug carbamazepine Stevens-Johnson syndrome in Han Chinese 15057820 Chung 2004, Nature
Expand Down
2 changes: 1 addition & 1 deletion docs/08-hla-typing.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ docker run --rm \

## What the Report Derives

The report (step 24 and the text report) lists each T1K row and then derives three things from it. The code is the HLA section of `og summary collect` (`rust/crates/og-summary/src/collect/group_c/hla.rs`); `tests/test_hla_report.py` checks every rule on synthetic T1K rows. The genotypes are unphased: T1K gives two alleles per gene, not which chromosome each sits on.
The report (step 24 and the text report) lists each T1K row and then derives three things from it. The code is the HLA section of `og summary collect` (`rust/crates/og-summary/src/collect/group_c/hla.rs`); `rust/crates/og-render/tests/port_report_pgx.rs` checks every rule on synthetic T1K rows. The genotypes are unphased: T1K gives two alleles per gene, not which chromosome each sits on.

**Coeliac DQ.** From the DQA1 and DQB1 rows:

Expand Down
4 changes: 4 additions & 0 deletions rust/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading