Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- Every `og` command answers `--help` and `-h` with its usage on stdout and exit 0. Before, 71 of the 158 commands `og --help` lists did not: 70 exited 2, among them `og clinvar hits`, `og pgx cpic-report` and `og bam contigs`, which read the word as a file name, and `og dev fixture comphet --help` took it as its output folder. `og clinvar --help` now also gives its defaults: `--window 10`, `--min-gq 30` and `--min-dp 10`. `og --help` now starts with how to run a genome (`./scripts/run-all.sh`, `./scripts/validate-setup.sh`, `og run` to inspect a run's progress, `og recheck`, `og lookup`, the reports), then lists the pipeline helpers; the developer and CI tools moved to `og dev --help`. `og run state` now says `Run state written: FILE` on stderr, not stdout, so its stdout stays empty when it writes a file; a script that read that line should read stderr or the file. `docs/rust.md` writes down the rules for help, output and exit codes. No option, output file or exit code of a run changes.
- A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again.
- VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37).
- The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2).
Expand Down
15 changes: 14 additions & 1 deletion docs/rust.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,27 @@ tests/rust/ the differential test against CPython, the image test

Each ported tool becomes a subcommand named after its file in `bin/` without `.py`, in kebab-case. Today `og` has `og qc sex-loss` (step 33), `og version`, `og --tree-hash`, `og map`, `og setup mask-fasta`, `og run state|tasks|eta|headroom|delta`, `og dev parity-compare`, `og pgx cyp2d6-bed|cyp2d6-measure|cyp2d6-check`, `og str motif|catalog`, `og cdna check|scan` (were `clip_check.py` and `cdna_scan.py`), `og comphet phase` (was `comphet_phase.py`), `og bam contigs`, `og setup unzip|genetic-map` and two hidden ones the tests drive: `og pycompat-probe` and `og bam-fetch-probe`. The filters of steps 04, 05 and 07 are grouped by step: `og sv svlen-fix` and `og sv inv-filter` (were `sv_svlen_fix.py` and `manta_inv_filter.py`), `og pgx refblock-fill` and `og pgx mito-merge` (were `pgx_refblock_fill.py` and `pgx_mito_merge.py`). Step 05 builds AnnotSV's input with `og sv add-callers` and checks its class 4 and 5 calls with `og sv postfilter` (were `cnvpytor_to_vcf.py` and `annotsv_postfilter.py`); both apply one read-depth rule for long Delly calls (`og-steps::sv`). Step 39's couple screen applies the same rule. `og str catalog` (was `strchive_convert.py`) builds the STRchive catalog of steps 09 and 9b; `og setup unzip` and `og setup genetic-map` replace two small Python programs inside `setup.sh`. Step 25 has `og prs format` (a port of `collect_summary.py prs-format`, named by what it works on) and `og prs table` (was `prs-table`). Step 33 has `og qc sample` (was `sample-qc`), and step 20 has `og mito annotate|deletions|copies` (were `mito-annotate`, `mito-deletions` and `mito-copies`). Steps 23 and 31 use `og vcf csq-rank` and `og table constraint-join` (ports of the awk programs `bin/csq_rank.awk` and `bin/constraint_join.awk`, named by what they work on). Step 27 uses `og pgx cpic-report` and `og pgx step36-outside` (were the two commands of `pgx_parse.py`). Step 49 runs `og chip prepare`, `og chip report` and `og chip check-table` (were the commands of `chip_report.py`). Steps 32, 35, 41 and 46 run `og cn bed`, `og cn call`, `og cn intervals`, `og cn check-bam`, `og cn marker-regions` and `og cn marker-read` ([Copy number from depth](cn-loci.md)); they replace `hba_copy_number.py`, `lpa_kiv2.py`, `copy_number.py` (the catalog tests materialise its pinned historical version as a differential oracle) and the marker reader of `traits_report.py`. Step 41 runs `og bloodtype report` (was `bloodtype_report.py`), which reads the marker sites through the VCF's index. Step 26b's six commands are `og ancestry-fine panel-map|check-map|order|sample-bed|chroms|report`. Step 43 runs `og archaic table|strict|genotypes|report` (were the four commands of `archaic_report.py`), and the archaic jobs of `public-evidence.yml` run `og archaic compare|genomes` (were the two commands of `archaic_compare.py`), and step 44 runs `og ancient select|lift|person-geno|align|report` (were the five commands of `ancient_report.py`); `og ancient lift` reads UCSC's chain file with `og-intervals::chain`. Step 40 runs `og consumer build-sites`, `og consumer export` and `og consumer check` (were the three commands of `export_consumer.py`). Step 38 and the Nextflow family workflow run `og family plan`, `prepare`, `check`, `report`, `segments` and `ibd` (were `family_report.py`, `family_segments.py` and `ibd_report.py`), and index the phased family VCF with `og vcf index` (was a `pysam.tabix_index` one-liner). `og family report` reads the joint VCF once, compresses each child's inheritance table on its own thread, and stops with exit 2 on a joint VCF that is not sorted. Step 39 and the Nextflow `COUPLE_CARRIER` process run `og couple screen`, and `og couple genes` rebuilds the carrier list (were the two commands of `couple_report.py`); the screen computes every couple before it writes a file. The opt-in steps read their tool's output with `og y str-bed|str-report` (45), `og viral report` (47), `og exomiser prepare|report` (48), `og mei report|annotsv-input` (50), `og mocha report` (51), `og vntyper report` (52) and `og locityper report` (53); they replaced `ystr_report.py`, `viral_report.py`, `exomiser_report.py`, `mei_report.py`, `mocha_report.py`, `vntyper_report.py` and `locityper_report.py`. Step 37 places the sample on the current YFull YTree with `og y tree-positions` and `og y tree-place` (were the two commands of `y_tree_place.py`), lists its private SNPs with `og y private-snps` (was `y_private_snps.py`) and writes Yleaf's marker positions with `og y yleaf-positions` (was `yleaf_run.py positions`; `yleaf_run.py predict` stays Python, as glue inside Yleaf's own code). Steps 08, 32 and 36 run `og pgx hla-concordance`, `og pgx pypgx-summary` and `og pgx outside-calls` (were the three commands of `pgx_outside_calls.py`), and step 35b runs `og paraphase check-bam|table` (were `paraphase_report.py`). Both write their two output files together or not at all. `scripts/owngenome-lookup.sh` runs `og lookup query` ([genotype lookup](lookup.md)), which answers every query of the command line in one process; `og lookup resolve` and `og lookup report` are its two halves, with the command lines of `genome_lookup.py`, which keeps only the query reading and the names rule that `results_index.py` imports. `og recheck` replaced `owngenome_recheck.py` ([Newer databases since a run](recheck.md)); the file stays one release as a shim that runs it, for crontabs. `og dev freshness` replaced `scripts/ci/freshness.py`, the monthly freshness issue of `freshness.yml`. `og dev build cn-intervals|callability|sv-rule-regions` rebuild the committed tables `bin/cn_loci.tsv` (its interval rows), `assets/callability_genes.tsv` and `assets/sv_rule_regions.tsv`; they replaced `scripts/ci/build-cn-intervals.py` and the Python inside `build-callability.sh` and `build-sv-rule-regions.sh`. Those scripts keep the downloads (`curl` through `scripts/ci/lib/fetch-pinned.sh`, each file checked against its sha256) and hand `og` the folder, so `og dev build` reads local files only. Both ask the network through the host's `curl` (the config on stdin, so no token reaches the process list), which keeps a TLS stack out of the static binary and its cross-builds.

Every verb the 1.0.0 release still has to port is already registered, each in its own module, as a stub that prints `og <verb>: not ported yet` on stderr and exits 2: `og report index|charts`, `og live serve|results|family|demo`, `og view serve`, `og vcf shift-back`, `og setup chrm-shift|revel|roh-af`, `og sv survivor-prep|consensus-check`, `og ancestry-fine freqs|project|paint`, `og longread clair3-model`, `og map svg`, `og dev demo-sample|demo-events|python-gate|everything-check|docs-check|steps-check` and `og dev build roh-af|acmg-lof`. A port fills its module's body and does not edit `og/src/main.rs`, the workspace member list or a crate's module list. `rust/og/tests/port_scaffold.rs` lists these verbs and fails when one has no line in `og --help` or reaches a router's "unknown" error. `og report render` is part way: it writes the frame of both reports and the quality-control and variant sections, leaves the other sections out, and no step calls it yet; `bin/render_report.py` still writes the reports.
Every verb the 1.0.0 release still has to port is already registered, each in its own module, as a stub that prints `og <verb>: not ported yet` on stderr and exits 2: `og report index|charts`, `og live serve|results|family|demo`, `og view serve`, `og vcf shift-back`, `og setup chrm-shift|revel|roh-af`, `og sv survivor-prep|consensus-check`, `og ancestry-fine freqs|project|paint`, `og longread clair3-model`, `og map svg`, `og dev demo-sample|demo-events|python-gate|everything-check|docs-check|steps-check` and `og dev build roh-af|acmg-lof`. A port fills its module's body and does not edit `og/src/main.rs`, the workspace member list or a crate's module list. `rust/og/tests/port_scaffold.rs` lists these verbs and fails when one has no line in `og --help` (`og dev --help` for an `og dev` verb) or reaches a router's "unknown" error. `og report render` is part way: it writes the frame of both reports and the quality-control and variant sections, leaves the other sections out, and no step calls it yet; `bin/render_report.py` still writes the reports.

`og ancestry-fine simulate` builds an admixed genome with known truth from held-out people of the fine-ancestry panel: an unphased VCF and a BED of each copy's ancestry. The engines that replace ADMIXTURE and RFMix test against it. It and the parts those engines share (the panel reader, the genetic map, `model/ref.geno` and a seeded xoshiro256** generator) live in `og-admix`.

A Python file whose tool moved to `og` but whose names other Python still imports stays as a *remnant*: a `# remnant:` header names its importers and the lane that deletes it, and `tests/parity/remnants.tsv` lists the names it may hold and the files that import it. `og dev remnants` (Guard, job `rust`) fails when the file defines a name the list does not allow, when an allowed name has no user left, when a listed importer no longer imports it, or when an unlisted file starts to. So a remnant only shrinks. `bin/family_report.py` is the first one: the pedigree and kinship rules that `pedigree_svg.py` imports. `bin/genome_lookup.py` keeps the query reading and the summary names rule that `results_index.py` imports.

The rules: no `unsafe` (the workspace forbids it), `cargo clippy -D warnings`, `cargo fmt`, and few, well-known crates (`indexmap`, `flate2`, `zlib-rs`, `memchr`, `smallvec`, `rustc-hash` and `rust-htslib` so far, declared in `rust/Cargo.toml` and locked in `Cargo.lock`). The musl builds also link `mimalloc` as their allocator: musl's own lets a long multi-threaded read hold several times its live memory.

## Command lines

Every `og` command keeps the help rules, and a new verb keeps all of them. `rust/og/tests/rv_og_cli_conventions.rs` checks the help rules, and that `og run state --out FILE` prints nothing on stdout.

- **Help.** `og <tool> [verb] --help` and `-h` print the command's usage on stdout and exit 0, without reading an input or writing a file. A help word after `--` is an argument, not a help request. A command whose own parser has a fuller usage is listed in `OWN_HELP` in `og/src/main.rs`; `og` answers any other command from its line of `og --help` or `og dev --help`. So a new verb answers `--help` as soon as it has its line there; under a tool-wide `OWN_HELP` entry (such as `clinvar` or `run`), the tool's own parser answers it instead.
- **`og --help`** starts with how to run a genome (`./scripts/run-all.sh`, `./scripts/validate-setup.sh`, `og run`, `og recheck`, `og lookup`, the reports), then lists the pipeline helpers the steps run. It lists no `og dev` line: `og dev --help` lists the developer and CI tools. `og run` inspects a run; it never starts one.
- **Output.** A result goes to stdout, or to the file `--out FILE` names. Messages, warnings and errors go to stderr only: `og run state` says `Run state written: FILE` on stderr. A verb that writes a fixed set of files, such as a step helper, takes their names as options.
- **Exit codes.** 0: done. 1: a file could not be read or written. 2: a refused command line, with the usage and the error on stderr. 3: a partial result or nothing found, such as `og lookup` for a query it cannot find or `og map` for a run with no map.

Some older verbs keep an older form for now. For example, `og run state` writes `<sample>/run_state.json` when no `--out` is given ([Where a run stands](run-state.md)); `og report render` prints its count of sections by state on stdout; `og dev freshness --out FILE` writes the report to the file and also prints it.

`og summary collect` exits 0 whenever it writes `summary.json`, whatever its sections say: a caller reads each section's `state` in `summary.json`, never the exit code.

## Building

```bash
Expand Down
33 changes: 30 additions & 3 deletions rust/crates/og-clinvar/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@
//! og clinvar deep-intronic [--no-gvcf] [--min-gq N] [--min-dp N] SITES VCF_CALLS GVCF_CALLS
//! ```
//!
//! A missing file, or `-`, is standard input; plain, gzip and BGZF input are
//! read alike. A refused command line is exit 2, a file that cannot be read
//! or written exit 1.
//! `join-adjacent` and `hits` read standard input when the VCF is missing, and
//! any VCF given as `-` is standard input; plain, gzip and BGZF input are read
//! alike. `-h` or `--help` before any `--` prints the usage and the defaults on
//! stdout, exit 0. A refused command line is exit 2, a file that cannot be
//! read or written exit 1.

use std::fmt::Display;
use std::io::{self, BufRead, BufWriter, Write};
Expand All @@ -22,6 +24,22 @@ const USAGE: &str = "usage: og clinvar join-adjacent --mode prepare|finish|tag|b
og clinvar hits [HITS_VCF]
og clinvar deep-intronic [--no-gvcf] [--min-gq N] [--min-dp N] SITES VCF_CALLS GVCF_CALLS";

/// The usage, then each option's meaning and default: what `--help` prints.
fn help() -> String {
format!(
"{USAGE}

--mode M join-adjacent: prepare, finish, tag or build
--fasta REF join-adjacent --mode build: the reference FASTA (indexed)
--window N join-adjacent: join two records when the second starts at most N bases after the first ends (default {WINDOW})
--min-gq N deep-intronic: the call floor, the lowest GQ of a plain reference call, never below 20 (default 30)
--min-dp N deep-intronic: the depth floor, the lowest DP of a plain reference call (default 10)
--no-gvcf deep-intronic: there is no gVCF, so a site with no record is a no call
join-adjacent and hits read standard input when the VCF is missing; any VCF given as - is standard input.
"
)
}

fn refuse(msg: &str) -> u8 {
eprintln!("{USAGE}");
eprintln!("og clinvar: error: {msg}");
Expand Down Expand Up @@ -108,6 +126,15 @@ fn input(path: Option<&str>) -> io::Result<Box<dyn BufRead>> {

/// Runs `og clinvar <verb> ...` and returns the exit code.
pub fn main(args: &[String]) -> u8 {
// A help word after `--` is an argument, as in og's own help rule.
if args
.iter()
.take_while(|a| *a != "--")
.any(|a| a == "-h" || a == "--help")
{
print!("{}", help());
return 0;
}
let Some(verb) = args.first() else {
return refuse("the following arguments are required: verb");
};
Expand Down
22 changes: 16 additions & 6 deletions rust/crates/og-fresh/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,12 @@ pub const USAGE: &str =
"usage: og dev freshness [--out FILE] [--sections a,b] [--versions F] [--setup F]
og dev freshness --update-issue BODY_FILE [--repo OWNER/NAME]
og dev freshness --self-test
sections: databases, clinvar (the ClinVar row alone), images, digests
sections: databases, clinvar (the ClinVar row alone), images, digests (default: all)
--out FILE also write the report to FILE; it is always printed on stdout
--versions F the pins to check (default: versions.env of the checkout above og)
--setup F the setup script with the download URLs (default: scripts/setup.sh)
--update-issue B put the report file B in the freshness issue of --repo
(default: $GITHUB_REPOSITORY)
";

fn usage_error(err: &mut dyn Write, msg: &str) -> u8 {
Expand Down Expand Up @@ -84,6 +89,15 @@ pub fn main(args: &[String]) -> u8 {
}

pub fn run(args: &[String], out: &mut dyn Write, err: &mut dyn Write) -> u8 {
// A help word after `--` is an argument, as in og's own help rule.
if args
.iter()
.take_while(|a| *a != "--")
.any(|a| a == "-h" || a == "--help")
{
let _ = write!(out, "{USAGE}");
return 0;
}
let p = match parse(
args,
&[
Expand All @@ -94,15 +108,11 @@ pub fn run(args: &[String], out: &mut dyn Write, err: &mut dyn Write) -> u8 {
"update-issue",
"repo",
],
&["self-test", "help"],
&["self-test"],
) {
Ok(p) => p,
Err(e) => return usage_error(err, &e),
};
if flag(&p, "help") {
let _ = write!(out, "{USAGE}");
return 0;
}
if flag(&p, "self-test") {
return crate::selftest::run(out);
}
Expand Down
Loading
Loading