diff --git a/CHANGELOG.md b/CHANGELOG.md index 4236a45d2..48b3da3ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- Every `og` command answers `--help` and `-h` with its usage on stdout and exit 0. Before, 71 of the 158 commands `og --help` lists did not: 70 exited 2, among them `og clinvar hits`, `og pgx cpic-report` and `og bam contigs`, which read the word as a file name, and `og dev fixture comphet --help` took it as its output folder. `og clinvar --help` now also gives its defaults: `--window 10`, `--min-gq 30` and `--min-dp 10`. `og --help` now starts with how to run a genome (`./scripts/run-all.sh`, `./scripts/validate-setup.sh`, `og run` to inspect a run's progress, `og recheck`, `og lookup`, the reports), then lists the pipeline helpers; the developer and CI tools moved to `og dev --help`. `og run state` now says `Run state written: FILE` on stderr, not stdout, so its stdout stays empty when it writes a file; a script that read that line should read stderr or the file. `docs/rust.md` writes down the rules for help, output and exit codes. No option, output file or exit code of a run changes. - A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again. - VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37). - The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2). diff --git a/docs/rust.md b/docs/rust.md index 8505e841e..ca23deb4e 100644 --- a/docs/rust.md +++ b/docs/rust.md @@ -42,7 +42,7 @@ tests/rust/ the differential test against CPython, the image test Each ported tool becomes a subcommand named after its file in `bin/` without `.py`, in kebab-case. Today `og` has `og qc sex-loss` (step 33), `og version`, `og --tree-hash`, `og map`, `og setup mask-fasta`, `og run state|tasks|eta|headroom|delta`, `og dev parity-compare`, `og pgx cyp2d6-bed|cyp2d6-measure|cyp2d6-check`, `og str motif|catalog`, `og cdna check|scan` (were `clip_check.py` and `cdna_scan.py`), `og comphet phase` (was `comphet_phase.py`), `og bam contigs`, `og setup unzip|genetic-map` and two hidden ones the tests drive: `og pycompat-probe` and `og bam-fetch-probe`. The filters of steps 04, 05 and 07 are grouped by step: `og sv svlen-fix` and `og sv inv-filter` (were `sv_svlen_fix.py` and `manta_inv_filter.py`), `og pgx refblock-fill` and `og pgx mito-merge` (were `pgx_refblock_fill.py` and `pgx_mito_merge.py`). Step 05 builds AnnotSV's input with `og sv add-callers` and checks its class 4 and 5 calls with `og sv postfilter` (were `cnvpytor_to_vcf.py` and `annotsv_postfilter.py`); both apply one read-depth rule for long Delly calls (`og-steps::sv`). Step 39's couple screen applies the same rule. `og str catalog` (was `strchive_convert.py`) builds the STRchive catalog of steps 09 and 9b; `og setup unzip` and `og setup genetic-map` replace two small Python programs inside `setup.sh`. Step 25 has `og prs format` (a port of `collect_summary.py prs-format`, named by what it works on) and `og prs table` (was `prs-table`). Step 33 has `og qc sample` (was `sample-qc`), and step 20 has `og mito annotate|deletions|copies` (were `mito-annotate`, `mito-deletions` and `mito-copies`). Steps 23 and 31 use `og vcf csq-rank` and `og table constraint-join` (ports of the awk programs `bin/csq_rank.awk` and `bin/constraint_join.awk`, named by what they work on). Step 27 uses `og pgx cpic-report` and `og pgx step36-outside` (were the two commands of `pgx_parse.py`). Step 49 runs `og chip prepare`, `og chip report` and `og chip check-table` (were the commands of `chip_report.py`). Steps 32, 35, 41 and 46 run `og cn bed`, `og cn call`, `og cn intervals`, `og cn check-bam`, `og cn marker-regions` and `og cn marker-read` ([Copy number from depth](cn-loci.md)); they replace `hba_copy_number.py`, `lpa_kiv2.py`, `copy_number.py` (the catalog tests materialise its pinned historical version as a differential oracle) and the marker reader of `traits_report.py`. Step 41 runs `og bloodtype report` (was `bloodtype_report.py`), which reads the marker sites through the VCF's index. Step 26b's six commands are `og ancestry-fine panel-map|check-map|order|sample-bed|chroms|report`. Step 43 runs `og archaic table|strict|genotypes|report` (were the four commands of `archaic_report.py`), and the archaic jobs of `public-evidence.yml` run `og archaic compare|genomes` (were the two commands of `archaic_compare.py`), and step 44 runs `og ancient select|lift|person-geno|align|report` (were the five commands of `ancient_report.py`); `og ancient lift` reads UCSC's chain file with `og-intervals::chain`. Step 40 runs `og consumer build-sites`, `og consumer export` and `og consumer check` (were the three commands of `export_consumer.py`). Step 38 and the Nextflow family workflow run `og family plan`, `prepare`, `check`, `report`, `segments` and `ibd` (were `family_report.py`, `family_segments.py` and `ibd_report.py`), and index the phased family VCF with `og vcf index` (was a `pysam.tabix_index` one-liner). `og family report` reads the joint VCF once, compresses each child's inheritance table on its own thread, and stops with exit 2 on a joint VCF that is not sorted. Step 39 and the Nextflow `COUPLE_CARRIER` process run `og couple screen`, and `og couple genes` rebuilds the carrier list (were the two commands of `couple_report.py`); the screen computes every couple before it writes a file. The opt-in steps read their tool's output with `og y str-bed|str-report` (45), `og viral report` (47), `og exomiser prepare|report` (48), `og mei report|annotsv-input` (50), `og mocha report` (51), `og vntyper report` (52) and `og locityper report` (53); they replaced `ystr_report.py`, `viral_report.py`, `exomiser_report.py`, `mei_report.py`, `mocha_report.py`, `vntyper_report.py` and `locityper_report.py`. Step 37 places the sample on the current YFull YTree with `og y tree-positions` and `og y tree-place` (were the two commands of `y_tree_place.py`), lists its private SNPs with `og y private-snps` (was `y_private_snps.py`) and writes Yleaf's marker positions with `og y yleaf-positions` (was `yleaf_run.py positions`; `yleaf_run.py predict` stays Python, as glue inside Yleaf's own code). Steps 08, 32 and 36 run `og pgx hla-concordance`, `og pgx pypgx-summary` and `og pgx outside-calls` (were the three commands of `pgx_outside_calls.py`), and step 35b runs `og paraphase check-bam|table` (were `paraphase_report.py`). Both write their two output files together or not at all. `scripts/owngenome-lookup.sh` runs `og lookup query` ([genotype lookup](lookup.md)), which answers every query of the command line in one process; `og lookup resolve` and `og lookup report` are its two halves, with the command lines of `genome_lookup.py`, which keeps only the query reading and the names rule that `results_index.py` imports. `og recheck` replaced `owngenome_recheck.py` ([Newer databases since a run](recheck.md)); the file stays one release as a shim that runs it, for crontabs. `og dev freshness` replaced `scripts/ci/freshness.py`, the monthly freshness issue of `freshness.yml`. `og dev build cn-intervals|callability|sv-rule-regions` rebuild the committed tables `bin/cn_loci.tsv` (its interval rows), `assets/callability_genes.tsv` and `assets/sv_rule_regions.tsv`; they replaced `scripts/ci/build-cn-intervals.py` and the Python inside `build-callability.sh` and `build-sv-rule-regions.sh`. Those scripts keep the downloads (`curl` through `scripts/ci/lib/fetch-pinned.sh`, each file checked against its sha256) and hand `og` the folder, so `og dev build` reads local files only. Both ask the network through the host's `curl` (the config on stdin, so no token reaches the process list), which keeps a TLS stack out of the static binary and its cross-builds. -Every verb the 1.0.0 release still has to port is already registered, each in its own module, as a stub that prints `og : not ported yet` on stderr and exits 2: `og report index|charts`, `og live serve|results|family|demo`, `og view serve`, `og vcf shift-back`, `og setup chrm-shift|revel|roh-af`, `og sv survivor-prep|consensus-check`, `og ancestry-fine freqs|project|paint`, `og longread clair3-model`, `og map svg`, `og dev demo-sample|demo-events|python-gate|everything-check|docs-check|steps-check` and `og dev build roh-af|acmg-lof`. A port fills its module's body and does not edit `og/src/main.rs`, the workspace member list or a crate's module list. `rust/og/tests/port_scaffold.rs` lists these verbs and fails when one has no line in `og --help` or reaches a router's "unknown" error. `og report render` is part way: it writes the frame of both reports and the quality-control and variant sections, leaves the other sections out, and no step calls it yet; `bin/render_report.py` still writes the reports. +Every verb the 1.0.0 release still has to port is already registered, each in its own module, as a stub that prints `og : not ported yet` on stderr and exits 2: `og report index|charts`, `og live serve|results|family|demo`, `og view serve`, `og vcf shift-back`, `og setup chrm-shift|revel|roh-af`, `og sv survivor-prep|consensus-check`, `og ancestry-fine freqs|project|paint`, `og longread clair3-model`, `og map svg`, `og dev demo-sample|demo-events|python-gate|everything-check|docs-check|steps-check` and `og dev build roh-af|acmg-lof`. A port fills its module's body and does not edit `og/src/main.rs`, the workspace member list or a crate's module list. `rust/og/tests/port_scaffold.rs` lists these verbs and fails when one has no line in `og --help` (`og dev --help` for an `og dev` verb) or reaches a router's "unknown" error. `og report render` is part way: it writes the frame of both reports and the quality-control and variant sections, leaves the other sections out, and no step calls it yet; `bin/render_report.py` still writes the reports. `og ancestry-fine simulate` builds an admixed genome with known truth from held-out people of the fine-ancestry panel: an unphased VCF and a BED of each copy's ancestry. The engines that replace ADMIXTURE and RFMix test against it. It and the parts those engines share (the panel reader, the genetic map, `model/ref.geno` and a seeded xoshiro256** generator) live in `og-admix`. @@ -50,6 +50,19 @@ A Python file whose tool moved to `og` but whose names other Python still import The rules: no `unsafe` (the workspace forbids it), `cargo clippy -D warnings`, `cargo fmt`, and few, well-known crates (`indexmap`, `flate2`, `zlib-rs`, `memchr`, `smallvec`, `rustc-hash` and `rust-htslib` so far, declared in `rust/Cargo.toml` and locked in `Cargo.lock`). The musl builds also link `mimalloc` as their allocator: musl's own lets a long multi-threaded read hold several times its live memory. +## Command lines + +Every `og` command keeps the help rules, and a new verb keeps all of them. `rust/og/tests/rv_og_cli_conventions.rs` checks the help rules, and that `og run state --out FILE` prints nothing on stdout. + +- **Help.** `og [verb] --help` and `-h` print the command's usage on stdout and exit 0, without reading an input or writing a file. A help word after `--` is an argument, not a help request. A command whose own parser has a fuller usage is listed in `OWN_HELP` in `og/src/main.rs`; `og` answers any other command from its line of `og --help` or `og dev --help`. So a new verb answers `--help` as soon as it has its line there; under a tool-wide `OWN_HELP` entry (such as `clinvar` or `run`), the tool's own parser answers it instead. +- **`og --help`** starts with how to run a genome (`./scripts/run-all.sh`, `./scripts/validate-setup.sh`, `og run`, `og recheck`, `og lookup`, the reports), then lists the pipeline helpers the steps run. It lists no `og dev` line: `og dev --help` lists the developer and CI tools. `og run` inspects a run; it never starts one. +- **Output.** A result goes to stdout, or to the file `--out FILE` names. Messages, warnings and errors go to stderr only: `og run state` says `Run state written: FILE` on stderr. A verb that writes a fixed set of files, such as a step helper, takes their names as options. +- **Exit codes.** 0: done. 1: a file could not be read or written. 2: a refused command line, with the usage and the error on stderr. 3: a partial result or nothing found, such as `og lookup` for a query it cannot find or `og map` for a run with no map. + +Some older verbs keep an older form for now. For example, `og run state` writes `/run_state.json` when no `--out` is given ([Where a run stands](run-state.md)); `og report render` prints its count of sections by state on stdout; `og dev freshness --out FILE` writes the report to the file and also prints it. + +`og summary collect` exits 0 whenever it writes `summary.json`, whatever its sections say: a caller reads each section's `state` in `summary.json`, never the exit code. + ## Building ```bash diff --git a/rust/crates/og-clinvar/src/cli.rs b/rust/crates/og-clinvar/src/cli.rs index 1dd988943..7132e92ae 100644 --- a/rust/crates/og-clinvar/src/cli.rs +++ b/rust/crates/og-clinvar/src/cli.rs @@ -6,9 +6,11 @@ //! og clinvar deep-intronic [--no-gvcf] [--min-gq N] [--min-dp N] SITES VCF_CALLS GVCF_CALLS //! ``` //! -//! A missing file, or `-`, is standard input; plain, gzip and BGZF input are -//! read alike. A refused command line is exit 2, a file that cannot be read -//! or written exit 1. +//! `join-adjacent` and `hits` read standard input when the VCF is missing, and +//! any VCF given as `-` is standard input; plain, gzip and BGZF input are read +//! alike. `-h` or `--help` before any `--` prints the usage and the defaults on +//! stdout, exit 0. A refused command line is exit 2, a file that cannot be +//! read or written exit 1. use std::fmt::Display; use std::io::{self, BufRead, BufWriter, Write}; @@ -22,6 +24,22 @@ const USAGE: &str = "usage: og clinvar join-adjacent --mode prepare|finish|tag|b og clinvar hits [HITS_VCF] og clinvar deep-intronic [--no-gvcf] [--min-gq N] [--min-dp N] SITES VCF_CALLS GVCF_CALLS"; +/// The usage, then each option's meaning and default: what `--help` prints. +fn help() -> String { + format!( + "{USAGE} + + --mode M join-adjacent: prepare, finish, tag or build + --fasta REF join-adjacent --mode build: the reference FASTA (indexed) + --window N join-adjacent: join two records when the second starts at most N bases after the first ends (default {WINDOW}) + --min-gq N deep-intronic: the call floor, the lowest GQ of a plain reference call, never below 20 (default 30) + --min-dp N deep-intronic: the depth floor, the lowest DP of a plain reference call (default 10) + --no-gvcf deep-intronic: there is no gVCF, so a site with no record is a no call + join-adjacent and hits read standard input when the VCF is missing; any VCF given as - is standard input. +" + ) +} + fn refuse(msg: &str) -> u8 { eprintln!("{USAGE}"); eprintln!("og clinvar: error: {msg}"); @@ -108,6 +126,15 @@ fn input(path: Option<&str>) -> io::Result> { /// Runs `og clinvar ...` and returns the exit code. pub fn main(args: &[String]) -> u8 { + // A help word after `--` is an argument, as in og's own help rule. + if args + .iter() + .take_while(|a| *a != "--") + .any(|a| a == "-h" || a == "--help") + { + print!("{}", help()); + return 0; + } let Some(verb) = args.first() else { return refuse("the following arguments are required: verb"); }; diff --git a/rust/crates/og-fresh/src/cli.rs b/rust/crates/og-fresh/src/cli.rs index 94c481b0a..9c9492661 100644 --- a/rust/crates/og-fresh/src/cli.rs +++ b/rust/crates/og-fresh/src/cli.rs @@ -12,7 +12,12 @@ pub const USAGE: &str = "usage: og dev freshness [--out FILE] [--sections a,b] [--versions F] [--setup F] og dev freshness --update-issue BODY_FILE [--repo OWNER/NAME] og dev freshness --self-test - sections: databases, clinvar (the ClinVar row alone), images, digests + sections: databases, clinvar (the ClinVar row alone), images, digests (default: all) + --out FILE also write the report to FILE; it is always printed on stdout + --versions F the pins to check (default: versions.env of the checkout above og) + --setup F the setup script with the download URLs (default: scripts/setup.sh) + --update-issue B put the report file B in the freshness issue of --repo + (default: $GITHUB_REPOSITORY) "; fn usage_error(err: &mut dyn Write, msg: &str) -> u8 { @@ -84,6 +89,15 @@ pub fn main(args: &[String]) -> u8 { } pub fn run(args: &[String], out: &mut dyn Write, err: &mut dyn Write) -> u8 { + // A help word after `--` is an argument, as in og's own help rule. + if args + .iter() + .take_while(|a| *a != "--") + .any(|a| a == "-h" || a == "--help") + { + let _ = write!(out, "{USAGE}"); + return 0; + } let p = match parse( args, &[ @@ -94,15 +108,11 @@ pub fn run(args: &[String], out: &mut dyn Write, err: &mut dyn Write) -> u8 { "update-issue", "repo", ], - &["self-test", "help"], + &["self-test"], ) { Ok(p) => p, Err(e) => return usage_error(err, &e), }; - if flag(&p, "help") { - let _ = write!(out, "{USAGE}"); - return 0; - } if flag(&p, "self-test") { return crate::selftest::run(out); } diff --git a/rust/crates/og-pgx/src/cli.rs b/rust/crates/og-pgx/src/cli.rs index 580901eaa..fa82d7a0b 100644 --- a/rust/crates/og-pgx/src/cli.rs +++ b/rust/crates/og-pgx/src/cli.rs @@ -29,6 +29,16 @@ const USAGE: &str = "usage: og pgx cpic-report --sample S --report R.json --outd og pgx step36-outside --report R.json "; +/// What `--help` adds to the usage: each verb in one line. +const VERBS: &str = " + cpic-report writes DIR/S_phenotypes.tsv and DIR/S_cpic_recommendations.txt from + PharmCAT's report, and with --pypgx and --comparison the PharmCAT/pypgx + comparison; --medications, --consensus and --contraindications add sections + step36-outside prints the step 36 genes (CYP2D6, HLA-A, HLA-B) the report calls from an + outside call, one per line + An optional option given as an empty string counts as not given. +"; + /// The options of `cpic-report`. An option given as an empty string counts /// as not given, as the Python's truth tests read it. #[derive(Debug, Clone, Default, PartialEq, Eq)] @@ -82,6 +92,15 @@ fn options(args: &[String], names: &[&str]) -> Result>, Strin /// `og pgx [args]`: the exit code. pub fn main(args: &[String]) -> u8 { + // A help word after `--` is an argument, as in og's own help rule. + if args + .iter() + .take_while(|a| *a != "--") + .any(|a| a == "-h" || a == "--help") + { + print!("{USAGE}{VERBS}"); + return 0; + } match args.first().map(String::as_str) { Some("cpic-report") => { const NAMES: [&str; 11] = [ diff --git a/rust/crates/og-runlog/src/cli.rs b/rust/crates/og-runlog/src/cli.rs index 375eb8b38..d7cbabd1b 100644 --- a/rust/crates/og-runlog/src/cli.rs +++ b/rust/crates/og-runlog/src/cli.rs @@ -257,7 +257,7 @@ pub fn run(args: &[String], env: &Env, out: &mut dyn Write, err: &mut dyn Write) Err(e) => return usage_error(err, spec.verb, &e), }; match spec.verb { - "state" => state(&p, env, out, err), + "state" => state(&p, env, err), "tasks" => tasks(&p, out, err), "eta" => eta(&p, env, out, err), "headroom" => headroom(&p, env, out, err), @@ -265,7 +265,7 @@ pub fn run(args: &[String], env: &Env, out: &mut dyn Write, err: &mut dyn Write) } } -fn state(p: &Parsed, env: &Env, out: &mut dyn Write, err: &mut dyn Write) -> u8 { +fn state(p: &Parsed, env: &Env, err: &mut dyn Write) -> u8 { let steps = match steps_of(p) { Ok(s) => s, Err(e) => return fail(err, e), @@ -292,7 +292,7 @@ fn state(p: &Parsed, env: &Env, out: &mut dyn Write, err: &mut dyn Write) -> u8 if let Err(e) = write_atomic(&target, &json_text(&st.to_json())) { return fail(err, format!("cannot write {}: {e}", target.display())); } - let _ = writeln!(out, "Run state written: {}", target.display()); + let _ = writeln!(err, "Run state written: {}", target.display()); 0 } diff --git a/rust/crates/og-runlog/tests/parity.rs b/rust/crates/og-runlog/tests/parity.rs index 7984736ee..b52d937ee 100644 --- a/rust/crates/og-runlog/tests/parity.rs +++ b/rust/crates/og-runlog/tests/parity.rs @@ -8,6 +8,8 @@ //! Outputs must agree byte for byte, except `generated_utc` (the clock), the //! disk numbers of `headroom` (free space moves between two calls) and the //! host memory line where Python can read the RAM and og cannot (not Linux). +//! `og run state` says its one sentence on stderr where the Python said it on +//! stdout. //! //! The Python is `OG_PYTHON`, else `python3`; it must be 3.9 or newer. A //! missing Python, or an oracle that cannot be materialised, fails the test: @@ -273,10 +275,14 @@ fn state_parity(sample: &str, dir: &Path, extra: &[&str]) { oa.extend(["--out", &oo]); let (p, o) = (py("run_state", &pa), og(&oa)); assert_eq!((p.code, o.code), (0, 0), "{}{}", p.stderr, o.stderr); + // og writes the file and says so on stderr; the Python said it on + // stdout. Same sentence, and nothing on og's stdout (docs/rust.md, + // "Command lines"). + assert_eq!(o.stdout, "", "og run state --out printed on stdout"); same( - "stdout", + "Run state written", &p.stdout.replace(&po, "OUT"), - &o.stdout.replace(&oo, "OUT"), + &o.stderr.replace(&oo, "OUT"), ); same( "run_state.json", diff --git a/rust/og/src/main.rs b/rust/og/src/main.rs index 2ff4c9bef..7509553f0 100644 --- a/rust/og/src/main.rs +++ b/rust/og/src/main.rs @@ -23,118 +23,245 @@ fn tree_hash() -> &'static str { &TREE_MARK["og-tree-hash:".len()..TREE_MARK.len() - 1] } -const USAGE: &str = "usage: og [args] - og map [args] - og map svg [args] - og run state|tasks|eta|headroom|delta [args] - og recheck --sample S [--genome-dir DIR] [--offline FILE] [--rerun [--refresh-clinvar]] | --affected KEYS | --launch-dir DIR - og setup mask-fasta SRC BED DEST - og setup unzip ZIP DEST - og setup genetic-map --zip ZIP --out DIR [--rfmix] CHROM... - og setup chrm-shift [args] - og setup revel [args] - og setup roh-af [args] - og vcf csq-rank [options] - og vcf index [args] - og vcf shift-back [args] - og table constraint-join --gene-col NAME [--constrained] CONSTRAINT TABLE - og sv svlen-fix|inv-filter [args] - og sv add-callers [args] - og sv postfilter [args] - og sv survivor-prep [args] - og sv consensus-check [args] - og pgx cpic-report|step36-outside [args] - og pgx refblock-fill|mito-merge|cyp2d6-bed|cyp2d6-measure|cyp2d6-check [args] - og pgx outside-calls|hla-concordance|pypgx-summary [args] - og paraphase check-bam|table [args] - og clinvar join-adjacent|hits|deep-intronic [args] - og str motif|catalog [args] - og cdna check [args] - og cdna scan [args] - og comphet phase [args] - og bam contigs BAM - og qc sex-loss --table TABLE [--regions REGIONS] [--vcf VCF] - og qc sample [args] - og prs format [args] - og prs table [args] - og mito annotate|deletions|copies [args] - og summary collect [--only {a,b,c}] [args] - og report render [args] - og report index [args] - og report charts [args] - og live serve [args] - og live results [args] - og live family [args] - og live demo [args] - og view serve [args] - og couple screen [args] - og couple genes [args] - og roh summary --sample SAMPLE --roh ROH [--out OUT] - og chip prepare|report|check-table [args] - og cn bed|call|intervals|check-bam|marker-regions|marker-read [args] - og traits table [args] - og traits phase-sites [args] - og traits check [args] - og bloodtype report|check-markers|systems [args] - og family plan|prepare|check|report|segments [args] - og family ibd [args] - og consumer build-sites|export|check [args] - og ancestry-fine panel-map|check-map|order|sample-bed|chroms|report [args] - og ancestry-fine freqs [args] - og ancestry-fine project [args] - og ancestry-fine paint [args] - og ancestry-fine simulate [args] - og archaic table|strict|genotypes|report [args] - og archaic compare [args] - og archaic genomes [args] - og ancient select|lift|person-geno|align|report [args] - og vntyper report [args] - og locityper report [args] - og mocha report [args] - og mei report [args] - og mei annotsv-input [args] - og viral report [args] - og y str-bed [args] - og y str-report [args] - og y tree-positions --tree-dir DIR OUT - og y tree-place [args] - og y private-snps [args] - og y yleaf-positions --data DATA OUT - og exomiser prepare [args] - og exomiser report [args] - og longread clair3-model [args] - og lookup query|resolve|report [args] - og dev parity-compare|contract|schema|report-only|remnants [args] - og dev scan personal-data|images [args] - og dev versions check-run|expand|coupled [args] - og dev pgs trait|label-match|metric-match [args] - og dev fixture comphet OUT [--swap] - og dev fixture trim-pharmcat REPORT OUT [MAX] - og dev evidence clashes RAW.vcf.gz SPLIT.vcf.gz DATA_DIR OUT.tsv - og dev evidence hg002-gvcf TRUTH.vcf.gz TRUTH.bed REF.fasta SITES.tsv OUT.vcf - og dev renovate-check LOG - og dev pharmcat diplotypes REPORT [--called|--called-lines] - og dev nf-trace requests|attempts TRACE [options] - og dev smoke mini-ref|long-reads|eh-catalog|csv-cell|check [args] - og dev helpers [args] - og dev build cn-intervals|callability|sv-rule-regions [args] - og dev build roh-af [args] - og dev build acmg-lof [args] - og dev demo-sample [args] - og dev demo-events [args] - og dev params-doc [args] - og dev python-gate [args] - og dev everything-check [args] - og dev licences [args] - og dev docs-check [args] - og dev stub-http [args] - og dev steps-check [args] - og dev traits-sources [args] - og dev freshness [--out FILE] [--sections a,b] | --update-issue BODY --repo O/N | --self-test - og version - og --tree-hash +const USAGE: &str = "usage: og [verb] [args] + og [verb] --help one command's usage, on stdout + +Run a genome (docs/getting-started.md): + GENOME_DIR=/data ./scripts/run-all.sh SAMPLE male|female [nextflow options] + start a run; run it again to resume, it reuses every finished step + ./scripts/validate-setup.sh [SAMPLE] + check the setup before a run + og run state|tasks|eta|headroom|delta [args] + inspect a run's progress and time left; og run never starts a run + og recheck --sample S [--genome-dir DIR] [--offline FILE] [--rerun [--refresh-clinvar]] | --affected KEYS | --launch-dir DIR + say which results of a finished run a newer ClinVar or catalog would change + og lookup query|resolve|report [args] + your genotype at an rsID, a position or in a gene, from a finished run + og report render [args] + the text and HTML report from a run's summary.json + og live serve [args] + the live site of a run + og view serve [args] + igv.js over one sample's reads and calls + +Pipeline helpers (the steps run these): + og map build --root DIR --run RUN --assets DIR --cache-dir DIR [--predicate-table] + og map genes --root DIR --cache-dir DIR (--chrom CHROM | --names) + og map svg [args] + og setup mask-fasta SRC BED DEST + og setup unzip ZIP DEST + og setup genetic-map --zip ZIP --out DIR [--rfmix] CHROM... + og setup chrm-shift [args] + og setup revel [args] + og setup roh-af [args] + og vcf csq-rank [options] + og vcf index [args] + og vcf shift-back [args] + og table constraint-join --gene-col NAME [--constrained] CONSTRAINT TABLE + og sv svlen-fix|inv-filter [args] + og sv add-callers [args] + og sv postfilter [args] + og sv survivor-prep [args] + og sv consensus-check [args] + og pgx cpic-report|step36-outside [args] + og pgx refblock-fill|mito-merge|cyp2d6-bed|cyp2d6-measure|cyp2d6-check [args] + og pgx outside-calls|hla-concordance|pypgx-summary [args] + og paraphase check-bam|table [args] + og clinvar join-adjacent|hits|deep-intronic [args] + og str motif|catalog [args] + og cdna check [args] + og cdna scan [args] + og comphet phase [args] + og bam contigs BAM + og qc sex-loss --table TABLE [--regions REGIONS] [--vcf VCF] + og qc sample [args] + og prs format [args] + og prs table [args] + og mito annotate|deletions|copies [args] + og summary collect [--only {a,b,c}] [args] + og report index [args] + og report charts [args] + og live results [args] + og live family [args] + og live demo [args] + og couple screen [args] + og couple genes [args] + og roh summary --sample SAMPLE --roh ROH [--out OUT] + og chip prepare|report|check-table [args] + og cn bed|call|intervals|check-bam|marker-regions|marker-read [args] + og traits table [args] + og traits phase-sites [args] + og traits check [args] + og bloodtype report|check-markers|systems [args] + og family plan|prepare|check|report|segments [args] + og family ibd [args] + og consumer build-sites|export|check [args] + og ancestry-fine panel-map|check-map|order|sample-bed|chroms|report [args] + og ancestry-fine freqs [args] + og ancestry-fine project [args] + og ancestry-fine paint [args] + og ancestry-fine simulate [args] + og archaic table|strict|genotypes|report [args] + og archaic compare [args] + og archaic genomes [args] + og ancient select|lift|person-geno|align|report [args] + og vntyper report [args] + og locityper report [args] + og mocha report [args] + og mei report [args] + og mei annotsv-input [args] + og viral report [args] + og y str-bed [args] + og y str-report [args] + og y tree-positions --tree-dir DIR OUT + og y tree-place [args] + og y private-snps [args] + og y yleaf-positions --data DATA OUT + og exomiser prepare [args] + og exomiser report [args] + og longread clair3-model [args] + +This og: + og version + og --tree-hash + +Developer and CI tools: og dev --help lists them. "; +/// The developer and CI tools, which `og dev --help` prints. They stay out of +/// [`USAGE`], so `og --help` lists only what a user or a step runs. +const DEV_USAGE: &str = " og dev parity-compare|contract|schema|report-only|remnants [args] + og dev scan personal-data|images [args] + og dev versions check-run|expand|coupled [args] + og dev pgs trait|label-match|metric-match [args] + og dev fixture comphet OUT [--swap] + og dev fixture trim-pharmcat REPORT OUT [MAX] + og dev evidence clashes RAW.vcf.gz SPLIT.vcf.gz DATA_DIR OUT.tsv + og dev evidence hg002-gvcf TRUTH.vcf.gz TRUTH.bed REF.fasta SITES.tsv OUT.vcf + og dev renovate-check LOG + og dev pharmcat diplotypes REPORT [--called|--called-lines] + og dev nf-trace requests|attempts TRACE [options] + og dev smoke mini-ref|long-reads|eh-catalog|csv-cell|check [args] + og dev helpers [args] + og dev build cn-intervals|callability|sv-rule-regions [args] + og dev build roh-af [args] + og dev build acmg-lof [args] + og dev demo-sample [args] + og dev demo-events [args] + og dev params-doc [args] + og dev python-gate [args] + og dev everything-check [args] + og dev licences [args] + og dev docs-check [args] + og dev stub-http [args] + og dev steps-check [args] + og dev traits-sources [args] + og dev freshness [--out FILE] [--sections a,b] | --update-issue BODY --repo O/N | --self-test +"; + +/// Commands whose own parser answers `-h`/`--help` with a fuller usage than +/// their line in [`USAGE`] or [`DEV_USAGE`]: `tool` for every verb of it, +/// `tool a|b` for those verbs. Any other command's help request is answered +/// from its lines there by [`help`], so a new verb answers `--help` with no +/// change here. +const OWN_HELP: &[&str] = &[ + "run", + "recheck", + "clinvar", + "lookup", + "family", + "chip", + "traits", + "bloodtype", + "cn", + "paraphase", + "ancient", + "pgx cpic-report|step36-outside|refblock-fill|mito-merge|outside-calls|hla-concordance|pypgx-summary", + "vcf csq-rank|index", + "table constraint-join", + "sv svlen-fix|inv-filter|add-callers|postfilter", + "str catalog", + "qc sex-loss", + "report render", + "couple screen|genes", + "ancestry-fine panel-map|check-map|order|sample-bed|chroms|report|simulate", + "archaic table|strict|genotypes|report", + "vntyper report", + "locityper report", + "mocha report", + "mei report|annotsv-input", + "viral report", + "y str-bed|str-report|tree-positions|tree-place|private-snps|yleaf-positions", + "exomiser prepare|report", + "dev freshness|contract|report-only|remnants|renovate-check|helpers|params-doc|traits-sources", +]; + +/// `og WORDS... -h|--help [...]`: the usage lines of the command named by +/// the words before the first option, on stdout, exit 0; the lines of its +/// verb when one matches, else every line of the tool. `None` leaves the +/// request to the command: it is in [`OWN_HELP`] (its `-h` then reaches it +/// as `--help`, which every one of them reads), names no tool with a line +/// (an unknown tool keeps its refusal), or holds no help word before `--`. +fn help(raw: &mut [String]) -> Option { + let end = raw.iter().position(|a| a == "--").unwrap_or(raw.len()); + let args: Vec<&str> = raw[..end].iter().map(String::as_str).collect(); + if !args.iter().any(|a| *a == "-h" || *a == "--help") { + return None; + } + // A root option with its own usage line (`og --tree-hash`) is a tool here. + let words: Vec<&str> = args + .iter() + .copied() + .enumerate() + .take_while(|(i, a)| !a.starts_with('-') || (*i == 0 && *a == "--tree-hash")) + .map(|(_, a)| a) + .collect(); + let (tool, verb) = (*words.first()?, words.get(1).copied()); + let own = OWN_HELP.iter().any(|e| match e.split_once(' ') { + None => *e == tool, + Some((t, vs)) => t == tool && verb.is_some_and(|v| vs.split('|').any(|x| x == v)), + }); + if own { + for a in &mut raw[..end] { + if a == "-h" { + *a = "--help".to_owned(); + } + } + return None; + } + let lines: Vec<(&str, Vec<&str>)> = USAGE + .lines() + .chain(DEV_USAGE.lines()) + .map(|l| l.trim_start_matches("usage:").trim()) + .filter_map(|l| { + let mut w = l.split(' '); + (w.next() == Some("og") && w.next() == Some(tool)).then(|| { + ( + l, + w.next().map_or_else(Vec::new, |v| v.split('|').collect()), + ) + }) + }) + .collect(); + let of_verb: Vec<&str> = lines + .iter() + .filter(|(_, vs)| verb.is_some_and(|v| vs.contains(&v))) + .map(|(l, _)| *l) + .collect(); + let show = if of_verb.is_empty() { + lines.iter().map(|(l, _)| *l).collect() + } else { + of_verb + }; + if show.is_empty() { + return None; + } + for (i, l) in show.iter().enumerate() { + println!("{}{l}", if i == 0 { "usage: " } else { " " }); + } + Some(ExitCode::SUCCESS) +} + /// argparse's way to refuse a command line: usage and the error on stderr, exit 2. fn usage_error(msg: &str) -> ExitCode { eprint!("{USAGE}"); @@ -175,7 +302,10 @@ fn main() -> ExitCode { eprintln!("ERROR: path is not valid UTF-8"); return ExitCode::from(2); } - let args: Vec = std::env::args().skip(1).collect(); + let mut args: Vec = std::env::args().skip(1).collect(); + if let Some(code) = help(&mut args) { + return code; + } let Some(tool) = args.first() else { return usage_error("the following arguments are required: tool"); }; diff --git a/rust/og/tests/port_scaffold.rs b/rust/og/tests/port_scaffold.rs index da22c7cbc..b96d5a8f8 100644 --- a/rust/og/tests/port_scaffold.rs +++ b/rust/og/tests/port_scaffold.rs @@ -1,7 +1,8 @@ //! Every verb the 1.0.0 plan adds is registered in `og`: listed on its own -//! line of `og --help` and routed to its module, never to the "unknown" -//! error of a router. A port replaces a stub's body, so this test keeps -//! holding after it; a verb that a later merge drops turns it red. +//! line of `og --help` (of `og dev --help` for an `og dev` verb) and routed +//! to its module, never to the "unknown" error of a router. A port replaces +//! a stub's body, so this test keeps holding after it; a verb that a later +//! merge drops turns it red. //! //! While a verb is still a stub it must say exactly `og : not ported //! yet` on stderr and exit 2, so no caller mistakes it for a result. @@ -87,12 +88,17 @@ fn og(args: &[&str]) -> (Option, String) { #[test] fn every_planned_verb_has_its_own_usage_line() { - let out = Command::new(env!("CARGO_BIN_EXE_og")) - .arg("--help") - .output() - .expect("og --help runs"); - assert!(out.status.success()); - let usage = String::from_utf8(out.stdout).expect("utf-8 usage"); + // `og --help` lists the user verbs and the pipeline helpers, `og dev + // --help` the developer and CI tools. + let mut usage = String::new(); + for args in [&["--help"][..], &["dev", "--help"]] { + let out = Command::new(env!("CARGO_BIN_EXE_og")) + .args(args) + .output() + .expect("og --help runs"); + assert!(out.status.success(), "og {}", args.join(" ")); + usage.push_str(&String::from_utf8(out.stdout).expect("utf-8 usage")); + } let lines: Vec<&str> = usage .lines() .map(|l| l.trim_start_matches("usage:").trim()) diff --git a/rust/og/tests/rv_og_cli_conventions.rs b/rust/og/tests/rv_og_cli_conventions.rs new file mode 100644 index 000000000..c56a79231 --- /dev/null +++ b/rust/og/tests/rv_og_cli_conventions.rs @@ -0,0 +1,305 @@ +//! The help rules every `og` command keeps (docs/rust.md, "Command +//! lines"): +//! +//! - every command on a line of `og --help` or `og dev --help` answers +//! `--help` and `-h` with its usage on stdout and exit 0, and writes no +//! file while doing so; +//! - `og --help` starts with how to run a genome and lists no `og dev` +//! line; `og dev --help` lists those, and nothing else; +//! - `og clinvar --help` names the defaults of its options. + +use std::io::Read; +use std::path::{Path, PathBuf}; +use std::process::{Command, Stdio}; +use std::time::{Duration, Instant}; + +struct Out { + code: Option, + stdout: String, + stderr: String, +} + +/// Runs `og ARGS` in `cwd` with no stdin; killed and failed after 30 s. +fn og_in(cwd: &Path, args: &[&str]) -> Out { + let mut child = Command::new(env!("CARGO_BIN_EXE_og")) + .args(args) + .current_dir(cwd) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .expect("og starts"); + let mut so = child.stdout.take().expect("piped stdout"); + let mut se = child.stderr.take().expect("piped stderr"); + let ro = std::thread::spawn(move || { + let mut b = Vec::new(); + so.read_to_end(&mut b).map(|_| b) + }); + let re = std::thread::spawn(move || { + let mut b = Vec::new(); + se.read_to_end(&mut b).map(|_| b) + }); + let start = Instant::now(); + let status = loop { + if let Some(status) = child.try_wait().expect("og waits") { + break status; + } + if start.elapsed() > Duration::from_secs(30) { + let _ = child.kill(); + let _ = child.wait(); + panic!("og {} still ran after 30 s", args.join(" ")); + } + std::thread::sleep(Duration::from_millis(10)); + }; + let text = |b: Vec| String::from_utf8_lossy(&b).into_owned(); + Out { + code: status.code(), + stdout: text(ro.join().expect("reader").expect("stdout")), + stderr: text(re.join().expect("reader").expect("stderr")), + } +} + +/// A new empty directory of this test. +fn scratch(name: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("og-cli-conventions-{}-{name}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).expect("scratch dir"); + d +} + +/// The `og ...` lines of a help text, without `usage:` and indentation. +fn og_lines(text: &str) -> Vec { + text.lines() + .map(|l| l.trim_start_matches("usage:").trim()) + .filter(|l| l.starts_with("og ")) + .map(str::to_owned) + .collect() +} + +fn is_verb(w: &str) -> bool { + w.starts_with(|c: char| c.is_ascii_lowercase()) + && w.chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') +} + +/// The commands a usage line names: the tool alone, and the tool with each +/// of the verbs of its second word. +fn commands(line: &str) -> Vec> { + let w: Vec<&str> = line.split_whitespace().collect(); + // A root option with its own line (`og --tree-hash`) is a command too. + let Some(tool) = w + .get(1) + .filter(|t| is_verb(t) || t.strip_prefix("--").is_some_and(is_verb)) + else { + return vec![]; + }; + let mut out = vec![vec![(*tool).to_owned()]]; + if let Some(verbs) = w.get(2).filter(|v| v.split('|').all(is_verb)) { + for v in verbs.split('|') { + out.push(vec![(*tool).to_owned(), v.to_owned()]); + } + } + out +} + +#[test] +fn every_command_answers_help_on_stdout_and_writes_nothing() { + let here = scratch("help"); + let root = og_in(&here, &["--help"]); + let dev = og_in(&here, &["dev", "--help"]); + assert_eq!(root.code, Some(0), "og --help: {}", root.stderr); + let mut lines = og_lines(&root.stdout); + lines.extend(og_lines(&dev.stdout)); + let mut seen = std::collections::BTreeSet::new(); + let mut failures = Vec::new(); + for line in &lines { + for cmd in commands(line) { + if !seen.insert(cmd.clone()) { + continue; + } + for flag in ["--help", "-h"] { + let mut args: Vec<&str> = cmd.iter().map(String::as_str).collect(); + args.push(flag); + let o = og_in(&here, &args); + if o.code != Some(0) || o.stdout.trim().is_empty() { + failures.push(format!( + "og {}: exit {:?}, {} bytes on stdout, stderr {:?}", + args.join(" "), + o.code, + o.stdout.len(), + o.stderr.lines().last().unwrap_or("") + )); + } + let left: Vec<_> = std::fs::read_dir(&here) + .expect("scratch dir") + .map(|e| e.expect("entry").file_name()) + .collect(); + if !left.is_empty() { + failures.push(format!("og {} wrote {left:?}", args.join(" "))); + let _ = std::fs::remove_dir_all(&here); + std::fs::create_dir_all(&here).expect("scratch dir"); + } + } + } + } + assert!( + seen.len() > 100, + "only {} commands read from the usage", + seen.len() + ); + assert!( + seen.contains(&vec!["--tree-hash".to_owned()]), + "og --tree-hash is not checked" + ); + assert!(failures.is_empty(), "{}", failures.join("\n")); + let _ = std::fs::remove_dir_all(&here); +} + +#[test] +fn the_help_fallback_names_the_verb_and_an_unknown_tool_is_still_refused() { + let here = scratch("fallback"); + // A verb answered from the root usage gets its own line, not the tool's. + let o = og_in(&here, &["sv", "consensus-check", "--help"]); + assert_eq!(o.code, Some(0)); + assert_eq!(o.stdout, "usage: og sv consensus-check [args]\n"); + // A help word after `--` is an argument, not a request. + let o = og_in(&here, &["sv", "consensus-check", "--", "--help"]); + assert_ne!(o.code, Some(0)); + // The same holds for a command that reads its own help word. + for args in [ + &["clinvar", "hits", "--", "--help"][..], + &["pgx", "cpic-report", "--", "--help"], + &["dev", "freshness", "--", "-h"], + ] { + let o = og_in(&here, args); + assert_ne!(o.code, Some(0), "og {}: {}", args.join(" "), o.stdout); + } + // The control: no line, no help. + let o = og_in(&here, &["no-such-tool", "--help"]); + assert_eq!(o.code, Some(2)); + assert!( + o.stderr.contains("unknown tool 'no-such-tool'"), + "{}", + o.stderr + ); + let _ = std::fs::remove_dir_all(&here); +} + +#[test] +fn og_help_starts_with_how_to_run_a_genome() { + let here = scratch("root"); + let root = og_in(&here, &["--help"]).stdout; + let head: Vec<&str> = root.lines().take(15).collect(); + assert!( + head.iter() + .any(|l| l.contains("./scripts/run-all.sh SAMPLE")), + "the first 15 lines of og --help do not say how to start a run:\n{}", + head.join("\n") + ); + assert!( + head.iter() + .any(|l| l.contains("./scripts/validate-setup.sh")) + ); + let run = root + .lines() + .position(|l| l.trim().starts_with("og run state")) + .expect("an og run line"); + let what = root.lines().nth(run + 1).unwrap_or(""); + assert!( + what.contains("progress") && what.contains("never starts a run"), + "og run is not described as inspecting a run: {what:?}" + ); + // No og dev line in og --help: it points to og dev --help, which lists + // them all and nothing else. + let in_root: Vec = og_lines(&root) + .into_iter() + .filter(|l| l.starts_with("og dev ")) + .collect(); + assert!(in_root.is_empty(), "og dev lines in og --help: {in_root:?}"); + assert!( + root.lines() + .any(|l| l.starts_with("Developer and CI tools") && l.contains("og dev --help")), + "og --help does not point to og dev --help" + ); + let dev = og_in(&here, &["dev", "--help"]); + assert_eq!(dev.code, Some(0)); + let in_dev = og_lines(&dev.stdout); + assert!(in_dev.len() > 20, "{in_dev:?}"); + let other: Vec<&String> = in_dev + .iter() + .filter(|l| !l.starts_with("og dev ")) + .collect(); + assert!( + other.is_empty(), + "og dev --help shows other tools: {other:?}" + ); + let _ = std::fs::remove_dir_all(&here); +} + +#[test] +fn og_clinvar_help_names_its_defaults() { + let here = scratch("clinvar"); + for args in [ + &["clinvar", "--help"][..], + &["clinvar", "hits", "--help"], + &["clinvar", "deep-intronic", "-h"], + ] { + let o = og_in(&here, args); + assert_eq!(o.code, Some(0), "og {}: {}", args.join(" "), o.stderr); + for want in [ + "og clinvar join-adjacent", + "og clinvar hits", + "og clinvar deep-intronic", + ] { + assert!(o.stdout.contains(want), "{want} not in:\n{}", o.stdout); + } + for (opt, default) in [ + ("--min-gq N", "30"), + ("--min-dp N", "10"), + ("--window N", "10"), + ] { + let line = o + .stdout + .lines() + .find(|l| l.trim_start().starts_with(opt)) + .unwrap_or_else(|| panic!("no {opt} line in:\n{}", o.stdout)); + assert!(line.ends_with(&format!("(default {default})")), "{line}"); + } + } + let _ = std::fs::remove_dir_all(&here); +} + +#[test] +fn og_run_state_with_out_prints_nothing_on_stdout() { + // Output convention (docs/rust.md, "Command lines"): the result goes to + // --out FILE, and the sentence that says so goes to stderr. + let here = scratch("run-state"); + let sample = here.join("S1"); + std::fs::create_dir_all(&sample).expect("sample dir"); + let steps = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../assets/steps.tsv"); + let out = here.join("state.json"); + let o = og_in( + &here, + &[ + "run", + "state", + "--sample", + "S1", + "--sample-dir", + sample.to_str().expect("utf-8 path"), + "--steps", + steps.to_str().expect("utf-8 path"), + "--out", + out.to_str().expect("utf-8 path"), + ], + ); + assert_eq!(o.code, Some(0), "{}", o.stderr); + assert_eq!(o.stdout, "", "og run state --out printed on stdout"); + assert_eq!( + o.stderr, + format!("Run state written: {}\n", out.display()), + "the sentence is not on stderr" + ); + assert!(out.is_file(), "no {}", out.display()); + let _ = std::fs::remove_dir_all(&here); +}