Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
2142fd0
test(docs): fail when a docs table puts an escaped pipe inside a code…
GeiserX Nov 17, 2026
ec90796
fix(docs): make the Nextflow install and three other table commands c…
GeiserX Nov 17, 2026
b777b7d
test(docs): plant the double-backtick row with a heredoc
GeiserX Nov 17, 2026
9935a5d
docs(privacy): say which servers the CPSR and MultiQC reports contact…
GeiserX Nov 17, 2026
21c1cba
test(docs): check the active menu item contrast and the code line anc…
GeiserX Nov 17, 2026
9a63979
fix(docs): give the active menu item the link colour of each scheme
GeiserX Nov 17, 2026
7b83aec
fix(docs): drop the empty line anchors from code blocks
GeiserX Nov 17, 2026
e0307ae
Merge remote-tracking branch 'origin/main' into pgp/ux-docs-table-pipes
GeiserX Nov 18, 2026
eab8e27
Merge remote-tracking branch 'origin/main' into pgp/ux-docs-report-pr…
GeiserX Nov 18, 2026
2f5a61c
fix(docs): keep the text on the accent readable in the dark scheme
GeiserX Nov 18, 2026
978c48b
Merge remote-tracking branch 'origin/main' into pgp/ux-docs-theme-con…
GeiserX Nov 18, 2026
8bcdbab
docs(privacy): name CPSR 2.3.0 as the checked example and say why the…
GeiserX Nov 18, 2026
d5efc9b
docs(updating): set GENOME_DIR before hashing the step 40 site table
GeiserX Nov 18, 2026
2dc458d
fix(docs): check borderless tables and real fence closers, and refuse…
GeiserX Nov 18, 2026
65253be
ci(guard): stop timing out the rust and parity jobs on a slow runner
GeiserX Nov 18, 2026
324dab3
Merge #493 ux-docs-table-pipes into the batch f tip
GeiserX Nov 18, 2026
b1b7640
Merge #496 ux-docs-theme-contrast-anchors into the batch f tip
GeiserX Nov 18, 2026
a301afd
Merge #494 ux-docs-report-privacy into the batch f tip
GeiserX Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions .github/workflows/guard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -257,11 +257,12 @@ jobs:
# unit-tests is close to its 10 minutes. Format, lint, unit tests with a
# planted failure that must go red, pycompat against CPython, then the two
# musl builds as scripts/setup.sh --og makes them and the stale-binary
# guard on the real binaries. It takes 15 to 18 minutes on a free runner,
# and a busy runner pushed it past 20, so the limit is 30.
# guard on the real binaries. It took 15 to 18 minutes on a free runner and
# 23 on main on 2026-11-18, and the same tree on the geiresx fork ran past
# 30 twice, so the limit is 45: it stops a hung job, not a slow one.
rust:
runs-on: ubuntu-latest
timeout-minutes: 30
timeout-minutes: 45
defaults:
run:
working-directory: rust
Expand Down Expand Up @@ -848,10 +849,11 @@ jobs:
# planted kind of difference must be reported, by oracle-diff.sh, by
# og dev parity-compare and by bin/og's shadow mode), then every case of
# tests/parity/cases.tsv with the oracle in its production image, then
# tests/test_fixitfelix.sh with og.
# tests/test_fixitfelix.sh with og. It took 19 minutes on main on
# 2026-11-18 and ran past 20 twice on the geiresx fork, so the limit is 35.
parity:
runs-on: ubuntu-latest
timeout-minutes: 20
timeout-minutes: 35
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- Four commands in docs tables now copy and run as written. The docs site printed the backslash that escaped each pipe for GitHub, so the Nextflow install line in Getting started read `curl -s https://get.nextflow.io \| ...`, which is not a pipeline. That row now points at the install command in Full run. The BAM build check of the FAQ and the site table hash of Keeping versions up to date moved into code blocks under their tables, and the `readings` row of Copy Number from Depth names its separator in words. `tests/test_docs_table_pipes.sh` fails when a table row under `docs/` holds an escaped pipe inside a code span.
- The docs site: the current page in the side menu is readable, and code blocks no longer add a Tab stop per line. The menu paints the current page in the accent colour, #2563eb, on a light tint of the same blue. That was 3.1:1 on the dark page and 4.49:1 on the light one, below the 4.5:1 that WCAG AA asks of normal text. The accent now takes the link colour of each scheme: #60a5fa on dark (6.3:1) and #1d4ed8 on light (5.8:1). On dark, the text Zensical writes on the accent (the "Back to top" button and the side menu button on hover or focus) takes the page colour, #1c1917, at 6.9:1: white on #60a5fa would be 2.5:1. `zensical.toml` no longer sets `anchor_linenums`: the site shows no line numbers, yet every code line got an empty link with no name, 122 of them on the getting-started page, on 88 pages in all. Code highlighting and the copy button do not change. A cargo test, `rust/crates/og-dev/tests/docs_theme.rs`, checks the colours both ways and the anchors.
- The privacy page now says what the CPSR (step 17) and MultiQC (step 28) reports contact when you open them: nothing. Both keep their stylesheets, scripts and images inside the HTML file, and their external addresses are plain links. The page said before that these two reports were not checked. Docs only; no output changes.
- A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again.
- VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37).
- The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2).
Expand Down
2 changes: 1 addition & 1 deletion docs/cn-loci.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Then the row's `call` column decides:
| `mapq0_min`, `mapq0_max` | the MAPQ 0 share guards, or `.` |
| `min_control_depth`, `max_control_ratio` | the control guards, or `.` |
| `call`, `interval` | as above |
| `readings` | `KEY=text`, separated by `\|`: the text for a count, a class, or `more` for any count above the keys |
| `readings` | `KEY=text`, separated by a vertical bar: the text for a count, a class, or `more` for any count above the keys |
| `input` | `bedcov` (this engine) or `cnvpytor-level` (the row records thresholds on CNVpytor's depth level, where 1.0 is two copies) |
| `owner` | the engine that calls the locus (`cn-depth`, this one) |
| `cross_check` | another engine that also reports the locus (`parascopy:RHCE`, `paraphase:hba`, `cnvpytor:cnvs` for step 18's CNVpytor calls) or `none` |
Expand Down
8 changes: 7 additions & 1 deletion docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,17 @@
| "Permission denied" writing output | Container runs as non-root | Add `--user root` to `docker run` (already done in all scripts) |
| VEP cache download fails/times out | 26 GB download over unreliable connection | Use `wget -c` (supports resume). See [docs/13-vep-annotation.md](13-vep-annotation.md) |
| DeepVariant crashes on Mac | amd64 emulation + memory pressure | `THREADS=2 DV_MEM=12g ./scripts/03-deepvariant.sh <sample> <sex>`, the sex being `male` or `female`. Will be slow. See [troubleshooting](troubleshooting.md#step-3-deepvariant-crashes-on-mac-amd64-emulation). |
| Wrong number of variants (too few) | Genome build mismatch | Ensure your BAM is aligned to GRCh38 (hg38), not hg19/GRCh37. Check with `samtools view -H your.bam \| grep SN:chr1` |
| Wrong number of variants (too few) | Genome build mismatch | Ensure your BAM is aligned to GRCh38 (hg38), not hg19/GRCh37. Check the BAM header with the command under this table |
| 0-byte output files | Missing input or wrong path | Check that all input files exist. Run the script with `bash -x` for debug output. |
| "No such image" on `docker pull` | Image name/tag changed | Check the exact image name in the step's documentation. Biocontainer tags change frequently. |
| Very slow on macOS | Rosetta 2 emulation overhead | Expected. Consider running on a Linux machine or cloud instance for heavy steps. |

To check the genome build of a BAM, print the chr1 line of its header. GRCh38 has `LN:248956422`, hg19 has `LN:249250621`, and GRCh37 names the contig `1`, so the command prints nothing:

```bash
samtools view -H your.bam | grep -w SN:chr1
```

For detailed solutions, see:
- [docs/chip-data-guide.md](chip-data-guide.md) — using 23andMe/MyHeritage/AncestryDNA data with this pipeline
- [docs/troubleshooting.md](troubleshooting.md) — comprehensive troubleshooting guide organized by symptom
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
| Docker | 20.10+ | [docs.docker.com/get-docker](https://docs.docker.com/get-docker/) |
| bash | 4.4+ | Pre-installed on Linux; macOS ships 3.2, install a newer one with `brew install bash` |
| Java *(for a full run)* | 17+ | Any OpenJDK build, e.g. `apt install openjdk-17-jre-headless` or [Adoptium Temurin](https://adoptium.net/) |
| Nextflow *(for a full run)* | `NEXTFLOW_VERSION` in `versions.env` | `curl -s https://get.nextflow.io \| NXF_VER=26.10.5 bash`, see [Full run](#full-run) |
| Nextflow *(for a full run)* | `NEXTFLOW_VERSION` in `versions.env` | The pinned install command in [Full run](#full-run) |
| wget or curl | Any | For downloading references |
| python3 *(optional)* | 3.6+ | Used by long-read alignment (02b) for symlink resolution. Falls back to `readlink -f` on GNU/Linux if absent |

Expand Down
16 changes: 14 additions & 2 deletions docs/stylesheets/theme.css
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,29 @@

/* Dark scheme: the page body uses the header colour too, so the header and
the page read as one surface. #2563eb on #1c1917 is 3.9:1, too faint for
body text; #60a5fa is the same hue at 7.4:1. */
body text; #60a5fa is the same hue at 7.4:1.
The current page in the side menu is painted in the accent, on the
transparent accent laid over the page (#1d202c). #2563eb there is 3.1:1,
so the accent takes the link colour too: #60a5fa is 6.3:1.
Zensical writes --md-accent-bg-color, white, on the accent: the "Back to
top" button and the side menu button on hover or focus. White on #60a5fa
is 2.5:1, so that text takes the page colour: #1c1917 on #60a5fa is
6.9:1, the same pair the "Copied" note uses. */
[data-md-color-scheme="slate"] {
--md-default-bg-color: #1c1917;
--md-code-bg-color: #292524;
--md-typeset-a-color: #60a5fa;
--md-accent-fg-color: #60a5fa;
--md-accent-bg-color: #1c1917;
}

/* Light scheme: #2563eb on white is 4.5:1, on the edge. #1d4ed8 is the same
hue at 6.3:1. */
hue at 6.3:1. On the current menu item's tint (#e9effd) #2563eb is 4.49:1,
under 4.5:1, so the accent takes #1d4ed8 as well: 5.8:1.
rust/crates/og-dev/tests/docs_theme.rs checks both schemes. */
[data-md-color-scheme="default"] {
--md-typeset-a-color: #1d4ed8;
--md-accent-fg-color: #1d4ed8;
}

/* Fixed logo size so it does not crowd the site name. */
Expand Down
11 changes: 10 additions & 1 deletion docs/updating.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,16 @@ This page is for maintainers. It says how a pinned version gets bumped, what che
| IPD-KIR release (step 8 with `KIR=true`) | `KIR_DB_RELEASE` in `versions.env` | nobody: the monthly freshness report says when a newer release exists |
| Parascopy data (step 35) | `PARASCOPY_DATA_VERSION` in `versions.env`, with the Zenodo record and the md5s in `scripts/setup.sh` | nobody: the monthly freshness report says when the record has a newer version |
| Cyrius lock (step 21) | `CYRIUS_VERSION` in `versions.env` and every wheel in `scripts/cyrius-constraints.txt` | nobody: resolved again by hand in `PYTHON_IMAGE`, as the lock's header says. The freshness report says when PyPI has a newer Cyrius |
| The site table of step 40: the 23andMe v5 site list commit, the dbSNP build and the table's hash | `CONSUMER_SITES_COMMIT`, `CONSUMER_DBSNP_BUILD`, their checksums and `CONSUMER_SITES_TABLE_SHA256` in `versions.env` | nobody: dbSNP releases a build every year or two, and the site list has not changed since 2019. A new build, a new commit or a change to `og consumer build-sites` needs one full build (`setup.sh --consumer-sites`, about 30 GB), then `gzip -dc <table> \| sha256sum` into `CONSUMER_SITES_TABLE_SHA256`. CI never builds the full table. The monthly freshness report says when NCBI publishes a newer build |
| The site table of step 40: the 23andMe v5 site list commit, the dbSNP build and the table's hash | `CONSUMER_SITES_COMMIT`, `CONSUMER_DBSNP_BUILD`, their checksums and `CONSUMER_SITES_TABLE_SHA256` in `versions.env` | nobody: dbSNP releases a build every year or two, and the site list has not changed since 2019. A new build, a new commit or a change to `og consumer build-sites` needs one full build: remove the old table from `reference/consumer/` (`setup.sh` keeps a table it finds), run `setup.sh --consumer-sites` (about 30 GB), then record the sha256 of the unpacked table in `CONSUMER_SITES_TABLE_SHA256` (the command is under this table). CI never builds the full table. The monthly freshness report says when NCBI publishes a newer build |

The hash of the site table of step 40 is the sha256 of its unpacked content, because the gzip wrapper holds the time it was written. Run this from the repository root after the build, with `GENOME_DIR` set to the folder you gave `setup.sh`. `gzip -t` stops it when the table is missing or broken, so it never prints the hash of an empty input:

```bash
export GENOME_DIR=/path/to/your/data
. ./versions.env
table="${GENOME_DIR}/reference/consumer/23andme_v5_sites_dbsnp${CONSUMER_DBSNP_BUILD}.tsv.gz"
gzip -t "$table" && gzip -dc "$table" | sha256sum
```

Three pins are floors, not versions that drift: Node 24 in `renovate-dry-run.yml`, Java 17 in the workflows that run Nextflow, and `manifest.nextflowVersion` in `nextflow.config`. Nothing proposes a bump for them, on purpose.

Expand Down
5 changes: 4 additions & 1 deletion docs/why-local.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,10 @@ A report that loads a file from the internet tells that server your IP address a
- **indexcov report (step 16):** loads Chart.js from `cdnjs.cloudflare.com` and jQuery from `code.jquery.com`. Offline, the page opens without its plots.
- **Nextflow report and timeline:** load nothing.

Not checked: the CPSR (step 17) and MultiQC (step 28) reports, which the test cannot produce.
The test's address listing does not see a CPSR or MultiQC report: the test never builds a real CPSR report, and its MultiQC report is written in another job. On 2026-11-17 we opened one of each in a browser that blocked and logged every request to the internet:

- **CPSR report (step 17):** loads nothing. Its stylesheets, scripts and images are inside the HTML file. Its external addresses are plain links to the sources it cites (ClinVar, PanelApp, PubMed and others), which open only when you click them. We checked the CPSR project's own example reports of versions 2.3.0 and 2.4.0. Version 2.3.0 is the CPSR inside the pinned PCGR 2.3.2 image, and its report template embeds every resource.
- **MultiQC report (step 28):** loads nothing. Its external addresses are plain links to MultiQC, Seqera and the papers of the tools it reports. We checked a report from the pinned MultiQC 1.35 image, run with the step's options.

**Recommendations for securing your data:**
- Store genomic data on an encrypted filesystem (LUKS on Linux, FileVault on macOS, BitLocker on Windows)
Expand Down
Loading
Loading