Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- Four commands in docs tables now copy and run as written. The docs site printed the backslash that escaped each pipe for GitHub, so the Nextflow install line in Getting started read `curl -s https://get.nextflow.io \| ...`, which is not a pipeline. That row now points at the install command in Full run. The BAM build check of the FAQ and the site table hash of Keeping versions up to date moved into code blocks under their tables, and the `readings` row of Copy Number from Depth names its separator in words. `tests/test_docs_table_pipes.sh` fails when a table row under `docs/` holds an escaped pipe inside a code span.
- A new page, [User journeys](docs/user-journeys.md), records where a new user gets stuck on each surface: the docs, the demo, the shell and the local site. It gives the outcome of each journey in the first test-as-user run (2026-11-17, commit `65095e97`), the fix for each finding, and what the run could not reach. A fix that lands has a journey to walk again.
- VCF intake now refuses a record with GT after another FORMAT key, such as `DP:GT:GQ`, before a reader can mistake depth for the genotype. The check exits 2 and names the sample, file, first offending position and FORMAT field. It tells users to reorder the keys and sample values together or get a corrected VCF. Records without GT remain allowed by this ordering check. Valid input and output formats are unchanged. Part of #411 (clin-37).
- The `archaic-same-person` job of `public-evidence.yml` puts step 43's Neanderthal totals beside Chen et al.'s released calls with `og archaic compare` and `og archaic genomes`, in Rust. They replace `bin/archaic_compare.py`, which is gone with its test, `tests/test_archaic_compare.py`; those checks are now cargo tests in og-ancestry. The tables are the same, byte for byte, on 20 parity cases and on the inputs of the last green run of the job (NA12878, chromosomes 19 to 22). No step of the pipeline runs them, and no option or output changes. What changes on purpose: a missing file, an unreadable number or an empty spans file stops with an `ERROR:` line and exit 1 instead of a Python traceback; an interval with a negative start or an end before its start stops the command, where the Python counted a negative length; and a `--mine` value without `=` is a usage error (exit 2).
Expand Down
2 changes: 1 addition & 1 deletion docs/cn-loci.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Then the row's `call` column decides:
| `mapq0_min`, `mapq0_max` | the MAPQ 0 share guards, or `.` |
| `min_control_depth`, `max_control_ratio` | the control guards, or `.` |
| `call`, `interval` | as above |
| `readings` | `KEY=text`, separated by `\|`: the text for a count, a class, or `more` for any count above the keys |
| `readings` | `KEY=text`, separated by a vertical bar: the text for a count, a class, or `more` for any count above the keys |
| `input` | `bedcov` (this engine) or `cnvpytor-level` (the row records thresholds on CNVpytor's depth level, where 1.0 is two copies) |
| `owner` | the engine that calls the locus (`cn-depth`, this one) |
| `cross_check` | another engine that also reports the locus (`parascopy:RHCE`, `paraphase:hba`, `cnvpytor:cnvs` for step 18's CNVpytor calls) or `none` |
Expand Down
8 changes: 7 additions & 1 deletion docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,17 @@
| "Permission denied" writing output | Container runs as non-root | Add `--user root` to `docker run` (already done in all scripts) |
| VEP cache download fails/times out | 26 GB download over unreliable connection | Use `wget -c` (supports resume). See [docs/13-vep-annotation.md](13-vep-annotation.md) |
| DeepVariant crashes on Mac | amd64 emulation + memory pressure | `THREADS=2 DV_MEM=12g ./scripts/03-deepvariant.sh <sample> <sex>`, the sex being `male` or `female`. Will be slow. See [troubleshooting](troubleshooting.md#step-3-deepvariant-crashes-on-mac-amd64-emulation). |
| Wrong number of variants (too few) | Genome build mismatch | Ensure your BAM is aligned to GRCh38 (hg38), not hg19/GRCh37. Check with `samtools view -H your.bam \| grep SN:chr1` |
| Wrong number of variants (too few) | Genome build mismatch | Ensure your BAM is aligned to GRCh38 (hg38), not hg19/GRCh37. Check the BAM header with the command under this table |
| 0-byte output files | Missing input or wrong path | Check that all input files exist. Run the script with `bash -x` for debug output. |
| "No such image" on `docker pull` | Image name/tag changed | Check the exact image name in the step's documentation. Biocontainer tags change frequently. |
| Very slow on macOS | Rosetta 2 emulation overhead | Expected. Consider running on a Linux machine or cloud instance for heavy steps. |

To check the genome build of a BAM, print the chr1 line of its header. GRCh38 has `LN:248956422`, hg19 has `LN:249250621`, and GRCh37 names the contig `1`, so the command prints nothing:

```bash
samtools view -H your.bam | grep -w SN:chr1
```

For detailed solutions, see:
- [docs/chip-data-guide.md](chip-data-guide.md) — using 23andMe/MyHeritage/AncestryDNA data with this pipeline
- [docs/troubleshooting.md](troubleshooting.md) — comprehensive troubleshooting guide organized by symptom
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
| Docker | 20.10+ | [docs.docker.com/get-docker](https://docs.docker.com/get-docker/) |
| bash | 4.4+ | Pre-installed on Linux; macOS ships 3.2, install a newer one with `brew install bash` |
| Java *(for a full run)* | 17+ | Any OpenJDK build, e.g. `apt install openjdk-17-jre-headless` or [Adoptium Temurin](https://adoptium.net/) |
| Nextflow *(for a full run)* | `NEXTFLOW_VERSION` in `versions.env` | `curl -s https://get.nextflow.io \| NXF_VER=26.10.5 bash`, see [Full run](#full-run) |
| Nextflow *(for a full run)* | `NEXTFLOW_VERSION` in `versions.env` | The pinned install command in [Full run](#full-run) |
| wget or curl | Any | For downloading references |
| python3 *(optional)* | 3.6+ | Used by long-read alignment (02b) for symlink resolution. Falls back to `readlink -f` on GNU/Linux if absent |

Expand Down
11 changes: 10 additions & 1 deletion docs/updating.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,16 @@ This page is for maintainers. It says how a pinned version gets bumped, what che
| IPD-KIR release (step 8 with `KIR=true`) | `KIR_DB_RELEASE` in `versions.env` | nobody: the monthly freshness report says when a newer release exists |
| Parascopy data (step 35) | `PARASCOPY_DATA_VERSION` in `versions.env`, with the Zenodo record and the md5s in `scripts/setup.sh` | nobody: the monthly freshness report says when the record has a newer version |
| Cyrius lock (step 21) | `CYRIUS_VERSION` in `versions.env` and every wheel in `scripts/cyrius-constraints.txt` | nobody: resolved again by hand in `PYTHON_IMAGE`, as the lock's header says. The freshness report says when PyPI has a newer Cyrius |
| The site table of step 40: the 23andMe v5 site list commit, the dbSNP build and the table's hash | `CONSUMER_SITES_COMMIT`, `CONSUMER_DBSNP_BUILD`, their checksums and `CONSUMER_SITES_TABLE_SHA256` in `versions.env` | nobody: dbSNP releases a build every year or two, and the site list has not changed since 2019. A new build, a new commit or a change to `og consumer build-sites` needs one full build (`setup.sh --consumer-sites`, about 30 GB), then `gzip -dc <table> \| sha256sum` into `CONSUMER_SITES_TABLE_SHA256`. CI never builds the full table. The monthly freshness report says when NCBI publishes a newer build |
| The site table of step 40: the 23andMe v5 site list commit, the dbSNP build and the table's hash | `CONSUMER_SITES_COMMIT`, `CONSUMER_DBSNP_BUILD`, their checksums and `CONSUMER_SITES_TABLE_SHA256` in `versions.env` | nobody: dbSNP releases a build every year or two, and the site list has not changed since 2019. A new build, a new commit or a change to `og consumer build-sites` needs one full build: remove the old table from `reference/consumer/` (`setup.sh` keeps a table it finds), run `setup.sh --consumer-sites` (about 30 GB), then record the sha256 of the unpacked table in `CONSUMER_SITES_TABLE_SHA256` (the command is under this table). CI never builds the full table. The monthly freshness report says when NCBI publishes a newer build |

The hash of the site table of step 40 is the sha256 of its unpacked content, because the gzip wrapper holds the time it was written. Run this from the repository root after the build, with `GENOME_DIR` set to the folder you gave `setup.sh`. `gzip -t` stops it when the table is missing or broken, so it never prints the hash of an empty input:

```bash
export GENOME_DIR=/path/to/your/data
. ./versions.env
table="${GENOME_DIR}/reference/consumer/23andme_v5_sites_dbsnp${CONSUMER_DBSNP_BUILD}.tsv.gz"
gzip -t "$table" && gzip -dc "$table" | sha256sum
```

Three pins are floors, not versions that drift: Node 24 in `renovate-dry-run.yml`, Java 17 in the workflows that run Nextflow, and `manifest.nextflowVersion` in `nextflow.config`. Nothing proposes a bump for them, on purpose.

Expand Down
207 changes: 207 additions & 0 deletions tests/test_docs_table_pipes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
#!/usr/bin/env bash
# A command inside a code span in a docs table must not hold an escaped pipe.
#
# GitHub needs `\|` to keep a pipe inside a table cell, but the docs site
# prints a code span as written, so the reader sees and copies the backslash:
# `curl -s https://get.nextflow.io \| NXF_VER=... bash` is not a pipeline.
# A command that needs a pipe goes in a code block under the table, and a
# table cell names a separator in words.
#
# Checks:
# 1. planted pages with such a row fail, and the scanner names the line:
# a bordered table, a double-backtick span, a table without border
# pipes, a span after an escaped backtick, and a table after a fenced
# block that holds a shorter fence line;
# 2. a planted page with a pipe only where it renders right passes: a
# fenced block, prose, a table cell outside a code span, a code span in
# prose, and backticks that do not pair;
# 3. every docs/**/*.md page passes.
#
# Run: bash tests/test_docs_table_pipes.sh [DIR] (DIR defaults to docs/)
set -euo pipefail

REPO=$(cd "$(dirname "$0")/.." && pwd)
DOCS=${1:-"${REPO}/docs"}
WORK=$(mktemp -d)
trap 'rm -rf "$WORK"' EXIT

FAILS=0
fail() { echo "FAIL: $*"; FAILS=$((FAILS + 1)); }
pass() { echo "ok: $*"; }

# scan FILE...: prints FILE:LINE: TEXT for each table row whose code span holds
# a backslash and a pipe; returns 1 when there is one. A table starts at a line
# with a pipe followed by a separator line (`|---|---|` or `--- | ---`, with or
# without border pipes) and ends at a blank line; its header and every row are
# checked. A fenced block opens with 3 or more backticks or tildes and closes
# only at a run of the same character at least as long with nothing after it;
# lines inside are skipped. A code span opens with a run of N backticks and
# closes at the next run of exactly N; a run with no partner is plain text, and
# a backtick after a backslash is a literal, not an opener.
scan() {
awk '
function runlen(s, c, i) { i = 0; while (substr(s, i + 1, 1) == c) i++; return i }
function bad(line, rest, tmp, n, s, k, off, found, code) {
rest = line
while (match(rest, /`+/)) {
n = RLENGTH; s = RSTART
k = 0; while (s - k > 1 && substr(rest, s - k - 1, 1) == "\\") k++
if (k % 2) { s++; n-- }
rest = substr(rest, s + n)
if (n == 0) continue
tmp = rest; off = 0; found = 0
while (match(tmp, /`+/)) {
if (RLENGTH == n) {
code = substr(rest, 1, off + RSTART - 1)
rest = substr(tmp, RSTART + RLENGTH)
found = 1
break
}
off += RSTART + RLENGTH - 1
tmp = substr(tmp, RSTART + RLENGTH)
}
if (found && index(code, "\\|") > 0) return 1
}
return 0
}
function hit(n, text) { print FILENAME ":" n ": " text; hits++ }
FNR == 1 { fence = 0; table = 0; prev = ""; prevn = 0 }
fence {
t = $0; sub(/^[ \t]*/, "", t); n = runlen(t, fch)
if (n >= flen && substr(t, n + 1) ~ /^[ \t]*$/) fence = 0
next
}
/^[ \t]*(```|~~~)/ {
t = $0; sub(/^[ \t]*/, "", t); fch = substr(t, 1, 1); flen = runlen(t, fch)
fence = 1; table = 0; prev = ""
next
}
/^[ \t]*$/ { table = 0; prev = ""; next }
table { if (bad($0)) hit(FNR, $0); next }
/\|/ && /^[ \t]*\|?[ \t]*:?-+:?[ \t]*(\|[ \t]*:?-+:?[ \t]*)*\|?[ \t]*$/ && index(prev, "|") > 0 {
table = 1
if (bad(prev)) hit(prevn, prev)
next
}
{ prev = $0; prevn = FNR }
END { exit hits > 0 }
' "$@"
}

# expect_hit NAME LINE WHAT: the planted page NAME must fail, and the scanner
# must name exactly one row, at line LINE. A scanner error that names no row
# is a failure too.
expect_hit() {
local name=$1 line=$2 what=$3 out rc=0
out=$(scan "${WORK}/${name}" 2>&1) || rc=$?
if [ "$rc" -eq 0 ]; then
fail "${what} passed"
elif [ "$(grep -c "^${WORK}/${name}:" <<<"$out")" -eq 1 ] && grep -q "^${WORK}/${name}:${line}: " <<<"$out"; then
pass "${what} fails and is named"
else
fail "${what} failed, but the scanner did not name only line ${line}: ${out}"
fi
}

# 1. Planted rows that must fail.
# A row like the old Nextflow line.
cat > "${WORK}/bad.md" <<'MD'
# Planted

| Software | Install |
|---|---|
| Docker | from docs.docker.com |
| Nextflow | `curl -s https://get.nextflow.io \| NXF_VER=1.0.0 bash` |
MD
expect_hit bad.md 6 "a planted table row with an escaped pipe in a code span"

# Double backticks open a span too.
cat > "${WORK}/double.md" <<'MD'
| a | b |
|---|---|
| a | ``x \| y`` |
MD
expect_hit double.md 3 "an escaped pipe in a double-backtick span"

# A table without border pipes is a table too.
cat > "${WORK}/borderless.md" <<'MD'
Tool | Install
--- | ---
Docker | from docs.docker.com
Nextflow | `curl -s https://get.nextflow.io \| bash`
MD
expect_hit borderless.md 4 "an escaped pipe in a code span of a table without border pipes"

# An escaped backtick is a literal, so the span starts at the next backtick.
cat > "${WORK}/escaped.md" <<'MD'
| a | b |
|---|---|
| a | \`example `x \| y` |
MD
expect_hit escaped.md 3 "an escaped pipe in a code span after an escaped backtick"

# A shorter fence line inside a four-backtick block does not close it, so the
# row inside stays skipped and the table after the block is checked.
cat > "${WORK}/fence.md" <<'MD'
````markdown
```
| `inside the block \| here` |
````

| a | b |
|---|---|
| a | `x \| y` |
MD
expect_hit fence.md 8 "an escaped pipe in a table after a block with a shorter fence line"

# 2. Pipes where they render right pass.
cat > "${WORK}/good.md" <<'MD'
# Planted

Prose may say `a \| b` outside a table.

| Column | Meaning |
|---|---|
| `readings` | `KEY=text`, separated by a vertical bar |
| choice | `left` \| `right` |
| stray | a lone `` run, then \| and `code` |

A line that starts with a pipe but has no separator under it is not a table:
| `not a table \| row` |

```bash
dmesg | grep -i "oom\|killed"
| `not a table \| row` |
```

~~~
| `still fenced \| here` |
~~~
MD
if out=$(scan "${WORK}/good.md"); then
pass "pipes in fences, prose, cells outside code spans and unpaired backticks pass"
else
fail "a page with no bad row failed: ${out}"
fi

# 3. The real docs. A find that cannot read part of the tree fails the test,
# so a partial list never passes as a clean scan.
if ! find "$DOCS" -name '*.md' -type f > "${WORK}/pages.raw"; then
fail "could not list the Markdown pages under ${DOCS}"
fi
pages=()
while IFS= read -r page; do pages+=("$page"); done < <(sort "${WORK}/pages.raw")
if [ "${#pages[@]}" -eq 0 ]; then
fail "no Markdown pages under ${DOCS}"
elif out=$(scan "${pages[@]}"); then
pass "no docs table row has an escaped pipe in a code span (${#pages[@]} pages)"
else
echo "$out"
fail "docs table rows with an escaped pipe in a code span; move the command into a code block under the table"
fi

if [ "$FAILS" -gt 0 ]; then
echo "${FAILS} check(s) failed"
exit 1
fi
echo "All checks passed."
Loading