Skip to content

fix(setup): setup lists what failed to install and exits non-zero instead of claiming it is complete - #481

Open
geiresx wants to merge 23 commits into
GeiserX:mainfrom
geiresx:pgp/rv-setup-partial-status
Open

geiresx wants to merge 23 commits into
GeiserX:mainfrom
geiresx:pgp/rv-setup-partial-status

Conversation

@geiresx

@geiresx geiresx commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

setup.sh said "Setup complete!" and exited 0 even when installs had failed. The sample QC, STRchive, tandem-repeat, AnnotSV and clinical installers each ran with || true. The other downloads in phase 4 also only printed a warning. After that came the same banner and exit status as a full setup. A user who missed one warning in a long log thought setup was done. A script that checked the exit status could not tell a partial setup from a full one. Meanwhile validate-setup.sh reported the same gaps as warnings.

Fix

  • When an install fails or is skipped, setup now adds it to a list with the exact command that retries it:
    • AnnotSV's data: --annotsv-data
    • the VEP 116 cache: --vep-cache
    • the VEP 115 cache and the CPSR bundle: --clinical-data
    • somalier's sites and VerifyBamID2's panel: --sample-qc-data
    • the STRchive catalog: --strchive-data
    • og: --og
    • the sequence dictionary, the pinned data files, gnomAD's chrM sites and the tandem-repeat map: setup again, because these have no flag of their own.
  • The summary prints "Setup complete!" and exits 0 only when the list is empty. Otherwise it prints "Setup finished: core ready, N items missing", lists each item with its retry command, and exits 3.
  • The retry commands quote the script path, so a checkout path with a space still pastes as one command.
  • --no-clinical-data still ends complete. Skipping the clinical data is a choice, not a gap.
  • docs/getting-started.md lists the three exit statuses: 0 complete, 3 partial, 1 a core failure. docs/troubleshooting.md names the new banner and the Missing list.

Decisions:

  • The PGS Catalog scores, pgsc_calc and its Nextflow plugin are not on the list. Their installer is warnings-only by design, because step 25 fetches what is missing when it runs.
  • The opt-in flags (--annotsv-data, --clinical-data and the others) keep their exit statuses.
  • Eight existing fake-docker cases run a plain setup whose fake downloads always fail their pinned checksums. That setup is partial now, so those cases expect exit 3 and the new banner. This is one line each, and it keeps main green. These eight files and docs/troubleshooting.md sit outside the package's first file list; they change here because the fix makes them wrong otherwise.

Output change: a full setup.sh run that misses something now exits 3 instead of 0, and its banner is different. Nothing changes for a complete setup.

How it was checked

  • New case tests/fake-docker/rv-setup-partial-status.sh covers five runs:
    • Everything is installed: "Setup complete!", exit 0.
    • The AnnotSV download fails: no banner, exactly one item listed (AnnotSV) with --annotsv-data, exit 3.
    • 40 GB free: both VEP caches are listed with their own retries, the bundle still installs, exit 3.
    • --no-clinical-data with everything else installed: complete, exit 0.
    • A plain fake setup: 10 items, each with a retry line, exit 3.
  • The new case is red without the fix. With scripts/ci/run-fake-docker-suite.sh in an ubuntu:24.04 container:
    • the case with main's setup.sh: ASSERT FAIL: annotsv exited 0, expected 3
    • a scratch copy with note_missing turned into a no-op, which leaves the list empty: ASSERT FAIL: annotsv exited 0, expected 3
  • CI on this head (eee0ee9, main merged in):
  • E2E runs on the batch tip that lands this PR.

Bead: F04

With a panel cut to regions, an autosome the panel had no file for was
cut out by `-r` before MoChA ran. It had no row in the sites table, so
the phase gate of og mocha report could not see it, and the sample read
assessed with that autosome never looked at. The bash step and the
Nextflow MOCHA_CALL process now put the phased genotypes onto every
prepared site, so such an autosome reaches MoChA unphased, keeps its
sites row, and the gate marks the sample not assessed and names it.
Both paths print a warning that names it.

Bead: i411-mocha-unpanelled-autosomes
…pped

HG002's fixture holds slices on several autosomes, and the e2e panel is
cut to the chr20 slice. The case asserted that MoChA read chr20 only,
which is the bug. It now checks that the other autosomes reach the sites
table with 0 phased sites, that the sample is not assessed, and that the
message names each of them; a red control runs the same check on the
sites table cut to chr20, which is all the step kept before. The
Nextflow mirror also compares the summary message.

Bead: i411-mocha-unpanelled-autosomes
An index built before the VCF was replaced can lack a contig the new file
has; the indexed read then returned no records and the lookup answered
no call. Compare the modification times, as og-bloodtype does, and stream
the file with a note when the index is older.

Bead: C15
…g setup is complete

setup.sh swallowed the failures of its sample QC, STRchive, tandem-repeat,
AnnotSV and clinical installers and of the phase 4 downloads, then printed
"Setup complete!" and exited 0. A caller could not tell a partial setup
from a full one.

Each failure or skip now goes on a MISSING list with the command that
retries it. The summary prints "Setup complete!" and exits 0 only when the
list is empty; otherwise "Setup finished: core ready, N items missing",
one line per item with its retry command, and exit status 3.

tests/fake-docker/rv-setup-partial-status.sh covers a complete setup, a
failed AnnotSV download, two VEP caches skipped for disk space, the
--no-clinical-data opt-out and a plain fake setup.

Bead: F04
@geiresx
geiresx requested a review from GeiserX as a code owner November 17, 2026 16:51
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 27 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2207350 and 5fa9969.

Files selected for processing: 25

  • CHANGELOG.md
  • docs/51-mocha.md
  • docs/getting-started.md
  • docs/lookup.md
  • docs/nextflow.md
  • docs/troubleshooting.md
  • modules/local/mocha/main.nf
  • rust/crates/og-io/src/lib.rs
  • rust/crates/og-io/tests/atomic_modes.rs
  • rust/crates/og-io/tests/atomic_planted_temp.rs
  • rust/crates/og-io/tests/atomic_root_owner.rs
  • rust/crates/og-lookup/src/vcf.rs
  • rust/crates/og-lookup/tests/rv_lookup_stale_index.rs
  • scripts/51-mocha.sh
  • scripts/setup.sh
  • tests/e2e/mocha-1.sh
  • tests/fake-docker/base-setup-clinvar-raw-only.sh
  • tests/fake-docker/base-setup-no-wget.sh
  • tests/fake-docker/base-setup.sh
  • tests/fake-docker/bash-step-setup-data.sh
  • tests/fake-docker/reference-setup.sh
  • tests/fake-docker/rv-setup-partial-status.sh
  • tests/fake-docker/setup-clinical-default.sh
  • tests/fake-docker/setup-gencode-one-fetch.sh
  • tests/fake-docker/single-source-downloads.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

AtomicFile created its temporary file with File::create on a name made of
the target and the pid. The mode followed the umask, so under umask 022 a
new output was 0644 and a replaced 0600 output became 0644. File::create
also opened whatever was already at that name, and two AtomicFiles on one
target in one process shared one temporary file.

The temporary file is now opened with create_new (O_CREAT | O_EXCL) and
mode 0600, or the replaced target's mode when that is stricter. Its name
carries a process-wide counter, and a taken name moves on to the next.

Bead: rv-io-owner-only-writes
… every slice

With the unpanelled autosomes kept, MoChA reads HG002's other fixture slices
unphased and may call a germline CNV there (such as the chr4 UGT2B17
deletion, untyped because the cut panel's cnps.bed holds chr20 only). The
zero-call check now counts chr20 only; the nothing-reported check stays. The
red control on MOSAIC counts chr20 too, and the docs and comments say a call
on another slice is reported no.
@geiresx

geiresx commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

…ds fail their checksums

The fake downloads fail their pinned checksums, so a plain fake setup is
partial: it now exits 3 and prints the 'Setup finished: core ready' banner.
The troubleshooting page names the new banner and the Missing list, and the
retry commands quote the script path so a path with a space pastes as is.
…some

A Nextflow outdir copies the small .tbi before the large VCF, so a fresh
index often carries the earlier time. Treating every older index as stale
streamed whole genomes for every region there. An older index is now
used when it lists the queried chromosome; only one that does not is
skipped, with a note.
…g runs as root

The docker profile runs containers as root over a work folder the host
user owns. Nextflow's fixOwnership never hands the files back there:
the biocontainers images ship a busybox chown without --from. A 0600
file og wrote as root was then unreadable to the user, and Nextflow
could not publish it. The atomic writer now gives its temporary file to
the owner and group of the nearest folder above it that root does not
own.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants