Fix PREP/UPDATE/SUBMIT_SUBMISSION for cloud executors (val→path staging, setup_logging, optional log outputs)

[smanda4]

Description

Please Describe the Bug(s) Fixed and/or the Feature(s) Added:

Three related fixes for tostadas on cloud executors (validated on Google Batch via Seqera Platform; applies to any executor with a URI-shaped workDir). The original scope was the FileNotFoundError in PREP_SUBMISSION / UPDATE_SUBMISSION; two additional bugs surfaced during end-to-end validation and are fixed alongside.

Fix 1: PREP_SUBMISSION and UPDATE_SUBMISSION — val vs path staging

PREP_SUBMISSION and UPDATE_SUBMISSION reference file paths through a val-typed meta/samples map (${meta.batch_tsv}, ${sample.fasta}, etc.). Because those are val inputs and not path inputs, Nextflow doesn't stage the referenced files into the task container. On cloud executors the paths resolve to raw workDir URIs (/tostadas-work/<hash>/...) that don't exist inside the container, and both modules fail before the submission scripts ever run.

Example failure on a cloud executor:

submission_prep.py --metadata_file /tostadas-work/<task-hash>/genbank/batch_1.tsv ...
FileNotFoundError: '/tostadas-work/<task-hash>/genbank/batch_1.tsv'

(where /tostadas-work/ is the workDir set in nextflow.config for the cloud executor)

Fix: lift each per-sample file type out of the meta/samples map into an explicit path input list. Nextflow then stages each list into a per-type subdirectory in the work dir, and the script walks samples in the same order the upstream workflow emitted them, popping the next entry from each file iterator whenever the sample declares that key. Same pattern used in 915d2b3 for submission_config staging in METADATA_VALIDATION. See commits febfe53, 2c2fe1b, c9980bb.

Fix 2: setup_logging silently no-ops when handlers pre-exist

submission_helper.py::setup_logging was guarded by if not logging.getLogger().handlers: and silently returned without configuring anything when handlers already existed. Inside the staphb/tostadas container an earlier import pre-registers a root-logger handler, so FileHandler was never attached and the *.log files that the Nextflow modules declare as outputs (prep_submission.log, update_submission.log, submission.log) were never actually written.

Fix: drop the guard and add force=True to logging.basicConfig so the root logger is unconditionally reconfigured. See commit 10e6544.

Fix 3: Drop optional log-file output declarations in PREP/UPDATE/SUBMIT_SUBMISSION

Even with Fix 2 applied, cloud-executor runs still failed with mv: cannot stat '.../*.log': No such file or directory. On the Google Batch executor, optional: true on path output declarations isn't fully respected during output staging — the staging step attempts mv on the declared file unconditionally and fails the task when the file isn't there.

Fix: remove the individual submission_log output declarations from prep_submission/main.nf, update_submission/main.nf, and submit_submission/main.nf. The catch-all path("${meta.batch_id}") directory output still captures the log file when produced, so downstream consumers don't lose it in successful runs. fetch_reports/main.nf was left as-is because it uses optional: false (log file is expected/required per the schema); with Fix 2, the log file should always exist by that point. See commit 383447d.

Drive-by fix

While in PREP_SUBMISSION I noticed the sample_args builder was checking sample.get("nnp") but referencing sample.nanopore. Upstream workflows populate the "nanopore" key, so the "nnp" check was always false and nanopore samples were silently omitted from sample_args. Fixed to check the correct "nanopore" key.

Files touched

Fix 1 (val/path staging):

• modules/local/prep_submission/main.nf — tuple input now includes staged batch_tsv plus five path lists (fastas, gffs, fq1s, fq2s, nnps); script uses iterators to zip samples with staged files
• modules/local/update_submission/main.nf — tuple input includes staged batch_tsv; script references $batch_tsv
• subworkflows/local/submission.nf — updated take comment
• workflows/genbank.nf — submission_batch_ch now emits a 9-item tuple
• workflows/biosample_and_sra.nf — same shape (fq1/fq2/nnp populated instead of fasta/gff)
• workflows/biosample_update.nf — rebatch_ch tuple includes tsv_file

Fix 2 (setup_logging force=True):

• bin/submission_helper.py — drop guard, add force=True

Fix 3 (drop optional log outputs):

• modules/local/prep_submission/main.nf
• modules/local/update_submission/main.nf
• modules/local/submit_submission/main.nf

Checklist

Go Through Checklist Below and Place A ✔️ (X Inside the Box) if Completed

General Checks

• Have you run appropriate tests (unit/integration/end-to-end) to check logic across run environments (Conda/Docker/Singularity on Scicomp/AWS/NF Tower/Local)?

  Validated end-to-end on Docker + Google Batch (via Seqera Platform) for the genbank workflow with all three fixes applied:

  • Single-sample measles VADR dry-run (AF266288, a public GenBank accession)
  • 8/8 pipeline tasks succeeded (CREATE_BATCH_TSVS → GENBANK_VALIDATION → VADR_MODEL_SETUP → VADR_TRIM → VADR_ANNOTATION → VADR_POST_CLEANUP → PREP_SUBMISSION → SUBMIT_SUBMISSION)
  • Output artifacts confirmed: .sqn, .tbl, .zip, VADR reports
  • Generated .sqn opens with a proper Seq-submit ::= { ASN.1 header and populated contact/affiliation blocks

  Did not test Conda / Singularity / Scicomp / AWS / NF Tower / Local execution paths, nor the biosample_and_sra or biosample_update workflows (no biosample test data available in the environment). Those workflows use the same channel-wiring pattern as genbank and the fixes were applied identically, so they should work, but happy to gate on additional testing if the reviewer wants.

• Have you conducted proper linting procedures?

  • Block comments added at each change explaining the WHY (why files need to be lifted out of val, why iterator coercion is needed, why the setup_logging guard is dropped, why the log outputs are removed)
  • Variable naming matches existing tostadas conventions
  • No Python formatting churn beyond the setup_logging change

• Have you updated existing documentation (README.md, etc.) or created new ones within docs?

  Bug fixes, not new features. Existing README doesn't need changes. Happy to add a "cloud executor support" note if the reviewer prefers.

CDC Checks

• Did you check for sensitive data, and remove any?

  Confirmed: no infrastructure identifiers, credentials, or environment-specific paths in the diff or PR body. Comments use generic examples.

• If you added or modified HTML, did you check that it was 508 compliant?

  N/A — no HTML in this PR.

Are additional approvals needed for this change? If so, please mention them below:

Not that I'm aware of, but please flag if there are.

Are there potential vulnerabilities or licensing issues with any new dependencies introduced? If so, please mention them below:

No new dependencies introduced.