From e99082921614f5e27a5d4dff77b22eb4489df560 Mon Sep 17 00:00:00 2001 From: Dane Kennedy Date: Thu, 8 Oct 2026 15:35:15 +0200 Subject: [PATCH] Restore sbatch --wait in Cromwell's submit-docker A user's docker tasks are failing since the flag was dropped, and they believe blocking submission fixes it. Put `--wait` back, as in the upstream docs example, so the task can be rerun against the rendered config and the cause narrowed down. With `--wait`, sbatch returns the job's exit status, so a job SLURM kills for time or memory fails the task at once rather than leaving Cromwell waiting forever for an rc file. The cost, recorded in a NOTE at the site, is that Cromwell learns no job id until the job ends: abort cannot scancel it and each running docker task holds a backend thread. `exit-code-timeout-seconds` is the non-blocking alternative to evaluate once the user's failure is understood. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01L5um94i6XqNc8EfAwJxEeF --- ansible/roles/compiled/templates/cromwell_slurm.conf | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/ansible/roles/compiled/templates/cromwell_slurm.conf b/ansible/roles/compiled/templates/cromwell_slurm.conf index 264f321..6aebdbd 100644 --- a/ansible/roles/compiled/templates/cromwell_slurm.conf +++ b/ansible/roles/compiled/templates/cromwell_slurm.conf @@ -80,10 +80,15 @@ backend { # Submit the script to SLURM. `singularity` here and above comes from the # unversioned `singularity` module that cromwell.lua loads; sbatch exports # the submitting environment into the job, so no path is hard-coded. - # No wait flag (unlike the upstream docs example): the submit returns at once - # so Cromwell learns the job id, can scancel it on abort, and detects - # completion through check-alive and the rc file exactly as `submit` does. + # The wait flag, as in the upstream docs example, blocks until the job ends + # and returns its exit status, so a job SLURM kills (time, memory) fails the + # task at once instead of waiting forever for an rc file that never comes. + # NOTE: this costs abort (no job id until the job ends, so kill-docker cannot + # scancel it) and holds a backend thread per running docker task. It was + # removed for those reasons and restored to diagnose a user's failures; + # exit-code-timeout-seconds is the non-blocking alternative to evaluate. sbatch \ + --wait \ -J ${job_name} \ -D ${cwd} \ -o ${cwd}/execution/stdout \