Skip to content

AWS Batch: Add timeout before canceling stuck jobs - #551

Open
victorlin wants to merge 1 commit into
masterfrom
victorlin/auto-cancel
Open

victorlin wants to merge 1 commit into
masterfrom
victorlin/auto-cancel

Conversation

@victorlin

@victorlin victorlin commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

This addresses an unexpected auto-cancel during one of our own scheduled workflow runs.

Closes #550

Checklist

  • Tested locally (just the detatch/attach and timeout part, unsure how to reproduce the exact scenario in #550)

     Attaching to Nextstrain AWS Batch Job ID: 8dc36b28-1a2d-4594-9316-7511b8acb1de
     …
     Job is stuck in RUNNABLE. Reason: MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT - The job resource requirement (vCPU/memory/GPU) is higher than that can be met by the CE(s) attached to the job queue.
     Automatically canceling stuck job…
     Job FAILED after 10.1 minutes (Canceled by Nextstrain CLI due to reason: MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT - The job resource requirement (vCPU/memory/GPU) is higher than that can be met by the CE(s) attached to the job queue.)
     
     Requested resources of 100 CPUs and/or 7400 MiB memory are likely beyond the limits of the largest instance size allowed by the job queue.
     
     Try adjusting your requested resources or the job queue in AWS.
    
  • Checks pass

  • Update changelog

@victorlin victorlin self-assigned this Oct 22, 2026
Comment thread nextstrain/cli/runner/aws_batch/__init__.py Outdated
This addresses an unexpected auto-cancel during one of our own scheduled
workflow runs.

<#550>
@victorlin
victorlin force-pushed the victorlin/auto-cancel branch from 41bb297 to 1cf303f Compare October 22, 2026 23:52
@victorlin

Copy link
Copy Markdown
Member Author

Will merge and release on Monday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AWS Batch: Transient MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT

2 participants