AWS Batch job stuck as RUNNABLE #546

Background

In AWS Batch, jobs are sent to a job queue before running. The job queue only allows certain instance sizes. This is specified in the associated compute environment (CE), and often not configured by the user, but instead someone who has set up the Batch runtime for them to use (following docs).

Problem

Jobs that request resources beyond the limits of the largest instance size available for the compute environment will be stuck as RUNNABLE indefinitely. This is documented behavior by AWS Batch.

The problem is that it's not clear to users why the job is stuck as RUNNABLE, and what can be done to address it.

Possible solutions

1. Nextstrain CLI: Detect when a job requests resources beyond the limits of the largest instance size available for the compute environment and exit with an error instead of submitting the job.
   • This would be the best user experience, but I'm not sure about feasibility. At the very least, users will need additional permissions to batch:DescribeComputeEnvironments and/or ec2:DescribeInstanceTypes. Job queues / compute environments can also be set up in a way where it is difficult to infer limits of the largest instance size.
2. Nextstrain CLI: Detect when a job is stuck in RUNNABLE and display a warning message or automatically cancel.
   • This only applies if the user is attached to the job.
3. Job queue config: set jobStateTimeLimitActions to automatically cancel a job stuck as RUNNABLE due to requesting resources beyond CE limits.
   • This relies on external configuration. Should be added to AWS Batch setup docs.

Additional context

This issue arose during debugging in @blab.