AWS Batch: Transient MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT #550

Description

The auto-cancel feature from #547 was triggered in today's scheduled run of forecasts-flu's "Run models" workflow (failed run, AWS Batch).

It was successful upon re-running. This means MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT can be transient!

Possible solution

Add a timeout before canceling stuck jobs. Maybe this is why jobStateTimeLimitActions has a minimum timeout of 10 minutes and Nextstrain CLI should mirror that in the client-side equivalent.

Investigation

The nextstrain build output lines in the GitHub Actions wait-1 job are jumbled and delayed due to buffering differences between stdout/stderr on non-TTY, so unfortunately it's hard to tell exactly when the job was set to RUNNABLE and reason to MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT.

However, we can tell from AWS that the Batch job was created at 1792697406772 (2026-10-22T19:30:06Z). CloudTrail logs around that time are revealing:

# search 5m window
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances \
  --start-time 1792697400 \
  --end-time 1792697700 \
  --query "Events[0].CloudTrailEvent" \
  --output text

# … "eventTime": "2026-10-22T19:32:42Z",
# … "instanceType": "c7a.8xlarge", …

A brief timeline:

• 19:30:06: AWS Batch job created.
• 19:32:42: AWS Batch invoked EC2 RunInstances for c7a.8xlarge
• 19:33:26: Nextstrain CLI exited with exit code 1.

Presumably if Nextstrain CLI waited longer, it would have moved on from RUNNABLE once the EC2 instance was ready.