Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/coverage/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ code does not implement. Machine-readable: [`coverage.json`](./coverage.json).
| `backup` | [Backup](./aws/backup.md) | - | - | - | 25 |
| `backupdr` | - | - | [BackupDR](./gcp/backupdr.md) | - | 6 |
| `bastion` | - | [Bastion](./azure/bastion.md) | - | - | 4 |
| `batch` | [Batch](./aws/batch.md) | - | - | - | 14 |
| `batch` | [Batch](./aws/batch.md) | - | - | - | 24 |
| `batch-azure` | - | [Batch](./azure/batch.md) | - | - | 17 |
| `bedrock` | [Bedrock](./aws/bedrock.md) | - | - | - | 65 |
| `bedrockagent` | [BedrockAgent](./aws/bedrockagent.md) | - | - | - | 32 |
Expand Down
2 changes: 1 addition & 1 deletion docs/coverage/aws/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Services cloudemu emulates for AWS, by native name. Back to the [cross-provider
| [AppSync](./appsync.md) | `appsync` | 17 |
| [Athena](./athena.md) | `athena` | 26 |
| [Backup](./backup.md) | `backup` | 25 |
| [Batch](./batch.md) | `batch` | 14 |
| [Batch](./batch.md) | `batch` | 24 |
| [Bedrock](./bedrock.md) | `bedrock` | 65 |
| [BedrockAgent](./bedrockagent.md) | `bedrockagent` | 32 |
| [BedrockAgentRuntime](./bedrockagentruntime.md) | `bedrockagentruntime` | 3 |
Expand Down
22 changes: 19 additions & 3 deletions docs/coverage/aws/batch.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,41 @@

AWS's `batch` service · portable interface `driver.Batch` · [AWS index](./README.md)

## Operations (14)
## Operations (24)

| Operation | Description |
| --- | --- |
| `CancelJob` | |
| `CreateComputeEnvironment` | Compute environments. |
| `CreateJobQueue` | Job queues. |
| `CreateSchedulingPolicy` | Scheduling policies. |
| `DeleteComputeEnvironment` | |
| `DeleteJobQueue` | |
| `DeleteSchedulingPolicy` | |
| `DeregisterJobDefinition` | |
| `DescribeComputeEnvironments` | |
| `DescribeJobDefinitions` | |
| `DescribeJobQueues` | |
| `DescribeJobs` | |
| `DescribeSchedulingPolicies` | |
| `ListJobs` | |
| `ListSchedulingPolicies` | |
| `ListTagsForResource` | |
| `RegisterJobDefinition` | Job definitions. |
| `TagResource` | Tags (by ARN), shared across all three resource kinds. |
| `SubmitJob` | Jobs. |
| `TagResource` | Tags (by ARN), shared across every taggable resource kind. |
| `TerminateJob` | |
| `UntagResource` | |
| `UpdateComputeEnvironment` | |
| `UpdateJobQueue` | |
| `UpdateSchedulingPolicy` | |

## Not in scope

_Not documented yet. See the [emulator boundary](../../../README.md) for cloudemu-wide non-goals._
- Jobs run no workload: a job always succeeds (container `exitCode` 0) once its dependencies succeed. By default it is `SUCCEEDED` as soon as it is submitted; with `--async-settle` it spends about one second each in `SUBMITTED`, `RUNNABLE` and `STARTING`, then five seconds in `RUNNING` (and stays `PENDING` while a dependency is unfinished). `attempts` is always empty, and retry strategies and timeouts are stored and echoed but never applied.
- `CancelJob` on a `PENDING` job takes effect at once. Real Batch waits for the job's dependencies to finish first.
- Array jobs: children are derived from the parent, not stored one by one. `SEQUENTIAL` and `N_TO_N` dependencies are honoured. An array parent reports `PENDING` while any child is unfinished. Multi-node parallel jobs run as a single job: `ListJobs` with `multiNodeJobId` returns an empty list.
- Fair-share scheduling policies are stored, validated and bound to queues, but they never change the order in which jobs run. Jobs never stay `RUNNABLE` because no capacity is free, so "Batch Job Queue Blocked" events are never sent.
- "Batch Job State Change" events go to the EventBridge default bus on the serve scheduler's tick (and on any Batch job API call), one per status the job passed through; none is sent for the initial submission. In library mode, nothing ticks, so events are published when a job API call sees the change or when `Tick` is called. A job that fails reports only `FAILED`, without the earlier steps. Array children send no events of their own. Real Batch sends no state-change events for compute environments or job queues, so the emulator sends none either.
- Error messages are written by the emulator. The exception type (`ClientException`, HTTP 400) is the documented one.
- `containerOverrides`, `ecsPropertiesOverride`, `eksPropertiesOverride` and `nodeOverrides` are merged into the job detail `DescribeJobs` returns (environment variables by name, resource requirements by type, other fields replaced). Overrides are not checked against compute-environment platform rules (for example `nodeOverrides` on Fargate), and multi-node jobs report no per-node `nodeDetails`.
34 changes: 33 additions & 1 deletion docs/coverage/coverage.json
Original file line number Diff line number Diff line change
Expand Up @@ -2655,6 +2655,9 @@
"service": "batch",
"interface": "Batch",
"operations": [
{
"name": "CancelJob"
},
{
"name": "CreateComputeEnvironment",
"doc": "Compute environments."
Expand All @@ -2663,12 +2666,19 @@
"name": "CreateJobQueue",
"doc": "Job queues."
},
{
"name": "CreateSchedulingPolicy",
"doc": "Scheduling policies."
},
{
"name": "DeleteComputeEnvironment"
},
{
"name": "DeleteJobQueue"
},
{
"name": "DeleteSchedulingPolicy"
},
{
"name": "DeregisterJobDefinition"
},
Expand All @@ -2681,16 +2691,35 @@
{
"name": "DescribeJobQueues"
},
{
"name": "DescribeJobs"
},
{
"name": "DescribeSchedulingPolicies"
},
{
"name": "ListJobs"
},
{
"name": "ListSchedulingPolicies"
},
{
"name": "ListTagsForResource"
},
{
"name": "RegisterJobDefinition",
"doc": "Job definitions."
},
{
"name": "SubmitJob",
"doc": "Jobs."
},
{
"name": "TagResource",
"doc": "Tags (by ARN), shared across all three resource kinds."
"doc": "Tags (by ARN), shared across every taggable resource kind."
},
{
"name": "TerminateJob"
},
{
"name": "UntagResource"
Expand All @@ -2700,6 +2729,9 @@
},
{
"name": "UpdateJobQueue"
},
{
"name": "UpdateSchedulingPolicy"
}
],
"providers": {
Expand Down
7 changes: 7 additions & 0 deletions docs/coverage/nongoals/batch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
- Jobs run no workload: a job always succeeds (container `exitCode` 0) once its dependencies succeed. By default it is `SUCCEEDED` as soon as it is submitted; with `--async-settle` it spends about one second each in `SUBMITTED`, `RUNNABLE` and `STARTING`, then five seconds in `RUNNING` (and stays `PENDING` while a dependency is unfinished). `attempts` is always empty, and retry strategies and timeouts are stored and echoed but never applied.
- `CancelJob` on a `PENDING` job takes effect at once. Real Batch waits for the job's dependencies to finish first.
- Array jobs: children are derived from the parent, not stored one by one. `SEQUENTIAL` and `N_TO_N` dependencies are honoured. An array parent reports `PENDING` while any child is unfinished. Multi-node parallel jobs run as a single job: `ListJobs` with `multiNodeJobId` returns an empty list.
- Fair-share scheduling policies are stored, validated and bound to queues, but they never change the order in which jobs run. Jobs never stay `RUNNABLE` because no capacity is free, so "Batch Job Queue Blocked" events are never sent.
- "Batch Job State Change" events go to the EventBridge default bus on the serve scheduler's tick (and on any Batch job API call), one per status the job passed through; none is sent for the initial submission. In library mode, nothing ticks, so events are published when a job API call sees the change or when `Tick` is called. A job that fails reports only `FAILED`, without the earlier steps. Array children send no events of their own. Real Batch sends no state-change events for compute environments or job queues, so the emulator sends none either.
- Error messages are written by the emulator. The exception type (`ClientException`, HTTP 400) is the documented one.
- `containerOverrides`, `ecsPropertiesOverride`, `eksPropertiesOverride` and `nodeOverrides` are merged into the job detail `DescribeJobs` returns (environment variables by name, resource requirements by type, other fields replaced). Overrides are not checked against compute-environment platform rules (for example `nodeOverrides` on Fargate), and multi-node jobs report no per-node `nodeDetails`.
10 changes: 10 additions & 0 deletions internal/settle/settle.go
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,16 @@ const (
DefaultRestoreExpeditedSettle = 1 * time.Second
DefaultRestoreStandardSettle = 3 * time.Second
DefaultRestoreBulkSettle = 5 * time.Second

// DefaultBatchJobPhase is how long an AWS Batch job dwells in SUBMITTED,
// RUNNABLE and STARTING before moving on; RUNNING lasts
// DefaultBatchJobRunDwell.
DefaultBatchJobPhase = 1 * time.Second

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With 1s phases, STARTING to SUCCEEDED takes 2s. Under normal CLI latency I couldn't land terminate-job while the job was STARTING or RUNNING: the job was already SUCCEEDED each time. A longer RUNNING dwell (5 to 10s) would let real CLI and SDK users exercise terminate vs cancel and dependency PENDING.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 689b550. Added settle.DefaultBatchJobRunDwell = 5s (additive; the other phases stay at 1s), stored per job as RunPhase so it survives snapshots. A record from an older snapshot without it falls back to the phase length. With the CLI I polled to RUNNING and terminate-job landed every time: FAILED, isTerminated: true, startedAt kept. Tests: TestJobLifecycleAdvancesWithClock (asserts stoppedAt - startedAt = 5s) and TestJobsAndPoliciesSurviveSnapshot (the dwell survives restore). I didn't make it configurable to keep the scope small; happy to add a flag later if it is wanted.


// DefaultBatchJobRunDwell is how long an AWS Batch job stays RUNNING. It is
// longer than the other phases so a real CLI or SDK client can observe a
// running job and call TerminateJob on it before it finishes.
DefaultBatchJobRunDwell = 5 * time.Second
)

// Window is a read-time overlay describing a resource still settling into its
Expand Down
3 changes: 2 additions & 1 deletion providers/aws/aws.go
Original file line number Diff line number Diff line change
Expand Up @@ -598,6 +598,7 @@ func wireLifecycleEvents(p *Provider) {
p.Glue.SetEventPublisher(p.EventBridge)
p.CloudWatch.SetEventPublisher(p.EventBridge)
p.SageMaker.SetEventPublisher(p.EventBridge)
p.Batch.SetEventPublisher(p.EventBridge)
}

// awsDrivers assembles the resource-discovery driver set from the provider's
Expand Down Expand Up @@ -639,7 +640,7 @@ func awsDrivers(p *Provider) *resourcediscovery.Drivers {
func (p *Provider) Tickables() []config.Tickable {
// SQS and EFS run before CloudWatch so the alarms evaluated in the same
// tick see the queue gauges and AWS/EFS metrics they just published.
return []config.Tickable{p.SQS, p.EFS, p.CloudWatch, p.SSM, p.CloudFormation}
return []config.Tickable{p.SQS, p.EFS, p.CloudWatch, p.SSM, p.CloudFormation, p.Batch}
}

// Close tears down any real engines wired into the provider via
Expand Down
49 changes: 37 additions & 12 deletions providers/aws/batch/batch.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
// Package batch provides an in-memory mock implementation of AWS Batch: the
// compute-environment, job-queue, and job-definition control plane. Resources
// provision synchronously (VALID/ACTIVE immediately), ARNs and ECS cluster ARNs
// are synthesized, and job-definition revisions increment on re-register.
// compute-environment, job-queue, job-definition and scheduling-policy control
// plane plus the job lifecycle. Resources provision synchronously
// (VALID/ACTIVE immediately), ARNs and ECS cluster ARNs are synthesized, and
// job-definition revisions increment on re-register. Jobs run no workload: a
// submitted job walks SUBMITTED -> RUNNABLE -> STARTING -> RUNNING ->
// SUCCEEDED on the configured clock (instantly unless AsyncSettle is on).
package batch

import (
Expand All @@ -11,6 +14,7 @@ import (
"sync"

"github.com/stackshy/cloudemu/v2/config"
"github.com/stackshy/cloudemu/v2/internal/awsevents"
"github.com/stackshy/cloudemu/v2/internal/idgen"
"github.com/stackshy/cloudemu/v2/internal/memstore"
"github.com/stackshy/cloudemu/v2/services/batch/driver"
Expand All @@ -24,6 +28,8 @@ const (
kindComputeEnvironment = "compute-environment"
kindJobQueue = "job-queue"
kindJobDefinition = "job-definition"
kindSchedulingPolicy = "scheduling-policy"
kindJob = "job"
)

// Mock is an in-memory implementation of AWS Batch. Each resource kind lives in
Expand All @@ -33,26 +39,45 @@ const (
// reads are copy-on-write (mutations build fresh values, never mutating stored
// ones in place).
type Mock struct {
computeEnvs *memstore.Store[driver.ComputeEnvironment]
jobQueues *memstore.Store[driver.JobQueue]
jobDefs *memstore.Store[driver.JobDefinition]
jobDefMaxRev *memstore.Store[int32]
computeEnvs *memstore.Store[driver.ComputeEnvironment]
jobQueues *memstore.Store[driver.JobQueue]
jobDefs *memstore.Store[driver.JobDefinition]
jobDefMaxRev *memstore.Store[int32]
schedPolicies *memstore.Store[driver.SchedulingPolicy]
jobs *memstore.Store[jobRecord]
jobSeq *memstore.Store[int64]

// registerMu serializes job-definition revision allocation so two concurrent
// RegisterJobDefinition calls for the same name can't claim the same revision.
registerMu sync.Mutex

// policyBindMu serializes binding a scheduling policy to a job queue
// (CreateJobQueue, UpdateJobQueue) against DeleteSchedulingPolicy, so a
// policy can't be deleted between the in-use check and a queue binding it.
policyBindMu sync.Mutex

// jobsMu serializes job submission (sequence allocation), cancel/terminate
// decisions and the event-emission bookkeeping on job records.
jobsMu sync.Mutex

// events publishes "Batch Job State Change" events to the EventBridge
// default bus once wired by the provider factory.
events awsevents.Emitter

opts *config.Options
}

// New creates a new AWS Batch mock with the given configuration options.
func New(opts *config.Options) *Mock {
return &Mock{
computeEnvs: memstore.New[driver.ComputeEnvironment](),
jobQueues: memstore.New[driver.JobQueue](),
jobDefs: memstore.New[driver.JobDefinition](),
jobDefMaxRev: memstore.New[int32](),
opts: opts,
computeEnvs: memstore.New[driver.ComputeEnvironment](),
jobQueues: memstore.New[driver.JobQueue](),
jobDefs: memstore.New[driver.JobDefinition](),
jobDefMaxRev: memstore.New[int32](),
schedPolicies: memstore.New[driver.SchedulingPolicy](),
jobs: memstore.New[jobRecord](),
jobSeq: memstore.New[int64](),
opts: opts,
}
}

Expand Down
6 changes: 4 additions & 2 deletions providers/aws/batch/batch_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -189,8 +189,10 @@ func TestRegisterJobDefinitionRevisionIncrements(t *testing.T) {
t.Fatalf("first revision/status wrong: %+v", first)
}

if len(first.PlatformCapabilities) != 1 || first.PlatformCapabilities[0] != "EC2" {
t.Fatalf("platformCapabilities default should be [EC2], got %v", first.PlatformCapabilities)
// No default is echoed when the caller sent none (it would drift in
// Terraform, which reads the list back as-is).
if len(first.PlatformCapabilities) != 0 {
t.Fatalf("platformCapabilities should be empty when none was sent, got %v", first.PlatformCapabilities)
}

second, err := m.RegisterJobDefinition(ctx, in)
Expand Down
4 changes: 4 additions & 0 deletions providers/aws/batch/compute_environment.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ func (m *Mock) CreateComputeEnvironment(
return nil, cerrors.New(cerrors.InvalidArgument, "computeEnvironmentName is required")
}

if err := validateTags(in.Tags); err != nil {
return nil, err
}

if in.Type != driver.CETypeManaged && in.Type != driver.CETypeUnmanaged {
return nil, cerrors.Newf(cerrors.InvalidArgument, "type must be MANAGED or UNMANAGED, got %q", in.Type)
}
Expand Down
Loading
Loading