Add suspended warm server pools

.github/workflows/ci.yml

@@ -70,4 +70,14 @@ jobs:
               docker compose version
               aws --version
               aws lambda-microvms help >/dev/null
+              for node_root in \
+                /opt/actions-runner/externals/node20 \
+                /opt/actions-runner/externals/node24; do
+                test "$("${node_root}/bin/node" \
+                  "${node_root}/lib/node_modules/npm/bin/npm-cli.js" \
+                  --version)" = "11.18.0"
+                test "$("${node_root}/bin/node" -p \
+                  "require(\"${node_root}/lib/node_modules/npm/node_modules/tar/package.json\").version")" = \
+                  "7.5.19"
+              done
             '

CHANGELOG.md

@@ -2,6 +2,19 @@
 
 ## Unreleased
 
+- Added experimental warm MicroVM reuse with one `server` lifecycle input and
+  output, fresh JIT registration per job, explicit no-DynamoDB resume, and
+  DynamoDB-backed cross-workflow pools with fenced leases.
+- Added request-local `server-capacity`, on-access reconciliation without a
+  scheduled garbage collector, `warm-hit` and deadline outputs, and a
+  `max-lifetime-seconds` input defaulting to two hours and capped at eight.
+- Added authenticated warm supervisor control, snapshot-safe Docker suspension,
+  and production `overlay2` validation with a copy-on-write `fuse-overlayfs`
+  fallback before the existing final `vfs` fallback.
+- Accept AWS's bodyless suspend, resume, and terminate lifecycle hook requests
+  while retaining strict request bodies for the run hook.
+- Quickstart now creates and configures an on-demand warm-state table and grants
+  its exact data-plane and MicroVM lifecycle permissions; teardown removes it.
 - Guarded Quickstart teardown script that previews by default and deletes
   generated GitHub repository config plus AWS resources with `--yes`.
 - Default runner image memory reduced to 2 GiB, plus a container-job and Redis
@@ -11,8 +24,8 @@
 - Deterministic launch idempotency and quota-aware retries/polling.
 - Partial-failure and explicit cleanup.
 - Snapshot-safe AL2023 ARM64 runner image with Docker, Buildx, and Compose.
-- Lifecycle supervisor with fresh Docker startup, automatic production `vfs`
-  fallback, and self-termination.
+- Lifecycle supervisor with fresh Docker startup, automatic production
+  `fuse-overlayfs` and `vfs` fallbacks, and self-termination.
 - Direct AWS CLI bootstrap, image build tooling, CI, release SBOMs, and
   examples.
 - One-command Classic PAT and static IAM-user Quickstart, with OIDC and GitHub

README.md

@@ -90,7 +90,7 @@ jobs:
           image-version: ${{ vars.MICROVM_RUNNER_IMAGE_VERSION }}
           execution-role-arn: ${{ vars.MICROVM_EXECUTION_ROLE_ARN }}
           cloudwatch-log-group: ${{ vars.MICROVM_RUNTIME_LOG_GROUP }}
-          maximum-duration-seconds: "3600"
+          max-lifetime-seconds: "3600"
 
   job:
     needs: start-runner
@@ -123,6 +123,26 @@ The start job emits a unique label for one target job. The runner is JIT-only
 and single-use. Its supervisor self-terminates after that job; the explicit stop
 job and platform maximum duration are independent cleanup backstops.
 
+### Experimental warm cache
+
+Warm mode reuses the MicroVM and its local Docker cache while creating a fresh
+JIT runner for every job. Set a human-readable `server` name on `start`, pass
+the opaque `server` output to `stop`, and configure the Quickstart-created
+`MICROVM_WARM_STATE_TABLE` for reuse across workflow runs. The informational
+`warm-hit` output reports whether an existing member was reused.
+
+> [!WARNING] A reused machine is a cache, not an isolation boundary. Enable warm
+> mode only for trusted workflows in the same private repository. Fork pull
+> requests are rejected. The Action uses authenticated MicroVM control traffic,
+> conditional DynamoDB leases, and the platform lifetime as its natural cleanup
+> backstop; it does not run a scheduled garbage collector.
+
+See [the warm-cache example](examples/warm-cache.yml) and the
+[implementation and testing design](docs/warm-cache.md). `server-capacity` is an
+optional request-local creation bound: an available member always wins; if all
+members are busy, omission permits another member, while a supplied bound fails
+once the current active count reaches it.
+
 ## Status
 
 The Action implements and tests:
@@ -136,10 +156,11 @@ The Action implements and tests:
 - typed GitHub and AWS adapters with mocked-boundary integration tests.
 
 The production AL2023 runner image is implemented and validated locally and
-through the AWS image build hooks. Docker prefers `overlay2` and always falls
-back to `vfs` if needed. The complete private-repository workflow is validated
-for success, job failure, cancellation, startup timeout, service containers, and
-the maximum-duration backstop.
+through the AWS image build hooks. Docker prefers `overlay2`, falls back to the
+copy-on-write `fuse-overlayfs`, and retains `vfs` as the final compatibility
+fallback. The complete private-repository workflow is validated for success, job
+failure, cancellation, startup timeout, service containers, and the
+maximum-duration backstop.
 
 ## Development
 
@@ -157,8 +178,8 @@ artifact directly.
 
 Version 1 is ARM64, JIT-only, repository-scoped, and intended for private
 repositories with trusted workflow changes. It has no webhook, queue,
-dispatcher, warm pool, shell ingress, persistent runner, or boot-time package
-installation.
+dispatcher, shell ingress, persistent GitHub runner registration, or boot-time
+package installation. Warm MicroVM reuse is experimental and opt-in.
 
 Detailed guides:
 
@@ -167,9 +188,15 @@ Detailed guides:
 - [security model](docs/security.md)
 - [operations and quotas](docs/operations.md)
 - [testing and release gates](docs/testing.md)
-- [proposed warm-cache implementation and testing plan](docs/warm-cache.md)
+- [warm-cache implementation and testing plan](docs/warm-cache.md)
 - [runner image](runner-image/README.md)
 
+### Performance evidence
+
+Benchmark harnesses, raw measurements, and research notes are maintained
+separately from the Action's product code. Public articles and reproducible
+summaries will be linked here as they are published.
+
 ## Credits and inspirations
 
 This project is a small, purpose-built variation on existing runner and Lambda

action.yml

@@ -33,11 +33,15 @@ inputs:
     description: Comma-separated additional runner labels
     required: false
     default: lambda-microvm,docker
+  max-lifetime-seconds:
+    description:
+      Platform-enforced MicroVM lifetime from 1 through 28,800 seconds; defaults
+      to 7,200 seconds
+    required: false
   maximum-duration-seconds:
     description:
-      Platform-enforced MicroVM lifetime from 1 through 28,800 seconds
+      Deprecated alias for max-lifetime-seconds; do not provide both inputs
     required: false
-    default: "3600"
   startup-timeout-seconds:
     description: Overall AWS and GitHub runner readiness timeout
     required: false
@@ -47,9 +51,10 @@ inputs:
     required: false
     default: INTERNET_EGRESS
   ingress-connectors:
-    description: JSON array or comma-separated ingress connector identifiers
+    description:
+      JSON array or comma-separated ingress connectors; defaults to NO_INGRESS
+      for ephemeral runners and ALL_INGRESS for warm control traffic
     required: false
-    default: NO_INGRESS
   cloudwatch-log-group:
     description: Optional CloudWatch log group for MicroVM runtime logs
     required: false
@@ -60,6 +65,26 @@ inputs:
   microvm-id:
     description: MicroVM identifier returned by start mode
     required: false
+  server:
+    description:
+      Warm pool name on start, or the opaque server value returned by start
+    required: false
+  server-capacity:
+    description:
+      Optional request-local ceiling when all members of a warm pool are busy
+    required: false
+  state-table:
+    description: DynamoDB table for cross-workflow warm pool discovery
+    required: false
+  lease-timeout-seconds:
+    description:
+      Recovery deadline for an abandoned warm lease; defaults to the MicroVM max
+      lifetime
+    required: false
+  reuse-safety-margin-seconds:
+    description: Minimum lifetime reserved before a MicroVM can be reused
+    required: false
+    default: "1800"
   debug:
     description: Enable non-secret diagnostic metadata
     required: false
@@ -78,6 +103,14 @@ outputs:
     description: Launch Region
   image-version:
     description: Resolved MicroVM image version
+  server:
+    description: Opaque value for resuming or releasing this exact warm lease
+  warm-hit:
+    description: Whether start reused an existing warm MicroVM
+  warm-expires-at:
+    description: Platform lifetime deadline as an ISO-8601 timestamp
+  reuse-deadline:
+    description: Last reuse deadline as an ISO-8601 timestamp
 
 runs:
   using: node24

docs/installation.md

@@ -51,11 +51,13 @@ log groups. Keep those generated files until teardown is complete.
 The script:
 
 1. uses the active local AWS credentials to create or reconcile the S3 bucket,
-   CloudWatch log groups, and image build and runtime IAM roles;
+   CloudWatch log groups, the on-demand warm-state DynamoDB table, and image
+   build and runtime IAM roles;
 2. packages, uploads, validates, and activates the runner image;
 3. configures the repository variables;
 4. creates a dedicated `lambda-microvm-github-runner-quickstart` IAM user;
-5. grants that user only image build and runner lifecycle permissions;
+5. grants that user only image build, runner lifecycle, and exact-table
+   warm-state data-plane permissions;
 6. rotates its access key directly into the `AWS_ACCESS_KEY_ID` and
    `AWS_SECRET_ACCESS_KEY` GitHub Actions secrets;
 7. sets the PAT as `GH_PERSONAL_ACCESS_TOKEN`.
@@ -80,6 +82,12 @@ For GitHub Actions job containers and service containers, copy
 steps inside a Node 24 container and verifies access to a Redis service
 container from that job container.
 
+For experimental cache reuse, copy
+[the warm-cache example](../examples/warm-cache.yml). The setup script already
+sets `MICROVM_WARM_STATE_TABLE`. Warm mode is only for trusted workflows in the
+same private repository: jobs sharing a server name share root-equivalent
+machine state. It is not an isolation boundary.
+
 ## Verify
 
 Run the workflow manually and confirm:
@@ -90,6 +98,10 @@ Run the workflow manually and confirm:
 4. the MicroVM reaches `TERMINATED`;
 5. no GitHub token, AWS credential, or JIT payload appears in logs.
 
+For warm mode, additionally confirm that `stop` reaches `SUSPENDED`, a later
+`start` reports `warm-hit: true`, the MicroVM ID is unchanged, and the JIT
+runner ID is new.
+
 ## Advanced credentials
 
 For short-lived credentials, use GitHub OIDC for AWS and a GitHub App

docs/operations.md

@@ -18,8 +18,8 @@ scripts/teardown-quickstart.sh --yes
 The script requires the generated `build/aws-setup.json` file and uses
 `build/microvm-image.json` when present. It deletes only the resource names and
 ARNs recorded there: repository secrets and variables, the dedicated Quickstart
-IAM user, project IAM roles, the MicroVM image, the versioned artifact bucket,
-and build/runtime log groups.
+IAM user, project IAM roles, the MicroVM image, the warm-state table, the
+versioned artifact bucket, and build/runtime log groups.
 
 ## Quickstart credential rotation
 
@@ -59,6 +59,20 @@ exhaustion. The defaults keep a simulated 200 simultaneous starts within the 100
 quota, request a quota increase or shape GitHub workflow concurrency. Do not add
 an internal queue to this product.
 
+Warm members consume regional MicroVM memory quota while running or suspended.
+`server-capacity` is a request-local creation ceiling, not persistent pool
+configuration: an available member is always reused; if all are busy, an omitted
+bound permits growth and a supplied bound rejects creation once the active count
+reaches it. Mixed bounds are intentionally allowed and the largest successful
+request can grow the pool. Use GitHub concurrency controls when a repository
+needs a stable operational bound.
+
+`max-lifetime-seconds` defaults to 7,200 and cannot exceed Lambda's
+28,800-second limit. A member inside the configured safety margin is terminated
+instead of reused. Untouched members naturally expire at their platform
+deadline; the next warm action reconciles stale table state. No scheduled
+garbage collector or table scan is required.
+
 ## Logs
 
 The AWS bootstrap creates build and runtime log groups with retention. Action
@@ -81,6 +95,11 @@ Investigate runners near their maximum duration and terminate confirmed orphans.
 Alert on repeated self-termination failures or VMs consistently reaching the
 duration backstop.
 
+For warm pools, inspect the exact `MICROVM_WARM_STATE_TABLE` partition when a
+pool reports no capacity. Do not edit lease IDs or generations manually while a
+workflow may still own them. Teardown removes the whole project table; preview
+its resource list before using `--yes`.
+
 ## Image updates and rollback
 
 The build script activates a version only after its `/ready` and `/validate`
@@ -107,7 +126,13 @@ version is active.
   status.
 - runner remains offline: inspect `/run`, Docker, DNS, and GitHub egress logs;
   start cleanup should terminate the VM.
-- Docker validation failure: inspect the supervisor log for both `overlay2` and
-  `vfs` startup failures. `vfs` is the automatic production fallback.
+- Docker validation failure: inspect the supervisor log for `overlay2`,
+  `fuse-overlayfs`, and `vfs` startup failures. `vfs` remains the final
+  production fallback.
+- `vfs` out of space: the fallback copies complete filesystem layers and is
+  substantially more storage-intensive than `overlay2`. On the 2 GiB runner,
+  prefer smaller base images and bounded caches; a large Node image plus service
+  images can exhaust the snapshot filesystem even though the same workload fits
+  with `overlay2`.
 - self-termination denied: correct the runtime role; the explicit stop job and
   maximum duration remain active.

docs/security.md

@@ -13,8 +13,9 @@ and trusted workflow changes only. Public fork pull requests are unsupported.
   resources.
 - Quickstart stores a classic PAT with `repo` scope and the dedicated IAM user's
   access key as GitHub Actions secrets. That IAM user can use the configured
-  image bucket, pass only the exact build/runtime roles, build images, and
-  manage runner MicroVMs. It cannot create or modify IAM resources.
+  image bucket, pass only the exact build/runtime roles, build images, manage
+  runner MicroVMs, and access the exact warm-state table. It cannot create or
+  modify IAM resources or DynamoDB tables.
 - Quickstart is limited to private repositories with trusted workflow changes.
   Rotate or delete both credentials when they are no longer needed.
 - Advanced setup uses a short-lived GitHub App installation token and obtains
@@ -35,9 +36,19 @@ the MicroVM.
 
 ## Network
 
-Normal launches use managed `NO_INGRESS` and `INTERNET_EGRESS` connectors. There
-is no public endpoint, shell token, or inbound debug path. Private VPC
-connectors require a separate network and IAM review.
+Normal launches use managed `NO_INGRESS` and `INTERNET_EGRESS` connectors. Warm
+launches require managed `ALL_INGRESS` so the Action can reach the dedicated
+control port. Every control request uses a short-lived AWS MicroVM auth token
+scoped to that port; the supervisor validates its bounded, versioned payload.
+The lifecycle-hook port and shell access are not exposed by this interface.
+Private VPC connectors require a separate network and IAM review.
+
+Warm reuse is intentionally a weaker isolation boundary: a trusted job has
+root-equivalent control through Docker and can poison files, images, caches, or
+memory consumed by a later job. A fresh GitHub JIT registration prevents stale
+scheduling but does not make the reused machine clean. Warm mode rejects
+fork-originated pull requests and must not be used with untrusted workflow
+changes.
 
 ## Cleanup
 
@@ -48,6 +59,13 @@ Each execution has independent backstops:
 3. start cleans up partial launches and unused JIT runners;
 4. Lambda enforces `maximumDurationInSeconds`.
 
+Warm mode changes the first two layers: runner exit returns the supervisor to
+idle, and `stop` suspends the owned lease (or terminates it at its reuse
+deadline). DynamoDB conditional generations fence stale owners. On-access
+reconciliation repairs abandoned metadata, DynamoDB TTL eventually deletes old
+items, and Lambda's maximum lifetime remains the resource cleanup backstop.
+There is no scheduled garbage collector.
+
 ## Supply chain
 
 - the Lambda AL2023 base is pinned by digest;