[isadeks]

P3 lets an agent wait for human approval while its MicroVM sleeps or retires with a verified checkpoint. A later decision wakes the original worker or admits one replacement with the saved conversation and workspace. Unanswered approvals remain available by default; explicit deadlines remain supported.

Linear users can reply approve or deny under the approval comment. The service verifies the actual comment through Linear’s API, rejects bots and the saved OAuth token’s own identity, and applies the task owner’s decision to that request. CLI decisions work on all backends; sleep and replacement are MicroVM-specific.

Area

• cdk — infrastructure, handlers and permissions
• agent — runtime and image
• cli — task and approval settings
• docs — guides and design sources
• tooling — contracts, packaging and CI

Related

ADR-021 P3 for #645, including remaining P2 prerequisites. Related follow-ups: #817, #841 and #857. AgentCore remains the default compute backend.

Changes

• Add approval-aware suspend/resume supervision, guest activity barriers, explicit lifecycle connection closure, credential renewal and correlated failure diagnostics.
• Preserve unanswered requests, with cancellation feedback, a ten-minute default sleep delay and task/deployment controls.
• Verify versioned conversation/workspace checkpoints before retirement, confirm worker shutdown before releasing capacity, and restore the pending action on one admitted replacement.
• Move approval creation and non-human timeout writes behind a trusted IAM API. The worker’s session tag restricts its API path to its own task. Workers can read approval records but cannot overwrite human decisions, notification markers or retention fields. MicroVM writes also require the active worker lease.
• Require an explicit MicroVM layout: true for new/already-nested installations, false for existing flat installations. Omission fails synthesis. Flat-to-nested migration tooling is out of scope for this PR (follow-up). Group stateless scheduled maintenance in a separate nested stack while retaining existing tables and MicroVM identities. Both layouts are covered across all image, gateway and vault configurations by the 490-resource budget.
• Strengthen terminal cleanup, transaction-conflict diagnostics, SDK-version checks, bootstrap permissions and real DynamoDB transaction tests in CI. Preserve stored serialization bytes.
• Reuse the existing Linear AgentCore Identity vault integration for MicroVM workers.

Current head (6c52cc6a)

• Merged main (fix(registry): name the Provider waiter state machine under the stack prefix (#865) #911). The conflict was in the registry waiter state machine name: this PR took main's <root-stack>-AgentRegistryWaiter naming and dropped its own Names.uniqueResourceName version (only one registry exists per deployment). Full build passed: 5,669 CDK tests (256 suites), 2,241 agent tests (86.80%), 1,005 CLI tests, lint.
• Rewrote the author on six commits that a test fixture had recorded as t <t@t> / Test <test@example.test> (test(agent): git fixtures leak into the shared .git/config under an inherited GIT_DIR — make the isolation structural (4th recurrence of #622/#720) #855). Trees are unchanged.

Hook-safe test fixtures (bf72bdf5)

• agent/tests/test_continuation_runtime.py git fixtures now strip inherited GIT_* and disable hooks. Under a linked-worktree pre-commit hook they had written into the real repository and re-triggered the hook (test(agent): git fixtures leak into the shared .git/config under an inherited GIT_DIR — make the isolation structural (4th recurrence of #622/#720) #855). Agent quality: 2,241 passed, 86.80%.

Post-wake stall fix (3b16c1ed)

Post-wake stall fix, found by the approve/deny test on the redeployed fixture:

• After a suspend longer than ~6 minutes, the first model call could hang for 246–276 s. Connections idle across the suspend are dropped silently (the VPC endpoint/NAT idle timeout is 350 s), and Claude Code 2.1.191 waits 300 s for response headers before retrying. A control task with a short sleep did not stall.
• MicroVM workers now default API_TIMEOUT_MS=120000 (agent/src/server.py; test and agent/README.md updated). Streamed responses keep their own idle watchdog, so long generations are not cut off.
• Deployed to the fixture (MicroVM image 4.0 ACTIVE). A re-test with ~7 minutes asleep had no stall: the approved task finished 58 s after its decision (PR opened), the denied task 18 s after; no api_retry events and no gaps over 60 s after wake.

Scope reduction (03136489)

Scope reduction after the fourth review, with no behaviour change:

• Removed the test-only migration helpers (cdk/src/migration/* and their tests) and three unreferenced diagnostic probes under agent/scripts/.
• Moved the duplicated approve/deny post-commit audit and wake code into cdk/src/handlers/shared/approval-decision.ts.
• Reverted the region-scoped dashboard/OAC renames; they are unrelated to P3 and will go in a separate PR.
• Verification docs now list flat-to-nested migration tooling as a follow-up instead of an open PR check.
• Merged main (fix(observability): allow CloudWatch SNS alarm publishing (#925) #926, fix(deps): clear urllib3, brace-expansion, fast-uri osv advisories #932).

Net vs bf7c271d before the merges: 17 files, +166/−1,385. Full local build passed: 5,669 CDK tests (256 suites), 2,241 agent tests (86.8% coverage), lint and synth; the docs check and link check passed. GitHub checks passed at 10777d5e.

Redeployed 10777d5e to the retained nested fixture in us-east-1 from a repackaged artifact: CloudFormation UPDATE_COMPLETE, MicroVM image 3.0 ACTIVE, approve/deny functions have the concurrency table. A post-deploy no-approval smoke task completed on MicroVM in 210 s and opened a PR. Its worker terminated.

Review fixes (bf7c271d)

• Configure both CLI/API decision functions with the concurrency table and the shared admission limit; Linear's separate wiring had hidden this gap.
• Clean terminal, fenced replacements that never obtained a launch receipt, checking matching continuation/slot/lease identities and no worker ID. DynamoDB Local tests drive failure through cleanup and reject a worker ID injected before the conditional write. Unknown launch outcomes retain their existing lifetime protection; valid pending checkpoints do not expire.
• Reproduce project-hook denial on pinned SDK 0.2.110 / CLI 2.1.191 using a synthetic loopback model. Retire only the tool ID in the CLI error result; do not clear all registrations at Stop.
• Preserve valid credentials on advisory refresh failures while keeping forced wake refresh mandatory. Default MicroVM uv installs to copies after reproducing Linux cache hardlinks. Confirm termination conflicts with GetMicrovm before treating cleanup as successful.
• Correct remaining timeout/stranded-request documentation, document all three live harness commands, and cover heartbeat grace/staleness boundaries. The checkpoint-code table and approval API's non-semantic validation wording were already present.

Local full build passed with DynamoDB Local and real SDK continuation probes enabled: 5,758 CDK tests (261 suites), 2,254 agent tests (86.84% coverage), and 1,005 CLI tests, plus lint, type checks, synthesis and documentation. The final agent diagnostic change passed another complete 2,254-test quality run. Final layout compatibility and heartbeat checks passed 25 tests, with CDK lint/compile passing. The documentation build and internal links passed across 82 pages; all three live-harness inspection commands ran without AWS calls.

Deployed bf7c271d to the retained nested fixture in us-east-1: CloudFormation UPDATE_COMPLETE, MicroVM image 3.0, AgentCore runtime 4, and coordinator version 5. Four deployed Lambda code hashes match the reviewed assets; approve/deny functions have the concurrency table and shared limit. Verified 26 baseline stateful resource identities were preserved. Latest-head GitHub checks passed.

Both live Cognito-authenticated API decision cases passed after a saved checkpoint, sleep, source-worker loss and PARKED: approval and denial started replacement workers in 11.7s and 12.3s. Approval performed exactly one Read; denial performed none and acknowledged the decision. Both tasks completed, their workers terminated, and scheduled cleanup closed leases and returned capacity to zero. These are the API routes used by CLI decisions; no interactive CLI command was used. Tests used an explicit 30-second sleep delay with a test-only policy, leaving the ten-minute default unchanged. The fixture configuration was restored exactly; final checks found no active tasks, open leases or held slots. Cleanup follows the five-minute reconciliation schedule. This already-nested update does not verify flat-to-nested migration. Earlier results below retain their recorded revisions.

Scope and remaining tradeoffs

This PR includes the trusted approval writer, Linear replies, retained-request default, shared maintenance stack and MicroVM layout selection. Test-only migration helpers, unreferenced probes and the unrelated dashboard/OAC renames were removed to reduce scope. Maintainer agreement on this expanded scope and the proposed ADRs remains open. The @aws-sdk/client-ssm dependency supports the runtime suspend setting; build-workflow changes make DynamoDB Local and the pinned SDK recovery probes required in CI. These are intentional dependency/CI changes, not incidental cleanup.

Nonblocking optimization remains: eager transcript append repeatedly copies/encodes history (a local synthetic 1,000-frame run took about 5.5 seconds cumulatively). Replacing checkpoint readback with HeadObject must preserve checksum/version validation and recovery after a lost upload reply. Broad type/handler refactors and restored-scope revalidation are deferred rather than mixed into lifecycle corrections. Known launch rejections still share the conservative unknown-outcome capacity hold; earlier release needs a durable, operation-specific rejection classification.

Validation

Review validation at 159576e6 (eight focused commits after the third review):

• Complete final-head GitHub build passed: 5,718 CDK tests, 2,231 agent tests (zero skips; 86.66% coverage), and 1,005 CLI tests. Includes real DynamoDB transaction checks and the real SDK approve/deny recovery probe, explicitly enabled in the build workflow. The full local build also passed before the final worker-token validation restoration, which passed its focused tests and this final full CI run.
• Cancellation-race tests exercise both actual pipeline paths. Restoring the false failure behavior produces two failing tests; bypassing both checked terminal writes produces three. Internal-whitespace/case replay tests reject an over-broad normalizer.
• Expanded flat/nested resource-budget, cdk-nag and bootstrap coverage remains in the full suite. Both layouts retain their original budget; widest configuration is below 490 resources.
• Final correction restores the original worker_attempt_id validator; 22 broker unit tests passed. Source tracing shows this field carries the platform launch/lease token, not AWS’s physical MicroVM identifier.
• A live probe using the worker's actual AgentCore Identity vault token successfully edited its own control comment, but Linear rejected editing a human-authored deny comment. The human body and author stayed unchanged. This verifies the current app-token installation; no historical approval threads were modified.
• The documentation build validates internal pages and anchors across 82 rendered pages, plus local existence of emitted GitHub repository-file targets. An injected missing repository target is rejected.
• Security/dependency/workflow, dead-code and PR-title checks passed at 159576e6. The silent-failure scan and its rule tests pass. Lint, type checks, contracts, documentation and infrastructure synth pass. The full build and all PR checks passed at that revision.
• Deployed 159576e6 successfully to the retained isolated fixture after pausing submissions and draining tasks, leases and capacity. Verified the new approval API, AgentCore runtime version 3, and MicroVM image 2.0 against the built image/artifact references. Seven live IAM simulation checks passed: own-task invocation/read allowed; other-task or untagged invocation and direct approval writes denied. The borrowed Linear vault and shared logging settings remain unchanged and functional.
• All six latest-image Linear coding-agent flows passed: AgentCore approve/deny, MicroVM same-worker wake approve/deny, and MicroVM replacement-worker recovery approve/deny. Verified actual owner-authored replies, matching decision sources and thread acknowledgments, approved reads exactly once, denied reads never succeeding, continued agent execution, terminated MicroVMs and scheduled capacity release. All four MicroVMs waited the full default 600 seconds before suspension; normal wake cases were observed asleep for over a minute. Replacement cases used controlled loss of a suspended worker with a verified checkpoint, then restored on different workers. No approval or continuation rows were manufactured. A separate automated wake probe measured 69.031 seconds between the suspend response and resume hook, inside the earlier 5–90 second failure window; resume returned HTTP 200 in 143 ms. That task also completed on the same worker. All seven test slots are released and active capacity is zero.

Live stress validation and capacity fix at 7384f9b0:

• A 12-task burst respected the 10-task cap, queued overflow, handled queued/startup cancellation and deliberate worker termination, and passed 50 concurrent status reads. Nine ordinary workloads completed without approval gates.
• Stress testing exposed simultaneous slot releases failing with DynamoDB TransactionConflict. Bounded retries preserve the exact transaction, idempotency token and worker-lease conditions. An isolated live reproduction went from 6/12 successful releases to 12/12; concurrent admission still accepted exactly 10/12. All 71 focused unit/transaction tests and the full local build passed: 5,723 CDK, 2,231 agent, 1,005 CLI tests.
• Deployed the fix and verified the actual Lambda packages. Two workers each passed 1 GiB memory, 512 MiB disk and 45 CPU-second benchmarks; a third was cancelled during its Python workload. All test workers terminated, leases closed and capacity returned to zero. The patched cleanup sweep reported no failures. Tests and reproducible harnesses remain private; no benchmark PRs were published.

The later main merge at 2a79c0e retains this fix; its CI is separate from the live deployment above.

Earlier live approval-boundary validation used the actual approval service and Python signing client: 34 AWS checks covered creation, field rejection, task/tag isolation, denied direct writes, timeout races and MicroVM lease fencing; nine IAM simulation checks also passed. These were isolated API tests with task records for all three backends, without coding-agent images.

Earlier coding-agent tests additionally covered explicit expiry/cancellation and AWS access after credential expiry. Those earlier configurations remain separate from the latest-image cases above. Retained Linear fixtures and reproducible integration harnesses stay outside this public PR.

Before marking ready

• Review-baseline GitHub checks complete (159576e6); full local build and live stress validation passed for capacity fix 7384f9b0.
• GitHub checks passed for the subsequent main merge (2a79c0e).
• GitHub checks passed for the nested-default change (35f05bbb).
• Complete GitHub checks for the review-fix head (bf7c271d).
• Complete GitHub checks for the scope-reduction head (10777d5e).
• Obtain maintainer acceptance of expanded feat(compute): add AWS Lambda MicroVMs as a ComputeStrategy backend #645 scope, ADR-021/ADR-023 and contributor assignment.
• Verify API approve/deny decisions on PARKED tasks in the latest review-fix deployment (the same routes used by CLI).
• Deploy and verify the trusted approval service with matching worker images after draining existing tasks.
• Complete the six latest-image Linear coding-agent approval/recovery cases.
• Complete the compatibility review: retain version-1 checkpoint field names and the distinct migration/runtime hash encodings so saved data remains readable.
• Out of scope: reusable flat-to-nested migration commands and an independently verified upgrade from current main are a follow-up. Until then, existing flat installations keep microvm_nested_stack=false.

Review decisions and limits

ADR-023 records the approval writer, rollout and retained-request decision. No default approval expiry is deliberate; zero is the no-expiry sentinel and the positive timeout range starts at 30 seconds. Worker/resource lifetime remains separate. A public SigV4 endpoint is intentional, with task-path IAM authorization.

This protects approval-record writes, not complete isolation from compromised ambient compute credentials: those credentials can still select session tags, and legacy channel bundles still include the webhook signing key. Action descriptions and policy metadata are worker assertions. TIMED_OUT currently also includes worker polling failures and is not proof that a human deadline elapsed. These limits are documented rather than presented as solved by the writer API.

The normal pipeline propagates failed terminal writes, while a superseding cancellation is benign and does not produce a false failure reaction. Actual pipeline entry points and the outer runtime's unhealthy response after a failed backup write are tested. Pure transaction conflicts get bounded retries; failed ownership checks do not. The continuation scan cursor uses compare-and-set to preserve progress when old and new schedules overlap during upgrade. Existing orchestrator heartbeat/backend monitoring handles lost running workers; a second broad RUNNING reaper was not added. Checkpoint field renames or hash-encoding consolidation require a versioned migration, not a style-only edit.

Deployment notes

Re-bootstrap to bundle 1.9.0 or later. For the approval permission change, pause submissions and drain existing tasks, including suspended workers and retained continuations; deploy CDK and the matching rebuilt image together. Old workers lose direct approval-write access and fail closed at their next gate. See the upgrade procedure.

Automatic suspension remains behind the deployment compatibility gate, which defaults off. Once enabled, the default sleep delay is ten minutes. MicroVM deployments must now explicitly select microvm_nested_stack: use true for new/already-nested installations and retain false for existing flat installations. Omission fails synthesis. Explicitly selecting true does not migrate a flat deployment. Regenerate old cloud assemblies before deploying. This PR does not include flat-to-nested migration tooling; that is a follow-up.

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution under the terms of the project license.

[isadeks]

@ayushtr-aws, thanks for the review at 548bdf48. Here's where each item stands at 33bdd809:

Blocking

• B1 (scope/governance): I reduced scope without splitting the PR. I removed the test-only cdk/src/migration/* helpers, three unreferenced diagnostic probes and the unrelated region-scoped dashboard/OAC renames, which will go in a separate PR. The approval-writer API, Linear replies, the approval_timeout_s 300 → 0 default and the ConcurrencyMaintenance stack are still in. Issue approval, assignment and ADR-021/ADR-023 acceptance are still waiting on maintainers; I'm happy to file separate issues for those three areas if you'd like to track them that way.
• B2 (nested by default): done as you suggested. Synth now throws when compute_type=lambda-microvm and microvm_nested_stack is unset (agent.ts:356-367). Migration tooling is no longer claimed by this PR: the verification docs and PR checklist list it as a follow-up, and existing flat installations keep false.
• B3 (approve/deny concurrency env): fixed (task-api.ts:1467-1468), with a CDK assertion on both decision functions' environment (microvm-continuation-manager.test.ts). Verified live: CLI/API approve and deny on PARKED tasks started replacements in about 12 s, and the redeployed functions have the variable.
• B4 (FENCED lease never launched): fixed. The reconciler treats ACTIVE/FENCED leases with no lease_microvm_id as never-launched, so cleanup and deleteClosedTaskContinuations now run, and DynamoDB Local tests cover it. I did not add the suggested defence-in-depth object expiration to the continuation bucket. Valid pending checkpoints must not expire, and the only lifecycle rule is the multipart-abort rule. Say if you'd still like a long-horizon expiration.
• B5 (second PreToolUse denier): I reproduced it on pinned SDK 0.2.110 / CLI 2.1.191. The fix (0352a7c8) retires only that tool ID when the SDK returns the error result, rather than clearing every registration at Stop. test_runner.py covers it.

Non-blocking, addressed: approve/deny duplication (now shared/approval-decision.ts), UV_LINK_MODE=copy default for MicroVM tasks, advisory STS refresh no longer forced mandatory, Terminate ConflictException confirmed via GetMicrovm, stale 300 s / built-in-timeout docs, checkpoint code table, a runner for cdk/test/live/*, heartbeat boundary tests, and the @aws-sdk/client-ssm / build.yml callouts in the PR body.

Non-blocking, deferred: the StartReceipt/ContinuationRecord type refactors, a no_deadline_sentinel contract field, eager transcript append (O(n²)), HeadObject readback, and early seat release on known launch rejections. These are listed under "Scope and remaining tradeoffs" in the PR body.

Validation:

• Full local build: 5,669 CDK and 2,241 agent tests passed, plus lint and synth.
• GitHub checks passed at 4a91db01.
• Redeployed to the nested fixture with MicroVM image 3.0. Live approve/deny wakes and an 11-task stress burst passed earlier on bf7c271d with no leaked VMs. A post-deploy no-approval smoke task completed on MicroVM in 210 s and opened a PR. Its worker terminated.

Ready for re-review when you have time.