Skip to content

fix(aws-lambda): serve the previous revision during InProgress updates; align alias/version not-found behaviour - #1683

Draft
Satyam-Trivedi-ZS wants to merge 7 commits into
stackshy:developmentfrom
Satyam-Trivedi-ZS:fix/aws-lambda-followups-1665
Draft

Satyam-Trivedi-ZS wants to merge 7 commits into
stackshy:developmentfrom
Satyam-Trivedi-ZS:fix/aws-lambda-followups-1665

Conversation

@Satyam-Trivedi-ZS

@Satyam-Trivedi-ZS Satyam-Trivedi-ZS commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #1646, so merge that first. Until #1646 merges, this diff also shows its commits; this PR's own changes are the commits after dca3dcf1.

Summary

  • Under --async-settle, Invoke of $LATEST now keeps running the previous code and configuration until an update completes. That is what the Lambda function states page says. Before this change the new code ran as soon as UpdateFunctionCode returned.
  • A version cut by UpdateFunctionCode --publish now reports Pending/Creating while the update is in progress, then Active/Successful.
  • DeleteAlias, UpdateAlias and DeleteFunction --qualifier now match real Lambda's not-found behaviour.
  • Invocations that race code updates under --async-settle no longer fail with ServiceException: a deployment is removed only once no invocation can still be using it.
  • $LATEST reports LastUpdateStatusReason/LastUpdateStatusReasonCode while an update is InProgress.

Part of #1665

Item status

# Item Status Evidence
1 (M) Invoke during an InProgress update runs the new code FIXED Lambda function states: "Invocations continue to use the function's previous code and configuration until the update completes."
2 (L) Version cut by UpdateFunctionCode --publish reports Active/Successful at once FIXED LocalStack's AWS-validated TestLambdaVersions::test_publish_with_update (last validated against AWS on 2025-11-25). The update-function-code --publish response and get-function --qualifier 1 both show Version: 1, State: Pending, StateReason: "The function is being created.", StateReasonCode: Creating, with no LastUpdateStatus. $LATEST is Active/InProgress at the same time. snapshot, validation dates
3 (L) DeleteAlias on a missing alias returns ResourceNotFoundException FIXED (now 204) The DeleteAlias API reference does not list ResourceNotFoundException. LocalStack's AWS-validated TestLambdaAlias::test_non_existent_alias_deletion (validated 2025-11-25) records HTTP 204 for a missing alias on an existing function. It was added in localstack#11878 ("AWS doesn't throw an error for delete_alias for non-existing alias"). The Go SDK has no modeled ResourceNotFoundException for DeleteAlias either.
4 (L) Not-found wording for UpdateAlias, DeleteAlias, DeleteFunction --qualifier FIXED where real behaviour is recorded; one case UNVERIFIABLE UpdateAlias: ResourceNotFoundException: Alias not found: <function ARN>:<alias> (test_non_existent_alias_update, validated 2025-11-25). DeleteAlias: no error text, since a missing alias is 204 (item 3). DeleteFunction --qualifier <missing numbered version>: real Lambda returns success and repeating it is idempotent; a missing function is Function not found: <unqualified function ARN>, which is already the emulator's text (TestLambdaFunction::test_delete_on_nonexisting_version, validated 2025-11-25). UNVERIFIABLE: a non-numeric qualifier that is neither a version nor an alias. No recording exists, so it stays ResourceNotFoundException; only its text moved to the same Function not found: <qualified ARN> form as the other not-found errors.

The real-AWS responses above come from LocalStack's parity suite. Those tests are marked @markers.aws.validated and are recorded against real AWS. I did not call real AWS myself.

Changes

fix(aws-lambda): keep serving the previous revision while an update is InProgress (d7ae180)

  • providers/aws/lambda:
    • Under AsyncSettle, commitUpdate saves the function as it was before the update in fd.prior: its configuration, AWS-only settings and engine deployment. While the update window is open, Invoke of $LATEST runs fd.prior.
    • Published versions are immutable and are never swapped.
    • Default mode never sets fd.prior, so nothing changes there.
  • Engine (real-runtime) path:
    • Under AsyncSettle, a code update deploys to a fresh engine key, <name>.update-<id>. Lambda names cannot contain ., so the key cannot collide with another function's key. The old deployment keeps serving until the window closes.
    • A deployment that no longer serves anything is listed in fd.retired and removed at the next code update or on delete. Both run while the function's engine reservation is held.
    • The delete-during-deploy path removes every key the deleted function held.
    • In default mode the code is still redeployed in place under the function name.
  • Locking:
    • Engine Deploy/Remove still never run under m.mu. UpdateFunction calls finalizeCodeUpdate, which takes m.mu, commits, and returns the stale keys; they are removed after unlock and before the reservation is released.
    • Every new funcData read-modify-write (keepPrior, takeRetired) runs under m.mu.
    • Invoke changes only its own copy of funcData. prior is immutable once stored, and retired is never appended in place.
  • Persistence: the live engine key (engineName) is in the snapshot. prior is not saved, like the existing settle windows, so a restored function serves its current revision. (The retired keys are persisted by a later commit, see below.)
  • Published versions: a version cut by PublishVersionOnWrite while the update window is open copies that window. It reports Pending/Creating until the update completes and then stays Active/Successful through later $LATEST updates. The copy comes from a new settle.Set.Window accessor, so the version and $LATEST share the same deadline.
  • config/engine.go: the FunctionDeployment.Name doc now mentions the AsyncSettle update key.

fix(aws-lambda): match real not-found behaviour for DeleteAlias, UpdateAlias and DeleteFunction --qualifier (2dc7124)

  • DeleteAlias: a missing alias is a no-op success. A missing function is still ResourceNotFoundException.
  • UpdateAlias: the error text is now Alias not found: <function ARN>:<alias>.
  • DeleteVersion (the DeleteFunction --qualifier path): a missing numbered version is a no-op success.
  • Two existing tests asserted the old emulator behaviour, TestDeleteAlias/alias not found and TestSDKDeleteFunctionMissingVersion. Both now assert the recorded AWS behaviour.

fix(aws-lambda): keep a deployment until the invocations that picked it finish (d0f99f2), from review:

  • priorRevision now carries the settle.Window of the update that replaced it, and servePrior checks only that window instead of the shared settle set, so an unlocked funcData copy is self-consistent.
  • Invocations pin the engine key they run on (deployPins in pins.go, a per-key count with a sync.Cond). After pinning, pinDeployment re-reads the function and checks it still serves that key (live or prior); if a code update retired it in the meantime, it refreshes the copy and retries. removeDeployments waits for a key's pins to drain before Remove. Since the update stores the function without the key before it drains, a pin that passes the check holds a deployment that stays until the invocation ends.
  • Locking is unchanged: engine Deploy/Remove and the drain run outside m.mu (under the function's engine reservation), and every funcData read-modify-write is still under m.mu. Invoke never takes m.mu.
  • setVersionLifecycle holds the published version's lifecycle fields. It keeps publishVersion at cyclomatic complexity 10 and reads the update window once.

fix(aws-lambda): report the update reason and keep retired deployments across a restore (461b606), from review:

  • $LATEST reports LastUpdateStatusReason: "The function is being created." and LastUpdateStatusReasonCode: Creating while an update is InProgress, and omits both once it is Successful. This matches all 36 InProgress responses in LocalStack's AWS-validated snapshots. A published version never carries them. Two optional fields were added to driver.FunctionInfo; go generate output is unchanged.
  • The snapshot now persists the retired engine keys, with the prior revision's key folded in, so after a restore the next code update or DeleteFunction removes those deployments.
  • DeleteFunction --qualifier <unknown non-numeric> uses the Function not found: <qualified ARN> text.

The business logic is in providers/aws/lambda. server/aws/lambda only renders the two new response fields.

Provider Coverage

  • AWS
  • Azure: not applicable. The in-progress window, published versions and these alias semantics are Lambda-specific.
  • GCP: not applicable, same reason.
  • OCI: not applicable.

Checklist

  • All tests pass: the full go test -p 1 run over all 660 packages, in chunks of 15, no failures; plus go test -race -p 1 on providers/aws/lambda/..., server/aws/lambda/..., server/aws/, persist/..., internal/settle/... and config/...
  • Linter: golangci-lint v2.14.0, full run (no --new-from-rev) on providers/aws/lambda, server/aws/lambda, internal/settle and config: 28 issues on this branch and 28 on the parent dca3dcf1, the same pre-existing set (26 goconst, 1 gocritic in policy.go, 1 wsl in execlogs.go). No new findings, and no function above gocyclo 10.
  • Every provider the change applies to implements the same behavior (AWS only, see above)
  • Integration tests added to cloudemu_test.go: not added; the provider tests and SDK wire tests below cover it
  • Unit tests added to provider test files
  • Regenerated docs: go generate ./... leaves no diff. No driver interface or operation changed; the additions are settle.Set.Window in internal/ and two optional fields on driver.FunctionInfo

Test Plan

First-round tests. Each wire test fails on the old parent f6700ed6 and passes on this branch. I checked this by running the new SDK test files against a f6700ed6 checkout.

  • server/aws/lambda/parity_1665_sdk_test.go, using the real aws-sdk-go-v2 client:
    • TestSDKInvokeRunsPreviousCodeWhileUpdateInProgress: runs with async=true and async=false, using an engine that returns its deployed code. Before the fix: invoke right after the update = "new", want "old".
    • TestSDKUpdateFunctionCodePublishVersionIsPending. Before the fix: state:Active ... lastUpdateStatus:Successful, but Pending/Creating is expected.
  • server/aws/lambda/alias_notfound_sdk_test.go: TestSDKAliasNotFoundParity. Before the fix: DeleteAlias gave 404, and UpdateAlias gave the message alias non-existent not found.
  • server/aws/lambda/qualified_name_sdk_test.go: TestSDKDeleteFunctionMissingVersion, updated. Before the fix it got a 404.
  • providers/aws/lambda/parity_1665_test.go:
    • TestInvokeRunsPreviousCodeWhileUpdateInProgress: old code during the window, new code after it. A second update retires the original deployment, and delete removes all deployments.
    • TestInvokeRunsNewCodeImmediatelyByDefault: default mode deploys in place.
    • TestInvokeUsesPreviousConfigurationWhileUpdateInProgress: the previous Timeout and DeadLetterConfig are used for $LATEST only.
    • TestEngineNameSurvivesSnapshot.
    • TestVersionPublishedWithUpdateIsPendingUntilUpdateCompletes.
  • providers/aws/lambda/alias_notfound_test.go: TestAliasAndVersionNotFoundParity, extended for the qualified-ARN text of an unknown non-numeric qualifier.

Review-round tests. Each fails before its fix and passes after it.

  • providers/aws/lambda/invoke_update_race_test.go (run under -race):
    • TestConcurrentInvokeDuringCodeUpdatesNeverHitsRemovedDeployment: the reviewer's reproduction, 4 goroutines invoking $LATEST while 1500 code updates run, each followed by fc.Advance(2s). Before the fix: 352 invocations failed during code updates; first: Internal: invoke function real-fn: "real-fn.update-..." not deployed. After: 0 failures over 10 runs, and at most 2 deployments remain.
    • TestCodeUpdateWaitsForRunningInvocationOfRetiredDeployment: deterministic. An invocation blocked inside the engine on the prior deployment; the next code update retires that deployment and must not return until the invocation finishes, which then succeeds with the old code. Before the fix the update returned while the invocation was still running.
  • providers/aws/lambda/parity_1665_test.go: TestRetiredDeploymentsAreRemovedAfterRestore (before: the original deployment survived the update and the delete), TestLatestInProgressReportsUpdateReason.
  • server/aws/lambda/parity_1665_sdk_test.go: TestSDKLatestInProgressReportsUpdateReason, plus a check in TestSDKUpdateFunctionCodePublishVersionIsPending that the Pending version carries no update reason.

Commands

  • go build ./...: ok
  • go test -race ./providers/aws/lambda/... ./server/aws/lambda/... ./server/aws/ ./persist/... ./internal/settle/... ./config/...: ok. This covers the persist completeness test.
  • go test -p 2 over all 657 packages, in chunks: ok
  • golangci-lint run --allow-parallel-runners --concurrency 2 --new-from-rev=origin/fix/aws-lambda-parity-1518 on the touched packages: 0 issues

Review round, real-user check on this branch's engines server (contrib/server -providers aws -aws-port 4608 -async-settle -functions subprocess), AWS CLI with --endpoint-url http://127.0.0.1:4608:

create-function r5 (old.zip)                         -> Pending; wait function-active-v2 -> ok
invoke before update                                 -> {"code": "old"}
update-function-code new.zip                         -> InProgress  The function is being created.  Creating
  invoke during window                               -> {"code": "old"}
  get-function-configuration during window           -> InProgress  The function is being created.  Creating
wait function-updated-v2                             -> ok
get-function-configuration after                     -> Successful  None  None
invoke after the window                              -> {"code": "new"}
4 parallel invoke loops while 6 code updates run back to back (update, wait function-updated-v2):
                                                     -> 102 invokes ok, 0 failed; every output is old, new or v3
delete-function                                      -> ok

Real-user AWS CLI check from the first round (aws-cli 1.44.87, always with --endpoint-url http://127.0.0.1:4608)

The lean cloudemu serve binary has no function runtime, so its Invoke echoes the payload and cannot show old versus new code. The code-swap check therefore ran on the engines server (contrib/server) with -providers aws -aws-port 4608 -async-settle -functions subprocess, using python3.12 zips that return {"code": "<tag>"}. The default settle window is 1 s and each CLI process takes about 1 s to start. So the in-window calls were started 0.4 s after the update and ran in parallel with it.

create-function swap2 (old.zip)                      -> Pending; wait function-active-v2 -> ok
invoke (before update)                               -> {"code": "old"}
update-function-code new.zip                         -> InProgress
  invoke during window                               -> {"code": "old"}
  get-function-configuration during window           -> LastUpdateStatus InProgress
wait function-updated-v2                             -> ok
invoke (after the window)                            -> {"code": "new"}
update-function-code v3.zip --publish                -> 1  Pending  Creating  None
  get-function --qualifier 1 during window           -> 1  Pending  Creating  None
  invoke $LATEST during window                       -> {"code": "new"}
wait function-active-v2 --qualifier 1                -> ok
get-function --qualifier 1 after                     -> 1  Active  Successful
invoke after                                         -> {"code": "v3"}

The same engines server without -async-settle (default mode, unchanged):

create / update-function-code                        -> Active Successful / Active Successful
invoke right after update                            -> {"code": "new"}
update-function-code --publish                       -> 1 Active Successful

The lean cloudemu serve -providers aws -aws-port 4608 -async-settle:

update-function-code --publish                       -> 1  Pending  Creating  None
wait function-active-v2 --qualifier 1 / function-updated-v2 -> ok / ok
delete-alias --name non-existent                     -> exit 0
update-alias --name non-existent                     -> ResourceNotFoundException: Alias not found: arn:aws:lambda:us-east-1:000000000000:function:l1:non-existent
delete-function --qualifier 99                       -> exit 0; list-versions-by-function -> $LATEST 1
delete-function --function-name ghost --qualifier 1  -> ResourceNotFoundException: Function not found: arn:aws:lambda:us-east-1:000000000000:function:ghost
delete-alias --function-name ghost                   -> ResourceNotFoundException: Function not found: arn:aws:lambda:us-east-1:000000000000:function:ghost

Terraform is not installed here, so no apply/plan/destroy run.

Known limits

  • Invoking the version that --publish cut while it is still Pending is not rejected. The docs say a Pending function cannot be invoked, but I found no recording of the exact error for a version. Tracked in AWS Lambda: follow-up parity gaps (versions, aliases, in-progress updates) #1665.
  • Published versions still run the function's live engine deployment. This predates this PR: versions do not keep their own code in the engine.
  • UpdateFunctionConfiguration does not redeploy to the engine, which also predates this PR. Environment and Timeout changes therefore never reach the real runtime. The in-progress window for configuration-only updates is visible in the in-memory view (for example the DeadLetterConfig used for async failure routing) but not on the engine path.
  • Under --async-settle, each code update keeps one extra engine deployment (the prior revision's) until the next code update or delete. A code update that retires a deployment waits for invocations still running on it before removing it.
  • DeleteAlias on a missing function still returns ResourceNotFoundException. I found no recording for that case.
  • A non-numeric DeleteFunction --qualifier that is neither a version nor an alias stays ResourceNotFoundException with the qualified-ARN text; the real response is unrecorded, as noted under item 4.

Related Issues

Part of #1665. Stacked on #1646.

@NitinKumar004 NitinKumar004 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I checked

  • The diff f6700ed6..e9b5fa8b, 14 files. The non-test changes are in providers/aws/lambda (engine_ops, lambda, lifecycle, versions, aliases, snapshot), internal/settle (new Set.Window) and a doc comment in config/engine.go. server/aws/lambda changes only in tests.
  • Gates on the PR head: go build ./... ok, go vet on the touched packages ok, go test -race -count=1 on providers/aws/lambda, server/aws/lambda, internal/settle, config, persist and server/aws ok, gofmt clean, go mod tidy no diff, go generate . no diff.
  • golangci-lint v2.14.0 on providers/aws/lambda, server/aws/lambda, internal/settle and config without --new-from-rev, compared with the same run on f6700ed: 36 issues on the base, 38 on the head (details under Medium 2).
  • Engines server (contrib/server -providers aws -async-settle -functions subprocess) driven with aws-cli 2.31 and python3.12 zips that return their own tag:
    • before update old; update-function-code new.zip gives InProgress; invoke during the window gives old; after function-updated-v2 it gives new.
    • update-function-code v3.zip --publish gives 1 Pending Creating None, and get-function --qualifier 1 during the window gives the same. Invoke of $LATEST during the window gives new. After the waiters, version 1 is Active Successful and both $LATEST and version 1 give v3.
    • After three more code updates, two deployment dirs exist (live and prior). After delete-function there are none, so nothing leaks.
  • Lean cloudemu serve -providers aws -async-settle with the AWS CLI:
    • delete-alias of a missing alias exits 0.
    • update-alias of a missing alias gives Alias not found: arn:...:function:l1:nope.
    • delete-alias on a missing function gives Function not found.
    • delete-function --qualifier 99 exits 0, and deleting version 1 twice exits 0 both times.
    • $LATEST, an alias qualifier, and a version an alias still references are rejected as before.
    • update-code --publish gives 2 Pending Creating. PublishVersion and UpdateFunctionConfiguration during the window give ResourceConflictException. After settling, both $LATEST and version 2 show Active/Successful.
  • Terraform hashicorp/aws 6.x against the same serve (async-settle), with aws_iam_role, aws_lambda_function (publish = true) and aws_lambda_alias: apply, plan exit 0, update (new zip and description), plan exit 0 (alias on version 2), destroy, then get-function gives ResourceNotFoundException.
  • The #1646 Medium (gocyclo in server/aws/lambda) is still there and unchanged: ServeHTTP 15, create 13, invoke 13, classify 12, Matches 12. This PR does not touch those functions, so it neither fixes that Medium nor makes it worse. It still has to be fixed in #1646.
  • No AI attribution in the commits or the PR body.

Items (a) and (c) are correct for the sequential case. A $LATEST invoke inside the window runs the previous code and configuration. Published versions are untouched. A version cut by --publish settles with the update through a copied window. Engine Deploy and Remove never run under m.mu, and every new funcData read-modify-write is under it. Invoke never takes m.mu, so no lock is held across the engine invoke. The problem is in (b), with concurrent invokes.

High

None.

Medium

  1. A concurrent invoke can pick a deployment that a code update then removes, so it fails with ServiceException (providers/aws/lambda/lifecycle.go:259, engine_ops.go:181, lambda.go:533). Invoke takes an unlocked copy of funcData and then asks the shared updateSettle set whether an update is running. Those are two separate reads. A copy taken just before commitUpdate stores the new fd (lambda.go:533) still holds the older prior. If the window opens at lambda.go:534 before the isUpdating check, that invoke serves the revision from two updates back. That revision's key is the one this same update just retired, and UpdateFunction removes it right after finalizing (engine_ops.go:181). The same failure happens without the stale copy: an invoke that resolved its key from the prior revision near the end of a window can reach the engine after the next code update has removed that key. Either way, funcengine.Invoke returns "not deployed", which invokeEngine turns into cerrors.Internal (HTTP 500).

    Reproduction: a throwaway provider test with the PR's own codeEngine. Four goroutines invoke $LATEST in a loop while the main goroutine runs 3000 code updates, each followed by fc.Advance(2s). Under AsyncSettle, 143 to 201 invokes failed with "not deployed" in each of three runs (about 85k to 128k invokes per run). The same test in default mode had 0 failures, as did the code before this PR, which deployed in place. So this is new, and it only happens under --async-settle. Under that flag, invoking while you deploy is exactly the case this PR models.

    Suggested fix, in two parts:

    • Make the copy self-consistent: store the update's settle.Window in priorRevision, the way versionData.updateWindow already does, and have servePrior check fd.prior.window.Settled(now) instead of the shared set.
    • Do not remove a retired key while an invocation that picked it may still be running. Two options: a per-key in-flight count that removeDeployments waits on, or retire a key only once it is at least two updates old and no window is open.

    A regression test like the one described would cover both parts.

  2. New lint findings in providers/aws/lambda (versions.go:39, versions.go:99). Without --new-from-rev, compared with f6700ed:

    • gocyclo: (*Mock).publishVersion goes from 10 to 11.
    • wsl: the append at versions.go:99 is cuddled after the new if block.

    --new-from-rev hides the gocyclo one because the function signature did not change. The repo keeps cyclomatic complexity at 10 or below. Moving the version's lifecycle fields into a small helper fixes both, and so would reading the window once (w := m.updateSettle.Window(functionName); if !gate && !w.Settled(now)). The helper is the better choice because it also takes cutWhilePending out of publishVersion.

Low

  1. Retired deployments leak after a Restore. prior and retired are not in the snapshot, so after a restore under AsyncSettle the prior and retired engine keys are no longer tracked. Those deployments stay until the process exits. The PR documents this. It would cost little to fold prior.engineKey into retired in snapshotFunc (or persist retired), so the next code update or delete removes them.
  2. The Pending version cut by --publish can be invoked. The docs say a Pending function cannot be invoked. Invoking version N while it reports Pending runs the new code. The PR lists this as a known limit. It is worth a line in #1665 so it is not lost.
  3. $LATEST InProgress omits LastUpdateStatusReason and LastUpdateStatusReasonCode. The PR notes this new gap. Please add it to #1665 as suggested.
  4. The unknown non-numeric qualifier keeps the emulator's wording. delete-function --qualifier bogus still returns "function version bogus not found", which is not real Lambda's Function not found: <ARN>:<qualifier> style. Real behaviour is unverified, so this is fine to leave, but the emulator's other not-found messages now use the ARN form. Aligning this one would keep them consistent.

Comment thread providers/aws/lambda/lifecycle.go Outdated
// configuration from before the update. Published versions are immutable and
// never affected.
func (m *Mock) servePrior(fd *funcData, executedVersion string) {
if executedVersion != latestVersion || fd.prior == nil || !m.isUpdating(fd.info.Name) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fd here is an unlocked copy, but isUpdating reads the shared settle set, so the two can come from different moments. A copy taken just before commitUpdate stores the new fd (lambda.go:533) still has the older prior. If the window opens (lambda.go:534) before this check, we serve the revision from two updates back, whose key this same update retires and removes at engine_ops.go:181. The invoke then gets "not deployed" and a 500. Storing the update's settle.Window in priorRevision (like versionData.updateWindow) and checking fd.prior.window.Settled(now) here would make the copy self-consistent.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. Fixed in d0f99f2: priorRevision now carries the settle.Window of the update that replaced it (set from beginUpdateSettle's return in commitUpdate), and servePrior checks fd.prior.window.Settled(now) only. A copy taken before a later update was committed now serves its own live key, which that update does not retire.

On top of that, pinDeployment re-reads the function after pinning the key and retries with a fresh copy if the key is no longer the live or prior one, so a stale copy can't run a retired deployment even in theory.


// The reservation is still held, so no other engine call for name runs
// while the deployments that no longer serve anything are removed.
m.removeDeployments(ctx, stale)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This removes the retired keys while invokes may still be about to use them: Invoke resolves its engine key without any reservation or lock. A stress test (4 goroutines invoking, 3000 code updates under AsyncSettle with the PR's codeEngine) gives about 150 to 200 "not deployed" ServiceExceptions per run, and 0 in default mode. Could removal wait for in-flight invocations on that key (a per-key count), or only retire keys that are at least two updates old?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I went with the per-key in-flight count (d0f99f2, pins.go). An invocation pins its engine key, then re-reads the function and checks it still serves that key (live or prior); if not, it unpins, refreshes and retries. removeDeployments waits on pins.wait(key) before Remove. Because the update stores the function without the key before it drains, any pin that passes the check holds a deployment that stays until the invocation ends.

The wait runs under the engine reservation, never under m.mu, and Invoke still never takes m.mu.

Your reproduction is now TestConcurrentInvokeDuringCodeUpdatesNeverHitsRemovedDeployment (4 invokers, 1500 updates with fc.Advance(2s), under -race): 352 "not deployed" failures before, 0 after, across 10 runs. TestCodeUpdateWaitsForRunningInvocationOfRetiredDeployment covers the still-running case deterministically with an engine that blocks mid-invoke.

Comment thread providers/aws/lambda/versions.go Outdated
// Likewise only Publish=true on UpdateFunctionCode cuts a version while an
// update is InProgress. That version settles together with the update.
if !gate && m.isUpdating(functionName) {
vd.updateWindow = m.updateSettle.Window(functionName)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This takes publishVersion to cyclomatic complexity 11 (gocyclo, hidden by --new-from-rev), and the append below now trips wsl. Moving cutWhilePending and updateWindow into a small versionLifecycle(functionName, gate) helper fixes both. It also reads the window once instead of calling isUpdating and then Window.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d0f99f2 as you suggested: setVersionLifecycle(vd, functionName, gate) sets cutWhilePending and updateWindow, reading the window once. publishVersion is back to gocyclo 10 and the wsl finding is gone. A full golangci-lint v2.14.0 run on the four packages shows the same 28 pre-existing issues as dca3dcf1.

- A published version now freezes its CodeSize, description, environment
  and AWS-only settings (VpcConfig, DeadLetterConfig, TracingConfig,
  Architectures, EphemeralStorage, Layers) at publish time, so
  UpdateFunctionCode/Configuration on $LATEST no longer changes what
  GetFunction --qualifier N or ListVersionsByFunction report for it.
- Under AsyncSettle a new function reports State Pending (StateReasonCode
  Creating, no LastUpdateStatus) and an update reports LastUpdateStatus
  InProgress. While Pending, Invoke, UpdateFunctionCode,
  UpdateFunctionConfiguration and PublishVersion return
  ResourceConflictException; while InProgress the updates, PublishVersion
  and TagResource do. The InProgress gate also covers an in-flight engine
  deploy. Publish=true on Create/UpdateFunctionCode is not blocked by the
  window its own write opened. Settle off keeps the old behaviour.
- Invoke with LogType=Tail returns X-Amz-Log-Result: the base64 of the
  last 4 KB of the execution log the provider already renders.
- GetAccountSettings (GET /2016-08-19/account-settings).
- Code signing configs: Create/Get/List/Update/DeleteCodeSigningConfig,
  Put/Get/DeleteFunctionCodeSigningConfig, ListFunctionsByCodeSigningConfig,
  CodeSigningConfigArn on CreateFunction, and tagging a code signing config
  ARN, with IAM checks per the Service Authorization Reference. State is
  in the Lambda snapshot.
- ResourceNotFoundException for a missing function, version or alias
  reads "Function not found: <function ARN>[:<qualifier>]".

Part of stackshy#1518
…CreateFunction on code signing attach failure

- A published version no longer inherits $LATEST's State, StateReason(Code),
  LastUpdateStatus and LastModified. The provider now reports each version's
  own lifecycle on FunctionVersion (Active/Successful, or Pending only for the
  version CreateFunction(Publish=true) cut, until creation completes), and the
  handler renders LastModified from the version's publish time. GetFunction
  --qualifier N and ListVersionsByFunction are unchanged by a later $LATEST
  update, in default mode and under --async-settle.
- CreateFunction with CodeSigningConfigArn deletes the just-created function
  when the attach fails (config deleted after the existence check) and reports
  CodeSigningConfigNotFoundException, so a retry does not hit
  ResourceConflictException. PutFunctionCodeSigningConfig maps the same race to
  CodeSigningConfigNotFoundException.
- GetAlias for a missing alias returns real Lambda's text:
  "Cannot find alias arn: <function ARN>:<alias>".
- Move resourceTags' doc comment back above its func.
Split the code signing and account settings branches out of ServeHTTP,
Matches and classify into small helpers, group the two concurrency routes,
and move CreateFunction's validation/config build and Invoke's DryRun and
synchronous response writing into their own functions. Name the repeated
response field strings as constants. No behaviour change.
…teAlias and DeleteFunction --qualifier

Real Lambda, as recorded by AWS-validated snapshot tests:
- DeleteAlias of an alias the function does not have returns 204 (no
  ResourceNotFoundException, which is not in the DeleteAlias error list).
- UpdateAlias of a missing alias is ResourceNotFoundException with the
  message "Alias not found: <function ARN>:<alias>".
- DeleteFunction with a numbered Qualifier the function does not have is an
  idempotent 204; a missing function is still "Function not found:
  <unqualified function ARN>".

A missing function stays ResourceNotFoundException for all three. A
non-numeric unknown DeleteFunction qualifier keeps its current error, as its
real behaviour is unverified.

Part of stackshy#1665
…s InProgress

Under --async-settle an UpdateFunctionCode/UpdateFunctionConfiguration opens
an InProgress window. Real Lambda documents that "invocations continue to use
the function's previous code and configuration until the update completes",
but the emulator ran the new code as soon as the update returned.

- The function as it was before the update (configuration, AWS-only
  settings, engine deployment) is kept as the prior revision, and Invoke of
  $LATEST runs it until the window closes. Published versions are immutable
  and unaffected.
- On the real-runtime path a code update deploys to a fresh engine key
  (<name>.update-<id>), so the old deployment keeps serving. Deployments that
  no longer serve anything are removed at the next code update or on delete,
  while the function's engine reservation is held; engine calls still never
  run under m.mu, and every funcData read-modify-write stays under it.
- The live engine key is persisted in snapshots; the prior revision is not,
  like the settle windows.
- The version UpdateFunctionCode --publish cuts during the window now reports
  State Pending, StateReasonCode Creating and no LastUpdateStatus until the
  update completes, then Active/Successful, as real Lambda reports it. It
  shares the update's exact deadline through settle.Set.Window.

Default mode (no --async-settle) is unchanged: updates complete at once and
the code is redeployed in place under the function name.

Part of stackshy#1665
…it finish

Under AsyncSettle an invocation could pick a deployment that a concurrent code
update then removed, and fail with ServiceException ("not deployed"):

- Invoke reads an unlocked copy of the function but checked the shared settle
  set to decide whether to serve the prior revision, so a copy taken just
  before an update was committed could serve the revision that update retired.
  The prior revision now carries its update's InProgress window, and Invoke
  checks only that, so a copy is self-consistent.
- A code update removed retired deployments while invocations on them could
  still be starting or running. An invocation now pins the deployment it runs
  on, re-checking after pinning that the function still serves it, and a code
  update removes a retired deployment only once its pins are released.

Also moves the published version's lifecycle fields into setVersionLifecycle,
which keeps publishVersion at cyclomatic complexity 10 and reads the update
window once.
…s across a restore

- $LATEST reports LastUpdateStatusReason "The function is being created." and
  LastUpdateStatusReasonCode "Creating" while an update is InProgress, as real
  Lambda does, and omits them once the update is Successful. A published
  version never carries them.
- A snapshot records the prior revision's deployment as retired (the prior
  revision itself is not persisted), and persists the retired list, so after a
  restore the next code update or DeleteFunction removes those deployments
  instead of leaking them.
- DeleteFunction with an unknown non-numeric qualifier stays
  ResourceNotFoundException but now uses the same "Function not found:
  <qualified ARN>" text as the other not-found errors.
@Satyam-Trivedi-ZS
Satyam-Trivedi-ZS force-pushed the fix/aws-lambda-followups-1665 branch from e9b5fa8 to 461b606 Compare October 24, 2026 17:59
@Satyam-Trivedi-ZS

Copy link
Copy Markdown
Collaborator Author

Thanks for the thorough review, @NitinKumar004. Pushed d0f99f2 and 461b606.

Restack. Rebased onto #1646's new head dca3dcf1 with git rebase --onto origin/fix/aws-lambda-parity-1518 f6700ed6. It applied cleanly: the #1646 refactor only touched server/aws/lambda handler code, and this PR's handler-side changes are tests plus two new response fields. The first-round commits are now 2dc7124 and d7ae180. go generate ./... leaves no diff.

Medium 1 (invoke vs. removed deployment): fixed in d0f99f2, with both parts of your suggestion:

  • priorRevision stores its update's settle.Window, and servePrior checks only that, so a funcData copy is self-consistent.
  • Invocations pin their engine key (a per-key count in pins.go), then re-check that the function still serves it and retry with a fresh copy if not. removeDeployments waits for the key's pins to drain before Remove. Locking is unchanged: no m.mu around engine Deploy/Remove or the drain, and funcData read-modify-writes are still under m.mu.
  • Your reproduction is a regression test under -race: 352 "not deployed" failures before the fix, 0 after (10 runs). A second, deterministic test blocks an invocation inside the engine and checks that the next code update waits for it.
  • Engines server with -async-settle -functions subprocess and the AWS CLI: 4 parallel invoke loops during 6 back-to-back code updates gave 102 invokes, 0 failures.

Medium 2 (lint): fixed. setVersionLifecycle holds the version lifecycle fields. golangci-lint v2.14.0, full run on providers/aws/lambda, server/aws/lambda, internal/settle, config: 28 on this branch, 28 on dca3dcf1, the same pre-existing set (26 goconst, 1 gocritic, 1 wsl). Nothing new and no function above gocyclo 10.

Low 1 (retired leak after Restore): fixed in 461b606. The snapshot persists the retired keys with prior's key folded in, so the next code update or delete removes them. Test: TestRetiredDeploymentsAreRemovedAfterRestore.

Low 2 (Pending --publish version is invocable): tracked. Added a Low item to #1665.

Low 3 (LastUpdateStatusReason): implemented in 461b606. $LATEST reports "The function is being created." / Creating while InProgress (as in all 36 InProgress responses in LocalStack's AWS-validated snapshots) and omits both once Successful. Published versions never carry them. CLI during the window: InProgress The function is being created. Creating; after: Successful None None.

Low 4 (--qualifier bogus wording): aligned. It stays ResourceNotFoundException (same code and status) and now reads Function not found: <ARN>:bogus, like the other not-found errors. I kept the error itself because real behaviour for that case is unrecorded; only the text changed.

Tests: go build ./...; go test -race -p 1 on the lambda provider/server, server/aws, persist, internal/settle, config; the full suite (660 packages) with no failures. The PR description is updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants