Repository navigation
fix(aws-lambda): serve the previous revision during InProgress updates; align alias/version not-found behaviour - #1683
Conversation
NitinKumar004
left a comment
There was a problem hiding this comment.
What I checked
- The diff
f6700ed6..e9b5fa8b, 14 files. The non-test changes are inproviders/aws/lambda(engine_ops, lambda, lifecycle, versions, aliases, snapshot),internal/settle(newSet.Window) and a doc comment inconfig/engine.go.server/aws/lambdachanges only in tests. - Gates on the PR head:
go build ./...ok,go veton the touched packages ok,go test -race -count=1onproviders/aws/lambda,server/aws/lambda,internal/settle,config,persistandserver/awsok, gofmt clean,go mod tidyno diff,go generate .no diff. - golangci-lint v2.14.0 on
providers/aws/lambda,server/aws/lambda,internal/settleandconfigwithout--new-from-rev, compared with the same run on f6700ed: 36 issues on the base, 38 on the head (details under Medium 2). - Engines server (
contrib/server -providers aws -async-settle -functions subprocess) driven with aws-cli 2.31 and python3.12 zips that return their own tag:- before update
old;update-function-code new.zipgives InProgress; invoke during the window givesold; afterfunction-updated-v2it givesnew. update-function-code v3.zip --publishgives1 Pending Creating None, andget-function --qualifier 1during the window gives the same. Invoke of$LATESTduring the window givesnew. After the waiters, version 1 isActive Successfuland both$LATESTand version 1 givev3.- After three more code updates, two deployment dirs exist (live and prior). After delete-function there are none, so nothing leaks.
- before update
- Lean
cloudemu serve -providers aws -async-settlewith the AWS CLI:- delete-alias of a missing alias exits 0.
- update-alias of a missing alias gives
Alias not found: arn:...:function:l1:nope. - delete-alias on a missing function gives
Function not found. delete-function --qualifier 99exits 0, and deleting version 1 twice exits 0 both times.$LATEST, an alias qualifier, and a version an alias still references are rejected as before.- update-code
--publishgives2 Pending Creating. PublishVersion and UpdateFunctionConfiguration during the window give ResourceConflictException. After settling, both$LATESTand version 2 show Active/Successful.
- Terraform hashicorp/aws 6.x against the same serve (async-settle), with
aws_iam_role,aws_lambda_function(publish = true) andaws_lambda_alias: apply, plan exit 0, update (new zip and description), plan exit 0 (alias on version 2), destroy, then get-function gives ResourceNotFoundException. - The #1646 Medium (gocyclo in
server/aws/lambda) is still there and unchanged:ServeHTTP15,create13,invoke13,classify12,Matches12. This PR does not touch those functions, so it neither fixes that Medium nor makes it worse. It still has to be fixed in #1646. - No AI attribution in the commits or the PR body.
Items (a) and (c) are correct for the sequential case. A $LATEST invoke inside the window runs the previous code and configuration. Published versions are untouched. A version cut by --publish settles with the update through a copied window. Engine Deploy and Remove never run under m.mu, and every new funcData read-modify-write is under it. Invoke never takes m.mu, so no lock is held across the engine invoke. The problem is in (b), with concurrent invokes.
High
None.
Medium
-
A concurrent invoke can pick a deployment that a code update then removes, so it fails with ServiceException (
providers/aws/lambda/lifecycle.go:259,engine_ops.go:181,lambda.go:533). Invoke takes an unlocked copy offuncDataand then asks the sharedupdateSettleset whether an update is running. Those are two separate reads. A copy taken just beforecommitUpdatestores the newfd(lambda.go:533) still holds the olderprior. If the window opens at lambda.go:534 before theisUpdatingcheck, that invoke serves the revision from two updates back. That revision's key is the one this same update just retired, andUpdateFunctionremoves it right after finalizing (engine_ops.go:181). The same failure happens without the stale copy: an invoke that resolved its key from the prior revision near the end of a window can reach the engine after the next code update has removed that key. Either way,funcengine.Invokereturns "not deployed", whichinvokeEngineturns intocerrors.Internal(HTTP 500).Reproduction: a throwaway provider test with the PR's own
codeEngine. Four goroutines invoke$LATESTin a loop while the main goroutine runs 3000 code updates, each followed byfc.Advance(2s). Under AsyncSettle, 143 to 201 invokes failed with "not deployed" in each of three runs (about 85k to 128k invokes per run). The same test in default mode had 0 failures, as did the code before this PR, which deployed in place. So this is new, and it only happens under--async-settle. Under that flag, invoking while you deploy is exactly the case this PR models.Suggested fix, in two parts:
- Make the copy self-consistent: store the update's
settle.WindowinpriorRevision, the wayversionData.updateWindowalready does, and haveservePriorcheckfd.prior.window.Settled(now)instead of the shared set. - Do not remove a retired key while an invocation that picked it may still be running. Two options: a per-key in-flight count that
removeDeploymentswaits on, or retire a key only once it is at least two updates old and no window is open.
A regression test like the one described would cover both parts.
- Make the copy self-consistent: store the update's
-
New lint findings in
providers/aws/lambda(versions.go:39,versions.go:99). Without--new-from-rev, compared with f6700ed:- gocyclo:
(*Mock).publishVersiongoes from 10 to 11. - wsl: the
appendat versions.go:99 is cuddled after the newifblock.
--new-from-revhides the gocyclo one because the function signature did not change. The repo keeps cyclomatic complexity at 10 or below. Moving the version's lifecycle fields into a small helper fixes both, and so would reading the window once (w := m.updateSettle.Window(functionName); if !gate && !w.Settled(now)). The helper is the better choice because it also takescutWhilePendingout ofpublishVersion. - gocyclo:
Low
- Retired deployments leak after a Restore.
priorandretiredare not in the snapshot, so after a restore under AsyncSettle the prior and retired engine keys are no longer tracked. Those deployments stay until the process exits. The PR documents this. It would cost little to foldprior.engineKeyintoretiredinsnapshotFunc(or persistretired), so the next code update or delete removes them. - The Pending version cut by
--publishcan be invoked. The docs say a Pending function cannot be invoked. Invoking version N while it reports Pending runs the new code. The PR lists this as a known limit. It is worth a line in #1665 so it is not lost. $LATESTInProgress omitsLastUpdateStatusReasonandLastUpdateStatusReasonCode. The PR notes this new gap. Please add it to #1665 as suggested.- The unknown non-numeric qualifier keeps the emulator's wording.
delete-function --qualifier bogusstill returns "function version bogus not found", which is not real Lambda'sFunction not found: <ARN>:<qualifier>style. Real behaviour is unverified, so this is fine to leave, but the emulator's other not-found messages now use the ARN form. Aligning this one would keep them consistent.
| // configuration from before the update. Published versions are immutable and | ||
| // never affected. | ||
| func (m *Mock) servePrior(fd *funcData, executedVersion string) { | ||
| if executedVersion != latestVersion || fd.prior == nil || !m.isUpdating(fd.info.Name) { |
There was a problem hiding this comment.
fd here is an unlocked copy, but isUpdating reads the shared settle set, so the two can come from different moments. A copy taken just before commitUpdate stores the new fd (lambda.go:533) still has the older prior. If the window opens (lambda.go:534) before this check, we serve the revision from two updates back, whose key this same update retires and removes at engine_ops.go:181. The invoke then gets "not deployed" and a 500. Storing the update's settle.Window in priorRevision (like versionData.updateWindow) and checking fd.prior.window.Settled(now) here would make the copy self-consistent.
There was a problem hiding this comment.
Thanks, good catch. Fixed in d0f99f2: priorRevision now carries the settle.Window of the update that replaced it (set from beginUpdateSettle's return in commitUpdate), and servePrior checks fd.prior.window.Settled(now) only. A copy taken before a later update was committed now serves its own live key, which that update does not retire.
On top of that, pinDeployment re-reads the function after pinning the key and retries with a fresh copy if the key is no longer the live or prior one, so a stale copy can't run a retired deployment even in theory.
|
|
||
| // The reservation is still held, so no other engine call for name runs | ||
| // while the deployments that no longer serve anything are removed. | ||
| m.removeDeployments(ctx, stale) |
There was a problem hiding this comment.
This removes the retired keys while invokes may still be about to use them: Invoke resolves its engine key without any reservation or lock. A stress test (4 goroutines invoking, 3000 code updates under AsyncSettle with the PR's codeEngine) gives about 150 to 200 "not deployed" ServiceExceptions per run, and 0 in default mode. Could removal wait for in-flight invocations on that key (a per-key count), or only retire keys that are at least two updates old?
There was a problem hiding this comment.
Agreed. I went with the per-key in-flight count (d0f99f2, pins.go). An invocation pins its engine key, then re-reads the function and checks it still serves that key (live or prior); if not, it unpins, refreshes and retries. removeDeployments waits on pins.wait(key) before Remove. Because the update stores the function without the key before it drains, any pin that passes the check holds a deployment that stays until the invocation ends.
The wait runs under the engine reservation, never under m.mu, and Invoke still never takes m.mu.
Your reproduction is now TestConcurrentInvokeDuringCodeUpdatesNeverHitsRemovedDeployment (4 invokers, 1500 updates with fc.Advance(2s), under -race): 352 "not deployed" failures before, 0 after, across 10 runs. TestCodeUpdateWaitsForRunningInvocationOfRetiredDeployment covers the still-running case deterministically with an engine that blocks mid-invoke.
| // Likewise only Publish=true on UpdateFunctionCode cuts a version while an | ||
| // update is InProgress. That version settles together with the update. | ||
| if !gate && m.isUpdating(functionName) { | ||
| vd.updateWindow = m.updateSettle.Window(functionName) |
There was a problem hiding this comment.
This takes publishVersion to cyclomatic complexity 11 (gocyclo, hidden by --new-from-rev), and the append below now trips wsl. Moving cutWhilePending and updateWindow into a small versionLifecycle(functionName, gate) helper fixes both. It also reads the window once instead of calling isUpdating and then Window.
There was a problem hiding this comment.
Done in d0f99f2 as you suggested: setVersionLifecycle(vd, functionName, gate) sets cutWhilePending and updateWindow, reading the window once. publishVersion is back to gocyclo 10 and the wsl finding is gone. A full golangci-lint v2.14.0 run on the four packages shows the same 28 pre-existing issues as dca3dcf1.
- A published version now freezes its CodeSize, description, environment and AWS-only settings (VpcConfig, DeadLetterConfig, TracingConfig, Architectures, EphemeralStorage, Layers) at publish time, so UpdateFunctionCode/Configuration on $LATEST no longer changes what GetFunction --qualifier N or ListVersionsByFunction report for it. - Under AsyncSettle a new function reports State Pending (StateReasonCode Creating, no LastUpdateStatus) and an update reports LastUpdateStatus InProgress. While Pending, Invoke, UpdateFunctionCode, UpdateFunctionConfiguration and PublishVersion return ResourceConflictException; while InProgress the updates, PublishVersion and TagResource do. The InProgress gate also covers an in-flight engine deploy. Publish=true on Create/UpdateFunctionCode is not blocked by the window its own write opened. Settle off keeps the old behaviour. - Invoke with LogType=Tail returns X-Amz-Log-Result: the base64 of the last 4 KB of the execution log the provider already renders. - GetAccountSettings (GET /2016-08-19/account-settings). - Code signing configs: Create/Get/List/Update/DeleteCodeSigningConfig, Put/Get/DeleteFunctionCodeSigningConfig, ListFunctionsByCodeSigningConfig, CodeSigningConfigArn on CreateFunction, and tagging a code signing config ARN, with IAM checks per the Service Authorization Reference. State is in the Lambda snapshot. - ResourceNotFoundException for a missing function, version or alias reads "Function not found: <function ARN>[:<qualifier>]". Part of stackshy#1518
…CreateFunction on code signing attach failure - A published version no longer inherits $LATEST's State, StateReason(Code), LastUpdateStatus and LastModified. The provider now reports each version's own lifecycle on FunctionVersion (Active/Successful, or Pending only for the version CreateFunction(Publish=true) cut, until creation completes), and the handler renders LastModified from the version's publish time. GetFunction --qualifier N and ListVersionsByFunction are unchanged by a later $LATEST update, in default mode and under --async-settle. - CreateFunction with CodeSigningConfigArn deletes the just-created function when the attach fails (config deleted after the existence check) and reports CodeSigningConfigNotFoundException, so a retry does not hit ResourceConflictException. PutFunctionCodeSigningConfig maps the same race to CodeSigningConfigNotFoundException. - GetAlias for a missing alias returns real Lambda's text: "Cannot find alias arn: <function ARN>:<alias>". - Move resourceTags' doc comment back above its func.
Split the code signing and account settings branches out of ServeHTTP, Matches and classify into small helpers, group the two concurrency routes, and move CreateFunction's validation/config build and Invoke's DryRun and synchronous response writing into their own functions. Name the repeated response field strings as constants. No behaviour change.
…teAlias and DeleteFunction --qualifier Real Lambda, as recorded by AWS-validated snapshot tests: - DeleteAlias of an alias the function does not have returns 204 (no ResourceNotFoundException, which is not in the DeleteAlias error list). - UpdateAlias of a missing alias is ResourceNotFoundException with the message "Alias not found: <function ARN>:<alias>". - DeleteFunction with a numbered Qualifier the function does not have is an idempotent 204; a missing function is still "Function not found: <unqualified function ARN>". A missing function stays ResourceNotFoundException for all three. A non-numeric unknown DeleteFunction qualifier keeps its current error, as its real behaviour is unverified. Part of stackshy#1665
…s InProgress Under --async-settle an UpdateFunctionCode/UpdateFunctionConfiguration opens an InProgress window. Real Lambda documents that "invocations continue to use the function's previous code and configuration until the update completes", but the emulator ran the new code as soon as the update returned. - The function as it was before the update (configuration, AWS-only settings, engine deployment) is kept as the prior revision, and Invoke of $LATEST runs it until the window closes. Published versions are immutable and unaffected. - On the real-runtime path a code update deploys to a fresh engine key (<name>.update-<id>), so the old deployment keeps serving. Deployments that no longer serve anything are removed at the next code update or on delete, while the function's engine reservation is held; engine calls still never run under m.mu, and every funcData read-modify-write stays under it. - The live engine key is persisted in snapshots; the prior revision is not, like the settle windows. - The version UpdateFunctionCode --publish cuts during the window now reports State Pending, StateReasonCode Creating and no LastUpdateStatus until the update completes, then Active/Successful, as real Lambda reports it. It shares the update's exact deadline through settle.Set.Window. Default mode (no --async-settle) is unchanged: updates complete at once and the code is redeployed in place under the function name. Part of stackshy#1665
…it finish
Under AsyncSettle an invocation could pick a deployment that a concurrent code
update then removed, and fail with ServiceException ("not deployed"):
- Invoke reads an unlocked copy of the function but checked the shared settle
set to decide whether to serve the prior revision, so a copy taken just
before an update was committed could serve the revision that update retired.
The prior revision now carries its update's InProgress window, and Invoke
checks only that, so a copy is self-consistent.
- A code update removed retired deployments while invocations on them could
still be starting or running. An invocation now pins the deployment it runs
on, re-checking after pinning that the function still serves it, and a code
update removes a retired deployment only once its pins are released.
Also moves the published version's lifecycle fields into setVersionLifecycle,
which keeps publishVersion at cyclomatic complexity 10 and reads the update
window once.
…s across a restore - $LATEST reports LastUpdateStatusReason "The function is being created." and LastUpdateStatusReasonCode "Creating" while an update is InProgress, as real Lambda does, and omits them once the update is Successful. A published version never carries them. - A snapshot records the prior revision's deployment as retired (the prior revision itself is not persisted), and persists the retired list, so after a restore the next code update or DeleteFunction removes those deployments instead of leaking them. - DeleteFunction with an unknown non-numeric qualifier stays ResourceNotFoundException but now uses the same "Function not found: <qualified ARN>" text as the other not-found errors.
e9b5fa8 to
461b606
Compare
|
Thanks for the thorough review, @NitinKumar004. Pushed d0f99f2 and 461b606. Restack. Rebased onto #1646's new head Medium 1 (invoke vs. removed deployment): fixed in d0f99f2, with both parts of your suggestion:
Medium 2 (lint): fixed. Low 1 (retired leak after Restore): fixed in 461b606. The snapshot persists the retired keys with Low 2 (Pending Low 3 ( Low 4 ( Tests: |
Stacked on #1646, so merge that first. Until #1646 merges, this diff also shows its commits; this PR's own changes are the commits after
dca3dcf1.Summary
--async-settle, Invoke of$LATESTnow keeps running the previous code and configuration until an update completes. That is what the Lambda function states page says. Before this change the new code ran as soon asUpdateFunctionCodereturned.UpdateFunctionCode --publishnow reportsPending/Creatingwhile the update is in progress, thenActive/Successful.DeleteFunction --qualifiernow match real Lambda's not-found behaviour.--async-settleno longer fail with ServiceException: a deployment is removed only once no invocation can still be using it.$LATESTreportsLastUpdateStatusReason/LastUpdateStatusReasonCodewhile an update isInProgress.Part of #1665
Item status
InProgressupdate runs the new codeUpdateFunctionCode --publishreportsActive/Successfulat onceTestLambdaVersions::test_publish_with_update(last validated against AWS on 2025-11-25). Theupdate-function-code --publishresponse andget-function --qualifier 1both showVersion: 1, State: Pending, StateReason: "The function is being created.", StateReasonCode: Creating, with noLastUpdateStatus.$LATESTisActive/InProgressat the same time. snapshot, validation datesTestLambdaAlias::test_non_existent_alias_deletion(validated 2025-11-25) records HTTP 204 for a missing alias on an existing function. It was added in localstack#11878 ("AWS doesn't throw an error for delete_alias for non-existing alias"). The Go SDK has no modeled ResourceNotFoundException for DeleteAlias either.DeleteFunction --qualifierResourceNotFoundException: Alias not found: <function ARN>:<alias>(test_non_existent_alias_update, validated 2025-11-25). DeleteAlias: no error text, since a missing alias is 204 (item 3). DeleteFunction--qualifier <missing numbered version>: real Lambda returns success and repeating it is idempotent; a missing function isFunction not found: <unqualified function ARN>, which is already the emulator's text (TestLambdaFunction::test_delete_on_nonexisting_version, validated 2025-11-25). UNVERIFIABLE: a non-numeric qualifier that is neither a version nor an alias. No recording exists, so it stays ResourceNotFoundException; only its text moved to the sameFunction not found: <qualified ARN>form as the other not-found errors.The real-AWS responses above come from LocalStack's parity suite. Those tests are marked
@markers.aws.validatedand are recorded against real AWS. I did not call real AWS myself.Changes
fix(aws-lambda): keep serving the previous revision while an update is InProgress(d7ae180)providers/aws/lambda:commitUpdatesaves the function as it was before the update infd.prior: its configuration, AWS-only settings and engine deployment. While the update window is open,Invokeof$LATESTrunsfd.prior.fd.prior, so nothing changes there.<name>.update-<id>. Lambda names cannot contain., so the key cannot collide with another function's key. The old deployment keeps serving until the window closes.fd.retiredand removed at the next code update or on delete. Both run while the function's engine reservation is held.Deploy/Removestill never run underm.mu.UpdateFunctioncallsfinalizeCodeUpdate, which takesm.mu, commits, and returns the stale keys; they are removed after unlock and before the reservation is released.funcDataread-modify-write (keepPrior,takeRetired) runs underm.mu.Invokechanges only its own copy offuncData.prioris immutable once stored, andretiredis never appended in place.engineName) is in the snapshot.prioris not saved, like the existing settle windows, so a restored function serves its current revision. (The retired keys are persisted by a later commit, see below.)PublishVersionOnWritewhile the update window is open copies that window. It reports Pending/Creating until the update completes and then stays Active/Successful through later$LATESTupdates. The copy comes from a newsettle.Set.Windowaccessor, so the version and$LATESTshare the same deadline.config/engine.go: theFunctionDeployment.Namedoc now mentions the AsyncSettle update key.fix(aws-lambda): match real not-found behaviour for DeleteAlias, UpdateAlias and DeleteFunction --qualifier(2dc7124)DeleteAlias: a missing alias is a no-op success. A missing function is still ResourceNotFoundException.UpdateAlias: the error text is nowAlias not found: <function ARN>:<alias>.DeleteVersion(theDeleteFunction --qualifierpath): a missing numbered version is a no-op success.TestDeleteAlias/alias not foundandTestSDKDeleteFunctionMissingVersion. Both now assert the recorded AWS behaviour.fix(aws-lambda): keep a deployment until the invocations that picked it finish(d0f99f2), from review:priorRevisionnow carries thesettle.Windowof the update that replaced it, andservePriorchecks only that window instead of the shared settle set, so an unlockedfuncDatacopy is self-consistent.deployPinsinpins.go, a per-key count with async.Cond). After pinning,pinDeploymentre-reads the function and checks it still serves that key (live or prior); if a code update retired it in the meantime, it refreshes the copy and retries.removeDeploymentswaits for a key's pins to drain beforeRemove. Since the update stores the function without the key before it drains, a pin that passes the check holds a deployment that stays until the invocation ends.m.mu(under the function's engine reservation), and everyfuncDataread-modify-write is still underm.mu. Invoke never takesm.mu.setVersionLifecycleholds the published version's lifecycle fields. It keepspublishVersionat cyclomatic complexity 10 and reads the update window once.fix(aws-lambda): report the update reason and keep retired deployments across a restore(461b606), from review:$LATESTreportsLastUpdateStatusReason: "The function is being created."andLastUpdateStatusReasonCode: Creatingwhile an update isInProgress, and omits both once it isSuccessful. This matches all 36 InProgress responses in LocalStack's AWS-validated snapshots. A published version never carries them. Two optional fields were added todriver.FunctionInfo;go generateoutput is unchanged.DeleteFunction --qualifier <unknown non-numeric>uses theFunction not found: <qualified ARN>text.The business logic is in
providers/aws/lambda.server/aws/lambdaonly renders the two new response fields.Provider Coverage
Checklist
go test -p 1run over all 660 packages, in chunks of 15, no failures; plusgo test -race -p 1onproviders/aws/lambda/...,server/aws/lambda/...,server/aws/,persist/...,internal/settle/...andconfig/...--new-from-rev) onproviders/aws/lambda,server/aws/lambda,internal/settleandconfig: 28 issues on this branch and 28 on the parentdca3dcf1, the same pre-existing set (26 goconst, 1 gocritic inpolicy.go, 1 wsl inexeclogs.go). No new findings, and no function above gocyclo 10.cloudemu_test.go: not added; the provider tests and SDK wire tests below cover itgo generate ./...leaves no diff. No driver interface or operation changed; the additions aresettle.Set.Windowininternal/and two optional fields ondriver.FunctionInfoTest Plan
First-round tests. Each wire test fails on the old parent
f6700ed6and passes on this branch. I checked this by running the new SDK test files against af6700ed6checkout.server/aws/lambda/parity_1665_sdk_test.go, using the realaws-sdk-go-v2client:TestSDKInvokeRunsPreviousCodeWhileUpdateInProgress: runs withasync=trueandasync=false, using an engine that returns its deployed code. Before the fix:invoke right after the update = "new", want "old".TestSDKUpdateFunctionCodePublishVersionIsPending. Before the fix:state:Active ... lastUpdateStatus:Successful, butPending/Creatingis expected.server/aws/lambda/alias_notfound_sdk_test.go:TestSDKAliasNotFoundParity. Before the fix: DeleteAlias gave 404, and UpdateAlias gave the messagealias non-existent not found.server/aws/lambda/qualified_name_sdk_test.go:TestSDKDeleteFunctionMissingVersion, updated. Before the fix it got a 404.providers/aws/lambda/parity_1665_test.go:TestInvokeRunsPreviousCodeWhileUpdateInProgress: old code during the window, new code after it. A second update retires the original deployment, and delete removes all deployments.TestInvokeRunsNewCodeImmediatelyByDefault: default mode deploys in place.TestInvokeUsesPreviousConfigurationWhileUpdateInProgress: the previous Timeout and DeadLetterConfig are used for$LATESTonly.TestEngineNameSurvivesSnapshot.TestVersionPublishedWithUpdateIsPendingUntilUpdateCompletes.providers/aws/lambda/alias_notfound_test.go:TestAliasAndVersionNotFoundParity, extended for the qualified-ARN text of an unknown non-numeric qualifier.Review-round tests. Each fails before its fix and passes after it.
providers/aws/lambda/invoke_update_race_test.go(run under-race):TestConcurrentInvokeDuringCodeUpdatesNeverHitsRemovedDeployment: the reviewer's reproduction, 4 goroutines invoking$LATESTwhile 1500 code updates run, each followed byfc.Advance(2s). Before the fix:352 invocations failed during code updates; first: Internal: invoke function real-fn: "real-fn.update-..." not deployed. After: 0 failures over 10 runs, and at most 2 deployments remain.TestCodeUpdateWaitsForRunningInvocationOfRetiredDeployment: deterministic. An invocation blocked inside the engine on the prior deployment; the next code update retires that deployment and must not return until the invocation finishes, which then succeeds with the old code. Before the fix the update returned while the invocation was still running.providers/aws/lambda/parity_1665_test.go:TestRetiredDeploymentsAreRemovedAfterRestore(before: the original deployment survived the update and the delete),TestLatestInProgressReportsUpdateReason.server/aws/lambda/parity_1665_sdk_test.go:TestSDKLatestInProgressReportsUpdateReason, plus a check inTestSDKUpdateFunctionCodePublishVersionIsPendingthat the Pending version carries no update reason.Commands
go build ./...: okgo test -race ./providers/aws/lambda/... ./server/aws/lambda/... ./server/aws/ ./persist/... ./internal/settle/... ./config/...: ok. This covers thepersistcompleteness test.go test -p 2over all 657 packages, in chunks: okgolangci-lint run --allow-parallel-runners --concurrency 2 --new-from-rev=origin/fix/aws-lambda-parity-1518on the touched packages: 0 issuesReview round, real-user check on this branch's engines server (
contrib/server -providers aws -aws-port 4608 -async-settle -functions subprocess), AWS CLI with--endpoint-url http://127.0.0.1:4608:Real-user AWS CLI check from the first round (aws-cli 1.44.87, always with
--endpoint-url http://127.0.0.1:4608)The lean
cloudemu servebinary has no function runtime, so its Invoke echoes the payload and cannot show old versus new code. The code-swap check therefore ran on the engines server (contrib/server) with-providers aws -aws-port 4608 -async-settle -functions subprocess, using python3.12 zips that return{"code": "<tag>"}. The default settle window is 1 s and each CLI process takes about 1 s to start. So the in-window calls were started 0.4 s after the update and ran in parallel with it.The same engines server without
-async-settle(default mode, unchanged):The lean
cloudemu serve -providers aws -aws-port 4608 -async-settle:Terraform is not installed here, so no
apply/plan/destroyrun.Known limits
--publishcut while it is stillPendingis not rejected. The docs say a Pending function cannot be invoked, but I found no recording of the exact error for a version. Tracked in AWS Lambda: follow-up parity gaps (versions, aliases, in-progress updates) #1665.UpdateFunctionConfigurationdoes not redeploy to the engine, which also predates this PR. Environment and Timeout changes therefore never reach the real runtime. The in-progress window for configuration-only updates is visible in the in-memory view (for example the DeadLetterConfig used for async failure routing) but not on the engine path.--async-settle, each code update keeps one extra engine deployment (the prior revision's) until the next code update or delete. A code update that retires a deployment waits for invocations still running on it before removing it.DeleteFunction --qualifierthat is neither a version nor an alias stays ResourceNotFoundException with the qualified-ARN text; the real response is unrecorded, as noted under item 4.Related Issues
Part of #1665. Stacked on #1646.