Skip to content

fix(aws-lambda): version immutability, async state, LogType=Tail, GetAccountSettings, code signing configs (#1518) - #1646

Open
Satyam-Trivedi-ZS wants to merge 3 commits into
stackshy:developmentfrom
Satyam-Trivedi-ZS:fix/aws-lambda-parity-1518
Open

Satyam-Trivedi-ZS wants to merge 3 commits into
stackshy:developmentfrom
Satyam-Trivedi-ZS:fix/aws-lambda-parity-1518

Conversation

@Satyam-Trivedi-ZS

@Satyam-Trivedi-ZS Satyam-Trivedi-ZS commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Part of #1518. Closes all six AWS Lambda parity items:

  • A published version keeps the CodeSize, Environment and AWS-only settings it was cut with after $LATEST changes. It also keeps its own lifecycle: LastModified is its publish time and it reports Active/Successful whatever $LATEST is doing (only the version cut by Publish=true on create is Pending, until creation completes).
  • Under --async-settle / config.WithAsyncSettle(), functions report State: Pending after create and LastUpdateStatus: InProgress after an update, and the operations AWS blocks in those states return ResourceConflictException. Default behaviour (settle off) is unchanged.
  • Invoke with LogType=Tail returns X-Amz-Log-Result.
  • GetAccountSettings is implemented.
  • Code signing configs: Create/Get/List/Update/Delete, Put/Get/DeleteFunctionCodeSigningConfig, ListFunctionsByCodeSigningConfig, CodeSigningConfigArn on CreateFunction, and tagging a code signing config ARN.
  • ResourceNotFoundException for a missing function now says Function not found: <function ARN>, and for a missing version or alias qualifier on GetFunction/GetFunctionConfiguration/Invoke it says Function not found: <function ARN>:<qualifier>. GetAlias for a missing alias says Cannot find alias arn: <function ARN>:<alias>. Other not-found texts (DeleteAlias/UpdateAlias for a missing alias, DeleteFunction --qualifier for a missing version, layers, statements, event source mappings) are unchanged because I couldn't verify real AWS text for them.

Item status

Item Status Evidence
(M) Published version reports $LATEST CodeSize after UpdateFunctionCode Fixed versionData (providers/aws/lambda/lambda.go) now freezes codeSize, description and the AWS-only config (VpcConfig, DeadLetterConfig, TracingConfig, Architectures, EphemeralStorage, Layers) at publish time. Environment was already snapshotted in config but never surfaced: driver.FunctionVersion gains CodeSize and Environment, and the handler renders a version from its snapshot through the new AWS-only GetVersionAWSConfig. Before this, get-function --qualifier 1 after an UpdateFunctionCode/UpdateFunctionConfiguration returned $LATEST's CodeSize, Environment and Architectures.
(M) State/LastUpdateStatus always Active/Successful Fixed (opt-in, like every other settle in the repo) providers/aws/lambda/lifecycle.go, using internal/settle (DefaultLambdaCreateSettle, DefaultLambdaUpdateSettle, 1 s each). Create: State: Pending, StateReason: "The function is being created.", StateReasonCode: Creating, no LastUpdateStatus (the API reference says it "is first set to Successful after function creation completes"). Update: State: Active, LastUpdateStatus: InProgress. Blocked operations, per Lambda function states: while Pending, Invoke, UpdateFunctionCode, UpdateFunctionConfiguration and PublishVersion; while InProgress, UpdateFunctionCode, UpdateFunctionConfiguration, PublishVersion and TagResource. Invoke is not blocked during InProgress. The InProgress gate is shared with the existing engine-deploy reservation from #1535 (checkNotUpdating = settle window or engineBusy, reusing updateInProgress). Publish=true on CreateFunction/UpdateFunctionCode goes through PublishVersionOnWrite, so the atomic publish AWS does with the write is not refused by the window the write itself opened.
(M) Invoke LogType=Tail never returns X-Amz-Log-Result Fixed The provider already rendered START / END / REPORT lines (plus captured engine stdout/stderr and the handler error) but only wrote them to CloudWatch Logs when a sink was wired. It now always renders them, keeps the last 4096 bytes as InvokeOutput.LogTail, and the handler base64-encodes that into X-Amz-Log-Result for synchronous invokes that ask for Tail (Invoke: "The last 4 KB of the execution log, which is base64-encoded"). Event and DryRun invokes don't get the header, and neither does a call without LogType=Tail.
(L) GetAccountSettings Fixed GET /2016-08-19/account-settings. Limits: ConcurrentExecutions 1000, UnreservedConcurrentExecutions = 1000 minus the sum of reserved concurrency (the same pool PutFunctionConcurrency already enforces), CodeSizeZipped 52428800 (50 MB), CodeSizeUnzipped 262144000 (250 MB), TotalCodeSize 322122547200 (300 GB, the current quota page). Usage: FunctionCount, and TotalCodeSize = $LATEST + every published version + every layer version ("Each function version and layer version consumes storage"). Note: the CLI reference example still shows the older 75 GB (80530636800). I used the quota page because it is the current documented limit.
(L) Code signing config ops Fixed Routes from the botocore/aws-sdk-go-v2 model: /2020-04-22/code-signing-configs (POST 201 / GET), /2020-04-22/code-signing-configs/{arn} (GET / PUT / DELETE 204), /2020-04-22/code-signing-configs/{arn}/functions (GET), /2020-06-30/functions/{name}/code-signing-config (GET / PUT / DELETE 204). Validation from the model: 1 to 20 SigningProfileVersionArns, UntrustedArtifactOnDeployment in `Warn
(L) ResourceNotFoundException message wording (no ARN) Fixed (verified from real responses, not from the API reference) The API reference doesn't give message text. Two independent real-AWS captures show Function not found: arn:aws:lambda:<region>:<account>:function:<name> (HashiCorp Discuss) and the qualified form ...:function:GetCustomerProfile:RAM256 for an alias (aws-lambda-power-tuning#93). Every function lookup in the provider uses this text through errFunctionNotFound, with the qualifier for a missing version or alias on Invoke and on GetFunction/GetFunctionConfiguration --qualifier. GetAlias for a missing alias uses the real GetAlias text Cannot find alias arn: <function ARN>:<alias> (captured from a real SDK v2 GetAlias error in lambroll#386). Other not-found messages (DeleteAlias/UpdateAlias alias, DeleteFunction-with-qualifier version, layer, statement, event source mapping) are unchanged because I couldn't find real text for them.

Changes

  • services/serverless/driver: FunctionVersion.CodeSize/Environment; FunctionInfo.StateReason/StateReasonCode/LastUpdateStatus; InvokeOutput.LogTail; the AWS-only data types CodeSigningConfig, CodeSigningConfigUpdate and AccountSettings. No portable interface method changed, so go generate ./... leaves docs/coverage unchanged. The new operations are AWS-only optional interfaces, the same pattern as event invoke config and Function URLs.
  • providers/aws/lambda: lifecycle.go (settle windows, gates, not-found error), code_signing.go, account_settings.go, the per-version snapshot in versions.go, and the log tail in execlogs.go. All business rules live here.
  • server/aws/lambda: routes, classify and IAM rules for the new operations; X-Amz-Log-Result; version rendering from the snapshot, including the version's own LastModified/State/LastUpdateStatus (FunctionVersion carries the lifecycle the provider computes); CreateFunction with CodeSigningConfigArn deletes the just-created function and returns CodeSigningConfigNotFoundException if the config disappears between the check and the attach; StateReason/StateReasonCode/LastUpdateStatus from the provider (falling back to Successful for a backend that doesn't report it).
  • Persistence: code signing configs, the function to config association, and the version's CodeSize, description and AWS config are in the Lambda snapshot. Old snapshots restore with those fields empty. Settle windows are not persisted, the same as every other provider, so a restored function reports its terminal state.

Provider Coverage

  • AWS
  • Azure: no equivalent (versions, code signing, account settings and log tail are Lambda concepts)
  • GCP: no equivalent
  • OCI: no equivalent

Checklist

  • All tests pass. A full go test -p 2 ./... run caught one stale expectation, TestAuthzMatrixLambdaUnroutedOps, which listed account-settings and code-signing-configs as unrouted. I moved those two into a new TestAuthzMatrixLambdaAccountAndCodeSigning, then re-ran go test -p 2 . ./server/aws/... ./providers/aws/... ./persist/... ./services/... ./compat/..., which is green.
  • Linter: no new findings in touched packages, checked with a full run (no --new-from-rev) against development as well as --new-from-rev
  • Every provider the change applies to implements the same behavior (AWS-only)
  • Integration tests added to cloudemu_test.go: not added, covered by SDK wire tests
  • Unit tests added to provider test files
  • Regenerated docs: go generate ./... and go run ./internal/compatgen produce no diff

Test Plan

New tests, all of which fail on development and pass here. I ran the wire file against an extracted copy of the merge base: every test fails there, for example version 1 CodeSize = 26, want 7, log tail missing "START RequestId:", GetAccountSettings ... StatusCode: 501, message = "function ghost not found", create = "Active"/""/"Successful".

  • providers/aws/lambda/parity_1518_test.go: TestPublishedVersionKeepsItsCodeSizeAndConfig (including a snapshot round-trip), TestAsyncSettlePendingThenActive, TestAsyncSettleUpdateInProgress, TestDefaultSettleIsImmediate, TestInvokeReturnsLogTail, TestLogTailKeepsLast4KB, TestGetAccountSettings, TestCodeSigningConfigLifecycle (including a snapshot round-trip), TestFunctionNotFoundMessageNamesARN, TestGetAliasNotFoundMessage, TestPublishedVersionReportsItsOwnLifecycle.
  • server/aws/lambda/parity_1518_sdk_test.go (real aws-sdk-go-v2 client): TestSDKPublishedVersionIsImmutable, TestSDKInvokeLogTypeTail, TestSDKGetAccountSettings, TestSDKCodeSigningConfigs (typed CodeSigningConfigNotFoundException, ResourceConflictException, ResourceNotFoundException, tags), TestSDKFunctionNotFoundMessage, TestSDKPublishedVersionKeepsItsOwnLifecycle (default and async-settle: GetFunction --qualifier 1 and ListVersionsByFunction unchanged across a $LATEST update), TestSDKPublishOnCreateVersionIsPendingUntilCreated, TestSDKCreateFunctionRollsBackWhenCodeSigningAttachFails, and TestSDKAsyncSettleWaiters, which drives the SDK's own FunctionActiveV2 and FunctionUpdatedV2 waiters through Pending to Active and InProgress to Successful and asserts that Invoke while Pending is a ResourceConflictException.
  • classify_test.go and iam_actions_test.go: rows for every new operation (action and resource ARN).

Commands:

  • go build ./...
  • go vet ./providers/aws/lambda ./server/aws/lambda
  • go test -p 2 ./... (plus the re-run above)
  • server/aws: TestAuthzMatrixLambdaAccountAndCodeSigning (signed requests under enforce-auth: GetAccountSettings allowed, CreateCodeSigningConfig denied then allowed, GetCodeSigningConfig denied on the config ARN)
  • go test -race ./providers/aws/lambda ./server/aws/lambda
  • go test ./persist/ (completeness check)
  • golangci-lint run --timeout=9m --new-from-rev=upstream/development --allow-parallel-runners --concurrency 2 ./providers/aws/lambda/... ./server/aws/lambda/... ./services/serverless/... ./internal/settle/...: 0 issues
  • Full golangci-lint run (no --new-from-rev) on ./providers/aws/lambda/... ./server/aws/lambda/...: 28 issues on this branch vs 32 on development with v2.14.0 (2 vs 6 with v2.4.0). Every finding on the branch is also on development; no function in server/aws/lambda is over gocyclo 10.

Real-user run (AWS CLI v1 / botocore 1.42 against cloudemu serve -aws-port 4608)

Default mode (no --async-settle):

aws lambda create-function --function-name demo --runtime python3.12 --handler main.handler --role arn:aws:iam::000000000000:role/r --zip-file fileb://v1.zip --environment Variables={STAGE=one}
  -> State Active, LastUpdateStatus Successful, CodeSize 196
aws lambda wait function-active-v2 --function-name demo            -> exit 0
aws lambda publish-version --function-name demo                    -> Version 1, CodeSize 196
aws lambda update-function-code --function-name demo --zip-file fileb://v2.zip   -> CodeSize 208
aws lambda update-function-configuration --function-name demo --environment Variables={STAGE=two}
aws lambda get-function --function-name demo --qualifier 1         -> [1, 196, "one", Active, Successful]
aws lambda get-function --function-name demo                       -> [$LATEST, 208, "two"]
aws lambda list-versions-by-function --function-name demo          -> $LATEST 208, 1 196, 2 208
aws lambda invoke --function-name demo --log-type Tail --payload '{"k":1}' out.json --query LogResult --output text | base64 -d
  -> START RequestId: ... Version: $LATEST / END RequestId: ... / REPORT RequestId: ...  Duration: 1.00 ms  Billed Duration: 1 ms
aws lambda invoke --function-name demo out.json                    -> no LogResult
aws lambda get-account-settings
  -> AccountLimit {TotalCodeSize 322122547200, CodeSizeUnzipped 262144000, CodeSizeZipped 52428800, ConcurrentExecutions 1000, UnreservedConcurrentExecutions 1000}, AccountUsage {TotalCodeSize 612, FunctionCount 1}
aws lambda create-code-signing-config --description demo --tags team=a --allowed-publishers SigningProfileVersionArns=arn:aws:signer:...   -> arn:...:code-signing-config:csc-<17 hex>
aws lambda create-code-signing-config ... --code-signing-policies UntrustedArtifactOnDeployment=Bogus   -> InvalidParameterValueException
aws lambda get-code-signing-config / update-code-signing-config (Enforce) / list-code-signing-configs   -> OK
aws lambda list-tags --resource <csc arn>                          -> {"team": "a"}
aws lambda put-function-code-signing-config --code-signing-config-arn <unknown>   -> CodeSigningConfigNotFoundException
aws lambda put-function-code-signing-config / get-function-code-signing-config / list-functions-by-code-signing-config   -> OK
aws lambda delete-code-signing-config (still attached)             -> ResourceConflictException
aws lambda delete-function-code-signing-config; delete-code-signing-config   -> exit 0
aws lambda get-code-signing-config (deleted)                       -> ResourceNotFoundException
aws lambda get-function --function-name ghost                      -> ResourceNotFoundException: Function not found: arn:aws:lambda:us-east-1:000000000000:function:ghost
aws lambda get-function --function-name demo --qualifier 9         -> ... Function not found: arn:aws:lambda:us-east-1:000000000000:function:demo:9
aws lambda invoke --function-name demo --qualifier nope out.json   -> ... Function not found: arn:aws:lambda:us-east-1:000000000000:function:demo:nope

With serve -async-settle:

create-function                          -> ["Pending", "Creating", null, 196]
invoke (right after create)              -> ResourceConflictException: The operation cannot be performed at this time. The resource arn:aws:lambda:us-east-1:000000000000:function:demo is currently in the following state: 'Pending'. StateReasonCode: 'Creating'
wait function-active-v2                  -> exit 0
update-function-code                     -> ["Active", "InProgress", 208]
publish-version (right after update)     -> ResourceConflictException: ... An update is in progress for resource: arn:...:function:demo
wait function-updated-v2                 -> exit 0
update-function-configuration            -> ["InProgress"], then wait function-updated-v2 exit 0
get-function --qualifier 1               -> [1, 196, "one", Active, Successful]

With serve -enforce-auth: an IAM user whose policy allows only lambda:GetAccountSettings and lambda:CreateCodeSigningConfig on * can call those two operations. list-code-signing-configs is denied with AccessDeniedException ... lambda:ListCodeSigningConfigs on resource: *, and get-code-signing-config is denied on resource: arn:aws:lambda:us-east-1:000000000000:code-signing-config:csc-....

The server was stopped after each run, and nothing is left listening on 4608. Terraform is not installed here, so I did not run it.

Known limits

  • Code signing configs are stored and attached, but deployment-package signatures are not verified (Enforce does not reject unsigned code).
  • During an InProgress update under --async-settle, Invoke runs the new code. Real Lambda keeps serving the previous code until the update completes (follow-up).
  • A version cut by UpdateFunctionCode --publish reports Active/Successful right away rather than following the update's InProgress window.
  • The REPORT log line keeps the existing Duration / Billed Duration fields. It does not add Memory Size / Max Memory Used.
  • Tag keys on code signing configs follow the same (unvalidated) rules as function tags today.

Related Issues

Part of #1518

@NitinKumar004 NitinKumar004 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is careful work. The six items are real gaps and the evidence in the description holds up. Here is what I ran against the pushed SHA:

  • Gates: build, gofmt, go mod tidy, go generate docs (no diff) and -race on ./providers/aws/lambda/... ./server/aws/lambda/... ./server/aws/ all pass. vet/test were re-run scoped and pass, and golangci-lint --new-from-rev reports 0 new issues.
  • Live Terraform (hashicorp/aws 6.66.0, default endpoint override, serve -async-settle): I used aws_lambda_code_signing_config, plus aws_lambda_function (publish=true, code_signing_config_arn, env, tags), aws_lambda_alias, aws_lambda_function_url on the alias, and aws_lambda_permission on the alias.
    • apply: 5 created, no waiter hang, version = "1".
    • plan -detailed-exitcode: 0.
    • Changed the zip and env, then apply: 2 changed, version = "2". The FunctionUpdatedV2/FunctionActiveV2 waiters completed.
    • plan: 0.
    • Afterwards list-versions-by-function shows v1 frozen (CodeSize 193, STAGE=one, its own CodeSha256) and v2/$LATEST at 196/two.
    • destroy: 5 destroyed. get-function then gives ResourceNotFoundException: Function not found: arn:aws:lambda:us-east-1:000000000000:function:tf1646, and list-code-signing-configs is empty.
  • AWS CLI against serve -async-settle:
    • create gives Pending/Creating/null, then wait function-active-v2 exits 0.
    • Invoke while Pending gives ResourceConflictException with the documented message.
    • --log-type Tail decodes to the START/END/REPORT lines.
    • An Event invoke gets no LogResult.
    • The get-account-settings shape matches API_GetAccountSettings exactly.
  • Regression probes for #1535 and #1530 (throwaway tests under -race), all green:
    • The existing TestDeleteAliasNotUndoneByConcurrentUpdate (2000 rounds) and TestEngineDeployDoesNotHoldLock.
    • A slow-engine probe. While a gated engine Deploy is in flight for one function, these calls on another function return in under 100 ms: GetFunction, ListFunctions, GetAccountSettings, GetFunctionCodeSigningConfig, ListCodeSigningConfigs, GetVersionAWSConfig. The new settle overlay (observe) only takes the settle.Set RWMutex, never m.mu.
    • A 3-way race of PutFunctionCodeSigningConfig, UpdateFunction and TagFunction (2000 rounds). The attachment is never lost, so there is no stale write-back of funcData.
    • PutFunctionCodeSigningConfig vs DeleteCodeSigningConfig (1000 rounds). A function is never left attached to a deleted config.
    • A PutFunctionCodeSigningConfig that lands during an engine code deploy survives the deploy's commit.
    • Every new funcData writer (PutFunctionCodeSigningConfig, DeleteFunctionCodeSigningConfig, the version snapshot in publishVersion) is a read-modify-write under m.mu. beginCreateSettle/beginUpdateSettle run inside the existing critical sections, and the engine deploy stays outside the lock.
  • Authz: the --enforce-auth qualifier→ARN mapping is untouched. tagChecks only diverts when the ARN contains :code-signing-config:, which a function ARN cannot. The authz matrix tests pass (including the new TestAuthzMatrixLambdaAccountAndCodeSigning).
  • Checked against the docs: the Pending-blocked set (Invoke, UpdateFunctionCode, UpdateFunctionConfiguration, PublishVersion) and the InProgress-blocked set (the same minus Invoke, plus TagResource) match Lambda function states exactly.

Medium

A published version still reports $LATEST's State, LastUpdateStatus and LastModified (server/aws/lambda/handler.go toVersionConfiguration, ~L1556).
toVersionConfiguration starts from toConfiguration(info, …) (the live $LATEST) and overrides only the version fields. State, StateReason(Code), LastUpdateStatus and LastModified therefore leak from $LATEST into every published version. Reproduced on serve -async-settle:

publish-version                          -> v1 LastModified 19:42:27Z, Successful
update-function-code ($LATEST)           -> InProgress, LastModified 19:42:30Z
get-function --qualifier 1               -> Version 1, Active, LastUpdateStatus InProgress, LastModified 19:42:30Z
list-versions-by-function                -> "1" InProgress 19:42:30Z

In real Lambda a published version is immutable. It keeps its own LastModified (the publish time) and reports Active/Successful whatever $LATEST is doing. Impacts:

  • The LastModified drift happens in default mode too, not only under --async-settle.
  • aws lambda wait function-updated-v2 --qualifier 1, or any tool polling a version or alias ARN, sees InProgress because of an unrelated $LATEST update.

This PR's stated goal is "a published version keeps … the AWS-only settings it was cut with", and versionData.createdAt already holds the right timestamp. Suggested fix:

  • In toVersionConfiguration, when ver.Version != $LATEST, set cfg.LastModified = ver.CreatedAt, cfg.LastUpdateStatus = Successful and cfg.State = Active, and clear StateReason/StateReasonCode.
  • Pending should only carry over for the version cut atomically by Publish=true on create. In real AWS that version is also Pending until creation completes, so that case is fine to keep.
  • Add a wire test asserting that GetFunction --qualifier 1 is unchanged across a $LATEST update.

Low

  1. Invoke during an InProgress update runs the new code (providers/aws/lambda/lambda.go ~L545). The docs say "Invocations continue to use the function's previous code and configuration until the update completes." This is already listed under Known limits and is opt-in only, so I'm not asking for it here. A tracking issue would be good so it doesn't get lost.
  2. CreateFunction with CodeSigningConfigArn is check → create → attach, with no rollback (server/aws/lambda/handler.go ~L1078). checkCodeSigningConfigExists runs before CreateFunction, but the attach runs afterwards. If the config is deleted between the two, the request returns 404 ResourceNotFoundException while the function already exists, and a retry then hits ResourceConflictException. The window is narrow. Either delete the function when the attach fails, or return CodeSigningConfigNotFoundException and keep the function (AWS validates before creating anything). The error type also differs: the provider's NotFound maps to ResourceNotFoundException here, not CodeSigningConfigNotFoundException.
  3. Not-found wording is only partly converted. GetAlias/DeleteAlias still return alias %s not found (providers/aws/lambda/aliases.go L134/L177), and DeleteFunction --qualifier 9 still returns function version %s not found (versions.go L217). The description scopes this deliberately to messages where real text was found. Still, the summary line ("a missing function, version or alias now says Function not found: …") overstates it for alias APIs and DeleteFunction-with-qualifier. Please tighten the description, or convert the version path, which uses the same Function not found: <arn>:<qualifier> form the PR already cites.
  4. Displaced doc comment (server/aws/lambda/iam_actions.go L149). The new codeSigningConfigTags was inserted between resourceTags' doc comment and its func. As a result, // resourceTags sets … now sits on top of the codeSigningConfigTags comment, and resourceTags has none.

Cascade / blast radius

  • services/serverless/driver: additive struct fields only, and the Serverless interface is unchanged. Azure/GCP leave LastUpdateStatus empty, and the handler falls back to Successful, so their output is identical. Only the AWS Lambda handler consumes it. docs/coverage has no diff.
  • internal/settle: two new constants only.
  • Persistence: new snapshot fields are omitempty, and old snapshots restore with an empty version AWSConfig. After such a restore, a published version reports empty AWS-only settings (e.g. no Architectures) instead of $LATEST's. That is acceptable and not a Terraform drift source, since TF reads $LATEST.
  • Mirror rule: AWS-only is justified, because versions, code signing configs and log tails have no Azure/GCP equivalent.

cfg.Role = ver.Role
cfg.MemorySize = ver.Memory
cfg.Timeout = ver.Timeout
cfg.CodeSize = ver.CodeSize

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A published version still inherits State / StateReason / LastUpdateStatus / LastModified from $LATEST via toConfiguration(info, …) above. On serve -async-settle: publish v1, then update-function-code, then get-function --qualifier 1 shows LastUpdateStatus: InProgress and LastModified = the $LATEST update time. Real Lambda versions are immutable: own LastModified (publish time), Active/Successful. Suggest, for ver.Version != $LATEST: cfg.LastModified = ver.CreatedAt, cfg.LastUpdateStatus = lastUpdateStatusSuccessful, cfg.State = Active, clear StateReason/StateReasonCode (keep Pending only for the version cut by Publish=true on create). The LastModified drift also happens with settle off.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, confirmed with your sequence. Fixed in f6700ed. The provider now reports each published version's own lifecycle on FunctionVersion (Active/Successful; Pending/Creating only for the version cut by Publish=true on create, and only until creation completes), and toVersionConfiguration uses it plus ver.CreatedAt (publish time) as LastModified for any non-$LATEST version. Tests: TestSDKPublishedVersionKeepsItsOwnLifecycle (default and async-settle; checks both GetFunction --qualifier 1 and ListVersionsByFunction across a $LATEST update), TestSDKPublishOnCreateVersionIsPendingUntilCreated, TestPublishedVersionReportsItsOwnLifecycle.

Comment thread server/aws/lambda/handler.go Outdated

if req.CodeSigningConfigArn != "" {
if csm, ok := h.fn.(codeSigningManager); ok {
if err := csm.PutFunctionCodeSigningConfig(r.Context(), req.FunctionName, req.CodeSigningConfigArn); err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function is already created when this attach runs. If the code signing config is deleted between checkCodeSigningConfigExists and here, the client gets a 404 ResourceNotFoundException (not CodeSigningConfigNotFoundException) while the function exists, and a retry then 409s. Narrow window, but either roll the function back on failure or map this error the same way as the pre-check.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f6700ed. If the attach fails, the handler deletes the just-created function (no m.mu held; it goes through the normal DeleteFunction) and reports the error the pre-check would have: CodeSigningConfigNotFoundException. The standalone PutFunctionCodeSigningConfig had the same check/attach window, so it maps that case the same way. Test: TestSDKCreateFunctionRollsBackWhenCodeSigningAttachFails uses a backend that deletes the config between the check and the attach, then asserts the typed error, that GetFunction gives ResourceNotFoundException, and that a retry succeeds.

Comment thread server/aws/lambda/iam_actions.go Outdated
@@ -147,6 +147,24 @@ func (c *checkSet) requestTags(tags map[string]string) {
}

// resourceTags sets aws:ResourceTag/* from the tags of the function name.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this resourceTags doc comment is now attached to codeSigningConfigTags (the new func was inserted between the comment and resourceTags). Move it back down above func (c *checkSet) resourceTags.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, moved back above func (c *checkSet) resourceTags in f6700ed.


// A function still Pending creation cannot be invoked
// (ResourceConflictException), matching real Lambda.
if err := m.checkInvocable(&fd); err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Matches the docs for Pending. For InProgress, the docs also say "Invocations continue to use the function's previous code and configuration until the update completes", but here the new code is already live. It's in Known limits, so this is fine for now. Worth a tracking issue so it doesn't get lost.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Leaving it as a known limit in this PR and tracking it as a follow-up: during an InProgress window Invoke should keep serving the previous code/config until the update completes.

@Satyam-Trivedi-ZS

Copy link
Copy Markdown
Collaborator Author

Thanks for the thorough review, @NitinKumar004. All addressed in f6700ed.

Medium: published version inherits $LATEST's lifecycle. Fixed. I put the rule in the provider: FunctionVersion now carries the version's own State/StateReason(Code)/LastUpdateStatus. That is Active/Successful, or Pending/Creating only for the version cut by Publish=true on create, and only until creation completes. The handler renders those fields and uses CreatedAt (the publish time) as LastModified for any non-$LATEST version. Tests:

  • TestSDKPublishedVersionKeepsItsOwnLifecycle: default and --async-settle. It asserts that GetFunction --qualifier 1 and ListVersionsByFunction are unchanged across a $LATEST update, and that $LATEST still reports its own update.
  • TestSDKPublishOnCreateVersionIsPendingUntilCreated
  • TestPublishedVersionReportsItsOwnLifecycle

The wire tests fail on 0e333c7 and pass now.

I re-ran your sequence on serve -async-settle with the AWS CLI:

publish-version                -> 1  Active Successful  LM 06:18:50Z
update-function-code           -> $LATEST Active InProgress LM 06:18:52Z
get-function --qualifier 1     -> 1  Active Successful  LM 06:18:50Z
list-versions-by-function      -> $LATEST InProgress 06:18:52Z, 1 Successful 06:18:50Z
wait function-updated-v2 --qualifier 1 -> exit 0
create-function --publish      -> 1 Pending/Creating; get-function --qualifier 1 Pending, then Active/Successful after wait function-active-v2

One thing I left alone: a version cut by UpdateFunctionCode --publish reports Active/Successful right away. It does not follow the update's InProgress window. I noted this under Known limits.

Low 1: Invoke during InProgress runs the new code. Follow-up. It stays under Known limits. The follow-up is to keep serving the previous code/config until the update completes.

Low 2: CreateFunction check → create → attach. Fixed. When the attach fails, the handler deletes the just-created function and returns CodeSigningConfigNotFoundException, the same error as the pre-check. The delete goes through the normal DeleteFunction, and no lock is held across the calls. The standalone PutFunctionCodeSigningConfig had the same window and now maps that case the same way. Test: TestSDKCreateFunctionRollsBackWhenCodeSigningAttachFails. Its backend deletes the config between the check and the attach. The test asserts the typed error and that GetFunction returns ResourceNotFoundException afterwards. It also checks that a retry succeeds, and that PutFunctionCodeSigningConfig returns CodeSigningConfigNotFoundException.

Low 3: not-found wording. Partly changed, and the description now matches:

  • GetAlias now returns Cannot find alias arn: <function ARN>:<alias>. That is the real GetAlias text from an SDK v2 error captured in lambroll#386. Test: TestGetAliasNotFoundMessage.
  • DeleteAlias/UpdateAlias for a missing alias and DeleteFunction --qualifier for a missing version are unchanged. I couldn't find real AWS text for them. The DeleteAlias API reference doesn't even list ResourceNotFoundException.
  • I tightened the summary line and the item row in the description.

Low 4: displaced doc comment. Fixed. resourceTags' comment is back above its func.

Locking is unchanged. The new version-lifecycle read only takes the settle.Set lock, the same as observe. The cut-on-create flag is set inside publishVersion's existing m.mu section, and the engine deploy path is untouched.

Checks: go build ./... passes. go test -race passes on ./providers/aws/lambda/... ./server/aws/lambda/... ./server/aws/ ./persist/.... The full go test -p 2 ./... (run in chunks) is green. go generate ./... gives no diff, and golangci-lint --new-from-rev=upstream/development on the touched packages reports 0 issues.

@NitinKumar004 NitinKumar004 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up. I re-checked every point from the last round against f6700ed and re-ran the gates and the live flows on the new head.

What I checked

  • Earlier findings:
    • Medium (published version reports $LATEST's State, LastUpdateStatus and LastModified): fixed. toVersionConfiguration now calls applyVersionLifecycle for any non-$LATEST version. That sets LastModified from ver.CreatedAt and takes State/StateReason/LastUpdateStatus from the provider's new observeVersion, which reports Active/Successful, or Pending/Creating only for the version cut by Publish=true on create while creation is still running. I re-ran my earlier sequence on serve -async-settle. After publish-version (v1 at 20:58:37Z) and then update-function-code ($LATEST InProgress at 20:58:41Z):
      • get-function --qualifier 1 gives 1, Active, Successful, 20:58:37Z, CodeSize 201, STAGE=one.
      • get-function-configuration --qualifier 1 gives the same.
      • list-versions-by-function shows $LATEST InProgress and 1 Successful.
      • wait function-updated-v2 --qualifier 1 exits 0 while $LATEST is still updating.
      • An alias on v1 (--qualifier live) reports v1's lifecycle and LastModified with the alias ARN.
    • Low 1 (Invoke during InProgress runs the new code): still open, now tracked in #1665. That's fine by me for this PR.
    • Low 2 (CreateFunction check, create, attach with no rollback): fixed. attachCreateCodeSigningConfig deletes the just-created function when the attach fails, then re-runs checkCodeSigningConfigExists, so the client gets CodeSigningConfigNotFoundException and not the function's ResourceNotFoundException. putFunctionCodeSigningConfig maps the same race the same way. TestSDKCreateFunctionRollsBackWhenCodeSigningAttachFails covers the typed error, the absent function and a successful retry.
    • Low 3 (not-found wording): addressed. GetAlias now returns Cannot find alias arn: <function ARN>:<alias> (confirmed via the CLI). The description now says plainly that DeleteAlias/UpdateAlias and DeleteFunction --qualifier keep emulator text (alias ghost not found, function version 9 not found, both confirmed). Those are tracked in #1665.
    • Low 4 (displaced doc comment in iam_actions.go): fixed. The resourceTags comment is back above its func.
  • Gates on f6700ed:
    • go build ./... passes.
    • go vet passes on settle, aws lambda provider and handler, and serverless.
    • Scoped go test -count=1 passes on internal/settle, providers/aws/lambda/..., server/aws/lambda/..., services/serverless/..., persist/ and server/aws -run Lambda.
    • -race passes on providers/aws/lambda/... and server/aws/lambda/....
    • go mod tidy and go generate ./... leave no diff, and no touched file fails gofmt.
  • Live Terraform (hashicorp/aws v6.70.0, default lambda endpoint override, serve -async-settle): I used aws_lambda_code_signing_config, plus aws_lambda_function (publish=true, code_signing_config_arn, env, tags), aws_lambda_alias and aws_lambda_function_url on the alias.
    • apply: 4 added, version = "1".
    • plan -detailed-exitcode: 0.
    • Changed the zip and set STAGE=two, then apply: 2 changed, version = "2".
    • plan: 0.
    • list-versions-by-function then shows each version with its own lifecycle:
      • 1: CodeSize 201, one, Active/Successful, LastModified 20:59:47Z.
      • 2 and $LATEST: CodeSize 211, two.
    • destroy: 4 destroyed. get-function then returns Function not found: arn:aws:lambda:us-east-1:000000000000:function:tf1646, and list-code-signing-configs is empty.
  • AWS CLI under -async-settle:
    • create-function gives Pending/Creating/null, then wait function-active-v2 exits 0.
    • create-function --publish returns v1 as Pending/Creating, and wait function-active-v2 --qualifier 1 exits 0.
    • update-function-code --publish gives v2 Active/Successful. That is the known limit listed in #1665.
    • publish-version during InProgress gives ResourceConflictException.
  • Architecture: the version lifecycle rule lives in the provider (observeVersion). The handler only maps it, with an Active/Successful fallback for a backend that reports nothing. FunctionVersion gains four additive fields and the Serverless interface is unchanged, so Azure/GCP are unaffected. cutWhilePending is not persisted, which is consistent with settle windows not being persisted (a restored function is terminal, so its version is Active). The create rollback is a compensating step in the handler. That is acceptable because the CodeSigningConfigArn on CreateFunction is itself composed in the handler (the portable CreateFunction has no such field).
  • No AI attribution in the PR body or commits.

High

None.

Medium

  1. New functions over the complexity limit of 10 (server/aws/lambda). I ran golangci-lint (v2.14.0) on ./providers/aws/lambda/... ./server/aws/lambda/... without --new-from-rev and compared it with the merge base: 32 issues there, 37 here.

    • gocyclo: ServeHTTP goes 12 to 15, create 11 to 13, invoke 11 to 13, and classify (12) and Matches (12) are new over the limit.
    • goconst: CodeSigningConfig, CodeSigningConfigArn and Tags in code_signing.go, and Tags in handler.go L442.

    --new-from-rev misses the gocyclo ones because the function signature lines did not change. The repo keeps cyclomatic complexity at 10 or below, so please split the code-signing and account-settings branches out of ServeHTTP/Matches/classify into small helpers (as classifyCodeSigningConfigs already does), and the same for create and invoke. This is the last thing I would like to see before approving.

Low

  1. Invoke during InProgress, UpdateFunctionCode --publish lifecycle, and DeleteAlias/UpdateAlias/DeleteFunction --qualifier wording remain as documented in #1665. No action needed in this PR.

Comment thread server/aws/lambda/handler.go Outdated
return
}

if isCodeSigningConfigARN(a.item) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: without --new-from-rev, golangci-lint shows ServeHTTP going from cyclomatic 12 to 15 with this branch and the account-settings one, and create/invoke going from 11 to 13. Matches and classify are new over 10. --new-from-rev hides these because the func signature lines didn't change. Moving the code-signing/account-settings dispatch into a helper, like classifyCodeSigningConfigs, would bring them back under the limit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in dca3dcf. The code signing and account settings branches now live in small helpers: isCodeSigningOrAccountPath (Matches), classifyCodeSigningOrAccount (classify), and serveCodeSigningOrAccount plus serveTagsRoute and serveConcurrencyRoute (ServeHTTP). create and invoke are split the same way. gocyclo, full run without --new-from-rev: ServeHTTP 15 → 10, Matches 12 → 10, classify 12 → 10, create 13 → 8, invoke 13 → 7. Behaviour is unchanged and the existing tests pass. Details are in the summary comment.

- A published version now freezes its CodeSize, description, environment
  and AWS-only settings (VpcConfig, DeadLetterConfig, TracingConfig,
  Architectures, EphemeralStorage, Layers) at publish time, so
  UpdateFunctionCode/Configuration on $LATEST no longer changes what
  GetFunction --qualifier N or ListVersionsByFunction report for it.
- Under AsyncSettle a new function reports State Pending (StateReasonCode
  Creating, no LastUpdateStatus) and an update reports LastUpdateStatus
  InProgress. While Pending, Invoke, UpdateFunctionCode,
  UpdateFunctionConfiguration and PublishVersion return
  ResourceConflictException; while InProgress the updates, PublishVersion
  and TagResource do. The InProgress gate also covers an in-flight engine
  deploy. Publish=true on Create/UpdateFunctionCode is not blocked by the
  window its own write opened. Settle off keeps the old behaviour.
- Invoke with LogType=Tail returns X-Amz-Log-Result: the base64 of the
  last 4 KB of the execution log the provider already renders.
- GetAccountSettings (GET /2016-08-19/account-settings).
- Code signing configs: Create/Get/List/Update/DeleteCodeSigningConfig,
  Put/Get/DeleteFunctionCodeSigningConfig, ListFunctionsByCodeSigningConfig,
  CodeSigningConfigArn on CreateFunction, and tagging a code signing config
  ARN, with IAM checks per the Service Authorization Reference. State is
  in the Lambda snapshot.
- ResourceNotFoundException for a missing function, version or alias
  reads "Function not found: <function ARN>[:<qualifier>]".

Part of stackshy#1518
…CreateFunction on code signing attach failure

- A published version no longer inherits $LATEST's State, StateReason(Code),
  LastUpdateStatus and LastModified. The provider now reports each version's
  own lifecycle on FunctionVersion (Active/Successful, or Pending only for the
  version CreateFunction(Publish=true) cut, until creation completes), and the
  handler renders LastModified from the version's publish time. GetFunction
  --qualifier N and ListVersionsByFunction are unchanged by a later $LATEST
  update, in default mode and under --async-settle.
- CreateFunction with CodeSigningConfigArn deletes the just-created function
  when the attach fails (config deleted after the existence check) and reports
  CodeSigningConfigNotFoundException, so a retry does not hit
  ResourceConflictException. PutFunctionCodeSigningConfig maps the same race to
  CodeSigningConfigNotFoundException.
- GetAlias for a missing alias returns real Lambda's text:
  "Cannot find alias arn: <function ARN>:<alias>".
- Move resourceTags' doc comment back above its func.
Split the code signing and account settings branches out of ServeHTTP,
Matches and classify into small helpers, group the two concurrency routes,
and move CreateFunction's validation/config build and Invoke's DryRun and
synchronous response writing into their own functions. Name the repeated
response field strings as constants. No behaviour change.
@Satyam-Trivedi-ZS
Satyam-Trivedi-ZS force-pushed the fix/aws-lambda-parity-1518 branch from f6700ed to dca3dcf Compare October 24, 2026 16:11
@Satyam-Trivedi-ZS

Copy link
Copy Markdown
Collaborator Author

Thanks @NitinKumar004. I rebased onto development and addressed the Medium in dca3dcf.

Medium: complexity and goconst in server/aws/lambda

I reproduced your numbers first: golangci-lint v2.14.0, full run with no --new-from-rev, gave 32 on development and 37 on f6700ed. Changes:

  • ServeHTTP: the tags branch moved to serveTagsRoute. The code signing and account settings routes go to serveCodeSigningOrAccount as one multi-value case, and the provisioned and reserved concurrency routes go to serveConcurrencyRoute. The switch is still exhaustive over route, so the exhaustive linter stays clean.
  • Matches: the code signing and account settings path checks moved to isCodeSigningOrAccountPath.
  • classify: the same three routes moved to classifyCodeSigningOrAccount, which is checked after provisioned concurrency, so the match order is unchanged.
  • create: split into validateCreate, which runs the same checks in the same order before any state change, and createFunctionConfig, which builds the config and resolves the code and layers.
  • invoke: split into invokeDryRun and writeInvokeResult.
  • goconst: added the bodyTags, bodyCodeSigningConfig, bodyCodeSigningConfigArn and bodyFunctionName constants. They are used in code_signing.go and in handler.go ListTags.
func base (development) f6700ed now
(*Handler).ServeHTTP 12 15 10
(*Handler).Matches ≤10 12 10
classify ≤10 12 10
(*Handler).create 11 13 8
(*Handler).invoke 11 13 7

The new helpers are all 6 or lower. Full lint on ./providers/aws/lambda/... ./server/aws/lambda/...:

  • v2.14.0: 32 on development, 28 on this branch.
  • v2.4.0: 6 on development, 2 on this branch.

Every finding left on the branch is also on development. The drop comes from the three pre-existing gocyclo hits (ServeHTTP/create/invoke) and one gocritic hugeParam in policy.go that the branch already removed.

The change is a refactor only, with no behaviour change. Testing:

  • go test -race passes on providers/aws/lambda/..., server/aws/lambda/..., server/aws (including the authz matrix), persist/... and internal/settle/....
  • The full go test -p 1 ./... passes, run in chunks.
  • go build ./... passes, and go generate ./... leaves no diff.
  • AWS CLI smoke test on serve -async-settle:
    • create gives Pending/Creating, then wait function-active-v2 exits 0, and invoke returns 200.
    • publish-version, then update-function-code ($LATEST InProgress), then get-function --qualifier 1 gives 1 Active Successful with its own LastModified.
    • get-account-settings works.
    • Code signing config create/get/update/list/tags/put/get/delete all behave as before.
    • DryRun returns 204, and an unknown qualifier gives ResourceNotFoundException.

Low

No change in this PR; it stays tracked in #1665.

Rebase

The branch is rebased onto development (a624a95). The only conflict was internal/settle/settle.go: I kept both the Batch constants (DefaultBatchJobPhase, DefaultBatchJobRunDwell) and the Lambda ones. Nothing else upstream touched providers/aws/lambda, server/aws/lambda or services/serverless, and go generate ./... gives no diff after the rebase. I also updated the lint lines in the description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants