Repository navigation
fix(aws-sqs): per-minute queue gauges, in-flight NotVisible, caller SenderId - #1539
Conversation
|
One project rule to fix before merge: please remove the AI attribution from this PR.
The repo doesn't accept AI attribution lines in commits or PR descriptions. Thanks! |
NitinKumar004
left a comment
NitinKumar004
left a comment
There was a problem hiding this comment.
Thanks for this. The SenderId half is solid and I could not break it. The gauge half computes the right values, but the way they are published (one sample per queue action instead of one per minute) breaks the two alarms people actually put on these metrics. Details below.
Issue coverage (#1514)
| Item | Status | Evidence |
|---|---|---|
| (Medium) Remaining SQS gauges | Partly delivered | Values are correct when sampled. Publishing per action means idle queues never report and busy queues over-report (see High and Medium below) |
| (Low) ApproximateNumberOfMessagesVisible | Partly delivered | Emitted on actions, stale while the queue is idle |
| (Low) ApproximateNumberOfMessagesNotVisible | Partly delivered | Counts in-flight only, matches the doc. Same staleness |
| (Low) ApproximateNumberOfMessagesDelayed | Partly delivered | Same staleness. A delayed message that becomes visible is never reflected until another action |
| (Low) ApproximateAgeOfOldestMessage | Partly delivered | Poison-pill skip and DLQ reset match the doc. The age never grows on a stuck queue, which is what this metric exists for |
| (Low) SenderId always the account id | Delivered | IAM user, second IAM user and assumed-role sessions each get their own id under --enforce-auth; auth off matches GetCallerIdentity; survives a --persist restart |
| (Low) NumberOfMessagesMoved | Correctly not added | Not an AWS/SQS metric per the metrics page; agreed |
Both new SDK tests fail against the base commit and pass on this branch.
High
1. Gauges are only sampled when someone touches the queue, so a stuck queue never alarms. providers/aws/sqs/metrics.go:16, call sites at sqs.go:544, 907, 1189, 1232, 1360, 1592
Real SQS publishes the queue-state gauges every minute for every active queue (a queue that holds messages, or was accessed in the last 6 hours). Here they are only written as a side effect of send/receive/delete/visibility/purge/move. The main reason anyone alarms on ApproximateAgeOfOldestMessage or ApproximateNumberOfMessagesVisible is to catch consumers that stopped, and when consumers stop nothing touches the queue, so nothing is published.
Reproduced on serve with the default 1s tick: I created a queue and two alarms (ApproximateAgeOfOldestMessage > 30, Maximum, 60s, and ApproximateNumberOfMessagesVisible > 0, Maximum, 60s), then sent one message with --delay-seconds 5 and did nothing else. After 140 seconds, GetQueueAttributes shows ApproximateNumberOfMessages=1, but the only datapoints are the ones from the send (Visible 0, Age 0), and both alarms are still OK. On AWS both would be in ALARM. The same gap shows up whenever a delay or visibility timeout runs out without any further traffic, as the PR notes say.
Fix: move the gauges onto the shared serve scheduler instead of the request paths. Give the SQS mock a Tick(now time.Time) bool (the same config.Tickable shape CloudWatch, SSM and CloudFormation use), add it to Provider.Tickables() in providers/aws/aws.go, and on each tick publish one sample per active queue whenever a full minute of provider-clock time has passed since that queue's last sample. Track a lastAccessedAt on the queue to apply the 6 hour rule. Keep the counters (NumberOfMessagesSent/Received/Deleted, NumberOfEmptyReceives, SentMessageSize) on the request path as they are now. Keep the "publish after releasing the lock" logic you added. Tests can drive Tick with the FakeClock.
Medium
2. Per-action sampling inflates Sum, Average and SampleCount, so backlog alarms fire too early. providers/aws/sqs/metrics.go:107
Every send, receive, delete and visibility change appends 3 to 5 gauge datapoints. A 60s CloudWatch bucket then holds one sample per action instead of one per minute. Statistics other than Maximum/Minimum stop meaning what they mean on AWS:
- Terraform
aws_cloudwatch_metric_alarmonApproximateNumberOfMessagesVisible,statistic = "Sum",threshold = 10. I sent 6 messages, so the real backlog was 6.GetMetricStatisticsreturnedSum=21, SampleCount=6, Maximum=6, and the alarm went toALARM("Threshold crossed"). On AWS the minute's single datapoint is 6 and the alarm staysOK. - 50 sends in a minute gave
SampleCount=52, Average=26.5, Maximum=52for one bucket.
There is also a cost problem. sampleGauges walks every message in the queue, so each send is now O(queue depth), and pushing N messages into one queue is O(N²). The CloudWatch mock appends datapoints with no retention cap (providers/aws/cloudwatch/cloudwatch.go:159), so a long-running serve now keeps about 3 to 5 extra datapoints per SQS operation forever, and each one also triggers alarm evaluation. The per-minute tick in finding 1 fixes all of this at once, so I would not patch it separately.
Low
3. Assumed-role SenderId is AROACLOUDEMU0000000000:<session> for every role. services/messagequeue/driver/driver.go:161, server/aws/sqs/handler.go:72
The comments say an assumed role gives "<role id>:<session>", and the ReceiveMessage reference says the same (the IAM role id). Under --enforce-auth I assumed role worker (RoleId AROAFIFADYUHAD3ARMCIY). SenderId came back as AROACLOUDEMU0000000000:sess-1, because STS mints every AssumedRoleId from the fixed assumedRoleIDPrefix (server/aws/sts/operations.go:101). That bug is in STS, not in this PR, and SenderId correctly matches GetCallerIdentity. But two different roles with the same session name produce the same SenderId. Either fix STS to use the role's real RoleId (a small follow-up), or reword the comments so they don't promise the role id.
4. Messages delivered by SNS or EventBridge keep the account id as SenderId. AWS reports the delivering service's principal. The PR notes this already. It's fine as a follow-up, but it would be good to track it on #1514 so it doesn't get lost.
5. GetQueueAttributes and the gauge can disagree on NotVisible. The PR notes say the attribute still counts delayed messages as not visible, while the gauge counts only in-flight messages. A user comparing the two will see different numbers. Again fine as a follow-up, but please track it on the issue.
AWS docs divergences
- Available CloudWatch metrics for Amazon SQS and Monitoring SQS with CloudWatch: AWS pushes the gauges once a minute for active queues, including idle queues that still hold messages. The PR samples on actions instead (findings 1 and 2).
- ReceiveMessage, SenderId: for a role, SenderId is the IAM role id plus the session. Here it's a fixed placeholder prefix, inherited from STS (finding 3).
- Metric names, units (Count, and Seconds for the age), the single
QueueNamedimension, the FIFO-onlyApproximateNumberOfGroupsWithInflightMessages, the standard-queue poison-pill exclusion and the DLQ age reset all match the doc.
Terraform and CLI
- Terraform (hashicorp/aws 6.66.0, default endpoint overrides for sqs, cloudwatch and sts). Resources:
aws_sqs_queue(standard with a redrive policy, a DLQ, and a FIFO queue) plus twoaws_cloudwatch_metric_alarmonApproximateNumberOfMessagesVisible. Apply created all 5 with no waiter hang.plan -detailed-exitcodereturned 0, and still 0 after sending traffic. I then changedvisibility_timeout_secondsand the alarm threshold: 2 resources changed, plan 0 again. Destroy removed all 5, after whichGetQueueUrlreturnedNonExistentQueueandDescribeAlarmsreturned 0 alarms. - AWS CLI with auth off:
receive-message --attribute-names SenderIdreturnedAIDA9F86D081884C7D65, the same assts get-caller-identityUserId.list-metricsshowed all four gauges, andget-metric-statisticsreturned the expected Count/Seconds values.send-message-batchworks. Sending to a missing queue returnsAWS.SimpleQueueService.NonExistentQueue. - AWS CLI with
--enforce-auth: the admin user, a second IAM user and an assumed-role session (send-message-batch) each got their own SenderId, matching IAMUserIdand GetCallerIdentity. A bad signature was rejected withSignatureDoesNotMatch. A role with no policy gotAccessDeniedExceptionfor sqs:SendMessage, so the auth gate is not bypassed. SenderIds came back unchanged after a--persistrestart.
What I verified
- Built and tested the pushed head
c2404867:go vetpasses;go test -racepasses onproviders/aws/sqs,server/aws/sqs,services/messagequeueandserver/wire/awsidentity; the SQS/STS/SNS/Lambda tests inserver/awspass. gofmtis clean,go mod tidyleaves no diff, andgo generateleaves no diff.golangci-lint2.14 reports no new issues on the touched packages against the merge base.- The new code is almost fully covered (metrics.go 100%). The
providers/aws/sqsandserver/aws/sqspackages overall are at 84% and 83%. - A throwaway concurrent test (send, batch send, receive, delete, change visibility, purge, move task, clock advance, with a monitoring backend that calls back into the queue) passed
-race -count=3with no deadlock. - Both new SDK tests fail on the base commit and pass here.
- CI is green on this head.
Verdict
Request changes. SenderId is ready. The gauges need to move to a per-minute tick (finding 1), which also fixes finding 2. After that I'm happy to approve.
| // | ||
| // Real SQS pushes these once a minute for every active queue, where any action | ||
| // on the queue (or any message in it) keeps it active. The emulator has no | ||
| // background publisher, so it samples the gauges whenever an action changes a |
There was a problem hiding this comment.
The gauges are only sampled when someone acts on the queue, so a queue whose consumers stopped never publishes again. I reproduced this on serve: one delayed message and nothing else for 140s. GetQueueAttributes shows 1 message, but Age/Visible alarms stay OK because the only datapoints are 0s from the send. AWS publishes every minute for active queues (holding messages or accessed in the last 6h). Could this become a Tick(now) on the SQS mock (same Tickable shape as CloudWatch/SSM/CloudFormation), added to Provider.Tickables(), emitting one sample per active queue per minute of provider-clock time? The counters can stay on the request path.
There was a problem hiding this comment.
Fixed in f1bb811. The gauges now come from (*Mock).Tick on the serve scheduler, registered in Provider.Tickables() ahead of CloudWatch. Each active queue (holding messages, or accessed in the last 6h) publishes one sample per minute of provider-clock time. Your repro on serve: one message with --delay-seconds 5 and no further traffic. Both alarms (Age > 30 and Visible > 0) were OK at t+60s and ALARM from t+90s. Age datapoints were 0/61/121/182s and Visible 0/1/1/1, each SampleCount 1. Covered by TestTickStuckQueueKeepsReportingAndAges and TestTickStopsForEmptyQueueIdleSixHours.
| // itself and publishes after releasing it, because a CloudWatch alarm on the | ||
| // gauge may notify an SNS topic that delivers straight back into this queue. | ||
| // Callers must not hold qd.mu. A no-op without a monitoring backend. | ||
| func (m *Mock) emitQueueGauges(qd *queueData) { |
There was a problem hiding this comment.
Because this runs on every send/receive/delete, a minute bucket holds one sample per action. With 6 sends, a TF alarm on ApproximateNumberOfMessagesVisible with statistic Sum and threshold 10 went to ALARM (Sum=21, SampleCount=6); on AWS the single per-minute datapoint is 6 and it stays OK. Each call also walks every message (O(depth) per send) and appends 3-5 datapoints that CloudWatch never trims. The per-minute tick fixes all three.
There was a problem hiding this comment.
Fixed by the same change (f1bb811). Requests no longer sample gauges, so each minute holds one datapoint. Six sends now give Visible Sum=6, SampleCount=1, and the Sum>10 alarm stays OK. Fifty batch sends give one sample of 50 per minute. The O(depth) walk now runs once per queue per minute on the tick, not on every send. Tests: TestTickPublishesOneSamplePerMinute, TestRequestsPublishCountersButNoGauges, TestSDKSQSGaugeSumIsOneSamplePerMinute.
| // ReceiveMessage, DeleteMessage against this or another queue) never | ||
| // deadlocks on it. | ||
| m.deliverToESM(ctx, qd) | ||
| m.emitQueueGauges(qd) |
There was a problem hiding this comment.
With the gauges moved to a periodic tick, this call (and the ones in receive, delete, change visibility, purge and move task) can go. Only a lastAccessedAt touch would be needed here, for the 6 hour active-queue rule.
There was a problem hiding this comment.
Done. All per-action gauge calls are removed (send, receive, delete, change visibility, purge, DLQ redrive, move task). Those paths now only record lastAccessed through accessQueue, and the counters stay as they were. lastAccessed and lastSampleAt are in the SQS snapshot (TestSnapshotKeepsGaugeSchedule). Publishing still happens after the queue locks are released, on both the tick and the counter paths (TestTickPublishesOutsideQueueLocks).
| // clock. The zero value means no schedule. Ignored by non-Azure providers. | ||
| ScheduledEnqueueTime time.Time | ||
| // SenderID is the IAM unique id of the caller that sent the message (an | ||
| // IAM user id, or "<role id>:<session name>" for an assumed role), which |
There was a problem hiding this comment.
Small one: under --enforce-auth an assumed role actually yields AROACLOUDEMU0000000000:<session> for every role, because STS mints AssumedRoleId from a fixed prefix (server/aws/sts/operations.go:101). I tested role worker with RoleId AROAFIFADYUHAD3ARMCIY and got AROACLOUDEMU0000000000:sess-1. That's not this PR's bug, and SenderId correctly matches GetCallerIdentity. Either a follow-up to make STS use the role's RoleId, or reword this so it doesn't promise the role id.
There was a problem hiding this comment.
Fixed in STS (b232ff9). AssumeRole, AssumeRoleWithWebIdentity and AssumeRoleWithSAML now return <role RoleId>:<session> when IAM knows the role, and fall back to the synthetic prefix only when it does not. The minted session identity carries that id, so GetCallerIdentity and SenderId follow it. On serve, roles worker and auditor with the same session sess-1 gave SenderIds AROATFI65QRLEGW5MUR7X:sess-1 and AROAEUYTHTYGJQPNI2IHR:sess-1, which match each role's RoleId. Tests: TestSDKAssumedRoleIDUsesRoleID, TestSDKAssumedRoleIDWithoutIAM, TestSDKSQSSenderIDMatchesCallerIdentity.
c240486 to
f1bb811
Compare
Satyam-Trivedi-ZS
changed the title
|
Thanks for the thorough review, @NitinKumar004. I've pushed High 1: gauges only sampled on activity. Fixed in
Medium 2: per-action sampling inflates Sum/Average/SampleCount. Fixed by the same commit.
Low 3: every assumed role gets
Low 4: SNS/EventBridge deliveries keep the account ID as SenderId. This is a follow-up.
Low 5: GetQueueAttributes NotVisible counted delayed messages. Fixed in
Attribution. History rewritten into two clean commits with no trailer, and the description footer is removed. Checks
|
NitinKumar004
left a comment
NitinKumar004
left a comment
There was a problem hiding this comment.
Thanks for the rework. The gauges now run on the serve scheduler, and the STS change fixes the assumed-role id. Both earlier blocking findings are fixed, and I found nothing new above Low.
Earlier findings
| # | Earlier finding | Status | Evidence |
|---|---|---|---|
| 1 (High) | Gauges only sampled on queue actions, so a stuck queue never alarms | Fixed | (*Mock).Tick is registered in Provider.Tickables() ahead of CloudWatch. On serve I sent 6 messages and 1 with --delay-seconds 20, received one, then left the queue alone. Datapoints kept coming once a minute: Visible 0, 5, 6, 6; NotVisible 0, 1, 1, 1; Delayed 0, 1, 0, 0; Age 14, 74, 135, 196, 257. The Age > 30 alarm went to ALARM with no further traffic |
| 2 (Medium) | Per-action sampling inflates Sum and SampleCount, O(depth) per send | Fixed | Every bucket has SampleCount 1. A TF alarm on Visible with statistic = "Sum" and threshold 10 stayed OK with a backlog of 6. Request paths no longer call sampleGauges. The walk runs once per queue per minute |
| 3 (Low) | Assumed-role SenderId used the fixed AROACLOUDEMU0000000000 prefix |
Fixed | Roles worker (AROAZFVPMPCH72HJTRAKF) and auditor (AROAEUWOZVH6WE2LTAVBW), both with session sess-1: AssumedRoleId, GetCallerIdentity UserId and the received SenderId are <RoleId>:sess-1 for each. AssumeRoleWithWebIdentity uses the RoleId too. An unknown role falls back to the synthetic prefix. requestedRole compares the full ARN, so a same-named role in another account can't lend its id |
| 4 (Low) | SNS and EventBridge deliveries keep the account id as SenderId | Open, tracking only | Still listed under Follow-ups in the PR body. #1514 has no comment that tracks it yet. Please add one, or open a ticket |
| 5 (Low) | GetQueueAttributes NotVisible disagrees with the gauge |
Fixed | Both now come from sampleGauges. On serve the attribute said NotVisible 1 and Delayed 0, the same values as the gauge in that minute |
What I checked
- The PR still merges cleanly onto current
origin/development(785367e, 46 commits ahead of the merge base). I ran the gates on the merge result and on the head. go build ./...passes.go vetpasses on the touched packages.go test -p 1passes onproviders/aws,providers/aws/sqs,server/aws,server/aws/sqs,server/aws/sts,services/messagequeue,persist,server/wire/awsidentityandserver/serverkit.-racepasses onproviders/aws/sqs,server/aws/sqs,server/aws/sts, and on the SQS and STS tests inserver/aws.gofmtis clean on the changed files.go mod tidyandgo generate ./...leave no diff.golangci-lint2.14 on the touched packages, with no--new-from-rev, compared to the merge base: one new finding (see Low 2). Thecloudemu-rolegoconst hit in STS goes away. There are no new gocyclo or funlen hits.- Terraform, hashicorp/aws 6.70.0, with a private provider download. The config had a DLQ, a standard queue with a redrive policy, a FIFO queue, and two
aws_cloudwatch_metric_alarm(Visible with Sum, Age with Maximum). Apply created 5 resources.plan -detailed-exitcodereturned 0, and still 0 after traffic. I changed the threshold andvisibility_timeout_seconds: 2 changed, plan 0 again. Destroy removed all 5. After thatGetQueueUrlreturnedNonExistentQueueandDescribeAlarmsreturned 0 alarms. - AWS CLI: SenderId for the default caller matched
get-caller-identityUserId (AIDA9F86D081884C7D65).send-message-batchunder assumed-role credentials carried the role session id. The FIFOApproximateNumberOfGroupsWithInflightMessageswent from 0 to 1 after one in-flight receive. The empty DLQ published Visible, NotVisible and Delayed but no Age, as the metrics page says ("reported if the queue contains at least one active message"). A send to a missing queue returnedAWS.SimpleQueueService.NonExistentQueue. --persistrestart: after a stop and restart on the same state file, the age series picked up at the next minute (196 then 257) with SampleCount 1. There was no duplicate sample, and SenderId came back unchanged.- Snapshot reads
lastSampleAtunderqd.mu.lastAccessedis atomic.Tickpublishes after it has released every queue lock. - No AI attribution lines in the commits or the PR body.
High
None.
Medium
None.
Low
- Gauge datapoints now pile up in CloudWatch forever on a long-running
serve.providers/aws/sqs/metrics.go:131. Every active queue now adds 3 to 5 datapoints a minute, about 5,000 to 7,000 a day per queue. A queue stays active while it holds messages, and a Lambda event source mapping keeps polling it, so in practice that is forever. The CloudWatch mock keeps every datapoint in one slice per (namespace, metric) with no retention (providers/aws/cloudwatch/cloudwatch.go:159). Each read and each alarm evaluation scans that slice in full. Each tick that publishes also returns true, so--persistrewrites the whole snapshot, growing datapoints included, once a minute. Real CloudWatch keeps 1-minute data for 15 days. This isn't this PR's bug, but it is the first producer that never stops. A follow-up ticket for CloudWatch retention (and maybe a per-series index) would cover it. - New goconst hit:
"QueueName"now appears 5 times in the package (metrics.go:158,sqs.go:541/691/962/1179). AdimQueueNameconst next tometricsNamespaceclears it. - Age can go past the retention period.
MessageRetentionPeriodis stored but never enforced (no expiry anywhere inproviders/aws/sqs). So on a stuck queueApproximateAgeOfOldestMessagekeeps growing past the retention period, and the message is never dropped. On AWS it would expire at the retention limit, and the age would fall back. This gap was there before this PR, but the per-minute age makes it visible. Worth a separate ticket. - The SNS and EventBridge SenderId follow-up (earlier finding 4) still needs a tracking comment on #1514.
| return false | ||
| } | ||
|
|
||
| _ = m.monitoring.PutMetricData(context.Background(), data) |
There was a problem hiding this comment.
Not blocking. This is now the first thing that writes CloudWatch datapoints with no end: 3 to 5 per active queue per minute. A queue with messages, or one an ESM polls, stays active forever. The CloudWatch mock never trims (cloudwatch.go:159) and scans the whole per-metric slice on every read and alarm evaluation. Tick also returns true each minute, so --persist rewrites the full snapshot every minute. Could we open a follow-up for CloudWatch retention (AWS keeps 1-minute data for 15 days)?
|
|
||
| // gaugeData builds the datums of one gauge sample. | ||
| func gaugeData(queue string, fifo bool, g queueGauges, now time.Time) []mondriver.MetricDatum { | ||
| dims := map[string]string{"QueueName": queue} |
There was a problem hiding this comment.
golangci-lint (goconst) flags "QueueName": it now appears 5 times in the package. A dimQueueName const next to metricsNamespace would cover this and the four uses in sqs.go.
AssumeRole, AssumeRoleWithWebIdentity and AssumeRoleWithSAML returned the fixed AssumedRoleId prefix AROACLOUDEMU0000000000 for every role. Real STS returns "<role id>:<session name>", where the role id is the role's unique RoleId. Use the IAM role's RoleId when the role is known and keep the synthetic prefix only when no IAM driver knows it. GetCallerIdentity and the SQS SenderId derive from this id, so two roles with the same session name no longer collapse onto one identity. Part of stackshy#1514
Publish the AWS/SQS queue-state gauges (ApproximateNumberOfMessagesVisible, ApproximateNumberOfMessagesNotVisible, ApproximateNumberOfMessagesDelayed, ApproximateAgeOfOldestMessage and, for FIFO queues, ApproximateNumberOfGroupsWithInflightMessages) from a Tick on the shared serve scheduler: one sample per active queue per minute of provider-clock time, where a queue is active while it holds messages or was accessed in the last six hours. A queue whose consumers stopped keeps reporting a growing backlog and age, and each minute holds one sample, so Sum/Average/SampleCount match real SQS. The counters stay on the request paths and every metric is published after the queue lock is released. The access and sample times are part of the SQS snapshot. GetQueueAttributes ApproximateNumberOfMessagesNotVisible now counts in-flight messages only, not delayed ones, matching the API reference and the gauge. SenderId is the caller's IAM unique id (user id, or role id:session for an assumed role) resolved by the shared awsidentity.Resolver, matching GetCallerIdentity's UserId; library and service-to-service sends keep the account id. Part of stackshy#1514
f1bb811 to
17d134b
Compare
Summary
Part of #1514. Adds the SQS CloudWatch queue gauges, published once a minute per active queue from the serve scheduler. Also makes
SenderIdthe caller's IAM unique id.ApproximateNumberOfGroupsWithInflightMessages, once a minute per active queueApproximateNumberOfMessagesVisiblenever emittedApproximateNumberOfMessagesNotVisiblenever emittedGetQueueAttributesnow uses the same definitionApproximateNumberOfMessagesDelayednever emittedApproximateAgeOfOldestMessagenot tracked or emittedSenderIdalways the bare account IDawsidentity.Resolver, so it matchesGetCallerIdentityUserId. Assumed roles now report the role's real RoleId (STS fix below)NumberOfMessagesMovednot emitted on DLQ redrive / StartMessageMoveTaskWhat real AWS does (sources)
QueueNamedimension, the in-flight and delayed definitions, the age rules, and the FIFO groups metric. The page has noNumberOfMessagesMoved. Move progress is theApproximateNumberOfMessagesMovedfield ofListMessageMoveTasks, which cloudemu already returns.ApproximateNumberOfMessagesNotVisibleis "the approximate number of messages that are in flight", and delayed messages are reported separately inApproximateNumberOfMessagesDelayed.ABCDE1F2GH3I4JK5LMNOP:i-a123b456) for a role.Changes
providers/aws/sqs/metrics.go(*Mock).Tick(now)has theconfig.Tickableshape that CloudWatch, SSM and CloudFormation already use. It is registered inProvider.Tickables(), ahead of CloudWatch, so alarms evaluated in the same tick see the fresh sample.PutMetricDatacall after all locks are released, so an alarm whose SNS action delivers back into a queue cannot deadlock.NumberOfMessagesSent/Received/Deleted,NumberOfEmptyReceives,SentMessageSize) stay where they were.DeleteMessageandReceiveMessagesWithOptionsnow publish their counters after releasing the queue lock.Queue state
lastAccessed, updated by every API action on it (create, send, receive, delete, visibility, attributes, tags, permissions, purge, list-DLQ-sources, move task), andlastSampleAt.--persistand snapshots keep the gauge schedule.GetQueueAttributesApproximateNumberOfMessagesNotVisibleno longer counts delayed messages. It and the gauge now come from the samesampleGaugespass.SenderIddriver.SendMessageInputanddriver.BatchSendEntrygain an optionalSenderID.awsidentity.Resolvershared with STS and EKS.STS (
server/aws/sts/operations.go)AssumeRole,AssumeRoleWithWebIdentityandAssumeRoleWithSAMLnow returnAssumedRoleId = <role's RoleId>:<session>when IAM knows the role. The syntheticAROACLOUDEMU0000000000prefix is kept only when no IAM driver knows it.GetCallerIdentity,aws:useridand SQSSenderIdall report it.go generate ./...produces no coverage diff.Provider Coverage
Checklist
go test -p 2 ./...)golangci-lint run --new-from-rev=<merge-base>overproviders/aws/...(sqs and the provider),server/aws/(sqs, sts and the server package) andservices/messagequeue/...reports 0 issues). The repo-wide run ondevelopmentalready has unrelated findingscloudemu_test.go: covered by the SDK wire tests below insteadgo generate ./..., no diff)Test Plan
Provider tests (
providers/aws/sqs/metrics_test.go, using FakeClock and callingTickdirectly):TestRequestsPublishCountersButNoGaugesTestTickPublishesOneSamplePerMinute: six sends give one sample of 6. Ticks inside the minute publish nothing.TestTickTracksVisibleInFlightDelayedAndAge: also checks thatGetQueueAttributesagrees with the gauge.TestTickStuckQueueKeepsReportingAndAges: one delayed send, then 8 hours of ticks with no traffic. Delayed→visible is reflected, the age grows every minute, and there is exactly one sample per minute.TestTickStopsForEmptyQueueIdleSixHours: an empty queue stops publishing after six idle hours, and any action wakes it up again.TestTickAgeSkipsStandardPoisonPills,TestTickFIFOCountsInflightGroups,TestTickFollowsDeadLetterRedriveAndMoveTask,TestTickFollowsLambdaESMDeadLetterRedriveTestTickPublishesOutsideQueueLocks: calls back into the queue from insidePutMetricData.TestTickWithoutMonitoringIsNoOpTestSnapshotKeepsGaugeSchedule: last access and last sample survive a snapshot restore.TestSenderIDFromCallerElseAccountproviders/aws/tickables_test.go: SQS is registered first inTickables().SDK wire tests (real aws-sdk-go-v2 clients):
server/aws/sqs_metrics_sdk_test.go:TestSDKSQSQueueGaugesReachCloudWatch: requests alone publish no gauges. After one tick there is a single sample, andGetQueueAttributesagrees with it.TestSDKSQSGaugeSumIsOneSamplePerMinute: six sends give Sum=6 and SampleCount=1.TestSDKSQSSenderIDMatchesCallerIdentity: an IAM user, and an assumed role whoseAssumedRoleIdis<RoleId>:worker-1.server/aws/sts/assumed_role_id_test.go:TestSDKAssumedRoleIDUsesRoleID: two roles with the same session name get different ids.TestSDKAssumedRoleIDWithoutIAM: falls back to the synthetic prefix.Full run
go build ./...passes.go test -p 2 ./...passes.go test -racepasses onproviders/aws/sqs.Real-user run:
cloudemu serve --aws-port 4604(default 1s tick), driven with the AWS CLI.stuck-qwith two alarms:ApproximateAgeOfOldestMessage > 30andApproximateNumberOfMessagesVisible > 0, both Maximum over 60s. Then I sent one message with--delay-seconds 5and did nothing else.OKat t+30s and t+60s, andALARMfrom t+90s on.sum-q, with an alarm on Visible using Sum, threshold 10.OK.burst-qgave per-minute samples of 20, 50 and 50, each with SampleCount 1.workerAROATFI65QRLEGW5MUR7XandauditorAROAEUYTHTYGJQPNI2IHR), calledassume-rolewith sessionsess-1for each, and sent a message with each role's credentials.AssumedRoleIdand the receivedSenderIdwereAROATFI65QRLEGW5MUR7X:sess-1andAROAEUYTHTYGJQPNI2IHR:sess-1respectively.Follow-ups (not in this PR)
SenderId. Real AWS reports the delivering service's principal there. This needs the delivery paths to pass a service principal.Related Issues
Part of #1514