Skip to content

fix(aws-elasticache): typed faults, control-plane lock, atomic creates - #1704

Merged
NitinKumar004 merged 2 commits into
developmentfrom
fix/elasticache-parity-1472
Oct 23, 2026
Merged

NitinKumar004 merged 2 commits into
developmentfrom
fix/elasticache-parity-1472

Conversation

@NitinKumar004

@NitinKumar004 NitinKumar004 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

This lays the foundation for the ElastiCache parity work in #1472. It doesn't add new operations. It fixes how errors reach the wire and makes the control plane safe under concurrent use.

Typed faults. Provider errors now carry their AWS fault code in a typed field (Fault, with AWSFault()). Before, the code was a prefix in the message. The wire layer takes the code from that field, picks the HTTP status from a table that follows the botocore model, and writes the message with no code prefix. This replaces the old substring matching, which had two bugs:

  • a duplicate parameter group came back as CacheClusterAlreadyExists; it is now CacheParameterGroupAlreadyExists;
  • CacheSubnetGroupNotFoundFault came back as 404; it is now 400, as in AWS.

Control-plane lock. Mock gets a mutex, mu, that guards the cluster, replication group, subnet group, parameter group and snapshot stores, and the tag map. The separate tag mutex is gone: every tag access already ran under mu. The lock order is documented on the struct: mu, then a settle set's lock. Settle sets have no callbacks. No lock is held across a call to monitoring, to the VPC subnet resolver or to a real cache engine.

  • A replication group that is still being created owns its member ids, so a cluster named <group>-001 is refused during that window. Creating a group whose member id is already used by a cluster returns CacheClusterAlreadyExists.
  • Creates are atomic, so concurrent creates of the same id produce exactly one winner.
  • The subnet-group and parameter-group in-use checks run in the same critical section as the delete. A cluster or group that is still being created counts as a user.
  • Persist Snapshot and Restore run under mu.
  • Parameter-group overrides are copied on write, so a Describe result handed out earlier is never mutated.

Id reservation around real engines. A create reserves the id while the real engine is being provisioned. A delete keeps the id reserved until Deprovision returns, so the same engine slot is never provisioned twice.

  • During those windows, Describe and List return the resource with status creating or deleting.
  • Mutating calls in those windows return InvalidCacheClusterState or InvalidReplicationGroupState, and a create of the same id returns AlreadyExists.
  • If a teardown fails, the resource is put back.

This also locks the remaining parameter-group operations and the snapshot operations, which covers #1690 and #1697. Each has its own race test. Tagging snapshot ARNs is still not supported and stays not-found (#1695). It adds docs/coverage/nongoals/cache.md, which also notes two behaviours: a persisted snapshot leaves out resources that are still creating or deleting, and a deleting group's member clusters drop out of DescribeCacheClusters first.

Tests:

  • 50-goroutine -race tests for duplicate creates of clusters, replication groups, subnet groups and parameter groups.
  • TestModifyVsResetParameterGroupRace (AWS ElastiCache: lock the remaining parameter-group ops #1690) and TestSnapshotVsDeleteReplicationGroupRace, which runs snapshot create, copy and delete against deleting the group (AWS ElastiCache: lock the snapshot ops #1697).
  • TestCreateCacheCollidingWithReservedGroupMember.
  • Delete-vs-create races for subnet groups and parameter groups.
  • Engine-gated tests: describing during creating and during deleting, mutating calls getting the invalid-state fault, recreate blocked until Deprovision returns, and Snapshot/Restore running alongside creates.
  • TestWriteErrStripsCodeFromMessage, TestWriteErrStatusFromTable and an SDK test that checks the typed fault types, the status codes and prefix-free messages.

Verified:

  • go build ./... passes.
  • -race -count=3 passes on the ElastiCache provider and server packages, plus TestWiringParity, persist and TestAWSErrorMessageShapeCompat.
  • golangci-lint on the touched packages shows no new issues against the development baseline.
  • go generate ./... is clean after the commit.
  • AWS CLI against cloudemu serve --async-settle: create-cache-cluster and create-replication-group report creating; duplicates return CacheClusterAlreadyExists and ReplicationGroupAlreadyExists with messages that carry no prefix; a missing subnet group returns HTTP 400; delete then recreate works.
  • Terraform (hashicorp/aws 6.70.0) with aws_elasticache_parameter_group, aws_elasticache_cluster and aws_elasticache_replication_group: apply, plan shows no changes, update (parameter value and tags), plan shows no changes, destroy.

Part of #1472

@NitinKumar004
NitinKumar004 merged commit ec39f28 into development Oct 23, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant