From f65a52cfebef196ce4f283e2ea9b3cbbb38001f2 Mon Sep 17 00:00:00 2001 From: Seer <2774413+chideat@users.noreply.github.com> Date: Thu, 10 Sep 2026 05:40:00 -0400 Subject: [PATCH] docs(redis): correct the cross-datacenter replication guides against a live cluster (#64) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs(redis): correct the Active-Active setup guide's ports, seeds, and credential Three corrections to functions/95-disaster-recovery/30-active-active/20-setup.mdx, verified against redis-operator and redis-modules release-5.1. announcePort is an identity value, not a transport. The guide described it as "the RESP port for the gossip control plane" and 7379 as the data plane. Both mesh planes actually dial the peer port: replication through connectPeerPort (redis-modules active-redis src/upeer.zig) and gossip through gossipOnePeerPort, which is the only gossip transport left since the plaintext redis-command fallback was removed with the legacy as.peersync path (src/mesh.zig, gossipOne). What announcePort is read for is self-recognition — a node matches a seed against its own announced addr+port (seedMatchesSelfLocked) — which is what lets every datacenter carry one uniform seed list. The guide was the outlier here; 10-architecture.mdx, 10-intro.mdx and 90-limitations.mdx already had it right. The 7379 requirement is unchanged, now stated with its mechanism: the proxy runs ONE listener, and the 7379 entry on the proxy Service is an alias onto it added in mesh mode only (internal/builder/activeredis/service.go, targetPort 6379). The module advertises its own local peer-port verbatim with no announce override, so an external load balancer must forward both numbers and must not remap 7379. Seeds are the remote member's announceAddress:announcePort, not its in-cluster proxy Service. When a load balancer or VIP fronts a member's proxy, that endpoint is what its peers must seed, written verbatim — a mismatch defeats the self-skip that makes a uniform seed list work. Step 1 leads with a dedicated custom RedisUser again, with the default account kept as the documented fallback rather than the opening path. This revisits the framing of 386e527, which made the default account primary after aapeerrepl was removed from the product; the account itself was never the problem, and ResolvePeerAuthCredential still records the dedicated account as the stronger choice because anything holding the default password can otherwise act as a mesh peer. The example creates its own Secret (password Secrets are 1:1 with RedisUser objects) and uses a plainly user-chosen username rather than the retired aapeerrepl. Requirements common to either account are stated once, from ResolvePeerAuthCredential and the RedisUser webhook: local instance, custom or default only, a password Secret, phase Success, and the same username and password value in every datacenter. Also added, from a redis-operator review of the same fields: - announceAddress is now a prerequisite. Nothing defaults it to a cross-datacenter address and admission only warns, so a member left without one advertises an address its peers cannot reach. - The empty-announceAddress fallback is two-tier, not a straight pod-IP fallback: with a managed DNS zone configured the operator derives .. and publishes it via ExternalDNS, and only with no zone does the module fall back to the pod IP. - announceAddress and announcePort must be set on the Redis instance. ActiveRedis exposes the same fields, but the operator rewrites that child's spec.proxy from the Redis CR on every reconcile, so an edit made there is overwritten. Known gap, tracked separately: docs/shared/crds/ is published as the site's API reference, and the AnnouncePort description generated into middleware.alauda.io_redis.yaml and redis.middleware.alauda.io_activeredis.yaml still carries the old "mesh CONTROL plane (gossip)" wording. It is corrected at source in redis-operator (!803 for release-5.1, !804 for master); those two files should be re-synced here once !803 merges. They are byte-identical to release-5.1 tip today, so that commit is the entire delta. * docs(redis): stop presenting the seed @peer-port suffix as a user knob The Seed syntax section described `@peer-port` as an optional suffix that "names the port to dial", carried over from the CRD description's framing that it is for "when an external LB remaps the module peer-port". That remap does not work, so the sentence advertised a knob the product does not support. DeriveMeshSeeds does accept and pass an explicit `@peer-port` through (internal/controller/middleware/activeredis/meshconfig.go), but a seed is only ever a gossip target: seeds are emitted as bare addr/port records for the gossip round (redis-modules active-redis src/mesh.zig, snapshotGossipTargetsRotated) and never enter the member table. Edges are built from LEARNED records instead — ensurePeer takes np.options.peer_port from node.peer_port (src/mesh_command.zig) — and every member advertises its own local peer-port verbatim, which the operator hardcodes to 7379. So a remapped seed suffix would steer the bootstrap gossip dial and nothing else, leaving replication on 7379. That is the same conclusion 90-limitations.mdx already states: the peer port must not be remapped. The suffix is also not optional in the sense the guide implied. gossipOne fails a round outright for a target with no peer_port ("A record without an advertised peer_port is undialable ... mesh-seeds require @peer-port"), because the peer-port one-shot is the only gossip transport left. The operator appending 7379 is what makes a plain host:port seed dialable at all. The section now tells the reader to write plain `host:port` and explains the completed `host:port@7379` form they will see back from `CONFIG GET activeredis.mesh-seeds`, without offering an override. Not fixed here: the same "when an external LB remaps the module peer-port" wording is generated into the ActiveRedisMesh CRD description and published through docs/shared/crds/, so it needs the same treatment at source in redis-operator as the AnnouncePort description in !803. * docs(redis): drop the module peer-port number from the setup guide The peer port is an internal module mechanism. Everything a datacenter reaches is the redis-tools proxy, which has a single listener and no notion of that number — it appears only as a backend dial target inside the pod. Naming it in a setup guide sent readers looking for a port they never configure. Removed from all four places it appeared: the prerequisite, the rejected announceAddress example, the announcePort directive, and the seed grammar. The announcePort correction is unchanged in substance — it is an identity value, and neither plane is steered by it — but is now stated without naming the port that does carry the traffic. The reachability REQUIREMENT is kept, stated by property instead of by number: forward every port the proxy Service publishes, each under its own number, and do not remap them. It still has to be said, because the operator's default Service type is NodePort and only Ports[0] carries a pinned node port (internal/builder/activeredis/service.go) — the second publishes on an arbitrary one, so an external LB or VIP fronting a NodePort member has to be told what to forward. Under type: LoadBalancer, which is what the guide's example uses, the LB is provisioned from the Service and publishes both ports on its own, so nothing is needed and the reader never meets the number. Seven mentions remain in sibling pages (release_notes.mdx, 90-limitations.mdx, 10-intro.mdx, 20-disaster-recovery/20-setup.mdx, 30-active-active/10-architecture.mdx); whether to extend this is a separate decision, since the architecture page is arguably where an internal mechanism does belong. * docs(redis): drop the module peer-port number from the remaining pages Completes the previous commit across the rest of the doc set, so the number appears nowhere: release notes, the mode comparison and port table in the Cross-Datacenter Replication intro, the limitations entry on address HA, the Disaster Recovery setup note, and the Active-Active architecture page. Everything a datacenter reaches is the redis-tools proxy, which has one listener; the peer port is a backend dial target inside the pod. A reader configuring an instance never sets it, so the docs describe the port by what it is rather than by its number, and point at the proxy Service for the value. Each page keeps its operational content: - The intro's port table still carries the row, so it stays visible that Active-Active publishes a second Service port and Disaster Recovery does not. The Port cell now says to read the number off the Service. - The "expose the peer port unchanged" directive becomes "publish every proxy port unchanged" — same requirement for an external LB or firewall, stated by property. The RESP port remains the documented exception, remappable through announcePort. - The architecture page still says which traffic uses which port and that the module fixes and advertises that number itself; only the literal is gone. The release-note section was included for the same reason 386e527 applied the product rename to historical sections: the docs should read consistently rather than preserve a superseded description of shipped behaviour. * docs(redis): refresh the published CRDs from redis-operator release-5.1 Syncs docs/shared/crds/ with redis-operator release-5.1 at 59c729d5, which merged the announcePort correction (!803, 222781b3). Two files change, 31 lines each; the other four CRDs are already identical. This retires the last copy of the claim that announcePort carries mesh gossip. docs/shared/crds/ is published as the site's API reference and generated into kubectl explain, so until now both were still telling users that gossip rides that port — the same error the prose fixes in this branch corrected two pages over. The regenerated description states it correctly: announcePort is the mesh IDENTITY value, it carries no traffic, and both planes dial the peer port. Generated files, copied whole from config/crd/bases/ — not hand-edited. Not resolved by this: the peer-port number still appears in four CRD field descriptions (redis, activeredis, activeredisconnections, activeredismeshes), which is the one place the prose policy from the previous two commits cannot reach, since these are generated. Fixing it means editing the Go field comments in redis-operator and regenerating. The activeredismeshes description additionally still frames the seed @peer-port suffix as being for "when an external LB remaps the module peer-port", which is the framing removed from the prose in 645925f. * docs(redis): correct the Active-Active setup guide against a live cluster Verified the guide line by line on business-1, running operator v5.1.0-rc.50.g59c729d5 (release-5.1 tip 59c729d5) with redis-modules v5.1.2 and Redis 7.2.16 — three sentinel instances meshed in namespace aa-doc-verify, all three ActiveRedisMesh CRs Healthy 3/3. Four defects, all reproduced: The default-account fallback did not work as written. The guide said every instance has one "so nothing has to be provisioned". True of the RedisUser, not of the credential: the default account holds a password Secret only when the instance was created with spec.passwordSecret, and the create-instance guide's own example sets none. Binding it there leaves ActiveRedis in Failed with "has no password secret: the module rejects credential-less peers" — reproduced on s72-dc1. The fallback now states the precondition and shows that message. The peer-auth password is policy-checked and the guide never said so. A placeholder value is refused at admission with "password should consists of letters, number and special characters"; the rules (8-32 chars, mixed classes) are now stated where the Secret is created. Enabling the mode restarts the data pods, which the guide did not mention, and that interacts with step order: the module's init container is added, so the instance rolls back through Initializing, and while it is not Ready a custom RedisUser cannot be created against it ("redis failover s72-dc1 is not ready"). Doing Step 2 before Step 1 therefore strands you until the roll finishes. Step 1 already came first; the guide now says why that matters. The Step 4 status sample did not match a real mesh: - epoch was shown as 7 at status level and on every member. It is absent from a converged mesh — zero occurrences across all three CRs, and the module reports epoch:0 behind them. Removed rather than left as a number nobody will see. - suspectCount and deadCount were printed as 0; they are omitted at zero. - address was shown as a bare host. It carries the port: activeredis-proxy-rfr-s72-dc1.aa-doc-verify.svc:6379. - every member was shown fully populated. The LOCAL member's entry is bare — uid, serviceID, shardID, state, version and nothing else. Gossip reports the other members, so the operator synthesizes the self record and has no source for address, clockOffsetMs or moduleVersion. The sample now shows three members with the local one short, and a directive says why, so a reader does not file a bug against the member that looks incomplete. moduleVersion 10 was correct for v5.1.2 and is unchanged. Confirmed correct and left alone: mesh rejected on Redis 6.0; serviceID bounds [0-15]; all four announceAddress rejections and the bare-IP warning; the default-account RedisUser naming and both printer-column samples; the proxy Service publishing its second port onto the proxy listener; the ActiveRedis child being overwritten from the Redis CR (patched it to a bogus announce address and watched the operator revert it); seeds completed and pushed identically to masters and replicas; and the Pending, Paused and Failed phases with their status messages. Not testable on one cluster: ExternalDNS binding of the announce host, and genuine cross-datacenter routing. Neither claim was changed. * docs(redis): correct the epoch claim and the operations-page mesh reads Follow-on from the live verification: two more claims outside the setup guide turned out to be wrong in the same ways. epoch does not track membership. The architecture page said the status carries "an epoch that advances as membership changes". It is not a change counter: ms.self.epoch is incremented in exactly two places in redis-modules active-redis src/mesh.zig at d224ac1 — forget (:1039) and forgetPermanent (:1066), the TTL'd-forget and permanent-decommission paths. Join, suspect, dead and recover never touch it, and ordinary per-round freshness is carried by heartbeat_seq instead. That matches the measurement: 0 across all three converged meshes, and epoch:0 from `as.mesh info` on the masters. The page now says it advances only on forget or decommission, so 0 on a group that has only gained members reads as correct rather than as a bug to file. The clock-skew jsonpath sample had the same defect as the Step 4 status sample: it showed the local member as a populated row. Running the doc's own command against a real mesh produces a blank first field and a blank offset, because the local record is synthesized and carries neither: alive activeredis-proxy-rfr-s72-dc2.aa-doc-verify.svc:6379 alive -9 activeredis-proxy-rfr-s72-dc3.aa-doc-verify.svc:6379 alive -12 The sample now shows that row and says the local member is the reference the other offsets are measured against. Remote addresses carry their port, as in the Step 4 fix. Both `redis-cli` reads on that page now say to run against the shard master. Only the master participates in gossip — a replica loads the module but must not advertise itself or initiate gossip (src/mesh.zig:286-292), and the operator agrees by construction, skipping any node that is not a ready master before reading as.mesh (activeredismesh_controller.go:150-152). Measured: on the same converged mesh, the dc2 replica reported alive:0 dead:2 while its master reported alive:2 dead:0. Someone debugging a skew alarm will otherwise run these on whichever pod they reach first and conclude the mesh is broken. Provenance worth recording: the numbers the Step 4 sample used — epoch 7 with clock_offset_ms -5 — are the fixture values in the module's own unit test for the node-line format (src/mesh_command.zig:1157-1168, which also carries hb=42, ver=8.4, mver=2). The sample appears to have been lifted from a test expectation rather than from a deployment, which is why it did not survive contact with one. * docs(redis): warn that Step 3 must wait for the Step 2 restart Found by re-walking the corrected guide end to end on business-1 with a fourth instance (s72-dc4, serviceID 3), following Step 1 -> 2 -> 3 verbatim. Creating the ActiveRedisMesh immediately after Step 2 puts it in Failed, not the Pending the phase table would lead you to expect, and the message says the module's own commands do not exist: set mesh config on pod rfr-s72-dc4-0: ERR Unknown option or number of arguments for CONFIG SET - 'activeredis.mesh-seeds'; shard 0 mesh info: ERR unknown command 'as.mesh', with args beginning with: 'info' The cause is the rolling restart Step 2 triggers. The module arrives with the new pods, and the roll takes them one at a time, so there is a window where the current master is still the pod that has NOT been rolled and carries no module. Observed directly at that moment: rfr-s72-dc4-1 had the install-activeredis-module init container and rfr-s72-dc4-0 did not, and the controller had picked the latter. It does clear on its own — Failed -> Pending once the master carries the module -> Healthy — but while it lasts it is indistinguishable from a real failure, and a first-time reader following the guide in order will hit it. Step 3 now says to wait for the instance to report Ready, shows the message, and says it clears. Also verified in the same walk, nothing to change: - the not-Ready refusal added in the previous commit, message for message: "redis failover s72-dc4 is not ready" - the Secret must exist before the RedisUser that references it; the guide's single create with a --- separator already handles that ordering - the architecture page's "adding a member does not require touching the existing ones": mesh-dc4 seeded ONE existing member (dc1) and reached Healthy 4/4; the other three learned dc4 through gossip and reached 4/4 with their seed lists untouched, confirmed by diffing them before and after - the local member's entry stays bare at four members, as the Step 4 directive now describes One cosmetic fix: the Step 1 sample's column spacing did not match what kubectl prints. * docs(redis): correct the Disaster Recovery setup guide against a live cluster Verified every claim in "Set Up Disaster Recovery Replication" against a Redis 7.2 peerof pair and a Redis 6.0 pair on a business cluster (operator v5.1.0-rc.50.g59c729d5, module image v5.1.2). Most of the guide held; these are the claims that did not. Step 1 now leads with a dedicated custom account, mirroring the Active-Active setup guide. The default account stays as a documented fallback, with the warning that it carries no credential at all unless the instance was created with spec.passwordSecret — binding it there leaves ActiveRedis in Failed. The old text said default passwords are "generated per instance", which they are not. Other corrections, each observed live: * Enabling replication rolls the data pods. Neither the 7.2 nor the 6.0 section said so, and the steps that follow fail against a half-rolled instance. Warned in Step 2, and Step 4 points at it for the downstream. * The Step 6 status sample showed service_metadata, which a 7.2 peerof link never populates — the module reports it empty. It also omitted versionState and was not in the order -o yaml prints. The 6.0 sample is correct as written and keeps service_metadata. * The ActiveRedis printer sample showed DPEERS 0; a zero count renders blank. Replaced with both sides of a live pair. * "Rotating an instance's own password Secret is independent and does not affect peer links" is only true when a custom account is bound. With the default account bound the two are the same Secret — rotating the upstream password made a new link fail WRONGPASS. * Decommission does not force a fresh full synchronization: a connection re-created straight after one resumed PartialSync. What it does is stop retaining the state that makes resuming possible. * A Secret shared by several instances is contended by their RedisUser controllers and never settles, so "the same default password" needs the same value in a Secret per member. * Rotation and the pre-flight inspection: an established link keeps replicating across a one-sided rotation, but a new link cannot be wired while the values differ. * One "master-slave" left in the 6.0 section, next to "master-replica". In the upgrade guide, peer admission matches on the whole credential, not the username alone, and a link wired while the two ends straddle the upgrade parks in Paused rather than failing — captured verbatim. * docs(redis): correct the 6.0-to-7.2 upgrade guide against a live rehearsal Ran the whole procedure on a business cluster: a Redis 6.0 Sentinel pair wired as a Disaster Recovery group, seeded with strings, a list and a hash, then taken through Steps 1-6 to Redis 7.2. The procedure works. Both members kept their data across the version change — verified independently on each member's own master, before any link was re-created — and the re-created link resumed as PartialSync from the offset the Redis 6.0 link had left off at, so Step 2's reason for choosing Detach holds across the generation change too. Corrections: * "Downgrading is not supported" read as though something enforced it. Nothing does: setting spec.version back to 6.0 is accepted and starts rolling the pods onto the older image straight away. Said so plainly. * The prerequisite about the proxy endpoint now states the fact it depends on — the proxy Service is not recreated, so its address, port and node port all survive the upgrade, and the recorded addresses stay valid. * Step 6 gains syncStatus to the checklist, with what a FullSync there would mean, and asks for the pre-upgrade keys as well as a new one. * A link wired while the two ends straddle the upgrade parks in Paused rather than failing or replicating — captured verbatim, since a reader who mistimes Step 5 will see it and it looks alarming. * docs(redis): tighten three claims from the Disaster Recovery pass Follow-ups on the previous two commits, each brought back to what was actually observed. The rolling-restart warning went into the Redis 7.2 section only, but the 6.0 section rolls its pods the same way and its next step — creating the connection — is rejected while the instance is still coming back. Both 6.0 patch steps now say so, with the rejection text. The Decommission directive said the retention that makes resuming possible is no longer kept, so a returning peer needs a full synchronization. The teardown runs on the instance that owns the connection and releases what *it* holds for that peer; the upstream's retention is untouched, which is why a link re-created straight afterwards resumed PartialSync. The directive now describes what the teardown does and says plainly that re-creating the link does not undo it — detach a peer you expect back. versionState was described as where a version-held link shows up. The module reports it as unknown | verified | violating, and a held shard is marked by a non-zero versionHoldSince instead. Also walked the newly-recommended custom-account path on the peerof pair rather than relying on the Active-Active run: both RedisUsers reach Success, switching spec.activeRedis.redisUserRef is a live change with no pod restart, and the link comes up Healthy and replicates. * docs(redis): document the custom peer-auth account on Redis 6.0 The Redis 6.0 section described only the automatic default-account binding, leaving open whether a dedicated account is usable there at all. It is — verified on a purpose-built 6.0 pair. The test was built so the result could not be ambiguous: the two instances were given different default passwords, and the custom account a third one. A link that comes up can therefore only be authenticating as the custom account. It came up Healthy and replicated, and the peer session on the upstream master shows `cmd=as.peerconf user=dr6-peer` — the custom ACL user, not default. Binding it in the same patch that enables replication is enough; the automatic binding leaves an existing one alone. What differs from Redis 7.2 is rotation, and it differs badly enough to warrant a warning of its own. A 6.0 link carries the credential inline in the module's peer record, fixed at wiring time, and the operator's self-healing re-wire is new-module-only. After rotating the password the running link kept working until its session was dropped, then sat at `shard 0 status is Disconnected` and never recovered on its own. Deleting and re-creating the ActiveRedisConnection restored it immediately. Both facts are now in the 6.0 section, and the credential-rotation section carries the 6.0 exception next to the 7.2 procedure it does not share. * docs(redis): record where the Redis 6.0 link credential used to live The docs stated that ActiveRedisConnection.spec.secretName "has been removed" without saying what it was, what replaced it, or what an existing Disaster Recovery group has to do about it. Anyone arriving from a release before v5.1 has that field in their manifests. The Redis 6.0 section now carries the history: until v5.1, Disaster Recovery on Redis 6.0 was the only cross-datacenter replication this product offered, and the link credential lived on the connection, in a Secret holding a password and no username. That is why the account has always been the instance's default one — the legacy module sends a one-argument AUTH and the proxy resolves it to default — and why both datacenters have always needed the same default password. v5.1 removes the field from ActiveRedisConnection and ActiveRedisInspection and takes the credential from spec.activeRedis.redisUserRef alone. Existing groups are migrated with no intervention: the operator binds each replicating instance to its own default-account RedisUser on its first reconcile, which changes nothing on the wire. Stored resources keep working; a manifest that still sets secretName is rejected on its next apply. Also in the release notes, which described the automatic binding as "deriving it from the credential the replication group already shares". Nothing is derived or provisioned — the account already exists and the binding only names it. Added the field removal as an entry of its own. The intro now dates the two generations, so "the legacy module" reads as the product's original Disaster Recovery (v4.1.0) rather than an unexplained alternative. * docs(redis): cover replicating instances in the upgrade to v5.1 The upgrade guide told a reader with a Disaster Recovery group what to do about their Redis version, but nothing about what the operator upgrade itself does to them — which is the question someone planning a 5.0 to 5.1 upgrade actually has. Confirmed by manual test of the upgrade: replication continues across it and the data on both sides is unaffected. The instances keep their Redis version, so they keep the module generation they were already running. What moves is where the link credential is declared: onto the instance's spec.activeRedis.redisUserRef, written by the operator on its first reconcile and naming the default account the link already authenticated as. No action required for the running group; the one thing that does need attention is manifests still setting the removed spec.secretName. * docs(redis): stop using the forbidden word "master" in the pages this pass edits The shared glossary maps master to control plane, so doom lint reports `Forbidden word: "master" (control plane)`. It is a Kubernetes node-naming rule and it fires on Redis's own replication role, but the rule is the rule and CI is red on it. Two things had hidden this. The local lint builds its flag list from the shared terms file, so with that fetch blocked it reports unknown words only and nothing else. And CI reports issues on changed lines, so the occurrences this pass did not touch have never failed a build. Checked against cspell-lib with the mapping applied rather than guessed at: the hyphenated forms are flagged too, because master-replica and master-slave tokenize to master. Only mymaster survives, as one token. That made one more line in this branch a latent failure, the Sentinel shard-0 note whose master-slave I had already changed to master-replica. Every line this branch touches now reads primary node or primary-replica pair and is clean. The remaining occurrences are in lines this branch never edited and are left alone; they are listed in TERMINOLOGY_CANDIDATES.md along with the reason a mechanical sweep would be wrong — the same files carry master as an identifier, in spec.replicas.sentinel.master and in mymaster / MasterName / setMasterName, where changing it breaks the examples. Whether primary node is the accepted replacement, and whether the glossary should carry an exemption for the Redis role sense, are noted there as open questions rather than decided here. (cherry picked from commit 80ee214ad7224a816ea09fd8bb4add078ecd25b3) --- TERMINOLOGY_CANDIDATES.md | 44 ++++ .../95-disaster-recovery/10-intro.mdx | 12 +- .../20-disaster-recovery/20-setup.mdx | 195 +++++++++++++++--- .../40-upgrade-6.0-to-7.2.mdx | 21 +- .../30-active-active/10-architecture.mdx | 4 +- .../30-active-active/20-setup.mdx | 155 +++++++++++--- .../30-active-active/30-operations.mdx | 12 +- .../95-disaster-recovery/90-limitations.mdx | 2 +- docs/en/release_notes.mdx | 12 +- docs/en/upgrade.mdx | 8 + .../crds/middleware.alauda.io_redis.yaml | 31 ++- ...edis.middleware.alauda.io_activeredis.yaml | 31 ++- 12 files changed, 423 insertions(+), 104 deletions(-) diff --git a/TERMINOLOGY_CANDIDATES.md b/TERMINOLOGY_CANDIDATES.md index e3b4cf2..79d734b 100644 --- a/TERMINOLOGY_CANDIDATES.md +++ b/TERMINOLOGY_CANDIDATES.md @@ -212,3 +212,47 @@ definitive answer needs a register search or counsel. | **Active-Passive** | Clear, same class as Active-Active. Considered and not chosen. | | **Multi-Active** | **Avoided.** Cockroach Labs presents "Multi-Active Availability" as its own coined term, explicitly contrasted with generic active-active. No ™ and no registration found (their registered mark is COCKROACHDB, Reg. 5307353), but adopting it would read as derivative. | | **Mesh** | No trademark issue, but collides with **Service Mesh** in the ACP console. Retained only as a mechanism word. | + +## `master` is a forbidden word in the shared glossary (added 2026-09-10) + +The shared terms list maps **`master` → `control plane`**, so `doom lint` reports +`Forbidden word: "master" (control plane)` on any occurrence. The mapping is a +Kubernetes node-naming rule, and it fires on Redis's own replication role, which +is genuinely called `master` — `INFO replication` reports `role:master`, and the +Redis CR carries a `spec.replicas.sentinel.master` field. + +Two things make this easy to miss: + +* **The local run does not catch it.** `flagWords` are built from the shared + terms file, which is fetched over the network; when that fetch is blocked the + local `yarn lint` has an empty flag list and reports only unknown words. +* **CI reports changed lines only,** so the existing occurrences below have never + failed a build. They will, on whichever pull request next edits those lines. + +Verified against `cspell-lib` with the mapping applied: the hyphenated forms +**are** flagged (`master-replica` and `master-slave` tokenize to `master`), while +`mymaster` is not, being a single token. + +Pages edited in the cross-datacenter replication pass now use **primary node** / +**primary-replica pair** and are clean. That is a local choice made to get the +pass through CI, not a registered term — confirm it, or choose another. + +| Still carrying `master` | Line | Kind | +|:---|:---|:---| +| `functions/95-disaster-recovery/90-limitations.mdx` | 50 | prose — "master-replica synchronization" | +| `intro.mdx` | 55 | prose — "master-slave relationships" | +| `functions/40-accessmethod.mdx` | 51 | prose — "Redis master-replica group" | +| `functions/75-monitor.mdx` | 35 | prose — alert name "Master-Slave Failover" | +| `how_to/access/10-sentinel.mdx` | 71 | prose — "master-slave cluster name" | + +**Do not rewrite these mechanically.** The same files also carry `master` as an +identifier, where changing it would be wrong: `spec.replicas.sentinel.master` in +`functions/10-create-instance.mdx`, and `mymaster` / `MasterName` / +`setMasterName` / `withSentinelMasterId` in `how_to/access/10-sentinel.mdx` — +Sentinel's fixed group name and client-library API surface. A blanket +find-and-replace breaks working examples. + +Decisions needed: whether `primary node` is the accepted replacement; whether the +prose rows above should be converted in one pass; and whether an exemption for +the Redis role sense belongs in the shared glossary instead, since the product +will keep meeting the word. diff --git a/docs/en/functions/95-disaster-recovery/10-intro.mdx b/docs/en/functions/95-disaster-recovery/10-intro.mdx index ed55346..b42a0b2 100644 --- a/docs/en/functions/95-disaster-recovery/10-intro.mdx +++ b/docs/en/functions/95-disaster-recovery/10-intro.mdx @@ -65,6 +65,8 @@ The `spec.activeRedis.mode` field takes the literal values `peerof` and `mesh`. The replication module ships in two generations. Which one an instance loads is decided by its Redis version, and the generation determines the feature set, the authentication model, and the peer transport. +Before v5.1 there was only one: Disaster Recovery on Redis `6.0`, introduced in v4.1.0. Redis `7.2` instances and the Active-Active mode arrived with the new module in v5.1, and the legacy generation is now maintained for existing instances only. + | | Redis `6.0` (legacy module) | Redis `7.2` (new module) | |:--|:--|:--| | Module | Frozen legacy implementation, maintained for existing instances only | Actively developed implementation | @@ -73,7 +75,7 @@ The replication module ships in two generations. Which one an instance loads is | Replicated command set | Smaller — commands introduced after Redis 6.0 are not replicated | Full; see [Command Support](./60-commands.mdx) | | Command guards | None — `MOVE`, `SWAPDB`, `RESTORE`, `FLUSHALL`, and `FLUSHDB` are replicated rather than refused | `MOVE`, `SWAPDB`, and `RESTORE` are always refused; `FLUSHALL` and `FLUSHDB` are refused while peers are attached | | Peer authentication | None at the module level. The link authenticates as the instance's default account, which the operator binds for you through `spec.activeRedis.redisUserRef` | Mandatory. The instance binds a `RedisUser` through `spec.activeRedis.redisUserRef` — normally its own default account, which must then carry the same password in every datacenter | -| Peer transport | Proxy Redis Serialization Protocol (RESP) port (`6379`) | The same proxy RESP port (`6379`) — one endpoint carries both transports. Active-Active additionally exposes the module peer port (`7379`) | +| Peer transport | Proxy Redis Serialization Protocol (RESP) port (`6379`) | The same proxy RESP port (`6379`) — one endpoint carries both transports. Active-Active publishes one additional Service port onto that same endpoint | | Persistence | RDB recommended | RDB only — `appendonly=yes` is rejected by admission | | Database count | Standard | `databases` must be `<= 16` | @@ -98,14 +100,14 @@ A proxy is deployed for both module generations. Enabling cross-datacenter repli | Port | Name | Purpose | Applies to | |:-----|:-----|:--------|:-----------| | `6379` | `proxy` | RESP endpoint, and the replication transport for both module generations. The proxy tells a peer session apart from a client session and routes it accordingly, so one endpoint carries both. This is the address recorded in mesh seed entries and in an `ActiveRedisConnection`'s `spec.addresses`. | All versions | -| `7379` | `peer` | The module peer port, aliased onto the same proxy listener. It is added to the proxy Service **only in Active-Active mode**, because a mesh member gossips its own local peer port and something must answer on it. | Redis `7.2`, Active-Active only | +| *Published by the Service* | `peer` | An alias onto the same proxy listener, added **only in Active-Active mode**. A mesh member gossips a peer address on a port the module fixes itself, and this entry is what answers there. Read the number from the proxy Service rather than assuming one. | Redis `7.2`, Active-Active only | -The proxy Service of a `peerof` instance carries no `7379` port, and none needs to be opened between datacenters. A Disaster Recovery link addresses the proxy RESP endpoint alone — the address recorded in the downstream's `ActiveRedisConnection`. +The proxy Service of a `peerof` instance publishes only the RESP port, and nothing else needs to be opened between datacenters. A Disaster Recovery link addresses that endpoint alone — the address recorded in the downstream's `ActiveRedisConnection`. - -When an external load balancer or firewall fronts the proxy of an Active-Active member, it must expose port `7379` as `7379`. The module always advertises its own local peer port for that traffic, so remapping it breaks the mesh. Only the RESP port may be remapped, through `announcePort`. + +When an external load balancer or firewall fronts the proxy of an Active-Active member, it must publish every port the proxy Service exposes, each under the same number. The mesh reaches members on a port the module fixes itself and advertises unchanged, so remapping that one breaks the mesh. The RESP port is the exception — it may be remapped, through `announcePort`. ### Instance ID: Service ID diff --git a/docs/en/functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx b/docs/en/functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx index ae18325..ec92509 100644 --- a/docs/en/functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx +++ b/docs/en/functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx @@ -22,7 +22,7 @@ The setup procedure differs between the two module generations. Follow the secti | Redis version | Procedure | Link authentication | |:--------------|:----------|:--------------------| -| `7.2` | [Redis 7.2 — new module](#redis-72--new-module) | The instance's `default` account, bound explicitly | +| `7.2` | [Redis 7.2 — new module](#redis-72--new-module) | A peer-auth `RedisUser`, bound explicitly — a dedicated `custom` account is recommended | | `6.0` | [Redis 6.0 — legacy module](#redis-60--legacy-module) | The instance's `default` account, bound automatically | @@ -33,15 +33,83 @@ Replication between Redis `6.0` and Redis `7.2` is not supported. See [Module ge The new module requires a **peer-auth credential**: every instance in the replication group binds a `RedisUser` through `spec.activeRedis.redisUserRef`. The operator pushes that credential to every node, and both inbound and outbound peer links authenticate with it. Enabling replication without a binding is rejected at admission, and the module rejects all inbound peer replication. -Bind the instance's own **default account**. Every instance already has one — its own reconcile creates it — so nothing has to be provisioned; the binding only names it. + +The module accepts an inbound peer only when the credential it presents matches the local peer-auth credential. Every member of the group therefore has to carry the **same username and the same password value**. Wiring a link between two datacenters whose credentials differ is refused at admission: - -The module accepts an inbound peer only when the credential it presents matches the local peer-auth credential. Binding the default account therefore requires every member of the group to carry the **same `default` password**. +``` +ActiveRedisConnection validation Connection failed with reason +ConnectionFailed(Failed to connect to the upstream: ERR WRONGPASS invalid username-password pair) +``` -This is not automatic: default passwords are generated per instance. Set the same default password on every member before you wire the links. +The bound account may be a `custom` one you create for the purpose, or the instance's own `default` account. A `system` account is always refused. -### Step 1: Confirm the default account in each datacenter \{#step-1-create-the-peer-auth-credential-in-each-datacenter} +### Step 1: Create the peer-auth credential in each datacenter \{#step-1-create-the-peer-auth-credential-in-each-datacenter} + +#### A dedicated account (recommended) + +A `custom` `RedisUser` keeps the replication credential separate from the instance password: binding the default account widens what that password grants — anything holding it can act as a replication peer — and ties credential rotation to instance-password rotation. + +Create the same account in **every** datacenter. Password Secrets are one-to-one with `RedisUser` objects, so each datacenter needs its own Secret object; what has to match across datacenters is the password *value*, not the Secret name. + +The password is policy-checked at admission: 8 to 32 characters, and it must mix letters, digits and special characters. A value that does not qualify is rejected when the `RedisUser` is created, with `password should consists of letters, number and special characters`. + +```bash +$ cat << EOF | kubectl -n default create -f - +apiVersion: v1 +kind: Secret +metadata: + name: dr-peer-auth-dc1 +stringData: + password: +--- +apiVersion: redis.middleware.alauda.io/v1 +kind: RedisUser +metadata: + name: dr-peer-auth-s72-dc1 +spec: + accountType: custom + arch: sentinel + redisName: s72-dc1 + username: dr-peer + aclRules: +@all ~* + passwordSecrets: + - dr-peer-auth-dc1 +EOF +``` + +| Field | Description | +|:------|:------------| +| `accountType` | `custom`. | +| `arch` | The architecture of the referenced instance — `sentinel`, `cluster`, or `standalone`. | +| `redisName` | The local instance this account belongs to. | +| `username` | The peer-auth username. **Identical in every datacenter.** | +| `aclRules` | `+@all ~*` — a peer replays the full command surface over every key. | +| `passwordSecrets` | One Secret, used by no other `RedisUser`, whose `password` key holds the value shared by every datacenter. | + +Wait for the account before binding it — the binding is rejected unless its phase is `Success`: + +```bash +$ kubectl -n default get redisusers dr-peer-auth-s72-dc1 +NAME INSTANCE USERNAME PHASE AGE +dr-peer-auth-s72-dc1 s72-dc1 dr-peer Success 1m +``` + +#### The instance's default account (fallback) + +The instance's own **default account** may be bound instead. It is the account Redis `6.0` links have always used, so an existing group already carries the matching password on every member. + + +Every instance has a default-account `RedisUser`, but it holds a password Secret only when the instance was created with `spec.passwordSecret`. On an instance created without one the account exists and reports `Success` while carrying no credential, and binding it leaves the instance's `ActiveRedis` in `Failed`: + +``` +peer-auth RedisUser "rfr-acl-s72-dc1-default" has no password secret: the module rejects credential-less peers +``` + +Set an instance password first, or bind a dedicated account as above. + + +Binding it requires every member of the group to carry the **same `default` password**. That is not automatic — each instance's password is whatever its own `spec.passwordSecret` holds — and each member needs its own Secret *object*: a Secret shared by several instances is contended by their `RedisUser` controllers and never settles. Set the same password *value* on every member before you wire the links. The default-account `RedisUser` is named after the instance: @@ -50,7 +118,7 @@ The default-account `RedisUser` is named after the instance: | Cluster | `drc-acl--default` | | Sentinel, standalone | `rfr-acl--default` | -Check that it is provisioned in **both** datacenters — the binding is rejected unless its phase is `Success`: +Check that it is provisioned in **both** datacenters: ```bash $ kubectl -n default get redisusers rfr-acl-s72-dc1-default @@ -60,21 +128,32 @@ rfr-acl-s72-dc1-default s72-dc1 default Success 3m The binding itself is set in Step 2, along with the rest of the replication configuration. - -Binding the default account widens what that password grants: anything holding it can act as a replication peer. A `custom` `RedisUser` narrows that, and lets the replication credential be rotated independently of the instance password. It must reference the local instance, carry a password Secret of its own that no other peer-auth `RedisUser` uses, reach `status.phase: Success`, and carry the **same username and the same password value in every datacenter**. A `system` account is always refused. - - ### Step 2: Enable replication on the upstream instance ```bash $ kubectl -n default patch redis s72-dc1 --type=merge \ - --patch='{"spec": {"activeRedis": {"serviceID": 0, "mode": "peerof", "redisUserRef": "rfr-acl-s72-dc1-default"}}}' + --patch='{"spec": {"activeRedis": {"serviceID": 0, "mode": "peerof", "redisUserRef": "dr-peer-auth-s72-dc1"}}}' ``` The range of `serviceID` is `[0-15]` and it must be unique within the replication group. It cannot be changed later, and replication cannot be turned off once enabled. + +The instance performs a **rolling restart** of its data pods when replication is first enabled — the proxy is created and the module is delivered to each pod. The instance leaves `Ready`, and its `ActiveRedis` sits in `Pending` while the pods roll: + +``` +instance s72-dc1 not ready +``` + +Wait for both to settle before continuing. The same applies to the downstream in Step 4, and Step 5 dials the upstream, so it needs the upstream settled too. + +```bash +$ kubectl -n default get redis s72-dc1 +$ kubectl -n default get activeredis +``` + + The instance must also satisfy the new module's constraints, which are enforced by admission: * `customConfig.appendonly` must not be `yes` — the new module is RDB-only. @@ -100,20 +179,22 @@ activeredis-proxy-rfr-s72-dc1 LoadBalancer 10.96.145.32 192.168.1.10 6 ``` -The proxy Service of a `peerof` instance exposes a single port, `6379`. The proxy carries the RESP control plane and the replication stream on it, telling the two apart per session, so the downstream needs to reach only that one address — the one you put in `spec.addresses`. No `7379` port is created in Disaster Recovery mode, and none has to be opened between datacenters. +The proxy Service of a `peerof` instance exposes a single port, `6379`. The proxy carries the RESP control plane and the replication stream on it, telling the two apart per session, so the downstream needs to reach only that one address — the one you put in `spec.addresses`. No additional port is created in Disaster Recovery mode, and none has to be opened between datacenters. Connection admission dials the endpoint it resolves for the replication stream and rejects the connection if it is unreachable. With `spec.peerPort` left unset, that is the address in `spec.addresses[0]` itself. ### Step 4: Enable replication on the downstream instance -Confirm the downstream's default account as in Step 1, and make sure its default password matches the upstream's. Then enable replication with a **different** `serviceID`: +Create the downstream's peer-auth account as in Step 1 — same username, same password value as the upstream's. Then enable replication with a **different** `serviceID`: ```bash $ kubectl -n default patch redis s72-dc2 --type=merge \ - --patch='{"spec": {"activeRedis": {"serviceID": 1, "mode": "peerof", "redisUserRef": "rfr-acl-s72-dc2-default"}}}' + --patch='{"spec": {"activeRedis": {"serviceID": 1, "mode": "peerof", "redisUserRef": "dr-peer-auth-s72-dc2"}}}' ``` +This restarts the downstream's data pods too. Wait for the instance to be `Ready` before Step 5. + ### Step 5: Create the connection on the downstream side The `ActiveRedisConnection` is created in the **downstream** cluster. `spec.instance` names the local (downstream) instance, and `spec.addresses` points at the upstream proxy's RESP endpoint. @@ -158,11 +239,12 @@ NAME INSTANCE STATUS MESSAGE AGE conn-dc2-to-dc1 s72-dc2 Healthy 40s $ kubectl -n default get activeredis -NAME INSTANCE SERVICEID SHARDS PEERS DPEERS PHASE MESSAGE AGE -s72-dc2-activeredis s72-dc2 1 1 1 0 Healthy 3m +NAME INSTANCE SERVICEID SHARDS PEERS DPEERS PHASE MESSAGE AGE +s72-dc1-activeredis s72-dc1 0 1 1 Healthy 16m +s72-dc2-activeredis s72-dc2 1 1 1 Healthy 12m ``` -The upstream instance's `ActiveRedis` resource reports the fan-out through `status.downstreamPeerCount`. +`PEERS` counts upstream links and `DPEERS` downstream ones, so each side prints one and leaves the other blank — a zero count is not rendered. The upstream instance's fan-out is `status.downstreamPeerCount`. Inspect the connection for per-shard synchronization detail: @@ -170,26 +252,27 @@ Inspect the connection for per-shard synchronization detail: $ kubectl -n default get activeredisconnections conn-dc2-to-dc1 -o yaml ... status: - instance: s72-dc2 credentialMode: peer-auth-global + instance: s72-dc2 shards: - index: 0 - offset: "15320" - opId: "412" + offset: "488" + opId: "20480" status: Connected syncStatus: PartialSync + versionState: verified status: Healthy upstreamPeer: service_id: 0 - service_metadata: - instance: default/s72-dc1 ``` -`status.shards[].status` indicates the connection status of the shard, and `status.shards[].syncStatus` indicates the data synchronization status: `PartialSync` (incremental synchronization of Oplog) or `FullSync` (RDB is being synchronized). +`status.shards[].status` indicates the connection status of the shard, and `status.shards[].syncStatus` indicates the data synchronization status: `PartialSync` (incremental synchronization of Oplog) or `FullSync` (RDB is being synchronized). `offset` and `opId` advance as the stream is applied — a stalled pair on a `Connected` shard is the signal to look further. + +`status.shards[].versionState` is the module's own verdict on the upstream's version — `unknown`, `verified`, or `violating`. A shard that is being held for version skew also carries a non-zero `versionHoldSince`. `status.credentialMode: peer-auth-global` records that the module-side peer records hold no inline credential — outbound dials read the node-local peer-auth credential, so a rotation is picked up on the next re-dial. -> The native Redis Sentinel mode does not have the concept of shards. Here, a master-replica pair of Sentinel is abstracted as `shard 0` to be compatible with the same data structure as the Redis cluster mode. +> The native Redis Sentinel mode does not have the concept of shards. Here, a primary-replica pair of Sentinel is abstracted as `shard 0` to be compatible with the same data structure as the Redis cluster mode. ## Redis 6.0 — legacy module \{#redis-60--legacy-module} @@ -210,6 +293,8 @@ You need to create a Redis instance first. $ kubectl -n default patch redis s6 --type=merge --patch='{"spec": {"activeRedis":{"serviceID":0 }}}' ``` + The command returns an admission warning recommending Redis `7.2`; the patch is applied. The instance then performs a **rolling restart** of its data pods to bring up the proxy, so wait for it to be `Ready` again before going on. + {/* lint ignore no-duplicate-headings-in-section */} @@ -258,11 +343,19 @@ Each disaster recovery instance will create a Proxy Service with a name that fol The range of `serviceID` is `[0-15]`. In the same disaster recovery cluster, the serviceID cannot be repeated. -#### Wait for the peer-auth binding +#### Wait for the instance and the peer-auth binding + + This side rolls its data pods too. The connection created next is rejected while the instance is still rolling — the `RedisUser` it authenticates as has to report `Success`, and it does not while its nodes are coming back: - The operator adds the binding on a following reconcile, and the connection is rejected while it is still empty. Wait until this prints a name: + ``` + peer-auth binding not usable: peer-auth RedisUser not ready: RedisUser + "rfr-acl-s6-dest-default" phase is "Pending", want Success + ``` + + The operator adds the binding on a following reconcile, and the connection is rejected while it is still empty. Wait until the instance is `Ready` and this prints a name: ```bash + $ kubectl -n default get redis s6-dest $ kubectl -n default get redis s6-dest -o jsonpath='{.spec.activeRedis.redisUserRef}' rfr-acl-s6-dest-default ``` @@ -285,11 +378,33 @@ EOF > Note to replace the upstream address. - A Redis `6.0` link authenticates as the instance's **default account**. The operator writes that down for you: the first time it reconciles an instance that has replication enabled and carries no binding, it sets `spec.activeRedis.redisUserRef` to the instance's own default-account `RedisUser` — `drc-acl--default` for a cluster instance, `rfr-acl--default` for a Sentinel one. Nothing is provisioned: that account already exists, and the password in it is the one the link uses. Both datacenters must therefore carry the same `default` password. + A Redis `6.0` link authenticates as the instance's **default account**. The operator writes that down for you: the first time it reconciles an instance that has replication enabled and carries no binding, it sets `spec.activeRedis.redisUserRef` to the instance's own default-account `RedisUser` — `drc-acl--default` for a cluster instance, `rfr-acl--default` for a Sentinel one. Nothing is provisioned: that account already exists, and the password in it is the one the link uses. Both datacenters must therefore carry the same `default` password — the same *value*, each in its own Secret object. There is no credential field on the connection. `spec.secretName` has been removed, and a manifest that still sets it is rejected with `unknown field "spec.secretName"`. + + Until v5.1, Disaster Recovery on Redis `6.0` was the only cross-datacenter replication this product offered, and the link credential lived on the connection: `ActiveRedisConnection.spec.secretName` named a Secret holding a `password` and nothing else. + + There was never a username in it. The legacy module sends a one-argument `AUTH ` when no username is set, and the proxy resolves that to the instance's **default account** — so a Redis `6.0` link has always authenticated as `default`, whatever the Secret was called. Both datacenters have therefore always had to carry the same `default` password. + + v5.1 removes `spec.secretName` from `ActiveRedisConnection` and `ActiveRedisInspection`, and takes the credential from `spec.activeRedis.redisUserRef` alone. **Existing groups need no intervention:** on its first reconcile after the upgrade the operator writes the binding on each replicating instance, pointing it at that instance's own default-account `RedisUser`. Nothing changes on the wire — same account, same password, read from somewhere else — and running links stay up across the upgrade. + + What does change is your manifests. A stored `ActiveRedisConnection` keeps working, but a manifest that still sets `secretName` is rejected on the next apply, and the field is gone from `kubectl explain`. Drop it, and manage the credential through the instance's binding instead. + + + + A `custom` `RedisUser` can be bound on Redis `6.0` as well — create it as shown in [Step 1 of the Redis 7.2 procedure](#step-1-create-the-peer-auth-credential-in-each-datacenter) and set `spec.activeRedis.redisUserRef` to it in the same patch that enables replication. A binding that is already set is never replaced by the automatic one, and the link then authenticates as that account rather than as `default`. + + What you do **not** get is rotation safety. A Redis `6.0` link carries its credential inside the module's peer record, frozen there when the link is wired, and the legacy path has no self-healing re-wire. Changing the password leaves the running link working until its session next drops — and then it stays down: + + ``` + Failed shard 0 status is Disconnected + ``` + + Recovering means deleting the `ActiveRedisConnection` and creating it again, which re-wires the peer with the current credential. Plan a rotation on Redis `6.0` as a short outage of the link, or upgrade to Redis `7.2`, where the credential is read per dial instead. + + #### Check Disaster Recovery Connection Status ```bash @@ -332,7 +447,7 @@ status: Here `status.shards[0].status` indicates the connection status of the shard, and `status.shards[0].syncStatus` indicates the data synchronization status. The synchronization status can be `PartialSync` (indicating incremental synchronization of Oplog) or `FullSync` (indicating that RDB is being synchronized). - > The native Redis Sentinel mode does not have the concept of shards. Here, a master-slave of Sentinel is abstracted as `shard 0` to be compatible with the same data structure as the Redis cluster mode. + > The native Redis Sentinel mode does not have the concept of shards. Here, a primary-replica pair of Sentinel is abstracted as `shard 0` to be compatible with the same data structure as the Redis cluster mode. @@ -381,11 +496,27 @@ Deleting an `ActiveRedisConnection` tears the link down according to its `spec.t | `Decommission` | Permanently decommissions the peer. In addition to removing the link, the module stops retaining Oplog, dead-key tombstones, and lag accounting for that peer. | No | -`Decommission` is the correct teardown for a datacenter that is gone for good — a detached-but-never-returning peer keeps holding garbage-collection floors on the survivors. It cannot be changed back to `Detach`, and a decommissioned peer that later returns is treated as a brand-new peer requiring a fresh full synchronization. +`Decommission` is the correct teardown for a datacenter that is gone for good — a detached-but-never-returning peer keeps holding garbage-collection floors on the survivors. It cannot be changed back to `Detach`: + +``` +spec.teardownPolicy is one-way: cannot change "Decommission" back to "Detach" +``` + +The decision is recorded against the peer's `serviceID` and survives a restart. What it changes is what **this** instance keeps for that peer: its Oplog retention, tombstone garbage-collection and lag accounting for that `serviceID` are released and never re-armed. Nothing stops a connection to that peer from being created again, and one created straight away still resumes from whatever the two ends happen to hold — so re-creating the link is not a way to undo the decommission, and a peer that is meant to come back should be detached, not decommissioned. ## Credential rotation -On Redis `7.2`, peer links carry no inline credential: outbound dials read the node-local peer-auth credential that the operator re-pushes on every reconcile. To rotate, update the password Secret of each datacenter's peer-auth `RedisUser` in place, to the same new value, one datacenter at a time. The `RedisUser` controller replays the ACL, the operator re-pushes the credential, and the proxy accepts the new credential on the next rebuild. Existing links stay healthy, and a link that drops after the rotation re-dials with the current credential. +On Redis `7.2`, peer links carry no inline credential: outbound dials read the node-local peer-auth credential that the operator re-pushes on every reconcile. To rotate, update the password Secret of each datacenter's peer-auth `RedisUser` in place, to the same new value, one datacenter at a time. The `RedisUser` controller replays the ACL, the operator re-pushes the credential, and the proxy accepts the new credential on the next rebuild. Established links stay healthy and keep replicating throughout — including the window in which the datacenters hold different values — and a link that drops after the rotation re-dials with the current credential. + +That window is not a free-for-all, though: while the values differ, *creating* a link is refused, because admission dials the upstream with the credential the data path will use. Finish the rotation on every member before wiring anything new. -Rotating an instance's own password Secret is independent and does not affect peer links. + +When a dedicated `custom` account is bound, rotating an instance's own password Secret is independent and does not affect peer links. + +When the instance's `default` account is bound, its password Secret **is** the peer-auth Secret: rotating the instance password rotates the replication credential with it, and the rest of the group has to follow. That is the coupling a dedicated account avoids. + + + +None of the above applies to Redis `6.0`. Its links hold the credential inline in the module's peer record, fixed when the link was wired, and there is no self-healing re-wire on that path — so a rotated link keeps running only until its session next drops, then reports `shard 0 status is Disconnected` and stays there. Delete and re-create the `ActiveRedisConnection` to re-wire it with the current credential. + diff --git a/docs/en/functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx b/docs/en/functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx index 3f768cb..6e3f418 100644 --- a/docs/en/functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx +++ b/docs/en/functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx @@ -50,7 +50,7 @@ Before starting, confirm each of the following for **every** member of the group $ kubectl -n default get redis s6-dc1 -o jsonpath='{.spec.customConfig.databases}' ``` -5. **The upstream proxy endpoint stays reachable.** The new module replicates over the same proxy endpoint the Redis `6.0` links already used, so a Disaster Recovery group needs no new port opened. Leave the existing firewall rules and load-balancer entries for the proxy's RESP port in place, so the links can be re-created at the end. +5. **The upstream proxy endpoint stays reachable.** The new module replicates over the same proxy endpoint the Redis `6.0` links already used, so a Disaster Recovery group needs no new port opened. The proxy Service is not recreated by the upgrade — it keeps its address, its port and, under NodePort, its node port — so leave the existing firewall rules and load-balancer entries for the proxy's RESP port in place, and the links can be re-created at the end against the same addresses you recorded in step 6. 6. **Record the current topology.** Note each member's `serviceID`, and which member holds the `ActiveRedisConnection` pointing at which upstream. You will re-create these links. ```bash @@ -104,6 +104,18 @@ Because the links were deleted in Step 2, there is no live replication to order The ordering rule matters for later, within-generation upgrades of a **live** group: the safe order is **downstream before upstream**, because a downstream can apply an equal-or-older upstream's Oplog but never a newer one. Bumping an upstream ahead of a live downstream returns an admission warning, and the module holds the affected edge — replication to that downstream pauses until the version skew is resolved. + +The same guard covers this upgrade. A connection created while the upstream is already on Redis `7.2` and the downstream is still on `6.0` is accepted, then held — it does not replicate, and it does not damage either side: + +```bash +$ kubectl -n default get activeredisconnections +NAME INSTANCE STATUS MESSAGE AGE +conn-dc2-to-dc1 s6-dc2 Paused upstream version 7.2 newer than local version 6.0; waiting for local upgrade (safe order upgrades the downstream first) 18s +``` + +It stays `Paused` until the local member is upgraded. Following the procedure — links deleted in Step 2, re-created in Step 5 — you will not see this; it is what a mistimed link looks like if you do. + + ### Step 4: Confirm the peer-auth binding The new module requires a peer-auth credential on every instance. On an instance that already had cross-datacenter replication enabled, the platform binds one automatically — see [Automatic peer-auth binding](#automatic-peer-auth-provisioning). @@ -122,7 +134,7 @@ rfr-acl-s6-dc1-default s6-dc1 default Success 8d If the binding is empty, or the `RedisUser` phase is not `Success`, resolve it as described in [If the binding is not there yet](#when-automatic-provisioning-is-on-hold) before continuing. -The module accepts an inbound peer only when the username it presents matches the local peer-auth username. The automatic binding uses each instance's `default` account, so members that were already replicating to each other converge on the same credential — that shared `default` password is exactly what their Redis `6.0` links were authenticating with. +The module accepts an inbound peer only when the credential it presents — username *and* password — matches the local peer-auth credential. The automatic binding uses each instance's `default` account, so members that were already replicating to each other converge on the same credential: that shared `default` password is exactly what their Redis `6.0` links were authenticating with. If you bound credentials manually on any member, verify that **every** member carries the same username and the same password value before creating any link. @@ -160,8 +172,9 @@ Confirm that: * the connection status is `Healthy`; * `status.credentialMode` is `peer-auth-global`; * `status.shards[].status` is `Connected` and the offsets advance; +* `status.shards[].syncStatus` is `PartialSync` — the new module picks the link up from the offset the Redis `6.0` link left off at, so a re-created link resumes rather than re-sending the dataset. A `FullSync` here means the retained state was not usable and the whole dataset is being re-sent; it will still converge, but it costs a full transfer; * the upstream's `ActiveRedis` reports the expected `status.downstreamPeerCount`; -* a key written on the upstream appears on each downstream. +* a key written on the upstream appears on each downstream, alongside the keys that were there before the upgrade. Then restore normal client traffic. @@ -212,5 +225,5 @@ Once the group is running on Redis `7.2`, the following become available: ## Known gaps in this draft * The procedure requires a full-group maintenance window. There is no supported way to upgrade a group while keeping replication live. -* Downgrading from Redis `7.2` back to Redis `6.0` is not supported. If the upgrade must be abandoned, restore the members from the backups taken in the prerequisites. +* Downgrading from Redis `7.2` back to Redis `6.0` is not supported — and **nothing prevents it**. Setting `spec.version` back to `6.0` is accepted by admission and starts rolling the pods onto the older image immediately. If the upgrade must be abandoned, restore the members from the backups taken in the prerequisites rather than reversing the version. * The Web Console does not provide a guided flow for this upgrade; the steps above are CLI-driven. diff --git a/docs/en/functions/95-disaster-recovery/30-active-active/10-architecture.mdx b/docs/en/functions/95-disaster-recovery/30-active-active/10-architecture.mdx index 3a384f5..ea8829c 100644 --- a/docs/en/functions/95-disaster-recovery/30-active-active/10-architecture.mdx +++ b/docs/en/functions/95-disaster-recovery/30-active-active/10-architecture.mdx @@ -41,13 +41,13 @@ The practical differences this produces: * **Adding a member does not require touching the existing ones.** A new member joins by naming any reachable existing member as a seed, and the whole group learns about it through gossip. Updating the other members' seed lists afterwards is still recommended, so that a member restarting while the original seeds are unreachable can still rejoin. * **Seeds are a bootstrap list, not a membership list.** One reachable seed per remote datacenter is enough, and the local member's own entry may be included — which lets every datacenter carry one identical seed list. -* **Membership is state that converges**, so it is reported rather than declared. The `ActiveRedisMesh` status carries the member list, each member's state, and an `epoch` that advances as membership changes. +* **Membership is state that converges**, so it is reported rather than declared. The `ActiveRedisMesh` status carries the member list and each member's state. It also carries an `epoch`, which is not a change counter: it advances only when a member is forgotten or permanently decommissioned, so a group that has only ever gained members reports `0`. ## What travels on which port A member is identified in the mesh by its **announced RESP address** — `announceAddress` with `announcePort`, normally `6379`. That is the address written in seed entries and carried in the membership records that gossip exchanges, and it must be routable from every other datacenter. -The traffic itself — both the gossip rounds and the replicated Oplog — is carried on the **peer port**, `7379`. The module always advertises its own local peer port for that traffic, which is why an external load balancer or firewall in front of the proxy must expose `7379` as `7379`. Only the RESP port may be remapped. +The traffic itself — both the gossip rounds and the replicated Oplog — is carried on the **peer port**: a second number the proxy Service publishes onto that same listener, which you can read off the Service. The module fixes that number itself and advertises it unchanged, which is why an external load balancer or firewall in front of the proxy must publish it under the same number. Only the RESP port may be remapped. As in Disaster Recovery mode, peers address the proxy of an instance rather than an individual pod, and the proxy resolves each connection to the primary that is current at that moment. diff --git a/docs/en/functions/95-disaster-recovery/30-active-active/20-setup.mdx b/docs/en/functions/95-disaster-recovery/30-active-active/20-setup.mdx index 6c092d9..ae9e638 100644 --- a/docs/en/functions/95-disaster-recovery/30-active-active/20-setup.mdx +++ b/docs/en/functions/95-disaster-recovery/30-active-active/20-setup.mdx @@ -18,12 +18,84 @@ Setting up Active-Active replication has two parts: enabling Active-Active mode * Every member runs Redis `7.2`. * Each member has a distinct `serviceID` in the range `[0-15]`. -* Each member's proxy is reachable from every other member on both port `6379` (RESP, used by seeds) and port `7379` (peer port, which carries both gossip and replication). +* Each member has an externally-routable `announceAddress` set explicitly — see [Step 2](#step-2-enable-active-active-mode). Nothing defaults it to a cross-datacenter address, and admission only warns, so a member left without one advertises an address its peers cannot reach. +* Each member's proxy endpoint is reachable from every other member on **every port its proxy Service publishes**, each under its own number — not only the announced one. * **The system clocks of all members are synchronized**, through a time-synchronization daemon (`ntpd`, `chronyd`, or the platform equivalent) on every node of every cluster in the group. This is a hard prerequisite, not a recommendation — see [Clock synchronization](./30-operations.mdx#clock-synchronization-requirement). -## Step 1: Confirm the default account in each datacenter \{#step-1-create-the-peer-auth-credential-in-each-datacenter} +## Step 1: Provision the peer-auth credential in each datacenter \{#step-1-create-the-peer-auth-credential-in-each-datacenter} -Every peer link in an Active-Active group authenticates, and every member must present the same credential. Bind the instance's own **default account**: every instance already has one, created by its own reconcile, so nothing has to be provisioned. This is the same credential Disaster Recovery uses — if these instances already carry a binding, skip to Step 2. +Every peer link in an Active-Active group authenticates, and every member must present the same credential: a member accepts an inbound peer only when the username it presents equals the member's own peer-auth username and the password validates against Redis ACL. Each instance binds that credential through `spec.activeRedis.redisUserRef`, which names a `RedisUser` **object** in the instance's own namespace. + +Whichever account you bind, it must: + +* reference the **local** instance — a `RedisUser` that targets another instance is refused; +* be a `custom` or a `default` account; a `system` account is always refused; +* own a password Secret carrying a `password` key; +* reach `status.phase: Success` — the binding is rejected until then; +* carry the **same username and the same password value in every datacenter**. + +### A dedicated account (recommended) + +Create a `custom` `RedisUser` for this purpose in **every** datacenter. It narrows what the credential grants — anything holding the default password can otherwise act as a mesh peer — and lets the replication credential be rotated independently of the instance password. + +Password Secrets are one-to-one with `RedisUser` objects, so each datacenter needs its own Secret object; what has to match across datacenters is the password *value*, not the Secret name. + +The password is policy-checked at admission: 8 to 32 characters, and it must mix letters, digits and special characters. A value that does not qualify is rejected when the `RedisUser` is created, with `password should consists of letters, number and special characters`. + +```bash +$ cat << EOF | kubectl -n default create -f - +apiVersion: v1 +kind: Secret +metadata: + name: aa-peer-auth-dc1 +stringData: + password: +--- +apiVersion: redis.middleware.alauda.io/v1 +kind: RedisUser +metadata: + name: aa-peer-auth-s72-dc1 +spec: + accountType: custom + arch: sentinel + redisName: s72-dc1 + username: aa-peer + aclRules: +@all ~* + passwordSecrets: + - aa-peer-auth-dc1 +EOF +``` + +| Field | Description | +|:------|:------------| +| `accountType` | `custom`. | +| `arch` | The architecture of the referenced instance — `sentinel`, `cluster`, or `standalone`. | +| `redisName` | The local instance this account belongs to. | +| `username` | The peer-auth username. **Identical in every datacenter.** | +| `aclRules` | `+@all ~*` — a peer replays the full command surface over every key. | +| `passwordSecrets` | One Secret, used by no other `RedisUser`, whose `password` key holds the value shared by every datacenter. | + +Wait for the account before binding it: + +```bash +$ kubectl -n default get redisusers aa-peer-auth-s72-dc1 +NAME INSTANCE USERNAME PHASE AGE +aa-peer-auth-s72-dc1 s72-dc1 aa-peer Success 1m +``` + +### The instance's default account (fallback) + +The instance's own **default account** may be bound instead — this is the same credential Disaster Recovery uses, so if these instances already carry a binding, you can skip to Step 2. + + +Every instance has a default-account `RedisUser`, but it holds a password Secret only when the instance was created with `spec.passwordSecret`. On an instance created without one the account exists and reports `Success` while carrying no credential, and binding it leaves the instance's `ActiveRedis` in `Failed`: + +``` +peer-auth RedisUser "rfr-acl-s72-dc1-default" has no password secret: the module rejects credential-less peers +``` + +Set an instance password first, or bind a dedicated account as above. + The default-account `RedisUser` is named after the instance: @@ -32,28 +104,26 @@ The default-account `RedisUser` is named after the instance: | Cluster | `drc-acl--default` | | Sentinel, standalone | `rfr-acl--default` | -Check that it is provisioned in each datacenter — the binding is rejected unless its phase is `Success`: - ```bash $ kubectl -n default get redisusers rfr-acl-s72-dc1-default NAME INSTANCE USERNAME PHASE AGE rfr-acl-s72-dc1-default s72-dc1 default Success 3m ``` - -The module accepts an inbound peer only when the credential it presents matches the local peer-auth credential. Binding the default account therefore requires every member of the mesh to carry the **same `default` password**, and default passwords are generated per instance — set them to a common value before enabling the mode. - - - -A `custom` `RedisUser` narrows what the credential grants — anything holding the default password can otherwise act as a mesh peer — and lets the replication credential be rotated independently of the instance password. It must reference the local instance, own its password Secret, reach `status.phase: Success`, and carry the **same username and the same password value in every datacenter**. A `system` account is always refused. + +Every member must then carry the **same `default` password**, and default passwords are generated per instance — set them to a common value before enabling the mode. Anything holding that password can act as a mesh peer, and rotating the instance password rotates the replication credential with it. A dedicated account avoids both. -## Step 2: Enable Active-Active mode on each instance +## Step 2: Enable Active-Active mode on each instance \{#step-2-enable-active-active-mode} These steps assume instances that are being placed into an Active-Active group from the start. Admission does not block changing `spec.activeRedis.mode` from `peerof` to `mesh` on an instance that is already replicating, but converting an established Disaster Recovery group to Active-Active is not a covered procedure and has not been validated. Build an Active-Active group from instances that are not already wired into a Disaster Recovery group. + +Turning on Active-Active adds the module's init container to the instance, so the operator rolls the Redis pods and the instance passes back through `Initializing` before it is `Ready` again. Complete Step 1 first: while an instance is not `Ready`, a `custom` `RedisUser` cannot be created against it — admission refuses with `redis failover is not ready` — so enabling the mode before the account exists leaves you waiting out the restart to create it. + + Members must advertise an externally-routable address, because a peer re-resolves it on every dial. This is the `announceAddress`, set on the proxy Service configuration. ```bash @@ -62,7 +132,7 @@ spec: activeRedis: serviceID: 0 mode: mesh - redisUserRef: rfr-acl-s72-dc1-default + redisUserRef: aa-peer-auth-s72-dc1 proxy: service: type: LoadBalancer @@ -75,24 +145,41 @@ spec: |:------|:------------| | `mode` | `mesh`. | | `serviceID` | Unique within the mesh, range `[0-15]`, immutable. | -| `redisUserRef` | The peer-auth `RedisUser` binding — normally the instance's default account. Mandatory. | +| `redisUserRef` | The peer-auth `RedisUser` binding from Step 1. Mandatory. | | `proxy.service.announceAddress` | The externally-reachable host peers dial to reach this member — the host on the external load balancer or VIP, decoupled from the in-cluster Service. Must be a **bare host**: no scheme, no port, no path. | -| `proxy.service.announcePort` | The externally-reachable RESP port for the gossip control plane. Defaults to the proxy Service port `6379`; override it only when an external load balancer exposes that port under a different number. | +| `proxy.service.announcePort` | The externally-reachable RESP port of this member's proxy endpoint. Together with `announceAddress` it forms the pair peers put in their seed lists for this member, and the pair the member matches to recognize its own seed entry. Defaults to the proxy Service port `6379`; override it only when an external load balancer exposes that port under a different number. | + + +`ActiveRedis` exposes the same `spec.proxy.service` fields, but that resource is derived: the operator rewrites its `spec.proxy` from the `Redis` instance on every reconcile — in mesh mode writing the *resolved* announce address into it — so an edit made there is overwritten. Always patch the `Redis` object. + * A **DNS name is recommended**: the module re-resolves it on every dial, so it survives IP changes, and a multi-A record can cover several NodePort addresses. When the announce address is a DNS name, the operator binds it to the proxy Service through ExternalDNS. * A **bare IP is accepted with a warning** — it will not re-resolve after an IP change, so it is only appropriate for a stable VIP. -* Leaving it **empty produces a warning** and the module advertises the pod IP, which is not routable across datacenters. -* An address containing a scheme (`tcp://…`), a port (`host:7379`), or a path is **rejected**. The port belongs in `announcePort`. +* Leaving it **empty produces a warning**, and what happens then depends on the operator: if it is configured with a managed DNS zone, it derives `..` and publishes that on the proxy Service through ExternalDNS; with no zone configured it sends no announce address at all and the module advertises the **pod IP**, which is not routable across datacenters. Nothing else fills the field in. +* An address containing a scheme (`tcp://…`), a port (`host:6379`), or a path is **rejected**. The port belongs in `announcePort`. - -`announcePort` applies to the gossip control plane only. The module always advertises its own local peer port (`7379`) for replication, so any external endpoint must expose `7379` as `7379`. + +`announcePort` is an identity value: it is how this member names its endpoint to its peers, and it carries no traffic of its own. Neither gossip nor replication is steered by it, and it does not override how the module addresses a peer. + +The proxy runs a **single listener** — application traffic, gossip, and replication all arrive on it and are told apart per session — but the proxy Service publishes more than one port onto it. When an external load balancer or firewall fronts the proxy, forward **every port that Service publishes**, each under its own number. Remapping any of them to a different number breaks the mesh. ## Step 3: Create an ActiveRedisMesh in each cluster -Each Kubernetes cluster owns one `ActiveRedisMesh` for its local instance. `spec.seeds` names the externally-routable proxy RESP addresses of members in other datacenters. +Each Kubernetes cluster owns one `ActiveRedisMesh` for its local instance. `spec.seeds` names the announced endpoints of members in other datacenters — the `announceAddress` and `announcePort` those members were given in Step 2. + + +Step 2 rolls the pods, and the module arrives with them. Create the `ActiveRedisMesh` before the roll reaches the shard's primary node, and the mesh lands in `Failed`, reporting that the module's own commands do not exist: + +``` +set mesh config on pod rfr-s72-dc1-0: ERR Unknown option or number of arguments for CONFIG SET +- 'activeredis.mesh-seeds'; shard 0 mesh info: ERR unknown command 'as.mesh' +``` + +This clears by itself — the mesh moves to `Pending` once the primary node carries the module, then to `Healthy` — but it is indistinguishable from a real failure while it lasts. Wait for `kubectl get redis` to report `Ready`. + ```bash $ cat << EOF | kubectl -n default create -f - @@ -110,7 +197,9 @@ EOF ### Seed syntax -A seed is `host:port`, where the port is the remote proxy's RESP endpoint — normally `6379`. An optional `@peer-port` suffix overrides the peer port when an external load balancer remaps it; when the suffix is omitted the operator appends the conventional `7379`. The operator normalizes the list before injecting it, so `redis-dc2.example.com:6379` becomes `redis-dc2.example.com:6379@7379`. +A seed is `host:port` — the remote member's **`announceAddress:announcePort`**, written exactly as that member advertises itself. It is an externally-routable endpoint, so when a load balancer or VIP fronts that member's proxy, the seed is the load balancer's address and port, not the in-cluster proxy Service. Write the pair verbatim: a member recognizes its own entry by comparing a seed's address and port with the pair it announces, and a mismatch defeats that self-skip. + +Write seeds as plain `host:port` and let the operator complete them. It normalizes the list into the module's internal seed grammar before injecting it, so `CONFIG GET activeredis.mesh-seeds` reports each entry in a longer form than the one you wrote. That completion is required for a seed to be usable at all, and there is nothing in it for you to set. One reachable seed per remote datacenter is enough — gossip discovers everything else. The local member's own entry may be included, so every datacenter can carry the same uniform seed list; the module skips its own record. @@ -141,41 +230,45 @@ $ kubectl -n default get activeredismesh mesh-dc1 -o yaml status: phase: Healthy instance: s72-dc1 - epoch: 7 memberCount: 3 aliveCount: 3 - suspectCount: 0 - deadCount: 0 fleetFloorVersion: "7.2" versionConverged: true members: - uid: 9f2c1a... serviceID: 0 shardID: 0 - address: redis-dc1.example.com state: alive - epoch: 7 - clockOffsetMs: 2 version: "7.2" - moduleVersion: 10 - uid: 4b81d3... serviceID: 1 shardID: 0 - address: redis-dc2.example.com + address: redis-dc2.example.com:6379 state: alive - epoch: 7 clockOffsetMs: -5 version: "7.2" moduleVersion: 10 + - uid: 07d65b... + serviceID: 2 + shardID: 0 + address: redis-dc3.example.com:6379 + state: alive + clockOffsetMs: -2 + version: "7.2" + moduleVersion: 10 ``` + +The first entry above is this instance's own, and it carries no `address`, `clockOffsetMs` or `moduleVersion`. That is expected, not a missing member: gossip reports the *other* members, so the operator fills the local record in from what the instance knows about itself, and those three fields have no local source. Every remote member carries them. Counts that are zero — `suspectCount`, `deadCount` — are omitted rather than printed. + + ### Mesh phases | Phase | Meaning | |:------|:--------| | `Healthy` | Membership is converged and every known member is alive. | | `Degraded` | The mesh is operating with suspect or dead members. | -| `Pending` | No ready master could be reached to read membership yet — the bootstrap window, or an outage. | +| `Pending` | No ready primary node could be reached to read membership yet — the bootstrap window, or an outage. | | `Paused` | `spec.pause` is `true`. | | `Failed` | The mesh could not be configured; `status.message` carries the reason (for example `invalid mesh seed`). | diff --git a/docs/en/functions/95-disaster-recovery/30-active-active/30-operations.mdx b/docs/en/functions/95-disaster-recovery/30-active-active/30-operations.mdx index 4cd37b4..aa373b1 100644 --- a/docs/en/functions/95-disaster-recovery/30-active-active/30-operations.mdx +++ b/docs/en/functions/95-disaster-recovery/30-active-active/30-operations.mdx @@ -49,12 +49,14 @@ Start from the resource, which reports the offset to every member the local inst ```bash $ kubectl -n default get activeredismesh mesh-dc1 \ -o jsonpath='{range .status.members[*]}{.address}{"\t"}{.state}{"\t"}{.clockOffsetMs}{"\n"}{end}' -redis-dc1.example.com alive 0 -redis-dc2.example.com alive -5 -redis-dc3.example.com alive 4180 + alive +redis-dc2.example.com:6379 alive -5 +redis-dc3.example.com:6379 alive 4180 ``` -For the aggregated gauges and the attribution flag, read them on the instance itself: +The first row is the local member. It carries no address and no offset — it is the reference the other offsets are measured against, so only the remote rows are readable here. + +For the aggregated gauges and the attribution flag, read them on the instance itself. Run this **against the shard's primary node**: only the primary takes part in gossip, so the mesh counters on a replica describe whatever reached it passively and are not a view of the mesh. ```bash $ redis-cli INFO activeredis | grep -E \ @@ -70,7 +72,7 @@ $ redis-cli INFO activeredis | grep -E \ | `clock_skew_critical_violations` | How many critical episodes have occurred. | | `local_clock_is_outlier` | `1` when **this** node's clock is the one that disagrees with all the others. | -Per-member offsets are also available directly from the module, as `clock_offset_ms` on each line: +Per-member offsets are also available directly from the module, as `clock_offset_ms` on each line. Run it against the shard's primary node, for the reason above: ```bash $ redis-cli as.mesh nodes diff --git a/docs/en/functions/95-disaster-recovery/90-limitations.mdx b/docs/en/functions/95-disaster-recovery/90-limitations.mdx index a6e6ca8..a217296 100644 --- a/docs/en/functions/95-disaster-recovery/90-limitations.mdx +++ b/docs/en/functions/95-disaster-recovery/90-limitations.mdx @@ -103,7 +103,7 @@ Run a time-synchronization daemon (`ntpd`, `chronyd`, or the platform equivalent The proxy address itself is not highly available. If you expose the proxy via NodePort, place it behind a front-end load balancer to ensure HA; alternatively, use MetalLB to provide a highly available virtual IP for the proxy. -In Active-Active mode on Redis `7.2` the address must expose **both** the RESP port (`6379`) and the peer port (`7379`), and the peer port must not be remapped — the module always advertises its own local peer port for gossip. A Disaster Recovery link uses the RESP endpoint alone. +In Active-Active mode on Redis `7.2` the address must expose **every port the proxy Service publishes**, each under its own number. The mesh reaches members on a port the module fixes itself and advertises unchanged, so remapping it breaks gossip. A Disaster Recovery link uses the RESP endpoint alone. ### Only IP connection addresses are supported (Redis 6.0) diff --git a/docs/en/release_notes.mdx b/docs/en/release_notes.mdx index 32641ee..ab17e5f 100644 --- a/docs/en/release_notes.mdx +++ b/docs/en/release_notes.mdx @@ -30,7 +30,7 @@ Redis `7.2` instances now run a redesigned replication module, offering two mode * **Disaster Recovery** (`peerof`, the default) is the hot-standby topology, with directed links from one upstream to up to 15 downstream instances, each ready to be promoted if the upstream datacenter is lost. It remains available on Redis `6.0`. * **Active-Active** (`mesh`) lets every datacenter serve reads and writes locally, with conflicts resolved automatically and membership maintained by a gossip mesh seeded from a handful of addresses. It is available on Redis `7.2` only, and is driven by the new `ActiveRedisMesh` resource. -Replication and gossip on Redis `7.2` ride the instance proxy, which tells the two transports apart on a single endpoint. Active-Active additionally exposes the module peer port (`7379`) on the proxy Service, so that mesh members can gossip. +Replication and gossip on Redis `7.2` ride the instance proxy, which tells the two transports apart on a single endpoint. Active-Active publishes one additional port on the proxy Service, aliased onto that same listener, so that mesh members can gossip. See [Cross-Datacenter Replication](./functions/95-disaster-recovery/10-intro.mdx). @@ -42,10 +42,18 @@ The credential is delivered to the nodes at runtime and is never written into th #### Upgrade Path from Redis 6.0 to Redis 7.2 for Replicating Instances -An instance with cross-datacenter replication enabled can now be upgraded in place from Redis `6.0` to Redis `7.2`. When such an instance carries no peer-auth binding, the platform provisions one automatically, deriving it from the credential the replication group already shares. +An instance with cross-datacenter replication enabled can now be upgraded in place from Redis `6.0` to Redis `7.2`. When such an instance carries no peer-auth binding, the platform writes one for it, naming the instance's own default-account `RedisUser` — the account a Redis `6.0` link has always authenticated as. Nothing is provisioned, and the credential does not change during the upgrade. The two module generations do not interoperate, so the whole group must be upgraded together within a maintenance window. See [Upgrade a Disaster Recovery Group from Redis 6.0 to Redis 7.2](./functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx). +#### The Link Credential Moves to the Instance + +Until this release, a Redis `6.0` replication link took its credential from the connection: `ActiveRedisConnection.spec.secretName` named a Secret holding a password. That Secret carried no username, so the link authenticated as the instance's default account regardless — which is now where the credential is declared. + +`spec.secretName` is **removed** from `ActiveRedisConnection` and `ActiveRedisInspection`. Both module generations take the link credential from `spec.activeRedis.redisUserRef` and nothing else. + +Existing Disaster Recovery groups are migrated without intervention: on its first reconcile after the upgrade the operator binds each replicating instance to its own default-account `RedisUser`, which changes nothing on the wire, and running links stay up. Stored resources keep working, but a manifest that still sets `secretName` is rejected on its next apply. + #### Teardown Policy for Replication Links `ActiveRedisConnection` now carries a `teardownPolicy`. `Detach` (the default) removes the link but keeps the peer's bookkeeping so the link can be re-created later. `Decommission` permanently retires a peer and releases the retention floors held for it — the correct teardown for a datacenter that is gone for good. diff --git a/docs/en/upgrade.mdx b/docs/en/upgrade.mdx index 96c85f8..80d19ae 100644 --- a/docs/en/upgrade.mdx +++ b/docs/en/upgrade.mdx @@ -64,6 +64,14 @@ Confirm that no instance reports `5.0` in the `VERSION` column (for example, run If an instance has cross-datacenter replication enabled, its Redis version cannot be changed in isolation. Follow [Upgrade a Disaster Recovery Group from Redis 6.0 to Redis 7.2](./functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx), which coordinates the whole replication group. + +Upgrading the operator to v5.1 does not interrupt an existing Disaster Recovery group. Replication continues across the upgrade and the data on both sides is unaffected — the instances stay on their current Redis version, so they keep running the module generation they were already using. + +One thing does move. In v5.1 the link credential is declared on the instance through `spec.activeRedis.redisUserRef` instead of on the connection through `spec.secretName`, and the operator writes that binding for each replicating instance on its first reconcile after the upgrade. It names the instance's own default account — the account a Redis `6.0` link has always authenticated as — so nothing changes on the wire and no action is required. + +Your manifests do need a change: `spec.secretName` is removed from `ActiveRedisConnection` and `ActiveRedisInspection`, and a manifest that still sets it is rejected on its next apply. See [Set Up Disaster Recovery Replication](./functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx#redis-60--legacy-module). + + ## Upgrade Strategies ### Minor Version Upgrade diff --git a/docs/shared/crds/middleware.alauda.io_redis.yaml b/docs/shared/crds/middleware.alauda.io_redis.yaml index d20573f..3f1e2c2 100644 --- a/docs/shared/crds/middleware.alauda.io_redis.yaml +++ b/docs/shared/crds/middleware.alauda.io_redis.yaml @@ -1567,17 +1567,26 @@ spec: type: string announcePort: description: |- - AnnouncePort is the externally-reachable RESP port peers dial at - AnnounceAddress for the mesh CONTROL plane (gossip) and the legacy - as.peersync transport — module-side it becomes the announced node - record's gossip/legacy dial port (mesh-announce-port). Behind the - operator this is the proxy's RESP listener, so it defaults to the proxy - Service port (6379); override it only when an external LB/F5 exposes - that port under a different number. It does NOT affect the AS_MESH data - plane: the module always advertises its own local peer-port (7379) for - that, so any external endpoint must expose 7379 as 7379. It is DECOUPLED - from the in-cluster Service Port/NodePort; the operator does not copy it - into the k8s Service NodePort. + AnnouncePort is the RESP port this instance publishes as its mesh + IDENTITY — module-side it becomes field 7 of the announced node record + (mesh-announce-port). Behind the operator that is the proxy's RESP + listener, so it defaults to the proxy Service port (6379); override it + only when an external LB/F5 exposes that port under a different number. + + It carries NO traffic. Both mesh planes dial the PEER port instead: the + AS_MESH data plane through connectPeerPort, and gossip through + gossipOnePeerPort, which is the only gossip transport left since the + plaintext redis-command fallback was removed with the legacy as.peersync + path (redis-modules active-redis src/mesh.zig, gossipOne). What this + value is actually read for is self-recognition: a node matches a seed + against its own announced addr+port (seedMatchesSelfLocked), which is + what lets every DC carry one identical mesh-seeds list. + + Because the module advertises its own local peer-port verbatim and there + is no announce override for it, any external endpoint must expose 7379 + as 7379 — remapping it leaves both planes dialling a port nothing serves. + This field is DECOUPLED from the in-cluster Service Port/NodePort; the + operator does not copy it into the k8s Service NodePort. format: int32 maximum: 65535 minimum: 0 diff --git a/docs/shared/crds/redis.middleware.alauda.io_activeredis.yaml b/docs/shared/crds/redis.middleware.alauda.io_activeredis.yaml index 331c62f..d2b5dad 100644 --- a/docs/shared/crds/redis.middleware.alauda.io_activeredis.yaml +++ b/docs/shared/crds/redis.middleware.alauda.io_activeredis.yaml @@ -1560,17 +1560,26 @@ spec: type: string announcePort: description: |- - AnnouncePort is the externally-reachable RESP port peers dial at - AnnounceAddress for the mesh CONTROL plane (gossip) and the legacy - as.peersync transport — module-side it becomes the announced node - record's gossip/legacy dial port (mesh-announce-port). Behind the - operator this is the proxy's RESP listener, so it defaults to the proxy - Service port (6379); override it only when an external LB/F5 exposes - that port under a different number. It does NOT affect the AS_MESH data - plane: the module always advertises its own local peer-port (7379) for - that, so any external endpoint must expose 7379 as 7379. It is DECOUPLED - from the in-cluster Service Port/NodePort; the operator does not copy it - into the k8s Service NodePort. + AnnouncePort is the RESP port this instance publishes as its mesh + IDENTITY — module-side it becomes field 7 of the announced node record + (mesh-announce-port). Behind the operator that is the proxy's RESP + listener, so it defaults to the proxy Service port (6379); override it + only when an external LB/F5 exposes that port under a different number. + + It carries NO traffic. Both mesh planes dial the PEER port instead: the + AS_MESH data plane through connectPeerPort, and gossip through + gossipOnePeerPort, which is the only gossip transport left since the + plaintext redis-command fallback was removed with the legacy as.peersync + path (redis-modules active-redis src/mesh.zig, gossipOne). What this + value is actually read for is self-recognition: a node matches a seed + against its own announced addr+port (seedMatchesSelfLocked), which is + what lets every DC carry one identical mesh-seeds list. + + Because the module advertises its own local peer-port verbatim and there + is no announce override for it, any external endpoint must expose 7379 + as 7379 — remapping it leaves both planes dialling a port nothing serves. + This field is DECOUPLED from the in-cluster Service Port/NodePort; the + operator does not copy it into the k8s Service NodePort. format: int32 maximum: 65535 minimum: 0