diff --git a/TERMINOLOGY_CANDIDATES.md b/TERMINOLOGY_CANDIDATES.md index e3b4cf2..79d734b 100644 --- a/TERMINOLOGY_CANDIDATES.md +++ b/TERMINOLOGY_CANDIDATES.md @@ -212,3 +212,47 @@ definitive answer needs a register search or counsel. | **Active-Passive** | Clear, same class as Active-Active. Considered and not chosen. | | **Multi-Active** | **Avoided.** Cockroach Labs presents "Multi-Active Availability" as its own coined term, explicitly contrasted with generic active-active. No ™ and no registration found (their registered mark is COCKROACHDB, Reg. 5307353), but adopting it would read as derivative. | | **Mesh** | No trademark issue, but collides with **Service Mesh** in the ACP console. Retained only as a mechanism word. | + +## `master` is a forbidden word in the shared glossary (added 2026-09-10) + +The shared terms list maps **`master` → `control plane`**, so `doom lint` reports +`Forbidden word: "master" (control plane)` on any occurrence. The mapping is a +Kubernetes node-naming rule, and it fires on Redis's own replication role, which +is genuinely called `master` — `INFO replication` reports `role:master`, and the +Redis CR carries a `spec.replicas.sentinel.master` field. + +Two things make this easy to miss: + +* **The local run does not catch it.** `flagWords` are built from the shared + terms file, which is fetched over the network; when that fetch is blocked the + local `yarn lint` has an empty flag list and reports only unknown words. +* **CI reports changed lines only,** so the existing occurrences below have never + failed a build. They will, on whichever pull request next edits those lines. + +Verified against `cspell-lib` with the mapping applied: the hyphenated forms +**are** flagged (`master-replica` and `master-slave` tokenize to `master`), while +`mymaster` is not, being a single token. + +Pages edited in the cross-datacenter replication pass now use **primary node** / +**primary-replica pair** and are clean. That is a local choice made to get the +pass through CI, not a registered term — confirm it, or choose another. + +| Still carrying `master` | Line | Kind | +|:---|:---|:---| +| `functions/95-disaster-recovery/90-limitations.mdx` | 50 | prose — "master-replica synchronization" | +| `intro.mdx` | 55 | prose — "master-slave relationships" | +| `functions/40-accessmethod.mdx` | 51 | prose — "Redis master-replica group" | +| `functions/75-monitor.mdx` | 35 | prose — alert name "Master-Slave Failover" | +| `how_to/access/10-sentinel.mdx` | 71 | prose — "master-slave cluster name" | + +**Do not rewrite these mechanically.** The same files also carry `master` as an +identifier, where changing it would be wrong: `spec.replicas.sentinel.master` in +`functions/10-create-instance.mdx`, and `mymaster` / `MasterName` / +`setMasterName` / `withSentinelMasterId` in `how_to/access/10-sentinel.mdx` — +Sentinel's fixed group name and client-library API surface. A blanket +find-and-replace breaks working examples. + +Decisions needed: whether `primary node` is the accepted replacement; whether the +prose rows above should be converted in one pass; and whether an exemption for +the Redis role sense belongs in the shared glossary instead, since the product +will keep meeting the word. diff --git a/docs/en/functions/95-disaster-recovery/10-intro.mdx b/docs/en/functions/95-disaster-recovery/10-intro.mdx index ed55346..b42a0b2 100644 --- a/docs/en/functions/95-disaster-recovery/10-intro.mdx +++ b/docs/en/functions/95-disaster-recovery/10-intro.mdx @@ -65,6 +65,8 @@ The `spec.activeRedis.mode` field takes the literal values `peerof` and `mesh`. The replication module ships in two generations. Which one an instance loads is decided by its Redis version, and the generation determines the feature set, the authentication model, and the peer transport. +Before v5.1 there was only one: Disaster Recovery on Redis `6.0`, introduced in v4.1.0. Redis `7.2` instances and the Active-Active mode arrived with the new module in v5.1, and the legacy generation is now maintained for existing instances only. + | | Redis `6.0` (legacy module) | Redis `7.2` (new module) | |:--|:--|:--| | Module | Frozen legacy implementation, maintained for existing instances only | Actively developed implementation | @@ -73,7 +75,7 @@ The replication module ships in two generations. Which one an instance loads is | Replicated command set | Smaller — commands introduced after Redis 6.0 are not replicated | Full; see [Command Support](./60-commands.mdx) | | Command guards | None — `MOVE`, `SWAPDB`, `RESTORE`, `FLUSHALL`, and `FLUSHDB` are replicated rather than refused | `MOVE`, `SWAPDB`, and `RESTORE` are always refused; `FLUSHALL` and `FLUSHDB` are refused while peers are attached | | Peer authentication | None at the module level. The link authenticates as the instance's default account, which the operator binds for you through `spec.activeRedis.redisUserRef` | Mandatory. The instance binds a `RedisUser` through `spec.activeRedis.redisUserRef` — normally its own default account, which must then carry the same password in every datacenter | -| Peer transport | Proxy Redis Serialization Protocol (RESP) port (`6379`) | The same proxy RESP port (`6379`) — one endpoint carries both transports. Active-Active additionally exposes the module peer port (`7379`) | +| Peer transport | Proxy Redis Serialization Protocol (RESP) port (`6379`) | The same proxy RESP port (`6379`) — one endpoint carries both transports. Active-Active publishes one additional Service port onto that same endpoint | | Persistence | RDB recommended | RDB only — `appendonly=yes` is rejected by admission | | Database count | Standard | `databases` must be `<= 16` | @@ -98,14 +100,14 @@ A proxy is deployed for both module generations. Enabling cross-datacenter repli | Port | Name | Purpose | Applies to | |:-----|:-----|:--------|:-----------| | `6379` | `proxy` | RESP endpoint, and the replication transport for both module generations. The proxy tells a peer session apart from a client session and routes it accordingly, so one endpoint carries both. This is the address recorded in mesh seed entries and in an `ActiveRedisConnection`'s `spec.addresses`. | All versions | -| `7379` | `peer` | The module peer port, aliased onto the same proxy listener. It is added to the proxy Service **only in Active-Active mode**, because a mesh member gossips its own local peer port and something must answer on it. | Redis `7.2`, Active-Active only | +| *Published by the Service* | `peer` | An alias onto the same proxy listener, added **only in Active-Active mode**. A mesh member gossips a peer address on a port the module fixes itself, and this entry is what answers there. Read the number from the proxy Service rather than assuming one. | Redis `7.2`, Active-Active only | -The proxy Service of a `peerof` instance carries no `7379` port, and none needs to be opened between datacenters. A Disaster Recovery link addresses the proxy RESP endpoint alone — the address recorded in the downstream's `ActiveRedisConnection`. +The proxy Service of a `peerof` instance publishes only the RESP port, and nothing else needs to be opened between datacenters. A Disaster Recovery link addresses that endpoint alone — the address recorded in the downstream's `ActiveRedisConnection`. - -When an external load balancer or firewall fronts the proxy of an Active-Active member, it must expose port `7379` as `7379`. The module always advertises its own local peer port for that traffic, so remapping it breaks the mesh. Only the RESP port may be remapped, through `announcePort`. + +When an external load balancer or firewall fronts the proxy of an Active-Active member, it must publish every port the proxy Service exposes, each under the same number. The mesh reaches members on a port the module fixes itself and advertises unchanged, so remapping that one breaks the mesh. The RESP port is the exception — it may be remapped, through `announcePort`. ### Instance ID: Service ID diff --git a/docs/en/functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx b/docs/en/functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx index ae18325..ec92509 100644 --- a/docs/en/functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx +++ b/docs/en/functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx @@ -22,7 +22,7 @@ The setup procedure differs between the two module generations. Follow the secti | Redis version | Procedure | Link authentication | |:--------------|:----------|:--------------------| -| `7.2` | [Redis 7.2 — new module](#redis-72--new-module) | The instance's `default` account, bound explicitly | +| `7.2` | [Redis 7.2 — new module](#redis-72--new-module) | A peer-auth `RedisUser`, bound explicitly — a dedicated `custom` account is recommended | | `6.0` | [Redis 6.0 — legacy module](#redis-60--legacy-module) | The instance's `default` account, bound automatically | @@ -33,15 +33,83 @@ Replication between Redis `6.0` and Redis `7.2` is not supported. See [Module ge The new module requires a **peer-auth credential**: every instance in the replication group binds a `RedisUser` through `spec.activeRedis.redisUserRef`. The operator pushes that credential to every node, and both inbound and outbound peer links authenticate with it. Enabling replication without a binding is rejected at admission, and the module rejects all inbound peer replication. -Bind the instance's own **default account**. Every instance already has one — its own reconcile creates it — so nothing has to be provisioned; the binding only names it. + +The module accepts an inbound peer only when the credential it presents matches the local peer-auth credential. Every member of the group therefore has to carry the **same username and the same password value**. Wiring a link between two datacenters whose credentials differ is refused at admission: - -The module accepts an inbound peer only when the credential it presents matches the local peer-auth credential. Binding the default account therefore requires every member of the group to carry the **same `default` password**. +``` +ActiveRedisConnection validation Connection failed with reason +ConnectionFailed(Failed to connect to the upstream: ERR WRONGPASS invalid username-password pair) +``` -This is not automatic: default passwords are generated per instance. Set the same default password on every member before you wire the links. +The bound account may be a `custom` one you create for the purpose, or the instance's own `default` account. A `system` account is always refused. -### Step 1: Confirm the default account in each datacenter \{#step-1-create-the-peer-auth-credential-in-each-datacenter} +### Step 1: Create the peer-auth credential in each datacenter \{#step-1-create-the-peer-auth-credential-in-each-datacenter} + +#### A dedicated account (recommended) + +A `custom` `RedisUser` keeps the replication credential separate from the instance password: binding the default account widens what that password grants — anything holding it can act as a replication peer — and ties credential rotation to instance-password rotation. + +Create the same account in **every** datacenter. Password Secrets are one-to-one with `RedisUser` objects, so each datacenter needs its own Secret object; what has to match across datacenters is the password *value*, not the Secret name. + +The password is policy-checked at admission: 8 to 32 characters, and it must mix letters, digits and special characters. A value that does not qualify is rejected when the `RedisUser` is created, with `password should consists of letters, number and special characters`. + +```bash +$ cat << EOF | kubectl -n default create -f - +apiVersion: v1 +kind: Secret +metadata: + name: dr-peer-auth-dc1 +stringData: + password: +--- +apiVersion: redis.middleware.alauda.io/v1 +kind: RedisUser +metadata: + name: dr-peer-auth-s72-dc1 +spec: + accountType: custom + arch: sentinel + redisName: s72-dc1 + username: dr-peer + aclRules: +@all ~* + passwordSecrets: + - dr-peer-auth-dc1 +EOF +``` + +| Field | Description | +|:------|:------------| +| `accountType` | `custom`. | +| `arch` | The architecture of the referenced instance — `sentinel`, `cluster`, or `standalone`. | +| `redisName` | The local instance this account belongs to. | +| `username` | The peer-auth username. **Identical in every datacenter.** | +| `aclRules` | `+@all ~*` — a peer replays the full command surface over every key. | +| `passwordSecrets` | One Secret, used by no other `RedisUser`, whose `password` key holds the value shared by every datacenter. | + +Wait for the account before binding it — the binding is rejected unless its phase is `Success`: + +```bash +$ kubectl -n default get redisusers dr-peer-auth-s72-dc1 +NAME INSTANCE USERNAME PHASE AGE +dr-peer-auth-s72-dc1 s72-dc1 dr-peer Success 1m +``` + +#### The instance's default account (fallback) + +The instance's own **default account** may be bound instead. It is the account Redis `6.0` links have always used, so an existing group already carries the matching password on every member. + + +Every instance has a default-account `RedisUser`, but it holds a password Secret only when the instance was created with `spec.passwordSecret`. On an instance created without one the account exists and reports `Success` while carrying no credential, and binding it leaves the instance's `ActiveRedis` in `Failed`: + +``` +peer-auth RedisUser "rfr-acl-s72-dc1-default" has no password secret: the module rejects credential-less peers +``` + +Set an instance password first, or bind a dedicated account as above. + + +Binding it requires every member of the group to carry the **same `default` password**. That is not automatic — each instance's password is whatever its own `spec.passwordSecret` holds — and each member needs its own Secret *object*: a Secret shared by several instances is contended by their `RedisUser` controllers and never settles. Set the same password *value* on every member before you wire the links. The default-account `RedisUser` is named after the instance: @@ -50,7 +118,7 @@ The default-account `RedisUser` is named after the instance: | Cluster | `drc-acl--default` | | Sentinel, standalone | `rfr-acl--default` | -Check that it is provisioned in **both** datacenters — the binding is rejected unless its phase is `Success`: +Check that it is provisioned in **both** datacenters: ```bash $ kubectl -n default get redisusers rfr-acl-s72-dc1-default @@ -60,21 +128,32 @@ rfr-acl-s72-dc1-default s72-dc1 default Success 3m The binding itself is set in Step 2, along with the rest of the replication configuration. - -Binding the default account widens what that password grants: anything holding it can act as a replication peer. A `custom` `RedisUser` narrows that, and lets the replication credential be rotated independently of the instance password. It must reference the local instance, carry a password Secret of its own that no other peer-auth `RedisUser` uses, reach `status.phase: Success`, and carry the **same username and the same password value in every datacenter**. A `system` account is always refused. - - ### Step 2: Enable replication on the upstream instance ```bash $ kubectl -n default patch redis s72-dc1 --type=merge \ - --patch='{"spec": {"activeRedis": {"serviceID": 0, "mode": "peerof", "redisUserRef": "rfr-acl-s72-dc1-default"}}}' + --patch='{"spec": {"activeRedis": {"serviceID": 0, "mode": "peerof", "redisUserRef": "dr-peer-auth-s72-dc1"}}}' ``` The range of `serviceID` is `[0-15]` and it must be unique within the replication group. It cannot be changed later, and replication cannot be turned off once enabled. + +The instance performs a **rolling restart** of its data pods when replication is first enabled — the proxy is created and the module is delivered to each pod. The instance leaves `Ready`, and its `ActiveRedis` sits in `Pending` while the pods roll: + +``` +instance s72-dc1 not ready +``` + +Wait for both to settle before continuing. The same applies to the downstream in Step 4, and Step 5 dials the upstream, so it needs the upstream settled too. + +```bash +$ kubectl -n default get redis s72-dc1 +$ kubectl -n default get activeredis +``` + + The instance must also satisfy the new module's constraints, which are enforced by admission: * `customConfig.appendonly` must not be `yes` — the new module is RDB-only. @@ -100,20 +179,22 @@ activeredis-proxy-rfr-s72-dc1 LoadBalancer 10.96.145.32 192.168.1.10 6 ``` -The proxy Service of a `peerof` instance exposes a single port, `6379`. The proxy carries the RESP control plane and the replication stream on it, telling the two apart per session, so the downstream needs to reach only that one address — the one you put in `spec.addresses`. No `7379` port is created in Disaster Recovery mode, and none has to be opened between datacenters. +The proxy Service of a `peerof` instance exposes a single port, `6379`. The proxy carries the RESP control plane and the replication stream on it, telling the two apart per session, so the downstream needs to reach only that one address — the one you put in `spec.addresses`. No additional port is created in Disaster Recovery mode, and none has to be opened between datacenters. Connection admission dials the endpoint it resolves for the replication stream and rejects the connection if it is unreachable. With `spec.peerPort` left unset, that is the address in `spec.addresses[0]` itself. ### Step 4: Enable replication on the downstream instance -Confirm the downstream's default account as in Step 1, and make sure its default password matches the upstream's. Then enable replication with a **different** `serviceID`: +Create the downstream's peer-auth account as in Step 1 — same username, same password value as the upstream's. Then enable replication with a **different** `serviceID`: ```bash $ kubectl -n default patch redis s72-dc2 --type=merge \ - --patch='{"spec": {"activeRedis": {"serviceID": 1, "mode": "peerof", "redisUserRef": "rfr-acl-s72-dc2-default"}}}' + --patch='{"spec": {"activeRedis": {"serviceID": 1, "mode": "peerof", "redisUserRef": "dr-peer-auth-s72-dc2"}}}' ``` +This restarts the downstream's data pods too. Wait for the instance to be `Ready` before Step 5. + ### Step 5: Create the connection on the downstream side The `ActiveRedisConnection` is created in the **downstream** cluster. `spec.instance` names the local (downstream) instance, and `spec.addresses` points at the upstream proxy's RESP endpoint. @@ -158,11 +239,12 @@ NAME INSTANCE STATUS MESSAGE AGE conn-dc2-to-dc1 s72-dc2 Healthy 40s $ kubectl -n default get activeredis -NAME INSTANCE SERVICEID SHARDS PEERS DPEERS PHASE MESSAGE AGE -s72-dc2-activeredis s72-dc2 1 1 1 0 Healthy 3m +NAME INSTANCE SERVICEID SHARDS PEERS DPEERS PHASE MESSAGE AGE +s72-dc1-activeredis s72-dc1 0 1 1 Healthy 16m +s72-dc2-activeredis s72-dc2 1 1 1 Healthy 12m ``` -The upstream instance's `ActiveRedis` resource reports the fan-out through `status.downstreamPeerCount`. +`PEERS` counts upstream links and `DPEERS` downstream ones, so each side prints one and leaves the other blank — a zero count is not rendered. The upstream instance's fan-out is `status.downstreamPeerCount`. Inspect the connection for per-shard synchronization detail: @@ -170,26 +252,27 @@ Inspect the connection for per-shard synchronization detail: $ kubectl -n default get activeredisconnections conn-dc2-to-dc1 -o yaml ... status: - instance: s72-dc2 credentialMode: peer-auth-global + instance: s72-dc2 shards: - index: 0 - offset: "15320" - opId: "412" + offset: "488" + opId: "20480" status: Connected syncStatus: PartialSync + versionState: verified status: Healthy upstreamPeer: service_id: 0 - service_metadata: - instance: default/s72-dc1 ``` -`status.shards[].status` indicates the connection status of the shard, and `status.shards[].syncStatus` indicates the data synchronization status: `PartialSync` (incremental synchronization of Oplog) or `FullSync` (RDB is being synchronized). +`status.shards[].status` indicates the connection status of the shard, and `status.shards[].syncStatus` indicates the data synchronization status: `PartialSync` (incremental synchronization of Oplog) or `FullSync` (RDB is being synchronized). `offset` and `opId` advance as the stream is applied — a stalled pair on a `Connected` shard is the signal to look further. + +`status.shards[].versionState` is the module's own verdict on the upstream's version — `unknown`, `verified`, or `violating`. A shard that is being held for version skew also carries a non-zero `versionHoldSince`. `status.credentialMode: peer-auth-global` records that the module-side peer records hold no inline credential — outbound dials read the node-local peer-auth credential, so a rotation is picked up on the next re-dial. -> The native Redis Sentinel mode does not have the concept of shards. Here, a master-replica pair of Sentinel is abstracted as `shard 0` to be compatible with the same data structure as the Redis cluster mode. +> The native Redis Sentinel mode does not have the concept of shards. Here, a primary-replica pair of Sentinel is abstracted as `shard 0` to be compatible with the same data structure as the Redis cluster mode. ## Redis 6.0 — legacy module \{#redis-60--legacy-module} @@ -210,6 +293,8 @@ You need to create a Redis instance first. $ kubectl -n default patch redis s6 --type=merge --patch='{"spec": {"activeRedis":{"serviceID":0 }}}' ``` + The command returns an admission warning recommending Redis `7.2`; the patch is applied. The instance then performs a **rolling restart** of its data pods to bring up the proxy, so wait for it to be `Ready` again before going on. + {/* lint ignore no-duplicate-headings-in-section */} @@ -258,11 +343,19 @@ Each disaster recovery instance will create a Proxy Service with a name that fol The range of `serviceID` is `[0-15]`. In the same disaster recovery cluster, the serviceID cannot be repeated. -#### Wait for the peer-auth binding +#### Wait for the instance and the peer-auth binding + + This side rolls its data pods too. The connection created next is rejected while the instance is still rolling — the `RedisUser` it authenticates as has to report `Success`, and it does not while its nodes are coming back: - The operator adds the binding on a following reconcile, and the connection is rejected while it is still empty. Wait until this prints a name: + ``` + peer-auth binding not usable: peer-auth RedisUser not ready: RedisUser + "rfr-acl-s6-dest-default" phase is "Pending", want Success + ``` + + The operator adds the binding on a following reconcile, and the connection is rejected while it is still empty. Wait until the instance is `Ready` and this prints a name: ```bash + $ kubectl -n default get redis s6-dest $ kubectl -n default get redis s6-dest -o jsonpath='{.spec.activeRedis.redisUserRef}' rfr-acl-s6-dest-default ``` @@ -285,11 +378,33 @@ EOF > Note to replace the upstream address. - A Redis `6.0` link authenticates as the instance's **default account**. The operator writes that down for you: the first time it reconciles an instance that has replication enabled and carries no binding, it sets `spec.activeRedis.redisUserRef` to the instance's own default-account `RedisUser` — `drc-acl--default` for a cluster instance, `rfr-acl--default` for a Sentinel one. Nothing is provisioned: that account already exists, and the password in it is the one the link uses. Both datacenters must therefore carry the same `default` password. + A Redis `6.0` link authenticates as the instance's **default account**. The operator writes that down for you: the first time it reconciles an instance that has replication enabled and carries no binding, it sets `spec.activeRedis.redisUserRef` to the instance's own default-account `RedisUser` — `drc-acl--default` for a cluster instance, `rfr-acl--default` for a Sentinel one. Nothing is provisioned: that account already exists, and the password in it is the one the link uses. Both datacenters must therefore carry the same `default` password — the same *value*, each in its own Secret object. There is no credential field on the connection. `spec.secretName` has been removed, and a manifest that still sets it is rejected with `unknown field "spec.secretName"`. + + Until v5.1, Disaster Recovery on Redis `6.0` was the only cross-datacenter replication this product offered, and the link credential lived on the connection: `ActiveRedisConnection.spec.secretName` named a Secret holding a `password` and nothing else. + + There was never a username in it. The legacy module sends a one-argument `AUTH ` when no username is set, and the proxy resolves that to the instance's **default account** — so a Redis `6.0` link has always authenticated as `default`, whatever the Secret was called. Both datacenters have therefore always had to carry the same `default` password. + + v5.1 removes `spec.secretName` from `ActiveRedisConnection` and `ActiveRedisInspection`, and takes the credential from `spec.activeRedis.redisUserRef` alone. **Existing groups need no intervention:** on its first reconcile after the upgrade the operator writes the binding on each replicating instance, pointing it at that instance's own default-account `RedisUser`. Nothing changes on the wire — same account, same password, read from somewhere else — and running links stay up across the upgrade. + + What does change is your manifests. A stored `ActiveRedisConnection` keeps working, but a manifest that still sets `secretName` is rejected on the next apply, and the field is gone from `kubectl explain`. Drop it, and manage the credential through the instance's binding instead. + + + + A `custom` `RedisUser` can be bound on Redis `6.0` as well — create it as shown in [Step 1 of the Redis 7.2 procedure](#step-1-create-the-peer-auth-credential-in-each-datacenter) and set `spec.activeRedis.redisUserRef` to it in the same patch that enables replication. A binding that is already set is never replaced by the automatic one, and the link then authenticates as that account rather than as `default`. + + What you do **not** get is rotation safety. A Redis `6.0` link carries its credential inside the module's peer record, frozen there when the link is wired, and the legacy path has no self-healing re-wire. Changing the password leaves the running link working until its session next drops — and then it stays down: + + ``` + Failed shard 0 status is Disconnected + ``` + + Recovering means deleting the `ActiveRedisConnection` and creating it again, which re-wires the peer with the current credential. Plan a rotation on Redis `6.0` as a short outage of the link, or upgrade to Redis `7.2`, where the credential is read per dial instead. + + #### Check Disaster Recovery Connection Status ```bash @@ -332,7 +447,7 @@ status: Here `status.shards[0].status` indicates the connection status of the shard, and `status.shards[0].syncStatus` indicates the data synchronization status. The synchronization status can be `PartialSync` (indicating incremental synchronization of Oplog) or `FullSync` (indicating that RDB is being synchronized). - > The native Redis Sentinel mode does not have the concept of shards. Here, a master-slave of Sentinel is abstracted as `shard 0` to be compatible with the same data structure as the Redis cluster mode. + > The native Redis Sentinel mode does not have the concept of shards. Here, a primary-replica pair of Sentinel is abstracted as `shard 0` to be compatible with the same data structure as the Redis cluster mode. @@ -381,11 +496,27 @@ Deleting an `ActiveRedisConnection` tears the link down according to its `spec.t | `Decommission` | Permanently decommissions the peer. In addition to removing the link, the module stops retaining Oplog, dead-key tombstones, and lag accounting for that peer. | No | -`Decommission` is the correct teardown for a datacenter that is gone for good — a detached-but-never-returning peer keeps holding garbage-collection floors on the survivors. It cannot be changed back to `Detach`, and a decommissioned peer that later returns is treated as a brand-new peer requiring a fresh full synchronization. +`Decommission` is the correct teardown for a datacenter that is gone for good — a detached-but-never-returning peer keeps holding garbage-collection floors on the survivors. It cannot be changed back to `Detach`: + +``` +spec.teardownPolicy is one-way: cannot change "Decommission" back to "Detach" +``` + +The decision is recorded against the peer's `serviceID` and survives a restart. What it changes is what **this** instance keeps for that peer: its Oplog retention, tombstone garbage-collection and lag accounting for that `serviceID` are released and never re-armed. Nothing stops a connection to that peer from being created again, and one created straight away still resumes from whatever the two ends happen to hold — so re-creating the link is not a way to undo the decommission, and a peer that is meant to come back should be detached, not decommissioned. ## Credential rotation -On Redis `7.2`, peer links carry no inline credential: outbound dials read the node-local peer-auth credential that the operator re-pushes on every reconcile. To rotate, update the password Secret of each datacenter's peer-auth `RedisUser` in place, to the same new value, one datacenter at a time. The `RedisUser` controller replays the ACL, the operator re-pushes the credential, and the proxy accepts the new credential on the next rebuild. Existing links stay healthy, and a link that drops after the rotation re-dials with the current credential. +On Redis `7.2`, peer links carry no inline credential: outbound dials read the node-local peer-auth credential that the operator re-pushes on every reconcile. To rotate, update the password Secret of each datacenter's peer-auth `RedisUser` in place, to the same new value, one datacenter at a time. The `RedisUser` controller replays the ACL, the operator re-pushes the credential, and the proxy accepts the new credential on the next rebuild. Established links stay healthy and keep replicating throughout — including the window in which the datacenters hold different values — and a link that drops after the rotation re-dials with the current credential. + +That window is not a free-for-all, though: while the values differ, *creating* a link is refused, because admission dials the upstream with the credential the data path will use. Finish the rotation on every member before wiring anything new. -Rotating an instance's own password Secret is independent and does not affect peer links. + +When a dedicated `custom` account is bound, rotating an instance's own password Secret is independent and does not affect peer links. + +When the instance's `default` account is bound, its password Secret **is** the peer-auth Secret: rotating the instance password rotates the replication credential with it, and the rest of the group has to follow. That is the coupling a dedicated account avoids. + + + +None of the above applies to Redis `6.0`. Its links hold the credential inline in the module's peer record, fixed when the link was wired, and there is no self-healing re-wire on that path — so a rotated link keeps running only until its session next drops, then reports `shard 0 status is Disconnected` and stays there. Delete and re-create the `ActiveRedisConnection` to re-wire it with the current credential. + diff --git a/docs/en/functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx b/docs/en/functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx index 3f768cb..6e3f418 100644 --- a/docs/en/functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx +++ b/docs/en/functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx @@ -50,7 +50,7 @@ Before starting, confirm each of the following for **every** member of the group $ kubectl -n default get redis s6-dc1 -o jsonpath='{.spec.customConfig.databases}' ``` -5. **The upstream proxy endpoint stays reachable.** The new module replicates over the same proxy endpoint the Redis `6.0` links already used, so a Disaster Recovery group needs no new port opened. Leave the existing firewall rules and load-balancer entries for the proxy's RESP port in place, so the links can be re-created at the end. +5. **The upstream proxy endpoint stays reachable.** The new module replicates over the same proxy endpoint the Redis `6.0` links already used, so a Disaster Recovery group needs no new port opened. The proxy Service is not recreated by the upgrade — it keeps its address, its port and, under NodePort, its node port — so leave the existing firewall rules and load-balancer entries for the proxy's RESP port in place, and the links can be re-created at the end against the same addresses you recorded in step 6. 6. **Record the current topology.** Note each member's `serviceID`, and which member holds the `ActiveRedisConnection` pointing at which upstream. You will re-create these links. ```bash @@ -104,6 +104,18 @@ Because the links were deleted in Step 2, there is no live replication to order The ordering rule matters for later, within-generation upgrades of a **live** group: the safe order is **downstream before upstream**, because a downstream can apply an equal-or-older upstream's Oplog but never a newer one. Bumping an upstream ahead of a live downstream returns an admission warning, and the module holds the affected edge — replication to that downstream pauses until the version skew is resolved. + +The same guard covers this upgrade. A connection created while the upstream is already on Redis `7.2` and the downstream is still on `6.0` is accepted, then held — it does not replicate, and it does not damage either side: + +```bash +$ kubectl -n default get activeredisconnections +NAME INSTANCE STATUS MESSAGE AGE +conn-dc2-to-dc1 s6-dc2 Paused upstream version 7.2 newer than local version 6.0; waiting for local upgrade (safe order upgrades the downstream first) 18s +``` + +It stays `Paused` until the local member is upgraded. Following the procedure — links deleted in Step 2, re-created in Step 5 — you will not see this; it is what a mistimed link looks like if you do. + + ### Step 4: Confirm the peer-auth binding The new module requires a peer-auth credential on every instance. On an instance that already had cross-datacenter replication enabled, the platform binds one automatically — see [Automatic peer-auth binding](#automatic-peer-auth-provisioning). @@ -122,7 +134,7 @@ rfr-acl-s6-dc1-default s6-dc1 default Success 8d If the binding is empty, or the `RedisUser` phase is not `Success`, resolve it as described in [If the binding is not there yet](#when-automatic-provisioning-is-on-hold) before continuing. -The module accepts an inbound peer only when the username it presents matches the local peer-auth username. The automatic binding uses each instance's `default` account, so members that were already replicating to each other converge on the same credential — that shared `default` password is exactly what their Redis `6.0` links were authenticating with. +The module accepts an inbound peer only when the credential it presents — username *and* password — matches the local peer-auth credential. The automatic binding uses each instance's `default` account, so members that were already replicating to each other converge on the same credential: that shared `default` password is exactly what their Redis `6.0` links were authenticating with. If you bound credentials manually on any member, verify that **every** member carries the same username and the same password value before creating any link. @@ -160,8 +172,9 @@ Confirm that: * the connection status is `Healthy`; * `status.credentialMode` is `peer-auth-global`; * `status.shards[].status` is `Connected` and the offsets advance; +* `status.shards[].syncStatus` is `PartialSync` — the new module picks the link up from the offset the Redis `6.0` link left off at, so a re-created link resumes rather than re-sending the dataset. A `FullSync` here means the retained state was not usable and the whole dataset is being re-sent; it will still converge, but it costs a full transfer; * the upstream's `ActiveRedis` reports the expected `status.downstreamPeerCount`; -* a key written on the upstream appears on each downstream. +* a key written on the upstream appears on each downstream, alongside the keys that were there before the upgrade. Then restore normal client traffic. @@ -212,5 +225,5 @@ Once the group is running on Redis `7.2`, the following become available: ## Known gaps in this draft * The procedure requires a full-group maintenance window. There is no supported way to upgrade a group while keeping replication live. -* Downgrading from Redis `7.2` back to Redis `6.0` is not supported. If the upgrade must be abandoned, restore the members from the backups taken in the prerequisites. +* Downgrading from Redis `7.2` back to Redis `6.0` is not supported — and **nothing prevents it**. Setting `spec.version` back to `6.0` is accepted by admission and starts rolling the pods onto the older image immediately. If the upgrade must be abandoned, restore the members from the backups taken in the prerequisites rather than reversing the version. * The Web Console does not provide a guided flow for this upgrade; the steps above are CLI-driven. diff --git a/docs/en/functions/95-disaster-recovery/30-active-active/10-architecture.mdx b/docs/en/functions/95-disaster-recovery/30-active-active/10-architecture.mdx index 3a384f5..ea8829c 100644 --- a/docs/en/functions/95-disaster-recovery/30-active-active/10-architecture.mdx +++ b/docs/en/functions/95-disaster-recovery/30-active-active/10-architecture.mdx @@ -41,13 +41,13 @@ The practical differences this produces: * **Adding a member does not require touching the existing ones.** A new member joins by naming any reachable existing member as a seed, and the whole group learns about it through gossip. Updating the other members' seed lists afterwards is still recommended, so that a member restarting while the original seeds are unreachable can still rejoin. * **Seeds are a bootstrap list, not a membership list.** One reachable seed per remote datacenter is enough, and the local member's own entry may be included — which lets every datacenter carry one identical seed list. -* **Membership is state that converges**, so it is reported rather than declared. The `ActiveRedisMesh` status carries the member list, each member's state, and an `epoch` that advances as membership changes. +* **Membership is state that converges**, so it is reported rather than declared. The `ActiveRedisMesh` status carries the member list and each member's state. It also carries an `epoch`, which is not a change counter: it advances only when a member is forgotten or permanently decommissioned, so a group that has only ever gained members reports `0`. ## What travels on which port A member is identified in the mesh by its **announced RESP address** — `announceAddress` with `announcePort`, normally `6379`. That is the address written in seed entries and carried in the membership records that gossip exchanges, and it must be routable from every other datacenter. -The traffic itself — both the gossip rounds and the replicated Oplog — is carried on the **peer port**, `7379`. The module always advertises its own local peer port for that traffic, which is why an external load balancer or firewall in front of the proxy must expose `7379` as `7379`. Only the RESP port may be remapped. +The traffic itself — both the gossip rounds and the replicated Oplog — is carried on the **peer port**: a second number the proxy Service publishes onto that same listener, which you can read off the Service. The module fixes that number itself and advertises it unchanged, which is why an external load balancer or firewall in front of the proxy must publish it under the same number. Only the RESP port may be remapped. As in Disaster Recovery mode, peers address the proxy of an instance rather than an individual pod, and the proxy resolves each connection to the primary that is current at that moment. diff --git a/docs/en/functions/95-disaster-recovery/30-active-active/20-setup.mdx b/docs/en/functions/95-disaster-recovery/30-active-active/20-setup.mdx index 6c092d9..ae9e638 100644 --- a/docs/en/functions/95-disaster-recovery/30-active-active/20-setup.mdx +++ b/docs/en/functions/95-disaster-recovery/30-active-active/20-setup.mdx @@ -18,12 +18,84 @@ Setting up Active-Active replication has two parts: enabling Active-Active mode * Every member runs Redis `7.2`. * Each member has a distinct `serviceID` in the range `[0-15]`. -* Each member's proxy is reachable from every other member on both port `6379` (RESP, used by seeds) and port `7379` (peer port, which carries both gossip and replication). +* Each member has an externally-routable `announceAddress` set explicitly — see [Step 2](#step-2-enable-active-active-mode). Nothing defaults it to a cross-datacenter address, and admission only warns, so a member left without one advertises an address its peers cannot reach. +* Each member's proxy endpoint is reachable from every other member on **every port its proxy Service publishes**, each under its own number — not only the announced one. * **The system clocks of all members are synchronized**, through a time-synchronization daemon (`ntpd`, `chronyd`, or the platform equivalent) on every node of every cluster in the group. This is a hard prerequisite, not a recommendation — see [Clock synchronization](./30-operations.mdx#clock-synchronization-requirement). -## Step 1: Confirm the default account in each datacenter \{#step-1-create-the-peer-auth-credential-in-each-datacenter} +## Step 1: Provision the peer-auth credential in each datacenter \{#step-1-create-the-peer-auth-credential-in-each-datacenter} -Every peer link in an Active-Active group authenticates, and every member must present the same credential. Bind the instance's own **default account**: every instance already has one, created by its own reconcile, so nothing has to be provisioned. This is the same credential Disaster Recovery uses — if these instances already carry a binding, skip to Step 2. +Every peer link in an Active-Active group authenticates, and every member must present the same credential: a member accepts an inbound peer only when the username it presents equals the member's own peer-auth username and the password validates against Redis ACL. Each instance binds that credential through `spec.activeRedis.redisUserRef`, which names a `RedisUser` **object** in the instance's own namespace. + +Whichever account you bind, it must: + +* reference the **local** instance — a `RedisUser` that targets another instance is refused; +* be a `custom` or a `default` account; a `system` account is always refused; +* own a password Secret carrying a `password` key; +* reach `status.phase: Success` — the binding is rejected until then; +* carry the **same username and the same password value in every datacenter**. + +### A dedicated account (recommended) + +Create a `custom` `RedisUser` for this purpose in **every** datacenter. It narrows what the credential grants — anything holding the default password can otherwise act as a mesh peer — and lets the replication credential be rotated independently of the instance password. + +Password Secrets are one-to-one with `RedisUser` objects, so each datacenter needs its own Secret object; what has to match across datacenters is the password *value*, not the Secret name. + +The password is policy-checked at admission: 8 to 32 characters, and it must mix letters, digits and special characters. A value that does not qualify is rejected when the `RedisUser` is created, with `password should consists of letters, number and special characters`. + +```bash +$ cat << EOF | kubectl -n default create -f - +apiVersion: v1 +kind: Secret +metadata: + name: aa-peer-auth-dc1 +stringData: + password: +--- +apiVersion: redis.middleware.alauda.io/v1 +kind: RedisUser +metadata: + name: aa-peer-auth-s72-dc1 +spec: + accountType: custom + arch: sentinel + redisName: s72-dc1 + username: aa-peer + aclRules: +@all ~* + passwordSecrets: + - aa-peer-auth-dc1 +EOF +``` + +| Field | Description | +|:------|:------------| +| `accountType` | `custom`. | +| `arch` | The architecture of the referenced instance — `sentinel`, `cluster`, or `standalone`. | +| `redisName` | The local instance this account belongs to. | +| `username` | The peer-auth username. **Identical in every datacenter.** | +| `aclRules` | `+@all ~*` — a peer replays the full command surface over every key. | +| `passwordSecrets` | One Secret, used by no other `RedisUser`, whose `password` key holds the value shared by every datacenter. | + +Wait for the account before binding it: + +```bash +$ kubectl -n default get redisusers aa-peer-auth-s72-dc1 +NAME INSTANCE USERNAME PHASE AGE +aa-peer-auth-s72-dc1 s72-dc1 aa-peer Success 1m +``` + +### The instance's default account (fallback) + +The instance's own **default account** may be bound instead — this is the same credential Disaster Recovery uses, so if these instances already carry a binding, you can skip to Step 2. + + +Every instance has a default-account `RedisUser`, but it holds a password Secret only when the instance was created with `spec.passwordSecret`. On an instance created without one the account exists and reports `Success` while carrying no credential, and binding it leaves the instance's `ActiveRedis` in `Failed`: + +``` +peer-auth RedisUser "rfr-acl-s72-dc1-default" has no password secret: the module rejects credential-less peers +``` + +Set an instance password first, or bind a dedicated account as above. + The default-account `RedisUser` is named after the instance: @@ -32,28 +104,26 @@ The default-account `RedisUser` is named after the instance: | Cluster | `drc-acl--default` | | Sentinel, standalone | `rfr-acl--default` | -Check that it is provisioned in each datacenter — the binding is rejected unless its phase is `Success`: - ```bash $ kubectl -n default get redisusers rfr-acl-s72-dc1-default NAME INSTANCE USERNAME PHASE AGE rfr-acl-s72-dc1-default s72-dc1 default Success 3m ``` - -The module accepts an inbound peer only when the credential it presents matches the local peer-auth credential. Binding the default account therefore requires every member of the mesh to carry the **same `default` password**, and default passwords are generated per instance — set them to a common value before enabling the mode. - - - -A `custom` `RedisUser` narrows what the credential grants — anything holding the default password can otherwise act as a mesh peer — and lets the replication credential be rotated independently of the instance password. It must reference the local instance, own its password Secret, reach `status.phase: Success`, and carry the **same username and the same password value in every datacenter**. A `system` account is always refused. + +Every member must then carry the **same `default` password**, and default passwords are generated per instance — set them to a common value before enabling the mode. Anything holding that password can act as a mesh peer, and rotating the instance password rotates the replication credential with it. A dedicated account avoids both. -## Step 2: Enable Active-Active mode on each instance +## Step 2: Enable Active-Active mode on each instance \{#step-2-enable-active-active-mode} These steps assume instances that are being placed into an Active-Active group from the start. Admission does not block changing `spec.activeRedis.mode` from `peerof` to `mesh` on an instance that is already replicating, but converting an established Disaster Recovery group to Active-Active is not a covered procedure and has not been validated. Build an Active-Active group from instances that are not already wired into a Disaster Recovery group. + +Turning on Active-Active adds the module's init container to the instance, so the operator rolls the Redis pods and the instance passes back through `Initializing` before it is `Ready` again. Complete Step 1 first: while an instance is not `Ready`, a `custom` `RedisUser` cannot be created against it — admission refuses with `redis failover is not ready` — so enabling the mode before the account exists leaves you waiting out the restart to create it. + + Members must advertise an externally-routable address, because a peer re-resolves it on every dial. This is the `announceAddress`, set on the proxy Service configuration. ```bash @@ -62,7 +132,7 @@ spec: activeRedis: serviceID: 0 mode: mesh - redisUserRef: rfr-acl-s72-dc1-default + redisUserRef: aa-peer-auth-s72-dc1 proxy: service: type: LoadBalancer @@ -75,24 +145,41 @@ spec: |:------|:------------| | `mode` | `mesh`. | | `serviceID` | Unique within the mesh, range `[0-15]`, immutable. | -| `redisUserRef` | The peer-auth `RedisUser` binding — normally the instance's default account. Mandatory. | +| `redisUserRef` | The peer-auth `RedisUser` binding from Step 1. Mandatory. | | `proxy.service.announceAddress` | The externally-reachable host peers dial to reach this member — the host on the external load balancer or VIP, decoupled from the in-cluster Service. Must be a **bare host**: no scheme, no port, no path. | -| `proxy.service.announcePort` | The externally-reachable RESP port for the gossip control plane. Defaults to the proxy Service port `6379`; override it only when an external load balancer exposes that port under a different number. | +| `proxy.service.announcePort` | The externally-reachable RESP port of this member's proxy endpoint. Together with `announceAddress` it forms the pair peers put in their seed lists for this member, and the pair the member matches to recognize its own seed entry. Defaults to the proxy Service port `6379`; override it only when an external load balancer exposes that port under a different number. | + + +`ActiveRedis` exposes the same `spec.proxy.service` fields, but that resource is derived: the operator rewrites its `spec.proxy` from the `Redis` instance on every reconcile — in mesh mode writing the *resolved* announce address into it — so an edit made there is overwritten. Always patch the `Redis` object. + * A **DNS name is recommended**: the module re-resolves it on every dial, so it survives IP changes, and a multi-A record can cover several NodePort addresses. When the announce address is a DNS name, the operator binds it to the proxy Service through ExternalDNS. * A **bare IP is accepted with a warning** — it will not re-resolve after an IP change, so it is only appropriate for a stable VIP. -* Leaving it **empty produces a warning** and the module advertises the pod IP, which is not routable across datacenters. -* An address containing a scheme (`tcp://…`), a port (`host:7379`), or a path is **rejected**. The port belongs in `announcePort`. +* Leaving it **empty produces a warning**, and what happens then depends on the operator: if it is configured with a managed DNS zone, it derives `..` and publishes that on the proxy Service through ExternalDNS; with no zone configured it sends no announce address at all and the module advertises the **pod IP**, which is not routable across datacenters. Nothing else fills the field in. +* An address containing a scheme (`tcp://…`), a port (`host:6379`), or a path is **rejected**. The port belongs in `announcePort`. - -`announcePort` applies to the gossip control plane only. The module always advertises its own local peer port (`7379`) for replication, so any external endpoint must expose `7379` as `7379`. + +`announcePort` is an identity value: it is how this member names its endpoint to its peers, and it carries no traffic of its own. Neither gossip nor replication is steered by it, and it does not override how the module addresses a peer. + +The proxy runs a **single listener** — application traffic, gossip, and replication all arrive on it and are told apart per session — but the proxy Service publishes more than one port onto it. When an external load balancer or firewall fronts the proxy, forward **every port that Service publishes**, each under its own number. Remapping any of them to a different number breaks the mesh. ## Step 3: Create an ActiveRedisMesh in each cluster -Each Kubernetes cluster owns one `ActiveRedisMesh` for its local instance. `spec.seeds` names the externally-routable proxy RESP addresses of members in other datacenters. +Each Kubernetes cluster owns one `ActiveRedisMesh` for its local instance. `spec.seeds` names the announced endpoints of members in other datacenters — the `announceAddress` and `announcePort` those members were given in Step 2. + + +Step 2 rolls the pods, and the module arrives with them. Create the `ActiveRedisMesh` before the roll reaches the shard's primary node, and the mesh lands in `Failed`, reporting that the module's own commands do not exist: + +``` +set mesh config on pod rfr-s72-dc1-0: ERR Unknown option or number of arguments for CONFIG SET +- 'activeredis.mesh-seeds'; shard 0 mesh info: ERR unknown command 'as.mesh' +``` + +This clears by itself — the mesh moves to `Pending` once the primary node carries the module, then to `Healthy` — but it is indistinguishable from a real failure while it lasts. Wait for `kubectl get redis` to report `Ready`. + ```bash $ cat << EOF | kubectl -n default create -f - @@ -110,7 +197,9 @@ EOF ### Seed syntax -A seed is `host:port`, where the port is the remote proxy's RESP endpoint — normally `6379`. An optional `@peer-port` suffix overrides the peer port when an external load balancer remaps it; when the suffix is omitted the operator appends the conventional `7379`. The operator normalizes the list before injecting it, so `redis-dc2.example.com:6379` becomes `redis-dc2.example.com:6379@7379`. +A seed is `host:port` — the remote member's **`announceAddress:announcePort`**, written exactly as that member advertises itself. It is an externally-routable endpoint, so when a load balancer or VIP fronts that member's proxy, the seed is the load balancer's address and port, not the in-cluster proxy Service. Write the pair verbatim: a member recognizes its own entry by comparing a seed's address and port with the pair it announces, and a mismatch defeats that self-skip. + +Write seeds as plain `host:port` and let the operator complete them. It normalizes the list into the module's internal seed grammar before injecting it, so `CONFIG GET activeredis.mesh-seeds` reports each entry in a longer form than the one you wrote. That completion is required for a seed to be usable at all, and there is nothing in it for you to set. One reachable seed per remote datacenter is enough — gossip discovers everything else. The local member's own entry may be included, so every datacenter can carry the same uniform seed list; the module skips its own record. @@ -141,41 +230,45 @@ $ kubectl -n default get activeredismesh mesh-dc1 -o yaml status: phase: Healthy instance: s72-dc1 - epoch: 7 memberCount: 3 aliveCount: 3 - suspectCount: 0 - deadCount: 0 fleetFloorVersion: "7.2" versionConverged: true members: - uid: 9f2c1a... serviceID: 0 shardID: 0 - address: redis-dc1.example.com state: alive - epoch: 7 - clockOffsetMs: 2 version: "7.2" - moduleVersion: 10 - uid: 4b81d3... serviceID: 1 shardID: 0 - address: redis-dc2.example.com + address: redis-dc2.example.com:6379 state: alive - epoch: 7 clockOffsetMs: -5 version: "7.2" moduleVersion: 10 + - uid: 07d65b... + serviceID: 2 + shardID: 0 + address: redis-dc3.example.com:6379 + state: alive + clockOffsetMs: -2 + version: "7.2" + moduleVersion: 10 ``` + +The first entry above is this instance's own, and it carries no `address`, `clockOffsetMs` or `moduleVersion`. That is expected, not a missing member: gossip reports the *other* members, so the operator fills the local record in from what the instance knows about itself, and those three fields have no local source. Every remote member carries them. Counts that are zero — `suspectCount`, `deadCount` — are omitted rather than printed. + + ### Mesh phases | Phase | Meaning | |:------|:--------| | `Healthy` | Membership is converged and every known member is alive. | | `Degraded` | The mesh is operating with suspect or dead members. | -| `Pending` | No ready master could be reached to read membership yet — the bootstrap window, or an outage. | +| `Pending` | No ready primary node could be reached to read membership yet — the bootstrap window, or an outage. | | `Paused` | `spec.pause` is `true`. | | `Failed` | The mesh could not be configured; `status.message` carries the reason (for example `invalid mesh seed`). | diff --git a/docs/en/functions/95-disaster-recovery/30-active-active/30-operations.mdx b/docs/en/functions/95-disaster-recovery/30-active-active/30-operations.mdx index 4cd37b4..aa373b1 100644 --- a/docs/en/functions/95-disaster-recovery/30-active-active/30-operations.mdx +++ b/docs/en/functions/95-disaster-recovery/30-active-active/30-operations.mdx @@ -49,12 +49,14 @@ Start from the resource, which reports the offset to every member the local inst ```bash $ kubectl -n default get activeredismesh mesh-dc1 \ -o jsonpath='{range .status.members[*]}{.address}{"\t"}{.state}{"\t"}{.clockOffsetMs}{"\n"}{end}' -redis-dc1.example.com alive 0 -redis-dc2.example.com alive -5 -redis-dc3.example.com alive 4180 + alive +redis-dc2.example.com:6379 alive -5 +redis-dc3.example.com:6379 alive 4180 ``` -For the aggregated gauges and the attribution flag, read them on the instance itself: +The first row is the local member. It carries no address and no offset — it is the reference the other offsets are measured against, so only the remote rows are readable here. + +For the aggregated gauges and the attribution flag, read them on the instance itself. Run this **against the shard's primary node**: only the primary takes part in gossip, so the mesh counters on a replica describe whatever reached it passively and are not a view of the mesh. ```bash $ redis-cli INFO activeredis | grep -E \ @@ -70,7 +72,7 @@ $ redis-cli INFO activeredis | grep -E \ | `clock_skew_critical_violations` | How many critical episodes have occurred. | | `local_clock_is_outlier` | `1` when **this** node's clock is the one that disagrees with all the others. | -Per-member offsets are also available directly from the module, as `clock_offset_ms` on each line: +Per-member offsets are also available directly from the module, as `clock_offset_ms` on each line. Run it against the shard's primary node, for the reason above: ```bash $ redis-cli as.mesh nodes diff --git a/docs/en/functions/95-disaster-recovery/90-limitations.mdx b/docs/en/functions/95-disaster-recovery/90-limitations.mdx index a6e6ca8..a217296 100644 --- a/docs/en/functions/95-disaster-recovery/90-limitations.mdx +++ b/docs/en/functions/95-disaster-recovery/90-limitations.mdx @@ -103,7 +103,7 @@ Run a time-synchronization daemon (`ntpd`, `chronyd`, or the platform equivalent The proxy address itself is not highly available. If you expose the proxy via NodePort, place it behind a front-end load balancer to ensure HA; alternatively, use MetalLB to provide a highly available virtual IP for the proxy. -In Active-Active mode on Redis `7.2` the address must expose **both** the RESP port (`6379`) and the peer port (`7379`), and the peer port must not be remapped — the module always advertises its own local peer port for gossip. A Disaster Recovery link uses the RESP endpoint alone. +In Active-Active mode on Redis `7.2` the address must expose **every port the proxy Service publishes**, each under its own number. The mesh reaches members on a port the module fixes itself and advertises unchanged, so remapping it breaks gossip. A Disaster Recovery link uses the RESP endpoint alone. ### Only IP connection addresses are supported (Redis 6.0) diff --git a/docs/en/release_notes.mdx b/docs/en/release_notes.mdx index 32641ee..ab17e5f 100644 --- a/docs/en/release_notes.mdx +++ b/docs/en/release_notes.mdx @@ -30,7 +30,7 @@ Redis `7.2` instances now run a redesigned replication module, offering two mode * **Disaster Recovery** (`peerof`, the default) is the hot-standby topology, with directed links from one upstream to up to 15 downstream instances, each ready to be promoted if the upstream datacenter is lost. It remains available on Redis `6.0`. * **Active-Active** (`mesh`) lets every datacenter serve reads and writes locally, with conflicts resolved automatically and membership maintained by a gossip mesh seeded from a handful of addresses. It is available on Redis `7.2` only, and is driven by the new `ActiveRedisMesh` resource. -Replication and gossip on Redis `7.2` ride the instance proxy, which tells the two transports apart on a single endpoint. Active-Active additionally exposes the module peer port (`7379`) on the proxy Service, so that mesh members can gossip. +Replication and gossip on Redis `7.2` ride the instance proxy, which tells the two transports apart on a single endpoint. Active-Active publishes one additional port on the proxy Service, aliased onto that same listener, so that mesh members can gossip. See [Cross-Datacenter Replication](./functions/95-disaster-recovery/10-intro.mdx). @@ -42,10 +42,18 @@ The credential is delivered to the nodes at runtime and is never written into th #### Upgrade Path from Redis 6.0 to Redis 7.2 for Replicating Instances -An instance with cross-datacenter replication enabled can now be upgraded in place from Redis `6.0` to Redis `7.2`. When such an instance carries no peer-auth binding, the platform provisions one automatically, deriving it from the credential the replication group already shares. +An instance with cross-datacenter replication enabled can now be upgraded in place from Redis `6.0` to Redis `7.2`. When such an instance carries no peer-auth binding, the platform writes one for it, naming the instance's own default-account `RedisUser` — the account a Redis `6.0` link has always authenticated as. Nothing is provisioned, and the credential does not change during the upgrade. The two module generations do not interoperate, so the whole group must be upgraded together within a maintenance window. See [Upgrade a Disaster Recovery Group from Redis 6.0 to Redis 7.2](./functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx). +#### The Link Credential Moves to the Instance + +Until this release, a Redis `6.0` replication link took its credential from the connection: `ActiveRedisConnection.spec.secretName` named a Secret holding a password. That Secret carried no username, so the link authenticated as the instance's default account regardless — which is now where the credential is declared. + +`spec.secretName` is **removed** from `ActiveRedisConnection` and `ActiveRedisInspection`. Both module generations take the link credential from `spec.activeRedis.redisUserRef` and nothing else. + +Existing Disaster Recovery groups are migrated without intervention: on its first reconcile after the upgrade the operator binds each replicating instance to its own default-account `RedisUser`, which changes nothing on the wire, and running links stay up. Stored resources keep working, but a manifest that still sets `secretName` is rejected on its next apply. + #### Teardown Policy for Replication Links `ActiveRedisConnection` now carries a `teardownPolicy`. `Detach` (the default) removes the link but keeps the peer's bookkeeping so the link can be re-created later. `Decommission` permanently retires a peer and releases the retention floors held for it — the correct teardown for a datacenter that is gone for good. diff --git a/docs/en/upgrade.mdx b/docs/en/upgrade.mdx index 96c85f8..80d19ae 100644 --- a/docs/en/upgrade.mdx +++ b/docs/en/upgrade.mdx @@ -64,6 +64,14 @@ Confirm that no instance reports `5.0` in the `VERSION` column (for example, run If an instance has cross-datacenter replication enabled, its Redis version cannot be changed in isolation. Follow [Upgrade a Disaster Recovery Group from Redis 6.0 to Redis 7.2](./functions/95-disaster-recovery/20-disaster-recovery/40-upgrade-6.0-to-7.2.mdx), which coordinates the whole replication group. + +Upgrading the operator to v5.1 does not interrupt an existing Disaster Recovery group. Replication continues across the upgrade and the data on both sides is unaffected — the instances stay on their current Redis version, so they keep running the module generation they were already using. + +One thing does move. In v5.1 the link credential is declared on the instance through `spec.activeRedis.redisUserRef` instead of on the connection through `spec.secretName`, and the operator writes that binding for each replicating instance on its first reconcile after the upgrade. It names the instance's own default account — the account a Redis `6.0` link has always authenticated as — so nothing changes on the wire and no action is required. + +Your manifests do need a change: `spec.secretName` is removed from `ActiveRedisConnection` and `ActiveRedisInspection`, and a manifest that still sets it is rejected on its next apply. See [Set Up Disaster Recovery Replication](./functions/95-disaster-recovery/20-disaster-recovery/20-setup.mdx#redis-60--legacy-module). + + ## Upgrade Strategies ### Minor Version Upgrade diff --git a/docs/shared/crds/middleware.alauda.io_redis.yaml b/docs/shared/crds/middleware.alauda.io_redis.yaml index d20573f..3f1e2c2 100644 --- a/docs/shared/crds/middleware.alauda.io_redis.yaml +++ b/docs/shared/crds/middleware.alauda.io_redis.yaml @@ -1567,17 +1567,26 @@ spec: type: string announcePort: description: |- - AnnouncePort is the externally-reachable RESP port peers dial at - AnnounceAddress for the mesh CONTROL plane (gossip) and the legacy - as.peersync transport — module-side it becomes the announced node - record's gossip/legacy dial port (mesh-announce-port). Behind the - operator this is the proxy's RESP listener, so it defaults to the proxy - Service port (6379); override it only when an external LB/F5 exposes - that port under a different number. It does NOT affect the AS_MESH data - plane: the module always advertises its own local peer-port (7379) for - that, so any external endpoint must expose 7379 as 7379. It is DECOUPLED - from the in-cluster Service Port/NodePort; the operator does not copy it - into the k8s Service NodePort. + AnnouncePort is the RESP port this instance publishes as its mesh + IDENTITY — module-side it becomes field 7 of the announced node record + (mesh-announce-port). Behind the operator that is the proxy's RESP + listener, so it defaults to the proxy Service port (6379); override it + only when an external LB/F5 exposes that port under a different number. + + It carries NO traffic. Both mesh planes dial the PEER port instead: the + AS_MESH data plane through connectPeerPort, and gossip through + gossipOnePeerPort, which is the only gossip transport left since the + plaintext redis-command fallback was removed with the legacy as.peersync + path (redis-modules active-redis src/mesh.zig, gossipOne). What this + value is actually read for is self-recognition: a node matches a seed + against its own announced addr+port (seedMatchesSelfLocked), which is + what lets every DC carry one identical mesh-seeds list. + + Because the module advertises its own local peer-port verbatim and there + is no announce override for it, any external endpoint must expose 7379 + as 7379 — remapping it leaves both planes dialling a port nothing serves. + This field is DECOUPLED from the in-cluster Service Port/NodePort; the + operator does not copy it into the k8s Service NodePort. format: int32 maximum: 65535 minimum: 0 diff --git a/docs/shared/crds/redis.middleware.alauda.io_activeredis.yaml b/docs/shared/crds/redis.middleware.alauda.io_activeredis.yaml index 331c62f..d2b5dad 100644 --- a/docs/shared/crds/redis.middleware.alauda.io_activeredis.yaml +++ b/docs/shared/crds/redis.middleware.alauda.io_activeredis.yaml @@ -1560,17 +1560,26 @@ spec: type: string announcePort: description: |- - AnnouncePort is the externally-reachable RESP port peers dial at - AnnounceAddress for the mesh CONTROL plane (gossip) and the legacy - as.peersync transport — module-side it becomes the announced node - record's gossip/legacy dial port (mesh-announce-port). Behind the - operator this is the proxy's RESP listener, so it defaults to the proxy - Service port (6379); override it only when an external LB/F5 exposes - that port under a different number. It does NOT affect the AS_MESH data - plane: the module always advertises its own local peer-port (7379) for - that, so any external endpoint must expose 7379 as 7379. It is DECOUPLED - from the in-cluster Service Port/NodePort; the operator does not copy it - into the k8s Service NodePort. + AnnouncePort is the RESP port this instance publishes as its mesh + IDENTITY — module-side it becomes field 7 of the announced node record + (mesh-announce-port). Behind the operator that is the proxy's RESP + listener, so it defaults to the proxy Service port (6379); override it + only when an external LB/F5 exposes that port under a different number. + + It carries NO traffic. Both mesh planes dial the PEER port instead: the + AS_MESH data plane through connectPeerPort, and gossip through + gossipOnePeerPort, which is the only gossip transport left since the + plaintext redis-command fallback was removed with the legacy as.peersync + path (redis-modules active-redis src/mesh.zig, gossipOne). What this + value is actually read for is self-recognition: a node matches a seed + against its own announced addr+port (seedMatchesSelfLocked), which is + what lets every DC carry one identical mesh-seeds list. + + Because the module advertises its own local peer-port verbatim and there + is no announce override for it, any external endpoint must expose 7379 + as 7379 — remapping it leaves both planes dialling a port nothing serves. + This field is DECOUPLED from the in-cluster Service Port/NodePort; the + operator does not copy it into the k8s Service NodePort. format: int32 maximum: 65535 minimum: 0