fix(hub): surface cluster leadership and make cluster mode startable - #168
fix(hub): surface cluster leadership and make cluster mode startable#168CoderMungan wants to merge 4 commits into
Conversation
Issue ActiveMemory#96 asks for three fields on StatusResponse. Reading the render path first turned up why the field is missing in the first place: ctx hub status already prints a role, a leader and a peer count, and all three answer from data that has nothing to do with Raft -- role from the listener count, leader from the address the CLI just dialed, peers from len(EntriesByProject). The spec covers the wiring the issue asks for plus the three lines it contradicts, and names the two cluster defects it deliberately leaves to the tracked H-12 / H-28 entries. Phase HL carries five implementation tasks and one follow-up for ctx hub stepdown, which prints "Leadership transferred" without calling Cluster.Stepdown(). Spec: specs/hub-status-cluster-leadership.md Signed-off-by: CoderMungan <codermungan@gmail.com>
The hub keeps a Raft Cluster on Server and shuts it down in
GracefulStop, but nothing read its leadership state. Status
reported store and listener counts only, and the three
cluster-shaped lines ctx hub status printed came from
elsewhere: Role from the listener count, so a leader with no
subscribers read "Follower"; Leader from the address the CLI
had just dialed, so every hub claimed to be one; Peers from
len(EntriesByProject), so a single-node hub holding entries
from three projects reported "Peers: 3".
StatusResponse gains ClusterEnabled, IsLeader, LeaderAddr and
ClusterPeers, filled by hubStatus from s.cluster. A failed
configuration read warns through a new HubClusterPeers format
and reports zero peers rather than failing the call: Status is
a diagnostic and the rest of the response is still worth
having. ClusterEnabled is the disambiguator -- without it a
standalone hub cannot be told apart from a clustered node that
has lost its leader, since both report no leadership.
ctx hub status now renders what the hub said. Standalone prints
the role and the entry count and stops; a clustered node adds
the leader -- or "Leader: unknown (election in progress)" while
Raft has none -- and the peer count. The fields travel as a
ClusterStatusInfo struct rather than a seventh positional
parameter. RoleActive labelled the listener-count heuristic and
goes away; RoleLeader and RoleStandalone replace it.
Cluster.LeaderAddr returned the ServerID that LeaderWithID's
second value carries, discarding the address its name and
docstring promise; it now returns the address. Cluster.Peers
reads the committed configuration and excludes self, counting
into a uint32 so the wire type needs no gosec narrowing
suppression. NewCluster's two bootstrap branches collapse into
one BootstrapCluster call whose error is checked, tolerating
ErrCantBootstrap -- the restart case. A node that failed to
bootstrap used to come back healthy-looking and never elect
anyone, which is exactly the state these fields exist to show.
Wiring the fields surfaced that the CLI could never reach them.
RunDaemon built its re-exec argv from --port and --data-dir
only, so ctx hub start --daemon --peers started a standalone
hub and reported success; the HA recipe's three commands
produced three unrelated hubs. Run the same command in the
foreground, where the flag did arrive, and Run advertised
fmt.Sprintf(":%d", port+1) to raft.NewTCPTransport, which
refuses to advertise an unspecified address: every cluster
start died on "local bind address is not advertisable". The
two defects hid each other, and cluster mode had never started
on any machine.
So --raft-bind lands here too, closing H-28: the address a node
binds its Raft transport to and advertises, with --peers the
other nodes' --raft-bind addresses, so every node bootstraps
the same {ID, Address} set. Wildcard, bare-port and host-less
values are rejected before raft sees them, by a message that
names the flag and the value instead of raft's "not
advertisable". --raft-bind with no peers runs a self-electing
single node, the cheapest way to see the new Status fields.
daemonArgs, extracted from RunDaemon so it can be tested
without forking, forwards both cluster flags.
Verified live before and after on two daemonized hubs: node A
reports Follower naming 127.0.0.1:19911, node B reports Leader,
both report Peers: 1. TestCluster_ThreeNodesElectOneLeader pins
it in-process -- three nodes, exactly one leader, all three
agreeing on its address, two peers each -- and cannot even
start on main. TestHubStatus_ClusterReportsLeadershipState and
TestCluster_LeaderAddrIsTheAddress were verified by mutation:
skipping the s.cluster block fails the first, restoring the
ID-returning LeaderAddr fails both.
The docs the mechanism falsified are corrected in the same
pass. The HA recipe's "expected output" was a per-peer table
with sync state and uptime no version of this code has printed;
the monitoring section documented a --exit-code flag that does
not exist and per-peer replication lag the response does not
carry. And ctx hub peer add|remove and ctx hub stepdown print a
confirmation and reach nothing -- Cluster.Stepdown() has no
caller -- while the recipe documented all three as working
cluster operations, so both doc pages now say what they do.
Wiring them needs new admin-gated RPCs; that stays in TASKS.
Closes ActiveMemory#96
Spec: specs/hub-status-cluster-leadership.md
Signed-off-by: CoderMungan <codermungan@gmail.com>
The daemon re-exec argv is a second flag surface, and it was hiding the startup crash the foreground path would have shown. Neither bug was visible from the other side alone, and the package tests passed throughout -- only running the built binary through the documented flow surfaced both. Spec: specs/hub-status-cluster-leadership.md Signed-off-by: CoderMungan <codermungan@gmail.com>
The three cluster commands printed a confirmation and returned. Cluster.Stepdown() had no caller anywhere in the tree, and raft's AddVoter and RemoveServer were never called at all, so an operator handing off leadership before maintenance got "Leadership transferred" from a process that had asked nobody anything, and ctx hub peer add reported a peer the cluster had never heard of. Two admin-token-gated RPCs replace the printing. Peer carries an action and a Raft address to Cluster.AddPeer or RemovePeer; Stepdown asks the leader to transfer. Both are gated the way Register and Revoke are, because reshaping a cluster is an operator action, not a client one, and both are leader-only: raft refuses a configuration change or a transfer on a follower, and clusterOpErr turns raft.ErrNotLeader into FailedPrecondition whose message names ctx hub status as the way to find the leader. A hub with no Raft node answers FailedPrecondition too, rather than confirming membership it cannot have. peer add needs somewhere to add a node to, which is what --join is: a node that brings up its Raft transport, bootstraps nothing, and waits for a leader to hand it a configuration. Without it a new node would bootstrap its own configuration and be a second cluster of one rather than a member of the first, so --join with --peers is an error. This is the AddVoter half of H-12; the designated-bootstrapper flow and the persisted flag stay with that task. NewCluster's four positional arguments become a ClusterConfig, which is what made room for Join without a fifth. Admin-token resolution (--token, then CTX_HUB_ADMIN_TOKEN) moves into core/admin.Token: revoke had it inline, and peer and stepdown would have been the second and third copy. Verified live end to end, on two daemonized hubs. Node B was started with --raft-bind ... --join; ctx hub peer add on the leader took it from Peers: 0 to Peers: 1 and B reported Role: Follower naming A. ctx hub stepdown on A moved leadership: A then reported Role: Follower and named B as leader. ctx hub peer remove on B took the count back to 0. Stepdown against the follower answered "not the leader: run this against the leader named by ctx hub status", and both the missing-token and --join-with---peers paths were exercised. The tests pin the same contracts against real Raft nodes: TestCluster_PeerAddJoinsNode, PeerRemoveShrinksCluster and StepdownHandsOffLeadership for the cluster half; TestPeer_FollowerIsPrecondition for the ErrNotLeader mapping; the admin-gate, no-cluster and malformed-request refusals for the handlers; TestDaemonArgs_ForwardsJoin because a boolean flag carries no value and is the easiest one to drop from a re-exec argv. The docs stop hedging. The "Not Wired Yet" warnings this branch added two commits ago are replaced by what the commands do, the HA recipe's membership section documents the two-step add and the leader-only rule, and the failure-modes entry claiming "ctx hub stop triggers stepdown first" is corrected: stop sends SIGTERM and the survivors elect after the heartbeat times out, so a planned restart should call stepdown first. Spec: specs/hub-status-cluster-leadership.md Signed-off-by: CoderMungan <codermungan@gmail.com>
|
Pushed 5ca1adb: the two commands I'd flagged as "Not Wired Yet" are wired in this PR after all, rather than left as a follow-up task. What they did before. What they do now. Two admin-token-gated RPCs —
Also folded in: Verified live, two daemonized hubs on loopback: The same contracts are pinned against real Raft nodes by Docs stop hedging. The "Not Wired Yet" warnings are gone; the HA recipe's membership section documents the two-step add and the leader-only rule; and one more false claim fell out on the way —
Ping me if you'd rather see this as a separate PR stacked on top — it is a self-contained commit (5ca1adb) and I can split it out. |
Closes #96.
The three fields the issue asks for (
ClusterEnabled,IsLeader,LeaderAddr), plus a fourth (ClusterPeers) and two defects that made thewhole surface unreachable. Spec:
specs/hub-status-cluster-leadership.md.What the issue asked for
StatusResponsegains the leadership fields,hubStatusfills them froms.cluster, andctx hub statusrenders them.ClusterEnabledis thedisambiguator the issue called for: without it, a standalone hub and a
clustered node that has lost its leader are the same response.
A failed Raft configuration read warns through a new
HubClusterPeersformatand reports zero peers instead of failing the RPC — Status is a diagnostic,
and the rest of the response is still worth having.
Why the render layer changed too
ctx hub statusalready printed three cluster-shaped lines. None of them camefrom Raft:
Role:ConnectedClients > 0 ? "Active" : "Follower"FollowerLeader:cfg.HubAddr, the address the CLI just dialedPeers:len(EntriesByProject)Peers: 3Adding a truthful cluster section next to those would have left the command
contradicting itself, so they now answer from the response:
and, while Raft has no leader for the term,
Leader: unknown (election in progress). TheDropped listeners:line keeps its non-zero condition, so ahealthy hub's output is unchanged apart from these three lines.
RoleActivelabelled the listener-count heuristic and is deleted;RoleLeaderandRoleStandalonereplace it. The render fields travel as aClusterStatusInfostruct rather than a seventh positional parameter.Two defects the wiring surfaced
Cluster.LeaderAddrreturned the ID, not the address.LeaderWithIDreturns
(ServerAddress, ServerID); the method namedLeaderAddrdiscardedthe address. Publishing that as
leader_addrwould have made the mismatch awire contract, so it returns the address now.
Cluster mode had never started, on any machine. Two bugs hiding each
other:
RunDaemonbuilt its re-exec argv from--portand--data-dironly.--peerswas parsed, never forwarded, never seen by the process thatactually ran — so
ctx hub start --daemon --peers …started a standalonehub and reported success. The HA recipe's three
--daemoncommands producedthree unrelated hubs.
In the foreground, where the flag did arrive,
Runpassedfmt.Sprintf(":%d", port+1)toraft.NewTCPTransportas both bind andadvertise address. Raft refuses to advertise an unspecified address:
So
--raft-bindlands here too, which closes H-28 from TASKS.md: theaddress a node binds its Raft transport to and advertises, with
--peersbecoming the other nodes'
--raft-bindaddresses so every node bootstraps thesame
{ID, Address}set. Wildcard, bare-port and host-less values are rejectedbefore raft sees them, by a message that names the flag and the value:
--raft-bindwith no peers runs a self-electing single node — the cheapest wayto see the leadership fields before adding nodes.
daemonArgsis extractedfrom
RunDaemonso the argv can be tested without forking, and forwards bothcluster flags.
NewCluster's two bootstrap branches also collapse into one checkedBootstrapClustercall that toleratesErrCantBootstrap(the restart case).A node that failed to bootstrap used to come back healthy-looking and never
elect anyone — precisely the state these fields exist to reveal.
Verification
Two daemonized hubs on loopback, each with the other as its peer:
and standalone, on the same binary:
Tests:
TestCluster_ThreeNodesElectOneLeader— three real Raft nodes: exactly oneleader, all three naming the same address, two peers each. On
mainthenodes cannot start at all.
TestHubStatus_StandaloneReportsClusterDisabled/TestHubStatus_ClusterReportsLeadershipState— the Status contract bothways. Verified by mutation: skipping the
s.clusterblock fails the second.TestCluster_LeaderAddrIsTheAddress— the node registers under an ID that isnot an address, so the old implementation fails here. Verified by mutation.
TestCluster_PeersExcludesSelf,TestValidateRaftBind,TestDaemonArgs_ForwardsClusterFlags/…_OmitsClusterFlags,TestRenderInfo_*,TestClusterStatus_{Standalone,Cluster,LeaderUnknown}plus the existing dropped-listener pair under the new struct argument.
CGO_ENABLED=0 go build ./...make testgo test -race ./internal/hub/make lint(golangci-lint)make auditmake lint-styleDocs
The recipe's "expected output" block was a per-peer table with sync state and
uptime that no version of this code has printed; the operations monitoring
section documented a
ctx hub status --exit-codeflag that does not exist andper-peer replication lag the response does not carry. Both corrected, and the
HA recipe now shows the two-port topology and the
--raft-bindstart commands.ctx hub peerandctx hub stepdown(third commit)Found on the way, and wired here rather than filed: all three cluster commands
printed a confirmation and returned.
Cluster.Stepdown()had no calleranywhere in the tree, and raft's
AddVoter/RemoveServerwere never calledat all — so a handoff before maintenance reported "Leadership transferred" from
a process that had asked nobody anything.
Two admin-token-gated RPCs replace the printing:
Peercarries an action and a Raft address toCluster.AddPeer/RemovePeer.Stepdownasks the leader to transfer leadership.Both are gated the way
RegisterandRevokeare — reshaping a cluster is anoperator action — and both are leader-only.
clusterOpErrturnsraft.ErrNotLeaderintoFailedPreconditionwhose message namesctx hub statusas the way to find the leader; a hub with no Raft node answersFailedPreconditiontoo, rather than confirming membership it cannot have.peer addneeds somewhere to add a node to, which is what--joinis: anode that brings up its Raft transport, bootstraps nothing, and waits to be
handed a configuration. Without it a new node bootstraps its own configuration
and is a second cluster of one, so
--joinwith--peersis an error. This isthe
AddVoterhalf of H-12; the designated-bootstrapper flow and thepersisted flag stay with that task.
Admin-token resolution (
--token, thenCTX_HUB_ADMIN_TOKEN) moves intocore/admin.Token—revokehad it inline, and these would have been copiestwo and three.
Live, on two daemonized hubs:
Pinned by
TestCluster_PeerAddJoinsNode,TestCluster_PeerRemoveShrinksCluster,TestCluster_StepdownHandsOffLeadership(real Raft nodes),TestPeer_FollowerIsPrecondition(theErrNotLeadermapping), the admin-gateand no-cluster refusals,
TestPeer_ValidatesRequest, andTestDaemonArgs_ForwardsJoin.The docs stop hedging: the "Not Wired Yet" warnings are gone, the HA recipe's
membership section documents the two-step add and the leader-only rule, and the
failure-modes entry claiming "
ctx hub stoptriggersstepdownfirst" iscorrected — stop sends SIGTERM and the survivors elect after the heartbeat
times out, so a planned restart should call
stepdownfirst.Deliberately out of scope, with reasons, at the bottom of the spec: the
Leadershipstreaming RPC andctx hub leadershortcut (non-goals in theissue), Raft term / commit index, the rest of deterministic bootstrap
(H-12), client-side failover to a new leader, and the unauthenticated Raft
transport (H-10/H-11).