Skip to content

fix(hub): surface cluster leadership and make cluster mode startable - #168

Open
CoderMungan wants to merge 4 commits into
ActiveMemory:mainfrom
CoderMungan:fix/hub-status-cluster-leadership
Open

fix(hub): surface cluster leadership and make cluster mode startable#168
CoderMungan wants to merge 4 commits into
ActiveMemory:mainfrom
CoderMungan:fix/hub-status-cluster-leadership

Conversation

@CoderMungan

@CoderMungan CoderMungan commented Sep 8, 2026

Copy link
Copy Markdown
Member

Closes #96.

The three fields the issue asks for (ClusterEnabled, IsLeader,
LeaderAddr), plus a fourth (ClusterPeers) and two defects that made the
whole surface unreachable. Spec: specs/hub-status-cluster-leadership.md.

What the issue asked for

StatusResponse gains the leadership fields, hubStatus fills them from
s.cluster, and ctx hub status renders them. ClusterEnabled is the
disambiguator the issue called for: without it, a standalone hub and a
clustered node that has lost its leader are the same response.

A failed Raft configuration read warns through a new HubClusterPeers format
and reports zero peers instead of failing the RPC — Status is a diagnostic,
and the rest of the response is still worth having.

Why the render layer changed too

ctx hub status already printed three cluster-shaped lines. None of them came
from Raft:

Line Where it came from What that means
Role: ConnectedClients > 0 ? "Active" : "Follower" a leader with no subscribers read Follower
Leader: cfg.HubAddr, the address the CLI just dialed every hub claimed to be the leader
Peers: len(EntriesByProject) a single-node hub holding three projects reported Peers: 3

Adding a truthful cluster section next to those would have left the command
contradicting itself, so they now answer from the response:

Role: Standalone          Role: Leader
Entries: 1248             Leader: 10.0.0.5:9901
                          Entries: 1248  Peers: 2

and, while Raft has no leader for the term, Leader: unknown (election in progress). The Dropped listeners: line keeps its non-zero condition, so a
healthy hub's output is unchanged apart from these three lines.

RoleActive labelled the listener-count heuristic and is deleted;
RoleLeader and RoleStandalone replace it. The render fields travel as a
ClusterStatusInfo struct rather than a seventh positional parameter.

Two defects the wiring surfaced

Cluster.LeaderAddr returned the ID, not the address. LeaderWithID
returns (ServerAddress, ServerID); the method named LeaderAddr discarded
the address. Publishing that as leader_addr would have made the mismatch a
wire contract, so it returns the address now.

Cluster mode had never started, on any machine. Two bugs hiding each
other:

  • RunDaemon built its re-exec argv from --port and --data-dir only.
    --peers was parsed, never forwarded, never seen by the process that
    actually ran — so ctx hub start --daemon --peers … started a standalone
    hub and reported success. The HA recipe's three --daemon commands produced
    three unrelated hubs.

  • In the foreground, where the flag did arrive, Run passed
    fmt.Sprintf(":%d", port+1) to raft.NewTCPTransport as both bind and
    advertise address. Raft refuses to advertise an unspecified address:

    Error: local bind address is not advertisable
    

So --raft-bind lands here too, which closes H-28 from TASKS.md: the
address a node binds its Raft transport to and advertises, with --peers
becoming the other nodes' --raft-bind addresses so every node bootstraps the
same {ID, Address} set. Wildcard, bare-port and host-less values are rejected
before raft sees them, by a message that names the flag and the value:

$ ctx hub start --port 9900 --peers 10.0.0.6:9901
Error: cluster mode needs --raft-bind: the address this node advertises to its peers, e.g. --raft-bind 10.0.0.5:9901

$ ctx hub start --port 9900 --raft-bind :9901
Error: --raft-bind ":9901" is not an address a peer can dial: give a concrete host and port, not a wildcard or bare port

--raft-bind with no peers runs a self-electing single node — the cheapest way
to see the leadership fields before adding nodes. daemonArgs is extracted
from RunDaemon so the argv can be tested without forking, and forwards both
cluster flags.

NewCluster's two bootstrap branches also collapse into one checked
BootstrapCluster call that tolerates ErrCantBootstrap (the restart case).
A node that failed to bootstrap used to come back healthy-looking and never
elect anyone — precisely the state these fields exist to reveal.

Verification

Two daemonized hubs on loopback, each with the other as its peer:

$ ctx hub status          # against node A
Role: Follower
Leader: 127.0.0.1:19911
Entries: 0  Peers: 1

$ ctx hub status          # against node B
Role: Leader
Leader: 127.0.0.1:19911
Entries: 0  Peers: 1

and standalone, on the same binary:

Role: Standalone
Entries: 0

Tests:

  • TestCluster_ThreeNodesElectOneLeader — three real Raft nodes: exactly one
    leader, all three naming the same address, two peers each. On main the
    nodes cannot start at all.
  • TestHubStatus_StandaloneReportsClusterDisabled /
    TestHubStatus_ClusterReportsLeadershipState — the Status contract both
    ways. Verified by mutation: skipping the s.cluster block fails the second.
  • TestCluster_LeaderAddrIsTheAddress — the node registers under an ID that is
    not an address, so the old implementation fails here. Verified by mutation.
  • TestCluster_PeersExcludesSelf, TestValidateRaftBind,
    TestDaemonArgs_ForwardsClusterFlags / …_OmitsClusterFlags,
    TestRenderInfo_*, TestClusterStatus_{Standalone,Cluster,LeaderUnknown}
    plus the existing dropped-listener pair under the new struct argument.
Check Result
CGO_ENABLED=0 go build ./... clean
make test 0 failures
go test -race ./internal/hub/ clean
make lint (golangci-lint) 0 issues
make audit all checks passed
make lint-style style checks passed

Docs

The recipe's "expected output" block was a per-peer table with sync state and
uptime that no version of this code has printed; the operations monitoring
section documented a ctx hub status --exit-code flag that does not exist and
per-peer replication lag the response does not carry. Both corrected, and the
HA recipe now shows the two-port topology and the --raft-bind start commands.

ctx hub peer and ctx hub stepdown (third commit)

Found on the way, and wired here rather than filed: all three cluster commands
printed a confirmation and returned. Cluster.Stepdown() had no caller
anywhere in the tree, and raft's AddVoter / RemoveServer were never called
at all — so a handoff before maintenance reported "Leadership transferred" from
a process that had asked nobody anything.

Two admin-token-gated RPCs replace the printing:

  • Peer carries an action and a Raft address to Cluster.AddPeer /
    RemovePeer.
  • Stepdown asks the leader to transfer leadership.

Both are gated the way Register and Revoke are — reshaping a cluster is an
operator action — and both are leader-only. clusterOpErr turns
raft.ErrNotLeader into FailedPrecondition whose message names
ctx hub status as the way to find the leader; a hub with no Raft node answers
FailedPrecondition too, rather than confirming membership it cannot have.

peer add needs somewhere to add a node to, which is what --join is: a
node that brings up its Raft transport, bootstraps nothing, and waits to be
handed a configuration. Without it a new node bootstraps its own configuration
and is a second cluster of one, so --join with --peers is an error. This is
the AddVoter half of H-12; the designated-bootstrapper flow and the
persisted flag stay with that task.

Admin-token resolution (--token, then CTX_HUB_ADMIN_TOKEN) moves into
core/admin.Tokenrevoke had it inline, and these would have been copies
two and three.

Live, on two daemonized hubs:

# B started with --raft-bind 127.0.0.1:19911 --join
$ ctx hub peer add 127.0.0.1:19911 --token ctx_adm_...
Added peer 127.0.0.1:19911
$ ctx hub status                 # leader A
Role: Leader
Leader: 127.0.0.1:19901
Entries: 0  Peers: 1
$ ctx hub status                 # joined node B
Role: Follower
Leader: 127.0.0.1:19901

$ ctx hub stepdown --token ctx_adm_...    # on the leader
Leadership transferred
$ ctx hub status                 # A, after the handoff
Role: Follower
Leader: 127.0.0.1:19911

$ ctx hub stepdown --token ctx_adm_...    # on a follower
Error: rpc error: code = FailedPrecondition desc = not the leader: run this against the leader named by ctx hub status

Pinned by TestCluster_PeerAddJoinsNode,
TestCluster_PeerRemoveShrinksCluster,
TestCluster_StepdownHandsOffLeadership (real Raft nodes),
TestPeer_FollowerIsPrecondition (the ErrNotLeader mapping), the admin-gate
and no-cluster refusals, TestPeer_ValidatesRequest, and
TestDaemonArgs_ForwardsJoin.

The docs stop hedging: the "Not Wired Yet" warnings are gone, the HA recipe's
membership section documents the two-step add and the leader-only rule, and the
failure-modes entry claiming "ctx hub stop triggers stepdown first" is
corrected — stop sends SIGTERM and the survivors elect after the heartbeat
times out, so a planned restart should call stepdown first.

Deliberately out of scope, with reasons, at the bottom of the spec: the
Leadership streaming RPC and ctx hub leader shortcut (non-goals in the
issue), Raft term / commit index, the rest of deterministic bootstrap
(H-12), client-side failover to a new leader, and the unauthenticated Raft
transport (H-10/H-11).

Issue ActiveMemory#96 asks for three fields on StatusResponse. Reading the
render path first turned up why the field is missing in the first
place: ctx hub status already prints a role, a leader and a peer
count, and all three answer from data that has nothing to do with
Raft -- role from the listener count, leader from the address the
CLI just dialed, peers from len(EntriesByProject). The spec covers
the wiring the issue asks for plus the three lines it contradicts,
and names the two cluster defects it deliberately leaves to the
tracked H-12 / H-28 entries.

Phase HL carries five implementation tasks and one follow-up for
ctx hub stepdown, which prints "Leadership transferred" without
calling Cluster.Stepdown().

Spec: specs/hub-status-cluster-leadership.md
Signed-off-by: CoderMungan <codermungan@gmail.com>
The hub keeps a Raft Cluster on Server and shuts it down in
GracefulStop, but nothing read its leadership state. Status
reported store and listener counts only, and the three
cluster-shaped lines ctx hub status printed came from
elsewhere: Role from the listener count, so a leader with no
subscribers read "Follower"; Leader from the address the CLI
had just dialed, so every hub claimed to be one; Peers from
len(EntriesByProject), so a single-node hub holding entries
from three projects reported "Peers: 3".

StatusResponse gains ClusterEnabled, IsLeader, LeaderAddr and
ClusterPeers, filled by hubStatus from s.cluster. A failed
configuration read warns through a new HubClusterPeers format
and reports zero peers rather than failing the call: Status is
a diagnostic and the rest of the response is still worth
having. ClusterEnabled is the disambiguator -- without it a
standalone hub cannot be told apart from a clustered node that
has lost its leader, since both report no leadership.

ctx hub status now renders what the hub said. Standalone prints
the role and the entry count and stops; a clustered node adds
the leader -- or "Leader: unknown (election in progress)" while
Raft has none -- and the peer count. The fields travel as a
ClusterStatusInfo struct rather than a seventh positional
parameter. RoleActive labelled the listener-count heuristic and
goes away; RoleLeader and RoleStandalone replace it.

Cluster.LeaderAddr returned the ServerID that LeaderWithID's
second value carries, discarding the address its name and
docstring promise; it now returns the address. Cluster.Peers
reads the committed configuration and excludes self, counting
into a uint32 so the wire type needs no gosec narrowing
suppression. NewCluster's two bootstrap branches collapse into
one BootstrapCluster call whose error is checked, tolerating
ErrCantBootstrap -- the restart case. A node that failed to
bootstrap used to come back healthy-looking and never elect
anyone, which is exactly the state these fields exist to show.

Wiring the fields surfaced that the CLI could never reach them.
RunDaemon built its re-exec argv from --port and --data-dir
only, so ctx hub start --daemon --peers started a standalone
hub and reported success; the HA recipe's three commands
produced three unrelated hubs. Run the same command in the
foreground, where the flag did arrive, and Run advertised
fmt.Sprintf(":%d", port+1) to raft.NewTCPTransport, which
refuses to advertise an unspecified address: every cluster
start died on "local bind address is not advertisable". The
two defects hid each other, and cluster mode had never started
on any machine.

So --raft-bind lands here too, closing H-28: the address a node
binds its Raft transport to and advertises, with --peers the
other nodes' --raft-bind addresses, so every node bootstraps
the same {ID, Address} set. Wildcard, bare-port and host-less
values are rejected before raft sees them, by a message that
names the flag and the value instead of raft's "not
advertisable". --raft-bind with no peers runs a self-electing
single node, the cheapest way to see the new Status fields.
daemonArgs, extracted from RunDaemon so it can be tested
without forking, forwards both cluster flags.

Verified live before and after on two daemonized hubs: node A
reports Follower naming 127.0.0.1:19911, node B reports Leader,
both report Peers: 1. TestCluster_ThreeNodesElectOneLeader pins
it in-process -- three nodes, exactly one leader, all three
agreeing on its address, two peers each -- and cannot even
start on main. TestHubStatus_ClusterReportsLeadershipState and
TestCluster_LeaderAddrIsTheAddress were verified by mutation:
skipping the s.cluster block fails the first, restoring the
ID-returning LeaderAddr fails both.

The docs the mechanism falsified are corrected in the same
pass. The HA recipe's "expected output" was a per-peer table
with sync state and uptime no version of this code has printed;
the monitoring section documented a --exit-code flag that does
not exist and per-peer replication lag the response does not
carry. And ctx hub peer add|remove and ctx hub stepdown print a
confirmation and reach nothing -- Cluster.Stepdown() has no
caller -- while the recipe documented all three as working
cluster operations, so both doc pages now say what they do.
Wiring them needs new admin-gated RPCs; that stays in TASKS.

Closes ActiveMemory#96

Spec: specs/hub-status-cluster-leadership.md
Signed-off-by: CoderMungan <codermungan@gmail.com>
@CoderMungan
CoderMungan marked this pull request as ready for review September 8, 2026 12:33
@CoderMungan CoderMungan changed the title fix(hub): surface cluster leadership in the Status RPC and ctx hub status fix(hub): surface cluster leadership and make cluster mode startable Sep 8, 2026
The daemon re-exec argv is a second flag surface, and it was
hiding the startup crash the foreground path would have shown.
Neither bug was visible from the other side alone, and the
package tests passed throughout -- only running the built
binary through the documented flow surfaced both.

Spec: specs/hub-status-cluster-leadership.md
Signed-off-by: CoderMungan <codermungan@gmail.com>
The three cluster commands printed a confirmation and returned.
Cluster.Stepdown() had no caller anywhere in the tree, and
raft's AddVoter and RemoveServer were never called at all, so
an operator handing off leadership before maintenance got
"Leadership transferred" from a process that had asked nobody
anything, and ctx hub peer add reported a peer the cluster had
never heard of.

Two admin-token-gated RPCs replace the printing. Peer carries
an action and a Raft address to Cluster.AddPeer or RemovePeer;
Stepdown asks the leader to transfer. Both are gated the way
Register and Revoke are, because reshaping a cluster is an
operator action, not a client one, and both are leader-only:
raft refuses a configuration change or a transfer on a
follower, and clusterOpErr turns raft.ErrNotLeader into
FailedPrecondition whose message names ctx hub status as the
way to find the leader. A hub with no Raft node answers
FailedPrecondition too, rather than confirming membership it
cannot have.

peer add needs somewhere to add a node to, which is what
--join is: a node that brings up its Raft transport,
bootstraps nothing, and waits for a leader to hand it a
configuration. Without it a new node would bootstrap its own
configuration and be a second cluster of one rather than a
member of the first, so --join with --peers is an error. This
is the AddVoter half of H-12; the designated-bootstrapper flow
and the persisted flag stay with that task.

NewCluster's four positional arguments become a ClusterConfig,
which is what made room for Join without a fifth.

Admin-token resolution (--token, then CTX_HUB_ADMIN_TOKEN)
moves into core/admin.Token: revoke had it inline, and peer and
stepdown would have been the second and third copy.

Verified live end to end, on two daemonized hubs. Node B was
started with --raft-bind ... --join; ctx hub peer add on the
leader took it from Peers: 0 to Peers: 1 and B reported
Role: Follower naming A. ctx hub stepdown on A moved
leadership: A then reported Role: Follower and named B as
leader. ctx hub peer remove on B took the count back to 0.
Stepdown against the follower answered "not the leader: run
this against the leader named by ctx hub status", and both the
missing-token and --join-with---peers paths were exercised.

The tests pin the same contracts against real Raft nodes:
TestCluster_PeerAddJoinsNode, PeerRemoveShrinksCluster and
StepdownHandsOffLeadership for the cluster half;
TestPeer_FollowerIsPrecondition for the ErrNotLeader mapping;
the admin-gate, no-cluster and malformed-request refusals for
the handlers; TestDaemonArgs_ForwardsJoin because a boolean
flag carries no value and is the easiest one to drop from a
re-exec argv.

The docs stop hedging. The "Not Wired Yet" warnings this branch
added two commits ago are replaced by what the commands do, the
HA recipe's membership section documents the two-step add and
the leader-only rule, and the failure-modes entry claiming
"ctx hub stop triggers stepdown first" is corrected: stop sends
SIGTERM and the survivors elect after the heartbeat times out,
so a planned restart should call stepdown first.

Spec: specs/hub-status-cluster-leadership.md
Signed-off-by: CoderMungan <codermungan@gmail.com>
@CoderMungan

Copy link
Copy Markdown
Member Author

Pushed 5ca1adb: the two commands I'd flagged as "Not Wired Yet" are wired in this PR after all, rather than left as a follow-up task.

What they did before. ctx hub peer add|remove and ctx hub stepdown each called a write helper and returned nil. Cluster.Stepdown() had no caller anywhere in the tree, and raft's AddVoter / RemoveServer were never called at all — so an operator handing off leadership before maintenance got "Leadership transferred" from a process that had asked nobody anything, and peer add reported a peer the cluster had never heard of. The HA recipe documented all three as working cluster operations.

What they do now. Two admin-token-gated RPCs — Peer (action + Raft address → Cluster.AddPeer / RemovePeer) and Stepdown (→ LeadershipTransfer) — gated the way Register and Revoke are, because reshaping a cluster is an operator action, not a client one. Both are leader-only: clusterOpErr maps raft.ErrNotLeader to FailedPrecondition with a message naming ctx hub status as the way to find the leader, and a hub with no Raft node answers FailedPrecondition too instead of confirming membership it cannot have.

--join. peer add needs somewhere to add a node to. A node started with --peers bootstraps its own configuration, which makes it a second cluster of one rather than a member of the first, so a node being added has to start with --join: Raft transport up, nothing bootstrapped, waiting to be handed a configuration. --join together with --peers is an error. This is the AddVoter half of H-12; the designated-bootstrapper flow and the persisted bootstrapped flag stay with that task, and I've noted that on the H-12 entry rather than closing it.

Also folded in: --token / CTX_HUB_ADMIN_TOKEN resolution moved to core/admin.Token, since revoke had it inline and these would have been copies two and three.

Verified live, two daemonized hubs on loopback:

# B started with --raft-bind 127.0.0.1:19911 --join
$ ctx hub peer add 127.0.0.1:19911 --token ctx_adm_...
Added peer 127.0.0.1:19911
$ ctx hub status                            # leader A: Peers 0 -> 1
Role: Leader
Leader: 127.0.0.1:19901
Entries: 0  Peers: 1
$ ctx hub status                            # joined node B
Role: Follower
Leader: 127.0.0.1:19901
Entries: 0  Peers: 1

$ ctx hub stepdown --token ctx_adm_...      # on the leader
Leadership transferred
$ ctx hub status                            # A, after the handoff
Role: Follower
Leader: 127.0.0.1:19911

$ ctx hub peer remove 127.0.0.1:19901 --token ctx_adm_...   # on the new leader
Removed peer 127.0.0.1:19901

$ ctx hub stepdown --token ctx_adm_...      # on a follower
Error: rpc error: code = FailedPrecondition desc = not the leader: run this against the leader named by ctx hub status
$ ctx hub peer add 127.0.0.1:19999          # no token
Error: admin token required: pass --token or set $CTX_HUB_ADMIN_TOKEN
$ ctx hub start --raft-bind 127.0.0.1:19961 --join --peers 127.0.0.1:19901
Error: --join and --peers are mutually exclusive: a joining node gets its configuration from the leader that adds it, ...

The same contracts are pinned against real Raft nodes by TestCluster_PeerAddJoinsNode, TestCluster_PeerRemoveShrinksCluster and TestCluster_StepdownHandsOffLeadership, plus TestPeer_FollowerIsPrecondition for the ErrNotLeader mapping, the admin-gate / no-cluster / malformed-request refusals, and TestDaemonArgs_ForwardsJoin (a boolean flag carries no value, so it is the easiest one to drop from a re-exec argv).

Docs stop hedging. The "Not Wired Yet" warnings are gone; the HA recipe's membership section documents the two-step add and the leader-only rule; and one more false claim fell out on the way — docs/operations/hub-failure-modes.md said "ctx hub stop triggers stepdown first", which it never did. stop sends SIGTERM and the survivors elect after the heartbeat times out, so the entry now says to run stepdown first for a planned restart.

make test, make lint, make audit and make lint-style are all green on the branch. The PR description is updated with the same material; the spec (specs/hub-status-cluster-leadership.md) carries the problem statement, the numbered solution and the out-of-scope list.

Ping me if you'd rather see this as a separate PR stacked on top — it is a self-contained commit (5ca1adb) and I can split it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hub status: surface cluster leadership (IsLeader, LeaderAddr) in Status RPC + ctx hub status

1 participant