Add command: cluster enroll to streamline node migration - #1110
Merged
Merged
Conversation
Two limits show up once the join is driven by a daemon rather than by a sysadmin at a shell. The api location was always built from the --node value, so joining a cluster whose nodenames this node can not resolve was impossible. Add --addr to give that location explicitly, in the [<scheme>://]<addr>[:<port>] format the client already parses, leaving --node as the nodename the join protocol carries. The token could only come from --token, and a token on the command line is readable by any user through the process table. Read OSVC_JOIN_TOKEN when --token is unset, and drop the required mark on --token so either source satisfies the command.
Moving a node from one cluster to another meant a shell on that node,
running om cluster join with a token pasted from the target cluster.
Nothing let an operator drive the move from the cluster the node joins.
Add two endpoints:
* POST /cluster/enroll, on a node of the target cluster. Its body
carries the location of the node to enroll and an access token with
the join role created on it. The ca claim of that token is the trust
anchor for its certificate: its cluster is not ours, a proxy client
only reaches a cluster node, and only a token carrying the join role
holds the claim.
* POST /node/name/{nodename}/daemon/action/join, on the node to
enroll, granted to the join role like the endpoint adding a node to
our own cluster nodes. It validates the order and forks a background
om cluster join, because the join stops the daemon serving the
request.
The enroll handler reads cluster.nodes on the candidate before it
creates anything. That request doubles as the preflight: it proves the
node is reachable, that the token is valid there, and that its ca claim
does anchor its certificate. A join token suffices for that read, which
assertGuest already accepts. A candidate that still has peers is refused
with a 409, because nothing in the join flow tells them to drop it from
their cluster.nodes and they would keep it forever. The single node
precondition also means cluster.nodes holds exactly the nodename the
join events will carry, which the response returns so the caller can
filter on it.
The order carries the location the candidate must reach us at, so it
does not have to resolve our nodename. The join_addr parameter gives it
explicitly and is refused when our certificate is not valid for its
host: VerifyHostname is the function the tls client of the candidate
runs, so we predict its verdict instead of letting a mismatch surface
later, inside the join forked on that node, where the operator sees
nothing. Without the parameter, the location is picked from the names our
certificate is valid for. A name designating us is preferred, since the
certificate is a cluster wide object whose plain names are usually those
of the node that bootstrapped the cluster; a wildcard is expanded with
our own label, which makes such a certificate designate each node of the
cluster in turn. The loopback address the certificate always carries is
skipped: the candidate cannot reach us there. The port comes from what
our listener actually bound when the name designates us, and from the
cluster configuration otherwise, a peer having bound a port of its own
that we do not know.
The token reaches the forked join through the environment, never through
its command line.
The candidate re-checks its own node count: the enroll side only sees a
snapshot taken over the network, and this endpoint is reachable on its
own.
Add o[mx] cluster enroll, driving POST /cluster/enroll from a node of the target cluster. It lives in commoncmd because it only calls the api, so both binaries get it, unlike cluster join and cluster leave which act on the local node. --token-file is preferred over --token: a token on the command line is kept in the shell history and readable through the process table. --wait, on by default, waits for NodeAlive rather than JoinSuccess. JoinSuccess only says the api node updated its cluster.nodes, which happens before the enrolled node has drained, stopped, reconfigured or restarted. NodeAlive says its heartbeat beats here, which is what proves the join step is completed. The event reader is opened before the post, since JoinSuccess can be published before the response reaches us, and the events are matched client side because the nodename is only known from that response. NodeAlive is labeled with the node observing the heartbeat, not the beating one, so that match is on the payload. A JoinSuccess never followed by a NodeAlive is reported as its own failure rather than a timeout: the cluster lists the node, its heartbeat is not beating, and the node is where to look.
cgalibern
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request introduces functionality to enroll a foreign node into an existing cluster, enabling seamless migration of a node from one cluster to another from the target cluster's perspective. It includes the following key changes:
Command Addition: Adds the
cluster enrollcommand, allowing operators to initiate the enrollment process via an API call.--token-filefor secure token provision, avoiding risks of exposing sensitive information via the process table.NodeAliveto ensure the node has successfully joined the cluster.API Enhancements:
POST /cluster/enrollendpoint for initiating the join request.POST /node/name/{nodename}/daemon/action/joinendpoint to accept and process the enrollment action on the candidate node.Cluster Join Updates:
--addrfor explicitly specifying the target node API location when resolving nodenames is not feasible.OSVC_JOIN_TOKENenvironment variable as an alternative to passing it via the command line.