Repository navigation
Fix a recovered range split dropped at replay finish: read leader and candidate terms as one pair - #579
Fix a recovered range split dropped at replay finish: read leader and candidate terms as one pair#579liangjchen wants to merge 3 commits into
Conversation
When log replay finishes, CcNode::FinishLogGroupReplay() stores the node group's leader term and then clears its candidate leader term. Code that read the leader term first and the candidate leader term second could load the leader term before the first store and the candidate leader term after the second, and conclude that the node was neither leader nor candidate. SplitFlushRangeOp::Forward() took that for an invalid recovered term and dropped the recovered range split without a log line. Its prepare log and its range write locks stayed behind with no owner: a new split of the range blocked at its first lock, orphan lock recovery looped because the log service kept answering "committed", and the node did not shut down. Sharder::LeaderTerms() now returns both terms, loading the candidate leader term first, the reverse of the store order, so the pair never shows both unset across the hand-over. Every site that read the leader term and then the candidate leader term goes through it: the split and data migration operations, TransactionExecution::CheckLeaderTerm(), CcNode::PinData(), Sharder::FetchLeaderTerm(), the catalog read of a recovering request, the "wait for replay" loops and the remaining checks. Each site keeps its comparisons. RangeBucketCcMap's read used one term per path and only drops its unused leader term load. The two operations now log when they drop a recovered transaction. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. WalkthroughThe change adds ChangesLeader-Term Replay Handling
Estimated code review effort: 3 (Moderate) | ~25 minutes Change: Bug fix Suggested reviewers: Merge Risk:
|
TransactionExecution::CheckLeaderTerm() accepted a recovered transaction whenever this node had a candidate leader term. If the node loses the node group and is elected again before a recovered transaction finishes, the replay in the new term recovers the transaction again, while the one recovered in the old term still passed the check until that replay finished. A recovered transaction now passes only while the candidate leader term equals its own term. The check keeps its original shape and reads both terms through one LeaderTerms() call. Sharder::LeaderTerms() moves into the header, with an overload that calls a function between its two loads, so that a test can promote the candidate leader at the one point where the load order matters. NodeGroupTerms::LeaderOrCandidate() is renamed LeadershipTerm(). The comments on the two term caches, LeaderTerms() and CcNode::PinData() now say plainly what each holds or returns. LeaderTerms-Test checks both changes: LeaderTerms() finds the term whether the promotion lands before, between or after its two loads, and CheckLeaderTerm() accepts a recovered transaction only while this node leads the group or replays its log in the transaction's term. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the conflict in tx_service/tests/CMakeLists.txt: LeaderTerms-Test goes after PayloadReservation-Test and CkptCycle-Test in OWN_MAIN_TESTS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Context
A node that restarts with a range split in flight can drop the recovered split at the moment log replay finishes. The split's prepare log and its range write locks then stay behind with no owner:
This was seen once in EloqDoc's multi-node test
ddl_split_recovery.py(cloud mode, last scenario: a node is killed with a split in flight while an index build is held). It was found while testing #578 and is independent of it.Cause. A node group's leader term and candidate leader term are two separate atomics in
Sharder. When replay finishes,CcNode::FinishLogGroupReplay()stores the leader term, writes a log line, and then clears the candidate leader term.SplitFlushRangeOp::Forward()read the leader term first and the candidate leader term second. A poll that loads the leader term before the first store and the candidate leader term after the second sees "not leader, not candidate", takes the "Recovered term is invalid" branch, and drops the split without logging anything. The term never changed.Behavior before and after
SplitFlushRangeOp::Forward()andDataMigrationOp::Forward()this dropped a recovered transaction whose term was still valid. InTransactionExecution::CheckLeaderTerm(),CcNode::PinData()andSharder::FetchLeaderTerm()it produced a false "this node holds no term".TransactionExecution::CheckLeaderTerm()accepted a recovered transaction while this node was candidate leader in any term. It now requires the candidate leader term to equal the transaction's term. Otherwise, if this node loses the node group and is elected again before a recovered transaction finishes, the old transaction keeps running next to the one the new term's replay recovers.Implementation
Sharder::LeaderTerms(ng_id)returns aNodeGroupTermspair. It loads the candidate leader term first and the leader term second, the reverse of the store order inFinishLogGroupReplay(). A load that sees the candidate leader term cleared is therefore followed by a load that sees the leader term set.NodeGroupTermshas two accessors for the common shapes:Replaying()(candidate, not yet leader) andLeadershipTerm()(the leader term if this node leads the group; otherwise the candidate leader term if this node is still replaying the group's log).tx_operation.cpp:SplitFlushRangeOp::Forward(),DataMigrationOp::Forward()tx_execution.h:TransactionExecution::CheckLeaderTerm()fault/cc_node.cpp:CheckLogGroupReplayFinished(),PinData(), and three checks on the standby subscription pathssharder.cpp:FetchLeaderTerm()checkpointer.cpp: the term validity checkcc/cc_req_misc.cpp: the catalog fetch's term checkcc/local_cc_shards.cpp: the assertion inCountSlices()remote/cc_node_service.cpp: two "wait for replay" loops andFetchNodeInfotx_index_operation.cpp: one "wait for replay" loopcc/catalog_cc_map.h: the read of a recovering request< 0,> 0,>= 0,== term); only the order of the loads changes. The one exception is the candidate leader term comparison inTransactionExecution::CheckLeaderTerm()(see above). Every recovered transaction takes the candidate leader term of the replay that creates it as its term (range_cc_map.h,range_bucket_cc_map.h,cluster_config_cc_map.h,catalog_cc_map.h), so the new comparison refuses only a transaction recovered in an earlier term.RangeBucketCcMap::Execute(ReadCc&)looked like the same pattern but uses one term per path (candidate leader term for a recovering request, leader term otherwise). It only loses its unused leader term load.FinishLogGroupReplay()gets a comment stating that readers depend on its store order.docs/06-distribution-and-clustering.mdrecords the invariant.LeaderTerms(ng_id, between_loads), an overload for tests, calls a function between the two loads;LeaderTerms(ng_id)is that overload with a function that does nothing. Both are inline insharder.h.tx_service/tests/LeaderTerms-Test.cpp, on one in-process test node:FinishLogGroupReplay()does) runs between the two loads, after both, and before both; the term is found every time.Design decisions and alternatives
Test plan
LeaderTerms-Test; existing suites)Commands and results:
Risk assessment
FinishLogGroupReplay()and the pathShardertakes without a log service both do this; a future writer that clears the candidate leader term first would reopen the race.PinData(),FetchLeaderTerm()andTransactionExecution::CheckLeaderTerm()now always load both terms; they loaded one when the node was already leader.TransactionExecution::CheckLeaderTerm()now refuses a recovered transaction whose term is not the candidate leader term. All creation sites set the term from the candidate leader term, so only a transaction left over from an earlier term is affected.LeaderTerms-Testperforms the promotion itself, in the orderFinishLogGroupReplay()uses; it does not check thatFinishLogGroupReplay()keeps that order. It sets the terms directly instead of driving a real election, so a node that loses the group and is elected again is covered only at the level ofCheckLeaderTerm().Rollback plan
Revert this PR.
Reviewer guide
Sharder::LeaderTerms()andNodeGroupTermsinsharder.h: the load order and why it is safe againstCcNode::FinishLogGroupReplay().SplitFlushRangeOp::Forward()andDataMigrationOp::Forward()intx_operation.cpp: the two sites that dropped work.TransactionExecution::CheckLeaderTerm()intx_execution.h: the one site whose comparison changes.tests/LeaderTerms-Test.cpp: the promotion between the loads and the five term states forCheckLeaderTerm().Follow-up work
TemplateCcMap::Execute(ReadCc&),RangeBucketCcMap::Execute(ReadCc&)andLocalCcHandler::ReadLocal()accept only the candidate leader term for a recovering request, whileCatalogCcMapandReadCc::ValidTermCheck()also accept the leader term. A recovering request that arrives after replay has finished would be refused there. Not changed here; no failure from it has been observed.RecoverDeadTxCc) looks a transaction up by its transaction entry, which a recovered transaction does not have, so it reports a running recovered transaction as not alive. Not verified to cause a failure.🤖 Generated with Claude Code