Skip to content

Fix a recovered range split dropped at replay finish: read leader and candidate terms as one pair - #579

Open
liangjchen wants to merge 3 commits into
mainfrom
fix/term-read-order
Open

liangjchen wants to merge 3 commits into
mainfrom
fix/term-read-order

Conversation

@liangjchen

@liangjchen liangjchen commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Context

A node that restarts with a range split in flight can drop the recovered split at the moment log replay finishes. The split's prepare log and its range write locks then stay behind with no owner:

  • a new split of the same range blocks forever at its first lock ("prepare acquire all");
  • orphan lock recovery repeats every 10 s, because the log service keeps answering that the transaction committed;
  • reads of the table block, and the node does not shut down on SIGTERM.

This was seen once in EloqDoc's multi-node test ddl_split_recovery.py (cloud mode, last scenario: a node is killed with a split in flight while an index build is held). It was found while testing #578 and is independent of it.

Cause. A node group's leader term and candidate leader term are two separate atomics in Sharder. When replay finishes, CcNode::FinishLogGroupReplay() stores the leader term, writes a log line, and then clears the candidate leader term. SplitFlushRangeOp::Forward() read the leader term first and the candidate leader term second. A poll that loads the leader term before the first store and the candidate leader term after the second sees "not leader, not candidate", takes the "Recovered term is invalid" branch, and drops the split without logging anything. The term never changed.

Behavior before and after

  • Before: code that read the leader term and then the candidate leader term could miss both during the hand-over at replay finish. In SplitFlushRangeOp::Forward() and DataMigrationOp::Forward() this dropped a recovered transaction whose term was still valid. In TransactionExecution::CheckLeaderTerm(), CcNode::PinData() and Sharder::FetchLeaderTerm() it produced a false "this node holds no term".
  • After: the two terms are read as one pair, candidate leader term first. The pair never shows both unset while the node passes from candidate leader to leader.
  • The two operations now log a warning when they drop a recovered transaction, with the transaction's term and both terms read.
  • TransactionExecution::CheckLeaderTerm() accepted a recovered transaction while this node was candidate leader in any term. It now requires the candidate leader term to equal the transaction's term. Otherwise, if this node loses the node group and is elected again before a recovered transaction finishes, the old transaction keeps running next to the one the new term's replay recovers.
  • No wire, log or storage format changes. No setting changes.

Implementation

  • Sharder::LeaderTerms(ng_id) returns a NodeGroupTerms pair. It loads the candidate leader term first and the leader term second, the reverse of the store order in FinishLogGroupReplay(). A load that sees the candidate leader term cleared is therefore followed by a load that sees the leader term set.
  • NodeGroupTerms has two accessors for the common shapes: Replaying() (candidate, not yet leader) and LeadershipTerm() (the leader term if this node leads the group; otherwise the candidate leader term if this node is still replaying the group's log).
  • Every site that read the leader term and then the candidate leader term now goes through the helper, 17 in all:
    • tx_operation.cpp: SplitFlushRangeOp::Forward(), DataMigrationOp::Forward()
    • tx_execution.h: TransactionExecution::CheckLeaderTerm()
    • fault/cc_node.cpp: CheckLogGroupReplayFinished(), PinData(), and three checks on the standby subscription paths
    • sharder.cpp: FetchLeaderTerm()
    • checkpointer.cpp: the term validity check
    • cc/cc_req_misc.cpp: the catalog fetch's term check
    • cc/local_cc_shards.cpp: the assertion in CountSlices()
    • remote/cc_node_service.cpp: two "wait for replay" loops and FetchNodeInfo
    • tx_index_operation.cpp: one "wait for replay" loop
    • cc/catalog_cc_map.h: the read of a recovering request
  • Each site keeps its original comparisons (< 0, > 0, >= 0, == term); only the order of the loads changes. The one exception is the candidate leader term comparison in TransactionExecution::CheckLeaderTerm() (see above). Every recovered transaction takes the candidate leader term of the replay that creates it as its term (range_cc_map.h, range_bucket_cc_map.h, cluster_config_cc_map.h, catalog_cc_map.h), so the new comparison refuses only a transaction recovered in an earlier term.
  • RangeBucketCcMap::Execute(ReadCc&) looked like the same pattern but uses one term per path (candidate leader term for a recovering request, leader term otherwise). It only loses its unused leader term load.
  • FinishLogGroupReplay() gets a comment stating that readers depend on its store order.
  • docs/06-distribution-and-clustering.md records the invariant.
  • LeaderTerms(ng_id, between_loads), an overload for tests, calls a function between the two loads; LeaderTerms(ng_id) is that overload with a function that does nothing. Both are inline in sharder.h.
  • New test tx_service/tests/LeaderTerms-Test.cpp, on one in-process test node:
    • "LeaderTerms never misses the term when the candidate leader is promoted between its two loads": the promotion (leader term stored, then candidate leader term cleared, as FinishLogGroupReplay() does) runs between the two loads, after both, and before both; the term is found every time.
    • "CheckLeaderTerm accepts a recovered transaction only in its own term": a recovered transaction and an ordinary one, checked while the node leads the group in their term, replays it in their term, replays it in a newer term, leads it in a newer term, and neither leads nor replays it.

Design decisions and alternatives

  • Fix the readers, not the writer. With either store order, a reader that loads the leader term first can miss both terms. Only the reader's order can close the window.
  • One helper instead of reordering each site by hand. The safe order is not obvious at a call site, and 17 sites had the unsafe one. The helper documents the rule once.
  • Not packing both terms into one atomic. That would remove the ordering dependency, but it touches every writer and every single-term reader. The helper is sufficient for the hand-over and much smaller.
  • Sites that already read the candidate leader term first are left as they are. They are safe today; converting them would widen the change without fixing anything.

Test plan

  • Unit/CTest coverage (new LeaderTerms-Test; existing suites)
  • Parent-project integration or manual validation
  • Formatting/build checks
  • Recovery, compatibility, or performance validation, when relevant (recovery)
  • Documentation updated, when behavior changed

Commands and results:

Runs 1-4 used main at #577 plus this change, inside the CI image, 2026-11-13.
Added after those runs: the documentation paragraph, the accessor rename to
LeadershipTerm(), comments, the candidate leader term comparison in
TransactionExecution::CheckLeaderTerm(), the LeaderTerms() overload for tests
and LeaderTerms-Test. These are covered by runs 5, 6 and 7. Run 7 used the
final code on main at #578, 2026-11-15; the branch, with main merged in, is
that code exactly.

1. Data substrate CI unit job (Debug, EloqStore data store, RocksDB log state)
   cmake -S . -B bld -DWITH_TESTS=ON -DWITH_DATA_STORE=ELOQDSS_ELOQSTORE \
     -DWITH_LOG_STATE=ROCKSDB -DCMAKE_BUILD_TYPE=Debug
   cmake --build bld -- -k ; (cd bld && ctest --output-on-failure --timeout 180)
   Result: 201 tests, 199 passed, 2 skipped by design, 0 failed.

2. EloqDoc main with data_substrate at this change (RelWithDebInfo, EloqStore,
   fault injection on)
   cmake --build <dir> --target eloqdoc eloqdoc-tests host_manager
   ctest --test-dir <dir> --label-exclude '^integration$'           passed
   Runtime targets eloqdoc-server-smoke, eloqdoc-ddl-recovery,
     eloqdoc-cluster-gate, eloqdoc-change-stream-drivers,
     eloqdoc-cdc-consumer                                            all passed
   Multi-node (cmake/tests/multinode), EloqStore local mode: smoke,
     dss_starting, replay_rerequest, cdc_step_down,
     cluster_gate_multinode, ddl_split_recovery                      6 of 6 passed
   Multi-node, EloqStore cloud mode: voter_takeover plus the same 6  7 of 7 passed
   ddl_split_recovery is the test that hung; all its scenarios passed in both
   modes.

3. The race, forced. A temporary patch pauses 500 us between the two loads in
   Sharder::LeaderTerms() when SplitFlushRangeOp::Forward() polls a recovered
   split. Scenario (vi) of ddl_split_recovery.py, local mode, 3 runs each:
   - before this change, the same pause between the old leader-then-candidate
     reads: hung 3 of 3, with "tx term 3, leader term now 3, candidate term
     now -1" printed from the drop branch;
   - with this change and the pause: passed 3 of 3. The recovered split logged
     "install dirty range" 175 us, 499 us and 308 us after "The leader of cc
     node group ng#0 ... has been recovered", and the new drop warning was
     not logged.
   The temporary patch is not part of this PR.

4. EloqKV main with data_substrate at this change: the ent-ci matrix,
   {Debug, RelWithDebInfo} x {ELOQDSS_ELOQSTORE, ELOQDSS_ROCKSDB_CLOUD_S3,
   ROCKSDB}, phases build / test / cluster through gh_ci_entry.sh
   Result: all 6 lanes passed all 3 phases on the first attempt.

5. Formatting and local build
   git-clang-format-18 --binary clang-format-18 --diff <merge base>  no changes
   ninja txservice TestNodeSmoke-Test CheckpointFlush-Test (Debug, g++-15)  built

6. LeaderTerms-Test (Debug, g++-15, local)
   ctest -L '^LeaderTerms-Test$'                                     2 of 2 passed
   TestNodeSmoke-Test                                                 passed
   Mutation check: the test built against headers with the two old
   behaviors put back (leader term loaded first; a recovered transaction
   accepted with any candidate leader term) fails exactly 4 assertions:
   the promotion between the loads yields leader -1 and candidate -1, and
   a transaction recovered in term 2 is accepted while the node replays
   term 3.

7. The final code on main at #578, inside the CI image (runs 1, 2 and 4
   repeated)
   Data substrate CI unit job: 207 tests, 0 failed, 2 skipped by design
     (LeaderTerms-Test's 2 cases included).
   EloqDoc at eloqdoc PR 507's branch, data_substrate at this change:
     unit tests, eloqdoc-server-smoke, eloqdoc-ddl-recovery,
     eloqdoc-change-stream-drivers, eloqdoc-cdc-consumer           passed
     eloqdoc-cluster-gate failed once: scenario (ix) measured a 0.93 s
     pause against a 1 s lower bound on its 1-3 s back-off. The back-off
     is timed on LocalCcShards' coarse clock, which a timer advances every
     100 ms, and the host was loaded. Reran alone: 5 of 5 passed.
   Multi-node, EloqStore local mode: ddl_split_recovery, cdc_step_down,
     cluster_gate_multinode, smoke, dss_starting, replay_rerequest,
     ckpt_cycles                                                    7 of 7 passed
   Multi-node, EloqStore cloud mode: voter_takeover plus the same 7  8 of 8 passed
   EloqKV ent-ci matrix: build and test phases passed in all 6 lanes;
     cluster phase passed in the 3 Debug lanes and RelWithDebInfo
     ROCKSDB. The RelWithDebInfo ELOQDSS_ROCKSDB_CLOUD_S3 lane passed on
     its fifth attempt; the RelWithDebInfo ELOQDSS_ELOQSTORE lane failed
     5 attempts. Every failure was a sub-second TTL test in expire.tcl
     (PSETEX, PEXPIRE or PEXPIREAT can set sub-second expires), known to
     fail on EloqKV main. expire.tcl alone in the cluster phase's first
     scenario, the two builds alternating, 10 runs each: 3 of 10 failed
     with this change and 3 of 10 on main without it.

Not run:
- No EloqSQL build or tests.
- No performance comparison (the change adds one atomic load on a few paths).

Risk assessment

  • Scope of behavior change. Outside the hand-over window the helper returns the same values the two separate reads returned, and each site keeps its comparisons, so steady-state behavior is unchanged.
  • Ordering dependency. The helper is correct only while the leader term is stored before the candidate leader term is cleared. FinishLogGroupReplay() and the path Sharder takes without a log service both do this; a future writer that clears the candidate leader term first would reopen the race.
  • Hot paths. The catalog read and the range bucket read still load one term for requests that are not recovering. PinData(), FetchLeaderTerm() and TransactionExecution::CheckLeaderTerm() now always load both terms; they loaded one when the node was already leader.
  • Stricter check for recovered transactions. TransactionExecution::CheckLeaderTerm() now refuses a recovered transaction whose term is not the candidate leader term. All creation sites set the term from the candidate leader term, so only a transaction left over from an earlier term is affected.
  • What the regression test does not cover. LeaderTerms-Test performs the promotion itself, in the order FinishLogGroupReplay() uses; it does not check that FinishLogGroupReplay() keeps that order. It sets the terms directly instead of driving a real election, so a node that loses the group and is elected again is covered only at the level of CheckLeaderTerm().

Rollback plan

Revert this PR.

Reviewer guide

  1. Sharder::LeaderTerms() and NodeGroupTerms in sharder.h: the load order and why it is safe against CcNode::FinishLogGroupReplay().
  2. SplitFlushRangeOp::Forward() and DataMigrationOp::Forward() in tx_operation.cpp: the two sites that dropped work.
  3. TransactionExecution::CheckLeaderTerm() in tx_execution.h: the one site whose comparison changes.
  4. The remaining sites are mechanical; check that each comparison is unchanged.
  5. tests/LeaderTerms-Test.cpp: the promotion between the loads and the five term states for CheckLeaderTerm().

Follow-up work

  • TemplateCcMap::Execute(ReadCc&), RangeBucketCcMap::Execute(ReadCc&) and LocalCcHandler::ReadLocal() accept only the candidate leader term for a recovering request, while CatalogCcMap and ReadCc::ValidTermCheck() also accept the leader term. A recovering request that arrives after replay has finished would be refused there. Not changed here; no failure from it has been observed.
  • The orphan lock liveness probe (RecoverDeadTxCc) looks a transaction up by its transaction entry, which a recovered transaction does not have, so it reports a running recovered transaction as not alive. Not verified to cause a failure.

🤖 Generated with Claude Code

When log replay finishes, CcNode::FinishLogGroupReplay() stores the node
group's leader term and then clears its candidate leader term. Code that
read the leader term first and the candidate leader term second could load
the leader term before the first store and the candidate leader term after
the second, and conclude that the node was neither leader nor candidate.

SplitFlushRangeOp::Forward() took that for an invalid recovered term and
dropped the recovered range split without a log line. Its prepare log and
its range write locks stayed behind with no owner: a new split of the range
blocked at its first lock, orphan lock recovery looped because the log
service kept answering "committed", and the node did not shut down.

Sharder::LeaderTerms() now returns both terms, loading the candidate leader
term first, the reverse of the store order, so the pair never shows both
unset across the hand-over. Every site that read the leader term and then
the candidate leader term goes through it: the split and data migration
operations, TransactionExecution::CheckLeaderTerm(), CcNode::PinData(),
Sharder::FetchLeaderTerm(), the catalog read of a recovering request, the
"wait for replay" loops and the remaining checks. Each site keeps its
comparisons. RangeBucketCcMap's read used one term per path and only drops
its unused leader term load.

The two operations now log when they drop a recovered transaction.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change adds NodeGroupTerms and Sharder::LeaderTerms() to read leader and candidate terms together. Catalog, transaction, checkpoint, replay, and remote-service paths now use paired terms or the shared replay-state check.

Changes

Leader-Term Replay Handling

Layer / File(s) Summary
Paired term contract and accessor
tx_service/include/sharder.h, tx_service/src/sharder.cpp, docs/06-distribution-and-clustering.md
Adds NodeGroupTerms and Sharder::LeaderTerms(). The accessor reads the candidate term before the leader term. The documentation describes this ordering and the leadership transition.
Term checks and service decisions
tx_service/include/cc/catalog_cc_map.h, tx_service/include/cc/range_bucket_cc_map.h, tx_service/include/tx_execution.h, tx_service/src/cc/cc_req_misc.cpp, tx_service/src/cc/local_cc_shards.cpp, tx_service/src/checkpointer.cpp, tx_service/src/remote/cc_node_service.cpp, tx_service/tests/LeaderTerms-Test.cpp, tx_service/tests/CMakeLists.txt
Uses paired term reads for catalog and transaction checks, checkpoint and slice checks, and node information. Recovery term selection in CatalogCcMap uses the maximum of leader and standby terms when positive, then falls back to the candidate term. Tests cover paired reads across promotion timing and transaction term matching.
Replay coordination and recovery operations
tx_service/src/fault/cc_node.cpp, tx_service/src/remote/cc_node_service.cpp, tx_service/src/tx_index_operation.cpp, tx_service/src/tx_operation.cpp
Uses paired terms in replay completion, pinning, subscription checks, and recovering transaction checks. Replay waits use Replaying().

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Suggested reviewers: liunyl

Merge Risk: Minimal · up to c4109

This change reads the leader and candidate terms together, so a node that is finishing log replay no longer briefly appears to hold no term. No concrete defect was found in the reviewed changes, and the earlier concerns about recovered transactions and term visibility were refuted by source evidence. It is ready to merge after normal checks.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a recovered range split dropped at replay finish is fixed by reading leader and candidate terms as one pair. The changeset adds Sharder::LeaderTerms() and updates the …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the repository template closely. It covers Context, Behavior before and after, Implementation, Design decisions and alternatives, Test plan with commands and results, Risk asse…
Full details: Title check

Explanation

The title names the main change: a recovered range split dropped at replay finish is fixed by reading leader and candidate terms as one pair. The changeset adds Sharder::LeaderTerms() and updates the affected sites to use it, so the title matches the primary fix.

Full details: Description check

Explanation

The description follows the repository template closely. It covers Context, Behavior before and after, Implementation, Design decisions and alternatives, Test plan with commands and results, Risk assessment, Rollback plan, Reviewer guide, and Follow-up work. It names the cause, the affected sites, the invariant, and the validation runs. The author also states that no automated regression test covers the store-order dependency, and that the test plan checkboxes were all ticked. The Context section links no issue number, and the template asks for a linked issue or RFC when applicable. This is a minor gap, not a critical one.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

A rabbit checks the leader’s term,
Then reads the candidate in turn.
Through replay paths the paired values flow,
Tests track each handover as they go.
The rabbit hops, pleased with the show.

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 145d852b-4291-4ca3-a934-b92aec8179ae
Commits

Reviewing files that changed from the base of the PR and between 2736fe9 and c410948.

Files selected for processing: 12

  • tx_service/include/cc/catalog_cc_map.h
  • tx_service/include/cc/range_bucket_cc_map.h
  • tx_service/include/sharder.h
  • tx_service/include/tx_execution.h
  • tx_service/src/cc/cc_req_misc.cpp
  • tx_service/src/cc/local_cc_shards.cpp
  • tx_service/src/checkpointer.cpp
  • tx_service/src/fault/cc_node.cpp
  • tx_service/src/remote/cc_node_service.cpp
  • tx_service/src/sharder.cpp
  • tx_service/tests/CMakeLists.txt
  • tx_service/tests/LeaderTerms-Test.cpp

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

liangjchen and others added 2 commits November 16, 2026 00:59
TransactionExecution::CheckLeaderTerm() accepted a recovered transaction
whenever this node had a candidate leader term. If the node loses the node
group and is elected again before a recovered transaction finishes, the
replay in the new term recovers the transaction again, while the one
recovered in the old term still passed the check until that replay
finished. A recovered transaction now passes only while the candidate
leader term equals its own term. The check keeps its original shape and
reads both terms through one LeaderTerms() call.

Sharder::LeaderTerms() moves into the header, with an overload that calls
a function between its two loads, so that a test can promote the candidate
leader at the one point where the load order matters.
NodeGroupTerms::LeaderOrCandidate() is renamed LeadershipTerm(). The
comments on the two term caches, LeaderTerms() and CcNode::PinData() now
say plainly what each holds or returns.

LeaderTerms-Test checks both changes: LeaderTerms() finds the term whether
the promotion lands before, between or after its two loads, and
CheckLeaderTerm() accepts a recovered transaction only while this node
leads the group or replays its log in the transaction's term.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the conflict in tx_service/tests/CMakeLists.txt: LeaderTerms-Test
goes after PayloadReservation-Test and CkptCycle-Test in OWN_MAIN_TESTS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant