fix(journal): repair a torn WAL tail instead of panicking the shard - #3976
Open
numinnex wants to merge 1 commit into
Open
fix(journal): repair a torn WAL tail instead of panicking the shard#3976numinnex wants to merge 1 commit into
numinnex wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3976 +/- ##
=============================================
- Coverage 84.20% 67.68% -16.53%
- Complexity 1398 1399 +1
=============================================
Files 1219 1218 -1
Lines 173564 151840 -21724
Branches 140663 118939 -21724
=============================================
- Hits 146150 102768 -43382
- Misses 23453 45177 +21724
+ Partials 3961 3895 -66
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FileStorage::truncateis the boot-time repair for a torn metadata WAL tail.It used compio's
set_len, which submitsIORING_OP_FTRUNCATE. That opcodelanded in kernel 6.9. Below it the driver probes the opcode as unsupported and
falls back to
push_blocking, but shard proactors are built withthread_pool_limit(0), so the fallback panics withthe thread pool is needed but no worker thread is running. The panic fires inside dispatch, outsidecatch_unwind_io.Net effect: a crash that tore the last WAL append made the next boot kill the
shard instead of repairing it, and the repair is idempotent, so the node stayed
down across restarts.
Two conditions have to coincide, so this is not a routine path:
kernel < 6.9 IORING_OP_FTRUNCATE absent, compio falls back
AND
torn WAL tail crash mid-append, so boot calls truncate_or_fail
The affected range is 5.19 through 6.8, not everything below 6.9. Ring setup
already requires IORING_SETUP_COOP_TASKRUN and IORING_SETUP_TASKRUN_FLAG,
which need 5.19, so RHEL 9 (5.14) and stock Ubuntu 22.04 (5.15) never start the
server at all and were never exposed. What this actually broke is Debian 12 and
AL2023 (6.1), and Ubuntu 24.04 and 22.04-HWE (6.8). macOS aarch64 is exempt
because create_shard_executor keeps a blocking pool there by design.
The fix
Truncate synchronously through std::fs on the stored path, which needs
neither the opcode nor the blocking pool. This mirrors what segment recovery
already does in truncate_to. The sync_all moves inside truncate, so the
repair is durable on its own and the caller no longer pairs it with a separate
fsync. sync_all rather than sync_data because the file length is metadata,
and without it a power cut right after the repair re-presents the torn tail.
truncate is no longer async. An async fn that never awaits trips
clippy::unused_async, and the journal crate denies clippy::pedantic.
Blocking the shard thread costs nothing here: the sole caller is boot-time
repair, before the shard serves traffic.