perf(storage): recover the WAL without blocking the async runtime

[zonotope]

Follow-up: #2001. Builds on #2003.

Summary

FileStorage::recover_wal replays a root's write-ahead log at startup. It takes the root gate with futures::executor::block_on and replays with synchronous file I/O, all on the calling thread. Async startup paths called it from a runtime thread, so that thread stalled until every in-flight file operation on the root finished, and then for the whole replay.

This PR adds an async recovery path and uses it wherever recovery runs in async code. It also makes the compiler enforce part of the rule the #2001 fix depends on: only blocking-pool threads hold the root gate.

Enforcing the root-gate rule

OperationHold holds a read guard on the root gate. It is now !Send. Returning one from a spawn_blocking task, or holding one across an .await in a Send future, no longer compiles. Both are ways an async task could hold the root gate while recover_wal blocks the thread that task needs. The shape of the #2001 bug, a guard returned through a spawn_blocking result, fails with "*const () cannot be sent between threads safely".

A static assertion next to the struct pins the marker. If OperationHold becomes Send, for example because the marker field is removed, the crate fails to compile with E0283.

The type does not cover a !Send future, such as a LocalSet task, holding one across an .await. The struct's docstring says so, and requires holders to take and release it on a blocking thread, which every current holder does.

Async recovery

recover_wal is split. recover_wal_locked does the replay and takes the root gate's write guard as a parameter, so it cannot run without the gate held. The sync recover_wal takes the gate with block_on and calls it.

FileStorage::recover_wal_async awaits the root gate and moves the write guard into a blocking task, which replays through recover_wal_locked and then releases the gate. No async task holds the gate. Dropping the future before it takes the gate replays nothing. Dropping it afterward leaves the replay to finish on the blocking pool.

run_blocking runs a closure on the blocking pool, resumes a panic on the caller, and returns only cancellation as an error. recover_wal_async and compare_and_swap both use it.

recover_wal's docstring now states that it blocks the calling thread until in-flight file operations on the root finish, and that the replay runs on that thread.

Call sites

Each call site uses the mechanism that fits its context.

Call site	Context	Recovery
create_sync_connection	sync connect	recover_wal
create_async_connection, with and without aws	async	recover_wal_async
FlureeBuilder::build_client and build_client_with_nameservice, used by the server	async	recover_wal_async, for the main root and file-backed address identifiers
FlureeBuilder::build, build_encrypted, build_encrypted_from_config, fluree_file	sync	recover_wal
FlureeBuilder::build_async (new)	async	recover_wal_async
CLI build_fluree	async callers on a multi-thread runtime	build_async
fluree-bench-virtual open	sync, not on a runtime worker	build

Without the aws feature, create_async_connection used to fall back to the sync path for file storage. It now recovers asynchronously too. The connection crate's two copies of the file arm are open_file_storage, file_connection, and file_connection_async.

File-backed connect_async and build_client now need a Tokio runtime context to recover, since recover_wal_async uses spawn_blocking. FileStorage's async I/O already needed one.

In the API crate, the file client, the memory client, and build_local_storage_from_config are async. The local and AWS address-identifier wrappers are one async wrap_address_identifiers, with its S3 arm behind the aws feature. Without aws, an S3 identifier gets the same configuration error as before. With aws, file- and memory-backed clients now open S3 storage for an S3 identifier, as S3-backed clients do. They used to reject it, because the local wrapper sent every identifier to build_local_storage_from_config. This is deliberate but not documented for users, so it is not a commitment yet.

FlureeBuilder::build_async builds the same file-backed instance as build. Both are open_file_storage, then recovery, then build_file.

The CLI's build_fluree is async and awaits build_async. Its 45 call sites gained an .await, and build_store, its one sync caller, is async too.

Docs

Examples that build a file-backed instance inside async code now use build_async().await. This covers the Rust API guide, the BM25, running, benches, encryption, and configuration docs, and the crate-level Quick Start.

Four examples did not compile: the two reindex examples called build().await, and the graph-source overview and R2RML pages called FlureeBuilder::default().build().await, which also has no storage path. They now build a file-backed instance with build_async().await.

The Rust API guide notes when to use build_async and build, including that build needs a Tokio runtime even from synchronous code. The encryption guide lists build_async among the build methods that apply a configured key. The build_encrypted* examples stay sync, since those methods have no async counterpart.

API changes

New public API: FileStorage::recover_wal_async and FlureeBuilder::build_async. FileStorage::crash_with_a_logged_write_for_test is a new #[doc(hidden)] test hook beside the existing ones.

One public signature changed: fluree_db_cli::context::build_fluree is now async. Nothing outside the CLI calls it.

Testing

New tests, each shown to fail with its protection removed:

• recover_wal_async_waits_for_the_root_gate_without_blocking_the_runtime (core). Another thread holds an operation on the root, and a timer on the same current-thread runtime must fire while recovery waits. With the sync recover_wal in its place, it times out.
• recover_wal_async_restores_acknowledged_writes_after_a_crash (core). It replays a crashed log and checks the rebuilt files on disk, since read_bytes replays on a miss by itself. Without the recovery call, it fails at the first disk read.
• file_connection_recovers_the_wal and async_file_connection_recovers_the_wal (connection). Each fails without its connection path's recovery call. Both pass with and without aws, which covers both create_async_connection variants.
• build_async_recovers_the_wal and build_client_recovers_the_wal (api, it_file_startup_recovery). Each fails without its build path's recovery call.

Restoring the pre-#2001 compare-and-swap shape against the !Send marker fails to compile, and removing the marker fails the static assertion.

[aaj3f]

@zonotope this is nice and a nice follow-up to #2001 & #2003. I think the only item from the review worth pointing out up-top (though in reality it feels quite minor) is that the structural guarantee put in place with the !Send marker (_not_send: std::marker::PhantomData<*const ()>), is only a guarantee insofar as a later refactor doesn't remove it. Which is to say, all tests pass green if you delete that field from OperationHold. Some kind of static assertion may be nice to better ensure (rather than merely with comments) that this is durable and not accidentally refactored away. Claude provided some more details below inline. Really nothing other than some nit items though (full review below):

---

Having recover_wal_locked take the write guard as a parameter means the replay can't run without the gate, moving that guard into the blocking task keeps it off async tasks entirely, and sharing open_file_storage/build_file between build and build_async (with the same split in the connection crate) leaves one copy of each startup path instead of two. I probed the parts that are easy to get subtly wrong and they hold: a panic in the replay comes back to the caller as the original panic and frees the root gate; a recover_wal_async dropped after its first poll still finishes the replay before anyone else gets the gate; and one aborted while it waits replays nothing. The !Send marker also does what the body says — the pre-#2001 compare-and-swap shape and a hold across .await in a Send future both fail with the *const () error.

Notes inline: one 🟠 to pin the !Send marker with a static assertion, one ❓ about S3 address identifiers on file/memory clients, and two docs 🟡s. One more with no line to sit on:

• This is more of a question than a suggestion. The title feeds the release notes, and build_async is new public API that embedders have to switch to before they see any of this (build_client* and the connection paths switch on their own). Something like "perf(storage): recover the WAL without blocking the async runtime; add FlureeBuilder::build_async" would tell them.

Adherence to repo commitments:

• Patterns/abstractions: ✔ splits recovery into a locked core with sync and async front doors, shares the setup between them in each crate, and extracts run_blocking from the join compare_and_swap already had; no parallel construct.
• Performance (speed first, memory second): ✔ startup-only. compare_and_swap keeps its single blocking hop with the same panic-resume, and recover_wal_async trades a parked runtime worker for one blocking-pool hop per root at startup. No performance-degradation risk.
• Deployment targets: ✔ recover_wal_async and run_blocking live in the native, non-wasm32 file.rs, the connection crate's file arms carry the same gate, and CI's wasm32 clippy command is clean locally. fluree-db-server builds through build_client, so it picks this up as is; the standalone solo binary opens its engine with build_client/build_client_with_nameservice too, so it gets async recovery at its next db pin with no code change there, and solo's Lambdas use S3. (Small behavior note: file-backed connect_async/build_client now need a Tokio runtime context to recover, which every FileStorage I/O call already needed.)
• Testing: ✔ six new tests across core, connection and api, all run by name here; the runtime test goes red with a block_on swapped back in, and the async connection and build_client tests go red without their recovery calls.
• Conventions: ⚠️ commits are subject-only (the body carries the rationale); fmt and clippy clean; docs updated, with the stragglers inline.

Verified locally at 6c336279f — CI hasn't run on this PR since it's stacked, so I ran what it would have for these crates: cargo fmt --check; clippy -D warnings --all-targets on core, connection, api and cli, with and without aws; CI's wasm32 clippy command; the fluree-db-core storage tests (137), the two connection tests with and without aws, and the two grp_misc startup-recovery tests — all green, with the new tests present by name.

Before merging. Since this is stacked on #2003's branch, CI hasn't run on it beyond the release workflow's plan job: no test or clippy run exists for 6c336279f. It's also based on #2003 as of ab3fe3d7f, before your two newer commits there; merging fix/cas-deadlock in again is clean, and on that merge all 138 core storage tests pass, with #2003's new panic test now pinning run_blocking. Once #2003 merges, I'd retarget this to main and push, because a retarget alone may not start CI (it arrives as an edited event, which isn't among the default activity types ci.yml's pull_request trigger uses), so those jobs actually run on this head before it lands. I also checked it against the other open PRs and there's nothing to coordinate, except one line against #2017 in fluree-db-api/src/lib.rs: your build_file refactor drops the Ok( wrapper, and #2017 renames attachment_provider_cell to ledger_manager_cell. That's trivial whichever of the two lands second.

Approving so you can merge when ready, but maybe worth folding in the marker assertion and the doc stragglers first — they're small, and I'd rather see them in this PR than lost in the backlog.

[aaj3f]

🟠 Should address — nothing pins this marker, so the rule the body says the compiler now enforces can be refactored away without a red build.

I confirmed the marker does what the body says: putting the pre-#2001 shape back (returning locked_read_blocking's result out of a spawn_blocking) fails with "*const () cannot be sent between threads safely", and so does holding an OperationHold across .await in a Send future. But it all hangs on this one field. I deleted _not_send and its initializer, and fluree-db-core and its tests compiled as before — so a cleanup pass that drops an unused-looking underscore field, or a later split of this struct, would quietly let the #2001 shape compile again.

The workspace doesn't depend on static_assertions, but its assert_not_impl_any! expands to something small enough to inline. I added this next to the struct: it compiles at this head, and with the marker removed it fails with E0283 ("type annotations needed"):

// Stops compiling if `OperationHold` becomes `Send`: both impls would then apply.
const _: fn() = || {
    trait AmbiguousIfSend<A> {
        fn some_item() {}
    }
    impl<T: ?Sized> AmbiguousIfSend<()> for T {}
    impl<T: ?Sized + Send> AmbiguousIfSend<u8> for T {}
    let _ = <OperationHold as AmbiguousIfSend<_>>::some_item;
};

Smaller, fwiw: !Send can't see a non-Send future. An OperationHold held across .await inside a LocalSet task (the kind of context #2001's test ran in) still compiles — I checked. So "Its release never waits on an async task to be polled" holds because every holder lives in this file on a blocking thread, not because of the type. It may be worth the doc comment saying so, so whoever adds the next holder knows the type only covers half of it.

Since the body leans on this as the enforcement, I'd rather see it pinned in this PR than lost in the backlog.

[zonotope]

Right on both counts. I added the assertion and updated the docstring in e1ac529

[aaj3f]

❓ Question — is the S3 arm for file- and memory-backed clients a deliberate change?

Before this PR, build_client_file and build_client_memory went through the local wrap_address_identifiers, which sent every identifier to build_local_storage_from_config — so an S3 address identifier on a file- or memory-backed client failed with "S3 storage in addressIdentifiers is only supported with 'aws' feature", even with aws on. With the two wrappers merged, an aws build now opens an S3 storage for that identifier instead.

I think that's what the old error message implied should happen, so this may well be intended. But the body only describes the no-aws side ("gets the same configuration error as before"), and I don't see a test with a local base and an S3 identifier. If it's deliberate, a sentence in the body and a small test would make it visible; if not, the local paths probably want to keep rejecting it.

[zonotope]

It was not purely intentional in that this wasn't a design goal from the beginning, but the constraint was only a byproduct of the old implementation. I chose to leave the behavior change in place because restricting it now would just be artificial. This relaxes a requirement, so nothing that previously worked stopped working. I also chose not to document for users or test it so we don't have to commit to this behavior until we have a specific use case and the extra information that brings.

[aaj3f]

🟡 Optional — the same build().await shape is in two more pages, and two file-backed build()? examples were missed.

docs/graph-sources/overview.md:193 and docs/graph-sources/r2rml.md:29 both have let fluree = FlureeBuilder::default().build().await?;. Like the reindex examples, .await on a Result doesn't compile — and default() has no storage path, so even a sync build() there returns "File storage requires a path". FlureeBuilder::file("/path/to/data").build_async().await? would match this page.

Two more still call build()? from what reads as async code: the without_ledger_caching() example at docs/getting-started/rust-api.md:772-774, and the remote-connection example at docs/operations/configuration.md:1129-1132 (the same example this PR converted at rust-api.md:1402-1408).

Four small edits, so if you agree, I'd rather see them in this PR than lost in the backlog.

Commenting here because those files are not in this diff.

[aaj3f]

🟡 Optional — "from synchronous code" can read as "without a runtime".

FlureeBuilder's own docs (fluree-db-api/src/lib.rs:1464-1466) say every build* method, the sync ones included, must be called inside a Tokio runtime, since the default ledger cache spawns its listener. I called FlureeBuilder::file(dir).build() from a plain thread to check, and it panics with "there is no reactor running, must be called from the context of a Tokio 1.x runtime". Someone reading this line could reasonably try it from a plain fn main. Maybe something like:

Suggested change

	`build()` builds the same instance from synchronous code, replaying the log on the calling thread.
	`build()` builds the same instance from synchronous code that has entered a Tokio runtime (for example with `Runtime::enter`), replaying the log on the calling thread.

It's a one-line edit to a line this PR adds, so I'd rather it land here.