Skip to content

Add bounded host storage transport for Tantivy - #22

Merged
kylebernhardy merged 7 commits into
mainfrom
codex/issue-7-harper-storage-proof
Sep 9, 2026
Merged

Add bounded host storage transport for Tantivy#22
kylebernhardy merged 7 commits into
mainfrom
codex/issue-7-harper-storage-proof

Conversation

@kylebernhardy

@kylebernhardy kylebernhardy commented Sep 8, 2026

Copy link
Copy Markdown
Member

Adds a bounded, versioned request/response bridge so Tantivy directory operations can run against caller-owned Harper storage. This is the transport and durability-policy foundation for Phase 0: prove the Harper-backed Tantivy storage and derived-index path; the real Harper/RocksDB adapter remains the next slice, so this PR does not close the issue.

For the human reviewer

  1. The native transport admits work against both operation and retained-byte limits before dispatch, while the JavaScript handler is total and converts storage failures into protocol responses. The alternative was allowing unbounded native waiters or exposing rocksdb-js internals; both violate the approved Harper-owned-storage boundary. Changing this later would alter the host-storage protocol and concurrency model.
  2. HostStorage.write() applies an atomic batch through Harper’s existing WAL-backed transaction primitive. Because current rocksdb-js does not expose per-write WAL sync, a Tantivy WAL_SYNC request is deliberately executed as atomic WAL write → explicit durability barrier. If the write succeeds and the barrier fails, the bytes are known-applied but not proven durable; the native commit fails and poisons that generation. The adapter must not claim rollback or continue writing.
  3. The durability barrier is database-wide with current supported primitives. It may include unrelated writes and its latency/stall impact is a required Phase 0 measurement; this PR does not claim that it meets the product performance gate.
  4. Read requests may time out individually, but mutation and durability operations wait for a definitive result. This prevents an ambiguous timeout from making the caller retry an already-applied write.

Verification

The Node proof constructs a host-backed KvDirectory, runs a real Tantivy create/write/commit/reopen lifecycle through the native transport, and verifies atomic WAL writes followed by explicit durability barriers. A fault test proves a failed barrier does not misreport the preceding write as rolled back.

/bin/zsh -lic with the pinned Rust toolchain passed formatting, TypeScript, the native build, and all 13 focused host-storage tests. Earlier full validation passed 38 Rust tests, 33 active Node tests, packed-package verification, and every CI job across Linux, macOS, and Windows; CI is rerunning for the corrected head. Three legacy rocksdb-js lease experiments remain intentionally skipped because that provider will not ship.

Local Claude and Gemini CLI review completed for the durability correction. Gemini reported no findings; Claude’s applicable concerns are captured above.

Comment generated by kAIle (GPT-5)

Complexity: complicated

Review-Coverage: authored=codex; ran=claude,gemini; declined=cursor-grok,cursor-composer,domain; rounds=6 @ 6f324a3

Human-Review-Need: 4 @ 6f324a3

@kylebernhardy kylebernhardy added this to the v5.3 milestone Sep 9, 2026
@kylebernhardy
kylebernhardy marked this pull request as ready for review September 9, 2026 00:08
@kylebernhardy
kylebernhardy force-pushed the codex/issue-7-harper-storage-proof branch from 625260c to 6f324a3 Compare September 9, 2026 04:16
@kylebernhardy
kylebernhardy merged commit a389041 into main Sep 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant