Use fixed chunks for Harper-backed Tantivy files

[kylebernhardy]

Summary

• Replace flush-sized fragment lists with directly addressed 256 KiB chunks and an immutable, revisioned tail.
• Publish a versioned binding atomically with its tail so readers see a complete file revision and existing handles remain immutable.
• Bound writer memory to one reusable chunk buffer and preserve Write retry semantics when staging fails.
• Reject host read-response budgets that cannot carry a complete chunk, while retaining exact per-request transport admission for variable key and metadata sizes.
• Exercise full-chunk, chunk-to-chunk, and chunk-to-tail reads through the host transport and RocksDB storage lease, with MmapDirectory as the oracle.

This is stacked on Add bounded host storage transport for Tantivy #22.

Architecture

The binding v2 value identifies an object generation, its immutable full-chunk count, tail revision and length, and total visible length. Range reads derive the intersecting chunk keys from offsets and fetch no preceding data. Writers stage complete chunks with WAL writes, then atomically publish the tail and binding; a failed publication leaves the prior binding readable.

The 256 KiB chunk size is the first benchmark candidate, not customer configuration or a frozen storage format.

Release boundaries

• The Harper-backed Directory remains test-only and has no public factory in this PR.
• Bounded orphan reclamation is required before release. Each replaced partial tail and deleted object remains until that reclaimer runs.
• The benchmark sweep must measure small-read amplification and retained OwnedBytes before deciding whether to copy small ranges or add a bounded cache.
• The production factory must bound namespace, logical-path, and atomic-metadata sizes before it can validate an aggregate transport budget at construction.

These remaining items stay in Implement Harper-backed Tantivy Directory mapping and atomic publication #11; this PR intentionally does not close it.

Verification

• npm run format
• npm run lint
• npm run test — 47 Rust tests, 35 active Node tests, and packed-package verification pass; three opt-in experimental rocksdb-js lease tests remain skipped without their external checkout.
• Claude reviewed the full change and remediation deltas. Its only carried items are the explicit release gates above.
• Gemini CLI reviewed the complete committed branch and returned no findings.

Comment generated by kAIle (GPT-5)

_{Review-Coverage: authored=codex; ran=claude; declined=gemini,cursor-grok,cursor-composer,domain; rounds=5 @ 0cb93b0}

_{Human-Review-Need: 4 @ 0cb93b0}