Prove native checkpoint publication and the Harper derived-index path

[kylebernhardy]

Outcome

Demonstrate real Harper source delivery through the shared runtime and Fulltext backend into native Tantivy files, with restart/replay correctness.

Acceptance criteria

• First expose generic native publish(payload) and committed-payload readback by reusing engine and existing publication machinery; version the ABI and test plain commit interactions.
• Exercise actual Harper record writes -> elected runner -> projection -> backend -> native apply -> payload-bearing commit -> reader reload -> query.
• Reuse valid files across restart and exact-resume from Harper's validated local cursor; missing/corrupt/incompatible files or source-log gaps rebuild.
• Inject source-commit-before-apply, apply-before-commit, commit-before-reload, and ownership-loss failures; assert exact IDs through upsert/delete/recreate.
• Prove real worker handoff/quiescence and cross-worker query visibility, including publisher loss before refresh notification.
• Verify replicated writes, initial copy/catch-up, eviction, source restore, and local rebuild before query readiness in the Harper-owned integration suite.
• Measure standalone native and integrated Harper with matching corpus/workload plus a no-index Harper control; include unrelated-table traffic and multiple indexes.
• Automate the reproducible cross-repo slice in CI against pinned artifacts; manual builds and process-kill tests do not establish release or power-loss coverage.

Harper integration code and source lifecycle live in Harper; this issue coordinates wrapper prerequisites and paired qualification. Hosted byte storage, chunk layouts, root-flush/WAL-only comparison, and transport feasibility are retired.

Design

Native Tantivy storage and Harper derived indexes. Native Tantivy files are the only delivery target for standalone use and Harper. Proposed APIs and integration gates are not claims of completed implementation.

Comment generated by kAIle (GPT-6)

[kylebernhardy]

Phase 0 now has a real end-to-end storage proof:\n\n- Add Harper RocksDB storage for derived indexes #2535 adds the Harper-owned RocksDB adapter. It maps a Tantivy mutation batch to one WAL-enabled RocksDB transaction and maps Tantivy sync to the existing database-wide RocksDB flush barrier.\n- Add bounded host storage transport for Tantivy #22 was loaded against that adapter, and the real Tantivy Directory conformance/lifecycle proof passed through the native host transport into Harper-owned RocksDB.\n- The adapter reuses Harper and rocksdb-js primitives; it does not introduce a rocksdb-js backend or require changes to rocksdb-js.\n- Schema activation and a runtime dependency on fulltext remain intentionally out of this slice.\n- The existing flush primitive is database-wide. Phase 0 must therefore measure and bound barrier coalescing, write stalls, p99 latency, and interference across multiple full-text indexes before production activation.\n\nComment generated by kAIle (GPT-5)