Implement bounded native execution and one-writer-per-index coordination

[kylebernhardy]

Outcome

Run native indexing, merging, publication, and search with bounded admission and deterministic ownership.

Acceptance criteria

• Exactly one Tantivy IndexWriter per canonical native directory/index/generation; independent indexes can progress in parallel.
• Use bounded native writer/commit lanes and search execution outside the JS event loop and shared libuv pool.
• Enforce aggregate memory, resident writer, native thread, merge, queue and search budgets; current per-index settings are not proof of process-wide bounds.
• Count retained native state during worker loss and preserve control/query completion under saturation.
• Distinguish queued cancellation from started non-cancelable native operations; panic/failure poisons the affected generation.
• Keep Harper runner election and epoch fencing in Harper; native directory locks remain the physical writer guard.
• No synchronous host-storage handoff or blocked-storage-thread pool is required for native MmapDirectory.

Design

Native Tantivy storage and Harper derived indexes. Native Tantivy files are the only delivery target for standalone use and Harper. Proposed APIs and integration gates are not claims of completed implementation.

Comment generated by kAIle (GPT-6)

[kylebernhardy]

One follow-up from the native checkpoint-publication review belongs in this lifecycle issue: add a deterministic test for writer admission racing with poison.

Runtime::enqueue_writer checks require_open() before acquiring the queue lock in try_push(). Runtime::poison changes the state and drains the queue separately. Verify that a request cannot pass the state check, enter the queue after that drain, and execute on a terminal generation. Cover mutations and publication while preserving the close/rollback control path.

This ordering is pre-existing and was identified by source inspection, not a stress-test reproduction. The checkpoint guard now remains armed after an ambiguous commit failure, but that does not make general admission and poison atomic. Please include the deterministic proof and any required fix in this issue before broad concurrent Harper integration.

Comment generated by kAIle (GPT-6)

[kylebernhardy]

Reject native writer operations after terminal failure addresses the admission/poison race recorded above and the related dirty-close race that could reopen a poisoned generation. Both are reproduced deterministically through the real addon; the fixed tests also verify native handle removal and exact document/checkpoint recovery after reopening.

The full gates pass: 116 Rust tests, 74 Node tests, and the packed-consumer test. Claude and Gemini CLI reviewed the plan and implementation. The matched 100k-document benchmark showed no clear slowdown; the PR includes the measurements and their limits.

This issue stays open. Process-wide execution budgets, broader worker-loss/teardown coverage, and remaining ownership qualification are not completed by this change. One teardown case to include is rollback failure after native handle release: check that the JavaScript wrapper can still reach a terminal closed state rather than repeatedly querying a removed handle. That follow-up comes from source inspection, not a fault-injection reproduction in this PR.

Comment generated by kAIle (GPT-6)