Shared derived-index runtime for native backends

[kriszyp]

The shared post-commit derived-index runtime for RocksDB tables: lock-elected delivery of committed transaction-log mutations to a native index backend, exact cursor resume, rebuild from primary records, cross-worker readiness, and opt-in writer backpressure. This PR carries the whole stack — #2533 (runtime foundation), #2535 and the native-backend additions — so one review covers the derived index end to end and one merge lands it. #2430 (native HNSW) is rebuilt on this branch and is the first consumer; nothing here has a production caller until it lands.

The durable design is the § Derived-index runtime section of the root DESIGN.md (invariants, ownership, cursor semantics, backend contract, bounded delivery, cadence, rebuild, fencing, shared readiness, lag policy). This description holds what is transient: scope, decisions, measurements, verification.

What is here

Runtime (resources/derivedIndexRuntime.ts). One DerivedIndexRuntime per database wakes on the root store's committed event; each registered backend has its own process-wide lock, runner, cursor vector (log name → completed transaction timestamp) and readiness. The elected runner reads committed entries from every physical log, collects identities per transaction, resolves each changed record once against the primary store after its last occurrence, projects only the registered attributes, and delivers bounded batches. Resume proves every saved cursor with exactStart; a boundary the log cannot resolve, a corrupt frame, a removed log or a reload marker condemns the generation and rebuilds. Nothing on the commit path is added beyond the log entry that already exists and a local-only eviction marker for registered caching tables.

Backend contract. One interface: attach(host) (the owner-epoch fence), deliver(batch) (accept/defer/fail), flush(reason) (barrier request), shutdown(epoch) (quiescence), getDurableCursor(), onStateChange(), and optional reset(epoch). batch.records is the last-write-wins view over distinct keys; batch.through is the cursor vector the batch completes and is withheld until an oversized transaction's closing chunk. Every hook is required — a backend that completes inside deliver() implements them trivially — because work that survives a method return is what an ownership handoff has to fence.

Rebuild. rebuilding is published, the previous epoch is quiesced, reset(newEpoch) runs, the committed tail of every log is captured, primary records are scanned and delivered in bounded chunks, the final chunk carries the tail as through, and replay resumes after it. Bounded retry with backoff; unavailable after the budget, honoured by peers.

Shared readiness. State, reason code, attempt count, rebuild request, lag flag and the owner-epoch counter in one getUserSharedBuffer allocation per index, read with plain Atomics.loads on any worker (readDerivedIndexReadiness). Same dependency as primary-key allocation, blob holds and HNSW node ids already carry.

Lag policy (opt-in, maxLagMilliseconds). The owner measures cursor distance, time parked, unread-log age and undurable-work age; past the budget every worker rejects local user writes to the index's tables with a retryable 503 (DERIVED_INDEX_LAGGING) at the staging layer (_writeUpdate / _writeDelete / _writeInvalidate / _writeRelocate), never canonical-source applies, replay or replication notifications. Cleared with hysteresis once the owner proves catch-up, and on every park the runtime cannot leave on its own.

Simplifications made in review

The design was cut down before asking for review, each item verified against source rather than inferred:

• One backend contract instead of a synchronous/asynchronous split with a discriminant, registration-time hook validation for one side, a runtime check that a "synchronous" flush returned no promise, and a quiescence path for that undeclared promise. No shipped or planned backend is synchronous.
• Plain words instead of a sequence lock. Nothing read the free-form reason string the seqlock existed to publish; the shared record carries a DerivedIndexReadinessReason code, the epoch counter moved into the same buffer, and the 512-byte record became 32 bytes with no spin. This removed the only construct in the runtime that was novel with respect to Harper's existing use of Atomics over rocksdb-js shared buffers.
• Committed-tail rebuild anchor instead of the oldest retained entry. rocksdb-js advances lastCommittedPosition only to the earliest still-uncommitted write (TransactionLogStore::commitFinished, uncommittedTransactionPositions.front()), so a committed read is a contiguous physical prefix and nothing committed after the capture can sit behind the tail. The whole-retention-window replay after every rebuild, the capture-time reload-marker suppression, its Date.now() comparison against transaction timestamps, the shared reload word and the backward-clock residual are gone; a reload marker is met exactly once.
• Lag latch clears on every terminal park, not only on unavailable: a backend without reset, a runtime without scanRecords, a condemnation marker that could not be written, and entering a rebuild (no cursor to guard; readers act on rebuilding). Before this, three paths released ownership with the latch set and nothing scheduled to clear it, so writes stayed 503 indefinitely; one of them needed a commit wake to retry, and commits were what was being shed.
• partial dropped from DerivedIndexTransaction (nothing consumed it; the withheld cursor already says what is certified), and the per-log cursor checks that could not fail after the whole-vector match removed.
• RocksDerivedIndexStorage (from Add Harper RocksDB storage for derived indexes #2535) removed. It existed to back Tantivy's KvDirectory with a Harper-owned column family. The HNSW plane persists in its own mmap file, and Prepare the Harper full-text benchmark for native storage #2568's benchmark shows the hosted path failing its own gates (~26 GB through ~108k JS read callbacks for 500 searches; a root-wide flush per publication) while Tantivy's native MmapDirectory passes, so Tantivy will persist the same way. The runtime never depended on it: the cursor is backend-owned. The derived-index column-family open/close/drop hooks in databases.ts go with it.
• A log that has never written a file retains its beginning. rocksdb-js reports oldestSequenceNumber: 0 with fileCount: 0 for it; the === 1 test read every brand-new database as a retention gap and failed every rebuild attempt. Found by restacking Native HNSW index: file-primary mmap graph as a backend on the shared derived-index runtime #2430 and defining a fresh table — the first real consumer exercising the runtime.

Decisions and alternatives

• Transaction log, not an aftercommit stream or a permanent worker-0 drainer. Same-thread aftercommit dispatch has no durable cursor for logs shared by all workers and retains every audit object in a transaction; a fixed drainer pins all indexes to one worker. A lock-elected runner per backend gives each index one serial stream, its own cursor, and parallelism across workers. transactionBroadcast's subscription registry was considered as the base and rejected: its cursors are live-subscription state, its lifecycle is customer-facing, and its aggregate results omit the log identity a cursor vector needs.
• Not a transactional dirty-key outbox (raised twice by planning review, overruled by the repo owner). It survives retention and works with auditing off, but adds a second durable write, a compaction stream and a cleanup protocol to every indexed mutation, a new column family and therefore a storage-format migration for every audited table, and still cannot commit an engine-specific native file atomically with RocksDB — the cursor protocol would be needed regardless.
• Not in-transaction native mutation. Neither Tantivy nor the mmap plane shares RocksDB rollback: pre-commit application leaves phantoms after abort and puts native construction on request latency.
• Rebuild in the runtime, not per backend. Reset → scan → project → deliver → replay is identical for vector and full-text and its duplication is how two runtimes came to exist.
• Resolve after the last collected occurrence, not on first encounter (adopted from planning review): a writer committing between two occurrences of a key inside one chunk would otherwise have its later state certified by the cursor while the earlier state stayed indexed.
• Eviction marker rather than keeping evicted documents and filtering candidates against the primary store, which would make this index the one exception to Harper's index lifecycle and add a primary read to every result.
• Timer-coalesced idle completion rather than a barrier at every idle pass, which would cost one barrier per write for arrivals spaced just beyond drain completion.
• Out of scope: a shared head reader for indexes whose cursors stay within a cohort (no measured N-scan bottleneck yet; the contracts allow it later), and retention pinning (rocksdb-js has no protected-position registration).

Measurements

Synthetic native-cost bench (derivedIndexRuntime.bench.js, 350 µs/apply, 5 ms barrier): 1,000 occurrences over 50 keys in one window — 1,000 applies / 359 ms through transactions, 50 applies / 24 ms through records; queued application held event-loop delay to 6.8 ms max versus 963 ms inline at the same ~2,800 mutations/s. At 1,500 arrivals/s, a 100 ms / 512-mutation cadence delivered 1,492/s with p99 write→durable 116 ms and 30 barriers in 3 s; the defaults (1 s / 4096) give p50 1.27 s and 4 barriers. On the native HNSW backend (#2430's ingest bench, 384-d): 265 native applies for 1,000 commits over 50 keys, foreground put 0.09 ms, apply 0.365 ms, barrier ~9 ms.

Verification

• Derived suites (derivedIndexRuntime, derivedIndexRuntimeNativeBackend, derivedIndexRuntimeRocks, derivedIndexRegistry): 81 passing. The native-backend suite covers coalescing, oversized transactions, deferral, cadence, rebuild/retry/exhaustion, handoff with an apply scheduled and with a flush pending, shutdown failure holding the lock, condemnation across a simulated restart, lag trip/clear/hysteresis, the three terminal-park cases (each proven red without the fix), tail-anchored rebuilds across several logs, and a real worker thread reading the owner's publication through the binding.
• End to end on real audited RocksDB tables: commit → runtime → queued backend → barrier → readiness; primary scan and replay; write shedding at the staging layer with canonical-source bypass.
• Native HNSW index: file-primary mmap graph as a backend on the shared derived-index runtime #2430 restacked on this branch: vectorIndexPlane.test.js 22 passing against @harperfast/hnsw 0.2.1 — two workers into one native file, a committed write replayed after its worker dies, reload markers, a cursor outside retention, an empty index answering no results, a populated index that lost its file answering 503.
• npm run build, tsc --noEmit, lint:required, prettier: clean.

Refs #2489. Supersedes #2533 and #2535 (whose storage adapter is dropped here, see above).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request implements Stage 1 of the derived-index runtime, establishing a lock-elected, cursor-based delivery system of committed RocksDB log mutations to derived-index backends. It introduces the DerivedIndexRuntime and DerivedIndexRunner to coordinate aggregate log iteration, transaction assembly, authoritative record resolution, and a robust rebuild phase. Additionally, it adds host-storage primitives via RocksDerivedIndexStorage, local-only cache eviction markers in Table.ts, and an opt-in writer backpressure lag policy. Comprehensive unit tests, benchmarks, and integration tests are also included to verify correctness and performance. No review comments were provided for this pull request, so there is no additional feedback to address.

[claude]

Reviewed; no blockers found.