Shared derived-index runtime for native backends

[kriszyp]

The shared post-commit derived-index runtime for RocksDB tables: lock-elected delivery of committed transaction-log mutations to a native index backend, exact cursor resume, rebuild from primary records, cross-worker readiness, and opt-in writer backpressure. This PR carries the whole stack — #2533 (runtime foundation), #2535 (RocksDB storage surface) and the native-backend additions — so one review covers the derived index end to end and one merge lands it. #2430 (native HNSW) is rebuilt on this branch and is the first consumer; nothing here has a production caller until it lands.

What is here

Runtime (resources/derivedIndexRuntime.ts). One DerivedIndexRuntime per database wakes on the root store's committed event; each registered backend has its own process-wide lock, runner, cursor vector (log name → completed transaction timestamp) and readiness. The elected runner reads committed entries from every physical log, collects identities per transaction, resolves each changed record once against the primary store after its last occurrence, projects only the registered attributes, and delivers bounded batches. Resume proves every saved cursor with exactStart; a boundary the log cannot resolve, a corrupt frame, a removed log or a reload marker condemns the generation and rebuilds. Nothing on the commit path is added beyond the log entry that already exists and a local-only eviction marker for registered caching tables.

Backend contract. One interface: attach(host) (the owner-epoch fence), deliver(batch) (accept/defer/fail), flush(reason) (barrier request), shutdown(epoch) (quiescence), getDurableCursor(), onStateChange(), and optional reset(epoch). batch.records is the last-write-wins view over distinct keys; batch.through is the cursor vector the batch completes and is withheld until an oversized transaction's closing chunk. Every hook is required — a backend that completes inside deliver() implements them trivially — because work that survives a method return is what an ownership handoff has to fence.

Rebuild. rebuilding is published, the previous epoch is quiesced, reset(newEpoch) runs, the committed tail of every log is captured, primary records are scanned and delivered in bounded chunks, the final chunk carries the tail as through, and replay resumes after it. Bounded retry with backoff; unavailable after the budget, honoured by peers.

Shared readiness. State, reason code, attempt count, rebuild request, lag flag and the owner-epoch counter in one getUserSharedBuffer allocation per index, read with plain Atomics.loads on any worker (readDerivedIndexReadiness). Same dependency as primary-key allocation, blob holds and HNSW node ids already carry.

Lag policy (opt-in, maxLagMilliseconds). The owner measures cursor distance, time parked, unread-log age and undurable-work age; past the budget every worker rejects local user writes to the index's tables with a retryable 503 (DERIVED_INDEX_LAGGING) at the staging layer (_writeUpdate / _writeDelete / _writeInvalidate / _writeRelocate), never canonical-source applies, replay or replication notifications. Cleared with hysteresis once the owner proves catch-up, and on every park the runtime cannot leave on its own.

Storage surface (resources/RocksDerivedIndexStorage.ts, from #2535). Binary, WAL-backed key/value for a native index that keeps its state in RocksDB (Tantivy); HNSW does not use it.

Simplifications made in review

The design was cut down before asking for review, each item verified against source rather than inferred:

• One backend contract instead of a synchronous/asynchronous split with a discriminant, registration-time hook validation for one side, a runtime check that a "synchronous" flush returned no promise, and a quiescence path for that undeclared promise. No shipped or planned backend is synchronous.
• Plain words instead of a sequence lock. Nothing read the free-form reason string the seqlock existed to publish; the shared record carries a DerivedIndexReadinessReason code, the epoch counter moved into the same buffer, and the 512-byte record became 32 bytes with no spin. This removed the only construct in the runtime that was novel with respect to Harper's existing use of Atomics over rocksdb-js shared buffers.
• Committed-tail rebuild anchor instead of the oldest retained entry. rocksdb-js advances lastCommittedPosition only to the earliest still-uncommitted write (TransactionLogStore::commitFinished, uncommittedTransactionPositions.front()), so a committed read is a contiguous physical prefix and nothing committed after the capture can sit behind the tail. The whole-retention-window replay after every rebuild, the capture-time reload-marker suppression, its Date.now() comparison against transaction timestamps, the shared reload word and the backward-clock residual are gone; a reload marker is met exactly once.
• Lag latch clears on every terminal park, not only on unavailable: a backend without reset, a runtime without scanRecords, a condemnation marker that could not be written, and entering a rebuild (no cursor to guard; readers act on rebuilding). Before this, three paths released ownership with the latch set and nothing scheduled to clear it, so writes stayed 503 indefinitely; one of them needed a commit wake to retry, and commits were what was being shed.
• partial dropped from DerivedIndexTransaction (nothing consumed it; the withheld cursor already says what is certified), and the per-log cursor checks that could not fail after the whole-vector match removed.
• A log that has never written a file retains its beginning. rocksdb-js reports oldestSequenceNumber: 0 with fileCount: 0 for it; the === 1 test read every brand-new database as a retention gap and failed every rebuild attempt. Found by restacking Native HNSW index: file-primary mmap graph as a backend on the shared derived-index runtime #2430 and defining a fresh table — the first real consumer exercising the runtime.

Verification

• Derived suites (derivedIndexRuntime, derivedIndexRuntimeNativeBackend, derivedIndexRuntimeRocks, derivedIndexRegistry, rocksDerivedIndexStorage): 95 passing. The native-backend suite covers coalescing, oversized transactions, deferral, cadence, rebuild/retry/exhaustion, handoff with an apply scheduled and with a flush pending, shutdown failure holding the lock, condemnation across a simulated restart, lag trip/clear/hysteresis, the three terminal-park cases (each proven red without the fix), tail-anchored rebuilds across several logs, and a real worker thread reading the owner's publication through the binding.
• End to end on real audited RocksDB tables: commit → runtime → queued backend → barrier → readiness; primary scan and replay; write shedding at the staging layer with canonical-source bypass.
• Native HNSW index: file-primary mmap graph as a backend on the shared derived-index runtime #2430 restacked on this branch: vectorIndexPlane.test.js 22 passing against @harperfast/hnsw 0.2.1 — two workers into one native file, a committed write replayed after its worker dies, reload markers, a cursor outside retention, an empty index answering no results, a populated index that lost its file answering 503.
• npm run build, tsc --noEmit, lint:required, prettier: clean.

Refs #2489. Supersedes #2533 and #2535, whose commits this branch contains.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces native-backend readiness additions to the derived-index runtime, implementing a coalesced delivery view, bounded chunked collection and resolution, scheduled durability cadence, a robust rebuild phase, generation fencing, and sequence-locked shared cross-worker readiness. It also includes comprehensive unit tests and benchmarks. The feedback suggests enhancing the documentation to explicitly describe what the 'attach' method provides to future implementers of the 'QueuedDerivedIndexBackend' interface.

[kriszyp]

Continued as #2567 against main: GitHub's stack membership kept refusing a base change on this PR even after #2533 and #2535 were closed, so the same head branch is re-opened there. Nothing else differs. — Claude Fable 5.1