diff --git a/DESIGN.md b/DESIGN.md index f5a3347b62..5a640a1abb 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -68,13 +68,16 @@ a version tie with the _executing_ node's name, and a fill from a shared source identity of its own, so two replicas resolving the same tie could keep different values at the same version — the one state anti-entropy cannot repair. On a tie the raced record wins on every replica. A RocksDB replacement whose candidate cannot advance the current version stores at the current version -and carries `VERSION_NOT_UNIQUE_FLAG`; rocksdb-js 2.8.0 ([#766](https://github.com/HarperFast/rocksdb-js/pull/766)) +and carries `VERSION_REUSED` (`resources/RecordEncoder.ts`, aliasing rocksdb-js's own +`VERSION_NOT_UNIQUE_FLAG`); rocksdb-js 2.8.0 ([#766](https://github.com/HarperFast/rocksdb-js/pull/766)) then refuses to publish or confirm that version through the VerificationTable. This avoids inventing an epsilon timestamp solely to force replacement while keeping stale record-cache values from being vouched as fresh. -The flag is also applied to ordinary resequenced RocksDB writes. Those records remain ineligible for -VerificationTable fast-path confirmation until a later write advances their version. +The flag is applied generically to every RocksDB record write whose version does not advance past +the record it replaces (`recordUpdater` in `RecordEncoder.ts`), not just this source-fill path — an +ordinary resequenced (out-of-order CRDT-merged) write gets it too. Those records remain ineligible +for VerificationTable fast-path confirmation until a later write advances their version. ## Blob orphan cleanup: pre-saved files outlive cancelled commits @@ -112,6 +115,12 @@ Opt-in deflate compression for file-backed blobs (harper#2443) has three load-be **A commit's conflict retries have their own deadline, separate from the open-transaction limit** (issue #2450). rocksdb-js ≥2.8 wakes a commit parked on another transaction's write intent after `ROCKSDB_JS_PARK_TIMEOUT_MS` (5s) and returns `RETRY_NOW_VALUE` even when the holder never releases, so a wedged intent presents as a stream of transient conflicts rather than one hung commit — and the `MAX_RETRIES` cap alone then keeps the request pending for ~40 park timeouts, minutes past the configured queue limit. `DatabaseTransaction.commitStartedAt` is stamped on the **chain root** at its first native submission and read at both retry decisions (the coordinated `RETRY_NOW_VALUE` resolve path and the `ERR_BUSY`/`ERR_TRY_AGAIN` rejection path); past `Math.max(STORAGE_MAXTRANSACTIONQUEUETIME, timeoutBudget)` the commit takes the existing `abortChainAfterRetries()` cleanup and throws a 503 `TransactionCommitConflictTimeoutError`. One clock per _logical_ commit, deliberately not per attempt: the per-attempt clock is `trackOutstandingCommit()`'s, which measures native liveness and drives `checkOverloaded()`'s thread-wide shedding, so back-dating it would let one uncapped `sourceApply` retry shed every unrelated request on the thread. The clock is released through the promise `commit()` returns (which settles only after the chained stores' commits), so a reused transaction's next batch starts fresh. `retryable` is true only on a chain root that has not rotated through a mid-scope commit — anywhere else an earlier store already wrote durable audit entries and ran its hooks that a replayed request would repeat. `sourceApply` is exempt, as it is from the attempt cap, for the harper-pro#348 divergence reason above. +## RocksDB range activity and snapshot expiration + +`PrimaryRocksDatabase.getRange` and `RocksIndexStore.getRange` pass native ranges through `trackReadRange`. The native transaction's owner is captured once when constructing the range; each `next()` records activity before native access, including entries later discarded by filters. The transaction monitor consumes that activity at its existing cadence and applies the same read-only idle policy as point reads. It never renews a pending write holder merely because a scan advances. Iterator references still own the original snapshot; the wrapper does not reopen or rotate it. When a handle with no outstanding reader references is handed to the native commit/retry loop, its read-owner association is removed: retry handlers can construct new synchronous ranges on that still-live write handle without being mistaken for expired readers. Existing wrapped ranges retain their captured owner and cannot resume after their read ownership ends. + +RocksDB 2.9.0 binds ranges to their supplied transaction and invalidates them when that transaction ends. An abandoned range is still bounded by the idle monitor. Resuming after its snapshot has been released throws an `ReadSnapshotExpiredError` (503) before accessing the native iterator; retry only the read, without replaying previously committed writes. A poisoned write-bearing transaction retains its existing 422 error and rollback behavior. Early iterator return remains a cleanup operation, including after expiration. + ## Repeat writes to the same key in one transaction carry their state forward (`DatabaseTransaction`/`Table`) A transaction can hold more than one write to the same record key — two `patch()` calls inside one `transaction()`, or a replicated transaction carrying two updates to a record. Each write captures `operation.entry` (its idea of the current record) when it is staged, and **neither engine can refresh that from a read**: LMDB queues staged puts and applies them only in the commit batch, so a `getEntry` inside that loop still returns the pre-transaction record (the exclusive `store.transaction()` fallback is no better), and RocksDB read-your-writes only sees writes already staged into the native transaction — which the source-apply path, staging its whole batch before `commit()`, hasn't done yet. @@ -1899,3 +1908,43 @@ degrades to the historical behavior rather than replacing it. Invariants that ar ## A worker that misses an ITC ack gets its OS thread state logged (`server/threads/manageThreads.js`) `broadcastWithAcknowledgement` already times out (30 s) on a worker whose port stays open but never acks, and that shape is almost always a blocked event loop — a native lock, a runaway synchronous call — which nothing inside the worker can report (harper-pro#788: a restarted node's single http worker went byte-silent while main kept serving `cluster_status`, and the app log only said "not acknowledged by worker thread(s) 2"). So each worker posts its Linux thread id (`readlink /proc/thread-self`) to main once at startup, before anything else runs on it, and the timeout branch reads that thread's kernel state from `/proc/self/task/`: state, `wchan`, the syscall number (the first token only — the rest of that file is argument registers and stack/instruction pointers), CPU ticks, and context-switch counts, plus two cross-platform signals main already has, `worker.performance.eventLoopUtilization()` and the age of the last 1 s resource report. It samples again a second later and logs the deltas: no CPU ticks, no context switches and `event loop active +1000ms` is "parked on a lock"; ticks climbing with state `R` is "spinning". It is deliberately main-thread-only and best-effort: `workers` and the tid live on the main thread's `Worker` objects, every `/proc` field is reported individually (a hardened container may deny `wchan`/`syscall` while `stat` stays readable), a follow-up sample whose `starttime` differs from the first is discarded (the tid may have been recycled), one diagnostic runs per worker with a 30 s cooldown so concurrent timeouts on the same worker don't multiply reads, and nothing here runs when acks arrive on time. It does not name the lock owner; that still needs a native stack from the next occurrence. + +## A table declaration lands where its application's `databases` binding resolves the name (`resources/databases.ts`) + +`table()` is the global instance of an internal target-bound factory, `declareTable(target, definition)`. +A `TableTarget` is the small binding the declaration body needs and nothing more: the root store, the +`tables` graph the class is published into, what to do after a lost create race (another thread created +the table first), and who owns the column-family wrappers the declaration opens. The global target is +`database()` / `databases[name]` / `resetDatabases()`; a branch target (harper#2264) is that branch's +`rootStore` / `tables` / `reloadBranch`, and it adopts every wrapper into `branch.openedStores` so +`close()` releases them. + +An application that declared `branchedDatabases` declares through `scopedTableFactory(branches)`, which +routes each declaration by database name — to the branch of that name, or to `table()` itself. GraphQL +`@table` (`graphql.ts`), `scope.ensureTable` (`components/Scope.ts`, `componentLoader.ts`) and +`defineTable` (`defineTableUsing`, through `security/jsLoader.ts`) all go through it. **An unbranched +application gets `table` and `defineTable` by identity** — `scopedTableFactory(undefined) === table` — +so the request path of every application that does not branch is untouched; only a branched +application pays for the routing, and only at declaration time. + +Consequences to preserve: + +- A branch root store's `databaseName` is its STORE identity (`initStores` stamps `storeName`), and the + branch's blob roots resolve from it. The create path may only fill the name in when it is unset + (`??=`), never overwrite it with the logical name. +- A branch Table class carries the base's logical name, so the Table statics that resolve the global + schema by name (`dropTable`, `addAttributes`, `removeAttributes`, audit-enabling `subscribe`) stay + refused through `assertSchemaMutable`. Schema evolution of a branch table is the factory's + existing-Table path — the re-declaration `@table`/`defineTable`/`ensureTable` perform on every reload — + which runs entirely against the branch's own store and catalog. +- A branch is scope-private: no `updateTable` event names a branch class (declaration, reload and + relationship hydration all pass the announcement policy through), so replication and analytics never + observe one. Cross-thread propagation still happens: the ITC schema-change signal carries + `branchPath`, and `syncSchemaMetadata` (`server/itc/serverHandlers.js`) hands such a message to + `reloadBranchAt`, which re-reads the catalog into the branch's `tables` on every thread that holds + that branch open — the same pre-backfill signal the base path relies on so a worker keeps a new index + maintained while another worker's backfill runs — instead of running the global rescan. +- The lost create race is handled per target: the global path rescans everything (`resetDatabases`); + a branch reloads only itself, and the relationships that reload queues are hydrated through the + application's own branch set (`branch.relatedBranches`, stamped by `prepareBranches`), never the + global map. diff --git a/bin/deploySetup.ts b/bin/deploySetup.ts index 0805c6116c..3fac217e4b 100644 --- a/bin/deploySetup.ts +++ b/bin/deploySetup.ts @@ -36,6 +36,7 @@ import { normalizeGitHost, projectNameFromPackage, GIT_HOST_PATTERN, + isReservedComponentName, PROJECT_NAME_PATTERN, } from '../utility/componentNames.ts'; @@ -74,6 +75,20 @@ export function resolveComponentName(req: any): string | undefined { return undefined; } +/** A credential sealed and granted to a name no deploy can run as is a secret nobody can ever use. */ +export function assertUsableComponentName(component: string): void { + if (!PROJECT_NAME_PATTERN.test(component)) { + throw cliError( + `"${component}" is not a usable component name — a deploy accepts letters, numbers, dashes and underscores.` + ); + } + if (isReservedComponentName(component)) { + throw cliError( + `"${component}" is reserved for Harper's "${component}" configuration section — deploying a component under that name is refused.` + ); + } +} + /** * The canonical bare host for a git credential — `https://github.com/owner/repo` and * `git@github.com` both identify `github.com`. Resolved once and then used for everything the host @@ -185,11 +200,7 @@ export async function deploySetup(req: any): Promise { }) ).project ?? '' ); - if (!PROJECT_NAME_PATTERN.test(component)) { - throw cliError( - `"${component}" is not a usable component name — a deploy accepts letters, numbers, dashes and underscores.` - ); - } + assertUsableComponentName(component); let credentialKey: string; // host (github) or registry (npm) — the credentials-entry discriminator let credentialEntry: Record; diff --git a/build-tools/check-shrinkwrap-pins.mjs b/build-tools/check-shrinkwrap-pins.mjs index bef7b35f29..52a8c49422 100644 --- a/build-tools/check-shrinkwrap-pins.mjs +++ b/build-tools/check-shrinkwrap-pins.mjs @@ -30,7 +30,9 @@ // Usage: node check-shrinkwrap-pins.mjs import { execFileSync } from 'node:child_process'; -import { existsSync, readFileSync } from 'node:fs'; +import { existsSync, readFileSync, realpathSync } from 'node:fs'; +import { createRequire } from 'node:module'; +import { resolve } from 'node:path'; const CHECKED_DEPS = ['@harperfast/rocksdb-js', 'fastify', '@aws-sdk/client-s3']; const ROCKSDB_SINGLE_INSTANCE_DEPS = ['@harperfast/extended-iterable', 'msgpackr']; @@ -51,6 +53,7 @@ if (packed.lockfileVersion !== 3) { process.exit(1); } const manifest = JSON.parse(readFileSync(`${pkgRoot}/package.json`, 'utf8')); +const requireFromRoot = createRequire(realpathSync(resolve(pkgRoot, 'package.json'))); let failed = false; const pins = {}; @@ -175,8 +178,14 @@ function verifyCanariesDiscriminate(pins) { function verifyRocksDbDependencyAlignment() { let rocksdbManifest; + let satisfies; + let validRange; + let requireFromRocksDb; + const rocksdbManifestPath = `${pkgRoot}/node_modules/@harperfast/rocksdb-js/package.json`; try { - rocksdbManifest = JSON.parse(readFileSync(`${pkgRoot}/node_modules/@harperfast/rocksdb-js/package.json`, 'utf8')); + rocksdbManifest = JSON.parse(readFileSync(rocksdbManifestPath, 'utf8')); + ({ satisfies, validRange } = requireFromRoot('semver')); + requireFromRocksDb = createRequire(realpathSync(rocksdbManifestPath)); } catch (e) { console.error(`::error::could not inspect rocksdb-js dependency alignment: ${e.message}`); failed = true; @@ -193,9 +202,23 @@ function verifyRocksDbDependencyAlignment() { failed = true; continue; } - if (rootSpec !== rocksdbSpec) { + if (rocksdbSpec == null) { console.error( - `::error::the root ${dep} pin ${rootSpec} does not match rocksdb-js ${rocksdbSpec ?? 'missing'} -- update these pins together to preserve one module instance` + `::error::rocksdb-js no longer declares ${dep} -- update this check for the new dependency contract before publishing an image` + ); + failed = true; + continue; + } + if (validRange(rocksdbSpec) == null) { + console.error( + `::error::rocksdb-js declares ${dep} with unsupported range ${rocksdbSpec} -- use a semver range or update this check for the new dependency contract` + ); + failed = true; + continue; + } + if (!satisfies(rootSpec, rocksdbSpec)) { + console.error( + `::error::the root ${dep} pin ${rootSpec} is outside rocksdb-js ${rocksdbSpec} -- update these specs together to preserve one module instance` ); failed = true; continue; @@ -212,15 +235,28 @@ function verifyRocksDbDependencyAlignment() { failed = true; } - const nestedManifest = `${pkgRoot}/node_modules/@harperfast/rocksdb-js/node_modules/${dep}/package.json`; - if (existsSync(nestedManifest)) { - let nestedVersion = 'unknown'; - try { - nestedVersion = JSON.parse(readFileSync(nestedManifest, 'utf8')).version; - } catch {} - console.error( - `::error::rocksdb-js loaded a nested ${dep}@${nestedVersion} -- root and rocksdb-js must share one module instance` - ); + try { + const rootResolution = realpathSync(requireFromRoot.resolve(dep)); + const rocksdbResolution = realpathSync(requireFromRocksDb.resolve(dep)); + if (rootResolution === rocksdbResolution) continue; + + const nestedManifest = `${pkgRoot}/node_modules/@harperfast/rocksdb-js/node_modules/${dep}/package.json`; + if (existsSync(nestedManifest)) { + let nestedVersion = 'unknown'; + try { + nestedVersion = JSON.parse(readFileSync(nestedManifest, 'utf8')).version; + } catch {} + console.error( + `::error::rocksdb-js loaded a nested ${dep}@${nestedVersion} -- root and rocksdb-js must share one module instance` + ); + } else { + console.error( + `::error::rocksdb-js resolves ${dep} from ${rocksdbResolution}, but the root resolves it from ${rootResolution} -- both must share one module instance` + ); + } + failed = true; + } catch (e) { + console.error(`::error::could not resolve the shared ${dep} instance: ${e?.message ?? e}`); failed = true; } } @@ -248,9 +284,6 @@ function reportMissingRange(dep, range) { failed = true; } -// Numeric major.minor.patch comparison, ignoring any prerelease/build suffix -- sufficient -// for the stable releases this check compares (avoids depending on a semver-parsing -// package that may not be resolvable from this script's own location). function compareVersions(a, b) { const partsA = a.split(/[-+]/)[0].split('.').map(Number); const partsB = b.split(/[-+]/)[0].split('.').map(Number); diff --git a/components/ApplicationScope.ts b/components/ApplicationScope.ts index d2a827c5a6..b007ee170d 100644 --- a/components/ApplicationScope.ts +++ b/components/ApplicationScope.ts @@ -1,4 +1,5 @@ import type { Resources } from '../resources/Resources.ts'; +import type { BranchDatabase } from '../resources/databases.ts'; import { type Server } from '../server/Server.ts'; import { forComponent } from '../utility/logging/harper_logger.ts'; import { scopedImport } from '../security/jsLoader.ts'; @@ -34,7 +35,7 @@ export class ApplicationScope { * the scoped `databases` binding, and an unbranched scope leaves it undefined so that binding * stays the process-wide singleton by identity. */ - branches?: Map; + branches?: Map; moduleCache: any; // used by the loader to retain a cache of modules, type is an internal detail of the loader #runtimeModules: RuntimeModuleTracker; constructor(name: string, resources: Resources, server: Server, isInternal = false) { diff --git a/components/Scope.ts b/components/Scope.ts index dcf998d86f..2ff45a37de 100644 --- a/components/Scope.ts +++ b/components/Scope.ts @@ -1,8 +1,7 @@ import { type Logger } from '../utility/logging/logger.ts'; import { loggerWithTag } from '../utility/logging/harper_logger.ts'; import { EventEmitter, once } from 'node:events'; -import { databaseEventsEmitter, table } from '../resources/databases.ts'; -import { assertTableTargetNotBranched } from '../resources/branchGuard.ts'; +import { databaseEventsEmitter, scopedTableFactory } from '../resources/databases.ts'; import { server, type Server } from '../server/Server.ts'; import { EntryHandler, type EntryHandlerEventMap, type onEntryEventHandler } from './EntryHandler.ts'; import { OptionsWatcher, OptionsWatcherEventMap } from './OptionsWatcher.ts'; @@ -241,9 +240,8 @@ export class Scope extends EventEmitter { } ensureTable(options: any): TableResourceType { - assertTableTargetNotBranched(this.applicationScope?.branches, options.database, options.table, 'ensureTable'); options.origin = this.#origin; - return table(options); + return scopedTableFactory(this.applicationScope?.branches)(options); } #handleOptionsWatcherReady(): void { diff --git a/components/componentLoader.ts b/components/componentLoader.ts index 27a304d70a..70e748e752 100644 --- a/components/componentLoader.ts +++ b/components/componentLoader.ts @@ -34,17 +34,16 @@ import { trackScopeClose } from './scopeShutdown.ts'; import { deployLifecycle } from './deployLifecycle.ts'; import { assertBranchedDatabases } from './Application.ts'; import { prepareBranches } from '../resources/branchDatabase.ts'; -import { assertTableTargetNotBranched } from '../resources/branchGuard.ts'; import { toScopeMount, nestScopeMount, type ScopeMount } from './scopeMount.ts'; import { scopedImport } from '../security/jsLoader.ts'; import { server } from '../server/Server.ts'; import { Resources } from '../resources/Resources.ts'; -import { table } from '../resources/databases.ts'; +import { scopedTableFactory } from '../resources/databases.ts'; import { getHdbBasePath } from '../utility/environment/environmentManager.ts'; import * as auth from '../security/auth.ts'; import * as mqtt from '../server/mqtt.ts'; import { getConfigObj, getConfigPath } from '../config/configUtils.ts'; -import { bootstrapModels } from '../resources/models/bootstrap.ts'; +import { bootstrapModels, startModelsConfigHotReload } from '../resources/models/bootstrap.ts'; import { ErrorResource } from '../resources/ErrorResource.ts'; import { Scope } from './Scope.ts'; import { ApplicationScope } from './ApplicationScope.ts'; @@ -790,7 +789,10 @@ export async function loadComponent( // methods. Per-entry errors are logged and skipped by `bootstrapModels`. // Awaited so module-backed entries (#1471) finish importing before the // per-component iteration below; built-in entries register synchronously. - if (isRoot) await bootstrapModels(config); + if (isRoot) { + await bootstrapModels(config); + startModelsConfigHotReload(); + } // The `env:` block declares the component's environment expectations (string literal → // process.env; object → declaration satisfied from the hdb_secret store / process.env). @@ -937,12 +939,8 @@ export async function loadComponent( // our own trusted modules can be directly retrieved from our map, otherwise use the (configurable) secure module loader const ensureTable = (options: any) => { - // Same fence as Scope.ensureTable: this legacy closure reaches the process-wide table() - // too, so without it a branched application's extension could still create the table in - // the base through its `start` / `startOnMainThread` hook. - assertTableTargetNotBranched(applicationScope.branches, options.database, options.table, 'ensureTable'); options.origin = origin; - return table(options); + return scopedTableFactory(applicationScope.branches)(options); }; // call the main start hook const network = diff --git a/components/operationsValidation.js b/components/operationsValidation.js index d2d9d09217..0a3945e2b4 100644 --- a/components/operationsValidation.js +++ b/components/operationsValidation.js @@ -13,7 +13,11 @@ const { ENV_ENCRYPTED_PREFIX } = require('../utility/envFile.ts'); // File and project names can only be alphanumeric, dash and underscores. Both patterns are shared // with the CLI (utility/componentNames.ts): `harper deploy setup=true` resolves a project name and a // credential host client-side, and has to reject exactly what these schemas would. -const { PROJECT_NAME_PATTERN: PROJECT_FILE_NAME_REGEX, GIT_HOST_PATTERN } = require('../utility/componentNames.ts'); +const { + PROJECT_NAME_PATTERN: PROJECT_FILE_NAME_REGEX, + GIT_HOST_PATTERN, + isReservedComponentName, +} = require('../utility/componentNames.ts'); const { assertBranchedDatabases } = require('./Application.ts'); // dotenv's accepted key character set. Restricting keys to this prevents a crafted key (e.g. one @@ -74,6 +78,31 @@ function checkProjectExists(checkExists, project, helpers) { } } +// Drop/package/file operations deliberately don't call this: an application deployed under the +// name before it was reserved has to stay removable. +function checkReservedProjectName(project, helpers) { + if (isReservedComponentName(project)) return helpers.message(HDB_ERROR_MSGS.RESERVED_PROJECT_NAME(project)); + return project; +} + +/** + * The file writers create the project directory as a side effect of writing into it, so they refuse + * only the creation — an application that already holds the name stays editable until it is + * migrated off it. + */ +function checkReservedProjectCreation(project, helpers) { + if (!isReservedComponentName(project)) return project; + let projectDir; + try { + const componentsRoot = configUtils.getConfigPath(hdbTerms.CONFIG_PARAMS.COMPONENTSROOT); + if (componentsRoot) projectDir = path.join(componentsRoot, project); + } catch (err) { + hdbLogger.error(err); + } + if (projectDir && fs.existsSync(projectDir)) return project; + return helpers.message(HDB_ERROR_MSGS.RESERVED_PROJECT_NAME(project)); +} + function checkFilePath(path, helpers) { if (path.includes('..')) return helpers.message('Invalid file path'); return path; @@ -155,6 +184,7 @@ function setComponentFileValidator(req) { const setCompSchema = Joi.object({ project: Joi.string() .pattern(PROJECT_FILE_NAME_REGEX) + .custom(checkReservedProjectCreation) .required() .messages({ 'string.pattern.base': HDB_ERROR_MSGS.BAD_PROJECT_NAME }), file: Joi.string().custom(checkFilePath).required(), @@ -215,6 +245,7 @@ function setEnvValueValidator(req) { const schema = Joi.object({ project: Joi.string() .pattern(PROJECT_FILE_NAME_REGEX) + .custom(checkReservedProjectCreation) .required() .messages({ 'string.pattern.base': HDB_ERROR_MSGS.BAD_PROJECT_NAME }), file: Joi.string().custom(checkFilePath).optional(), @@ -331,6 +362,7 @@ function addComponentValidator(req) { const addFuncSchema = Joi.object({ project: Joi.string() .pattern(PROJECT_FILE_NAME_REGEX) + .custom(checkReservedProjectName) .custom(checkProjectExists.bind(null, false)) .required() .messages({ 'string.pattern.base': HDB_ERROR_MSGS.BAD_PROJECT_NAME }), @@ -456,6 +488,7 @@ function deployComponentValidator(req) { const deployProjSchema = Joi.object({ project: Joi.string() .pattern(PROJECT_FILE_NAME_REGEX) + .custom(checkReservedProjectName) .required() .messages({ 'string.pattern.base': HDB_ERROR_MSGS.BAD_PROJECT_NAME }), package: Joi.string().optional(), diff --git a/config/RootConfigWatcher.ts b/config/RootConfigWatcher.ts index d440ea08a8..ebc27f0787 100644 --- a/config/RootConfigWatcher.ts +++ b/config/RootConfigWatcher.ts @@ -15,6 +15,17 @@ import { } from '../utility/watcherFallback.ts'; import { resolveWatchTarget } from '../utility/watchPath.ts'; +let sharedWatcher: RootConfigWatcher | undefined; + +/** + * The isolate-wide watcher for the root config file. Consumers (logging, models) subscribe to this + * one instance instead of each opening their own native watcher — per-worker duplicates double the + * inotify/FD footprint and the synchronous read+parse work on every config write. + */ +export function getSharedRootConfigWatcher(): RootConfigWatcher { + return (sharedWatcher ??= new RootConfigWatcher()); +} + export class RootConfigWatcher extends EventEmitter { #configFilePath: string; #watchPath: string; @@ -26,9 +37,9 @@ export class RootConfigWatcher extends EventEmitter { #partialRead: PartialReadRetry; ready: Promise; - constructor() { + constructor(configFilePath: string = getConfigFilePath()) { super(); - this.#configFilePath = getConfigFilePath(); + this.#configFilePath = configFilePath; const watchTarget = resolveWatchTarget(this.#configFilePath); this.#watchPath = watchTarget.path; this.#partialRead = new PartialReadRetry(this.#configFilePath); diff --git a/config/configUtils.ts b/config/configUtils.ts index 6257d3f108..1474597b89 100644 --- a/config/configUtils.ts +++ b/config/configUtils.ts @@ -1,7 +1,12 @@ import * as hdbTerms from '../utility/hdbTerms.ts'; import * as hdbUtils from '../utility/common_utils.ts'; import logger from '../utility/logging/harper_logger.ts'; -import { configValidator, getDomainSocketPathLengthWarning } from '../validation/configValidator.ts'; +import { + configValidator, + getDomainSocketPathLengthWarning, + isLegacySqlApplicationEntry, +} from '../validation/configValidator.ts'; +import { isReservedComponentName } from '../utility/componentNames.ts'; import fs from 'fs-extra'; import YAML from 'yaml'; import path from 'path'; @@ -757,6 +762,10 @@ function validateConfig(configDoc, skipFsValidation = false) { configDoc.setIn(['operationsApi', 'network', 'domainSocket'], domainSocket); const domainSocketWarning = getDomainSocketPathLengthWarning(validation.value.rootPath, domainSocket); if (domainSocketWarning) logger.warn(domainSocketWarning); + if (isLegacySqlApplicationEntry(configJson.sql)) + logger.warn( + "The root config entry 'sql' is an application, but 'sql' now configures Harper's SQL engine. Redeploy that application under a different name and remove the 'sql' entry; until then the SQL engine settings cannot be configured." + ); } /** @@ -826,12 +835,34 @@ function lookupConfigParam(arg: string): string | undefined { return Object.hasOwn(CONFIG_PARAM_MAP, name) ? CONFIG_PARAM_MAP[name] : undefined; } +const COMPONENT_PARAM_SUFFIXES = ['_package', '_port']; + +function suffixEscapedComponentName(arg: string): string | undefined { + if (typeof arg !== 'string') return undefined; + const suffix = COMPONENT_PARAM_SUFFIXES.find((candidate) => arg.endsWith(candidate)); + if (suffix === undefined) return undefined; + const component = arg.slice(0, -suffix.length); + return component === '' ? undefined : component; +} + /** * Component entries (`my-component_package`, `my-component_port`) are operator-named, so they - * cannot be enumerated in CONFIG_PARAM_MAP and bypass it. + * cannot be enumerated in CONFIG_PARAM_MAP and bypass it. This escape is the one way to write a + * root component entry without going through deploy_component, so a reserved name is excluded. */ function isSuffixEscapedParam(arg: string): boolean { - return typeof arg === 'string' && (arg.endsWith('_package') || arg.endsWith('_port')); + const component = suffixEscapedComponentName(arg); + return component !== undefined && !isReservedComponentName(component); +} + +function findReservedComponentParams(args: object): string[] { + const reserved = []; + for (const arg in args) { + if (!Object.hasOwn(args, arg)) continue; + const component = suffixEscapedComponentName(arg); + if (component !== undefined && isReservedComponentName(component)) reserved.push(arg); + } + return reserved; } const MAX_REPORTED_UNRECOGNIZED = 10; @@ -1225,6 +1256,17 @@ export async function setConfiguration(setConfigJson) { true ); } + const reservedComponentParams = findReservedComponentParams(configFields); + if (reservedComponentParams.length > 0) { + throw handleHDBError( + new Error(), + `Unable to update config, cannot configure a component whose name is reserved for Harper's own configuration section: ${describeUnrecognized(reservedComponentParams)}`, + HTTP_STATUS_CODES.BAD_REQUEST, + undefined, + undefined, + true + ); + } // Before any local write: the writer skips names it cannot resolve, so a request mixing // recognized and unrecognized names would otherwise apply the recognized half and still report // success. diff --git a/docs/derived-index-runtime-stage-1.md b/docs/derived-index-runtime-stage-1.md index 52cdbe6ee7..f3861a1256 100644 --- a/docs/derived-index-runtime-stage-1.md +++ b/docs/derived-index-runtime-stage-1.md @@ -139,8 +139,11 @@ The returned iterable also exposes `failedLogs`, naming any physical iterator th unexpected non-corruption error; a derived-index runner treats either that signal or the existing corrupt-frame signal as an availability failure and never advances its cursor through it. A runner starts the aggregate with its backend's `startByLog` vector and preserves the physical log -name on every result. Entries from one physical log are assembled through `endTxn`; a drain budget -is checked only between complete transactions, never in the middle of one. +name on every result. Entries from one physical log are assembled through `endTxn`. The transaction +count and byte budgets are checked only between complete transactions; an oversized transaction is +cut into chunks by the distinct-record and wall-time bounds described in +[Bounded collection, resolution and delivery](#bounded-collection-resolution-and-delivery), and no +cursor is published for it until its `endTxn` entry has been delivered. On construction or reconstruction, the aggregate reader exact-seeks each saved log cursor, validates and consumes exactly one complete anchor transaction per saved log, then merges from those same @@ -165,8 +168,9 @@ than a shared scan whose start is pinned to the slowest cursor. Sharing may be a cohort whose cursor distance is bounded and whose independent fallback is preserved. Within a runner batch, repeated mutations for one `(tableId, recordId)` share one authoritative -primary read and one backend projection. The runtime never calls the resource `get()` path: a cache -miss must not fetch from an origin inside the drain. The configured projection runs in Harper and +primary read and one backend projection, and the batch's `records` view lists each such key once +(see [Coalesced delivery view](#coalesced-delivery-view)). The runtime never calls the resource +`get()` path: a cache miss must not fetch from an origin inside the drain. The configured projection runs in Harper and returns only declared derived-index attributes; record bodies, credentials, and unrelated fields do not cross the backend boundary or enter diagnostic logs. @@ -198,15 +202,21 @@ hide work. Once a full vector becomes durable, older offered vectors and their d window are discarded. The backend must therefore publish the `through` vector atomically with the index state made durable by that barrier. -Accepted-but-not-durable progress is capped at 64 batches by default. At the cap, the runner retains -ownership but stops reading and enters `waiting-durable`; database commit wakes do not retry it. A -backend state-change wake reconciles its cursor and resumes only after a complete offered vector has -become durable. Backend-returned `deferred` batches follow the same wake discipline, preventing an -unrelated database write stream from repeatedly probing a saturated native queue. +Accepted-but-not-durable progress is capped at 64 cursor-advancing batches by default +(`maxAcceptedBatchesAhead`, settable per registration). At the cap, the runner retains ownership +but stops reading and enters `waiting-durable`; database commit wakes do not retry it. A backend +state-change wake reconciles its cursor and resumes only after a complete offered vector has become +durable. The cap never blocks while a transaction is open, because the durable cursor cannot move +until that transaction closes; memory pressure from an oversized transaction is the backend's +`deferred`. Backend-returned `deferred` batches follow the same wake discipline, preventing an +unrelated database write stream from repeatedly probing a saturated native queue. The cap is a +ceiling on durability lag, not a flush schedule; the schedule is the +[durability cadence](#durability-cadence). The fake Stage 1 backend acknowledges each complete transaction durably before returning `accepted`. This proves the cursor and cross-worker mechanics without pretending to implement the -later Tantivy queue and barrier. +later Tantivy queue and barrier. The native-backend suite adds a queue-and-accept fake whose apply +and barrier run asynchronously, which is the shape a native backend is expected to use. ### Authoritative record resolution @@ -292,8 +302,12 @@ unsupported filesystem mutation. A saved boundary normally makes recreation fail therefore rebuild. Harper does not pin transaction-log retention in Stage 1. A backend lagging past retention rebuilds -when its next exact anchor fails. Runtime status records cursor age and log-floor distance, and emits -one error and one metric per transition to `needs-rebuild`, so this availability loss is visible. +when its next exact anchor fails. `getMetrics()` reports `cursorLagMilliseconds` (latest transaction this runner has read minus the +durable position, per log — it cannot see transactions the runner has not read, so a parked +runner's lag is reported through `stalledMilliseconds`, the time spent on backend backpressure or +the durability ceiling) separately from backend backpressure (`deferredBytes`, the `deferred` +status) so retention lag and queue memory pressure are distinguishable, and the runtime emits one error per transition to `needs-rebuild`, so this +availability loss is visible. Writer backpressure above a lag threshold is the opt-in [lag policy](#lag-policy). ### Backend boundary @@ -314,42 +328,94 @@ type DerivedIndexTransaction = { type DerivedIndexBatch = { ownerEpoch: bigint; transactions: DerivedIndexTransaction[]; - through: DerivedIndexCursor; + records: DerivedIndexMutation[]; // coalesced last-write-wins view; non-enumerable + through?: DerivedIndexCursor; // absent only on a rebuild scan chunk + bytes: number; // estimated payload bytes; non-enumerable + rebuild?: true; }; type DerivedIndexMutation = { tableId: number; recordId: Id; logVersion: number; - state: { kind: 'record'; version: number; projection: unknown } | { kind: 'absent' }; + state: + | { kind: 'record'; version: number; projection: unknown } + | { kind: 'absent' } + | { kind: 'unindexable'; version: number; reason: string }; }; +interface DerivedIndexBackendHost { + isOwnerEpoch(epoch: bigint): boolean; + getReadiness(): DerivedIndexReadiness; +} + interface DerivedIndexBackend { readonly id: string; + attach(host: DerivedIndexBackendHost): void; // the epoch fence getDurableCursor(): DerivedIndexCursor | undefined; - deliver(batch: DerivedIndexBatch): DerivedIndexDeliveryResult; + deliver(batch: DerivedIndexBatch): DerivedIndexDeliveryResult; // enqueues; may apply asynchronously + flush(reason: 'age' | 'threshold' | 'shutdown'): void | Promise; // the barrier request + shutdown(ownerEpoch: bigint): void | Promise; // the quiescence handshake onStateChange(wake: (change?: 'changed' | 'accepted-work-lost' | 'failed') => void): () => void; + reset?(ownerEpoch: bigint): void | Promise; } type DerivedIndexRegistration = { backend: DerivedIndexBackend; projections: ReadonlyMap unknown>; + options?: DerivedIndexRunnerOptions; // per-index turn, chunk, cadence and rebuild bounds }; ``` +`records` and `bytes` are non-enumerable properties so the enumerable batch shape stays the Stage 1 +`{ ownerEpoch, transactions, through }` contract; a backend reads them like any other field. There is +one backend contract, and it assumes **asynchronous effects** — work or publication that survives a +method return — because that, not the apply style, is what can outlive an ownership handoff. Every +backend therefore provides the epoch fence (`attach`), the barrier request (`flush`) and the +quiescence handshake (`shutdown`); registration rejects one that does not. A backend that happens to +apply and make each batch durable inside `deliver()` implements them trivially, which costs nothing +and leaves no backend able to compile without the fence. Release awaits the shutdown flush and any +in-flight reset before quiescing the epoch and unlocking. `reset` is optional: a backend that omits +it keeps Stage 1's terminal `needs-rebuild`; a backend that implements it owns its crash safety — its +first durable action must invalidate the cursor or its generation before anything destructive, so an +interrupted reset reopens as cursorless rather than as a valid cursor over partially destroyed state +(shared readiness is process memory and is no evidence after a restart). A condemnation is therefore also written to the root store under the index's marker +key (`derived-index::condemned`, through the audit store's symbol-keyed `putSync`, so its +durability follows the root store's WAL setting): a process that restarts after condemning a +cursor but before the rebuild's `reset` has durably invalidated it finds the marker on acquisition +and rebuilds instead of trusting the still-format-valid cursor. The marker clears only at the first +durable `ready` after the rebuild, so a crash before that costs one extra rebuild. A marker that +cannot be written issues no `reset` — without it a crash mid-reset would reopen on the condemned +cursor: the condemnation stays a shared `needs-rebuild`, the lock is released, and whichever runner +acquires next retries the write before any reset, spending no rebuild attempt on it; a marker that +cannot be read counts as present, and the condemnation that follows still has to reach the store. +Residual: a condemnation the root store never accepted before a process restart is lost with the +process, the same exposure as any root-store metadata write under a full disk. A backend that cannot rebuild parks on the marker until `requestRebuild` or a +rebuild-capable registration. Registration that fails part-way (an `attach` or `onStateChange` that throws) leaves +no readiness subscription or table admission behind. + `DerivedIndexRegistration` belongs to Harper. Its projection functions are compiled from schema attributes and execute before `deliver()`, so the backend receives only its declared materialized view. They are not customer callbacks. `DerivedIndexDeliveryResult` uses the exported numeric constants `DERIVED_INDEX_ACCEPTED`, `DERIVED_INDEX_DEFERRED`, and `DERIVED_INDEX_FAILED` rather than allocating result objects on the -drain path. A `deliver()` call is included in the runner's wall-time budget and may not wait on a -writer mutex or merge. `accepted` means the backend owns the batch; it does not authorize durable -cursor advancement until the backend's barrier includes its `through` vector. `deferred` preserves -the batch and requires a later state-change wake. A backend reports discarded accepted-but-not- -durable queue contents as `accepted-work-lost`, causing the current owner to reconstruct from the -durable cursor. A permanent failure transitions the backend to `needs-rebuild` and emits one -contextual error outside the write path. +drain path. A `deliver()` call runs after the runner's wall-time budget has been spent on collection +and resolution and is **not** bounded by it: the runtime cannot bound work it does not perform, and a +native backend cannot honour a 5 ms budget for a batch of milliseconds-per-mutation applies. The +shape a native backend is expected to use is therefore: accept the batch into its own queue and +return `DERIVED_INDEX_ACCEPTED` immediately; apply asynchronously in its own bounded time slices; +advance `getDurableCursor()` at its own barrier; and return `DERIVED_INDEX_DEFERRED` when its queue +is full. `deliver()` may not wait on a writer mutex or merge. `accepted` means the backend owns the +batch; it does not authorize durable cursor advancement until the backend's barrier includes its +`through` vector. `deferred` preserves the batch and requires a later state-change wake. A backend +reports discarded accepted-but-not-durable queue contents as `accepted-work-lost`, causing the +current owner to reconstruct from the durable cursor. A permanent failure reported as `'failed'` or +`DERIVED_INDEX_FAILED` transitions the backend to `needs-rebuild`, emits one contextual error outside +the write path, and enters the bounded [rebuild](#rebuild-as-a-runtime-phase) when the backend +supports it. A record the backend itself cannot index (a malformed vector, an oversized document) is +never a permanent failure: the backend skips it, counts it, and continues, so the same record cannot +abort every future rebuild. The cursor remains backend-owned because its atomic durability mechanism differs by engine: Tantivy includes it in published index state, while a future HNSW implementation stores it with the @@ -365,24 +431,319 @@ flowchart TD W --> L{acquire backend runner lock?} L -->|no| W L -->|yes| Q[merge and validate exact anchors in-stream] - Q --> T[discard anchors, assemble bounded complete transactions] - T --> P[resolve and project current state] + Q --> T[collect bounded transaction identities] + T --> P[resolve and project each distinct key once] P --> O{backend outcome} O -->|accepted| N[advance offered progress] - O -->|deferred| H[hold backend at cursor] + O -->|deferred| H[hold chunk at cursor] O -->|failed| R N --> D{backend barrier durable?} D -->|later| C[persist durable cursor and wake] D -->|not yet| T C --> T H --> W + R --> X{backend has reset and a record scan?} + X -->|no| Z[terminal: release lock, index unavailable] + X -->|yes| Y[publish rebuilding, shutdown old epoch, reset, scan, boundary, replay] + Y -->|ready after final barrier| T + Y -->|failure| K{attempts below cap?} + K -->|yes, after backoff| Y + K -->|no| U[publish unavailable, release lock] ``` +## Native-backend readiness + +The additions below make the runtime safe and efficient for a backend whose apply costs 0.2–1.4 ms +of synchronous CPU per mutation and whose durability barrier is an `msync` (measured on the HNSW +native plane at 384 dimensions: apply 0.356 ms/mutation, barrier 13.5 ms/call independent of delta +size, event loop blocked 62 ms max during ordinary ingest). They are runtime changes only; nothing +here is specific to one backend. + +### Coalesced delivery view + +`DerivedIndexBatch.records` lists each distinct `(tableId, writeKeyId(recordId))` of the batch once, +in first-occurrence order, with the last `logVersion` in batch order and the same resolved `state` +object its occurrences in `transactions` carry. `transactions` is unchanged, so consumers that need +per-transaction metadata keep it; a backend that materializes latest state iterates `records`. The +primary read was already shared; what the view removes is the per-occurrence mutation wrapper and +the repeated backend apply. Bench (`derivedIndexRuntime.bench.js`, 350 µs/apply): 1000 transactions +over 50 keys in one window cost 1000 applies / 362 ms through `transactions` and 50 applies / 26 ms +through `records`. + +### Bounded collection, resolution and delivery + +A drain turn has two phases. **Collection** reads transaction identities from the log iterator — +`(tableId, recordId, logVersion)` per eligible entry, no primary read — until one of the turn budgets +is met: `maxTransactionsPerTurn` and `maxBytesPerTurn` (log entry bytes) between complete +transactions, `maxMillisecondsPerTurn` after any entry, and `maxChunkRecords` distinct keys +(default 4096) after any entry. **Resolution** then reads the current primary entry once per distinct +key and projects it, checking `maxChunkBytes` and the same wall-time budget after every key; when +either is reached with at least one record already in the chunk, the remaining collected identities +(including the rest of the transaction being resolved) are carried to the next turn, so neither +phase can hold the event loop for more than one budget plus one record. Resolution happens after +every collected occurrence of the key has been read, so the delivered state is never older than a +log entry the batch's cursor certifies; a concurrent writer that commits between the two +occurrences of a key is reflected, not skipped. This ordering is the reason resolution is not done +inline as entries are read. + +An oversized transaction — one that meets the record or time bound before its `endTxn`, or whose +resolution meets the byte or time bound — is delivered across several chunks: each carries the +transaction's `logName` and `timestamp` with the mutations resolved so far, `through` stays at the +last complete transaction, and the unread or unresolved remainder is carried to the next turn. The +chunk that delivers the transaction's last key after its `endTxn` was read advances `through`. +Nothing marks such a chunk: a backend cannot act on the distinction, and the cursor already says +what is certified. Query-visible atomicity of one transaction across chunks is not promised. A +chunk that advances no cursor is accepted work whose durability the next cursor-advancing batch +certifies; it does not count against `maxAcceptedBatchesAhead`, and the backend bounds its memory +with `deferred`, which the runtime honours by holding the chunk until a backend wake. A key repeated +across chunks is resolved again (idempotent latest state); repeats within a chunk are coalesced. + +Payload bytes are an **estimate, not an admission bound**: each resolved record contributes its +stored size when the resolver reports one (`DerivedIndexRecord.size`; the projection is a subset of +the record) and the log entry's size otherwise, and nothing is serialized to compute it. The hard +bound on a chunk is `maxChunkRecords`; `maxChunkBytes` (default 4 MiB) stops resolution once the +estimate is reached, carrying the remaining collected identities to the next turn. `getMetrics()` +reports deferred (held chunk) and accepted-not-durable bytes. + +All bounds are settable per `DerivedIndexRegistration.options`, falling back to the runtime-wide +values, because a vector backend and a full-text backend want different turn sizes. + +### Durability cadence + +`maxAcceptedBatchesAhead` is a ceiling; the schedule below is what obliges a backend to flush. The +runtime is the scheduler because it already tracks accepted-not-durable work; the backend supplies +the barrier through the `flush(reason)` request and reports completion through the +existing `onStateChange` wake. `flush` is a request, not a barrier call: the backend runs it +asynchronously, coalesces requests that arrive while a barrier is in flight into one following +barrier, and publishes the `through` vector atomically with the state that barrier makes durable. + +| trigger | option | default | +| ------------------------------------------------------- | ------------------------- | ------- | +| first accepted batch since the last request is this old | `maxFlushAgeMilliseconds` | 1000 ms | +| accepted mutations since the last request reach | `flushAfterMutations` | 4096 | +| accepted estimated bytes since the last request reach | `flushAfterBytes` | 8 MiB | +| runner release or runtime stop | always (`'shutdown'`) | | + +The age timer is armed by the first accepted batch after a request and re-armed after every +request while accepted work is not yet durable, so an isolated write becomes durable within +`maxFlushAgeMilliseconds`, a burst amortizes to one barrier per threshold, and a backend that +coalesced a request into a barrier already running is asked again rather than left with a +non-durable tail. Idle completion is the age timer: reaching the end of the log does not request an +extra barrier, because arrivals spaced just beyond drain completion would otherwise pay one barrier +per write. +`getMetrics().oldestAcceptedAgeMilliseconds` exposes a backend that ignores requests. Bench at +1500 arrivals/s over 200 keys (5 ms barrier): flush-every-batch gives write→durable p50 21 ms with +281 barriers in 3 s; `maxFlushAgeMilliseconds: 100` / `flushAfterMutations: 512` gives p50 73 ms +with 29 barriers; the defaults give p50 1.27 s with 4 barriers. A backend chooses through its +registration options. + +At release the runner calls `flush('shutdown')` and then `shutdown(epoch)`, and unlocks only after +that settles. + +### Rebuild as a runtime phase + +When the backend implements `reset(ownerEpoch)` and the runtime was constructed with +`scanRecords(tableId)`, `needs-rebuild` is no longer terminal. The lock holder runs, with an +ownership check after every `await`: + +1. publish shared readiness `rebuilding` — before anything destructive; +2. `await backend.shutdown(previousEpoch)` so work accepted under the previous epoch is quiescent, + then mint a new owner epoch, republish `rebuilding` under it, and `backend.reset(newEpoch)`; + afterwards `getDurableCursor()` must be `undefined`; +3. capture the **committed tail**: for every physical log, the last complete committed transaction + the committed reader yields (`getRange({ log, start: 0 })` walked to its end); a log with no + committed transaction is omitted and must retain its beginning (`oldestSequenceNumber === 1`), + otherwise the attempt fails closed; +4. scan every registered table through `scanRecords` (opened after the capture; a record whose + `value` is null is a tombstone and one whose key is a symbol is a Harper-internal store entry + such as id allocation — both are skipped; on the live path only a missing entry resolves to + `absent`, and a present entry that decodes to null reaches the projection and fails closed), + project, and deliver chunks bounded by `maxChunkRecords`, `maxChunkBytes` and `maxMillisecondsPerTurn` with + `through` absent, yielding between chunks and waiting for a backend wake on `deferred` + (re-offering after one flush age at most, so a dropped wake cannot park the rebuild forever); +5. deliver one final chunk (possibly empty) carrying `through` = boundary. Until that batch is + durable the backend's cursor stays `undefined`, so a crash mid-rebuild resumes as a fresh + rebuild rather than a partial index with a certified cursor; +6. install the boundary as offered progress and open the log iterator from it with the existing + `exactStart` / `resumeAfterExactStart` validation (the anchor transaction is already reflected + in the scan because it committed before the capture), replay to the head through the ordinary + drain, and publish `ready` on the first durable advance past the boundary or the first idle + pass whose durable cursor equals offered progress, whichever comes first — under sustained + ingest there may never be an idle pass, and a durable advance already certifies a complete + prefix. + +The tail is a safe anchor because a committed read is a contiguous physical prefix. rocksdb-js keeps +the physically-written-but-uncommitted start offsets in a sorted set and advances +`lastCommittedPosition` only to the earliest of them (`TransactionLogStore::commitFinished`, +`uncommittedTransactionPositions.front()`): a transaction that wrote at offset 200 and committed +before one still pending at offset 100 stays invisible until 100 commits, so nothing committed after +the capture can sit behind the captured tail. Everything committed before the tail is in the scan +(the scan reads current records after the capture); everything after it is replayed. A `reload` +marker is therefore met exactly once — the one that triggered the rebuild is behind the tail, and +one committed during the scan is replayed and demands its own rebuild — with no capture-time +bookkeeping and no clock comparison. Walking a log to its tail decodes every retained frame once, +without record resolution or projection; the rocksdb-js follow-up that exposes the committed +`(logId, offset)` per log makes the capture O(1). An interior corrupt frame stops the walk and fails +the attempt closed: a log that cannot be read to its committed tail cannot be replayed from any +anchor. + +Failure anywhere in the phase, or a `'failed'` report before the index reaches `ready`, retries +with capped exponential backoff (`rebuildBackoffMilliseconds` 1 s doubling to +`maxRebuildBackoffMilliseconds` 5 min) while holding the lock. After `maxRebuildAttempts` (8) +consecutive attempts the index publishes `unavailable` with the reason, releases the lock, and +stops; the attempt count travels in the shared readiness record so a peer that acquires afterwards +honours the exhausted budget instead of starting its own. Only `requestRebuild(backendId)` or +reaching `ready` resets it. An owner that acquires while the shared state is `needs-rebuild` or +`rebuilding` rebuilds rather than trusting a format-valid durable cursor: a previous owner +condemned that generation. `requestRebuild` from a non-owning worker sets a request word in the +shared record that the owner consumes on its next drain turn and any acquisition consumes first, +so a request reaches an owner that never idles or is parked on backpressure or backoff (the +requesting worker also notifies the buffer, which wakes the owner directly, and the word bypasses +those wake gates at the owner's next wake of any kind); a request arriving during a +rebuild is absorbed by it (the rebuild consumes the word when it starts and again when it +completes). A non-owning worker never writes the readiness record itself: only the lock holder +publishes. A budget a previous owner already exhausted is honoured without one more attempt, and +a backend that cannot rebuild parks on a condemned generation instead of resuming from its cursor. A projection that throws a 4xx-classified error (`ClientError`) for one +record yields `state: { kind: 'unindexable' }` whose `reason` is the error's class and status only, +never its message (validation messages can quote record values) — the backend removes any entry and +counts it — in live delivery and rebuild alike, so one malformed record cannot loop a rebuild; any +other exception stays fail-closed. A chunk, or a whole scan, in which the projection rejected every +record is not a fault: a bulk write of records that lack the projected attribute, or a table indexed +before its data carries it, is a legitimate state, and condemning it would turn user data into an +outage that the rebuild's replay re-trips. The runtime logs one warning per such streak and the +count stays visible in `unindexableRecords`. + +### Generation fencing and cancellation + +Asynchronous acceptance makes ownership handoff unsafe without a handshake: worker A can accept a +batch, release the lock, and later run a scheduled apply or a flush completion after worker B has +reset the index. Three mechanisms close it: + +- **Shutdown before unlock.** `#release()` drops ownership immediately, calls `flush('shutdown')` + then `shutdown(epoch)`, and unlocks only when that settles. One `shutdown` runs per epoch: a + release that overlaps a rebuild attempt's own quiescence shares its promise instead of calling + the backend twice. A rejected `shutdown` keeps the lock and publishes `unavailable` with the + reason: a backend that cannot prove its queue is quiescent must not hand the index to another + owner. `DerivedIndexRuntime.stop()` and the unregister function return one cached promise per + runtime or runner — repeated calls return the same promise — that resolves after every backend + settled (a still-draining backend is waited for even when another already failed) and + **rejects** when any shutdown failed, so a caller cannot close storage on a fulfilled promise + while a backend is still draining into it. A release that overlaps an in-flight `reset` waits for + the reset before quiescing and unlocking. Table registrations are released only after the + runner's backend settled, so an eviction committed during the drain still writes the marker the + next owner replays. The runner that holds a lock after a failed shutdown can be revived by + `requestRebuild`, which resumes under the same epoch and retries that epoch's quiescence before + minting a successor and resetting; a stopped or unregistered runner in that state stays reachable + through the runtime's `requestRebuild`, which retries the release so a re-registered runner can + acquire. There is no automatic bounded hold. A failed shutdown stays in the runtime-wide + `stop()` wait, so a later `stop()` keeps reporting it. +- **Epoch fence.** `attach(host)` gives the backend `isOwnerEpoch(epoch)`, an `Atomics` read of the + shared owner-epoch counter. The backend checks it before each apply, after each await, and in + flush completions; a completion for a superseded epoch is dropped. Each rebuild attempt mints a + new epoch after quiescing the previous one, so a late completion from a failed attempt cannot + pass the fence either. +- **Runtime checks.** The runner tracks a generation that changes on every acquisition, discard, + reset and release, and ignores any delivery result, wake or `await` continuation that belongs to + an earlier generation. + +### Shared cross-worker readiness + +`indexStore.isIndexing` is per worker and `getStatus()` is only meaningful on the owner. The owner +publishes readiness — `ready`, `rebuilding`, `needs-rebuild` or `unavailable`, a reason **code**, and +the rebuild-attempt count — into one small shared buffer per backend (`getUserSharedBuffer`, +`READINESS_BYTES`): five independently read `Int32` words (state, reason, attempts, rebuild +request, lag exceeded) followed by the `BigInt64` owner-epoch counter that `#mintEpoch` increments +and `isOwnerEpoch` compares against. Each word is self-consistent on its own and nothing needs to +observe two of them atomically, so there is no sequence lock: the owner stores reason and attempts +before state, and a reader that sees a new state sees values at least as new. +`DerivedIndexRuntime.getReadiness(id)` and the exported `readDerivedIndexReadiness(logStore, id)` +read it with four `Atomics.load`s on any worker, so a query path can choose between a 503 and a +stale-but-usable answer without holding the runner lock. `unknown` means no runtime in this process +has evaluated the index yet. The shared reason is one of `DerivedIndexReadinessReason` — never a +message, because backend and validation messages can quote record content; the full message stays +in the owner's local status and log, and a successor that parks on an inherited condemnation reports +the code it inherited. A runner that latched a shared `unavailable` drops the latch on its next wake +once the shared state has moved on, so a peer's revival does not strand the other workers. `ready` is +published on a validated acquisition and after a rebuild's final barrier; `rebuilding` before the +destructive reset. rocksdb-js (`DBDescriptor::getUserSharedBuffer`) copies the default buffer into +one native allocation per key on the first call and, on every later call from any thread, wraps that +same allocation in a new external `ArrayBuffer`; it never returns a `SharedArrayBuffer`, never +re-seeds an existing entry, and keeps the allocation while any wrapper is alive. The runtime +therefore fetches the view once per runner and holds it. `Atomics` over these wrappers is the same +dependency Harper's primary-key allocation (`Table.ts`), blob hold table (`blob.ts`) and HNSW node-id +allocation already carry, and the rocksdb-js README documents it as the intended use; wakes go +through the binding's own `notify()`, so nothing here needs `Atomics.wait`. A fault detected in the +middle of a drain turn (a corrupt frame surfacing from the iterator) starts the rebuild from inside +that turn; the turn's generation check prevents its end-of-log path from publishing `ready` over the +`rebuilding` just written. + +### Lag policy + +Opt-in writer backpressure, per registration (`maxLagMilliseconds`, 0 = no policy; a budget below +two flush ages is raised to that, since catch-up is only proven at a durable barrier). The owner +measures lag as the longest of four terms — cursor distance behind what it has read, time parked +on backpressure or the durability ceiling, time since the oldest commit it may not have read yet +(cleared each time a drain reaches the end of the log, and bounded by how far the newest entry it +has read trails the clock, so a reader that never quite empties a steadily fed log is behind by +that distance rather than by the age of its first unread commit), and the age of the oldest accepted work the +backend has not yet made durable — because a reader too slow to reach the end of the log cannot +hide from the third term and a backend that accepts but never barriers cannot hide from the +fourth, while a caught-up owner sitting idle reports zero and a runner keeping up under sustained +ingest stays within drain latency plus flush age. Catch-up is proven by a durable advance or an +idle pass with durable == offered. It +samples on every drain turn, idle pass and age tick and on its own lag timer while parked, and +publishes a lag-exceeded word in the shared readiness buffer: set at `lag >= budget`, cleared only +once this owner has proved catch-up — a durable advance and the end of the log both reached since it +acquired — and lag is below half the budget, so the policy neither flaps nor clears on an ownership +handoff before the successor has drained the inherited backlog. Discarding progress drops the +accepted work with it, so a rebuild or lost accepted work does not turn the owner's age into +fabricated lag. The word is owned by whoever holds the runner lock, and every transition out of +"behind and still reading" clears it: entering a rebuild (there is no durable cursor to guard, and +the replay anchor is captured fresh; readers act on `rebuilding`), becoming `unavailable`, parking +in a `needs-rebuild` the runtime cannot leave (no `reset` or no `scanRecords`), a condemnation +marker that could not be written (its retry needs a commit wake, and commits were what was being +shed), and a failed shutdown holding the lock. An ordinary handoff preserves it, so a successor +cannot admit writes before proving catch-up itself. + +Every worker's runner registers an admission check for the index's tables +(`registerDerivedIndexTables(store, tableIds, admission)`); `derivedIndexWriteRejection(store, +tableId)` costs one WeakMap miss on tables without a derived index and one `Atomics.load` per +policy-enabled index otherwise. The check sits at the staging layer — `_writeUpdate`, +`_writeDelete`, `_writeInvalidate` and `_writeRelocate` — where every local write converges — put, patch, post and `create()`, +`loadAsInstance: false` writes, held-lock saves, and per-row query deletes — and it bypasses +canonical-source applies (`transaction.sourceApply`: replication peers and external caching +sources), crash-recovery replay (`transaction.isReplay`) and replication notifications +(`options.isNotification`), because a rejected canonical write would advance the source cursor +past a write that never landed; origin cache fills call `updateRecord` directly and are not gated. +The check is one function call, one `WeakMap` lookup and no allocation on every local write, +including tables with no derived index — low, not zero. A shed write fails +with `DerivedIndexLagError`, a `ServerError` with status 503 and `code: 'DERIVED_INDEX_LAGGING'`; +the status and code are the wire contract on every surface, while `retryable: true` is carried on +the error object and serialized only where a surface already serializes it. + +What the policy guarantees is that local user writes are shed after the configured budget. It does +not by itself guarantee the cursor is never lost: the budget is chosen by the registration, audit +retention and emergency storage reclamation are configured separately, and replicated writes keep +advancing the log; a registration should keep its budget well inside the effective retention +window. Why not pin retention to the slowest cursor: rocksdb-js has no protected-position +registration (`purgeLogs()` is time/name filtered and configured retention applies independently). +Why not metrics only: sustained overload runs the cursor past retention, rebuilds, and falls behind +again; the bounded retry makes that loop finite, not harmless. Rollout is two-phase: deploy with +the policy off, confirm every worker runs the new runtime (an old worker has no admission check and +keeps accepting writes after a new owner trips), then enable it per registration. + +Regardless of the policy, lag and backpressure remain separate observable signals +(`cursorLagMilliseconds` and `stalledMilliseconds` versus `deferredBytes` and the `deferred` +status), a `'failed'` report and every rebuild failure go through the bounded retry above, and +the runtime never lets an exception from a backend call escape the scheduled drain. + ## Approaches considered **Invariant:** every committed mutation relevant to a derived index is eventually reflected in the index, or that index is explicitly unavailable pending rebuild; cursor advancement can never hide -unapplied work. +unapplied work. For the native-backend additions: a published cursor certifies a complete durable +prefix for that index generation, every committed change outside that prefix stays replayable, and +no work accepted under one owner epoch is applied or published once another epoch owns the index. ### Different layer @@ -444,6 +805,70 @@ common aligned case without pinning healthy indexes to the slowest cursor. Stage backend and no measured N-scan bottleneck, so it establishes the independent runner fallback first; the cursor and batch contracts do not prevent adding cohorts after comparative benchmarks. +### Native-backend additions + +Two implementations of #2489's protocol existed in parallel: this runtime and the HNSW-shaped +runtime on the native-plane branch. The additions above converge on this one. + +**Different layer.** Keep both runtimes and share only the transaction-log reader changes. +Rejected: ownership election, exact-cursor validation and recovery stay duplicated, which is the +failure #2489 exists to prevent. + +**Deeper cause.** Move native insertion onto a thread pool so delivery cost stops being event-loop +cost. It is the deeper fix for the event-loop term, but the installed native package exposes only a +synchronous insert, it removes neither the duplicated runtime nor the repeated-key work, and it is a +later phase of the native plane. A second deeper-cause candidate, raised by the planning review and +**adopted**: collect bounded mutation identities first and resolve each key after its last collected +occurrence, instead of resolving on first encounter and reusing the result. The first draft resolved +inline, and a concurrent writer committing between two occurrences of a key inside one chunk would +have had its later state certified by the cursor while the earlier state stayed indexed — permanently, +since replay skips both. Identity-first collection prevents that state rather than detecting it, at +no extra primary reads. + +**Do less.** Adopt the runtime unchanged and put coalescing, chunking and rebuild inside each +backend. Rejected for rebuild, which is not backend-specific (reset → scan → project → deliver → +replay is identical for full-text and vector backends) and whose duplication is how two runtimes came +to exist. Coalescing in the runtime only removes repeated wrappers and applies, since the primary read +was already shared — still worth doing once. Within bounded delivery, "collect an oversized +transaction across yields but deliver it whole" was the do-less candidate and is rejected because it +materializes the whole resolved payload (100k records × a 1.5 KB projection is 150 MB) before the +backend can defer. Timer-coalesced idle flushing, also raised by the planning review and **adopted**, +is the do-less form of idle completion: an immediate barrier at every idle pass would cost one barrier +per write for arrivals spaced just beyond drain completion. + +**Different layer, revisited (adopted from two planning rechecks, then simplified).** Enforce the +handoff invariant at the backend contract rather than by documentation: the fence, barrier request +and quiescence handshake are checked at registration. The rechecks first split the contract on an +`asynchronous` discriminant so a backend with no asynchronous effects could omit the hooks; that +split was removed once it was clear no shipped or planned backend is synchronous — the HNSW plane +and Tantivy both queue and barrier — and the split cost a registration-time validation branch, a +runtime check that a "synchronous" `flush` did not return a promise, and a quiescence path for that +undeclared promise. One contract, three required hooks, trivially implemented by a fake that +completes in `deliver()`. Their remaining suggestions were declined on facts: a commit-time +admission recheck only shrinks a staging-to-commit window that the budget and hysteresis already +dwarf; a native-backend restart test needs a native backend, which #2430 owns. + +**Different layer, for the lag policy (adopted from its planning gate).** Gate at the staging layer +(`_writeUpdate` / `_writeDelete`) rather than at the public verbs: `create()`, `loadAsInstance: +false` writes and held-lock saves reach the staging layer without passing `update()`, and +replication already marks its writes (`isNotification`) there. Gating `updateRecord` itself was +rejected because origin cache fills share it. + +**Chosen.** Coalesced view, identity-first bounded collection with no cursor publication +mid-transaction, runtime-scheduled durability cadence with the age timer as idle completion, rebuild +phase anchored at the committed tail with bounded retry and an observable `unavailable` end state +carried across owners, shutdown-before-unlock plus a shared epoch fence, and plain-word shared +readiness with reason codes. The first draft anchored the rebuild at the oldest retained entry and +suppressed already-covered reload markers by capture time; reading rocksdb-js's commit watermark +(above) showed the tail is a contiguous-prefix boundary, which removed the whole-log replay, the +capture clock, the shared reload word and the backward-clock residual together. Excluded: a tighter +anchor from staged or uncommitted positions (unnecessary once the tail is safe) and the transactional +dirty-key outbox, rejected on the facts under _Deeper cause_ above: a second durable write plus a +compaction stream and cleanup protocol on every indexed mutation, a new column family and therefore +a storage-format migration for every audited table, and no ability to commit an engine-specific +native file (an mmap plane) atomically with RocksDB in any case, so the cursor protocol would still +be needed. + ## Verification ### Correctness @@ -507,8 +932,31 @@ the cursor and batch contracts do not prevent adding cohorts after comparative b iterator reuse, native enqueue throughput, deferred recovery, and independent runner parallelism. - Benchmark aligned and intentionally divergent index cursors before considering a shared scan cohort; no cohort optimization is part of Stage 1. -- Exercise a large transaction to prove memory is bounded by the configured drain batch plus one - complete transaction, not by all writes retained until commit. +- Exercise a large transaction to prove memory is bounded by the configured chunk, not by the whole + transaction: the native-backend suite delivers one transaction across chunks that advance no + cursor, lets the backend defer after the first chunk, and checks that only the closing chunk + advances `through`. + +### Native-backend suite and bench + +`unitTests/resources/derivedIndexRuntimeNativeBackend.test.js` proves, with a queue-and-accept fake +backend: coalescing with the last `logVersion` and preserved key identity while `transactions` and +the enumerable batch shape are unchanged; resolution after the last collected occurrence under a +concurrent write; oversized-transaction chunking with deferral; per-registration option override; +flush requests by threshold, age and shutdown; a rebuild driven through reset → scan → boundary → +replay with `rebuilding` observed before the reset and `ready` only after the final barrier; +ownership handoff while an apply is scheduled and while a flush is pending, with the old epoch +fenced; a rejected shutdown holding the lock; a non-owning worker reading the shared readiness; a +a 4xx projection rejection delivered as +`unindexable`; a backend failing every rebuild settling into `unavailable` with the budget honoured +by a peer and revived by `requestRebuild`; a boundary lost to retention during the scan; and a +reload marker handled once. Every test asserts no unhandled rejection. + +`unitTests/resources/derivedIndexRuntime.bench.js` (excluded from `test:unit:resources`) reports the +numbers quoted in the sections above: applies and wall time with and without the coalesced view, +event-loop delay for inline versus queued application of a 5,000-mutation window (964.8 ms max +inline, 7.0 ms max queued, same throughput), and write→durable latency, indexed throughput, peak +queued bytes and barrier count for independently paced arrivals under three cadences. ### Repository gates diff --git a/integrationTests/components/branched-database-schema-gate.test.ts b/integrationTests/components/branched-database-schema-gate.test.ts deleted file mode 100644 index dfe975794a..0000000000 --- a/integrationTests/components/branched-database-schema-gate.test.ts +++ /dev/null @@ -1,71 +0,0 @@ -/** - * A branched application cannot declare its schema into the base (harper#643). - * - * GraphQL `@table`, `scope.ensureTable` and `defineTable` all register in the process-wide catalog. - * For a branched name that means the table would be created in the BASE — replicated, visible to - * every other application, and bound to this application's own REST routes — while the application's - * own JavaScript read and wrote its branch. It is refused until harper#2264 makes these land in the - * branch instead. - * - * Reproduction: - * npm run test:integration -- "integrationTests/components/branched-database-schema-gate.test.ts" - */ -import { suite, test, before, after } from 'node:test'; -import { ok } from 'node:assert'; -import { resolve, join, basename } from 'node:path'; -import { cp, mkdtemp } from 'node:fs/promises'; -import { tmpdir } from 'node:os'; - -import { - startHarper, - killHarper, - teardownHarper, - sendOperation, - type ContextWithHarper, -} from '@harperfast/integration-testing'; - -const FIXTURE_PATH = resolve(import.meta.dirname, 'fixtures/branched-database-gated'); - -// Which databases an application forks is a deployment decision, so it is declared on the -// application's root-config entry, next to host/urlPath — not in its own config.yaml. -const BRANCHED = { config: { 'branched-database-gated': { branchedDatabases: ['data'] } } }; - -suite('a branched application declaring a table in the base', (ctx: ContextWithHarper) => { - before(async () => { - const dataRootDir = await mkdtemp( - join(process.env.HARPER_INTEGRATION_TEST_INSTALL_PARENT_DIR || tmpdir(), 'harper-integration-test-') - ); - ctx.harper = { dataRootDir } as any; - await startHarper(ctx, BRANCHED); - await sendOperation(ctx.harper, { operation: 'create_database', database: 'data' }); - await sendOperation(ctx.harper, { - operation: 'create_table', - database: 'data', - table: 'Branched', - primary_key: 'id', - }); - await killHarper(ctx); - await cp(FIXTURE_PATH, join(dataRootDir, 'components', basename(FIXTURE_PATH)), { - recursive: true, - dereference: true, - }); - await startHarper(ctx, BRANCHED); - }); - - after(async () => { - await teardownHarper(ctx); - }); - - test('is refused, so the base schema is untouched', async () => { - // The fixture's schema.graphql declares `GatedByBranch @table(database: "data")` and the - // application branches `data`. Dropping `branchedDatabases` from that same fixture is what - // makes this table appear here, so the assertion is about the gate and not about the - // application failing to load for some other reason. - const described = await sendOperation(ctx.harper, { operation: 'describe_database', database: 'data' }); - ok(!Object.keys(described).includes('GatedByBranch'), 'a branched application must not create a base table'); - - // It still loads and still reaches its branch: the refusal is scoped to the declaration. - const probe = await sendOperation(ctx.harper, { operation: 'branch_probe', id: 'anything' }); - ok(probe.found === false, 'the application itself is still running against its branch'); - }); -}); diff --git a/integrationTests/components/branched-database-schema.test.ts b/integrationTests/components/branched-database-schema.test.ts new file mode 100644 index 0000000000..b4202c5010 --- /dev/null +++ b/integrationTests/components/branched-database-schema.test.ts @@ -0,0 +1,124 @@ +/** + * A branched application's schema lands in its branch (harper#2264). + * + * GraphQL `@table`, `scope.ensureTable` and `defineTable` all register through the application's + * own table factory. For a branched name that is the branch's store, so the table the application + * declares is created in its private fork: served by its exported REST route, reachable through + * `databases.` from `harper`, and absent from the base -- which every other application and the + * operations API still see untouched. + * + * Reproduction: + * npm run test:integration -- "integrationTests/components/branched-database-schema.test.ts" + */ +import { suite, test, before, after } from 'node:test'; +import { ok, rejects, strictEqual } from 'node:assert'; +import { resolve, join, basename } from 'node:path'; +import { cp, mkdtemp } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; + +import { + startHarper, + killHarper, + teardownHarper, + sendOperation, + type ContextWithHarper, +} from '@harperfast/integration-testing'; + +const FIXTURE_PATH = resolve(import.meta.dirname, 'fixtures/branched-database-schema'); + +// Which databases an application forks is a deployment decision, so it is declared on the +// application's root-config entry, next to host/urlPath — not in its own config.yaml. +const BRANCHED = { config: { 'branched-database-schema': { branchedDatabases: ['data'] } } }; + +suite('a branched application declaring a table in schema.graphql', (ctx: ContextWithHarper) => { + let authorization: string; + + before(async () => { + // The base must exist before it can be branched, so it is built on a first start without the + // application, and the restart is what branches it. + const dataRootDir = await mkdtemp( + join(process.env.HARPER_INTEGRATION_TEST_INSTALL_PARENT_DIR || tmpdir(), 'harper-integration-test-') + ); + ctx.harper = { dataRootDir } as any; + await startHarper(ctx, BRANCHED); + await sendOperation(ctx.harper, { operation: 'create_database', database: 'data' }); + await sendOperation(ctx.harper, { + operation: 'create_table', + database: 'data', + table: 'Branched', + primary_key: 'id', + }); + await killHarper(ctx); + await cp(FIXTURE_PATH, join(dataRootDir, 'components', basename(FIXTURE_PATH)), { + recursive: true, + dereference: true, + }); + await startHarper(ctx, BRANCHED); + authorization = `Basic ${Buffer.from(`${ctx.harper.admin.username}:${ctx.harper.admin.password}`).toString('base64')}`; + }); + + after(async () => { + await teardownHarper(ctx); + }); + + test('serves the table through its exported route, out of the branch', async () => { + // The fixture's schema.graphql declares `DeclaredInBranch @table(database: "data") @export`. + const put = await fetch(`${ctx.harper.httpURL}/DeclaredInBranch/via-rest`, { + method: 'PUT', + headers: { 'Content-Type': 'application/json', authorization }, + body: JSON.stringify({ note: 'written through the exported route' }), + }); + strictEqual(put.status, 204, `PUT through the exported route: ${put.status} ${await put.text()}`); + + const get = await fetch(`${ctx.harper.httpURL}/DeclaredInBranch/via-rest`, { headers: { authorization } }); + strictEqual(get.status, 200); + strictEqual(((await get.json()) as any).note, 'written through the exported route'); + + // The same row through `databases.data.DeclaredInBranch` imported from `harper`: the route + // and the binding resolve to one store, the branch. + const probe = await sendOperation(ctx.harper, { + operation: 'branch_probe', + table: 'DeclaredInBranch', + id: 'via-rest', + }); + strictEqual(probe.found, true, 'the imported binding reads what the route wrote'); + strictEqual(probe.note, 'written through the exported route'); + }); + + test('the base database has no such table', async () => { + const described = await sendOperation(ctx.harper, { operation: 'describe_database', database: 'data' }); + ok(!Object.keys(described).includes('DeclaredInBranch'), 'a branched declaration must not create a base table'); + ok(Object.keys(described).includes('Branched'), 'sanity: the base still describes its own table'); + + await rejects( + () => + sendOperation(ctx.harper, { + operation: 'search_by_id', + database: 'data', + table: 'DeclaredInBranch', + ids: ['via-rest'], + get_attributes: ['*'], + }), + /not exist|invalid/i, + 'the operations API resolves the base and must not find the table' + ); + }); + + test("the application's own writes to a base-created table still go to the branch", async () => { + await sendOperation(ctx.harper, { + operation: 'branch_probe', + table: 'Branched', + action: 'put', + id: 'app-only', + note: 'from the application', + }); + const throughBase = await sendOperation(ctx.harper, { + operation: 'search_by_id', + database: 'data', + table: 'Branched', + ids: ['app-only'], + get_attributes: ['*'], + }); + strictEqual(throughBase.length, 0, 'the base never sees the application’s write'); + }); +}); diff --git a/integrationTests/components/branched-database.test.ts b/integrationTests/components/branched-database.test.ts index 26d9e19bfa..a7d390f019 100644 --- a/integrationTests/components/branched-database.test.ts +++ b/integrationTests/components/branched-database.test.ts @@ -8,7 +8,7 @@ * * Two phases, because a branch is a checkpoint: the base must already hold the schema when the * branch is taken. The first start creates `Branched` in the base (the application's own - * `@table` still registers there — scoping that is #2264) and seeds it; the restart is the one + * `@table` would land in its branch, harper#2264) and seeds it; the restart is the one * that branches a base with data in it. * * Reproduction: diff --git a/integrationTests/components/fixtures/branched-database-gated/schema.graphql b/integrationTests/components/fixtures/branched-database-gated/schema.graphql deleted file mode 100644 index 1a94170ec7..0000000000 --- a/integrationTests/components/fixtures/branched-database-gated/schema.graphql +++ /dev/null @@ -1,3 +0,0 @@ -type GatedByBranch @table(database: "data") @export { - id: ID @primaryKey -} diff --git a/integrationTests/components/fixtures/branched-database-gated/config.yaml b/integrationTests/components/fixtures/branched-database-schema/config.yaml similarity index 85% rename from integrationTests/components/fixtures/branched-database-gated/config.yaml rename to integrationTests/components/fixtures/branched-database-schema/config.yaml index 4c07184cb3..d24270537d 100644 --- a/integrationTests/components/fixtures/branched-database-gated/config.yaml +++ b/integrationTests/components/fixtures/branched-database-schema/config.yaml @@ -1,3 +1,5 @@ +rest: true + graphqlSchema: files: '*.graphql' diff --git a/integrationTests/components/fixtures/branched-database-gated/resources.js b/integrationTests/components/fixtures/branched-database-schema/resources.js similarity index 58% rename from integrationTests/components/fixtures/branched-database-gated/resources.js rename to integrationTests/components/fixtures/branched-database-schema/resources.js index ee0aff9291..01f77f26b6 100644 --- a/integrationTests/components/fixtures/branched-database-gated/resources.js +++ b/integrationTests/components/fixtures/branched-database-schema/resources.js @@ -1,8 +1,10 @@ /** - * Fixture for harper#643: an application that declared `branchedDatabases: [data]`. + * Fixture for harper#2264: an application that declared `branchedDatabases: [data]` and declares + * its own table in schema.graphql. * - * It reads and writes `databases.data.Branched` exactly as an unbranched application would — the - * scoped binding the loader hands it is what redirects those names. + * It reads and writes `databases.data.` exactly as an unbranched application would — the + * scoped binding the loader hands it is what redirects those names, and the `@table` declaration + * lands in the same branch that binding resolves. * * The `import` is load-bearing. A branch is delivered through the module loader's `harper` exports, * so the bare `databases` global — which `vm-current-context` shares process-wide and cannot scope — @@ -13,12 +15,12 @@ import { databases, server } from 'harper'; server.registerOperation({ name: 'branch_probe', execute: async function branchProbe(op) { - const { Branched } = databases.data; + const Table = databases.data[op.table ?? 'Branched']; if (op.action === 'put') { - await Branched.put({ id: op.id, note: op.note }); + await Table.put({ id: op.id, note: op.note }); return { wrote: op.id }; } - const record = await Branched.get(op.id); + const record = await Table.get(op.id); return { found: record != null, note: record?.note ?? null }; }, }); diff --git a/integrationTests/components/fixtures/branched-database-schema/schema.graphql b/integrationTests/components/fixtures/branched-database-schema/schema.graphql new file mode 100644 index 0000000000..18de2272d9 --- /dev/null +++ b/integrationTests/components/fixtures/branched-database-schema/schema.graphql @@ -0,0 +1,4 @@ +type DeclaredInBranch @table(database: "data") @export { + id: ID @primaryKey + note: String +} diff --git a/integrationTests/database/eviction-secondary-index.test.ts b/integrationTests/database/eviction-secondary-index.test.ts index 255c30100e..7daa5b16db 100644 --- a/integrationTests/database/eviction-secondary-index.test.ts +++ b/integrationTests/database/eviction-secondary-index.test.ts @@ -222,6 +222,15 @@ suite(`QA-179 TTL eviction sweep vs secondary index [${ENGINE}]`, { skip: skipSu }); // ---- Q1: mid-eviction snapshot — index/base consistent WHILE rows are being evicted ----- + test('a progressing scan survives monitor ticks during the eviction window', async () => { + const response = await fetch(`${httpURL}/PacedDumpP/`, { headers: client.headers }); + const body = await response.text(); + strictEqual(response.status, 200, body); + const ids = JSON.parse(body); + strictEqual(ids.length, ROWS_PERMANENT); + strictEqual(new Set(ids).size, ROWS_PERMANENT); + }); + test('Q1 mid-eviction: index/base stay consistent during the sweep', { timeout: 60_000 }, async () => { // expiration:4s, scanInterval:2s. Rows become evictable at ~t=4s after load; the sweep // fires every ~2s. Sample partway through the sweep (some evicted, some not) so a split diff --git a/integrationTests/database/eviction-secondary-index/resources.js b/integrationTests/database/eviction-secondary-index/resources.js index 143a78027b..74fa88bd35 100644 --- a/integrationTests/database/eviction-secondary-index/resources.js +++ b/integrationTests/database/eviction-secondary-index/resources.js @@ -1,3 +1,5 @@ +import { setTimeout as delay } from 'node:timers/promises'; + // QA-179 — TTL background expiration/eviction sweep vs secondary-index consistency under // long-transaction force-commit. // @@ -71,3 +73,16 @@ export class DumpP extends Resource { return out; } } + +export class PacedDumpP extends Resource { + static loadAsInstance = false; + async get() { + const out = []; + for await (const row of tables.Permanent.search({})) { + out.push(row.id); + // Model a streaming consumer spanning several 5ms idle-monitor ticks. + await delay(2); + } + return out; + } +} diff --git a/integrationTests/database/txnlog-restart-reclaim.test.ts b/integrationTests/database/txnlog-restart-reclaim.test.ts index 9517b7dda1..e68c37c3b6 100644 --- a/integrationTests/database/txnlog-restart-reclaim.test.ts +++ b/integrationTests/database/txnlog-restart-reclaim.test.ts @@ -22,9 +22,9 @@ const FIXTURE_PATH = resolve(import.meta.dirname, 'txnlog-restart-reclaim'); const AUDIT_RETENTION_SECONDS = 2; const AUDIT_RETENTION_MS = AUDIT_RETENTION_SECONDS * 1000; const RETENTION_MARGIN_MS = 500; -const MIN_RECLAIM_BYTES = 64 * 1024 * 1024; +const MIN_RECLAIM_BYTES = 100_000_000; const MIN_RECLAIM_RATIO = 0.75; -const VOLUME_RECORDS = 20_000; +const VOLUME_RECORDS = 24_000; const CHURN_BATCH_RECORDS = 500; const PAYLOAD = 'p'.repeat(5000); const RECLAIM_FILESYSTEM_ROOT = '/dev/shm'; @@ -223,7 +223,8 @@ suite( await startHarper(ctx, { config: CONFIG, env: ENV }); const restartState = await waitForReclaimState(ctx); - strictEqual(restartState.purgeRuns, 1, 'expected the restart purge to be the only cleanup pass'); + // Native recovery and Harper startup can both purge; the filesystem oracle below guards reclamation. + ok(restartState.purgeRuns >= 1, 'expected a startup transaction-log purge'); const postBootLogs = transactionLogsUnder(ctx.harper.dataRootDir); const postBootPaths = new Set(postBootLogs.map((file) => file.path)); diff --git a/integrationTests/resources/ttl-rate-limiter-convergence.test.ts b/integrationTests/resources/ttl-rate-limiter-convergence.test.ts new file mode 100644 index 0000000000..a2eb06e2af --- /dev/null +++ b/integrationTests/resources/ttl-rate-limiter-convergence.test.ts @@ -0,0 +1,214 @@ +/** + * QA-431 companion — same fixture as ttl-rate-limiter-concurrent.test.ts, with the two + * ingredients that make version-reuse cache staleness reproducible: GET pressure DURING each + * burst (cold reads re-seed the VerificationTable), and per-window convergence classification — + * CONVERGED_LATE (a read raced still-landing merges) vs STUCK_SHORT (a durably lost increment). + */ + +import { suite, test, before, after } from 'node:test'; +import { ok } from 'node:assert'; +import { resolve } from 'node:path'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { setupHarperWithFixture, teardownHarper, type ContextWithHarper } from '@harperfast/integration-testing'; +// @ts-expect-error utils/client.mjs has no type declarations +import { createApiClient } from '../apiTests/utils/client.mjs'; + +const FIXTURE_PATH = resolve(import.meta.dirname, 'ttl-rate-limiter-concurrent'); +const WORKERS = 4; +const ROUNDS = 10; +const WINDOWS = 10; +const HITS = 50; + +const skipSuite = process.platform === 'win32' || process.env.HARPER_RUNTIME === 'bun'; + +suite( + `QA-431 convergence [${ROUNDS} rounds × ${WINDOWS} windows × ${HITS} hits, ${WORKERS} workers]`, + { skip: skipSuite }, + (ctx: ContextWithHarper) => { + let httpURL: string; + let auth: string; + + before(async () => { + await setupHarperWithFixture(ctx, FIXTURE_PATH, { + config: { threads: { count: WORKERS } }, + env: {}, + }); + const client = createApiClient(ctx.harper); + httpURL = ctx.harper.httpURL; + auth = client.headers.Authorization; + const deadline = Date.now() + 30_000; + let ready = false; + while (Date.now() < deadline) { + try { + const r = await fetch(`${httpURL}/RateCounter/`, { + headers: { Authorization: auth }, + signal: AbortSignal.timeout(3_000), + }); + if (r.status !== 503) { + ready = true; + break; + } + } catch { + /* not ready */ + } + await sleep(200); + } + if (!ready) throw new Error('RateCounter never left 503 within the 30s readiness deadline'); + }); + + after(async () => { + await teardownHarper(ctx); + }); + + function hdrs() { + return { 'Content-Type': 'application/json', 'Authorization': auth }; + } + + async function increment(id: string): Promise { + try { + const r = await fetch(`${httpURL}/RateIncrement/`, { + method: 'POST', + headers: hdrs(), + body: JSON.stringify({ id }), + signal: AbortSignal.timeout(6_000), + }); + return r.status; + } catch { + return 'error'; + } + } + + async function put(id: string, hits: number): Promise { + try { + const r = await fetch(`${httpURL}/RateCounter/${id}`, { + method: 'PUT', + headers: hdrs(), + body: JSON.stringify({ id, hits }), + signal: AbortSignal.timeout(5_000), + }); + return r.status; + } catch { + return 'error'; + } + } + + async function getHits(id: string): Promise<{ status: number | 'error'; hits: number | null }> { + try { + const r = await fetch(`${httpURL}/RateCounter/${id}`, { + headers: { Authorization: auth }, + signal: AbortSignal.timeout(5_000), + }); + if (r.status !== 200) return { status: r.status, hits: null }; + const body = await r.json(); + return { status: 200, hits: Number(body?.hits ?? -1) }; + } catch { + return { status: 'error', hits: null }; + } + } + + test('concurrent bursts with in-burst GET pressure converge to exact counts', async () => { + let clean = 0; + let convergedLate = 0; + let stuckShort = 0; + let over = 0; + let inconclusive = 0; + const anomalyLogs: string[] = []; + + for (let round = 0; round < ROUNDS; round++) { + const ids = Array.from({ length: WINDOWS }, (_, w) => `r${round}w${w}`); + await Promise.all(ids.map((id) => put(id, 0))); + + const perWindow = await Promise.all( + ids.map(async (id) => { + let bursting = true; + const reader = (async () => { + while (bursting) await getHits(id); + })(); + const results = await Promise.all(Array.from({ length: HITS }, () => increment(id))); + bursting = false; + await reader; + return { + id, + acked: results.filter((s) => s === 200).length, + errs: results.filter((s) => s === 'error').length, + }; + }) + ); + + for (const { id, acked, errs } of perWindow) { + // a window whose burst was wholly rejected measures nothing + if (acked === 0) { + inconclusive++; + continue; + } + const first = await getHits(id); + if (first.status !== 200) { + inconclusive++; + continue; + } + // a timed-out request may still have been applied, so anything up to acked+errs is + // exact-or-explained; only beyond that is a double-apply + if (first.hits! >= acked && first.hits! <= acked + errs) { + clean++; + continue; + } + if (first.hits! > acked + errs) { + over++; + anomalyLogs.push(`${id}: OVER first=${first.hits} acked=${acked} errs=${errs}`); + continue; + } + const seen: (number | string)[] = [first.hits!]; + let finalHits: number | null = first.hits; + let expired = false; + const deadline = Date.now() + 400; + while (Date.now() < deadline) { + await sleep(20); + const g = await getHits(id); + if (g.status === 404) { + expired = true; + seen.push('404'); + break; + } + if (g.status !== 200) { + seen.push(String(g.status)); + continue; + } + seen.push(g.hits!); + finalHits = g.hits; + if (g.hits! >= acked) break; + } + if (finalHits! >= acked && finalHits! <= acked + errs) { + convergedLate++; + anomalyLogs.push(`${id}: CONVERGED_LATE acked=${acked} seen=[${seen.join(',')}]`); + } else if (finalHits! > acked + errs) { + over++; + anomalyLogs.push( + `${id}: OVER-LATE final=${finalHits} acked=${acked} errs=${errs} seen=[${seen.join(',')}]` + ); + } else if (expired) { + inconclusive++; + anomalyLogs.push(`${id}: EXPIRED-WHILE-SHORT acked=${acked} seen=[${seen.join(',')}]`); + } else { + stuckShort++; + anomalyLogs.push(`${id}: STUCK_SHORT final=${finalHits} acked=${acked} seen=[${seen.join(',')}]`); + } + } + // let the round's records expire so rounds stay independent + await sleep(700); + } + + console.log( + `\n[QA-431-CONVERGENCE] ${ROUNDS}×${WINDOWS}×${HITS} (${WORKERS} workers)\n` + + ` clean=${clean} converged-late=${convergedLate} stuck-short=${stuckShort} over=${over} inconclusive=${inconclusive}\n` + + (anomalyLogs.length ? ` ${anomalyLogs.join('\n ')}` : ' (no anomalies)') + ); + + ok(stuckShort === 0, `QA-431-CONVERGENCE: ${stuckShort} window(s) with durably lost increments`); + ok(over === 0, `QA-431-CONVERGENCE: ${over} window(s) over-counted (double-apply)`); + ok( + clean + convergedLate + stuckShort + over >= (ROUNDS * WINDOWS) / 2, + `QA-431-CONVERGENCE: only ${clean + convergedLate + stuckShort + over}/${ROUNDS * WINDOWS} windows measurable` + ); + }); + } +); diff --git a/package-lock.json b/package-lock.json index b24510e024..bb8dfb7553 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "harper", - "version": "5.2.5", + "version": "5.3.0-alpha.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "harper", - "version": "5.2.5", + "version": "5.3.0-alpha.1", "license": "Apache-2.0", "dependencies": { "@aws-sdk/client-s3": "^3.1012.0", @@ -17,7 +17,7 @@ "@fastify/cors": "^11.2.0", "@fastify/static": "^9.1.3", "@harperfast/extended-iterable": "1.0.3", - "@harperfast/rocksdb-js": "2.8.0", + "@harperfast/rocksdb-js": "2.9.0", "@harperfast/skills": "^1.10.8", "@turf/area": "6.5.0", "@turf/boolean-contains": "6.5.0", @@ -2495,14 +2495,14 @@ } }, "node_modules/@harperfast/rocksdb-js": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js/-/rocksdb-js-2.8.0.tgz", - "integrity": "sha512-czswCG+1KCRMYe6XQcsh5u28IQ6EbIx9eXPQd0kmjiLbtoMjiszYE92756bZJSbDykbR2OJLOazMZg7+qfFSJA==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js/-/rocksdb-js-2.9.0.tgz", + "integrity": "sha512-EOQnnDo4aoxLidosuQGZ2xO4kDCqwridUQdJ7UF7Ui7d5gTh6tUfTkqRSYk2CbFehVJDgU1qQLJZePGsXRRaEg==", "license": "Apache-2.0", "dependencies": { - "@harperfast/extended-iterable": "1.0.3", - "msgpackr": "2.0.6", - "ordered-binary": "1.6.1" + "@harperfast/extended-iterable": "^1.0.3", + "msgpackr": "^2.0.6", + "ordered-binary": "^1.6.1" }, "bin": { "rocksdb-js": "bin/rocksdb-js.mjs" @@ -2511,20 +2511,20 @@ "node": "^22.18.0 || >=24.0.0" }, "optionalDependencies": { - "@harperfast/rocksdb-js-darwin-arm64": "2.8.0", - "@harperfast/rocksdb-js-darwin-x64": "2.8.0", - "@harperfast/rocksdb-js-linux-arm64-glibc": "2.8.0", - "@harperfast/rocksdb-js-linux-arm64-musl": "2.8.0", - "@harperfast/rocksdb-js-linux-x64-glibc": "2.8.0", - "@harperfast/rocksdb-js-linux-x64-musl": "2.8.0", - "@harperfast/rocksdb-js-win32-arm64": "2.8.0", - "@harperfast/rocksdb-js-win32-x64": "2.8.0" + "@harperfast/rocksdb-js-darwin-arm64": "2.9.0", + "@harperfast/rocksdb-js-darwin-x64": "2.9.0", + "@harperfast/rocksdb-js-linux-arm64-glibc": "2.9.0", + "@harperfast/rocksdb-js-linux-arm64-musl": "2.9.0", + "@harperfast/rocksdb-js-linux-x64-glibc": "2.9.0", + "@harperfast/rocksdb-js-linux-x64-musl": "2.9.0", + "@harperfast/rocksdb-js-win32-arm64": "2.9.0", + "@harperfast/rocksdb-js-win32-x64": "2.9.0" } }, "node_modules/@harperfast/rocksdb-js-darwin-arm64": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-darwin-arm64/-/rocksdb-js-darwin-arm64-2.8.0.tgz", - "integrity": "sha512-v7cj3bGpRptZHXSbhVzYJ0k+jyMD0Z0kX0u8YNZH6Xq+ifZ6zBkBbyoSdvQ2waGLJXxBORylUxTgYMvTZWiQBA==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-darwin-arm64/-/rocksdb-js-darwin-arm64-2.9.0.tgz", + "integrity": "sha512-zgBjJKkHlyjiKJP+FkstFj1znZ/O8qXE9yIXIyKhTtxDeayobFY2x6M/KOWd9N1kaa8GrAQXGawUEXZnV1U+qw==", "cpu": [ "arm64" ], @@ -2538,9 +2538,9 @@ } }, "node_modules/@harperfast/rocksdb-js-darwin-x64": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-darwin-x64/-/rocksdb-js-darwin-x64-2.8.0.tgz", - "integrity": "sha512-9mMPPvRhxjLorE7u1gN3n+InzT4XHXQkHYdNKE6qN4iwZy0vM8iHoQLwIiN9GpzUiueEqBEilAP/9mIOHhMgBw==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-darwin-x64/-/rocksdb-js-darwin-x64-2.9.0.tgz", + "integrity": "sha512-OfAtdB7GDjkKYaq6Db3mDfoRKnrnjXS1X4OaF4xFsQUoZRtmMvF3OTP9w4N8c2vvEff2kjlGqpj2Dewab1uGNQ==", "cpu": [ "x64" ], @@ -2554,9 +2554,9 @@ } }, "node_modules/@harperfast/rocksdb-js-linux-arm64-glibc": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-linux-arm64-glibc/-/rocksdb-js-linux-arm64-glibc-2.8.0.tgz", - "integrity": "sha512-H1nNimVbIB4/6Sxv5DrxmP1asHLbxTCHuZ2ZN/zudqPkp/bTTSs5RsczOH6JeKTEUxUsJDTXM4nGRcn6LwFaOg==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-linux-arm64-glibc/-/rocksdb-js-linux-arm64-glibc-2.9.0.tgz", + "integrity": "sha512-/CnIr6vy+D5M4c2WOAyoC9/2TKSEYCvvsyzz0AoWw3taUmEX6fR308eHMA0iKFHPp7xSI3fyz0fBOd+3TO250A==", "cpu": [ "arm64" ], @@ -2573,9 +2573,9 @@ } }, "node_modules/@harperfast/rocksdb-js-linux-arm64-musl": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-linux-arm64-musl/-/rocksdb-js-linux-arm64-musl-2.8.0.tgz", - "integrity": "sha512-EqG4lkYptNviIcPPaQjK+X4ufjqzSImIvVxP89GHFHrbOJkGgEwRpiya0gWfGu3vJ1iGL1XL1nsU/kEjeDOugg==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-linux-arm64-musl/-/rocksdb-js-linux-arm64-musl-2.9.0.tgz", + "integrity": "sha512-MBM4Vv4ERdRgbmXZXQrGPhgmTmb0EX22dwDEFrq57mpn5IEuuXU4S/RXGAcsoEmYqYb86lvtukD5AnvxcefoSA==", "cpu": [ "arm64" ], @@ -2592,9 +2592,9 @@ } }, "node_modules/@harperfast/rocksdb-js-linux-x64-glibc": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-linux-x64-glibc/-/rocksdb-js-linux-x64-glibc-2.8.0.tgz", - "integrity": "sha512-MZPKZ6gF8kCy/KXTav4tc7T2uq+iOunxDOYq7nifZkCCy+0+C9U7g9CZx0NErORRQRP29x3z0zEKkCQ/go5c6Q==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-linux-x64-glibc/-/rocksdb-js-linux-x64-glibc-2.9.0.tgz", + "integrity": "sha512-rhIoInrDAzwMx7N2H0K1mM8x/hY2D/NF2rhd8ERT2w2GoK4NqdBtPldVY8ein8fmtDdYaS6CpHmRSmq+xlhqhw==", "cpu": [ "x64" ], @@ -2611,9 +2611,9 @@ } }, "node_modules/@harperfast/rocksdb-js-linux-x64-musl": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-linux-x64-musl/-/rocksdb-js-linux-x64-musl-2.8.0.tgz", - "integrity": "sha512-PnMn1f/UxGOZKxlayR82tzsFCGoaMu0mBGKMmP/r2rbxzX5T89T0c+aVhuhpgNrFlWXTdawlA10Oh/YjfT/2qQ==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-linux-x64-musl/-/rocksdb-js-linux-x64-musl-2.9.0.tgz", + "integrity": "sha512-D/MfsqycP7ijIAAptUcITsMMzn7cnawdL3H1XUqe+h9fgNP7+y0gXk/9KxVA5D6cKPw1TkEDaeLuuBsiI2vFUA==", "cpu": [ "x64" ], @@ -2630,9 +2630,9 @@ } }, "node_modules/@harperfast/rocksdb-js-win32-arm64": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-win32-arm64/-/rocksdb-js-win32-arm64-2.8.0.tgz", - "integrity": "sha512-4khqmGbIebiCwY3FVIgcUXEmMapgWP8f97aNrbThPDCIkuJscB3yoAG4A8OJo+Jln/TdjJZOgy8e2ir9F926dA==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-win32-arm64/-/rocksdb-js-win32-arm64-2.9.0.tgz", + "integrity": "sha512-76PILBAErw0R9enoprg83FdyHk0UNG63pII4klFV5Xlrfj0gmJ0iqLEbocEwC+d2RTGkpgkUPjoNMw4BbMpyTA==", "cpu": [ "arm64" ], @@ -2646,9 +2646,9 @@ } }, "node_modules/@harperfast/rocksdb-js-win32-x64": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-win32-x64/-/rocksdb-js-win32-x64-2.8.0.tgz", - "integrity": "sha512-E5HpBAnRwotITq++534WCinMvYidkkPoaAD5uwBNHiQbNt6YYvNZllJAIh/bukXD+uO6DL6qUMmVkeBh3kWK5A==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@harperfast/rocksdb-js-win32-x64/-/rocksdb-js-win32-x64-2.9.0.tgz", + "integrity": "sha512-BRPIzbvx1NlOhA6vwAnUwFYuIQykmdsjGxnblb5ejvdf9HFw0b6durdKQfPyk8L+RO2uiXqpfkK4Da8V0vhgdQ==", "cpu": [ "x64" ], diff --git a/package.json b/package.json index c8f8435a37..a86fa618ff 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "harper", "description": "Harper is an open-source Node.js performance platform that unifies database, cache, application, and messaging layers into one in-memory process.", - "version": "5.2.5", + "version": "5.3.0-alpha.1", "license": "Apache-2.0", "homepage": "https://harper.fast", "bugs": { @@ -180,7 +180,7 @@ "@fastify/cors": "^11.2.0", "@fastify/static": "^9.1.3", "@harperfast/extended-iterable": "1.0.3", - "@harperfast/rocksdb-js": "2.8.0", + "@harperfast/rocksdb-js": "2.9.0", "@harperfast/skills": "^1.10.8", "@turf/area": "6.5.0", "@turf/boolean-contains": "6.5.0", diff --git a/resources/DESIGN.md b/resources/DESIGN.md index ae3a287e5f..59281e87c4 100644 --- a/resources/DESIGN.md +++ b/resources/DESIGN.md @@ -12,27 +12,27 @@ See also: `../DESIGN.md` for cross-cutting non-obvious internals (RecordObject p ## File overview -| File | Purpose | -| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -| `Resource.ts` | Base class; `transactional()` wrapper; method routing | -| `Table.ts` | Table-as-Resource implementation. Factory `makeTable()` returns a `TableResource` subclass per table. **See section markers below.** | -| `Resources.ts` | Registry mapping URL paths → Resource classes | -| `RequestTarget.ts` | Parses path/query into a structured target | -| `ResourceInterface.ts` | Type definitions (`Context`, `Record`, etc.) | -| `RecordEncoder.ts` | msgpack encoding + `entryMap` (record → storage entry) | -| `IterableEventQueue.ts` | Async iterable used for subscriptions and streaming responses | -| `transaction.ts` | Per-request transaction object stored in `contextStorage` | -| `auditStore.ts` | Append-only audit log records | -| `derivedIndexRuntime.ts` | Lock-elected, cursor-based delivery of committed RocksDB log mutations to derived-index backends | -| `derivedIndexRegistry.ts` | Worker-local registration counts used to emit cache-eviction markers only for tables with derived indexes | -| `RocksDerivedIndexStorage.ts` | Binary, WAL-backed host storage used by native derived indexes; batches through one RocksDB transaction and syncs through the shared root | -| `recordLock.ts` | Exclusive record locks (harper#483): option contract, native key lock primitives (`lockAttemptKey`, `makeKeyLockHandle`, `acquireRecordKey`) | -| `nodeIdMapping.ts` | Maps node IDs ↔ timestamps for replication ordering | -| `openApi.ts` | Generates OpenAPI/JSON Schema from `@export` schemas | -| `defineTable.ts` | Code-first table authoring (`defineTable` + `types`) — a TS front-end to the canonical `table()` model | -| `defineResource.ts` | Per-method request contract (`defineResource` / `Resource.withSchema`, `t`, `schemaOf`) — typed handlers + edge validation | -| `jsonSchemaTypes.ts` | Shared `JsonSchemaFragment` IR + `attributeToFragment` projector (one vocabulary for validation/OpenAPI/MCP) | -| `analytics/` | Telemetry recording (separate from monitoring) | +| File | Purpose | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Resource.ts` | Base class; `transactional()` wrapper; method routing | +| `Table.ts` | Table-as-Resource implementation. Factory `makeTable()` returns a `TableResource` subclass per table. **See section markers below.** | +| `Resources.ts` | Registry mapping URL paths → Resource classes | +| `RequestTarget.ts` | Parses path/query into a structured target | +| `ResourceInterface.ts` | Type definitions (`Context`, `Record`, etc.) | +| `RecordEncoder.ts` | msgpack encoding + `entryMap` (record → storage entry) | +| `IterableEventQueue.ts` | Async iterable used for subscriptions and streaming responses | +| `transaction.ts` | Per-request transaction object stored in `contextStorage` | +| `auditStore.ts` | Append-only audit log records | +| `derivedIndexRuntime.ts` | Lock-elected, cursor-based delivery of committed RocksDB log mutations to derived-index backends; chunked collection, flush cadence, rebuild phase, epoch fencing, shared readiness | +| `derivedIndexRegistry.ts` | Worker-local registration counts used to emit cache-eviction markers only for tables with derived indexes | +| `RocksDerivedIndexStorage.ts` | Binary, WAL-backed host storage used by native derived indexes; batches through one RocksDB transaction and syncs through the shared root | +| `recordLock.ts` | Exclusive record locks (harper#483): option contract, native key lock primitives (`lockAttemptKey`, `makeKeyLockHandle`, `acquireRecordKey`) | +| `nodeIdMapping.ts` | Maps node IDs ↔ timestamps for replication ordering | +| `openApi.ts` | Generates OpenAPI/JSON Schema from `@export` schemas | +| `defineTable.ts` | Code-first table authoring (`defineTable` + `types`) — a TS front-end to the canonical `table()` model | +| `defineResource.ts` | Per-method request contract (`defineResource` / `Resource.withSchema`, `t`, `schemaOf`) — typed handlers + edge validation | +| `jsonSchemaTypes.ts` | Shared `JsonSchemaFragment` IR + `attributeToFragment` projector (one vocabulary for validation/OpenAPI/MCP) | +| `analytics/` | Telemetry recording (separate from monitoring) | --- diff --git a/resources/DatabaseTransaction.ts b/resources/DatabaseTransaction.ts index c088f06b67..732773fe42 100644 --- a/resources/DatabaseTransaction.ts +++ b/resources/DatabaseTransaction.ts @@ -26,6 +26,7 @@ import { } from './longLivedTransactions.ts'; const trackedTxns = new Set(); +const readTransactionOwners = new WeakMap(); // Read options for a rotated generation's native transactions; shared because they never vary. const SNAPSHOT_FREE = Object.freeze({ disableSnapshot: true }); // Logical transactions the monitor supervises for their WRITES, kept apart from trackedTxns because the @@ -292,6 +293,76 @@ export function transactionOpenTooLongError(): ServerError { ); } +class ReadSnapshotExpiredError extends ServerError { + constructor() { + super('Read scan snapshot expired; retry the read without replaying previously committed writes', 503); + this.name = 'ReadSnapshotExpiredError'; + } +} + +export function trackReadRange(transaction: ReadTransaction, createRange: () => any): any { + const owner = readTransactionOwners.get(transaction); + if (!owner) return createRange(); + function checkActive() { + if (owner.timedOut) throw transactionOpenTooLongError(); + if (owner.transaction !== transaction) { + throw new ReadSnapshotExpiredError(); + } + } + checkActive(); + const range = createRange(); + const iterate = range.iterate; + range.iterate = function (options) { + const iterator = iterate.call(this, options); + let done = false; + // Closing the underlying iterator is the one step here that can throw for a reason the caller + // must not see: `next()` only reaches it once the snapshot is already gone, which is the + // likeliest moment for the native layer to object, and an error from cleanup would replace the + // named 503 with exactly the raw iterator error this wrapper exists to stop surfacing. The + // closure reference rather than `this` so a destructured `next` still cleans up. + const wrapper = { + [Symbol.iterator]() { + return this; + }, + next() { + if (done) return { done: true, value: undefined }; + try { + checkActive(); + owner.rangeReadActive = true; + const result = iterator.next(); + done = result.done === true; + return result; + } catch (error) { + closeQuietly(); + throw error; + } + }, + return(value?: any) { + if (!done) { + done = true; + iterator.return?.(value); + } + return { done: true, value }; + }, + throw(error) { + // Not delegated to `iterator.throw`: it closes and rethrows the same error anyway, and + // delegating after the close below would run it against an iterator already closed. + closeQuietly(); + throw error; + }, + }; + function closeQuietly() { + try { + wrapper.return(); + } catch { + // the error being propagated is the actionable one + } + } + return wrapper; + }; + return range; +} + type MaybePromise = T | Promise; export type CommitOptions = { @@ -381,6 +452,9 @@ export type TransactionWrite = { // this settles (fire-and-forget from the if-branch), so Table.save()'s lock-writable path awaits // it to ensure the write is durable before resolving to the caller. innerCommit?: MaybePromise; + // the commit derives stored state (folds, index diffs, residency) from its base entry, so + // save() must reload that base through the committing transaction's snapshot + reloadCommitBase?: boolean; }; export function getAppliedWriteVersion(recordVersion: number | undefined, txnLogKey: number): number { @@ -534,8 +608,9 @@ export class DatabaseTransaction implements Transaction { } } - getReadTxn(disableSnapshot?: boolean): ReadTransaction { - this.readTxnRefCount = (this.readTxnRefCount || 0) + 1; + rangeReadActive = false; + + renewReadTimeout(): void { // The limit is an IDLE limit. Writes always re-arm it (see addWrite), but reads only do so // while no uncommitted writes are held: staged writes hold write intents that other writers' // coordinated-retry commits park on, so a handler that wrote once and then only reads — an @@ -548,6 +623,11 @@ export class DatabaseTransaction implements Transaction { if ((this.writes.length === 0 && !this.next) || this.open !== TRANSACTION_STATE.OPEN || !this.hasPendingWrites()) { this.timeout = Math.max(txnExpiration, this.timeoutBudget); } + } + + getReadTxn(disableSnapshot?: boolean): ReadTransaction { + this.readTxnRefCount = (this.readTxnRefCount || 0) + 1; + this.renewReadTimeout(); if (this.transaction) { if ((this.transaction as any).openTimer) (this.transaction as any).openTimer = 0; return this.transaction; @@ -582,6 +662,8 @@ export class DatabaseTransaction implements Transaction { // Monitor state is not ownership state: it stays with `trackedTxns.add` in getReadTxn(). private attachOwnedTransaction(transaction: RocksTransactionWithRetry): void { this.transaction = transaction; + readTransactionOwners.set(transaction, this); + this.rangeReadActive = false; this.readTxnsUsed = 1; this.baseReadRefConsumed = false; this.handleOpenedAt = performance.now(); @@ -623,6 +705,7 @@ export class DatabaseTransaction implements Transaction { trackedTxns.delete(this); this.endWriteSupervision(); this.transaction = null; + this.rangeReadActive = false; this.readTxnsUsed = 0; this.readTxnRefCount = 0; this.handleOpenedAt = 0; @@ -1060,8 +1143,17 @@ export class DatabaseTransaction implements Transaction { // flags so an ordinary write never reads the property (harper#2412). const writeVersion = this.sourceApply || this.isReplay ? getAppliedWriteVersion(operation.recordVersion, txnTime) : txnTime; - if (reloadEntry || operation.entry === undefined) { - operation.entry = operation.store.getEntry(operation.key, { transaction }); + // A base that feeds stored state must come from this transaction's snapshot, never the + // cross-worker cache vouch (stale when a resequenced write reused a version). That closes the + // lost-update window only when this transaction holds a snapshot to validate the later Put + // against; a snapshot-free transaction (this.snapshotFree, after a mid-scope-commit rotation) + // has no snapshot for rocksdb-js to validate the Put against, so it only narrows the window to + // the read-to-put span rather than closing it (open follow-up, tracked in the PR description). + // Replays keep their pre-read base — their convergence contract is the replay pass itself. + const reloadsCommitBase = operation.reloadCommitBase && !operation.saved && !this.isReplay; + if (reloadEntry || operation.entry === undefined || reloadsCommitBase) { + const uncachedRead = (!!operation.reloadCommitBase && !this.isReplay) || reloadEntry; + operation.entry = operation.store.getEntry(operation.key, { transaction, uncachedRead }); } if (!operation.saved) { operation.saved = true; @@ -1234,6 +1326,8 @@ export class DatabaseTransaction implements Transaction { if (transaction) { this.writes = this.writes.filter((write) => write); // filter out removed entries if (this.writes.length > 0) { + // Commit retries can construct fresh ranges on this live handle after read ownership ends. + readTransactionOwners.delete(transaction); // The transaction was created with coordinatedRetry:true (see // getReadTxn), so commit() can resolve to RETRY_NOW_VALUE. That // sentinel (a number) is why commitResolution is typed @@ -2032,6 +2126,10 @@ function startMonitoringTxns() { reportNow: number, reportBudget: LongLivedHolderReportBudget ) { + if (txn.rangeReadActive) { + txn.rangeReadActive = false; + txn.renewReadTimeout(); + } reportIfLongLived(txn, reportThresholdMs, reportNow, reportBudget); { const commitChainHead = txn.commitChainHead ?? txn; diff --git a/resources/PrimaryRocksDatabase.ts b/resources/PrimaryRocksDatabase.ts index a045342a83..bd92ae33f4 100644 --- a/resources/PrimaryRocksDatabase.ts +++ b/resources/PrimaryRocksDatabase.ts @@ -1,9 +1,10 @@ +import { trackReadRange } from './DatabaseTransaction.ts'; import { RocksDatabase, type RocksDatabaseOptions, constants, type Store, Transaction } from '@harperfast/rocksdb-js'; const FRESH_VERSION_FLAG = constants.FRESH_VERSION_FLAG; import { WeakLRUCache } from 'weak-lru-cache'; import { when } from '../utility/when.ts'; -import { assignStoredFields, entryMap, METADATA, type Entry } from './RecordEncoder.ts'; +import { assignStoredFields, entryMap, METADATA, VERSION_REUSED, type Entry } from './RecordEncoder.ts'; /** * RocksDatabase subclass that owns all primary-store behaviour for Harper tables: @@ -121,6 +122,13 @@ export class PrimaryRocksDatabase extends RocksDatabase { */ getEntry(id: any, options?: any): any { this.readCount++; + // Commit-path base reads must reflect the caller's transaction snapshot; the cache vouch + // answers "latest committed" — the wrong question there, and wrong outright for a version a + // resequenced write reused — so read the store directly, touching neither cache nor VT. + if (options?.uncachedRead) { + if (options.async) return when(super.get(id, options), (result) => this.#processEntry(result, id)); + return this.#processEntry(super.getSync(id, options), id); + } const cache = this.#cache; // The cache stores the record *value* (weakly, via setValue) rather than // the Entry: a WeakRef-wrapped value lets the LRFU expirer release it once @@ -158,11 +166,16 @@ export class PrimaryRocksDatabase extends RocksDatabase { if (cache && cachedValue !== undefined) cache.delete(id); return undefined; } - // Only object values can be weakly cached and mapped back to their Entry; - // primitive/empty values fall through uncached (no fast path, still correct). - if (entry.version != null && cache && entry.value != null && typeof entry.value === 'object') { - entryMap.set(entry.value, entry); - cache.setValue(id, entry.value, (entry.size ?? 0) >> 10); + if (entry.version != null && cache) { + // its version no longer identifies its value, so drop any copy already held + if (entry.metadataFlags & VERSION_REUSED) { + if (cachedValue !== undefined) cache.delete(id); + } else if (entry.value != null && typeof entry.value === 'object') { + // Only object values can be weakly cached and mapped back to their Entry; + // primitive/empty values fall through uncached (no fast path, still correct). + entryMap.set(entry.value, entry); + cache.setValue(id, entry.value, (entry.size ?? 0) >> 10); + } } return entry; }); @@ -178,7 +191,7 @@ export class PrimaryRocksDatabase extends RocksDatabase { } getRange(options?: any): any { - const iterable = super.getRange(options); + const iterable = trackReadRange(options?.transaction, () => super.getRange(options)); if (options?.valuesForKey) return iterable.map((v: any) => v?.value); if (options?.values === false || options?.onlyCount) return iterable; if (!this.#enc.isRocksDB) return iterable; diff --git a/resources/RecordEncoder.ts b/resources/RecordEncoder.ts index 2de8d2d913..a956915189 100644 --- a/resources/RecordEncoder.ts +++ b/resources/RecordEncoder.ts @@ -30,7 +30,7 @@ import { } from './blob.ts'; import { getThisNodeId } from './nodeIdMapping.ts'; import { recordAction } from './analytics/write.ts'; -import { RocksDatabase } from '@harperfast/rocksdb-js'; +import { constants, RocksDatabase } from '@harperfast/rocksdb-js'; import { when } from '../utility/when.ts'; import { CONFIG_PARAMS } from '../utility/hdbTerms.ts'; import * as envMngr from '../utility/environment/environmentManager.js'; @@ -124,7 +124,26 @@ export const HAS_NODE_ID = 64; export const PENDING_LOCAL_TIME = 1; export const HAS_STRUCTURE_UPDATE = 0x100; export const HAS_ADDITIONAL_AUDIT_REFS = 0x80; -export const VERSION_NOT_UNIQUE_FLAG = 0x10000; +// A resequenced write keeps the (newer) version it merged onto, so one version identifies two +// different stored values and version equality proves nothing. The metadata word this is set in is +// the same header word the VerificationTable reads at value offset 8 — ACTION_32_BIT is the tag +// byte its predicate requires — so the bit is what stops the native layer vouching for the version. +export const VERSION_REUSED = constants.VERSION_NOT_UNIQUE_FLAG; +// The bit is persisted in every resequenced record, so a rocksdb-js that moved it down onto one of +// the flags above (or up into the tag byte) would silently change what records already on disk +// mean: it must stay a single bit strictly between them. +if ( + typeof VERSION_REUSED !== 'number' || + (VERSION_REUSED & (VERSION_REUSED - 1)) !== 0 || + VERSION_REUSED < 0x10000 || + VERSION_REUSED > 0x800000 +) + throw new Error( + `rocksdb-js VERSION_NOT_UNIQUE_FLAG (${VERSION_REUSED}) is not a single bit in the range Harper record metadata reserves for it — requires @harperfast/rocksdb-js >= 2.8.0` + ); +function versionIsReused(newVersion: number, existingEntry: { version?: number } | undefined): boolean { + return existingEntry?.version != null && newVersion <= existingEntry.version; +} const TRACKED_WRITE_TYPES = new Set(['put', 'patch', 'delete', 'message', 'publish']); // For now we use this as the private property mechanism for mapping records to entries. @@ -885,11 +904,12 @@ export function recordUpdater(store, tableId, auditStore) { : NO_TIMESTAMP; const expiresAt = options?.expiresAt; if (expiresAt >= 0) assignMetadata |= HAS_EXPIRATION; - if (isRocksDB && record !== undefined && existingEntry?.version != null && newVersion <= existingEntry.version) { - assignMetadata = Math.max(assignMetadata, 0) | VERSION_NOT_UNIQUE_FLAG; - } metadataInNextEncoding = assignMetadata; expiresAtNextEncoding = expiresAt; + // Math.max normalizes the -1 "no metadata word" sentinel to 0 first: OR-ing the flag into + // -1 directly would stay -1 and silently drop the metadata word (and the flag with it). + if (isRocksDB && record !== undefined && versionIsReused(newVersion, existingEntry)) + metadataInNextEncoding = Math.max(metadataInNextEncoding, 0) | VERSION_REUSED; const putOptions: { version: number; instructedWrite?: boolean; diff --git a/resources/RocksIndexStore.ts b/resources/RocksIndexStore.ts index 5696c4d862..c08f695752 100644 --- a/resources/RocksIndexStore.ts +++ b/resources/RocksIndexStore.ts @@ -1,3 +1,4 @@ +import { trackReadRange } from './DatabaseTransaction.ts'; import { type CountEstimate, type CountEstimateOptions, @@ -48,7 +49,7 @@ export class RocksIndexStore extends RocksDatabase { * @param options */ getRange(options: StoreIteratorOptions): Iterable { - return super.getRange(translateIndexBounds(options)).map(({ key }) => { + return trackReadRange(options.transaction, () => super.getRange(translateIndexBounds(options))).map(({ key }) => { return { key: key[0], value: key.length > 2 ? key.slice(1) : key[1] }; }); } diff --git a/resources/Table.ts b/resources/Table.ts index 497792bcec..fabce7cc73 100644 --- a/resources/Table.ts +++ b/resources/Table.ts @@ -53,6 +53,7 @@ import { import * as envMngr from '../utility/environment/environmentManager.ts'; import { addSubscription } from './transactionBroadcast.ts'; import { + DerivedIndexLagError, handleHDBError, ClientError, ServerError, @@ -81,7 +82,7 @@ import { transaction, contextStorage } from './transaction.ts'; import { MAXIMUM_KEY, writeKey, compareKeys } from 'ordered-binary'; import { getWorkerIndex, getWorkerCount } from '../server/threads/manageThreads.js'; import { HAS_BLOBS, LOCAL_ONLY, auditRetention, removeAuditEntry } from './auditStore.ts'; -import { hasDerivedIndexRegistration } from './derivedIndexRegistry.ts'; +import { derivedIndexWriteRejection, hasDerivedIndexRegistration } from './derivedIndexRegistry.ts'; import { buildEmbedBefore, createDefaultEmbedder, type EmbedAttribute, type Embedder } from './models/embedHook.ts'; import { autoCast, autoCastBooleanStrict } from '../utility/common_utils.ts'; import { @@ -747,6 +748,13 @@ export function makeTable(options) { } return { txnLogKey: version, nodeId }; } + // Canonical-source applies (sourceApply), replay and replication notifications are never shed; + // dropping one would advance the source cursor past a write that never landed. + function assertDerivedIndexAdmission(options: any, transaction: any) { + if (options?.isNotification || transaction?.sourceApply || transaction?.isReplay) return; + const reason = derivedIndexWriteRejection(auditStore, tableId); + if (reason) throw new DerivedIndexLagError(reason); + } function stageDerivedIndexEviction(transaction: RocksTransaction, id: Id, version: number) { if (!hasDerivedIndexRegistration(auditStore, tableId)) return; const nodeId = getThisNodeId(auditStore) ?? 0; @@ -2272,6 +2280,7 @@ export function makeTable(options) { const context = this.getContext(); checkValidId(id); const transaction = txnForContext(this.getContext()); + assertDerivedIndexAdmission(options, transaction); const write: any = { key: id, store: primaryStore, @@ -2279,6 +2288,7 @@ export function makeTable(options) { entry: this.#entry, recordVersion: options?.version, lockHandle: this.#lockHandle && this.#lockHandle.keyId === writeKeyId(id) ? this.#lockHandle : undefined, + reloadCommitBase: true, commit: (txnTime, existingEntry, _retry, transaction: any) => { const txnLogKey = isRocksDB && options?.version != null ? (transaction?.getTimestamp?.() ?? txnTime) : txnTime; @@ -2330,6 +2340,7 @@ export function makeTable(options) { const context = this.getContext(); checkValidId(id); const transaction = txnForContext(this.getContext()); + assertDerivedIndexAdmission(options, transaction); const write: any = { key: id, store: primaryStore, @@ -2337,6 +2348,7 @@ export function makeTable(options) { entry: this.#entry, recordVersion: options?.version, lockHandle: this.#lockHandle && this.#lockHandle.keyId === writeKeyId(id) ? this.#lockHandle : undefined, + reloadCommitBase: true, before: (this.constructor as any).source?.relocate && !(context as any)?.source ? (this.constructor as any).source.relocate.bind((this.constructor as any).source, id, undefined, context) @@ -2888,6 +2900,7 @@ export function makeTable(options) { const context = this.getContext(); const transaction = txnForContext(context); const replaying = transaction.isReplay === true; + assertDerivedIndexAdmission(options, transaction); checkValidId(id); if (fullUpdate && recordUpdate == null && options?.isNotification) { // A source/replication-applied put must carry the record; these applies skip record @@ -2930,6 +2943,8 @@ export function makeTable(options) { entry, nodeName: (context as any)?.nodeName, fullUpdate, + // copy-apply rows keep their pre-read base: one read per row, healed by the post-copy replay + reloadCommitBase: options?.isCopyApply !== true, deferSave: true, // the origin's record version on an applied write; absent for a locally-originated one recordVersion: options?.version, @@ -2990,10 +3005,13 @@ export function makeTable(options) { : txnTime; } if (createdTimeProperty) { - if (entry?.value) { + // the reloaded commit base, not the pre-read one: a full PUT racing a create + // would otherwise stamp a fresh created time over the real one + const base = write.entry; + if (base?.value) { if (fullUpdate || recordUpdate[createdTimeProperty.name]) { // make sure to retain original created time - recordUpdate[createdTimeProperty.name] = entry?.value[createdTimeProperty.name]; + recordUpdate[createdTimeProperty.name] = base.value[createdTimeProperty.name]; } } else { // new entry, set created time @@ -3512,8 +3530,8 @@ export function makeTable(options) { if (recordToStore && recordToStore.getRecord) throw new Error('Can not assign a record to a record, check for circular references'); if (residencyId == undefined) { - if (entry?.residencyId) - (context as any).previousResidency = TableResource.getResidencyRecord(entry.residencyId); + if (existingEntry?.residencyId) + (context as any).previousResidency = TableResource.getResidencyRecord(existingEntry.residencyId); const residency = residencyFromFunction(TableResource.getResidency(recordToStore, context)); if (residency) { if (!residency.includes(server.hostname)) { @@ -3753,6 +3771,7 @@ export function makeTable(options) { this.#assertLiveHandle(id); const context = this.getContext(); const transaction = txnForContext(context); + assertDerivedIndexAdmission(options, transaction); checkValidId(id); const entry = this.#entry ?? primaryStore.getEntry(id, { transaction: transaction.getReadTxn() }); @@ -3761,6 +3780,7 @@ export function makeTable(options) { store: primaryStore, entry, chainsStagedState: true, + reloadCommitBase: true, nodeName: (context as any)?.nodeName, recordVersion: options?.version, lockHandle: this.#lockHandle && this.#lockHandle.keyId === writeKeyId(id) ? this.#lockHandle : undefined, @@ -3984,6 +4004,7 @@ export function makeTable(options) { // objects. Entries are small and shallow; the clone is cheap next to the query. conditions = cloneConditions(conditions); let orderAlignedCondition; + let syntheticOrderCondition; const filtered = {}; function prepareConditions(conditions: any[], operator: string) { @@ -4122,7 +4143,7 @@ export function makeTable(options) { // if it is indexed, we add a pseudo-condition to align with the natural sort order of the index. // the primary key has no secondary index, but the primary store is itself keyed in // primary-key order, so scanning it is already aligned with the sort - orderAlignedCondition = { ...sort, comparator: 'sort' }; + orderAlignedCondition = syntheticOrderCondition = { ...sort, comparator: 'sort' }; conditions.push(orderAlignedCondition); } else if (conditions.length === 0 && !target.allowFullScan) throw handleHDBError( @@ -4152,8 +4173,10 @@ export function makeTable(options) { }; } } else { - // if we had to add an aligned condition that isn't first, we remove it and do ordering later - if (orderAlignedCondition) conditions.splice(conditions.indexOf(orderAlignedCondition), 1); + // if we had to add an aligned condition that isn't first, we remove it and do ordering later — + // only the one we added; a caller's own condition on the sort attribute is still a filter + const syntheticIndex = syntheticOrderCondition ? conditions.indexOf(syntheticOrderCondition) : -1; + if (syntheticIndex >= 0) conditions.splice(syntheticIndex, 1); postOrdering = sort; } } diff --git a/resources/branchDatabase.ts b/resources/branchDatabase.ts index 10f459be05..af574b18fc 100644 --- a/resources/branchDatabase.ts +++ b/resources/branchDatabase.ts @@ -571,7 +571,7 @@ function branchRootOf(branchPath: string): string { * shared by every worker thread that loaded the application, so a thread that gives up its handle -- * a failed load, most often -- must not delete storage another thread is serving queries from. */ -async function closeBranchAt(branchPath: string): Promise { +export async function closeBranchAt(branchPath: string): Promise { const pending = branchesByPath.get(branchPath); branchesByPath.delete(branchPath); const opened = await pending?.catch(() => null); @@ -888,11 +888,10 @@ export async function prepareBranches( getDatabases(); if (branchedDatabases === true) { - // A snapshot, not a subscription: this is every database that exists at THIS load. One created - // afterward -- by another application, or by this one once schema declarations can target a - // branch -- is not retroactively branched. `system` is excluded the same way an explicit - // declaration of it is refused (assertBranchedDatabases): it carries the instance's own catalog, - // users and jobs, not application data. + // A snapshot, not a subscription: every database that exists at THIS load. One created afterward + // is not retroactively branched, and this application's own declarations into an unbranched + // database land in the base. `system` is excluded the same way an explicit declaration of it is + // refused (assertBranchedDatabases): it carries the instance's catalog, users and jobs. branchedDatabases = Object.keys(databases).filter((name) => name !== 'system'); } if (!branchedDatabases.length) return branches; @@ -924,8 +923,12 @@ export async function prepareBranches( if (isNew) opened.push(branchPath); } // Only now: a relationship whose target this application also branched has to resolve to that - // branch, and the whole set has to exist before any of them can resolve that way. - for (const branch of branches.values()) hydrateBranchRelationships(branch, branches); + // branch, and the whole set has to exist before any of them can resolve that way. The set stays + // on each branch for the reloads its own schema declarations trigger later. + for (const branch of branches.values()) { + branch.relatedBranches = branches; + hydrateBranchRelationships(branch, branches); + } } catch (error) { // A partially branched application is worse than one that failed to load: some of its names // would resolve to a branch and the rest to the base. Only this application's handles go, and diff --git a/resources/branchGuard.ts b/resources/branchGuard.ts deleted file mode 100644 index d5cfd5255a..0000000000 --- a/resources/branchGuard.ts +++ /dev/null @@ -1,30 +0,0 @@ -import { DEFAULT_DATABASE_NAME } from '../utility/hdbTerms.ts'; - -/** - * Refuse a table declaration that would land in the BASE of a database this application branched. - * - * `table()` registers in the process-wide catalog, so a branched application declaring a table -- - * through GraphQL `@table`, `scope.ensureTable`, or `defineTable` -- would create it in the base: - * replicated, visible to every other application, and bound to this application's own routes, while - * its JavaScript read and wrote the branch. Refusing is temporary; making these land in the branch - * is harper#2264. Databases this application did not branch are untouched. - */ -export function assertTableTargetNotBranched( - branches: Map | undefined, - databaseName: string | undefined | null, - tableName: string, - how: string -): void { - if (!branches?.size) return; - // Falsy, not nullish: `table()` resolves every falsy name to the default database, so a guard that - // only defaulted null and undefined would let `database: ''` past the fence and then land in the - // base as `data`. - const target = databaseName || DEFAULT_DATABASE_NAME; - if (!branches.has(target)) return; - const error: any = new Error( - `Cannot declare table '${tableName}' in branched database '${target}' through ${how}: it would be created ` + - `in the base database's schema rather than in this application's branch` - ); - error.statusCode = 400; - throw error; -} diff --git a/resources/databases.ts b/resources/databases.ts index 8c4308f33c..008523ca8a 100644 --- a/resources/databases.ts +++ b/resources/databases.ts @@ -1,4 +1,5 @@ import { EventEmitter } from 'node:events'; +import { randomBytes } from 'node:crypto'; import { initSync, getHdbBasePath, get as envGet } from '../utility/environment/environmentManager.ts'; import { INTERNAL_DBIS_NAME } from '../utility/lmdb/terms.ts'; import { open, compareKeys, type Database, type RootDatabase } from 'lmdb'; @@ -529,6 +530,21 @@ function applyDurableDeclaration(attribute: any, descriptor: any) { else delete attribute[field]; } } + +/** + * True when a descriptor claims an index build no live operation in this process can own. The PID and + * worker generation cannot answer that alone: a container reuses PID 1 and starts the in-memory + * generation back at 1 while the persisted one is higher. A descriptor with no incarnation was written + * before the field existed, so it belongs to an earlier process; a thread started without one of its + * own cannot judge, and falls back rather than declaring a live build dead. + */ +function isAbandonedIndexBuild(descriptor: any, currentRestartGeneration: number): boolean { + if (!descriptor) return false; + if (descriptor.indexingPID && descriptor.indexingPID !== process.pid) return true; + if (descriptor.restartNumber < currentRestartGeneration) return true; + const incarnation = manageThreads.processIncarnation; + return !!descriptor.indexingPID && incarnation != null && descriptor.indexingIncarnation !== incarnation; +} // How many times the schema load will try to finish a tombstoned drop before // giving up for the rest of this process's lifetime. A drop that fails once // almost always fails identically forever - the usual cause is a RocksDB @@ -795,9 +811,11 @@ export function hydrateBranchRelationships(branch: BranchDatabase, branches: Map // data -- the fallback belongs to a database the application did not branch, never to one it did. return targetBranch ? targetBranch.tables?.[target.table] : databases[target.database]?.[target.table]; }; - for (const hydration of branch.pendingRelationships.splice(0)) { + // Kept, not drained, like the global list: a target declared later (on this or another thread) is + // picked up by the next pass, and `hydrateTableRelationships` is a no-op once everything resolves. + for (const hydration of branch.pendingRelationships) { try { - hydrateTableRelationships(hydration, resolveTarget); + hydrateTableRelationships(hydration, resolveTarget, false); } catch (error) { logger.error( `Unable to hydrate persisted relationships for branch table ${hydration.databaseName}.${hydration.tableName}`, @@ -830,7 +848,8 @@ const resolveTargetGlobally: ResolveRelationshipTarget = (target) => databases[t function hydrateTableRelationships( { table, databaseName, tableName, definitions }: RelationshipHydration, - resolveTarget: ResolveRelationshipTarget = resolveTargetGlobally + resolveTarget: ResolveRelationshipTarget = resolveTargetGlobally, + announce = true ): void { const hydratable: { definition: PersistedRelationship; targetTable: any }[] = []; for (let index = 0; index < definitions.length; index++) { @@ -879,7 +898,7 @@ function hydrateTableRelationships( table.attributes.splice(0, table.attributes.length, ...attributes); table.schemaVersion++; table.updatedAttributes(); - databaseEventsEmitter.emit('updateTable', table); + if (announce) databaseEventsEmitter.emit('updateTable', table); } function validRelationshipDefinition(definition: any, definitions: unknown[], index: number): boolean { @@ -990,6 +1009,7 @@ export function readMetaDb( lmdbDatabaseEnvs.set(path, rootStore); } + rootStore.dbisDb?.resetReadTxn(); return initStores(path, rootStore, databaseName, { defaultTable, auditPath, isLegacy }); } catch (error) { error.message += ` opening database ${path}`; @@ -1075,6 +1095,7 @@ function initStores( } else { attributesDbi = rootStore.openDB(INTERNAL_DBIS_NAME, internalDbiInit as any); } + openedStores?.push(attributesDbi); rootStore.dbisDb = markInternalDbiNonVersioned(attributesDbi); } @@ -1283,6 +1304,8 @@ function initStores( indices[attribute.name] = dbi; indices[attribute.name].indexNulls = attribute.indexNulls; } + // the only way a thread that never declares the schema reaches Table.indices + indices[attribute.name].isIndexing = !!attribute.indexingPID; const existingAttribute = existingAttributes.find( (existingAttribute) => existingAttribute.name === attribute.name ); @@ -1433,6 +1456,19 @@ export function resolveBranchPath(baseName: string, appName: string): string { export interface BranchDatabase { tables: Tables; rootStore: RootDatabaseKind; + /** The realpath of the branch directory; what a schema-change signal names to address this branch. */ + path: string; + /** The logical name the application uses (`data`); every Table class in `tables` carries it. */ + databaseName: string; + /** The branch's own store identity, which its blob roots resolve from. */ + storeName: string; + /** + * Every column-family wrapper opened on this store -- by the open, by a reload, or by a table + * declaration -- so `close()` can release them all. Recorded at acquisition rather than + * reconstructed from `tables` at close: a re-declaration displaces the index and catalog wrappers it + * replaces, and a failed declaration can leave one that no class ever held. + */ + openedStores: any[]; /** * Relationships this branch's tables declared, still un-hydrated. They cannot be resolved at open * time: a branch's definitions name the BASE database (its tables carry the base's logical names), @@ -1440,6 +1476,11 @@ export interface BranchDatabase { * base. `hydrateBranchRelationships` finishes the job once the whole branch set is known. */ pendingRelationships: RelationshipHydration[]; + /** + * The application's whole branch set, once `prepareBranches` has opened it, so a reload can hydrate + * a relationship whose target the application also branched against that branch. + */ + relatedBranches?: Map; close(): void; } @@ -1645,10 +1686,10 @@ function branchDirectoryExistsFor(storeName: string): boolean { * worker's shutdown path, so a branch left open on an exiting worker does not linger in the * process-global RocksDB registry. * - * NOT SAFE FOR SCHEMA MUTATION. A branch's Table classes carry the base's logical name, so a - * `dropTable()` or equivalent through one resolves against the global schema and would delete the - * live base Table class — which is why schema operations through a branch are refused - * (branchGuard.ts). + * Schema changes reach a branch only through its own bound factory (`scopedTableFactory`): a + * declaration re-asserted against the branch's store. A branch's Table classes carry the base's + * logical name, so the Table statics (`dropTable()`, `addAttributes()`) — which resolve the global + * schema by that name and would act on the live base table — stay refused (`assertSchemaMutable`). * * A branch's blob roots are a hard-link clone of the base's, taken with the checkpoint, so a row * whose blob predates the branch reads back normally and the branch allocates new file ids in its own @@ -1713,13 +1754,17 @@ export function openBranchDatabase( releaseBranchIdentity(storeName); const stranded = rocksdbDatabaseEnvs.get(path); rocksdbDatabaseEnvs.delete(path); - closeBranchHandles(path, stranded, openedStores); + closeBranchHandles(path, stranded, openedStores, tables); throw error; } let closed = false; const branch: BranchDatabase = { tables, rootStore, + path, + databaseName, + storeName, + openedStores, pendingRelationships: relationshipsToHydrate.splice(queuedRelationshipsAt), close() { // guard on the handle, not on the registrations: those are keyed by path, and a closed @@ -1729,7 +1774,7 @@ export function openBranchDatabase( openBranches.delete(path); releaseBranchIdentity(storeName); rocksdbDatabaseEnvs.delete(path); - closeBranchHandles(path, rootStore, openedStores); + closeBranchHandles(path, rootStore, openedStores, tables); }, }; openBranches.set(path, branch); @@ -1745,17 +1790,32 @@ export function openBranchDatabase( * memoized blob roots in `databasePaths`. A real database is opened once per thread; harper#643 * makes branch open/close routine, so both would grow with branch churn. */ -function closeBranchHandles(path: string, rootStore?: RootDatabaseKind, openedStores: any[] = []): void { +function closeBranchHandles( + path: string, + rootStore?: RootDatabaseKind, + openedStores: any[] = [], + tables: Tables = {} +): void { const reclamationPaths = new Set([path]); (rootStore as any)?.auditStore?.stopAuditCleanup?.(); const closeStore = (store: any, description: string) => { - if (store?.path) reclamationPaths.add(store.path); + if (!store || store.status === 'closed') return; + if (store.path) reclamationPaths.add(store.path); try { - store?.close?.(); + store.close?.(); } catch (error) { logger.warn(`Error closing ${description} for branch database at ${path}`, error); } }; + // the class, before its stores: an expiration timer or a reclamation handler on a closed store + // would otherwise keep firing against it for the life of the process + for (const tableName in tables) { + try { + tables[tableName]?.cleanup?.(); + } catch (error) { + logger.warn(`Error releasing table ${tableName} of branch database at ${path}`, error); + } + } for (const store of openedStores) closeStore(store, 'column family'); if (rootStore) closeDerivedIndexStores(rootStore, `for branch database at ${path}`); closeStore((rootStore as any)?.dbisDb, 'attributes store'); @@ -2312,6 +2372,102 @@ function openIndex(dbiKey: string, rootStore: RootDatabaseKind, attribute: any) * @param replicate */ export function table(tableDefinition: TableDefinition): TableResourceType { + return declareTable(GLOBAL_TARGET, tableDefinition); +} + +/** + * Where a declaration lands. `table()` is bound to the global catalog; a branched application's + * declarations are bound to its branch (`scopedTableFactory`). Everything in `declareTable` that is + * global by construction -- the root store, the `tables` graph a class is published into, the reload + * after a lost create race, who owns the handles it opens -- goes through this, and nothing else does, + * so the unbranched path is the same code with the same objects behind it. + */ +interface TableTarget { + rootStore(databaseName: string, tableName: string): RootDatabaseKind; + tables(databaseName: string): Tables; + /** Another thread created the table this declaration was about to; make `tables` reflect it. */ + reload(databaseName: string): void; + /** Records a column-family wrapper the declaration opened, for whoever closes the store. */ + adopt(store: any): void; + /** Set for a branch: its Table classes refuse DDL, and its schema signals address it by path. */ + branch?: BranchDatabase; +} + +const GLOBAL_TARGET: TableTarget = { + rootStore: (databaseName, tableName) => database({ database: databaseName, table: tableName }), + tables: (databaseName) => databases[databaseName], + reload: () => resetDatabases(), + // a real database's stores live until the process (or `closeDatabase`, which walks the graph) ends + adopt: () => {}, +}; + +/** + * The factory a branched application declares tables through: each declaration goes to the branch + * of the database it names, or to `table()` itself for a database the application did not branch. + * An unbranched application gets `table` by identity -- no wrapper, no per-call routing. + */ +export function scopedTableFactory(branches?: Map): typeof table { + if (!branches?.size) return table; + return function scopedTable(tableDefinition: TableDefinition): TableResourceType { + // `||`, not `??`: `table()` resolves every falsy name to the default database + const branch = branches.get(tableDefinition.database || DEFAULT_DATABASE_NAME); + return branch ? declareTable(branchTarget(branch), tableDefinition) : table(tableDefinition); + }; +} + +function branchTarget(branch: BranchDatabase): TableTarget { + return { + rootStore: () => branch.rootStore, + tables: () => branch.tables, + reload: () => reloadBranch(branch), + adopt: (store) => branch.openedStores.push(store), + branch, + }; +} + +/** + * Re-read a branch's catalog into its `tables`: tables and indexes another thread declared since the + * open (or since the last reload) are opened here, the same way a schema-change rescan does for a + * real database. Existing classes are kept and their attribute lists refreshed. + */ +function reloadBranch(branch: BranchDatabase): void { + const { rootStore, tables, databaseName, storeName, openedStores } = branch; + const queuedRelationshipsAt = relationshipsToHydrate.length; + try { + initStores(rootStore.path, rootStore, databaseName, { destination: tables, storeName, openedStores }); + } finally { + for (const hydration of relationshipsToHydrate.splice(queuedRelationshipsAt)) + queueBranchHydration(branch, hydration); + } + // Until `prepareBranches` has the whole set, a cross-database target cannot be resolved without + // falling through to the base; it hydrates the complete set once. After that, every sibling is + // re-hydrated: the table this reload brought in may be the target a sibling's relationship waited for. + if (!branch.relatedBranches) return; + for (const sibling of branch.relatedBranches.values()) hydrateBranchRelationships(sibling, branch.relatedBranches); +} + +/** One pending hydration per table: a re-declaration replaces the entry the earlier declaration queued. */ +function queueBranchHydration(branch: BranchDatabase, hydration: RelationshipHydration): void { + const existing = branch.pendingRelationships.findIndex( + (pending) => pending.databaseName === hydration.databaseName && pending.tableName === hydration.tableName + ); + if (existing >= 0) branch.pendingRelationships[existing] = hydration; + else branch.pendingRelationships.push(hydration); +} + +/** + * The receiving side of a branch's schema-change signal: a thread that holds this branch open reloads + * it, any other thread has nothing to do. Returns the branch's tables so the caller can address the + * table the signal named. + */ +export function reloadBranchAt(path: string): Tables | undefined { + const branch = openBranches.get(path); + if (!branch) return undefined; + reloadBranch(branch); + return branch.tables; +} + +function declareTable(target: TableTarget, tableDefinition: TableDefinition): TableResourceType { let { table: tableName, database: databaseName, @@ -2350,8 +2506,8 @@ export function table(tableDefinition: TableDefinition): Tabl if (isBranchIdentity(databaseName)) { throw new ClientError(`'${databaseName}' is in use as a branch store identity and cannot be a database name`); } - const rootStore = database({ database: databaseName, table: tableName }); - const tables = databases[databaseName]; + const rootStore = target.rootStore(databaseName, tableName); + const tables = target.tables(databaseName); logger.trace(`Defining ${tableName} in ${databaseName}`); let Table = tables?.[tableName]; if (rootStore.status === 'closed') { @@ -2517,6 +2673,7 @@ export function table(tableDefinition: TableDefinition): Tabl internalDbiInit as any ); } + target.adopt(attributesDbi); markInternalDbiNonVersioned(attributesDbi); exclusiveLock(); // get an exclusive lock on the database so we can verify that we are the only thread creating the table (and assigning the table id) @@ -2525,8 +2682,8 @@ export function table(tableDefinition: TableDefinition): Tabl // table was created while we were setting up; the lock is not reentrant, so release // before the recursive reload releaseLock(); - resetDatabases(); - return table(tableDefinition); + target.reload(databaseName); + return declareTable(target, tableDefinition); } let primaryStore; @@ -2556,9 +2713,12 @@ export function table(tableDefinition: TableDefinition): Tabl } else { primaryStore = (rootStore as any).openDB(dbiName, dbiInit as any); } + target.adopt(primaryStore); unpublishedPrimaryStore = primaryStore; primaryStore = handleLocalTimeForGets(primaryStore, rootStore); - rootStore.databaseName = databaseName; + // only a store no table has loaded yet is unnamed; a branch's store carries its own store + // identity here, which its blob roots resolve from, and must not take the logical name + rootStore.databaseName ??= databaseName; primaryStore.tableId = attributesDbi.getSync(NEXT_TABLE_ID); logger.trace(`Assigning new table id ${primaryStore.tableId} for ${tableName}`); if (!primaryStore.tableId) primaryStore.tableId = 1; @@ -2566,6 +2726,7 @@ export function table(tableDefinition: TableDefinition): Tabl primaryKeyAttribute.tableId = primaryStore.tableId; Table = makeTable({ + isBranch: Boolean(target.branch), primaryStore, auditStore, audit, @@ -2604,6 +2765,7 @@ export function table(tableDefinition: TableDefinition): Tabl } else { (rootStore as any).dbisDb = (rootStore as any).openDB(INTERNAL_DBIS_NAME, internalDbiInit as any); } + target.adopt((rootStore as any).dbisDb); attributesDbi = markInternalDbiNonVersioned((rootStore as any).dbisDb); } Table.dbisDB = attributesDbi; @@ -2698,8 +2860,7 @@ export function table(tableDefinition: TableDefinition): Tabl const abandonedIndexBuild = attribute.indexed && (attributeDescriptor.indexingFailed || - (attributeDescriptor.indexingPID && attributeDescriptor.indexingPID !== process.pid) || - attributeDescriptor.restartNumber < (workerData?.restartNumber ?? manageThreads.restartNumber)); + isAbandonedIndexBuild(attributeDescriptor, workerData?.restartNumber ?? manageThreads.restartNumber)); if (abandonedIndexBuild) { // Recovery is the exception to skipping the handling below, because without it `isIndexing` // stays pinned on with nothing left to clear it and every query on the attribute fails with @@ -2710,6 +2871,7 @@ export function table(tableDefinition: TableDefinition): Tabl } else { if (attribute.indexed) { const dbi = openIndex(dbiKey, rootStore, attribute); + target.adopt(dbi); // Persisting the indexFormat openIndex just resolved adds a field the descriptor lacks // rather than rewriting one it has. Without it an empty index resolves 'versioned', writes // versioned nodes, then re-derives 'legacy' on the next load — see indexFormatNeedsPersist. @@ -2773,6 +2935,7 @@ export function table(tableDefinition: TableDefinition): Tabl // on the main thread, where workerData is undefined (and it is initialized to 1). const currentRestartGeneration = workerData?.restartNumber ?? manageThreads.restartNumber; const dbi = openIndex(dbiKey, rootStore, attribute); + target.adopt(dbi); if (deferredPrimaryRow) indices[attribute.name] = dbi; // private until published; lets the rollback close it // openIndex resolves and stamps attribute.indexFormat for a versioned-capable (RocksDB // custom-object) index. An index created before this field existed has no indexFormat on @@ -2787,8 +2950,7 @@ export function table(tableDefinition: TableDefinition): Tabl changed || indexFormatNeedsPersist || attributeDescriptor?.indexingFailed || - (attributeDescriptor?.indexingPID && attributeDescriptor?.indexingPID !== process.pid) || - attributeDescriptor?.restartNumber < currentRestartGeneration + isAbandonedIndexBuild(attributeDescriptor, currentRestartGeneration) ) { hasChanges = true; exclusiveLock(); @@ -2796,8 +2958,7 @@ export function table(tableDefinition: TableDefinition): Tabl if ( structurallyChanged || attributeDescriptor?.indexingFailed || - (attributeDescriptor?.indexingPID && attributeDescriptor?.indexingPID !== process.pid) || - attributeDescriptor?.restartNumber < currentRestartGeneration + isAbandonedIndexBuild(attributeDescriptor, currentRestartGeneration) ) { hasChanges = true; if (attribute.indexNulls === undefined) attribute.indexNulls = true; @@ -2817,9 +2978,17 @@ export function table(tableDefinition: TableDefinition): Tabl // resumes rather than restarts. Canonicalized to match structurallyChanged above. const indexOptionsChanged = canonicalIndexKey(attributeDescriptor?.indexed) !== canonicalIndexKey(attribute.indexed); - attribute.lastIndexedKey = indexOptionsChanged - ? undefined - : (attributeDescriptor?.lastIndexedKey ?? undefined); + // Only a checkpoint runIndexing stamped with its own key resumes: earlier releases advanced + // lastIndexedKey past failed and unflushed index writes, so any other is a full rebuild. + const uncertifiedCheckpoint = + attributeDescriptor?.lastIndexedKey !== undefined && + (attributeDescriptor.checkpointCertified === undefined || + compareKeys(attributeDescriptor.checkpointCertified, attributeDescriptor.lastIndexedKey) !== 0); + attribute.lastIndexedKey = + indexOptionsChanged || uncertifiedCheckpoint + ? undefined + : (attributeDescriptor?.lastIndexedKey ?? undefined); + if (attribute.lastIndexedKey !== undefined) attribute.checkpointCertified = attribute.lastIndexedKey; // Explicit reindex is the upgrade path from a legacy (un-versioned) custom-index // object store to the versioned, VT-cacheable format. A full rebuild from scratch // (lastIndexedKey === undefined) clears the store and rewrites every node, so the @@ -2843,6 +3012,9 @@ export function table(tableDefinition: TableDefinition): Tabl // the new process reuses the old PID. Cleared on clean completion; left in place // on failure/crash so the next, higher-numbered restart re-triggers the backfill. attribute.restartNumber = currentRestartGeneration; + if (manageThreads.processIncarnation != null) + attribute.indexingIncarnation = manageThreads.processIncarnation; + attribute.indexingBuildId = randomBytes(8).toString('hex'); delete attribute.indexingFailed; // clear failure flag for the new run dbi.isIndexing = true; Object.defineProperty(attribute, 'dbi', { value: dbi, configurable: true, enumerable: false }); @@ -2856,6 +3028,13 @@ export function table(tableDefinition: TableDefinition): Tabl if (attributeDescriptor?.indexingPID && attributeDescriptor.indexingPID !== process.pid) reindexReasons.push(`crash-recovery(pid=${attributeDescriptor.indexingPID})`); if (attributeDescriptor?.restartNumber < currentRestartGeneration) reindexReasons.push('restart-number'); + if (uncertifiedCheckpoint) reindexReasons.push('uncertified-checkpoint'); + if ( + attributeDescriptor?.indexingPID === process.pid && + manageThreads.processIncarnation != null && + attributeDescriptor.indexingIncarnation !== manageThreads.processIncarnation + ) + reindexReasons.push('abandoned-build(previous process incarnation)'); logger.info( `reindex ${databaseName}.${tableName}.${attribute.name}: reason=${reindexReasons.join(',') || 'unknown'}` ); @@ -2869,9 +3048,13 @@ export function table(tableDefinition: TableDefinition): Tabl // workers / a reload would treat the still-partial index as ready and return incomplete results. attribute.indexingPID = attributeDescriptor.indexingPID; attribute.lastIndexedKey = attributeDescriptor.lastIndexedKey; + if (attributeDescriptor.checkpointCertified !== undefined) + attribute.checkpointCertified = attributeDescriptor.checkpointCertified; // Carry the in-progress restart generation too, so persisting this metadata-only // change doesn't drop it and break the crash-recovery trigger for the running backfill. attribute.restartNumber = attributeDescriptor.restartNumber; + attribute.indexingIncarnation = attributeDescriptor.indexingIncarnation; + attribute.indexingBuildId = attributeDescriptor.indexingBuildId; if (attributeDescriptor.indexingFailed) attribute.indexingFailed = attributeDescriptor.indexingFailed; } attributesDbi.put(dbiKey, attribute); @@ -2934,15 +3117,23 @@ export function table(tableDefinition: TableDefinition): Tabl Table.updatedAttributes(); } logger.trace(`${tableName} table loading, running index`); + const branchPath = target.branch?.path; if (attributesToIndex.length > 0 || indicesToRemove.length > 0) { - Table.indexingOperation = runIndexing(Table, attributesToIndex, indicesToRemove); + // captured before the backfill can rewrite the attributes + const buildIds = new Map(attributesToIndex.map((attribute) => [attribute, attribute.indexingBuildId])); + const markSettled = () => markAbandonedIndexBuild(Table, rootStore, buildIds); + Table.indexingOperation = runIndexing(Table, attributesToIndex, indicesToRemove, branchPath).then( + markSettled, + markSettled + ); } else if (hasChanges) signalling.signalSchemaChange( - new SchemaEventMsg(process.pid, 'schema-change', Table.databaseName, Table.tableName) + new SchemaEventMsg(process.pid, 'schema-change', Table.databaseName, Table.tableName, undefined, branchPath) ); Table.origin = origin; - if (hasChanges || refreshRelationshipAttributes) { + // scope-private: replication and other global subscribers must not learn of a branch class + if ((hasChanges || refreshRelationshipAttributes) && !target.branch) { databaseEventsEmitter.emit('updateTable', Table, origin !== 'cluster'); } if (expiration || eviction || scanInterval) @@ -3064,29 +3255,122 @@ export function canonicalizeIndexOptions(value: any): any { } const MAX_OUTSTANDING_INDEXING = 1000; const MIN_OUTSTANDING_INDEXING = 10; -async function runIndexing(Table, attributes, indicesToRemove) { +const INDEXING_YIELD_INTERVAL = 100; +// A resumable checkpoint is written only after a flush (see flushIndexStores), at most once per period +// and never before this many more records: the flush seals every column family in the database, so a +// slow backfill must not impose the period's flush rate on unrelated tables. +let indexingCheckpointPeriodMs = 5000; +let indexingCheckpointMinRecords = 10000; +export function setIndexingCheckpointPeriod(ms: number, minRecords = indexingCheckpointMinRecords) { + const previous = { ms: indexingCheckpointPeriodMs, minRecords: indexingCheckpointMinRecords }; + indexingCheckpointPeriodMs = ms; + indexingCheckpointMinRecords = minRecords; + return previous; +} +const yieldEventTurn = () => new Promise((resolve) => setImmediate(resolve)); +// RocksDB index stores have no WAL (openRocksDatabase defaults disableWAL), so a flush is what makes the +// entries a checkpoint certifies durable. A flush only covers writes issued before it started, so a caller +// never joins one in flight: it joins the next one, which every backfill on that database asking meanwhile +// shares — at most one in flight and one queued. +const indexingFlushes = new WeakMap; queued?: Promise }>(); +function flushIndexStores(rootStore: any): Promise | undefined { + if (!(rootStore instanceof RocksDatabase)) return; + let flushes = indexingFlushes.get(rootStore); + if (!flushes) indexingFlushes.set(rootStore, (flushes = {})); + if (flushes.queued) return flushes.queued; + const start = () => { + flushes.queued = undefined; + const flush = rootStore.flush().finally(() => { + if (flushes.inFlight === flush) flushes.inFlight = undefined; + }); + flushes.inFlight = flush; + return flush; + }; + if (!flushes.inFlight) return start(); + return (flushes.queued = flushes.inFlight.then(start, start)); +} +export function resumeStartKey(attributes: { lastIndexedKey?: any }[]): any { + let start: any; + for (const attribute of attributes) { + if (attribute.lastIndexedKey == undefined) return undefined; + if (start === undefined || compareKeys(attribute.lastIndexedKey, start) < 0) start = attribute.lastIndexedKey; + } + return start; +} + +/** + * Persists the failure marker for a build that ended without running one of runIndexing's own exit + * paths, so something re-triggers it. Fenced on `indexingBuildId` inside the storage engine's catalog + * serialization boundary, because a replacement generation (or another thread declaring different index + * options) can claim the attribute before an outgoing build's promise settles, and marking that would fail + * a live build. The fence read and write stay synchronous, and nothing here may throw because + * `Table.indexingOperation` reaches operations-API callers. + */ +async function markAbandonedIndexBuild(Table, rootStore, buildIds: Map) { + for (const [attribute, buildId] of buildIds) { + try { + let marked; + if (buildId == null || Table.dbisDB.getSync(attribute.key)?.indexingBuildId !== buildId) continue; + const markIfOwned = () => { + const descriptor = Table.dbisDB.getSync(attribute.key); + if (descriptor?.indexingBuildId === buildId && !descriptor.indexingFailed) { + Table.dbisDB.putSync(attribute.key, { ...descriptor, indexingFailed: true }); + marked = true; + } + }; + if (rootStore instanceof RocksDatabase) { + acquireUpdateAttributesLock(rootStore, `abandoned index build '${Table.tableName}.${attribute.name}'`); + try { + markIfOwned(); + } finally { + releaseUpdateAttributesLock(rootStore); + } + } else { + rootStore.transactionSync(markIfOwned); + } + if (marked) + logger.warn( + `Indexing of ${Table.databaseName}.${Table.tableName}.${attribute.name} ended without completing. ` + + `The index stays incomplete and every query on the attribute reports it as not indexed yet; ` + + `the next load of the table retries the backfill from the last checkpoint (indexingFailed=true).` + ); + } catch (error) { + // A store closed by shutdown is the common case, and it cannot be written to at all. + try { + logger.debug(`Could not mark the abandoned index build of ${Table.tableName}.${attribute.name}`, error); + } catch {} + } + } +} +async function runIndexing(Table, attributes, indicesToRemove, branchPath?: string) { + let checkpointing; + let hadIndexingErrors = false; + const attributeErrorReported = {}; + const onIndexPutRejected = (property, error) => { + hadIndexingErrors = true; + if (attributeErrorReported[property]) return; + attributeErrorReported[property] = true; + logger.error(`Error indexing attribute ${property}`, error); + }; + const putRejectionHandlers = attributes.map((attribute) => (error) => onIndexPutRejected(attribute.name, error)); try { logger.info(`Indexing ${Table.tableName} attributes`, attributes); await signalling.signalSchemaChange( - new SchemaEventMsg(process.pid, 'schema-change', Table.databaseName, Table.tableName) + new SchemaEventMsg(process.pid, 'schema-change', Table.databaseName, Table.tableName, undefined, branchPath) ); let lastResolution; for (const index of indicesToRemove) { lastResolution = index.drop(); + if (lastResolution?.then) lastResolution.then(undefined, (error) => onIndexPutRejected(index.name, error)); } let interrupted; - let hadIndexingErrors = false; - const attributeErrorReported = {}; let indexed = 0; const attributesLength = attributes.length; await new Promise((resolve) => setImmediate(resolve)); // yield event turn, indexing should consistently take at least one event turn if (attributesLength > 0) { - let start: any; - for (const attribute of attributes) { - // if we are resuming, we need to start from the last key we indexed by all attributes - if (compareKeys(attribute.lastIndexedKey, start) < 0) start = attribute.lastIndexedKey; - if (attribute.lastIndexedKey == undefined) { - // if we are starting from the beginning, clear out any previous index entries since we are rewriting + const start = resumeStartKey(attributes); + if (start === undefined) { + for (const attribute of attributes) { if (attribute.dbi.clearAsync) { // LMDB, note that we don't need to wait for this to complete, just gets enqueued in front of the other writes attribute.dbi.clearAsync(); @@ -3096,6 +3380,26 @@ async function runIndexing(Table, attributes, indicesToRemove) { } } let outstanding = 0; + // A resumed scan starts at the checkpoint, so it must only name a key whose every predecessor is + // durably indexed: persisted once the writes it covers have settled and flushed, frozen after any + // record fails so the retry re-covers it, and stamped with its own key (see the trigger in table()). + const persistCheckpoint = async (key) => { + if (hadIndexingErrors) return; + try { + await flushIndexStores(Table.primaryStore.rootStore); + const puts = []; + for (const attribute of attributes) { + attribute.lastIndexedKey = key; + attribute.checkpointCertified = key; + puts.push(Table.dbisDB.put(attribute.key, attribute)); + } + await Promise.all(puts); + } catch (error) { + logger.warn(`Could not persist the indexing checkpoint for ${Table.tableName}`, error); + } + }; + let nextCheckpointAt = performance.now() + indexingCheckpointPeriodMs; + let nextCheckpointRecord = indexingCheckpointMinRecords; // this means that a new attribute has been introduced that needs to be indexed for (const { key, value: record } of Table.primaryStore.getRange({ start, @@ -3103,7 +3407,7 @@ async function runIndexing(Table, attributes, indicesToRemove) { versions: true, snapshot: false, // don't hold a read transaction this whole time })) { - if (!record) continue; // deletion entry + const atInterval = ++indexed % INDEXING_YIELD_INTERVAL === 0; // TODO: Do we ever need to interrupt due to a schema change that was not a restart? //if (Table.schemaVersion !== schemaVersion) return; // break out if there are any schema changes and let someone else pick it up outstanding++; @@ -3115,71 +3419,77 @@ async function runIndexing(Table, attributes, indicesToRemove) { // we index, that's fine because indexing is idempotent, we can just put the same values again. If it changes // during the indexing, the indexing here will fail. This is also fine because it means the other thread will have // performed indexing and we don't need to do anything further - for (let i = 0; i < attributesLength; i++) { - const attribute = attributes[i]; - const property = attribute.name; - const index = attribute.dbi; - try { - const resolver = attribute.resolve; - const value = record && (resolver ? resolver(record) : record[property]); - if (index.customIndex) { - index.customIndex.index(key, value); - didSynchronousIndexing = true; - continue; - } - const values = getIndexedValues(value, index.indexNulls); - if (values) { - for (let i = 0, l = values.length; i < l; i++) { - lastResolution = index.put(values[i], key); + if (record) { + for (let i = 0; i < attributesLength; i++) { + const attribute = attributes[i]; + const property = attribute.name; + const index = attribute.dbi; + const onPutRejected = putRejectionHandlers[i]; + try { + const resolver = attribute.resolve; + const value = record && (resolver ? resolver(record) : record[property]); + if (index.customIndex) { + index.customIndex.index(key, value); + didSynchronousIndexing = true; + continue; + } + const values = getIndexedValues(value, index.indexNulls); + if (values) { + for (let i = 0, l = values.length; i < l; i++) { + lastResolution = index.put(values[i], key); + if (lastResolution?.then) lastResolution.then(undefined, onPutRejected); + } + } + } catch (error) { + hadIndexingErrors = true; + if (!attributeErrorReported[property]) { + // just report an indexing error once per attribute so we don't spam the logs. + // A store closed by worker shutdown surfaces here as "Database not open"; that is + // a benign interruption (the next generation re-runs the backfill), so don't log + // it as an error — the outer catch returns quietly once the iterator also throws. + attributeErrorReported[property] = true; + if (Table.primaryStore?.rootStore?.status === 'closed') + logger.debug(`Indexing attribute ${property} interrupted by store shutdown`, error); + else logger.error(`Error indexing attribute ${property}`, error); } - } - } catch (error) { - hadIndexingErrors = true; - if (!attributeErrorReported[property]) { - // just report an indexing error once per attribute so we don't spam the logs. - // A store closed by worker shutdown surfaces here as "Database not open"; that is - // a benign interruption (the next generation re-runs the backfill), so don't log - // it as an error — the outer catch returns quietly once the iterator also throws. - attributeErrorReported[property] = true; - if (Table.primaryStore?.rootStore?.status === 'closed') - logger.debug(`Indexing attribute ${property} interrupted by store shutdown`, error); - else logger.error(`Error indexing attribute ${property}`, error); } } } when( lastResolution, () => outstanding--, - (error) => { - outstanding--; - hadIndexingErrors = true; - logger.error(error); - } + () => outstanding-- ); if (workerData && workerData.restartNumber !== manageThreads.restartNumber) { interrupted = true; } - if (++indexed % 100 === 0 || interrupted) { - // occasionally update our progress so if we crash, we can resume - for (const attribute of attributes) { - attribute.lastIndexedKey = key; - Table.dbisDB.put(attribute.key, attribute); + if (interrupted) { + try { + await lastResolution; + } catch { + // already counted and logged by the rejection handler above } - if (interrupted) return; + await checkpointing; + await persistCheckpoint(key); + return; + } + if (atInterval && indexed >= nextCheckpointRecord && performance.now() >= nextCheckpointAt) { + nextCheckpointAt = performance.now() + indexingCheckpointPeriodMs; + nextCheckpointRecord = indexed + indexingCheckpointMinRecords; + await checkpointing; + checkpointing = when( + lastResolution, + () => persistCheckpoint(key), + () => {} + ); } if (outstanding > MAX_OUTSTANDING_INDEXING) await lastResolution; - else if (outstanding > MIN_OUTSTANDING_INDEXING) - await new Promise((resolve) => setImmediate(resolve)); // yield event turn, don't want to use all computation - else if (didSynchronousIndexing) await new Promise((resolve) => setImmediate(resolve)); // custom indexes (e.g. HNSW) index synchronously and never raise `outstanding`; without this yield a large backfill runs in a single event-loop turn, starving keepalive/replication and queries and never letting the isIndexing flag be observed + if (atInterval || didSynchronousIndexing || outstanding > MIN_OUTSTANDING_INDEXING) await yieldEventTurn(); } } - // Await the last pending put. If it rejects, that is also an indexing error. - // Note: the when() calls above already attach rejection handlers to each record's - // last-put promise; this try-catch specifically handles the case where lastResolution - // itself rejects (i.e. the very last put in the loop failed) which would otherwise - // throw past the hadIndexingErrors check to the outer catch. The broader issue of - // unhandled rejections from non-last puts in multi-value attributes is pre-existing - // and out of scope for this fix. + await checkpointing; + // Await the last pending put. If it rejects, that is also an indexing error (already counted by + // onIndexPutRejected); catching it here keeps it from escaping to the outer catch. try { await lastResolution; } catch (error) { @@ -3190,6 +3500,16 @@ async function runIndexing(Table, attributes, indicesToRemove) { // microtasks when their tracked promise settles) have a chance to set hadIndexingErrors // before we decide whether to mark indexing as complete. await new Promise((resolve) => setImmediate(resolve)); + // the tail since the last checkpoint is not durable until flushed; announcing the index complete + // before that would outlive a crash that loses it + if (!hadIndexingErrors) { + try { + await flushIndexStores(Table.primaryStore.rootStore); + } catch (error) { + hadIndexingErrors = true; + logger.error(`Could not flush the indexes of ${Table.tableName} before marking them complete`, error); + } + } if (hadIndexingErrors) { // Some records failed to index. Persist the failure marker in the descriptor so // the next call to table() (including after a restart with a fresh PID) re-triggers @@ -3218,9 +3538,12 @@ async function runIndexing(Table, attributes, indicesToRemove) { // update the attributes to indicate that we are finished for (const attribute of attributes) { delete attribute.lastIndexedKey; + delete attribute.checkpointCertified; delete attribute.indexingPID; delete attribute.indexingFailed; delete attribute.restartNumber; + delete attribute.indexingIncarnation; + delete attribute.indexingBuildId; attribute.dbi.isIndexing = false; // Also clear isIndexing on the currently-active dbi in Table.indices, which may // differ from attribute.dbi if a resetDatabases() call during this migration @@ -3232,11 +3555,12 @@ async function runIndexing(Table, attributes, indicesToRemove) { await lastResolution; // now notify all the threads that we are done and the index is ready to use await signalling.signalSchemaChange( - new SchemaEventMsg(process.pid, 'indexing-finished', Table.databaseName, Table.tableName) + new SchemaEventMsg(process.pid, 'indexing-finished', Table.databaseName, Table.tableName, undefined, branchPath) ); logger.info(`Finished indexing ${Table.tableName} attributes`, attributes); } } catch (error) { + await checkpointing; // A worker shutting down closes its stores mid-backfill, so the range iterator or a // put throws (e.g. "Database not open" / "Iterator not initialized"). This is an // interruption, not a data error: the next worker generation re-runs the backfill via diff --git a/resources/defineTable.ts b/resources/defineTable.ts index dd43541ab6..e1b1d8b361 100644 --- a/resources/defineTable.ts +++ b/resources/defineTable.ts @@ -403,8 +403,21 @@ function compileTypeDef(name: string, shape: Shape, options: DefineTableOptions) * index changes) through the same evolution path GraphQL reloads take. */ export function defineTable(name: string, shape: S, options: DefineTableOptions = {}): TableHandle { + return defineTableUsing(table, name, shape, options); +} + +/** + * `defineTable` through a specific table factory: the one a branched application's scope hands out + * (`scopedTableFactory`), so the table lands in its branch. Internal -- the public entry is `defineTable`. + */ +export function defineTableUsing( + tableFactory: typeof table, + name: string, + shape: S, + options: DefineTableOptions = {} +): TableHandle { const typeDef = compileTypeDef(name, shape, options); - const tableClass = table(typeDef); + const tableClass = tableFactory(typeDef); typeDef.tableClass = tableClass; return tableClass as TableHandle; } diff --git a/resources/derivedIndexRegistry.ts b/resources/derivedIndexRegistry.ts index 040e26dde2..22fd7b7c1e 100644 --- a/resources/derivedIndexRegistry.ts +++ b/resources/derivedIndexRegistry.ts @@ -1,10 +1,26 @@ const registrations = new WeakMap>(); +const admissions = new WeakMap string | undefined>>>(); -export function registerDerivedIndexTables(auditStore: object, tableIds: Iterable): () => void { +/** `admission` returns a reason while writes to these tables must be rejected. */ +export function registerDerivedIndexTables( + auditStore: object, + tableIds: Iterable, + admission?: () => string | undefined +): () => void { const registeredTableIds = new Set(tableIds); let counts = registrations.get(auditStore); if (!counts) registrations.set(auditStore, (counts = new Map())); for (const tableId of registeredTableIds) counts.set(tableId, (counts.get(tableId) ?? 0) + 1); + let checks: Map string | undefined>> | undefined; + if (admission) { + checks = admissions.get(auditStore); + if (!checks) admissions.set(auditStore, (checks = new Map())); + for (const tableId of registeredTableIds) { + let byTable = checks.get(tableId); + if (!byTable) checks.set(tableId, (byTable = [])); + byTable.push(admission); + } + } let registered = true; return () => { if (!registered) return; @@ -13,11 +29,28 @@ export function registerDerivedIndexTables(auditStore: object, tableIds: Iterabl const count = counts!.get(tableId); if (count === 1) counts!.delete(tableId); else if (count) counts!.set(tableId, count - 1); + if (admission && checks) { + const byTable = checks.get(tableId); + const index = byTable?.indexOf(admission) ?? -1; + if (byTable && index >= 0) byTable.splice(index, 1); + if (byTable?.length === 0) checks.delete(tableId); + } } if (counts!.size === 0) registrations.delete(auditStore); + if (checks?.size === 0) admissions.delete(auditStore); }; } export function hasDerivedIndexRegistration(auditStore: object, tableId: number): boolean { return registrations.get(auditStore)?.has(tableId) ?? false; } + +/** The reason a write to this table must currently be rejected, or undefined when writes are admitted. */ +export function derivedIndexWriteRejection(auditStore: object, tableId: number): string | undefined { + const byTable = admissions.get(auditStore)?.get(tableId); + if (!byTable) return; + for (let i = 0; i < byTable.length; i++) { + const reason = byTable[i](); + if (reason) return reason; + } +} diff --git a/resources/derivedIndexRuntime.ts b/resources/derivedIndexRuntime.ts index 95c5bfa737..a4b91732f0 100644 --- a/resources/derivedIndexRuntime.ts +++ b/resources/derivedIndexRuntime.ts @@ -19,7 +19,11 @@ export type DerivedIndexCursor = { logs: Record; }; -export type DerivedIndexState = { kind: 'record'; version: number; projection: unknown } | { kind: 'absent' }; +export type DerivedIndexState = + | { kind: 'record'; version: number; projection: unknown } + | { kind: 'absent' } + /** The projection rejected the record (a 4xx-classified error); the backend removes any entry and counts it. */ + | { kind: 'unindexable'; version: number; reason: string }; export type DerivedIndexMutation = { tableId: number; @@ -37,45 +41,223 @@ export type DerivedIndexTransaction = { export type DerivedIndexBatch = { ownerEpoch: bigint; transactions: DerivedIndexTransaction[]; - through: DerivedIndexCursor; + /** + * Last-write-wins view over the distinct `(tableId, writeKeyId(recordId))` keys of the batch, in + * first-occurrence order, each carrying the last `logVersion` and the same resolved `state` + * object as its occurrences in `transactions`. + */ + records: DerivedIndexMutation[]; + /** + * Cursor vector this batch completes. Absent on a rebuild scan chunk: such a batch advances no + * cursor and the backend's durable cursor must stay `undefined` until a batch carrying `through` + * has been made durable. + */ + through?: DerivedIndexCursor; + bytes: number; + rebuild?: true; }; +export type DerivedIndexFlushReason = 'age' | 'threshold' | 'shutdown'; + +export type DerivedIndexReadinessState = 'unknown' | 'ready' | 'rebuilding' | 'needs-rebuild' | 'unavailable'; + +/** + * Why an index is not `ready`, as a code every worker can read from shared memory. The owner's log + * line carries the full message; the code is what a peer or a successor can act on. + */ +export type DerivedIndexReadinessReason = + | 'none' + | 'cursor-missing' + | 'cursor-unoffered' + | 'log-missing' + | 'log-retention' + | 'log-corrupt' + | 'reload' + | 'backend-failed' + | 'runner-failed' + | 'condemned' + | 'shutdown-failed' + | 'rebuild-failed' + | 'rebuild-exhausted' + | 'rebuild-requested'; + +export type DerivedIndexReadiness = { + state: DerivedIndexReadinessState; + reason?: DerivedIndexReadinessReason; + /** The most recently minted owner epoch; a backend fences queued work against it through `isOwnerEpoch`. */ + ownerEpoch: bigint; + rebuildAttempts: number; +}; + +export interface DerivedIndexBackendHost { + /** True while `epoch` is the most recently minted owner epoch for this backend. */ + isOwnerEpoch(epoch: bigint): boolean; + getReadiness(): DerivedIndexReadiness; +} + +/** + * A backend queues expensive work and publishes durability later: `deliver()` may only enqueue, a + * barrier completes asynchronously, and the durable cursor trails delivery. Work that survives a + * method return is the safety boundary, so every backend provides the epoch fence, the barrier + * request and the quiescence handshake the handoff protocol needs, even one that happens to complete + * everything before returning. + */ export interface DerivedIndexBackend { readonly id: string; + /** Receives the epoch fence and readiness reader before any delivery. */ + attach(host: DerivedIndexBackendHost): void; getDurableCursor(): DerivedIndexCursor | undefined; deliver(batch: DerivedIndexBatch): DerivedIndexDeliveryResult; + /** Request a durability barrier; the backend completes it and wakes through `onStateChange`. */ + flush(reason: DerivedIndexFlushReason): void | Promise; + /** + * Stop accepting work for `ownerEpoch`, settle or discard what is queued, and resolve once nothing + * further will be applied or published for it. A rejection keeps the runner lock held. + */ + shutdown(ownerEpoch: bigint): void | Promise; onStateChange(wake: (change?: DerivedIndexBackendStateChange) => void): () => void; + /** + * Destroy index state and the durable cursor; `getDurableCursor()` must return `undefined` + * afterwards. Crash safety is the backend's: its first durable action must invalidate the cursor + * (or the generation the cursor belongs to) before anything destructive, so an interrupted reset + * reopens as cursorless rather than as a valid cursor over partially destroyed state. Shared + * readiness is process memory and is no evidence after a restart. + */ + reset?(ownerEpoch: bigint): void | Promise; } export type DerivedIndexBackendStateChange = 'changed' | 'accepted-work-lost' | 'failed'; +export type DerivedIndexRunnerOptions = { + maxTransactionsPerTurn?: number; + maxBytesPerTurn?: number; + maxMillisecondsPerTurn?: number; + /** Hard bound on distinct records resolved per chunk; an oversized transaction is cut here. */ + maxChunkRecords?: number; + /** Estimated payload bytes after which a chunk stops adding complete transactions. */ + maxChunkBytes?: number; + maxAcceptedBatchesAhead?: number; + maxFlushAgeMilliseconds?: number; + flushAfterMutations?: number; + flushAfterBytes?: number; + rebuildBackoffMilliseconds?: number; + maxRebuildBackoffMilliseconds?: number; + maxRebuildAttempts?: number; + /** + * Opt-in writer backpressure: while the index is further behind than this, user writes to its + * tables fail with a retryable 503 on every worker. 0 (the default) means no policy. + */ + maxLagMilliseconds?: number; +}; + export type DerivedIndexRegistration = { backend: DerivedIndexBackend; projections: ReadonlyMap unknown>; + options?: DerivedIndexRunnerOptions; }; -export type DerivedIndexRecord = { version: number; value: unknown } | undefined; +/** + * `size` is the stored byte size of the record when known; it bounds the projection's size without + * serializing it. Resolve a missing, deleted or evicted record as `undefined`; a present entry is + * projected as-is, so an undecodable body fails closed instead of silently leaving the index. + */ +export type DerivedIndexRecord = { version: number; value: unknown; size?: number } | undefined; -export type DerivedIndexRuntimeOptions = { - maxTransactionsPerTurn?: number; - maxBytesPerTurn?: number; - maxMillisecondsPerTurn?: number; - maxAcceptedBatchesAhead?: number; +/** A scan record whose `value` is null or undefined is a tombstone, and one whose `recordId` is a symbol is a Harper-internal store entry; neither is indexed. */ +export type DerivedIndexScanRecord = { recordId: Id; version: number; value: unknown; size?: number }; + +export type DerivedIndexRuntimeOptions = DerivedIndexRunnerOptions & { idleGraceMilliseconds?: number; now?: () => number; + /** Iterates every current record of a table for the rebuild scan; without it a rebuild cannot run. */ + scanRecords?: (tableId: number) => Iterable; }; export type DerivedIndexRunnerStatus = - | { state: 'idle' | 'running' | 'deferred' | 'waiting-durable' | 'stopped'; ownerEpoch?: bigint } - | { state: 'needs-rebuild'; reason: string; ownerEpoch?: bigint }; + | { + state: 'idle' | 'running' | 'deferred' | 'waiting-durable' | 'stopped' | 'rebuilding'; + ownerEpoch?: bigint; + } + | { state: 'needs-rebuild' | 'unavailable'; reason: string; ownerEpoch?: bigint }; + +export type DerivedIndexRunnerMetrics = { + readiness: DerivedIndexReadiness; + acceptedBatches: number; + acceptedBytes: number; + acceptedMutations: number; + deferredBytes: number; + oldestAcceptedAgeMilliseconds: number; + /** Lag between the latest transaction this runner has read and the durable cursor; blind while parked. */ + cursorLagMilliseconds: number; + /** How long the runner has been parked on backend backpressure or the durability ceiling. */ + stalledMilliseconds: number; + unindexableRecords: number; + rebuildAttempts: number; + rebuiltRecords: number; + /** How long the current epoch's quiescence (backend shutdown) has been pending. */ + quiescenceAgeMilliseconds: number; +}; + +type ResolvedRunnerOptions = Required & { + idleGraceMilliseconds: number; + now: () => number; +}; const ELIGIBLE_ACTIONS = new Set(['put', 'patch', 'delete', 'invalidate', 'relocate', 'evict']); +const READINESS_STATES: DerivedIndexReadinessState[] = [ + 'unknown', + 'ready', + 'rebuilding', + 'needs-rebuild', + 'unavailable', +]; +const READINESS_REASONS: DerivedIndexReadinessReason[] = [ + 'none', + 'cursor-missing', + 'cursor-unoffered', + 'log-missing', + 'log-retention', + 'log-corrupt', + 'reload', + 'backend-failed', + 'runner-failed', + 'condemned', + 'shutdown-failed', + 'rebuild-failed', + 'rebuild-exhausted', + 'rebuild-requested', +]; +const CONDEMNED_MARKER = new Uint8Array([1]); +// One shared allocation per backend: five independently read Int32 words, then the owner-epoch +// counter. Each word is self-consistent on its own; nothing needs to observe two of them atomically. +const READINESS_WORDS = 6; +const READINESS_EPOCH_OFFSET = READINESS_WORDS * 4; +export const READINESS_BYTES = READINESS_EPOCH_OFFSET + 8; +const READINESS_STATE = 0; +const READINESS_REASON = 1; +const READINESS_ATTEMPTS = 2; +const READINESS_REBUILD_REQUEST = 3; +const READINESS_LAG_EXCEEDED = 4; + +/** A failure raised inside the collector that already knows its shareable reason. */ +class RunnerError extends Error { + code: DerivedIndexReadinessReason; + constructor(code: DerivedIndexReadinessReason, message: string) { + super(message); + this.code = code; + } +} + export class DerivedIndexRuntime { #logStore: RocksTransactionLogStore; #resolveRecord: (tableId: number, recordId: Id) => DerivedIndexRecord; - #options: Required; + #scanRecords?: (tableId: number) => Iterable; + #options: ResolvedRunnerOptions; #runners = new Map(); + #pendingStops = new Set>(); + #heldRunners = new Map }>(); + #stopping?: Promise; #onCommit = () => this.wake(); #listening = false; #stopped = false; @@ -87,22 +269,28 @@ export class DerivedIndexRuntime { ) { this.#logStore = logStore; this.#resolveRecord = resolveRecord; + this.#scanRecords = options.scanRecords; this.#options = { - maxTransactionsPerTurn: options.maxTransactionsPerTurn ?? 256, - maxBytesPerTurn: options.maxBytesPerTurn ?? 4 * 1024 * 1024, - maxMillisecondsPerTurn: options.maxMillisecondsPerTurn ?? 5, - maxAcceptedBatchesAhead: Math.max(1, options.maxAcceptedBatchesAhead ?? 64), + ...resolveRunnerOptions(options), idleGraceMilliseconds: options.idleGraceMilliseconds ?? 30_000, now: options.now ?? Date.now, }; } - register(registration: DerivedIndexRegistration): () => void { + register(registration: DerivedIndexRegistration): () => Promise { if (this.#stopped) throw new Error('Derived index runtime is stopped'); if (!registration.backend.id) throw new Error('Derived index backend id is required'); + for (const hook of ['attach', 'flush', 'shutdown'] as const) { + if (typeof registration.backend[hook] !== 'function') + throw new TypeError(`Derived index backend '${registration.backend.id}' must implement ${hook}()`); + } if (this.#runners.has(registration.backend.id)) throw new Error(`Derived index backend '${registration.backend.id}' is already registered`); - const runner = new DerivedIndexRunner(this.#logStore, this.#resolveRecord, registration, this.#options); + const runner = new DerivedIndexRunner(this.#logStore, this.#resolveRecord, this.#scanRecords, registration, { + ...resolveRunnerOptions(registration.options, this.#options), + idleGraceMilliseconds: this.#options.idleGraceMilliseconds, + now: this.#options.now, + }); this.#runners.set(registration.backend.id, runner); if (!this.#listening) { this.#logStore.rootStore.on('committed', this.#onCommit); @@ -110,13 +298,25 @@ export class DerivedIndexRuntime { } runner.wake(true); return () => { - if (this.#runners.get(registration.backend.id) !== runner) return; - this.#runners.delete(registration.backend.id); - runner.stop(); - this.#stopListeningIfIdle(); + if (this.#runners.get(registration.backend.id) === runner) { + this.#runners.delete(registration.backend.id); + this.#stopListeningIfIdle(); + } + return this.#track(runner, runner.stop()); }; } + #track(runner: DerivedIndexRunner, stopped: Promise): Promise { + this.#pendingStops.add(stopped); + // A failed shutdown stays pending and its runner stays reachable, so a later stop() keeps + // reporting the held lock and requestRebuild() can retry releasing it. + stopped.then( + () => this.#pendingStops.delete(stopped), + () => this.#heldRunners.set(runner.id, { runner, stopped }) + ); + return stopped; + } + wake() { if (this.#stopped) return; for (const runner of this.#runners.values()) runner.wake(); @@ -126,12 +326,49 @@ export class DerivedIndexRuntime { return this.#runners.get(backendId)?.status; } - stop() { - if (this.#stopped) return; + /** Shared readiness published by whichever worker owns the index; readable on every worker. */ + getReadiness(backendId: string): DerivedIndexReadiness { + return this.#runners.get(backendId)?.getReadiness() ?? readDerivedIndexReadiness(this.#logStore, backendId); + } + + getMetrics(backendId: string): DerivedIndexRunnerMetrics | undefined { + return this.#runners.get(backendId)?.getMetrics(); + } + + /** Force a rebuild (or retry one that became `unavailable`). Returns false when the backend cannot be rebuilt by the runtime. */ + requestRebuild(backendId: string): boolean { + const held = this.#heldRunners.get(backendId); + if (held) { + const released = held.runner.retryRelease(); + this.#pendingStops.add(released); + released.then( + () => { + this.#pendingStops.delete(released); + this.#pendingStops.delete(held.stopped); + if (this.#heldRunners.get(backendId) === held) this.#heldRunners.delete(backendId); + }, + () => {} + ); + } + const runner = this.#runners.get(backendId); + if (runner) return runner.requestRebuild(); + return held !== undefined; + } + + /** Resolves once every runner has released ownership and its backend shutdown has settled. */ + stop(): Promise { + if (this.#stopping) return this.#stopping; this.#stopped = true; - for (const runner of this.#runners.values()) runner.stop(); + for (const runner of this.#runners.values()) this.#track(runner, runner.stop()); this.#runners.clear(); this.#stopListening(); + this.#stopping = Promise.allSettled([...this.#pendingStops]).then((results) => { + const failures = results.filter((result) => result.status === 'rejected').map((result) => result.reason); + if (failures.length === 1) throw failures[0]; + if (failures.length) throw new AggregateError(failures, 'derived index backends failed to shut down'); + }); + this.#stopping.catch(() => {}); + return this.#stopping; } #stopListeningIfIdle() { @@ -145,50 +382,218 @@ export class DerivedIndexRuntime { } } +function resolveRunnerOptions( + options: DerivedIndexRunnerOptions | undefined, + defaults?: Required +): Required { + const base: Required = defaults ?? { + maxTransactionsPerTurn: 256, + maxBytesPerTurn: 4 * 1024 * 1024, + maxMillisecondsPerTurn: 5, + maxChunkRecords: 4096, + maxChunkBytes: 4 * 1024 * 1024, + maxAcceptedBatchesAhead: 64, + maxFlushAgeMilliseconds: 1000, + flushAfterMutations: 4096, + flushAfterBytes: 8 * 1024 * 1024, + rebuildBackoffMilliseconds: 1000, + maxRebuildBackoffMilliseconds: 300_000, + maxRebuildAttempts: 8, + maxLagMilliseconds: 0, + }; + if (!options) return base; + return { + maxTransactionsPerTurn: options.maxTransactionsPerTurn ?? base.maxTransactionsPerTurn, + maxBytesPerTurn: options.maxBytesPerTurn ?? base.maxBytesPerTurn, + maxMillisecondsPerTurn: options.maxMillisecondsPerTurn ?? base.maxMillisecondsPerTurn, + maxChunkRecords: Math.max(1, options.maxChunkRecords ?? base.maxChunkRecords), + maxChunkBytes: options.maxChunkBytes ?? base.maxChunkBytes, + maxAcceptedBatchesAhead: Math.max(1, options.maxAcceptedBatchesAhead ?? base.maxAcceptedBatchesAhead), + maxFlushAgeMilliseconds: options.maxFlushAgeMilliseconds ?? base.maxFlushAgeMilliseconds, + flushAfterMutations: options.flushAfterMutations ?? base.flushAfterMutations, + flushAfterBytes: options.flushAfterBytes ?? base.flushAfterBytes, + rebuildBackoffMilliseconds: options.rebuildBackoffMilliseconds ?? base.rebuildBackoffMilliseconds, + maxRebuildBackoffMilliseconds: options.maxRebuildBackoffMilliseconds ?? base.maxRebuildBackoffMilliseconds, + maxRebuildAttempts: options.maxRebuildAttempts ?? base.maxRebuildAttempts, + maxLagMilliseconds: Math.max(0, options.maxLagMilliseconds ?? base.maxLagMilliseconds), + }; +} + +/** Catch-up is proven only at a durable barrier, so a lag budget below two flush ages would trip on cadence alone. */ +function effectiveLagBudget(options: Required): number { + return options.maxLagMilliseconds > 0 ? Math.max(options.maxLagMilliseconds, 2 * options.maxFlushAgeMilliseconds) : 0; +} + +type OfferedProgress = { cursor: DerivedIndexCursor; bytes: number; mutations: number; acceptedAt: number }; + +type CollectedKey = { recordId: Id; logVersion: number; sizeHint: number | undefined }; + +type CollectedTransaction = { + logName: string; + timestamp: number; + keys: Map>; + keyCount: number; + complete: boolean; +}; + +type Chunk = { + batch: DerivedIndexBatch; + resolved: Map>; + started: number; +}; + +const CONTINUE = null; + class DerivedIndexRunner { #logStore: RocksTransactionLogStore; #resolveRecord: (tableId: number, recordId: Id) => DerivedIndexRecord; + #scanRecords?: (tableId: number) => Iterable; #registration: DerivedIndexRegistration; - #options: Required; + #options: ResolvedRunnerOptions; #lockKey: string; + #markerKey: symbol; + #markersSupported: boolean; #iterator?: Iterator; #iterable?: TransactionLogIterable; #knownLogs = new Set(); #pendingTimestamps = new Map(); #seenTimestamps = new Map>(); - #offeredCursors: DerivedIndexCursor[] = []; #offered?: DerivedIndexCursor; + #offeredCursors: OfferedProgress[] = []; + #unanchoredBytes = 0; + #unanchoredMutations = 0; + #unanchoredAcceptedAt = 0; #pendingBatch?: DerivedIndexBatch; + #carried: CollectedTransaction[] = []; + #latestSeen = new Map(); + #stalledSince?: number; + #lastCaughtUpAt?: number; + #lagTimer?: NodeJS.Timeout; + #lockRetryTimer?: NodeJS.Timeout; + #lagBudget: number; #scheduled = false; #waitingForLock = false; #owned = false; #stopped = false; + #generation = 0; #idleTimer?: NodeJS.Timeout; + #flushTimer?: NodeJS.Timeout; + #rebuildTimer?: NodeJS.Timeout; + #unflushedBytes = 0; + #unflushedMutations = 0; + #releasing?: Promise; + #releasingSince?: number; + #releaseFailure?: Error; + #stopResult?: Promise; + #heldLock = false; + #quiescing?: { epoch: bigint; promise: Promise; since: number }; + #condemned = false; + #unreadSince?: number; + #reachedEndOfLog = false; + #rebuilding = false; + #rebuildRequested = false; + #boundaryPending = false; + #rebuildAttempts = 0; + #rebuiltRecords = 0; + #unindexableRecords = 0; + #allUnindexableWarned = false; + #rebuildWaiter?: () => void; + #rebuildWakePending = false; #unsubscribeBackend: () => void; #unregisterTables: () => void; #ownerEpoch?: bigint; + #readinessBuffer: SharedReadinessBuffer; + #sharedViews: SharedViews; + #resetting?: Promise; status: DerivedIndexRunnerStatus = { state: 'idle' }; + get id() { + return this.#registration.backend.id; + } + + retryRelease(): Promise { + if (!this.#heldLock) return this.#stopResult ?? Promise.resolve(); + this.#heldLock = false; + this.#releaseFailure = undefined; + this.#owned = true; + this.#release(); + this.#stopResult = (this.#releasing ?? Promise.resolve()).then(() => { + if (this.#releaseFailure) throw this.#releaseFailure; + this.#unregisterTables(); + }); + this.#stopResult.catch(() => {}); + return this.#stopResult; + } + constructor( logStore: RocksTransactionLogStore, resolveRecord: (tableId: number, recordId: Id) => DerivedIndexRecord, + scanRecords: ((tableId: number) => Iterable) | undefined, registration: DerivedIndexRegistration, - options: Required + options: ResolvedRunnerOptions ) { this.#logStore = logStore; this.#resolveRecord = resolveRecord; + this.#scanRecords = scanRecords; this.#registration = registration; this.#options = options; this.#lockKey = `derived-index:${registration.backend.id}:runner`; - this.#unsubscribeBackend = registration.backend.onStateChange((change = 'changed') => - this.#backendStateChanged(change) + this.#markerKey = Symbol.for(`derived-index:${registration.backend.id}:condemned`); + const root = logStore.rootStore as { getSync?: unknown; removeSync?: unknown } | undefined; + this.#markersSupported = + typeof logStore.putSync === 'function' && + typeof root?.getSync === 'function' && + typeof root?.removeSync === 'function'; + this.#lagBudget = effectiveLagBudget(options); + this.#readinessBuffer = readinessBuffer(logStore, registration.backend.id, () => { + if (this.#owned) this.wake(true); + }); + this.#sharedViews = sharedViewsOf(this.#readinessBuffer); + try { + registration.backend.attach({ + isOwnerEpoch: (epoch) => Atomics.load(this.#sharedViews.epoch, 0) === epoch, + getReadiness: () => this.getReadiness(), + }); + this.#unsubscribeBackend = registration.backend.onStateChange((change = 'changed') => + this.#backendStateChanged(change) + ); + } catch (error) { + this.#readinessBuffer.cancel?.(); + throw error; + } + this.#unregisterTables = registerDerivedIndexTables( + logStore, + registration.projections.keys(), + this.#lagBudget > 0 ? () => this.#writeRejection() : undefined ); - this.#unregisterTables = registerDerivedIndexTables(logStore, registration.projections.keys()); + } + + /** + * rocksdb-js wraps one process-wide native allocation per key in a new external ArrayBuffer on every + * call and never re-seeds an existing entry, so each view is fetched once and held for the runner's + * life; the held wrapper keeps the allocation alive. + */ + #shared(): SharedViews { + return this.#sharedViews; } wake(fromBackend = false) { - if (this.#stopped || this.status.state === 'needs-rebuild') return; - if (!fromBackend && (this.status.state === 'deferred' || this.status.state === 'waiting-durable')) return; + if (this.#stopped || this.#rebuilding) return; + if (!fromBackend && this.#lagBudget > 0) this.#unreadSince ??= this.#options.now(); + if (this.status.state === 'unavailable') { + if ( + this.#heldLock || + Atomics.load(this.#shared().words, READINESS_STATE) === READINESS_STATES.indexOf('unavailable') + ) + return; + this.status = { state: 'idle' }; + } + // A shared rebuild request must reach an owner parked on backpressure or backoff at its next wake. + const requested = Atomics.load(this.#shared().words, READINESS_REBUILD_REQUEST) === 1; + if (!requested) { + if (this.status.state === 'needs-rebuild' && (this.#rebuildTimer || !this.#rebuildRequested)) return; + if (!fromBackend && (this.status.state === 'deferred' || this.status.state === 'waiting-durable')) return; + } if (this.#idleTimer) { clearTimeout(this.#idleTimer); this.#idleTimer = undefined; @@ -203,18 +608,183 @@ class DerivedIndexRunner { }); } - stop() { - if (this.#stopped) return; + /** + * Rejects when the backend could not prove its queued work quiescent; the runner lock stays held + * then. Callers await it before closing storage: it resolves only once nothing can still write. + */ + stop(): Promise { + if (this.#stopResult) return this.#stopResult; this.#stopped = true; this.status = { state: 'stopped', ownerEpoch: this.#ownerEpoch }; if (this.#idleTimer) clearTimeout(this.#idleTimer); - this.#unsubscribeBackend?.(); - this.#unregisterTables(); + if (this.#rebuildTimer) clearTimeout(this.#rebuildTimer); + if (this.#lockRetryTimer) clearTimeout(this.#lockRetryTimer); + this.#rebuildTimer = undefined; + try { + this.#unsubscribeBackend?.(); + this.#readinessBuffer.cancel?.(); + } catch (error) { + logger.warn?.(`Derived index '${this.id}' cleanup hook threw`, error); + } this.#release(); + this.#stopResult = (this.#releasing ?? Promise.resolve()).then(() => { + if (this.#releaseFailure) throw this.#releaseFailure; + this.#unregisterTables(); + }); + this.#stopResult.catch(() => {}); + return this.#stopResult; + } + + getReadiness(): DerivedIndexReadiness { + return readReadiness(this.#shared()); + } + + getMetrics(): DerivedIndexRunnerMetrics { + const now = this.#options.now(); + let acceptedBytes = this.#unanchoredBytes; + let acceptedMutations = this.#unanchoredMutations; + const oldestAcceptedAt = this.#oldestAcceptedAt(); + for (let i = 1; i < this.#offeredCursors.length; i++) { + acceptedBytes += this.#offeredCursors[i].bytes; + acceptedMutations += this.#offeredCursors[i].mutations; + } + const cursorLag = this.#cursorLag(); + return { + readiness: this.getReadiness(), + acceptedBatches: Math.max(0, this.#offeredCursors.length - 1), + acceptedBytes, + acceptedMutations, + deferredBytes: this.#pendingBatch?.bytes ?? 0, + oldestAcceptedAgeMilliseconds: oldestAcceptedAt === undefined ? 0 : Math.max(0, now - oldestAcceptedAt), + cursorLagMilliseconds: cursorLag, + stalledMilliseconds: this.#stalledSince === undefined ? 0 : Math.max(0, now - this.#stalledSince), + unindexableRecords: this.#unindexableRecords, + rebuildAttempts: this.#rebuildAttempts, + rebuiltRecords: this.#rebuiltRecords, + quiescenceAgeMilliseconds: + this.#releasingSince === undefined && this.#quiescing === undefined + ? 0 + : Math.max(0, now - Math.min(this.#releasingSince ?? Infinity, this.#quiescing?.since ?? Infinity)), + }; + } + + #cursorLag(): number { + let cursorLag = 0; + const durable = this.#offeredCursors[0]?.cursor; + if (durable) { + for (const [logName, latest] of this.#latestSeen) { + const position = durable.logs[logName]; + if (position !== undefined && latest > position) cursorLag = Math.max(cursorLag, latest - position); + } + } + return cursorLag; + } + + #writeRejection(): string | undefined { + if (Atomics.load(this.#shared().words, READINESS_LAG_EXCEEDED) !== 1) return; + return `derived index '${this.id}' is more than ${this.#lagBudget} ms behind; retry this write`; + } + + /** + * Time since the oldest commit this runner may not have read, bounded by how far the newest + * entry it has read trails the clock: a reader that never quite empties a steadily fed log is + * behind by that distance, not by the age of its first unread commit. + */ + #unreadAge(now: number): number { + if (this.#unreadSince === undefined) return 0; + let newestRead = -Infinity; + for (const latest of this.#latestSeen.values()) if (latest > newestRead) newestRead = latest; + return Math.max(0, Math.min(now - this.#unreadSince, now - newestRead)); + } + + #oldestAcceptedAt(): number | undefined { + if (this.#offeredCursors.length > 1) return this.#offeredCursors[1].acceptedAt; + return this.#unanchoredMutations > 0 ? this.#unanchoredAcceptedAt : undefined; + } + + /** + * Owner-only. Lag is the longest of four terms: cursor distance behind what this runner has read, + * time parked on backpressure, time since the oldest commit this runner may not have read yet (a + * reader too slow to reach the end of the log cannot hide), and the age of the oldest accepted + * work not yet durable (a backend that accepts but never barriers cannot hide). All four are zero + * for a caught-up owner sitting idle and stay within drain latency plus flush age for a runner + * keeping up under sustained ingest. The trip survives discard and handoff: a successor clears it + * only after proving catch-up itself — a durable advance and the end of the log both reached since + * it acquired, so an inherited backlog cannot be cleared by one barrier — below half the budget. + */ + #publishLag() { + const max = this.#lagBudget; + if (max <= 0 || !this.#owned || this.#rebuilding) return; + const now = this.#options.now(); + const oldestAccepted = this.#oldestAcceptedAt(); + const lag = Math.max( + this.#cursorLag(), + this.#stalledSince === undefined ? 0 : now - this.#stalledSince, + this.#unreadAge(now), + oldestAccepted === undefined ? 0 : now - oldestAccepted + ); + const words = this.#shared().words; + const tripped = Atomics.load(words, READINESS_LAG_EXCEEDED) === 1; + if (!tripped && lag >= max) { + Atomics.store(words, READINESS_LAG_EXCEEDED, 1); + logger.warn?.(`Derived index '${this.id}' is ${Math.round(lag)} ms behind; rejecting writes until it catches up`); + } else if (tripped && lag < max / 2 && this.#lastCaughtUpAt !== undefined && this.#reachedEndOfLog) { + Atomics.store(words, READINESS_LAG_EXCEEDED, 0); + logger.info?.(`Derived index '${this.id}' caught up; admitting writes again`); + } + if (this.#lagTimer) return; + this.#lagTimer = setTimeout( + () => { + this.#lagTimer = undefined; + if (this.#owned && !this.#stopped) this.#publishLag(); + }, + Math.min(1000, Math.max(1, max / 4)) + ); + this.#lagTimer.unref?.(); + } + + requestRebuild(): boolean { + if (this.#stopped || !this.#canRebuild()) return false; + if (this.#rebuilding) return true; + this.#rebuildAttempts = 0; + if (this.#rebuildTimer) { + clearTimeout(this.#rebuildTimer); + this.#rebuildTimer = undefined; + } + if (this.status.state === 'unavailable') { + this.status = { state: 'needs-rebuild', reason: this.status.reason, ownerEpoch: this.#ownerEpoch }; + } + if (this.#owned) { + this.#startRebuild(); + return true; + } + if (this.#heldLock) { + this.#rebuildRequested = true; + this.#acquired(true); + return true; + } + // The owner may be another worker that never idles: leave the request where every runner looks, + // and notify whoever holds the buffer's callback. + Atomics.store(this.#shared().words, READINESS_REBUILD_REQUEST, 1); + this.#readinessBuffer.notify?.(); + this.wake(true); + return true; + } + + #takeSharedRebuildRequest(): boolean { + return Atomics.exchange(this.#shared().words, READINESS_REBUILD_REQUEST, 0) === 1; + } + + #canRebuild(): boolean { + return typeof this.#registration.backend.reset === 'function' && this.#scanRecords !== undefined; } #acquire() { - if (this.#waitingForLock) return; + if (this.#waitingForLock || this.#lockRetryTimer) return; + if (this.#releasing) { + this.#releasing.then(() => this.wake(true)); + return; + } this.#waitingForLock = true; const retry = () => { this.#waitingForLock = false; @@ -226,51 +796,122 @@ class DerivedIndexRunner { }; try { if (!this.#logStore.tryLock(this.#lockKey, retry)) return; + } catch (error) { this.#waitingForLock = false; - this.#owned = true; - this.#ownerEpoch = this.#nextOwnerEpoch(); + logger.error(`Derived index '${this.id}' could not attempt the runner lock; retrying`, error); + if (!this.#lockRetryTimer) { + this.#lockRetryTimer = setTimeout(() => { + this.#lockRetryTimer = undefined; + this.wake(true); + }, this.#options.rebuildBackoffMilliseconds); + this.#lockRetryTimer.unref?.(); + } + return; + } + this.#waitingForLock = false; + this.#acquired(); + } + + #acquired(reviving = false) { + this.#heldLock = false; + this.#releaseFailure = undefined; + this.#owned = true; + this.#generation++; + this.#lastCaughtUpAt = undefined; + this.#unreadSince = this.#options.now(); + this.#reachedEndOfLog = false; + try { + if (!reviving) this.#ownerEpoch = this.#mintEpoch(); this.status = { state: 'running', ownerEpoch: this.#ownerEpoch }; + const condemned = this.#readCondemnation(); + const shared = this.getReadiness(); + if (this.#takeSharedRebuildRequest()) { + this.#rebuildRequested = true; + this.#rebuildAttempts = 0; + } else if (!this.#rebuildRequested) this.#rebuildAttempts = shared.rebuildAttempts; + if (this.#rebuildRequested && this.#canRebuild()) { + this.#startRebuild(); + return; + } + if (shared.state === 'unavailable') { + this.status = { + state: 'unavailable', + reason: `unavailable by a previous owner (${shared.reason ?? 'none'})`, + ownerEpoch: this.#ownerEpoch, + }; + this.#admitWrites(); + this.#release(); + return; + } + if (shared.state === 'needs-rebuild' || shared.state === 'rebuilding') { + if (this.#canRebuild()) this.#startRebuild(); + else { + if (this.#writeCondemnation()) this.#rebuildRequested = false; + this.status = { + state: 'needs-rebuild', + reason: `condemned by a previous owner (${shared.reason ?? 'none'})`, + ownerEpoch: this.#ownerEpoch, + }; + this.#admitWrites(); + this.#release(); + } + return; + } + if (condemned) { + this.#needsRebuild('condemned before a restart; the durable cursor is not trusted', 'condemned'); + return; + } this.#resetFromDurableCursor(); - if (this.#owned) this.#drain(); + if (this.#owned && !this.#rebuilding) this.#drain(); } catch (error) { - this.#waitingForLock = false; - this.#fail('failed to acquire or initialize the runner', error); + this.#fail('failed to initialize the runner', error); } } - #nextOwnerEpoch(): bigint { - const buffer = this.#logStore.getUserSharedBuffer( - `derived-index:${this.#registration.backend.id}:owner-epoch`, - new ArrayBuffer(8) - ); - return Atomics.add(new BigInt64Array(buffer), 0, 1n) + 1n; + #mintEpoch(): bigint { + return Atomics.add(this.#sharedViews.epoch, 0, 1n) + 1n; } #resetFromDurableCursor() { const durable = this.#registration.backend.getDurableCursor(); if (!isValidCursor(durable)) { - this.#needsRebuild(durable ? 'backend returned an invalid durable cursor' : 'backend has no durable cursor'); + this.#needsRebuild( + durable ? 'backend returned an invalid durable cursor' : 'backend has no durable cursor', + 'cursor-missing' + ); return; } - this.#validateLogSet(durable); - if (this.status.state === 'needs-rebuild') return; - this.#offered = cloneCursor(durable); - this.#offeredCursors = [cloneCursor(durable)]; + if (!this.#installCursor(durable)) return; + this.#publishReadiness('ready'); + } + + #installCursor(cursor: DerivedIndexCursor): boolean { + this.#validateLogSet(cursor); + if (!this.#owned || this.status.state === 'needs-rebuild' || this.status.state === 'unavailable') return false; + this.#offered = cloneCursor(cursor); + this.#offeredCursors = [{ cursor: cloneCursor(cursor), bytes: 0, mutations: 0, acceptedAt: this.#options.now() }]; + this.#unanchoredBytes = 0; + this.#unanchoredMutations = 0; + this.#unflushedBytes = 0; + this.#unflushedMutations = 0; + this.#pendingBatch = undefined; + this.#carried = []; + this.#latestSeen.clear(); this.#pendingTimestamps.clear(); this.#seenTimestamps.clear(); - for (const [logName, timestamp] of Object.entries(durable.logs)) { + for (const [logName, timestamp] of Object.entries(cursor.logs)) { this.#pendingTimestamps.set(logName, [timestamp]); this.#seenTimestamps.set(logName, new Set([timestamp])); } this.#iterable = this.#logStore.getRange({ - startByLog: new Map(Object.entries(durable.logs)), + startByLog: new Map(Object.entries(cursor.logs)), exactStart: true, exclusiveStart: true, resumeAfterExactStart: true, includeLogName: true, }); this.#iterator = this.#iterable[Symbol.iterator](); - this.#checkRangeHealth(); + return this.#checkRangeHealth(); } #validateLogSet(cursor: DerivedIndexCursor) { @@ -278,23 +919,41 @@ class DerivedIndexRunner { const current = new Set(currentLogs); for (const logName of Object.keys(cursor.logs)) { if (!current.has(logName)) { - this.#needsRebuild(`saved transaction log '${logName}' is missing`); + this.#needsRebuild(`saved transaction log '${logName}' is missing`, 'log-missing'); return; } } this.#knownLogs = current; for (const logName of currentLogs) { if (cursor.logs[logName] !== undefined) continue; - const oldestSequenceNumber = this.#logStore.rootStore.useLog(logName).getStats().oldestSequenceNumber; - if (oldestSequenceNumber !== 1) { - this.#needsRebuild(`new transaction log '${logName}' no longer retains its beginning`); + if (!this.#retainsBeginning(logName)) { + this.#needsRebuild(`new transaction log '${logName}' no longer retains its beginning`, 'log-retention'); return; } } } + /** A log that never wrote a file has nothing to have lost; one with files must still hold the first. */ + #retainsBeginning(logName: string): boolean { + const stats = this.#logStore.rootStore.useLog(logName).getStats(); + return stats.fileCount === 0 || stats.oldestSequenceNumber === 1; + } + #drain() { - if (!this.#owned || this.#stopped || this.status.state === 'needs-rebuild') return; + if (!this.#owned || this.#stopped || this.#rebuilding) return; + if (this.#canRebuild() && this.#takeSharedRebuildRequest()) { + if (this.#rebuildTimer) { + clearTimeout(this.#rebuildTimer); + this.#rebuildTimer = undefined; + } + this.#rebuildAttempts = 0; + this.#startRebuild(); + return; + } + if (this.status.state === 'needs-rebuild' || this.status.state === 'unavailable') return; + const generation = this.#generation; + const now = this.#options.now(); + this.#publishLag(); try { if (!this.#checkNewLogs() || !this.#checkRangeHealth()) return; if (this.status.state === 'waiting-durable') { @@ -302,40 +961,36 @@ class DerivedIndexRunner { if (this.#offeredCursors.length - 1 >= this.#options.maxAcceptedBatchesAhead) return; this.status = { state: 'running', ownerEpoch: this.#ownerEpoch }; } - const batch = this.#pendingBatch ?? this.#collectBatch(); - if (!this.#owned) return; - if (!batch) { - this.#finishIdlePass(); + const batch = this.#pendingBatch ?? this.#collectChunk(); + if (!this.#live(generation)) return; + if (batch === CONTINUE) { + this.status = { state: 'running', ownerEpoch: this.#ownerEpoch }; + this.wake(); return; } - let result: DerivedIndexDeliveryResult; - const deliveryIterator = this.#iterator; - try { - result = this.#registration.backend.deliver(batch); - } catch (error) { - this.#fail('backend delivery threw', error); + if (!batch) { + this.#finishIdlePass(); return; } - if (!this.#owned || this.#iterator !== deliveryIterator) return; + const result = this.#deliver(batch); + if (result === undefined) return; if (result === DERIVED_INDEX_DEFERRED) { this.#pendingBatch = batch; this.status = { state: 'deferred', ownerEpoch: this.#ownerEpoch }; - return; - } - if (result !== DERIVED_INDEX_ACCEPTED) { - this.#needsRebuild( - result === DERIVED_INDEX_FAILED ? 'backend rejected a delivery batch' : 'backend returned an invalid result' - ); + this.#stalledSince ??= now; return; } this.#pendingBatch = undefined; - this.#offered = cloneCursor(batch.through); - this.#offeredCursors.push(cloneCursor(batch.through)); + this.#noteAccepted(batch); + if (!this.#live(generation)) return; if (!this.#reconcileDurableCursor()) return; - if (this.#offeredCursors.length - 1 >= this.#options.maxAcceptedBatchesAhead) { + this.#publishLag(); + if (!lastOpen(this.#carried) && this.#offeredCursors.length - 1 >= this.#options.maxAcceptedBatchesAhead) { this.status = { state: 'waiting-durable', ownerEpoch: this.#ownerEpoch }; + this.#stalledSince ??= now; return; } + this.#stalledSince = undefined; this.status = { state: 'running', ownerEpoch: this.#ownerEpoch }; this.wake(); } catch (error) { @@ -343,114 +998,312 @@ class DerivedIndexRunner { } } - #collectBatch(): DerivedIndexBatch | undefined { - const started = this.#options.now(); - let transactions = 0; - let bytes = 0; - let progressed = false; - const through = cloneCursor(this.#offered!); - const pending: Array<{ - logName: string; - timestamp: number; - records: Map>; - }> = []; - while (true) { - const first = this.#iterator!.next(); - if (first.done) break; - const firstRecord = first.value; - let records: Map> | undefined; - let current = firstRecord; - this.#assertRecord(current); - const logName = current.logName!; - const timestamp = current.txnLogKey; - let seen = this.#seenTimestamps.get(logName); - if (!seen) this.#seenTimestamps.set(logName, (seen = new Set())); - if (seen.has(timestamp)) - throw new Error(`transaction log '${logName}' repeated completed timestamp ${timestamp}`); - while (true) { - if (current.logName !== logName || current.txnLogKey !== timestamp) - throw new Error(`transaction ${timestamp} from '${logName}' ended without an endTxn boundary`); - bytes += current.size ?? 0; - const projection = this.#registration.projections.get(current.tableId); - if (projection) { - if (current.type === 'reload') throw new Error(`table ${current.tableId} requires a derived-index rebuild`); - if (ELIGIBLE_ACTIONS.has(current.type)) { - records ??= new Map(); - let byRecord = records.get(current.tableId); - if (!byRecord) records.set(current.tableId, (byRecord = new Map())); - byRecord.set(writeKeyId(current.recordId), { - recordId: current.recordId, - logVersion: current.version, - }); - } - } - if (current.endTxn) break; - const next = this.#iterator!.next(); - if (next.done) throw new Error(`transaction ${timestamp} from '${logName}' is incomplete`); - current = next.value; - this.#assertRecord(current); - } - seen.add(timestamp); - let pendingTimestamps = this.#pendingTimestamps.get(logName); - if (!pendingTimestamps) this.#pendingTimestamps.set(logName, (pendingTimestamps = [])); - pendingTimestamps.push(timestamp); - through.logs[logName] = timestamp; - progressed = true; - transactions++; - if (records) pending.push({ logName, timestamp, records }); - if ( - transactions >= this.#options.maxTransactionsPerTurn || - bytes >= this.#options.maxBytesPerTurn || - this.#options.now() - started >= this.#options.maxMillisecondsPerTurn - ) - break; + #deliver(batch: DerivedIndexBatch): typeof DERIVED_INDEX_ACCEPTED | typeof DERIVED_INDEX_DEFERRED | undefined { + const generation = this.#generation; + let result: DerivedIndexDeliveryResult; + try { + result = this.#registration.backend.deliver(batch); + } catch (error) { + this.#fail('backend delivery threw', error, 'backend-failed'); + return; + } + if (!this.#live(generation)) return; + if (result === DERIVED_INDEX_DEFERRED || result === DERIVED_INDEX_ACCEPTED) return result; + this.#needsRebuild( + result === DERIVED_INDEX_FAILED ? 'backend rejected a delivery batch' : 'backend returned an invalid result', + 'backend-failed' + ); + } + + #noteAccepted(batch: DerivedIndexBatch) { + const now = this.#options.now(); + if (batch.through && !sameCursor(batch.through, this.#offered)) { + this.#offered = cloneCursor(batch.through); + this.#offeredCursors.push({ + cursor: cloneCursor(batch.through), + bytes: this.#unanchoredBytes + batch.bytes, + mutations: this.#unanchoredMutations + batch.records.length, + acceptedAt: this.#unanchoredMutations > 0 ? this.#unanchoredAcceptedAt : now, + }); + this.#unanchoredBytes = 0; + this.#unanchoredMutations = 0; + } else { + if (this.#unanchoredMutations === 0) this.#unanchoredAcceptedAt = now; + this.#unanchoredBytes += batch.bytes; + this.#unanchoredMutations += batch.records.length; + } + this.#unflushedBytes += batch.bytes; + this.#unflushedMutations += batch.records.length; + if ( + this.#unflushedMutations >= this.#options.flushAfterMutations || + this.#unflushedBytes >= this.#options.flushAfterBytes + ) { + this.#requestFlush('threshold'); + } else this.#armFlushTimer(); + } + + #requestFlush(reason: DerivedIndexFlushReason) { + if (this.#flushTimer) { + clearTimeout(this.#flushTimer); + this.#flushTimer = undefined; } + this.#unflushedBytes = 0; + this.#unflushedMutations = 0; + const generation = this.#generation; + try { + const result = this.#registration.backend.flush(reason); + if (result && typeof result.then === 'function') { + result.then(undefined, (error: unknown) => { + if (this.#live(generation)) this.#fail('backend flush request rejected', error, 'backend-failed'); + }); + } + } catch (error) { + this.#fail('backend flush request threw', error, 'backend-failed'); + return; + } + // A backend may coalesce this into a barrier already running; keep asking while work is not durable. + if (this.#hasNonDurableWork()) this.#armFlushTimer(); + } + + #hasNonDurableWork(): boolean { + return this.#offeredCursors.length > 1 || this.#unanchoredMutations > 0 || this.#boundaryPending; + } + + #armFlushTimer() { + if (this.#flushTimer) return; + this.#flushTimer = setTimeout(() => { + this.#flushTimer = undefined; + if (!this.#owned) return; + this.#publishLag(); + this.#requestFlush('age'); + }, this.#options.maxFlushAgeMilliseconds); + this.#flushTimer.unref?.(); + } + + #collectChunk(): DerivedIndexBatch | typeof CONTINUE | undefined { + const chunk = this.#newChunk(false); + const collected = this.#collectIdentities(chunk.started); if (!this.#checkRangeHealth()) return; - if (!progressed) return; - return { ownerEpoch: this.#ownerEpoch!, transactions: this.#resolveTransactions(pending), through }; - } - - #resolveTransactions( - pending: Array<{ - logName: string; - timestamp: number; - records: Map>; - }> - ): DerivedIndexTransaction[] { - const resolved = new Map>(); - for (const transaction of pending) { - for (const [tableId, records] of transaction.records) { - let byRecord = resolved.get(tableId); - if (!byRecord) resolved.set(tableId, (byRecord = new Map())); - for (const [key, record] of records) { - if (byRecord.has(key)) continue; - const current = this.#resolveRecord(tableId, record.recordId); - const state: DerivedIndexState = current - ? { - kind: 'record', - version: current.version, - projection: this.#registration.projections.get(tableId)!(current.value), - } - : { kind: 'absent' }; - byRecord.set(key, state); + if (collected.length === 0) return; + return this.#resolveCollected(chunk, collected); + } + + #collectIdentities(started: number): CollectedTransaction[] { + const options = this.#options; + const iterator = this.#iterator!; + const projections = this.#registration.projections; + const collected = this.#carried; + this.#carried = []; + let keyCount = 0; + for (const transaction of collected) keyCount += transaction.keyCount; + let current = lastOpen(collected); + let transactions = 0; + let readBytes = 0; + let entries = 0; + while (keyCount < options.maxChunkRecords) { + const next = iterator.next(); + if (next.done) { + if (current) + throw new RunnerError( + 'log-corrupt', + `transaction ${current.timestamp} from '${current.logName}' is incomplete` + ); + break; + } + const entry = next.value; + this.#assertRecord(entry); + if (!current) { + const logName = entry.logName!; + const timestamp = entry.txnLogKey; + let seen = this.#seenTimestamps.get(logName); + if (!seen) this.#seenTimestamps.set(logName, (seen = new Set())); + if (seen.has(timestamp)) + throw new RunnerError( + 'log-corrupt', + `transaction log '${logName}' repeated completed timestamp ${timestamp}` + ); + current = { logName, timestamp, keys: new Map(), keyCount: 0, complete: false }; + collected.push(current); + } else if (entry.logName !== current.logName || entry.txnLogKey !== current.timestamp) { + throw new RunnerError( + 'log-corrupt', + `transaction ${current.timestamp} from '${current.logName}' ended without an endTxn boundary` + ); + } + readBytes += entry.size ?? 0; + entries++; + const projection = projections.get(entry.tableId); + if (projection) { + if (entry.type === 'reload') { + // A rebuild anchors its replay at the committed tail captured before its scan, so a marker + // is met exactly once: here, before the rebuild it demands. + throw new RunnerError('reload', `table ${entry.tableId} requires a derived-index rebuild`); + } else if (ELIGIBLE_ACTIONS.has(entry.type)) { + let byRecord = current.keys.get(entry.tableId); + if (!byRecord) current.keys.set(entry.tableId, (byRecord = new Map())); + const key = writeKeyId(entry.recordId); + const known = byRecord.get(key); + if (known) known.logVersion = entry.version; + else { + byRecord.set(key, { recordId: entry.recordId, logVersion: entry.version, sizeHint: entry.size }); + current.keyCount++; + keyCount++; + } } } + if (entry.endTxn) { + current.complete = true; + this.#latestSeen.set(current.logName, current.timestamp); + this.#seenTimestamps.get(current.logName)!.add(current.timestamp); + let pendingTimestamps = this.#pendingTimestamps.get(current.logName); + if (!pendingTimestamps) this.#pendingTimestamps.set(current.logName, (pendingTimestamps = [])); + pendingTimestamps.push(current.timestamp); + current = undefined; + transactions++; + if ( + transactions >= options.maxTransactionsPerTurn || + readBytes >= options.maxBytesPerTurn || + options.now() - started >= options.maxMillisecondsPerTurn + ) + break; + } else if ((entries & 15) === 0 && options.now() - started >= options.maxMillisecondsPerTurn) break; } - return pending.map(({ logName, timestamp, records }) => { + return collected; + } + + #resolveCollected(chunk: Chunk, collected: CollectedTransaction[]): DerivedIndexBatch | typeof CONTINUE { + const options = this.#options; + const through = cloneCursor(this.#offered!); + let completed = 0; + let visited = 0; + for (let i = 0; i < collected.length; i++) { + const transaction = collected[i]; const mutations: DerivedIndexMutation[] = []; - for (const [tableId, byRecord] of records) { - for (const key of byRecord.keys()) { - const record = byRecord.get(key)!; - mutations.push({ tableId, ...record, state: resolved.get(tableId)!.get(key)! }); + let remaining: CollectedTransaction | undefined; + for (const [tableId, byRecord] of transaction.keys) { + for (const [key, collectedKey] of byRecord) { + if ( + !remaining && + chunk.batch.records.length > 0 && + (chunk.batch.bytes >= options.maxChunkBytes || + ((++visited & 15) === 0 && options.now() - chunk.started >= options.maxMillisecondsPerTurn)) + ) { + remaining = { ...transaction, keys: new Map(), keyCount: 0 }; + } + if (remaining) { + let rest = remaining.keys.get(tableId); + if (!rest) remaining.keys.set(tableId, (rest = new Map())); + rest.set(key, collectedKey); + remaining.keyCount++; + continue; + } + const record = this.#addMutation(chunk, tableId, key, collectedKey); + mutations.push({ + tableId, + recordId: collectedKey.recordId, + logVersion: collectedKey.logVersion, + state: record.state, + }); } } - return { logName, timestamp, mutations }; + if (remaining) { + if (mutations.length) + chunk.batch.transactions.push({ logName: transaction.logName, timestamp: transaction.timestamp, mutations }); + this.#carried = [remaining, ...collected.slice(i + 1)]; + break; + } + if (transaction.complete) { + through.logs[transaction.logName] = transaction.timestamp; + completed++; + if (mutations.length) + chunk.batch.transactions.push({ logName: transaction.logName, timestamp: transaction.timestamp, mutations }); + } else { + if (mutations.length) + chunk.batch.transactions.push({ logName: transaction.logName, timestamp: transaction.timestamp, mutations }); + this.#carried = [ + { + logName: transaction.logName, + timestamp: transaction.timestamp, + keys: new Map(), + keyCount: 0, + complete: false, + }, + ]; + } + } + this.#noteChunkProjection(chunk); + if (completed === 0 && chunk.batch.records.length === 0) return CONTINUE; + chunk.batch.through = through; + return chunk.batch; + } + + #newChunk(rebuild: boolean): Chunk { + const batch = { ownerEpoch: this.#ownerEpoch!, transactions: [] } as unknown as DerivedIndexBatch; + Object.defineProperties(batch, { + records: { value: [], writable: true, configurable: true }, + bytes: { value: 0, writable: true, configurable: true }, }); + if (rebuild) batch.rebuild = true; + return { batch, resolved: new Map(), started: this.#options.now() }; + } + + #addMutation(chunk: Chunk, tableId: number, key: unknown, collectedKey: CollectedKey): DerivedIndexMutation { + let byRecord = chunk.resolved.get(tableId); + if (!byRecord) chunk.resolved.set(tableId, (byRecord = new Map())); + let record = byRecord.get(key); + if (record) { + record.logVersion = collectedKey.logVersion; + return record; + } + const current = this.#resolveRecord(tableId, collectedKey.recordId); + const state: DerivedIndexState = current + ? this.#project(chunk, tableId, current.value, current.version, current.size ?? collectedKey.sizeHint) + : { kind: 'absent' }; + record = { tableId, recordId: collectedKey.recordId, logVersion: collectedKey.logVersion, state }; + byRecord.set(key, record); + chunk.batch.records.push(record); + return record; + } + + /** A chunk the projection rejected outright is worth one warning per streak, never an outage. */ + #noteChunkProjection(chunk: Chunk) { + const records = chunk.batch.records; + if (records.length === 0) return; + if (records.some((record) => record.state.kind !== 'unindexable')) { + this.#allUnindexableWarned = false; + return; + } + if (this.#allUnindexableWarned) return; + this.#allUnindexableWarned = true; + logger.warn?.( + `Derived index '${this.#registration.backend.id}' could not project any of the ${records.length} records in a chunk; they are counted in unindexableRecords` + ); + } + + #project( + chunk: Chunk, + tableId: number, + value: unknown, + version: number, + size: number | undefined + ): DerivedIndexState { + chunk.batch.bytes += size ?? 0; + try { + return { kind: 'record', version, projection: this.#registration.projections.get(tableId)!(value) }; + } catch (error) { + const statusCode = (error as { statusCode?: unknown })?.statusCode; + if (typeof statusCode !== 'number' || statusCode < 400 || statusCode >= 500) throw error; + // Validation messages can quote record values, which must not reach the backend or the log. + const reason = `${error instanceof Error && error.name ? error.name : 'Error'} (${statusCode})`; + if (this.#unindexableRecords++ === 0) + logger.warn?.(`Derived index '${this.#registration.backend.id}' skipped a record it cannot project: ${reason}`); + return { kind: 'unindexable', version, reason }; + } } #assertRecord(record: AuditRecord) { if (!record || record.logName === undefined || record.tableId === undefined || record.type === undefined) - throw new Error('transaction log yielded an undecodable audit entry'); + throw new RunnerError('log-corrupt', 'transaction log yielded an undecodable audit entry'); } #checkNewLogs(): boolean { @@ -458,15 +1311,14 @@ class DerivedIndexRunner { const currentSet = new Set(current); for (const logName of this.#knownLogs) { if (!currentSet.has(logName)) { - this.#needsRebuild(`transaction log '${logName}' was removed`); + this.#needsRebuild(`transaction log '${logName}' was removed`, 'log-missing'); return false; } } for (const logName of current) { if (this.#knownLogs.has(logName)) continue; - const oldest = this.#logStore.rootStore.useLog(logName).getStats().oldestSequenceNumber; - if (oldest !== 1) { - this.#needsRebuild(`new transaction log '${logName}' no longer retains its beginning`); + if (!this.#retainsBeginning(logName)) { + this.#needsRebuild(`new transaction log '${logName}' no longer retains its beginning`, 'log-retention'); return false; } this.#knownLogs.add(logName); @@ -474,102 +1326,509 @@ class DerivedIndexRunner { return true; } - #checkRangeHealth(): boolean { - if (!this.#iterable) return true; - if (this.#iterable.corruptFrameStop.breaks > 0) { - this.#needsRebuild('transaction log contains a corrupt frame'); + #checkRangeHealth(iterable = this.#iterable): boolean { + if (!iterable) return true; + if (iterable.corruptFrameStop.breaks > 0) { + this.#needsRebuild('transaction log contains a corrupt frame', 'log-corrupt'); return false; } - if (this.#iterable.failedLogs.size > 0) { - this.#needsRebuild(`transaction log iterator failed for '${this.#iterable.failedLogs.values().next().value}'`); + if (iterable.failedLogs.size > 0) { + this.#needsRebuild( + `transaction log iterator failed for '${iterable.failedLogs.values().next().value}'`, + 'log-corrupt' + ); return false; } - if (this.#iterable.exactStartFailures.size > 0) { - const [logName, failure] = this.#iterable.exactStartFailures.entries().next().value; - this.#needsRebuild(`transaction log '${logName}' has a ${failure} durable cursor boundary`); + if (iterable.exactStartFailures.size > 0) { + const [logName, failure] = iterable.exactStartFailures.entries().next().value; + this.#needsRebuild(`transaction log '${logName}' has a ${failure} durable cursor boundary`, 'log-retention'); return false; } return true; } #finishIdlePass() { + this.#stalledSince = undefined; + this.#unreadSince = undefined; + this.#reachedEndOfLog = true; + if (this.#rebuilding || !this.#offered) return; const durable = this.#registration.backend.getDurableCursor(); + if (durable === undefined && this.#boundaryPending) { + this.#armFlushTimer(); + return; + } if (!isValidCursor(durable)) { - this.#needsRebuild('backend lost its durable cursor'); + this.#needsRebuild('backend lost its durable cursor', 'cursor-missing'); return; } if (!this.#reconcileDurableCursor(durable)) return; - if (!sameCursor(durable, this.#offered!)) return; + if (sameCursor(durable, this.#offered!)) this.#lastCaughtUpAt = this.#options.now(); + this.#publishLag(); + if (!sameCursor(durable, this.#offered!)) { + this.#armFlushTimer(); + return; + } + this.#settleReady(); if (this.#idleTimer) return; this.status = { state: 'idle', ownerEpoch: this.#ownerEpoch }; this.#idleTimer = setTimeout(() => { this.#idleTimer = undefined; - if (!this.#stopped && sameCursor(this.#registration.backend.getDurableCursor(), this.#offered!)) this.#release(); + if (this.#stopped) return; + try { + if (sameCursor(this.#registration.backend.getDurableCursor(), this.#offered!)) this.#release(); + } catch (error) { + this.#fail('backend cursor read threw at idle release', error, 'backend-failed'); + } }, this.#options.idleGraceMilliseconds); } #reconcileDurableCursor(cursor = this.#registration.backend.getDurableCursor()): boolean { + if (cursor === undefined && this.#boundaryPending) return true; if (!isValidCursor(cursor)) { - this.#needsRebuild('backend returned an invalid durable cursor'); + this.#needsRebuild('backend returned an invalid durable cursor', 'cursor-missing'); return false; } - const offeredIndex = this.#offeredCursors.findIndex((offered) => sameCursor(cursor, offered)); + const offeredIndex = this.#offeredCursors.findIndex((offered) => sameCursor(cursor, offered.cursor)); if (offeredIndex < 0) { - this.#needsRebuild('backend advanced to an unoffered cursor vector'); + this.#needsRebuild('backend advanced to an unoffered cursor vector', 'cursor-unoffered'); return false; } + // Every timestamp of an offered vector was pushed to its log's pending list, so the durable one is + // present; everything before it is no longer needed for repeat detection. for (const [logName, timestamp] of Object.entries(cursor.logs)) { - const pending = this.#pendingTimestamps.get(logName); - if (!pending) { - this.#needsRebuild(`backend advanced unknown transaction log '${logName}'`); - return false; - } + const pending = this.#pendingTimestamps.get(logName)!; const index = pending.indexOf(timestamp); - if (index < 0) { - this.#needsRebuild(`backend advanced '${logName}' to an unoffered cursor`); - return false; - } if (index > 0) { const retained = pending.slice(index); this.#pendingTimestamps.set(logName, retained); this.#seenTimestamps.set(logName, new Set(retained)); } } - if (offeredIndex > 0) this.#offeredCursors.splice(0, offeredIndex); + this.#boundaryPending = false; + if (offeredIndex > 0) { + this.#offeredCursors.splice(0, offeredIndex); + if (!this.#rebuilding && this.status.state !== 'needs-rebuild') this.#settleReady(); + } + if (offeredIndex > 0 || sameCursor(cursor, this.#offered)) this.#lastCaughtUpAt = this.#options.now(); return true; } + #settleReady() { + this.#rebuildAttempts = 0; + if (this.#condemned) this.#clearCondemnation(); + if (Atomics.load(this.#shared().words, READINESS_STATE) !== READINESS_STATES.indexOf('ready')) + this.#publishReadiness('ready'); + } + + /** + * Shared readiness is process memory, so a condemnation is also written to the root store under + * the index's marker key; a restart before the rebuild's `reset` has durably invalidated the + * cursor then still rebuilds instead of trusting it. Durability follows the root store's WAL + * setting. The marker clears only at the first durable `ready` after the rebuild, so a crash + * before that costs one extra rebuild, never a trusted condemned cursor. A log store without a + * root-store key-value surface (test fakes) keeps Stage 1's process-memory condemnation only. + */ + #writeCondemnation(): boolean { + if (this.#condemned || !this.#markersSupported) return true; + try { + this.#logStore.putSync(this.#markerKey, CONDEMNED_MARKER, {}); + this.#condemned = true; + return true; + } catch (error) { + // Without the marker a crash mid-reset would reopen on the condemned cursor, so no reset runs. + logger.error(`Derived index '${this.id}' could not persist its condemnation`, error); + return false; + } + } + + #readCondemnation(): boolean { + if (!this.#markersSupported) return false; + try { + this.#condemned = this.#logStore.rootStore.getSync(this.#markerKey) !== undefined; + return this.#condemned; + } catch (error) { + logger.error(`Derived index '${this.id}' could not read its condemnation marker`, error); + return true; + } + } + + /** No rebuild attempt is spent on a refused marker; the next acquirer retries it before any reset. */ + #deferForCondemnation(code: DerivedIndexReadinessReason) { + logger.error(`Derived index '${this.id}' condemnation could not be persisted; retrying at the next wake`); + const reason = this.status.state === 'needs-rebuild' ? this.status.reason : 'condemnation not persisted'; + this.status = { state: 'needs-rebuild', reason, ownerEpoch: this.#ownerEpoch }; + this.#publishReadiness('needs-rebuild', code); + this.#rebuildRequested = true; + // The retry needs a wake, and wakes come from commits: shedding them would be the only thing + // keeping this park from ever ending. + this.#admitWrites(); + this.#release(); + } + + #clearCondemnation() { + if (!this.#markersSupported) { + this.#condemned = false; + return; + } + try { + this.#logStore.rootStore.removeSync(this.#markerKey); + this.#condemned = false; + } catch (error) { + logger.error(`Derived index '${this.id}' could not clear its condemnation marker`, error); + } + } + #backendStateChanged(change: DerivedIndexBackendStateChange) { - if (this.#stopped || this.status.state === 'needs-rebuild') return; + if (this.#stopped || this.status.state === 'unavailable') return; if (change === 'failed') { - this.#needsRebuild('backend reported a permanent failure'); + this.#needsRebuild('backend reported a permanent failure', 'backend-failed'); + return; + } + if (this.#rebuilding) { + if (change === 'accepted-work-lost') { + this.#rebuildFailed('backend lost accepted rebuild work', 'backend-failed'); + return; + } + if (this.#rebuildWaiter) this.#rebuildWaiter(); + else this.#rebuildWakePending = true; return; } + if (this.status.state === 'needs-rebuild') return; if (change === 'accepted-work-lost' && this.#owned) { - this.#pendingBatch = undefined; + this.#discardProgress(); try { this.#resetFromDurableCursor(); } catch (error) { - this.#fail('failed to reset lost accepted work', error); + this.#fail('failed to reset lost accepted work', error, 'backend-failed'); return; } } this.wake(true); } - #fail(reason: string, error: unknown) { + /** + * The error's message stays in the local status and log, since backend messages can quote record + * content; only the code reaches shared memory. A collector error carries its own code. + */ + #fail(reason: string, error: unknown, code: DerivedIndexReadinessReason = 'runner-failed') { const detail = error instanceof Error && error.message ? `${reason}: ${error.message}` : reason; - this.#needsRebuild(detail, error); + this.#needsRebuild(detail, error instanceof RunnerError ? error.code : code, error); } - #needsRebuild(reason: string, error?: unknown) { + #needsRebuild(reason: string, code: DerivedIndexReadinessReason, error?: unknown) { + if (this.#rebuilding) { + this.#rebuildFailed(reason, code, error); + return; + } if (this.status.state !== 'needs-rebuild') logger.error(`Derived index '${this.#registration.backend.id}' needs rebuild: ${reason}`, error); this.status = { state: 'needs-rebuild', reason, ownerEpoch: this.#ownerEpoch }; + this.#discardProgress(); + if (!this.#owned) return; + if (!this.#writeCondemnation()) { + this.#deferForCondemnation(code); + return; + } + if (this.#canRebuild()) { + // A failure after a rebuild but before `ready` is that rebuild failing late; it counts against the cap. + if (this.#rebuildAttempts >= this.#options.maxRebuildAttempts) { + this.#becomeUnavailable(reason, code, error); + return; + } + this.#publishReadiness('needs-rebuild', code); + this.#rebuildRequested = true; + this.#scheduleRebuild(); + return; + } + this.#publishReadiness('needs-rebuild', code); + this.#admitWrites(); + this.#release(); + } + + #becomeUnavailable(reason: string, code: DerivedIndexReadinessReason, error?: unknown) { + logger.error( + `Derived index '${this.#registration.backend.id}' is unavailable after ${this.#rebuildAttempts} rebuild attempts: ${reason}`, + error + ); + this.status = { state: 'unavailable', reason, ownerEpoch: this.#ownerEpoch }; + this.#publishReadiness('unavailable', code); + this.#admitWrites(); + this.#release(); + } + + /** An index no owner will catch up must not keep shedding writes. */ + #admitWrites() { + if (this.#lagBudget > 0) Atomics.store(this.#shared().words, READINESS_LAG_EXCEEDED, 0); + } + + #discardProgress() { + this.#generation++; + this.#stalledSince = undefined; + this.#lastCaughtUpAt = undefined; + this.#unreadSince = this.#options.now(); + this.#reachedEndOfLog = false; + this.#offered = undefined; this.#pendingBatch = undefined; + this.#carried = []; + try { + this.#iterator?.return?.(); + } catch (error) { + logger.warn?.(`Derived index '${this.#registration.backend.id}' log iterator close threw`, error); + } this.#iterator = undefined; this.#iterable = undefined; - this.#release(); + this.#boundaryPending = false; + if (this.#flushTimer) { + clearTimeout(this.#flushTimer); + this.#flushTimer = undefined; + } + } + + #scheduleRebuild() { + if (this.#rebuildTimer || this.#stopped) return; + const attempt = this.#rebuildAttempts; + if (attempt === 0) { + this.#startRebuild(); + return; + } + const delay = Math.min( + this.#options.rebuildBackoffMilliseconds * 2 ** (attempt - 1), + this.#options.maxRebuildBackoffMilliseconds + ); + this.#rebuildTimer = setTimeout(() => { + this.#rebuildTimer = undefined; + if (this.#stopped) return; + if (this.#owned) this.#startRebuild(); + else this.wake(true); + }, delay); + this.#rebuildTimer.unref?.(); + } + + #startRebuild() { + if (!this.#owned || this.#rebuilding || this.#stopped) return; + this.#rebuildRequested = false; + this.#takeSharedRebuildRequest(); + this.#rebuilding = true; + if (this.#lagTimer) { + clearTimeout(this.#lagTimer); + this.#lagTimer = undefined; + } + this.#rebuildWakePending = false; + if (this.#idleTimer) { + clearTimeout(this.#idleTimer); + this.#idleTimer = undefined; + } + this.#discardProgress(); + // The lag policy guards a durable cursor against retention; a rebuild has none to guard, and its + // replay anchor is captured fresh after the scan starts. Readers act on `rebuilding` instead. + this.#admitWrites(); + if (!this.#writeCondemnation()) { + this.#rebuilding = false; + this.#deferForCondemnation('rebuild-requested'); + return; + } + if (this.#rebuildAttempts >= this.#options.maxRebuildAttempts) { + this.#rebuilding = false; + this.#becomeUnavailable('rebuild budget exhausted by a previous owner', 'rebuild-exhausted'); + return; + } + const generation = this.#generation; + this.status = { state: 'rebuilding', ownerEpoch: this.#ownerEpoch }; + this.#rebuildAttempts++; + this.#publishReadiness('rebuilding'); + this.#runRebuild(generation).then( + () => { + if (!this.#live(generation)) return; + this.#rebuilding = false; + this.#rebuildRequested = false; + this.#takeSharedRebuildRequest(); + this.status = { state: 'running', ownerEpoch: this.#ownerEpoch }; + this.#drain(); + }, + (error) => { + if (!this.#live(generation)) return; + this.#rebuildFailed( + error instanceof Error && error.message ? error.message : String(error), + error instanceof RunnerError ? error.code : 'rebuild-failed', + error + ); + } + ); + } + + #live(generation: number): boolean { + return this.#owned && !this.#stopped && this.#generation === generation; + } + + async #runRebuild(generation: number) { + const backend = this.#registration.backend; + await this.#quiesce(this.#ownerEpoch!); + if (!this.#live(generation)) return; + this.#ownerEpoch = this.#mintEpoch(); + this.status = { state: 'rebuilding', ownerEpoch: this.#ownerEpoch }; + this.#publishReadiness('rebuilding'); + this.#resetting = Promise.resolve(backend.reset!(this.#ownerEpoch)); + try { + await this.#resetting; + } finally { + this.#resetting = undefined; + } + if (!this.#live(generation)) return; + if (backend.getDurableCursor() !== undefined) throw new Error('backend kept a durable cursor after reset'); + const boundary = this.#captureBoundary(); + const options = this.#options; + let chunk = this.#newChunk(true); + let indexed = 0; + for (const [tableId] of this.#registration.projections) { + for (const record of this.#scanRecords!(tableId)) { + if (this.#addScanRecord(chunk, tableId, record)) indexed++; + // Filtered entries (tombstones, symbol keys) count against the turn too: a long run of them + // must yield without delivering an empty chunk. + if ( + chunk.batch.records.length >= options.maxChunkRecords || + chunk.batch.bytes >= options.maxChunkBytes || + options.now() - chunk.started >= options.maxMillisecondsPerTurn + ) { + if (chunk.batch.records.length > 0) { + this.#noteChunkProjection(chunk); + await this.#deliverRebuildChunk(chunk, generation); + } else await new Promise((resolve) => setImmediate(resolve)); + if (!this.#live(generation)) return; + chunk = this.#newChunk(true); + } + } + } + this.#noteChunkProjection(chunk); + chunk.batch.through = boundary; + await this.#deliverRebuildChunk(chunk, generation); + if (!this.#live(generation)) return; + this.#rebuiltRecords = indexed; + if (!this.#installCursor(boundary)) return; + this.#boundaryPending = true; + logger.info?.(`Rebuilt derived index '${backend.id}' from ${indexed} records; replaying what committed since`); + } + + #addScanRecord(chunk: Chunk, tableId: number, record: DerivedIndexScanRecord): DerivedIndexMutation | undefined { + if (record.value == null || typeof record.recordId === 'symbol') return; + const key = writeKeyId(record.recordId); + let byRecord = chunk.resolved.get(tableId); + if (!byRecord) chunk.resolved.set(tableId, (byRecord = new Map())); + if (byRecord.has(key)) return; + const mutation: DerivedIndexMutation = { + tableId, + recordId: record.recordId, + logVersion: record.version, + state: this.#project(chunk, tableId, record.value, record.version, record.size), + }; + byRecord.set(key, mutation); + chunk.batch.records.push(mutation); + return mutation; + } + + async #deliverRebuildChunk(chunk: Chunk, generation: number) { + while (true) { + if (!this.#live(generation)) return; + const result = this.#deliver(chunk.batch); + if (result === undefined) { + if (this.#live(generation)) throw new Error('rebuild delivery was rejected'); + return; + } + if (result === DERIVED_INDEX_ACCEPTED) break; + this.#stalledSince ??= this.#options.now(); + await this.#waitForBackend(); + } + this.#stalledSince = undefined; + this.#noteAccepted(chunk.batch); + await new Promise((resolve) => setImmediate(resolve)); + } + + #waitForBackend(): Promise { + if (this.#rebuildWakePending) { + this.#rebuildWakePending = false; + return Promise.resolve(); + } + return new Promise((resolve) => { + const timer = setTimeout(() => this.#rebuildWaiter?.(), Math.max(1, this.#options.maxFlushAgeMilliseconds)); + timer.unref?.(); + this.#rebuildWaiter = () => { + clearTimeout(timer); + this.#rebuildWaiter = undefined; + resolve(); + }; + }); + } + + /** + * The committed tail of every log, captured before the scan. A committed read is a contiguous + * physical prefix (rocksdb-js advances `lastCommittedPosition` only to the earliest still-uncommitted + * write), so nothing committed after this point can sit behind it: the scan covers everything up to + * the tail and the replay from it covers everything after, including any reload marker committed + * during the scan, which then demands its own rebuild. A log with no committed transaction is left + * out of the cursor and read from its beginning, which it must still retain. + */ + #captureBoundary(): DerivedIndexCursor { + const boundary: DerivedIndexCursor = { format: 1, logs: {} }; + for (const logName of this.#logStore.rootStore.listLogs()) { + let tail: number | undefined; + const range = this.#logStore.getRange({ log: logName, start: 0 }); + for (const entry of range) if (entry.endTxn) tail = entry.txnLogKey; + if (range.corruptFrameStop.breaks > 0 || range.failedLogs.size > 0) + throw new RunnerError('log-corrupt', `transaction log '${logName}' cannot be read to its committed tail`); + if (tail === undefined) { + if (!this.#retainsBeginning(logName)) + throw new RunnerError( + 'log-retention', + `transaction log '${logName}' retains no committed transaction and has lost its beginning` + ); + continue; + } + boundary.logs[logName] = tail; + } + return boundary; + } + + #rebuildFailed(reason: string, code: DerivedIndexReadinessReason, error?: unknown) { + this.#rebuilding = false; + this.#rebuildWaiter?.(); + this.#discardProgress(); + if (this.#rebuildAttempts >= this.#options.maxRebuildAttempts) { + this.#becomeUnavailable(reason, code, error); + return; + } + logger.error( + `Derived index '${this.#registration.backend.id}' rebuild attempt ${this.#rebuildAttempts} failed: ${reason}`, + error + ); + this.status = { state: 'needs-rebuild', reason, ownerEpoch: this.#ownerEpoch }; + this.#publishReadiness('needs-rebuild', code); + this.#rebuildRequested = true; + if (this.#owned) this.#scheduleRebuild(); + } + + /** One `shutdown(epoch)` per epoch, shared by a rebuild attempt and a release that overlap. */ + #quiesce(epoch: bigint): Promise { + if (this.#quiescing?.epoch === epoch) return this.#quiescing.promise; + let promise: Promise; + try { + promise = Promise.resolve(this.#registration.backend.shutdown(epoch)); + } catch (error) { + promise = Promise.reject(error); + } + const quiescing = { epoch, promise, since: this.#options.now() }; + this.#quiescing = quiescing; + const settle = () => { + if (this.#quiescing === quiescing) this.#quiescing = undefined; + }; + promise.then(settle, settle); + return promise; + } + + #publishReadiness(state: DerivedIndexReadinessState, reason: DerivedIndexReadinessReason = 'none') { + const { words } = this.#shared(); + // State last: a reader that sees the new state sees a reason and attempt count at least as new. + Atomics.store(words, READINESS_REASON, READINESS_REASONS.indexOf(reason)); + Atomics.store(words, READINESS_ATTEMPTS, state === 'ready' ? 0 : this.#rebuildAttempts); + Atomics.store(words, READINESS_STATE, READINESS_STATES.indexOf(state)); } #release() { @@ -578,17 +1837,109 @@ class DerivedIndexRunner { clearTimeout(this.#idleTimer); this.#idleTimer = undefined; } + if (this.#rebuildTimer) { + clearTimeout(this.#rebuildTimer); + this.#rebuildTimer = undefined; + } this.#owned = false; + this.#rebuilding = false; + this.#rebuildWaiter?.(); + if (this.#lagTimer) { + clearTimeout(this.#lagTimer); + this.#lagTimer = undefined; + } + this.#discardProgress(); + const backend = this.#registration.backend; + const epoch = this.#ownerEpoch; + this.#releasingSince = this.#options.now(); + const unlock = () => { + this.#releasing = undefined; + this.#releasingSince = undefined; + try { + this.#logStore.unlock(this.#lockKey); + } catch (error) { + logger.error(`Failed to release derived index runner '${backend.id}'`, error); + } + }; + const hold = (error: unknown) => { + this.#releasing = undefined; + this.#releasingSince = undefined; + this.#heldLock = true; + const reason = `backend shutdown failed; runner lock held: ${error instanceof Error ? error.message : String(error)}`; + logger.error(`Derived index '${backend.id}' ${reason}`, error); + this.#releaseFailure = new Error(reason, { cause: error }); + this.status = { state: 'unavailable', reason, ownerEpoch: epoch }; + this.#publishReadiness('unavailable', 'shutdown-failed'); + this.#admitWrites(); + }; + let flushed: void | Promise; try { - this.#logStore.unlock(this.#lockKey); + flushed = backend.flush('shutdown'); } catch (error) { - logger.error(`Failed to release derived index runner '${this.#registration.backend.id}'`, error); + logger.warn?.(`Derived index '${backend.id}' shutdown flush request threw`, error); } - this.#iterator = undefined; - this.#iterable = undefined; + const settling = Promise.allSettled([this.#resetting, flushed]).then(() => undefined); + this.#releasing = settling.then(() => (epoch === undefined ? undefined : this.#quiesce(epoch))).then(unlock, hold); } } +function lastOpen(collected: CollectedTransaction[]): CollectedTransaction | undefined { + const last = collected[collected.length - 1]; + return last && !last.complete ? last : undefined; +} + +type SharedReadinessBuffer = ArrayBufferLike & { notify?: () => void; cancel?: () => void }; + +function readinessBuffer( + logStore: RocksTransactionLogStore, + backendId: string, + callback?: () => void +): SharedReadinessBuffer { + return logStore.getUserSharedBuffer( + `derived-index:${backendId}:readiness`, + new ArrayBuffer(READINESS_BYTES), + callback ? { callback } : undefined + ) as SharedReadinessBuffer; +} + +type SharedViews = { + words: Int32Array; + epoch: BigInt64Array; +}; + +function sharedViewsOf(buffer: ArrayBufferLike): SharedViews { + return { + words: new Int32Array(buffer, 0, READINESS_WORDS), + epoch: new BigInt64Array(buffer, READINESS_EPOCH_OFFSET, 1), + }; +} + +const readinessViews = new WeakMap>(); + +function readReadiness({ words, epoch }: SharedViews): DerivedIndexReadiness { + const state = READINESS_STATES[Atomics.load(words, READINESS_STATE)] ?? 'unknown'; + const reason = READINESS_REASONS[Atomics.load(words, READINESS_REASON)] ?? 'none'; + const readiness: DerivedIndexReadiness = { + state, + ownerEpoch: Atomics.load(epoch, 0), + rebuildAttempts: Atomics.load(words, READINESS_ATTEMPTS), + }; + if (reason !== 'none') readiness.reason = reason; + return readiness; +} + +/** Read an index's shared readiness on any worker, without a registered runtime. */ +export function readDerivedIndexReadiness( + logStore: RocksTransactionLogStore, + backendId: string +): DerivedIndexReadiness { + let byBackend = readinessViews.get(logStore); + if (!byBackend) readinessViews.set(logStore, (byBackend = new Map())); + let views = byBackend.get(backendId); + if (!views) byBackend.set(backendId, (views = sharedViewsOf(readinessBuffer(logStore, backendId)))); + return readReadiness(views); +} + function isValidCursor(cursor: DerivedIndexCursor | undefined): cursor is DerivedIndexCursor { if ( !cursor || @@ -609,8 +1960,8 @@ function cloneCursor(cursor: DerivedIndexCursor): DerivedIndexCursor { return { format: 1, logs: { ...cursor.logs } }; } -function sameCursor(left: DerivedIndexCursor | undefined, right: DerivedIndexCursor): boolean { - if (!isValidCursor(left)) return false; +function sameCursor(left: DerivedIndexCursor | undefined, right: DerivedIndexCursor | undefined): boolean { + if (!isValidCursor(left) || !isValidCursor(right)) return false; const leftNames = Object.keys(left.logs); const rightNames = Object.keys(right.logs); if (leftNames.length !== rightNames.length) return false; diff --git a/resources/graphql.ts b/resources/graphql.ts index 301e8bd6c1..edeeda029e 100644 --- a/resources/graphql.ts +++ b/resources/graphql.ts @@ -1,7 +1,6 @@ import { dirname } from 'path'; import { Script } from 'node:vm'; -import { table } from './databases.ts'; -import { assertTableTargetNotBranched } from './branchGuard.ts'; +import { scopedTableFactory, table } from './databases.ts'; import { getWorkerIndex } from '../server/threads/manageThreads.js'; import { Resources } from './Resources.ts'; import type { NamedTypeNode, StringValueNode, ValueNode } from 'graphql'; @@ -80,7 +79,7 @@ export function handleApplication(scope: import('../components/Scope.ts').Scope) entry.urlPath, entry.absolutePath, scope.resources, - scope.applicationScope?.branches, + scopedTableFactory(scope.applicationScope?.branches), scope.logger ); }); @@ -102,7 +101,7 @@ async function processGraphQLSchema( urlPath, filePath, resources, - branches?: Map, + declareTable: typeof table = table, logger: { warn?: (...args: any[]) => void; error?: (...args: any[]) => void } = harperLogger ) { // lazy load the graphql package so we don't load it for users that don't use graphql @@ -369,16 +368,7 @@ async function processGraphQLSchema( for (const typeDef of tables) { // with graphql database definitions, this is a declaration that the table should exist and that it // should be created if it does not exist - try { - assertTableTargetNotBranched(branches, typeDef.database, typeDef.table, 'a GraphQL @table directive'); - } catch (error) { - // Reported and skipped rather than thrown: the refusal is scoped to this one branched table, - // and skipping leaves the base schema untouched, which is the point of it. The rest of the - // schema — and the rest of the application — still loads. - logger.error?.((error as Error).message); - continue; - } - typeDef.tableClass = table(typeDef); + typeDef.tableClass = declareTable(typeDef); if (getWorkerIndex() === 0) { // Post-Phase-2: typeDef.properties is the canonical Record (no .find); read the Array form. const pk = (typeDef.attributes as any[])?.find((p) => p.isPrimaryKey)?.name ?? 'id'; diff --git a/resources/models/backendRegistry.ts b/resources/models/backendRegistry.ts index 3a350a2dc9..fc4fdbacd6 100644 --- a/resources/models/backendRegistry.ts +++ b/resources/models/backendRegistry.ts @@ -1,3 +1,4 @@ +import { AsyncLocalStorage } from 'node:async_hooks'; import { ServerError } from '../../utility/errors/hdbError.ts'; import type { DefineBackendSpec, GenerateResult, ModelBackend, ModelCapabilities, ToolCall } from './types.ts'; @@ -24,14 +25,95 @@ type ModelKind = 'embedding' | 'generative'; const embedding: Map = new Map(); const generative: Map = new Map(); +/** A registration a factory made during construction, deferred to the caller. */ +export interface CapturedInstall { + kind: ModelKind; + logicalName: string; + backend: ModelBackend; +} + +interface CaptureSlot { + kind: ModelKind; + logicalName: string; + backend?: ModelBackend; + /** Deferred with the primary, so no request observes a new helper next to an old primary. */ + extras: CapturedInstall[]; + /** Async work spawned by a factory retains the ALS context past construction; once construction + * ends the scope deactivates so a late same-slot registration installs normally. */ + active: boolean; +} + +// Async-context scoped: a module-global slot would divert unrelated registrations and collide +// concurrent constructions. +const captureScope = new AsyncLocalStorage(); + +function install(kind: ModelKind, logicalName: string, backend: ModelBackend): void { + const slot = captureScope.getStore(); + if (slot?.active) { + if (slot.kind === kind && slot.logicalName === logicalName) slot.backend = backend; + else slot.extras.push({ kind, logicalName, backend }); + return; + } + (kind === 'embedding' ? embedding : generative).set(logicalName, backend); +} + /** Map `logicalName` to a backend for embedding calls. Re-set replaces. */ export function setEmbedding(logicalName: string, backend: ModelBackend): void { - embedding.set(logicalName, backend); + install('embedding', logicalName, backend); } /** Map `logicalName` to a backend for generative calls. Re-set replaces. */ export function setGenerative(logicalName: string, backend: ModelBackend): void { - generative.set(logicalName, backend); + install('generative', logicalName, backend); +} + +/** + * Build a backend through its normal registration path but return it instead of installing it, so a + * config reload can install it conditionally. A scratch logical name would be briefly visible + * through `listBackends`, which backs the public `GET /v1/models`. + */ +export async function constructBackend( + kind: ModelKind, + logicalName: string, + register: () => void | Promise +): Promise<{ backend?: ModelBackend; extras: CapturedInstall[] }> { + const slot: CaptureSlot = { kind, logicalName, extras: [], active: true }; + try { + await captureScope.run(slot, async () => { + await register(); + }); + } finally { + slot.active = false; + } + return { backend: slot.backend, extras: slot.extras }; +} + +/** + * Replace `logicalName`'s backend for `kind` only if it is still `expected`. A plain `set` is atomic + * but unconditional, so concurrent writers resolve by arrival order rather than by which value is + * newer. A caller that loses the swap must re-derive from current state rather than overwrite. + */ +export function replaceIfCurrent( + kind: ModelKind, + logicalName: string, + expected: ModelBackend | undefined, + next: ModelBackend +): boolean { + const map = kind === 'embedding' ? embedding : generative; + if (map.get(logicalName) !== expected) return false; + map.set(logicalName, next); + return true; +} + +/** + * Remove `logicalName`'s backend for `kind` only if it is still `expected`, returning whether it was + * removed. Withdrawing a credential must not delete a slot another writer has since taken over. + */ +export function removeIfCurrent(kind: ModelKind, logicalName: string, expected: ModelBackend | undefined): boolean { + const map = kind === 'embedding' ? embedding : generative; + if (map.get(logicalName) !== expected) return false; + map.delete(logicalName); + return true; } /** Non-throwing lookup of the backend mapped to `logicalName` for `kind`, or `undefined`. Used by the router to assemble + filter candidate lists without exceptions. */ diff --git a/resources/models/bootstrap.ts b/resources/models/bootstrap.ts index 551efa0fd5..583f5ee990 100644 --- a/resources/models/bootstrap.ts +++ b/resources/models/bootstrap.ts @@ -1,5 +1,8 @@ /** - * YAML→registry boot bridge (#629 / #630 of #510). + * YAML→registry boot bridge (#629 / #630 of #510), plus hot reload of the + * `models` block when the root config file changes (#2344) — so an + * orchestrator can rotate a credential or re-point a baseUrl by rewriting + * harperdb-config.yaml, without a process restart. * * Reads the top-level `models` block from the root config and dispatches each * `models.embedding.` / `models.generative.` entry to the matching @@ -22,7 +25,8 @@ * convention from `@harperfast/oauth`'s config loader. * * Errors per entry are logged and skipped, not thrown — one misconfigured - * backend should not block Harper boot. + * backend should not block Harper boot, and one bad entry in a rewritten file + * should not tear down the entries that were fine. */ import harperLogger from '../../utility/logging/harper_logger.ts'; import { expandEnvVarsDeep, isUnresolvedEnvVarPlaceholder } from '../../utility/expandEnvVar.ts'; @@ -35,6 +39,16 @@ import { pathToFileURL } from 'node:url'; import { createRequire } from 'node:module'; import { getHdbBasePath } from '../../utility/environment/environmentManager.ts'; import { setFallbackGroup, clearFallbackGroups } from './routing.ts'; +import { + constructBackend, + getBackend, + removeIfCurrent, + replaceIfCurrent, + type CapturedInstall, +} from './backendRegistry.ts'; +import { getSharedRootConfigWatcher, RootConfigWatcher } from '../../config/RootConfigWatcher.ts'; +import { validateModelsBlock } from '../../validation/configValidator.ts'; +import type { ModelBackend } from './types.ts'; /** * Field names treated as credentials. When present in config as a literal @@ -79,36 +93,149 @@ const FACTORIES: Record = { bedrock: (args) => registerBedrockBackend({ ...args, config: args.config as BedrockBackendConfig }), }; +// What the config projection currently has installed, per kind+logicalName slot. `backend` is the +// exact instance this module put in the registry: reload swaps compare against it, so a +// programmatic `registerBackend` that overrode a config entry keeps its documented precedence, and +// removal can never delete a slot another writer has since taken over. +interface InstalledSlot { + kind: ModelKind; + logicalName: string; + /** The instance this projection installed; absent while an application override owns the slot. */ + backend?: ModelBackend; + /** Raw entry as applied, for change detection — an unchanged entry is not reconstructed. */ + entryJson: string; + /** The entry's fallback group as applied, so a retained backend keeps its routing. */ + fallback?: string[]; + /** Whether the entry's backend is a built-in. Module-backed entries are restart-managed: reload + * refuses to add, change, OR remove them, so a rename cannot half-apply as a bare removal. */ + builtin?: boolean; + /** Helper registrations the entry's factory made, installed and removed with the entry. + * `suppressed`: a config entry claimed this helper's name; the record is kept so the helper is + * restored when that entry is removed — matching a restart with the final config. */ + extras?: Array; +} + +const installedSlots = new Map(); + +/** + * Let a config entry claim a name held by a projection-installed helper. The record is suppressed + * rather than deleted, so removing the claiming entry later restores the helper — the live + * registry then matches a restart with the same final config. + */ +function claimHelperOccupant(kind: ModelKind, logicalName: string): ModelBackend | undefined { + for (const slot of installedSlots.values()) { + const extra = slot.extras?.find((e) => !e.suppressed && e.kind === kind && e.logicalName === logicalName); + if (extra) { + extra.suppressed = true; + return extra.backend; + } + } + return undefined; +} + +/** Restore a suppressed helper once the entry that claimed its name is removed. */ +function restoreSuppressedHelper(kind: ModelKind, logicalName: string): void { + for (const slot of installedSlots.values()) { + const extra = slot.extras?.find((e) => e.suppressed && e.kind === kind && e.logicalName === logicalName); + if (extra && replaceIfCurrent(extra.kind, extra.logicalName, undefined, extra.backend)) { + extra.suppressed = false; + return; + } + } +} + +const slotKey = (kind: ModelKind, logicalName: string) => `${kind} ${logicalName}`; + /** * Populate the model registry from `rootConfig.models`. No-op if the block * is absent or empty. Idempotent within a process: each entry overwrites any - * prior registration under the same logical name (registry uses `.set()`). + * prior registration under the same logical name. */ export async function bootstrapModels(rootConfig: RootConfig | undefined | null): Promise { - // Rebuild fallback groups from scratch each (re)load so a removed/changed `fallback:` - // (or a removed `models:` block) doesn't leave stale routing behind (#1326). - clearFallbackGroups(); - const block = rootConfig?.models; - if (!block) return; - await registerKind('embedding', block.embedding); - await registerKind('generative', block.generative); + // The projection is NOT reset here: boot rides the same serialized queue as a reload, so a + // re-bootstrap in a long-lived process removes entries the new config dropped, keeps retained + // state (a malformed entry's routing), and cannot corrupt an in-flight apply's swaps. + return queueModelsApply(rootConfig?.models, true); } -function warnOnLiteralCredentials(kind: ModelKind, logicalName: string, entry: ModelEntry): void { - for (const field of CREDENTIAL_FIELDS) { - const value = (entry as Record)[field]; - if (typeof value !== 'string' || value.length === 0) continue; - if (isUnresolvedEnvVarPlaceholder(value)) continue; // operator is using ${VAR} indirection - harperLogger.warn( - `models.${kind}.${logicalName}: '${field}' is a literal value in harperdb-config.yaml; ` + - `prefer \${ENV_VAR} indirection for credentials to keep them off disk` - ); +/** Forget everything the projection installed. Test isolation only — never part of a reload. */ +export function resetModelsProjection(): void { + installedSlots.clear(); +} + +/** + * Apply a `models:` block as a hot reload: changed entries are rebuilt through the same factories + * boot uses and swapped in atomically, removed entries stop serving, and programmatic + * registrations that overrode a config entry are left in place. Serialized and coalesced — + * concurrent calls settle on the latest block. + */ +export function applyModelsConfig(models: ModelsConfig | undefined): Promise { + return queueModelsApply(models, false); +} + +// Interleaved applies could install an older credential last; serialized + coalesced, they cannot. +// Boot and reload coalesce SEPARATELY: merging them either loses boot's overwrite contract or +// launders a raw watcher block through boot semantics, past reload validation and the missing-key +// no-op. Boot drains first; the newest reload then refines it under its own rules. +let pendingBoot: { block: ModelsConfig | undefined } | undefined; +let pendingReload: { block: ModelsConfig | null | undefined } | undefined; +let applyChain: Promise | undefined; + +function queueModelsApply(block: ModelsConfig | null | undefined, isBoot: boolean): Promise { + if (isBoot) { + pendingBoot = { block: block ?? undefined }; + // A boot supersedes any reload queued before it — draining that older snapshot after the + // newer boot would refine it backwards. A reload queued after the boot still refines. The + // same applies one layer down: a watcher snapshot OBSERVED before this boot whose settle + // timer has not fired yet is also pre-boot content — boot read the same file, so dropping + // the timer loses nothing newer. + pendingReload = undefined; + clearTimeout(pendingSettle); + pendingSettle = undefined; + } else { + pendingReload = { block }; } + applyChain ??= (async () => { + try { + while (pendingBoot || pendingReload) { + const isBootTurn = pendingBoot !== undefined; + const next = isBootTurn ? pendingBoot! : pendingReload!; + if (isBootTurn) pendingBoot = undefined; + else pendingReload = undefined; + try { + await applyModels(next.block, isBootTurn); + } catch (err) { + // Per-entry failures are handled inside applyModels; this catches a failure of the apply + // itself. The chain must not reject — the watcher calls fire-and-forget, so a rejection + // would be unhandled — and the loop continues so a newer pending snapshot still applies. + harperLogger.error(`models: config apply failed (${(err as Error)?.message ?? err})`); + } + } + } finally { + applyChain = undefined; + } + })(); + return applyChain; } -async function registerKind(kind: ModelKind, entries: Record | undefined): Promise { +interface DesiredEntry { + kind: ModelKind; + logicalName: string; + entry: ModelEntry; + entryJson: string; +} + +function collectKind( + kind: ModelKind, + entries: Record | undefined, + out: DesiredEntry[], + present: Set +): void { if (!entries) return; for (const [logicalName, entry] of Object.entries(entries)) { + // Present even when invalid: a malformed entry keeps its previously applied backend (like a + // failed rebuild does) rather than reading as removed. Only true absence removes. + present.add(slotKey(kind, logicalName)); if (!entry || typeof entry !== 'object') { // Schema validation (configValidator.ts) catches this before bootstrap // runs, so reaching here means config was loaded by an unusual path @@ -120,6 +247,125 @@ async function registerKind(kind: ModelKind, entries: Record harperLogger.error(`models.${kind}.${logicalName}: 'backend' must be a non-empty string; skipping`); continue; } + out.push({ kind, logicalName, entry, entryJson: JSON.stringify(entry) }); + } +} + +function publishEntry( + desiredEntry: DesiredEntry, + backend: ModelBackend, + extras: CapturedInstall[], + isBoot: boolean +): void { + const { kind, logicalName, entry, entryJson } = desiredEntry; + const key = slotKey(kind, logicalName); + // Boot overwrites occupants (the documented contract); a reload replaces only what this + // projection installed — or a helper the projection installed under this name, which a config + // entry outranks — so genuine application overrides survive. + const previous = installedSlots.get(key); + let expected = isBoot ? getBackend(kind, logicalName) : previous?.backend; + // Claimed at boot too, or the parent record keeps an installable claim on a name it lost. + const claimedHelper = claimHelperOccupant(kind, logicalName); + if (!isBoot && expected === undefined && claimedHelper !== undefined) expected = claimedHelper; + // Helpers rotate and retire with their entry even when the primary swap is lost to an + // override — a frozen helper would keep serving a revoked credential. + const installedExtras: Array = []; + for (const extra of extras) { + const extraExpected = isBoot + ? getBackend(extra.kind, extra.logicalName) + : previous?.extras?.find((e) => e.kind === extra.kind && e.logicalName === extra.logicalName)?.backend; + if (replaceIfCurrent(extra.kind, extra.logicalName, extraExpected, extra.backend)) { + installedExtras.push(extra); + } else { + harperLogger.warn( + `models.${kind}.${logicalName}: helper '${extra.logicalName}' is owned by another registration; leaving it in place` + ); + } + } + for (const old of previous?.extras ?? []) { + if (old.suppressed) continue; + if (!extras.some((e) => e.kind === old.kind && e.logicalName === old.logicalName)) { + removeIfCurrent(old.kind, old.logicalName, old.backend); + } + } + const builtin = Boolean(FACTORIES[entry.backend as string]); + if (replaceIfCurrent(kind, logicalName, expected, backend)) { + installedSlots.set(key, { + kind, + logicalName, + backend, + entryJson, + builtin, + fallback: entry.fallback, + extras: installedExtras, + }); + } else { + // Record the ask with no installed instance, so unchanged reloads skip instead of + // re-losing this swap every apply; installed helpers stay recorded and removable. + installedSlots.set(key, { + kind, + logicalName, + entryJson, + builtin, + fallback: entry.fallback, + extras: installedExtras, + }); + harperLogger.warn(`models.${kind}.${logicalName}: another registration owns this entry; leaving it in place`); + } +} + +async function applyModels(block: ModelsConfig | null | undefined, isBoot: boolean): Promise { + if (!isBoot) { + if (block === undefined) { + // A snapshot with no `models` key is not evidence of intent: a non-atomic in-place rewrite + // can be observed as a valid YAML prefix that simply hasn't reached the block yet, and + // removing every backend on that would be catastrophic. Removal is expressed by an empty + // or shrunken block — `models: {}`. A bare `models:` key (null) is rejected by validation + // below, exactly as boot's validator rejects it, so reload cannot accept a file a restart + // would refuse. + harperLogger.debug?.('models: watched config has no models block; projection left as-is'); + return; + } + // Boot validates the composed config (configValidator); a reload must enforce the same + // schema, or a typo boot would reject hot-applies silently (e.g. a misspelled baseUrl + // falling back to the public endpoint). + const { error } = validateModelsBlock(block); + if (error) { + harperLogger.error(`models: rejecting config reload (${error.message}); keeping the previous projection`); + return; + } + } + const desired: DesiredEntry[] = []; + const presentKeys = new Set(); + collectKind('embedding', block?.embedding, desired, presentKeys); + collectKind('generative', block?.generative, desired, presentKeys); + + const staged = new Map(); + const failedKeys = new Set(); + for (const desiredEntry of desired) { + const { kind, logicalName, entry, entryJson } = desiredEntry; + const key = slotKey(kind, logicalName); + const slot = installedSlots.get(key); + // Unchanged: skip while ANY occupant serves (rebuilding under an override would run the factory + // only to lose the swap). Boot is stricter — it must overwrite occupants, so it only skips when + // the projection's own instance is installed; an emptied registry rebuilds either way. + const installed = getBackend(kind, logicalName); + if (slot && slot.entryJson === entryJson && (isBoot ? installed === slot.backend : installed !== undefined)) + continue; + // Reload runs factories for built-ins only: they are independent, pure constructors. A module + // factory may compose with other entries (wrap an earlier backend), which staged construction + // cannot honor — changing one needs a restart, exactly as before this feature. The guard covers + // the incoming backend AND a currently-installed module-backed slot: a module→built-in rewrite + // is still a change of a restart-managed entry, so it must not live-replace the module (which + // would drop its helpers with none of the disposal a restart performs). + if (!isBoot && (!FACTORIES[entry.backend as string] || slot?.builtin === false)) { + harperLogger.warn( + `models.${kind}.${logicalName}: module-backed entries require a restart to add or change; ` + + `keeping the previous projection for this entry` + ); + failedKeys.add(key); + continue; + } // Warn before expansion: literal credentials in `harperdb-config.yaml` // land on disk, in backups, and (depending on deployment) in replicated // config tables. The `${VAR}` indirection pattern from @@ -132,19 +378,218 @@ async function registerKind(kind: ModelKind, entries: Record // (env var unset) pass through unchanged — backend's required-field // validation catches them with a meaningful error. const config = expandEnvVarsDeep(entry); - const builtin = FACTORIES[entry.backend]; - if (builtin) { - await builtin({ logicalName, kind, config }); - } else { - // Not a built-in name: treat `backend` as a module specifier (#1471). - await registerFromModule(kind, logicalName, entry.backend, config); + const { backend, extras } = await constructBackend(kind, logicalName, async () => { + const builtin = FACTORIES[entry.backend as string]; + if (builtin) { + await builtin({ logicalName, kind, config }); + } else { + // Not a built-in name: treat `backend` as a module specifier (#1471). + await registerFromModule(kind, logicalName, entry.backend as string, config); + } + }); + if (!backend) { + harperLogger.error( + `models.${kind}.${logicalName}: registration installed no backend; skipping ` + + `(a factory must register before its returned promise resolves)` + ); + failedKeys.add(key); + continue; } - // Record the ordered fallback group (other logical names) for the router (#1326). - if (entry.fallback?.length) setFallbackGroup(kind, logicalName, entry.fallback); + // Boot publishes each entry as it is built, in config order, so a later module factory + // observes earlier entries exactly as it always has (a wrapper resolves its base). Reload + // defers everything to one synchronous publish, so a request never sees a partial rebuild. + if (isBoot) publishEntry(desiredEntry, backend, extras, true); + else staged.set(key, { desiredEntry, backend, extras }); } catch (err) { + failedKeys.add(key); harperLogger.error(`models.${kind}.${logicalName}: registration failed (${(err as Error)?.message ?? err})`); } } + + for (const { desiredEntry, backend, extras } of staged.values()) { + publishEntry(desiredEntry, backend, extras, isBoot); + } + const desiredKeys = new Set(desired.map((d) => slotKey(d.kind, d.logicalName))); + // Module-backed slots retained across a removal are absent from both `desiredKeys` and `presentKeys`, + // so neither routing loop below would restore them after the clear; track them so their recorded + // fallback survives — else the primary keeps serving while its failover is silently dropped. + const retainedModuleKeys = new Set(); + for (const [key, slot] of [...installedSlots]) { + if (presentKeys.has(key)) continue; + // Module-backed entries are restart-managed on reload in BOTH directions: refusing an added + // rename target while removing its old name would turn the rename into a bare removal. + if (!isBoot && slot.builtin === false) { + harperLogger.warn( + `models.${slot.kind}.${slot.logicalName}: module-backed entries require a restart to remove; keeping it` + ); + retainedModuleKeys.add(key); + continue; + } + if (slot.backend) removeIfCurrent(slot.kind, slot.logicalName, slot.backend); + for (const extra of slot.extras ?? []) { + if (!extra.suppressed) removeIfCurrent(extra.kind, extra.logicalName, extra.backend); + } + installedSlots.delete(key); + // A removed entry may have been shadowing a factory helper of the same name; put the + // helper back, so the live registry matches a restart with this final config. Not gated on + // slot.backend: the claim suppressed the helper before this entry's swap was known to win, so + // a lost swap must still release it (a no-op while the name is held or nothing is suppressed). + restoreSuppressedHelper(slot.kind, slot.logicalName); + } + // Rebuild fallback routing from scratch each apply so a removed/changed `fallback:` (or a + // removed `models:` block) doesn't leave stale routing behind (#1326). + clearFallbackGroups(); + for (const { kind, logicalName, entry } of desired) { + // A failed rebuild keeps its previous backend serving, so it keeps the routing that was + // applied WITH that backend — not the new entry's group, which belongs to the build that + // did not happen. + const key = slotKey(kind, logicalName); + const group = failedKeys.has(key) ? installedSlots.get(key)?.fallback : entry.fallback; + if (group?.length && getBackend(kind, logicalName)) setFallbackGroup(kind, logicalName, group); + } + // Routing follows the name, not the instance. + for (const [key, slot] of installedSlots) { + if (desiredKeys.has(key) || !presentKeys.has(key)) continue; + if (slot.fallback?.length && getBackend(slot.kind, slot.logicalName)) { + setFallbackGroup(slot.kind, slot.logicalName, slot.fallback); + } + } + // Retained module-backed slots keep serving under their old name, so they keep the fallback group + // they were built with — a restart-managed removal must not half-apply as a silent failover loss. + for (const key of retainedModuleKeys) { + const slot = installedSlots.get(key); + if (slot?.fallback?.length && getBackend(slot.kind, slot.logicalName)) { + setFallbackGroup(slot.kind, slot.logicalName, slot.fallback); + } + } +} + +// ── Hot reload wiring ───────────────────────────────────────────────────────── + +let modelsConfigWatcher: RootConfigWatcher | undefined; +let ownsModelsConfigWatcher = false; +let modelsApplyListener: ((config: unknown) => void) | undefined; +let pendingSettle: NodeJS.Timeout | undefined; + +/** + * Watch the root config file and hot-apply `models:` changes. Follows `harper_logger`'s + * root-config-watch pattern: one watcher per worker, each worker reprojecting its own registry. + * + * The watcher sees the FILE, but `HARPER_DEFAULT_CONFIG` / `HARPER_CONFIG` / `HARPER_SET_CONFIG` + * compose with it at boot — so if any of those names `models`, the file alone is not authoritative + * for the block and hot reload stays off, preserving boot semantics unchanged. (That is also the + * compatibility gate: an orchestrator still injecting models through `HARPER_SET_CONFIG` gets + * today's restart behavior; one that writes the file instead gets live reload.) + * + * Returns whether the watch is active. + */ +export function startModelsConfigHotReload(options?: { + configFilePath?: string; + debounceMs?: number; + /** Test seam: invoked when a watcher snapshot is observed, before its settle timer is armed. */ + onSnapshotObserved?: () => void; + /** Test seam: subscribe to this instance (caller-owned) instead of constructing or sharing. */ + watcher?: RootConfigWatcher; +}): boolean { + if (modelsConfigWatcher) return true; + const pinnedBy = envLayerNamingModels(); + if (pinnedBy) { + harperLogger.info( + `models: hot reload of the config file is disabled: ${pinnedBy} also defines 'models', so the file alone is not authoritative for it` + ); + return false; + } + // One watcher per isolate: logging already opens one, and a second per worker doubles the + // native-watcher/FD footprint and the read+parse work on every config write. A test-supplied + // path gets a private instance, owned (and closed) by this module. + ownsModelsConfigWatcher = options?.watcher === undefined && options?.configFilePath !== undefined; + modelsConfigWatcher = + options?.watcher ?? + (ownsModelsConfigWatcher ? new RootConfigWatcher(options?.configFilePath) : getSharedRootConfigWatcher()); + // An 'error' event with no listener would take the worker down; and `ready` is an events.once + // promise that rejects on a pre-ready 'error', so it must be observed too. + modelsConfigWatcher.on('error', modelsWatcherErrorListener); + modelsConfigWatcher.ready.catch(() => {}); + // 'ready' carries the file state at watch start: a rewrite that lands during the watcher's + // initial scan arrives there rather than as 'change', and an unchanged block is a no-op anyway. + // Settle window: a non-atomic in-place rewrite emits an event per write() and an early snapshot + // can be a schema-valid subset of the final file (a truncated map removes the unwritten + // entries). Applying only the last event in a quiet window adopts the completed file. The real + // contract for orchestrators remains an atomic tmp+rename. + const debounceMs = options?.debounceMs ?? 150; + const applyFromFile = (config: unknown) => { + const models = (config as RootConfig | undefined)?.models; + options?.onSnapshotObserved?.(); + clearTimeout(pendingSettle); + pendingSettle = setTimeout(() => { + void queueModelsApply(models, false); + }, debounceMs); + pendingSettle.unref?.(); + }; + modelsConfigWatcher.on('ready', applyFromFile); + modelsConfigWatcher.on('change', applyFromFile); + modelsApplyListener = applyFromFile; + // The shared watcher's one-time 'ready' may predate this subscription (logging usually + // constructs the singleton first, and EventEmitter does not replay). A snapshot it already + // holds is applied directly, so a rewrite landing before this subscription is not invisible + // until the next write. + if (modelsConfigWatcher.config !== undefined) applyFromFile(modelsConfigWatcher.config); + return true; +} + +/** Stop watching the config file. The current projection keeps serving. */ +export function stopModelsConfigHotReload(): void { + clearTimeout(pendingSettle); + pendingSettle = undefined; + if (modelsConfigWatcher) { + if (modelsApplyListener) { + modelsConfigWatcher.off('ready', modelsApplyListener); + modelsConfigWatcher.off('change', modelsApplyListener); + } + modelsConfigWatcher.off('error', modelsWatcherErrorListener); + // The shared watcher belongs to the isolate (logging still consumes it); close only a + // private, test-supplied instance. + if (ownsModelsConfigWatcher) modelsConfigWatcher.close(); + } + modelsApplyListener = undefined; + modelsConfigWatcher = undefined; +} + +function modelsWatcherErrorListener(error: unknown): void { + harperLogger.warn(`models: config watcher error: ${(error as Error)?.message ?? error}`); +} + +function envLayerNamingModels(): string | undefined { + for (const name of ['HARPER_SET_CONFIG', 'HARPER_CONFIG', 'HARPER_DEFAULT_CONFIG']) { + const raw = process.env[name]; + if (!raw) continue; + try { + const parsed = JSON.parse(raw); + // Dotted top-level keys ("models.embedding.default") compose into the models block too + // (harperConfigEnvVars flattens/expands them), so they pin exactly like a nested key. + if ( + parsed && + typeof parsed === 'object' && + Object.keys(parsed).some((k) => k === 'models' || k.startsWith('models.')) + ) + return name; + } catch { + // Malformed env config is reported by config loading itself; not this watcher's job. + } + } + return undefined; +} + +function warnOnLiteralCredentials(kind: ModelKind, logicalName: string, entry: ModelEntry): void { + for (const field of CREDENTIAL_FIELDS) { + const value = (entry as Record)[field]; + if (typeof value !== 'string' || value.length === 0) continue; + if (isUnresolvedEnvVarPlaceholder(value)) continue; // operator is using ${VAR} indirection + harperLogger.warn( + `models.${kind}.${logicalName}: '${field}' is a literal value in harperdb-config.yaml; ` + + `prefer \${ENV_VAR} indirection for credentials to keep them off disk` + ); + } } /** diff --git a/resources/search.ts b/resources/search.ts index e4ec82d7b5..fb9d34dae2 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -1103,7 +1103,7 @@ export function filterByType(searchCondition, Table, context, filtered, isPrimar canUseIndex = canUseIndex && // is it a comparator that makes sense to use index !isPrimaryKey && // no need to use index for primary keys, since we will be iterating over the primary keys - Table?.indices[attribute] && // is there an index for this attribute + !!(Table && usableIndex(Table, attribute)) && // is there a usable (complete) index for this attribute estimatedIncomingCount > 3; // do we have a valid estimate of multiple incoming records (that is worth using an index for) if (canUseIndex) { if (searchCondition.estimated_count == undefined) estimateCondition(Table)(searchCondition); @@ -1169,7 +1169,7 @@ export function filterByType(searchCondition, Table, context, filtered, isPrimar function estimateRangeCondition(table, condition, searchType, fraction) { const attributeName = condition[0] ?? condition.attribute; const isPrimaryKey = attributeName === table.primaryKey; - const store = isPrimaryKey ? table.primaryStore : table.indices[attributeName]; + const store = isPrimaryKey ? table.primaryStore : usableIndex(table, attributeName); // LMDB-backed and custom index stores do not implement estimateCount if (typeof store?.estimateCount !== 'function') return undefined; let value = condition[1] ?? condition.value; @@ -1245,6 +1245,28 @@ function estimateRangeCondition(table, condition, searchType, fraction) { return Math.max(1, Math.round(confidence * count + (1 - confidence) * heuristic)); } +/** The index a condition can be driven by; searchByIndex refuses a rebuilding one, so it reads as absent. */ +function usableIndex(table, attributeName): any { + const index = attributeName == null ? undefined : table.indices[attributeName]; + return index?.isIndexing ? undefined : index; +} + +/** + * True when an index this attribute would have to be driven by is still being built, following a + * relationship path to the local join index and on to the related table's leaf index. + */ +function drivesRebuildingIndex(table, attributeName, relationshipOffset = 0): boolean { + if (!Array.isArray(attributeName)) + return attributeName != null && attributeName !== table.primaryKey && !!table.indices[attributeName]?.isIndexing; + if (relationshipOffset >= attributeName.length - 1) + return drivesRebuildingIndex(table, attributeName[relationshipOffset]); + const attribute = findAttribute(table.attributes, attributeName[relationshipOffset]); + if (!attribute) return false; + if (table.indices[attribute.relationship?.from]?.isIndexing) return true; + const relatedTable = attribute.definition?.tableClass || attribute.elements?.definition?.tableClass; + return !!relatedTable && drivesRebuildingIndex(relatedTable, attributeName, relationshipOffset + 1); +} + export function estimateCondition(table) { function estimateConditionForTable(condition) { if (condition.estimated_count === undefined) { @@ -1274,7 +1296,10 @@ export function estimateCondition(table) { // skip if it is cached let searchType = condition.comparator || condition.search_type; searchType = ALTERNATE_COMPARATOR_NAMES[searchType] || searchType; - if (condition.negated) { + // before comparator dispatch: several branches fall back to a finite table-fraction heuristic + if (drivesRebuildingIndex(table, condition[0] ?? condition.attribute)) { + condition.estimated_count = Infinity; + } else if (condition.negated) { // a negated condition always executes as a full scan (searchByIndex forces // needFullScan), so follow the filter-only convention used by contains/ends_with: // estimate Infinity so its positive-range estimate can never win the driving-condition @@ -1307,12 +1332,12 @@ export function estimateCondition(table) { } } else { // we only attempt to estimate count on equals operator because that's really all that LMDB supports (some other key-value stores like libmdbx could be considered if we need to do estimated counts of ranges at some point) - const index = table.indices[attribute_name]; + const index = usableIndex(table, attribute_name); condition.estimated_count = index ? index.getValuesCount(condition[1] ?? condition.value) : Infinity; } } else if (searchType === 'contains' || searchType === 'ends_with' || searchType === 'ne') { const attribute_name = condition[0] ?? condition.attribute; - const index = table.indices[attribute_name]; + const index = usableIndex(table, attribute_name); if (condition.value === null && searchType === 'ne') { condition.estimated_count = Math.max( estimatedEntryCount(table.primaryStore) - (index ? index.getValuesCount(null) : 0), @@ -1321,7 +1346,7 @@ export function estimateCondition(table) { } else condition.estimated_count = Infinity; } else if (searchType === 'in') { const attribute_name = condition[0] ?? condition.attribute; - const index = table.indices[attribute_name]; + const index = usableIndex(table, attribute_name); if (Array.isArray(condition.value) && index) { // Sum of per-value matches (over-counts duplicates but is a fine ceiling) let estimate = 0; @@ -1342,7 +1367,7 @@ export function estimateCondition(table) { BETWEEN_ESTIMATE * estimatedEntryCount(table.primaryStore) + 1; else if (searchType === 'sort') { const attribute_name = condition[0] ?? condition.attribute; - const index = table.indices[attribute_name]; + const index = usableIndex(table, attribute_name); if (index?.customIndex?.estimateCountAsSort) // allow custom index to define its own estimation of counts condition.estimated_count = index.customIndex.estimateCountAsSort(condition); @@ -1350,7 +1375,7 @@ export function estimateCondition(table) { } else { // for the search types that use the broadest range, try do them last const attribute_name = condition[0] ?? condition.attribute; - const index = table.indices[attribute_name]; + const index = usableIndex(table, attribute_name); if (index?.customIndex?.estimateCount) // allow custom index to define its own estimation of counts condition.estimated_count = index.customIndex.estimateCount(condition.value); diff --git a/security/jsLoader.ts b/security/jsLoader.ts index 32dc100dc0..271a14118e 100644 --- a/security/jsLoader.ts +++ b/security/jsLoader.ts @@ -1,9 +1,9 @@ import { Resource } from '../resources/Resource.ts'; import { contextStorage, transaction } from '../resources/transaction.ts'; import { RequestTarget } from '../resources/RequestTarget.ts'; -import { tables, databases } from '../resources/databases.ts'; +import { tables, databases, scopedTableFactory } from '../resources/databases.ts'; import { models as harperModelsSingleton } from '../resources/models/Models.ts'; -import { defineTable, types } from '../resources/defineTable.ts'; +import { defineTable, defineTableUsing, types } from '../resources/defineTable.ts'; import { defineResource, t, schemaOf, projectTableFragment } from '../resources/defineResource.ts'; import { readFile } from 'node:fs/promises'; import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path'; @@ -16,7 +16,7 @@ import { createRequire } from 'node:module'; import * as env from '../utility/environment/environmentManager'; import * as child_process from 'node:child_process'; import { CONFIG_PARAMS, DEFAULT_DATABASE_NAME } from '../utility/hdbTerms.ts'; -import { assertTableTargetNotBranched } from '../resources/branchGuard.ts'; + import { contentTypes } from '../server/serverHelpers/contentTypes.ts'; import type {} from 'ses'; import { @@ -893,15 +893,15 @@ function scopedDatabaseBindings(scope: ApplicationScope): { databases: any; tabl } /** - * `defineTable` registers into the process-wide catalog, so for a branched name it is refused rather - * than silently misdirected onto the base. See `assertTableTargetNotBranched`. + * `defineTable` for a branched application registers through that application's table factory, so + * a branched name lands in its branch; an unbranched application gets `defineTable` itself. */ function scopedDefineTable(scope: ApplicationScope): typeof defineTable { const branches = scope.branches; if (!branches?.size) return defineTable; - return function (name: string, shape: any, options: any = {}) { - assertTableTargetNotBranched(branches, options.database, name, 'defineTable'); - return defineTable(name, shape, options); + const declareTable = scopedTableFactory(branches); + return function (name: string, shape: any, options?: any) { + return defineTableUsing(declareTable, name, shape, options); } as typeof defineTable; } diff --git a/server/itc/serverHandlers.js b/server/itc/serverHandlers.js index 06c3eb9c0a..99bd486181 100644 --- a/server/itc/serverHandlers.js +++ b/server/itc/serverHandlers.js @@ -12,7 +12,7 @@ const harperBridge = require('../../dataLayer/harperBridge/harperBridge.ts'); const process = require('process'); const { isMainThread, threadId, workerData } = require('node:worker_threads'); -const { resetDatabases, closeDatabase } = require('../../resources/databases.ts'); +const { resetDatabases, closeDatabase, reloadBranchAt } = require('../../resources/databases.ts'); /** * This object/functions are passed to the ITC client instance and dynamically added as event handlers. @@ -76,6 +76,13 @@ schemaHandler.addListener = function (listener) { */ async function syncSchemaMetadata(msg) { try { + // A change to a scope-private branch is not a change to any database in the global map, so the + // rescan below has nothing to find; a thread holding that branch open reloads it instead. + if (msg.branchPath) { + // No write barrier here: the symbol-keyed put below has never been one (harper#2522). + reloadBranchAt(msg.branchPath); + return; + } // TODO: Eventually should indicate which database/table changed so we don't have to scan everything let databases = resetDatabases(); if (msg.table && msg.database) diff --git a/server/threads/itc.js b/server/threads/itc.js index 992237f4e4..1300e87bbe 100644 --- a/server/threads/itc.js +++ b/server/threads/itc.js @@ -72,14 +72,24 @@ function validateEvent(event) { * @param schema * @param table * @param attribute + * @param branchPath the branch directory when the change is to a scope-private branch of `schema` + * rather than to the database itself (resources/databases.ts `reloadBranchAt`) * @constructor */ -function SchemaEventMsg(originator, operation, schema, table = undefined, attribute = undefined) { +function SchemaEventMsg( + originator, + operation, + schema, + table = undefined, + attribute = undefined, + branchPath = undefined +) { this.originator = originator; this.operation = operation; this.schema = schema; this.table = table; this.attribute = attribute; + if (branchPath) this.branchPath = branchPath; } /** diff --git a/server/threads/manageThreads.js b/server/threads/manageThreads.js index f6cf60f7c8..24044b6aa0 100644 --- a/server/threads/manageThreads.js +++ b/server/threads/manageThreads.js @@ -173,6 +173,11 @@ module.exports = { isThreadRunning, waitUntilConfirmedGone, restartNumber: workerData?.restartNumber || 1, + // Identifies this process incarnation, where the PID cannot: a container reuses PID 1. Minted once + // on the main thread and carried to workers, so live siblings agree on it — one derived per thread + // would not. `undefined` on a worker started without it; consumers must fall back, not treat that + // as a mismatch. + processIncarnation: workerData ? workerData.processIncarnation : randomBytes(8).toString('hex'), }; connectedPorts.onMessageByType = onMessageByType; @@ -239,6 +244,7 @@ const RESERVED_WORKER_DATA_KEYS = [ 'workerCount', 'name', 'restartNumber', + 'processIncarnation', 'ticketKeys', 'noServerStart', '__proto__', // never a legitimate payload name; spread would define it as an own property @@ -433,6 +439,7 @@ function startWorker(path, options = {}) { workerCount: (workerCount = options.threadCount), name: options.name, restartNumber: module.exports.restartNumber, + processIncarnation: module.exports.processIncarnation, ticketKeys: getTicketKeys(), }, transferList: portsToSend, diff --git a/sqlEngine/PLAN.md b/sqlEngine/PLAN.md index a9a2c88c68..9ba94c9d73 100644 --- a/sqlEngine/PLAN.md +++ b/sqlEngine/PLAN.md @@ -119,9 +119,9 @@ type LogicalPlan = | `PhysicalProject` | per-row evaluation of projection list. | | `PhysicalRelationshipJoin` | single `Table.search` using the array-attribute relationship syntax (`core/resources/search.ts:138-196`) — fastest join path when the join is on a declared `relationship` attribute. | | `PhysicalIndexNestedLoopJoin` | outer side streams via index scan; per outer row, probe inner side with `Table.search({ conditions: [{ attribute: innerKey, value: outerRow[outerKey], comparator: 'equals' }] })`. LEFT OUTER fills nulls when probe returns 0 rows. | -| `PhysicalHashJoin` | build smaller side into `Map`; probe the other; cap build size by `sql.engine.maxHashRows`. | +| `PhysicalHashJoin` | build smaller side into `Map`; probe the other; cap build size by `sql.maxHashRows`. | | `PhysicalNestedLoopJoin` | only for cross / non-equi join. Last resort. | -| `PhysicalSort` | in-memory sort with `sql.engine.maxSortRows` cap. | +| `PhysicalSort` | in-memory sort with `sql.maxSortRows` cap. | | `PhysicalLimit` | counts rows; calls `child.return()` when limit is reached. | | `PhysicalStreamingAggregate` | requires input sorted on group keys (set by `outputOrder` properties on physical operators); O(1) memory per group. | | `PhysicalHashAggregate` | `Map`; capped by `maxHashRows`. | @@ -217,7 +217,7 @@ core/sqlEngine/ **Modified files (small edits):** - `core/sqlTranslator/index.js`: in `processAST` (line 130), after the switch resolves `sqlFunction`, route through `core/sqlEngine/router.ts` instead of calling `search`/`convertInsert`/`cbUpdateUpdate`/`deleteTranslator` directly. -- `config/...`: add the `sql.engine`, `sql.engine.allowFullScan`, `sql.engine.maxSortRows`, `sql.engine.maxHashRows` settings. +- `config/...`: add the `sql.engine`, `sql.allowFullScan`, `sql.maxSortRows`, `sql.maxHashRows` settings (siblings under `sql`, not nested under `engine` — matches `SqlEngineConfig` in `config.ts`). - No changes to `core/dataLayer/SQLSearch.js`, `SelectValidator.js`, `sql_statement_bucket.js` until the final cutover phase. ## Custom function porting diff --git a/sqlEngine/config.ts b/sqlEngine/config.ts index b7e87cf5c3..fd7b349c5a 100644 --- a/sqlEngine/config.ts +++ b/sqlEngine/config.ts @@ -9,9 +9,7 @@ * EngineUnsupportedError (default). * * The flag is read from the HARPER_SQL_ENGINE environment variable, then - * harperConfig.sql?.engine, then defaults to 'auto'. We deliberately keep - * this resolution lazy and lightweight so the router can be invoked without a - * fully booted Harper config (e.g., in unit tests). + * sql.engine in Harper config, then defaults to 'auto'. * * Phase 5 cutover: the default is now 'auto' — the new engine handles every SQL * request it can plan and silently falls back to legacy on anything it can't, so @@ -26,6 +24,9 @@ * flipping to 'new' and deleting the legacy path. See PLAN.md phase-5 notes. */ +import { getConfigValue } from '../config/configUtils.ts'; +import { CONFIG_PARAMS } from '../utility/hdbTerms.ts'; + export type SqlEngineMode = 'legacy' | 'new' | 'auto'; export interface SqlEngineConfig { @@ -48,22 +49,26 @@ function envEngine(): SqlEngineMode | undefined { return undefined; } -function harperConfigEngine(): Partial { - try { - const harperConfig = (globalThis as { harperConfig?: { sql?: Partial } }).harperConfig; - return harperConfig?.sql ?? {}; - } catch { - return {}; - } +function configEngine(): SqlEngineMode | undefined { + const v = getConfigValue(CONFIG_PARAMS.SQL_ENGINE); + if (v === 'legacy' || v === 'new' || v === 'auto') return v; + return undefined; } export function getSqlEngineConfig(): SqlEngineConfig { - const fromConfig = harperConfigEngine(); const fromEnv = envEngine(); + const fromConfig = configEngine(); + const allowFullScan = getConfigValue(CONFIG_PARAMS.SQL_ALLOWFULLSCAN); + const maxSortRows = getConfigValue(CONFIG_PARAMS.SQL_MAXSORTROWS); + const maxHashRows = getConfigValue(CONFIG_PARAMS.SQL_MAXHASHROWS); return { - engine: fromEnv ?? fromConfig.engine ?? DEFAULTS.engine, - allowFullScan: fromConfig.allowFullScan ?? DEFAULTS.allowFullScan, - maxSortRows: fromConfig.maxSortRows ?? DEFAULTS.maxSortRows, - maxHashRows: fromConfig.maxHashRows ?? DEFAULTS.maxHashRows, + engine: fromEnv ?? fromConfig ?? DEFAULTS.engine, + allowFullScan: typeof allowFullScan === 'boolean' ? allowFullScan : DEFAULTS.allowFullScan, + maxSortRows: isPositiveInteger(maxSortRows) ? maxSortRows : DEFAULTS.maxSortRows, + maxHashRows: isPositiveInteger(maxHashRows) ? maxHashRows : DEFAULTS.maxHashRows, }; } + +function isPositiveInteger(value: unknown): value is number { + return typeof value === 'number' && Number.isInteger(value) && value >= 1; +} diff --git a/unitTests/bin/deploySetup.test.js b/unitTests/bin/deploySetup.test.js index c24a68bbc4..570ad90e26 100644 --- a/unitTests/bin/deploySetup.test.js +++ b/unitTests/bin/deploySetup.test.js @@ -7,11 +7,31 @@ // resolves its inputs. const assert = require('node:assert'); const path = require('node:path'); -const { resolveComponentName, resolveGitHost, storeSealedSecret } = require('#src/bin/deploySetup'); +const { + assertUsableComponentName, + resolveComponentName, + resolveGitHost, + storeSealedSecret, +} = require('#src/bin/deploySetup'); const { directoryProjectName } = require('#src/utility/componentNames'); const cliOperationsModule = require('#src/bin/cliOperations'); describe('deploySetup', () => { + describe('assertUsableComponentName', () => { + it('accepts a name a deploy would accept', () => { + assert.doesNotThrow(() => assertUsableComponentName('sql-tools')); + }); + + it('rejects a name outside the deploy grammar', () => { + assert.throws(() => assertUsableComponentName('has space'), /not a usable component name/); + }); + + // Sealing a credential for a name deploy_component refuses leaves a secret nobody can use. + it('rejects a name reserved by the root config', () => { + assert.throws(() => assertUsableComponentName('sql'), /is reserved for Harper's "sql" configuration section/); + }); + }); + describe('resolveComponentName', () => { it('canonicalizes an explicit project like deploy_component does', () => { assert.strictEqual(resolveComponentName({ project: 'web' }), 'web'); diff --git a/unitTests/build-tools/checkShrinkwrapPins.test.mjs b/unitTests/build-tools/checkShrinkwrapPins.test.mjs index e8298280a0..e88b5c2c80 100644 --- a/unitTests/build-tools/checkShrinkwrapPins.test.mjs +++ b/unitTests/build-tools/checkShrinkwrapPins.test.mjs @@ -1,6 +1,6 @@ import assert from 'node:assert'; import { spawnSync } from 'node:child_process'; -import { chmod, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { chmod, cp, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { dirname, join, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -10,13 +10,26 @@ const script = join(root, 'build-tools/check-shrinkwrap-pins.mjs'); const dependencies = ['@harperfast/rocksdb-js', 'fastify', '@aws-sdk/client-s3']; const alignedDependencies = { '@harperfast/extended-iterable': '1.0.3', - 'msgpackr': '2.0.5', + 'msgpackr': '2.0.6', +}; +const rocksdbDependencyRanges = { + '@harperfast/extended-iterable': '^1.0.3', + 'msgpackr': '^2.0.6', }; describe('shrinkwrap pin canaries', function () { it('keeps the checked canaries present and ranged in the real manifest', async function () { const manifest = JSON.parse(await readFile(join(root, 'package.json'), 'utf8')); - const fixture = await createFixture(manifest.dependencies); + const rocksdbManifestPath = join(root, 'node_modules/@harperfast/rocksdb-js/package.json'); + let rocksdbManifest; + try { + rocksdbManifest = JSON.parse(await readFile(rocksdbManifestPath, 'utf8')); + } catch (error) { + assert.fail( + `the installed rocksdb-js manifest is required at ${rocksdbManifestPath}: ${error?.message ?? error}` + ); + } + const fixture = await createFixture(manifest.dependencies, {}, false, '', 3, {}, rocksdbManifest.dependencies); try { const result = runCheck(fixture); assert.strictEqual(result.status, 0, result.stderr); @@ -25,7 +38,51 @@ describe('shrinkwrap pin canaries', function () { } }); - it('fails when a root encoder pin diverges from rocksdb-js', async function () { + it('fails when a root encoder pin is outside the rocksdb-js range', async function () { + const fixture = await createFixture({ + '@harperfast/rocksdb-js': '2.7.1', + 'fastify': '^5.8.2', + '@aws-sdk/client-s3': '^3.1012.0', + }); + try { + await writeFile( + join(fixture.packageRoot, 'node_modules/@harperfast/rocksdb-js/package.json'), + JSON.stringify({ + version: '1.0.0', + dependencies: { ...rocksdbDependencyRanges, msgpackr: '^3.0.0' }, + }) + ); + const result = runCheck(fixture); + assert.strictEqual(result.status, 1); + assert.match(result.stderr, /root msgpackr pin 2\.0\.6 is outside rocksdb-js \^3\.0\.0/); + } finally { + await fixture.cleanup(); + } + }); + + it('fails when rocksdb-js no longer declares a guarded dependency', async function () { + const fixture = await createFixture({ + '@harperfast/rocksdb-js': '2.7.1', + 'fastify': '^5.8.2', + '@aws-sdk/client-s3': '^3.1012.0', + }); + try { + await writeFile( + join(fixture.packageRoot, 'node_modules/@harperfast/rocksdb-js/package.json'), + JSON.stringify({ + version: '1.0.0', + dependencies: { '@harperfast/extended-iterable': '^1.0.3' }, + }) + ); + const result = runCheck(fixture); + assert.strictEqual(result.status, 1); + assert.match(result.stderr, /rocksdb-js no longer declares msgpackr/); + } finally { + await fixture.cleanup(); + } + }); + + it('fails when rocksdb-js uses a non-semver dependency spec', async function () { const fixture = await createFixture({ '@harperfast/rocksdb-js': '2.7.1', 'fastify': '^5.8.2', @@ -36,12 +93,12 @@ describe('shrinkwrap pin canaries', function () { join(fixture.packageRoot, 'node_modules/@harperfast/rocksdb-js/package.json'), JSON.stringify({ version: '1.0.0', - dependencies: { ...alignedDependencies, msgpackr: '2.0.6' }, + dependencies: { ...rocksdbDependencyRanges, msgpackr: 'workspace:*' }, }) ); const result = runCheck(fixture); assert.strictEqual(result.status, 1); - assert.match(result.stderr, /root msgpackr pin 2\.0\.5 does not match rocksdb-js 2\.0\.6/); + assert.match(result.stderr, /rocksdb-js declares msgpackr with unsupported range workspace:\*/); } finally { await fixture.cleanup(); } @@ -52,30 +109,90 @@ describe('shrinkwrap pin canaries', function () { '@harperfast/rocksdb-js': '2.7.1', 'fastify': '^5.8.2', '@aws-sdk/client-s3': '^3.1012.0', - 'msgpackr': '^2.0.5', + 'msgpackr': '^2.0.6', + }); + try { + const result = runCheck(fixture); + assert.strictEqual(result.status, 1); + assert.match(result.stderr, /root msgpackr spec must be exact, received \^2\.0\.6/); + } finally { + await fixture.cleanup(); + } + }); + + for (const [dependency, version] of Object.entries(alignedDependencies)) { + it(`fails when rocksdb-js installs a nested ${dependency} instance`, async function () { + const fixture = await createFixture({ + '@harperfast/rocksdb-js': '2.7.1', + 'fastify': '^5.8.2', + '@aws-sdk/client-s3': '^3.1012.0', + }); + try { + const nestedDir = join(fixture.packageRoot, 'node_modules/@harperfast/rocksdb-js/node_modules', dependency); + await mkdir(nestedDir, { recursive: true }); + await writeFile(join(nestedDir, 'package.json'), JSON.stringify({ version, main: 'index.js' })); + await writeFile(join(nestedDir, 'index.js'), ''); + const result = runCheck(fixture); + assert.strictEqual(result.status, 1); + assert(result.stderr.includes(`rocksdb-js loaded a nested ${dependency}@${version}`), result.stderr); + } finally { + await fixture.cleanup(); + } + }); + } + + it('follows a linked rocksdb-js package to detect its private dependency instance', async function () { + const fixture = await createFixture({ + '@harperfast/rocksdb-js': '2.7.1', + 'fastify': '^5.8.2', + '@aws-sdk/client-s3': '^3.1012.0', }); try { + const rocksdbDir = join(fixture.packageRoot, 'node_modules/@harperfast/rocksdb-js'); + const linkedDir = join(dirname(fixture.packageRoot), 'linked-rocksdb-js'); + const linkedMsgpackrDir = join(linkedDir, 'node_modules/msgpackr'); + await mkdir(linkedMsgpackrDir, { recursive: true }); + await writeFile( + join(linkedDir, 'package.json'), + JSON.stringify({ version: '1.0.0', dependencies: rocksdbDependencyRanges }) + ); + await writeFile(join(linkedMsgpackrDir, 'package.json'), JSON.stringify({ version: '2.0.6', main: 'index.js' })); + await writeFile(join(linkedMsgpackrDir, 'index.js'), ''); + await rm(rocksdbDir, { recursive: true, force: true }); + await symlink(linkedDir, rocksdbDir, 'junction'); + const result = runCheck(fixture); assert.strictEqual(result.status, 1); - assert.match(result.stderr, /root msgpackr spec must be exact, received \^2\.0\.5/); + assert.match(result.stderr, /rocksdb-js loaded a nested msgpackr@2\.0\.6/); } finally { await fixture.cleanup(); } }); - it('fails when rocksdb-js installs a nested encoder instance', async function () { + it('reports divergent resolutions outside the classic nested layout', async function () { const fixture = await createFixture({ '@harperfast/rocksdb-js': '2.7.1', 'fastify': '^5.8.2', '@aws-sdk/client-s3': '^3.1012.0', }); try { - const nestedDir = join(fixture.packageRoot, 'node_modules/@harperfast/rocksdb-js/node_modules/msgpackr'); - await mkdir(nestedDir, { recursive: true }); - await writeFile(join(nestedDir, 'package.json'), JSON.stringify({ version: '2.0.5' })); + const rocksdbDir = join(fixture.packageRoot, 'node_modules/@harperfast/rocksdb-js'); + const linkedDir = join(dirname(fixture.packageRoot), 'linked-rocksdb-js'); + const siblingMsgpackrDir = join(dirname(linkedDir), 'node_modules/msgpackr'); + await mkdir(linkedDir, { recursive: true }); + await mkdir(siblingMsgpackrDir, { recursive: true }); + await writeFile( + join(linkedDir, 'package.json'), + JSON.stringify({ version: '1.0.0', dependencies: rocksdbDependencyRanges }) + ); + await writeFile(join(siblingMsgpackrDir, 'package.json'), JSON.stringify({ version: '2.0.6', main: 'index.js' })); + await writeFile(join(siblingMsgpackrDir, 'index.js'), ''); + await rm(rocksdbDir, { recursive: true, force: true }); + await symlink(linkedDir, rocksdbDir, 'junction'); + const result = runCheck(fixture); assert.strictEqual(result.status, 1); - assert.match(result.stderr, /rocksdb-js loaded a nested msgpackr@2\.0\.5/); + assert.match(result.stderr, /rocksdb-js resolves msgpackr from .* but the root resolves it from /); } finally { await fixture.cleanup(); } @@ -310,7 +427,8 @@ async function createFixture( allRangeVersionsCurrent = false, failedRange = '', failedAttempts = 3, - registryResponses = {} + registryResponses = {}, + rocksdbDependencies = rocksdbDependencyRanges ) { const tempDir = await mkdtemp(join(tmpdir(), 'harper-shrinkwrap-canary-')); const packageRoot = join(tempDir, 'package'); @@ -318,6 +436,7 @@ async function createFixture( const queryLog = join(tempDir, 'queries.log'); await mkdir(packageRoot, { recursive: true }); await mkdir(binDir, { recursive: true }); + await cp(join(root, 'node_modules/semver'), join(packageRoot, 'node_modules/semver'), { recursive: true }); await writeFile(queryLog, ''); const packageDependencies = { ...alignedDependencies, ...manifestDependencies }; await writeFile(join(packageRoot, 'package.json'), JSON.stringify({ dependencies: packageDependencies })); @@ -339,11 +458,12 @@ async function createFixture( (dependency in alignedDependencies ? packageDependencies[dependency] : '1.0.0'), }; if (dependency === '@harperfast/rocksdb-js') { - dependencyManifest.dependencies = Object.fromEntries( - Object.keys(alignedDependencies).map((dep) => [dep, packageDependencies[dep]]) - ); + dependencyManifest.dependencies = rocksdbDependencies; + } else if (dependency in alignedDependencies) { + dependencyManifest.main = 'index.js'; } await writeFile(join(dependencyDir, 'package.json'), JSON.stringify(dependencyManifest)); + if (dependency in alignedDependencies) await writeFile(join(dependencyDir, 'index.js'), ''); } await writeFile( join(binDir, 'npm'), diff --git a/unitTests/components/reservedComponentName.test.js b/unitTests/components/reservedComponentName.test.js new file mode 100644 index 0000000000..c4cba48704 --- /dev/null +++ b/unitTests/components/reservedComponentName.test.js @@ -0,0 +1,94 @@ +'use strict'; + +const assert = require('node:assert'); +const fs = require('fs-extra'); +const path = require('node:path'); +const os = require('node:os'); +const env = require('#src/utility/environment/environmentManager'); +const { CONFIG_PARAMS } = require('#src/utility/hdbTerms'); +const validator = require('#js/components/operationsValidation'); +const operations = require('#js/components/operations'); +const { isReservedComponentName } = require('#src/utility/componentNames'); + +// The validators return undefined when valid and an Error when invalid. +describe('reserved component names', () => { + let ROOT; + + before(() => { + env.initTestEnvironment(); + ROOT = path.join(os.tmpdir(), `harper-reserved-names-${process.pid}`); + env.setProperty(CONFIG_PARAMS.COMPONENTSROOT, ROOT); + fs.ensureDirSync(ROOT); + }); + + after(() => { + fs.removeSync(ROOT); + }); + + it("reserves 'sql', the root config key that configures the SQL engine", () => { + assert.ok(isReservedComponentName('sql')); + assert.ok(!isReservedComponentName('sql-tools')); + assert.ok(!isReservedComponentName('mysql')); + // The reserved thing is the YAML root key, which is case-sensitive: `SQL:` is a different + // entry and collides with nothing. + assert.ok(!isReservedComponentName('SQL')); + }); + + it('add_component refuses to create a project under a reserved name', () => { + const error = validator.addComponentValidator({ project: 'sql' }); + assert.ok(error); + assert.match(error.message, /Component name 'sql' is reserved/); + }); + + it('add_component does not refuse a name that merely contains a reserved one', () => { + const error = validator.addComponentValidator({ project: 'sql-tools' }); + // May still be rejected for already existing; only the reservation is under test. + if (error) assert.doesNotMatch(error.message, /is reserved/); + }); + + describe('the file writers, which create the project directory by writing into it', () => { + const setComponentFile = { project: 'sql', file: 'resources.js', payload: '' }; + const setEnvValue = { project: 'sql', key: 'A', value: 'b' }; + + afterEach(() => { + fs.removeSync(path.join(ROOT, 'sql')); + }); + + it('refuse to bring a reserved-name project into existence', () => { + for (const error of [ + validator.setComponentFileValidator(setComponentFile), + validator.setEnvValueValidator(setEnvValue), + ]) { + assert.ok(error); + assert.match(error.message, /Component name 'sql' is reserved/); + } + }); + + it('still write to an application that already holds the name, so it can be migrated', () => { + fs.ensureDirSync(path.join(ROOT, 'sql')); + assert.strictEqual(validator.setComponentFileValidator(setComponentFile), undefined); + assert.strictEqual(validator.setEnvValueValidator(setEnvValue), undefined); + }); + }); + + // The handler derives `project` (canonicalized, or from `package`) before it validates, so each + // of these is a distinct way to reach the reserved name — all refused before the deploy writes + // config, ingests credentials, stages a payload, or records a deployment. + const deployRequests = [ + ['an explicit project name', { project: 'sql', package: '@org/sql-app' }], + ['a canonicalized project name', { project: 'sql.tgz', package: '@org/sql-app' }], + ['a project name derived from the package', { package: 'sql' }], + ['a payload deploy, which writes no root config entry', { project: 'sql', payload: 'ZmFrZQ==' }], + ['force, which cannot buy a reserved name', { project: 'sql', package: '@org/sql-app', force: true }], + ]; + + for (const [description, request] of deployRequests) { + it(`deploy_component rejects ${description}`, async () => { + await assert.rejects(operations.deployComponent(request), (error) => { + assert.match(error.message, /Component name 'sql' is reserved/); + assert.strictEqual(error.statusCode, 400); + return true; + }); + }); + } +}); diff --git a/unitTests/config/setConfigurationSql.test.js b/unitTests/config/setConfigurationSql.test.js new file mode 100644 index 0000000000..31d2d2bc7b --- /dev/null +++ b/unitTests/config/setConfigurationSql.test.js @@ -0,0 +1,43 @@ +'use strict'; + +const assert = require('node:assert'); +const { CONFIG_PARAMS, CONFIG_PARAM_MAP } = require('#src/utility/hdbTerms'); +const { setConfiguration } = require('#src/config/configUtils'); + +describe('sql.* config param registration', () => { + const SQL_PARAMS = { + SQL_ENGINE: 'sql_engine', + SQL_ALLOWFULLSCAN: 'sql_allowFullScan', + SQL_MAXSORTROWS: 'sql_maxSortRows', + SQL_MAXHASHROWS: 'sql_maxHashRows', + }; + + for (const [key, expected] of Object.entries(SQL_PARAMS)) { + it(`${key} is registered in CONFIG_PARAMS so getConfigValue can resolve it`, () => { + assert.strictEqual(CONFIG_PARAMS[key], expected); + }); + + it(`${key} is reachable through CONFIG_PARAM_MAP, so set_configuration accepts it`, () => { + assert.strictEqual(CONFIG_PARAM_MAP[expected.toLowerCase()], expected); + }); + } + + // Rejected before any file read, so these run without a harper-config.yaml fixture on disk. + it('set_configuration still rejects a key outside the four registered sql.* settings', async () => { + await assert.rejects( + setConfiguration({ operation: 'set_configuration', sql_bogus: true }), + /Unable to update config, unrecognized config parameter/ + ); + }); + + // The `_package` / `_port` escape would otherwise put back the very entry + // deploy_component now refuses to create. + for (const param of ['sql_package', 'sql_port']) { + it(`set_configuration rejects ${param}, which would write a component entry over the sql section`, async () => { + await assert.rejects( + setConfiguration({ operation: 'set_configuration', [param]: 'x' }), + /cannot configure a component whose name is reserved/ + ); + }); + } +}); diff --git a/unitTests/resources/branchDatabase.test.js b/unitTests/resources/branchDatabase.test.js index c939bc7ed1..ebd5308d13 100644 --- a/unitTests/resources/branchDatabase.test.js +++ b/unitTests/resources/branchDatabase.test.js @@ -546,45 +546,421 @@ describeUnlessLmdb('branch rollback is scoped to the failing application (harper }); }); -describe('declaring a table from a branched application (harper#643)', () => { - const { assertTableTargetNotBranched } = require('#src/resources/branchGuard'); - - it('refuses every declaration path that would land in the base', function () { - // GraphQL @table, scope.ensureTable and defineTable all funnel into the process-wide table(), - // so gating only one of them leaves the base reachable through the other two -- and @table is - // the path most applications actually use. - const branches = new Map([['gatedbase', {}]]); - for (const how of ['a GraphQL @table directive', 'ensureTable', 'defineTable']) { - assert.throws(() => assertTableTargetNotBranched(branches, 'gatedbase', 'T', how), /branched database/); - } +describeUnlessLmdb('declaring a table from a branched application (harper#2264)', () => { + const { scopedTableFactory, reloadBranchAt, databaseEventsEmitter } = require('#src/resources/databases'); + const { closeBranchAt, prepareBranches } = require('#src/resources/branchDatabase'); + const APP = 'declApp'; + const BASE = 'declbase'; + const id = { name: 'id', type: 'ID', isPrimaryKey: true }; + // every `updateTable` a global subscriber (replication, analytics) would have seen + let announced; + const subscriber = (Table) => announced.push(Table); + + before(async function () { + this.timeout(30000); + setupTestDBPath(); + setMainIsWorker(true); + const Existing = table({ + table: 'Existing', + database: BASE, + schemaDefined: true, + attributes: [id, { name: 'note' }], + }); + await Existing.put({ id: 'base-row', note: 'from the base' }); + // the default database has to exist to be branched + table({ table: 'DefaultSource', database: 'data', attributes: [id] }); + databaseEventsEmitter.on('updateTable', subscriber); }); - it('defaults every falsy database name the way table() does', function () { - // `table()` resolves any falsy name to the default database (`if (!databaseName)`), so a guard - // that only defaulted nullish names would let `database: ''` past the fence and then land in - // the base as `data`. - for (const falsy of ['', null, undefined]) { - assert.throws( - () => assertTableTargetNotBranched(new Map([['data', {}]]), falsy, 'T', 'defineTable'), - /branched database 'data'/, - `${JSON.stringify(falsy)} must resolve to the default database` + after(function () { + databaseEventsEmitter.off('updateTable', subscriber); + }); + + beforeEach(function () { + announced = []; + }); + + afterEach(async function () { + await removeBranches(); + }); + + async function branchedFactory() { + const branches = await prepareBranches(APP, [BASE], 'vm-current-context'); + const branch = branches.get(BASE); + return { branch, branches, declare: scopedTableFactory(branches) }; + } + + function assertNothingAnnouncedFrom(branch) { + for (const Table of announced) { + assert.notStrictEqual( + Table, + branch.tables[Table.tableName], + `a branch class (${Table.tableName}) must never reach a global updateTable subscriber` ); } + } + + it('hands an unbranched application the global factory itself', function () { + // This is the path every application that does not branch takes: it has to be the same + // function, not an equivalent one, so nothing on it changes. + assert.strictEqual(scopedTableFactory(undefined), table); + assert.strictEqual(scopedTableFactory(new Map()), table); }); - it('defaults an unnamed database to the default one', function () { - // `@table` and defineTable both omit `database` to mean `data`, so a branch of `data` has to - // catch the omitted case or the most common declaration of all slips through. - assert.throws( - () => assertTableTargetNotBranched(new Map([['data', {}]]), undefined, 'T', 'defineTable'), - /branched database 'data'/ + it('declares into the branch and leaves the base without the table', async function () { + const { branch, declare } = await branchedFactory(); + + const Declared = declare({ + table: 'Declared', + database: BASE, + schemaDefined: true, + attributes: [id, { name: 'note' }], + }); + + assert.strictEqual(branch.tables.Declared, Declared, 'published into the branch graph'); + assert.strictEqual(databases[BASE].Declared, undefined, 'and never into the base'); + assert.strictEqual(Declared.databaseName, BASE, 'carrying the logical name the application uses'); + assert.strictEqual( + branch.rootStore.databaseName, + `${APP.length}_${APP}__${BASE}`, + "the store keeps its own identity, which the branch's blob roots resolve from" + ); + await Declared.put({ id: 'a', note: 'in the branch' }); + assert.strictEqual((await Declared.get('a'))?.note, 'in the branch'); + // a class the factory built is a branch class: the base-directed statics stay refused + await assert.rejects(() => Declared.dropTable(), /branched database/); + assertNothingAnnouncedFrom(branch); + }); + + it('defaults an unnamed database the way table() does', async function () { + // `@table` and defineTable omit `database` to mean the default database, so a branch of it has + // to catch the omitted -- and the empty-string -- case or the most common declaration slips through + const branches = await prepareBranches(APP, ['data'], 'vm-current-context'); + const branch = branches.get('data'); + const declare = scopedTableFactory(branches); + const Unnamed = declare({ table: 'Unnamed', attributes: [id] }); + const Empty = declare({ table: 'Empty', database: '', attributes: [id] }); + assert.strictEqual(branch.tables.Unnamed, Unnamed); + assert.strictEqual(branch.tables.Empty, Empty); + assert.strictEqual(databases.data.Unnamed, undefined); + assert.strictEqual(databases.data.Empty, undefined); + }); + + it('routes a database the application did not branch to the global factory, same class and all', async function () { + const { declare } = await branchedFactory(); + const definition = { table: 'Elsewhere', database: 'declother', attributes: [id] }; + + const Elsewhere = declare(definition); + + assert.strictEqual(Elsewhere, databases.declother.Elsewhere); + assert.strictEqual(Elsewhere, table(definition), 'the identical object the global factory returns'); + }); + + it('re-declares a table the base already has in the branch only', async function () { + const { branch, declare } = await branchedFactory(); + const baseClass = databases[BASE].Existing; + const baseAttributes = baseClass.attributes.map((attribute) => attribute.name); + + const Evolved = declare({ + table: 'Existing', + database: BASE, + schemaDefined: true, + attributes: [id, { name: 'note' }, { name: 'added', type: 'String', indexed: true }], + }); + + assert.strictEqual(Evolved, branch.tables.Existing, 'the alter lands on the branch class'); + assert.notStrictEqual(Evolved, baseClass); + assert.ok(Evolved.attributes.some((attribute) => attribute.name === 'added')); + assert.ok(Evolved.indices.added, 'with its index opened on the branch store'); + assert.deepStrictEqual( + databases[BASE].Existing.attributes.map((attribute) => attribute.name), + baseAttributes, + 'the base schema is untouched' + ); + assert.strictEqual(databases[BASE].Existing, baseClass); + assert.strictEqual((await baseClass.get('base-row'))?.note, 'from the base'); + assert.strictEqual( + (await Evolved.get('base-row'))?.note, + 'from the base', + 'the branch still serves the checkpoint' + ); + assertNothingAnnouncedFrom(branch); + }); + + it('persists a declared table in the branch: it is there again after close and reopen', async function () { + this.timeout(30000); + const { branch, declare } = await branchedFactory(); + const Persisted = declare({ + table: 'Persisted', + database: BASE, + schemaDefined: true, + attributes: [id, { name: 'note' }], + }); + await Persisted.put({ id: 'kept', note: 'survives the reopen' }); + + await closeBranchAt(branch.path); + const reopened = await getOrCreateBranch(BASE, APP); + + assert.notStrictEqual(reopened, branch, 'sanity: a fresh handle over the same directory'); + assert.ok(reopened.tables.Persisted, 'the catalog in the branch store is the only record needed'); + assert.notStrictEqual(reopened.tables.Persisted, Persisted); + assert.strictEqual((await reopened.tables.Persisted.get('kept'))?.note, 'survives the reopen'); + assert.strictEqual(databases[BASE].Persisted, undefined); + }); + + it('hydrates a relationship between two branch-declared tables to the branch after reopen', async function () { + this.timeout(30000); + const { branch, declare } = await branchedFactory(); + const Target = declare({ + table: 'RelTarget2', + database: BASE, + schemaDefined: true, + attributes: [id, { name: 'label' }], + }); + declare({ + table: 'RelHost2', + database: BASE, + schemaDefined: true, + schemaRelationshipsDefined: true, + attributes: [ + id, + { name: 'targetId', type: 'ID', indexed: {} }, + { + name: 'target', + type: 'RelTarget2', + relationship: { from: 'targetId' }, + relationshipReference: { database: BASE, table: 'RelTarget2' }, + definition: { tableClass: Target }, + }, + ], + }); + assertNothingAnnouncedFrom(branch); + + await closeBranchAt(branch.path); + const reopened = (await prepareBranches(APP, [BASE], 'vm-current-context')).get(BASE); + + const attribute = reopened.tables.RelHost2.attributes.find((a) => a.name === 'target'); + assert.ok(attribute, 'the persisted relationship comes back with the table'); + const targetClass = (attribute.definition || attribute.elements?.definition)?.tableClass; + assert.strictEqual(targetClass, reopened.tables.RelTarget2, "resolved within the branch's own graph"); + assertNothingAnnouncedFrom(reopened); + }); + + /** What another thread's create leaves in the shared catalog: the rows, with no class in THIS thread. */ + function createElsewhere(branch, tableName) { + const catalog = branch.rootStore.dbisDb; + const tableId = catalog.getSync(Symbol.for('next-table-id')) ?? 1; + catalog.putSync(Symbol.for('next-table-id'), tableId + 1); + catalog.putSync(`${tableName}/note`, { name: 'note', attribute: 'note' }); + catalog.putSync(`${tableName}/`, { name: 'id', type: 'ID', isPrimaryKey: true, tableId, schemaDefined: true }); + assert.strictEqual(branch.tables[tableName], undefined, 'sanity: this thread has no class yet'); + } + + it('loads a table another thread created into the branch instead of re-creating it, or creating it in the base', async function () { + const { branch, declare } = await branchedFactory(); + createElsewhere(branch, 'Raced'); + + const Raced = declare({ table: 'Raced', database: BASE, schemaDefined: true, attributes: [id, { name: 'note' }] }); + + assert.strictEqual(Raced, branch.tables.Raced, 'the class the reload built is what the declaration returns'); + assert.strictEqual(databases[BASE].Raced, undefined, 'the reload after a lost race must not be the global rescan'); + await Raced.put({ id: 'r', note: 'usable' }); + assert.strictEqual((await Raced.get('r'))?.note, 'usable'); + }); + + it('reloads the branch on a schema-change signal that names it, where the branch is open', async function () { + const { branch, declare } = await branchedFactory(); + const { schema: schemaHandler } = require('#js/server/itc/serverHandlers'); + const ITCEventObject = require('#js/server/itc/utility/ITCEventObject'); + const { SchemaEventMsg } = require('#js/server/threads/itc'); + const { ITC_EVENT_TYPES } = require('#src/utility/hdbTerms'); + // the handler has no way to remove a listener; this one only records into a local array + const signalled = []; + schemaHandler.addListener((message) => signalled.push(message)); + + // the signal is fire-and-forget from the declaration, so its local delivery is a turn behind + const { waitFor } = require('../waitFor.js'); + const signalFor = (tableName) => signalled.find((message) => message.table === tableName); + + declare({ table: 'Signalled', database: BASE, schemaDefined: true, attributes: [id] }); + await waitFor(() => signalFor('Signalled'), { + timeout: 5_000, + message: 'a declaration into the branch still signals the other threads', + }); + const forBranch = signalFor('Signalled'); + assert.strictEqual(forBranch.branchPath, branch.path, 'and the signal addresses the branch, not the base'); + assert.strictEqual(forBranch.schema, BASE); + + table({ table: 'GlobalSignalled', database: BASE, schemaDefined: true, attributes: [id] }); + await waitFor(() => signalFor('GlobalSignalled'), { timeout: 5_000, message: 'a base declaration signals' }); + assert.strictEqual(signalFor('GlobalSignalled').branchPath, undefined, 'a base declaration carries no branch path'); + + // what a receiving thread does with it: the branch it holds open learns the new table + createElsewhere(branch, 'FromSignal'); + await schemaHandler( + new ITCEventObject( + ITC_EVENT_TYPES.SCHEMA, + new SchemaEventMsg(process.pid, 'schema-change', BASE, 'FromSignal', undefined, branch.path) + ) + ); + assert.ok(branch.tables.FromSignal, 'the receiver reloaded the branch'); + assert.strictEqual(databases[BASE].FromSignal, undefined); + assert.strictEqual( + reloadBranchAt('/nowhere/such/branch'), + undefined, + 'a thread without the branch has nothing to do' + ); + assertNothingAnnouncedFrom(branch); + }); + + it('releases every store a declaration opened when the branch closes', async function () { + const { branch, declare } = await branchedFactory(); + declare({ + table: 'Released', + database: BASE, + schemaDefined: true, + attributes: [id, { name: 'tag', type: 'String', indexed: true }], + }); + // a re-declaration with an index displaces the wrappers the first one opened; both generations are owned + declare({ + table: 'Released', + database: BASE, + schemaDefined: true, + attributes: [ + id, + { name: 'tag', type: 'String', indexed: true }, + { name: 'other', type: 'String', indexed: true }, + ], + }); + const Released = branch.tables.Released; + const held = [Released.primaryStore, Released.indices.tag, Released.indices.other, ...branch.openedStores]; + assert.ok(held.length > 4, 'sanity: the declarations opened stores'); + + await closeBranchAt(branch.path); + + for (const store of held) { + assert.strictEqual(store.status, 'closed', `a store the branch owned is still open: ${store.name ?? store.path}`); + } + }); + + it('a second, unbranched application alongside sees the untouched base', async function () { + const { scopedBindings } = require('#src/security/jsLoader'); + const { branch, declare } = await branchedFactory(); + declare({ table: 'Private', database: BASE, schemaDefined: true, attributes: [id] }); + + const other = scopedBindings({}); + assert.strictEqual(other.databases, databases); + assert.strictEqual(other.databases[BASE].Private, undefined); + assert.ok(other.databases[BASE].Existing, 'the base table the branch started from is still the base one'); + assert.notStrictEqual(other.databases[BASE].Existing, branch.tables.Existing); + }); +}); + +describeUnlessLmdb('schema authoring paths from a branched scope (harper#2264)', () => { + const { mkdtempSync, writeFileSync, rmSync } = require('node:fs'); + const { tmpdir } = require('node:os'); + const { basename } = require('node:path'); + const { stringify } = require('yaml'); + let Scope, ApplicationScope, Resources, handleApplication, scopedBindings, prepareBranches; + const APP = 'authoringApp'; + const BASE = 'authorbase'; + let directory; + let openScope; + + before(function () { + ({ Scope } = require('#src/components/Scope')); + ({ ApplicationScope } = require('#src/components/ApplicationScope')); + ({ Resources } = require('#src/resources/Resources')); + ({ handleApplication } = require('#src/resources/graphql')); + ({ scopedBindings } = require('#src/security/jsLoader')); + ({ prepareBranches } = require('#src/resources/branchDatabase')); + setupTestDBPath(); + setMainIsWorker(true); + table({ table: 'AuthorSource', database: BASE, attributes: [{ name: 'id', isPrimaryKey: true }] }); + }); + + afterEach(async function () { + try { + await openScope?.close(); + } finally { + openScope = undefined; + if (directory) rmSync(directory, { recursive: true, force: true }); + directory = undefined; + await removeBranches(); + } + }); + + async function branchedApplicationScope() { + const applicationScope = new ApplicationScope(APP, new Resources(), {}); + applicationScope.branches = await prepareBranches(APP, [BASE], 'vm-current-context'); + return applicationScope; + } + + it('lands a GraphQL @table declaration in the branch, exported route and imported binding alike', async function () { + this.timeout(30000); + const applicationScope = await branchedApplicationScope(); + const branch = applicationScope.branches.get(BASE); + directory = mkdtempSync(join(tmpdir(), 'harper.unit-test.branched-graphql-')); + writeFileSync( + join(directory, 'schema.graphql'), + `type GqlDeclared @table(database: "${BASE}") @export {\n\tid: ID @primaryKey\n\tnote: String\n}\n` ); + const configFilePath = join(directory, 'config.yaml'); + writeFileSync(configFilePath, stringify({ graphqlSchema: { files: 'schema.graphql' } })); + const scope = new Scope(basename(directory), 'graphqlSchema', directory, configFilePath, applicationScope); + openScope = scope; + await scope.ready; + + await handleApplication(scope); + + const Declared = branch.tables.GqlDeclared; + assert.ok(Declared, 'the @table declaration created the table in the branch'); + assert.strictEqual(databases[BASE].GqlDeclared, undefined, 'and not in the base'); + const exported = [...applicationScope.resources.values()].find((entry) => entry.Resource === Declared); + assert.ok(exported, 'the exported route is backed by the branch class'); + // what `import { databases } from 'harper'` resolves to inside the application + assert.strictEqual(scopedBindings(applicationScope).databases[BASE].GqlDeclared, Declared); + await Declared.put({ id: 'g', note: 'through the branch' }); + assert.strictEqual((await Declared.get('g'))?.note, 'through the branch'); + }); + + it('lands scope.ensureTable in the branch', async function () { + this.timeout(30000); + const applicationScope = await branchedApplicationScope(); + const branch = applicationScope.branches.get(BASE); + directory = mkdtempSync(join(tmpdir(), 'harper.unit-test.branched-ensure-')); + const configFilePath = join(directory, 'config.yaml'); + writeFileSync(configFilePath, stringify({ jsResource: { files: 'resources.js' } })); + const scope = new Scope(basename(directory), 'jsResource', directory, configFilePath, applicationScope); + openScope = scope; + await scope.ready; + + const Ensured = scope.ensureTable({ + table: 'Ensured', + database: BASE, + attributes: [{ name: 'id', isPrimaryKey: true }], + }); + + assert.strictEqual(branch.tables.Ensured, Ensured); + assert.strictEqual(databases[BASE].Ensured, undefined); + assert.strictEqual(Ensured.origin, basename(directory), 'the origin stamp is unchanged'); }); - it('leaves unbranched targets and unbranched applications alone', function () { - assert.doesNotThrow(() => assertTableTargetNotBranched(new Map([['gatedbase', {}]]), 'other', 'T', 'defineTable')); - assert.doesNotThrow(() => assertTableTargetNotBranched(undefined, 'gatedbase', 'T', 'defineTable')); - assert.doesNotThrow(() => assertTableTargetNotBranched(new Map(), 'gatedbase', 'T', 'defineTable')); + it('leaves an unbranched scope on the global factory and the global objects', async function () { + const applicationScope = new ApplicationScope('plain', new Resources(), {}); + directory = mkdtempSync(join(tmpdir(), 'harper.unit-test.plain-ensure-')); + const configFilePath = join(directory, 'config.yaml'); + writeFileSync(configFilePath, stringify({ jsResource: { files: 'resources.js' } })); + const scope = new Scope(basename(directory), 'jsResource', directory, configFilePath, applicationScope); + openScope = scope; + await scope.ready; + + const definition = { table: 'PlainEnsured', database: BASE, attributes: [{ name: 'id', isPrimaryKey: true }] }; + const Ensured = scope.ensureTable(definition); + + assert.strictEqual(Ensured, databases[BASE].PlainEnsured); + assert.strictEqual(Ensured, table(definition)); + assert.strictEqual(scopedBindings(applicationScope).databases, databases); }); }); @@ -603,15 +979,25 @@ describe('defineTable through a branched application (harper#643)', () => { await removeBranches(); }); - itUnlessLmdb('refuses to define into a branched database rather than defining into the base', async function () { - // defineTable registers in the process-wide catalog, so without this the table would appear in - // the base -- replicated and visible to every other application -- while this application's - // own reads and writes went to its branch. Landing it in the branch is harper#2264. + itUnlessLmdb('defines into the branch rather than into the base (harper#2264)', async function () { + // Without this the table would appear in the base -- replicated and visible to every other + // application -- while this application's own reads and writes went to its branch. + const { types } = require('#src/resources/defineTable'); const branch = await getOrCreateBranch('defbase', 'defApp'); - const { defineTable } = scopedBindings({ branches: new Map([['defbase', branch]]) }); + const { defineTable, databases: scoped } = scopedBindings({ branches: new Map([['defbase', branch]]) }); + const shape = { id: types.id.primaryKey, note: types.string.nullable }; + + const Defined = defineTable('Defined', shape, { database: 'defbase' }); - assert.throws(() => defineTable('Defined', { id: 'string' }, { database: 'defbase' }), /branched database/); - assert.strictEqual(databases.defbase.Defined, undefined, 'and nothing must be created in the base'); + assert.strictEqual(branch.tables.Defined, Defined, 'the returned handle is the branch class'); + assert.strictEqual(scoped.defbase.Defined, Defined, "and what the application's own `databases` resolves"); + assert.strictEqual(databases.defbase.Defined, undefined, 'nothing is created in the base'); + await Defined.put({ id: 'd', note: 'branch-defined' }); + assert.strictEqual((await Defined.get('d'))?.note, 'branch-defined'); + // re-declaring through defineTable evolves the branch table, the way a reload does + const Evolved = defineTable('Defined', { ...shape, extra: types.string.nullable }, { database: 'defbase' }); + assert.strictEqual(Evolved, Defined); + assert.ok(Evolved.attributes.some((attribute) => attribute.name === 'extra')); }); itUnlessLmdb('still defines into databases the application did not branch', async function () { @@ -1819,3 +2205,71 @@ describeUnlessLmdb('branch control paths cannot be spelled as a database name (h ); }); }); + +describeUnlessLmdb('a table declared into a branch on one thread reaches another thread (harper#2264)', () => { + const { Worker } = require('node:worker_threads'); + const { scopedTableFactory } = require('#src/resources/databases'); + const { prepareBranches } = require('#src/resources/branchDatabase'); + const APP = 'threadApp'; + const BASE = 'threadbase'; + + before(function () { + setupTestDBPath(); + setMainIsWorker(true); + table({ table: 'ThreadSource', database: BASE, attributes: [{ name: 'id', isPrimaryKey: true }] }); + }); + + afterEach(async function () { + await removeBranches(); + }); + + it('is loaded by the reload the schema-change signal triggers, not by re-declaring', async function () { + this.timeout(60000); + const branches = await prepareBranches(APP, [BASE], 'vm-current-context'); + const branch = branches.get(BASE); + const phase = new Int32Array(new SharedArrayBuffer(4)); + const worker = new Worker(__dirname + '/branchDeclare-thread.js', { + workerData: { phase, baseName: BASE, appName: APP, tableName: 'Threaded', addPorts: [] }, + }); + try { + const failure = new Promise((_, reject) => worker.once('error', reject)); + const message = (type) => + Promise.race([ + failure, + new Promise((resolve) => { + worker.on('message', function onMessage(received) { + if (received.type !== type) return; + worker.off('message', onMessage); + resolve(received); + }); + }), + ]); + const opened = message('opened'); + const reloaded = message('reloaded'); + assert.strictEqual( + (await opened).loaded, + false, + 'sanity: the other thread opened the branch before the table existed' + ); + + const Threaded = scopedTableFactory(branches)({ + table: 'Threaded', + database: BASE, + schemaDefined: true, + attributes: [{ name: 'id', type: 'ID', isPrimaryKey: true }, { name: 'note' }], + }); + await Threaded.put({ id: 'declared-elsewhere', note: 'seen across threads' }); + Atomics.store(phase, 0, 1); + Atomics.notify(phase, 0); + + const seen = await reloaded; + assert.strictEqual(seen.loaded, true, 'the other thread has the class after its reload'); + assert.deepStrictEqual(seen.attributes, ['id', 'note']); + assert.strictEqual(seen.note, 'seen across threads', 'and reads the row through it'); + assert.strictEqual(databases[BASE].Threaded, undefined, 'the base has nothing on either thread'); + assert.ok(branch.tables.Threaded); + } finally { + await worker.terminate(); + } + }); +}); diff --git a/unitTests/resources/branchDeclare-thread.js b/unitTests/resources/branchDeclare-thread.js new file mode 100644 index 0000000000..e54fbc75ec --- /dev/null +++ b/unitTests/resources/branchDeclare-thread.js @@ -0,0 +1,28 @@ +const { parentPort, workerData } = require('node:worker_threads'); +const { setupTestDBPath } = require('../testUtils'); +const { reloadBranchAt } = require('#src/resources/databases'); +const { getOrCreateBranch } = require('#src/resources/branchDatabase'); +const { setMainIsWorker } = require('#js/server/threads/manageThreads'); + +// phase (main -> here): 0 open the branch, 1 the main thread has declared the table. Absent when +// mocha loads this file itself. +const { phase, baseName, appName, tableName } = workerData ?? {}; +if (phase) run(); + +async function run() { + setupTestDBPath(); + setMainIsWorker(true); + const branch = await getOrCreateBranch(baseName, appName); + parentPort.postMessage({ type: 'opened', loaded: Boolean(branch.tables[tableName]) }); + Atomics.wait(phase, 0, 0); + // what the ITC schema-change handler does on a thread that holds the branch open + reloadBranchAt(branch.path); + const Table = branch.tables[tableName]; + const row = Table ? await Table.get('declared-elsewhere') : undefined; + parentPort.postMessage({ + type: 'reloaded', + loaded: Boolean(Table), + attributes: Table ? Table.attributes.map((attribute) => attribute.name) : [], + note: row?.note ?? null, + }); +} diff --git a/unitTests/resources/cache-probe.bench.js b/unitTests/resources/cache-probe.bench.js new file mode 100644 index 0000000000..6dc36199ed --- /dev/null +++ b/unitTests/resources/cache-probe.bench.js @@ -0,0 +1,34 @@ +require('../testUtils'); +const { setupTestDBPath } = require('../testUtils'); +const { table } = require('#src/resources/databases'); +const { setMainIsWorker } = require('#js/server/threads/manageThreads'); + +describe('warm cached read', function () { + it('measures warm getEntry throughput', async function () { + this.timeout(120_000); + await main(); + }); +}); + +async function main() { + setupTestDBPath(); + setMainIsWorker(true); + const TestTable = table({ + table: 'CacheProbeBench', + database: 'test', + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'name' }], + }); + const store = TestTable.primaryStore; + await TestTable.put(1, { name: 'warm', payload: 'x'.repeat(200) }); + await TestTable.get(1); // warm cache + VT + store.getEntry(1); + + const N = 2_000_000; + for (let i = 0; i < 100_000; i++) store.getEntry(1); + const t0 = process.hrtime.bigint(); + for (let i = 0; i < N; i++) store.getEntry(1); + const t1 = process.hrtime.bigint(); + const warmNs = Number(t1 - t0) / N; + + console.log(`warm getEntry: ${warmNs.toFixed(0)} ns/op (${(1e9 / warmNs / 1e6).toFixed(2)} M ops/s)`); +} diff --git a/unitTests/resources/caching-rocks-database.test.js b/unitTests/resources/caching-rocks-database.test.js index 6f63715566..4cb4f5eb81 100644 --- a/unitTests/resources/caching-rocks-database.test.js +++ b/unitTests/resources/caching-rocks-database.test.js @@ -2,9 +2,9 @@ require('../testUtils'); const assert = require('assert'); const { setupTestDBPath } = require('../testUtils'); const { table } = require('#src/resources/databases'); -const { VERSION_NOT_UNIQUE_FLAG } = require('#src/resources/RecordEncoder'); +const { VERSION_REUSED } = require('#src/resources/RecordEncoder'); const { setMainIsWorker } = require('#js/server/threads/manageThreads'); -const { RocksDatabase, constants } = require('@harperfast/rocksdb-js'); +const { RocksDatabase } = require('@harperfast/rocksdb-js'); const { PrimaryRocksDatabase } = require('#src/resources/PrimaryRocksDatabase'); const isLMDB = process.env.HARPER_STORAGE_ENGINE === 'lmdb'; @@ -116,33 +116,74 @@ describe('PrimaryRocksDatabase', function () { assert.equal(result.name, 'eight updated'); }); - it('marks a record whose version was reused by a resequenced write', async function () { + it('marks a record whose version was reused by a resequenced write, preserving its expiresAt', async function () { const now = Date.now(); const expiresAt = now + 60_000; - await TestTable.put(9, { name: 'base', count: 0 }); - await TestTable.patch(9, { count: { __op__: 'add', value: 1 } }, { timestamp: now + 100 }); - const inOrder = TestTable.primaryStore.getEntry(9); - await TestTable.patch(9, { count: { __op__: 'add', value: 1 } }, { timestamp: now + 50, expiresAt }); - const resequenced = TestTable.primaryStore.getEntry(9); + await TestTable.put(13, { name: 'base', count: 0 }); + await TestTable.patch(13, { count: { __op__: 'add', value: 1 } }, { timestamp: now + 100 }); + const inOrder = TestTable.primaryStore.getEntry(13); + await TestTable.patch(13, { count: { __op__: 'add', value: 1 } }, { timestamp: now + 50, expiresAt }); + const resequenced = TestTable.primaryStore.getEntry(13); assert.equal(resequenced.version, inOrder.version); assert.equal(resequenced.value.count, 2); assert.equal(resequenced.expiresAt, expiresAt); - assert(resequenced.metadataFlags & VERSION_NOT_UNIQUE_FLAG); + assert(resequenced.metadataFlags & VERSION_REUSED); }); - it('does not vouch for stale cached data after a version-reusing write', async function () { - // A drift between Harper's flag and the native constant would silently disarm the assertions below. - assert.equal(constants.VERSION_NOT_UNIQUE_FLAG, VERSION_NOT_UNIQUE_FLAG); + it('VT does not vouch for a version a resequenced write reused', async function () { + const store = TestTable.primaryStore; const now = Date.now(); - await TestTable.put(11, { name: 'base', count: 0 }); - await TestTable.patch(11, { count: { __op__: 'add', value: 1 } }, { timestamp: now + 100 }); - await TestTable.get(11); - await TestTable.get(11); - const inOrder = TestTable.primaryStore.getEntry(11); - assert(TestTable.primaryStore.verifyVersion(11, inOrder.version)); + await TestTable.put(9, { name: 'base', count: 0 }); + await TestTable.patch(9, { name: 'newer', count: { __op__: 'add', value: 1 } }, { timestamp: now + 100 }); + await TestTable.get(9); + const inOrder = store.getEntry(9); + assert(store.verifyVersion(9, inOrder.version), 'VT should vouch for an in-order version'); + + // out-of-order: merges onto the newer record and stores under its (reused) version + await TestTable.patch(9, { count: { __op__: 'add', value: 1 } }, { timestamp: now + 50 }); + const resequenced = store.getEntry(9); + assert.equal(resequenced.version, inOrder.version, 'resequenced write keeps the existing version'); + assert.equal(resequenced.value.count, 2, 'both increments are applied'); + assert(resequenced.metadataFlags & VERSION_REUSED, 'the stored record is marked non-unique for the native layer'); + assert(!store.verifyVersion(9, resequenced.version), 'VT must not vouch for a version shared by two stored values'); + assert.notEqual(store.getEntry(9).value, resequenced.value, 'a flagged record is never served from cache'); + assert(!store.verifyVersion(9, resequenced.version), 'reading a flagged record must not publish its version'); + const client = await TestTable.get(9); + assert.equal(client.count, 2, 'a client-level read sees the merged value, not a stale cache vouch'); + }); + it('an uncachedRead bypasses the cache vouch and returns the stored record', async function () { + const store = TestTable.primaryStore; + await TestTable.put(12, { name: 'stored', count: 5 }); + await TestTable.get(12); // warm the cache and seed the VT slot + const cached = store.getEntry(12); + assert(store.verifyVersion(12, cached.version), 'VT vouches for the in-order version'); + // the vouch fast path returns the cached object itself, so identity is what distinguishes them + assert.equal(store.getEntry(12).value, cached.value, 'the vouch path serves the cached object'); + const direct = store.getEntry(12, { uncachedRead: true }); + assert.notEqual(direct.value, cached.value, 'uncachedRead must decode from the store, not serve the cache'); + assert.equal(direct.value.name, 'stored'); + assert.equal(direct.value.count, 5); + assert.equal(direct.version, cached.version, 'the decoded entry carries the stored version'); + }); + + it('an in-order write after a resequenced one restores vouching', async function () { + const store = TestTable.primaryStore; + const now = Date.now(); + await TestTable.put(11, { name: 'base', count: 0 }); + await TestTable.patch(11, { name: 'newer', count: { __op__: 'add', value: 1 } }, { timestamp: now + 100 }); await TestTable.patch(11, { count: { __op__: 'add', value: 1 } }, { timestamp: now + 50 }); - assert.equal((await TestTable.get(11)).count, 2); - assert(!TestTable.primaryStore.verifyVersion(11, inOrder.version)); + const reused = store.getEntry(11); + assert(reused.metadataFlags & VERSION_REUSED, 'the reused version is marked'); + assert(!store.verifyVersion(11, reused.version), 'the reused version is not vouched for'); + + // the next in-order write gives the record a version of its own; vouching resumes with the + // same one-read lag as any cold key + await TestTable.patch(11, { name: 'later' }, { timestamp: now + 200 }); + const advanced = store.getEntry(11); + assert(advanced.version > reused.version, 'the in-order write advances the version'); + assert.equal(advanced.value.count, 2, 'the merged count survives the in-order write'); + store.getEntry(11); + assert(store.verifyVersion(11, advanced.version), 'vouching resumes for a version of its own'); }); }); diff --git a/unitTests/resources/caching.test.js b/unitTests/resources/caching.test.js index 31610311b2..27cd52dc8a 100644 --- a/unitTests/resources/caching.test.js +++ b/unitTests/resources/caching.test.js @@ -6,7 +6,7 @@ const { table } = require('#src/resources/databases'); const { Resource } = require('#src/resources/Resource'); const { setMainIsWorker } = require('#js/server/threads/manageThreads'); const { RequestTarget } = require('#src/resources/RequestTarget'); -const { VERSION_NOT_UNIQUE_FLAG } = require('#src/resources/RecordEncoder'); +const { VERSION_REUSED } = require('#src/resources/RecordEncoder'); const { INVALIDATED } = require('#src/resources/Table'); const { exportIdMapping } = require('#src/resources/nodeIdMapping'); const { transaction } = require('#src/resources/transaction'); @@ -599,7 +599,7 @@ describe('Caching', () => { assert.equal(ConflictCachingTable.primaryStore.getSync(id).name, 'refreshed'); const refreshedEntry = ConflictCachingTable.primaryStore.getEntry(id); assert.equal(refreshedEntry.version, invalidatedVersion); - assert(refreshedEntry.metadataFlags & VERSION_NOT_UNIQUE_FLAG); + assert(refreshedEntry.metadataFlags & VERSION_REUSED); }); it('uses a source-reported version before the transaction timestamp', async function () { @@ -668,7 +668,7 @@ describe('Caching', () => { await waitFor(() => !ConflictCachingTable.primaryStore.hasLock(id)); const refreshedEntry = ConflictCachingTable.primaryStore.getEntry(id); assert.equal(refreshedEntry.version, invalidatedVersion); - assert(refreshedEntry.metadataFlags & VERSION_NOT_UNIQUE_FLAG); + assert(refreshedEntry.metadataFlags & VERSION_REUSED); }); it('falls back from an invalid source-reported version', async function () { diff --git a/unitTests/resources/crdt.test.js b/unitTests/resources/crdt.test.js index 4ac3c29e30..3e4fc937e1 100644 --- a/unitTests/resources/crdt.test.js +++ b/unitTests/resources/crdt.test.js @@ -122,6 +122,24 @@ describe('crdt getRecordAtTime', () => { assert.deepStrictEqual(getRecordAtTime(current, 250, store, 1, 'D'), { id: 'D', count: 3 }); }); + it('falls back to the reported position when no ref resolves the head', () => { + // A ref that names an audit key whose entry carries a different record version resolves nothing, + // so the walk has to start from the position the record reports for itself. Only the RocksDB + // shape reaches this: LMDB records carry no refs. + const events = [ + { version: 10, type: 'put', value: { id: 'F', count: 1 }, previousVersion: 0 }, + { version: 20, type: 'patch', value: { count: { __op__: 'add', value: 2 } }, previousVersion: 10 }, + { txnLogKey: 900, version: 999, type: 'put', value: { id: 'F', count: 999 }, previousVersion: 0 }, + ]; + const store = makeStore(events); + const current = currentEntry({ id: 'F', count: 3 }, 20, { + version: 20, + nodeId: 1, + additionalAuditRefs: [{ version: 900, nodeId: 1 }], + }); + assert.deepStrictEqual(getRecordAtTime(current, 10, store, 1, 'F'), { id: 'F', count: 1 }); + }); + describe('record deleted then re-inserted under the same key (issue #1330)', () => { // put(n:1) -> patch(n:2) -> patch(n:3) -> delete -> put(n:4, re-insert, current) const events = [ diff --git a/unitTests/resources/derivedIndexRegistry.test.js b/unitTests/resources/derivedIndexRegistry.test.js index 0f447a68fc..1bb5edbe54 100644 --- a/unitTests/resources/derivedIndexRegistry.test.js +++ b/unitTests/resources/derivedIndexRegistry.test.js @@ -1,5 +1,9 @@ const assert = require('node:assert'); -const { hasDerivedIndexRegistration, registerDerivedIndexTables } = require('#src/resources/derivedIndexRegistry'); +const { + derivedIndexWriteRejection, + hasDerivedIndexRegistration, + registerDerivedIndexTables, +} = require('#src/resources/derivedIndexRegistry'); describe('derived index registration tracking', () => { it('counts registrations independently by audit store and table', () => { @@ -25,4 +29,20 @@ describe('derived index registration tracking', () => { releaseOtherStore(); assert.strictEqual(hasDerivedIndexRegistration(secondStore, 1), false); }); + + it('returns the first admission reason for a table and none once released', () => { + const store = {}; + let reason; + const releaseGated = registerDerivedIndexTables(store, [1], () => reason); + const releaseOpen = registerDerivedIndexTables(store, [1, 2]); + assert.strictEqual(derivedIndexWriteRejection(store, 1), undefined); + reason = 'behind'; + assert.strictEqual(derivedIndexWriteRejection(store, 1), 'behind'); + assert.strictEqual(derivedIndexWriteRejection(store, 2), undefined); + releaseGated(); + assert.strictEqual(derivedIndexWriteRejection(store, 1), undefined); + assert.strictEqual(hasDerivedIndexRegistration(store, 1), true); + releaseOpen(); + assert.strictEqual(hasDerivedIndexRegistration(store, 1), false); + }); }); diff --git a/unitTests/resources/derivedIndexRuntime.bench.js b/unitTests/resources/derivedIndexRuntime.bench.js new file mode 100644 index 0000000000..09df0a88aa --- /dev/null +++ b/unitTests/resources/derivedIndexRuntime.bench.js @@ -0,0 +1,414 @@ +/** + * Benchmark: the shared derived-index runtime feeding a backend with a synthetic per-mutation cost + * and a fixed-cost durability barrier. Run via: npx mocha unitTests/resources/derivedIndexRuntime.bench.js + */ +const { EventEmitter } = require('node:events'); +const { performance } = require('node:perf_hooks'); +const { + DERIVED_INDEX_ACCEPTED, + DERIVED_INDEX_DEFERRED, + DerivedIndexRuntime, +} = require('#src/resources/derivedIndexRuntime'); +const { derivedIndexWriteRejection, registerDerivedIndexTables } = require('#src/resources/derivedIndexRegistry'); + +const APPLY_MICROS = Number(process.env.DERIVED_BENCH_APPLY_MICROS ?? 350); +const BARRIER_MILLIS = Number(process.env.DERIVED_BENCH_BARRIER_MILLIS ?? 5); +const DIMENSIONS = 384; +const RECORD_BYTES = DIMENSIONS * 4; + +function busyWait(micros) { + const until = performance.now() + micros / 1000; + while (performance.now() < until); +} + +class LiveLogStore { + constructor() { + this.log = []; + this.locks = new Set(); + this.sharedBuffers = new Map(); + this.rootStore = new EventEmitter(); + this.rootStore.listLogs = () => ['local']; + this.rootStore.useLog = (name) => ({ name, getStats: () => ({ oldestSequenceNumber: 1 }) }); + this.nextTimestamp = 1; + } + + commit(recordId, size = RECORD_BYTES) { + const timestamp = this.nextTimestamp++; + this.log.push({ + logName: 'local', + txnLogKey: timestamp, + version: timestamp, + recordId, + tableId: 1, + type: 'put', + endTxn: true, + size, + committedAt: performance.now(), + }); + this.rootStore.emit('committed'); + return timestamp; + } + + getRange(options) { + const start = options.startByLog?.get('local') ?? 0; + const log = this.log; + let index = log.findIndex((entry) => entry.txnLogKey > start); + if (index < 0) index = log.length; + return { + corruptFrameStop: { breaks: 0, truncatedVersions: new Set(), midLogBreak: false }, + failedLogs: new Set(), + exactStartFailures: new Map(), + [Symbol.iterator]() { + return { + next: () => (index < log.length ? { value: log[index++], done: false } : { value: undefined, done: true }), + return: () => ({ value: undefined, done: true }), + }; + }, + }; + } + + tryLock(key) { + if (this.locks.has(key)) return false; + this.locks.add(key); + return true; + } + + unlock(key) { + this.locks.delete(key); + } + + getUserSharedBuffer(key, defaultBuffer) { + let buffer = this.sharedBuffers.get(key); + if (!buffer) this.sharedBuffers.set(key, (buffer = new SharedArrayBuffer(defaultBuffer.byteLength))); + return buffer; + } +} + +class InlineBackend { + constructor(id, { useRecords = true } = {}) { + this.id = id; + this.cursor = { format: 1, logs: {} }; + this.applies = 0; + this.useRecords = useRecords; + } + attach() {} + flush() {} + shutdown() {} + getDurableCursor() { + return this.cursor; + } + deliver(batch) { + const mutations = this.useRecords + ? batch.records + : batch.transactions.flatMap((transaction) => transaction.mutations); + for (let i = 0; i < mutations.length; i++) { + busyWait(APPLY_MICROS); + this.applies++; + } + busyWait(BARRIER_MILLIS * 1000); + this.barriers = (this.barriers ?? 0) + 1; + this.cursor = batch.through; + return DERIVED_INDEX_ACCEPTED; + } + onStateChange(wake) { + this.wake = wake; + return () => {}; + } +} + +class QueueBackend { + constructor(id, { sliceMillis = 4, capacityBytes = 64 * 1024 * 1024 } = {}) { + this.id = id; + this.cursor = { format: 1, logs: {} }; + this.queue = []; + this.queuedBytes = 0; + this.peakQueuedBytes = 0; + this.applies = 0; + this.barriers = 0; + this.appliedCursor = this.cursor; + this.appliedAt = new Map(); + this.durableAt = new Map(); + this.sliceMillis = sliceMillis; + this.capacityBytes = capacityBytes; + this.scheduled = false; + this.flushRequested = false; + this.flushing = false; + this.position = 0; + } + attach(host) { + this.host = host; + } + getDurableCursor() { + return this.cursor; + } + deliver(batch) { + if (this.queuedBytes >= this.capacityBytes) return DERIVED_INDEX_DEFERRED; + this.queue.push(batch); + this.queuedBytes += batch.bytes; + this.peakQueuedBytes = Math.max(this.peakQueuedBytes, this.queuedBytes); + this.schedule(); + return DERIVED_INDEX_ACCEPTED; + } + schedule() { + if (this.scheduled) return; + this.scheduled = true; + setImmediate(() => { + this.scheduled = false; + const until = performance.now() + this.sliceMillis; + while (this.queue.length && performance.now() < until) { + const batch = this.queue[0]; + if (!this.host.isOwnerEpoch(batch.ownerEpoch)) { + this.queue.shift(); + continue; + } + while (this.position < batch.records.length && performance.now() < until) { + busyWait(APPLY_MICROS); + this.applies++; + this.appliedAt.set(batch.records[this.position].logVersion, performance.now()); + this.position++; + } + if (this.position < batch.records.length) break; + this.queue.shift(); + this.queuedBytes -= batch.bytes; + this.position = 0; + if (batch.through) this.appliedCursor = batch.through; + } + if (this.queue.length) this.schedule(); + else if (this.flushRequested) this.runFlush(); + if (this.queuedBytes < this.capacityBytes) this.wake?.('changed'); + }); + } + flush() { + this.flushRequested = true; + if (!this.queue.length) this.runFlush(); + } + runFlush() { + if (this.flushing) return; + this.flushRequested = false; + this.flushing = true; + const through = this.appliedCursor; + setImmediate(() => { + busyWait(BARRIER_MILLIS * 1000); + this.barriers++; + this.flushing = false; + this.cursor = through; + const now = performance.now(); + for (const version of this.appliedAt.keys()) { + if (version <= (through.logs.local ?? 0) && !this.durableAt.has(version)) this.durableAt.set(version, now); + } + this.wake?.('changed'); + }); + } + shutdown() { + this.queue.length = 0; + this.queuedBytes = 0; + } + onStateChange(wake) { + this.wake = wake; + return () => {}; + } +} + +function makeRuntime(store, options) { + const vector = new Float32Array(DIMENSIONS); + return new DerivedIndexRuntime( + store, + (tableId, recordId) => ({ version: 1, value: { id: recordId, vector }, size: RECORD_BYTES }), + { idleGraceMilliseconds: 60_000, ...options } + ); +} + +const registration = (backend, options) => ({ + backend, + projections: new Map([[1, (record) => record.vector]]), + options, +}); + +const until = (condition, timeout = 120_000) => + new Promise((resolve, reject) => { + const deadline = Date.now() + timeout; + const poll = () => { + if (condition()) return resolve(); + if (Date.now() > deadline) return reject(new Error('bench condition timed out')); + setTimeout(poll, 5); + }; + poll(); + }); + +function eventLoopProbe() { + const probe = { max: 0, samples: [] }; + let last = performance.now(); + const interval = setInterval(() => { + const now = performance.now(); + const delay = now - last - 1; + probe.samples.push(delay); + probe.max = Math.max(probe.max, delay); + last = now; + }, 1); + probe.stop = () => { + clearInterval(interval); + probe.samples.sort((a, b) => a - b); + probe.p99 = probe.samples[Math.floor(probe.samples.length * 0.99)] ?? 0; + return probe; + }; + return probe; +} + +const fmt = (n, digits = 1) => Number(n).toFixed(digits); + +describe('Benchmark: derived-index runtime with a costly native-shaped backend', function () { + this.timeout(0); + + before(() => { + console.log(`\n apply cost ${APPLY_MICROS} µs/mutation, barrier ${BARRIER_MILLIS} ms, ${DIMENSIONS}-d records`); + }); + + it('admission check: cost per local write with no index, an index without a policy, and an admitting policy', () => { + const store = {}; + const words = new Int32Array(new ArrayBuffer(64)); + const measure = (label) => { + const rounds = 5_000_000; + let hits = 0; + const started = performance.now(); + for (let i = 0; i < rounds; i++) if (derivedIndexWriteRejection(store, 1) !== undefined) hits++; + const nanos = ((performance.now() - started) * 1e6) / rounds; + console.log(` ${label.padEnd(36)} | ${fmt(nanos)} ns per write (${hits} rejections)`); + }; + measure('no derived index on the table'); + const releasePlain = registerDerivedIndexTables(store, [1]); + measure('index registered, no lag policy'); + releasePlain(); + const releaseGated = registerDerivedIndexTables(store, [1], () => + Atomics.load(words, 5) === 1 ? 'behind' : undefined + ); + measure('lag policy on, admitting'); + releaseGated(); + }); + + it('admission check on a real table: put throughput with the staging-layer guard live and stubbed', async function () { + if (process.env.HARPER_STORAGE_ENGINE === 'lmdb') return this.skip(); + const { setupTestDBPath } = require('../testUtils'); + setupTestDBPath(); + require('#js/server/threads/manageThreads').setMainIsWorker(true); + const { table } = require('#src/resources/databases'); + const registry = require('#src/resources/derivedIndexRegistry'); + const Plain = table({ + database: 'derived-index-admission-bench', + table: 'Plain', + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'title' }], + }); + const live = registry.derivedIndexWriteRejection; + const measure = async (label, rounds = 20_000) => { + const started = performance.now(); + for (let i = 0; i < rounds; i++) await Plain.put(`${label}-${i}`, { title: label }); + const seconds = (performance.now() - started) / 1000; + console.log(` ${label.padEnd(36)} | ${fmt(rounds / seconds, 0)} puts/s`); + }; + await measure('warm-up', 5_000); + await measure('guard live (no derived index)'); + // The staging layer reads the export at call time, so replacing it replaces the guard; prove it. + registry.derivedIndexWriteRejection = () => 'probe'; + try { + await Plain.put('probe', { title: 'probe' }); + throw new Error('the stub did not reach the staging layer'); + } catch (error) { + if (error.code !== 'DERIVED_INDEX_LAGGING') throw error; + } + registry.derivedIndexWriteRejection = () => undefined; + try { + await measure('guard stubbed out'); + } finally { + registry.derivedIndexWriteRejection = live; + } + await measure('guard live again'); + }); + + it('coalescing: repeated keys within one delivery window', async () => { + for (const useRecords of [false, true]) { + const store = new LiveLogStore(); + for (let round = 0; round < 20; round++) for (let key = 0; key < 50; key++) store.commit(`k${key}`); + const backend = new InlineBackend(`coalesce-${useRecords}`, { useRecords }); + const runtime = makeRuntime(store, { maxTransactionsPerTurn: 1000, maxMillisecondsPerTurn: 1000 }); + const started = performance.now(); + runtime.register(registration(backend)); + await until(() => backend.cursor.logs.local === store.nextTimestamp - 1); + const elapsed = performance.now() - started; + console.log( + ` ${useRecords ? 'records (coalesced)' : 'transactions (per occurrence)'}`.padEnd(38) + + `| applies ${String(backend.applies).padStart(5)} | ${fmt(elapsed)} ms for 1000 transactions over 50 keys` + ); + await runtime.stop(); + } + }); + + it('queue-and-accept: event-loop delay while a 5,000-mutation window is applied', async () => { + for (const shape of ['inline', 'queued']) { + const store = new LiveLogStore(); + for (let i = 0; i < 5000; i++) store.commit(`r${i}`); + const backend = shape === 'inline' ? new InlineBackend(shape) : new QueueBackend(shape, { sliceMillis: 4 }); + const runtime = makeRuntime(store, { maxTransactionsPerTurn: 5000, maxMillisecondsPerTurn: 50 }); + const probe = eventLoopProbe(); + const started = performance.now(); + runtime.register(registration(backend, { flushAfterMutations: 5000, maxFlushAgeMilliseconds: 50 })); + await until(() => backend.cursor.logs.local === store.nextTimestamp - 1); + const elapsed = performance.now() - started; + const { max, p99 } = probe.stop(); + console.log( + ` ${shape}`.padEnd(38) + + `| loop delay max ${fmt(max).padStart(7)} ms, p99 ${fmt(p99).padStart(6)} ms | ${fmt(elapsed)} ms total, ${fmt((5000 / elapsed) * 1000, 0)} mutations/s` + ); + await runtime.stop(); + } + }); + + it('independently paced arrivals: latency, throughput, queue bytes, durability age per cadence', async () => { + const RATE = Number(process.env.DERIVED_BENCH_RATE ?? 1500); + const SECONDS = Number(process.env.DERIVED_BENCH_SECONDS ?? 3); + const cadences = [ + ['flush every batch', { flushAfterMutations: 1, maxFlushAgeMilliseconds: 1 }], + ['age 100 ms / 512 mutations', { flushAfterMutations: 512, maxFlushAgeMilliseconds: 100 }], + ['default (age 1 s / 4096)', {}], + ]; + console.log(` arrivals at ${RATE}/s for ${SECONDS} s, 200 distinct keys`); + for (const [name, options] of cadences) { + const store = new LiveLogStore(); + const backend = new QueueBackend(`paced-${name}`, { sliceMillis: 4 }); + const runtime = makeRuntime(store, { maxMillisecondsPerTurn: 5 }); + let peakAccepted = 0; + runtime.register(registration(backend, options)); + const total = RATE * SECONDS; + let committed = 0; + const startedAt = performance.now(); + await new Promise((resolve) => { + const tick = () => { + const due = Math.min(total, Math.floor(((performance.now() - startedAt) / 1000) * RATE)); + while (committed < due) store.commit(`k${committed++ % 200}`); + peakAccepted = Math.max(peakAccepted, runtime.getMetrics(backend.id).acceptedBytes); + if (committed >= total) resolve(); + else setTimeout(tick, 1); + }; + tick(); + }); + await until(() => backend.cursor.logs.local === store.nextTimestamp - 1); + const finishedAt = performance.now(); + const latencies = []; + let maxAge = 0; + for (const entry of store.log) { + const durableAt = backend.durableAt.get(entry.txnLogKey); + if (durableAt === undefined) continue; + latencies.push(durableAt - entry.committedAt); + maxAge = Math.max(maxAge, durableAt - entry.committedAt); + } + latencies.sort((a, b) => a - b); + const p50 = latencies[Math.floor(latencies.length * 0.5)] ?? 0; + const p99 = latencies[Math.floor(latencies.length * 0.99)] ?? 0; + console.log( + ` ${name}`.padEnd(30) + + `| write→durable p50 ${fmt(p50).padStart(7)} ms p99 ${fmt(p99).padStart(7)} ms max ${fmt(maxAge).padStart(7)} ms` + + ` | indexed ${fmt((total / (finishedAt - startedAt)) * 1000, 0).padStart(5)}/s, applies ${backend.applies}` + + ` | peak queue ${fmt(Math.max(peakAccepted, backend.peakQueuedBytes) / 1024, 0)} KiB | barriers ${backend.barriers}` + ); + await runtime.stop(); + } + }); +}); diff --git a/unitTests/resources/derivedIndexRuntime.test.js b/unitTests/resources/derivedIndexRuntime.test.js index 9243665305..dae5dc3faa 100644 --- a/unitTests/resources/derivedIndexRuntime.test.js +++ b/unitTests/resources/derivedIndexRuntime.test.js @@ -66,6 +66,12 @@ class FakeBackend { this.deliverImpl = deliver; } + attach() {} + + flush() {} + + shutdown() {} + getDurableCursor() { return this.cursor; } diff --git a/unitTests/resources/derivedIndexRuntimeNativeBackend.test.js b/unitTests/resources/derivedIndexRuntimeNativeBackend.test.js new file mode 100644 index 0000000000..03a1a8eb64 --- /dev/null +++ b/unitTests/resources/derivedIndexRuntimeNativeBackend.test.js @@ -0,0 +1,2019 @@ +require('../testUtils'); +const assert = require('node:assert'); +const { EventEmitter } = require('node:events'); +const { waitFor } = require('../waitFor'); +const { setupTestDBPath } = require('../testUtils'); +const { ClientError } = require('#src/utility/errors/hdbError'); +const { + derivedIndexWriteRejection, + hasDerivedIndexRegistration, + registerDerivedIndexTables, +} = require('#src/resources/derivedIndexRegistry'); +const { + DERIVED_INDEX_ACCEPTED, + DERIVED_INDEX_DEFERRED, + DerivedIndexRuntime, + READINESS_BYTES, + readDerivedIndexReadiness, +} = require('#src/resources/derivedIndexRuntime'); + +// A shared fake of the RocksDB transaction-log store: `entriesByCursor` maps a resume timestamp to +// the entries physically after it, `logEntries` is the retained log used by the rebuild boundary +// capture, and every worker (runtime) sharing one instance shares its locks and shared buffers. +class FakeLogStore { + constructor( + entriesByCursor, + { logNames = ['local'], logEntries = new Map(), onNext, live = false, markers = new Map() } = {} + ) { + this.entriesByCursor = entriesByCursor; + this.logEntries = logEntries; + // Root-store markers survive a "restart" (a new FakeLogStore sharing this map); shared buffers do not. + this.markers = markers; + this.onNext = onNext; + this.live = live; + this.locks = new Set(); + this.waiters = new Map(); + this.sharedBuffers = new Map(); + this.bufferLookups = 0; + this.rangeCalls = []; + this.exactStartFailures = new Map(); + this.rootStore = new EventEmitter(); + this.rootStore.listLogs = () => logNames.slice(); + // Like rocksdb-js: a log that never wrote a file reports no files and oldest sequence 0. + this.rootStore.useLog = (name) => ({ + name, + getStats: () => { + const retained = this.logEntries.get(name); + if (retained !== undefined && retained.length === 0) return { fileCount: 0, oldestSequenceNumber: 0 }; + return { fileCount: 1, oldestSequenceNumber: 1 }; + }, + }); + this.rootStore.getSync = (key) => this.markers.get(key); + this.rootStore.removeSync = (key) => this.markers.delete(key); + } + + putSync(key, value) { + this.markers.set(key, value); + } + + getRange(options) { + this.rangeCalls.push(options); + let entries; + if (options.log !== undefined) { + entries = (this.logEntries.get(options.log) ?? []).map((entry) => ({ ...entry })); + } else { + const start = options.startByLog.get('local'); + const source = this.entriesByCursor.get(start) ?? []; + entries = this.live ? source : source.map((entry) => ({ ...entry })); + } + const store = this; + const iterable = { + corruptFrameStop: { breaks: 0, truncatedVersions: new Set(), midLogBreak: false }, + failedLogs: new Set(), + exactStartFailures: new Map(options.log === undefined ? this.exactStartFailures : []), + [Symbol.iterator]() { + let index = 0; + return { + next() { + store.onNext?.(entries[index], index); + return index < entries.length ? { value: entries[index++], done: false } : { value: undefined, done: true }; + }, + return() { + return { value: undefined, done: true }; + }, + }; + }, + }; + return iterable; + } + + tryLock(key, onUnlocked) { + if (!this.locks.has(key)) { + this.locks.add(key); + return true; + } + if (onUnlocked) { + let waiters = this.waiters.get(key); + if (!waiters) this.waiters.set(key, (waiters = [])); + waiters.push(onUnlocked); + } + return false; + } + + unlock(key) { + this.locks.delete(key); + for (const waiter of this.waiters.get(key) ?? []) setImmediate(waiter); + this.waiters.delete(key); + } + + getUserSharedBuffer(key, defaultBuffer, options) { + this.bufferLookups++; + let memory = this.sharedBuffers.get(key); + if (!memory) { + memory = { buffer: new SharedArrayBuffer(defaultBuffer.byteLength), callbacks: new Set() }; + this.sharedBuffers.set(key, memory); + } + // Like the native binding, each lookup returns its own wrapper over the same shared memory with + // notification and cancellation bound to that lookup's subscription. + const wrapper = structuredClone(memory.buffer); + const { callback } = options ?? {}; + if (callback) memory.callbacks.add(callback); + wrapper.callbacks = memory.callbacks; + wrapper.notify = () => { + for (const listener of memory.callbacks) setImmediate(listener); + }; + wrapper.cancel = () => { + if (callback) memory.callbacks.delete(callback); + }; + return wrapper; + } +} + +// A queue-and-accept backend shaped like a native index: deliver() only enqueues, an applier drains +// the queue asynchronously, and the cursor becomes durable at flush(). Every apply and flush is +// fenced by the owner epoch the runtime handed it through attach(). +class AsyncBackend { + constructor(id, { cursor, applyDelay = 0, capacity = Infinity, onReset, applyRecord } = {}) { + this.id = id; + this.cursor = cursor; + this.deliveries = []; + this.queue = []; + this.applied = new Map(); + this.appliedCursor = cursor; + this.flushes = []; + this.resets = []; + this.shutdowns = []; + this.fenced = 0; + this.applyDelay = applyDelay; + this.capacity = capacity; + this.onReset = onReset; + this.applyRecord = applyRecord; + this.pendingFlush = undefined; + this.applying = false; + } + + attach(host) { + this.host = host; + } + + getDurableCursor() { + return this.cursor; + } + + deliver(batch) { + this.deliveries.push(batch); + if (this.queue.length >= this.capacity) return DERIVED_INDEX_DEFERRED; + this.currentEpoch = batch.ownerEpoch; + this.queue.push(batch); + this.scheduleApply(); + return DERIVED_INDEX_ACCEPTED; + } + + scheduleApply() { + if (this.applying || this.queue.length === 0) return; + this.applying = true; + setTimeout(() => { + this.applying = false; + const batch = this.queue.shift(); + if (!this.host.isOwnerEpoch(batch.ownerEpoch)) { + this.fenced++; + } else { + try { + for (const record of batch.records) { + if (this.applyRecord) this.applyRecord(record); + if (record.state.kind === 'record') this.applied.set(record.recordId, record.state); + else this.applied.delete(record.recordId); + } + if (batch.through) this.appliedCursor = batch.through; + } catch { + this.queue.length = 0; + this.stateChange?.('failed'); + return; + } + } + const hadCapacity = this.queue.length < this.capacity; + this.scheduleApply(); + if (!hadCapacity || this.queue.length === 0) this.stateChange?.('changed'); + }, this.applyDelay); + } + + flush(reason) { + this.flushes.push(reason); + if (this.pendingFlush) return; + const epoch = this.currentEpoch; + this.pendingFlush = new Promise((resolve) => + setTimeout(() => { + this.pendingFlush = undefined; + if (epoch !== undefined && !this.host.isOwnerEpoch(epoch)) this.fenced++; + else if (this.queue.length === 0 && !this.applying) { + this.cursor = this.appliedCursor; + this.stateChange?.('changed'); + } else this.flush('age'); + resolve(); + }, this.applyDelay + 1) + ); + } + + reset(ownerEpoch) { + this.onReset?.(ownerEpoch); + this.resets.push(ownerEpoch); + this.queue.length = 0; + this.applied.clear(); + this.cursor = undefined; + this.appliedCursor = undefined; + } + + async shutdown(ownerEpoch) { + this.shutdowns.push(ownerEpoch); + while (this.pendingFlush) await this.pendingFlush; + this.queue.length = 0; + } + + onStateChange(wake) { + this.stateChange = wake; + return () => { + if (this.stateChange === wake) this.stateChange = undefined; + }; + } +} + +// A backend that applies and makes each batch durable inside deliver(): the hooks are trivial, but +// the contract still requires them so no backend can compile without the fence and the handshake. +class SyncBackend { + constructor(id, cursor, deliver) { + this.id = id; + this.cursor = cursor; + this.deliveries = []; + this.deliverImpl = deliver; + this.flushes = []; + } + + attach(host) { + this.host = host; + } + + flush(reason) { + this.flushes.push(reason); + } + + shutdown() {} + + getDurableCursor() { + return this.cursor; + } + + deliver(batch) { + this.deliveries.push(batch); + if (this.deliverImpl) return this.deliverImpl(batch, this); + this.cursor = batch.through; + return DERIVED_INDEX_ACCEPTED; + } + + onStateChange(wake) { + this.stateChange = wake; + return () => { + if (this.stateChange === wake) this.stateChange = undefined; + }; + } +} + +const cursor = (timestamp) => ({ format: 1, logs: { local: timestamp } }); +const audit = ({ timestamp, recordId, tableId = 1, version = timestamp, type = 'put', endTxn = true, size = 32 }) => ({ + logName: 'local', + txnLogKey: timestamp, + version, + recordId, + tableId, + type, + endTxn, + size, +}); + +function runtimeFor(store, records, options) { + let reads = 0; + const runtime = new DerivedIndexRuntime( + store, + (tableId, recordId) => { + reads++; + return records.get(`${tableId}:${recordId}`); + }, + { + idleGraceMilliseconds: 5, + scanRecords: (tableId) => + [...records.entries()] + .filter(([key]) => key.startsWith(`${tableId}:`)) + .map(([key, record]) => ({ recordId: key.slice(key.indexOf(':') + 1), ...record })), + ...options, + } + ); + return { runtime, getReads: () => reads }; +} + +const registration = (backend, options) => ({ + backend, + projections: new Map([[1, (record) => ({ title: record.title })]]), + options, +}); + +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +describe('DerivedIndexRuntime for native backends', () => { + const rejections = []; + const onRejection = (reason) => rejections.push(reason); + before(() => process.on('unhandledRejection', onRejection)); + after(() => process.off('unhandledRejection', onRejection)); + afterEach(() => { + assert.deepStrictEqual(rejections, [], 'runtime work must settle without unhandled rejections'); + }); + + it('coalesces repeated keys into one last-write-wins record beside the unchanged transactions', async () => { + const store = new FakeLogStore( + new Map([ + [ + 10, + [ + audit({ timestamp: 20, recordId: 'a', version: 100 }), + audit({ timestamp: 30, recordId: 'b', version: 200 }), + audit({ timestamp: 40, recordId: 'a', version: 300 }), + ], + ], + ]) + ); + const backend = new SyncBackend('coalesce', cursor(10)); + const { runtime, getReads } = runtimeFor( + store, + new Map([ + ['1:a', { version: 300, value: { title: 'a' } }], + ['1:b', { version: 200, value: { title: 'b' } }], + ]) + ); + runtime.register(registration(backend)); + + await waitFor(() => backend.deliveries.length === 1); + const [batch] = backend.deliveries; + assert.deepStrictEqual(batch.through, cursor(40)); + assert.strictEqual(batch.transactions.length, 3); + assert.deepStrictEqual( + batch.records.map(({ recordId, logVersion }) => [recordId, logVersion]), + [ + ['a', 300], + ['b', 200], + ] + ); + assert.strictEqual(batch.records[0].state, batch.transactions[0].mutations[0].state); + assert.strictEqual(batch.records[0].state, batch.transactions[2].mutations[0].state); + assert.strictEqual(batch.transactions[0].mutations[0].logVersion, 100); + assert.strictEqual(getReads(), 2); + assert.deepStrictEqual(Object.keys(batch), ['ownerEpoch', 'transactions', 'through']); + await runtime.stop(); + }); + + it('resolves a key after its last collected occurrence so a concurrent write cannot be certified stale', async () => { + const records = new Map([['1:a', { version: 100, value: { title: 'v1' } }]]); + const store = new FakeLogStore( + new Map([ + [ + 10, + [ + audit({ timestamp: 20, recordId: 'a', version: 100 }), + audit({ timestamp: 30, recordId: 'a', version: 200 }), + ], + ], + ]), + { + onNext: (entry) => { + // Another worker commits a=v2 after the first occurrence has been read. + if (entry?.txnLogKey === 30) records.set('1:a', { version: 200, value: { title: 'v2' } }); + }, + } + ); + const backend = new SyncBackend('ordered', cursor(10)); + const { runtime, getReads } = runtimeFor(store, records); + runtime.register(registration(backend)); + + await waitFor(() => backend.deliveries.length === 1); + const [batch] = backend.deliveries; + assert.deepStrictEqual(batch.through, cursor(30)); + assert.deepStrictEqual(batch.records[0].state, { kind: 'record', version: 200, projection: { title: 'v2' } }); + assert.strictEqual(getReads(), 1); + await runtime.stop(); + }); + + it('delivers an oversized transaction in partial chunks that advance no cursor until it closes', async () => { + const ids = ['a', 'b', 'c', 'd', 'e', 'f', 'g']; + const entries = ids.map((id, index) => audit({ timestamp: 20, recordId: id, endTxn: index === ids.length - 1 })); + entries.push(audit({ timestamp: 30, recordId: 'z' })); + const store = new FakeLogStore(new Map([[10, entries]])); + const records = new Map([...ids, 'z'].map((id) => [`1:${id}`, { version: 20, value: { title: id } }])); + let defer = true; + const backend = new SyncBackend('oversized', cursor(10), (batch, target) => { + if (defer) return DERIVED_INDEX_DEFERRED; + target.cursor = batch.through; + return DERIVED_INDEX_ACCEPTED; + }); + const { runtime } = runtimeFor(store, records, { maxChunkRecords: 3 }); + runtime.register(registration(backend)); + + await waitFor(() => runtime.getStatus('oversized')?.state === 'deferred'); + assert.strictEqual(backend.deliveries.length, 1, 'the backend can defer after the first chunk'); + await sleep(5); + assert(runtime.getMetrics('oversized').stalledMilliseconds > 0, 'a parked runner reports how long it has stalled'); + assert.strictEqual(backend.deliveries[0].records.length, 3); + assert.deepStrictEqual(backend.deliveries[0].through, cursor(10), 'an open transaction advances no cursor'); + assert.strictEqual(runtime.getMetrics('oversized').deferredBytes, 96); + defer = false; + backend.stateChange(); + + await waitFor(() => backend.cursor.logs.local === 30); + const chunks = backend.deliveries.filter((batch, index) => index === 0 || batch !== backend.deliveries[index - 1]); + assert.deepStrictEqual( + chunks.map((batch) => [batch.records.map((record) => record.recordId).join(''), batch.through.logs.local]), + [ + ['abc', 10], + ['def', 10], + ['gz', 30], + ] + ); + assert.deepStrictEqual( + chunks.map((batch) => batch.transactions.map((transaction) => transaction.timestamp)), + [[20], [20], [20, 30]], + 'each chunk carries the transaction it is part of' + ); + await runtime.stop(); + }); + + it('lets a registration override the runtime-wide turn and durability options', async () => { + const store = new FakeLogStore( + new Map([ + [ + 10, + [ + audit({ timestamp: 20, recordId: 'a' }), + audit({ timestamp: 30, recordId: 'b' }), + audit({ timestamp: 40, recordId: 'c' }), + ], + ], + ]) + ); + const records = new Map(['a', 'b', 'c'].map((id) => [`1:${id}`, { version: 40, value: { title: id } }])); + const backend = new SyncBackend('per-registration', cursor(10), () => DERIVED_INDEX_ACCEPTED); + const { runtime } = runtimeFor(store, records, { maxTransactionsPerTurn: 256, maxAcceptedBatchesAhead: 64 }); + runtime.register(registration(backend, { maxTransactionsPerTurn: 1, maxAcceptedBatchesAhead: 2 })); + + await waitFor(() => runtime.getStatus('per-registration')?.state === 'waiting-durable'); + assert.strictEqual(backend.deliveries.length, 2); + assert.deepStrictEqual( + backend.deliveries.map((batch) => batch.through.logs.local), + [20, 30] + ); + await sleep(5); + assert(runtime.getMetrics('per-registration').stalledMilliseconds > 0); + backend.cursor = backend.deliveries[1].through; + backend.stateChange(); + await waitFor(() => backend.deliveries.length === 3); + backend.cursor = backend.deliveries[2].through; + backend.stateChange(); + await waitFor(() => runtime.getStatus('per-registration').state === 'idle'); + assert.strictEqual( + runtime.getMetrics('per-registration').stalledMilliseconds, + 0, + 'leaving the ceiling clears the stall clock' + ); + await runtime.stop(); + }); + + it('requests flushes by threshold, by age, and at shutdown', async () => { + const store = new FakeLogStore( + new Map([ + [ + 10, + [ + audit({ timestamp: 20, recordId: 'a' }), + audit({ timestamp: 30, recordId: 'b' }), + audit({ timestamp: 40, recordId: 'c' }), + ], + ], + ]) + ); + const records = new Map(['a', 'b', 'c'].map((id) => [`1:${id}`, { version: 40, value: { title: id } }])); + const backend = new AsyncBackend('cadence', { cursor: cursor(10) }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register( + registration(backend, { maxTransactionsPerTurn: 1, flushAfterMutations: 2, maxFlushAgeMilliseconds: 20 }) + ); + + await waitFor(() => backend.flushes.includes('threshold')); + await waitFor(() => backend.flushes.includes('age'), { timeout: 2000 }); + await waitFor(() => backend.cursor.logs.local === 40 && runtime.getStatus('cadence').state === 'idle'); + assert.strictEqual(runtime.getMetrics('cadence').acceptedBatches, 0); + const stopped = runtime.stop(); + assert.strictEqual(backend.flushes.at(-1), 'shutdown'); + await stopped; + }); + + it('drives an async-accept backend through a rebuild and publishes ready only after the final barrier', async () => { + const records = new Map([ + ['1:a', { version: 5, value: { title: 'a' } }], + ['1:b', { version: 6, value: { title: 'b' } }], + ]); + // The retained log ends at transaction 8, so the scan covers `c` and the replay resumes after 8. + const store = new FakeLogStore(new Map([[8, [audit({ timestamp: 9, recordId: 'ignored', tableId: 2 })]]]), { + logEntries: new Map([ + ['local', [audit({ timestamp: 7, recordId: 'a' }), audit({ timestamp: 8, recordId: 'c' })]], + ]), + }); + records.set('1:c', { version: 8, value: { title: 'c' } }); + const observed = []; + const backend = new AsyncBackend('rebuild', { + applyDelay: 5, + onReset: () => observed.push(runtime.getReadiness('rebuild').state), + }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register(registration(backend, { maxChunkRecords: 1, maxFlushAgeMilliseconds: 10 })); + + await waitFor(() => runtime.getReadiness('rebuild').state === 'ready', { timeout: 5000 }); + assert.deepStrictEqual(observed, ['rebuilding'], 'rebuilding is published before the destructive reset'); + assert.deepStrictEqual(backend.cursor, cursor(9)); + assert.deepStrictEqual([...backend.applied.keys()].sort(), ['a', 'b', 'c']); + const scanChunks = backend.deliveries.filter((batch) => batch.rebuild); + assert.strictEqual(scanChunks.length, 4, 'one-record chunks plus the boundary chunk'); + assert.deepStrictEqual( + scanChunks.map((batch) => batch.through), + [undefined, undefined, undefined, cursor(8)], + 'the boundary chunk carries the committed tail captured before the scan' + ); + assert.strictEqual(runtime.getMetrics('rebuild').rebuiltRecords, 3); + assert.strictEqual(runtime.getMetrics('rebuild').rebuildAttempts, 0); + await runtime.stop(); + }); + + it('does not publish ready while accepted rebuild work is not yet durable', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const backend = new AsyncBackend('slow-barrier', { applyDelay: 30 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 5 })); + + await waitFor(() => backend.deliveries.some((batch) => batch.through)); + assert.strictEqual(runtime.getReadiness('slow-barrier').state, 'rebuilding'); + assert.strictEqual(backend.cursor, undefined); + await waitFor(() => runtime.getReadiness('slow-barrier').state === 'ready', { timeout: 5000 }); + assert.deepStrictEqual(backend.cursor, cursor(7)); + await runtime.stop(); + }); + + it('orders backend shutdown before lock release and fences the old epoch during handoff', async () => { + const records = new Map([['1:a', { version: 20, value: { title: 'a' } }]]); + const store = new FakeLogStore( + new Map([ + [10, [audit({ timestamp: 20, recordId: 'a' })]], + [20, []], + ]) + ); + let releaseShutdown; + const backend = new AsyncBackend('handoff', { cursor: cursor(10), applyDelay: 50 }); + backend.shutdown = (epoch) => { + backend.shutdowns.push(epoch); + return new Promise((resolve) => (releaseShutdown = resolve)); + }; + const first = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }).runtime; + first.register(registration(backend)); + await waitFor(() => backend.deliveries.length === 1); + const firstEpoch = backend.deliveries[0].ownerEpoch; + assert.strictEqual(backend.host.isOwnerEpoch(firstEpoch), true); + + const stopped = first.stop(); + await waitFor(() => backend.shutdowns.length === 1); + assert.strictEqual(store.locks.size, 1, 'the lock is held until the backend settles its queue'); + + const second = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }).runtime; + second.register(registration(backend)); + await sleep(20); + assert.strictEqual(backend.host.isOwnerEpoch(firstEpoch), true, 'the second owner waits for the lock'); + + backend.cursor = cursor(20); + releaseShutdown(); + await stopped; + backend.shutdown = AsyncBackend.prototype.shutdown; + await waitFor(() => backend.host.isOwnerEpoch(firstEpoch) === false); + assert(backend.deliveries.every((batch) => batch.ownerEpoch === firstEpoch || batch.ownerEpoch > firstEpoch)); + await waitFor(() => second.getStatus('handoff').state === 'idle' && store.locks.size === 1); + await waitFor(() => backend.fenced >= 1, { timeout: 2000 }); + assert.strictEqual(backend.applied.size, 0, 'the old owner apply must not publish into the new generation'); + await second.stop(); + }); + + it('waits for a pending flush before releasing ownership', async () => { + const records = new Map([['1:a', { version: 20, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[10, [audit({ timestamp: 20, recordId: 'a' })]]])); + const backend = new AsyncBackend('flush-pending', { cursor: cursor(10), applyDelay: 30 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 1 })); + + await waitFor(() => backend.pendingFlush !== undefined); + const stopped = runtime.stop(); + assert.strictEqual(store.locks.size, 1); + await stopped; + assert.strictEqual(store.locks.size, 0); + assert.strictEqual(backend.pendingFlush, undefined); + }); + + it('keeps the lock and rejects stop() when the backend cannot prove its queue is quiescent', async () => { + const store = new FakeLogStore(new Map([[10, []]])); + const backend = new AsyncBackend('held', { cursor: cursor(10) }); + backend.shutdown = () => Promise.reject(new Error('native queue did not drain')); + const { runtime } = runtimeFor(store, new Map(), { idleGraceMilliseconds: 1000 }); + runtime.register(registration(backend)); + await waitFor(() => store.locks.size === 1 && runtime.getStatus('held').state === 'idle'); + + await assert.rejects(runtime.stop(), /native queue did not drain/); + assert.strictEqual(store.locks.size, 1); + const shared = readDerivedIndexReadiness(store, 'held'); + assert.strictEqual(shared.state, 'unavailable'); + assert.strictEqual(shared.reason, 'shutdown-failed', 'the backend message stays local; the code is shared'); + }); + + it('revives an index whose lock was held by a failed shutdown once the backend can settle', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const backend = new AsyncBackend('held-revive', { cursor: cursor(7), applyDelay: 2 }); + let settle = false; + backend.shutdown = async (epoch) => { + backend.shutdowns.push(epoch); + if (!settle) throw new Error('native queue did not drain'); + }; + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 5 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => runtime.getStatus('held-revive').state === 'unavailable'); + assert.strictEqual(store.locks.size, 1); + + settle = true; + assert.strictEqual(runtime.requestRebuild('held-revive'), true); + await waitFor(() => runtime.getReadiness('held-revive').state === 'ready', { timeout: 5000 }); + assert.strictEqual(backend.resets.length, 1); + const heldEpoch = backend.shutdowns[0]; + assert.deepStrictEqual( + backend.shutdowns.slice(0, 2), + [heldEpoch, heldEpoch], + 'the held epoch is quiesced again first' + ); + assert(backend.resets[0] > heldEpoch, 'reset runs under a successor epoch only after the held one settled'); + await runtime.stop(); + assert.strictEqual(store.locks.size, 0); + }); + + it('returns one cleanup promise for repeated stop() and waits for every backend', async () => { + const store = new FakeLogStore(new Map([[10, []]])); + const failing = new AsyncBackend('cleanup-failing', { cursor: cursor(10) }); + failing.shutdown = () => Promise.reject(new Error('native queue did not drain')); + const draining = new AsyncBackend('cleanup-draining', { cursor: cursor(10) }); + let releaseDrain; + draining.shutdown = () => new Promise((resolve) => (releaseDrain = resolve)); + const { runtime } = runtimeFor(store, new Map(), { idleGraceMilliseconds: 1000 }); + runtime.register(registration(failing)); + runtime.register(registration(draining)); + await waitFor(() => store.locks.size === 2); + + const first = runtime.stop(); + const second = runtime.stop(); + assert.strictEqual(second, first); + await sleep(20); + assert.strictEqual(store.locks.size, 2, 'stop() must not settle while a backend is still draining'); + releaseDrain(); + await assert.rejects(first, /native queue did not drain/); + assert.strictEqual(store.locks.size, 1, 'the drained backend released; the failed one keeps its lock'); + await assert.rejects(runtime.stop(), /native queue did not drain/); + }); + + it('keeps a failed unregister shutdown in the runtime-wide stop() wait', async () => { + const store = new FakeLogStore(new Map([[10, []]])); + const failing = new AsyncBackend('unregister-failing', { cursor: cursor(10) }); + failing.shutdown = () => Promise.reject(new Error('native queue did not drain')); + const { runtime } = runtimeFor(store, new Map(), { idleGraceMilliseconds: 1000 }); + const unregister = runtime.register(registration(failing)); + await waitFor(() => store.locks.size === 1); + await assert.rejects(unregister(), /native queue did not drain/); + await assert.rejects(runtime.stop(), /native queue did not drain/); + assert.strictEqual(store.locks.size, 1); + }); + + it('delivers a peer rebuild request to an owner parked on backend backpressure', async () => { + const records = new Map([['1:a', { version: 20, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[10, [audit({ timestamp: 20, recordId: 'a' })]]]), { + logEntries: new Map([['local', [audit({ timestamp: 10, recordId: 'a' })]]]), + }); + const ownerBackend = new AsyncBackend('parked', { cursor: cursor(10), capacity: 0, applyDelay: 2 }); + const peerBackend = new AsyncBackend('parked', { cursor: cursor(10) }); + const owner = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + const peer = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + owner.register(registration(ownerBackend, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => owner.getStatus('parked').state === 'deferred'); + peer.register(registration(peerBackend)); + ownerBackend.capacity = Infinity; + assert.strictEqual(peer.requestRebuild('parked'), true); + await waitFor(() => ownerBackend.resets.length === 1, { timeout: 5000 }); + await waitFor(() => owner.getReadiness('parked').state === 'ready', { timeout: 5000 }); + await peer.stop(); + await owner.stop(); + }); + + it('does not run one more attempt for a budget a previous owner already exhausted', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const words = new Int32Array( + store.getUserSharedBuffer('derived-index:inherited:readiness', new ArrayBuffer(READINESS_BYTES)) + ); + Atomics.store(words, 0, 2); // rebuilding + Atomics.store(words, 2, 2); // attempts + const backend = new AsyncBackend('inherited'); + const { runtime } = runtimeFor(store, records); + runtime.register(registration(backend, { maxRebuildAttempts: 2 })); + await waitFor(() => runtime.getStatus('inherited')?.state === 'unavailable'); + assert.strictEqual(backend.resets.length, 0); + await runtime.stop(); + }); + + it('parks a backend that cannot rebuild when a previous owner condemned the generation', async () => { + const store = new FakeLogStore(new Map([[10, []]])); + const words = new Int32Array( + store.getUserSharedBuffer('derived-index:condemned:readiness', new ArrayBuffer(READINESS_BYTES)) + ); + Atomics.store(words, 0, 3); // needs-rebuild + const backend = new SyncBackend('condemned', cursor(10)); + const { runtime } = runtimeFor(store, new Map(), { scanRecords: undefined }); + runtime.register(registration(backend)); + await waitFor(() => runtime.getStatus('condemned')?.state === 'needs-rebuild'); + assert.strictEqual(backend.deliveries.length, 0); + await waitFor(() => store.locks.size === 0); + await runtime.stop(); + }); + + it('waits for an in-flight reset before quiescing and releasing on stop()', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const events = []; + let finishReset; + const backend = new AsyncBackend('reset-race', { applyDelay: 2 }); + backend.reset = (epoch) => { + backend.resets.push(epoch); + events.push('reset-start'); + return new Promise((resolve) => (finishReset = () => (events.push('reset-end'), resolve()))); + }; + backend.shutdown = async (epoch) => { + events.push(`shutdown-${epoch === backend.resets[0] ? 'new' : 'old'}`); + }; + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register(registration(backend)); + await waitFor(() => events.includes('reset-start')); + const stopped = runtime.stop(); + await sleep(10); + assert.strictEqual(store.locks.size, 1, 'the lock is held while the reset is in flight'); + assert.deepStrictEqual(events, ['shutdown-old', 'reset-start']); + finishReset(); + await stopped; + assert.deepStrictEqual(events, ['shutdown-old', 'reset-start', 'reset-end', 'shutdown-new']); + assert.strictEqual(store.locks.size, 0); + }); + + it('lets requestRebuild release a lock held by a stopped runner whose shutdown failed', async () => { + const store = new FakeLogStore(new Map([[10, []]])); + const stuck = new AsyncBackend('held-stopped', { cursor: cursor(10) }); + let settle = false; + stuck.shutdown = async () => { + if (!settle) throw new Error('native queue did not drain'); + }; + const { runtime } = runtimeFor(store, new Map(), { idleGraceMilliseconds: 1000 }); + const unregister = runtime.register(registration(stuck)); + await waitFor(() => store.locks.size === 1); + await assert.rejects(unregister(), /native queue did not drain/); + + const replacement = new AsyncBackend('held-stopped', { cursor: cursor(10), applyDelay: 2 }); + runtime.register(registration(replacement, { maxFlushAgeMilliseconds: 5 })); + await sleep(20); + assert.strictEqual( + runtime.getStatus('held-stopped').ownerEpoch, + undefined, + 'the replacement waits on the held lock' + ); + settle = true; + assert.strictEqual(runtime.requestRebuild('held-stopped'), true); + await waitFor( + () => + runtime.getStatus('held-stopped').ownerEpoch !== undefined && + runtime.getStatus('held-stopped').state === 'idle', + { timeout: 5000 } + ); + assert.strictEqual(replacement.resets.length, 1, 'the request also rebuilds under the new owner'); + await runtime.stop(); + assert.strictEqual(store.locks.size, 0); + }); + + it('clears a latched unavailable status once a peer has revived the index', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const words = new Int32Array( + store.getUserSharedBuffer('derived-index:latched:readiness', new ArrayBuffer(READINESS_BYTES)) + ); + Atomics.store(words, 0, 4); // unavailable + const latched = runtimeFor(store, records, { idleGraceMilliseconds: 5 }).runtime; + latched.register(registration(new AsyncBackend('latched', { applyDelay: 2 }))); + await waitFor(() => latched.getStatus('latched').state === 'unavailable'); + await waitFor(() => store.locks.size === 0); + + const reviver = runtimeFor(store, records, { idleGraceMilliseconds: 5 }).runtime; + const reviverBackend = new AsyncBackend('latched', { applyDelay: 2 }); + reviver.register(registration(reviverBackend, { maxFlushAgeMilliseconds: 5 })); + assert.strictEqual(reviver.requestRebuild('latched'), true); + await waitFor(() => reviver.getReadiness('latched').state === 'ready', { timeout: 5000 }); + await reviver.stop(); + + store.rootStore.emit('committed'); + await waitFor(() => latched.getStatus('latched').state !== 'unavailable' && store.locks.size === 1, { + timeout: 5000, + }); + await latched.stop(); + }); + + it('lets a successor that takes over mid-rebuild replay from its own tail without meeting the marker again', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const reload = { ...audit({ timestamp: 8, type: 'reload' }), recordId: null }; + const store = new FakeLogStore( + new Map([ + [7, [reload, audit({ timestamp: 9, recordId: 'a' })]], + [8, [audit({ timestamp: 9, recordId: 'a' })]], + [9, []], + ]), + { logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' }), reload]]]) } + ); + const first = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + const firstBackend = new AsyncBackend('reload-handoff', { cursor: cursor(7), applyDelay: 2, capacity: 0 }); + first.register(registration(firstBackend, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => firstBackend.resets.length === 1 && firstBackend.deliveries.length >= 1); + // The first owner leaves mid-rebuild; the successor rebuilds the condemned generation itself. + firstBackend.capacity = Infinity; + await first.stop(); + + const second = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + const secondBackend = new AsyncBackend('reload-handoff', { cursor: cursor(7), applyDelay: 2 }); + second.register(registration(secondBackend, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => second.getReadiness('reload-handoff').state === 'ready', { timeout: 5000 }); + assert.strictEqual(secondBackend.resets.length, 1, 'one rebuild for the condemned generation, none for the marker'); + assert.deepStrictEqual(secondBackend.cursor, cursor(9)); + await second.stop(); + }); + + it('keeps tables registered until the backend has settled its shutdown', async () => { + const store = new FakeLogStore(new Map([[10, []]])); + const backend = new AsyncBackend('registered-until-settled', { cursor: cursor(10) }); + let releaseShutdown; + backend.shutdown = () => new Promise((resolve) => (releaseShutdown = resolve)); + const { runtime } = runtimeFor(store, new Map(), { idleGraceMilliseconds: 1000 }); + const unregister = runtime.register(registration(backend)); + await waitFor(() => store.locks.size === 1); + + const unregistered = unregister(); + assert.strictEqual(unregister(), unregistered); + await sleep(10); + assert.strictEqual( + hasDerivedIndexRegistration(store, 1), + true, + 'an eviction during the drain must still write its marker' + ); + releaseShutdown(); + await unregistered; + assert.strictEqual(hasDerivedIndexRegistration(store, 1), false); + await runtime.stop(); + }); + + it('never publishes ready between a fault found mid-drain and the destructive reset', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, [audit({ timestamp: 8, recordId: 'a' })]]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + let breakOnce = true; + store.onNext = () => { + if (breakOnce) { + breakOnce = false; + store.lastIterable.corruptFrameStop.breaks = 1; + } + }; + const originalGetRange = store.getRange.bind(store); + store.getRange = (options) => (store.lastIterable = originalGetRange(options)); + const readinessLog = []; + const backend = new AsyncBackend('mid-drain', { + cursor: cursor(7), + applyDelay: 2, + onReset: () => readinessLog.push(runtime.getReadiness('mid-drain').state), + }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + const seen = new Set(); + const probe = setInterval(() => seen.add(runtime.getReadiness('mid-drain').state), 0); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 5 })); + + await waitFor(() => backend.resets.length === 1); + clearInterval(probe); + assert.deepStrictEqual(readinessLog, ['rebuilding']); + assert.strictEqual(seen.has('ready') && backend.resets.length === 1 && seen.size === 1, false); + await waitFor(() => runtime.getReadiness('mid-drain').state === 'ready', { timeout: 5000 }); + assert.strictEqual(runtime.getMetrics('mid-drain').rebuildAttempts, 0); + await runtime.stop(); + }); + + it('does not leave a rebuild request dangling when one arrives during a rebuild', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore( + new Map([ + [7, []], + [8, []], + ]), + { logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]) } + ); + const backend = new AsyncBackend('dangling', { applyDelay: 2, capacity: 0 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 5 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => runtime.getStatus('dangling').state === 'rebuilding' && backend.deliveries.length >= 1); + assert.strictEqual(runtime.requestRebuild('dangling'), true); + backend.capacity = Infinity; + backend.stateChange('changed'); + await waitFor(() => runtime.getReadiness('dangling').state === 'ready', { timeout: 5000 }); + await waitFor(() => store.locks.size === 0); + + const releasedEpoch = runtime.getStatus('dangling').ownerEpoch; + backend.cursor = cursor(8); + store.rootStore.emit('committed'); + await waitFor( + () => runtime.getStatus('dangling').ownerEpoch > releasedEpoch && runtime.getStatus('dangling').state === 'idle' + ); + assert.strictEqual(backend.resets.length, 1, 'a healthy re-acquisition must not rebuild again'); + await runtime.stop(); + }); + + it('routes a rebuild request from a non-owning worker to a busy owner', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const ownerBackend = new AsyncBackend('routed', { cursor: cursor(7), applyDelay: 2 }); + const peerBackend = new AsyncBackend('routed', { cursor: cursor(7) }); + const owner = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + const peer = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + owner.register(registration(ownerBackend, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => owner.getStatus('routed').state === 'idle' && store.locks.size === 1); + peer.register(registration(peerBackend)); + + assert.strictEqual(peer.requestRebuild('routed'), true); + store.rootStore.emit('committed'); + await waitFor(() => ownerBackend.resets.length === 1, { timeout: 5000 }); + await waitFor(() => owner.getReadiness('routed').state === 'ready', { timeout: 5000 }); + assert.strictEqual(peerBackend.resets.length, 0); + await peer.stop(); + await owner.stop(); + }); + + it('absorbs a peer rebuild request that arrives during a rebuild and never publishes from the peer', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const ownerBackend = new AsyncBackend('absorbed', { applyDelay: 2, capacity: 0 }); + const peerBackend = new AsyncBackend('absorbed'); + const owner = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + const peer = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + owner.register(registration(ownerBackend, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => owner.getStatus('absorbed').state === 'rebuilding' && ownerBackend.deliveries.length >= 1); + peer.register(registration(peerBackend)); + const published = readDerivedIndexReadiness(store, 'absorbed'); + assert.strictEqual(peer.requestRebuild('absorbed'), true); + assert.deepStrictEqual( + readDerivedIndexReadiness(store, 'absorbed'), + published, + 'a non-owner never writes the shared record' + ); + + ownerBackend.capacity = Infinity; + ownerBackend.stateChange('changed'); + await waitFor(() => owner.getReadiness('absorbed').state === 'ready', { timeout: 5000 }); + await sleep(30); + assert.strictEqual(ownerBackend.resets.length, 1, 'the in-flight rebuild absorbs the request'); + assert.strictEqual(owner.getReadiness('absorbed').state, 'ready'); + await peer.stop(); + await owner.stop(); + }); + + it('ignores a superseded flush rejection and keeps asking a backend that coalesced a request', async () => { + const records = new Map([['1:a', { version: 20, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[10, [audit({ timestamp: 20, recordId: 'a' })]]])); + const backend = new AsyncBackend('flush-liveness', { cursor: cursor(10) }); + let rejectSuperseded; + let dropped = 0; + const realFlush = backend.flush.bind(backend); + backend.flush = (reason) => { + if (dropped++ === 0) { + backend.flushes.push(reason); + return new Promise((_resolve, reject) => (rejectSuperseded = reject)); + } + return realFlush(reason); + }; + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 10 })); + + await waitFor(() => backend.cursor.logs.local === 20 && runtime.getStatus('flush-liveness').state === 'idle'); + assert(backend.flushes.length >= 2, 'the age timer re-arms while accepted work is not durable'); + await runtime.stop(); + rejectSuperseded(new Error('cancelled by shutdown')); + await sleep(10); + assert.notStrictEqual(readDerivedIndexReadiness(store, 'flush-liveness').state, 'needs-rebuild'); + }); + + it('rebuilds across several physical logs, omitting a log that has never written a file', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logNames: ['local', 'remote'], + logEntries: new Map([ + ['local', [audit({ timestamp: 7, recordId: 'a' })]], + ['remote', []], + ]), + }); + const backend = new AsyncBackend('multi-log', { applyDelay: 2 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 5 })); + + await waitFor(() => runtime.getReadiness('multi-log').state === 'ready', { timeout: 5000 }); + assert.deepStrictEqual(backend.cursor, { format: 1, logs: { local: 7 } }); + assert.strictEqual(runtime.getMetrics('multi-log').rebuildAttempts, 0); + await runtime.stop(); + }); + + it('does not spend rebuild attempts on reload markers behind the captured tail', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const reloads = [8, 9, 10].map((timestamp) => ({ ...audit({ timestamp, type: 'reload' }), recordId: null })); + const store = new FakeLogStore( + new Map([ + [7, [...reloads, audit({ timestamp: 11, recordId: 'a' })]], + [10, [audit({ timestamp: 11, recordId: 'a' })]], + [11, []], + ]), + { logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' }), ...reloads]]]) } + ); + const backend = new AsyncBackend('reloads', { cursor: cursor(7), applyDelay: 2 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register( + registration(backend, { maxRebuildAttempts: 2, rebuildBackoffMilliseconds: 5, maxFlushAgeMilliseconds: 5 }) + ); + + await waitFor(() => runtime.getReadiness('reloads').state === 'ready', { timeout: 5000 }); + assert.strictEqual(backend.resets.length, 1); + assert.deepStrictEqual(backend.cursor, cursor(11)); + await runtime.stop(); + }); + + it('splits resolution of the first collected transaction at the chunk byte bound', async () => { + const store = new FakeLogStore( + new Map([ + [ + 10, + [ + { ...audit({ timestamp: 20, recordId: 'a' }), endTxn: false }, + { ...audit({ timestamp: 20, recordId: 'b' }), endTxn: false }, + audit({ timestamp: 20, recordId: 'c' }), + ], + ], + ]) + ); + const records = new Map(['a', 'b', 'c'].map((id) => [`1:${id}`, { version: 20, value: { title: id }, size: 100 }])); + const backend = new SyncBackend('split', cursor(10)); + const { runtime } = runtimeFor(store, records, { maxChunkBytes: 150 }); + runtime.register(registration(backend)); + + await waitFor(() => backend.cursor.logs.local === 20); + assert.deepStrictEqual( + backend.deliveries.map((batch) => [ + batch.records.map((record) => record.recordId).join(''), + batch.through.logs.local, + ]), + [ + ['ab', 10], + ['c', 20], + ] + ); + await runtime.stop(); + }); + + it('fails closed on a rejected flush request without an unhandled rejection', async () => { + const store = new FakeLogStore(new Map([[10, [audit({ timestamp: 20, recordId: 'a' })]]])); + const backend = new AsyncBackend('flush-reject', { cursor: cursor(10) }); + backend.flush = () => Promise.reject(new Error('msync failed')); + const { runtime } = runtimeFor(store, new Map([['1:a', { version: 20, value: { title: 'a' } }]]), { + scanRecords: undefined, + }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 1 })); + + await waitFor(() => runtime.getStatus('flush-reject')?.state === 'needs-rebuild'); + // The runner may already have re-acquired and parked on its own condemnation; either way the + // local reason attributes the fault to the backend. + assert.match(runtime.getStatus('flush-reject').reason, /backend flush request rejected|\(backend-failed\)/); + const shared = runtime.getReadiness('flush-reject'); + assert.strictEqual(shared.state, 'needs-rebuild'); + assert.strictEqual(shared.reason, 'backend-failed', 'the backend message never reaches the shared record'); + await runtime.stop(); + }); + + it('exposes the owner-published readiness to a non-owning worker', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const backend = new AsyncBackend('shared-readiness', { applyDelay: 20 }); + const owner = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }).runtime; + const peer = new DerivedIndexRuntime(store, () => undefined); + assert.strictEqual(peer.getReadiness('shared-readiness').state, 'unknown'); + owner.register(registration(backend, { maxFlushAgeMilliseconds: 10 })); + + await waitFor(() => peer.getReadiness('shared-readiness').state === 'rebuilding'); + assert.strictEqual(readDerivedIndexReadiness(store, 'shared-readiness').state, 'rebuilding'); + await waitFor(() => peer.getReadiness('shared-readiness').state === 'ready', { timeout: 5000 }); + assert.strictEqual(peer.getReadiness('shared-readiness').ownerEpoch, backend.deliveries.at(-1).ownerEpoch); + await owner.stop(); + }); + + it('cancels the rebuild-request notification when a runner is unregistered', async () => { + const store = new FakeLogStore(new Map([[10, []]])); + const { runtime } = runtimeFor(store, new Map(), { idleGraceMilliseconds: 1000 }); + for (let cycle = 0; cycle < 3; cycle++) { + const unregister = runtime.register(registration(new SyncBackend('cycled', cursor(10)))); + await waitFor(() => runtime.getStatus('cycled').state === 'idle' && store.locks.size === 1); + await unregister(); + } + assert.strictEqual(store.sharedBuffers.get('derived-index:cycled:readiness').callbacks.size, 0); + await runtime.stop(); + }); + + it('cancels only its own subscription when two runners share a readiness buffer', async () => { + const store = new FakeLogStore(new Map([[10, []]])); + const first = runtimeFor(store, new Map(), { idleGraceMilliseconds: 1000 }).runtime; + const second = runtimeFor(store, new Map(), { idleGraceMilliseconds: 1000 }).runtime; + first.register(registration(new SyncBackend('shared-buffer', cursor(10)))); + second.register(registration(new SyncBackend('shared-buffer', cursor(10)))); + const callbacks = store.sharedBuffers.get('derived-index:shared-buffer:readiness').callbacks; + assert.strictEqual(callbacks.size, 2); + await first.stop(); + assert.strictEqual(callbacks.size, 1); + await second.stop(); + assert.strictEqual(callbacks.size, 0); + }); + + it('trips the opt-in lag policy on every worker while the index is behind and clears it with hysteresis', async () => { + const records = new Map(['a', 'b', 'c'].map((id) => [`1:${id}`, { version: 1, value: { title: id } }])); + const store = new FakeLogStore( + new Map([ + [ + 10, + [ + audit({ timestamp: 1_000, recordId: 'a' }), + audit({ timestamp: 2_000, recordId: 'b' }), + audit({ timestamp: 3_000, recordId: 'c' }), + ], + ], + ]) + ); + const backend = new SyncBackend('lagging', cursor(10), () => DERIVED_INDEX_ACCEPTED); + const owner = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + const peer = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + assert.strictEqual(derivedIndexWriteRejection(store, 1), undefined, 'nothing registered: writes admitted'); + owner.register(registration(backend, { maxLagMilliseconds: 500, maxFlushAgeMilliseconds: 5 })); + peer.register(registration(new SyncBackend('lagging', cursor(10)), { maxLagMilliseconds: 500 })); + + await waitFor(() => derivedIndexWriteRejection(store, 1) !== undefined); + assert.match(derivedIndexWriteRejection(store, 1), /'lagging' is more than 500 ms behind/); + assert.match(derivedIndexWriteRejection(store, 2) ?? '', /^$/, 'tables without a derived index are never gated'); + + backend.cursor = cursor(3_000); + backend.stateChange(); + await waitFor(() => derivedIndexWriteRejection(store, 1) === undefined, { timeout: 5000 }); + await owner.stop(); + await peer.stop(); + assert.strictEqual(derivedIndexWriteRejection(store, 1), undefined, 'unregistering removes the admission check'); + }); + + it('publishes ready on the first durable advance even when the runner never idles', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, [audit({ timestamp: 8, recordId: 'a' })]]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + live: true, + }); + let next = 9; + store.onNext = (entry) => { + // Keep the log ahead of the runner so no idle pass ever happens. + if (entry === undefined && next < 2000) + store.entriesByCursor.get(7).push(audit({ timestamp: next++, recordId: 'a' })); + }; + const backend = new AsyncBackend('never-idle', { applyDelay: 1 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 5, maxTransactionsPerTurn: 1 })); + await waitFor(() => runtime.getReadiness('never-idle').state === 'ready', { timeout: 5000 }); + assert.notStrictEqual(runtime.getStatus('never-idle').state, 'idle'); + assert.strictEqual(runtime.getMetrics('never-idle').rebuildAttempts, 0); + await runtime.stop(); + }); + + it('skips and counts every record of a chunk the projection rejects instead of condemning the index', async () => { + const ids = Array.from({ length: 40 }, (_, i) => `r${i}`); + const store = new FakeLogStore(new Map([[10, ids.map((id, i) => audit({ timestamp: 20 + i, recordId: id }))]])); + const backend = new SyncBackend('all-rejected', cursor(10)); + const { runtime } = runtimeFor(store, new Map(ids.map((id) => [`1:${id}`, { version: 1, value: { title: 1 } }])), { + scanRecords: undefined, + }); + runtime.register({ + backend, + projections: new Map([ + [ + 1, + () => { + throw new ClientError('title must be a string', 400); + }, + ], + ]), + }); + await waitFor(() => backend.deliveries.length > 0); + const states = backend.deliveries.flatMap((batch) => batch.records.map((record) => record.state)); + assert.strictEqual(states.length, 40); + assert(states.every((state) => state.kind === 'unindexable' && /\(400\)$/.test(state.reason))); + assert.strictEqual(runtime.getMetrics('all-rejected').unindexableRecords, 40); + await waitFor(() => backend.cursor.logs.local === 59); + assert.strictEqual(runtime.getStatus('all-rejected').state, 'idle'); + assert.strictEqual(runtime.getMetrics('all-rejected').rebuildAttempts, 0); + await runtime.stop(); + }); + + it('admits writes again when the index becomes unavailable with no owner left to catch up', async () => { + const records = new Map([['1:a', { version: 8, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, [audit({ timestamp: 8, recordId: 'a' })]]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const backend = new AsyncBackend('shed-then-dead', { cursor: cursor(7) }); + // Accept but never make anything durable, so catch-up is never proven and the policy trips. + backend.flush = () => {}; + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register( + registration(backend, { + maxLagMilliseconds: 20, + maxFlushAgeMilliseconds: 5, + maxRebuildAttempts: 1, + rebuildBackoffMilliseconds: 5, + }) + ); + await waitFor(() => derivedIndexWriteRejection(store, 1) !== undefined, { timeout: 5000 }); + backend.applyRecord = () => { + throw new Error('native capacity exhausted'); + }; + backend.stateChange('failed'); + await waitFor(() => runtime.getStatus('shed-then-dead')?.state === 'unavailable', { timeout: 5000 }); + assert.strictEqual(derivedIndexWriteRejection(store, 1), undefined, 'an unavailable index must not shed forever'); + await runtime.stop(); + }); + + it('admits writes again when a lagging index parks in needs-rebuild it can never leave', async () => { + const records = new Map([['1:a', { version: 8, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[10, [audit({ timestamp: 11, recordId: 'a' })]]]), { + logEntries: new Map([['local', [audit({ timestamp: 10, recordId: 'a' })]]]), + }); + // No reset(): the runtime cannot rebuild this backend, so needs-rebuild is a terminal park. + const backend = new SyncBackend('parked-lagging', cursor(10), () => DERIVED_INDEX_ACCEPTED); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register(registration(backend, { maxLagMilliseconds: 20, maxFlushAgeMilliseconds: 5 })); + await waitFor(() => derivedIndexWriteRejection(store, 1) !== undefined, { timeout: 5000 }); + backend.stateChange('failed'); + await waitFor(() => runtime.getStatus('parked-lagging')?.state === 'needs-rebuild', { timeout: 5000 }); + assert.strictEqual( + derivedIndexWriteRejection(store, 1), + undefined, + 'an index parked in needs-rebuild with no way to rebuild must not shed writes forever' + ); + await runtime.stop(); + }); + + it('arms one retry timer while tryLock keeps throwing under a commit stream', async () => { + const store = new FakeLogStore(new Map([[10, [audit({ timestamp: 20, recordId: 'a' })]]])); + let attempts = 0; + const tryLock = store.tryLock.bind(store); + store.tryLock = (key, onUnlocked) => { + if (attempts++ < 2) throw new Error('lock table busy'); + return tryLock(key, onUnlocked); + }; + const backend = new SyncBackend('lock-storm', cursor(10)); + const { runtime } = runtimeFor(store, new Map([['1:a', { version: 20, value: { title: 'a' } }]])); + runtime.register(registration(backend, { rebuildBackoffMilliseconds: 200 })); + await waitFor(() => attempts === 1); + for (let i = 0; i < 20; i++) { + store.rootStore.emit('committed'); + await sleep(1); + } + assert.strictEqual(attempts, 1, 'commit wakes do not re-enter tryLock while the retry timer is armed'); + await waitFor(() => backend.deliveries.length === 1, { timeout: 5000 }); + assert.strictEqual(attempts, 3, 'one attempt per timer firing'); + await runtime.stop(); + }); + + it('rebuilds after a restart when a condemnation was recorded before the reset could invalidate the cursor', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const logEntries = new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]); + const first = new FakeLogStore(new Map([[7, []]]), { logEntries }); + const condemned = new AsyncBackend('condemn-restart', { cursor: cursor(7), applyDelay: 1 }); + condemned.reset = undefined; + const before = runtimeFor(first, records, { idleGraceMilliseconds: 60_000 }).runtime; + before.register(registration(condemned, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => before.getReadiness('condemn-restart').state === 'ready'); + condemned.stateChange('failed'); + await waitFor(() => before.getStatus('condemn-restart')?.state === 'needs-rebuild'); + assert.strictEqual(first.markers.size, 1, 'the condemnation is written to the root store'); + await before.stop(); + + // "Restart": fresh shared buffers (readiness reads unknown), same root store markers and log. + const restarted = new FakeLogStore(new Map([[7, []]]), { logEntries, markers: first.markers }); + const rebuilt = new AsyncBackend('condemn-restart', { cursor: cursor(7), applyDelay: 1 }); + const after = runtimeFor(restarted, records, { idleGraceMilliseconds: 60_000 }).runtime; + assert.strictEqual(after.getReadiness('condemn-restart').state, 'unknown'); + after.register(registration(rebuilt, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => after.getReadiness('condemn-restart').state === 'ready', { timeout: 5000 }); + assert.strictEqual(rebuilt.resets.length, 1, 'the still-valid cursor is rebuilt, not trusted'); + assert.strictEqual(restarted.markers.size, 0, 'the marker clears at the durable ready'); + await after.stop(); + + const again = new FakeLogStore(new Map([[7, []]]), { logEntries, markers: first.markers }); + const trusted = new AsyncBackend('condemn-restart', { cursor: cursor(7), applyDelay: 1 }); + const third = runtimeFor(again, records, { idleGraceMilliseconds: 60_000 }).runtime; + third.register(registration(trusted, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => third.getReadiness('condemn-restart').state === 'ready', { timeout: 5000 }); + assert.strictEqual(trusted.resets.length, 0, 'a cleared marker lets the cursor be trusted'); + await third.stop(); + }); + + it('does not reset the backend when the condemnation marker cannot be persisted', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + store.putSync = () => { + throw new Error('root store is read-only'); + }; + const backend = new AsyncBackend('marker-fails', { cursor: cursor(7), applyDelay: 1 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => runtime.getReadiness('marker-fails').state === 'ready'); + backend.stateChange('failed'); + await waitFor(() => runtime.getStatus('marker-fails')?.state === 'needs-rebuild' && store.locks.size === 0); + assert.strictEqual(backend.resets.length, 0, 'no destructive reset without a durable condemnation'); + assert.strictEqual(runtime.getReadiness('marker-fails').state, 'needs-rebuild'); + store.rootStore.emit('committed'); + await sleep(30); + assert.strictEqual(backend.resets.length, 0, 'a wake retries the marker and still issues no reset'); + assert.strictEqual( + runtime.getMetrics('marker-fails').rebuildAttempts, + 0, + 'no attempt is spent on a refused marker' + ); + // The store recovers: the next wake writes the marker, and only then does the rebuild run. + delete store.putSync; + store.rootStore.emit('committed'); + await waitFor(() => runtime.getReadiness('marker-fails').state === 'ready', { timeout: 5000 }); + assert.strictEqual(store.markers.size, 0, 'written, then cleared at the durable ready'); + assert.strictEqual(backend.resets.length, 1); + await runtime.stop(); + }); + + it('trips the lag policy for a reader too slow to reach the end of the log, even with a caught-up backend', async () => { + const entries = Array.from({ length: 400 }, (_, i) => audit({ timestamp: 11 + i, recordId: `r${i}` })); + const records = new Map(entries.map((entry, i) => [`1:r${i}`, { version: 11 + i, value: { title: 'x' } }])); + const store = new FakeLogStore(new Map([[10, entries]]), { + onNext: (entry) => { + if (!entry) return; + const until = performance.now() + 1; + while (performance.now() < until); + }, + }); + const backend = new SyncBackend('slow-reader', cursor(10)); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register( + registration(backend, { + maxLagMilliseconds: 100, + maxFlushAgeMilliseconds: 5, + maxMillisecondsPerTurn: 2, + maxChunkRecords: 4, + }) + ); + await waitFor(() => derivedIndexWriteRejection(store, 1) !== undefined, { timeout: 5000 }); + assert.strictEqual( + runtime.getMetrics('slow-reader').cursorLagMilliseconds, + 0, + 'durable trails what was read by nothing' + ); + await waitFor(() => derivedIndexWriteRejection(store, 1) === undefined, { timeout: 10_000 }); + assert.strictEqual(backend.cursor.logs.local, 410); + await runtime.stop(); + }); + + it('keeps an inherited lag trip until the successor has drained the backlog, not just advanced once', async () => { + const entries = Array.from({ length: 300 }, (_, i) => audit({ timestamp: 11 + i, recordId: `r${i}` })); + const records = new Map(entries.map((entry, i) => [`1:r${i}`, { version: 11 + i, value: { title: 'x' } }])); + const store = new FakeLogStore(new Map([[10, entries]]), { + onNext: (entry) => { + if (!entry) return; + const until = performance.now() + 1; + while (performance.now() < until); + }, + }); + const first = new SyncBackend('inherited', cursor(10), () => DERIVED_INDEX_ACCEPTED); + const owner = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + owner.register(registration(first, { maxLagMilliseconds: 100, maxFlushAgeMilliseconds: 5, maxChunkRecords: 4 })); + await waitFor(() => derivedIndexWriteRejection(store, 1) !== undefined, { timeout: 5000 }); + await owner.stop(); + + const second = new SyncBackend('inherited', cursor(10)); + const successor = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }).runtime; + successor.register( + registration(second, { + maxLagMilliseconds: 100, + maxFlushAgeMilliseconds: 5, + maxMillisecondsPerTurn: 2, + maxChunkRecords: 4, + }) + ); + assert.notStrictEqual(derivedIndexWriteRejection(store, 1), undefined, 'the trip survives the handoff'); + await waitFor(() => second.deliveries.length >= 3); + assert.notStrictEqual(derivedIndexWriteRejection(store, 1), undefined, 'durable advances alone do not clear it'); + await waitFor(() => derivedIndexWriteRejection(store, 1) === undefined, { timeout: 10_000 }); + assert.strictEqual(second.cursor.logs.local, 310, 'cleared only once the end of the log was reached'); + await successor.stop(); + }); + + it('parks a backend that cannot rebuild on a refused condemnation and retries the marker without a reset', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const backend = new AsyncBackend('marker-fails-no-reset', { cursor: cursor(7), applyDelay: 1 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 5 })); + await waitFor(() => runtime.getReadiness('marker-fails-no-reset').state === 'ready'); + backend.reset = undefined; + store.putSync = () => { + throw new Error('root store is read-only'); + }; + backend.stateChange('failed'); + await waitFor( + () => runtime.getStatus('marker-fails-no-reset')?.state === 'needs-rebuild' && store.locks.size === 0 + ); + store.rootStore.emit('committed'); + await sleep(30); + assert.strictEqual(runtime.getStatus('marker-fails-no-reset').state, 'needs-rebuild'); + assert.strictEqual(store.markers.size, 0); + delete store.putSync; + store.rootStore.emit('committed'); + await waitFor(() => store.markers.size === 1, { timeout: 5000 }); + assert.strictEqual( + runtime.getStatus('marker-fails-no-reset').state, + 'needs-rebuild', + 'still parked: it cannot rebuild' + ); + assert.strictEqual(runtime.getReadiness('marker-fails-no-reset').state, 'needs-rebuild'); + let lockAttempts = 0; + const tryLock = store.tryLock.bind(store); + store.tryLock = (key, onUnlocked) => (lockAttempts++, tryLock(key, onUnlocked)); + store.rootStore.emit('committed'); + await sleep(20); + assert.strictEqual(lockAttempts, 0, 'a parked runner whose marker is written stays parked'); + await runtime.stop(); + }); + + it('keeps proving catch-up mid-stream so a backend that keeps up under sustained ingest never trips', async () => { + const entries = []; + const records = new Map(); + const store = new FakeLogStore(new Map([[10, entries]]), { live: true }); + const backend = new AsyncBackend('sustained', { cursor: cursor(10), applyDelay: 0 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register(registration(backend, { maxLagMilliseconds: 60, maxFlushAgeMilliseconds: 5 })); + await waitFor(() => runtime.getStatus('sustained')?.state === 'idle'); + let next = 11; + const started = Date.now(); + while (Date.now() - started < 300) { + for (let i = 0; i < 5; i++) { + const id = `r${next}`; + records.set(`1:${id}`, { version: next, value: { title: id } }); + entries.push(audit({ timestamp: next++, recordId: id })); + } + store.rootStore.emit('committed'); + await sleep(2); + assert.strictEqual(derivedIndexWriteRejection(store, 1), undefined, 'a backend that keeps up is never shed'); + } + assert(backend.flushes.length > 5, 'catch-up was proven at barriers, not at an idle pass'); + await runtime.stop(); + }); + + it('retries the lock instead of parking when tryLock throws once', async () => { + const store = new FakeLogStore(new Map([[10, [audit({ timestamp: 20, recordId: 'a' })]]])); + let attempts = 0; + const tryLock = store.tryLock.bind(store); + store.tryLock = (key, onUnlocked) => { + if (attempts++ === 0) throw new Error('lock table busy'); + return tryLock(key, onUnlocked); + }; + const backend = new SyncBackend('lock-throw', cursor(10)); + const { runtime } = runtimeFor(store, new Map([['1:a', { version: 20, value: { title: 'a' } }]])); + runtime.register(registration(backend, { rebuildBackoffMilliseconds: 5 })); + await waitFor(() => backend.deliveries.length === 1); + assert.strictEqual(attempts, 2); + await runtime.stop(); + }); + + it('completes a rebuild whose scan the projection rejects entirely, counting the records', async () => { + const ids = Array.from({ length: 5 }, (_, i) => `r${i}`); + const records = new Map(ids.map((id) => [`1:${id}`, { version: 1, value: { title: 1 } }])); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'r0' })]]]), + }); + const backend = new AsyncBackend('scan-rejected', { applyDelay: 1 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register({ + backend, + projections: new Map([ + [ + 1, + () => { + throw new ClientError('title must be a string', 400); + }, + ], + ]), + options: { maxChunkRecords: 2, maxRebuildAttempts: 1, maxFlushAgeMilliseconds: 5 }, + }); + await waitFor(() => runtime.getReadiness('scan-rejected').state === 'ready', { timeout: 5000 }); + assert.strictEqual(runtime.getMetrics('scan-rejected').unindexableRecords, 5); + assert.strictEqual(runtime.getMetrics('scan-rejected').rebuiltRecords, 5); + assert.strictEqual(runtime.getMetrics('scan-rejected').rebuildAttempts, 0); + assert.strictEqual(backend.applied.size, 0); + await runtime.stop(); + }); + + it('yields the event loop through a long run of tombstones during a rebuild scan', async () => { + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'kept' })]]]), + }); + const backend = new AsyncBackend('tombstone-scan', { applyDelay: 1 }); + let clock = 0; + let yielded = false; + let yieldedBefore = -1; + const { runtime } = runtimeFor(store, new Map(), { + idleGraceMilliseconds: 60_000, + now: () => clock, + scanRecords: function* () { + setImmediate(() => (yielded = true)); + for (let i = 0; i < 2000; i++) { + if (yielded && yieldedBefore < 0) yieldedBefore = i; + clock += 1; + yield { recordId: `dead${i}`, version: 1, value: null }; + } + yield { recordId: 'kept', version: 2, value: { title: 'kept' } }; + }, + }); + runtime.register(registration(backend, { maxMillisecondsPerTurn: 5, maxFlushAgeMilliseconds: 5 })); + await waitFor(() => runtime.getReadiness('tombstone-scan').state === 'ready', { timeout: 5000 }); + assert(yieldedBefore >= 0 && yieldedBefore < 2000, `scan held the event loop through ${yieldedBefore} tombstones`); + assert.strictEqual(runtime.getMetrics('tombstone-scan').rebuiltRecords, 1); + assert.deepStrictEqual([...backend.applied.keys()], ['kept']); + const scanChunks = backend.deliveries.filter((batch) => batch.rebuild); + assert( + scanChunks.every((batch) => batch.records.length > 0 || batch.through), + 'no empty chunk before the boundary' + ); + assert.deepStrictEqual(scanChunks.at(-1).through, cursor(7)); + const lookups = store.bufferLookups; + await runtime.stop(); + assert.strictEqual(store.bufferLookups, lookups, 'shared-memory views are fetched once per runner'); + }); + + it('does not trip the lag policy for a caught-up owner that idles past the budget', async () => { + const records = new Map([['1:a', { version: 1, value: { title: 'a' } }]]); + const entries = []; + const store = new FakeLogStore(new Map([[10, entries]]), { live: true }); + const backend = new SyncBackend('idle-caught-up', cursor(10), () => DERIVED_INDEX_ACCEPTED); + let clock = 1_000_000; + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000, now: () => clock }); + runtime.register(registration(backend, { maxLagMilliseconds: 400, maxFlushAgeMilliseconds: 5 })); + await waitFor(() => runtime.getStatus('idle-caught-up')?.state === 'idle'); + clock += 5_000; + await sleep(350); + assert.strictEqual(derivedIndexWriteRejection(store, 1), undefined, 'a caught-up idle owner has no lag'); + entries.push(audit({ timestamp: 11, recordId: 'a' })); + backend.deliverImpl = () => DERIVED_INDEX_DEFERRED; + store.rootStore.emit('committed'); + await waitFor(() => runtime.getStatus('idle-caught-up').state === 'deferred'); + clock += 1_000; + await waitFor(() => derivedIndexWriteRejection(store, 1) !== undefined, { timeout: 5000 }); + await runtime.stop(); + }); + + it('keeps writes admitted when the registration sets no lag policy', async () => { + const records = new Map([['1:a', { version: 1, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[10, [audit({ timestamp: 5_000_000, recordId: 'a' })]]])); + const backend = new SyncBackend('unbounded-lag', cursor(10), () => DERIVED_INDEX_ACCEPTED); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register(registration(backend)); + await waitFor(() => backend.deliveries.length === 1); + await sleep(10); + assert.strictEqual(derivedIndexWriteRejection(store, 1), undefined); + await runtime.stop(); + }); + + it('rejects a backend that lacks the fence, barrier or quiescence hooks', () => { + const store = new FakeLogStore(new Map([[10, []]])); + const { runtime } = runtimeFor(store, new Map()); + for (const hook of ['attach', 'flush', 'shutdown']) { + const incomplete = new SyncBackend(`incomplete-${hook}`, cursor(10)); + incomplete[hook] = undefined; + assert.throws(() => runtime.register(registration(incomplete)), new RegExp(`must implement ${hook}\\(\\)`)); + assert.strictEqual(runtime.getStatus(incomplete.id), undefined); + } + }); + + it('waits for the shutdown flush of an asynchronous backend before releasing the lock', async () => { + const records = new Map([['1:a', { version: 20, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[10, [audit({ timestamp: 20, recordId: 'a' })]]])); + const backend = new AsyncBackend('flush-before-unlock', { cursor: cursor(10), applyDelay: 2 }); + let finishFlush; + const events = []; + backend.flush = (reason) => { + backend.flushes.push(reason); + if (reason !== 'shutdown') return; + return new Promise((resolve) => (finishFlush = () => (events.push('flush-done'), resolve()))); + }; + backend.shutdown = async () => { + events.push('shutdown'); + }; + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 60_000 }); + runtime.register(registration(backend, { maxFlushAgeMilliseconds: 1000 })); + await waitFor(() => backend.deliveries.length === 1); + const stopped = runtime.stop(); + await sleep(10); + assert.deepStrictEqual(events, []); + assert.strictEqual(store.locks.size, 1); + finishFlush(); + await stopped; + assert.deepStrictEqual(events, ['flush-done', 'shutdown']); + assert.strictEqual(store.locks.size, 0); + }); + + it('leaves no readiness subscription or table admission behind when registration fails', () => { + const store = new FakeLogStore(new Map([[10, []]])); + const { runtime } = runtimeFor(store, new Map()); + const throwing = new SyncBackend('partial-registration', cursor(10)); + throwing.onStateChange = () => { + throw new Error('subscribe failed'); + }; + assert.throws(() => runtime.register(registration(throwing, { maxLagMilliseconds: 50 })), /subscribe failed/); + assert.strictEqual(store.sharedBuffers.get('derived-index:partial-registration:readiness').callbacks.size, 0); + assert.strictEqual(hasDerivedIndexRegistration(store, 1), false); + }); + + it('skips and counts a record the projection rejects instead of rebuilding', async () => { + const store = new FakeLogStore( + new Map([[10, [audit({ timestamp: 20, recordId: 'bad' }), audit({ timestamp: 30, recordId: 'good' })]]]) + ); + const backend = new SyncBackend('unindexable', cursor(10)); + const { runtime } = runtimeFor( + store, + new Map([ + ['1:bad', { version: 20, value: { title: 7 } }], + ['1:good', { version: 30, value: { title: 'good' } }], + ]) + ); + runtime.register({ + backend, + projections: new Map([ + [ + 1, + (record) => { + if (typeof record.title !== 'string') throw new ClientError('title must be a string', 400); + return { title: record.title }; + }, + ], + ]), + }); + + await waitFor(() => backend.deliveries.length === 1); + assert.deepStrictEqual(backend.deliveries[0].records[0].state, { + kind: 'unindexable', + version: 20, + reason: 'Error (400)', + }); + assert.strictEqual(backend.deliveries[0].records[1].state.kind, 'record'); + assert.strictEqual(runtime.getMetrics('unindexable').unindexableRecords, 1); + assert.deepStrictEqual(backend.cursor, cursor(30)); + await runtime.stop(); + }); + + it('settles a backend that fails every rebuild into an observable unavailable state', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + const backend = new AsyncBackend('exhausted', { + applyRecord: () => { + throw new Error('native capacity exhausted'); + }, + }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register( + registration(backend, { maxRebuildAttempts: 3, rebuildBackoffMilliseconds: 5, maxFlushAgeMilliseconds: 5 }) + ); + + await waitFor(() => runtime.getStatus('exhausted')?.state === 'unavailable', { timeout: 5000 }); + assert.strictEqual(backend.resets.length, 3); + await waitFor(() => store.locks.size === 0, { message: 'an unavailable index releases the lock' }); + const readiness = readDerivedIndexReadiness(store, 'exhausted'); + assert.strictEqual(readiness.state, 'unavailable'); + assert.strictEqual(readiness.rebuildAttempts, 3); + assert.strictEqual(readiness.reason, 'backend-failed'); + + // A peer worker (its own backend instance) honours the shared budget instead of starting its own attempts. + const peerBackend = new AsyncBackend('exhausted'); + const peer = runtimeFor(store, records).runtime; + peer.register(registration(peerBackend)); + await waitFor(() => peer.getStatus('exhausted')?.state === 'unavailable'); + assert.strictEqual(peerBackend.resets.length, 0); + await peer.stop(); + + backend.applyRecord = undefined; + assert.strictEqual(runtime.requestRebuild('exhausted'), true); + await waitFor(() => runtime.getReadiness('exhausted').state === 'ready', { timeout: 5000 }); + assert.deepStrictEqual(backend.cursor, cursor(7)); + await runtime.stop(); + }); + + it('retries a rebuild whose boundary was lost to retention during the scan', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const store = new FakeLogStore(new Map([[7, []]]), { + logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' })]]]), + }); + store.exactStartFailures.set('local', 'missing'); + const backend = new AsyncBackend('retention', { applyDelay: 2 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register(registration(backend, { rebuildBackoffMilliseconds: 10, maxFlushAgeMilliseconds: 5 })); + + await waitFor(() => runtime.getStatus('retention')?.state === 'needs-rebuild'); + assert.match(runtime.getStatus('retention').reason, /missing durable cursor boundary/); + assert.strictEqual(readDerivedIndexReadiness(store, 'retention').rebuildAttempts, 1); + store.exactStartFailures.clear(); + await waitFor(() => runtime.getReadiness('retention').state === 'ready', { timeout: 5000 }); + assert.strictEqual(backend.resets.length, 2); + await runtime.stop(); + }); + + it('meets the reload marker that triggered a rebuild once, because the replay resumes after the tail', async () => { + const records = new Map([['1:a', { version: 5, value: { title: 'a' } }]]); + const reload = { ...audit({ timestamp: 8, recordId: undefined, type: 'reload' }), recordId: null }; + const store = new FakeLogStore( + new Map([ + [7, [reload, audit({ timestamp: 9, recordId: 'a' })]], + [8, [audit({ timestamp: 9, recordId: 'a' })]], + [9, []], + ]), + { logEntries: new Map([['local', [audit({ timestamp: 7, recordId: 'a' }), reload]]]) } + ); + const backend = new AsyncBackend('reload', { cursor: cursor(7), applyDelay: 2 }); + const { runtime } = runtimeFor(store, records, { idleGraceMilliseconds: 1000 }); + runtime.register(registration(backend, { rebuildBackoffMilliseconds: 5, maxFlushAgeMilliseconds: 5 })); + + await waitFor(() => runtime.getReadiness('reload').state === 'ready', { timeout: 5000 }); + assert.strictEqual(backend.resets.length, 1); + assert.deepStrictEqual(backend.cursor, cursor(9)); + await runtime.stop(); + }); +}); + +describe('DerivedIndexRuntime rebuild against an audited RocksDB table', () => { + if (process.env.HARPER_STORAGE_ENGINE === 'lmdb') return; + let runtime; + before(() => { + setupTestDBPath(); + require('#js/server/threads/manageThreads').setMainIsWorker(true); + }); + after(() => runtime?.stop()); + + it('sheds user writes end to end once a real runner exceeds its lag budget, and admits them after catch-up', async () => { + const { table } = require('#src/resources/databases'); + const Gated = table({ + database: 'derived-index-lag-rocks', + table: 'Gated', + audit: true, + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'title' }], + }); + await Gated.put('seed', { title: 'seed' }); + const anchor = [...Gated.auditStore.getRange({ start: 1 })] + .filter((entry) => entry.tableId === Gated.tableId) + .at(-1).txnLogKey; + const backend = new SyncBackend('gated', { format: 1, logs: { local: anchor } }, () => DERIVED_INDEX_ACCEPTED); + const gatedRuntime = new DerivedIndexRuntime(Gated.auditStore, (tableId, recordId) => { + const entry = Gated.primaryStore.getEntry(recordId); + return entry?.value ? { version: entry.version, value: entry.value } : undefined; + }); + gatedRuntime.register({ + backend, + projections: new Map([[Gated.tableId, (record) => ({ title: record.title })]]), + options: { maxLagMilliseconds: 40, maxFlushAgeMilliseconds: 5 }, + }); + await Gated.put('g1', { title: 'first' }); + await waitFor(() => backend.deliveries.length >= 1); + // The backend accepts but never makes anything durable, so catch-up is never proven. + await waitFor(() => derivedIndexWriteRejection(Gated.auditStore, Gated.tableId) !== undefined, { timeout: 5000 }); + let rejected; + try { + await Gated.put('g2', { title: 'blocked' }); + } catch (error) { + rejected = error; + } + assert.strictEqual(rejected?.code, 'DERIVED_INDEX_LAGGING'); + // A canonical-source apply is never shed: dropping it would advance the source cursor past it. + const { transaction } = require('#src/resources/transaction'); + const canonical = { sourceApply: true }; + await transaction(canonical, () => Gated.put('g-source', { title: 'canonical' }, canonical)); + assert.strictEqual((await Gated.get('g-source'))?.title, 'canonical'); + assert.notStrictEqual(derivedIndexWriteRejection(Gated.auditStore, Gated.tableId), undefined, 'still tripped'); + await waitFor(() => backend.deliveries.length >= 2, { timeout: 5000 }); + + backend.cursor = backend.deliveries.at(-1).through; + backend.stateChange(); + await waitFor(() => derivedIndexWriteRejection(Gated.auditStore, Gated.tableId) === undefined, { timeout: 5000 }); + await Gated.put('g2', { title: 'admitted' }); + await gatedRuntime.stop(); + }); + + it('rebuilds from the primary store on the retained log boundary and replays to the head', async () => { + const { table } = require('#src/resources/databases'); + const Product = table({ + database: 'derived-index-rebuild-rocks', + table: 'Product', + audit: true, + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'title' }], + }); + for (const id of ['p1', 'p2', 'p3']) await Product.put(id, { title: `title ${id}` }); + await Product.delete('p2'); + + const backend = new AsyncBackend('rocks-rebuild', { applyDelay: 2 }); + runtime = new DerivedIndexRuntime( + Product.auditStore, + (tableId, recordId) => { + const entry = Product.primaryStore.getEntry(recordId); + return entry?.value ? { version: entry.version, value: entry.value } : undefined; + }, + { + idleGraceMilliseconds: 60_000, + scanRecords: () => + Product.primaryStore.getRange({ versions: true }).map((entry) => ({ + recordId: entry.key, + version: entry.version, + value: entry.value, + })), + } + ); + runtime.register({ + backend, + projections: new Map([[Product.tableId, (record) => ({ title: record.title })]]), + options: { maxFlushAgeMilliseconds: 10, maxChunkRecords: 2 }, + }); + + await waitFor(() => runtime.getReadiness('rocks-rebuild').state === 'ready', { timeout: 10_000 }); + assert.deepStrictEqual([...backend.applied.keys()].sort(), ['p1', 'p3']); + let tail; + for (const entry of Product.auditStore.getRange({ log: 'local', start: 0 })) + if (entry.endTxn) tail = entry.txnLogKey; + const scanChunks = backend.deliveries.filter((batch) => batch.rebuild); + assert.strictEqual(scanChunks.at(-1).through.logs.local, tail, 'the boundary is the committed tail at capture'); + assert.deepStrictEqual(backend.cursor, { format: 1, logs: { local: tail } }, 'nothing to replay after the tail'); + + await Product.put('p4', { title: 'title p4' }); + await waitFor(() => backend.applied.has('p4'), { timeout: 5000 }); + assert.deepStrictEqual(backend.applied.get('p4').projection, { title: 'title p4' }); + assert.strictEqual(runtime.getMetrics('rocks-rebuild').rebuildAttempts, 0); + + // The write path honours an admission check with a retryable 503; unrelated tables are untouched. + const release = registerDerivedIndexTables(Product.auditStore, [Product.tableId], () => 'derived index is behind'); + // The guard fires before the first await, so the write may throw synchronously or reject. + const failure = async (write) => { + try { + await write(); + } catch (error) { + return error; + } + assert.fail('expected the write to be rejected'); + }; + const blocked = await failure(() => Product.put('p5', { title: 'blocked' })); + assert.strictEqual(blocked.statusCode, 503); + assert.strictEqual(blocked.code, 'DERIVED_INDEX_LAGGING'); + assert.strictEqual(blocked.retryable, true); + assert.strictEqual((await failure(() => Product.delete('p4'))).statusCode, 503); + assert.strictEqual((await failure(() => Product.create({ title: 'created' }))).statusCode, 503); + release(); + await Product.put('p5', { title: 'admitted' }); + await waitFor(() => backend.applied.has('p5'), { timeout: 5000 }); + + // A peer runtime on the same real store: shared readiness, the request word and the buffer + // notification all go through the native binding here, not the fake. + const peer = new DerivedIndexRuntime(Product.auditStore, () => undefined, { scanRecords: () => [] }); + assert.strictEqual(peer.getReadiness('rocks-rebuild').state, 'ready'); + // The binding hands every caller a plain ArrayBuffer over one process-wide allocation: a second + // wrapper of the same key observes the owner's publication, and never a SharedArrayBuffer. + const wrapper = Product.auditStore.getUserSharedBuffer( + 'derived-index:rocks-rebuild:readiness', + new ArrayBuffer(READINESS_BYTES) + ); + assert(!(wrapper instanceof SharedArrayBuffer)); + assert.strictEqual(readDerivedIndexReadiness(Product.auditStore, 'rocks-rebuild').state, 'ready'); + assert.strictEqual(new Int32Array(wrapper)[0], 1, 'the wrapper sees the published state word'); + // A real worker thread, through the binding alone, reads the owner's publication. + const { Worker } = require('node:worker_threads'); + const worker = new Worker( + `const { parentPort, workerData } = require('node:worker_threads'); + const { RocksDatabase } = require(workerData.binding); + const db = new RocksDatabase(workerData.path).open(); + const words = new Int32Array(db.getUserSharedBuffer(workerData.key, new ArrayBuffer(workerData.bytes)), 0, 6); + parentPort.postMessage({ state: Atomics.load(words, 0) }); + db.close();`, + { + eval: true, + workerData: { + binding: require.resolve('@harperfast/rocksdb-js'), + path: Product.auditStore.rootStore.path, + key: 'derived-index:rocks-rebuild:readiness', + bytes: READINESS_BYTES, + }, + } + ); + const seen = await new Promise((resolve, reject) => { + worker.once('message', resolve); + worker.once('error', reject); + }); + await new Promise((resolve) => worker.once('exit', resolve)); + assert.strictEqual(seen.state, 1, 'a worker thread reads the ready state the owner published'); + // The marker is written through the audit store's symbol-keyed putSync and read back through the + // root store: one keyspace on the real binding. + const condemnable = new AsyncBackend('rocks-condemn', { applyDelay: 2 }); + const condemning = new DerivedIndexRuntime(Product.auditStore, () => undefined, { scanRecords: () => [] }); + condemning.register({ backend: condemnable, projections: new Map([[Product.tableId, (record) => record]]) }); + await waitFor(() => condemning.getReadiness('rocks-condemn').state === 'ready', { timeout: 5000 }); + condemnable.reset = undefined; + condemnable.stateChange('failed'); + await waitFor(() => condemning.getStatus('rocks-condemn')?.state === 'needs-rebuild'); + const markerKey = Symbol.for('derived-index:rocks-condemn:condemned'); + assert.notStrictEqual( + Product.auditStore.rootStore.getSync(markerKey), + undefined, + 'marker readable via the root store' + ); + await condemning.stop(); + const rebuildable = new AsyncBackend('rocks-condemn', { applyDelay: 2 }); + const rebuilding = new DerivedIndexRuntime( + Product.auditStore, + (tableId, recordId) => { + const entry = Product.primaryStore.getEntry(recordId); + return entry?.value ? { version: entry.version, value: entry.value } : undefined; + }, + { scanRecords: () => [] } + ); + rebuilding.register({ + backend: rebuildable, + projections: new Map([[Product.tableId, (record) => ({ title: record.title })]]), + options: { maxFlushAgeMilliseconds: 5 }, + }); + await waitFor(() => rebuilding.getReadiness('rocks-condemn').state === 'ready', { timeout: 5000 }); + assert.strictEqual(rebuildable.resets.length, 1); + assert.strictEqual( + Product.auditStore.rootStore.getSync(markerKey), + undefined, + 'marker removed at the durable ready' + ); + await rebuilding.stop(); + + const peerBackend = new AsyncBackend('rocks-rebuild', { applyDelay: 2 }); + const unregisterPeer = peer.register({ + backend: peerBackend, + projections: new Map([[Product.tableId, (record) => ({ title: record.title })]]), + }); + assert.strictEqual(peer.requestRebuild('rocks-rebuild'), true); + await waitFor(() => backend.resets.length === 2, { timeout: 5000 }); + await waitFor(() => runtime.getReadiness('rocks-rebuild').state === 'ready', { timeout: 10_000 }); + // create() left Harper's internal id-allocation entry in the primary store; the scan must skip it. + assert.deepStrictEqual([...backend.applied.keys()].sort(), ['p1', 'p3', 'p4', 'p5']); + assert.strictEqual(peerBackend.resets.length, 0); + await unregisterPeer(); + await peer.stop(); + }); +}); diff --git a/unitTests/resources/derivedIndexRuntimeRocks.test.js b/unitTests/resources/derivedIndexRuntimeRocks.test.js index a79b92e4c1..8455b230ff 100644 --- a/unitTests/resources/derivedIndexRuntimeRocks.test.js +++ b/unitTests/resources/derivedIndexRuntimeRocks.test.js @@ -46,6 +46,9 @@ describe('DerivedIndexRuntime with an audited RocksDB table', () => { id: 'products', cursor: { format: 1, logs: { local: anchor } }, deliveries: [], + attach() {}, + flush() {}, + shutdown() {}, getDurableCursor() { return this.cursor; }, diff --git a/unitTests/resources/indexBackfillConvergence-crash.js b/unitTests/resources/indexBackfillConvergence-crash.js new file mode 100644 index 0000000000..8ea2a54c4e --- /dev/null +++ b/unitTests/resources/indexBackfillConvergence-crash.js @@ -0,0 +1,69 @@ +// Child-process half of the crash cases in indexBackfillConvergence.test.js; the mocha glob loads it +// too, hence the entry guard. +const path = require('node:path'); +const { mkdirSync, writeFileSync } = require('node:fs'); + +if (require.main === module) { + const [rootPath, databasePath, database, tableName, markerPath, rowCount, mode] = process.argv.slice(2); + const env = require('#src/utility/environment/environmentManager'); + const terms = require('#src/utility/hdbTerms'); + // A private root keeps this process off the parent's system database (RocksDB's lock is + // per process); only the database under test is shared, at the path the parent chose. + env.setProperty(terms.HDB_SETTINGS_NAMES.HDB_ROOT_KEY, rootPath); + env.setProperty(terms.CONFIG_PARAMS.STORAGE_PATH, path.join(rootPath, 'database')); + env.setProperty(terms.CONFIG_PARAMS.DATABASES, { [database]: { path: databasePath } }); + const { table, resetDatabases, setIndexingCheckpointPeriod } = require('#src/resources/databases'); + const { setMainIsWorker } = require('#js/server/threads/manageThreads'); + setMainIsWorker(true); + // kill-at-checkpoint: die at the first persisted checkpoint (checkpoint on every interval); + // kill-after-complete: never checkpoint, die once the ready descriptor is persisted + setIndexingCheckpointPeriod(mode === 'kill-after-complete' ? 3600000 : 0, 0); + + mkdirSync(path.join(rootPath, 'database'), { recursive: true }); + const seed = async () => { + const Tbl = table({ + table: tableName, + database, + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'tag' }], + }); + let last; + for (let i = 0; i < Number(rowCount); i++) { + last = Tbl.put({ id: 'c-' + String(i).padStart(6, '0'), tag: 't-' + (i % 7) }); + } + await last; + }; + + const dieAtFirstCheckpoint = () => { + resetDatabases(); + const Tbl = table({ + table: tableName, + database, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + ], + }); + const prefix = tableName + '/'; + const poll = () => { + for (const { key, value } of Tbl.dbisDB.getRange({ start: false })) { + if (value?.name !== 'tag' || !key.toString().startsWith(prefix)) continue; + if (value.lastIndexedKey !== undefined) { + writeFileSync(markerPath, value.lastIndexedKey); + process.kill(process.pid, 'SIGKILL'); + } + if (!value.indexingPID) { + writeFileSync(markerPath, 'COMPLETED'); + if (mode === 'kill-after-complete') process.kill(process.pid, 'SIGKILL'); + process.exit(0); + } + } + setImmediate(poll); + }; + setImmediate(poll); + }; + + seed().then(dieAtFirstCheckpoint, (error) => { + console.error(error); + process.exit(1); + }); +} diff --git a/unitTests/resources/indexBackfillConvergence.test.js b/unitTests/resources/indexBackfillConvergence.test.js new file mode 100644 index 0000000000..cf01150893 --- /dev/null +++ b/unitTests/resources/indexBackfillConvergence.test.js @@ -0,0 +1,743 @@ +/** + * Regression coverage for harper#2536: a secondary-index backfill that could not converge on a + * large table because runIndexing (resources/databases.ts) (1) never used its resume checkpoint — + * `start` stayed undefined and every retrigger rescanned from the first record — and (2) never + * yielded the event loop on a plain index whose put resolves synchronously, so the whole backfill + * ran as one uninterrupted turn. + */ +require('../testUtils'); +const assert = require('node:assert'); +const path = require('node:path'); +const { readFileSync, rmSync } = require('node:fs'); +const { spawn } = require('node:child_process'); +const { setupTestDBPath } = require('../testUtils'); +const env = require('#src/utility/environment/environmentManager'); +const terms = require('#src/utility/hdbTerms'); +const { + table, + resetDatabases, + closeDatabase, + resumeStartKey, + setIndexingCheckpointPeriod, +} = require('#src/resources/databases'); +const { setMainIsWorker } = require('#js/server/threads/manageThreads'); + +const DB = 'test'; +const LMDB = process.env.HARPER_STORAGE_ENGINE === 'lmdb'; +const INDEXING_YIELD_INTERVAL = 100; + +async function collect(iter) { + const out = []; + for await (const x of iter) out.push(x); + return out; +} + +function pad(i) { + return String(i).padStart(4, '0'); +} + +// The per-attribute descriptor lives in Table.dbisDB under the table's key prefix; the dbisDB is +// shared by every table in the database, so scope the scan to this table. +function findDescriptor(Tbl, attrName) { + const prefix = Tbl.tableName + '/'; + for (const { key, value } of Tbl.dbisDB.getRange({ start: false })) { + if (value && value.name === attrName && key.toString().startsWith(prefix)) return { key, value }; + } + return null; +} + +// LMDB commits checkpoint writes asynchronously; every checkpoint put is queued by the time +// runIndexing resolves, so waiting for the queue to flush makes the read authoritative. +async function settledCheckpoint(Tbl, attrName) { + await Tbl.dbisDB.flushed; + return findDescriptor(Tbl, attrName).value.lastIndexedKey; +} + +// Run the crash fixture as a child process; it is expected to SIGKILL itself, so a child that never +// reaches its marker is killed by the parent instead of wedging the run (.mocharc.json has timeout 0). +async function runCrashChild(args) { + const child = spawn(process.execPath, [path.join(__dirname, 'indexBackfillConvergence-crash.js'), ...args], { + stdio: ['ignore', 'ignore', 'pipe'], + }); + let stderr = ''; + child.stderr.on('data', (chunk) => (stderr += chunk)); + const timer = setTimeout(() => child.kill('SIGTERM'), 60000); + try { + return await new Promise((resolve, reject) => { + child.once('error', reject); + child.once('exit', (code, signal) => resolve({ code, signal, stderr })); + }); + } finally { + clearTimeout(timer); + } +} + +// Wrap Table.primaryStore.getRange so the test can observe the range runIndexing actually opens +// (its `start` option and every key it visits) and optionally abort the scan partway. runIndexing +// only reads the store after awaiting a schema-change signal and an event turn, so wrapping right +// after table() returns is early enough. +function observeRange(Tbl, { onKey, abortAfter } = {}) { + const store = Tbl.primaryStore; + const original = store.getRange; + const observed = { start: undefined, keys: [] }; + store.getRange = function (options) { + observed.start = options?.start; + const inner = original.call(this, options); + return { + [Symbol.iterator]() { + const iterator = inner[Symbol.iterator](); + return { + next: () => { + if (abortAfter !== undefined && observed.keys.length >= abortAfter) { + iterator.return?.(); + throw new Error('simulated primary-store iterator failure'); + } + const result = iterator.next(); + if (!result.done) { + observed.keys.push(result.value.key); + onKey?.(result.value.key); + } + return result; + }, + return: () => iterator.return?.(), + }; + }, + }; + }; + observed.restore = () => { + store.getRange = original; + }; + return observed; +} + +describe('resumeStartKey: minimum resume checkpoint across the attributes being built (#2536)', () => { + it('returns the shared checkpoint when every attribute checkpointed at the same key', () => { + assert.strictEqual(resumeStartKey([{ lastIndexedKey: 'k-0500' }, { lastIndexedKey: 'k-0500' }]), 'k-0500'); + }); + + it('returns the minimum when the attributes checkpointed at different keys', () => { + assert.strictEqual( + resumeStartKey([{ lastIndexedKey: 'k-0700' }, { lastIndexedKey: 'k-0300' }, { lastIndexedKey: 'k-0500' }]), + 'k-0300' + ); + assert.strictEqual(resumeStartKey([{ lastIndexedKey: 42 }, { lastIndexedKey: 7 }]), 7); + }); + + it('returns undefined (full scan) when any attribute has never checkpointed', () => { + assert.strictEqual(resumeStartKey([{ lastIndexedKey: 'k-0700' }, {}]), undefined); + assert.strictEqual(resumeStartKey([{ lastIndexedKey: 'k-0700' }, { lastIndexedKey: undefined }]), undefined); + }); + + it('returns the checkpoint of a single attribute', () => { + assert.strictEqual(resumeStartKey([{ lastIndexedKey: 'k-0900' }]), 'k-0900'); + }); +}); + +describe('index backfill convergence (#2536)', () => { + // checkpoint at every yield interval instead of every few seconds, so small tables checkpoint + let checkpointPolicy; + before(() => { + checkpointPolicy = setIndexingCheckpointPeriod(0, 0); + }); + after(() => { + setIndexingCheckpointPeriod(checkpointPolicy.ms, checkpointPolicy.minRecords); + }); + + it('resumes an interrupted backfill from its persisted checkpoint, not from the first record', async () => { + const TABLE = 'BackfillResume'; + const N = 600; + const ABORT_AFTER = 250; + setupTestDBPath(); + setMainIsWorker(true); + + let Tbl = table({ + table: TABLE, + database: DB, + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'tag' }, { name: 'group' }], + }); + let last; + for (let i = 0; i < N; i++) last = Tbl.put({ id: 'k-' + pad(i), tag: 't-' + (i % 3), group: 'g-' + (i % 2) }); + await last; + + resetDatabases(); + Tbl = table({ + table: TABLE, + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + { name: 'group', indexed: true }, + ], + }); + assert.ok(Tbl.indexingOperation, 'adding indexed attributes should trigger a backfill'); + const firstPass = observeRange(Tbl, { abortAfter: ABORT_AFTER }); + try { + await Tbl.indexingOperation; + } finally { + firstPass.restore(); + } + assert.strictEqual(firstPass.keys.length, ABORT_AFTER, 'the first pass should have been aborted partway'); + // runIndexing checkpoints every 100 entries it visits (LMDB yields a leading structures entry + // too), once the index writes the checkpoint covers have settled; LMDB commits those + // asynchronously, so the last checkpoint can land after the interruption itself was recorded. + const checkpoint = firstPass.keys[199]; + assert.strictEqual(await settledCheckpoint(Tbl, 'tag'), checkpoint, 'the last checkpoint should be persisted'); + for (const name of ['tag', 'group']) { + const parked = findDescriptor(Tbl, name); + assert.strictEqual(parked?.value.indexingFailed, true, `${name}: interrupted backfill should be parked`); + assert.strictEqual(parked.value.lastIndexedKey, checkpoint, `${name}: checkpoint should be persisted`); + assert.strictEqual(parked.value.checkpointCertified, checkpoint, `${name}: checkpoint should be stamped`); + } + + // The parked descriptor retriggers the backfill; it must open its scan at the checkpoint. + resetDatabases(); + const Tbl2 = table({ + table: TABLE, + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + { name: 'group', indexed: true }, + ], + }); + assert.ok(Tbl2.indexingOperation, 'a parked backfill should retrigger'); + const resumed = observeRange(Tbl2); + try { + await Tbl2.indexingOperation; + } finally { + resumed.restore(); + } + + assert.strictEqual(resumed.start, checkpoint, 'the resumed scan should start at the persisted checkpoint'); + assert.strictEqual(resumed.keys[0], checkpoint, 'the first key visited after resume should be the checkpoint'); + assert.strictEqual( + resumed.keys.length, + N - Number(checkpoint.slice(2)), + 'the resumed scan should only cover the checkpoint and the records after it' + ); + + for (const name of ['tag', 'group']) { + const done = findDescriptor(Tbl2, name); + assert.strictEqual(done.value.indexingFailed, undefined, `${name}: indexingFailed cleared after completion`); + assert.strictEqual(done.value.lastIndexedKey, undefined, `${name}: checkpoint cleared after completion`); + assert.strictEqual(done.value.checkpointCertified, undefined, `${name}: stamp cleared after completion`); + } + let total = 0; + for (const v of ['t-0', 't-1', 't-2']) { + total += (await collect(Tbl2.search({ conditions: [{ attribute: 'tag', value: v }] }))).length; + } + assert.strictEqual(total, N, 'every row should be indexed once the resumed backfill completes'); + }); + + // A failed index write must freeze the checkpoint before that record whether it throws + // synchronously (RocksDB) or a non-last value's put rejects asynchronously (LMDB), since only a + // record's last put is awaited. + for (const { failure, tagOf, failPut, secondAttribute } of [ + { + failure: 'throws synchronously', + tagOf: (i) => 't-' + (i % 3), + failPut: () => { + throw new Error('simulated transient index put failure'); + }, + }, + { + failure: 'rejects asynchronously on a non-last value', + tagOf: (i) => ['t-' + (i % 3), 'u-' + (i % 5)], + failPut: () => + new Promise((_, reject) => setImmediate(() => reject(new Error('simulated async index put failure')))), + }, + { + failure: "rejects asynchronously while a later attribute's put resolves", + tagOf: (i) => 't-' + (i % 3), + failPut: () => + new Promise((_, reject) => setImmediate(() => reject(new Error('simulated async index put failure')))), + secondAttribute: true, + }, + ]) { + it(`does not advance the checkpoint past a record whose index write ${failure}, so the retry re-covers it`, async () => { + const TABLE = 'BackfillFailedRecord' + (Array.isArray(tagOf(0)) ? 'Multi' : secondAttribute ? 'Two' : ''); + const N = 600; + const FAILING_ID = 'k-' + pad(250); + const failingValue = [].concat(tagOf(250))[0]; + const indexedAttributes = [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + { name: 'group', indexed: !!secondAttribute }, + ]; + setupTestDBPath(); + setMainIsWorker(true); + + let Tbl = table({ + table: TABLE, + database: DB, + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'tag' }, { name: 'group' }], + }); + let last; + for (let i = 0; i < N; i++) last = Tbl.put({ id: 'k-' + pad(i), tag: tagOf(i), group: 'g-' + (i % 2) }); + await last; + + resetDatabases(); + Tbl = table({ table: TABLE, database: DB, attributes: indexedAttributes }); + assert.ok(Tbl.indexingOperation, 'adding an indexed attribute should trigger a backfill'); + const tagIndex = Tbl.indices.tag; + const originalPut = tagIndex.put; + tagIndex.put = function (indexedValue, primaryKey, options) { + if (primaryKey === FAILING_ID && indexedValue === failingValue) return failPut(); + return originalPut.call(this, indexedValue, primaryKey, options); + }; + const firstPass = observeRange(Tbl); + try { + await Tbl.indexingOperation; + } finally { + tagIndex.put = originalPut; + firstPass.restore(); + } + const failedAt = firstPass.keys.indexOf(FAILING_ID); + const lastSafeCheckpoint = firstPass.keys[Math.floor(failedAt / 100) * 100 - 1]; + const parked = findDescriptor(Tbl, 'tag'); + assert.strictEqual(parked?.value.indexingFailed, true, 'a backfill with a failed record should be parked'); + const persisted = await settledCheckpoint(Tbl, 'tag'); + if (LMDB) { + // checkpoints wait for their writes to commit, so a failure that lands first withholds them + const safe = [undefined, ...firstPass.keys.slice(0, failedAt).filter((_, i) => i % 100 === 99)]; + assert.ok(safe.includes(persisted), `checkpoint ${persisted} must not pass the failed record`); + } else { + assert.strictEqual( + persisted, + lastSafeCheckpoint, + 'the checkpoint must stop at the last one written before the failed record' + ); + } + + resetDatabases(); + const Tbl2 = table({ table: TABLE, database: DB, attributes: indexedAttributes }); + assert.ok(Tbl2.indexingOperation, 'a parked backfill should retrigger'); + const resumed = observeRange(Tbl2); + try { + await Tbl2.indexingOperation; + } finally { + resumed.restore(); + } + assert.strictEqual(resumed.start, persisted, 'the retry should resume from the persisted safe checkpoint'); + assert.ok(resumed.keys.includes(FAILING_ID), 'the retry must revisit the record that failed'); + assert.strictEqual( + findDescriptor(Tbl2, 'tag').value.indexingFailed, + undefined, + 'the retry should complete cleanly' + ); + const viaIndex = await collect(Tbl2.search({ conditions: [{ attribute: 'tag', value: failingValue }] })); + assert.ok( + viaIndex.some((row) => row.id === FAILING_ID), + 'the record whose index write failed must be indexed after the retry' + ); + }); + } + + it('resumes from the minimum of unequal persisted checkpoints, and scans everything when one is absent', async () => { + const TABLE = 'BackfillUnequalCheckpoints'; + const N = 500; + setupTestDBPath(); + setMainIsWorker(true); + + let Tbl = table({ + table: TABLE, + database: DB, + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'tag' }, { name: 'group' }], + }); + let last; + for (let i = 0; i < N; i++) last = Tbl.put({ id: 'k-' + pad(i), tag: 't-' + (i % 3), group: 'g-' + (i % 2) }); + await last; + const indexedAttributes = [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + { name: 'group', indexed: true }, + ]; + resetDatabases(); + Tbl = table({ table: TABLE, database: DB, attributes: indexedAttributes }); + await Tbl.indexingOperation; + + // Park both indexes at different checkpoints, the way two attributes whose checkpoint writes + // straddled an interruption would be left. + const park = (Tbl, checkpoints, { certified = true } = {}) => { + for (const [name, lastIndexedKey] of Object.entries(checkpoints)) { + const { key, value } = findDescriptor(Tbl, name); + value.indexingFailed = true; + delete value.checkpointCertified; + if (lastIndexedKey === undefined) delete value.lastIndexedKey; + else { + value.lastIndexedKey = lastIndexedKey; + if (certified) value.checkpointCertified = lastIndexedKey; + } + Tbl.dbisDB.putSync(key, value); + } + }; + park(Tbl, { tag: 'k-' + pad(300), group: 'k-' + pad(200) }); + resetDatabases(); + let Tbl2 = table({ table: TABLE, database: DB, attributes: indexedAttributes }); + assert.ok(Tbl2.indexingOperation, 'parked indexes should retrigger'); + let resumed = observeRange(Tbl2); + try { + await Tbl2.indexingOperation; + } finally { + resumed.restore(); + } + assert.strictEqual(resumed.start, 'k-' + pad(200), 'the scan should start at the lower checkpoint'); + assert.strictEqual(resumed.keys[0], 'k-' + pad(200)); + + park(Tbl2, { tag: 'k-' + pad(300), group: undefined }); + resetDatabases(); + Tbl2 = table({ table: TABLE, database: DB, attributes: indexedAttributes }); + assert.ok(Tbl2.indexingOperation, 'parked indexes should retrigger'); + resumed = observeRange(Tbl2); + try { + await Tbl2.indexingOperation; + } finally { + resumed.restore(); + } + assert.strictEqual(resumed.start, undefined, 'an attribute with no checkpoint forces a full scan'); + assert.strictEqual( + resumed.keys.find((key) => typeof key === 'string'), + 'k-' + pad(0) + ); + for (const name of ['tag', 'group']) { + assert.strictEqual(findDescriptor(Tbl2, name).value.lastIndexedKey, undefined, `${name}: completed`); + } + const evens = await collect(Tbl2.search({ conditions: [{ attribute: 'group', value: 'g-0' }] })); + assert.strictEqual(evens.length, N / 2, 'the cleared index should be fully repopulated'); + + // A checkpoint written by a release without the stamp advanced past failed and unflushed index + // writes, so it must not be resumed: full rebuild, including the clear of the existing entries. + park(Tbl2, { tag: 'k-' + pad(300), group: 'k-' + pad(300) }, { certified: false }); + resetDatabases(); + Tbl2 = table({ table: TABLE, database: DB, attributes: indexedAttributes }); + assert.ok(Tbl2.indexingOperation, 'a parked legacy checkpoint should retrigger'); + resumed = observeRange(Tbl2); + try { + await Tbl2.indexingOperation; + } finally { + resumed.restore(); + } + assert.strictEqual(resumed.start, undefined, 'an unstamped legacy checkpoint must not be resumed'); + assert.strictEqual( + resumed.keys.find((key) => typeof key === 'string'), + 'k-' + pad(0) + ); + const odds = await collect(Tbl2.search({ conditions: [{ attribute: 'group', value: 'g-1' }] })); + assert.strictEqual(odds.length, N / 2, 'the rebuilt index should be complete'); + }); + + it('resumes from the checkpoint a process killed mid-backfill left behind', async () => { + const DATABASE = 'backfillcrash'; + const TABLE = 'BackfillCrash'; + const N = 10000; + const dbPath = setupTestDBPath(); + setMainIsWorker(true); + // The database under test lives outside storage.path and is opened only by the child until + // it is dead, so no store is ever shared between the two processes. + const crashDir = path.join(dbPath, 'backfill-crash'); + rmSync(crashDir, { recursive: true, force: true }); + const markerPath = path.join(crashDir, 'checkpoint.marker'); + const databasesConfig = env.get(terms.CONFIG_PARAMS.DATABASES); + env.setProperty(terms.CONFIG_PARAMS.DATABASES, { + ...databasesConfig, + [DATABASE]: { path: path.join(crashDir, 'shared') }, + }); + + const { code, signal, stderr } = await runCrashChild([ + path.join(crashDir, 'child-root'), + path.join(crashDir, 'shared'), + DATABASE, + TABLE, + markerPath, + String(N), + 'kill-at-checkpoint', + ]); + assert.strictEqual( + signal, + 'SIGKILL', + `the child should have killed itself at its first checkpoint (exit ${code}): ${stderr}` + ); + const checkpoint = readFileSync(markerPath, 'utf8'); + assert.match(checkpoint, /^c-\d{6}$/, 'the child should have recorded a durable checkpoint'); + + // The dead process's PID on the descriptor is the crash-recovery trigger. + const Tbl = table({ + table: TABLE, + database: DATABASE, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + ], + }); + try { + assert.ok(Tbl.indexingOperation, 'reopening after the crash should retrigger the backfill'); + const resumed = observeRange(Tbl); + try { + await Tbl.indexingOperation; + } finally { + resumed.restore(); + } + if (LMDB) { + // the child read a committed checkpoint, but LMDB's write thread can commit the next one + // (already queued) before the kill lands + assert.ok(resumed.start >= checkpoint, `the resumed scan should start at or after ${checkpoint}`); + } else { + assert.strictEqual( + resumed.start, + checkpoint, + "the resumed scan should start at the crashed process's checkpoint" + ); + } + assert.strictEqual(resumed.keys[0], resumed.start); + assert.strictEqual( + findDescriptor(Tbl, 'tag').value.indexingPID, + undefined, + 'the resumed backfill should complete' + ); + let total = 0; + for (let i = 0; i < 7; i++) { + total += (await collect(Tbl.search({ conditions: [{ attribute: 'tag', value: 't-' + i }] }))).length; + } + assert.strictEqual(total, N, 'every row should be indexed after the resumed backfill'); + } finally { + closeDatabase(DATABASE); + env.setProperty(terms.CONFIG_PARAMS.DATABASES, databasesConfig); + } + }); + + it('flushes the tail written since the last checkpoint before announcing the index complete', async () => { + const DATABASE = 'backfillcomplete'; + const TABLE = 'BackfillComplete'; + const N = 10000; + const dbPath = setupTestDBPath(); + setMainIsWorker(true); + const crashDir = path.join(dbPath, 'backfill-complete'); + rmSync(crashDir, { recursive: true, force: true }); + const markerPath = path.join(crashDir, 'complete.marker'); + const databasesConfig = env.get(terms.CONFIG_PARAMS.DATABASES); + env.setProperty(terms.CONFIG_PARAMS.DATABASES, { + ...databasesConfig, + [DATABASE]: { path: path.join(crashDir, 'shared') }, + }); + + // a long period means the whole index is the unflushed tail when the ready descriptor lands + const { code, signal, stderr } = await runCrashChild([ + path.join(crashDir, 'child-root'), + path.join(crashDir, 'shared'), + DATABASE, + TABLE, + markerPath, + String(N), + 'kill-after-complete', + ]); + assert.strictEqual( + signal, + 'SIGKILL', + `the child should have killed itself once complete (exit ${code}): ${stderr}` + ); + assert.strictEqual(readFileSync(markerPath, 'utf8'), 'COMPLETED'); + + const Tbl = table({ + table: TABLE, + database: DATABASE, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + ], + }); + try { + assert.strictEqual(Tbl.indexingOperation, undefined, 'a completed index must not retrigger'); + let total = 0; + for (let i = 0; i < 7; i++) { + total += (await collect(Tbl.search({ conditions: [{ attribute: 'tag', value: 't-' + i }] }))).length; + } + assert.strictEqual(total, N, 'every index entry must survive a kill right after completion'); + } finally { + closeDatabase(DATABASE); + env.setProperty(terms.CONFIG_PARAMS.DATABASES, databasesConfig); + } + }); + + it('parks the index instead of certifying a checkpoint or completing when the flush fails', async function () { + if (LMDB) return this.skip(); // LMDB commits in order and never flushes + const TABLE = 'BackfillFlushFails'; + const N = 300; + setupTestDBPath(); + setMainIsWorker(true); + + let Tbl = table({ + table: TABLE, + database: DB, + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'tag' }], + }); + let last; + for (let i = 0; i < N; i++) last = Tbl.put({ id: 'k-' + pad(i), tag: 't-' + (i % 3) }); + await last; + + resetDatabases(); + Tbl = table({ + table: TABLE, + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + ], + }); + assert.ok(Tbl.indexingOperation, 'adding an indexed attribute should trigger a backfill'); + const rootStore = Tbl.primaryStore.rootStore; + const originalFlush = rootStore.flush; + rootStore.flush = () => Promise.reject(new Error('simulated flush failure')); + try { + await Tbl.indexingOperation; + } finally { + rootStore.flush = originalFlush; + } + const parked = findDescriptor(Tbl, 'tag'); + assert.strictEqual(parked.value.indexingFailed, true, 'the index must stay parked when it cannot be flushed'); + assert.strictEqual(parked.value.lastIndexedKey, undefined, 'no checkpoint may be certified without a flush'); + + resetDatabases(); + const Tbl2 = table({ + table: TABLE, + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + ], + }); + assert.ok(Tbl2.indexingOperation, 'the parked index should retrigger'); + await Tbl2.indexingOperation; + assert.strictEqual(findDescriptor(Tbl2, 'tag').value.indexingFailed, undefined, 'the retry should complete'); + }); + + it('does not persist a checkpoint before the record floor, whatever the period', async () => { + const TABLE = 'BackfillCheckpointFloor'; + const N = 25000; + const FLOOR = 10000; + setupTestDBPath(); + setMainIsWorker(true); + + let Tbl = table({ + table: TABLE, + database: DB, + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'tag' }], + }); + let last; + for (let i = 0; i < N; i++) last = Tbl.put({ id: 'f-' + String(i).padStart(5, '0'), tag: 't-' + (i % 3) }); + await last; + + const policy = setIndexingCheckpointPeriod(0, FLOOR); + try { + resetDatabases(); + Tbl = table({ + table: TABLE, + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + ], + }); + assert.ok(Tbl.indexingOperation, 'adding an indexed attribute should trigger a backfill'); + const checkpoints = []; + const originalPut = Tbl.dbisDB.put; + Tbl.dbisDB.put = function (key, value, options) { + if (value?.name === 'tag' && value.lastIndexedKey !== undefined) checkpoints.push(value.lastIndexedKey); + return originalPut.call(this, key, value, options); + }; + try { + await Tbl.indexingOperation; + } finally { + Tbl.dbisDB.put = originalPut; + } + assert.ok(checkpoints.length > 0, 'a 25k-row backfill should checkpoint'); + const stringKeys = (key) => typeof key === 'string'; + const visitedBefore = (key) => Number(key.slice(2)) + (LMDB ? 1 : 0) + 1; + assert.ok( + visitedBefore(checkpoints[0]) >= FLOOR, + `the first checkpoint ${checkpoints[0]} should come after ${FLOOR} records` + ); + for (let i = 1; i < checkpoints.length; i++) { + assert.ok( + visitedBefore(checkpoints[i]) - visitedBefore(checkpoints[i - 1]) >= FLOOR, + `checkpoints ${checkpoints[i - 1]} and ${checkpoints[i]} are closer than ${FLOOR} records` + ); + } + assert.ok(checkpoints.every(stringKeys)); + } finally { + setIndexingCheckpointPeriod(policy.ms, policy.minRecords); + } + }); + + it('yields the event loop at a bounded record interval on a plain index whose put resolves synchronously', async () => { + const TABLE = 'BackfillYield'; + const N = 2000; + setupTestDBPath(); + setMainIsWorker(true); + + let Tbl = table({ + table: TABLE, + database: DB, + attributes: [{ name: 'id', isPrimaryKey: true }, { name: 'tag' }], + }); + let last; + for (let i = 0; i < N; i++) last = Tbl.put({ id: 'y-' + pad(i), tag: 't-' + (i % 5) }); + await last; + + resetDatabases(); + Tbl = table({ + table: TABLE, + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'tag', indexed: true }, + ], + }); + assert.ok(Tbl.indexingOperation, 'adding an indexed attribute should trigger a backfill'); + + // A setImmediate ticker advances once per event-loop turn the backfill gives up; stamp each + // visited key with the current tick so the longest run of keys on one tick is the longest + // stretch the loop ran without yielding. + let tick = 0; + let running = true; + const ticksPerKey = []; + const ticker = () => { + if (!running) return; + tick++; + setImmediate(ticker); + }; + setImmediate(ticker); + const observed = observeRange(Tbl, { + onKey: (key) => { + if (typeof key === 'string') ticksPerKey.push(tick); + }, + }); + try { + await Tbl.indexingOperation; + } finally { + running = false; + observed.restore(); + } + + assert.strictEqual(ticksPerKey.length, N, 'the backfill should visit every record'); + let longestRun = 0; + let run = 0; + for (let i = 0; i < ticksPerKey.length; i++) { + run = i > 0 && ticksPerKey[i] === ticksPerKey[i - 1] ? run + 1 : 1; + if (run > longestRun) longestRun = run; + } + // LMDB index puts are asynchronous, so the pre-existing backpressure branch yields more often there + if (LMDB) { + assert.ok( + longestRun <= INDEXING_YIELD_INTERVAL, + `backfill ran ${longestRun} records without yielding the event loop (bound ${INDEXING_YIELD_INTERVAL})` + ); + } else { + assert.ok( + longestRun >= INDEXING_YIELD_INTERVAL / 2 && longestRun <= INDEXING_YIELD_INTERVAL, + `backfill should yield the event loop every ${INDEXING_YIELD_INTERVAL} records, ran ${longestRun}` + ); + } + const complete = await collect(Tbl.search({ conditions: [{ attribute: 'tag', value: 't-0' }] })); + assert.strictEqual(complete.length, N / 5, 'the backfill should still index every row'); + }); +}); diff --git a/unitTests/resources/indexBuildAbandonment.test.js b/unitTests/resources/indexBuildAbandonment.test.js new file mode 100644 index 0000000000..de3774e488 --- /dev/null +++ b/unitTests/resources/indexBuildAbandonment.test.js @@ -0,0 +1,313 @@ +/** + * harper#2537 / harper#2536. A backfill can end without running either of runIndexing's own exit paths + * — the restart-interrupt return inside the loop, and the closed-store return in its catch — leaving a + * descriptor that claims an armed, in-progress build with no `indexingFailed`, nothing logged above + * debug, and nothing to re-trigger it. Both reach the same settle handler; the cases below simulate + * the closed-store return, which is the one reproducible without a live worker generation. Two guards + * cover that: + * + * 1. the operation's settle handler persists the failure marker for the exact build it scheduled; + * 2. the trigger treats a build whose process incarnation is not this process's — including a + * descriptor written before that field existed — as abandoned, which is the only way to detect a + * process restart when Harper is PID 1 and the restart generation resets to 1 in memory. + */ + +require('../testUtils'); +const assert = require('node:assert'); +const { RocksDatabase } = require('@harperfast/rocksdb-js'); +const { setupTestDBPath } = require('../testUtils'); +const { table } = require('#src/resources/databases'); +const manageThreads = require('#js/server/threads/manageThreads'); +const { forComponent } = require('#src/utility/logging/harper_logger'); + +// resources/Table.ts keys the exclusive schema lock on these bytes +const UPDATE_ATTRIBUTES_LOCK_KEY = Buffer.from('update-attributes'); + +describe('an index build that ends without completing is marked and recovered', function () { + this.timeout(60000); + + before(() => { + setupTestDBPath(); + manageThreads.setMainIsWorker(true); + }); + + async function catalogFlushed(Table) { + if (Table.dbisDB.committed) await Table.dbisDB.committed; + } + + function seed(tableName, indexed) { + const Table = table({ + table: tableName, + database: 'test', + schemaDefined: true, + attributes: [ + { name: 'id', type: 'ID', isPrimaryKey: true }, + { name: 'tag', type: 'String', indexed }, + ], + }); + return Table; + } + + it('persists a failure marker when the backfill returns through its silent store-shutdown path', async () => { + const tableName = 'IndexAbandonShutdown'; + const Seeded = seed(tableName, false); + let lastPut; + for (let i = 0; i < 10; i++) lastPut = Seeded.put({ id: `k-${i}`, tag: i % 2 ? 'odd' : 'even' }); + await lastPut; + + const storageLogger = forComponent('storage'); + const originalWarn = storageLogger.warn; + const warnings = []; + storageLogger.warn = (...args) => warnings.push(args); + + // table() reads the primary store to decide there is data to backfill, so the interruption is + // installed only once it has returned — runIndexing suspends at its first await before iterating. + const Rebuilding = seed(tableName, true); + const primaryStore = Rebuilding.primaryStore; + const rootStore = primaryStore.rootStore; + const originalGetRange = primaryStore.getRange; + const originalStatus = Object.getOwnPropertyDescriptor(rootStore, 'status'); + primaryStore.getRange = () => { + throw new Error('Database not open'); + }; + Object.defineProperty(rootStore, 'status', { value: 'closed', configurable: true, writable: true }); + try { + await Rebuilding.indexingOperation; + } finally { + primaryStore.getRange = originalGetRange; + if (originalStatus) Object.defineProperty(rootStore, 'status', originalStatus); + else delete rootStore.status; + storageLogger.warn = originalWarn; + } + + await catalogFlushed(Rebuilding); + const descriptor = Rebuilding.dbisDB.getSync(`${tableName}/tag`); + assert.strictEqual( + descriptor.indexingFailed, + true, + 'a backfill that returned without completing must leave a durable failure marker, or nothing re-triggers it' + ); + assert.ok(descriptor.indexingPID, 'the build must still read as incomplete so queries keep refusing'); + assert.strictEqual( + Rebuilding.indices.tag.isIndexing, + true, + 'the index must stay marked as rebuilding after an abandoned build' + ); + const reported = warnings + .map(([message]) => message) + .filter((message) => typeof message === 'string' && message.includes(`${tableName}.tag`)); + assert.strictEqual( + reported.length, + 1, + `the abandoned build must be reported above debug: ${JSON.stringify(warnings)}` + ); + + const Recovered = seed(tableName, true); + assert.ok(Recovered.indexingOperation, 'the persisted marker must re-trigger the backfill on the next load'); + await Recovered.indexingOperation; + await catalogFlushed(Recovered); + assert.strictEqual( + Recovered.dbisDB.getSync(`${tableName}/tag`).indexingPID, + undefined, + 'the recovered build must complete and clear the descriptor' + ); + assert.strictEqual(Recovered.indices.tag.isIndexing, false, 'the recovered index must be usable again'); + }); + + it('does not mark a build a replacement claimed between the settle handler check and its write', async () => { + const tableName = 'IndexAbandonNotOwned'; + const key = `${tableName}/tag`; + const Seeded = seed(tableName, false); + let lastPut; + for (let i = 0; i < 10; i++) lastPut = Seeded.put({ id: `k-${i}`, tag: i % 2 ? 'odd' : 'even' }); + await lastPut; + + const Rebuilding = seed(tableName, true); + const ownBuildId = Rebuilding.dbisDB.getSync(key).indexingBuildId; + assert.ok(ownBuildId, 'the trigger must stamp a build id for the settle handler to fence on'); + + // The settle handler re-reads under the exclusive catalog lock; report a replacement's claim on + // that read, which is the window a replacement worker generation actually claims the build in. + const dbisDB = Rebuilding.dbisDB; + const rootStore = Rebuilding.primaryStore.rootStore; + const isRocksDatabase = rootStore instanceof RocksDatabase; + const originalGetSync = dbisDB.getSync.bind(dbisDB); + const originalTransactionSync = rootStore.transactionSync; + let reads = 0; + let heldOnReread = null; + if (!isRocksDatabase) { + rootStore.transactionSync = function (...args) { + heldOnReread = true; + return originalTransactionSync.apply(this, args); + }; + } + dbisDB.getSync = (readKey, ...rest) => { + const value = originalGetSync(readKey, ...rest); + if (readKey === key && ++reads === 2) { + // tryLock fails even for the thread already holding it, so this observes the locked section + if (isRocksDatabase) { + heldOnReread = !rootStore.tryLock(UPDATE_ATTRIBUTES_LOCK_KEY); + if (!heldOnReread) rootStore.unlock(UPDATE_ATTRIBUTES_LOCK_KEY); + } + return { ...value, indexingBuildId: 'a-replacement-build' }; + } + return value; + }; + const originalStatus = Object.getOwnPropertyDescriptor(rootStore, 'status'); + const originalGetRange = Rebuilding.primaryStore.getRange; + Rebuilding.primaryStore.getRange = () => { + throw new Error('Database not open'); + }; + Object.defineProperty(rootStore, 'status', { value: 'closed', configurable: true, writable: true }); + try { + await Rebuilding.indexingOperation; + } finally { + Rebuilding.primaryStore.getRange = originalGetRange; + if (originalStatus) Object.defineProperty(rootStore, 'status', originalStatus); + else delete rootStore.status; + dbisDB.getSync = originalGetSync; + if (!isRocksDatabase) rootStore.transactionSync = originalTransactionSync; + } + + assert.ok(reads >= 2, 'the settle handler must re-read the descriptor after its first check'); + assert.strictEqual( + heldOnReread, + true, + 'the re-read and the write must happen under the exclusive catalog lock the declaration takes' + ); + await catalogFlushed(Rebuilding); + assert.strictEqual( + originalGetSync(key).indexingFailed, + undefined, + 'the settle handler marked a build that a replacement had already claimed' + ); + assert.strictEqual( + originalGetSync(key).indexingBuildId, + ownBuildId, + 'the settle handler must not write its own stale descriptor snapshot back over the catalog' + ); + }); + + it('aborts the LMDB catalog transaction when persisting the failure marker throws', async function () { + if (process.env.HARPER_STORAGE_ENGINE !== 'lmdb') this.skip(); + const tableName = 'IndexAbandonMarkerAbort'; + const key = `${tableName}/tag`; + const Seeded = seed(tableName, false); + let lastPut; + for (let i = 0; i < 10; i++) lastPut = Seeded.put({ id: `k-${i}`, tag: i % 2 ? 'odd' : 'even' }); + await lastPut; + + const Rebuilding = seed(tableName, true); + const dbisDB = Rebuilding.dbisDB; + const originalPut = dbisDB.put; + const originalPutSync = dbisDB.putSync; + function throwAfterMarkerWrite(write) { + return function (writeKey, value, ...rest) { + const result = write.call(this, writeKey, value, ...rest); + if (writeKey === key && value?.indexingFailed) throw new Error('marker write failed after staging'); + return result; + }; + } + dbisDB.put = throwAfterMarkerWrite(originalPut); + dbisDB.putSync = throwAfterMarkerWrite(originalPutSync); + const rootStore = Rebuilding.primaryStore.rootStore; + const originalStatus = Object.getOwnPropertyDescriptor(rootStore, 'status'); + const originalGetRange = Rebuilding.primaryStore.getRange; + Rebuilding.primaryStore.getRange = () => { + throw new Error('Database not open'); + }; + Object.defineProperty(rootStore, 'status', { value: 'closed', configurable: true, writable: true }); + try { + await Rebuilding.indexingOperation; + } finally { + Rebuilding.primaryStore.getRange = originalGetRange; + if (originalStatus) Object.defineProperty(rootStore, 'status', originalStatus); + else delete rootStore.status; + dbisDB.put = originalPut; + dbisDB.putSync = originalPutSync; + } + + assert.strictEqual( + dbisDB.getSync(key).indexingFailed, + undefined, + 'a failed marker write must abort instead of committing its staged catalog change' + ); + }); + + it('re-triggers a build whose process incarnation is not this process, including one that has none', async () => { + const tableName = 'IndexAbandonIncarnation'; + const Seeded = seed(tableName, true); + let lastPut; + for (let i = 0; i < 10; i++) lastPut = Seeded.put({ id: `k-${i}`, tag: i % 2 ? 'odd' : 'even' }); + await lastPut; + if (Seeded.indexingOperation) await Seeded.indexingOperation; + await catalogFlushed(Seeded); + const completedBuild = Seeded.indexingOperation; + const key = `${tableName}/tag`; + const complete = Seeded.dbisDB.getSync(key); + + // Both shapes a container restart leaves behind: the PID is reused (Harper is PID 1) and the + // in-memory restart generation is back to its starting value, so neither existing check fires. + for (const [label, incarnation] of [ + ['written by an older version, with no incarnation at all', undefined], + ['written by a previous incarnation of this same PID', 'not-this-process'], + ]) { + const armed = { + ...complete, + indexingPID: process.pid, + restartNumber: manageThreads.restartNumber ?? 1, + lastIndexedKey: 'k-4', + }; + if (incarnation === undefined) delete armed.indexingIncarnation; + else armed.indexingIncarnation = incarnation; + const written = Seeded.dbisDB.put(key, armed); + if (written?.then) await written; + + const Recovered = seed(tableName, true); + assert.notStrictEqual( + Recovered.indexingOperation, + completedBuild, + `an armed build ${label} must be re-triggered, not trusted` + ); + await Recovered.indexingOperation; + await catalogFlushed(Recovered); + assert.strictEqual( + Recovered.dbisDB.getSync(key).indexingPID, + undefined, + `the recovered build (${label}) must complete and clear the descriptor` + ); + assert.strictEqual(Recovered.indices.tag.isIndexing, false, `the recovered index (${label}) must be usable`); + const odds = []; + for await (const record of Recovered.search({ conditions: [{ attribute: 'tag', value: 'odd' }] })) + odds.push(record); + assert.strictEqual(odds.length, 5, `the recovered backfill (${label}) must index every record`); + } + }); + + it('leaves a build owned by this process incarnation alone', async () => { + const tableName = 'IndexAbandonLive'; + const Seeded = seed(tableName, true); + let lastPut; + for (let i = 0; i < 10; i++) lastPut = Seeded.put({ id: `k-${i}`, tag: i % 2 ? 'odd' : 'even' }); + await lastPut; + if (Seeded.indexingOperation) await Seeded.indexingOperation; + await catalogFlushed(Seeded); + const completedBuild = Seeded.indexingOperation; + + // A second thread of this process declaring the same table while the build is genuinely in flight + // must not start a duplicate backfill. + const key = `${tableName}/tag`; + const written = Seeded.dbisDB.put(key, { + ...Seeded.dbisDB.getSync(key), + indexingPID: process.pid, + restartNumber: manageThreads.restartNumber ?? 1, + indexingIncarnation: manageThreads.processIncarnation, + lastIndexedKey: 'k-4', + }); + if (written?.then) await written; + + const Live = seed(tableName, true); + assert.strictEqual(Live.indexingOperation, completedBuild, 'a live build in this process must not be re-triggered'); + assert.strictEqual(Live.indices.tag.isIndexing, true, 'a live build must still read as incomplete'); + }); +}); diff --git a/unitTests/resources/indexRebuildThreadConsistency-thread.js b/unitTests/resources/indexRebuildThreadConsistency-thread.js new file mode 100644 index 0000000000..02e7da52c8 --- /dev/null +++ b/unitTests/resources/indexRebuildThreadConsistency-thread.js @@ -0,0 +1,57 @@ +const { parentPort, workerData } = require('node:worker_threads'); +const { setupTestDBPath } = require('../testUtils'); +const { resetDatabases } = require('#src/resources/databases'); +const manageThreads = require('#js/server/threads/manageThreads'); +const { setMainIsWorker } = manageThreads; + +// A thread that never declares the schema: it reaches Table.indices only through the catalog reload +// (resetDatabases -> initStores), which is what the main and operations threads do in a running node. +// Probing is driven by the test with Atomics rather than by an ITC schema event, so the backfill can be +// held at a known point; the reload it performs is the same one the ITC handler performs. +const { phase, ack, tableName, attributeName, probeValue, probeId } = workerData ?? {}; +if (phase) run(); + +async function run() { + setupTestDBPath(); + setMainIsWorker(true); + for (let step = 0; step < 3; step++) { + Atomics.wait(phase, 0, step); + await probe(step + 1); + Atomics.store(ack, 0, step + 1); + Atomics.notify(ack, 0); + } +} + +async function probe(step) { + const message = { + step, + loaded: false, + isIndexing: null, + hits: null, + searchError: null, + foundById: false, + processIncarnation: manageThreads.processIncarnation, + }; + try { + const Table = resetDatabases().test?.[tableName]; + message.loaded = Boolean(Table); + if (Table) { + message.isIndexing = Table.indices[attributeName]?.isIndexing ?? null; + try { + const hits = []; + for await (const record of Table.search({ + allowFullScan: false, + conditions: [{ attribute: attributeName, value: probeValue }], + })) + hits.push(record); + message.hits = hits.length; + } catch (error) { + message.searchError = error?.message ?? String(error); + } + message.foundById = Boolean(await Table.get(probeId)); + } + } catch (error) { + message.failure = error?.message ?? String(error); + } + parentPort.postMessage(message); +} diff --git a/unitTests/resources/indexRebuildThreadConsistency.test.js b/unitTests/resources/indexRebuildThreadConsistency.test.js new file mode 100644 index 0000000000..4398dc234a --- /dev/null +++ b/unitTests/resources/indexRebuildThreadConsistency.test.js @@ -0,0 +1,203 @@ +/** + * harper#2537: a thread that reaches Table.indices through the schema load path (resetDatabases -> + * initStores) rather than by declaring the schema held `isIndexing === false` for a rebuilding index + * and served it — 200 with rows missing, while a primary-key read returned the same record. + * + * The backfill is held by blocking the declaring thread's event loop in `Atomics.wait`: runIndexing + * suspends at its first await, so the index is empty there and no timing window is needed. + */ + +require('../testUtils'); +const assert = require('node:assert'); +const { Worker } = require('node:worker_threads'); +const { setupTestDBPath } = require('../testUtils'); +const { table } = require('#src/resources/databases'); +const manageThreads = require('#js/server/threads/manageThreads'); +const { setMainIsWorker } = manageThreads; + +const WAIT_MS = 30000; +const PROBE_ID = 'seed-3'; +const PROBE_VALUE = '/p/3'; + +describe('an index being rebuilt is incomplete on every thread (harper#2537)', function () { + this.timeout(60000); + + before(() => { + setupTestDBPath(); + setMainIsWorker(true); + }); + + /** + * Runs the three-step probe on a second thread: before the rebuild is triggered, while it is held, + * and after it has completed. `trigger` runs on this thread between the first and second probe and + * must return without awaiting the backfill. + */ + async function withReader(tableName, attributeName, trigger) { + const phase = new Int32Array(new SharedArrayBuffer(4)); + const ack = new Int32Array(new SharedArrayBuffer(4)); + const worker = new Worker(__dirname + '/indexRebuildThreadConsistency-thread.js', { + workerData: { + phase, + ack, + tableName, + attributeName, + probeValue: PROBE_VALUE, + probeId: PROBE_ID, + addPorts: [], + // startWorker sends this key with every worker it spawns; the reader must adopt it, not mint + // its own, or two live threads would disagree about which builds this process owns + processIncarnation: manageThreads.processIncarnation, + }, + }); + const probes = {}; + const failure = new Promise((_, reject) => worker.once('error', reject)); + worker.on('message', (message) => (probes[message.step] = message)); + const probed = (step) => + Promise.race([ + failure, + new Promise((resolve) => { + if (probes[step]) return resolve(probes[step]); + worker.on('message', function onMessage(message) { + if (message.step !== step) return; + worker.off('message', onMessage); + resolve(message); + }); + }), + ]); + // every exit from here must terminate the worker, or it stays blocked in Atomics.wait + try { + // Waits on the acknowledged step rather than on Atomics.wait's return: a reader that acks + // before this thread reaches the wait makes it return 'not-equal', which is success, not failure. + const release = (step) => { + Atomics.store(phase, 0, step); + Atomics.notify(phase, 0); + const deadline = Date.now() + WAIT_MS; + while (Atomics.load(ack, 0) < step) { + const remaining = deadline - Date.now(); + if (remaining <= 0) return false; + Atomics.wait(ack, 0, step - 1, remaining); + } + return true; + }; + + assert.ok(release(1), 'the reader thread never finished its pre-rebuild load'); + const before = await probed(1); + + const Table = trigger(); + assert.ok(Table.indexingOperation, 'the rebuild was not triggered, so nothing is held'); + // Blocking here keeps the backfill suspended at runIndexing's first await, so the reader + // observes an armed descriptor over an index with nothing written to it yet. + assert.ok(release(2), 'the reader thread never finished its mid-rebuild load'); + const during = await probed(2); + + await Table.indexingOperation; + assert.ok(release(3), 'the reader thread never finished its post-rebuild load'); + const after = await probed(3); + + for (const probe of [before, during, after]) + assert.strictEqual(probe.failure, undefined, `reader thread failed at step ${probe.step}: ${probe.failure}`); + return { before, during, after }; + } finally { + await worker.terminate(); + } + } + + function seed(tableName, attributes) { + const Table = table({ table: tableName, database: 'test', schemaDefined: true, attributes }); + let lastPut; + for (let i = 0; i < 8; i++) lastPut = Table.put({ id: `seed-${i}`, path: `/p/${i}` }); + return { Table, lastPut }; + } + + it('a thread that opened the table before the rebuild must not serve the partial index', async () => { + const tableName = 'IndexThreadNewHandle'; + const { Table, lastPut } = seed(tableName, [ + { name: 'id', type: 'ID', isPrimaryKey: true }, + { name: 'path', type: 'String' }, + ]); + await lastPut; + if (Table.indexingOperation) await Table.indexingOperation; + + const { before, during, after } = await withReader(tableName, 'path', () => + table({ + table: tableName, + database: 'test', + schemaDefined: true, + attributes: [ + { name: 'id', type: 'ID', isPrimaryKey: true }, + { name: 'path', type: 'String', indexed: true }, + ], + }) + ); + + assert.ok(before.loaded, 'the reader thread must have loaded the table before the rebuild'); + assert.strictEqual( + before.processIncarnation, + manageThreads.processIncarnation, + 'a worker must adopt the incarnation it was started with, so live threads agree on build ownership' + ); + assert.strictEqual(before.isIndexing, null, 'there is no index on the attribute before the rebuild'); + + assert.ok( + during.foundById, + 'the probe record must be readable by primary key while the rebuild is held, or the test proves nothing' + ); + assert.strictEqual( + during.isIndexing, + true, + 'a thread that loaded the table before the rebuild was triggered held isIndexing = false and served the partial index' + ); + assert.match( + during.searchError ?? '', + /not indexed yet/, + `a read of a rebuilding index must refuse, not return ${during.hits} rows for a record that exists` + ); + + assert.strictEqual(after.isIndexing, false, 'the reload after completion must clear isIndexing'); + assert.strictEqual(after.searchError, null, 'the completed index must serve reads'); + assert.strictEqual(after.hits, 1, 'the completed index must return the seeded record'); + }); + + it('a handle the reader already holds is re-stamped, not left at its previous state', async () => { + const tableName = 'IndexThreadReusedHandle'; + const indexedAttributes = (indexed) => [ + { name: 'id', type: 'ID', isPrimaryKey: true }, + { name: 'path', type: 'String', indexed }, + ]; + const { Table, lastPut } = seed(tableName, indexedAttributes(true)); + await lastPut; + if (Table.indexingOperation) await Table.indexingOperation; + + // A structural index-option change re-triggers the backfill over an attribute the reader thread + // already holds an open index handle for, so its handle is reused by the reload rather than opened. + const { before, during, after } = await withReader(tableName, 'path', () => + table({ + table: tableName, + database: 'test', + schemaDefined: true, + attributes: indexedAttributes({ indexNulls: false }), + }) + ); + + assert.strictEqual(before.isIndexing, false, 'the completed index must be usable before the rebuild'); + assert.strictEqual(before.hits, 1, 'the completed index must return the seeded record before the rebuild'); + + assert.strictEqual( + during.isIndexing, + true, + 'a handle already open on the reader thread must be re-stamped as rebuilding' + ); + assert.match( + during.searchError ?? '', + /not indexed yet/, + 'a reused handle must also refuse reads while rebuilding' + ); + + assert.strictEqual( + after.isIndexing, + false, + 'the reload after completion must clear isIndexing on the reused handle' + ); + assert.strictEqual(after.hits, 1, 'the rebuilt index must return the seeded record'); + }); +}); diff --git a/unitTests/resources/indexRestartNumber.test.js b/unitTests/resources/indexRestartNumber.test.js index 42e0101252..8a70e77dd0 100644 --- a/unitTests/resources/indexRestartNumber.test.js +++ b/unitTests/resources/indexRestartNumber.test.js @@ -211,7 +211,7 @@ describe('indexing crash-recovery: restartNumber re-trigger (#1359)', () => { assert.equal(total, N, 'all rows should be indexed after the restartNumber-triggered re-run'); }); - it('treats a store closed by worker shutdown as a benign interruption (resolves, no indexingFailed)', async () => { + it('treats a store closed by worker shutdown as a benign interruption, and still marks the build (harper#2537)', async () => { const TABLE = 'RN_ShutdownInterrupt'; setupTestDBPath(); setMainIsWorker(true); @@ -272,17 +272,16 @@ describe('indexing crash-recovery: restartNumber re-trigger (#1359)', () => { 'a store closed by shutdown must be handled as a benign interruption, not a rejection' ); - // The early-return is a pure no-op on persisted state: it must NOT mark the index - // indexingFailed (the old path tried to persist that against the closed store, which - // both failed loudly and was unnecessary — recovery comes from the restartNumber/PID - // trigger, covered by the tests above). Recovery markers, if the trigger set them, are - // left untouched because the fix writes nothing. + // harper#1359 left this return writing nothing, because recovery came from the restartNumber/PID + // trigger; harper#2536 showed that trigger cannot fire after a process restart under PID 1, so + // declareTable's settle handler persists the marker instead. runIndexing still writes nothing here. const desc = findDescriptor(Tbl, 'tag'); assert.ok(desc, 'tag descriptor should exist after a shutdown-interrupted backfill'); - assert.notEqual( + assert.equal( desc.value.indexingFailed, true, - 'a shutdown-interrupted backfill must NOT be marked indexingFailed' + 'a backfill that returned without completing must be marked, or nothing re-triggers it under PID 1' ); + assert.ok(desc.value.indexingPID, 'the build must still read as incomplete so queries keep refusing'); }); }); diff --git a/unitTests/resources/longLivedTransactions.test.js b/unitTests/resources/longLivedTransactions.test.js index 35a2eaf659..9ddb1d3b00 100644 --- a/unitTests/resources/longLivedTransactions.test.js +++ b/unitTests/resources/longLivedTransactions.test.js @@ -632,19 +632,55 @@ describe('Long-lived transaction reporting (#2471)', () => { } it('names a chain link reachable only through the root under its own native id', async function () { - this.timeout(15000); + this.timeout(30000); if (process.env.HARPER_STORAGE_ENGINE === 'lmdb') this.skip(); await withChainLinks(async (links, childLine, childId, refreshChildWrite) => { resetLongLivedTransactionReportsForTests(); - warnings.length = 0; - await refreshChildWrite(); - await waitFor(() => childLine() !== undefined, 10000); + const missingActiveReport = 'the child must be reported on the first monitor tick after a write'; + let lastChildLine; + let reportedChildLine; + try { + reportedChildLine = await waitFor( + async () => { + const trackedTxns = setTxnExpiration(30000); + warnings.length = 0; + await refreshChildWrite(); + assert.ok(!trackedTxns.has(links[1]), 'the child must remain reachable only through the root chain'); + setTxnExpiration(20); + try { + const monitorRan = await waitFor( + () => warningsMatching('Harper transaction has held').length > 0, + 2000 + ).then( + () => true, + (error) => { + if (error?.code !== 'ERR_ASSERTION') throw error; + return false; + } + ); + if (!monitorRan) return false; + lastChildLine = childLine(); + return /state: [^,]*active/.test(lastChildLine) && lastChildLine; + } finally { + setTxnExpiration(30000); + } + }, + { + timeout: 10000, + message: missingActiveReport, + } + ); + } catch (error) { + if (error?.message === missingActiveReport && lastChildLine) + assert.match(lastChildLine, /state: [^,]*active/, 'the last reported child state must be active'); + throw error; + } assert.match( - childLine(), + reportedChildLine, new RegExp(`transaction ${childId}\\b`), 'the link must be named under its own native id, which is what the sweep line joins to' ); - assert.match(childLine(), /state: [^,]*active/); + assert.match(reportedChildLine, /state: [^,]*active/); }); }); diff --git a/unitTests/resources/models/backendRegistry.test.js b/unitTests/resources/models/backendRegistry.test.js index 774c1a48a6..d69e6216d5 100644 --- a/unitTests/resources/models/backendRegistry.test.js +++ b/unitTests/resources/models/backendRegistry.test.js @@ -11,6 +11,10 @@ const { defineBackend, ModelBackendNotFoundError, ModelBackendRegistrationError, + getBackend, + replaceIfCurrent, + removeIfCurrent, + constructBackend, } = require('#src/resources/models/backendRegistry'); function fakeBackend(name) { @@ -152,4 +156,157 @@ describe('registerBackend', () => { it('throws when the backend lacks a name or capabilities()', () => { assert.throws(() => registerBackend('embedding', 'x', { embed: async () => ({}) }), ModelBackendRegistrationError); }); + + // Config hot reload depends on these (#2344): build what boot would have built, then install it + // only if nothing else claimed the slot meanwhile. + describe('conditional replacement (#2344)', () => { + it('replaces the entry when it is still the expected instance', () => { + const original = fakeBackend('original'); + const next = fakeBackend('next'); + setEmbedding('default', original); + + assert.equal(replaceIfCurrent('embedding', 'default', original, next), true); + assert.equal(getBackend('embedding', 'default'), next); + }); + + it('refuses, and writes nothing, when another writer already changed the entry', () => { + // The slot-clobber race: a plain set resolves by arrival order, so an older credential could + // otherwise overwrite a newer registration. + const original = fakeBackend('original'); + const interloper = fakeBackend('interloper'); + const next = fakeBackend('next'); + setEmbedding('default', original); + setEmbedding('default', interloper); // a concurrent structural reload or registerBackend + + assert.equal(replaceIfCurrent('embedding', 'default', original, next), false); + assert.equal(getBackend('embedding', 'default'), interloper, 'the other writer survives'); + }); + + it('treats an absent entry as expected-undefined', () => { + const next = fakeBackend('next'); + assert.equal(replaceIfCurrent('generative', 'fresh', undefined, next), true); + assert.equal(getBackend('generative', 'fresh'), next); + }); + + it('does not install the entry it was asked to replace when the expectation fails', () => { + const original = fakeBackend('original'); + setEmbedding('default', original); + assert.equal( + replaceIfCurrent('embedding', 'default', fakeBackend('never-installed'), fakeBackend('next')), + false + ); + assert.equal(getBackend('embedding', 'default'), original); + }); + + it('builds a backend through its registration path without installing it', async () => { + // Rotation needs the constructed instance so the install can be conditional; the factory would + // otherwise install unconditionally as its last act. + const built = fakeBackend('built'); + const { backend } = await constructBackend('embedding', 'default', () => setEmbedding('default', built)); + + assert.equal(backend, built, 'the constructed backend is handed back'); + assert.equal(getBackend('embedding', 'default'), undefined, 'and NOT installed'); + }); + + it("defers a factory's secondary registrations to the caller instead of installing mid-construction", async () => { + // A helper installed live during a slow construction would pair a NEW helper with the OLD + // primary for any request arriving in the window. + const helper = fakeBackend('helper'); + const built = fakeBackend('built'); + const { backend, extras } = await constructBackend('embedding', 'default', () => { + setEmbedding('default-helper', helper); + setEmbedding('default', built); + }); + + assert.equal(backend, built); + assert.deepEqual(extras, [{ kind: 'embedding', logicalName: 'default-helper', backend: helper }]); + assert.equal(getBackend('embedding', 'default-helper'), undefined, 'helper deferred, not installed'); + assert.equal(getBackend('embedding', 'default'), undefined); + }); + + it('lets async work spawned by a factory install normally after construction ends', async () => { + // The ALS context outlives the run() for async descendants; deactivation is what keeps a + // late same-slot registration from being silently swallowed into a dead capture. + const built = fakeBackend('built'); + const lateBackend = fakeBackend('late'); + let late; + await constructBackend('embedding', 'default', () => { + late = (async () => { + await new Promise((resolve) => setImmediate(resolve)); + setEmbedding('default', lateBackend); + })(); + setEmbedding('default', built); + }); + await late; + + assert.equal(getBackend('embedding', 'default'), lateBackend, 'the late install reaches the registry'); + }); + + it('clears the capture scope even when the factory throws', async () => { + await assert.rejects(() => + constructBackend('embedding', 'default', () => { + throw new Error('factory blew up'); + }) + ); + // A leaked scope would silently swallow the next registration for this name. + const after = fakeBackend('after'); + setEmbedding('default', after); + assert.equal(getBackend('embedding', 'default'), after); + }); + + it('does not divert an unrelated registration that lands while a construction is awaiting', async () => { + // The capture is scoped to the async context: only the factory's OWN installs defer. A + // concurrent registration from elsewhere must install live, not be swallowed or deferred. + const unrelated = fakeBackend('unrelated'); + const built = fakeBackend('built'); + let release; + const gate = new Promise((resolve) => (release = resolve)); + const constructing = constructBackend('embedding', 'default', async () => { + await gate; + setEmbedding('default', built); + }); + + setEmbedding('other', unrelated); + assert.equal(getBackend('embedding', 'other'), unrelated, 'installed live, mid-construction'); + + release(); + const { backend, extras } = await constructing; + assert.equal(backend, built); + assert.deepEqual(extras, [], 'the outside registration was not captured'); + assert.equal(getBackend('embedding', 'default'), undefined); + }); + + it('keeps two concurrent constructions separate', async () => { + // Two entries rotating at once must not collide; a module-global capture slot made the second + // one fail and drop its event. + const first = fakeBackend('first'); + const second = fakeBackend('second'); + const [a, b] = await Promise.all([ + constructBackend('embedding', 'one', async () => { + await Promise.resolve(); + setEmbedding('one', first); + }), + constructBackend('embedding', 'two', async () => { + setEmbedding('two', second); + }), + ]); + + assert.equal(a.backend, first); + assert.equal(b.backend, second); + assert.equal(getBackend('embedding', 'one'), undefined); + assert.equal(getBackend('embedding', 'two'), undefined); + }); + + it('removes an entry only while it is still the expected instance', () => { + const mine = fakeBackend('mine'); + setEmbedding('default', mine); + assert.equal(removeIfCurrent('embedding', 'default', mine), true); + assert.equal(getBackend('embedding', 'default'), undefined); + + const theirs = fakeBackend('theirs'); + setEmbedding('default', theirs); + assert.equal(removeIfCurrent('embedding', 'default', mine), false, 'not ours to remove'); + assert.equal(getBackend('embedding', 'default'), theirs); + }); + }); }); diff --git a/unitTests/resources/models/fixtures/counting-backend-module.cjs b/unitTests/resources/models/fixtures/counting-backend-module.cjs new file mode 100644 index 0000000000..172e99a876 --- /dev/null +++ b/unitTests/resources/models/fixtures/counting-backend-module.cjs @@ -0,0 +1,19 @@ +'use strict'; +// Reload-test fixture: counts factory invocations so a test can assert an entry was (or was not) +// reconstructed. Same register-function shape as embed-backend-module.cjs. +const { registerBackend, defineBackend } = require('#src/resources/models/backendRegistry'); + +module.exports = function register({ logicalName, kind, config }) { + globalThis.__countingBackendBuilds = (globalThis.__countingBackendBuilds ?? 0) + 1; + registerBackend( + kind, + logicalName, + defineBackend({ + name: `counting:${config.model ?? 'test'}`, + embed: async (input) => { + const texts = Array.isArray(input) ? input : [input]; + return { status: 'completed', output: texts.map(() => Float32Array.from([1, 2, 3])) }; + }, + }) + ); +}; diff --git a/unitTests/resources/models/fixtures/helper-backend-module.cjs b/unitTests/resources/models/fixtures/helper-backend-module.cjs new file mode 100644 index 0000000000..d657549fe3 --- /dev/null +++ b/unitTests/resources/models/fixtures/helper-backend-module.cjs @@ -0,0 +1,18 @@ +'use strict'; +// Reload-test fixture: registers a helper backend under a second name before its primary, +// optionally parking on globalThis.__helperGate so a test can observe mid-construction state. +const { registerBackend, defineBackend } = require('#src/resources/models/backendRegistry'); + +const embed = async (input) => ({ + status: 'completed', + output: (Array.isArray(input) ? input : [input]).map(() => Float32Array.from([1, 2, 3])), +}); + +module.exports = async function register({ logicalName, kind, config }) { + registerBackend(kind, config.helperName ?? `${logicalName}-helper`, defineBackend({ name: 'helper', embed })); + if (globalThis.__helperGate) { + globalThis.__helperGateReached = true; + await globalThis.__helperGate; + } + registerBackend(kind, logicalName, defineBackend({ name: `primary:${config.model ?? 'test'}`, embed })); +}; diff --git a/unitTests/resources/models/fixtures/wrapping-backend-module.cjs b/unitTests/resources/models/fixtures/wrapping-backend-module.cjs new file mode 100644 index 0000000000..1f4dd21035 --- /dev/null +++ b/unitTests/resources/models/fixtures/wrapping-backend-module.cjs @@ -0,0 +1,20 @@ +'use strict'; +// Boot-composition fixture: records whether the entry named by config.wraps was already +// installed when this factory ran — the order-dependent pattern boot must preserve. +const { registerBackend, defineBackend, getBackend } = require('#src/resources/models/backendRegistry'); + +module.exports = function register({ logicalName, kind, config }) { + const base = getBackend(kind, config.wraps ?? 'base'); + globalThis.__wrapperSawBase = base !== undefined; + registerBackend( + kind, + logicalName, + defineBackend({ + name: `wrapper:${config.wraps ?? 'base'}`, + embed: async (input) => { + const texts = Array.isArray(input) ? input : [input]; + return { status: 'completed', output: texts.map(() => Float32Array.from([9])) }; + }, + }) + ); +}; diff --git a/unitTests/resources/models/modelsConfigReload.test.js b/unitTests/resources/models/modelsConfigReload.test.js new file mode 100644 index 0000000000..e4ca83d4ab --- /dev/null +++ b/unitTests/resources/models/modelsConfigReload.test.js @@ -0,0 +1,871 @@ +'use strict'; + +// Hot reload of the `models:` config block (#2344): changed entries are rebuilt through the same +// factories boot uses and swapped in atomically, removed entries stop serving, application +// overrides keep their precedence, and a file rewrite reaches live requests with no restart. + +const assert = require('node:assert'); +const { tmpdir } = require('node:os'); +const { join } = require('node:path'); +const { writeFileSync, mkdtempSync, rmSync } = require('node:fs'); +const { stringify } = require('yaml'); +const { waitFor } = require('../../waitFor.js'); +const { getSharedRootConfigWatcher, RootConfigWatcher } = require('#src/config/RootConfigWatcher'); +const { + bootstrapModels, + applyModelsConfig, + startModelsConfigHotReload, + stopModelsConfigHotReload, + resetModelsProjection, +} = require('#src/resources/models/bootstrap'); +const { + clearRegistry, + getBackend, + setEmbedding, + removeIfCurrent, + resolveEmbedding, + ModelBackendNotFoundError, +} = require('#src/resources/models/backendRegistry'); +const { getRouter, clearRouting } = require('#src/resources/models/routing'); + +// Captures the Authorization header a real embed call sends. The backend takes `globalThis.fetch` +// at construction time, so this must be installed before the backend under test is built. +function installFetchCapture() { + const sent = []; + const original = globalThis.fetch; + globalThis.fetch = async (url, init) => { + sent.push(init?.headers?.Authorization); + return new Response(JSON.stringify({ data: [{ embedding: [0.1, 0.2] }] }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }); + }; + return { + sent, + restore() { + globalThis.fetch = original; + }, + }; +} + +const openaiEntry = (apiKey, extra = {}) => ({ backend: 'openai', model: 'text-embedding-3-small', apiKey, ...extra }); +const block = (entries) => ({ embedding: entries }); + +describe('models config hot reload (#2344)', () => { + beforeEach(() => { + clearRegistry(); + clearRouting(); + resetModelsProjection(); + }); + + afterEach(() => { + stopModelsConfigHotReload(); + }); + + describe('applyModelsConfig', () => { + it('serves later requests with the rewritten credential, through the normal resolution path', async () => { + const captured = installFetchCapture(); + try { + await bootstrapModels({ models: block({ default: openaiEntry('sk-first') }) }); + await resolveEmbedding('default').embed('hello', { model: 'text-embedding-3-small' }); + assert.equal(captured.sent.at(-1), 'Bearer sk-first'); + + await applyModelsConfig(block({ default: openaiEntry('sk-second') })); + + // Resolved the same way a request resolves it — no restart, no caller re-registration. + await resolveEmbedding('default').embed('hello', { model: 'text-embedding-3-small' }); + assert.equal(captured.sent.at(-1), 'Bearer sk-second', 'the new credential reaches the provider'); + } finally { + captured.restore(); + } + }); + + it('does not reconstruct an unchanged entry', async () => { + await bootstrapModels({ models: block({ default: openaiEntry('sk-1'), other: openaiEntry('sk-2') }) }); + const defaultBefore = getBackend('embedding', 'default'); + const otherBefore = getBackend('embedding', 'other'); + + await applyModelsConfig(block({ default: openaiEntry('sk-1'), other: openaiEntry('sk-2b') })); + + assert.equal(getBackend('embedding', 'default'), defaultBefore, 'unchanged entry keeps its instance'); + assert.notEqual(getBackend('embedding', 'other'), otherBefore, 'changed entry was rebuilt'); + }); + + it('stops serving an entry removed from the block, leaving the others alone', async () => { + await bootstrapModels({ models: block({ default: openaiEntry('sk-1'), retired: openaiEntry('sk-2') }) }); + const defaultBefore = getBackend('embedding', 'default'); + + await applyModelsConfig(block({ default: openaiEntry('sk-1') })); + + assert.throws(() => resolveEmbedding('retired'), ModelBackendNotFoundError); + assert.equal(getBackend('embedding', 'default'), defaultBefore); + }); + + it('leaves an application override in place on reload AND on removal', async () => { + await bootstrapModels({ models: block({ default: openaiEntry('sk-1') }) }); + const appOwned = { name: 'app-policy-backend', capabilities: () => ({ embed: true }) }; + setEmbedding('default', appOwned); + + await applyModelsConfig(block({ default: openaiEntry('sk-2') })); + assert.equal(getBackend('embedding', 'default'), appOwned, 'reload must not clobber the override'); + + await applyModelsConfig(block({})); + assert.equal(getBackend('embedding', 'default'), appOwned, 'removal must not delete the override'); + }); + + it("publishes a factory's helper registration with its primary, not mid-construction", async () => { + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + let release; + let applying; + globalThis.__helperGate = new Promise((resolve) => (release = resolve)); + globalThis.__helperGateReached = false; + try { + applying = bootstrapModels({ models: block({ default: { backend: helperModule, model: 'm1' } }) }); + await waitFor(() => globalThis.__helperGateReached, { message: 'construction never started' }); + + // Mid-construction: neither the helper nor the primary may be visible yet. + assert.equal(getBackend('embedding', 'default-helper'), undefined, 'helper must not publish early'); + assert.equal(getBackend('embedding', 'default'), undefined); + + release(); + await applying; + assert.ok(getBackend('embedding', 'default'), 'primary published'); + assert.ok(getBackend('embedding', 'default-helper'), 'helper published with it'); + } finally { + // A thrown assertion above must not leave the apply chain parked on the gate — every + // later test's applies queue behind it and the whole suite hangs. + release(); + await applying?.catch(() => {}); + delete globalThis.__helperGate; + delete globalThis.__helperGateReached; + } + }); + + it('removes a helper together with its removed entry on re-bootstrap; reload refuses both', async () => { + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + await bootstrapModels({ models: block({ default: { backend: helperModule, model: 'm1' } }) }); + const entry = getBackend('embedding', 'default'); + assert.ok(getBackend('embedding', 'default-helper'), 'helper installed with its entry'); + + // Reload: module entries are restart-managed in both directions. + await applyModelsConfig(block({})); + assert.equal(getBackend('embedding', 'default'), entry, 'reload keeps the module entry'); + assert.ok(getBackend('embedding', 'default-helper'), 'and its helper'); + + // Re-bootstrap (the restart-shaped event): removal takes effect, helper cascades. + await bootstrapModels({ models: block({}) }); + assert.equal(getBackend('embedding', 'default'), undefined); + assert.equal(getBackend('embedding', 'default-helper'), undefined, 'helper removed with its entry'); + }); + + it('removes a stale helper when a re-bootstrap registers a different one', async () => { + // Module factories run only at boot now, so helper rotation is a restart-shaped event. + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + await bootstrapModels({ + models: block({ default: { backend: helperModule, model: 'm1', helperName: 'helper-a' } }), + }); + assert.ok(getBackend('embedding', 'helper-a')); + + await bootstrapModels({ + models: block({ default: { backend: helperModule, model: 'm2', helperName: 'helper-b' } }), + }); + + assert.equal(getBackend('embedding', 'helper-a'), undefined, 'stale helper removed'); + assert.ok(getBackend('embedding', 'helper-b'), 'replacement helper installed'); + }); + + it('a module entry rename on reload keeps the old name serving instead of half-applying', async () => { + // Refusing the added name while removing the old one would turn a rename into a bare + // removal; module entries are restart-managed in both directions. + const counting = join(__dirname, 'fixtures', 'counting-backend-module.cjs'); + await bootstrapModels({ models: block({ old: { backend: counting, model: 'm1' } }) }); + const before = getBackend('embedding', 'old'); + + await applyModelsConfig(block({ renamed: { backend: counting, model: 'm1' } })); + + assert.equal(getBackend('embedding', 'old'), before, 'old name keeps serving'); + assert.equal(getBackend('embedding', 'renamed'), undefined, 'new name waits for a restart'); + }); + + it('a removed module-backed primary keeps serving AND keeps its fallback routing', async () => { + // A module-backed entry is restart-managed on removal, so the primary keeps serving. Its + // fallback group must survive the routing rebuild too, or removing/renaming it silently + // disables failover while the primary still answers — the half-apply this path prevents. + const counting = join(__dirname, 'fixtures', 'counting-backend-module.cjs'); + await bootstrapModels({ + models: block({ + default: { backend: counting, model: 'm1', fallback: ['backup'] }, + backup: openaiEntry('sk-backup'), + }), + }); + const primary = getBackend('embedding', 'default'); + assert.ok(primary, 'module-backed primary installed at boot'); + + await applyModelsConfig(block({ backup: openaiEntry('sk-backup') })); + + assert.equal(getBackend('embedding', 'default'), primary, 'module-backed primary keeps serving'); + const candidates = getRouter().route({ kind: 'embedding', logicalName: 'default', requires: [] }); + assert.strictEqual(candidates.length, 2, 'fallback routing survives the restart-managed removal'); + assert.strictEqual(candidates[0], primary, 'the retained primary still leads'); + assert.strictEqual( + candidates[1], + getBackend('embedding', 'backup'), + 'router still returns the previous fallback candidate' + ); + }); + + it('refuses a module-backed→built-in change on reload, keeping the module and its fallback', async () => { + // Rewriting a module-backed entry to a built-in is still a change of a restart-managed + // entry: it must not live-replace the custom backend (which would drop its helpers with no + // disposal). The old module keeps serving with its routing until a restart. + const counting = join(__dirname, 'fixtures', 'counting-backend-module.cjs'); + await bootstrapModels({ + models: block({ + default: { backend: counting, model: 'm1', fallback: ['backup'] }, + backup: openaiEntry('sk-backup'), + }), + }); + const primary = getBackend('embedding', 'default'); + assert.ok(primary, 'module-backed default installed at boot'); + + await applyModelsConfig( + block({ default: openaiEntry('sk-new', { fallback: ['backup'] }), backup: openaiEntry('sk-backup') }) + ); + + assert.equal(getBackend('embedding', 'default'), primary, 'module kept; the built-in was not swapped in'); + const candidates = getRouter().route({ kind: 'embedding', logicalName: 'default', requires: [] }); + assert.strictEqual(candidates.length, 2, 'the retained module keeps its fallback routing'); + assert.strictEqual(candidates[1], getBackend('embedding', 'backup'), 'router still returns the fallback'); + }); + + it('refuses to run a module factory on reload, retaining the previous projection', async () => { + // A module factory may compose with other entries; staged reload construction cannot + // honor that ordering, so changing one keeps restart semantics (review decision). + const counting = join(__dirname, 'fixtures', 'counting-backend-module.cjs'); + globalThis.__countingBackendBuilds = 0; + try { + await bootstrapModels({ models: block({ default: { backend: counting, model: 'm1' } }) }); + const before = getBackend('embedding', 'default'); + assert.equal(globalThis.__countingBackendBuilds, 1); + + await applyModelsConfig(block({ default: { backend: counting, model: 'm2' } })); + + assert.equal(globalThis.__countingBackendBuilds, 1, 'factory not re-run on reload'); + assert.equal(getBackend('embedding', 'default'), before, 'previous backend retained'); + } finally { + delete globalThis.__countingBackendBuilds; + } + }); + + it('boot installs entries sequentially, so a later module factory can wrap an earlier one', async () => { + // The order-dependent composition boot has always allowed: `cached` resolves `base` at + // factory time. Staged boot construction broke this (review finding); per-entry publish + // restores it. + const wrapping = join(__dirname, 'fixtures', 'wrapping-backend-module.cjs'); + globalThis.__wrapperSawBase = undefined; + try { + await bootstrapModels({ + models: block({ + base: openaiEntry('sk-base'), + cached: { backend: wrapping, wraps: 'base' }, + }), + }); + + assert.equal(globalThis.__wrapperSawBase, true, 'the wrapper factory saw its base installed'); + assert.ok(getBackend('embedding', 'cached')); + } finally { + delete globalThis.__wrapperSawBase; + } + }); + + it('does not clobber an application override of a helper name', async () => { + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + await bootstrapModels({ models: block({ default: { backend: helperModule, model: 'm1' } }) }); + const appOwned = { name: 'app-helper', capabilities: () => ({ embed: true }) }; + setEmbedding('default-helper', appOwned); + + await applyModelsConfig(block({ default: { backend: helperModule, model: 'm2' } })); + + assert.equal(getBackend('embedding', 'default-helper'), appOwned, 'helper override survives the rebuild'); + assert.ok(getBackend('embedding', 'default'), 'the primary itself still rotated'); + }); + + it('treats a snapshot with no models key as a no-op, not a total removal', async () => { + // A non-atomic in-place rewrite can be observed as a valid YAML prefix that has not reached + // the models block yet; adopting that as authoritative would remove every backend. + await bootstrapModels({ models: block({ default: openaiEntry('sk-1') }) }); + const before = getBackend('embedding', 'default'); + + await applyModelsConfig(undefined); + + assert.equal(getBackend('embedding', 'default'), before, 'projection untouched'); + }); + + it('rejects a reload the boot schema would reject, keeping the previous projection', async () => { + await bootstrapModels({ models: block({ default: openaiEntry('sk-1') }) }); + const before = getBackend('embedding', 'default'); + + // Boot would refuse both of these in configValidator; a hot reload must not be laxer. + await applyModelsConfig(block({ default: openaiEntry('sk-2', { requestTimeoutMs: -1 }) })); + assert.equal(getBackend('embedding', 'default'), before, 'invalid field value rejected'); + + await applyModelsConfig(block({ default: { backend: 'openai', apiKey: 'sk-2', baseUrI: 'typo' } })); + assert.equal(getBackend('embedding', 'default'), before, 'unknown field name rejected'); + }); + + it('rejects models: null on reload exactly as boot validation would, and {} is the clear', async () => { + // Boot's validator refuses `models: null`, so accepting it live would leave a file on disk + // that the next restart rejects. The sole off-switch is an empty (or shrunken) block. + await bootstrapModels({ models: block({ default: openaiEntry('sk-1') }) }); + const before = getBackend('embedding', 'default'); + + await applyModelsConfig(null); + assert.equal(getBackend('embedding', 'default'), before, 'null rejected, prior projection kept'); + + await applyModelsConfig({ embedding: {} }); + assert.equal(getBackend('embedding', 'default'), undefined, 'an explicit empty block clears'); + }); + + it('tolerates a boot-legal sibling key next to embedding/generative on reload', async () => { + // Boot validates with allowUnknown, so `models.debug: true` boots; a reload rejecting it + // would silently block every future rotation in that file. The capture must precede the + // apply — the backend takes globalThis.fetch at construction. + const captured = installFetchCapture(); + try { + await bootstrapModels({ models: block({ default: openaiEntry('sk-1') }) }); + + await applyModelsConfig({ ...block({ default: openaiEntry('sk-2') }), debug: true }); + + await resolveEmbedding('default').embed('x', { model: 'text-embedding-3-small' }); + assert.equal(captured.sent.at(-1), 'Bearer sk-2', 'the rotation applied despite the sibling key'); + } finally { + captured.restore(); + } + }); + + it('restores a suppressed helper the moment its claiming entry is removed', async () => { + // Boot with a factory helper AND a config entry of the same name: the entry wins, the + // helper's record is suppressed rather than forgotten, and dropping the entry restores the + // helper — the live registry matches a restart with the same final config. + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + await bootstrapModels({ + models: block({ + 'default': { backend: helperModule, model: 'm1' }, + 'default-helper': openaiEntry('sk-own'), + }), + }); + const winner = getBackend('embedding', 'default-helper'); + assert.ok(winner && winner.name !== 'helper', 'the config entry owns the name at boot'); + + await applyModelsConfig(block({ default: { backend: helperModule, model: 'm1' } })); + + const restored = getBackend('embedding', 'default-helper'); + assert.ok(restored, 'the helper is back the moment the claiming entry is removed'); + assert.notEqual(restored, winner, 'and it is the factory helper, not the removed entry'); + }); + + it('releases a helper whose claiming entry never won the name, once that entry is removed', async () => { + // The claim suppresses the helper before the entry's own swap is known to win. When an + // application override already holds the name the swap loses, and dropping the entry later + // must still hand the name back to the helper — otherwise its record stays suppressed forever. + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + await bootstrapModels({ models: block({ default: { backend: helperModule, model: 'm1' } }) }); + const helper = getBackend('embedding', 'default-helper'); + assert.ok(helper && helper.name === 'helper', 'the factory helper serves its name at boot'); + + const appOwned = { name: 'app-policy-backend', capabilities: () => ({ embed: true }) }; + setEmbedding('default-helper', appOwned); + await applyModelsConfig( + block({ 'default': { backend: helperModule, model: 'm1' }, 'default-helper': openaiEntry('sk-own') }) + ); + assert.equal(getBackend('embedding', 'default-helper'), appOwned, 'the config entry lost to the override'); + + // The override retires, then the config drops the entry that claimed the name. + assert.ok(removeIfCurrent('embedding', 'default-helper', appOwned)); + await applyModelsConfig(block({ default: { backend: helperModule, model: 'm1' } })); + + const restored = getBackend('embedding', 'default-helper'); + assert.ok(restored && restored.name === 'helper', 'the helper is back once its claimant is gone'); + }); + + it('lets a config entry claim a name held by a projection-installed helper', async () => { + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + await bootstrapModels({ models: block({ default: { backend: helperModule, model: 'm1' } }) }); + const helperInstance = getBackend('embedding', 'default-helper'); + assert.ok(helperInstance); + + // Config outranks the projection's own helpers — but never an application override. + await applyModelsConfig( + block({ 'default': { backend: helperModule, model: 'm1' }, 'default-helper': openaiEntry('sk-own') }) + ); + + const claimed = getBackend('embedding', 'default-helper'); + assert.ok(claimed && claimed !== helperInstance, 'the config entry now owns the name'); + + // The parent no longer tracks it: removing the parent must not remove the claimed entry. + await applyModelsConfig(block({ 'default-helper': openaiEntry('sk-own') })); + assert.equal(getBackend('embedding', 'default-helper'), claimed); + }); + + it('a module entry change under an override is refused, leaving override and helper intact', async () => { + // Module factories no longer run on reload, so nothing rotates here by design; the refusal + // must leave every installed piece exactly as it was. + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + await bootstrapModels({ + models: block({ default: { backend: helperModule, model: 'm1', helperName: 'helper-a' } }), + }); + const helperA = getBackend('embedding', 'helper-a'); + const appOwned = { name: 'app-policy-backend', capabilities: () => ({ embed: true }) }; + setEmbedding('default', appOwned); + + await applyModelsConfig(block({ default: { backend: helperModule, model: 'm2', helperName: 'helper-b' } })); + + assert.equal(getBackend('embedding', 'default'), appOwned, 'primary override intact'); + assert.equal(getBackend('embedding', 'helper-a'), helperA, 'existing helper untouched'); + assert.equal(getBackend('embedding', 'helper-b'), undefined, 'no factory ran on reload'); + }); + + it('a failed rebuild keeps the routing that was applied with the serving backend', async () => { + await bootstrapModels({ + models: block({ + default: openaiEntry('sk-1', { fallback: ['safe'] }), + safe: openaiEntry('sk-s'), + other: openaiEntry('sk-o'), + }), + }); + const before = getBackend('embedding', 'default'); + + // Schema-valid but unbuildable (module cannot be imported), with a re-pointed fallback: the + // old backend keeps serving, so it keeps the routing it was applied with. + await applyModelsConfig( + block({ + default: { backend: './nonexistent-backend.cjs', fallback: ['other'] }, + safe: openaiEntry('sk-s'), + other: openaiEntry('sk-o'), + }) + ); + + assert.equal(getBackend('embedding', 'default'), before, 'previous backend retained'); + const candidates = getRouter().route({ kind: 'embedding', logicalName: 'default', requires: [] }); + assert.strictEqual(candidates.length, 2); + assert.strictEqual(candidates[1], getBackend('embedding', 'safe'), 'old fallback retained, not the new one'); + }); + + it('a re-bootstrap with a shrunk config removes entries the new config dropped', async () => { + // A long-lived process can re-run boot (a component reload cycle); the projection survives + // bootstrapModels, so absence stays authoritative there too. + await bootstrapModels({ models: block({ default: openaiEntry('sk-1'), extra: openaiEntry('sk-2') }) }); + assert.ok(getBackend('embedding', 'extra')); + + await bootstrapModels({ models: block({ default: openaiEntry('sk-1') }) }); + + assert.equal(getBackend('embedding', 'extra'), undefined, 'dropped entry removed on re-boot'); + assert.ok(getBackend('embedding', 'default'), 'kept entry still serves'); + }); + + it('keeps config-defined fallback routing when an override owns a now-malformed entry', async () => { + await bootstrapModels({ + models: block({ default: openaiEntry('sk-1', { fallback: ['alt'] }), alt: openaiEntry('sk-alt') }), + }); + const appOwned = { + name: 'app-policy-backend', + capabilities: () => ({ embed: true }), + embed: async () => ({ status: 'completed', output: [Float32Array.from([1])] }), + }; + setEmbedding('default', appOwned); + + await applyModelsConfig(block({ default: { backend: '' }, alt: openaiEntry('sk-alt') })); + + const candidates = getRouter().route({ kind: 'embedding', logicalName: 'default', requires: [] }); + assert.strictEqual(candidates[0], appOwned, 'the override serves'); + assert.strictEqual(candidates.length, 2, 'and keeps its config-defined fallback'); + }); + + it('does not rebuild an overridden entry on every unrelated reload', async () => { + const countingModule = join(__dirname, 'fixtures', 'counting-backend-module.cjs'); + const countingEntry = { backend: countingModule, model: 'm1' }; + globalThis.__countingBackendBuilds = 0; + try { + await bootstrapModels({ models: block({ default: countingEntry }) }); + assert.equal(globalThis.__countingBackendBuilds, 1); + + const appOwned = { name: 'app-policy-backend', capabilities: () => ({ embed: true }) }; + setEmbedding('default', appOwned); + + // An UNCHANGED entry under an override must not rebuild at all — any occupant satisfies a + // reload's unchanged-skip. + const buildsBeforeUnchanged = globalThis.__countingBackendBuilds; + await applyModelsConfig(block({ default: countingEntry })); + assert.equal(globalThis.__countingBackendBuilds, buildsBeforeUnchanged, 'no rebuild when unchanged'); + + // A change under an override loses the swap once (and records that), then unchanged + // reloads stop reconstructing. + await applyModelsConfig(block({ default: { ...countingEntry, model: 'm2' } })); + const buildsAfterChange = globalThis.__countingBackendBuilds; + await applyModelsConfig(block({ default: { ...countingEntry, model: 'm2' } })); + await applyModelsConfig(block({ default: { ...countingEntry, model: 'm2' } })); + + assert.equal(globalThis.__countingBackendBuilds, buildsAfterChange, 'no rebuild on unchanged reloads'); + assert.equal(getBackend('embedding', 'default'), appOwned, 'override still in place'); + } finally { + delete globalThis.__countingBackendBuilds; + } + }); + + it('keeps the previous backend and its fallback routing when a rebuild fails', async () => { + await bootstrapModels({ + models: block({ + default: openaiEntry('sk-1', { fallback: ['alt'] }), + alt: openaiEntry('sk-alt'), + }), + }); + const before = getBackend('embedding', 'default'); + + // apiKey removed → openai's constructor rejects → the old entry must keep serving. + await applyModelsConfig( + block({ default: openaiEntry(undefined, { fallback: ['alt'] }), alt: openaiEntry('sk-alt') }) + ); + + assert.equal(getBackend('embedding', 'default'), before, 'previous backend retained'); + const candidates = getRouter().route({ kind: 'embedding', logicalName: 'default', requires: [] }); + assert.strictEqual(candidates.length, 2, 'fallback routing survives a failed rebuild of a still-serving entry'); + assert.strictEqual(candidates[0], before); + assert.strictEqual(candidates[1], getBackend('embedding', 'alt')); + }); + + it('updates fallback routing with the block', async () => { + await bootstrapModels({ + models: block({ + default: openaiEntry('sk-1', { fallback: ['a'] }), + a: openaiEntry('sk-a'), + b: openaiEntry('sk-b'), + }), + }); + + await applyModelsConfig( + block({ default: openaiEntry('sk-1', { fallback: ['b'] }), a: openaiEntry('sk-a'), b: openaiEntry('sk-b') }) + ); + + const candidates = getRouter().route({ kind: 'embedding', logicalName: 'default', requires: [] }); + assert.strictEqual(candidates.length, 2); + assert.strictEqual(candidates[0], getBackend('embedding', 'default')); + assert.strictEqual(candidates[1], getBackend('embedding', 'b'), 'the group now routes to b, not a'); + }); + + it('keeps boot semantics when a reload coalesces over a queued boot', async () => { + // While an apply is in flight, a queued bootstrapModels can be overwritten by a watcher + // event; the newest block wins, but the boot flag must stick or the re-bootstrap silently + // loses its overwrite contract. + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + const appOwned = { name: 'app-policy-backend', capabilities: () => ({ embed: true }) }; + let release; + let inFlight; + globalThis.__helperGate = new Promise((resolve) => (release = resolve)); + globalThis.__helperGateReached = false; + try { + setEmbedding('victim', appOwned); + inFlight = bootstrapModels({ models: block({ slow: { backend: helperModule, model: 'm1' } }) }); + await waitFor(() => globalThis.__helperGateReached, { message: 'in-flight apply never started' }); + + const boot = bootstrapModels({ models: block({ victim: openaiEntry('sk-boot') }) }); + const reload = applyModelsConfig(block({ victim: openaiEntry('sk-boot') })); + + release(); + await Promise.all([inFlight, boot, reload]); + + assert.notEqual(getBackend('embedding', 'victim'), appOwned, 'boot overwrote the occupant'); + } finally { + release(); + await inFlight?.catch(() => {}); + delete globalThis.__helperGate; + delete globalThis.__helperGateReached; + } + }); + + it('a boot supersedes a reload queued before it, and is refined by one queued after', async () => { + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + let release; + let inFlight; + globalThis.__helperGate = new Promise((resolve) => (release = resolve)); + globalThis.__helperGateReached = false; + try { + inFlight = bootstrapModels({ models: block({ slow: { backend: helperModule, model: 'm1' } }) }); + await waitFor(() => globalThis.__helperGateReached, { message: 'in-flight apply never started' }); + + // Queued in this order: stale reload, then a NEWER boot. Draining the stale reload after + // the boot would resurrect `stale` and refine the newer truth backwards. + const staleReload = applyModelsConfig( + block({ slow: { backend: helperModule, model: 'm1' }, stale: openaiEntry('sk-stale') }) + ); + const boot = bootstrapModels({ + models: block({ slow: { backend: helperModule, model: 'm1' }, keeper: openaiEntry('sk-boot') }), + }); + + release(); + await Promise.all([inFlight, staleReload, boot]); + + assert.ok(getBackend('embedding', 'keeper'), 'the boot applied'); + assert.equal(getBackend('embedding', 'stale'), undefined, 'the older reload was discarded'); + + // And the other half of the contract: a reload queued AFTER the boot still refines it. + await applyModelsConfig( + block({ + slow: { backend: helperModule, model: 'm1' }, + keeper: openaiEntry('sk-boot'), + late: openaiEntry('sk-late'), + }) + ); + assert.ok(getBackend('embedding', 'late'), 'a post-boot reload applied its distinct state'); + } finally { + release(); + await inFlight?.catch(() => {}); + delete globalThis.__helperGate; + delete globalThis.__helperGateReached; + } + }); + + it('a watcher block coalescing over a queued boot is NOT laundered through boot semantics', async () => { + // The two lanes must stay separate: boot applies its own block with overwrite authority, and + // the newer reload block still faces reload validation and the missing-key no-op — a sticky + // flag merging them would let a partial-write prefix tear everything down as "boot". + const helperModule = join(__dirname, 'fixtures', 'helper-backend-module.cjs'); + let release; + let inFlight; + globalThis.__helperGate = new Promise((resolve) => (release = resolve)); + globalThis.__helperGateReached = false; + try { + inFlight = bootstrapModels({ models: block({ slow: { backend: helperModule, model: 'm1' } }) }); + await waitFor(() => globalThis.__helperGateReached, { message: 'in-flight apply never started' }); + + // Boot's block is authoritative for the whole map, so it must carry `slow` itself — + // what's under test is the reload lane, not boot's removal semantics. + const boot = bootstrapModels({ + models: block({ keeper: openaiEntry('sk-boot'), slow: { backend: helperModule, model: 'm1' } }), + }); + // A raw watcher snapshot with no models key lands on top of the queued boot. + const reload = applyModelsConfig(undefined); + + release(); + await Promise.all([inFlight, boot, reload]); + + assert.ok(getBackend('embedding', 'keeper'), 'boot applied its own block'); + // The missing-key snapshot was a no-op, not a boot-authority removal of everything. + assert.ok(getBackend('embedding', 'slow'), 'the missing-key reload removed nothing'); + } finally { + release(); + await inFlight?.catch(() => {}); + delete globalThis.__helperGate; + delete globalThis.__helperGateReached; + } + }); + + it('coalesces rapid applies to the latest block', async () => { + const captured = installFetchCapture(); + try { + await bootstrapModels({ models: block({ default: openaiEntry('sk-1') }) }); + + const first = applyModelsConfig(block({ default: openaiEntry('sk-2') })); + const second = applyModelsConfig(block({ default: openaiEntry('sk-3') })); + await Promise.all([first, second]); + + await resolveEmbedding('default').embed('x', { model: 'text-embedding-3-small' }); + assert.equal(captured.sent.at(-1), 'Bearer sk-3', 'the newest queued block is what applied'); + } finally { + captured.restore(); + } + }); + + it('rejects a block with a malformed entry wholesale, retaining backends AND routing', async () => { + // Schema validation front-runs per-entry handling on reload: one malformed entry rejects the + // snapshot, so nothing is removed and routing is untouched — a broken rewrite cannot + // half-apply. + await bootstrapModels({ + models: block({ default: openaiEntry('sk-1', { fallback: ['alt'] }), alt: openaiEntry('sk-alt') }), + }); + const before = getBackend('embedding', 'default'); + + await applyModelsConfig(block({ default: { backend: '' }, alt: openaiEntry('sk-alt') })); + + assert.equal(getBackend('embedding', 'default'), before, 'nothing removed'); + assert.ok(getBackend('embedding', 'alt'), 'sibling untouched'); + const candidates = getRouter().route({ kind: 'embedding', logicalName: 'default', requires: [] }); + assert.strictEqual(candidates.length, 2, 'routing untouched'); + assert.strictEqual(candidates[0], before); + }); + + it('removes true absence through a valid block', async () => { + await bootstrapModels({ models: block({ default: openaiEntry('sk-1'), retired: openaiEntry('sk-2') }) }); + + await applyModelsConfig(block({ default: openaiEntry('sk-1') })); + + assert.throws(() => resolveEmbedding('retired'), ModelBackendNotFoundError); + assert.ok(getBackend('embedding', 'default'), 'kept entry still serves'); + }); + + it('survives a poisoned block and still applies the next one', async () => { + await bootstrapModels({ models: block({ default: openaiEntry('sk-1') }) }); + const poisoned = {}; + Object.defineProperty(poisoned, 'embedding', { + enumerable: true, + get() { + throw new Error('poisoned block'); + }, + }); + + // Must resolve, not reject: the watcher calls fire-and-forget, so a rejection here would be + // an unhandled rejection in production. + await applyModelsConfig(poisoned); + + const rotated = block({ default: openaiEntry('sk-after-poison') }); + await applyModelsConfig(rotated); + assert.notEqual(getBackend('embedding', 'default'), undefined, 'later applies still work'); + }); + }); + + describe('startModelsConfigHotReload', () => { + const ENV_LAYERS = ['HARPER_SET_CONFIG', 'HARPER_CONFIG', 'HARPER_DEFAULT_CONFIG']; + let savedEnv; + + beforeEach(() => { + savedEnv = {}; + for (const name of ENV_LAYERS) { + savedEnv[name] = process.env[name]; + delete process.env[name]; + } + }); + + afterEach(() => { + for (const name of ENV_LAYERS) { + if (savedEnv[name] === undefined) delete process.env[name]; + else process.env[name] = savedEnv[name]; + } + }); + + it('stays off when an env layer names models through a dotted top-level key', () => { + process.env.HARPER_SET_CONFIG = JSON.stringify({ 'models.embedding.default': { backend: 'openai' } }); + + assert.equal(startModelsConfigHotReload(), false, 'dotted keys compose into models and pin it'); + }); + + it('applies a snapshot the watcher already holds, when ready pre-fired the subscription', async function () { + this.timeout(10000); + // The shared singleton's one-time 'ready' usually fires for the logger before models + // subscribes; a pre-warmed watcher's snapshot must be applied directly or a rewrite in + // that gap stays invisible until the next write. + const fixture = mkdtempSync(join(tmpdir(), 'harper.unit-test.models-reload-')); + const configFilePath = join(fixture, 'config.yaml'); + try { + writeFileSync(configFilePath, stringify({ models: block({ early: openaiEntry('sk-early') }) })); + // Pre-warm the instance models will subscribe to, consuming its one-time 'ready' first — + // exactly the shared-singleton situation where logging constructed the watcher earlier. + const prewarmed = new RootConfigWatcher(configFilePath); + prewarmed.ready.catch(() => {}); + await waitFor(() => prewarmed.config !== undefined, { message: 'watcher never became ready' }); + + await bootstrapModels({ models: block({}) }); + assert.equal(startModelsConfigHotReload({ watcher: prewarmed, debounceMs: 10 }), true); + + await waitFor(() => getBackend('embedding', 'early') !== undefined, { + message: 'the pre-subscription snapshot never applied', + }); + prewarmed.close(); + } finally { + stopModelsConfigHotReload(); + rmSync(fixture, { recursive: true, force: true }); + } + }); + + it('subscribes to the isolate-shared watcher and unsubscribes without closing it', () => { + // Logging already opens one root-config watcher per worker; models must ride the same + // instance instead of doubling the native-watcher footprint (review finding). + const shared = getSharedRootConfigWatcher(); + shared.ready.catch(() => {}); + const before = shared.listenerCount('change'); + + assert.equal(startModelsConfigHotReload(), true); + assert.equal(shared.listenerCount('change'), before + 1, 'models subscribed to the shared watcher'); + + stopModelsConfigHotReload(); + assert.equal(shared.listenerCount('change'), before, 'stop unsubscribes without closing'); + }); + + for (const layer of ENV_LAYERS) { + it(`stays off when ${layer} also defines models, so boot semantics keep ruling`, () => { + // The compatibility gate: an orchestrator still injecting models through an env layer + // keeps today's restart behavior; the file is not authoritative for the block. Each + // layer is asserted independently — a typo in one name would silently un-gate it. + process.env[layer] = JSON.stringify({ models: { embedding: {} } }); + + assert.equal(startModelsConfigHotReload(), false); + }); + } + + it('a boot discards a watcher snapshot whose settle timer has not fired yet', async function () { + this.timeout(10000); + // The exact race the debounce guard exists for: content observed BEFORE the boot must not + // apply after it. onSnapshotObserved makes the observation moment deterministic. + const fixture = mkdtempSync(join(tmpdir(), 'harper.unit-test.models-reload-')); + const configFilePath = join(fixture, 'config.yaml'); + try { + writeFileSync(configFilePath, stringify({ models: block({ keeper: openaiEntry('sk-a') }) })); + await bootstrapModels({ models: block({ keeper: openaiEntry('sk-a') }) }); + let observations = 0; + assert.equal( + startModelsConfigHotReload({ + configFilePath, + debounceMs: 300, + onSnapshotObserved: () => observations++, + }), + true + ); + await waitFor(() => observations >= 1, { message: 'watcher never became ready' }); + const observedBeforeBoot = observations; + + writeFileSync( + configFilePath, + stringify({ models: block({ keeper: openaiEntry('sk-a'), stale: openaiEntry('sk-stale') }) }) + ); + await waitFor(() => observations > observedBeforeBoot, { message: 'rewrite never observed' }); + + // Observed, timer armed, not yet fired: the boot must cancel it. + await bootstrapModels({ models: block({ keeper: openaiEntry('sk-boot') }) }); + + await new Promise((resolve) => setTimeout(resolve, 500)); + assert.equal(getBackend('embedding', 'stale'), undefined, 'pre-boot snapshot discarded'); + assert.ok(getBackend('embedding', 'keeper'), 'boot content stands'); + } finally { + stopModelsConfigHotReload(); + rmSync(fixture, { recursive: true, force: true }); + } + }); + + it('applies a config file rewrite to live requests, with no restart', async function () { + this.timeout(10000); + const fixture = mkdtempSync(join(tmpdir(), 'harper.unit-test.models-reload-')); + const configFilePath = join(fixture, 'config.yaml'); + const captured = installFetchCapture(); + try { + writeFileSync(configFilePath, stringify({ models: block({ default: openaiEntry('sk-boot') }) })); + await bootstrapModels({ models: block({ default: openaiEntry('sk-boot') }) }); + assert.equal(startModelsConfigHotReload({ configFilePath, debounceMs: 10 }), true); + + await resolveEmbedding('default').embed('hello', { model: 'text-embedding-3-small' }); + assert.equal(captured.sent.at(-1), 'Bearer sk-boot'); + + writeFileSync(configFilePath, stringify({ models: block({ default: openaiEntry('sk-rewritten') }) })); + await waitFor( + async () => { + // Observed through the request path itself: the rewrite has landed once an embed + // call presents the new credential. + await resolveEmbedding('default').embed('hello', { model: 'text-embedding-3-small' }); + return captured.sent.at(-1) === 'Bearer sk-rewritten'; + }, + { timeout: 8000, message: 'the rewritten credential never reached requests' } + ); + } finally { + captured.restore(); + rmSync(fixture, { recursive: true, force: true }); + } + }); + }); +}); diff --git a/unitTests/resources/rangeReadActivity.test.js b/unitTests/resources/rangeReadActivity.test.js new file mode 100644 index 0000000000..12b4590d4e --- /dev/null +++ b/unitTests/resources/rangeReadActivity.test.js @@ -0,0 +1,208 @@ +const assert = require('node:assert'); +const { setupTestDBPath } = require('../testUtils'); +const { waitFor } = require('../waitFor.js'); +const { table } = require('#src/resources/databases'); +const { setMainIsWorker } = require('#js/server/threads/manageThreads'); +const { DatabaseTransaction, setTxnExpiration } = require('#src/resources/DatabaseTransaction'); +const { transaction } = require('#src/resources/transaction'); + +const isLMDB = process.env.HARPER_STORAGE_ENGINE === 'lmdb'; + +describe('RocksDB range read activity', function () { + let Rows; + const opened = []; + before(async function () { + if (isLMDB) this.skip(); + setupTestDBPath(); + setMainIsWorker(true); + Rows = table({ + database: 'rangeActivity', + table: 'Rows', + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'bucket', indexed: true }, + ], + }); + if (Rows.indexingOperation) await Rows.indexingOperation; + for (let id = 0; id < 20; id++) await Rows.put({ id, bucket: 'before' }); + }); + afterEach(function () { + setTxnExpiration(30000); + for (const txn of opened.splice(0)) txn.abort(); + }); + + function openRead() { + const txn = new DatabaseTransaction(); + txn.db = Rows.primaryStore; + opened.push(txn); + const native = txn.useReadTxn(); + return { txn, native }; + } + + for (const kind of ['primary', 'index']) { + function range(native) { + return (kind === 'primary' ? Rows.primaryStore : Rows.indices.bucket).getRange({ transaction: native }); + } + + it(`${kind}: retains a committed snapshot while the scan crosses monitor ticks`, async function () { + setTxnExpiration(20); + const { txn, native } = openRead(); + const iterator = range(native)[Symbol.iterator](); + await txn.commit(); + let count = 0; + while (!iterator.next().done) { + count++; + await waitFor(() => !txn.rangeReadActive, { interval: 1 }); + assert.strictEqual(txn.transaction, native, 'progress must retain the original snapshot'); + } + assert.equal(count, 20); + txn.doneReadTxn(); + assert.equal(txn.transaction, null); + }); + + it(`${kind}: counts progress even when every row is filtered out`, async function () { + setTxnExpiration(20); + const { txn, native } = openRead(); + const filtered = range(native).filter(async () => { + await waitFor(() => !txn.rangeReadActive, { interval: 1 }); + assert.strictEqual(txn.transaction, native); + return false; + }); + await txn.commit(); + let count = 0; + for await (const _entry of filtered) count++; + assert.equal(count, 0); + txn.doneReadTxn(); + assert.equal(txn.transaction, null); + }); + + for (const started of [false, true]) { + it(`${kind}: reports expired ${started ? 'started' : 'unstarted'} scans before native access`, async function () { + setTxnExpiration(10); + const { txn, native } = openRead(); + const iterator = range(native)[Symbol.iterator](); + if (started) assert.notEqual(iterator.next().done, true); + await txn.commit(); + await waitFor(() => txn.transaction === null, { interval: 1 }); + const expired = (error) => error.statusCode === 503 && error.name === 'ReadSnapshotExpiredError'; + assert.throws(() => iterator.next(), expired); + assert.throws(() => range(native), expired); + assert.equal(iterator.return().done, true); + assert.equal(iterator.return().done, true); + assert.equal(iterator.next().done, true); + }); + } + + it(`${kind}: closing or throwing early does not retain an iterator reference`, async function () { + for (const throws of [false, true]) { + const { txn, native } = openRead(); + const iterator = range(native)[Symbol.iterator](); + iterator.next(); + await txn.commit(); + if (throws) { + const error = new Error('consumer failed'); + assert.throws( + () => iterator.throw(error), + (caught) => caught === error + ); + } else iterator.return(); + txn.doneReadTxn(); + assert.equal(txn.transaction, null); + assert.equal(iterator.return().done, true); + } + }); + } + + it('key-only primary range iterators remain iterable themselves', function () { + const { native } = openRead(); + const iterator = Rows.primaryStore.getRange({ transaction: native, values: false })[Symbol.iterator](); + assert.strictEqual(iterator[Symbol.iterator](), iterator); + assert.equal([...iterator].length, 20); + }); + + it('Table.search releases the retained snapshot on early consumer return', async function () { + const context = {}; + await transaction(context, async (txn) => { + const results = Rows.search({}, context); + await txn.commit(); + assert.ok(txn.transaction); + for await (const _row of results) break; + assert.equal(txn.transaction, null); + }); + }); + + it('keeps primary and index reads on the original snapshot after another request commits', async function () { + const { native } = openRead(); + const primary = Rows.primaryStore.getRange({ transaction: native }); + const index = Rows.indices.bucket.getRange({ + transaction: native, + start: 'before', + end: 'before', + inclusiveEnd: true, + }); + await Rows.put({ id: 19, bucket: 'after' }); + try { + assert.equal([...primary].find(({ key }) => key === 19).value.bucket, 'before'); + assert.ok([...index].some(({ value }) => value === 19)); + } finally { + await Rows.put({ id: 19, bucket: 'before' }); + } + }); + + it('preserves read-your-writes through primary and index searches', async function () { + const context = {}; + await transaction(context, async () => { + await Rows.put({ id: 100, bucket: 'own' }, context); + for (const conditions of [[], [{ attribute: 'bucket', value: 'own' }]]) { + const ids = []; + for await (const row of Rows.search({ conditions }, context)) ids.push(row.id); + assert.ok(ids.includes(100)); + } + }); + await Rows.delete(100); + }); + + it('commit retry handlers can scan a live handle after read ownership is handed off', async function () { + const txn = new DatabaseTransaction(); + txn.db = Rows.primaryStore; + opened.push(txn); + let attempts = 0; + txn.addWrite({ + key: 19, + store: Rows.primaryStore, + commit(_version, _entry, _retry, native) { + attempts++; + const iterator = Rows.primaryStore.getRange({ transaction: native })[Symbol.iterator](); + assert.notEqual(iterator.next().done, true); + iterator.return(); + Rows.primaryStore.putSync(19, { id: 19, bucket: 'before' }, { transaction: native }); + }, + }); + await Rows.put({ id: 19, bucket: 'before' }); + await txn.commit(); + assert.ok(attempts > 1, 'the concurrent write must force a real native conflict retry'); + assert.equal((await Rows.get(19)).bucket, 'before'); + }); + + it('range activity cannot extend an idle write holder and its write is rolled back', async function () { + setTxnExpiration(20); + const context = {}; + await assert.rejects( + transaction(context, async (txn) => { + await Rows.put({ id: 101, bucket: 'uncommitted' }, context); + const native = Rows._readTxnForContext(context); + const iterator = Rows.primaryStore.getRange({ transaction: native })[Symbol.iterator](); + while (!txn.timedOut) { + assert.notEqual(iterator.next().done, true); + await waitFor(() => !txn.rangeReadActive || txn.timedOut, { interval: 1 }); + } + assert.throws( + () => iterator.next(), + (error) => error.statusCode === 422 + ); + }), + (error) => error.statusCode === 422 + ); + assert.equal(await Rows.get(101), null); + }); +}); diff --git a/unitTests/resources/searchPlannerRebuildingIndex.test.js b/unitTests/resources/searchPlannerRebuildingIndex.test.js new file mode 100644 index 0000000000..b85d9cf0d1 --- /dev/null +++ b/unitTests/resources/searchPlannerRebuildingIndex.test.js @@ -0,0 +1,167 @@ +/** + * harper#2537. `searchByIndex` refuses a rebuilding index with IndexRebuildingError, but the planner + * used to rank its condition by the partially-built index's cardinality, so the narrowest-first + * ordering could hand the lead to a condition the executor was about to refuse — a 503 for a query a + * sibling index could have answered completely. The planner has to see a rebuilding index as absent. + */ + +require('../testUtils'); +const assert = require('node:assert'); +const { setupTestDBPath } = require('../testUtils'); +const { table } = require('#src/resources/databases'); +const { estimateCondition } = require('#src/resources/search'); +const { setMainIsWorker } = require('#js/server/threads/manageThreads'); + +describe('the query planner treats a rebuilding index as unavailable (harper#2537)', function () { + this.timeout(60000); + + let Table; + let Parent; + + before(async () => { + setupTestDBPath(); + setMainIsWorker(true); + Table = table({ + table: 'PlannerRebuildingIndex', + database: 'test', + schemaDefined: true, + attributes: [ + { name: 'id', type: 'ID', isPrimaryKey: true }, + { name: 'rare', type: 'String', indexed: true }, + { name: 'common', type: 'String', indexed: true }, + ], + }); + let lastPut; + for (let i = 0; i < 20; i++) + lastPut = Table.put({ id: `k-${i}`, rare: i === 7 ? 'needle' : `r-${i}`, common: i < 10 ? 'left' : 'right' }); + await lastPut; + if (Table.indexingOperation) await Table.indexingOperation; + + Parent = table({ + table: 'PlannerRebuildingParent', + database: 'test', + schemaDefined: true, + schemaRelationshipsDefined: true, + attributes: [ + { name: 'id', type: 'ID', isPrimaryKey: true }, + { name: 'status', type: 'String', indexed: true }, + { name: 'childId', type: 'ID', indexed: true }, + { + name: 'child', + type: 'PlannerRebuildingIndex', + relationship: { from: 'childId' }, + relationshipReference: { database: 'test', table: 'PlannerRebuildingIndex' }, + definition: { tableClass: Table }, + }, + ], + }); + for (let i = 0; i < 20; i++) + lastPut = Parent.put({ id: `p-${i}`, status: i < 10 ? 'open' : 'closed', childId: `k-${i}` }); + await lastPut; + if (Parent.indexingOperation) await Parent.indexingOperation; + }); + + /** Stand in for a build in flight without holding one open for the length of the suite. */ + async function whileRebuilding(attribute, body) { + Table.indices[attribute].isIndexing = true; + try { + return await body(); + } finally { + Table.indices[attribute].isIndexing = false; + } + } + + it('estimates a rebuilding index at Infinity for every comparator that would otherwise use it', async () => { + const estimates = await whileRebuilding('rare', () => { + const estimate = estimateCondition(Table); + return { + equals: estimate({ attribute: 'rare', value: 'needle' }), + range: estimate({ attribute: 'rare', comparator: 'starts_with', value: 'r-' }), + between: estimate({ attribute: 'rare', comparator: 'between', value: ['r-0', 'r-9'] }), + in: estimate({ attribute: 'rare', comparator: 'in', value: ['needle', 'r-1'] }), + sort: estimate({ attribute: 'rare', comparator: 'sort' }), + }; + }); + for (const [comparator, estimate] of Object.entries(estimates)) + assert.strictEqual( + estimate, + Infinity, + `a "${comparator}" condition on a rebuilding index must not rank as usable (got ${estimate})` + ); + }); + + it('still estimates a complete index by its cardinality', () => { + const estimate = estimateCondition(Table); + assert.ok( + estimate({ attribute: 'rare', value: 'needle' }) < Infinity, + 'a complete index must still produce a finite estimate' + ); + }); + + it('leads with the available index and answers the query completely', async () => { + const rows = await whileRebuilding('rare', async () => { + const found = []; + for await (const record of Table.search({ + allowFullScan: false, + conditions: [ + { attribute: 'rare', value: 'needle' }, + { attribute: 'common', value: 'left' }, + ], + })) + found.push(record); + return found; + }); + assert.deepStrictEqual( + rows.map((row) => row.id), + ['k-7'], + 'the sibling index must lead so the rebuilding attribute is applied as a record filter' + ); + }); + + it('still refuses a query whose only condition is on the rebuilding index', async () => { + await whileRebuilding('rare', () => { + assert.throws( + () => Table.search({ allowFullScan: false, conditions: [{ attribute: 'rare', value: 'needle' }] }), + /not indexed yet/, + 'a search that can only be driven by the rebuilding index must refuse rather than return partial results' + ); + }); + }); + + it('follows a relationship path for every comparator, not only equality', async () => { + const estimates = await whileRebuilding('rare', () => { + const estimate = estimateCondition(Parent); + return { + equals: estimate({ attribute: ['child', 'rare'], value: 'needle' }), + between: estimate({ attribute: ['child', 'rare'], comparator: 'between', value: ['r-0', 'r-9'] }), + starts_with: estimate({ attribute: ['child', 'rare'], comparator: 'starts_with', value: 'r-' }), + }; + }); + for (const [comparator, estimate] of Object.entries(estimates)) + assert.strictEqual( + estimate, + Infinity, + `a relationship "${comparator}" whose leaf index is rebuilding must not rank as usable (got ${estimate})` + ); + }); + + it('treats a rebuilding local join index as unusable too', () => { + Parent.indices.childId.isIndexing = true; + try { + assert.strictEqual( + estimateCondition(Parent)({ attribute: ['child', 'rare'], comparator: 'between', value: ['r-0', 'r-9'] }), + Infinity, + 'the join is driven by the local from-index, so a rebuild of it must rank the condition as unusable' + ); + } finally { + Parent.indices.childId.isIndexing = false; + } + }); + + it('still estimates a relationship finitely when every index it needs is complete', () => { + assert.ok( + estimateCondition(Parent)({ attribute: ['child', 'rare'], value: 'needle' }) < Infinity, + 'a complete relationship path must still produce a finite estimate' + ); + }); +}); diff --git a/unitTests/resources/sortAlignedCondition.test.js b/unitTests/resources/sortAlignedCondition.test.js new file mode 100644 index 0000000000..dbb0b1028c --- /dev/null +++ b/unitTests/resources/sortAlignedCondition.test.js @@ -0,0 +1,85 @@ +/** + * A sort on an attribute that also carries a caller's condition used to drop that condition whenever + * the planner did not give it the lead: `orderConditions` reorders narrowest-first, and the branch that + * un-does the sort alignment spliced whatever `orderAlignedCondition` pointed at — the caller's + * condition, not just the pseudo-condition the planner had added. The query then returned rows the + * condition excludes. Reached far more often now that a rebuilding index's condition is ranked last + * (harper#2537), which is how it surfaced. + */ + +require('../testUtils'); +const assert = require('node:assert'); +const { setupTestDBPath } = require('../testUtils'); +const { table } = require('#src/resources/databases'); +const { setMainIsWorker } = require('#js/server/threads/manageThreads'); + +describe('a sort does not drop a condition on the sorted attribute', function () { + this.timeout(60000); + + let Table; + + before(async () => { + setupTestDBPath(); + setMainIsWorker(true); + Table = table({ + table: 'SortAlignedCondition', + database: 'test', + schemaDefined: true, + attributes: [ + { name: 'id', type: 'ID', isPrimaryKey: true }, + { name: 'rare', type: 'String', indexed: true }, + { name: 'common', type: 'String', indexed: true }, + ], + }); + let lastPut; + for (let i = 0; i < 20; i++) + lastPut = Table.put({ id: `k-${i}`, rare: i === 7 ? 'needle' : `r-${i}`, common: i < 10 ? 'left' : 'right' }); + await lastPut; + if (Table.indexingOperation) await Table.indexingOperation; + }); + + async function ids(request) { + const found = []; + for await (const record of Table.search(request)) found.push(record.id); + return found; + } + + it('applies a condition on the sort attribute even when another condition leads', async () => { + // `rare` matches one row and leads; `common` carries both the sort and a condition that excludes it + assert.deepStrictEqual( + await ids({ + conditions: [ + { attribute: 'rare', value: 'needle' }, + { attribute: 'common', value: 'right' }, + ], + sort: { attribute: 'common' }, + }), + [], + 'the only row matching "rare" has common=left, so a common=right condition must exclude it' + ); + assert.deepStrictEqual( + await ids({ + conditions: [ + { attribute: 'rare', value: 'needle' }, + { attribute: 'common', value: 'left' }, + ], + sort: { attribute: 'common' }, + }), + ['k-7'], + 'a matching condition on the sort attribute must still return the row' + ); + }); + + it('still orders by the sort attribute when its condition does not lead', async () => { + const ordered = await ids({ + conditions: [ + { attribute: 'common', value: 'right' }, + { attribute: 'rare', comparator: 'greater_than', value: 'r-1' }, + ], + sort: { attribute: 'rare', descending: true }, + }); + assert.ok(ordered.length > 1, 'the fixture must return several rows for the ordering to be observable'); + const sorted = [...ordered].sort().reverse(); + assert.deepStrictEqual(ordered, sorted, 'the results must still be ordered by the sort attribute, descending'); + }); +}); diff --git a/unitTests/resources/vectorIndex.test.js b/unitTests/resources/vectorIndex.test.js index cc550a56d1..7dd100bde3 100644 --- a/unitTests/resources/vectorIndex.test.js +++ b/unitTests/resources/vectorIndex.test.js @@ -935,6 +935,7 @@ describe('HNSW construction ef auto-scale (#2180)', () => { describe('HNSW greedy routing above layer 0 (ROUTING_EF)', () => { if (process.env.HARPER_STORAGE_ENGINE === 'lmdb') return; const N = 600; + const ROUTING_EF = 1; // must track ROUTING_EF in resources/indexes/HierarchicalNavigableSmallWorld.ts // Each graph's level assignment is pinned with a seeded PRNG (mulberry32) so a run is // reproducible: greedy-vs-full equality is only statistically true over random graphs — ~2-3% // of random 600-node graphs legitimately route to a different entry point and change the @@ -965,7 +966,9 @@ describe('HNSW greedy routing above layer 0 (ROUTING_EF)', () => { ], }); let seedState = seed; + let draws = 0; T.indices.vector.customIndex.random = () => { + draws++; seedState = (seedState + 0x6d2b79f5) | 0; let t = Math.imul(seedState ^ (seedState >>> 15), 1 | seedState); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; @@ -976,6 +979,10 @@ describe('HNSW greedy routing above layer 0 (ROUTING_EF)', () => { const b = ((i * 7) % N) / N; await T.put(i, { vector: [Math.cos(a), Math.sin(a), b, (i % 11) / 11] }); } + // A seed only names a graph while each node takes exactly one draw: a second consumer of the + // stream shifts every level after it and silently re-pins all eight graphs to something the + // measurements below were never taken on. + assert.strictEqual(draws, N, 'the pinned stream must serve one level draw per node and nothing else'); return T; } @@ -993,15 +1000,33 @@ describe('HNSW greedy routing above layer 0 (ROUTING_EF)', () => { // accuracy. Compare against the same graph searched with the full ef at every layer — the // pre-change behaviour — rather than against a fixed expectation. it('returns the same neighbours as searching every layer at the full ef', async () => { + const descentSensitive = []; for (const seed of SEEDS) { const label = '0x' + (seed >>> 0).toString(16); const T = await buildGraph(seed); try { const customIndex = T.indices.vector.customIndex; + const originalSearchLayer = customIndex.searchLayer; const greedy = []; - for (const target of targets) { - greedy.push(await topTenIds(T, target)); + const routingEfs = new Set(); + customIndex.searchLayer = function (v, epId, ep, ef, level, ...rest) { + if (level > 0) routingEfs.add(ef); + return originalSearchLayer.call(this, v, epId, ep, ef, level, ...rest); + }; + try { + for (const target of targets) { + greedy.push(await topTenIds(T, target)); + } + } finally { + customIndex.searchLayer = originalSearchLayer; } + // Hand the upper layers the full ef and both sides of the greedy-vs-full comparison + // below search identically, so only this assertion notices the optimization going away. + assert.deepStrictEqual( + [...routingEfs], + [ROUTING_EF], + `the layers above 0 must route at ROUTING_EF (seed ${label})` + ); // Every layer at the ef layer 0 actually resolves to — what search() passed down before // greedy descent. Read it from a real query rather than efConstructionSearch, which is @@ -1009,7 +1034,6 @@ describe('HNSW greedy routing above layer 0 (ROUTING_EF)', () => { // auto-scaled path. const resolvedLayer0Ef = await captureLayer0Ef(T, { limit: 10 }); assert(resolvedLayer0Ef > 1, `expected an auto-scaled layer-0 ef, got ${resolvedLayer0Ef} (seed ${label})`); - const originalSearchLayer = customIndex.searchLayer; customIndex.searchLayer = function (v, epId, ep, ef, level, ...rest) { return originalSearchLayer.call(this, v, epId, ep, level > 0 ? resolvedLayer0Ef : ef, level, ...rest); }; @@ -1024,10 +1048,32 @@ describe('HNSW greedy routing above layer 0 (ROUTING_EF)', () => { } finally { customIndex.searchLayer = originalSearchLayer; } + + // On many graphs layer 0 alone reaches the true neighbours from wherever it starts, and + // there the comparison above passes with the descent deleted outright. That is a + // property of the graph, not of this seed list, so the sweep as a whole has to contain + // at least one graph the descent decides — otherwise nothing here tests the descent. + customIndex.searchLayer = function (v, epId, ep, ef, level, ...rest) { + return level > 0 ? [] : originalSearchLayer.call(this, v, epId, ep, ef, level, ...rest); + }; + try { + for (let i = 0; i < targets.length; i++) { + if ((await topTenIds(T, targets[i])) !== greedy[i]) { + descentSensitive.push(label); + break; + } + } + } finally { + customIndex.searchLayer = originalSearchLayer; + } } finally { await T.dropTable(); } } + assert( + descentSensitive.length > 0, + 'no seed routes differently without the descent: check whether the graph still has a hierarchy worth descending (mL, MAX_LEVEL) before re-picking SEEDS against this control' + ); }); }); diff --git a/unitTests/sqlEngine/aggregate.test.js b/unitTests/sqlEngine/aggregate.test.js index 546140ac95..2eee8aef30 100644 --- a/unitTests/sqlEngine/aggregate.test.js +++ b/unitTests/sqlEngine/aggregate.test.js @@ -3,7 +3,7 @@ /** * End-to-end pipeline tests for the new SQL engine, phase 2: aggregates. * - * Sets globalThis.harperConfig.sql.allowFullScan = true so the mock table + * Sets sql.allowFullScan = true (via configUtils) so the mock table * does not require an indexed WHERE condition (aggregate queries legitimately * scan all rows). */ @@ -13,6 +13,8 @@ const alasql = require('alasql'); const router = require('#src/sqlEngine/router'); const binder = require('#src/sqlEngine/binder/bind'); +const configUtils = require('#src/config/configUtils'); +const { CONFIG_PARAMS } = require('#src/utility/hdbTerms'); function makeMockTable({ primaryKey = 'id', attributes = [], rows = [] } = {}) { const table = { @@ -81,7 +83,7 @@ const ORDERS = [ describe('sqlEngine phase 2: aggregates', () => { let originalEngine; - let savedHarperConfig; + let originalAllowFullScan; let mockTable; beforeEach(() => { @@ -89,8 +91,8 @@ describe('sqlEngine phase 2: aggregates', () => { process.env.HARPER_SQL_ENGINE = 'new'; // Allow full scans — aggregate queries legitimately read all rows. - savedHarperConfig = globalThis.harperConfig; - globalThis.harperConfig = { sql: { allowFullScan: true } }; + originalAllowFullScan = configUtils.getConfigValue(CONFIG_PARAMS.SQL_ALLOWFULLSCAN); + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, true); mockTable = makeMockTable({ primaryKey: 'id', @@ -108,7 +110,7 @@ describe('sqlEngine phase 2: aggregates', () => { afterEach(() => { if (originalEngine === undefined) delete process.env.HARPER_SQL_ENGINE; else process.env.HARPER_SQL_ENGINE = originalEngine; - globalThis.harperConfig = savedHarperConfig; + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, originalAllowFullScan); binder._setDatabasesLoader(null); }); diff --git a/unitTests/sqlEngine/join.test.js b/unitTests/sqlEngine/join.test.js index 74ef175e3b..fae06c6611 100644 --- a/unitTests/sqlEngine/join.test.js +++ b/unitTests/sqlEngine/join.test.js @@ -16,6 +16,8 @@ const alasql = require('alasql'); const router = require('#src/sqlEngine/router'); const binder = require('#src/sqlEngine/binder/bind'); const { EngineUnsupportedError } = require('#src/sqlEngine/errors'); +const configUtils = require('#src/config/configUtils'); +const { CONFIG_PARAMS } = require('#src/utility/hdbTerms'); function makeMockTable({ primaryKey = 'id', attributes = [], rows = [] } = {}) { const table = { @@ -93,6 +95,7 @@ function sortByJson(rows) { describe('sqlEngine phase 3: joins', () => { let originalEngine; + let originalAllowFullScan; let users; let orders; let products; @@ -100,7 +103,8 @@ describe('sqlEngine phase 3: joins', () => { beforeEach(() => { originalEngine = process.env.HARPER_SQL_ENGINE; process.env.HARPER_SQL_ENGINE = 'new'; - globalThis.harperConfig = { sql: { allowFullScan: true } }; + originalAllowFullScan = configUtils.getConfigValue(CONFIG_PARAMS.SQL_ALLOWFULLSCAN); + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, true); users = makeMockTable({ primaryKey: 'id', @@ -148,7 +152,7 @@ describe('sqlEngine phase 3: joins', () => { afterEach(() => { if (originalEngine === undefined) delete process.env.HARPER_SQL_ENGINE; else process.env.HARPER_SQL_ENGINE = originalEngine; - delete globalThis.harperConfig; + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, originalAllowFullScan); binder._setDatabasesLoader(null); }); @@ -319,7 +323,6 @@ describe('sqlEngine phase 3: joins', () => { }); it('hash join does not match NaN keys (NaN never equals itself)', async () => { - globalThis.harperConfig = { sql: { allowFullScan: true } }; const left = makeMockTable({ primaryKey: 'id', attributes: [ @@ -342,7 +345,6 @@ describe('sqlEngine phase 3: joins', () => { }); it('hash join emits matched rows when both sides share a non-indexed equi key', async () => { - globalThis.harperConfig = { sql: { allowFullScan: true } }; const left = makeMockTable({ primaryKey: 'id', attributes: [ @@ -382,7 +384,6 @@ describe('sqlEngine phase 3: joins', () => { }); it('hash join LEFT null-fills a non-indexed-key row with no match', async () => { - globalThis.harperConfig = { sql: { allowFullScan: true } }; const left = makeMockTable({ primaryKey: 'id', attributes: [ @@ -415,7 +416,7 @@ describe('sqlEngine phase 3: joins', () => { }); it('indexNL join passes with allowFullScan off when the outer has an indexed filter', async () => { - globalThis.harperConfig = { sql: { allowFullScan: false } }; + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, false); const data = await runSql( 'SELECT u.name, o.amount FROM dev.user u JOIN dev.orders o ON u.id = o.user_id WHERE u.id = 1' ); @@ -429,7 +430,7 @@ describe('sqlEngine phase 3: joins', () => { }); it('rejects a join whose outer side requires a full scan when allowFullScan is off', async () => { - globalThis.harperConfig = { sql: { allowFullScan: false } }; + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, false); await assert.rejects( runSql('SELECT u.name, o.amount FROM dev.user u JOIN dev.orders o ON u.id = o.user_id'), EngineUnsupportedError diff --git a/unitTests/sqlEngine/mutation.test.js b/unitTests/sqlEngine/mutation.test.js index 6cf5cb6f80..3bfed029c0 100644 --- a/unitTests/sqlEngine/mutation.test.js +++ b/unitTests/sqlEngine/mutation.test.js @@ -17,6 +17,8 @@ const alasql = require('alasql'); const router = require('#src/sqlEngine/router'); const binder = require('#src/sqlEngine/binder/bind'); const mutation = require('#src/sqlEngine/executor/runMutation'); +const configUtils = require('#src/config/configUtils'); +const { CONFIG_PARAMS } = require('#src/utility/hdbTerms'); function evalConditions(row, conditions, operator) { const fn = operator === 'or' ? 'some' : 'every'; @@ -122,13 +124,15 @@ function runSql(sql) { describe('sqlEngine phase 4: mutations', () => { let originalEngine; + let originalAllowFullScan; let widgets; let txnCalls; beforeEach(() => { originalEngine = process.env.HARPER_SQL_ENGINE; process.env.HARPER_SQL_ENGINE = 'new'; - globalThis.harperConfig = { sql: { allowFullScan: true } }; + originalAllowFullScan = configUtils.getConfigValue(CONFIG_PARAMS.SQL_ALLOWFULLSCAN); + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, true); widgets = makeWritableTable({ primaryKey: 'id', @@ -159,7 +163,7 @@ describe('sqlEngine phase 4: mutations', () => { afterEach(() => { if (originalEngine === undefined) delete process.env.HARPER_SQL_ENGINE; else process.env.HARPER_SQL_ENGINE = originalEngine; - delete globalThis.harperConfig; + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, originalAllowFullScan); binder._setDatabasesLoader(null); mutation._setTransactionRunner(null); }); diff --git a/unitTests/sqlEngine/router.test.js b/unitTests/sqlEngine/router.test.js index 872a038dc4..53fe02dba2 100644 --- a/unitTests/sqlEngine/router.test.js +++ b/unitTests/sqlEngine/router.test.js @@ -15,19 +15,25 @@ const sinon = require('sinon'); const router = require('#src/sqlEngine/router'); const config = require('#src/sqlEngine/config'); +const configUtils = require('#src/config/configUtils'); +const { CONFIG_PARAMS } = require('#src/utility/hdbTerms'); const { EngineUnsupportedError } = require('#src/sqlEngine/errors'); describe('sqlEngine router', () => { let originalEngine; + let originalConfigEngine; beforeEach(() => { originalEngine = process.env.HARPER_SQL_ENGINE; + originalConfigEngine = configUtils.getConfigValue(CONFIG_PARAMS.SQL_ENGINE); delete process.env.HARPER_SQL_ENGINE; + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ENGINE, undefined); }); afterEach(() => { if (originalEngine === undefined) delete process.env.HARPER_SQL_ENGINE; else process.env.HARPER_SQL_ENGINE = originalEngine; + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ENGINE, originalConfigEngine); }); it('defaults to auto mode (phase-5 cutover)', () => { @@ -124,3 +130,86 @@ describe('sqlEngine router', () => { ); }); }); + +describe('sqlEngine config: harper config integration', () => { + const SQL_KEYS = [ + CONFIG_PARAMS.SQL_ENGINE, + CONFIG_PARAMS.SQL_ALLOWFULLSCAN, + CONFIG_PARAMS.SQL_MAXSORTROWS, + CONFIG_PARAMS.SQL_MAXHASHROWS, + ]; + let originalValues; + let originalEnvEngine; + + beforeEach(() => { + originalValues = SQL_KEYS.map((key) => configUtils.getConfigValue(key)); + originalEnvEngine = process.env.HARPER_SQL_ENGINE; + delete process.env.HARPER_SQL_ENGINE; + // Clear to a known-unset state so a default-value assertion below can't go red on a + // machine whose own harper-config.yaml happens to set one of these already. + SQL_KEYS.forEach((key) => configUtils.updateConfigObject(key, undefined)); + }); + + afterEach(() => { + if (originalEnvEngine === undefined) delete process.env.HARPER_SQL_ENGINE; + else process.env.HARPER_SQL_ENGINE = originalEnvEngine; + SQL_KEYS.forEach((key, i) => configUtils.updateConfigObject(key, originalValues[i])); + }); + + it('reads sql.engine from Harper config when no env var is set', () => { + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ENGINE, 'legacy'); + assert.strictEqual(config.getSqlEngineConfig().engine, 'legacy'); + }); + + it('HARPER_SQL_ENGINE env var still takes precedence over sql.engine config', () => { + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ENGINE, 'legacy'); + process.env.HARPER_SQL_ENGINE = 'new'; + assert.strictEqual(config.getSqlEngineConfig().engine, 'new'); + }); + + it('reads sql.allowFullScan from Harper config (default is false)', () => { + assert.strictEqual(config.getSqlEngineConfig().allowFullScan, false); + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, true); + assert.strictEqual(config.getSqlEngineConfig().allowFullScan, true); + }); + + it('reads sql.maxSortRows from Harper config (default is 1_000_000)', () => { + assert.strictEqual(config.getSqlEngineConfig().maxSortRows, 1_000_000); + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_MAXSORTROWS, 42); + assert.strictEqual(config.getSqlEngineConfig().maxSortRows, 42); + }); + + it('reads sql.maxHashRows from Harper config (default is 1_000_000)', () => { + assert.strictEqual(config.getSqlEngineConfig().maxHashRows, 1_000_000); + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_MAXHASHROWS, 7); + assert.strictEqual(config.getSqlEngineConfig().maxHashRows, 7); + }); + + it('ignores an unknown sql.engine value and falls back to the default', () => { + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ENGINE, 'gibberish'); + assert.strictEqual(config.getSqlEngineConfig().engine, 'auto'); + }); + + it('ignores a wrong-typed sql.allowFullScan value and falls back to the default', () => { + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_ALLOWFULLSCAN, 'true'); + assert.strictEqual(config.getSqlEngineConfig().allowFullScan, false); + }); + + it('ignores wrong-typed sql.maxSortRows/maxHashRows values and falls back to the defaults', () => { + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_MAXSORTROWS, 'unlimited'); + configUtils.updateConfigObject(CONFIG_PARAMS.SQL_MAXHASHROWS, 'unlimited'); + const cfg = config.getSqlEngineConfig(); + assert.strictEqual(cfg.maxSortRows, 1_000_000); + assert.strictEqual(cfg.maxHashRows, 1_000_000); + }); + + it('flattenConfig() derives the four flat sql.* keys from a nested sql block', () => { + const flat = configUtils.flattenConfig({ + sql: { engine: 'legacy', allowFullScan: true, maxSortRows: 5, maxHashRows: 7 }, + }); + assert.strictEqual(flat[CONFIG_PARAMS.SQL_ENGINE.toLowerCase()], 'legacy'); + assert.strictEqual(flat[CONFIG_PARAMS.SQL_ALLOWFULLSCAN.toLowerCase()], true); + assert.strictEqual(flat[CONFIG_PARAMS.SQL_MAXSORTROWS.toLowerCase()], 5); + assert.strictEqual(flat[CONFIG_PARAMS.SQL_MAXHASHROWS.toLowerCase()], 7); + }); +}); diff --git a/unitTests/validation/configValidator.test.js b/unitTests/validation/configValidator.test.js index 2ff8b8d470..179601ad38 100644 --- a/unitTests/validation/configValidator.test.js +++ b/unitTests/validation/configValidator.test.js @@ -271,6 +271,89 @@ describe('Test configValidator module', () => { expect(configValidator(config).error.message).to.include("'replication.blobGapEscalationMs' must be an integer"); }); + it('accepts a well-formed sql config section', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = { engine: 'new', allowFullScan: true, maxSortRows: 500, maxHashRows: 500 }; + expect(configValidator(config).error).to.be.undefined; + }); + + it('rejects an unknown sql.engine value rather than silently keeping the default', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = { engine: 'gibberish' }; + expect(configValidator(config).error.message).to.include("'sql.engine' must be one of [legacy, new, auto]"); + }); + + it('rejects a quoted-string sql.allowFullScan value (convert is off for this section)', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = { allowFullScan: 'true' }; + expect(configValidator(config).error.message).to.include("'sql.allowFullScan' must be a boolean"); + }); + + it('rejects non-positive/non-integer sql.maxSortRows and sql.maxHashRows', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = { maxSortRows: 0, maxHashRows: 2.5 }; + expect(configValidator(config).error.message).to.include("'sql.maxSortRows' must be greater than or equal to 1"); + expect(configValidator(config).error.message).to.include("'sql.maxHashRows' must be an integer"); + }); + + it('rejects a quoted-number sql.maxSortRows value (convert is off for this section)', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = { maxSortRows: '500' }; + expect(configValidator(config).error.message).to.include("'sql.maxSortRows' must be a number"); + }); + + it('rejects an unknown key inside sql (typos fail loudly instead of being silently ignored)', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = { allowFullScan: true, allwoFullScan: true }; + expect(configValidator(config).error.message).to.include("'sql.allwoFullScan' is not allowed"); + }); + + it('accepts an empty sql section', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = {}; + expect(configValidator(config).error).to.be.undefined; + }); + + it('accepts a disabled entry, the loader spelling for an application turned off', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = false; + expect(configValidator(config).error).to.be.undefined; + config.sql = null; + expect(configValidator(config).error).to.be.undefined; + }); + + it('accepts an application deployed under the sql name before it was reserved', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = { package: '@org/sql-app', urlPath: '/sql', install: { timeout: 1000 } }; + expect(configValidator(config).error).to.be.undefined; + }); + + it('rejects sql engine settings on an entry that is a deployed application', () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = { package: '@org/sql-app', engine: 'new' }; + expect(configValidator(config).error.message).to.include( + "'sql.engine' cannot be set while 'sql' names a deployed application" + ); + }); + + it("rejects a typo'd sql setting rather than reading it as an application entry", () => { + const config = testUtils.deepClone(FAKE_CONFIG); + config.sql = { allwoFullScan: true }; + expect(configValidator(config).error.message).to.include("'sql.allwoFullScan' is not allowed"); + }); + + // The predicate configUtils.validateConfig() uses to decide whether to warn that 'sql' holds + // an application; the schema branch above is driven by the same key list. + it('identifies an application-shaped sql entry positively, by the keys a deploy writes', () => { + const { isLegacySqlApplicationEntry } = config_val; + expect(isLegacySqlApplicationEntry({ package: '@org/sql-app' })).to.be.true; + expect(isLegacySqlApplicationEntry({ host: 'sql.example.com' })).to.be.true; + expect(isLegacySqlApplicationEntry({ engine: 'new' })).to.be.false; + expect(isLegacySqlApplicationEntry({ allwoFullScan: true })).to.be.false; + expect(isLegacySqlApplicationEntry({})).to.be.false; + expect(isLegacySqlApplicationEntry(undefined)).to.be.false; + }); + it('rejects a URL / port / numeric node.hostname, and accepts a bare host (#2218)', () => { const config = testUtils.deepClone(FAKE_CONFIG); for (const [bad, reason] of [ diff --git a/utility/componentNames.ts b/utility/componentNames.ts index 1c44b5edc8..f19f14eda5 100644 --- a/utility/componentNames.ts +++ b/utility/componentNames.ts @@ -105,3 +105,15 @@ export function deriveGitSecretName(component: string, host: string): string { const componentKey = String(component).replace(/[^\w.-]+/g, '_'); return `deploy.${componentKey}.git.${hostKey}`; } + +/** + * The root config namespace holds core settings sections and application entries side by side, so + * an application deployed under one of these names overwrites the section and fails the next boot's + * config validation. Only `sql` is listed: it is the one section whose schema is closed to unknown + * keys, so it is the one a stray application entry breaks. + */ +export const RESERVED_COMPONENT_NAMES = new Set(['sql']); + +export function isReservedComponentName(project: string): boolean { + return typeof project === 'string' && RESERVED_COMPONENT_NAMES.has(project); +} diff --git a/utility/errors/commonErrors.ts b/utility/errors/commonErrors.ts index 98c7a209d4..69096d2065 100644 --- a/utility/errors/commonErrors.ts +++ b/utility/errors/commonErrors.ts @@ -236,6 +236,8 @@ const CUSTOM_FUNCTIONS_ERROR_MSGS = { NO_FILE: 'File does not exist', BAD_FILE_NAME: 'File name can only contain alphanumeric, dash and underscore characters', BAD_PROJECT_NAME: 'Project name can only contain alphanumeric, dash and underscores characters', + RESERVED_PROJECT_NAME: (project: string) => + `Component name '${project}' is reserved for Harper's '${project}' configuration section; deploy under a different name`, BAD_PACKAGE: 'Packaged project must be base64-encoded tar file of project directory', DROP_FUNCTION: 'Error dropping custom function, check the log for more details', ADD_FUNCTION: 'Error adding custom function project, check the log for more details', diff --git a/utility/errors/hdbError.ts b/utility/errors/hdbError.ts index efd15140b5..8eacc4e75a 100644 --- a/utility/errors/hdbError.ts +++ b/utility/errors/hdbError.ts @@ -66,6 +66,22 @@ export class ServerError extends Error { } } +/** + * Thrown when a write targets a table whose derived index has fallen further behind than its + * registration allows. A distinct, retryable 503 so writers back off before the index's cursor is + * lost to transaction-log retention, rather than reading a generic 503 as a permanent failure. + */ +export class DerivedIndexLagError extends ServerError { + code: string; + retryable: boolean; + constructor(message: string) { + super(message, 503); + this.name = 'DerivedIndexLagError'; + this.code = 'DERIVED_INDEX_LAGGING'; + this.retryable = true; + } +} + /** * Thrown when a query targets an attribute whose secondary index is still being (re)built. It is a * distinct, retryable 503 so callers can tell a transient "index rebuilding" condition apart from a diff --git a/utility/hdbTerms.ts b/utility/hdbTerms.ts index 9f0fe07e01..d586b166ea 100644 --- a/utility/hdbTerms.ts +++ b/utility/hdbTerms.ts @@ -742,6 +742,10 @@ export const CONFIG_PARAMS = { CLONED: 'cloned', NODE_HOSTNAME: 'node_hostname', NODE_URL: 'node_url', + SQL_ENGINE: 'sql_engine', + SQL_ALLOWFULLSCAN: 'sql_allowFullScan', + SQL_MAXSORTROWS: 'sql_maxSortRows', + SQL_MAXHASHROWS: 'sql_maxHashRows', } as const; /** diff --git a/utility/logging/harper_logger.ts b/utility/logging/harper_logger.ts index db6206afbd..ea7a64d614 100644 --- a/utility/logging/harper_logger.ts +++ b/utility/logging/harper_logger.ts @@ -168,7 +168,7 @@ function resolveLogPath(configPath: string, rootPath: string) { async function updateLogSettings() { if (!rootConfig) { // set up the initial watcher - rootConfig = new RootConfigWatcher(); + rootConfig = getSharedRootConfigWatcher(); // wait for it to be ready await rootConfig.ready; // TODO: Any way to differentiate changes that we can and can't handle? @@ -1883,7 +1883,7 @@ export function AuthAuditLog( this.path = path; } // we have to load this at the end to avoid circular dependencies problems -import { RootConfigWatcher } from '../../config/RootConfigWatcher.ts'; +import { getSharedRootConfigWatcher } from '../../config/RootConfigWatcher.ts'; export const getLogFilePath = () => logFilePath; export const forComponent = (name: string, isExternal?: boolean) => mainLogger.forComponent(name, isExternal); diff --git a/validation/configValidator.ts b/validation/configValidator.ts index 727da8b991..95e4036639 100644 --- a/validation/configValidator.ts +++ b/validation/configValidator.ts @@ -30,6 +30,31 @@ const INVALID_RETENTION_VALUE_MSG = const VALID_ROTATION_DURATION_UNITS = ['D', 'd', 'H', 'h', 'M', 'm']; const UNDEFINED_OPS_API = 'rootPath config parameter is undefined'; +/** + * What `deploy_component` writes into a root-config entry, plus the deployment-level keys + * componentLoader reads off it — the shape of a `sql` application deployed before the name was + * reserved. Closed to new keys: nothing new may take this shape. + */ +export const LEGACY_SQL_APPLICATION_KEYS = [ + 'package', + 'files', + 'path', + 'install', + 'credentials', + 'loadComponent', + 'urlPath', + 'host', + 'branchedDatabases', + 'network', + 'port', + 'securePort', +]; + +export function isLegacySqlApplicationEntry(value: unknown): boolean { + if (typeof value !== 'object' || value === null) return false; + return LEGACY_SQL_APPLICATION_KEYS.some((key) => Object.hasOwn(value, key)); +} + // Directory-path validation. The previous `([...]+)+$` nested quantifier // backtracked catastrophically (ReDoS), hanging the CLI at 100% CPU on any // value with a character outside its allow-list after a run of valid ones — a @@ -107,6 +132,88 @@ export const routeConstraints = Joi.alternatives([ array.items(string), ]); +// Models — `models:` block opts a deployment into the per-backend registry. +// Per-backend shape is validated by a discriminated alternative on the +// `backend` field. Phase 2 (#629) lands ollama; Phase 3 (#630) lands openai. +// +// `.unknown(false)` on each known backend's schema turns field-name typos +// (`bakend: ollama`, `hsot: ...`) into boot-blocking validation errors. +// Without it, Joi's top-level `allowUnknown: true` propagates and typos +// silently survive into bootstrap. Unknown backend types (anything not in +// the `switch` list) fall through to a permissive schema so future Harper +// versions or third-party components can register their own backends +// without core schema edits — `bootstrapModels` logs+skips at runtime. +// +// `requestTimeoutMs: min(1)` (not `min(0)`) so the meaning is unambiguous: +// omit the field for "no timeout". `0` would validate but `composeSignal` +// treats it as "no timeout" via `if (!timeoutMs)`, surprising a test that +// sets 0 to mean "fail immediately". +const commonEntryFields = { + model: string.optional(), + requestTimeoutMs: number.min(1).optional(), + // Ordered fallback group — other logical names tried, in order, after this one (#1326). + fallback: Joi.array().items(string).optional(), +}; +const ollamaEntrySchema = Joi.object({ + backend: string.valid('ollama').required(), + host: string.optional(), + ...commonEntryFields, +}).unknown(false); +const openaiEntrySchema = Joi.object({ + backend: string.valid('openai').required(), + // `apiKey` may be a literal secret or a `${ENV_VAR}` placeholder; both + // are syntactically strings. `bootstrap.ts` runs `expandEnvVarsDeep` + // before construction; the backend rejects unresolved placeholders + // with an explicit error pointing at the env-var name. + apiKey: string.required(), + baseUrl: string.optional(), + organization: string.optional(), + ...commonEntryFields, +}).unknown(false); +const anthropicEntrySchema = Joi.object({ + backend: string.valid('anthropic').required(), + // Same secret-handling posture as openai's `apiKey`. + apiKey: string.required(), + baseUrl: string.optional(), + ...commonEntryFields, +}).unknown(false); +const bedrockEntrySchema = Joi.object({ + backend: string.valid('bedrock').required(), + // AWS credentials resolve via the SDK chain (env / shared file / IAM + // roles for service accounts) — no apiKey field. `region` is + // effectively required (Bedrock is regional) but the backend can + // fall back to AWS_REGION env, so we leave it optional here. + region: string.optional(), + ...commonEntryFields, +}).unknown(false); +const unknownBackendEntrySchema = Joi.object({ + backend: string.required(), +}).unknown(true); +const modelEntrySchema = Joi.alternatives().conditional('.backend', { + switch: [ + { is: 'ollama', then: ollamaEntrySchema }, + { is: 'openai', then: openaiEntrySchema }, + { is: 'anthropic', then: anthropicEntrySchema }, + { is: 'bedrock', then: bedrockEntrySchema }, + ], + otherwise: unknownBackendEntrySchema, +}); +const modelsSchema = Joi.object({ + embedding: Joi.object().pattern(Joi.string(), modelEntrySchema).optional(), + generative: Joi.object().pattern(Joi.string(), modelEntrySchema).optional(), +}); + +/** + * Validate a `models:` block on its own — the hot-reload path applies a watched file's block + * without the full boot validation, and must enforce the same schema boot does. + */ +export function validateModelsBlock(models) { + // `allowUnknown: true` mirrors the boot path, whose top-level option propagates into this + // subtree: a sibling of embedding/generative that boots must not block a reload. Entry-level + // strictness is preserved by each backend schema's `.unknown(false)`. + return modelsSchema.validate(models, { abortEarly: false, allowUnknown: true, errors: { wrap: { label: "'" } } }); +} + let hdbRoot; let skipFsVal = false; @@ -162,76 +269,42 @@ export function configValidator(configJson, skipFsValidation = false) { session: mcpSessionSchema.optional(), }); - // Models — `models:` block opts a deployment into the per-backend registry. - // Per-backend shape is validated by a discriminated alternative on the - // `backend` field. Phase 2 (#629) lands ollama; Phase 3 (#630) lands openai. - // - // `.unknown(false)` on each known backend's schema turns field-name typos - // (`bakend: ollama`, `hsot: ...`) into boot-blocking validation errors. - // Without it, Joi's top-level `allowUnknown: true` propagates and typos - // silently survive into bootstrap. Unknown backend types (anything not in - // the `switch` list) fall through to a permissive schema so future Harper - // versions or third-party components can register their own backends - // without core schema edits — `bootstrapModels` logs+skips at runtime. - // - // `requestTimeoutMs: min(1)` (not `min(0)`) so the meaning is unambiguous: - // omit the field for "no timeout". `0` would validate but `composeSignal` - // treats it as "no timeout" via `if (!timeoutMs)`, surprising a test that - // sets 0 to mean "fail immediately". - const commonEntryFields = { - model: string.optional(), - requestTimeoutMs: number.min(1).optional(), - // Ordered fallback group — other logical names tried, in order, after this one (#1326). - fallback: Joi.array().items(string).optional(), - }; - const ollamaEntrySchema = Joi.object({ - backend: string.valid('ollama').required(), - host: string.optional(), - ...commonEntryFields, - }).unknown(false); - const openaiEntrySchema = Joi.object({ - backend: string.valid('openai').required(), - // `apiKey` may be a literal secret or a `${ENV_VAR}` placeholder; both - // are syntactically strings. `bootstrap.ts` runs `expandEnvVarsDeep` - // before construction; the backend rejects unresolved placeholders - // with an explicit error pointing at the env-var name. - apiKey: string.required(), - baseUrl: string.optional(), - organization: string.optional(), - ...commonEntryFields, - }).unknown(false); - const anthropicEntrySchema = Joi.object({ - backend: string.valid('anthropic').required(), - // Same secret-handling posture as openai's `apiKey`. - apiKey: string.required(), - baseUrl: string.optional(), - ...commonEntryFields, - }).unknown(false); - const bedrockEntrySchema = Joi.object({ - backend: string.valid('bedrock').required(), - // AWS credentials resolve via the SDK chain (env / shared file / IAM - // roles for service accounts) — no apiKey field. `region` is - // effectively required (Bedrock is regional) but the backend can - // fall back to AWS_REGION env, so we leave it optional here. - region: string.optional(), - ...commonEntryFields, - }).unknown(false); - const unknownBackendEntrySchema = Joi.object({ - backend: string.required(), - }).unknown(true); - const modelEntrySchema = Joi.alternatives().conditional('.backend', { - switch: [ - { is: 'ollama', then: ollamaEntrySchema }, - { is: 'openai', then: openaiEntrySchema }, - { is: 'anthropic', then: anthropicEntrySchema }, - { is: 'bedrock', then: bedrockEntrySchema }, - ], - otherwise: unknownBackendEntrySchema, - }); - const modelsSchema = Joi.object({ - embedding: Joi.object().pattern(Joi.string(), modelEntrySchema).optional(), - generative: Joi.object().pattern(Joi.string(), modelEntrySchema).optional(), - }); + // `convert: false` — validateConfig() only writes the coerced value back into configDoc for + // threads/componentsRoot/logging/storage/operationsApi, not sql, so leaving Joi's default + // convert:true on here would accept a quoted `allowFullScan: "true"` and then silently drop + // it (getSqlEngineConfig()'s typeof guard rejects the still-unconverted string). + const sqlSettingsSchema = Joi.object({ + engine: string.valid('legacy', 'new', 'auto').optional(), + allowFullScan: boolean.optional(), + maxSortRows: number.integer().min(1).optional(), + maxHashRows: number.integer().min(1).optional(), + }) + .unknown(false) + .prefs({ convert: false }); + + // An application deployed under the name before it was reserved still boots; validateConfig() + // warns to rename it. Selected positively so a typo'd setting is still held to the settings + // schema instead of passing as an application. + const legacySqlApplicationSchema = Joi.object({ + engine: Joi.any().forbidden(), + allowFullScan: Joi.any().forbidden(), + maxSortRows: Joi.any().forbidden(), + maxHashRows: Joi.any().forbidden(), + }) + .unknown(true) + .messages({ + 'any.unknown': `{#label} cannot be set while 'sql' names a deployed application; redeploy that application under a different name`, + }); + const sqlSchema = Joi.alternatives() + .conditional( + Joi.object() + .or(...LEGACY_SQL_APPLICATION_KEYS) + .unknown(true), + { then: legacySqlApplicationSchema, otherwise: sqlSettingsSchema } + ) + // `false`/null is how componentLoader spells a disabled entry, so an operator who had already + // turned a pre-reservation `sql` application off keeps booting. + .allow(false, null); const configSchema = Joi.object({ authentication: Joi.alternatives( @@ -411,6 +484,7 @@ export function configValidator(configJson, skipFsValidation = false) { }).required(), mcp: mcpSchema.optional(), models: modelsSchema.optional(), + sql: sqlSchema.optional(), ignoreScripts: boolean.optional(), tls: Joi.alternatives([Joi.array().items(tlsConstraints), tlsConstraints]), });