diff --git a/DESIGN.md b/DESIGN.md index 1a9781c4d4..14df723f7f 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -649,10 +649,117 @@ with its journal intact. Only the sweep itself is best-effort, because it costs decision. For the same reason, a swap whose rename cannot be confirmed on storage skips both the retire and the journal removal: the journal is what would carry the activation forward after a power loss. -Three limits are deliberate and tracked separately: activation is two renames, so the live _pathname_ is +Two limits are deliberate and tracked separately: activation is two renames, so the live _pathname_ is briefly absent (in-memory resources are unaffected, but a component that opens its own files during a -request can still see a gap); validation does not run on the main-thread deploy path; and config -publication is not yet an effect of this transaction, as above. +request can still see a gap); and config publication is not yet an effect of this transaction, as above. + +## Certification: `.complete` requires a verdict, and the mint enforces it + +**Certification is OFF by default** (`HARPER_CERTIFY_DEPLOYS`). The mechanism below is complete and +reviewed; where the candidate load _runs_ is not settled, and that is why it is gated rather than shipped — +see "No host satisfies both requirements" below. With it off, `deploy_component` behaves as it did before +this work: the candidate is built aside and swapped in, and it earns no `.complete`. + +`.complete` is what recovery treats as proof that a candidate both built and validated, so the function +that writes it requires the verdict rather than trusting its caller to have asked for one. `validateCandidate` +used to be an optional callback on `prepareApplication` and only one of its four production call sites +supplied it — the same _one rule, N sites_ shape that produced most of this area's defects. The record of +which candidates a validator certified is module-internal: a proof passed as an argument is one an external +caller can forge, or a future caller can forget. + +Every outcome other than an explicit passing verdict is failure — a throw, an exit without a verdict, a +malformed message, a closed channel, a deadline — because the alternative is minting authority from +silence. A spawn failure is a deploy failure, not a success. The worker is terminated and its exit awaited +before its tree is swept, so a still-running candidate cannot race the sweep. + +Two cases earn no authority rather than being refused — the rule is _no verdict means no authority_, never +_no verdict means no deploy_: + +- **Safe mode** deploys uncertified. It may not execute configured code, so no validator can vouch for the + candidate. An earlier draft staged without activating, but nothing resumes a journal-less staged tree: + `recoverInterruptedActivations` removes it as build residue while the operation has already returned + success and replicated. +- **A branch-configured component** deploys uncertified. A branch's location is derived only from the + application and database names, so a certification load would open the store the live version is serving + from: a candidate could mutate rows, throw, be rejected, and leave the live version serving the mutation. + +The guarantee is scoped to the lifetime of a preparation. A package deploy's root-config entry is still +written before the build and never rolled back, so a rejected v2 can be re-prepared and activated after a +restart; closing that needs config staged with activation. + +### No host satisfies both requirements + +A certification load has two non-negotiable properties, and no available host has both. This is the reason +for the switch, and it is a real constraint rather than an unfinished implementation: + +| Host | Serving-equivalent load | Can be force-killed | +| ------------------------ | ------------------------------------------ | ------------------- | +| A thread in this process | Yes — shares the process's RocksDB handles | **No** under Bun | +| A separate process | **No** — RocksDB's lock is exclusive | Yes | + +- **A thread cannot be killed under Bun.** `terminate()` triggers a NAPI segfault there (`manageThreads` and + `jobProcess.ts` both avoid it, the latter draining its event loop instead of calling `process.exit`), so + the parent can only _ask_ the thread to exit. A candidate that blocks its event loop, or removes the + `parentPort` listener, never processes the ask: the thread never exits and its concurrency slot is held for + the life of the process. Two of those stop the node deploying. +- **A separate process cannot open the databases.** RocksDB takes an exclusive per-process file lock, so a + helper process fails with `IO error: While lock file: … Resource temporarily unavailable` the moment + `loadRootPlugins` reaches `getTables()` — and `security/auth.ts` calls `table()` at module scope, so this + is not avoidable by loading fewer plugins. Opening `readOnly` takes a shared lock and would work, but then + any candidate that writes during load — creating a table, seeding a record — is rejected by certification + and fine in production, which is a worse failure than the one being prevented. + +Two things that are easy to get wrong about the thread host, learned the expensive way: + +- **What ships today is a bare `new Worker`, and it does not load on Windows.** The thread dies _inside its + import graph_ — before its first statement, exit code 0, no `error` event — so every certification there + fails identically (HarperFast/harper#2494). The certification unit tests skip the validator-dependent + cases on Windows for that reason, and removing that skip is #2494's acceptance test, not a cleanup. +- **Moving to `startWorker` is the intended fix, and is not done.** The original reason for avoiding it was + half wrong: `isEligibleBroadcastRecipient` already excludes a job-type worker (`name: THREAD_TYPES.JOB`) + from broadcasts, and the per-peer `MessageChannel` construction is O(workers) for one slow-path deploy — + cheap for something a human triggers. The bootstrap plumbing that migration needs (`extraWorkerData`, + `extraTransferList`, `noServerStart`) landed with this work and is currently unused by certification. + What is **not** established is that the standard path repairs Windows: the evidence proves the bespoke + import graph fails, not that mesh membership or the standard bootstrap is what fixes it. #2494 names the + narrow experiment that would isolate the two. +- **`startWorker` cannot take `workerData`.** `...options` is spread into the `Worker` constructor after the + bootstrap `workerData`, so passing it replaces `addPorts`/`addThreadIds` and the thread comes up with no + ITC wiring at all. Use `extraWorkerData` + `extraTransferList` (merged, reserved keys refused), and + `options.noServerStart` for the reserved key a validator needs. + +### Diagnosing a validator that reports nothing + +The verdict travels through a `SharedArrayBuffer` flag, not a message: the validator exits the instant it +has posted, and on Windows the parent observes that exit before the queued message. A second slot carries +how far the load got, and the validator's own `exit` handler marks it — so a thread ended _from outside_ +(`terminate()`, a native abort) is distinguishable from one that ended itself, which no exit code shows. + +Do not diagnose such a thread with `console.error`: a worker's stderr is piped to the parent +asynchronously, so anything written on the way out loses the same race the verdict message loses. That +mistake cost a full CI round. Shared memory is the only channel that survives the exit. + +Isolation contains the JS heap, the module registry, process-global registrations and component status. It +does **not** contain databases, the filesystem, the network or native addons: a candidate can write before +it throws. Component authors are administrators, so this is a correctness and recovery-authority boundary, +not a security sandbox. + +Certification also loads for real, so it leaves the footprint a load leaves. `symlinkHarperModule` links the +running install into `node_modules/harper` on every non-root load — that is what makes `import 'harper'` +resolve to the live instance — so certifying writes into the tree it is only supposed to read, and that tree +is then renamed into the live path. The packer dereferences symlinks and recurses into linked directories, +so a component carrying that link packages the entire Harper install: `package_component` on a freshly added +component spent 46s tarring and then failed with "Maximum response size reached". `certifyCandidate` +therefore snapshots the candidate's `node_modules` before the load and removes only the links its own load +created, which matters because a `file:` deploy stages a symlink to the developer's own source +tree — deleting a link they already had would be certification reaching outside the candidate. A serving +worker recreates the link the next time it loads the component. Packaging a component that a worker HAS +loaded still follows the link; that is pre-existing (HarperFast/harper#2487), and `scanPackageDirectory` +documents the missing cycle protection behind it. + +A validator must also release what it opened: `loadRootPlugins` reaches `getTables()`, and a thread that +exits without `closeLoadedDatabases()` leaks process-global RocksDB handles and blocks an online +`restore_backup` from confirming a database is closed. ## Component preparation is serialized across worker threads diff --git a/components/Application.ts b/components/Application.ts index d5a6f16fb9..40fc29a1ba 100644 --- a/components/Application.ts +++ b/components/Application.ts @@ -1020,6 +1020,72 @@ export async function extractApplication( // non-recursive `rm` of a directory. `assertApplicationConfig` rejects any name `isJoinableComponentName` // rejects, so the collision is unreachable from a root-config key as well as from a deploy. const CANDIDATE_COMPLETE_MARKER = '.complete'; + +/** + * What a `.complete` marker must contain to count as roll-forward authority. + * + * The marker used to be written empty, after a validation that was a no-op on the main thread — so an + * empty marker is one minted without any verdict at all. Recovery now requires this payload, which means + * a candidate staged by an older build and found after an upgrade rolls BACK to the committed tree rather + * than forward onto something nothing certified. That is the conservative direction and it costs only an + * in-flight deploy across an upgrade. + */ +const CANDIDATE_CERTIFIED_MARKER_BODY = 'certified:1'; + +/** + * Candidates a validator has certified, by `\0`. + * + * `.complete` is what recovery treats as authority for a "build AND validation complete" candidate — it + * will roll such a candidate forward after a crash. So the function that writes it has to require the + * verdict, rather than trusting its caller to have asked for one: `validateCandidate` was an optional + * callback and only one of `prepareApplication`'s four production call sites supplied it, which is the + * one-rule-N-sites shape that produced most of the previous step's defects. + * + * Module-internal on purpose. A proof passed in as an argument is a proof an external caller can forge or + * a future caller can forget; nothing crosses this boundary, so there is nothing to get wrong. + */ +const certifiedCandidates = new Set(); + +function certificationKey(componentDirPath: string, deploymentId: string): string { + return `${componentDirPath}\0${deploymentId}`; +} + +/** + * Whether the on-disk marker says a validator certified this candidate. + * + * Content, not mere existence: an empty marker predates certification and was minted after a validation + * that did nothing on the main thread. An unreadable marker is not authority either — the same + * fail-closed reading the rest of this protocol uses. + */ +async function candidateIsCertifiedOnDisk(deploymentDirPath: string): Promise { + try { + const body = await readFile(join(deploymentDirPath, CANDIDATE_COMPLETE_MARKER), 'utf8'); + return body.trim() === CANDIDATE_CERTIFIED_MARKER_BODY; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') { + logger.warn( + `Treating the completion marker in ${deploymentDirPath} as absent because it could not be read: ` + + errorMessage(error) + ); + } + return false; + } +} + +/** Whether a validator returned a passing verdict for exactly this candidate. */ +function isCandidateCertified(componentDirPath: string, deploymentId: string): boolean { + return certifiedCandidates.has(certificationKey(componentDirPath, deploymentId)); +} + +/** Record that a validator returned a passing verdict for exactly this candidate. */ +function recordCandidateCertified(componentDirPath: string, deploymentId: string): void { + certifiedCandidates.add(certificationKey(componentDirPath, deploymentId)); +} + +/** Forget a candidate's certification once its deployment directory is gone, so the set cannot grow. */ +function forgetCandidateCertification(componentDirPath: string, deploymentId: string): void { + certifiedCandidates.delete(certificationKey(componentDirPath, deploymentId)); +} // Records activation intent beside the candidate, so recovery finishes or undoes the whole transaction — // tree and configuration together — instead of inferring intent from filesystem shape alone. const ACTIVATION_JOURNAL = '.activation.json'; @@ -1812,7 +1878,7 @@ async function settleInterruptedActivation( ); const liveExists = await exists(liveDirPath); const candidateExists = await exists(candidateDirPath); - const candidateComplete = await exists(join(deploymentDirPath, CANDIDATE_COMPLETE_MARKER)); + const candidateComplete = await candidateIsCertifiedOnDisk(deploymentDirPath); const asideRecords = await inProgressAsideRecords(asideStagingDir); const rollForward = async () => { @@ -2021,18 +2087,18 @@ async function syncTreeContents(rootPath: string, foreignTree = false): Promise< await syncDirectory(rootPath); } -export async function markCandidateComplete( - componentDirPath: string, - deploymentId: string, - componentName: string -): Promise { - // Contents first: `.complete` is roll-forward AUTHORITY, so it must not be durable before the tree it - // vouches for. - // - // A `file:` candidate IS a symlink to a tree this deploy does not own, but the dependency - // install writes THROUGH it — so the tree still has to be walked, or the install output `.complete` - // vouches for is never made durable. Only the foreign files alongside it are tolerated: see - // `syncTreeContents`. +/** + * Make a candidate's contents durable, before anything vouches for them. + * + * Separate from `markCandidateComplete` so it can be exercised without also being a way to write + * `.complete`: that marker is gated on certification, and a test seam that bypassed the gate would be a + * seam a caller could bypass it through too. + * + * A `file:` candidate IS a symlink to a tree this deploy does not own, but the dependency + * install writes THROUGH it — so the tree still has to be walked, or the install output is never made + * durable. Only the foreign files alongside it are tolerated: see `syncTreeContents`. + */ +export async function syncCandidateTree(componentDirPath: string, deploymentId: string): Promise { const candidatePath = candidateApplicationPath(componentDirPath, deploymentId); const candidateIsLink = await lstat(candidatePath).then( (stats) => stats.isSymbolicLink(), @@ -2042,13 +2108,32 @@ export async function markCandidateComplete( } ); await syncTreeContents(candidatePath, candidateIsLink); +} + +export async function markCandidateComplete( + componentDirPath: string, + deploymentId: string, + componentName: string +): Promise { + // Everything below writes the marker recovery trusts, so an uncertified candidate must not get here. The + // check lives at the mint rather than at the caller for the reason `certifiedCandidates` documents. + if (!isCandidateCertified(componentDirPath, deploymentId)) { + throw new Error( + `Refusing to mark the ${componentName} candidate ${deploymentId} complete: no validator has ` + + `certified it, and \`${CANDIDATE_COMPLETE_MARKER}\` is what recovery treats as proof that one did` + ); + } + try { await writeControlFileDurably(candidateComponentFilePath(componentDirPath, deploymentId), componentName); } catch (error) { if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error; } try { - await writeControlFileDurably(candidateCompleteMarkerPath(componentDirPath, deploymentId), ''); + await writeControlFileDurably( + candidateCompleteMarkerPath(componentDirPath, deploymentId), + CANDIDATE_CERTIFIED_MARKER_BODY + ); } catch (error) { if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error; } @@ -2064,6 +2149,104 @@ export async function markCandidateComplete( * rename is the COMMIT POINT: nothing after it may compensate, because the live path holds the candidate and * renaming the aside back over it cannot succeed. */ +/** + * Decide and obtain a candidate's certification, immediately before activation. + * + * Three outcomes, and the difference between the last two is the whole reason this is one function: + * + * - **Certified** — a validator loaded this exact tree under the real application identity and mount. + * Activation mints `.complete`. + * - **Uncertified, activate anyway** — safe mode. It may not execute configured code, so it certifies + * nothing, and it activates without minting `.complete`. + * - **Uncertified, activate anyway** — a branch-configured component. A branch's location is derived only + * from the application and database names, so a certification load would open the store the LIVE version + * is serving from; a candidate could mutate rows, throw, be rejected, and leave the live version serving + * the mutation. Not deferrable — certification cannot succeed for these until validation-scoped branch + * storage exists — so "pending" would be permanent limbo reported as success. It deploys exactly as it + * does today and simply earns no authority. + * + * The invariant across all three is *no verdict means no authority*, never *no verdict means no deploy*. + */ +/** Environment switch for the certification mechanism. Off unless explicitly set. */ +export const CERTIFY_DEPLOYS_ENV = 'HARPER_CERTIFY_DEPLOYS'; + +/** + * Whether a candidate load may run at all. **Off by default**, which is a deliberate, temporary state. + * + * The mechanism below is complete and reviewed, but WHERE the candidate load runs is unresolved, and both + * available hosts fail a requirement the guarantee depends on: + * + * - A thread shares this process's RocksDB handles, so the load is genuinely serving-equivalent — but under + * Bun `terminate()` triggers a NAPI segfault, so the parent can only ask it to exit, which a candidate + * blocking its event loop defeats. That leaks a thread and its concurrency slot for the process lifetime. + * - A separate process can be SIGKILLed, but cannot open the databases at all: RocksDB takes an exclusive + * per-process lock, so a helper fails with `IO error: While lock file: … Resource temporarily unavailable` + * the moment `loadRootPlugins` reaches `getTables()`. Opening read-only would reject any candidate that + * writes during load, which is a false-rejection class worse than the problem. + * + * So this lands off rather than shipping a guarantee that only holds on some runtimes, or a load that is not + * the load a serving worker performs. With it off, `deploy_component` behaves exactly as it did before this + * work: the candidate is built aside and swapped in, and it earns no `.complete`. Tests that exercise + * certification set the variable explicitly. + */ +function certificationEnabled(): boolean { + const value = process.env[CERTIFY_DEPLOYS_ENV]; + return value !== undefined && value !== '' && value !== 'false' && value !== '0'; +} + +async function certifyPreparedCandidate( + application: Application, + deploymentId: string, + candidateDirPath: string, + options: PrepareApplicationOptions +): Promise<{ certified: boolean }> { + if (!certificationEnabled()) { + application.logger.debug( + `Deploying ${application.name} without certification: no validator host is enabled (${CERTIFY_DEPLOYS_ENV})` + ); + return { certified: false }; + } + // Safe mode ACTIVATES, uncertified, rather than staging for later: a staged tree carries no journal, so + // `recoverInterruptedActivations` removes it as build residue at the next start — while the operation has + // already returned success, replicated, and run its restart phase. So it takes the same shape as the + // branch-configured case below: the deploy happens and earns no authority, and with no `.complete` a + // crash mid-swap rolls back to the committed tree. + const safeMode = + process.env.HARPER_SAFE_MODE && process.env.HARPER_SAFE_MODE !== 'false' && process.env.HARPER_SAFE_MODE !== '0'; + if (safeMode) { + application.logger.warn( + `Deploying ${application.name} without certification: safe mode must not execute configured code, so ` + + `no validator can vouch for this candidate` + ); + return { certified: false }; + } + + const { rootApplicationLoadOptions } = await import('./componentLoader.ts'); + const loadOptions = rootApplicationLoadOptions(application.name, { forCertification: true }); + // Answered here rather than by spawning a thread that will fail on the same thing a moment later. + if (!loadOptions.ok) { + throw new Error(`Cannot certify ${application.name}: its root-config mount could not be resolved`); + } + if (loadOptions.branchConfigured) { + application.logger.warn( + `Deploying ${application.name} without certification: a certification load would open the same ` + + `database branch the live version is serving from, so it is skipped until validation-scoped ` + + `branch storage exists` + ); + return { certified: false }; + } + + options.emitPhase?.('load', 'start'); + const { certifyCandidate } = await import('./certifyCandidate.ts'); + const outcome = await certifyCandidate(candidateDirPath, application.name, { + timeoutMs: options.certificationTimeoutMs, + }); + if (!outcome.certified) throw outcome.error ?? new Error(`${application.name} could not be certified`); + recordCandidateCertified(application.dirPath, deploymentId); + options.emitPhase?.('load', 'done'); + return { certified: true }; +} + export async function activateCandidateApplication(application: Application, deploymentId: string): Promise { const liveDirPath = application.dirPath; const candidateDirPath = candidateApplicationPath(liveDirPath, deploymentId); @@ -2080,7 +2263,32 @@ export async function activateCandidateApplication(application: Application, dep throw new Error(`Cannot activate ${application.name}: no candidate build at ${candidateDirPath}`); } - await markCandidateComplete(liveDirPath, deploymentId, application.name); + // Durability first, and for EVERY activation, certified or not: `syncRenameParents` syncs only directory + // entries, so an unflushed swap can leave the live path holding zero-length files after a power loss, + // with the aside already retired. Certification decides what the tree MEANS, not whether it is durable, + // so this must not sit behind the mint. + await syncCandidateTree(liveDirPath, deploymentId); + + // The RECORD decides, never the caller: a `certified` argument on an exported function is proof any + // caller can forge, which is the invariant this step exists for. + // + // So: certified candidates get `.complete`; the deliberately uncertified ones (a branch-configured + // component) simply do not, and a crash mid-swap rolls them back to the committed tree rather than + // forward onto something no validator vouched for. + if (isCandidateCertified(liveDirPath, deploymentId)) { + await markCandidateComplete(liveDirPath, deploymentId, application.name); + } else if (await candidateIsCertifiedOnDisk(candidateDeploymentDirPath(liveDirPath, deploymentId))) { + // A resumed activation after a restart. The in-memory record is empty then, but the marker a validator + // already earned is on disk, and recovery is rolling this candidate FORWARD on the strength of it — + // so the warning below would be the opposite of what is happening. The marker is not re-minted from + // its own presence; nothing is written here. + application.logger.info(`Resuming the interrupted activation of ${application.name}, already certified`); + } else { + application.logger.warn( + `Activating ${application.name} without a certification marker: nothing certified this candidate, so ` + + `recovery will roll it back rather than forward` + ); + } const journalPath = activationJournalPath(liveDirPath, deploymentId); try { await writeControlFileDurably( @@ -2188,6 +2396,9 @@ export async function activateCandidateApplication(application: Application, dep await rm(deploymentDirPath, { recursive: true, force: true, maxRetries: 3, retryDelay: 100 }).catch((error) => application.logger.warn(`Deployed ${application.name} but could not clean up its staging directory:`, error) ); + // Same reason as `discardCandidate`: the tree is gone, so its verdict goes too. Both sites, because + // one site remembering and its sibling forgetting is the defect shape this whole area keeps producing. + forgetCandidateCertification(liveDirPath, deploymentId); await rmdir(dirname(deploymentDirPath)).catch(() => {}); } } @@ -2328,6 +2539,10 @@ function stripExtendedLengthPrefix(target: string): string { /** Remove a candidate's whole deployment directory, best-effort — it is never the last good copy. */ async function discardCandidate(application: Application, deploymentId: string): Promise { const deploymentDirPath = candidateDeploymentDirPath(application.dirPath, deploymentId); + // The tree this certification vouched for is going, so the certification goes with it — otherwise a + // later deploy reusing the id would inherit a verdict that was never about its tree, and the set would + // grow for the life of the process. + forgetCandidateCertification(application.dirPath, deploymentId); await rm(deploymentDirPath, { recursive: true, force: true, maxRetries: 3, retryDelay: 100 }).catch((error) => application.logger.warn(`Failed to remove the abandoned deploy candidate at ${deploymentDirPath}:`, error) ); @@ -3365,7 +3580,14 @@ export type PrepareApplicationOptions = { * behavior, where the swap committed first and a load failure was reported over an already-live * broken release. */ - validateCandidate?: (candidateDirPath: string) => Promise; + /** + * Emit the operation's progress phases. Certification moved in here from `deploy_component`, which was + * the only place emitting the `load` phases — SSE clients key progress off them, so the stream has to + * keep coming from wherever certification now runs. + */ + emitPhase?: (phase: string, status: 'start' | 'done') => void; + /** Certification deadline. Callers with no request budget of their own (boot, `addComponent`) rely on the default. */ + certificationTimeoutMs?: number; }; export async function prepareApplication(application: Application, options: PrepareApplicationOptions = {}) { @@ -3415,9 +3637,14 @@ export async function prepareApplication(application: Application, options: Prep await application.cleanupGitCredentialSession(); } try { - // Validated while the previous version is still the one serving, so a candidate that + // Certified while the previous version is still the one serving, so a candidate that // installs cleanly but throws at load is rejected without ever having been live. - await options.validateCandidate?.(candidateDirPath); + // + // Here rather than in `deploy_component`, because this is the function that goes on to mint + // `.complete` — the marker recovery treats as proof a validation happened. Leaving it to the + // caller meant three of this function's four production call sites minted that authority + // without ever asking for a verdict. + await certifyPreparedCandidate(application, deploymentId, candidateDirPath, options); if (!application.isNewComponent) { application.packageMetadataChanged = installedRuntimeChanged( previousPackageMetadata, diff --git a/components/certificationProtocol.ts b/components/certificationProtocol.ts new file mode 100644 index 0000000000..cc24f78306 --- /dev/null +++ b/components/certificationProtocol.ts @@ -0,0 +1,52 @@ +/** + * The wire between a deploy validator and whatever supervises it. + * + * Its own module, with NO Harper imports, because both ends need it and one of them is a process that must + * be able to read a verdict without having loaded (or survived loading) the module graph it is testing. + * Importing `certifyCandidate` for these would pull `manageThreads` into the validator. + */ + +export const VERDICT_NO_ANSWER = 0; +export const VERDICT_CERTIFIED = 1; +export const VERDICT_REJECTED = 2; + +/** + * Slots in the shared buffer. Slot 0 carries the verdict; slot 1 carries how far the validator got. + * + * Progress travels through shared memory because that is the only channel that survives this thread's exit: + * a worker's `console.error` is piped to the parent asynchronously, so output written on the way out loses + * the same race the verdict message loses. An exit code alone cannot say which phase the thread was in, nor + * whether it ended itself. + */ +export const SLOT_VERDICT = 0; +export const SLOT_PROGRESS = 1; +export const VERDICT_SLOTS = 2; + +/** Phases the validator records in `SLOT_PROGRESS`, each strictly later than the last. */ +export const PROGRESS_NOTHING = 0; +export const PROGRESS_MODULE_SCOPE = 1; +export const PROGRESS_CERTIFY_ENTERED = 2; +export const PROGRESS_ROOT_PLUGINS_LOADED = 3; +export const PROGRESS_CANDIDATE_LOADED = 4; +export const PROGRESS_TEARDOWN_DONE = 5; +/** Added to the phase when the validator's own `exit` handler runs, distinguishing a self-exit from a teardown. */ +export const PROGRESS_EXIT_OBSERVED = 100; + +const PROGRESS_NAMES: Record = { + [PROGRESS_NOTHING]: 'never ran its module body', + [PROGRESS_MODULE_SCOPE]: 'reached module scope but not the certification body', + [PROGRESS_CERTIFY_ENTERED]: 'entered certification but did not finish loading root plugins', + [PROGRESS_ROOT_PLUGINS_LOADED]: 'loaded root plugins but did not finish loading the candidate', + [PROGRESS_CANDIDATE_LOADED]: 'loaded the candidate but did not finish tearing it down', + [PROGRESS_TEARDOWN_DONE]: 'finished teardown but reported no verdict', +}; + +/** Render `SLOT_PROGRESS` for an operator: which phase, and whether the thread ended itself. */ +export function describeProgress(progress: number): string { + const selfExited = progress >= PROGRESS_EXIT_OBSERVED; + const phase = selfExited ? progress - PROGRESS_EXIT_OBSERVED : progress; + const described = PROGRESS_NAMES[phase] ?? `reached an unknown phase (${phase})`; + // A thread torn down from outside — `terminate()`, a native abort, the process going away — never runs + // its own exit handler, so the absence of that mark is the interesting half. + return selfExited ? `it ${described} and then exited itself` : `it ${described} and was ended from outside`; +} diff --git a/components/certifyCandidate.ts b/components/certifyCandidate.ts new file mode 100644 index 0000000000..4eac941bb6 --- /dev/null +++ b/components/certifyCandidate.ts @@ -0,0 +1,460 @@ +'use strict'; + +import { readdir, lstat, realpath, rm, rmdir } from 'node:fs/promises'; +import { join } from 'node:path'; +import { MessageChannel, Worker, type MessagePort } from 'node:worker_threads'; + +import harperLogger from '../utility/logging/harper_logger.ts'; +import { PACKAGE_ROOT } from '../utility/packageUtils.js'; +import { + buildWorkerExecArgv, + buildWorkerResourceLimits, + isProcessShuttingDown, +} from '../server/threads/manageThreads.js'; + +/** + * How long a certification load may take before the candidate is rejected. + * + * Its OWN budget, measured from validator spawn — deliberately not the operations-API timeout. A component + * may legally declare a longer load timeout than that, so borrowing it would reject a candidate the + * serving workers would happily load; and a clock started at the request would already be overdue after a + * long `npm install`. Callers with no request budget at all (boot, `addComponent`) get this default. + */ +const DEFAULT_CERTIFICATION_TIMEOUT_MS = 120_000; + +/** + * The maximum number of certification workers alive at once. + * + * Deploys are already serialized per component by the preparation lock, but not across components, and a + * certification thread is a whole module graph. Bounded so a burst of deploys cannot spawn one thread per + * component simultaneously. + */ +const MAX_CONCURRENT_CERTIFICATIONS = 2; + +/** + * Verdict slots in the shared buffer below. A message can be lost — the validator exits the instant it has + * posted, and on Windows the parent observes that exit before the queued message — so the pass/fail bit + * travels through shared memory, which needs no event-loop turn and cannot be outrun by the exit. The + * message still carries the candidate's error text, which is detail rather than authority. + * + * `NO_ANSWER` is the initial value, so silence remains a failure rather than becoming a pass. + */ +export { + VERDICT_NO_ANSWER, + VERDICT_CERTIFIED, + VERDICT_REJECTED, + SLOT_VERDICT, + SLOT_PROGRESS, + VERDICT_SLOTS, + describeProgress, +} from './certificationProtocol.ts'; +import { + VERDICT_NO_ANSWER, + VERDICT_CERTIFIED, + SLOT_VERDICT, + SLOT_PROGRESS, + VERDICT_SLOTS, + describeProgress, +} from './certificationProtocol.ts'; + +/** How long to wait for a validator to actually go away before giving up and saying so. */ +const TERMINATION_GRACE_MS = 5000; + +/** + * How long a rejection waits for the validator's queued message after its exit. + * + * The shared flag says a candidate was rejected; only the MESSAGE carries the candidate's own error text, + * and on Windows the exit consistently beats it. Settling on the flag alone reported every Windows rejection + * as "exited before reporting why" and then closed the channel, discarding the syntax error the operator + * needed. The flag is already the authority, so this waits only for the detail — and briefly. + */ +const REJECTION_DETAIL_GRACE_MS = 250; + +/** The module links `symlinkHarperModule` maintains inside a component's `node_modules`. */ +const HARPER_MODULE_LINKS = ['harper', 'harperdb']; + +/** + * What the candidate's `node_modules` held BEFORE certification loaded it. + * + * Taken so the cleanup below can put the tree back exactly as the deploy staged it, rather than removing a + * link that was already there. That distinction is not academic: a `file:` deploy stages a + * SYMLINK to the developer's own source directory, so certification reads and writes through it — and a + * developer working on that component very likely already has `node_modules/harper` pointing at this + * install. Deleting theirs would be certification reaching outside the candidate to modify a working tree. + */ +async function snapshotHarperModuleLinks( + candidateDirPath: string, + installRoot: string +): Promise<{ hadNodeModules: boolean; preexisting: Set }> { + const nodeModulesDir = join(candidateDirPath, 'node_modules'); + const preexisting = new Set(); + let hadNodeModules = false; + try { + await lstat(nodeModulesDir); + hadNodeModules = true; + } catch { + return { hadNodeModules, preexisting }; + } + for (const name of HARPER_MODULE_LINKS) { + try { + if ((await realpath(join(nodeModulesDir, name))) === installRoot) preexisting.add(name); + } catch {} + } + return { hadNodeModules, preexisting }; +} + +/** + * Remove the `node_modules/harper` (and `harperdb`) link the certification load created. + * + * `symlinkHarperModule` links the running install into a component's `node_modules` on EVERY non-root load, + * so certifying a candidate writes into the tree it is only supposed to read — and that tree is then renamed + * into the live path. The link is not part of what the deploy staged, and a serving worker recreates it the + * next time it loads the component, so removing it restores the staged bytes rather than taking anything + * away. Left behind it turns every walk of the component into a walk of the whole Harper install: the packer + * dereferences and recurses into symlinked directories, so `package_component` packaged the install. + * + * Only a link this certification created, and only one resolving to THIS install, is removed — a component + * that ships or installs a `node_modules/harper` of its own keeps it. + */ +async function removeCertificationLinks( + candidateDirPath: string, + appName: string, + installRoot: string, + before: { hadNodeModules: boolean; preexisting: Set } +): Promise { + const nodeModulesDir = join(candidateDirPath, 'node_modules'); + for (const name of HARPER_MODULE_LINKS) { + if (before.preexisting.has(name)) continue; + const linkPath = join(nodeModulesDir, name); + try { + if (!(await lstat(linkPath)).isSymbolicLink()) continue; + if ((await realpath(linkPath)) !== installRoot) continue; + await rm(linkPath, { force: true }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') continue; + harperLogger.warn( + `Could not remove the ${name} module link certification left in the ${appName} candidate; ` + + `packaging or copying this component will follow it into the Harper install:`, + error + ); + } + } + // The load creates `node_modules` when the candidate has none, and an empty directory left behind is + // still a difference from the staged tree. + if (before.hadNodeModules) return; + try { + if ((await readdir(nodeModulesDir)).length === 0) await rmdir(nodeModulesDir); + } catch {} +} + +let active = 0; +/** Queued waiters. Each returns whether it accepted the slot being offered; a lapsed one declines. */ +const waiting: (() => boolean)[] = []; + +async function acquireSlot(timeoutMs: number): Promise<() => void> { + // A released slot is HANDED to the next waiter without `active` ever dipping. Decrementing and then + // waking a waiter that increments a microtask later leaves a window a fresh caller can claim through + // synchronously, sending the woken waiter to the back of its own queue; and a release offered to a + // waiter that has already timed out would be swallowed, leaving a free slot nobody is woken for. + // + // The wait is bounded because a validator that will not die keeps its slot deliberately (see the + // termination path), and an unbounded queue behind it would hold every later deploy inside the + // preparation lock with nothing to report. + const deadline = Date.now() + timeoutMs; + while (active >= MAX_CONCURRENT_CERTIFICATIONS) { + const remaining = deadline - Date.now(); + if (remaining <= 0) { + const error: any = new Error( + `No certification slot became available within ${timeoutMs}ms; ${MAX_CONCURRENT_CERTIFICATIONS} ` + + `validator(s) are still running` + ); + error.statusCode = 503; + throw error; + } + let settled = false; + let resolveWait!: (inherited: boolean) => void; + const waited = new Promise((resolve) => (resolveWait = resolve)); + const waiter = () => { + if (settled) return false; + settled = true; + resolveWait(true); + return true; + }; + waiting.push(waiter); + const timer = setTimeout(() => { + if (settled) return; + settled = true; + const index = waiting.indexOf(waiter); + if (index !== -1) waiting.splice(index, 1); + resolveWait(false); + }, remaining); + timer.unref?.(); + const inherited = await waited; + // Cleared either way, so an admitted waiter does not keep its closure registered until a deadline + // that no longer applies to it. + clearTimeout(timer); + // The slot came from the releaser with `active` already accounting for it, so this caller holds it + // without incrementing — which is what makes the handoff a handoff. + if (inherited) return makeRelease(); + } + active++; + return makeRelease(); +} + +function makeRelease(): () => void { + let released = false; + return () => { + if (released) return; + released = true; + // Offer the slot along the queue until someone takes it. Only if nobody does does the count fall. + while (waiting.length > 0) { + if (waiting.shift()!()) return; + } + active--; + }; +} + +export interface CertificationOutcome { + certified: boolean; + /** Why not, when `certified` is false. Always present in that case. */ + error?: Error; +} + +/** + * Load a candidate in an ephemeral worker and report whether it loaded. + * + * Spawned with a bare `new Worker` rather than `startWorker()`, which is a known problem rather than a + * settled choice: this thread does not load at all on Windows (HarperFast/harper#2494), and `startWorker` + * is the intended fix. The original argument for avoiding it — that a certification worker would join the + * ITC mesh, letting a candidate's top-level `server.registerOperation()` announce itself to main, at + * O(deploys × workers) channels — is only half right: `isEligibleBroadcastRecipient` already excludes a + * job-type worker from broadcasts, and the channel cost is O(workers) for one slow-path deploy. See + * DESIGN.md's certification section for what is and is not established about the Windows failure. + * + * Every outcome other than an explicit passing verdict is a failure, because `.complete` — which this + * gates — is what crash recovery treats as proof that a validation happened. + */ +export async function certifyCandidate( + candidateDirPath: string, + appName: string, + { timeoutMs = DEFAULT_CERTIFICATION_TIMEOUT_MS }: { timeoutMs?: number } = {} +): Promise { + // The same guard `startWorker` applies. A deploy racing shutdown would otherwise spawn a thread the + // shutdown path does not know about, and then wait out its deadline on a process that is leaving. + if (isProcessShuttingDown()) { + const error: any = new Error(`Cannot certify ${appName} while the Harper process is shutting down`); + error.statusCode = 503; + return { certified: false, error }; + } + let releaseSlot: () => void; + try { + releaseSlot = await acquireSlot(timeoutMs); + } catch (error) { + return { certified: false, error: error as Error }; + } + // The COMPILED sibling, referenced the way `jobRunner` references `jobProcess.js`: workers load from the + // build output, and `__dirname` resolves there without assuming where the package root is. + const entry = join(__dirname, './deployValidator.js'); + let worker: Worker | undefined; + // Must be installed the moment the worker exists, never in the `finally`: the validator `realExit`s + // immediately after posting, so a listener attached after the exit has already fired never runs, and + // `certifyCandidate` then never returns while holding the preparation lock. + let exited: Promise | undefined; + // Set when a validator outlives its termination grace, so its slot is deliberately not returned. + let slotHeld = false; + let settled = false; + let timer: NodeJS.Timeout | undefined; + // Declared out here, ASSIGNED inside the `try`. The slot is already held by this point, so anything + // fallible between here and the `finally` — channel or buffer allocation under resource pressure, a + // `realpath` that throws — would leak it permanently, since nothing else decrements `active`. + let verdicts: MessagePort | undefined; + let installRoot: string | undefined; + let linksBefore: Awaited> | undefined; + + try { + // A channel of its own, never `parentPort`: that carries Harper's ITC traffic, so an unrelated message + // arriving first reads as a malformed verdict. On a dedicated channel, anything non-conforming really is. + const channel = new MessageChannel(); + verdicts = channel.port1; + const verdictPort = channel.port2; + const verdictFlag = new Int32Array(new SharedArrayBuffer(VERDICT_SLOTS * 4)); + // Before the load, so the cleanup in the `finally` can tell what it created from what was already there. + installRoot = await realpath(PACKAGE_ROOT).catch(() => undefined); + linksBefore = installRoot ? await snapshotHarperModuleLinks(candidateDirPath, installRoot) : undefined; + + // Re-checked here, not only at entry. The slot wait above is unbounded up to `timeoutMs` and the two + // filesystem calls after it are awaited, so shutdown can begin while this caller is queued — and a + // validator started then is absent from `manageThreads.workers`, so shutdown neither terminates nor + // awaits it while it loads databases and components into a process that is tearing down. + if (isProcessShuttingDown()) { + const error: any = new Error(`Cannot certify ${appName} while the Harper process is shutting down`); + error.statusCode = 503; + return { certified: false, error }; + } + + return await new Promise((resolve) => { + // Exactly one settlement, whichever of the outcomes below happens first. A candidate with a + // syntax error emits `error` AND then `exit`; a candidate that posts a verdict and then throws + // during teardown does the reverse. Either way the first answer stands. + const settle = (outcome: CertificationOutcome) => { + if (settled) return; + settled = true; + if (timer) clearTimeout(timer); + resolve(outcome); + }; + const fail = (message: string) => settle({ certified: false, error: new Error(message) }); + + try { + const started = new Worker(entry, { + workerData: { + candidateDirPath, + appName, + verdictPort, + verdictFlag, + // `server/DESIGN.md`: "Workers receive `workerData.noServerStart = true` — never start the + // server inside a worker." Without it `threadServer` boots at module scope and loads every + // root component, so the validator would serve traffic and certify the wrong thing. + noServerStart: true, + }, + transferList: [verdictPort], + // The same interpreter setup every Harper worker gets. Without it this thread cannot load + // Harper's own module graph at all — a module that imports JSON fails outright — so this is + // shared with `startWorker` rather than reconstructed. + // Without the configured APM preloads: a validator is a throwaway thread, and resolving the + // preload list from here would memoize it earlier than the first serving worker — see + // `buildWorkerExecArgv`. + execArgv: buildWorkerExecArgv({ preloads: false }), + // Bounded like every other Harper worker. Without limits a candidate whose top-level load + // builds a large in-memory index balloons a thread nothing constrains, and the OOM killer + // takes the whole process down mid-deploy — while the previous release was healthy. + resourceLimits: buildWorkerResourceLimits(), + argv: process.argv.slice(2), + }); + worker = started; + exited = new Promise((resolve) => started.once('exit', () => resolve())); + } catch (error) { + // A synchronous spawn throw is a deploy failure, not a candidate failure — and specifically + // not a success. Under thread pressure a node will refuse deploys rather than publish + // uncertified trees. + const spawnError = error instanceof Error ? error : new Error(String(error)); + spawnError.message = `Could not start a validator to certify ${appName}: ${spawnError.message}`; + (spawnError as any).statusCode = 503; + settle({ certified: false, error: spawnError }); + return; + } + + timer = setTimeout(() => { + fail(`Certification of ${appName} did not finish within ${timeoutMs}ms; the candidate was not ` + `published`); + }, timeoutMs); + // A hung candidate must not keep the process alive once the parent has stopped caring. + timer.unref?.(); + + verdicts.on('message', (message: unknown) => { + if (!message || typeof message !== 'object' || typeof (message as any).ok !== 'boolean') { + fail(`Certification of ${appName} returned a malformed verdict`); + return; + } + if ((message as any).ok) { + settle({ certified: true }); + return; + } + const error = new Error((message as any).message || `${appName} failed to load`); + if (typeof (message as any).stack === 'string') error.stack = (message as any).stack; + settle({ certified: false, error }); + }); + worker.on('error', (error) => { + const failure = error instanceof Error ? error : new Error(String(error)); + settle({ certified: false, error: failure }); + }); + worker.on('exit', (code) => { + // Only reached when no verdict MESSAGE arrived first. The shared flag is written before the + // validator exits, so it is still authoritative here — this is the ordinary path on Windows, + // where the exit consistently beats the queued message. + const flag = Atomics.load(verdictFlag, SLOT_VERDICT); + if (flag === VERDICT_NO_ANSWER) { + fail( + `Certification of ${appName} exited with code ${code} without reporting a verdict: ` + + describeProgress(Atomics.load(verdictFlag, SLOT_PROGRESS)) + ); + return; + } + if (flag === VERDICT_CERTIFIED) { + settle({ certified: true }); + return; + } + // Rejected for certain; what is still in flight is WHY. The message handler settles with the + // candidate's own error if it lands first, and `settle` takes the first answer either way, so + // this is the fallback rather than a race the detail can lose outright. + setTimeout(() => { + fail(`${appName} failed to load during certification (its validator exited before reporting why)`); + }, REJECTION_DETAIL_GRACE_MS).unref?.(); + }); + }); + } finally { + // Terminated and AWAITED before the caller may delete the candidate tree, so a still-running + // candidate cannot race the sweep. + // + // `terminate()` triggers a NAPI segfault under Bun — `manageThreads` avoids it the same way — so + // there the worker is asked to exit itself and its exit awaited. A termination that FAILS is + // reported rather than treated as cleanup done: the caller is about to remove a tree this thread + // may still be reading. + if (worker) { + // Every wait here is bounded: this `finally` blocks the caller's deploy, so a termination that never + // settles would hold the preparation lock indefinitely. + const grace = () => new Promise((resolve) => setTimeout(resolve, TERMINATION_GRACE_MS).unref?.()); + try { + // Asking is what can fail; waiting for the exit must happen either way. A synchronous throw from + // the ask must not skip the wait, or the caller sweeps a tree whose thread is still terminating. + let asked: Promise = Promise.resolve(); + try { + asked = + typeof (globalThis as any).Bun !== 'undefined' + ? // `terminate()` triggers a NAPI segfault under Bun; `manageThreads` asks the worker to + // exit itself for the same reason. + (worker.postMessage({ type: 'force-exit' }), Promise.resolve()) + : worker.terminate().then( + () => {}, + () => {} + ); + } catch (error) { + harperLogger.warn(`Could not ask the validator for ${appName} to exit; waiting for it anyway:`, error); + } + // ONE grace for the whole thing: racing terminate and then the exit separately could wait twice + // over before calling a hung validator hung, inside a deploy holding the preparation lock. + const outcome = await Promise.race([ + Promise.all([asked, exited ?? Promise.resolve()]).then(() => 'exited' as const), + grace().then(() => 'still-running' as const), + ]); + // A validator that would not die keeps its slot: releasing it would admit another thread while a + // runaway one still holds the candidate tree open, and the caller is about to sweep that tree. + if (outcome === 'still-running') { + // Held until it actually goes away, not forever: permanently held slots would shrink the cap + // for the life of the process, trading a bounded overcommit for an unbounded outage. + slotHeld = true; + harperLogger.error( + `The validator certifying ${appName} did not exit within ${TERMINATION_GRACE_MS}ms; holding its ` + + `slot until it does, and its candidate tree may still be open` + ); + void (exited ?? Promise.resolve()).then(() => { + harperLogger.warn(`The validator certifying ${appName} exited late; returning its slot`); + releaseSlot(); + }); + } + } catch (error) { + harperLogger.warn( + `Could not terminate the validator for ${appName}; its candidate tree may still be open:`, + error + ); + } + } + if (timer) clearTimeout(timer); + // After the termination above, so this is not racing a thread that could relink. Runs for every + // outcome: a rejected candidate is swept and a certified one is renamed live, and neither should + // carry the link. + if (installRoot && linksBefore) await removeCertificationLinks(candidateDirPath, appName, installRoot, linksBefore); + // Both ends, or the channel keeps this thread's event loop referenced. Optional because a throw + // before the assignment above means there is nothing to close. + verdicts?.close(); + if (!slotHeld) releaseSlot(); + } +} diff --git a/components/componentLoader.ts b/components/componentLoader.ts index 27a304d70a..5116ac1785 100644 --- a/components/componentLoader.ts +++ b/components/componentLoader.ts @@ -157,6 +157,35 @@ function rootConfigMount(appName: string): ScopeMount | undefined { * branchedDatabases: [data] * ``` */ +/** + * The load options boot uses for a root-config application, so a deploy's certification load can be built + * from the same place instead of hand-plumbing a subset and drifting from it. + * + * `branchedDatabases` is REPORTED, not applied, when `forCertification` is set. A branch's location is + * derived only from the application and database names (`resources/branchDatabase.ts`), so a certification + * load that resolved branches the way boot does would open the very store the live application is serving + * from — a candidate could mutate rows, then throw, then be rejected, leaving the live version serving the + * mutation. Certifying against the base store instead is no better: it is not a rehearsal of the real + * thing and can write to base. So the caller is told the component is branch-configured and skips + * certification for it rather than certifying something it did not test. + */ +export function rootApplicationLoadOptions( + appName: string, + { forCertification = false }: { forCertification?: boolean } = {} +): { ok: true; options: Record; branchConfigured: boolean } | { ok: false } { + const mountResult = tryRootConfigMount(appName); + if (!mountResult.ok) return { ok: false }; + const branchedDatabases = rootConfigBranchedDatabases(appName); + const options: Record = { + isRoot: false, + autoReload: false, + appName, + mount: mountResult.mount, + }; + if (!forCertification) options.branchedDatabases = branchedDatabases; + return { ok: true, options, branchConfigured: branchedDatabases !== undefined }; +} + function rootConfigBranchedDatabases(appName: string): string[] | true | undefined { if (Object.hasOwn(TRUSTED_RESOURCE_PLUGINS, appName)) return undefined; return (getConfigObj()?.[appName] as any)?.branchedDatabases; @@ -269,15 +298,11 @@ export async function loadComponentDirectories( } return; } - const mountResult = tryRootConfigMount(appName); - if (!mountResult.ok) return; + const loadOptions = rootApplicationLoadOptions(appName); + if (!loadOptions.ok) return; const loadedModules = new Set(); await loadComponent(appFolder, cycleResources, HDB_ROOT_DIR_NAME, { - isRoot: false, - autoReload: false, - appName, - mount: mountResult.mount, - branchedDatabases: rootConfigBranchedDatabases(appName), + ...loadOptions.options, collectLoadedModules: loadedModules, }); await readyComponentModules(loadedModules, readyComponentPromises); diff --git a/components/deployValidator.ts b/components/deployValidator.ts new file mode 100644 index 0000000000..7f47be7abc --- /dev/null +++ b/components/deployValidator.ts @@ -0,0 +1,238 @@ +'use strict'; + +// The guard first, exactly as `jobProcess.ts` does it: from here on this thread runs a CANDIDATE's code, +// which must not be able to terminate the thread out from under the verdict protocol. +import { realExit } from '../server/threads/workerProcessGuard.ts'; + +import { stat } from 'node:fs/promises'; +import { basename, join } from 'node:path'; +import { parentPort, workerData } from 'node:worker_threads'; + +import { HDB_ROOT_DIR_NAME } from '../utility/hdbTerms.ts'; +import harperLogger from '../utility/logging/harper_logger.ts'; +import { + PROGRESS_CANDIDATE_LOADED, + PROGRESS_CERTIFY_ENTERED, + PROGRESS_EXIT_OBSERVED, + PROGRESS_MODULE_SCOPE, + PROGRESS_ROOT_PLUGINS_LOADED, + PROGRESS_TEARDOWN_DONE, + SLOT_PROGRESS, + SLOT_VERDICT, + VERDICT_CERTIFIED, + VERDICT_REJECTED, +} from './certificationProtocol.ts'; +import { loadComponent, rootApplicationLoadOptions, setErrorReporter } from './componentLoader.ts'; +import type { Scope } from './Scope.ts'; + +/** + * Entry point for an ephemeral deploy-certification worker. + * + * Loads ONE candidate tree under its real application identity and root-config mount, and reports whether + * it loaded. Nothing else: this thread never joins the serving topology, serves a request, or outlives its + * verdict. It exists so that `deploy_component` can certify a candidate on any thread — the in-process + * check it replaces was gated on `!isMainThread`, and the operations API deploys on main, so an operator + * deploy was certified by nothing at all. + * + * The verdict is posted exactly once and the thread then exits. Every other outcome — a throw, an exit + * without a verdict, a malformed message — is failure at the parent, which is what makes "inability to + * obtain a verdict is failure" true rather than aspirational. + */ +// Captured and REMOVED from `workerData` before any candidate code runs: `workerData` is reachable from +// the candidate via `require('node:worker_threads')`, so a port left there would let a candidate post its +// own passing verdict and certify itself. +const { candidateDirPath, appName } = workerData ?? {}; +const verdictPort = workerData?.verdictPort; +const verdictFlag: Int32Array | undefined = workerData?.verdictFlag; +if (workerData) { + delete workerData.verdictPort; + delete workerData.verdictFlag; +} + +/** Record the furthest phase reached, in the one channel that survives this thread's exit on every platform. */ +function markProgress(phase: number): void { + if (verdictFlag) Atomics.store(verdictFlag, SLOT_PROGRESS, phase); +} + +markProgress(PROGRESS_MODULE_SCOPE); + +/** Whether a candidate declares anything a load should act on, so "loaded nothing" can be judged. */ +async function declaresLoadableContent(dirPath: string): Promise { + for (const name of ['config.yaml', 'config.yml', 'package.json']) { + try { + await stat(join(dirPath, name)); + return true; + } catch { + // Absent is the only answer that matters here; an unreadable candidate fails the load itself. + } + } + return false; +} + +/** The bootstrap's own bound, well inside the parent's certification deadline so this error is the one seen. */ +const BOOTSTRAP_DEADLINE_MS = 60_000; + +/** Reject with a phase-naming error if `work` does not settle in time, so a hang is diagnosable. */ +async function withPhaseDeadline(work: Promise, ms: number, phase: string): Promise { + let timer: NodeJS.Timeout | undefined; + try { + return await Promise.race([ + work, + new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`Certification timed out after ${ms}ms while ${phase}`)), ms); + }), + ]); + } finally { + if (timer) clearTimeout(timer); + } +} + +/** + * Release the RocksDB handles this thread opened, whatever happened. + * + * `loadRootPlugins` reaches `getTables()`, so the database graph is open before the candidate is touched — + * which means a bootstrap failure or its phase deadline must not skip this. `closeLoadedDatabases` + * documents that a thread exiting without it leaks handles PROCESS-wide, because rocksdb-js's registry is + * process-global, and that the leak blocks an online `restore_backup` from confirming a database is closed. + * + * A failure here is logged, not turned into a rejection: the candidate's own teardown failing is the + * candidate's fault and does reject (see the scope closes), but Harper's teardown failing is not, and + * rejecting a working component for it would fail a good deploy without un-leaking anything. + */ +async function releaseDatabases(componentName: string): Promise { + try { + const { closeLoadedDatabases } = await import('../resources/databases.ts'); + closeLoadedDatabases(); + } catch (error) { + harperLogger.warn(`Could not release database handles after certifying ${componentName}:`, error); + } +} + +async function certify(): Promise { + markProgress(PROGRESS_CERTIFY_ENTERED); + const componentName = appName || basename(candidateDirPath); + const loadOptions = rootApplicationLoadOptions(componentName, { forCertification: true }); + if (!loadOptions.ok) { + throw new Error(`Cannot certify ${componentName}: its root-config mount could not be resolved`); + } + // The global plugins first: they create the surfaces applications extend, so a component assigning + // `server.mqtt.authorizeClient` needs the mqtt plugin loaded or the assignment throws on `undefined` and + // certification rejects something a serving worker loads fine. Loading stops before the other + // applications, which is why this is a separate entry point from `loadRootComponents`. + const { loadRootPlugins } = await import('../server/loadRootComponents.js'); + try { + // Bounded, and it says WHICH phase did not finish. This bootstrap loads Harper's global plugins, parts + // of which expect to be a member of the worker topology a validator deliberately is not — so it can + // wait on something that will never arrive here. Without a bound that is indistinguishable from a + // candidate that hangs, and on Windows it presented as a silent exit. + const resources = await withPhaseDeadline( + loadRootPlugins(true), + BOOTSTRAP_DEADLINE_MS, + `loading Harper's global plugins` + ); + markProgress(PROGRESS_ROOT_PLUGINS_LOADED); + + // Installed AFTER the bootstrap so it only ever sees the candidate. Earlier, it captures the first error + // from anything the root config names — including the candidate's own live path, which `deploy_component` + // writes before building and which does not exist yet on a first deploy. + // + // A reporter is needed at all because the loader reports some failures through it while still resolving. + let reportedError: Error | undefined; + setErrorReporter((error: Error) => (reportedError ??= error)); + // Collected so teardown happens BEFORE the verdict. A scope that fails to close is a rejected + // validation, not a warning: `close()` stops at the throwing listener, leaving the scope partially live, + // and the thread exits either way so nothing downstream would learn of a failure reported after a pass. + const scopes = new Set(); + const modules = new Set(); + let loaded = false; + try { + await loadComponent(candidateDirPath, resources, HDB_ROOT_DIR_NAME, { + ...loadOptions.options, + collectScopes: scopes, + collectLoadedModules: modules, + }); + if (reportedError) throw reportedError; + // A load that did nothing is not a pass: a run that neither opened a scope nor loaded a module has + // not exercised the candidate, which is how a platform-specific no-op would read as a clean verdict. + // A static-only component stays clear of this because its load still opens a scope, which + // `deployCertification.test.js` pins. + if (!scopes.size && !modules.size && (await declaresLoadableContent(candidateDirPath))) { + throw new Error( + `Certification of ${componentName} loaded nothing: it declares component configuration, so a run ` + + `that opened no scope and loaded no module has not exercised it` + ); + } + loaded = true; + markProgress(PROGRESS_CANDIDATE_LOADED); + } finally { + const closes = await Promise.allSettled(Array.from(scopes, (scope) => scope.close())); + const failed = closes.filter((result) => result.status === 'rejected'); + // Only when the load itself succeeded. A throw from `loadComponent` — a syntax error, an unreadable + // file — reaches this block too, and a teardown failure there would replace the candidate's real + // error with a note about its scopes: the operator would get the symptom instead of the cause. + if (failed.length && loaded) { + throw new AggregateError( + failed.map((result) => (result as PromiseRejectedResult).reason), + `${componentName} loaded but ${failed.length} scope(s) failed to tear down` + ); + } + if (loaded) markProgress(PROGRESS_TEARDOWN_DONE); + } + } finally { + await releaseDatabases(componentName); + } +} + +// The receiver for the parent's Bun force-exit, registered here rather than inherited as an import side +// effect of `manageThreads`: `terminate()` segfaults under Bun, so this message is the only way the parent +// can end the thread, and that must not depend on which modules happened to load. +parentPort?.on('message', (message: any) => { + if (message?.type === 'force-exit') realExit(0); +}); + +function report(verdict: { ok: true } | { ok: false; message: string; stack?: string }): void { + // The flag first, and synchronously: this thread exits immediately after, and a posted message loses + // that race on Windows. Shared memory needs no event-loop turn. + if (verdictFlag) Atomics.store(verdictFlag, SLOT_VERDICT, verdict.ok ? VERDICT_CERTIFIED : VERDICT_REJECTED); + // A closed or absent channel is the parent's problem to detect — it treats a missing verdict as failure — + // so this must not throw its way out of the exit path. + try { + verdictPort?.postMessage(verdict); + } catch (error) { + harperLogger.warn('Deploy certification could not post its verdict:', error); + } +} + +// Provenance for an exit that reported nothing, written where it CANNOT be lost. +// +// A first attempt logged a stack from this handler and produced nothing on Windows: a worker's +// `console.error` is piped to the parent asynchronously, so anything written on the way out loses the same +// race the verdict message loses. Shared memory needs no event-loop turn, so the phase markers below travel +// through it instead, and this handler marks that the thread ended ITSELF — a thread torn down from outside +// (`terminate()`, a native abort, the process going away) never runs it, and that absence is the evidence. +process.on('exit', () => { + if (!verdictFlag) return; + const progress = Atomics.load(verdictFlag, SLOT_PROGRESS); + if (progress < PROGRESS_EXIT_OBSERVED) Atomics.store(verdictFlag, SLOT_PROGRESS, progress + PROGRESS_EXIT_OBSERVED); +}); + +// A ref'd handle for the whole certification: a worker's event loop draining ends the thread even with a +// promise still pending, which would reach the parent as "exited without a verdict" rather than as a hang, +// with no timeout left alive to fire. +const keepAlive = setInterval(() => {}, 1000); + +void (async () => { + try { + await certify(); + report({ ok: true }); + realExit(0); + } catch (error) { + const failure = error instanceof Error ? error : new Error(String(error)); + // The message travels rather than the Error: an Error does not survive `postMessage` with its + // prototype, and the parent only needs what it will put in the operation's own error. + report({ ok: false, message: failure.message, stack: failure.stack }); + realExit(1); + } finally { + clearInterval(keepAlive); + } +})(); diff --git a/components/operations.js b/components/operations.js index d1ac41076b..b08ca88fdb 100644 --- a/components/operations.js +++ b/components/operations.js @@ -28,7 +28,6 @@ const { scanPackageDirectory, streamPackagedDirectory, } = require('../components/packageComponent.ts'); -const { Resources } = require('../resources/Resources.ts'); const { Application, prepareApplication, @@ -437,107 +436,6 @@ async function packageComponent(req) { return { project, payload }; } -/** - * Load the built candidate to surface load-time errors. A load-ERROR PROBE, not a safety guarantee: it runs - * the component's own top-level code with incomplete side-effect isolation. - * - * A no-op on the main thread, and the operations API deploys there — so operator deploys are unvalidated - * (#2315 step 2). What this guarantees is ORDER: where validation runs, a rejected candidate never goes live. - */ -// `componentLoader.setErrorReporter` is ONE process-global callback, so two components validating -// concurrently on the same worker cross-attribute their failures: B installs its reporter while A is -// loading, A's load error lands in B, and A then activates broken code while B rejects a good candidate. -// Validation is serialized (it is already the slow path) and the previous reporter is restored, so the -// global is only ever owned by one in-flight validation. -let validationChain = Promise.resolve(); - -async function validateComponentLoads(candidateDirPath, emit) { - const run = validationChain.then( - () => validateComponentLoadsExclusive(candidateDirPath, emit), - () => validateComponentLoadsExclusive(candidateDirPath, emit) - ); - validationChain = run.then( - () => {}, - () => {} - ); - return run; -} - -async function validateComponentLoadsExclusive(candidateDirPath, emit) { - // now we attempt to actually load the component in case there is - // an error we can immediately detect and report, but app code should not run on the main thread - if (!isMainThread && !process.env.HARPER_SAFE_MODE) { - const pseudoResources = new Resources(); - pseudoResources.isWorker = true; - - const componentLoader = require('./componentLoader.ts').default || require('./componentLoader.ts'); - const { trackScopeClose } = require('./scopeShutdown.ts'); - let lastError; - const priorErrorReporter = componentLoader.getErrorReporter?.(); - componentLoader.setErrorReporter((error) => (lastError = error)); - emit('phase', { phase: 'load', status: 'start' }); - // This load exists only to surface load-time errors early; the Scopes it creates are - // throwaway. They are collected (instead of registered for worker-shutdown auto-close) so we - // can close them here once validation completes — otherwise each deploy leaks the Scope's - // deploy-lifecycle listeners on this worker, eventually tripping MaxListenersExceededWarning - // (#1462). - const validationScopes = new Set(); - // Process-wide `server.*` registrations (registerOperation, setMcpQuotaHandler) are not owned by - // a Scope, so a candidate's top-level registration during this throwaway load would otherwise - // outlive it and pollute the live worker on a failed/rolled-back deploy. The guard makes those - // registration methods no-op for the duration of the load. - const { runWithDeployValidationGuard } = require('../server/serverHelpers/deployValidationState.ts'); - // The candidate loads under the REAL component's name, so a candidate that throws would mark the live - // component ERROR. Its status writes are diverted into the guard's throwaway sink instead — see - // `deployValidationState.ts` for why this is context-scoped rather than captured and reverted here. - const componentName = path.basename(candidateDirPath); - // Extension modules the candidate load pulls in are registered in the loader's module registry keyed by - // module, so forgetting only the candidate's realpath leaves those behind — one set per deploy. Their - // identities are collected by the load itself; diffing the global registry instead would delete a live - // module registered by an interleaving real load, since validations serialize only with each other. - const validationModules = new Set(); - const validation = runWithDeployValidationGuard(async () => { - try { - await componentLoader.loadComponent(candidateDirPath, pseudoResources, undefined, { - collectScopes: validationScopes, - collectLoadedModules: validationModules, - }); - } finally { - const closeResults = await Promise.allSettled(Array.from(validationScopes, (scope) => scope.close())); - const failedCloses = closeResults.filter((result) => result.status === 'rejected'); - for (const result of failedCloses) { - log.warn('Failed to close a deploy-validation Scope', result.reason); - } - // A rejected close is a REJECTED VALIDATION, not a warning. `Scope.close()` stops at the - // throwing listener, so its remaining internal listener removal and subscription-hold release - // never run and the throwaway scope stays partially live — one leak per deploy, on the worker - // that serves the component. - if (failedCloses.length) { - throw new AggregateError( - failedCloses.map((result) => result.reason), - `Could not tear down deploy validation for ${componentName}: ${failedCloses.length} scope(s) failed to close` - ); - } - } - }); - // Track the load+close so a concurrent worker shutdown waits for these scopes to finish - // disposing — a plugin may start a native runtime in handleApplication — before realExit. - trackScopeClose(validation); - try { - await validation; - } finally { - componentLoader.setErrorReporter(priorErrorReporter); - // The candidate path is unique per deploy, so leaving it in the loader's realpath registry leaks - // one dead entry per deploy for the life of the process. - componentLoader.forgetLoadedPath?.(candidateDirPath); - componentLoader.forgetLoadedModules?.(validationModules); - } - emit('phase', { phase: 'load', status: 'done' }); - - if (lastError) throw lastError; - } -} - const BRANCH_STORAGE_RETAINED = '. Any branched database storage this application owns was left in place; drop it again with restart: true to discard that data'; @@ -755,12 +653,20 @@ async function deployComponent(req) { // committed" — so a later failure arrives after both phases reported success. The operation's error // is the authority on whether the deploy landed, not the phase stream. emit('phase', { phase: 'prepare', status: 'start' }); + let prepareDoneEmitted = false; await prepareApplication(application, { - validateCandidate: async (candidateDirPath) => { - emit('phase', { phase: 'prepare', status: 'done' }); - await validateComponentLoads(candidateDirPath, emit); + // `prepare` closes when the candidate is built, then `load` brackets certification. + emitPhase: (phase, status) => { + if (phase === 'load' && status === 'start' && !prepareDoneEmitted) { + prepareDoneEmitted = true; + emit('phase', { phase: 'prepare', status: 'done' }); + } + emit('phase', { phase, status }); }, }); + // A candidate that is deliberately not certified (safe mode, or a branch-configured component) + // emits no `load` phase, so `prepare` still has to be closed out. + if (!prepareDoneEmitted) emit('phase', { phase: 'prepare', status: 'done' }); const rollingRestart = req.restart === 'rolling'; // if doing a rolling restart set restart to false so that other nodes don't also restart. req.restart = rollingRestart ? false : req.restart; diff --git a/integrationTests/deploy/certified-deploy.test.ts b/integrationTests/deploy/certified-deploy.test.ts new file mode 100644 index 0000000000..9bc6c6f21b --- /dev/null +++ b/integrationTests/deploy/certified-deploy.test.ts @@ -0,0 +1,130 @@ +/** + * `deploy_component` never publishes a candidate that fails to load — through the real operations API, + * which deploys on the MAIN thread (#2315 step 2). + * + * The regression this guards: the load check used to be gated on `!isMainThread`, and the operations API + * deploys on main, so for an operator deploy it ran nothing at all. Step 1 put the check between build and + * swap, but on this path there was no check to order — a candidate that installed cleanly and threw at + * load was published anyway, while the operation reported an error. + * + * Deliberately end to end rather than a unit test of the certification helper: what was broken was the + * WIRING on this thread, and a helper test cannot see that. So this drives the API, then asserts the + * previous release still ANSWERS REQUESTS — which step 1's availability test explicitly did not cover, + * since it sampled the component directory on disk and exercised no route. + */ +import { suite, test, before, after } from 'node:test'; +import { ok, strictEqual } from 'node:assert'; +import { join } from 'node:path'; +import { mkdtemp, writeFile, rm, readFile } from 'node:fs/promises'; +import { existsSync } from 'node:fs'; +import { tmpdir } from 'node:os'; + +import { startHarper, teardownHarper, targz, type ContextWithHarper } from '@harperfast/integration-testing'; +import { operation, readVersion } from './redeploy-restart-flag-helpers.ts'; + +const PROJECT = 'certified-deploy'; + +/** A component exposing its version over REST. `throwsAtLoad` makes its resource module throw on import. */ +async function buildPayload(version: number, { throwsAtLoad = false } = {}): Promise { + const dir = await mkdtemp(join(tmpdir(), 'certified-deploy-fixture-')); + try { + await writeFile(join(dir, 'package.json'), JSON.stringify({ name: PROJECT, version: `${version}.0.0` })); + await writeFile(join(dir, 'version.txt'), String(version)); + await writeFile(join(dir, 'config.yaml'), 'rest: true\njsResource:\n files: resource.js\n'); + await writeFile( + join(dir, 'resource.js'), + throwsAtLoad + ? `throw new Error('v${version} cannot load');\n` + : `export class Version extends Resource {\n\tget() {\n\t\treturn { version: ${version} };\n\t}\n}\n` + ); + return await targz(dir); + } finally { + await rm(dir, { recursive: true, force: true }); + } +} + +suite('deploy_component certifies a candidate before publishing it', (ctx: ContextWithHarper) => { + before(async () => { + // Certification is off by default — see `certificationEnabled` in `components/Application.ts` — so this + // suite asks for it. Without the switch the deploys below all succeed and prove nothing. + await startHarper(ctx, { env: { HARPER_CERTIFY_DEPLOYS: 'true' } }); + }); + + after(async () => { + await teardownHarper(ctx); + }); + + test('a candidate that throws at load is rejected, and the previous release keeps answering', async () => { + const componentsRoot = join(ctx.harper.dataRootDir, 'components'); + const livePath = join(componentsRoot, PROJECT); + + // v1: a component that loads and serves. + await operation(ctx, { + operation: 'deploy_component', + project: PROJECT, + payload: await buildPayload(1), + restart: true, + }); + strictEqual(await readFile(join(livePath, 'version.txt'), 'utf8'), '1', 'v1 is live'); + + // v2: installs cleanly, throws the moment it is loaded. + let rejection: string | undefined; + try { + await operation(ctx, { + operation: 'deploy_component', + project: PROJECT, + payload: await buildPayload(2, { throwsAtLoad: true }), + restart: true, + }); + } catch (error) { + rejection = error instanceof Error ? error.message : String(error); + } + + // Non-2xx. The helper does not surface the response body, so the candidate's own message is asserted + // in the unit test instead; what matters here is that the operation failed rather than reporting + // success over a component that cannot load. + ok(rejection, 'the deploy is reported as failed'); + + // The point, and what step 1's availability test explicitly could not show: v1 still ANSWERS, not + // merely still exists on disk. + strictEqual(await readVersion(ctx), 1, 'v1 still answers requests'); + strictEqual(await readFile(join(livePath, 'version.txt'), 'utf8'), '1', 'and is still the live tree'); + ok(!existsSync(join(componentsRoot, '.deploy-staging')), 'and the rejected candidate was swept'); + }); + + test('a candidate that loads cleanly is still published', async () => { + // The gate has to accept as well as refuse, or the previous assertion proves only that deploys break. + const livePath = join(ctx.harper.dataRootDir, 'components', PROJECT); + await operation(ctx, { + operation: 'deploy_component', + project: PROJECT, + payload: await buildPayload(3), + restart: true, + }); + strictEqual(await readFile(join(livePath, 'version.txt'), 'utf8'), '3', 'v3 is published'); + strictEqual(await readVersion(ctx), 3, 'and answers requests'); + }); + + test('the published tree does not carry the module link the certification load created', async () => { + // Certifying LOADS the candidate, and every non-root load links the running install into the + // component's `node_modules/harper` so `import 'harper'` resolves to the live instance. That link is + // not part of what the deploy staged, and the candidate tree is renamed into the live path — so + // without cleanup every walk of the component follows it into the whole Harper install. + // `package_component` did exactly that and packaged the install: 46s of tarring and then + // "Maximum response size reached". + // + // Deployed WITHOUT a restart deliberately: a serving worker legitimately recreates the link the next + // time it loads the component, so the invariant is about what the DEPLOY leaves behind. + const livePath = join(ctx.harper.dataRootDir, 'components', PROJECT); + await operation(ctx, { operation: 'deploy_component', project: PROJECT, payload: await buildPayload(4) }); + strictEqual(await readFile(join(livePath, 'version.txt'), 'utf8'), '4', 'v4 is published'); + ok(!existsSync(join(livePath, 'node_modules', 'harper')), 'no link to the Harper install was left behind'); + + // And the user-visible consequence: packaging sees the component, not the install behind the link. + const estimate = await operation(ctx, { operation: 'package_component', project: PROJECT, estimate: true }); + ok( + estimate.total_size < 1_000_000, + `packaging walks only the component (got ${estimate.total_size} bytes; the install is orders of magnitude larger)` + ); + }); +}); diff --git a/server/loadRootComponents.js b/server/loadRootComponents.js index e6773dd7ab..4be43e33ff 100644 --- a/server/loadRootComponents.js +++ b/server/loadRootComponents.js @@ -63,25 +63,42 @@ async function loadRootComponents(isWorkerThread = false) { console.error(errorForLog(error)); } - let resources = resetResources(); + const resources = await loadRootPlugins(isWorkerThread); + if (!process.env.HARPER_SAFE_MODE) { + // once the global plugins are loaded, we now load all the CF and run applications (and their components) + const readyComponentPromises = new WeakMap(); + await loadComponentDirectories(loadedComponents, resources, readyComponentPromises, interruptedActivationFailures); + await readyComponentModules(loadedComponents.keys(), readyComponentPromises); + return; + } + await readyComponentModules(loadedComponents.keys()); +} + +/** + * Load the Harper root component — the global plugins — and nothing that depends on them. + * + * This is the boundary the deploy certification validator needs. Plugins are what establish the surfaces + * applications extend (`server.mqtt.authorizeClient` and friends), so a certification load without them + * fails a component that works perfectly on a serving worker: the plugin surface is simply absent. But the + * validator must NOT go on to load the other applications, which is exactly where this function stops. + * + * Extracted rather than duplicated so the two callers cannot drift on what "the plugins are loaded" means. + * Listeners are not bound here: plugins register handlers on the scope's server object, and the port + * binding belongs to `threadServer`'s startup, which a validator suppresses with `workerData.noServerStart`. + */ +async function loadRootPlugins(isWorkerThread = false) { + const resources = resetResources(); getTables(); resources.isWorker = isWorkerThread; await loadCertificates(); - // the Harper root component await loadComponent(dirname(configUtils.getConfigFilePath()), resources, 'hdb', { isRoot: true, providedLoadedComponents: loadedComponents, autoReload: false, }); - if (!process.env.HARPER_SAFE_MODE) { - // once the global plugins are loaded, we now load all the CF and run applications (and their components) - const readyComponentPromises = new WeakMap(); - await loadComponentDirectories(loadedComponents, resources, readyComponentPromises, interruptedActivationFailures); - await readyComponentModules(loadedComponents.keys(), readyComponentPromises); - return; - } - await readyComponentModules(loadedComponents.keys()); + return resources; } module.exports.loadRootComponents = loadRootComponents; +module.exports.loadRootPlugins = loadRootPlugins; diff --git a/server/threads/manageThreads.js b/server/threads/manageThreads.js index d16befaee3..016c363277 100644 --- a/server/threads/manageThreads.js +++ b/server/threads/manageThreads.js @@ -142,6 +142,9 @@ const listenersByType = new Map(); const messagesQueuedByType = new Map(); module.exports = { + buildWorkerExecArgv, + buildWorkerResourceLimits, + isProcessShuttingDown, startWorker, restartWorkers, shutdownWorkers, @@ -338,12 +341,131 @@ listenersByType.set(hdbTerms.ITC_EVENT_TYPES.OPERATION_EXECUTE_RESPONSE, null); listenersByType.set(THREAD_INFO, null); listenersByType.set(PROCESS_GROUP_TERMINATION_CONFIRMED, null); +/** + * The interpreter flags and preloads every Harper worker needs to load Harper's own module graph. + * + * Exported because a worker that must NOT join the serving topology still needs exactly these: a bare + * `new Worker()` cannot even load a module that imports JSON. Shared rather than copied so the two spawn + * paths cannot drift on something this load-bearing. + * + * `preloads: false` omits the configured APM agents. Two reasons, and the second is the load-bearing one: + * a throwaway certification thread is not something an APM should see one of per deploy; and + * `getImportModules()`/`getRequireModules()` MEMOIZE on first call, so resolving them from a spawn that + * happens earlier than a serving worker would freeze the list against whatever config was live then. Before + * certification existed, nothing resolved them until the first real worker started. + */ +/** Whether the process is tearing down, so no new worker of any kind should be started. */ +function isProcessShuttingDown() { + return processShuttingDown; +} + +/** + * The heap bounds every Harper worker runs under. Shared with the non-topology spawn path so a validator + * thread is constrained like a serving one — an unbounded module graph in a candidate's top-level load + * would otherwise be the OOM killer's problem, taken out on the whole process. + */ +function buildWorkerResourceLimits(threadCount) { + let availableMemory = process.constrainedMemory?.() || totalmem(); + availableMemory = Math.min(availableMemory, totalmem(), 20000 * MB); + const maxOldMemory = + resolveThreadHeapMemoryMb(envMgr.get(hdbTerms.CONFIG_PARAMS.THREADS_MAXHEAPMEMORY)) ?? + Math.max(Math.floor(availableMemory / MB / (10 + (threadCount || 1) / 4)), 512); + return { + maxOldGenerationSizeMb: maxOldMemory, + maxYoungGenerationSizeMb: Math.min(Math.max(maxOldMemory >> 6, 16), 64), + }; +} + +function buildWorkerExecArgv({ preloads = true } = {}) { + const isBun = typeof globalThis.Bun !== 'undefined'; + const execArgv = isBun + ? [] + : [ + '--enable-source-maps', + '--experimental-vm-modules', // used for giving applications their own top level scope + '--disable-warning=ExperimentalWarning', // yeah, yeah, we know it is experimental + '--expose-internals', // expose Node.js internal utils so jsLoader can use `decorateErrorStack()` + ]; + if (!isBun && envMgr.get(hdbTerms.CONFIG_PARAMS.THREADS_HEAPSNAPSHOTNEARLIMIT)) + execArgv.push('--heapsnapshot-near-heap-limit=1'); + // Preload configured modules (e.g. an APM agent like dd-trace) before the worker's entry + // script so they can instrument all subsequent Harper and app module loads. Resolved once + // (config and installed components are fixed for the process lifetime). `threads.preload` + // uses --import (ESM/loader-hook registration, e.g. dd-trace/register.js — the entry that + // instruments worker threads); `threads.preloadRequire` uses --require for CJS agents that + // document that path (e.g. dd-trace/init, Dynatrace OneAgent). --import is URL-based, so + // resolved paths are passed as file URLs. Not supported under Bun, which does not use + // execArgv here. Safe mode also omits preloads because they are configured code, + // which safe mode must not resolve or execute. + const isSafeMode = + process.env.HARPER_SAFE_MODE && process.env.HARPER_SAFE_MODE !== 'false' && process.env.HARPER_SAFE_MODE !== '0'; + if (!isBun && !isSafeMode && preloads) { + for (const importPath of getImportModules()) execArgv.push('--import', pathToFileURL(importPath).href); + for (const requirePath of getRequireModules()) execArgv.push('--require', requirePath); + } + return execArgv; +} + function startWorker(path, options = {}) { if (processShuttingDown) { const error = new Error('Cannot start a worker while the Harper process is shutting down'); error.code = 'ERR_HARPER_PROCESS_SHUTTING_DOWN'; throw error; } + // Validated BEFORE anything with a side effect. `buildWorkerExecArgv` resolves the configured preload + // list through `getImportModules()`, which MEMOIZES — so rejecting bad input after that point would + // freeze the list for the whole process on a spawn that never happened. + // A caller's own workerData, merged rather than substituted. `...options` is spread into the + // constructor LAST, so an `options.workerData` would replace the object below wholesale and take + // `addPorts`/`addThreadIds` with it — the worker would come up with no ITC wiring at all, and + // `workerDataProviders` cannot carry a `MessagePort` because it `structuredClone`s. Callers that need + // both use `extraWorkerData`/`extraTransferList`; reserved keys are refused here, before the spawn, + // so a collision is an error rather than a worker missing the bootstrap it did not know it lost. + const { + extraWorkerData, + extraTransferList, + execArgvOptions: _execArgvOptions, + noServerStart: _noServerStart, + ...workerOptions + } = options; + // `in`, not truthiness: `workerData: null` is present, replaces the bootstrap object, and would sail + // past a truthy check leaving the thread with no ITC wiring and no error. + if ('workerData' in workerOptions) { + throw new Error( + `startWorker does not accept 'workerData' — it would replace the thread's ITC bootstrap. Use 'extraWorkerData'` + ); + } + // Same hazard, other half: a raw `transferList` is spread last and REPLACES the merged list, dropping + // `portsToSend` so the thread comes up with no ports to its peers. + if ('transferList' in workerOptions) { + throw new Error( + `startWorker does not accept 'transferList' — it would drop the thread's ITC ports. Use 'extraTransferList'` + ); + } + if (extraWorkerData) { + for (const key of Object.keys(extraWorkerData)) { + if (RESERVED_WORKER_DATA_KEYS.includes(key)) { + throw new Error(`extraWorkerData may not set '${key}': it is owned by the thread bootstrap`); + } + // A registered provider owns its key too. `extraWorkerData` is spread AFTER provider output, so + // without this a caller could silently replace an authoritative value — `configOverrides` most + // consequentially, which would leave the thread reading on-disk config while its parent runs on + // `setProperty()` overrides. `registerWorkerDataProvider` already refuses name collisions; this is + // the same ownership rule on the other path into `workerData`. + if (workerDataProviders.has(key)) { + throw new Error(`extraWorkerData may not set '${key}': a registered workerData provider owns it`); + } + } + } + // A transferred port is single-use, and the unexpected-exit path below re-invokes `startWorker` with the + // SAME options object — so a restartable worker carrying transferred ports throws `DataCloneError` on + // its second spawn, synchronously, inside an `exit` listener with no handler around it. That does not + // fail one deploy; it takes the process down. A caller transferring ports must own the thread's lifetime. + if (extraTransferList?.length && options.autoRestart !== false) { + throw new Error( + `startWorker with 'extraTransferList' requires 'autoRestart: false': transferred ports cannot be reused by a restart` + ); + } // Take a percentage of total memory to determine the max memory for each thread. The percentage is based // on the thread count. Generally, it is unrealistic to efficiently use the majority of total memory for a single // NodeJS worker since it would lead to massive swap space usage with other processes and there is significant @@ -354,18 +476,12 @@ function startWorker(path, options = {}) { // 16 threads: 20% of total memory per thread // 64 threads: 11% of total memory per thread // (and then limit to their license limit, if they have one) - let availableMemory = process.constrainedMemory?.() || totalmem(); // used constrained memory if it is available - // and lower than total memory - availableMemory = Math.min(availableMemory, totalmem(), 20000 * MB); - const maxOldMemory = - resolveThreadHeapMemoryMb(envMgr.get(hdbTerms.CONFIG_PARAMS.THREADS_MAXHEAPMEMORY)) ?? - Math.max(Math.floor(availableMemory / MB / (10 + (options.threadCount || 1) / 4)), 512); // Max young memory space (semi-space for scavenger) is 1/128 of max memory (limited to 16-64). For most of our m5 // machines this will be 64MB (less for t3's). This is based on recommendations from: // https://www.alibabacloud.com/blog/node-js-application-troubleshooting-manual---comprehensive-gc-problems-and-optimization594965 // https://github.com/nodejs/node/issues/42511 // https://plaid.com/blog/how-we-parallelized-our-node-service-by-30x/ - const maxYoungMemory = Math.min(Math.max(maxOldMemory >> 6, 16), 64); + const resourceLimits = buildWorkerResourceLimits(options.threadCount); const channelsToConnect = []; const portsToSend = []; @@ -378,43 +494,16 @@ function startWorker(path, options = {}) { if (!extname(path)) path += '.js'; - const isBun = typeof globalThis.Bun !== 'undefined'; - const execArgv = isBun - ? [] - : [ - '--enable-source-maps', - '--experimental-vm-modules', // used for giving applications their own top level scope - '--disable-warning=ExperimentalWarning', // yeah, yeah, we know it is experimental - '--expose-internals', // expose Node.js internal utils so jsLoader can use `decorateErrorStack()` - ]; - if (!isBun && envMgr.get(hdbTerms.CONFIG_PARAMS.THREADS_HEAPSNAPSHOTNEARLIMIT)) - execArgv.push('--heapsnapshot-near-heap-limit=1'); - // Preload configured modules (e.g. an APM agent like dd-trace) before the worker's entry - // script so they can instrument all subsequent Harper and app module loads. Resolved once - // (config and installed components are fixed for the process lifetime). `threads.preload` - // uses --import (ESM/loader-hook registration, e.g. dd-trace/register.js — the entry that - // instruments worker threads); `threads.preloadRequire` uses --require for CJS agents that - // document that path (e.g. dd-trace/init, Dynatrace OneAgent). --import is URL-based, so - // resolved paths are passed as file URLs. Not supported under Bun, which does not use - // execArgv here. Safe mode also omits preloads because they are configured code, - // which safe mode must not resolve or execute. - const isSafeMode = - process.env.HARPER_SAFE_MODE && process.env.HARPER_SAFE_MODE !== 'false' && process.env.HARPER_SAFE_MODE !== '0'; - if (!isBun && !isSafeMode) { - for (const importPath of getImportModules()) execArgv.push('--import', pathToFileURL(importPath).href); - for (const requirePath of getRequireModules()) execArgv.push('--require', requirePath); - } + const execArgv = buildWorkerExecArgv(options.execArgvOptions); const worker = new Worker(isAbsolute(path) ? path : join(PACKAGE_ROOT, path), { - resourceLimits: { - maxOldGenerationSizeMb: maxOldMemory, - maxYoungGenerationSizeMb: maxYoungMemory, - }, + resourceLimits, execArgv, argv: process.argv.slice(2), // pass these in synchronously to the worker so it has them on startup: workerData: { ...collectProvidedWorkerData(options), + ...extraWorkerData, addPorts: portsToSend, addThreadIds: channelsToConnect.map((channel) => channel.existingPort.threadId), addPortIsJobWorkers: channelsToConnect.map((channel) => channel.existingPort.isJobWorker === true), @@ -423,9 +512,13 @@ function startWorker(path, options = {}) { name: options.name, restartNumber: module.exports.restartNumber, ticketKeys: getTicketKeys(), + // `noServerStart` is a RESERVED key, so a provider or `extraWorkerData` cannot supply it — but a + // thread that must not serve (a deploy validator) has to. Added only when asked for, so the + // default spawn's workerData is byte-identical to before. + ...(options.noServerStart ? { noServerStart: true } : undefined), }, - transferList: portsToSend, - ...options, + transferList: extraTransferList ? [...portsToSend, ...extraTransferList] : portsToSend, + ...workerOptions, }); // now that we have the new thread ids, we can finishing connecting the channel and notify the existing // worker of the new port with thread id. @@ -443,7 +536,15 @@ function startWorker(path, options = {}) { } addPort(worker, true, isJobWorker); worker.unexpectedRestarts = options.unexpectedRestarts || 0; + // A one-shot thread cannot be copied: its transferred ports are detached after the first spawn, so + // re-spawning from the same options throws `DataCloneError`. The restart loop skips these workers, so + // this is the backstop for a direct caller — a named error rather than a clone failure from deep inside + // `new Worker`. + worker.isOneShot = Boolean(extraTransferList?.length); worker.startCopy = () => { + if (worker.isOneShot) { + throw new Error(`Cannot restart a one-shot ${options.name ?? 'worker'}: its transferred ports are spent`); + } // in a shutdown sequence we use overlapping restarts, starting the new thread while waiting for the old thread // to die, to ensure there is no loss of service and maximum availability. return startWorker(path, options); @@ -561,7 +662,11 @@ async function restartWorkers( const worker = restarting[index]; // Terminal shutdown: stop replacing workers mid-loop — the guard for every replacement start below. if (processShuttingDown && startReplacementThreads) break; - if ((name && worker.name !== name) || worker.wasShutdown) continue; // filter by type, if specified + // One-shot threads are ephemeral by contract (transferred ports force `autoRestart: false`), so they + // are never restarted OR replaced: `startCopy` would re-spawn from spent ports and throw + // `DataCloneError` synchronously, inside this loop, taking the rest of the restart with it. Left + // alone, such a thread finishes its single task or hits its own deadline. + if ((name && worker.name !== name) || worker.wasShutdown || worker.isOneShot) continue; // filter by type, if specified const overlapping = OVERLAPPING_RESTART_TYPES.indexOf(worker.name) > -1; if (overlapping && startReplacementThreads && canPreStartReplacement) { // Overlapping restart: start the replacement and wait until it is accepting connections diff --git a/unitTests/components/deployActivation.test.js b/unitTests/components/deployActivation.test.js index a6b55251f2..4fa1b58f2a 100644 --- a/unitTests/components/deployActivation.test.js +++ b/unitTests/components/deployActivation.test.js @@ -40,7 +40,10 @@ async function stageState(root, component, id, state) { const deploymentDir = path.join(root, DEPLOY_STAGING_DIR, id); await fs.mkdir(deploymentDir, { recursive: true }); if (state.candidate) await writeTree(path.join(deploymentDir, component), state.candidate); - if (state.complete) await fs.writeFile(path.join(deploymentDir, '.complete'), ''); + // The marker's CONTENT is the authority now: an empty one predates certification and was minted after a + // validation that did nothing on the main thread, so recovery treats it as uncertified. + if (state.complete) await fs.writeFile(path.join(deploymentDir, '.complete'), 'certified:1'); + if (state.legacyComplete) await fs.writeFile(path.join(deploymentDir, '.complete'), ''); if (state.componentFile !== false) await fs.writeFile(path.join(deploymentDir, '.component'), component); if (state.journal !== undefined) { await fs.writeFile( @@ -869,4 +872,16 @@ describe('activation transaction', () => { assert.strictEqual(await readLive(root, 'web'), 'CURRENT\n', 'the live component is untouched'); await fs.rm(root, { recursive: true, force: true }); }); + + it('rolls back a candidate whose completion marker predates certification', async () => { + // An empty `.complete` is what an older build wrote, after a validation that was a no-op on the main + // thread. Found after an upgrade it must NOT be roll-forward authority: nothing certified that tree. + const root = await newRoot('legacy-marker'); + await stageState(root, 'web', 'd1', { candidate: 'NEW\n', legacyComplete: true, journal: true, aside: 'OLD\n' }); + + const failures = await recoverInterruptedActivations(root); + + assert.strictEqual(failures.size, 0, 'it settles rather than failing closed'); + assert.strictEqual(await readLive(root, 'web'), 'OLD\n', 'the committed tree comes back, not the candidate'); + }); }); diff --git a/unitTests/components/deployCandidateBuild.test.js b/unitTests/components/deployCandidateBuild.test.js index bd0a7b7950..1771f9f812 100644 --- a/unitTests/components/deployCandidateBuild.test.js +++ b/unitTests/components/deployCandidateBuild.test.js @@ -109,7 +109,7 @@ describe('deploy candidate builds', () => { // chmod cannot make a file unreadable to root, so as root this would assert nothing at all. if (process.platform === 'win32' || process.getuid?.() === 0) return this.skip(); this.timeout(20000); - const { markCandidateComplete } = require('#src/components/Application'); + const { syncCandidateTree } = require('#src/components/Application'); const componentsRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'candidate-filelink-')); const dirPath = await makeLiveComponent(componentsRoot, 'web', 'LIVE v1\n'); // A `file:` candidate is a symlink into a developer's own tree, which can hold a file the Harper uid @@ -127,10 +127,10 @@ describe('deploy candidate builds', () => { await fs.mkdir(deploymentDir, { recursive: true }); await fs.symlink(source, path.join(deploymentDir, 'web'), 'dir'); - await markCandidateComplete(dirPath, 'dep-link', 'web'); - - assert.ok( - existsSync(path.join(deploymentDir, '.complete')), + // The durability step specifically, not the marker: `.complete` is gated on certification now, and a + // test seam that could write it without one would be a seam production could use too. + await assert.doesNotReject( + () => syncCandidateTree(dirPath, 'dep-link'), 'the deploy is not failed by a file it never wrote and cannot read' ); await fs.chmod(unreadable, 0o600); diff --git a/unitTests/components/deployCertification.test.js b/unitTests/components/deployCertification.test.js new file mode 100644 index 0000000000..e132d722f1 --- /dev/null +++ b/unitTests/components/deployCertification.test.js @@ -0,0 +1,384 @@ +'use strict'; + +const assert = require('node:assert'); +const { mkdir, mkdtemp, readFile, rm, writeFile } = require('node:fs/promises'); +const { existsSync } = require('node:fs'); +const { tmpdir } = require('node:os'); +const { join } = require('node:path'); + +const testUtils = require('../testUtils.js'); +testUtils.preTestPrep(); + +const { + Application, + prepareApplication, + markCandidateComplete, + CERTIFY_DEPLOYS_ENV, +} = require('#src/components/Application'); +const { certifyCandidate } = require('#src/components/certifyCandidate'); +const { packageDirectory } = require('#src/components/packageComponent'); +const { rootApplicationLoadOptions } = require('#src/components/componentLoader'); +const { getConfigObj } = require('#src/config/configUtils'); + +/** A component payload whose `resource.js` runs `body` when the component is loaded. */ +async function payloadThatRunsOnLoad(rootDir, name, version, body) { + const sourceDir = await mkdtemp(join(rootDir, `${name}-${version}-`)); + await writeFile(join(sourceDir, 'package.json'), JSON.stringify({ name, version, main: 'resource.js' })); + await writeFile(join(sourceDir, 'config.yaml'), 'jsResource:\n files: resource.js\n'); + await writeFile(join(sourceDir, 'resource.js'), body); + return packageDirectory(sourceDir, { skip_node_modules: true }); +} + +// Skipped on Windows: a validator thread dies inside its own import graph there — before its first +// statement, exit code 0, no `error` event — so every certification fails identically +// (HarperFast/harper#2494). The mechanism is off by default, so nothing ships broken; this suite opts in, +// which is why it is the thing that fails. Removing this skip is the acceptance test for that issue. +const SKIP_ON_WINDOWS = process.platform === 'win32'; + +describe('deploy certification', () => { + // Narrowed to the cases that actually SPAWN a validator. A suite-level skip also suppressed the + // default-off case, the mint gate, safe mode and the branch-configured options — none of which start a + // worker — and the default-off case is precisely the Windows coverage worth keeping, since it guards the + // behaviour Windows operators actually get. + function skipWithoutValidator(test) { + if (SKIP_ON_WINDOWS) test.skip(); + } + + // A separate precondition from the validator one, and stated as the dependency rather than as a platform: + // `getConfigObj()` is undefined in the Windows unit-test process, so a test that injects a root-config + // entry cannot run there. Naming the requirement means this stops skipping the day that changes, without + // anyone remembering to revisit a `win32` check. + function skipWithoutRootConfig(test) { + if (!getConfigObj()) test.skip(); + } + // Certification is OFF by default — see `certificationEnabled` for why — so every test that expects a + // verdict has to ask for one. Without this the suite would still pass while proving nothing: an + // uncertified deploy publishes, so the acceptance cases would go green and only the rejection case + // would notice. + let priorCertifyDeploys; + before(() => { + priorCertifyDeploys = process.env[CERTIFY_DEPLOYS_ENV]; + process.env[CERTIFY_DEPLOYS_ENV] = 'true'; + }); + after(() => { + if (priorCertifyDeploys === undefined) delete process.env[CERTIFY_DEPLOYS_ENV]; + else process.env[CERTIFY_DEPLOYS_ENV] = priorCertifyDeploys; + }); + + it('deploys without certifying when no validator host is enabled', async function () { + // The default state, and the one that would otherwise be untested: with certification off, a + // candidate that throws at load is published exactly as it was before this work, and earns no + // `.complete`. Asserting it here is what would catch the switch being flipped on by accident — + // and it is the inverse of the headline test below, which enables it. + this.timeout(30000); + const restore = process.env[CERTIFY_DEPLOYS_ENV]; + delete process.env[CERTIFY_DEPLOYS_ENV]; + const rootDir = await mkdtemp(join(tmpdir(), 'certify-off-')); + const componentDirPath = join(rootDir, 'shop'); + await mkdir(componentDirPath, { recursive: true }); + await writeFile(join(componentDirPath, 'package.json'), JSON.stringify({ name: 'shop', version: '1.0.0' })); + + const application = new Application({ + name: 'shop', + payload: await payloadThatRunsOnLoad(rootDir, 'shop', '2.0.0', "throw new Error('candidate blew up at load');\n"), + }); + application.dirPath = componentDirPath; + + try { + await prepareApplication(application); + assert.strictEqual( + JSON.parse(await readFile(join(componentDirPath, 'package.json'), 'utf8')).version, + '2.0.0', + 'an uncertified deploy publishes, which is the pre-certification behaviour' + ); + assert.ok( + !existsSync(join(componentDirPath, '.complete')), + 'and mints no authority, so a crash mid-swap rolls back rather than forward' + ); + } finally { + if (restore === undefined) delete process.env[CERTIFY_DEPLOYS_ENV]; + else process.env[CERTIFY_DEPLOYS_ENV] = restore; + await rm(rootDir, { recursive: true, force: true }); + } + }); + + it('does not publish a candidate that throws at load — ON THE MAIN THREAD', async function () { + skipWithoutValidator(this); + // The whole point of this step. The in-process check this replaces was gated on `!isMainThread`, and + // the operations API deploys on main, so this exact deploy used to succeed and publish a broken + // component while reporting an error. Tests run on the main thread, so this is that path. + this.timeout(30000); + const rootDir = await mkdtemp(join(tmpdir(), 'certify-rejects-')); + const componentDirPath = join(rootDir, 'shop'); + await mkdir(componentDirPath, { recursive: true }); + await writeFile(join(componentDirPath, 'package.json'), JSON.stringify({ name: 'shop', version: '1.0.0' })); + await writeFile(join(componentDirPath, 'index.js'), 'module.exports = { live: 1 };\n'); + + const application = new Application({ + name: 'shop', + payload: await payloadThatRunsOnLoad(rootDir, 'shop', '2.0.0', "throw new Error('candidate blew up at load');\n"), + }); + application.dirPath = componentDirPath; + + try { + await assert.rejects(() => prepareApplication(application), /candidate blew up at load/); + // v1 is still the live tree, byte for byte. + assert.strictEqual( + JSON.parse(await readFile(join(componentDirPath, 'package.json'), 'utf8')).version, + '1.0.0', + 'the previous version still serves' + ); + assert.strictEqual(await readFile(join(componentDirPath, 'index.js'), 'utf8'), 'module.exports = { live: 1 };\n'); + // And nothing was left claiming the rejected candidate was validated. + assert.ok(!existsSync(join(rootDir, '.deploy-staging')), 'the rejected candidate was swept'); + } finally { + await rm(rootDir, { recursive: true, force: true }); + } + }); + + it('publishes a candidate that loads cleanly', async function () { + skipWithoutValidator(this); + this.timeout(30000); + const rootDir = await mkdtemp(join(tmpdir(), 'certify-accepts-')); + const componentDirPath = join(rootDir, 'shop'); + await mkdir(componentDirPath, { recursive: true }); + await writeFile(join(componentDirPath, 'package.json'), JSON.stringify({ name: 'shop', version: '1.0.0' })); + + const application = new Application({ + name: 'shop', + payload: await payloadThatRunsOnLoad(rootDir, 'shop', '2.0.0', 'module.exports = { fine: true };\n'), + }); + application.dirPath = componentDirPath; + + try { + await prepareApplication(application); + assert.strictEqual( + JSON.parse(await readFile(join(componentDirPath, 'package.json'), 'utf8')).version, + '2.0.0', + 'a certified candidate is published' + ); + } finally { + await rm(rootDir, { recursive: true, force: true }); + } + }); + + it('publishes a static-only component, which legitimately loads no module', async function () { + skipWithoutValidator(this); + // The "loaded nothing is not a pass" guard fires on a candidate that declares loadable content and + // then opens no scope and loads no module. A static-only component declares content (every component + // ships a `package.json`) and loads no JS module, so whether the guard rejects it comes down to + // whether a static load still opens a scope. It does — this test is what pins that, so a change to + // scope creation cannot silently turn valid static deploys into certification failures. + this.timeout(30000); + const rootDir = await mkdtemp(join(tmpdir(), 'certify-static-')); + const componentDirPath = join(rootDir, 'brochure'); + await mkdir(componentDirPath, { recursive: true }); + await writeFile(join(componentDirPath, 'package.json'), JSON.stringify({ name: 'brochure', version: '1.0.0' })); + + const sourceDir = await mkdtemp(join(rootDir, 'brochure-2.0.0-')); + await writeFile(join(sourceDir, 'package.json'), JSON.stringify({ name: 'brochure', version: '2.0.0' })); + await writeFile(join(sourceDir, 'config.yaml'), "static:\n files: 'web/**'\n"); + await mkdir(join(sourceDir, 'web'), { recursive: true }); + await writeFile(join(sourceDir, 'web', 'index.html'), 'hi\n'); + + const application = new Application({ + name: 'brochure', + payload: await packageDirectory(sourceDir, { skip_node_modules: true }), + }); + application.dirPath = componentDirPath; + + try { + await prepareApplication(application); + assert.strictEqual( + JSON.parse(await readFile(join(componentDirPath, 'package.json'), 'utf8')).version, + '2.0.0', + 'a static-only component is published rather than rejected for loading nothing' + ); + } finally { + await rm(rootDir, { recursive: true, force: true }); + } + }); + + it('rejects a candidate whose load never finishes, rather than waiting on it', async function () { + skipWithoutValidator(this); + this.timeout(30000); + const rootDir = await mkdtemp(join(tmpdir(), 'certify-timeout-')); + const candidateDirPath = join(rootDir, 'hangs'); + await mkdir(candidateDirPath, { recursive: true }); + await writeFile(join(candidateDirPath, 'package.json'), JSON.stringify({ name: 'hangs', version: '1.0.0' })); + await writeFile(join(candidateDirPath, 'config.yaml'), 'jsResource:\n files: resource.js\n'); + // Blocks the validator thread outright — `Atomics.wait` rather than a spin loop, so it holds without + // burning CPU. The in-process check had no answer for a load that never returns: it held the + // validation chain for the life of the process. An isolated thread can simply be given a deadline. + await writeFile( + join(candidateDirPath, 'resource.js'), + 'const shared = new Int32Array(new SharedArrayBuffer(4));\nAtomics.wait(shared, 0, 0);\n' + ); + + try { + const outcome = await certifyCandidate(candidateDirPath, 'hangs', { timeoutMs: 2000 }); + assert.strictEqual(outcome.certified, false); + assert.match(outcome.error.message, /did not finish within 2000ms/); + } finally { + await rm(rootDir, { recursive: true, force: true }); + } + }); + + it('fails a deploy that cannot get a certification slot, rather than queueing it forever', async function () { + skipWithoutValidator(this); + // The concurrency cap deliberately withholds the slot of a validator that will not die, so the queue + // behind it has to be bounded too — otherwise one stuck thread turns every later deploy into a wait + // inside the preparation lock with nothing to report. + this.timeout(30000); + const rootDir = await mkdtemp(join(tmpdir(), 'certify-slots-')); + const candidateDirPath = join(rootDir, 'hangs'); + await mkdir(candidateDirPath, { recursive: true }); + await writeFile(join(candidateDirPath, 'package.json'), JSON.stringify({ name: 'hangs', version: '1.0.0' })); + await writeFile(join(candidateDirPath, 'config.yaml'), 'jsResource:\n files: resource.js\n'); + await writeFile( + join(candidateDirPath, 'resource.js'), + 'const shared = new Int32Array(new SharedArrayBuffer(4));\nAtomics.wait(shared, 0, 0);\n' + ); + + try { + // `acquireSlot` claims synchronously when a slot is free — no await before `active++` — so both of + // these hold slots by the time the third call runs, without sleeping to arrange it. + const occupying = [ + certifyCandidate(candidateDirPath, 'first', { timeoutMs: 4000 }), + certifyCandidate(candidateDirPath, 'second', { timeoutMs: 4000 }), + ]; + const queued = await certifyCandidate(candidateDirPath, 'third', { timeoutMs: 300 }); + assert.strictEqual(queued.certified, false); + assert.match(queued.error.message, /No certification slot became available within 300ms/); + assert.strictEqual(queued.error.statusCode, 503); + for (const outcome of await Promise.all(occupying)) assert.strictEqual(outcome.certified, false); + } finally { + await rm(rootDir, { recursive: true, force: true }); + } + }); + + it('refuses to mint .complete for a candidate no validator certified', async function () { + // The gate itself. `.complete` is what recovery treats as proof a validation happened, so the + // function that writes it has to require the verdict rather than trust its caller — three of + // `prepareApplication`'s four call sites never asked for one. + const rootDir = await mkdtemp(join(tmpdir(), 'certify-gate-')); + const componentDirPath = join(rootDir, 'shop'); + await mkdir(join(rootDir, '.deploy-staging', 'd1', 'shop'), { recursive: true }); + try { + await assert.rejects( + () => markCandidateComplete(componentDirPath, 'd1', 'shop'), + /no validator has certified it/ + ); + assert.ok(!existsSync(join(rootDir, '.deploy-staging', 'd1', '.complete')), 'and no authority was written'); + } finally { + await rm(rootDir, { recursive: true, force: true }); + } + }); + + it('deploys in safe mode without certifying, rather than staging a candidate nothing resumes', async function () { + // Safe mode may not execute configured code, so it certifies nothing — and it deploys anyway, minting + // no `.complete`. + // + // An earlier draft staged WITHOUT activating, reasoning that safe mode is transient so the candidate + // could wait. Nothing resumes it: the staged tree carries no journal, so recovery removes it as build + // residue at the next start — while the operation had already reported success, replicated, and run + // its restart phase. An operator booting into safe mode to replace the component crashing the node + // would have got a 200 and a node that came back running the broken component with the fix deleted. + this.timeout(30000); + const rootDir = await mkdtemp(join(tmpdir(), 'certify-safe-mode-')); + const componentDirPath = join(rootDir, 'shop'); + await mkdir(componentDirPath, { recursive: true }); + await writeFile(join(componentDirPath, 'package.json'), JSON.stringify({ name: 'shop', version: '1.0.0' })); + + const application = new Application({ + name: 'shop', + payload: await payloadThatRunsOnLoad(rootDir, 'shop', '2.0.0', 'module.exports = { fine: true };\n'), + }); + application.dirPath = componentDirPath; + + const priorSafeMode = process.env.HARPER_SAFE_MODE; + process.env.HARPER_SAFE_MODE = '1'; + try { + await prepareApplication(application); + assert.strictEqual( + JSON.parse(await readFile(join(componentDirPath, 'package.json'), 'utf8')).version, + '2.0.0', + 'the deploy takes effect — safe mode is when an operator most needs it to' + ); + } finally { + if (priorSafeMode === undefined) delete process.env.HARPER_SAFE_MODE; + else process.env.HARPER_SAFE_MODE = priorSafeMode; + await rm(rootDir, { recursive: true, force: true }); + } + }); + + describe('branch-configured components', () => { + // A branch's location is derived only from the application and database names, so a certification + // load that resolved branches the way boot does would open the store the LIVE version is serving + // from — a candidate could mutate rows, throw, be rejected, and leave the live version serving the + // mutation. So certification never applies them, and the caller is told to skip certifying instead. + const appName = 'branch_certify_probe'; + + afterEach(() => { + // `getConfigObj()` is undefined in the Windows unit-test process, and mocha runs this hook even for + // tests skipped in `beforeEach` — so an unguarded delete fails the hook rather than the test. + const config = getConfigObj(); + if (config) delete config[appName]; + }); + + it('reports the component as branch-configured and withholds the branch settings', function () { + skipWithoutRootConfig(this); + getConfigObj()[appName] = { branchedDatabases: ['data'] }; + + const forBoot = rootApplicationLoadOptions(appName); + const forCertification = rootApplicationLoadOptions(appName, { forCertification: true }); + + assert.deepStrictEqual(forBoot.options.branchedDatabases, ['data'], 'boot still gets its branches'); + assert.ok( + !('branchedDatabases' in forCertification.options), + 'certification is never handed the live branch settings' + ); + assert.strictEqual(forCertification.branchConfigured, true, 'and the caller is told to skip certifying'); + }); + + it('still deploys a branch-configured component, it just earns no authority', async function () { + skipWithoutRootConfig(this); + // The half of this decision that is observable end to end: nothing is refused. That certification + // is skipped is covered by the option test above, and that an unminted candidate rolls back + // rather than forward is covered by the recovery suite — the composition of the two (activate, + // mint nothing, then crash) is not covered here, and needs a crash the unit harness cannot stage. + this.timeout(30000); + const rootDir = await mkdtemp(join(tmpdir(), 'certify-branch-deploy-')); + const componentDirPath = join(rootDir, appName); + await mkdir(componentDirPath, { recursive: true }); + await writeFile(join(componentDirPath, 'package.json'), JSON.stringify({ name: appName, version: '1.0.0' })); + getConfigObj()[appName] = { branchedDatabases: ['data'] }; + + const application = new Application({ + name: appName, + payload: await payloadThatRunsOnLoad(rootDir, appName, '2.0.0', 'module.exports = { fine: true };\n'), + }); + application.dirPath = componentDirPath; + + try { + await prepareApplication(application); + assert.strictEqual( + JSON.parse(await readFile(join(componentDirPath, 'package.json'), 'utf8')).version, + '2.0.0', + 'a branch-configured component still deploys — no capability is removed' + ); + } finally { + await rm(rootDir, { recursive: true, force: true }); + } + }); + + it('reports an ordinary component as not branch-configured', function () { + skipWithoutRootConfig(this); + getConfigObj()[appName] = { package: 'npm:whatever@1.0.0' }; + + const forCertification = rootApplicationLoadOptions(appName, { forCertification: true }); + + assert.strictEqual(forCertification.branchConfigured, false, 'so it is certified normally'); + }); + }); +}); diff --git a/unitTests/server/threads/workerDataExtra-fixture.js b/unitTests/server/threads/workerDataExtra-fixture.js new file mode 100644 index 0000000000..00c17a7091 --- /dev/null +++ b/unitTests/server/threads/workerDataExtra-fixture.js @@ -0,0 +1,20 @@ +'use strict'; + +// Fixture for workerDataExtra.test.js: report what a per-call `extraWorkerData` spawn actually +// received, including whether the thread's own ITC bootstrap survived the merge. +const { parentPort, workerData } = require('node:worker_threads'); + +const port = workerData.certification?.verdictPort; +if (port) port.postMessage({ type: 'through-transferred-port', nonce: workerData.certification.nonce }); + +parentPort.postMessage({ + type: 'extra-report', + candidateDirPath: workerData.certification?.candidateDirPath, + nonce: workerData.certification?.nonce, + sawPort: Boolean(port), + noServerStart: workerData.noServerStart, + // The bootstrap the merge must not displace. + hasAddPorts: Array.isArray(workerData.addPorts), + hasTicketKeys: Boolean(workerData.ticketKeys), + name: workerData.name, +}); diff --git a/unitTests/server/threads/workerDataExtra.test.js b/unitTests/server/threads/workerDataExtra.test.js new file mode 100644 index 0000000000..521b704f3b --- /dev/null +++ b/unitTests/server/threads/workerDataExtra.test.js @@ -0,0 +1,167 @@ +'use strict'; + +const assert = require('assert'); +const path = require('node:path'); +const { MessageChannel } = require('node:worker_threads'); +const { startWorker, registerWorkerDataProvider } = require('#js/server/threads/manageThreads'); + +const FIXTURE = path.join(__dirname, 'workerDataExtra-fixture.js'); +const WORKER_NAME = 'workerData-extra-test'; + +/** Spawn the fixture and resolve its report. */ +function spawnAndReport(options) { + return new Promise((resolve, reject) => { + // `preloads: false` is not incidental: `getImportModules()` memoizes, so a spawn from a test that + // resolves the configured preload list freezes it for the whole process and breaks + // preloadSafeMode.test.js when they share one mocha run. This is the same hazard the deploy + // validator avoids, which is why the option is threaded through startWorker at all. + const worker = startWorker(FIXTURE, { + autoRestart: false, + name: WORKER_NAME, + execArgvOptions: { preloads: false }, + ...options, + }); + const timer = setTimeout(() => reject(new Error('fixture never reported')), 20000); + worker.on('message', (message) => { + if (message?.type !== 'extra-report') return; + clearTimeout(timer); + resolve(message); + worker.terminate().catch(() => {}); + }); + worker.on('error', (error) => { + clearTimeout(timer); + reject(error); + }); + }); +} + +describe('startWorker per-call workerData', () => { + it('refuses input that would silently cost the thread its ITC bootstrap', () => { + // `...options` is spread into the constructor last, so this would REPLACE the bootstrap object + // wholesale — the worker would come up with no addPorts and no way to reach its peers. + assert.throws( + () => startWorker(FIXTURE, { name: WORKER_NAME, workerData: { mine: 1 } }), + /does not accept 'workerData'/ + ); + // `workerData: null` is present but falsy: it still replaces the bootstrap object, so a truthiness + // check would let it through and the thread would come up with no ITC wiring and no error. + assert.throws(() => startWorker(FIXTURE, { name: WORKER_NAME, workerData: null }), /does not accept 'workerData'/); + // The other half of the same hazard: a raw transferList is spread last and replaces the merged list, + // dropping the ports to this thread's peers. + assert.throws( + () => startWorker(FIXTURE, { name: WORKER_NAME, transferList: [] }), + /does not accept 'transferList'/ + ); + // A transferred port is single-use, and the unexpected-exit path re-invokes startWorker with the SAME + // options — so a restartable worker carrying one throws DataCloneError on its second spawn, + // synchronously, inside an `exit` listener with nothing to catch it. That is a process-wide crash, + // not a failed spawn, so the contract is refused up front. + const { port1, port2 } = new MessageChannel(); + try { + assert.throws( + () => startWorker(FIXTURE, { name: WORKER_NAME, extraTransferList: [port2] }), + /requires 'autoRestart: false'/ + ); + } finally { + port1.close(); + port2.close(); + } + // A registered provider owns its key on this path too. `extraWorkerData` is spread after provider + // output, so without this a caller could quietly replace `configOverrides` and leave the thread + // reading on-disk config while its parent runs on `setProperty()` overrides. + assert.throws( + () => startWorker(FIXTURE, { name: WORKER_NAME, extraWorkerData: { configOverrides: { 'a.b': 1 } } }), + /a registered workerData provider owns it/ + ); + const unregister = registerWorkerDataProvider('extraCollisionProbe', () => undefined); + try { + assert.throws( + () => startWorker(FIXTURE, { name: WORKER_NAME, extraWorkerData: { extraCollisionProbe: 1 } }), + /a registered workerData provider owns it/ + ); + } finally { + unregister(); + } + // ...and once the provider is gone, the name is free again. + assert.doesNotThrow(() => + startWorker(FIXTURE, { + autoRestart: false, + name: WORKER_NAME, + execArgvOptions: { preloads: false }, + extraWorkerData: { extraCollisionProbe: 1 }, + }).terminate() + ); + for (const key of ['addPorts', 'ticketKeys', 'workerCount', 'noServerStart', '__proto__']) { + assert.throws( + () => startWorker(FIXTURE, { name: WORKER_NAME, extraWorkerData: { [key]: 'x' } }), + /is owned by the thread bootstrap/, + `${key} must be refused` + ); + } + }); + + it('merges extraWorkerData and a transferred port alongside the bootstrap, not instead of it', async function () { + this.timeout(30000); + const { port1, port2 } = new MessageChannel(); + const throughPort = new Promise((resolve) => port1.once('message', resolve)); + const report = await spawnAndReport({ + noServerStart: true, + extraWorkerData: { certification: { candidateDirPath: '/tmp/candidate', nonce: 'n-1', verdictPort: port2 } }, + extraTransferList: [port2], + }); + + assert.strictEqual(report.candidateDirPath, '/tmp/candidate'); + assert.strictEqual(report.nonce, 'n-1'); + assert.strictEqual(report.sawPort, true, 'the MessagePort survived the transfer'); + // `noServerStart` is a reserved key precisely so a provider cannot forge it; the bootstrap + // supplies it on request instead. + assert.strictEqual(report.noServerStart, true); + // The point of the merge: the thread still got everything it normally gets. + assert.strictEqual(report.hasAddPorts, true, 'addPorts survived'); + assert.strictEqual(report.hasTicketKeys, true, 'ticketKeys survived'); + assert.strictEqual(report.name, WORKER_NAME); + + assert.deepStrictEqual(await throughPort, { type: 'through-transferred-port', nonce: 'n-1' }); + port1.close(); + }); + + it('never restarts or copies a one-shot worker, whose transferred ports are spent', async function () { + // The crash this guards: `restartWorkers` replaces workers through `worker.startCopy()`, which + // re-spawns from the SAME options object — so a worker carrying transferred ports would hit + // `DataCloneError` synchronously, inside the restart loop, taking the rest of the restart with it. + this.timeout(30000); + const { port1, port2 } = new MessageChannel(); + const worker = startWorker(FIXTURE, { + autoRestart: false, + name: WORKER_NAME, + execArgvOptions: { preloads: false }, + extraWorkerData: { certification: { candidateDirPath: '/tmp/c', nonce: 'n', verdictPort: port2 } }, + extraTransferList: [port2], + }); + + try { + assert.strictEqual(worker.isOneShot, true, 'a port-carrying spawn is marked one-shot'); + // The backstop for a direct caller: a named refusal rather than a DataCloneError from inside + // `new Worker`. + assert.throws(() => worker.startCopy(), /Cannot restart a one-shot/); + + // The restart loop's own filter is NOT driven here. `restartWorkers` runs a real node restart — + // it reinstalls applications, which shells out to `npm pack` — so calling it from a unit test + // exercises far more than the one-line `isOneShot` filter and would be slow and fragile for it. + // What is asserted above is the mechanism that filter depends on: the flag is set, and the copy + // refuses by name. The filter itself is covered by reading, and the crash it prevents is + // described in the commit that added it. + } finally { + port1.close(); + await worker.terminate().catch(() => {}); + } + }); + + it('leaves noServerStart absent unless asked for', async function () { + this.timeout(30000); + const report = await spawnAndReport({}); + assert.strictEqual(report.noServerStart, undefined); + assert.strictEqual(report.sawPort, false); + assert.strictEqual(report.hasAddPorts, true); + }); +});