diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9d69b22..8f7b180 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,7 +6,7 @@ Contributors are encouraged to communicate with maintainers in issues or other c Source files are located in `src/`. These are built to the `dist/` directory. The published package includes `dist/`, `scripts/`, and the regular npm metadata and documentation files. -The `src/index.ts` is the source for the main export. This is the public re-export of all the various utilities from `src/harperLifecycle.ts`, `targz.ts`, and more. The `src/run.ts` is the source for the `harper-integration-test-run` bin script. And the `scripts/setup-loopback.sh` is the source for the `harper-integration-test-setup-loopback` bin script. +The `src/index.ts` is the source for the main export. This is the public re-export of all the various utilities from `src/harperLifecycle.ts`, `targz.ts`, and more. The `src/run.ts` is the source for the `harper-integration-test-run` bin script. The internal `src/harperInstanceRegistry.ts` publishes each running Harper instance to a shared on-disk registry, and `src/harperMonitor.ts` is the singleton monitor process that reads it and reaps instances whose test runner died without cleaning up (see README's *Orphaned Instance Monitor* section). A registry record covers a whole process group, so the monitor — the only half that can see when that group is finished — is also the only writer that removes one; the lifecycle side registers and never deregisters. And the `scripts/setup-loopback.sh` is the source for the `harper-integration-test-setup-loopback` bin script. The package is `"type": "module"` — all source files are ESM by default. diff --git a/README.md b/README.md index 9ee90f1..aaecbcb 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,8 @@ The lifecycle and utility APIs below are framework-agnostic. They manage Harper Allocates a loopback address from the pool, creates a temporary install directory, starts a Harper process, and waits for it to be ready. Populates `ctx.harper` with the instance details. Call in a setup/`before()` hook. +Harper runs as its own detached process group, so a runner that dies without running cleanup would otherwise leave it alive holding the fixed ports. On POSIX the instance is registered with a shared [instance monitor](#orphaned-instance-monitor-posix) that reaps it in that case. + The Harper binary is resolved in the following order: 1. `harperBinPath` option passed directly to `startHarper()` @@ -204,6 +206,38 @@ suite('my suite', (ctx: ContextWithHarper) => { If you are not using `node:test`, use `createHarperContext()` to create a plain `HarperTestContext` instead. +### Orphaned Instance Monitor (POSIX) + +A test runner that dies without running its teardown — `SIGKILL`, a hard crash, a cancelled CI job — cannot reap the Harper instances it started, because those are deliberately detached into their own process groups so that whole-tree teardown works. Left alone, they hold their loopback address's fixed ports until the machine is rebooted. + +To close that gap, `startHarper()` registers each instance in a small on-disk registry and makes sure a single shared **monitor** process is running. The monitor is not a per-instance sidecar: one is started on demand per registry directory (per user on the machine, by default), every concurrent runner reuses it, and it exits once the registry has been empty for a while. Registry updates are published by renaming a complete file into place, so a runner killed mid-write leaves the previous registry — and therefore every other runner's reap targets — intact. It scans the registry on an interval and terminates — `SIGTERM`, then `SIGKILL` after a grace period — the process group of any instance whose owning runner is gone, or which has outlived its lifetime budget. Instances are matched by PID *and* process start time, which narrows PID reuse rather than eliminating it — a recycled PID that `ps` can still describe is rejected, and on a host whose `ps` cannot report a start time (busybox/Alpine) this degrades to a PID-only check, which the monitor logs on startup. + +A record describes a process *group*, not one process, and the monitor is the only thing that removes one — when that group has no members left. Harper exiting is not the end of the group: a child that ignored the `SIGTERM` stays in it, still holding the ports, and the record is what remembers the group long enough to escalate to `SIGKILL` or to reap it later when its runner dies. + +The runner's own `exit`/`SIGINT`/`SIGTERM`/`SIGHUP` handlers still reap instances immediately on any exit it can observe; the monitor only handles the deaths it cannot. + +Registry directory layout (`${TMPDIR}/harper-integration-test-monitor-${uid}` by default — per-user, because signalling another user's process group fails with `EPERM` and could never have reaped it): + +| File | Contents | +| --- | --- | +| `registry.json` | The current monitor and every registered instance (PID, start time, owning runner, loopback address, deadline) | +| `registry.lock` | Cross-process mutex guarding `registry.json` | +| `monitor.log` | Append-only record of monitor start/exit and every reap, with the reason | +| `registry.json.*.pending` | A registry update being written, renamed over `registry.json` once complete so no reader ever sees a partial one. Only present transiently, or left behind by a writer that was killed mid-update | + +Each managed Harper process also carries `HARPER_IT_KIND=harper-instance`, `HARPER_IT_INSTANCE_ID`, and `HARPER_IT_OWNER_PID` in its environment, and the monitor's command line contains `--harper-integration-test-monitor`, so both are identifiable from `ps` / `/proc` without consulting the registry. + +This is POSIX-only: reaping relies on process groups, which Windows does not have. On Windows, registration is skipped and the runner-side cleanup handlers are the only protection. + +**Environment Variables:** + +- `HARPER_INTEGRATION_TEST_MONITOR` - Set to `off` to disable registration and the monitor entirely. Default on (POSIX only). +- `HARPER_INTEGRATION_TEST_MONITOR_DIR` - Registry directory. Default `${TMPDIR}/harper-integration-test-monitor-${uid}`. Point separate runs at separate directories to give them separate monitors. +- `HARPER_INTEGRATION_TEST_MONITOR_INTERVAL_MS` - How often the monitor rescans the registry. Default `2000`. +- `HARPER_INTEGRATION_TEST_MONITOR_REAP_GRACE_MS` - Grace period between the monitor's SIGTERM and SIGKILL. Default `5000`. +- `HARPER_INTEGRATION_TEST_MONITOR_IDLE_MS` - How long the registry must stay empty before the monitor shuts down. Default `60000`. +- `HARPER_INTEGRATION_TEST_INSTANCE_MAX_LIFETIME_MS` - Backstop lifetime after which an instance is reaped even if its runner still looks alive. Default `14400000` (4h). + ### Server Log Capture When `HARPER_INTEGRATION_TEST_LOG_DIR` is set, each Harper instance writes its logs to a per-suite subdirectory. Logs are preserved for the lifetime of the log directory. In CI, combine with artifact upload steps that run on failure to capture logs from failing runs. diff --git a/src/harperInstanceRegistry.ts b/src/harperInstanceRegistry.ts new file mode 100644 index 0000000..e0b553c --- /dev/null +++ b/src/harperInstanceRegistry.ts @@ -0,0 +1,451 @@ +import { spawn, spawnSync } from 'node:child_process'; +import { randomBytes } from 'node:crypto'; +import { mkdir, open, readFile, rename, stat, unlink } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { fileURLToPath } from 'node:url'; + +/** + * Cross-process registry of running Harper test instances, and the singleton monitor process that + * reaps them. + * + * Every Harper instance the harness starts is detached (its own process-group leader) so that a + * test runner dying without running cleanup — `SIGKILL`, a hard crash, a cancelled CI job — leaves + * Harper alive holding its fixed ports. Instead of pairing every instance with its own sidecar, + * each instance publishes explicit metadata here and a single shared monitor (one per registry + * directory, i.e. per user on the machine by default) reaps whatever is orphaned or overdue. + * + * The registry is the contract between the two halves: `harperLifecycle.ts` registers instances, + * `harperMonitor.ts` scans, reaps, and removes them. Removal belongs to the monitor alone: a record + * describes a process *group*, and a runner watching only its direct child cannot tell when that + * group is finished. Both agree on tunables through the environment, which the monitor inherits + * from whichever runner first spawned it. + * + * POSIX only. Reaping is `kill(-pgid)`, which Windows has no equivalent for; on Windows + * registration is skipped and the runner-side cleanup handlers in `harperLifecycle.ts` remain the + * only protection. + */ + +/** Marker passed on the monitor's command line so `ps -ef | grep` finds it. */ +export const MONITOR_ARGV_MARKER = '--harper-integration-test-monitor'; + +/** Marks the environment of a Harper process the harness owns (visible in `/proc//environ`). */ +export const INSTANCE_ENV_KIND = 'HARPER_IT_KIND'; +export const INSTANCE_ENV_KIND_VALUE = 'harper-instance'; +export const INSTANCE_ENV_ID = 'HARPER_IT_INSTANCE_ID'; +export const INSTANCE_ENV_OWNER_PID = 'HARPER_IT_OWNER_PID'; + +const LOCK_STALE_TIMEOUT_MS = 10000; +const LOCK_RETRY_DELAY_MS = 50; +/** Ceiling on waiting for the lock, well past a stale reclaim so only a pathological holder hits it. */ +const LOCK_ACQUIRE_TIMEOUT_MS = 30000; +/** Ceiling on the `ps` lookup, so a call wedged in D-state cannot stall a caller indefinitely. */ +const PS_TIMEOUT_MS = 5000; + +let lockTokenCounter = 0; + +function envInt(name: string, fallback: number): number { + const parsed = parseInt(process.env[name] || '', 10); + return Number.isNaN(parsed) || parsed <= 0 ? fallback : parsed; +} + +/** + * Directory holding the registry, its lock, and the monitor log. Resolved on every call rather + * than at module load so a test (or a caller isolating a run) can point it somewhere private + * after import, and so the value the monitor inherits always matches its parent's. + * + * Per-user, because reaping across users cannot work anyway: signalling another user's process + * group fails with `EPERM`, which reads as "still alive", so a foreign record would never be + * pruned and would keep the monitor running forever. It also keeps the shared-`/tmp` path from + * being one another account can plant records in. + * + * Override with `HARPER_INTEGRATION_TEST_MONITOR_DIR`. + */ +export function getRegistryDir(): string { + if (process.env.HARPER_INTEGRATION_TEST_MONITOR_DIR) return process.env.HARPER_INTEGRATION_TEST_MONITOR_DIR; + const uid = process.getuid?.(); + return join(tmpdir(), uid === undefined ? 'harper-integration-test-monitor' : `harper-integration-test-monitor-${uid}`); +} + +export function getRegistryPath(): string { + return join(getRegistryDir(), 'registry.json'); +} + +function getLockPath(): string { + return join(getRegistryDir(), 'registry.lock'); +} + +export function getMonitorLogPath(): string { + return join(getRegistryDir(), 'monitor.log'); +} + +/** How often the monitor rescans the registry. Default 2s. */ +export function getMonitorScanIntervalMs(): number { + return envInt('HARPER_INTEGRATION_TEST_MONITOR_INTERVAL_MS', 2000); +} + +/** How long the registry must stay empty before the monitor shuts itself down. Default 60s. */ +export function getMonitorIdleExitMs(): number { + return envInt('HARPER_INTEGRATION_TEST_MONITOR_IDLE_MS', 60000); +} + +/** Grace period between the monitor's `SIGTERM` and its `SIGKILL` escalation. Default 5s. */ +export function getReapGraceMs(): number { + return envInt('HARPER_INTEGRATION_TEST_MONITOR_REAP_GRACE_MS', 5000); +} + +/** + * Backstop lifetime for a registered instance. The sharp signal is owner death; this only catches + * the pathological case where the runner's PID was recycled by a long-lived process, so it is + * deliberately far longer than any plausible suite. Default 4h. + */ +export function getInstanceMaxLifetimeMs(): number { + return envInt('HARPER_INTEGRATION_TEST_INSTANCE_MAX_LIFETIME_MS', 4 * 60 * 60 * 1000); +} + +/** Whether instances should be registered with a monitor at all. */ +export function isInstanceMonitorEnabled(): boolean { + return process.platform !== 'win32' && process.env.HARPER_INTEGRATION_TEST_MONITOR !== 'off'; +} + +/** Identifies a process well enough to survive PID reuse: the PID plus its wall-clock start time. */ +export interface ProcessIdentity { + pid: number; + /** + * `ps` start time (e.g. `Mon Aug 24 09:09:38 2026`). Absolute wall clock, so it stays valid + * across reboots, unlike the boot-relative tick counts in `/proc`. Undefined when `ps` was + * unavailable, in which case liveness degrades to a plain PID check. + */ + startTime?: string; +} + +export interface HarperInstanceRecord extends ProcessIdentity { + /** Stable id, also exported to the process as `HARPER_IT_INSTANCE_ID`. */ + id: string; + /** The test runner that started this instance; its death is what makes the instance an orphan. */ + owner: ProcessIdentity; + /** Loopback address the instance was assigned, for diagnostics in the monitor log. */ + hostname?: string; + registeredAt: number; + /** Wall-clock deadline after which the monitor reaps the instance regardless of owner liveness. */ + expiresAt: number; +} + +export interface InstanceRegistry { + /** The monitor currently responsible for this registry, if any. */ + monitor?: ProcessIdentity; + instances: HarperInstanceRecord[]; +} + +/** + * Reads the wall-clock start time of each PID in a single `ps` call. + * + * Absence from the result means the PID does not exist. A present-but-different value means the + * PID was recycled — the case that makes a bare `kill(-pgid)` dangerous, since the group we + * recorded may now belong to something unrelated. + */ +export function readProcessStartTimes(pids: number[]): Map { + const startTimes = new Map(); + const uniquePids = [...new Set(pids)].filter((pid) => Number.isInteger(pid) && pid > 0); + if (uniquePids.length === 0) return startTimes; + const result = spawnSync('ps', ['-o', 'pid=,lstart=', '-p', uniquePids.join(',')], { + encoding: 'utf8', + timeout: PS_TIMEOUT_MS, + killSignal: 'SIGKILL', + // `lstart` is rendered in the caller's timezone and locale, so runners configured differently + // would record different strings for one process and read each other's records as PID reuse — + // discarding live instances. Pin both so the string is a property of the process alone. + env: { ...process.env, TZ: 'UTC', LC_ALL: 'C' }, + }); + // A non-zero status just means none of the PIDs exist; only a missing `ps` is worth noticing, + // and there the empty map degrades callers to a plain PID check rather than reporting deaths. + if (result.error || typeof result.stdout !== 'string') return startTimes; + for (const line of result.stdout.split('\n')) { + const parsed = line.trim().match(/^(\d+)\s+(.*\S)$/); + if (parsed) startTimes.set(Number(parsed[1]), parsed[2]); + } + return startTimes; +} + +export function readProcessIdentity(pid: number): ProcessIdentity { + return { pid, startTime: readProcessStartTimes([pid]).get(pid) }; +} + +/** + * Whether `identity` still refers to the same live process, given start times already collected + * for this scan. When `ps` produced nothing at all (no start times for any PID) we cannot + * distinguish "gone" from "unmeasurable", so fall back to a signal-0 existence check. + */ +export function isSameProcessAlive(identity: ProcessIdentity, startTimes: Map): boolean { + const currentStartTime = startTimes.get(identity.pid); + if (currentStartTime === undefined) return startTimes.size === 0 && pidExists(identity.pid); + return identity.startTime === undefined || identity.startTime === currentStartTime; +} + +/** + * Whether the PID now belongs to a *different* process than the one recorded — distinct from a PID + * that has simply gone, and the difference matters wherever a record outlives its own process. + */ +export function isProcessIdentityReused(identity: ProcessIdentity, startTimes: Map): boolean { + const currentStartTime = startTimes.get(identity.pid); + return currentStartTime !== undefined && identity.startTime !== undefined && currentStartTime !== identity.startTime; +} + +function pidExists(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (error) { + // EPERM means the process exists but belongs to someone else — still alive. + return (error as NodeJS.ErrnoException).code === 'EPERM'; + } +} + +/** + * Acquires the registry lock, mirroring the `wx` create-or-fail mutex the loopback pool uses. + * Critical sections here are a read/modify/write of one small file, so the retry delay is much + * shorter than the pool's. + */ +async function acquireLock(): Promise { + const lockPath = getLockPath(); + const token = `${process.pid}-${++lockTokenCounter}-${randomBytes(8).toString('hex')}`; + await mkdir(getRegistryDir(), { recursive: true, mode: 0o700 }); + // Bounded so a pathological holder — one refreshing the lock faster than it goes stale — surfaces + // as a failed registration the caller can warn about, rather than a caller that never settles. + const deadline = Date.now() + LOCK_ACQUIRE_TIMEOUT_MS; + while (true) { + try { + const lockFileHandle = await open(lockPath, 'wx'); + try { + await lockFileHandle.writeFile(token); + } finally { + await lockFileHandle.close(); + } + return token; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error; + try { + const lockFileStat = await stat(lockPath); + // A holder that died mid-section would otherwise wedge every runner sharing this registry. + if (Date.now() - lockFileStat.mtimeMs > LOCK_STALE_TIMEOUT_MS) await unlink(lockPath); + } catch { + // Another process removed it first; just retry. + } + if (Date.now() >= deadline) { + throw new Error(`Timed out after ${LOCK_ACQUIRE_TIMEOUT_MS}ms waiting for the Harper instance registry lock at ${lockPath}`); + } + await sleep(LOCK_RETRY_DELAY_MS); + } + } +} + +export async function withRegistryLock(callback: () => Promise): Promise { + const token = await acquireLock(); + try { + return await callback(); + } finally { + try { + // Release only a lock we still hold. A critical section that overran the stale timeout has + // already been superseded, and unlinking then would remove the *new* holder's lock and let + // a third process into the section alongside it. + if ((await readFile(getLockPath(), 'utf-8')) === token) await unlink(getLockPath()); + } catch { + // Already released (e.g. reclaimed as stale). + } + } +} + +/** + * Reads the registry. Only call while holding the lock. An absent file reads as empty — the first + * registration by this user, or a registry directory someone cleared. `writeRegistryFile` + * publishes by rename, so a half-written file is never observable here. + */ +export async function readRegistryFile(): Promise { + let contents: string; + try { + contents = await readFile(getRegistryPath(), 'utf-8'); + } catch (error) { + // Only a missing file means an empty registry. Every other read failure has to propagate: + // reporting one as empty is what turns a transient problem into the caller writing that + // emptiness back, erasing live instances other runners are relying on us to reap. + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return { instances: [] }; + throw error; + } + let parsed: InstanceRegistry; + try { + parsed = JSON.parse(contents) as InstanceRegistry; + } catch (error) { + throw new Error( + `Harper instance registry at ${getRegistryPath()} is not valid JSON (delete it to reset monitoring): ${(error as Error).message}` + ); + } + if (!Array.isArray(parsed.instances)) { + // Same erasure the parse failure above guards against: reporting this as empty would have the + // caller write that emptiness back over whatever the file really described. + throw new Error(`Harper instance registry at ${getRegistryPath()} has no instance list (delete it to reset monitoring)`); + } + return { monitor: parsed.monitor, instances: parsed.instances }; +} + +let pendingWriteCounter = 0; + +/** + * Writes the registry. Only call while holding the lock. + * + * Publishes by rename so readers see either the old registry or the new one. An in-place write + * truncates first, and a writer killed in that window — the `SIGKILL` this whole mechanism exists + * to survive — leaves torn JSON, which used to read as an empty registry and take every reap + * target on the machine with it. The pending name is unpredictable and unique per write: a fixed + * one could be truncated by a writer that reclaimed the lock as stale, and one reused after PID + * reuse would collide with a leftover file and fail the registration outright. + */ +export async function writeRegistryFile(registry: InstanceRegistry): Promise { + await mkdir(getRegistryDir(), { recursive: true, mode: 0o700 }); + const registryPath = getRegistryPath(); + const pendingPath = `${registryPath}.${process.pid}.${++pendingWriteCounter}.${randomBytes(8).toString('hex')}.pending`; + try { + // `wx` (O_CREAT|O_EXCL) refuses to follow a symlink planted at the pending name, so a shared + // registry directory cannot be turned into an arbitrary-write primitive. + const pendingFileHandle = await open(pendingPath, 'wx'); + try { + await pendingFileHandle.writeFile(JSON.stringify(registry)); + } finally { + await pendingFileHandle.close(); + } + await rename(pendingPath, registryPath); + } catch (error) { + await unlink(pendingPath).catch(() => {}); + throw error; + } +} + +function getMonitorScript(): string { + const extension = import.meta.url.endsWith('.ts') ? 'ts' : 'js'; + return fileURLToPath(new URL(`./harperMonitor.${extension}`, import.meta.url)); +} + +/** + * Starts a monitor. Safe to call speculatively: a monitor that finds a live one already recorded + * in the registry exits immediately, so a race between two runners costs one short-lived process + * rather than a second reaper. + * + * Detached and unref'd so it outlives the runner that happened to start it — the whole point is to + * still be there when that runner dies. + */ +function spawnMonitor(): void { + try { + const monitor = spawn(process.execPath, [getMonitorScript(), MONITOR_ARGV_MARKER], { + detached: true, + stdio: 'ignore', + env: process.env, + }); + monitor.on('error', (error) => { + console.warn(`[harper-monitor] Failed to start the Harper instance monitor: ${error.message}`); + }); + monitor.unref(); + } catch (error) { + console.warn(`[harper-monitor] Failed to start the Harper instance monitor: ${(error as Error).message}`); + } +} + +let instanceCounter = 0; + +/** + * Environment markers identifying a Harper process as harness-owned. The registry is the + * authoritative list; these make an individual process self-describing for anyone inspecting it + * directly (`tr '\0' '\n' < /proc//environ | grep HARPER_IT_`). + */ +export function buildInstanceEnv(instanceId: string): Record { + return { + [INSTANCE_ENV_KIND]: INSTANCE_ENV_KIND_VALUE, + [INSTANCE_ENV_ID]: instanceId, + [INSTANCE_ENV_OWNER_PID]: String(process.pid), + }; +} + +/** + * Allocates the id used for both the instance's environment markers and its registry record. + * + * Uniqueness comes from the random suffix, not the counter: worker threads share `process.pid` and + * each holds its own copy of this module, so both first starts would claim `-1` and + * registration — which replaces same-id records — would leave one live instance unreapable. The + * PID and counter stay for legibility in `ps` and the monitor log. + */ +export function nextInstanceId(): string { + return `${process.pid}-${++instanceCounter}-${randomBytes(6).toString('hex')}`; +} + +/** + * Registers a running Harper instance with the shared monitor, starting one if none is live. + * + * The liveness check and the insert happen in a single critical section, which is what keeps the + * monitor's idle shutdown from racing a new registration: the monitor only exits while holding the + * lock with an empty registry, so either we see it alive and it sees our instance, or we see it + * gone and start a replacement. + */ +export async function registerHarperInstance(instance: { + id: string; + pid: number; + hostname?: string; +}): Promise { + if (!isInstanceMonitorEnabled()) return; + const startTimes = readProcessStartTimes([instance.pid, process.pid]); + const now = Date.now(); + const record: HarperInstanceRecord = { + id: instance.id, + pid: instance.pid, + startTime: startTimes.get(instance.pid), + owner: { pid: process.pid, startTime: startTimes.get(process.pid) }, + hostname: instance.hostname, + registeredAt: now, + expiresAt: now + getInstanceMaxLifetimeMs(), + }; + + const monitorNeeded = await withRegistryLock(async () => { + const registry = await readRegistryFile(); + registry.instances = registry.instances.filter((existing) => existing.id !== record.id); + registry.instances.push(record); + await writeRegistryFile(registry); + return !(registry.monitor && isSameProcessAlive(registry.monitor, readProcessStartTimes([registry.monitor.pid]))); + }); + + // Outside the lock: the monitor claims its slot under the same lock and would otherwise wait + // out our critical section before it could start. + if (monitorNeeded) spawnMonitor(); +} + +/** + * Whether a process group still has members. A group outlives its leader: Harper exiting on + * `SIGTERM` leaves any child that ignored the signal running in the same group, still holding the + * ports. POSIX reserves a group id for the lifetime of the group that holds it, so this answers for + * one continuous group — see `groupOutlivedLeader` for what that does and does not establish about + * whose group it is. + */ +export function processGroupExists(pgid: number): boolean { + // Group 0 is "the caller's own group" and 1 is init's; neither can be an instance we recorded. + if (!Number.isInteger(pgid) || pgid <= 1) return false; + try { + process.kill(-pgid, 0); + return true; + } catch (error) { + // EPERM means members exist that we may not signal — still a live group. + return (error as NodeJS.ErrnoException).code === 'EPERM'; + } +} + +/** + * Signals a whole process group. Instances are spawned detached, so the instance PID is also its + * process-group id and this reaches Harper plus anything it spawned. + */ +export function signalProcessGroup(pgid: number, signal: 'SIGTERM' | 'SIGKILL'): void { + // The registry is on-disk state a corrupt or hostile writer can reach, and `kill(-1)` broadcasts + // to every process this user may signal while `kill(-0)` hits our own group. No instance we + // registered is ever either, so refuse rather than translate a bad record into a wide signal. + if (!Number.isInteger(pgid) || pgid <= 1) return; + try { + process.kill(-pgid, signal); + } catch { + // The group is already gone. + } +} diff --git a/src/harperLifecycle.ts b/src/harperLifecycle.ts index 11f733b..65a656c 100644 --- a/src/harperLifecycle.ts +++ b/src/harperLifecycle.ts @@ -8,6 +8,7 @@ import { getNextAvailableLoopbackAddress, releaseLoopbackAddress } from './loopb import { waitForPortsFree } from './portUtils.ts'; import { ok, equal } from 'node:assert'; import { createRequire } from 'node:module'; +import { buildInstanceEnv, nextInstanceId, registerHarperInstance } from './harperInstanceRegistry.ts'; /** * Minimal context interface required by startHarper/teardownHarper. @@ -120,7 +121,7 @@ export const LOG_DIR_MARKER_PREFIX = '[Harper] Logs for this instance will be st export interface StartHarperOptions { /** * Maximum time (ms) to wait between chunks of startup output before treating Harper as hung. - * Resets on every chunk of output, so it bounds silence, not total boot time. + * Resets on every chunk of output, so it bounds silence rather than total boot time. * Falls back to {@link DEFAULT_STARTUP_TIMEOUT_MS} (60s locally, 150s under CI). */ startupTimeoutMs?: number; @@ -185,7 +186,7 @@ export interface HarperContext { operationsAPIURL: string; /** Assigned loopback IP address (e.g., '127.0.0.2') */ hostname: string; - /** Child process for the Harper instance */ + /** Child process handle for the Harper instance */ process: ChildProcess; /** Absolute path to the log directory for this suite (only set when HARPER_INTEGRATION_TEST_LOG_DIR is configured) */ logDir?: string; @@ -358,6 +359,8 @@ interface RunHarperCommandOptions { timeoutMs?: number; /** Absolute timeout (ms): ceiling on total time regardless of output. Falls back to DEFAULT_STARTUP_MAX_MS. */ maxMs?: number; + /** Loopback address this instance is bound to; recorded with the instance monitor for diagnostics. */ + hostname?: string; } interface RunHarperCommandResult { @@ -386,6 +389,7 @@ export function runHarperCommand({ harperBinPath, timeoutMs, maxMs, + hostname, }: RunHarperCommandOptions): Promise { const harperScript = getHarperScript(harperBinPath); const runtime = HARPER_RUNTIME; @@ -393,18 +397,20 @@ export function runHarperCommand({ runtime === 'bun' ? [harperScript, ...args] : ['--trace-warnings', '--force-node-api-uncaught-exceptions-policy=true', harperScript, ...args]; + const instanceId = nextInstanceId(); const proc = spawn(runtime, runtimeArgs, { - env: { ...process.env, ...env }, - // On POSIX, run Harper as its own process-group leader so teardown can signal the whole - // group (parent + any worker children), not just the direct child. Windows has no process - // groups; killHarper uses `taskkill /T` there instead. stdio stays piped (not detached), - // and we never unref, so output capture and lifetime management are unchanged. + env: { ...process.env, ...env, ...buildInstanceEnv(instanceId) }, + // On POSIX, run Harper as its own process-group leader so both teardown and the shared + // instance monitor can signal the whole group (parent + any worker children), not just the + // direct child. Windows has no process groups; killHarper uses `taskkill /T` there instead. + // stdio stays piped (not detached), and we never unref, so output capture and lifetime + // management are unchanged. detached: process.platform !== 'win32', }); - // Reap this process's tree if the runner exits/is interrupted before teardown (it's detached, - // so it would otherwise survive signals sent to the runner's group). - trackHarperProcess(proc); + // Publishes the instance to the shared monitor, which reaps it if this runner dies without + // running cleanup, and installs the runner-side handlers covering cooperative exits. + const trackedProcess = trackHarperProcess(proc, instanceId, hostname); let stdoutStream: WriteStream | undefined; let stderrStream: WriteStream | undefined; @@ -420,6 +426,7 @@ export function runHarperCommand({ let stdout = ''; let stderr = ''; let settled = false; + let readinessDetected = false; let idleTimer: NodeJS.Timeout; let maxTimer: NodeJS.Timeout; @@ -433,22 +440,34 @@ export function runHarperCommand({ settled = true; clearTimers(); reject(new HarperStartupError(message, stdout, stderr)); - // Harper is spawned detached (its own process group), so `proc.kill()` would only hit the - // direct child and orphan any worker children still holding ports. Reap the whole tree. + // Harper is the detached group leader, so reap the whole group rather than only it. signalHarperTree(proc, 'SIGKILL'); }; const succeed = () => { - if (settled) return; - settled = true; + if (settled || readinessDetected) return; + readinessDetected = true; + // Left armed across registration, these would let registry-lock contention time out — and + // kill — an instance that already booted successfully. clearTimers(); - resolve({ process: proc, stdout, stderr }); + // Resolve only once the instance is durably registered, so a runner killed the moment + // startHarper returns still leaves the monitor a record to act on. Registration always + // settles: `acquireLock` is bounded and `trackHarperProcess` absorbs its failures. + void trackedProcess.registered.then(() => { + if (settled) return; + settled = true; + resolve({ process: proc, stdout, stderr }); + }); }; + // Startup ends at readiness, not at resolution: the watchdog and the `startupOutput` snapshot + // both stop here, while resolution waits on registration for a little longer. + const startupFinished = () => settled || readinessDetected; + // Reset on every chunk of output so the limit is time-since-last-progress, not total boot // time: a slow-but-healthy boot that keeps logging never trips it, only true silence does. const resetIdleTimer = () => { - if (settled) return; + if (startupFinished()) return; clearTimeout(idleTimer); idleTimer = setTimeout( () => failStartup(`Harper produced no startup output for ${idleTimeoutMs}ms before reporting ready (likely hung)`), @@ -469,7 +488,7 @@ export function runHarperCommand({ // Once ready, keep streaming logs to disk but stop the watchdog and capture: the // returned startupOutput is a snapshot taken at readiness, and the server may run // (and log) for the rest of the suite. - if (settled) return; + if (startupFinished()) return; resetIdleTimer(); stdout += dataString; // Match against the accumulated output, not just this chunk, so a marker split across @@ -482,7 +501,7 @@ export function runHarperCommand({ proc.stderr?.on('data', (data: Buffer) => { const dataString = stripAnsi(data.toString()); stderrStream?.write(dataString); - if (settled) return; + if (startupFinished()) return; resetIdleTimer(); stderr += dataString; }); @@ -497,9 +516,9 @@ export function runHarperCommand({ reject(error); }); proc.on('exit', (statusCode, signal) => { - clearTimers(); if (!settled) { settled = true; + clearTimers(); if (statusCode === 0) { resolve({ process: proc, stdout, stderr }); } else { @@ -629,6 +648,7 @@ export async function startHarper(ctx: HarperTestContext, options?: StartHarperO harperBinPath: options?.harperBinPath, timeoutMs: options?.startupTimeoutMs, maxMs: options?.startupMaxMs, + hostname: loopbackAddress, }); publishHarperNode(ctx, { @@ -648,27 +668,25 @@ export async function startHarper(ctx: HarperTestContext, options?: StartHarperO return ctx as StartedHarperTestContext; } +function signalWindowsProcessTree(pid: number, signal: 'SIGTERM' | 'SIGKILL'): void { + const args = ['/pid', String(pid), '/T']; + if (signal === 'SIGKILL') args.push('/F'); + spawn('taskkill', args, { stdio: 'ignore' }).on('error', () => {}); +} + /** * Signals Harper's entire process tree, not just the direct child. * - * Harper runs its listeners in worker threads (in-process), but components can also spawn - * child processes, so we target the whole tree to be safe: - * - POSIX: Harper is spawned as its own process-group leader (`detached`), so a negative PID - * signals the group (parent + descendants). - * - Windows: there are no process groups, so we shell out to `taskkill /T` (tree). `/F` (force) - * is required to actually terminate console processes, so it is used for the SIGKILL step. - * - * Best-effort: errors (e.g. the process already exited) are swallowed; teardown's port assertion - * and the wait-for-exit are the safety nets. + * Harper is the detached POSIX process-group leader, with anything it spawns in that group, so a + * negative PID reaches the whole tree. Windows has no process groups, so `taskkill /T` is used + * there. Errors are best-effort because the target may already have exited. */ function signalHarperTree(proc: ChildProcess, signal: 'SIGTERM' | 'SIGKILL'): void { const pid = proc.pid; if (pid === undefined) return; if (process.platform === 'win32') { - const args = ['/pid', String(pid), '/T']; - if (signal === 'SIGKILL') args.push('/F'); try { - spawn('taskkill', args, { stdio: 'ignore' }).on('error', () => {}); + signalWindowsProcessTree(pid, signal); } catch { try { proc.kill(signal); @@ -691,36 +709,60 @@ function signalHarperTree(proc: ChildProcess, signal: 'SIGTERM' | 'SIGKILL'): vo } } +interface TrackedHarperProcess { + /** Settles once the instance is recorded with the shared monitor (or registration was skipped). */ + registered: Promise; +} + /** * Tracks live Harper processes so that if the test runner exits or is interrupted before teardown * runs (Ctrl+C, or a CI job killing the runner), their process trees are reaped instead of left - * orphaned holding the fixed ports. This matters specifically because Harper is spawned `detached` - * (its own process group), so it would otherwise survive signals delivered to the runner's group. + * orphaned holding the fixed ports. Harper is spawned detached (its own process group), so it + * would otherwise survive signals delivered to the runner's group. + * + * These parent-side hooks cover every exit the runner can still run JavaScript for. Death it + * cannot observe — `SIGKILL`, a hard crash, a cancelled CI job — is covered by the shared instance + * monitor, which reaps whatever the registry says is orphaned. See `harperInstanceRegistry.ts`. * - * Best-effort: on POSIX the group SIGKILL is delivered synchronously (works even from the 'exit' - * handler); on Windows the `taskkill` shell-out may not complete before the runner exits. + * On POSIX the group `SIGKILL` is delivered synchronously, so it works even from the 'exit' + * handler. On Windows the `taskkill` shell-out may not complete before the runner exits, and there + * is no monitor to fall back on. */ const liveHarperProcesses = new Set(); let runnerCleanupRegistered = false; -function trackHarperProcess(proc: ChildProcess): void { +function trackHarperProcess(proc: ChildProcess, instanceId: string, hostname?: string): TrackedHarperProcess { liveHarperProcesses.add(proc); + // A failed spawn leaves no PID and nothing to reap; the caller rejects on the 'error' event. + const registered = + proc.pid === undefined + ? Promise.resolve() + : registerHarperInstance({ id: instanceId, pid: proc.pid, hostname }).catch((error: Error) => { + console.warn(`[harper-monitor] Failed to register Harper instance ${instanceId}: ${error.message}`); + }); + const trackedProcess: TrackedHarperProcess = { registered }; + + // Only the direct child is untracked here; its registry record covers the group, which can + // outlive it, and the monitor owns removing that. proc.once('exit', () => liveHarperProcesses.delete(proc)); - if (runnerCleanupRegistered) return; + if (runnerCleanupRegistered) return trackedProcess; runnerCleanupRegistered = true; const reapAll = () => { for (const child of liveHarperProcesses) signalHarperTree(child, 'SIGKILL'); }; process.once('exit', reapAll); - // SIGINT/SIGTERM don't fire 'exit'; reap, then re-raise so the runner still terminates normally. - for (const signal of ['SIGINT', 'SIGTERM'] as const) { + // These signals don't fire 'exit'; reap, then re-raise so the runner still terminates normally. + const cleanupSignals: NodeJS.Signals[] = ['SIGINT', 'SIGTERM']; + if (process.platform !== 'win32') cleanupSignals.push('SIGHUP'); + for (const signal of cleanupSignals) { process.once(signal, () => { reapAll(); process.kill(process.pid, signal); }); } + return trackedProcess; } /** Identifies the objects published as `ctx.harper`; a shallow copy of a node does not carry it. */ diff --git a/src/harperMonitor.ts b/src/harperMonitor.ts new file mode 100644 index 0000000..7ea4678 --- /dev/null +++ b/src/harperMonitor.ts @@ -0,0 +1,231 @@ +import { constants, existsSync } from 'node:fs'; +import { open } from 'node:fs/promises'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { + getMonitorIdleExitMs, + getMonitorLogPath, + getMonitorScanIntervalMs, + getReapGraceMs, + getRegistryPath, + isProcessIdentityReused, + isSameProcessAlive, + processGroupExists, + readProcessIdentity, + readProcessStartTimes, + readRegistryFile, + signalProcessGroup, + withRegistryLock, + writeRegistryFile, + type HarperInstanceRecord, +} from './harperInstanceRegistry.ts'; + +/** + * The singleton Harper instance monitor. + * + * One of these runs per registry directory, shared by every concurrent test runner using it. + * It periodically scans the registry written by `harperLifecycle.ts` and reaps any instance whose + * owning runner has died or which has outlived its budget, then shuts itself down once the + * registry has been empty long enough. See `harperInstanceRegistry.ts` for the shared contract. + * + * Runs detached with no stdio, so diagnostics go to `monitor.log` in the registry directory. + */ + +const scanIntervalMs = getMonitorScanIntervalMs(); +const idleExitMs = getMonitorIdleExitMs(); +const reapGraceMs = getReapGraceMs(); + +/** Instances already sent `SIGTERM`, mapped to the time we escalate to `SIGKILL`. */ +const escalationDeadlines = new Map(); +let idleSince: number | undefined; +let running = true; + +// O_NOFOLLOW so a symlink planted at the log path in a shared registry directory cannot redirect +// these appends into a file the runner can write. +const LOG_FLAGS = constants.O_WRONLY | constants.O_CREAT | constants.O_APPEND | (constants.O_NOFOLLOW ?? 0); + +async function log(message: string): Promise { + try { + const logFileHandle = await open(getMonitorLogPath(), LOG_FLAGS, 0o600); + try { + await logFileHandle.writeFile(`${new Date().toISOString()} [${process.pid}] ${message}\n`); + } finally { + await logFileHandle.close(); + } + } catch { + // The registry directory is gone; the next loop iteration notices and shuts us down. + } +} + +/** + * Takes ownership of the registry, or reports that someone else already has it. + * + * Two runners starting at once can each decide a monitor is needed; the loser exits here rather + * than double-reaping. Recording our own identity (not just our PID) means a later monitor can + * tell "still running" from "PID recycled". + */ +async function claimMonitorSlot(): Promise { + // A registrant always writes the registry before starting us, so a missing file means the run + // we were spawned for was already torn down. Stand down rather than recreate its directory. + if (!existsSync(getRegistryPath())) return false; + return withRegistryLock(async () => { + const registry = await readRegistryFile(); + const existing = registry.monitor; + if (existing && isSameProcessAlive(existing, readProcessStartTimes([existing.pid]))) return false; + registry.monitor = readProcessIdentity(process.pid); + await writeRegistryFile(registry); + return true; + }); +} + +/** Gives up ownership so the next registration starts a fresh monitor rather than trusting a dead one. */ +async function releaseMonitorSlot(): Promise { + try { + await withRegistryLock(async () => { + const registry = await readRegistryFile(); + if (registry.monitor?.pid !== process.pid) return; + delete registry.monitor; + await writeRegistryFile(registry); + }); + } catch { + // Best-effort: a stale slot is detected by the liveness check on the next registration. + } +} + +interface ReapTarget { + instance: HarperInstanceRecord; + reason: string; +} + +/** + * Whether the instance's group is still running, now that the process we recorded as its leader is + * not. A leader PID `ps` still describes, with a different start time, is a reused id whose group + * is somebody else's. + * + * PID *absence* is the best evidence available here, not proof of ownership: a group id is reserved + * only for the lifetime of one group, so a group of ours that ended while nothing was watching, had + * its id reused, and then lost its own leader is indistinguishable from ours. Reaping is best-effort + * against PID reuse throughout this registry, and this is that same bar, widened by however long the + * monitor was not looking. + */ +function groupOutlivedLeader(instance: HarperInstanceRecord, startTimes: Map): boolean { + return !isProcessIdentityReused(instance, startTimes) && processGroupExists(instance.pid); +} + +/** Why an instance should be reaped, or undefined while it is still legitimately running. */ +function reapReason(instance: HarperInstanceRecord, startTimes: Map, now: number): string | undefined { + if (!isSameProcessAlive(instance.owner, startTimes)) return `owning runner ${instance.owner.pid} is gone`; + if (now > instance.expiresAt) { + return `exceeded its ${Math.round((instance.expiresAt - instance.registeredAt) / 1000)}s lifetime budget`; + } + return undefined; +} + +/** + * Prunes records whose process group is gone — the monitor is the only writer that removes one — + * and returns the ones that should be reaped. A monitor killed mid-grace therefore leaves a target + * its successor picks straight back up. + */ +async function scanRegistry(): Promise<{ live: HarperInstanceRecord[]; targets: ReapTarget[] }> { + return withRegistryLock(async () => { + const registry = await readRegistryFile(); + const startTimes = readProcessStartTimes( + registry.instances.flatMap((instance) => [instance.pid, instance.owner.pid]) + ); + const live: HarperInstanceRecord[] = []; + const targets: ReapTarget[] = []; + const now = Date.now(); + for (const instance of registry.instances) { + // Retention is the group's lifetime, deliberately not the leader's and not whether a reap + // is due: the survivors of a leader that exited — on our SIGTERM or on its own — still + // hold the ports, and this record is the only thing that remembers them, whether they are + // reaped now, when their runner dies, or at the lifetime budget. + if (!isSameProcessAlive(instance, startTimes) && !groupOutlivedLeader(instance, startTimes)) continue; + live.push(instance); + const reason = reapReason(instance, startTimes, now); + if (reason !== undefined) targets.push({ instance, reason }); + } + if (live.length !== registry.instances.length) { + registry.instances = live; + await writeRegistryFile(registry); + } + return { live, targets }; + }); +} + +/** + * Terminates an instance's process group: `SIGTERM` first so Harper can flush and release its + * ports cleanly, escalating to `SIGKILL` on a later scan if it is still alive after the grace + * period. Killing by group id is safe here because the record's start time already proved the + * group leader is the process we registered, not a recycled PID — or, once that leader is gone, on + * the narrower evidence `groupOutlivedLeader` describes. + */ +async function reap({ instance, reason }: ReapTarget): Promise { + const escalateAt = escalationDeadlines.get(instance.id); + if (escalateAt === undefined) { + await log(`Reaping Harper ${instance.pid}${instance.hostname ? ` (${instance.hostname})` : ''}: ${reason}. SIGTERM.`); + escalationDeadlines.set(instance.id, Date.now() + reapGraceMs); + signalProcessGroup(instance.pid, 'SIGTERM'); + return; + } + if (Date.now() < escalateAt) return; + await log(`Harper ${instance.pid} survived SIGTERM for ${reapGraceMs}ms; escalating to SIGKILL.`); + // Push the next escalation past this scan's horizon so a process stuck in D-state does not + // produce a SIGKILL (and a log line) on every tick. + escalationDeadlines.set(instance.id, Date.now() + reapGraceMs); + signalProcessGroup(instance.pid, 'SIGKILL'); +} + +/** Shuts down once the registry has been empty for the idle window, under the lock so a concurrent registration wins. */ +async function exitIfIdle(): Promise { + if (idleSince === undefined || Date.now() - idleSince < idleExitMs) return false; + const released = await withRegistryLock(async () => { + const registry = await readRegistryFile(); + if (registry.instances.length > 0) return false; + if (registry.monitor?.pid === process.pid) delete registry.monitor; + await writeRegistryFile(registry); + return true; + }); + if (!released) idleSince = undefined; + return released; +} + +for (const signal of ['SIGINT', 'SIGTERM', 'SIGHUP'] as const) { + process.once(signal, () => { + running = false; + void releaseMonitorSlot().then(() => process.exit(0)); + }); +} + +if (!(await claimMonitorSlot())) process.exit(0); +await log(`Monitor started (scan ${scanIntervalMs}ms, idle exit ${idleExitMs}ms, reap grace ${reapGraceMs}ms).`); +// `ps -o lstart=` is absent on busybox, so identity checks silently degrade to a bare PID test and +// a recycled PID can be mistaken for its predecessor. Say so once rather than reaping on a +// guarantee this host cannot provide. +if (readProcessStartTimes([process.pid]).size === 0) { + await log('WARNING: `ps -o pid=,lstart=` returned nothing; falling back to PID-only liveness checks (PID reuse is undetectable).'); +} + +while (running) { + // Someone removed the registry out from under us (a cleaned-up run, or a developer clearing + // tmp): there is nothing left to supervise, and recreating it would strand an empty directory. + if (!existsSync(getRegistryPath())) process.exit(0); + try { + const { live, targets } = await scanRegistry(); + for (const id of escalationDeadlines.keys()) { + if (!live.some((instance) => instance.id === id)) escalationDeadlines.delete(id); + } + for (const target of targets) await reap(target); + + if (live.length > 0) idleSince = undefined; + else idleSince ??= Date.now(); + if (await exitIfIdle()) { + await log('Monitor exiting: no registered Harper instances.'); + process.exit(0); + } + } catch (error) { + // Never let one bad scan take the monitor down — it is the last line of defence against + // orphaned instances, and the next scan may well succeed. + await log(`Scan failed: ${(error as Error).stack || (error as Error).message}`); + } + await sleep(scanIntervalMs); +} diff --git a/test/harperLifecycle.test.ts b/test/harperLifecycle.test.ts index 0f1ba75..4e7a70a 100644 --- a/test/harperLifecycle.test.ts +++ b/test/harperLifecycle.test.ts @@ -1,11 +1,13 @@ import { test, before, after } from 'node:test'; -import { ok, strictEqual, match, rejects } from 'node:assert'; +import { ok, strictEqual, match, doesNotMatch, rejects } from 'node:assert'; import { spawn, type ChildProcess } from 'node:child_process'; import { once } from 'node:events'; import { mkdtempSync, writeFileSync, rmSync, existsSync } from 'node:fs'; +import { readFile, writeFile as writeFileAsync } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { setTimeout as sleep } from 'node:timers/promises'; +import { fileURLToPath } from 'node:url'; import { killHarper, teardownHarper, @@ -18,12 +20,24 @@ import { buildHarperChildEnv, type StartedHarperTestContext, } from '../src/harperLifecycle.ts'; +import { + MONITOR_ARGV_MARKER, + readProcessStartTimes, + readRegistryFile, + signalProcessGroup, + withRegistryLock, + type InstanceRegistry, +} from '../src/harperInstanceRegistry.ts'; +import { isPortFree, waitForPortsFree } from '../src/portUtils.ts'; // Standalone scripts used as a fake "Harper binary" (passed via harperBinPath) to drive // runHarperCommand's startup watchdog through specific timing scenarios without a real Harper. const FIXTURE_SOURCES: Record = { // Reports ready, then stays alive like a server. 'ready.cjs': "process.stdout.write('booting\\n');\nsetTimeout(() => process.stdout.write('successfully started\\n'), 50);\nsetInterval(() => {}, 1000);\n", + // Reports ready, then logs once more and goes quiet — what a real Harper does after startup, + // and what re-arms a startup watchdog that keys on resolution instead of readiness. + 'ready-then-log.cjs': "process.stdout.write('booting\\n');\nsetTimeout(() => process.stdout.write('successfully started\\n'), 50);\nsetTimeout(() => process.stdout.write('post-readiness log line\\n'), 150);\nsetInterval(() => {}, 1000);\n", // Emits one line, then goes silent forever (hung during startup). 'idle-hang.cjs': "process.stdout.write('booting\\n');\nsetInterval(() => {}, 1000);\n", // Emits output continuously but never reports ready. @@ -33,8 +47,123 @@ const FIXTURE_SOURCES: Record = { 'idle-reset.cjs': "let n = 0;\nconst t = setInterval(() => {\n n++;\n process.stdout.write('progress ' + n + '\\n');\n if (n >= 8) { clearInterval(t); process.stdout.write('successfully started\\n'); }\n}, 100);\nsetInterval(() => {}, 1000);\n", // Exits non-zero. 'exit-nonzero.cjs': "process.stderr.write('boom\\n');\nprocess.exit(1);\n", + 'process-tree.cjs': ` +const { spawn } = require('node:child_process'); +const { createServer } = require('node:net'); +const host = '127.0.0.1'; +if (process.env.HARPER_FAKE_DESCENDANT === '1') { + // A child that ignores SIGTERM outlives the group leader the monitor's SIGTERM does kill, and + // keeps its listener — the case where reaping has to continue after the leader is gone. + if (process.env.HARPER_DESCENDANT_IGNORE_TERM === '1') process.on('SIGTERM', () => {}); + const server = createServer(); + server.listen(0, host, () => { + process.stdout.write('descendant-ready:' + process.pid + ':' + server.address().port + '\\n'); + }); +} else { + const descendant = spawn(process.execPath, [__filename], { + env: { ...process.env, HARPER_FAKE_DESCENDANT: '1' }, + stdio: ['ignore', 'pipe', 'inherit'], + }); + let descendantOutput = ''; + descendant.stdout.on('data', (chunk) => { + descendantOutput += chunk; + const descendantMatch = descendantOutput.match(/descendant-ready:(\\d+):(\\d+)/); + if (!descendantMatch) return; + const server = createServer(); + server.listen(0, host, () => { + process.stdout.write('tree-ready:' + process.pid + ':' + descendant.pid + ':' + server.address().port + ':' + descendantMatch[2] + '\\n'); + process.stdout.write('successfully started\\n'); + }); + const delay = Number(process.env.HARPER_TERM_DELAY_MS || 0); + if (delay > 0) process.on('SIGTERM', () => setTimeout(() => server.close(() => process.exit(0)), delay)); + }); +} +`, + // Rewrites the registry back to back so a reader — or the SIGKILL the test sends — is almost + // always looking at a write in flight. The padded shape makes each write big enough to span + // several of the reader's polls. + 'registry-writer.mjs': ` +const { writeRegistryFile } = await import(process.env.HARPER_REGISTRY_URL); +// Never a real process group: a bogus start time means any monitor that somehow read this private +// registry would treat the record as gone rather than signalling the PID's group. +const seeded = { + id: 'seeded', + pid: process.pid, + startTime: 'writer-fixture', + owner: { pid: process.pid, startTime: 'writer-fixture' }, + registeredAt: 0, + expiresAt: 0, +}; +const small = { instances: [seeded] }; +const padded = { instances: [seeded, { ...seeded, id: 'padding', hostname: 'x'.repeat(2_000_000) }] }; +await writeRegistryFile(small); +process.stdout.write('writer-ready\\n'); +for (;;) { + await writeRegistryFile(padded); + await writeRegistryFile(small); +} +`, + 'orphan-runner.mjs': ` +const { runHarperCommand, killHarper } = await import(process.env.HARPER_LIFECYCLE_URL); +const result = await runHarperCommand({ + args: [], + env: { + HARPER_TERM_DELAY_MS: process.env.HARPER_TERM_DELAY_MS, + HARPER_DESCENDANT_IGNORE_TERM: process.env.HARPER_DESCENDANT_IGNORE_TERM, + }, + completionMessage: 'successfully started', + harperBinPath: process.env.HARPER_FAKE_SCRIPT, + timeoutMs: 5000, + maxMs: 10000, + hostname: '127.0.0.1', +}); +const match = result.stdout.match(/tree-ready:(\\d+):(\\d+):(\\d+):(\\d+)/); +if (!match) throw new Error('Missing fake Harper process markers: ' + result.stdout); +process.stdout.write('runner-ready:' + result.process.pid + ':' + match[1] + ':' + match[2] + ':' + match[3] + ':' + match[4] + '\\n'); +if (process.env.HARPER_RUNNER_MODE === 'teardown') { + await killHarper({ harper: { process: result.process } }, { graceMs: 2000 }); + let harperGone = false; + try { process.kill(result.process.pid, 0); } catch { harperGone = true; } + process.stdout.write('teardown-complete:' + harperGone + '\\n'); +} else { + setInterval(() => {}, 1000); +} +`, + // One runner process starting a Harper instance from each of two worker threads — the shape a + // worker-pooled test runner (vitest threads, and anything else sharing one process) produces. + // Both workers share process.pid, so only an id unique per module copy keeps both records. + 'worker-runner.mjs': ` +import { Worker } from 'node:worker_threads'; +for (const index of [0, 1]) { + const worker = new Worker(process.env.HARPER_WORKER_SCRIPT, { workerData: { index } }); + worker.on('message', (message) => process.stdout.write(message + '\\n')); + worker.on('error', (error) => { + process.stderr.write('worker ' + index + ' failed: ' + (error.stack || error) + '\\n'); + process.exit(1); + }); +} +setInterval(() => {}, 1000); +`, + 'worker-harper.mjs': ` +import { parentPort, workerData } from 'node:worker_threads'; +const { runHarperCommand } = await import(process.env.HARPER_LIFECYCLE_URL); +const result = await runHarperCommand({ + args: [], + env: {}, + completionMessage: 'successfully started', + harperBinPath: process.env.HARPER_FAKE_SCRIPT, + timeoutMs: 5000, + maxMs: 10000, + hostname: '127.0.0.1', +}); +const match = result.stdout.match(/tree-ready:(\\d+):(\\d+):(\\d+):(\\d+)/); +if (!match) throw new Error('Missing fake Harper process markers: ' + result.stdout); +parentPort.postMessage('worker-ready:' + workerData.index + ':' + match[1] + ':' + match[2] + ':' + match[3] + ':' + match[4]); +`, }; +const isPosix = process.platform !== 'win32'; + let fixtureDir: string; const fixtures: Record = {}; @@ -45,6 +174,10 @@ before(() => { writeFileSync(path, src); fixtures[name] = path; } + // This process starts fake Harper instances directly in several tests; keep them out of the + // shared registry so they can never outlive the run or perturb the monitor tests below, each of + // which opts back in against its own private registry directory. + process.env.HARPER_INTEGRATION_TEST_MONITOR = 'off'; }); after(() => { @@ -59,22 +192,53 @@ function fakeCtx(process?: ChildProcess): StartedHarperTestContext { function waitForOutput(child: ChildProcess, needle: string): Promise { return new Promise((resolve) => { let buffer = ''; - child.stdout?.on('data', (chunk: Buffer) => { + const onData = (chunk: Buffer) => { buffer += chunk.toString(); - if (buffer.includes(needle)) resolve(); - }); + if (!buffer.includes(needle)) return; + child.stdout?.off('data', onData); + resolve(); + }; + child.stdout?.on('data', onData); }); } /** Resolves with the regex match once it appears on the child's stdout. */ -function waitForMatch(child: ChildProcess, regex: RegExp): Promise { - return new Promise((resolve) => { +function waitForMatch(child: ChildProcess, regex: RegExp, timeoutMs = 5000): Promise { + return new Promise((resolve, reject) => { let buffer = ''; - child.stdout?.on('data', (chunk: Buffer) => { + const onData = (chunk: Buffer) => { buffer += chunk.toString(); const matched = buffer.match(regex); - if (matched) resolve(matched); - }); + if (!matched) return; + clearTimeout(timeout); + child.stdout?.off('data', onData); + resolve(matched); + }; + const timeout = setTimeout(() => { + child.stdout?.off('data', onData); + reject(new Error(`Timed out waiting for ${regex}; output: ${buffer}`)); + }, timeoutMs); + child.stdout?.on('data', onData); + }); +} + +/** Resolves with all `count` matches of a global regex once they have appeared on the child's stdout. */ +function waitForMatches(child: ChildProcess, regex: RegExp, count: number, timeoutMs = 15000): Promise { + return new Promise((resolve, reject) => { + let buffer = ''; + const onData = (chunk: Buffer) => { + buffer += chunk.toString(); + const matched = [...buffer.matchAll(regex)]; + if (matched.length < count) return; + clearTimeout(timeout); + child.stdout?.off('data', onData); + resolve(matched); + }; + const timeout = setTimeout(() => { + child.stdout?.off('data', onData); + reject(new Error(`Timed out waiting for ${count} matches of ${regex}; output: ${buffer}`)); + }, timeoutMs); + child.stdout?.on('data', onData); }); } @@ -92,6 +256,130 @@ async function waitProcessGone(pid: number, timeoutMs: number): Promise } } +const MONITOR_SCRIPT = fileURLToPath(new URL('../src/harperMonitor.ts', import.meta.url)); + +interface RunningFakeHarperTree { + runner: ChildProcess; + harperPid: number; + descendantPid: number; + ports: number[]; + /** Private registry directory this runner's monitor owns. */ + monitorDir: string; + /** True when this tree created `monitorDir` and is therefore responsible for removing it. */ + ownsMonitorDir: boolean; +} + +interface FakeHarperTreeOptions { + mode?: 'teardown'; + /** Make the descendant ignore `SIGTERM`, so it outlives the leader the monitor's `SIGTERM` kills. */ + descendantIgnoresTerm?: boolean; + /** Share an existing registry directory (and therefore an existing monitor) with another tree. */ + monitorDir?: string; + /** `HARPER_INTEGRATION_TEST_MONITOR_IDLE_MS` for the monitor this runner may start. */ + idleMs?: string; + /** `HARPER_INTEGRATION_TEST_INSTANCE_MAX_LIFETIME_MS` for this runner's instance. */ + maxLifetimeMs?: string; +} + +/** Restores an environment variable, removing it when it was previously unset — assigning `undefined` would set the string `"undefined"`. */ +function restoreEnv(name: string, value: string | undefined): void { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; +} + +/** Reads a registry directly. Registry writes are atomic, so this only reads empty before the first one. */ +async function readMonitorRegistry(monitorDir: string): Promise { + try { + return JSON.parse(await readFile(join(monitorDir, 'registry.json'), 'utf-8')) as InstanceRegistry; + } catch { + return { instances: [] }; + } +} + +/** Waits for a monitor to claim `monitorDir` and returns its PID. */ +async function waitForMonitor(monitorDir: string, timeoutMs = 10000): Promise { + const deadline = Date.now() + timeoutMs; + for (;;) { + const monitorPid = (await readMonitorRegistry(monitorDir)).monitor?.pid; + if (monitorPid !== undefined) return monitorPid; + if (Date.now() >= deadline) throw new Error(`No monitor claimed ${monitorDir} within ${timeoutMs}ms`); + await sleep(50); + } +} + +/** + * Starts a separate runner process that launches a fake Harper tree (a process-group leader plus a + * descendant, each holding a TCP port) through `runHarperCommand`, so tests can kill the runner + * the way CI does and observe what happens to the tree. + */ +async function startFakeHarperTree(options: FakeHarperTreeOptions = {}): Promise { + const monitorDir = options.monitorDir ?? mkdtempSync(join(tmpdir(), 'harper-it-monitor-')); + const runner = spawn(process.execPath, [fixtures['orphan-runner.mjs']], { + env: { + ...process.env, + HARPER_LIFECYCLE_URL: new URL('../src/harperLifecycle.ts', import.meta.url).href, + HARPER_FAKE_SCRIPT: fixtures['process-tree.cjs'], + HARPER_TERM_DELAY_MS: options.mode === 'teardown' ? '300' : '0', + HARPER_DESCENDANT_IGNORE_TERM: options.descendantIgnoresTerm ? '1' : '0', + HARPER_RUNNER_MODE: options.mode, + // Re-enable monitoring (this test process disables it) against a private registry, with + // timings compressed so an orphan is reaped in well under a test timeout. + HARPER_INTEGRATION_TEST_MONITOR: 'on', + HARPER_INTEGRATION_TEST_MONITOR_DIR: monitorDir, + HARPER_INTEGRATION_TEST_MONITOR_INTERVAL_MS: '200', + HARPER_INTEGRATION_TEST_MONITOR_REAP_GRACE_MS: '500', + HARPER_INTEGRATION_TEST_MONITOR_IDLE_MS: options.idleMs ?? '60000', + HARPER_INTEGRATION_TEST_INSTANCE_MAX_LIFETIME_MS: options.maxLifetimeMs ?? '', + }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + const tree: RunningFakeHarperTree = { + runner, + harperPid: 0, + descendantPid: 0, + ports: [], + monitorDir, + ownsMonitorDir: options.monitorDir === undefined, + }; + let match: RegExpMatchArray; + try { + match = await waitForMatch(runner, /runner-ready:(\d+):(\d+):(\d+):(\d+):(\d+)/); + } catch (error) { + await cleanupFakeHarperTree(tree); + throw error; + } + strictEqual(Number(match[1]), Number(match[2]), 'the lifecycle handle should be the Harper process itself'); + tree.harperPid = Number(match[2]); + tree.descendantPid = Number(match[3]); + tree.ports = [Number(match[4]), Number(match[5])]; + return tree; +} + +function forceKill(pid: number | undefined): void { + // A tree whose readiness timed out still carries its initial 0s, and `process.kill(0, ...)` + // signals the test runner's own process group. + if (pid === undefined || pid <= 0) return; + try { + process.kill(pid, 'SIGKILL'); + } catch { + // The process already exited. + } +} + +async function cleanupFakeHarperTree(tree: Partial): Promise { + forceKill(tree.runner?.pid); + forceKill(tree.harperPid); + forceKill(tree.descendantPid); + if (tree.monitorDir) { + // Monitors outlive their runner by design, so tests must reap their own. Wait for the slot + // to be claimed first, otherwise a monitor still starting up survives the cleanup. + const monitorPid = await waitForMonitor(tree.monitorDir, 5000).catch(() => undefined); + forceKill(monitorPid); + if (tree.ownsMonitorDir) rmSync(tree.monitorDir, { recursive: true, force: true }); + } + if (tree.ports?.length) await waitForPortsFree('127.0.0.1', tree.ports, 2000, 50); +} + // --- Startup watchdog (Race 1) --- test('runHarperCommand resolves when the completion message appears', async () => { @@ -152,7 +440,8 @@ test('runHarperCommand keeps a slow-but-progressing boot alive past the idle win env: {}, completionMessage: 'successfully started', harperBinPath: fixtures['idle-reset.cjs'], - timeoutMs: 400, + // The idle window has to cover process launch before the fixture's first output. + timeoutMs: 700, maxMs: 10000, }); try { @@ -162,6 +451,429 @@ test('runHarperCommand keeps a slow-but-progressing boot alive past the idle win } }); +// --- Orphan reaping via the shared instance monitor (POSIX only) --- + +test('runner SIGKILL leaves the monitor to reap the orphaned Harper tree and release its ports', { skip: !isPosix }, async () => { + const tree = await startFakeHarperTree(); + try { + await waitForMonitor(tree.monitorDir); + strictEqual(await isPortFree('127.0.0.1', tree.ports[0]), false); + strictEqual(await isPortFree('127.0.0.1', tree.ports[1]), false); + // SIGKILL is precisely the death the runner's own cleanup handlers cannot observe, so + // everything below is the monitor's doing. + tree.runner.kill('SIGKILL'); + await once(tree.runner, 'exit'); + ok(await waitProcessGone(tree.harperPid, 10000), `Harper ${tree.harperPid} should be reaped with its runner`); + ok(await waitProcessGone(tree.descendantPid, 10000), `descendant ${tree.descendantPid} should be reaped with its runner`); + ok(await waitForPortsFree('127.0.0.1', tree.ports, 5000, 50), 'Harper tree ports should be reusable'); + } finally { + await cleanupFakeHarperTree(tree); + } +}); + +test('the monitor finishes reaping a group whose leader dies before its children', { skip: !isPosix }, async () => { + // The monitor's SIGTERM lands on the whole group, so the leader (no handler) exits first while a + // child that ignores SIGTERM keeps running — and keeps the port. Dropping the record the moment + // its leader is gone would cancel the SIGKILL escalation and strand exactly that child. + const tree = await startFakeHarperTree({ descendantIgnoresTerm: true }); + try { + await waitForMonitor(tree.monitorDir); + strictEqual(await isPortFree('127.0.0.1', tree.ports[1]), false); + tree.runner.kill('SIGKILL'); + await once(tree.runner, 'exit'); + ok(await waitProcessGone(tree.harperPid, 10000), `leader ${tree.harperPid} should exit on the monitor's SIGTERM`); + ok( + await waitProcessGone(tree.descendantPid, 15000), + `TERM-ignoring descendant ${tree.descendantPid} should still be escalated to SIGKILL` + ); + ok(await waitForPortsFree('127.0.0.1', tree.ports, 5000, 50), 'the surviving child must not keep its port'); + } finally { + await cleanupFakeHarperTree(tree); + } +}); + +test('runner SIGHUP reaps the Harper tree and releases its ports', { skip: !isPosix }, async () => { + const tree = await startFakeHarperTree(); + try { + tree.runner.kill('SIGHUP'); + await once(tree.runner, 'exit'); + ok(await waitProcessGone(tree.harperPid, 5000), `Harper ${tree.harperPid} should die on runner SIGHUP`); + ok(await waitProcessGone(tree.descendantPid, 5000), `descendant ${tree.descendantPid} should die on runner SIGHUP`); + ok(await waitForPortsFree('127.0.0.1', tree.ports, 5000, 50), 'Harper tree ports should be reusable'); + } finally { + await cleanupFakeHarperTree(tree); + } +}); + +test('concurrent runners share one monitor', { skip: !isPosix }, async () => { + const monitorDir = mkdtempSync(join(tmpdir(), 'harper-it-monitor-')); + let first: RunningFakeHarperTree | undefined; + let second: RunningFakeHarperTree | undefined; + try { + first = await startFakeHarperTree({ monitorDir }); + const monitorPid = await waitForMonitor(monitorDir); + second = await startFakeHarperTree({ monitorDir }); + + const registry = await readMonitorRegistry(monitorDir); + strictEqual(registry.monitor?.pid, monitorPid, 'the second runner should reuse the running monitor'); + strictEqual(registry.instances.length, 2, 'both instances should be registered with that one monitor'); + + // Two runners registering at the same instant can both decide a monitor is needed; the + // loser must stand down rather than become a second reaper. + const redundant = spawn(process.execPath, [MONITOR_SCRIPT, MONITOR_ARGV_MARKER], { + env: { ...process.env, HARPER_INTEGRATION_TEST_MONITOR_DIR: monitorDir }, + stdio: 'ignore', + }); + const exited = await Promise.race([ + once(redundant, 'exit').then(([code]) => code as number | null), + sleep(5000).then(() => 'timeout' as const), + ]); + forceKill(redundant.pid); + strictEqual(exited, 0, 'a redundant monitor should exit immediately'); + strictEqual((await readMonitorRegistry(monitorDir)).monitor?.pid, monitorPid, 'it must not take over the registry'); + } finally { + if (first) await cleanupFakeHarperTree(first); + if (second) await cleanupFakeHarperTree(second); + forceKill((await readMonitorRegistry(monitorDir)).monitor?.pid); + rmSync(monitorDir, { recursive: true, force: true }); + } +}); + +test('instances started from two worker threads of one runner are registered and reaped separately', { skip: !isPosix }, async () => { + // Worker threads share process.pid and each gets its own copy of the registry module, so an id + // built from pid + a module-local counter collides on their first starts — and registration + // replaces same-id records, leaving one of two live instances with nothing to reap it. + const monitorDir = mkdtempSync(join(tmpdir(), 'harper-it-monitor-')); + const runner = spawn(process.execPath, [fixtures['worker-runner.mjs']], { + env: { + ...process.env, + HARPER_WORKER_SCRIPT: fixtures['worker-harper.mjs'], + HARPER_LIFECYCLE_URL: new URL('../src/harperLifecycle.ts', import.meta.url).href, + HARPER_FAKE_SCRIPT: fixtures['process-tree.cjs'], + HARPER_INTEGRATION_TEST_MONITOR: 'on', + HARPER_INTEGRATION_TEST_MONITOR_DIR: monitorDir, + HARPER_INTEGRATION_TEST_MONITOR_INTERVAL_MS: '200', + HARPER_INTEGRATION_TEST_MONITOR_REAP_GRACE_MS: '500', + HARPER_INTEGRATION_TEST_MONITOR_IDLE_MS: '60000', + HARPER_INTEGRATION_TEST_INSTANCE_MAX_LIFETIME_MS: '', + }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + const pids: number[] = []; + const ports: number[] = []; + try { + const ready = await waitForMatches(runner, /worker-ready:\d+:(\d+):(\d+):(\d+):(\d+)/g, 2, 20000); + for (const worker of ready) { + pids.push(Number(worker[1]), Number(worker[2])); + ports.push(Number(worker[3]), Number(worker[4])); + } + await waitForMonitor(monitorDir); + const registry = await readMonitorRegistry(monitorDir); + strictEqual(registry.instances.length, 2, 'each worker should get its own registry record'); + strictEqual(new Set(registry.instances.map((instance) => instance.id)).size, 2, 'their ids must not collide'); + + runner.kill('SIGKILL'); + await once(runner, 'exit'); + for (const pid of pids) ok(await waitProcessGone(pid, 15000), `${pid} should be reaped with its runner`); + ok(await waitForPortsFree('127.0.0.1', ports, 5000, 50), 'both trees should release their ports'); + } finally { + forceKill(runner.pid); + for (const pid of pids) forceKill(pid); + forceKill(await waitForMonitor(monitorDir, 5000).catch(() => undefined)); + rmSync(monitorDir, { recursive: true, force: true }); + if (ports.length) await waitForPortsFree('127.0.0.1', ports, 2000, 50); + } +}); + +test('an instance that outlives its lifetime budget is reaped while its runner is still alive', { skip: !isPosix }, async () => { + // Owner death is the sharp signal; this budget is the backstop for a runner PID recycled by + // some other long-lived process, which would otherwise look alive forever. + const tree = await startFakeHarperTree({ maxLifetimeMs: '250' }); + try { + ok(await waitProcessGone(tree.harperPid, 10000), `overdue Harper ${tree.harperPid} should be reaped`); + strictEqual(tree.runner.exitCode, null, 'the owning runner should be untouched'); + ok(await waitForPortsFree('127.0.0.1', tree.ports, 5000, 50), 'Harper tree ports should be reusable'); + } finally { + await cleanupFakeHarperTree(tree); + } +}); + +test('a reap in flight is finished even though its runner is alive to see the leader exit', { skip: !isPosix }, async () => { + // The leader's exit is all the live runner can see of this reap, and removing the record on it + // dropped the escalation while the TERM-ignoring descendant kept the port. + const tree = await startFakeHarperTree({ maxLifetimeMs: '250', descendantIgnoresTerm: true }); + try { + ok(await waitProcessGone(tree.harperPid, 10000), `overdue Harper ${tree.harperPid} should be reaped`); + ok( + await waitProcessGone(tree.descendantPid, 15000), + `its TERM-ignoring descendant ${tree.descendantPid} should still be escalated to SIGKILL` + ); + strictEqual(tree.runner.exitCode, null, 'the owning runner should be untouched'); + ok(await waitForPortsFree('127.0.0.1', tree.ports, 5000, 50), 'Harper tree ports should be reusable'); + } finally { + await cleanupFakeHarperTree(tree); + } +}); + +test('a group that outlives its leader stays reapable until its runner dies', { skip: !isPosix }, async () => { + // Nothing is due to be reaped yet, and the runner's own handlers dropped the leader from their + // live set the moment it exited, so the record is all that is left to remember the survivors. + const tree = await startFakeHarperTree(); + try { + await waitForMonitor(tree.monitorDir); + process.kill(tree.harperPid, 'SIGKILL'); + ok(await waitProcessGone(tree.harperPid, 5000), `leader ${tree.harperPid} should be gone`); + await sleep(1000); // several scan intervals + strictEqual( + (await readMonitorRegistry(tree.monitorDir)).instances.length, + 1, + 'the record should survive the leader while the group still holds a port' + ); + strictEqual(await isPortFree('127.0.0.1', tree.ports[1]), false); + + tree.runner.kill('SIGKILL'); + await once(tree.runner, 'exit'); + ok(await waitProcessGone(tree.descendantPid, 15000), `descendant ${tree.descendantPid} should be reaped with its runner`); + ok(await waitForPortsFree('127.0.0.1', tree.ports, 5000, 50), 'its port should come back'); + } finally { + await cleanupFakeHarperTree(tree); + } +}); + +test('a record whose leader PID has been reused is dropped, not signalled', { skip: !isPosix }, async () => { + // A PID running something else means our group ended and its id was handed out again. + const monitorDir = mkdtempSync(join(tmpdir(), 'harper-it-monitor-')); + const bystander = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: true, stdio: 'ignore' }); + const exitedOwner = spawn(process.execPath, ['-e', '']); + await once(exitedOwner, 'exit'); + let monitor: ChildProcess | undefined; + try { + await writeFileAsync( + join(monitorDir, 'registry.json'), + JSON.stringify({ + instances: [ + { + id: 'stale', + pid: bystander.pid, + // Not the bystander's start time: this record describes whatever held the PID before it. + startTime: 'Mon Jan 1 00:00:00 2001', + owner: { pid: exitedOwner.pid, startTime: 'Mon Jan 1 00:00:00 2001' }, + registeredAt: Date.now(), + expiresAt: Date.now() + 3600000, + }, + ], + }) + ); + monitor = spawn(process.execPath, [MONITOR_SCRIPT, MONITOR_ARGV_MARKER], { + env: { + ...process.env, + HARPER_INTEGRATION_TEST_MONITOR_DIR: monitorDir, + HARPER_INTEGRATION_TEST_MONITOR_INTERVAL_MS: '200', + HARPER_INTEGRATION_TEST_MONITOR_REAP_GRACE_MS: '200', + HARPER_INTEGRATION_TEST_MONITOR_IDLE_MS: '60000', + }, + stdio: 'ignore', + }); + await waitForMonitor(monitorDir); + await sleep(1500); + strictEqual(bystander.signalCode, null, 'an unrelated process group must not be signalled for a recycled PID'); + strictEqual(bystander.exitCode, null, 'the bystander should still be running'); + strictEqual((await readMonitorRegistry(monitorDir)).instances.length, 0, 'the stale record should be pruned instead'); + } finally { + forceKill(monitor?.pid); + forceKill(bystander.pid); + rmSync(monitorDir, { recursive: true, force: true }); + } +}); + +test('a process identity does not depend on the environment that read it', { skip: !isPosix }, () => { + // `ps -o lstart=` renders in the caller's timezone and locale. Two runners configured differently + // would record different strings for one process and read each other's live records as PID reuse. + const previousTz = process.env.TZ; + try { + process.env.TZ = 'UTC'; + const asUtc = readProcessStartTimes([process.pid]).get(process.pid); + process.env.TZ = 'America/Denver'; + const asDenver = readProcessStartTimes([process.pid]).get(process.pid); + ok(asUtc, 'this host should report a start time'); + strictEqual(asUtc, asDenver); + } finally { + restoreEnv('TZ', previousTz); + } +}); + +test('a reap cannot be widened into a broadcast by a bad record', { skip: !isPosix }, () => { + // The registry is on-disk state outside this process, so a corrupt or planted record can name + // PID 1 — and `kill(-1)` reaches every process the monitor may signal. + const realKill = process.kill; + const attempted: number[] = []; + process.kill = ((pid: number) => { + attempted.push(pid); + return true; + }) as typeof process.kill; + try { + signalProcessGroup(1, 'SIGKILL'); + signalProcessGroup(0, 'SIGTERM'); + signalProcessGroup(-1, 'SIGKILL'); + } finally { + process.kill = realKill; + } + strictEqual(attempted.length, 0, `no signal should be sent for these group ids, got ${attempted.join()}`); +}); + +test('the monitor shuts down once no instances remain', { skip: !isPosix }, async () => { + const tree = await startFakeHarperTree({ idleMs: '500' }); + try { + const monitorPid = await waitForMonitor(tree.monitorDir); + tree.runner.kill('SIGKILL'); + await once(tree.runner, 'exit'); + ok(await waitProcessGone(tree.harperPid, 10000), `Harper ${tree.harperPid} should be reaped`); + ok(await waitProcessGone(monitorPid, 10000), `monitor ${monitorPid} should exit once the registry empties`); + strictEqual((await readMonitorRegistry(tree.monitorDir)).monitor, undefined, 'it should release its slot on the way out'); + } finally { + await cleanupFakeHarperTree(tree); + } +}); + +test('a registry write survives the abrupt death of its writer', { skip: !isPosix }, async () => { + const monitorDir = mkdtempSync(join(tmpdir(), 'harper-it-monitor-')); + const registryPath = join(monitorDir, 'registry.json'); + const writer = spawn(process.execPath, [fixtures['registry-writer.mjs']], { + env: { + ...process.env, + HARPER_REGISTRY_URL: new URL('../src/harperInstanceRegistry.ts', import.meta.url).href, + HARPER_INTEGRATION_TEST_MONITOR_DIR: monitorDir, + }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + /** Parses the registry the way a monitor would, failing loudly instead of reading torn JSON as empty. */ + async function readCompleteRegistry(context: string): Promise { + const raw = await readFile(registryPath, 'utf-8'); + try { + return JSON.parse(raw) as InstanceRegistry; + } catch { + throw new Error(`Torn registry ${context}: ${raw.length} bytes, ending ${JSON.stringify(raw.slice(-40))}`); + } + } + try { + await waitForMatch(writer, /writer-ready/); + // Read while the writer rewrites the file back to back. A truncate-in-place write is visible + // here as an empty or half-written file, which `readRegistryFile` maps to an empty registry — + // after which the next writer persists that emptiness, dropping every runner's reap targets. + let reads = 0; + for (const deadline = Date.now() + 500; Date.now() < deadline; reads++) { + const observed = await readCompleteRegistry(`while the writer was running (read ${reads})`); + strictEqual(observed.instances[0]?.id, 'seeded', 'a visible registry must never lose its instances'); + } + ok(reads > 20, `expected to catch many writes in flight, only managed ${reads} reads`); + + // SIGKILL mid-write: whatever is on disk afterwards must still be one of the two complete + // shapes the writer alternates between, never a partial one. + writer.kill('SIGKILL'); + await once(writer, 'exit'); + const survivor = await readCompleteRegistry('after the writer was killed'); + strictEqual(survivor.instances[0]?.id, 'seeded', 'a killed writer must leave the reap target behind'); + ok( + survivor.instances.length === 1 || survivor.instances.length === 2, + `expected one of the writer's two complete shapes, got ${survivor.instances.length} instances` + ); + } finally { + forceKill(writer.pid); + rmSync(monitorDir, { recursive: true, force: true }); + } +}); + +test('a superseded lock holder does not release the lock that replaced it', { skip: !isPosix }, async () => { + const monitorDir = mkdtempSync(join(tmpdir(), 'harper-it-monitor-')); + const previousDir = process.env.HARPER_INTEGRATION_TEST_MONITOR_DIR; + process.env.HARPER_INTEGRATION_TEST_MONITOR_DIR = monitorDir; + try { + await withRegistryLock(async () => { + // What a critical section that overran the stale timeout comes back to: another process + // reclaimed the lock and is now inside the section itself. + await writeFileAsync(join(monitorDir, 'registry.lock'), 'another-holder'); + }); + strictEqual( + await readFile(join(monitorDir, 'registry.lock'), 'utf-8'), + 'another-holder', + 'releasing must not evict the holder that superseded us — that admits a third process alongside it' + ); + } finally { + restoreEnv('HARPER_INTEGRATION_TEST_MONITOR_DIR', previousDir); + rmSync(monitorDir, { recursive: true, force: true }); + } +}); + +test('an unreadable registry is reported rather than silently read as empty', { skip: !isPosix }, async () => { + const monitorDir = mkdtempSync(join(tmpdir(), 'harper-it-monitor-')); + const previousDir = process.env.HARPER_INTEGRATION_TEST_MONITOR_DIR; + process.env.HARPER_INTEGRATION_TEST_MONITOR_DIR = monitorDir; + try { + // An empty registry is the one answer that must never be invented: the caller writes it back, + // and every live instance on the machine loses the record that would have got it reaped. + await writeFileAsync(join(monitorDir, 'registry.json'), '{"instances":[{"id":"trunc'); + await rejects(readRegistryFile(), /is not valid JSON/); + await writeFileAsync(join(monitorDir, 'registry.json'), '{"instances":null}'); + await rejects(readRegistryFile(), /has no instance list/); + } finally { + restoreEnv('HARPER_INTEGRATION_TEST_MONITOR_DIR', previousDir); + rmSync(monitorDir, { recursive: true, force: true }); + } +}); + +test('neither startup watchdog can time out a Harper that already reported ready', { skip: !isPosix }, async () => { + const monitorDir = mkdtempSync(join(tmpdir(), 'harper-it-monitor-')); + const previousDir = process.env.HARPER_INTEGRATION_TEST_MONITOR_DIR; + const previousEnabled = process.env.HARPER_INTEGRATION_TEST_MONITOR; + process.env.HARPER_INTEGRATION_TEST_MONITOR_DIR = monitorDir; + process.env.HARPER_INTEGRATION_TEST_MONITOR = 'on'; + // Held for longer than maxMs but well inside the stale timeout, so registration blocks on a lock + // that is legitimately someone else's — what a machine full of concurrent runners looks like. + const lockPath = join(monitorDir, 'registry.lock'); + await writeFileAsync(lockPath, 'other-runner'); + let started: Awaited> | undefined; + try { + const starting = runHarperCommand({ + args: [], + env: {}, + completionMessage: 'successfully started', + // Both windows expire while registration is still blocked on the lock: the absolute one + // counting from launch, the idle one if the post-readiness log line re-arms it. + harperBinPath: fixtures['ready-then-log.cjs'], + timeoutMs: 2000, + maxMs: 2000, + hostname: '127.0.0.1', + }); + await sleep(3000); + rmSync(lockPath, { force: true }); + started = await starting; + match(started.stdout, /successfully started/); + doesNotMatch(started.stdout, /post-readiness log line/, 'startupOutput is a snapshot taken at readiness'); + } finally { + if (started) started.process.kill('SIGKILL'); + restoreEnv('HARPER_INTEGRATION_TEST_MONITOR_DIR', previousDir); + restoreEnv('HARPER_INTEGRATION_TEST_MONITOR', previousEnabled); + forceKill(await waitForMonitor(monitorDir, 5000).catch(() => undefined)); + rmSync(monitorDir, { recursive: true, force: true }); + } +}); + +test('killHarper waits for Harper shutdown and does not keep the runner alive', { skip: !isPosix }, async () => { + const tree = await startFakeHarperTree({ mode: 'teardown' }); + try { + await waitForMatch(tree.runner, /teardown-complete:true/); + await Promise.race([ + once(tree.runner, 'exit'), + sleep(3000).then(() => { + throw new Error('Runner stayed alive after supervised teardown'); + }), + ]); + ok(await waitForPortsFree('127.0.0.1', tree.ports, 2000, 50), 'ports should be free when killHarper resolves'); + } finally { + await cleanupFakeHarperTree(tree); + } +}); + test('runHarperCommand rejects when the process exits non-zero', async () => { await rejects( runHarperCommand({ @@ -186,8 +898,6 @@ test('runHarperCommand rejects when the process exits non-zero', async () => { // leader), so killHarper's group signal (negative PID) targets the whole tree. Windows has no // process groups — killHarper uses `taskkill /T` there — and no real POSIX signals, so the // signal-specific assertions below are guarded to POSIX. -const isPosix = process.platform !== 'win32'; - test('killHarper terminates a process that exits on SIGTERM, before the grace deadline', async () => { const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: isPosix }); await once(child, 'spawn');