Release a purged transaction-log segment's mapping instead of refusing to read it

[kriszyp]

purgeLogs() removes a retired segment's pathname, but an iterator that already holds its immutable mapping can safely finish reading that committed history. This PR keeps that behavior while making the mapping collectible and ensuring readers resume at the next real survivor after a purged run.

What changed

• TransactionLog._currentLogBuffer is weak, matching the per-segment cache, so a long-lived iterator no longer pins a deleted mapping after it advances.
• nextReadableLogBuffer() walks the registered successor set instead of deriving a jump from the newest contiguous run. It crosses multiple holes without skipping an intervening survivor, and stops rather than skipping a registered segment that is only temporarily unmappable.
• Read probes use TransactionLogFile::openExisting(), backed by O_RDWR without O_CREAT on POSIX and OPEN_EXISTING on Windows. A segment removed between discovery and the OS open therefore remains absent instead of being recreated as a header-only ghost. TransactionLogStore::load()'s two startup opens (activating the surviving current segment, and the backward recovery-boundary scan) now use the same non-creating open, closing the two remaining ghost-recreate paths this round's review found.
• A mapping carries the append-owned readable extent, exposed through a type-tagged N-API getter. readableExtent() requires that exact bound, so purge cannot make a reader infer authority from frame-shaped bytes left by a failed append or from unused mapped capacity. publishReadableExtentLocked(), which runs on every append, now reads the live map through a raw pointer instead of copying the shared_ptr, so the commit hot path pays no refcount traffic in the common case.
• An all-segment purge advances a store generation and clears old flush correlations. writeFlushedPosition() reopens and verifies txn.state by pathname, preventing a pre-purge callback or an unlinked/replaced stream from publishing stale retention state.
• Purge-vs-mapping is now tested as a contract rather than a platform outcome: a reader keeps every entry it mapped and retention reclaims the segment once nothing maps it, and an unwritable store directory forces the refused-unlink branch — the segment stays registered, the log.warn line is asserted end to end through the public global-event API, and the next run reclaims it. lastRemoveError is read through a fileMutex-held accessor rather than unlocked.
• README.md's memory-map section describes the actual ownership split: only the current file's map is held natively; a frozen (rotated) file's map is weakly cached on POSIX and survives solely through JS Buffer references, so stats.memory.activeMaps counts only the current file there. On Windows the weak-for-frozen optimization does not apply — every frozen read re-pins the mapping strongly for the file's life, and activeMaps/mappedBytes count it.

For the human reviewer

1. A purged segment intentionally remains readable through a mapping acquired before purge. The mapping is immutable committed history; the fix releases it after the iterator advances rather than invalidating it mid-read. The open decision underneath it is the reader-gap policy: a subscription that falls behind retention now skips the deleted run and keeps yielding with no signal — no throw, no log.warn, no marker on the resumed entry — so a follower that lost segments 6–8 believes its history is contiguous. Stopping instead is the wedge this PR fixes, so the choice is between silent skip and a new reader-visible signal Harper would have to handle; it is cheaper to decide now than to retrofit.
2. The hole policy intentionally distinguishes absence from temporary unavailability: absent registered paths are skipped through the successor lookup, while a segment that still has a durable extent but cannot currently map causes the iterator to stop and retry later.
3. purgeLogs({ destroy: true }) can restart segment numbering while a JavaScript buffer cache from the prior store generation survives. That pre-existing cache-identity problem is outside this PR; Harper does not use destructive log-store purge in production.
4. Windows purge-vs-mapping is unexplained, and is no longer asserted. A Windows-only test here claimed a live reader mapping refuses the unlink; it passed on all three Windows Node jobs at 9a8606a8 and failed on all three (both attempts) at c8e790a0, with nothing in between changing how long the mapping lives. Only the convergence is asserted cross-platform now; the first-run deletion is asserted on POSIX only. Worth a word if you know which Windows rule applies here.
5. Real, deliberately not fixed here — a refused unlink of the current segment strands a reader on a stale mapping. removeFileLocked() drops its strong memoryMap and closes the fd before it knows whether the unlink succeeded. If purgeLogs({ destroy: true }) fails to delete the current segment (EACCES, or a Windows sharing violation) while a bound transaction keeps the store alive, the next append reopens the file and builds a new MemoryMap; a reader still holding the old buffer never sees the new entries, because its extent and overlay stop advancing. Confirmed independently by codex, cursor-grok and the harper-domain adjudicator across five rounds now (all "Scope: in-scope"), but reachable only through purgeLogs({ destroy: true }), which Harper does not run in production, and squarely inside the destroy path this PR has deliberately left untouched since an earlier scope correction — every attempt at hardening that path in earlier rounds kept surfacing a new destroy-path race and was reverted. Filing this as a separate issue rather than reopening that path here.
6. Pre-existing, narrow — txn.state's 8-byte record is overwritten in place with no fsync, no temp-then-rename, and no checksum. A crash mid-write can leave a torn LogPosition (e.g. a newer sequence number paired with an older offset); retention would then trust it as the flushed boundary and could delete a segment whose tail never reached RocksDB. This predates this PR (the old code overwrote the same record the same way) and the practical exposure is narrow — a torn write inside one 8-byte range is rare on filesystems with ordered/journaled metadata — but it is a real gap in the hardening this PR added around txn.state. Reviewed and left alone rather than restructuring that file's durability inside this PR; a proper fix is write-temp-then-atomic-rename.
7. Real, new this round, deliberately not fixed here — a partial purgeLogs({ destroy: true }) can leave a stale flushed-state generation authoritative. The generation bump, correlation-ring reset, and txn.state clear only run when the destructive purge empties sequenceFiles entirely. If one segment's unlink is refused (the same EACCES/Windows-sharing-violation case as item 5) while a later, already-flushed segment is successfully deleted, the registered set stays non-empty and that whole block is skipped — txn.state survives naming the deleted segment's sequence number. After a restart the refused segment becomes current, a later append can reuse the now-vacant sequence number the stale txn.state still names, and retention can then delete that reused segment as "already flushed" before it reaches RocksDB. Confirmed independently by codex and the harper-domain adjudicator (severity settled at minor after two rounds: the trigger needs a destroy, a refused unlink below the flushed segment, continued writes, and a restart before the next flush — narrow, and again destroy: true-only). The fix is a one-branch policy choice — stop the destructive purge at the first refused unlink (restoring main's effective behavior, since main threw there), or invalidate the generation whenever the segment txn.state names is removed regardless of whether the whole set emptied — and belongs with item 5 in the same follow-up rather than reopening the destroy path here.
8. transaction-log-reader.ts:419-429 caches logBuffer.size after an advance without re-checking latestLogId > logBuffer.logId, unlike its sibling at :289-296. If the advance lands on the still-current segment, a HEADER_SIZE-sized snapshot is stamped onto the shared _logBuffers entry and every entry later written to that segment is skipped once it rotates. This is pre-existing — main has the identical shape with getLogFileSize() in place of readableExtent() — so it is left for a separate issue rather than widened into this PR; flagging it here because it is the same silent-entry-loss class this PR is about.
9. Pre-existing, stats-only — memory.overlayBytes over-reports after a rotated segment's map is weakly released. collectStats() sums every registered file's lastOverlaySize regardless of whether native still holds the strong map, while mappedBytes/activeMaps (the adjacent counters) are correctly gated on it. Once downgradeMapToFrozen() drops the strong reference (pre-existing on main), the overlay watermark can describe an already-unmapped region until retention unregisters the segment. This branch's weak _currentLogBuffer makes the staleness surface more often in practice. Stats/diagnostics only — no data-safety impact. The domain adjudicator ruled it pre-existing and out of this diff's scope; left for separate follow-up rather than fixed here.

Rebase / maintenance history (this session)

origin/main advanced twice during this maintenance pass, each time landing its own new AGENTS.md invariant that collided with this branch's purge invariant number — the same renumbering dance as every prior rebase round:

• First rebase: origin/main had merged PR Serialize database destruction with concurrent opens #787 ("Serialize database destruction with concurrent opens") among 15 other commits, adding invariants 26–28. Rebased cleanly (git merge-tree --write-tree origin/main HEAD matched HEAD^{tree}); the purge invariant moved 25→29, databaseFlushed 26→30. Re-grepped the whole tree afterward per this PR's own recorded lesson and fixed three stale in-code "invariant 25" cross-references the collision left behind.
• Second rebase, mid-review: origin/main advanced again (PR Keep user shared buffers alive for the life of the column family instead of freeing them on garbage collection #868, "Keep user shared buffers for the life of the column family") and added its own invariant 29, colliding a second time. Rebased cleanly again; purge moved 29→30, databaseFlushed 30→31. Same whole-tree re-grep, same fix pattern.

No semantic conflict either time — git merge-tree --write-tree origin/main HEAD equals HEAD^{tree} at the final head; origin/main is a full ancestor.

Independent review (this session)

Four rounds: round 1 (full, on the first rebase) — codex+gemini+cursor-grok+harper-domain re-raised the standing findings (items 5, 6, 8 above) plus two genuinely new items from a fresh read: item 7 above (partial-destroy stale flushed-state generation, new this session) and two factually stale in-code comments a prior rebase's renumbering had left behind (a test comment citing the wrong invariant number entirely, and a corruptFrame() docstring naming a function the code no longer calls) — both fixed. Round 2 (delta) confirmed both fixes and caught a mis-attribution in my own fix, which I corrected. The second rebase forced round 3 (full, per this PR's standing rule that a rebase re-review is always full): re-confirmed the same standing set with no new actionable findings; a Gemini "blocker" (unlocked shared_ptr read) was refuted directly from source (fileMutex is held by every caller). Round 4 (delta, on a trailing comment trim) converged: no new findings, cursor-grok's own leg failed on a formatting error in its response (not a coverage gap — codex+gemini+domain all completed). Receipt independent=true, reviewers=codex,gemini,harper-domain, covers the pushed head exactly.

Verification

• pnpm build clean; pnpm check (type-check, oxlint, oxfmt) clean throughout
• pnpm test:native — 243 passed
• node --expose-gc vitest run (transaction-log, backup-transaction-logs, destroy, transaction-log-crash-recovery) — 162 passed, 0 failed, on the final pushed head
• Full pnpm test — 1059 passed / 10 skipped / 0 failed (71 files), on the final pushed head
• Merge integrity: origin/main is an ancestor of HEAD; git merge-tree --write-tree origin/main HEAD equals HEAD^{tree} at the pushed head
• Independent review: 4 rounds this session (codex + gemini + cursor-grok + harper-domain), independent=true, receipt keyed exactly to the pushed head

Refs HarperFast/harper#2337

Complexity: moderate

_{Generated by GPT-5 Codex; maintained by Claude Sonnet 5}

Origin — the dispatch brief this PR was written from

Release a purged transaction-log segment's mapping instead of refusing to read it

Maintain #820 (Release a purged transaction-log segment's mapping instead of refusing to read it) on branch fix/txnlog-purge-read-coherence. Read the current PR head, mergeability, and latest check runs before acting; the dispatch observation may be stale. First read the remote head for refs/heads/fix/txnlog-purge-read-coherence without updating local refs and require it to equal this task's observed head 5396e00; if it differs, stop with needs-input because remote history changed after dispatch. Then fetch only refs/heads/main from origin into refs/remotes/origin/main. If Git reports a paused rebase for this task's exact generation branch, require that rebase's recorded original head to equal 5396e00 and its recorded onto commit to equal the fetched origin/main tip; stop with needs-input on either mismatch. Only then resume it without repeating the local-HEAD ancestry check or starting another rebase. Otherwise record the remote SHA, verify it is an ancestor of local HEAD, and rebase onto the fetched base. If Context carries companion instructions, follow them exactly for every named gitlink; they override ordinary conflict handling. Preserve both sides' intent for every other conflict. Stop with needs-input on semantic conflicts. Force-with-lease is authorized only for this rebase. Immediately before pushing, re-read the PR's base ref and body plus every companion PR head/state named in Context. The base must still be main, and the declarations and submodule bindings must still match Context. If an open companion advanced, update its named gitlink to the new exact head and rerun relevant tests. If it merged, point the named gitlink at the companion repository's current default-branch tip after verifying that tip contains the merge, then rerun relevant tests. If it closed without merging, stop with needs-input and do not publish its abandoned head. If the base, declaration, binding or any other companion state changed ambiguously, stop with needs-input. Then push the rebased head with git push --force-with-lease=refs/heads/fix/txnlog-purge-read-coherence:EXPECTED_HEAD_SHA origin HEAD:refs/heads/fix/txnlog-purge-read-coherence, substituting the recorded SHA. If the remote head changes, stop with needs-input; never overwrite intervening remote work. Do this before evaluating CI. Then inspect CI for the resulting current PR head, not old-head failures. Wait for relevant pending checks, diagnose remaining failures, fix them, run relevant tests, and push in THIS task. Do not create a separate CI-fix task. Stop with needs-input for judgment calls or an unavailable CI result; report exactly what was verified. No merge is authorized.

_{Dispatch: task pr-maint-68bd037170be49b2a01e194607ecae40 · queued by automation · ran by claude/sonnet/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=gemini,claude,codex,cursor-grok; adjudicated=domain; blocked=claude(fallback)(out-of-budget); declined=cursor-composer; rounds=39; full=2 @ 3a6242e}

_{Human-Review-Need: 4 (decisions: destroy-refusal-policy, mapping-carried-extent, windows-first-run-unexplained, txn-state-reopen-per-flush, gc-driven-map-release) @ 3a6242e}

[gemini-code-assist]

Code Review

This pull request implements a robust cache revalidation mechanism for JS memory-map caches against the native store's purge epoch, preventing stale reads of purged transaction-log segments. It also hardens databaseFlushed() to verify the physical existence of txn.state on disk rather than relying solely on the stream's open status. The feedback recommends using performance.now() instead of Date.now() in tests to ensure a monotonic clock, and replacing new Uint32Array with readUInt32LE on read buffers to avoid potential alignment errors.

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	24.17K ops/sec	41.37	39.83	606.92	0.112	120,863
🥈 rocksdb	2	11.54K ops/sec	86.65	83.52	23,942.922	0.967	57,707

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	28.49K ops/sec	35.09	33.91	548.999	0.107	142,471
🥈 rocksdb	2	11.75K ops/sec	85.09	83.27	555.685	0.048	58,761

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	22.62K ops/sec	44.20	38.63	1,811.706	0.280	113,119
🥈 rocksdb	2	15.60K ops/sec	64.11	56.50	1,089.161	0.121	77,985

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	348.43 ops/sec	2,870.008	68.30	67,406.572	14.23	698
🥈 lmdb	2	26.88 ops/sec	37,197.622	460.996	1,168,454.918	136.315	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	36.09K ops/sec	27.71	11.62	20,513.668	0.843	180,439
🥈 lmdb	2	444.17 ops/sec	2,251.415	113.189	25,470.298	1.54	2,221

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	616.66K ops/sec	1.62	1.40	4,942.504	0.204	3,083,276
🥈 lmdb	2	460.73K ops/sec	2.17	1.21	9,972.754	0.572	2,303,650

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	829.25 ops/sec	1,205.915	1,047.263	2,064.763	0.302	1,659
🥈 lmdb	2	1.16 ops/sec	861,597.44	785,294.281	924,747.607	3.43	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	22.10K ops/sec	45.25	29.19	20,422.185	2.08	44,201
🥈 lmdb	2	816.95 ops/sec	1,224.067	103.125	14,296.427	5.54	1,634

---

Results from commit 8c2b320

[cb1kenobi]

The new commit only renumbers AGENTS.md invariants after the rebase onto main. Prior blocking issues remain fixed: readers walk registered successors instead of jumping past a survivor, read probes cannot recreate a deleted segment, purged reads use the mapping-carried append-owned extent, and destroy invalidates stale flush correlations. No new blocking defect showed up on the changed lines.

—
Reviewed 162cf81

[cb1kenobi]

The new commit only corrects three invariant references in comments after the rebase. It introduces no blocking defect on changed lines.

—
Reviewed 8216e2a

[cb1kenobi]

The merge of main onto this head does not reintroduce the fixed purge bugs or add a new one. Readers still walk registered successors, read probes still cannot recreate a deleted segment, and flushed-state writes still drop stale pre-purge correlations. No blocking defect showed up on the changed lines.

—
Reviewed eee417f