Record a write-ahead audit retention floor, raised before every prune (infrastructure for #2448)

[dawsontoth]

Records a write-ahead audit retention floor for every database — the point at or after which no prune has removed audit history — so that Table.subscribe can tell a complete startTime catch-up from a silently truncated one (#2448). This PR is the write side only: the floor is recorded, verified, and raised first by every path that prunes. Nothing reads it publicly.

Re-scoped 2026-09-10, per kriszyp's review. The branch originally shipped Table.oldestRetainedAuditTime() as a public accessor with a documented contract. That accessor is removed. Three reasons, all his: it had no caller except #2448; a public single-number floor binds a contract that a per-origin resume vector could not honor; and it created a second resume-validity path where replication already checks inside the operation (shouldForceBaseCopyForRetention). So the check belongs inside Table.subscribe itself — one path, and the only shape where the floor cannot move between being read and being acted on. Nothing consumes the floor in this PR — no resume path reads it, and silently truncated catch-up is unchanged until #2448 calls getAuditFloor(auditStore) (internal, resources/auditStore.ts) from inside subscribe. HarperFast/documentation#666 is closed and the section #660 added is being removed; consumer-facing docs land with #2448.

The gap this is infrastructure for is live today: Table.subscribe's startTime replay begins wherever the audit log now begins, and MQTT durable sessions hand it a persisted per-topic startTime on every resume, so a client offline longer than logging.auditRetention loses messages — QoS 1/2 included — with no signal at all.

Making the floor trustworthy was most of the work. All five paths that prune audit history now raise the floor before removing anything, monotonically, in a store transaction whose commit is actually verified; only one path did before, and it recorded afterwards, so a crash in between left a floor certifying history that was already gone. The floor lives under a new key whose presence is the trust marker: a store without one has retention history that cannot be accounted for, so it gets a one-time resync epoch rather than a permissive baseline. Untrustworthy metadata resolves to Infinity rather than to a number, so a consumer spelling the check as cursor < floor cannot read corrupt bytes as safe.

Two fail-open paths this uncovered are worth naming, because neither depends on the floor existing. A prune bound of NaN, a negative, or -0 was accepted by the range — audit keys are raw float64, so those values set the sign bit or the quiet-NaN pattern and sort above every real timestamp, making getRange({ start: 1, end: NaN }) span the whole log. delete_transaction_logs_before reached exactly that through Number.parseInt on a non-numeric timestamp (and '9999999999999oops' parsed to a year-2286 bound that purged everything), so it now validates at its own boundary and reports a 400, and the bound guard itself throws rather than declining silently. A finite bound above everything reachable — Infinity, Date.now() * 1000, a bare '9999999999999' — is the same defect by degree: recorded verbatim, it pins the whole database's floor in the far future forever, since a floor only rises. boundedAuditPruneEnd clamps every operator-supplied bound to just above the newest key in the log, and the prune uses that same clamped value as its range end, so it cannot remove an entry the floor does not cover. And RocksDB's transactionSync returns undefined on a swallowed abort rather than throwing; the floor write requires an explicit true.

The branch originally retired getLastRemoved/updateLastRemoved, which had no consumers and did not work: both read through the audit store's value decoder, so on LMDB the raw eight float bytes decoded as an audit entry and the function returned a stale module global — measured, it stored 1234567.5 and returned 1 — while on RocksDB the same call threw in msgpack decode. That retirement was reverted when this branch merged #2338, which hardened the marker's write path and added five tests around it. Both markers are live now and the code says why they are separate keys: last-removed records where the LMDB retention loop got to, after the fact, while the floor is written ahead of all five prune paths with its commit verified.

For the human reviewer

Judgment calls, mirroring the decision slugs in the Human-Review-Need footer. Mechanics went through five cross-model rounds (three full, two delta; final adjudicated severity nit) and human review; these are the scope and contract choices that remain yours.

1. Write side first, read side in Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 — re-scoped. Originally "ship the public primitive before its consumer," which kriszyp's review overturned: there is no public primitive now. What remains separable is that this PR touches every prune path plus bootstrap and the bridge, while Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 touches subscribe — which also serves WebSocket/SSE reconnects and sourcedFrom caching-table subscriptions, so its failure mode for a fallen-off cursor is its own design question. Nothing here presumes how Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 answers it.

2. One database-scoped floor, not per-table. The audit store is per-database and entries carry a tableId, so an exact per-table floor needs a scan for that table's oldest entry. The cost: deleteHistory on one table raises the floor for every sibling, forcing resyncs of history that still exists. Cheap to change now, costly once Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 depends on the meaning.

3. Infinity is the unknown sentinel — internal representation now, so no longer a public-type concern. It makes both cursor >= floor and cursor < floor fail closed, which is why it beat undefined/null/throw. It is also absorbing (a floor at Infinity never comes down), which is why every operator bound is clamped before it can be recorded. Open (Chris's thread): with every caller now clamped, nothing can pass Infinity to raiseAuditFloor, so accepting it only preserves a footgun for a future caller; reversing to a throw is trivial.

4. Upgrade stamps a resync epoch — decided (Dawson), and its one exception is now named in the contract. Every existing store lacking the record gets max(Date.now(), newest retained audit key) on first open, recorded as a guess under its own key. Never stamping would make every upgraded deployment fail closed forever. The cost, per kriszyp's review, is stated rather than denied: the stamp is bounded by what survives, so a legacy deleteHistory that removed a table's newest entries, followed by a clock rollback, stamps a floor below history that is gone, and a cursor in that window is certified over the gap. The guarantee is therefore one-directional for every prune that ran with a floor recorded, and silent about history removed before one existed. Only a database generation closes that (restore_backup rolls a database back in time with no epoch, so audit cursors, record versions, and replication sequence state all read as valid afterwards #2451); the recorded epoch is what tells that repair how far the guess reached. On RocksDB getKeys() is unimplemented, so the clock-rollback guard reduces to Date.now() there.

5. A floor that cannot be written blocks the prune — routed to A failed audit-floor write blocks the boot purge exactly when the disk is full, making #1115's reclamation unreachable #2486. raiseAuditFloor throws, and it is called first precisely so the throw stops the prune. On a full volume the thing that fails is the 8-byte floor write, so Resync re-delivery (~6.7×) + cleanup starvation balloon a far-behind node's transaction logs (compounds #1114 OOM) #1115's boot purge is skipped in exactly the condition it exists for. You cannot record that you pruned without recording — the unknown sentinel is itself a write — so the escapes are reserved headroom, an in-memory poison, or accepting an inaccurate floor. Nothing reads the floor yet, so this is reversible per call site.

6. deleteHistory raises unprobed. The retention loop probes for an eligible entry before raising; deleteHistory raises to its (clamped) bound before knowing whether this table has any entry below it, so a routine per-table trim advances the database floor even when it deletes nothing. Symmetry would mean a tableId-filtered scan — the per-table cost decision 2 avoids.

7. Table.deleteHistory now rejects non-number bounds (reject-vs-coerce-deletehistory-bounds). A numeric string, 'Infinity', or a Date used to coerce through > into an accepted bound — the whole-log prune Kris caught — and now throws Invalid audit prune bound, matching the bridge's 400. A contract tightening on a public Table static, trivially reversible by coercing through the bridge's validation instead. Nothing that worked before stops working except the coercion that was the bug: a no-arg call was always an empty range (endTime defaults to 0), and a Date was never a valid range key.

8. On RocksDB the floor tracks the configured horizon, not surviving reality (rocksdb-floor-tracks-configured-horizon). Every retention pass advances the floor to the horizon whether a log file dropped or not, because whole-file purge granularity cannot say beforehand what will drop and the floor must be written first. Cursors over entries still on disk are told to resync — the safe direction, and the deliberate one; reversible only if the native purge ever reports what it will drop. Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448's consumer must not read "floor advanced" as "history removed".

9. Power-loss durability of the floor write on RocksDB is trusted, not enforced (engine-commit-durability-trusted). The write-ahead guarantee rests on the floor's transactionSync being durable before the same-thread purgeLogs unlinks files. It is crash-safe — a process death leaves OS buffers intact for both — but rocksdb-js exposes no per-commit WAL sync; the only gate is a full memtable flushSync, which its own docs describe as database-wide and stall-prone. So a power cut in that window can persist the unlink and lose the record. One flush per retention pass is cheap in frequency, not in blast radius; this wants a ruling on the engine contract before a sync is added.

10. The reopen test wraps root.close (test-monkey-patches-store-close). closeDatabase fire-and-forgets LMDB's asynchronous close() while the drop path awaits the same call; the durability test captures the discarded promise rather than changing closeDatabase. The helper still fire-and-forgets for any future caller — returning the close promise would fix it at the root, but that is outside this PR's write-side scope.

One deferred limitation stated in the internal contract: copying a database's state without its history — restore_backup reinstalls the backup's floor, and a RocksDB checkpoint copies the floor but no transaction logs — needs a database-level generation, because the same copy rolls back record versions and per-node replication sequence state too (#2451). The former check/use race limitation disappears by construction once #2448 puts the check inside subscribe — there is then no separate read to race. In this PR there is no read at all.

Verification

Storage-layer behavior, reachable from unit tests on both engines.

• unitTests/resources/auditFloor.test.js (new): floor establishment and the resync epoch, the exact resume predicate at F-1/F/F+1, six shapes of untrustworthy metadata all resolving to Infinity, monotonicity, database scoping and non-leakage, NaN/-0/non-number bounds refused before anything is deleted — including numeric strings, 'Infinity', and a future Date, which > would otherwise coerce into an accepted bound (kriszyp) — deleteHistory(Infinity) and far-future finite bounds clamped rather than recorded, bootstrap provenance, a real legacy standalone audit root with close-and-reopen durability, floor-before-purge ordering observed from inside purgeLogs, and the empty-pass no-write case. Tests read the floor through getAuditFloor(table.auditStore), the call Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 will make.
• unitTests/resources/auditPurge.test.js (extended): purgeAgedLogs records the floor at the same cutoff, before purgeLogs.
• unitTests/resources/deleteTransactionLogsBeforeRocks.test.js (extended): the whole-database purge advances the floor database-scoped and clamped; the bound guard refuses thirteen bad timestamp shapes with a 400 while accepting Date, numeric strings including '1e3' and '1234.5', a number, 0 and "0"; a far-future bound does not pin the floor.
• Every fix has a failing test, verified by ablation — restoring each pre-fix guard fails its test on both engine branches where the code path exists (the LMDB-only deleteHistory path is covered on LMDB, the RocksDB-only whole-database purge on RocksDB).
• unitTests/resources at the re-scope commit: 1731 passing on RocksDB, 1252 on LMDB. The one RocksDB failure is longLivedTransactions "names a chain link reachable only through the root", an order-dependent flake of main's own: it fails in the full suite with this branch's test files excluded, and passes in isolation and on re-run.
• Build clean, npm run lint:required 0 errors, prettier --check clean.
• Merged origin/main repeatedly rather than rebased; the DESIGN.md cheat-sheet conflicts were resolved as unions and verified row-by-row with an escaped-pipe-aware parser after one resolution silently reverted main's Indexed multi-value (elements) attributes return one result per matching element — duplicate records from range/contains queries #2434 row.

Docs: HarperFast/documentation#660 documented the now-removed accessor and is being reverted by a follow-up; #666 (corrections to that page) is closed. Consumer-facing docs land with #2448.

Deferred to its own issue, per the review's disposition: on RocksDB, Table.deleteHistory() still scans the whole audit range and reports a nonzero entriesDeleted for no-op removals. Pre-existing on main, not this diff — #2566.

History: squashed to one commit on current main on 2026-09-10 after the re-scope (43 commits over repeated main merges collapsed; tree verified byte-identical to the last tested tip before the force-push). The per-round review record — including the eight rounds on the docs page and the three findings that were the engine's own — is in this PR's threads and comments, not in the commit history.

Complexity: complicated

_{Review-Coverage: authored=claude; ran=codex; declined=gemini,cursor-grok,cursor-composer,domain; rounds=18; full=1 @ 167c67f}

_{Human-Review-Need: 4 @ 167c67f}

[gemini-code-assist]

Code Review

This pull request implements an audit retention floor mechanism to track the oldest retained audit history time, ensuring consumers resuming incremental audit-log consumption can reliably detect if their history has been pruned. The floor is raised before any pruning occurs across all prune paths, with updates made to ResourceBridge.ts, Table.ts, auditStore.ts, and databases.ts, alongside new comprehensive unit tests and design documentation. The review feedback suggests returning 0 immediately in Table.ts's deleteHistory when using RocksDB to avoid unnecessary iteration, and using strict assertions (assert.strictEqual and assert.deepStrictEqual) in the unit tests to prevent type-coercion bugs.

[dawsontoth]

Docs companion: HarperFast/documentation#660.

[claude]

Reviewed; no blockers found.

[dawsontoth]

The full-scope review round I ran to audit this function as a whole produced no coverage — the Codex leg was SIGKILLed on timeout and the fallback ran out of budget, so there is no receipt behind it. Rather than re-run and hope, here is the audit done by hand, so the concurrency and failure reasoning is checkable without re-deriving it.

Audit-floor failure matrix

Written because three consecutive fail-opens in updateAuditFloor were each introduced by the fix for the one before it, and because delta review kept grading the changed lines while the bugs lived in the interaction. This enumerates the states instead. The invariant every cell is checked against:

A caller must never prune when the floor was not durably recorded at or above the prune cutoff, and the floor must never move backwards.

raiseAuditFloor is called before every prune and throws on anything it cannot record, so "throws" in the table means the prune does not happen — the safe outcome.

Record state × operation

Recorded floor	raiseAuditFloor(cutoff)	establishAuditFloor()	Covered by
absent	persists Infinity (unknown) in-transaction, then the caller may prune	stamps max(Date.now(), newest retained key)	persists the unknown sentinel when a prune finds no floor record at all; gives a new database a floor…
absent at pre-check, established before the transaction	ordinary monotonic raise to cutoff	leaves the raced-in record alone	raises normally when a floor appears between the pre-check and the transaction
present, finite, < cutoff	raises to cutoff	no-op	never lowers an established floor; the five prune-path tests
present, finite, >= cutoff	lock-free skip, no write	no-op	never lowers an established floor
present, decodes unknown (Infinity, corrupt, wrong length, -0, NaN, negative)	stays unknown — cutoff > Infinity is false	does not stamp over it	leaves an unknown floor unknown when a prune tries to raise it; does not stamp over a record that decodes to unknown; 7 metadata shapes

Cutoff domain

Cutoff	Behaviour	Why it matters	Covered by
finite > current	recorded	—	prune-path tests
finite <= current	skipped	monotonic	never lowers an established floor
NaN, negative, -0, non-number	throws	audit keys are raw float64, so these sort above every timestamp and the prune range would span the whole log	throws on … as a cutoff… (5 shapes) + refuses a deleteHistory whose bound the range would honor
Infinity	recorded, reads back as unknown	deleteHistory(Infinity) legitimately removes everything	takes an Infinity cutoff and reports the floor as unknown

Write outcome × engine

Write outcome	LMDB	RocksDB	Covered by
succeeds	read-back matches → true	read-back matches → true	all prune-path tests
fails silently	read-back mismatch → false → throws	read-back mismatch → false → throws	does not report a commit when the floor write fails silently (LMDB); refuses the purge when the RocksDB floor write does not stick (RocksDB)
throws	propagates out of transactionSync → prune stops	same	does not prune when the floor cannot be recorded
put replaced by a rejecting stub	rejection contained, read-back still catches it	n/a (native putSync)	does not report a commit…; #2338's marker fixtures
transaction aborted, transactionSync → undefined	!== true → throws	!== true → throws	— (see gaps)
no audit store at all	throws with a named error	same	throws rather than skipping the write when there is no audit store

Known gaps, stated rather than papered over

1. Rocks clock-rollback bootstrap is a no-op. establishAuditFloor bounds a rolled-back clock with the newest retained key, but RocksTransactionLogStore.getKeys() returns [] (unimplemented, RocksTransactionLogStore.ts:457), so on RocksDB the bootstrap reduces to Date.now(). Accepted, documented in the code, and the ruled-on trade for not leaving upgraded deployments permanently fail-closed.
2. The swallowed-abort cell has no test. Nothing forces transactionSync to return undefined; the === true guard is reasoned from RecordEncoder.saveStructures' documented contract, not exercised.
3. 8 of 36 floor tests skip on RocksDB, 2 on LMDB — the engine-specific paths (deleteHistory is a no-op on RocksDB; getKeys is unimplemented there; the log purge exists only there). Every skip is a path that genuinely does not exist on that engine, but it means neither engine runs the whole file.
4. No cross-worker test. The raced-in-initialization cell is covered by stubbing the pre-check read, not by two real workers; the transaction is what makes it correct and that is reasoned, not measured.
5. Restore and RocksDB checkpoint copy a floor without its history — out of scope by decision, tracked in restore_backup rolls a database back in time with no epoch, so audit cursors, record versions, and replication sequence state all read as valid afterwards #2451.

Why this exists: three consecutive fail-opens in updateAuditFloor were each introduced by the fix for the previous one, and every one was found by human review after the automated rounds had cleared the same code as nits. If you review one thing here, review this table against the code rather than the diff.

[kriszyp]

I think we want to ensure that this uses the same mechanism for determining complete catch-up, as replication uses for determining base copy (should be exactly the same condition/path).
I'm also wondering if this could be more succinctly/narrowly handled by having a subscribe() return a flag indicating the start time preceded the audit retention window (try to subscribe, fail if it can't reliably subscribe, less prone to race conditions).
_{🤖 Reviewed with Codex}

[cb1kenobi]

Legacy bootstrap can certify a selectively pruned gap — Severity: blocker. A floorless LMDB store from an older version may already have an audit entry removed by table-specific deleteHistory that is newer than every surviving key. After clock rollback, max(Date.now(), newest) can remain below that missing entry, so a persisted cursor above this epoch passes the resume check and silently skips history. The simplest safe fix is to store Infinity for an unmarked store until a full resync or database generation records a trustworthy epoch.

—
Reviewed a184017

[dawsontoth]

Confirmed, and reachable — thank you, this is the sharpest form of the bootstrap objection so far. deleteHistory(endTime) removes one table's entries below endTime out of the shared log, so a table whose entries were the newest and all fell below that bound leaves the newest surviving key being a sibling's older entry. getKeys({reverse:true, limit:1}) therefore bounds what survives, not what existed, and a clock rolled back to between the survivor and the removed entries stamps an epoch below them. A cursor in that window then resumes over the gap. Four conditions have to coincide, but none of them are exotic.

What I have changed is the comment, which claimed more than the guard delivers — it now says the bound narrows the rollback hole rather than closing it, and why nothing in surviving state can distinguish the case (25bfa1a).

What I have not done is take the recommended fix, because AUDIT_FLOOR_UNKNOWN for every unmarked store is the option Dawson already ruled against — the residual question from an earlier round was exactly stamp, or preserve an unknown floor for any store with no record, and the cost of not stamping is that every upgraded deployment fails closed forever, for all consumers, permanently, since the floor is monotonic and Infinity is its top. That trade is recorded as decision 4 in the PR body. Your finding raises the price of that ruling and deserves to be weighed against it rather than settled by me, so I have escalated it to him rather than resolving this thread.

One option neither the ruling nor this comment considers, if it helps the decision: the missing ingredient is any persisted artifact of the pre-rollback clock, and the database file's mtime is one. max(Date.now(), newest surviving key, mtimeMs) would cover your scenario, since the removed entry was written before the last write to the file. It fails only when a copy resets mtimes, and its error direction is safe (too high costs one resync; too low degrades to today's bound). Narrows, still does not close.

🤖 Addressed by Claude Code

[dawsontoth]

Following up with a severity question rather than a fix, and it is genuinely a question — your call, not something I want to assume.

The case for re-targeting this at #2448 instead of this PR. As of 2a0d9ba00 the accessor promises nothing: it is explicitly diagnostic, and cursor >= floor is stated in the JSDoc, DESIGN.md, the docs page and the PR body as not a certificate that resuming is safe. Nothing in the product reads the floor — Table.subscribe does not consume it; that is #2448. So there is no code path today in which a wrong bootstrap epoch produces wrong behavior. It is a latent metadata bug, and #2448 is what makes it live.

That is now recorded structurally rather than promised: #2448 is marked blocked by #2451, with your finding and Kris's both written in as gating criteria, and #2451 carries a new acceptance criterion that came directly out of your comment — a database that reads unknown must be able to earn a real floor back — because AUDIT_FLOOR_UNKNOWN turns out to be absorbing (raiseAuditFloor cannot lift it, establishAuditFloor skips any existing record), which makes the generation stamp the only route back to a finite floor.

The strongest argument against me, which I would rather raise than have you find. Deferring is not free, because the bootstrap runs at first open by the new version, not when #2448 lands. Ship this and any affected store has the suspect epoch persisted immediately, months before anything reads it. A later release cannot repair it either, because nothing distinguishes a bootstrap-stamped floor from a legitimately-pruned one.

Which suggests a third option, if you would rather not wave this through. That last objection is fixable cheaply and additively: have establishAuditFloor also record the bootstrap epoch under its own key. The 8-byte floor format and its strict decode stay untouched, nothing reads the new key in this PR, and a later release can then identify exactly the suspect population — floor still equal to its bootstrap value, never raised by a real prune — and repair it. That converts "defer and hope" into "defer with a repair path," which I think is the actual disagreement here.

So, three ways I am happy to go, in my order of preference:

1. Add the bootstrap-provenance key now (~10 lines plus a test), and re-target the epoch fix itself at restore_backup rolls a database back in time with no epoch, so audit cursors, record versions, and replication sequence state all read as valid afterwards #2451. Deferral becomes repairable.
2. Re-target as-is, on the grounds that nothing acts on the floor until Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 and Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 cannot land without restore_backup rolls a database back in time with no epoch, so audit cursors, record versions, and replication sequence state all read as valid afterwards #2451.
3. Add the mtime term now — max(Date.now(), newest surviving key, mtimeMs) covers your scenario, since the removed entry was written before the last write to the file. Narrows, does not close, and fails when a copy resets mtimes.

Tell me which and I will implement it, or hold the PR if you would rather none of them. Not resolving this thread either way — that is yours to close.

🤖 Addressed by Claude Code

[dawsontoth]

Implemented option 1 — fbb96f673. Dawson's call was to take it, so the epoch fix itself stays with #2451 and this PR makes the deferral repairable rather than permanent.

establishAuditFloor now writes the epoch under Symbol.for('audit-floor-bootstrap') before the floor, stamps the floor from what that record holds, and never touches the record again. The reading a later release gets:

Floor	Bootstrap record	Reading
finite, === bootstrap	present	still the unverified guess — suspect, repair by raising it
finite, > bootstrap	present	a real prune raised it; earned, not guessed
finite	absent	provenance write was lost — suspect, so this fails closed
AUDIT_FLOOR_UNKNOWN	either	already fail-closed; nothing to repair

Absent reads as suspect because repair means raising a floor, which is always safe, so an over-broad reading costs one resync. Ordering carries the rest: a crash between the two writes leaves a record with no floor, which the next open retries because the early return tests the floor. updateAuditFloor takes the key now rather than hardcoding it, so the record gets the same verified commit as the floor instead of a second write path.

Two things worth flagging because they are the kind of decision that looks arbitrary later:

• Adoption is not bounded by the newest surviving key. Re-deriving max(Date.now(), newest) on a store that already has a record would be more conservative, and it would also destroy the signal — floor === bootstrap is the entire basis of the table above. So the record wins, which is sound because a prune in the meantime would have written the floor the early return checks.
• There is no read-the-record-first fast path, though it is the obvious optimization and I wrote it that way first. Ablating the write showed the orphan-adoption test still passing, because the initial read was silently covering for it — two routes to one value, the second untested. That is the same shape as the four fail-opens earlier in this PR, so it is now one path; everything past the early return runs once in a store's life, so the transaction it saves is not worth it.

Coverage checked by ablation on both engines rather than assuming the branch is shared: with the record write and adoption removed, the same four tests fail on RocksDB and on LMDB, and the fifth is a negative test that correctly still passes. Suites 1975 RocksDB / 1634 LMDB, zero failures.

Your original scenario is still reachable in the sense that matters — the epoch can still be wrong. What changed is that it can now be found and fixed. Leaving this thread for you to close or hold.

🤖 Addressed by Claude Code

[dawsontoth]

Correction to the table I posted above — the cross-model round at 1f5ab5b broke the middle row, and it was the load-bearing one.

floor > bootstrap ⇒ a real prune raised it; earned, not guessed does not follow. A prune raising the floor above the epoch certifies only the history that prune removed. It says nothing about history removed before tracking began, which may sit above the epoch — which is exactly the case your finding is about, since that is what the guess cannot see. Concretely, using your own scenario: a v4-era deleteHistory removes tableA up to t=1000 while sibling tableB's newest survivor is 900; the rolled-back clock stamps bootstrap=900 and floor=900; a later retention pass raises the floor to 950. A repair keyed on floor > bootstrap reads 950 > 900, calls it earned, and skips the store — leaving a consumer at cursor 970 certified over tableA's missing 950–1000. The reading would have skipped precisely the databases the record exists to find.

So the record's presence is the signal, and no comparison against the floor retires it. A store carrying the record has an unverified pre-tracking window for as long as the record exists, however far the floor has since moved; only a database generation (#2451) can retire the mark. What the recorded value is for is telling that repair how far the guess reached. Fixed in 0f479e1f6 — the contract in DESIGN.md and the docstring, since nothing reads the record yet.

Same round found a second defect in it, which matters more for your original concern: an undecodable provenance record pinned the floor to unknown permanently. The resolver skipped the write whenever a record existed, so my comment claiming "the floor stays absent, so a later open retries" could not come true — every later open failed the read back identically. A torn 4-byte write would have produced the fail-closed-forever state this whole bootstrap exists to avoid. Undecodable bytes are now overwritten, which is safe for this record and would not be for the floor, where a present record may be a deliberate AUDIT_FLOOR_UNKNOWN. I had weighed that exact asymmetry when I wrote it and chose wrong.

Your finding is unchanged by any of this: the epoch stamped here can still be below history a legacy selective prune removed. What has changed is that the store is now durably marked as carrying a guess, and the mark cannot be cleared by anything short of the generation work.

🤖 Addressed by Claude Code

[dawsontoth]

On the review summary — the inline finding is addressed in 2a0d9ba (the API is now explicitly diagnostic; thread resolved). The two points here:

"Same mechanism replication uses for determining base copy." The existing predicate is shouldForceBaseCopyForRetention (harper-pro replication/replicationConnection.ts), and you are right that the floor is not that condition today: replication's bound is requestedStartTime < max(oldestRetainedTime ?? 0, Date.now() - auditRetention), computed live from a scan of one peer's log, while the floor is a persisted, monotonic, database-scoped watermark written ahead of each prune. I have written the five concrete divergences into #2448 as a requirement so it is not rediscovered — the sharp one is that replication maxes in the nominal cutoff unconditionally while the floor's LMDB branch raises off the first actually eligible entry, so when cleanup has not run recently the two give different answers for the same database right now, with replication the conservative one. Also flagged there: that difference is not purely accidental, since replication's bound doubles as the unbounded-replay heap guard (#1114) and is deliberately willing to force a base copy for entries still on disk. So unification most likely means the floor becomes the authoritative retention term with replication keeping its heap bound on top, rather than either one adopting the other wholesale.

"Could subscribe() just return a flag." Agreed, and that is #2448's design arrived at independently — including your reason for preferring it. Validating inside the resume is the only thing that closes the check/use window, since retention can advance between a caller's read and its subscribe. The one design note recorded there: it wants to be opt-in rather than a default error, because Table.subscribe also serves MQTT durable resume, SSE/WS reconnects, and sourcedFrom caching tables.

Which leaves the fair question your summary implies: if subscribe() returning a flag is the real ergonomics, is a public accessor worth merging first? The primitive-only scope was Dawson's call, on the reasoning that the flag needs a decision on default behavior across those four consumers, while the retention plumbing underneath it — five prune paths that now raise the floor before removing anything, in a transaction whose commit is verified — is needed by either shape and is what this PR actually is. If you would rather see the flag land before any public accessor, that is worth settling with him directly; I am not going to relitigate the scoping decision on his behalf.

🤖 Addressed by Claude Code

[claude]

What: cb1kenobi's thread here ("Legacy bootstrap can certify a selectively pruned gap — Severity: blocker") is still open (isResolved: false) as of this push. 25bfa1af5 only rewords this comment to admit the gap — establishAuditFloor's max(Date.now(), newest surviving key) bootstrap is unchanged: a legacy deleteHistory that pruned one table's entries out of the shared log can leave the newest surviving key older than history that's actually gone, and a clock rolled back into that window still stamps an epoch that certifies a cursor over the silently-lost entries.

Why it matters: Both the MEMBER reviewer and the author-agent agree this is real and reachable ("Confirmed, and reachable... Four conditions have to coincide, but none of them are exotic"). The author-agent explicitly declined to implement a fix and escalated the stamp-vs-unknown tradeoff to a human decision-maker rather than resolving the thread. Flagging so this doesn't get lost under a "no blockers" verdict from any reviewer pass — it needs the maintainer decision the author is waiting on, not a rediscovery.

Suggested fix: None of mine to add on top of the thread — the two candidates are already on the table there (stamp AUDIT_FLOOR_UNKNOWN for every unmarked store, or the mtime-based narrowing floated in the latest reply). This is a decision, not an implementation gap.

[dawsontoth]

Accurate on all counts, and the read on 25bfa1af5 is the right one: it rewords the comment to stop the guard over-claiming, and changes no behavior. The bootstrap is unchanged and the decision is live, not lost — leaving both this thread and cb1kenobi's unresolved on purpose.

Two pieces of state worth having here so a later pass does not rediscover them:

1. AUDIT_FLOOR_UNKNOWN for unmarked stores is absorbing. raiseAuditFloor cannot lift it (cutoff > Infinity is false) and establishAuditFloor skips any existing record, so under that candidate a legacy store never earns a finite floor back by any route. The only mechanism that could is a persisted marker of when the floor-maintaining version took over, at which point the oldest surviving key becomes a sound bound — i.e. the database generation in restore_backup rolls a database back in time with no epoch, so audit cursors, record versions, and replication sequence state all read as valid afterwards #2451. So both open blockers on this PR (this one and Kris's restore/checkpoint hole) reduce to the same missing primitive.
2. The maintainer's current question is whether to pivot to enforcement inside subscribe (Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448's shape, which Kris independently proposed) rather than a public accessor. That does not make the epoch trustworthy — location does not fix metadata — but it changes this trade's price: as an opt-in subscribe option, the unknown sentinel only costs the subscribers who ask for the guarantee, instead of every consumer on every upgraded deployment. That is the weighing in progress.

Nothing here blocks behavior today: after 2a0d9ba00 the accessor is diagnostic and no product path acts on the floor, so this is a latent metadata bug that becomes live when #2448 lands. Whether that makes it a blocker for this PR or for #2448 is cb1kenobi's call, not mine to assume.

🤖 Addressed by Claude Code

[kriszyp]

I think my concerns are:

• I don't think there should be any inconsistency between how we determine a valid subscription resume in replication vs how we do it with the Table.subscribe. Is there any compelling reason to create different paths? Or should these be aligned/consistent?
• I know this is an acknowledged open question, but I am not sure I followed the justification for separating this from #2448; the specific concern with separation and defining a Table.oldestRetainedAuditTime() is that we are binding ourselves to a contract with a new public API (HarperFast/documentation#66 indicates it is supposed to be publicly documented), yet this API hasn't been shown to have any external purpose, its purpose is only for 2448, AFAICT. Furthermore, that contract seems like it could be tenuous. I think we have acknowledged elsewhere that higher precision subscription resumption may involve a vector of resume timestamps (for each origin), which doesn't seem to align with a single number returning floor function in this iteration (or maybe I am missing something?).
  _{🤖 Reviewed with Codex}

[dawsontoth]

Re-scoped to infrastructure only, per this review — Dawson's call, 2026-09-10, at 04caa8f.

• Table.oldestRetainedAuditTime() is removed. The floor itself stays — write-ahead, monotonic, verified on commit, raised first by all five prune paths — but nothing reads it publicly. The single-number-vs-per-origin-vector point is what settled it: inside subscribe() the check can go per-origin whenever it needs to, with no public shape to preserve.
• The consumer is Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448. Table.subscribe reads getAuditFloor inside the resume, so the check and the replay cannot drift apart. That is the same shape as replication's shouldForceBaseCopyForRetention, which answers the consistency question by making it one path rather than two aligned ones.
• docs(resources): the audit floor detects pruning, not a restored rollback documentation#666 is closed — there is no public API to document. Docs for the behavior land with Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448.

Both inline findings are fixed in the same push; replies are on the threads. PR title and body are being updated to match the new scope.

🤖 Addressed by Claude Code

[cb1kenobi]

Reviewed the full diff at 167c67f for correctness, concurrency, engine parity, and test coverage, and found no new blocking issues on changed lines. The write-ahead ordering holds because every prune path records the clamped bound as its floor and then uses that same value as its range end, so nothing can be removed that the floor does not cover. Earlier blocking findings (string/Date coercion through the clamp, phantom commits on floorless stores, far-future bounds pinning the floor) are all fixed with tests. Remaining open threads are scope and contract decisions for the maintainers, not defects.

—
Reviewed 167c67f

[kriszyp]

I think this looks good, nice work. I am filing an issue for the follow-up work of using a shared function between this subscription and replication handler for determining the audit/txn log floor.
_{🤖 Reviewed with Codex}