Skip to content

lance-graph-dt64-lance: DR's map table made durable through the fire-and-forget cycle writer - #1993

Merged
AdaWorldAPI merged 16 commits into
mainfrom
ccr-8062bcc7-l7lkz9-dt64-lance
Nov 10, 2026
Merged

AdaWorldAPI merged 16 commits into
mainfrom
ccr-8062bcc7-l7lkz9-dt64-lance

Conversation

@AdaWorldAPI

@AdaWorldAPI AdaWorldAPI commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Makes DR's versioned global → local table (dt64::dr::MapTable) durable. This is the storage half of D-DT64-7 O5 / D10′ (D-LITHO Q5 asked which table).

The code is in an excluded crate with its own workspace. dt64 stays free of lance and arrow. Outside the new crate this PR adds three things:

  • dt64: MapTable::claim, MapTable::advance_at and MapTable::with_clock_at;
  • the supervisor: SealTicket::wait(&mut self), which is cancel-safe;
  • wal_shootout: a dr64 subcommand.

Design

The owner never waits for storage. There is no custom WAL, no confirmation ledger, and no arbitration between writers. This follows .claude/v3/knowledge/d-mbx-a6-owner-consume-and-persistence.md.

owner:   advance ── claim v ── casts ──try_submit(v, casts)──▶ returns v
writer:                                   │ one background task owns the sink
                                          ▼
                      seal cycle v (LanceCycleWriter: one Lance version)
owner:   settle ── ticket Ok ── advance_at(v)        ticket Err ── v is a gap
  • The WAL is LanceCycleWriter behind the supervisor's CycleWriter (D-DR64-WAL-1, EXO-225). Measured at DR64 sizes, with 1, 4, 16 and 64 tiles per advance over three runs:

    • it commits an advance in 2.2–5.7 ms p50, against 7.4–8.3 ms for MemWAL with a 10 ms flush;
    • it keeps one Lance version per advance, which DR's reads as of a route version need.

    The full table is in .claude/plans/wal-shootout-v1.md §8.

  • One advance is one cycle, and the cycle id is the route version.

    • The version is claimed when the advance is built (MapTable::claim).
    • It is applied at that same version, and only after the seal ticket is Ok (MapTable::advance_at). Ticks of a shared clock in between do not move it.
    • A failed seal applies nothing, so its version is a gap in both the table and the store. Every later advance is then refused (Poisoned), because the writer seals nothing after a failure.
    • Waiting for a seal is cancel-safe: a pending advance stays queued until its result is in.
  • The table is handed out only when drained. MapLog::into_table returns Undrained, carrying the log back, in two cases:

    • a submitted cycle is still unsettled;
    • the store may hold a cycle the table lacks. That covers a durable cycle the table refused (Settled::Refused) and a seal whose outcome is unknown (Settled::OutcomeUnknown: an ambiguous commit, a hash conflict, or a writer gone before answering).

    Both also poison the log. A failure that proves nothing was published stays NotDurable, and the log still drains.

  • Ownership travels in the casts. Every cast is written on behalf of the mailbox that owns the table. restore refuses any landing written on behalf of another mailbox.

  • One 512-byte cast per 64 ids. A W64 tile is one cast, a W256 tile four, and a removal one. Each advance also gets a marker cast carrying its version, source and relation-map revisions.

  • Restore replays exactly. restore reads every sealed cycle (scan_sealed + scan_image) and re-applies each advance at its own route version.

    • It starts at the stored genesis (with_clock_at), so a shared clock that has already moved on is fine.
    • Images that do not form a valid log are refused.

The first version of this PR wrote its own dataset synchronously, refused stale writers by dataset version, and reported a post-commit Race / ClockMoved. Codex's two P1 findings were concurrency holes in that design, and the rebuild above answers both. Its two later P2 findings (replay onto a clock that has moved on, and cancelling settle_next) are fixed in 0a89ed9f6. The CI job's missing ndarray checkout (P1) and the ambiguous-seal case (P1) are fixed as well. All six threads are resolved.

Tests

The new crate has 5 unit tests and 12 integration tests, and dt64 gains a unit test for with_clock_at. The round trip runs on a real LanceCycleWriter. The interleavings are driven deterministically by a gated in-memory sink: commits wait on a semaphore that the test opens, and a chosen cycle can be made to fail or to answer ambiguously.

Test Shows
a_restored_table_answers_every_past_version_exactly Six advances: extension, reorder, removal, an empty advance, re-add, and an empty map. After restore every version answers the same, generation and event counts match, and there is one Lance version per cycle.
the_owner_returns_before_the_seal_and_applies_only_after_it Two advances are submitted while commits are held. Nothing is applied until the seals land; then both apply in order.
a_clock_tick_between_claim_and_seal_lands_at_the_claimed_version The shared clock ticks twice while the cycle seals. The advance lands at the claimed version, and the restored table agrees.
a_failed_seal_applies_nothing_and_poisons_what_follows The failed seal hands its casts back, and the seal queued behind it is poisoned. Neither is applied and no further advance is claimed. The log still drains. The store replays to genesis plus v1.
every_landing_is_on_behalf_of_the_owner Every landing carries the owner, and a restore on behalf of another mailbox is refused.
a_reserved_id_or_a_repeated_tile_is_refused_before_a_version_is_claimed Both are refused before any version is claimed. A silence twin checks that a valid advance still goes through.
a_restored_log_continues_and_restores_again Resume, advance, and restore again.
a_restore_onto_a_clock_that_has_moved_on_replays_exactly A restore onto a clock already at 5 answers every version like the live table, and the next advance is issued at 6.
a_cancelled_settle_keeps_the_advance_pending A timed-out settle_next leaves the advance queued. It applies once the seal lands.
a_frame_from_before_a_restore_is_foreign Pins the open frame question.
into_table_refuses_while_a_submitted_cycle_is_unsettled into_table refuses while cycles are pending; after settling, it returns the applied table.
an_ambiguous_seal_blocks_the_table_until_a_restore The sink commits, then answers Ambiguous. The log is poisoned and into_table refuses; the restore shows the store held the advance.
a_durable_cycle_the_table_refuses_poisons_the_log (unit) A refused durable cycle blocks later advances and into_table.

Disable runs, each restored afterwards:

Disabled Failed
Apply even when the seal failed a_failed_seal_applies_nothing_and_poisons_what_follows
The owner check on restore every_landing_is_on_behalf_of_the_owner
advance_at fed the clock's current version instead of the claimed one 4 tests, including the clock-tick test
The poison gate the failed-seal test
The reserved-id check the reserved-id test
Replay built with with_clock instead of with_clock_at a_restore_onto_a_clock_that_has_moved_on_replays_exactly
Pop the pending advance before waiting a_cancelled_settle_keeps_the_advance_pending
into_table always returns the table into_table_refuses_while_a_submitted_cycle_is_unsettled
No poison, or no divergence record, on Refused a_durable_cycle_the_table_refuses_poisons_the_log (two separate runs)
Treat no seal error as an unknown outcome an_ambiguous_seal_blocks_the_table_until_a_restore
Treat every seal error as an unknown outcome a_failed_seal_applies_nothing_and_poisons_what_follows (its drain assertion)

Other checks:

  • cargo fmt and cargo clippy --all-targets -D warnings are clean for the new crate, dt64 and the supervisor.
  • The dt64 suite passes, and so do the supervisor's cycle_writer tests.
  • The new crate needs lance-graph's ndarray-hpc feature, because lance-graph does not build without it. Its CI job checks out the sibling ndarray.

Open

  • Frames are not restored. A frame is one table instance's witness. Issuing a frame at a past version after a restore is a DR owner decision.
  • Only the map is stored. A table whose clock is shared with a RouteLog needs the route log to be durable too.
  • Compaction. The map table's files and scan time grow with every advance (EXO-225), so it needs a compaction schedule. Restore currently runs one filtered scan per sealed cycle, so its cost grows roughly with cycles × fragments; it has not been measured.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk

…dataset

Stores dt64's dr::MapTable (the versioned global -> local table, D-DT64-7
O5 / D10') in Lance, in a new excluded crate so dt64 stays free of lance and
arrow:

- one MapTable::advance is one Lance commit, written before the advance is
  applied, so a refused or failed commit leaves the table unchanged;
- one row per local position, plus genesis, advance and removal markers;
- the DR route version travels in the rows; the Lance dataset version is the
  commit counter, used to refuse a stale writer, detect a race, and read the
  table as of a past commit (restore_at, via checkout_version);
- restore replays every advance at its own route version, including gaps from
  a shared clock, so every past version answers exactly as before; rows that
  do not form a log are refused.

Frames are not restored: a frame from before a restore is foreign to the
restored table. Executing at a past version after a restore is left open
for the DR owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 93e9d3dc-b1d2-4ccb-9f67-56116d490abe
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@AdaWorldAPI
AdaWorldAPI marked this pull request as ready for review November 10, 2026 18:16
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T20:35:38.881974Z 4621338 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 050fa942d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A DR64 tile map is 64 global ids x 8 bytes, one 512-byte payload, so a DR
advance is the shoot-out's SoA-512 cycle with one cast per changed tile.
The existing sizes start at N = 64; dr64 runs the same loop at N = 1, 4,
16 and 64 changed tiles per advance, both arms, to pick the WAL for the
DR map table (#1993).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
MapTable::claim takes the next version off the (possibly shared) clock;
MapTable::advance_at applies an advance at exactly that version once its
rows are durable, even if other clock holders ticked in between. It
refuses, changing nothing, a version at or before the last applied one
and a version the clock never issued. advance() is now claim + apply, so
its behaviour is unchanged. This is what lets a WAL-backed map log carry
the route version on the cast instead of re-deriving it after the seal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
N = 1, 4, 16, 64 tiles per advance, three processes: L commits in
2.2-5.7 ms p50 against 7.4-8.3 ms for MemWAL at a 10 ms flush, and keeps
one Lance version per advance, which DR's as-of-route-version reads need.
Its cost is files and scan time, which need a compaction schedule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
…ter (not yet compiled)

Advances claim a route version, become casts on behalf of the owning
mailbox, and are submitted to CycleWriter over LanceCycleWriter
(D-DR64-WAL-1). advance_at runs only after the seal ticket is Ok; a failed
seal is a gap and poisons further submissions. Restore replays sealed
cycles and refuses landings on behalf of another mailbox. The synchronous
Lance write loop and StaleWriter/Race/ClockMoved are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
12 tests: the round trip on LanceCycleWriter answers every past version
exactly (one Lance version per cycle); gated in-memory interleavings cover
submit-before-seal, a clock tick between claim and seal, a failed seal
followed by a poisoned one, ownership on every landing, refusals before a
claim, resume + restore again, and foreign frames after a restore.
Requires lance-graph's ndarray-hpc feature (its fallback does not build).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
…t rebuild

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
@AdaWorldAPI AdaWorldAPI changed the title lance-graph-dt64-lance: DR's map table in a separate Lance-versioned dataset lance-graph-dt64-lance: DR's map table made durable through the fire-and-forget cycle writer Nov 10, 2026

Copy link
Copy Markdown
Owner Author

@codex review

The crate was rebuilt on the fire-and-forget cycle writer, which replaces the design the earlier review read. The description covers the new design and its tests.


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 542dd6bece

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ss a cancelled settle

Codex P2s on #1993.
- restore built the table with MapTable::with_clock, which takes its last
  version from the clock, so a shared clock that had moved on made a valid
  log fail with NotAfterLast. New MapTable::with_clock_at starts the table
  at the stored genesis and keeps the clock for later issuance.
- settle_next popped the pending advance before awaiting its seal, so a
  cancelled wait dropped a durable advance. New SealTicket::wait(&mut self)
  waits without consuming the ticket; the entry stays queued until its
  result is in.
Tests: a_restore_onto_a_clock_that_has_moved_on_replays_exactly,
a_cancelled_settle_keeps_the_advance_pending, and a dt64 unit test for
with_clock_at.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk

Copy link
Copy Markdown
Owner Author

@codex review

Both P2 findings are fixed in 0a89ed9f6 (replay at the stored genesis, and a cancel-safe settle), each with a test and a disable run.


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0a89ed9f69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…le cycle poisons it

MapLog::into_table returned the table while sealed cycles were still
unsettled, dropping them silently. It now returns Undrained (carrying the
log back) while anything is pending or a durable cycle was refused. A
durable cycle the table refuses (Settled::Refused) now poisons the log:
later advances are refused, as after a failed seal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk

Copy link
Copy Markdown
Owner Author

@codex review

Since the last review:

  • c2ae0b15d: MapLog::into_table now returns Undrained, which carries the log back, while any submitted cycle is unsettled or a durable cycle was refused. A sealed cycle can no longer be dropped unapplied.
  • In the same commit, a durable cycle the table refuses (Settled::Refused) poisons the log: later advances return Poisoned, as after a failed seal.
  • 8ce17afcf: the CI job checks out the sibling ndarray.

Tests:

  • into_table_refuses_while_a_submitted_cycle_is_unsettled (integration): a refused into_table, then settle, then a drained table.
  • a_durable_cycle_the_table_refuses_poisons_the_log (unit).

Disable runs, each red: into_table always Ok; no poison on Refused; no divergence record on Refused.


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8ce17afcfb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…f reading as a gap

An ambiguous commit (SealRecovery::ResubmitFrozen), a hash conflict
(Escalate) or a writer gone before answering may have published the cycle.
Settled::OutcomeUnknown reports it; the log is poisoned and into_table
refuses until a restore. A proven-unpublished failure stays NotDurable and
still drains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMMWgnUU9g7Ldyn6sbk

Copy link
Copy Markdown
Owner Author

@codex review

462133860: a seal of unknown outcome (an ambiguous commit, a hash conflict, or a writer gone before answering) is now Settled::OutcomeUnknown. It poisons the log, and into_table refuses until a restore. A proven-unpublished failure still drains.


Generated by Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 4621338600

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@AdaWorldAPI
AdaWorldAPI merged commit b34d195 into main Nov 10, 2026
15 checks passed
AdaWorldAPI pushed a commit that referenced this pull request Nov 10, 2026
…ter, #1993 merged

Mapping for #1997 found three errors. authorize_identity reaches
authorize_memberships (MembershipDecision), not authorize_scoped. The
production LanceCycleWriter (WalSink::commit_cycle) is on main beside the
probe's stand-in, so G4 composes shipped pieces. #1993 merged as b34d195.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMPJjVvasjwqhjJ6c9p
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants