lance-graph-dt64-lance: DR's map table made durable through the fire-and-forget cycle writer

.claude/board/entries/2026-11-10-dt64-map-table-lance.md

@@ -0,0 +1,51 @@
+# 2026-11-10 — DR map table made durable through the fire-and-forget cycle writer
+
+**Status:** VERIFIED-IN-CODE (12 tests; the round trip runs on a real
+`LanceCycleWriter`). Answers the storage half of D-DT64-7 O5 / D10'
+(D-LITHO Q5: "which table"). WAL choice: D-DR64-WAL-1 (EXO-225).
+
+> ⊘ **SUPERSEDED the same day:** the first version of this crate wrote its
+> own Lance dataset synchronously, compared dataset versions to refuse a
+> stale writer, and reported a post-commit `Race` / `ClockMoved`. That is a
+> custom WAL with arbitration, which the persistence doctrine forbids
+> (`.claude/v3/knowledge/d-mbx-a6-owner-consume-and-persistence.md`). It was
+> rebuilt on the cycle writer.
+
+**What.** `crates/lance-graph-dt64-lance` (excluded, own workspace) makes
+`dr::MapTable` durable. dt64 stays free of lance and arrow.
+
+- `MapLog::advance` claims the route version (`MapTable::claim`), turns the
+  advance into casts on behalf of the owning mailbox, submits them with
+  `CycleWriterHandle::try_submit` (cycle id = route version) and returns at
+  once. One background task owns the `LanceCycleWriter`.
+- `MapLog::settle` applies an advance with `MapTable::advance_at` only when
+  its seal ticket is `Ok`. A failed seal applies nothing and its version is a
+  gap; every later advance is refused (`Poisoned`), matching the writer,
+  which seals nothing after a failure.
+- One 512-byte cast per 64 ids: a W64 tile is one cast, a W256 tile four, a
+  removal one; each advance adds one marker cast (version, source,
+  relation-map revisions).
+- `restore` reads every sealed cycle (`scan_sealed` + `scan_image`) and
+  re-applies each advance at its own route version. It refuses a landing on
+  behalf of another mailbox and any image that is not a valid log.
+- There is no arbitration: one owner holds the `MapLog`, which owns its
+  table mutably. Two writers on one dataset are outside the contract
+  (`LanceCycleWriter` fences on its own cached head).
+
+**Tests.** 4 unit + 8 integration. Disable runs, each restored:
+- applying without a sealed ticket → `a_failed_seal_applies_nothing_and_poisons_what_follows`;
+- no owner check on restore → `every_landing_is_on_behalf_of_the_owner`;
+- applying at the clock's current version instead of the claimed one → 4 tests,
+  including `a_clock_tick_between_claim_and_seal_lands_at_the_claimed_version`;
+- no poison gate → the failed-seal test;
+- no reserved-id check → `a_reserved_id_or_a_repeated_tile_is_refused_before_a_version_is_claimed`.
+
+**Open.**
+- Frames are not restored: a frame is one table instance's witness, so a
+  frame from before a restore is `ForeignFrame` (pinned by a test). Issuing a
+  frame at a past version after a restore is a DR owner decision.
+- A table whose clock is shared with a `RouteLog` needs the route log
+  durable too; only the map is stored here.
+- The map table's compaction schedule (L's files and scan time grow with
+  advances; EXO-225).
+- lance-graph does not build without `ndarray-hpc`; this crate enables it.

.claude/board/entries/README.md

@@ -25,13 +25,14 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
 opposite directions; the stranding this convention prevents shows up in
 exactly one of them, never both.
 
-594 entries, 2026-08-06 .. 2026-11-10.
+595 entries, 2026-08-06 .. 2026-11-10.
 
 | date | entry id | finding | file |
 |---|---|---|---|
 | 2026-11-10 | `lab-resolve-borrow-all` |  | [2026-11-10-lab-resolve-borrow-all.md](2026-11-10-lab-resolve-borrow-all.md) |
 | 2026-11-10 | `int8-direct-packed-b` |  | [2026-11-10-int8-direct-packed-b.md](2026-11-10-int8-direct-packed-b.md) |
 | 2026-11-10 | `D-FOUNDRY-PARITY-1` |  | [2026-11-10-foundry-parity-1-lab.md](2026-11-10-foundry-parity-1-lab.md) |
+| 2026-11-10 | `dt64-map-table-lance` |  | [2026-11-10-dt64-map-table-lance.md](2026-11-10-dt64-map-table-lance.md) |
 | 2026-11-10 | `dt64-lane-borrow-all` |  | [2026-11-10-dt64-lane-borrow-all.md](2026-11-10-dt64-lane-borrow-all.md) |
 | 2026-11-10 | `dt64-int8-facet-stride` |  | [2026-11-10-dt64-int8-facet-stride.md](2026-11-10-dt64-int8-facet-stride.md) |
 | 2026-11-10 | `D-DT64-GEMM-SHAPE-1` |  | [2026-11-10-dt64-gemm-shape.md](2026-11-10-dt64-gemm-shape.md) |

.claude/plans/d-dt64-7-behaviour-addressing-registration-v1.md

@@ -120,6 +120,8 @@ Preserved throughout: the `0xC401` name-bound reading, the `FnIndex` opcode cont
 
 **Status:** core shipped (`dr::MapTable`, T1–T4 green, both disable runs red). Stored selections wait on O4.
 
+**Storage (2026-11-10):** the versioned table is made durable by `crates/lance-graph-dt64-lance` (`MapLog`) through the fire-and-forget cycle writer over `LanceCycleWriter` (D-DR64-WAL-1). An advance claims its route version, is submitted as casts on behalf of the owning mailbox, and is applied with `advance_at` only after its seal ticket is `Ok`; a failed seal is a gap. `restore` replays every sealed cycle at its own route version. (⊘ An earlier same-day version wrote its own dataset synchronously with stale-writer/race checks; replaced.) OPEN: frames are not restored (a frame is one table instance's witness), so executing at a past version after a restore needs a DR decision; a table sharing its clock with a `RouteLog` needs the route log stored too.
+
 O5 is a BindStore plan item (D-DT64-6 Amendment A). Litho is the mental model it came from, not a component.
 
 **What exists (VERIFIED-IN-CODE, `lance-graph-dt64/src/dr.rs`):**

.claude/plans/wal-shootout-v1.md

@@ -252,3 +252,37 @@ compactor) does to both curves.
 - **D-LNC-7, decided:** `LanceCycleWriter` stays the default with the capped session;
   MemWAL stays an option. The 60-minute soak is deferred.
 
+
+## 8. DR64 sizes (2026-11-10) — the WAL for DR/DT64 map advances
+
+A DR64 tile is 64 global ids × 8 B = one 512-byte cast, so a DR advance that
+changes N tiles is one cycle of N casts. `examples/wal_shootout.rs dr64` runs
+N = 1, 4, 16, 64 at 200 cycles each, on one AVX-512 VM, three separate
+processes. The arms are the same as §1: L = `LanceCycleWriter`, M10 / M100 =
+lancedb MemWAL flushed every 10 / 100 ms. Ranges below cover the three runs.
+
+| N tiles | L p50 | L p99 | M10 p50 | M10 p99 | M100 p50 |
+|---|---|---|---|---|---|
+| 1 | 2.2–4.4 ms | 4.0–7.1 ms | 8.1–8.3 ms | 10.7–11.2 ms | 97.7–98.2 ms |
+| 4 | 2.8–5.7 ms | 5.6–9.2 ms | 7.5–8.3 ms | 10.8–11.8 ms | 97.2–98.3 ms |
+| 16 | 3.4–5.4 ms | 5.2–9.0 ms | 7.8–8.3 ms | 10.9–11.1 ms | 97.4–97.5 ms |
+| 64 | 4.3–5.4 ms | 6.2–9.3 ms | 7.4–7.9 ms | 11.0–11.2 ms | 97.0–97.4 ms |
+
+| N tiles | L MiB / files / versions / scan | M10 MiB / files / versions / scan |
+|---|---|---|
+| 1 | 2.7 / 601 / 200 / 23–28 ms | 1.0 / 209 / 3 / 6–7 ms |
+| 64 | 15.6 / 601 / 200 / 23–25 ms | 14.9 / 209 / 3 / 8–9 ms |
+
+- **L is faster per advance at every N measured**, at p50 and at p99, in all
+  three runs. M's commit time is its flush interval, not its write.
+- **L keeps one Lance version per advance** (200 versions for 200 cycles). M
+  folds 200 advances into 3 versions. DR reads the map table as of a route
+  version, so a version per advance is the shape DR needs, not overhead.
+- **L's cost is files and scan time:** 601 files and a 23–32 ms scan after 200
+  advances, against 209 files and 5–9 ms for M. Both grow with the number of
+  advances, and L's needs compaction (an owner-scheduled step; not part of the
+  advance path).
+
+**D-DR64-WAL-1, decided:** DR/DT64 map advances go through `LanceCycleWriter`
+(L) behind `CycleWriter`, fire and forget. MemWAL is not used for DR. Open:
+the compaction schedule for the map table.

.github/workflows/dt64-map-table-lance.yml

@@ -0,0 +1,84 @@
+name: dt64 map table in Lance
+# lance-graph-dt64-lance makes dt64's DR map table (dr::MapTable) durable
+# through the supervisor's cycle writer over LanceCycleWriter.
+# It is workspace-excluded (own [workspace] root), so no other job builds it.
+# Without this one a change to dt64's dr module or a lance bump could break it
+# unnoticed.
+#
+# lance's build compiles .proto files: `protoc` alone is not enough, the
+# well-known includes (libprotobuf-dev) are needed too.
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    paths:
+      - crates/lance-graph-contract/**
+      - crates/lance-graph-dt64/**
+      - crates/lance-graph-dt64-lance/**
+      # The map log is written through the cycle writer over LanceCycleWriter.
+      - crates/lance-graph-supervisor/**
+      - crates/lance-graph-planner/**
+      - crates/lance-graph/Cargo.toml
+      - crates/lance-graph/src/graph/cycle_sink.rs
+      - .github/workflows/dt64-map-table-lance.yml
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+env:
+  CARGO_TERM_COLOR: always
+  CARGO_INCREMENTAL: "0"
+  CARGO_PROFILE_DEV_DEBUG: "0"
+  CARGO_PROFILE_TEST_DEBUG: "0"
+  RUST_BACKTRACE: "1"
+
+jobs:
+  map-table:
+    name: DR map table in Lance
+    runs-on: ubuntu-24.04
+    timeout-minutes: 45
+    defaults:
+      run:
+        working-directory: lance-graph
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          path: lance-graph
+          persist-credentials: false
+      # lance-graph's ndarray-hpc feature and the planner path-depend on
+      # ../../../ndarray.
+      - name: Checkout AdaWorldAPI/ndarray (sibling dependency)
+        uses: actions/checkout@v4
+        with:
+          repository: AdaWorldAPI/ndarray
+          path: ndarray
+          persist-credentials: false
+      - name: Install protoc and the protobuf includes (for lance)
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler libprotobuf-dev
+      - name: Setup rust toolchain (pinned by rust-toolchain.toml)
+        run: rustup show
+      # See dir-sim-persist.yml: spell target-cpu=native out as this runner's
+      # feature list so the cache never reuses an artifact on another CPU.
+      - name: Resolve target-cpu=native for this runner
+        id: cpu
+        run: |
+          flags=$(.github/scripts/resolve-native-rustflags.sh)
+          echo "RUSTFLAGS=$flags" >> "$GITHUB_ENV"
+          hash=$(printf %s "$flags" | sha256sum | cut -c1-16)
+          echo "flags=$hash" >> "$GITHUB_OUTPUT"
+      - uses: Swatinem/rust-cache@v2
+        with:
+          key: cpu-${{ steps.cpu.outputs.flags }}
+          workspaces: lance-graph/crates/lance-graph-dt64-lance
+      - name: fmt
+        run: cargo fmt --manifest-path crates/lance-graph-dt64-lance/Cargo.toml --check
+      - name: clippy
+        run: cargo clippy --manifest-path crates/lance-graph-dt64-lance/Cargo.toml --all-targets -- -D warnings
+      - name: test
+        run: cargo test --manifest-path crates/lance-graph-dt64-lance/Cargo.toml

Cargo.toml

@@ -176,6 +176,10 @@ exclude = [
     # (ogar-dir-core / ogar-ad / ogar-dir-sim). Verify via
     # `cargo test --manifest-path crates/lance-graph-dir-sim/Cargo.toml`.
     "crates/lance-graph-dir-sim",
+    # DR's versioned global -> local map table stored in Lance (D-DT64-7
+    # O5). Own [workspace], lance + arrow; dt64 stays lance-free. Verify via
+    # `cargo test --manifest-path crates/lance-graph-dt64-lance/Cargo.toml`.
+    "crates/lance-graph-dt64-lance",
     # Cognitive Compilation loop (claude/cognitive-compilation-lance-graph-h8sgym).
     # The NEW piece is the Elixir-shaped template — thinking styles, JITson, and
     # i4-32D thinking-style vectors already exist; the template was the gap. Four

crates/lance-graph-dt64-lance/Cargo.toml

@@ -0,0 +1,36 @@
+# lance-graph-dt64-lance — DR's versioned global -> local table (dt64
+# `dr::MapTable`) stored as a separate Lance-versioned dataset
+# (D-DT64-7 O5 / D10', D-LITHO Q5). EXCLUDED from the lance-graph workspace
+# (own [workspace] root), like lance-graph-dir-sim-lance: dt64 stays free of
+# lance and arrow.
+#
+# Advances are fire and forget: each one is a cycle submitted to the
+# supervisor's CycleWriter, whose single background task owns a
+# LanceCycleWriter (D-DR64-WAL-1). The advance is applied to the table only
+# after its seal ticket resolves Ok; a failed seal is a gap.
+#
+# Build/verify:
+#   cargo test --manifest-path crates/lance-graph-dt64-lance/Cargo.toml
+
+[package]
+name = "lance-graph-dt64-lance"
+version = "0.1.0"
+edition = "2021"
+publish = false
+license = "Apache-2.0"
+description = "DR's versioned global -> local map table made durable through the fire-and-forget cycle writer, replayed exactly at restore."
+
+[workspace]
+
+[dependencies]
+lance-graph-dt64 = { path = "../lance-graph-dt64" }
+lance-graph-contract = { path = "../lance-graph-contract" }
+lance-graph-planner = { path = "../lance-graph-planner" }
+lance-graph-supervisor = { path = "../lance-graph-supervisor", features = ["cycle-driver"] }
+# LanceCycleWriter, the WAL (graph::cycle_sink, behind `planner`). `ndarray-hpc`
+# is required: lance-graph does not build without it (its fallback is stale).
+lance-graph = { path = "../lance-graph", default-features = false, features = ["planner", "ndarray-hpc"] }
+
+[dev-dependencies]
+tokio = { version = "1", features = ["macros", "rt-multi-thread", "sync", "time"] }
+tempfile = "3"