Skip to content

DDIR: add the complete LDBC-derived SNB read benchmark through the server - #864

Merged
frankmcsherry merged 8 commits into
master-nextfrom
codex/ddir-ldbc-benchmark-stack
Sep 7, 2026
Merged

DDIR: add the complete LDBC-derived SNB read benchmark through the server#864
frankmcsherry merged 8 commits into
master-nextfrom
codex/ddir-ldbc-benchmark-stack

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Sep 7, 2026

Copy link
Copy Markdown
Member

DDIR: a complete LDBC-derived SNB read benchmark through the server

Based directly on master-next at b83dc32d, after #863 was squash-merged.
Both the engine prerequisites (#863) and chunk-merge correctness fix (#862)
are now upstream. This PR's own diff is 26 files / 2,916 added lines; no engine
source changes or remaining prerequisite stack.

Add a runnable read catalogue for the pinned SNB specification: IS1–7,
IC1–14 v2 and BI1–20. All 41 definitions are present, expressed through named
relational steps that emit ordinary DDP. No selected read is silently skipped.

The runner uses private real TCP ddir_server processes. BI bindings stand;
interactive bindings vary by request batch, join against installed dataflows,
and are retracted before graph changes. A maintained mode instead keeps all
bindings live. Shared graph inputs, bounded retraction/restoration, answer
checks, phase timings, hashes and failure artifacts make this a baseline for
subsequent engine/query-plan improvements. It adds no new prepared-query server
capability.

CI uses only a 199-row hand-authored witness, both backends, mixed one/four
workers and maintained four workers. It asserts answers, not timing thresholds,
and needs no downloaded data, generator, external database or Python packages.
Existing BI CSV composite-merged-fk snapshots can be supplied for larger runs.
The earlier four-query panel remains as a small independent-oracle control.

This is not an official LDBC score, driver workload or conformance claim.
Plans are installed once rather than reparsed per request; data oscillates
between two snapshots rather than growing or following an official arrival
stream. Default bindings are smoke parameters. Full-suite answers are checked
against fresh evaluation of the same logical plan, not an independent spec
implementation. Fourteen queries also have independent hand-checked expectations
(not necessarily a complete oracle per query); the other 27 have no independent
semantic oracle here. General aggregates, recursive hop bounds and unpacked strings
remain visible baseline costs. SF1 resource failures are not skipped or sold
as successful throughput measurements.

The emitted DDP plans for all 41 queries and the graph are byte-identical to the
post-squash baseline. Query logic and fixture rows are unchanged. Review fixes
change the harness and make the tiny IC3 alternate requester meaningfully
different, rather than hand-optimizing query definitions. BI19 deliberately
retains its unused unbounded rank as an optimizer-elimination target. BI13's
endpoint policy and inclusive calendar-month count are unchanged; the endpoint
interpretation remains documented follow-up work.

Documentation warns that RSS can fall under compression or swapping and
excludes Python (raising that ceiling alone does not ensure safe larger runs),
and that violating a shape contract can take down the shared server on either
backend. Feed acknowledgements do not validate shapes; the benchmark assumes
trusted programs and shape-correct data and starts private servers.
No overnight optimizer, selective-loader or profiling
prototypes have been folded into these definitions.

The six original benchmark commits were replayed onto the actual foundation
squash, with #862 included through that base. Two review-fix commits follow:

  • 1b37122f: one shared Spark BI CSV reader (quoted fields and quoted-field
    backslash escapes, plain/gzip partition support), setup-only snapshot delta
    computation, and separate timing components. Each phase now summarizes
    preparation, encoding, client-observed request/response, Value decoding and
    total client time. Comparisons default to wire_ms, not server CPU time;
    --metric client_ms selects the client total. Reports use version 2 /
    snb-suite-2; old reports must not be compared as if timing semantics and
    loading were unchanged. Establish a fresh baseline.
  • b07b86ed: IC3 hand-checked requester and retraction witnesses, an
    out-of-neighborhood requester in the tiny server bank, and accurate coverage/
    optimizer-opportunity documentation. No query plan optimization is included.

Fresh local validation of the review fixes passes: seven Python checks,
the four-query panel at one/four workers, all 41 reads in mixed mode at one/four
workers, and all 41 in maintained mode at four workers. Both backends are checked
in every benchmark configuration using the preserved optimized post-squash
server; Rust sources are unchanged. All five comparison metrics work for both
report families, and legacy reports are rejected. Loader-only checks confirm
all seven common projections agree on the existing SF0.003 snapshot (35,588
full-suite projected rows); a seeded 2,000-value quoted/unquoted round-trip
check also passes. The permanent CSV fixture covers escapes, delimiters, empty
and multiline fields, and compressed/plain partition copies.

The earlier eight server tests and both chunk-merge regressions passed on this
same Rust source; they were not rebuilt for Python-only corrections. All fresh
server runs use tiny fixtures under a physical-footprint guard. No SF1 run or
performance improvement claim is part of this revision. Hosted CI remains a
separate gate.

@frankmcsherry
frankmcsherry force-pushed the codex/ddir-ldbc-benchmark-stack branch from cf19a71 to 6ff8a78 Compare September 7, 2026 15:31
@frankmcsherry
frankmcsherry changed the base branch from codex/ddir-ldbc-prerequisites to master-next September 7, 2026 15:31
Use one Spark BI CSV reader, compute signed graph deltas once at setup, and summarize client preparation, encoding, request/response, decoding and total timings separately. Version the reports and make comparisons select an explicit component, defaulting to request/response time.
Add a hand-checked IC3 request and retraction witness and vary the tiny server bank to an out-of-neighborhood requester. Keep BI19's unused rank as an optimizer target and leave BI13's endpoint policy unchanged, documenting both.
@frankmcsherry
frankmcsherry merged commit 75fba2b into master-next Sep 7, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant