perf(rust): make the uuid_on_collision merge pass near-linear

[SkyeAv]

The uuid_on_collision: merge fold re-canonicalized and re-sorted every already-stored list item on each incoming row — quadratic in divergent rows per edge — which made merge-mode graphs a build bottleneck: 37.5 s for a 100k-line workload, >98% of it in the fold. This makes the fold near-linear and the canonical serializer allocation-free, with byte-identical output proven against frozen copies of both old implementations: the worst-case shape is 28.6x faster and the default error mode is 14.6% faster.

Merge fold (rust/src/ndjson.rs)

• Quadratic kill: merge_records re-canonicalized all stored list items, linear-scanned Vec::contains membership, and re-sorted the whole array on every fold. Each array field now carries a ListState — membership FxHashSet<SharedBytes>, parallel ordered Vec<SharedBytes>, unioned flag: canonical bytes are computed once when an item arrives, every fold is an O(1) membership check per incoming item, and a single deferred stable sort runs at write-out only for fields that actually went through a union (copied-never-unioned lists keep source order).
• One allocation per distinct item: SharedBytes is Rc<[u8]> shared between the set and the vec — Hash/Eq delegate to the full canonical bytes, never a bare hash (matching the record_if_new precedent). Peak RSS on the scenario-C shape: 1931 MB → 1256 MB (−35%).
• O(1) repeat suppression: MergeIndex::absorb content-hash membership is now an FxHashSet<u64>; exact byte repeats are still suppressed without incrementing merged or re-running the fold.
• Carried over verbatim: the number_of_cases recompute with its −1 conflict unwind, first-wins scalars, id-skip, supporting_case_ids carrier strip (including serde_json preserve_order's swap_remove key ordering), and fail-loudly malformed-JSON handling.
• New fail-loudly guard: MergedRecord::finish raises a structured merge-state-desync runtime_error instead of silently truncating if the bytes/items parallel invariant ever breaks; pinned by a negative test.

Canonical serialization (rust/src/json.rs)

• Direct-to-buffer: canonical_json_bytes writes sorted-key JSON straight into the output buffer with the same key comparator; scalars delegate to serde_json::to_writer, so escaping and number formatting are byte-identical by construction. The deep Value clone is gone from production.
• Default mode benefits too: the error-mode edge content hash flows through this serializer — 598k-line error-mode workload: ~3.54 s → ~3.03 s (−14.6%). The node path never touches it (finalize_record early-returns for !is_edges; record_if_new keys on emitted_json_bytes) and measured flat (−1.7%) as a regression guard.

Equivalence proof

• Frozen oracles: byte-verbatim #[cfg(test)] copies of the old fold (merge_records_reference / MergeIndexReference) and old serializer (canonical_json_bytes_reference) are the differential targets.
• Seeded fuzz: merge_fold_matches_reference_on_fuzz and canonical_json_bytes_matches_reference_on_fuzz drive old-vs-new on reproducible randomized streams — divergent groups, object/scalar list items, key-order variants, late fields including copied-never-unioned lists, scalar-vs-array conflicts, exact repeats, unicode/exponent number forms — asserting byte-identical records, first-seen order, and (merged, scalar_conflicts) counters.
• Python end-to-end: tests/test_rs.py::test_dedup_edges_merge_matches_python_reference re-implements the semantics in pure Python and asserts byte-identical rs.dedup_ndjson output with matching counters.
• Executable speed floor: merge_fold_speedup_bound_vs_reference times the new fold against the frozen quadratic reference in-process and fails below 5x (measured 21–31x) — machine-independent; no absolute wall-clock assertions in CI.

Measurement

Release builds (uv run maturin develop --release), seeded corpora (seed 42) from the committed harness tests/bench_merge_bench.py (opt-in TABLASSERT_BENCH=1), same machine before/after:

Workload	Before	After
100k lines · 1k triples × 100 divergent rows · 50 case ids/row	37.45 s	1.311 s	28.6x
100k lines · 5k triples × 20 rows · 100 case ids/row	20.69 s	2.155 s	9.6x
600k lines · 200k triples × 3 rows	6.71 s	5.184 s	1.29x
100k byte-identical rows (no fold)	0.59 s	0.495 s	1.19x
598k lines · default error mode	~3.54 s	~3.03 s	−14.6%

merged/scalar_conflicts counters identical on every scenario; the after wall-clocks are conservative (post-Rc re-timing measured C at 1.114–1.185 s).

Design

• Algorithms, not parallelism: first-seen output order and first-wins scalars are inherently sequential; with the quadratic killed the worst shape lands under 1.5 s, so sharding or threading the pass is not justified. Deferred: parallelization, spill-to-disk.
• Accepted memory tradeoff: the buffered state holds ~one canonical-bytes copy per distinct list item — merge mode is opt-in and already buffers full records, and the Rc sharing halves the naive two-copy shape (RSS −35% measured).
• No API/config/dependency change: dedup_ndjson's signature and counters are unchanged; rust/src/uuid.rs (golden UUID vectors) and rust/src/fullmap.rs are untouched.

Docs

• CHANGELOG.md: ### Performance entry under Unreleased with the measured numbers, the reproduction command, and the memory note. No other docs change — behavior is identical, so docs/configuration/graph.md and models.py remain accurate.

Testing

• make check → EXIT=0
• uv run pytest → 1165 passed, 44 skipped, 0 failed
• cargo test --manifest-path rust/Cargo.toml → 131 passed, 0 failed (1 ignored: regenerate_golden)
• cargo test --manifest-path rust/Cargo.toml fold_speedup -- --nocapture → new 114.9 ms vs reference 3.43 s → 29.9x (bound 5x)
• uv run ruff check . / uv run ruff format --check . / uv run pyright / cargo fmt --check / cargo clippy --all-targets -- -D warnings → all clean
• TABLASSERT_BENCH=1 uv run --no-sync pytest tests/bench_merge_bench.py -s -n 0 -q → 6 passed (A/B/C/DUP/E/N)
• Independent full-audit code review → PASS after two remediation rounds (error-mode measurement added; desync-guard hardened); the auditor's own 600-seed adversarial differential fuzz → 0 divergences

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 3a22b401-a75f-45a2-a94b-de17bb40ec1f

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}