Repository navigation
feat(theta,tuple): serialize union results directly - #336
Merged
tisonkun merged 9 commits intoNov 10, 2026
Merged
Conversation
Add compact serialization to ThetaUnion and TupleUnion with shared result filtering, trimming, and ordering. Reuse each family's compact encoder and borrow Tuple entries to avoid cloning summaries during serialization. Cover canonical results, custom seeds, summary clone avoidance, and subsequent merges. Compare direct serialization with compact-sketch serialization for scalar and heap-backed summaries. Co-authored-by: Nash Aschenbath <nashaschenbath@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ZENOTME
reviewed
Nov 9, 2026
|
|
||
| pub(super) fn serialize_uncompressed<E>( | ||
| state: &CompactSketchState<E>, | ||
| hash: impl Fn(&E) -> u64, |
Contributor
There was a problem hiding this comment.
How about
pub(super) fn serialize_uncompressed<E: SketchEntry>(
state: &CompactSketchState<E>,
) -> Vec<u8>
to omit hash
| } | ||
|
|
||
| let entries = state.retained_entries(); | ||
| let pre_longs = if state.is_estimation_mode() { |
Contributor
There was a problem hiding this comment.
Better extract preamble_longs as before
| pub const EMPTY_SKETCH_BYTES: [u8; 8] = [1, 3, 9, 1, 0, 0x1e, 0, 0]; | ||
|
|
||
| /// Serializes compact Tuple state whose retained entries are owned or borrowed. | ||
| pub(super) fn serialize_compact_state<S, E>(compact_state: &CompactSketchState<E>) -> Vec<u8> |
Contributor
There was a problem hiding this comment.
I can't get the meaning of the name serialize_compact_state and serialize_uncompressed. Looks like the diff is how they serialize entry. Maybe need to refine.
tisonkun
marked this pull request as ready for review
November 10, 2026 06:59
Implement the internal SketchEntry trait for retained u64 hashes so the compact encoder can read both compact-sketch hashes and union entries through the same interface without a hash projection callback.
Name both family encoders serialize_compact_uncompressed to describe the compact representation and the uncompressed wire format consistently. Keep the family-specific headers and entry encoding in their respective serialization modules.
Derive the compact preamble length from the canonical sketch state in one place for both family encoders: one 8-byte word for empty images and exact singletons, two for other exact results, and three for estimation results. The helper is specific to uncompressed formats. Compressed Theta has a different header layout, so its size calculation belongs with that codec rather than behind a compression flag on the shared helper.
Add a larger exact result and estimation workloads with lg_k 8 and 16 alongside the default-capacity cases. Compare ordered and unordered serialization for scalar and heap-backed summaries with fixed rounds so changes to result preparation can be checked across retained sizes.
Global comparison sorting repeatedly dereferences the union hash table when ordering borrowed entries. Partition large ordered results by their top active hash bits, scaled to the result theta, and sort within each bucket to reduce those indirect comparisons without cloning summaries. Keep small and unordered results on the existing compaction path and keep shared compaction inlineable. Large ordered results use one extra reference buffer, or 32 KiB for 4096 retained entries on 64-bit targets. Cover ordering with spread and clustered hashes, small-result boundaries, and larger serialized results whose heap-backed summaries must not be cloned. Size the lg_k 8 benchmark rounds for the measured workload.
Use shared compaction to trim and sort borrowed entries in one path. Remove the bucket partition, size cutoff, temporary reference buffer, and associated inline hints to keep result construction straightforward. Retain summary borrowing, expanded serialization benchmarks, and integration coverage for large results, zero summary clones, and subsequent merges.
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
ThetaUnion::serialize(ordered)andTupleUnion::serialize(ordered)so callers can serialize union results directly. Tuple serialization borrows retained summaries, avoiding the clones and associated allocations required byto_sketch(ordered).serialize().Owned and borrowed result paths share theta filtering, nominal-capacity trimming, and comparison sorting for ordered results. Each family's compact encoder serves both compact sketches and unions. Theta output is byte-identical to
to_sketch(ordered).serialize(); Tuple output is byte-identical when ordered and semantically equivalent when unordered, where entry order may differ.Tuple serialization collects references to retained entries, trims and optionally sorts those references, and writes the summaries through the shared encoder. It does not clone summaries.
Combines #322 and #333 by @Aschenbath, retaining contributor attribution in the commit.
Closes #320.
Closes #321.
Validation
cargo x checkacross the feature matrix passed.cargo x test: 767 tests, including doctests, passed with the pinned C++, Go, and Java fixtures.cargo x lintpassed.cargo x bench -- serde --testpassed, including all 64 Theta/Tuple implementation cases.lg_kvalues of 8, 12, and 16.Performance
Measured at
1f8c314d1f614c66a00bf607b715d5569a27245fon Apple M4 Max, macOS 26.6.2, Rust 1.101.0-nightly (8006091eb). Each scenario below uses 80,000 inputs and the defaultlg_k=12, retaining 4,096 entries. The labels summary holds four strings per key.Run each scenario with
cargo x bench -- <scenario> --exact --raw, usingtuple::serde::u64::ordered/80000,tuple::serde::labels::ordered/80000,tuple::serde::u64::unordered/80000, andtuple::serde::labels::unordered/80000. Each scenario reuses its union for the fixed number of serializations; timing includes fixture construction and output destruction. Results are median milliseconds and min–max over five interleaved repetitions per scenario. Different rows use different round counts and are not a throughput ranking.Heap-backed summaries benefit from avoiding clones: direct serialization takes about 20% of the compact-sketch path's time when ordered and 15% when unordered in these workloads. Direct serialization is not universally faster: scalar summaries are inexpensive to clone, and the measured ordered and unordered scalar cases take about 5% and 17% longer, respectively. These results are workload- and toolchain-dependent and do not establish a library-wide speedup.