Skip to content

perf(theta): pack compressed sketches word by word - #342

Merged
tisonkun merged 3 commits into
apache:mainfrom
davecromberge:perf-theta-compressed
Nov 10, 2026
Merged

tisonkun merged 3 commits into
apache:mainfrom
davecromberge:perf-theta-compressed

Conversation

@davecromberge

@davecromberge davecromberge commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Replaces the unrolled per-width bit-packing module with word-based packing and unpacking for compressed (serial version 4) Theta sketches. The serialized bytes are unchanged.

  • Compressed deserialize 2.2–4.2× and serialize 1.1–2.2× faster across sketch sizes.
  • Removes bit_pack.rs (5,401 lines) and adds a round-trip test for every entry width.
  • A clean build of the crate with only the theta feature takes 0.45 s instead of 1.35 s in release mode, and 0.38 s instead of 0.78 s in debug mode.
  • The compiled .rlib (theta feature only) shrinks from 1.66 MB to 0.86 MB.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tisonkun
tisonkun requested review from tisonkun and a balanced review from Copilot November 10, 2026 06:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the compressed Theta serialization changes.

No open issues 🎉

What changed in this PR

This PR optimizes compressed Theta sketch serialization and deserialization with word-based bit packing while preserving the serial version 4 byte format.

Changes:

  • Replaces block-based packing and unpacking with word-based operations.
  • Adds round-trip and expected-payload checks for all entry widths from 1 to 63 bits.
  • Removes the obsolete module declaration and documents the performance improvement.
File Description
datasketches/​src/​thetafamily/​theta/​sketch.rs Implements word-based packing and unpacking and adds entry-width coverage.
datasketches/​src/​thetafamily/​theta/​mod.rs Removes the bit_pack module declaration.
CHANGELOG.md Records faster compressed Theta serialization and deserialization.

Review effort: Balanced

@tisonkun tisonkun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@tisonkun
tisonkun merged commit c344dad into apache:main Nov 10, 2026
10 checks passed
@davecromberge

Copy link
Copy Markdown
Member Author

@tisonkun thanks for your review and for adding the extra safeguard for invalid hashes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants