Skip to content

Benchmark target, README that matches it, and release 1.1.0 - #56

Merged
marcobambini merged 2 commits into
mainfrom
docs/benchmark
Aug 24, 2026
Merged

Benchmark target, README that matches it, and release 1.1.0#56
marcobambini merged 2 commits into
mainfrom
docs/benchmark

Conversation

@marcobambini

Copy link
Copy Markdown
Member

Two commits: a reproducible benchmark plus the README changes its numbers force,
then the 1.1.0 version bump and changelog entry.

make benchmark

New test/benchmark.c and target. k=20 over 1,000,000 vectors of dimension 768,
cosine, across every storage and quantization mode, with recall scored against
the exact full-precision scan
— speed without recall says nothing. Built at
-O3 with the same per-translation-unit ISA flags the shipped extension uses,
so it measures the kernels that actually ship rather than the scalar fallback.
Fixed xorshift seed, so two machines compare like for like.

make benchmark
make benchmark NVECS=100000 DIM=384 K=10 DISTANCE=l2

The README carries the Apple M5 Pro run and invites results from other CPUs.

What the numbers changed in the README

For cosine, INT8 is the mode to use and UINT8 is not. Same size, same
speed, 33.8% recall against 99.5%. Unsigned quantization subtracts the dataset
minimum before scaling, and cosine measures angle, which that shift destroys.
Omitting qtype selects UINT8 for non-negative data — correct for L2, wrong
for cosine — so this is a real trap and the README now says so in both places
someone would look.

1BIT is a pre-filter, not a ranking. 377 Mvec/s and 30× less memory reads
like a headline; it is 10% recall on this data. Framed accordingly.

TurboQuant's argument is memory, not speed. TURBO4 is marginally slower
than the exact scan here while using 8× less.

That last one required touching an existing section. It claimed 15× and 38×
speedups for TurboQuant. Those were measured file-backed, where the baseline
reads 3 GB off disk and the comparison is dominated by I/O rather than
arithmetic, and before the kernel rewrites made the full-precision scan much
faster. I have not deleted them or called them wrong — both measurements are
real and answer different questions, and the section now says which is which. If
your working set does not fit in RAM, the file-backed comparison is the one that
describes your deployment.

The benchmark data is uniform random, the worst case for every quantizer;
real embeddings have structure quantization exploits. Both the tool's output and
the README say so, because a recall column without that caveat reads as a
prediction rather than a floor.

Also documents the quantization modes as a table (previously discoverable only
from API.md), that HAMMING is rejected for non-BIT types, that BIT columns
accept only 1BIT, and that normalized=1 does something.

1.1.0

SQLITE_VECTOR_VERSION drives both make version and vector_version().
Package.swift is left alone — its release URL and checksum are rewritten by the
[auto-update] commit after a tag is published.

The changelog entry covers the audit and flags the two behaviour changes worth
knowing before upgrading: tie-breaking among equal distances is different, and
qtype=AUTO on a BIT column now means 1BIT instead of failing or silently
recording UINT8.

🤖 Generated with Claude Code

marcobambini and others added 2 commits August 24, 2026 23:02
…sures

Adds test/benchmark.c and a `make benchmark` target: k=20 over 1,000,000
vectors of dimension 768, cosine, across every storage and quantization mode,
with recall scored against the exact full-precision scan. Built at -O3 with the
same per-translation-unit ISA flags the shipped extension uses, so it measures
the kernels that actually ship rather than the scalar fallback. Data comes from
a fixed xorshift seed so two machines compare like for like.

Measured on an Apple M5 Pro, and the numbers change three things the README
said or implied.

* For cosine, INT8 is the mode to use and UINT8 is not: same size, same speed,
  33.8% recall against 99.5%. Unsigned quantization subtracts the dataset
  minimum before scaling, and cosine measures angle, which that shift destroys.
  Since omitting qtype picks UINT8 for non-negative data, this is a real trap,
  so the README now says it in the two places someone would look.

* 1BIT reads as a headline number - 377 Mvec/s, 30x less memory - and is 10%
  recall on this data. It is a pre-filter to re-rank, and the README now frames
  it that way rather than as a ranking mode.

* TurboQuant's argument is memory, not speed. TURBO4 is slightly slower than the
  exact scan here while using 8x less. The existing section claimed 15x and 38x
  speedups; those were measured file-backed, where the baseline is reading 3 GB
  off disk rather than doing arithmetic, and before the kernel rewrites made the
  full-precision scan itself much faster. Both measurements are real and answer
  different questions, so that section now says which is which instead of
  quietly leaving a number that no longer describes an in-memory deployment.

The data is uniform random, which is the worst case for every quantizer - real
embeddings have structure quantization exploits. The section says so, because a
recall column without that caveat reads as a prediction rather than a floor.

Also documents the quantization modes as a table (they were only discoverable
from API.md), that HAMMING is now rejected for non-BIT types rather than
crashing, that BIT columns only accept 1BIT, and that normalized=1 has an
effect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bumps SQLITE_VECTOR_VERSION, which `make version` and vector_version() both
read, and writes the 1.1.0 changelog entry covering the audit: thirteen defects
including two crashes and four memory-safety issues, the x86 builds that had
been shipping scalar code, the kernel and top-k rewrites, and the two behaviour
changes worth knowing about before upgrading — tie-breaking among equal
distances, and qtype=AUTO on a BIT column now meaning 1BIT.

Package.swift is left alone: its release URL and checksum are rewritten
automatically after a tag is published.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@marcobambini
marcobambini merged commit 7cd6b6e into main Aug 24, 2026
17 checks passed
@marcobambini
marcobambini deleted the docs/benchmark branch August 24, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant