Skip to content

Benchmark: pivot the table around hardware, and stop measuring in memory #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
marcobambini merged 3 commits into from
Aug 25, 2026
Merged

Benchmark: pivot the table around hardware, and stop measuring in memory #58

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a675b0d
docs: pivot the benchmark table around hardware
marcobambini Aug 25, 2026
7391418
docs: benchmark on a file-backed database, never in memory
marcobambini Aug 25, 2026
5e2fb94
docs: make a wrong benchmark row hard to produce
marcobambini Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,11 +188,14 @@ unittest-simd: $(BUILD_DIR)/backend $(BUILD_DIR)/test_vector_simd
# make benchmark k=20 over 1M vectors of dim 768
# make benchmark NVECS=100000 DIM=384 K=10 smaller, for a quick look
# make benchmark DISTANCE=l2 a different metric
# make benchmark HARDWARE="Apple M5 Pro" name the machine; the backend is appended
NVECS ?= 1000000
DIM ?= 768
K ?= 20
NQUERIES ?= 20
DISTANCE ?= cosine
# names the row this run contributes to the README hardware table
HARDWARE ?=

BENCH_OBJ = $(patsubst %.c, $(BUILD_DIR)/bm-%.o, $(notdir $(SRC_FILES))) $(BUILD_DIR)/bm-sqlite3.o

Expand All @@ -203,7 +206,7 @@ $(BUILD_DIR)/bm-%.o: %.c
$(CC) $(CFLAGS) $(ISA_CFLAGS) -DSQLITE_CORE -O3 -c $< -o $@

$(BUILD_DIR)/benchmark: test/benchmark.c $(BENCH_OBJ)
$(CC) $(CFLAGS) -DSQLITE_CORE -O3 -DNVECS=$(NVECS) -DDIM=$(DIM) -DK=$(K) -DNQUERIES=$(NQUERIES) -DDISTANCE='"$(DISTANCE)"' $< $(BENCH_OBJ) -o $@ -lm -lpthread
$(CC) $(CFLAGS) -DSQLITE_CORE -O3 -DNVECS=$(NVECS) -DDIM=$(DIM) -DK=$(K) -DNQUERIES=$(NQUERIES) -DDISTANCE='"$(DISTANCE)"' -DHARDWARE='"$(HARDWARE)"' $< $(BENCH_OBJ) -o $@ -lm -lpthread

benchmark: $(BUILD_DIR)/benchmark
./$(BUILD_DIR)/benchmark
Expand Down
155 changes: 97 additions & 58 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,67 +147,108 @@ SELECT e.id, v.distance FROM images AS e

## Benchmark

Every number below comes from one command, so you can reproduce it and compare machines:
To add your machine to the table, one command — pass the CPU name and nothing else:

```bash
make benchmark
make benchmark HARDWARE="Apple M5 Pro"
```

That builds `test/benchmark.c` at `-O3` with the same per-translation-unit SIMD flags the
shipped extension uses, then searches **k=20 over 1,000,000 vectors of dimension 768**
with cosine distance, 20 queries, reporting the best. Recall is the overlap with the exact
full-precision top-20. Override any of it:
It builds `test/benchmark.c` at `-O3` with the same per-translation-unit SIMD flags the
shipped extension uses, runs **k=20 over 1,000,000 vectors of dimension 768** with cosine
distance and 20 queries reporting the best, and prints two rows ready to paste, unedited,
into the table below.

Two things it does so a pasted row cannot be wrong. The backend is **appended by the
binary** from what the build actually selected, not typed by hand, so a row cannot claim
`AVX512` on a build that fell back to `SSE2`. And a run whose parameters differ from the
ones the table is built on **prints an explanation instead of rows**, because a row
measured on a different workload would sit in that table looking comparable without being
comparable.

That second guard exists because the parameters *are* adjustable, just not for this table:

```bash
make benchmark NVECS=100000 DIM=384 K=10 DISTANCE=l2
```

### Apple M5 Pro (6P+12E, 64 GB, macOS 26.6.2) — NEON backend
That run prints the mode table for whatever you asked for, and no paste-ready rows.

Common to every row: the database is **a file, never `:memory:`** — an in-memory database
puts the whole index in the process no matter how it is configured, which makes any memory
figure meaningless. Vectors are uniform random with a fixed seed, so two machines measure
the same data. The `INT8` index is **740 MB** on disk against 2930 MB of raw `FLOAT32`.
Recall is the overlap with the exact `FLOAT32` scan, the baseline everything is compared
against, 100% by definition; on the reference machine that scan takes **484 ms/query**,
because it reads 3 GB per query.

The two rows per machine are the two ways the same index gets deployed. *Preloaded* holds
it in the process after `vector_quantize_preload()`. *Streamed* walks it through a bounded
buffer set by `max_memory=30MB`, the default, and what a device with less RAM than the
index actually does. **Max memory** is measured, not the parameter echoed back: it is the
peak the extension and SQLite had allocated during the scan.

### Hardware

| Hardware | Vectors | Index | Max memory | ms/query | Mvec/s | Recall@20 |
| --- | ---: | --- | ---: | ---: | ---: | ---: |
| Apple M5 Pro - NEON | 1,000,000 | `INT8` preloaded | 740 MB | 37.6 | 26.6 | 99.5% |
| Apple M5 Pro - NEON | 1,000,000 | `INT8` streamed | 30 MB | 114.4 | 8.7 | 99.5% |

*Results from other CPUs welcome — run the command above and open a PR adding the two rows
it prints.*

The trade is **25x less memory for 3x the latency**, with recall untouched: both rows read
the same index, only how much of it is resident differs.

| Mode | Index | ms/query | Mvec/s | Recall@20 |
Two things the timings do not show. They are best-of-20, so the file is in the operating
system's page cache by then — that cache lives outside the process and is evicted under
pressure, so it is not in the memory column, but it is why the streamed row is not paying
for storage reads. On a device where the index genuinely does not fit in RAM, add
`index size / storage bandwidth` to the streamed number. And the preloaded row is almost
unaffected by where the database lives, because after the one-time preload the scan reads
the extension's own buffer and never goes back to SQLite.

Recall is repeated on every row on purpose: it depends on the data, not the hardware, so a
row that disagrees with the others is a sign that machine selected a different SIMD
backend than it should have.

### Choosing a mode

`INT8` is in the table above because it is the mode to reach for first. The others, same
machine, same data, all preloaded:

| Mode | Index | ms/query | vs exact | Recall@20 |
| --- | ---: | ---: | ---: | ---: |
| `FLOAT32` exact | 2930 MB | 147.8 | 6.8 | 100.0% |
| `UINT8` | 740 MB | 55.4 | 18.0 | 33.8% |
| `UINT8` preloaded | 740 MB | 37.2 | 26.9 | 33.8% |
| `INT8` | 740 MB | 56.4 | 17.7 | 99.5% |
| **`INT8` preloaded** | **740 MB** | **37.7** | **26.5** | **99.5%** |
| `1BIT` | 99 MB | 5.3 | 187.5 | 10.0% |
| `1BIT` preloaded | 99 MB | 2.7 | 377.6 | 10.0% |
| `TURBO2` | 195 MB | 53.0 | 18.9 | 45.2% |
| `TURBO2` preloaded | 195 MB | 48.2 | 20.7 | 45.2% |
| `TURBO4` | 378 MB | 160.4 | 6.2 | 81.8% |
| `TURBO4` preloaded | 378 MB | 151.8 | 6.6 | 81.8% |

*Contributions from other CPUs welcome — run the command above and open a PR adding a
section.*

### Reading the table

**The data is uniform random**, which is the worst case for every quantizer: real
embeddings have structure that quantization exploits, so recall on your own vectors will
be higher, often much higher. Treat the recall column as a floor and a way to rank the
modes against each other, not as a prediction for your dataset.

Three things are worth knowing before you pick a mode.

**For cosine, use `INT8`, not `UINT8`.** They cost exactly the same and store exactly the
same number of bytes, but `UINT8` recall collapses to 33.8% while `INT8` holds 99.5%.
| `FLOAT32` exact | 2930 MB | 484.4 | 1.0x | 100.0% |
| `UINT8` | 740 MB | 37.3 | 13.0x | 33.8% |
| `INT8` | 740 MB | 37.6 | 12.9x | 99.5% |
| `1BIT` | 99 MB | 2.5 | 195x | 10.0% |
| `TURBO2` | 195 MB | 48.0 | 10.1x | 45.2% |
| `TURBO4` | 378 MB | 151.5 | 3.2x | 81.8% |

**The data is uniform random**, the worst case for every quantizer: real embeddings have
structure quantization exploits, so recall on your own vectors will be higher, often much
higher. Read that column as a floor and a way to rank the modes, not as a prediction.

Three things are worth knowing before choosing.

**For cosine, use `INT8`, not `UINT8`.** Same size, same speed, 33.8% recall against 99.5%.
Unsigned quantization subtracts the dataset minimum before scaling, and cosine measures
angle, which that shift destroys. `UINT8` is the right choice for L2, where a common
translation cancels out. If you do not set `qtype`, the extension picks `UINT8` for
non-negative data and `INT8` otherwise — which is the correct call for L2 and the wrong
one for cosine, so set it explicitly when you use cosine.

**`1BIT` is a filter, not an answer.** 377 Mvec/s and 30x less memory, at 10% recall on
this data. It earns its place as a first pass whose survivors you re-rank at full
precision, not as the final ranking.

**TurboQuant trades speed for size, not for speed.** `TURBO4` here is *slower* than the
exact scan (160 ms against 148 ms) while using 8x less memory and returning 81.8% recall.
The lookup-table scan is one table gather per row, and at dimension 768 that is 384
gathers into a 393 KB table for every vector — already about one lookup per cycle, so
there is no headroom left in the current storage layout. Choose TurboQuant when the
memory budget is what binds; choose `INT8` when throughput is.
angle, which that shift destroys. `UINT8` is right for L2, where a common translation
cancels. If you omit `qtype` the extension picks `UINT8` for non-negative data — correct
for L2, wrong for cosine — so set it explicitly when you use cosine.

**`1BIT` is a filter, not an answer.** 195x faster than exact and 30x smaller, at 10%
recall here. It earns its place as a first pass whose survivors you re-rank at full
precision.

**TurboQuant buys memory against `INT8`, not speed.** `TURBO4` is 3.2x faster than the
exact scan, so it is a real win over brute force — but `INT8` is 4x faster again at twice
the size, and `TURBO2` is both smaller and faster than `TURBO4` if 45% recall is enough.
TurboQuant's lookup scan is one table gather per row: at dimension 768 that is 384 gathers
into a 384 KB table per vector, already about one lookup per cycle, so its current storage
layout has no headroom left ([#57](https://github.com/sqliteai/sqlite-vector/issues/57)).
Reach for it when the memory budget is what binds.

## TurboQuant Benchmark and Recall

Expand All @@ -221,15 +262,13 @@ SELECT vector_quantize('images', 'embedding', 'qtype=TURBO,qbits=4');
SELECT vector_quantize('images', 'embedding', 'qtype=TURBO2');
```

An earlier synthetic benchmark reported speedups of 15x for 4-bit and 38x for 2-bit
against `vector_full_scan()`. Those numbers were measured with a **file-backed** database,
where the full scan reads 3 GB of raw vectors off disk and the comparison is dominated by
I/O rather than by arithmetic — and before the distance kernels were rewritten, which made
the full-precision scan itself substantially faster. Against an in-memory baseline on
current code the picture is different: see [Benchmark](#benchmark) below, where `TURBO4`
is marginally slower than the exact scan and its argument is memory, not speed. Both
measurements are real; they answer different questions. If your working set does not fit
in RAM, the file-backed comparison is the one that describes your deployment.
An earlier synthetic benchmark on this dataset reported speedups of 15x for 4-bit and 38x
for 2-bit against `vector_full_scan()`, with DOT distance and k=10. The [Benchmark](#benchmark)
section above measures the same shape with cosine and k=20 and lands lower — 3.2x for
4-bit, 10.2x for 2-bit — mostly because the distance kernels have since been rewritten,
which made the full-precision baseline it is compared against substantially faster. The
direction is the same: TurboQuant beats brute force, and `INT8` beats TurboQuant on speed
while costing twice the memory.

For comparison, the raw `FLOAT32` vectors alone are about **3.07 GB** for 1M x 768 before SQLite row/page overhead. TurboQuant 4-bit reduces the scan representation to about **13%** of that raw vector payload, TurboQuant 3-bit to about **10%**, and TurboQuant 2-bit to about **7%**. Actual resident memory depends on whether the database is in-memory or file-backed, SQLite cache settings, preloading, page cache behavior, and the host allocator.

Expand Down
Loading
Loading