[eterceros]

At a glance

SQLite search by token, quantity, composite and date parameters reaches about 30 times the throughput at 30 virtual users (about 11 times lower p50 at 1 user), with identical results and no large change in write cost.

What	Result	main	this PR	Evidence
Search, 30 VU, requests/s	IMPROVES x31 (x33 on the current main)	13.9	436.1	4 pairs, every B beat every A; 1 pair on current main
Search, 30 VU, p50 / p95	IMPROVES -96.6 % / -96.8 %	1,596 / 5,463 ms	54 / 174 ms	4 pairs; current main: -95.1 % / -97.4 %
Search, HFS CPU per request	IMPROVES -96.8 %	479 ms	15.1 ms	4 pairs
Search, 1 VU, p50 / p95	IMPROVES -91.7 % / -98.8 %	11.4 / 752 ms	1.0 / 8.8 ms	4 pairs
Peak resident memory (search)	IMPROVES -56.6 %	4,655 MB	2,021 MB	4 pairs
CRUD, 300 VU, HFS CPU per request	IMPROVES -2.4 % (small, mean of 7 pairs)			better in 6 of 7 pairs; the exception is one pair on the current main: EMPEORA +5.5 %
CRUD, 300 VU, requests/s	neutral	566	605	+7.5 %, pairs -12.4 to +29.1
CRUD, bytes written per request	neutral	403,776	390,818	-3.1 % +- 4.3
Search results	unchanged			22 URLs x 3 pages, 0 differences, twice

Fixes #2212
Part of #2180

What changed

14 commits, 7 files, all under crates/persistence (SQLite backend and README); SQLite schema 41 -> 46.

• Covering search indexes (v42). idx_search_quantity, idx_search_token and the new idx_search_composite_cov carry resource_key, so a search reads the index alone. idx_search_token is keyed code-first (value_token_code, resource_key, value_token_system, ...), so a token search without a system can seek it. Stale shapes are rebuilt at startup.
• Bounded first-page probe for one-parameter searches: it reads at most 2,001 index rows and, when the key list is short, drives the page from it and reads resources by rowid. HFS_SQLITE_SEARCH_PROBE=0 (false, off, no) turns it off; read once, default on; documented in the README.
• Early-exit recency window for broad one-parameter first pages (newest 512 resources, retried on the newest 4,096 when the page cannot be filled), now including UNION ALL date shapes, which used to materialise 146k-292k keys.
• Code-first composite pairing (EXISTS) for token+quantity composites.
• Drop the two prefix indexes (v45): idx_resources_type and idx_history_resource, only when their stored SQL is exactly the historical one.
• Write path: create and restore_deleted run in one BEGIN IMMEDIATE transaction (it used to auto-commit ~17 statements; a failure could leave a resource without its index) and a resource's main search rows go in with multi-row INSERTs (up to 512 rows per statement) instead of groups of eight plus single rows.
• v43 and v44 are deliberate no-ops that keep the version numbers: an earlier build of this work created two more indexes there, which cost about a third more WAL per create and were replaced by reshaping idx_search_token in v46. v46 drops them if present. No migration step v2-v41 is edited; the fresh-database base schema (create_indexes) no longer creates the two prefix indexes.

Why it is safe

• Results: 0 differences against main over 22 URLs x 3 pages (11 distinct totals, 16 distinct first ids, so the check is not vacuous), on the original base and again on the current main.
• Migrations are DDL that can be repeated: the version is stamped after each step, a crash between a DROP INDEX and its CREATE INDEX leaves the database at the old version and the next start redoes it, and the startup check recreates a missing index.
• Prefix indexes are dropped only when their stored definition matches exactly; an index of the same name with another shape is left alone (tests cover both).
• No file other than the SQLite backend and the persistence README changed. The backend code is the code that was measured: after series 1 only comment, README and rustfmt commits were added; series 2 measured the tip with the comment and README commit, and the final formatting-only commit came after it.

Evidence

Local shared 8-core box, 30 % corpus slice, ratios only, fresh copy of the same database template per run, runs one at a time under a FIFO lock. A = origin/main, B = this change.

Series 1: 4 ABAB pairs per suite, A = origin/main 678fc50 (v41), B = the same 11 SQLite commits before the rebase (v46). Search 30 VU x 120 s and 1 VU x 60 s, CRUD 300 VU x 90 s. 0 HTTP errors, 0 database is locked, foreign CPU <= 0.2 cores except one run (0.64).

Search metric	A	B	Change	Pairs
30 VU requests/s	13.9	436.1	IMPROVES +3,046 % (x31)	4/4, every B beat every A (A 13.0-14.7, B 423-444)
30 VU p50	1,596 ms	54 ms	IMPROVES -96.6 %	4/4
30 VU p95	5,463 ms	174 ms	IMPROVES -96.8 %	4/4
30 VU HFS CPU per request	479 ms	15.1 ms	IMPROVES -96.8 %	4/4
30 VU peak resident memory	4,655 MB	2,021 MB	IMPROVES -56.6 %	4/4
1 VU requests/s	7.0	494.4	IMPROVES x70	4/4
1 VU p50 / p95	11.4 / 752 ms	1.0 / 8.8 ms	IMPROVES -91.7 % / -98.8 %	4/4

All 24 request shapes improve (p50 -94 % to -98 % at 30 VU), including Observation?component-code-value-quantity (p50 2,656 -> 155 ms, -94 %). A is slow because a token search without a system reads the parameter's whole index range and each candidate costs a table read; under 30 VU this saturates 6.5 cores (4.7 GB resident memory for 14 requests/s) and slows every shape.

CRUD metric (300 VU)	A	B	Change	Reading
Requests/s	566	605	+7.5 % +- 17 (pairs -12.4, +6.8, +6.4, +29.1)	neutral: inside the A-to-A spread (502-620)
p50	158.8 ms	146.6 ms	IMPROVES -7.8 % +- 4.5	4/4 pairs
p95	1,227 ms	1,265 ms	+3.5 % +- 20.5	noise
HFS CPU per request	1.12 ms	1.08 ms	IMPROVES -3.7 % +- 2.9	4/4 pairs, small
Bytes written per request	403,776	390,818	-3.1 % +- 4.3	not conclusive
Write syscalls per request	175.0	162.7	IMPROVES -7.0 % +- 2.3

Per-shape p95 in CRUD scatters widely (for example PUT Patient +110 %, POST Organization -26 %) with ranges across pairs of 100 to 400 %; it is not used for any claim.

Series 2: one A/B pair per suite on the current main, A = origin/main 064d977, B = this branch at 4ced50edc (after the rebase), same template and protocol, shorter windows (search 30 VU x 60 s and 1 VU x 30 s, CRUD 300 VU x 60 s). Both release binaries were built back to back in one target directory (462 s and 482 s). 0 HTTP errors, foreign CPU <= 0.13 cores.

Metric	A	B	Change
Search 30 VU requests/s	13.7	453.0	IMPROVES +3,207 % (x33)
Search 30 VU p50 / p95	1,048 / 6,611 ms	51.6 / 168.7 ms	IMPROVES -95.1 % / -97.4 %
Search 30 VU HFS CPU per request	491.6 ms	14.8 ms	IMPROVES -97.0 %
Search 1 VU requests/s	8.3	533.5	IMPROVES x64
Peak resident memory	4,815 MB	1,791 MB	IMPROVES -62.8 %
CRUD requests/s	509.7	646.6	+26.9 %, noise (A fell in the low part of its spread)
CRUD p50	132.6 ms	143.9 ms	EMPEORA +8.5 %, outside the series 1 pair range (-14.3 to -4.8 %, all improvements)
CRUD HFS CPU per request	1.01 ms	1.06 ms	EMPEORA +5.5 %, outside the series 1 pair range (-0.8 to -7.2 %, all improvements)
CRUD p95	1,341 ms	1,359 ms	+1.4 %, neutral
CRUD bytes written per request	379,040	380,861	+0.5 %, neutral

One pair cannot prove that CRUD is neutral; it shows that there is no large regression on the current main. The two CRUD numbers that moved against this change in series 2 (CPU +5.5 %, p50 +8.5 %) are small and are the only results with the opposite sign to series 1, so a real small write-side cost cannot be excluded with this data.

Measurement base: the SQLite backend directory is identical between 678fc50, 064d977 and the current main 3281155 (git diff empty; the only change since 064d977 is #2210, which touches Postgres files and the README); main changed outside it (SQL-on-FHIR handlers, memory trimming, Postgres and Elasticsearch work), and series 2 re-checked the search and CRUD path on that base.

Series 3: a repeat of the CRUD pair on a verified quiet machine, order reversed (B first), same binaries as series 2, CRUD 300 VU x 60 s. Gate: no compiler running and load1 <= 2.5 on three samples 30 s apart (0.60 at the last), idle baseline 0.04 busy cores and 18 GB free, load1 0.51 / 2.78 at the start of the two runs, foreign CPU 0.09-0.11 cores (about 1 % of the machine). The first pair (B1/A1) is the official one; B2/A2 is an extra pair produced when the harness discarded each run once by mistake.

CRUD metric	A (v41)	B (v46)	Change B vs A
HFS CPU per request	1.10 ms	1.05 ms	IMPROVES -4.7 %
p50 / p95	162.5 / 1,317 ms	150.0 / 952 ms	IMPROVES -7.7 % / -27.7 %
Bytes written per request	423,364	397,725	IMPROVES -6.1 %
Write syscalls per request	179.3	164.4	IMPROVES -8.3 %
Requests/s	632	892	+41 % (noisy)

The extra pair: CPU per request -2.5 %, bytes -4.5 %, write syscalls -7.1 %, p50 EMPEORA +4.9 %, requests/s -9 %.

All seven CRUD pairs (B vs A, negative = B better; simple mean of the pair values):

Series	HFS CPU per request	p50
1 (4 pairs)	-2.1, -0.8, -4.8, -7.2 %	-14.3, -7.0, -4.9, -4.8 %
2 (1 pair, A first)	+5.5 %	+8.5 %
3 (2 pairs, B first)	-4.7, -2.5 %	-7.7, +4.9 %
Mean of 7	-2.4 %	-3.6 %

CPU per request is better in 6 of 7 pairs, p50 in 5 of 7, and bytes and write syscalls per request are lower with B in every pair (-3 % to -8 %). The data does not say whether the two pairs that went the other way are noise or a small real cost, and seven pairs cannot rule one out; they do not show a regression of any size.

Known limitations

• Startup cost after upgrade. v42 rebuilds three indexes under the write lock before the server listens: the benchmark database needed 16-40 s to become ready on B against 1.1 s on A. The code comments estimate about 16 s per 6 GB for the covering indexes plus (only for databases that already ran an earlier build of this work) about 10 s per GB of token index, and temporary space about the size of the index being built. Not measured on a large production database.
• system| regression. A token search with a system and no code lost its seek (the index is now code-first) and reads the parameter's range.
• Rolling back to an older binary keeps results correct but is slower for system| searches, still maintains idx_search_composite_cov, and a bulk rebuild by the old binary recreates the old index shapes, which the new binary rebuilds on the next upgrade. An older binary opens a newer database without complaint (no version check exists today).
• Two empty migrations (v43, v44) stay in the ladder; collapsing everything into one v42 would be cleaner but changes code that was measured, so it is left for a reviewer's call.
• Window attempts ignore errors. The recency-window statements run with .ok(), so a SQLITE_BUSY or interrupt there falls back to the full page statement instead of failing; results stay correct.
• Statement cache. The multi-row INSERT prepares one statement per row count; with 64 cache slots, many distinct counts could evict hot statements. Not observed in the CRUD runs.
• Open PR perf(bulk-submit): parallel ingest and a parallel, scoped rebuild on every backend (1.9x-5.4x at 900k) #1941 also edits the SQLite schema; the second to land renumbers.
• 7 CRUD pairs and 4 search pairs plus 1 on the current main, one 30 % corpus slice, shared machine (series 3 on a verified quiet one), one empty-database CRUD scenario.

Test plan

What was run on this branch (14 commits on origin/main 3281155; the first 13 commits were carried over unchanged from a branch on 064d977, where series 2 and 3 were measured; the 14th only fixes two lint errors, in test-only code and an unused mut):

• cargo test -p helios-persistence --features sqlite --lib sqlite: 572 passed, 0 failed.
• cargo test -p helios-persistence --features sqlite --test sqlite_tests: 118 passed, 0 failed.
• rustfmt --edition 2024 --check on schema.rs, search_impl.rs and storage.rs: clean after the last commit (a formatting fix, its own commit).
• Equivalence, A against B: 22 URLs x 3 pages, 0 differences (series 1 and series 2).
• Earlier, before the rebase (base 678fc50, tip 109da4fed): cargo test -p helios-persistence --features sqlite, all suites: 2,416 passed, 0 failed, 31 ignored. Not repeated on the rebased branch.
• cargo clippy -p helios-persistence --all-targets --features sqlite -- -D warnings with the allow-list of the CI Linting job: clean after the last commit. The first push failed that job on two errors (choose_first_page_plan unused outside tests, an unused mut in a storage.rs test); the last commit fixes both. Not run with --all-features over the whole workspace as CI does.
• The full helios-persistence suite on the rebased branch, and the PostgreSQL, MongoDB and Elasticsearch suites: not run (this change touches only the SQLite backend).

🤖 Generated with Claude Code

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

[codecov]

Codecov Report

❌ Patch coverage is 98.79615% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tes/persistence/src/backends/sqlite/search_impl.rs	99.07%	6 Missing ⚠️
crates/persistence/src/backends/sqlite/storage.rs	96.85%	4 Missing ⚠️
crates/persistence/src/backends/sqlite/schema.rs	99.10%	3 Missing ⚠️
...stence/src/backends/sqlite/search/query_builder.rs	98.11%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!