Skip to content

perf(server): storage cleanup no longer freezes the server, and freezes are logged from 2 seconds - #1036

Merged
badcuban merged 4 commits into
mainfrom
perf/storage-maintenance-yield
Nov 21, 2026
Merged

badcuban merged 4 commits into
mainfrom
perf/storage-maintenance-yield

Conversation

@badcuban

Copy link
Copy Markdown
Collaborator

The storage cleanup job (StorageMaintenance) runs 2 minutes after boot and every 6 hours, on the main thread with the synchronous node:sqlite driver. It deleted 5,000 rows per transaction, the batches ran back to back because Effect only yields every couple of thousand operations, and a wal_checkpoint(TRUNCATE) followed. So the server couldn't answer for most of the run. On a real 9.5 GB install the freeze monitor logged an 11.3s freeze in the same second as an 11.2s run, and one 29.6s run as two freezes of 14.1s and 15.8s. Of 75 logged runs, 26 took 5s or more.

Cleanup job

  • 25 rows per transaction (20 for diff blobs), with Effect.yieldNow after every batch, which is a full event-loop turn on the default scheduler.
  • The per-thread checkpoint_diff_blobs delete had no row limit. It is batched now.
  • The explicit TRUNCATE checkpoint is gone. Small batches keep the WAL near the 1,000-page autocheckpoint.
  • What gets deleted is unchanged.

Measured on a copy of that database:

Before After
Normal run (14,700 rows), longest loop hold 3.1s 53ms
Heavy run (123,600 rows), longest loop hold 19.7s 0.55s
Heavy run, holds over 100ms 40 2
Heavy run, total time 158s 45s

WAL size limit (every database). PRAGMA journal_size_limit = 32 MB in Sqlite.ts. Without it, a reader holding a snapshot during many commits left the WAL file at its high-water size. It reached 252 MB in a test and stayed there. With the limit, SQLite trims the file once a checkpoint resets the WAL.

Freeze records (StallMonitor)

  • Freezes of 2s or more go to the log. Clients are still told only about freezes of 5s or more.
  • For overloaded and unknown freezes the log line also names the five busiest programs. It reads them with ps in its own fiber, with a timeout and a kill, so a failing ps can't drop or delay the line. Program names and CPU share only: no arguments, and paths are cut to the last part. Skipped on Windows.

Tests. A new test fails on the old code (zero loop turns between batches) and passes now. Parser and failure cases are covered for the process reader. The persistence and observability sets pass, along with fmt, lint and typecheck.

Review. GPT-6.2-Astra reviewed the diff. Fixed: the WAL high-water size, a ps failure suppressing the warning, relative paths in a Linux comm, and a test that could pass with a turn only every other batch. Left as is: a 4,999.6ms freeze rounds to 5s and reaches clients.

Not solved here. With a cold cache a single batch can still hold the loop 100 to 300ms, and the job's first scan can take up to 2s. Those need SQLite off the main thread, which is planned separately.

The cleanup job deleted old rows in 5,000-row transactions that ran back
to back on the main thread, so the server could not answer for the whole
prune (3 to 20 seconds on a 9.5 GB database). It now deletes 25 rows per
transaction and yields a full event-loop turn after each one, batches the
diff-blob delete too, and drops the final TRUNCATE checkpoint, which small
batches make unnecessary.
…rams

The stall monitor now logs freezes from 2 seconds (clients still hear of
them from 5). When a freeze's cause is overloaded or unknown, the log line
also lists the five busiest programs by CPU, read with ps after the freeze:
program names only, never arguments or paths.
Sets journal_size_limit to 32 MiB so SQLite trims the WAL once a
checkpoint resets it. Without it, a read held open across many commits
(such as during a storage prune) grew the WAL to hundreds of megabytes
and it stayed that size. The cleanup test now counts one loop turn per
delete transaction, the partial last one included.
Catches a spawn that throws, gives up after a hard timeout, kills the
child with SIGKILL, and cuts relative paths down to the program name.
@vercel

vercel Bot commented Nov 21, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
threadlines Skipped Skipped Nov 21, 2026 5:19am UTC

Request Review

@badcuban
badcuban added this pull request to the merge queue Nov 21, 2026
Merged via the queue into main with commit 5ed11b5 Nov 21, 2026
23 checks passed
@badcuban
badcuban deleted the perf/storage-maintenance-yield branch November 21, 2026 05:30

This branch was previously deployed

1 inactive deployment
Preview — dc530bfc Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant