Repository navigation
perf(server): storage cleanup no longer freezes the server, and freezes are logged from 2 seconds - #1036
Merged
Conversation
The cleanup job deleted old rows in 5,000-row transactions that ran back to back on the main thread, so the server could not answer for the whole prune (3 to 20 seconds on a 9.5 GB database). It now deletes 25 rows per transaction and yields a full event-loop turn after each one, batches the diff-blob delete too, and drops the final TRUNCATE checkpoint, which small batches make unnecessary.
…rams The stall monitor now logs freezes from 2 seconds (clients still hear of them from 5). When a freeze's cause is overloaded or unknown, the log line also lists the five busiest programs by CPU, read with ps after the freeze: program names only, never arguments or paths.
Sets journal_size_limit to 32 MiB so SQLite trims the WAL once a checkpoint resets it. Without it, a read held open across many commits (such as during a storage prune) grew the WAL to hundreds of megabytes and it stayed that size. The cleanup test now counts one loop turn per delete transaction, the partial last one included.
Catches a spawn that throws, gives up after a hard timeout, kills the child with SIGKILL, and cuts relative paths down to the program name.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The storage cleanup job (
StorageMaintenance) runs 2 minutes after boot and every 6 hours, on the main thread with the synchronousnode:sqlitedriver. It deleted 5,000 rows per transaction, the batches ran back to back because Effect only yields every couple of thousand operations, and awal_checkpoint(TRUNCATE)followed. So the server couldn't answer for most of the run. On a real 9.5 GB install the freeze monitor logged an 11.3s freeze in the same second as an 11.2s run, and one 29.6s run as two freezes of 14.1s and 15.8s. Of 75 logged runs, 26 took 5s or more.Cleanup job
Effect.yieldNowafter every batch, which is a full event-loop turn on the default scheduler.checkpoint_diff_blobsdelete had no row limit. It is batched now.Measured on a copy of that database:
WAL size limit (every database).
PRAGMA journal_size_limit = 32 MBinSqlite.ts. Without it, a reader holding a snapshot during many commits left the WAL file at its high-water size. It reached 252 MB in a test and stayed there. With the limit, SQLite trims the file once a checkpoint resets the WAL.Freeze records (
StallMonitor)overloadedandunknownfreezes the log line also names the five busiest programs. It reads them withpsin its own fiber, with a timeout and a kill, so a failingpscan't drop or delay the line. Program names and CPU share only: no arguments, and paths are cut to the last part. Skipped on Windows.Tests. A new test fails on the old code (zero loop turns between batches) and passes now. Parser and failure cases are covered for the process reader. The persistence and observability sets pass, along with fmt, lint and typecheck.
Review. GPT-6.2-Astra reviewed the diff. Fixed: the WAL high-water size, a
psfailure suppressing the warning, relative paths in a Linuxcomm, and a test that could pass with a turn only every other batch. Left as is: a 4,999.6ms freeze rounds to 5s and reaches clients.Not solved here. With a cold cache a single batch can still hold the loop 100 to 300ms, and the job's first scan can take up to 2s. Those need SQLite off the main thread, which is planned separately.