Repository navigation
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
#24003 tracked this. #36233 fixed its thread.visited part and closed it, while the comment that closed #34909 said the broader write amplification is separate work. When #19105 closed, the reply asked for a new issue for its remaining part only: "High write amplification from one commit per event and page fan-out". This is that issue, reproduced on a fresh install and measured on today's main.
- Check out
bench/event-write-costfrom our fork (64c16f9246): today'smainplus a measurement script and two exports it needs. - Make a fresh T3 home and a small git project, and run one turn in it so the database has thread events.
- Run one live turn and count its WAL:
This starts the server from the checkout and runs "Run
node apps/server/scripts/bench-event-writes.ts live --home <T3 home> --project <git repo> --model claude-haiku-5-5 --out out
ls, read README.md, then write notes.txt with a one-line summary of it" throught3 thread start. Meanwhile it reads every WAL frame header as the server writes it, following WAL restarts. Finally it groups frames by commit marker, checks that it read every new event row, and maps each page to its b-tree withdbstat.
Expected behavior
A commit writes about the pages its new rows need: the event row with its indexes, and the projection rows whose values changed. The write volume stays within a small multiple of the event data.
Actual behavior
Three measurements of today's write path:
- Fresh database, one live turn (the steps above,
mainat5dd8cde083): the turn appended 56 events with about 70 KB ofpayload_json+metadata_json, and the server made 38 commits, 31 of them with events. Those commits wrote 961 WAL frames, 3.9 MB, which is 56× the event data: 17.2 pages per event, and 25 pages per event commit at p50 (47 at p90). - Real 11.8 GB database, replayed on
main: the script'sruncommand replays recorded events throughmain's ownEventSinkonto a page-for-page copy of a desktop install'sstatev2.sqlite, migrated tomain's schema. Two windows recorded on that install, 420 and 4,131 events, gave 34.0 and 33.4 WAL frames per event and 33 pages per commit at p50 (47 and 49 at p90). That is 74–77× the event data. Checkpoints then copy another 11.1–11.7 pages per event into the database file. Counts are identical over 3 runs. - The same install, live:
0.1.10-nightly.20261116.4774, built from16695c812f, which is 17 commits behind5dd8cde083with none of them on the write path. Over 10 minutes of normal use, reading WAL frame headers read-only: 135 events in 118 commits, 28 pages per commit at p50 (46 at p90), 26.8 frames per event, 85× the event data.
Where the frames go:
| B-trees | Fresh database, live turn | Real database, 4,131 replayed events |
|---|---|---|
orchestration_events + its 7 indexes |
33.2% | 40.5% |
| turn item projection + its indexes, positions and counts | 15.6% | 26.6% |
orchestration_v2_projection_threads + 3 indexes |
13.0% | 13.4% |
| node projection + 5 indexes | 8.1% | 7.4% |
sqlite_sequence, projection metadata |
6.5% | 5.7% |
| provider turns, runs, provider threads | 10.3% | 0.8% |
Fewer commits and fewer pages per commit are what lower this; smaller payloads barely do. These parts are known, each measured with the same script replaying recorded events through the server's own EventSink, where counts repeat exactly between runs:
- Open PRs:
- fix(server): Claude publishes each background roster change once #34952: Claude publishes each background roster change two or three times. On the install above, 20% of
provider-thread.updatedevents in two days were byte-identical repeats. - perf(server): streamed projection updates no longer rewrite unchanged index entries #36861 (taken over from perf(server): streamed projection updates no longer rewrite unchanged index entries #35026): streamed projection updates rewrite index entries whose values did not change.
- perf(server): store a subagent's record and its item in one commit #37624: each subagent progress line is
subagent.updatedplusturn_item.updatedin two commits, which is 14–18% of all events. The PR stores both in one transaction: −4.2% to −5.3% WAL frames per event in live turns, −4.3% to −4.6% on recorded windows.
- fix(server): Claude publishes each background roster change once #34952: Claude publishes each background roster change two or three times. On the install above, 20% of
- On our fork's branches, not yet PRs, because of the three-open-PR limit:
perf/event-redundant-index(was perf(server): events stop maintaining the stream sequence index #35023, closed to free a slot for a connection fix and to be reopened):idx_orch_events_stream_sequenceis maintained on every insert, and no read needs it. That is 3.6% of frames on a fresh database and about 6% on a 5.7 GB one.perf/group-commit: once a root turn reports its terminal, every snapshot of a background Claude subagent commits alone. A model from this install's trace timings put the gain at 14–24% of event commits, but a live run with two background subagents on a fresh database measured only about 3% fewer commits after the root turn ended and no change in frames per event, because there the snapshots rarely arrive while a commit is still running.perf/short-event-id:event_idis 230–290 bytes with a random tail, which spreads the autoindex. The branch usesevent:<uuidv7>: −3.4% to −3.5% WAL frames and −17% to −20% checkpoint pages per event.perf/checkpoint-interval: the branch raisesCHECKPOINT_FRAMESfrom 1,000 to 4,000, so hot pages are copied about every 130 commits instead of every 33. Checkpoint writes drop 19–32%. The cost is that an OS crash can lose up to ~16 MB of commits instead of ~4 MB undersynchronous = NORMAL.
Impact
Minor bug or occasional failure
Version or commit
main at 5dd8cde083; desktop 0.1.10-nightly.20261116.4774
Environment
Linux (NixOS), ext4 on NVMe, Node 26.13.0 (SQLite 3.53.3), Claude Code 2.1.322
Logs or stack traces
No response
Screenshots, recordings, or supporting files
No response
Workaround
None.
Model and harness: Claude Opus 5.5, Claude Code in T3 Code
Activity
Dropje97 commentedon Nov 16, 2026
A data point from a larger install, for scale. Our
statev2.sqliteis 49 GB, with about 20 agents working at once, on nightly 4787 plus local patches.Over one hour of normal use, the server process wrote 6.6 GB (
write_bytesin/proc/<pid>/io). In the same hour it appended 19,794 events with 50 MB ofpayload_json+metadata_json, across 46 threads. That is 131× the event data: 81 pages of 4 KiB per event, or about 160 GB a day.The count covers everything the server process writes: WAL frames, checkpoints into the database file, and its logs, which are at most 0.2 GB of it. Our build also adds about 25 local read indexes on the projection tables, so part of the excess is ours.
orchestration_eventsand its 7 indexes are stock. I did not map pages to b-trees, because that needs a scan of the live database.Measured and written by Claude (Opus 5.5) in T3 Code; posted with the owner's approval.
MicahLyle commentedon Nov 17, 2026
Note
Written by Claude Opus 5.5 (high effort) in Claude Code; reviewed and approved by me.
Another data point, from a different setup: stable 0.1.9 (AppImage, Electron 44.10.1) on Linux,
statev2.sqliteon btrfs (zstd, ssd, discard=async), about 7 Claude agent threads plus subagents, about 3 provider events/s. Read-only bpftrace of the server's WALpwrites over 60 s:ProviderCommitBatch.flushcalls hadwork_item_count = 1,batch_wait_ms0–1.sqlite_sequence,orchestration_v2_projection_metadata, theprojection_threads(project_id, updated_at)index leaf, page 1 and oneprojection_nodesleaf. 860 distinct pages were rewritten about 8 times each per minute.wal_checkpoint(PASSIVE)ran about every 8 s, about 3,000 pages a minute back into the main file. fsyncs were low (about 7 a minute), so the problem is bytes, not syncs.What btrfs adds: on a copy-on-write filesystem every small page rewrite lands in a new extent with its own metadata updates. The 300 MB database file was in about 41,000 extents, and device writes were 1.5–1.8× the logical writes. On the busiest machine the database alone came to about 30–45 GB/day at the device. So on btrfs (and probably ZFS/bcachefs) the fixes listed above pay off about 1.5× more. A one-line note in the docs,
chattr +Con the data directory before first start, would help btrfs users until then.One more vote for
perf/checkpoint-interval. At our rate, 1,000 frames is about 35 commits, so the same hot pages are copied into the main file about 7 times a minute.JarenKempton commentedon Nov 19, 2026
Note
🤖 GPT-6.1 Sol on behalf of JarenKempton
The same per-commit page fan-out is present on Linux with the SQLite database already stored in a NOCOW directory, so btrfs copy-on-write is not required to reproduce it.
Environment:
0.1.11-nightly.20261118.4903, Linux x86_64, btrfs over LUKS/NVMe; approximately 13.3 GB database with 4 KiB pages.A 180-second read-only WAL capture during normal agent activity measured:
sqlite_sequenceand projection metadata each had 294 frames targeting one page.This ratio counts WAL bytes only, not checkpoint writes or physical-device amplification, and the payload denominator may differ from other reports.
These observations support the existing diagnosis of per-event commit and index/projection fan-out. We also observed frequent WAL restarts; I’m reporting that separately on #38159 rather than conflating the two costs.