Skip to content

[Bug]: Each event commit rewrites 25–33 SQLite pages, so the server writes 56–85× its event data #37600

Open

Description

@Vantrongs

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

#24003 tracked this. #36233 fixed its thread.visited part and closed it, while the comment that closed #34909 said the broader write amplification is separate work. When #19105 closed, the reply asked for a new issue for its remaining part only: "High write amplification from one commit per event and page fan-out". This is that issue, reproduced on a fresh install and measured on today's main.

  1. Check out bench/event-write-cost from our fork (64c16f9246): today's main plus a measurement script and two exports it needs.
  2. Make a fresh T3 home and a small git project, and run one turn in it so the database has thread events.
  3. Run one live turn and count its WAL:
    node apps/server/scripts/bench-event-writes.ts live --home <T3 home> --project <git repo> --model claude-haiku-5-5 --out out
    This starts the server from the checkout and runs "Run ls, read README.md, then write notes.txt with a one-line summary of it" through t3 thread start. Meanwhile it reads every WAL frame header as the server writes it, following WAL restarts. Finally it groups frames by commit marker, checks that it read every new event row, and maps each page to its b-tree with dbstat.

Expected behavior

A commit writes about the pages its new rows need: the event row with its indexes, and the projection rows whose values changed. The write volume stays within a small multiple of the event data.

Actual behavior

Three measurements of today's write path:

  1. Fresh database, one live turn (the steps above, main at 5dd8cde083): the turn appended 56 events with about 70 KB of payload_json + metadata_json, and the server made 38 commits, 31 of them with events. Those commits wrote 961 WAL frames, 3.9 MB, which is 56× the event data: 17.2 pages per event, and 25 pages per event commit at p50 (47 at p90).
  2. Real 11.8 GB database, replayed on main: the script's run command replays recorded events through main's own EventSink onto a page-for-page copy of a desktop install's statev2.sqlite, migrated to main's schema. Two windows recorded on that install, 420 and 4,131 events, gave 34.0 and 33.4 WAL frames per event and 33 pages per commit at p50 (47 and 49 at p90). That is 74–77× the event data. Checkpoints then copy another 11.1–11.7 pages per event into the database file. Counts are identical over 3 runs.
  3. The same install, live: 0.1.10-nightly.20261116.4774, built from 16695c812f, which is 17 commits behind 5dd8cde083 with none of them on the write path. Over 10 minutes of normal use, reading WAL frame headers read-only: 135 events in 118 commits, 28 pages per commit at p50 (46 at p90), 26.8 frames per event, 85× the event data.

Where the frames go:

B-trees Fresh database, live turn Real database, 4,131 replayed events
orchestration_events + its 7 indexes 33.2% 40.5%
turn item projection + its indexes, positions and counts 15.6% 26.6%
orchestration_v2_projection_threads + 3 indexes 13.0% 13.4%
node projection + 5 indexes 8.1% 7.4%
sqlite_sequence, projection metadata 6.5% 5.7%
provider turns, runs, provider threads 10.3% 0.8%

Fewer commits and fewer pages per commit are what lower this; smaller payloads barely do. These parts are known, each measured with the same script replaying recorded events through the server's own EventSink, where counts repeat exactly between runs:

Impact

Minor bug or occasional failure

Version or commit

main at 5dd8cde083; desktop 0.1.10-nightly.20261116.4774

Environment

Linux (NixOS), ext4 on NVMe, Node 26.13.0 (SQLite 3.53.3), Claude Code 2.1.322

Logs or stack traces

No response

Screenshots, recordings, or supporting files

No response

Workaround

None.

Model and harness: Claude Opus 5.5, Claude Code in T3 Code

Activity

  1. added
    bugSomething is broken or behaving incorrectly.
    needs-triageIssue needs maintainer review and initial categorization.
    on Nov 16, 2026
  2. Dropje97 commented on Nov 16, 2026

    @Dropje97

    A data point from a larger install, for scale. Our statev2.sqlite is 49 GB, with about 20 agents working at once, on nightly 4787 plus local patches.

    Over one hour of normal use, the server process wrote 6.6 GB (write_bytes in /proc/<pid>/io). In the same hour it appended 19,794 events with 50 MB of payload_json + metadata_json, across 46 threads. That is 131× the event data: 81 pages of 4 KiB per event, or about 160 GB a day.

    The count covers everything the server process writes: WAL frames, checkpoints into the database file, and its logs, which are at most 0.2 GB of it. Our build also adds about 25 local read indexes on the projection tables, so part of the excess is ours. orchestration_events and its 7 indexes are stock. I did not map pages to b-trees, because that needs a scan of the live database.

    Measured and written by Claude (Opus 5.5) in T3 Code; posted with the owner's approval.

  3. MicahLyle commented on Nov 17, 2026

    @MicahLyle

    Note

    Written by Claude Opus 5.5 (high effort) in Claude Code; reviewed and approved by me.

    Another data point, from a different setup: stable 0.1.9 (AppImage, Electron 44.10.1) on Linux, statev2.sqlite on btrfs (zstd, ssd, discard=async), about 7 Claude agent threads plus subagents, about 3 provider events/s. Read-only bpftrace of the server's WAL pwrites over 60 s:

    • 242 WAL commits for 260 events carrying 0.43 MB of JSON: 99 % of ProviderCommitBatch.flush calls had work_item_count = 1, batch_wait_ms 0–1.
    • 28.5 WAL frames per commit (median 27, p90 40), about 115 KB of WAL per event, about 66× the payload. This matches your numbers on ext4.
    • The same pages are in about 97 % of commits: sqlite_sequence, orchestration_v2_projection_metadata, the projection_threads(project_id, updated_at) index leaf, page 1 and one projection_nodes leaf. 860 distinct pages were rewritten about 8 times each per minute.
    • wal_checkpoint(PASSIVE) ran about every 8 s, about 3,000 pages a minute back into the main file. fsyncs were low (about 7 a minute), so the problem is bytes, not syncs.
    • Subagent progress rewrote each subagent turn item about 60 times.

    What btrfs adds: on a copy-on-write filesystem every small page rewrite lands in a new extent with its own metadata updates. The 300 MB database file was in about 41,000 extents, and device writes were 1.5–1.8× the logical writes. On the busiest machine the database alone came to about 30–45 GB/day at the device. So on btrfs (and probably ZFS/bcachefs) the fixes listed above pay off about 1.5× more. A one-line note in the docs, chattr +C on the data directory before first start, would help btrfs users until then.

    One more vote for perf/checkpoint-interval. At our rate, 1,000 frames is about 35 commits, so the same hot pages are copied into the main file about 7 times a minute.

  4. JarenKempton commented on Nov 19, 2026

    @JarenKempton

    Note

    🤖 GPT-6.1 Sol on behalf of JarenKempton

    The same per-commit page fan-out is present on Linux with the SQLite database already stored in a NOCOW directory, so btrfs copy-on-write is not required to reproduce it.

    Environment: 0.1.11-nightly.20261118.4903, Linux x86_64, btrfs over LUKS/NVMe; approximately 13.3 GB database with 4 KiB pages.

    A 180-second read-only WAL capture during normal agent activity measured:

    • Approximately 380 events/minute and 0.93 MB/minute of event payload.
    • 10,969 WAL frames over 180 seconds: approximately 15 MB/minute of WAL, or roughly 15× the payload.
    • Approximately 32 pages per commit.
    • Repeated writes to hot pages: in a 90-second window, the application-sequence index had 329 frames across just four distinct pages; sqlite_sequence and projection metadata each had 294 frames targeting one page.

    This ratio counts WAL bytes only, not checkpoint writes or physical-device amplification, and the payload denominator may differ from other reports.

    These observations support the existing diagnosis of per-event commit and index/projection fan-out. We also observed frequent WAL restarts; I’m reporting that separately on #38159 rather than conflating the two costs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaving incorrectly.needs-triageIssue needs maintainer review and initial categorization.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions