Repository navigation
Description
What happened
After the service auto-updated, T3 Code "would not come back up": systemd showed active (running), but the server was unusable for ~7 minutes and nothing said a database upgrade was running, so the user assumed it hung.
Diagnosis
#35421 (startup migrations 82/83 taking 17 min on a 32 GB statev2.sqlite) was closed by #35486. That PR stopped requests from hanging (immediate 503), committed each migration on its own and logged "N of M" progress. Its own description says it does not make readiness faster and that the 503 has an empty body. On a build with the fix, that gap is the whole user-visible problem:
- Readiness is still blocked by every pending migration. On a 15 GB database the server answered nothing useful for 6 min 48 s. Index builds over the turn_items / nodes projection tables still finish before the server serves anything.
- The 503 carries no information.
rejectRequestsUntilServing(apps/server/src/httpResponseErrorGuard.ts) answers every HTTP request and WebSocket upgrade with503,Retry-After: 1and an empty body. The progress lines exist only inboot-service.log. Users can't tell an upgrade from a hang, and restarting throws away the migration in progress. - Each release adds another expensive startup migration. On this build migration 86 (
86_QueuedTurnInputIndex, a partial index onjson_extract(payload_json, '$.inputIntent')) took as long as 82. [Bug]: 0.1.10 startup migrations 82/83 take 17 min on a 32 GB statev2.sqlite while the server listens but answers nothing #35421 only covered 82/83, and more full-table index builds keep shipping. - The root cause keeps growing.
statev2.sqlitegrows without bound ([Bug]: ClaudeReadimages stay inline as base64 in both event and projection rows (1.4 GB, 46% of statev2.sqlite, in one week) #26591, [Bug]: Deleting a thread says it permanently clears its history, but V2 keeps every event and projection row #17845). A long-lived install hits this on every update that adds an index.
Not a duplicate of #35421: the hang is fixed. What's left is the server staying unusable for the whole migration with no user-visible progress, which #35486 left out of scope.
Steps to reproduce
- Standalone systemd service with a ~15 GB
statev2.sqlite, on0.1.8-nightly.20261108.4353. - Update to
0.1.10-nightly.20261113.4616(migrations 81–87 pending); service restarts. - Open the app or
curl -i http://127.0.0.1:3773/.well-known/t3/environment. - Empty-body
503for ~7 min; progress only inboot-service.log.
Version
Bug seen on 0.1.10-nightly.20261113.4616 (updated from 0.1.8-nightly.20261108.4353). Triage CLI: 0.1.10-nightly.20261113.4616.
Environment
Linux x64 on WSL2 (kernel 6.18.40.1-microsoft-standard-WSL2), Node v26.12.1. Standalone server as a systemd user service bound to 0.0.0.0:3773. statev2.sqlite 15.1 GB.
Evidence
`boot-service.log` (the per-migration progress added by #35486):
[05:53:28.988] Running database migrations
[05:53:28.995] 81_ProjectionMessageSearchText 1 of 7
[05:53:45.644] 82_TurnItemConversationIndex 2 of 7
[05:56:16.942] 83_CommandReadIndexes 3 of 7
[05:57:34.330] 84_LastExecutedRunIndex 4 of 7
[05:57:35.645] 85_TurnItemCounts 5 of 7
[05:57:41.971] 86_QueuedTurnInputIndex 6 of 7
[06:00:11.420] 87_ProjectionProjectsCustomTitle 7 of 7
[06:00:11.443] Migrations ran successfully
[06:00:12.811] Listening on http://0.0.0.0:3773
`GET /.well-known/t3/environment` returned `503` at 05:56 and 05:58 and `200` at 06:00:18 (`databaseMigration: 87`). Service start was 05:53:24.Related issues
- [Bug]: 0.1.10 startup migrations 82/83 take 17 min on a 32 GB statev2.sqlite while the server listens but answers nothing #35421 / fix(server): long startup migrations log progress and resume after a restart #35486: same symptom; fix removed the hang, not the blocking or missing progress.
- Desktop shows a plain loading screen during long startup migrations, and closing it rolls them back #35313: desktop loading screen during migrations; web/standalone has no equivalent.
- Desktop stays windowless for 8+ minutes after an update while startup recovery holds a write transaction open #23372: desktop windowless 8+ min during startup recovery.
- [Bug]: Claude
Readimages stay inline as base64 in both event and projection rows (1.4 GB, 46% of statev2.sqlite, in one week) #26591, [Bug]: Deleting a thread says it permanently clears its history, but V2 keeps every event and projection row #17845: unboundedstatev2.sqlitegrowth.
Suggestions: build indexes after ready or in batches (note ProjectionStore.ts pins reads with INDEXED BY on the migration-82 index, so deferring needs fallback queries); put "upgrading database, N of M" in the startup 503 body so clients can show it.
Fix applied or workaround
None. Waited it out; nothing changed on the install or database.
Filed by
Claude Code (Claude Opus 5.5), via t3 triage.
Activity
fdubayteq commentedon Nov 13, 2026
Same problem on the desktop app (macOS, bundled local server), on a build newer than the one in this report. The desktop has its own version of the visibility gap: the window sits on the white splash with the T3 logo and no text or progress the whole time. The user took it for a hang and quit once mid-migration.
Environment
9a35ab0a), updated from 0.1.9-nightly.20261110.4478statev2.sqlite: 24.4 GB (page_size 4096, 5,952,314 pages, freelist_count 2,374,046, so ~9.7 GB / 40% free and ~14.7 GB live)Migration timeline from
server.trace.ndjson(UTC, 2026-11-13)From the first migration to ready took about 12 min 20 s. Migration 88 (new since this issue was filed) is another full-table index build on
orchestration_v2_projection_turn_itemsand costs as much as 82 and 86. The per-migration commits from #35486 worked: the quit during 82 lost only 82's in-progress work, and 81 was kept.What the user saw
Ustate, ~10% CPU). Noserver-child*.logwas written, because it never crashed.Desktop-side gap (in addition to the 503 body point above): #35056 made the desktop keep waiting while migration progress lines appear in the backend output tail. In this build that information stays in the main process: the splash renderer gets nothing to display. Passing the "Upgrading database (N of M)" line the desktop already parses through to the splash would have prevented the quit. A warning when closing during a migration (suggested in #35313) would have too.
Diagnosed via
t3 triagewith Claude Code (Claude Opus 5.5).AlexChambers commentedon Nov 14, 2026
Same on the macOS desktop app, 0.1.10-nightly.20261113.4656 (updated from 20261111.4544). The window sat on the logo splash for about 5 minutes with no progress, and I assumed it was broken.
statev2.sqliteis 53 GB, but 27 GB of that is free pages (freelist_count6.68M of 13.9M pages,auto_vacuum= 0). Timings fromserver.trace.ndjson:runMigrations: 148 s in total(The per-migration spans overlap, so they add up to more than the total.) The first
ProjectionStore.readShellSnapshotRowsafter startup then took 17 to 26 s.Live data is about 30 GB:
turn_items17 GB (command output 7.4 GB, dynamic tool output 4.2 GB, spread across about 10k threads) andnodes5 GB. Settings > Storage offers "Reclaim freed database space", but nothing prompts you to use it before an update rebuilds indexes over the whole file.The same 8 migrations took about 4 s on a Linux server with a smaller database.
Snowy7 commentedon Nov 14, 2026
Same on the Windows desktop app, on a newer build than the ones above: 0.1.10-nightly.20261114.4682, auto-updated from
0.1.9-nightly.20261111.4501. The window sat on a black splash with only the T3 logo, no text or progress, for about 4 minutes. The user thought it was stuck, relaunched once mid-upgrade, and started at3 triagesession before it finished.Environment: Windows 11 Pro (10.0.26300) x64, desktop app with its bundled local server on
:3773.statev2.sqliteis 20.7 GB: page_size 4096, 5,052,748 pages, freelist_count 2,064,566, so about 8.5 GB (41%) is free pages.Migration timeline from
server.trace.ndjson(UTC, 2026-11-14):Throughout, the desktop main process polled
GET http://127.0.0.1:3773/.well-known/t3/environmentabout every 100 ms and got an empty503 (StatusCode error)each time. Nothing from that reached the renderer, so the splash showed no state. So as of build 4682, the desktop still shows nothing like "upgrading database, N of M" on any platform, and each new nightly keeps adding migrations (89–91 here).Nothing was changed on the install. Waiting it out worked.
Filed by Claude Code (Claude Opus 5.5) via
t3 triage.EtanHey commentedon Nov 15, 2026
Same on the macOS desktop app, on today's build, with one extra symptom: quitting mid-migration led to "This T3 home is unavailable" on relaunch.
Environment: T3 Code (Nightly) desktop 0.1.10-nightly.20261115.4723 (
11c425b4), auto-updated from 20261114. macOS (Darwin 27.0.0), arm64.statev2.sqliteis 19.9 GB.Timeline (UTC, from
desktop.trace.ndjsonandserver.trace.ndjson):92_QueuedTurnInputIndexJsonValidtook 67.0 s,93_ProviderTurnAttemptIndex1.9 s. The splash showed no progress, so the user quit at 06:07:46.94_TurnItemMessageOrdinalIndex(CREATE INDEX … json_extract(payload_json,'$.messageId')) started. The user force-quit around 06:08:11.bootstrap found another server owning the T3 home.awaitHomeOwnershipReleasegave up after its 10 s grace, and the app showed the "T3 home is unavailable" dialog, which points at a background service. None is installed.Migrator 94endedInterruptedafter 62.6 s, which rolled the index back and released the lock.The new part:
launcherWatch.awaitLauncherLosspollsppidevery 2 s on the JS thread. The synchronousCREATE INDEXblocks that thread, so the orphan can't notice the desktop is gone until the statement returns. It keeps the ownership lock for the whole build: about 47 s here, longer than the desktop's 10 s grace (BACKEND_QUIT_STOP_TIMEOUT+ 3 s). Then the interrupt throws the finished index away. This is a gap in the #13747 fix while a migration is running. On large databases, a misleading home-ownership dialog then follows a mid-upgrade quit.Reported via T3 triage by Claude Code (Opus 5.5).
WhiteWarrior625 commentedon Nov 16, 2026
Same on the Linux desktop app (AppImage, KDE Wayland), nightly 0.1.10-nightly.20261116.4774, auto-updated from the previous nightly. statev2.sqlite is 41 GB.
Timeline (local time, from desktop.trace.ndjson, server.trace.ndjson and effect_sql_migrations):
So the quit cost ~6 minutes, and migration 95 is a new long step on large databases beyond the index builds named above. Nightly .4787 has no commits touching migrations or the splash.
Found via
t3 triagewith Claude Code (Claude Opus 5.5).kkauf commentedon Nov 17, 2026
Another data point (macOS, desktop 0.1.9 → 0.1.10,
statev2.sqlite9 GB, 517k turn items).The window showed nothing while migrations ran, so I assumed the app was hung and quit it. That interrupted
83_CommandReadIndexesafter 19 s (InterruptErrorinserver.trace.ndjson), and the next launch had to start again. Index builds took 37–45 s each (82, 86, 88).The server already emits "Upgrading database N of 15" events. Showing them in the window, with a "don't quit" note and a quit warning while a migration runs, would have prevented the restart.
Posted via Claude Code.
murmaider227 commentedon Nov 18, 2026
On Windows, T3 Code Desktop Nightly
0.1.11-nightly.20261118.4864showed the loading screen for about 7 min 13 s. Migration 95 took 5 min 37 s, and committing its transaction took another 1 min 20 s. The local endpoint returned HTTP 503 during startup, then HTTP 200. The app opened on its own.KaramanisDev commentedon Nov 18, 2026
Same symptom on a Windows desktop install, reporting from
t3 triageon 0.1.10. Same mechanism as #35995 and #35421, on a larger database.Environment: Windows 11 Pro 10.0.26300, Node v26.13.1, desktop app with local backend on 127.0.0.1:3773. Upgraded from 0.1.9. Exact desktop build string not recorded.
Evidence:
statev2.sqliteis 35.9 GB (page_count 8764197, freelist_count 3066454, so about 12.6 GB of free pages), with a 6.7 GB WAL that has not been checkpointed.orchestration_eventsmax rowid is 5,005,116.effect_sql_migrations: migrations 82–94 applied in one batch, 08:19:55 to 08:23:09 UTC.server.trace.ndjson:Migrator 83_CommandReadIndexes77.98 s,85_TurnItemCounts20.13 s,88_TurnItemTypeStatusIndex7.23 s,94_TurnItemMessageOrdinalIndex6.64 s.desktop.trace.ndjson:shared.httpReadiness.waitForHttpReadytimed out after 180000 ms, 1644 attempts against/.well-known/t3/environment(503 and transport errors). The desktop then reportedDesktopLocalEnvironmentAuthBackendNotConfiguredError: Local backend is not configured.apps/server/src/httpResponseErrorGuard.ts:54-60returns an empty 503 until serving is attached, which matches the report.Not explained by the traces: about 3 minutes before migration 82 started (no span over 1 s), and readiness probes still failing for about 3 minutes after migration 94 finished at 08:23:09 (no slow span). Someone with the full logs may want to check those windows.
Filed via
t3 triage.lol2x commentedon Nov 20, 2026
Same on the Windows desktop app, 0.1.11-nightly.20261119.4954 (auto-updated from 0.1.11-nightly.20261117.4851). The window sat on the splash with no progress. The user quit after ~134 s, which rolled back the migration, and assumed the update had broken the app again.
Environment: Windows 11 Pro 10.0.26300 x64, desktop app with bundled local server on 127.0.0.1:3773.
statev2.sqliteis 6.1 GB, with 308k turn items and 1.6 GB ofpayload_json.Evidence (from
server.trace.ndjson):BEGINand inserted the row for99_BackgroundRecoveryIndexes. No later span until the user quit at 01:10:06, andeffect_sql_migrationsstayed at 98.orchestration_v2_projection_turn_itemsthat callsjson_valid/json_extracton every row. A read-only full scan ofpayload_jsontook 109.5 s cold and 2.4 s warm, so this build is dominated by cold disk reads.databaseMigration: 100.Adds a smaller-database data point: even at 6 GB, one cold index build is longer than users will wait at a splash screen with no progress, and each new nightly with an index migration triggers it again.
Reported via
t3 triageby Claude Code (Claude Opus 5.5).nachlielp commentedon Nov 21, 2026
Same thing on the macOS desktop app, 0.1.9 → 0.1.11 stable auto-update, with a 13.2 GiB
statev2.sqlite(Apple M3 Pro, 18 GB RAM). Migrations 82–102 were pending.On the launch that finally completed,
runMigrationstook 382.8 s and the window had no backend for 6 min 24 s. The desktop log recorded twoBackendReadinessTimeoutError: Timed out after 180000msentries in that time, and about 1,400 polls of/.well-known/t3/environmentreturned 503.Migrations that took more than a few seconds:
Nothing on screen said an upgrade was running, so the app was quit and relaunched three times in the first two minutes. Each quit interrupted the migration in progress: 83 was rolled back twice (21.3 s and 20.9 s of work lost) before it succeeded on the third try.
There was also a gap of about 109 s between the end of 95 and the start of 96 with no spans. A read-only
dbstatquery was running against the database during part of that window, so that gap may be partly self-inflicted.Most of the table these indexes cover is inline base64 from Claude
Readimages (numbers in #26591).