Skip to content

Startup still blocks for ~7 minutes on 0.1.10 migrations after the #35421 fix, and clients only get an empty 503 #35995

Open

Description

@lwalejko

What happened

After the service auto-updated, T3 Code "would not come back up": systemd showed active (running), but the server was unusable for ~7 minutes and nothing said a database upgrade was running, so the user assumed it hung.

Diagnosis

#35421 (startup migrations 82/83 taking 17 min on a 32 GB statev2.sqlite) was closed by #35486. That PR stopped requests from hanging (immediate 503), committed each migration on its own and logged "N of M" progress. Its own description says it does not make readiness faster and that the 503 has an empty body. On a build with the fix, that gap is the whole user-visible problem:

  1. Readiness is still blocked by every pending migration. On a 15 GB database the server answered nothing useful for 6 min 48 s. Index builds over the turn_items / nodes projection tables still finish before the server serves anything.
  2. The 503 carries no information. rejectRequestsUntilServing (apps/server/src/httpResponseErrorGuard.ts) answers every HTTP request and WebSocket upgrade with 503, Retry-After: 1 and an empty body. The progress lines exist only in boot-service.log. Users can't tell an upgrade from a hang, and restarting throws away the migration in progress.
  3. Each release adds another expensive startup migration. On this build migration 86 (86_QueuedTurnInputIndex, a partial index on json_extract(payload_json, '$.inputIntent')) took as long as 82. [Bug]: 0.1.10 startup migrations 82/83 take 17 min on a 32 GB statev2.sqlite while the server listens but answers nothing #35421 only covered 82/83, and more full-table index builds keep shipping.
  4. The root cause keeps growing. statev2.sqlite grows without bound ([Bug]: Claude Read images stay inline as base64 in both event and projection rows (1.4 GB, 46% of statev2.sqlite, in one week) #26591, [Bug]: Deleting a thread says it permanently clears its history, but V2 keeps every event and projection row #17845). A long-lived install hits this on every update that adds an index.

Not a duplicate of #35421: the hang is fixed. What's left is the server staying unusable for the whole migration with no user-visible progress, which #35486 left out of scope.

Steps to reproduce

  1. Standalone systemd service with a ~15 GB statev2.sqlite, on 0.1.8-nightly.20261108.4353.
  2. Update to 0.1.10-nightly.20261113.4616 (migrations 81–87 pending); service restarts.
  3. Open the app or curl -i http://127.0.0.1:3773/.well-known/t3/environment.
  4. Empty-body 503 for ~7 min; progress only in boot-service.log.

Version

Bug seen on 0.1.10-nightly.20261113.4616 (updated from 0.1.8-nightly.20261108.4353). Triage CLI: 0.1.10-nightly.20261113.4616.

Environment

Linux x64 on WSL2 (kernel 6.18.40.1-microsoft-standard-WSL2), Node v26.12.1. Standalone server as a systemd user service bound to 0.0.0.0:3773. statev2.sqlite 15.1 GB.

Evidence

`boot-service.log` (the per-migration progress added by #35486):


[05:53:28.988] Running database migrations
[05:53:28.995] 81_ProjectionMessageSearchText   1 of 7
[05:53:45.644] 82_TurnItemConversationIndex     2 of 7
[05:56:16.942] 83_CommandReadIndexes            3 of 7
[05:57:34.330] 84_LastExecutedRunIndex          4 of 7
[05:57:35.645] 85_TurnItemCounts                5 of 7
[05:57:41.971] 86_QueuedTurnInputIndex          6 of 7
[06:00:11.420] 87_ProjectionProjectsCustomTitle 7 of 7
[06:00:11.443] Migrations ran successfully
[06:00:12.811] Listening on http://0.0.0.0:3773



`GET /.well-known/t3/environment` returned `503` at 05:56 and 05:58 and `200` at 06:00:18 (`databaseMigration: 87`). Service start was 05:53:24.

Related issues

Suggestions: build indexes after ready or in batches (note ProjectionStore.ts pins reads with INDEXED BY on the migration-82 index, so deferring needs fallback queries); put "upgrading database, N of M" in the startup 503 body so clients can show it.

Fix applied or workaround

None. Waited it out; nothing changed on the install or database.

Filed by

Claude Code (Claude Opus 5.5), via t3 triage.

Activity

  1. fdubayteq commented on Nov 13, 2026

    @fdubayteq

    Same problem on the desktop app (macOS, bundled local server), on a build newer than the one in this report. The desktop has its own version of the visibility gap: the window sits on the white splash with the T3 logo and no text or progress the whole time. The user took it for a hang and quit once mid-migration.

    Environment

    • T3 Code (Nightly) desktop 0.1.10-nightly.20261113.4643 (tag commit 9a35ab0a), updated from 0.1.9-nightly.20261110.4478
    • macOS 27.2 (Darwin 27.2.0), arm64, Mac mini (Macmini9,1)
    • statev2.sqlite: 24.4 GB (page_size 4096, 5,952,314 pages, freelist_count 2,374,046, so ~9.7 GB / 40% free and ~14.7 GB live)

    Migration timeline from server.trace.ndjson (UTC, 2026-11-13)

    Migration Start Duration
    81_ProjectionMessageSearchText 16:12:25 13.0 s
    (app quit at 16:14:10 during 82; relaunched 16:14:20)
    82_TurnItemConversationIndex 16:14:22 152.5 s
    83_CommandReadIndexes 16:16:54 132.6 s
    84_LastExecutedRunIndex 16:19:11 2.1 s
    85_TurnItemCounts 16:19:13 13.1 s
    86_QueuedTurnInputIndex 16:19:26 151.1 s
    87_ProjectionProjectsCustomTitle 16:21:57 0.0 s
    88_TurnItemTypeStatusIndex 16:21:57 151.9 s
    89_ShellThreadReadIndexes 16:24:34 3.0 s
    server.startup 16:24:38 6.2 s, ready 16:24:44

    From the first migration to ready took about 12 min 20 s. Migration 88 (new since this issue was filed) is another full-table index build on orchestration_v2_projection_turn_items and costs as much as 82 and 86. The per-migration commits from #35486 worked: the quit during 82 lost only 82's in-progress work, and 81 was kept.

    What the user saw

    1. Update the desktop nightly, launch.
    2. White splash with the logo for 10+ minutes. No error, no "upgrading database" text, no progress.
    3. Quit and relaunch once, thinking it was hung. Same splash.
    4. During this the backend process was in uninterruptible disk wait (U state, ~10% CPU). No server-child*.log was written, because it never crashed.

    Desktop-side gap (in addition to the 503 body point above): #35056 made the desktop keep waiting while migration progress lines appear in the backend output tail. In this build that information stays in the main process: the splash renderer gets nothing to display. Passing the "Upgrading database (N of M)" line the desktop already parses through to the splash would have prevented the quit. A warning when closing during a migration (suggested in #35313) would have too.

    Diagnosed via t3 triage with Claude Code (Claude Opus 5.5).

  2. AlexChambers commented on Nov 14, 2026

    @AlexChambers
    Contributor

    Same on the macOS desktop app, 0.1.10-nightly.20261113.4656 (updated from 20261111.4544). The window sat on the logo splash for about 5 minutes with no progress, and I assumed it was broken.

    statev2.sqlite is 53 GB, but 27 GB of that is free pages (freelist_count 6.68M of 13.9M pages, auto_vacuum = 0). Timings from server.trace.ndjson:

    • runMigrations: 148 s in total
    • 88_TurnItemTypeStatusIndex: 138 s
    • 86_QueuedTurnInputIndex: 133 s
    • 83_CommandReadIndexes: 123 s
    • 85_TurnItemCounts: 11 s

    (The per-migration spans overlap, so they add up to more than the total.) The first ProjectionStore.readShellSnapshotRows after startup then took 17 to 26 s.

    Live data is about 30 GB: turn_items 17 GB (command output 7.4 GB, dynamic tool output 4.2 GB, spread across about 10k threads) and nodes 5 GB. Settings > Storage offers "Reclaim freed database space", but nothing prompts you to use it before an update rebuilds indexes over the whole file.

    The same 8 migrations took about 4 s on a Linux server with a smaller database.

  3. Snowy7 commented on Nov 14, 2026

    @Snowy7
    Contributor

    Same on the Windows desktop app, on a newer build than the ones above: 0.1.10-nightly.20261114.4682, auto-updated from 0.1.9-nightly.20261111.4501. The window sat on a black splash with only the T3 logo, no text or progress, for about 4 minutes. The user thought it was stuck, relaunched once mid-upgrade, and started a t3 triage session before it finished.

    Environment: Windows 11 Pro (10.0.26300) x64, desktop app with its bundled local server on :3773. statev2.sqlite is 20.7 GB: page_size 4096, 5,052,748 pages, freelist_count 2,064,566, so about 8.5 GB (41%) is free pages.

    Migration timeline from server.trace.ndjson (UTC, 2026-11-14):

    Migration Start Duration
    81_ProjectionMessageSearchText 10:47:57 7.5 s
    (app relaunched ~10:49:25)
    82_TurnItemConversationIndex 10:49:26 3.3 s
    83_CommandReadIndexes 10:49:29 67.2 s
    84_LastExecutedRunIndex 10:50:40 1.3 s
    85_TurnItemCounts 10:50:41 6.9 s
    86_QueuedTurnInputIndex 10:50:48 42.3 s
    87_ProjectionProjectsCustomTitle 10:51:30 0.0 s
    88_TurnItemTypeStatusIndex 10:51:30 4.2 s
    89_ShellThreadReadIndexes / 90_AuthConnectDevices / 91_AuthConnectDeniedRequests 10:51:39 <1 s
    server.startup 10:51:43 3.7 s

    Throughout, the desktop main process polled GET http://127.0.0.1:3773/.well-known/t3/environment about every 100 ms and got an empty 503 (StatusCode error) each time. Nothing from that reached the renderer, so the splash showed no state. So as of build 4682, the desktop still shows nothing like "upgrading database, N of M" on any platform, and each new nightly keeps adding migrations (89–91 here).

    Nothing was changed on the install. Waiting it out worked.

    Filed by Claude Code (Claude Opus 5.5) via t3 triage.

  4. EtanHey commented on Nov 15, 2026

    @EtanHey
    Contributor

    Same on the macOS desktop app, on today's build, with one extra symptom: quitting mid-migration led to "This T3 home is unavailable" on relaunch.

    Environment: T3 Code (Nightly) desktop 0.1.10-nightly.20261115.4723 (11c425b4), auto-updated from 20261114. macOS (Darwin 27.0.0), arm64. statev2.sqlite is 19.9 GB.

    Timeline (UTC, from desktop.trace.ndjson and server.trace.ndjson):

    • 06:06:33: Launch after the update. Migrations ran: 92_QueuedTurnInputIndexJsonValid took 67.0 s, 93_ProviderTurnAttemptIndex 1.9 s. The splash showed no progress, so the user quit at 06:07:46.
    • 06:07:55: Relaunch. 94_TurnItemMessageOrdinalIndex (CREATE INDEX … json_extract(payload_json,'$.messageId')) started. The user force-quit around 06:08:11.
    • 06:08:16 and 06:08:30: Each relaunch logged bootstrap found another server owning the T3 home. awaitHomeOwnershipRelease gave up after its 10 s grace, and the app showed the "T3 home is unavailable" dialog, which points at a background service. None is installed.
    • 06:08:58: The orphaned backend's Migrator 94 ended Interrupted after 62.6 s, which rolled the index back and released the lock.
    • 06:08:51 → 06:10:11: The next launch rebuilt migration 94 from scratch (68.7 s) and was ready about 80 s after it started.

    The new part: launcherWatch.awaitLauncherLoss polls ppid every 2 s on the JS thread. The synchronous CREATE INDEX blocks that thread, so the orphan can't notice the desktop is gone until the statement returns. It keeps the ownership lock for the whole build: about 47 s here, longer than the desktop's 10 s grace (BACKEND_QUIT_STOP_TIMEOUT + 3 s). Then the interrupt throws the finished index away. This is a gap in the #13747 fix while a migration is running. On large databases, a misleading home-ownership dialog then follows a mid-upgrade quit.


    Reported via T3 triage by Claude Code (Opus 5.5).

  5. WhiteWarrior625 commented on Nov 16, 2026

    @WhiteWarrior625

    Same on the Linux desktop app (AppImage, KDE Wayland), nightly 0.1.10-nightly.20261116.4774, auto-updated from the previous nightly. statev2.sqlite is 41 GB.

    Timeline (local time, from desktop.trace.ndjson, server.trace.ndjson and effect_sql_migrations):

    • 10:03:03 relaunch after the update. 90 and 91 committed at 10:03:04.
    • 92_QueuedTurnInputIndexJsonValid ran 5 min 43 s without finishing (cold page cache, full scan of turn_items). The splash showed only the logo, so the app was quit at 10:08:47, rolling 92 back.
    • 10:09:21 relaunch: 92 took 3.7 s, 93 1.0 s, 94 4.5 s (warm cache).
    • 95_TurnItemInlineWorkOutput: the backfill UPDATE (LIKE scans over dynamic_tool payload_json) took 209.8 s. The WAL grew to ~7 GB during it, and the backend RSS reached ~9.4 GB.
    • 96 started 10:13:33 and the server answered 200 shortly after.

    So the quit cost ~6 minutes, and migration 95 is a new long step on large databases beyond the index builds named above. Nightly .4787 has no commits touching migrations or the splash.

    Found via t3 triage with Claude Code (Claude Opus 5.5).

  6. kkauf commented on Nov 17, 2026

    @kkauf

    Another data point (macOS, desktop 0.1.9 → 0.1.10, statev2.sqlite 9 GB, 517k turn items).

    The window showed nothing while migrations ran, so I assumed the app was hung and quit it. That interrupted 83_CommandReadIndexes after 19 s (InterruptError in server.trace.ndjson), and the next launch had to start again. Index builds took 37–45 s each (82, 86, 88).

    The server already emits "Upgrading database N of 15" events. Showing them in the window, with a "don't quit" note and a quit warning while a migration runs, would have prevented the restart.

    Posted via Claude Code.

  7. murmaider227 commented on Nov 18, 2026

    @murmaider227

    On Windows, T3 Code Desktop Nightly 0.1.11-nightly.20261118.4864 showed the loading screen for about 7 min 13 s. Migration 95 took 5 min 37 s, and committing its transaction took another 1 min 20 s. The local endpoint returned HTTP 503 during startup, then HTTP 200. The app opened on its own.

  8. KaramanisDev commented on Nov 18, 2026

    @KaramanisDev

    Same symptom on a Windows desktop install, reporting from t3 triage on 0.1.10. Same mechanism as #35995 and #35421, on a larger database.

    Environment: Windows 11 Pro 10.0.26300, Node v26.13.1, desktop app with local backend on 127.0.0.1:3773. Upgraded from 0.1.9. Exact desktop build string not recorded.

    Evidence:

    • statev2.sqlite is 35.9 GB (page_count 8764197, freelist_count 3066454, so about 12.6 GB of free pages), with a 6.7 GB WAL that has not been checkpointed. orchestration_events max rowid is 5,005,116.
    • effect_sql_migrations: migrations 82–94 applied in one batch, 08:19:55 to 08:23:09 UTC.
    • server.trace.ndjson: Migrator 83_CommandReadIndexes 77.98 s, 85_TurnItemCounts 20.13 s, 88_TurnItemTypeStatusIndex 7.23 s, 94_TurnItemMessageOrdinalIndex 6.64 s.
    • desktop.trace.ndjson: shared.httpReadiness.waitForHttpReady timed out after 180000 ms, 1644 attempts against /.well-known/t3/environment (503 and transport errors). The desktop then reported DesktopLocalEnvironmentAuthBackendNotConfiguredError: Local backend is not configured.
    • apps/server/src/httpResponseErrorGuard.ts:54-60 returns an empty 503 until serving is attached, which matches the report.

    Not explained by the traces: about 3 minutes before migration 82 started (no span over 1 s), and readiness probes still failing for about 3 minutes after migration 94 finished at 08:23:09 (no slow span). Someone with the full logs may want to check those windows.

    Filed via t3 triage.

  9. lol2x commented on Nov 20, 2026

    @lol2x

    Same on the Windows desktop app, 0.1.11-nightly.20261119.4954 (auto-updated from 0.1.11-nightly.20261117.4851). The window sat on the splash with no progress. The user quit after ~134 s, which rolled back the migration, and assumed the update had broken the app again.

    Environment: Windows 11 Pro 10.0.26300 x64, desktop app with bundled local server on 127.0.0.1:3773. statev2.sqlite is 6.1 GB, with 308k turn items and 1.6 GB of payload_json.

    Evidence (from server.trace.ndjson):

    • Relaunch after the update at 01:07:54 UTC. The migration transaction did BEGIN and inserted the row for 99_BackgroundRecoveryIndexes. No later span until the user quit at 01:10:06, and effect_sql_migrations stayed at 98.
    • Migration 99 builds a partial index over orchestration_v2_projection_turn_items that calls json_valid/json_extract on every row. A read-only full scan of payload_json took 109.5 s cold and 2.4 s warm, so this build is dominated by cold disk reads.
    • After a warm-cache relaunch, 99 and 100 committed in ~20 s and the server reported databaseMigration: 100.

    Adds a smaller-database data point: even at 6 GB, one cold index build is longer than users will wait at a splash screen with no progress, and each new nightly with an index migration triggers it again.

    Reported via t3 triage by Claude Code (Claude Opus 5.5).

  10. nachlielp commented on Nov 21, 2026

    @nachlielp

    Same thing on the macOS desktop app, 0.1.9 → 0.1.11 stable auto-update, with a 13.2 GiB statev2.sqlite (Apple M3 Pro, 18 GB RAM). Migrations 82–102 were pending.

    On the launch that finally completed, runMigrations took 382.8 s and the window had no backend for 6 min 24 s. The desktop log recorded two BackendReadinessTimeoutError: Timed out after 180000ms entries in that time, and about 1,400 polls of /.well-known/t3/environment returned 503.

    Migrations that took more than a few seconds:

    Migration Time
    82_TurnItemConversationIndex 28.6 s
    83_CommandReadIndexes 20.9 s
    86_QueuedTurnInputIndex 32.5 s
    88_TurnItemTypeStatusIndex 29.6 s
    92_QueuedTurnInputIndexJsonValid 28.4 s
    94_TurnItemMessageOrdinalIndex 27.3 s
    95_TurnItemInlineWorkOutput 124.1 s
    99_BackgroundRecoveryIndexes 29.1 s

    Nothing on screen said an upgrade was running, so the app was quit and relaunched three times in the first two minutes. Each quit interrupted the migration in progress: 83 was rolled back twice (21.3 s and 20.9 s of work lost) before it succeeded on the third try.

    There was also a gap of about 109 s between the end of 95 and the start of 96 with no spans. A read-only dbstat query was running against the database during part of that window, so that gap may be partly self-inflicted.

    Most of the table these indexes cover is inline base64 from Claude Read images (numbers in #26591).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions