Repository navigation
Description
Note
Written by Claude Opus 5.5 (Claude Code) on behalf of the reporter, from the host's logs, its live process state and the v0.1.7 source.
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Same symptom as #24897, but its leading theory doesn't fit: Node's file-system thread pool was idle here. See "Ruled out".
Area
apps/server
Steps to reproduce
I can't reproduce this on demand. It has been seen once.
- Run T3 Code as a background service on an always-on Linux host, with several agents running and T3 Connect on.
- Stay connected from a desktop client over T3 Connect. The client renews its credential about once an hour.
- At some point
POST /api/t3-connect/mint-credentialstops completing. The client is disconnected when its current session expires.
Expected behavior
mint-credential completes or returns an error, and a stuck credential path recovers without a restart.
Actual behavior
The client shows relay_error, HTTP 504, "Could not connect relay environment." Every mint-credential request hung until the caller gave up at about 8.8 s. Other server work stayed healthy, so the host looked half alive. Only a service restart cleared it.
Timeline (UTC, 2026-11-06):
| Time | Event |
|---|---|
| 19:59:55 | Last successful credential before the incident |
| 20:10 onward | Relay requests start being cancelled, mostly scheduled-task webhooks. Per 10 minutes: 8, 15, 21, 15, 23, 30, 39 |
| 20:58:56 | First failed mint-credential, 59 minutes after the last success |
| 20:59:56 | Client disconnected, about 1 hour after the last success (sessions here have a 1 hour TTL, see #9884) |
| 20:58:56 to 21:24:27 | 53 of 53 mint-credential spans ended Interrupted with status 499 after about 8.8 s |
| 21:26:07 | t3code.service restarted |
| 21:26:49 | mint-credential succeeded in 1.6 s and the client reconnected |
No auth_pairing_links row was written between 19:59:55 and the restart, so the insert never committed. The hang could have started any time after 19:59:55, because no renewal was attempted until 20:58:56.
Impact
Major for remote use. The host can't be reached over T3 Connect until someone restarts it, and that needs a separate way in. Scheduled-task webhooks were failing at a rising rate in the same period, 115 cancelled in total. The restart cancelled 3 running agent threads.
Version or commit
Server and client 0.1.7. The service launcher process was still 0.1.6 (t3 __service-launcher).
Environment
- Ubuntu 24.04, kernel 6.8, 8 vCPU, 16 GB RAM, systemd user service, server uptime 2.3 days
- Bundled cloudflared 2026.10.4
statev2.sqlite9.81 GiB.t3 doctor --dry-runafterwards: 0 KiB reclaimable, "Nothing to repair"- Busy factory-style use: about 50 threads in the project, several running at once
Logs or stack traces
Stack from a failed span, read in full for one of the 53 (paths shortened):
InterruptError: All fibers interrupted without error
at PairingGrantStore.issueOneTimeToken
at EnvironmentAuth.issuePairingLink
at EnvironmentAuth.createPairingLink
at environment.cloud.mintCredential
at server.runServerCommand
[cause]: InterruptCause: The fiber was interrupted by: at fiber (#818)
cloudflared logged this for each attempt:
ERR Request failed error="Incoming request ended abruptly: context canceled" dest=.../api/t3-connect/mint-credential
Measured while the host was stuck:
| Check | Result |
|---|---|
GET /.well-known/t3/environment from localhost |
200 in 5.7 ms |
Thread states, read from /proc, sampled once about 2 minutes before the restart |
All 4 libuv-worker threads idle in futex_wait_queue |
| T3 server CPU, one 5 s sample | 2% |
Slowest sql.transaction in the retained traces (about 20:58 to 21:22) |
2.6 s, all Success |
| Event-loop stall warnings, 19:50 to 21:25 | Only 2: 2.2 s and 2.9 s, both before 20:40 |
| WAL file | About 4 MB, far below the 10,000-frame backpressure ceiling |
| Memory, swap, disk, OOM kills | Healthy (12 GB available, disk 66% used, no OOM) |
| Tunnel | cloudflared process up 2 days, no tunnel errors logged near the incident |
Ruled out
Measured:
- File-system thread pool exhausted (the theory in [Bug]: Host stops answering mint-credential/assets after a client sleeps mid-request: ServerSecretStore.get blocked 17.6 h behind dpop.verifyRequestDpopProof #24897): the workers were idle, though this is a single sample.
- Event loop frozen (Server event loop freezes 6–28 s on a normal-sized database (5.9 GB), so remote devices lose Send #26224): it answered in milliseconds and logged only short stalls.
- Relay or tunnel failure: requests reached the server, and the failed spans are from the server itself.
From reading the v0.1.7 source only, not tested:
- Token generation:
uuidV4andrandomBytesappear to be synchronous. - Publishing the change:
changesPubSubis unbounded, sopublishshouldn't block. - WAL admission gate (
nodeSqliteClient.ts:674):BEGINpasses through it too (:937), so a closed gate should have stopped the transactions that kept completing. For the same reason I don't expect fix(server): keep writes moving with a larger WAL limit #31804 to explain this. prepareCachestuck entry (:982):Cache.getremoves interrupted entries, andprepareis synchronous.
Root cause
Not found. By elimination, pairingLinks.create is the only step in issueOneTimeToken left that looks able to wait, but I can't show that it was waiting, or on what.
Two limits on that:
- A waiter that is interrupted never produces a completed
sql.transactionspan, so the 2.6 s maximum only describes transactions that got the writer lock. I can't fully rule out starvation of a queued writer, sinceborrowTargetfirst trieswithPermitsIfAvailable. I also can't explain why it would always hit this one statement. - The
mint-credentialspans have no retained child spans, even on success, so the logs don't show which step it reached.
The detailed traces from before the restart were overwritten: 11 files of about 10 MB last only about 20 minutes on this host, and I hadn't saved them. The figures above come from what I read before they rotated out.
After the restart the host was noticeably busier: load about 15 to 18 on 8 vCPUs, one mint-credential at 4.8 s, one sql.transaction at 15 s, and 20 waits over 9 s for the writer lock (the worst, about 10 s, at 21:35). That could be post-restart catch-up. It's worth noting because it leaves little margin against the caller's 9 s limit, but it doesn't match the stuck period, when CPU and transaction times were low.
What would help
- What should I capture if this recurs? Is there a way to dump live fibers, or to see what the writer lock is waiting on, from a running service? I'm now watching for a recurrence and will save traces and thread states straight away, and will post them here.
- A span or a timeout around
pairingLinks.create, or child spans onmint-credential, would show which step hangs.
Workaround
Restart the service on the host (systemctl --user restart t3code.service, or t3 service restart as in #24897). This clears it immediately, but stops running agent processes.
Activity
juliusmarminge commentedon Nov 6, 2026
Tried to reproduce this on main at b90e554. It did not reproduce.
POST /api/auth/pairing-token. That route goes throughEnvironmentAuth.createPairingLink, thenPairingGrantStore.issueOneTimeToken, the same path asmint-credentialafter the proof checks. The load was 17 storms of 200 to 500 concurrent requests, with most aborted at random points between 0 and 2 s. A plain mint after each storm completed in 2 to 300 ms.INSERTalways completed afterwards. The only state I could produce where writes stop while reads stay healthy is a WAL at the 32,768-frame ceiling plus a read snapshot held open. That doesn't match your ~4 MB WAL, and it cleared as soon as the snapshot was released.Your stuck spans only show that this insert never finished. They don't show whether every write was stuck or only this one. The cancellations of scheduled-task webhooks rising from 20:10 suggest it may not have been this insert alone. If it happens again, these captures would tell the two cases apart:
T3CODE_TRACE_MAX_FILES=100(and optionally raiseT3CODE_TRACE_MAX_BYTES) in the service's environment, so the files still cover the start of the hang, which can be up to an hour before the first failed renewal.t3 pairon the host. It inserts the same kind of row into the same table from a separate process. Then note whether it prints a token or hangs.ls -l statev2.sqlite*in the userdata directory (the WAL size).sqlite/sql.transactionspans from the stuck window that carrydb.connection: writer. Writer spans that completed during the window mean the writer was free. Theirdb.lock_owner_at_start.*attributes show who held it.T3 has no built-in live fiber dump, so the separate-process
t3 paircheck is the quickest way to tell a stuck in-process lock from a stuck database file.AI-assisted; verified with the tests and tools listed above.
juliusmarminge commentedon Nov 12, 2026
@nhicks00 If it happens again, before restarting can you run
t3 pairon the host and say whether it prints a token or hangs, and attach the trace files (with T3CODE_TRACE_MAX_FILES=100 set beforehand) covering the stuck window, especially any sql.transaction spans with db.connection: writer?ctrl-cheeb-del commentedon Nov 15, 2026
Note
Written by Claude Opus 5.5 (Claude Code) on behalf of a separate reporter, from the host's logs and live process state.
Hit what looks like the same root cause on 0.1.10-nightly.20261114.4682 (Linux, systemd user service). There's more evidence this time, including the commit error at shutdown.
Symptom differs slightly:
mint-credentialand/oauth/tokendid not hang. They returned 200 (slow, 3–8.6 s). The next request,/api/auth/websocket-ticket/rpc, then returned 401 withSessionStore.verify→UnknownSessionTokenError/ServerAuthInvalidCredentialError. Clients that already had a session were rejected the same way. The desktop showed "This device could not prepare the connection" on direct routes and "T3 Connect returned an error" over T3 Connect. Removing and re-adding the environment didn't help.The writer had an open transaction that never committed, for ~15 h:
statev2.sqliteandstatev2.sqlite-walstopped changing on disk at 18:47:23 UTC (statmtime and size frozen, same inode as the process's open fds) and stayed that way until the restart ~14.5 h later.sql.transaction/sqlitespans kept ending inSuccess, so the writes went into a transaction that was never committed. New sessions were then checked on areaderconnection that couldn't see them.sql.transactionspan withdb.connection: writerstarted 18:19:16 UTC. After that, onlyreadertransaction spans appear. The writer transaction that started after that point never finished. Two writer transactions shortly before took 5.7 s and 6.0 s.On
systemctl --user restart t3code, shutdown logged:After the restart the database was written again, pairing worked at once, and everything written since 18:19–18:47 was lost.
Trigger: not found. The last trace activity before the WAL froze was a 3.3 s sqlite statement at 18:47:20. The span had no statement text. No failure spans around then. The last agent run on the host ended 18:19.
To your question: I didn't get a chance to run
t3 pairbefore restarting, sorry. Given the above, I'd expect it to print a token that then fails to verify, not hang. Thecannot commit transaction - SQL statements in progresserror suggests a statement or cursor left open inside the writer transaction (an unfinished iterator or stream?) blocked the COMMIT, and the transaction was never rolled back or surfaced.juliusmarminge commentedon Nov 19, 2026
This is the same problem as #36635, so I'm closing it in favour of that one. If it is different, say how and it will be reopened.