Skip to content

[Bug]: T3 Connect host stops issuing credentials: mint-credential hangs in PairingGrantStore.issueOneTimeToken while other work stays healthy #32203

Closed as duplicate

Description

@nhicks00

Note

Written by Claude Opus 5.5 (Claude Code) on behalf of the reporter, from the host's logs, its live process state and the v0.1.7 source.

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Same symptom as #24897, but its leading theory doesn't fit: Node's file-system thread pool was idle here. See "Ruled out".

Area

apps/server

Steps to reproduce

I can't reproduce this on demand. It has been seen once.

  1. Run T3 Code as a background service on an always-on Linux host, with several agents running and T3 Connect on.
  2. Stay connected from a desktop client over T3 Connect. The client renews its credential about once an hour.
  3. At some point POST /api/t3-connect/mint-credential stops completing. The client is disconnected when its current session expires.

Expected behavior

mint-credential completes or returns an error, and a stuck credential path recovers without a restart.

Actual behavior

The client shows relay_error, HTTP 504, "Could not connect relay environment." Every mint-credential request hung until the caller gave up at about 8.8 s. Other server work stayed healthy, so the host looked half alive. Only a service restart cleared it.

Timeline (UTC, 2026-11-06):

Time Event
19:59:55 Last successful credential before the incident
20:10 onward Relay requests start being cancelled, mostly scheduled-task webhooks. Per 10 minutes: 8, 15, 21, 15, 23, 30, 39
20:58:56 First failed mint-credential, 59 minutes after the last success
20:59:56 Client disconnected, about 1 hour after the last success (sessions here have a 1 hour TTL, see #9884)
20:58:56 to 21:24:27 53 of 53 mint-credential spans ended Interrupted with status 499 after about 8.8 s
21:26:07 t3code.service restarted
21:26:49 mint-credential succeeded in 1.6 s and the client reconnected

No auth_pairing_links row was written between 19:59:55 and the restart, so the insert never committed. The hang could have started any time after 19:59:55, because no renewal was attempted until 20:58:56.

Impact

Major for remote use. The host can't be reached over T3 Connect until someone restarts it, and that needs a separate way in. Scheduled-task webhooks were failing at a rising rate in the same period, 115 cancelled in total. The restart cancelled 3 running agent threads.

Version or commit

Server and client 0.1.7. The service launcher process was still 0.1.6 (t3 __service-launcher).

Environment

  • Ubuntu 24.04, kernel 6.8, 8 vCPU, 16 GB RAM, systemd user service, server uptime 2.3 days
  • Bundled cloudflared 2026.10.4
  • statev2.sqlite 9.81 GiB. t3 doctor --dry-run afterwards: 0 KiB reclaimable, "Nothing to repair"
  • Busy factory-style use: about 50 threads in the project, several running at once

Logs or stack traces

Stack from a failed span, read in full for one of the 53 (paths shortened):

InterruptError: All fibers interrupted without error
    at PairingGrantStore.issueOneTimeToken
    at EnvironmentAuth.issuePairingLink
    at EnvironmentAuth.createPairingLink
    at environment.cloud.mintCredential
    at server.runServerCommand
  [cause]: InterruptCause: The fiber was interrupted by: at fiber (#818)

cloudflared logged this for each attempt:

ERR Request failed error="Incoming request ended abruptly: context canceled" dest=.../api/t3-connect/mint-credential

Measured while the host was stuck:

Check Result
GET /.well-known/t3/environment from localhost 200 in 5.7 ms
Thread states, read from /proc, sampled once about 2 minutes before the restart All 4 libuv-worker threads idle in futex_wait_queue
T3 server CPU, one 5 s sample 2%
Slowest sql.transaction in the retained traces (about 20:58 to 21:22) 2.6 s, all Success
Event-loop stall warnings, 19:50 to 21:25 Only 2: 2.2 s and 2.9 s, both before 20:40
WAL file About 4 MB, far below the 10,000-frame backpressure ceiling
Memory, swap, disk, OOM kills Healthy (12 GB available, disk 66% used, no OOM)
Tunnel cloudflared process up 2 days, no tunnel errors logged near the incident

Ruled out

Measured:

From reading the v0.1.7 source only, not tested:

  • Token generation: uuidV4 and randomBytes appear to be synchronous.
  • Publishing the change: changesPubSub is unbounded, so publish shouldn't block.
  • WAL admission gate (nodeSqliteClient.ts:674): BEGIN passes through it too (:937), so a closed gate should have stopped the transactions that kept completing. For the same reason I don't expect fix(server): keep writes moving with a larger WAL limit #31804 to explain this.
  • prepareCache stuck entry (:982): Cache.get removes interrupted entries, and prepare is synchronous.

Root cause

Not found. By elimination, pairingLinks.create is the only step in issueOneTimeToken left that looks able to wait, but I can't show that it was waiting, or on what.

Two limits on that:

  • A waiter that is interrupted never produces a completed sql.transaction span, so the 2.6 s maximum only describes transactions that got the writer lock. I can't fully rule out starvation of a queued writer, since borrowTarget first tries withPermitsIfAvailable. I also can't explain why it would always hit this one statement.
  • The mint-credential spans have no retained child spans, even on success, so the logs don't show which step it reached.

The detailed traces from before the restart were overwritten: 11 files of about 10 MB last only about 20 minutes on this host, and I hadn't saved them. The figures above come from what I read before they rotated out.

After the restart the host was noticeably busier: load about 15 to 18 on 8 vCPUs, one mint-credential at 4.8 s, one sql.transaction at 15 s, and 20 waits over 9 s for the writer lock (the worst, about 10 s, at 21:35). That could be post-restart catch-up. It's worth noting because it leaves little margin against the caller's 9 s limit, but it doesn't match the stuck period, when CPU and transaction times were low.

What would help

  • What should I capture if this recurs? Is there a way to dump live fibers, or to see what the writer lock is waiting on, from a running service? I'm now watching for a recurrence and will save traces and thread states straight away, and will post them here.
  • A span or a timeout around pairingLinks.create, or child spans on mint-credential, would show which step hangs.

Workaround

Restart the service on the host (systemctl --user restart t3code.service, or t3 service restart as in #24897). This clears it immediately, but stops running agent processes.

Activity

  1. juliusmarminge commented on Nov 6, 2026

    @juliusmarminge
    Member

    Tried to reproduce this on main at b90e554. It did not reproduce.

    • Real server. Ran the dev server on a fresh database and sent POST /api/auth/pairing-token. That route goes through EnvironmentAuth.createPairingLink, then PairingGrantStore.issueOneTimeToken, the same path as mint-credential after the proof checks. The load was 17 storms of 200 to 500 concurrent requests, with most aborted at random points between 0 and 2 s. A plain mint after each storm completed in 2 to 300 ms.
    • SQLite client in isolation. Ran it with the checkpoint and reader workers on, about 20k mixed writes, write transactions, read-only transactions and reads per round, with interrupts at random points. A plain INSERT always completed afterwards. The only state I could produce where writes stop while reads stay healthy is a WAL at the 32,768-frame ceiling plus a read snapshot held open. That doesn't match your ~4 MB WAL, and it cleared as soon as the snapshot was released.

    Your stuck spans only show that this insert never finished. They don't show whether every write was stuck or only this one. The cancellations of scheduled-task webhooks rising from 20:10 suggest it may not have been this insert alone. If it happens again, these captures would tell the two cases apart:

    1. Before it recurs, keep more trace history. Set T3CODE_TRACE_MAX_FILES=100 (and optionally raise T3CODE_TRACE_MAX_BYTES) in the service's environment, so the files still cover the start of the hang, which can be up to an hour before the first failed renewal.
    2. While it is stuck, before restarting, run t3 pair on the host. It inserts the same kind of row into the same table from a separate process. Then note whether it prints a token or hangs.
    3. Also while it is stuck: do running threads still update in the UI, and do new events keep arriving? Note ls -l statev2.sqlite* in the userdata directory (the WAL size).
    4. After the restart, copy the whole trace set and attach any sqlite/sql.transaction spans from the stuck window that carry db.connection: writer. Writer spans that completed during the window mean the writer was free. Their db.lock_owner_at_start.* attributes show who held it.

    T3 has no built-in live fiber dump, so the separate-process t3 pair check is the quickest way to tell a stuck in-process lock from a stuck database file.

    AI-assisted; verified with the tests and tools listed above.

  2. juliusmarminge commented on Nov 12, 2026

    @juliusmarminge
    Member

    @nhicks00 If it happens again, before restarting can you run t3 pair on the host and say whether it prints a token or hangs, and attach the trace files (with T3CODE_TRACE_MAX_FILES=100 set beforehand) covering the stuck window, especially any sql.transaction spans with db.connection: writer?

  3. ctrl-cheeb-del commented on Nov 15, 2026

    @ctrl-cheeb-del

    Note

    Written by Claude Opus 5.5 (Claude Code) on behalf of a separate reporter, from the host's logs and live process state.

    Hit what looks like the same root cause on 0.1.10-nightly.20261114.4682 (Linux, systemd user service). There's more evidence this time, including the commit error at shutdown.

    Symptom differs slightly: mint-credential and /oauth/token did not hang. They returned 200 (slow, 3–8.6 s). The next request, /api/auth/websocket-ticket/rpc, then returned 401 with SessionStore.verify → UnknownSessionTokenError / ServerAuthInvalidCredentialError. Clients that already had a session were rejected the same way. The desktop showed "This device could not prepare the connection" on direct routes and "T3 Connect returned an error" over T3 Connect. Removing and re-adding the environment didn't help.

    The writer had an open transaction that never committed, for ~15 h:

    • statev2.sqlite and statev2.sqlite-wal stopped changing on disk at 18:47:23 UTC (stat mtime and size frozen, same inode as the process's open fds) and stayed that way until the restart ~14.5 h later.
    • During that time sql.transaction / sqlite spans kept ending in Success, so the writes went into a transaction that was never committed. New sessions were then checked on a reader connection that couldn't see them.
    • In the trace files, the last sql.transaction span with db.connection: writer started 18:19:16 UTC. After that, only reader transaction spans appear. The writer transaction that started after that point never finished. Two writer transactions shortly before took 5.7 s and 6.0 s.
    • The process held 10.7 GB RSS (peak 17.3 GB), which looks like the uncommitted pages building up.
    • Load was idle. This wasn't CPU, disk or memory pressure on the host.

    On systemctl --user restart t3code, shutdown logged:

    V2 orchestration shutdown reconciliation failed
      [cause]: effect/sql/SqlError/LockTimeoutError: Failed to execute statement
        at classifySqliteError ...
        [cause]: Error: cannot commit transaction - SQL statements in progress
    

    After the restart the database was written again, pairing worked at once, and everything written since 18:19–18:47 was lost.

    Trigger: not found. The last trace activity before the WAL froze was a 3.3 s sqlite statement at 18:47:20. The span had no statement text. No failure spans around then. The last agent run on the host ended 18:19.

    To your question: I didn't get a chance to run t3 pair before restarting, sorry. Given the above, I'd expect it to print a token that then fails to verify, not hang. The cannot commit transaction - SQL statements in progress error suggests a statement or cursor left open inside the writer transaction (an unfinished iterator or stream?) blocked the COMMIT, and the transaction was never rolled back or surfaced.

  4. added
    needs more infoInitial triage showed no bug. Awaiting more info
    and removed
    needs more infoInitial triage showed no bug. Awaiting more info
    on Nov 19, 2026
  5. juliusmarminge commented on Nov 19, 2026

    @juliusmarminge
    Member

    This is the same problem as #36635, so I'm closing it in favour of that one. If it is different, say how and it will be reopened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions