Repository navigation
Description
Follow-up to #16. That issue covers the reconnect loop. The connection supervisor starts three more background tasks next to the loop, and reviewers of the #16 fix noticed they share its weakness. This issue is the reproduction.
The problem. Every saved connection has a supervisor. Besides the reconnect loop it runs three small tasks. One listens for the device going online or offline. One listens for wakeups, such as the app coming back to the foreground. One checks whether a better route has opened up, for example the LAN instead of T3 Connect. If any of the three fails in a way nobody planned for, it stops and never restarts. Nothing is logged. The reconnect loop keeps running, so the app looks alive while one of its inputs is gone.
What the user sees. All three were reproduced in tests with an injected fault. The first was also reproduced in the web app.
| Task that died | What happens | What gets it going again |
|---|---|---|
| Network listener | The supervisor keeps its last network status forever. If that was offline, the connection stays on "This device is offline" after the network is back. | Only Retry. Returning to the app does nothing, and neither does turning the connection off and on. After a Retry the next dropped session parks it on offline again. |
| Wakeup listener | Returning to the app no longer checks the connection. A session whose socket went dead while the app was away stays "Connected". Ten simulated minutes later it is still the same dead session. | Nothing in the test. |
| Route checker | The connection stays on the slower route for as long as the session lasts. In the test it is still on T3 Connect an hour after the LAN answered again. | A dropped session. The fresh connect tries the best route first, so this one heals by accident. |
A fourth result, also tested: when the network listener dies while the status is online, the supervisor never parks. It keeps retrying on the backoff ladder, and nothing cuts the wait short when the network returns. In the test the reconnect came 209 s after the network was back, against 1 s with the listener alive. The ladder tops out at 5 minutes.
Proof. Three tests on main (2109962), one per task. Each makes the task's input fail once, then gives the supervisor every chance to recover.
× hears the network return after its connectivity stream died
AssertionError: expected 'offline' to be 'connected'
× replaces a stalled session on return to the app after its wakeup stream died
AssertionError: expected { desired: true, …(8) } to match object { phase: 'connected', generation: 2 }
- "generation": 2,
+ "generation": 1,
× still moves to a better route after one route check crashed
AssertionError: expected 'RelayConnectionTarget' to be 'BearerConnectionTarget'
All three pass when the injected fault is taken out, so the dead task is the only difference. The thrown errors' texts appear nowhere in the output. The test logger is on, because it prints the assertion errors.
In the web app. A scenario per docs/operations/scenarios.md pairs a second server as a saved environment in a real browser. The fault is injected, and this is all of it:
navigator.onLineis overridden so the browser reports offline when the page loads.window.addEventListener("online", …)throws for one of the page's four Connectivity subscriptions, the one that belongs to the saved environment's supervisor. That kills its network listener at startup.navigator.onLinegoes back to true and theonlineevent fires.
With the fault, the row still reads "This device is offline" 30 seconds later, where the scenario stops waiting. Without it, the row reads "Connected" 105 ms after the event. The browser console shows no error in either run. Throwing on each of the other three subscriptions instead leaves the environment able to reconnect, so the stuck row comes from the supervisor's own listener.
The saved environment's row in each run, after the network returned. The top one is the run without the fault.
The run with the fault, at 3x speed. Pairing, the reload that arms the fault, then the row that never leaves offline.
How likely it is. We know of no real trigger today. I read what feeds the three tasks in apps/web/src/connection/platform.ts, apps/mobile/src/connection/platform.ts, wakeups.ts and driver.ts. The listeners register DOM and React Native callbacks, and nothing in them fails once registered. The tasks' own code only updates refs and offers to a queue.
The route checker comes closest. A route check on a saved address that does not parse as a URL dies in new URL (remoteApiBaseUrl in rpc/http.ts), and Effect.option around it does not catch a defect. I confirmed that with the real fetchRemoteEnvironmentDescriptor. Unproven as a trigger: both places that save an address parse it first (normalizeHttpBaseUrl in onboarding.ts, the try in routes.ts), and I found no way to get a bad address saved.
What was not reproduced.
- The wakeup listener and the route checker in the web app. Breaking the page's
visibilitychangeregistration kills the wakeup stream for every consumer at once (thread and shell resubscription, relay discovery), so a capture would not isolate the supervisor. The route checker needs one environment with two routes, which means T3 Connect. - Mobile. Nothing was run on a device.
- The wakeup listener also carries the "network path changed" and "T3 Connect account changed" wakeups. They go through the same stream, so they would be lost too. I did not test those two.
What to do. Give the three forks the treatment #16 gives the loop: catch an unexpected failure, log its cause, start the task again. They sit right above the loop's fork in supervisor.ts. The network listener needs one more step. When it restarts it should read connectivity.status again, or a change that happened while it was down stays lost.
Related, not a duplicate: upstream PR 39223 makes the offline wait end after 30 seconds. It would soften the first row of the table without restarting the listener.
The failing tests
describe("EnvironmentSupervisor background fibers", () => {
it.effect("hears the network return after its connectivity stream died", () =>
Effect.gen(function* () {
const harness = yield* makeHarness();
const status = yield* SubscriptionRef.make<NetworkStatus>("offline");
const subscriptions = yield* Ref.make(0);
const dying = Connectivity.Connectivity.of({
status: SubscriptionRef.get(status),
changes: Stream.unwrap(
Ref.updateAndGet(subscriptions, (count) => count + 1).pipe(
Effect.map((count) =>
count === 1
? Stream.die(new Error("platform connectivity stream died"))
: SubscriptionRef.changes(status),
),
),
),
});
const supervisor = yield* EnvironmentSupervisor.make(RELAY_ENTRY, {
initiallyDesired: true,
}).pipe(
Effect.provideService(Connectivity.Connectivity, dying),
Effect.provide(harness.dependencies),
);
yield* TestClock.adjust("10 minutes");
expect((yield* SubscriptionRef.get(supervisor.state)).phase).toBe("offline");
yield* SubscriptionRef.set(status, "online");
yield* TestClock.adjust("10 minutes");
const state = yield* SubscriptionRef.get(supervisor.state);
expect(state.phase).toBe("connected");
}).pipe(Effect.provide(TestClock.layer())),
);
it.effect("replaces a stalled session on return to the app after its wakeup stream died", () =>
Effect.gen(function* () {
const harness = yield* makeHarness({ probe: () => Effect.never });
const returns = yield* SubscriptionRef.make(0);
const subscriptions = yield* Ref.make(0);
const dying = ConnectionWakeups.ConnectionWakeups.of({
changes: Stream.unwrap(
Ref.updateAndGet(subscriptions, (count) => count + 1).pipe(
Effect.map((count) =>
count === 1
? Stream.die(new Error("platform wakeup stream died"))
: SubscriptionRef.changes(returns).pipe(
Stream.drop(1),
Stream.map(() => "application-active" as const),
),
),
),
),
});
const supervisor = yield* EnvironmentSupervisor.make(TARGET_ENTRY, {
initiallyDesired: true,
}).pipe(
Effect.provideService(ConnectionWakeups.ConnectionWakeups, dying),
Effect.provide(harness.dependencies),
);
yield* TestClock.adjust("10 minutes");
expect(yield* SubscriptionRef.get(supervisor.state)).toMatchObject({
phase: "connected",
generation: 1,
});
// The user returns to the app; the session's probe never answers.
yield* SubscriptionRef.update(returns, (count) => count + 1);
yield* TestClock.adjust("15 seconds");
yield* TestClock.adjust("15 seconds");
yield* TestClock.adjust("10 minutes");
const state = yield* SubscriptionRef.get(supervisor.state);
expect(state).toMatchObject({ phase: "connected", generation: 2 });
}).pipe(Effect.provide(TestClock.layer())),
);
it.effect("still moves to a better route after one route check crashed", () =>
Effect.gen(function* () {
const lanReachable = yield* Ref.make(false);
const preflights = yield* Ref.make(0);
const harness = yield* makeHarness({
checkRoute: (route) =>
route.target._tag === "BearerConnectionTarget"
? Ref.get(lanReachable).pipe(
Effect.map((reachable) => (reachable ? "answered" : "silent")),
)
: Effect.succeed("unchecked"),
preflight: () =>
Ref.updateAndGet(preflights, (count) => count + 1).pipe(
Effect.flatMap((count) =>
count === 1
? Effect.die(new Error("route check threw"))
: Effect.succeed("ready" as const),
),
),
prepare: (_attempt, target) => Effect.succeed(preparedFor(target)),
});
const supervisor = yield* EnvironmentSupervisor.make(LAN_THEN_RELAY_ENTRY, {
initiallyDesired: true,
}).pipe(Effect.provide(harness.dependencies));
yield* awaitState(
supervisor.state,
(state) => state.phase === "connected" && state.generation === 1,
);
expect(Option.getOrThrow(yield* SubscriptionRef.get(supervisor.prepared)).target._tag).toBe(
"RelayConnectionTarget",
);
// The LAN answers again. The first check of it crashes, every later one would pass.
yield* Ref.set(lanReachable, true);
yield* harness.setNetworkStatus("unknown");
yield* harness.setNetworkStatus("online");
yield* TestClock.adjust("30 minutes");
yield* harness.wake("application-active");
yield* TestClock.adjust("30 minutes");
expect(Option.getOrThrow(yield* SubscriptionRef.get(supervisor.prepared)).target._tag).toBe(
"BearerConnectionTarget",
);
}).pipe(Effect.provide(TestClock.layer())),
);
});Add them at the end of packages/client-runtime/src/connection/supervisor.test.ts.
The web scenario
Save as scripts/scenarios/supervisor-connectivity-fiber.e2e.ts, then from scripts run FAULT=dead e2e run scenarios/supervisor-connectivity-fiber.e2e.ts --target web, and again with FAULT=control. It is a reproduction with an injected fault, not a regression test. Only the listener's first start throws, so a fix that restarts it should turn the dead run green. That is untested.
// Reproduction, not a regression test: it injects a fault into the page.
//
// FAULT=dead one Connectivity subscription fails to start (the saved environment's
// supervisor, see FAULT_TARGET), so that supervisor's connectivity fiber dies.
// FAULT=control the same page overrides, and no subscription fails.
//
// Both modes override navigator.onLine so the browser reports offline at page load, then
// report online again and fire the `online` event.
import { expect } from "e2e";
import { SECOND_HOME, startSecondServer } from "./second-server.ts";
import { makeState, test, WEB } from "./t3.ts";
const FAULT = process.env.FAULT ?? "dead";
// Which of the page's Connectivity subscriptions fails, counted from 0 in the order they start.
// With one saved environment there are four. Failing 0, 1 or 3 leaves the environment able to
// reconnect; 2 is the saved environment's supervisor.
const TARGET = Number(process.env.FAULT_TARGET ?? 2);
test(
"A saved environment connects again when the network returns",
WEB,
async ({ t3, app, screen, browser }) => {
await t3.project("demo");
await browser.emulateMedia({ colorScheme: "dark", reducedMotion: "reduce" });
await browser.addInitScript(
({ target }: { target: number }) => {
// Armed by the reload below, so pairing runs on an untouched page.
const mode = window.sessionStorage.getItem("fault");
const state = { online: mode === null, subscriptions: [] as Array<string> };
Object.assign(window, { __fault: state });
if (mode === null) return;
Object.defineProperty(Navigator.prototype, "onLine", {
configurable: true,
get: () => state.online,
});
const original = window.addEventListener;
window.addEventListener = function (this: unknown, type: string, ...rest: Array<never>) {
// Only the Connectivity service in apps/web/src/connection/platform.ts feeds supervisors.
if (type === "online" && (new Error().stack ?? "").includes("/connection/platform.ts")) {
const index = state.subscriptions.length;
const fails = mode === "dead" && index === target;
state.subscriptions.push(`${index} ${fails ? "failed (injected)" : "started"}`);
if (fails) throw new Error("injected: addEventListener('online') failed");
}
return Reflect.apply(original, this, [type, ...rest]);
} as typeof window.addEventListener;
},
{ target: TARGET },
);
const server = startSecondServer(app.baseUrl ?? "");
try {
const { host, token } = await server.ready;
// Seeding waits until the second server listens.
await makeState(SECOND_HOME).state.project("remote-demo");
// The saved environment's row reads "<address> · <status>".
const status = (text: string) => screen.getByText(`${host}/ · ${text}`, { exact: false });
await t3.open("/settings/connections");
await screen.getByRole("button", "Add environment").tap();
const dialog = screen.getByRole("dialog");
await dialog.getByPlaceholder("backend.example.com").fill(host);
await dialog.getByPlaceholder("PAIRCODE").fill(token);
await dialog.getByRole("button", "Add environment").tap();
await expect(status("Connected")).toBeVisible({ timeout: 60_000 });
await app.screenshot("1-connected-before-fault");
// The page loads again with the browser reporting offline.
await browser.evaluate((mode) => {
window.sessionStorage.setItem("fault", mode);
window.location.reload();
}, FAULT);
await expect(screen.getByText("This device is offline")).toBeVisible({ timeout: 30_000 });
await app.screenshot("2-browser-reports-offline");
// The network returns: navigator.onLine reads true and the online event fires.
await browser.evaluate(() => {
(window as unknown as { __fault: { online: boolean } }).__fault.online = true;
window.dispatchEvent(new Event("online"));
});
const report = await browser.evaluate(() => ({
subscriptions: (window as unknown as { __fault: { subscriptions: Array<string> } }).__fault
.subscriptions,
onLine: navigator.onLine,
}));
console.log(
`[fault=${FAULT}] Connectivity subscriptions: ${report.subscriptions.join(", ")}; navigator.onLine=${report.onLine}`,
);
try {
await expect(status("Connected")).toBeVisible({ timeout: 30_000 });
} finally {
await app.screenshot("3-after-network-returned");
}
} finally {
await server.stop();
}
},
);
Activity
viktorce commentedon Nov 19, 2026
Filed upstream as pingdotgg#39419. It follows pingdotgg#39418, the upstream copy of #16.
viktorce commentedon Nov 19, 2026
The upstream copy, pingdotgg#39419, is closed as not planned. I closed it on purpose: it has no fix and no known real trigger. This issue stays open here.
viktorce commentedon Nov 19, 2026
Update: the upstream copy, pingdotgg#39419, is open again. A maintainer opened pingdotgg#39431 to fix it, so my earlier closure there no longer fit.
viktorce commentedon Nov 20, 2026
Closing here. This is tracked upstream as pingdotgg#39419, and a maintainer is fixing it in pingdotgg#39431, so there is nothing left to do on the fork.