Repository navigation
fix(client-runtime): connection supervisor restarts its background listeners after a defect #39431
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1332,40 +1332,66 @@ export const make = Effect.fn("EnvironmentSupervisor.make")(function* ( | |
| // Registered before the fibers below so closing the scope interrupts them first. | ||
| // A loop still running when the queue shuts down would report that as a failure. | ||
| yield* Effect.addFinalizer(() => Queue.shutdown(signals).pipe(Effect.andThen(clearLease))); | ||
| yield* connectivity.changes.pipe( | ||
| Stream.runForEach((network) => | ||
| Ref.modify(intent, (current) => { | ||
| const next = targetNetwork(network); | ||
| return current.network === next | ||
| ? [false, current] | ||
| : ([true, { ...current, network: next }] as const); | ||
| }).pipe( | ||
| Effect.flatMap((changed) => | ||
| changed | ||
| ? (network === "offline" ? Effect.void : forgetSilentRoutes).pipe( | ||
| Effect.andThen(signal({ _tag: "NetworkChanged", network })), | ||
| ) | ||
| : Effect.void, | ||
| ), | ||
| // A defect in a background listener would otherwise end it for good, silently. | ||
| // Untraced: a span here would stay open for the supervisor's whole lifetime. | ||
| const keepRunning = Effect.fnUntraced( | ||
| function* (task: string, effect: Effect.Effect<void>, onRestart?: Effect.Effect<void>) { | ||
| let current = effect; | ||
| while (true) { | ||
| const exit = yield* Effect.exit(current); | ||
| if (Exit.isSuccess(exit) || Cause.hasInterruptsOnly(exit.cause)) return yield* exit; | ||
| yield* Effect.logError("Connection supervisor task failed; restarting it.").pipe( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Keep diagnostic logging inside the listener recovery boundary. The failure log at line 1343 runs outside the Guard the diagnostic construction and emission. Recover only defects, so interruption still propagates and the retry delay still runs: Suggested fix
Add a case that makes the restart-message logger throw once. Assert that the task still restarts after the delay.
|
||
| Effect.annotateLogs({ | ||
| "environment.id": target.environmentId, | ||
| "environment.label": target.label, | ||
| task, | ||
| ...safeErrorLogAttributes(exit.cause.reasons.find(Cause.isDieReason)?.defect), | ||
| }), | ||
| ); | ||
| yield* Effect.sleep(RETRY_BASE_DELAY_MS); | ||
| current = onRestart ? onRestart.pipe(Effect.andThen(effect)) : effect; | ||
| } | ||
| }, | ||
| (self) => Effect.forkScoped(self), | ||
| ); | ||
|
|
||
| const applyNetwork = (network: NetworkStatus) => | ||
| Ref.modify(intent, (current) => { | ||
| const next = targetNetwork(network); | ||
| return current.network === next | ||
| ? [false, current] | ||
| : ([true, { ...current, network: next }] as const); | ||
| }).pipe( | ||
| Effect.flatMap((changed) => | ||
| changed | ||
| ? (network === "offline" ? Effect.void : forgetSilentRoutes).pipe( | ||
| Effect.andThen(signal({ _tag: "NetworkChanged", network })), | ||
| ) | ||
| : Effect.void, | ||
| ), | ||
| ), | ||
| Effect.forkScoped, | ||
| ); | ||
| // A restarted listener may have missed changes while it was down. | ||
| yield* keepRunning( | ||
| "network-listener", | ||
| connectivity.changes.pipe(Stream.runForEach(applyNetwork)), | ||
| connectivity.status.pipe(Effect.flatMap(applyNetwork)), | ||
| ); | ||
| yield* wakeups.changes.pipe( | ||
| Stream.filter(ConnectionWakeups.reachesConnectionSupervisor), | ||
| Stream.runForEach((reason) => | ||
| // A long suspension can hide a network change the platform never reported. | ||
| (reason === "network-changed" || reason === "application-active-long" | ||
| ? forgetSilentRoutes | ||
| : Effect.void | ||
| ).pipe(Effect.andThen(signal({ _tag: "Wakeup", reason }))), | ||
| yield* keepRunning( | ||
| "wakeup-listener", | ||
| wakeups.changes.pipe( | ||
| Stream.filter(ConnectionWakeups.reachesConnectionSupervisor), | ||
| Stream.runForEach((reason) => | ||
| // A long suspension can hide a network change the platform never reported. | ||
| (reason === "network-changed" || reason === "application-active-long" | ||
| ? forgetSilentRoutes | ||
| : Effect.void | ||
| ).pipe(Effect.andThen(signal({ _tag: "Wakeup", reason }))), | ||
| ), | ||
| ), | ||
| Effect.forkScoped, | ||
| ); | ||
| yield* Queue.take(betterRouteChecks).pipe( | ||
| Effect.flatMap(checkBetterRoutes), | ||
| Effect.forever, | ||
| Effect.forkScoped, | ||
| yield* keepRunning( | ||
| "better-route-checker", | ||
| Queue.take(betterRouteChecks).pipe(Effect.flatMap(checkBetterRoutes), Effect.forever), | ||
| ); | ||
| // A restarted loop has forgotten its retry ladder, so it waits out the | ||
| // ladder's ceiling unless a signal cuts the wait short. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Await route recovery instead of advancing an hour.
Use a
Deferredto mark the injected preflight failure. Advance the restart delay and the next required check interval. Then await the connected replacement state before checking its route.The current loop drives sixty periodic checks without identifying the recovery milestone. The final
preparedassertion can also observe connection preparation before connection completes.As per path instructions, “await a Deferred, take from a Queue, or drain the worker.” Use
TestClock.adjustfor the timers that the behavior requires.Source: Path instructions