Skip to content

[Internal] Fixes for merged streams - #848

Open
rkistner wants to merge 9 commits into
mainfrom
fix-merged-return
Open

rkistner wants to merge 9 commits into
mainfrom
fix-merged-return

Conversation

@rkistner

@rkistner rkistner commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

This is a rewrite of MergedAsyncIterator to avoid issues with returning.

This is currently used for two use cases:

  1. For sync responses, this injects keepalives and token-expiry warnings into the data stream.
  2. For mongodb storage checkpoint watching, this injects a periodic timer to refresh mongodb snapshot references even when there are no changes.

Conceptually, when we merge two iterators, if one closes we need to close the other one. But with how iterators work in JavaScript, if we call return() on an iterator while it is already blocking on a next(), that return waits for the next to complete first. So the current implementation did not block on return(), since it could result in deadlocks. The caller is then responsible for ensuring both iterators are aborted afterwards. An example of such an issue is in #773. The the non-blocking return meant that an uncaught error in return() could result in an uncaught rejection, that kills the process.

This re-designs the API slightly: The MergedAsyncIterator now passes in a signal to each iterator. When the MergedAsyncIterator closes, it aborts the sources directly, meaning it can safely await return. It does mean that each source must act on the AbortSignal to abort when requested.

There are no actual known issues with uncaught rejections fixed by this - it is merely a better structuring of the APIs.

This now also removes the ix dependency, which was barely used. The remaining use cases were mostly its AbortError, which we now replace with a DOMException to be consistent with the errors used by AbortController. This also improves propagation of the underlying reason on abort errors, but that does not have an actual improvement in logs that I could find.

AI Usage

Implemented using Codex gpt-6.1-sol; manually verified the approach and implementation.

@chatgpt-codex-connector

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

@changeset-bot

changeset-bot Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 4d666dd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages
Name Type
@powersync/service-core Patch
@powersync/service-module-mongodb-storage Patch
@powersync/service-module-postgres-storage Patch
@powersync/service-module-convex Patch
@powersync/service-module-core Patch
@powersync/service-module-mongodb Patch
@powersync/service-module-mssql Patch
@powersync/service-module-mysql Patch
@powersync/service-module-postgres Patch
@powersync/service-image Patch
test-client Patch
@powersync/service-schema Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ff27ec7464

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/service-core/src/streams/merge.ts
@rkistner
rkistner requested a review from simolus3 October 21, 2026 08:13
this.signal.throwIfAborted();
if (!this.initialized) {
this.initialized = true;
for (const source of this.sources) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that MergedAsyncIterator is only constructed in a Symbol.asyncIterator call, creating the inner iterators synchronously seems better.

Can we move the initializer to run in the constructor? In the unlikely event of [Symbol.asyncIterator]() itself throwing, I think we should report that synchronously instead of leaving MergedAsyncIterator in a weird half state.


function wrapPromiseWithIndex<T>(promise: Promise<T>, index: number) {
return promise.then((value) => ({ value, index })) as Promise<MergeResult<T>>;
export function mergeAsyncIterables<T>(sources: IterableSource<T>[], signal?: AbortSignal): AsyncIterable<T> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Outside of tests, we always pass an abort signal to this. So I wonder if it would be cleaner to mark it as required, but that doesn't seem to simplify much. So probably fine as-is.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants