Skip to content

fix: explicitly close inner dispatcher stream in arun_many - #2176

Closed
Battleplus wants to merge 1 commit into
unclecode:developfrom
Battleplus:fix/2083-arun-many-stream-cleanup
Closed

fix: explicitly close inner dispatcher stream in arun_many#2176
Battleplus wants to merge 1 commit into
unclecode:developfrom
Battleplus:fix/2083-arun-many-stream-cleanup

Conversation

@Battleplus

Copy link
Copy Markdown

What

The outer result_transformer generator wraps the inner dispatcher.run_urls_stream() generator but doesn't retain a reference to it. When the outer generator is closed, the inner generator's cleanup is deferred to asynchronous-generator finalization.

Now we retain a reference to the inner generator and explicitly call aclose() on it in the finally block, ensuring cleanup completes before the outer generator returns.

Why

Fixes #2083 - AsyncWebCrawler.arun_many() stream closure does not await dispatcher cleanup.

Changes

  • Retain reference to inner dispatcher stream generator
  • Explicitly call aclose() on inner generator in finally block

Testing

  • Verified that stream cleanup now completes before outer generator returns
  • Verified that dispatcher tasks are properly cancelled and awaited

Fixes #2083

@Battleplus
Battleplus force-pushed the fix/2083-arun-many-stream-cleanup branch from e051fb2 to dfdebae Compare August 22, 2026 11:00
@Battleplus
Battleplus changed the base branch from main to develop August 22, 2026 11:00
Retain and explicitly close the dispatcher stream so arun_many(stream=True)
waits for dispatcher-owned tasks to be cancelled and awaited before the
public stream's aclose() returns. Keep proxy-session release ordered after
that cleanup, even if closing the inner stream raises.

Add a public-API regression test covering immediate task cleanup.

Fixes unclecode#2083

Signed-off-by: Battleplus <3559424769@qq.com>
@Battleplus
Battleplus force-pushed the fix/2083-arun-many-stream-cleanup branch from dfdebae to a2925ce Compare August 22, 2026 11:01
@Battleplus

Battleplus commented Aug 22, 2026

Copy link
Copy Markdown
Author

Follow-up audit completed. I rebased the PR onto the required develop branch, kept dispatcher cleanup ahead of proxy-session release even if inner.aclose() raises, and added a regression test through the public AsyncWebCrawler.arun_many(..., stream=True) API.

Validation: python -m pytest tests/async/test_dispatchers.py -q -k stream_closure -> 2 passed.

@Battleplus

Copy link
Copy Markdown
Author

Closing after the cross-PR audit found the earlier #2136, which targets develop and already implements the same inner-stream ownership fix with contextlib.aclosing() plus focused cleanup coverage. Keeping a second implementation would only split review attention.

@Battleplus Battleplus closed this Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: AsyncWebCrawler.arun_many() stream closure does not await dispatcher cleanup in 0.9.2

1 participant