fix: explicitly close inner dispatcher stream in arun_many - #2176
Closed
Battleplus wants to merge 1 commit into
Closed
fix: explicitly close inner dispatcher stream in arun_many#2176Battleplus wants to merge 1 commit into
Battleplus wants to merge 1 commit into
Conversation
Battleplus
force-pushed
the
fix/2083-arun-many-stream-cleanup
branch
from
August 22, 2026 11:00
e051fb2 to
dfdebae
Compare
Retain and explicitly close the dispatcher stream so arun_many(stream=True) waits for dispatcher-owned tasks to be cancelled and awaited before the public stream's aclose() returns. Keep proxy-session release ordered after that cleanup, even if closing the inner stream raises. Add a public-API regression test covering immediate task cleanup. Fixes unclecode#2083 Signed-off-by: Battleplus <3559424769@qq.com>
Battleplus
force-pushed
the
fix/2083-arun-many-stream-cleanup
branch
from
August 22, 2026 11:01
dfdebae to
a2925ce
Compare
Author
|
Follow-up audit completed. I rebased the PR onto the required Validation: |
Author
|
Closing after the cross-PR audit found the earlier #2136, which targets |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The outer
result_transformergenerator wraps the innerdispatcher.run_urls_stream()generator but doesn't retain a reference to it. When the outer generator is closed, the inner generator's cleanup is deferred to asynchronous-generator finalization.Now we retain a reference to the inner generator and explicitly call
aclose()on it in the finally block, ensuring cleanup completes before the outer generator returns.Why
Fixes #2083 - AsyncWebCrawler.arun_many() stream closure does not await dispatcher cleanup.
Changes
aclose()on inner generator in finally blockTesting
Fixes #2083