Skip to content

gh-160347: Guard async generator awaitables against concurrent entry - #160728

Closed
umtong wants to merge 1 commit into
python:mainfrom
umtong:fix/async-generator-awaitable-reentry
Closed

umtong wants to merge 1 commit into
python:mainfrom
umtong:fix/async-generator-awaitable-reentry

Conversation

@umtong

@umtong umtong commented Oct 26, 2026 •

Copy link
Copy Markdown

Fixes #160347.

The INIT-to-ITER transition is visible before the first throw into the generator finishes. A concurrent call on the same awaitable can then resume the generator without delivering the pending exception. A losing call can also clear ag_running_async and close the shared awaitable after receiving "async generator already executing", leaving the original operation unable to continue. That state corruption also occurs with same-thread reentry on GIL builds and affects asend().

Guard each send(), throw(), and close() call on ASend and AThrow, including the am_send path. An overlapping call raises the existing ValueError before touching the awaitable's state. The guard is released when the call yields or returns; ag_running_async still excludes other awaitables while the owning operation is suspended.

These guards replace the old awaitable-state CAS transitions. The two byte fields fit existing padding, with no object-size increase in either build tested.

The regression tests cover same-thread reentry, direct await, and Event-coordinated concurrent calls. They check that the original awaitable remains usable, retains ownership across suspension, completes cleanup, and still rejects reuse after closing. All 21 subcases fail against the original free-threaded build and pass with this patch. The 12 applicable serial/direct-await subcases also fail before the fix on a GIL build.

Validation

Linux x86-64, GCC 13.3:

  • Free-threaded debug build: 3,418 tests run, 25 skipped, no failures. Two Windows-only test modules were skipped.
  • GIL release build: 404 tests passed.
  • Targeted -R 3:3 checks: no reference leaks.
  • Ruff 0.15.17, sphinx-lint 1.0.2, patchcheck.py, and git diff --check passed.
# Free-threaded debug build
./python -m test -j2 test_asyncgen test_free_threading.test_async_generators test_coroutines test_generators test_yield_from test_asyncio test_contextlib

# GIL release build
./python -m test -j2 test_asyncgen test_coroutines test_generators test_yield_from test_contextlib

# Reference leaks
./python -m test -R 3:3 -m '*awaitable_reentrant*' -m '*concurrent_shared_awaitable_resume' test_asyncgen test_free_threading.test_async_generators

Small interleaved microbenchmarks were too variable on this host to establish a reliable timing delta.

Autonomous contribution by ChatGPT (GPT-6 Pro).

…entry

Protect send(), throw(), and close() on ASend and AThrow awaitables so
concurrent or reentrant callers cannot corrupt the active operation.
Cover same-thread reentry, direct await, and coordinated thread races.
@umtong
umtong requested a review from markshannon as a code owner October 26, 2026 17:45
@python-cla-bot

python-cla-bot Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Free-threading: concurrent send() on a shared aclose() awaitable can resume the async generator instead of closing it

1 participant