Skip to content

gh-83055: Prevent put_nowait item in asyncio.Queue when there are woken tasks and free slot. - #160983

Closed
YvesDup wants to merge 4 commits into
python:mainfrom
YvesDup:queue-asyncio-keep-insertion-order-fix
Closed

YvesDup wants to merge 4 commits into
python:mainfrom
YvesDup:queue-asyncio-keep-insertion-order-fix

Conversation

@YvesDup

@YvesDup YvesDup commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

The proposal fixes this issue that can break the FIFO order of the queue.

With a finite size queue, when the queue is full after the retrieval of an item, a new slot is free and a pending putter is awaken. This free slot that can be stole by put_nowait statement that will insert an item before them of the pending putters including the woken putter.
IMO, a QueueFull exception should be raise with an explicit message

In same way, there is a similar bug when queue is empty. After the insertion of an item in an empty queue with pending getters, one of the pending getter is awaken to get this fresh item at the next iteration.
But this item can be get via a get_nowait statement before the awaken task.
IMO, a QueueEmpty exception should be raise with an explicit message.

+ add 3 private attributes,
+ add new "QueueWithPendingTasks" exception,
+ update put_nowait, getnowait methods, could raise this new exception,
+ update _wakeup_next method,
+ update __repr__,
+ add 2 unit tests,
@YvesDup YvesDup changed the title gh-83055: Prevent get/put item in asyncio.Queue when there are pending task or woken tasks gh-83055: Prevent get/put item in asyncio.Queue when there are pending task or woken tasks Oct 30, 2026
@YvesDup YvesDup changed the title gh-83055: Prevent get/put item in asyncio.Queue when there are pending task or woken tasks gh-83055: Prevent get/put item in asyncio.Queue when there are pending task or woken tasks Oct 30, 2026
@YvesDup

YvesDup commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

@asvetlov : A draft PR with an another proposal.

This PR also fixes this issue for the following example:

async def simplier_get_case():
    async def get_and_print(q):
        item = await q.get()
        print(f"\tGET: {item}, {str(q) = }")
 
    q = asyncio.Queue(1)
    g = 3
    for n in range(g):
        asyncio.create_task(get_and_print(q))
    await asyncio.sleep(0.0)

    # queue full: 1 item in queue, 2 items waiting into the _putters
    n = 1
    print(f'PUT: {await q.put(n) or n}')
    n += 1
    # queue not empty: 1 item in queue, 2 tasks the _putters
 
    # steal the free solt
    try:
        print(f"\tGET_NOWAIT steal a free slot: {q.get_nowait()}")
    except:
        g -= 1
        print("Failed to GET_NOWAIT item".center(60, '-'))
  
    # get all items
    for _ in range(g):
        print(f'PUT: {await q.put(n) or n}')
        n += 1
    await asyncio.sleep(0.0)

asyncio.run(simplier_get_case())

@YvesDup YvesDup changed the title gh-83055: Prevent get/put item in asyncio.Queue when there are pending task or woken tasks gh-83055: Prevent get/put[_nowait] item in asyncio.Queue when there woken tasks Nov 1, 2026
@YvesDup YvesDup changed the title gh-83055: Prevent get/put[_nowait] item in asyncio.Queue when there woken tasks gh-83055: Prevent get/put[_nowait] item in asyncio.Queue when there are woken tasks Nov 1, 2026
+ in Queue class
  - remove ``QueueWithpendingTasks`` exception and replace with existing ``QueueFull`` and ``QueueEmpty`` with a specific message.
  - simplify new tests that raise these exceptions.
  - fix update to woken counters.
+ rewrite tests.
@YvesDup YvesDup changed the title gh-83055: Prevent get/put[_nowait] item in asyncio.Queue when there are woken tasks gh-83055: Prevent put_nowait item in asyncio.Queue when there are woken tasks and free slot. Nov 4, 2026
@YvesDup YvesDup closed this Nov 4, 2026
@YvesDup

YvesDup commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Closed in favor of #160967

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant