Skip to content

API scorers: predict() fails inside a running event loop; LLMDescriptionScorer leaks an unclosed loop per instance on Python 3.14 #353

Description

@voorhs

Both API scorers run their requests with a loop kept on the instance: _init_event_loop does asyncio.get_event_loop() (falling back to new_event_loop()) and _compute_similarities calls self._event_loop.run_until_complete(aiometer.run_all(...))LLMDescriptionScorer llm_encoder.py:282-291, :247-252; TypeSafeDescriptionScorer typesafe.py:259-270, :331-340. Two consequences:

1. predict() fails inside a running event loop (both scorers)

In a notebook, a FastAPI/uvicorn handler, or any async def caller, get_event_loop() returns the running loop and run_until_complete raises RuntimeError: This event loop is already running. A pipeline whose scoring node is one of these modules cannot be served from an async server without the max_concurrent=None (sequential, sync-client) escape hatch.

2. LLMDescriptionScorer leaks an unclosed event loop per instance on Python 3.14

On 3.14 asyncio.get_event_loop() raises RuntimeError when no loop is set, so the fallback new_event_loop() runs — without set_event_loop — for every instance; each HPO trial constructs a new module, and clear_cache delattrs the loop without closing it (:274-280). Reproduced with the _init_event_loop body verbatim on CPython 3.14.3:

$ python3.14 -W always repro.py     # three "trials"
ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>   (×3)
distinct loops: 3 closed: [False, False, False]

One selector fd per trial, never released until GC. On ≤3.13 the main-thread get_event_loop() creates and sets one loop, so it is shared and this does not show. TypeSafeDescriptionScorer calls set_event_loop and reuses the loop, so it only has problem 1.

Proposed

One helper in a shared base (#357) instead of a loop on the instance: no running loop → run the batch on a fresh loop that is closed afterwards (asyncio.run semantics); running loop → run it on a worker thread with its own loop (or expose an apredict coroutine for async callers). That also removes the Dumper.dump(..., exclude=[asyncio.BaseEventLoop, dict]) special-casing in both dump methods.

Deferred from #350.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions