Conversation
The scorer ran its async batch with `run_until_complete` on a loop taken from `asyncio.get_event_loop()`. Called from async code (notebook, async web handler) that returned the running loop and raised "This event loop is already running". On Python 3.14 `get_event_loop()` raises when no loop is set, so every instance created a new loop that `clear_cache` never closed, leaking one loop per HPO trial. Each scorer now owns a private loop, which the generator's async client stays bound to. Batches run on it directly from sync code, or on a worker thread when the caller is already inside a running loop. The loop is closed on refit and in `clear_cache`. Closes deeppavlov#353
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved event-loop thread-safety, concurrency, and lifecycle concerns remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
What changed in this PR
Updates LLMDescriptionScorer to support synchronous prediction from running asyncio loops and manage private event-loop cleanup.
Changes:
- Adds worker-thread execution for running-loop callers.
- Closes loops during refit and cache clearing.
- Adds regression tests for async prediction and loop closure.
| File | Summary |
|---|---|
tests/modules/scoring/test_description_llm.py |
Tests running-loop prediction and loop closure. |
src/autointent/modules/scoring/_description/llm_encoder.py |
Implements private loop management and worker-thread execution. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+304
to
+308
| return self._event_loop.run_until_complete(coro) | ||
| # Called from async code (notebook, async web handler): this thread's loop is busy, | ||
| # so drive the scorer's loop from a worker thread and block until it finishes. | ||
| with ThreadPoolExecutor(max_workers=1) as executor: | ||
| return executor.submit(self._event_loop.run_until_complete, coro).result() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Closes #353 (the
LLMDescriptionScorerpart.TypeSafeDescriptionScorerisn't ondevyet).Problem
_init_event_looptook the loop fromasyncio.get_event_loop(). Whenpredict()is called from async code (a notebook, a FastAPI handler), that returns the running loop, andrun_until_completeraisesRuntimeError: This event loop is already running.get_event_loop()raises when no loop is set, so every instance fell back tonew_event_loop(), andclear_cacheremoved it withdelattrwithout closing it: one leaked loop per HPO trial.Fix
asyncio.new_event_loop()) and never touches the caller's loop. I kept one loop per instance on purpose rather than callingasyncio.runper batch: the generator'sAsyncOpenAIclient pools connections bound to the loop they were opened on, so running every batch on the same loop keeps that pool valid._run_async(coro): with no running loop in the current thread, it runs the batch directly on the scorer's loop. Inside a running loop, it runs the batch on a one-off worker thread and blocks until it finishes.predict()stays synchronous._init_event_loop) and inclear_cache.Tests
test_description_scorer_llm_predict_inside_running_loop:predict()called from a coroutine returns the same result as the sync call, and the scorer still works from sync code afterwards.test_description_scorer_llm_closes_its_event_loop: refit andclear_cacheclose the loop.devand pass with this change.tests/modulesandtests/pipelinefiltered to description/LLM: 24 passed.ruffandmypyon the changed files pass.This change touches the same lines as #361, so whichever merges second will need a one-line conflict resolved. I can rebase if needed.