[Bug]: AsyncWebCrawler leaks Playwright driver process when browser startup fails in __aenter__

[Zzzz1111]

crawl4ai version

0.8.0

Expected Behavior

If browser startup fails while entering AsyncWebCrawler, all resources that were partially initialized during __aenter__() / start() should be cleaned up.

In particular, the Playwright node ... cli.js run-driver subprocess should exit before the exception is propagated.

Current Behavior

When Playwright starts successfully but Chromium launch fails, AsyncWebCrawler.__aenter__() propagates the exception without closing the partially initialized crawler.

Each failed attempt leaves one live direct child process:

.../playwright/driver/node .../playwright/driver/package/cli.js run-driver

Repeated failures therefore grow the process count linearly:

attempt=1 playwright_drivers=1
attempt=2 playwright_drivers=2
attempt=3 playwright_drivers=3

On Linux/Python 3.11, each leaked child can also retain an asyncio-waitpid-* watcher thread until that child exits.

The relevant lifecycle currently looks like:

async def __aenter__(self):
    return await self.start()

and browser startup initializes Playwright before launching Chromium:

self.playwright = await async_playwright().start()
self.browser = await self.playwright.chromium.launch(**browser_args)

If the second line raises, __aenter__() never completes, so Python does not call __aexit__(). The already-started Playwright driver is not rolled back.

Is this reproducible?

Yes

Inputs Causing the Bug

• Any failure after Playwright has started but before browser startup completes.
• The reproducer below makes this deterministic by pointing PLAYWRIGHT_BROWSERS_PATH at an empty temporary directory.

Steps to Reproduce

1. Install Crawl4AI 0.8.0.
2. Save and run the script below.
3. Observe that the number of Playwright driver child processes grows from 1 to 3.
4. The script terminates only the child processes it created before exiting.

Code snippets

import asyncio
import os
import tempfile
import time

import psutil

# Force BrowserType.launch() to fail after the Playwright driver has started.
empty_browsers = tempfile.mkdtemp(prefix="crawl4ai-empty-browsers-")
os.environ["PLAYWRIGHT_BROWSERS_PATH"] = empty_browsers

from crawl4ai import AsyncWebCrawler, BrowserConfig  # noqa: E402


parent = psutil.Process()
initial_pids = {p.pid for p in parent.children(recursive=True)}


def new_children():
    return [
        p
        for p in parent.children(recursive=True)
        if p.pid not in initial_pids
    ]


def playwright_drivers():
    result = []
    for process in new_children():
        try:
            command = " ".join(process.cmdline())
        except (psutil.NoSuchProcess, psutil.AccessDenied):
            continue
        if "playwright/driver" in command and "run-driver" in command:
            result.append(process)
    return result


async def fail_during_enter():
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)):
        pass


try:
    for attempt in range(1, 4):
        try:
            asyncio.run(fail_during_enter())
        except Exception as exc:
            print(f"attempt={attempt} error={type(exc).__name__}: {exc}")

        time.sleep(0.5)
        drivers = playwright_drivers()
        print(
            f"attempt={attempt} playwright_drivers={len(drivers)} "
            f"pids={[p.pid for p in drivers]}"
        )
finally:
    # Do not leave processes behind after running the reproducer.
    children = new_children()
    for process in reversed(children):
        try:
            process.terminate()
        except psutil.NoSuchProcess:
            pass

    _, alive = psutil.wait_procs(children, timeout=3)
    for process in alive:
        try:
            process.kill()
        except psutil.NoSuchProcess:
            pass
    psutil.wait_procs(alive, timeout=3)

Supporting Information

An application-level control using explicit lifecycle cleanup does not leak:

crawler = AsyncWebCrawler(config=BrowserConfig(headless=True))
try:
    await crawler.start()
finally:
    await crawler.close()

With the same forced launch failure, three attempts produce:

attempt=1 playwright_drivers=0
attempt=2 playwright_drivers=0
attempt=3 playwright_drivers=0

A possible library-level fix would be to make partial startup exception-safe, ideally in BrowserManager.start(), and add a regression test asserting that a failed browser launch leaves no Playwright driver child process:

try:
    self.playwright = await async_playwright().start()
    self.browser = await self.playwright.chromium.launch(**browser_args)
except BaseException:
    await self.close()
    raise

Related but not identical: #1242 reports an exception through the same AsyncWebCrawler.__aenter__() -> BrowserManager.start() path followed by BaseSubprocessTransport cleanup against a closed event loop.

OS

macOS 15.6.1, arm64

Python version

3.11.1

Browser

Chromium (launch intentionally forced to fail)

Browser version

N/A

Error logs & Screenshots (if applicable)

Representative error:

playwright._impl._errors.Error: BrowserType.launch:
Executable doesn't exist at <empty PLAYWRIGHT_BROWSERS_PATH>

Loop <_UnixSelectorEventLoop running=False closed=True debug=False>
that handles pid <pid> is closed

[SohamKukreti]

Thanks for the detailed report and the reproducer script. I reproduced the bug on the latest develop (0.9.2). Root cause is what you pointed out: BrowserManager.start() starts the Playwright driver first and launches the browser later with nothing catching a launch failure, so __aenter__ never completes and __aexit__ never runs. Your suggested fix is in the right direction. Will raise a PR shortly.