Repository navigation
[Bug]: flaky: cli-webmcp iframe tests intermittently miss the iframe's tools on Linux/macOS Chrome #42869
Description
Version
main (observed on 1.64.0-next between 2026-09-21 and 2026-09-22)
Steps to reproduce
The three iframe-based tests in tests/mcp/cli-webmcp.spec.ts register a WebMCP tool in the main frame and another in an <iframe>, run cli('open', url), then immediately webmcp-list / webmcp-call:
webmcp-call disambiguates same-name tools in identical same-origin frameswebmcp-call disambiguates duplicate tool names by framewebmcp-list stitches tools across frames
Expected behavior
The listing includes the iframe's tools every time.
Actual behavior
Intermittently only the main frame's tools are collected:
Expected substring: "is registered in multiple frames, retry with the frame parameter"
Received string: "### Result
Called WebMCP tool "echo" in http://localhost:8907/. ... "text": "main"
Expected substring: "- frame: http://localhost:8907/widget.html (frame 1)"
Received string: "### Result
Found 1 WebMCP tool(s). ...
From the CI results database (last ~470 runs per test, expected_status = passed):
| Test | runs | failed runs |
|---|---|---|
| disambiguates same-name tools in identical same-origin frames | 469 | 11 |
| disambiguates duplicate tool names by frame | 469 | 6 |
| webmcp-list stitches tools across frames | 469 | 3 |
Failures occurred on main (2026-09-22 00:12, 00:14, 16:32) and on unrelated PRs (#42787, #42789, #42818, #42821, #42832, #42835, #42845), only on mcp-ubuntu-latest-chrome/chromium and mcp-macos-latest-chrome/chromium, never on Windows.
Additional context
listWebMCPTools evaluates collectToolsInPage in every frame and treats a rejected or timed-out evaluation as "no tools here". open waits for load before the snapshot collects tools, so the iframe has loaded and its script has run by then. The remaining suspect is that Chromium's getTools() reflects a freshly registered tool from a child frame asynchronously, so a listing taken right after navigation can still miss it.
I can send a PR that polls the snapshot header for the expected tool count before asserting in these three tests, unless a product-side wait is preferred.
Environment
- CI: mcp-ubuntu-latest-chrome, mcp-ubuntu-latest-chromium, mcp-macos-latest-chrome, mcp-macos-latest-chromium
- Playwright: main
Activity