` (use `select_option()`).
+
+## Cost
+
+Human input is slower than native input. These timings are from the headful
+extension E2E on a background tab, on one machine; the artifact records the
+numbers for each run:
+
+| Call | Approx. wall time |
+| --- | --- |
+| `click()` on a target in view (first use, pointer enters from an edge) | ~4 s |
+| `fill()` of 9 characters (includes its click and select-all/delete) | ~3 s |
+| `click()` on a target ~2000 px below the fold (wheel scroll + click) | ~5 s |
+
+Most of the time goes to the pointer approach. It takes at least 32 steps of
+2–4 samples, with an 8–15 ms pause between steps, and each step waits for the
+facade to forward its samples to the tab. Typing costs about 40–110 ms per
+character.
+
+## Opt-outs
+
+- Per call: `locator.click(..., human=False)`, `locator.fill(..., human=False)`,
+ and the same keyword on `page.click`/`page.fill`/`frame.click`/`frame.fill`.
+ These run exactly the native Playwright behavior.
+- Globally: `BW_HUMAN_INPUT=0` (or `false`/`off`) in the executor's
+ environment, set on the daemon before it spawns executors.
+
+Screenshot caret defaults are separate. They live in `repl/screenshot_defaults.py`
+and are not affected by these switches.
+
+## Verification
+
+```bash
+bash tests/daemon/e2e/run.sh \
+ tests/daemon/e2e/test_human_input.py \
+ tests/daemon/e2e/test_human_scroll.py \
+ tests/daemon/e2e/test_human_pointer_layout.py \
+ tests/daemon/e2e/test_human_input_no_focus.py -v
+```
+
+Artifacts in `tests/daemon/e2e/_artifacts/`:
+
+- `human-input.json`: trusted key/pointer events, coalesced samples, opt-outs
+ and input-type compatibility.
+- `human-scroll.json` / `human-scroll-bottom.json`: wheel→scroll ordering for
+ the document, nested scrollers and frames.
+- `human-pointer-layout.json`: pixel-grid snapping when the target moves
+ during the approach.
+- `human-input-no-focus.json`: Chrome tab/window state before and after,
+ activation/focus events, and per-call timings on a background tab.
+
+The first three use headless Chrome with only the extension channel, which
+keeps desktop cursor movement out of the event stream. The no-focus test is
+headful by default.
diff --git a/src/browserwright/_executor/process.py b/src/browserwright/_executor/process.py
index 49eaa1b..622feb7 100644
--- a/src/browserwright/_executor/process.py
+++ b/src/browserwright/_executor/process.py
@@ -1186,10 +1186,12 @@ def _bind_live_surface(self, g: dict[str, Any]) -> None:
Called by ``build_globals`` BEFORE ``_load_agent_helpers``, so the
helper modules are seeded from the live surface too.
"""
+ from ..repl.human_input import install_human_input
from ..repl.markdown import make_read_markdown
from ..repl.screenshot_defaults import install_screenshot_defaults
install_screenshot_defaults(self._context)
+ install_human_input(self._context)
g.pop("__bw_playwright_handle__", None)
g["page"] = self._page
diff --git a/src/browserwright/daemon/server/extension_upstream.py b/src/browserwright/daemon/server/extension_upstream.py
index e5035cd..95a6401 100644
--- a/src/browserwright/daemon/server/extension_upstream.py
+++ b/src/browserwright/daemon/server/extension_upstream.py
@@ -14,7 +14,7 @@
- `Target.setDiscoverTargets` / `Target.setAutoAttach` → silent ack
(we don't need Chrome's discover stream — ghost targets come from the
extension via "attached"/"detached" event types instead)
- - `Browser.getVersion` → daemon-stamped result, used for heartbeat
+ - `Browser.getVersion` → extension-reported browser identity (heartbeat shape)
- `Browser.crash`, `Browser.close` and other unsupported browser-level
methods → -32601 ("method not implemented in extension backend")
@@ -29,6 +29,7 @@
import json
import logging
import os
+import re
import secrets
import time
from typing import Any, Awaitable, Callable, TYPE_CHECKING
@@ -1070,16 +1071,9 @@ async def send_text(self, frame: str) -> None:
return
if method == "Browser.getVersion":
- # Heartbeat — daemon-internal. Return a stable shape so the
- # proxy doesn't choke on the heartbeat loop in UpstreamConnection
- # land (not used in extension backend, but symmetric).
- await self._respond(req_id, {
- "product": f"browserwright-daemon-extension/{__version__}",
- "userAgent": "extension-relay",
- "protocolVersion": "1.3",
- "revision": "0",
- "jsVersion": "0",
- })
+ # Same stable CDP shape heartbeat callers rely on, carrying the
+ # extension-reported browser identity when there is one.
+ await self._respond(req_id, self._browser_version())
return
if isinstance(method, str) and method in _UNSUPPORTED_BROWSER_METHODS:
@@ -1444,15 +1438,30 @@ async def send_command(self, method: str, params: dict | None = None,
if method == "Target.setDiscoverTargets":
return {}
if method == "Browser.getVersion":
- return {
- "product": f"browserwright-daemon-extension/{__version__}",
- "userAgent": "extension-relay",
- "protocolVersion": "1.3",
- "revision": "0",
- "jsVersion": "0",
- }
+ return self._browser_version()
return {}
+ def _browser_version(self) -> dict:
+ """`Browser.getVersion`, which chrome.debugger cannot answer itself.
+
+ Playwright derives the browser platform from ``userAgent``
+ (``Macintosh`` → mac) and uses it to resolve ``ControlOrMeta`` and to
+ attach macOS editing commands to shortcuts such as Meta+A. The
+ extension's service worker reports its own user agent in the relay
+ hello; no page is read. Older extensions that don't report one keep
+ the legacy identity.
+ """
+ user_agent = self._relay.browser_user_agent()
+ version = re.search(r"\b(?:Headless)?Chrome/[\d.]+", user_agent)
+ return {
+ "product": version.group(0) if version else
+ f"browserwright-daemon-extension/{__version__}",
+ "userAgent": user_agent or "extension-relay",
+ "protocolVersion": "1.3",
+ "revision": "0",
+ "jsVersion": "0",
+ }
+
# ---- helpers ---------------------------------------------------------
# ---- relay lifecycle events (ADR-0013 inputs) ------------------------
diff --git a/src/browserwright/daemon/server/facade_extension.py b/src/browserwright/daemon/server/facade_extension.py
index 7c3bbc2..bf22f09 100644
--- a/src/browserwright/daemon/server/facade_extension.py
+++ b/src/browserwright/daemon/server/facade_extension.py
@@ -57,6 +57,7 @@
import contextlib
import json
import logging
+import math
import os
import secrets
from collections import deque
@@ -208,6 +209,14 @@ def __init__(
# debugger. Reusing a primary session crashes Playwright's CDP router.
self._aux_sessions: dict[str, int] = {}
self._closed = False
+ # In-flight pointer-move forwards (see `_queue_mouse_move`).
+ self._mouse_moves: set[asyncio.Task] = set()
+ # Device-pixel grid leases for the executor's generated clicks
+ # (`Browserwright.setPointerGrid`). Scale, owning CDP session and the
+ # last dispatched point live here, outside every page execution world.
+ self._pointer_grids: dict[int, float] = {}
+ self._pointer_grid_owners: dict[int, str] = {}
+ self._pointer_grid_points: dict[int, dict[str, float]] = {}
# Guard concurrent synthesis (autoAttach replay vs fan-out attach) so we
# never announce the same tab twice.
self._lock = asyncio.Lock()
@@ -305,10 +314,54 @@ async def run(self) -> None:
continue
text = raw if isinstance(raw, str) else raw.decode(
"utf-8", errors="replace")
+ if self._queue_mouse_move(text):
+ continue
+ # Pointer-move ACKs arrive only after renderer dispatch. Let a
+ # run of consecutive samples reach Chrome together (so it can
+ # coalesce them like a real device), then finish the run
+ # before a press or any other protocol operation.
+ await self._drain_mouse_moves()
await self._handle_client_frame(text)
finally:
+ await self._drain_mouse_moves()
await self.aclose()
+ def _queue_mouse_move(self, frame: str) -> bool:
+ """Forward a session-scoped `mouseMoved` without waiting for its ACK.
+
+ Only plain pointer motion is parallelized; presses, keys and every
+ stateful handshake stay strictly sequential through
+ `_handle_client_frame`. Each response keeps its own id/sessionId.
+ """
+ try:
+ msg = json.loads(frame)
+ except (ValueError, TypeError):
+ return False
+ if not isinstance(msg, dict) or msg.get("method") != "Input.dispatchMouseEvent":
+ return False
+ params = msg.get("params")
+ sid = msg.get("sessionId")
+ if (not isinstance(params, dict) or params.get("type") != "mouseMoved"
+ or not isinstance(sid, str) or self._tab_id_for_session(sid) is None):
+ return False
+ req_id = msg.get("id") if isinstance(msg.get("id"), int) else None
+ task = asyncio.create_task(self._forward_session_command(
+ req_id, sid, "Input.dispatchMouseEvent", params))
+ self._mouse_moves.add(task)
+ task.add_done_callback(self._mouse_moves.discard)
+ return True
+
+ async def _drain_mouse_moves(self) -> None:
+ if self._mouse_moves:
+ moves = self._mouse_moves.copy()
+ await asyncio.gather(*moves, return_exceptions=True)
+ self._mouse_moves.difference_update(moves)
+
+ def _drop_pointer_grid(self, tab_id: int) -> None:
+ self._pointer_grids.pop(tab_id, None)
+ self._pointer_grid_owners.pop(tab_id, None)
+ self._pointer_grid_points.pop(tab_id, None)
+
async def aclose(self) -> None:
if self._closed:
return
@@ -321,6 +374,9 @@ async def aclose(self) -> None:
self._context_refresh_tasks.clear()
self._context_refresh_pending.clear()
self._frame_generations.clear()
+ self._pointer_grids.clear()
+ self._pointer_grid_owners.clear()
+ self._pointer_grid_points.clear()
self._pending_console_text.clear()
# NOTE: we deliberately do NOT call self._ext.close() — ExtensionUpstream
# .close() calls relay.set_event_handler(None), which would clobber the
@@ -463,7 +519,9 @@ async def _handle_client_frame(self, frame: str) -> None:
if session_id is None:
if method == "Target.detachFromTarget" and params.get("sessionId") in self._aux_sessions:
detached_sid = params["sessionId"]
- self._aux_sessions.pop(detached_sid)
+ detached_tab = self._aux_sessions.pop(detached_sid)
+ if self._pointer_grid_owners.get(detached_tab) == detached_sid:
+ self._drop_pointer_grid(detached_tab)
await self._respond(req_id, {})
await self._send_to_client(json.dumps({
"method": "Target.detachedFromTarget",
@@ -502,6 +560,41 @@ async def _handle_client_frame(self, frame: str) -> None:
# it here because it talks raw flat-session CDP to Playwright. ---
await self._forward_session_command(req_id, session_id, method, params)
+ async def _set_pointer_grid(self, req_id: int | None, session_id: str,
+ tab_id: int, params: dict) -> None:
+ """`Browserwright.setPointerGrid` — a facade-only method.
+
+ ``{"devicePixelRatio": r}`` leases dispatch-time rounding of this tab's
+ `Input.dispatchMouseEvent` coordinates to the physical pixel grid;
+ ``{"devicePixelRatio": null}`` releases the lease and returns the last
+ dispatched point. Only the executor's generated clicks use it (see
+ `repl/human_input.py`): Playwright's final actionability pass may move
+ the target after the pointer approach, so the snap happens here, at
+ native dispatch. The lease belongs to the CDP session that took it;
+ detaching that session or losing the tab drops it.
+ """
+ scale = params.get("devicePixelRatio")
+ if scale is None:
+ point = None
+ if self._pointer_grid_owners.get(tab_id) == session_id:
+ point = self._pointer_grid_points.get(tab_id)
+ self._drop_pointer_grid(tab_id)
+ await self._respond(req_id, {"point": point}, session_id=session_id)
+ return
+ value = 0.0
+ if isinstance(scale, (int, float)) and not isinstance(scale, bool):
+ with contextlib.suppress(OverflowError, ValueError):
+ value = float(scale)
+ if not (math.isfinite(value) and value > 0):
+ await self._error(req_id, -32602,
+ "devicePixelRatio must be positive and finite",
+ session_id=session_id)
+ return
+ self._pointer_grids[tab_id] = value
+ self._pointer_grid_owners[tab_id] = session_id
+ self._pointer_grid_points.pop(tab_id, None)
+ await self._respond(req_id, {}, session_id=session_id)
+
def _command_wait_s(self) -> float | None:
"""How long a forwarded command may wait in the relay: derived from
the deadline of the agent call currently running on this session
@@ -521,6 +614,11 @@ async def _forward_session_command(self, req_id: int | None,
f"unknown sessionId {session_id!r}",
session_id=session_id)
return
+ if method == "Browserwright.setPointerGrid":
+ await self._set_pointer_grid(req_id, session_id, tab_id, params)
+ return
+ if method == "Input.dispatchMouseEvent" and tab_id in self._pointer_grids:
+ params = _snap_to_grid(params, self._pointer_grids[tab_id])
# PR3: a command scoped to the synthetic main-frame id (which equals the
# targetId we handed Playwright) must target the REAL Chrome frame id.
self._rewrite_command_frame_id(tab_id, params)
@@ -531,6 +629,9 @@ async def _forward_session_command(self, req_id: int | None,
try:
result = await self._relay.send_cdp(
tab_id, method or "", params, timeout=self._command_wait_s())
+ if method == "Input.dispatchMouseEvent" and tab_id in self._pointer_grids:
+ self._pointer_grid_points[tab_id] = {
+ axis: params[axis] for axis in ("x", "y") if axis in params}
if method == "Runtime.enable":
self._native_runtime_tabs.add(tab_id)
elif method == "Runtime.disable":
@@ -1431,6 +1532,7 @@ def _evict_tab(self, tab_id: int) -> None:
"""Drop all per-tab state for a closed/detached tab and wake any
outstanding Runtime.enable barrier so it doesn't hang on a dead tab."""
self._tab_sessions.pop(tab_id, None)
+ self._drop_pointer_grid(tab_id)
self._quiet_runtime_tabs.discard(tab_id)
self._native_runtime_tabs.discard(tab_id)
self._execution_contexts.pop(tab_id, None)
@@ -1473,3 +1575,20 @@ async def _error(self, req_id: int | None, code: int, msg: str,
async def _noop_close(self, reason: str) -> None:
return
+
+
+def _snap_to_grid(params: dict, scale: float) -> dict:
+ """Round a mouse event's x/y to the device-pixel grid of ``scale``.
+
+ Malformed coordinates pass through untouched so Chrome reports its normal
+ invalid-parameter error.
+ """
+ try:
+ if all(isinstance(params.get(axis), (int, float))
+ and not isinstance(params[axis], bool)
+ and math.isfinite(params[axis] * scale) for axis in ("x", "y")):
+ return {**params, **{axis: round(params[axis] * scale) / scale
+ for axis in ("x", "y")}}
+ except (OverflowError, ValueError):
+ pass
+ return params
diff --git a/src/browserwright/daemon/server/relay.py b/src/browserwright/daemon/server/relay.py
index 89d590a..590993c 100644
--- a/src/browserwright/daemon/server/relay.py
+++ b/src/browserwright/daemon/server/relay.py
@@ -230,6 +230,7 @@ class _ExtensionConn:
version: str = ""
browserwright_version: str = ""
extension_protocol_version: str = ""
+ user_agent: str = ""
version_drift: str = VersionDrift.UNKNOWN.value
# Where this extension came from, classified at ws handshake from the
# Origin header: "store" (Chrome Web Store item id), "development"
@@ -846,6 +847,11 @@ def status_payload(self) -> dict:
"tab_count": sum(len(e.tabs) for e in self._extensions.values()),
}
+ def browser_user_agent(self) -> str:
+ """Browser user agent reported by the active extension's service worker."""
+ ext = self._pick_active_extension()
+ return ext.user_agent if ext is not None else ""
+
async def reload_extensions(
self,
*,
@@ -1345,6 +1351,7 @@ async def _dispatch_from_extension(self, ext: _ExtensionConn,
ext.connection_generation = self._connection_generation
ext.install_id = str(msg.get("installId") or "")
ext.browser = str(msg.get("browser") or "")
+ ext.user_agent = str(msg.get("userAgent") or "")
ext.version = str(msg.get("version") or "")
ext.browserwright_version = str(msg.get("browserwrightVersion") or ext.version)
ext.extension_protocol_version = str(
diff --git a/src/browserwright/repl/human_input.py b/src/browserwright/repl/human_input.py
new file mode 100644
index 0000000..812c9b7
--- /dev/null
+++ b/src/browserwright/repl/human_input.py
@@ -0,0 +1,675 @@
+"""Input defaults for the executor's live Playwright surface.
+
+Install once on the resident context, before handing it to agent helpers. Real
+Playwright objects remain real objects (including chained locators and frames).
+Only registered contexts use these defaults. ``human=False`` on click/fill, or
+``BW_HUMAN_INPUT=0``, restores native defaults. Native Playwright still
+performs the final click and its actionability/navigation checks; fill
+validates editability and clears existing text before keyboard typing.
+
+Nothing here activates a tab or focuses a window: the extension backend drives
+the user's own Chrome, and browser UI activation steals the desktop's active
+window. Every step works on a background tab (see ``docs/human-input.md``).
+Screenshot defaults live in ``screenshot_defaults.py``.
+"""
+from __future__ import annotations
+
+import asyncio
+import functools
+import json
+import math
+import os
+import random
+import time
+import weakref
+from datetime import timedelta
+from fractions import Fraction
+from typing import Any
+
+_CONTEXTS: weakref.WeakSet[Any] = weakref.WeakSet()
+_SCROLL_CONTEXTS: weakref.WeakSet[Any] = weakref.WeakSet()
+_POINTERS: weakref.WeakKeyDictionary[Any, tuple[float, float]] = weakref.WeakKeyDictionary()
+_BUTTONS: weakref.WeakKeyDictionary[Any, set[str]] = weakref.WeakKeyDictionary()
+_INSTALLED = False
+_RNG = random.SystemRandom()
+_TEXT_TYPES = {None, "", "text", "search", "email", "url", "tel", "password"}
+_SCROLL_ENGINE = '_browserwright_scroll'
+_SCROLL_SOURCE = r'''({
+ query(root, selector) { return this.queryAll(root, selector)[0] || null; },
+ queryAll(root, selector) {
+ if (!(root instanceof Element)) return [];
+ const rect = e => {
+ const r = e.getBoundingClientRect();
+ return {x:r.x,y:r.y,width:r.width,height:r.height};
+ };
+ const parent = e => e.parentElement || e.getRootNode().host || null;
+ const canScroll = (e, axis) => {
+ if (e === document.scrollingElement)
+ return axis === 'x' ? e.scrollWidth > innerWidth : e.scrollHeight > innerHeight;
+ const s = getComputedStyle(e);
+ return /^(auto|scroll|overlay)$/.test(axis === 'x' ? s.overflowX : s.overflowY) &&
+ (axis === 'x' ? e.scrollWidth > e.clientWidth : e.scrollHeight > e.clientHeight);
+ };
+ const jobs = [];
+ let target = rect(root);
+ const position = JSON.parse(selector || '{}').position;
+ if (position) target = {x:target.x+root.clientLeft+position.x,
+ y:target.y+root.clientTop+position.y,width:0,height:0};
+ for (let e = parent(root); e; e = parent(e)) {
+ if (e === document.scrollingElement) continue;
+ const sx = canScroll(e,'x'), sy = canScroll(e,'y');
+ if (!sx && !sy) continue;
+ const r = rect(e), scaleX = e.offsetWidth ? r.width/e.offsetWidth : 1;
+ const scaleY = e.offsetHeight ? r.height/e.offsetHeight : 1;
+ jobs.push({target,clip:{x:r.x+e.clientLeft*scaleX,y:r.y+e.clientTop*scaleY,
+ width:e.clientWidth*scaleX,height:e.clientHeight*scaleY},sx,sy,element:e});
+ target = r;
+ }
+ jobs.push({target,clip:{x:0,y:0,width:innerWidth,height:innerHeight},
+ sx:true,sy:true,element:document.scrollingElement});
+ const adjustment = (start,size,low,length) => {
+ if (size <= length && start >= low && start+size <= low+length) return 0;
+ const margin = Math.min(8,length/10);
+ if (size > length-2*margin) return start+size/2-low-length/2;
+ if (start < low+margin) return start-low-margin;
+ if (start+size > low+length-margin) return start+size-low-length+margin;
+ return 0;
+ };
+ for (const job of jobs) {
+ const {target:t,clip:c} = job;
+ job.dx = job.sx ? adjustment(t.x,t.width,c.x,c.width) : 0;
+ job.dy = job.sy ? adjustment(t.y,t.height,c.y,c.height) : 0;
+ const left=Math.max(1,c.x),top=Math.max(1,c.y);
+ const right=Math.min(innerWidth-1,c.x+c.width),bottom=Math.min(innerHeight-1,c.y+c.height);
+ job.point = null;
+ if (right > left && bottom > top) {
+ // Prefer exposed space belonging to this scroller; a nested scroller
+ // can otherwise consume the wheel before its parent sees it.
+ for (const fy of [.5,.1,.9,.02,.98]) for (const fx of [.5,.1,.9,.02,.98]) {
+ const x=left+(right-left)*fx,y=top+(bottom-top)*fy;
+ let hit=document.elementFromPoint(x,y);
+ while (hit && hit !== job.element &&
+ !((job.dx && canScroll(hit,'x')) || (job.dy && canScroll(hit,'y')))) hit=parent(hit);
+ if (hit === job.element || (!hit && job.element === document.scrollingElement)) {
+ job.point={x,y}; break;
+ }
+ }
+ if (!job.point) job.point={x:(left+right)/2,y:(top+bottom)/2};
+ }
+ delete job.element;
+ }
+ const r=rect(root), out=document.createElement('span');
+ let caret=null;
+ if (root.isContentEditable) {
+ const s=root.ownerDocument.getSelection();
+ caret={inside:!!s && root.contains(s.anchorNode) && root.contains(s.focusNode),point:null};
+ for (const fx of [.95,.05,.5]) for (const fy of [.5,.1,.9]) {
+ const x=r.x+r.width*fx,y=r.y+r.height*fy;
+ if (root.ownerDocument.elementFromPoint(x,y) === root) {
+ caret.point={x:r.width*fx-root.clientLeft,y:r.height*fy-root.clientTop};break;
+ }
+ }
+ }
+ const active=root.getRootNode().activeElement || root.ownerDocument.activeElement;
+ out.textContent=JSON.stringify({rect:r,jobs,focused:active === root || root.contains(active),
+ viewport:{width:innerWidth,height:innerHeight,
+ dpr:devicePixelRatio || 1},clientLeft:root.clientLeft,clientTop:root.clientTop,
+ offsetWidth:root.offsetWidth,offsetHeight:root.offsetHeight,caret});
+ // A detached transport node never changes the document or its styles.
+ return [out];
+ }
+})'''
+_SHIFT_KEYS = dict(zip('~!@#$%^&*()_+{}|:"<>?', (
+ 'Backquote', 'Digit1', 'Digit2', 'Digit3', 'Digit4', 'Digit5', 'Digit6',
+ 'Digit7', 'Digit8', 'Digit9', 'Digit0', 'Minus', 'Equal', 'BracketLeft',
+ 'BracketRight', 'Backslash', 'Semicolon', 'Quote', 'Comma', 'Period', 'Slash',
+)))
+
+
+def _enabled(page: Any, human: bool | None) -> bool:
+ return (page.context in _CONTEXTS and human is not False
+ and os.environ.get("BW_HUMAN_INPUT", "1").lower() not in {"0", "false", "off"})
+
+
+class _Budget:
+ """Share one operation timeout across preparation, input, and native action."""
+
+ def __init__(self, page: Any, timeout: Any):
+ if isinstance(timeout, timedelta):
+ timeout = timeout.total_seconds() * 1000
+ ms = page._impl_obj._timeout_settings.timeout(timeout)
+ self.end = time.monotonic() + ms / 1000 if ms else None
+
+ def remaining(self) -> float:
+ if self.end is None:
+ return 0
+ ms = (self.end - time.monotonic()) * 1000
+ if ms <= 0:
+ from playwright.sync_api import TimeoutError
+ raise TimeoutError("Timeout exceeded during browserwright input")
+ return ms
+
+ def pause(self, seconds: float) -> None:
+ remaining = self.remaining()
+ if remaining and seconds * 1000 > remaining:
+ seconds = remaining / 1000
+ time.sleep(seconds)
+ self.remaining()
+
+
+def _mouse_samples(page: Any, points: list[tuple[float, float]], budget: _Budget) -> None:
+ """Queue one small group of native samples before waiting for their ACKs.
+
+ Playwright's public synchronous mouse.move waits for each dispatched event.
+ Real pointer devices can supply several samples before Chrome paints; the
+ implementation API keeps Playwright's button/modifier and drag handling
+ while allowing Chrome to coalesce samples through its normal input queue.
+ """
+ remaining = budget.remaining()
+ if _BUTTONS.get(page.mouse):
+ # Playwright's drag interception owns shared state while a button is
+ # held. Keep those moves sequential, as its public API normally does.
+ for px, py in points:
+ budget.remaining()
+ page.mouse.move(px, py)
+ budget.remaining()
+ return
+
+ async def send() -> None:
+ tasks = [asyncio.create_task(page.mouse._impl_obj.move(px, py))
+ for px, py in points]
+ try:
+ batch = asyncio.gather(*tasks)
+ if remaining:
+ await asyncio.wait_for(batch, timeout=remaining / 1000)
+ else:
+ await batch
+ except BaseException:
+ for task in tasks:
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+ raise
+
+ try:
+ page._sync(send())
+ except asyncio.TimeoutError as exc:
+ from playwright.sync_api import TimeoutError
+ raise TimeoutError('Timeout exceeded during browserwright input') from exc
+ _POINTERS[page.mouse] = points[-1]
+ budget.remaining()
+
+
+def _approach(page: Any, x: float, y: float, budget: _Budget, metrics: dict[str, float]) -> None:
+ ratio = metrics['dpr']
+ previous = _POINTERS.get(page.mouse)
+ if previous is None:
+ # Until input has been sent, the pointer is outside this viewport.
+ # Enter at a screen edge rather than inventing a previous (0, 0).
+ start_x = 0 if _RNG.random() < .5 else metrics['width'] - 1
+ start_y = _RNG.uniform(.15, .85) * metrics['height']
+ else:
+ start_x, start_y = previous
+ distance = math.hypot(x - start_x, y - start_y)
+ # Sample by distance, with enough cadence for short paths too. A small
+ # sideways arc keeps the approach smooth rather than jumping between a
+ # handful of remote points.
+ steps = min(512, max(32, math.ceil(distance / 6)))
+ bend = _RNG.uniform(-min(20, max(3, distance / 8)), min(20, max(3, distance / 8)))
+ for step in range(1, steps + 1):
+ # Retain the visible path cadence and density. Each frame-sized group
+ # contains several finer device samples, rather than stretching the
+ # original waypoints farther apart when the browser coalesces them.
+ count = _RNG.randint(2, 4)
+ points = []
+ for sample in range(1, count + 1):
+ fraction = (step - 1 + sample / count) / steps
+ eased = fraction * fraction * (3 - 2 * fraction)
+ arc = math.sin(math.pi * fraction) * bend
+ px = start_x + (x - start_x) * eased
+ py = start_y + (y - start_y) * eased
+ if distance:
+ px += arc * (y - start_y) / distance
+ py -= arc * (x - start_x) / distance
+ else:
+ px += arc
+ points.append((round(max(0, px) * ratio) / ratio,
+ round(max(0, py) * ratio) / ratio))
+ _mouse_samples(page, points, budget)
+ budget.pause(_RNG.uniform(0.008, 0.015))
+
+
+def _border_width(locator: Any, property_name: str, budget: _Budget) -> float:
+ # Read computed CSS via Playwright's utility-world expectation transport.
+ # Locator.evaluate runs in the main world and would expose internal reads.
+ result = locator._sync(locator._impl_obj._expect('to.have.css', {
+ 'expressionArg': property_name, 'expectedText': [{'regexSource': '.*', 'regexFlags': ''}],
+ 'isNot': False, 'timeout': budget.remaining(),
+ }))
+ received = result['received']
+ if isinstance(received, dict) and 'value' in received:
+ received = received['value']
+ if isinstance(received, dict):
+ # Older Playwright releases return the wire-format value here.
+ from playwright._impl._js_handle import parse_value
+ received = parse_value(received)
+ if not isinstance(received, str):
+ raise ValueError(f'Unexpected computed CSS result for {property_name}')
+ # Match Playwright's parseInt border offset, including scaled screens.
+ return int(float(received.removesuffix('px')))
+
+
+def _scroll_plan(handle: Any, budget: _Budget, position: Any = None) -> dict[str, Any]:
+ budget.remaining()
+ transport = handle.query_selector(f'{_SCROLL_ENGINE}={json.dumps({"position": position})}')
+ if transport is None:
+ from playwright.sync_api import Error
+ raise Error('Unable to inspect scrolling target')
+ try:
+ result = json.loads(transport.text_content())
+ finally:
+ transport.dispose()
+ budget.remaining()
+ return result
+
+
+def _wheel_target(page: Any, handle: Any, frame: Any, budget: _Budget, position: Any = None) -> None:
+ """Reveal a target using native wheel input, including frame ancestors."""
+ offset_x = offset_y = 0.0
+ scale_x = scale_y = 1.0
+ if frame.parent_frame is not None:
+ owner = frame.frame_element()
+ try:
+ _wheel_target(page, owner, frame.parent_frame, budget)
+ owner_plan = _scroll_plan(owner, budget)
+ box = owner.bounding_box()
+ if box is not None:
+ scale_x = box['width'] / (owner_plan['offsetWidth'] or box['width'])
+ scale_y = box['height'] / (owner_plan['offsetHeight'] or box['height'])
+ offset_x = box['x'] + owner_plan['clientLeft'] * scale_x
+ offset_y = box['y'] + owner_plan['clientTop'] * scale_y
+ finally:
+ owner.dispose()
+ from .isolated_world import isolated_evaluate
+ metrics = isolated_evaluate(page, '() => ({width:innerWidth,height:innerHeight,dpr:devicePixelRatio || 1})')
+ while True:
+ plan = _scroll_plan(handle, budget, position)
+ if plan['rect']['width'] < 2 or plan['rect']['height'] < 2:
+ return # Native click supplies its normal hidden-element error.
+ job = next((j for j in reversed(plan['jobs'])
+ if abs(j['dx']) > .5 or abs(j['dy']) > .5), None)
+ if job is None:
+ return
+ point = job['point']
+ if point is None:
+ # Unreachable clipping cannot be repaired by moving scrollTop.
+ budget.pause(.05)
+ continue
+ x = round((offset_x + point['x'] * scale_x) * metrics['dpr']) / metrics['dpr']
+ y = round((offset_y + point['y'] * scale_y) * metrics['dpr']) / metrics['dpr']
+ previous = _POINTERS.get(page.mouse)
+ if previous is None or math.hypot(x-previous[0],y-previous[1]) > 2:
+ _approach(page, x, y, budget, metrics)
+ amount = _RNG.uniform(160, 280)
+ dx = max(-amount,min(amount,job['dx'])) * scale_x
+ dy = max(-amount,min(amount,job['dy'])) * scale_y
+ budget.remaining()
+ page.mouse.wheel(dx,dy)
+ # wheel ACK precedes compositor scrolling. Read fresh geometry after
+ # Chrome has had a frame to process the actual input.
+ budget.pause(_RNG.uniform(.035,.065))
+
+
+def _wheel_into_view(locator: Any, budget: _Budget, position: Any = None) -> None:
+ handle = locator.element_handle(timeout=budget.remaining())
+ try:
+ # The channel owner is the exact frame resolved by Playwright's
+ # selector engine, including FrameLocator chains. owner_frame() would
+ # evaluate ownerDocument in the page's main world unnecessarily.
+ from playwright._impl._sync_base import mapping
+ frame = mapping.from_impl(handle._impl_obj._frame)
+ _wheel_target(locator.page,handle,frame,budget,position)
+ finally:
+ handle.dispose()
+
+
+def _enable_pointer_grid(page: Any, ratio: float) -> Any:
+ """Lease dispatch-time rounding for generated clicks on the local facade.
+
+ Native Chrome CDP has no such method; it keeps the existing input path.
+ This state belongs to the connection and tab, never to the document.
+ """
+ if getattr(page, '_browserwright_pointer_grid_supported', None) is False:
+ return None
+ from playwright.sync_api import Error
+ from .isolated_world import page_cdp_session
+ session = page_cdp_session(page)
+ try:
+ session.send('Browserwright.setPointerGrid', {'devicePixelRatio': ratio})
+ except Error as exc:
+ if any(message in str(exc).lower() for message in
+ ("wasn't found", 'method not found', 'unknown method', 'not implemented')):
+ page._browserwright_pointer_grid_supported = False
+ return None
+ raise
+ page._browserwright_pointer_grid_supported = True
+ return session
+
+
+def _click(native: Any, locator: Any, kwargs: dict[str, Any], *, generated_position: bool = False) -> Any:
+ page = locator.page
+ budget = _Budget(page, kwargs.get("timeout"))
+ if kwargs.get("trial"):
+ return native(locator, **kwargs)
+ # Final native click owns stability, enabled/receives-events checks,
+ # force semantics, modifiers, click_count and navigation.
+ _wheel_into_view(locator, budget, kwargs.get('position'))
+ box = locator.bounding_box(timeout=budget.remaining())
+ if box is None or box["width"] < 2 or box["height"] < 2:
+ return native(locator, **{**kwargs, "timeout": budget.remaining()})
+ position = kwargs.get("position")
+ from .isolated_world import isolated_evaluate
+ metrics = isolated_evaluate(
+ page, "() => ({width: innerWidth, height: innerHeight, dpr: devicePixelRatio || 1})")
+ if position is None:
+ viewport = page.viewport_size or metrics
+ left = max(0, -box["x"])
+ top = max(0, -box["y"])
+ right = min(box["width"], viewport["width"] - box["x"])
+ bottom = min(box["height"], viewport["height"] - box["y"])
+ if right - left < 2 or bottom - top < 2:
+ return native(locator, **{**kwargs, "timeout": budget.remaining()})
+ # A comfortable interior point, deliberately away from dead center.
+ fraction_x = _RNG.uniform(0.23, 0.40) if _RNG.random() < .5 else _RNG.uniform(.60, .77)
+ border_x = _border_width(locator, 'border-left-width', budget)
+ border_y = _border_width(locator, 'border-top-width', budget)
+ ratio = metrics['dpr']
+ # Native click truncates to hundredths of a CSS pixel. Choose points
+ # representable on both that grid and the physical pixel grid (e.g.
+ # quarter-scale screens need multiples of four CSS pixels).
+ scale = Fraction(str(ratio)).limit_denominator(100)
+ grid = scale.denominator / math.gcd(scale.numerator, 100)
+ x = round((box['x'] + left + (right - left) * fraction_x) / grid) * grid
+ y = round((box['y'] + top + (bottom - top) * _RNG.uniform(.28, .72)) / grid) * grid
+ if not (box['x'] + left < x < box['x'] + right
+ and box['y'] + top < y < box['y'] + bottom):
+ return native(locator, **{**kwargs, 'timeout': budget.remaining()})
+ # Playwright adds border widths to padding-relative positions and
+ # truncates the resulting coordinates to two decimals. The epsilon
+ # avoids floating-point cancellation moving an exact grid point down.
+ position = {'x': x - box['x'] - border_x + .00001,
+ 'y': y - box['y'] - border_y + .00001}
+ else:
+ x = math.trunc((box['x'] + _border_width(locator, 'border-left-width', budget)
+ + position['x']) * 100) / 100
+ y = math.trunc((box['y'] + _border_width(locator, 'border-top-width', budget)
+ + position['y']) * 100) / 100
+ _approach(page, x, y, budget, metrics)
+ # Native actionability can observe a newer target box than the approach.
+ # Correct its final absolute coordinates at the dispatch boundary instead
+ # of assuming the old padding-relative offset still lands on the grid.
+ grid_session = (_enable_pointer_grid(page, metrics['dpr'])
+ if kwargs.get('position') is None or generated_position else None)
+ point = None
+ options = {**kwargs, "position": position}
+ if options.get("delay") is None:
+ options["delay"] = _RNG.uniform(35, 95)
+ try:
+ result = native(locator, **{**options, 'timeout': budget.remaining()})
+ finally:
+ if grid_session is not None:
+ from playwright.sync_api import Error
+ try:
+ point = grid_session.send('Browserwright.setPointerGrid', {
+ 'devicePixelRatio': None}).get('point')
+ if point:
+ # Native input can succeed before navigation waiting
+ # times out. Retain its real position on that path too.
+ _POINTERS[page.mouse] = (point['x'], point['y'])
+ except Error:
+ # A disconnected/closed target also retires its facade lease.
+ # Cleanup must not replace the native action's exception.
+ pass
+ _BUTTONS.get(page.mouse, set()).discard(kwargs.get('button') or 'left')
+ _POINTERS[page.mouse] = (point['x'], point['y']) if point else (x, y)
+ return result
+
+
+def _select_all_with_keyboard(keyboard: Any, budget: _Budget) -> None:
+ """Select the focused editor through its platform's native shortcut."""
+ # Use Playwright's ControlOrMeta shortcut rather than guessing an OS key.
+ # Physical events let the browser perform selection itself, including
+ # ranges containing replaced contenteditable elements without any text.
+ keyboard.down('ControlOrMeta')
+ try:
+ budget.pause(_RNG.uniform(.025, .065))
+ keyboard.down('KeyA')
+ try:
+ budget.pause(_RNG.uniform(.045, .100))
+ finally:
+ keyboard.up('KeyA')
+ budget.pause(_RNG.uniform(.025, .055))
+ finally:
+ keyboard.up('ControlOrMeta')
+
+
+def _ensure_editor_caret(locator: Any, budget: _Budget, force: Any) -> bool:
+ """Keep select-all inside the focused editor after clicking rich media."""
+ handle = locator.element_handle(timeout=budget.remaining())
+ if handle is None:
+ return False
+ try:
+ plan = _scroll_plan(handle, budget)
+ if not plan['focused']:
+ return False
+ caret = plan.get('caret')
+ if caret is None:
+ return False
+ if caret['inside']:
+ return True
+ point = caret['point']
+ if point is not None:
+ # A replaced child can receive focus without updating the caret.
+ # Exposed editor space accepts an ordinary pointer caret gesture.
+ _click(type(locator).click.__wrapped__, locator,
+ {'position': point, 'timeout': budget.remaining(), 'force': force},
+ generated_position=True)
+ caret = _scroll_plan(handle, budget).get('caret')
+ if caret is None:
+ return False
+ if not caret['inside']:
+ # With no exposed blank space, native focus traversal initializes
+ # an editor caret. This can fire blur/focus just as physical Tab
+ # navigation does; only use it when the current caret is outside.
+ keyboard = locator.page.keyboard
+ keyboard.down('Tab')
+ try:
+ budget.pause(_RNG.uniform(.045, .100))
+ finally:
+ keyboard.up('Tab')
+ keyboard.down('Shift')
+ try:
+ budget.pause(_RNG.uniform(.025, .065))
+ keyboard.down('Tab')
+ try:
+ budget.pause(_RNG.uniform(.045, .100))
+ finally:
+ keyboard.up('Tab')
+ budget.pause(_RNG.uniform(.025, .055))
+ finally:
+ keyboard.up('Shift')
+ final = _scroll_plan(handle, budget)
+ return bool(final['focused'] and (final.get('caret') or {}).get('inside'))
+ finally:
+ handle.dispose()
+
+
+def _fill(native: Any, locator: Any, value: str, kwargs: dict[str, Any]) -> Any:
+ budget = _Budget(locator.page, kwargs.get("timeout"))
+ if not isinstance(value, str):
+ return native(locator, value, **kwargs)
+ # Native validation handles non-strings, select elements and specialized
+ # inputs (number/date/time/etc.). Textarea and contenteditable have no type.
+ kind = locator.get_attribute("type", timeout=budget.remaining())
+ if kind is not None:
+ kind = kind.lower()
+ if kind not in _TEXT_TYPES:
+ return native(locator, value, **{**kwargs, "timeout": budget.remaining()})
+ # Keep native fill's errors/actionability for invalid or noneditable
+ # targets, and its forced hidden-element behavior. Valid visible text
+ # controls receive a trusted pointer gesture before selecting existing text.
+ from playwright.sync_api import Error
+ try:
+ editable = locator.is_editable(timeout=budget.remaining())
+ except Error:
+ return native(locator, value, **{**kwargs, "timeout": budget.remaining()})
+ if (not editable or locator.locator("xpath=self::select").count()
+ or locator.bounding_box(timeout=budget.remaining()) is None):
+ return native(locator, value, **{**kwargs, "timeout": budget.remaining()})
+ locator.click(timeout=budget.remaining(), force=kwargs.get("force"))
+ if kwargs.get('force'):
+ handle = locator.element_handle(timeout=budget.remaining())
+ try:
+ focused = handle is not None and _scroll_plan(handle, budget)['focused']
+ finally:
+ if handle is not None:
+ handle.dispose()
+ if not focused:
+ # A forced pointer action can hit an overlay. Native fill retains
+ # its forced focus/replacement semantics for the requested editor.
+ return native(locator, value, **{**kwargs, 'timeout': budget.remaining()})
+ contenteditable = False
+ try:
+ current = locator.input_value(timeout=budget.remaining())
+ except Error:
+ contenteditable = True
+ current = locator.inner_text(timeout=budget.remaining())
+ if not current:
+ # Replaced elements can be selected/removed even without text.
+ current = bool(locator.locator(
+ 'img,video,audio,iframe,object,embed,input,textarea,select,hr').count())
+ if contenteditable and not _ensure_editor_caret(locator, budget, kwargs.get('force')):
+ # Custom focus/keyboard handlers may prevent caret placement. Keep
+ # native contenteditable replacement rather than select the page.
+ return native(locator, value, **{**kwargs, 'timeout': budget.remaining()})
+ if current:
+ keyboard = locator.page.keyboard
+ _select_all_with_keyboard(keyboard, budget)
+ keyboard.down('Backspace')
+ try:
+ budget.pause(_RNG.uniform(.045, .100))
+ finally:
+ keyboard.up('Backspace')
+ for character in value:
+ budget.remaining()
+ remaining = budget.remaining()
+ delay = _RNG.uniform(35, 105)
+ delay = min(delay, remaining) if remaining else delay
+ # Use physical Shift chords for capitals and shifted punctuation;
+ # Keyboard.type alone accepts an uppercase key without holding Shift.
+ shifted = (f"Key{character}" if 'A' <= character <= 'Z'
+ else _SHIFT_KEYS.get(character))
+ if shifted:
+ keyboard = locator.page.keyboard
+ keyboard.down('Shift')
+ try:
+ budget.pause(_RNG.uniform(.025, .065))
+ keyboard.down(shifted)
+ try:
+ budget.pause(delay / 1000)
+ finally:
+ keyboard.up(shifted)
+ budget.pause(_RNG.uniform(.025, .055))
+ finally:
+ keyboard.up('Shift')
+ else:
+ # Retain Playwright's insertion path for non-keyboard Unicode.
+ locator.page.keyboard.type(character, delay=delay)
+ budget.remaining()
+ return None
+
+
+def install_human_input(context: Any) -> None:
+ """Register the executor context and patch its sync API input methods."""
+ global _INSTALLED
+ from playwright.sync_api import BrowserContext, Frame, Locator, Mouse, Page
+ if not isinstance(context, BrowserContext):
+ return
+ if context not in _SCROLL_CONTEXTS:
+ context._sync(context._impl_obj._channel.send('registerSelectorEngine', None, {
+ 'selectorEngine': {'name': _SCROLL_ENGINE, 'source': _SCROLL_SOURCE,
+ 'contentScript': True},
+ }))
+ _SCROLL_CONTEXTS.add(context)
+ _CONTEXTS.add(context)
+ if _INSTALLED:
+ return
+ _INSTALLED = True
+
+ for name, operation in (("click", _click), ("fill", _fill)):
+ native = getattr(Locator, name)
+
+ def wrap(native: Any, operation: Any) -> Any:
+ @functools.wraps(native)
+ def input_method(self: Any, *args: Any, human: bool | None = None, **kwargs: Any) -> Any:
+ if not _enabled(self.page, human):
+ return native(self, *args, **kwargs)
+ return operation(native, self, *args, kwargs)
+ return input_method
+
+ setattr(Locator, name, wrap(native, operation))
+
+ # Selector convenience methods must use the same defaults as chained
+ # locators, including within child frames. Page/Frame default to first match;
+ # Locator's native strictness is retained.
+ for cls in (Page, Frame):
+ for name in ("click", "fill"):
+ native = getattr(cls, name)
+
+ def wrap_selector(native: Any, name: str, cls: Any) -> Any:
+ @functools.wraps(native)
+ def input_method(self: Any, selector: str, *args: Any,
+ human: bool | None = None, **kwargs: Any) -> Any:
+ page = self if cls is Page else self.page
+ if not _enabled(page, human):
+ return native(self, selector, *args, **kwargs)
+ locator = self.locator(selector)
+ if not kwargs.pop("strict", False):
+ locator = locator.first
+ return getattr(locator, name)(*args, human=human, **kwargs)
+ return input_method
+
+ setattr(cls, name, wrap_selector(native, name, cls))
+
+ # Track explicit agent mouse calls too; pointer state stays in Python.
+ for name in ("move", "click", "dblclick"):
+ native = getattr(Mouse, name)
+
+ def wrap_mouse(native: Any, name: str) -> Any:
+ @functools.wraps(native)
+ def move(self: Any, x: float, y: float, **kwargs: Any) -> Any:
+ result = native(self, x, y, **kwargs)
+ _POINTERS[self] = (x, y)
+ if name != 'move':
+ _BUTTONS.get(self, set()).discard(kwargs.get('button') or 'left')
+ return result
+ return move
+
+ setattr(Mouse, name, wrap_mouse(native, name))
+
+ for name in ('down', 'up'):
+ native = getattr(Mouse, name)
+
+ def wrap_button(native: Any, name: str) -> Any:
+ @functools.wraps(native)
+ def button(self: Any, **kwargs: Any) -> Any:
+ result = native(self, **kwargs)
+ held = _BUTTONS.setdefault(self, set())
+ pressed = kwargs.get('button') or 'left'
+ if name == 'down':
+ held.add(pressed)
+ else:
+ held.discard(pressed)
+ return result
+ return button
+
+ setattr(Mouse, name, wrap_button(native, name))
diff --git a/tests/daemon/e2e/test_human_input.py b/tests/daemon/e2e/test_human_input.py
new file mode 100644
index 0000000..d536926
--- /dev/null
+++ b/tests/daemon/e2e/test_human_input.py
@@ -0,0 +1,419 @@
+"""Real executor input behavior, observed by an ordinary page's event listeners.
+
+Repeat with ``uv run pytest tests/daemon/e2e/test_human_input.py -m real_chrome -v``.
+The JSON artifact includes the browser's trusted keyboard and pointer events,
+native opt-out, actionability failures, and input-type compatibility results.
+"""
+from __future__ import annotations
+
+import json
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+import textwrap
+import threading
+import uuid
+
+import pytest
+
+from .helpers import bs_home, drop_ledger_session, run_skill, seed_ledger_session
+
+
+@pytest.fixture
+def e2e_chrome(extension_only_chrome):
+ return extension_only_chrome
+
+
+_HTML = """
+Input behavior fixture
+
+
+
+"""
+
+
+@pytest.fixture
+def human_input_site():
+ # getCoalescedEvents is restricted to secure contexts. A loopback origin
+ # is trustworthy; about:blank set_content cannot exercise this API.
+ class Handler(BaseHTTPRequestHandler):
+ def do_GET(self):
+ body = _HTML.encode()
+ self.send_response(200)
+ self.send_header('Content-Type', 'text/html; charset=utf-8')
+ self.send_header('Content-Length', str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *args):
+ pass
+
+ server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
+ worker = threading.Thread(target=server.serve_forever, daemon=True)
+ worker.start()
+ try:
+ yield f'http://127.0.0.1:{server.server_port}'
+ finally:
+ server.shutdown()
+ server.server_close()
+ worker.join(timeout=2)
+
+
+def _assert_typed(record: dict, expected: str) -> None:
+ """Verify actual event order and final replacement, not wrapper internals."""
+ assert record["value"] == expected
+ assert record["return_none"] is True
+ events = record["events"]
+ # Clearing a preexisting value can emit Delete/Backspace before typing.
+ starts = [i for i, event in enumerate(events)
+ if event["type"] == "keydown" and len(event["key"] or "") == 1
+ and not (event.get('ctrl') or event.get('meta'))]
+ assert len(starts) == len(expected), events
+ for index, character in zip(starts, expected):
+ char_events = events[index:index + 4]
+ assert [event["type"] for event in char_events] == [
+ "keydown", "keypress", "input", "keyup",
+ ], char_events
+ assert [event["key"] for event in char_events if event["key"]] == [
+ character, character, character,
+ ], char_events
+ assert char_events[2]["data"] == character
+ assert all(event["trusted"] for event in char_events)
+ if "shift" in char_events[0] and character.isascii() and character.isupper():
+ assert char_events[0]["shift"] is True, char_events
+ assert any(event["type"] == "keydown" and event["key"] == "Shift"
+ for event in events[:index]), events
+
+
+@pytest.mark.parametrize("extension_only_chrome", ["headless"], indirect=True)
+def test_executor_human_input_defaults_and_playwright_semantics(
+ e2e_daemon, e2e_chrome, ext_ready, e2e_artifacts_dir, human_input_site,
+):
+ """Agent-created locators/pages/frames use normal browser input events."""
+ pytest.importorskip("playwright.sync_api")
+ assert not (e2e_chrome.profile_path / 'DevToolsActivePort').exists()
+ sid = f"e2e-input-{uuid.uuid4().hex}"
+ seed_ledger_session(bs_home("extension"), sid, backend="extension", name="input")
+ artifact = e2e_artifacts_dir / "human-input.json"
+ artifact.unlink(missing_ok=True)
+ script = "import json\n" + f"page.goto({human_input_site!r}, wait_until='load')\n"
+ script += "from pathlib import Path\n" + f"artifact = Path({str(artifact)!r})\n"
+ script += textwrap.dedent("""
+ from playwright.sync_api import Error, TimeoutError
+ result = {}
+ result['browser_version'] = {'reported': context.browser.version,
+ 'actual_user_agent': page.evaluate('navigator.userAgent')}
+ def take():
+ return page.evaluate('events.splice(0)')
+ field = page.get_by_role('region', name='Form').get_by_role('textbox', name='Name')
+ returned = field.fill('Alice')
+ result['locator_fill'] = {'value': field.input_value(), 'return_none': returned is None,
+ 'events': take()}
+ returned = field.fill('A!b?')
+ result['shift_fill'] = {'value': field.input_value(), 'return_none': returned is None,
+ 'events': take()}
+ empty = page.get_by_role('textbox', name='Empty', exact=True)
+ returned = empty.fill('Eve')
+ result['initially_empty_fill'] = {'value': empty.input_value(),
+ 'return_none': returned is None, 'events': take()}
+ notes = page.get_by_role('textbox', name='Notes')
+ returned = notes.fill('New notes', force=True)
+ result['textarea_fill'] = {'value': notes.input_value(),
+ 'return_none': returned is None, 'events': take()}
+
+ returned = page.get_by_role('button', name='Submit').click()
+ result['click'] = {'return_none': returned is None, 'events': take()}
+ metrics_session = context.new_cdp_session(page)
+ metrics_session.send('Emulation.setDeviceMetricsOverride', {
+ 'width': 1280, 'height': 900, 'deviceScaleFactor': 1.75, 'mobile': False})
+ page.get_by_role('button', name='Submit').click()
+ result['scaled_click'] = {'events': take()}
+ metrics_session.send('Emulation.clearDeviceMetricsOverride')
+ metrics_session.detach()
+ returned = page.click('#submit', position={'x': 9, 'y': 11}, force=True)
+ result['position_click'] = {'return_none': returned is None, 'events': take()}
+
+ returned = field.fill('native', human=False)
+ result['native_fill'] = {'value': field.input_value(), 'return_none': returned is None,
+ 'events': take()}
+ page.click('#submit', human=False)
+ result['native_click'] = {'events': take()}
+
+ rich = page.get_by_role('textbox', name='Rich text')
+ returned = rich.fill('Rich')
+ result['rich_fill'] = {'value': rich.inner_text(), 'return_none': returned is None,
+ 'events': take()}
+ empty_rich = page.get_by_role('textbox', name='Empty editor')
+ returned = empty_rich.fill('Empty rich')
+ result['empty_editor_fill'] = {'value': empty_rich.inner_text(),
+ 'return_none': returned is None, 'events': take()}
+ media = page.get_by_role('textbox', name='Rich media')
+ returned = media.fill('Media')
+ result['media_fill'] = {'value': media.inner_text(), 'return_none': returned is None,
+ 'children': media.locator('img').count(), 'events': take()}
+ returned = page.fill('#name', 'Page')
+ result['page_fill'] = {'value': field.input_value(), 'return_none': returned is None,
+ 'events': take()}
+ returned = field.fill('')
+ result['empty_fill'] = {'value': field.input_value(), 'return_none': returned is None,
+ 'events': take()}
+ # A forced click can land on an overlay instead of the intended input.
+ # Replacement still belongs to the requested field, and must not select
+ # or erase another editor that was focused before the call.
+ empty.click(human=False)
+ page.evaluate('''() => {
+ const r=document.querySelector('#name').getBoundingClientRect();
+ const overlay=document.createElement('div');overlay.id='overlay';
+ Object.assign(overlay.style,{position:'fixed',left:r.left+'px',top:r.top+'px',
+ width:r.width+'px',height:r.height+'px',zIndex:'1000'});
+ document.body.append(overlay);
+ }''')
+ take()
+ returned = field.fill('Forced', force=True)
+ result['forced_overlay_fill'] = {'value': field.input_value(),
+ 'other_value': empty.input_value(),
+ 'return_none': returned is None, 'events': take()}
+ page.evaluate("document.querySelector('#overlay').remove()")
+
+ for selector in ['#readonly', '#disabled']:
+ try:
+ page.locator(selector).fill('x', timeout=180)
+ result[selector] = {'error': None}
+ except TimeoutError as exc:
+ result[selector] = {'error': type(exc).__name__, 'message': str(exc)}
+ try:
+ page.locator('#select').fill('bad', timeout=180)
+ result['select'] = {'error': None}
+ except Error as exc:
+ result['select'] = {'error': type(exc).__name__, 'message': str(exc)}
+ for selector, value in [('#number', '42'), ('#date', '2026-10-07')]:
+ returned = page.locator(selector).fill(value)
+ result[selector] = {'value': page.locator(selector).input_value(),
+ 'return_none': returned is None, 'events': take()}
+ try:
+ page.locator('#number').fill('not a number')
+ result['invalid_number'] = {'error': None}
+ except Error as exc:
+ result['invalid_number'] = {'error': type(exc).__name__, 'message': str(exc)}
+ # Keep observations available if later page/frame setup fails.
+ artifact.write_text(json.dumps(result, indent=2) + '\\n')
+
+ # A held button enters Playwright's native drag handling. Preserve its
+ # state throughout the approach and let native click release normally.
+ page.mouse.move(20, 20)
+ page.mouse.down()
+ take()
+ returned = page.get_by_role('button', name='Submit').click(force=True)
+ result['held_button_click'] = {'return_none': returned is None, 'events': take()}
+ page.mouse.up()
+ take()
+
+ other = context.new_page()
+ other.set_content('Other ')
+ other.evaluate('''() => {
+ window.events = [];
+ for (const type of ['keydown', 'keypress', 'input', 'keyup'])
+ document.addEventListener(type, e => events.push({type, key:e.key || null,
+ data:e.data ?? null, trusted:e.isTrusted,
+ ctrl:e.ctrlKey, meta:e.metaKey}));
+ }''')
+ returned = other.get_by_role('textbox', name='Other').fill('New')
+ result['new_page_fill'] = {'value': other.locator('#other').input_value(),
+ 'return_none': returned is None,
+ 'events': other.evaluate('events')}
+ frame = other.main_frame
+ frame.evaluate('events.splice(0)')
+ returned = frame.fill('#other', 'Frame')
+ result['frame_fill'] = {'value': frame.locator('#other').input_value(),
+ 'return_none': returned is None,
+ 'events': frame.evaluate('events')}
+ other.close()
+ artifact.write_text(json.dumps(result, indent=2) + '\\n')
+ print('HUMAN_INPUT=' + ','.join(result))
+ """)
+ try:
+ execution = run_skill(script, backend="extension", runtime_dir=e2e_daemon.runtime_dir,
+ extra_env={"BD_SESSION": sid}, timeout=90)
+ output = {"returncode": execution.returncode, "stdout": execution.stdout,
+ "stderr": execution.stderr}
+ if artifact.exists():
+ output["observations"] = json.loads(artifact.read_text())
+ artifact.write_text(json.dumps(output, indent=2) + "\n")
+ assert execution.returncode == 0, output
+ observations = output["observations"]
+ assert 'Chrome/' + observations['browser_version']['reported'] in \
+ observations['browser_version']['actual_user_agent']
+ _assert_typed(observations['initially_empty_fill'], 'Eve')
+ assert not any(e['key'] in ('Delete', 'Backspace')
+ for e in observations['initially_empty_fill']['events'])
+ assert not any(e.get('ctrl') or e.get('meta')
+ for e in observations['initially_empty_fill']['events'])
+ _assert_typed(observations['empty_editor_fill'], 'Empty rich')
+ assert not any(e['key'] in ('Delete', 'Backspace') or e.get('ctrl') or e.get('meta')
+ for e in observations['empty_editor_fill']['events'])
+ # Replacements must select through an actual trusted keyboard shortcut,
+ # with a full selection visible before deletion. This also covers a
+ # multiline textarea and contenteditable containing only an image.
+ for name, old in [('locator_fill', 'replace me'), ('shift_fill', 'Alice'),
+ ('textarea_fill', 'old multiline\nnotes'),
+ ('rich_fill', 'old rich text'), ('media_fill', None),
+ ('page_fill', 'native'), ('empty_fill', 'Page')]:
+ events = observations[name]['events']
+ shortcut = next(e for e in events if e['type'] == 'keyup'
+ and (e['key'] or '').lower() == 'a'
+ and (e['ctrl'] or e['meta']))
+ assert shortcut['trusted'], shortcut
+ assert any(e['type'] == 'keydown' and e['key'] in ('Control', 'Meta')
+ and e['trusted'] for e in events[:events.index(shortcut)]), events
+ selection = shortcut['selection']
+ if 'start' in selection:
+ assert selection == {'start': 0, 'end': len(old)}, shortcut
+ else:
+ assert selection['collapsed'] is False, shortcut
+ if old is not None:
+ assert selection['text'] == old, shortcut
+ assert any(e['type'] == 'keyup' and e['key'] in ('Control', 'Meta')
+ for e in events[events.index(shortcut) + 1:]), events
+ for name in ('locator_fill', 'shift_fill', 'initially_empty_fill'):
+ pressed = {}
+ shift_down = None
+ for event in observations[name]['events']:
+ if event['type'] == 'keydown':
+ pressed[event['key']] = event['time']
+ if event['key'] == 'Shift':
+ shift_down = event['time']
+ elif event['shift'] and shift_down is not None:
+ assert event['time'] - shift_down >= 15, event
+ elif event['type'] == 'keyup' and event['key'] in pressed:
+ assert event['time'] - pressed.pop(event['key']) >= 20, event
+ first_moves = [e for e in observations['locator_fill']['events']
+ if e['type'] == 'pointermove']
+ assert first_moves and (first_moves[0]['x'] > 5 or first_moves[0]['y'] > 5), first_moves
+ for name in ('locator_fill', 'shift_fill', 'initially_empty_fill', 'click', 'scaled_click'):
+ for event in observations[name]['events']:
+ if event['type'] == 'pointermove':
+ assert abs(event['x'] * event['dpr'] - round(event['x'] * event['dpr'])) < .01, event
+ assert abs(event['y'] * event['dpr'] - round(event['y'] * event['dpr'])) < .01, event
+ for sample in event['coalesced'] or []:
+ assert abs(sample['x'] * event['dpr'] - round(sample['x'] * event['dpr'])) < .01, event
+ assert abs(sample['y'] * event['dpr'] - round(sample['y'] * event['dpr'])) < .01, event
+ focus = next(event for event in observations["locator_fill"]["events"]
+ if event["type"] == "focus" and event["target"] == "name")
+ assert focus["activated"] is True, focus
+ assert any(event["type"] == "mousedown" and event["target"] == "name"
+ for event in observations["locator_fill"]["events"]), observations["locator_fill"]
+ for key, value in [("locator_fill", "Alice"), ("rich_fill", "Rich"),
+ ("textarea_fill", "New notes"),
+ ("page_fill", "Page"), ("new_page_fill", "New"),
+ ("frame_fill", "Frame")]:
+ _assert_typed(observations[key], value)
+ _assert_typed(observations["shift_fill"], "A!b?")
+ assert all(event["shift"] for event in observations["shift_fill"]["events"]
+ if event["type"] == "keydown" and event["key"] in ("A", "!", "?"))
+ _assert_typed(observations['media_fill'], 'Media')
+ assert observations['media_fill']['children'] == 0
+ assert observations["empty_fill"]["value"] == ""
+ assert observations["empty_fill"]["return_none"] is True
+ forced = observations['forced_overlay_fill']
+ assert forced['value'] == 'Forced' and forced['other_value'] == 'Eve', forced
+ assert forced['return_none'] is True
+ assert not any(e.get('ctrl') or e.get('meta') for e in forced['events']), forced
+ assert observations["native_fill"]["value"] == "native"
+ assert observations["native_fill"]["return_none"] is True
+ assert not any(event["type"] in ("keydown", "keypress", "keyup")
+ for event in observations["native_fill"]["events"])
+ assert observations["click"]["return_none"] is True
+ held = observations['held_button_click']
+ assert held['return_none'] is True
+ assert any(e['type'] == 'mousemove' and e['buttons'] == 1 for e in held['events']), held
+ assert any(e['type'] == 'click' and e['target'] == 'submit' for e in held['events']), held
+ assert any(e['type'] == 'mouseup' and e['buttons'] == 0 for e in held['events']), held
+ pointer = observations["click"]["events"]
+ coalesced = [event for event in pointer if event['type'] == 'pointermove'
+ and len(event['coalesced'] or []) > 1]
+ assert coalesced, pointer
+ for event in coalesced:
+ assert all(sample['trusted'] for sample in event['coalesced']), event
+ assert all(a['time'] <= b['time'] for a, b in
+ zip(event['coalesced'], event['coalesced'][1:])), event
+ assert sum(event["type"] == "mousemove" for event in pointer) >= 3, pointer
+ visible_moves = [event for event in pointer if event['type'] == 'mousemove']
+ assert len(visible_moves) >= 15, visible_moves
+ assert all(((b['x'] - a['x']) ** 2 + (b['y'] - a['y']) ** 2) ** .5 <= 15
+ for a, b in zip(visible_moves, visible_moves[1:])), visible_moves
+ # Chrome may deliver several physical samples in one pointermove.
+ # Check path density using those original samples, rather than requiring
+ # the browser to dispatch every sample as a separate DOM event.
+ moves = [sample for event in pointer if event['type'] == 'pointermove'
+ for sample in (event['coalesced'] or [event])]
+ assert len(moves) >= 15, moves
+ assert all(((b["x"] - a["x"]) ** 2 + (b["y"] - a["y"]) ** 2) ** .5 <= 15
+ for a, b in zip(moves, moves[1:])), moves
+ click = next(event for event in pointer if event["type"] == "click")
+ rect = click["rect"]
+ assert rect["x"] < click["x"] < rect["x"] + rect["width"]
+ assert rect["y"] < click["y"] < rect["y"] + rect["height"]
+ assert (abs(click["x"] - rect["x"] - rect["width"] / 2) > 0.5 or
+ abs(click["y"] - rect["y"] - rect["height"] / 2) > 0.5), click
+ down = next(event for event in pointer if event["type"] == "mousedown")
+ up = next(event for event in pointer if event["type"] == "mouseup")
+ assert up["time"] - down["time"] >= 5, pointer
+ positioned = next(event for event in observations["position_click"]["events"]
+ if event["type"] == "click")
+ assert abs(positioned["x"] - positioned["rect"]["x"] -
+ positioned["rect"]["borderX"] - 9) <= 1
+ assert abs(positioned["y"] - positioned["rect"]["y"] -
+ positioned["rect"]["borderY"] - 11) <= 1
+ native = next(event for event in observations["native_click"]["events"]
+ if event["type"] == "click")
+ assert abs(native["x"] - native["rect"]["x"] - native["rect"]["width"] / 2) <= 1
+ assert abs(native["y"] - native["rect"]["y"] - native["rect"]["height"] / 2) <= 1
+ for selector in ("#readonly", "#disabled"):
+ assert observations[selector]["error"] == "TimeoutError", observations[selector]
+ assert observations["select"]["error"] == "Error"
+ assert observations["invalid_number"]["error"] == "Error"
+ assert observations["#number"]["value"] == "42"
+ assert observations["#date"]["value"] == "2026-10-07"
+ finally:
+ drop_ledger_session(bs_home("extension"), sid)
diff --git a/tests/daemon/e2e/test_human_input_no_focus.py b/tests/daemon/e2e/test_human_input_no_focus.py
new file mode 100644
index 0000000..a9c4ac7
--- /dev/null
+++ b/tests/daemon/e2e/test_human_input_no_focus.py
@@ -0,0 +1,252 @@
+"""Human input on a background tab must never activate a tab or focus a window.
+
+The extension backend drives the user's own Chrome. Activating the session's
+tab or raising its window steals the desktop's active window from the user.
+This test puts the user somewhere else first — a newly focused window, plus a
+newer active tab in the session tab's own window — then runs executor
+click + fill + scroll-into-view on the (now background) session tab.
+
+It asserts (a) the page saw the actions as trusted input and they took effect,
+and (b) Chrome's own tab/window state is unchanged: the same active tab in
+every window, the same focused window, and no `tabs.onActivated` /
+`windows.onFocusChanged` event for the session tab or its window at any point
+during the actions (so a transient activate-then-restore is caught too).
+Chrome state is read from the extension's service worker over the harness's
+CDP port, independently of the daemon.
+
+Repeat with ``bash tests/daemon/e2e/run.sh
+tests/daemon/e2e/test_human_input_no_focus.py -v``. The artifact is
+``tests/daemon/e2e/_artifacts/human-input-no-focus.json``. Headful by default
+(the meaningful case); ``BW_E2E_HEADLESS=1`` still exercises the contract.
+"""
+from __future__ import annotations
+
+import json
+import textwrap
+import threading
+import time
+import uuid
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+import pytest
+
+from browserwright.cdp import CDPSession
+
+from .helpers import bs_home, drop_ledger_session, run_skill, seed_ledger_session
+from .test_l2_multisession import _extension_id_from_path, _extension_worker_target_id
+
+_SESSION_HTML = """Session page
+
+Name
+Near
+
+Far
+"""
+
+_USER_HTML = """User page
+The user is reading this tab.
"""
+
+_WATCH_JS = """(() => {
+ globalThis.__bwFocusLog = [];
+ const log = (kind) => (...args) => __bwFocusLog.push({kind, args, t: Date.now()});
+ globalThis.__bwFocusListeners = [
+ [chrome.tabs.onActivated, log('tabs.onActivated')],
+ [chrome.tabs.onHighlighted, log('tabs.onHighlighted')],
+ [chrome.windows.onFocusChanged, log('windows.onFocusChanged')],
+ ];
+ for (const [event, fn] of __bwFocusListeners) event.addListener(fn);
+ return true;
+})()"""
+
+_STATE_JS = """(async () => {
+ const windows = await chrome.windows.getAll({populate: true});
+ let lastFocused = null;
+ try { lastFocused = (await chrome.windows.getLastFocused()).id; } catch (e) {}
+ return {
+ lastFocusedWindow: lastFocused,
+ windows: windows.map(w => ({id: w.id, focused: w.focused,
+ activeTab: (w.tabs.find(t => t.active) || {}).id ?? null,
+ tabs: w.tabs.map(t => ({id: t.id, active: t.active, url: t.url}))})),
+ };
+})()"""
+
+
+class _Worker:
+ """Evaluate in the extension's service worker over the browser CDP port."""
+
+ def __init__(self, chrome, extension_id: str):
+ self._cdp = CDPSession(chrome.ws_url)
+ self._sid = self._cdp.attach(_extension_worker_target_id(self._cdp, extension_id))
+
+ def eval(self, expression: str):
+ result = self._cdp.send("Runtime.evaluate", session=self._sid,
+ expression=expression, returnByValue=True,
+ awaitPromise=True)
+ if "exceptionDetails" in result:
+ raise AssertionError(f"service-worker evaluation failed: {result!r}")
+ return result.get("result", {}).get("value")
+
+ def close(self) -> None:
+ self._cdp.close()
+
+
+@pytest.fixture
+def pages_site():
+ class Handler(BaseHTTPRequestHandler):
+ def do_GET(self):
+ body = (_USER_HTML if self.path.startswith("/user") else _SESSION_HTML).encode()
+ self.send_response(200)
+ self.send_header("Content-Type", "text/html; charset=utf-8")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *args):
+ pass
+
+ server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+ worker = threading.Thread(target=server.serve_forever, daemon=True)
+ worker.start()
+ try:
+ yield f"http://127.0.0.1:{server.server_port}"
+ finally:
+ server.shutdown()
+ server.server_close()
+ worker.join(timeout=2)
+
+
+def _wait_loaded(worker: _Worker, tab_id: int, timeout: float = 15.0) -> None:
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ if worker.eval(f"chrome.tabs.get({tab_id}).then(t => t.status)") == "complete":
+ return
+ time.sleep(0.1)
+ raise AssertionError(f"tab {tab_id} did not finish loading")
+
+
+def test_background_tab_human_input_never_steals_focus(
+ e2e_daemon, e2e_chrome, ext_ready, patched_ext_dir, e2e_artifacts_dir, pages_site,
+):
+ pytest.importorskip("playwright.sync_api")
+ artifact = e2e_artifacts_dir / "human-input-no-focus.json"
+ artifact.unlink(missing_ok=True)
+ token = uuid.uuid4().hex
+ session_url = f"{pages_site}/session?{token}"
+ sid = f"e2e-nofocus-{token}"
+ seed_ledger_session(bs_home("extension"), sid, backend="extension", name="nofocus")
+ env = {"BD_SESSION": sid}
+ worker = _Worker(e2e_chrome, _extension_id_from_path(patched_ext_dir))
+ record: dict = {"session_url": session_url}
+ try:
+ # 1. Bind the session tab to the fixture page.
+ setup = run_skill(f"page.goto({session_url!r}, wait_until='load')\n",
+ backend="extension", runtime_dir=e2e_daemon.runtime_dir,
+ extra_env=env, timeout=60)
+ assert setup.returncode == 0, (setup.stdout, setup.stderr)
+ tabs = worker.eval(f"chrome.tabs.query({{}}).then(ts => ts.filter("
+ f"t => (t.url || '').includes({token!r})).map(t => t.id))")
+ assert len(tabs) == 1, tabs
+ session_tab = tabs[0]
+ session_window = worker.eval(f"chrome.tabs.get({session_tab}).then(t => t.windowId)")
+
+ # 2. The user moves on: a newer active tab in the session tab's own
+ # window, then a separate window that takes focus.
+ user_tab = worker.eval(
+ f"chrome.tabs.create({{windowId: {session_window}, "
+ f"url: {pages_site + '/user-tab'!r}, active: true}}).then(t => t.id)")
+ user_window = worker.eval(
+ f"chrome.windows.create({{url: {pages_site + '/user-window'!r}, "
+ f"focused: true}}).then(w => w.id)")
+ user_window_tab = worker.eval(
+ f"chrome.tabs.query({{windowId: {user_window}}}).then(ts => ts[0].id)")
+ for tab in (user_tab, user_window_tab):
+ _wait_loaded(worker, tab)
+ time.sleep(0.5) # let window-manager focus settle before the baseline
+ before = worker.eval(_STATE_JS)
+ record["before"] = before
+ session_win_state = next(w for w in before["windows"] if w["id"] == session_window)
+ assert session_win_state["activeTab"] == user_tab, before
+ assert not any(t["active"] for t in session_win_state["tabs"]
+ if t["id"] == session_tab), before
+ assert worker.eval(_WATCH_JS) is True
+
+ # 3. Human input on the background session tab.
+ script = textwrap.dedent("""
+ import json, time
+ out = {'visibility': page.evaluate('document.visibilityState')}
+ t0 = time.monotonic()
+ page.get_by_role('button', name='Near').click()
+ out['click_s'] = time.monotonic() - t0
+ t0 = time.monotonic()
+ page.get_by_role('textbox', name='Name').fill('New Name!')
+ out['fill_s'] = time.monotonic() - t0
+ t0 = time.monotonic()
+ page.get_by_role('button', name='Far').click()
+ out['scroll_click_s'] = time.monotonic() - t0
+ out['value'] = page.locator('#name').input_value()
+ out['clicks'] = page.evaluate('clicks')
+ out['scrollY'] = page.evaluate('scrollY')
+ out['events'] = page.evaluate('events')
+ from pathlib import Path
+ Path(RESULT_PATH).write_text(json.dumps(out))
+ """)
+ result_path = e2e_artifacts_dir / f"human-input-no-focus-{token}.tmp.json"
+ script = f"RESULT_PATH = {str(result_path)!r}\n" + script
+ run = run_skill(script, backend="extension", runtime_dir=e2e_daemon.runtime_dir,
+ extra_env=env, timeout=90)
+ record["run"] = {"returncode": run.returncode, "stdout": run.stdout,
+ "stderr": run.stderr}
+ focus_log = worker.eval("globalThis.__bwFocusLog ?? null")
+ after = worker.eval(_STATE_JS)
+ record.update(focus_log=focus_log, after=after)
+ assert run.returncode == 0, record["run"]
+ result = json.loads(result_path.read_text())
+ result_path.unlink()
+ record["result"] = result
+
+ # (a) The actions happened, through trusted browser input.
+ assert result["clicks"] == ["near", "far"], result["clicks"]
+ assert result["value"] == "New Name!", result["value"]
+ assert result["scrollY"] > 0, result["scrollY"]
+ events = result["events"]
+ for kind in ("pointermove", "mousedown", "click", "keydown", "input", "wheel"):
+ seen = [e for e in events if e["type"] == kind]
+ assert seen and all(e["trusted"] for e in seen), (kind, seen)
+ typed = "".join(e["key"] for e in events
+ if e["type"] == "keydown" and len(e["key"] or "") == 1)
+ assert typed.endswith("New Name!"), typed
+
+ # (b) Chrome's focus state is exactly what the user left it as.
+ assert focus_log is not None, "service worker restarted; focus log lost"
+
+ def activations(state):
+ return {w["id"]: w["activeTab"] for w in state["windows"]}
+
+ assert activations(after) == activations(before), (before, after)
+ assert after["lastFocusedWindow"] == before["lastFocusedWindow"], (before, after)
+ assert ([w["focused"] for w in after["windows"]]
+ == [w["focused"] for w in before["windows"]]), (before, after)
+ assert not any(e["kind"] == "tabs.onActivated"
+ and e["args"][0].get("tabId") == session_tab
+ for e in focus_log), focus_log
+ assert not any(e["kind"] == "windows.onFocusChanged"
+ and e["args"][0] == session_window
+ for e in focus_log), focus_log
+ assert not any(e["kind"] == "tabs.onHighlighted"
+ and session_tab in (e["args"][0].get("tabIds") or [])
+ for e in focus_log), focus_log
+ finally:
+ artifact.write_text(json.dumps(record, indent=2) + "\n")
+ try:
+ worker.eval("""(() => { for (const [e, f] of globalThis.__bwFocusListeners || [])
+ e.removeListener(f); return true; })()""")
+ finally:
+ worker.close()
+ drop_ledger_session(bs_home("extension"), sid)
diff --git a/tests/daemon/e2e/test_human_pointer_layout.py b/tests/daemon/e2e/test_human_pointer_layout.py
new file mode 100644
index 0000000..a7ff368
--- /dev/null
+++ b/tests/daemon/e2e/test_human_pointer_layout.py
@@ -0,0 +1,114 @@
+"""Native pointer coordinates after target geometry changes during an approach.
+
+Repeat with ``mise exec -- uv run pytest tests/daemon/e2e/test_human_pointer_layout.py -q``.
+The artifact records trusted browser events and the final target geometry.
+"""
+from __future__ import annotations
+
+import json
+import textwrap
+
+import pytest
+
+from .helpers import run_skill
+
+
+@pytest.fixture
+def e2e_chrome(extension_only_chrome):
+ return extension_only_chrome
+
+
+@pytest.mark.parametrize('extension_only_chrome', ['headless'], indirect=True)
+def test_click_after_layout_shift_stays_on_physical_pixel_grid(
+ e2e_daemon, e2e_chrome, ext_ready, e2e_artifacts_dir,
+):
+ artifact = e2e_artifacts_dir / 'human-pointer-layout.json'
+ artifact.unlink(missing_ok=True)
+ script = 'import json\nfrom pathlib import Path\n'
+ script += f'artifact=Path({str(artifact)!r})\n'
+ script += textwrap.dedent('''
+ result={}
+ metrics=context.new_cdp_session(page)
+ for ratio in [2,1.75]:
+ page.goto('about:blank')
+ metrics.send('Emulation.setDeviceMetricsOverride', {
+ 'width':1000,'height':700,'deviceScaleFactor':ratio,'mobile':False})
+ page.set_content("""Moving
+ """)
+ returned=page.get_by_role('button',name='Moving').click()
+ result[str(ratio)]={'return_none':returned is None,
+ 'observation':page.evaluate("""() => ({clicked,shifted,events,
+ box:document.querySelector('button').getBoundingClientRect().toJSON()})""")}
+ # The framework lease must end with the generated click. Explicit
+ # padding-relative positions and human=False retain native CDP
+ # coordinates, even when those are between physical pixels.
+ result[str(ratio)]['explicit']=[]
+ for native in [False,True]:
+ page.evaluate('events=[]')
+ page.get_by_role('button',name='Moving').click(
+ position={'x':20.123,'y':20.123},human=not native)
+ result[str(ratio)]['explicit'].append(page.evaluate("""() => ({events,
+ box:document.querySelector('button').getBoundingClientRect().toJSON(),
+ border:parseFloat(getComputedStyle(document.querySelector('button')).borderLeftWidth)})"""))
+ artifact.write_text(json.dumps(result,indent=2))
+ metrics.send('Emulation.clearDeviceMetricsOverride')
+ # Detaching a CDP handle must retire its temporary input state without
+ # affecting the live page or native calls made through another handle.
+ lease=context.new_cdp_session(page)
+ lease.send('Browserwright.setPointerGrid', {'devicePixelRatio':2})
+ lease.detach()
+ page.evaluate('events=[]')
+ page.get_by_role('button',name='Moving').click(
+ position={'x':20.123,'y':20.123},human=False)
+ result['detached_lease']=page.evaluate("""() => ({events,
+ box:document.querySelector('button').getBoundingClientRect().toJSON(),
+ border:parseFloat(getComputedStyle(document.querySelector('button')).borderLeftWidth)})""")
+ artifact.write_text(json.dumps(result,indent=2))
+ metrics.detach()
+ ''')
+ execution = run_skill(script, backend='extension', runtime_dir=e2e_daemon.runtime_dir,
+ timeout=40)
+ output = {'returncode':execution.returncode,'stdout':execution.stdout,'stderr':execution.stderr}
+ if artifact.exists():
+ output['observations'] = json.loads(artifact.read_text())
+ artifact.write_text(json.dumps(output,indent=2)+'\n')
+ assert execution.returncode == 0, output
+ detached = output['observations'].pop('detached_lease')
+ press = next(event for event in detached['events'] if event['type']=='pointerdown')
+ assert press['trusted'] and press['target']=='target', detached
+ for axis in ('x','y'):
+ expected = int((detached['box'][axis]+detached['border']+20.123)*100)/100
+ assert abs(press[axis]-expected) < .001, detached
+ for result in output['observations'].values():
+ observed = result['observation']
+ assert result['return_none'] and observed['clicked'] and observed['shifted'], observed
+ box = observed['box']
+ assert box['x'] == 360.625, box
+ pointer = [event for event in observed['events'] if event['type'].startswith('pointer')]
+ assert any(event['type']=='pointerdown' and event['target']=='target' for event in pointer)
+ for event in pointer:
+ assert event['trusted'], event
+ for sample in [event, *(event['coalesced'] or [])]:
+ assert sample['trusted'], sample
+ for axis in ('x','y'):
+ scaled = sample[axis]*event['dpr']
+ assert abs(scaled-round(scaled)) < .01, event
+ for explicit in result['explicit']:
+ press = next(event for event in explicit['events'] if event['type']=='pointerdown')
+ assert press['trusted'] and press['target']=='target', explicit
+ for axis in ('x','y'):
+ expected = int((explicit['box'][axis]+explicit['border']+20.123)*100)/100
+ assert abs(press[axis]-expected) < .001, explicit
diff --git a/tests/daemon/e2e/test_human_scroll.py b/tests/daemon/e2e/test_human_scroll.py
new file mode 100644
index 0000000..edfdf41
--- /dev/null
+++ b/tests/daemon/e2e/test_human_scroll.py
@@ -0,0 +1,137 @@
+"""Trusted wheel preparation through the extension-only executor.
+
+Repeat with ``mise exec -- uv run pytest tests/daemon/e2e/test_human_scroll.py -q``.
+The artifact records real wheel/scroll order for document, nested and frame input.
+"""
+from __future__ import annotations
+
+import json
+import textwrap
+import pytest
+
+from .helpers import run_skill
+
+
+@pytest.fixture
+def e2e_chrome(extension_only_chrome):
+ return extension_only_chrome
+
+
+_HTML = """Top
+Below
+
+
+"""
+
+
+@pytest.mark.parametrize('extension_only_chrome', ['headless'], indirect=True)
+def test_scroll_target_at_document_bottom(e2e_daemon, e2e_chrome, ext_ready, e2e_artifacts_dir):
+ artifact = e2e_artifacts_dir / 'human-scroll-bottom.json'
+ script = textwrap.dedent('''
+ import json
+ page.set_content("""
+
Edge
+ """)
+ returned=page.get_by_role('button',name='Edge').click(timeout=8000)
+ result={'return_none':returned is None,'clicked':page.evaluate('window.clicked'),
+ 'events':page.evaluate('events')}
+ ''')
+ script += f'\nfrom pathlib import Path\nPath({str(artifact)!r}).write_text(json.dumps(result,indent=2))\n'
+ execution = run_skill(script, backend='extension', runtime_dir=e2e_daemon.runtime_dir,timeout=20)
+ assert execution.returncode == 0, (execution.stdout,execution.stderr)
+ result = json.loads(artifact.read_text())
+ assert result['return_none'] and result['clicked'],result
+ assert any(e['type']=='wheel' and e['trusted'] for e in result['events']),result
+
+
+@pytest.mark.parametrize('extension_only_chrome', ['headless'], indirect=True)
+def test_executor_scroll_has_trusted_wheel_input(
+ e2e_daemon, e2e_chrome, ext_ready, e2e_artifacts_dir,
+):
+ artifact = e2e_artifacts_dir / 'human-scroll.json'
+ artifact.unlink(missing_ok=True)
+ script = 'import json\nfrom pathlib import Path\n'
+ script += f'artifact=Path({str(artifact)!r})\npage.set_content({_HTML!r})\n'
+ script += textwrap.dedent('''
+ from playwright.sync_api import TimeoutError
+ result={}
+ def save(): artifact.write_text(json.dumps(result,indent=2))
+ returned=page.get_by_role('button',name='Top').click()
+ result['visible']={'return_none':returned is None,'events':page.evaluate('events.splice(0)')}
+ save()
+ child_frame=page.frames[1]
+ child_frame.evaluate("""() => {
+ window.events=[]; window.clicked=false;
+ for(const type of ['wheel','scroll','click']) document.addEventListener(type,e=>{
+ events.push({type,trusted:e.isTrusted,time:performance.now()});
+ if(type==='click' && e.target.id==='child') clicked=true;
+ },true);
+ }""")
+ returned=page.get_by_role('textbox',name='Below').fill('Ada')
+ result['document']={'return_none':returned is None,'value':page.locator('#below').input_value(),
+ 'events':page.evaluate('events.splice(0)')}
+ save()
+ returned=page.get_by_role('button',name='Nested').click(position={'x':12,'y':11},force=True)
+ result['nested']={'return_none':returned is None,'clicked':page.evaluate('clicked.slice()'),
+ 'events':page.evaluate('events.splice(0)')}
+ save()
+ child=page.frame_locator('#frame').get_by_role('button',name='Child')
+ returned=child.click()
+ result['frame']={'return_none':returned is None,'events':page.evaluate('events.splice(0)'),
+ 'child_events':child_frame.evaluate('events'),
+ 'child_clicked':child_frame.evaluate('clicked')}
+ save()
+ page.get_by_role('button',name='Top').click(human=False)
+ page.evaluate("events.splice(0); document.addEventListener('wheel',e=>e.preventDefault(),{passive:false})")
+ try:
+ page.get_by_role('textbox',name='Below').click(timeout=3000)
+ result['blocked']={'timeout':False}
+ except TimeoutError:
+ result['blocked']={'timeout':True,'events':page.evaluate('events.splice(0)')}
+ artifact.write_text(json.dumps(result,indent=2))
+ ''')
+ execution = run_skill(script, backend='extension', runtime_dir=e2e_daemon.runtime_dir,
+ timeout=90)
+ output = {'returncode':execution.returncode,'stdout':execution.stdout,'stderr':execution.stderr}
+ if artifact.exists():
+ output['observations'] = json.loads(artifact.read_text())
+ artifact.write_text(json.dumps(output,indent=2)+'\n')
+ assert execution.returncode == 0, output
+ observations = output['observations']
+ assert observations['document']['value'] == 'Ada'
+ assert 'nested' in observations['nested']['clicked']
+ assert observations['visible']['return_none']
+ assert not any(e['type'] in ['scroll','wheel'] for e in observations['visible']['events'])
+ assert observations['frame']['child_clicked']
+ for name in ['document','nested','frame']:
+ events = observations[name]['events']
+ assert observations[name]['return_none'], observations[name]
+ assert any(e['type']=='wheel' and e['trusted'] for e in events), events
+ for i, event in enumerate(events):
+ if event['type']=='scroll':
+ assert any(prior['type']=='wheel' and prior['trusted']
+ and event['time']-prior['time']<200 for prior in events[:i]), events
+ child_events=observations['frame']['child_events']
+ assert any(e['type']=='wheel' and e['trusted'] for e in child_events), child_events
+ for i,event in enumerate(child_events):
+ if event['type']=='scroll':
+ assert any(prior['type']=='wheel' and prior['trusted']
+ and event['time']-prior['time']<200 for prior in child_events[:i]), child_events
+ assert observations['blocked']['timeout'], observations['blocked']
+ assert any(e['type']=='wheel' and e['trusted'] for e in observations['blocked']['events'])
+ assert not any(e['type']=='scroll' for e in observations['blocked']['events'])