[tanishqbaweja]

Problem

Snapshot answers with the UI tree (UI Tree:), so an element left out of the tree is left out of the answer, and with the default use_vision=False the tree is the only list of elements the agent gets. For a browser window, Tree.get_nodes() walks the window with no semantic parent (window_sem_node was created only if not is_browser) and builds the window's tree afterwards from the page's DOM lists alone. Everything else the walk finds is captured, numbered for label= and boxed on the annotated screenshot, but not listed:

• Chrome and Edge: the tabs, the address bar, the toolbar buttons and the browser's own dialogs.
• Firefox: everything. Its page has no RootWebArea (as the IA2 fallback's comment says), so it is walked like the rest of the window, and the IA2 fallback that fills the DOM lists runs only with use_dom=True. Its tree is the window line alone.

Snapshot's description says "Set use_dom=True for browser content to get web page elements instead of browser UI", and the README says DOM mode is "filtering out browser UI elements". Without use_dom, though, a browser window's tree is the same page-only tree. Before 8c6c670 replaced it with the UI tree, Snapshot printed the flat list of interactive_nodes, which still holds these controls.

Reproduction

At 106b50c on Windows 11 (build 22631) with Python 3.14.9, with mocks only. A script built two browser windows out of MagicMock UIA elements that hold the cached properties the walk reads: a toolbar ("App bar") with Back and the address bar, above a page with a text and a link. The Chrome window's page has the AutomationId RootWebArea; the Firefox window's has none. The script ran them through the real Tree.get_state() and tree_traversal(), with ControlFromHandle, the cache requests and the children lookup patched, and printed the labels and semantic_tree_to_string(), which Snapshot prints under UI Tree:. Output on main:

== Google Chrome, Snapshot(use_dom=False)
labels: 0: edit "Address and search bar", 1: button "Back", 2: link "Learn more"
UI Tree:
desktop
    └── window "Example Domain - Google Chrome"
        ├── (200,310) link "Learn more"  [action: click]
        └── text "This domain is for use in documentation examples."
== Mozilla Firefox, Snapshot(use_dom=False)
labels: 0: link "Learn more", 1: edit "Address and search bar", 2: button "Back"
UI Tree:
desktop
    └── window "Example Domain - Mozilla Firefox"
== Google Chrome, Snapshot(use_dom=True)
labels: 0: link "Learn more"
UI Tree:
desktop
    └── window "Example Domain - Google Chrome"
        ├── (200,310) link "Learn more"  [action: click]
        └── text "This domain is for use in documentation examples."

Fix

tree/service.py, get_nodes(): every window gets its semantic node before the walk.

• Without use_dom, the walk attaches a browser's own controls to it, as it does for any app. The page's elements and text are then added from the DOM lists, in page order, as before. The walk enters RootWebArea without a semantic parent, so nothing is listed twice.
• With use_dom=True, the walk attaches nothing to a browser window, so its tree is still the page alone.
• Unchanged: other windows, the labels, the annotated screenshot, WaitFor and Scrape.

With the change, the reproduction prints the same labels and the same use_dom=True tree. The two other trees become:

desktop
    └── window "Example Domain - Google Chrome"
        ├── tool bar "App bar"
        │   ├── (25,60) button "Back"  [action: click]
        │   └── (510,60) edit "Address and search bar"  [action: fill]
        ├── (200,310) link "Learn more"  [action: click]
        └── text "This domain is for use in documentation examples."

desktop
    └── window "Example Domain - Mozilla Firefox"
        ├── tool bar "App bar"
        │   ├── (25,60) button "Back"  [action: click]
        │   └── (510,60) edit "Address and search bar"  [action: fill]
        └── (200,310) link "Learn more"  [action: click]

The browser's controls come first, as the window lays them out above the page. If you would rather keep the default tree to the page too, the use_dom description would need to change, and the Firefox window would still need the elements its walk finds.

Tests

New tests/test_tree_browser_controls.py, mocks only: the same two windows go through the real Tree.get_state() and tree_traversal().

• test_browser_controls_are_listed_above_the_page: the Chrome window without use_dom. The labels are the address bar, Back and the link, and the tree lists the toolbar with Back and the address bar, then the link and the text.
• test_page_without_root_web_area_is_listed: the Firefox window without use_dom. The tree lists the toolbar and the link.
• test_use_dom_lists_the_page_alone: the Chrome window with use_dom=True. The labels and the tree hold the page alone, as on main.

I ran each file on its own, in the project's environment:

python -m pytest tests/test_tree_browser_controls.py -v
python -m pytest tests/test_tree_service.py
python -m pytest tests/test_tree_closed_window.py
python -m pytest tests/test_tree_retry_backoff.py
python -m pytest tests/test_tree_walk_z_order.py
python -m pytest tests/test_uia_timeouts.py
python -m pytest tests/test_tree_views.py

On main, the first two new tests fail at the tree, after their label check passes: At index 2 diff: ' ├── (200,310) link "More info" [action: click]' != ' ├── tool bar "App bar"' and Right contains 4 more items, first extra item: ' ├── tool bar "App bar"'. test_use_dom_lists_the_page_alone passes on both. All 3 pass on this branch.

These pass on main and on this branch with the same counts: tests/test_tree_service.py (50 passed), tests/test_tree_closed_window.py (4), tests/test_tree_retry_backoff.py (3), tests/test_tree_walk_z_order.py (3), tests/test_uia_timeouts.py (17) and tests/test_tree_views.py (29). I did not run the whole suite.

ruff check and ruff format --check are clean on the new test file. On tree/service.py, ruff check reports the same two W293 lines (650 and 749) on main and on this branch, and the file is not formatted on main either; on the lines this change edits, ruff format --diff shows only what it shows there on main (the single quotes of the window node and the long tree_traversal() call).

Not tested: a real browser. Everything ran against mocks. To check it on a real machine, open a page in Chrome or Edge and call Snapshot without use_dom: with this change the UI tree should list the browser's tabs and toolbar (the address bar, Back) above the page's elements, and on main only the page's. With use_dom=True, both should list the page alone. In Firefox, Snapshot without use_dom should show only the window line on main.