[tanishqbaweja]

Problem

UIA reports an element that is scrolled out of view as off-screen, and a provider can still give its real position, past the edge of the view. tree_traversal() drops off-screen elements, except edit fields and a browser's list items:

is_visible = (area > 0) and (not is_offscreen or control_type_name=="EditControl" or (control_type_name=="ListItemControl" and is_browser)) and is_control_element

So a text box further down a web page, or below the bottom of a long form's window, is taken as interactive. iou_bounding_box() then clips its box to its window (in a browser, to the page) and to the screen. Nothing of it is left, so the box becomes (0,0,0,0) and the center (0,0), and a named field is still listed:

• Snapshot's UI tree prints it as (0,0) edit "Comment" [action: fill].
• get_coordinates_from_label() returns (0, 0) for its label, so Click and Type with that label click the top-left corner of the primary display, and Type types there.

#592 described this (0,0) result for elements outside a stale screen box and fixed the stale box. Elements outside their own window or page still get it.

Reproduction

At 106b50c on Windows 11 (build 22631) with Python 3.14.9, with mocks only. A script built two windows out of MagicMock UIA elements that hold the cached properties the walk reads: a Chrome window whose page (RootWebArea) has a link "Sign in" and, further down, an edit "Comment" at Rect(100, 1500, 700, 1530); and a "Settings" window at (0,0)-(800,600) with a button "OK" and an edit "Notes" at Rect(100, 900, 700, 930), below the window. Both edits report IsOffscreen. The script ran them through the real Tree.get_state() and tree_traversal(), with ControlFromHandle, the cache requests and the children lookup patched, and printed each label with the point Desktop.get_coordinates_from_label() gives for it (what Click and Type use for label=), then semantic_tree_to_string(). Output on main:

== Chrome, Snapshot(use_dom=True)
label 0: link 'Sign in' -> (200, 310)
label 1: edit 'Comment' -> (0, 0)
UI Tree:
desktop
    └── window "News - Google Chrome"
        ├── (200,310) link "Sign in"  [action: click]
        └── (0,0) edit "Comment"  [action: fill]
== Settings window, Snapshot()
label 0: edit 'Notes' -> (0, 0)
label 1: button 'OK' -> (540, 515)
UI Tree:
desktop
    └── window "Settings"
        ├── (540,515) button "OK"  [action: click]
        └── (0,0) edit "Notes"  [action: fill]

Without use_dom the Chrome window gives the same labels and tree.

Fix

tree/service.py, tree_traversal(): before an element is taken as interactive, its box is clipped as it would be listed (to the page in a browser's DOM walk, else to the window, and to the screen). When nothing is left, the element is left out, as the IA2 walk (_record() in tree/ia2.py) and the word nodes (_append_word_nodes()) already do. The reproduction then gives:

== Chrome, Snapshot(use_dom=True)
label 0: link 'Sign in' -> (200, 310)
UI Tree:
desktop
    └── window "News - Google Chrome"
        └── (200,310) link "Sign in"  [action: click]
== Settings window, Snapshot()
label 0: button 'OK' -> (540, 515)
UI Tree:
desktop
    └── window "Settings"
        └── (540,515) button "OK"  [action: click]

• Only elements that were listed at (0,0) change. An element with any part on screen is listed as before, with the center of that part, including an edit that reports IsOffscreen while its box is on screen.
• The labels after a left-out element are one lower. The annotated screenshot never drew it (its box was empty), and Snapshot with region or display already dropped it: the region filter drops empty boxes.
• WaitFor reads the same list, so such a field no longer satisfies its conditions, as an off-screen button already does not.
• The walk no longer reads the value, words and text selection of a field that is not on screen.

Tests

New TestTreeTraversal::test_field_scrolled_out_of_view_is_not_listed_at_the_origin in tests/test_tree_service.py, 2 cases (a window, and a page in a browser's DOM walk). A form holds two edits that report IsOffscreen: "Name", on screen, and "Comment", below the window or the visible page. The walk is the real tree_traversal(), with the children lookup patched. Only "Name" is listed, at (210,30); in the window case the rendered tree has that one element.

I ran each file on its own, in the project's environment:

python -m pytest tests/test_tree_service.py -v
python -m pytest tests/test_tree_views.py
python -m pytest tests/test_tree_closed_window.py
python -m pytest tests/test_tree_retry_backoff.py
python -m pytest tests/test_tree_walk_z_order.py
python -m pytest tests/test_uia_timeouts.py
python -m pytest tests/test_tree_cache_utils.py

On main both new cases fail: assert [('Comment', '(0,0)'), ('Name', '(210,30)')] == [('Name', '(210,30)')] (window) and assert [('Name', '(210,30)'), ('Comment', '(0,0)')] == [('Name', '(210,30)')] (web page). On this branch the file passes: 52 passed, the 50 tests on main and the 2 new cases. The other files pass on main and on this branch with the same counts: test_tree_views.py (29 passed), test_tree_closed_window.py (4), test_tree_retry_backoff.py (3), test_tree_walk_z_order.py (3), test_uia_timeouts.py (17) and test_tree_cache_utils.py (3). I did not run the whole suite.

This merges cleanly with #614, which changes get_nodes() in the same file; with both, tests/test_tree_browser_controls.py (3 passed) and tests/test_tree_service.py (52) pass.

ruff check is clean on the test file; on tree/service.py it reports the two W293 lines that main has (650 and 749 there, 650 and 762 here). Neither file is formatted on main: ruff format leaves the added lines as they are, and on the test file it shows only the hunk it shows on main.

Not tested

A real browser or app: everything ran against mocks. To check it on a real machine, open a page in Chrome or Edge with a labelled text box below the visible part (a sign-up form longer than the window, say) and call Snapshot. On main the UI tree should list that box at (0,0) with [action: fill]; with this change it should not be listed until it is scrolled into view. An edit field scrolled out of view in a long desktop form should behave the same.