FastUIA 0.1.0 [ALPHA-2026-06-14]: Native Windows UI Automation API for Java

---

⚡ Direct COM-level Windows UI Automation (UIA) access, zero-copy element queries, and event-driven focus tracking for Java.

FastUIA provides low-latency native UI Automation for Java by communicating directly with the Windows IUIAutomation COM interfaces via JNI, bypassing the marshaling overhead of standard frameworks. Designed for AI agents, RPA bots, and desktop automation systems that need to read and interact with UI elements at millisecond speed using FastCore for seamless native library loading.

Watch Showcase Demo (YouTube)

---

Quick Start

import fastuia.FastUIA;
import fastuia.FastUIAElement;

public class Demo {
    public static void main(String[] args) {
        FastUIA uia = new FastUIA();

        // 1. Query the currently focused UI element
        FastUIAElement el = uia.getFocusedElement();
        if (el != null) {
            System.out.println("Focused: " + el.getName());
            System.out.println("Type:    " + el.getControlType());
            if (el.supportsValue()) {
                System.out.println("Value: " + el.getValue());
            }
        }

        // 2. Get element at specific screen coordinates
        FastUIAElement atPoint = uia.getElementFromPoint(960, 540);
        if (atPoint != null) {
            System.out.println("At (960,540): " + atPoint.getName());
        }

        // 3. Traverse the UI tree
        FastUIAElement root = uia.getRootElement();
        FastUIAElement first = root != null ? root.getFirstChild() : null;
        if (first != null) {
            System.out.println("First desktop child: " + first.getName());
        }
    }
}

---

Table of Contents

• Why FastUIA?
• Quick Start
• Key Features
• Real-World Use Cases
• API Quick Reference
• Technical Demos & Benchmarks
• Installation
• Documentation
• Platform Support
• Related Projects
• License

---

Why FastUIA?

Standard Java UI automation stacks (Microsoft's own UIAutomationClient, third-party wrappers like WinAppDriver, Selenium) introduce multiple abstraction layers that severely degrade performance for tight automation loops:

1. COM Marshaling Overhead: Every UIA call crosses process boundaries through heavyweight COM proxy stubs, adding 1-5 ms of latency per element query in framework wrappers.
2. No Direct Event Dispatch: Standard Java UIA wrappers poll element state or piggyback on slow managed event queues. True event-driven focus and structure change callbacks require native COM sinks.
3. Heap Allocation on Every Query: Wrapper APIs typically box all COM results into managed objects before returning them to Java, causing heap churn and GC pauses inside automation hot paths.

FastUIA bypasses all of these by binding directly to IUIAutomation COM interfaces via JNI:

Feature	Standard Java UIA Wrappers	FastUIA
Element Access Latency	1-5 ms per call (proxy COM marshaling)	Sub-millisecond direct IUIAutomation COM query
Screen-Point Lookup	Framework polling or slow HitTest	Direct ElementFromPoint via native COM call
Focus Events	Polling or heavy event listener overhead	Native IUIAutomationFocusChangedEventHandler sink
Tree Traversal	Object allocation per node	Zero-copy handle-based traversal
Heap Pressure	Boxed COM results on every call	GetPrimitiveArrayCritical pinning, no heap churn
Pattern Support	Generic pattern factory with reflection	Typed native pattern check per interface

---

Key Features

• ⚡ Direct COM-Level Access: JNI binds directly to IUIAutomation and IUIAutomationElement COM interfaces without managed marshaling proxies.
• 🎯 Zero-Copy Element Queries: Bounding rectangles and primitive properties are transferred via GetPrimitiveArrayCritical pinning with no intermediate object allocation.
• 📡 Event-Driven Callbacks: Register native FocusChangedListener, TextChangedListener, and StructureChangedListener sinks backed by real Windows UIA event handlers.
• 🌳 Full Tree Traversal: Navigate the full desktop UI automation tree via getParent(), getFirstChild(), getNextSibling(), getPreviousSibling() with handle-level zero allocation.
• 🧩 Pattern Introspection: Typed per-pattern support checks (supportsValue(), supportsInvoke(), supportsExpandCollapse(), supportsScroll(), and 9 more) backed by native COM QueryInterface.
• 🔗 FastCore Integration: Automated zero-dependency native DLL extraction and loading via FastCore.loadLibrary.

---

Real-World Use Cases

• 🤖 Autonomous AI Desktop Agents: Read the focused element, control type, and bounding rectangle at each step of an AI agent loop to build a real-time UI state model for LLM-driven desktop automation.
• 🕵️ RPA & Workflow Automation: Drive enterprise desktop applications (SAP, thick-client ERP, legacy WinForms) by querying and interacting with UI elements at native COM speed.
• 🧪 High-Speed UI Testing: Poll and validate element states in tight integration test loops without the 5-15 ms overhead of COM-proxied automation frameworks.
• 🖥️ Live UI Inspection & Overlay: Pair FastUIA element queries with FastOverlay to draw real-time bounding boxes around focused or hovered elements.

---

API Quick Reference

FastUIA (Session)

Method	Return Type	Description	Docs
getFocusedElement()	FastUIAElement	Returns the currently keyboard-focused UI element.	Reference
getRootElement()	FastUIAElement	Returns the root desktop automation element.	Reference
getElementFromPoint(x, y)	FastUIAElement	Returns the element at the given screen coordinates.	Reference
setClickThrough(title, enabled)	void	Toggles Win32 click-through on a window by title.	Reference
addFocusChangedListener(l)	void	Registers a native focus-change event sink.	Reference
addTextChangedListener(l)	void	Registers a native text-change event sink.	Reference
addStructureChangedListener(l)	void	Registers a native UI structure-change event sink.	Reference

FastUIAElement

Method	Return Type	Description	Docs
getName()	String	Returns the accessible name of the element.	Reference
getControlType()	ControlType	Returns the typed ControlType enum value.	Reference
getBoundingRect()	Rect	Returns the screen bounding rectangle (x, y, w, h).	Reference
getValue() / setValue(s)	String / void	Gets or sets the element's ValuePattern text.	Reference
getSelection() / setSelection(s)	String / void	Gets or sets the element's TextPattern selection.	Reference
invoke()	void	Triggers the element's default action via InvokePattern.	Reference
expand() / collapse()	void	Controls ExpandCollapsePattern state.	Reference
scroll(h, v)	void	Scrolls element by horizontal and vertical percent.	Reference
getParent() / getFirstChild()	FastUIAElement	Navigates the UI automation tree.	Reference
getNextSibling() / getPreviousSibling()	FastUIAElement	Sibling-level tree traversal.	Reference
getAutomationId()	String	Returns the automation ID string of the element.	Reference
getFrameworkId()	String	Returns the UI framework name (e.g. "Win32", "WPF").	Reference
getProcessId()	int	Returns the owning process ID.	Reference
supportsValue() / supportsInvoke()	boolean	Pattern support introspection via native QueryInterface.	Reference
release()	void	Releases the underlying native COM element handle.	Reference

---

Technical Demos & Benchmarks

Case	Java Example	Launcher	Description
Focus Inspector Demo	Demo.java	run-demo.bat	Queries the focused element, control type, bounding rect, and pattern support at each keystroke.
JMH Microbenchmark Suite	Benchmark.java	run-benchmark.bat	JMH throughput measurements for element queries, tree traversal, and pattern introspection.

---

Installation

Option 1: Maven (Recommended)

Add the JitPack repository and the dependencies to your pom.xml:

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>

<dependencies>
    <!-- FastUIA - Native Windows UI Automation -->
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastUIA</artifactId>
        <version>0.1.0</version>
    </dependency>

    <!-- FastCore - Required Native JNI Loader -->
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastCore</artifactId>
        <version>0.1.0</version>
    </dependency>
</dependencies>

Option 2: Gradle (via JitPack)

repositories {
    maven { url 'https://jitpack.io' }
}

dependencies {
    implementation 'com.github.andrestubbe:FastUIA:0.1.0'
    implementation 'com.github.andrestubbe:FastCore:0.1.0'
}

Option 3: Direct Download (No Build Tool)

Download the release JARs directly from GitHub Releases:

1. 📦 FastUIA-0.1.0.jar (Core UI Automation Library)
2. ⚙️ FastCore-0.1.0.jar (Mandatory Native Loader)

Important

Both JARs must be included in your classpath for the JNI calls to function correctly.

---

Documentation

• COMPILE.md: Full native compilation guide (MSVC C++17 build chain + JNI setup).
• REFERENCE.md: Comprehensive API specification, COM architecture, and event system.
• PHILOSOPHY.md: Engineering rationale for direct COM-level UI Automation binding.
• ROADMAP.md: Planned milestones, pattern expansion, and cross-platform support.
• CHANGELOG.md: Complete version history and release notes.

---

Platform Support

Platform	Architecture	Status	Driver / Subsystem
Windows 10 / 11	x64	✅ Fully Supported	IUIAutomation COM via native Win32 JNI
Linux	x64 / AArch64	🚧 Planned	AT-SPI2 / Atspi via D-Bus
macOS	Apple Silicon / x64	🚧 Planned	Accessibility API (AXUIElement)

---

Related Projects

• FastCore: Native Library Loader & JNI Utilities for Java
• FastOverlay: High-Performance Native DirectComposition Transparent Overlay API for Java
• FastRobot: Low-Latency Native Input & Bot Automation Substrate
• FastScreen: High-Speed DXGI Screen Capture Engine (240-2000 FPS)
• FastWindow: Native Win32 Window Management & Styling Substrate
• FastKeyboard: Ultra-Fast Native RawInput Keyboard Engine

---

License

MIT License. See LICENSE file for details.

---

Part of the FastJava Ecosystem — Making the JVM faster. 🚀