Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,9 @@ docs-check:
@python3 docs/quickstart/build.py --check >/dev/null \
|| { python3 docs/quickstart/build.py --check; exit 1; }
@sh docs/checks/lib-selftest.sh >/dev/null || { sh docs/checks/lib-selftest.sh; exit 1; }
@echo "docs: spelling, anchors, quick-start pages and the checks' prelude clean"
@python3 docs/checks/private-data.py selftest >/dev/null || { python3 docs/checks/private-data.py selftest; exit 1; }
@python3 docs/checks/private-data.py scan $$(git ls-files) || exit 1
@echo "docs: spelling, anchors, quick-start pages, the checks' prelude and private data clean"

ci: fmt-check vet lint test clients crosscompile java api-check flags-check license-check docs-check dotnet

Expand Down
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
*Mutatis mutandis.*

**Mobile app automation for AI agents and people.** Android emulators, Android
phones, iOS simulators and iPhones — and TVs, Wear OS and Meta Quest's 2D app
panels — driven through one tool layer from a single Go binary with no runtime
phones, iOS simulators and iPhones — and TVs, Wear OS, Android Automotive and
Meta Quest's 2D app panels — driven through one tool layer from a single Go binary with no runtime
dependencies.

```sh
Expand Down Expand Up @@ -168,8 +168,8 @@ app may, and reporting "empty" would be a different and usually false claim.
Verified on devices, not only compiled: a Pixel 7 emulator on Android 15 and
Android 17, a Pixel 8 Pro on Android 17, an iPhone 17 Pro simulator on iOS
26.5 and an iPhone 18 Pro simulator on iOS 27.0, and an iPhone 15 Plus on iOS
26.6.2 and, since its update, iOS 27.0.1. TVs, Wear OS and Meta Quest are
below. The end-to-end flows in
26.6.2 and, since its update, iOS 27.0.1. TVs, Wear OS, cars and Meta Quest
are below. The end-to-end flows in
[docs/checks/](docs/checks/) each drive a real device and assert on what
happened rather than on an exit code.

Expand Down Expand Up @@ -211,6 +211,14 @@ answers — with swiping right as a watch's back
([docs/checks/wear-os.sh](docs/checks/wear-os.sh)). Not yet a real watch,
and not the crown: `press` has no rotary input.

**A car's screen** is Android too: since 2026-11-11 the UiAutomator2 backend
drives the Android Automotive 15 emulator — the system bars and the app
between them, the app's dialogs and the car's permission prompt, with the
keyboard left out of `map` as a phone leaves it — and acts for user 10, the
driver, as the car's apps do ([docs/checks/automotive.sh](docs/checks/automotive.sh)).
Not yet a real car, nor the instrument cluster's displays, and the
`uiautomator` driver reads only the app's window there.

**Meta Quest's 2D apps** are Android too: since 2026-10-25 the Android
backends drive the panels on the Meta Spatial Simulator, Meta's Horizon OS
emulator, where every app is a panel and every panel stays open. Not yet a
Expand Down
2 changes: 1 addition & 1 deletion cmd/mobium/boot.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ func newBootCmd() *cobra.Command {
return runTool("app_boot", call)
},
}
cmd.Flags().BoolVar(&window, "window", false, "Show the emulator's window (default: headless)")
cmd.Flags().BoolVar(&window, "window", false, "Show the emulator's window, or the simulator in DeviceHub or Simulator.app (default: headless)")
return cmd
}

Expand Down
245 changes: 234 additions & 11 deletions docs/CHALLENGES.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/RELEASE-CHECKLIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
[CI workflow](../.github/workflows/ci.yml) runs on every push — formatting,
vet, both modules' tests, the four non-Go clients' own tests, and six
cross-compile targets. This file is the rest, and the rest is where most
serious defects in this project have come from: 314 of 392 were found only by
serious defects in this project have come from: 327 of 407 were found only by
running against a real device.

Work through it before tagging a release, on three substrates: an Android
Expand Down
12 changes: 12 additions & 0 deletions docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,18 @@ this is what is not.
which `press` cannot send — Wear's lists also scroll by swipe, so
nothing needs it yet. HarmonyOS, KaiOS and BlackBerry: no plans to
implement at the moment; anyone can write a driver.
**Android Automotive measured 2026-11-11** (CHALLENGES 393–404): the
UiAutomator2 backend drives the car emulator (`automotive.sh`). Not
tried: the `uiautomator` driver on a car, which reads one window; the
instrument cluster's displays; a car's portrait screen; Android Auto,
whose head unit has no hierarchy.
**CarPlay is not reachable in Xcode 27.1's simulator** (looked for
2026-11-11, iPhone 18 Pro on iOS 27.0): DeviceHub, which replaces
Simulator.app, has no CarPlay or external-display item in any menu or
panel, and `simctl io enumerate` lists LCD, Resizable, TVOut and
Wireless screens, none of them CarPlay; a screenshot of the external
one hangs. Left: Apple's standalone CarPlay Simulator, which drives a
real iPhone, or an older Xcode's Simulator.app.
3. **`press-tap` and `press-drag` on Android 16 and later**, refused
since CHALLENGES 86: the refusals and comments said a restart would
recover a device left with fingers down, which 86 measured it does
Expand Down
20 changes: 11 additions & 9 deletions docs/SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ emulator -avd mobium-test -no-window -no-snapshot-save -no-boot-anim -no-audio &
```

**Unlike an iOS simulator, the window cannot be attached afterwards.** `simctl
boot` is always headless and `open -a Simulator` shows it later; the Android
boot` is always headless and `mobium boot <udid> --window` shows it later; the Android
emulator decides at launch, and the console (`adb emu help`) has no command to
raise a window. To see the screen on a `-no-window` emulator, take a
screenshot — or restart it without the flag.
Expand Down Expand Up @@ -385,15 +385,15 @@ Android 11+. The phone and the Mac must be on the same network.

```sh
adb mdns services
# adb-3C19...-FIOvFv _adb-tls-pairing._tcp 10.0.0.96:33169
# adb-3C19...-FIOvFv _adb-tls-connect._tcp 10.0.0.96:43943
# adb-XXXXXXXX-abcdef _adb-tls-pairing._tcp 192.0.2.20:33169
# adb-XXXXXXXX-abcdef _adb-tls-connect._tcp 192.0.2.20:43943
```

4. Pair against the **pairing** port, which is not the one on the main
Wireless debugging screen:

```sh
adb pair 10.0.0.96:33169 123456
adb pair 192.0.2.20:33169 123456
```

`adb pair` may answer `protocol fault (couldn't read status message)` **and
Expand Down Expand Up @@ -455,7 +455,7 @@ xcodebuild -downloadPlatform iOS # the runtime, if it is missing

xcrun simctl list devices available
xcrun simctl boot <udid> # boots headless — no window appears
open -a Simulator # only this puts it on screen
mobium boot <udid> --window # only this puts it on screen
mobium --device <udid> map
```

Expand All @@ -464,9 +464,11 @@ has, and `mobium shutdown "iPhone 17 Pro"` ends the session and shuts it down.

**`simctl boot` shows you nothing.** It starts the runtime with no window, so
a simulator can be fully booted and driveable while the screen stays empty.
`open -a Simulator` attaches the GUI to whatever is already booted. Mobium
neither needs nor opens it — `xcrun simctl io <udid> screenshot` works
headless — but it is the first thing anyone asks about.
`mobium boot <udid> --window` opens it on screen: in Simulator.app, or from
Xcode 27.1, which has none, in DeviceHub. A DeviceHub already open is brought
forward and left on the device it shows, and the answer names the one to pick.
Mobium needs no window — `xcrun simctl io <udid> screenshot` works headless —
but it is the first thing anyone asks about.

**Install Xcode from the Mac App Store.** `xcodes install` has been broken
since 2026-09-11: Apple removed the `/olympus/v1/app/config` endpoint it uses
Expand Down Expand Up @@ -544,7 +546,7 @@ xcodebuild -downloadPlatform tvOS # or Xcode > Settings > Components
xcrun simctl create "Apple TV" \
com.apple.CoreSimulator.SimDeviceType.Apple-TV-4K-3rd-generation-1080p \
com.apple.CoreSimulator.SimRuntime.tvOS-26-5
xcrun simctl boot <udid> && open -a Simulator
mobium boot <udid> --window
```

`mobium devices` lists it as an `apple tv simulator`, and the first command
Expand Down
1 change: 1 addition & 0 deletions docs/checks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ checks do not: `clean-stop.sh`, which stops the caller's daemon, and
| [flutter.sh](flutter.sh) | **A Flutter app** — MobiumApp's `flutter/` demo, installed by hand — on any device, through the semantics tree and no driver: `map` names every control; typing reaches the app, judged by what the app says it got, the password not echoed; a Semantics identifier taps its button; check, uncheck, a counted tap; `scroll-to` the fortieth row; back from a pushed screen. Passed 2026-10-01/02 on the Pixel 7 AVD, the iPhone 17 Pro simulator, the Pixel 8 Pro and the iPhone 15 Plus ([CHALLENGES 206](../CHALLENGES.md)) |
| [tv-app.sh](tv-app.sh) | **A TV app driven with a remote** — mobium-app's `tv/` app, MobiumTV — on any Android device: a D-pad press says where focus went and the app agrees; select and a tap arrive as select and as touch; a tap on a tile that opens on focus reaches it as focus, not a click; nine presses right scroll a row to Card 10; `app_alert` sees a dialog, accept is its positive button and back cancels it; play-pause, fast-forward and stop arrive as the app counts them. Passed 2026-10-08 on the Android TV emulator (API 34) and on a Fire TV (Fire OS 7.7.1.7) over Wi-Fi, in 2 minutes 9 seconds; the TV found that a read can land before the screen it expects, so every screen and line is waited for |
| [wear-os.sh](wear-os.sh) | **A watch** — Wear OS's own Settings, by locator: `text="Gestures"` is one match though a chip repeats its title's text; map lists a toggle chip's switch once, not three times; `app_alert` reads Wear's ✓/✕ dialog, dismiss leaves Tilt-to-wake off and accept presses OK; swiping right goes back. Refuses anything that is not a watch. Passed 2026-10-13 three times in three on a Wear OS 5.1 emulator (API 35, round 454×454), about 20 seconds; on `main` it failed at its first tap (CHALLENGES 315–317) |
| [automotive.sh](automotive.sh) | **A car** — Android Automotive, its screen of system bars, app and keyboard windows, its apps run as user 10: `map` lists the bars and the launcher; a tap in the bottom bar moves the temperature, read back and put back, pressed again if the car drops the press; an untargeted swipe; each app tile mapped once and car-ui's marker not read as text; `devices` says car bars and a back gesture is refused; the car's back button mapped, and a tap on it goes back; letterboxed MobiumApp is the app in front and no dialog; `app_alert` quotes the app's dialog from its own words, not the instrument cluster, and the car's permission prompt by its named buttons, `dismiss` refused and `--button` pressing; with the keyboard up no key is mapped and nothing reads as a dialog; `grant`, `logs --app` and `clear-data` act for user 10, each read back from the device or the app. Refuses anything that is not a car, or a car in front for user 0. Passed 2026-11-11 on the Android Automotive 15 emulator (1024×768); on `main` it failed at its first step (CHALLENGES 393–404) |
| [twa.sh](twa.sh) | **A PWA from the Play Store** — a Trusted Web Activity, OYO Lite by default, installed by hand — on an Android phone: checked to be a TWA by its task, `launch` saying so, its page attached by the site's host and standalone, a tap counted by the page, and back through a tapped link's history and then out, each naming the app. Passed on the Pixel 8 Pro on 2026-10-01 ([CHALLENGES 205](../CHALLENGES.md)) |
| [chrome.sh](chrome.sh) | **A page in Chrome on Android**, on an emulator or a phone, with the network: listed by its title and URL, `text`, `eval` and `map` answered by the page, a tap on a ref counted by a button put on the page, and the page's own link followed to iana.org. On an emulator it starts Chrome fresh first ([CHALLENGES 200](../CHALLENGES.md)); on a phone it leaves Chrome running, prints no listing, and closes its tab |
| [pwa.sh](pwa.sh) | **A progressive web app**, Squoosh, on an emulator, an Android phone or a simulator, with the network: installed through Chrome's menu or Safari's share sheet when its icon is absent, launched from the icon, running standalone with an active service worker, its context found by its start URL. iOS: a tap in the web context, placed from text the page and the native tree both report, and one from `NATIVE_APP`, each counted by the page ([CHALLENGES 248, 290](../CHALLENGES.md)). Android phone: installed as a WebAPK, launched by its package with `app_launch` saying so ([CHALLENGES 202](../CHALLENGES.md)), the tap counted by the page, and the WebAPK and the tabs removed afterwards. Emulator: the tap counted, or refused because Chrome stopped reporting its WebView and the page counted nothing ([CHALLENGES 200](../CHALLENGES.md)). An iPhone refused |
Expand Down
4 changes: 2 additions & 2 deletions docs/checks/audit.sh
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ if [ "$KIND" = android ]; then
[ "$got" = "LabelInName:testid=sendMismatch,RedundantDescription:testid=saveRedundant,SpeakableTextPresent:testid=iconUnnamed" ] ||
fail "the Screen Reader Demo's findings were $got, want its nameless icon, its Save button and its Submit form alone"
echo "$out" | json '",".join(d.get("unchecked", []))' | grep -q TextContrast || fail "contrast is not named as unchecked"
$M find testid=iconUnnamed >/dev/null && $M find testid=saveRedundant >/dev/null || fail "a finding's locator does not resolve"
shows testid=iconUnnamed && shows testid=saveRedundant || fail "a finding's locator does not resolve"
row "spoken" "the nameless icon, Save button and Submit form, not twins"
$M terminate $APP >/dev/null 2>&1 || true
echo PASS; exit 0
Expand All @@ -84,7 +84,7 @@ types=$(echo "$out" | json '",".join(sorted(set(f["type"] for f in d["findings"]
nameless=$(echo "$out" | json 'sum(1 for f in d["findings"] if not f.get("bounds"))')
loc=$(echo "$out" | json 'next((f["locator"] for f in d["findings"] if f.get("locator","").startswith("testid=")), "")')
if [ -n "$loc" ]; then
$M find "$loc" >/dev/null 2>&1 || fail "a finding's locator does not resolve"
shows "$loc" || fail "a finding's locator does not resolve"
fi
row "settings" "$n finding(s): $types; $nameless with no element; a locator resolves"
$M terminate com.apple.Preferences >/dev/null 2>&1 || true
Expand Down
Loading
Loading