diff --git a/docs/CHALLENGES.md b/docs/CHALLENGES.md index 39cad8e..b758808 100644 --- a/docs/CHALLENGES.md +++ b/docs/CHALLENGES.md @@ -7,7 +7,7 @@ the source. The pattern across them is the reason the document exists: **almost nothing here was found by reading code or by a test written from imagination.** Of -390 defects, 312 were found only by running against a real device. Seventeen +392 defects, 314 were found only by running against a real device. Seventeen more — 255, 280, 282, 327, 329, 353, 355, 356, 359, 371, 375, 380, 382, 384, 386, 387 and 388 — were met while running against one, though the fault lay in a check, the host or Mobium's own design rather than in the platform. The other sixty-one — 4, 5, 6, 14, 23, @@ -8644,6 +8644,36 @@ kHz at -42 dBFS, and above 1320 Hz neither version split, since a break bends a higher tone less. Not measured: a tone from anything but MobiumApp (Chrome played nothing), and any other host. Not run on a phone, which captures no samples. +### 391. A wait for something to go said it had, on a locked iPhone simulator + +**Found by:** a person verifying 389 on an Intel iMac, 2026-11-05 +(mobiumdev/mobium#327), who saw `wait` time out saying only "it is not on screen" +with an iOS simulator locked. Trying every form of `wait` then found worse. iOS +reads a lock screen, so a target is simply absent from it: `wait --for hidden`, +`--not` and `--count 0` answered `is hidden`, exit 0, about an app nobody could +see. Android cannot read a sleeping screen, and refused them all. + +**Fixed:** a wait that holds because its target is absent, or that times out, asks +whether the device is locked when the system shell is in front, and answers +`device_not_ready`. So do `find` and `text` on a miss. A wait to show still waits +out its timeout, for a person unlocking by hand. On an unlocked home screen all of +these answer as they did, and `lock.sh` fails on `main` and passes here. + +### 392. A list's testID on iOS gave no ref, where Android's gave the list + +**Found by:** the same verification (mobiumdev/mobium#329). `wait testid=table` +answered `@e2 table (list)` on Android and `(no ref) table` on the iOS 26.5 +simulator. Not 387's doing: before it, iOS answered with no element at all. React +Native on iOS puts the testID on a plain view around its ScrollView, on exactly the +same rectangle, 16,213 370x611 points, and on Android on the ScrollView itself. + +**Fixed:** a match the map filters out takes the ref of the one mapped element +inside it on its own rectangle, in `wait`, `scroll-to` and `find`. Only on the same +rectangle, or a ref would touch something other than what matched, which is 387 +from the other side: a wrapper around a smaller button still has no ref. A sweep of +test IDs in 14 screens of real apps on both platforms gave list refs only to such +containers. iOS still labels it `ScrollView` where Android says `table`. + ## Findings that were not defects Worth recording because each one closed off an approach that looked obvious. diff --git a/docs/RELEASE-CHECKLIST.md b/docs/RELEASE-CHECKLIST.md index ec5f290..555d847 100644 --- a/docs/RELEASE-CHECKLIST.md +++ b/docs/RELEASE-CHECKLIST.md @@ -4,7 +4,7 @@ [CI workflow](../.github/workflows/ci.yml) runs on every push — formatting, vet, both modules' tests, the four non-Go clients' own tests, and six cross-compile targets. This file is the rest, and the rest is where most -serious defects in this project have come from: 312 of 390 were found only by +serious defects in this project have come from: 314 of 392 were found only by running against a real device. Work through it before tagging a release, on three substrates: an Android diff --git a/docs/checks/README.md b/docs/checks/README.md index abfd92f..596b178 100644 --- a/docs/checks/README.md +++ b/docs/checks/README.md @@ -76,7 +76,7 @@ checks do not: `clean-stop.sh`, which stops the caller's daemon, and | [audio.sh](audio.sh) | **Audio capture on an Android emulator**, on MobiumApp's Audio Demo: 440 Hz for 2 s heard as such, 440 then a second of silence then 880 in order, 660 Hz played as an alarm, a tone ended at Stop, and the silence control heard as silence while `dumpsys audio` reports a player started for it — the contrast the capture exists for. Each assertion was shown to fail on a wrong expectation. Refuses a phone and an iOS device by name | | [audio-sim.sh](audio-sim.sh) | **Audio on an iOS simulator**, on MobiumApp's Audio Demo, through a process tap on the app alone: a tone heard as 440 Hz for about two seconds, the sequence as 440 then 880 Hz, the Silent button's player writing zeros heard as silence, an expect of 660 Hz for a 440 tone failing with what was heard and the WAV kept, and Settings, which opens no audio, said never to have. Needs Xcode and System Audio Recording allowed for what runs mobium. Passed three of three on the iPhone 17 Pro simulator, 2026-10-22 | | [record.sh](record.sh) | **Screen recording**: something moving recorded, saved where the command ran, and judged by the video's own header — frames and duration — then a still screen, fewer frames and still a valid file (on a real iPhone, whose stream sends frames at a steady rate, a steady rate of them), the refusals, and nothing left recording or on the device | -| [lock.sh](lock.sh) | **A locked device**, straight after `lock lock`: a tap is refused as device_not_ready and reaches nothing, on Android a read names the lock and a touch at a point is refused because the screen is off, and on iOS `scroll-to` leaves the lock screen unswiped. Unlocked, the same tap lands. Emulators and simulators only (CHALLENGES 389) | +| [lock.sh](lock.sh) | **A locked device**, straight after `lock lock`: a tap is refused as device_not_ready and reaches nothing, a read names the lock and a wait for absence is refused rather than claimed, on Android a touch at a point is refused because the screen is off, and on iOS `scroll-to` leaves the lock screen unswiped. Unlocked, the same tap lands. Emulators and simulators only (CHALLENGES 389, 391) | | [locale-ios.sh](locale-ios.sh) | **Per-app language on iOS**, read back from the app: Settings pinned to ja-JP reads 一般 for General — relaunched by the pin, and again on a later launch — and General once cleared; a malformed tag is refused. iOS has no stored per-app language, so this holds Mobium's launch arguments to what the app shows. Android's half is `device-state.sh` | | [notifications-ios.sh](notifications-ios.sh) | **Notifications on an iOS simulator**, read back from Notification Center: a post as MobiumApp confirmed by its banner with MobiumApp still in front; the read finding it by app and body and closing Notification Center again; the shade opened so `map` reads Notification Center and finds it, then closed back to MobiumApp; and a post with the home screen in front refused. On a real iPhone, only the refusal is checked | | [timezone-ios.sh](timezone-ios.sh) | **The time zone on iOS**, read back from Calendar, which marks the hour in progress: the device's hour, then Asia/Tokyo's once set — the app in front relaunched, and a later launch too — and the device's again once set back; an unknown zone refused. Only hour rows are read, so a phone's events are never printed | @@ -113,7 +113,7 @@ checks do not: `clean-stop.sh`, which stops the caller's daemon, and | [speech.sh](speech.sh) | **What the screen reader says**, `mobium speech`, on MobiumApp's Login Demo and Screen Reader Demo: refused with the screen reader off, naming the setting; a sentence about a typed password field withheld; a walk hearing the icon with no name as a bare "Button" and Send as "Submit form" — on Android "Save button" with its kind said twice and every sentence beside its element, on an iPhone both Save buttons said alike and neither named — and ending back round on Android, where the cursor stops on an iPhone; listening around a wrong sign-in hearing the app's announcement on both, and on Android around the carts hearing "Live cart: 1" as a change and never the quiet count — on an iPhone neither, since React Native's live region is Android's; the screen reader off after the session. On Android it sets TalkBack's log output level to VERBOSE through TalkBack's own settings and back as found, and closes the Accessibility Suite's notification prompt with back, answering nothing. A simulator is refused; a phone needs `ALLOW_PHONE=1` | | [login.sh](login.sh) | **The login demo** — mobile automation's hello world, on MobiumApp's Login screen, and the script to show when showing Mobium. Negative paths first, each judged by the message the app shows: an empty form, a username too short, one with characters it does not allow, a password too short, a wrong password and an unknown user, told apart by nothing. Then positive: one field valid while the other is not, a username sanitized on submit, the welcome screen waited for rather than slept for, and log-out. A typed password is never printed. If iOS raises "Save Password?" over the app, log-out is skipped and the dialog quoted, rather than a swallowed tap reported as a timeout | | [otp.sh](otp.sh) | **A one-time code**, on MobiumApp's OTP Demo: the code read where it arrives — the notification shade on Android, the banner on iOS, read through rather than landing on SpringBoard (CHALLENGES 155) — six one-digit boxes that move focus on, filled digit by digit and then as one string into the first box, which either arrives whole or names the digit that never did (156); no code, verified, already used, wrong twice, locked, and expired after `background 62`; and Resend refused while the screen disables it | -| [wait-states.sh](wait-states.sh) | **Waiting for a state**, on MobiumApp's Form and Login Demos: `wait --for` checked and unchecked on a custom checkbox, a radio and a real Switch; focused, arriving on a tap and moving to the next field — on iOS by asking WebDriverAgent which element is active, since the tree says every field is unfocused; and value, the whole of it where `text` is a part, with a field showing its placeholder holding "". Each timed out, saying what it saw, before it succeeded. A checked state on a text field and a password's value are refused at once, and a refusal's `details.check` is read from a real one (CHALLENGES 158). A plain text in a list answers with no ref, while a row and the list itself keep theirs (CHALLENGES 387) | +| [wait-states.sh](wait-states.sh) | **Waiting for a state**, on MobiumApp's Form and Login Demos: `wait --for` checked and unchecked on a custom checkbox, a radio and a real Switch; focused, arriving on a tap and moving to the next field — on iOS by asking WebDriverAgent which element is active, since the tree says every field is unfocused; and value, the whole of it where `text` is a part, with a field showing its placeholder holding "". Each timed out, saying what it saw, before it succeeded. A checked state on a text field and a password's value are refused at once, and a refusal's `details.check` is read from a real one (CHALLENGES 158). a plain text in a list answers with no ref, while a row, the list itself and a wrapper on the list's rectangle keep theirs (CHALLENGES 387, 392) | | [network.sh](network.sh) | **Network conditions**, judged by the traffic: on an emulator, latency by ping and download and upload by timing 2MB and 1MB against servers the script runs on the Mac, each before, shaped and after; offline by a connection that must fail; the download limit airplane mode removes, put back on coming online; and a session ended offline and shaped leaving the device online and unshaped. On a phone, offline and back, and shaping refused. The emulator console's own `network speed`/`delay` read back as set and change nothing — see CHALLENGES, findings | | [proxy.sh](proxy.sh) | **A proxy**, judged by what the app received: MobiumApp's Network Demo fetches a port nothing on the device listens on, through a proxy the script runs on the Mac that answers every request itself — failed with no proxy, the proxy's body with `--proxy 127.0.0.1:` (reversed with `adb reverse`, and logged by the proxy), failed again with `--no-proxy`, every proxy key unset and the reverse gone; set again and the session ended, the device as found; a proxy with a scheme refused. An emulator; a phone with `ALLOW_PHONE=1`. Passed 2026-10-22 on the Android 15 emulator and the Pixel 8 Pro | | [test-runner.sh](test-runner.sh) | **`mobium test`**, held to its controls: a project whose device is unset refused before any test runs; the suite in `tests/mobiumapp` passing on every project, the projects at once and faster than one after another; every test in `tests/controls/must-fail.test.json` failing with its step, code and a screenshot, exit 1, and the JUnit file read by Python's XML parser agreeing; `--last-failed` running exactly those; and the flaky control failing with no retry and reported flaky with one. Iteration 2: the soft control reporting both soft failures and still reaching its last step, a retain-on-failure trace kept for a failed test and not a passing one, with its recording in Vibium's record format, and `--debug` stopping before each step and quitting. The suite's login test uses `each`; `--ui` is `test-ui.sh`'s. Emulators and simulators (the flaky control clears app data). [the test runner guide](../guides/test-runner.md) | diff --git a/docs/checks/lock.sh b/docs/checks/lock.sh index 7d532e5..a815904 100755 --- a/docs/checks/lock.sh +++ b/docs/checks/lock.sh @@ -55,11 +55,18 @@ if $M find testid=username 2>/dev/null | grep -q 'username (input)'; then fi row "tap" "refused, and it reached nothing" +# A lock screen holds nothing of the app: a target missing from it has not gone. +fresh +refused "a wait to be hidden" wait "label=Login Demo" --for hidden --timeout 2s +refused "a wait for it not to show" wait "label=Login Demo" --not --timeout 2s +refused "a wait for none" wait "label=Login Demo" --for count --count 0 --timeout 2s +refused "a wait to show" wait "label=Login Demo" --timeout 2s +refused "a find" find "label=Login Demo" +refused "a text" text "label=Login Demo" +row "read" "names the lock, and absence is not claimed" + fresh if [ "$PLATFORM" = android ]; then - refused "a read" find testid=username - row "read" "names the lock, not a changing screen" - fresh refused "a swipe at a point" swipe 540 1500 540 1000 refused "a long press at a point" long-press 540 1500 row "at a point" "refused: the screen is off" diff --git a/docs/checks/wait-states.sh b/docs/checks/wait-states.sh index 648fa32..6925b46 100755 --- a/docs/checks/wait-states.sh +++ b/docs/checks/wait-states.sh @@ -126,5 +126,14 @@ r=$($M wait role=list) || fail "the list itself: $r" echo "$r" | grep -q '@e[0-9].*(list' || fail "the list itself lost its ref: $r" row "answer" "a text has no ref; a row and a list keep theirs" +# React Native on iOS puts this testID on a plain view around the list, on the +# same rectangle (CHALLENGES 392). Android puts it on the list. +open "Table Demo" +r=$($M wait testid=table) || fail "a list's wrapper: $r" +echo "$r" | grep -q '@e[0-9].*(list' || fail "a wrapper on its list's rectangle has no ref: $r" +r=$($M find testid=table) || fail "find a list's wrapper: $r" +echo "$r" | grep -q '@e[0-9].*(list' || fail "find gave a wrapper on its list's rectangle no ref: $r" +row "wrapper" "on its list's rectangle, it has the list's ref" + $M terminate $APP >/dev/null 2>&1 || true echo PASS diff --git a/internal/agent/handlers.go b/internal/agent/handlers.go index 2f63149..8c421cf 100644 --- a/internal/agent/handlers.go +++ b/internal/agent/handlers.go @@ -1064,6 +1064,9 @@ func (h *Handlers) text(ctx context.Context, args map[string]interface{}) (*Tool } node, err := pickToRead(loc, tree) if err != nil { + if matchedNothing(err) && h.lockedNote(ctx, s, tree) != "" { + return nil, lockedOver(loc) + } // Android leaves out what the keyboard covers, so a label under it // is a plain miss there, and "run app_map again" cannot find it; // hiding the keyboard can. @@ -1130,11 +1133,19 @@ func (h *Handlers) find(ctx context.Context, args map[string]interface{}) (*Tool view.Elements = append(view.Elements, elementView(e)) continue } + if e, ok := sameRectEntry(byNode, n); ok { + lines = append(lines, e.Line()) + view.Elements = append(view.Elements, elementView(e)) + continue + } line, ev := noRefElement(n) lines = append(lines, line) view.Elements = append(view.Elements, ev) } if len(lines) == 0 { + if h.lockedNote(ctx, s, tree) != "" { + return nil, lockedOver(loc) + } return Result(fmt.Sprintf("No element matches %s", loc), view), nil } return Result(strings.Join(lines, "\n"), view), nil diff --git a/internal/agent/lock_test.go b/internal/agent/lock_test.go index 269acff..748a592 100644 --- a/internal/agent/lock_test.go +++ b/internal/agent/lock_test.go @@ -171,3 +171,39 @@ func TestScrollToDoesNotSwipeALockScreen(t *testing.T) { t.Errorf("err = %v, want device_not_ready", err) } } + +// A lock screen holds nothing of the app, so a target absent from it has not +// gone. iOS answered "is hidden", exit 0. CHALLENGES 391. +func TestAWaitForAbsenceIsRefusedOnALockScreen(t *testing.T) { + for name, args := range map[string]map[string]interface{}{ + "hidden": {"target": "text=Login", "condition": "hidden", "timeout_ms": 100}, + "not": {"target": "text=Login", "not": true, "timeout_ms": 100}, + "none": {"target": "text=Login", "condition": "count", "count": 0, "timeout_ms": 100}, + "timed": {"target": "text=Login", "timeout_ms": 100}, + } { + h, s, _ := withLock(t, true, keyguard(t)) + _, err := wait(h, s, args) + if mobiumerr.CodeOf(err) != mobiumerr.DeviceNotReady || !strings.Contains(err.Error(), "mobium lock unlock") { + t.Errorf("%s: err = %v, want device_not_ready naming the unlock", name, err) + } + } +} + +func TestAWaitForAbsenceStaysOnAnUnlockedShell(t *testing.T) { + h, s, _ := withLock(t, false, keyguard(t)) + if _, err := wait(h, s, map[string]interface{}{"target": "text=Login", "condition": "hidden", "timeout_ms": 100}); err != nil { + t.Errorf("hidden: %v", err) + } + if _, err := wait(h, s, map[string]interface{}{"target": "text=Login", "timeout_ms": 100}); mobiumerr.CodeOf(err) != mobiumerr.Timeout { + t.Errorf("visible: %v, want timeout", err) + } +} + +func TestAFindMissOnALockScreenNamesTheLock(t *testing.T) { + h, s, _ := withLock(t, true, keyguard(t)) + h.defaultDevice = s.dev.Serial + _, err := h.find(context.Background(), map[string]interface{}{"locator": "text=Login", "device": s.dev.Serial}) + if mobiumerr.CodeOf(err) != mobiumerr.DeviceNotReady { + t.Errorf("err = %v", err) + } +} diff --git a/internal/agent/wait.go b/internal/agent/wait.go index 133deb6..7b24237 100644 --- a/internal/agent/wait.go +++ b/internal/agent/wait.go @@ -312,6 +312,12 @@ func (h *Handlers) waitOn(ctx context.Context, s *session, args map[string]inter }) waited := time.Since(started) + // A lock screen holds nothing of the app: a target absent from it has not + // gone, and one the wait timed out on was never going to come. CHALLENGES 391. + absent := err == nil && matched == nil && (cond != condCount || count == 0) + if (absent || errors.Is(err, errPollTimeout)) && tree != nil && h.lockedNote(ctx, s, tree) != "" { + return nil, lockedOver(loc) + } if err != nil { if errors.Is(err, errPollTimeout) { return nil, mobiumerr.New(mobiumerr.Timeout, "timed out after %s waiting for %s to %s — %s", @@ -552,7 +558,31 @@ func (h *Handlers) rememberRefs(serial string, tree *uitree.Tree, want *uitree.N } return e, true } - return uitree.Entry{}, false + return sameRectEntry(byNode, want) +} + +// sameRectEntry is the one mapped element inside n on n's own rectangle: a plain +// wrapper is that element, as React Native on iOS puts a testID on a view around +// its ScrollView. A smaller one is not, since its ref would touch something other +// than what matched. CHALLENGES 392. +func sameRectEntry(byNode map[*uitree.Node]uitree.Entry, n *uitree.Node) (uitree.Entry, bool) { + if n == nil || n.Bounds.Empty() { + return uitree.Entry{}, false + } + var found uitree.Entry + count := 0 + var walk func(*uitree.Node) + walk = func(c *uitree.Node) { + for _, k := range c.Children { + if e, ok := byNode[k]; ok && k.Bounds == n.Bounds { + found = e + count++ + } + walk(k) + } + } + walk(n) + return found, count == 1 } // noRefElement reports a match the map filters out, a label say, which has no diff --git a/internal/agent/wait_test.go b/internal/agent/wait_test.go index 52e0b3c..68c3fa3 100644 --- a/internal/agent/wait_test.go +++ b/internal/agent/wait_test.go @@ -546,3 +546,45 @@ func TestScrollToAPlainTextInAListHasNoRef(t *testing.T) { t.Errorf("element = %+v, want the text with no ref", got) } } + +// wrapperScreen is React Native on iOS's shape: a testID on a plain view around +// the control. CHALLENGES 392. +func wrapperScreen(t *testing.T) *uitree.Tree { + t.Helper() + tree, err := uitree.ParseAndroid([]byte(`` + + `` + + `` + + `` + + `` + + `` + + `` + + ``)) + if err != nil { + t.Fatalf("parse fixture: %v", err) + } + return tree +} + +func TestAWrapperOnItsControlsRectangleHasItsRef(t *testing.T) { + h, sess, _ := withFake(t, wrapperScreen(t)) + res, err := wait(h, sess, map[string]interface{}{"target": "testid=same", "timeout_ms": 1000}) + if err != nil { + t.Fatalf("wait: %v", err) + } + if got := res.StructuredContent.(WaitView).Element; got == nil || got.Ref == "" || got.Role != "list" { + t.Errorf("element = %+v, want the list's ref", got) + } +} + +// Its ref would touch the button, not the wrapper that matched: #329's defect +// from the other side. +func TestAWrapperAroundASmallerControlHasNoRef(t *testing.T) { + h, sess, _ := withFake(t, wrapperScreen(t)) + res, err := wait(h, sess, map[string]interface{}{"target": "testid=around", "timeout_ms": 1000}) + if err != nil { + t.Fatalf("wait: %v", err) + } + if got := res.StructuredContent.(WaitView).Element; got == nil || got.Ref != "" { + t.Errorf("element = %+v, want no ref", got) + } +}