Skip to content

Add Maestro UI tests - #383

Open
stephanlindauer wants to merge 7 commits into
masterfrom
maestro-ui-tests
Open

stephanlindauer wants to merge 7 commits into
masterfrom
maestro-ui-tests

Conversation

@stephanlindauer

@stephanlindauer stephanlindauer commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

What

Adds UI tests that run without any manual work: Maestro flows that drive the release APK on an emulator, on every push to master and every PR. Testing the release APK also covers R8.

The 13 flows cover first launch, navigation, chat input, settings, about, rotation, dark mode, GPX import, camera, the notification, observer mode, offline mode and publishing a position. .maestro/README.md explains how to run them and the rules for new flows.

These tests use the production server

  • Every CI run publishes a test rider on the public map for a few minutes. It sits at a random spot in the open Atlantic, and the flow confirms it through the public feed.
  • Chat messages are not part of CI. manual/chat_send posts a real message, so it only runs when started by hand: ./scripts/run-ui-tests.sh --public-chat.
  • A photo upload can't reach the gallery. The upload is only tapped after the flow has asserted that the device is offline.

Maestro grants every permission by default, which would make the app start publishing positions. All flows therefore set permissions explicitly.

Because the job depends on the production server and the map tile server, an outage there turns it red without a code problem. I suggest not making it a required check, at least for now.

CI setup

GitHub's runners have no GPU, so the emulator renders everything in software. That shaped the configuration:

  • Android 16 (API 36) instead of 17. On the Android 17 x86_64 image, Android restarted itself shortly after the map was reopened, right after graphics errors in the emulator log. No app crash showed up, but I didn't get as far as identifying the crashing process.
  • A small screen (Nexus 4, 768×1280). At 1080×2400, System UI and the launcher stopped responding, and their dialogs blocked every flow.
  • Error dialogs hidden, 4 cores, 4 GB RAM. A crashed app still fails its flow.

A run takes about 16 minutes, of which the flows take about 12. Maestro's screenshots and logs, logcat and tombstones are kept as artifacts for 14 days.

Testing

  • CI: All 13 flows pass on GitHub. That is one green run so far, so flakiness is still unknown.
  • Locally on Android 17 (arm64) at 1080×2400, with the dependencies from Update dependencies, Gradle and CI actions #382: All flows passed two full runs. The two flows added before the second run also passed on their own.
  • Failure detection: known-bugs/back_navigation fails as intended. It documents the Back button bug described in Update dependencies, Gradle and CI actions #382 and moves into the regular run together with a fix. run-ui-tests.sh passes a failing test run on as its exit code (checked with stubs).
  • manual/chat_send: It passed its one run: the message reached the server and appeared in the app.

This branch builds master's dependencies, so the CI setup hasn't run with #382 yet. The two PRs don't touch the same files, and the first push to master with both will run that combination.

Not covered: zoom, activity recreation with "Don't keep activities", languages and external links. The dark mode and GPX flows check that the app keeps working, not how it looks.

🤖 Generated with Claude Code

Run 13 flows against the release APK on an emulator for every push to
master and every pull request. They use the production server, so each
run publishes a test rider in the open Atlantic for a few minutes.

Posting to the public chat is not part of CI and only runs by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stephanlindauer
stephanlindauer requested a review from a team as a code owner November 1, 2026 16:36
Stephan Lindauer and others added 6 commits November 1, 2026 17:42
The command line tools preinstalled on the runner don't have pixel_8 yet.
pixel_6 has the same screen resolution.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Maestro's device server died on the emulator with two cores. Use the same
number of cores as the local runs where the flows pass, and keep Maestro's
debug output and logcat as artifacts to make failures diagnosable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the 37.0 x86_64 image Android restarts itself shortly after the map is
reopened, which kills Maestro's device server. Also keep more logcat and
the tombstones, so a crash like this can be traced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
System UI is slow with software rendering. Its "isn't responding" dialog
blocked every flow, although the app itself was fine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
System apps stopped responding because everything renders in software.
Use a smaller screen, and make hiding error dialogs take effect by
triggering a configuration change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant