Skip to content

Feature: native iOS support for simulators and paired iPhone/iPad devices #174

Open

Description

@banggaoo

Feature: Native iOS support for simulators and paired iPhone/iPad devices

Problem

Artemis's Android device driver and ADB-specific execution paths cannot drive
iOS targets. Selecting iOS needs to work consistently across task submission,
device ownership, observation, actions, and recorded traces, rather than as a
standalone simulator-only integration.

Proposed Scope

Add opt-in, local iOS support on macOS with Xcode 27 or newer:

  • iOS simulators: Xcode MCP device-interaction sessions plus simctl.
  • Paired physical iPhones/iPads: devicectl for discovery/app lifecycle and a
    user-provisioned WebDriverAgent runner for accessibility observation and input.
  • CLI run/batch, embedded Python SDK, dependency-free remote SDK, daemon queue,
    MCP tools, and web console.
  • Screenshots, hierarchy, taps/swipes/text, supported hardware keys, app
    install/launch/stop, recording, replay, and live screen streaming.
  • Per-device iOS execution leases isolated from Android identifiers.
  • Remote iOS submissions reject legacy hosts that lack the platform.ios
    capability, before a task can accidentally execute on Android.

Android remains the default. Android-specific features must reject or be
excluded from iOS execution rather than reporting synthetic success.

Boundaries

  • No automatic Xcode permission grants, device trust, Developer Mode changes,
    passcode entry, or WDA signing/provisioning.
  • Physical devices require an explicit identifier and an available WDA server.
  • Physical recording uses screenshot polling, not a simulator-quality video
    stream. iOS recordings have no audio.
  • ADB shell/logcat, Android Back/Delete semantics, and verified Android app
    locking are not claimed as iOS equivalents.
  • No cloud iOS execution or new third-party runtime dependencies.

Acceptance

  • Deterministic Python suite passes without devices or real provider
    credentials: 2900 passed, 11 skipped, 8 deselected.
  • Formatting, lint, protected-core typing, and unchanged quality ratchets pass.
  • Web console tests and production build pass (139 karma tests, Chrome
    Headless; production bundle builds cleanly).
  • Built distributions include the iOS modules and pass installed-CLI
    smoke (artemis --help, iOS driver imports, platform CLI options).
  • Dedicated real simulator (iOS 27.0): boot, enumeration, app install/list/
    launch/terminate, native recordVideo to valid MP4, and clean uninstall/
    shutdown verified. With Xcode agent consent granted, the full MCP path also
    passes: real DeviceInteraction session, 248-element hierarchy, tap/swipe
    input, and clean session teardown. Without consent it refuses cleanly
    (XcodeApprovalRequiredError) and releases all resources.
  • Physical WDA observation/input/recording verified on connected hardware
    (iPhone 15 Pro, Xcode 27, LAN tunnel): WDA session via driver-launched
    runner, 319-element hierarchy, tap/swipe/home, WDA + devicectl screenshots,
    pid-verified terminate, physical recording to valid MP4, clean disconnect,
    zero orphaned processes.
  • Existing Android regressions remain green; the Android driver source is
    unchanged.
  • Android path smoke-tested on this head via AndroidAdbDriver on a live Android 36 emulator (emulator-5554): connect, real 1080x2400 screenshot, Settings launch, tap, swipe, clean disconnect. No physical handset is attached, so physical-hardware smoke remains open.

Current Verification Limits

  • No physical Android handset is attached; Android smoke ran on an API 36 emulator (see evidence below).
  • Physical coverage is one iPhone over a LAN CoreDevice tunnel; iPad and USB
    transports are not separately exercised.
  • Real provider-driven task quality is not established by driver-level
    acceptance.
  • Xcode binds agent approval to the client binary's code signature:
    unsigned/adhoc interpreters receive ~24-hour grants that need periodic
    re-approval; signed agents can hold persistent grants (documented in
    docs/ios.md).

Implementation PR: #175

Activity

  1. banggaoo commented on Oct 9, 2026

    @banggaoo
    Author

    High-level architecture — how iOS fits the system

    Implementation: feat/ios-support → PR #175.

    One platform-agnostic agent pipeline, two driver backends

                            ┌──────────── user goal (platform=ios) ─────────────┐
                            │                                                   │
            CLI ── SDK (embedded/remote) ── Admin Console ── MCP tools ─────────┘
                            │
                    daemon task queue          (platform-aware payload)
                            │
                    worker + device lock       (ios/<UDID> namespace)
                            │
                 ┌──────────┴──────────┐
                 ▼                     ▼
          ┌─────────────┐      ┌──────────────┐
          │  Simulator   │      │   Physical   │      ← same observation/action contract
          │ simctl +     │      │ devicectl +  │        → the agent layer cannot tell
          │ Xcode MCP    │      │ WebDriverAgent│       which backend it drives
          └─────────────┘      └──────────────┘
                 │                     │
                 └──────────┬──────────┘
                            ▼
         perception loop (screenshot + UI hierarchy) → agent → tool calls → actions
                            │
             recording (segmented video) → trace → replay / live stream
    

    What this means concretely

    • One feature, one flag: platform: ios on the task payload is the only switch. Everything downstream — queueing, locking, execution leases, streaming, replay — already had a place for it.
    • The agent stack is untouched: planner/operator/checker/flash see the same screenshots + element trees and emit the same tool calls. iOS only changes who executes the actions.
    • Two backends, one contract: simulator rides Xcode's own DeviceInteraction bridge; physical rides WebDriverAgent over the CoreDevice tunnel. Both produce identical observation/action semantics.
    • Android is untouched: Android remains the default; iOS targets carry no ADB endpoint; Android-only operations reject iOS explicitly instead of silently misbehaving.

    Verified end-to-end (live runs)

    Flash + Pro profiles on simulator, Flash on physical iPhone 15 Pro (WDA), multi-app flows (Safari → URL → scroll → home → Settings), recording + PASS traces — evidence in PR #175 comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions