Skip to content

[tests] Add infrastructure to run monotouch-test inside app extensions - #26265

Draft
rolfbjarne wants to merge 15 commits into
mainfrom
dev/rolf/static-registrar-appextensions
Draft

[tests] Add infrastructure to run monotouch-test inside app extensions#26265
rolfbjarne wants to merge 15 commits into
mainfrom
dev/rolf/static-registrar-appextensions

Conversation

@rolfbjarne

@rolfbjarne rolfbjarne commented Jul 23, 2026

Copy link
Copy Markdown
Member

TODO:

  • Early prototype, no reviewing necessary.

Early prototype that runs the monotouch-test suite from inside an app
extension (an AUv3 audio-unit extension), so we can exercise the test
suite in the more constrained app-extension environment (e.g. to validate
the trimmable static registrar there).

Highlights:

  • A new audio-unit AUv3 extension test harness under
    tests/monotouch-test/dotnet/extensions/audio-unit/, with a container
    app that instantiates the extension out-of-process and per-platform
    projects (macOS/iOS/tvOS/MacCatalyst) that share almost all logic via
    AppExtension-shared.csproj.

  • Test results are streamed back over a localhost TCP connection, reusing
    the existing Touch.Client reporting infrastructure. The run script
    (scripts/run-audio-unit-extension-tests) opens a TcpListener, configures
    the sandboxed extension through NSUserDefaults (defaults write), and
    reads the NUnit XML result back over the socket.

  • The extension is configured entirely through NSUserDefaults, which — unlike
    environment variables set on the container host — reliably reach the
    OS-spawned, sandboxed extension process.

  • Runtime/registrar/configuration variations are selected with
    TEST_VARIATION (consistent with the rest of the test infrastructure).

Still an early prototype; xharness/CI wiring is proposed but not yet
included here.

🤖 Pull request created by Copilot

@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@rolfbjarne rolfbjarne mentioned this pull request Jul 30, 2026
Draft
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
rolfbjarne and others added 5 commits September 1, 2026 19:30
@rolfbjarne
[tests] Add infrastructure to run monotouch-test inside app extensions
c96f0f9 
Add a test harness that runs the monotouch-test NUnit suite inside an
Audio Unit (AUv3) app extension process, verifying that .NET code works
correctly in the extension sandbox.

Key changes:

- New test projects: AudioUnitExtension and SpotlightImportExtension
  (tests/dotnet/) with ContainerApp + AppExtension layout for iOS and
  macOS.
- Convert ExtensionProject from Share/Thumbnail to QuickLook Preview
  (iOS) and Audio Unit (macOS) for more reliable triggering.
- Runner script (scripts/run-audio-unit-extension-tests/) that builds,
  registers, launches the host, instantiates the AU out-of-process,
  runs NUnit tests inside the extension, and collects results.
- Makefiles (tests/monotouch-test/dotnet/extensions/audio-unit/) with
  run-monovm / run-coreclr targets and TEST_FILTER support.
- TouchRunner modifications to support extension-hosted execution.
- Test for issue #24869: extensions with managed-static registrar.

Fixes #24869
@rolfbjarne
[tests] Move ExtensionTestRunner into Touch.Client and use NativeHand…
85404c3 
…le ctors

Move ExtensionTestRunner from the monotouch-test project into the shared
Touch.Client runner (namespace MonoTouch.NUnit.UI), so it can be reused by any
Touch.Unit-based test app. Since the shared runner can't reference the per-app
generated TestLoader, CreateHeadlessRunner now takes the test assemblies as an
argument, and the caller passes TestLoader.GetTestAssemblies ().

Also change the app extension `(IntPtr handle)` base constructors to
`(NativeHandle handle)` across the extension test projects.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3f83ba49-2d01-4cee-b4ca-8fbb6912c597
@rolfbjarne
[tests] Consolidate the audio-unit extension Info.plist and entitleme…
d5e1de3 
…nts into the csproj

Set AssemblyName=monotouchtest so the extension's native executable and bundle
name match the container app conventions, which lets BundleTest use the same
expected executable name on all platforms again.

Replace the six per-platform Info.plist/Entitlements.plist files with:
 * a single shared AppExtension-Info.plist (referenced via PartialAppManifest)
   containing only the keys with no MSBuild property equivalent (NSExtension,
   CFBundleDisplayName/CFBundleName).
 * CustomEntitlements items in AppExtension-shared.csproj (the network client/server
   entitlements come from the shared monotouch-test project).

The remaining Info.plist keys (bundle identifier, version, package type, etc.) are
derived automatically by CompileAppManifest from the csproj properties.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3f83ba49-2d01-4cee-b4ca-8fbb6912c597
@rolfbjarne
[tests] Select the audio-unit extension runtime/registrar with TEST_V…
60e3982 
…ARIATION

Remove the CoreCLR/MonoVM split (the XAMARIN_RUNTIME variable and the
build/run-monovm/coreclr convenience targets) from the audio-unit extension
makefiles, and select the runtime, registrar and configuration using
TEST_VARIATION instead (the same mechanism the other .NET test projects use, see
tests/common/test-variations.csproj). The per-platform makefiles now just include
the shared makefile.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3f83ba49-2d01-4cee-b4ca-8fbb6912c597
@rolfbjarne
[tests] Stream audio-unit extension test results over localhost TCP
be42346 
Rework the audio-unit extension test harness to report results the same
way the regular Touch.Unit test apps do: over a localhost TCP connection.

* The run script (run-audio-unit-extension-tests) now opens a TcpListener
  on a free 127.0.0.1 port, configures the extension via NSUserDefaults
  (network.enabled/host.name/host.port/transport=TCP + xml.enabled), and
  reads the NUnit XML result stream back over the socket (until the
  '<!-- the end -->' marker). Pass/fail is parsed from the <test-results>
  element and the raw NUnit XML is persisted for CI consumption. The
  system log is still captured, but for diagnostics only.

* Subprocess launches use Xamarin.Utils.Execution (tools/common/Execution.cs)
  instead of hand-rolled Process helpers.

* The extension's test filter now comes solely from NSUserDefaults
  (test.name); the environment-variable, bundle-file and Info.plist
  fallbacks are removed, along with the build-time MonotouchExtensionTestName
  property.

This relies on the empirically-verified fact that 'defaults write
<extension-bundle-id> ...' reaches the sandboxed extension's
NSUserDefaults.StandardUserDefaults, whereas environment variables set on
the container host do not propagate to the OS-spawned extension process.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3f83ba49-2d01-4cee-b4ca-8fbb6912c597
@rolfbjarne
rolfbjarne force-pushed the dev/rolf/static-registrar-appextensions branch from 82f6494 to be42346 Compare September 2, 2026 07:48
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@rolfbjarne
[tests] Preserve monotouch-test bundle name
365a05b 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@rolfbjarne
[tests] Run app-hosted tests on the main thread by default
30d09c3 
Restore the test runner default changed by the app-extension work so UIKit tests execute on the UI thread, while retaining the opt-out property for specialized runners.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
rolfbjarne and others added 5 commits September 8, 2026 12:22
@rolfbjarne
Fix nullability
03a4323
@rolfbjarne
Nullability fixes.
4304ea1
@rolfbjarne
[tests] Run monotouch-test app extensions in CI
16a3e8b 
Add dedicated xharness tasks for the macOS and Mac Catalyst audio-unit extension hosts. The tasks follow the existing monotouch platform selection, exercise MonoVM and CoreCLR managed-static registrar configurations, and run serially because extension registration is machine-global.

Allow the extension runner to write NUnit output directly to a CI-selected vsts-prefixed path, and make the run target build the app before launching it. Clear the platform job's category TEST_FILTER because the extension runner currently accepts an exact test name instead of a category expression.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rolfbjarne
[tests] Add app extension CI variations
fd6a507 
Add a dedicated appextensions label and platform-split CI matrix entries for iOS, tvOS, macOS, and Mac Catalyst. Run the CoreCLR trimmable-static registrar variation without MonoVM coverage, and use a 10-minute extension result timeout with a larger watchdog for the encompassing clean build and cleanup.

Add simulator installation, launch, preferences, result collection, and lifecycle handling. Make TCP result reporting safe when UIApplication.SharedApplication is unavailable inside an app extension.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 9bf659bd-8b21-4e13-8d63-553db1de36c9
@rolfbjarne
[tests] Run app extension tests with TV Top Shelf
2618a84 
Use the modern com.apple.tv-top-shelf extension point for the tvOS variation because tvOS does not support Audio Unit app extensions. The container invalidates top-shelf content, which causes tvOS to instantiate the registered TVTopShelfContentProvider and run the shared monotouch-test extension host.

Keep the existing Audio Unit extension and activation path for iOS, macOS, and Mac Catalyst.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 9bf659bd-8b21-4e13-8d63-553db1de36c9
@rolfbjarne
[ci] Consolidate app extension test job
9e54968 
Run the iOS, tvOS, macOS, and Mac Catalyst app-extension variations from a single appextensions matrix entry instead of creating a separate CI job for each platform.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 9bf659bd-8b21-4e13-8d63-553db1de36c9
@rolfbjarne
Merge remote-tracking branch 'origin/main' into dev/rolf/static-regis…
1743db5 
…trar-appextensions
@vs-mobiletools-engineering-service2

vs-mobiletools-engineering-service2 commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

✅ API diff for current PR / commit

NET (empty diffs)

✅ API diff vs stable

NET (empty diffs)

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: 1743db515845510fb3bcf6ceaf61afa69951bf54 [PR build]

@vs-mobiletools-engineering-service2

vs-mobiletools-engineering-service2 commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

🔥 [CI Build #1743db5] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

25 tests crashed, 1 tests failed, 17 tests passed.

Failures

❌ appextensions tests

🔥 Failed catastrophically on VSTS: test results - appextensions (no summary found).

Html Report (VSDrops) Download

❌ assembly-processing tests

🔥 Failed catastrophically on VSTS: test results - assembly-processing (no summary found).

Html Report (VSDrops) Download

❌ cecil tests

🔥 Failed catastrophically on VSTS: test results - cecil (no summary found).

Html Report (VSDrops) Download

❌ dotnettests tests (iOS)

🔥 Failed catastrophically on VSTS: test results - dotnettests_ios (no summary found).

Html Report (VSDrops) Download

❌ dotnettests tests (MacCatalyst)

🔥 Failed catastrophically on VSTS: test results - dotnettests_maccatalyst (no summary found).

Html Report (VSDrops) Download

❌ dotnettests tests (macOS)

🔥 Failed catastrophically on VSTS: test results - dotnettests_macos (no summary found).

Html Report (VSDrops) Download

❌ dotnettests tests (Multiple platforms)

🔥 Failed catastrophically on VSTS: test results - dotnettests_multiple (no summary found).

Html Report (VSDrops) Download

❌ dotnettests tests (tvOS)

🔥 Failed catastrophically on VSTS: test results - dotnettests_tvos (no summary found).

Html Report (VSDrops) Download

❌ framework tests

🔥 Failed catastrophically on VSTS: test results - framework (no summary found).

Html Report (VSDrops) Download

❌ fsharp tests

🔥 Failed catastrophically on VSTS: test results - fsharp (no summary found).

Html Report (VSDrops) Download

❌ generator tests

🔥 Failed catastrophically on VSTS: test results - generator (no summary found).

Html Report (VSDrops) Download

❌ interdependent-binding-projects tests

🔥 Failed catastrophically on VSTS: test results - interdependent-binding-projects (no summary found).

Html Report (VSDrops) Download

❌ introspection tests

🔥 Failed catastrophically on VSTS: test results - introspection (no summary found).

Html Report (VSDrops) Download

❌ linker tests (iOS)

🔥 Failed catastrophically on VSTS: test results - linker_ios (no summary found).

Html Report (VSDrops) Download

❌ linker tests (MacCatalyst)

🔥 Failed catastrophically on VSTS: test results - linker_maccatalyst (no summary found).

Html Report (VSDrops) Download

❌ linker tests (macOS)

🔥 Failed catastrophically on VSTS: test results - linker_macos (no summary found).

Html Report (VSDrops) Download

❌ linker tests (tvOS)

🔥 Failed catastrophically on VSTS: test results - linker_tvos (no summary found).

Html Report (VSDrops) Download

❌ monotouch tests (iOS)

🔥 Failed catastrophically on VSTS: test results - monotouch_ios (no summary found).

Html Report (VSDrops) Download

❌ monotouch tests (MacCatalyst)

🔥 Failed catastrophically on VSTS: test results - monotouch_maccatalyst (no summary found).

Html Report (VSDrops) Download

❌ monotouch tests (macOS)

🔥 Failed catastrophically on VSTS: test results - monotouch_macos (no summary found).

Html Report (VSDrops) Download

❌ monotouch tests (tvOS)

🔥 Failed catastrophically on VSTS: test results - monotouch_tvos (no summary found).

Html Report (VSDrops) Download

❌ msbuild tests

🔥 Failed catastrophically on VSTS: test results - msbuild (no summary found).

Html Report (VSDrops) Download

❌ sharpie tests

🔥 Failed catastrophically on VSTS: test results - sharpie (no summary found).

Html Report (VSDrops) Download

❌ windows tests

1 tests failed, 2 tests passed.

Failed tests

  • Remote .NET tests/Xamarin.Tests.DotNetProjectTest.AppWithLibraryWithResourcesReferenceOnRemoteWindows(iOS,"ios-arm64",True): Failed: 'dotnet build' failed with exit code 1

Html Report (VSDrops) Download

❌ xcframework tests

🔥 Failed catastrophically on VSTS: test results - xcframework (no summary found).

Html Report (VSDrops) Download

❌ xtro tests

🔥 Failed catastrophically on VSTS: test results - xtro (no summary found).

Html Report (VSDrops) Download

Successes

macOS tests

✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 1743db515845510fb3bcf6ceaf61afa69951bf54 [PR build]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dalexsoto dalexsoto Awaiting requested review from dalexsoto dalexsoto will be requested when the pull request is marked ready for review dalexsoto is a code owner
Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

copilot

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rolfbjarne @vs-mobiletools-engineering-service2