Skip to content

ui-tests.md

Latest commit

 

History

History
282 lines (212 loc) · 12.6 KB

ui-tests.md

File metadata and controls

282 lines (212 loc) · 12.6 KB
Raw

UI tests framework

PowerToys provides UI-test frameworks for modules and Settings. New tests should use Microsoft.PowerToys.UITest.Next, which drives Windows UI Automation through winappcli and runs as a Microsoft.Testing.Platform executable. The legacy Microsoft.PowerToys.UITest framework uses WinAppDriver/Selenium and remains documented for existing suites and migration baselines.

Agent-assisted workflows

Two repository skills cover the complete implementation and validation loop:

  • UI-tests migration skill: create new .Next test projects, port legacy WinAppDriver tests, design stable selectors/waits/lifecycle, and prepare tests for CI.
  • Local-VM UI-tests skill: create persistent Windows 10 and Windows 11 Hyper-V guests, stage current build/test artifacts, execute tests in a standard-user interactive desktop, and collect durable TRX/log/screenshot/video evidence.

For new or migrated tests, use both skills. Build first, then use the local VMs as the default live agentic loop: run one deterministic test, diagnose and fix it, and finally widen to the complete module suite on both supported Windows versions.

Module-specific constraints are documented with the module; for example, see the PowerRename UI-test notes for command-line selection, Boost engine lifetime, and signed shell-extension requirements. Read the Environment Variables safety notes before running that suite: it temporarily replaces the real User PATH and must run only in a disposable VM, not on a working machine.

Before running tests

.Next tests

  • Build the PowerToys runtime and .UITests.Next test executable.
  • Install the pinned winappcli runtime or set WINAPP_CLI_PATH. The pipeline helper is .pipelines/InstallWinAppCli.ps1.
  • Use a live interactive desktop. UIA, foreground input, Explorer, hotkeys, and rendering do not work in session 0.
  • Exit an existing PowerToys instance before a host-desktop run. The harness owns the runner and module lifecycle.

Legacy tests

Running tests

.Next tests

Build the focused project with the repository script, then run the produced Microsoft.Testing.Platform executable directly:

tools\build\build.cmd `
  -Path src\modules\<Module>\Tests\<Module>.UITests.Next `
  -Platform x64 `
  -Configuration Debug

$exe = 'x64\Debug\tests\<Module>.UITests.Next\net10.0-windows10.0.26100.0\<Module>.UITests.Next.exe'
& $exe `
  --filter 'TestCategory=<Module>' `
  --report-trx `
  --report-trx-filename module.trx `
  --results-directory .\TestResults\<Module> `
  --timeout 7m

Use explicit filter properties such as Name=, Name~, FullyQualifiedName~, or TestCategory=. A bare display name can select zero tests. The 7m timeout above is a focused-filter example; choose a larger value for a module or project-wide run.

Legacy tests

  • Exit PowerToys if it's running.

  • Open PowerToys.slnx in Visual Studio and build the solution.

  • Run tests in the Test Explorer (Test > Test Explorer or Ctrl+E, T).

Running .Next tests in persistent local VMs

The supported local backend is a pair of persistent Hyper-V guests driven through PowerShell Direct: Windows 10 and Windows 11, each with an already logged-on standard-user desktop. The VMs reveal first-run, profile, Explorer, WebView2, foreground, and lifecycle assumptions without modifying the host profile, while retaining staged payloads for a fast edit/build/rerun loop.

One-time host setup

Scaffold a VM root outside the repository, then follow the generated next steps to create the untracked configuration:

pwsh .github\skills\ui-tests-local-vm\scripts\Initialize-LocalVm.ps1 `
  -DestinationRoot C:\PowerToysUiTestVm

pwsh .github\skills\ui-tests-local-vm\scripts\Initialize-LocalVmHost.ps1 `
  -VmRoot C:\PowerToysUiTestVm `
  -CheckOnly

If -CheckOnly reports IsReady=false, a human must run the elevated setup command it prints. Hyper-V group membership, the DPAPI-protected guest administrator credential, and guest creation cannot be completed by an agent. See the setup reference for install media, vm.config.psd1, Windows 10/11 guest creation, and baseline checkpoints.

Run the agentic loop

Create a module exchange containing ui-tests.zip, powertoys-runtime.zip, winappcli.zip, and dotnet-runtime.zip as described in the agentic-loop reference. Payloads are extracted to guest-local storage; tests are never run directly from a host share.

$vmRoot = 'C:\PowerToysUiTestVm'
$exchange = "$vmRoot\shared\PowerToysUiTests\<Module>"

pwsh .github\skills\ui-tests-local-vm\scripts\Invoke-LocalVmUiTest.ps1 `
  -VmName PowerToysUiTest-Win11 `
  -ConfigurationPath "$vmRoot\vm.config.psd1" `
  -VmRoot $vmRoot `
  -ExchangeRoot $exchange `
  -TestExecutable '<Module>.UITests.Next.exe' `
  -Filter 'Name=<Module>.FocusedTest' `
  -Platform x64Win11 `
  -BuildLabel (git rev-parse HEAD) `
  -SuiteTimeout 15m `
  -TimeoutMinutes 25 `
  -ReuseStagedPayload

The controller starts the guest if needed, validates the standard-user token, Explorer session, and desktop size, then runs the test through a limited interactive scheduled task. It streams progress and returns status.json, TRX counters, per-test failures, logs, screenshots, and retained failure recordings under <ExchangeRoot>\LocalVmResults\<runId>.

After each source change, rebuild and replace only the changed archive, then rerun the same focused filter with -ReuseStagedPayload. Widen only after that behavior is understood. A module is complete only after the full category filter passes with executed == total on both Windows 10 and Windows 11; restore the baseline checkpoint for the final clean-profile confirmation. See the local-VM troubleshooting guide for desktop, PowerShell Direct, payload, shell-extension signing, and evidence failures.

Running tests in pipeline

The PowerToys UI test pipeline provides flexible options for building and testing:

Pipeline Options

  • buildSource: Select the build type for testing:

    • latestMainOfficialBuild: Downloads and uses the latest official PowerToys build from main branch
    • buildNow: Builds PowerToys from current source code and uses it for testing
    • specificBuildId: Downloads a specific PowerToys build using the build ID specified in specificBuildId parameter

    Default value: latestMainOfficialBuild

  • specificBuildId: When buildSource is set to specificBuildId, specify the exact PowerToys build ID to download and test against.

    Default value: "xxxx" (placeholder, enter actual build ID when using specificBuildId option)

    When to use this:

    • Testing against a specific known build for reproducibility
    • Regression testing against a particular build version
    • Validating fixes in a specific build before release

    Usage: Enter the build ID number (e.g., 12345) to download that specific build. Only used when buildSource is set to specificBuildId.

  • uiTestModules: Specify which UI test modules to build and run. This parameter controls both the .csproj projects to build and the .dll test assemblies to execute. Examples:

    • ['UITests-FancyZones'] - Only FancyZones UI tests
    • ['MouseUtils.UITests'] - Only MouseUtils UI tests
    • ['UITests-FancyZones', 'MouseUtils.UITests'] - Multiple specific modules
    • Leave empty to build and run all UI test modules

    Important: The uiTestModules parameter values must match both the test project names (for .csproj selection during build) and the test assembly names (for .dll execution during testing).

Build Modes

  1. Official Build Testing (buildSource = latestMainOfficialBuild or specificBuildId)

    • Downloads and installs official PowerToys build (latest from main or specific build ID)
    • Builds only UI test projects (all or specific based on uiTestModules)
    • Runs UI tests against installed PowerToys
    • Tests both machine-level and per-user installation modes automatically

  2. Current Source Build Testing (buildSource = buildNow)

    • Builds entire PowerToys solution from current source code
    • Builds UI test projects (all or specific based on uiTestModules)
    • Runs UI tests against freshly built PowerToys
    • Uses artifacts from current pipeline build

Note: All modes support the uiTestModules parameter to control which specific UI test modules to build and run. Both machine-level and per-user installation modes are tested automatically when using official builds.

Pipeline Access

How to add the first UI tests for your modules

Use the UI-tests migration skill for new .Next projects and ports. It contains the current executable project scaffold, API mapping, naming, CI-stability checklist, and validated examples.

The project sample below describes the legacy WinAppDriver framework and is retained for existing legacy suites. Do not use it as the starting point for a new .Next project.

  • Follow the naming convention: {ModuleFolder}/Tests/{ModuleName}-{TestType(Fuzz/UI/Unit)}Tests

  • Create a new project and add the following references to the project file. Change the OutputPath to your own module's path.

      <Project Sdk="Microsoft.NET.Sdk">
  <!-- Look at Directory.Build.props in root for common stuff as well -->
  <Import Project="..\..\..\Common.Dotnet.CsWinRT.props" />

  <PropertyGroup>
      <ProjectGuid>{4E0AE3A4-2EE0-44D7-A2D0-8769977254A0}</ProjectGuid>
      <RootNamespace>PowerToys.Hosts.UITests</RootNamespace>
      <AssemblyName>PowerToys.Hosts.UITests</AssemblyName>
      <IsPackable>false</IsPackable>
      <IsTestProject>true</IsTestProject>
      <Nullable>enable</Nullable>
      <OutputType>Library</OutputType>

      <!-- This is a UI test, so don't run as part of MSBuild -->
      <RunVSTest>false</RunVSTest>
      </PropertyGroup>
      <PropertyGroup>
      <OutputPath>$(SolutionDir)$(Platform)\$(Configuration)\tests\Hosts.UITests\</OutputPath>
      </PropertyGroup>

      <ItemGroup>
      <PackageReference Include="MSTest" />
      <ProjectReference Include="..\..\..\common\UITestAutomation\UITestAutomation.csproj" />
      </ItemGroup>
  </Project>

  • Inherit your test class from UITestBase.

    Set Scope: The default scope starts from the PowerToys settings UI. If you want to start from your own module, set the constructor as shown below:

    Specify Scope:

      [TestClass]
  public class HostModuleTests : UITestBase
  {
      public HostModuleTests()
          : base(PowerToysModule.Hosts, WindowSize.Small_Vertical)
      {
      }
  }

  • Then you can start performing the UI operations.

Example

[TestMethod("Hosts.Basic.EmptyViewShouldWork")]
[TestCategory("Hosts File Editor #4")]
public void TestEmptyView()
{
    this.CloseWarningDialog();
    this.RemoveAllEntries();

    // 'Add an entry' button (only show-up when list is empty) should be visible
    Assert.IsTrue(this.HasOne<HyperlinkButton>("Add an entry"), "'Add an entry' button should be visible in the empty view");

    VisualAssert.AreEqual(this.TestContext, this.Find("Entries"), "EmptyView");

    // Click 'Add an entry' from empty-view for adding Host override rule
    this.Find<HyperlinkButton>("Add an entry").Click();

    this.AddEntry("192.168.0.1", "localhost", false, false);

    // Should have one row now and not more empty view
    Assert.IsTrue(this.Has<Button>("Delete"), "Should have one row now");
    Assert.IsFalse(this.Has<HyperlinkButton>("Add an entry"), "'Add an entry' button should be invisible if not empty view");

    VisualAssert.AreEqual(this.TestContext, this.Find("Entries"), "NonEmptyView");
}

Extra tools and information

Accessibility Tools: While working on tests, you may need a tool that helps you to view the element's accessibility data, e.g. for finding the button to click. For this purpose, you could use AccessibilityInsights.