Skip to content

Run the SOS test harness on Helix - #6002

Open
steveisok wants to merge 4 commits into
steveisok-sos-coverage-migrationfrom
steveisok-sos-helix-integration
Open

Run the SOS test harness on Helix#6002
steveisok wants to merge 4 commits into
steveisok-sos-coverage-migrationfrom
steveisok-sos-helix-integration

Conversation

@steveisok

@steveisok steveisok commented Sep 3, 2026

Copy link
Copy Markdown
Member

Depends on #6001

This is the top (layer 4) PR in the SOS harness stack. It adds the remaining SOS.Tests Helix integration on top of the coverage-migration layer without registering stack metadata.

Changes

  • Shard the matrix deterministically by capture family using a stable hash. The key includes target, flavor, runtime version, GC type, dump kind, and liveness while deliberately excluding Host and DAC so dump reuse remains local to one shard.
  • Stage one shared correlation payload per OS/RID/configuration and execute named work items: 8 dump + 2 live generally, and 32 dump + 2 live on macOS.
  • Add Windows and Unix launchers with writable runtime/native/executable overlays, bounded macOS parallelism, MTP result formats, and uploaded logs, replays, LLDB traces, and host crash diagnostics.
  • Translate HELIX_WORKITEM_UPLOAD_ROOT in launchers to the harness-owned SOSHARNESS_UPLOAD_ROOT; production harness and test code do not read HELIX_* variables.
  • Submit SOS.Tests through the existing Helix template for Windows, Linux x64/arm64, Alpine/musl, and macOS x64/arm64 while preserving all non-SOS local tests and skipping only duplicate local SOS.Tests execution.
  • Preserve the lower-layer validity/capability filters, .NET 11 SingleFile+cDAC support, command-coverage evidence, and current Arcade dependency version.

Validation

  • Built SOS.TestHarness and SOS.Tests directly.
  • Passed 42 focused sharding, validity, capability, and payload tests.
  • Verified an empty macOS-style shard is skipped rather than failing theory discovery.
  • Parsed changed XML, JSON, and YAML files; validated Unix launcher syntax and executable mode.
  • Evaluated PrintSOSHelixWorkItems without submission for Windows x64, Linux x64, Alpine/musl x64, macOS x64, and macOS arm64, including a single-work-item smoke selection.
  • Verified 10 non-macOS and 34 macOS work items, one shared correlation payload, expected names/commands, payload closure, queues, and pipeline mappings.
  • Completed a cumulative code review against 0a60d3a638186671a8862110c25d27499931742e and fixed all high-confidence findings.

@steveisok
steveisok requested a review from a team as a code owner September 3, 2026 00:25
@steveisok steveisok mentioned this pull request Sep 3, 2026
Closed
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from 3ac4ddb to df3d87e Compare September 3, 2026 12:31
@steveisok steveisok mentioned this pull request Sep 4, 2026
Merged
Copilot AI lite review requested due to automatic review settings September 4, 2026 02:12
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from df3d87e to c6775f5 Compare September 4, 2026 02:12
Copilot AI reviewed Sep 4, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The new Helix MSBuild project has a payload-staging target gated by a likely incorrect Helix queue property name, which can prevent correlation payload staging and break Helix submissions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 Medium severity

New issues introduced by this change (1)
Severity Finding
Medium severity eng/​helix/​SOS.Tests.Helix.proj — PrepareSOSHelixPayload is currently conditioned on the MSBuild property HelixTargetQueue, but the…
What changed in this PR

This PR integrates SOS.Tests with Helix execution across Windows, Linux (including musl), and macOS by adding deterministic sharding and Helix payload/launcher infrastructure, while keeping local CI test coverage intact (skipping only duplicate local SOS.Tests runs when Helix submission is enabled).

Changes:

  • Add capture-family-based deterministic sharding (stable hash) and shard filtering throughout the SOS test matrix.
  • Introduce Helix submission project + OS-specific launchers that stage a shared correlation payload and run named shard work items, with writable overlays for executables/runtime/native bits and enriched artifact/log uploads.
  • Update pipeline templates to optionally submit SOS.Tests to Helix (including Helix-only legs for cross-built architectures) while preserving other local test execution.
File Description
src/​tests/​SOS.Tests/​WindowsTheoryAttribute.cs Allow empty theory data when matrix partitioning/sharding is in effect.
src/​tests/​SOS.Tests/​SosTheoryAttribute.cs Same empty-matrix behavior adjustment for cross-platform theories.
src/​tests/​SOS.Tests/​SosReplayAttribute.cs Route replay output under a harness-owned upload root when provided (Helix-friendly).
src/​tests/​SOS.Tests/​TestMatrices.cs Apply shard filtering consistently; introduce unsharded helpers for composition.
src/​tests/​SOS.Tests/​TestConfigShardTests.cs Add unit tests for sharding/environment parsing and hash stability.
src/​tests/​SOS.Tests/​UnixPayloadTests.cs Add tests for writable overlay behavior and upload-root routing on Unix.
src/​tests/​SOS.Tests/​SOS.Tests.csproj Add a pipeline-controlled “skip SOS.Tests locally” switch; avoid chmod step when tests are skipped.
src/​tests/​SOS.Tests/​README.md Document Helix sharding and new harness environment variables and execution model.
src/​tests/​SOS.TestHarness/​TestConfig.cs Implement capture-family key, stable hash, shard selection parsing, and shard filtering entry points.
src/​tests/​SOS.TestHarness/​SnapshotStore.cs Support executable overlays and enforce “prebuilt-only” mode for Helix payload execution.
src/​tests/​SOS.TestHarness/​RepoLayout.cs Add environment-variable overrides for repo/runtime/native/executable/scratch roots and directory resolution helper.
src/​tests/​SOS.TestHarness/​HostDiagnostics.cs Add upload-root routing for host crash dump diagnostics.
src/​tests/​SOS.TestHarness/​ToolPaths.cs Add dbgeng root override and use configurable dotnet root for host runtime discovery.
global.json Add Microsoft.DotNet.Helix.Sdk MSBuild SDK reference.
eng/​helix/​SOS.Tests.Helix.proj New Helix submission project: defines shard work items and stages the correlation payload.
eng/​helix/​sos/​run-sos-tests.sh New Unix Helix work-item launcher: overlays runtime/executables/native as needed, configures LLDB, runs tests with MTP reporters.
eng/​helix/​sos/​run-sos-tests.cmd New Windows Helix work-item launcher: sets harness env, manages signature-check toggling, runs tests with MTP reporters.
eng/​helix/​sos/​debuggee-entitlements.plist New macOS entitlements used for codesigning staged debuggee executables.
eng/​pipelines/​build.yml Add parameters/logic to submit SOS.Tests to Helix and support additional-architecture builds (macOS) and helix-only legs.
diagnostics.yml Wire new Helix-enabled build/test legs for public CI (Windows/Linux/musl/macOS), including helix-only cross-arch scenarios.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread eng/helix/SOS.Tests.Helix.proj
Comment on lines +109 to +112
<Target Name="PrepareSOSHelixPayload"
DependsOnTargets="ValidateSOSHelixConfiguration"
BeforeTargets="CoreTest"
Condition="'$(HelixTargetQueue)' == ''">
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from c6775f5 to e4f72a5 Compare September 4, 2026 12:31
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from e4f72a5 to d67533d Compare September 4, 2026 12:46
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from d67533d to 76f45ff Compare September 4, 2026 13:13
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from 76f45ff to 6634824 Compare September 4, 2026 18:07
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from 6634824 to 192da26 Compare September 4, 2026 19:05
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from 192da26 to 33fe748 Compare September 4, 2026 19:49
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from 33fe748 to 01629ce Compare September 4, 2026 20:37
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from 01629ce to dba425f Compare September 4, 2026 21:25
steveisok added a commit that referenced this pull request Sep 4, 2026
@steveisok
Reapply new SOS test harness (#6008)
36ad68a 
## Summary

- Restores the complete reviewed SOS test harness from
[#5979](#5979) after its exact
revert in [#6006](#6006).
- Re-establishes this change as the new bottom layer beneath
[#5999](#5999),
[#6000](#6000),
[#6001](#6001), and
[#6002](#6002).
- Temporarily excludes only `SOS.Tests` from local CI via `SkipTests`
when `ContinuousIntegrationBuild` is true. Normal local builds and
discovery remain enabled until the upper Helix layer moves execution out
of local CI.

## Reconstruction proof

- The baseline restoration commit
`2c7616acc2be6b20cf377574fa52e40196f16ca7` has tree
`5d2ab7a6990a8cdc2ff8a3d2ec7a29e373b99d54`, exactly matching the
pre-revert commit `ec5af2a1a56c4d9348e7e92438bb2faf8684f15f`.
- The only subsequent delta is five added lines in
`src/tests/SOS.Tests/SOS.Tests.csproj` for the temporary CI-only skip.

## Validation

- `./dotnet.sh build src/tests/SOS.Tests/SOS.Tests.csproj --no-restore
--verbosity minimal` — succeeds with 0 warnings and 0 errors.
- Focused Microsoft.Testing.Platform discovery for
`SOS.Tests.PrintExceptionTests.PrintException_Data` with
DotnetDump/Core/net10 constraints — 2 expected rows discovered without
`ContinuousIntegrationBuild`.
- MSBuild property evaluation: default `SkipTests` is empty;
`-p:ContinuousIntegrationBuild=true` evaluates `SkipTests=true`.

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c34cd9f4-d3b5-4b46-b7fc-e34b34bc88c3
steveisok and others added 2 commits September 4, 2026 19:11
@steveisok
Run the SOS test harness on Helix
62580ab 
Add deterministic capture-family sharding, a shared SOS Helix payload, cross-platform launchers, and pipeline scheduling while preserving the lower harness layers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@steveisok
Scope SOS Helix payload validation
1b8cc89 
Validate the project-owned shared payload independently from SDK-added correlation payloads, and retain the Azure Pipelines reporter in CI submissions.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: 48106f1a-69fc-430d-917a-89bfe78a5f71
@steveisok
Serialize Linux ARM64 SOS dump shards
35e88f2 
Avoid deterministic native LLDB and dotnet-dump host crashes when dump tests run concurrently on Linux ARM64, while preserving existing shard counts and concurrency elsewhere.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: 48106f1a-69fc-430d-917a-89bfe78a5f71
@steveisok
Restrict Alpine Helix dump kinds
ad927d1 
Avoid intermittent .NET 8 Mini dump capture failures in the unprivileged Alpine Helix container while preserving Heap, Full, and live coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 48106f1a-69fc-430d-917a-89bfe78a5f71
@steveisok
steveisok force-pushed the steveisok-sos-helix-integration branch from dba425f to ad927d1 Compare September 4, 2026 23:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@steveisok