MSTest: #124's process-exit safety net races vstest teardown — finalize pipeline often never completes

[xping-admin]

Follow-up to #124. The fix there (process-exit safety net in Xping.Sdk.MSTest/XpingContext.cs, mirroring the xUnit adapter's HandleTestAssemblyFinished) reliably detects and logs the missing-[AssemblyCleanup] condition, but empirical testing shows it frequently does not prevent the underlying data loss when run via dotnet test. The safety net fires, but the finalize-and-upload work usually loses a race against vstest's teardown of the testhost process.

Summary

• AppDomain.CurrentDomain.ProcessExit does fire reliably in the exact repro from MSTest: sessions silently lost with [assembly: Parallelize(Scope = MethodLevel)] — [AssemblyCleanup] never runs, nothing is flushed or uploaded #124 (method-level [Parallelize]), and the warning log ("Process exiting with session ... still active ... Finalizing now as a safety net") prints every time.
• A bare, minimal HttpClient POST issued directly from that ProcessExit handler completes successfully in ~10ms — proving the handler itself gets a real execution window and that async/await + thread-pool continuations are not fundamentally broken during this shutdown.
• But the actual XpingContext.FinalizeAndShutdownAsync() → FinalizeSessionAsync() pipeline (environment detection, PR-context detection, DI resolution, the Polly-resilience-wrapped IXpingUploader, etc.) never completed in any of several real dotnet test runs — no success log, no error log, no exception, nothing. The process is torn down mid-flight.
• Net effect: the safety net upgrades silent, undiagnosable data loss into a logged but often still-lost session. It's a real improvement (the warning alone turns this from invisible to visible in CI logs) but not a complete fix for the data-loss part of MSTest: sessions silently lost with [assembly: Parallelize(Scope = MethodLevel)] — [AssemblyCleanup] never runs, nothing is flushed or uploaded #124.

Reproduction

Same setup as #124:

[assembly: Parallelize(Workers = 4, Scope = ExecutionScope.MethodLevel)]

XPING_APIKEY=test XPING_APIENDPOINT=http://127.0.0.1:8787/v1 \
  dotnet test samples/SampleApp.MSTest/SampleApp.MSTest.csproj --logger "console;verbosity=detailed"

against a local stub HTTP listener on 127.0.0.1:8787 (any listener that returns 200 with a small JSON body).

Evidence

Instrumented XpingContext.cs temporarily with File.AppendAllText checkpoints (bypassing the logger, so the marker file survives even if the process dies before flushing console output) at each stage of the finalize sequence, then ran the repro three separate times.

Isolated control test (no MSTest/vstest involved — a throwaway console app that calls Environment.Exit(0) from a background thread and does an async HttpClient POST inside its own AppDomain.ProcessExit handler): completes every time, ~10ms round trip. This rules out "async work inside ProcessExit is fundamentally unreliable in this .NET runtime" as an explanation.

Real dotnet test runs, with checkpoints inside FinalizeOnProcessExit / FinalizeAndShutdownAsync:

Checkpoint	Reached?
start (handler entered)	✅ every run
raw/bare HttpClient POST to stub server, issued directly from the handler	✅ completes, ~10ms, confirmed received server-side
before FinalizeAsync	✅ every run
after FinalizeAsync	❌ never reached in any run
before ShutdownAsync / after ShutdownAsync	❌ never reached (never got past FinalizeAsync)

So execution enters XpingContextOrchestrator.FinalizeSessionAsync() and never returns — no exception is logged, no local .xping/runs/*.jsonl.gz file is written (confirmed by diffing .xping/runs/ before/after each run), and in most runs the stub server received zero or only a partial/incomplete request (one early run recorded a truncated 2-byte body on a single POST, suggesting the connection was cut mid-request rather than the request never starting).

This points at something inside the real orchestrator pipeline — as opposed to a bare HTTP call — taking long enough that it loses the race against vstest killing the testhost process. Candidates not yet isolated further (would need instrumentation inside Xping.Sdk.Core, out of scope for the MSTest-only fix in #124):

• IEnvironmentDetector.BuildEnvironmentInfoAsync() (CI/git/environment detection)
• IPullRequestContextDetector.Detect() / PR-context detection
• DI container resolution cost the first time these services are touched via this code path
• The Polly resilience pipeline wrapping the upload HttpClient (retry/circuit-breaker warm-up overhead vs. a bare HttpClient)
• IHost.DisposeAsync() inside ShutdownAsync() (generic host shutdown, default HostOptions.ShutdownTimeout) — though this wasn't even reached, so it's not the culprit for these runs specifically

Why this matters

vstest's testhost teardown appears to give very little grace period once it decides the run is complete — on the order of low tens of milliseconds, based on the bare-POST timing above. Any safety net that goes through the full production finalize path (environment detection + PR detection + resilience-wrapped upload + host disposal) is very unlikely to finish in that window. A safety net that has a realistic chance of completing would need to be substantially cheaper than the normal finalize path, e.g.:

• Skip environment/PR re-detection and reuse whatever was already captured at session start.
• Issue a minimal, direct upload (or just persist to the local .xping/runs store, which is far cheaper than a network round trip and matches the "at least don't lose data locally" bar) instead of routing through the full IXpingUploader + Polly pipeline.
• Investigate whether AssemblyLoadContext.Unloading fires with more lead time than ProcessExit under vstest specifically (untested here — Xping.Sdk.MSTest targets netstandard2.0, where AssemblyLoadContext isn't available without additional multi-targeting work).

Scope note

This is intentionally filed separately from #124: the fix there is correct and complete for what it targets (turning total silence into a logged, diagnosable event, mirroring the xUnit adapter's existing safeguard pattern), and is scoped to Xping.Sdk.MSTest. Actually preventing the data loss itself would mean either making the safety net cheaper (MSTest-scoped) or changing the shared finalize pipeline in Xping.Sdk.Core (cross-framework, higher risk, needs its own design discussion) — both beyond what #124 asked for.

Environment: MSTest.TestFramework/MSTest.TestAdapter 3.7.2, Microsoft.NET.Test.Sdk 17.12.0, net10.0, macOS (darwin 25.5.0). Classic VSTest adapter (not the newer Microsoft.Testing.Platform runner) — confirmed via samples/SampleApp.MSTest/SampleApp.MSTest.csproj.