Skip to content

Latest commit

 

History

History
444 lines (344 loc) · 23.2 KB

dev-guide.md

File metadata and controls

444 lines (344 loc) · 23.2 KB

Development Guide

This document contains all the required information to build, test, and consume MSTest.

Prerequisites

To build and test all functionalities of MSTest, we recommend installing Visual Studio 2026 with the following workloads:

  • .NET desktop development
  • Universal Windows Platform development
  • .NET Core cross-platform development

Recommended workflow

We recommend the following overall workflow when developing this repository:

  • Fork this repository.
  • Always work on your fork.
  • Always keep your fork up to date.

Starting with your fork

Before updating your fork, run this command:

git remote add upstream https://github.com/Microsoft/testfx.git

This will make management of multiple forks and your work easier over time.

Updating your fork

We recommend the following commands to update your fork:

git checkout main
git clean -dfx
git fetch upstream
git rebase upstream/main
git push

Or more succinctly:

git checkout main && git clean -xdf && git fetch upstream && git rebase upstream/main && git push

This will update your fork with the latest from microsoft/testfx on your machine and push those updates to your remote fork.

Building

An experimental native Bazel build is available for the portable .NET 8-11 development graph, selected test suites, and native MTP/TRX package consumers. It does not replace the MSBuild/Arcade shipping build or its complete framework/workload and acceptance matrix.

An experimental BuildXL build can schedule and cache the existing MSBuild projects on Windows x64. Its default entry point is TestFx.slnx, preserving project-specific target frameworks except classic UAP, which requires desktop MSBuild. Separate Debug and Release pipeline legs build the solution and retain engine counters and project binlogs for later analysis. After a successful build, each leg also runs the same 13 net10.0 unit suites that the Bazel dogfood workload runs, from the BuildXL outputs, and retains restore, build and test timings plus per-suite test-identity hashes so the Bazel and BuildXL phases can be compared; see running the matched unit suites. Outputs are isolated; the Arcade build remains the shipping authority. The driver enables tracked shared compilation, sizes concurrency to the available CPUs, and automatically falls back to full input hashing when change journals are unavailable without modifying local volume settings. It never rewrites the checked-in XLF files, and -Pack (opt-in, not used by CI) also packs the 33 NuGet packages in the same graph; see opt-in pack. CI initializes the checkout volume's change journal before restoring the action cache between compatible runs; existing journals are preserved, and initialization failures fail the job. The Windows CI agent must run elevated. The BuildXL performance comparison remains an explicit local experiment for matched cold, changed-input, and no-op TRX builds. CI runs real solution builds, not repeated benchmark samples.

The easiest and recommended solution is to build the repository with the provided scripts at the repo root.

For Windows:

build.cmd

For Linux and macOS:

./build.sh

Common building options

By default, the script generates a Debug build type, which is not optimized code and includes asserts. As its name suggests, this makes it easier and friendlier to debug the code. If you want to make performance measurements, you ought to build the Release version instead, which doesn't have any asserts and has all code optimizations enabled. Likewise, if you plan on running tests, the Release configuration is more suitable since it's considerably faster than the Debug one. For this, you add the flag -configuration release (or -c release). For example:

For Windows:

build.cmd -configuration release

For Linux and macOS:

./build.sh --configuration release

Another common flag is -pack which will produce the NuGet packages of MSTest. These packages are required for the acceptance tests (see testing section).

On Windows, -pack also runs eng/verify-nupkgs.ps1, which checks package metadata and compares the files of each MSTest package against its expected shape in eng/package-shapes/<PackageId>.txt. A missing or unexpected file fails the build and is listed in the error. When a package layout change is intentional, regenerate the expected shapes after packing and commit them with your change:

.\eng\verify-nupkgs.ps1 -configuration Debug -updateBaselines

For more information about all the different options available, supply the argument -help|-h when invoking the build script. On Unix-like systems, non-abbreviated arguments can be passed in with a single - or double hyphen --.

Repository builds use MSBuild's experimental multithreaded mode by default, including desktop MSBuild on Windows and public/official CI. Windows builds require Visual Studio 2026 18.12 or newer. To diagnose a build without multithreading, use build.cmd -mt:0 on Windows or ./build.sh --mt false on Linux/macOS. Direct MSBuild invocations can use /mt:false to override the inherited MSBUILDENABLEMULTITHREADED default. Compatibility tests that deliberately opt out and BuildXL's separate native scheduler retain their existing behavior.

Build layout

MSTest uses Microsoft common infrastructure called arcade as such all outputs follow this structure:

artifacts
  bin
    $(MSBuildProjectName)
      $(Configuration)
  packages
    $(Configuration)
      Shipping
        $(MSBuildProjectName).$(PackageVersion).nupkg
      NonShipping
        $(MSBuildProjectName).$(PackageVersion).nupkg
      Release
      PreRelease
  TestResults
    $(Configuration)
      $(MSBuildProjectName)_$(TargetFramework)_$(TestArchitecture).(xml|html|log|error.log)
  SymStore
    $(Configuration)
      $(MSBuildProjectName)
  log
    $(Configuration)
      Build.binlog
  tmp
    $(Configuration)
  obj
    $(MSBuildProjectName)
      $(Configuration)
  toolset

with

directory description
bin Build output of each project.
obj Intermediate directory for each project.
packages NuGet packages produced by all projects in the repo.
TestResults Test results produced by test runs.
SymStore Storage for converted Windows PDBs
log Build binary log and other logs.
tmp Temp files generated during build.
toolset Files generated during toolset restore.

Testing

MSTest uses the following 3 kinds of tests:

  • Unit tests
    • Very fast tests primarily validating individual units.
    • Named as <ProjectUnderTest>.UnitTests where <ProjectUnderTest> is the project under test.
  • Integration tests
    • Slightly slower tests with File system interactions.
    • Named either as <ProjectUnderTest>.IntegrationTests or as <PackageUnderTest>.Acceptance.IntegrationTests where <ProjectUnderTest> is the project under test <PackageUnderTest> is the package under test
  • Performance tests
    • Focused tests that ensure the performance of specific workflows of the application

The easiest way to run the tests is to call

For Windows:

build.cmd -pack -test -integrationTest

For Linux and macOS:

./build.sh -pack -test -integrationTest

Note that -test allows to run the unit tests and -integrationTest allows to run the two kinds of integration tests. Acceptance integration tests require the NuGet packages to have been produced hence the -pack flag.

Retained OpenTelemetry traces

The Windows, Linux and macOS CI test jobs save existing MTP activities under artifacts/TestResults/<Configuration>/OpenTelemetry. Download the corresponding TestResults_<OS>_<configuration or collection kind>_Attempt<job attempt> build artifact to retrieve them. Failed jobs publish partial traces too. Availability follows the build's Azure Pipelines retention policy; no guaranteed number of days is configured in this repository.

The Windows app-model job includes the same directory inside Windows_App_Model_Diagnostics_Attempt<job attempt>/artifacts/TestResults/<Configuration>/OpenTelemetry.

Local collection is opt-in. After building the desired test host with the repository scripts, set TESTFX_OTEL_TRACE_DIRECTORY to an absolute directory before running it. For example, in PowerShell:

$previous = $env:TESTFX_OTEL_TRACE_DIRECTORY
try {
    $env:TESTFX_OTEL_TRACE_DIRECTORY = Join-Path $PWD 'artifacts\TestResults\Debug\OpenTelemetry'
    & .\artifacts\bin\Microsoft.Testing.Extensions.UnitTests\Debug\net10.0\Microsoft.Testing.Extensions.UnitTests.exe --filter 'FullyQualifiedName~TestFxOpenTelemetryTests'
} finally {
    $env:TESTFX_OTEL_TRACE_DIRECTORY = $previous
}

OTEL_SDK_DISABLED=true disables collection. Without the directory variable the existing instrumentation-only dogfooding remains active without writing files. The helper does not configure or contact an OTLP endpoint. The existing Native AOT exclusion remains unchanged.

Each uniquely named .jsonl file is a UTF-8 TestFx archive, not an OTLP request:

  • The first resource record declares format: "testfx-otel-jsonl", schemaVersion: 1, a unique archive runId, and resource attributes.
  • Each span record preserves IDs/parents, trace state/flags, source, name/kind, exact UTC start/end, durationTicks (100 ns ticks), status, attributes, events and links. Attribute lists contain { "key": ..., "value": ... } entries; JSON strings, booleans, numbers and arrays retain their value types. Readers must preserve 64-bit integers. Non-finite floats and unsupported attribute types are represented as invariant strings.
  • The last completion record contains status, spanCount and failedExports. Require status: "complete" and zero failures before treating a file as a complete export. A crash/forced termination can leave no completion and a partial last line; earlier complete lines remain useful, but are not a full run.

Completion describes archive delivery, not test-session success. For example, --help and --info produce complete archives with an informative TestHostBuilder operation (HostType: InformativeCommandLineHost) but no test results or test.run.exit_code verdict. Do not count those as healthy execution samples. Likewise, a complete archive can describe a failed or zero-result run.

CI checks completion with eng/verify-test-traces.ps1 before the existing artifact upload. Incomplete files fail that check on successful jobs and remain diagnostic warnings on failed/canceled jobs. Empty selection or explicit disablement is not proof that traces were collected.

Module version/binary MVID, build/SDK/TFM, machine/OS/runtime/CPU, CI/source, logical execution and supplied retry/job-attempt provenance are retained on the resource. Do not mix different attempts or use an absent phase duration as zero. Existing test.step.* attributes are preserved; canonical span elapsed time and reported result durations are not interchangeable, and fixture intervals may overlap.

The sink omits captured stdout/stderr attributes, but names, parameter values, paths, metadata and exceptions can still contain sensitive information. Treat these files like the other test artifacts; do not upload them to a new destination without reviewing their contents. Synchronous span flushing adds collection overhead and uses no dropping batch queue. See the research, contract and validation record.

Inspect the value of the retained data

Generate an offline cost/coverage snapshot from downloaded test-artifact ZIPs, directories or individual JSONL archives (Python 3.12+, no additional packages):

python .github\scripts\report_test_traces.py `
    --input C:\evidence\TestResults_Windows_Release_Attempt1.zip `
    --input C:\evidence\TestResults_Linux_Release_Attempt1.zip `
    --output C:\evidence\report\test-traces

The command writes an HTML report, Markdown and JSON with the same filename prefix. It reads trace members in ZIPs without extracting or running artifact code. Failed/incomplete/unsupported archives are named and excluded; the command returns exit 2 when input was rejected or no executing workers were found. Root CTRF reports are selected consistently from an unwrapped artifact or a single common artifact wrapper, in ZIPs and extracted directories; child and merged reports are not included. A damaged trace member does not discard healthy members. Artifact labels remain literal in both Markdown and HTML output.

The snapshot ranks existing fixture invocations and test-occurrence elapsed times, preserves TestContainer step timings, and shows the observation cohorts. Controllers and non-executing operations do not inflate test counts. Host protocol types are shown separately, but either protocol can be used by a test-launched subprocess; compare execution occurrences with the independent root CTRF counts. Shared fixture work and its overlapping interval union are distinct measures, neither CPU time nor time that can be subtracted to obtain native method-body duration.

This is not yet a weekly regression detector. One build can expose expensive setup, but cannot distinguish a regression from machine load. Binary MVIDs keep platform-specific variants separate when their exported TFM labels are identical; those identifiers are not stable cross-build cohort keys. See the first-report research and evidence.

Code coverage

Code coverage is enabled by default for repository tests. Each test project's generated coverage.config includes only its directly referenced production assemblies, excluding shared test-host infrastructure, NuGet package sources, and generated artifacts. Microsoft.Testing.TestInfrastructure is explicitly excluded at both the assembly and source levels.

Compatibility shims in Polyfills directories are excluded from coverage. Ensure.cs and ProcessExtensions.cs remain included because their argument validation and process-lifetime behavior are repository-owned logic worth testing. Coverage exclusions do not change which tests execute.

Mutation testing

The repository uses Stryker.NET for mutation testing. The checked-in MutationTesting.slnx provides a focused local run for the server-mode client sources, while the weekly workflow generates isolated solutions for every enabled module in eng/mutation-testing/modules.json. Unsupported targets are explicitly listed with a reason rather than silently omitted. First run .\build.cmd on Windows or ./build.sh on Linux and macOS to provision the repository-local .dotnet SDK. Then install the pinned tool to a dedicated path and run it from the repository root:

On Windows PowerShell:

$strykerVersion = (Get-Content .config/stryker/dotnet-tools.json | ConvertFrom-Json).tools.'dotnet-stryker'.version
$toolCommand = if (Test-Path artifacts/tools/stryker/dotnet-stryker.exe) { "update" } else { "install" }
$previousMutationTesting = $env:MutationTesting
$previousDotnetRoot = $env:DOTNET_ROOT
$previousTelemetryOptOut = $env:DOTNET_CLI_TELEMETRY_OPTOUT
$previousPath = $env:PATH
try {
    $env:DOTNET_ROOT = (Resolve-Path .dotnet).Path
    $env:DOTNET_CLI_TELEMETRY_OPTOUT = "1"
    $env:PATH = "$env:DOTNET_ROOT;$env:PATH"
    if ($toolCommand -eq "update") {
        & ./.dotnet/dotnet tool update dotnet-stryker --tool-path artifacts/tools/stryker --version $strykerVersion --configfile .config/stryker/NuGet.config --allow-downgrade
    }
    else {
        & ./.dotnet/dotnet tool install dotnet-stryker --tool-path artifacts/tools/stryker --version $strykerVersion --configfile .config/stryker/NuGet.config
    }
    if ($LASTEXITCODE -ne 0) {
        throw "Failed to install the pinned Stryker.NET tool."
    }
    $env:MutationTesting = "true"
    ./artifacts/tools/stryker/dotnet-stryker --config-file stryker-config.json --solution MutationTesting.slnx --output artifacts/mutation-testing
    if ($LASTEXITCODE -ne 0) {
        throw "Stryker.NET mutation testing failed."
    }
}
finally {
    $env:MutationTesting = $previousMutationTesting
    $env:DOTNET_ROOT = $previousDotnetRoot
    $env:DOTNET_CLI_TELEMETRY_OPTOUT = $previousTelemetryOptOut
    $env:PATH = $previousPath
}

On Linux and macOS:

(
  set -e
  export DOTNET_ROOT="$PWD/.dotnet"
  export DOTNET_CLI_TELEMETRY_OPTOUT=1
  export PATH="$DOTNET_ROOT:$PATH"
  stryker_version=$(sed -n 's/^[[:space:]]*"version":[[:space:]]*"\([^"]*\)".*/\1/p' .config/stryker/dotnet-tools.json)
  test -n "$stryker_version"
  if [ -x artifacts/tools/stryker/dotnet-stryker ]; then
    "$DOTNET_ROOT/dotnet" tool update dotnet-stryker --tool-path artifacts/tools/stryker --version "$stryker_version" --configfile .config/stryker/NuGet.config --allow-downgrade
  else
    "$DOTNET_ROOT/dotnet" tool install dotnet-stryker --tool-path artifacts/tools/stryker --version "$stryker_version" --configfile .config/stryker/NuGet.config
  fi
  MutationTesting=true ./artifacts/tools/stryker/dotnet-stryker --config-file stryker-config.json --solution MutationTesting.slnx --output artifacts/mutation-testing
)

The opt-in property runs unit-test projects on net10.0 and selects Arcade's open strong-name key for mutated assemblies and their friend assemblies because Stryker's in-memory compiler cannot complete Microsoft delay signing. Native MSTest hosts also consume the adapter/framework from source in this mode; a shipped Microsoft-key adapter cannot access the open-key friend declarations in the locally built platform and would fail before test discovery.

The HTML and JSON reports are written to artifacts/mutation-testing. The mutation testing workflow runs weekly and can be started manually. It reads the enabled modules from eng/mutation-testing/modules.json, runs each production/test-project mapping on an isolated runner, and uploads its detailed Stryker report. Every Microsoft.Testing.Extensions.* project must have an enabled module or an explicit excludedProjects entry with a reason, so newly added extensions cannot silently skip mutation testing. The Windows-only PackagedApp controller and build-only PackagedApp.MSBuild package are excluded from the Ubuntu/net10.0 Stryker shards, not from their acceptance tests.

Generated shard solutions must be passed to Arcade's --projects option using an absolute path; relative paths resolve under Arcade's toolset directory, not the repository root. Shards retain module failure metadata and build-console logs even when the build fails before Stryker starts. An aggregate job verifies that every enabled module used the same source revision, computes the weighted enabled-module score from the mutant counts, reports onboarding coverage and exclusion reasons against the known mutation targets, and uploads the mutation-testing-full-report artifact with a Markdown summary, an HTML index, aggregate JSON, logs, and all module reports. If any expected module is missing or failed, the aggregate is explicitly marked partial and the workflow fails after publishing the available diagnostics.

The Mutation Test Improver remains scoped to the ServerMode client report. Its trusted pre-agent step resolves the upstream run and downloads the binary artifact before the sandbox's text-only CLI proxy starts. It stages validated report data and run identity under /tmp/gh-aw/mutation-input; missing/expired reports and download/validation failures still reach monthly reporting as unavailable evidence, never a fresh zero score. Failed aggregates are usable only when their ServerMode module metadata proves success at the upstream source commit.

Working with Visual Studio

If you are working with Visual Studio, we recommend opening it through the open-vs.cmd script at the repo root. This script will set all the required environment variables required so that Visual Studio picks up the locally downloaded version of the .NET SDK. If you prefer to use your machine-wide configuration, you can open Visual Studio directly.

Inside Visual Studio, all projects can be built normally. All but acceptance tests can be tested directly from Visual Studio. The acceptance tests will always use the version of the NuGet packages produced in the artifacts/packages/shipping folder so if you have made some changes and run these tests, it's likely that the changes will not be applied.

Developing MSTest.Sdk

Do not use IsImplicitlyDefined="true" on PackageReference items in the MSTest.Sdk .targets files. The package would be defined twice, which can produce NU1009 warnings that are commonly treated as errors.

Do not use VersionOverride on those PackageReference items. Although it can override a version under Central Package Management (CPM), it is forbidden when CentralPackageVersionOverrideEnabled is false and causes NU1013.

Instead, split version specification based on CPM:

  • When ManagePackageVersionsCentrally is not true, set Version directly on the PackageReference.
  • When ManagePackageVersionsCentrally is true, leave the PackageReference unversioned and add a matching PackageVersion item.

This supports both values of CentralPackageVersionOverrideEnabled without producing NU1009.

Layering MSTest.Sdk on another base SDK

MSTest.Sdk implicitly imports Microsoft.NET.Sdk. To combine it with another base SDK, such as Microsoft.NET.Sdk.Web, import both SDKs manually and list the other SDK first:

<Project>
  <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk.Web" />
  <Import Project="Sdk.props" Sdk="MSTest.Sdk" />

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
  </PropertyGroup>

  <Import Project="Sdk.targets" Sdk="MSTest.Sdk" />
  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk.Web" />
</Project>

Because an <Import> cannot specify the SDK version in the same way as <Project Sdk="MSTest.Sdk/x.y.z">, pin it in global.json:

{
  "msbuild-sdks": {
    "MSTest.Sdk": "x.y.z"
  }
}

Sdk.props and Sdk.targets guard their Microsoft.NET.Sdk imports with _MSTestSdkImportsMicrosoftNETSdk. MSTest.Sdk imports the base SDK only when another SDK has not already set UsingMicrosoftNETSdk, avoiding MSB4011 duplicate-import warnings.

Visual Studio version requirement

Windows command-line builds require Visual Studio 2026 18.12 or newer for multithreaded desktop MSBuild, as specified by tools.vs.version in global.json. The repository uses the XML-based TestFx.slnx solution format, which these versions also support. The command-line multithreading default does not change Visual Studio's in-process IDE build scheduling.