Add support for merging coverage results from multiple test assemblies

[picolino]

Summary

Add built-in support to Microsoft.Testing.Extensions.CodeCoverage for merging coverage results produced by multiple Microsoft Testing Platform test applications or test assemblies into a single coverage report.

The extension should provide a way to specify multiple input reports or a shared merged output and produce one combined result in the selected format, including Cobertura.

This would allow solutions containing multiple test projects to produce a single overall coverage percentage and a single coverage artifact without requiring an additional global tool or third-party report merger.

Background and Motivation

I am using Microsoft Testing Platform with Microsoft.Testing.Extensions.CodeCoverage in a solution that contains multiple test projects.

Each test project is a separate Microsoft Testing Platform executable and produces its own coverage report:

TestResults/UnitTests/coverage.cobertura.xml
TestResults/IntegrationTests/coverage.cobertura.xml
TestResults/FunctionalTests/coverage.cobertura.xml

These files contain coverage collected from different test suites, but they frequently cover the same production assemblies.

To obtain the actual overall coverage of the solution, the hit information from all test runs must be combined.

Calculating an average of the percentages is not correct. For example:

Unit tests:        80 of 100 lines covered = 80%
Integration tests: 20 of 100 lines covered = 20%

If the two suites cover different lines, the combined result may be 100%, not 50%. A correct merge must combine sequence points, branches, methods, and hit counts before recalculating the totals.

GitLab displays a single coverage value for the pipeline or merge request widget. It can also consume a Cobertura report for line-by-line visualization. For a solution-level result, the CI pipeline therefore needs one correctly merged coverage result representing all test projects.

At present, Microsoft.Testing.Extensions.CodeCoverage generates a separate report for each test application and does not provide an integrated merge operation.

The current workflow requires an additional step such as:

dotnet-coverage merge \
  --output merged.cobertura.xml \
  --output-format cobertura \
  TestResults/**/*.coverage

or the equivalent short options:

dotnet-coverage merge \
  -o merged.cobertura.xml \
  -f cobertura \
  TestResults/**/*.coverage

The dotnet-coverage tool already supports merging multiple reports in the following input formats:

coverage
cobertura
xml

It also supports glob patterns and options including:

-o, --output
-f, --output-format
--remove-input-files

ReportGenerator solves a similar problem at the report level. It accepts multiple reports through the -reports argument, supports semicolon-separated paths and globbing, and merges them into one generated report:

reportgenerator \
  -reports:"TestResults/**/coverage.cobertura.xml" \
  -targetdir:"CoverageReport" \
  -reporttypes:"Cobertura;Html"

Coverlet also provides incremental merging through:

--merge-with <path>

The input for this option is a previous Coverlet JSON report, which is merged with the current run before the requested output format is generated.

These tools demonstrate that merging is a common requirement for solutions with multiple test projects.

Because Microsoft.Testing.Extensions.CodeCoverage is already backed by Microsoft Code Coverage functionality, similar merge support should be available directly as part of the Microsoft Testing Platform extension.

Proposed Feature

Add a merge mode to Microsoft.Testing.Extensions.CodeCoverage that produces a single report from multiple coverage inputs.

There are several possible command-line designs.

Explicit merge command

The package could expose a command through the Microsoft Testing Platform executable:

dotnet test \
  -- \
  --coverage-merge \
  --coverage-merge-input "TestResults/**/*.coverage" \
  --coverage-output merged.cobertura.xml \
  --coverage-output-format cobertura

Possible options:

--coverage-merge
--coverage-merge-input <path-or-glob>
--coverage-output <path>
--coverage-output-format <coverage|xml|cobertura>
--coverage-remove-merge-inputs

Multiple input arguments should be supported:

--coverage-merge-input TestResults/UnitTests.coverage \
--coverage-merge-input TestResults/IntegrationTests.coverage

Globbing should also be supported:

--coverage-merge-input "TestResults/**/*.coverage"

This design would be similar to the existing dotnet-coverage merge command:

dotnet-coverage merge \
  -o merged.cobertura.xml \
  -f cobertura \
  TestResults/**/*.coverage

Merge into an existing output

Another design would be an option similar to Coverlet's --merge-with:

dotnet test \
  -- \
  --coverage \
  --coverage-output coverage.json \
  --coverage-merge-with combined-coverage.json

After each test application completes, its coverage would be merged into the specified report.

This design is convenient for sequential test execution, but safe concurrent access would need to be considered when test projects run in parallel.

Shared collection or orchestration mode

When dotnet test is invoked for a solution, project, directory, or multiple test applications, the Microsoft Testing Platform orchestration layer could collect all coverage outputs and automatically merge them after every test application finishes.

For example:

dotnet test Solution.sln \
  -- \
  --coverage \
  --coverage-merge \
  --coverage-output TestResults/coverage.cobertura.xml \
  --coverage-output-format cobertura

The expected result would be one file:

TestResults/coverage.cobertura.xml

containing the combined coverage from every test application participating in the invocation.

The merge must combine coverage at the source-document, module, method, sequence-point, and branch level before recalculating the overall percentages.

It should not calculate an arithmetic average of the percentages from the input reports.

The implementation should also define behavior for:

• the same module appearing in multiple reports;
• different builds or module identifiers;
• the same source file represented with different relative or absolute paths;
• duplicate test runs;
• reports using different supported input formats;
• partially missing source files or symbols;
• parallel test application execution;
• deterministic output ordering;
• failed or incomplete test runs.

Ideally, the package should support both collection and merging without requiring a separate installation of the dotnet-coverage global tool:

<PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" />

The goal is for the complete solution-level workflow to be available through the Microsoft Testing Platform extension:

run tests → collect per-test-app data → merge → write one Cobertura report

Alternative Designs

Use dotnet-coverage merge

The official dotnet-coverage global tool already supports merging:

dotnet-coverage merge \
  -o merged.cobertura.xml \
  -f cobertura \
  TestResults/**/*.coverage

This produces the required result, but it requires users to:

1. Install and version a separate .NET tool.
2. Preserve every intermediate coverage file.
3. Locate the files produced by separate test applications.
4. Run an additional pipeline command.
5. Publish the merged output instead of the original reports.

Since the merge implementation already exists in the Microsoft coverage tooling, exposing it through Microsoft.Testing.Extensions.CodeCoverage would provide a more integrated Microsoft Testing Platform experience.

Use ReportGenerator

ReportGenerator can consume multiple reports using:

-reports:<report>[;<report>]

and supports globbing:

reportgenerator \
  -reports:"TestResults/**/coverage.cobertura.xml" \
  -targetdir:"CoverageReport" \
  -reporttypes:"Cobertura"

ReportGenerator supports merging several coverage files into one report and is a useful workaround.

However, it is a third-party dependency and primarily operates as a report generator. Users should not need an external reporting package merely to combine data created by the official Microsoft coverage extension.

Use Coverlet --merge-with

Coverlet supports incremental merging with:

--merge-with <path>

The referenced input must use Coverlet's JSON format. This approach works well when multiple coverage runs are controlled by the same script.

However, switching to Coverlet is not always desirable or possible for projects that have standardized on Microsoft Testing Platform and Microsoft Code Coverage.

Collect all tests in one test application

Tests could theoretically be reorganized into one executable so that coverage is collected in a single run.

This is generally not practical. Separate test projects may target different frameworks, have different dependencies, require different environments, or represent unit, integration, and functional test suites with distinct configurations.

The coverage tooling should support standard multi-project solution layouts without requiring tests to be consolidated artificially.

Publish multiple reports and rely on the CI system

Some CI systems can accept multiple coverage artifacts or combine values from several jobs.

However, this does not always produce a correct solution-level coverage result, particularly when multiple test suites cover overlapping production assemblies.

A correct combined percentage requires merging the underlying coverage data, not averaging percentages reported by independent test jobs.

Producing a single merged report also improves portability across GitLab, GitHub Actions, Azure Pipelines, and other CI systems.

Related

#233 should not forget to print merged coverage results to standard output