Dotnet Unit test with Coverlet- How to get coverage for entire solution and not just a project

Question

We are using coverlets (https://github.com/tonerdo/coverlet) for measuring code coverage of unit tests in a .NET solution containing multiple projects. The results are appearing separately for every project in the solution. What we want is to have a consolidated result for the entire solution. Can anyone suggest the best way to get that? If by any chance it is not possible by coverlet can you suggest any alternate open source tool that can do this using a CLI. We essentially need to integrate it with a CI tool, which should warn if the coverage is below a threshold.

Answer 1

This is how we are generating code coverage for the entire solution using coverlet.msbuild.

1. Reference coverlet.msbuild in each test project in your solution.
2. In your CI script, navigate to the directory containing your solution file. Then,
3. Run a command similar to the following (syntax is bash)

dotnet test {solution_filename.sln} --logger:trx --results-directory ../TestResults \
   "/p:CollectCoverage=true" \
   "/p:CoverletOutput=../TestResults/" \
   "/p:MergeWith=../TestResults/coverlet.json" \
   "/p:CoverletOutputFormat=\"json,cobertura\"" 

If running this on Windows, you may need to escape some characters passed in to these arguments such as comma (%2c).

To merge the results across several projects, we generate two output formats, json and cobertura. See the parameter /p:CoverletOutputFormat.

When generating code coverage for each project, coverlet will use /p:MergeWith to merge the coverlet.json for the current project with the previous coverlet.json.

This approach yielded one cobertura results file for the solution that we could use later in our CI build.

Answer 2

If you are using the coverage.collector nuget, it will generate separate test result file for each project.

You can then use reportgenerator tool to merge multiple results into one.

Here is how we are doing it in our CI:

dotnet test <solution-file> --collect:"XPlat Code Coverage"
dotnet tool install --global dotnet-reportgenerator-globaltool --version <version-number>
reportgenerator -reports:<base-directory>/**/coverage.cobertura.xml -targetdir:<output-directory>/CoverageReport -reporttypes:Cobertura

This will generate a combined report for all your test projects.

Answer 3

Recently I bumped into the same problem and Joel answer works perfectly. Unless you need xml instead of json. In that case you have to run test projects one by one, producing json output and merging with a previous BUT run the last one in a format you need.

Here is an example of how I did it:

RUN dotnet test "tests/[project name].Test.Integration/[project name].Test.Integration.csproj" \
    --configuration Release \
    --no-restore \
    --no-build \
    --verbosity=minimal \
    -p:CollectCoverage=true \
    -p:CoverletOutputFormat="json" \
    -p:CoverletOutput=/src/cover.json

RUN dotnet test "tests/[project name].Test.Unit/[project name].Test.Unit.csproj" \
    --configuration Release \
    --no-restore \
    --no-build \
    --verbosity=minimal \
    -p:CollectCoverage=true \
    -p:CoverletOutputFormat="opencover" \
    -p:CoverletOutput=/src/cover.xml \
    -p:MergeWith=../../cover.json

I run it in Docker and my folder structure might look a little messed up =). Here it is, to avoid any confusion:

src
---src
---tests
------[project name].Test.Integration
---------[project name].Test.Integration.csproj
------[project name].Test.Unit
---------[project name].Test.Unit.csproj
---[project name].sln
---cover.json <- this file gets created after the first command
---cover.xml <- this file gets created after the second command

So I run coverage for my integration tests first and then I merge it with unit test coverage in desired format (opencover since I need it for SonarQube)

Answer 4

None of the above-mentioned answers can help you with failing a build in CI, when the overall coverage of the solution falls below a certain threshold. All you get is a JSON, XML or HTML report which you need to parse with an additional script and that's too much work.

Instead, this is what I did.

First, run dotnet test and tell coverlet to merge all your coverage results. (You need to reference coverlet.msbuild in your test projects)

dotnet test your_solution.sln /p:CollectCoverage=true /p:CoverletOutput=../TestResults/ /p:CoverletOutputFormat="json%2ccobertura" /p:MergeWith=../TestResults/coverage.json

If you add your threshold switches (/p:Threshold, etc) to the same command, it will try to apply it for each project individually, but that's not what we want.

So to go around that limitation, what we can do is to run the above command one more time to merge the existing result with one of the test projects you have. Merging the total with a subsection is not going to change your total.

dotnet test one_of_your_test_projects.csproj /p:CollectCoverage=true /p:CoverletOutput=../TestResults/ /p:CoverletOutputFormat="json%2ccobertura" /p:MergeWith=../TestResults/coverage.json /p:ThresholdType=branch /p:Threshold=80 /p:ThresholdStat=total

Now, that command fails only when your solution's coverage falls below your threshold. Here are some YAML tasks for AzureDevOps which I use. It should be somewhat similar for any other too.

- task: DotNetCoreCLI@2
  displayName: 'Calculate code coverage'
  continueOnError: false
  inputs:
    command: test
    projects: '**/*.Test.csproj'
    arguments: '/p:CollectCoverage=true /p:CoverletOutput=../TestResults/ /p:CoverletOutputFormat="json%2ccobertura" /p:MergeWith=../TestResults/coverage.json  /m:1'
    publishTestResults: false

- task: DotNetCoreCLI@2
  displayName: 'Checkpoint: Total branch coverage >= $(codeCoverageBranchThreshold)'
  continueOnError: false
  inputs:
    command: test
    projects: './one_of_your_test_projects_dir/one_of_your_test_projects.csproj'
    arguments: '/p:CollectCoverage=true /p:CoverletOutput=../TestResults/ /p:CoverletOutputFormat="json%2ccobertura" /p:MergeWith=../TestResults/coverage.json /p:ThresholdType=branch /p:Threshold=$(codeCoverageBranchThreshold) /p:ThresholdStat=total'
    publishTestResults: false