VS2015 Code Coverage not working with tests in ASP.NET Core 1.0 (formerly known as ASP.NET 5)

Question

I have a an ASP.NET Core 1.0 (previously known as ASP.NET 5) solution with a couple of Class Library (Package)'s and an ASP.NET MVC6 project.

I have a test library using the new XUnit 2.0 which supports Core 1.0.

However, for some reason my code coverage is producing zero results when running it on all of my tests which are passing.

By default, ASP.NET Core 1.0 projects are built in-memory by the runtime and no artifacts are persisted to disk. So in the settings I enabled "Produce all outputs on build" for each project in the solution. Now when I build I see the pdb and dll files being output to the artifacts folder. I thought for sure code coverage would work after this, but still no results.

Maybe code coverage just simply doens't work with the new .NET Core 1.0. If anyone has some information on this that would be great.

Answer 1

I'm able to get code coverage with OpenCover 4.6.166 using this configuration :

project.json sample

{
"version": "1.0.0-*",
"dependencies": {
    "<project under test>": "1.0.0-*",
    "dotnet-test-xunit": "2.2.0-preview2-build1029",
    "xunit": "2.2.0-beta2-build3300"
},
"frameworks": {
    "net451": {
        "dependencies": {
            "Moq": "4.5.16"
        }
    },
    "netcoreapp1.0": {
        "dependencies": {
            "Microsoft.NETCore.App": {
                "version": "1.0.0",
                "type": "platform"
            },
            "moq.netcore": "4.4.0-beta8",
            "System.Diagnostics.Process": "4.1.0",
            "System.Diagnostics.TraceSource": "4.0.0"
        },
        "imports": [
            "dnxcore50",
            "portable-net451+win8"
        ]
    }
},
"testRunner": "xunit"

}

I use the myget CI feed my NuGet.config :

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
        <clear />
    <add key="api.nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
    <!--<add key="aspnetmaster" value="https://www.myget.org/F/aspnetmaster/api/v3/index.json" />-->
    <!--<add key="aspnetlatest" value="https://www.myget.org/F/aspnetvnext/api/v2" />-->
        <add key="AspNetVNext" value="https://www.myget.org/F/aspnetcidev/api/v3/index.json" />
  </packageSources>
</configuration>

Restore dependencies with --infer-runtimes option:

dotnet restore --infer-runtimes

Then use OpenCover with this command line :

OpenCover.exe -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:" test ""<your project path>\project.json"" -f net451 -xml ""test\ChatLe.Repository.Test\xunit-results.xml"" -nologo -parallel none" -output:"<your coverage .xml path>" -register:user -filter:"+[*]* -[xunit*]* -[*]*Migrations.*" -returntargetcode

Note the -f net451 switch is important, it says to dotnet to use the net451 framework. To cover netcoreapp1.0 use -f netcoreapp1.0.

In my console output, I can see those result :

Answer 2

I got a .NET Core project with working code coverage using xUnit 2.*, Appveyor, OpenCover and Coveralls, but it had to be extended to .NET Platform Standard. You can see the results on the relevant open source library here.

Some notable conclusions:

1. I couldn't find a way to get OpenCover to pay attention to 'portable' debugType *.pdb files. I had to use the legacy, Windows-only 'full' *.pdb format. This is done by setting 'debugType' to 'full' under 'buildOptions' in your project's 'project.json' file.

2. I had to run OpenCover on Windows, and it had to run via some version of the traditional .NET Framework. If you haven't yet, configure your non-test project to use .NET Platform Standard instead of .NET Core explicitly, like I did here. Then, in your test project, break apart your .NET Core and your traditional .NET Frameworks, like I did here. Assuming you use .NET Platform Standard 1.6, you would use net463 where I have net46.

3. Run OpenCover using the traditional .NET Framework via the dotnet test command line tool, not the xUnit console runner. You can see my code coverage generation (and submission) powershell script here. Where I specify net46, you would specify the version you used in your test project (presumably net463).

Here's the script pulled from source. You may or may not need to adjust the filter argument with respect to your dependencies.

nuget install OpenCover -Version 4.6.519 -OutputDirectory tools
nuget install coveralls.net -Version 0.7.0 -OutputDirectory tools
.\tools\OpenCover.4.6.519\tools\OpenCover.Console.exe -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:" test ""test\Invio.Extensions.DependencyInjection.Tests\project.json"" -f net46" -register:user -filter:"+[*]* -[xunit*]*" -returntargetcode -output:opencover_results.xml
.\tools\coveralls.net.0.7.0\tools\csmacnz.Coveralls.exe --opencover -i .\opencover_results.xml

Technically, this isn't testing .NET Core. This is testing the traditional .NET Framework's interpretation of an otherwise .NET Core project. However, if you do not do any conditional compilation based upon which framework you're using, the code coverage results will be identical.

A minor word of caution: if you look at the example repository, you'll see that I have a debugType of portable, not full, for my project.json file. That's because I am using Travis CI for Linux & Mac OS X builds, and that fails when you use full debugTypes. I got around this by setting mine programmatically via a powershell script running after my build on appveyor.

Answer 3

I've got it working XUnit.

Important things to note:

• xunit.runner.json is where your xunit configurations go
• Microsoft.CodeCoverage package is needed to run code coverage in visual studio

You will get all tests shown in the Test Runner Explorer. You can also analyze code coverage with the Visual Studio menu item. Only thing I haven't got working is code coverage highlighting: code coverage highlighting in .net core visual studio

This is my project.json

{
  "version": "1.0.0-*",
  "buildOptions": {
    "debugType": "portable",
    "copyToOutput": {
      "include": [ "xunit.runner.json" ]
    }
  },
  "dependencies": {
    "System.Runtime.Serialization.Primitives": "4.1.1",
    "xunit": "2.1.0",
    "dotnet-test-xunit": "2.2.0-preview2-build1029",
    "ClanService": { "target": "project" },
    "Utilities": { "target": "project" },
    "UnitTests.Configuration": { "target": "project" },
    "Microsoft.CodeCoverage": "1.0.2"
  },
  "testRunner": "xunit",
  "frameworks": {
    "netcoreapp1.0": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.1"
        }
      },
      "imports": [
        "dotnet5.4",
        "portable-net451+win8"
      ]
    }
  }
}