Unit testing code targeting several frameworks at once with .NET Core and xUnit

Question

I have some libraries on .NET Core which target net45 and netstandard1.6 and I want to unit-test them. I did dotnet new -t xunittest and that created a new test project targeting netcoreapp1.0 so that it will tests only the .NET Core code.

I tried to also compile it to target also net45 but then I got a series of errors on test discovering. My questions is

Is there a way to test the code targeting both (maybe more later) frameworks with a single test project or should I make a test project for each one I target?

Edit: here are my project.json and the messages I'm getting:

{
  "version": "1.0.0-*",
  "buildOptions": {
    "debugType": "portable"
  },
  "dependencies": {
    "xunit": "2.1.0"
  },
  "testRunner": "xunit",
  "frameworks": {
    "net45": {
      "frameworkAssemblies": {
        "System.Runtime": "4.0.0.0"
      }
    },
    "netcoreapp1.0": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.0"
        },
        "System.Runtime.Serialization.Primitives": "4.1.1",
        "dotnet-test-xunit": "2.2.0-preview2-build1029"
      },
      "imports": [
        "dotnet5.4",
        "portable-net451+win8"
      ]
    }
  }
}

these are the errors I get once the project is compiled:

dotnet-test Error: 0 : [ReportingChannel]: Waiting for message failed System.IO.IOException: Unable to read data om the transport connection: An established connection was aborted by the software in your host machine. ---> stem.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.DotNet.Tools.Test.ReportingChannel.ReadMessages()
>dotnet-test Error: 0 : Unhandled Exception: System.IO.IOException: Unable to read data from the transport nnection: An established connection was aborted by the software in your host machine. ---> stem.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.DotNet.Tools.Test.ReportingChannel.ReadMessages()
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object ate)
[ReportingChannel]: Error sending System.IO.IOException: Unable to write data to the transport connection: Cannot cess a disposed object.
Object name: 'System.Net.Sockets.Socket'.. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, cketError& errorCode)
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BinaryWriter.Write7BitEncodedInt(Int32 value)
   at System.IO.BinaryWriter.Write(String value)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.Send(Message message)
dotnet-test Error: 0 : System.IO.IOException: Unable to write data to the transport connection: Cannot access a sposed object.
Object name: 'System.Net.Sockets.Socket'.. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, cketError& errorCode)
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BinaryWriter.Write7BitEncodedInt(Int32 value)
   at System.IO.BinaryWriter.Write(String value)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.Send(Message message)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.SendError(String error)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.SendError(Exception ex)
   at Microsoft.DotNet.Tools.Test.DesignTimeRunner.HandleDesignTimeMessages(ProjectContext projectContext, tnetTestParams dotnetTestParams)
   at Microsoft.DotNet.Tools.Test.DesignTimeRunner.DoRunTests(ProjectContext projectContext, DotnetTestParams tnetTestParams)
   at Microsoft.DotNet.Tools.Test.BaseDotnetTestRunner.RunTests(ProjectContext projectContext, DotnetTestParams tnetTestParams, BuildWorkspace workspace)
   at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)

and finalizes with this line

========== Discover test finished: 0 found (0:00:03.0417342) ==========

although I know I have one test (so far at least).

however If I remove the net 45 references all works like charm, here is my working project.json:

{
   "version": "1.0.0-*",
   "buildOptions": {
      "debugType": "portable"
   },
   "dependencies": {
      "System.Runtime.Serialization.Primitives": "4.1.1",
      "xunit": "2.1.0",
      "dotnet-test-xunit": "2.2.0-preview2-build1029"
   },
   "testRunner": "xunit",
   "frameworks": {
      "netcoreapp1.0": {
         "dependencies": {
            "Microsoft.NETCore.App": {
               "type": "platform",
               "version": "1.0.0"
             }
         },
         "imports": [
         "dotnet5.4",
         "portable-net451+win8"
         ]
      }
   }
}

Answer 1

According to the xUnit .NET Core docs,

You can target both net4xx and netcoreapp simply by adding both frameworks together in your project.json file. When you run dotnet test with multiple framework entries, the system will run all your framework tests, one after another.

This doesn't mean that you can have a single test project reference and test project that target different platforms (like a .NET 4.5 project and a .NET Core project). But it does mean that you can test projects that target the same set of platforms.

For example, I have a class library that targets both .NET 4.5.1 and .NET Core:

{
  "dependencies": {
    "NETStandard.Library": "1.6.0"
  },

  "frameworks": {
    "netstandard1.6": {},
    "net451":  {} 
  },
  "version": "1.0.0-*"
}

I can target net451 and netcoreapp1.0 in my test project, and reference this class library:

{
  "buildOptions": {
    "debugType": "portable"
  },
  "dependencies": {
    "xunit": "2.2.0-beta2-build3300",
    "dotnet-test-xunit": "2.2.0-preview2-build1029",
    "NetStandardClassLibrary": {
      "target": "project"
    }
  },
  "frameworks": {
    "net451": {},
    "netcoreapp1.0": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.0"
        }
      },
      "imports": [
        "dotnet5.4",
        "portable-net451+win8"
      ]
    }
  },
  "testRunner": "xunit",
  "version": "1.0.0-*"
}

When dotnet test is executed, tests will run twice (once for each platform target):

λ dotnet test
Project NetStandardClassLibrary (.NETStandard,Version=v1.6) was previously compiled. Skipping compilation.
Project XunitBothFrameworks (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
xUnit.net .NET CLI test runner (64-bit .NET Core win81-x64)
  Discovering: XunitBothFrameworks
  Discovered:  XunitBothFrameworks
  Starting:    XunitBothFrameworks
  Finished:    XunitBothFrameworks
=== TEST EXECUTION SUMMARY ===
   XunitBothFrameworks  Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 0.244s
Project NetStandardClassLibrary (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project XunitBothFrameworks (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
xUnit.net .NET CLI test runner (64-bit Desktop .NET win81-x64)
  Discovering: XunitBothFrameworks
  Discovered:  XunitBothFrameworks
  Starting:    XunitBothFrameworks
  Finished:    XunitBothFrameworks
=== TEST EXECUTION SUMMARY ===
   XunitBothFrameworks  Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 0.201s
SUMMARY: Total: 2 targets, Passed: 2, Failed: 0.

When I tried to run the tests inside Visual Studio, I got a similar error message as you. I think it's a bug, because dotnet test on the command line worked fine.

Answer 2

For me, the below project.json worked nicely, I didn't have to run the test from cmd.

{
  "version": "0.1.0-*",
  "dependencies": {
    "Moq": "4.5.22",
    "xunit": "2.2.0-beta2-build3300",
    "dotnet-test-xunit": "2.2.0-preview2-build1029",
    "IntegraPay.Domain": { "version": "1.0.0-*","target": "project" },
    "Integrapay.RegistrationApplication": { "version": "", "target": "project" }
  },
  "frameworks": {
    "netcoreapp1.0": {
      "imports": [
        "net451"
      ],
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.0"
        }
      }
    }
  },
  "testRunner": "xunit"
}