Using MSTest, xUnit or NUnit with dotnet core

Question

TLDR - Jump to the answer for details on a working xUnit or MSTest setup. xUnit was the least painful option.

I've been unable to get NUnit working on my build server so I've resorted to trying both xUnit and MSTest - Neither have worked for me.

MSTest I just dont get - I expected this to work but when I try to install dotnet-test-mstest from nuget I get dependency issues as all of it's dependent libraries are not compatible for netcoreapp1 for example....

The dependency MSTest.ObjectModel 1.0.1-preview does not support framework .NETCoreApp,Version=v1.0.

The dependency MSTest.TestAdapter.Dotnet 1.0.4-preview does not support framework.

Can anyone offer any advice on why this could be happening?

Answer 1

I managed to get this working by creating brand new projects following these instructions...

https://docs.microsoft.com/en-us/dotnet/articles/core/testing/using-mstest-on-windows

It seems that MSTest dotnet core projects need to be created as console applications to work..?

In the aim of helping all others, here's the project.json that I ended up with.

{
    "version": "1.0.0-*",
    "testRunner": "mstest",

    "dependencies": {
        "dotnet-test-mstest": "1.1.1-preview",
        "MyLibrary": {
            "target": "project"
        },
        "Microsoft.NETCore.App": {
            "type": "platform",
            "version": "1.0.1"
        },
        "MSTest.TestFramework": "1.0.4-preview",
        "NSubstitute": "2.0.0-rc"
    },

    "frameworks": {
        "netcoreapp1.0": {
            "imports": "dnxcore50"
        }
    }
}

I seem to be doing 2 key things differently. For a start my key dependency was previously...

"NETStandard.Library": "1.6.0"

while in the new project.json it is

"Microsoft.NETCore.App": {
        "type": "platform",
         "version": "1.0.1"
     }

(in the original project.json this was placed in the frameworks section of the file as a subdependency).

the other is in the imports section under 'newtcoreapp1.0'. In the original project.json I used this included itself, whereas the working one imports dnxcore50 (although I'm still not convinced this is actually right).

It appears that there is a bug with 1.0.5-preview of MSTest.TestFramework - I ended up changing the version for an earlier one. With 1.0.5 the tests would load but every one would fail with an error...

Could not load type 'LogMessageHandler' from assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework

I looked around for somewhere to submit this as a bug but no luck yet. If anyone finds this with the same problems and knows where to log it please let me know and I'll '+1' it.

For completeness, here's a project.json file that works for NUnit (although it's NUnit 3 and reading it's output files is (at time of writing) an issue on VSTS...

{
    "dependencies": {
        "MyLibrary": {
            "target": "project"
        },
        "NETStandard.Library": "1.6.0",
        "NUnit": "3.5.0",
        "dotnet-test-nunit": "3.4.0-beta-1"
    },
    "frameworks": {
        "netcoreapp1.0": {
            "imports": [
                "netcoreapp1.0",
                "portable-net45+win8"
            ],
            "dependencies": {
                "Microsoft.NETCore.App": {
                    "version": "1.0.1-*",
                    "type": "platform"
                },
                "NSubstitute": "2.0.0-rc"
            }
        }
    },
    "testRunner": "nunit",
    "version": "1.0.0-*"
}

And here's xUnit - I'll be playing with that a bit more as it seems more viable on VSTS...

{
    "version": "1.0.0-*",
    "buildOptions": {
        "debugType": "portable"
    },
    "dependencies": {
        "System.Runtime.Serialization.Primitives": "4.1.1",
        "xunit": "2.1.0",
        "dotnet-test-xunit": "1.0.0-rc2-build10015",
        "NSubstitute": "2.0.0-rc",
        "MyLibrary": {
            "target": "project"
        }

    },
    "testRunner": "xunit",
    "frameworks": {
        "netcoreapp1.0": {
            "dependencies": {
                "Microsoft.NETCore.App": {
                    "type": "platform",
                    "version": "1.0.1"
                }
            },
            "imports": [
                "dotnet5.4",
                "portable-net451+win8"
            ]
        }
    }
}

More really useful xUnit info here - https://docs.microsoft.com/en-us/dotnet/articles/core/testing/unit-testing-with-dotnet-test

Overall I found, once taking getting VSTS working as well into account, that xUnit was far and above the easiest option to use - you just have to set up the build server tasks to use dotnet test command line and then use a publish test files task to show the results. After much pain I still couldn't get the MS test run to pass on the build server.

Answer 2

The issue with dotnet-test-mstest 1.1.1-preview and MSTest.TestFramework 1.0.5-preview that @TTCG brought up has hence been fixed with a later release. Feel free to try this out on dotnet-test-mstest 1.1.2-preview and MSTest.TestFramework 1.0.6-preview.