No xunit tests discovered by vstest.console.exe

Question

I'm putting together a new stack of unit tests to be run together as a CI job. I'm using vstest.console.exe instead of mstest.exe mainly for its ability to run tests from several frameworks, but right now the focus is a few xUnit dlls. The jobs are run as part of a Jenkins pipeline.

I have tested everything successfully on a couple of dev boxes, but annoyingly test discovery is not working on any of the CI build boxes so far. This is after the addition of the 0.99.8 xUnit test adapter vsix (also tested with 0.99.7). The xUnit dlls are being built against 4.5 with the 2.0.0.2378 beta nuget version of xUnit.

I have reproduced the symptoms with the simplest possible dll, with a single public test method, working fine on my own box, not working on any build boxes. The deployment enviroment is pretty straight forward, with a VS2012 install and the xUnit test adapter, on Windows 2012.

I have enabled the TpTrace logging via the vstest exe config files and everything looks fine. I guess I'm looking for a way to troubleshoot the issue further (maybe tracing for the xUnit discovery process) or a workaround the problem. I'd prefer to retain the use of vstest console for the simplicity of running several frameworks.

I have written up this issue via the xUnit codeplex site as well.

I've checked out this SO post but none of the suggested solutions make sense here.

Answer 1

It takes me quite some time to find the solution of how to use vstest.console.exe and xunit for VS2013, so I think it worth the time to explain here how I did for everyone...

The first step is to follow what is explained here to install the pre-release xunit.runner.visualstudio nuget package in the xunit projects that need it to be able to run the xunit test from Visual Studio.

Then, when you run your vstest.console.exe command, you MUST use the parameter /TestAdapterPath.

Your command line should look like something (the path toward the xunit adapter is here relative so, you could put it in absolute or adapt depending on the active directory):

"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" c:\path\to\your\assembly.to.test.dll /TestAdapterPath:".\packages\xunit.runner.visualstudio.0.99.9-build1021\build\_common\"

edit: Because the adapter dlls are copied to the output folder, we could simplify the command line giving the path "." to the /TestAdapterPath option:

"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" c:\path\to\your\assembly.to.test.dll /TestAdapterPath:"."

For information, it works also with NUnit, nuget package NUnitTestAdapter and the command:

"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" c:\path\to\your\assembly.to.test.dll /TestAdapterPath:"."

Answer 2

OK so problem solved, but after some frustrating troubleshooting that I will cover in case its useful for someone. The problem was that xunit.execution.dll was not available in the same folder as the dlls containing tests. This is required for xunit discovery. I only got here by:

• Setting HKCU\Software\Outercurve Foundation\xUnit.net\Visual Studio Test Plugin\MessageDisplay = Diagnostic (this should be possible via the runsettings file but is not being picked up, and not possible via the VS tools options xunit page because its failing to open)
• vstest now spits out 'Skipping xunitTests.dll (no reference to xUnit.net)'
• This message actually means that xunit.dll and xunit.execution.dll weren't found in the folder

Problem solved by making sure that dll gets copied into that folder on the build boxes.