How do I tell MSTEST to run all test projects in a Solution?

Question

I need to know how to tell MSTEST to run all test projects in a solution file. This needs to be done from the command line. Right now I have to pass it a specific project file, I'm trying to get it to run from a SOLUTION file.

I'm hoping this is possible, because in Visual Studio, hitting Ctrl+R, A, runs ALL tests in the currently opened solution.

The way I've interpretted the help files, you have to pass in each DLL specifically.

I want to run this from the command line from my CruiseControl.NET server, so I can write other utilities to make this happen. If there is a wierd way of getting this to happen through some OTHER method, let me know.

How do I tell MSTEST to run all test projects for a solution?

<exec>
    <!--MSTEST seems to want me to specify the projects to test -->
    <!--I should be able to tell it a SOLUTION to test!-->
    <executable>mstest.exe</executable>
    <baseDirectory>C:\projects\mysolution\</baseDirectory>
    <buildArgs>/testcontainer:testproject1\bin\release\TestProject1.dll 
    /runconfig:localtestrun.Testrunconfig 
    /resultsfile:C:\Results\testproject1.results.trx</buildArgs>
    <buildTimeoutSeconds>600</buildTimeoutSeconds>
</exec>

Answer 1

To elaborate on VladV's answer and make things a bit more concrete, following the suggested naming convention running your tests can be easily be automated with MSBuild. The following snippet from the msbuild file of my current project does exactly what you asked.

<Target Name="GetTestAssemblies">
    <CreateItem
        Include="$(WorkingDir)\unittest\**\bin\$(Configuration)\**\*Test*.dll"
        AdditionalMetadata="TestContainerPrefix=/testcontainer:">
       <Output
           TaskParameter="Include"
           ItemName="TestAssemblies"/>
    </CreateItem>
</Target>
<!-- Unit Test -->
<Target Name="Test" DependsOnTargets="GetTestAssemblies">
    <Message Text="Normal Test"/>
<Exec 
    WorkingDirectory="$(WorkingDir)\unittest"
    Command="MsTest.exe @(TestAssemblies->'%(TestContainerPrefix)%(FullPath)',' ') /noisolation /resultsfile:$(MSTestResultsFile)"/>
    <Message Text="Normal Test Done"/>
</Target>

Furthermore integrating MsBuild with CruiseControl is a piece of cake.

Edit
Here's how you can 'call' msbuild from your ccnet.config.

First if you do not already use MSBuild for your build automation add the following xml around the snippet presented earlier:

<Project DefaultTargets="Build"
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    ..... <insert snippet here> .....
</Project>

Save this in e.g. RunTests.proj next to your solution in your source tree. Now you can modify the bit of ccnet.config above to the following:

<msbuild>
  <executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable>
  <workingDirectory>C:\projects\mysolution\</workingDirectory>
  <baseDirectory>C:\projects\mysolution\</baseDirectory>  
  <projectFile>RunTests.proj</projectFile>
  <targets>Test</targets>
  <timeout>600</timeout>
  <logger>C:\Program Files\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
</msbuild>

Answer 2

This is an old thread, but I have been struggling with the same issue and I realized that you can really just run MSTest on every dll in the whole solution and it doesn't really cause any problems. MSTest is looking for methods in the assemblies marked with the [TestMethod] attribute, and assemblies that aren't "test" assemblies just won't have any methods decorated with that attribute. So you get a "No tests to execute." message back and no harm done.

So for example in NAnt you can do this:

<target name="default">
    <foreach item="File" property="filename">
        <in>
            <items>
                <include name="**\bin\Release\*.dll" />
            </items>
        </in>
        <do>
            <echo message="${filename}" />
            <exec program="C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe">
                <arg value="/testcontainer: ${filename}" />
                <arg value="/nologo" />
            </exec>
        </do>
    </foreach>
</target>

and it will run all the test methods in every dll in every bin\Release folder in the solution. Those which are not test dlls will return a "No tests to execute." and those that have tests will have the tests run. The only part I haven't figured out yet is that (in NAnt) execution stops the first time a command returns a non-zero value. So if any unit tests fail it doesn't keep going to execute any tests in subsequent assemblies. That is not great, but if all the tests pass, then they will all run.