vstest.console.exe with /EnableCodeCoverage just "hangs"... how debug, and how to fix?

Question

I'm using vstest.console.exe (VS2012) to run tests with /EnableCodeCoverage, and with a .runsettings that defines a "Code Coverage" DataCollector (see CodeCoverage.runsettings in code block below).

I'm running from a powershell build script, that invokes:

vstest.console.exe /inIsolation /Logger:trx /EnableCodeCoverage /Settings:CodeCoverage.runsettings /TestCaseFilter:"TestCategory=Customers" bin\Release\Sdm.Test.IntegTest.dll

Previously this command was working, however a recent new project that integrated some old legacy code has brought in a lot of new dependencies/DLLs.

What I see is that the command just "hangs", and never seems to run any of the tests. When I use the SysInternals Process Explorer I do see some activity in vstest.executionengine.exe ... my best guess is that it is attempting to instrument a whole bunch of DLLs that my .runsettings file say should be excluded. But that's only a guess.

Any help in figuring out how to diagnose the issue would be appreciated.

CodeCoverage.runsettings below:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- Configurations that affect the Test Framework -->
  <RunConfiguration>
    <!-- Path relative to solution directory -->
    <ResultsDirectory>.\TestResults</ResultsDirectory>

    <!-- [x86] | x64  
      - You can also change it from menu Test, Test Settings, Default Processor Architecture -->
    <TargetPlatform>x64</TargetPlatform>

    <!-- Framework35 | [Framework40] | Framework45 -->
    <TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
  </RunConfiguration>

  <!-- Configurations for data collectors -->
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>

            <!--
            Additional paths to search for .pdb (symbol) files. Symbols must be found for modules to be instrumented.
            If .pdb files are in the same folder as the .dll or .exe files, they are automatically found. Otherwise, specify them here.
            Note that searching for symbols increases code coverage runtime. So keep this small and local.
            -->

            <!--           
            <SymbolSearchPaths>              
                   <Path>C:\Users\User\Documents\Visual Studio 2012\Projects\ProjectX\bin\Debug</Path>
                   <Path>\\mybuildshare\builds\ProjectX</Path>
            </SymbolSearchPaths>
            -->

            <!--
            About include/exclude lists:
            Empty "Include" clauses imply all; empty "Exclude" clauses imply none.
            Each element in the list is a regular expression (ECMAScript syntax). See http://msdn.microsoft.com/library/2k3te2cs.aspx.
            An item must first match at least one entry in the include list to be included.
            Included items must then not match any entries in the exclude list to remain included.
            -->

            <!-- Match assembly file paths: -->
            <ModulePaths>
              <Include>
                <ModulePath>.*\.dll$</ModulePath>
                <ModulePath>.*\.exe$</ModulePath>
              </Include>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
                <ModulePath>.*[uU]nit[tT]est\.dll</ModulePath>
                <ModulePath>.*[iI]nteg[tT]est\.dll</ModulePath>
                <ModulePath>.*bomicustomautopoco\.dll</ModulePath>
                <ModulePath>.*Common\.Logging\.dll</ModulePath>
                <ModulePath>.*Common\.Logging\.Log4Net129.dll</ModulePath>
                <ModulePath>.*fluentassertions\.dll</ModulePath>
                <ModulePath>.*x509publickeyparser\.dll</ModulePath>
                <ModulePath>.*EFCachingProvider\.dll</ModulePath>
                <ModulePath>.*EFProviderWrapperToolkit\.dll</ModulePath>
                <ModulePath>.*log4net\.dll</ModulePath>
                <ModulePath>.*DatahelperDTCBridge\.dll</ModulePath>
                <ModulePath>.*\.ni\.dll</ModulePath>
                <ModulePath>.*mscorlib\.dll</ModulePath>
                <ModulePath>.*vjslib\.dll</ModulePath>
                <ModulePath>.*Microsoft\..*dll</ModulePath>
                <ModulePath>.*System\.EnterpriseServices\..*dll</ModulePath>
                <ModulePath>.*System\.ComponentModel\..*dll</ModulePath>
                <ModulePath>.*System\.Configuration\..*dll</ModulePath>
                <ModulePath>.*System\.Core\..*dll</ModulePath>
                <ModulePath>.*System\.Data\..*dll</ModulePath>
                <ModulePath>.*System\.Entity\..*dll</ModulePath>
                <ModulePath>.*System\.IdentityModel\..*dll</ModulePath>
                <ModulePath>.*System\.Numerics\..*dll</ModulePath>
                <ModulePath>.*System\.Runtime\..*dll</ModulePath>
                <ModulePath>.*System\.ServiceModel\..*dll</ModulePath>
                <ModulePath>.*System\.Transactions\..*dll</ModulePath>
                <ModulePath>.*System\.Web\..*dll</ModulePath>
                <ModulePath>.*System\.Xml\..*dll</ModulePath>
                <ModulePath>.*msdia110typelib_clr0200\.dll</ModulePath>
                <ModulePath>.*vstest.executionengine.exe</ModulePath>
                <ModulePath>.*BOMi2Service\.dll</ModulePath>
                <ModulePath>.*NakedObjects\..*dll</ModulePath>
                <ModulePath>.*nakedobjects\..*dll</ModulePath>
                <ModulePath>.*sdm\.corejava\.dll</ModulePath>
                <ModulePath>.*sdm\.datahelper\.dll</ModulePath>
                <ModulePath>.*sdm\.events\.dll</ModulePath>
                <ModulePath>.*Sdm\.Infrastructure\.dll</ModulePath>
                <ModulePath>.*Sdm\.Infrastructure\.Attributes\.dll</ModulePath>
                <ModulePath>.*sdm\.systems\.application\.dll</ModulePath>
                <ModulePath>.*sdm\.systems\.distribution\.library\.dll</ModulePath>
                <ModulePath>.*sdm\.systems\.distribution\.server\.dll</ModulePath>
                <ModulePath>.*sdm\.objectstore\.dll</ModulePath>
                <ModulePath>.*sdm\.profiler\.dll</ModulePath>
                <ModulePath>.*sdm\.resultsprocessor\.dll</ModulePath>
                <ModulePath>.*sdm\.systems\.reflector\.dll</ModulePath>
                <ModulePath>.*Sdm\.Test\.Fixtures\.dll</ModulePath>
                <ModulePath>.*sdm\.utilities\.dll</ModulePath>
                <ModulePath>.*Spring\.Core\.dll</ModulePath>
                <ModulePath>.*TechTalk\.SpecFlow\.dll</ModulePath>
              </Exclude>
            </ModulePaths>

            <!-- Match fully qualified names of functions: -->
            <!-- (Use "\." to delimit namespaces in C# or Visual Basic, "::" in C++.)  -->
            <Functions>
              <Exclude>
                <Function>^Fabrikam\.UnitTest\..*</Function>
                <Function>^std::.*</Function>
                <Function>^ATL::.*</Function>
                <Function>.*::__GetTestMethodInfo.*</Function>
                <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
                <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
              </Exclude>
            </Functions>

            <!-- Match attributes on any code element: -->
            <Attributes>
              <Exclude>
                <!-- Don’t forget "Attribute" at the end of the name -->
                <Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>
                <Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>
                <Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>
                <Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>
                <Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
              </Exclude>
            </Attributes>

            <!-- Match the path of the source files in which each method is defined: -->
            <Sources>
              <Exclude>
                <Source>.*\\atlmfc\\.*</Source>
                <Source>.*\\vctools\\.*</Source>
                <Source>.*\\public\\sdk\\.*</Source>
                <Source>.*\\microsoft sdks\\.*</Source>
                <Source>.*\\vc\\include\\.*</Source>
              </Exclude>
            </Sources>

            <!-- Match the company name property in the assembly: -->
            <CompanyNames>
              <Exclude>
                <CompanyName>.*microsoft.*</CompanyName>
              </Exclude>
            </CompanyNames>

            <!-- Match the public key token of a signed assembly: -->
            <PublicKeyTokens>
              <!-- Exclude Visual Studio extensions: -->
              <Exclude>
                <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
                <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
                <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
                <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
                <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
                <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
                <PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
              </Exclude>
            </PublicKeyTokens>


            <!-- We recommend you do not change the following values: -->
            <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>          


          </CodeCoverage>
        </Configuration>
      </DataCollector>

    </DataCollectors>
  </DataCollectionRunSettings>

  <!-- Adapter Specific sections -->

  <!-- MSTest adapter -->
  <MSTest>
    <MapInconclusiveToFailed>True</MapInconclusiveToFailed>
    <CaptureTraceOutput>false</CaptureTraceOutput>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>False</DeploymentEnabled>
  </MSTest>


</RunSettings>

Answer 1

The only clue I could find that eventually led to a solution was in the Event Viewer, specifically:

.NET Runtime version 2.0.50727.5477 - Failed to CoCreate profiler.

This did ultimately led me to a fix, which is to add:

<startup useLegacyV2RuntimeActivationPolicy="true">
  <supportedRuntime version="v4.0.30319" />
</startup>        

to vstest.executionengine.exe.config (if running 64bit) or vstest.executionengine.x86.exe.config (if running 32bit).

This works under both VS (when spawned by devenv.exe) and from the command line (when spawned by vstest.console.exe)

Here's some notes that got me to this solution:

The new DLLs that are referenced include some pretty old code built against .NET 2.0. After a lot of searching, I've pieced together that:

• vstest.executionengine.exe uses the .NET 4 profiling stuff to do the dynamic code coverage instrumentation
• the event viewer error message is an obscure clue that vstest.executionengine.exe is failing to kick off the .NET 2.0 profiler
• as noted above, the fix is to update the .config (that is, vstest.executionengine.exe.config).

It isn't possible to add this stuff to the app.config of the project containing the tests; it must go into the vstest.executionengine's config (because this is the exe actually running).

Other things I tried (none of which, ultimately, helped):

Diagnostics in the registry

Computer\HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\11.0\EnterpriseTools\QualityTools\Diagnostics

• EnableTracing DWORD = 1
• TraceLevel DWORD = 4

Diagnostics in the *.config files

for vstest.console.exe, vstest.discoveryengine.*.exe, vstest.executionengine.*.exe

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow

added:

  <system.diagnostics>
    <switches>
      <add name="TpTraceLevel" value="4" />
    </switches>
  </system.diagnostics>

... this causes log files to be written to %TEMP%

Red herring: Tried setting environment variables to disable

- COR_ENABLE_PROFILING=0
- COMPLUS_ProfAPI_ProfilerCompatibilitySetting=DisableV2Profiler

... in an attempt to disable the .NET 2 profiler, but did nothing.

Further investigation (using procexp.exe) showed that vstest.executionengine.exe always sets COR_ENABLE_PROFILING=1 irrespective of env var.

Moreover, COR_PROFILER's set to guid

• for me, was COR_PROFILER={B19F184A-CC62-4137-9A6F-AF0F91730165}
• via regedit, HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID{B19F184A-CC62-4137-9A6F-AF0F91730165}\InprocServer32
• corresponds to "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\IntelliTrace\11.0.0\x64\Microsoft.IntelliTrace.Profiler.11.0.0.dll"

Looking at the event viewer I saw info messages that the .NET 4 profiler was successfully started:

.NET Runtime version 4.0.30319.18063 - The profiler was loaded successfully.  Profiler CLSID: '{b19f184a-cc62-4137-9a6f-af0f91730165}'. Process ID (decimal): 7700.  Message ID: [0x2507].

The log files in %TEMP% also suggested no issues.

Red herring: Tried to go the other way, and enable via env vars:

COMPLUS_ProfAPI_ProfilerCompatibilitySetting=EnableV2Profiler
and also explicitly set
COR_PROFILER={B19F184A-CC62-4137-9A6F-AF0F91730165}

made no difference either.

Other red herrings

• occasional messages in Event Viewer of form:

  engine::notify_process_attach failed with exception: Session "MTM_7d145e0c-1c26-44b0-89e5-acc448aaae6d" does not exist.

• vstest.discoveryengine.TpTrace.log... the error "AddProcess : Failed to AddProcess 5" also seems to be irrelevant.

  'System.EventHandler`1[Microsoft.VisualStudio.TestTools.Execution.SessionStartEventArgs]' to 'Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector' I, 2800, 11, 2014/07/01, 10:59:11.875, PCKMA0419\vstest.discoveryengine.exe, Started Vangaurd process with command line unregister /wildcard /session:MTM_* I, 2800, 11, 2014/07/01, 10:59:11.880, PCKMA0419\vstest.discoveryengine.exe, Add Vangaurd process to the project object W, 2800, 11, 2014/07/01, 10:59:11.882, PCKMA0419\vstest.discoveryengine.exe, AddProcess : Failed to AddProcess 5 I, 2800, 11, 2014/07/01, 10:59:11.882, PCKMA0419\vstest.discoveryengine.exe, Started Vangaurd process with command line collect /session:MTM_64f33307-c936-469e-b068-482ec0ea45cf /output:"C:\Users\danhaywood\AppData\Local\Temp\MTM_64f33307-c936-469e-b068-482ec0ea45cf\c44e78af-2475-4747-99f3-e0fc3ca41d51\DanHaywood_PCKMA0419 2014-07-01 10_59_11.coverage" /config:

"C:\Users\danhaywood\AppData\Local\Temp\MTM_64f33307-c936-469e-b068-482ec0ea45cf\CodeCoverage.config" ~~~

Blogs consulted along the way:

• What 'additional configuration' is necessary to reference a .NET 2.0 mixed mode assembly in a .NET 4.0 project?
• http://blogs.msdn.com/b/dougste/archive/2009/12/30/failed-to-cocreate-profiler.aspx
• http://social.msdn.microsoft.com/Forums/en-US/cf079584-54b0-44df-a157-620cc613fca6/failed-to-cocreate-profiler?forum=netfxtoolsdev
• http://blogs.msdn.com/b/davbr/archive/2007/12/11/debugging-your-profiler-i-activation.aspx
• http://bytes.com/topic/c-sharp/answers/813648-net-crashing
• http://social.msdn.microsoft.com/Forums/en-US/ab48e868-528d-4d40-b04f-b8a39ba8bf0c/vs2010-on-windows-2008-wrong-setting-of-corprofiler-environment-variable-in-target-process?forum=vstsprofiler
• http://msdn.microsoft.com/en-us/library/dd778910(v=vs.110).aspx
• http://blogs.msdn.com/b/davbr/archive/2009/05/26/run-your-v2-profiler-binary-on-clr-v4.aspx
• http://cvpcs.org/blog/2011-06-23/getting_ncover_and_nunit_to_play_nicely_with_.net_4.0
• Force NCover 1.5.8 to use v4 framework like testdriven.net does?