Bug: VS2017 Live Unit Testing - only minus's - doesn't work

Question

TL;DR; Edit 6: I have narrowed it down and provided 5 steps to reproduce the problem/bug.

1. Create a VS2017 c# Console App (.Net Full Framework)
2. Add a method to the Program.cs and make the class public:

 

public class Program {     static void Main(string[] args)     {     }      public int Add(int a, int b)     {          return a + b;     } } } 

3. Right click the Add Method and choose Create Unit Test:

4. Use these settings to create a new Unit Test Project with the test:

5. Add a Unit Test

 

[TestMethod()] public void AddTest() {     Program p = new Program();     var r = p.Add(1, 2);      Assert.IsTrue(3 == r); } 

• Test Menu > Live Unit Testing > Start
• Context menu over the AddTest() > Live Testing > Include

Now change the Add's + symbol to -

The Live Unit Testing works, change the - symbol back to a + and the Live Test passes!!

Now Add a reference to a Nuget DLL (anyone will do) - Live Unit Testing no longer works! Or simply close the Solution, re-open it and Live Unit Testing no longer works!

---

I've setup a MSUnitTest v2 project and started Live Unit Testing. I am using a Web Api 2 .Net project (Full Framework v4.5.2).

I see the Minus signs everywhere but no Tick's or Crosses.

I change code the same as they do in the Live Unit Testing in Visual Studio 2017 video and the clocks appear over the minus's but then nothing...

The unit test will pass when I debug it:

When I hover the mouse over the blue minus's the message is "Covered by 0 Tests"

I have found that you need the MSTest.TestAdapter and MSTest.TestFramework as per https://developercommunity.visualstudio.com/content/problem/5520/live-unit-test-are-only-showing-minuses-using-micr.html which I have done and am referencing both V1.18:

• Microsoft.VisualStudio.TestPlatform.TestFramework
• Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions

There are a bunch of beta testers who reported the same problem:

https://developercommunity.visualstudio.com/content/problem/4510/live-unit-testing-doesnt-do-anything-1.html

https://developercommunity.visualstudio.com/content/problem/2737/live-unit-testing-doesnt-do-anything.html

https://developercommunity.visualstudio.com/content/problem/4376/live-unit-testing-doesnt-work.html

https://developercommunity.visualstudio.com/content/problem/2527/live-unit-testing-doesnt-work-for-projects-that-im.html

All posts are resolved or wont fix.

This Stackoverflow user got an issue with Live Unit Testing but its a Null Exception problem. I tried the solution in comments to remove the Microsoft.VisualStudio.QualityTools.UnitTestFramework from the Gac, when I tried to uninstall from C:\Windows\assembly I got an access denied and uninstalling via a Developer Command Prompt:

GacUtil /u Microsoft.VisualStudio.QualityTools.UnitTestFramework 

Results in:

Assembly: Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL Unable to uninstall: assembly is required by one or more applications

Could someone help me get Live Unit Testing to work?

Edit 1:

In VS > Tools > Options > Live Unit testing I turned logging to Verbose.

Looking at the output I saw one warning:

[15:48:26.521 Verbose] - TestDriver - TestDiscoverer log message: Warning - Configuration system failed to initialize

Following a few answers in this I still wasn't able to get rid of the "Configuration system failed to initialize" - not sure if this is a red-herring or not, its the only clue I've found so far...

Edit 2:

I ran VS2017 as Admin and that got rid of the "Configuration system failed to initialize". Unfortunately running as Admin doesn't make a difference, it still doesn't work but doesn't fail with any exceptions.

---

Here is Verbose log. Its interesting that "Preparing to start test discovery" looks like its found the x86 Test I have included.

However, it clearly cannot discover the test; "Discovered 0 tests"

[16:52:55.110 Verbose] - Aggregator - Calculating the set of tests that cover 1 files   [16:52:55.110 Verbose] - Aggregator - [Workflow 4] Sending TestRequestEvent with 6 assemblies and 0 tests   [16:52:55.110 Verbose] - TestDriver - [Workflow 4] Received TestRequestEvent   [16:52:55.110 Verbose] - TestDriver - [Workflow 4] Preparing to start test discovery (X64: 0, X86: 1).   [16:52:55.110 Verbose] - TestDriver - [Workflow 4] Start discovering tests from group 0, which contains 1 assemblies.   [16:52:55.599 Verbose] - TestDriver - [Workflow 4] Finished discovering test from group 0.   [16:52:55.599 Verbose] - TestDriver - [Workflow 4] Discovered 0 tests from XYZ.API.ClientService\bin\Debug\XYZ.API.ClientService.dll, updating cached data.   [16:52:55.599 Verbose] - TestDriver - [Workflow 4] Notify Aggregator about completion of test run. (Passed: 0, Failed: 0, Run succeeded: True)   [16:52:55.599 Verbose] - Aggregator - [Workflow 4] Received TestResponseEvent (TestRunCompleted)   [16:52:55.599 Verbose] - Aggregator - [Workflow 4] Coverage data for 0 tests was received during this workflow   [16:52:55.600 Verbose] - BuildManager - Allow to send a single build event.   [16:52:55.599 Verbose] - Aggregator - Starting coverage aggregation cycle - work list has 3 methods   [16:52:55.599 Verbose] - Aggregator - Finished coverage aggregation cycle - work list had 3 methods of which 3 methods were defined in 2 open files   [16:52:55.600 Verbose] - BuildManager - Interrupting build queue -> no new assemblies.   [16:52:55.600 Verbose] - StatusMargin - Received file coverage result   [16:52:55.599 Verbose] - Aggregator - Sending FileCoverageResult for 'C:\XYZProjectCollection\XYZ\XYZ.API.ClientService\Repositories\ValueRepository.cs - C:\XYZProjectCollection\Lincoln\XYZ.API.ClientService\XYZ.API.MiscService.csproj' [16:52:55.599 Verbose] - Aggregator - [Workflow 4] Sending TestRunCompletedEvent   

Edit 3:

I had a look through ProcessMonitor traces today.. nothing really stood out, I could have missed something but I'm starting to feel this is a bug not a config problem.

I'm working in a really locked down environment so it could be that something was blocked during vs2017 install. I know Unity3D domain is not whitelisted for downloads. Although all core Visual Studio Microsoft stuff is installed. I'm starting to run out of diagnostic troubleshooting strategy's. Any ideas very welcome!

Edit 4:

This morning I decided to get rid of the Microsoft.VisualStudio.QualityTools.UnitTestFramework DLL from the GAC. I did it following these instructions to overcome the Access Denied:

This led to finding one of the potential root causes. I started seeing the Live Unit Testing Logging outputing:

[09:44:59.053 Verbose] - BuildManager - C:\Program Files x86\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(3245,5): error MSB3491: Could not write lines to file "C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\i\XYZ.API.MiscService\Debug\CoreCompileInputs.cache". Access to the path 'C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\i\XYZ.API.MiscService\Debug\CoreCompileInputs.cache' is denied.    [09:44:59.053 Verbose] - BuildManager - C:\Program Files x86\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(4874,5): error MSB3491: Could not write lines to file "C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\i\XYZ.API.MiscService\Debug\XYZ.API.MiscService.csproj.FileListAbsolute.txt". Access to the path 'C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\i\XYZ.API.MiscService\Debug\XYZ.API.MiscService.csproj.FileListAbsolute.txt' is denied. 

I gave myself permission to all those directories, just to rule out any permissions issues:

This still doesn't work, however it seems I am getting closer as the Live Unit Tests are taking a little longer to not work (ie the clocks that appear on top of the minus's take longer to process).

Then I tried ProcMon again and it showed some interesting logs but no smoking gun.

Here is the current Verbose Log:

[10:22:07.363 Info] - BuildManager - Build completed (succeeded). [10:22:07.363 Verbose] - BuildManager - Interrupting build queue -> switching 'build' and 'test' directories. [10:22:07.363 Verbose] - BuildManager - New 'test' directory: C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\1\ [10:22:07.363 Verbose] - BuildManager - New 'build' directory: C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\0\ [10:22:07.363 Verbose] - Aggregator - [Workflow 4] Received BuildEvent. [10:22:07.363 Verbose] - Aggregator - assembly: C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API.Tests\bin\Debug\XYZ.API.Tests.dll [10:22:07.363 Verbose] - Aggregator - assembly: C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API.AccountClientService\bin\Debug\XYZ.API.AccountClientService.dll [10:22:07.363 Verbose] - Aggregator - assembly: C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API\bin\XYZ.API.dll [10:22:07.363 Verbose] - Aggregator - assembly: C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API.Helpers\bin\Debug\XYZ.API.Global.dll [10:22:07.363 Verbose] - Aggregator - assembly: C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API.MiscService\bin\Debug\XYZ.API.MiscService.dll [10:22:07.363 Verbose] - Aggregator - file: 'C:\XYZProjectCollection\Lincoln\XYZ.API.MiscService\Repositories\ValueRepository.cs - C:\XYZProjectCollection\Lincoln\XYZ.API.MiscService\XYZ.API.MiscService.csproj' [10:22:07.363 Verbose] - Aggregator - Skip reading spans for assembly C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API.Tests\bin\Debug\XYZ.API.Tests.dll with MVID c96c0f4b-b21e-47be-a71e-97ebf8a3d493 since we already have this information from a previous build [10:22:07.363 Verbose] - Aggregator - Skip reading spans for assembly C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API.AccountClientService\bin\Debug\XYZ.API.AccountClientService.dll with MVID 6772f896-04ab-4804-bb18-3ed2c7aeb9b9 since we already have this information from a previous build [10:22:07.363 Verbose] - Aggregator - Skip reading spans for assembly C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API\bin\XYZ.API.dll with MVID 9c862440-c16a-4efe-8574-76e8c1453c4d since we already have this information from a previous build [10:22:07.363 Verbose] - Aggregator - Skip reading spans for assembly C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API.Helpers\bin\Debug\XYZ.API.Global.dll with MVID a8899d5d-4730-46bf-b4f2-3c2b1b75b0b1 since we already have this information from a previous build [10:22:07.363 Verbose] - Aggregator - Removing MVID f337ca44-aae6-42ca-8df5-3776ff962372 for project C:\XYZProjectCollection\Lincoln\XYZ.API.MiscService\XYZ.API.MiscService.csproj from the file span cache [10:22:07.363 Verbose] - Aggregator - Adding MVID 8e944276-0eec-43f4-aff3-07e40f8611dc for project C:\XYZProjectCollection\Lincoln\XYZ.API.MiscService\XYZ.API.MiscService.csproj to the file span cache [10:22:07.363 Verbose] - Aggregator - Calculating the set of tests that cover 0 files [10:22:07.363 Verbose] - Aggregator - Reading spans for assembly C:\XYZProjectCollection\Lincoln\.vs\XYZ.API\18528\t\XYZ.API.MiscService\bin\Debug\XYZ.API.MiscService.dll with MVID 8e944276-0eec-43f4-aff3-07e40f8611dc [10:22:07.363 Verbose] - Aggregator - Adding 3 methods from 'C:\XYZProjectCollection\Lincoln\XYZ.API.MiscService\Repositories\ValueRepository.cs - C:\XYZProjectCollection\Lincoln\XYZ.API.MiscService\XYZ.API.MiscService.csproj' to the coverage aggregation work list [10:22:07.363 Verbose] - Aggregator - Calculating the set of tests that cover 1 files [10:22:07.363 Verbose] - Aggregator - [Workflow 4] Sending TestRequestEvent with 5 assemblies and 0 tests [10:22:07.363 Verbose] - TestDriver - [Workflow 4] Received TestRequestEvent [10:22:07.363 Verbose] - TestDriver - [Workflow 4] Preparing to start test discovery (X64: 0, X86: 1). [10:22:07.363 Verbose] - TestDriver - [Workflow 4] Start discovering tests from group 0, which contains 1 assemblies. [10:22:07.834 Verbose] - TestDriver - [Workflow 4] Finished discovering test from group 0. [10:22:07.834 Verbose] - TestDriver - [Workflow 4] Discovered 0 tests from XYZ.API.MiscService\bin\Debug\XYZ.API.MiscService.dll, updating cached data. [10:22:07.834 Verbose] - TestDriver - [Workflow 4] Notify Aggregator about completion of test run. (Passed: 0, Failed: 0, Run succeeded: True) [10:22:07.834 Verbose] - Aggregator - [Workflow 4] Received TestResponseEvent (TestRunCompleted) [10:22:07.834 Verbose] - Aggregator - [Workflow 4] Coverage data for 0 tests was received during this workflow [10:22:07.834 Verbose] - Aggregator - Starting coverage aggregation cycle - work list has 3 methods [10:22:07.834 Verbose] - Aggregator - Finished coverage aggregation cycle - work list had 3 methods of which 3 methods were defined in 5 open files [10:22:07.834 Verbose] - BuildManager - Allow to send a single build event. [10:22:07.834 Verbose] - BuildManager - Interrupting build queue -> no new assemblies. [10:22:07.834 Verbose] - Aggregator - Sending FileCoverageResult for 'C:\XYZProjectCollection\Lincoln\XYZ.API.MiscService\Repositories\ValueRepository.cs - C:\XYZProjectCollection\Lincoln\XYZ.API.MiscService\XYZ.API.MiscService.csproj' [10:22:07.834 Verbose] - Aggregator - [Workflow 4] Sending TestRunCompletedEvent [10:22:07.850 Verbose] - StatusMargin - Received file coverage result 

Edit 5:

According to this blog: https://blogs.msdn.microsoft.com/visualstudio/2017/03/09/live-unit-testing-in-visual-studio-2017-enterprise/

Q: Why does nothing happen after I include or exclude tests from Live Test Set?

A: This is a known issue which we were not able to fix in Visual Studio 2017 version 15.0. It is now fixed in Visual Studio 2017 version 15.2.

Turns out I have Visual Studio/15.0.0 +26228.9. Upgrading to Visual Studio 2017 (v15.2) DOES NOT fix the problem.

I have reported the issue to https://developercommunity.visualstudio.com and will keep everyone posted. It properly didn't get reported the environment is locked.

Answer 1

Update: MS say it's fixed in 15.3.2

https://social.msdn.microsoft.com/Forums/vstudio/en-US/9f5a5c01-052b-4b6a-94c5-70aae9cdf843/vs2017-live-unit-testing-only-minuss?forum=vsunittest

---

Ok, I have narrowed it down.

When you add a NuGet reference to the Unit Test Project, it adds a packages.config AND in some scenarios might add an App.Config file.

When I add Oracle.ManagedDataAccess the following is added to the App.Config:

<?xml version="1.0" encoding="utf-8"?> <configuration>   <!--<configSections>     <section name="oracle.manageddataaccess.client"       type="OracleInternal.Common.ODPMSectionHandler, Oracle.ManagedDataAccess, Version=4.122.1.0, Culture=neutral, PublicKeyToken=89b483f429c47342"/>     </configSections>-->   <system.data>     <DbProviderFactories>       <remove invariant="Oracle.ManagedDataAccess.Client"/>       <add name="ODP.NET, Managed Driver" invariant="Oracle.ManagedDataAccess.Client" description="Oracle Data Provider for .NET, Managed Driver"         type="Oracle.ManagedDataAccess.Client.OracleClientFactory, Oracle.ManagedDataAccess, Version=4.122.1.0, Culture=neutral, PublicKeyToken=89b483f429c47342"/>     </DbProviderFactories>   </system.data>   <runtime>     <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">       <dependentAssembly>         <publisherPolicy apply="no"/>         <assemblyIdentity name="Oracle.ManagedDataAccess" publicKeyToken="89b483f429c47342" culture="neutral"/>         <bindingRedirect oldVersion="4.121.0.0 - 4.65535.65535.65535" newVersion="4.122.1.0"/>       </dependentAssembly>     </assemblyBinding>   </runtime>   <oracle.manageddataaccess.client>     <version number="*">       <dataSources>         <dataSource alias="SampleDataSource" descriptor="(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=ORCL))) "/>       </dataSources>     </version>   </oracle.manageddataaccess.client> </configuration> 

Notice how I commented out the <!--<configSections ... configSections>-->, purely through a process of elimination I found its this specific App.Config tag that causes Live Unit Testing to fail.

Answer 2

As others have mentioned, this feature is still a bit buggy yet. I have had fairly good luck however with just turning on the feature, and paying attention to the Test Explorer window - At least on my box, it runs the tests, even if there are the "minus signs" in the gutter of the code editor. I have seen the feature work properly, however.

I would keep using the feature, and keep giving feedback to Microsoft any way you can (there should be a Send Feedback button in the Visual Studio menu bar).

UPDATE:
Using an existing project, I see no UI in the editor's gutter at all... I have version 15.3.4 installed. I include an entire solution in Live Unit Tests, and Rebuild... I get this message in the Output window:

[11:42:17.584 Info] No supported test adapters are referenced by this solution. If you have a test project, add a NuGet reference to a test adapter corresponding to the test framework used in order to run the tests. If you already have the required NuGet reference in your test project, performing a NuGet restore may resolve the issue. 

UPDATE 2: Creating a new project as a test case, and following your steps to include a NuGet package that adds a configsections tag, I was able to reproduce the problem. Hopefully this doesn't add to the confusion, but I only had the excluded tests/minus signs problem when the configsections tag was commented out (and the Oracle package was installed).