Test data "disappearing" when using MSTest DynamicData

Question

I am using VS 2022 and have a c# library that targets framework 4.5 up to .net 6. I have a MS Test project for the library. I use dynamic data heavily and recently noticed that not all of my test data was appearing in my test results. I have broken out the offending problem into as small as I can to prove out what I had been seeing.

If I put a breakpoint in HasExpectedItems2() and debug the test, I see it hit 3x and index is 0, 1, 2 respectively, as I would expect. However, if I put a breakpoint in HasExpectedItems1() and debug the test, it only hits 2x, and items contains 3 items, and is null, respectively. I am not seeing items ever contain 0 items, which should be one of the data rows. If I comment out the 3rd List in DumbItems (which contains the 3 objects), HasExpectedItems1() still hits 2x, and items contains 0 items, and is null, respectively, as expected. So it appears by having a populated list, it somehow ignores the empty list. I've been trying to resolve this for days and am completely baffled on what is going on here.

using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using Microsoft.VisualStudio.TestTools.UnitTesting;
#if NET45_OR_GREATER
using System.Web.Mvc;
#elif NETCOREAPP3_0_OR_GREATER
using Microsoft.AspNetCore.Mvc.Rendering;
#endif

[TestClass]
[ExcludeFromCodeCoverage]
public class StaticTestingUnitTests
{
    protected static readonly IEnumerable<SelectListItem>[] DumbItems = new IEnumerable<SelectListItem>[]
    {
        null,
        new List<SelectListItem>(),
        new List<SelectListItem>
        {
            new SelectListItem { Text = "One", Value = "1" },
            new SelectListItem { Text = "Two", Value = "2" },
            new SelectListItem { Text = "Three", Value = "3" },
        },
    };

    protected static IEnumerable<object[]> GetDumbTest1()
    {
        foreach (object item in DumbItems)
        {
            yield return new object[] { item };
        }
    }

    protected static IEnumerable<object[]> GetDumbTest2()
    {
        for (int i = 0; i < DumbItems.Length; i++)
        {
            yield return new object[] { i };
        }
    }

    [DataTestMethod]
    [DynamicData(nameof(GetDumbTest1), DynamicDataSourceType.Method)]
    public void HasExpectedItems1(IEnumerable<SelectListItem> items)
    {
        Assert.AreEqual(5, 5);
    }

    [DataTestMethod]
    [DynamicData(nameof(GetDumbTest2), DynamicDataSourceType.Method)]
    public void HasExpectedItems2(int index)
    {
        Assert.AreEqual(5, 5);
    }
}

Answer 1

This is happening due to an undocumented breaking change in MsTest 2.2.4 which changes the default behavior for discovering ITestDataSource test cases (i.e. including DynamicData) to happen during Test Discovery instead of during Test Execution.

To restore the previous behavior, upgrade to MsTest 2.2.6 or newer, and add the following assembly attribute:

[assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)]