Is there a better way to pass dynamic inputs in-line to a DataTestMethod? I.e. How to programmatically create test inputs for a data-driven test

Question

I've been looking for this for years and years, and I think I've finally found a real way in "MSTest V2" (meaning the one that comes with .netcore, and is only really handled correctly in Visual Studio 2017). See my answer for my solution.

The problem this solves for me is that my input data is not easily serialized, but I have logic that needs to be tested with many of these inputs. There are lots of reasons why it's better to do it this way, but that was the show-stopper for me; I was forced to have one giant unit test with a for loop going through my inputs. Until now.

Answer 1

So the new DataTestMethodAttribute class is overridable, and it allows for overriding a method with this signature:

public override TestResult[] Execute(ITestMethod testMethod);

Once I discovered that, it was easy: I just derive, figure out my inputs, and then loop through them in my Execute method. I went a few steps further though, in order to make this easily re-usable.

So, first a base class that overrides that Execute method, and exposes an abstract GetTestInputs() method which returns an IEnumerable. You can derive from this any type which can implement that method.

public abstract class DataTestMethodWithProgrammaticTestInputs : DataTestMethodAttribute
{
    protected Lazy<IEnumerable> _items;

    public DataTestMethodWithProgrammaticTestInputs()
    {
        _items = new Lazy<IEnumerable>(GetTestInputs, true);
    }

    protected abstract IEnumerable GetTestInputs();

    public override TestResult[] Execute(ITestMethod testMethod)
    {
        var results = new List<TestResult>();
        foreach (var testInput in _items.Value)
        {
            var result = testMethod.Invoke(new object[] { testInput });
            var overriddenDisplayName = GetDisplayNameForTestItem(testInput);
            if (!string.IsNullOrEmpty(overriddenDisplayName))
                result.DisplayName = overriddenDisplayName;
            results.Add(result);
        }
        return results.ToArray();
    }

    public virtual string GetDisplayNameForTestItem(object testItem)
    {
        return null;
    }
}

Next, I created a derived type that uses reflection to instantiate a type, and then calls a property's get method on the created instance. This type can be used directly as an attribute, though deriving from it, implementing the GetDisplayNameForTestItem method, and tying to a specific type is a good idea, especially if you have more than one test where you are using the same data.

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class DataTestMethodWithTestInputsFromClassPropertyAttribute : DataTestMethodWithProgrammaticTestInputs
{
    private Type _typeWithIEnumerableOfDataItems;
    private string _nameOfPropertyWithData;

    public DataTestMethodWithTestInputsFromClassPropertyAttribute(
        Type typeWithIEnumerableOfDataItems,
        string nameOfPropertyWithData)
        : base()
    {
        _typeWithIEnumerableOfDataItems = typeWithIEnumerableOfDataItems;
        _nameOfPropertyWithData = nameOfPropertyWithData;
    }

    protected override IEnumerable GetTestInputs()
    {
        object instance;
        var defaultConstructor = _typeWithIEnumerableOfDataItems.GetConstructor(Type.EmptyTypes);
        if (defaultConstructor != null)
            instance = defaultConstructor.Invoke(null);
        else
            instance = FormatterServices.GetUninitializedObject(_typeWithIEnumerableOfDataItems);

        var property = _typeWithIEnumerableOfDataItems.GetProperty(_nameOfPropertyWithData);
        if (property == null)
            throw new Exception($"Failed to find property named {_nameOfPropertyWithData} in type {_typeWithIEnumerableOfDataItems.Name} using reflection.");
        var getMethod = property.GetGetMethod(true);
        if (property == null)
            throw new Exception($"Failed to find get method on property named {_nameOfPropertyWithData} in type {_typeWithIEnumerableOfDataItems.Name} using reflection.");
        try
        {
            return getMethod.Invoke(instance, null) as IEnumerable;
        }
        catch (Exception ex)
        {
            throw new Exception($"Failed when invoking get method on property named {_nameOfPropertyWithData} in type {_typeWithIEnumerableOfDataItems.Name} using reflection.  Exception was {ex.ToString()}");
        }
    }
}

Finally, here's an example of a derived attribute type in use which can easily be used for many tests:

[TestClass]
public class MyTestClass
{
    public class MyTestInputType{public string Key; public Func<string> F; }
    public IEnumerable TestInputs 
    {
        get
        {
            return new MyTestInputType[] 
            { 
                new MyTestInputType(){ Key = "1", F = () => "" }, 
                new MyTestInputType() { Key = "2", F = () => "2" } 
            };
        }
    }

    [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
    public class DataTestMethodWithTestInputsFromThisTestProjectAttribute : DataTestMethodWithTestInputsFromClassPropertyAttribute
    {
        public DataTestMethodWithTestInputsFromThisTestProjectAttribute() 
            : base(typeof(MyTestClass), nameof(MyTestClass.TestInputs)) { }

        public override string GetDisplayNameForTestItem(object testItem)
        {
            var asTestInput = testItem as MyTestInputType;
            if (asTestInput == null)
                return null;
            return asTestInput.Key;
        }
    }

    [DataTestMethodWithTestInputsFromThisTestProject]
    public void TestMethod1(MyTestInputType testInput)
    {
         Assert.IsTrue(testInput.Key == testInput.F());
    }

    [DataTestMethodWithTestInputsFromThisTestProject]
    public void TestMethod2(MyTestInputType testInput)
    {
        Assert.IsTrue(string.IsNullOrEmpty(testInput.F()));
    }
}

And that's it. Anyone have a better way with mstest?

Answer 2

You can now use the DynamicDataAttribute:

[DynamicData(nameof(TestMethodInput))]
[DataTestMethod]
public void TestMethod(List<string> list)
{
    Assert.AreEqual(2, list.Count);
}

public static IEnumerable<object[]> TestMethodInput
{
    get
    {
        return new[]
        {
            new object[] { new List<string> { "one" } },
            new object[] { new List<string> { "one", "two" } },
            new object[] { new List<string> { "one", "two", "three" } }
        };
    }
}

There is a good short into at https://dev.to/frannsoft/mstest-v2---new-old-kid-on-the-block

There is more gory detail at https://blogs.msdn.microsoft.com/devops/2017/07/18/extending-mstest-v2/