NUnit: Running multiple assertions in a single test

Question

I have been asked to write a testing application that needs to test a new stored procedure on multiple rows in a database, in essence I want to do something like this:

[Test]
public void TestSelect()
{
    foreach(id in ids)
    {
        DataTable old = Database.call("old_stored_proc",id);
        DataTable new_ = Database.call("new_stored_proc",id);

        Assert.AreEqual(old.Rows[0]["column"],ne_.Rows[0]["column"]);
    }
}

When I run this test, if 1 row doesn't match the other, the entire test fails; instead I would like to count how many times the assertion was passed and how many times it has failed. Is there a way to do this with NUnit?

I realize that NUnit might be overkill and this is a simple task without it...I just wanted to learn it. ;)

Answer 1

I know that the question is specifically about NUnit, but interestingly enough, Gallio/MbUnit has a feature which allows to run and catch several assertions at once.

[Test]
public void MultipleTest()
{
    Assert.Multiple(() =>
    {
       Assert.IsTrue(blabla);
       Assert.AreEqual(pik, pok);
       // etc.
    }
}

The Assert.Multiple is catching all the failing assertions and is going to report them at the end of the test.

Answer 2

Seems like you are just Asserting the wrong thing. If you want to check all the values and then assert that there are no errors (or show the number of errors) then try this:

[Test]
public void TestSelect()
{
    int errors = 0;
    foreach(id in ids)
    {
        DataTable old = Database.call("old_stored_proc",id);
        DataTable new_ = Database.call("new_stored_proc",id);

        if (old.Rows[0]["column"] != new_.Rows[0]["column"])
        {
            errors++;
        }            
    }

    Assert.AreEqual(0, errors, "There were " + errors + " errors.");
}

Answer 3

1) If the id's are constant and not looked up at test run time, create a separate unit test fixture for each id. That way you will know which id's are actually failing. See here for a write up on the problems with data driven tests:
http://googletesting.blogspot.com/2008/09/tott-data-driven-traps.html

2) If you need to dynamically look up the id's making it impossible to create a fixture for each id, use akmad's suggestion with one change. Keep a list of id's where the values are not equal and add the list to the error message. It will be extremely difficult to diagnose a failing test that only states the number of errors, as you won't know what id's cause the errors.

3) I don't know how difficult it would be to do in NUnit, but in PyUnit, when we need to run tests on dynamically generated data, we dynamically create tests fixtures and attach them to the TestCase class so that we have a failed test for each piece of data that does not pass. Though I imagine this would be much more difficult without python's dynamic abilities.