NUnit: Why Doesn't Assert.Throws<T> Catch My ArgumentNullException?

Question

I am posting this question anew at the behest of the distinguished Mr. John Skeet, who suggested I devise a simple test program that isolates and demonstrates the issue I am encountering and repost the question. This question grew out of this one, so please forgive me if it all sounds very familiar. You can potentially glean extra details about this question from that one.

The issue I am encountering regards Assert.Throws<T> from NUnit 2.5.9. It will, on occasion, fail to catch any exceptions thrown in the method invoked by the TestDelegate. I have pinned down this behavior in a reproducible manner in the code below. (Though this may, admittedly, be a case of Fails On My Machine™.

To reproduce the error, I've created a solution with two C# DLL projects:

• The first contains one class, with a single public method. That method is an extension method that encapsulates the logic required to create a SqlCommand, populate its parameters and invoke ExecuteScalar on it. This project includes no other references.
• The second contains a single class with two methods that test whether or not the method in the first DLL is working as expected. This project references the first, and includes a reference to the NUnit Framework. No other assemblies are referenced.

When I step through the tests in the debugger, I observe the following:

1. Assert.Throws correctly invokes the ExecuteScalar<T> extension method.
2. The parameter values are null, as expected.
3. ExecuteScalar<T> tests its parameters for null values.
4. The debugger does hit and execute the line containing throw new ArgumentNullException(...).
5. After executing the throw, control of the application is not immediately transferred to Assert.Throws. Instead, it continues on the next line in ExecuteScalar<T>.
6. As soon as the next line of code executes, the debugger breaks, and displays the error "Argument null exception was unhandled by user code."

The source code that isolates this behavior is given below.

THE EXTENSION METHOD

namespace NUnit_Anomaly
{
    using System;
    using System.Data;
    using System.Data.SqlClient;

    public static class Class1
    {
        public static T ExecuteScalar<T>(this SqlConnection connection, string sql)
        {
            if (connection == null)
            {
                throw new ArgumentNullException("connection");
            }

            if (sql == null)
            {
                throw new ArgumentNullException("sql");
            }

            using (var command = connection.CreateCommand())
            {
                command.CommandType = CommandType.Text;
                command.CommandText = sql;
                return (T)command.ExecuteScalar();
            }
        }
    }
}

THE TEST CASES

namespace NUnit_Tests
{
    using System;
    using System.Data.SqlClient;
    using System.Diagnostics;

    using NUnit.Framework;

    using NUnit_Anomaly;

    [TestFixture]
    public class NUnitAnomalyTest
    {

        [Test]
        public void ExecuteDataSetThrowsForNullConnection()
        {
            Assert.Throws<ArgumentNullException>(() => ((SqlConnection)null).ExecuteScalar<int>(null));
        }

        [Test]
        public void ExecuteDataSetThrowsForNullSql()
        {

            const string server = "MY-LOCAL-SQL-SERVER";
            const string instance = "staging";
            string connectionString = String.Format("Data Source={0};Initial Catalog={1};Integrated Security=True;",
                                                    server,
                                                    instance);

            using (var connection = new SqlConnection(connectionString))
            {
                Assert.Throws<ArgumentNullException>(() => connection.ExecuteScalar<int>(null));
            }
        }
    }
}

The net effect is that the tests fail when they shouldn't. To the best of my understanding, Assert.Throws<T> should catch my exception and the test should pass.

UPDATE

I took Hans' advice and checked the Exceptions dialog. I wasn't breaking on thrown exceptions, but I was breaking on unhandled user exceptions. Apparently, that's why the debugger breaks into the IDE when the exception is thrown. Clearing the checkbox fixed the problem, and Assert.Throws<T> picked it up. However, if I haven't done this, I can't just press F5 to continue execution, or the exception will become a NullReferenceException.

So now the question is: Can I configure exception breaks on a per-project basis? I only want to do this when I'm testing, but not in general.

Answer 1

What actually happens is that Assert.Throws does catch your exception, however Visual Studio stops on the first-chance exception anyway. You can check this by just pressing F5; Visual Studio will happily carry on executing.

As the exception helper tells you, the exception was unhandled by user code. So we know that Visual Studio doesn’t consider NUnit to be user code for some reason.

Visual Studio actually tells you this in plain text, if you know where to look:

There is also evidence of this fact in the stack trace:

Solution 1: Use a debug build of NUnit with debugging symbols. That will get Visual Studio to regard NUnit as user code, and thus stop treating your exceptions as "unhandled by user code". This isn’t trivial, but might work better in the long term.

Solution 2: Turn off the "Enable Just My Code" checkbox in Visual Studio’s debugging settings:

P.S. I’m not considering work-arounds whereby you avoid the use of Assert.Throws<T> altogether, but there are of course ways of doing that.