Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Unit test passes when in debug but fails when run

A search method returns any matching Articles and the most recent Non-matching articles up to a specified number.

Prior to being returned, the IsMatch property of the matching articles is set to true as follows:

articles = matchingArticles.Select(c => { c.IsMatch = true; return c; }).ToList();

In a test of this method,

    [Test]
    public void SearchForArticle1Returns1MatchingArticleFirstInTheList()
    {
        using (var session = _sessionFactory.OpenSession())
        {
            var maxResults = 10;
            var searchPhrase = "Article1";
            IArticleRepository articleRepository = new ArticleRepository(session);
            var articles = articleRepository.GetSearchResultSet(searchPhrase, maxResults);
            Assert.AreEqual(10, articles.Count);
            Assert.AreEqual(1, articles.Where(a => a.Title.Contains(searchPhrase)).Count());
            var article = articles[0];
            Assert.IsTrue(article.Title.Contains(searchPhrase));
            Assert.IsTrue(article.IsMatch);
        }
    }

All assertions pass when the test is run in debug, however the final assertion fails when the test is run in release:

Expected: True But was: False

In the app itself the response is correct.

Any ideas as to why this is happening?

Edit:

I figured out what the problem is. It's essentially a race condition. When I am setting up the tests, I am dropping the db table, recreating it and populating it with the test data. Since the search relies on Full Text search, I am creating a text index on the relevant columns and setting it to auto populate. When this is run in debug, there appears to be sufficient time to populate the text index and the search query returns matches. When I run the test I don't think the index has been populated in time, no matches are returned and the test fails. It's similar to issues with datetimes. If I put a delay between creating the catalog and running the test the test passes.

like image 748
Pones Avatar asked Jun 19 '11 09:06

Pones


People also ask

Do assertions in unit tests fail when debugged?

Assertions in unit tests were failing when the tests were run but would pass when the unit tests were debugged. This was occurring for a handful of unit tests but not all of them.

What is the difference between debug mode and Run mode?

Basically when you are running in debug mode, you are running a single test only. When you are running in run mode, you are running multiple tests in addition to the one you are having a problem with. In my situation the problem was those other tests writing to a global list that I was not explicitly clearing in my test setup.

Why do my tests fail when I combine them?

The failing tests share a resource that affects them all when tested together. Recheck the affected tests and their subjects. You should also look into static fields or properties in the subjects. They tends to cause issues if not used properly when designing your classes. Thanks @Nkosi.

How do I fix a test that is not working?

I fixed the issue by clearing the list at the beginning of the test. My advice to see if this is the type of problem you are facing would be to disable all other tests and only 'run' the test you have an issue with. If it works when ran by itself, but not with others, you'll know you have some dependency between tests.


1 Answers

Pones, you have since clarified that the unit test fails when not debugging.

At this stage it could be anything however you should continue to run the unit test not debugging and insert the following statement somewhere you know (or think you know) is true

 if(condition)
            Debugger.Launch();

This will do the obvious and allow you to zone in on whats going wrong. 1 Place i suggest is on the IsMatch property (for starters)

Another common place you can run into issues like this is using DateTime's. If your unit test is running 'too fast' then it may break an assumption you had.

like image 155
wal Avatar answered Oct 24 '22 07:10

wal