How to run only failed nunit tests in teamcity build

Question

I have many tests in selenium that took almost 6 hours to complete. After the execution, few of them failed due to time-out error or unavoidable errors. I want to run only failed tests in that case. I am using teamcity to run the nunit tests. Any idea how can I rerun only failed tests.

Answer 1

There's an option to run failed tests first as part of TeamCity's nUnit integration - but you cannot limit it to only failing tests, nothing like this is built in.

You have a couple options here in my opinion:

• Use a Selenium grid / other infrastructure to run your Selenium tests in parallel to reduce your test run time. This seems to be the root of your problem: Currently the time to do a full test run is prohibitive. If you can run your Selenium tests fast, you wouldn't need to run them selectively. Generally using the built in nUnit runner is a bad choice long term because it doesn't allow for parallelism.
• Based on the failures you are seeing determine the nUnit test fixtures and do a custom nUnit run where you just pass in those test fixtures to rerun - again there is no integration for this and you would have to use the command line nUnit runner.

Answer 2

There are a few options depending on the testing framework you are using so you can make teamcity launch tests on more flexible criteria by using command line options. We used this workaround in our instable test environment.

With NUnit it is possible to use the filtered outputfile (according to failed test) as the given testlist for the next run. There are various ways to fetch and set the list and you can find details here: https://github.com/nunit/docs/wiki/Console-Command-Line

There are two quick options:

• you save/add the list with failed tests in a file (with use of powershell script)
• read the last test-result file and use the where clause to get the failed tests

With each run the list with tests should shrink.

Be aware:

• builds are not easy comparable (teamcity estimates the execution time or if you have failure conditions based on metric changes)
• (new) errors in other tests will stay undetected until you launch all tests again