Rerun failed .NET unit test from PowerShell script or .NET

Question

The Visual Studio Test task in Azure Devops has a really cool feature, the ability to retry the failed unit tests. This is a great feature when you have a long test time and some tests that are flaky. This Test task in Azure Devops works for various test platforms like xUnit, NUnit & MSTest. (So tests written for .NET)

Would it be possible to get the same behavior from script? I prefer xUnit or NUnit and running the script in PowerShell.

For xUnit there a -method "name":

run a given test method (can be fully specified or use a wildcard; i.e., 'MyNamespace.MyClass.MyTestMethod' or '*.MyTestMethod') if specified more than once, acts as an OR operation

NUnit has a --where=EXPRESSION syntax source:

An expression indicating which tests to run. It may specify test names, classes, methods, categories or properties comparing them to actual values with the operators ==, !=, =~ and !~. See Test Selection Language for a full description of the syntax.

But not sure how to collect the failed test for xUnit or NUnit to get it all working.

Of course, fixing the flaky test would be better, but that's sometimes not that easy.

Update: running from . NET/C# (which could be triggered in PowerShell) is also acceptable

Answer 1

You can do a little of "manual work" to get the result using regular expressions in powershell.

the example is with XUnit. so what you have to do is store the result of the dotnet test project.csproj in a variable. so an example will be like the next

Test run for C:\Users\Tigrex\source\repos\ConsoleApp1\XUnitTestProject1\bin\Debug\netcoreapp2.2\XUnitTestProject1.dll(.NETCoreApp,Version=v2.2) Microsoft (R) Test Execution Command Line Tool Version 16.3.0 Copyright (c) Microsoft Corporation. All rights reserved. Starting test execution, please wait... A total of 1 test files matched the specified pattern. X XUnitTestProject1.UnitTest1.ThisIsAnotherFailedTestYesAgain [11ms] Error Message: Assert.Equal() Failure Expected: 2 Actual: 1 Stack Trace: at XUnitTestProject1.UnitTest1.ThisIsAnotherFailedTestYesAgain() in C:\Users\Tigrex\source\repos\ConsoleApp1\XUnitTestProject1\UnitTest1.cs:line 33 X XUnitTestProject1.UnitTest1.ThisIsAnotherFAiledTest [1ms] Error Message: Assert.Equal() Failure Expected: 2 Actual: 1 Stack Trace: at XUnitTestProject1.UnitTest1.ThisIsAnotherFAiledTest() in C:\Users\Tigrex\source\repos\ConsoleApp1\XUnitTestProject1\UnitTest1.cs:line 22 X XUnitTestProject1.UnitTest1.TestToFail [1ms] Error Message: Assert.Equal() Failure Expected: 2 Actual: 1 Stack Trace: at XUnitTestProject1.UnitTest1.TestToFail() in C:\Users\Tigrex\source\repos\ConsoleApp1\XUnitTestProject1\UnitTest1.cs:line 16 Total tests: 5 Passed: 2 Failed: 3 Total time: 1.2764 Seconds

as you can see there is some common patterns which mainly is Error Message that gives you the hint to know where to look for, in this case, xUnit indicates the error ones by X testname [{time}ms] Error Message

if you match that text against a regular expression you can get the desired response: I used this one: X\s*(\S*)\s\[\d*ms\]\s*Error Message I'm sure it can be improved (I'm not a master on regex) but it does its job. you can remove Error Message for example. anyway, I keep going.

once you match the result you only need to get the group for each result, which in this case I stored it in TestName. and call the dotnet test ...

$result = dotnet test XUnitTestProject1/XUnitTestProject1.csproj 

$regex = 'X\s*(?<TestName>\S*)\s\[\d*ms\]\s*'

$matches = [regex]::Matches($result, $regex)
Foreach ($failedTest IN $matches)
{
    $failedTestName = $failedTest.Groups['TestName'].Value
   dotnet test --filter   "FullyQualifiedName=$failedTestName" 
}

this line $failedTestName = $failedTest.Groups['TestName'].Value is necessary, if you try to pass the .Groups.. in the FullyQualifiedName string, PowerShell understand them as a literal string.

you need to do the same to calculate the times and the percentage.

Also for the first iteration is easier because you can ran all test in one go, but from the second and far you cant. so necessary list (to keep the tests that are failing) is necessary.

something like this will do the job.

$times = 1

$result = dotnet test XUnitTestProject1/XUnitTestProject1.csproj 
$regexFailedtests = 'X\s*(?<TestName>\S*)\s\[\d*ms\]\s*'
$FailedTestMatches = [regex]::Matches($result, $regexFailedtests)

$totalTestExecutedRegex = 'Total tests:\s*(?<TotalTest>\d*)'
$totalTests = [regex]::Matches($result, $totalTestExecutedRegex)[0].Groups['TotalTest'].Value -as [int]

$totalTesPassedRegex = 'Passed:\s*(?<Passed>\d*)'
$totalTestsPassed = [regex]::Matches($result, $totalTesPassedRegex)[0].Groups['Passed'].Value -as [int]


#convert the failed test into a list of string, so it can be looped.
$listFailedTest = New-Object Collections.Generic.List[string]
Foreach ($failedTest IN $FailedTestMatches)
{
    $failedTestName = $failedTest.Groups['TestName'].Value
    $listFailedTest.Add($failedTestName)
}

$percentage = ($totalTestsPassed*100)/$totalTests #Calculate the percentage

while($times -lt 5 -and $percentage -lt 70) {#5 loops or > 70% of test working

    $listFailedTestInsideDo = New-Object Collections.Generic.List[string]
    $listFailedTestInsideDo = $listFailedTest;  #do a copy of the main list
    $listFailedTest = New-Object Collections.Generic.List[string] ##empty the main list.
    Foreach ($failedTestName IN $listFailedTestInsideDo)
    {

       $result2 = dotnet test --filter   "FullyQualifiedName=$failedTestName" 

       if($result2 -match'Passed:\s*\d*') #if contains passed then it worked
       {
            totalTestsPassed++

        }else{
        $listFailedTest.Add($failedTestName) #add in new List for the new loop
       }
    }

    $percentage = ($totalTestsPassed*100)/$totalTests

    $times++
}