Console output missing in Azure DevOps release when enabling "Specify a batch size"

Question

We have a large test set running (based on NUnit) within Azure DevOps. Recently we enabled the "Rerun failed tests" option within the Visual Studio Test task. At first, this didn't work due to a bug in VSTest with handling custom test display names (which are required for our tests). Tests would still only run once. However, by setting a batch size, this issue is fixed and tests are finally retried correctly.

It works wonderful, except for one strange effect... for no apparent reason, the detailed logs are no longer showing our own custom output (generated by Console.WriteLine). This worked without problems before. All we get now is default output and the final test result.

Logs without "Specify a batch size" enabled are way more informative:

The logs themselves are still written: we know that as (almost) all the information is also included per-test. This is shown in the test results:

Also, simply disabling the batch size option makes the logs show up again.

Does anyone have an idea what causes this behavior, and how to fix it? So far, switching around between Console.Writeline/Trace.Writeline/Debug.Writeline etc. hasn't helped, and I haven't found much other info about this specific issue... and there are certain situations where having one complete log file is necessary (or way more practical), so it would be nice if we manage to have both retries and full logging.

Thanks in advance!

Answer 1

I posted the question on developercommunity.visualstudio.com as well. The discussion took a bit of time, but summarized, the following response was given:

"The supported behavior is to have the Trace information from the tests be present as part of the test results file (trx)/Standard console logs and not part of the build logs (as you have mentioned in your question). Do note this is also dependent on the test framework being used. There’s no plan to have the trace information flow to the build logs from the test. The different behavior you are seeing when you turn on Batching option is due to the fact the internal flow of execution changes a bit (and eventually all flows will converge to same as batch option in coming days). we recommend not to take dependency on build logs. Instead the test tab is the place you will get in better logs in context to test case/test run."

A full test run output should be available in the trx files found in the test run. Upon check, this is indeed the case:

When opening the file in Notepad++, I finally see the logs of my complete run:

- All these lines are written by using Console.WriteLine().

- If there are more trx files than one, then the largest file holds the logs of the complete run: the small file only has the logs for the selected test.

NOTE: We found that during a test run timeout, this file is not generated. This has been reported to Microsoft and is acknowledged as an issue:

"got the point here. we are working on Advanced diagnostics in vstest task. for instance we will abort the test if a test is taking more time to complete. in this case a dump of the test process will be created as well and ofcourse we will have trx also getting uploaded. the second scenario where not a perticular test is taking time but overall run ends time out we will take a dump of test process and abort the run. Dumps will help you debug the issue."

Somewhere in the future, this dump should be available to use.