Parse Fitnesse RESTFul XML output into TFS Test format

Question

I'm integrating a Fitnesse Acceptance test suite into a TFS based CI process.

I can run the Fitnesse Test suite in a RESTful manner (http://fitnesse.org/FitNesse.UserGuide.RestfulTests):

http://myfitnesseserver/MyTestSuite?suite&format=xml

and get back an XML document of test results.

I'd like to transform that into a format that TFS can interpret as number of tests passed / failed.

Any pointers?

Thanks

Answer 1

I have a similar goal at work, so I created a generic command-line Fitnesse test runner that executes a test or suite as a web request, parses the resulting XML and transforms it using the style sheet below, and finally writes the result to a file called "results.xml" in the %TestOutputDirectory% as specified by a Generic Test in Visual Studio.

The results file is loaded by Visual Studio and parsed as a summary results file that reports the number of child tests that pass or fail in a Fitnesse test or suite. Details of the output file format are documented here, but a simple example looks like this when run against Fitnesse's two minute example in the default Fitnesse wiki:

<?xml version="1.0" encoding="utf-8"?>
<SummaryResult>
  <TestName>TwoMinuteExample</TestName>
  <TestResult>Failed</TestResult>
  <ErrorMessage>6 right, 1 wrong, 0 ignores and 0 exceptions.</ErrorMessage>
  <InnerTests>
    <InnerTest>
      <TestName>TwoMinuteExample</TestName>
      <TestResult>Failed</TestResult>
      <ErrorMessage>6 right, 1 wrong, 0 ignores and 0 exceptions.</ErrorMessage>
    </InnerTest>
  </InnerTests>
</SummaryResult>

So, now it is possible to create a Visual Studio "Generic Test" in a test project for each Fitnesse test/suite you want to execute from Visual Studio or as part of a build. The generic test must specify the path to the generic test runner executable, the Fitnesse server, port and test/suite name in the wiki. It requires checking the box for the "summary results file" and entering "Results.xml" to get the detail to show up in the output of the test run or build.

I can share this code with you, or you could use the generic command line test runner and pipe the output into a small app that transforms the results using the style sheet below.

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet
    version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:msxsl="urn:schemas-microsoft-com:xslt"
    exclude-result-prefixes="msxsl"
    >
  <xsl:output method="xml" indent="yes"/>

  <xsl:variable name="GlobalRightCount" select="sum(//result/counts/right)"/>
  <xsl:variable name="GlobalIgnoresCount" select="sum(//result/counts/ignores)"/>
  <xsl:variable name="GlobalWrongCount" select="sum(//result/counts/wrong)"/>
  <xsl:variable name="GlobalExceptionsCount" select="sum(//result/counts/exceptions)"/>
  <xsl:variable name="GlobalFailureCount" select="$GlobalWrongCount + $GlobalExceptionsCount"/>

  <xsl:template match="testResults">
    <SummaryResult>
      <TestName>
        <xsl:value-of select="rootPath"/>
      </TestName>
      <xsl:choose>
        <xsl:when test="$GlobalFailureCount = 0">
          <TestResult>Passed</TestResult>
          <xsl:call-template name="GlobalErrorMessage"/>
        </xsl:when>
        <xsl:otherwise>
          <TestResult>Failed</TestResult>
          <xsl:call-template name="GlobalErrorMessage"/>
        </xsl:otherwise>
      </xsl:choose>
      <InnerTests>
        <xsl:for-each select="result">
          <InnerTest>
            <TestName>
              <xsl:value-of select="relativePageName"/>
            </TestName>
            <xsl:choose>
              <xsl:when test="sum(counts/wrong) + sum(counts/exceptions) = 0">
                <TestResult>Passed</TestResult>
                <xsl:call-template name="ResultErrorMessage"/>
              </xsl:when>
              <xsl:otherwise>
                <TestResult>Failed</TestResult>
                <xsl:call-template name="ResultErrorMessage"/>
              </xsl:otherwise>
            </xsl:choose>
          </InnerTest>
        </xsl:for-each>
      </InnerTests>
    </SummaryResult>
  </xsl:template>


  <xsl:template name="GlobalErrorMessage">
    <ErrorMessage><xsl:value-of select ="$GlobalRightCount"/> right, <xsl:value-of select ="$GlobalWrongCount"/> wrong, <xsl:value-of select ="$GlobalIgnoresCount"/> ignores and <xsl:value-of select ="$GlobalExceptionsCount"/> exceptions.</ErrorMessage>
  </xsl:template>

  <xsl:template name="ResultErrorMessage">
    <ErrorMessage><xsl:value-of select ="sum(counts/right)"/> right, <xsl:value-of select ="sum(counts/wrong)"/> wrong, <xsl:value-of select ="sum(counts/ignores)"/> ignores and <xsl:value-of select ="sum(counts/exceptions)"/> exceptions.</ErrorMessage>
  </xsl:template>
</xsl:stylesheet>

Update: Adding generic test runner code

This is definitely just a proof of concept and not necessarily a final solution. You may be able to combine this technique with Martin Woodward's answer to get the complete picture where the test list itself is dynamic. Or, you could alter the test runner to run all the tests it finds in an entire (sub)wiki. There are probably several other options here.

The following code is far from optimized, but shows the general process. You can paste this into a new console application project. Note that it requires command line parameters, and you can provide defaults for these in the project properties:

Sample command line: localhost 80 FitNesse.UserGuide.TwoMinuteExample

class Program
{
    static int Main(string[] args)
    {
        //Default to error unless proven otherwise
        int returnValue = 1; 

        try
        {
            List<string> commandLineArgs = args.ToList<string>();

            string host = args[0];
            int port = int.Parse(args[1]);
            string path = args[2];
            string testCommand = "suite";

            XmlDocument fitnesseResults = GetFitnesseResult(host, port, path, testCommand);
            XmlDocument visualStudioResults = TransformFitnesseToVisualStudioResults(fitnesseResults);
            visualStudioResults.Save("Results.xml");

            var testResultNode = visualStudioResults.DocumentElement.SelectSingleNode("TestResult");
            var value = testResultNode.InnerText;
            if (value == "Success")
            {
                returnValue = 0;
            }
        }
        catch (System.Exception ex)
        {
            Console.WriteLine(ex.ToString());
        }

        return returnValue;
    }

    private static XmlDocument GetFitnesseResult(string host, int port, string path, string testCommand)
    {
        UriBuilder uriBuilder = new UriBuilder("http", host, port, path, "?" + testCommand + "&format=xml");
        WebRequest request = HttpWebRequest.Create(uriBuilder.Uri);
        request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);

        WebResponse response = request.GetResponse();
        Stream responseStream = response.GetResponseStream();
        StreamReader responseReader = new StreamReader(responseStream);
        string responseString = responseReader.ReadToEnd();

        XmlDocument rawResults = new XmlDocument();
        rawResults.LoadXml(responseString);

        return (rawResults);
    }

    private static XmlDocument TransformFitnesseToVisualStudioResults(XmlDocument fitnesseResults)
    {
        XslCompiledTransform transformer = new XslCompiledTransform(false);
        string codeBase = Assembly.GetEntryAssembly().CodeBase;
        string directory = Path.GetDirectoryName(codeBase);
        string xsltPath = Path.Combine(directory, "FitnesseToSummaryResult.xslt");
        transformer.Load(xsltPath);

        MemoryStream resultsStream = new MemoryStream();
        transformer.Transform(fitnesseResults, null, resultsStream);
        resultsStream.Position = 0;
        XmlDocument results = new XmlDocument();
        results.Load(resultsStream);

        return (results);
    }
}