I want Canopy web testing results to show in VS 2013 test explorer... and I'm SO CLOSE

Question

I'm trying to figure out how to get the test results for Canopy to show in the VS test explorer. I can get my tests to show up and it will run them but it always shows a pass. It seems like the Run() function is "eating" the results so VS never sees a failure.

I'm sure it is a conflict between how Canopy is nicely interpreting the exceptions it gets into test results because normally you'd want Run() to succeed regardless of the outcome and report its results using its own reports.

Maybe I should be redirecting output and interpreting that in the MS testing code?

So here is how I have it set up right now...

The Visual Studio Test Runner looks at this file for what it sees as tests, these call the canopy methods that do the real testing.

open canopy
open runner
open System
open Microsoft.VisualStudio.TestTools.UnitTesting

[<TestClass>]
type testrun() = 

    // Look in the output directory for the web drivers    
    [<ClassInitialize>]
    static member public setup(context : TestContext) =
        // Look in the output directory for the web drivers    
        canopy.configuration.ieDir <- "."
        canopy.configuration.chromeDir <- "."

        // start an instance of the browser
        start ie
        ()

    [<TestMethod>]
    member x.LocationNoteTest() =
        let myTestModule = new myTestModule()
        myTestModule.all()
        run()


    [<ClassCleanup>]
    static member public cleanUpAfterTesting() =
        quit() 
        ()

myTestModule looks like

open canopy
open runner
open System

type myTestModule() =

    // some helper methods

    member x.basicCreate() =
        context "The meat of my tests"

        "Test1" &&& fun _ ->
            // some canopy test statements like...
            url "http://theURL.com/"

            "#title" == "The title of my page"

            //Does the text of the button match expectations?
            "#addLocation" == "LOCATION"

            // add a location note
            click ".btn-Location"

     member x.all() = 
        x.basicCreate()
        // I could add additional tests here or I could decide to call them individually

Answer 1

I have it working now. I put the below after the run() for each test.

    Assert.IsTrue(canopy.runner.failedCount = 0,results.ToString())

so now my tests look something like:

    [<TestMethod>]
    member x.LocationNoteTest() =
            let locationTests = new LocationNote()

            // Add the test to the canopy suite
            // Note, this just defines the tests to run, the canopy portion
            // of the tests do not actually execute until run is called.
            locationTests.all()

            // Tell canopy to run all the tests in the suites.
            run()

            Assert.IsTrue(canopy.runner.failedCount = 0,results.ToString())

Canopy and the UnitTesting infrastructure have some overlap in what they want to take care of. I want the UnitTesting infrasturcture to be the thing "reporting" the summary of all tests and details so I needed to find a way to "reset" the canopy portion so that I didn't have to track the last known state from canopy and then compare. So for this to work your canopy suite can only have one test but we want to have as many as we want at the UnitTesting level. To adjust for that we do the below in the [].

    runner.suites <- [new suite()]
    runner.failedCount <- 0
    runner.passedCount <- 0

It might make sense to have something within canopy that could be called or configured when the user wants to use a different unit testing infrastructure around canopy.

Additionally I wanted the output that includes the error information to appear as it normally does when a test fails so I capture the console.out in a stringBuilder and clear that in []. I set it up in by including the below [] where common.results is the StringBuilder I then use in the asserts.

    System.Console.SetOut(new System.IO.StringWriter(common.results))