Automated Browser Testing using selenium, Nunit, Selenium Grid, C#, webdriver/remote control

Question

I have been researching on how to automate browser testing for a number of weeks now using all kind of different methods. Seleniums website is very vague on which is the best route to take.

I have installed

1. Selenium Webdriver
2. Remote Control
3. Selenium Grid
4. Apache Ant
5. Nunit (and pretty much everything else you could need to do this)

I finally give up trying on my own and want to know the best way to do this.

I want to be able to

1. Load the same webpage on a number of different browsers
2. Load the same webpage on a number of virtual machines(which I have set up)
3. Be able to take snapshots comparing the different browser results.

I have knowledge of programming in C# and would prefer to run my tests through Nunit.

Can anyone give me directions or point me to a website that already has them? Thank you.

Answer 1

I have built up a test framework using junit with Selenium WebDriver that satisfies every one of your points. While its not exactly what you're asking for, I feel it may be beneficial to you regardless.

Load the same webpage on a number of different browsers

Using Selenium's grid, this is very simple to set up. Set up some virtual machines with the environments you're looking to test in. In our environment, for example, we have a grid running with four nodes (as virtual machines) with a setup like the following

• Windows with IE7 and FireFox
• Windows with IE8 and FireFox
• Windows with IE9 and Firefox
• Linux with FireFox

Note that Selenium recommends that only one instance of IE be allowed to run on the Windows nodes. On each of the aforementioned nodes, there is one instance of the specified IE and five instances of the specified FF allowed to run at any given time. With the grid setup and the hub configured, firing off tests is a breeze. In WebDriver, use the DesiredCapabilities object to set up the desired environment and then just send the test off and wait for the result to return.

Platform desiredPlatform;
DesiredCapabilities desiredCapabilities;
desiredPlatform = Platform.LINUX;
desiredCapabilities = DesiredCapabilities.firefox();
desiredCapabilities.setPlatform(desiredPlatform);
desiredCapabilities.setVersion("11");

WebDriver driver = new RemoteWebDriver("http://hubURL", desiredCapabilities);

Load the same webpage on a number of virtual machines(which I have set up)

I solved this one by forcing the tests to run, albeit in an unconvential way, in a threaded manner. Each JUnit test uses a shared thread library I put together which creates all the necessary RemoteWebDrivers needed in separate threads. Each of these threads runs simultaneously on its node while the parent thread sits and waits for all to terminate. Then on to the next test which is run multithreaded as well.

There were a couple problems I encountered such as retrieving the Junit stack traces in all of the child threads. I solved this by redirecting Std.err to a bytestream on the parent thread. All errors get routed to that stream which I then convert to a string and print out to Std.out at the end of each test. The html pages generated at the end of the tests include Std.out which worked out perfectly.

Be able to take snapshots comparing the different browser results

While I have gotten this to work, there are some inherent problems with grabbing screenshots remotely. IE will return black screenshots if the process is running as a service. The workaround was to just run the jar from the command line and keep the user logged in, in which case the screenshots return correctly. This is a known issue in the browser and there really is no nice solution to the problem. Taking screenshots works roughly like this

WebDriver augmentedDriver = new Augmenter().augment(driver);
TakesScreenshot ss = (TakesScreenshot) augmentedDriver;
String base64Screenshot = ss.getScreenshotAs(OutputType.BASE64);
byte[] decodedScreenshot = Base64.decodeBase64(base64Screenshot.getBytes());
FileOutputStream fos = new FileOutputStream(new File(imageName));
fos.write(decodedScreenshot);
fos.close();

which saves the captured screenshot from the remote machine's running browser onto the local machine.

In reality, browser automation is still struggling to stabilize itself. There are a number of important features, such as the ones you're asking about, that just aren't implemented solidly that I know of in any framework. With time, though, I'm sure a lot of this will settle down and QA developers everywhere will rejoice.