Automated Windows UI Testing Approach

Question

We are looking to set up automated UI tests and were wondering what the best approach is, what are the potential pitfalls, is it expensive to set up?

Thanks in advance.

B

Answer 1

The biggest expense of automated testing is probably time. There are a lot of very expensive tools, but there are also free tools. The cost of even an expensive tool is not likely to match the cost of the time required to properly set up automated testing. As long as management understands that a significant upfront cost is required, and is willing to go through with it, then look out for these when actually automating the tests:

Pitfalls

Maintainability

There is a long term cost in automation in the form of maintenance. Remember that automated testing is software development. That means you have all the same potential problems as any other software. That also means that the same methods of improving maintainability in software apply to automated testing. (This is why I consider all those "professional" tools using vbscript instead of a proper OO language to be trash.)

Automated testing also has its own special maintainability problems. The biggest problem is that you're using a UI instead of an API. If you've ever had to use an unstable API you might begin to understand the pain of running a program through a constantly changing UI. Fortunately the solution to this is known, if not always well implemented: Object Mapping. You basically have some layer that connects to the UI on one side and code on the other. The code side remains as stable as possible while the UI side can change as often as needed.

An example from the framework I work on:

public Image GoImage
{
  get { return Browser.Image(Find.ById("BtnGo")); }
}

This example uses WatiN. With this I write lines in scripts such as GoImage.Click(), and if the id of the image tag ever changes, I don't change all of my scripts, I only update the mapping.

Tests that should not be automated

Not every test should be automated. Sometimes it will take longer to automate a test than it would to run it manually. If it's a test that you're only going to run once or just a few times, it's better to not automate it at all. You can mitigate this by creating ways of quickly creating automated tests. If appropriate, data-driven test automation is a great way of doing this. With our test automation framework, we can create new tests by modifying a dozen or so lines in an excel spreadsheet.

You should also be hesitant to create test scripts that are only partially automated. This most often occurs when you can automate the steps of the test, but the verification has to be done manually. In theory you could get some speed gain by letting the automation fly through the UI and stop while the user does their checks, but psychology gets in the way. Most people would tune out while the automation is running and require as much time to figure out what they're supposed to be checking as they would have to just run the entire test manually.

The Approach

Like I said before, automated testing is software development, so that's how you approach it. Here's the most basic design for test automation that I've found:

Interface Library

You need something that lets you programatically control the UI. This is the part that commercial tools tend to do well, but recently open source projects have appeared that also nicely handle this. For Windows UIs, there's white. I've never used it, but I like the API. Web automation really belongs to open source tools like watir and WatiN.

Framework

Framework is a general term for everything that you need to create yourself. Object mapping, helper functions, data-driven script runners. Commercial tools try to supply these for you, but I've never found a tool that does exactly what I need. I always roll my own here. This is where most of the maintenance work is, which is why I'm so down on tools that use weak languages like vbscript. I much prefer to create a framework using .NET.

Test Runner

You need something to actually run the tests. Commercial tools provide this as well, and here they do well enough. But they're really no better than a unit testing program. Yes, NUnit is just as useful for UI automated testing as for unit testing. You can also write a custom test runner fairly easily.

Logging and Results

You need some way to know if the test was successful or not. Most any existing logging library like log4n/log4j would work. Test runners also usually have this built in. Commercial tools usually do well at results too, as long as you avoid proprietary formats.

Test Scripts

Obviously you need the tests themselves.

And one final thing I want to say. Test automation can reduce the time it takes to do the same amount of testing, but it works better when you have the mindset of doing more testing in the same amount of time.

Answer 2

We've been doing Automated UI testing successfully using our own home-grown tools and libraries based on the Microsoft UI Automation framework. It has been a bit of steep learning curve, but worth it.

I recommend reading up on the Michael's Automation Stack to learn how you might structure your code. You might also find my Introduction to UI Automation useful.