Automated testing of GUI [closed]

Question

This question isn't about unit-testing. And it is for a desktop product.

This is about testing of the gui and testing that the right stuff is input in the right text box at the right time.

A company I used to work at used WinRunner (different department so I don't know much more that that), but that has now been shutdown by HP but they don't seem that bothered whether you stay with HP or go elsewhere. You can't read about the product until you've signed up which is annoying.

The tool has to work with MFC (non-negotiable) and the ideal tool will also...

• be automated.
• be scriptable.
• work with different screen resolutions automatically.
• be able to 'spy' on individual static text boxes, etc.
• intuitive enough so non-programmers can create the scripts.
• have reporting tools, including email of individual users.

What do other SO users do for automated GUI testing?

Answer 1

We use the SAFS framework for Rational Robot (RRAFS). There are also SAFS implementations for WinRunner (WRAFS) and it looks like they have a new "Image-Based Testing" implementation, which I'm not familiar with.

This framework does a nice job of seperating the UI implementation from the test scripts. I've tested four releases of a web application developed by two different teams (one team using classic ASP, one using ASP.NET) and I only had to change the application map of my UI objects, the tests themselves didn't need to change.

That said, the language of the framework is cumbersome and takes getting used to. It's not very robust, in terms of language constructs, but with some effort you can do anything you need to. It's sort of like "programming" in Windows Batch language, but for tests ;)

To address your individual requirements above:

1) The tool has to work with MFC (non-negotiable). The SAFS framework uses a 3rd party "record-playback" tool to drive the tests, like Rational Robot or Mercury WinRunner. If that tool can interact with MFC apps, then the framework can. I don't know how the "Image-Based Testing" implementation drives the tests, but I'd guess it can also work with MFC.

2) be automated. The SAFS framework integrates with the STAF framework, which will allow you to automate the execution of your tests. I have a proof-of-concept test that uses STAF to start a VM image from a pool of images, install the application under test, run the RRAFS test, and put the results on a web server for others to get at.

3) be scriptable. Yes, but as mentioned, it's not the most robust programming language. I wrote an Excel add-in that our testers use to write their tests that simplifies things a little bit.

4) work with different screen resolutions automatically. Yes, since it's looking "under the covers" at the UI objects and not the screen. Except for maybe the "Image-based Testing" option...

5) be able to 'spy' on individual static text boxes, etc. Yes, you can wait for a UI object to appear, disapper, to have a value, for a value to be changed, etc.

6) intuitive enough so non-programmers can create the scripts. With some training. We've had limited success. Some QA folks can write the tests, some struggle.

7) have reporting tools, including email of individual users. Yes, using the STAF framework you can post results to a web server, send out emails, etc.

Answer 2

Lots of good answers here, but I wanted to address this goal statement, specifically:

• intuitive enough so non-programmers can create the scripts

I can understand why you'd want this, but it's a lot harder than you might think. While you can find any number of tools out there that'll claim to make writing scripts easy, in practice, you're going to need at least some people on your automation team who understand programming. Writing scripts that are reasonably robust is going to involve one or more of looping, if/then/else, and subroutine calls. Not the kind of thing that non-programmers are going to find intuitive.

Be especially wary of the idea that you can "record" a person using a tool, then play it back for testing. That sort of "automation" is often so brittle that you'll end up modifying or re-recording the script for nearly every change in the software.

Answer 3

Coming from a strong Mercury/HP background, I would highly recommend using QuickTest Professional for your GUI testing. It has a lot of the same functionality as WinRunner, but without a lot of code. Simple GUI checks can be done through the QTP interface with minimal, if any, custom VB code. Checks for text next to boxes can be done with simple compares using the datasheet in QTP.

If you are used to WinRunner, and know VBScript (not so much TSL), then I would definantly look at QTP.

As far as your other requirements, QTP also has the Spy feature, like WinRunner, that will list all properties and actions you can perform on objects. And as for simplicity of use, at my old job, we would have business or system testers create simple smoke scripts, then I would take them and code them for more in-depth testing (multiple data values, error checking, etc). And as for reporting, QTP will do simple reporting of pass/fail/warning on tags that you put in, along with custom data you can input. So you could use a case statement to populate your output values based on your results. It won't do e-mail naitevly, but if you integrate with TestDirector/QualityCenter, you can setup through there, along with automating the kick-off of your scripts, and parameterizing data right from there (which is nice to send back to testers to have populate data without being involved in the script itseld).

Pat