Swing UI Testing Library Comparisons: FEST, WindowTester Pro, etc

Question

I'm not trying duplicate questions such as this one:

Unit testing framework for a Swing UI

What I'd like to know is, does anyone have any good comparisons for the various Swing Unit testing libraries such as:

• WindowTester Pro
• FEST
• etc...

We've never done any GUI testing so we're not familiar with the gotchas that may lie ahead.

Thanks in advance.

Answer 1

I have had some rather good experience with Abbot and FEST, both open source libraries for Swing UI testing.

Abbot seems not supported anymore; it was a bit hard to get into, because the recorder wasn't generating scripts "good enough". Actually, I've used the recorder to "learn" the script language (XML tags), and I finally wrote the scripts myself directly with a simple text editor. This worked quite well.

FEST takes another approach where you have to code (in Java) your UI tests. That makes it a tool reserved for Java developers, whereas Abbot could be used by other people (e.g. QA Team testers).

The main issues with both tools, and probably with any UI testing tool, are:

• to find a way to identify components uniquely without using their position or text content (which can change from one revision to another or makes it difficult to test the same application in a different Locale)
• to use correct timing in scripts: those testing tools can run your UI much faster than a human user, thus your UI may not be fast enough for them (e.g. it may take several dozens ms to open a dialog, even more to populate a table from a database)

For both issues, there is a solution though.

For components identification, I strongly advise to name all Swing Components (using Component.setName()) in the UI and use a naming strategy for that, which can ensure that there are never 2 components with the same name visible at the same time. In the guts-gui library, I have even developed a strategy that automatically names Swing components that are stored as fields in panels, this helps adding component names after the application has been coded.

For script timing, both frameworks accept a timeout value while waiting for a dialog to appear; it's up to you to choose the best value, considering the facts that your tests may run on different kinds of machine with more or less available power. You should use a timeout that is large enough to ensure that the script won't report false negatives (e.g. a dialog that appears after 1sec, whereas the script waits for only 500ms), but also not too long so that when there is a real error (e.g. an expected dialog never appears). I suggest to use timeouts ranging from 2 to 5 seconds, that should fit most testing platforms and most applications.

Hope this helps.

Answer 2

Jemmy provides a reasonably good capabilities for UI testing. Although, it wouldn't be a staight out of the box solution for JUnit testing, it could be easily extended to suit your purposes.

I am not sure about other UI testing tools, but when compared with RFT it provides you a handle of the actual UI object (RFT returns a proxy object). This could be handy from my experience.

It is an open-source project (licensed under CDDL) and is actively under development.

I think other popular (or used to be??) was jfcUnit. Though I don't think this is under active development.

Answer 3

There are many factors to consider. Record /Replay, Unit test support, Nature of code change, Licensing, Cost, multi-platform support, Testing with multiple look and feel, support for i18n testing ... what is your list look like?

Some comments on the tools we used.

IBM Rational Functional Tester :

This has ability to record scripts, and play back. It supports verification points. One of the biggest plus point is no code change is required. RFT modifies the JVM and uses java accessibility extensions to record and test. We use it mainly for Java (swing/awt with lot of 2D and dialog operations). It works with Browsers as well.

RFT expose two mechanisms to identify GUI elements. One uses object map. This very weak and has trouble in long term maintainability. Using "find" API is more programmer friendly, though it requires code change. Having all objects with proper name helps too.

Not at all suitable for unit testing.

Works with Windows and Linux.

Very costly floating license is in the range of 12000USD, fixed licenses will cost half of that. All nodes (recording the tests and running the tests) need license. Pricing is approximate and old, but it will give an idea.

Needs real GUI session, on windows. (It may be OK on linux with VNC)

Jemmy: We shifted to jemmy for new testing. Supports windows, linux. It used to be free, not sure what Oracle's plan on it. We added our test layer on top of jemmy - for assertions and other verification mechanisms. This presentation on 'jemmy testing toolkit' has some more details on jemmy.

Answer 4

In 2012 @AlexRuiz, the main developer behind FEST decided to stop development of FEST

The FEST codebase was later forked into AssertJ-Swing.

Skip FEST and start with AssertJ-Swing.