Selenium + JUnit: test order/flow?

Question

I am using Selenium to test my java web app's html pages (JSPs actually). My web app requires a flow to access each pages (it is a small online game web app), as in: to get to page B you'd need to go to page A, enter some text and press a button to get to page B. Obviously I already have some tests to verify that page A is working properly.

I would like to be able to write more tests in order to check that after the tests for page A run I'll get my tests for page B running (and so on for the rest of the app). So in short: define some order in my tests somehow.

After doing lots of reading about testing in the last few days, I cannot find anything interesting on this specific subject. Hence I'm asking for advice now.

Possible Solutions I have identified:

1. Define (in the same test class) test methods for page A, then test methods for test B. Then order the execution of the test methods. But we know JUnit (but TestNG does) does not allow test methods execution ordering, see SO question selenium-junit-tests-how-do-i-run-tests-within-a-test-in-sequential-order

2. Grouping all the tests (for page A, page B, & so on) under one test method. But I've read it's bad, see SO question: junit-one-test-case-per-method-or-multiple-test-cases-per-method. Is it that bad when doing selenium test? I've seen some code doing it so I assume it may not be.

3. Grouping all the tests (for page A, page B, & so on) under one test method but use the JUnit's ErrorCollector class: ErrorCollector allows you to execute ordered checks in the same method and yields a specific error message if one fails BUT let the method (hence the checks) running until the end. This solution seems too "brutal" to me.

4. Use JUnit's TestSuite class: it runs test listed in the suite in the order test classes are defined in the suite. So that would involve having independent test methods to test page A in a test class (let's say TestA), then all the test methods to test page B in a test class (let's say TestB), and so on. Then insert those in a test suite such as @SuiteClasses( {TestA.class, TestB.class, TestC.class, ...} )

5. Use JUnit's TestSuite class in combination with JUnit's ErrorCollector class. Oh well, since we can, you may want to Group test per page in different classes, & on top of that group page tests "zones" using ErrorCollector. This solution may be very usefull if you have a very dense web page or other reasons.

6. Quite radical: Use another tool such as TestNG to have access to features such as test method ordering.

Note: I imagine some would recommend the last solution (migrate to TestNG) but I'd like to hear other ideas/opinions tied to JUnit too. As in, if I'm working in a team that is not able (for some reason) to migrate to another testing framework, then how would they address this test ordering issue?

Answer 1

Why to migrate? You can use JUnit for unit-testing and another framework for higher-level testing. In your case it is a kind of acceptance or functional or end-to-end, it is not that important how you name it. But important is to understand that these tests are not unit. They stick to different rules: they are more complex, run longer and less often, they require complex setup, external dependencies and may sporadically fail. Why not use another framework for them (or even another programming language)?

Possible variants are:

• BDD frameworks: already mentioned Cucumber, JDave, JBehave, Spock. For instance, Spock is JUnit based and Groovy =)
• TestNG

If adding another framework is not an option: you enumerated more options for JUnit then I could imagine =) I would put the whole test script for the flow in one test method and would organize test code into "Drivers". That means that your end-to-end tests do not call the methods of your application or Selenium API directly, but wrap them into methods of Driver components which hide API complexity and look like statements of what happens or what is expected. Look at the example:

@Test 
public void sniperWinsAnAuctionByBiddingHigher() throws Exception {
    auction.startSellingItem();

    application.startBiddingIn(auction);
    auction.hasReceivedJoinRequestFrom(ApplicationRunner.SNIPER_XMPP_ID);

    auction.reportPrice(1000, 98, "other bidder");
    application.hasShownSniperIsBidding(auction, 1000, 1098);

    auction.hasReceivedBid(1098, ApplicationRunner.SNIPER_XMPP_ID);

    auction.reportPrice(1098, 97, ApplicationRunner.SNIPER_XMPP_ID);
    application.hasShownSniperIsWinning(auction, 1098);

    auction.announceClosed();
    application.hasShownSniperHasWonAuction(auction, 1098);
} 

A snippet is taken from the "Growing Object-Oriented Software Guided by Tests". The book is really great and I highly recommend to read it.

This is real end-to-end test that uses real XMPP connection, Openfire jabber server and WindowLicker Swing GUI-testing framework. But all this stuff if offloaded to Driver components. And in your test you just see how different actors communicate. And it is ordered: after application started bidding we check that auction server received join request, then we instruct auction server to report new price and check that it is reflected in UI and so on. The whole code is available on github.

The example on github is complex, because the application is not as trivial as it usually happens with book examples. But that book gives it gradually and I was able to built the whole application from scratch following the book guide. In fact, it is the sole book I ever read on TDD and automated developer testing that gives such a thorough and complete example. And I've read quite a lot of them. But note, that Driver approach does not make your tests unit. It just allows you hide complexity. And it can (and should) be used with other frameworks too. They just give you additional possibilities to split your tests into sequential steps if you need; to write a user readable test cases; to externalize test data into CSV,Excel tables, XML files or database, to timeout your tests; to integrate with external systems, servlet and DI containers; to define and run separately test groups; to give more user-friendly reports and so on.

And about making all your tests unit. It is not possible for anything excluding something like utility libraries for math, string processing and so on. If you have application that is completely unit tested that it means either that you test not all application or you do not understand what tests are unit and what are not. The first case may be OK, but everything that is not covered must be tested and retested manually by developers, testers, users or whoever. It is quite common but it better to be conscious decision rather than casual one. Why you cannot unit test everything?

There are a lot of definitions of unit tests and it leads to holy war) I prefer the following: "Unit test is test for program unit in isolation". Some people say: "hey, unit is my application! I test login and it is simple unit function". But there is also pragmatics that hides in isolation. Why do we need to differ unit tests from others? Because it is our first safety net. They must be fast. You commit often (to git, for example) and you run them at least before each commit. But imagine, that "unit" tests takes 5 minutes to run. You will either run them less often or you will commit less often or you will run just one test case or even one test method at the time, or you will wait say each 2 minutes for tests to complete in 5 minutes. An in that 5 minutes you'll go to Coding Horror where you'll spend the next 2 hours =) And unit tests must never fail sporadically. If they do that - you will not trust them. Hence, the isolation: you must isolate slowness and sources of sporadic failures from your unit tests. Hence, isolation means that unit tests should not use:

• File system
• Network, Sockets, RMI and so on
• GUI
• Multithreading
• External libraries expecting test framework and supporting simple libraries like Hamcrest

And unit tests must be local. You want to have just one or so tests failing when you've made a defect within 2 minutes of coding, not a half of the whole suite. That means that you are very limited in testing stateful behavior in unit tests. You should not make a test setup that makes 5 state transitions to reach preconditions. Because fail in first transition will break at least 4 tests for following transitions and one more test that you currently write for the 6th transition. And any non-trivial application has quite a lot of flows and state transitions in it. So this cannot be unit tested. For the same reason unit tests must not use changeable shared state in database, static fields, Spring context or whatever. This is exactly the reason why JUnit creates new instance of test class for every test method.

So, you see, you cannot fully unit test a web app, no matter how you recode it. Because it has flows, JSPs, servlet container and probably more. Of course, you can just ignore this definition, but it is damn useful) If you agree that distinguishing unit tests from other tests is useful, and this definition helps to achieve that then you'll go for another framework or at least another approach for tests that are not unit, you'll create separate suites for separate kinds of test and so on.

Hope, this will help)

Answer 2

You can implement your own Runner that will probably wrap other Runner and sort tests according to criteria yo9u defined. This is if you really need this.

But I do not think it is so necessary. I understand that if page A does not work the pass to B will not work too. So you want to run the test A first and then run test A->B. But does it really make sense? If for example test A->B runs first and fails because it cannot arrive to page A other test that verifies test A will fail too. So, Both tests will fail and id does not depend on the tests order.

If however you mean that you want to use test A as a set up operation of test B it is very bad practice. You can use the logic of test A as a start of test B but you should not couple between 2 tests. One obvious reason is that this makes debugging very hard. To debug test A->B you have to run both A and A->B tests that means that you probably have to run all tests (at least within one test case).