Is it worth the effort to write tests that work with the browser (like using selenium or something)?

Question

It's pretty clearly worth it to write tests for stuff that happens on the server side, but I've heard that it is really hard to write unit test for UI stuff and that they are fragile and unreliable. I would love to have more confidence that changes that I make don't break major parts of import pages on my site though.

Any thoughts or experiences?

Answer 1

UI tests can be slow, fragile and painful to maintain, but some bugs can only be caught in UI tests. The important question isn't whether it is worth it to write UI tests, but how to keep your UI tests useful, stable and maintainable.

A common mistake is to use UI tests as a substitute for other tests. Theoretically, you can test a lot of functionality through UI tests, but there are many problems with that approach. For starters, some functionality may be very hard to test directly in the UI (especially exceptional conditions). Secondly, if a test fails, it often hard to see what the source of the problem is. Finally, the more code paths you test in UI tests, the slower the UI tests get. If you rely only on slow tests, your productivity gets worse, which increases the temptation to just "temporary" turn off broken tests.

My advice is to test as much as possible in unit tests and integration tests, create a good separation between your UI and your business logic, and use your UI tests to catch/prevent bugs that cannot be tested in other kinds of tests.

If you have many tests, consider creating multiple suites. I create one suite for UI tests, one for integration tests, and one for unit tests. The unit tests are very fast, so I run them as I develop my code (often via TDD). These are the tests that help me be productive.

The integration tests I run less often (perhaps after I've done implementing a bit of changes). The UI tests I run when I'm getting ready to submit a change (or when I write more UI tests, obviously).

One final bit of advice: consider writing your UI tests with a Domain Specific Language. This makes it easier to understand the tests (because they read as a set of user steps and not as a bunch of low-level browser actions). It also can make the code easier to maintain. For instance, instead of having every test go through the step-by-step browser actions to log the user in, you might see:

LoginPage loginPage = new LoginPage(selenium);
HomePage homePage = loginPage
    .enterUserName(TestUsers.Alice.USER_NAME)
    .enterPassword(TestUsers.Alice.PASSWORD)
    .submit();
assertThat(homePage.getBreadcrumbs(), equalTo("Home"));

Answer 2

It's easy, could be done by not-so-techy person and definitely worth the effort. UI testing is hard for desktop application, not so hard for web apps because you can move controls around and selenium will still be able to find them in HTML. No such luck for poor desktop developers.

Of course there's always a possibility you overdo it to the point where it doesn't give you much back compare to the effort but it's the same with normal unit testing. You should know when to stop.

Answer 3

I used them as part of development. cool thing about them is that you can execute them also in IE(you need selenium server for that). I used them only on development stage and dont treat them like unit tests. if i do some UI specific logic they do help alot.

the most important thing is to assign id to all used html elements. without it tests are very fragile. using comments also help.