Automated acceptance testing - UI or API?

Question

I've been researching automated acceptance testing over the last few days, learning about BDD & JBehave, FitNesse & Slim, Selenium & WebDriver, etc.

I've just viewed this video by Robert C. Martin where he demonstrates how to use FitNesse to write and maintain such tests. Towards the end, somebody asks if these tests hit the UI. Martin goes on to explain that coupling acceptance tests to the UI can be costly since changes to the UI are quite frequent. I could guess also that such tests could only be written after the UI has been developed, which would bring testers behind schedule by definition.

I have to ask: what is the alternative? Martin seems to be implying that tests should be hitting a hidden layer that would manipulate the application's business layer. My understanding is that this would require additional work, not to mention that it would expose a new API which would need to be secured once in a production environment.

Could hitting the business layer through application services be sufficient?

What has been your experience?

Thanks for sharing!

Answer 1

Testing through the UI or hitting the business layer directly can be seen as two different types of tests, with different advantages and disadvantages.

If you're testing the UI directly, then you're testing what the user sees, and you don't have to change code in order to be able to test it. However, it becomes quite hard to test corner cases, or how the system reacts to exceptional conditions (such as exceptions). It is as Robert Martin says, brittle. If your interface changes, you need to change your tests. So testing through the UI depends upon the maturity of your UI. Later on in the project, the UI is more stable, so testing through the UI makes more sense. Also, testing some stuff through the UI is hard or convoluted.

If you're testing the business layer, you can then more easily test corner conditions, and you are less susceptible to changes in the UI. However, as you say, the software has to be written in such a way as to allow testing of this kind. You may even have to expose a new interface to allow it, which then has to be maintained. In my experience, it is sometimes quite hard to get the developers to support this sort of interface. It's seen as not as important as the real interface. But it's always possible. This sort of interface needs buy-in from the developers, otherwise you risk it being 'not supported' over time.

If you don't have the external interface, then try asking for it. You never know, you might be able to persuade them that it's a good idea. It's easier at the start of a project.

If you have the external interface, then use that to test your business logic.

Otherwise, you'll have to use the UI to test these things. One approach would be to use the UI to do smoke testing, to answer the following question(s): Is this software testable? Think of the common things you have to test every time you get a build from dev. Can I login, can I logout, does the main page appear, can I do a simple order? Pick 5 or 6 of these things, and build an automated test suite to test these things. Use these tests as a guide as to how much functionality you can actually test through the UI, and how useful it is.

You can then use this as an argument when you go to the developers and ask for the external interface.

Answer 2

Unfortunately you need both. In general you would want to automate the business layer tests with something like fitness. Avoiding the UI basically gives you assurance that the defined business rules always work. Automating thru the UI can require a lot more maintenance. But on the plus side since much of the UI uses mechanisms provided by the platform (e.g. c#/Winforms) the bulk of your UI tests can be only when changes are made if you are well covered with other types of tests (like business layer). Automated tests need to be maintained and updated but UI tests tend to require more maintenance and are often less reliable over time.