Automated Testing Framework

Question

I am trying build an automated testing framework for some of our internal web services (java) at work. Each service has a set of APIs (3-5), although this could be relatively easy to achieve, the problem comes with some APIs which do not behave as pure functions, ex: something like persistX, this could store something in a database and returns an exception incase of failure. There is no easy way to validate since there is no output here.

So I was wondering if this could a a bit generalized, say while testing the API, the user could provide a simple plugin or script to the framework of some sort that could validate the test. This is just one idea would be great if someone can tell me some better ones or any resources about the same.

Thanks

Answer 1

I recommend the robot framework. It is a keyword driven framework written in python. Because of that, you can run it in the JVM with jython which means you can extend it with java code (or python, of course). I've sucessfully used it to call APIs, then verify the result by peeking into a database, or querying the file system.

It also works on the .NET platform, has a selenium module for testing the front end, a jenkins plugin, and several other tools. It's very extensible and flexible.

Answer 2

What you are looking at is the application of black-box and white-box testing and the tools that support both.

For the web services that return a proper response you can perform black-box testing by verifying the data in returned response. SoapUI is the best tool for this.

For the APIs which do not behave as pure functions, you do white-box testing by verifying its side-effects like persistence, event generation, logging etc. For this you like programmable tools and SoapUI may or may not be the correct option.

We do both at our work and after evaluating multiple tools/frameworks (SoapUI, RSSPec, Robotframework), I chose Spock. Why spock?

1. It allows you to write intention revealing tests in BDD style
2. We are Java shop and we want to use the same familiar language for automation as well but with simplified syntatic sugar. And spock is all groovy based.
3. Excellent Webdriver/Selenium 2 support (including PageFactory) with Geb
4. It is built on top of JUNIT so all the plugins of JUNIT can be leveraged (code coverage, hudson/jenkins integration, etc)
5. Lot of webservice APIs and XML DSLs (no need to work with XPATH for simple scenarios)
6. Simplified setup (unlike robotframework it doesnt require python, jython setup)

etc....