Using Robot Framework for ATDD

Question

I would like to hear other people's experience with Robot Framework for automated acceptance testing.

What are its major strengths and weaknesses as well as any comparison with other frameworks (mainly Fitnesse and Selenium)?

The code that will be tested is real-time, legacy code, mainly in C++.

Answer 1

I have used Robot Framework at three different companies spanning about six years at the time that I write this, and all have been successful in one way or another.

My experiences

Company 1

The first place I used Robot was a Java-based web application for a top-tier Internet travel company. We used Robot with Jython, which let us create keywords in Java and directly act with the system under test. We used Selenium to drive the web browser, with most of our testing being on Firefox. While the testing effort was largely successful with the QA organization, the development organization failed to embrace it -- they preferred to use JUnit rather than Robot.

Company 2

My second company I feel was an unqualified success. We used Robot in a multitude of ways. The primary use was to drive Internet Explorer for acceptance and regression testing of a highly successful commercial .NET web application.

We also used it to test an iPad app by combining Selenium with Appium. We used Robot to test the RESTful services that supplied data to the app. We wrote specialized keywords that let us do image analysis, and we also used Robot tests to do quick analysis of our training equipment before each training session. We had keywords that let us snapshot a database before a test, and to restore the database after a test.

We also started using Robot to help with manual testing. We put manual test cases in Robot, which let us leverage Robot's reporting and tagging features. When these tests would run, they would prompt the user to perform manual steps which was proven to be much more efficient than when we had testers reading manual steps out of a test case management tool or Word document.

Company 3

The third company was a large company ($1B revenue) with a fairly large IT staff. They had testers with very low technical skill (I remember one who had no idea what a command line was). We had one team dedicated to writing a core set of keywords, and to providing training and mentoring to the other teams. I think the use of Robot was instrumental in getting some use out of the least-skilled testers, though even with high level keywords it was a struggle with them.

Company 4

Most recently I moved to a very small company with just a handful of developers and no dedicated testers. We embraced the use of page objects with Robot Framework, and we now have an exceptionally stable suite of easily readable high level acceptance tests. The use of Robot at this company has been an unqualified success.

Strengths

The biggest strength of Robot is its flexibility. We've used Robot to support manual testing, testing of SOAP and REST services, web-based UI testing, database testing, testing of images, and the testing of mobile apps. Because Robot is so easy to extend with additional libraries, there's almost nothing you can't test if you're willing to roll up your sleeves and write some keywords. Depending on your setup, you can write keywords in Python, Java, .NET, or really just about any language through the Robot remote API.

Because Robot test cases and keywords are written in plain text, you're not locked in to using a proprietary tool to create or view tests. Users are able to pick the tool of their choice -- Visual Studio, Eclipse, Emacs, Notepad, etc. There is also a Robot-specific IDE (RIDE), though I don't recommend it. Also, because the files are plain text, they integrate well with other software tools -- they are easy to diff and merge, to search, etc.

Weaknesses

Robot makes it easy to write low-quality tests. While there are facilities to document keywords and test cases, and to use human-readable names for keywords, test cases and variables, there's no good way to enforce best practices. Writing a large body of tests and keywords requires discipline. As the saying goes, Robot gives you plenty of rope to hang yourself with.

Another weakness is that the pace of progress on Robot is fairly slow. On the plus side, Robot is robust and relatively bug free, so there's not a big need for frequent patches. However, there are feature requests that languish in their issue tracker for years with no movement, which can be discouraging.

Summary

In all companies we've enjoyed being able to leverage the flexibility provided by Robot's syntax to create data-driven tests, BDD-style tests, as well as simple procedural tests. And in all cases, because the tests are plain text files, the test assets were easy to manage with our SCM tools (Mercurial, Subversion, and Git)

For me, Robot has proven to be easy to use, extremely easy to extend, and useful for a wide range of testing duties, from unit-testing of Python functions, to testing of web services, browser-based and tablet UI testing, the testing of images, the testing of databases, and even to improve the efficiency of manual testing.