Where to store expected output of a test?

Question

Writing a test I expect the tested method to return certain outputs. Usually I'm checking that for a given database operation I get a certain output. My practice has usually been that of writing an array as a quick map/properties file in the test itself. This solution is quick, and is not vulnerable to run-time changes of an external file to load the expected results from.

A solution is to place the data in a java source file, so I bloat less the test and still get a compile-time checked test. How about this?

Or is loading the exepected results as resources a better approach? A .properties file is not good enough since I can have only one value per key. Is commons-config the way to go?

I'd prefer a simple solution where I name the properties per key, so for each entry I might have a doc-length and numFound property value (sounds like the elements of an xml node)?

How do you go about this?

Answer 1

You must remember about maintaining such tests. After writing several web services tests with Spring-WS test support I must admit that storing requests (test setup) and expected responses in external XML files wasn't such a good idea. Each request-response pair had the same name prefix as test case so everything was automated and very clean. But still refactoring and diagnosing test failures becomes painful. After a while I realized that embedding XML in test case as String, although ugly, is much easier to maintain.

In your case, I assume you invoke some database query and you get a list of maps in response. What about writing some nice DSL to make assertions on these structures? Actually, FEST-Assert is quite good for that.

Let's say you test the following query (I know it's an oversimplification):

List<Map<String, Object>> rs = db.query("SELECT id, name FROM Users");

then you can simply write:

assertThat(rs).hasSize(1);
assertThat(rs.get(0))
  .hasSize(2)
  .includes(
    entry("id", 7),
    entry("name", "John")
  )
);

Of course it can and should be further simplified to fit your needs better. Isn't it easier to have a full test scenario on one screen rather than jump from one file to another?

Or maybe you should try Fitnesse (looks like you are no longer doing unit testing, so acceptance testing framework should be fine), where tests are stored in wiki-like documents, including tables?

Answer 2

Yes, using resources for expected results (and also setup data) works well and is pretty common.

XML may well be a useful format for you - being hierarchical can certainly help (one element per test method). It depends on the exact situation, but it's definitely an option. Alternatively, JSON may be easier for you. What are you comfortable with, in terms of serialization APIs?