Solid Java unit test automation? (JUnit/Hamcrest/...)

Question

Intent

I am looking for the following:

• A solid unit testing methodology
  1. What am I missing from my approach?
  2. What am I doing wrong?
  3. What am I doing which is unnecessary?
• A way to get as much as possible done automatically

Current environment

• Eclipse as IDE
• JUnit as a testing framework, integrated into Eclipse
• Hamcrest as a "matchers" library, for better assertion readability
• Google Guava for precondition validation

Current approach

Structure

• One test class per class to test
• Method testing grouped in static nested classes
• Test method naming to specify behaviour tested + expected result
• Expected exceptions specified by Java Annotation, not in method name

Methodology

• Watch out for null values
• Watch out for empty List<E>
• Watch out for empty String
• Watch out for empty arrays
• Watch out for object state invariants altered by code (post-conditions)
• Methods accept documented parameter types
• Boundary checks (e.g. Integer.MAX_VALUE, etc...)
• Documenting immutability through specific types (e.g. Google Guava ImmutableList<E>)
• ... is there a list for this? Examples of nice-to-have testing lists:
  • Things to check in database projects (e.g. CRUD, connectivity, logging, ...)
  • Things to check in multithreaded code
  • Things to check for EJBs
  • ... ?

Sample code

This is a contrived example to show some techniques.

---

MyPath.java

import static com.google.common.base.Preconditions.checkArgument; import static com.google.common.base.Preconditions.checkNotNull; import java.util.Arrays; import com.google.common.collect.ImmutableList; public class MyPath {   public static final MyPath ROOT = MyPath.ofComponents("ROOT");   public static final String SEPARATOR = "/";   public static MyPath ofComponents(String... components) {     checkNotNull(components);     checkArgument(components.length > 0);     checkArgument(!Arrays.asList(components).contains(""));     return new MyPath(components);   }   private final ImmutableList<String> components;   private MyPath(String[] components) {     this.components = ImmutableList.copyOf(components);   }   public ImmutableList<String> getComponents() {     return components;   }   @Override   public String toString() {     StringBuilder stringBuilder = new StringBuilder();     for (String pathComponent : components) {       stringBuilder.append("/" + pathComponent);     }     return stringBuilder.toString();   } } 

---

MyPathTests.java

import static org.hamcrest.Matchers.is; import static org.hamcrest.collection.IsCollectionWithSize.hasSize; import static org.hamcrest.collection.IsEmptyCollection.empty; import static org.hamcrest.collection.IsIterableContainingInOrder.contains; import static org.hamcrest.core.IsEqual.equalTo; import static org.hamcrest.core.IsNot.not; import static org.hamcrest.core.IsNull.notNullValue; import static org.junit.Assert.assertThat; import org.junit.Test; import org.junit.experimental.runners.Enclosed; import org.junit.runner.RunWith; import com.google.common.base.Joiner; @RunWith(Enclosed.class) public class MyPathTests {   public static class GetComponents {     @Test     public void componentsCorrespondToFactoryArguments() {       String[] components = { "Test1", "Test2", "Test3" };       MyPath myPath = MyPath.ofComponents(components);       assertThat(myPath.getComponents(), contains(components));     }   }   public static class OfComponents {     @Test     public void acceptsArrayOfComponents() {       MyPath.ofComponents("Test1", "Test2", "Test3");     }     @Test     public void acceptsSingleComponent() {       MyPath.ofComponents("Test1");     }     @Test(expected = IllegalArgumentException.class)     public void emptyStringVarArgsThrows() {       MyPath.ofComponents(new String[] { });     }     @Test(expected = NullPointerException.class)     public void nullStringVarArgsThrows() {       MyPath.ofComponents((String[]) null);     }     @Test(expected = IllegalArgumentException.class)     public void rejectsInterspersedEmptyComponents() {       MyPath.ofComponents("Test1", "", "Test2");     }     @Test(expected = IllegalArgumentException.class)     public void rejectsSingleEmptyComponent() {       MyPath.ofComponents("");     }     @Test     public void returnsNotNullValue() {       assertThat(MyPath.ofComponents("Test"), is(notNullValue()));     }   }   public static class Root {     @Test     public void hasComponents() {       assertThat(MyPath.ROOT.getComponents(), is(not(empty())));     }     @Test     public void hasExactlyOneComponent() {       assertThat(MyPath.ROOT.getComponents(), hasSize(1));     }     @Test     public void hasExactlyOneInboxComponent() {       assertThat(MyPath.ROOT.getComponents(), contains("ROOT"));     }     @Test     public void isNotNull() {       assertThat(MyPath.ROOT, is(notNullValue()));     }     @Test     public void toStringIsSlashSeparatedAbsolutePathToInbox() {       assertThat(MyPath.ROOT.toString(), is(equalTo("/ROOT")));     }   }   public static class ToString {     @Test     public void toStringIsSlashSeparatedPathOfComponents() {       String[] components = { "Test1", "Test2", "Test3" };       String expectedPath =           MyPath.SEPARATOR + Joiner.on(MyPath.SEPARATOR).join(components);       assertThat(MyPath.ofComponents(components).toString(),           is(equalTo(expectedPath)));     }   }   @Test   public void testPathCreationFromComponents() {     String[] pathComponentArguments = new String[] { "One", "Two", "Three" };     MyPath myPath = MyPath.ofComponents(pathComponentArguments);     assertThat(myPath.getComponents(), contains(pathComponentArguments));   } } 

---

Question, phrased explicitly

• Is there a list of techniques to use to build a unit test? Something much more advanced than my oversimplified list above (e.g. check nulls, check boundaries, check expected exceptions, etc.) perhaps available in a book to buy or a URL to visit?

• Once I have a method that takes a certain type of parameters, can I get any Eclipse plug-in to generate a stub for my tests for me? Perhaps using a Java Annotation to specify metadata about the method and having the tool materialise the associated checks for me? (e.g. @MustBeLowerCase, @ShouldBeOfSize(n=3), ...)

I find it tedious and robot-like to have to remember all of these "QA tricks" and/or apply them, I find it error-prone to copy and paste and I find it not self-documenting when I code things as I do above. Admittedly, Hamcrest libraries go in the general direction of specialising types of tests (e.g. on String objects using RegEx, on File objects, etc) but obviously do not auto-generate any test stubs and do not reflect on the code and its properties and prepare a harness for me.

Help me make this better, please.

PS

Do not tell me that I am just presenting code which is a silly wrapper around the concept of creating a Path from a list of path steps supplied in a static factory method please, this is a totally made-up example but it shows a "few" cases of argument validation... If I included a much longer example, who would really read this post?

Answer 1

1. Consider using ExpectedException instead of @Test(expected.... This is because if for example you expect a NullPointerException and your test throws this exception in your setup (before calling the method under test) your test will pass. With ExpectedException you put the expect immediately before the call to the method under test so there is no chance of this. Also, ExpectedException allows you to test the exception message which is helpful if you have two different IllegalArgumentExceptions that might be thrown and you need to check for the correct one.

2. Consider isolating your method under test from the setup and verify, this will ease in test review and maintenance. This is especially true when methods on the class under test are invoked as part of setup which can confuse which is the method under test. I use the following format:

   public void test() {    //setup    ...     // test (usually only one line of code in this block)    ...     //verify    ... } 

3. Books to look at: Clean Code, JUnit In Action, Test Driven Development By Example

   Clean Code has an excellent section on testing

4. Most example I have seen (including what Eclipse autogenerates) have the method under test in the title of the test. This facilitates review and maintenance. For example: testOfComponents_nullCase. Your example is the first I have seen that uses the Enclosed to group methods by method under test, which is really nice. However, it adds some overhead as @Before and @After do not get shared between enclosed test classes.

5. I have not started using it, but Guava has a test library: guava-testlib. I have not had a chance to play with it but it seems to have some cool stuff. For example: NullPointerTest is quote:

• A test utility that verifies that your methods throw {@link * NullPointerException} or {@link UnsupportedOperationException} whenever any * of their parameters are null. To use it, you must first provide valid default * values for the parameter types used by the class.

Review: I realize the test above was just an example but since a constructive review might be helpful, here you go.

1. In testing getComponents, test the empty list case as well. Also, use IsIterableContainingInOrder.

2. In testing of ofComponents, it seems that it would make sense to call getComponents or toString to validate that it properly handled the various non-error cases. There should be a test where no argument is passed to ofComponents. I see that this is done with ofComponents( new String[]{}) but why not just do ofComponents()? Need a test where null is one of the values passed: ofComponents("blah", null, "blah2") since this will throw an NPE.

3. In testing ROOT, as has been pointed out before, I suggest calling ROOT.getComponents once and doing all three verifications on it. Also, ItIterableContainingInOrder does all three of not empty, size and contains. The is in the tests is extraineous (although it is linguistic) and I feel is not worth having (IMHO).

4. In testing toString, I feel it is very helpful to isolate the method under test. I would have written toStringIsSlashSeparatedPathOfComponents as follows. Notice that I do not use the constant from the class under test. This is because IMHO, ANY functional change to the class under test should cause the test to fail.

   @Test      public void toStringIsSlashSeparatedPathOfComponents() {           //setup     String[] components = { "Test1", "Test2", "Test3" };           String expectedPath =  "/" + Joiner.on("/").join(components);       MyPath path = MyPath.ofComponents(components)     // test    String value = path.toStrign();     // verify    assertThat(value, equalTo(expectedPath));    }  

5. Enclosed will not run any unit test that is not in an inner class. Therefore testPathCreationFromComponents would not be run.

Finally, use Test Driven Development. This will ensure that your tests are passing for the right reason and will fail as expected.

Answer 2

I see you put a lot of effort to really test your classes. Good! :)

My comments/questions would be:

• what about mocking? you do not mention any tool for this
• it seems to me that you care a lot about the nitty-gritty details (I do not say they are not important!), while neglecting the business purpose of tested class. I guess it comes from the fact you code code-first (do you?). What I would suggest is more TDD/BDD approach and focus on business responsibilities of the tested class.
• not sure what this gives you: "Method testing grouped in static nested classes"?
• regarding auto-generation of test stubs etc. Simply put: don't. You will end up testing implementation instead of behaviour.