Delete or comment out non-working JUnit tests?

Question

I'm currently building a CI build script for a legacy application. There are sporadic JUnit tests available and I will be integrating a JUnit execution of all tests into the CI build. However, I'm wondering what to do with the 100'ish failures I'm encountering in the non-maintained JUnit tests. Do I:

1) Comment them out as they appear to have reasonable, if unmaintained, business logic in them in the hopes that someone eventually uncomments them and fixes them

2) Delete them as its unlikely that anyone will fix them and the commented out code will only be ignored or be clutter for evermore

3) Track down those who have left this mess in my hands and whack them over the heads with the printouts of the code (which due to long-method smell will be sufficently suited to the task) while preaching the benefits of a well maintained and unit tested code base

Answer 1

If you use Junit 4 you can annotate that tests with @Ignore annotation.

If you use JUnit 3 you can just rename tests so they don't start with test.

Also, try to fix tests for functionality you are modifying in order to not make code mess larger.

Answer 2

Follow the no broken window principle and take some action towards a solution of the problem. If you can't fix the tests, at least:

1. Ignore them from the unit tests (there are different ways to do this).
2. Enter as many issue as necessary and assign people to fix the tests.

Then to prevent such situation from happening in the future, install a plug in similar to Hudson Game Plugin. People gets assigned points during continuous integration, e.g.

• -10 break the build <-- the worse
• -1 break a test
• +1 fix a test
• etc.

Really cool tool to create a sense of responsibility about unit tests within a team.

Answer 3

The failing JUnit tests indicate that either

1. The source code under test has been worked on without the tests being maintained. In this case option 3 is definitely worth considering, or
2. You have a genuine failure.

Either way you need to fix/review the tests/source. Since it sounds like your job is to create the CI system and not to fix the tests, in your position i would leave a time-bomb in the tests. You can get very fancy with annotated methods with JUnit 4 (something like @IgnoreUntil(date="2010/09/16")) and a custom runner, so or you can simply add an an if statement to the first line of each test:

  if (isBeforeTimeBomb()) {
    return;
  }

Where isBeforeTimeBomb() can simply check the current date against a future date of your choosing. Then you follow the advice given by others here and notify your development team that the build is green now, but is likely to explode in X days unless the timebombed tests are fixed.

Answer 4

• Comment them out so that they can be fixed later.
• Generate test coverage reports (with Cobertura for example). The methods that were supposed to be covered by the tests that you commented out will then be indicated as not covered by tests.

Answer 5

If they compile but fail: leave them in. That will get you a good history of test improvements over time when using CI. If the tests do not compile but break the build, comment them out and poke the developers to fix them.

This obviously does not preclude using option 3 (hitting them over the head), you should do that anyway, regardless of what you do with the tests.