Finding @Ignore'd tests that are now passing

Question

For a decent sized open source project where developers come and go, someone may fix a bug without realizing that someone else a while back committed a disabled unit test (a la @Ignore). We'd like to find the passing tests that are disabled so we can enable them and update the bug tracker, CC list, and anything else downstream.

What is the best way to occasionally run all @Ignore'd tests and identify the ones that are now passing? We are using Java 1.6 with JUnit4, building our project with ant and transitioning to gradle. We use Jenkins for CI.

A few ideas:

1. Permanently replace all of our @Ignore annotations with a conditional ignore http://www.codeaffine.com/2013/11/18/a-junit-rule-to-conditionally-ignore-tests/

2. Run a custom JUnit4 class runner that changes the behavior of @Ignore. https://stackoverflow.com/a/42520871

3. Temporarily comment out all @Ignore annotations so that they run. However we'd need a way to negate the failures.

Answer 1

Sorry, this is not a solution, but rather another alternative that has worked for me: My key point was to not modify existing (1000s) of unit tests. So no broad code changes. No new Annotations, certainly not temporarily.

What I did was override the JUnit @Ignore detection and make that conditional, via classpath prepends: Check in a separate control file if that test/class is listed or disabled. This is based on package/FQCN/method name and Regexp patterns. If covered, run it even though it still has @Ignore in the unchanged original JUNit Test source.

Log the outcome, amend the control file. Rinse and repeat.