JUnit tests pass but PIT says the suite isn't green

Question

While trying to run a PIT mutation test I get the following error:

mutationCoverage failed: All tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.

The tests run just fine when I do a normal test build but while running the mutation tests phase they supposedly fail but no details are provided as to why. I have gone through the reasons listed on the PIT Testing FAQ but I still have no clue what could be wrong.

I tried:

• adding the -Dthreads=1 option to rule of any multi threading issue
• could not find any system properties unique the couple tests that are failing
• the tests are not ignored under normal runs

What are some other things I should try? Or other ways to debug what could be going on here?

Answer 1

The common causes of tests failing at the coverage stage are

1. PIT picking up tests not included/are excluded in the normal test config
2. Tests rely on an environment variable or other property set in the test config, but not set in the pitest config
3. Tests have a hidden order dependency that is not revealed during the normal test run
4. PIT doesn't like something in your tech stack - possibly a JUnit test runner

It sounds like you've eliminated 1 & 2. so that leaves 3 and 4.

Test order dependencies can be hard to spot. If the answer is yes to any of these you may have one.

• Does your codebase include mutable static state? (e.g in singletons)
• Do your tests hit a database (in memory or otherwise) where it is possible for state to persist between tests?
• Do your tests modify files on disk?

There are probably also many other causes not listed above.

If you are confident that order dependencies are impossible in your code base, that leaves a problem with these particular tests.

It's hard to guess what this might be without some code. Can you post a simplified version of the test that still fails?