Unit tests sometimes failing, sometimes passing

Question

I have a mocha unit test suite.

When I run them locally, everything works fine. When I run them on our Jenkins CI server, they sometimes fail, sometimes pass.

I'm just not able to reproduce why they fail. What might cause this behavior?

Answer 1

Tests can fail intermittently for a number of reasons and identifying why they fail is often revealing about your codebase and environment.

Here are a few possible causes:

• Shared objects - singletons that hold state can cause problems between tests if the test environment isn't reset to a well known state. if if your test runner executes tests in a non-deterministic order you may see random errors that are actually exposing corrupted state issues

• Environmental and external dependencies - any external object that can hold state can cause unpredictable results

• Timing - sometimes tests are written with timeouts or thread sleeps that are too specific. If the build server is operating under heavy load these timeouts may not be long enough

As general guidance, tests must be:

• isolated: tests focus on one unit at a time
• repeatable: produces the same results each time
• independent: the order in which tests are executed should not matter

Answer 2

I would try to narrow down the problem by reducing the number of executors to 1. If the tests are still failing intermittently, then you have a Test Run War, otherwise (given they run fine locally) it looks like a Resource Leakage.

More info at http://xunitpatterns.com/Erratic%20Test.html