What does Flaky: Hypothesis test produces unreliable results mean?

Question

I am using the hypothesis python package for testing.

I am getting the following error:

Flaky: Hypothesis test_visiting produces unreliable results: Falsified on the first call but did not on a subsequent one

As far as I can tell, the test is working correctly.

How do I get around this?

Answer 1

It means more or less what it says: You have a test which failed the first time but succeeded the second time when rerun with the same example. This could be a Hypothesis bug, but it usually isn't. The most common cause of this is that you have a test which depends on some external state - e.g. if you're using a system random number generator rather than a Hypothesis provided one, or if your test creates some files and only fails if the files did not exist at the start of the test. The second most common cause of this is that your failure is a recursion error and the example which triggered it at one level of function calls did not at another.

You haven't really provided enough information to say what's actually happening, so it's hard to provide more specific advice than that. If you're running a recent version of Hypothesis (e.g. 1.9.0 certainly does it) you should have been given quite detailed diagnostics about what is going on - it will tell you what the original exception you got was and it will report if the values passed in seemed to change between calls.

Answer 2

One thing that I haven't seen mentioned a lot, and it might be a relatively new behavior, is that you may want to raise the deadline of your tests. In my experience, if one test case fails due to a missed deadline and the second one passes, you'll see it as a "flaky" test failure.

@hypothesis.settings(deadline=500)

It's been hard for me to find some proper documentation about this behavior that I could personally understand fully, but this seems to fix it for me.