Python/numpy locating Runtime Warning

Question

I have a numpy script sitting on an app server, it gets called thousands of times and once in a blue moon I get a Runtime Warning:

/usr/local/lib/python2.7/dist-packages/scipy/stats/stats.py:2417: RuntimeWarning: invalid value encountered in double_scalars
r = (r_num / r_den)

1. Not sure where this occurs.
2. Why this occurs.
3. The consequence it has on the code if any. everything passes the eye test and unit test.

But again I'm not sure if I'm looking at the right place because the chance of this warrning happening is less than 1%

how can I get python to print out the location of the warning?

Answer 1

If you put

np.seterr(all='raise')

near the beginning of your script, exceptions will be raised instead of warnings. That will halt your script with a nice traceback which will give you information about where the error is occurring.

You could then put a try...except around the line in your code that raises the exception, and use the except clause to log the value of relevant variables.

---

Also, the RuntimeWarning you posted says the warning is originating in stats.py, line 2417. This seems to be in the pearsonr function. Googling "invalid value encountered in double_scalars" yielded this SO question which suggests

from scipy.stats.stats import pearsonr

X = [4, 4, 4, 4, 4, 4]
Y = [4, 5, 5, 4, 4, 4]

pearsonr(X, Y)

raise the RuntimeWarning. This suggests that you are occasionally calling pearsonr with inputs that result in division by zero (as user3453425 stated)-- maybe due to one of the inputs being constant, and hence having a standard deviation of zero.

In this case pearsonr(X, Y) returns (nan, 1.0). So make sure you handle the case when the pearson correlation coefficient is undefined (nan).