Random nan errors when importing matplotlib.pyplot

Question

I am implementing a Kalman filter in numpy. It works fine, except for when I import matplotlib.pyplot to visualize the result:

import numpy as np 
import matplotlib.pyplot as plt # adding this line breaks things
import sys

The complete code is here. Let me emphasize that it seems to work correctly before I do the import; it prints a 100 x 2 array of numbers that are plausible. After adding the import, even without using anything from pyplot, all rows after a specific row are nan. The numbers that are not nan are the same as before.

My first thought was that it might be a name conflict, but it's not. You can easily see that the code doesn't have anything named "plt", besides, it's not consistent with the behavior described below.

The number of nan lines differs when I execute the file from Sublime Text instead of from the command line or when I add import matplotlib as mpl before the pyplot import. Again, non-nan numbers are the same as in the correctly functioning version.

Trying to debug only made me more confused. I added print statements to the problematic iteration of the main loop, which at first gave only nan matrices. When I added one more statement, print yt, however, printing the matrices that were nan suddenly had not-nan values. Also, moving the import sys statement before the import numpy as np one changes the number of nan rows. During experiments along these lines, I observed that when executing the same file several times, the values changed (by a lot, e.g. from 77 to 3.32686992e+297), and upon further repeated execution back to the original values, oscillating randomly between these two outputs. There is no saved state, file operations consist only of one call to np.loadtxt.

Further information that might help: I have Python 2.7.6 and Ubuntu 14.04, although on someone else's computer with Python 2.7.8 and spyder the behavior is similar.

What could be possible sources for this behavior? Right now, I'm thinking either witchcraft, coincidential mysterious hardware failures on both our computers, or an evil virus designed to frustrate Python programmers.

Answer 1

I can't reproduce the errors you're seeing*, so it's hard for me to pinpoint the cause. Having said that, one obvious source of numerical instability in your code is the matrix inversion operation on line 39.

In practice there are very few situations where you need to invert a matrix. In particular, you should never use matrix inversion to solve systems of linear equations - it is always faster and more numerically stable to use factorization instead.

You can replace your call to np.linalg.inv with np.linalg.solve like this:

# aux_k = np.linalg.inv(psi.dot(sigmatt1[t]).dot(psi.T) + rmat)
# k[t] = sigmatt1[t].dot(psi.T).dot(aux_k)

A = psi.dot(sigmatt1[t]).dot(psi.T) + rmat
B = sigmatt1[t].dot(psi.T)
k[t] = np.linalg.solve(A, B.T).T

See if that helps with your stability issues.

---

Update

You mentioned in the comments above that your numpy.__version__ == '1.8.2', but your matplotlib.__version__numpy__ == '1.5'. This probably means that matplotlib was built against an older (and probably incompatible) version of numpy (how did you install these libraries?).

I would recommend you try removing and reinstalling matplotlib.

---

* I've tried using numpy v1.8.2 and v1.10.0.dev-8bcb756, linked against either OpenBLAS 0.2.12 compiled from source or the standard CBLAS lib from the Ubuntu repositories. I've also tried both matplotlib v1.3.1 and v1.5.x.