Cubic spline memory error

Question

On a computer with 4GB of memory this simple interpolation leads to a memory error:

(based on: http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html)

import numpy as np
from scipy.interpolate import interp1d

x = np.linspace(0, 10, 80000)
y = np.cos(-x**2/8.0)
f2 = interp1d(x, y, kind='cubic')

I thought about cutting the data into chunks, but is there a way I can perform this cubic spline interpolation without requiring so much memory? Why does it even get in trouble?

Answer 1

If you look at the traceback when the error occurs, you'll see something like:

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-4-1e538e8d766e> in <module>()
----> 1 f2 = interp1d(x, y, kind='cubic')

/home/warren/local_scipy/lib/python2.7/site-packages/scipy/interpolate/interpolate.py in __init__(self, x, y, kind, axis, copy, bounds_error, fill_value)
    390         else:
    391             minval = order + 1
--> 392             self._spline = splmake(x, y, order=order)
    393             self._call = self.__class__._call_spline
    394 

/home/warren/local_scipy/lib/python2.7/site-packages/scipy/interpolate/interpolate.py in splmake(xk, yk, order, kind, conds)
   1754 
   1755     # the constraint matrix
-> 1756     B = _fitpack._bsplmat(order, xk)
   1757     coefs = func(xk, yk, order, conds, B)
   1758     return xk, coefs, order

MemoryError: 

The function that is failing is scipy.interpolate._fitpack._bsplmat(order, xk). This function creates a 2-d array of 64-bit floats with shape (len(xk), len(xk) + order - 1). In your case, this is over 51GB.

Instead of interp1d, see if InterpolatedUnivariateSpline works for you. For example,

import numpy as np
from scipy.interpolate import InterpolatedUnivariateSpline

x = np.linspace(0, 10, 80000)
y = np.cos(-x**2/8.0)
f2 = InterpolatedUnivariateSpline(x, y, k=3)

I don't get a memory error with this.