Python-Predicting/Extrapolating future data given a data set

Question

I'm very new to Python. I have a data set and I'm trying to use numPy/sciPy to predict/extrapolate future data points. Is there a simple way to come up with a mathematical function(say, a Sine function) that fits my current data, and then I could pass new values into that function to get my prediction?

Here's what I have but I don't think it's doing what I want:

import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

def main():

    y = [8.3, 8.3, 8.3, 8.3, 7.2, 7.8, 7.8, 8.3, 9.4, 10.6, 10.0, 10.6, 11.1, 12.8,
         12.8, 12.8, 11.7, 10.6, 10.6, 10.0, 10.0, 8.9, 8.9, 8.3, 7.2, 6.7, 6.7, 6.7,
         7.2, 8.3, 7.2, 10.6, 11.1, 11.7, 12.8, 13.3, 15.0, 15.6, 13.3, 15.0, 13.3,
         11.7, 11.1, 10.0, 10.6, 9.4, 8.9, 8.3, 8.9, 6.7, 6.7, 6.0, 6.1, 8.3, 8.3,
         10.6, 11.1, 11.1, 11.7, 12.2, 13.3, 14.4, 16.7, 14.4, 13.3, 12.2, 11.7,
         11.1, 10.0, 8.3, 7.8, 7.2, 8.0, 6.7, 7.2, 7.2, 7.8, 10.0, 12.2, 12.8,
         12.8, 13.9, 15.0, 16.7, 16.7, 16.7, 15.6, 13.9, 12.8, 12.2, 10.6, 9.0,
         8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 10.0, 10.6, 11.1, 12.0, 11.7,
         11.1, 13.0, 13.3, 13.0, 11.1, 10.6, 10.6, 10.0, 10.0, 10.0, 9.4, 9.4,
         8.9, 8.3, 9.0, 8.9, 9.4, 9.0, 9.4, 10.6, 11.7, 11.1, 11.7, 12.8, 12.8,
         12.8, 13.0, 11.7, 10.6, 10.0, 10.0, 8.9, 9.4, 7.8, 7.8, 8.3, 7.8, 8.9,
         8.9, 8.9, 9.4, 10.0, 10.0, 10.6, 11.0, 11.1, 11.1, 12.2, 10.6, 10.0, 8.9,
         8.9, 9.0, 8.9, 8.3, 8.9, 8.9, 9.4, 9.4, 9.4, 8.9, 8.9, 8.9, 9.4, 10.0,
         11.1, 11.7, 11.7, 11.7, 11.7, 12.0, 11.7, 11.7, 12.0, 11.7, 11.0, 10.6,
         9.4, 10.0, 8.3, 8.0, 7.2, 5.6, 6.1, 5.6, 6.1, 6.7, 8.0, 10.0, 10.6, 11.1,
         13.3, 12.8, 12.8, 12.2, 11.1, 10.0, 10.0, 10.0, 10.0, 9.4, 8.3]    
    x = np.array(np.arange(len(y)))        

    fitting_parameters, covariance = curve_fit(fit, x, y)
    a = fitting_parameters[0]
    b = fitting_parameters[1]
    c = fitting_parameters[2]
    d = fitting_parameters[3]

    for x_predict in range(len(y) + 1, len(y) + 24):
        next_x = x_predict
        next_y = fit(next_x, a, b, c, d)

        print("next_x: " + str(next_x))
        print("next_y: " + str(next_y))
        y.append(next_y)

    plt.plot(y)
    plt.show()

def fit(x, a, b, c, d):
    return a*np.sin(b*x + c) + d

I tried to curve_fit and univariatespline my data but that only fits my current data and smooths my points, respectively. My point is that these tools are only "fitting" my data but aren't actually giving me a function that I can use to get future points.

I thought I could use Discrete Fourier Transforms because my data is periodic and looked like it could be described as a summation of Sines and Cosines. But once I got a frequency domain from the time domain, I was stuck on how to "extrapolate" in order to predict future periods and points in the time domain:

import numpy as np
import matplotlib.pyplot as plt

mydata = [8.3, 8.3, 8.3, 8.3, 7.2, 7.8, 7.8, 8.3, 9.4, 10.6, 10.0, 10.6, 11.1, 12.8,
         12.8, 12.8, 11.7, 10.6, 10.6, 10.0, 10.0, 8.9, 8.9, 8.3, 7.2, 6.7, 6.7, 6.7,
         7.2, 8.3, 7.2, 10.6, 11.1, 11.7, 12.8, 13.3, 15.0, 15.6, 13.3, 15.0, 13.3,
         11.7, 11.1, 10.0, 10.6, 9.4, 8.9, 8.3, 8.9, 6.7, 6.7, 6.0, 6.1, 8.3, 8.3,
         10.6, 11.1, 11.1, 11.7, 12.2, 13.3, 14.4, 16.7, 14.4, 13.3, 12.2, 11.7,
         11.1, 10.0, 8.3, 7.8, 7.2, 8.0, 6.7, 7.2, 7.2, 7.8, 10.0, 12.2, 12.8,
         12.8, 13.9, 15.0, 16.7, 16.7, 16.7, 15.6, 13.9, 12.8, 12.2, 10.6, 9.0,
         8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 10.0, 10.6, 11.1, 12.0, 11.7,
         11.1, 13.0, 13.3, 13.0, 11.1, 10.6, 10.6, 10.0, 10.0, 10.0, 9.4, 9.4,
         8.9, 8.3, 9.0, 8.9, 9.4, 9.0, 9.4, 10.6, 11.7, 11.1, 11.7, 12.8, 12.8,
         12.8, 13.0, 11.7, 10.6, 10.0, 10.0, 8.9, 9.4, 7.8, 7.8, 8.3, 7.8, 8.9,
         8.9, 8.9, 9.4, 10.0, 10.0, 10.6, 11.0, 11.1, 11.1, 12.2, 10.6, 10.0, 8.9,
         8.9, 9.0, 8.9, 8.3, 8.9, 8.9, 9.4, 9.4, 9.4, 8.9, 8.9, 8.9, 9.4, 10.0,
         11.1, 11.7, 11.7, 11.7, 11.7, 12.0, 11.7, 11.7, 12.0, 11.7, 11.0, 10.6,
         9.4, 10.0, 8.3, 8.0, 7.2, 5.6, 6.1, 5.6, 6.1, 6.7, 8.0, 10.0, 10.6, 11.1,
         13.3, 12.8, 12.8, 12.2, 11.1, 10.0, 10.0, 10.0, 10.0, 9.4, 8.3] 

sp = np.fft.rfft(mydata)
freq = np.fft.rfftfreq(len(mydata), d= 1.0)

plt.subplot(211)
plt.plot(mydata)
plt.subplot(212)
plt.plot(freq, sp, 'r')
plt.show()

I understand that extrapolation can be dangerous and unreliable, but for the purposes of this project, I'm just trying to get a working prediction function that I can graph.

Your help is greatly appreciated.

Answer 1

Here is a way to interpolate by representing your periodic data as a Fourier Series. The coefficients used in the Fourier Series are obtained by taking the discrete FFT.

I don't recommend this -- you can see below that the interpolation is not what one would consider intuitively as very good -- but since I had mentioned it in the comments, I'll follow through and show some code :)

import numpy as np
import matplotlib.pyplot as plt
import scipy.fftpack as fftpack

def fft_inverse(Yhat, x):
    """Based on http://stackoverflow.com/a/4452499/190597 (mtrw)"""
    Yhat = np.asarray(Yhat)
    x = np.asarray(x).reshape(-1, 1)
    N = len(Yhat)
    k = np.arange(N)
    total = Yhat * np.exp(1j * x * k * 2 * np.pi / N)
    return np.real(total.sum(axis=1))/N

mydata = [8.3, 8.3, 8.3, 8.3, 7.2, 7.8, 7.8, 8.3, 9.4, 10.6, 10.0, 10.6, 11.1, 12.8,
         12.8, 12.8, 11.7, 10.6, 10.6, 10.0, 10.0, 8.9, 8.9, 8.3, 7.2, 6.7, 6.7, 6.7,
         7.2, 8.3, 7.2, 10.6, 11.1, 11.7, 12.8, 13.3, 15.0, 15.6, 13.3, 15.0, 13.3,
         11.7, 11.1, 10.0, 10.6, 9.4, 8.9, 8.3, 8.9, 6.7, 6.7, 6.0, 6.1, 8.3, 8.3,
         10.6, 11.1, 11.1, 11.7, 12.2, 13.3, 14.4, 16.7, 14.4, 13.3, 12.2, 11.7,
         11.1, 10.0, 8.3, 7.8, 7.2, 8.0, 6.7, 7.2, 7.2, 7.8, 10.0, 12.2, 12.8,
         12.8, 13.9, 15.0, 16.7, 16.7, 16.7, 15.6, 13.9, 12.8, 12.2, 10.6, 9.0,
         8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 10.0, 10.6, 11.1, 12.0, 11.7,
         11.1, 13.0, 13.3, 13.0, 11.1, 10.6, 10.6, 10.0, 10.0, 10.0, 9.4, 9.4,
         8.9, 8.3, 9.0, 8.9, 9.4, 9.0, 9.4, 10.6, 11.7, 11.1, 11.7, 12.8, 12.8,
         12.8, 13.0, 11.7, 10.6, 10.0, 10.0, 8.9, 9.4, 7.8, 7.8, 8.3, 7.8, 8.9,
         8.9, 8.9, 9.4, 10.0, 10.0, 10.6, 11.0, 11.1, 11.1, 12.2, 10.6, 10.0, 8.9,
         8.9, 9.0, 8.9, 8.3, 8.9, 8.9, 9.4, 9.4, 9.4, 8.9, 8.9, 8.9, 9.4, 10.0,
         11.1, 11.7, 11.7, 11.7, 11.7, 12.0, 11.7, 11.7, 12.0, 11.7, 11.0, 10.6,
         9.4, 10.0, 8.3, 8.0, 7.2, 5.6, 6.1, 5.6, 6.1, 6.7, 8.0, 10.0, 10.6, 11.1,
         13.3, 12.8, 12.8, 12.2, 11.1, 10.0, 10.0, 10.0, 10.0, 9.4, 8.3] 

Yhat = fftpack.fft(mydata)

fig, ax = plt.subplots(nrows=2, sharex=True)
xs = np.arange(len(mydata))
ax[0].plot(xs, mydata)

new_xs = np.linspace(xs.min(), xs.max(), len(mydata)*1.5)
new_ys = fft_inverse(Yhat, new_xs)
ax[1].plot(new_xs, new_ys)

plt.xlim(xs.min(), xs.max())
plt.show()

---

Here is how to use scipy.optimize to find parameters to fit a model function which can then be used to interpolate at arbitrary x-coordinates. The fit using a single sin is still pretty horrible, but I'll post the code just to show how to use scipy.optimize anyway:

import numpy as np
import matplotlib.pyplot as plt
import scipy.optimize as optimize

mydata = np.array(
    [8.3, 8.3, 8.3, 8.3, 7.2, 7.8, 7.8, 8.3, 9.4, 10.6, 10.0, 10.6, 11.1, 12.8,
     12.8, 12.8, 11.7, 10.6, 10.6, 10.0, 10.0, 8.9, 8.9, 8.3, 7.2, 6.7, 6.7, 6.7,
     7.2, 8.3, 7.2, 10.6, 11.1, 11.7, 12.8, 13.3, 15.0, 15.6, 13.3, 15.0, 13.3,
     11.7, 11.1, 10.0, 10.6, 9.4, 8.9, 8.3, 8.9, 6.7, 6.7, 6.0, 6.1, 8.3, 8.3,
     10.6, 11.1, 11.1, 11.7, 12.2, 13.3, 14.4, 16.7, 14.4, 13.3, 12.2, 11.7,
     11.1, 10.0, 8.3, 7.8, 7.2, 8.0, 6.7, 7.2, 7.2, 7.8, 10.0, 12.2, 12.8,
     12.8, 13.9, 15.0, 16.7, 16.7, 16.7, 15.6, 13.9, 12.8, 12.2, 10.6, 9.0,
     8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 10.0, 10.6, 11.1, 12.0, 11.7,
     11.1, 13.0, 13.3, 13.0, 11.1, 10.6, 10.6, 10.0, 10.0, 10.0, 9.4, 9.4,
     8.9, 8.3, 9.0, 8.9, 9.4, 9.0, 9.4, 10.6, 11.7, 11.1, 11.7, 12.8, 12.8,
     12.8, 13.0, 11.7, 10.6, 10.0, 10.0, 8.9, 9.4, 7.8, 7.8, 8.3, 7.8, 8.9,
     8.9, 8.9, 9.4, 10.0, 10.0, 10.6, 11.0, 11.1, 11.1, 12.2, 10.6, 10.0, 8.9,
     8.9, 9.0, 8.9, 8.3, 8.9, 8.9, 9.4, 9.4, 9.4, 8.9, 8.9, 8.9, 9.4, 10.0,
     11.1, 11.7, 11.7, 11.7, 11.7, 12.0, 11.7, 11.7, 12.0, 11.7, 11.0, 10.6,
     9.4, 10.0, 8.3, 8.0, 7.2, 5.6, 6.1, 5.6, 6.1, 6.7, 8.0, 10.0, 10.6, 11.1,
     13.3, 12.8, 12.8, 12.2, 11.1, 10.0, 10.0, 10.0, 10.0, 9.4, 8.3]) 


def fit(x, a, b, c, d):
    return a*np.sin(b*x + c) + d

xs = np.linspace(0, 2*np.pi, len(mydata))

guess = (mydata.ptp()/2, 10, 0, mydata.mean())
fitting_parameters, covariance = optimize.curve_fit(fit, xs, mydata, p0=guess)
a, b, c, d = fitting_parameters
print(a, b, c, d)

fig, ax = plt.subplots(nrows=2, sharex=True)
ax[0].plot(xs, mydata)

new_xs = np.linspace(xs.min(), xs.max(), len(mydata)*1.5)
new_ys = fit(new_xs, a, b, c, d)
ax[1].plot(new_xs, new_ys)

plt.xlim(xs.min(), xs.max())
plt.show()

You may be able to improve the fit by choosing a better model function (in place of fit). What to choose is a matter of creativity and intuition guided by a priori knowledge of your problem domain. What is better depends not only on the goodness of the fit, but also on how simple or complex you wish your model to be, and/or on how much predictive power it has when applied to new data sets.