Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python Smooth Time Series Data

I have some data in python that is unixtime, value:

[(1301672429, 274), (1301672430, 302), (1301672431, 288)...]

Time constantly steps by one second. How might I reduce this data so the timestamp is every second, but the value is the average of the surrounding 10 values?

Fancier rolling averages would be good too, but this data is graphed so it is mostly to smooth out the graph.

Follow up of ( TSQL Rolling Average of Time Groupings after coming to the conclusion that trying to do this in SQL is a route of pain).

like image 911
Kyle Brandt Avatar asked Apr 01 '11 15:04

Kyle Brandt


People also ask

How do you smooth data on a time series?

The smoothing techniques are the members of time series forecasting methods or algorithms, which use the weighted average of a past observation to predict the future values or forecast the new value. These techniques are well suited for time-series data having fewer deviations with time.

How do you smooth a Pandas time series?

To make time series data more smooth in Pandas, we can use the exponentially weighted window functions and calculate the exponentially weighted average.

How smoothing methods can be used to forecast a time series?

Exponential Smoothing Methods are a family of forecasting models. They use weighted averages of past observations to forecast new values. The idea is to give more importance to recent values in the series. Thus, as observations get older in time, the importance of these values get exponentially smaller.


1 Answers

Using http://www.scipy.org/Cookbook/SignalSmooth:

import numpy
def smooth(x,window_len=11,window='hanning'):
        if x.ndim != 1:
                raise ValueError, "smooth only accepts 1 dimension arrays."
        if x.size < window_len:
                raise ValueError, "Input vector needs to be bigger than window size."
        if window_len<3:
                return x
        if not window in ['flat', 'hanning', 'hamming', 'bartlett', 'blackman']:
                raise ValueError, "Window is on of 'flat', 'hanning', 'hamming', 'bartlett', 'blackman'"
        s=numpy.r_[2*x[0]-x[window_len-1::-1],x,2*x[-1]-x[-1:-window_len:-1]]
        if window == 'flat': #moving average
                w=numpy.ones(window_len,'d')
        else:  
                w=eval('numpy.'+window+'(window_len)')
        y=numpy.convolve(w/w.sum(),s,mode='same')
        return y[window_len:-window_len+1]

I get what seems to be good results with (Not that I understand the math):

   if form_results['smooth']:
            a = numpy.array([x[1] for x in results])
            smoothed = smooth(a,window_len=21)
            results = zip([x[0] for x in results], smoothed)
like image 131
Kyle Brandt Avatar answered Oct 06 '22 07:10

Kyle Brandt