I have some data in python that is unixtime, value:
[(1301672429, 274), (1301672430, 302), (1301672431, 288)...]
Time constantly steps by one second. How might I reduce this data so the timestamp is every second, but the value is the average of the surrounding 10 values?
Fancier rolling averages would be good too, but this data is graphed so it is mostly to smooth out the graph.
Follow up of ( TSQL Rolling Average of Time Groupings after coming to the conclusion that trying to do this in SQL is a route of pain).
The smoothing techniques are the members of time series forecasting methods or algorithms, which use the weighted average of a past observation to predict the future values or forecast the new value. These techniques are well suited for time-series data having fewer deviations with time.
To make time series data more smooth in Pandas, we can use the exponentially weighted window functions and calculate the exponentially weighted average.
Exponential Smoothing Methods are a family of forecasting models. They use weighted averages of past observations to forecast new values. The idea is to give more importance to recent values in the series. Thus, as observations get older in time, the importance of these values get exponentially smaller.
Using http://www.scipy.org/Cookbook/SignalSmooth:
import numpy
def smooth(x,window_len=11,window='hanning'):
if x.ndim != 1:
raise ValueError, "smooth only accepts 1 dimension arrays."
if x.size < window_len:
raise ValueError, "Input vector needs to be bigger than window size."
if window_len<3:
return x
if not window in ['flat', 'hanning', 'hamming', 'bartlett', 'blackman']:
raise ValueError, "Window is on of 'flat', 'hanning', 'hamming', 'bartlett', 'blackman'"
s=numpy.r_[2*x[0]-x[window_len-1::-1],x,2*x[-1]-x[-1:-window_len:-1]]
if window == 'flat': #moving average
w=numpy.ones(window_len,'d')
else:
w=eval('numpy.'+window+'(window_len)')
y=numpy.convolve(w/w.sum(),s,mode='same')
return y[window_len:-window_len+1]
I get what seems to be good results with (Not that I understand the math):
if form_results['smooth']:
a = numpy.array([x[1] for x in results])
smoothed = smooth(a,window_len=21)
results = zip([x[0] for x in results], smoothed)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With