Normalising a data set with irregular intervals in Python

Question

I have some data in Python that eventually gets plotted on a graph (Highcharts) the issue is that intervals between data points on the x axis (time) are not regular. While accurate this makes the graph seem a bit erratic, visually.

The data is in the following format:

data = {"points": [[1335360000, 1335361920, 93374739787], [1335361920, 1335447840, 11738851087.0]......]} 

That is: timestamp from, timestamp to, value

What I need to do is modify the data so that it normalises to the lowest frequency/longest time interval so that it will appear consistant when plotted on a graph.

Are there any ideas on the most efficient way to do this?

Update

I can't really use any 3rd party libaries in this situation.

Graphs usually are rendered like this, appearing more jagged where the data points are closer together:

Answer 1

If you haven't tried it, you may find the pandas library useful for transforming irregular to regular time-series (and other types of data-manipulation-jui-jitsu, in general). It's efficient to program with (clean, reusable idioms once you learn them) and fast at runtime (cython-optimized).

To give you a taste, here are a few pandas examples based on the data format you described.

1. Read data into a pandas.DataFrame. (A DataFrame acts like a dict of columns, where the values are numpy arrays.)

   In [33]: df = pandas.DataFrame(data['points'], columns=['from', 'to', 'value'])
   
   In [34]: df
   Out[34]: 
            from          to  value
   0  1335360000  1335360004      3
   1  1335360004  1335360008     32
   2  1335360008  1335360009      4
   3  1335360009  1335360011     36
   4  1335360011  1335360014     38
   

2. Convert existing columns and add derived columns

   In [46]: utcfromtimestamp = datetime.datetime.utcfromtimestamp
   
   In [47]: df['from'] = df['from'].map(utcfromtimestamp)
   
   In [48]: df['to'] = df['to'].map(utcfromtimestamp)
   
   In [49]: df['delta'] = [x.total_seconds() for x in (df['to'] - df['from'])]
   
   In [50]: df['avg/s'] = df['value'] / df['delta']
   
   In [51]: df
   Out[51]: 
                     from                   to  value  delta      avg/s
   0  2012-04-25 13:20:00  2012-04-25 13:20:04      3      4   0.750000
   1  2012-04-25 13:20:04  2012-04-25 13:20:08     32      4   8.000000
   2  2012-04-25 13:20:08  2012-04-25 13:20:09      4      1   4.000000
   3  2012-04-25 13:20:09  2012-04-25 13:20:11     36      2  18.000000
   4  2012-04-25 13:20:11  2012-04-25 13:20:14     38      3  12.666667
   

3. Group and select information to plot

   In [78]: df.groupby('from')['avg/s'].mean()
   Out[78]: 
   from
   2012-04-25 13:20:00     0.750000
   2012-04-25 13:20:04     8.000000
   2012-04-25 13:20:08     4.000000
   2012-04-25 13:20:09    18.000000
   2012-04-25 13:20:11    12.666667
   Name: avg/s
   

See this link for information on up- or down-sampling time-series. The next release (0.8), still in development, is slated to provide even cleaner methods to resample a time-series.

Answer 2

I suppose you could do some form of curve fitting (least squares or whatever), but maybe you should just stick with the irregular intervals for the sake of accuracy?

If you turn it into a line graph, you'll probably be fine with your original data.