How to efficiently resample a DatetimeIndex

Question

Pandas has a resample method on a series/dataframe but there seems no way to resample a DatetimeIndex on its own?

Concretely, I have a daily Datetimeindex with possibly missing dates and I want to resample it at an hourly freq but only including days which are in the original daily index.

Is there a better way than my attempt below?

In [56]: daily_index = pd.period_range('01-Jan-2017', '31-Jan-2017', freq='B').asfreq('D')

In [57]: daily_index
Out[57]: 
PeriodIndex(['2017-01-02', '2017-01-03', '2017-01-04', '2017-01-05',
             '2017-01-06', '2017-01-09', '2017-01-10', '2017-01-11',
             '2017-01-12', '2017-01-13', '2017-01-16', '2017-01-17',
             '2017-01-18', '2017-01-19', '2017-01-20', '2017-01-23',
             '2017-01-24', '2017-01-25', '2017-01-26', '2017-01-27',
             '2017-01-30', '2017-01-31'],
            dtype='int64', freq='D')

In [58]: daily_index.shape
Out[58]: (22,)

In [59]: hourly_index = pd.DatetimeIndex([]).union_many(
    ...:     pd.date_range(day.to_timestamp('H','S'), day.to_timestamp('H','E'), freq='H')
    ...:     for day in daily_index
    ...: )

In [60]: hourly_index
Out[60]: 
DatetimeIndex(['2017-01-02 00:00:00', '2017-01-02 01:00:00',
               '2017-01-02 02:00:00', '2017-01-02 03:00:00',
               '2017-01-02 04:00:00', '2017-01-02 05:00:00',
               '2017-01-02 06:00:00', '2017-01-02 07:00:00',
               '2017-01-02 08:00:00', '2017-01-02 09:00:00',
               ...
               '2017-01-31 14:00:00', '2017-01-31 15:00:00',
               '2017-01-31 16:00:00', '2017-01-31 17:00:00',
               '2017-01-31 18:00:00', '2017-01-31 19:00:00',
               '2017-01-31 20:00:00', '2017-01-31 21:00:00',
               '2017-01-31 22:00:00', '2017-01-31 23:00:00'],
              dtype='datetime64[ns]', length=528, freq=None)

In [61]: 22*24
Out[61]: 528

In [62]: %%timeit
    ...: hourly_index = pd.DatetimeIndex([]).union_many(
    ...:     pd.date_range(day.to_timestamp('H','S'), day.to_timestamp('H','E'), freq='H')
    ...:     for day in daily_index
    ...: )
100 loops, best of 3: 13.7 ms per loop

UPDATE:

I went with a slight variation of @NTAWolf's answer which has similar performance but won't reorder the input dates in case they're not sorted

def resample_index(index, freq):
    """Resamples each day in the daily `index` to the specified `freq`.

    Parameters
    ----------
    index : pd.DatetimeIndex
        The daily-frequency index to resample
    freq : str
        A pandas frequency string which should be higher than daily

    Returns
    -------
    pd.DatetimeIndex
        The resampled index

    """
    assert isinstance(index, pd.DatetimeIndex)
    start_date = index.min()
    end_date = index.max() + pd.DateOffset(days=1)
    resampled_index = pd.date_range(start_date, end_date, freq=freq)[:-1]
    series = pd.Series(resampled_index, resampled_index.floor('D'))
    return pd.DatetimeIndex(series.loc[index].values)

In [184]: %%timeit
     ...: hourly_index3 = pd.date_range(daily_index.start_time.min(), 
     ...:                               daily_index.end_time.max() + 1, 
     ...:                               normalize=True, freq='H')
     ...: hourly_index3 = hourly_index3[hourly_index3.floor('D').isin(daily_index.start_time)]
100 loops, best of 3: 2.97 ms per loop

In [185]: %timeit resample_index(daily_index.to_timestamp('D','S'), freq='H')
100 loops, best of 3: 2.93 ms per loop

Answer 1

|             Method              |  Time   | Relative |
|---------------------------------|---------|----------|
| OP's updated approach           | 1.31 ms |  17.6 %  |
| Generate daterange, np.in1d     | 1.75 ms |  23.5 %  |
| Generate daterange, Series.isin | 1.90 ms |  25.5 %  |
| Resample with dummy Series      | 4.37 ms |  58.7 %  |
| OP's initial approach           | 7.45 ms | 100.0 %  |

Update 2: Generate daterange, np.in1d

Again, @IanS inspired more optimisation! This is a little less readable, but a bit faster:

%%timeit -r 10
hourly_index4 = pd.date_range(daily_index.start_time.min(), 
                              daily_index.end_time.max() + pd.DateOffset(days=1), 
                              normalize=True, freq='H')
overlap = np.in1d(np.array(hourly_index4.values, dtype='datetime64[D]'),
                  np.array(daily_index.start_time.values, dtype='datetime64[D]'))
hourly_index4 = hourly_index4[overlap]

1000 loops, best of 10: 1.75 ms per loop

Here, speedups are gained by converting the values of both Series to the same numpy datetime kind (flooring hourly_index in the process). Passing .values to numpy sped it up a little bit.

Update 1: Generate daterange, Series.isin

Faster approach than the initial bid, inspired by @IanS's approach: Generate daterange for the complete range of dates in your data, per hour, and select only those entries that match existing dates in your data:

%%timeit
hourly_index3 = pd.date_range(daily_index.start_time.min(), 
                              # The following line should use 
                              # +pd.DateOffset(days=1) in place of +1
                              # but is left as is to show the option.
                              daily_index.end_time.max() + 1, 
                              normalize=True, freq='H')
hourly_index3 = hourly_index3[hourly_index3.floor('D').isin(daily_index.start_time)]

100 loops, best of 3: 1.9 ms per loop

which cuts off about 75% processing time.

Original answer: Resample with dummy Series

Using a dummy series, you can avoid the looping. On my computer, it cuts off about 40% of the run time.

I get the following time for your approach:

In [14]: %%timeit -o -r 10
   ....: hourly_index = pd.DatetimeIndex([]).union_many(   
   ....:     pd.date_range(day.to_timestamp('H','S'), day.to_timestamp('H','E'), freq='H')
   ....:     for day in daily_index
   ....: )
   ....: 
100 loops, best of 10: 7.45 ms per loop

And for the faster approach:

In [13]: %%timeit -o -r 10
s = pd.Series(0, index=daily_index)
s = s.resample('H').last()
s = s[s.index.start_time.floor('D').isin(daily_index.start_time)]
hourly_index2 = s.index.start_time
   ....: 
100 loops, best of 10: 4.37 ms per loop

Note that we don't really care about the value put into the series; here I just default to int.

The expression s.index.start_time.floor('D').isin(daily_index.start_time) gives us a boolean vector for which values in s.index that match days in daily_index.