Pandas: Quickly add variable number of months to a timestamp column

Question

Here's the setup:

I have two (integer-indexed) columns, start and month_delta. start has timestamps (its internal type is np.datetime64[ns]) and month_delta is integers.

I want to quickly produce the column that consists of the each datetime in start, offset by the corresponding number of months in month_delta. How do I do this?

Things I've tried that don't work:

• apply is too slow.
• You can't add a series of DateOffset objects to a series of datetime64[ns] dtype (or a DatetimeIndex).
• You can't use a Series of timedelta64 objects either; Pandas silently converts month-based timedeltas to nanosecond-based timedeltas that are ~30 days long. (Yikes! What happened to not failing silently?)

Currently I'm iterating over all different values of month_delta and doing a tshift by that amount on the relevant part of a DatetimeIndex I created, but this is a horrible kludge:

new_dates = pd.Series(pd.Timestamp.now(), index=start.index)
date_index = pd.DatetimeIndex(start)
for i in xrange(month_delta.max()):
    mask = (month_delta == i)
    cur_dates = pd.Series(index=date_index[mask]).tshift(i, freq='M').index
    new_dates[mask] = cur_dates

Yuck! Any suggestions?

Answer 1

Here is a way to do it (by adding NumPy datetime64s with timedelta64s) without calling apply:

import pandas as pd
import numpy as np
np.random.seed(1)

def combine64(years, months=1, days=1, weeks=None, hours=None, minutes=None,
              seconds=None, milliseconds=None, microseconds=None, nanoseconds=None):
    years = np.asarray(years) - 1970
    months = np.asarray(months) - 1
    days = np.asarray(days) - 1
    types = ('<M8[Y]', '<m8[M]', '<m8[D]', '<m8[W]', '<m8[h]',
             '<m8[m]', '<m8[s]', '<m8[ms]', '<m8[us]', '<m8[ns]')
    vals = (years, months, days, weeks, hours, minutes, seconds,
            milliseconds, microseconds, nanoseconds)
    return sum(np.asarray(v, dtype=t) for t, v in zip(types, vals)
               if v is not None)

def year(dates):
    "Return an array of the years given an array of datetime64s"
    return dates.astype('M8[Y]').astype('i8') + 1970

def month(dates):
    "Return an array of the months given an array of datetime64s"
    return dates.astype('M8[M]').astype('i8') % 12 + 1

def day(dates):
    "Return an array of the days of the month given an array of datetime64s"
    return (dates - dates.astype('M8[M]')) / np.timedelta64(1, 'D') + 1

N = 10
df = pd.DataFrame({
   'start': pd.date_range('2000-1-25', periods=N, freq='D'),
   'months': np.random.randint(12, size=N)})
start = df['start'].values
df['new_date'] = combine64(year(start), months=month(start) + df['months'], 
                           days=day(start))

print(df)

yields

   months      start   new_date
0       5 2000-01-25 2000-06-25
1      11 2000-01-26 2000-12-26
2       8 2000-01-27 2000-09-27
3       9 2000-01-28 2000-10-28
4      11 2000-01-29 2000-12-29
5       5 2000-01-30 2000-06-30
6       0 2000-01-31 2000-01-31
7       0 2000-02-01 2000-02-01
8       1 2000-02-02 2000-03-02
9       7 2000-02-03 2000-09-03