KeyError in pandas to_datetime using custom format

Question

The index of my DataFrame (TradeData) is in string format:

In [30]: TradeData.index
Out[30]: Index(['09/30/2013 : 04:14 PM', '09/30/2013 : 03:53 PM', ... ], dtype=object)

And I would like it to be in Datetime. But the conversion does not seem to work:

In [31]: TradeDataIdxd = pd.to_datetime(TradeData.index, format="%m/%d/%Y : %I:%M %p")
Traceback (most recent call last):

File "<ipython-input-31-1191c22cd132>", line 1, in <module>
TradeDataIdxd = pd.to_datetime(TradeData.index, format="%m/%d/%Y : %I:%M %p")

File "C:\WinPython-64bit-3.3.2.3\python-3.3.2.amd64\lib\site-packages\pandas\tseries\tools.py", line 128, in to_datetime
return _convert_listlike(arg, box=box)

File "C:\WinPython-64bit-3.3.2.3\python-3.3.2.amd64\lib\site-packages\pandas\tseries\tools.py", line 104, in _convert_listlike
result = tslib.array_strptime(arg, format)

File "tslib.pyx", line 1137, in pandas.tslib.array_strptime (pandas\tslib.c:18543)

KeyError: 'p'

None of the elements of TradeData.index are 'p'. Any ideas what could be the matter? Thanks in advance.

Answer 1

You can circumvent this to_datetime issue by resetting the index, manipulating the series via map/lambda/strptime, and then finally setting the index again.

In [1058]: TradeData.index
Out[1058]: Index([u'09/30/2013 : 04:14 PM', u'09/30/2013 : 03:53 PM', u'09/30/2013 : 03:53 PM'], dtype=object)

In [1059]: index_name = TradeData.index.name

In [1060]: TradeData = TradeData.reset_index()

In [1061]: TradeData[index_name] = TradeData[index_name].map(lambda x: datetime.strptime(x, "%m/%d/%Y
: %I:%M %p"))

In [1062]: TradeData = TradeData.set_index(index_name)

In [1063]: TradeData.index
Out[1063]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2013-09-30 16:14:00, ..., 2013-09-30 15:53:00]
Length: 3, Freq: None, Timezone: None

Not quite as concise, but it has the same effect. Or, to package it up in a function:

def df_index_to_datetime(df, datetime_format):
    index_name = df.index.name
    df = df.reset_index()
    df[index_name] = df[index_name].map(lambda x: datetime.strptime(x, datetime_format))
    df = df.set_index(index_name)
    return df