Pandas: how to use between_time with milliseconds?

Question

Consider this:

import pandas as pd
import numpy as np

idx2=[pd.to_datetime('2016-08-31 22:08:12.000'), 
     pd.to_datetime('2016-08-31 22:08:12.200'),
     pd.to_datetime('2016-08-31 22:08:12.400')]

test=pd.DataFrame({'value':[1,1,3], 'groups' : ['A',np.NaN,'A']},index=idx2)
    test
Out[27]: 
                        groups  value
2016-08-31 22:08:12.000      A      1
2016-08-31 22:08:12.200    NaN      1
2016-08-31 22:08:12.400      A      3

I need to only keep data between 22:08:12.200 and 22:08:12.400, so I naturally use between_time:

test.between_time('22:08:12.200','22:08:12.400')

gives

ValueError: Cannot convert arg ['22:08:12.200'] to a time

What is wrong here? How can I slice my dataframe based on time with millisecond information?

Answer 1

I am not sure why the direct string does not work, but it looks like something to do with a time conversion from the datetime which came from the string. But you can workaround with an explicit conversion to time as:

Code:

test.between_time(*pd.to_datetime(['22:08:12.200', '22:08:12.400']).time)

Test Code:

import pandas as pd
import numpy as np

idx2 = [
    pd.to_datetime('2016-08-31 22:08:12.000'),
    pd.to_datetime('2016-08-31 22:08:12.200'),
    pd.to_datetime('2016-08-31 22:08:12.400')]

test = pd.DataFrame(
    {'value': [1, 1, 3], 'groups': ['A', np.NaN, 'A']}, index=idx2)

print(test.between_time(
    *pd.to_datetime(['22:08:12.200', '22:08:12.400']).time))

Results:

                        groups  value
2016-08-31 22:08:12.200    NaN      1
2016-08-31 22:08:12.400      A      3

Answer 2

you can use standard datetime :

test.between_time(datetime.time(22,8,12,200000),datetime.time(22,8,12,400000),include_start=True,include_end=True)