Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Merging/combining two dataframes with different frequency time series indexes in Pandas?

Using pandas 0.15.1. Suppose I have the following two dataframes:

daily
2014-11-20 00:00:00 Rain
2014-11-21 00:00:00 Cloudy
2014-11-22 00:00:00 Sunny

.

minutely
2014-11-20 12:45:00     51
2014-11-20 12:46:00     43
2014-11-20 12:47:00     44
...
2014-11-21 12:45:00     44
2014-11-21 12:46:00     46
2014-11-21 12:47:00     48
...
2014-11-22 12:45:00     38
2014-11-22 12:46:00     32
2014-11-22 12:47:00     37

I'd like to combine the two dataframes such that the day values get propagated to each minute row that have the corresponding day.

And since the minute rows do not actually have data at 00:00:00 I do not want that time included in the resulting dataframe. Desired output:

2014-11-20 12:45:00     51  Rain
2014-11-20 12:46:00     43  Rain
2014-11-20 12:47:00     44  Rain
...
2014-11-21 12:45:00     44  Cloudy
2014-11-21 12:46:00     46  Cloudy
2014-11-21 12:47:00     48  Cloudy
...
2014-11-22 12:45:00     38  Sunny
2014-11-22 12:46:00     32  Sunny
2014-11-22 12:47:00     37  Sunny

How can I achieve this? Do I need to use merge, concat, or join?

like image 695
User Avatar asked Nov 22 '14 17:11

User


1 Answers

starting with:

>>> left
                     minutely
2014-11-20 12:45:00        51
2014-11-20 12:46:00        43
2014-11-20 12:47:00        44
2014-11-21 12:45:00        44
2014-11-21 12:46:00        46
2014-11-21 12:47:00        48
2014-11-22 12:45:00        38
2014-11-22 12:46:00        32
2014-11-22 12:47:00        37
>>> right
             daily
2014-11-20    Rain
2014-11-21  Cloudy
2014-11-22   Sunny

you may do:

>>> left['day'] = left.index.date
>>> right.index = right.index.date
>>> left.join(right, on='day', how='left')
                     minutely         day   daily
2014-11-20 12:45:00        51  2014-11-20    Rain
2014-11-20 12:46:00        43  2014-11-20    Rain
2014-11-20 12:47:00        44  2014-11-20    Rain
2014-11-21 12:45:00        44  2014-11-21  Cloudy
2014-11-21 12:46:00        46  2014-11-21  Cloudy
2014-11-21 12:47:00        48  2014-11-21  Cloudy
2014-11-22 12:45:00        38  2014-11-22   Sunny
2014-11-22 12:46:00        32  2014-11-22   Sunny
2014-11-22 12:47:00        37  2014-11-22   Sunny
like image 138
behzad.nouri Avatar answered Oct 24 '22 04:10

behzad.nouri