Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

data.table outer join by group

Tags:

r

data.table

I am trying to use data.table to fill missing observations in a large unbalanced multi-dimensional panel that I have. Below is an example of the data with some comments as to what I want:

mydat <- structure(list(fund = c(1, 1, 1, 1, 2, 2, 2, 3, 3), holdingid = c(10,                                                                                                                                        
 10, 11, 11, 15, 15, 14, 20, 20), yearqtr = structure(c(2000,                                                                                                                                                 
 2000.5, 2000, 2000.25, 2000, 2000.75, 2000.25, 2000.25, 2000.5                                                                                                                                               
 ), class = "yearqtr"), shares = c(20, 25, 30, 30, 34, 34, 4,                                                                                                                                                 
 8, 10)), .Names = c("fund", "holdingid", "yearqtr", "shares"), row.names = c(NA,                                                                                                                             
 -9L), class = "data.frame")

allqtrs <- structure(c(2000, 2000.25, 2000.5, 2000.75), class = "yearqtr")

#note that there are missing yearqtrs for some fund-holding series
#if a fund-holding series is missing an observation I want to create 
#that fund-holding-quarter and fill it with NA

I am trying to balance the panel with the end goal of lagging (or differencing) each fund-holdingid series properly (in the sense that the irregularity of the data is taken care of). Obviously I could use zooreg for each fund-holdingid group and lag using this, but my data is >20 million rows and I am trying to write a more efficient solution. Thanks for the help.

EDIT To clarify a bit more I am looking to do something similar to what can be done with Oracle SQL's partition by outer joins as demonstrated here http://st-curriculum.oracle.com/obe/db/10g/r2/prod/bidw/outerjoin/outerjoin_otn.htm

EDIT-2 I used a lot of time series terms in the description. To be more specific, for each fund-holding pair I want to have an observation for every yearqtr in allqtrs. So in this case since there are 3 funds with 3, 2, and 1 holdings respectively, there should be (2+2+1)*4 total rows in the output since there are 4 possible quarters for each fund-holding. Another important point is that the holdingids are very diverse. Something like expand.grid(unique(fund),unique(holdingid),unique(allqtrs)) will lead to far too many rows since each fund will only have a small subset of the possible holdings.

like image 948
Rob Richmond Avatar asked Jun 02 '13 01:06

Rob Richmond


1 Answers

It's always good to post the answer you're expecting as well to avoid any ambiguity/miscommunication. Is this what you're looking for?

require(data.table)
dt <- as.data.table(mydat)
setkey(dt, "yearqtr")
dt[, .SD[J(allqtrs)], by = list(fund, holdingid)]
#     fund holdingid yearqtr shares
#  1:    1        10 2000.00     20
#  2:    1        10 2000.25     NA
#  3:    1        10 2000.50     25
#  4:    1        10 2000.75     NA
#  5:    1        11 2000.00     30
#  6:    1        11 2000.25     30
#  7:    1        11 2000.50     NA
#  8:    1        11 2000.75     NA
#  9:    2        15 2000.00     34
# 10:    2        15 2000.25     NA
# 11:    2        15 2000.50     NA
# 12:    2        15 2000.75     34
# 13:    2        14 2000.00     NA
# 14:    2        14 2000.25      4
# 15:    2        14 2000.50     NA
# 16:    2        14 2000.75     NA
# 17:    3        20 2000.00     NA
# 18:    3        20 2000.25      8
# 19:    3        20 2000.50     10
# 20:    3        20 2000.75     NA
like image 100
Arun Avatar answered Sep 30 '22 14:09

Arun