I am trying to use data.table to fill missing observations in a large unbalanced multi-dimensional panel that I have. Below is an example of the data with some comments as to what I want:
mydat <- structure(list(fund = c(1, 1, 1, 1, 2, 2, 2, 3, 3), holdingid = c(10,
10, 11, 11, 15, 15, 14, 20, 20), yearqtr = structure(c(2000,
2000.5, 2000, 2000.25, 2000, 2000.75, 2000.25, 2000.25, 2000.5
), class = "yearqtr"), shares = c(20, 25, 30, 30, 34, 34, 4,
8, 10)), .Names = c("fund", "holdingid", "yearqtr", "shares"), row.names = c(NA,
-9L), class = "data.frame")
allqtrs <- structure(c(2000, 2000.25, 2000.5, 2000.75), class = "yearqtr")
#note that there are missing yearqtrs for some fund-holding series
#if a fund-holding series is missing an observation I want to create
#that fund-holding-quarter and fill it with NA
I am trying to balance the panel with the end goal of lagging (or differencing) each fund-holdingid series properly (in the sense that the irregularity of the data is taken care of). Obviously I could use zooreg for each fund-holdingid group and lag using this, but my data is >20 million rows and I am trying to write a more efficient solution. Thanks for the help.
EDIT To clarify a bit more I am looking to do something similar to what can be done with Oracle SQL's partition by outer joins as demonstrated here http://st-curriculum.oracle.com/obe/db/10g/r2/prod/bidw/outerjoin/outerjoin_otn.htm
EDIT-2 I used a lot of time series terms in the description. To be more specific, for each fund-holding pair I want to have an observation for every yearqtr in allqtrs. So in this case since there are 3 funds with 3, 2, and 1 holdings respectively, there should be (2+2+1)*4 total rows in the output since there are 4 possible quarters for each fund-holding. Another important point is that the holdingids are very diverse. Something like expand.grid(unique(fund),unique(holdingid),unique(allqtrs)) will lead to far too many rows since each fund will only have a small subset of the possible holdings.
It's always good to post the answer you're expecting as well to avoid any ambiguity/miscommunication. Is this what you're looking for?
require(data.table)
dt <- as.data.table(mydat)
setkey(dt, "yearqtr")
dt[, .SD[J(allqtrs)], by = list(fund, holdingid)]
# fund holdingid yearqtr shares
# 1: 1 10 2000.00 20
# 2: 1 10 2000.25 NA
# 3: 1 10 2000.50 25
# 4: 1 10 2000.75 NA
# 5: 1 11 2000.00 30
# 6: 1 11 2000.25 30
# 7: 1 11 2000.50 NA
# 8: 1 11 2000.75 NA
# 9: 2 15 2000.00 34
# 10: 2 15 2000.25 NA
# 11: 2 15 2000.50 NA
# 12: 2 15 2000.75 34
# 13: 2 14 2000.00 NA
# 14: 2 14 2000.25 4
# 15: 2 14 2000.50 NA
# 16: 2 14 2000.75 NA
# 17: 3 20 2000.00 NA
# 18: 3 20 2000.25 8
# 19: 3 20 2000.50 10
# 20: 3 20 2000.75 NA
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With