Modify usual hpfilter function to ignore na's

Question

I'm a new R user, trying to learn quickly, but I couldn't crack this myself. I work mostly with economic time series – hence, try to maintain my dataset in xts multi-column format, e.g.:

> head(USDATAq)
         tq   ngdp    rgdp  profit
1947 Q1   0  237.2  1770.7    20.7
1947 Q2   1  240.4  1768.0    23.9
1947 Q3   2  244.5  1766.5    23.8
1947 Q4   3  254.3  1793.3    25.5
1948 Q1   4  260.3  1821.8    29.4
1948 Q2   5  267.3  1855.3    31.2

I apply the hpfilter function for filtering. Elsewhere on this site, I found this implementation which uses the coredata function to apply hpfilter to xts objects:

hpfilter <- function(x, lambda=2){
  eye <- diag(length(x))
  dcrossprod <- crossprod(diff(eye, lag=1, d=2))
  coredata(x) <- solve(eye + lambda * dcrossprod, coredata(x))
  return(x)
}

My question is:

How can I modify the function so that it will work with variables having NA observations (at present, it calculates NA for the entire date range if there is any NA)?

I can pass on the dataset as na.omit(USDATAq), which works, but this curtails all variables in dataset to the minimum observations. But, different variables are available until different dates, followed by NA's. I would like to eventually apply the function to every column of the dataset in a loop or mapply, so that the function returns each filtered series using all available observations of that series.

Answer 1

Thanks @ran2. I worked on your suggestion and managed to solve the problem – but, in a rather inelegant way. First of all, I could not get any of the 'apply family' functions to work correctly on an xts object, maintaining its structure. Plain apply with the apply(x, MARGIN=2,..) for column-wise application showed promise, but stalled at the 'coredata' statement. lapply etc yielded mangled lists.

I then went to the for loop. But because the x<-na.omit(x) changes the length of the variable, it cannot replace the original within loop.

> for(i in 1:ncol(USDATAq)) {
+ USDATAq[,i]<-hpfilter(USDATAq[,i])
+ }

Error in NextMethod(.Generic) : number of items to replace is not a multiple of replacement length

So, I had to add unseemly code to hpfilter to ‘merge’ the result back to the original (with NA) and then return the variable. This merging matches the 2 variables by date (hence, length) filling NA’s into the result. Then, this result can replace the original in a loop. In conclusion, I had to modify hpfilter to:

hpfilter <- function(x,lambda=2){
y<-na.omit(x)
eye <- diag(length(y))
coredata(y) <- solve(eye + lambda * crossprod(diff(eye, lag=1, d=2)), coredata(y))
xy<-merge(x,y) 
return(xy[,2])
}

and then use the loop above, to finally get error free results. My knowledge of R is so rudimentary, though, that there probably are easier ways to do this. But, at least, I can proceed now. Thanks to all for pointing me in the right direction. I'd still welcome further corrections to my code above.