putting `mclapply` results back onto data.frame

Question

I have a very large data.frame that I want to apply a fairly complicated function to, calculating a new column. I want to do it in parallel. This is similar to the question posted over on the r listserve, but the first answer is wrong and the second is unhelpful.

I've gotten everything figured out thanks to the parallel package, except how to put the output back onto the data frame. Here's a MWE that shows what I've got:

library(parallel)

# Example Data
data <- data.frame(a = rnorm(200), b = rnorm(200),  
                   group = sample(letters, 200, replace = TRUE))

# Break into list
datagroup <- split(data, factor(data$group))

# execute on each element in parallel
options(mc.cores = detectCores())
output <- mclapply(datagroup, function(x) x$a*x$b)

The result in output is a list of numeric vectors. I need them in a column that I can append to data. I've been looking along the lines of do.call(cbind, ...), but I have two lists with the same names, not a single list that I'm joining. melt(output) gets me a single vector, but its rows are not in the same order as data.

Answer 1

Converting from comment to answer..

This seems to work:

data <- 
  do.call(
    rbind, mclapply(
      split(data, data$group), 
       function(x){
         z <- x$a*x$b
         x <- as.data.frame(cbind(x, newcol = z))
         return(x)
         }))
rownames(data) <- seq_len(nrow(data))
head(data)
#           a          b group      newcol
#1 -0.6482428  1.8136254     a -1.17566963
#2  0.4397603  1.3859759     a  0.60949714
#3 -0.6426944  1.5086339     a -0.96959055
#4 -1.2913493 -2.3984527     a  3.09724030
#5  0.2260140  0.1107935     a  0.02504087
#6  2.1555370 -0.7858066     a -1.69383520

---

Since you are working with a "very large" data.frame (how large roughly?), have you considered using either dplyr or data.table for what you do? For a large data set, performance may be even better with one of these than with mclapply. The equivalent would be:

library(dplyr)
data %>%
  group_by(group) %>%
  mutate(newcol = a * b)

library(data.table) 
setDT(data)[, newcol := a*b, by=group]