add columns to data frame using foreach and %dopar%

Question

In Revolution R 2.12.2 on Windows 7 and Ubuntu 64-bit 11.04 I have a data frame with over 100K rows and over 100 columns, and I derive ~5 columns (sqrt, log, log10, etc) for each of the original columns and add them to the same data frame. Without parallelism using foreach and %do%, this works fine, but it's slow. When I try to parallelize it with foreach and %dopar%, it will not access the global environment (to prevent race conditions or something like that), so I cannot modify the data frame because the data frame object is 'not found.'

My question is how can I make this faster? In other words, how to parallelize either the columns or the transformations?

Simplified example:

require(foreach)    
require(doSMP)
w <- startWorkers()
registerDoSMP(w)

transform_features <- function()
{    
    cols<-c(1,2,3,4) # in my real code I select certain columns (not all)

    foreach(thiscol=cols, mydata) %dopar% { 
        name <- names(mydata)[thiscol]
        print(paste('transforming variable ', name))
        mydata[,paste(name, 'sqrt', sep='_')] <<- sqrt(mydata[,thiscol])
            mydata[,paste(name, 'log', sep='_')] <<- log(mydata[,thiscol])
    }
}


n<-10 # I often have 100K-1M rows
mydata <- data.frame(
    a=runif(n,1,100),
    b=runif(n,1,100),
    c=runif(n,1,100),
    d=runif(n,1,100)
    )

ncol(mydata) # 4 columns

transform_features()

ncol(mydata) # if it works, there should be 8

Notice if you change %dopar% to %do% it works fine

Answer 1

Try the := operator in data.table to add the columns by reference. You'll need with=FALSE so you can put the call to paste on the LHS of :=.

See When should I use the := operator in data.table?

Answer 2

Might it be easier if you did something like

n<-10
mydata <- data.frame(
    a=runif(n,1,100),
    b=runif(n,1,100),
    c=runif(n,1,100),
    d=runif(n,1,100)
    )

mydata_sqrt <- sqrt(mydata)  
colnames(mydata_sqrt) <- paste(colnames(mydata), 'sqrt', sep='_')

mydata <- cbind(mydata, mydata_sqrt)

producing something like

> mydata
           a         b         c        d   a_sqrt   b_sqrt   c_sqrt   d_sqrt
1  29.344088 47.232144 57.218271 58.11698 5.417018 6.872565 7.564276 7.623449
2   5.037735 12.282458  3.767464 40.50163 2.244490 3.504634 1.940996 6.364089
3  80.452595 76.756839 62.128892 43.84214 8.969537 8.761098 7.882188 6.621340
4  39.250277 11.488680 38.625132 23.52483 6.265004 3.389496 6.214912 4.850240
5  11.459075  8.126104 29.048527 76.17067 3.385126 2.850632 5.389669 8.727581
6  26.729365 50.140679 49.705432 57.69455 5.170045 7.081008 7.050208 7.595693
7  42.533937  7.481240 59.977556 11.80717 6.521805 2.735186 7.744518 3.436157
8  41.673752 89.043099 68.839051 96.15577 6.455521 9.436265 8.296930 9.805905
9  59.122106 74.308573 69.883037 61.85404 7.689090 8.620242 8.359607 7.864734
10 24.191878 94.059012 46.804937 89.07993 4.918524 9.698403 6.841413 9.438217