How to assign fixed memory size to a variable in R

Question

I have a variable which grows in size from few MB to 3 GB in a loop. and I get out of memory error. I've tried some of solutions like increasing the amount of memory R can use,also using rm() and gc(). I have thought if it maybe solved if I assign 3GB to this variable at first. Now:

1. Is it possible in R?

2. If so, will it improve speed?

3. Is it probable that solve out of memory error

I have a 64-bit Windows 7 OS. my code is more than thousand lines. but key lines are

1.getting data from an access file by odbcConnectAccess2007 and sqlFetch functions and putting the table in a temp variable

2.merging the Master Variable with temp variable

Answer 1

Without seeing specific code, it's hard to know what would help. But if you're calling rbind/cbind/merge within a for loop, that is extremely inefficient. What you can do is throw everything into a list and then use do.call at the end. Compare:

data_list <- list(); 
length(data_list) <- 2000

for(i in 1:2000) {
    data_list[[i]] <- data.frame(matrix(runif(11*10), ncol=11, nrow=10))
}


sequentialRbind<-function() {
    res <- data_list[[1]]
    for(i in 2:length(data_list)) {
        res <- rbind(res, data_list[[i]])
    }
    return(res)
}

> system.time(res1 <- do.call(rbind,data_list))
   user  system elapsed 
   0.78    0.00    0.78 
> 
> system.time(res2 <- Reduce(rbind,data_list))
   user  system elapsed 
   8.24    0.00    8.27 
> 
> system.time(res3 <- sequentialRbind())
   user  system elapsed 
   8.25    0.00    8.27