Parallel processing in R with H2O

Question

I am setting up a piece of code to parallel processes some computations for N groups in my data using foreach.

I have a computation that involves a call to h2o.gbm.

In my current, sequential set-up, I use up to about 70% of my RAM.

How do I correctly set-up my h2o.init() within the parallel piece of code? I am afraid that I might run out of RAM when I use multiple cores.

My Windows 10 machine has 12 cores and 128GB of RAM.

Would something like this pseudo-code work?

library(foreach)
library(doParallel)

#setup parallel backend to use 12 processors
cl<-makeCluster(12)
registerDoParallel(cl)

#loop
df4 <-foreach(i = as.numeric(seq(1,999)), .combine=rbind) %dopar% {
  df4 <- data.frame()
  #bunch of computations
  h2o.init(nthreads=1, max_mem_size="10G")
  gbm <- h2o.gbm(train_some_model)
  df4 <- data.frame(someoutput)
   }

fwrite(df4, append=TRUE)

stopCluster(cl)

Answer 1

The way your code is currently set up won't be the best option. I understand what you are trying to do -- execute a bunch of GBMs in parallel (each on a single core H2O cluster), so you can maximize the CPU usage across the 12 cores on your machine. However, what your code will do is try to run all the GBMs in your foreach loop in parallel on the same single-core H2O cluster. You can only connect to one H2O cluster at a time from a single R instance, however the foreach loop will create a new R instance.

Unlike most machine learning algos in R, the H2O algos are all multi-core enabled so the training process will already be parallelized at the algorithm level, without the need for a parallel R package like foreach.

You have a few options (#1 or #3 is probably best):

1. Set h2o.init(nthreads = -1) at the top of your script to use all 12 of your cores. Change the foreach() loop to a regular loop and train each GBM (on a different data partition) sequentially. Although the different GBMs are trained sequentially, each single GBM will be fully parallelized across the H2O cluster.
2. Set h2o.init(nthreads = -1) at the top of your script, but keep your foreach() loop. This should run all your GBMs at once, with each GBM parallelized across all cores. This could overwhelm the H2O cluster a bit (this is not really how H2O is meant to be used) and could be a bit slower than #1, but it's hard to say without knowing the size of your data and the number of partitions of you want to train on. If you are already using 70% of your RAM for a single GBM, then this might not be the best option.
3. You can update your code to do the following (which most closely resembles your original script). This will preserve your foreach loop, creating a new 1-core H2O cluster at a different port on your machine. See below.

Updated R code example which uses the iris dataset and returns the predicted class for iris as a data.frame:

library(foreach)
library(doParallel)
library(h2o)
h2o.shutdown(prompt = FALSE)

#setup parallel backend to use 12 processors
cl <- makeCluster(12)
registerDoParallel(cl)

#loop
df4 <- foreach(i = seq(20), .combine=rbind) %dopar% {
  library(h2o)
  port <- 54321 + 3*i
  print(paste0("http://localhost:", port))
  h2o.init(nthreads = 1, max_mem_size = "1G", port = port)
  df4 <- data.frame()
  data(iris)
  data <- as.h2o(iris)
  ss <- h2o.splitFrame(data)
  gbm <- h2o.gbm(x = 1:4, y = "Species", training_frame = ss[[1]])
  df4 <- as.data.frame(h2o.predict(gbm, ss[[2]]))[,1]
}

In order to judge which option is best, I would try running this on a few data partitions (maybe 10-100) to see which approach seems to scale the best. If your training data is small, it's possible that #3 will be faster than #1, but overall, I'd say #1 is probably the most scalable/stable solution.

Answer 2

Following Erin LeDell's answer, I just wanted to add that in many cases a decent practical solution can be something in between #1 and #3. To increase CPU utilization and still save RAM you can use multiple H2O instances in parallel, but they each can use multiple cores without much performance loss relative to running more instances with only one core.

I ran an experiment using a relatively small 40MB dataset (240K rows, 22 columns) on a 36 core server.

• Case 1: Use all 36 cores (nthreads=36) to estimate 120 GBM models (with default hyper-parameters) sequentially.

• Case 2: Use foreach to run 4 H2O instances on this machine, each using 9 cores to estimate 30 GBM default models sequentially (total = 120 estimations).

• Case 3: Use foreach to run 12 H2O instances on this machine, each using 3 cores to estimate 10 GBM default models sequentially (total = 120 estimations).

Using 36 cores estimating a single GBM model on this dataset is very inefficient. CPU utilization in Case 1 is jumping around a lot, but is on average below 50%. So there is definitely something to gain using more than one H2O instance at a time.

• Runtime Case 1: 264 seconds
• Runtime Case 2: 132 seconds
• Runtime Case 3: 130 seconds

Given the small improvement from 4 to 12 H2O instances, I did not even run 36 H2O instances each using one core in parallel.