How do you create a progress bar when using the "foreach()" function in R?

Question

there are some informative posts on how to create a counter for loops in an R program. However, how do you create a similar function when using the parallelized version with "foreach()"?

Answer 1

Edit: After an update to the doSNOW package it has become quite simple to display a nice progress bar when using %dopar% and it works on Linux, Windows and OS X

doSNOW now officially supports progress bars via the .options.snow argument.

library(doSNOW) cl <- makeCluster(2) registerDoSNOW(cl) iterations <- 100 pb <- txtProgressBar(max = iterations, style = 3) progress <- function(n) setTxtProgressBar(pb, n) opts <- list(progress = progress) result <- foreach(i = 1:iterations, .combine = rbind,                    .options.snow = opts) %dopar% {     s <- summary(rnorm(1e6))[3]     return(s) } close(pb) stopCluster(cl)  

Yet another way of tracking progress, if you keep in mind the total number of iterations, is to set .verbose = T as this will print to the console which iterations have been finished.

Previous solution for Linux and OS X

On Ubuntu 14.04 (64 bit) and OS X (El Capitan) the progress bar is displayed even when using %dopar% if in the makeCluster function oufile = "" is set. It does not seem to work under Windows. From the help on makeCluster:

outfile: Where to direct the stdout and stderr connection output from the workers. "" indicates no redirection (which may only be useful for workers on the local machine). Defaults to ‘/dev/null’ (‘nul:’ on Windows).

Example code:

library(foreach) library(doSNOW) cl <- makeCluster(4, outfile="") # number of cores. Notice 'outfile' registerDoSNOW(cl) iterations <- 100 pb <- txtProgressBar(min = 1, max = iterations, style = 3) result <- foreach(i = 1:iterations, .combine = rbind) %dopar%  {       s <- summary(rnorm(1e6))[3]       setTxtProgressBar(pb, i)        return(s) } close(pb) stopCluster(cl)  

This is what the progress bar looks like. It looks a little odd since a new bar is printed for every progression of the bar and because a worker may lag a bit which causes the progress bar to go back and forth occasionally.

Answer 2

This code is a modified version of the doRedis example, and will make a progress bar even when using %dopar% with a parallel backend:

#Load Libraries library(foreach) library(utils) library(iterators) library(doParallel) library(snow)  #Choose number of iterations n <- 1000  #Progress combine function f <- function(){   pb <- txtProgressBar(min=1, max=n-1,style=3)   count <- 0   function(...) {     count <<- count + length(list(...)) - 1     setTxtProgressBar(pb,count)     Sys.sleep(0.01)     flush.console()     c(...)   } }  #Start a cluster cl <- makeCluster(4, type='SOCK') registerDoParallel(cl)  # Run the loop in parallel k <- foreach(i = icount(n), .final=sum, .combine=f()) %dopar% {   log2(i) }  head(k)  #Stop the cluster stopCluster(cl) 

You have to know the number of iterations and the combination function ahead of time.