can't open sockets for parallel cluster

Question

I am trying to use the parallel package, and found that makeCluster fails to complete. I've traced the hang to the following line in newPSOCKnode :

con <- socketConnection("localhost", port = port, server = TRUE, 
    blocking = TRUE, open = "a+b", timeout = timeout)

That command stalls (granted the default timeout is a large value). My suspicion is this is due to some "overzealous IT rules" laid down on our work computers, but would welcome any suggestions as to how to trace (and fix) the source of the problem. This is Windows7-64, "Enterprise", R 3.0.1 .

More info: inside debugging session, I set timeout < - 10, but it still hangs -- as though socketConnection is getting trapped somewhere that it can't even check the timeout value.

Here's my dump at the same point as Richie Cotton's data:

Browse[3]> ls.str()
arg :  chr "parallel:::.slaveRSOCK()"
cmd :  chr "\"C:/Users/carl.witthoft/Documents/R/R-3.0.1/bin/x64/Rscript\" -e \"parallel:::.slaveRSOCK()\" MASTER=localhost PORT=11017 OUT="| __truncated__
env :  chr "MASTER=localhost PORT=11017 OUT=/dev/null TIMEOUT=2592000 METHODS=TRUE XDR=TRUE"
machine :  chr "localhost"
manual :  logi FALSE
master :  chr "localhost"
methods :  logi TRUE
options : <environment: 0x000000000ccac6a0> 
outfile :  chr "/dev/null"
port :  int 11017
rank :  int 1
renice :  int NA
rscript :  chr "\"C:/Users/carl.witthoft/Documents/R/R-3.0.1/bin/x64/Rscript\""
timeout :  num 2592000
useXDR :  logi TRUE

So aside from a different port number, I think everything matches up.

Next trick: I opened a shell and ran netsh advfirewall firewall add rule name="Open Port 11017" dir=in action=allow protocol=TCP localport=11017 and got an "OK" response. I ran netstat -a -n and found the following line:

TCP 0.0.0.0:11017 0.0.0.0:0 LISTENING

But running makePSOCKcluster still hangs at the same place.

NEXT: I tried running R from the command line (via cygwin bash), and the error message I get is Error in loadhistory(file) : no history mechanism available Execution halted , after which -C returns me to the R-prompt.

Answer 1

What you're describing is the classic problem with PSOCK clusters: makeCluster hangs. It can hang for dozens of reasons because it has to create all of the processes, called "worker" processes, that will perform the actual work of the "cluster", and that involves starting new R sessions using the Rscript command that will execute the .slaveRSOCK function, which will create a socket connection back to the master and then execute the slaveLoop function where it will eventually execute the tasks sent to it by the master. If anything goes wrong starting any of the worker processes (and trust me: a lot can go wrong), the master will hang while executing socketConnection, waiting for the worker to connect to it even though that worker may have died or never even been created successfully.

For many failure scenarios, using the outfile argument is great because it often reveals the error that causes the worker process to die and thus the master to hang. But if that reveals nothing, I go to manual mode. In manual mode, the master prints the command to start each worker instead of executing the command itself. It's more work, but it gives you complete control, and you can even debug into the workers if you need to.

Here's an example:

> library(parallel)
> cl <- makePSOCKcluster(1, manual=TRUE, outfile='log.txt')
Manually start worker on localhost with
   '/usr/lib/R/bin/Rscript' -e 'parallel:::.slaveRSOCK()' MASTER=localhost
PORT=10187 OUT=log.txt TIMEOUT=2592000 METHODS=TRUE XDR=TRUE 

At this point, your R session is hung because it's executing socketConnection, just as you described. It's now your job to open a new terminal window (command prompt, or whatever), and paste in that Rscript command. As soon as you've executed it, makePSOCKcluster should return since we only requested one worker. Of course, if something goes wrong, it won't return, but if you're lucky, you'll get an error message in your terminal window and you'll have an important clue that will hopefully lead to a solution to your problem. If you're not so lucky, the Rscript command will also hang, and you'll have to dive in even deeper.

To debug the worker, you don't execute the displayed Rscript command because you need an interactive session. Instead, you start an R session with a command such as:

$ R --vanilla --args MASTER=localhost PORT=10187 OUT=log.txt TIMEOUT=2592000 METHODS=TRUE XDR=TRUE

In that R session, you can put a breakpoint on the .slaveRSOCK function and then execute it:

> debug(parallel:::.slaveRSOCK)
> parallel:::.slaveRSOCK()

Now you can start stepping through the code, possibly setting breakpoints on the slaveLoop and makeSOCKmaster functions. In your case, I assume that it will hang trying to create the socket connection, in which case the title of your question will be appropriate.

For more information on this kind of problem, see my answer to a similar question.

UPDATE

Now that this particular problem has been resolved, I can add two tips for debugging makePSOCKcluster problems:

• Check to see if anything in your .Rprofile only works in interactive mode
• On Windows, use the Rterm command rather than Rgui so that you're more likely to see error messages and output from using outfile=''.

Answer 2

Test 1: Does the obvious command work?

library(parallel)    
cluster <- makePSOCKcluster("localhost")    
parSapply(cluster, 1:5, sqrt)
stopCluster(cluster)

Test 2: Is your port blocked?

According to ?makeCluster, the default port is 10187. Check with your network admin to see if that port is open.

Test 3: Do the variables passed in to socketConnection look right?

If I do debugonce(parallel:::newPSOCKnode) and then stepping to through to just before the call to socketConnection, the workspace look this this:

ls.str()
arg :  chr "parallel:::.slaveRSOCK()"
## cmd :  chr "\"C:/PROGRA~1/R/R-215~1.2/bin/x64/Rscript\" -e \"parallel:::.slaveRSOCK()\" MASTER=localhost PORT=10187 OUT=/dev/null TIMEOUT=2"| __truncated__
## env :  chr "MASTER=localhost PORT=10187 OUT=/dev/null TIMEOUT=2592000 METHODS=TRUE XDR=TRUE"
## machine :  chr "localhost"
## manual :  logi FALSE
## master :  chr "localhost"
## methods :  logi TRUE
## options : <environment: 0x0000000010bf2518> 
## outfile :  chr "/dev/null"
## port :  num 10187
## rank :  int 1
## renice :  int NA
## rscript :  chr "\"C:/PROGRA~1/R/R-215~1.2/bin/x64/Rscript\""
## timeout :  num 2592000
## useXDR :  logi TRUE

Are you getting the same things passed in?