Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

R and snow on amazon EC2 using starcluster

I'm trying to run analysis in parrallel in R on an AWS EC2 cluster. I am using starcluster to setup and manage the EC2 cluster, and am trying to use snow and foreach in R. To start off, I have 2 nodes in the cluster, 1 master and 1 worker.

starcluster start mycluster 
starcluster listinstances

-----------------------------------------
mycluster (security group: @sc-mycluster)
-----------------------------------------
....
Cluster nodes:
     master running i-xxxxxxxxx masterIP.compute-1.amazonaws.com
    node001 running i-xxxxxxxxx node001IP.compute-1.amazonaws.com
Total nodes: 2

starcluster sshmaster mycluster

I then start R and load the snow package and try to create a cluster object.

R
library("snow")

cl = makeCluster(c("masterIP.compute-1.amazonaws.com", "node001IP.compute-1.amazonaws.com"), type = "SOCK")

This, however, gives me the following error message:

The authenticity of host 'masterIP.compute-1.amazonaws.com (xx.xxx.xx.xx)' can't be established.
ECDSA key fingerprint is xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'masterIP.compute-1.amazonaws.com,xx.xxx.xx.xx' (ECDSA) to the list of known hosts.
Permission denied (publickey).

So I tried copying my ssh key (keyname.rsa to be specific) to the .ssh file on EC2 and trying again. That still didn't work; I received the same Permission denied (publickey). error. It was my thought that starcluster handled the setup of ssh and communication between nodes, so I'm a little confused as to why I'm not able to set this up. I also tried to just add node001, so cl = makeCluster(c("node001IP.compute-1.amazonaws.com"), type = "SOCK"), but the same error occurs.

like image 777
user1074057 Avatar asked Jan 31 '13 23:01

user1074057


1 Answers

It turns out, after much tinkering, that all that was needed was an update to R version 2.15. The command cl = makeCluster(c("masterIP.compute-1.amazonaws.com", "node001IP.compute-1.amazonaws.com"), type = "SOCK") worked perfectly after that.

like image 139
user1074057 Avatar answered Sep 23 '22 15:09

user1074057