Unable to launch SparkR in RStudio

Question

After long and difficult installation process of SparkR i getting into new problems of launching SparkR.

My Settings

R 3.2.0    
RStudio 0.98.1103    
Rtools 3.3    
Spark 1.4.0
Java Version 8
SparkR 1.4.0
Windows 7 SP 1  64 Bit

Now i try to use following code in R:

library(devtools)
library(SparkR)
Sys.setenv(SPARK_MEM="1g")
Sys.setenv(SPARK_HOME="C:/spark-1.4.0")
sc <- sparkR.init(master="local")

I recieve following:

JVM is not ready after 10 seconds

I was also trying to add some system variables like spark path or java path.

Do you have any advices for me to fix that problems.

The next step for me after testing local host would be to start tests on my running hadoop cluster.

Answer 1

I think it was a bug that has now been resolved. Try the following,

Sys.setenv(SPARK_HOME="C:\\spark-1.4.0")

.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))

library("SparkR", lib.loc="C:\\spark-1.4.0\\lib") # The use of \\ is for windows environment.

library(SparkR)

sc=sparkR.init(master="local")

Launching java with spark-submit command C:\spark-1.4.0/bin/spark-submit.cmd sparkr-shell

C:\Users\Ashish\AppData\Local\Temp\RtmpWqFsOB\backend_portbdc329477c6

Hope this helps.

Answer 2

I had the same issue and my spark-submit.cmd file was also not executing from the command line. Following steps worked for me

Go to your environment variables and in the system variables select variable name PATH. Along with other values add c:/Windows/System32/ separated by a semicolon. This made my spark-submit.cmd run from command line and eventually from the Rstudio.

I have realized that we get the above issue only if all the required path values are not specified. Ensure all your path values(R, Rtools) are specified in the environment variables. For instance my Rtools path was c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin

I hope this helps.

Answer 3

That didn't work for me. If anyone has the same problem, try to give execute permissions to c:/sparkpath/bin/spark-submit.cmd.