SparkR Error in sparkR.init(master="local") in RStudio

Question

I have installed the SparkR package from Spark distribution into the R library. I can call the following command and it seems to work properly: library(SparkR)

However, when I try to get the Spark context using the following code,

sc <- sparkR.init(master="local")

It fails after some time with the following message:

Error in sparkR.init(master = "local") :
   JVM is not ready after 10 seconds

I have set JAVA_HOME, and I have a working RStudio where I can access other packages like ggplot2. I don't know why it is not working, and I don't even know where to investigate the issue.

Answer 1

I had the same issue and my spark-submit.cmd file was also not executing from the command line. Following steps worked for me

Go to your environment variables and in the system variables select variable name PATH. Along with other values add c:/Windows/System32/ separated by a semicolon. This made my spark-submit.cmd run from command line and eventually from the Rstudio.

I have realized that we get the above issue only if all the required path values are not specified. Ensure all your path values(R, Rtools) are specified in the environment variables. For instance my Rtools path was c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin

I hope this helps.